1 Monitoring

Xrootd provides two types of monitoring: 1) summary monitoring and 2) detail monitoring. Summary monitoring is controlled by the xrd.report directive while detail monitoring is controlled by the xrootd.monitor directive.

In order to provide real-time information with minimal impact, monitor data is sent as UDP messages. Each directive specifies what information is to be sent as well as the destinations. Because UDP is used, information is sent whether or not the receiving host is listening for the records. You should not activate monitoring if you do not activate the receiving end, as well.

Summary monitoring is suitable for providing a broad over-view of an xrootd cluster. The information is typically rendered by agents such as Ganglia or Mona Lisa, among others.

Detail monitoring is suitable for deep analysis of access and usage patterns of an xrootd cluster. Since such information is necessarily complex, specialized renderers must be used.

The following sections describe the data formats of each monitoring stream.

2 Summary Monitoring Data Format

The xrd.report directive specifies the parameters as well as the hosts that are to receive the summary information, Summary records are sent as UDP datagrams. Therefore, the information is sent whether or not the receiving host is enabled for the records. Summary information is formatted as an XML record and is described in the following sections. When dealing with XML formats you must:

Be insensitive to the XML tag order within a phrase, and
Ignore undocumented tags.

Normally, multiple xrootd servers transmit summary information to a collector (i.e., a process accepting messages on a specific port). In order to simplify the processing of summary information, a UDP multiplexing and XML parsing program, called mpxstats, is provided. This program accepts data on a selectable port, multiplexes received the datagrams into a single stream, and optionally parses the XML into either a CGI format or a flat key-value format. The output is sent to standard out for further processing.

2.1 The mpxstats Command

mpxstats [-f {cgi | flat | xml}] –p port [-s]

Function

Multiplex UDP datagrams into a single stream and optionally parse the data.

Options & Parameters

-f Parses the received data into the specified format:

cgi Computer Gateway Interface

flat Simple keyword-value format

xml Original format (i.e., input is not parsed)

-p port is the port to use for accepting UDP datagrams.

-s includes the actual sender in cgi and flat format output.

Defaults

By default, xml output format is used. The UDP port must be specified.

Notes

1) The cgi and flat formats are based on the input the xml tags, without interpretation. Therefore, un-described tags may appear in the output and should be ignored.

2) The cgi format generally produces: “var=value[&var=value[. . .]]\n”. Each var is based on an xml format item and the value is the item’s associated value. One new-line terminated string is generated for each UDP packet.

3) The cgi format is suitable for input to an XrdOucEnv class object which converts cgi strings into environment variable store. The class provides a simple value look-up scheme; much like getenv().

4) The flat format generally produces: “var value\n[var value\n[. . .]]\n”. Each var is based on an xml format item and the value is the item’s associated value. Each var-value pair is a new-line terminated string. A null line is generated at the end for each UDP packet.

5) The flat format is suitable for input to Perl and Python scripts and can easily be used to construct var-value hashes for further processing.

6) The mpxstats program writes its output to standard out. Error messages are written to standard error.

2.1.1 Quick Guide Example

The following picture illustrates the general scheme most installations use to gather summary statistics and insert them into their monitoring framework. Here a number of xrootd servers send their statistics to a collector machine listening at port 3333. The collector merges all of the data streams and sends a selection of the desired data to the monitoring infrastructure.

To implement such a scheme, follow these steps:

1. In the configuration file for each xrootd insert the following directive

if exec xrootd

xrd.report collector_host_name:3333

Where collector_host_name is the name of the machine that collects and formats the summary data. The if/fi construct only allows xrootd to report statistics as the cmsd does not currently report meaningful statistic.

2. Start the data multiplexing program and feed its output to program or script that can inject the data into the monitoring infrastructure. For instance,

mpxstats –f flat –p 3333 | send2monitor

The send2monitor script is, of course, installation dependent. Below is a simple perl script that reads the statistical data from standard in, places it a hash, and then calls a subroutine that can use the values in the hash to feed Ganglia.

#!/bin/perl

do {undef %StatsData;

while (($Line = <STDIN>) ne "\n")

{exit if !chomp($Line);

($Var,$Val) = split(' ',$Line);

$StatsData{$Var} = $Val;

}

Ganglia(); # Inject data into the monitoring system

} while(1);

send2monitor: Place Data In a Hash Indexed By the Data’s Variable Name

2.2 Summary Data

<statistics

tod="int64" ver="chars" src=”chars”

tos=”int64” pgm=”chars” ins=”chars” pid=”int”> • • •

</statistics toe=”int64”>

Variable	Type	Explanation Of Value
host	char	The name of the host that actually sent the UDP packet.[*]
ins	char	The instance name specified via –n option (anon if none).
pgm	char	The name of the program.
pid	int	The program’s process ID.
src	char	The host and port reporting data, specified as “hostname:port”
tod	int64	Unix time when statistics gathering started.
toe	int64	Unix time when statistics gathering ended[†].
tos	int64	Unix time when the program was started.
ver	char	The version name of the server’s code.

2.2.1 Buff Summary Data

</stats>

Variable	Type	Explanation Of Value
buff.adj	int	Number of adjustments to the buffer profile.
buff.buffs	int	Number of allocated buffers.
buff.mem	int64	Bytes allocated to buffers.
buff.reqs	int	Number of requests for a buffer.

2.2.2 Info Summary Data

<host>chars</host><port>int</port><name>chars</name>

</stats>

Variable	Type	Explanation Of Value[‡]
info.host	char	The name of the host that generated the information.
info.name	char	The instance name specified via –n option (anon if none).
info.port	int	The port used for server requests.

2.2.3 Link Summary Data

</stats>

Variable	Type	Explanation Of Value
link.ctime	int64	Cumulative number of connect seconds. ctime/tot gives the average session time per connection.
link.in	int64	Total number of bytes received.
link.maxn	int	Maximum number of simultaneous connections.
link.num	int	Number of current connections.
link.out	int64	Total number of bytes sent.
link.sfps	int	Number of partial sendfile() operations.
link.stall	int	Number of times partial data was received.
link.tmo	int	Number of request read timeouts.
link.tot	int64	Total number of connections since start-up.

2.2.4 Ofs Summary Data

<role>chars</role><opr>int</opr><opw>int</opw>

</stats>

Variable	Type	Explanation Of Value
ofs.bxq	int	Number of background tasks processed.
ofs.dly	int	Number of times a delay was imposed.
ofs.err	int	Number of errors encountered.
ofs.han	int	Number of active file handles.
ofs.opp	int	Number files open in read/write POSC mode.
ofs.opr	int	Number files open in read-mode.
ofs.opw	int	Number files open in read/write mode.
ofs.rdr	int	Number of redirects processed.
ofs.rep	int	Number of background replies processed.
ofs.role	char	Reporter’s role (e.g., manager, server, etc).
ofs.ser	int	Number of events received that indicated failure.
ofs.sok	int	Number of events received that indicated success.
ofs.ups	int	Number of times a POSC mode file was un-persisted.

2.2.5 Oss Summary Data

<paths>int

<lp>”chars”</lp><rp>”chars”</rp>

</stats> • • •

</paths>

<space>int

<name>chars</name>

<usg>int64</usg>[<qta>int64</qta>]

</stats> • • •

</space>

</stats>

Variable	Type	Explanation Of Value
oss.paths	int	Number of subsequent paths stats (0 <= i < n).
oss.paths.i.free	int64	Total number of kilobytes available.
oss.paths.i.ifr	int64	Total number of free inodes.
oss.paths.i.ino	int64	Total number of inodes.
oss.paths.i.lp	char	The minimally reduced logical file system path.
oss.paths.i.rp	char	The minimally reduced real file system path.
oss.paths.i.tot	int64	Total number of kilobytes allocated.
oss.space	int	Number of subsequent space stats (0 <= i < n).
oss.space.i.free	int64	Total number of kilobytes available.
oss.space.i.fsn	int	Number of file system extents.
oss.space.i.maxf	int64	Maximum number of kilobytes available in a filesystem extent.
oss.space.i.name	char	Name for the space.
oss.space.i.qta	int64	Total space quota[§], if supported.
oss.space.i.tot	int64	Total number of kilobytes allocated.
oss.space.i.usg	int64	Usage associated with space name, if supported.

2.2.6 Poll Summary Data

</stats>

Variable	Type	Explanation Of Value
poll.att	int	Total number of file descriptors attached for polling.
poll.en	int	Number of poll enable operations.
poll.ev	int	Number of polling events.
poll.int	int	Number of unsolicited polling events.

2.2.7 Proc Summary Data

</stats>

Variable	Type	Explanation Of Values Reported by getrusage()
proc.sys.s	int	Seconds of system-time.
proc.sys.u	int	Microseconds of system-time.
proc.usr.s	int	Seconds of user-time.
proc.usr.u	int	Microseconds of user-time.

2.2.8 Rootd Protocol Summary Data

</stats>

Variable	Type	Explanation Of Value
rootd.num	int	Total number of times the protocol was selected.

2.2.9 Sched Summary Data

</stats>

Variable	Type	Explanation Of Value
sched.idle	int	Number of scheduler threads waiting for work.
sched.inq	int	Number of jobs that are currently in the run-queue.[**]
sched.jobs	int	Total number of jobs requiring a thread.
sched.maxinq	int	Longest run-queue length
sched.tcr	int	Total thread creations.
sched.tde	int	Total thread destructions.
sched.threads	int	Total number of current scheduler threads.
sched.tlimr	int	Number of times the thread limit was reached.

2.2.10 Sgen Summary Data

Variable	Type	Explanation Of Value
sgen.as	int	One if data was asynchronously gathered, 0 otherwise.
sgen.et	long	Elapsed milliseconds from start to completion of statistics.

2.2.11 Xrootd Protocol Summary Data

<ops>

</ops>

<aio>

</aio>

</stats>

Variable	Type	Explanation Of Value
xrootd.num	int	Total number of times the protocol was selected.
xrootd.aio.max	int	Maximum simultaneous async I/O requests.
xrootd.aio.num	int	Total number of async I/O requests processed.
xrootd.aio.rej	int	Number async I/O requests converted to sync I/O.
xrootd.ops.getf	int	Total number of getfile requests.
xrootd.ops.misc	int	Total number of “other” requests.
xrootd.ops.open	int	Total number of file open requests.
xrootd.ops.pr	int	Total number of pre-read requests.
xrootd.ops.putf	int	Total number of putfile requests.
xrootd.ops.rd	int	Total number of read requests.
xrootd.ops.rf	int	Total number of cache refresh requests.
xrootd.ops.sync	int	Total number of sync requests.
xrootd.ops.wr	int	Total number of write requests.

3 Detailed Monitoring Data Format

The xrootd.monitor directive specifies the monitor parameters as well as the hosts that are to receive the monitoring information, Monitor records are sent as UDP datagrams. Therefore, the information is sent whether or not the receiving host is enabled for the records. Because the amount of information is potentially large and monitoring should incur the least possible overhead, statistical timing as well as special format records are used. The following paragraphs explain each of these points.

In order to maintain a low overhead, each connection collects its own I/O event data in a local buffer and sends the data when the buffer is full or when the connection is closed. Non-I/O events (e.g., open and close) are collected globally in a single buffer, and is sent when it is full or when the specified full timeout occurs. I/O and non-I/O events may be intermixed when the configuration specifies a particular recipient for such a combination. Low overhead is also maintained by not time-stamping each event. That is, the information is collected within a statistical window. While the order of events is maintained, it is impossible to tell precisely when the event actually happened within this window. The receiver should uniformly distribute the events across the window.

Since each connection maintains its own local buffer of I/O events, multiple datagrams may be sent with disparate, possibly overlapping, windows. The receiver must merge all of these windows into a uniform coherent time stream. This is possible because precise times are always given for the start and end of the window in which the events were collected. Care should be taken to appropriately order the packets, as UDP packets can arrive in any order. To assist in ordering packets, each packet carries a time-stamp as well as a sequence number so that the receiver can easily order packets as well as discover if any packets were lost due to network congestion. Below is one such example.

Packet physical order:

Packet 0: t=4 seq=2 window=tod+a:tod+b <I/O requests>

Packet 1: t=4 seq=1 window=tod+x:tod+y <I/O requests>

Packet 2: t=1 seq=0 window=tod+j:tod+k <I/O requests>

Packet 3: t=5 seq=3 window=tod+d:tod+e <I/O requests>

Packet logical order:

Packet 2: t=1 seq=0 window=tod+j:tod+k <I/O requests>

Packet 1: t=4 seq=1 window=tod+x:tod+y <I/O requests>

Packet 0: t=4 seq=2 window=tod+a:tod+b <I/O requests>

Packet 3: t=5 seq=3 window=tod+d:tod+e <I/O requests>

Window logical order:

Packet 0: t=4 seq=2 window=tod+a:tod+b <I/O requests>

Packet 3: t=5 seq=3 window=tod+d:tod+f <I/O requests>

Packet 2: t=1 seq=0 window=tod+j:tod+k <I/O requests>

Packet 1: t=4 seq=1 window=tod+x:tod+y <I/O requests>

The previous figure illustrates the physical and logical sequence of packets from a single server. You should note that the window start and end times do not correlate with the packet send time.

In order to maximize the amount of information that can be stored in a single data-gram as well as to minimize redundancy, a dense encoding scheme is used. The scheme works as follows:

i. Each time a user enables monitoring and assigns an application string to the action, the system assigns the combination a unique dictionary ID (dictid). The mapping between the dictionary ID and the user/application pair generates a separate monitor record that is sent to the destination host.

ii. Each time a user opens a particular file, the system assigns the combination a unique dictionary ID (dictid). The mapping between the dictionary ID and the user/path pair generates a separate monitor record that is sent to the destination host.

iii. Every request that deals with the user/path pair is henceforth tagged with the dictionary ID. Thus, it is critical for the receiver to maintain the mapping.

iv. Monitor records are formatted as structured binary records. All numeric fields within the record are sent in network byte order.

v. Each datagram is self-consistent. That is, information is never logically split across datagrams. Mapping requests are always fully contained within a datagram. I/O request information is always bracketed by window timing marks.

The following figures describe each type of data-gram that is sent by xrootd.

struct XrdXrootdMonHeader

{kXR_char code; // 'd'|'i'|’s’|'t'|'u'|’v’

kXR_char pseq; // packet sequence

kXR_int16 plen; // packet length

kXR_int32 stod; // Unix time at Server start

};

Header for Each Monitor Message Data-gram

Actual information structures follow the header in the same data-gram. There are four types of information structures,

· d – dictid mapping to a user/path combination

· i – dictid mapping to a user/information combination

· s – staging event

· t – a file or I/O request trace

· u – dictid mapping to the user login name

· v – dictid mapping to user authentication information

The record type is placed in the header’s code variable . The pseq variable is an ascending, wrapping, packet sequence number, whose value ranges from 0 to 255. This provides a gross mechanism to order packets. Timing marks within the packet provide more accurate information. The plen variable contains the packet’s length. This value can be used to verify that the system’s reported length equals the intended length. The stod, defined as Unix time, indicates when the Unix time when the server was started. Thus, each stod/dictid combination will be unique across all time.

3.1 Monitor Map Message Format

struct XrdXrootdMonMap

{XrdXrootdMonHeader hdr;

kXR_int32 dictid;

char info[];

};

A map message ‘d’, ‘i’, ‘s’, ‘u’, or ‘v’ in hdr.code is generated when a client:

· user logs in (type ‘u’),

· authenticates (type ‘v’),

· stages a file (type ‘s’),

· opens a file (type ‘d’), and

· associates information with the session (type ‘i’).

For each record other than ‘s’, xrootd generates a unique dictionary ID and assigns it to the user/information, user/path combination, username, or user/authinfo. The MonMap record describes this mapping. It starts with a standard header. Following the header is the binary dictionary ID, dictid. This ID is unique within the server’s boot-session. That is, every time the server is restarted, the dictid value is reset to zero. For ‘s’ records, the dictid is always zero.

3.1.1 Message Info Field

Code Contents of info Code Contents of info

d userid\npath u userid

i userid\nappinfo v userid\nauthinfo

s userid\nstginfo

userid: user.pid:fd@host

stginfo: lfn\n&tod=tod&sz=bytes&tm=sec

authinfo: &p=prot&n=[name]&h=[hname]&o=[org]&r=[role]

Where:

path The full path name of the file being opened.

appinfo

Un-interpreted application supplied information.

user The Unix username of the user.

pid The user’s process number that issued the request.

fd The server’s file descriptor number associated with the connection to user:pid at host.

host The host name, or IP address, where the user’s request originated.

lfn The logical name of the file just staged in.

tod The Unix seconds, as returned by time(), when the record was produced.

bytes The size of the staged in file in bytes.

sec The number of seconds it took to stage in the file (i.e., the time between the start of the stage-in request to the time the complete file was written to disk).

prot The protocol name used for authentication.

name The client’s distinguished name as reported by prot. If no name is present, the variable data is null.

host The client’s host’s name as reported by prot. If no host name is present, the variable data is null.

org The client’s organization as reported by prot. If no organization is present, the tag variable is null.

role The client’s role name as reported by prot. If no role name is present, the variable data is null.

Notes

1) The ‘d’ , ‘i’, and ‘v’ records contain two ASCII text strings, separated by a new-line (\n) character.

2) The ‘s’ record contains three ASCII text strings, separated by a new-line (\n) character.

3) The ‘u’ records do not end with a new-line character.

4) Mapping records can be sent at any time. Interspersed with the mapping records are trace records which detail I/O requests.

3.2 Monitor Trace Message Format

struct XrdXrootdMonTrace

{union {kXR_int64 val;

kXR_char id[8];

kXR_unt32 rTot[2];

} arg0;

union {kXR_int32 buflen;

kXR_unt32 wTot;

kXR_int32 Window;

} arg1;

union {kXR_unt32 dictid;

kXR_int32 Window;

} arg2;

};

struct XrdXrootdMonIO

{XrdXrootdMonHeader hdr;

XrdXrootdMonTrace info[];

};

The MonTrace record is highly encoded and repeated as often as possible in a single data-gram, as shown in the MonIO structure. Each instance of info represents either a read, write, open, close request, application id, or a window timing mark. All binary data is appears in network byte order. The info[].arg0.id[0] character identifies the type of information the entry contains, as follows:

Definition	Value	Meaning
XROOTD_MON_OPEN	0x80	File has been opened
XROOTD_MON_APPID	0xa0	Application provided marker
XROOTD_MON_CLOSE	0xc0	File has been closed
XROOTD_MON_DISC	0xd0	Client has disconnected
XROOTD_MON_WINDOW	0xe0	Window timing mark
----	<=0x7f[††]	Read or write request

The following fields are used for each type of record:

Field	Contents
info[].arg0.id[1…3]	Not used.
info[].arg0.id[4…15]	Up to 12 characters of application identification.

Appid Request Entries

Field	Contents
info[].arg0.id[1]	Number of bits Info[].arg0.rTot[1] has been right shifted to fit into a 32-bit unsigned int.
info[].arg0.id[2]	Number of bits Info[].arg1.wTot has been right shifted to fit into a 32-bit unsigned int.
info[].arg0.id[3]	Not used.
info[].arg0.rTot[1]	Scaled number of bytes read from the file.
info[].arg1.wTot	Scaled number of bytes written to the file.
info[].arg2.dictid	The dictionary ID associated with this request.

Close Request Entries

Field	Contents
info[].arg0.id[1…7]	Not used.
info[].arg1.buflen	Number of seconds that client was connected.
info[].arg2.dictid	The dictionary ID associated with this request.

Disconnect Request Entries

Field	Contents
info[].arg0.id[1…7]	Size of the file in bytes.
info[].arg1	Not used.
info[].arg2.dictid	The dictionary ID associated with this request.

Open Request Entries

Field	Contents
info[].arg0.id[1-7]	Not used.
info[].arg1.Window	Unix time of when the previous window ended.
info[].arg2.Window	Unix time of when this window has started.

Window Entry

Field	Contents
info[].offset.val	Read or write offset (see below).
info[].arg1.bufflen	Length of the read when non-negative. When negative, this is the length of a write request.
info[].arg2.dictid	The dictionary ID associated with this request

Read/Write Request Entry

Since each datagram is self-consistent, a trace message will always start and end with a window entry. Additional window entries may be placed within the record should requests cross window boundaries within the same data-gram. Because request timing is variable, window start and end times are rarely adjacent. That is, a window may end at time x but the new window may start at a time that is many windows away from the end time. This is because xrootd compresses adjacent empty windows.

A window entry may also be forced should the buffer fill or the connection is closed before the window actually ends. In this case, the window may be substantially smaller than configured window size. The receiver should not count that each window will be the same size. The receiver should internally time-stamp each entry using an appropriate distribution curve within the reported window.

Definitions of these structures can be found in “XrdXrootdMonData.hh” file.

[*] This information is provided by the Operating System’s recvfrom() function, not the data stream. It is present only when the –s mpxstats option has been specified.

[†] The quantity (toe-tod) is the number of seconds it took to gather the statistics. The sgen format item provides millisecond accuracy.

[‡] The info tag is deprecated and normally does not get included as this information is present in the header tag. It is documented here for backwards compatability.

[§] This tag may be missing if quotas have not been configured.

[**] The number of active requests is represented by (sched.threads – sched.idle + sched.inq).

[††]Indicates that if the high order bit is zero, then this is a read/write request.

4 Document Change History