Xrootd is capable of providing a real-time monitoring. Basic (light) monitoring includes the following information:

• sessions:
  • user name
  • process id
  • client host name
  • duration
  • close time
• files:
  • file path name
  • associated session
  • open time
  • close time
  • number of bytes read
  • number of bytes written
  • file size in bytes
• xrootd servers:
  • restart time

Setup

Monitoring data is sent from xrootd servers via UDP packets to a collector, therefore it is non-disruptive to the xrootd no matter what happens to the collector. The collector is capable of collecting and decoding in real time packets from multiple xrootd servers. Decoded data, in the form of ASCII files is formated such that it can easily be loaded into a relational database.

Setting up the light monitoring involves several simple steps:

1. Prepare the database server
2. Configure xrootd
3. Start the collector
4. Set up the MySQL database
5. Run the MySQL loader
6. Run Application to Prepare Statistics
7. Set up Apache tomcat server
8. Download the monitoring web application

Preparing Database Server

Configuring xrootd

Optionally pick up a port number <ctrPort> for the collector (default = 9930) and add the following line to the xrootd config file:

xrootd.monitor all dest files info user <dbHostName>:<crtPort>

For instance:

xrootd.monitor all dest files info user xrootd-stats:9930

Starting Collector

Simply run the command xrdmonCollector.pl < configFile> start.
If you used a non-default port number in xrootd config file, you have to use the same number in the configFile . Above script runs the binary xrdmonCollector which has to be in your $PATH. The log file is writen to
<baseDir>/<thisSite>/logs/out/xrdmonCollector.<date>

If you decide to run multiple collectors per machine (e.g. one for each group of hosts), you should repeat all of the steps described in this document for each cluster and use a different <baseDir> for each cluster in addition to using a different port number (in xrootd config and < configFile>)

To stop the collector, use the command xrdmonCollector.pl < configFile> stop
The collector buffers some data in memory, and killing it in any other way will result in losing buffered data. Please keep in mind that it can take a few minutes for the collector to completely shut down.

Setting up MySQL

To load data into MySQL you will need:

• MySQL (recommended: 5.0 or higher)
• perl
• DBI module for perl. If you don't have (chances are you do, it is very popular), see http://dbi.perl.org/.

You need to do (if you run multiple collectors, you need to repeat these steps for each collector):

1. Pick a MySQL server<dbServerName>, user name <MySQLUser> and database name <dbName>
2. Make sure the mysqld server is running
on the MySQL server <dbServerName>
3. Make sure the time_zone tables are loaded into MySQL. If not use the appropriate command for your platform to load them.
4. Grant the user appropriate privileges using the account with GRANT privileges. At MySQL prompt enter:
   grant all on <dbName> to <MySQLUser>
5. Run xrdmonCreateMySQL.pl < configFile> .

Platform	command to load time_zone tables
Solaris	mysql_tzinfo_to_sql /usr/share/lib/zoneinfo/ | mysql -uroot mysql
Linux	mysql_tzinfo_to_sql /usr/share/zoneinfo/ | mysql -uroot mysql

Running MySQL Loader

To load data coming out of the collector into the MySQL database start the MySQL loader using the command:

But first you have to replace the dummy subroutine getFileTypes in xrdmonLoadMySQL.pl with a suitable one for your file types. (see example for BaBar experiment.)

xrdmonLoadMySQL.pl will run indefinitely. To stop it use the command

This will ensure that the program loads everything in the buffer before terminating. Killing it in any other way might leave the database tables in an inconsitent state. Also the loader sets certain semaphore files that might be left behind if it is not stopped properly.

Preparing Statistics Tables

The data loaded into the database needs further processing to produce the statistics tables that are published on the web. You do this with the command:

xrdmonPrepareStats.pl will run indefinitely. To stop it use the command

Killing it in any other way might leave the database tables in an inconsitent state.

Configuration File

All perl scripts in xrootd monitoring system use a single config file. Make sure that you do not alter the config file after you have created the database unless you are sure that the changes are not incompatible with the existing database and the applications that are already running.

Two of the definitions used in the config file are specific to this project and need some explanation. These are file types and Job

File Types

getFileTypes

Definition of Job

1. running: Number of open sessions > 0
2. finished: Number of open sessions = 0 AND latest disconnect time is at least <maxJobIdleTime> before current time.
3. dormant: When none of above is true. A dormant job can resume running if a new session is started before the <maxJobIdleTime> limit is reached.

Contents of Configuration File

The config file contains lines with the following format:

<token>: <Value 1> [ <Value 2> [ <Value 3> ] ] ]
(no space between token and colon and all values are separated by single space)

The valid tokens and explanation of values are given in the following table.

Time and interval values are integers followed by MINUTE | HOUR | DAY | WEEK | MONTH | YEAR

The date format is: YYYY-MM-DD<space>HH:MM:SS

Location of Perl Scripts

The perl scripts are currently located in xrootd/src/XrdMon. This is a temporary location and will be fixed very soon.
You may have to tweak the very first line in each script to point to the correct location of your perl on your system.

Setting up Tomcat Server

To deploy the web application for xrootd monitoring you need to set up Apache Tomcat.

1. Choose a server <webServerName>, preferably other than the one for MySQL database (e.g. you might want to run MySQL and the collector inside internet-free-zone for security reasons)
2. Download Apache Tomcat 5.5.* (we are using 5.5.7 at SLAC)
3. Install the administration Web Application which has to be dowloaded separately
4. Create a user with manager and admin privileges by editting the file conf/tomcat-users.xml. Add in the line:
   <user username="<userName>" password="<password>" roles="manager,admin"/>
5. Configure the database connection by editing and adding the following lines to conf/server.xml

   <Resource
   name="jdbc/xrdmon-database"
   type="javax.sql.DataSource"
   password="<password>"
   driverClassName=" org.gjt.mm.mysql.Driver "
   maxIdle="2"
   maxWait="5000"
   username="<webUser>"
   url="jdbc:mysql://<dbServerName>/<dbName>"
   maxActive="4"/>

6. Start the tomcat server by running the startup script in the bin directory

Deploying Web Application

Downloaded (location to be decided)/xrdmon.war (6 MB) and deploy the application through the manager tomcat application by loading the war file.

Additional Informtion

More on real time

So how "real-time" is this system? Most of the information on the xrootd server side is flushed after a udp packet is filled up, or when a specified time elapses (currently 1 min). Once sent, data should be (a) collected, (b) decoded and (c) flushed to the real time log file within few seconds (depends on the RT flush setting, default is 10 sec). Loading to MySQL introduces another short delay, at slac it is 1 minute. So in practice, the information is available in MySQL a couple of minutes after the event happens.

More on log files

By default collector will decode all the light data and produce 3 different types of log files:

Collected packets' logs contain segregated raw packets exactly as they arrive from xrootd servers, packets from each xrootd instance in a separate directory.

Decoded real time data logs contain information about current snapshot of the system, for instance current number of opened sessions, users, or files. If you are not interested in watching current "snapshot" of the system you may consider turning it off (xrdmonCollector -rt off). This data is regarded as temporary/scratch, e.g. it becomes invalid when the collector is stopped.

History data logs contain information about closed sessions and closed files. It should be treated as a permanent record.

Collected packets

Collected packets' logs are segregated according to xrootd host and port. Log name: active.ctr. Default maximum size for the log file is 1GB. When active.ctr reaches that size it is renamed to <timestamp>.ctr, and a new active.ctr is opened. Never delete active.ctr while collector is running. You can backup/copy/rm closed logs.

Real time logs

Real time data is kept in rtLog_verXXX.txt. If you use the provided script for loading data to MySQL it will remove the rtLog_verXXX.txt each time it loads the data to MySQL. It backs up the data in the backup directory for each site by appending it to the current backup file. A new backup file is started at intervals given by <backup interval>. You will also see lock files rtLog_verXXX.lock which is used to synchronize access to rtLog_verXXX.txt. The lock file should not be touched.

Format of the log file:
u <uniqueSessionId> <userName> <processId> <clientHostName> <xrootdHostName>
d <uniqueSessionId> <durationInSec> disconnectTime>
o <uniqueFileId> <userName> <processId> <clientHostName> <filePath> <openTime> <fileSize> <xrootdHostName>
c <uniqueFileId> <bytesRead> <bytesWritten> <closeTime>

Lines starting with 'u' indicate 'user connect' (session start).
Lines starting with 'd' indicate 'user disconnect' (session stop).
Use <uniqueSessionId> to correlate 'd' lines with the corresponding 'u' lines.

Lines starting with 'o' indicate 'file open'.
Lines starting with 'c' indicate 'file close'.
Use <uniqueFileId> to correlate 'o' lines with the corresponding 'c' lines.

All values are separated by TAB.
Format of timestamp is: YYYY-MM-DD<space>HH:MM:SS

Example:

u 115 dgustaf 12078 barb0045 kan012.slac.stanford.edu
o 120 owen 1960 barb0154 /store/SP/R12/001237/200010/12.4.0j/SP_001237_003641.02E.root 2005-03-16 22:43:07 186067283 kan006.slac.stanford.edu
c 66 0 0 2005-03-16 22:41:40
d 84 0 2005-03-16 22:42:40
o 107 palombo 26144 barb0077 /store/SP/R14/001237/200401/14.4.2a/SP_001237_007640.01.root 2005-03-16 22:43:00 1276663486 kan031.slac.stanford.edu
o 128 roethel 7553 barb0131 /store/SP/R12/000998/200301/12.6.0b/SP_000998_013189.01.root 2005-03-16 22:43:20 1353696936 kan031.slac.stanford.edu
o 131 owen 8791 barb0093 /store/SP/R12/001237/200007/12.4.0j/SP_001237_003653.01.root 2005-03-16 22:43:30 16643497 kan031.slac.stanford.edu
u 116 dgustaf 20445 noma0065 kan032.slac.stanford.edu

Future Developments

This is a first attempt in providing a real time monitoring for xrootd. More work is in progress including:

• Expansion to more than one site running the collector
• Handling database backup
• Ability to reload the database from scratch using the backup files
• Ability to choose the time zone for the web pages. At the moment the times are always in GMT
• Utilites to add new file types and sites to the database
• Query optimization
• Staging statistics

Questions or comments? Send email to Tofigh Azemoon or Jacek Becla.