1.2 Starting the xrootd Daemon

Use the following command to start the xrd-based xrootd daemon:

xrootd [ options ] [ path [ path [ . . . ] ] [piargs]

options: [-c cfn] [-l largs] [-k {num | sz{k|m|g} | sig}]

[esoteric]

esoteric: [-b] [-d] [-h] [-I {v4 | v6}] [-L protlib]

[-n name] [-[-p {port | any}] [-P protocol]

[-R user] [-s pfn] [-S site] [-z]

largs: [=]fn | - |

@lib[,bsz=sz][,cse={0|1|2}][,logfn=[=]fn]

piargs: -+[tag] [args] [piargs]

Parameters

path An absolute file system path prefix. All requests will be restricted to files with this prefix. You may specify any number of path prefixes. If no path is specified, operations will be restricted to paths starting with /tmp.

Options

-c fn The name of the configuration file. If one is not specified, no configuration file is processed.

-l [=] fn

Specified how messages are to be handled. Options are:

fn Directs messages and any trace output to the indicated file, fn, possibly qualified by the instance name (see the fencing section). If fn is a dash (-), output is sent to standard error; the default.

=fn Same as fn but the fn is not qualified by the instance name, if any. This allows log files to be handled in an arbitrary manual way. For more information see the section on fencing.

@lib Directs messages to a plug-in that is defined in the shared library specified by lib (see the section on log file plug-ins). Additional comma-separated parameters may follow lib, as follows:

bsz=sz Specifies the size of the speed matching buffer. The default is 64K. Messages are placed in the buffer and then forwarded to the plug-in as time permits. A value of 0 disables speed matching and messages are handed off to the plug-in as they occur. See the section on log file plug-ins for more information. A positive value less than 8K is forced to be 8K. The maximum allowed in one megabyte. The sz may be suffixed by k or m to indicate kilobytes or megabyte, respectively.

cse={0|1|2} Specifies how standard error output should be handled:

0 Does not capture standard error output. All such output is sent to the logfn destination, if specified, or is otherwise lost. This is the default.

1 Captures standard error but only forwards it to the logging plug-in if it starts with a standard time stamp. This option may cause an infinite loop. Refer to the logging plug-in section for more information.

2 Captures standard error output and forwards it to the logging plug-in without inspection. Refer to the logging plug-in section for more information.

logfn=[=]fn Specifies that messages are also to be routed to a local log file. The parameter is identical to that described above. To use standard error, specify a dash (-) for fn.

-k num | sz{k|m|g} | sig

Keep no more than num old log files. If sz is specified, the number of log files kept (excluding the current log file) is trimmed to not exceed sz bytes. The sz must be suffixed by k, m, or g to indicate kilobytes, megabyte, or gigbytes, respectively. If a sig value is specified (i.e. hup etc), then an external program is expected to handle log file rotation (e.g. logrotate). Except for fifo, the argument specifies signal that causes the daemon to close and re-open the log file to allow rotation to occur. When fifo is specified, the daemon waits for data to appear on a fifo whose path is identical to the log file path but whose name is prefixed by a dot. Refer to the notes for manual rotation caveats.

Esoteric Options

-b Runs the program in the background. You should also specify -l.

-d Turns on debugging.

-h Displays help information.

-I {v4 | v6}

Restricts the server’s internet address protocol. When v4 is specified, only hosts with IPV4 addresses can connect or be connected to. When v6 is specified, the default, hosts using IPV6 or IPV4 addresses can connect or be connected to. This option is only useful for systems that have misbehaving IPV6 network stacks. The default is established by the network interface configuration on the machine at the time the program starts.

-L protlib

Specifies the shared library that holds the implementation of the default protocol specified by the –P option.

-n name

Assigns name to the xrootd instance. By default, the xrootd instance is unnamed. See the notes on how to use this option.

-p port

The TCP port, or service name associated with a port, that xrootd is use for new connections. The default is “xrootd” or port 1094, if the TCP service xrootd cannot be found /etc/services.

-p any

Uses any available port.

-P protocol

The name of the default protocol. Use this option when the name of the executable differs from the name of the default protocol. You may need to specify the –L option as well. See the notes for more information.

-R user

The user name or numeric uid of the user whose effective identity is to be assumed. See the usage notes for more information. The specified user may not have super-user privileges. This option may not be specified unless the program is running as super-user.

-s pfn Specifies the name of the file that is to hold the process id upon start-up.

-S site Specifies a 1- to 15-character site name that is to be included in monitoring records. The name may only contain letters, digits and the symbols “_-:.”; any other characters are converted to a period.

-z provides microsecond resolution for log file message timestamps.

-+ provides a mechanism to pass command line arguments to plug-ins. See the section “Passing Plug-In Command Line Arguments” for more information.

Defaults

xrootd –l - –I v6 –p xrootd –P xrootd /tmp

General Notes

1) For security purposes, only files in /tmp are allowed to be accessed unless you specify otherwise. You may specify other paths either on the command line or using the xrootd export configuration directive.

2) Do not prefix any export path with the oss localroot directive path, if any.

3) If a log file is specified without a signal -k option, the file is closed at midnight, renamed to have a date suffix (i.e., fn.yyyymmdd) and possible sequence number (i.e. fn.yyyymmdd.n), and a new log file is opened.

4) When a signal value is specified, log files are not automatically renamed at midnight. Instead an external program must be used to properly rotate log files. Make sure to choose a signal that is not in use by any plugin. If unsure, choose one of the obscure signal names and monitor for any odd behavior. Otherwise, use the fifo option. Be aware that on some non-Linux platforms the fifo file descriptor may leak.

5) When fifo is specified the fifo file name must not exists or exist as a fifo file. A simple “echo x >> /path/.lfn” causes the logfile to close and reopen.

6) The sig names, except for fifo, be fully capitalized as well prefixed by “sig” or “SIG” when capitalized.

Notes on Esoteric Options

1) The default port service name, default protocol, and pidfile name is normally determined by the prefix-name of the executable. The prefix-name is defined to be all of the characters in the base filename (i.e., the directory path removed) up to but not including the first dot in the name, if any. If the name starts with a dot, the prefix-name is the complete base filename.

2) The way the prefix-name is derived allows you to maintain several versions of a particular xrd executable (e.g., xrootd and xrootd.debug) without changing the intrinsic way default names (e.g, protocol) are determined.

3) You must use the –P option to set the default protocol name as well as other related naming aspects (e.g., port service name) when the name of the executable does not correspond to the name of the default protocol.

4) The –n option allows you to run multiple instances of the xrootd on the same machine. See the next section on instances and fencing.

5) Use -p any for protocols that manage their own port numbers. This is the case for redirection target xrootd/cmsd combinations. Only the initial point of contact needs a well-known port number. All other connections between clients and servers are routed using whatever port numbers are currently in effect. This allows you to keep a simple configuration file for servers and to run more than one server on the same machine without worrying about conflicting port numbers.

6) The –b option forces the program into the background. If –l is not specified; all output messages are discarded.

7) The -R option allows the program to run under the super user’s account. This is allowed because the effective user is set to specified user and the effective group to user’s primary group. Thus, the program is not effectively running as super-user. However, the real and saved user ids may still be “root”, depending on how the program was started.

8) The -R option provides a minimal increase in security since it is possible for a loaded protocol to switch back to super user mode. You should not use the -R option unless absolutely necessary.

9) The –b, –p, and –s command line options are meant to be used by start-up scripts (e.g. init.d).

10) Warning: Command line options, except for –s, over-ride corresponding configuration file directives.

Example

xrootd –c /opt/xrootd/xrootd.cf

1.2.1 Multiple Instances and Automatic Fencing

You can run multiple instances of xrootd on the same physical machine. This is useful when you want to overlay more than one cluster on top of a file system (e.g. production and test). In order to prevent instances from interfering with each other, you must provide each xrootd that is running on the same hardware a unique instance name. One of the daemons need not have an instance name as it assumes the name “anon” (i.e. anonymous).

Once an instance name is assigned to a daemon using the –n option, the system automatically fences in the daemon so that it does not interfere with any other xrootd processes running with it. Automatic fencing consists of threse actions:

· The instance name is suffixed to the adminpath to create a unique location for temporary server files. For instance, if –n is not specified, xrootd creates /tmp/.xrootd/admin as the path for the administrative interface. If “-n test” is specified, xrootd creates /tmp/test/.xrootd/admin instead. Even the path specified with the adminpath configuration directive is modified.

· The instance name is used to create a new directory in the current working directory. The current working directory is changed to this newly created path. So, if “/home/xrootd” is the current working directory and “-n test” is specified; the current working directory becomes “/home/xrootd/test”. This allows core files to be segregated by instance name.

· The instance name is automatically inserted into the log file path specified via the –l command line directive to create a unique location for server log files files. For instance, if “–l /var/adm/xrootd/xrdlog” is specified along with “-n test”, xrootd modifies the –l argument to be /var/adm/xrootd/test/xrdlog.

Automatic fencing of log files may, for some installations, run counter to the way log files are commonly handled. You can disable fencing of log files by prefix the log file path by an equals sign. However, you are then responsible to make sure that each instance uses a different log file path or name.

1.2.2 Passing Plug-In Command Line Arguments

You can pass command line arguments to various plug-ins that support the feature using the -+ option. The option must be specified after all XRootD options and parameters as plug-in arguments are stripped from the command line. For example,

xrootd [ options ] [ path [ path [ . . . ] ] -+mypi arg1 arg2 -+urpi arg3

places a pointer to the vector containing “argv[0] arg1 arg2 0” into the internal environment passed to the plug-in using the variable “mypi.argv**” and the count in “mypi.argc” (i.e. 3). Similarly, “argv[0] arg3 0” are pointed to by “urpi.argv**” with the count in “urpi.argc” (i.e. 2). If no tag is specified, the leading prefix is missing (i.e. the variables are “.argv**” and “.argc”). Note that argv[0] is the actual value of argv[0] (i.e. the executable name) passed to XRootD.

It is up to the plug-in to extract the appropriate arguments using a documented tag. Tag values that start with “xrd” should not be used as these are reserved for plug-ins normally distributed with the XRootD package.

1.2.3 Log File Plug-Ins

XRootD allows you to specify a plug-in to handle messages that would otherwise be sent to a regular file or standard error. You do this using the ‘@’ qualifier with the –l option. Logging messages is a critical function in the server and any delay will severely impact server performance. The default logging path is very efficient and any plug-in placed in the path should be just as efficient. To help, a speed matching buffer is used to minimize plug-in vagaries. However, if you choose to not use a speed matching buffer (i.e. a bsz of zero for synchronous operation) then the plug-in becomes the choke point in server performance.

You may also choose to capture standard error output using the cse parameter. However, this option will result in an infinite loop if your logging plug-in writes to standard error for any reason. This may be mitigated by specifying cse=1 which only sends standard error output to the plug-in if it starts with a timestamp of the form “yymmdd hh:mm:ss”. All debugging output starts with such a timestamp.

The details on how you write a plug-in is detailed in the XrdSysLogPI.hh header file. It is important to realize that if you use the XrdSysLogger object to route a message from your plug-in, an infinite loop will result. Additionally, one log file plug-in is used to all XrdSysLogger instances.

1.2.4 Files created by xrootd

The following directories and files are created by xrootd:

Default File	Changed by	Contents
<stderr>	-l and -n command line options	Informational and error messages
/tmp//[name/].xrootd/	-n command line option and the adminpath directive	Directory for various server-related files.
<cwd>//[name/]core[.pid]	-n command line options	Core file
/tmp/[name/]xrootd.pid	pidpath and –n option	Holds the process id
/tmp/xrootd.name.env	pidpath and –n option or –s directory	Holds environmental information (see next section).

1.2.4.1 Environmental Information File

The daemon writes environmental information in the directory infered by the –s command line directive and if not specified, in /tmp. This information can be used to automatically collect all relevant information about a daemon to facilitate automatic problem resolution.

The environmental file is named “xrootd.name.env” where name is the instance name and anon if no instance name was specified. The format of the information is shown below. When parsing this information, you should not depend on the order shown below.

pid=pid&host=host&inst=inst&ver=ver&cfgfn=cfgfn&cwd=cwd&logfn=logfn

Parameters

cfgfn The configuration file used.

cwd The current working directory.

host The host name.

inst The instance name.

logfn The log file being used.

pid The process id.

ver The version string.

1.2.5 Exported Environment Variables

The following table shows the environment variable exported by xrootd. These may be used by external programs and plugins, as needed. They should never be modified.

XRD Variable	Contents
XRDADMINPATH	Is the directory for log files. By default, it is XRDBASE/logs.
XRDCONFIGFN	The effective administrative path used for server management files.
XRDDEBUG	Set to one when the –d command line option is specified.
XRDHOST	The current host’s DNS name.
XRDINSTANCE	Is the string of the form “execname instance@hostname”. Where execname is the executable’s name, instance is the name specified via –n or anon if no instance name was specified, and hostname is the current host’s DNS name.
XRDLOGDIR	Is the directory where log files are written.
XRDNAME	The name specified via –n or anon if no instance name was specified.
XRDPROG	The executable’s name.
XRDSITE	The site name specified either via the –s command line option or the all.sitename directive.

If the standard cms client plugin is being used, the following additional environment variables are exported.

CMS Variable	Contents
XRDCMSMAN	The space separated list of managers for this host each in the form of “host:port”.
XRDCMSCLUSTERID	The globally unique cluster identification for this host.

If the standard ofs plugin is being used, the following additional environment variables are exported.

OFS Variable	Contents
XRDROLE	The effective value specified on the all.role directive.
XRDTPC	Is set when TPC (Third Party Copy) has been configured and represents the version number of the protocol. If the first character is a plus sign, authentication is required.

If the standard oss plugin is being used, the following additional environment variables are exported.

OSS Variable	Contents
XRDN2NLIB	The path and name of the name-to plugin, if specified via the oss.namelib directive.
XRDRMTROOT	The local root path specified by the oss.remoteroot directive.
XRDLCLROOT	The local root path specified by the oss.localroot directive.
XRDOSSQUOTAFILE	The name and location of the file handling disk space quotas.
XRDOSSUSAGEFILE	The name and location of the file handling disk space usage.

If the standard xrootd protocol plugin is being used, the following additional environment variables are exported.

XROOTD Variable	Contents
XRDOFSLIB	The path and name of the OFS plugin specified by the xrootd.fslib directive.
XRDMONRDR	If monitoring is enabled, how often server identification records are sent.

2 Common xrd Configuration Directives

2.1 adminpath

all.adminpath path [ group ]

Function

Specify the location of the administrative communications path.

Parameters

path The absolute path to a directory that is to hold the Unix named socket used to communicate administrative commands to xrd and it related components (e.g., xrootd).

group

Allows any user in xrd’s group to use a socket in path by setting r/w group access on the path

Default (see warning in the notes)

/tmp

Notes

1) The adminpath directive allows you to specify the location of the local TCP socket used for the command-line administrative functions.

2) Warning: if idle /tmp directories and socket files are automatically deleted by the system, you should neither accept the default path nor allow the adminpath path to reside in /tmp.

3) Local TCP socket names are limited to 108 characters. Up to 32 characters are needed to define actual socket files; leaving 76 characters that may be specified as the path.

4) The following steps are taken when creating a Unix named socket in path:

· If –n is specified, a subdirectory corresponding to the instance name (i.e., -n argument) is created in path, if one does not exist. This becomes the new path.

· Subdirectories “.xrd” and “.xrootd” are created in path, if they do not exist.

· Mode bits for these directories are set to 0700 (rwx for owner).

· If group was specified, the mode setting is extended to 0770 (rwx for owner and group).

· A Unix named socket with the name “admin” is created in the “.xrootd” subdirectory.

5) If the “.xrd” or “.xrootd” directories already exist, the mode settings are reset to correspond the to the adminpath directive.

6) The adminpath value is passed to all protocols so that they can create their respective administrative files.

7) Refer to the section “Administrative Interface” for details on how to use the Unix named socket created by this directive.

Example

xrd.adminpath /var/adm/xrd group

2.1.1 Administrative Interface

The adminpath directive is used to construct the path where local TCP sockets, called named sockets, are created. These sockets are used to communicate requests and receive responses via the administrative interface. Unix domain sockets function identically as INET domain sockets. The only differences are in how the socket is created and the domain in which is operates. Care should also be given as to who creates the socket. Following the next steps will allow the successful use of the administrative interface.

Wait until the server creates that the socket file (e.g., “/tmp/.xrootd/admin”). This can be done by polling for the socket using stat().
Create a stream socket using socket(PF_UNIX, SOCK_STREAM, 0)
Properly fill out the sockaddr_un structure with the path name of the socket (e.g., “/tmp/.xrootd/admin”). This structure is normally defined in the <sys/un.h> include file.
Issue a connect() call to connect the newly created socket to the path.
Use write() to issue requests to the server and read() to read responses.

The Xrootd Protocol Reference defines the administrative protocol used for the socket interface.

2.2 allow

xrd.allow { host | netgroup } name

Function

Restrict the hosts that can connect to xrootd.

Parameters

host name

The DNS host name allowed to connect to xrootd. Substitute for name a host name or IP address. A host name may contain a single asterisk anywhere in the name. This lets you allow a range of hosts should the names follow a regular pattern. . IP addresses may be specified in IPV4 format (i.e. “a.b.c.d”) or in IPV6 format (i.e. “[x:x:x:x:x:x]”).

netgroup name

The NIS netgroup allowed to connect to xrootd. Substitute for name a valid NIS netgroup. Only hosts that are members of the specified netgroup are allowed to connect to xrootd.

Defaults

None. If allow is not specified, any host is allowed to connect.

Notes

1) You may specify any number of hosts and netgroups. Any host matching a specified name or is a member of a specified netgroup is allowed to connect to xrootd.

2) Warning! Using hostname based security relies on the security of the DNS server and the inability of other hosts spoofing and successfully using the “allowed” IP addresses. The two security assumptions have severe limitations.

Example

xrd.allow host objyana*.slac.stanford.edu

3.2 network

xrd.network [buffsz blen[k | m | g]] [cache sec]

[[no]dnr] [kaparms idle[,itvl[,cnt]]

[[no]keepalive]

[routes {split|common|local} [use if1[,if2]]]

[[no]rpipa] [wan]

Function

Specify network parameters.

Parameters

buffsz blen

The buffer size to be set for each connected socket. The blen can be suffixed by k, m, or g to indicate kilo-, mega-, or giga-bytes; respectively. The default is determined by the operating system.

cache sec

The maximum number of seconds that an address to hostname translation can be locally cached for future use. The sec can be suffixed by s, m, h or d to indicate seconds, minutes, or hours, or days; respectively. The default is 3 hours.

dnr Uses Domain Name Resolution to convert IP addresses to host name for connecting clients. Host names are displayed in various messages.

nodnr Avoids using Domain Name Resolution to convert IP addresses to host name for connecting clients. Client IP addresses are displayed in various messages.

kaparms idle[,itvl[,cnt]

Specifies TCP keepalive parameters. The kaparms option is only effective for Linux. Up to three parameters may be specified. Omitted parameters, as well as parameter values of zero, use the system default. The parameters are:

idle The time the connection needs to remain idle before TCP starts sending keepalive probes. The idle value may be optionally suffixed by m for minutes, s for seconds (the default), or h for hours.

itvl The time between individual keepalive probes. The itvl value may be optionally suffixed by m for minutes, s for seconds (the default), or h for hours.

cnt The maximum number of keepalive probes TCP should send before dropping the connection.

keepalive

Uses the operating system’s keep-alive mechanism to determine whether or not a client is still connected to the daemon. This is the default. See the usage notes for other ways of simulating keepalive.

nokeepalive

Does not use the operating system’s keep-alive mechanism to determine whether or not a client is still connected to the daemon.

routes

Specifies that a dual network exists and how public and private addresses are routed within a site. Select one of the below optionally followed by the use keyword and up to two interface names (i.e. if1 and optionally if2 immediately preceded by a comma; use ifconfig to display interfaces and their names):

split Two separate networks exist. Clients connecting with a private address can only be redirected to a server’s private address. Clients connecting with a public address can only be redirected to a server’s public address. If clients can use both types of addresses, all servers must be dual homed with a public and private address. Typically, you must specify the interfaces you are using.

common Two common networks exist. Clients connecting with a private address are preferentially redirected to a server’s private address but may be redirected to a server’s public address if need be. However, clients connecting with a public address can only be redirected to a server’s public address. All servers must have at least a public address. If the machine is dual-homed you must specify the interfaces you are using.

local Two cross-routable networks exist. Clients connecting with a private address are preferentially redirected to a server’s private address but may be redirected to a server’s public address if need be. Clients connecting with a public address are preferentially redirected to a server’s public address but may be redirected to a server’s private address if need be. This is the default mode of operation in order to be compatible with previous releases. If the machine is dual-homed it is advisable to specify the interfaces you are using for predictable access.

Warning, such a network configuration is not suitable for external access and a proxy server must be used for out of domain clients.

rpipa

Tries to resolve private IP addresses to host names. See the section on resolving private IP address for more information.

norpipa

Does not resolve private IP addresses to host names. This is the default.

wan Indicates that the subsequent specifications apply to the special Wide Area Network port. This option is currently not supported by newer clients.

Defaults

xrd.network cache 3h dnr norpipa

Notes

1) For systems that support TCP buffer auto-tuning as a manual option, specify a buffsz blen of 0 to turn on auto-tuning.

2) Setting the buffer size to a large value may cause the operating system’s default value to be used. You should determine the maximum valid value for your system before specifying values greater that 64k.

3) Normally, the best performance is obtained by using TCP buffer auto-tuning.

4) Even if you specific nodnr, domain Resolution may be forced on if you specify an allow directive using a host name or host name fragment or authorize file access via host names, netgroups, or domain names. Resolution is still optimized by caching the results for future use.

5) The daemon’s internal timeout mechanism can be used to discover unconnected clients instead of TCP keepalive and may be more responsive. See the timeout directive. You should avoid using both mechanisms at the same time.

6) For aggressive keepalive processing you can use “kaparms 300,10,6”.

7) The wan port is created only if there is a protocol that can handle wan connections. See the protocol directive.

8) By default, public-private networking support is disabled. You must specify the routes option to enable it. If you do specify for one server you must specify for all nodes in your cluster (i.e. servers and redirectors). Failure to do so may result in unreachable nodes..

9) If the supplied information is inconsistent with the server network settings, warning messages are printed and the public-private network support may be turned off for that server. You should verify that your specification is consistent with the server’s networking configuration.

10) Interface names are arbitrary but usually are of the form enx, ethx, etc. One interface must be assigned a public address and the other the private address. This mechanism works best when all of the servers in a cluster use the same interface naming conventions; though the address assignments may differ. If this is not the case, you will need to use the if-else-fi configuration syntax to special case particular nodes.

11) Dual public/private network are fully supported in Linux, MacOS, and in Solaris 11.

Example

xrd.network nokeepalive nodnr

xrd.network wan buffsz 512k

3.2.1 Dual Public/Private Network Guidelines

Before you configure a dual public/private network, determine if such a network is actually necessary. Administering dual networks is difficult and problem resolution rather onerous. Typical reasons and alternatives to running a dual network are:

· Conserving IPV4 addressees: consider using IPV6, which does not suffer from this problem.

· Performance: if private addresses are routed along with public addresses over the same networking hardware then the switch becomes the bottleneck and performance improvements are moot.

· Security: if you desire to restrict certain kinds of access you may be able to achieve the same result using router and switch settings (e.g. file walls, routing restrictions, etc) thus avoiding a dual network.

If you still wish to run a dual network, you should use the following guidelines to avoid access surprises and mysterious performance issues.

· Never register private addresses in a publicly accessible DNS server. This exposes your private network configuration and is considered bad practice. Private addresses should only be registered in a private DNS or zone registered to prevent external leakage.

· Never register a server’s private and public address under a single host name unless both addresses have equal connectivity. While this practice works for simple applications, complex applications like XRootD and Proof attempt to use all addresses assigned to a particular host name. If connectivity is unequal, performance issues and connection failures are likely to occur.

· Never cluster servers with only a public address with servers that only have a private address if their name spaces overlap. While this may work for certain network routing topologies, it invariably introduces inconsistencies.

· Carefully review the routes option on the xrd.network directive to make sure your network routing topology and interfaces are correctly stated.

3.2.2 Resolving Private IP Addresses

By default, private IP addresses are not resolved to host names. This is commonly accepted practice to avoid DNS timeouts as private addresses are usually not registered in DNS. However, some installations may opt to register such addresses in either a private DNS or in a zoned public DNS. This actually may be necessary in cloud deployments where nodes within the cloud receive private addresses and a NAT box is installed to provide outside access. In such cases, here is usually a 1-to1 mapping between a public address (e.g. IPv6) to a corresponding private address (e.g. IPv4). The public address is registered in a public DNS while the private address is registered in a private DNS within the cloud. Both addresses are registered with the same host name. However, to avoid leaking private addresses you must specify the xrd.network rpipa option so that only host names are returned and not actual addresses. This allows clients on the public network to accesses nodes in the private network using the NAT box. Nodes in the private network also receive host names but since these names are registered within the private network, proper address resolution occurs.

3.5 sched

xrd.sched parms

parms: [avlt avlt] [core {asis | max | off}]

[idle idle] [maxt maxt] [mint mint] [stksz size]

Function

Specify when threads are created, how many can be created, and when they should be destroyed.

Parameters

avlt avlt

The number of threads that must always be available to service a request. These threads are never bound to any connection. Excess threads above this quantity will be allowed to bind with a socket until either the socket becomes idle or the number of available threads falls under avlt. The default is one half of the mint value.

core {assis | max | off}

Sets the limit for core file production. Choices are

assis - leave the current setting alone.

max - allow core files up the hard limit maximum (the default).

off - turn off core file production.

idle idle

The interval between checks for underused threads. Underused threads in excess of the mint value are terminated. Specify a number, optionally suffixed by m for minutes, s for seconds (the default), or h for hours. The default is every 13 minutes (i.e., 13m). Specifying a value of zero prevents threads from being terminated even if they are idle.

maxt maxt

The maximum number of threads that may be created to service requests. The number of threads will dynamically vary between mint and maxt.

mint mint

The minimum number of threads that must exist to handle requests. Once this number has been created, it is never reduced.

stksz size

The default thread stack size. Specify the number of bytes, optionally suffixed by k for kilobytes or m for megabytes. The default is controlled by the target operating system.

Defaults

xrd.sched mint 8 maxt 2048 avlt 512 idle 780

Notes

1) The mint-number of threads are eventually created.

2) The stack size is controlled by the target operating system used to create xrootd. For instance, in Solaris when sizeof(long) is 4 (indicating a 32-bit architecture), a 1M stack size is used. When sizeof(long) is 8 (indicating a 64-bit architecture or an LP64 data model) a 2M stack size is used. In Linux, the stack grows, as needed, to a maximum of 2M.

3) You should periodically review whether or not you have sufficient number of threads. The daemon prints a warning message the first time the maxt value is reached and more threads are needed.

4) If the daemon indicates that the thread limit was reached but less than maxt threads were created; then the target operating system maximum has been reached. This now becomes the new maxt value.

5) Warning, you should not change the thread parameters unless there is an overpowering reason to do so. The system is optimized for having many thread readily available. Constraining the number of threads may yield random failures that are hard to explain.

Example

xrd.sched mint 10 maxt 100 avlt 20

3.6 sitename

all.sitename sname

Function

Specify the site name to be included in monitoring records.

Parameters

sname

The 1- to 63-character name of the site. The name may include letters, digits, and the symbols “_-:.”. Other symbols are converted to a period. If the name is longer than 63 characters it is truncated to 63 characters.

Defaults

None.

Notes

1) The –S command line option overrides the sitename directive.

2) The first sitename directive takes precedence over any subsequent sitename directives.

Example

all.sitename slac

5 Esoteric xrootd Configuration Directives

5.1 async

xrootd.async parms

parms: [force] [limit clim] [maxsegs smax]

[maxstalls mstall] [maxtot slim]

[minsz reqsz[k | m | g]] [minsfsz sfsz[k | m | g]]

[off] [nosf] [segsz segsz[k | m | g]] [syncw]

Function

Specify how asynchronous I/O is to be handled.

Parameters

force Uses asynchronous I/O for all requests, even if the client did not ask for asynchronous handling.

limit clim

The maximum allowed number of outstanding asynchronous requests per client connection. Any additional requests past clim are synchronously handled. The default is eight (8).

maxsegs smax

The maximum number of simultaneous asynchronous operations that may any one request may have in progress. The default is eight (8).

maxstalls mstall

The maximum number of times a client may fail to deliver data at a sufficient rate to keep up with asynchronous I/O needs before future requests from the client are synchronously handled. Asynchronous handling is tried again after mstall number of synchronously handled requests. The default is eight (8).

maxtot slim

The maximum number of simultaneous asynchronous operations the server may have in progress. The default is 4,096.

minsz reqsz

The minimum number of bytes that must be read or written in a single client request for that request to be asynchronously handled. I/O requests smaller than reqsz are always synchronously handled. The reqsz can be suffixed by k, m, or g to indicate kilo-, mega-, or giga-bytes; respectively. The default is one half of the initial segsz.

minsfsz sfsz

The minimum number of bytes that must be read in a single client request for that request to be handled using sendfile(). I/O requests smaller than sfsz are always handled in the standard way. The sfsz can be suffixed by k, m, or g to indicate kilo-, mega-, or giga-bytes; respectively. The default 8192 for Linux, 1 otherwise.

off Disables asynchronous I/O for all requests.

nosf Disables using sendfile(), where available, for all read requests.

segsz segsz

The ideal asynchronous I/O segment size. The server attempts to quantize asynchronous I/O request into segsz pices. The segsz can be suffixed by k, m, or g to indicate kilo-, mega-, or giga-bytes; respectively. The default is 64k. See the usage notes on how the system dynamically adjusts this value.

syncw

Uses synchronous I/O for all fsync requests. Otherwise, asynchronous I/O is used for fsync requests if the client requested asynchronous I/O or if the force has been specified.

Defaults

xrootd.async limit 8 maxsegs 8 maxstalls 5 maxtot 4096 segsz 64k

Notes

1) Asynchronous requests allow the client to start a number of read operations at one time and wait for the request to complete in optimal order. When properly employed, asynchronous requests may substantially improve overall client processing speed.

2) Asynchronous processing represents a substantial resource commitment on part of the daemon. Each operation requires the dispatching of a separate thread. Rampant asynchronous processing may exhaust resource limits. Use the maxpl and maxps values to limit the amount of asynchronous processing.

3) Asynchronous processing is effective when the disk transfer rate approaches the network transfer rate. Thus, asynchronous processing is enabled only when a sufficiently large amount of data is requested by the client at one time. Use the minsz parameter to control the point where asynchronous operation is effective.

4) The segsz parameter specifies the ideal i/o size for asynchronous operations in order to maintain continuous disk/network transfer overlap. Requests are broken into segsz units. If the requested size is smaller than segsz, the server will use half the segsz. If the number of units is more than twice maxsegs, the server will use twice the segsz.

5) Conversion of asynchronous requests to synchronous requests is transparent to the client.

6) The sendfile() interface allows data to be transferred directly from the kernel’s file system memory cache to a client. Generally, this significantly reduced system overhead. The sendfile() function is available in Solaris 5.8 and above, as well as Linux 2.6 and above.

7) Use the nosf option in cases where you suspect that the sendfile() interface is causing data transfer problems.

Example

xrootd.async minsz 1M

5.2 chksum

xrootd.chksum [chkcgi] [max num] digest [path [args]]

digest: algorithim [digest]

Function

Specify how file check sums are computed.

Parameters

chkcgi

Always checks the cgi information, if any, for the cks.type element that can be used to select a checksum algorithm. The cgi information is not checked if only one digest is specified for backward compatibility. It is always checked if more than one digest is specified, making chkcgi unnecessary. See the usage notes for more information.

num Maximum number of checksum calculations that may run at the same time. Specifying 0 prevents real-time check summing. See the notes for more information.

algorithm

The name of the checksum digest (e.g., md5) used for the check summing. Specify one or more supported digests, each separated by a space. The first algorithm becomes the default algorithm. See the usage notes on how multiple digests are supported,

path The absolute path of the program that computes the check sum. If path is not specified, checksums are internally performed.

args Initial arguments to be passed to the program identified by path, if any.

Defaults

The default max is 4; otherwise. If path is not specified, checksums are internally performed. File check summing is not supported unless the directive is specified.

Notes

1) When a client issues an xrootd query checksum request, the following steps are performed:

a. A checksum digest is selected as follows:

i. If a single algorithm is specified and chkcgi was not specified, the digest in the configuration file is used.

ii. If a single algorithm is specified and chkgi is specified, a cgi scan is made for cks.type and, if specified, its argument must match the single algorithm in the configuration file or an error results. In any case, the digest in the configuration file is used.

iii. If more than one algorithm has been specified, a cgi scan is made for cks.type and, if specified, its argument must match one of the specified algorithms in the configuration file or an error results. If there is a match, the cks.type argument is used as the desired digest. If cks.type is not found, then the first algorithm specified in the configuration file is used.

b. A check is made that the client has lookup privileges for the file and that a valid checksum has been recorded for the file. If both are true, that checksum is sent back to the client. If the client lacks lookup privileges, an access error is sent back to the client.

c. Since a checksum needs to be computed the max value applies. If it is zero, the client is told that the checksum is not available.

d. If the checksum is natively supported and no program path has been specified, a new checksum is locally computed and recorded for future queries. Otherwise, the program named in path is executed to compute a new checksum and it is not recorded for future queries. Of course, the program may record the checksum in some way for future queries.

e. Either the previously recorded checksum or the computed checksum is provided to the client.

2) Native checksums are adler32, crc32, and md5.

3) Use the ofs.ckslib directive to add new digests or improve the performance of the native digests.

4) Since computation of multiple checksums is CPU and memory intensive choose the max with circumspection. You can control memory usage via the ofs.cksrdsz directive.

5) The ofs directives are documented on the OFS/OSS reference manual.

6) When the program identified by path is invoked, it is passed the path to the file that is to be processed. The actual argument list varies depending on whether or not a single algorithm has been specified, as follows:

a. When a single algorithm has been specified, the program is passed the file path as the last argument and is the only argument if no args have been specified.

b. When multiple algorithms have been specified, the program is passed the file path as the last argument and the checksum algorithm name as the second to the last argument. If no args have been specified, these are the only two arguments that are passed.

7) The program must output on standard out a single checksum value; normally ending with a new-line (‘\n’) character and terminate with a status code of zero. If the program terminates with a non-zero status code or returns no output, the client’s request fails.

8) Upon success, the returned checksum value is passed back to the client, prefixed by the digest token, digest.

9) Warning: If an external checksum program is specified (i.e. path is specified), then neither the oss.localroot nor oss.namelib directives are applied to the logical file name before passing the file name to the specified program that computes the checksum. Hence, the program is responsible for converting a logical file name to a physical file name.

10) When checksums are natively computed (i.e., path is not specified), then the oss.localroot and oss.namelib directives are applied to the logical file name. The checksum is computed against the resulting physical file name.

11) The chkcgi option is provided for backward compatibility. In previous releases only one algorithm could be specified and cgi information was immaterial. This processing mode remains the default when only a single algorithm is specified. You may wish to verify that a client as not requesting an unsupported digest in the case where some servers support multiple checksums and others do not.

12) The administrator’s interface allows you to list and cancel checksum jobs. This applies to external as well as internal computation of the checksum.

13) When max is zero, checksums on demand are prohibited. This requires that checksums to be pre-computed. This can be done using the frm_admin chksum command. See the File Residency Manager reference.

Example

xrootd.chksum max 2 md5

5.3 diglib

xrootd.diglib * authpath

Function

Enable remote debugging via the digFS read-only file system.

Parameters

* Loads the built-in version of digFS; the only the version currently supported.

authpath

The path to the authorization file that describes who is allowed to access digFS and what kind of information they may view.

Defaults

Bu default, digFS is disabled.

Notes

1) The digFS provides a virtual read-only file system view of key information about xrootd and cmsd that is valuable to remotely debug system problems.

2) Since digFS exposes system information an authorization file describing access permissions is required. See the next section.

3) When diglib is specified, the /=/ directory is automatically exported and available to authorized users. You must not list /=/ in the all.export list.

4) The /=/ path always refers to local storage regardless of server role and is never subject to redirection.

5) Only close, dirlist, locate, open, read, and stat requests can be vectored to digFS. Other requests referring to /=/ are disallowed.

6) The digFS accepts configuration directives starting with dig. Refer to subsequent sections for a description of these directives.

Example

xrootd.diglib * /etc/xrootd/digauth.cf

5.3.1 Authorizing digFS Access

The file describing digFS access permissions is composed of newline delimited records. Each record describes a single entity that is authorized to access certain information. The format of each record is

info allow aprot ident

aprot: gsi | host | krb5 | pwd | sss | unix

Parameters

info Authorizes the entity described in the record to access certain information. Use the word all to allow access to all information. If you specify all, you can remove specific information by specifying subsequent information keywords prefixed by a minus sign. Alternatively, list the info keywords to enable access to the associated information described below.

Keyword	Information	Keyword	Information
conf	configuration file	logs	log files
core	core files	proc	process information (Linux only)

aprot Specifies the authentication protocol that must be used in order to use digFS. Only one protocol per entity description may be specified. Since the information provided by digFS is sensitive in nature you should use the strongest authentication protocol consistent with site policies. The following table lists the default protocols[1] from strongest to weakest. Additionally, the rightmost column lists the ident tags that can be successfully specified relative to that protocol since not every protocol identifies clients in the same way. See the XRootD security reference for more detailed information.

Protocol	Descritption	*Meaningful ident* Codes**
krb5	Kerberos Version 5	h n
gsi	Grid Security Infrastructure (i.e. x.509)	g h n o r
sss	Simple Shared Secret	g h n
pwd	Password	g h n o r
host	DNS resolved hostname	h
unix	NFS V2-Style authentication	g h n

ident Authorizes the aprot authenticated entity possessing the specified identity values access to info digFS information. Identity values are specified as key value pairs. The entity must match all the specified pairs in order to be granted access. An imbedded space in a value must be designated as a \s (i.e. two character sequence). Specifying an inappropriate key relative to an authentication protocol prohibits access. The following table describes the possible key value pairs.

Key	Value
g	Group name
h	Fully qualified hostname
n	Authentication-specific protocol client identity string (see notes)
o	Organization name
r	Role name

Notes

1) Valid entries in the authfile are used and syntactically incorrect entries are discarded. At least one valid entry must exist for digFS to be enabled.

2) If the modification time of the authfile changes outside of a 5 second window it is reprocessed. This allows you to modify the authfile on a running system. However, you must atomically update the file as follows:

a. Create a copy of the file.

b. Modify the copy as needed.

c. Rename the copy to be the original name (i.e. use mv).

Distributing a modified copy of the file to other hosts should also use rename to install the new authfile.

3) Each authentication protocol has a specific way of identifying a client. For instance, x.509 (i.e. gsi) uses distinguished name (i.e. dn). Depending on the security configuration the protocol-specific name may be mapped to a Unix name. If so, you must use the mapped name not the original name.

4) If an entity is associated with more than one group name then the specified group (i.e. g=) must match one of the associated group names.

5) Starting in version 4.2 you are able to enable digFS but prevent its use by simply not creating an authorization file or commenting out all authorization entries. This allows you to enable its use in real-time without restarting XRootD by either creating an authorization file or adding authorization lines to an existing file.

6) Prior to 4.2 you must have a valid authorization file with at least one authorization entry. However, that entry may be unsatisfiable. This also allows you to selective enable or disable digFS without an XRootD restart.

Example

all –core allow krb5 h=test.org n=xtestor

conf logs allow gsi g=atlas n=theuser

5.3.2 Optional digFS Directives

The digFS accepts the following directives in the configuration file.

5.3.2.1 addconf

dig.addconf path [ fname ]

Function

Add a configuration file reference to the digFS namespace.

Parameters

path The absolute path to a regular file that is to be added to “/=/conf/etc”. The name of the file will be the same as the last component of path unless fname is specified.

fname The name that is to appear in “/=/conf/etc” but refers to path.

Defaults

None.

Notes

1) The path is only added if it is readable by the xrootd server.

2) This directive allows you to make other server related configuration files available via digFS.

Example

dig.addconf /etc/sysconfig/xrootd

5.3.2.2 log

dig.log parm [ parm ]

parm: deny | grant | none

Function

Control the level of logging.

Parameters

parm The level of logging; specify one or more of:

deny - log file access denials

grant - log file access approvals

none - turn off logging

Defaults

dig.log deny grant

Notes

1) To enable logging of denials and approvals you must specify both deny and grant parameters.

Example

dig.log deny

5.3.3 Using digFS

The digFS file system can be accessed using standard file system applications. All information is rooted in the /=/ directory and follows a standard layout. The following table describes directory tree.

Directory	Subdirectory	Contents
/=/conf		Configuration files cmsd.cf and xrootd.cf
	/etc	Other site selected configuration files.
/=/core		Core files
	/cmsd	Directory holding cmsd core files.
	/xrootd	Directory holding xrootd core files.
/=/logs		Log files
	/cmsd	Directory holding cmsd log files.
	/xrootd	Directory holding xrootd log files.
/=/proc		/proc files (Linux only)
	/cmsd	Directory holding the cmsd proc files.
	/xrootd	Directory holding the xrootd proc files.

If the cmsd and the xrootd share the same configuration file the /=/conf/cmsd.cf and /=/conf/xrootd.cf will be identical. If they share the core file directory or the log file directory; the same files may appear in the cmsd and xrootd subdirectories. As other components are added to digFS, additional executable names may appear in each root subdirectory.

To find out your access rights, simply list the entries in the /=/ directory. Only the subdirectories for which you are authorized are displayed.

5.4 fslib

xrootd.fslib [throttle | [-2] path2] {default | [-2] path1}

Function

Specify the location of the file system interface layer.

Parameters

throttle

Loads “libXrdThrottle.so” to wrap the subsequent library specification.

path2 The path to the shared library that is to be used as the wrapper for the subsequent library specification.

-2 Uses version two of the interface for file system object instantiation.

default

Loads a built-in version of “libXrdOfs.so” as the file system implementation.

path1 The path to the shared library that contains an implementation of the Open File System (ofs) interface that xrootd is to use for file system specific operations (e.g., open, close, read, write, rename, etc).

Defaults

xrootd.fslib default

Notes

1) When you only specify the shared library filename, the library is located using the standard platform-dependent loader rules (e.g. well know places followed by the LD_LIBRARY_PATH envar setting).

2) The sfs interface allows you to provide an arbitrary file system implementation. It is documented in XrdSfsInterface.hh. Refer to this include file for differences between version 1 and 2 instantiation.

Example

xrootd.fslib /opt/xrootd/lib/libofs.so

5.5 fsoverload

xrootd.fsoverload [[no]bypass] [redirect target]

[stall sec]

target: host:port[%host:port]

host: dnsname | [ipv6addr] | ipv4addr

Function

Specify how to handle file system overload.

Parameters

[no]bypass

Specifies whether or not clients should be redirected to the client-specified forwarding destination when the file system indicates it is overloaded. This option is only meaningful for servers configured as forwarding proxies and is ignored if that is not the case. It is only applicable when the client specifies a forwarding destination.

redirect target

The name or address of the host and port number where clients are to be redirected when the file system indicates that it is overloaded. The target consists of one or two host:port specifications with the second separated by a percent sign (%). When a second host:port is specified, then clients connecting using a private IP address are redirected to the second host:port while clients connecting with a public IP address are redirected to the first host:port. If only one host:port is specified, all clients are redirected to that host.

stall sec

Specifies how long the client should be stalled when a redirect target is not available and the file system indicates that it is overloaded. After the stall, clients will re-issue the request. A value of zero passes back an overload error to the client when the file system indicates an overload.

Defaults

xrootd.fsoverload nobypass stall 33

Notes

1) The fsoverload directive is most effective for disk caching proxy servers. Refer to the “Proxy Storage Services Configuration Reference” for additional information on how to effectively use this directive.

2) Currently, the fsoverload directive only applies to file open requests. All other requests encountering a file system overload event fail with an overload error.

3) The bypass directive only applies to release 4.0 or higher clients. Older clients that specify a forwarding path are subject to the stall option should a file system overload event occur.

Example

xrootd.fsoverload bypass redirect foo.proxy.edu:1094

5.6 log

xrootd.log [-]levent [ [-]levent ] [• • •]

levent: all | disc | login

Function

Specify event logging options.

Parameters

levent Specifies the events to be logged level. One ore more events may be specified. The specifications are cumulative and processed left to right. Each event may be optionally prefixed by a minus sign to turn off the setting. Valid events are:

all logs all possible events, the default

disc disconnect events

login login events

Defaults

xrootd.log all

Notes

1) Events messages are routed to the xrootd log file.

Example

xrootd.log all -login

5.7 monitor

xrootd.monitor [ options ] dest [ dest ]

options: [all] [auth] [flush [io] intvl[m|s|h]]

[fstat intvl[m|s|h] [lfn] [ops] [ssq] [xfr cnt]]

[ident sec] [mbuff size[k] [rbuff size[k]]

[rnums cnt] [window intvl[m|s|h]]

dest: dest events host:port

events: [files] [fstat] [io] [info] [redir] [user]

Function

Enable I/O monitoring.

Parameters

all Automatically enables monitoring for all connections. If all is not specified, monitoring is only enabled upon client request.

auth includes authentication information along with user information, when user is specified and authentication has been configured.

flush [io] intvl

The maximum time event data may be internally buffered before it is sent to the monitoring destination. Specify a number optionally suffixed by h for hours, m for minutes, or s for seconds, the default. When io is specified, io event data is also subject to flushing. Otherwise, only non-io events are flushed. The default only applies to non-io events and is 10 minutes.

fstat intvl [lfn] [ops] [ssq] [xfr cnt]

Enables file activity monitoring using a special “f” stream. The intvl is the maximum time event data may be internally buffered before it is sent to the monitoring destination. Specify a number optionally suffixed by h for hours, m for minutes, or s for seconds. A value of zero disables the “f” stream. The intvl is also used as the basis for xfr event data. By default, only file open and close events are inserted into the stream. Additional information may be requested as follows:

lfn includes the user’s dictionary identifier along with the logical file name being opened in the open event record.

ops includes detailed operation count information along with minimum and maximum values in the close event record.

ssq includes the sum of squares count for read and write sizes in the close event record. Specifying ssq automatically includes ops. This option will impact server performance. See the notes for more information.

xfr cnt inserts the number of bytes read and written from each open file every intvl*cnt elapsed time. The cnt must be 1 or more.

ident sec

The number of seconds between each server identity transmissions (i.e., the ‘=’ map record). Specify a number optionally suffixed by h for hours, m for minutes, or s for seconds, the default. A value of zero transmits the identity only once at start-up time. The default is 1 hour (i.e. 3600 seconds).

mbuff size

The size of the monitoring datagram for file and I/O events. Specify no less than 1024 and no more than 64k as the maximum message size. The size can be suffixed by k to indicate kilo-bytes. The default size is 16k.

rbuff size

The size of the monitoring datagram for redirection events. Specify no less than 2048 and no more than 64k as the maximum message size. The size can be suffixed by k to indicate kilo-bytes. The default size is 32k.

rnums cnt

The number of redirection monitoring streams to start. Specify no less than 1 and no more than 8. The default size is 3.

window intvl

The monitoring window size. Data collected within the window is not differentiated by time. Thus, the window represents the undifferentiated sampling interval. Specify a number optionally suffixed by h for hours, m for minutes, or s for seconds, the default. The default is 60 seconds.

dest host:port

The name of the host where monitoring messages should be sent. The receiving port number must be specified after the colon. All monitoring messages are sent as datagrams (i.e., UDP protocol). The dest parameter must be specified as the last parameter. Up two destinations are allowed. By default, only file event information is sent. The actual events may be specified after the dest keyword. These events are:

files file-related request monitoring (i.e., open and close requests).

fstat the specified “f” stream information (i.e., open and close requests).

io I/O request monitoring (read and write requests plus files).

iov same as io above but also include details on readv vector elements.

info client specified monitoring data submitted using xrootd protocol.

redir redirection events.

user client login and disconnect events.

Defaults

While flush 10m ident 1h mbuff 8k rbuff 32k rnums 3 window 60 is in effect; monitoring is not enabled.

Notes

1) Use the monitor directive to enable statistical gathering of file event and I/O requests.

2) The fstat ssq option impacts performance since floating point operations must be carried out for each read and write request in order to accurately compute the sum of squares. The counts can be used to compute the standard deviation for read and write sizes. Do not specify this option unless there is a clear need for such information.

3) The fstat ssq counts are available on platforms that use IEEE 754 floating point format. The fstat ssq option is ignored on non-conforming platforms.

4) The io option reports individual seeks for each read and write request. While this data may be used to determine access patterns or used in I/O trace simulation studies, it reduces server performance by about 7% and generates a large amount of monitoring data. Normally, fstat provides sufficient information about client I/O efficiency at a much lower cost.

5) The flush parameter does not apply to monitor streams that include io event data unless io is specified. By default, monitor streams that include io event data are flushed only when the internal monitor buffer becomes full or when the user owning the stream being monitored disconnects.

6) You may specify two monitoring destinations. This allows you to isolate data high volume streams (i.e., io monitoring) and provide real-time display for low-volume streams (i.e., info, files, fstat, and user).

7) The all option forces monitor data to be collected for all connections. If all is not specified, each client must enable monitoring manually using the xrootd set request code (see the xrootd protocol specification). This allows selective monitoring and gives each client the opportunity to tag io monitor data with the relevant application name.

8) The iov option inserts a read entry for every element in a readv vector. This may explode the amount of monitoring information that is generated. By default, when only io is specified, a summary readv entry is placed in the monitoring stream.

9) Clients cannot enable monitoring that has not been enabled by the monitor directive.

10) Specifying a small datagram buffer size (e.g. less than 8k) increases the number of datagrams that need to be sent and, consequently, adds to server overhead. Large datagram buffer sizes reduce the number of datagrams as well as server overhead but increase memory utilization as each connection allocates a buffer.

11) Approximately 61 requests can fit into a 1K mbuff.

12) On average, 64 to 128 redirection events can fit into a 32K rbuff.

13) Increasing the number of redirection monitoring streams (rnums) reduces the bottlenecks in the monitoring path.

14) Specifying a small window increases the timing accuracy of any individual request entry at the expense of additional datagrams and significantly increased server overhead. Conversely, large window sizes reduce timing accuracy but also reduce server overhead.

15) Refer to the “Scalla Monitoring” reference for a detailed explanation on the datagram format used by the monitoring subsystem. It is especially important to understand the different between files, fstat, and io monitoring as the information overlaps and there is rarely a need to specify all three.

Example

xrootd.monitor all fstat 5m dest fstat datacoll:5050

5.9 prep

xrootd.prep parms

parms: [ keep ksec ] [ scrub time ] [ logdir ldir ]

Function

Specify how prepare request tracking is done.

Parameters

keep ksec

The time that prepare request tracking record are to be held. The time may be suffixed by s (the default), m , or h to indicate seconds, minutes, and hours, respectively. The default is 24 hours.

scrub time

The time between scrubs of the tracking log directory. The time may be suffixed by s (the default), m , or h to indicate seconds, minutes, and hours, respectively. The default is 1 hour.

logdir ldir

The absolute path of the directory that is to hold the preparation tracking records. A directory must be specified, otherwise preparation request tracking is disabled.

Defaults

None. Preparation request tracking is normally disabled. When a logdir directory is specified, the keep and scrub defaults of 24H and 1H apply, respectively.

Notes

1) This directive allows server to track prepare requests. When request tracking is enabled, each prepare is logged in the logdir directory. It then becomes possible to list the requests and cancel them, if need be.

2) Since there can be more than one redirecting xrootd server, prepare requests may be scattered across several servers. It is the client’s responsibility to collect information from each server in order to create a composite preparation request history.

3) Each server uniquely names the files in the logdir directory. When multiple xrootd redirecting servers exist, it is possible to collect full preparation history from any server, if the logdir directory is located in a shared file system (e.g., NFS).

4) When running multiple xrootd servers on the same machine, the instance name (-n command line option) is used to differentiate logdir directories among all instances by appending the instance name to the path.

Example

xrootd.prep keep 12H logdir /nfs/xrootd/preplog

5.10 redirect

xrootd.redirect target {[-]foper | [?] path [path […]]}

foper: {all | chmod | chksum | dirlist | locate | mkdir

| trunc} [[-]foper]

target: host:port[%host:port]

host: dnsname | [ipv6addr] | ipv4addr

Function

Specify request forwarding.

Parameters

target The name or address of the host and port number where clients are to be redirected based on the subsequent parameters. The target consists of one or two host:port specifications with the second separated by a percent sign (%). When a second host:port is specified, then clients connecting using a private IP address are redirected to the second host:port while clients connecting with a public IP address are redirected to the first host:port. If only one host:port is specified, all clients are redirected to that host.

foper Specifies which metadata operations are to be immediately redirected. One or more operations may be specified. The specifications are cumulative and processed left to right. Each operation may be optionally prefixed by a minus sign to turn off the setting. Valid operations are:

all redirect all possible operations

chmod redirect change mode requests

chksum redirect checksum requests

dirlist redirect directory content listing requests

locate redirect path location requests

mkdir redirect create directory requests

mv redirect rename requests

prepare redirect prepare requests that do not need file staging

prepstage redirect prepare requests that may need file staging

rm redirect file removal requests

rmdir redirect directory removal requests

stat redirect file attribute requests

trunc redirect file truncate requests using a path

path Specifies that when a file open request occurs on the specified path prefix, the client should be redirected to the specified host and port. One or more paths may be specified. However, no more than four different host-port combinations may be specified.

? path Specifies that any client operation on the specified path prefix that ends with a “not found” error (i.e., EONOENT) and has not been specifically covered by another redirect directive, the client should be redirected to the specified host and port. All subsequently specified paths, if any, on the line fall under the “not found” provision.

Defaults

xrootd.redirect -all

Notes

4) Request redirection is typically applicable to the cluster manager. Refer to the role directive in the “Clustering Configuration Reference” for additional information, especially on inter-related directives.

5) Normally, meta-data requests are performed on the local host. However, certain clustered environments may be controlled by a central manager that records the exact state of every file. In such environments, the central manager may perform meta-data requests. When the redirect directive is not specified, the client is directed to perform the operation on a single host, normally the one that has the file. When the request is redirected, the target host is responsible for performing the operation.

6) The redirect path prefixes are always matched from most- to least-specific prefix (i.e., longest to shortest).

Example

xrootd.redirect all -prepare

5.11 trace

xrootd.trace [-]option [ [-]option ] [• • •]

off | stall | redirect | request | response

Function

Specify execution tracing options.

Parameters

option The tracing level. One ore more options may be specified. The specifications are cumulative and processed left to right. Each option may be optionally prefixed by a minus sign to turn off the setting. Valid options are:

all selects all possible trace levels

debug traces internal activities for debugging purposes

emsg traces errors sent back to the client

fs traces file system requests

login traces login and authentication steps

mem traces memory management functions

none traces nothing

off a synonym for none

stall traces client deferrals due to resource limitations

redirect traces client redirections to other servers

request traces client request information

response traces request response information

Defaults

Tracing is disabled.

Notes

1) All tracing is enabled when the daemon is invoked with the –d option.

2) All previous trace settings are discarded when none or off is encountered.

Example

xrd.trace all -debug

6 Enabling HTTP Access

XRootD supports HTTP access via a protocol plugin. The HTTP protocol can run alongside of standard XRootD protocol without any interference; providing an additional access mode. To Enable HTTP access, add the following configuration parameter to the configuration file.

if exec xrootd

xrd.protocol http[:port] path/libXrdHttp.so [cfgfile]

Parameters

port The port number http is to use for incoming requests. Specify a number, the name of a TCP service, or the word any. If you do not specify port number, port 1094 is used. This is also the default port for XRootD protocol.

path The path to the shared library libXrdHttp.so that contains the code the implements the HTTP protocol.

cfgfile is the path to an external configuration file specific to HTTP. If not specified, the configuration file at daemon start-up is used.

Defaults

Not applicable.

Notes

1) You should surround the xrd.protocol directive with the shown if-fi if you are using a common configuration file for xrootd and cmsd daemons. Failure to do so will prevent the cmsd from starting.

2) All HTTP requests are bound by any “xrd.” and “xrootd.” Directives that exist in the start-up configuration file. Hence, HTTP access cannot exceed any restriction imposed by those directives.

3) Monitoring of HTTP requests is handled as if they are XRootD requests. Consequently, the monitoring stream includes all HTTP requests as well. However, monitoring information is tagged with the fact that the information was generated by the HTTP protocol.

4) While the default port number is the same as for XRootD; the framework directs each request to appropriate protocol and these requests are never intermixed. To avoid any confusion you may wish to use a more standard HTTP port number such as 8080. This is especially true if connection are made using HTTPS.

5) Additional configuration file directives specific to HTTP are always prefixed with “http.” and the following sections describe these directives.

6) HTTP support also includes WebDav support enabling a wider range of access abilities beyond simple get and post capabilities.

7) When HTTP is used in a clustered XRootD deployment, all servers in that deployment must have HTTP enabled. Failure to do so typically results in access failure when a client is redirected to a serer that holds the desired file but for which HTTP was not enabled.

8) Not all HTTP clients support all HTTP features. While the plugin does not violate the HTTP or WebDav standards, it does implement a wide range of allowable features (e.g. redirection on POST requests) that may not be supported by the HTTP client being used.

6.1 Enabling HTTPS

When a server is configured to use HTTPS, each server processes the client’s credentials from the connection. This allows the client to be authenticated and makes authorization possible. On the other hand, HTTPS is very resource demanding because it encrypts and decrypts all of the TCP traffic. Additionally, the process of establishing an encrypted connection requires several network inter-changes that increase connection latency which can be substantial on a wide area network.

While HTTP is much faster it is impossible to authenticate the client using HTTP. However, the HTTP plugin allows a client to initially connect with HTTPS, extract the authentication information, encode that information in a low-overhead encrypted security token and redirect the connection to use HTTP. This is known as HTTPS to HTTP conversion and is much less resource intensive. When a server is configured to do HTTPS to HTTP conversion, it always expects a security token to be present when a client connects via HTTP. If the token is missing or cannot be decrypted the connection is rejected. This mechanism provides a relatively secure authentication but at the expense of privacy as no traffic is encrypted past the authentication stage.

HTTPS to HTTP conversion is especially attractive in clustered environments where a client typically makes contact with a particular node (i.e. redirector) that then redirects the client to a particular server that holds the requested file. Using HTTPS everywhere incurs identical overhead at each contact point. This can be eliminated by using HTTPS at the initial contact point and converting HTTPS to HTTP for subsequent connections. This incurs the overhead just once.

When enabling HTTPS you should consider the following points:

If only HTTPS is configured, then the server only accepts HTTPS connections (see the cadir, cafile, cert and key directives).
If HTTPS to HTTP conversion is configured a server accepts an HTTPS connection or an HTTP connection that provides a valid security token (see the secretkey directive)
If self-conversion of HTTPS to HTTP is configured, the server unconditionally redirects any HTTPS incoming connections to itself; using HTTP and a security token (see the selfhttps2http directive). This allows greater performance for subsequent requests.

When using HTTP in an XRootD cluster, additional considerations apply on how the cluster redirector interacts with data servers in that cluster. The following table provides reasonable possibilities, depending on the degree of security that is desired.

Redirector Accepts	Server Accepts	Configuration	Remarks
HTTP	HTTP	The is the default	No security.
HTTPS	HTTP with security token	Specify cadir, cert, key, and secretkey directives.	Central authentication, fast unencrypted data access but higher CPU load in the redirector
HTTP	HTTPS	Specify cadir, cert, key directives only in data servers. Specify desthttps in redirectors.	Fast redirection, distributed authentication, slow encrypted data access
HTTP	HTTPS with self redirection using HTTP with security token	Specify cadir, cert, key, selfhttps2http and secretkey directives only in data servers. Specify desthttps in redirectors.	Fast redirection, distributed authentication, fast unencrypted data access
HTTPS	HTTPS	Specify cadir, cert, key directives in servers and redirectors.	Fully authenticated but authentication occurs twice, slow encrypted data access, resource consumption can be high

2) Specifying a secret key automatically enables HTTPS to HTTP conversion.

Example

http.secretkey 1agq45jkl3400vfghtqzz963

6.2.8 selfhttps2http

http.selfhttps2http {no | yes}

Function

Specify whether or not a server may redirect an HTTPS connection to itself using HTTP plus a security token.

Parameters

no A server should continue to use HTTPS for all communications. The word false and the number 0 are synonyms.

yes A server should convert an HTTPS session to an HTTP session by redirecting the client to itself using HTTP plus a security token. The word true and the number 1 are synonyms. You must also specify the secretkey directive.

Defaults

http.selfhttps2http no

Notes

1) This option is meant to control the level of data privacy that is desired. Normally, HTTPS connections are converted to HTTP connections after authentication information is extracted from the HTTP stream. This greatly reduces overhead as no data past the authentication stage has to be encrypted.

2) When an HTTPS connection is converted to an HTTP connection, the redirection includes a security token encrypted with the key specified by with the secretkey directive. The HTTP connection is only accepted if the token can be decrypted using the same key.

Example

http.selfhttps2http yes

http.listingdeny yes

6.3.3 listingredir

http.listingredir desturl

Function

Specify the node to which to redirect clients requesting a directory listing.

Parameters

desturl The redirection URL to use when a directory listing is requested.

Defaults

None.

Notes

None.

Example

http.listingredir http://hostwhichprovideslistings:80/

6.3.4 staticpreload

http.staticpreload url path

Function

Preload a static resource file into memory.

Parameters

url The URL naming a static resource (e.g. style sheet or icon).

path The path to the local file containing the resource.

Defaults

None.

Notes

1) Static resources named in a URL must start with “/static” in order to be recognized.

2) The contents of the static resource contained in path may not exceed 64K. If it does, it is truncated to 64K.

3) Typical static resources are URL resources ending with “.css” and “.ico”.

4) This directive is ineffective if the staticredir directive is specified.

Example

http.staticpreload http://static/mycss.css /etc/mycss

6.3.5 staticredir

http.staticredir newurl

Function

Preload a static resource file into memory.

Parameters

newurl The URL the client is to be redirected to when requesting a non-local or unsupported static resource.

Defaults

None.

Notes

1) The staticredir directive is only effective when a) embeddedstatic processing is disabled, or b) the resource is neither a content style sheet nor an icon.

Example

http.staticredir

Component	Purpose
acc	Access control (i.e., authorization)
cms	Cluster Management Services
ofs	Open File System
oss	Open storage system (i.e., file system implementation)
sec	Security authentication
xrd	Extended Request Daemon
xrootd	The xrootd protocol implementation.
http	The HTTP protocol implementation.
all	Applies the directive to all of the above components.

2.1 adminpath

2.2 allow

2.3 port

3.1 buffers

3.2 network

3.2.2 Resolving Private IP Addresses

3.3 protocol

3.4 report

3.5 sched

3.6 sitename

3.7 timeout

3.8 trace

4.1 export

4.2 seclib

5.1 async

5.2 chksum

5.3 diglib

5.3.2.1 addconf

5.3.2.2 log

5.4 fslib

5.5 fsoverload

5.6 log

5.7 monitor

5.8 pidpath

5.9 prep

5.10 redirect

5.11 trace

6.2.1 cadir (required or use cafile)

6.2.2 cafile (required or use cadir)

6.2.3 cert (optional)

6.2.4 desthttps

6.2.5 gridmap (optional)

6.2.6 key (optional)

6.2.7 secretkey

6.2.9 secxtractor

6.3.1 embeddedstatic

6.3.2 listingdeny

6.3.3 listingredir

6.3.4 staticpreload

6.3.5 staticredir