The Scalla-xrootd Protocol

Version 2.9.6

Andrew Hanushevsky

14-July-2009

Produced under contract DE-AC02-76-SFO0515 with the Department of Energy

An implementation is available under a BSD-style license allowing minimally restricted use.

2 Request/Response Protocol

2.1 Format of Client-Server Initial Handshake

When a client first connects to the XRootd server, it must perform a special handshake. This handshake will determine whether the client is communicating with an XRootd server or a rootd server.

The handshake consists of the client sending 20 bytes, as follows:

kXR_int32 0

kXR_int32 4 (network byte order)

kXR_int32 2012 (network byte order)

The first twelve bytes are zero. The next eight bytes correspond to a standard rootd server protocol request (i.e., kROOTD_PROTOCOL). Both, rootd and XRootd, servers will respond, as follows:

rootd Response XRootd Response

streamid: kXR_char smid[2]

status: kXR_unt16 0

msglen: kXR_int32 8 msglen: kXR_int32 rlen

msgtype: kXR_int32 2012 msgval1: kXR_int32 pval

msgval: kXR_int32 pval msgval2: kXR_int32 flag

Where:

smid is the initial streamid. The smid for the initial response is always two null characters (i.e., ‘\0’);

rlen is the binary response length (e.g., 8 for the indicated response).

pval is the binary protocol version number.

flag is additional bit-encoded information about the server; as follows:

kXR_DataServer - This is a data server.

KXR_LBalServer - This is a load-balancing server.

Notes

1) All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2) The first four bytes of the reply determine whether a client is communicating with rootd (has a value of 8) or XRootd (has a value of 0).

3) All twenty bytes must be received by the server at one time. All known TCP implementations will guarantee that the first message is sent intact if all twenty bytes are sent in a single system call. Using multiple system calls for the first message may cause unpredictable results.

2.2 Data Serialization

All data sent and received is serialized (i.e., marshaled) in three ways:

1. Bytes are sent unaligned without any padding,

2. Data type characteristics are predefined (see table below), and

3. All integer quantities are sent in network byte order (i.e, big endian).

XRootd Type	Sign	Bit Length	Bit Alignment	Typical Host Type
kXR_char8	unsigned	8	8	unsigned char
kXR_unt16	unsigned	16	16	unsigned short
kXR_int32	signed	32	32	long[1]
kXR_int64	signed	64	64	long long

Table 1: XRootd Protocol Data Types

Network byte order is defined by the Unix htons() and htonl() macros for host to network short and host to network long, respectively. The reverse is defined by the ntohs() and ntohl() macros. Many systems do not define the long long versions of these macros. XRootd protocol requires that the POSIX version of long long serialization be used, as defined in the following figures. The OS-dependent isLittleEndian() function returns true if the underlying hardware using little endian integer representation.

unsigned long long htonll(unsigned long long x)

{unsigned long long ret_val;

if (isLittleEndian())

{*( (unsigned long *)(&ret_val) + 1) =

htonl(*( (unsigned long *)(&x)));

*(((unsigned long *)(&ret_val))) =

htonl(*( ((unsigned long *)(&x))+1) );

} else {

*( (unsigned long *)(&ret_val)) =

htonl(*( (unsigned long *)(&x)));

*(((unsigned long *)(&ret_val)) + 1) =

htonl(*( ((unsigned long *)(&x))+1) );

}

return ret_val;

};

Figure 1: POSIX Host to Network Byte Order Serialization

unsigned long long ntohll(unsigned long long x)

{unsigned long long ret_val;

if (isLittleEndian())

{*( (unsigned long *)(&ret_val) + 1) =

ntohl(*( (unsigned long *)(&x)));

*(((unsigned long *)(&ret_val))) =

ntohl(*(((unsigned long *)(&x))+1));

} else {

*( (unsigned long *)(&ret_val)) =

ntohl(*( (unsigned long*)(&x)));

*(((unsigned long*)(&ret_val)) + 1) =

ntohl(*(((unsigned long*)(&x))+1));

}

return ret_val;

};

Figure 2: POSIX Network to Host Byte Order Serialization

More compact and efficient, though OS restricted (i.e., Solaris and Linux), versions of 64-bit network byte ordering routines are given in the following figure.

#if defined(__sparc) || __BYTE_ORDER==__BIG_ENDIAN

#ifndef htonll

#define htonll(x) x

#endif

#ifndef ntohll

#define ntohll(x) x

#endif

#else

#ifndef htonll

#define htonll(x) __bswap_64(x)

#endif

#ifndef ntohll

#define ntohll(x) __bswap_64(x)

#endif

Figure 3: Network and Host Byte Ordering Macros

2.3 Client Request Format

Requests sent to the server are a mixture of ASCII and binary. All requests, other than the initial handshake request, have the same format, as follows:

kXR_char streamid[2]

kXR_unt16 requestid

kXR_char parms[16]

kXR_int32 dlen

kXR_char data[dlen]

Where:

streamid

is the binary identifier that is associated with this request stream. This identifier will be echoed along with any response to the request.

requestid

is the binary identifier of the operation to be performed by the server.

parms are parameters specific to the requestid.

dlen is the binary length of the data portion of the message. If no data is present, then the value is zero.

data are data specific to the requestid. Not all requests have associated data. If the request does have data, the length of this field is recorded in the dlen field.

Notes

2) All XRootd client requests consist of a standard 24-byte fixed length message. The 24-byte header may then be optionally followed by request specific data.

3) Stream id’s are arbitrary and are assigned by the client. Typically these id’s correspond to logical connections multiplexed over a physical connection established to a particular server.

4) The client may send any number of requests to the same server. The order in which requests are performed is undefined. Therefore, each request should have a different streamid so that returned results may be paired up with associated requests.

5) Requests sent by a client over a single physical connection may be processed in an arbitrary order. Therefore the client is responsible for serializing requests, as needed.

2.3.1 Valid Client Requests

The following table lists all possible requests and their arguments. Grayed rows represent requests that are not currently supported.

Requestid	Login?	Auth?	Redirect?	Arguments
kXR_admin	yes	yes	no	args
kXR_auth	y	n	n	authtype, authinfo
KXR_bind	n	n	n	sessid
kXR_chmod	y	y	yes	mode, path
kXR_close	y	y	n	fdnum
KXR_dirlist	y	y	y	path
KXR_endsess	y	y	n	sessid
kXR_getfile	y	y	y*	path
kXR_locate	y	y	y	path
kXR_login	n	n	n	userid, token
kXR_ls	y	y	y	options path
kXR_mkdir	y	y	y	mode, path
kXR_mv	y	y	y	old_name, new_name
kXR_open	y	y	y*	mode, flags, path
kXR_ping	y	n	n
kXR_prepare	y	y	n	paths
kXR_protocol	n	n	n
kXR_putfile	y	y	y*	mode, flags, path
kXR_query	y	y	n	args
kXR_read	y	y	y	fdnum, length, offset
kXR_readv	y	y	y	fdnum, length, offset
kXR_rm	y	y	y	path
kXR_rmdir	y	y	y	path
kXR_set	y	y	n	info
kXR_stat	y	y	y	path
kXR_statx	y	y	n	pathlist
kXR_truncate	y	y	y	Fdnum , length, offset, path
kXR_write	y	y	y	fdnum, length, offset, data
kXR_verifyw	y	y	y	fdnum, length, offset, data
Table 2: Valid Client Requests

2.3.2 Valid Client Paths

The XRootd server accepts only absolute paths where a path may be specified. Relative paths must be resolved by the client interface prior to sending them to XRootd. This means that the interface must handle a virtual “current working directory” to resolve relative paths should they arise.

Path names are restricted to the following set of characters:

Letters (upper or lower case),
Digits (0-9), and
Special characters: !@#%^_-+=:./

In general, paths may not contain shell meta-characters or imbedded spaces.

2.3.3 Client Recovery From Server Failures

A server failure should be recognized when the server unexpectedly closes it’s TCP/IP connection. Should this happen, the client may recover all operations by treating the termination of the connection as a redirection request (see page 30) to the initial XRootd server for all streams associated with the closed TCP/IP connections.

Because many clients are likely to be affected by a server failure, it is important that clients pace their reconnection to the initial XRootd server. One effective way to do this is to use the last three bits of the client’s IP address as the number of seconds to wait before attempting a reconnection. It is up to the client to determine either the number of times or the time window in which reconnections should be attempted before failure is declared. Typical values are 16 attempts or 3 minutes, whichever is longer.

Note that it may not be possible to recover in this way for files that were opened in update mode. Clients who do not provide proper transactional support generally cannot recover via redirection for any read/write resources.

2.3.4 Client Recovery From File Location Failures

A file location failure should be recognized when a server returns kXR_error status code with a kXR_NotFound error code and is either a load balancing server or the target of a redirect. The recovery steps are:

The client should contact the last load balancer used.
For kXR_Open requests, the request should be re-issued with the kXR_refresh option set. For other requests, no recovery is currently possible.
If the same result is encountered again, the client should consider the file missing and not attempt any further recovery actions.

2.4 Server Response Format

All responses, including the initial handshake response, have the same format, as follows:

kXR_char streamid[2]

kXR_unt16 status

kXR_int32 dlen

kXR_char data[dlen]

Where:

streamid

is the binary identifier that is associated with this request stream corresponding to a previous request.

status is the binary status code indicating how the request completed. The next section describes possible status codes.

dlen is the binary length of the data portion of the message. If no data is present, then the value is zero.

data are data specific to the requestid. Not all responses have associated data. If the response does have data, the length of this field is recorded in the dlen field.

Notes

2) Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3) Unsolicited responses are server requests for client configuration changes to make better use of the overall system. Since these responses do not correspond to any request, the streamid value has no meaning.

4) Unsolicited responses must be immediately acted upon. They should not be paired with any previous request.

2.4.1 Valid Server Response Status Codes

The following table lists all possible requests and their arguments. Grayed rows represent requests that are not currently supported.

Status	Response Data
kXR_attn	Parameters to direct immediate client action
kXR_authmore	Authentication specific data
kXR_error	Error number and corresponding ASCII message text
kXR_ok	Depends on request (this is predefined to be the value 0)
KXR_oksofar	Depends on request
kXR_redirect	Target port number and ASCII host name
kXR_wait	Binary number of seconds and optional ASCII message
kXR_waitresp	Binary number of seconds

Notes

1) Any request may receive any of the previous status codes.

2) The following sections detail the response format used for each status code.

2.4.2 Server kXR_attn Response Format

kXR_char pad[2]

kXR_unt16 kXR_attn

kXR_int32 plen

kXR_int32 actnum

kXR_char parms[plen-4]

Where:

plen is two bytes of padding required by the standard response format. These two bytes can be ignored for this particular response code.

plen is the binary length of the parms portion of the message (i.e., the subsequent bytes).

actnum

is the binary action code describing the action that the client is to take. These are:

kXR_asyncav - The file or file(s) the client previously requested to be

prepared are now available.

kXR_asyncab - The client should immediately disconnect (i.e., close

the socket connection) from the server and abort further

execution.

kXR_asyncdi - The client should immediately disconnect (i.e., close

the socket connection) from the server. Parameters

indicate when a reconnect may be attempted.

kXR_asyncgo - The client may start sending requests. This code is sent

to cancel the effects of a previous kXR_asyncwt code.

kXR_asyncms - The client should send the indicated message to the

console. The parameters contain the message text.

kXR_asyncrd - The client should immediately disconnect (i.e., close the

socket connection) and reconnect to the indicated

server.

kXR_asynresp - The client should use the response data in the message to complete the request associated with the indicated streamid.

kXR_asynunav - The file or file(s) the client previously requested to be

prepared cannot be made available.

kXR_asyncwt - The client should hold off sending any new requests

until the indicated amount of time has passed or until

receiving a kXR_asyncgo action code.

parms is the parameter data, if any, that is to steer client action.

Notes

2) Servers use the kXR_attn response code to optimize overall system performance and to notify clients of any impending events. All responses except for kXR_asynresp, do not correspond to any client request and should not be paired up with any request.

3) When kXR_attn is received, the client must perform the requested action and indicated by the actnum value.

2.4.2.1 Server kXR_attn Response for kXR_asyncab Client Action

kXR_char pad[2]

kXR_unt16 kXR_attn

kXR_int32 mlen

kXR_int32 kXR_asyncab

kXR_char msg[mlen-4]

Where:

mlen is the binary length of the following action code and message.

msg is the message to be sent to the terminal. The mlen value, less four, indicates the length of the message. The ending null byte (‘\0’) is transmitted and included in the message length.

Notes

2) Servers use the kXR_attn response code to optimize overall system performance and to notify clients of any impending events. This response does not correspond to any client request and should not be paired up with any request.

3) When kXR_attn is received with the kXR_asyncab action code, the client should close all physical connections, write the message (msg), if any, to standard error, and terminate execution.

2.4.2.2 Server kXR_attn Response for kXR_asyncdi Client Action

kXR_char pad[2]

kXR_unt16 kXR_attn

kXR_int32 12

kXR_int32 kXR_asyncdi

kXR_int32 wsec

kXR_int32 msec

Where:

wsec is the number of seconds the client should wait before attempting to reconnect to the server.

msec is the maximum number of seconds the client should wait before declaring reconnect failure.

Notes

3) When kXR_attn is received with the kXR_asyncdi action code, the client should close the physical connection, wait wsec seconds, and attempt to reconnect to the server.

4) If a server reconnect fails, the client should wait either an additional wsec seconds or some other predetermined time and try again. If msec seconds have gone since the initial wait and the client has not reconnected to the server, a reconnect failure should be declared.

5) When a reconnect failure is declared, the client may either terminate the program or perform an internal redirection to a load-balancing server.

6) A reconnect is essentially a delayed redirect to the same server. The actions that must be carried out when reconnecting are identical to those that must be performed when reconnecting to a different server. Refer to the description of the kXR_asyncrd action for the set steps that the client must take to successfully reconnect.

2.4.2.3 Server kXR_attn Response for kXR_asyncgo Client Action

kXR_char pad[2]

kXR_unt16 kXR_attn

kXR_int32 4

kXR_int32 kXR_asyncgo

Notes

3) When kXR_attn is received with the kXR_asyncgo action code, the client may resume sending requests to the server.

4) The kXR_asyncgo code is sent to cancel the effects of a previously sent kXR_asyncwt code. Therefore, if the client is still waiting for the kXR_asyncwt interval to expire, the interval should be cancelled.

2.4.2.4 Server kXR_attn Response for kXR_asyncms Client Action

kXR_char pad[2]

kXR_unt16 kXR_attn

kXR_int32 mlen

kXR_int32 kXR_asyncms

kXR_char msg[mlen-4]

Where:

mlen is the binary length of the following action code and message.

msg is the message to be sent to the terminal. The mlen value, less four, indicates the length of the message. The ending null byte (‘\0’) is transmitted and included in the message length.

Notes

3) When kXR_attn is received with the kXR_asyncms action code, the client should simply write the indicated message to the terminal.

2.4.2.5 Server kXR_attn Response for kXR_asyncrd Client Action

kXR_char pad[2]

kXR_unt16 kXR_attn

kXR_int32 plen

kXR_int32 kXR_asyncrd

kXR_int32 port

kXR_char host[?token][plen-8]

Where:

plen is the binary length of the parameter portion of the message (i.e., the subsequent bytes).

port is the binary port number to which the client must connect. If the value is zero, the default XRootd port number must be used.

host is the ASCII name of the to which the client must connect. The host does not end with a null (\0) byte.

token is an optional ASCII token that, when present, must be delivered to the new host during the login phase. The token, if present, is separated from the host by a single question mark. The token does not end with a null (\0) byte.

Notes

3) When kXR_attn is received with the kXR_asyncrd action code, the client should perform the following steps:

a. Decompose the response to extract the port number, host name, and possible token value.

b. Physically close the connection to the current host, regardless of type.

c. Establish a new physical connection with the indicated host at the specified or default port number.

d. Perform the initial handshake, login with token (see kXR_login description), and authentication (see kXR_auth description).

e. Re-establish all open files, as needed. Previously opened files may be re-opened all at once or when a request attempts to use the file.

f. Re-issue any requests that were sent to the previous server but have not received a response.

4) Since XRootd allows multiple open files per physical connection, a kXR_asyncrd response can become somewhat complicated to handle. The client can re-open files immediately after a new connection is made or can re-open files as they are needed. In either case, the client must:

g. Issue a kXR_open request using the same file name and options as was originally used.

h. Use the returned file handle for all subsequent requests for that file (i.e., substitute the new fhandle for the old fhandle).

5) An XRootd server will never redirect a physical connection to a rootd server. This differs for logical connections where a logical connection may be so redirected.

6) After 256 redirect responses within 10 minutes on the same physical connection, the client should declare an internal system error since it is obvious that effective work is not being performed.

2.4.2.6 Server kXR_attn Response for kXR_asynresp Client Action

kXR_char pad[2]

kXR_unt16 kXR_attn

kXR_int32 rlen

kXR_int32 kXR_asynresp

kXR_char reserved[4]

kXR_char streamid[2]

kXR_unt16 status

kXR_int32 dlen

kXR_char data[dlen]

Where:

rlen is the binary length of the following action code and response.

streamid

is the stream identifier associated with a previously issued request that received a kXR_waitresp response.

status is the binary status code indicating how the request completed. The codes definitions are identical as to those described for synchronous responses.

dlen is the binary length of the data portion of the message. If no data is present, then the value is zero.

data are data specific to the request. Not all responses have associated data. If the response does have data, the length of this field is recorded in the dlen field.

Notes

2) Servers use the kXR_attn response code to optimize overall system performance and to notify clients of any impending events.

3) Unlike other asynchronous events, this response is associated with a previous request and the response data must be used to complete that request.

4) The rlen-dlen is always 16.

5) When kXR_attn is received with the kXR_asynresp action code, the client should remove the request paired with streamid from wait state and complete it using the response data.

2.4.2.7 Server kXR_attn Response for kXR_asyncwt Client Action

kXR_char pad[2]

kXR_unt16 kXR_attn

kXR_int32 8

kXR_int32 kXR_asyncwt

kXR_int32 wsec

Where:

wsec is the number of seconds the client should wait before sending any more requests to the server.

Notes

3) When kXR_attn is received with the kXR_asyncwt action code, the client should queue any new requests (i.e., not send new requests) until wsec seconds have elapsed.

4) While waiting, the client should still be receiving messages from the server. It is possible for the server to send additional unsolicited responses even after a kXR_asyncwt has been sent. For example, the server may send a kXR_asyncgo request to cancel the effects of the kXR_asyncwt request before the wsec interval has gone by.

2.4.3 Server kXR_authmore Response Format

kXR_char streamid[2]

kXR_unt16 kXR_authmore

kXR_int32 dlen

kXR_char data[dlen]

Where:

streamid

is the binary identifier that is associated with this request stream corresponding to a previous request.

dlen is the binary length of the data portion of the message (i.e., the subsequent bytes).

data is the data, if any, required to continue the authentication process.

Notes

2) Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3) The kXR_authmore response code is issued only for those authentication schemes that require several handshakes in order to complete (e.g., .x500).

4) When a kXR_authmore response is received, the client must call the appropriate authentication continuation method and pass it data, if present. The output of the continuation method should be sent to the server using another kXR_auth request. This handshake continues until either the continuation method fails or the server returns a status code of kXR_error or kXR_ok.

5) Refer to the description of the security framework for detailed information.

2.4.4 Server kXR_error Response Format

kXR_char streamid[2]

kXR_unt16 kXR_error

kXR_int32 dlen

kXR_int32 errnum

kXR_char errmsg[dlen-4]

Where:

streamid

is the binary identifier that is associated with this request stream corresponding to a previous request.

dlen is the binary length of the data portion of the message (i.e., the subsequent bytes).

errnum

is the binary error number indicating the nature of the problem encountered when processing the request.

errmsg

is the human-readable null-terminated message that describes the error. This message may be displayed for informational purposes.

Notes

2) Since the error message is null-terminated, dlen includes the null byte in its count of bytes that were sent.

3) Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

2.4.4.1 Server kXR_error Sub-Codes

The following table lists possible error sub-codes included in the errnum field as part of the kXR_error response:

Status	Meaning
kXR_ArgInvalid	A request argument was not valid
kXR_ArgMissing	Required request argument was not provided
kXR_ArgTooLong	A request argument was too long (e.g., path)
kXR_Cancelled	The operation was cancelled by the administrator
kXR_ChkLenErr	The close length does not equal the file size
kXR_ChkSumErr	The kXR_verifyw checksum does not match
kXR_FileLocked