The XRootD Protocol

Version 4.0.0

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Andrew Hanushevsky

17-January-2020

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

©2004-2020 by the Board of Trustees of the Leland Stanford, Jr., University

All Rights Reserved

Produced under contract DE-AC02-76-SFO0515 with the Department of Energy

The protocol specification described in this document falls under BSD license terms.

The specification may be used for any purpose whatsoever.

Use of this specification must cite the original source -- xrootd.org.

Binary definition in header file XProtocol.hh superceed any such definitions in this document.

 

 


1         Contents

1       Contents. 3

2       Request/Response Protocol 7

2.1        Format of Client-Server Initial Handshake. 7

2.2        Data Serialization.. 9

2.3        Client Request Format 11

2.3.1         Valid Client Requests. 13

2.3.2         Valid Client Paths. 14

2.3.3         Client Recovery from Server Failures. 15

2.4        Server Response Format 17

2.4.1         Valid Server Response Status Codes. 18

2.4.2         Server kXR_attn Response Format 19

2.4.2.1      Server kXR_attn Response for kXR_asyncms Client Action.. 20

2.4.2.2      Server kXR_attn Response for kXR_asyninfo Client Action.. 21

2.4.2.3      Server kXR_attn Response for kXR_asynresp Client Action.. 23

2.4.3         Server kXR_authmore Response Format 25

2.4.4         Server kXR_error Response Format 26

2.4.4.1      Server kXR_error Sub-Codes & Recovery Actions. 27

2.4.5         Server kXR_ok Response Format 29

2.4.6         Server kXR_oksofar Response Format 30

2.4.7         Server kXR_redirect Response Format 31

2.4.8         Server kXR_wait Response Format 34

2.4.9         Server kXR_waitresp Response Format 35

2.5        Binary Definitions of Status and Error codes. 37

2.5.1         Status Codes. 37

2.5.2         kXR_attn Status Subcodes. 37

2.5.3         Error Codes. 38

3       Transport Layer Security (TLS) Support 39

3.1        Client-Server interactions to unilaterally use TLS. 40

3.2        Client-Server interactions to use TLS only when required.. 40

4       Server Request Format 41

4.1        kXR_auth Request 41

4.2        kXR_bind Request 43

4.2.1         TLS Considerations. 44

4.3        kXR_chkpoint Request 45

4.4        kXR_chmod Request 47

4.5        kXR_close Request 49

4.6        kXR_dirlist Request 51

4.7        kXR_endsess Request 54

4.8        kXR_fattr Request 55

4.8.1         Layout of namevec. 56

4.8.2         Layout of valuvec. 57

4.8.3         kXR_fattr Request – Delete Subcode. 59

4.8.4         kXR_fattr Request – Get Subcode. 61

4.8.5         kXR_fattr Request – List Subcode. 63

4.8.6         kXR_fattr Request – Set Subcode. 65

4.9        kXR_gpfile Request 67

4.10     kXR_locate Request 71

4.11     kXR_login Request 75

4.11.1      Additional Login CGI Tokens. 78

4.12     kXR_mkdir Request 79

4.13     kXR_mv Request 81

4.14     kXR_open Request 83

4.14.1      Additional Open CGI Tokens. 87

4.15     kXR_ping Request 89

4.16     kXR_pgread Request 91

4.16.1      Backward Compatability. 94

4.17     kXR_pgwrite Request 95

4.17.1      Backward Compatability. 97

4.17.2      Error Recovery. 98

4.18     kXR_prepare Request 99

4.19     kXR_protocol Request 103

4.19.1      Protocol Security Requirements vs Response Implications. 111

4.20     kXR_query Request 113

4.20.1      KXR_query Checksum Cancellation Request 117

4.20.2      KXR_query Checksum Request 119

4.20.2.1    Additional Query Checksum CGI Tokens. 120

4.20.3      KXR_query Configuration Request 121

4.20.3.1    Format for Query Config cms. 124

4.20.3.2    Format for Query Config role. 125

4.20.3.3    Format for Query Config xattrs. 125

4.20.4      KXR_query Opaque Request 127

4.20.5      KXR_query Space Request 129

4.20.6      KXR_query Statistics Request 131

4.20.7      KXR_query Visa Request 135

4.20.8      KXR_query Xattr Request 137

4.21     kXR_read Request 139

4.22     kXR_readv Request 143

4.23     kXR_rm Request 147

4.24     kXR_rmdir Request 148

4.25     kXR_set Request 149

4.25.1      Valid kXR_set Values. 151

4.26     kXR_sigver Request 153

4.26.1      Signing a request 155

4.26.2      Verifying a signed request 156

4.27     kXR_stat Request 157

4.27.1      Additional Stat CGI Tokens. 160

4.28     kXR_statx Request 161

4.29     kXR_sync Request 163

4.30     kXR_truncate Request 165

4.31     kXR_write Request 167

4.32     kXR_writev Request 169

5       The Security Framework. 171

5.1        Framework for Transport Layer Protocols. 175

5.2        Request Verification.. 176

6       Document Change History. 177

 


2         Request/Response Protocol

2.1       Format of Client-Server Initial Handshake

When a client first connects to the XRootD server, it must perform a special handshake. This handshake should determine whether the client is communicating using XRootD protocol or another protocol hosted by the server.

 

The handshake consists of the client sending 20 bytes, as follows:

 

kXR_int32

              0

 

kXR_int32

              0

 

kXR_int32

              0

 

kXR_int32

              4

(network byte order)

kXR_int32

        2012

(network byte order)

 

XRootD protocol, servers should respond, as follows:

 

streamid:

kXR_char

smid[2]

status:

kXR_unt16

0

msglen:

kXR_int32

rlen

msgval1:

kXR_int32

pval

msgval2:

kXR_int32

flag

 

Where:

 

smid     is the initial streamid. The smid for the initial response is always two null characters (i.e., ‘\0’);

 

rlen      is the binary response length (e.g., 8 for the indicated response).

 

pval     is the binary protocol version number.

 

flag      is additional bit-encoded information about the server; as follows:

            kXR_DataServer - 0x00 00 00 01 This is a data server.

            KXR_LBalServer - 0x00 00 00 00 This is a load-balancing server.

 


Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      The particular response format was developed for protocol version 2.0 and does not convey all of the information to capture features provided by subsequent protocol versions. In order to provide backward compatibility, this response format has been kept. The recommended mechanism to obtain all of the information that may be needed is to “piggy-back” a kXR_protocol Request with the handshake (i.e. send the handshake and the request with a single write).

3)      All twenty bytes must be received by the server at one time. All known TCP implementations should guarantee that the first message is sent intact if all twenty bytes are sent in a single system call. Using multiple system calls for the first message may cause unpredictable results.


 

2.2       Data Serialization

All data sent and received is serialized (i.e., marshaled) in three ways:

1.      Bytes are sent unaligned without any padding,

2.      Data type characteristics are predefined (see table below), and

3.      All integer quantities are sent in network byte order (i.e, big endian).

 

XRootD Type

Sign

Bit Length

Bit Alignment

Typical Host Type

kXR_char8

unsigned

 8

 8

unsigned char

kXR_unt16

unsigned

16

16

unsigned short

kXR_int32

  signed

32

32

long[1]

kXR_int64

  signed

64

64

long long

Table 1: XRootD Protocol Data Types

Network byte order is defined by the Unix htons() and htonl() macros for host to network short and host to network long, respectively. The reverse is defined by the ntohs() and ntohl() macros. Many systems do not define the long long versions of these macros. XRootD protocol requires that the POSIX version of long long serialization be used, as defined in the following figures. The OS-dependent isLittleEndian() function returns true if the underlying hardware using little endian integer representation.

 

 

unsigned long long htonll(unsigned long long x)

       {unsigned long long ret_val;

        if (isLittleEndian())

          {*( (unsigned long *)(&ret_val) + 1) =

                     htonl(*(  (unsigned long *)(&x)));

           *(((unsigned long *)(&ret_val))) =

                     htonl(*( ((unsigned long *)(&x))+1) );

           } else {

           *( (unsigned long *)(&ret_val)) =

                     htonl(*(  (unsigned long *)(&x)));

           *(((unsigned long *)(&ret_val)) + 1) =

                     htonl(*( ((unsigned long *)(&x))+1) );

           }

       return ret_val;

      };

 

Figure 1: POSIX Host to Network Byte Order Serialization


 

 

unsigned long long ntohll(unsigned long long x)

       {unsigned long long ret_val;

        if (isLittleEndian())

           {*( (unsigned long *)(&ret_val) + 1) =

                            ntohl(*( (unsigned long *)(&x)));

            *(((unsigned long *)(&ret_val))) =

                            ntohl(*(((unsigned long *)(&x))+1));

           } else {

            *( (unsigned long *)(&ret_val)) =

                            ntohl(*( (unsigned long*)(&x)));

            *(((unsigned long*)(&ret_val)) + 1) =

                            ntohl(*(((unsigned long*)(&x))+1));

           }

        return ret_val;

       };

 

Figure 2: Network and Host Byte Order Seialization

 

More compact and efficient, though OS restricted (i.e., Solaris and Linux), versions of 64-bit network byte ordering routines are given in the following figure.

 

 
#if defined(__sparc) || __BYTE_ORDER==__BIG_ENDIAN
#ifndef htonll
#define htonll(x) x
#endif
#ifndef ntohll
#define ntohll(x) x
#endif
#else
#ifndef htonll
#define htonll(x) __bswap_64(x)
#endif
#ifndef ntohll
#define ntohll(x) __bswap_64(x)
#endif
 

Figure 3: Network and Host Byte Ordering Macros


2.3       Client Request Format

Requests sent to the server are a mixture of ASCII and binary. All requests, other than the initial handshake request, have the same format, as follows:

 

kXR_char

streamid[2]

kXR_unt16

requestid

kXR_char

parms[16]

kXR_int32

dlen

kXR_char

data[dlen]

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

 

requestid

            is the binary identifier of the operation to be performed by the server.

 

parms  are parameters specific to the requestid.

 

dlen     is the binary length of the data portion of the message. If no data is present, then the value is zero.

 

data     are data specific to the requestid. Not all requests have associated data. If the request does have data, the length of this field is recorded in the dlen field.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      All XRootD client requests consist of a standard 24-byte fixed length message. The 24-byte header may then be optionally followed by request specific data.

3)      Stream id’s are arbitrary and are assigned by the client. Typically these id’s correspond to logical connections multiplexed over a physical connection established to a particular server.

4)      The client may send any number of requests to the same server. The order in which requests are performed is undefined. Therefore, each request should have a different streamid so that returned results may be paired up with associated requests.

5)      Requests sent by a client over a single physical connection may be processed in an arbitrary order. Therefore the client is responsible for serializing requests, as needed.


 

2.3.1        Valid Client Requests

 

Requestid

Value

Login?

Auth?

Redirect?

Arguments

kXR_auth

3000

y

n

n

authtype, authinfo

KXR_bind

3024

n

n

n

sessid

kXR_chkpoint

3012

y

-

n

fhandle, length, offset

kXR_chmod

3002

y

y

yes

mode, path

kXR_close

3003

y

-

n

fhandle

KXR_dirlist

3004

y

y

y

path

KXR_endsess

3023

y

-

n

sessid

kXR_fattr

3020

y

y

y

Arguments vary by subcode

kXR_gpfile

3005

y

optional

y

Arguments vary by subcode

kXR_locate

3027

y

y

y

path

kXR_login

3007

n

n

n

userid, token

kXR_mkdir

3008

y

y

y

mode, path

kXR_mv

3009

y

y

y

old_name, new_name

kXR_open

3010

y

y

y

mode, flags, path

kXR_pgread

3030

y

-

y

fhandle, pathid, length, offset

kXR_pgwrite

3026

y

-

y

fhandle, pathid, length, offset

kXR_ping

3011

y

n

n

 

kXR_prepare

3021

y

y

n

paths

kXR_protocol

3006

n

n

n

 

kXR_query

3001

y

y

y

args

kXR_read

3013

y

-

y

fhandle, pathid, length, offset

kXR_readv

3025

y

-

y

fhandle, pathid, length, offset

kXR_rm

3014

y

y

y

path

kXR_rmdir

3014

y

y

y

path

kXR_set

3018

y

y

y

info

kXR_sigver

3029

y

y

n

signature

kXR_stat

3017

y

-

n

fhandle

kXR_stat

3017

y

y

y

path

kXR_statx

3022

y

y

n

pathlist

kXR_sync

3016

y

-

n

fhandle

kXR_truncate

3028

y

-

n

fhandle, length

kXR_truncate

3028

y

-

y

path, length

kXR_write

3019

y

-

y

fhandle, pathid, length, offset, data

kXR_writev

3031

y

y

n

fhandle, length, offset

 

Table 2: Valid Client Requests

*


2.3.2        Valid Client Paths

The XRootD server accepts only absolute paths where a path may be specified. Relative paths must be resolved by the client interface prior to sending them to XRootD. This means that the interface must handle a virtual “current working directory” to resolve relative paths should they arise.

 

Path names are restricted to the following set of characters:

 

In general, paths may not contain shell meta-characters.

 

Any path may be suffixed by CGI information. The format corresponds to that defined in RFC 3875. However, the protocol does not allow URL encoded characters (i.e. %xx). The meaning of any CGI element that is not specified in this document is implementation specific.

 


 

2.3.3        Client Recovery from Server Failures

A server failure should be recognized when the server unexpectedly closes its TCP/IP connection or does not respond for an extended period of time. Should this happen, the client may recover all operations by treating the termination of the connection or unresponsiveness as a redirection request (see page 31) to the initial XRootD server for all streams associated with the closed TCP/IP connections.

 

The initial XRootD server is defined as the first manager or the last meta-manager encountered. In the absence of any manager, the first data server encountered. See the kXR_protocol request on how to determine a node’s type.

 

Because many clients are likely to be affected by a server failure, it is important that clients pace their reconnection to the initial XRootD server. One effective way to do this is to use the last three bits of the client’s IP address as the number of seconds to wait before attempting a reconnection. It is up to the client to determine either the number of times or the time window in which reconnections should be attempted before failure is declared. Typical values are 16 attempts or 3 minutes, whichever is longer.

 

Note that it may not be possible to recover in this way for files that were opened in update mode. Clients who do not provide proper transactional support generally cannot recover via redirection for any read/write resources.


2.4       Server Response Format

All responses, including the initial handshake response, have the same format, as follows:

 

kXR_char

streamid[2]

kXR_unt16

status

kXR_int32

dlen

kXR_char

data[dlen]

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream corresponding to a previous request.

 

status   is the binary status code indicating how the request completed. The next section describes possible status codes.

 

dlen     is the binary length of the data portion of the message. If no data is present, then the value is zero.

 

data     are data specific to the requestid. Not all responses have associated data. If the response does have data, the length of this field is recorded in the dlen field.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3)      Unsolicited responses are server requests for client configuration changes to make better use of the overall system. Since these responses do not correspond to any request, the streamid value has no meaning.

4)      Unsolicited responses must be immediately acted upon. They should not be paired with any previous request.

 


2.4.1        Valid Server Response Status Codes

The following table lists all possible responses and their arguments.

 

Status

Response Data

kXR_attn

Parameters to direct immediate client action

kXR_authmore

Authentication specific data

kXR_error

Error number and corresponding ASCII message text

kXR_ok

Depends on request (this is predefined to be the value 0)

KXR_oksofar

Depends on request

kXR_redirect

Target port number and ASCII host name

kXR_wait

Binary number of seconds and optional ASCII message

kXR_waitresp

Binary number of seconds

 

Notes

1)      Any request may receive any of the previous status codes.

2)      The following sections detail the response format used for each status code.


2.4.2        Server kXR_attn Response Format

 

kXR_char

pad[2]

kXR_unt16

kXR_attn

kXR_int32

plen

kXR_int32

actnum

kXR_char

parms[plen-4]

 

Where:

 

plen     is two bytes of padding required by the standard response format. These two bytes can be ignored for this particular response code.

 

plen     is the binary length of the parms portion of the message (i.e., the subsequent bytes).

 

actnum

            is the binary action code describing the action that the client is to take. These are:

            kXR_asyncms     -  The client should send the indicated message to the

                                             console. The parms contain the message text.

         kXR_asyninfo    -  The client should use the response data in the message to gauge the status of a in-progress request.

         kXR_asynresp    -  The client should use the response data in the message to complete the request associated with the indicated streamid.

 

parms          is the parameter data, if any, that is to steer client action.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Servers use the kXR_attn response code to optimize overall system performance and to notify clients of any impending events. All responses except for kXR_asyninfo and kXR_asynresp, do not correspond to any client request and should not be paired up with any request.

3)      When kXR_attn is received, the client must perform the requested action and indicated by the actnum value.


2.4.2.1       Server kXR_attn Response for kXR_asyncms Client Action

 

kXR_char

pad[2]

kXR_unt16

kXR_attn

kXR_int32

mlen

kXR_int32

kXR_asyncms

kXR_char

msg[mlen-4]

 

Where:

 

mlen    is the binary length of the following action code and message.

 

msg      is the message to be sent to the terminal. The mlen value, less four, indicates the length of the message. The ending null byte (‘\0’) should be transmitted and included in the message length.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Servers use the kXR_attn response code to optimize overall system performance and to notify clients of any impending events. This response does not correspond to any client request and should not be paired up with any request.

3)      When kXR_attn is received with the kXR_asyncms action code, the client should simply write the indicated message to the terminal.


 

2.4.2.2       Server kXR_attn Response for kXR_asyninfo Client Action

 

kXR_char

pad[2]

kXR_unt16

kXR_attn

kXR_int32

plen

kXR_int32

kXR_asyninfo

kXR_char

streamid[2]

kXR_unt16

reqcode

kXR_char

data[plen-8]

 

Where:

 

plen     is the binary length of the subsequent bytes.

 

streamid

            is the stream identifier associated with a previously issued request. The request may or may not have prevoiusly received a  kXR_waitresp response.

reqcode

            is the request code to which this response applies. The request defines the format of the data response. See the notes as to which request codes send kXR_attn+kXR_asyninfo responses.

 

data     is the data relevant to the request identified by streamed. Refer to specific request descriptions for the information contained in data for those requests that may send kXR_asyninfo responses.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      The following request codes may send a kXR_attn+kXR_asyninfo response: kXR_gpfile and kXR_pgwrite. Refer to the specific request for the format of the data that is sent.

3)      When this response is received, the client should reset the wait timeout to its original value.


2.4.2.3       Server kXR_attn Response for kXR_asynresp Client Action

 

kXR_char

pad[2]

kXR_unt16

kXR_attn

kXR_int32

plen

kXR_int32

kXR_asynresp

kXR_char

reserved[4]

kXR_char

streamid[2]

kXR_unt16

status

kXR_int32

dlen

kXR_char

data[dlen]

 

Where:

 

plen     is the binary length of the following action code and response.

 

streamid

            is the stream identifier associated with a previously issued request that received a  kXR_waitresp response.

 

status   is the binary status code indicating how the request completed. The codes definitions are identical as to those described for synchronous responses.

 

dlen     is the binary length of the data portion of the message. If no data is present, then the value is zero.

 

data     are data specific to the request. Not all responses have associated data. If the response does have data, the length of this field is recorded in the dlen field.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Servers use the kXR_attn response code to optimize overall system performance and to notify clients of any impending events.

3)      Unlike other asynchronous events, this response is associated with a previous request and the response data must be used to complete that request.

4)      The rlen-dlen is always 16.

5)      When kXR_attn is received with the kXR_asynresp action code, the client should remove the request paired with streamid from wait state and complete it using the response data.


2.4.3         Server kXR_authmore Response Format

 

kXR_char

streamid[2]

kXR_unt16

kXR_authmore

kXR_int32

dlen

kXR_char

data[dlen]

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream corresponding to a previous request.

 

dlen     is the binary length of the data portion of the message (i.e., the subsequent bytes).

 

data     is the data, if any, required to continue the authentication process.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3)      The kXR_authmore response code is issued only for those authentication schemes that require several handshakes in order to complete (e.g., .x500).

4)      When a kXR_authmore response is received, the client must call the appropriate authentication continuation method and pass it data, if present. The output of the continuation method should be sent to the server using another kXR_auth request. This handshake continues until either the continuation method fails or the server returns a status code of kXR_error or kXR_ok.

5)      Refer to the description of the security framework for detailed information.

 


2.4.4        Server kXR_error Response Format

 

kXR_char

streamid[2]

kXR_unt16

kXR_error

kXR_int32

dlen

kXR_int32

errnum

kXR_char

errmsg[dlen-4]

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream corresponding to a previous request.

 

dlen     is the binary length of the data portion of the message (i.e., the subsequent bytes).

 

errnum

            is the binary error number indicating the nature of the problem encountered when processing the request.

 

errmsg

            is the human-readable null-terminated message that describes the error. This message may be displayed for informational purposes.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Since the error message is null-terminated, dlen includes the null byte in its count of bytes that were sent.

3)      Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.


2.4.4.1       Server kXR_error Sub-Codes & Recovery Actions

 

The following table lists possible error sub-codes included in the errnum field as part of the kXR_error response:

 

 

Status

 

Meaning

Redirector

Recovery

Server

Recovery

kXR_ArgInvalid

A request argument was not valid

n/a

n/a

kXR_ArgMissing

Required request argument was not provided

n/a

n/a

kXR_ArgTooLong

A request argument was too long (e.g., path)

n/a

n/a

kXR_AttrNotFound

The requested file attribute does not exist

n/a

n/a

kXR_BadPayload

The request arguments were malformed

n/a

n/a

kXR_Cancelled

The operation was cancelled by the administrator

n/a

n/a

kXR_ChkLenErr

The close length does not equal the file size

n/a

n/a

kXR_ChkSumErr

The kXR_verifyw checksum does not match

n/a

n/a

kXR_DecryptErr

Data could not be decrypted

n/a

n/a

kXR_FileLocked

File is locked, open request was rejected

n/a

n/a

kXR_FileNotOpen

File if not open for the request (e.g., read)

n/a

n/a

kXR_FSError

The file system indicated an error

n/a

A

kXR_fsReadOnly

The file system is marked read-only.

n/a

n/a

kXR_inProgress

Operation already in progress

B

B

kXR_InvalidRequest

The request code is invalid

n/a

n/a

kXR_IOError

An I/O error has occurred

n/a

A

kXR_isDirectory

Object being opened with kXR_open is a directory

n/a

n/a

kXR_NoMemory

Insufficient memory to complete the request

C

B

kXR_NoSpace

Insufficient disk space to write data

n/a

n/a

kXR_NotAuthorized

Client is not authorized for the request

n/a

E

kXR_NotFile

Object being opened with kXR_open is not a file.

n/a

n/a

kXR_NotFound

The requested file was not found

n/a

D

kXR_noReplicas

No more replicas exist.

n/a

n/a

kXR_noserver

There are no servers available to process the request

n/a

n/a

kXR_overQuota

Space quota exceeded

n/a

n/a

kXR_overloaded

Server is overloaded

C

D

kXR_ServerError

An internal server error has occurred

C

A

kXR_SigVerErr

Request signature could not be verified

n/a

n/a

kXR_TLSRequired

Request requires a TLS connection

n/a

n/a

kXR_Unsupported

The request is valid but not supported

n/a

E

 


 

A.    Go back to the redirector and ask for a different server. kXR_refresh should not be turned on. The “tried=” CGI value should indicate the hostname of the failing server.

B.     Generally, this represents a programming error. However, should an operation subject to a callback response be retried prior to the callback, this status code may be returned. Clients should honor server’s callback requests and wait for a callback response. Therefore, this error can be ignored as long as a callback is outstanding. Otherwise, it should be treated as a fatal error.

C.    If the redirector is replicated, a different redirector should be tried. If all redirectors provide the same response, a fatal error should be reported. In the case of intermediate redirectors (i.e., a redirector transferring the request to another redirector), the recovery may be attempted by treating the intermediate as a server and performing the action outline in A.

D.    Go back to the redirector and ask for a different server. kXR_refresh should be turned on. The “tried=” CGI value should indicate the hostname of the failing server. This should normally be done only once.

E.     If the redirector is a meta-manager or is virtual (i.e. actually a metalink) then go back to the redirector and ask for a different server. The “tried=” CGI value should indicate the hostname of the failing server. The kXR_refresh should not be turned on. For kXR_NotAuthorized, recovery should be attempted no more than three times.

 


2.4.5        Server kXR_ok Response Format

 

kXR_char

streamid[2]

kXR_unt16

kXR_ok

kXR_int32

dlen

kXR_char

data[dlen]

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream corresponding to a previous request.

 

dlen     is the binary length of the data portion of the message (i.e., the subsequent bytes).

 

data     is the result, if any, of the corresponding request.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3)      The kXR_ok response indicates that the request fully completed and no additional responses will be forthcoming.


2.4.6        Server kXR_oksofar Response Format

 

kXR_char

streamid[2]

kXR_unt16

kXR_oksofar

kXR_int32

dlen

kXR_char

data[dlen]

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream corresponding to a previous request.

 

dlen     is the binary length of the data portion of the message (i.e., the subsequent bytes).

 

data     is the result, if any, of the corresponding request.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3)      The kXR_oksofar response indicates that the server is providing partial results and the client should be prepared to receive additional responses on the same stream. This response is primarily used when a read request would transmit more data than the internal server segment size.

4)      Sending requests using the same streamid when a kXR_oksofar status code has been returned may produced unpredictable results. A client must serialize all requests using the streamid in the presence of partial results.

5)      Any status code other than kXR_oksofar indicates the end of transmission

 


2.4.7        Server kXR_redirect Response Format

 

kXR_char

streamid[2]

kXR_unt16

kXR_redirect

kXR_int32

dlen

kXR_int32

port | 0xffffffff

kXR_char

host[?[opaque][?token]][dlen-4] | url

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream corresponding to a previous request.

 

dlen     is the binary length of the data portion of the message (i.e., the subsequent bytes).

 

port      is the binary port number to which the client must connect. If the value is zero, the default XRootD port number must be used. If the value is negative, then the text after port contains a standard URL that must be used to effect a new connection. This should only occur if the client has indicated that URL redirection responses are acceptable during the most recent kXR_login request to the redirecting server.

 

host      is the ASCII name of the to which the client must connect. The host does not end with a null (\0) byte. The host should be interpreted as a standard URL if port is negative (see above).

 

opaque is an optional ASCII token that, when present, must be delivered to the new host as opaque information added to the file name[2] associated with the operation being redirected. The opaque, if present, is separated from the host by a single question mark. The opaque does not end with a null (\0) byte but may end with a question mark (see token below). Therefore, opaque may never contain a question mark.


 

token    is an optional ASCII token that, when present, must be delivered to the new host during the login phase, if one is needed (i.e. established connections to the specified host may be re-used without a login). The token, if present, is separated from the host by a two question marks. The first question mark may be followed by opaque information. If none is present, another question mark immediately follows the first one. The token does not end with a null (\0) byte.

 

url       when a client indicates that it supports multi-protocol redirects, the server may respond with an actual url. In this case, the port value is set to -1.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit  length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3)      After 256 redirect responses within 10 minutes on the same logical connection, the client should declare an internal system error since it is obvious that effective work is not being performed.

4)      The client must be prepared to handle a redirect response at any time. A redirect response requires that the client

a.   Decompose the response to extract the port number, host name, and possible token value.

b.   Possibly close the connection of the current host, if the current host is a data server and this is the last logical connection to the server. Otherwise, if this is the first load-balancing server encountered in the operation sequence, the connection should remain open since a load-balancing server always responds with a redirect.

c.    Establish a new logical connection with the indicated host at the specified or default port number. If a physical connection already exists and is session compatible with the new logical connection; the existing physical connection should be reused and the next step (i.e. handshake and login) should be skipped.

d.  Perform the initial handshake, login with token (see kXR_login description), and authentication (see kXR_auth description).

e.   If the redirection occurred for a request using a file handle (i.e., fhandle) then a new file handle must be obtained.

                                                              i.      A kXR_open request must be issued using the same file name and options as was originally used.

                                                           ii.      The returned file handle must be used for the request that is to be re-issued as well as all subsequent requests relating o the file.

f.     Re-issue the request that was redirected.

5)      Opaque data must be treated as truly opaque. The client should not inspect nor modify the data in any way.

 


 

2.4.8        Server kXR_wait Response Format

 

kXR_char

streamid[2]

kXR_unt16

kXR_wait

kXR_int32

dlen

kXR_int32

seconds

kXR_char

infomsg[dlen-4]

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream corresponding to a previous request.

 

dlen     is the binary length of the data portion of the message (i.e., the subsequent bytes).

 

seconds

            is the maximum binary number of seconds that the client needs to wait before re-issuing the request.

 

infomsg

            is the human-readable message that describes the reason of why the wait is necessary. The message does not end with a null (\0) byte. This message may be displayed for informational purposes.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3)      The client should wait the indicated number of seconds and retry the request.

4)      Nothing prohibits the client from waiting for less time than the indicated number of seconds.

 


2.4.9        Server kXR_waitresp Response Format

 

kXR_char

streamid[2]

kXR_unt16

kXR_waitresp

kXR_int32

4

kXR_int32

seconds

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream corresponding to a previous request.

 

seconds

            is the estimated maximum binary number of seconds that the client needs to wait for the response.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Since requests may be completed in any order, the ordering of responses is undefined. The client must appropriately pair responses with requests using the streamid value.

3)      The client should wait the indicated number of seconds for the response. The response should be returned via an unsolicited response (kXR_attn with kXR_asynresp) at some later time which may be earlier than the time indicated in seconds. When the response arrives, the client must use the response data to complete the request that received the kXR_waitresp.

4)      Nothing prohibits the client from waiting for different time than the indicated number of seconds. Generally, if no response is received after at least seconds have elapsed; the client should treat the condition as a fatal error.

 


2.5       Binary Definitions of Status and Error codes

2.5.1        Status Codes

 

Status Code

Value

kXR_ok

0

kXR_oksofar

4000

kXR_attn

4001

kXR_authmore

4002

kXR_error

4003

kXR_redirect

4004

kXR_wait

4005

kXR_waitresp

4006

 

2.5.2        kXR_attn Status Subcodes

 

kXR_attn Subcode

Value

kXR_asyncms

5002

kXR_asyninfo

5009

kXR_asynresp

5008

 


 

2.5.3        Error Codes

 

Error

Value

Corresponding POSIX errno Value

kXR_ArgInvalid

3000

EINVAL

kXR_ArgMissing

3001

EINVAL

kXR_ArgTooLong

3002

ENAMETOOLONG

kXR_FileLocked

3003

EDEADLK

kXR_FileNotOpen

3004

EBADF

kXR_FSError

3005

EIO

kXR_InvalidRequest

3006

EEXIST

kXR_IOError

3007

EIO

kXR_NoMemory

3008

ENOMEM

kXR_NoSpace

3009

ENOSPC

kXR_NotAuthorized

3010

EACCES

kXR_NotFound

3011

ENOENT

kXR_ServerError

3012

ENOMSG

kXR_Unsupported

3013

ENOTSUP

kXR_noserver

3014

EHOSTUNREACH

kXR_NotFile

3015

ENOTBLK

kXR_isDirectory

3016

EISDIR

kXR_Cancelled

3017

ECANCELED

kXR_ChkLenErr

3018

EDOM

kXR_ChkSumErr

3019

EDOM

kXR_inProgress

3020

EINPROGRESS

kXR_overQuota

3021

EDQUOT

kXR_SigVerErr

3022

EILSEQ

kXR_DecryptErr

3023

ERANGE

kXR_Overloaded

3024

EUSERS

kXR_fsReadOnly

3025

EROFS

kXR_BadPayload

3026

EINVAL

kXR_AttrNotFound

3027

ENOATTR

kXR_TLSRequired

3028

EPROTOTYPE

kXR_noReplicas

3029

EADDRNOTAVAIL


3         Transport Layer Security (TLS) Support

 

The XRootD protocol supports TLS mode connections in two explcit ways:

1)      client request using the kXR_protocol request, and

2)      server request using the kXR_protocol response.

 

This mechanism provides several features:

·         A single port can be used for TLS and non-TLS connections.

·         The request channel can be split from the data channel using the kXR_bind request so that control information flows on a TLS connection while data flows on a non-TLS connection. Such an arrangement may significantly improve performance.

·         The number of interactions can be reduced when a connection needs to use TLS.

·         The server may independently enforce TLS requirements in for broad categories:

o   logins and all subsequent interactions,

o   all post-login interactions,

o   third party copy requests, and

o   data transfers.

 

Currently, once a connection switches to TLS mode it cannot switch back. This is not a protocol requirement but a practical side-effect of current TLS implementations that buffer an inditerminant amount of data making it problematic to deterministically switch modes. However, the XRootD protocol is sufficiently open to allow such switches if and when the TLS protocol can do so in the future.

 

A server is not required to support TLS. If it does, it should follow the protocol specifications described in the kXR_protocol and kXR_bind requests.

 

TLS may be considered a replacement for request signing in most circumstances. However, for certain workflows, request signing may offer better performance. Be ware, that XRootD request signing, as defined, does not protect data while TLS., when used for data, does so.

 


 

3.1       Client-Server interactions to unilaterally use TLS

·         The client should connect to the server using a non-TLS connection and send the handshake packet.

·         The client should then send a kXR_protocol request indicating that it wants to use TLS. For reduced latency, the handshake and the kXR_protocol request may be sent together.

·         If the server supports TLS it should indicate in the kXR_protocol response that the connection will be switched to use TLS after the response is sent.

·         The client should check if the server switched the connection to use TLS and do the same if so indicated.

·         All communications from then on use TLS.

 

3.2       Client-Server interactions to use TLS only when required

·         The client should connect to the server using a non-TLS connection and send the handshake packet.

·         The client should then send a kXR_protocol request indicating that it is able to use TLS. For reduced latency, the handshake and the kXR_protocol request may be sent together. In the kXR_protocol request the client should also indicate the expected next operation (i.e. login, data transfer, or third party copy).

·         If the server supports TLS it should indicate in the kXR_protocol response that the connection has been switched to use TLS if the client’s subsequent operation requires TLS. Note that it is also possible for the server to indicate that TLS is required after the kXR_login request (i.e. login does not require TLS).

·         The client should check if the server switched the connection to use TLS and do the same if so indicated. If the next reqest is a kXR_login and the server indicated that TLS is not required until after the login, the client should defer switching the connection to TLS until after the login and all authentication interactions (i.e. kXR_auth requests).

 

 

 

 

 


4         Server Request Format

4.1       kXR_auth Request

Purpose: Authenticate client’s username to the server.

 

Request

Normal Response

kXR_char

streamid[2]

kXR_char

streamid[2]

kXR_unt16

kXR_auth

kXR_unt16

kXR_ok

kXR_char

reserved[12]

kXR_int32

0

kXR_char

credtype[4]

 

 

kXR_int32

credlen

 

 

kXR_char

cred[credlen]

 

 

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream. This identifier should be echoed akXR_int32 with any response to the request.

 

reserved

            is an area reserved for future use and must be initialized to null characters (i.e., ‘\0’).

 

credtype

            the first four characters of the protocol name. If the protocol name is less than four characters, the name should be null terminated.

 

credlen

            is the binary length of the supplied credentials, cred.

 

cred     are the credentials used to provide authentication information.

 


 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      Authentication credentials may be supplied by many means. The common mechanism used by XRootD is to use the classes in the libXrdSec.so library. See the “Authentication & Access Control Configuration Reference” for more information.

3)      Refer to the description of the security framework on how a client authenticates to an XRootD server.

 

Binary Definitions

 

Request

Modifiers

Value

Explanation

kXR_auth

 

3000

Perform authenication

 



4.2       kXR_bind Request

Purpose: Bind a socket to a pre-existing session.

 

Request

Normal Response

kXR_char

streamid[2]

kXR_char

streamid[2]

kXR_unt16

kXR_bind

kXR_unt16

kXR_ok

kXR_char

sessid[16]

kXR_int32

1

kXR_int32

0

kXR_char

pathid

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

 

sessid   is the session identifier returned by a previous kXR_login request.

 

pathid  is the socket identifier associated with this connection. The pathid may be used in subqsequent kXR_read, kXR_readv, and kXR_write requests to indicate which socket should be used for a response or as a source of data.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      The sessid value should be treated as opaque data.

3)      The socket issuing the kXR_bind request must neither have a session id (i.e., be logged in) nor be already bound.

4)      Once a socket is bound to a session, if may only supply data for kXR_write requests or receive responses for kXR_read and kXR_readv requests.

5)      Each login session is limited to the number of bound sockets. Use the kXR_Qconfig sub-request code of kXR_query to determine the maximum number of sockets that can be bound to a login session.

6)      Bound sockets are meant to support parallel data transfer requests across wide-area networks. They are also meant to split control information from data allowing control to flow on a TLS connection while data flows on a non-TLS connection. See TLS Considerations for more information.


 

Binary Definitions

 

Request

Modifiers

Value

Explanation

kXR_bind

 

3024

Bind additional sockets to session

 

4.2.1        TLS Considerations

A server may indicate in the response to the kXR_protocol request that all data must flow across a TLS connection. The kXR_protocol request is normally sent by the client immediately after the handshake. If the the server’s response indicates that TLS must be used for data then the connection to be bound should be set to TLS mode in order for the request to succeed. There are two ways to achieve this.

 

The client may record whether or not the bound connections must use TLS. If bound connections must use TLS the kXR_bind request should be prefixed by a kXR_protocol request indicating that the connection should be switched to TLS. To reduce latency, the kXR_protocol and kXR_bind requests should be sent together. This method is preferred.

 

Alternatively, the client may always send a kXR_protocol request ahead of the kXR_bind request indicating in the request that it is able to use TLS and the next request will be kXR_bind. If the server requires the data to use TLS it should respond that the connection will switch to using TLS after the kXR_protocol response is sent. If the connection was switched to use TLS the client should do the same and then send the kXR_bind request. Since this involves additional interactions, it is not the preferred method.

 

The client is also free to switch the connection to use TLS whether or not the server requires it to do so.

 


4.3       kXR_chkpoint Request

Purpose: Create a checkpoint for an open file.

 

Request

Normal Response

kXR_char

streamid[2]

kXR_char

streamid[2]

kXR_unt16

kXR_chkpoint

kXR_unt16

kXR_ok

kXR_char

Fhandle[4]

kXR_int32

dlen

kXR_char

reserved[11]

kXR_char

data[dlen]

kXR_char

opcode

 

 

kXR_int32

alen

 

 

 

ckp_list[n]

 

 

n = alen/16 with no remainder

 

ckp_list

 

kXR_char

reserved[4]

 

 

kXR_int31

ckplen

 

 

kXR_int64

offset

 

 

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

 

fhandle

            is the file handle value supplied by the successful response to the associated kXR_open request that is to be used for the checkpoint request.

 

opcode         is the checkpoint operation wanted:

            kXR_ckpAdd        -  Create a checkpoint.

            kXR_ckpDel          -  Delete any existing checkpoint.

            kXR_ckpQMax     -  Query the maximum size of a checkpoint.

            kXR_ckpRestore   -  Restore file data and delete the checkpoint.

 

alen      is the binary length of the arguments that follow the request header. These arguments specify what is to be checkpointed via struct ckp_list. The maximum allowed value for alen is 8200. This allows up to 512 checkpoint segments.

 

 

ckplen  the amount of data to checkpoint.

 

offset    is the binary offset of the data to checkpoint. The offset plus the corresponding ckplen should be wholly contained within the file. If it is not, the request should fail.

 

dlen     is the binary length of the of the response data, data. It is should be zero except in response to kXR_ckpQMax opcode where is should be 4.

 

data     is the response data. For kXR_ckpQMax opcode is should be a kXR_int32 value conaining the maximum number of bytes a checkpoint can have.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      The fhandle value should be treated as opaque data.

3)      The fhandle should refere to a file opened for writing. If it does not, the request should fail.

4)      A checkpoint size is limited. The protocol specifies that the minimum limit is kXR_ckpMinMax. Implementation may allow for larger checkpoints.

5)      Once a checkpoint is establish a new should not be allowed until the existing checkpoint is deleted or restored.

6)      Should the client loose connectivity to the server, all outstanding checkpoints should be restored.

7)      When the client closes a file with an outstanding checkpoint, the checkpoint should be deleted.

 

Binary Definitions

 

Request

Modifiers

Value

Explanation

kXR_chkpoint

 

3012

Checkpoint file data.

 

opcode

 

 

 

kXR_ckpCreate

0x01

Create a new checkpoint.

 

kXR_ckpDelete

0x02

Delete the current checkpoint.

 

kXR_ckpQMax

0x03

Query checkpoint limit.

 

kXR_ckpRestore

0x04

Restore the current checkpoint.

 

 


4.4       kXR_chmod Request

Purpose: Change the access mode on a directory or a file.

 

Request

Normal Response

kXR_char

streamid[2]

kXR_char

streamid[2]

kXR_unt16

kXR_chmod

kXR_unt16

kXR_ok

kXR_char

reserved[14]

kXR_int32

0

kXR_int16

mode

 

 

kXR_int32

plen

 

 

kXR_char

path[plen]

 

 

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

 

reserved

            is an area reserved for future use and must be initialized to null characters (i.e., ‘\0’).

 

mode    is the access mode to be set for path. The access mode is an “or’d” combination of the following values:

 

Access

Readable

Writeable

Executable

Owner

kXR_ur

kXR_uw

not supported

Group

kXR_gr

kXR_gw

not supported

Other

kXR_or

not supported

not supported

 

plen     is the binary length of the supplied path, path.

 

path     is the path whose mode is to be set. It may be suffixed with CGI information.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      No umask is applied to the specified mode.

 

 Binary Definitions

 

Request

Modifiers

Value

Explanation

kXR_chmod

 

3002

Change directory or file permissions

 

mode

 

 

 

kXR_ur

0x01 00

Owner readable

 

kXR_uw

0x00 80

Owner writable

 

kXR_ux

0x00 40

Owner searchable (directories)

 

kXR_gr

0x00 20

Group readable

 

kXR_gw

0x00 10

Group writable

 

kXR_gx

0x00 08

Group searchable (directories)

 

kXR_or

0x00 04

Other readable

 

kXR_ow

0x00 02

Other writable

 

kXR_ox

0x00 01

Other searchable (directories)

 

 


4.5       kXR_close Request

Purpose: Close a previously opened file, communications path, or path group.

 

Request

Normal Response

kXR_char

streamid[2]

kXR_char

streamid[2]

kXR_unt16

kXR_close

kXR_unt16

kXR_ok

kXR_char

fhandle[4]

kXR_int32

0

kXR_char

reserved[12]

 

 

kXR_int32

0

 

 

 

Where:

 

streamid

            is the binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

 

reserved

            is an area reserved for future use and must be initialized to null characters (i.e., ‘\0’).

 

fhandle

            is the file handle value supplied by the successful response to the associated kXR_open request.

 

Notes

1)      All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields must be initialized to binary zero.

2)      The fhandle value should be treated as opaque data.

 

Binary Definitions

 

Request

Modifiers

Value

Explanation

kXR_close

 

3003

Close an open file


4.6       kXR_dirlist Request

Purpose: Enumerate the contents of a directory.

 

Request

Normal Response

kXR_char

streamid[2]

kXR_char

streamid[2]

kXR_unt16

kXR_dirlist

kXR_unt16

kXR_ok

kXR_char

reserved[15]

kXR_int32

dlen

kXR_char

options

kXR_char

Dirname0\n

kXR_int32

plen

 

kXR_char

path[plen]

 

 

 

 

kXR_char

kXR_char

dirnamen

<