2 Request/Response Protocol

2.1 Format of Client-Server Initial Handshake

When a client first connects to the XRootD server, it should perform a special handshake. This handshake should determine whether the client is communicating using XRootD protocol or another protocol hosted by the server.

The handshake consists of the client sending 20 bytes, as follows:

kXR_int32	0
kXR_int32	0
kXR_int32	0
kXR_int32	4	(network byte order)
kXR_int32	2012	(network byte order)

XRootD protocol, servers should respond, as follows:

streamid:	kXR_char	smid[2]
status:	kXR_unt16	0
msglen:	kXR_int32	rlen
msgval1:	kXR_int32	pval
msgval2:	kXR_int32	flag

Where:

smid initial streamid. The smid for the initial response is always two null characters (i.e., ‘\0’);

rlen binary response length (e.g., 8 for the indicated response).

pval binary protocol version number.

flag additional bit-encoded information about the server; as follows:

kXR_DataServer - 0x00 00 00 01 This is a data server.

KXR_LBalServer - 0x00 00 00 00 This is a load-balancing server.

Notes

1) All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values. All reserved fields should be initialized to binary zero.

2) The particular response format was developed for protocol version 2.0 and does not convey all of the information to capture features provided by subsequent protocol versions. In order to provide backward compatibility, this response format has been kept. The recommended mechanism to obtain all of the information that may be needed is to “piggy-back” a kXR_protocol Request with the handshake (i.e. send the handshake and the request with a single write).

3) All twenty bytes should be received by the server at one time. All known TCP implementations should guarantee that the first message is sent intact if all twenty bytes are sent in a single system call. Using multiple system calls for the first message may cause unpredictable results.

2.2 Data Serialization

All data sent and received is serialized (i.e., marshaled) in three ways:

1. Bytes are sent unaligned without any padding,

2. Data type characteristics are predefined (see table below), and

3. All integer quantities are sent in network byte order (i.e, big endian).

XRootD Type	Sign	Bit Length	Bit Alignment	Typical Host Type
kXR_char8	unsigned	8	8	unsigned char
kXR_unt16	unsigned	16	16	unsigned short
kXR_int32	signed	32	32	long[1]
kXR_int64	signed	64	64	long long

Table 1: XRootD Protocol Data Types

Network byte order is defined by the Unix htons() and htonl() macros for host to network short and host to network long, respectively. The reverse is defined by the ntohs() and ntohl() macros. Many systems do not define the long long versions of these macros. XRootD protocol requires that the POSIX version of long long serialization be used, as defined in the following figures. The OS-dependent isLittleEndian() function returns true if the underlying hardware using little endian integer representation.

unsigned long long htonll(unsigned long long x)

{unsigned long long ret_val;

if (isLittleEndian())

{*( (unsigned long *)(&ret_val) + 1) =

htonl(*( (unsigned long *)(&x)));

*(((unsigned long *)(&ret_val))) =

htonl(*( ((unsigned long *)(&x))+1) );

} else {

*( (unsigned long *)(&ret_val)) =

htonl(*( (unsigned long *)(&x)));

*(((unsigned long *)(&ret_val)) + 1) =

htonl(*( ((unsigned long *)(&x))+1) );

}

return ret_val;

};

Figure 1: POSIX Host to Network Byte Order Serialization

unsigned long long ntohll(unsigned long long x)

{unsigned long long ret_val;

if (isLittleEndian())

{*( (unsigned long *)(&ret_val) + 1) =

ntohl(*( (unsigned long *)(&x)));

*(((unsigned long *)(&ret_val))) =

ntohl(*(((unsigned long *)(&x))+1));

} else {

*( (unsigned long *)(&ret_val)) =

ntohl(*( (unsigned long*)(&x)));

*(((unsigned long*)(&ret_val)) + 1) =

ntohl(*(((unsigned long*)(&x))+1));

}

return ret_val;

};

Figure 2: Network and Host Byte Order Seialization

More compact and efficient, though OS restricted (i.e., Solaris and Linux), versions of 64-bit network byte ordering routines are given in the following figure.

#if defined(__sparc) || __BYTE_ORDER==__BIG_ENDIAN

#ifndef htonll

#define htonll(x) x

#endif

#ifndef ntohll

#define ntohll(x) x

#endif

#else

#ifndef htonll

#define htonll(x) __bswap_64(x)

#endif

#ifndef ntohll

#define ntohll(x) __bswap_64(x)

#endif

Figure 3: Network and Host Byte Ordering Macros

2.3 Client Request Format

Requests sent to the server are a mixture of ASCII and binary. All requests, other than the initial handshake request, have the same format, as follows:

kXR_char	streamid[2]
kXR_unt16	requestid
kXR_char	parms[16]
kXR_int32	dlen
kXR_char	data[dlen]

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

requestid

binary identifier of the operation to be performed by the server.

parms parameters specific to the requestid.

dlen binary length of the data portion of the message. If no data is present, then the value is zero.

data data specific to the requestid. Not all requests have associated data. If the request does have data, the length of this field is recorded in the dlen field.

Notes

2) All XRootD client requests consist of a standard 24-byte fixed length message. The 24-byte header may then be optionally followed by request specific data.

3) Stream id’s are arbitrary and are assigned by the client. Typically these id’s correspond to logical connections multiplexed over a physical connection established to a particular server.

4) The client may send any number of requests to the same server. The order in which requests are performed is undefined. Therefore, each request should have a different streamid so that returned results may be paired up with associated requests.

5) Requests sent by a client over a single physical connection may be processed in an arbitrary order. Therefore the client is responsible for serializing requests, as needed.

2.3.1 Valid Client Requests

Requestid		Value	Login?	Auth?	Redirect?	Arguments
kXR_auth		3000	y	n	n	authtype, authinfo
KXR_bind		3024	n	n	n	sessid
kXR_chkpoint		3012	y	-	n	fhandle, length, offset
kXR_chmod		3002	y	y	yes	mode, path
kXR_close		3003	y	-	n	fhandle
KXR_dirlist		3004	y	y	y	path
KXR_endsess		3023	y	-	n	sessid
kXR_fattr		3020	y	y	y	Arguments vary by subcode
kXR_gpfile		3005	y	optional	y	Arguments vary by subcode
kXR_locate		3027	y	y	y	path
kXR_login		3007	n	n	n	userid, token
kXR_mkdir		3008	y	y	y	mode, path
kXR_mv		3009	y	y	y	old_name, new_name
kXR_open		3010	y	y	y	mode, flags, path
kXR_pgread		3030	y	-	y	fhandle, pathid, length, offset
kXR_pgwrite		3026	y	-	y	fhandle, pathid, length, offset
kXR_ping		3011	y	n	n
kXR_prepare		3021	y	y	n	paths
kXR_protocol		3006	n	n	n
kXR_query		3001	y	y	y	args
kXR_read		3013	y	-	y	fhandle, pathid, length, offset
kXR_readv		3025	y	-	y	fhandle, pathid, length, offset
kXR_rm		3014	y	y	y	path
kXR_rmdir		3014	y	y	y	path
kXR_set		3018	y	y	y	info
kXR_sigver		3029	y	y	n	signature
kXR_stat		3017	y	-	n	fhandle
kXR_stat		3017	y	y	y	path
kXR_statx		3022	y	y	n	pathlist
kXR_sync		3016	y	-	n	fhandle
kXR_truncate		3028	y	-	n	fhandle, length
kXR_truncate		3028	y	-	y	path, length
kXR_write		3019	y	-	y	fhandle, pathid, length, offset, data
kXR_writev		3031	y	y	n	fhandle, length, offset
	Table 2: Valid Client Requests

2.3.2 Valid Client Paths

The XRootD server accepts only absolute paths where a path may be specified. Relative paths should be resolved by the client interface prior to sending them to XRootD. This means that the interface should handle a virtual “current working directory” to resolve relative paths should they arise.

Path names are restricted to the following set of characters:

Letters (upper or lower case),
Digits (0-9), and
Special characters: !@#%^_-+=:./

In general, paths may not contain shell meta-characters.

Any path may be suffixed by CGI information. The format corresponds to that defined in RFC 3875. However, the protocol does not allow URL encoded characters (i.e. %xx). The meaning of any CGI element that is not specified in this document is implementation specific.

2.3.3 Client Recovery from Server Failures

A server failure should be recognized when the server unexpectedly closes its TCP/IP connection or does not respond for an extended period of time. Should this happen, the client may recover all operations by treating the termination of the connection or unresponsiveness as a redirection request (see page 29) to the initial XRootD server for all streams associated with the closed TCP/IP connections.

The initial XRootD server is defined as the first manager or the last meta-manager encountered. In the absence of any manager, the first data server encountered. See the kXR_protocol request on how to determine a node’s type.

Because many clients are likely to be affected by a server failure, it is important that clients pace their reconnection to the initial XRootD server. One effective way to do this is to use the last three bits of the client’s IP address as the number of seconds to wait before attempting a reconnection. It is up to the client to determine either the number of times or the time window in which reconnections should be attempted before failure is declared. Typical values are 16 attempts or 3 minutes, whichever is longer.

Note that it may not be possible to recover in this way for files that were opened in update mode. Clients who do not provide proper transactional support generally cannot recover via redirection for any read/write resources.

2.4.4.1 Error Codes and Recovery Actions

kXR_Error Status Code in errnum	Meaning	Redirector Recovery	Server Recovery
kXR_ArgInvalid	A request argument was not valid	n/a	n/a
kXR_ArgMissing	Required request argument was not provided	n/a	n/a
kXR_ArgTooLong	A request argument was too long (e.g., path)	n/a	n/a
kXR_AttrNotFound	The requested file attribute does not exist	n/a	n/a
kXR_AuthFailed	Authentication failed	H	H
kXR_BadPayload	The request arguments were malformed	n/a	n/a
kXR_Cancelled	The operation was cancelled by the system	n/a	n/a
kXR_ChkSumErr	The checksum does not match	n/a	n/a
kXR_Conflict	Request cannot be executed due to a conflict	n/a	n/a
kXR_DecryptErr	Data could not be decrypted	n/a	n/a
kXR_FileLocked	File is locked, open request was rejected	n/a	n/a
kXR_FileNotOpen	File if not open for the request (e.g., read)	n/a	n/a
kXR_FSError	The file system indicated an error	n/a	A
kXR_fsReadOnly	The file system is marked read-only.	n/a	n/a
kXR_Impossible	The request cannot be executed due to exigent conditions	n/a	n/a
kXR_inProgress	Operation already in progress	B	B
kXR_InvalidRequest	The request code is invalid	n/a	n/a
kXR_IOError	An I/O error has occurred	n/a	A
kXR_isDirectory	Object being opened with kXR_open is a directory	n/a	n/a
kXR_ItExists	Cannot create new object as it already exists	n/a	n/a
kXR_NoMemory	Insufficient memory to complete the request	C	B
kXR_NoSpace	Insufficient disk space to write data	n/a	n/a
kXR_NotAuthorized	Client is not authorized for the request	n/a	E
kXR_NotFile	Object being opened with kXR_open is not a file.	n/a	n/a
kXR_NotFound	The requested file was not found	n/a	D
kXR_noReplicas	No more replicas exist.	n/a	n/a
kXR_noserver	There are no servers available to process the request	F	n/a
kXR_overQuota	Space quota exceeded	n/a	n/a
kXR_overloaded	Server is overloaded	C	D
kXR_ReqTimedOut	Request could not be completed in time	n/a	D
kXR_ServerError	An internal server error has occurred	C	A
kXR_SigVerErr	Request signature could not be verified	G	G
kXR_TooManyErrs	Request has excessive errors to continue	n/a	D
kXR_TLSRequired	Request requires a TLS connection	n/a	n/a
kXR_Unsupported	The request is valid but not supported	n/a	E

A. Go back to the redirector and ask for a different server. kXR_refresh should not be turned on. The “tried=” CGI value should indicate the hostname of the failing server.

B. Generally, this represents a programming error. However, should an operation subject to a callback response be retried prior to the callback, this status code may be returned. Clients should honor server’s callback requests and wait for a callback response. Therefore, this error can be ignored as long as a callback is outstanding. Otherwise, it should be treated as a fatal error.

C. If the redirector is replicated, a different redirector should be tried. If all redirectors provide the same response, a fatal error should be reported. In the case of intermediate redirectors (i.e., a redirector transferring the request to another redirector), the recovery may be attempted by treating the intermediate as a server and performing the action outline in A.

D. Go back to the redirector and ask for a different server. kXR_refresh should be turned on. The “tried=” CGI value should indicate the hostname of the failing server. This should normally be done only once.

E. If the redirector is a meta-manager or is virtual (i.e. actually a metalink) then go back to the redirector and ask for a different server. The “tried=” CGI value should indicate the hostname of the failing server. The kXR_refresh should not be turned on. For kXR_NotAuthorized, recovery should be attempted no more than three times.

F. If the redirector is virtual (i.e. actually a metalink), the follow the actions listed under E. Real redirectors have a real-time view of all available resources and the inability to allocate a resource indicates that none are useable for a request. Retrying the request is highly likely to be ineffective. Virtual redirectors only have a static view of resources and cannot determine if using another resource will succeed without actually trying to use that resource. Thus, all failures are retryable.

G. Signature verification errors due to transport corruption are retryable as such corruptions are transient. There is no way to determine if a failure is due to corruption or active compromise. The request should be retried once or twice.

H. Authentication failures may be due to server missconfiguration. If another server or redirector is available, the operation may be retried.

2.4.5 kXR_ok Response Format

kXR_char	streamid[2]
kXR_unt16	kXR_ok
kXR_int32	dlen
kXR_char	data[dlen]

Where:

streamid

binary identifier that is associated with this request stream corresponding to a previous request.

dlen binary length of the data portion of the message (i.e., the subsequent bytes).

data result, if any, of the corresponding request.

Notes

2) Since requests may be completed in any order, the ordering of responses is undefined. The client should appropriately pair responses with requests using the streamid value.

3) The kXR_ok response indicates that the request fully completed and no additional responses should be forthcoming.

2.4.6 kXR_oksofar Response Format

kXR_char	streamid[2]
kXR_unt16	kXR_oksofar
kXR_int32	dlen
kXR_char	data[dlen]

Where:

streamid

binary identifier that is associated with this request stream corresponding to a previous request.

dlen binary length of the data portion of the message (i.e., the subsequent bytes).

data result, if any, of the corresponding request.

Notes

2) Since requests may be completed in any order, the ordering of responses is undefined. The client should appropriately pair responses with requests using the streamid value.

3) The kXR_oksofar response indicates that the server is providing partial results and the client should be prepared to receive additional responses on the same stream. This response is primarily used when a read request would transmit more data than the internal server segment size.

4) Sending requests using the same streamid when a kXR_oksofar status code has been returned may produced unpredictable results. A client should serialize all requests using the streamid in the presence of partial results.

5) Any status code other than kXR_oksofar indicates the end of transmission

2.4.7 kXR_redirect Response Format

kXR_char	streamid[2]
kXR_unt16	kXR_redirect
kXR_int32	dlen
kXR_int32	port \| 0xffffffff \| < 0
kXR_char	host[?[fcgi][?lcgi]][dlen-4] \| url

Where:

streamid

binary identifier that is associated with this request stream corresponding to a previous request.

dlen binary length of the data portion of the message (i.e., the subsequent bytes).

port binary port number to which the client should connect. If the value is zero, the default XRootD port number should be used. If the value is negative, then the text after port contains a standard URL that should be used to effect a new connection. This should only occur if the client has indicated that URL redirection responses are acceptable during the most recent kXR_login request to the redirecting server. See the usage notes when 0xffffffff should be used as a negative port number.

host ASCII name of the to which the client should connect. The host does not end with a null (\0) byte. The host should be interpreted as a standard URL if port is negative (see above). See the usage notes describing the format of host.

fcgi optional ASCII CGI string that, when present, should be added to the end of any existing CGI information appened to the file name[2] associated with the operation being redirected. The fcgi, if present, is separated from the host by a single question mark. The fcgi does not end with a null (\0) byte but may end with a question mark (see token below). Therefore, fcgi may never contain a question mark. See the usage notes for more information.

lcgi optional ASCII CGI string that, when present, should be delivered to the new host during the login phase as additional login parameters. However, if an established connection to the specified host already exists it may be re-used without a login. The lcgi, if present, is separated from the host by a two question marks. The first question mark may be followed by fcgi information. If none is present, another question mark immediately follows the first one. The lcgi does not end with a null (\0) byte. See the usage notes for more information.

url when a client indicates that it supports multi-protocol redirects, the server may respond with an actual url. In this case, the port value is set to -1.

Notes

2) The host, either in its shortened form or in a URL redirect, may either be an FQDN or an IP address. IP addresses should be in IPv6 bracket format or in IPv4 octet format. However, using IP addresses should be avoided as they conflict with NAT devicess common in frwewalled sites, Kubernetes clusters using virtual networks, and addresses are not usually present in a host certificates SAN extension which makes TLS connection impossible.

3) CGI information (i.e. fcgi and lcgi) is a stream of 7-bit clean (ASCII) characters that encode zero or more assertions. These assertions are separated by the ‘&’ character. Assertions are key-value pairs in the form key = value. Both key and value text must not contain ‘?’, ‘&’ or ‘=’ characters. Hex encoding of characters (i.e. sequences such as % <hex> <hex>, where <hex> is a character in 0123456789abcdef ) should not be supported. When parsing a CGI string, any key that is not understood should be ignored. Keys that start xrd. are reserved. Implementations are free to include other keys. It is recommended for keys to start with a reverse-DNS name under the developer's control; e.g., (org.example.my-new-key). For historic reasons a server should process a login string (i.e. lcgi) that starts with the ‘?’ character by ignoring that character and processing the remaining login string.

4) Since requests may be completed in any order, the ordering of responses is undefined. The client should appropriately pair responses with requests using the streamid value.

5) After 256 redirect responses within 10 minutes on the same logical connection, the client should declare an internal system error since it is obvious that effective work is not being performed.

6) The client should be prepared to handle a redirect response at any time. A redirect response requires that the client

a. Decompose the response to extract the port number, host name, and possible token value.

b. Possibly close the connection of the current host, if the current host is a data server and this is the last logical connection to the server. Otherwise, if this is the first load-balancing server encountered in the operation sequence, the connection should remain open since a load-balancing server always responds with a redirect.

c. Establish a new logical connection with the indicated host at the specified or default port number. If a physical connection already exists and is session compatible with the new logical connection; the existing physical connection should be reused and the next step (i.e. handshake and login) should be skipped.

d. Perform the initial handshake, login with token (see kXR_login description), and authentication (see kXR_auth description).

e. If the redirection occurred for a request using a file handle (i.e., fhandle) then a new file handle should be obtained.

i. A kXR_open request should be issued using the same file name and options as was originally used.

ii. The returned file handle should be used for the request that is to be re-issued as well as all subsequent requests relating o the file.

f. Re-issue the request that was redirected.

7) Historically, clients tested the port for the exact value of -1 (i.e. 0xffffffff) to determine whether a redirect URL or a shortened host specification was present. This prevented additional information frm being passed in the port field. To provide backward compatability, a special kXR_login capability was introduced, kXR_redirflags, that indicates the client simply checks for a negative value and the low order 31 bits may be used as redirect flags. Servers should always use -1 unless the client indicates that it is capable of handling any negative value by setting the kXR_redirflags capability in the login request.

8) The following redirect flags are defined:

Flag	Meaning
kXR_recoverWrts	Write recovery for copy targets is possible at the server that set this redirect flag.
kXR_collapseRedir	If the redirect target is in the same address group as the redirecting server, make the target the primary address for all future contacts for this address group.

9) Normally, the protocol limits write recovery to the server to which the write was directed. The kXR_recoverWrts flag allows write recovery to occur at a different server as long as that server is in the redirect path leading to a server executing a write operation.

10) A DNS host name may be assigned multiple addresses, each of which is a different physical endpoint. All servers under that DNS name should be considered to belong to the same address group. Any server in that address group may request that one of those servers be designated as the primary target using the kXR_collapseRedir flag. This option is meant to support replicated services in which there is a primary leader chosen by consensus.

11) Opaque data should be treated as truly opaque. The client should not inspect nor modify the data in any way.

2.4.8 kXR_status Response Format

kXR_char	streamid[2]
kXR_unt16	kXR_status
kXR_int32 kXR_unt32 kXR_char kXR_char kXR_char kXR_char kXR_int32 kXR_char	resplen (should be >= 16) crc32c streamid[2] requestid resptype reserved[4] dlen info[resplen-16]
kXR_char	data[dlen]

Where:

streamid

binary identifier that is associated with this request stream corresponding to a previous request. It is repeated to allow for a quick integrity check of the streamid before doing more extensive checks.

resplen

binary length of the response portion of the message (i.e., the subsequent bytes not including any data portion).

crc32c CRC32-C as defined by the IETF RFC 7143 standard (see the notes for details) of the resplen-sizeof(crc32c) bytes immediately after crc32c. This means that the data portion, if any, should not be included in the cr32c calculation.

requestid

identifier of the original request. The requestid+kXR_1stRequest should equal the original request code.

resptype

binary code identifying the response type. See the subsequent section for details.

dlen binary length of the data portion of the message, if any. If there is no data portion then dlen should be zero.

info optional additional response information whose contents should be interpreted in the context of the requestid and resptype codes. Refer to each corresponding request to see how to interpret the info, if present. The length should be calculated as resplen- kXR_statusBodyLen and should result in a value >= 0.

data result, if any, of the corresponding request.

Notes

2) Since requests may be completed in any order, the ordering of responses is undefined. The client should appropriately pair responses with requests using the streamid value.

3) The crc32c should use the CRC32-C polynomial specified in the IETF RFC 7143 standard. This corresponds to the polynomial 0x1edc6f41 or

x³²+x²⁸+x²⁷+x²⁶+x²⁵+x²³+x²²+x²⁰+x¹⁹+x¹⁸+x¹⁴+x¹³+x¹¹+x¹⁰+x⁹+x⁸+x⁶+1.

4) When kXR_status is received the client should perform an integrity check on the response, as follows:

a. Verify that the two streamid values are identical, and

b. calculate the CRC32C value of the response and verify that it matches the value sent by the server in crc32c.

5) When an integrity check fails, the only recourse is to close the connection and start with a new connection. The reason is that there is no way to know how much and what kind of data may be in transit, should any of the length fields be corrupted. Be aware that closing a connection with active requests causes those requests to be terminated.

2.4.8.1 Valid ResponseTypes

The resptype codes as defined in struct ServerResponseStatus are:

resptype	datalen	Explanation
kXR_FinalResult	>= 0	Request completed as indicated in the response.
kXR_PartialResult	>= 0	Request has partially completed as indicated.
kXR_ProgressInfo	== 0	Request is ongoing this is a progress report only.

Notes

1) The presence of info and data is determined by the particular request being performed. Refer to the requests returning kXR_status for details.

2) Sending requests using the same streamid when a kXR_status with a PatrialResult or ProgressInfo restype code has been returned may produce unpredictable results. A client should serialize all requests using the streamid until a FinalResult restype is returned by the request.

3) Currently, only kXR_gpfile, kXR_pgread and kXR_pgwrite return kXR_status. However, clients implementing this version of the protocol should be implemented to handle any request returning kXR_status.

4) Requests employing kXR_status should never return kXR_ok and kXR_oksofar as these are essentially subsumed by kXR_status. The use of other response types is allowed.

5) When kXR_PartialResult or kXR_ProgressInfo is received, the client should reset the wait timeout to its original value.

2.4.9 kXR_wait Response Format

kXR_char	streamid[2]
kXR_unt16	kXR_wait
kXR_int32	dlen
kXR_int32	seconds
kXR_char	infomsg[dlen-4]

Where:

streamid

binary identifier that is associated with this request stream corresponding to a previous request.

dlen binary length of the data portion of the message (i.e., the subsequent bytes).

seconds

maximum binary number of seconds that the client needs to wait before re-issuing the request.

infomsg

human-readable message that describes the reason of why the wait is necessary. The message does not end with a null (\0) byte. This message may be displayed for informational purposes.

Notes

2) Since requests may be completed in any order, the ordering of responses is undefined. The client should appropriately pair responses with requests using the streamid value.

3) The client should wait the indicated number of seconds and retry the request.

4) Nothing prohibits the client from waiting for less time than the indicated number of seconds.

2.4.10 kXR_waitresp Response Format

kXR_char	streamid[2]
kXR_unt16	kXR_waitresp
kXR_int32	4
kXR_int32	seconds

Where:

streamid

binary identifier that is associated with this request stream corresponding to a previous request.

seconds

estimated maximum binary number of seconds that the client needs to wait for the response.

Notes

2) Since requests may be completed in any order, the ordering of responses is undefined. The client should appropriately pair responses with requests using the streamid value.

3) The client should wait the indicated number of seconds for the response. The response should be returned via an unsolicited response (kXR_attn with kXR_asynresp) at some later time which may be earlier than the time indicated in seconds. When the response arrives, the client should use the response data to complete the request that received the kXR_waitresp.

4) Nothing prohibits the client from waiting for different time than the indicated number of seconds. Generally, if no response is received after at least seconds have elapsed; the client should treat the condition as a fatal error.

2.5 Binary Definitions of Status, Error and Response Subcodes

2.5.1 Response Status Codes

Status Code	Value
kXR_ok	0
kXR_oksofar	4000
kXR_attn	4001
kXR_authmore	4002
kXR_error	4003
kXR_redirect	4004
kXR_status	4007
kXR_wait	4005
kXR_waitresp	4006

2.5.2 kXR_attn Subcodes

kXR_attn Subcode	Value
kXR_asyncms	5002
kXR_asynresp	5008

2.5.3 kXR_redirect Subcodes

kXR_redirect Subcode	Value
kXR_recoverWrts	0x00001000
kXR_collapseRedir	0x00002000

2.5.4 kXR_status Subcodes and Other Values

kXR_status subcode in XrdProto::	Value
kXR_FinalResult	0x00
kXR_PartialResult	0x01
kXR_ProgressInfo	0x02

kXR_status value in XrdProto::	Value
kXR_statusBodyLen	16

2.5.5 Error Codes

Error	Value	Corresponding POSIX errno Value
kXR_ArgInvalid	3000	EINVAL
kXR_ArgMissing	3001	EINVAL
kXR_ArgTooLong	3002	ENAMETOOLONG
kXR_FileLocked	3003	EDEADLK
kXR_FileNotOpen	3004	EBADF
kXR_FSError	3005	ENODEV
kXR_InvalidRequest	3006	EBADRQC
kXR_IOError	3007	EIO
kXR_NoMemory	3008	ENOMEM
kXR_NoSpace	3009	ENOSPC
kXR_NotAuthorized	3010	EACCES
kXR_NotFound	3011	ENOENT
kXR_ServerError	3012	EFAULT
kXR_Unsupported	3013	ENOTSUP
kXR_noserver	3014	EHOSTUNREACH
kXR_NotFile	3015	ENOTBLK
kXR_isDirectory	3016	EISDIR
kXR_Cancelled	3017	ECANCELED
kXR_ItExists	3018	EEXIST
kXR_ChkSumErr	3019	EDOM
kXR_inProgress	3020	EINPROGRESS
kXR_overQuota	3021	EDQUOT
kXR_SigVerErr	3022	EILSEQ
kXR_DecryptErr	3023	ERANGE
kXR_Overloaded	3024	EUSERS
kXR_fsReadOnly	3025	EROFS
kXR_BadPayload	3026	EINVAL
kXR_AttrNotFound	3027	ENOATTR
kXR_TLSRequired	3028	EPROTOTYPE
kXR_noReplicas	3029	EADDRNOTAVAIL
kXR_AuthFailed	3030	EAUTH (preferable) or EBADE
kXR_Impossible	3031	EIDRM
kXR_Conflict	3032	ENOTTY
kXR_TooManyErrs	3033	ETOOMANYREFS
kXR_ReqTimedOut	3034	ETIMEDOUT

4.3.1 kXR_ckpBegin, kXR_ckpCommit, and kXR_ckpRollback Subcodes

Purpose: Create, delete or restore a checkpoint.

Request		Normal Response
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_chkpoint	kXR_unt16	kXR_ok
kXR_char	fhandle[4]	kXR_int32	0
kXR_char	reserved[11]
kXR_char	opcode
kXR_int32	0

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

fhandle

file handle value supplied by the successful response to the associated kXR_open request that is to be used for the checkpoint request. The file should be opened in write mode.

opcode checkpoint operation wanted:

kXR_ckpBegin - Create a checkpoint.

kXR_ckpCommit - Delete an existing checkpoint to commit changes.

kXR_ckpRollback - Restore file data and delete the checkpoint.

Notes

2) The fhandle value should be treated as opaque data.

3) The fhandle should refere to a file opened for writing. If it does not, the request should fail.

4) The file should be opened in update mode without the POSC option. Creation of a checkpoint should fail if this is not the case.

5) Once a checkpoint is establish a new one should not be allowed until the existing checkpoint is committed or rolled back.

6) Should the client loose connectivity to the server, all outstanding checkpoints should be restored.

7) Should the client close a file with an outstanding checkpoint, the checkpoint should be restored.

8) Upon server restart, all outstanding checkpoints should be restored.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_chkpoint		3012	Checkpoint file data.
	opcode
	kXR_ckpBegin	0x00	Create a new checkpoint.
	kXR_ckpCommit	0x01	Delete the current checkpoint.
	kXR_ckpRollback	0x03	Restore the current checkpoint.

4.3.2 kXR_ckpQuery Subcode

Purpose: Create and execute a checkpointed operation on an open file.

Request		Normal Response
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_chkpoint	kXR_unt16	kXR_ok
kXR_char	fhandle[4]	kXR_int32	8
kXR_char	reserved[11]	kXR_int32	limit
kXR_char	kXR_ckpQuery	kXR_int32	used
kXR_int32	0

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

fhandle

file handle value supplied by the successful response to the associated kXR_open request that is to be used for the checkpoint request. The file should be opened for write mode.

limit the maximum number of data bytes that may be recorded in a checkpoint.

used the number of data bytes already recorded in the checkpoint.

Notes

2) The fhandle value should be treated as opaque data.

3) The fhandle should refere to a file opened for writing. If it does not, the request should fail.

4) When the client closes a file with an outstanding checkpoint, the checkpoint should be deleted.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_chkpoint		3012	Checkpoint file data.
	opcode
	kXR_ckpQuery	0x02	Query checkpoint limit and usage.

4.6 kXR_dirlist Request

Purpose: Enumerate the contents of a directory.

Request		Normal Response
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_dirlist	kXR_unt16	kXR_ok
kXR_char	reserved[15]	kXR_int32	dlen
kXR_char	options	kXR_char	Dirname₀\n
kXR_int32	plen		•
kXR_char	path[plen]		•
		kXR_char kXR_char	• dirname_n 0
Normal Response w/ kXR_dcksm		Normal Response w/ kXR_dstat
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_ok	kXR_unt16	kXR_ok
kXR_int32	dlen	kXR_int32	dlen
kXR_char	“.\n”	kXR_char	“.\n”
kXR_char	“0 0 0 0\n”	kXR_char	“0 0 0 0\n”
kXR_char	dirname₀\n	kXR_char	dirname₀\n
kXR_char	statinfo₀	kXR_char	statinfo₀\n
kXR_char	[ ctype:csval ]₀\n		•
	•		•
	•		•
kXR_char kXR_char	• dirname_n\n statinfo_n	kXR_char kXR_char kXR_char	dirname_n\n statinfo_n 0
kXR_char	[ ctype:csval ]_n
kXR_char	0

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

options

optionally, one or more of the following:

kXR_dstat - return stat information with each entry (protocol version 3+).

kXR_dcksm - return stat information and checksum with each entry

(protocol version 4+).

reserved

area reserved for future use and should be initialized to null characters (i.e. ‘\0’).

plen binary length of the supplied path, path.

path path of a directory whose entries are to be listed. It may be suffixed with CGI information.

dlen binary length of the data that follows dlen.

dirname

entry in the directory whose listing was requested.

statinfo

the kXR_stat information for the preceeding dirname. Refer to kXR_stat for details. The statinfo is only returned when kXR_dcksm or kXR_dstat is set and the server implements the protocol version indicated in options.

ctype the name of the checksum algorithm.

csval the checksum value as a hexadecimal ASCII text string.

Notes

2) A directory may have multiple entries and the response contains all of the entries.

3) Each directory entry should be suffixed by a new-line character; except for the last entry which should be suffixed by a null character. When kXR_dstat is specified, the last entry is the couplet “dirname\nstatinfo”.

4) When kXR_dcksm is specified, kXR_dstat should be assumed and the checksum information as “[ ctype:csval ]” (e.g. “[ adler32:d395bc71 ]”) appended to the corresponding statinfo separated by a single space. If there is no checksum value associated with the entry the csval should be set to the word none (e.g. “[ adler32:none]”).

5) If a file has more than one checksum associated with it and the request does not specify which checksum should be returned, then the default checksum should be retruned.

6) Similar to kXR_query with the kXR_Qcksum subcode, the directory path may contain the cks.ctype CGI element specifying the particular checksum that should be returned. Refer to kXR_Qcksum for details.

7) Since more entries may exist than is possible to send at one time, the kXR_oksofar protocol may be used to segment the response. Under no circumstances should a directory name be split across a response packet.

8) The server should not return the entries “.” and “..” except when kXR_dstat is specified, in which case only the “.” entry is returned.

9) An empty directory should return the eight-byte triplet {streamid, 0, 0} unless kXR_dstat is specified; in which case “{streamid,0,8}.\n0 0 0\0” should be returned.

10) Clients should always check if the server supports kXR_dstat. If the option is supported, the first entry should be a dot entry followed the zero stat information.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_dirlist		3004	List a directory
	options
	kXR_dcksm	0x04	Return checksum with entry
	kXR_dstat	0x02	Return stat information with entry
	kXR_online	0x01	Only list online entries

4.8 kXR_fattr Request

The kXR_fattr request code is used to delete, list, retrieve, and set file attributes (also known as extended attributes). This is accomplished using request subcodes. File attributes are specific to the file system being exported by the server. The exported file system has its own specific limits on the length of attribute names and the amount of data that may associated with a name. Some even limit the total amount of attribute data that may be associated with a file. Finally, not all file systems support extended attributes. The kXR_query request using the kXR_QConfig subcode with the xattr argument may be used to ascertain limits for any particular server.

The kXR_fattr request imposes its own limits on the maximum length of an attribute name (i.e. kXR_faMaxNlen, currently 248 bytes) and attributes value (i.e. kXR_faMaxVlen, currently 65536 bytes or 64K). Be aware that smaller limits may apply, depending on the underlying file system.

The kXR_fattr request supports deleting, retrieving, and setting multiple attributes with one request. However, the operation should not be considered atomic when multiple attributes are specified. A maximum kXR_faMaxVars (currently 16) attribute vales may be deleted, set, or retrieved per request.

For delete and retrieve requests, only attribute names are specified. For set requests, the attribute names are followed by the corresponding values (i.e. in 1-to-correspondence to the names) to be used for each attribute. Regardless of the subcode, the first string in each request is the path name of the file to which the request applies; which may be a null string.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_fattr		3020	Perform file attribute function
	kXR_fattrDel	0	Delete one or more attributes
	kXR_fattrGet	1	Get one or more attributes
	kXR_fattrList	2	List file attribute names
	kXR_fattrSet	3	Set one or more attributes
	isNew	0x01	Attribute must not exist
	aData	0x10	Include attribute value

4.8.1 Layout of namevec

Subsequent sections refer to namevec which is a vector whose elements are laid out as follows:


kXR_unt16	rc
kXR_char	name[]
kXR_char	0

Where:

rc as an argument is should be set to zero. In the response, it holds the status code associated with the attribute name. A status code not equal to kXR_ok indicates that the requested operation with respect to the attribute name was not completed.

name name of an attribute. The length of each name, excluding the null byte, should not be greater than kXR_faMaxNlen. Notice that the name is followed by a null byte. Attribute names are null terminated strings. These elements are concatenated together to produce a vector of names.

Notes

1) All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values.

2) There is no alignment requirement in the for the namevec element. That is, namevec elements should be streamed together irrespective of byte boundaries.

3) A namevec element should not be split across kXR_oksofar responses.

4.8.2 Layout of valuvec

Subsequent sections refer to valuvec which is a vector whose elements are laid out as follows:


kXR_int32	vlen
kXR_char	value[vlen]

Where:

vlen length of the subsequent value.

value value that the attribute is to have when issuing a kXR_fattrSet

request or the actual value of the attribute when issuing a kXR_fattrGet request.

Notes

1) All binary fields are transmitted in network byte order using an explicit length. The kXR_char and kXR_unt16 data types are treated as unsigned values.

2) There is no alignment requirement in the for the valuvec element. That is, valuvec elements should be streamed together irrespective of byte boundaries.

3) A valuvec element should not be split across kXR_oksofar responses.

4.9 kXR_gpfile Request

Purpose: Direct a server to get or put a complete file.

Request		Normal Response
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_gpfile	kXR_unt16	kXR_waitresp
kXR_unt16	options	kXR_int32	4
kXR_char	sources	kXR_int32	seconds
kXR_char	streams	Async Attn	Status Update Response
kXR_char	reserved[10]	kXR_char	pad[2]
kXR_unt16	srclen	kXR_unt16	kXR_attn
kXR_int32	totlen	kXR_int32	18
kXR_char	[cstype:csval\s]^#	kXR_unt32	kXR_asyncinfo
kXR_char	src[srclen]	kXR_char	streamid[2]
kXR_char	\s	kXR_unt16	kXR_gpfile
kXR_char	dst[]	kXR_int64	xfrbytes
		kXR_char	pctdone
		kXR_char	status
^#Should only be present when kXR_gpfcsver is specified in options.

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

options

request options:

kXR_gpfcsver - verify that specified checksum matches.

kXR_gpfdlgid - use identity delegation for the retrieval otherwise

token based authorization should be assumed.

kXR_gpfforce - remove any existing file with the same name prior to the

retrieval.

kXR_gpfkeep - do not remove any partial file upon failure.

kXR_gpfhush - do not send status updates.

kXR_gpfPut - this is a request to put a file; otherwise, get the file.

kXR_gpftls - transfer the data using TLS.

sources

the binary number of the maximum number of sources to use for the copy. A value of zero should use the default number of sources.

streams

the number of parallel streams to use for the retrieval specified in binary. A value of zero should use the default number of streams.

srclen binary length of the source URL argument, src.

totlen binary length of the arguments that follow.

cstype:csval

Specified the the checksum of cstype (e.g. adler32, crc32, md5, etc) and the corresponding value, csval that the file should have at the destination. The cstype should be a supported checksum algorithm and csval should be specified as an ASCII text hex string without the leading ‘0x’. A single space character should follow csval. A valid csval consists of an even number of characters whose length divided by two equals the algorithm’s result in binary. The retrival should fail if the checksum of the transferred file does not equal the specified value.

src URL of the source file along with any CGI information releant to the source’s location.

dst URL of the destination file along with any CGI information relevant to the destination’s location. The URL should idenify the name (i.e. path), that the file is to have at the destination server (i.e. the server to which the request is directed). The path may include CGI information to modify file creation. Elements that can be specified in the kXR_open request may be specified in the CGI. See the notes on possible restrictions.

xfrbytes

the binary number of bytes that have been processed so far reported via an asynchronous kXR_attn plus kXR_asyninfo response.

pctdone

the binary number indicating the percentage, 0 to 100, of the of operation that has been completed reported via an asynchronous kXR_attn plus kXR_asyninfo response.

status the status of the retrieval request as one of:

kXR_gpfpend - not started

kXR_gpfxfr - transferring data

kXR_gpfver - performing checksum verification

Notes

2) Support for kXR_gpfile may be determined from the kXR_protocol response.

3) It is up to the implementation whether or not a third party transfer is cancelled when network connectivity is lost to the client. Minimally, the desired action is for pending requests be removed from the transfer queue.

4) It is up to the implementation whether or not protocols beyond file, xroot and xroots are supported for the retrieval or sending of src. Ostensibly, the protocol specification allows the client to specify an arbitrary protocol to be used (e.g. http, s3, etc) in the src and dst URLs. Should a specified protocol not be supported the request should fail.

5) The kXR_gpfile request is primarily geared for token based authorization retrieval. However, it does allow delegated identity retrieval. An implementation should support token based authorization if it supports kXR_gpfile. Delegated identity retrieval is an optional extension. However, an error should be reported if the kXR_gpfdlgid option is set but not supported.

6) An implementation should assure that if a transfer fails for any reason whatsoever, the destination file is removed.

7) The kXR_gpfkeep is meant for debugging pruposes to allow failing transfer to be better diagnosed.

8) The client should handle a kXR_auhmore response to the the kXR_gpfile request. This may occur if the server needs to obtain delegated credentials to continue the request (e.g. kXR_gpfdlgid was specified). This is independent of any previous kXR_authmore response that the client may have handled (e.g. during a kXR_login request).

9) The general response to a successful kXR_gpfile request should be kXR_waitresp. This allows the retrieval to occur asynchronously to client execution with possible asynchronous status updates. When the request completes the client should receive a final response indicating success or failure.

10) Status updates should be handled by an asynchronous kXR_attn plus kXR_asyninfo response. The frequency is implementation dependent but typically should be spaced between 3 to 5 seconds.

11) The final response should be provided via an asynchronous kXR_attn plus kXR_asynresp response.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_gpfile		3005	n/a
	options
	kXR_gpfcsver	0x0001	Check supplied ofr verification.
	kXR_gpfdlgid	0x0002	Use delegated identity.
	kXR_gpfforce	0x0004	Remove file at destination first.
	kXR_gpfkeep	0x0008	Keep file upon failure.
	kXR_gpfhush	0x0010	Do not send status updates.
	kXR_gpfPut	0x0020	Send the file to the destination.
	kXR_gpftls	0x0040	Send the data using TLS.
	status
	kXR_gpfpend	0x00	Request is pending.
	kXR_gpfxfr	0x01	Request is transferring data.
	kXR_gpfver	0x02	Request is verifying the checksum.

4.10 kXR_locate Request

Purpose: Locate a file.

Request		Normal Response
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_locate	kXR_unt16	kXR_ok
kXR_int16	options	kXR_int32	rlen
kXR_char	reserved[14]	kXR_char	info[rlen]
kXR_int32	plen
kXR_char	path[plen]

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

options

options to apply when path is opened. The options are an “or’d” combination of the following values:

kXR_nowait - provide information as soon as possible

kXR_prefname - hostname response is prefered

kXR_refresh - update cached information on the file’s location

(see notes)

reserved

area reserved for future use and should be initialized to null characters (i.e., ‘\0’).

plen binary length of the supplied path, path.

path path of the file to be located. CGI information appended to the path does not affect the request. Path may also start with an asterisk or be only an asterisk with the following meaning:

* - return all connected managers and servers

*path - return all managers and servers exporting path

rlen byte length of the response that follows

info zero or more node types, IPV6 hybrid addresses, and port numbers of nodes that have the file. The port number is to be used to contact the node.

Node Entry Response Format

xy[::aaa.bbb.ccc.ddd.eee]:ppppp

xyhostname:ppppp

Where:

x is a single character that identifies the type of node whose IP address follows. Valid characters are:

M - Manager node where the file is online

m - Manager node where the file is pending to be online.

S - Server node where the file is online

s - Server node where the file is pending to be online.

y is a single character that identifies the file access mode at the node whose IP address follows. Valid characters are:

r - Read access allowed

w - Read and write access allowed.

aaa.bbb.ccc.ddd.eee

IPv4 portion of the IPV6 node address, for IPV4 environments. Otherwise, a true IPV6 address is returned.

hostname

hostname for the node address. This format may only be returned when kXR_prefname is specified, but does not forbid an address reply.

ppppp port number to be used for contacting the node.

Notes

2) Option flags are the same as those defined for the kXR_open request.

3) The kXR_refresh voids the kXR_nowait option.

4) If the file resides in more than one location, each location is separated by a space.

5) The kXR_nowait option provides a location as soon as one becomes known. This means that not all locations are necessarily returned. If the file does not exist, a wait is still imposed.

6) If available, use the inet_ntop() and inet_pton() function to convert addresses to suitable format as these accepts traditional IPV4 address as well as IPV6 addresses.

7) Nodes identified as M or m, do not actually hold the file. These are manager nodes that know other locations for the file. To obtain the real file location, the client should contact each M(m) node and issue a kXR_locate request. The processes is iterative, as the response from an M(m) node may identified other M(m) nodes.

8) Clients should guard against circular references by setting an absolute depth limit in the number of M(m) to M(m) references they will accept before declaring an error. A limit of 4 covers a range of 16,777,216 possible locations.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_locate		3027	Perform location operation
	options
	kXR_compress	0x00 01	Return unique hosts
	kXR_nowait	0x20 00	Return immediate information
	kXR_prefname	0x01 00	Preferentially return DNS names
	kXR_refresh	0x00 80	Refresh cached information

4.11 kXR_login Request

Purpose: Initialize a server connection.

Request		Normal Response
			server < 2.4.0 \| client < 1.0
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_login	kXR_unt16	kXR_ok
kXR_int32	pid	kXR_int32	slen
kXR_char	username[8]	kXR_char	sec[slen]
kXR_char	reserved		server >= 2.4.0 & client >= 1.0
kXR_char	ability	kXR_char	streamid[2]
kXR_char	capver	kXR_unt16	kXR_ok
kXR_char	reserved	kXR_int32	slen+16
kXR_int32	tlen	kXR_char	sessid[16]
kXR_char	token[tlen]	kXR_char	sec[slen]

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

reserved

area reserved for future use and should be initialized to null characters (i.e., ‘\0’).

pid process number associated with this connection.

username

unauthenticated name of the user to be associated with the connection on which the login is sent.

ability client’s extended capabilities represented as bit flags, as follows:

0b00000001 the client accepts full standard URL’s in a redirection response. Unless the following ability is set, the protocol in the URL should remain xroot. This bit is also identified as kXR_fullurl.

0b00000011 the client accepts protocol changes in a full standard URL’s in a redirection response. Unless the this ability is set, the protocol in the URL should remain xroot. This bit is also identified as kXR_multipr.

0b00000100 the client accepts protocol redirects during a kXR_read and kXR_readv requests.This bit is also identified as kXR_readrdok.

0b00001000 the client is dual-stacked and supports IPv4 and IPv4 connections.This bit is also identified as kXR_hasipv64.

0b00010000 the client only supports IPv4 connections.This bit is also identified as kXR_onlyprv4.

0b00100000 the client only supports IPv6 connections.This bit is also identified as kXR_onlyprv6.

0b01000000 the client only supports local file access.This bit is also identified as kXR_lclfile.

0b10000000 the client supports redirect flags in the kXR_redirect response.This bit is also identified as kXR_redirflags.

capver

client’s capabilities combined with the binary protocol version number of the client. The capabilities reside in the top-most two bits while the protocol version number is encoded in the lower 6 bits. Currently, for capabilities two values are possible:

0b00vvvvvv - client only supports synchronous responses

0b10vvvvvv - (kXR_asyncap) client supports asynchronous responses

tlen binary length of the supplied token, token. If no token is present, tlen is zero.

token token supplied by the previous redirection response that has initiated this login request plus other optional elements.

slen binary length of the information, sec, that follows slen.

sessid opaque session identifier associated with this login. The sessid is always present when the server protocol version if greater than or equal to 2.4.0 and the client protocol version if greater than 0.

sec null-terminated security information. The information should be treated as opaque and is meant to be used as input to the security protocol creation routine XrdSecGetProtocol().

Notes

2) If no security information is returned (i.e., slen is zero), the XRootD server does not require that the client authenticate.

3) If security information is returned, then the client should create the security context allowed by the security information, obtain credentials, and send them using the kXR_auth request.

4) Authentication should occur prior to any operation that requires authentication. See the table on page 13 for a list of requests that should be authenticated.

5) A subsequent kXR_auth request may revert the login into a normal user login should XRootD find that the authenticated user cannot assume the role of administrator.

6) Sending a kXR_login request on a previously authenticated connection destroys the authentication context; requiring that the connection be re-authenticated.

7) The sessid is used in kXR_bind and kXR_endsess requests.

8) When the client indicates kXR_lclfile along with kXR_fullurl then the client should accept redirects to a local file the via file:// protocol indicator.

9) The kXR_redirflags should be only used in conjunction with the kXR_redirect server response.

10) Opaque information should be treated as truly opaque. The client should not inspect nor modify opaque information in any way.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_login		3007	Perform server login
	ability
	kXR_fullurl	0x01	Accepts full URL redirect
	kXR_hasipv64	0x08	IPv4 and IPv6 capable
	kXR_multipr	0x03	Accepts non-root protocol redirects
	kXR_nothing	0x00	No special abilities
	kXR_onlyprv4	0x10	Only accepts private IPv4 addresses
	kXR_onlyprv6	0x20	Only accepts private IPv6 addresses
	kXR_lclfile	0x40	Supports local file access.
	kXR_redirflags	0x80	Supports kXR_redirect flags.
	capver
	kXR_asyncap	0x80	Supports asynchronous responses
	kXR_vermask	0x3f	Mask to isolate kXR_vernnn
	kXR_ver000	0x00	Predates 2005 protocol
	kXR_ver001	0x01	Implements original 2005 protocol
	kXR_ver002	0x02	Implements above + async responses
	kXR_ver003	0x03	Implements above + 2011 extensions
	kXR_ver004	0x04	Implements above + request signing
	kXR_ver005	0x05	Implements above + TLS

4.11.1 Additional Login CGI Tokens

The following table lists additional CGI tokens that may be passed to further identify the client. They are passed in the token argument.

Token	Token Value
xrd.cc	the two character country code of the client’s location
xrd.if	the client’s interface speed in gigabits gggg[.mm]
xrd.ll	the comma separated latitude and longtitude of the client in degree [-]DDD[.dddddd] format
xrd.tz	signed timezone relative to UDT of client’s location

4.12 kXR_mkdir Request

Purpose: Create a directory.

Request		Normal Response
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_mkdir	kXR_unt16	kXR_ok
kXR_char	options	kXR_int32	0
kXR_char	reserved[13]
kXR_unt16	mode
kXR_int32	plen
kXR_char	path[plen]

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

reserved

area reserved for future use and should be initialized to null characters (i.e., ‘\0’).

options

options to apply when path is created. The options are an “or’d” combination of the following values:

kXR_mkdirpath - create directory path if it does not already exist

mode access mode to be set for path. The access mode is an “or’d” combination of the following values:

Access	Readable	Writeable	Searchable
Owner	kXR_ur	kXR_uw	kXR_ux
Group	kXR_gr	kXR_gw	kXR_gx
Other	kXR_or	not supported	kXR_ox

plen binary length of the supplied path, path.

path path of the of the directory to be created. The path may be suffixed with CGI information.

Notes

2) When a directory path is created, as requested by the kXR_mkdirpath option, the directory permission specified in mode are propagated along the newly created path.

3) No umask applies to the specified mode.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_mkdir		3008	Create a directory
	mode
	kXR_ur	0x01 00	Owner readable
	kXR_uw	0x00 80	Owner writable
	kXR_ux	0x00 40	Owner searchable (directories)
	kXR_gr	0x00 20	Group readable
	kXR_gw	0x00 10	Group writable
	kXR_gx	0x00 08	Group searchable (directories)
	kXR_or	0x00 04	Other readable
	kXR_ow	0x00 02	Other writable (not allowed)
	kXR_ox	0x00 01	Other searchable (directories)
	options
	kXR_mkdirpath	0x01	Create missing directories in path

4.14 kXR_open Request

Purpose: Open a file or a communications path.

Request		Normal Response
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_open	kXR_unt16	kXR_ok
kXR_unt16	mode	kXR_int32	rlen
kXR_unt16	options	kXR_char	fhandle[4]
kXR_unt16	optiont		*optional addition*
kXR_char	reserved[6]	kXR_int32	cpsize
kXR_char	fhtemplt[4]	kXR_char	cptype[4]
kXR_int32	plen	kXR_char	info[resplen-12]
kXR_char	path[plen]

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

mode advisory mode in which path is to be opened. The mode is an “or’d” combination of the following values:

Access	Readable	Writeable	Executable
Owner	kXR_ur	kXR_uw	kXR_ux
Group	kXR_gr	kXR_gw	kXR_gx
Other	kXR_or	not supported	kXR_ox

options

options to apply when path is opened. The options are an “or’d” combination of the following values:

kXR_async - open the file for asynchronous i/o (see notes)

kXR_compress - open a file even when compressed (see notes)

kXR_delete - open a new file, deleting any existing file

kXR_force - ignore file usage rules

kXR_mkpath - create directory path if it does not already exist

kXR_new - open a new file only if it does not already exist

kXR_open_apnd - open only for appending

kXR_open_read - open only for reading

kXR_open_updt - open for reading and writing

kXR_posc - enable Persist On Successful Close (POSC) processing

kXR_refresh - update cached information on the file’s location

(see notes)

kXR_replica - the file is being opened for replica creation

kXR_retstat - return file status information in the response

kXR_seqio - file will be read or written sequentially (see notes)

optiont

additional options to apply when path is opened. The optiont are an “or’d” combination of the following values:

kXR_dup - the opened file should have the same contents as the

one referenced by fhtemplt (see notes).

kXR_samefs - the opened file should be in the same file system as

The one referenced by fhtemplt (see notes).

reserved

area reserved for future use and should be initialized to null characters (i.e., ‘\0’).

fhtemplt

is the file handle returned by a previous kXR_open request to a file to be used as the template file when kXR_dup or kXR_samefs is set in optiont. If neither option is set then this field should be set to null characters.

plen binary length of the supplied path, path.

path path of the file to be opened. The path may be suffixed with CGI information to provide additional information necessary to properly process the request. See the following section on CGI information for more information.

resplen

byte length of the response that follows. At least four bytes should be returned.

fhandle

file handle for the associated file. The file handle should be treated as opaque data. It should be used for subsequent kXR_close, kXK_read, kXR_sync, and kXR_write requests.

cpsize compression page size. The cpsize field is returned when the kXR_compress or kXR_retstat have been specified. Subsequent reads should be equal to this value and read offsets should be an integral multiple of this value. If cpsize is zero, the file is not compressed and subsequent reads may use any offset and read length.

cptype name of the compression algorithm used to compress the file (e.g. lz4). The cptype field is returned when the kXR_compress or kXR_retstat have been specified. If the file is not compressed, the first byte of the four byte field is a null byte (\0). For compressed files, subsequent reads should use the returned algorithm to decompress each cpsize worth of data data.

info same information that kXR_stat returns for the file. This information is returned only if kXR_retstat is set and the server is at protocol version 2.4.0 or greater. The cpsize and cptype fields are always returned and are only meaningful if kXR_compress has been specified. Otherwise, cpsize and cptype are set to values indicating that the file is not compressed.

Notes

2) Open fails if the path designates a directory.

3) No umask applies to the specified mode.

4) The kXR_async option tells the server to overlap file i/o with network requests as much as possible for this file. For instance, read requests may be done in parallel with other read requests sent on the same link. This option is only useful if the client is able to issue multiple requests (i.e., is not serializing the requests-response stream).

5) While the kXR_async option applies to write operations, as well. Server-side asynchronous opportunities are far more limited. The client needs to perform appropriate multiplexing of write requests with other requests to gain improved parallelism.

6) The kXR_async option imposes additional overhead on the server and should only be specified when the client can take advantage of request-response parallelism.

7) The kXR_refresh option imposes additional overhead on the server because it requires that the server obtain the most current information on the file’s location before attempting to process the open request. This option should only be used as part of the error recovery process outlined in section “Client Recovery From File Location Failures”.

8) The kXR_refresh option is ignored by any server not functioning as a primary redirecting server.

9) When a directory path is created, as requested by the kXR_mkpath option, the directory permission of 0775 (i.e., rwxrwxr-x) are propagated along the newly created path.

10) Only files may be opened using the kXR_open request code.

11) The kXR_retstat option is meant to eliminate an additional server request for file status information for applications that always need such information.

12) The kXR_seqio option is meant to be advisory. A server may choose to optimize data layout or access based on this hint. Misusing the hint may lead to degraded performance.

13) The kXR_posc option requests safe file persistence which persists the file only when it has been explicitly closed.

14) The kXR_dup option allows the initial contents of a file to be a duplicate of the file referenced in fhtemplt (i.e. the template file). This option may imply kXR_samefs, depending on the implementation.

15) The kXR_samefs allows you to open a new file in the same file system where the file referenced by fhtemplt resides.

Binary Definitions

Request	Modifiers	Value	Explanation
kXR_open		3010	Open a file
	mode
	kXR_ur	0x01 00	Owner readable
	kXR_uw	0x00 80	Owner writable
	kXR_ux	0x00 40	Owner searchable (directories)
	kXR_gr	0x00 20	Group readable
	kXR_gw	0x00 10	Group writable
	kXR_gx	0x00 08	Group searchable (directories)
	kXR_or	0x00 04	Other readable
	kXR_ow	0x00 02	Other writable
	kXR_ox	0x00 01	Other searchable (directories)
	options
	kXR_async	0x00 40	Allow asynchronous I/O
	kXR_compress	0x00 01	Open without inflating files
	kXR_delete	0x00 02	Delete any existing file
	kXR_force	0x00 04	Disregard locking rules
	kXR_mkpath	0x01 00	Create any missing directories
	kXR_new	0x00 08	Create a new file
	kXR_open_apnd	0x02 00	Open only for appending
	kXR_open_read	0x00 10	Open only for reading
	kXR_open_updt	0x00 20	Open for reading and writing
	kXR_open_wrto	0x80 00	Open only for writing
	kXR_posc	0x10 00	Persist on successful close
	kXR_refresh	0x00 80	Refresh cached information
	kXR_replica	0x08 00	Open for replication
	kXR_retstat	0x04 00	Return file stat information
	kXR_seqio	0x40 00	Open for sequential I/O

optiont
kXR_dup	0x00 01	Create duplicate file
kXR_samefs	0x00 02	Create file in a particular filesys

4.14.1 Additional Open CGI Tokens

The kXR_open request allows a client to pass CGI information to properly steer the open. The information may or may not be acted upon, depending on the server’s capabilities. The following table lists the defined CGI tokens.

Token	Token Value
ofs.posc	When set to a value of 1 requests “persist on successful close” processing. This is historical as the kXR_posc option should be preferentially used.
oss.asize	The mber of bytes to reserve for a new file.
oss.cgroup	The desired space name (a.k.a space token).

Notes

1) Unrecognized CGI tokens should be ignored.

2) Invalid arguments to a recognized CGI token should result in the termination of the request.

Example

/tmp/foo&oss.cgroup=index

4.16 kXR_pgread Request

Purpose: Read one or more integrity protected data pages from an open file.

Request		Normal Response
kXR_char	streamid[2]	kXR_char	streamid[2]
kXR_unt16	kXR_pgread	kXR_unt16	kXR_status
kXR_char	fhandle[4]	kXR_int32	resplen
kXR_int64	offset	kXR_unt32	crc32c
kXR_int32	rlen	kXR_char	streamid[2]
kXR_int32	alen	kXR_char	pgrid
		kXR_char	pgrtype
		kXR_char	reserved[4]
Arguments	when alen > 0	kXR_int32	dlen
kXR_char	pathid alen>0	kXR_int64	offset
kXR_char	reqflags alen=2	kXR_char	data[dlen]

Where:

streamid

binary identifier that is associated with this request stream. This identifier should be echoed along with any response to the request.

offset binary offset from which the data is to be read. For best performance, the offset should be an integral multiple of the page size. However, unaligned offsets are allowed but require special data framing as described in a subsequent section. In the response, it is the file offset from which the returned data was read.

rlen binary maximum amount of data that is to be read.

alen binary length of the arguments that follow the request header.

pathid

when alen is > 0, this is a path identifier returned by kXR_bind. The response data is sent via this path, if possible. If pathid is not specified or is zero, the login stream should be used to deliver the response. When pathid is set to kXR_AnyPath then the server can use any bound path to return the response.

reqflags

when alen >= 2, these are request flags, as follows:

kXR_pgRetry - request is a retry of a previous request.

resplen

binary length of the response that follows excluding the data portion.

crc32c CRC32-C as defined by the IETF RFC 7143 standard (see the kXR_status response for details) of the resplen-sizeof(crc32c) bytes immediately after crc32c. This means that the data portion, if any, should not be included in the cr32c calculation.

pgrid response signature and should be equal to kXR_pgread-kXR_1stRequest.

pgrtype

indicates the type of status being reported. Only the following type codes are allowed relative to namespace XrdProto:

kXR_FinalResult - All of the data has been transmitted.

kXR_PartialResult - Partial data has been transmitted; additional data

should be expected on this stream.

datalen

binary length of the of the data, data, that was actually read plus associated checksums.

data data that was read. Each page or page segment should be preceeded by a 4 byte CRC23C checksum. The first page may be shorter than a full page if the offset is not page aligned. The last page may be shorter than a full page if it is the last one in the file being read and is incomplete.

Notes

1) All binary fields are transmitted in network byte order using an explicit length. The kXR_char, kXR_unt16 and kXR_unt32 data types are treated as unsigned values. All reserved fields should be initialized to binary zero.

2) Support for kXR_pgread may be determined from the kXR_protocol response and testing for the presence of the kXR_suppgrw flag.

3) The fhandle value should be treated as opaque data.

4) The kXR_pgPageSZ defines the page size (currently 4096 bytes).

Status	Response Data
kXR_attn	Parameters to direct immediate client action
kXR_authmore	Authentication specific data
kXR_error	Error number and corresponding ASCII message text
kXR_ok	Depends on request (this is predefined to be the value 0)
KXR_oksofar	Depends on request
kXR_redirect	Target port number and ASCII host name or URL
kXR_status	Depends on request
kXR_wait	Binary number of seconds & optional ASCII message
kXR_waitresp	Binary number of seconds

kXR_char	streamid[2]
kXR_unt16	kXR_authmore
kXR_int32	dlen
kXR_char	data[dlen]

2.4.2.1 kXR_attn Response for kXR_asyncms Client Action

2.4.3 kXR_authmore Response Format

2.4.4 kXR_error Response Format

2.4.5 kXR_ok Response Format

2.4.6 kXR_oksofar Response Format

2.4.8 kXR_status Response Format

2.4.10 kXR_waitresp Response Format

4.2.1 TLS Considerations

4.8.1 Layout of namevec

4.8.2 Layout of valuvec

4.11 kXR_login Request