xrdcopy(1)

📦
version 6.2.0 · 13 versions
, in epel 10, fedora 44

💖 xrdcopy(1) 💖

okay bestie, so this is xrdcopy(1) from the General Commands Manual section, and it is literally the glow-up era of file copying. buckle up 💅

✨ NAME ✨

xrdcp — copy files 🦋

like. just. copy. files. iconic behavior honestly.

🌸 SYNOPSIS 🌸

xrdcp [options] source destination

options: [--cksum args] [--debug lvl]
[--coerce] [--force] [--help] [--license]
[--nopbar] [--posc] [--proxy ipaddr:port]
[--recursive] [--retry time] [--server]
[--silent] [--sources num] [--streams num]
[--tpc [delegate] [push] first|only]
[--tpc-token-file file] [--verbose]
[--version] [--xrate rate] [--zip file]
[--dynamic-src] [--infiles fn]
[--parallel n] [--xattr]
[--notlsok] [--tlsnodata] [--tlsmetalink]
[--zip-mtln-cksum] [--rm-bad-cksum] [--continue]
[--xrate-threshold] [--retry-policy]
[--header name: value]

legacy options: [-adler] [-DSparm string] [-DIparm number]
[-md5] [-np] [-ODcgi] [-OScgi] [-x]

💫 DESCRIPTION 💫

so, the xrdcp utility copies one or more files from one location to another. the data source and destination may be a local or remote file or directory. and plot twist — the data source may also reside on multiple servers. multitasking queen 👑

🎀 OPTIONS 🎀

  • -C | --cksum type[:value|print|source] — grabs the checksum of type (like adler32, crc32, md5 or zcrc32) from the source, computes the checksum at the destination, and verifies they're the same. if you pick auto as the checksum type, xrdcp will try to automatically infer the right checksum type based on source/destination configuration, source file type (metalink, ZIP), and available checksum plug-ins. if a value is specified, it's used as the source checksum. and when print is specified, the checksum at the destination gets printed but is not verified. trust but verify? not here bestie 💀
  • -d | --debug lvl — debug level: 1 (low), 2 (medium), 3 (high)
  • -F | --coerce — ignores locking semantics on the destination file. > ⚠️ this option may lead to file corruption if not properly used. not me warning you, that's literally in the docs.
  • -f | --force — re-creates a file if it's already present. it said "refresh" and meant it ✨
  • -H | --header name: value — adds the given header to the requests sent to an http or https endpoint. may be repeated to add more than one header. requires that the source or the destination is an http or https URL. the Connection, Content-Length, Expect, Host, Keep-Alive, Proxy-Authenticate, Proxy-Authorization, Proxy-Connection, TE, Trailer, Transfer-Encoding and Upgrade headers are reserved and cannot be overridden. each of these names prepended with TransferHeader is reserved as well.

in a third party copy (--tpc), the headers go to the server the client connects to: the destination server in pull mode, or the source server in push mode. to send a header to the other server — the one the client does not connect to — prepend TransferHeader to the header name. for example, in pull mode:

-H "X-Request-Id: <destination-id>"
-H "TransferHeaderX-Request-Id: <source-id>"

the destination server gets the X-Request-Id header, and sends the value of TransferHeaderX-Request-Id as the X-Request-Id header of its request to the source server. lowkey elegant i fear 🧚

if the option is not specified, the XRD_HTTPHEADERS environment variable is used instead. it holds the headers as a newline separated list of "name: value" entries.

  • -h | --help — displays usage information.
  • --license — displays license terms and conditions. the legal lore 📜
  • -N | --nopbar — does not display the progress bar. for when the bar is too much main character energy.
  • -P | --posc — requests POSC (persist-on-successful-close) processing to create a new file. files are automatically deleted should they not be successfully closed. all or nothing, periodt 💅
  • -D | --proxy proxyaddr:proxyport — [NOT YET IMPLEMENTED]

use proxyaddr:proxyport as a SOCKS4 proxy. only numerical addresses are supported.

  • -r | --recursive — recursively copy all files starting at the given source directory. go off, do the whole tree 🌳
  • --retry — retry failed copy-jobs. it gets back up again 🫶
  • --server — runs as if in a server environment. used only for server-side third party copy support.
  • -s | --silent — neither produces summary information nor displays the progress bar. zero noise, all vibes.
  • -y | --sources num — uses up to num sources to copy the file.
  • -S | --streams num — use num parallel data streams to do the transfer (the main stream is in this case not used to carry out the transfer). the maximum value is 15. the default is 0 (i.e., use only the control stream).
  • --tpc [delegate] [push] first|only — copies the file from remote server to remote server using third-party-copy protocol (i.e., data flows from server to server). the source and destination servers must support third party copies. additional security restrictions may apply and may cause the copy to fail if they cannot be satisfied. the argument first tries tpc and if it fails, does a normal copy; while only fails the copy unless tpc succeeds. when delegate is specified, the copy delegates the command issuer's credentials to the target server which uses those credentials to authenticate with the source server. delegation is ignored if the target server is not configured to use delegated credentials. currently, only gsi credentials can be delegated. when push is specified, the copy is driven by the source server which pushes the data to the destination. by default, the copy is driven by the destination server which pulls the data from the source. push copies only apply to third-party copies performed over HTTP(S). server-to-server, no cap 💅
  • --tpc-token-file file — specifies the file holding the bearer tokens used to authenticate the third-party copy. the first line of file contains the token presented to the source and the second line the token presented to the destination. alternatively, file may hold a JSON object whose 'src' property is the token presented to the source and whose 'dst' property is the token presented to the destination, as in '{ "src": "<token>", "dst": "<token>" }'. in both formats a token may be left out, by an empty line or by an omitted property, in which case the corresponding side is given no token, so a pull or a push may well be authenticated with a single token. each token is sent as an 'Authorization: Bearer' header, so the option only applies to third-party copies performed over HTTP(S). if the option is not specified, the file named by the BEARER_TOKEN_FILE environment variable is used instead. the copy fails if file cannot be opened or if it holds neither a source nor a destination token.
  • -v | --verbose — displays summary output. tell me everything 🔥
  • -V | -version — displays version information and immediately exits. hi and bye 👋
  • -z | --zip file — copy given file from a ZIP archive (same as xrdcl.unzip opaque info).
  • -X | --xrate rate — limits the copy speed to the specified rate. the rate may be qualified with the letter k, m, or g to indicate kilo, mega, or giga bytes, respectively. the option only applies when the source or destination is local. speed limit but make it fashion.
  • -X | --xrate-threshold rate — if the transfer rate drops below given threshold force the client to use different source or if no more sources are available fail the transfer.
  • -Z | --dynamic-src — file size may change during the copy. chaotic, we love that for it 🙈
  • -I | --infiles fn — specifies the file that contains a list of input files.
  • --parallel n — number of copy jobs to be run simultaneously. multitasking icon.
  • --xattr — preserve extended attributes.
  • --notlsok — if server is too old to support TLS encryption fallback to unencrypted communication.
  • --tlsnodata — in case of roots/xroots protocol, encrypt only the control stream and leave the data streams unencrypted.
  • --tlsmetalink — treat all URLs in metalink as roots/xroots.
  • --zip-mtln-cksum — use the checksum available in a metalink file even if a file is being extracted from a ZIP archive.
  • --rm-bad-cksum — remove the target file if checksum verification failed (enables also POSC semantics). failing the vibe check = deleted 💀
  • --continue — continue copying a file from the point where the previous copy was interrupted. we don't restart, we resume ✨

🦋 LEGACY OPTIONS 🦋

legacy options are provided for backward compatibility. these are now deprecated and should be avoided. i'm not saying anything, i'm just saying 👀

  • -adler — equivalent to "--cksum adler32:source".
  • -DIpname numberval — set the internal parameter pname with the numeric value numberval.
  • -DSpname stringval — set the internal parameter pname with the string value stringval.
  • -md5 — equivalent to "--cksum md5:source".
  • -np — equivalent to "--nopbar".
  • -ODcgi — add cgi information cgi to any destination xrootd URL. you should specify the opaque information directly on the destination URL.
  • -OScgi — add cgi information cgi to any source xrootd URL.
  • -x — equivalent to "--sources 12".

💜 OPERANDS 💜

  • source — a dash (i.e. -) indicating stanard in, a local file, a local directory name suffixed by /, or an xrootd URL in the form of
xroot://[user@]host[:port]/absolutepath

the absolutepath can be a directory.

  • destination — a dash (i.e. -) indicating stanard out, a local file, a local directory name suffixed by /, or an xrootd URL in the form
xroot://[user@]host[:port]/absolutepath

the absolutepath can be a directory.

bestie there's more where that came from, hold on 😭💖

🌿 ENVIRONMENT 🌿

okay bestie, this is the part where the client reads your ✨vibes✨ from the environment. All these variables apply to xrdfs and every other app that's running the libXrdCl library, unless the description specifically says otherwise. And see the stuff in the brackets? That's the name of the matching xrdcp commandline parameter. iykyk 🫶

  • XRD_LOGLEVEL — this one controls how much diagnostic tea gets spilled. Valid values are: Dump, Debug, Info, Warning, and Error.
  • XRD_LOGFILE — set this and all those diagnostics get written to the file you name instead of going to stderr. no messy terminal, we're organized 💅
  • XRD_LOGMASK — decides which diagnostic topics are printed at all, at every level. It's a "|" separated list of topics. The first element can be "All", in which case every topic is enabled and the elements after it can turn some back off — or "None", in which case everything is disabled and the following flags can turn things back on. If a topic name is prefixed with "^", that means the topic should be disabled. If there's no prefix, that means the topic should be enabled.

You can also handle the log mask per diagnostic level, separately, by setting one or more of these bad boys: XRD_LOGMASK_ERROR, XRD_LOGMASK_WARNING, XRD_LOGMASK_INFO, XRD_LOGMASK_DEBUG, and XRD_LOGMASK_DUMP. The default for each level is "All", except for the Dump level, where the default is "All|^PollerMsg". Which means: at Dump level, every topic except "PollerMsg" is enabled. slay ✨

Available topics: AppMsg, UtilityMsg, FileMsg, PollerMsg, PostMasterMsg, XRootDTransportMsg, TaskMgrMsg, XRootDMsg, FileSystemMsg, AsyncSockMsg, TlsMsg

  • XRD_TLSDBGLVL — sets the debug level for the TLS component. Valid values are: OFF, CTX, SOK, SIO, ALL and OUT. The default value is OFF.
  • XRD_PARALLELEVTLOOP — the number of event loops aka the number of threads handling requests. Default number is 10. ten threads living their truth 🧚
  • XRD_READRECOVERY — turns read recovery on or off (on by default, bestie).
  • XRD_WRITERECOVERY — turns write recovery on or off (on by default).
  • XRD_OPENRECOVERY — turns open recovery on or off for mutable (truncate or create) opens (on by default).
  • XRD_CONNECTIONWINDOW (-DIConnectionWindow) — a time window for establishing the connection. If the connection isn't established inside that window, it's officially a connection failure. And if a connection failure happens earlier than that, another connection attempt only gets made at the beginning of the next window. patience is a virtue ig 😭
  • XRD_CONNECTIONRETRY (-DIConnectionRetry) — how many connection attempts (aka how many available connection windows) get made before the failure is declared permanent. fr fr, after that it's over.
  • XRD_REQUESTTIMEOUT (-DIRequestTimeout) — the default time after which an error is declared when it was impossible to get a response to a request.
  • XRD_STREAMTIMEOUT (-DIStreamTimeout) — the default time after which a connection error is declared (and a recovery gets attempted) if there are unfulfilled requests and there's no socket activity or a registered wait timeout.
  • XRD_SUBSTREAMSPERCHANNEL (-DISubStreamsPerChannel) — number of streams per session.
  • XRD_TIMEOUTRESOLUTION (-DITimeoutResolution) — the resolution for timeout events. i.e. timeout events only get processed every XRD_TIMEOUTRESOLUTION seconds.
  • XRD_STREAMERRORWINDOW (-DIStreamErrorWindow) — the time after which the permanent failure flags get cleared out and a new connection may be attempted if needed. fresh start energy 🌸
  • XRD_RUNFORKHANDLER (-DIRunForkHandler) — decides whether the fork handlers should be enabled, which is what makes the API fork safe.
  • XRD_REDIRECTLIMIT (-DIRedirectLimit) — the maximum number of allowed redirections.
  • XRD_NOTAUTHORIZEDRETRYLIMIT (-dNotAuthorizedRetryLimit) — the maximum number of allowed retries at a meta-manager for a not-authorized error.
  • XRD_POLLERPREFERENCE (-DSPollerPreference) — a comma separated list of poller implementations, in order of preference. The default is: built-in.
  • XRD_CLIENTMONITOR (-DSClientMonitor) — path to the client monitor library.
  • XRD_CLIENTMONITORPARAM (-DSClientMonitorParam) — extra optional parameters that get passed to the monitoring object on initialization.
  • XRD_WORKERTHREADS (-DIWorkerThreads) — number of threads processing user callbacks.
  • XRD_CPPARALLELCHUNKS (-DICPParallelChunks) — the maximum number of asynchronous requests being processed by the xrdcp command per connected channel substream (it gets adjusted in real-time, she's adaptive 💅).
  • XRD_CPCHUNKSIZE (-DICPChunkSize) — the size of a single data chunk handled by xrdcp.
  • XRD_NETWORKSTACK (-DSNetworkStack) — the network stack the client should use to connect to the server. Possible values are:
    • IPAuto — automatically detect which IP stack to use
    • IPAll — use IPv6 stack (AF_INET6 sockets) and both IPv6 and IPv4 (mapped to IPv6) addresses
    • IPv6 — use only IPv6 stack and addresses
    • IPv4 — use only IPv4 stack (AF_INET sockets) and addresses
    • IPv4Mapped6 — use IPv6 stack and mapped IPv4 addresses
  • XRD_DATASERVERTTL (-DIDataServerTTL) — the time period after which an idle connection to a data server should be closed.
  • XRD_LOADBALANCERTTL (-DILoadBalancerTTL) — the time period after which an idle connection to a manager or a load balancer should be closed.
  • XRD_APPNAME (-DSAppName) — override the application name that gets reported to the server. main character rebrand ✨
  • XRD_PLUGINCONFDIR — a custom location containing client plug-in config files.
  • XRD_PLUGIN — a default client plug-in to be used.
  • XRD_CPINITTIMEOUT (-DICPInitTimeout) — the maximum time allowed for the copy process to initialize, i.e. open the source and destination files.
  • XRD_CPTPCTIMEOUT (-DICPTPCTimeout) — the maximum time allowed for a third-party copy operation to finish.
  • XRD_TCPKEEPALIVE (-DITCPKeepAlive) — enable/disable the TCP keep alive functionality
  • XRD_TCPKEEPALIVETIME (-DITCPKeepAliveTime) — the time between the last data packet sent and the first keepalive probe (Linux only)
  • XRD_TCPKEEPALIVEINTERVAL (-DITCPKeepAliveInterval) — the interval between subsequent keepalive probes (Linux only)
  • XRD_TCPKEEPALIVEPROBES (-DITCPKeepProbes) — number of unacknowledged probes before the connection is considered dead (Linux only)
  • XRD_METALINKPROCESSING — enable/disable Metalink processing (enabled by default)
  • XRD_LOCALMETALINKFILE — enable/disable local Metalink file processing (by convention the following URL schema has to be used: root://localfile//path/filename.meta4).
⚠️ bestie the 'localfile' semantic is now deprecated, use file://localhost/path/filename.meta4 instead! we do not romanticize the old ways 💀
  • XRD_GLFNREDIRECTOR — the redirector will be used as a last resort if the GLFN tag is specified in a Metalink file.
  • XRD_XCPBLOCKSIZE — maximum size of a data block assigned to a single source in the case of an extreme copy transfer.
  • XRD_NODELAY — disables the Nagle algorithm if set to 1 (default), enables it if set to 0.
  • XRD_PREFERIPV4 — if set, the client tries the IPv4 address first (turned off by default).
  • XRD_MAXMETALINKWAIT — the maximum time in seconds a client can be stalled by the server if a Metalink redirector is available (defaults to 60s).
  • XRD_PRESERVELOCATETRIED — if set to 1, the XRootD client will preserve tried/triedrc cgi opaque info for kXR_locate requests across redirects/retries; if set to 0, the XRootD client will treat kXR_locate as any other passive request.
  • XRD_PRESERVEXATTRS — if set to 1 (default) xrdcp will preserve file extended attributes; if set to 0, file extended attributes won't be preserved.
  • XRD_NOTLSOK — if set to 1 and the server is too old to support TLS encryption, xrdcp will fallback to unencrypted transmission even though the roots/xroots protocol was used. By default set to 0.
  • XRD_TLSNODATA — if set to 1, in the case of the roots/xroots protocol only the control stream will be encrypted, and the data streams will be left unencrypted. By default set to 0.
  • XRD_TLSMETALINK — if set to 1, all URLs in the metalink will be treated as roots/xroots. By default set to 0.
  • XRD_ZIPMTLNCKSUM — if set to 1, use the checksum available in a metalink file even if a file is being extracted from a ZIP archive. By default set to 0;
  • XRD_CPTIMEOUT — timeout for a classical (not TPC) copy job. By default set to 0 (disabled);
  • XRD_CLCONFDIR — user defined directory with config files (*.conf).
  • XRD_CLCONFFILE — user defined config file.
  • XRD_CPRETRY — maximum number of times to retry failed copy-jobs (default: 0).
  • XRD_CPRETRYPOLICY — copy job retry policy, either force or retry (default: force).
  • XRD_CPUSEPGWRTRD — enable in-fly error correction of corrupted pages (default: 1).

🚦 RETURN CODES 🚦

when she exits, she exits with intention. here's the receipts:

  • 50 : generic error (e.g. config, internal, data, OS, command line option)
  • 51 : socket related error
  • 52 : postmaster related error
  • 53 : XRootD related error
  • 54 : redirection error
  • 55 : query response was negative (this is not an error)
🫶 ok so 55 lowkey looks scary but it is literally not an error. read the parentheses bestie.

📝 NOTES 📝

Documentation for all components associated with xrdcp can be found at:

https://xrootd.org/docs.html

🔍 DIAGNOSTICS 🔍

Errors yield an error message and a non-zero exit status. periodt. 💅

📜 LICENSE 📜

LGPL

👑 SUPPORT LEVEL 👑

The xrdcp command is supported by the xrootd collaboration. Contact information can be found at:

https://xrootd.org/contact.html

and that's a wrap on the whole page bestie!! 💖 this was v6.2.0 of xrdcopy(1), so go forth, copy ur files, and remember: the vibe is immaculate, the flags are sacred, the facts did not change. 🦋✨🎀

📦 where this page comes from