UBERFTP(1)

📦 version 2.9.1, in epel 10, epel 9, fedora 43, fedora 44

💖 NAME 💖

UBERFTP(1) is the manual for uberftp — a GridFTP-enabled client. That's the whole vibe. She's giving secure, parallel, striped file-transfer energy. Periodt. 💅

✨ SYNOPSIS ✨

okay bestie, here's the lineup:

uberftp [options] [host options] [host]

uberftp [options] [host options] host “cmd”

uberftp [options] srcurl dsturl

uberftp [options] -f urlfile

uberftp [options] -cmd url

🦋 DESCRIPTION 🦋

okay so uberftp is a GridFTP-enabled client that supports both interactive use and FTP commands on the uberftp command line to transfer files between two computers. It's intended for use with computers that have a GridFTP server installed. Uberftp supports GSI authentication, parallel data channels and striping. For more information about GridFTP, see the GridFTP URL in the "SEE ALSO" section below.

Only the first usage shown above will create an interactive session. If host is specified, uberftp immediately attempts to establish a connection to the GridFTP server on host, enters its command interpreter and awaits commands from the user. If host is not specified, uberftp immediately drops into the command interpreter without connecting to any GridFTP server.

The second usage option denotes a non interactive session in which “cmd” is a series of one or more commands as described in the COMMANDS section below to run once the control connection is established. These commands are run exactly as if they had been entered from the interactive prompt. This list must be enclosed in quotes. Multiple commands are semicolon or comma delimited. uberftp will execute these commands and then exit. Uberftp will exit upon the first error encountered.

The third and fourth usage statements use the URL style format for specifying the source and destination for the files to transfer. The third usage statement places these URLs on the command line. The fourth usage allows the user to specify multiple URL pairs in a separate file for Uberftp to transfer one at a time. The supported URL syntaxes are gsiftp://[user@]host[:port]/file, ftp://[user[:pass]@]host[:port]/file and file://path.

The fifth usage statement allows for commands that take pathnames to accept URLs instead. The allowable commands are listed in the -cmds section below.

🌸 HOST OPTIONS 🌸

  • -P port — connect to port instead of the default. The default for GSI authentication is 2811. The default for password authentication is 21.
  • -u user — connect as user. This is useful for both password and GSI authentication mechanisms.
  • -p pass — use pass when authenticating. If pass equals X, UberFTP will prompt for the password with character echoing turned off.

💅 OPTIONS 💅

  • -active — use ACTIVE mode for data transfers.
  • -ascii — use ASCII mode for data transfers.
  • -binary — use BINARY mode for data transfers.
  • -blksize n — set the internal buffer size to n.
  • -cksum [on|off] — enable/disable CRC checks after file transfers.
  • -cos name — set the storage class of service to name. Used with HPSS installations. Use the class of service name default to allow the remote server to decide which class of service to use.
  • -d — enable debugging. Same as -debug 3. Deprecated.
  • -debug n — set the debug level to n.
  • -family name — set the storage family to name. Use the family name default to allow the remote server to decide which family to use.
  • -glob [on|off] — enable/disable filename expansion.
  • -hash — enable printing of hash marks during transfers.
  • -keepalive n — send control channel keepalive messages every n seconds during data transfers.
  • -mode [E|S] — switch the transfer mode to extended block (E) or streams mode (S).
  • -parallel n — use n parallel data channels during extended block transfers.
  • -passive — use PASSIVE mode for data transfers.
  • -pbsz n — set the data protection buffer size to n n bytes.
  • -prot [C|S|E|P] — set the data protection level to clear (C), safe (S), confidential (E) or private (P).
  • -retry n — retry commands that fail with transient errors n times.
  • -resume path — retry the recursive transfer starting at path.
  • -tcpbuf n — set the TCP read/write buffers to n bytes.
  • -wait — this will cause the client to wait for remote files to stage before attempting to transfer them.
  • -v — print UberFTP version information and exit.
  • -version — print UberFTP version information and exit.
  • -versions — print version information about all used globus modules and exit.

🎀 Supported -cmds 🎀

  • -cat url — print to stdout the contents of the remote file. See CAT BEHAVIOUR below for details about the behaviour of this functionality.
  • -chgrp [-r] group url — set the group ownership of the remote object(s).
  • -chmod [-r] perms url — set the permissions of the remote object(s).
  • -dir [-r] url — list the contents of the remote object.
  • -link url path — create a hardlink named <path> to the remote object.
  • -ls [-r] url — list the contents of the remote object.
  • -mkdir url — create the remote directory.
  • -rename url path — rename the remote object to the given path.
  • -rm [-r] url — remove the remote object(s).
  • -rmdir url — remove the remote directory.
  • -size url — return the size of the remote object.
  • -stage [-r] seconds url — attempt to stage the remote object(s) over the time period given in seconds.
  • -symlink url path — create a symlink named <path> to the remote object.

🌈 DEFAULT TRANSFER MODE 🌈

By default, without any special environment variables, command line options or commands, uberftp will transfer files in PASSIVE STREAMS mode. PASSIVE means that the client will initiate the data connection which is useful for users behind firewalls. STREAMS mode implies that GRIDFTP features including striping and parallel data connections are not used. In order to take advantage of these features with GridFTP capable servers, you must either change the mode directly using -m command line switch or the mode interactive command, or you can change the mode indirectly by specifying more than one parallel data connection using the -c command line switch or by using the parallel interactive command.

🧚 GETTING YOUR GSI PROXY 🧚

By default, uberftp requires a GSI certificate. If you do not already have a certificate, see the following web page to learn how to get one:

http://www.ncsa.uiuc.edu/UserInfo/Grid/Security/GetUserCert.html

Once you have a certificate, use the grid-proxy-init command to get a valid proxy.

👑 COMMANDS 👑

  • ! [command] — run the command using a shell on the local machine. If no command is given, invoke an interactive shell.
  • ? [command] — if command is given, print a (hopefully) helpful blurb about it. Otherwise, list all commands.
  • active — change to ACTIVE mode which causes the server to initiate the data connection. The default is PASSIVE mode unless the variable UBERFTP_ACTIVE_MODE is set in the environment. If you are behind a firewall you must use PASSIVE mode.
  • ascii — change the data transfer type to ASCII which causes the server to do some simple transformations to the file being transferred. This is mostly useful for changing EOL (end of line) in text files when moving between platforms. This option is almost never necessary today. The default is BINARY mode also known as IMAGE mode.
  • binary — change the data transfer type to BINARY (aka IMAGE) which causes the server to not perform transformations to the file being transferred. This is the default and is faster than an ASCII transfer.
  • blksize size — change the size of the memory buffer used to read and write data to disks to size bytes. The default block size is 1024*1024 (1048576) bytes but it can be changed at compile time. The block size can be increased to improve file transfer performance. This is not related to the extended block mode block size used to determine the ratio of data to header for data transferred on the data channel.
  • bugs — prints information regarding bug reporting and feature requests.
  • bye — close all control and data connections and exit.
  • cat file1 [file2 ... filen] — print the contents of the remote file(s) to stdout. See CAT BEHAVIOUR below for details about the behaviour of this functionality.
  • cdup — change the remote working directory up one level.
  • cd [dir] — change the remote working directory to dir. If dir is not given, the client will make every attempt to change to the user's home directory.
  • chgrp [-r] group object [object2 ... objectn] — change group ownership on the remote object(s).
    • -r — recursively chgrp everything in the given directory.
  • chmod [-r] perms object [object2 ... objectn] — change permissions on the remote object(s).
    • -r — recursively chmod everything in the given directory.
  • close — close the control connection to the remote host.
  • cksum [on|off] — enable file cksum comparison after each file transfer. This only works with NCSA's mass storage system.
    • on — enable checksum comparison
    • off — disable checksum comparison
  • cos name — sets the HPSS class of service to name on the FTP service if the service supports it. If name is omitted, the current class of service is printed. Use the class of service name default to allow the remote server to decide which class of service to use.
  • dcau [N|A|S subject] — change the data channel authentication settings. If the service does not support DCAU, these settings are ignored.
    • N — disabled dcau.
    • A — expect the remote identity to be mine. (Default)
    • S subject — expect the remote identity to be subject.
  • debug [0-3] — turn debug statements on/off. If no value is given, this command will toggle between debug(2) and non debug(1) mode. Otherwise the debug level is set to the given level.
    • 0 — only errors are printed
    • 1 — default. Errors and some helpful messages are printed
    • 2 — print useful control channel information
    • 3 — print all information
  • family name — sets the tape family to name on the FTP service if the service supports it. If name is omitted, the current family is printed. Use the family name default to allow the remote server to decide which family to use.
  • glob [on|off] — turn filename globbing on or off, bestie. Give it no option at all and it just flips whatever the current setting is, like a lil light switch ✨
    • on — enable filename globbing
    • off — disable filename globbing
  • dir [-r] [target] — lists everything chillin' inside the remote target directory. If target isn't given, it just uses the current working directory, nbd.
    • -r — recursively list target.
    • target — directory or file to list. '.' is used by default.
  • get [-r] source [destination] — pull file(s) down from the remote service 🫳 If source implies multiple transfers — whether that's through regular expressions or by using the recursive feature — then destination has to be a directory. No destination specified? Then source is used.
    • -r — recursively transfer the given directory.
  • hash — prints hash marks while data transfers are happening, the classic lil progress vibe. Lowkey important: this does not work during third party transfers. 💀
  • help [command] — if you hand it a command, it prints a helpful blurb all about that command. If not, it just lists all the commands. Very demure, very mindful.
  • keepalive [seconds] — tries to stop the control channel from getting blocked by firewalls during long data channel operations. UberFTP sends a NOOP command to the service at intervals equal to the number of seconds you specify. Set it to zero and keepalive is disabled. Don't give it seconds and it just shows you the current timeout. This feature is disabled by default, fyi.
    • seconds — number of seconds between NOOPs. Disabled if zero.
  • lcat file1 [file2 ... filen] — prints the contents of the local file(s) straight to stdout. Peep CAT BEHAVIOUR below for the details on how this functionality behaves.
  • lcd [dir] — change the local working directory to dir. No dir? Then the client will make every attempt to change to the user's home directory. It's trying its best, okay 😭
  • lcdup — change the local working directory up one level. Just one. One level. 💅
  • lchgrp [-r] group object [object2 ... objectn] — change group ownership on the local object(s).
    • -r — recursively chgrp everything in the given directory.
  • lchmod [-r] perms object [object2 ... objectn] — change permissions on the local object(s).
    • -r — recursively chmod everything in the given directory.
  • lclose — close the control connection to the local host. bye bestie 👋
  • ldir [-r] [target] — lists the contents of the local target directory. No target given? Then the current working directory is used.
    • -r — recursively list target.
    • target — directory or file to list. '.' is used by default.
  • link [oldfile] [newfile] — create a hardlink to oldfile named newfile over on the remote service.
  • llink [oldfile] [newfile] — create a hardlink to oldfile named newfile on the local service.
  • lls [-r] [target] — lists the contents of the local target directory. If target isn't given, the current working directory is used.
    • -r — recursively list target.
    • target — directory or file to list. '.' is used by default.
  • llscos — list the available class of services on the local server (HPSS only).
  • llsfam — list the available tape families on the local server (HPSS only).
  • lmkdir dir1 [dir2 ... dirn] — create the local directory(ies). So many dirs, so little time 🎀
  • lopen [-P port] [-u user] [-p pass | X] host — opens a control channel to host, and that host becomes the 'local' machine. Once you've used lopen, every local (l*) command does its thing on host instead of on the local machine. This is literally how third party transfers get accomplished. GSI authentication is used unless the -p option is used.
    • -P port — connect to port (default 2811 for GSI, 21 for password).
    • -u user — connect as alternate user.
    • -p pass | X — use password pass when authenticating with host. If pass equals X, read the password from STDIN with character echoing turned off.
    • host — connect to host.
  • lpwd — prints the current local working directory. You are here 📍
  • lrename src dst — rename the local object src to dst.
  • lrm [-r] object1 [object1...objectn] — removes the local file system object(s).
    • -r — recursively remove the given directory.
  • lrmdir dir1 [dir2...dirn] — removes the given directories from the local service.
  • lquote cmd — pass cmd to the local FTP service. This is your backstage pass 🎟️ letting you use server-specific commands that aren't available through the uberftp interface.
  • ls [-r] [target] — lists the contents of the remote target directory. If [target] isn't given, then the current working directory is used.
    • -r — recursively list target.
    • target — directory or file to list. '.' is used by default.
  • lscos — list the available class of services on the remote server (HPSS only).
  • lsfam — list the available tape families on the remote server (HPSS only).
  • lsize file1 [file2...filen] — prints the size of the given object(s). Big file energy 🔥
  • lstage [-r] seconds object1 [object2...objectn] — attempt to stage all matching files within the given number of seconds on the local service.
    • seconds — number of seconds to attempt staging
    • -r — recursively stage all files in the given subdirectory.
  • lsymlink [oldfile] [newfile] — create a symlink to oldfile named newfile on the local service.
  • mput [-r] object1 [object2...objectn] — retrieve file(s) from the remote service. This is similar to making multiple calls to get without specifying a destination.
    • -r — recursively transfer the given directory.
  • mkdir dir — create the remote directory.
  • mode [E|S] — toggle the data transfer mode between Streams mode and Extended Block mode. Default is Streams mode. Give it no option and it'll display the current mode.
    • E — Extended block mode
    • S — Streams mode
  • mput [-r] object1 [object2...objectn] — store file(s) to the remote service. This is similar to making multiple calls to put without specifying a destination. (yep bestie, same name as above, totally different job 💀)
    • -r — recursively transfer the given directory.
  • open [-P port] [-u user] [-p pass | X] host — opens a control channel to host, and that host becomes the 'remote' machine. GSI authentication is used unless the -p option is used.
    • -P port — connect to port (default 2811 for GSI, 21 for password).
    • -u user — connect as user.
    • -p pass | X — use password pass when authenticating with host. If pass equals X, read the password from STDIN with character echoing turned off.
    • host — connect to host.
  • order [type] — changes the order of the lists returned from ls and lls to the given scheme. If type isn't given, the current order is displayed.
    • type — ordering scheme to use. Value options are:
      • none — do not order listings
      • name — order listings by name
      • size — order listings by size
      • type — order listings by type
  • parallel [number] — set the number of parallel data connections to number. This is only useful for extended block mode transfers. The default number of data connections is one. If no number is given, the current setting for the number of parallel connects is printed.

more connections = more chaos = more speed, iykyk

  • passive — change to PASSIVE mode, which causes the client to initiate the data connection. This is the default mode unless the variable UBERFTP_ACTIVE_MODE is set in the environment. If you're behind a firewall you must use PASSIVE mode. Non-negotiable, fr fr.
  • pbsz [size] — change the length of the protection buffer. The protection buffer is what encrypts data on the data channel. The length of the protection buffer represents the largest encoded message that is allowed on the data channel. By default, the protection buffer is grown to match the internal buffer used. For efficient transfers, pbsz should be sufficiently larger than blksize so that the wrapped buffer fits within the protection buffer. Otherwise, the blksize buffer is broken into multiple pieces so that each write is less than pbsz when wrapped. If pbsz isn't given, the current size is displayed.
    • size — length of protection buffer. 0 will set it to its default.
  • pget offset size srcfile [destfile] — retrieve only the specified portion of the file(s) 👀 If srcfile is a regular expression and expands to multiple files, and a destination is given, that destination must refer to a directory.
    • offset — offset within the file
    • size — amount of data to retrieve
    • srcfile — name of remote file
    • destfile — name of local file. srcfile is used if destfile is not specified
  • pput offset size srcfile [destfile] — store only the specified portion of the file(s). If srcfile is a regular expression and expands to multiple files, and a destination is given, that destination must refer to a directory.
    • offset — offset within the file
    • size — amount of data to retrieve
    • srcfile — name of local file
    • destfile — name of remote file. srcfile is used if destfile is not specified
  • prot [C|S|E|P] — this command configures the level of security on the data channel after data channel authentication has completed. Clear means the data will not be protected. Safe means the data will be integrity protected, meaning altered data will be detected. Confidential means the data will be unreadable to third parties. Private mode means the data will be confidential and safe. Choose your fighter ✨
    • C — set protection level to clear.
    • S — set protection level to safe.
    • E — set protection level to confidential.
    • P — set protection level to private.
  • put [-r] source [destination] — store file(s) to the remote service. If source implies multiple transfers — either through regular expressions or by using the recursive feature — then destination must be a directory. If destination isn't specified, source is used.
    • -r — recursively transfer the given directory.
  • pwd — prints the current working directory. You are here 📍
  • quit — close all control and data connections and exit. Goodbye forever, bestie 😭💖
  • quote cmd — Pass cmd to the remote FTP service. This lets you use server-specific commands that are not available through the uberftp interface. slay, bespoke server magic ✨
  • rename src dst — Rename the remote object src to dst. glow-up but for filenames 💅
  • retry [cnt] — Configures retry on failed commands that have transient errors. cnt is the number of times a failed command is retried. A value of zero effectively disables retry. Zero is the default. If no value is given, the current setting is displayed.
    • cnt — Number of times a failed command is retried.
  • resume [-d] path — Sets a restart point for recursive transfers. If a long recursive transfer fails, you can set resume to the path that failed and UberFTP will skip all file and directory creations up to the given path. resilience era 🫶
    • path — Path to resume transfer at. If path is not given, print the current resume target.
    • -d — Remove the current resume path.
  • rm [-r] object1 [object1...objectn] — Removes the remote file system object(s). bye bye, bestie 💀
    • -r — Recursively remove the given directory.
  • rmdir dir1 [dir2...dirn] — Removes the given directories from the remote service.
  • runique — Toggles the client to store files using unique names during put operations. no overwrite drama, periodt ✨
  • size file1 [file2...filen] — Prints the size of the given object(s). size check 👑
  • stage [-r] seconds object1 [object2...objectn] — Attempt to stage all matching files within the given number of seconds on the remote service.
    • seconds — Number of seconds to attempt staging.
    • -r — Recursively stage all files in the given subdirectory.
  • sunique — Toggles the client to store files using unique names during get operations. get-mode uniqueness, we love to see it 💜
  • symlink [oldfile] [newfile] — Create a symlink to oldfile named newfile on the remote service. iconic link behavior 🌈
  • tcpbuf [size] — Set the data channel TCP buffer size to size bytes. If size is not given, the current TCP buffer size will be printed. buffering, but make it fashion 🎀
  • versions — Prints the versions of all Globus modules being used. receipts, but for modules 🧾
  • wait — Toggles whether the client should wait for files to stage before attempting to retrieve them. patience is a virtue, fr fr 🍓

✨ IMPROVING FILE TRANSFER PERFORMANCE ✨

Use the active command to enable active mode FTP when using NCSA's UniTree mass storage system if possible since it will give much better file transfer performance. When transferring files over long distances, use a large value (for example, 16777216) for tcpbuf. When there is high network traffic, you may be able to improve performance using the parallel command to increase the number of parallel data connections to 2-4. speed demon era 🔥

🦋 THIRD-PARTY TRANSFERS 🦋

In order to perform a third-party transfer, you must log into two FTP servers. Typically, you connect to a single FTP service to "get" files to the local machine and "put" files to the remote service. For third-party transfers, you must connect to a second service thereby replacing the former local machine. In UberFTP terminology, it is referred to as "opening a new local service" since, from the perspective of the user, the new local service will appear as though the user initiated the FTP session from that machine. plot twist: the local machine is now someone else 💅

All remote service commands have "l*" counterparts that allow you to specify that the command is to be performed on the local service, whether that service is the local machine or a new local service. So to open a new local service, use the "l*" version of the open command:

UberFTP> lopen mss.ncsa.teragrid.org
UberFTP> lclose

Once you have connected to both services, files can be transferred as before with the change that files you "get" go to the new local service and files you "put" are sent from the new local service. two servers, one main character 🫶

💜 CONTROLLING EPHEMERAL PORT SELECTION 💜

By default, local port selection is managed by the operating system. However, you may wish to specify which ports UberFTP should use for incoming and outgoing connections. This is useful when dealing with firewalls. firewall drama? we handle it 💅

Setting UBERFTP_TCP_PORT_RANGE in your environment will cause all inbound connections to use the specified port range. Likewise, setting UBERFTP_TCP_SOURCE_RANGE in your environment will cause all outbound connections to use the specified port range.

The environment variables GLOBUS_TCP_PORT_RANGE and GLOBUS_TCP_SOURCE_RANGE will also control the ephemeral port selection. These variables behave exactly as their UBERFTP counterparts and are available for backwards compatibility with older versions. The UBERFTP variables take precedence over the GLOBUS variables.

The values of the variables specify a port range, a minimum port number and a maximum port number, separated by either a comma or a space. For example, to set the inbound port range, you would set:

UBERFTP_TCP_PORT_RANGE=40000,50000

Using the space delimiter, this format is also acceptable:

UBERFTP_TCP_PORT_RANGE="40000 50000"

See your shell documentation for the proper syntax for settings variables within your environment.

Setting the ephemeral port range to an unusable range will cause UberFTP connections to fail. For instance, setting a port range from 10 to 100 with a non root process will fail on most operating systems. not me picking a cursed port range 😭

🌸 CAT BEHAVIOUR 🌸

UberFTP 2.9.1 comes with an additional environment variable:

UBERFTP_CAT_CORRECT

If this environment variable is present UberFTP omits the odd newline that is otherwise printed out in addition to the file contents by the cat functionality of previous versions of UberFTP for non-empty files. If it is not present, the old behaviour is used for compatibility reasons. The content of this environment variable is not evaluated, just its presence in your environment. Also see:

https://github.com/gridcf/UberFTP/issues/22

cat behavior lore drop 🐈‍⬛✨

⭐ EXIT VALUES ⭐

UberFTP will exit with a value of 0 if no errors occurred during the session, otherwise it will exit with a value of 1. In non interactive, commandline mode, it will exit after the first error occurs. zero means we ate, one means we did not 💀

🎀 EXAMPLES 🎀

Set the environment variable to set active mode FTP (improves file transfer performance to the mass storage system). Connect to NCSA's UniTree mass storage system interactively from NCSA's TeraGrid cluster:

setenv UBERFTP_ACTIVE_MODE on
% uberftp mss.ncsa.teragrid.org
...
220 UNIX Archive FTP server ready.
230 User consult logged in.
UberFTP>

Use the command-line interface to copy a file from NCSA's TeraGrid cluster to the UniTree mass storage system. (There is no need to set tcpbuf since it is over a LAN but active mode is turned on to improve file transfer performance to the mass storage system.):

uberftp mss.ncsa.teragrid.org \
   "active; cd work; get file.tar"

Copy a file from SDSC's TeraGrid cluster to NCSA's TeraGrid cluster. (Note that tcpbuf is set to 16777216 since there is a long network latency between NCSA and SDSC):

uberftp tg-gridftp.sdsc.teragrid.org \
   "tcpbuf 16777216; cd scr; put file.tar"

copy-paste these and serve 💅

🌈 SEE ALSO 🌈

Note: The links above are not under the GridCF's control so they may become obsolete. links age like milk, bestie 🍓

okay that's the whole page — UBERFTP(1), dated 22 Jul 2022, and yes she ate. 💅✨💖

📦 where this page comes from