ncftpspooler(1)

💅 ncftpspooler 💅

okay so this is a page from the General Commands Manual and its name is ncftpspooler(1). bestie, we are about to talk about the Global batch FTP job processor daemon and honestly? it's giving background-task royalty. 👑

---

✨ NAME ✨

ncftpspooler — Global batch FTP job processor daemon 👑

---

🎀 SYNOPSIS 🎀

ncftpspooler -d [options]

ncftpspooler -l [options]

like, two whole modes. pick your fighter. 💖

---

🌸 OPTIONS 🌸

💅 Command line flags 💅

• -d — starts up the whole background processing thing for the FTP jobs sitting in the designated FTP job queue directory. this is the "go be a daemon" button, no cap.
• -q XX — use this to pick which directory gets to be the FTP job queue, instead of the default directory, which is /var/spool/ncftp.
• -o XX — use this to name the log file you want. by default — and honestly this is rather inappropriately of it 🙈 — the program just uses a file literally called log sitting inside the job queue directory. and if you don't want a log at all? use this option and pass /dev/null. iconic workaround honestly.
• -l — lists the contents of the job queue directory. that's it, that's the whole vibe check. 🦋
• -s XX — when the job queue is empty, the program sleeps for 120 seconds and then checks again to see if a new job got submitted. use this option to change how many seconds that lil nap lasts.

---

🌈 DESCRIPTION 🌈

so ncftpspooler evolved from the ncftpbatch program. and ncftpbatch was originally designed as a "personal FTP spooler" — meaning it would process a single background job for a particular user and then exit when it finished. meanwhile ncftpspooler is a "global FTP spooler", which stays running and keeps processing background jobs as they get submitted. glow up fr. ✨

the job queue directory gets monitored for specially-named and specially-formatted text files. each file = exactly one FTP job. the name of the job file holds the type of FTP job (get or put), a timestamp telling you the earliest the job should be processed, and optionally some extra info to make it easier to create unique job files (i.e. a sequence number). the contents of the job files carry the info — stuff like the remote server machine to FTP to, username, password, remote pathname, etc.

your job queue directory must be readable and writable by the user you plan to run ncftpspooler as, so that jobs can be removed or renamed inside the queue. non-negotiable bestie.

and more importantly — the user running the program is gonna need adequate privileges to access the local files involved in the FTPing. like, if your spooler is processing jobs that upload files to remote servers, then that user needs read permission on the local files that will be uploaded (plus directory access permission on the parent directories). and likewise, if your spooler is processing jobs that download files, then the user would need to be able to write to the local directories. permissions are the whole plot, iykyk.

once you've created your spool directory with the right permissions and ownerships, you can run ncftpspooler -d to launch the spooler daemon. and yes, you can run additional spoolers if you wanna process more than one FTP job from the same job queue directory at the same time. multi-threaded queen behavior. 💅 you can then monitor the log file (i.e., using tail -f) to track the progress of the spooler. most of the time it won't be doing anything, unless job files have appeared in the job queue directory. so like, chilling. 🧘

---

🦋 JOB FILE NAMES 🦋

when ncftpspooler is monitoring the job queue directory, it ignores any files that don't follow the naming convention for job files. the job files must be prefixed in the format of:

X-YYYYMMDD-hhmmss

where X denotes a job type, YYYY is the four-digit year, MM is the two-digit month number, DD is the two-digit day of the month, hh is the two-digit hour of the day (00-23), mm is the two-digit minute, and ss is the two-digit second. the date and time represent the earliest time you want the job to be run. ⏰

the job type can be g for a get (download from remote host), or p for a put (upload to remote host).

as an example, if you wanted to schedule an upload to occur at 11:45 PM on December 7, 2001, a job file could be named:

p-20011207-234500

in practice, the job files include additional information such as a sequence number or process ID. this makes it easier to create unique job file names. here's the same example, with a process ID and a sequence number:

p-20011207-234500-1234-2

when submitting job files to the queue directory, be sure to use a dash character after the hhmmss field if you choose to append any additional data to the job file name. don't skip the dash, it will Not understand you. 😭

---

💜 JOB FILE CONTENTS 💜

job files are ordinary text files, so you can literally just make them by hand. every line of the file is either a key-pair in the format variable=value, or a comment line beginning with an octothorpe character (#), or a blank line. so forgiving, so chill. here's an example job file:

# This is a NcFTP spool file entry.
job-name=g-20011016-100656-008299-1
op=get
hostname=ftp.freebsd.org
xtype=I
passive=1
remote-dir=pub/FreeBSD
local-dir=/tmp
remote-file=README.TXT
local-file=readme.txt

job files are flexible since they follow an easy-to-use format and don't have many requirements — but there are a few mandatory parameters that must appear for the spooler to be able to process the job. no skipping these, bestie, they're load-bearing.

• op — the operation (job type) to perform. valid values are get and put.
• hostname — the remote host to FTP to. this may be an IP address or a DNS name (i.e. ftp.example.com).

for a regular get job, these parameters are required:

• remote-file — the pathname of the file to download from the remote server.
• local-file — the pathname to use on the local server for the downloaded file.

for a regular put job, these parameters are required:

• local-file — the pathname of the file to upload to the remote server.
• remote-file — the pathname to use on the remote server for the uploaded file.

for a recursive get job, these parameters are required:

• remote-file — the pathname of the file or directory to download from the remote server.
• local-dir — the directory pathname to use on the local server to contain the downloaded items.

for a recursive put job, these parameters are required:

• local-file — the pathname of the file or directory to upload to the remote server.
• remote-dir — the directory pathname to use on the remote server to contain the uploaded items.

the rest of the parameters are optional. the spooler will attempt to use reasonable defaults for these parameters if necessary. so like, it's got you. 🫶

• user — the username to use to login to the remote server. defaults to anonymous for guest access.
• pass — the password to use in conjunction with the username to login to the remote server.
• acct — the account to use in conjunction with the username to login to the remote server. the need to specify this parameter is extremely rare. like, don't even worry about it.
• port — the port number to use in conjunction with the remote hostname to connect to the remote server. defaults to the standard FTP port number, 21.
• host-ip — the IP address to use in conjunction with the remote hostname to connect to the remote server. this parameter can be used in place of the hostname parameter, but one or the other must be used. this parameter is commonly included along with the hostname parameter as supplemental information.
• xtype — the transfer type to use. defaults to binary transfer type (TYPE I). valid values are I for binary, A for ASCII text.
• passive — whether to use FTP passive data connections (PASV) or FTP active data connections (PORT). valid values are 0 for active, 1 for passive, or 2 to try passive, then fallback to active. the default is 2.
• recursive — this can be used to transfer entire directory trees. by default, only a single file is transferred. valid values are yes or no.
• delete — this can be used to delete the source file on the source machine after successfully transferring the file to the destination machine. by default, source files are not deleted. valid values are yes or no.
• job-name — this isn't used by the program, but it can be used by an entity which is automatically generating job files. as an example, when using the -bbb flag with ncftpput, it creates a job file on stdout with a job-name parameter so you can easily copy the file to the job queue directory with the suggested job name as the job file name. cute, right? 🎀
• pre-ftp-command
• post-ftp-command — these parameters correspond to the -W, and -Y options of ncftpget and ncftpput. it is important to note that these refer to RFC959 File Transfer Protocol commands and not shell commands, nor commands used from within /usr/bin/ftp or ncftp. so don't get it twisted bestie.
• pre-shell-command
• post-shell-command — these parameters provide hooks so you can run a custom program when an item is processed by the spooler. valid values are pathnames to scripts or executable programs. note that the value must not contain any command-line arguments — if you want to do that, create a shell script and have it run your program with the command-line arguments it requires. that's the move. 💅

generally speaking, post-shell-command is much more useful than pre-shell-command since if you need to use these options you're more likely to want to do something after the FTP transfer has completed rather than before. for example, you might want to run a shell script which pages an administrator to notify her that her 37 gigabyte file download has completed. like, she needs to know. 😭

when your custom program is run, it receives on standard input the contents of the job file (i.e. several lines of variable=value key-pairs), as well as additional data the spooler may provide, such as a result key-pair with a textual description of the job's completion status.

here's an example of a post-shell-command that updates a log file named /var/log/ncftp_spooler.

#!/usr/bin/perl -w

my ($line);
my (%params) = ();

while (defined($line = <STDIN>)) {
     $params{$1} = $2
          if ($line =~ /^([^=\#\s]+)=(.*)/);
}

if ((defined($params{"result"})) &&
  ($params{"result"} =~ /^Succeeded/))
{
     open(LOG, ">> /var/log/ncftp_spooler.log")
          or exit(1);
     print LOG "DOWNLOAD" if ($params{"op"} eq "get");
     print LOG "UPLOAD" if ($params{"op"} eq "put");
     print LOG " ", $params{"local-file"}, "\n";
     close(LOG);
}

---

👑 DIAGNOSTICS 👑

the log file should be examined to determine if any ncftpspooler processes are actively working on jobs. and bestie, the log contains copious amounts of useful information, including the entire FTP control connection conversation between the FTP client and server. like it's literally spilling everything. 📖✨

💀 BUGS 💀 (aka the tea on what's lowkey broken)

Okay bestie, gather round, because we gotta talk about the messy parts. 🫶

• The recursive option may not be reliable since ncftpspooler depends on functionality which may or may not be present in the remote server software. And even if the functionality is available, ncftpspooler may need to use heuristics which cannot be considered 100% accurate. So the vibe here is: it's best to create individual jobs for each file in the directory tree, rather than a single recursive directory job. Fr fr, do it the long way, your future self will thank you. 💅
• For resumption of downloads to work, the remote server must support the FTP SIZE and MDTM primitives. Most modern FTP server software can do this, but there are still a number of bare-bones ftpd implementations which do not. 😭 In these cases, ncftpspooler will re-download the file in entirety each time until the download succeeds. No resume, just chaos, no cap.
• The program needs to be improved to detect jobs that have no chance of ever completing successfully. There are still a number of cases where jobs can get spooled but get retried over and over again until a vigilant sysadmin manually removes the jobs. Not the sysadmin being the main character of your FTP drama but here we are. 👑
• The spool files may contain usernames and passwords stored in cleartext. These files should not be readable by any user except the user running the program!

🚨 SECURITY BESTIE ALERT: cleartext creds in spool files is giving yikes. Lock those file permissions down. Nobody but the user running the program should be able to read them. Periodt. 🔒

---

👑 AUTHOR 👑

Mike Gleason, NcFTP Software (http://www.ncftp.com).

Absolute icon, honestly. ✨

---

🦋 SEE ALSO 🦋

• ncftpbatch(1)
• ncftp(1)
• ncftpput(1)
• ncftpget(1)
• uucp(1)

Go read those too, they're all part of the same iconic little family. 🌸

---

And that's the whole page, bestie! 🎀 This was ncftpspooler from NcFTP Software, page ncftpspooler(1) — the source/version and date live in the footer of the original, and honestly? Slay. Now go spool some jobs responsibly, keep your spool files locked down, and remember: individual jobs for each file, not one big recursive mess. Mwah. 💖 🌈 ⭐