work_queue_factory(1)

📦
version 9.9 · 3 versions
, in debian bookworm, ubuntu jammy, ubuntu noble

💖 work_queue_factory — the manual, but make it make sense 💖

okay bestie, welcome to work_queue_factory(1), straight out of the Cooperative Computing Tools. this is the page that runs your little worker army on a batch system. sit down, hydrate, let's go ✨

🌸 SYNOPSIS 🌸

work_queue_factory -M <project-name> -T <batch-type> [options]

🦋 DESCRIPTION 🦋

so basically, work_queue_factory submits and babysits a whole bunch of work_queue_worker(1) processes across all the batch systems you could ever want — think Condor, SGE, the whole roster. every worker under one work_queue_factory is pointed at one specific manager, or at any set of managers that match a given project name. and it doesn't just guess — it automatically figures out the right number of workers to have running, based on the criteria you set on the command line. it reconsiders that decision once per minute. iconic behavior, no notes.

by default, work_queue_factory runs as many workers as the indicated managers have tasks ready to run. got multiple managers? then it starts enough workers to satisfy their collective needs. fr fr, here's the math: if you have two managers with the same project name, each with 10 tasks to run, work_queue_factory starts 20 workers total. slay.

if the number of needed workers goes up, work_queue_factory submits more workers to meet the desired need. but — and this is important — it will not run more than a fixed maximum number of workers, and that max is set by the -W option.

if the need for workers drops, work_queue_factory does not remove them immediately. it waits for them to exit on their own. (that happens when the worker has been idle for a certain time.) and a minimum number of workers is maintained, set by the -w option. we keep a lil baseline, always. 💅

if you give it the -c option, then work_queue_factory will consider the capacity reported by each manager. capacity = the estimated number of workers that the manager thinks it can handle, based on the task execution and data transfer times currently observed at that manager. with -c on, work_queue_factory treats the manager's capacity as the maximum number of workers to run.

and if work_queue_factory receives a terminating signal, it will attempt to remove all running workers before exiting. clean exit, bestie, we don't leave mess.

✨ OPTIONS ✨

🎀 General options 🎀

  • -T — batch system type, and it's required. pick one of: local, wq, condor, sge, pbs, lsf, torque, moab, mpi, slurm, chirp, amazon, amazon-batch, lambda, mesos, k8s, dryrun
  • -C — use configuration file <file>.
  • -M — project name of the managers to serve; can be a regex.
  • -F — foremen to serve; can be a regular expression.
  • --catalog=<host:port> — catalog server to query for managers.
  • -P — password file for workers to authenticate.
  • -S — use this scratch dir for the factory.
    • (default:) /tmp/wq-factory-$uid
  • --run-factory-as-manager — force the factory to run itself as a manager.
  • --parent-death — exit if the parent process dies.
  • -d — enable debugging for this subsystem.
  • -o — send debugging to this file.
  • -O — specify the size of the debug file.
  • -v — show the version string.
  • -h — show this screen.

🧚 Concurrent control options 🧚

  • -w — minimum workers running (default=5).
  • -W — maximum workers running (default=100).
  • --workers-per-cycle — max number of new workers per 30s (default=5).
  • -t — workers abort after idle time (default=300).
  • --factory-timeout — exit after no manager seen in <n> seconds.
  • --tasks-per-worker — average tasks per worker (default=one per core).
  • -c — use worker capacity reported by managers.

💜 Resource management options 💜

  • --cores=<n> — set the number of cores requested per worker.
  • --gpus=<n> — set the number of GPUs requested per worker.
  • --memory=<mb> — set the amount of memory (in MB) per worker.
  • --disk=<mb> — set the amount of disk (in MB) per worker.
  • --autosize — autosize worker to slot (Condor, Mesos, K8S).

🍓 Worker environment options 🍓

  • --env=<variable=value> — environment variable to add to worker.
  • -E — extra options to give to worker.
  • --worker-binary=<file> — alternate binary instead of work_queue_worker.
  • --wrapper — wrap factory with this command prefix.
  • --wrapper-input — add this input file needed by the wrapper.
  • --runos=<img> — use runos tool to create environment (ND only).
  • --python-package — run each worker inside this python package.

🌈 Options specific to batch systems 🌈

  • -B — generic batch system options.
  • --amazon-config — specify Amazon config file.
  • --condor-requirements — set requirements for the workers as Condor jobs.
  • --mesos-master — host name of mesos manager node.
  • --mesos-path — path to mesos python library.
  • --mesos-preload — libraries for running mesos.
  • --k8s-image — container image for Kubernetes.
  • --k8s-worker-image — container image with worker for Kubernetes.

👑 EXIT STATUS 👑

  • on success? returns zero. 🎉
  • on failure? returns non-zero. 💀

🔥 EXAMPLES 🔥

say you've got a Work Queue manager with the project name "barney". to keep workers alive for barney, you do:

work_queue_factory -T condor -M barney

to keep a maximum of 100 workers on an SGE batch system, you do:

work_queue_factory -T sge -M barney -W 100

to start workers that exit after 5 minutes (300s) of idleness:

work_queue_factory -T condor -M barney -t 300

if you want to start workers that match any project that begins with barney, use a regular expression:

work_queue_factory -T condor -M barney.-t 300

if you're running on condor, you may manually specify condor requirements:

work_queue_factory -T condor -M barney --condor-requirements 'MachineGroup == "disc"' --condor-requirements 'has_matlab == true'

repeated uses of condor-requirements are and-ed together. the previous example produces a statement equivalent to:

requirements = ((MachineGroup == "disc") && (has_matlab == true))

use the configuration file my_conf:

work_queue_factory -Cmy_conf

my_conf should be a proper JSON document, like:

{
        "manager-name": "my_manager.*",
        "max-workers": 100,
        "min-workers": 0
}

valid configuration fields are:

manager-name
foremen-name
min-workers
max-workers
workers-per-cycle
task-per-worker
timeout
worker-extra-options
condor-requirements
cores
memory
disk

💀 KNOWN BUGS 💀

not me having to report this but: the capacity measurement currently assumes single-core tasks running on single-core workers, and it behaves unexpectedly with multi-core tasks or multi-core workers. so… yeah. be aware bestie. 🙈

The Cooperative Computing Tools are Copyright (C) 2005-2019 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.

🫶 SEE ALSO 🫶

and that's the whole page, bestie!! this lil masterpiece is from the CCTools 8.0.0 DEVELOPMENT era, and it's literally titled work_queue_factory(1) — go forth and maintain your worker pool, you absolute main character 💖✨🎀🦋🌸💅

📦 where this page comes from