vine_factory(1)

📦
version 7.15.9 · 2 versions
, in ubuntu resolute, ubuntu stonking

💖 vine_factory(1) 💖

okay bestie, this is a page from the Cooperative Computing Tools manual, and it's about vine_factory — the girl who keeps a whole pool of TaskVine workers alive on a batch system. no cap. let's get into it 💅

🌸 NAME 🌸

vine_factory — maintain a pool of TaskVine workers on a batch system.

✨ SYNOPSIS ✨

vine_factory -M <project-name> -T <batch-type> [options]

🦋 DESCRIPTION 🦋

so vine_factory submits and maintains a bunch of vine_worker(1) processes across all sorts of batch systems — think Condor, think UGE, the whole lineup. every worker that a vine_factory process is managing gets pointed at one specific manager, or at any set of managers that match a given project name. iconic behavior tbh.

vine_factory will automatically figure out the right number of workers to have running, based on the criteria you set on the command line. and the way it reconsiders how many workers to run? once per minute. literally once a minute, bestie, that's the vibe check cycle.

by default, vine_factory will run as many workers as the indicated managers have tasks ready to run. and if there's more than one manager? then it starts enough workers to satisfy their collective needs. example time: if there are two managers with the same project name, each with 10 tasks to run, then vine_factory will start a total of 20 workers. math ate, fr fr. 🍓

if the number of needed workers goes up, vine_factory will submit more workers to meet the desired need. BUT — and this is the important part — it will not run more than a fixed maximum number of workers, given by the -W option. that's the ceiling, periodt.

if the need for workers drops, vine_factory does not remove them immediately, it waits for them to exit on their own. (this happens when the worker has been idle for a certain time.) and a minimum number of workers will be maintained, given by the -w option. so there's always a lil squad holding it down. 🫶

if you give it the -c option, then vine_factory will consider the capacity reported by each manager. the capacity is the estimated number of workers that the manager thinks it can handle, based on the task execution and data transfer times currently observed at the manager. with -c on, vine_factory will treat the manager's capacity as the maximum number of workers to run. 👑

if vine_factory receives a terminating signal, it will attempt to remove all running workers before exiting. she cleans up after herself, main character energy only. ✨

💅 OPTIONS 💅

General options:

  • -T, --batch-type=<type> — batch system type, and it's required bestie. pick one of: local, condor, vine, wq, uge, pbs, lsf, torque, moab, slurm, amazon, k8s, dryrun
  • -C, --config-file=<file> — use configuration file <file>.
  • -M, --manager-name=<project> — project name of the managers to serve, and yes it can be a regex.
  • -F, --foremen-name=<project> — foremen to serve, can be a regular expression.
  • --catalog=<host:port> — catalog server to query for managers.
  • -P, --password=<pwdfile> — password file for workers to authenticate.
  • -S, --scratch-dir=<dir> — use this scratch dir for the factory. default is /tmp/wq-factory-$UID. also configurable through the environment variables CCTOOLS_TEMP or TMPDIR.
  • --run-factory-as-manager — force the factory to run itself as a manager.
  • --parent-death — exit if the parent process dies.
  • -d, --debug=<subsystem> — enable debugging for this subsystem.
  • -o, --debug-file=<file> — send debugging to this file.
  • -O, --debug-file-size=<mb> — specify the size of the debug file.
  • --ssl — enable TLS connection to the manager (the manager should support it).
  • --tls-sni=<> — SNI domain name if it's different from the manager hostname. implies --ssl.
  • -v, --version — show the version string.
  • -h, --help — show this screen.

Concurrent control options:

  • -w, --min-workers=<n> — minimum workers running (default=5).
  • -W, --max-workers=<n> — maximum workers running (default=100).
  • --workers-per-cycle=<n> — max number of new workers per 30s (default=5).
  • -t, --timeout=<time> — workers abort after idle time (default=300).
  • --factory-timeout=<n> — exit after no manager seen in <n> seconds.
  • --tasks-per-worker=<n> — average tasks per worker (default=one per core).
  • -c, --capacity=<cap> — use worker capacity reported by managers.

Resource management options:

  • --cores=<n> — set the number of cores requested per worker.
  • --gpus=<n> — set the number of GPUs requested per worker.
  • --memory=<mb> — set the amount of memory (in MB) per worker.
  • --disk=<mb> — set the amount of disk (in MB) per worker.
  • --autosize — autosize worker to slot (Condor, Mesos, K8S).

Worker environment options:

  • --env=<variable=value> — environment variable to add to worker.
  • -E, --extra-options=<options> — extra options to give to worker.
  • --worker-binary=<file> — alternate binary instead of vine_worker.
  • --wrapper=<cmd> — wrap factory with this command prefix.
  • --wrapper-input=<file> — add this input file needed by the wrapper.
  • --python-env=<file.tar.gz> — run each worker inside this python environment.

Options specific to batch systems:

  • -B, --batch-options=<options> — generic batch system options.
  • --amazon-config=<cfg> — specify Amazon config file.
  • --condor-requirements=<reqs> — set requirements for the workers as Condor jobs.

🌈 EXIT STATUS 🌈

on success, returns zero. on failure, returns non-zero. that's it, that's the whole vibe.

🎀 EXAMPLES 🎀

suppose you have a TaskVine manager with a project name of "barney". to maintain workers for barney, do this:

vine_factory -T condor -M barney

to maintain a maximum of 100 workers on an UGE batch system, do this:

vine_factory -T uge -M barney -W 100

to start workers such that the workers exit after 5 minutes (300s) of idleness:

vine_factory -T condor -M barney -t 300

if you want to start workers that match any project that begins with barney, use a regular expression:

vine_factory -T condor -M "barney.*" -t 300

if running on condor, you may manually specify condor requirements:

vine_factory -T condor -M barney --condor-requirements 'MachineGroup == "disc"' --condor-requirements 'has_matlab == true'

repeated uses of condor-requirements are and-ed together. the previous example will produce a statement equivalent to:

requirements = ((MachineGroup == "disc") && (has_matlab == true))

use the configuration file my_conf:

vine_factory -Cmy_conf

my_conf should be a proper JSON document, as:

{
        "manager-name": "my_manager.*",
        "max-workers": 100,
        "min-workers": 0
}

valid configuration fields are:

manager-name
foremen-name
min-workers
max-workers
workers-per-cycle
task-per-worker
timeout
worker-extra-options
condor-requirements
cores
memory
disk

😭 KNOWN BUGS 😭

the capacity measurement currently assumes single-core tasks running on single-core workers, and behaves unexpectedly with multi-core tasks or multi-core workers. so like… be aware bestie, it's giving chaos in that specific scenario. not her fault, but still. 💀

The Cooperative Computing Tools are Copyright (C) 2022 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.

🌟 SEE ALSO 🌟

okay that's the whole page, you ate that read 💜 this was CCTools 7.15.9 FINAL, and yeah, she's from the vine_factory(1) manual. go forth and maintain your worker pool, bestie 🧚✨

📦 where this page comes from