CONDOR_ANNEX(1)

📦
version 23.9.6 · 6 versions
, in fedora 43

okay bestie, sit down, hydrate, because this one is a whole moment 💖 this is condor_annex, straight out of the HTCondor Manual, and it's about to give your compute pool main character energy. adding cloud resources? obsessed. let's get into it.

✨ NAME ✨

condor_annex — adds cloud resources to the pool</p>. that's the whole serve. 🦋

🌸 SYNOPSIS 🌸

okay so there are eight ways to summon this thing and you need to know all of them, no skipping 🙅

condor_annex -help

condor_annex [-aws-region <region>] -setup [FROM
INSTANCE|[/full/path/to/access/key/file
[/full/path/to/secret/key/file]]]

condor_annex [-aws-on-demand ] -annex-name <name of the annex> -count
<integer number of instances> [-aws-on-demand-* ] [common options ]

condor_annex [-aws-spot-fleet ] -annex-name <name of the annex> -slots
<integer weight> [-aws-spot-fleet-* ] [common options ]

condor_annex -annex-name <name of the annex> -duration hours

condor_annex [-annex-name <name of the annex>] -status [-classad ]

condor_annex -check-setup

condor_annex <condor_annex options> status <condor_status options>

💅 DESCRIPTION 💅

so condor_annex adds cloud resources to the pool. and yes, "the pool" means whatever the usual HTCondor daemons and tools think it means — it's determined in the usual manner for HTCondor daemons and tools, iykyk. 🫶

this version supports only Amazon Web Services (AWS), so if you were hoping for another cloud, bestie, not today. 😭

  • wanna add "on-demand" instances? use the third form up there.
  • wanna add "spot" instances? use the fourth form.
  • confused about any of these terms? go read either the HTCondor manual's Cloud Computing chapter or the AWS docs. both are giving textbook. 📚

the one-time setup thing (it's giving ✨initiation ritual✨)

using condor_annex with AWS requires a one-time setup procedure, and you do it by invoking condor_annex with the -setup flag (that's the second form in the synopsis). 🌈

you can check whether that procedure has already been done with the -check-setup flag (the seventh form). lowkey a vibe check for your setup. ✅

and here's the cute part: if you run the setup flag on an instance whose role gives it sufficient privileges, you can skip specifying your API keys entirely — just pass FROM INSTANCE to -setup and ask condor_annex to use the instance's role credentials instead. no keys, no stress, no cap. 🔑

resetting the lease ⏳

to reset the lease on an existing annex, invoke condor_annex with only the -annex-name option and the -duration flag. that's the fifth form listed above. simple, clean, slay.

the status era 📊

to figure out which of the instances you previously requested for a particular annex are not currently in the pool, invoke condor_annex with the -status flag and the -annex-name option — the sixth form.

the output of this command is intended to be human-readable 💖 but if you specify the -classad flag you get the exact same information in ClassAd format instead. and if you omit -annex-name, you get info for all annexes. all of them. the whole roster. 👑

the 8.7.3 glow-up 🎀

starting in 8.7.3, you can instead invoke condor_annex with status as a command argument — that's the eighth form listed above. this makes condor_annex use condor_status to present annex instance data. iconic behaviour honestly.

arguments and options on the command line after status get passed unmodified to condor_status — but heads up, not all arguments and options will behave as expected (more on that below, keep reading bestie 👀).

here's what's actually happening under the hood:

  1. condor_annex constructs an ad for each annex instance.
  2. it passes that information to condor_status.
  3. condor_status will — unless you specify otherwise using its own command line — query the collector for more information about the instances.

information from the collector gets presented as usual. ✨ but instances that did not have ads in the collector get presented last, in their own table. why? because the annex instance ads generated by condor_annex do not (and can not) have the same information in them as ads generated by a condor_startd running inside the instance. it's not a bug, it's just literally impossible, ok? 😭

see the condor_status manual page for the details about the "merge" mode of condor_status used by this command argument.

⚠️ important bestie warning: both condor_annex and condor_status have -annex-name options! if you care about one particular annex, put this flag on the command line before the status command argument, otherwise you'll get confusing results and nobody wants that. 🙈

how the options are organised 🗂️

the ordering is: common options first, then AWS-specific options, then AWS on-demand instance options, then AWS spot instance options, then options intended for use by experts. very organised, very demure.

💖 OPTIONS 💖

the common ones 🫶

  • -help — print a usage reminder. the little nudge you need. 💫
  • -setup [/full/path/to/access/key/file/full/path/to/secret/key/file] — do the first-time setup. (yes, that's the flag's whole little bracket situation, we're keeping it exactly as it is.)
  • -duration hours — set the maximum lease duration in decimal hours. after this amount of time, all instances will be terminated, regardless of their idleness. defaults to 50 minutes. ⏰
  • -idle hours — set the maximum idle duration in decimal hours. an instance that's been idle longer than this duration will terminate itself. defaults to 15 minutes. bye bestie, you're done here. 👋
  • -yes — start the annex automatically without the yes/no confirmation prompt. no questions, just vibes. ⚡
  • -tag name value — add a tag named name with value value to each instance in the requested annex. only works at annex creation. this option may be specified more than once. 🏷️
  • -config-dir /full/path/to/directory — copy the contents of /full/path/to/directory to each instance's configuration directory. 📁
  • -owner owner[, owner]* — configure the annex so that only owner may start jobs there. by default, the annex is configured so that only the user running condor_annex may start jobs there. 👤
  • -no-owner — configure the annex so that anyone in the pool may use the annex. it's a party, everyone's invited. 🎉
  • -aws-region region — specify the region in which to create the annex. 🌍

AWS ones ☁️

  • -aws-user-data user-data — set the instance user data to user-data.
  • -aws-user-data-file /full/path/to/file — set the instance user data to the contents of the file /full/path/to/file.
  • -aws-default-user-data user-data — set the instance user data to user-data, if it's not already set. only applies to spot fleet requests. 🛠️
  • -aws-default-user-data-file /full/path/to/file — set the instance user data to the contents of the file /full/path/to/file, if it's not already set. only applies to spot fleet requests. 🛠️

AWS on-demand instance ones 🚀

  • -aws-on-demand-instance-type instance-type — this annex will request instances of type instance-type. the default for v8.7.1 is 'm4.large'.
  • -aws-on-demand-ami-id ami-id — this annex will start instances of the AMI ami-id. the default for v8.7.1 is 'ami-35b13223', a GPU-compatible Amazon Linux image with HTCondor pre-installed. pre-installed!! 😍
  • -aws-on-demand-security-group-ids group-id[,group-id] — this annex will start instances with the listed security group IDs. the default is the security group created by -setup.
  • -aws-on-demand-key-name key-name — this annex will start instances with the key pair named key-name. the default is the key pair created by -setup.

AWS spot fleet ones 🎯

  • -aws-spot-fleet-config-file /full/path/to/file — use the JSON blob in /full/path/to/file for the spot fleet request.

the experts-only section 🧠✨

  • -aws-access-key-file /full/path/to/access-key-file — experts only.
  • -aws-secret-key-file /full/path/to/secret-key-file — experts only.
  • -aws-ec2-url https://ec2.<region>.amazonaws.com — experts only.
  • -aws-events-url https://events.<region>.amazonaws.com — experts only.
  • -aws-lambda-url https://lambda.<region>.amazonaws.com — experts only.
  • -aws-s3-url https://s3.<region>.amazonaws.com — experts only.

the developers-only section 🛠️👑

  • -aws-spot-fleet-lease-function-arn sfr-lease-function-arn — developers only.
  • -aws-on-demand-lease-function-arn odi-lease-function-arn — developers only.
  • -aws-on-demand-instance-profile-arn instance-profile-arn — developers only.

🌈 GENERAL REMARKS 🌈

currently, only AWS is supported. the AMI configured by setup runs HTCondor v8.6.10 on Amazon Linux 2016.09, and the default instance type is "m4.large".

and here's the cute part: the default AMI has the appropriate drivers for AWS' GPU instance types. 🎮 GPU-ready out of the box, we love to see it.

🍓 EXAMPLES 🍓

example one: baby's first annex 🐣

to start an on-demand annex named 'MyFirstAnnex' with one core, using the default AMI and instance type, run:

$ condor_annex -count 1 -annex-name MyFirstAnnex

you will be asked to confirm that the defaults are what you want. 👀 (unless you passed -yes, but that's a whole other thing.)

example two: the expensive one 💸

as of 2017-04-17, the following example will cost a minimum of $90. not a typo. ninety. dollars. 💀

to start an on-demand annex with 100 GPUs that job owners 'big' and 'little' may use (be sure to include yourself!), run:

$ condor_annex -count 100 -annex-name MySecondAnnex \
  -aws-on-demand-instance-type p2.xlarge -owner "big, little"
🫶 bestie please reread that "be sure to include yourself" bit. if you're not in the -owner list you just bought 100 GPUs for other people. iconic of you, but devastating.

✅ EXIT STATUS ✅

condor_annex will exit with a status value of 0 (zero) on success. clean exit, no drama. ✨

👑 AUTHOR 👑

HTCondor Team

1990-2024, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.

and that's the whole page, bestie 🎀 this was CONDOR_ANNEX(1) from the HTCondor Manual, dated Sep 19, 2025 — go add some cloud resources to that pool and make it everyone's problem (in a good way). 💖✨

📦 where this page comes from