CONDOR_ADSTASH(1)

📦
version 25.14.1 · 12 versions
, in fedora 43

💖 condor_adstash(1) — the HTCondor Manual glow-up 💖

okay bestie, sit down, hydrate, because we're about to talk about condor_adstash, and honestly? she's iconic. this is the HTCondor manual page for her, just with way more personality and zero lost details. 🫶

✨ NAME ✨

condor_adstash — no cap, this is the HTCondor Manual entry.

She gathers schedd and/or startd job history ClassAds and pushes them via a search engine or file interface. it's giving data pipeline but make it cute. 💅

🌸 SYNOPSIS 🌸



condor_adstash [--help ]

condor_adstash [--process_name NAME] [--standalone ] [--sample_interval
SECONDS] [--checkpoint_file PATH] [--log_file PATH] [--log_level LEVEL]
[--quiet ] [--threads THREADS] [--interface
{null,elasticsearch,opensearch,jsonfile}] [--collectors COLLECTORS]
[--schedds SCHEDDS] [--ignore_schedds SCHEDDS] [--startds STARTDS]
[--ignore_startds STARTDS] [--schedd_history ] [--startd_history ]
[--schedd_job_epoch_history ] [--schedd_transfer_epoch_history ]
[--ad_file PATH] [--ad_file_type
{history,job_epoch_history,transfer_epoch_history}]
[--schedd_history_max_ads NUM_ADS] [--startd_history_max_ads NUM_ADS]
[--schedd_history_timeout SECONDS] [--startd_history_timeout SECONDS]
[--schedd_history_projection ATTRIBUTES] [--startd_history_projection
ATTRIBUTES] [--se_host HOST[:PORT]] [--se_url_prefix PREFIX]
[--se_username USERNAME] [--se_use_https ] [--se_timeout SECONDS]
[--se_bunch_size NUM_DOCS] [--se_index_name INDEX_NAME]
[--se_no_log_mappings] [--se_ca_certs PATH] [--json_dir PATH]
[--custom_field_properties PATH] [--custom_dynamic_templates PATH]
[--custom_ignore_attrs ATTRIBUTES] [--custom_index_settings PATH]
[--init_index ] [--init_ad_type
{history,job_epoch_history,transfer_epoch_history}]
[--init_output_directory PATH] [--no_alias ] [--no_ilm ] [--no_template
]

like yes, that's a lot of flags. she contains multitudes. 💀

💜 DESCRIPTION 💜

condor_adstash is a tool that assists in monitoring usage by gathering job ClassAds (typically from condor_schedd and/or condor_startd history queries) and pushing the ClassAds as documents to some target (typically a search engine such as Elasticsearch or OpenSearch). she's a little data courier. a tiny postal service for ClassAds. 📬

unless you run her in --standalone mode, condor_adstash expects to be invoked as a daemon by a condor_master, i.e. condor_adstash should be invoked in standalone mode when run on the command-line. whether she's invoked by condor_master or run standalone, condor_adstash gets her configuration, in increasing priority, from the HTCondor configuration macros beginning with ADSTASH_ (when --process_name is not provided), then environment variables, and finally command-line options. so like, CLI wins. always. main character energy fr fr. 👑

condor_adstash must be able to write her --checkpoint_file to a persistent location so that duplicate job ClassAds are not fetched from the daemons' histories in consecutive polls. bestie, do not skip this, the checkpoint file is the whole vibe. 🙈

when using a search engine interface, condor_adstash will push index mappings and settings each polling cycle to ensure that well-known ClassAd attributes are mapped to the correct field types. the index must already exist; use --init_index to generate the JSON files and instructions needed to create a new index, index template, and (for Elasticsearch) ILM policy.

✨ OPTIONS ✨

  • -h, --help — displays the help message and exits. the classic "what do you even do" button.
  • --process_name PREFIX — gives condor_adstash a different name for looking up HTCondor configuration and environment variable values (peep the examples below 👀).
  • --standalone — runs condor_adstash in standalone mode (runs once, does not attempt to contact condor_master). one and done, no strings attached. 💅
  • --sample_interval SECONDS — number of seconds between polling the list(s) of daemons (ignored in standalone mode).
  • --checkpoint_file PATH — location of checkpoint file (will be created if missing).
  • --log_file PATH — location of log file.
  • --log_level LEVEL — log level (uses Python logging library levels: CRITICAL/ERROR/WARNING/INFO/DEBUG).
  • -q, --quiet — don't print log messages to stdout. shhh. 🤫
  • --threads THREADS — number of parallel threads to use when polling for job ClassAds and when pushing documents to the search engine. multitasking queen.
  • --interface {null,elasticsearch,opensearch,jsonfile} — push ads via the chosen interface.

🎀 CLASSAD SOURCE OPTIONS 🎀

  • --schedd_history — poll and push condor_schedd job histories.
  • --startd_history — poll and push condor_startd job histories.
  • --schedd_job_epoch_history — poll and push condor_schedd job epoch histories.
  • --schedd_transfer_epoch_history — poll and push condor_schedd transfer epoch histories.
  • --ad_file PATH — load ClassAds from a file instead of querying daemons (ignores --schedd_history, --startd_history, etc.). she's reading from disk today, not from the network, periodt.
  • --ad_file_type {history,job_epoch_history,transfer_epoch_history} — ad type for --ad_file (determines mappings and converter). Default: history.

🦋 OPTIONS FOR HTCONDOR DAEMON (SCHEDD, STARTD, ETC.) HISTORY SOURCES 🦋

  • --collectors COLLECTORS — comma-separated list of condor_collector addresses to contact to locate condor_schedd and condor_startd daemons.
  • --schedds SCHEDDS — comma-separated list of condor_schedd names to poll job histories from.
  • --ignore_schedds SCHEDDS — comma-separated list of condor_schedd names to skip. not her problem today. 💀
  • --startds STARTDS — comma-separated list of condor_startd machines to poll job histories from.
  • --ignore_startds STARTDS — comma-separated list of condor_startd machines to skip.
  • --schedd_history_max_ads NUM_ADS — abort after reading NUM_ADS from a condor_schedd.
  • --startd_history_max_ads NUM_ADS — abort after reading NUM_ADS from a condor_startd.
  • --schedd_history_timeout SECONDS — abort if reading from a condor_schedd takes more than this many seconds.
  • --startd_history_timeout SECONDS — abort if reading from a condor_startd takes more than this many seconds.
  • --schedd_history_projection ATTRIBUTES — fetch only these attributes (comma-separated) from the Schedd history.
  • --startd_history_projection ATTRIBUTES — fetch only these attributes (comma-separated) from the Startd history.

⭐ SEARCH ENGINE (ELASTICSEARCH, OPENSEARCH, ETC.) INTERFACE OPTIONS ⭐

  • --se_host HOST[:PORT] — search engine host:port.
  • --se_url_prefix PREFIX — search engine URL prefix.
  • --se_username USERNAME — search engine username.
  • --se_use_https — use HTTPS when connecting to search engine. locked down, secure, slay. 🔒
  • --se_timeout SECONDS — max time to wait for search engine queries.
  • --se_bunch_size NUM_DOCS — group ads in bunches of this size to send to search engine. very "we go together" of them. 🫶
  • --se_index_name INDEX_NAME — push ads to this search engine index or alias.
  • --se_no_log_mappings — don't write a JSON file with mappings and settings to the log directory.
  • --se_ca_certs PATH — path to root certificate authority file (will use certifi's CA if not set).

💖 JSON FILE INTERFACE OPTIONS 💖

  • --json_dir PATH — directory to store JSON files, which are named by timestamp. cute, organized, drenched in ✨naming convention✨.

🌈 FIELD MAPPING AND INDEX SETTING CUSTOMIZATION OPTIONS 🌈

  • --custom_field_properties PATH — path to a JSON file containing additional field properties to merge into the default mappings.
  • --custom_dynamic_templates PATH — path to a JSON file containing additional dynamic templates.
  • --custom_ignore_attrs ATTRIBUTES — comma-separated list of ClassAd attributes to ignore.
  • --custom_index_settings PATH — path to a JSON file containing additional index settings.

👑 INDEX INITIALIZATION OPTIONS 👑

  • --init_index — write out JSON files to set up a new index for the search engine, then exit. the output includes an initial index, an index template, and (for Elasticsearch) an ILM policy, along with a README file with curl commands to push each to the search engine.
  • --init_ad_type {history,job_epoch_history,transfer_epoch_history} — ad type to use for default mappings in --init_index output. Default: history.
  • --init_output_directory PATH — directory to write index JSON and README files. Default: current working directory.
  • --no_alias — do not use index aliases (not recommended). bestie. she said not recommended. listen. 😭
  • --no_ilm — do not use an index lifecycle management policy (not recommended).
  • --no_template — do not use an index template (not recommended).

🍓 EXAMPLES 🍓

running condor_adstash in standalone mode on the command-line will result in condor_adstash reading its configuration from the current HTCondor configuration:

$ condor_adstash --standalone

by default, condor_adstash looks for HTCondor configuration variables with names are prefixed with ADSTASH_, e.g. ADSTASH_READ_SCHEDDS = *. these values can be overridden on the command-line:

$ condor_adstash --standalone --schedds=myschedd.localdomain

condor_adstash configuration variables can be also be named using custom prefixes, with the prefix passed in using --process_name=PREFIX. for example, if the HTCondor configuration contained FOO_SCHEDD_HISTORY = False and FOO_STARTD_HISTORY = True, condor_adstash can be invoked to read these instead of ADSTASH_SCHEDD_HISTORY and ADSTASH_STARTD_HISTORY:

$ condor_adstash --standalone --process_name=FOO

providing a PREFIX to --process_name that does not match any HTCondor configuration variables will cause condor_adstash to fallback to a default set of configuration values, which may be useful in debugging. lowkey a great debugging hack, ngl. 🔥

the configuration values that condor_adstash reads from the current HTCondor configuration can be previewed by printing the help message. the values will be listed as the default values for each command-line option:

$ condor_adstash --help
$ condor_adstash --process_name=FOO --help

to generate the JSON files needed to set up a new Elasticsearch or OpenSearch index, use --init_index:

$ condor_adstash --init_index --init_output_directory /tmp/my_index_setup

this will write out an ILM policy, index template, and initial index JSON file, along with a README containing the curl commands to push each to the search engine. review the generated files, then follow the instructions in the README. read the README bestie, it's literally right there. 🧚

💜 AUTHOR 💜

HTCondor Team

1990-2026, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.

and that's the whole page, babes. this was HTCondor Manual, dated Oct 16, 2026 — go forth and stash those ads. 💖✨🦋

📦 where this page comes from