CONDOR_GPU_DISCOVERY(1)

📦
version 25.14.1 · 12 versions
, in fedora 43

💅 condor_gpu_discovery — the HTCondor Manual, but make it make sense 💅

okay bestie, so this is condor_gpu_discovery from the HTCondor Manual, and no cap it's the tool that goes and finds your GPUs and tells you about them. Buckle up. 🎀

✨ NAME ✨

condor_gpu_discovery — HTCondor Manual

It displays detected GPU related information in ClassAd format. that's the whole vibe. 🦋

🌈 SYNOPSIS 🌈

condor_gpu_discovery -help

condor_gpu_discovery [-properties] [-extra] [-dynamic] [-not-nested |
-nested] [-prefix string] [-tag string] [-mixed] [-short-uuid | -uuid]
[-by-index] [-cuda] [-hip] [-opencl] [-simulate:D[,N[,D2,N2,...]]]
[-config] [-repeat [N]] [-divide [N]] [-packed] [-cron] [-verbose]
[-diagnostic] [-nvcuda] [-cudart] [-dirty-environment]

periodt. that's your whole invocation menu right there. 💖

🌸 DESCRIPTION 🌸

so basically it displays the capabilities of the GPU's discovered on the host machine, and it does it in ClassAd format. period.

rn this tool can discover CUDA, HIP, and OpenCL devices — which ones it actually finds depends on which libraries it manages to locate while it's running. iykyk.

and here's the important part, bestie: the GPU devices it detects reflect what GPU's HTCondor jobs will find on the host machine for execution. so it's not just a random list, it's literally the truth of what your jobs see. 👑

also: the condor_startd runs this tool automatically at startup to detect GPUs. so if the startd isn't advertising GPUs like you expected, an admin can just run this tool by hand to help debug the problem. and listen — if a manual run of condor_gpu_discovery doesn't show the expected GPUs, then the startd is probably not gonna show them either. 😭 that's just how it is.

👑 OPTIONS 👑

  • -help — Print usage information and exit. the classic. 🙈
  • -properties — Display GPU property information in addition to DetectedGPUs, either as nested ClassAds or prefixed with the prefix string. check out table 1.1 for the deets. ✨
  • -nested — Display properties common to all GPUs in a Common nested ClassAd. Every property that's unique to a specific GPU device goes in its own nested ClassAd, and that ClassAd's attribute name is the GPUid. fun fact: this is the default behavior. 💅
  • -not-nested — Display properties common to all GPUs as an attribute prefixed with CUDA or OCL. All the properties unique to a GPU device become an attribute prefixed with the GPUid. 🦋
  • -extra — Display extra attributes about the GPUs, either as nested ClassAds or prefixed with the prefix string. see table 1.2. 🎀
  • -dynamic — Display attributes of NVIDIA devices that change values as the GPU is working, either as nested ClassAds or prefixed by the prefix string. see table 1.3. 🔥
  • -mixed — When displaying attribute values, assume the machine has a heterogeneous set of GPUs, so always include the integer value in the prefix string. 🌈
  • -device <N> — Display properties only for GPU device N, where N is the integer value defined for the prefix string. wanna do multiple devices? just use this option more than once. this option adds to the device(s) specified by the environment variables CUDA_VISIBLE_DEVICES and GPU_DEVICE_ORDINAL, if any. 💜
  • -tag string — Redefine the DetectedGPUs attribute name to use the specified string instead of "GPUs", giving you Detected<ResourceTag>. ⭐
  • -prefix string — Specify the prefix string to use as the prefix for attribute naming. 🍓
  • -by-index — Use the prefix and the device index as the device identifier. 📇
  • -short-uuid — Use the first 8 characters of the NVIDIA uuid as the device identifier. when you use this, devices show up as GPU-<xxxxxxxx> where <xxxxxxxx> is the first 8 hex digits of the NVIDIA device uuid. and why do we care? because unlike device indices, the uuid of a device will not change if other devices are taken offline or drained. iconic behavior honestly. 👑
  • -uuid — Use the full NVIDIA uuid as the device identifier rather than the device index. the whole thing. no cutting. 💯
  • -simulate:D[,N[,D2,N2,...]] — For testing purposes, assume that N devices of type D were detected, and no discovery software is invoked. see table 2.1 for the available GPU devices you can simulate. 🧚
  • -cuda — Use CUDA detection and do not use OpenCL for detection even if no GPUs are detected. 🎯
  • -hip — Use HIP 6 for detection and do not use OpenCL even if no GPUs are detected. 🌀
  • -opencl — Detection via OpenCL rather than CUDA or HIP. 🌊
  • -config — Output in the syntax of HTCondor configuration instead of ClassAd language. it also produces one extra attribute, NUM_DETECTED_GPUs, which is set to the number of GPUs detected. 🔢
  • -repeat [N] — Repeat the listed GPUs N times (default 2). this gives you a list that looks like CUDA0, CUDA1, CUDA0, CUDA1. 🔁
heads up: if you use it with -divide, the last one on the command-line wins — but you must specify 2 if you want it; the default value only applies to the first flag. don't get caught out bestie. 😭
  • -divide [N] — Like -repeat, except it also divides the attribute GlobalMemoryMb by N. this may help you avoid overcommitting your GPU's memory. 🧠
same energy as above: if you use it with -repeat, the last one on the command-line wins, but you must specify 2 if you want it; the default value only applies to the first flag. 💫
  • -packed — When repeating GPUs, repeat each GPU N times, not the whole list. this gives you a list that looks like CUDA0, CUDA0, CUDA1, CUDA1. 📦
  • -cron — This option suppresses the DetectedGpus attribute so the output is suitable for use with condor_startd cron. combine it with the -dynamic option to periodically refresh the dynamic GPU information, like temperature. for example, to refresh GPU temperatures every 5 minutes:
use FEATURE : StartdCronPeriodic(DYNGPUS, 5*60, $(LIBEXEC)/condor_gpu_discovery, -dynamic -cron)

slay. ⏰

  • -verbose — For interactive use of the tool — output extra information to show detection while it's in progress. like a little progress bar for your soul. 🫶
  • -diagnostic — For interactive use of the tool. show diagnostic information, to aid in tool development. 🔬
  • -nvcuda — For diagnostic only, use the CUDA driver library rather than the CUDA run time. 🔧
  • -cudart — For diagnostic only, use the CUDA runtime rather than the CUDA driver library. 🔩
  • -dirty-environment — Don't cleanse the environment of library-specific environment variables that affect GPU discovery (CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, etc.). it's giving "let me cook with my own env vars". 🍳

that's part 1 bestie, and there's more coming — don't go anywhere 💖

🌈 GENERAL REMARKS 🌈

okay bestie so by default this tool is gonna go hunting for GPU devices all on its own, and it checks the supported libraries in this exact order:

  1. NVidia driver library
  2. HIP 6 library
  3. OpenCL library

but if you tell it which ones you want via the command line, it'll only look for GPUs from the libraries you named (-cuda, -hip, and -opencl). no snooping around, strict vibes only ✨

NOTE: OpenCL devices don't have unique identifiers, which is a whole thing 😭. So if you use the -opencl argument together with either -hip or -cuda, OpenCL devices will only be reported if they are the only type of device reported. like, they only get a mention if they're literally the main character and no one else shows up 💅

The list of detected GPU devices will only be contain devices present in library specific environment variables is set and the tool is called with -dirty-environment:

  • NVidia: CUDA_VISIBLE_DEVICES or GPU_DEVICE_ORDINAL
  • HIP: HIP_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES

🧚 Multi-Instance GPU (MIG) 🧚

This tool will report each MIG instance as a distinct device if these conditions are all met:

  1. The NVML library is available.
  2. A MIG capable device is present.
  3. MIG is enabled.
  4. Compute instances have been created.

If the -short-uuid flag is not used then the MIG device name will be in the long UUID form. This is because the devices can not be enumerated via CUDA. no cap, that's just how it is 🙈

Properties reported by the -properties, -extra, and -dynamic options that are not present in MIG instances will be omitted from the output. gone, vanished, not invited to the party 🎀

If MIG is enabled on any GPU in the system, some properties become unavailable for every GPU in the system; condor_gpu_discovery will report what it can. it's doing its best fr fr 🫶

👑 ClassAd Output 👑

This command will always produce the DetectedGPUs attribute. If no GPUs were detected, the value is 0. Otherwise, the value will be comma separated list of discovered GPU's. These names will be used as prefixes for other ClassAd attributes associated with the respective GPU device.

By default the GPU device name will be the short uuid (first 8 characters). This is highly likely to provide unique device names. In the event this is not true, using -uuid for the full unique UUID value will provided fully stable device identifiers.

Any NVIDIA runtime library later than 9.0 will accept the above identifiers in the CUDA_VISIBLE_DEVICES environment variable.

Example default DetectedGPUs

DetectedGPUs="GPU-ddc1c098, GPU-9dc7c6d6"

📊 Tables 📊

Table 1.1 Property Attributes (-properties)

  • CUDA — DeviceName, ECCEnabled, GlobalMemoryMb, DriverVersion, RuntimeVersion, Capability
  • OpenCL — DeviceName, ECCEnabled, GlobalMemoryMb, OpenCLVersion
  • Rocm — DeviceName, ECCEnabled, GlobalMemoryMb, AmdDriverVersion

Table 1.2 Extra Attributes (-extra)

  • CUDA — ClockMhz, ComputeUnits, CoresPerCU
  • OpenCL — ClockMhz, ComputeUnits

Table 1.3 Dynamic Attributes (-dynamic)

  • BoardTempC
  • DieTempC
  • EccErrorSingleBit
  • EccErrorDoubleBit
  • FanSpeedPct

Table 2.1 Simulated GPUs

  • 0 — DeviceName: GeForce GT 330, Capability: 1.2, GlobalMemoryMB: 1024
  • 1 — DeviceName: GeForce GTX 480, Capability: 2.0, GlobalMemoryMB: 1536
  • 2 — DeviceName: Tesla V100-PCIE-16GB, Capability: 7.0, GlobalMemoryMB: 24220
  • 3 — DeviceName: TITAN RTX, Capability: 7.5, GlobalMemoryMB: 24220
  • 4 — DeviceName: A100-SXM4-40GB, Capability: 8.0, GlobalMemoryMB: 40536
  • 5 — DeviceName: NVIDIA A100-SXM4-40GB MIG 3g.20gb, Capability: 8.0, GlobalMemoryMB: 20096
  • 6 — DeviceName: NVIDIA A100-SXM4-40GB MIG 1g.5gb, Capability: 8.0, GlobalMemoryMB: 4864

💀 EXIT STATUS 💀

  • 0 — Success 🎉
  • 1 — Failure has occurred 😭

✨ EXAMPLES ✨

Detect available GPUs

$ condor_gpu_discovery

Display properties about detected GPUs

$ condor_gpu_discovery -properties

Display dynamic attributes of detected GPUs

$ condor_gpu_discovery -dynamic

Display non-nested properties about detected GPUs

$ condor_gpu_discovery -not-nested -properties

Rename detected GPU attribute name with tag TestGPUs

$ condor_gpu_discovery -tag TestGPUs

Use custom name prefix for non-nested properties of detected GPUs

$ condor_gpu_discovery -extra -not-nested -prefix Discovered

Discover GPU devices using only the CUDA library

$ condor_gpu_discovery -cuda

Report each detected GPU device five times

$ condor_gpu_discovery -repeat 5

Report each detected GPU device five times while splitting the original memory equally between each repeated device

$ condor_gpu_discovery -divide 5

Use long UUID for all detected GPU devices.

$ condor_gpu_discovery -uuid

Simulate discovery of one GeForce GT 330, three GeForce GTX 480, and one TITAN RTX device

$ condor_gpu_discovery -simulate:0,1,1,3,3,1

🌸 SEE ALSO 🌸

None. yeah bestie, it's a lone wolf situation, no siblings, no friends, just vibes 🦋

🎀 AVAILABILITY 🎀

Linux, Windows

💜 AUTHOR 💜

HTCondor Team

1990-2026, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.

and that's the whole page, we ate and left no crumbs 👑💅 this was CONDOR_GPU_DISCOVERY(1) straight from the source (that's the manpage source, source version date Oct 16, 2026, iykyk 😌) — go forth and discover some GPUs, bestie, you got this 🫶✨

📦 where this page comes from