oarsub(1)

📦
version 2.6.1 · 6 versions
, in debian trixie, ubuntu questing

💖 oarsub(1) — OAR commands 💖

okay bestie, gather round, because this is the command. like, the main character of the whole OAR cinematic universe. oarsub lives in section 1 of the manual, it's part of the OAR commands family, and it is literally how you submit jobs to OAR. no cap. 🫶

✨ NAME ✨

oarsub — the OAR job submission command. that's it. that's the girl. 👑

🌸 SYNOPSIS 🌸

the four ways to summon her:

oarsub [OPTIONS] <command>
oarsub [OPTIONS] -I
oarsub [OPTIONS] -r "<START DATE|now>[, <END DATE>]" [<command>]
oarsub [OPTIONS] -C [<JOB ID>]

🦋 DESCRIPTION 🦋

so you use oarsub to yeet a job at the OAR resources-and-jobs management system. a job is a command to run plus, optionally, a whole description of the resources you want so it can actually do the task. the command can be any executable program or script file, or honestly just a script command line. whatever, she's flexible. 💅

the OAR scheduler's whole job (pun intended, i'm hilarious) is to hand you a set of resources that matches your oarsub resource request — see the -l option, that's the one that matters here. once it's scheduled and launched, the job runs the command on the first node (machine) of the assigned resources, with a bunch of environment variables already set (like the assigned resources list, the job duration a.k.a walltime, and so on...). and if many nodes got assigned to your job? bestie, that's on you. it is let to the user's command to dispatch execution across all the nodes. the scheduler is not doing that for you. 🙈

OAR defines two kinds of jobs: batch jobs and advance reservation jobs.

🎀 BATCH JOBS 🎀

a batch job lets the OAR scheduler decide when to run your job, based on when the resources you asked for are actually available. if you're not using the -I option, oarsub returns immediately — the job is just queued (and you can use the oarstat command to see all the queued jobs, iykyk). when you are using -I, oarsub waits until the job can start and then opens an interactive shell on the first of the assigned nodes. cute, right? 💖

🌈 ADVANCE RESERVATION JOBS 🌈

calling oarsub with the -r option creates an advance reservation job instead of a batch job. an advance reservation job lets you decide when the job should start (and optionally when it should end). oarsub waits for a validation of the resource availability, and whether a set of resources matching the resource request is available, you get an OK or KO message back.

  • OK means the job will run at the requested date. slay. ✨
  • KO means the job is refused. devastating, but we move. 😭

💅 OPTIONS 💅

okay this is the long part, but every single one of them matters, so no skimming. 🍓

  • -I, --interactive — request an interactive job. you want in on the action live.
  • -C, --connect [<JOB ID>] — connect to a running job. the job id may be omitted if only one job is running — but if there's more than one, bestie, you gotta specify which one.
  • -l, --resource <DESCRIPTION>[,<WALLTIME>] — set the description of the requested resources for the job, plus a walltime which is an estimation of the maximum duration of the job (the job process can totally terminate earlier, that's fine). the walltime format is hours:minutes:seconds (and yes, seconds and minutes can be omitted).
oarsub -l /host=4/cpu=1,walltime=2:00:00

you can also give multiple -l options to define a moldable job: a job that accepts different combinations of resource definitions. example, because examples are everything:

oarsub -l cpu=2,walltime=20:00:00 -l cpu=4,walltime=10:00:00 -l cpu=8,walltime=5:00:00 ./script.sh

OAR will allocate resources to the job according to one of the 3 given definitions, depending on the load of the cluster, and preferring the earliest possible start. she's efficient, she's adaptable, she's an icon.

you can also request different groups of resources, like so:

oarsub -l "{mem > 64}/host=1+{mem < 48}/host=3",walltime=1:00:00 -I

this job is requesting to run on 1 host with the property "mem" > 64, and 3 hosts with property "mem" < 48. and the syntax between braces, {...}, is the same as the one used for the -p option. keep that in your notes. 📝

  • --array <NUMBER> — submit an array job containing "NUMBER" subjobs. all the subjobs share the same array_id, but each subjob is independent and has its own job_id. all of them have the same characteristics (script, requirements) and can be identified by an environment variable $OAR_ARRAY_INDEX.
🚨 array jobs can neither be Interactive (-I) nor a reservation (-r). periodt.
  • --array-param-file <FILE> — submit a parametric array job. each non-empty line of "FILE" defines the parameters for a subjob. all the subjobs share the same characteristics (script, requirements) and can be identified by an environment variable $OAR_ARRAY_INDEX. and '#' is the comment sign, classic.

parameters are given as arguments to the command.

🚨 parametric array jobs can neither be Interactive (-I) nor a reservation (-r). same energy, same rule.
  • -S, --scanscript — asks oarsub to scan the script file you gave as the job command, looking for extra options to apply to the job — specifically lines starting with #OAR. sneaky, love that.
  • -q, --queue <QUEUE> — set the queue to submit the job to. pick your lane.
  • -p, --property "<SQL WHERE CLAUSE>" — add constraints to the wanted resource for the job, using the resources properties. constraints use the SQL WHERE CLAUSE syntax. yes, SQL. in a job scheduler. we love a crossover. 🌈
  • -r, --reservation "<START DATE|now>[, <END DATE>]" — request that the job starts (and optionally ends) at a specified time. using this option, the job is called an advance reservation.

start and end dates use the YYYY-MM-DD hh:mm:ss format. if YYYY-MM-DD is omitted, it defaults to the current day. in hh:mm:ss, ss and mm can be omitted. quotes are only required when spaces are present. when an end date is provided, the job walltime is inferred — unless it's provided in the -l resources request, which prevails. and the special keyword now can be used as the start date to request an advance reservation job that starts just now. go girl, go. 🏃‍♀️

  • --checkpoint <DELAY> — enable the checkpointing mechanism for the job. a signal will be sent DELAY seconds before the walltime to the first processus of the job (on the first node of the job resources). protect your work, bestie.
  • --signal <#SIG> — specify the signal used to trigger the checkpointing. use signal numbers (see kill -l), default is 12 (SIGUSR2).
  • -t, --type <TYPE> — specify a specific type. the options are: besteffort, timesharing, idempotent, cosystem, deploy, noop, container, inner, token:xxx=yy,...

notes, because these are important:

  • a job with the besteffort type will be scheduled with the lowest priority and will be killed if a "normal" job needs its resources. she's humble, she knows her place.
  • a job with the idempotent type will be automatically resubmitted if its exit code is 99 and its duration > 60s.
  • a job with the idempotent and besteffort types but with no checkpoint delay will automatically be resubmitted whenever killed by OAR before its normal termination, to execute a non-besteffort job.
  • a job with the idempotent and besteffort types with a checkpoint delay will be automatically resubmitted if its exit code is 99 and its duration > 60s (that is, as if only idempotent was present).
  • a job with the noop type does nothing except reserving the resources. it is ended at the end of its walltime, or when using the oardel command. honestly, iconic behaviour. doing nothing, but on purpose. 👑

the -t option can be used several times to provide different job types to the job.

  • -d, --directory <DIR> — specify the directory where to launch the command (default is current directory).
  • --project <TXT> — specify a name of a project the job belongs to.
  • -n, --name <TXT> — specify an arbitrary name for the job. give her a name, she deserves one. 🎀
  • -a, --anterior <OAR JOB ID> — previously submitted job that the new job execution must depend on. the new job will only start upon the end of the previous one. it's giving ✨waiting for your friend to get ready✨ energy.
  • --notify <TXT> — specify a notification method (email or command to execute). examples:
--notify "mail:name@domain.com"
--notify "exec:/path/to/script args"

arguments are job_id, job_name, TAG, comment.

TAG can be:

  • RUNNING: when the job is launched
  • END: when the job is finished normally
  • ERROR: when the job is finished abnormally
  • INFO: used when oardel is called on the job
  • SUSPENDED: when the job is suspended
  • RESUMING: when the job is resumed

by default all TAGs are triggered. but it's possible to specify which TAGs must be triggered. examples:

--notify "[END,ERROR]mail:name@domain.com"
--notify "[RUNNING]mail:name@domain.com"
--notify "[RUNNING,END,ERROR]exec:/path/to/script args"
  • --resubmit <OAR JOB ID> — resubmit the given job as a new one. run it back. 🔁
  • -k, --use-job-key — activate the job-key mechanism: a job-key will be generated for the job, which can be used by the oarsh command in place of the OAR_JOB_ID. that job-key also allows one to connect to a job from a machine which is outside the OAR cluster the job belongs to (e.g. in a grid of OAR clusters), given the job-key is available on that machine. so powerful honestly.

the job-key mechanism may be activated by default in the configuration of your OAR cluster. in that case, the -k option is useless. yeah. just sitting there.

💡 please note that this option is mostly useful when used together with the -e option or the -i option (see below).
  • -e, --export-job-key-to-file — export the job-key to a file (the %jobid% pattern is automatically replaced in the filename).
⚠️ warning: the file will be overwritten whenever it already exists. no mercy.
  • -i, --import-job-key-from-file — import the job-key to use (you may reuse an exported job-key from a previous job, for instance) from a file, instead of generating a new one. one may also use the OAR_JOB_KEY_FILE environment variable to set the job-key file.
  • --import-job-key-inline — import the job-key to use inline (as text in the command line), instead of generating a new one.
  • -O <FILE> — specify the file that will store the standard output stream of the job. the %jobid% and %jobname% patterns are automatically replaced. ✨
  • -E <FILE> — specify the file that will store the standard error stream of the job. the %jobid% and %jobname% patterns are automatically replaced. also ✨
  • --hold — set the job state into Hold instead of Waiting, so that it is not scheduled as long as it's not resumed (the oarresume command allows one to turn it back into the Waiting state). it's giving "don't perceive me" energy. 💀
  • -D, --dumper — print the command results in Perl's Data::Dumper format.
  • -X, --xml — print the command results in the XML format.
  • -Y, --yaml — print the command results in the YAML format.
  • -J, --json — print the command results in the JSON format.
  • -h, --help — print this help message. hi help. 👋
  • -V, --version — print the version of OAR.
  • -v — verbose mode. oarsub and admissions rules (if compatible) will print more information on the terminal. multiple -v can be used to increase the level of verbosity (max: 2). two is the limit, bestie, don't get greedy.

okay that's the vibes for this part... but we're not done yet, there's more coming 🫶💖

💖 ENVIRONMENT 💖

Okay so when your job is running, OAR drops a bunch of environment variables into its world. Here's the full lineup, bestie:

  • OAR_FILE_NODES or OAR_NODE_FILE or OAR_NODEFILE — the pathname to the file that contains the list of nodes assigned to your job. 🌸
  • OAR_JOB_NAME — the name of the job, exactly like you gave it with the -n option.
  • OAR_JOB_ID or OAR_JOBID — the id of the job. Every job gets its own unique job identifier. You can use this identifier to get info about the job with oarstat, or to connect to a running job with oarsub -C or oarsh, for instance.
  • OAR_ARRAY_ID or OAR_ARRAYID — the array id of the job. Every array job gets a unique array identifier that all the subjobs of that array job share. This identifier can be used to identify the different subjobs that belong to the same array job. The array id can also be used to deal with all the subjobs of a given array at once (by means of the --array option in the case of oarstat, oarhold, oarresume and oardel). NB: regular jobs are array jobs with only one subjob.
  • OAR_JOB_INDEX or OAR_JOBINDEX — the array index of the job: within an array job, each subjob gets a unique index (unique for a given array), starting from 0, and that's what you use to identify the subjob.
  • OAR_JOB_WALLTIME and OAR_JOB_WALLTIME_SECONDS — the walltime of the job in the hh:mm:ss format and in seconds.
  • OAR_RESOURCE_PROPERTIES_FILE — the pathname to the file containing the list of all resources assigned for the job, and their value. See oarprint too.
  • OAR_PROJECT_NAME — the name of the project the job is part of, as given with the --project option.
  • OAR_STDOUT and OAR_STDERR — the pathname to the files storing the standard output and standard error of the job executable, if it's not running in interactive mode.
  • OAR_WORKING_DIRECTORY aka OAR_WORKDIR aka OAR_O_WORKDIR — the working directory for the job. The job executable will be executed in that directory, on the first node that is allocated to the job.
  • OAR_JOB_KEY_FILE — the key file to use for the submission (or for oarsh) if you're using the job-key mechanism (-k or --use-job-key option). You can also provide the job-key to import using the -i or --import-job-key-from-file option as well.

✨ SCRIPT ✨

When you submit a job using a script file, that script file can hold some extra OAR options. Those lines start with #OAR and use the same option syntax described above. 💅

🔥 EXAMPLES 🔥

🌈 Job submission with arguments 🌈

oarsub -l /nodes=4 -I

oarsub -q default -l /nodes=10/cpu=3,walltime=50:30:00 -p "switch = 'sw1'" /home/username/path/to/my/prog

oarsub -r "2009-04-27 11:00:00" -l /nodes=12/cpu=2

oarsub -C 154

🦋 Array job submission with 10 identical subjobs 🦋

oarsub -l /nodes=4 /home/usename/path/to/my/prog --array 10

🍓 Parametric array job submission 🍓

oarsub /home/users/toto/prog --array-param-file /home/username/path/to/params.txt

With /home/username/path/to/param.txt containing for instance:

# my param file
# a subjob with a single parameter
p100
# a subjob without parameter
""
# a subjob with two strings as parameters
"arg1a arg1b arg1c" "arg2a arg2b"

💜 Job submission with a job script 💜

oarsub -S /home/username/path/to/my/script.sh

With /home/username/path/to/my/script.sh containing for instance:

#!/bin/bash
#OAR -l /nodes=4/cpu=1,walltime=3:15:00
#OAR -p switch = 'sw3' or switch = 'sw5'
#OAR -t besteffort
#OAR -t type2
#OAR -k
#OAR -e /path/to/job/key
#OAR --stdout stdoutfile.log
/home/username/path/to/my/prog

👑 SEE ALSO 👑

🎀 COPYRIGHTS 🎀

Copyright 2003-2021 Laboratoire d'Informatique de Grenoble (http://www.liglab.fr). This software is licensed under the GNU General Public License Version 2 or above. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And the footer of this page says source/version oarsub(1), date 2025-03-24 — periodt. 💖✨

📦 where this page comes from