TOIL(1)

✨ NAME ✨

Hiii, welcome to the TOIL(1) manual 🎀 this one is for toil, aka Toil Documentation 💖

Toil is an open-source, pure-Python workflow engine that lets people write better pipelines ✨ no cap, it's giving main character energy 💅

Okay so, check out the website http://toil.ucsc-cgl.org/ for a full list of Toil's features 🌸 and read the paper http://biorxiv.org/content/early/2016/07/07/062497 to see what Toil can actually do in the real world 🌍. Please subscribe to the low-volume announce mailing list at https://groups.google.com/forum/#!forum/toil-announce 📬 and feel free to also join us on GitHub (https://github.com/BD2KGenomics/toil) and Gitter (https://gitter.im/bd2k-genomics-toil/Lobby) 🫶.

If you're using Toil for your research, bestie, please cite this 📚

Vivian, J., Rao, A. A., Nothaft, F. A., Ketchum, C., Armstrong, J., Novak, A., … Paten, B. (2017). Toil enables reproducible, open source, big biomedical data analyses. Nature Biotechnology, 35(4), 314–316. http://doi.org/10.1038/nbt.3772

---

✨ QUICKSTART EXAMPLES ✨

🌈 Running a basic CWL workflow

The Common Workflow Language (http://www.commonwl.org/), or CWL for short, is an emerging standard for writing workflows that are portable across multiple workflow engines and platforms 🦋. Running CWL workflows using Toil is easy, literally 💖.

1. Copy and paste the following code block into example.cwl:

cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
stdout: output.txt
inputs:
  message:
    type: string
    inputBinding:
      position: 1
outputs:
  output:
    type: stdout

and this code into example-job.yaml:

message: Hello world!

2. To run the workflow, simply enter:

$ toil-cwl-runner example.cwl example-job.yaml

Your output will be in output.txt:

$ cat output.txt
Hello world!

Congratulations, bestie! 🎉 You've run your first Toil workflow using the default Batch System (#batchsysteminterface), single_machine, and the default file job store (which was placed in a temporary directory for you by toil-cwl-runner) 👑.

Toil uses batch systems to manage the jobs it creates 🧁.

The single_machine batch system is mainly used to prep and debug workflows on a local machine 🔧. Once it's validated, try running them on a full-fledged batch system (see Batch System API (#batchsysteminterface)). Toil supports many different batch systems, such as Kubernetes (https://kubernetes.io/) and Grid Engine, so its versatility makes it easy to run your workflow in all kinds of places 🌸.

Toil's CWL runner is totally customizable! 💅 Run toil-cwl-runner --help to see a complete list of available options.

To learn more about CWL, see the CWL User Guide (https://www.commonwl.org/user_guide/), which is where this example was shamelessly borrowed from 🙈. For info on using CWL with Toil, check out the section CWL in Toil (#cwl). And for an example of CWL on an AWS cluster, have a look at Running a CWL Workflow on AWS ✨.

🧚 Running a basic WDL workflow

The Workflow Description Language (https://software.broadinstitute.org/wdl/), or WDL, is another emerging language for writing workflows that are portable across multiple workflow engines and platforms 🌷. Running WDL workflows using Toil is still in alpha and currently experimental 🧪. Toil currently supports basic workflow syntax (see WDL in Toil (#wdl) for more details and examples). Here we go over running a basic WDL helloworld workflow 💖.

1. Copy and paste the following code block into wdl-helloworld.wdl:

workflow write_simple_file {
  call write_file
}
task write_file {
  String message
  command { echo ${message} > wdl-helloworld-output.txt }
  output { File test = "wdl-helloworld-output.txt" }
}

and this code into wdl-helloworld.json:

{
  "write_simple_file.write_file.message": "Hello world!"
}

2. To run the workflow, simply enter:

$ toil-wdl-runner wdl-helloworld.wdl wdl-helloworld.json

Your output will be in wdl-helloworld-output.txt:

$ cat wdl-helloworld-output.txt
Hello world!

Just like the CWL example above, this will use the single_machine batch system and an automatically-located file job store by default 🫶. You can customize how Toil runs the workflow with command-line options. Run toil-wdl-runner --help to learn about them 🎀.

To learn more about WDL in general, see the Terra WDL documentation (https://support.terra.bio/hc/en-us/sections/360007274612-WDL-Documentation) 📖. For more on using WDL in Toil, see WDL in Toil (#wdl) 💜.

🐍 Running a basic Python workflow

Besides workflow languages like CWL and WDL, Toil also supports running workflows written against its Python API 🍓.

An example Toil Python workflow can be run with just three steps, bestie 🦋:

1. Install Toil (see Installation (#installation-ref)).
2. Copy and paste the following code block into a new file called helloWorld.py:

from toil.common import Toil
from toil.job import Job

def helloWorld(message, memory="1G", cores=1, disk="1G"):
    return f"Hello, world!, here's a message: {message}"

if __name__ == "__main__":
    parser = Job.Runner.getDefaultArgumentParser()
    options = parser.parse_args()
    options.clean = "always"
    with Toil(options) as toil:
        output = toil.start(Job.wrapFn(helloWorld, "You did it!"))
    print(output)

3. Specify the name of the job store (#jobstoreoverview) and run the workflow:

$ python3 helloWorld.py file:my-job-store

For something beyond a "Hello, world!" example, refer to Example: sorting 🔥.

Toil's customization options are available in Python workflows too. Run python3 helloWorld.py --help to see a complete list of available options 💅.

🧬 Example: mitochondrial variant calling

Let's run a more realistic workflow with Toil 🔥. This one is mitochondrial variant calling. It takes the human reference genome and sequenced reads from an individual, then figures out how that person's mitochondrial DNA differs from the reference genome 🧬.

We'll run an example workflow from Dockstore (https://dockstore.org/workflows/github.com/broadinstitute/gatk/MitochondriaPipeline:master?tab=info). First, grab an example workflow input:

(venv) $ wget https://toil-datasets.s3.us-west-2.amazonaws.com/MitochondriaInputs.zip && unzip MitochondriaInputs.zip

Then, change your current working directory:

(venv) $ cd MitochondriaInputs

This workflow will take approximately 30 minutes to run ⏳.

Since Toil supports Dockstore TRS IDs, which lets the WDL runner run any workflow on Dockstore, we'll run it directly 💖:

(venv) $ toil-wdl-runner '#workflow/github.com/broadinstitute/gatk/MitochondriaPipeline:master' -i ExampleInputsMitochondriaPipeline.json --logInfo --container docker --quantCheck false --outputFile mitochondria.json

Note:

• --logInfo — runs the workflow with INFO level logging. For different levels of logging, see --logLevel, --logCritical, --logError, --logWarning, --logDebug, and --logTrace 📝.
• --container docker — uses Docker as the container backend. By default, Toil will run with Singularity. To set it explicitly, use --container singularity 🐳.
• --outputFile — puts the workflow JSON outputs into a file. If you leave it out, Toil will put the workflow outputs onto the commandline 💌.
• --quantCheck false — disables certain type checks. This is useful for Cromwell compatibility 🙈.

⚠️ Unless fakeroot support is set up for Singularity, this particular workflow must be run with Docker, because it assumes commands in the container will run as root. Also, WDL workflows sometimes depend on non-spec compliant behaviors. To see if Toil has a workaround option, see Toil WDL Runner Options (#wdloptions) 💀.

Once the workflow is done running, you can look at your JSON output with jq . mitochondria.json 👀. For example, if we want the out_vcf output from the workflow, we can run jq -r '.["MitochondriaPipeline.out_vcf"]' mitochondria.json to get its path:

/private/groups/patenlab/toil-dev/mitochondria/wdl-out-c6o9mjop/MitochondriaPipeline.AlignAndCall.FilterContamination/HG02571.GRCh38.chrM.vcf

To open the VCF file, we can run less $(jq -r '.["MitochondriaPipeline.out_vcf"]' mitochondria.json) 🌈.

Note:

For outputs that aren't files, their values are right there in the JSON 🫶. For example, with jq '.["MitochondriaPipeline.median_coverage"]' mitochondria.json you can fetch the median_coverage output's value:

4183.5

Toil uses a jobstore to store all of a workflow's files and to communicate between workers 🗂️. If you don't specify one, Toil will use an ephemeral directory that gets deleted after Toil is done running. To control where those files go, or to allow a workflow to be restarted, you can use the --jobStore option. If you specify a jobstore explicitly, it will stick around if the workflow fails. To keep the jobstore after a successful completion, use --clean never. To remove the jobstore even after a failing run, use --clean always 🧹.

On a cluster, the jobstore must be somewhere accessible to all worker nodes 🌐. Here's an example of running the workflow with a specified jobstore:

(venv) $ toil-wdl-runner MitochondriaPipeline.wdl -i ExampleInputsMitochondriaPipeline.json --logInfo --container docker --outputFile mitochondria.json --jobstore mitochondriaJobstore

Toil supports several batch systems 🧁. By default, Toil uses single_machine, which runs everything on the local machine. Other batch systems are available too. For example, you can use --batchSystem slurm to run on a Slurm cluster 🚀:

(venv) $ toil-wdl-runner MitochondriaPipeline.wdl -i ExampleInputsMitochondriaPipeline.json --logInfo --container docker --outputFile mitochondria.json --jobStore mitochondriaJobstore --batchSystem slurm

See Running on Slurm (#runningslurm) for more information, including how to specify time limits and partitions ⏰.

Sometimes a workflow fails 😭. If that happens, the workflow can be restarted from the point of failure with --restart, as long as you still have the jobstore 💖:

(venv) $ toil-wdl-runner MitochondriaPipeline.wdl -i ExampleInputsMitochondriaPipeline.json --logInfo --container docker --outputFile mitochondria.json --jobStore mitochondriaJobstore --batchSystem slurm

🫧 Example: sorting

For a more detailed example and explanation, we've built a sample pipeline that merge-sorts a temporary file 🧚. This is not supposed to be an efficient sorting program. It's more of a fully worked example of what Toil is capable of 💜.

🎀 Running the example

1. Download the example code 📥.
2. Run it with the default settings:

$ python3 sort.py file:jobStore

Okay so the workflow just dropped a file called sortedFile.txt into your current directory 📂 Go peek at it, bestie, because it's literally full of sorted lines! 🤩💖

Lowkey the coolest part is that this workflow runs a smart merge sort on a file it generates, fileToSort.txt ✨ It's "smart" because every step (splitting the file into chunks 🔪, sorting those chunks 🧹, and merging them back together 🫶) gets its own little job. Each job can ask for its own resource requirements, and it only runs after the jobs it depends on have finished. Jobs with no dependencies? They run in parallel, no cap 🔥

💅 Note: Delete fileToSort.txt before moving on to #3. This example adds options that set the size of fileToSort.txt, but only if it doesn't already exist yet. If it does exist, the workflow just uses that file, and you'll get the same results as #2.

3. Run with custom options 🎀

$ python3 sort.py file:jobStore \
             --numLines=5000 \
             --lineLength=10 \
             --overwriteOutput=True \
             --workDir=/tmp/

Here we see that we can add our own options to a Toil Python workflow 🦋 As we mentioned above, the first two options, --numLines and --lineLength, decide how many lines there are and how many characters go in each line. --overwriteOutput makes the current contents of sortedFile.txt get overwritten, if it already exists. The last option, --workDir, is one that's built into Toil, and it's where the temporary files unique to each job are kept 🧚

---

🔍 Describing the source code 🔍

Want to know what's actually happening under the hood? 👀 Let's start with the main() function. It looks like a LOT of code, but don't worry, we'll break it down piece by piece 🧩

def main(options=None):
    if not options:
        # deal with command line arguments
        parser = ArgumentParser()
        Job.Runner.addToilOptions(parser)
        parser.add_argument(
            "--numLines",
            default=defaultLines,
            help="Number of lines in file to sort.",
            type=int,
        )
        parser.add_argument(
            "--lineLength",
            default=defaultLineLen,
            help="Length of lines in file to sort.",
            type=int,
        )
        parser.add_argument("--fileToSort", help="The file you wish to sort")
        parser.add_argument("--outputFile", help="Where the sorted output will go")
        parser.add_argument(
            "--overwriteOutput",
            help="Write over the output file if it already exists.",
            default=True,
        )
        parser.add_argument(
            "--N",
            dest="N",
            help="The threshold below which a serial sort function is used to sort file. "
            "All lines must of length less than or equal to N or program will fail",
            default=10000,
        )
        parser.add_argument(
            "--downCheckpoints",
            action="store_true",
            help="If this option is set, the workflow will make checkpoints on its way through"
            'the recursive "down" part of the sort',
        )
        parser.add_argument(
            "--sortMemory",
            dest="sortMemory",
            help="Memory for jobs that sort chunks of the file.",
            default=None,
        )

        parser.add_argument(
            "--mergeMemory",
            dest="mergeMemory",
            help="Memory for jobs that collate results.",
            default=None,
        )

        options = parser.parse_args()
    if not hasattr(options, "sortMemory") or not options.sortMemory:
        options.sortMemory = sortMemory
    if not hasattr(options, "mergeMemory") or not options.mergeMemory:
        options.mergeMemory = sortMemory

    # do some input verification
    sortedFileName = options.outputFile or "sortedFile.txt"
    if not options.overwriteOutput and os.path.exists(sortedFileName):
        print(
            f"Output file {sortedFileName} already exists.  "
            f"Delete it to run the sort example again or use --overwriteOutput=True"
        )
        exit()

    fileName = options.fileToSort
    if options.fileToSort is None:
        # make the file ourselves
        fileName = "fileToSort.txt"
        if os.path.exists(fileName):
            print(f"Sorting existing file: {fileName}")
        else:
            print(
                f"No sort file specified. Generating one automatically called: {fileName}."
            )
            makeFileToSort(
                fileName=fileName, lines=options.numLines, lineLen=options.lineLength
            )
    else:
        if not os.path.exists(options.fileToSort):
            raise RuntimeError("File to sort does not exist: %s" % options.fileToSort)

    if int(options.N) <= 0:
        raise RuntimeError("Invalid value of N: %s" % options.N)

    # Now we are ready to run
    with Toil(options) as workflow:
        sortedFileURL = "file://" + os.path.abspath(sortedFileName)
        if not workflow.options.restart:
            sortFileURL = "file://" + os.path.abspath(fileName)
            sortFileID = workflow.importFile(sortFileURL)
            sortedFileID = workflow.start(
                Job.wrapJobFn(
                    setup,
                    sortFileID,
                    int(options.N),
                    options.downCheckpoints,
                    options=options,
                    memory=sortMemory,
                )
            )
        else:
            sortedFileID = workflow.restart()
        workflow.exportFile(sortedFileID, sortedFileURL)

First we make a parser to process command line arguments using the argparse module (https://docs.python.org/2.7/library/argparse.html) 🛠️ It's really important that we add the call to Job.Runner.addToilOptions() so the parser gets initialized with all of Toil's default options. Then we add the command line arguments unique to this workflow, and parse the input. The help message listed with each argument should give you a pretty good idea of what it can do 💜

Next we do a little bit of input verification 🕵️‍♀️ The option --fileToSort lets you specify a file that needs to be sorted. If that option isn't given, this is where we make our own file, with the call to makeFileToSort().

Finally we get to the context manager that initializes the workflow 🎉 We create a path to the input file, prepended with file:// as per the documentation for toil.common.Toil() (#toil.common.Toil), when staging a file that is stored locally. Notice that we have to check whether or not the workflow is restarting, so that we don't import the file more than once. Then we can kick off the workflow by calling toil.common.Toil.start() (#toil.common.Toil.start) on the job setup. When the workflow ends, we capture its output (the sorted file's fileID) and use that in toil.common.Toil.exportFile() (#toil.common.Toil.exportFile) to move the sorted file from the job store back into "userland" 🏡

Next let's look at the job that begins the actual workflow, setup 🌸

def setup(job, inputFile, N, downCheckpoints, options):
    """
    Sets up the sort.
    Returns the FileID of the sorted file
    """
    RealtimeLogger.info("Starting the merge sort")
    return job.addChildJobFn(
        down,
        inputFile,
        N,
        "root",
        downCheckpoints,
        options=options,
        preemptible=True,
        memory=sortMemory,
    ).rv()

setup really only does two things 🙌 First it writes to the logs using Job.log(), and then it calls addChildJobFn(). Child jobs run directly after the current job. This function turns the 'job function' down into an actual job and passes in the inputs, including an optional resource requirement, memory. The job doesn't actually get run until the call to Job.rv(). Once the job down finishes, its output is returned here 🦋

Now we can look at what down does 👀

def down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory):
    """
    Input is a file, a subdivision size N, and a path in the hierarchy of jobs.
    If the range is larger than a threshold N the range is divided recursively and
    a follow on job is then created which merges back the results else
    the file is sorted and placed in the output.
    """

    RealtimeLogger.info("Down job starting: %s" % path)

    # Read the file
    inputFile = job.fileStore.readGlobalFile(inputFileStoreID, cache=False)
    length = os.path.getsize(inputFile)
    if length > N:
        # We will subdivide the file
        RealtimeLogger.critical(
            "Splitting file: %s of size: %s" % (inputFileStoreID, length)
        )
        # Split the file into two copies
        midPoint = getMidPoint(inputFile, 0, length)
        t1 = job.fileStore.getLocalTempFile()
        with open(t1, "w") as fH:
            fH.write(copySubRangeOfFile(inputFile, 0, midPoint + 1))
        t2 = job.fileStore.getLocalTempFile()
        with open(t2, "w") as fH:
            fH.write(copySubRangeOfFile(inputFile, midPoint + 1, length))
        # Call down recursively. By giving the rv() of the two jobs as inputs to the follow-on job, up,
        # we communicate the dependency without hindering concurrency.
        result = job.addFollowOnJobFn(
            up,
            job.addChildJobFn(
                down,
                job.fileStore.writeGlobalFile(t1),
                N,
                path + "/0",
                downCheckpoints,
                checkpoint=downCheckpoints,
                options=options,
                preemptible=True,
                memory=options.sortMemory,
            ).rv(),
            job.addChildJobFn(
                down,
                job.fileStore.writeGlobalFile(t2),
                N,
                path + "/1",
                downCheckpoints,
                checkpoint=downCheckpoints,
                options=options,
                preemptible=True,
                memory=options.mergeMemory,
            ).rv(),
            path + "/up",
            preemptible=True,
            options=options,
            memory=options.sortMemory,
        ).rv()
    else:
        # We can sort this bit of the file
        RealtimeLogger.critical(
            "Sorting file: %s of size: %s" % (inputFileStoreID, length)
        )
        # Sort the copy and write back to the fileStore
        shutil.copyfile(inputFile, inputFile + ".sort")
        sort(inputFile + ".sort")
        result = job.fileStore.writeGlobalFile(inputFile + ".sort")

    RealtimeLogger.info("Down job finished: %s" % path)
    return result

Okay so down is the recursive part of the workflow 💅 First we read the file into the local filestore by calling job.fileStore.readGlobalFile(). This drops a copy of the file into the temp directory for this particular job 📂, and that storage disappears once this job ends. For a detailed explanation of the filestore, the job store, and their interfaces, have a look at Managing files within a workflow (see #managingfiles).

Next, down checks the base case of the recursion: is the length of the input file less than N? (Remember, N was an option we added to the workflow in main.) In the base case, we just sort the file and return the file ID of this new sorted file ✨.

If the base case fails, the file gets split into two new tempFiles using job.fileStore.getLocalTempFile() and the helper function copySubRangeOfFile. Then we add a follow-on Job called up with job.addFollowOnJobFn(). We've already seen child jobs, but a follow-on Job is a job that runs after the current job and all of its children (and their children and follow-ons) have completed. A follow-on makes total sense here because up is responsible for merging the files together, and we don't want to merge anything until we know it's sorted 🫶. Again, the return value of the follow-on job is requested using Job.rv().

Looking at up 👀

def up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory):
    """
    Merges the two files and places them in the output.
    """

    RealtimeLogger.info("Up job starting: %s" % path)

    with job.fileStore.writeGlobalFileStream() as (fileHandle, outputFileStoreID):
        fileHandle = codecs.getwriter("utf-8")(fileHandle)
        with job.fileStore.readGlobalFileStream(inputFileID1) as inputFileHandle1:
            inputFileHandle1 = codecs.getreader("utf-8")(inputFileHandle1)
            with job.fileStore.readGlobalFileStream(inputFileID2) as inputFileHandle2:
                inputFileHandle2 = codecs.getreader("utf-8")(inputFileHandle2)
                RealtimeLogger.info(
                    "Merging %s and %s to %s"
                    % (inputFileID1, inputFileID2, outputFileStoreID)
                )
                merge(inputFileHandle1, inputFileHandle2, fileHandle)
        # Cleanup up the input files - these deletes will occur after the completion is successful.
        job.fileStore.deleteGlobalFile(inputFileID1)
        job.fileStore.deleteGlobalFile(inputFileID2)

        RealtimeLogger.info("Up job finished: %s" % path)

        return outputFileStoreID

Here we see the two input files getting merged together 🔥, with the output written to a brand new file using job.fileStore.writeGlobalFileStream(). After a little cleanup, the output file is returned.

Once the final up finishes and all of the rv() promises are fulfilled, main receives the sorted file's ID, which it uses in exportFile to send it to the user 🎀.

There are other things in this example that we didn't go over, like Checkpoints (see #checkpoints) and the details of much of the Toil Class API (see #api) 🧚.

At the end of the script, these lines:

if __name__ == '__main__'
    main()

are included to make sure the main function only runs once, in the '__main__' process that you, the user, invoked 👑. In Toil terms, by running the script you created the leader process, which is where main() runs. A worker process is a separate process whose only job (pun intended 💅) is hosting the execution of one or more jobs defined in that script. In any Toil workflow there is always one leader process and potentially many worker processes 🌈.

When you use the single-machine batch system (the default), the worker processes run on the same machine as the leader process. With full-fledged batch systems like Kubernetes, the worker processes will typically be started on separate machines 🦋. The boilerplate makes sure the pipeline only starts once (on the leader) and not when its job functions get imported and executed on the individual workers. Iconic behaviour, honestly 😭.

Typing python3 sort.py --help will show the complete list of arguments for the workflow, which includes both Toil's arguments and the ones defined inside sort.py. A complete explanation of Toil's arguments can be found in Commandline Options (see #commandref) 📖.

---

Logging 📜

By default, Toil logs a lot of information about the current environment, plus messages from the batch system and jobs 🙈. You can configure this with the --logLevel flag. For example, to only log CRITICAL level messages to the screen:

$ python3 sort.py file:jobStore \
             --logLevel=critical \
             --overwriteOutput=True

This hides most of the information we get from the Toil run 🤫. For more detail, run the pipeline with --logLevel=debug to see a comprehensive output. For more information, see Commandline Options (see #workflowoptions) 💜.

---

Error Handling and Resuming Pipelines 💀

With Toil, you can recover gracefully from a bug in your pipeline without losing any progress from jobs that already completed successfully 🫶. To show you how this works, let's add a bug to our example code and see how Toil handles the failure and how we can resume the pipeline afterwards. Add a bad assertion at line 52 of the example (the first line of down()):

def down(job, inputFileStoreID, N, downCheckpoints, memory=sortMemory):
    ...
    assert 1 == 2, "Test error!"

When we run the pipeline, Toil will show a detailed failure log with a traceback 😬:

$ python3 sort.py file:jobStore
...
---TOIL WORKER OUTPUT LOG---
...
m/j/jobonrSMP    Traceback (most recent call last):
m/j/jobonrSMP      File "toil/src/toil/worker.py", line 340, in main
m/j/jobonrSMP        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
m/j/jobonrSMP      File "toil/src/toil/job.py", line 1270, in _runner
m/j/jobonrSMP        returnValues = self._run(jobGraph, fileStore)
m/j/jobonrSMP      File "toil/src/toil/job.py", line 1217, in _run
m/j/jobonrSMP        return self.run(fileStore)
m/j/jobonrSMP      File "toil/src/toil/job.py", line 1383, in run
m/j/jobonrSMP        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
m/j/jobonrSMP      File "toil/example.py", line 30, in down
m/j/jobonrSMP        assert 1 == 2, "Test error!"
m/j/jobonrSMP    AssertionError: Test error!

If we try running the pipeline again, Toil will give us an error saying that a job store of the same name already exists. By default, when something fails, the job store is preserved so the workflow can be restarted from the previously failed jobs. We can restart the pipeline by running:

$ python3 sort.py file:jobStore \
             --restart \
             --overwriteOutput=True

We can also change the number of times Toil will attempt to retry a failed job:

$ python3 sort.py file:jobStore \
             --retryCount 2 \
             --restart \
             --overwriteOutput=True

You'll now see Toil attempt to rerun the failed job until it runs out of tries 🔁. --retryCount is useful for non-systemic errors, like downloading a file that might hit a sporadic interruption, or some other non-deterministic failure.

To successfully restart our pipeline, we can edit our script to comment out line 30, or remove it entirely, and then run:

$ python3 sort.py file:jobStore \
             --restart \
             --overwriteOutput=True

The pipeline will run successfully, and the job store will be removed on the pipeline's completion 🎉✨.

---

Collecting Statistics 📊

Please see the Status Command (see #cli-status) section for more on gathering runtime and resource info on jobs 💖.

---

Launching a Toil Workflow in AWS ☁️

After you've installed the aws extra for Toil during the Installation (see #installation-ref) and set up AWS (see Preparing your AWS environment, #prepareaws), you can run the basic helloWorld.py script (Running a basic Python workflow) on a VM in AWS just by modifying the run command 🦋🌸.

Okay so, on AWS you've got two options: run the workflow on a single instance, or run it on a cluster, which is basically a bunch of containers spread across multiple AWS instances 🌸. For the full tea on clusters, check out Running in AWS (#runningaws) 💅.

Also!! Don't forget the Destroy-Cluster Command (#destroycluster) when you're finished, bestie 🙈. Otherwise things may not get cleaned up properly 😭.

1. Launch a cluster in AWS with the Launch-Cluster Command (#launchcluster) ✨:

$ toil launch-cluster <cluster-name> \
             --clusterType kubernetes \
             --keyPairName <AWS-key-pair-name> \
             --leaderNodeType t2.medium \
             --nodeTypes t2.medium -w 1 \
             --zone us-west-2a

The arguments keyPairName, leaderNodeType, and zone are required to launch a cluster 🔥.

2. Copy helloWorld.py to the /tmp directory on the leader node using the Rsync-Cluster Command (#rsynccluster) 🦋:

$ toil rsync-cluster --zone us-west-2a <cluster-name> helloWorld.py :/tmp

Note that the command needs you to define the file you're copying, plus the target location on the cluster leader node 🎀.

3. Log in to the cluster leader node with the Ssh-Cluster Command (#sshcluster) 💜:

$ toil ssh-cluster --zone us-west-2a <cluster-name>

Note that this command will log you in as the root user 👑.

4. Run the workflow on the cluster 🚀:

$ python3 /tmp/helloWorld.py aws:us-west-2:my-S3-bucket

In this particular case, we create an S3 bucket called my-S3-bucket in the us-west-2 availability zone to store intermediate job results 🧚.

Along with some other INFO log messages, you should get the following output in your terminal window: Hello, world!, here's a message: You did it!. Slay 💖.

5. Exit from the SSH connection 👋.

$ exit

6. Use the Destroy-Cluster Command (#destroycluster) to destroy the cluster 🧹:

$ toil destroy-cluster --zone us-west-2a <cluster-name>

Note that this command will destroy the cluster leader node and any resources created to run the job, including the S3 bucket 💀.

---

✨ Running a CWL Workflow on AWS ✨

After you've installed the aws and cwl extras for Toil during the Installation (#installation-ref) and set up AWS (see Preparing your AWS environment (#prepareaws)), you can run a CWL workflow with Toil on AWS 🌈.

Also!! Remember to use the Destroy-Cluster Command (#destroycluster) when you're finished, or things may not be cleaned up properly 😭.

1. First, launch a node in AWS with the Launch-Cluster Command (#launchcluster) 🎀:

$ toil launch-cluster <cluster-name> \
             --clusterType kubernetes \
             --keyPairName <AWS-key-pair-name> \
             --leaderNodeType t2.medium \
             --nodeTypes t2.medium -w 1 \
             --zone us-west-2a

2. Copy example.cwl and example-job.yaml from the CWL example to the node with the Rsync-Cluster Command (#rsynccluster) 🦋:

toil rsync-cluster --zone us-west-2a <cluster-name> example.cwl :/tmp
toil rsync-cluster --zone us-west-2a <cluster-name> example-job.yaml :/tmp

3. SSH into the cluster's leader node with the Ssh-Cluster Command (#sshcluster) utility 💜:

$ toil ssh-cluster --zone us-west-2a <cluster-name>

4. Once you're on the leader node, command line tools like kubectl will be available to you 🧑‍💻. It's also a good idea to update and install the following 🫶:

sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y dist-upgrade
sudo apt-get -y install git

5. Now create a new virtualenv with the --system-site-packages option and activate it 🌸:

virtualenv --system-site-packages venv
source venv/bin/activate

6. Now run the CWL workflow with the Kubernetes batch system 🔥:

(venv) $ toil-cwl-runner \
             --provisioner aws \
             --batchSystem kubernetes \
             --jobStore aws:us-west-2:any-name \
             /tmp/example.cwl /tmp/example-job.yaml

Tip: When running a CWL workflow on AWS, input files can be provided either on the local file system or in S3 buckets using s3:// URI references. Final output files will be copied to the local file system of the leader node 🍓.

7. Finally, log out of the leader node and from your local computer, then destroy the cluster 💅:

$ toil destroy-cluster --zone us-west-2a <cluster-name>

---

✨ Running a Workflow with Autoscaling - Cactus ✨

Cactus (https://github.com/ComparativeGenomicsToolkit/cactus) is a reference-free, whole-genome multiple alignment program that can run on any of the cloud platforms Toil supports 🧬.

Note: Cloud Independence 🌈 This example gives you a "cloud agnostic" view of running Cactus with Toil. Most options won't change between cloud providers. However, each provisioner has unique inputs for --leaderNodeType, --nodeType and --zone. We recommend the following: - --leaderNodeType (used in launch-cluster) — AWS: t2.medium, Google: n1-standard-1 - --zone (used in launch-cluster) — AWS: us-west-2a, Google: us-west1-a - --zone (used in cactus) — AWS: us-west-2, Google: no value listed - --nodeType (used in cactus) — AWS: c3.4xlarge, Google: n1-standard-8 When executing toil launch-cluster with gce specified for --provisioner, the option --boto must be specified and given a path to your .boto file. See Running in Google Compute Engine (GCE) (#runninggce) for more information about the --boto option 🎀.

Also!! Remember to use the Destroy-Cluster Command (#destroycluster) when you're finished, or things may not be cleaned up properly 😭.

1. Download pestis.tar.gz 📦.
2. Launch a cluster with the Launch-Cluster Command (#launchcluster) 🚀:

$ toil launch-cluster <cluster-name> \
             --provisioner <aws, gce> \
             --keyPairName <key-pair-name> \
             --leaderNodeType <type> \
             --nodeType <type> \
             -w 1-2 \
             --zone <zone>

Note: A Helpful Tip 💡 When using AWS, setting the environment variable eliminates having to specify the --zone option for each command. This will be supported for GCE in the future. $ export TOIL_AWS_ZONE=us-west-2c

3. Create an appropriate directory for uploading files 📁:

$ toil ssh-cluster --provisioner <aws, gce> <cluster-name>
$ mkdir /root/cact_ex
$ exit

4. Copy the required files up to the leader node 🧁. That means seqFile.txt (a text file containing the locations of the input sequences as well as their phylogenetic tree, see here (https://github.com/ComparativeGenomicsToolkit/cactus#seqfile-the-input-file)), the organisms' genome sequence files in FASTA format, and configuration files (e.g. blockTrim1.xml, if desired):

$ toil rsync-cluster --provisioner <aws, gce> <cluster-name> pestis-short-aws-seqFile.txt :/root/cact_ex
$ toil rsync-cluster --provisioner <aws, gce> <cluster-name> GCF_000169655.1_ASM16965v1_genomic.fna :/root/cact_ex
$ toil rsync-cluster --provisioner <aws, gce> <cluster-name> GCF_000006645.1_ASM664v1_genomic.fna :/root/cact_ex
$ toil rsync-cluster --provisioner <aws, gce> <cluster-name> GCF_000182485.1_ASM18248v1_genomic.fna :/root/cact_ex
$ toil rsync-cluster --provisioner <aws, gce> <cluster-name> GCF_000013805.1_ASM1380v1_genomic.fna :/root/cact_ex
$ toil rsync-cluster --provisioner <aws, gce> <cluster-name> setup_leaderNode.sh :/root/cact_ex
$ toil rsync-cluster --provisioner <aws, gce> <cluster-name> blockTrim1.xml :/root/cact_ex
$ toil rsync-cluster --provisioner <aws, gce> <cluster-name> blockTrim3.xml :/root/cact_ex

5. Log in to the leader node 🔐:

$ toil ssh-cluster --provisioner <aws, gce> <cluster-name>

6. Set up the leader node's environment so it can run Cactus 🧪:

$ bash /root/cact_ex/setup_leaderNode.sh
$ source cact_venv/bin/activate
(cact_venv) $ cd cactus
(cact_venv) $ pip install --upgrade .

7. Run Cactus (https://github.com/ComparativeGenomicsToolkit/cactus) as an autoscaling workflow 🦋:

(cact_venv) $ cactus \
                  --retry 10 \
                  --batchSystem kubernetes \
                  --logDebug \
                  --logFile /logFile_pestis3 \
                  --configFile \
                  /root/cact_ex/blockTrim3.xml <aws, google>:<zone>:cactus-pestis \
                  /root/cact_ex/pestis-short-aws-seqFile.txt \
                  /root/cact_ex/pestis_output3.hal

Note: Pieces of the Puzzle 🧩 --logDebug — equivalent to --logLevel DEBUG. --logFile /logFile_pestis3 — writes logs in a file named logFile_pestis3 under / folder. --configFile — this is not required, depending on whether a specific configuration file is intended to run the alignment.

<aws, google>:<zone>:cactus-pestis 🪣 this one makes a bucket called cactus-pestis with your cloud provider of choice, and it stores the intermediate job files and metadata. 💅 NOTE: if you want a GCE-based jobstore, bestie, you type google here, NOT gce. 🙈

Your result file, pestis_output3.hal, gets saved under /root/cact_ex on the leader node. 🦋

Use cactus --help to see all the Cactus and Toil flags you can use. 🌸

8. Log out of the leader node: 🚪

(cact_venv) $ exit

9. Download the finished output to your local machine: 📥

(venv) $ toil rsync-cluster \
             --provisioner <aws, gce> <cluster-name> \
             :/root/cact_ex/pestis_output3.hal \
             <path-of-folder-on-local-machine>

10. Destroy the cluster, it's giving closure 💀✨

(venv) $ toil destroy-cluster --provisioner <aws, gce> <cluster-name>

---

✨ CWL IN TOIL ✨

okay so the Common Workflow Language (CWL) is an emerging standard for writing workflows that are portable across multiple workflow engines and platforms. 🌈 Toil is literally obsessed with it and has full support for the CWL v1.0, v1.1, and v1.2 standards. 👑

You can use Toil to run CWL workflows <#runcwl> or develop and test new ones. 🧚

---

✨ RUNNING CWL WORKFLOWS ✨

The toil-cwl-runner command handles CWL parsing using cwltool, and it borrows the job-scheduling and batch system support from Toil. 💖 You can use it to run CWL workflows locally or out in the cloud. ☁️

🎀 Running CWL Locally

To run in local batch mode, give it the CWL file and the input object file:

$ toil-cwl-runner example.cwl example-job.yml

For a simple example of CWL with Toil, check out Running a basic CWL workflow <#cwlquickstart>. 🍓

🙈 Note for macOS + Docker + Toil

Okay, so when you're invoking CWL documents that use Docker containers and you see errors that look like this 😭

docker: Error response from daemon: Mounts denied:
The paths /var/...tmp are not shared from OS X and are not known to Docker.

you may need to add this 👇

export TMPDIR=/tmp/docker_tmp

either in your startup file (.bashrc) or manually in your shell before you invoke toil. 💅

💅 Detailed Usage Instructions

Need help? Just run this toil command and you'll get all the info 🫶:

$ toil-cwl-runner -h

A more detailed example shows how you can specify both Toil and cwltool arguments for your workflow, and it's so good:

$ toil-cwl-runner \
    --singularity \
    --jobStore my_jobStore \
    --batchSystem lsf \
    --workDir `pwd` \
    --outdir `pwd` \
    --logFile cwltoil.log \
    --writeLogs `pwd` \
    --logLevel DEBUG \
    --retryCount 2 \
    --maxLogFileSize 20000000000 \
    --stats \
    standard_bam_processing.cwl \
    inputs.yaml

In this example we set the following options, which are all passed to Toil: ✨

• --singularity — tells Toil that every job with a Docker format container should run on the Singularity container engine instead of the Docker container engine. 🐝
• --jobStore — the path to a folder that doesn't exist yet. It'll hold the Toil jobstore and all the related job-tracking info. 📂
• --batchSystem — picks the HPC or Cloud-based cluster platform you want to use. ☁️
• --workDir — the directory where all the temporary files for the workflow get created. A subdirectory of this becomes the $TMPDIR environment variable, and you can reference that subdirectory with the CWL parameter reference $(runtime.tmpdir) in CWL tools and workflows. 🧸
• --outdir — the directory where your final File and Directory outputs get written. References to these and other output types show up in the JSON object printed to the stdout stream after the workflow runs. 🌷
• --logFile — the path to the main logfile. 📝
• --writeLogs — the directory where job logs get stored. At DEBUG log level, this includes logs for each Toil job run, plus stdout/stderr logs for each CWL CommandLineTool that didn't use the stdout/stderr directives to redirect output. 🔍
• --retryCount — how many times to retry each Toil job. 🔁
• --maxLogFileSize — logs that get bigger than this value get truncated. ✂️
• --stats — saves resource usages in json files, which you can collect with the toil stats command once the workflow is done. 📊

🌸 Extra Toil CWL Options

Besides the normal Toil options and the options cwltool supports, toil-cwl-runner adds a few of its own, bestie:

• --bypass-file-store — skips Toil's file store system and assumes all paths are accessible in place from all nodes. This can dodge possibly-redundant file copies into Toil's job store storage, and it's required for CWL's InplaceUpdateRequirement. ⚠️ BUT it lets a failed job execution leave behind a partially-modified state, which means a restarted workflow might not work correctly. 💀
• --reference-inputs — doesn't copy remote inputs into Toil's file store and assumes they're accessible in place from all nodes. 📌
• --disable-streaming — doesn't allow streaming of job input files. By default, files marked with streamable True get streamed from remote job stores. 🌊
• --cwl-default-ram — applies the CWL specification default ramMin. 🧠
• --no-cwl-default-ram — does NOT apply the CWL specification default ramMin, so Toil --defaultMemory kicks in instead. This can help jobs get to Slurm with no memory limit assigned. 🧁
• --cwl-min-ram BYTES — sets a minimum memory allocation for all tasks. If --no-cwl-default-ram is passed, this doesn't apply to tools that don't specify a memory requirement, and --defaultMemory gets used for those tools instead. 🎯

☁️ Running CWL in the Cloud

To run in cloud and HPC configurations, you may need to give additional command line parameters to select and configure the batch system you want to use. 🫧

To run a CWL workflow in AWS with toil, check out Running a CWL Workflow on AWS <#awscwl>. ✈️

🔥 Running CWL workflows with InplaceUpdateRequirement

Some CWL workflows use the InplaceUpdateRequirement feature, which requires that operations on files have visible side effects that Toil's file store can't support. 😤 If you need to run a workflow like this, make sure all your worker nodes have a shared filesystem, then use the --bypass-file-store option with toil-cwl-runner. This leaves all CWL intermediate files on disk and shares them between jobs using file paths, instead of storing them in the file store and downloading them whenever jobs need them. 🗂️

🍓 Toil & CWL Tips

🔎 See logs for just one job by using the full log file

This requires knowing the job's toil-generated ID, which you can find in the log files. 🪪

cat cwltoil.log | grep jobVM1fIs

🧵 Grep for full tool commands from toil logs

This gives you a more concise view of the commands being run. (Note that this info is only available from Toil when you run with --logDebug.) 💅

pcregrep -M "\[job .*\.cwl.*$\n(.*        .*$\n)*" cwltoil.log
#         ^allows for multiline matching

🎞️ Find Bams that have been generated for a specific step while the pipeline is running

find . | grep -P '^./out_tmpdir.*_MD\.bam$'

🧾 See what jobs have been run

cat log/cwltoil.log | grep -oP "\[job .*.cwl\]" | sort | uniq

or:

cat log/cwltoil.log | grep -i "issued job"

📡 Get status of a workflow

$ toil status /home/johnsoni/TEST_RUNS_3/TEST_run/tmp/jobstore-09ae0acc-c800-11e8-9d09-70106fb1697e
<hostname> 2018-10-04 15:01:44,184 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
<hostname> 2018-10-04 15:01:44,185 MainThread INFO toil.utils.toilStatus: Parsed arguments
<hostname> 2018-10-04 15:01:47,081 MainThread INFO toil.utils.toilStatus: Traversing the job graph gathering jobs. This may take a couple of minutes.

Of the 286 jobs considered, there are 179 jobs with children, 107 jobs ready to run, 0 zombie jobs, 0 jobs with services, 0 services, and 0 jobs with log files currently in file:/home/user/jobstore-09ae0acc-c800-11e8-9d09-70106fb1697e.

📊 Toil Stats

You can get run statistics broken down by CWL file. ✨ This only works once the workflow is finished:

$ toil stats /path/to/jobstore

This reports resource usage information for all the CWL jobs the workflow executed. See Stats Command <#cli-stats> for an explanation of what the different fields mean. 🧐

🪵 Understanding toil log files

There's a worker_log.txt file for each Toil job. It gets written to while the job is running, and it's uploaded at the end if the job finishes or if you're running at debug log level. If it gets uploaded, the contents are printed to the main log file and transferred to a log file in the --logDir folder. 🗃️

The new log file will be named something like:

CWLJob_<name of the CWL job>_<attempt number>.log

Standard output/error files will be named like:

<name of the CWL job>.stdout_<attempt number>.log

Say you have a workflow revsort.cwl with a step rev that calls the tool revtool.cwl. The CWL job name ends up being all those parts strung together with dots: revsort.cwl.rev.revtool.cwl. 🧩

---

✨ CWL CONFORMANCE ✨

⚠️ Note: To fully support inplace_update, --bypass-file-store should be enabled. See Running CWL workflows with InplaceUpdateRequirement <# inplaceupdaterequirement>.

The badges below show Toil's support for different aspects of the CWL specification. 🏅 They describe the current development version of Toil, not necessarily the release documented here. 💜

💖 CWL 1.2 Required Features

• [image: required] [image]

🌈 CWL 1.2 Optional Features

• [image: command_line_tool] [image]
• [image: conditional] [image]
• [image: docker] [image]
• [image: expression_tool] [image]
• [image: format_checking] [image]
• [image: initial_work_dir] [image]
• [image: inline_javascript] [image]
• [image: inplace_update] [image]
• [image: input_object_requirements] [image]
• [image: json_schema_invalid] [image]
• [image: load_listing] [image]
• [image: multiple_input] [image]
• [image: networkaccess] [image]
• [image: resource] [image]
• [image: scatter] [image]
• [image: schema_def] [image]
• [image: step_input] [image]
• [image: timelimit] [image]
• [image: workflow] [image]
• [image: work_reuse] [image]

✨ WDL IN TOIL ✨

okay so bestie, the Workflow Description Language (WDL) <https://openwdl.org/> is a programming language built for writing workflows 💅 It runs a whole set of tasks in a pipeline spread across multiple computers, and it's giving reproducibility 🧪 You wrap up an entire sequence of commands, where the outputs feed into the next commands, into one workflow that runs the exact same way every single time. no cap, that's how scientific analyses stay reproducible 🔁✨

Toil can run <#runwdl> WDL workflows and help you develop <#devwdl> them too 🫶 The Toil team also maintains a set of WDL conformance tests <#conformancewdl> for checking how well Toil and other WDL runners behave 🎀

---

🦋 RUNNING WDL WORKFLOWS 🦋

Toil has beta support for running WDL workflows through the toil-wdl-runner command 🌸 It comes with the [wdl] extra. If you don't have it yet, check out Installing Toil with Extra Features <#extras> to see how to install it 💖

You can run WDL workflows with toil-wdl-runner 💜 Under the hood, it uses MiniWDL <https://github.com/chanzuckerberg/miniwdl/#miniwdl> to parse and interpret the workflow, and it supports WDL 1.0 or later. Those workflows have to declare a version and use inputs and outputs sections, so keep that in mind bestie 🙈

Tip: The last release of Toil that supported unversioned, draft-2 WDL workflows was 5.12.0 <https://github.com/DataBiosphere/toil/releases/tag/releases%2F5.12.0> 🍓

Toil is a drop-in replacement for the Cromwell <https://github.com/broadinstitute/cromwell#readme> WDL runner, at least for compatible workflows ✨ So instead of running a workflow with Cromwell like this:

java -jar Cromwell.jar run myWorkflow.wdl --inputs myWorkflow_inputs.json

you can run the same workflow with toil-wdl-runner like this:

toil-wdl-runner myWorkflow.wdl --input myWorkflow_inputs.json

(We're using --input here with Toil, but it also accepts the Cromwell-style --inputs. iykyk 👑)

By default, this runs on the current machine, with a job store in an automatically determined temporary location. But you can add a few Toil options to use other Toil-supported batch systems, like Kubernetes 🧚

toil-wdl-runner --jobStore aws:us-west-2:wdl-job-store --batchSystem kubernetes myWorkflow.wdl --input myWorkflow_inputs.json

For Toil, --input is optional, and you can pass inputs as a positional argument instead 🦋

toil-wdl-runner myWorkflow.wdl myWorkflow_inputs.json

You can also run workflows straight from URLs! For example, to run the MiniWDL self test workflow, you can do this 🌈

toil-wdl-runner https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/self_test.wdl https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/inputs.json

---

💅 Toil WDL Runner Options 💅

• --jobStore — This is where Toil keeps its state info while the workflow runs. It has to be accessible from all machines, bestie 🏠
• -o, --outputDirectory — The folder or URI prefix where workflow output files get saved. It defaults to a new directory in the current directory 📂
• -m, --outputFile — A JSON file name or URI where workflow output values get saved. It defaults to standard output 🧾
• -i, --input, --inputs — An alternative to the positional argument for the input JSON file. Mostly here so it plays nice with other WDL runners 🤝
• --outputDialect — Picks an output format dialect. Use cromwell to just return the workflow's output values as JSON, or miniwdl to nest them under an outputs key and include a dir key 💁‍♀️
• --referenceInputs — Decides whether input files get passed around by URL reference instead of being imported into Toil's storage. It defaults to off, and it can be True or False or other similar words 🔗
• --container — Picks the container engine used to run tasks. By default it's auto, which tries Singularity if it's installed and Docker if it isn't. You can also set it to docker or singularity explicitly 🐳
• --allCallOutputs — Decides whether outputs from all calls in a workflow get included alongside the outputs from the output section, when one is defined. For strict WDL spec compliance, set it to False. It usually defaults to False. But if the workflow includes metadata for the Cromwell Output Organizer (croo) <https://github.com/ENCODE-DCC/croo>, it defaults to True 📦
• --strict — Decides whether Toil should immediately exit on a lint warning. By default it's false 🚨
• --quantCheck — Decides whether quantifier validation type checking should be enabled. Turning it off is useful for workflows that Cromwell can run but that don't fully comply with the WDL spec. By default it's true ✅
• --runImportsOnWorkers — Runs file imports on workers. This is handy if the leader isn't network optimized and a lot of downloads are needed. By default it's false 🏃‍♀️
• --importWorkersBatchSize — Requires --runImportsOnWorkers to be true. Sets the target batch size in bytes for batched imports. As many files as can fit go into each batch import job. This also accepts abbreviations, like G or Gi 📦
• --importWorkersDisk — Requires --runImportsOnWorkers to be true. Sets the disk size each import worker gets. You usually won't need to set this, because Toil will try to use file streaming when downloading files. If streaming isn't possible, for example when downloading from AWS to a GCE job store, set it to the largest file size of all the files to import. By default it's 1 MiB 💾

Any number of other Toil options may also be specified. For the defined Toil options, see Commandline Options <#commandref> 📖

---

🧾 Managing Workflow Logs 🧾

At the default settings, if a WDL task succeeds, its standard output and standard error get printed in the toil-wdl-runner output. The exception is when the workflow captures them with the stdout() and stderr() WDL built-in functions. If a WDL task fails, they get printed whether they were meant to be captured or not. Complete logs from Toil for failed jobs get printed too 😭

If you want to save logs organized by WDL task, use the --writeLogs or --writeLogsGzip options to specify a directory where the log files should go 🗂️ Log files get named after the same dotted, hierarchical workflow and task names used to set values from the input JSON. The only difference is that scatters add an extra numerical component. Toil job logs for failed jobs also show up here when you're running at the default log level 🍒

For example, if you run:

toil-wdl-runner --writeLogs logs https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/self_test.wdl https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/inputs.json

You'll end up with a logs/ directory containing:

hello_caller.0.hello.stderr_000.log
hello_caller.1.hello.stderr_000.log
hello_caller.2.hello.stderr_000.log

That final number is a sequential counter 🔢 If a step has to be retried, or if you run the workflow multiple times without clearing out the logs directory, it goes up. lowkey satisfying 🥰

---

🌷 Enabling WDL Call Cache 🌷

Toil can cache task and workflow outputs so it can reuse the outputs of tasks and workflows that already ran 🧠 This can save a ton of time when you're debugging a long running workflow where a later task fails. However, it's only guaranteed for running locally, and it can eat up a considerable amount of disk space 💀

To use it, set these environment variables before running the workflow:

export MINIWDL__CALL_CACHE__PUT=True
export MINIWDL__CALL_CACHE__GET=True
export MINIWDL__CALL_CACHE__DIR=/absolute_path/to/cache

The path to the cache directory has to be an absolute path, so don't skip that part 🫶

For setting up the call cache permanently, check out the MiniWDL call cache <https://miniwdl.readthedocs.io/en/latest/runner_reference.html#call-cache> documentation 🌈

✨ DEVELOPING WDL WORKFLOWS ✨

okay so Toil is a total dev tool for writing and testing WDL workflows locally 💖 Once your workflow slays on your machine, you can run it on Toil against a cloud or cluster backend, or take it to other WDL implementations like Terra (https://support.terra.bio/hc/en-us/sections/360004147011-Workflows), Cromwell (https://github.com/broadinstitute/cromwell#readme), or MiniWDL (https://github.com/chanzuckerberg/miniwdl/#miniwdl). Main character energy 👑

📖 Learning WDL

To learn how to write WDL workflows in general, you should take the WDL Tutorial (<#tutorialwdl>) 🎀

There are other tutorials in the resources section (<resourcesWdl>) 🌸

🐛 Debugging WDL Workflows

When a WDL workflow breaks, lowkey a lot of the general Toil debugging strategies (<debugging>) are super helpful, like finding failed jobs with toil status --failed or running individual jobs locally with toil debug-job. These are way easier if you use a manually specified job store by passing --jobStore to toil-wdl-runner, so you have the job store path handy for the debugging tools 🧰

There are also WDL-specific debugging strategies too 💅

📜 Reading WDL Workflow Logs

When a WDL workflow fails, you're likely to see a message like this 😭

WDL.runtime.error.CommandFailed: task command failed with exit status 1
[2023-07-16T16:23:54-0700] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host phoenix-15.prism

This means the command line command from one of your WDL tasks exited with a failing (aka nonzero) exit code. That happens either when the command line command is written wrong, or when the error detection code in the tool you're trying to run catches something and reports an error 🙈

Go up higher in the log until you find lines that look like:

[2024-01-16T20:12:19-0500] [Thread-3 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] hello_caller.0.hello.stderr follows:

And:

[2024-01-16T20:12:19-0500] [Thread-3 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] hello_caller.0.hello.stdout follows:

These will be followed by the standard error and standard output log data from the task's command. There may be useful info in there, like an error message from the underlying tool 🔍

If you want individual task logs saved separately for later reference, you can use the --writeLogs option to specify a directory to store them in. For more info, see Managing Workflow Logs (<#logging>) 📁

🗂️ Finding Uploaded Files

If you want to find files that were uploaded from a WDL job, look for lines like this in the job's debug log (<debuggingLog>):

[2023-07-16T15:58:39-0700] [MainThread] [D] [toil.wdl.wdltoil] Virtualized /data/tmp/2846b6012e3e5535add03b363950dd78/cb23/197c/work/bamPerChrs/Sample.chr14.bam as WDL file toilfile:2703483274%3A0%3Afiles%2Ffor-job%2Fkind-WDLTaskJob%2Finstance-b4c5x6hq%2Ffile-c4e4f1b16ddf4c2ab92c2868421f3351%2FSample.chr14.bam/Sample.chr14.bam

If you're using a file job store, you can grab that long URI:

toilfile:2703483274%3A0%3Afiles%2Ffor-job%2Fkind-WDLTaskJob%2Finstance-b4c5x6hq%2Ffile-c4e4f1b16ddf4c2ab92c2868421f3351%2FSample.chr14.bam/Sample.chr14.bam

Then URL-decode it with, for example, https://www.urldecoder.io/, and you'll get this:

toilfile:2703483274:0:files/for-job/kind-WDLTaskJob/instance-b4c5x6hq/file-c4e4f1b16ddf4c2ab92c2868421f3351/Sample.chr14.bam/Sample.chr14.bam

Then take the part after the last colon:

files/for-job/kind-WDLTaskJob/instance-b4c5x6hq/file-c4e4f1b16ddf4c2ab92c2868421f3351/Sample.chr14.bam/Sample.chr14.bam

That's the path relative to the job store directory where this file can be found ✨

Another approach is to use find -iname 'Sample.chr14.bam' on a file job store, if you know the name the file had when it was uploaded 🕵️

💡 Workflow Authoring Tips

Here are some tips for writing WDL workflows. You can also consult the OpenWDL Cookbook (https://github.com/openwdl/cookbook) 🍓

🗑️ Deleting Files

WDL doesn't have a built-in way to delete files. If you run a task that deletes a file, it will still exist in Toil's job store storage 💀

Toil recently gained support for deleting files at the end of WDL workflows. If you have a large file that you only need for part of your workflow, consider writing the part that creates and uses it as a separate child workflow and invoking it with call. Then the file gets cleaned up when the child workflow ends, leaving more space for files created in the parent workflow 🧹

🌍 External WDL Resources

Here are some other resources a WDL developer might find useful 🫶

🎓 Official WDL Tutorials

You can also learn to write WDL workflows for Toil by following the official WDL quickstart guide (https://docs.openwdl.org/getting-started/quickstart.html) ✏️

Once you have your workflow .wdl file and your input .json, you can run the workflow on the inputs with toil-wdl-runner:

toil-wdl-runner myWorkflow.wdl --input myWorkflow_inputs.json

🎬 Learn WDL Video Tutorials

For people who prefer video tutorials, Lynn Langit has a Learn WDL Video Course (https://www.youtube.com/playlist?list=PL4Q4HssKcxYv5syJKUKRrD8Fbd-_CnxTM) that will teach you how to write and run WDL workflows. The course is taught using Cromwell, but Toil should also be compatible with the course's workflows 📺

📚 OpenWDL Cookbook

The OpenWDL Cookbook (https://github.com/openwdl/cookbook) contains example solutions to a variety of tricky workflow authoring problems 🧠

📝 WDL Specifications

WDL language specifications can be found here: https://github.com/openwdl/wdl/blob/main/versions/1.1/SPEC.md

Toil is not yet fully conformant with the WDL specification (see WDL Conformance (<#conformancewdl>)), but it inherits most of the functionality of MiniWDL (https://github.com/chanzuckerberg/miniwdl/#miniwdl) 💜

🏫 UCSC Genomics Institute Wiki

The UCSC Genomics Institute (home of the Toil project) has tips on debugging WDL workflows with Toil (https://giwiki.gi.ucsc.edu/index.php?title=Phoenix_WDL_Tutorial#Debugging_Workflows). They're designed for one particular Slurm cluster environment, but might be applicable elsewhere 🌈

---

✨ WDL CONFORMANCE ✨

The Toil team maintains a set of WDL Conformance Tests (https://github.com/DataBiosphere/wdl-conformance-tests). Much like the CWL Conformance Tests (https://github.com/common-workflow-language/cwl-v1.2/blob/main/CONFORMANCE_TESTS.md) for CWL, the WDL Conformance Tests are useful for figuring out if a WDL implementation actually follows the WDL specification (https://github.com/openwdl/wdl/blob/cb875867d86f868fa08f6eb2be179a50097ba440/versions/1.1/SPEC.md) 🔥

The WDL Conformance Tests include a runner harness that can test toil-wdl-runner, as well as Cromwell and MiniWDL, and it supports testing conformance with the 1.1, 1.0, and draft-2 versions of WDL.

If you want to evaluate Toil's WDL conformance for yourself, first make sure you have toil-wdl-runner installed. It comes with the [wdl] extra; see Installing Toil with Extra Features (<#extras>) 🧩

Then, check out the test repository:

$ git clone https://github.com/DataBiosphere/wdl-conformance-tests
$ cd wdl-conformance-tests

Most tests need a Docker daemon available, so make sure yours is working properly:

$ docker info
$ docker run --rm docker/whalesay cowsay "Docker is working"

Then you can test toil-wdl-runner against a particular WDL spec version, say 1.1:

$ python3 run.py --runner toil-wdl-runner --versions 1.1

For any failed tests, the test number and the log of the failing test will be reported 🙈

After the tests run, you can clean up intermediate files with:

$ make clean

For more options, see:

$ python3 run.py --help

Or, consult the conformance test documentation (https://github.com/DataBiosphere/wdl-conformance-tests/tree/master#wdl-workflow-description-language-spec-conformance-tests) 📖

---

✨ WDL TUTORIAL ✨

This tutorial will walk you through writing a workflow in WDL that you can run with Toil (<runWdl>). We're going to write a workflow for Fizz Buzz (https://en.wikipedia.org/wiki/Fizz_buzz) 🎀

✍️ Writing the File

Let's go step by step through the process of creating a single-file WDL workflow 🦋

🏷️ Version

All WDL files need to start with a version statement (unless they're very old draft-2 files). Toil supports draft-2, WDL 1.0, and WDL 1.1, while Cromwell (another popular WDL runner used on Terra) supports only draft-2 and 1.0 💅

So let's start a new WDL 1.0 workflow. Open up a file named fizzbuzz.wdl and start with a version statement:

version 1.0

🧺 Workflow Block

Then, add an empty workflow named FizzBuzz.

version 1.0
workflow FizzBuzz {
}

🎛️ Input Block

Workflows usually need some kind of user input, so let's give our workflow an input section ✨

version 1.0
workflow FizzBuzz {
    input {
        # How many FizzBuzz numbers do we want to make?
        Int item_count
        # Every multiple of this number, we produce "Fizz"
        Int to_fizz = 3
        # Every multiple of this number, we produce "Buzz"
        Int to_buzz = 5
        # Optional replacement for the string to print when a multiple of both
        String? fizzbuzz_override
    }
}

Notice that each input has a type, a name, and an optional default value. If the type ends in ?, the value is optional, and it may be null. If an input is not optional and there is no default value, then the user's inputs file must specify a value for it in order for the workflow to run 🧐

🧱 Body

Now we'll start on the body of the workflow, to be inserted just after the inputs section 🛠️

The first thing we're going to need to do is create an array of all the numbers up to the item_count. We can do this by calling the WDL range() function and assigning the result to an Array[Int] variable.

Array[Int] numbers = range(item_count)

WDL 1.0 has a wide variety of functions in its standard library (https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#standard-library), and WDL 1.1 has even more 🤯

🔁 Scattering

Once we create an array of all the numbers, we can use a scatter to operate on each one. WDL does not have loops; instead it has scatters, which work a bit like map() in Python. The body of the scatter runs for each value in the input array, all in parallel. We're going to increment all the numbers, since FizzBuzz starts at 1 but WDL range() starts at 0 🌸

Array[Int] numbers = range(item_count)
scatter (i in numbers) {
    Int one_based = i + 1
}

💖 Conditionals 💖

Okay so inside the body of the scatter, we're gonna throw in some conditionals to decide whether we spit out "Fizz", "Buzz", or "FizzBuzz" 🙈. To support our fizzbuzz_override, we make an array of it plus a default value, then use the WDL select_first() function to grab the first non-null value in that array ✨.

Each run of a scatter is allowed to declare variables, and outside the scatter those variables get squished together into arrays of all the results 🧚. BUT each variable can only be declared once in the scatter, even with conditionals (rude, I know 💀). So we use select_first() at the end and lean on the fact that variables from conditionals that never ran are null 🦋.

Note that WDL has conditional expressions with a then and an else, but conditional statements only have a body, no else branch 😭. If you need an else, you'll have to check the negated condition instead 💅.

So first, let's handle the special cases 👑:

Array[Int] numbers = range(item_count)
scatter (i in numbers) {
    Int one_based = i + 1
    if (one_based % to_fizz == 0) {
        String fizz = "Fizz"
        if (one_based % to_buzz == 0) {
            String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
        }
    }
    if (one_based % to_buzz == 0) {
        String buzz = "Buzz"
    }
    if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
        # Just a normal number.
    }
}

🎀 Calling Tasks 🎀

Now for the normal numbers, we need to turn our number into a string 🍓. In WDL 1.1, and in WDL 1.0 on Cromwell, you can use a ${} substitution syntax in quoted strings anywhere, not just in command line commands. Toil technically will support this too, but it's not in the spec, and the tutorial needs an excuse for you to call a task 🙈. So we're gonna insert a call to a stringify_number task, which we'll write later.

To call a task (or another workflow), we use a call statement and give it some inputs. Then we can fish the output values out of the task with . access, only if we don't make a noise instead 🙊.

Array[Int] numbers = range(item_count)
scatter (i in numbers) {
    Int one_based = i + 1
    if (one_based % to_fizz == 0) {
        String fizz = "Fizz"
        if (one_based % to_buzz == 0) {
            String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
        }
    }
    if (one_based % to_buzz == 0) {
        String buzz = "Buzz"
    }
    if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
        # Just a normal number.
        call stringify_number {
            input:
                the_number = one_based
        }
    }
    String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string])
}

We can put the code into the workflow now, and set about writing the task 💖.

version 1.0
workflow FizzBuzz {
    input {
        # How many FizzBuzz numbers do we want to make?
        Int item_count
        # Every multiple of this number, we produce "Fizz"
        Int to_fizz = 3
        # Every multiple of this number, we produce "Buzz"
        Int to_buzz = 5
        # Optional replacement for the string to print when a multiple of both
        String? fizzbuzz_override
    }
    Array[Int] numbers = range(item_count)
    scatter (i in numbers) {
        Int one_based = i + 1

        if (one_based % to_fizz == 0) {
            String fizz = "Fizz"
            if (one_based % to_buzz == 0) {
                String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
            }
        }
        if (one_based % to_buzz == 0) {
            String buzz = "Buzz"
        }
        if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
            # Just a normal number.
            call stringify_number {
                input:
                    the_number = one_based
            }
        }
        String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
    }
}

🦋 Writing Tasks 🦋

Our task should go after the workflow in the file. It looks a lot like a workflow, except it uses task ✨.

task stringify_number {
}

We're gonna want it to take in an integer the_number, and we want it to output a string the_string. So let's fill that in with input and output sections 💅.

task stringify_number {
    input {
        Int the_number
    }
    # ???
    output {
        String the_string # = ???
    }
}

Now, unlike workflows, tasks can have a command section, which gives a command to run 🔥. This section is now usually set off with triple angle brackets, <<< and >>>, and inside it you can use ~{}, which is Bash-like substitution but with a tilde, to place WDL variables into your command script 🌸. So let's add a command that echoes back the number so we can see it as a string.

task stringify_number {
    input {
        Int the_number
    }
    command <<<
       # This is a Bash script.
       # So we should do good Bash script things like stop on errors
       set -e
       # Now print our number as a string
       echo ~{the_number}
    >>>
    output {
        String the_string # = ???
    }
}

Now we need to capture the result of the command script. The WDL stdout() function returns a WDL File containing the standard output printed by the task's command 😭. We want to read that back into a string, which we can do with the WDL read_string() function. It also removes trailing newlines, per https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#string-read_stringstringfile 🫶.

task stringify_number {
    input {
        Int the_number
    }
    command <<<
       # This is a Bash script.
       # So we should do good Bash script things like stop on errors
       set -e
       # Now print our number as a string
       echo ~{the_number}
    >>>
    output {
        String the_string = read_string(stdout())
    }
}

We're also gonna add a runtime section to our task to specify resource requirements 🧚. We're also telling it to run in a Docker container, to make sure absolutely nothing can go wrong with our delicate echo command 💀. In a real workflow you probably want to set up optional inputs for all the tasks so you can control the resource requirements, but here we'll just hardcode them 🙈.

task stringify_number {
    input {
        Int the_number
    }
    command <<<
        # This is a Bash script.
        # So we should do good Bash script things like stop on errors
        set -e
        # Now print our number as a string
        echo ~{the_number}
    >>>
    output {
        String the_string = read_string(stdout())
    }
    runtime {
        cpu: 1
        memory: "0.5 GB"
        disks: "local-disk 1 SSD"
        docker: "ubuntu:24.04"
    }
}

The disks section is a little weird 🥹. It isn't in the WDL spec, but Toil supports Cromwell-style strings that ask for a local-disk of a certain number of gigabytes, which may suggest that it be SSD storage.

Then we can put our task into our WDL file 💖.

version 1.0
workflow FizzBuzz {
    input {
        # How many FizzBuzz numbers do we want to make?
        Int item_count
        # Every multiple of this number, we produce "Fizz"
        Int to_fizz = 3
        # Every multiple of this number, we produce "Buzz"
        Int to_buzz = 5
        # Optional replacement for the string to print when a multiple of both
        String? fizzbuzz_override
    }
    Array[Int] numbers = range(item_count)
    scatter (i in numbers) {
        Int one_based = i + 1
        if (one_based % to_fizz == 0) {
            String fizz = "Fizz"
            if (one_based % to_buzz == 0) {
                String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
            }
        }
        if (one_based % to_buzz == 0) {
            String buzz = "Buzz"
        }
        if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
            # Just a normal number.
            call stringify_number {
                input:
                    the_number = one_based
            }
        }
        String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
    }
}
task stringify_number {
    input {
        Int the_number
    }
    command <<<
        # This is a Bash script.
        # So we should do good Bash script things like stop on errors
        set -e
        # Now print our number as a string
        echo ~{the_number}
    >>>
    output {
        String the_string = read_string(stdout())
    }
    runtime {
        cpu: 1
        memory: "0.5 GB"
        disks: "local-disk 1 SSD"
        docker: "ubuntu:24.04"
    }
}

👑 Output Block 👑

Now the only thing missing is a workflow-level output section. Technically, in WDL 1.0 you aren't supposed to need this, but you do need it in 1.1, and Toil doesn't actually send your outputs anywhere yet if you don't have one 😭. So we're making one. We need to collect together all the strings that came out of the different tasks in our scatter into an Array[String] ✨. We'll add the output section at the end of the workflow section, above the task 🫶.

version 1.0
workflow FizzBuzz {
    input {
        # How many FizzBuzz numbers do we want to make?
        Int item_count
        # Every multiple of this number, we produce "Fizz"
        Int to_fizz = 3
        # Every multiple of this number, we produce "Buzz"
        Int to_buzz = 5
        # Optional replacement for the string to print when a multiple of both
        String? fizzbuzz_override
    }
    Array[Int] numbers = range(item_count)
    scatter (i in numbers) {
        Int one_based = i + 1
        if (one_based % to_fizz == 0) {
            String fizz = "Fizz"
            if (one_based % to_buzz == 0) {
                String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
            }
         }
        if (one_based % to_buzz == 0) {
            String buzz = "Buzz"
        }
        if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
            # Just a normal number.
            call stringify_number {
                input:
                    the_number = one_based
            }
        }
        String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
    }
    output {
       Array[String] fizzbuzz_results = result
    }
}
task stringify_number {
    input {
        Int the_number
    }
    command <<<
        # This is a Bash script.
        # So we should do good Bash script things like stop on errors
        set -e
        # Now print our number as a string
        echo ~{the_number}
    >>>
    output {
        String the_string = read_string(stdout())
    }
    runtime {
        cpu: 1
        memory: "0.5 GB"
        disks: "local-disk 1 SSD"
        docker: "ubuntu:24.04"
    }
}

Okay so because result is defined inside a scatter, when we reference it outside the scatter it shows up as an array 🫶

🚀 Running the Workflow

Now all that's left is to actually run it, bestie 💅 First, make an inputs file to specify the workflow inputs:

echo '{"FizzBuzz.item_count": 20}' >fizzbuzz.json

Then run it with Toil. If you're on a Slurm cluster and you're currently in a shared directory that's available on all your nodes, you can run:

toil-wdl-runner --jobStore ./fizzbuzz_store --batchSystem slurm --slurmTime 00:10:00 --caching false --batchLogsDir ./logs fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json

Or if you want to run it locally, just do:

toil-wdl-runner fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json

🌸 Next Steps

• Try breaking your workflow up into multiple files and using import statements 🧚
• Publish your workflow on Dockstore: https://docs.dockstore.org/en/stable/getting-started/dockstore-workflows.html 🦋
• Read up on Toil-specific WDL development considerations in Developing WDL Workflows (#devwdl) 💜

---

✨ INTRODUCTION ✨

Toil runs basically everywhere, locally or in the cloud (#cloudoverview), specifically Amazon Web Services and Google Compute Engine. It also speaks two workflow DSLs, CWL (#cwl) and WDL (#wdl), plus plain Python workflows (see Developing a Python Workflow, #devpython) 🌈

Toil is built in a modular way, so it can run on lots of different systems with different configurations. The three configurable pieces are:

• Job Store 📦: a filepath or URL that hosts and centralizes all the files for a workflow, like a local folder or an AWS s3 bucket url.
• Batch System 🖥️: either a local single machine or one of the supported HPC environments (lsf, mesos, slurm, torque, htcondor, kubernetes, or grid_engine).
• Provisioner ☁️: only for running in the cloud. It decides which cloud provider supplies the instances that do the "work" of your workflow.

Job Store

The job store is a storage abstraction that holds all of the information used in a Toil run. It centralizes every file the jobs in the workflow use, plus the details of how far the run has progressed. If a workflow crashes or fails, the job store has everything it needs to resume with minimal repeated work 🔁

Several different job stores are supported, including the file job store and cloud job stores. For info on developing job stores, see Job Store API (#jobstoreinterface).

File Job Store

The file job store is for local use. It keeps the workflow info in a directory on the machine where the workflow is launched. It's the simplest and most convenient job store for testing or small runs 🎀

For an example that uses the file job store, see Running a basic CWL workflow (#quickstart).

Cloud Job Stores

Toil currently supports these cloud storage systems as job stores:

• AWS Job Store (#awsjobstore): an AWS S3 bucket formatted as aws:<zone>:<bucketname>, where only numbers, letters, and dashes are allowed in the bucket name. Example: aws:us-west-2:my-aws-jobstore-name.
• Google Job Store (#googlejobstore): a Google Cloud Storage bucket formatted as gce:<zone>:<bucketname>, where only numbers, letters, and dashes are allowed in the bucket name. Example: gce:us-west2-a:my-google-jobstore-name.

These use cloud buckets to house all of the files. This is super useful when several different worker machines are all running jobs and need to access the same job store 🫶

Batch System

A Toil batch system is either a local single machine (one computer) or a currently supported cluster of computers (lsf, mesos, slurm, torque, htcondor, or grid_engine). These environments manage individual worker nodes under a leader node to process the work a workflow needs. The leader and its workers all coordinate their tasks and files through one centralized job store location.

For a more detailed description of the different batch systems, or info on developing them, see Batch System API (#batchsysteminterface).

Provisioner

The Toil provisioner is a tool set for running a Toil workflow on a particular cloud platform ☁️

The Toil Cluster Utilities (#clusterutils) are the command line tools you use to provision nodes on your desired cloud platform. They let you launch nodes, ssh to the leader, and rsync files back and forth.

For detailed instructions on using the provisioner, see Running in AWS (#runningaws) or Running in Google Compute Engine (GCE) (#runninggce).

---

✨ COMMANDLINE OPTIONS ✨

Want to see every single one of Toil's commandline options? Quick way to do it is to run this on a workflow language front-end 👑

$ toil-wdl-runner --help

Or for a Toil Python workflow:

$ python3 example.py --help

For a basic Toil workflow, there's exactly one mandatory argument: the job store. Everything else is optional 💖

The Config File

Instead of typing out arguments on the command line every time, Toil lets you use a configuration file. Lowkey a lifesaver 😭

Options get applied with priority, in this order:

1. Command line options
2. Environmental Variables
3. Config file values
   • a. Provided config file through --config
   • b. Default config value in $HOME/.toil/default.yaml
4. Defaults

You can manually generate an example configuration file at any path you pick. To make one, run:

$ toil config [filename].yaml

Then uncomment whatever options you need and change or add values.

After editing the config file, you can run Toil with those settings by passing it on the command line:

$ python3 example.py --config=[filename].yaml

Alternatively, you can edit the default config file, which lives at $HOME/.toil/default.yaml 🌸

If you use CLI options in addition to the configuration file, the CLI options overwrite the configuration file options. For example:

$ python3 example.py --config=[filename].yaml --defaultMemory 80Gi

Heads up bestie ⚠️ this will result in a default memory per job of 80GiB no matter what's in the configuration file provided. The CLI wins every time.

The Job Store

Running Toil workflows requires a file path or URL pointing to a central location for all of the workflow's intermediate files: the job store 📦 For toil-cwl-runner and toil-wdl-runner, a job store can often be selected automatically, or you can specify one with the --jobStore option. Toil Python workflows generally need the job store as a positional command line argument.

To run the Python quickstart (#pyquickstart) example, say you're on a node with a big /scratch volume. You can have the job store created there by running python3 HelloWorld.py /scratch/my-job-store, or more explicitly, python3 HelloWorld.py file:/scratch/my-job-store ✨

Syntax for specifying different job stores:

• Local: file:job-store-name
• AWS: aws:region-here:job-store-name
• Google: google:projectID-here:job-store-name

Different types of job store options can be found below 🧚

Commandline Options

Core Toil Options

Options to specify the location of the Toil workflow and turn on stats collation about the performance of jobs 📊

• --workDir WORKDIR — Pick an absolute path to the directory where Toil's temporary files go. Standard output and error from batch system jobs land here too (unless --noStdOutErr is set), and a cache directory may live here as well. Temp files and folders go into a directory called toil-<workflowID> inside workDir. Toil generates the workflowID itself and reports it in the workflow logs. If you don't set this, the default comes from the TMPDIR, TEMP, and TMP variables via mkdtemp. For CWL, the temporary output directory is used instead (see CWL option --tmp-outdir-prefix). This directory needs to exist on all machines running jobs. If you want standard output and error from batch system jobs captured, it'll generally need to be on a shared file system. When sharing a cache between containers on a host, this directory must be shared between the containers too 💅
• --coordinationDir COORDINATION_DIR — Pick an absolute path to the directory where Toil keeps its state and lock files. When sharing a cache between containers on a host, this directory must be shared between the containers 🔒
• --noStdOutErr — Don't capture standard output and error from batch system jobs 🙈
• --stats — okay so this one saves stats about your toil workflow so you can pull them up later with toil stats 📊✨
• --clean=STATE — this decides whether the jobStore gets deleted when the program wraps up 🧹 Your choices are 'always', 'onError', 'never', or 'onSuccess'. Heads up, bestie: --stats needs info from the jobStore at the end, so with that flag on the jobStore is never deleted 🫶 Wanna be able to restart the run? Go with 'never' or 'onSuccess' 🔄 The default is 'never' if stats is enabled, and 'onSuccess' otherwise 💅
• --cleanWorkDir STATE — controls when the temporary worker directory gets cleaned up after a job finishes 🗑️ Choices: 'always', 'onError', 'never', or 'onSuccess'. Default = always.

⚠️ WARNING: this one is really only for debugging, babe. Running a full pipeline with it changed could fill your disk with intermediate data 💀

• --clusterStats FILEPATH — if enabled, it writes out JSON resource usage stats to a file 📝 The default spot is the current working directory, but you can pass an absolute path to pick exactly where the file lands 🗺️ This only applies when using scalable batch systems ✨
• --restart — if you pass this, toil will try to restart the existing workflow at the location pointed to by --jobStore 🔁 If the workflow doesn't exist, it'll raise an exception 🙅‍♀️

---

✨ Logging Options ✨

Okay so toil keeps stdout and stderr hidden by default 🙈 except when a job fails. Log levels follow priority from the logging module, so here's the vibe 📚

• --logOff — only CRITICAL log messages show up 🤫 Same as --logLevel=OFF or --logLevel=CRITICAL.
• --logCritical — only CRITICAL log messages show up 🚨 Same as --logLevel=OFF or --logLevel=CRITICAL.
• --logError — only ERROR and CRITICAL messages show up 😬 Equivalent to --logLevel=ERROR.
• --logWarning — WARN, ERROR, and CRITICAL messages show up ⚠️ Equivalent to --logLevel=WARNING.
• --logInfo — all the non-debugging messages show up 💁 Equivalent to --logLevel=INFO.
• --logDebug — DEBUG messages and everything above them show up 🔍 Equivalent to --logLevel=DEBUG.
• --logTrace — TRACE messages and everything above them show up 🦋 Equivalent to --logLevel=TRACE.
• --logLevel=LOGLEVEL — pick your level! It can be set to OFF (or CRITICAL), ERROR, WARN (or WARNING), INFO, DEBUG, or TRACE 🎀
• --logFile FILEPATH — the file path where the logging output gets written 📄
• --rotatingLogging — turns on rotating logging so your log files don't get too huge 🔄 (you set the size with --maxLogFileSize BYTESIZE).
• --maxLogFileSize BYTESIZE — the max size of a job log file you wanna keep, in bytes ✂️ Bigger log files get truncated down to the last X bytes. Setting this to zero means no truncation at all 🙌 Setting it to a negative value truncates from the beginning instead. Default=100MiB. This only sets the max size in bytes when --rotatingLogging is active.
• --log-dir DIRPATH — for CWL and local file system only 📂 Logs stdout and stderr (if the tool asks for stdout/stderr) into the DIRPATH.
• --logColors BOOL — turns colored logging on or off 🌈 Default=True.

---

✨ Batch System Options ✨

• --batchSystem BATCHSYSTEM — the type of batch system you wanna run your jobs with 🖥️ Default = single_machine.
• --disableAutoDeployment — this turns off auto-deployment of Toil Python workflows 🙅‍♀️ If True, the workflow's Python code has to be sitting in the same location on every single worker. Default = False.
• --maxJobs MAXJOBS — the max number of jobs you can submit to the backing scheduler at once 🚦 Not supported on Mesos or AWS Batch. Use 0 for unlimited. Defaults to unlimited.
• --maxLocalJobs MAXLOCALJOBS — the max number of housekeeping jobs that can run at the same time on your local system 🧹 Use 0 for unlimited. Defaults to the number of local cores.
• --manualMemArgs — skips the default arguments 'hv=MEMORY' & 'h_vmem=MEMORY' in the qsub call 🙅‍♀️ and relies on TOIL_GRIDGENGINE_ARGS to supply your own arguments instead. Requires that TOIL_GRIDGENGINE_ARGS be set.
• --memoryIsProduct — if your batch system understands requested memory as a product of the requested memory and the number of cores, set this flag so memory gets allocated properly 🧮 This is pretty common with grid engine clusters (Ex: SGE, PBS, Torque).
• --runCwlInternalJobsOnWorkers — decides whether CWL internal jobs (e.g. CWLScatter) run on the worker nodes instead of the primary node 🏃‍♀️ If false (default), all of those jobs run on the primary node. Setting this to true can speed things up for very large workflows with lots of sub-workflows and/or scatters, as long as your worker pool is big enough 🚀
• --statePollingWait STATEPOLLINGWAIT — how many seconds to wait before doing a scheduler query for job state ⏳ If you're still inside that waiting period, you get cached results. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf.
• --statePollingTimeout STATEPOLLINGTIMEOUT — how many seconds to keep retrying against a broken scheduler 🔧 Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf.
• --batchLogsDir BATCHLOGSDIR — the directory the backing batch system logs into 📒 It should be available on both the leader and the workers if the batch system writes logs to the worker machines' filesystems, which lots of HPC schedulers do. If unset, the Toil work directory gets used. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf.
• --mesosEndpoint MESOSENDPOINT — the host and port of your Mesos server, separated by a colon 🌐 (default: <leader IP>:5050)
• --mesosFrameworkId MESOSFRAMEWORKID — use a specific Mesos framework ID 🆔
• --mesosRole MESOSROLE — use a Mesos role 👑
• --mesosName MESOSNAME — the Mesos name to use 🏷️ (default: toil)
• --scale SCALE — a scaling factor that changes the value of all submitted tasks' submitted cores 🍓 Used in the single_machine batch system. It's super handy for running workflows on smaller machines than they were designed for, by setting a value less than 1. (default: 1)
• --slurmAllocateMem SLURM_ALLOCATE_MEM — if False, don't use --mem 🩹 This is a workaround for Slurm clusters that reject jobs with memory allocations.
• --slurmTime SLURM_TIME — the Slurm job time limit, in [DD-]HH:MM:SS format ⏰
• --slurmPartition SLURM_PARTITION — the partition you wanna send Slurm jobs to 🗂️
• --slurmGPUPartition SLURM_GPU_PARTITION — the partition to send Slurm jobs to if they ask for GPUs 🎮
• --slurmPE SLURM_PE — a special partition to send Slurm jobs to if they ask for more than 1 CPU 💜 Useful for Slurm clusters that don't offer a partition accepting both single-core and multi-core jobs.
• --slurmArgs SLURM_ARGS — extra arguments to pass along to Slurm ➕
• --kubernetesHostPath KUBERNETES_HOST_PATH — the path on Kubernetes hosts to use as a shared inter-pod temp directory 📁
• --kubernetesOwner KUBERNETES_OWNER — the username to mark Kubernetes jobs with 🏷️
• --kubernetesServiceAccount KUBERNETES_SERVICE_ACCOUNT — the service account to run jobs as 🪪
• --kubernetesPodTimeout KUBERNETES_POD_TIMEOUT — how many seconds to wait for a scheduled Kubernetes pod to start running ⌛ (default: 120s)
• --kubernetesPrivileged BOOL — whether Kubernetes pods are allowed to run as privileged 🔓 This can be used to enable FUSE mounts for faster runtimes with Singularity. When launching Toil-managed clusters, this gets set to true by --allowFuse. (default: False)
• --kubernetesPodSecurityContext KUBERNETES_POD_SECURITY_CONTEXT — the path to a YAML defining a pod security context to apply to all pods 📜
• --kubernetesSecurityContext KUBERNETES_SECURITY_CONTEXT — the path to a YAML defining a security context to apply to all containers 🛡️
• --awsBatchRegion AWS_BATCH_REGION — the AWS region holding the AWS Batch queue you're submitting to ☁️
• --awsBatchQueue AWS_BATCH_QUEUE — the name or ARN of the AWS Batch queue you're submitting to 📬
• --awsBatchJobRoleArn AWS_BATCH_JOB_ROLE_ARN — the ARN of an IAM role to run AWS Batch jobs as 🎭 so they can e.g. access a job store. It must be assumable by ecs-tasks.amazonaws.com.

---

✨ Data Storage Options ✨

Allows configuring Toil's data storage 🗄️

• --symlinkImports BOOL — with a filesystem-based job store, CWL input files get symlinked in by default 🔗 Setting this to True copies the files into the job store instead, which may protect them from being modified externally. When set to False and caching is enabled, Toil protects the file automatically by changing its permissions to read-only 🔒 (Default=True)
• --moveOutputs BOOL — with a filesystem-based job store, output files get moved to the output directory by default, and a symlink to the moved file gets created at the original spot 🚚 Setting this to True copies the files into the output directory instead. Applies to filesystem-based job stores only. (Default=False)
• --caching BOOL — turns worker level file caching on or off 💾 Set it to "true" if you want caching. By default, caching is enabled on supported batch systems. It doesn't affect CWL or WDL task caching.
• --symlinkJobStoreReads BOOL — bestie, this lets jobs read from a JobStore's shared filesystem, and do container mounts, directly through a symlink 🔗. If the shared filesystem can't handle every job hammering it at once, you can turn this off and use --caching=True so each node reads from its own local cache storage instead 🫶. (Default=True)

---

✨ Autoscaling Options ✨

This is where you set the minimum and maximum number of nodes in an autoscaled cluster, plus the knobs for how much it provisions 🎀.

• --provisioner CLOUDPROVIDER — the provisioner for cluster auto-scaling, and the main Toil --provisioner option 💅. It defaults to None for running on single_machine and non-auto-scaling batch systems. The currently supported choices are 'aws' or 'gce'.
• --nodeTypes NODETYPES — a list of comma-separated node types, where each one is made of slash-separated instance types 🦋. You can add an optional spot bid after a colon, which makes that node type preemptible. Instance types can show up in multiple node types, and the same node type can be both preemptible and non-preemptible, no cap.
  • Valid argument specifying two node types: c5.4xlarge/c5a.4xlarge:0.42,t2.large
  • Node types: c5.4xlarge/c5a.4xlarge:0.42 and t2.large
  • Instance types: c5.4xlarge, c5a.4xlarge, and t2.large
  • Semantics: bid $0.42/hour for either c5.4xlarge or c5a.4xlarge instances, treated interchangeably, while they're available at that price, and buy t2.large instances at full price 💸
• --minNodes MINNODES — the minimum number of nodes of each type in the cluster when auto-scaling 📝. Give it as a comma-separated list the same length as your list of node types. default=0
• --maxNodes MAXNODES — the maximum number of nodes of each type when autoscaling, also as a comma-separated list 🧚. If the list is shorter than the number of nodeTypes, the first value is used as the default. default=10
• --targetTime TARGETTIME — how rapidly you aim to finish jobs, in seconds 🔥. Shorter times mean more aggressive parallelization. The autoscaler scales up or down so it expects all queued jobs to finish within targetTime seconds. (Default: 1800)
• --betaInertia BETAINERTIA — a smoothing parameter that stops the number of provisioned nodes from wobbling around for no reason 🌊. It controls an exponentially weighted moving average of the estimated number of nodes. A value of 0.0 disables smoothing completely, and a value of 0.9 smooths so much that hardly any changes ever get made. Must be between 0.0 and 0.9. (Default: 0.1)
• --scaleInterval SCALEINTERVAL — the gap in seconds between checks on whether the cluster's size needs to change ⏰. (Default: 60)
• --preemptibleCompensation PREEMPTIBLECOMPENSATION — how much the autoscaler prefers swapping in non-preemptible nodes when preemptible nodes can't be started for some reason 🫶. Defaults to 0.0, and it must be between 0.0 and 1.0, inclusive. A value of 0.0 turns compensation off, a value of 0.5 covers two missing preemptible nodes with one non-preemptible node, and a value of 1.0 replaces every missing preemptible node with a non-preemptible one.
• --nodeStorage NODESTORAGE — the size of the root volume of worker nodes when they launch, in gigabytes 💾. Set this if your jobs need a lot of disk space. The default value is 50.
• --nodeStorageOverrides NODESTORAGEOVERRIDES — a comma-separated list of nodeType:nodeStorage pairs that override the --nodeStorage default for the specified nodeType(s) 🌈. This is super useful for heterogeneous jobs where some tasks need way more disk than others.
• --metrics — turns on the prometheus/grafana dashboard so you can watch CPU/RAM usage, queue size, and issued jobs 📊.
• --assumeZeroOverhead — ignores scheduler and OS overhead, and assumes jobs can use every last byte of memory and disk on a node when autoscaling 🙈.

---

✨ Service Options ✨

This lets you cap the maximum number of service jobs in a cluster. Keeping it limited means nodes don't get clogged up with services, which can cause deadlocks 🔒. (Not for CWL).

• --maxServiceJobs MAXSERVICEJOBS — the max number of service jobs that can run concurrently, not counting ones on preemptible nodes 🧁. default=9223372036854775807
• --maxPreemptibleServiceJobs MAXPREEMPTIBLESERVICEJOBS — the max number of service jobs that can run concurrently on preemptible nodes 🦋. default=9223372036854775807
• --deadlockWait DEADLOCKWAIT — how many seconds the workflow can run only the same service jobs, with no jobs to use them, before it's declared deadlocked and stopped 💀. default=60
• --deadlockCheckInterval DEADLOCKCHECKINTERVAL — how many seconds to wait between checks on whether the workflow is stuck running only service jobs with nothing to use them 🧠. It should be shorter than --deadlockWait. You might need to raise it if the batch system can't enumerate running jobs fast enough, or if polling for running jobs puts an unacceptable load on a shared cluster. default=30

---

✨ Resource Options ✨

These set the default cores/memory requirements (if the jobs don't specify their own), and let you limit the total memory/cores requested from the batch system 💪.

• --defaultMemory INT — the default amount of memory to request for a job 🧠. It only applies to jobs that don't set an explicit value for this requirement. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. Default is 2.0Gi
• --defaultCores FLOAT — the default number of CPU cores to dedicate to a job ⚡. It only applies to jobs without an explicit value for this requirement. Fractions of a core (like 0.1) work on some batch systems, namely Mesos and single_machine. Default is 1.0
• --defaultDisk INT — the default amount of disk space to dedicate to a job 💽. It only applies to jobs without an explicit value for this requirement. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. Default is 2.0Gi
• --defaultAccelerators ACCELERATOR — the default amount of accelerators to request for a job 🎮. It only applies to jobs without an explicit value for this requirement. Each accelerator spec can have a type (gpu [default], nvidia, amd, cuda, rocm, opencl, or a specific model like nvidia-tesla-k80), and a count [default: 1]. If you use both a type and a count, they must be separated by a colon. If you use multiple types of accelerators, separate the specs with commas. Default is [].
• --defaultPreemptible BOOL — makes all jobs able to run on preemptible (spot) nodes by default 🦋.
• --maxCores INT — the maximum number of CPU cores to request from the batch system at any one time 🚧. Standard suffixes like K, Ki, M, Mi, G or Gi are supported.
• --maxMemory INT — the maximum amount of memory to request from the batch system at any one time 🚧. Standard suffixes like K, Ki, M, Mi, G or Gi are supported.
• --maxDisk INT — the maximum amount of disk space to request from the batch system at any one time 🚧. Standard suffixes like K, Ki, M, Mi, G or Gi are supported.

---

✨ Options for rescuing/killing/restarting jobs ✨

These are for jobs that run too long, fail, or get lost, because some batch systems have issues! 🙃

• --retryCount INT — how many times to retry a failing job before giving up and labeling it failed 🔁. default=1
• --stopOnFirstFailure BOOL — stops the workflow at the first complete job failure 🛑.
• --enableUnlimitedPreemptibleRetries — if set, preemptible failures (or any failure caused by an instance getting unexpectedly terminated) won't count toward job failures or --retryCount 🌸.
• --doubleMem — if set, batch jobs that die from hitting the memory limit on batch schedulers get their memory doubled and are retried 💪. The remaining retry count is reduced by 1. Currently only supported by LSF. default=False.
• --maxJobDuration INT — the max runtime of a job in seconds before it gets killed ⏱️. This is a lower bound, so the actual time before the job is killed may be longer.
• --rescueJobsFrequency INT — how many seconds to wait between checks for missing or overlong jobs, meaning jobs the batch system lost 🕵️. Expert parameter.
• --jobStoreTimeout FLOAT — the max number of seconds to wait for a job's update to the job store before declaring it failed 🧃.

---

✨ Log Management Options ✨

• --maxLogFileSize MAXLOGFILESIZE — the max size of a job log file to keep, in bytes 📜. Log files bigger than this get truncated to the last X bytes. Setting this to zero prevents any truncation. Setting it to a negative value truncates from the beginning. Default=62.5 K
• --writeLogs FILEPATH — okay so this one takes the worker logs the leader collects and saves each into its own file at the path you give 📁. Any non-empty standard output and error from failed batch system jobs gets written into files at that path too 💀. No path? The current working directory is used 🌸. Ngl, by default the leader only gets logs back from failed jobs 🙈. If you want logs from successful jobs too, set the log level to 'debug' or turn on --writeLogsFromAllJobs, and tweak --maxLogFileSize to control the truncation limit for worker logs ✨.
• --writeLogsGzip FILEPATH — literally the same energy as --writeLogs, except the log files get gzipped on the leader 🎁.
• --writeMessages FILEPATH — the file where messages from the leader's message bus get sent 📨.
• --realTimeLogging BOOL — turns on real-time logging from workers to the leader ⚡.

🎀 Miscellaneous Options 🎀

• --disableChaining BOOL — turns off job chaining 🔗. Chaining is when one job reuses its resource allocation for its successor job, if that's possible 🫶.
• --disableJobStoreChecksumVerification — turns off checksum verification for files moved to and from the job store 🙅. Checksum verification is a safety check that makes sure your data doesn't get corrupted during transfer 🛡️. Currently only supported for non-streaming AWS files.
• --sseKey SSEKEY — give it the path to a file containing a 32 character key 🔐. That key gets used for server-side encryption on awsJobStore or googleJobStore. SSE won't be used at all if you don't pass this flag 💅.
• --setEnv NAME, -e NAME — NAME=VALUE, NAME, -e NAME=VALUE, or just NAME are all valid, bestie 💫. This sets an environment variable early on in the worker. If you leave out VALUE, it gets looked up in the current environment. Separately from this, the worker tries to emulate the leader's environment before running a job, except for some variables known to vary across systems 🌈. With this option, a variable can be injected straight into the worker process itself before it starts 🧚.
• --servicePollingInterval SERVICEPOLLINGINTERVAL — how long service jobs wait between checking whether the keep-alive flag exists ⏰ (default=60)
• --forceDockerAppliance — disables the sanity check for whether the docker image specified by TOIL_APPLIANCE_SELF exists 🐳. Toil uses that image to provision mesos for autoscaling, so this is the check you're skipping 🙈.
• --statusWait INT — seconds to wait between reports of running jobs ⏳ (default=3600)
• --disableProgress — turns off the progress bar that shows up when standard error is a terminal 🫠.
• --publishWorkflowMetrics PUBLISHWORKFLOWMETRICS — decides whether workflow metrics reports get published to Dockstore when a workflow completes 📊. Those reports include unique workflow and task run IDs, job names, and version and Toil feature use information. Selecting "current" publishes metrics for the current workflow. Selecting "all" also publishes prior workflow runs from the Toil history database, even if they were run with "no" 👑.

⚠️ Main character warning: once published, workflow metrics CANNOT be deleted or un-published. They will stay published forever! 😭

🔍 Debug Options 🔍

Debug options for finding problems or helping with testing 🕵️‍♀️.

• --debugWorker — an experimental no forking mode for local debugging 🧪. Workers aren't forked, and stderr/stdout aren't redirected to the log. (default=False)
• --disableWorkerOutputCapture — lets worker output go to the worker's standard out/error instead of per-job logs 📣.
• --badWorker BADWORKER — for testing purposes, this randomly kills a --badWorker proportion of jobs using SIGKILL 💀. (Default: 0.0)
• --badWorkerFailInterval BADWORKERFAILINTERVAL — when killing a job, it picks uniformly within the interval from 0.0 to --badWorkerFailInterval seconds after the worker starts 🎲. (Default: 0.01)
• --kill_polling_interval KILL_POLLING_INTERVAL — the interval (in seconds) the leader waits between polling for the kill flag inside the job store. That flag is set by the "toil kill" command 🪦. (default=5)

🔄 Restart Option 🔄

In the event of failure, Toil can resume the pipeline by adding the argument --restart and rerunning the workflow 💖. Toil Python workflows (but not CWL or WDL workflows) can even be edited and resumed, which is super useful for development or troubleshooting 🛠️.

🧸 Running Workflows with Services 🧸

Toil supports jobs, or clusters of jobs, that run as services to other accessor jobs. Example services include server databases or Apache Spark Clusters 🦋. Service jobs exist to provide services to accessor jobs, so their runtime depends on their accessor jobs running at the same time. Those dependencies can create potential deadlock scenarios 😵. That's when the workflow hangs because only service jobs are running, and their accessor jobs can't be scheduled because resources are too limited to run both at once. Toil tries to schedule services and accessors intelligently, but to avoid a deadlock when running service jobs, it's advisable to use these parameters:

• --maxServiceJobs: the maximum number of service jobs that can be run concurrently, excluding service jobs running on preemptible nodes.
• --maxPreemptibleServiceJobs: the maximum number of service jobs that can run concurrently on preemptible nodes.

If you set these so that at a maximum cluster size there are enough resources to run accessors alongside services, a deadlock can't happen 🙌.

If too low a limit is specified, a deadlock can occur. Toil can't schedule enough service jobs at once to finish the workflow 🥲. Toil will detect this if it happens and throw a toil.DeadlockException exception. Increasing the cluster size and these limits will fix it 💅.

🐍 Setting Options directly in a Python Workflow 🐍

It's good to remember that commandline options can be overridden in the code of a Python workflow 💜. For example, toil.job.Job.Runner.getDefaultOptions() can be used to get the default Toil options, ignoring what was passed on the command line. In this example, that's used to ignore command-line options and always run with the "./toilWorkflow" directory as the jobstore:

options = Job.Runner.getDefaultOptions("./toilWorkflow") # Get the options object

with Toil(options) as toil:
    toil.start(Job())  # Run the root job

However, each option can be explicitly set within the workflow by modifying the options object. In this example, we're setting logLevel = "DEBUG" (all log statements are shown) and clean="ALWAYS" (always delete the jobstore) like so:

options = Job.Runner.getDefaultOptions("./toilWorkflow") # Get the options object
options.logLevel = "DEBUG" # Set the log level to the debug level.
options.clean = "ALWAYS" # Always delete the jobStore after a run

with Toil(options) as toil:
    toil.start(Job())  # Run the root job

However, the usual incantation is to accept commandline args from the user with the following:

parser = Job.Runner.getDefaultArgumentParser() # Get the parser
options = parser.parse_args() # Parse user args to create the options object

with Toil(options) as toil:
    toil.start(Job())  # Run the root job

We can also have code in the workflow overwrite user supplied arguments:

parser = Job.Runner.getDefaultArgumentParser() # Get the parser
options = parser.parse_args() # Parse user args to create the options object
options.logLevel = "DEBUG" # Set the log level to the debug level.
options.clean = "ALWAYS" # Always delete the jobStore after a run

with Toil(options) as toil:
    toil.start(Job())  # Run the root job

---

🧰 TOIL UTILITIES 🧰

Toil includes some utilities for inspecting or manipulating workflows during and after their execution 🔎. (There are additional Toil Cluster Utilities available for working with Toil-managed clusters in the cloud ☁️.)

The generic toil subcommand utilities are:

• stats — reports runtime and resource usage for all jobs in a specified jobstore. The workflow must have originally been run using the --stats option 📈.
• status — inspects a job store to see which jobs have failed, run successfully, etc. 👀
• debug-job — runs a failing job on your local machine 🧑‍💻.
• clean — deletes the job store used by a previous Toil workflow invocation 🧹.
• kill — kills any running jobs in a rogue toil 🚨.

For information on a specific utility, run it with the --help option:

toil stats --help

📊 Stats Command 📊

To use the stats command, a workflow must first be run using the --stats option. Using this command makes certain that toil does not delete the job store, no matter what other options are specified. Normally the option --clean=always would delete the job store, but --stats overrides this 🙌.

🧪 Running an Example 🧪

We can run an example workflow and record stats:

python3 tutorial_stats.py file:my-jobstore --stats

Where tutorial_stats.py is the following:

import math
import time
from multiprocessing import Process

from toil.common import Toil
from toil.job import Job

def think(seconds):
    start = time.time()
    while time.time() - start < seconds:
        # Use CPU
        math.sqrt(123456)

class TimeWaster(Job):
    def __init__(self, time_to_think, time_to_waste, space_to_waste, *args, **kwargs):
        self.time_to_think = time_to_think
        self.time_to_waste = time_to_waste
        self.space_to_waste = space_to_waste
        super().__init__(*args, **kwargs)

    def run(self, fileStore):
        # Waste some space
        file_path = fileStore.getLocalTempFile()
        with open(file_path, "w") as stream:
            for i in range(self.space_to_waste):
                stream.write("X")

        # Do some "useful" compute
        processes = []
        for core_number in range(max(1, self.cores)):
            # Use all the assigned cores to think
            p = Process(target=think, args=(self.time_to_think,))
            p.start()
            processes.append(p)
        for p in processes:
            p.join()

        # Also waste some time
        time.sleep(self.time_to_waste)

def main():
    options = Job.Runner.getDefaultArgumentParser().parse_args()

    job1 = TimeWaster(0, 0, 0, displayName="doNothing")
    job2 = TimeWaster(10, 0, 4096, displayName="efficientJob")
    job3 = TimeWaster(10, 0, 1024, cores=4, displayName="multithreadedJob")
    job4 = TimeWaster(1, 9, 65536, displayName="inefficientJob")

    job1.addChild(job2)
    job1.addChild(job3)
    job3.addChild(job4)

    with Toil(options) as toil:
        if not toil.options.restart:
            toil.start(job1)
        else:
            toil.restart()

if __name__ == "__main__":
    main()

Notice the displayName key 👀, which can rename a job and give it an alias when it finally shows up in stats. So cute, right? 🎀

---

✨ Displaying Stats ✨

okay bestie, this part is my fave 💖. To see the runtime and resources each job used when it ran, just type:

toil stats file:my-jobstore

It should spit out something like this 📊:

Batch System: single_machine
Default Cores: 1  Default Memory: 2097152KiB
Max Cores: unlimited
Local CPU Time: 55.54 core·s  Overall Runtime: 26.23 s
Worker
    Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
        n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
        3 |     0.34   10.83   10.80   21.23   32.40 |     0.33   10.43   17.94   43.07   53.83 |     0.01    0.40   14.08   41.85   42.25 |  177168Ki 179312Ki 178730Ki 179712Ki 536192Ki |      0Ki     4Ki    22Ki    64Ki    68Ki
Job
 Worker Jobs  |     min    med    ave    max
              |       1      1 1.3333      2
    Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
        n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
        4 |     0.33   10.83    8.10   10.85   32.38 |     0.33   10.43   13.46   41.70   53.82 |     0.01    1.68    2.78    9.02   11.10 |  177168Ki 179488Ki 178916Ki 179696Ki 715664Ki |      0Ki     4Ki    18Ki    64Ki    72Ki
 multithreadedJob
    Total Cores: 4.0
    Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
        n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
        1 |    10.85   10.85   10.85   10.85   10.85 |    41.70   41.70   41.70   41.70   41.70 |     1.68    1.68    1.68    1.68    1.68 |  179488Ki 179488Ki 179488Ki 179488Ki 179488Ki |      4Ki     4Ki     4Ki     4Ki     4Ki
 efficientJob
    Total Cores: 1.0
    Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
        n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
        1 |    10.83   10.83   10.83   10.83   10.83 |    10.43   10.43   10.43   10.43   10.43 |     0.40    0.40    0.40    0.40    0.40 |  179312Ki 179312Ki 179312Ki 179312Ki 179312Ki |      4Ki     4Ki     4Ki     4Ki     4Ki
 inefficientJob
    Total Cores: 1.0
    Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
        n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
        1 |    10.38   10.38   10.38   10.38   10.38 |     1.36    1.36    1.36    1.36    1.36 |     9.02    9.02    9.02    9.02    9.02 |  179696Ki 179696Ki 179696Ki 179696Ki 179696Ki |     64Ki    64Ki    64Ki    64Ki    64Ki
 doNothing
    Total Cores: 1.0
    Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
        n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
        1 |     0.33    0.33    0.33    0.33    0.33 |     0.33    0.33    0.33    0.33    0.33 |     0.01    0.01    0.01    0.01    0.01 |  177168Ki 177168Ki 177168Ki 177168Ki 177168Ki |      0Ki     0Ki     0Ki     0Ki     0Ki

This report tells you about the resources your workflow used 📊🫶. Heads up though: right now it does NOT track CPU and memory used inside Docker containers 🐳💀, only Singularity containers.

There are three parts to this report 🎀.

✨ Overall Summary ✨

At the top is a section with the overall summary stats for the run 💅:

Batch System: single_machine
Default Cores: 1  Default Memory: 2097152KiB
Max Cores: unlimited
Local CPU Time: 55.54 core·s  Overall Runtime: 26.23 s

This lists some important settings for the Toil batch system that actually ran your jobs. It also gives you:

• 🧠 The CPU time used on the local machine, in core seconds. This counts the Toil leader itself (excluding some startup time) and any jobs running under the leader, which for the single_machine batch system means every job. It does not include CPU used by jobs that ran on other machines.
• ⏱️ The overall wall-clock runtime of the workflow in seconds, as measured by the leader.

These last two numbers skip some startup and shutdown time spent loading and saving files, so you may still want to use the time shell built-in to time your Toil runs overall 🕰️.

👑 Worker Summary

After the overall summary, there's a section with stats about the Toil worker processes that Toil used to run your workflow's jobs 🧚:

Worker
    Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
        n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
        3 |     0.34   10.83   10.80   21.23   32.40 |     0.33   10.43   17.94   43.07   53.83 |     0.01    0.40   14.08   41.85   42.25 |  177168Ki 179312Ki 178730Ki 179712Ki 536192Ki |      0Ki     4Ki    22Ki    64Ki    68Ki

• 🔢 Count: this shows that Toil had to submit 3 Toil worker processes to the backing scheduler to run this workflow. (In this case, it ran them all on the local machine.)
• ⏳ Real Time: stats about the wall clock times all the worker processes took. Every sub-column value is in seconds.
• 🧮 CPU Time: stats about the CPU usage of all the worker processes. Every sub-column value is in core seconds.
• 😴 CPU Wait: stats about CPU time that was reserved for worker processes but never actually used. Here, the max and total are pretty high compared to both real time and CPU time, which means a lot of reserved CPU went unused 😭. That can mean the workflow is overestimating how many cores it needs, that small jobs are running inside the same resource reservations as large jobs via chaining, or that the workflow is stuck waiting around for slow disk I/O.
• 🧠 Memory: the peak memory usage of each worker process and its child processes.
• 💾 Disk: the disk usage in each worker. It's polled at the end of each job the worker runs, so it may not always show the actual peak disk usage.

🦋 Job Breakdown

Finally, there's the breakdown of resource usage by jobs. It starts with a table summarizing how many jobs ran on each worker:

Job
 Worker Jobs  |     min    med    ave    max
              |       1      1 1.3333      2

In this example, most workers ran one job each, but one worker managed to run two jobs through chaining 🔗. (Jobs chain when a job has only one dependent job, which in turn depends only on that first job, and the second job needs no more resources than the first job did.)

Next up, we have stats for resource usage across all jobs together 💜:

Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
    n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
    4 |     0.33   10.83    8.10   10.85   32.38 |     0.33   10.43   13.46   41.70   53.82 |     0.01    1.68    2.78    9.02   11.10 |  177168Ki 179488Ki 178916Ki 179696Ki 715664Ki |      0Ki     4Ki    18Ki    64Ki    72Ki

And finally, for each kind of job (as determined by the job's displayName), we get stats summarizing the resources used by the instances of that kind of job 🌸:

multithreadedJob
   Total Cores: 4.0
   Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
       n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
       1 |    10.85   10.85   10.85   10.85   10.85 |    41.70   41.70   41.70   41.70   41.70 |     1.68    1.68    1.68    1.68    1.68 |  179488Ki 179488Ki 179488Ki 179488Ki 179488Ki |      4Ki     4Ki     4Ki     4Ki     4Ki
efficientJob
   Total Cores: 1.0
   Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
       n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
       1 |    10.83   10.83   10.83   10.83   10.83 |    10.43   10.43   10.43   10.43   10.43 |     0.40    0.40    0.40    0.40    0.40 |  179312Ki 179312Ki 179312Ki 179312Ki 179312Ki |      4Ki     4Ki     4Ki     4Ki     4Ki
inefficientJob
   Total Cores: 1.0
   Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
       n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
       1 |    10.38   10.38   10.38   10.38   10.38 |     1.36    1.36    1.36    1.36    1.36 |     9.02    9.02    9.02    9.02    9.02 |  179696Ki 179696Ki 179696Ki 179696Ki 179696Ki |     64Ki    64Ki    64Ki    64Ki    64Ki
doNothing
   Total Cores: 1.0
   Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
       n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
       1 |     0.33    0.33    0.33    0.33    0.33 |     0.33    0.33    0.33    0.33    0.33 |     0.01    0.01    0.01    0.01    0.01 |  177168Ki 177168Ki 177168Ki 177168Ki 177168Ki |      0Ki     0Ki     0Ki     0Ki     0Ki

okay so for each job we slap its name up top 💅, then the total cores it asked for (summed across every single instance of it 🫶), and then the stats table. ✨

Here the * marker in the table headers is the main character 👑. It shows that jobs are sorted by the median of the real time used. You can switch that up with the --sortCategory option 🦋.

The columns mean the same thing as they did for the workers, so here's the tea:

• Count — how many jobs of each type ran 🌸
• Real Time — stats on the wall clock times that instances of the job type took. All the sub-column values are in seconds ⏱️
• CPU Time — stats on the CPU usage amounts of each job. Lowkey iconic: multithreadedJob managed to use CPU time at faster than one core second per second 🔥, because it reserved multiple cores and ran multiple threads
• CPU Wait — stats on CPU time that was reserved for but not consumed by jobs. Not me noticing inefficientJob used hardly any of the cores it requested for most of its real time 💀
• Memory — the peak memory usage of each job 🧠
• Disk — the disk usage at the end of each job. It may not always reflect the actual peak disk usage 🙈

✨ Example Cleanup ✨

Once we're done looking at the stats, we can clean up the job store by running:

toil clean file:my-jobstore

💖 Status Command 💖

Continuing the example from the stats section above, if we ran our workflow with the command:

python3 tutorial_stats.py file:my-jobstore --stats

we can poke at our jobstore with the status command, for example:

toil status file:my-jobstore

If the run was successful, this won't return much valuable info, something like:

2018-01-11 19:31:29,739 - toil.lib.bioio - INFO - Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
2018-01-11 19:31:29,740 - toil.utils.toilStatus - INFO - Parsed arguments
2018-01-11 19:31:29,740 - toil.utils.toilStatus - INFO - Checking if we have files for Toil
The root job of the job store is absent, the workflow completed successfully.

Otherwise, toil status will return something like the following:

Of the 3 jobs considered, there are 1 completely failed jobs, 1 jobs
with children, 2 jobs ready to run, 0 zombie jobs, 0 jobs with
services, 0 services, and 0 jobs with log files currently in
FileJobStore(/Users/anovak/workspace/toil/tree).

The toil status command supports several useful flags, including:

• --perJob to get per-job status information 🎀
• --logs to print stored worker logs 📜
• --failed to list all failed jobs in the workflow ❌

For more info, run toil status --help 🫶.

One use case of toil status is the --printStatus argument. Running toil status --printStatus file:my-jobstore at any point in the workflow's lifecycle can tell you the progress of the workflow. Note: this command will output all current running jobs but not any finished or failed jobs 🦋.

For example, after running workflow.py in another terminal:

$ toil status --printStatus file:my-jobstore
[2024-05-31T13:59:13-0700] [MainThread] [I] [toil.utils.toilStatus] Traversing the job graph gathering jobs. This may take a couple of minutes.
Of the 2 jobs considered, there are 0 completely failed jobs, 1 jobs with children, 1 jobs ready to run, 0 zombie jobs, 0 jobs with services, 0 services, and 0 jobs with log files currently in FileJobStore(/path/to/my-jobstore).

Message bus path:  /tmp/tmp9cnaq3bm
Job ID kind-TimeWaster/instance-zvdsdkm_ with name TimeWaster is running on SingleMachineBatchSystem as ID 101349.
Job ID kind-TimeWaster/instance-7clm8cv2 with name TimeWaster is running on SingleMachineBatchSystem as ID 101350.

At this exact moment, two jobs named "TimeWaster" are running on my local machine 🧚

🧹 Clean Command 🧹

If a Toil pipeline didn't finish successfully, or it was run using --clean=always or --stats, the job store will stick around until it's deleted. toil clean <jobStore> makes sure all the artifacts associated with a job store are removed 🫧. This is especially useful for deleting AWS job stores, which reserve an SDB domain as well as an S3 bucket.

The deletion of the job store can be changed with the --clean argument. It can be set to always, onError, never, or onSuccess (default) 💅.

Temporary directories where jobs are running can also be saved from deletion using --cleanWorkDir, which has the same options as --clean.

⚠️ Only use --cleanWorkDir when debugging, bestie. Intermediate jobs will fill up disk space 💀

🐛 Debug Job Command 🐛

If a Toil workflow fails and it wasn't run with --clean=always, the failing job will be waiting in the job store to be debugged 🔍. (With WDL or CWL workflows, you may have needed to manually set a --jobStore location you can find again.)

You can use toil debug-job on a job in the job store to run it on your local machine and reproduce any error that may have happened during a remote workflow 🎀.

The toil debug-job command takes a job store, plus the ID or name of a job in it. If multiple jobs match a job name, and only one seems to have run out of retries and completely failed, it'll run that one 👑.

You can also pass the --printJobInfo flag to dump information about the job instead of running it 📋.

🛑 Kill Command 🛑

To kill all currently running jobs for a given jobstore, use the command:

toil kill file:my-jobstore

🌸 TOIL DEBUGGING 🌸

Toil has a bunch of tools to help with debugging, and we're about to go through them together 🫶. Here we get into how to work through potential problems a user might run into while trying to run a workflow.

🏃 Failing Fast 🏃

If you have a workflow you're testing, you can pass --stopOnFirstFailure=True to have Toil stop the workflow as soon as any job is completely failed (i.e. it runs out of retry attempts) 🛑. If you don't want Toil to retry failing jobs at all, pass --retryCount=0 🙅‍♀️.

📜 Reading the Log 📜

Usually, at the end of a failed Toil workflow, Toil will reproduce the job logs for the jobs that failed. You can look at the end of your workflow log and use those job logs to figure out which jobs are failing and why 🕵️‍♀️.

🔊 Using DEBUG Logging 🔊

When you're debugging a broken workflow, run it with --logDebug to set the log level to DEBUG 🧠.

When debug logging is on, the log from every Toil job gets inserted into the main Toil log between these markers:

=========>
     Toil job log is here
<=========

Normally, only the logs of failing jobs and the output of commands run from WDL get reproduced like this 💅.

🔎 Finding Failed Jobs in the Jobstore 🔎

The toil status command (see Status Command above) can be used with the --failed option to list all failed jobs in a Toil job store 📋.

You can also use it with the --logs option to retrieve per-job logs from the job store, for failed jobs that left them behind. These logs might be super useful for diagnosing and fixing the problem 🩺.

🧪 Running a Job Locally 🧪

If you have a failing job's ID or name, you can reproduce its failure on your local machine with toil debug-job. See Debug Job Command above 🫧.

For example, say you have this WDL workflow in test.wdl. This workflow cannot succeed, due to the typo in the echo command:

version 1.0
workflow test {
  call hello
}
task hello {
  input {
  }
  command <<<
    set -e
    echoo "Hello"
  >>>
  output {
  }
}

You could try to run it with:

toil-wdl-runner --jobStore ./store test.wdl --retryCount 0

But it will fail 💀.

If you want to reproduce the failure later, or on another machine, you can first find out what jobs failed with toil status:

toil status --failed --noAggStats ./store

This will produce something like:

[2024-03-14T17:45:15-0400] [MainThread] [I] [toil.utils.toilStatus] Traversing the job graph gathering jobs. This may take a couple of minutes.
Failed jobs:
'WDLTaskJob' test.hello.command kind-WDLTaskJob/instance-r9u6_dcs v6

And we can see a failed job with the display name test.hello.command 🔍. That tells you exactly where the job lives in the WDL workflow: the command section of the hello task, called from the test workflow. (If you're writing a Toil Python script, this is the job's displayName 💅.) We can rerun that job locally by name with:

toil debug-job ./store test.hello.command

If there were multiple failed jobs with that name (maybe because of a WDL scatter 🙈), we'd have to pick one by Toil job ID instead:

toil debug-job ./store kind-WDLTaskJob/instance-r9u6_dcs

And if there's only one failed WDL task, we can just tell Toil to rerun the failed WDLTaskJob by its Python class name:

toil debug-job ./store WDLTaskJob

Any of these will run the job (containers included!) on your local machine, where you can watch it live or poke at it with a debugger 🔥.

---

✨ Fetching Job Inputs ✨

The --retrieveTaskDirectory option to toil debug-job is so useful 😭. It lets you send a job's input files to a directory and then stops the job from running. It works for CWL and WDL jobs, and for Python workflows that call toil.job.Job.files_downloaded_hook() after downloading their files. It makes the worker work in the directory you specify, so the job's temporary directory ends up at worker/job inside it. For WDL and CWL jobs that mount files into containers, there's also an inside directory full of symlinks to the files as they'd look from the root of the container's filesystem 🧚.

For example, say you have a broken WDL workflow named example_alwaysfail_with_files.wdl, like this:

version 1.0
workflow test {
    call make_file as f1
    call make_file as f2
    call hello {
        input:
            name_file=f1.out,
            unused_file=f2.out
    }
}
task make_file {
    input {
    }
    command <<<
        echo "These are the contents" >test.txt
    >>>
    output {
        File out = "test.txt"
    }
}
task hello {
    input {
        File name_file
        File? unused_file
    }
    command <<<
        set -e
        echoo "Hello" "$(cat ~{name_file})"
    >>>
    output {
        File out = stdout()
    }
}

You can try and fail to run it like this:

toil-wdl-runner --jobStore ./store example_alwaysfail_with_files.wdl --retryCount 0

If you dump the files from the failing job:

toil debug-job ./store WDLTaskJob --retrieveTaskDirectory dumpdir

You'll end up with a directory tree that, according to tree, looks something like this:

dumpdir
├── inside
│   └── mnt
│       └── miniwdl_task_container
│           └── work
│               └── _miniwdl_inputs
│                   ├── 0
│                   │   └── test.txt -> ../../../../../../worker/job/2c6b3dc4-1d21-4abf-9937-db475e6a6bc2/test.txt
│                   └── 1
│                       └── test.txt -> ../../../../../../worker/job/e3d724e1-e6cc-4165-97f1-6f62ab0fb1ef/test.txt
└── worker
    └── job
        ├── 2c6b3dc4-1d21-4abf-9937-db475e6a6bc2
        │   └── test.txt
        ├── e3d724e1-e6cc-4165-97f1-6f62ab0fb1ef
        │   └── test.txt
        ├── tmpr2j5yaic
        ├── tmpxqr9__y4
        └── work

15 directories, 4 files

You can see where Toil downloaded the job's input files into the worker's temporary directory, and how they'd be mounted into the container 💖.

---

✨ Interactively Investigating Running Jobs ✨

Say you have a broken WDL workflow that can't complete. Whenever you run tutorial_debugging_hangs.wdl, it hangs 😭:

version 1.1

workflow TutorialDebugging {

    input {
        Array[String] messages = ["Uh-oh!", "Oh dear", "Oops"]
    }

    scatter(message in messages) {

        call WhaleSay {
            input:
                message = message
        }

        call CountLines {
            input:
                to_count = WhaleSay.result
        }
    }

    Array[File] to_compress = flatten([CountLines.result, WhaleSay.result])

    call CompressFiles {
        input:
            files = to_compress
    }

    output {
        File compressed = CompressFiles.result
    }

}

# Draw ASCII art
task WhaleSay {
    input {
        String message
    }

    command <<<
        cowsay "~{message}"
    >>>

    output {
        File result = stdout()
    }

    runtime {
        container: "docker/whalesay"
    }
}

# Count the lines in a file
task CountLines {
    input {
        File to_count
    }

    command <<<
        wc -l ~{to_count}
    >>>

    output {
        File result = stdout()
    }

    runtime {
        container: ["ubuntu:latest", "https://gcr.io/standard-images/ubuntu:latest"]
    }
}

# Compress files into a ZIP
task CompressFiles {
    input {
        Array[File] files
    }

    command <<<
    set -e
    cat >script.py <<'EOF'
    import sys
    from zipfile import ZipFile
    import os

    # Interpret command line arguments
    to_compress = list(reversed(sys.argv[1:]))

    with ZipFile("compressed.zip", "w") as z:
        while to_compress != []:
            # Grab the file to add off the end of the list
            input_filename = to_compress[-1]
            # Now we need to write this to the zip file.
            # What internal filename should we use?
            basename = os.path.basename(input_filename)
            disambiguation_number = 0
            while True:
                target_filename = str(disambiguation_number) + basename
                try:
                    z.getinfo(target_filename)
                except KeyError:
                    # Filename is free
                    break
                # Otherwise try another name
                disambiguation_number += 1
            # Now we can actually make the compressed file
            with z.open(target_filename, 'w') as out_stream:
                with open(input_filename) as in_stream:
                    for line in in_stream:
                        # Prefix each line of text with the original input file
                        # it came from.
                        # Also remember to encode the text as the zip file
                        # stream is in binary mode.
                        out_stream.write(f"{basename}: {line}".encode("utf-8"))
    EOF
    python script.py ~{sep(" ", files)}
    >>>

    output {
        File result = "compressed.zip"
    }

    runtime {
        container: "python:3.11"
    }
}

You can try to run it like this, using Docker containers. Pretend this was actually a run on a large cluster 🌈:

$ toil-wdl-runner --jobStore ./store tutorial_debugging_hangs.wdl --container docker

If you run this, it'll hang at the TutorialDebugging.CompressFiles.command step:

[2024-06-18T12:12:49-0400] [MainThread] [I] [toil.leader] Issued job 'WDLTaskJob' TutorialDebugging.CompressFiles.command kind-WDLTaskJob/instance-y0ga_907 v1 with job batch system ID: 16 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False

Workflow Progress  94%|██████████▎| 15/16 (0 failures) [00:36<00:02, 0.42 jobs/s]

Say you want to find out why it's stuck. First, you need to kill the workflow. Open a new shell in the same directory and run:

# toil kill ./store

You can also hit Control+C in its terminal window and wait for it to stop.

Then you need to use toil debug-job to run the stuck job on your local machine:

$ toil debug-job ./store TutorialDebugging.CompressFiles.command

This produces some more informative logging messages, showing that the Docker container is managing to start up, but that it stays running indefinitely, with a repeating message:

[2024-06-18T12:18:00-0400] [MainThread] [N] [MiniWDLContainers] docker task running :: service: "lhui2bdzmzmg", task: "sg371eb2yk", node: "zyu9drdp6a", message: "started"
[2024-06-18T12:18:01-0400] [MainThread] [D] [MiniWDLContainers] docker task status :: Timestamp: "2024-06-18T16:17:58.545272049Z", State: "running", Message: "started", ContainerStatus: {"ContainerID": "b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a", "PID": 36510, "ExitCode": 0}, PortStatus: {}
[2024-06-18T12:18:03-0400] [MainThread] [D] [MiniWDLContainers] docker task status :: Timestamp: "2024-06-18T16:17:58.545272049Z", State: "running", Message: "started", ContainerStatus: {"ContainerID": "b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a", "PID": 36510, "ExitCode": 0}, PortStatus: {}
[2024-06-18T12:18:04-0400] [MainThread] [D] [MiniWDLContainers] docker task status :: Timestamp: "2024-06-18T16:17:58.545272049Z", State: "running", Message: "started", ContainerStatus: {"ContainerID": "b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a", "PID": 36510, "ExitCode": 0}, PortStatus: {}
...

This also gives you the Docker container ID of the running container, b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a. You can use that to get a shell inside the running container 🕵️‍♀️:

$ docker exec -ti b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a bash
root@b7210b346637:/mnt/miniwdl_task_container/work#

Your shell is already in the working directory of the task, so we can inspect the files there to get an idea of how far it's gotten. Has it managed to create script.py? Has the script managed to create compressed.zip? Let's check 👀:

# ls -lah
total 6.1M
drwxrwxr-x 6 root root  192 Jun 18 16:17 .
drwxr-xr-x 3 root root 4.0K Jun 18 16:17 ..
drwxr-xr-x 3 root root   96 Jun 18 16:17 .toil_wdl_runtime
drwxrwxr-x 8 root root  256 Jun 18 16:17 _miniwdl_inputs
-rw-r--r-- 1 root root 6.0M Jun 18 16:23 compressed.zip
-rw-r--r-- 1 root root 1.3K Jun 18 16:17 script.py

So we can see that the script exists, and the zip file also exists 📂✨ bestie, so maybe the script is still running?? We can check with ps, but we need the -x option to pull in processes that aren't under the current shell 👀. We can also add the -u option to get the stats too 📊:

# ps -xu
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2316   808 ?        Ss   16:17   0:00 /bin/sh -c /bin/
root         7  0.0  0.0   4208  3056 ?        S    16:17   0:00 /bin/bash ../com
root         8  0.1  0.0   4208  1924 ?        S    16:17   0:00 /bin/bash ../com
root        20 95.0  0.4  41096 36428 ?        R    16:17   7:09 python script.py
root       645  0.0  0.0   4472  3492 pts/0    Ss   16:21   0:00 bash
root      1379  0.0  0.0   2636   764 ?        S    16:25   0:00 sleep 1
root      1380  0.0  0.0   8584  3912 pts/0    R+   16:25   0:00 ps -xu

Here we can see that python is literally running and eating 95% of a CPU core 🔥💀. So we can surmise that Python is probably stuck spinning around in an infinite loop 🌀. Let's look at our files again 🧐:

# ls -lah
total 8.1M
drwxrwxr-x 6 root root  192 Jun 18 16:17 .
drwxr-xr-x 3 root root 4.0K Jun 18 16:17 ..
drwxr-xr-x 3 root root   96 Jun 18 16:17 .toil_wdl_runtime
drwxrwxr-x 8 root root  256 Jun 18 16:17 _miniwdl_inputs
-rw-r--r-- 1 root root 7.6M Jun 18  2024 compressed.zip
-rw-r--r-- 1 root root 1.3K Jun 18 16:17 script.py

Note that, while we've been investigating, our compressed.zip file has grown from 6.0M to 7.6M 😭. So we now know that, not only is the Python script stuck in a loop, it is also writing to the ZIP file inside that loop 💀🫠.

Let's inspect the inputs 🎀:

# ls -lah _miniwdl_inputs/*
_miniwdl_inputs/0:
total 4.0K
drwxrwxr-x 3 root root  96 Jun 18 16:17 .
drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
-rw-r--r-- 1 root root  65 Jun 18 16:15 stdout.txt

_miniwdl_inputs/1:
total 4.0K
drwxrwxr-x 3 root root  96 Jun 18 16:17 .
drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
-rw-r--r-- 1 root root  65 Jun 18 16:15 stdout.txt

_miniwdl_inputs/2:
total 4.0K
drwxrwxr-x 3 root root  96 Jun 18 16:17 .
drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
-rw-r--r-- 1 root root  65 Jun 18 16:15 stdout.txt

_miniwdl_inputs/3:
total 4.0K
drwxrwxr-x 3 root root  96 Jun 18 16:17 .
drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
-rw-r--r-- 1 root root 384 Jun 18 16:15 stdout.txt

_miniwdl_inputs/4:
total 4.0K
drwxrwxr-x 3 root root  96 Jun 18 16:17 .
drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
-rw-r--r-- 1 root root 387 Jun 18 16:15 stdout.txt

_miniwdl_inputs/5:
total 4.0K
drwxrwxr-x 3 root root  96 Jun 18 16:17 .
drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
-rw-r--r-- 1 root root 378 Jun 18 16:15 stdout.txt

These are the files that are meant to be compressed into that ZIP file. But hang on... there are only six of these files, and none of them is over 400 bytes in size 🤨. How did we get a multi-megabyte ZIP file? The script must be putting more data than we expected into the ZIP file it is writing 😳.

Taking what we know, we can inspect the Python script again and see if we can find a way in which it could get stuck in an infinite loop, writing much more data to the ZIP than is actually in the input files. We can also inspect it for WDL variable substitutions (there aren't any 🙅‍♀️). Let's look at it with line numbers using the nl tool, numbering even blank lines with -b a 🔢:

# nl -b a script.py
 1  import sys
 2  from zipfile import ZipFile
 3  import os
 4
 5  # Interpret command line arguments
 6  to_compress = list(reversed(sys.argv[1:]))
 7
 8  with ZipFile("compressed.zip", "w") as z:
 9      while to_compress != []:
10          # Grab the file to add off the end of the list
11          input_filename = to_compress[-1]
12          # Now we need to write this to the zip file.
13          # What internal filename should we use?
14          basename = os.path.basename(input_filename)
15          disambiguation_number = 0
16          while True:
17              target_filename = str(disambiguation_number) + basename
18              try:
19                  z.getinfo(target_filename)
20              except KeyError:
21                  # Filename is free
22                  break
23              # Otherwise try another name
24              disambiguation_number += 1
25          # Now we can actually make the compressed file
26          with z.open(target_filename, 'w') as out_stream:
27              with open(input_filename) as in_stream:
28                  for line in in_stream:
29                      # Prefix each line of text with the original input file
30                      # it came from.
31                      # Also remember to encode the text as the zip file
32                      # stream is in binary mode.
33                      out_stream.write(f"{basename}: {line}".encode("utf-8"))

We have three loops here: while to_compress != [] on line 9, while True on line 16, and for line in in_stream on line 28 🌸.

The while True loop is immediately suspicious 👀, but none of the code inside it writes to the ZIP file, so we know we can't be stuck in there 🙅‍♀️.

The for line in in_stream loop contains the only call that writes data to the ZIP, so we must be spending time inside it, but it is constrained to loop over a single file at a time, so it can't be the infinite loop we're looking for 💅.

So then we must be infinitely looping at while to_compress != [], and indeed we can see that to_compress is never modified, so it can never become [] 🤯.

So now we have a theory as to what the problem is, and we can exit out of our shell in the container, and stop toil debug-job with Control+C 🛑. Then we can make the following change to our workflow, adding code to the script to actually pop the handled files off the end of the list 🫶:

--- tutorial_debugging_works.wdl   2024-06-18 12:03:32
+++ tutorial_debugging_hangs.wdl   2024-06-18 12:03:53
@@ -112,9 +112,6 @@
                        # Also remember to encode the text as the zip file
                        # stream is in binary mode.
                        out_stream.write(f"{basename}: {line}".encode("utf-8"))
-            # Even though we got distracted by zip file manipulation, remember
-            # to pop off the file we just did.
-            to_compress.pop()
     EOF
     python script.py ~{sep(" ", files)}
     >>>

If we apply that change and produce a new file, tutorial_debugging_works.wdl, we can clean up from the old failed run and run a new one 🧹:

$ toil clean ./store
$ toil-wdl-runner --jobStore ./store tutorial_debugging_works.wdl --container docker

This will produce a successful log, ending with something like 🎉:

[2024-06-18T12:42:20-0400] [MainThread] [I] [toil.leader] Finished toil run successfully.

Workflow Progress 100%|███████████| 17/17 (0 failures) [00:24<00:00, 0.72 jobs/s]
{"TutorialDebugging.compressed": "/Users/anovak/workspace/toil/src/toil/test/docs/scripts/wdl-out-u7fkgqbe/f5e16468-0cf6-4776-a5c1-d93d993c4db2/compressed.zip"}
[2024-06-18T12:42:20-0400] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/Users/anovak/workspace/toil/src/toil/test/docs/scripts/store)

Note the line to standard output giving us the path on disk where the TutorialDebugging.compressed output from the workflow is 📍. If you look at that ZIP file, you can see it contains the expected files, such as 3stdout.txt, which should contain this suitably prefixed dismayed whale 🐳:

stdout.txt:  ________
stdout.txt: < Uh-oh! >
stdout.txt:  --------
stdout.txt:     \
stdout.txt:      \
stdout.txt:       \
stdout.txt:                     ##        .
stdout.txt:               ## ## ##       ==
stdout.txt:            ## ## ## ##      ===
stdout.txt:        /""""""""""""""""___/ ===
stdout.txt:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
stdout.txt:        \______ o          __/
stdout.txt:         \    \        __/
stdout.txt:           \____\______/

When we're done inspecting the output, and satisfied that the workflow now works, we might want to clean up all the auto-generated WDL output directories from the successful and failed run(s) 🗑️:

$ rm -Rf wdl-out-*

🔎 Introspecting the Job Store 🔎

Note: Currently these features are only implemented for use locally (single machine) with the fileJobStore.

To view what files currently reside in the jobstore, run the following command 👀:

$ toil debug-file file:path-to-jobstore-directory \
      --listFilesInJobStore

When run from the commandline, this should generate a file containing the contents of the job store (in addition to displaying a series of log messages to the terminal). This file is named "jobstore_files.txt" by default and will be generated in the current working directory 📝.

If one wishes to copy any of these files to a local directory, one can run for example 📦:

$ toil debug-file file:path-to-jobstore \
      --fetch overview.txt *.bam *.fastq \
      --localFilePath=/home/user/localpath

To fetch overview.txt, and all .bam and .fastq files. This can be used to recover previously used input and output files for debugging or reuse in other workflows, or use in general debugging to ensure that certain outputs were imported into the jobStore 🫶.

📊 Stats and Status 📊

See Stats Command <#cli-stats> and Status Command <#cli-status> for more about gathering statistics about job success, runtime, and resource usage from workflows ✨.

🐍 Using a Python debugger 🐍

If you execute a workflow using the --debugWorker flag, or if you use toil debug-job, Toil will run the job in the process you started from the command line. This means you can either use pdb https://docs.python.org/3/library/pdb.html, or an IDE that supports debugging Python https://wiki.python.org/moin/PythonDebuggingTools#IDEs_with_Debug_Capabilities to interact with the Python process as it runs your job 🦋.

Heads up: Note that the --debugWorker flag will only work with the single_machine batch system (the default), and not any of the custom job schedulers. 💀

---

☁️ RUNNING IN THE CLOUD ☁️

Toil supports Amazon Web Services (AWS) and Google Compute Engine (GCE) in the cloud and has autoscaling capabilities that can adapt to the size of your workflow, whether your workflow requires 10 instances or 20,000 🌈🚀

Toil does this by spinning up a virtual cluster running Kubernetes 💖 (https://kubernetes.io/). Kubernetes needs a leader node to coordinate the workflow, plus worker nodes to actually run the tasks inside it. As the workflow goes, Kubernetes will "autoscale" 🦋, creating and terminating workers whenever the workflow needs them. Historically, toil spun up clusters with Apache Mesos (https://mesos.apache.org/gettingstarted/), but ngl that's no longer recommended 💀.

Once you're comfy running toil locally (picking a jobStore, see #jobstoreoverview, and writing a workflow), you can check out the guides below to turn those workflows into cloud ready ones ✨.

✨ Managing a Cluster of Virtual Machines (Provisioning) ✨

Toil can launch and manage a whole cluster of virtual machines 🖥️ using the provisioner, so your workflow runs across several nodes. The provisioner can also automatically scale the cluster up or down (autoscaling) when computational demand changes. Right now we have working provisioners for AWS and GCE. Azure support has been deprecated, so no more Azure, bestie 🙈.

Toil uses Kubernetes (https://kubernetes.io/) as the Batch System (see #batchsystemoverview).

See here for instructions for Running in AWS (#runningaws) 🌸.

See here for instructions for Running in Google Compute Engine (GCE) (#runninggce) 🌈.

Toil also has a whole suite of commands for using the provisioners to manage clusters 💅.

🎀 Toil Cluster Utilities 🎀

On top of the generic Toil Utilities (#utils), there are several utilities for starting and managing a Toil cluster with the AWS or GCE provisioners. They're installed via the [aws] or [google] extra. For installation details, see Toil Provisioner (#installprovisioner).

The toil cluster subcommands are:

• destroy-cluster — For autoscaling. Terminates the specified cluster and its associated resources 🧹.
• launch-cluster — For autoscaling. Used to launch a toil leader instance with the specified provisioner 🚀.
• rsync-cluster — For autoscaling. Transfers files to a cluster launched with toil launch-cluster 📦.
• ssh-cluster — SSHs into the toil appliance container running on the leader of the cluster 💜.

For info on a specific utility, run it with the --help option:

toil launch-cluster --help

The cluster utilities can be used for Running in Google Compute Engine (GCE) (#runninggce) and Running in AWS (#runningaws) 🫶.

Tip: By default, all of the cluster utilities expect to be running on AWS. To run with Google, you'll need to specify the --provisioner gce option for each utility.

Note: Boto must be configured (http://boto3.readthedocs.io/en/latest/guide/quickstart.html#configuration) with AWS credentials before using cluster utilities. Running in Google Compute Engine (GCE) (#runninggce) contains instructions for

🦋 Launch-Cluster Command 🦋

Running toil launch-cluster starts up a leader for a cluster. You can add workers to the initial cluster by specifying the -w option. An example would be:

$ toil launch-cluster my-cluster \
      --leaderNodeType t2.small -z us-west-2a \
      --keyPairName your-AWS-key-pair-name \
      --nodeTypes m3.large,t2.micro -w 1,4

The options are listed below. You can also pull them up by running:

$ toil launch-cluster --help

The main positional argument for launch-cluster is the clusterName. This is simply the name of your cluster. If it doesn't exist yet, Toil will create it for you 🥰.

🌈 Launch-Cluster Options 🌈

• --help, -h — also accepted. Displays this help menu 📖.
• --tempDirRoot TEMPDIRROOT — The path to the temporary directory where all temp files get created. By default, it uses the current working directory as the base.
• --version — Displays the version.
• --provisioner CLOUDPROVIDER, -p CLOUDPROVIDER — also accepted. The provisioner for cluster auto-scaling. Both AWS and GCE are currently supported.
• --zone ZONE, -z ZONE — also accepted. The availability zone of the leader. This can also be set via the TOIL_AWS_ZONE or TOIL_GCE_ZONE environment variables, or by the ec2_region_name parameter in your .boto file if you're using AWS. If you're using this utility on an existing EC2 instance, it's derived from the instance metadata instead.
• --leaderNodeType LEADERNODETYPE — The non-preemptable node type for the cluster leader.
• --keyPairName KEYPAIRNAME — The name of the AWS or ssh key pair to include on the instance 🔑.
• --owner OWNER — The owner tag for all instances. If it isn't given, the value in TOIL_OWNER_TAG is used, or else the value of --keyPairName.
• --boto BOTOPATH — The path to the boto credentials directory. It gets transferred to all nodes so they can access the AWS jobStore from non-AWS instances.
• --tag KEYVALUE, -t KEY=VALUE — KEYVALUE is specified as KEY=VALUE. -t KEY=VALUE is also accepted. Tags get added to the AWS cluster for this node and all of its children. Tags look like -t key1=value1 --tag key2=value2. You can add multiple tags, but each one needs its own flag. By default, the cluster is tagged with: { "Name": clusterName, "Owner": IAM username } 🏷️.
• --vpcSubnet VPCSUBNET — The VPC subnet ID to launch the cluster leader in. It uses the default subnet if you don't specify one. This subnet needs to have auto assign IPs turned on.
• --nodeTypes NODETYPES — A comma-separated list of node types to create while launching the leader. The syntax for each node type depends on the provisioner you use. For the AWS provisioner, it's the name of an EC2 instance type, followed by a colon and the dollar price to bid for a spot instance, like 'c3.8xlarge:0.42'. You must also provide the --workers argument to say how many workers of each node type to create.
• --workers WORKERS, -w WORKERS — also accepted. A comma-separated list of how many workers of each node type to launch alongside the leader when the cluster is created. This is handy if you're running toil without auto-scaling but need more hardware support 💪.
• --leaderStorage LEADERSTORAGE — The size (in gigabytes) of the root volume for the leader instance. This is an EBS volume 💾.
• --nodeStorage NODESTORAGE — The size (in gigabytes) of the root volume for any worker instances created when using the -w flag. This is an EBS volume.
• --nodeStorageOverrides NODESTORAGEOVERRIDES — A comma-separated list of nodeType:nodeStorage pairs that override the default from --nodeStorage for the specified nodeType(s). This is super useful for heterogeneous jobs where some tasks need way more disk than others.
• --allowFuse BOOL — Whether to allow FUSE mounts for faster runtimes with Singularity. Heads up, this one is a lot 👇.

Warning: Allowing FUSE mounts results in the Toil container running as privileged. For Kubernetes, pods will be asked to run as privileged. If this isn't allowed, Singularity containers will use sandbox directories instead 🚨.

🔮 Logging Options 🔮

• --logOff — Same as --logCritical.
• --logCritical — Turns on logging at level CRITICAL and above. (default is INFO)
• --logError — Turns on logging at level ERROR and above. (default is INFO)
• --logWarning — Turns on logging at level WARNING and above. (default is INFO)
• --logInfo — Turns on logging at level INFO and above. (default is INFO)
• --logDebug — Turns on logging at level DEBUG and above. (default is INFO)
• --logDebug — Turns on logging at level TRACE and above. (default is INFO) 🙈 yes, --logDebug shows up twice in the original, and this second one is the TRACE one.
• --logLevel LOGLEVEL — Logs at the given level. It can be OFF (or CRITICAL), ERROR, WARN (or WARNING), INFO, DEBUG, or TRACE. (default is INFO)
• --logFile LOGFILE — The file to log in 📝.
• --rotatingLogging — Turns on rotating logging, which keeps log files from getting too big 🔄.

💜 Ssh-Cluster Command 💜

Toil lets you ssh into the leader of the cluster. Here's how:

$ toil ssh-cluster CLUSTER-NAME-HERE

This opens a shell on the Toil leader, and it's where you start an Running a Workflow with Autoscaling (#autoscaling) run. Docker issues stop you from using screen and tmux when you ssh into the cluster, because the shell doesn't know it's a TTY, which means it can't allocate a new screen session 😭. You can work around this with:

$ script
$ screen

Just running screen inside script will get things working properly again ✨.

Finally, you can run remote commands with this syntax:

$ toil ssh-cluster CLUSTER-NAME-HERE remoteCommand

Warning: It's not advised to run your Toil workflow with remote execution like this, unless you use a tool like nohup (https://linux.die.net/man/1/nohup). That keeps the process from dying if the SSH connection gets interrupted.

For an example, check out Running a Workflow with Autoscaling (#autoscaling) 🎬.

🧚 Rsync-Cluster Command 🧚

The most common use case for the rsync-cluster utility is deploying your workflow code to the Toil leader. The syntax is the same as traditional rsync (https://linux.die.net/man/1/rsync), except for the hostname before the colon. You don't need that in toil rsync-cluster, since Toil automatically figures out the hostname 🫶.

Here's an example of how it's used:

$ toil rsync-cluster CLUSTER-NAME-HERE \
      ~/localFile :/remoteDestination

🧹 Destroy-Cluster Command 🧹

The destroy-cluster command is the advised way to get rid of any Toil cluster launched using the Launch-Cluster Command command. It makes sure all attached nodes, volumes, security groups, etc. get deleted. If you shut a node or cluster down using Amazon's online portal, residual resources may still be running in the background 👀. To delete a cluster, run:

$ toil destroy-cluster CLUSTER-NAME-HERE

🫧 Storage (Toil jobStore) 🫧

Toil can use cloud storage, like AWS or Google buckets, to handle its storage needs ☁️.

This is super useful when you run Toil in single machine mode on any cloud platform, because you get to use their integrated storage systems 💖.

For an overview of the job store, peep Job Store (#jobstoreoverview) 👀✨ that's the big-picture tea.

If you want the step-by-step for a specific job store, check these out:

• AWS Job Store (#awsjobstore) 🦋
• Google Job Store (#googlejobstore) 🌈

---

✨ CLOUD PLATFORMS ✨

🌸 Running on Kubernetes

Okay so Kubernetes (https://kubernetes.io/) is literally the most popular container orchestration tool, and it's basically become the de facto cross-cloud-provider API for getting at cloud resources. 😭 Major providers like Amazon (https://aws.amazon.com/kubernetes/), Microsoft (https://azure.microsoft.com/en-us/overview/kubernetes-getting-started/), Kubernetes owner Google (https://cloud.google.com/kubernetes-engine/), and DigitalOcean (https://www.digitalocean.com/products/kubernetes/) have all put serious effort into making Kubernetes work well on their platforms. They wrote their own deployment docs and built provider-managed Kubernetes products. And if you want it on just one machine, minikube (https://github.com/kubernetes/minikube) can run Kubernetes on a single machine. Iconic. 💅

Toil supports running Toil workflows against a Kubernetes cluster, either in the cloud or deployed on user-owned hardware. 🫶

🧚 Preparing your Kubernetes environment

1. Get a Kubernetes cluster

To run Toil workflows on Kubernetes, you need a Kubernetes cluster set up. This isn't covered here, but there are loads of options, and which one you pick depends on which cloud ecosystem you already use (if any) and on pricing. If you're just following along with the docs, use minikube on your local machine. 🍓

Alternatively, Toil can set up a Kubernetes cluster for you with the Toil provisioner (#installprovisioner). Follow this guide (#provisioningwithkubernetes) to get started with a Toil-managed Kubernetes cluster on AWS. 🎀

Heads up, bestie 💀: right now the only way to run a Toil workflow on Kubernetes is with the AWS Job Store. So your Kubernetes workflow has to store its data in Amazon's cloud no matter where you run it. This can result in significant egress charges from Amazon if you run it outside of Amazon.

Kubernetes Cluster Providers:

• Your own institution
• Amazon EKS (https://aws.amazon.com/eks/)
• Microsoft Azure AKS (https://docs.microsoft.com/en-us/azure/aks/)
• Google GKE (https://cloud.google.com/kubernetes-engine/)
• DigitalOcean Kubernetes (https://www.digitalocean.com/docs/kubernetes/)
• minikube (https://kubernetes.io/docs/tasks/tools/install-minikube/)

2. Get a Kubernetes context on your local machine

There are two main ways to run Toil workflows on Kubernetes. You can run the Toil leader on a machine outside the cluster, with jobs submitted to and run on the cluster, or you can submit the Toil leader itself as a job and let it run inside the cluster. Either way, your own machine needs to be set up so it can submit jobs to the Kubernetes cluster. Generally, that means creating and populating a file named .kube/config in your user's home directory, and specifying the cluster to connect to, the certificate and token info needed for mutual authentication, and the Kubernetes namespace to work within. However, Kubernetes configuration can also be picked up from other files in the .kube directory, environment variables, and the enclosing host when running inside a Kubernetes-managed container. 🧠

You'll have to do different things here depending on where you got your Kubernetes cluster:

• Configuring for Amazon EKS: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
• Configuring for Microsoft Azure AKS: https://docs.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-get-credentials
• Configuring for Google GKE: https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl
• Configuring for DigitalOcean Kubernetes Clusters: https://www.digitalocean.com/docs/kubernetes/how-to/connect-to-cluster/
• Configuring for minikube: https://kubernetes.io/docs/setup/learning-environment/minikube/#kubectl

Toil's internal Kubernetes configuration logic mirrors that of the kubectl command. Toil workflows will use the current kubectl context to launch their Kubernetes jobs. ✨

3. If running the Toil leader in the cluster, get a service account

If you're going to run your workflow's leader inside the Kubernetes cluster (see Option 1: Running the Leader Inside Kubernetes), you'll need a service account in your chosen Kubernetes namespace. Most namespaces should have one named default, which should work fine. If your cluster requires a different service account, get its name and use it when launching the Kubernetes job containing the Toil leader. 🌈

4. Set up appropriate permissions

Your local Kubernetes context and/or the service account you're using to run the leader in the cluster need certain permissions to run the workflow. Toil has to be able to interact with jobs and pods in the cluster, and to retrieve pod logs. You as a user may need permission to set up an AWS credentials secret, if one isn't already available. Additionally, it's super useful for you to have permission to interact with nodes and to shell into pods. 👑

The appropriate permissions may already be available to you and your service account by default, especially in managed or ease-of-use-optimized setups like EKS or minikube.

However, if the appropriate permissions aren't already available, you or your cluster administrator will have to grant them manually. The following Role (toil-user) and ClusterRole (node-reader), to be applied with kubectl apply -f filename.yaml, should grant sufficient permissions to run Toil workflows when bound to your account and the service account used by Toil workflows. Be sure to replace YOUR_NAMESPACE_HERE with the namespace you're running your workflows in.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: YOUR_NAMESPACE_HERE
  name: toil-user
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["explain", "get", "watch", "list", "describe", "logs", "attach", "exec", "port-forward", "proxy", "cp", "auth"]
- apiGroups: ["batch"]
  resources: ["*"]
  verbs: ["get", "watch", "list", "create", "run", "set", "delete"]
- apiGroups: [""]
  resources: ["secrets", "pods", "pods/attach", "podtemplates", "configmaps", "events", "services"]
  verbs: ["patch", "get", "update", "watch", "list", "create", "run", "set", "delete", "exec"]
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "describe"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "describe"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["*"]
  verbs: ["*"]

To bind a user or service account to the Role or ClusterRole and actually grant the permissions, you'll need a RoleBinding and a ClusterRoleBinding, respectively. Make sure to fill in the namespace, username, and service account name, and add more user stanzas if your cluster is going to support multiple Toil users. 🦋

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: toil-developer-member
  namespace: toil
subjects:
- kind: User
  name: YOUR_KUBERNETES_USERNAME_HERE
  apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
  name: YOUR_SERVICE_ACCOUNT_NAME_HERE
  namespace: YOUR_NAMESPACE_HERE
roleRef:
  kind: Role
  name: toil-user
  apiGroup: rbac.authorization.k8s.io

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-nodes
subjects:
- kind: User
  name: YOUR_KUBERNETES_USERNAME_HERE
  apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
  name: YOUR_SERVICE_ACCOUNT_NAME_HERE
  namespace: YOUR_NAMESPACE_HERE
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

🦋 AWS Job Store for Kubernetes

Currently, the only job store that works with jobs running on Kubernetes is the AWS Job Store. The job store is what Toil uses to exchange data between jobs, and this one requires that the Toil leader and Kubernetes jobs be able to connect to and use Amazon S3 and Amazon SimpleDB. It also requires that you have an Amazon Web Services account. ☁️

1. Get access to AWS S3 and SimpleDB

In your AWS account, you need to create an AWS access key. First, go to the IAM dashboard. For "us-west1", the link would be:

https://console.aws.amazon.com/iam/home?region=us-west-1#/home

Then create an access key, and save the Access Key ID and the Secret Key. As documented in the AWS documentation (https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html):

1. On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.
2. Expand the Access keys (access key ID and secret access key) section.
3. Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can't retrieve this secret access key again.

Make sure that, if your AWS infrastructure requires your user to authenticate with a multi-factor authentication (MFA) token, you obtain a second secret key and access key that don't have this requirement. The secret key and access key used to populate the Kubernetes secret that allows the jobs to contact the job store need to be usable without human intervention. 🙈

2. Configure AWS access from the local machine

This only really needs to happen if you run the leader on the local machine. But we need the files in place to fill in the secret in the next step. Run:

$ aws configure

Then when it asks, punch in your secret key and access key 🔑✨ That should spit out a file at ~/.aws/credentials that looks like this:

[default]
aws_access_key_id =  BLAH
aws_secret_access_key =  blahblahblah

3. Create a Kubernetes secret to give jobs access to AWS 🔐 Hop into the folder where your credentials file lives:

$ cd ~/.aws

Then make a Kubernetes secret that holds it. We're calling it aws-credentials 💖

$ kubectl create secret generic aws-credentials --from-file credentials

---

✨ Configuring Toil for your Kubernetes environment ✨

Okay bestie, so to run your workflow on Kubernetes, you'll need to set up a bunch of environment variables 🌸 and pass the --batchSystem kubernetes option too. Figuring out what values to use for these might mean chatting with your cluster provider, so don't be shy 🫶

1. TOIL_AWS_SECRET_NAME is the MAIN one 👑 It has to be set to the secret holding your AWS credentials file, if your cluster nodes can't already reach S3 and SimpleDB (like through IAM roles). It's required for the AWS job store to work, and that's currently the only job store you can use on Kubernetes. In this example we're using aws-credentials.
2. TOIL_KUBERNETES_HOST_PATH lets Toil jobs on the same physical host share a cache 🦋 Set it to a path on the host where the shared cache should live. Inside the container it gets mounted as /var/lib/toil, or at TOIL_WORKDIR if you've specified that. The path has to already exist on the host, and it needs as much free space as your Kubernetes node offers to jobs. In this example we're using /data/scratch. To actually make use of caching, make sure you don't use --disableCaching.
3. TOIL_KUBERNETES_OWNER should be the username of whoever is running the Toil workflow 💅 Toil's jobs include this name, so they're easy to spot and clean up if anything happens to the Toil leader. In this example we're using demo-user.
4. TOIL_KUBERNETES_PRIVILEGED can be True or False 🔥 When it's True, pods are allowed to run as privileged, which enables FUSE mounts for Singularity for faster runtimes. If it isn't set to true, Singularity extracts images into sandbox directories instead. It's unset/False by default, except in Toil-managed clusters.

⚠️ Heads up, bestie: Docker containers cannot be run inside unprivileged Kubernetes pods (which are themselves containers) 💀 The Docker daemon doesn't (yet) support this. Other tools, like Singularity in its user-namespace mode, can run containers from within containers.

If you're using Singularity to run containerized tools and want downloaded container images to stick around between Toil jobs, you might need some setup 🧚

On non-Toil managed clusters: You'll want to set TOIL_KUBERNETES_HOST_PATH too, and make sure Singularity downloads its containers under the Toil work directory (/var/lib/toil by default) by setting SINGULARITY_CACHEDIR.

On Toil-managed clusters: Clusters made with the launch-cluster command need no setup 🎀 TOIL_KUBERNETES_HOST_PATH is already set to /var/lib/toil, and SINGULARITY_CACHEDIR is set to /var/lib/toil/singularity, which is a shared location. You may still need to implement Singularity locking (shown below) or move the Singularity cache somewhere else.

If you're using toil-wdl-runner, all the necessary Singularity locking is already in place, so you shouldn't have to do anything 🙌 Otherwise, for both Toil managed and non-Toil managed clusters, you need to make sure that no two jobs try to download the same container at the same time. Singularity has no synchronization or locking around its cache, and the cache isn't safe for simultaneous access by multiple Singularity invocations 😭 Some Toil workflows use their own custom workaround for this; for example, see this section in toil-wdl-runner: https://github.com/DataBiosphere/toil/blob/1821782b9ec40431e8afa36030fb8ee5df943003/src/toil/wdl/wdltoil.py#L1675-L1679

---

✨ Running workflows ✨

To run the workflow, you'll need to run the Toil leader process somewhere 🚀 It can either run inside Kubernetes as a Kubernetes job, or outside Kubernetes as a normal command.

✨ Option 1: Running the Leader Inside Kubernetes ✨

Once you've figured out your environment variable values for the workflow run, write a YAML file that defines a Kubernetes job to run your workflow with that configuration 📝 Some config items (like your username and the name of your AWS credentials secret) need to be written into the YAML so the leader can use them too.

The leader pod will need your workflow, its other dependencies, and Toil, all installed 🍓 The easiest way to get Toil installed is to start from the Toil appliance image for the version of Toil you want. In this example we use quay.io/ucsc_cgl/toil:5.5.0.

Here's an example YAML file to run a test workflow 👑

apiVersion: batch/v1
kind: Job
metadata:
  # It is good practice to include your username in your job name.
  # Also specify it in TOIL_KUBERNETES_OWNER
  name: demo-user-toil-test
# Do not try and rerun the leader job if it fails

spec:
 backoffLimit: 0
 template:
   spec:
     # Do not restart the pod when the job fails, but keep it around so the
     # log can be retrieved
     restartPolicy: Never
     volumes:
     - name: aws-credentials-vol
       secret:
         # Make sure the AWS credentials are available as a volume.
         # This should match TOIL_AWS_SECRET_NAME
         secretName: aws-credentials
     # You may need to replace this with a different service account name as
     # appropriate for your cluster.
     serviceAccountName: default
     containers:
     - name: main
       image: quay.io/ucsc_cgl/toil:5.5.0
       env:
       # Specify your username for inclusion in job names
       - name: TOIL_KUBERNETES_OWNER
         value: demo-user
       # Specify where to find the AWS credentials to access the job store with
       - name: TOIL_AWS_SECRET_NAME
         value: aws-credentials
       # Specify where per-host caches should be stored, on the Kubernetes hosts.
       # Needs to be set for Toil's caching to be efficient.
       - name: TOIL_KUBERNETES_HOST_PATH
         value: /data/scratch
       volumeMounts:
       # Mount the AWS credentials volume
       - mountPath: /root/.aws
         name: aws-credentials-vol
       resources:
         # Make sure to set these resource limits to values large enough
         # to accommodate the work your workflow does in the leader
         # process, but small enough to fit on your cluster.
         #
         # Since no request values are specified, the limits are also used
         # for the requests.
         limits:
           cpu: 2
           memory: "4Gi"
           ephemeral-storage: "10Gi"
       command:
       - /bin/bash
       - -c
       - |
         # This Bash script will set up Toil and the workflow to run, and run them.
         set -e
         # We make sure to create a work directory; Toil can't hot-deploy a
         # Python file from the root of the filesystem, which is where we start.
         mkdir /tmp/work
         cd /tmp/work
         # We make a virtual environment to allow workflow dependencies to be
         # hot-deployed.
         #
         # We don't really make use of it in this example, but for workflows
         # that depend on PyPI packages we will need this.
         #
         # We use --system-site-packages so that the Toil installed in the
         # appliance image is still available.
         virtualenv --python python3 --system-site-packages venv
         . venv/bin/activate
         # Now we install the workflow. Here we're using a demo workflow
         # from Toil itself.
         wget https://raw.githubusercontent.com/DataBiosphere/toil/releases/4.1.0/src/toil/test/docs/scripts/tutorial_helloworld.py
         # Now we run the workflow. We make sure to use the Kubernetes batch
         # system and an AWS job store, and we set some generally useful
         # logging options. We also make sure to enable caching.
         python3 tutorial_helloworld.py \
             aws:us-west-2:demouser-toil-test-jobstore \
             --batchSystem kubernetes \
             --realTimeLogging \
             --logInfo

Save this YAML as leader.yaml, then run it on your Kubernetes installation with:

$ kubectl apply -f leader.yaml

To keep an eye on the leader job's progress, read its logs 👀 If you're using a Kubernetes dashboard like k9s (https://github.com/derailed/k9s), just find the pod created for the job in the dashboard and view its logs there. If not, you'll have to locate the pod by hand 🦋

✨ Monitoring and Debugging Kubernetes Jobs and Pods ✨

These techniques are most useful for the pod that holds the Toil leader, but they also work on individual Toil jobs on Kubernetes, even when the leader is outside the cluster 💜

Kubernetes names pods for jobs by appending a short random string to the job's name. To find the name of your job's pod, do:

$ kubectl get pods | grep demo-user-toil-test
demo-user-toil-test-g5496                                         1/1     Running     0          2m

As long as you've set TOIL_KUBERNETES_OWNER correctly, you should be able to find all of your workflow's pods by searching for your username 🔍

$ kubectl get pods | grep demo-user

If a pod's status is anything other than Pending, you can view its logs with:

$ kubectl logs demo-user-toil-test-g5496

This dumps the pod's logs from the beginning until now, then terminates. To follow along with the logs from a running pod, add the -f option 🌈

$ kubectl logs -f demo-user-toil-test-g5496

Okay so if your pod is stuck in ImagePullBackoff 😭, that's giving "bestie, you asked for an image that doesn't exist." If you're looking at a leader, check the image section of your YAML. If you're dealing with a worker job, check the value of TOIL_APPLIANCE_SELF instead. Also check your Kubernetes node's Internet connectivity and DNS 🌐. In Kubernetes, DNS depends on system-level pods, and those can get terminated or evicted when resources are oversubscribed, just like your own workloads. Not fair, but it's real 💀.

If your pod seems stuck in Pending or ContainerCreating 🥺, you can get the tea on what's wrong with kubectl describe pod:

$ kubectl describe pod demo-user-toil-test-g5496

Pay extra attention to the Events: section at the very end of the output. If you see FailedScheduling events, that means your job is too big for the nodes on your cluster, or your cluster is too busy for your jobs 😩:

Type     Reason            Age                  From               Message
----     ------            ----                 ----               -------
Warning  FailedScheduling  13s (x79 over 100m)  default-scheduler  0/4 nodes are available: 1 Insufficient cpu, 1 Insufficient ephemeral-storage, 4 Insufficient memory.

If a pod is running but acting erratically, or looks stuck, you can shell in and snoop around 🕵️‍♀️:

$ kubectl exec -ti demo-user-toil-test-g5496 /bin/bash

One super common reason pods get stuck is trying to use more memory than Kubernetes allows (or more than the Toil job's memory resource requirement), but in a way that doesn't trigger the Linux OOM killer to terminate the pod's processes. In that case, the pod can sit at nearly 100% memory usage more or less forever, and trying to shell in will fail, because starting a process inside the pod needs some of that memory too. The recommended fix is to kill the offending pod and increase its memory requirement (or its Toil job's), or reduce its memory needs by adapting your user code 🧠.

---

✨ When Things Go Wrong ✨

The Toil Kubernetes batch system has cleanup code that terminates worker jobs when the leader shuts down. But if the leader pod gets removed by Kubernetes, is forcibly killed, or otherwise suffers a sudden existence failure, it can vanish while its worker jobs keep living their best life 🙃.

It is not recommended to restart a workflow in this state. Jobs from the previous invocation will remain running and will be trying to modify the job store concurrently with jobs from the new invocation. 🚫

To clean up dangling jobs, use this snippet:

$ kubectl get jobs | grep demo-user | cut -f1 -d' ' | xargs -n10 kubectl delete job

This deletes all jobs with demo-user's username in their names, in batches of 10. You can also put the UUID that Toil assigns to a particular workflow invocation in the filter, so you only clean up the jobs for that one invocation 🧹.

---

✨ Option 2: Running the Leader Outside Kubernetes ✨

If you don't want to run your Toil leader inside Kubernetes, you can run it locally instead. This is super useful when developing a workflow, since files can be hot-deployed from your local machine straight to Kubernetes 🚀. But your local machine needs access to AWS (ideally role-assumption- and MFA-free 🔐) and access to Kubernetes too. Real time logging won't work unless your local machine can listen for incoming UDP packets on arbitrary ports on the address it uses to contact the IPv4 Internet. Toil does no NAT traversal or detection, so keep that in mind.

Note that if you set TOIL_WORKDIR when running your workflow like this, it needs to be a directory that exists both on the host and in the Toil appliance.

Here's an example of running our test workflow leader locally, outside of Kubernetes:

$ export TOIL_KUBERNETES_OWNER=demo-user  # This defaults to your local username if not set
$ export TOIL_AWS_SECRET_NAME=aws-credentials
$ export TOIL_KUBERNETES_HOST_PATH=/data/scratch
$ virtualenv --python python3 --system-site-packages venv
$ . venv/bin/activate
$ wget https://raw.githubusercontent.com/DataBiosphere/toil/releases/4.1.0/src/toil/test/docs/scripts/tutorial_helloworld.py
$ python3 tutorial_helloworld.py \
      aws:us-west-2:demouser-toil-test-jobstore \
      --batchSystem kubernetes \
      --realTimeLogging \
      --logInfo

---

✨ Running CWL Workflows ✨

Running CWL workflows on Kubernetes can be a lil' bit challenging 😤, because executing CWL can require toil-cwl-runner to orchestrate containers of its own, inside a Kubernetes job that's running in the Toil appliance container.

Normally, running a CWL workflow should Just Work ✨, as long as the workflow's Docker containers can be executed with Singularity, your Kubernetes cluster doesn't impose extra capability-based confinement (i.e. SELinux, AppArmor) that interferes with Singularity's use of user-mode namespaces, and you configure Toil so its workers know where to store their data within the Kubernetes pods (which is done for you if you're using a Toil-managed cluster). For example, you should be able to run a CWL workflow like this:

$ export TOIL_KUBERNETES_OWNER=demo-user  # This defaults to your local username if not set
$ export TOIL_AWS_SECRET_NAME=aws-credentials
$ export TOIL_KUBERNETES_HOST_PATH=/data/scratch
$ virtualenv --python python3 --system-site-packages venv
$ . venv/bin/activate
$ pip install toil[kubernetes,cwl]==5.8.0
$ toil-cwl-runner  \
     --jobStore  aws:us-west-2:demouser-toil-test-jobstore \
     --batchSystem kubernetes \
     --realTimeLogging \
     --logInfo \
     --disableCaching \
     path/to/cwl/workflow \
     path/to/cwl/input/object

Additional cwltool options that your workflow might require, such as --no-match-user, can be passed to toil-cwl-runner, which inherits most cwltool options 🎀.

---

✨ AppArmor and Singularity ✨

Kubernetes clusters based on Ubuntu hosts often have AppArmor enabled on the host. AppArmor is a capability-based security enhancement system that plugs into the Linux kernel to enforce lists of things programs may or may not do, called profiles. For example, an AppArmor profile could be applied to a web server process to stop it from using the mount() system call to manipulate the filesystem. It has no business doing that under normal circumstances, but it might try if hackers compromised it 👀.

Kubernetes clusters also often use Docker as the backing container runtime to run pod containers. When AppArmor is enabled, Docker loads an AppArmor profile and applies it to all of its containers by default, with the ability for the profile to be overridden on a per-container basis. Unfortunately, this profile blocks some of the mount() system calls that Singularity uses to set up user-mode containers from working inside the pod, even though these calls would be allowed for an unprivileged user under normal circumstances 😭.

On the UCSC Kubernetes cluster, we configure our Ubuntu hosts with an alternative default AppArmor profile for Docker containers (https://github.com/adamnovak/gi-kubernetes-autoscaling-config/blob/e1350ac9ad17d94b5073b20db3c75620957926e3/kubenode.ubuntu.cloud-config.yaml#L27-L67), which allows these calls. Other solutions include turning off AppArmor on the host, configuring Kubernetes with a container runtime other than Docker, or using Kubernetes's AppArmor integration (https://kubernetes.io/docs/tutorials/security/apparmor/) to apply a more permissive profile or the unconfined profile to pods that Toil launches.

Toil does not yet have a way to apply a container.apparmor.security.beta.kubernetes.io/runner-container: unconfined annotation to its pods, as described in the Kubernetes AppArmor documentation (https://kubernetes.io/docs/tutorials/security/apparmor/#securing-a-pod). This feature is tracked in issue #4331 (https://github.com/DataBiosphere/toil/issues/4331) 🦋.

---

✨ Running in AWS ✨

Toil jobs can be run on a variety of cloud platforms. Of these, Amazon Web Services (AWS) is currently the best-supported solution 🏆. Toil provides the Toil Cluster Utilities (#clusterutils) to conveniently create AWS clusters, connect to the leader of the cluster, and then launch a workflow. The leader handles distributing the jobs over the worker nodes and autoscaling to optimize costs 💸.

The Running a Workflow with Autoscaling section details how to create a cluster and run a workflow that will dynamically scale depending on the workflow's needs.

The Static Provisioning section explains how a static cluster (one that won't automatically change in size) can be created and provisioned (grown, shrunk, destroyed, etc.).

---

✨ Preparing your AWS environment ✨

To use Amazon Web Services (AWS) to run Toil, or just to use S3 to host the files during the computation of a workflow, first set up and configure an account with AWS 💖:

1. If necessary, create and activate an AWS account (https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/).
2. Next, generate a key pair for AWS with this command (do NOT generate your key pair with the Amazon browser 🙅‍♀️):

$ ssh-keygen -t rsa

3. This should prompt you to save your key. Please save it in ~/.ssh/id_rsa.
4. Now move it to where your OS can see it as an authorized key:

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

5. Next, you'll need to add your key to the ssh-agent:

$ eval `ssh-agent -s`
$ ssh-add

If your key has a passphrase, you'll be prompted to enter it here once 🔑.

6. You'll also need to chmod your private key (good practice, but also enforced by AWS):

$ chmod 400 id_rsa

7. Now you'll need to add the key to AWS via the browser. For example, on us-west1, this address would be accessible at:

https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#KeyPairs:sort=keyName

8. Now click on the "Import Key Pair" button to add your key. (The original page has a screenshot here titled Adding an Amazon Key Pair, linked at https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#KeyPairs:sort=keyName.)
9. Next, you need to create an AWS access key. First go back to the IAM dashboard. For "us-west1", the example link would be here:

https://console.aws.amazon.com/iam/home?region=us-west-1#/home

10. The directions (transcribed from https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html) are now:
    1. On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.
2. Expand the Access keys section (that's your access key ID and secret access key) 🔑
   3. Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. 💾 Once you close the dialog box, you can't retrieve this secret access key again.

🚨 Bestie, listen: once that dialog closes, the secret access key is gone forever. Save the file before you close anything! 😭

11. Okay, you should now have a brand new "AWS Access Key ID" and "AWS Secret Access Key" 🥳 Next we install the AWS CLI and make sure it has the proper credentials:

$ pip install awscli --upgrade --user

12. Now configure your AWS credentials with:

$ aws configure

13. Add your "AWS Access Key ID" and "AWS Secret Access Key" from earlier, plus your region and output format. ✨

" AWS Access Key ID [****************Q65Q]: "
" AWS Secret Access Key [****************G0ys]: "
" Default region name [us-west-1]: "
" Default output format [json]: "

This creates the files ~/.aws/config and ~/.aws/credentials. 🗂️

14. If you haven't already, install toil 🧰 (this example uses version 5.12.0, but we recommend the latest release):

$ virtualenv venv
$ source venv/bin/activate
$ pip install toil[all]==5.12.0

15. Now that toil is installed and you're running a virtualenv, here's an example of launching a toil leader node. 🚀 Heads up: in this example we set TOIL_APPLIANCE_SELF to toil version 5.3.0, but please set the version to the one you actually have installed if you're using a different version. 💅

$ toil launch-cluster <cluster-name> \
      --clusterType kubernetes \
      --leaderNodeType t2.medium \
      --nodeTypes t2.medium -w 1 \
      --zone us-west-1a \
      --keyPairName id_rsa

To break down each of these commands, bestie: 🫶

• toil launch-cluster — the base command for launching a cluster in toil. 🚀
• <cluster-name> — just pick a name for your cluster. 🏷️
• --clusterType kubernetes — picks the type of cluster that coordinates and runs your workflow. Kubernetes is the recommended option. 👑
• --leaderNodeType t2.medium — sets the leader node type. A t2.medium is 2CPU, 4Gb RAM, and $0.0464/Hour. 💰 Here's the full list of available AWS instances: https://aws.amazon.com/ec2/pricing/on-demand/
• --nodeTypes t2.medium -w 1 — sets the worker node type and how many worker nodes to launch. A Kubernetes cluster needs at least 1 worker node. 🐝
• --zone us-west-1a — the AWS zone you want to launch the instance in. It has to share the same prefix as the zone in your awscli credentials (in this tutorial, that's us-west-1). 📍
• --keyPairName id_rsa — the name of your key pair. It should be id_rsa if you followed this tutorial. 🔑

🏷️ Tag it, bestie: you can set the TOIL_AWS_TAGS environment variable to a JSON object to add arbitrary tags to your AWS resources. For example, if you run export TOIL_AWS_TAGS='{"project-name": "variant-calling"}' in your shell before using Toil, any AWS resources Toil creates get a project-name tag with the value variant-calling. 🦋

🐳 Image swap: you can also set the TOIL_APPLIANCE_SELF environment variable to one of the Toil project's Docker images at https://quay.io/repository/ucsc_cgl/toil?tag=latest&tab=tags if you want to launch a cluster using a different version of Toil than the one you have installed. 🌈

---

✨ AWS Job Store ✨

Using the AWS job store is super straightforward once you've finished Preparing your AWS environment (#awsprep). All you have to do is specify the prefix for the job store name. 🫧

To run the sort example (#sortexample) with the AWS job store, you'd type:

$ python3 sort.py aws:us-west-2:my-aws-sort-jobstore

---

🌸 Toil Provisioner 🌸

The Toil provisioner is the component responsible for creating resources in Amazon's cloud. ☁️ It's included in Toil alongside the [aws] extra, and it lets us spin up a cluster. 🚀

Getting started with the provisioner is simple, I promise: 💖

1. Make sure you have Toil installed with the AWS extras. For detailed instructions, see Installing Toil with Extra Features (#extras).
2. You'll need an AWS account, and you'll need to save your AWS credentials on your local machine. 🔐 For help setting up an AWS account, see here: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html. For setting up your AWS credentials, follow the instructions here: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files.

The Toil provisioner makes heavy use of the Toil Appliance, a Docker image that bundles Toil and all its requirements (e.g. Kubernetes). 🐳 This makes deployment simple across platforms, and you can even simulate a cluster locally. (See Developing with Docker (#appliance-dev) for details.) 🧚

🎀 Choosing Toil Appliance Image 🎀

When you use the Toil provisioner, the appliance image gets chosen automatically based on the pip-installed version of Toil on your system. 🫶 You can override that choice by setting the environment variables TOIL_DOCKER_REGISTRY and TOIL_DOCKER_NAME or TOIL_APPLIANCE_SELF. See Environment Variables (#envars) for more information on these variables. If you're developing with autoscaling and want to test and build your own appliance, have a look at Developing with Docker (#appliance-dev).

For information on using the Toil Provisioner, have a look at Running a Workflow with Autoscaling. 🌈

---

🚀 Details about Launching a Cluster in AWS 🚀

Using the provisioner to launch a Toil leader instance is simple with the launch-cluster command. For example, to launch a Kubernetes cluster named "my-cluster" with a t2.medium leader in the us-west-2a zone, run:

(venv) $ toil launch-cluster my-cluster \
             --clusterType kubernetes \
             --leaderNodeType t2.medium \
             --nodeTypes t2.medium -w 1 \
             --zone us-west-2a \
             --keyPairName <AWS-key-pair-name>

The cluster name uniquely identifies your cluster, and it's used to fill in the instance's Name tag. 🏷️ The Toil provisioner also automatically tags your cluster with an Owner tag that matches your keypair name, which helps with cost tracking. 💸 On top of that, the ToilNodeType tag lets you filter "leader" vs. "worker" nodes in your cluster. 🔍

The leaderNodeType is an EC2 instance type (https://aws.amazon.com/ec2/instance-types/). It only affects the leader node. 👑

The --zone parameter specifies which EC2 availability zone to launch the cluster in. You can also set this with the TOIL_AWS_ZONE environment variable. 📍 Note: a zone is not the same thing as an EC2 region! 🙈 A region is a geographical area like us-west-2 (Oregon), and availability zones are partitions of that area, like us-west-2a.

By default, Toil creates an IAM role for each cluster with enough permissions to do cluster stuff (e.g. full S3, EC2, and SDB access). 🛡️ If the default permissions aren't enough for your use case (say you need access to ECR), you can create a custom IAM role with all the permissions you need and set the --awsEc2ProfileArn parameter when launching the cluster. Your custom role must at least have these permissions (#minawspermissions) for the Toil cluster to function properly.

Toil also creates a new security group with the same name as the cluster name, with default rules (e.g. it opens port 22 for SSH access). 🔓 If you need additional security groups, you can use the --awsEc2ExtraSecurityGroupId parameter when launching the cluster.

💀 Danger zone, bestie: do not use the same name as the cluster name for your extra security groups! Any security group matching the cluster name will be deleted once the cluster is destroyed. 😱

For more info on options, try:

(venv) $ toil launch-cluster --help

---

🧚 Static Provisioning 🧚

Toil can manage a cluster in the cloud using the Toil Cluster Utilities (#clusterutils). The cluster utilities also make it super easy to run a toil workflow directly on that cluster. 🫶 We call this static provisioning because the size of the cluster never changes. That's in contrast to Running a Workflow with Autoscaling. 📏

To launch worker nodes alongside the leader, use the -w option:

(venv) $ toil launch-cluster my-cluster \
             --clusterType kubernetes \
             --leaderNodeType t2.small -z us-west-2a \
             --keyPairName <AWS-key-pair-name> \
             --nodeTypes m3.large,t2.micro -w 1,4 \
             --zone us-west-2a

This spins up a t2.small leader node with five additional workers: one m3.large instance and four t2.micro. 🐝

Currently, static provisioning is only possible during the cluster's creation. 🙃 Adding and removing nodes through the native provisioner is still in development. Of course, you can always delete the cluster with the Destroy-Cluster Command (#destroycluster) utility. 🗑️

---

📦 Uploading Workflows 📦

Now that our cluster is launched, we use the Rsync-Cluster Command (#rsynccluster) utility to copy the workflow to the leader. For a simple workflow in a single file, it might look like this:

(venv) $ toil rsync-cluster -z us-west-2a my-cluster toil-workflow.py :/

📚 If your toil workflow has dependencies, have a look at the Auto-Deployment (#autodeploying) section for a detailed explanation of how to include them.

---

🌈 Running a Workflow with Autoscaling 🌈

Toil can create an autoscaling Kubernetes cluster for you using the AWS provisioner. 🤩 Autoscaling is a feature of running Toil in the cloud where extra cloud instances get launched as needed to run the workflow. 📈

🛠️ Make sure you've done the AWS setup in Preparing your AWS environment first!

To set up a Kubernetes cluster, just use the --clusterType=kubernetes command line option with toil launch-cluster. To make it autoscale, specify a range of possible node counts for a node type (such as -w 1-4). The cluster will automatically add and remove nodes within that range, depending on how many it needs to run the jobs submitted to the cluster. 🦋

For example, to launch a Toil cluster with a Kubernetes scheduler, run:

Okay so the full launch command looks like this, bestie 💅

(venv) $ toil launch-cluster <cluster-name> \
        --provisioner=aws \
        --clusterType kubernetes \
        --zone us-west-2a \
        --keyPairName <AWS-key-pair-name> \
        --leaderNodeType t2.medium \
        --leaderStorage 50 \
        --nodeTypes t2.medium -w 1-4 \
        --nodeStorage 20 \
        --logDebug

Behind the scenes, Toil slides in kubeadm and sets up the kubelet on the Toil leader and every worker node 🔧 Once that's done, your cluster can schedule jobs with Kubernetes ✨

Note: You need at least one worker node, or Kubernetes won't have anywhere to schedule any jobs 😭 And yes, it's totally normal for this step to take a while.

For our demo we're bringing back sort.py 🦋 This time we run it on a Toil cluster with Kubernetes. First, download the file and drop it in your current working directory.

Next, copy the workflow file over and SSH into the cluster:

(venv) $ toil rsync-cluster -z us-west-2a <cluster-name> sort.py :/root
(venv) $ toil ssh-cluster -z us-west-2a <cluster-name>

Swap <cluster-name> for your real cluster name 💖 You can totally use your own cluster config and workflow files too. For more on this step, check the corresponding section of the Static Provisioning tutorial.

Important: There are some important caveats about kicking off a toil run through an ssh session. They're covered in the Ssh-Cluster Command section (#sshcluster), so go read that before you do anything wild 🙈

Now that you're inside the cluster, a Kubernetes environment should already be configured and running ✨ To check, just run:

$ kubectl get nodes

You should see a leader node showing the Ready status. Depending on how many worker nodes you asked for upfront, you should see them listed here too 🌸

You can also check that the metrics server is running:

$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

If you get a JSON response (kinda like the output below) and no errors, the metrics server is set up and running, and you're good to start running workflows 💅

{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1", ...}

Note: All the nodes take a bit to get set up, so you might not see every single one running right away. You can start running workflows anyway, but Toil might throw a fit until the resources it needs are up and running 🙃

Time to run the workflow:

$ python3 sort.py \
        --batchSystem kubernetes \
        aws:<region>:<job-store-name>

Make sure you replace <region> and <job-store-name> 🫶 When you're using the Kubernetes batch system, you have to use a cloud-accessible job store, like AWS or Google.

The sort workflow should kick off on the Kubernetes cluster Toil set up for you ✨ This one takes a while to finish, so you can stick the job in the background and keep an eye on things with kubectl. For example, you can check out the pods that are running:

$ kubectl get pods

You should see something like this:

NAME                                                      READY   STATUS              RESTARTS   AGE
root-toil-a864e1b0-2e1f-48db-953c-038e5ad293c7-11-4cwdl   0/1     ContainerCreating   0          85s
root-toil-a864e1b0-2e1f-48db-953c-038e5ad293c7-14-5dqtk   0/1     Completed           0          18s
root-toil-a864e1b0-2e1f-48db-953c-038e5ad293c7-7-gkwc9    0/1     ContainerCreating   0          107s
root-toil-a864e1b0-2e1f-48db-953c-038e5ad293c7-9-t7vsb    1/1     Running             0          96s

If a pod fails for whatever reason, or you just want to make sure one isn't stuck, use kubectl describe pod <pod-name> or kubectl logs <pod-name> to inspect it 🔍

If everything went well, you should see an output file from the sort workflow:

$ head sortedFile.txt

You can now run your own workflows! 🎀👑

---

✨ Preemptibility ✨

Toil can run on a heterogeneous cluster that mixes preemptible and non-preemptible nodes 🌈 A preemptible node is basically one that can get shut down at any time, even while jobs are running. Those jobs can then be restarted later somewhere else 🦋

To make a node type preemptible, add a spot bid in dollars after a colon, right on its entry in the list of node types you pass to the --nodeTypes flag. The bid info lives at https://aws.amazon.com/ec2/spot/pricing/. If spot instance prices go above your bid, the preemptible nodes get shut down 💀

For example, this cluster has both preemptible and non-preemptible nodes:

(venv) $ toil launch-cluster <cluster-name> \
        --provisioner=aws \
        --clusterType kubernetes \
        --zone us-west-2a \
        --keyPairName <AWS-key-pair-name> \
        --leaderNodeType t2.medium \
        --leaderStorage 50 \
        --nodeTypes t2.medium -w 1-4 \
        --nodeTypes t2.large:0.20 -w 1-4 \
        --nodeStorage 20 \
        --logDebug

Individual jobs can explicitly say whether they should run on preemptible nodes, using the boolean preemptible resource requirement in Toil's Python API 🧚 In CWL, this shows up as a hint https://doc.arvados.org/user/cwl/cwl-extensions.html called UsePreemptible in the http://arvados.org/cwl# namespace (usually imported as arv). In WDL, it's a runtime attribute https://cromwell.readthedocs.io/en/stable/RuntimeAttributes/#preemptible called preemptible, as recognized by Cromwell. Toil's Kubernetes batch system will prefer to put preemptible jobs on preemptible nodes 💜

If a job doesn't say it's preemptible, it won't run on preemptible nodes, even when they're available, unless you run the workflow with the --defaultPreemptible flag. That flag lets jobs without an explicit preemptible requirement run on preemptible machines. For example:

$ python3 /root/sort.py aws:us-west-2:<my-jobstore-name> \
        --batchSystem kubernetes \
        --defaultPreemptible

Specify Preemptibility Carefully 😭 Make sure your choices for --nodeTypes and --maxNodes <> actually make sense for your workflow, or it could hang. The provisioner has to be able to create nodes big enough for the largest job in the workflow, and non-preemptible node types have to be allowed if the workflow has non-preemptible jobs.

---

✨ Using MinIO and S3-Compatible object stores ✨

Toil can access files stored in an S3-compatible object store, like https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services or https://min.io/ (MinIO) 🌸 Use these environment variables to configure the S3 connection:

• TOIL_S3_HOST — the IP address or hostname to use for connecting to S3
• TOIL_S3_PORT — the port number to use for connecting to S3, if needed
• TOIL_S3_USE_SSL — turns SSL on or off for connecting to S3 (True by default)

Examples:

TOIL_S3_HOST=127.0.0.1
TOIL_S3_PORT=9010
TOIL_S3_USE_SSL=False

---

✨ In-Workflow Autoscaling with Mesos ✨

Instead of the usual Kubernetes-based autoscaling, you can use Toil's older Mesos-based method, where the scaling logic runs inside the Toil workflow itself 🧠 With this approach, a Toil cluster can only run one workflow at a time. It also doesn't work on the ARM architecture 🙈

In this mode, the --preemptibleCompensation flag helps when preemptible nodes aren't available but your workflow needs them. With this flag on, the autoscaler tries to make up for a shortage of preemptible nodes of a certain type by creating non-preemptible nodes of that type, but only if non-preemptible nodes of that type were specified in --nodeTypes.

Note: This approach is deprecated, because the Mesos project isn't publishing up-to-date builds anymore 🥀

1. Download sort.py
2. Launch a Mesos leader node in AWS using the Launch-Cluster Command (#launchcluster) command, without using any ranges of node counts:

(venv) $ toil launch-cluster <cluster-name> \
        --clusterType mesos \
        --keyPairName <AWS-key-pair-name> \
        --leaderNodeType t2.medium \
        --zone us-west-2a

3. Copy the sort.py workflow up to the leader node:

(venv) $ toil rsync-cluster -z us-west-2a <cluster-name> sort.py :/root

4. Log in to the leader node:

(venv) $ toil ssh-cluster -z us-west-2a <cluster-name>

5. Run the workflow with in-workflow autoscaling, specifying a provisioner, node types, and counts as workflow arguments:

$ python3 /root/sort.py aws:us-west-2:<my-jobstore-name> \
        --provisioner aws \
        --nodeTypes c3.large \
        --maxNodes 2 \
        --batchSystem mesos

Note: In this example, the autoscaling Toil code creates up to two c3.large instances and launches Mesos agent containers inside them 🫶 Those containers can then run the jobs defined by the sort.py workflow. Toil also creates an S3 bucket called aws:us-west-2:autoscaling-sort-jobstore to store intermediate job results. The Toil autoscaler can provision several different node types too, which is super useful for workflows with jobs that need different resources. For example, you could run the workflow with --nodeTypes c3.large,r3.xlarge --maxNodes 5,1, which lets the provisioner create up to five c3.large nodes and one r3.xlarge node for the memory-hungry jobs. In that case, the autoscaler holds off on creating the pricier r3.xlarge node until it's actually needed, and runs most jobs on the c3.large nodes 💅

1. View the generated file to sort:

$ head fileToSort.txt

2. View the sorted file:

$ head sortedFile.txt

---

✨ Dashboard ✨

Toil has a dashboard for checking the RAM and CPU usage of each node, the number of issued jobs of each type, the number of failed jobs, and the size of the jobs queue 📊 To launch it for a Toil workflow, pass the --metrics flag on the workflow's command line. Then, while you're connected to the leader node through toil ssh-cluster, you can view the dashboard in your browser at localhost:3000 👑

To change the default port, use the --grafana_port argument:

(venv) $ toil ssh-cluster -z us-west-2a --grafana_port 8000 <cluster-name>

On AWS, the dashboard is literally keeping tabs on every node in your cluster 👀 watching CPU and RAM usage like a hawk 🦅 But ngl, it also works when you're running a workflow on just one machine 💅 The dashboard uses Grafana as the front end for real-time plots ✨ and Prometheus for tracking the metrics toil exports 📊: [image]

Bestie, if you want the dashboard for a non-released toil version 🙈 you'll have to build the containers locally with make docker. The reason is that the prometheus, grafana, and mtail containers used in the dashboard are tied to a specific toil version 🔒

---

🌸 Running in Google Compute Engine (GCE) 🌸

Okay so toil has a provisioner for Google 🫶 and a Google Job Store too! To get started, follow the instructions for Preparing your Google environment.

---

🔑 Preparing your Google environment

Toil supports using the Google Cloud Platform https://cloud.google.com/storage/. Setting this up is easy-peasy 🧚

1. Make sure the google extra (Installing Toil with Extra Features #extras) is installed
2. Follow Google's Instructions (https://cloud.google.com/docs/authentication/getting-started) to download credentials and set the GOOGLE_APPLICATION_CREDENTIALS environment variable
3. Create a new ssh key with the proper format. To make one, run this command:

$ ssh-keygen -t rsa -f ~/.ssh/id_rsa -C [USERNAME]

where [USERNAME] is something like jane@example.com. Make sure to leave your password blank.

Warning: This command could overwrite an old ssh key you may be using. If you have an existing ssh key you would like to use, it will need to be called id_rsa and it needs to have no password set.

Make sure only you can read the SSH keys 🔐:

$ chmod 400 ~/.ssh/id_rsa ~/.ssh/id_rsa.pub

4. Add your newly formatted public key to Google 💜 Log into your Google Cloud account and head to the metadata section (https://console.cloud.google.com/compute/metadata) under the Compute tab. [image]

Near the top of the screen, click on 'SSH Keys', then edit, add item, and paste the key. Then save: [image]

For more details, look at Google's instructions for adding SSH keys (https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys).

---

🗝️ Google Job Store

To use the Google Job Store, you'll need to set GOOGLE_APPLICATION_CREDENTIALS by following Google's instructions (https://cloud.google.com/docs/authentication/getting-started).

Then, to run the sort example with the Google job store, you would type:

$ python3 sort.py google:my-project-id:my-google-sort-jobstore

---

🚀 Running a Workflow with Autoscaling

Warning: Google Autoscaling is in beta!

The steps to run a GCE workflow are super similar to the AWS ones (Running a Workflow with Autoscaling #autoscaling), except you'll need to explicitly specify the --provisioner gce option. Otherwise it defaults to aws, so don't skip it 🙅‍♀️

1. Download sort.py
2. Launch the leader node in GCE using the Launch-Cluster Command (#launchcluster) command:

(venv) $ toil launch-cluster <CLUSTER-NAME> \
             --provisioner gce \
             --leaderNodeType n1-standard-1 \
             --keyPairName <SSH-KEYNAME> \
             --zone us-west1-a

Where <SSH-KEYNAME> is the first part of [USERNAME] you used when setting up your ssh key. For example, if [USERNAME] was <jane@example.com>, then <SSH-KEYNAME> should be jane.

The --keyPairName option is for an SSH key that was added to the Google account. If your ssh key [USERNAME] was jane@example.com, then your key pair name will be just jane 💖

3. Upload the sort example and ssh into the leader:

(venv) $ toil rsync-cluster --provisioner gce <CLUSTER-NAME> sort.py :/root
(venv) $ toil ssh-cluster --provisioner gce <CLUSTER-NAME>

4. Run the workflow 🔥:

$ python3 /root/sort.py  google:<PROJECT-ID>:<JOBSTORE-NAME> \
      --provisioner gce \
      --batchSystem mesos \
      --nodeTypes n1-standard-2 \
      --maxNodes 2

5. Clean up 🧹:

$ exit  # this exits the ssh from the leader node
(venv) $ toil destroy-cluster --provisioner gce <CLUSTER-NAME>

✨ HPC ENVIRONMENTS ✨

Okay so Toil is a super flexible framework 💅 that can be used in a bunch of different environments, including high-performance computing (HPC) environments 🖥️ It supports a whole squad of batch systems, like Grid Engine (http://www.univa.com/oracle), Slurm (https://www.schedmd.com/), Torque (http://www.adaptivecomputing.com/products/open-source/torque/), and LSF (https://en.wikipedia.org/wiki/Platform_LSF). These are the popular schedulers you'll find in HPC land ✨ Toil also supports HTCondor (https://research.cs.wisc.edu/htcondor/), which is a fave for high-throughput computing (HTC) 🧚 To use one of these batch systems, specify the --batchSystem argument to the workflow.

Due to the cost and complexity of keeping support for these schedulers up, we currently consider everything except Slurm to be "community supported" 🫶 That means the core dev team doesn't regularly test or develop support for them. BUT there are members of the Toil community deploying Toil in all kinds of HPC environments right now, and we 100% welcome external contributions 💜

Want to add support for a new or existing batch system? Then you'll be extending the abstract batch system class toil.batchSystems.abstractBatchSystem.AbstractBatchSystem 🔥

🌊 Running on Slurm 🌊

When you run Toil workflows on Slurm, you usually want to run the workflow itself from the head node 👑 Toil takes care of running all the required sbatch commands for you. You probably do not want to submit the Toil workflow as a Slurm job with sbatch (although you can if you have a large number of workflows to run 🙈). You also probably do not want to manually allocate resources with sallocate 💀

To run a Toil workflow on Slurm, include --batchSystem slurm in your command line arguments. Slurm clusters generally have shared filesystems, which means the file job store would be appropriate 🦋 Just make sure your job store location is shared across your whole Slurm cluster. You'll also likely want another shared directory via the --batchLogsDir option, so Toil can retrieve the Slurm job logs if something goes wrong with a job 🍓

For example, to run the sort example on Slurm (assuming you're in a shared directory), you'd type this on the cluster head node:

$ mkdir -p logs
$ python3 sort.py ./store --batchSystem slurm --batchLogsDir ./logs

By default, this does not include any time limit or particular Slurm partition. If your Slurm cluster requires time limits, add the --slurmTime option to set the time limit for jobs ⏰

If you do specify a time limit, a partition will be automatically selected that can fit jobs of that duration, and a partition will be automatically selected for jobs that need GPUs 🎀 To use a particular partition, use the --slurmPartition argument. If your GPU jobs need to go to a different partition, use the --slurmGPUPartition argument. For example, to run a WDL workflow from Dockstore using GPUs on Slurm, with a 4 hour time limit per job and partitions manually specified, you'd run:

$ toil-wdl-runner '#workflow/github.com/vgteam/vg_wdl/GiraffeDeepVariantFromGAF:gbz' \
  https://raw.githubusercontent.com/vgteam/vg_wdl/refs/heads/gbz/params/giraffe_and_deepvariant_gaf.json \
  --jobStore ./store --batchSystem slurm --batchLogsDir ./logs \
  --slurmTime 4:00:00 --slurmPartition medium --slurmGPUPartition gpu

Any extra sbatch arguments your cluster needs that Toil doesn't directly support can be passed via the Toil --slurmArgs option, or the TOIL_SLURM_ARGS environment variable 🧚 (Heads up: there's special handling for some Slurm options Toil needs to interact with, like --time or --partition, if they get passed this way ⚠️)

💅 Slurm Tips 💅

1. If you're running containers with Singularity on Slurm (like in WDL workflows), make sure Singularity caching and Toil's MiniWDL caching use a shared directory across your cluster nodes 🌈 By default, Toil configures Singularity to cache per-workflow and per-node, but on Slurm a shared filesystem is almost always available. Assuming your home directory is shared, you can set this up like so:

$ echo 'export SINGULARITY_CACHEDIR="${HOME}/.singularity/cache"' >>~/.bashrc
$ echo 'export MINIWDL__SINGULARITY__IMAGE_CACHE="${HOME}/.cache/miniwdl"' >>~/.bashrc

Then log out and back in again so the setting takes effect 🔄

2. If your home directory is not shared across the cluster nodes, make sure you've installed Toil in such a way that it's in your PATH on the cluster nodes 🗺️
3. Slurm sandboxing and resource limits do not apply to Docker containers 💀 There's no relationship between the sandbox cgroup your Toil job runs in and the sandbox cgroup the Docker daemon creates for the container your job requested. If you want your Toil jobs' containers to actually live inside their Slurm job resource allocations, run containers with Singularity or another user-mode or daemon-less containerization system 🦋
4. Slurm can sometimes report that a job has finished before that job's changes to the cluster's shared filesystem are visible to other nodes or the head node 😭 Toil tries to anticipate and compensate for this, but there's no amount of waiting or retrying that could guarantee correct behavior in theory. In theory, the shared filesystem could be days or months behind! In practice the delay is usually no more than a few seconds, and Toil can handle that 💖 But if you're seeing odd behavior from Toil where files don't exist when they should, or still exist when they shouldn't, your cluster's filesystem might just be unusually slow to reach consistency across nodes 🐢
5. If you see warnings about XDG_RUNTIME_DIR, your Slurm cluster might not be managing XDG login sessions correctly for Slurm jobs 🥺 Toil can work around this, but as a result the workaround might have trouble finding an appropriate "coordination directory" where it can store state files local to each Slurm node. If you're seeing unusual stuff like Toil jobs on one node waiting on operations on a different node, you can try giving Toil a path to a per-node, writable directory with the --coordinationDir option, so it knows exactly where to put those files 🗂️
6. With a shared filesystem, Toil's caching system is not necessarily going to help your workflow 🤷‍♀️ Try running and timing test workflows with --caching true and with --caching false to figure out whether copying files from the shared filesystem to local storage on each node is worth it for your workload ⏱️
7. If you're running CWL workflows on Slurm with a shared filesystem, you can try the --bypass-file-store option to toil-cwl-runner ✨ It may speed up your workflow, but you may also need to change Toil's work directory to a shared directory provided with the --workDir option so it works properly across machines 🏃‍♀️

📂 Standard Output/Error from Batch System Jobs 📂

Standard output and error from batch system jobs (except for the Mesos batch system) get redirected to files in the toil-<workflowID> directory, which is created inside the temporary directory specified by the --workDir option 🧁 (See COMMANDLINE OPTIONS for more on that one 💜) Each file is named like this: toil_job_<Toil job ID>_batch_<name of batch system>_<job ID from batch system>_<file description>.log, where <file description> is std_output for standard output and std_error for standard error. HTCondor will also write job event log files, with <file description> set to job_events 📜

If you want to capture standard output and error, --workDir will generally need to be on a shared file system 🗄️ Otherwise, if these get written to local temporary directories on each node (like /tmp), Toil won't be able to retrieve them 😭 Alternatively, the --noStdOutErr option forces Toil to discard all standard output and error from batch system jobs 🗑️

---

🌐 WORKFLOW EXECUTION SERVICE (WES) 🌐

The GA4GH Workflow Execution Service (WES) is a standardized API for submitting and monitoring workflows 🦋 Toil has experimental support for setting up a WES server and executing CWL, WDL, and Toil workflows through the WES API 🧚 You can read more about the WES API specification here: https://ga4gh.github.io/workflow-execution-service-schemas/docs/

To get started with the Toil WES server, make sure the server extra is installed (see Installing Toil with Extra Features) 🎀

🐰 Preparing your WES environment 🐰

The WES server needs Celery (https://docs.celeryproject.org/en/stable/getting-started/introduction.html) to distribute and execute workflows 💖 To set up Celery:

1. Start RabbitMQ, which is the broker between the WES server and the Celery workers 🐇

docker run -d --name wes-rabbitmq -p 5672:5672 rabbitmq:3.9.5

2. Start the Celery workers 👷‍♀️

celery -A toil.server.celery_app worker --loglevel=INFO

🚀 Starting a WES server 🚀

To start a WES server on the default port 8080, run the Toil command:

$ toil server

The WES API will be hosted at this URL:

http://localhost:8080/ga4gh/wes/v1

Want a different port, like 3000? Just specify the --port argument:

$ toil server --port 3000

There are lots of other command line options too 🌸 To find help info, run this command:

$ toil server --help

Here's a detailed rundown of all the server-specific options 👑

• --debug — Turns on debug mode 🐞
• --bypass_celery — Skips sending workflows to Celery and just runs them under the server. This one's for testing 🧪
• --host HOST — The host interface the Toil server binds to (default: "127.0.0.1") 🏠
• --port PORT — The port the Toil server listens on (default: 8080) 🔌
• --swagger_ui — If True, the swagger UI gets enabled and hosted at the {api_base_path}/ui endpoint (default: False) 📖
• --cors — Enables Cross Origin Resource Sharing (CORS). Only turn this on if the server is meant to be used by a website or domain 🌍 (default: False).
• --cors_origins CORS_ORIGIN — Ignored if --cors is False. This sets the allowed origins for CORS. For details about CORS and its security risks, check the GA4GH docs on CORS at https://w3id.org/ga4gh/product-approval-support/cors ⚠️ (default: "*").
• --workers WORKERS, -w WORKERS — Ignored if --debug is True 🙈 Otherwise this sets how many worker processes the WSGI server launches. Default is 2 💖.
• --work_dir WORK_DIR — 📁 This is where your workflows get stored. Ideally the folder is empty, or only has old workflows in it 🧹. Default is './workflows' 🌸.
• --state_store STATE_STORE — ☁️ The local path or S3 URL where workflow state metadata gets saved. Default is in --work_dir ✨.
• --opt OPT, -o OPT — 🎀 Set the default parameters that get sent to the workflow engine for every run. Options that take arguments have to use = syntax. You can pass this more than once. Example: --opt=--logLevel=CRITICAL --opt=--workDir=/tmp 💅.
• --dest_bucket_base DEST_BUCKET_BASE — 🪣 Tells CWL workflows to save output files to dynamically generated unique paths under the URL you give. Supports AWS S3 🦋.
• --wes_dialect DIALECT — 🧚 Limits WES responses to a dialect that works with clients that don't fully implement the WES standard. Default is 'standard' 👑.

---

✨ Running the Server with docker-compose ✨

Instead of setting up the server pieces by hand (toil server, RabbitMQ, and Celery), you can use the docker-compose.yml file below to orchestrate and link them all together 🫶.

Make sure to change the credentials for basic authentication by updating the traefik.http.middlewares.auth.basicauth.users label 🔐. You can generate the passwords with tools like htpasswd. Here's the guide: https://doc.traefik.io/traefik/v2.0/middlewares/basicauth/#configuration-examples. (Heads up: single $ signs need to be replaced with $$ in the yaml file 🙈.)

When you're running on a host other than localhost, change the Host to your target host in the traefik.http.routers.wes.rule and traefik.http.routers.wespublic.rule labels 🌈.

You can also change /tmp/toil-workflows if you want Toil workflows to live somewhere else. Just create the directory before starting the server 📂.

To run workflows that need Docker, the docker.sock socket has to be mounted as a volume for Celery 🐳. Also, the TOIL_WORKDIR directory (defaults to: /var/lib/toil) and /var/lib/cwl (if you're running CWL workflows with DockerRequirement) should exist on the host and get mounted as volumes too 💜.

🚨 Run this behind a firewall, bestie! It opens up the Toil server on port 8080 to anyone who connects. Not a vibe 💀.

# docker-compose.yml
version: "3.8"

services:
  rabbitmq:
    image: rabbitmq:3.9.5
    hostname: rabbitmq
  celery:
    image: ${TOIL_APPLIANCE_SELF}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker:/var/lib/docker
      - /var/lib/toil:/var/lib/toil
      - /var/lib/cwl:/var/lib/cwl
      - /tmp/toil-workflows:/tmp/toil-workflows
    command: celery --broker=amqp://guest:guest@rabbitmq:5672// -A toil.server.celery_app worker --loglevel=INFO
    depends_on:
      - rabbitmq
  wes-server:
    image: ${TOIL_APPLIANCE_SELF}
    volumes:
      - /tmp/toil-workflows:/tmp/toil-workflows
    environment:
      - TOIL_WES_BROKER_URL=amqp://guest:guest@rabbitmq:5672//
    command: toil server --host 0.0.0.0 --port 8000 --work_dir /tmp/toil-workflows
    expose:
      - 8000
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.wes.rule=Host(`localhost`)"
      - "traefik.http.routers.wes.entrypoints=web"
      - "traefik.http.routers.wes.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=test:$$2y$$12$$ci.4U63YX83CwkyUrjqxAucnmi2xXOIlEF6T/KdP9824f1Rf1iyNG"
      - "traefik.http.routers.wespublic.rule=Host(`localhost`) && Path(`/ga4gh/wes/v1/service-info`)"
    depends_on:
      - rabbitmq
      - celery
  traefik:
    image: traefik:v2.2
    command:
      - "--providers.docker"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:8080"
    ports:
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Want more customization? Go for it 💅. For example, if you have a domain, you can set up HTTPS with Let's Encrypt. The guide is here: https://doc.traefik.io/traefik/user-guides/docker-compose/acme-http/ 🔒.

Once everything is configured, just run docker-compose up to start the containers 🚀. Run docker-compose down to stop and remove all of them 🧹.

📝 Note: docker-compose is not installed on the Toil appliance by default. Check out the next section to set up the WES server on a Toil cluster 🫶.

---

✨ Running on a Toil cluster ✨

To run the server on a Toil leader instance on EC2:

1. Launch a Toil cluster with the toil launch-cluster command with the AWS provisioner 🚀.
2. SSH into your cluster with the --sshOption=-L8080:localhost:8080 option to forward port 8080 🦋.
3. Install Docker Compose by running the following commands from the Docker docs. The guide is here: https://docs.docker.com/compose/install/#install-compose 🐳.

curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

# check installation
docker-compose --version

Or, want a different version of Docker Compose? Just change "1.29.2" to whatever version you want 🌈.

4. Copy the docker-compose.yml file from (Running the Server with docker-compose) into an empty directory, then tweak the configuration as needed 💖.
5. Now run docker-compose up -d to start the WES server in detach mode on the Toil appliance 🌙.
6. To stop the server, run docker-compose down 🛑.

---

✨ WES API Endpoints ✨

The GA4GH WES API spec defines these endpoints, all with base path ga4gh/wes/v1/, and Toil supports every single one 👑:

• GET /service-info — ℹ️ Get info about the Workflow Execution Service.
• GET /runs — 📋 List the workflow runs.
• POST /runs — 🚀 Run a workflow. This creates a new workflow run and returns a run_id so you can keep an eye on its progress 👀.
• GET /runs/{run_id} — 🔍 Get detailed info about a workflow run.
• POST /runs/{run_id}/cancel — 🛑 Cancel a running workflow.
• GET /runs/{run_id}/status — 🚦 Get the status (overall state) of a workflow run.

When you run the WES server with the docker-compose setup above, most endpoints (everything except GET /service-info) are protected with basic authentication 🔐. Make sure to set the Authorization header with the correct credentials when you submit or retrieve a workflow 💜.

---

✨ Submitting a Workflow ✨

Now that the WES API is up and running, you can submit and monitor workflows remotely with the WES API endpoints 🧚. Submit a workflow for execution with the POST /runs endpoint 🏃‍♀️.

As a quick example, we can submit the example CWL workflow from Running a basic CWL workflow to our WES API 🎀:

# example.cwl
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
stdout: output.txt
inputs:
 message:
   type: string
   inputBinding:
     position: 1
outputs:
 output:
   type: stdout

Using cURL 🌸:

$ curl --location --request POST 'http://localhost:8080/ga4gh/wes/v1/runs' \
    --user test:test \
    --form 'workflow_url="example.cwl"' \
    --form 'workflow_type="cwl"' \
    --form 'workflow_type_version="v1.0"' \
    --form 'workflow_params="{\"message\": \"Hello world!\"}"' \
    --form 'workflow_attachment=@"./toil_test_files/example.cwl"'
{
  "run_id": "4deb8beb24894e9eb7c74b0f010305d1"
}

The --user argument attaches your basic authentication credentials to the request 🔑. Make sure to change test:test to the username and password you configured for your WES server 🫶. Or, if you'd rather, set the Authorization header yourself as "Authorization: Basic base64_encoded_auth" 💅.

If the workflow gets submitted successfully, you'll get back a JSON object containing a run_id 🎉. That run_id is a unique identifier for your requested workflow, and you can use it to monitor or cancel the run 🦋.

There are a few required parameters that have to be set for every workflow submission, so don't skip any of them, bestie 🎀

• workflow_url — the URL of the workflow you want to run. It can point to a file from workflow_attachment 🔗
• workflow_type — the workflow language. Toil currently supports "CWL", "WDL", or "py". If you're running a Toil Python workflow, set this to "py" 🐍
• workflow_type_version — the version of the workflow language. To see which versions your WES server supports, check the GET /service-info endpoint ✨
• workflow_params — a JSON object that specifies the inputs of the workflow 💖

Then there are some optional parameters you can add too 💅

• workflow_attachment — a list of files tied to the workflow run 📎
• workflow_engine_parameters — a JSON key-value map of workflow engine parameters that gets sent to the runner. Example: {"--logLevel": "INFO", "--workDir": "/tmp/"} 🦋
• tags — a JSON key-value map of metadata attached to the workflow 🏷️

For the full scoop on these parameters, check out the Run Workflow section of the WES API spec at https://ga4gh.github.io/workflow-execution-service-schemas/docs/#operation/RunWorkflow 📚

✨ Upload multiple files ✨

Okay so, look at the body of the request from the previous example. The workflow_url is a relative URL that points at the example.cwl file, which gets uploaded from the local path ./toil_test_files/example.cwl 📂

If you want to choose the file name (or subdirectory) of the remote destination file, set the filename field in the Content-Disposition header. You can also upload more than one file by passing the workflow_attachment parameter multiple times, each with a different file. It's giving organized 🌸

Here's what that looks like:

$ curl --location --request POST 'http://localhost:8080/ga4gh/wes/v1/runs' \
    --user test:test \
    --form 'workflow_url="example.cwl"' \
    --form 'workflow_type="cwl"' \
    --form 'workflow_type_version="v1.0"' \
    --form 'workflow_params="{\"message\": \"Hello world!\"}"' \
    --form 'workflow_attachment=@"./toil_test_files/example.cwl"' \
    --form 'workflow_attachment=@"./toil_test_files/2.fasta";filename=inputs/test.fasta' \
    --form 'workflow_attachment=@"./toil_test_files/2.fastq";filename=inputs/test.fastq'

On the server, the execution directory ends up with this structure after that request 🧚

execution/
├── example.cwl
├── inputs
│     ├── test.fasta
|     └── test.fastq
└── wes_inputs.json

✨ Specify Toil options ✨

To pass Toil-specific parameters to the workflow, include the workflow_engine_parameters parameter with your request 🎀

For example, if you want to set the logging level to INFO and change the working directory of the workflow, just add this as your workflow_engine_parameters:

{"--logLevel": "INFO", "--workDir": "/tmp/"}

These options get appended to the end of the existing parameters during command construction, which means they override the default parameters if you provide them. (Default parameters that can be passed multiple times won't be overridden 🙈)

✨ Monitoring a Workflow ✨

Once you submit a workflow, you get back a run_id. With that, you can check the status or pull the full logs of the workflow run 👀

✨ Checking the state ✨

The GET /runs/{run_id}/status endpoint gives you a simple result with the overall state of your run 💜

$ curl --user test:test http://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1/status
{
  "run_id": "4deb8beb24894e9eb7c74b0f010305d1",
  "state": "RUNNING"
}

The possible states are: QUEUED, INITIALIZING, RUNNING, COMPLETE, EXECUTOR_ERROR, SYSTEM_ERROR, CANCELING, and CANCELED 🌈

✨ Getting the full logs ✨

To get the detailed info about a workflow run, hit the GET /runs/{run_id} endpoint 🍓

$ curl --user test:test http://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1
{
  "run_id": "4deb8beb24894e9eb7c74b0f010305d1",
  "request": {
    "workflow_attachment": [
      "example.cwl"
    ],
    "workflow_url": "example.cwl",
    "workflow_type": "cwl",
    "workflow_type_version": "v1.0",
    "workflow_params": {
      "message": "Hello world!"
    }
  },
  "state": "RUNNING",
  "run_log": {
    "cmd": [
      "toil-cwl-runner --outdir=/home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/outputs --jobStore=file:/home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/toil_job_store /home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/execution/example.cwl /home/workflows/4deb8beb24894e9eb7c74b0f010305d1/execution/wes_inputs.json"
    ],
    "start_time": "2021-08-30T17:35:50Z",
    "end_time": null,
    "stdout": null,
    "stderr": null,
    "exit_code": null
  },
  "task_logs": [],
  "outputs": {}
}

✨ Canceling a run ✨

Want to stop a run? Use the POST /runs/{run_id}/cancel endpoint 🛑

$ curl --location --request POST 'http://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1/cancel' \
      --user test:test
{
  "run_id": "4deb8beb24894e9eb7c74b0f010305d1"
}

---

✨ DEVELOPING A PYTHON WORKFLOW ✨

This tutorial walks you through the Toil features you need to develop a workflow with the Toil Python API. 🐍💖

✨ Scripting Quick Start ✨

To start, check out this short Toil Python workflow, which shows how to define a workflow 👑

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

def helloWorld(message):
    return f"Hello, world!, here's a message: {message}"

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_quickstart")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "OFF"
    options.clean = "always"

    hello_job = Job.wrapFn(helloWorld, "Woot")

    with Toil(options) as toil:
        print(toil.start(hello_job))  # prints "Hello, world!, ..."

This workflow is made up of a single job. You can (optionally) set that job's resource requirements with the keyword arguments memory, cores, and disk. The workflow runs through toil.job.Job.Runner.getDefaultOptions() 🌟 We'll break down the components of this code in detail below.

✨ Job Basics ✨

The atomic unit of work in a Toil workflow is a Job. Your code extends this base class, or uses helper methods like toil.job.Job.addChildJobFn(), to define units of work. For example, here's a longer, class-based version of the job from the quick start example 🦋

from toil.job import Job

class HelloWorld(Job):
    def __init__(self, message):
        Job.__init__(self,  memory="2G", cores=2, disk="3G")
        self.message = message

    def run(self, fileStore):
        return f"Hello, world! Here's a message: {self.message}"

In this example, a class called HelloWorld is defined. The constructor requests 2 gigabytes of memory, 2 cores, and 3 gigabytes of local disk to complete the work 💜

The toil.job.Job.run() method is the function you override to get work done. Here, it just returns a message 🫶

You can also log a message using toil.job.Job.log(), which gets registered in the log output of the leader process of the workflow 📝

...
    def run(self, fileStore):
        self.log(f"Hello, world! Here's a message: {self.message}")

✨ Invoking a Workflow ✨

okay so we can take the previous example and make it a whole complete workflow 💅 by adding the function calls that create a HelloWorld instance and run it as a workflow with one single job. like this:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

class HelloWorld(Job):
    def __init__(self, message):
        Job.__init__(self)
        self.message = message

    def run(self, fileStore):
        return f"Hello, world!, here's a message: {self.message}"

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_invokeworkflow")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "OFF"
    options.clean = "always"

    hello_job = HelloWorld("Woot")

    with Toil(options) as toil:
        print(toil.start(hello_job))

🙈 Heads up, bestie: Do not include a . in the name of your python script (besides .py at the end). This is so toil can import the types and functions defined in your file while starting a new process.

This uses the toil.common.Toil class, and it's the thing that actually runs and resumes Toil workflows 🦋. You use it as a context manager, and it lets you do preliminary setup, like staging files into the job store on the leader node. You make an instance of it by handing it an options object. Then you invoke the actual workflow by calling toil.common.Toil.start() and passing it the root job. But if a workflow is being restarted, you use toil.common.Toil.restart() instead. Also, the context manager should have explicit if else branches for the restart and non-restart cases, and the boolean for those branches is toil.options.restart 🫶.

Here's how that looks:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

class HelloWorld(Job):
    def __init__(self, message):
        Job.__init__(self)
        self.message = message

    def run(self, fileStore):
        return f"Hello, world!, I have a message: {self.message}"

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_invokeworkflow2")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        if not toil.options.restart:
            job = HelloWorld("Woot!")
            output = toil.start(job)
        else:
            output = toil.restart()
    print(output)

The call to toil.job.Job.Runner.getDefaultOptions() is giving main character energy 👑, because it creates a whole set of default options for the workflow. The only argument is a description of how to store the workflow's state in what we call a job-store. In this example the job-store lives in a directory inside the current working directory called "toilWorkflowRun". You can also make this string encode other ways to store the necessary state, like an S3 bucket object store location 🍓. By default, the job-store gets deleted if the workflow completes successfully.

The workflow runs on the final line, where we make a HelloWorld and run it as a workflow. Every Toil workflow starts from a single starting job, called the root job 🌸. The return value of the root job becomes the result of the completed workflow. (Check out promises below to see why this is so useful!)

---

✨ Specifying Commandline Arguments ✨

If you want to control the options from the command line, you can use toil.job.Job.Runner.getDefaultArgumentParser() to make a configargparse.ArgParser object. That object can parse command line options for a Toil Python workflow, and it's honestly so clean 🧚. Here's an example:

from toil.common import Toil
from toil.job import Job

class HelloWorld(Job):
    def __init__(self, message):
        Job.__init__(self)
        self.message = message

    def run(self, fileStore):
        return "Hello, world!, here's a message: %s" % self.message

if __name__ == "__main__":
    parser = Job.Runner.getDefaultArgumentParser()
    options = parser.parse_args()
    options.logLevel = "OFF"
    options.clean = "always"

    hello_job = HelloWorld("Woot")

    with Toil(options) as toil:
        print(toil.start(hello_job))

This makes a fully fledged Toil Python workflow with every option Toil exposes as a command line argument. Running this program with --help will print the full list of options 💖.

Alternatively, any existing argparse.ArgumentParser object (see /usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser) can get Toil command line options added to it with the toil.job.Job.Runner.addToilOptions() method. BUT 😭 if it isn't also a configargparse.ArgParser, it won't read the Toil config file or the environment variables that Toil uses as alternatives to command-line options, and that can lead to unexpected behavior. Don't be that person 💀.

---

✨ Resuming a Workflow ✨

Sometimes a workflow fails. Maybe a job had a programmatic error, or maybe a node died. Either way, the workflow can be resumed 🔥. The only time it can't be reliably resumed is if the job-store itself becomes corrupt.

Here's the critical part: jobs have to be rerunnable, even if they look like they finished successfully. Put simply, a user-defined job should never corrupt its input arguments. That way, no matter whether a node, the network or the leader fails, the job can restart and the workflow can pick back up 🦋.

To resume, specify the "restart" option in the options object you pass to toil.common.Toil.start(). If you expect node failures, it can also help to use the integer "retryCount" option, which tries to rerun a job retryCount number of times before marking it fully failed.

If a small subset of jobs fails (retry attempts included), Toil keeps running the other jobs until it can't do anything else. At that point, toil.common.Toil.start() raises a toil.exceptions.FailedJobsException exception ⭐. Usually this is when you decide whether to fix the script and resume, or delete the job-store manually and rerun the whole workflow from scratch.

---

✨ Functions and Job Functions ✨

Making jobs out of class definitions means writing a constructor every single time, and ugh, boilerplate 🙄. So to skip that, toil.job.FunctionWrappingJob and toil.job.JobFunctionWrappingTarget let you turn plain functions straight into jobs. Here's the quick start example, repeated for your viewing pleasure:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

def helloWorld(message):
    return f"Hello, world!, here's a message: {message}"

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_quickstart")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "OFF"
    options.clean = "always"

    hello_job = Job.wrapFn(helloWorld, "Woot")

    with Toil(options) as toil:
        print(toil.start(hello_job))  # prints "Hello, world!, ..."

This is basically the same as the previous example, just using a function to define the job ✨.

The function call that does the wrapping is:

Job.wrapFn(helloWorld, "Woot")

It creates an instance of the toil.job.FunctionWrappingTarget that wraps the function 🧚.

The keyword arguments memory, cores and disk let you set resource requirements, same as before. Even if they aren't in the function header as keyword arguments, you can still pass them when you wrap the function as a job, and they'll be used to set the resource requirements 💅.

You can also wrap a job function, which is a function whose first argument is a reference to the wrapping job. Just like a self argument in a class, this gives you access to the methods of the wrapping job (see toil.job.JobFunctionWrappingTarget). Here's an example:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

def helloWorld(job, message):
    job.log(f"Hello world, I have a message: {message}")

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_jobfunctions")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    hello_job = Job.wrapJobFn(helloWorld, "Woot!")

    with Toil(options) as toil:
        toil.start(hello_job)

Here helloWorld() is a job function. It uses toil.job.Job.log() to log a message that gets printed to the output console 🎀. The only subtle difference to note is this line:

hello_job = Job.wrapJobFn(helloWorld, "Woot")

It uses toil.job.Job.wrapJobFn() to wrap the job function, instead of toil.job.Job.wrapFn(), which wraps a vanilla function 🔥.

---

✨ Workflows with Multiple Jobs ✨

A parent job can have child jobs and follow-on jobs 🌈. These relationships get set up with methods on the job class, like toil.job.Job.addChild() and toil.job.Job.addFollowOn().

Think of a set of jobs as the nodes in a job graph, and the child and follow-on relationships as its directed edges. If a job B sits on a directed path of child/follow-on edges from a job A, then B is a successor of A, and A is a predecessor of B. Lowkey the vocab you need for graph talk 🧠.

A parent job's child jobs run directly after the parent finishes, and they run in parallel 💜. The follow-on jobs of a job run after its child jobs and their successors have completed, and they also run in parallel. Follow-ons are super handy for cleanup tasks that should happen after a bunch of parallel child tasks are done 🧹. Here's a simple example that uses the earlier helloWorld() job function:

from toil.common import Toil
from toil.job import Job

def helloWorld(job, message):
    job.log(f"Hello world, I have a message: {message}")

if __name__ == "__main__":
    parser = Job.Runner.getDefaultArgumentParser()
    options = parser.parse_args()
    options.logLevel = "INFO"
    options.clean = "always"

    j1 = Job.wrapJobFn(helloWorld, "first")
    j2 = Job.wrapJobFn(helloWorld, "second or third")
    j3 = Job.wrapJobFn(helloWorld, "second or third")
    j4 = Job.wrapJobFn(helloWorld, "last")

    j1.addChild(j2)
    j1.addChild(j3)
    j1.addFollowOn(j4)

    with Toil(options) as toil:
        toil.start(j1)

Okay so in this example we make four jobs 💖 j1 runs first, then j2 and j3 run in parallel as children of j1 ✨, and finally j4 runs as a follow-on of j1 🦋 It's giving organized, no cap.

There are multiple shorthand functions to achieve the same workflow, for example: 🧚

from toil.common import Toil
from toil.job import Job

def helloWorld(job, message):
    job.log(f"Hello world, I have a message: {message}")

if __name__ == "__main__":
    parser = Job.Runner.getDefaultArgumentParser()
    options = parser.parse_args()
    options.logLevel = "INFO"
    options.clean = "always"

    j1 = Job.wrapJobFn(helloWorld, "first")
    j2 = j1.addChildJobFn(helloWorld, "second or third")
    j3 = j1.addChildJobFn(helloWorld, "second or third")
    j4 = j1.addFollowOnJobFn(helloWorld, "last")

    with Toil(options) as toil:
        toil.start(j1)

Equivalently defines the workflow, where the functions toil.job.Job.addChildJobFn() and toil.job.Job.addFollowOnJobFn() are used to create job functions as children or follow-ons of an earlier job. Bestie, the shorthand is chef's kiss 🎀

Job graphs are not limited to trees, and can express arbitrary directed acyclic graphs. For a precise definition of legal graphs, see toil.job.Job.checkJobGraphForDeadlocks(). The previous example could be specified as a DAG as follows: 🌸

from toil.common import Toil
from toil.job import Job

def helloWorld(job, message):
    job.log(f"Hello world, I have a message: {message}")

if __name__ == "__main__":
    parser = Job.Runner.getDefaultArgumentParser()
    options = parser.parse_args()
    options.logLevel = "INFO"
    options.clean = "always"

    j1 = Job.wrapJobFn(helloWorld, "first")
    j2 = j1.addChildJobFn(helloWorld, "second or third")
    j3 = j1.addChildJobFn(helloWorld, "second or third")
    j4 = j2.addChildJobFn(helloWorld, "last")
    j3.addChild(j4)

    with Toil(options) as toil:
        toil.start(j1)

Note the use of an extra child edge to make j4 a child of both j2 and j3. Lowkey the graph flex of the century 👑

---

Dynamic Job Creation ✨

The previous examples show a workflow being defined outside of a job. However, Toil also allows jobs to be created dynamically within jobs, which is so iconic. For example: 💅

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

def binaryStringFn(job, depth, message=""):
    if depth > 0:
        job.addChildJobFn(binaryStringFn, depth - 1, message + "0")
        job.addChildJobFn(binaryStringFn, depth - 1, message + "1")
    else:
        job.log(f"Binary string: {message}")

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_dynamic")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        toil.start(Job.wrapJobFn(binaryStringFn, depth=5))

The job function binaryStringFn logs all possible binary strings of length n (here n=5), creating a total of 2^(n+2) - 1 jobs dynamically and recursively. 🤯 Static and dynamic creation of jobs can be mixed in a Toil workflow, with jobs defined within a job or job function being created at run time. Main character energy, literally. 🦋

---

Promises 🔮

The previous example of dynamic job creation shows variables from a parent job being passed to a child job. Such forward variable passing is naturally specified by recursive invocation of successor jobs within parent jobs. This can also be achieved statically by passing around references to the return variables of jobs. In Toil this is achieved with promises, as illustrated in the following example: 🧚

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

def fn(job, i):
    job.log("i is: %s" % i, level=100)
    return i + 1

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_promises")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    j1 = Job.wrapJobFn(fn, 1)
    j2 = j1.addChildJobFn(fn, j1.rv())
    j3 = j1.addFollowOnJobFn(fn, j2.rv())

    with Toil(options) as toil:
        toil.start(j1)

Running this workflow results in three log messages from the jobs: i is 1 from j1, i is 2 from j2 and i is 3 from j3. 🎀

The return value from the first job is promised to the second job by the call to toil.job.Job.rv() in the following line: 💖

j2 = j1.addChildFn(fn, j1.rv())

The value of j1.rv() is a promise, rather than the actual return value of the function, because j1 for the given input has at that point not been evaluated. A promise (toil.job.Promise) is essentially a pointer to the return value that gets replaced by the actual return value once it has been evaluated. Therefore, when j2 is run, the promise becomes 2. Bestie, it's like a placeholder that glows up 🌈

Promises also support indexing of return values: 🌸

def parent(job):
    indexable = Job.wrapJobFn(fn)
    job.addChild(indexable)
    job.addFollowOnFn(raiseWrap, indexable.rv(2))

def raiseWrap(arg):
    raise RuntimeError(arg) # raises "2"

def fn(job):
    return (0, 1, 2, 3)

Promises can be quite useful 🫶 For example, we can combine dynamic job creation with promises to achieve a job creation process that mimics the functional patterns possible in many programming languages: 👑

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

def binaryStrings(job, depth, message=""):
    if depth > 0:
        s = [
            job.addChildJobFn(binaryStrings, depth - 1, message + "0").rv(),
            job.addChildJobFn(binaryStrings, depth - 1, message + "1").rv(),
        ]
        return job.addFollowOnFn(merge, s).rv()
    return [message]

def merge(strings):
    return strings[0] + strings[1]

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_promises2")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.loglevel = "OFF"
    options.clean = "always"

    with Toil(options) as toil:
        print(toil.start(Job.wrapJobFn(binaryStrings, depth=5)))

The return value l of the workflow is a list of all binary strings of length 10, computed recursively. Although it's a toy example, it shows how closely Toil workflows can mimic typical programming patterns. Functional programming in my Python era 💀

---

Promised Requirements 💜

Promised requirements are a special case of Promises that allow a job's return value to be used as another job's resource requirements. 🦋

This is useful when, for example, a job's storage requirement is determined by a file staged to the job store by an earlier job: 🍓

import os

from toil.common import Toil
from toil.job import Job, PromisedRequirement
from toil.lib.io import mkdtemp

def parentJob(job):
    downloadJob = Job.wrapJobFn(
        stageFn,
        "file://" + os.path.realpath(__file__),
        cores=0.1,
        memory="32M",
        disk="1M",
    )
    job.addChild(downloadJob)

    analysis = Job.wrapJobFn(
        analysisJob,
        fileStoreID=downloadJob.rv(0),
        disk=PromisedRequirement(downloadJob.rv(1)),
    )
    job.addFollowOn(analysis)

def stageFn(job, url):
    importedFile = job.fileStore.import_file(url)
    return importedFile, importedFile.size

def analysisJob(job, fileStoreID):
    # now do some analysis on the file
    pass

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_requirements")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        toil.start(Job.wrapJobFn(parentJob))

Note that this also makes use of the size attribute of the FileID object. 🧚 This promised requirements mechanism can also be used in combination with an aggregator for multiple jobs' output values: 💅

def parentJob(job):
    aggregator = []
    for fileNum in range(0, 10):
        downloadJob = Job.wrapJobFn(stageFn, "file://" + os.path.realpath(__file__), cores=0.1, memory='32M', disk='1M')
        job.addChild(downloadJob)
        aggregator.append(downloadJob)

    analysis = Job.wrapJobFn(analysisJob,
                             fileStoreID=downloadJob.rv(0),
                             disk=PromisedRequirement(lambda xs: sum(xs), [j.rv(1) for j in aggregator]))
    job.addFollowOn(analysis)

Limitations 🙈

Just like regular promises, the return value must be determined prior to scheduling any job that depends on the return value. In our example above, notice how the dependent jobs were follow ons to the parent while promising jobs are children of the parent. This ordering ensures that all promises are properly fulfilled. Don't skip this part, bestie 😭

---

FileID 🎀

The toil.fileStore.FileID class is a small wrapper around Python's builtin string class. It's used to represent a file's ID in the file store, and it has a size attribute that is the file's size in bytes. This object is returned by importFile and writeGlobalFile. Tiny but mighty 🌸

---

Managing files within a workflow 🦋

It is frequently the case that a workflow will want to create files, both persistent and temporary, during its run. The toil.fileStores.abstractFileStore.AbstractFileStore class is used by jobs to manage these files in a manner that guarantees cleanup and resumption on failure. Cleanup AND resumption, the dream team 🧹✨

okay bestie, so the toil.job.Job.run() method gets a file store instance as an argument 🧚 The example below shows how you can make temp files that stick around for the whole life of the job 💖 get placed on a specified local disk of the node 🌸 and get cleaned up regardless of failure when the job finishes 🫶

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

class LocalFileStoreJob(Job):
    def run(self, fileStore):
        # self.tempDir will always contain the name of a directory within the allocated disk space reserved for the job
        scratchDir = self.tempDir

        # Similarly create a temporary file.
        scratchFile = fileStore.getLocalTempFile()

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_managing")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    # Create an instance of FooJob which will have at least 2 gigabytes of storage space.
    j = LocalFileStoreJob(disk="2G")

    # Run the workflow
    with Toil(options) as toil:
        toil.start(j)

Job functions get access to the file store too 👑 The equivalent of the LocalFileStoreJob class is localFileStoreJobFn, shown here:

def localFileStoreJobFn(job):
    scratchDir = job.tempDir
    scratchFile = job.fileStore.getLocalTempFile()

pls note that fileStore is accessed as an attribute of the job argument, not passed in directly 💅

Beyond temp files that last for one job, the file store also lets you create files in a global store 🌍 These persist for the whole workflow and are globally accessible (hence the name) between jobs. Here's a whole example:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

def globalFileStoreJobFn(job):
    job.log(
        "The following example exercises all the methods provided "
        "by the toil.fileStores.abstractFileStore.AbstractFileStore class"
    )

    # Create a local temporary file.
    scratchFile = job.fileStore.getLocalTempFile()

    # Write something in the scratch file.
    with open(scratchFile, "w") as fH:
        fH.write("What a tangled web we weave")

    # Write a copy of the file into the file-store; fileID is the key that can be used to retrieve the file.
    # This write is asynchronous by default
    fileID = job.fileStore.writeGlobalFile(scratchFile)

    # Write another file using a stream; fileID2 is the
    # key for this second file.
    with job.fileStore.writeGlobalFileStream(cleanup=True) as (fH, fileID2):
        fH.write(b"Out brief candle")

    # Now read the first file; scratchFile2 is a local copy of the file that is read-only by default.
    scratchFile2 = job.fileStore.readGlobalFile(fileID)

    # Read the second file to a desired location: scratchFile3.
    scratchFile3 = os.path.join(job.tempDir, "foo.txt")
    job.fileStore.readGlobalFile(fileID2, userPath=scratchFile3)

    # Read the second file again using a stream.
    with job.fileStore.readGlobalFileStream(fileID2) as fH:
        print(fH.read())  # This prints "Out brief candle"

    # Delete the first file from the global file-store.
    job.fileStore.deleteGlobalFile(fileID)

    # It is unnecessary to delete the file keyed by fileID2 because we used the cleanup flag,
    # which removes the file after this job and all its successors have run (if the file still exists)

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_managing2")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        toil.start(Job.wrapJobFn(globalFileStoreJobFn))

This example runs through the global read, write and delete moves 🔥 using both local copies of the files and streams. It covers every method in the file store interface, so it's basically the whole menu 🍓

Now, the tea ☕ The file-store has no way to update an existing "global" file, so barring deletion, files are immutable 🙈 There's also no file system hierarchy for files in the global file store. Lowkey limiting, but these restrictions are what let Toil support different object stores pretty easily and use caching to cut down on network file transfer between jobs 🦋

🧳 Staging of Files into the Job Store

External files can be imported into or exported out of the job store before a workflow runs ✨ This works when the toil.common.Toil context manager is used on the leader. The context manager gives you toil.common.Toil.importFile() and toil.common.Toil.exportFile() for this. You pass URLs to these two methods to describe where the files are coming from and going to. Local files can be imported and exported as relative paths, and those paths should be relative to the directory where the toil workflow is initially run from 📂

Using absolute paths and an appropriate schema where you can (like prefixing with file:// or s3:/) makes imports and exports less ambiguous, and it's recommended 💅

Want the list of currently supported URLs? Check toil.jobStores.abstractJobStore.AbstractJobStore.importFile(). And if you wanna import an external file into the job store as a shared file, pass the optional sharedFileName parameter to that method 🎀

If a workflow fails for any reason, an imported file acts like any other file in the job store 🧚 If the workflow was configured so it doesn't get cleaned up on a failed run, the file persists in the job store, and it does not need to be staged again when the workflow is resumed.

Here's an example:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

class HelloWorld(Job):
    def __init__(self, id):
        Job.__init__(self)
        self.inputFileID = id

    def run(self, fileStore):
        with fileStore.readGlobalFileStream(self.inputFileID, encoding="utf-8") as fi:
            with fileStore.writeGlobalFileStream(encoding="utf-8") as (
                fo,
                outputFileID,
            ):
                fo.write(fi.read() + "World!")
        return outputFileID

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_staging")
    tmp: str = mkdtemp("tutorial_staging_tmp")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        if not toil.options.restart:
            # Prepare an input file
            path = os.path.join(tmp, "in.txt")
            with open(path, "w") as f:
                f.write("Hello,\n")
            # In a real workflow, you would obtain an input file path from the
            # user.

            # Stage it into the Toil job store.
            #
            # Note: this may create a symlink depending on the value of the
            # --linkImports command line option, in which case the original
            # input file needs to still exist if the workflow is restarted.
            inputFileID = toil.importFile(f"file://{path}")

            # Run the workflow
            outputFileID = toil.start(HelloWorld(inputFileID))
        else:
            outputFileID = toil.restart()

        toil.exportFile(
            outputFileID,
            "file://" + os.path.join(tmp, "out.txt"),
        )

heads up 🙈 the code above notes that --linkImports may create a symlink, so if the workflow gets restarted, the original input file still has to exist.

🐳 Using Docker Containers in Toil

Docker containers are super common with Toil 💜 Put them together and your pipelines become fully portable between any platform that has both Toil and Docker installed 🌈 Docker means the user doesn't have to do any other tool installation or environment setup. iconic, honestly ✨

To use Docker containers with Toil, Docker has to be installed on all workers of the cluster 🫶 Installation instructions are on the Docker website at https://docs.docker.com/engine/getstarted/step_one/.

If you're using Toil-based autoscaling, Docker gets set up automatically on the cluster's worker nodes, so no extra installation steps are needed 🤩 For more on autoscaling, check out the Running a Workflow with Autoscaling documentation.

To use Docker containers in a Toil workflow, the container can be built locally or downloaded in real time from an online Docker repository like Quay (quay.io). If the container isn't in a repository, its layers have to be accessible on each node of the cluster 📦

When you invoke Docker containers from inside a Toil workflow, it's strongly recommended that you use dockerCall(), a Toil job function provided in toil.lib.docker 👑 dockerCall leverages Docker's own Python API and provides container cleanup on job failure. Without that feature, failed jobs can result in resource leaks 💀 Docker's API can be found at docker-py (https://docker-py.readthedocs.io/en/stable/).

To use dockerCall, your Docker installation has to be set up to run without sudo. Instructions for that are at https://docs.docker.com/engine/installation/linux/ubuntulinux/#/create-a-docker-group.

Here's a basic dockerCall example:

dockerCall(job=job,
            tool='quay.io/ucsc_cgl/bwa',
            workDir=job.tempDir,
            parameters=['index', '/data/reference.fa'])

Notice the assumption that reference.fa lives in /data 🧚 That's Toil's standard convention for a mount location, which cuts down on boilerplate when you call dockerCall. If you want your own mount locations, you can supply a volumes kwarg to dockerCall, like volumes={working_dir: {'bind': '/data', 'mode': 'rw'}}, where working_dir is an absolute path on your filesystem 🎀

dockerCall can also be added to workflows like any other job function 💖

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.docker import apiDockerCall
from toil.lib.io import mkdtemp

align = Job.wrapJobFn(
    apiDockerCall, image="ubuntu", working_dir=os.getcwd(), parameters=["ls", "-lha"]
)

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_docker")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        toil.start(align)

okay so cgl-docker-lib (https://github.com/BD2KGenomics/cgl-docker-lib/blob/master/README.md) is a whole stash of dockerCall-compatible Dockerized tools that bioinformatics people use constantly 🧬✨ literally go look, it's so useful.

The docs also walk you through making your own Docker containers that work with Toil and dockerCall 🐳 bestie, the rule is that your container needs an ENTRYPOINT set to a wrapper script, like the cgl-docker-lib containerization standards say. You set that by passing the optional keyword argument 'entrypoint'. For example: entrypoint=["/bin/bash","-c"] 💅

dockerCall currently supports 75 keyword arguments from the python Docker API (https://docker-py.readthedocs.io/en/stable/containers.html), under the 'run' command. Lowkey a lot of options, so you're covered 🔥

🧚 Services 🧚

Sometimes you need a service, like a database or a server, running at the same time as your workflow 🌸 That's where toil.job.Job.Service comes in. It gives you a simple way to spawn a service inside a Toil workflow, with precise control over when it starts and ends, plus start and end methods for setup and cleanup. Here's a simple, conceptual example so you can see how services vibe:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

class DemoService(Job.Service):
    def start(self, fileStore):
        # Start up a database/service here
        # Return a value that enables another process to connect to the database
        return "loginCredentials"

    def check(self):
        # A function that if it returns False causes the service to quit
        # If it raises an exception the service is killed and an error is reported
        return True

    def stop(self, fileStore):
        # Cleanup the database here
        pass

j = Job()
s = DemoService()
loginCredentialsPromise = j.addService(s)

def dbFn(loginCredentials):
    # Use the login credentials returned from the service's start method to connect to the service
    pass

j.addChildFn(dbFn, loginCredentialsPromise)

if __name__ == "__main__":
    jobstore: str = mkdtemp("tutorial_services")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        toil.start(j)

In this example, DemoService starts a database in its start method and returns an object that tells a client job how to access it 🦋 The stop method cleans the database up, and the check method gets polled regularly to make sure the service is still alive.

DemoService gets added as a service of the root job j, with resource requirements specified. The return value of toil.job.Job.addService() is a promise for whatever the service's start method returns. Once that promise is fulfilled, it tells you how to connect to the database. The promise gets passed into a child job of j, which uses it to make the connection 💖 A job's services start before any of its successors run, and they stop after all of that job's successors finish successfully.

You can make multiple services per job, and they all run in parallel 🎀 Services can also define sub-services with toil.job.Job.Service.addChild(), so you can build complex networks of services inside a workflow, like Apache Spark clusters 🤯

🔁 Checkpoints 🔁

Okay, services make resuming a workflow after a failure kinda messy 😭 They create complicated dependencies between jobs. Say a service provides a database that multiple jobs update. If that database fails and loses its state, it's not clear that just restarting the service lets the workflow pick back up, because the jobs that made that state might already be finished.

To fix this, Toil supports checkpoint jobs ✨ You specify them with the boolean keyword argument checkpoint on a job or wrapped function, like this:

j = Job(checkpoint=True)

A checkpoint job gets rerun if one or more of its successors fails its retry attempts, and it keeps going until it has used up its own retry attempts too. When you restart a checkpoint job, all of its existing successors get deleted first, and then the job reruns to define new successors. So if you checkpoint a job that defines a service, then when the service fails, the database and the jobs that access it can be redefined and rerun 🧠

Heads up though, to keep the implementation simple, a job can only be a checkpoint if it has no successors when it's first defined. In other words, it can only define successors inside its run method 🙈

🫧 Encapsulation 🫧

Let's say A is a root job that might have children and follow-ons. Without an encapsulated job, the simplest way to make a job B run after A and all of its successors is to create a parent of A, called Ap, and then make B a follow-on of Ap. For example:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

if __name__ == "__main__":
    # A is a job with children and follow-ons, for example:
    A = Job()
    A.addChild(Job())
    A.addFollowOn(Job())

    # B is a job which needs to run after A and its successors
    B = Job()

    # The way to do this without encapsulation is to make a parent of A, Ap, and make B a follow-on of Ap.
    Ap = Job()
    Ap.addChild(A)
    Ap.addFollowOn(B)

    jobstore: str = mkdtemp("tutorial_encapsulations")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        print(toil.start(Ap))

An encapsulated job E(A) of A saves you from making Ap at all 🙌 Instead, you can write:

import os

from toil.common import Toil
from toil.job import Job
from toil.lib.io import mkdtemp

if __name__ == "__main__":
    # A
    A = Job()
    A.addChild(Job())
    A.addFollowOn(Job())

    # Encapsulate A
    A = A.encapsulate()

    # B is a job which needs to run after A and its successors
    B = Job()

    # With encapsulation A and its successor subgraph appear to be a single job, hence:
    A.addChild(B)

    jobstore: str = mkdtemp("tutorial_encapsulations2")
    os.rmdir(jobstore)
    options = Job.Runner.getDefaultOptions(jobstore)
    options.logLevel = "INFO"
    options.clean = "always"

    with Toil(options) as toil:
        print(toil.start(A))

Note that calling toil.job.Job.encapsulate() creates the toil.job.Job.EncapsulatedJob 💜

📦 Depending on Toil 📦

If you're packing your workflow as a pip-installable distribution on PyPI, you might be tempted to declare Toil as a dependency in your setup.py, using the install_requires keyword argument to setup(). Sadly, this doesn't work, and there are two reasons 😤 First, Toil uses Setuptools' extra mechanism to manage its own optional dependencies. If you explicitly declare a dependency on Toil, you'd have to hard-code a particular combination of extras (or no extras at all), which takes away the user's choice of which Toil extras to install. Second, and more importantly, declaring a dependency on Toil would only get Toil installed on the leader node of a cluster, not the worker nodes. Auto-deployment doesn't save you here, because Toil can't auto-deploy itself. It's the classic "Which came first, chicken or egg?" problem 🐣

So the move is: don't explicitly depend on Toil. Document the dependency instead (something like "This workflow needs Toil version X.Y.Z to be installed"), and optionally add a version check to your setup.py. For an example, check out the check_version() function in the toil-lib project's setup.py (https://github.com/BD2KGenomics/toil-lib/blob/master/setup.py). Or you can just depend on toil-lib, and you'll get that check for free 🫶

If your workflow depends on one of Toil's own dependencies, consider not making that dependency explicit either. If you do, you risk a version conflict between your project and Toil. The pip utility may silently ignore that conflict, which could break either Toil or your workflow. The safest bet is to assume Toil installs that dependency for you. The only downside is that you're locked into the exact version Toil declares. But that's just life with Python, which, unlike Java, has no way of letting dependencies belonging to different software components live in the same process, and whose favored distribution utility can't properly resolve overlapping dependencies or detect conflicts (https://github.com/pypa/pip/issues/988) 😭

🐳 Best Practices for Dockerizing Toil Workflows 🐳

Computational Genomics Lab's Dockstore-based production system (https://cgl.genomics.ucsc.edu/ and https://dockstore.org/docs) gives workflow authors a way to run Dockerized versions of their pipelines in an automated, scalable way ✨ To be compatible with this system, your workflow should meet the following requirements. Besides the Docker container, you also need a common workflow language descriptor file (https://dockstore.org/docs/getting-started-with-cwl). For inputs:

• Only command line arguments should be used to configure the workflow. If your workflow relies on a configuration file, like Toil-RNAseq (https://github.com/BD2KGenomics/toil-rnaseq) or ProTECT (https://github.com/BD2KGenomics/protect), you can use a wrapper script inside the Docker container to parse the CLI and generate the necessary configuration file 💅
• All inputs should be explicitly enumerated rather than implicit. For example, don't rely on one FASTQ read's path to figure out where its pair lives. This matters because all inputs get mapped to their own isolated directories when Docker is called via Dockstore 🥹
• All inputs must be documented in the CWL descriptor file, bestie 📝 Real examples live in Toil-RNAseq (https://github.com/BD2KGenomics/toil-rnaseq) and ProTECT (https://github.com/BD2KGenomics/protect), so go peek at those. 👀

For outputs: 📦

• All outputs should be written to a local path rather than S3. 🏠
• Take care to package outputs in a local and user-friendly way. For example, don't tar up all the output if there are specific files people will care to see individually. 🎀
• All output file names should be deterministic and predictable. For example, don't prepend the name of an output file with PASS/FAIL depending on the outcome of the pipeline. 🙅‍♀️ That's a vibe-killer for anyone looking for their files later. 💀
• All outputs must be documented in the CWL descriptor file. Again, the examples are in Toil-RNAseq (https://github.com/BD2KGenomics/toil-rnaseq) and ProTECT (https://github.com/BD2KGenomics/protect). 💖

---

✨ TOIL CLASS API ✨

The Toil class is the thing that configures and starts a Toil run. 🚀 It's giving main character energy. 👑

class toil.common.Toil(options, workflow_name=None, trs_spec=None)

A context manager that represents a Toil workflow. 🧚 It's where the batch system, the job store, and their configuration all get set up.

Parameters:

• options (Namespace)
• workflow_name (str | None)
• trs_spec (str | None)

🌸 __init__

__init__(options, workflow_name=None, trs_spec=None)

This initializes a Toil object from the options you give it. 💅 It's super light-weight, so don't expect much to happen here. The heavy lifting happens when the context is entered. ✨

Parameters:

• options (Namespace) — the command line options the user specified. 🎀
• workflow_name (str | None) — a human-readable name for the workflow being run (probably a filename, URL, or TRS specifier). Toil uses it for history storage. 📚
• trs_spec (str | None) — a TRS id:version string for the workflow being run, if there is one. Toil uses it for history storage and for publishing workflow execution metrics to Dockstore. 📈

Return type: None

🦋 start(rootJob)

start(rootJob)

This invokes a Toil workflow, using the given job as the root for an initial run. 🔥

This method must be called in the body of a with Toil(...) as toil: statement. It should not be called more than once for a workflow that has not finished. 🙈

Parameters:

• rootJob (Job) — the root job of the workflow. 🌳

Return type: Any

Returns: the root job's return value. 🎁

🍓 restart()

restart()

This restarts a workflow that got interrupted. Bring it back, bestie. 🔄

Return type: Any

Returns: the root job's return value. 🎁

💜 getJobStore(locator)

classmethod getJobStore(locator)

This creates an instance of the concrete job store implementation that matches the given locator. 🗄️ It's lowkey iconic.

Parameters:

• locator (str) — the location of the job store the instance will represent. 📍
• locator — listed again with no further description. 🤷‍♀️

Return type: AbstractJobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Returns: an instance of a concrete subclass of AbstractJobStore. 💖

🌈 parseLocator(locator)

static parseLocator(locator)

This parses a job store locator into a type string, plus the data that implementation needs to connect to it. 🧩 It does not validate the set of possible job store types, so it's not checking that part for you. 🙅‍♀️

Raises RuntimeError if the locator is not in the appropriate syntax. 💀

Parameters:

• locator (str)

Return type: tuple[str, str]

🫶 canonical_locator(locator)

classmethod canonical_locator(locator)

This turns a job store locator into one that works from any directory, and that always includes the explicit type of job store. 🗺️ Very reliable, no cap.

Parameters:

• locator (str)

Return type: str

⭐ createBatchSystem(config)

static createBatchSystem(config)

This creates an instance of the batch system specified in the given config. 🖥️

Parameters:

• config (Config) — the current configuration. ⚙️

Return type: AbstractBatchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem)

Returns: an instance of a concrete subclass of AbstractBatchSystem. 🔥

🌷 import_file(src_uri, shared_file_name=None, symlink=True, check_existence=True)

import_file(src_uri, shared_file_name=None, symlink=True, check_existence=True)

This imports the file at the given URL into the job store. 📥 By default, it returns None if the file doesn't exist. 🤔

Heads up, bestie: check_existence also defaults to True, which means it raises FileNotFoundError when the file is missing. Read the flags before you vibe. 💀

For the full breakdown, see toil.jobStores.abstractJobStore.AbstractJobStore.importFile(). 📖

Parameters:

• check_existence (bool) — if true, raise FileNotFoundError when the file doesn't exist. If false, return None when the file doesn't exist. 🫣
• src_uri (str)
• shared_file_name (str | None)
• symlink (bool)

The same parameters get listed again with the same types, in a different order: src_uri (str), shared_file_name (str | None), symlink (bool), and check_existence (bool), with no further descriptions. 🔁

Return type: FileID | None

🌺 export_file(file_id, dst_uri)

export_file(file_id, dst_uri)

This exports a file to the destination pointed at by the destination URL. 📤

For the full description, see toil.jobStores.abstractJobStore.AbstractJobStore.exportFile(). 📖

Parameters:

• file_id (FileID)
• dst_uri (str)

Return type: None

🧁 normalize_uri(uri, check_existence=False, dir_path=None)

static normalize_uri(uri, check_existence=False, dir_path=None)

Given a URI with no scheme, this turns it into a properly quoted file: URI. 🎀 Very tidy, very satisfying. 🫧

Parameters:

• check_existence (bool) — if set, raise FileNotFoundError when a URI points to a local file that doesn't exist. 🚨
• dir_path (str | None) — if specified, relative paths get interpreted relative to the given directory path instead of the current one. 🗂️
• uri (str)

Return type: str

Okay so this is the part where the Toil class gets really specific, bestie 💅 We're doing the directory helpers first, and they are giving organized 📁✨

getToilWorkDir is a static method that returns a path to a writable directory where the per-workflow directories live 🌸

static getToilWorkDir(configWorkDir=None)

This directory has to exist on every machine, even if the Toil worker hasn't run yet 🫶 If your workers and your leader use different temp directories, you might need to set TOIL_WORKDIR. Lowkey important, so don't skip it 😭

Parameters

• configWorkDir (str | None) — the value passed to the program with the --workDir flag

Return type

str

Returns

The path to the Toil work directory, which stays the same across all machines 💜

---

get_toil_coordination_dir is a classmethod that returns a path to a writable directory. It'll be in memory when that's convenient, and it's the one you should use for file locking and coordination 🔒

classmethod get_toil_coordination_dir(config_work_dir,
config_coordination_dir)

Parameters

• config_work_dir (str | None) — the value passed to the program with the --workDir flag
• config_coordination_dir (str | None) — the value passed to the program with the --coordinationDir flag
• workflow_id — used when a tmpdir_prefix exists, to build full directory paths that are unique per workflow ✨

Return type

str

Returns

The path to the Toil coordination directory. It should sit on a POSIX filesystem that lets you delete directories containing open files. That's a must-have vibe 🦋

---

get_workflow_path_component is a static method that gives you a safe filesystem path component for a workflow 🧚

static get_workflow_path_component(workflow_id)

It's consistent for every process on a given machine, and different for every process on different machines. Main character energy 👑

Parameters

• workflow_id (str) — the ID of the current Toil workflow

Return type

str

---

getLocalWorkflowDir is a classmethod that returns the directory where the worker directories and the cache will live for this workflow on this machine 🍓

classmethod getLocalWorkflowDir(workflowID, configWorkDir=None)

Parameters

• configWorkDir (str | None) — the value passed to the program with the --workDir flag
• workflowID (str)

Return type

str

Returns

The path to the local workflow directory on this machine 🌈

---

get_local_workflow_coordination_dir is a classmethod that returns the directory where coordination files should live for this workflow on this machine. That includes Toil's internal databases and the machine's lock files 🔐

classmethod get_local_workflow_coordination_dir(workflow_id,
config_work_dir, config_coordination_dir)

If an in-memory filesystem is available, it gets used. If not, it falls back to the local workflow directory, which might be on a shared network filesystem 🫠

Parameters

• workflow_id (str) — the unique ID of the current workflow
• config_work_dir (str | None) — the value used for the work directory in the current Toil Config
• config_coordination_dir (str | None) — the value used for the coordination directory in the current Toil Config

Return type

str

Returns

The path to the local workflow coordination directory on this machine ✨

---

✨ JOB STORE API ✨

Okay, the job store is the abstraction layer that hides all the messy details of file storage 🗂️ That could be a standard filesystem, S3, or something else. The AbstractJobStore API is what gets implemented to support a given file store, like S3. If you want to add support for a new file store, this is the API you implement 💖

✨ AbstractJobStore ✨

class toil.jobStores.abstractJobStore.AbstractJobStore(locator)

This class represents the physical storage for the jobs and files in a Toil workflow 🌸

Job stores are responsible for storing toil.job.JobDescription objects, which relate jobs to each other, and for storing files. The actual toil.job.Job objects are stored in files and get referenced by their JobDescriptions. Every non-file CRUD method the JobStore offers deals in JobDescriptions, not full executable Jobs 🦋

If you want to actually get your hands on a toil.job.Job, call toil.job.Job.loadJob() with a JobStore and the relevant JobDescription 🫶

Parameters

• locator (str)

✨ __init__(locator) ✨

__init__(locator)

This creates an instance of the job store. The instance won't be fully functional until either initialize() or resume() is invoked 🙈 Heads up: destroy() can be called on the object with or without calling either of those two methods first ⚠️

It takes and stores the locator string for the job store, which is accessible through self.locator.

Parameters

• locator (str)

Return type

None

✨ initialize(config) ✨

initialize(config)

This initializes the job store. It creates the physical storage for the job store, allocates a workflow ID, and saves the given Toil configuration to the store 💾

Parameters

• config (Config) — the Toil configuration to initialize this job store with. The given configuration gets updated with the newly allocated workflow ID.

Raises

• JobStoreExistsException — if the physical storage for this job store already exists 💀

Return type

None

✨ write_config() ✨

write_config()

This saves the value of the AbstractJobStore.config attribute to the job store, so other instances of this class can retrieve it later 🎀

Return type

None

✨ resume() ✨

resume()

This connects the instance to the physical storage it represents and loads the Toil configuration into the AbstractJobStore.config attribute 🔥

Raises

• NoSuchJobStoreException — if the physical storage for this job store doesn't exist 😭

Return type

None

✨ config property ✨

property config: Config

This returns the Toil configuration associated with this job store 💅

Raises

• AttributeError — if the config hasn't been assigned yet (for example, during resume())

Return type

toil.common.Config

✨ locator property ✨

property locator: str

This gets the locator that defines the job store. You use it to connect to the job store 🔗

Return type

str

✨ setRootJob(rootJobStoreID) ✨

setRootJob(rootJobStoreID)

This sets the root job of the workflow backed by this job store 👑

Okay bestie, the cut-off method's one parameter is rootJobStoreID 🥹 It's a FileID, and the return type is None 💅

Parameters

• rootJobStoreID (FileID) 🎀

Return type: None ✨

set_root_job(job_id) 👑

Okay so this one sets the root job of the workflow backed by this job store 🌸 Main character energy fr.

Parameters

• job_id (FileID) — the ID of the job you want to make the root 💖

Return type: None 🫶

load_root_job() 🦋

This one loads the JobDescription for the root job in the current job store, so you can grab it and go 🧚

💀 It raises toil.job.JobException if no root job is set, OR if the root job doesn't exist in this job store. Not it being ghosted 😭

Return type: JobDescription

Returns: the root job 👑

create_root_job(job_description) 🌈

Slay, this creates the given JobDescription and sets it as the root job in this job store ✨

Parameters

• job_description (JobDescription) — the JobDescription to save and make the root job 🍓

Return type: JobDescription 💜

get_root_job_return_value() 🔮

This one parses the return value from the root job ✨

⚠️ It raises an exception if the root job hasn't fulfilled its promise yet. Bestie, be patient 💀

Return type: Any 🌸

import_file(src_uri, shared_file_name=None, hardlink=False, symlink=True) 📥

Okay so this one imports the file at the given URL into the job store 🎀 The ID of the newly imported file gets returned. If you give it a shared file name, the file gets imported as that name and None is returned instead 🫶 Also, if an executable file from your local filesystem gets uploaded, its executability is preserved when it's downloaded again 💅

Currently supported schemes are:

• s3 for objects in Amazon S3, e.g. s3://bucket/key 🍓
• file for local files, e.g. file:///local/file/path 🦋
• http, e.g. http://someurl.com/path 🌈
• gs, e.g. gs://bucket/file ⭐

Raises FileNotFoundError if the file does not exist. Not cute 😭

Parameters

• src_uri (str) — a URL pointing to a file or object in the storage mechanism of a supported URL scheme, like a blob in an AWS s3 bucket. It must be a file, not a directory or prefix 🙈
• shared_file_name (str | None) — optional name to assign to the imported file within the job store 🧚
• src_uri
• shared_file_name
• hardlink (bool)
• symlink (bool)

Returns: the jobStoreFileID of the imported file, or None if shared_file_name was given 💖

Return type: toil.fileStores.FileID or None 💜

export_file(file_id, dst_uri) 📤

This one exports a file to the destination pointed at by the destination URL 🌸 The exported file will be executable if and only if it was originally uploaded from an executable file on the local filesystem. Iykyk 🔥

Refer to AbstractJobStore.import_file() for the currently supported URL schemes 🫶

Note that the helper method _exportFile is what reads from the source and writes to the destination. If you want to implement any optimizations that get around this, the _exportFile method should be overridden by subclasses of AbstractJobStore 👑

Parameters

• file_id (FileID) — the ID of the file in the job store that should be exported 🎀
• dst_uri (str) — a URL pointing to a file or object in the storage mechanism of a supported URL scheme, like a blob in an AWS s3 bucket. It may also be a local path 🦋
• file_id
• dst_uri

Return type: None ✨

abstractmethod destroy() 💥

The inverse of initialize(), this one deletes the physical storage represented by this instance 😭 It's not atomic, but it is idempotent, which is there to help with possible issues from eventual consistency in the underlying storage. So if it fails (raises an exception), it may (and should be) invoked again 🫶

⚠️ If the underlying storage is eventually consistent, even a successful call isn't an ironclad guarantee that the physical storage vanished completely and immediately. A successful call only guarantees that the deletion will eventually happen. So it's recommended to not immediately reuse the same job store location for a new Toil workflow 💀

Return type: None 🌈

get_env() 🧚

This one returns a dictionary of environment variables that this job store needs set in order to function properly on a worker 💜

Return type: dict[str, str] 🍓

clean(jobCache=None) 🧹

Okay so this is the cleanup function for a job store after a restart ✨ It fixes jobs that might have been partially updated, resets the try counts, and removes jobs that are not successors of the current root job 💅

Parameters

• jobCache (dict[str | TemporaryID, JobDescription] | None) — if a value is given, it must be a dict from job ID keys to JobDescription object values. Jobs get loaded from the cache (which can be downloaded from the job store in a batch) instead of piecemeal when recursed into 🌸

Return type: JobDescription 🦋

abstractmethod assign_job_id(job_description) 🪪

This one gets a new jobStoreID to be used by the described job, and assigns it to the JobDescription 🎀 Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated. Lowkey wild 👀

Parameters

• job_description (JobDescription) — the JobDescription to give an ID to 🍓

Return type: None 💖

abstractmethod create_job(job_description) 📝

This one writes the given JobDescription to the job store ✨ The job must already have an ID assigned.

⚠️ You must call jobDescription.pre_update_hook() 🙈

Returns: the JobDescription that was passed in 🫶

Return type: toil.job.JobDescription 💜

Parameters

• job_description (JobDescription) 🌈

abstractmethod job_exists(job_id) 🔍

This one checks whether a description of the job with the specified jobStoreID exists in the job store 👑

Return type: bool ✨

Parameters

• job_id (str) 🦋

abstractmethod get_public_url(file_name) 🌍

This one returns a publicly accessible URL to the given file in the job store 🫶

⚠️ The returned URL may expire as early as 1h after it's been returned. It also throws an exception if the file does not exist 😭

Parameters

• file_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀 — the jobStoreFileID of the file you want a URL for
• file_name ✨

Raises NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) 💀 — if the specified file doesn't exist in this job store

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🦋 get_shared_public_url(shared_file_name) (abstractmethod)

okay so this one is the sibling of getPublicUrl() (toil.jobStores.abstractJobStore.AbstractJobStore.getPublicUrl), but it's specifically for shared files written with writeSharedFileStream() (toil.jobStores.abstractJobStore.AbstractJobStore.writeSharedFileStream) 🌸

You get back a publicly accessible URL for the file in the job store. The URL always starts with 'http:', 'https:' or 'file:' 🔗. Heads up, bestie: that link can expire as early as 1h after it's returned 😭. If the file doesn't exist, it throws an exception 💅.

Parameters

• shared_file_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓 — the name of the shared file you want a publicly accessible URL for
• shared_file_name 🧚

Raises NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) — raised if the specified file doesn't exist in the store 🙈

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🌈 load_job(job_id) (abstractmethod)

This one loads the description of the job referenced by the given ID, assigns it the job store's config, and then returns it ✨

Plot twist: it may declare the job to have failed (see toil.job.JobDescription.setupJobAfterFailure()) if there's evidence of a failed update attempt 👀.

Parameters

• job_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🌸 — the ID of the job to load

Raises NoSuchJobException (toil.jobStores.abstractJobStore.NoSuchJobException) — if there is no job with the given ID 💀

Return type: JobDescription (toil.job.JobDescription)

🧚 batch()

If the batch system supports it, calls to create() made while this context manager is active get performed in a batch once the context manager is released 🫶

Return type: Iterator (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator)[None (/usr/share/doc/python3/html/library/constants.html#None)]

💜 update_job(job_description) (abstractmethod)

This persists changes to the state of the given JobDescription in this store, and it does it atomically 💖. It must call jobDescription.pre_update_hook() 🙌.

Parameters

• job (toil.job.JobDescription) — the job to write to this job store 🎀
• job_description (JobDescription) ✨

Return type: None (/usr/share/doc/python3/html/library/constants.html#None)

🗑️ delete_job(job_id) (abstractmethod)

This removes the JobDescription from the store, atomically. After that, you may not call load(), write(), update(), etc. with the same jobStoreID, or with any JobDescription bearing it 🚫.

Good news though: this operation is idempotent 🫧. That means deleting a job twice, or deleting a job that doesn't exist, will succeed silently 🤫.

Parameters

• job_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the job to delete from this job store 🍓
• job_id 🌷

Return type: None (/usr/share/doc/python3/html/library/constants.html#None)

🌸 jobs()

This is a best-effort attempt to return an iterator over JobDescriptions for all jobs in the store. ngl, it might not return every job 😬, and it may also contain orphaned jobs that already finished successfully and shouldn't be rerun. If you need to guarantee you get any and all jobs that can be run, construct a more expensive ToilState object instead 💅.

Returns

An iterator on the jobs in the store. It may or may not contain all jobs, and it may contain invalid jobs 🙈.

Return type: Iterator[toil.job.jobDescription]

🔥 write_file(local_path, job_id=None, cleanup=False) (abstractmethod)

This takes a file (as a path) and places it in this job store 📦. It returns an ID you can use to retrieve the file later. The write is atomic, so the file won't show up in the jobStore until the write has successfully completed 🦋.

Parameters

• local_path (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🌈 — the path to the local file that will be uploaded to the job store. The last path component (the basename of the file) stays associated with the file in the file store, if supported, so the file can be searched for by name or name glob 🔍.
• job_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str, or None, /usr/share/doc/python3/html/library/constants.html#None) 🧚 — the id of a job, or None. If specified, the file may be associated with that job in a job-store-specific way. This may influence the returned ID.
• cleanup (bool, /usr/share/doc/python3/html/library/functions.html#bool) 💜 — whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

Raises

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException) ⚠️ — if the file was modified concurrently during an invocation of this method
• NoSuchJobException (toil.jobStores.abstractJobStore.NoSuchJobException) 💀 — if the job specified via jobStoreID doesn't exist

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

FIXME: some implementations may not raise this 🛠️

Returns

An ID referencing the newly created file, which you can use to read the file in the future 🎀.

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Parameters

• local_path (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🌸
• job_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str, or str | None, /usr/share/doc/python3/html/library/constants.html#None) 🫶
• cleanup (bool, /usr/share/doc/python3/html/library/functions.html#bool) ✨

🎀 write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) (abstractmethod)

This is similar to writeFile, but instead it returns a context manager that yields a tuple of 1) a file handle you can write to and 2) the ID of the resulting file in the job store 🧁. You don't need to close the yielded file handle yourself, and you shouldn't 🙅‍♀️. Same deal as before: the write is atomic, so the file won't appear in the jobStore until it's fully written 🦋.

Parameters

• job_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the id of a job, or None. If specified, the file may be associated with that job in a job-store-specific way. This may influence the returned ID 🌷
• cleanup (bool, /usr/share/doc/python3/html/library/functions.html#bool) — whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing 💜
• basename (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — if the implementation supports it, use the given file basename so that when you search the job store with a query matching that basename, the file gets detected 🔍
• encoding (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None, which represents binary mode 🤓
• errors (str) — okay so this one is an optional string 🫶 that tells Toil how to handle encoding errors. It works exactly like it does in open(), and if you set an encoding but skip this, it defaults to 'strict' 💅

💥 Raises

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file got modified concurrently during an invocation of this method 😭
• NoSuchJobException (toil.jobStores.abstractJobStore.NoSuchJobException) — if the job you pass in via jobStoreID doesn't exist 💀

Return type

Iterator[tuple[IO[bytes], str]]

FIXME: some implementations may not raise this 🙈

Returns

a context manager yielding a file handle that you can write to, plus an ID that points to the newly created file so you can read it back later 🦋

Return type

Iterator[Tuple[IO[bytes], str]]

Parameters

• job_id (str | None)
• cleanup (bool)
• basename (str | None)
• encoding (str | None)
• errors (str | None)

---

✨ abstractmethod get_empty_file_store_id(job_id=None, cleanup=False, basename=None)

Okay this one is so iconic 💖 It creates an empty file in the job store and hands you back its ID. After that, fileExists(getEmptyFileStoreID(jobStoreID)) will return True, no cap 🌸

Parameters

• job_id (str | None) — the id of a job, or None. If you give one, the file may be linked to that job in a job-store-specific way, and that can change the ID you get back 🧚
• cleanup (bool) — whether Toil should try to delete the file when the job whose jobStoreID was given as jobStoreID gets deleted with jobStore.delete(job). If no jobStoreID was given, this does absolutely nothing 🫠
• basename (str | None) — if the implementation supports it, this uses the basename you gave, so when you search the job store with a query matching that basename, the file shows up 🔥

Returns

a jobStoreFileID that points to the newly created file, so you can reference it later 🎀

Return type

str

---

✨ abstractmethod read_file(file_id, local_path, symlink=False)

Bestie, this one is a whole vibe 🦋 It copies or hard links the file referenced by jobStoreFileID to your local file path. The version you get matches the last copy that was written or updated. If the job store file gets changed later via updateFile or updateFileStream, whether those writes show up at localFilePath is up to each implementation 🤷‍♀️

The copy happens atomically, so the file won't show up in your local file system until the copy is totally done 💅

The file at the given local path may not be modified after this method returns! 🚫

Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs. 🔥

Parameters

• file_id (str) — the ID of the file you want copied 💜
• local_path (str) — the local path that tells Toil where to put the contents of the file from the job store 🗺️
• symlink (bool) — whether the reader can handle a symlink. If you set this to true, the job store might make a symlink instead of a full copy, or a hard link ✨
• file_id
• local_path
• symlink

Return type

None

---

✨ abstractmethod read_file_stream(file_id, encoding=None, errors=None)

This is basically readFile's cute cousin 🌈 It returns a context manager that gives you a file handle you can read from. You don't need to close the handle yourself, and you actually shouldn't 🙅‍♀️

Parameters

• file_id (FileID | str) — the ID of the file you want a readable file handle for 🍓
• encoding (str | None) — the name of the encoding used to decode the file. Encodings work the same as they do for decode(). It defaults to None, which means binary mode 🤓
• errors (str | None) — an optional string that tells Toil how to handle encoding errors. Errors work the same as they do for open(), and it defaults to 'strict' when you set an encoding 💖
• file_id
• encoding
• errors

Returns

a context manager that gives you a file handle you can read from 📖

Return type

Iterator[Union[IO[bytes], IO[str]]]

---

✨ abstractmethod delete_file(file_id)

Deleting a file from the job store is idempotent, which means deleting it twice, or deleting something that doesn't exist, just succeeds silently 🫶 No drama, no errors, just vibes ✨

Parameters

• file_id (str) — the ID of the file to delete 🗑️
• file_id

Return type

None

---

✨ fileExists(jobStoreFileID)

Checks whether a file exists in this job store 👀

Parameters

jobStoreFileID (str)

Return type

bool

---

✨ abstractmethod file_exists(file_id)

Same energy as fileExists, but this version is the abstract one 🧚 It checks whether a file exists in this job store 🔍

Parameters

• file_id (str) — an ID that references the file you want to check 💜

Return type

bool

---

✨ getFileSize(jobStoreFileID)

Gets the size of the given file in bytes 📏 Lowkey the easiest one of the whole bunch 🌸

Parameters

jobStoreFileID (str)

Return type

int

💖 get_file_size(file_id)

abstractmethod get_file_size(file_id)

Okay so this one gets the size of a file in bytes 📏. If the file doesn't exist when you ask, you get 0 instead. Lowkey iconic.

ngl, job stores that encrypt files might give you overestimates of the size 🙈. The encrypted file may have been padded out to the nearest block, and an initialization vector may have been tacked on, etc.

Parameters 🎀

• file_id (str) — an ID referencing the file to be checked
• file_id

Return type ✨ int

🦋 updateFile(jobStoreFileID, localFilePath)

updateFile(jobStoreFileID, localFilePath)

This one replaces the existing version of a file in the job store. Bestie, it's giving "overwrite the old one" energy 💅.

Parameters 🎀

• jobStoreFileID (str)
• localFilePath (str)

Return type ✨ None

🌸 update_file(file_id, local_path)

abstractmethod update_file(file_id, local_path)

Same vibe as updateFile, just with the newer snake_case name 🐍. It replaces the existing version of a file in the job store, and it throws an exception if the file does not exist 💀. So make sure the file is actually there before you try.

Parameters 🎀

• file_id (str) — the ID of the file in the job store to be updated
• local_path (str) — the local path to a file that will overwrite the current version in the job store

Raises 😭

• ConcurrentFileModificationException — if the file was modified concurrently during an invocation of this method
• NoSuchFileException — if the specified file does not exist

Return type ✨ None

🧚 update_file_stream(file_id, encoding=None, errors=None)

abstractmethod update_file_stream(file_id, encoding=None, errors=None)

This is giving write_shared_file_stream realness, but for replacing a file that already exists 🫶. It replaces the existing version of a file in the job store, and it returns a context manager that yields a file handle you can write to.

Important bestie moment 🚨: the yielded file handle does not need to be closed explicitly, and it should not be closed explicitly. Don't do it!!

Parameters 🎀

• file_id (str) — the ID of the file in the job store to be updated
• encoding (str | None) — the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None, which means binary mode.
• errors (str | None) — an optional string that tells Python how to handle encoding errors. Errors work the same as for open(). Defaults to 'strict' when an encoding is specified.
• file_id
• encoding
• errors

Raises 😭

• ConcurrentFileModificationException — if the file was modified concurrently during an invocation of this method
• NoSuchFileException — if the specified file does not exist

Return type ✨ Iterator[IO[Any]]

🔥 write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)

abstractmethod write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)

This one returns a context manager that yields a writable file handle to the global file referenced by the name you give it 📝. The file gets created in an atomic manner, which means it's all or nothing, no half-written messes 👑.

Parameters 🎀

• shared_file_name (str) — a file name matching AbstractJobStore.fileNameRegex, and it has to be unique within this job store
• encrypted (bool | None) — True if the file must be encrypted, None if it may or may not be encrypted, or False if it must be stored in the clear
• encoding (str | None) — the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None, which means binary mode.
• errors (str | None) — an optional string that tells Python how to handle encoding errors. Errors work the same as for open(). Defaults to 'strict' when an encoding is specified.
• shared_file_name
• encrypted
• encoding
• errors

Raises 😭

• ConcurrentFileModificationException — if the file was modified concurrently during an invocation of this method

Returns 💜 a context manager yielding a writable file handle

Return type ✨ Iterator[IO[bytes]]

🌈 read_shared_file_stream(shared_file_name, encoding=None, errors=None)

abstractmethod read_shared_file_stream(shared_file_name, encoding=None, errors=None)

The reading buddy of the one above 📖. It returns a context manager that yields a readable file handle to the global file referenced by the given name.

Parameters 🎀

• shared_file_name (str) — a file name matching AbstractJobStore.fileNameRegex, and it has to be unique within this job store
• encoding (str | None) — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which means binary mode.
• errors (str | None) — an optional string that tells Python how to handle encoding errors. Errors work the same as for open(). Defaults to 'strict' when an encoding is specified.
• shared_file_name
• encoding
• errors

Return type ✨ Iterator[IO[str]] | Iterator[IO[bytes]]

Returns 💜 a context manager yielding a readable file handle

🍓 write_logs(msg)

abstractmethod write_logs(msg)

Slay, this one stores a message as a log in the jobstore 📋. It's the place to drop your notes so future-you can find them 🧚.

Parameters 🎀

• msg (str) — the string to be written
• msg

Raises 😭

• ConcurrentFileModificationException — if the file was modified concurrently during an invocation of this method

Return type ✨ None

abstractmethod read_logs(callback, read_all=False)

okay so this one is the log reader 📖💅 It reads the logs that write_logs() has been stacking up. For each log, it calls your callback function with the message as an argument. It won't hand you the logs back directly, bestie 🙅‍♀️ you have to pass in a callback that processes the messages ✨.

Only unread logs get read, unless you set the read_all parameter 👀

Parameters

• callback (Callable[..., Any]) — a function to be applied to each of the stats file handles found 💖
• read_all (bool) — a boolean that decides whether the already processed stats files get read too, on top of the unread ones 🌸
• callback
• read_all

Raises toil.jobStores.abstractJobStore.ConcurrentFileModificationException — if the file got modified concurrently during an invocation of this method 😭

Returns the number of stats files processed 🎀

Return type int

---

write_leader_pid() 🧚

Writes the pid of this process to a file in the job store.

Overwriting the current contents of pid.log is a feature, not a bug 💜 Other methods rely on always having the most current pid available. So far there's no reason to store any old pids.

Return type None

read_leader_pid() 🦋

Reads the pid of the leader process from a file in the job store.

Raises toil.jobStores.abstractJobStore.NoSuchFileException — if the PID file doesn't exist 💀

Return type int

write_leader_node_id() 🌈

Writes the leader node id to the job store. This should only be called by the leader 👑

Return type None

read_leader_node_id() 🫶

Reads the leader node id stored in the job store.

Raises toil.jobStores.abstractJobStore.NoSuchFileException — if the node ID file doesn't exist 😭

Return type str

write_kill_flag(kill=False) 🚨

Writes a file inside the job store that acts as a kill flag ✨

The initialized file contains the characters "NO". This should only be changed when the user runs the "toil kill" command. Changing the file to "YES" triggers a kill of the leader process, and the workers are expected to be cleaned up by the leader 🔥

Parameters

• kill (bool)

Return type None

read_kill_flag() 🙈

Reads the kill flag from the job store. Returns True if the leader has been killed, and False otherwise.

Return type bool

default_caching() 🍓

The job store's preference on caching: whether it likes caching or doesn't care either way. Some job stores benefit from caching, but on some local configurations it can be flaky 🥲

see https://github.com/DataBiosphere/toil/issues/4218

Return type bool

---

👑 TOIL JOB API 👑

Functions to wrap jobs and return values (promises). Main character energy 💅✨

🎀 FunctionWrappingJob 🎀

The subclass of Job for wrapping user functions.

class toil.job.FunctionWrappingJob(userFunction, *args, **kwargs)

A job used to wrap a function. In its run method, the wrapped function gets called 🦋

Parameters

• userFunction (Callable[..., Any])
• args (Any)
• kwargs (Any)

__init__(userFunction, *args, **kwargs) 🌸

Parameters

• userFunction (callable) — the function to wrap. It will be called with *args and **kwargs as arguments 💖
• args (Any)
• kwargs (Any)

Return type None

The keywords memory, cores, disk, accelerators, preemptible and checkpoint are reserved keyword arguments. If you specify them, they get used to determine the resources the job requires, same as toil.job.Job.__init__() 🧠 If they're keyword arguments to the function, they'll be pulled out of the function definition, but you can override them (as you would expect, obviously) 💅

Parameters

• userFunction (Callable[..., Any])
• args (Any)
• kwargs (Any)

Return type None

run(fileStore) 🧚‍♀️

Override this function to perform work and dynamically create successor jobs ✨

Parameters

• fileStore (AbstractFileStore) — used to create local and globally sharable temporary files, and to send log messages to the leader process 🫶 (the full name is toil.fileStores.abstractFileStore.AbstractFileStore)

Return type Any

Returns The return value of the function can be passed to other jobs by means of toil.job.Job.rv() 🌈

💜 JobFunctionWrappingJob 💜

The subclass of FunctionWrappingJob for wrapping user job functions. It's giving organized 🍓

class toil.job.JobFunctionWrappingJob(userFunction, *args, **kwargs)

A job function is a function whose first argument is a Job instance, which is the wrapping job for the function. This lets the function add successor jobs and use everything the Job class provides 👑

To give the job function access to the toil.fileStores.abstractFileStore.AbstractFileStore instance (see toil.job.Job.run()), it's made into a variable of the wrapping job called fileStore 🦋

To specify a job's resource requirements, these default keyword arguments are available:

• memory
• disk
• cores
• accelerators
• preemptible

For example, to wrap a function into a job, you'd call:

Job.wrapJobFn(myJob, memory='100k', disk='1M', cores=0.1)

Parameters

• userFunction (Callable[..., Any])
• args (Any)
• kwargs (Any)

run(fileStore) ✨

Override this function to perform work and dynamically create successor jobs 💅

Parameters

• fileStore (AbstractFileStore) — used to create local and globally sharable temporary files, and to send log messages to the leader process 🫶 (the full name is toil.fileStores.abstractFileStore.AbstractFileStore)

Return type Any

Okay so the return value of your function can be passed along to other jobs using toil.job.Job.rv(), which means it's basically a gift for your downstream jobs 🎁✨

🌸 EncapsulatedJob

EncapsulatedJob is the Job subclass for wrapping up a job, so a whole subgraph of jobs can act like one single job 🫶

💖 class toil.job.EncapsulatedJob(job, unitName=None)

This is a convenience Job class that makes a job subgraph look like a single job. Bestie, it's giving main character energy 👑

Say A is the root job of a subgraph and B is another job we want to run after A and all of its successors are finished. Here's how you encapsulate it:

#  Job A and subgraph, Job B
A, B = A(), B()
Aprime = A.encapsulate()
Aprime.addChild(B)
#  B will run after A and all its successors have completed, A and its subgraph of
# successors in effect appear to be just one job.

If the job being encapsulated has predecessors (so it isn't the root job), the encapsulated job inherits those predecessors. But if predecessors get added to the job being encapsulated after the encapsulated job is created, the encapsulating job will NOT inherit them automatically 😭 So make sure the encapsulated job has the right set of predecessors, ngl this part matters a lot.

🙈 Watch out: predecessors added after the encapsulated job exists are not inherited automatically. Double-check the predecessor set yourself!

The return value of an encapsulated job (the one you get from toil.job.Job.rv()) is the return value of the root job. So A().encapsulate().rv() and A().rv() resolve to the same value once A or A.encapsulate() has run. Iconic consistency 💅

Parameters

• job (Job | None)
• unitName (str | None)

🌷 __init__(job, unitName=None)

Parameters

• job (Job | None) — the job to encapsulate.
• unitName (str | None) — a human-readable name to identify this job instance.

Return type: None

💅 addChild(childJob)

Adds a childJob to be run as a child of this job. Child jobs run directly after this job's toil.job.Job.run() method has finished, so they're up next once the run is done ✨

Return type: Job

Returns: childJob, for call chaining 🔗

Parameters:

• childJob (Job)

🧚 addService(service, parentService=None)

Adds a service. The toil.job.Job.Service.start() method of the service gets called after the run method has completed but before any successors run. The service's toil.job.Job.Service.stop() method gets called once the job's successors have run 🌈

Services are how things like databases and servers get started up and accessed by jobs in a workflow. Lowkey so useful 🍓

⚠️ Raises toil.job.JobException: if the service has already been made the child of a job or of another service.

Parameters

• service — the Service to add.
• parentService — the Service that will be started before service is started. This lets you build trees of services. parentService must be a service of this job.

Returns: a promise that will be replaced with the return value from toil.job.Job.Service.start() of service in any successor of the job 🦋

🎀 addFollowOn(followOnJob)

Adds a follow-on job. Follow-on jobs run after the child jobs and their successors have been run, so they're the last to slay the stage 👑

Return type: Job

Returns: followOnJob, for call chaining

Parameters:

• followOnJob (Job)

🫧 rv(*path)

Creates a promise (toil.job.Promise). This is the "promise" that represents a return value of the job's run method or, if it's a function-wrapping job, the wrapped function's return value 🎁

Parameters:

• path (Any) — optional path for selecting a component of the promised return value. If it's absent or empty, the entire return value gets used. Otherwise, the first element of the path selects an individual item from the return value. For that to work, the return value must be a list, dictionary, or any other type implementing the __getitem__() magic method. If the selected item is another composite value, the second element of the path can select an item from it, and so on. For example, if the return value is [6,{'a':42}], .rv(0) would select 6, rv(1) would select {'a':3}, and rv(1,'a') would select 3. To select a slice from a return value that's sliceable (like a tuple or list), the path element should be a slice object. For example, if the return value is [6, 7, 8, 9], then .rv(slice(1, 3)) would select [7, 8]. Note that slicing really only makes sense at the end of the path 🔪

Return type: Promise

Returns: a promise representing the return value of this job's toil.job.Job.run() method 🌸

🦄 prepareForPromiseRegistration(jobStore)

Sets things up so this job's promises can register themselves. This prepares this job (the promisor) so its promises can register with it when the jobs they're promised to (the promisees) get serialized 🧩

The promisee holds the reference to the promise, usually as part of its job arguments. When it's being pickled, so are the promises it refers to. Pickling a promise triggers it to register with the promisor. It's a whole chain reaction 💖

🎁 Promise

Promise is the class used to reference return values of jobs and services that haven't run or started yet 🔮

💜 class toil.job.Promise(job, path)

This references a return value from a method as a promise before that method has actually run. Specifically, it covers a return value from a toil.job.Job.run() or toil.job.Job.Service.start() method 🌙

Let T be a job. Instances of Promise (called a promise) are returned by T.rv(), which references the return value of T's run function. When the promise is passed to the constructor (or as an argument to a wrapped function) of a different, successor job, the promise gets replaced by the actual referenced return value. This is the magic that lets one job's run method return value be an input argument to a job before that first job's run function has executed. Big brain energy 🧠

Parameters

• job (Job)
• path (Any)

Return type: Promise

🌟 filesToDelete = {}

A set of IDs of files containing promised values, for when we know we won't need them anymore 🗑️

🪄 __init__(job, path)

Initializes this promise.

Parameters

• job (Job) — the job whose return value this promise references.
• path (Any) — see Job.rv().

🦋 class toil.job.PromisedRequirement(valueOrCallable, *args)

This is a class for dynamically allocating a job function's resource requirements, and it involves toil.job.Promise instances. Use it when resource requirements depend on the return value of a parent function 💅

A PromisedRequirement can be modified by passing in a function that takes the Promise as input.

For example, let f, g, and h be functions. Then a Toil workflow can be defined like this:

A = Job.wrapFn(f)
B = A.addChildFn(g, cores=PromisedRequirement(A.rv())
C = B.addChildFn(h, cores=PromisedRequirement(lambda x: 2*x, B.rv()))

Parameters

• valueOrCallable (Any)
• args (Any)

🎀 __init__(valueOrCallable, *args)

Initializes this Promised Requirement.

Parameters

• valueOrCallable (Any) — a single Promise instance or a function that takes args as input parameters. 🫶

🎀 Part 39: Promises, Job Methods, and the Goodies 🎀

• args (int (/usr/share/doc/python3/html/library/functions.html#int) or .Promise): a variable-length argument list, so you can toss in as many as you need 🧚✨

Return type None

🌸 getValue()

Okay, this one's simple and it slays 💅 getValue() gives you back the PromisedRequirement value.

Return type Any (/usr/share/doc/python3/html/library/typing.html#typing.Any)

🫶 static convertPromises(kwargs)

Bestie, this one checks whether a reserved resource keyword is a Promise or PromisedRequirement instance. If it is, it returns True. It also converts a Promise instance into a PromisedRequirement, so it's doing the most 💖

Parameters

• kwargs (dict (/usr/share/doc/python3/html/library/stdtypes.html#dict)[str (/usr/share/doc/python3/html/library/stdtypes.html#str), Any (/usr/share/doc/python3/html/library/typing.html#typing.Any)]): the function keyword arguments 🌈

Return type bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

✨ JOB METHODS API ✨

Okay, this section is the heart of it all 🫶 Jobs are the units of work in Toil, and workflows are built by composing them together. Main character energy 👑

🧚 toil.job.Job

class toil.job.Job(memory=None, cores=None, disk=None,
accelerators=None, preemptible=None, preemptable=None, unitName='',
checkpoint=False, displayName='', descriptionClass=None, local=None,
files=None)

This class represents a unit of work in Toil. It's the building block, iykyk 🧱

Parameters

• memory (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | int (/usr/share/doc/python3/html/library/functions.html#int) | None (/usr/share/doc/python3/html/library/constants.html#None))
• cores (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | int (/usr/share/doc/python3/html/library/functions.html#int) | float (/usr/share/doc/python3/html/library/functions.html#float) | None (/usr/share/doc/python3/html/library/constants.html#None))
• disk (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | int (/usr/share/doc/python3/html/library/functions.html#int) | None (/usr/share/doc/python3/html/library/constants.html#None))
• accelerators (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | int (/usr/share/doc/python3/html/library/functions.html#int) | Mapping (/usr/share/doc/python3/html/library/typing.html#typing.Mapping)[str (/usr/share/doc/python3/html/library/stdtypes.html#str), Any (/usr/share/doc/python3/html/library/typing.html#typing.Any)] | AcceleratorRequirement (#toil.job.AcceleratorRequirement) | Sequence (/usr/share/doc/python3/html/library/typing.html#typing.Sequence)[str (/usr/share/doc/python3/html/library/stdtypes.html#str) | int (/usr/share/doc/python3/html/library/functions.html#int) | Mapping (/usr/share/doc/python3/html/library/typing.html#typing.Mapping)[str (/usr/share/doc/python3/html/library/stdtypes.html#str), Any (/usr/share/doc/python3/html/library/typing.html#typing.Any)] | AcceleratorRequirement (#toil.job.AcceleratorRequirement)] | None (/usr/share/doc/python3/html/library/constants.html#None))
• preemptible (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | int (/usr/share/doc/python3/html/library/functions.html#int) | bool (/usr/share/doc/python3/html/library/functions.html#bool) | None (/usr/share/doc/python3/html/library/constants.html#None))
• preemptable (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | int (/usr/share/doc/python3/html/library/functions.html#int) | bool (/usr/share/doc/python3/html/library/functions.html#bool) | None (/usr/share/doc/python3/html/library/constants.html#None))
• unitName (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | None (/usr/share/doc/python3/html/library/constants.html#None))
• checkpoint (bool (/usr/share/doc/python3/html/library/functions.html#bool) | None (/usr/share/doc/python3/html/library/constants.html#None))
• displayName (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | None (/usr/share/doc/python3/html/library/constants.html#None))
• descriptionClass (type (/usr/share/doc/python3/html/library/functions.html#type) | None (/usr/share/doc/python3/html/library/constants.html#None))
• local (bool (/usr/share/doc/python3/html/library/functions.html#bool) | None (/usr/share/doc/python3/html/library/constants.html#None))
• files (set (/usr/share/doc/python3/html/library/stdtypes.html#set)[FileID (#toil.fileStores.FileID)] | None (/usr/share/doc/python3/html/library/constants.html#None))

🦋 __init__

__init__(memory=None, cores=None, disk=None, accelerators=None,
preemptible=None, preemptable=None, unitName='',
checkpoint=False, displayName='', descriptionClass=None,
local=None, files=None)

This is the Job initializer, so it's where the magic starts ✨ Any overriding constructor must call this method, no exceptions 🙅‍♀️

Parameters

• memory (int (/usr/share/doc/python3/html/library/functions.html#int) or a string convertible by toil.lib.conversions.human2bytes to an int): the maximum number of bytes of memory the job will need to run 🧠
• cores (float (/usr/share/doc/python3/html/library/functions.html#float), int (/usr/share/doc/python3/html/library/functions.html#int), or a string convertible by toil.lib.conversions.human2bytes to an int): the number of CPU cores required 💪
• disk (int (/usr/share/doc/python3/html/library/functions.html#int) or a string convertible by toil.lib.conversions.human2bytes to an int): the amount of local disk space the job needs, expressed in bytes 💾
• accelerators (int (/usr/share/doc/python3/html/library/functions.html#int), string, dict (/usr/share/doc/python3/html/library/stdtypes.html#dict), or list (/usr/share/doc/python3/html/library/stdtypes.html#list) of those. Strings and dicts must be parseable by parse_accelerator): the computational accelerators the job needs. If it's a string, it can be a string of a number, or a string naming a model, brand, or API (with an optional colon-delimited count) 🚀
• preemptible (bool (/usr/share/doc/python3/html/library/functions.html#bool), int in {0, 1}, or string in {'false', 'true'} in any case): whether the job can run on a preemptible node 🌀
• preemptable (str (/usr/share/doc/python3/html/library/stdtypes.html#str) | int (/usr/share/doc/python3/html/library/functions.html#int) | bool (/usr/share/doc/python3/html/library/functions.html#bool) | None (/usr/share/doc/python3/html/library/constants.html#None)): a legacy preemptible parameter, kept for backwards compatibility with workflows that don't use the preemptible keyword 🕰️
• unitName (str (/usr/share/doc/python3/html/library/stdtypes.html#str)): a human-readable name for this instance of the job 🏷️
• checkpoint (bool (/usr/share/doc/python3/html/library/functions.html#bool)): if any of this job's successor jobs completely fails and exhausts all their retries, remove those successor jobs and rerun this job to restart the subtree. The job must be a leaf vertex in the job graph when it's initially defined. See toil.job.Job.checkNewCheckpointsAreCutVertices() 🔁
• displayName (str (/usr/share/doc/python3/html/library/stdtypes.html#str)): a human-readable display name for the job type 🎀
• descriptionClass (class): an override for the JobDescription class used to describe the job 📋
• local (bool (/usr/share/doc/python3/html/library/functions.html#bool) | None (/usr/share/doc/python3/html/library/constants.html#None)): whether the job can run on the leader 👑
• files (set (/usr/share/doc/python3/html/library/stdtypes.html#set)[FileID (#toil.fileStores.FileID)] | None (/usr/share/doc/python3/html/library/constants.html#None)): a set of Files the job will want to use 📁

Return type None

💜 check_initialized()

This one double-checks that Job.__init__() has been called by any subclass __init__(). It works because the self._description instance variable should always be set after __init__(). If __init__() never got called, it raises an error, so no sneaky skipping 🙅‍♀️

Return type None (/usr/share/doc/python3/html/library/constants.html#None)

🎀 property jobStoreID: str | TemporaryID

okay so this one hands you the ID of this Job 🆔✨ main character ID energy fr

Return type: Union[str, TemporaryID]

🌸 property description: JobDescription

this gives you the JobDescription that describes this job 👀🫶 you can literally peek behind the curtain

Return type: JobDescription

💾 property disk: int

the maximum number of bytes of disk the job needs to run. it's giving "bring enough snacks for the trip" 🍓

Return type: int

🧠 property memory: int

the maximum number of bytes of memory the job needs to run. no cap, same energy as disk but for RAM 💜

Return type: int

👑 property cores: int | float

the number of CPU cores required. sometimes a whole number, sometimes a float, we don't judge 🦋

Return type: Union[int, float]

🔥 property accelerators: list[AcceleratorRequirement]

any accelerators the job needs, like GPUs. the glow-up hardware 💅⭐

Return type: list[AcceleratorRequirement]

🦋 property preemptible: bool

whether the job is allowed to run on a preemptible node. bestie, this is the "okay you can get kicked off but we'll still vibe" setting 🌈

Return type: bool

🧚 property checkpoint: bool

figures out whether the job is a checkpoint job or not. iykyk 🙈

Return type: bool

✨ assignConfig(config)

assigns the given config object, and then the various actions inside the Job class get to use it. it's the group chat that everyone reads from 💖

Parameters

• config (Config) — the config object to query

Return type: None

🌈 run(fileStore)

this is the one you override when you want the job to actually do the work and dynamically create successor jobs. main character moment 🔥👑

Parameters

• fileStore (AbstractFileStore) — this is what you use to make local and globally sharable temporary files, and to send log messages to the leader process 📝

Return type: Any

Returns

The return value of the function can be passed to other jobs using toil.job.Job.rv(). passing the tea along 🫖💅

🌸 addChild(childJob)

adds a childJob to run as a child of this job. child jobs run right after this job's toil.job.Job.run() method finishes, so no waiting around 🏃‍♀️💨

Return type: Job

Returns

childJob, so you can keep chaining calls 🔗

Parameters

• childJob (Job)

🦋 hasChild(childJob)

checks whether childJob is already a child of this job. no double-booking the vibes 🙅‍♀️

Return type: bool

Returns

True if childJob is a child of the job, otherwise False 💜

Parameters

• childJob (Job)

🍓 addFollowOn(followOnJob)

adds a follow-on job. follow-ons run after the child jobs and their successors have all run, so they're the encore 🎤✨

Return type: Job

Returns

followOnJob, for call chaining 🔗

Parameters

• followOnJob (Job)

🫶 hasPredecessor(job)

checks whether a given job is already a predecessor of this job. the family tree check 🌳

Parameters

• job (Job)

Return type: bool

💖 hasFollowOn(followOnJob)

checks whether the given job is already a follow-on of this job. lowkey the same energy as hasChild, just for encores 🎶

Return type: bool

Returns

True if followOnJob is a follow-on of this job, otherwise False ✨

Parameters

• followOnJob (Job)

🎀 addService(service, parentService=None)

adds a service. the service's toil.job.Job.Service.start() method gets called after run finishes but before any successors run, and its toil.job.Job.Service.stop() method gets called once the successors of the job have run 🧘‍♀️

services are how you get things like databases and servers started up and accessed by jobs in a workflow. the backstage crew 🎭🌟

heads up: if the service has already been made the child of a job or another service, this raises toil.job.JobException. no stealing other people's services bestie 🙅‍♀️💀

Parameters

• service (Service) — the service to add 💅
• parentService (Service | None) — a service that gets started before service is started. this is how you build trees of services 🌳 the parentService has to be a service of this job

Return type: Promise

Returns

a promise that gets replaced with the return value from toil.job.Job.Service.start() of service in any successor of the job 🦋

🌈 hasService(service)

returns True if the given Service is a service of this job, and False otherwise. yes or no, no vibes-based answers 💅

Parameters

• service (Service)

Return type: bool

🧚 addChildFn(fn, *args, **kwargs)

adds a function as a child job 🌸

Parameters

• fn (Callable) — the function to run as a child job, with *args and **kwargs passed in as its arguments. check out toil.job.FunctionWrappingJob for the reserved keyword arguments you use to set resource requirements 📦

Return type: FunctionWrappingJob

Returns

the new child job that wraps fn 👑

🍓 addFollowOnFn(fn, *args, **kwargs)

adds a function as a follow-on job, same vibe as addChildFn but for the encore slot 🎤

Parameters

• fn (Callable) — the function to run as a follow-on job, with *args and **kwargs passed in as its arguments. the reserved keyword arguments for resource requirements live in toil.job.FunctionWrappingJob 💅

Return type: FunctionWrappingJob

Returns

the new follow-on job that wraps fn ✨

🦋 addChildJobFn(fn, *args, **kwargs)

adds a job function as a child job. if you're wondering what a job function is, peep toil.job.JobFunctionWrappingJob for the full definition 🫶

Parameters

• fn (Callable) — the job function to run as a child job, with *args and **kwargs passed in as its arguments. the reserved keyword arguments for resource requirements are covered in toil.job.JobFunctionWrappingJob 🌈

Return type: FunctionWrappingJob

Returns

the new child job that wraps fn 💖

💜 addFollowOnJobFn(fn, *args, **kwargs)

adds a follow-on job function. for the official definition of a job function, check out toil.job.JobFunctionWrappingJob. it's giving encore with a job function twist 🎶💅

Okay so this first bit is about follow-on jobs 🦋 You hand it a job function, and it runs right after as a follow-on, with your *args and **kwargs going straight into that function ✨

Parameters

• fn (Callable) — the job function you want to run as a follow-on, with *args and **kwargs as its arguments 💖 For the reserved keyword arguments that set resource requirements, peek at toil.job.JobFunctionWrappingJob 🎀

Return type: FunctionWrappingJob

Returns: the brand new follow-on job that wraps fn 🌸

---

property tempDir: str

This one is a shortcut for job.fileStore.getLocalTempDir() 🧚 The temp dir gets created on the very first call, and that same dir comes back for that call and every call after it ✨

Returns: the path to the tempDir. See job.fileStore.getLocalTempDir 🗂️

Return type: str

---

log(text, level=20)

Logs stuff using fileStore.log_to_leader() 📣

Parameters

• text (str)

Return type: None

---

static wrapFn(fn, *args, **kwargs)

Turns a plain function into a Job 🍓 It's a convenience wrapper for the constructor of toil.job.FunctionWrappingJob 💅

Parameters

• fn — the function to run, with *args and **kwargs as its arguments. For reserved keyword arguments that set resource requirements, check out toil.job.JobFunctionWrappingJob 🎀

Return type: FunctionWrappingJob

Returns: the new function that wraps fn 🦋

---

static wrapJobFn(fn, *args, **kwargs)

Turns a job function into a Job 👑 Another convenience wrapper, this time for the constructor of toil.job.JobFunctionWrappingJob 💖

Parameters

• fn — the job function to run, with *args and **kwargs as its arguments. Reserved keyword arguments for setting resource requirements are described in toil.job.JobFunctionWrappingJob 🌈

Return type: JobFunctionWrappingJob

Returns: the new job function that wraps fn ✨

---

encapsulate(name=None)

Encapsulates the job, see toil.job.EncapsulatedJob 🫶 It's a convenience function for the constructor of toil.job.EncapsulatedJob 💜

Parameters

• name (str | None) — a human-readable name for the encapsulated job 🏷️

Return type: EncapsulatedJob

Returns: an encapsulated version of this job 🎁

---

rv(*path)

This makes a promise (toil.job.Promise) 🔮 The promise stands in for the return value of the job's run method, or, if it's a function-wrapping job, the wrapped function's return value 🧠

Parameters

• path (Any) — an optional path for picking out a piece of the promised return value 🎯 If it's absent or empty, you get the whole return value. Otherwise, the first element of the path picks out one item of the return value. For that to work, the return value has to be a list, a dictionary, or any other type that implements the __getitem__() magic method. If the picked item is another composite value, the second element of the path can pick an item from that, and so on. For example, if the return value is [6,{'a':42}], .rv(0) selects 6, rv(1) selects {'a':3}, and rv(1,'a') selects 3. To grab a slice from a slicable return value, like a tuple or list, the path element should be a slice object. For example, if the return value is [6, 7, 8, 9], then .rv(slice(1, 3)) selects [7, 8].

Heads up bestie: slicing only really makes sense at the end of the path 🙈

Return type: Promise

Returns: a promise representing the return value of this job's toil.job.Job.run() method 💅

---

prepareForPromiseRegistration(jobStore)

This sets things up so this job's promises can register themselves with it 🦋 It preps this job (the promisor) so its promises can sign up with it when the jobs they're promised to (the promisees) get serialized 🌸

The promisee holds the reference to the promise, usually inside its job arguments. When it gets pickled, so do the promises it refers to, and pickling a promise triggers it to register with the promisor 📦

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore), which you'll see as AbstractJobStore 💾

Return type: None

---

checkJobGraphForDeadlocks()

This makes sure a graph of Jobs (one that hasn't been saved to the JobStore yet) doesn't have any pathological relationships that would cause deadlocks if we tried to run the jobs 💀 Not on my watch 😤

For more info, see toil.job.Job.checkJobGraphConnected(), toil.job.Job.checkJobGraphAcyclic() and toil.job.Job.checkNewCheckpointsAreLeafVertices() 🔍

Raises: toil.job.JobGraphDeadlockException if the job graph is cyclic, contains multiple roots, or has checkpoint jobs that aren't leaf vertices when they're defined (see toil.job.Job.checkNewCheckpointsAreLeaves()) 🚨

---

getRootJobs()

This returns the set of root job objects that contain this job 👑 A root job is a job with no predecessors, meaning it isn't a child, follow-on, or service of anything 🌱

It only deals with jobs created right here, not jobs loaded from the job store 🫶

Return type: set[Job]

---

checkJobGraphConnected()

Raises: toil.job.JobGraphDeadlockException if toil.job.Job.getRootJobs() doesn't contain exactly one root job 😬

Return type: None

Execution always starts from one root job, so having multiple root jobs will cause a deadlock to occur.

Warning: multiple root jobs = deadlock 💀 Keep it to exactly one root, periodt 💅

It only deals with jobs created here, not jobs loaded from the job store 🌸

Return type: None

---

checkJobGraphAcylic()

Raises: toil.job.JobGraphDeadlockException if the connected component of jobs containing this job has any cycles of child/followOn dependencies in the augmented job graph (see below) 🔁 Cycles like that aren't allowed in valid job graphs, so don't even try 🙅‍♀️

Return type: None

Here's the vibe on implied edges: a follow-on edge (A, B) between two jobs A and B is equivalent to adding a child edge to B from (1) A, (2) each child of A, and (3) the successors of each child of A. We call each of those an "implied" edge 🧩 The augmented job graph is a job graph that includes all the implied edges ✨

For a job graph G = (V, E), the algorithm is O(|V|^2). It's O(|V| + |E|) for a graph with no follow-ons. The former follow-on case could be improved! 📈

It only deals with jobs created here, not jobs loaded from the job store 🫶

Return type: None

---

checkNewCheckpointsAreLeafVertices()

So a checkpoint job is one that gets restarted if it fails, or if any of its successors completely fails and exhausts their retries 🔄

A job is a leaf if it has no successors 🍃

A checkpoint job has to be a leaf when it's first added to the job graph. Once its run method is invoked, it can then create direct successors. This restriction is there to simplify the implementation 🧚

Only works on connected components of jobs that haven't been added to the JobStore yet, bestie 💅

• Raises toil.job.JobGraphDeadlockException 😭 — if there's a job being added to the graph where checkpoint=True and it's not a leaf.

Return type: None 🫶

---

✨ defer(function, *args, **kwargs) ✨

okay so this one registers a deferred function, which is just a callable that gets invoked after the current attempt at running this job wraps up 🦋 An attempt is considered concluded when the job function (or the toil.job.Job.run() method for class-based jobs) returns, raises an exception, or after the process running it terminates abnormally. Main character energy for cleanup 💖

A deferred function gets called on the node that tried to run the job, even if a later attempt happens on another node 🌸 It should be idempotent, because it may be called multiple times on the same node or even in the same process. So make sure running it twice is totally fine 🙈

You can register more than one deferred function per job attempt by calling this method repeatedly with different arguments. If the same function is registered twice with the same or different arguments, it'll be called twice per job attempt 💀

Examples of deferred functions are ones that handle cleanup of resources external to Toil, like Docker containers, files outside the work directory, etc. 🧹✨

Parameters

• function (callable) 🎀 — the function to be called after this job concludes.
• args (list) 🌈 — the arguments to the function.
• kwargs (dict) 🍓 — the keyword arguments to the function.

Return type: None 💅

---

✨ getTopologicalOrderingOfJobs() ✨

Return type: list[Job] 👑

Returns: a list of jobs such that for all pairs of indices i, j for which i < j, the job at index i can be run before the job at index j. It's giving ordering 💜

This only considers jobs in this job's subgraph that are newly added, not loaded from the job store. And it ignores service jobs 🙈

---

✨ saveBody(jobStore) ✨

this saves the execution data for just this job to the JobStore, and fills in the JobDescription with the information needed to retrieve it later 🦋

Heads up: the Job's JobDescription must have already been given a real jobStoreID first, or this won't work 😭

It does not save the JobDescription itself 🙅‍♀️

Parameters

• jobStore (AbstractJobStore) 🧚 — the job store to save the job body into.

Return type: None ✨

---

✨ saveAsRootJob(jobStore) ✨

this saves the job to the given jobStore as the root job of the workflow 👑 The main character of the whole thing 💅

Return type: JobDescription 🌸

Returns: the JobDescription describing this job.

Parameters

• jobStore (AbstractJobStore) 🍓

---

✨ classmethod loadJob(job_store, job_description) ✨

this retrieves a toil.job.Job instance from a JobStore, so it's basically the reverse of saving 🪄

Parameters

• job_store (AbstractJobStore) 🌈 — the job store.
• job_description (JobDescription) 🫶 — the JobDescription of the job to retrieve.

Return type: Job 💖

Returns: the job referenced by the JobDescription.

---

✨ set_debug_flag(flag) ✨

turns on the given debug option for the job 🔥

Parameters

• flag (str) 💜

Return type: None 🎀

---

✨ has_debug_flag(flag) ✨

returns true if the given debug flag is set, so you can check before you do something 👀

Parameters

• flag (str) 🌸

Return type: bool 💅

---

✨ files_downloaded_hook(host_and_job_paths=None) ✨

this is a function subclasses can call once they've downloaded their input files 📥

⚠️ If the "download_only" debug flag is set, this will abort the job. Be careful with that one, bestie 😭

It can also be hinted a list of file path pairs, one outside and one inside the job container, in which case the container environment can be reconstructed 🧩

Parameters

• host_and_job_paths (list[tuple[str, str]] | None) 🦋

Return type: None ✨

---

💖 JobDescription 💖

okay this is the class used to store all the information that the Toil Leader ever needs to know about a Job 👑

class toil.job.JobDescription(requirements, jobName, unitName='', displayName='', local=None, files=None)

It stores all the info the Toil Leader could ever want, which includes:

• Resource requirements. 💪
• Which jobs are children, follow-ons, or predecessors of this job. 🌳
• A reference to the Job object in the job store. 🗄️

It can be obtained from an actual (executable) Job object, and it can be used to get the Job object back from the JobStore 🔄

It never contains other Jobs or JobDescriptions. Everything is referenced by ID, no cap 🫶 It also gets subclassed into variants for checkpoint jobs and service jobs, each with their own specific parameters 🌈

Parameters

• requirements (Mapping[str, int | str | float | bool | list]) 🎀
• jobName (str) 🌸
• unitName (str | None) 🍓
• displayName (str | None) 🧚
• local (bool | None) 🦋
• files (set[FileID] | None) 💜

✨ __init__(requirements, jobName, unitName='', displayName='', local=None, files=None) ✨

this creates a new JobDescription 🎉

Parameters

• requirements (Mapping[str, int | str | float | bool | list]) 💅 — a dict from string to number, string, or bool describing the resource requirements of the job. The 'cores', 'memory', 'disk', and 'preemptible' fields, if set, get parsed and broken out into properties. If unset, the relevant property is unspecified, and it gets pulled from the assigned Config object when queried (see toil.job.Requirer.assignConfig()) 🔥
• jobName (str) — 💖 the kind of job this is! It's used in job store IDs and logging, and the cluster scaler uses it to learn how long this kind of job takes. If there's no real user-defined name, it ought to be the job class's name ✨
• unitName (str | None) — the name of this particular instance of the job type. It can show up next to jobName in logs 🦋
• displayName (str | None) — a human-readable name for this specific job instance 🌸 If there's no real user-defined name, it ought to be the job class's name.
• local (bool | None) — 🏃 if True, the job uses minimal resources but is sensitive to latency, so the leader should run it.
• files (set[FileID] | None) — a set of FileID objects the job plans to use 📁

Return type: None

get_names()

Gets the names and ID of this job as a named tuple 🏷️

Return type: Names

get_chain()

Gets every job that ran in this job's chain, in order, so you can see the whole lineup 👑

For each job, you get a named tuple with its various names and its original job store ID. The jobs are listed in execution order. If the job hasn't run yet or it didn't chain, you just get a one-item list.

Return type: list[Names]

serviceHostIDsInBatches()

Finds all the batches of service host job IDs that can start at the same time 🧚

Batches come back in the order they need to start in.

Return type: Iterator[list[str]]

successorsAndServiceHosts()

Gives you an iterator over all child, follow-on, and service job IDs 💅

Return type: Iterator[str]

allSuccessors()

Gives you an iterator over all child, follow-on, and chained, inherited successor job IDs ✨ Follow-ons come before children.

Return type: Iterator[str]

successors_by_phase()

Gives you an iterator over all child, follow-on, and chained inherited successor job IDs, plus their phase number on the stack 🦋

Phases run from higher numbers down to lower numbers.

Return type: Iterator[tuple[int, str]]

services (property)

Gets a collection of the IDs of service host jobs for this job, in arbitrary order 🌈 It's empty if the job has no unfinished services.

has_body()

Returns True if a job body is attached, and False otherwise 🫶

Return type: bool

attach_body(file_store_id, user_script)

Attaches a job body to this JobDescription 🎀

It takes the file store ID that the body is stored at, plus the required user script module. The file store ID can also be "firstJob" for the root job, which is stored as a shared file instead.

Parameters:

• file_store_id (str)
• user_script (ModuleDescriptor)

Return type: None

detach_body()

Drops the body reference from a JobDescription 🙈

Return type: None

get_body()

Gets the info needed to load the job body 🔥

Return type: tuple[str, ModuleDescriptor]

Returns: a file store ID (or the magic shared file name "firstJob") and a user script module.

⚠️ This one fails if no body is attached, so check has_body() first, bestie!

nextSuccessors()

Returns the collection of job IDs for this job's successors that are ready to run 🌸 Just because they're ready doesn't mean they're free, though. If they have multiple predecessor relationships, they may still be blocked on other jobs.

It returns None when we're at the final phase (all successors are done). It returns an empty collection if there are more phases but they can't be entered yet, like when we're waiting for the job itself to run.

Return type: set[str] | None

filterSuccessors(predicate)

Keeps only the successor jobs that the given predicate function approves 💅

The predicate is called with each job's ID. Every other successor gets treated as complete and forgotten 💀

Parameters:

• predicate (Callable[[str], bool])

Return type: None

filterServiceHosts(predicate)

Keeps only the services that the given predicate approves 🧚

The predicate is called with the service host job's ID. Every other service gets treated as complete and forgotten 👋

Parameters:

• predicate (Callable[[str], bool])

Return type: None

clear_nonexistent_dependents(job_store)

Removes all references to child, follow-on, and associated service jobs that don't exist anymore 🗑️ In other words, all the ones that have been completed and removed.

Parameters:

• job_store (AbstractJobStore)

Return type: None

clear_dependents()

Removes all references to successor and service jobs. Full clean slate 🧹

Return type: None

is_subtree_done()

Checks whether the subtree is done 🌈

Return type: bool

Returns: True if the job appears to be done, and all related child, follow-on, and service jobs appear to be finished and removed.

replace(other)

Takes on the ID of another JobDescription, while keeping our own state and type 🦋 When it gets updated in the JobStore, we save over the other JobDescription.

This is super useful for chaining jobs: the chained-to job can replace the parent job 👑

Okay so this one merges the cleanup state and successors (everything except this job itself) from the job being replaced into this one 🧹✨ glow-up energy, bestie 💅

Parameters

• other (JobDescription) — the job description being replaced 🎀

Return type

• None

assert_is_not_newer_than(other)

Vibe check time 🕰️ this makes sure this JobDescription is not newer than a prospective new version of it. No time travelling allowed 💜

Parameters

• other (JobDescription)

Return type

• None

is_updated_by(other)

This returns True if the passed JobDescription is a distinct, newer version of this one 👑 it's giving main character update.

Parameters

• other (JobDescription)

Return type

• bool

addChild(childID)

Makes the job with the given ID a child of the described job 🧚

Parameters

• childID (str)

Return type

• None

addFollowOn(followOnID)

Makes the job with the given ID a follow-on of the described job 🦋

Parameters

• followOnID (str)

Return type

• None

addServiceHostJob(serviceID, parentServiceID=None)

Makes the ServiceHostJob with the given ID a service of the described job 🌸 If you pass a parent ServiceHostJob ID, that parent service starts first, and it has to have been added already. No line-cutting 🙅‍♀️

hasChild(childID)

Returns True if the job with the given ID is a child of the described job 🫶

Parameters

• childID (str)

Return type

• bool

hasFollowOn(followOnID)

Tests if the job with the given ID is a follow-on of the described job 🔍

Parameters

• followOnID (str)

Return type

• bool

hasServiceHostJob(serviceID)

Tests if the ServiceHostJob is a service of the described job 🧐

Return type

• bool

renameReferences(renames)

Applies the given dict of ID renames to all references to jobs 🔁

Okay but it does not touch our own ID or the IDs of finished predecessors, so those stay exactly as they are 🙅‍♀️ and any IDs that aren't in the renames dict also get left alone 🌷

Parameters

• renames (dict[TemporaryID, str]) — the rename operations to apply 💖

Return type

• None

addPredecessor()

Lets the JobDescription know that a predecessor has been added to its Job 🧩

Return type

• None

onRegistration(jobStore)

Does setup work that needs the JobStore ✨ It gets called by the Job saving logic once this JobDescription meets the JobStore and has its ID assigned.

This is also the method you override when you need setup (like hooking up flag files for service jobs) that depends on the JobStore 🚩

Parameters

• jobStore (AbstractJobStore) — the job store we are being placed into 🏠

Return type

• None

setupJobAfterFailure(exit_status=None, exit_reason=None)

Configures the job after a failure 🤕 It reduces remainingTryCount if it's greater than zero, and sets the memory to be at least as big as the default memory, since running out of memory is super common 💀

This one needs a configuration to already be assigned, so check out toil.job.Requirer.assignConfig() first 🚨

Parameters

• exit_status (int | None) — the exit code from the job 🔢
• exit_reason (BatchJobExitReason | None) — the reason the job stopped, if the batch system can tell us 📋

Return type

• None

getLogFileHandle(jobStore)

Creates a context manager that yields a file handle to the log file 📜 It assumes logJobStoreFileID is set.

remainingTryCount (property)

Gets the number of tries remaining ⏳ You get the try count that's set on the JobDescription, or the default based on the retry count from the config if none is set.

clearRemainingTryCount()

Clears remainingTryCount and resets it back to its default value 🧽

Return type

• bool

It returns True if a modification to the JobDescription was made, and False otherwise 💅

reserve_versions(count)

Reserves a job version number for later, so it can be journaled asynchronously 📚

Parameters

• count (int)

Return type

• None

pre_update_hook()

Runs before a created or updated version of this job gets pickled and saved 💾 It's called by the job store.

Return type

• None

✨ JOB.RUNNER API ✨

okay so the Runner is literally the bestie who sets up and kicks off your whole Toil run 💅 it has every method you need to get things configured and started 🚀

💖 class Job.Runner 💖

Used to set up and run a Toil workflow. ✨

🎀 getDefaultArgumentParser() 🎀

static getDefaultArgumentParser(jobstore_as_flag=False)

This gives you an argument parser with all the Toil workflow options already baked in 🍓 ngl this is the Right Way to grab an argument parser in a Toil Python workflow 👑

Parameters

• jobstore_as_flag (bool) — makes the job store option a --jobStore flag instead of a required jobStore positional argument 🦋

Return type

• ArgumentParser

Returns

• the argument parser a Toil workflow uses, with the Toil options added on top ✨

🌸 getDefaultOptions() 🌸

static getDefaultOptions(jobStore=None, jobstore_as_flag=False)

Gets the default options for a Toil workflow, so you don't have to start from scratch 💜

Parameters

• jobStore (str | PathLike[str] | None) — a string describing the job store for the workflow 🧚
• jobstore_as_flag (bool) — makes the job store option a --jobStore flag instead of a required jobStore positional argument 🦋

Return type

• Namespace

Returns

• the options a Toil workflow uses 💅

🦋 addToilOptions() 🦋

static addToilOptions(parser, jobstore_as_flag=False)

This adds the default Toil options to an optparse or argparse parser object 🌈

⚠️ Heads up bestie: you should probably use getDefaultArgumentParser() instead, since it makes a parser of the correct class so Toil's config file and environment variables actually work. If the parser you pass here is just an argparse.ArgumentParser and not a configargparse.ArgParser, Toil's config file and environment variables will not be respected 💀

Parameters

• parser (OptionParser | ArgumentParser) — the options object you want to add Toil options to 🫶
• jobstore_as_flag (bool) — makes the job store option a --jobStore flag instead of a required jobStore positional argument 🦋

Return type

• None

🔥 startToil() 🔥

static startToil(job, options)

🚨 Deprecated! This has been deprecated by toil.common.Toil.start, so use that one instead 🙈

This runs the Toil workflow using the given options. The options are built with getDefaultOptions and addToilOptions, and the workflow starts with this job ✨

Parameters

• job (Job) — the root job of the workflow 👑

Raises

• toil.exceptions.FailedJobsException if, at the end of the function, there are still failed jobs 😭

Return type

• Any

Returns

• the return value of the root job's run function 🎀

---

✨ JOB.FILESTORE API ✨

okay the AbstractFileStore is basically an abstraction of a Toil run's shared storage, and it's giving organized 🗂️💖

🧚 class toil.fileStores.abstractFileStore.AbstractFileStore 🧚

class toil.fileStores.abstractFileStore.AbstractFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)

This is the interface that lets user code run by Toil read and write files 📂 It also gives that code access to other Toil facilities, including:

• normal (non-real-time) logging 📝
• finding the correct temporary directory for scratch work 🧹
• importing and exporting files into and out of the workflow 📦

It stores user files in the jobStore, but keeps them separate from the actual jobs 🦋 It may implement caching 🌈

It gets passed as an argument to the toil.job.Job.run() method 💅

🔒 Careful! Access to files is only permitted inside the context manager provided by toil.fileStores.abstractFileStore.AbstractFileStore.open(). Anything outside that is a no-go 🙅‍♀️

It's also responsible for committing completed jobs back to the job store with an update operation, and it lets that commit operation be waited for ⏳

Parameters

• jobStore (AbstractJobStore)
• jobDesc (JobDescription)
• file_store_dir (str)
• waitForPreviousCommit (Callable[[], Any])

🎀 __init__() 🎀

__init__(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)

This creates a brand new file store object 🌸

Parameters

• jobStore (AbstractJobStore) — the job store in use for the current Toil run 🏠
• jobDesc (JobDescription) — the JobDescription object for the currently running job 🎯
• file_store_dir (str) — the per-worker local temporary directory where the file store should keep local files. The file store creates per-job directories under here 🗃️
• waitForPreviousCommit (Callable[[], Any]) — the waitForCommit method of the previous job's file store, for when jobs run in sequence on the same worker. It stops this file store's startCommit and the previous job's startCommit from running at the same time and racing. If they did race, the later job could be fully marked as completed in the job store before the earlier one 😱

Return type

• None

🌈 createFileStore() 🌈

static createFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit, caching)

This creates a concrete FileStore 💖

Parameters

• jobStore (AbstractJobStore)
• jobDesc (JobDescription)
• file_store_dir (str)
• waitForPreviousCommit (Callable[[], Any])
• caching (bool | None)

Return type

• NonCachingFileStore | CachingFileStore

💜 shutdownFileStore() 💜

static shutdownFileStore(workflowID, config_work_dir, config_coordination_dir)

This does any filestore-specific cleanup that's needed 🧽

💀 Destructive operation! This is a destructive operation, so it's super important that no other running processes on the system are modifying or using the file store for this workflow. Otherwise, it's a whole mess 😭

This is meant to be the last call to the file store in a Toil run. The batch system cleanup function calls it when the batch system shuts down 🌙

Parameters

• workflowID (str) — the workflow ID for this invocation of the workflow 🆔
• config_work_dir (str | None) — the path to the work directory in the Toil Config 🗂️
• config_coordination_dir (str | None) — the path to the coordination directory in the Toil Config 📂✨

Return type: None 🌸

---

✨ open(job) 🧚

Okay so this is the context manager that wraps around a job, handling the stuff that happens right before and right after it runs 🎀

Heads up, bestie: file operations are ONLY allowed inside the context manager 🙅‍♀️

Implementations must only yield from within with super().open(job):. 💅

Parameters

• job (Job) — the Toil job instance that's about to run 💜

Return type: Generator[None, None, None]

---

💖 get_disk_usage()

This gives you the number of bytes of disk used by the last job that ran under open() 📊

Disk usage gets measured at the very end of the job. TODO: sample periodically and record peak usage 👀

Return type: int | None

---

🦋 getLocalTempDir()

Get a brand new local temporary directory to dump your files into 🗂️

It only sticks around for the duration of the job, lowkey a temporary era 🌷

Return type: str

Returns: the absolute path to a new local temporary directory. It exists for the duration of the job only, and it's guaranteed to get deleted once the job terminates, along with every file inside it, recursively 🧹✨

---

🌈 getLocalTempFile(suffix=None, prefix=None)

Get a new local temporary file that persists for the duration of the job 📄

Parameters

• suffix (str | None) — if it's not None, the file name ends with this string. Otherwise, the default ".tmp" is used 🍓
• prefix (str | None) — if it's not None, the file name starts with this string. Otherwise, the default "tmp" is used 🧚

Return type: str

Returns: the absolute path to a local temporary file. It exists for the duration of the job only, and is guaranteed to be deleted once the job terminates 💅

---

🔥 getLocalTempFileName(suffix=None, prefix=None)

Get a valid name for a new local file, but don't actually create anything at that path 🙈

Parameters

• suffix (str | None) — if it's not None, the file name ends with this string. Otherwise, the default ".tmp" is used 🍓
• prefix (str | None) — if it's not None, the file name starts with this string. Otherwise, the default "tmp" is used 🧚

Return type: str

Returns: the path to a valid file 🫶

---

👑 abstractmethod writeGlobalFile(localFileName, cleanup=False)

Upload a file (given as a path) to the job store 🚀

If the file lives in a FileStore-managed temporary directory (the kind you get from toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir()), it becomes a local copy of the file. That copy is eligible for deletion by toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile() 🧼

If you upload an executable file from the local filesystem, its executability is preserved when it gets downloaded again. Main character energy 🎀⚡

Parameters

• localFileName (str) — the path to the local file you want to upload. The last path component (the basename of the file) stays associated with the file in the file store, if the backing JobStore supports it. That way the file can be searched for by name or name glob 🔍
• cleanup (bool) — if True, the copy of the global file gets deleted once the job and all its successors have finished running. If not, the global file has to be deleted manually, so don't forget 😭

Return type: FileID (toil.fileStores.FileID)

Returns: an ID you can use to retrieve the file 🎟️

---

💜 writeGlobalFileStream(cleanup=False, basename=None, encoding=None, errors=None)

This is basically writeGlobalFile, but it lets you write a stream straight into the job store 🌊 The yielded file handle doesn't need to be closed explicitly, and you should not close it explicitly 🙅‍♀️

Parameters

• encoding (str | None) — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which means binary mode 🔢
• errors (str | None) — how encoding errors get handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified ⚠️
• cleanup (bool) — works the same as in toil.fileStores.abstractFileStore.AbstractFileStore.writeGlobalFile() 🧹
• basename (str | None) — if the backing JobStore supports it, this basename gets used so that when you search the job store with a query matching that basename, the file will be detected 🔎

Return type: Iterator[tuple[WriteWatchingStream, FileID]]

Returns: a context manager that yields a tuple of 1) a file handle you can write to, and 2) the toil.fileStores.FileID of the resulting file in the job store 🦋

---

🧚 logAccess(fileStoreID, destination=None)

Record that the given file was read by the job 📝 (to be announced if the job fails 📣)

If destination is not None, it gives the path the file was downloaded to. Otherwise, it assumes the file was streamed 🌸

Must be called by the readGlobalFile() and readGlobalFileStream() implementations. No skipping this step, periodt 💅

Parameters

• fileStoreID (FileID | str)
• destination (str | None)

Return type: None 💅

🦋 readGlobalFile

abstractmethod readGlobalFile(fileStoreID, userPath=None,
cache=True, mutable=False, symlink=False)

Okay so this one is how you make a file from the job store show up locally. 🫶 It's giving "I need that file on my machine, now."

If mutable is True, you get a local copy, so the original stays untouched and doesn't change the file for other jobs. ✨ If mutable is False, Toil can make a link to the file instead, which saves disk space. 💾 The downloaded file is executable only if it was originally uploaded from an executable file on the local filesystem. 🔥

If you give a userPath, that's where the file goes. If you don't, it lands in the local temp directory with an encoded name. 🧚

⚠️ Don't delete the destination file yourself! 💀 Only deleteLocalFile is allowed to remove it.

Implementations must call logAccess() to report the download. 📝

Parameters

• fileStoreID (str) — the job store id for the file 🆔
• userPath (str | None) — the path and name where the global file gets copied or hard-linked (see above) 📁
• cache (bool) — described in toil.fileStores.CachingFileStore.readGlobalFile() 🧠
• mutable (bool) — also described in toil.fileStores.CachingFileStore.readGlobalFile() 🪞
• symlink (bool) — True if the caller can take a symlink, False if it can only take a normal file or a hardlink 🔗

Return type

str

Returns

An absolute path to a local, temporary copy of the file keyed by fileStoreID. 🌸

🌈 readGlobalFileStream

abstractmethod readGlobalFileStream(fileStoreID, encoding=None,
errors=None)

This is basically readGlobalFile's chill cousin: it reads a stream straight from the job store. 🎀

The file handle you get back doesn't need to be closed, and you should not close it yourself. 🙅‍♀️

Parameters

• encoding (str | None) — the name of the encoding used to decode the file. The encodings are the same as for decode(). It defaults to None, which means binary mode. 🔤
• errors (str | None) — an optional string that says how encoding errors get handled. These work the same as for open(). If you specify an encoding, it defaults to 'strict'. 🚨
• fileStoreID (str) 🆔

Return type

ContextManager[IO[bytes] | IO[str], bool | None]

Implementations must call logAccess() to report the download. 📝

Return type

ContextManager[IO[bytes] | IO[str], bool | None]

Returns

A context manager that yields a file handle you can read from. 📖

Parameters

• fileStoreID (str) 🆔
• encoding (str | None) 🔤
• errors (str | None) 🚨

📏 getGlobalFileSize

getGlobalFileSize(fileStoreID)

This one gets the size of the file behind the given ID, in bytes. 🧮

If it's a FileID or anything else with a non-None size field, Toil just uses that. ⚡

Otherwise, it asks the job store to poll the file's size. 🕵️‍♀️

Heads up: the job store might overestimate the size. This can happen if the file is encrypted and had an IV or other encryption framing added. 🔐

Parameters

• fileStoreID (FileID | str) — the File ID for the file 🆔

Return type

int

Returns

The file's size in bytes, as stored in the job store. 💜

🗑️ deleteLocalFile

abstractmethod deleteLocalFile(fileStoreID)

This deletes the local copies of files linked to the job store ID you pass in. 🧹

If no such local copies exist, it raises an OSError with an errno of errno.ENOENT. 💀 That means you can't call it several times in a row. 🙈

The files it deletes are all of these:

• Everything the current job read from this file ID with readGlobalFile into the job's file-store-provided temp directory. 📂
• The file that was written to create the given file ID, if the current job wrote it from the job's file-store-provided temp directory. ✍️

Parameters

• fileStoreID (FileID | str) — the File Store ID of the file to be deleted 🗑️

Return type

None

💥 deleteGlobalFile

abstractmethod deleteGlobalFile(fileStoreID)

This one deletes the local files and then permanently removes them from the job store. 😤

To keep the job restartable if it needs to be, the delete doesn't happen until after the job's run method has finished. 🔁

Parameters

• fileStoreID (FileID | str) — the File Store ID of the file to be deleted 🆔

Return type

None

📣 log_to_leader

log_to_leader(text, level=20)

This sends a logging message to the leader. 📨 The worker logs the same message at the same level too. 🪵

It doesn't depend on the commit system, so it's safe to use during an asynchronous commit, or when there's no commit afterward. 🫧

Parameters

• text (str) — the string to log ✏️
• level (int) — the logging level 📶

Return type

None

🌊 log_user_stream

log_user_stream(name, stream)

This sends a stream of UTF-8 text to the leader as a named log stream. 🌸

It's super useful for things like the error logs of Docker containers. 🐳 The leader either shows it to the user or organizes it nicely for user-level log info. 👑

Parameters

• name (str) — a hierarchical, .-delimited string 🌳
• stream (IO[bytes]) — a stream of encoded text. Encoding errors are tolerated, so it won't crash over them. 🫶

Return type: None · /usr/share/doc/python3/html/library/constants.html#None 💅

abstractmethod startCommit(jobState=False) 🔥 Okay bestie, this one updates the job's status on disk ✨ It's allowed to bump the job's version number, and it may kick off an asynchronous process. If it does, call waitForCommit() to wait on that process. You must waitForCommit() before committing any more updates to the job, no cap 🚨 While the async process is running, it's totally safe to modify the job, but anything you change after this call won't get committed until the next call 🫶

Parameters

• jobState (bool · /usr/share/doc/python3/html/library/functions.html#bool) — If True, it commits the state of the FileStore's job and file deletes. Otherwise, it only commits file creates and updates 🌸

Return type: None · /usr/share/doc/python3/html/library/constants.html#None

abstractmethod waitForCommit() ⏳ This one blocks while startCommit is running, so it's giving patience. The job's successor calls it to make sure it doesn't start touching the job store until this job is finished 🦋 It might get called even when startCommit was never called on that particular instance, and in that case it just doesn't block. Chill 😌

Return type: bool · /usr/share/doc/python3/html/library/functions.html#bool

Returns: Always returns True ✅

abstractmethod classmethod shutdown(shutdown_info) 🌙 Shuts down the filestore on this node. It's meant to be called on batch system shutdown, so it's the closing ceremony 👑

Parameters

• shutdown_info (Any · /usr/share/doc/python3/html/library/typing.html#typing.Any) — The implementation-specific shutdown info, used for shutting down the file store and removing all its state and all job local temp directories from the node 🧹

Return type: None · /usr/share/doc/python3/html/library/constants.html#None

---

🧚 class toil.fileStores.FileID(fileStoreID, size, executable=False)

A small wrapper around Python's builtin string class, and honestly it's iconic 💖 It represents a file's ID in the file store and has a size attribute, which is the file's size in bytes. This object gets returned by importFile and writeGlobalFile 🎀

Calls into the file store can use bare strings too. If the size isn't in the ID, it gets queried from the job store, so you're covered either way 🫶

Parameters

• fileStoreID (str · /usr/share/doc/python3/html/library/stdtypes.html#str)
• size (int · /usr/share/doc/python3/html/library/functions.html#int)
• executable (bool · /usr/share/doc/python3/html/library/functions.html#bool)
• args (Any · /usr/share/doc/python3/html/library/typing.html#typing.Any)
• kwargs (dict · /usr/share/doc/python3/html/library/stdtypes.html#dict[str · /usr/share/doc/python3/html/library/stdtypes.html#str, Any · /usr/share/doc/python3/html/library/typing.html#typing.Any])

Return type: FileID · #toil.fileStores.FileID

__init__(fileStoreID, size, executable=False) 🌷

Parameters

• fileStoreID (str · /usr/share/doc/python3/html/library/stdtypes.html#str)
• size (int · /usr/share/doc/python3/html/library/functions.html#int)
• executable (bool · /usr/share/doc/python3/html/library/functions.html#bool)

Return type: None

pack() 📦 Packs the FileID into a string so it can be passed through external code. Slay 💅

Return type: str · /usr/share/doc/python3/html/library/stdtypes.html#str

classmethod unpack(packedFileStoreID) 🔓 Unpacks the result of pack() back into a FileID object. It's the undo button, basically 🦋

Parameters

• packedFileStoreID (str · /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: FileID · #toil.fileStores.FileID

---

✨ BATCH SYSTEM API ✨

Okay so the batch system interface is how Toil stays flexible about where jobs run 🌈 It abstracts over different ways of running batches of jobs, like on Slurm clusters, Kubernetes clusters, or a single node. The toil.batchSystems.abstractBatchSystem.AbstractBatchSystem API gets implemented to run jobs using a given job management system, and that's the main character 👑

🍓 Batch System Environment Variables

Environment variables let you pass scheduler-specific parameters along. It's giving customization 💜

SLURM has two environment variables. The first one applies to all jobs, and the second one defines the partition to use for parallel jobs 🌸

export TOIL_SLURM_ARGS="-t 1:00:00 -q fatq"
export TOIL_SLURM_PE='multicore'

TORQUE also has two. One is for everything except the resource requirements, and the other is for the resource requirements themselves, written without the -l prefix 🎀

export TOIL_TORQUE_ARGS="-q fatq"
export TOIL_TORQUE_REQS="walltime=1:00:00"

GridEngine (SGE, UGE) needs one more environment variable to define the parallel environment (see http://www.softpanorama.org/HPC/Grid_engine/parallel_environment.shtml#Important_details) for running multicore jobs 🔥

export TOIL_GRIDENGINE_PE='smp'
export TOIL_GRIDENGINE_ARGS='-q batch.q'

HTCondor lets you include extra parameters in the submit file that gets passed to condor_submit 🧚

export TOIL_HTCONDOR_PARAMS='requirements = TARGET.has_sse4_2 == true; accounting_group = test'

The environment variable is parsed as a semicolon-separated string of parameter = value pairs. Get the semicolons right or it won't parse, bestie 🙈

🦋 Batch System API

💖 class toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

This is an abstract base class that represents the interface the batch system has to provide to Toil. Basically it's the contract ✨

abstractmethod classmethod supportsAutoDeployment() 🚀 Tells you whether the batch system supports auto-deployment of the user script itself. If it does, setUserScript() can be invoked to set the resource object representing the user script 🫶

Note to implementors: If your implementation returns True here, it should also override… and then the sentence just stops there 🙈

Return type: bool · /usr/share/doc/python3/html/library/functions.html#bool

abstractmethod classmethod supportsWorkerCleanup() 🧹 Tells you whether the batch system supports worker cleanup. It indicates whether the batch system invokes BatchSystemSupport.workerCleanup() (#toil.batchSystems.abstractBatchSystem.BatchSystemSupport.workerCleanup) after the last job for a particular workflow invocation finishes 💅

Quick vibe check on terms: "worker" means an entire node, not just a worker process. A worker process can run more than one job sequentially, and more than one concurrent worker process can exist on a worker node for the same workflow. The batch system is said to shut down after the last worker process terminates 👑

Return type: bool · /usr/share/doc/python3/html/library/functions.html#bool

setUserScript(userScript) 📜 Sets the user script for this workflow. This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() (#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem.supportsAutoDeployment) returns True. Otherwise it will raise an exception 💀

Parameters

• userScript (Resource · #toil.resource.Resource) — The resource object representing the user script or module, plus the modules it depends on 🌈

Return type: None · /usr/share/doc/python3/html/library/constants.html#None

set_message_bus(message_bus) 📣 Gives the batch system a chance to connect directly to the message bus. That way it can send informational messages about the jobs it's running to other Toil components. Communication is caring 🫶

Parameters

• message_bus (MessageBus · #toil.bus.MessageBus)

Return type: None · /usr/share/doc/python3/html/library/constants.html#None

abstractmethod issueBatchJob(command, job_desc, job_environment=None) 🚀 Issues a job with the specified command to the batch system and returns a unique job ID number. Iconic behavior, honestly 👑

Parameters

• command (str · /usr/share/doc/python3/html/library/stdtypes.html#str) — the command to execute somewhere to run the Toil worker process
• job_desc (JobDescription · #toil.job.JobDescription) — the JobDescription for the job being run
• job_environment (dict · /usr/share/doc/python3/html/library/stdtypes.html#dict[str · /usr/share/doc/python3/html/library/stdtypes.html#str, str · /usr/share/doc/python3/html/library/stdtypes.html#str] | None · /usr/share/doc/python3/html/library/constants.html#None) — a collection of job-specific environment variables to be set on the worker 🌸

Return type: int · /usr/share/doc/python3/html/library/functions.html#int

Returns: you get back a unique job ID number 🎀 that you can use to reference the newly issued job later. Literally your receipt, bestie 🧾✨

💅 killBatchJobs(jobIDs)

abstractmethod killBatchJobs(jobIDs) is an abstract method, so every batch system has to write its own version 🫶

Okay so this one kills the given job IDs 💀. Once it returns, the killed jobs won't show up in getRunningBatchJobIDs anymore, and they won't come back from getUpdatedBatchJob either 🙈

• Parameters
  • jobIDs (list[int], docs: /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/functions.html#int), the list of IDs of the jobs you want gone 🗑️
• Return type: None (/usr/share/doc/python3/html/library/constants.html#None), nothing to see here 💅

🔮 getIssuedBatchJobIDs()

abstractmethod getIssuedBatchJobIDs() is the "show me everything" method 👀

It gets all the jobs that are currently issued, meaning the ones that are running right now or just waiting for their turn ⏳

• Return type: list[int] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/functions.html#int)
• Returns: a list of issued jobs, given as job ID numbers. They may be running, or they may be waiting to be run. The list is technically a list, but please don't depend on the ordering 🙅‍♀️

🏃‍♀️ getRunningBatchJobIDs()

abstractmethod getRunningBatchJobIDs() is the actually running right now check. Waiting doesn't count here, bestie 🚫⏳

It gives you a map of job ID numbers for jobs that are running at this moment, along with how long each one has been running in seconds ⏱️

• Return type: dict[int, float] (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/functions.html#float)
• Returns: a dictionary where the keys are running job ID numbers and the values are how many seconds each one has been running 🌸

🦋 getUpdatedBatchJob(maxWait)

abstractmethod getUpdatedBatchJob(maxWait) is the one that tells you which jobs have finished (ceased running, either successfully or with an error) 🎉 Each job only gets returned exactly once, so don't expect a second look 🫠

Jobs killed by killBatchJobs won't be reported here. Heads up though: those can make it return None earlier than maxWait 👀

• Parameters
  • maxWait (int, docs: /usr/share/doc/python3/html/library/functions.html#int), the number of seconds it's allowed to block while it waits for a result ⏲️
• Return type: UpdatedBatchJobInfo <#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo> | None (docs: /usr/share/doc/python3/html/library/constants.html#None)
• Returns: if there's a result, you get an UpdatedBatchJobInfo. If there isn't, you get None 🫶
  • wallTime is how many seconds of wall-clock time the job ran for. It's always a strictly positive float. If the batch system doesn't track wall time, it's None instead ⌚

📢 getSchedulingStatusMessage()

getSchedulingStatusMessage() is the gossip button 🗣️ It gets a log message fragment for the user about anything that might be going wrong in the batch system, but only if there's something useful to say.

If there's nothing useful, it returns None 🙃 It can also tell you what resource is the limiting factor when jobs are being scheduled. If the leader thinks the workflow is stuck, this message can be shown to the user so they can figure out why it's stuck 🔍

• Return type: str | None (docs: /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/constants.html#None)
• Returns: a user-directed message about the scheduling state 💬

🛑 shutdown()

abstractmethod shutdown() gets called when a toil invocation wraps up 🎬 It should cleanly terminate all the worker threads, so nothing gets left hanging 🧵

• Return type: None (/usr/share/doc/python3/html/library/constants.html#None)

🌈 setEnv(name, value=None)

setEnv(name, value=None) lets you set an environment variable for the worker process before it launches 🚀

Workers usually inherit the environment of whatever machine they're running on. This method lets you override specific variables in that inherited environment before the worker starts. Note that this is a different mechanism from the one the worker uses internally to set up the environment of a job, so don't mix them up 🙅‍♀️

A call to this method affects all jobs issued after it returns. Note to implementors: that means you'll usually need to copy the variables before enqueuing a job 📋

If you don't give a value, it gets looked up from the current environment 👀

• Parameters
  • name (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
  • value (str | None, docs: /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/constants.html#None)
• Return type: None (/usr/share/doc/python3/html/library/constants.html#None)

🎀 add_options(parser)

classmethod add_options(parser) is for batch systems that come with their own command line options 🛠️ If yours does, it adds them to the parser you give it.

• Parameters
  • parser (ArgumentParser | _ArgumentGroup, docs: /usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser)
• Return type: None (/usr/share/doc/python3/html/library/constants.html#None)

⚙️ setOptions(setOption)

classmethod setOptions(setOption) processes the command line or configuration options that are relevant to this batch system 🧩

• Parameters
  • setOption (OptionSetter <#toil.batchSystems.options.OptionSetter>), a function with the signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None). It returns nothing and updates the run configuration as a side effect ✨
• Return type: None (/usr/share/doc/python3/html/library/constants.html#None)

🧚 getWorkerContexts()

getWorkerContexts() gets a list of picklable context manager objects that wrap the worker's work, in order 🎁

These let the Toil worker do things in-process, like configuring environment variables, hot-deploying user scripts, or cleaning up a node. Without them, you'd need a wrapping "executor" process to handle that 🙌

• Return type: list[ContextManager[Any, bool | None]] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/typing.html#typing.ContextManager, /usr/share/doc/python3/html/library/typing.html#typing.Any, /usr/share/doc/python3/html/library/functions.html#bool, /usr/share/doc/python3/html/library/constants.html#None)

---

✨ JOB.SERVICE API ✨

okay bestie, the Service class is how you spawn databases and servers inside a Toil workflow 💾🦋 it's literally giving main character energy.

💖 class Job.Service

class Job.Service(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, unitName='')

Abstract class used to define the interface to a service. You have to subclass it yourself to actually define your services 💅

It's not executed as a job. It runs inside a ServiceHostJob instead 🧚

🌸 Parameters

• memory (str | int | None)
• cores (str | int | float | None)
• disk (str | int | None)
• accelerators (str | int | Mapping[str, Any] | AcceleratorRequirement | Sequence[str | int | Mapping[str, Any] | AcceleratorRequirement] | None)
• preemptible (str | int | bool | None)
• unitName (str | None)

🎀 __init__

__init__(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, unitName='')

Memory, core and disk requirements are specified exactly the same way as in toil.job.Job.__init__() 🫶

🌸 Parameters

• memory (str | int | None)
• cores (str | int | float | None)
• disk (str | int | None)
• accelerators (str | int | Mapping[str, Any] | AcceleratorRequirement | Sequence[str | int | Mapping[str, Any] | AcceleratorRequirement] | None)
• preemptible (str | int | bool | None)
• unitName (str | None)

Return type: None

🔥 abstractmethod start(job)

Start the service.

🌸 Parameters

• job (ServiceHostJob) — the underlying host job the service is running in. You can use it to register deferred functions, or to get at the fileStore for creating temporary files ✨

Return type: Any

💜 Returns

An object describing how to access the service. It has to be pickleable, because jobs use it to reach the service (see toil.job.Job.addService()) 🦋

🔥 abstractmethod stop(job)

Stops the service. The function can block until it's complete. 🛑

🌸 Parameters

• job (ServiceHostJob) — the same underlying host job as above, again usable for deferred functions or the fileStore 💖

Return type: None

🌈 check()

Checks that the service is still running. 👀

💀 Raises

• exceptions.RuntimeError — if the service failed, the service job gets labeled failed 😭

Return type: bool

💜 Returns

True if the service is still running, otherwise False. If it returns False, the service job gets terminated and counts as a success 🎉

⚠️ Important point: if the service job exits because of a failure, it should raise a RuntimeError, not return False! Returning False means "all good, shut it down," so a failure dressed up as False will be treated as a success 💀

✨ EXCEPTIONS API ✨

okay so this section is all about the Toil-specific exceptions 💅 these are the custom errors Toil throws when something goes sideways, so you can catch them by name. no cap, super useful 🫶

---

🧨 toil.job.JobException

exception toil.job.JobException(message)

this is the general job exception 💖 basically the catch-all for when a job goes wrong.

🛠️ __init__(message)

__init__(message)

• message (str)

Return type: None

---

🌀 toil.job.JobGraphDeadlockException

exception toil.job.JobGraphDeadlockException(string)

bestie, this one gets raised when your workflow has an unresolvable dependency, like a cycle 🔄 it's giving stuck forever energy. go peep toil.job.Job.checkJobGraphForDeadlocks() for more on that.

🛠️ __init__(string)

__init__(string)

---

📁 toil.jobStores.abstractJobStore.ConcurrentFileModificationException

exception toil.jobStores.abstractJobStore.ConcurrentFileModificationException(jobStoreFileID)

this one shows up when multiple processes try to modify the same file at once 😭 two cooks in one kitchen, and the file is paying the price.

• jobStoreFileID (FileID | str)

🛠️ __init__(jobStoreFileID)

__init__(jobStoreFileID)

• jobStoreFileID (FileID | str) — the ID of the file that got modified by multiple workers or processes concurrently

---

🏠 toil.jobStores.abstractJobStore.JobStoreExistsException

exception toil.jobStores.abstractJobStore.JobStoreExistsException(locator, prefix)

this just means the job store you asked for already exists 🙅‍♀️ it's already living its best life there.

• locator (str)
• prefix (str)

🛠️ __init__(locator, prefix)

__init__(locator, prefix)

• locator (str) — the location of the job store
• locator
• prefix (str)

---

👻 toil.jobStores.abstractJobStore.NoSuchFileException

exception toil.jobStores.abstractJobStore.NoSuchFileException(jobStoreFileID, customName=None, *extra)

this one means the file does not exist 👻 we were SURE it was there. the vibes are gone.

• jobStoreFileID (FileID | str)
• customName (str | None)
• extra (Any)

🛠️ __init__(jobStoreFileID, customName=None, *extra)

__init__(jobStoreFileID, customName=None, *extra)

• jobStoreFileID (FileID | str) — the ID of the file that we wrongly assumed was there
• customName (str | None) — optionally, an alternate name for the file that isn't there
• extra (Any) — optional extra info to tack onto the error message
• extra

---

🔍 toil.jobStores.abstractJobStore.NoSuchJobException

exception toil.jobStores.abstractJobStore.NoSuchJobException(jobStoreID)

this means the job you're looking for does not exist 🫠 it ghosted you.

• jobStoreID (FileID | str)

🛠️ __init__(jobStoreID)

__init__(jobStoreID)

• jobStoreID (FileID | str) — the jobStoreID that we mistakenly assumed was real
• jobStoreID

---

🚫 toil.jobStores.abstractJobStore.NoSuchJobStoreException

exception toil.jobStores.abstractJobStore.NoSuchJobStoreException(locator, prefix)

this one means the job store does not exist at all 🏚️ there's nothing at that address, bestie.

• locator (str)
• prefix (str)

🛠️ __init__(locator, prefix)

__init__(locator, prefix)

• locator (str) — the location of the job store
• prefix (str) — the type of job store
• locator
• prefix

✨ RUNNING TESTS ✨

Okay bestie, you run Toil's tests with make targets, like $ make <target> 💅 Which ones actually do stuff depends on which environment variables you've set (more on that in Running Integration Tests) 🧪

• test — invokes all the tests 💖
• integration_test — invokes only the integration tests 🔗
• test_offline — skips building the Docker appliance and only invokes tests with no Docker dependencies 🦋
• integration_test_local — makes integration tests way easier to debug locally 🔍 It runs them serially and doesn't redirect output, so everything shows up on your terminal like it should ✨

Before you run tests for the first time, set up your virtual environment by following the steps in Installing Plugins (see buildfromsource) 🌸

Run all tests, slow ones included:

$ make test

Run only the quick tests (as of Jul 25, 2018, this was ~ 20 minutes) ⏰:

$ export TOIL_TEST_QUICK=True; make test

Run an individual test with:

$ make test tests=src/toil/test/sort/sortTest.py::SortTest::testSort

The default value for tests is "src", which includes all tests in the src/ subdirectory of the project root. Tests that need a particular feature get skipped implicitly 🙈 If you want to explicitly skip tests that depend on a feature that's currently installed, use:

$ make test tests="-m 'not aws' src"

This runs only the tests that don't depend on the aws extra, even if that extra is currently installed. Note the difference between a feature and an extra 🧚 Every extra is a feature, but some features aren't extras, like the gridengine feature. To skip tests involving both the gridengine feature and the aws extra, use the following:

$ make test tests="-m 'not aws and not gridengine' src"

---

🐍 Running Tests with pytest

Honestly, it's often simpler to just use pytest directly instead of the make wrapper 💜 This usually works as expected, but some tests need a little manual prep. To run a specific test with pytest, use the following:

python3 -m pytest src/toil/test/sort/sortTest.py::SortTest::testSort

For more info, check out the pytest documentation at https://docs.pytest.org/en/latest/ 📚

🏎️ Running Integration Tests

These tests usually only run in our CI workflow because of their resource requirements and cost 💸 But you can make them available for local testing:

• Tests that use Docker (like the autoscaling tests and the Docker tests) need an appliance image to be hosted 🐳 First, make sure you've gone through the setup in Using Docker with Quay. Then build and host the appliance image by running the make target push_docker:

$ make push_docker

• Integration tests need to be activated with an environment variable, plus you have to export info relevant to the tests you want to run 🔓 Enable them with:

$ export TOIL_TEST_INTEGRATIVE=True

• Last step: set the environment variables for your keyname and desired zone 🗝️

$ export TOIL_X_KEYNAME=[Your Keyname]
$ export TOIL_X_ZONE=[Desired Zone]

Where X is one of our currently supported cloud providers (GCE, AWS) 🌈

• Check the sections above for guidance on running tests 🫶

🌈 Test Environment Variables

Here's the cast of test environment variables, bestie 👑

• TOIL_TEST_TEMP — an absolute path to a directory where Toil tests write their temporary files 📁 If you don't set it, it defaults to the system's standard temporary directory (see https://docs.python.org/2/library/tempfile.html#tempfile.gettempdir).
• TOIL_TEST_INTEGRATIVE — if True, this lets the integration tests run 🏃‍♀️ It's only valid when running the tests from the source directory via make test or make test_parallel.
• TOIL_AWS_KEYNAME — an AWS keyname (see Preparing your AWS environment, prepareaws) 🔑 It's required to run the AWS tests.
• TOIL_GOOGLE_PROJECTID — a Google Cloud account projectID (see Running in Google Compute Engine (GCE), runninggce) ☁️ It's required to run the Google Cloud tests.
• TOIL_TEST_QUICK — if True, long-running tests get skipped 🏃‍♀️💨

Partial install and failing tests

Heads up: some tests may fail with an ImportError if the required extras aren't installed 😭 Install Toil with all of the extras to prevent those errors.

🐳 Using Docker with Quay

Some of the tests need Docker 🐋 Follow the installation instructions for your system on the Docker website (https://www.docker.com/products/docker) to get started.

When you run make test, you might still get this error:

$ make test
Please set TOIL_DOCKER_REGISTRY, e.g. to quay.io/USER.

To fix it, make an account with Quay (https://quay.io/) and specify it like so, where USER is your Quay username:

$ TOIL_DOCKER_REGISTRY=quay.io/USER make test

Pro move: you can add this variable to your bashrc by running:

$ echo 'export TOIL_DOCKER_REGISTRY=quay.io/USER' >> $HOME/.bashrc

🍓 Running Mesos Tests

If you're running Toil's Mesos tests, be sure to create the virtualenv with --system-site-packages so it includes the Mesos Python bindings 🥹 Verify this by activating the virtualenv and running pip list | grep mesos. On macOS, that may come up empty. To fix it, run the following:

for i in /usr/local/lib/python2.7/site-packages/*mesos*; do ln -snf $i venv/lib/python2.7/site-packages/; done

✨ DEVELOPING WITH DOCKER ✨

okay so if you're building features that rely on the Toil Appliance (that's the docker image toil uses for AWS autoscaling) 🐳, bestie, you should lowkey set up a personal registry on Quay https://quay.io/ or Docker Hub https://hub.docker.com/ 💅. Here's the tea: Toil Appliance images are tagged with the Git commit they're based on, and only commits on our master branch trigger an appliance build on Quay. So the second you make a commit or dirty your working copy, Toil can't automatically find the right Toil Appliance image anymore 😭. If you want to test appliance changes in autoscaling, you need to build and push your own appliance image to a personal Docker registry. The next section walks you through it 🦋.

Making Your Own Toil Docker Image

Note! Toil checks if the docker image specified by TOIL_APPLIANCE_SELF exists before launching, using the docker v2 schema. This should work for any major docker repository, but you can override this check with --forceDockerAppliance if you want 🙈.

Here's a general workflow (similar instructions apply when using Docker Hub) 🌸:

1. Make some changes to the provisioner of your local version of Toil ✨
2. Go to the location where you installed the Toil source code and run

$ make docker

This automatically builds a docker image that's ready to upload to your personal Quay https://quay.io/ account. On Docker Desktop, containerd may have to be enabled https://docs.docker.com/desktop/containerd/#enable-the-containerd-image-store. If you haven't installed the Toil source code yet, check out Installing Plugins #buildfromsource.

3. You'll need Docker installed and need to log into Quay https://docs.quay.io/solution/getting-started.html if you haven't already. Also make sure your Quay account is public 🎀.
4. Set the environment variable TOIL_DOCKER_REGISTRY to your Quay account. If you find yourself doing this a lot, you might want to add

export TOIL_DOCKER_REGISTRY=quay.io/<MY_QUAY_USERNAME>

to your .bashrc or equivalent 💜.

5. Now run

$ make push_docker

which uploads the docker image to your Quay account. Take note of the image's tag for the next step 🏷️.

6. Finally, tell Toil where to pull the Appliance image you just created (it uses the Toil release you have installed by default). To do this, set the environment variable TOIL_APPLIANCE_SELF to the url of your image. For more info, see Environment Variables #envars.
7. Now you can launch your cluster! For more information, see Running a Workflow with Autoscaling #autoscaling 🚀.

Running a Cluster Locally

The Toil Appliance container is also super useful as a test environment, since it can simulate a Toil cluster locally 🧚. An important caveat: autoscaling only works on an EC2 instance and cannot (at this time) run on a local machine 💀.

To spin up a local cluster, start by using this Docker run command to launch a Toil leader container:

docker run \
    --entrypoint=mesos-master \
    --net=host \
    -d \
    --name=leader \
    --volume=/home/jobStoreParentDir:/jobStoreParentDir \
    quay.io/ucsc_cgl/toil:3.6.0 \
    --registry=in_memory \
    --ip=127.0.0.1 \
    --port=5050 \
    --allocation_interval=500ms

A couple notes on this one: the -d flag tells Docker to run in daemon mode, so the container runs in the background 🌙. To check that the container is running, run docker ps to see all containers. If you want to run your own container instead of the official UCSC container, just swap quay.io/ucsc_cgl/toil:3.6.0 for your own container name 🫶.

Also, we're not mounting the job store directory itself, but rather the location where the job store will be written. Due to complications with running Docker on MacOS, I recommend only mounting directories within your home directory 🍓. The next command launches the Toil worker container with similar parameters:

docker run \
    --entrypoint=mesos-slave \
    --net=host \
    -d \
    --name=worker \
    --volume=/home/jobStoreParentDir:/jobStoreParentDir \
    quay.io/ucsc_cgl/toil:3.6.0 \
    --work_dir=/var/lib/mesos \
    --master=127.0.0.1:5050 \
    --ip=127.0.0.1 \
    —-attributes=preemptable:False \
    --resources=cpus:2

Here we're specifying 2 CPUs and a non-preemptable worker. You can easily change either or both of these in a logical way ⭐. To change the number of cores, swap the 2 for whatever number you like. To make the worker preemptable, change preemptable:False to preemptable:True. Also, the same volume is mounted into the worker. This is needed since both the leader and worker read and write to the job store 👑.

Now that your cluster is running, you can run

docker exec -it leader bash

to get a shell in your leader "node" 💅. You can also replace the leader parameter with worker to get shell access on your worker.

Docker-in-Docker issues

If you want to run Docker inside this Docker cluster (dockerized tools, perhaps), you should also mount in the Docker socket via -v /var/run/docker.sock:/var/run/docker.sock. This gives the Docker client inside the Toil Appliance access to the Docker engine on the host 🔥. Client/engine version mismatches have been known to cause issues, so we recommend using Docker version 1.12.3 on the host to match the Docker client installed in the Appliance.

Finally, be careful where you write files inside the Toil Appliance 😬. "Child" Docker containers launched in the Appliance will actually be siblings to the Appliance, since the Docker engine lives on the host. This means the child container can only mount in files from the Appliance if those files are in a directory that was originally mounted into the Appliance from the host. That way, the files are accessible to the sibling container. Note: if Docker can't find the file or directory on the host, it will silently fail and mount in an empty directory.

Enabling FUSE

When running toil-wdl-runner with Singularity, Singularity decompresses images to sandbox directories by default. This can take a while if a workflow has lots of images 🐢. To avoid this, you can give the Docker container access to FUSE at startup. There are 2 main ways to do this. You can either run all the Docker containers in privileged mode:

docker run \
    -d \
    --name=toil_leader \
    --privileged \
    quay.io/ucsc_cgl/toil:6.2.0

Or pass the /dev/fuse device node through into the container:

docker run \
    -d \
    --name=toil_leader \
    --device=/dev/fuse \
    quay.io/ucsc_cgl/toil:6.2.0

toil-wdl-runner will handle the logic from there 🧚‍♀️.

✨ MAINTAINER'S GUIDELINES ✨

In general, as developers and maintainers of the code, we follow these guidelines 🎀:

• We strive to never break the build on master. All development should happen on branches, either in the main Toil repository or in developers' forks.
• Pull requests should be used for any and all changes (except truly trivial ones) 💖.
• Pull requests should be in response to issues. If you find yourself making a pull request without an issue, create the issue first 🙈.

Naming Conventions

• Commit messages should be great https://chris.beams.io/posts/git-commit/#seven-rules. Most importantly, they must:
  • Have a short subject line. If you need more space, drop down two lines and write a body explaining what is changing and why it has to change.
  • Write the subject line as a command: Destroy all humans, not All humans destroyed.
  • Reference the issue being fixed in a Github-parseable format, such as (resolves #1234) at the end of the subject line, or This will fix #1234. somewhere in the body. If no single commit on its own fixes the issue, the cross-reference must appear in the pull request title or body instead 🔥.
• Branches in the main Toil repository must start with issues/, followed by the issue number (or numbers, separated by a dash), followed by a short, lowercase, hyphenated description of the change. (There can be many open pull requests with their branches at any given time, and this convention keeps branches easy to identify 🦋.)

Say there's an issue numbered #123 titled Foo does not work. The branch name would be issues/123-fix-foo, and the commit title would be Fix foo in case of bar (resolves #123) 💅.

Pull Requests

• All pull requests must be reviewed by a person other than the request's author. Review the PR by following the Reviewing Pull Requests #reviewingprs checklist 👑.
• Modified pull requests must be re-reviewed before merging. Note that Github does not enforce this! 😭
• Merge pull requests by following the Merging Pull Requests #mergingprs checklist ✨.
• When merging a pull request, make sure to update the Draft Changelog https://github.com/DataBiosphere/toil/wiki/Draft-Changelog on the Github wiki. We use it to produce the changelog for the next release. The PR template tells you to do this, so don't forget 🌈. New entries go at the bottom.
• Pull requests won't be merged unless CI tests pass. Gitlab tests only run on code in the main Toil repository on some branch, so it's the responsibility of the approving reviewer to make sure pull requests from outside repositories are copied to branches in the main repository. This can be done with (from a Toil clone):

./contrib/admin/test-pr theirusername their-branch issues/123-fix-description-here

You have to repeat this every time the PR submitter updates their PR, after checking that the update is not malicious 🫶.

If there's no issue corresponding to the PR (the branch gets named after it), the reviewer should first create the issue 🙈.

Developers who have push access to the main Toil repository are encouraged to make their pull requests from within the repository, so they can skip this step 🧚.

• 💅 okay bestie, when you merge a PR into master, lowkey always pick "Squash and marge" 🔥 especially when the PR is one single unit of work. If you rewrote the PR from scratch with all the fixes baked in, it would be exactly one commit ✨. That keeps the commit history on master super readable and way easier to debug if something breaks 🫶
• When you squash a PR with multiple authors, please add Co-authored-by <https://github.blog/2018-01-29-commit-together-with-co-authors/> so everyone gets their credit 💖
• See Issue #2816 <https://github.com/DataBiosphere/toil/issues/2816> for more deets 🦋

💅 Publishing a Release 💅

okay so here's how you publish a Toil release, step by step 👑

• Figure out the release version X.Y.Z 🎀 It should follow semantic versioning <https://semver.org/>. If user-workflow-breaking changes are made, X gets incremented, and Y and Z go back to zero 💀 If non-breaking changes are made but new functionality is added, X stays the same as the last release, Y gets incremented, and Z goes back to zero. If only patches are released, X and Y stay the same as the last release and Z gets incremented 🌸
• If it doesn't exist already, make a release branch in the Toil repo named X.Y.x, where x is a literal lower-case "x" 🧚 For patch releases, find the existing branch and make sure it's up to date with the patch commits that are going out. They can be cherry-picked over <https://trunkbaseddevelopment.com/branch-for-release/> from master 🫶
• On the release branch, edit version_template.py in the root of the repository. Find the line that looks like this (slightly different for patch releases):

baseVersion = 'X.Y.0a1'

Make it look like this instead:

baseVersion = 'X.Y.Z'

• Commit your change to the branch ✨
• Tag the current state of the release branch as releases/X.Y.Z 🏷️
• Make the Github release here <https://github.com/DataBiosphere/toil/releases/new>, referencing that tag 🎀 For a non-patch release, fill in the description with the changelog from the wiki page <https://github.com/DataBiosphere/toil/wiki/Draft-Changelog>, which you should clear afterwards 🧹 For a patch release, just describe the patch 💅
• For a non-patch release, set up the main branch so development builds declare themselves as alpha versions of what the next release will probably be 🦋 Edit version_template.py in the root of the repository on the main branch to set baseVersion like this:

baseVersion = 'X.Y+1.0a1'

Make sure to replace X and Y+1 with actual numbers 🔢

🌈 Using Git Hooks 🌈

in the contrib/hooks directory, there are two scripts, mypy-after-commit.py and mypy-before-push.py 🧚 They can be set up as Git hooks so you don't accidentally push commits that would immediately fail type-checking. They're supposed to eliminate the need to run make mypy constantly 🙈 You can install them into your Git working copy like this:

ln -rs ./contrib/hooks/mypy-after-commit.py .git/hooks/post-commit
ln -rs ./contrib/hooks/mypy-before-push.py .git/hooks/pre-push

After you make a commit, the post-commit script starts type-checking it, and if that takes too long it re-launches the process in the background ⏳ When you push, the pre-push script checks whether the commit you're pushing type-checked successfully. If it hasn't been type-checked but is currently checked out, it gets type-checked then ✨ If type-checking fails, the push is aborted 🛑

Type-checking only happens if you're in a Toil development virtual environment. If you aren't, the scripts won't do anything 🫠

To bypass or override the pre-push hook (if it's wrong, or if you need to push something that doesn't typecheck), you can run git push --no-verify 🙅‍♀️ If the scripts get confused about whether a commit actually typechecks, you can clear out the type-checking result cache. It lives in /var/run/user/<your UID>/.mypy_toil_result_cache on Linux and in .mypy_toil_result_cache in the Toil repo on Mac 🗑️

To uninstall the scripts, delete .git/hooks/post-commit and .git/hooks/pre-push 👋

🎀 Adding Retries to a Function 🎀

see toil.lib.retry <https://github.com/DataBiosphere/toil/blob/master/src/toil/lib/retry.py> 💖

retry() can decorate any function, based on the list of errors you want it to retry on 🔁

That list of errors can contain normal Exception objects, and/or RetryCondition objects wrapping Exceptions to include extra conditions 🧠

For example, retrying on one Exception (HTTPError):

from requests import get
from requests.exceptions import HTTPError

@retry(errors=[HTTPError])
def update_my_wallpaper():
    return get('https://www.deviantart.com/')

Or:

from requests import get
from requests.exceptions import HTTPError

@retry(errors=[HTTPError, ValueError])
def update_my_wallpaper():
    return get('https://www.deviantart.com/')

The examples above retry for the default interval on any errors listed in the errors= arg list 🌸

To retry on specifically 500/502/503/504 errors, you can specify an ErrorCondition object instead, like this:

from requests import get
from requests.exceptions import HTTPError

@retry(errors=[
    ErrorCondition(
               error=HTTPError,
               error_codes=[500, 502, 503, 504]
           )])
def update_my_wallpaper():
    return requests.get('https://www.deviantart.com/')

To retry on specifically errors containing the phrase "NotFound":

from requests import get
from requests.exceptions import HTTPError

@retry(errors=[
    ErrorCondition(
        error=HTTPError,
        error_message_must_include="NotFound"
    )])
def update_my_wallpaper():
    return requests.get('https://www.deviantart.com/')

To retry on all HTTPError errors EXCEPT an HTTPError containing the phrase "NotFound":

from requests import get
from requests.exceptions import HTTPError

@retry(errors=[
    HTTPError,
    ErrorCondition(
               error=HTTPError,
               error_message_must_include="NotFound",
               retry_on_this_condition=False
           )])
def update_my_wallpaper():
    return requests.get('https://www.deviantart.com/')

To retry on boto3's specific status errors, an example of the implementation is:

import boto3
from botocore.exceptions import ClientError

@retry(errors=[
    ErrorCondition(
               error=ClientError,
               boto_error_codes=["BucketNotFound"]
           )])
def boto_bucket(bucket_name):
    boto_session = boto3.session.Session()
    s3_resource = boto_session.resource('s3')
    return s3_resource.Bucket(bucket_name)

Any combination of these works too, as long as the codes are matched to the correct exceptions 🧐 A ValueError will not return a 404, for example 🙅‍♀️

The retry function as a decorator makes retrying functions way easier and clearer 🥰 It also encourages smaller, independent functions, instead of lumping a bunch of different things that might need different retry conditions into one function 👑

The ErrorCondition object does a lot of the heavy lifting for you 💪 It boils writing specific retry conditions down to an API that covers all the common use-cases, so you never have to write any new bespoke functions 🫶

Use-cases covered currently:

1. Retrying on a normal error, like a KeyError.
2. Retrying on HTTP error codes (use ErrorCondition).
3. Retrying on boto's specific status errors, like "BucketNotFound" (use ErrorCondition).
4. Retrying when an error message contains a certain phrase (use ErrorCondition).
5. Explicitly NOT retrying on a condition (use ErrorCondition).

If you need new functionality, the best practice in Toil right now is to add it to the ErrorCondition itself, rather than making a brand new custom retry method 🌈✨

✨ PULL REQUEST CHECKLISTS ✨

okay so this section is basically your PR cheat sheet, bestie 💅 it's got checklists for dealing with PRs. for the general PR stuff, go check out Pull Requests (#prs) 🦋

👀 Reviewing Pull Requests 👀

this checklist has to stay in sync with the checklist in the pull request template 🫶

when you're reviewing a PR, do the following:

• Check where it's coming from. Make sure it's coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it's from an external repo, pull it in for CI with:

contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing

• If there's no associated issue, create one at https://github.com/DataBiosphere/toil/issues/new 📝

• Read through the code changes. Make sure it doesn't have:
  • Addition of trailing whitespace. 🙈
  • New variable or member names in camelCase that want to be in snake_case. 🐍
  • New functions without type hints (https://docs.python.org/3/library/typing.html).
  • New functions or classes without informative docstrings.
  • Changes to semantics that aren't reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that aren't reflected in docs/running/cliOptions.rst.
  • New features without tests. 😭
• Leave comments. Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines ✨
• Wrap it up. Finish the review with an overall description of your opinion 👑

---

🎀 Merging Pull Requests 🎀

this checklist has to stay in sync with the checklist in the pull request template too, so no skipping 💖

when you're merging a PR, do the following:

• Check the tests. Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch. 🧪
• Check the review. Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed 🦋
• Squash and merge. Merge with the Github "Squash and merge" feature 🫧
  • If there are multiple authors' commits, add Co-authored-by (see https://github.blog/2018-01-29-commit-together-with-co-authors/) to give credit to all the contributing authors. the people deserve their flowers 🌸
• Copy the changelog entry. Copy its recommended changelog entry to the Draft Changelog (https://github.com/DataBiosphere/toil/wiki/Draft-Changelog) 📋
• Tag it. Append the issue number in parentheses to the changelog entry 🔥

---

🏗️ TOIL ARCHITECTURE 🏗️

okay this diagram is the whole vibe of how Toil is built, so let's look at it together 🌈 it lays out the software architecture of Toil.

[image: Toil's architecture is composed of the leader, the job store, the worker processes, the batch system, the node provisioner, and the stats and logging monitor.]

Figure 1: The basic components of Toil's architecture. 🖼️

these are the components, and here's the tea on each one ☕

• the leader 👑 — the leader is responsible for deciding which jobs should be run. to do this it traverses the job graph. right now it's a single threaded process, but we take aggressive steps to keep it from becoming a bottleneck (see Read-only Leader described below) 💪
• the job-store 🗄️ — handles all the files shared between the components. files in the job-store are the means by which the state of the workflow is maintained. each job is backed by a file in the job store, and atomic updates to this state are used to make sure the workflow can always be resumed upon failure. the job-store can also store all user files, so they can be shared between jobs. it's defined by the AbstractJobStore class (toil.jobStores.abstractJobStore.AbstractJobStore). multiple implementations of this class let Toil support different back-end file stores, e.g.: S3, network file systems, Google file store, etc. ☁️
• workers 🐝 — the workers are temporary processes responsible for running jobs, one at a time per worker. each worker process is invoked with a job argument that it's responsible for running. the worker monitors this job and reports back success or failure to the leader by editing the job's state in the file-store. if the job defines successor jobs, the worker may choose to run them immediately (see Job Chaining below) ⚡
• the batch-system 🚦 — responsible for scheduling the jobs given to it by the leader, running a worker command for each job. it's defined by the AbstractBatchSystem class (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem). Toil uses multiple existing batch systems to schedule jobs, including Apache Mesos, GridEngine, and a multi-process single node implementation that lets workflows run without any of these frameworks. so Toil can therefore fairly easily be made to run a workflow using an existing cluster 🫶
• the node provisioner 🏭 — creates the worker nodes that the batch system schedules workers onto. it's defined by the AbstractProvisioner class (toil.provisioners.abstractProvisioner.AbstractProvisioner).
• the statistics and logging monitor 📊 — monitors the logging and statistics produced by the workers and reports them. it uses the job-store to gather this information 🧐

🧩 Jobs and JobDescriptions 🧩

okay so as noted in Job Basics (#jobbasics), a job is the atomic unit of work in a Toil workflow ✨ workflows extend the Job class (toil.job.Job) to define units of work. these jobs get pickled and stored in the job-store by the leader, and the worker retrieves and un-pickles them when they're scheduled to run.

during scheduling, Toil doesn't work with the actual Job objects. instead, JobDescription objects (toil.job.JobDescription) are used to store all the information that the Toil leader ever needs to know about the job. this includes requirements information, dependency information, the body object to run, the worker command to issue, etc. 💅

internally, the JobDescription object is referenced by its jobStoreID, which is often not human readable. however, the Job and JobDescription objects contain several human-readable names that are super useful for logging and identification 🦋

• jobName 🏷️ — the name of the kind of job this is. this may be used in job store IDs and logging. it's also used to let the cluster scaler learn a model for how long the job will take. defaults to the job class's name if no real user-defined name is available.
  • for a FunctionWrappingJob (toil.job.FunctionWrappingJob), the jobName is replaced by the wrapped function's name.
  • for a CWL workflow, the jobName is the class name of the internal job that's running the CWL workflow, such as "CWLJob".
• unitName 🎀 — the name of this particular instance of this kind of job. if the user sets it, it will appear with the jobName in logging.
  • for a CWL workflow, the unitName is the dotted path from the workflow down to the task being run, including numbers for scatter steps.
• displayName 💖 — a human-readable name to identify this particular job instance. it's used as an identifier of the job class in the stats report. defaults to the job class's name if no real user-defined name is available.
  • for CWL workflows, this includes the jobName and the unitName. main character energy 👑

📊 Statistics and Logging 📊

okay so the stats and logging system is a whole vibe 💅 It's one joint class called StatsAndLogging (toil.statsAndLogging.StatsAndLogging) 🎀 You can instantiate it and run it as a thread on the leader, where it keeps polling the job store for new log files with read_logs() 👀 Those log files are JSON, so they hold structured data 🧚

Structured log messages from user Python code get stored under workers.logs_to_leader. They come from the file store's log_to_leader() method and get logged at the appropriate level 💖 The text the worker captured for all its chained jobs (that's logs.messages) gets logged at debug level in the worker's output 🌸 And if --writeLogs or --writeLogsGzip is provided, the StatsAndLogging thread also saves the received worker logs into per-job files inside the job store using writeLogFiles() 🦋

Ngl, the worker only fills that in when debug logging is on, or when --writeLogsFromAllJobs is set. Otherwise, logs from successful jobs just aren't persisted 💀 Logs from failed jobs are handled differently. They get written to the file store, and the log file is available through toil.job.JobDescription.getLogFileHandle(). The leader thread grabs those and calls back into StatsAndLogging to print them or save them locally, whichever is appropriate ✨

The CWL and WDL interpreters use log_user_stream() to inject their task-level logs into the stats and logging system 🎀 The full text of those logs lands in the JSON stats files, and when the StatsAndLogging thread spots them, it reports and saves them, kind of like how it handles regular Toil job logs 🫶

To pass the stats and the non-failed-job logs around, the job store has a logs mailbox system 📬 write_logs() drops in a string, and read_logs() on the leader hands those strings to a callback. The mailbox tracks new versus old based on whether anyone has read a string yet, so you can read only the new values or every value it's observed 👑 The stats and logging system uses this to move around structured JSON that holds both log data and worker-measured stats. It expects StatsAndLogging to be the only live reader 🔥

---

⚡ Optimizations ⚡

Toil has a bunch of optimizations built for scalability, and here are some of the key ones 💅

👑 Read-only leader

The leader process is a single thread for now 🧵 Most of its job is processing the state of jobs, and each job is stored as a file in the job-store 📁 To keep that one thread from getting overloaded, each worker does as much as it can to manage the state of the job it's running 💪 So, with a couple of minor exceptions, the leader never needs to write or update a job's state in the job-store 🙈

For example, when a job is complete and has no further successors, the responsible worker deletes the job from the job-store to mark it complete ✅ The leader then only has to check whether the file exists when it gets a signal from the batch-system that the job is done 🫶 This off-loading of state management is orthogonal to any future parallelization of the leader 🌈

🔗 Job chaining

The worker helps schedule successor jobs too, so the leader has fewer individual jobs to process 🧚 Right now it's pretty simple. If there is a single next successor job to run, and its resources fit within the resources of the current job and closely match them, then that job runs immediately on the worker without going back to the leader ⚡

More extensions of this strategy are possible. For lots of workflows that define a series of serial successors (e.g. map sequencing reads, post-process mapped reads, etc.), this pattern is super effective at reducing leader workload 😤

🫧 Preemptable node support

Running at large scale means dealing with intermittent node failures 🥹 So Toil is designed to always be resumable, as long as the job-store doesn't become corrupt 🛟 That robustness is what lets Toil run on preemptible nodes, which are only available when others aren't willing to pay more to use them 💸

Designing workflows that divide into many short individual jobs that can use preemptable nodes lets them be scheduled and executed efficiently 🦋

🗄️ Caching

Running bioinformatic pipelines often means passing large datasets between jobs 🧬 Toil caches the results from jobs, so child jobs running on the same node can directly use the same file objects 🎀 That eliminates the need for an intermediary transfer to the job store 🏃‍♀️ Caching also takes pressure off local disks, because multiple jobs can share a single file 💿 The drop in I/O lets pipelines run faster, and by sharing files, users can run more jobs in parallel since overall disk requirements go down 🚀

To demonstrate how efficient caching is, we ran an experimental internal pipeline on 3 samples from the TCGA Lung Squamous Carcinoma (LUSC) dataset 🔬 The pipeline takes the tumor and normal exome fastqs, plus the tumor rna fastq and input, and predicts MHC presented neoepitopes in the patient that are potential targets for T-cell based immunotherapies 💖 It was run on each sample individually on c3.8xlarge machines on AWS (60GB RAM, 600GB SSD storage, 32 cores) 🖥️ The pipeline aligns the data to hg19-based references, predicts MHC haplotypes using PHLAT, calls mutations using 2 callers (MuTect and RADIA), and annotates them using SnpEff. Then it predicts MHC:peptide binding using the IEDB suite of tools, before running an in-house rank boosting algorithm on the final calls 🧪

To optimize time, the pipeline is written so mutations are called on a per-chromosome basis from the whole-exome bams and merged into a complete vcf 🧩 Running mutect in parallel on whole exome bams would make each mutect job download the complete Tumor and Normal Bams to its working directory. That quickly fills the disk and limits how parallel the jobs can be 😭 The workflow ran in Toil, with and without caching, and Figure 2 shows it finishes faster with caching while using less disk on average than the uncached run 📉 We believe the benefits of caching from file transfers will be much higher on magnetic disk-based storage systems than on the SSD systems we tested 💜

[image: Graph outlining the efficiency gain from caching.]

Figure 2: Efficiency gain from caching. The lower half of each plot describes the disk used by the pipeline, recorded every 10 minutes over the duration of the pipeline, and the upper half shows the corresponding stage of the pipeline being processed. Since jobs requesting the same file shared the same inode, the effective load on the disk is considerably lower than in the uncached case, where every job downloads a personal copy of every file it needs 🫧 In all cases, the uncached run uses almost 300-400GB more than the cached run in the resource heavy mutation calling step. We also see a benefit in wall time for each stage, since we eliminate the time taken for file transfers ⏱️

---

📜 Toil support for Common Workflow Language 📜

The CWL document and input document get loaded using the 'cwltool.load_tool' module 📜 That does normalization and URI expansion (so relative file references become absolute file URIs 🔗), validates the document against the CWL schema ✅, initializes Python objects for the major document elements (command line tools, workflows, workflow steps) 🧱, and performs static type checking that sources and sinks have compatible types 🔍

Input files referenced by the CWL document and input document get imported into the Toil file store 📥 CWL documents can use any URI scheme the Toil file store supports, including local files and object storage ☁️

The location field of File references gets updated to reflect the import token returned by the Toil file store 🎟️

For directory inputs, the directory listing is stored in a Directory object 📂 Each individual file gets imported into the Toil file store too 💖

An initial workflow Job is created from the toplevel CWL document 🌱 Then control passes to the Toil engine, which schedules that initial workflow job to run 🚦

When the toplevel workflow job runs, it traverses the CWL workflow and creates a toil job for each step 👣 The dependency graph is expressed by making downstream jobs children of upstream jobs, and initializing the child jobs with an input object containing the promises of output from upstream jobs 🪄

Toil jobs have a single output, but CWL lets steps have multiple output parameters that can feed into multiple other steps 🤯 So the input to a CWLJob is expressed as an "indirect dictionary" 📖 This is a dictionary of input parameters, where each entry value is a tuple of a promise and a promise key. When the job runs, the indirect dictionary is turned into a concrete input object. Each promise gets resolved into its actual value (which is always a dict), and then the promise key is used to look up the actual value for the input parameter ✨

If a workflow step specifies a scatter, a scatter job is created and connected into the workflow graph as described above 🧩 When the scatter step runs, it creates child jobs for each parameterization of the scatter. A gather job is added as a follow-on to gather the outputs into arrays 🧺

When running a command line tool, it first creates output and temporary directories under the Toil local temp dir 🗂️ It then runs the command line tool using single_job_executor from CWLTool, providing a Toil-specific constructor for filesystem access and overriding the default PathMapper to use ToilPathMapper 🏃‍♀️

The ToilPathMapper keeps track of a file's symbolic identifier (the Toil FileID), its local path on the host (the value returned by readGlobalFile), and the location of the file inside the Docker container 🐳 it's giving organized 💜

okay so once single_job_executor from CWLTool finishes doing its thing 💅, Toil gets the output object and the status back. If the underlying job failed, Toil raises an exception 🙈 no sneaking past that. Then every file in the output object gets tossed into the file store with writeGlobalFile 📦, and the 'location' field of each File reference gets updated to match the token the Toil file store hands back ✨.

When the whole workflow wraps up 🎀, it returns an indirect dictionary that links to the outputs of the job steps feeding into the final output. That's the value toil.start() or toil.restart() gives you, and you resolve it to get the final output object 🦋. The files in that object get exported from the file store to outdir on the host file system, and their 'location' fields get updated to point at the final exported spot 💖.

---

✨ MINIMUM AWS IAM PERMISSIONS ✨

Toil needs at least these permissions in an IAM role to run things on a cluster 🔐. They're added by default when you launch a cluster, BUT if you make your own custom IAM role and pass it in with --awsEc2ProfileArn when launching a cluster <#launchawsclusterdetails>, double-check they're all there 🫶.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:*",
                "s3:*",
                "iam:PassRole"
            ],
            "Resource": "*"
        }
    ]
}

---

✨ AUTO-DEPLOYMENT ✨

If you want to run a Toil Python workflow in a distributed environment 🌸, on multiple worker machines, either in the cloud or on a bare-metal cluster, the Python code has to be made available to those other machines. If the workflow's main module imports other modules, those need to be on the workers too. Toil can automatically do that for you, with a little help from you 🫶. We call this feature auto-deployment of a workflow.

Let's look at the different auto-deploying scenarios first. Some of them, as we'll see shortly, can't be auto-deployed 💀. Last but not least, we'll deal with declaring Toil as a dependency (see #depending-on-toil) for a workflow that's packaged as a setuptools distribution 📦.

Toil can be easily deployed to a remote host 🚀. First, assuming you followed our Preparing your AWS environment section (#prepareaws) to install Toil and use it to create a remote leader node on (in this example) AWS, you can log into it using the Ssh-Cluster Command (#sshcluster). Once you're on the remote host, create and activate a virtualenv (and make sure to use the --system-site-packages option!!) 🧚:

$ virtualenv --system-site-packages venv
$ . venv/bin/activate

Note the --system-site-packages option, which makes sure globally installed packages are accessible inside the virtualenv 🌈. Do not (re)install Toil after this! The --system-site-packages option has already brought over Toil and its dependencies from your local installation of Toil for you 😭.

From here, you can install a project and its dependencies:

$ tree
.
├── util
│   ├── __init__.py
│   └── sort
│       ├── __init__.py
│       └── quick.py
└── workflow
    ├── __init__.py
    └── main.py

3 directories, 5 files
$ pip install matplotlib
$ cp -R workflow util venv/lib/python3.9/site-packages

Ideally, your project would have a setup.py file (see setuptools http://setuptools.readthedocs.io/en/latest/index.html) which streamlines the installation process ✨:

$ tree
.
├── util
│   ├── __init__.py
│   └── sort
│       ├── __init__.py
│       └── quick.py
├── workflow
│   ├── __init__.py
│   └── main.py
└── setup.py

3 directories, 6 files
$ pip install .

Or, if your project has been published to PyPI:

$ pip install my-project

In each case, we've created a virtualenv with the --system-site-packages flag in the venv subdirectory, then installed the matplotlib distribution from PyPI along with the two packages our project consists of. (Again, both Python and Toil are assumed to be present on the leader and all worker nodes 💜.)

We can now run our workflow 🎉:

$ python3 main.py --batchSystem=kubernetes …

Important: If the workflow's external dependencies contain native code (i.e. are not pure Python), then they must be manually installed on each worker.

Warning: Neither python3 setup.py develop nor pip install -e . can be used in this process, as instead of copying the source files, they create .egg-link files that Toil can't auto-deploy. Similarly, python3 setup.py install doesn't work either, as it installs the project as a Python .egg, which is also not currently supported by Toil (though it could be https://github.com/BD2KGenomics/toil/issues/1367 in the future). Also note that using the --single-version-externally-managed flag with setup.py will prevent the installation of your package as an .egg. It will also disable the automatic installation of your project's dependencies.

✨ Auto Deployment with Sibling Python Files ✨

This scenario applies if a Python workflow imports files that are its siblings 👯:

$ cd my_project
$ ls
userScript.py utilities.py
$ ./userScript.py --batchSystem=kubernetes …

Here userScript.py imports additional functionality from utilities.py. Toil detects that userScript.py has sibling Python files and copies them to the workers, right alongside the main Python file. Heads up 👀: sibling Python files get auto-deployed regardless of whether they're actually imported by the workflow. All .py files living in the same directory as the main workflow Python file will be auto-deployed automatically.

This structure is a super suitable way to organize the source code of reasonably complicated workflows 🍓.

✨ Auto-Deploying a Package Hierarchy ✨

Recall that in Python, a package (https://docs.python.org/2/tutorial/modules.html#packages) is a directory containing one or more .py files, one of which must be called __init__.py, and optionally other packages. For more involved workflows with a significant amount of code, this is the recommended way to organize your source code 🧠. Because we use a package hierarchy, the main workflow file is actually a Python module. It's merely one of the modules in the package hierarchy. We need to tell Toil that we want to use a package hierarchy by invoking Python's -m option. This lets Toil identify the entire set of modules belonging to the workflow and copy all of them to each worker. Note that while using the -m option is optional in the scenarios above, it is mandatory in this one 🚨.

The following shell session illustrates this 👇:

$ cd my_project
$ tree
.
├── utils
│   ├── __init__.py
│   └── sort
│       ├── __init__.py
│       └── quick.py
└── workflow
    ├── __init__.py
    └── main.py

3 directories, 5 files
$ python3 -m workflow.main --batchSystem=kubernetes …

Here the workflow entry point module main.py doesn't reside in the current directory. It's part of a package called util, in a subdirectory of the current directory 🗂️. Additional functionality is in a separate module called util.sort.quick, which corresponds to util/sort/quick.py. Because we invoke the workflow via python3 -m workflow.main, Toil can determine the root directory of the hierarchy (my_project in this case) and copy all Python modules underneath it to each worker. The -m option is documented here (https://docs.python.org/2/using/cmdline.html#cmdoption-m) 📖.

When -m is passed, Python adds the current working directory to sys.path, the list of root directories considered when resolving a module name like workflow.main. Without that added convenience, we'd have to run the workflow as PYTHONPATH="$PWD" python3 -m workflow.main. This also means Toil can detect the root directory of the invoked module's package hierarchy even if it isn't the current working directory. In other words, we could do this 🦋:

$ cd my_project
$ export PYTHONPATH="$PWD"
$ cd /some/other/dir
$ python3 -m workflow.main --batchSystem=kubernetes …

Also note that the root directory itself must not be a package, i.e. it must not contain an __init__.py 🙅.

✨ Relying on Shared Filesystems ✨

Bare-metal clusters typically mount a shared file system like NFS on each node. If every node has that file system mounted at the same path, you can place your project on that shared filesystem and run your Python workflow from there. Additionally, you can clone the Toil source tree into a directory on that shared file system, and you won't even need to install Toil on every worker 🙌. Be sure to add both your project directory and the Toil clone to PYTHONPATH. Toil replicates PYTHONPATH from the leader to every worker 💅.

Using a shared filesystem: Toil currently only supports a tempdir set to a local, non-shared directory.

✨ Toil Appliance ✨

The term Toil Appliance refers to the Ubuntu-based Docker image that Toil uses for the machines in clusters managed by Toil, and for executing jobs on Kubernetes. It's easily deployed, only needs Docker, and allows a consistent environment on all Toil clusters 🐳. To specify a different image, see the Toil Environment Variables section (#envars). For more information on the Toil Appliance, see the Running in AWS section (#runningaws) ☁️.

🌸 ENVIRONMENT VARIABLES 🌸

okay bestie, there are a bunch of environment variables that change how Toil runs, so let's go through them 💅✨

• TOIL_CHECK_ENV — 🦋 this is a flag that decides whether Toil looks back at the Python virtual environment it's installed in when it builds commands that might run on other hosts. If it's set to True and Toil is installed in the current virtual environment, Toil uses absolute paths to its own executables. The catch is that the virtual environment has to live at the same path on every node 👀. If it's not set that way, internal commands like _toil_worker get resolved using the PATH on whichever node they run on. This is super useful in a shared HPC environment, where people might have their own Toil installs in virtual environments 🫶
• TOIL_WORKDIR — 🎀 this is an absolute path to the directory where Toil writes its temporary files. The directory has to exist on each worker node, and every worker can have a different value 🌈 The --workDir command line option beats this one. When you're using the Toil docker container (like on Kubernetes), it defaults to /var/lib/toil. With Toil autoscaling on Mesos, it's somewhere inside the Mesos sandbox. In every other case, it falls back to the system's standard temporary directory, which you can read about at https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir 💜
• TOIL_WORKDIR_OVERRIDE — 👑 same deal as TOIL_WORKDIR, an absolute path where Toil writes its temporary files, but this one is the boss. It overrides both TOIL_WORKDIR and the --workDir command line option 💖
• TOIL_COORDINATION_DIR — 🧚 an absolute path to the directory where Toil writes its lock files. It has to exist on each worker node, and it can be set to something different on each worker. The --coordinationDir command line option overrides this one 🌸
• TOIL_COORDINATION_DIR_OVERRIDE — 🔥 the absolute path for lock files, same as above, but it overrides both TOIL_COORDINATION_DIR and the --coordinationDir command line option. Main character energy 👑
• TOIL_BATCH_LOGS_DIR — 📝 a directory where batch system logs get saved, somewhere the leader can get to them 🫶 The --batchLogsDir option overrides this. Heads up: it only works for grid engine batch systems, which are gridengine, htcondor, torque, slurm, and lsf 🙈
• TOIL_HISTORY — 🍓 decides whether the Toil history database is turned on. It defaults to true ✨
• TOIL_JOB_HISTORY — 🦋 decides whether per-job history gets recorded in the Toil history database. It defaults to false, so you have to opt in 💅
• TOIL_KUBERNETES_HOST_PATH — 🌈 a path on the Kubernetes hosts that gets mounted as the Toil work directory inside the workers, which lets everyone share a cache 🤝 If the path doesn't exist yet, it gets created for you 🎀
• TOIL_KUBERNETES_OWNER — 💜 a name prefix that makes your Kubernetes jobs easy to spot in a crowd 👀 If you don't set it, Toil uses your current user name 🫶
• TOIL_KUBERNETES_SERVICE_ACCOUNT — 🔐 the service account name that gets applied when Toil creates Kubernetes pods 💖

Okay so the list keeps going and it's giving even more env vars 💅✨ let's gooo bestie 🫶

• TOIL_KUBERNETES_POD_TIMEOUT — how many seconds Toil waits for a scheduled Kubernetes pod to actually start running ⏰🌸
• TOIL_KUBERNETES_POD_SECURITY_CONTEXT — point this at a YAML file that defines a pod security context, and Toil slaps it on every pod 🔐💜
• TOIL_KUBERNETES_SECURITY_CONTEXT — same vibe, but this YAML file is a security context for all the containers instead 📦🦋
• KUBE_WATCH_ENABLED — a boolean that lets you use the Kubernetes watch stream feature instead of polling for running jobs. It's lowkey so much more efficient 👀🧚 The default is False, so you have to opt in.
• TOIL_APPLIANCE_SELF — the fully qualified reference for the Toil Appliance you want to use, written as REPO/IMAGE:TAG. Valid examples are quay.io/ucsc_cgl/toil:3.6.0 and cket/toil:3.5.0. Also, since Docker defaults to Dockerhub, only quay.io repos need to include their registry 🐳🍓
• TOIL_DOCKER_REGISTRY — the URL of the registry where your Toil Appliance image lives. Docker uses Dockerhub by default, but quay.io is also super popular, and you can get it by setting this to quay.io 🌈
• TOIL_DOCKER_NAME — the name of the Toil Appliance image you want. It's usually just toil, but this option exists so you can override it, since the image can be built with totally arbitrary names 🎀
• TOIL_AWS_SECRET_NAME — for the Kubernetes batch system, this is the name of a Kubernetes secret holding a credentials file that grants access to AWS resources. It gets mounted as ~/.aws inside the Kubernetes-managed Toil containers. With the right credentials (access to S3 and SimpleDB), this lets the AWSJobStore work with the Kubernetes batch system 🔑☁️
• TOIL_AWS_ZONE — the zone to use when you're on AWS. It also determines the region, and it overrides TOIL_AWS_REGION 🗺️👑
• TOIL_AWS_REGION — the region to use when you're on AWS 🌎
• TOIL_AWS_AMI — the ID of the AMI used for node provisioning. If you're unsure, just don't set this one 🙈
• TOIL_AWS_NODE_DEBUG — decides whether nodes that failed health checks get kept around. If it's set to True, nodes that fail EC2 health checks won't be terminated right away, so you can poke at them and figure out what went wrong 🔍💅

Heads up: if any EC2 nodes get left behind this way, the security group gets left behind too. That's by necessity, since it can't be deleted until all its associated nodes are terminated. 🫠

• TOIL_AWS_BATCH_QUEUE — the name or ARN of an AWS Batch Queue, for use with the AWS Batch batch system 📬✨
• TOIL_AWS_BATCH_JOB_ROLE_ARN — the ARN of an IAM role to run AWS Batch jobs as, for use with the AWS Batch batch system 🪪🌸

Warning, bestie: if the jobs aren't run with an IAM role, or on machines that can't access S3 and SimpleDB, the AWS job store won't be usable at all. 💀

• TOIL_GOOGLE_PROJECTID — 💖 the Google project ID that gets used when making Google job store names for tests or CWL workflows.
• TOIL_SLURM_ALLOCATE_MEM — ✨ decides whether Slurm gets told to allocate memory with --mem. It's True by default, so it's on unless you turn it off.
• TOIL_SLURM_ARGS — 🎀 extra arguments for sbatch when you're using the slurm batch system. ngl, don't put CPU or memory specs in here. Define resource requirements for the job instead. There's no default value. If neither --export nor --export-file shows up in your argument list, --export=ALL gets added for you 🫶
• TOIL_SLURM_PARTITION — 🦋 the partition your Slurm jobs get sent to.
• TOIL_SLURM_GPU_PARTITION — 🔥 the partition your Slurm jobs go to if they ask for GPUs.
• TOIL_SLURM_PE — 🌸 the name of the slurm partition to use for parallel jobs. This is lowkey a lifesaver for clusters that don't have a partition taking both single-core and multi-core jobs. There's no default value.
• TOIL_SLURM_TIME — ⏰ the Slurm job time limit, in [DD-]HH:MM:SS format. For example, 2-07:15:30 means 2 days, 7 hours, 15 minutes and 30 seconds, and 4:00:00 means 4 hours.
• TOIL_GRIDENGINE_ARGS — 🧚 extra arguments for qsub on the gridengine batch system. Same rule as before: no CPU or memory specs here, define resource requirements for the job instead. There's no default value.
• TOIL_GRIDENGINE_PE — 💜 parallel environment arguments for qsub and the gridengine batch system. There's no default value.
• TOIL_TORQUE_ARGS — 👑 arguments for qsub on the Torque batch system. Do not pass CPU or memory specs here, because that's giving chaos 💀 Instead, define extra job parameters like the queue. Example: -q medium. Use TOIL_TORQUE_REQS to pass extra values for the -l resource requirements parameter. There's no default value.
• TOIL_TORQUE_REQS — 🌈 arguments for the resource requirements on the Torque batch system. Again, no CPU or memory specs. Instead, define extra resource requirements as a string that goes after the -l argument to qsub. Example: walltime=2:00:00,file=50gb. There's no default value.
• TOIL_LSF_ARGS — 🍓 additional arguments for LSF's bsub command. Instead, define extra job parameters like the queue. Example: -q medium. There's no default value.
• TOIL_HTCONDOR_PARAMS — 🦋 additional parameters to include in the HTCondor submit file that gets passed to condor_submit. Do not pass CPU or memory specs here. Instead, define any extra parameters HTCondor might need. This variable is parsed as a semicolon-separated string of parameter = value pairs. Example: requirements = TARGET.has_sse4_2 == true;

Like this: accounting_group = test. There is no default value for this variable 🫶

• TOIL_CUSTOM_DOCKER_INIT_COMMAND — okay so this is your chance to run any custom bash command inside the Toil docker container before the Toil services start 🐳. It's perfect for setup on the worker and/or primary nodes, like private docker authentication 🔐. For AWS ECR, the example is pip install awscli && eval $(aws ecr get-login --no-include-email --region us-east-1). Slay 💅
• TOIL_CUSTOM_INIT_COMMAND — any custom bash command you want to run before the Toil appliance starts ✨. Same vibe as the one above: custom setup on the worker and/or primary nodes, like private docker authentication for the Toil appliance itself (aka whatever you pulled from TOIL_APPLIANCE_SELF) 🎀
• TOIL_S3_HOST — the IP address or hostname you want to use when connecting to S3 🦋. Example: TOIL_S3_HOST=127.0.0.1
• TOIL_S3_PORT — a port number for connecting to S3 🌸. Example: TOIL_S3_PORT=9001
• TOIL_S3_USE_SSL — you can turn SSL on or off for S3 connections 🔒. It's on by default (True), so only touch this if you need to. Example: TOIL_S3_USE_SSL=False
• TOIL_FTP_USER — the FTP username that overrides every FTP login 👑. Example: TOIL_FTP_USER=ftp_user
• TOIL_FTP_PASSWORD — the FTP password that overrides every FTP login too 🙈. Example: TOIL_FTP_PASSWORD=ftp_password
• TOIL_WES_BROKER_URL — an optional broker URL so the WES server and the Celery task queue can talk to each other 🧚. If you don't set it, the default is amqp://guest:guest@localhost:5672// 💜
• TOIL_WES_JOB_STORE_TYPE — the default job store type for workflows run through the WES server 🍓. It can be file, aws, or google.
• TOIL_OWNER_TAG — this tags your cloud resources with a tag that reads "Owner: $TOIL_OWNER_TAG" 🏷️. Real talk: UCSC uses this internally to keep a bot from terminating untagged resources 🤖💀
• TOIL_AWS_PROFILE — the name of the AWS profile you want Toil to run with 🌈
• TOIL_AWS_TAGS — tags your cloud resources with any arbitrary tags you give in JSON format ⭐. When you use launch cluster, the CLI options win and overwrite these. For info on valid AWS tags, check out AWS Tags at https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html 📚
• TOIL_TRS_ROOT — an override for the Tool Registry Service endpoint or Dockstore deployment that Toil talks to for workflow lookups and metrics publication 🔍. This lets you point those somewhere else, like https://qa.dockstore.org, for debugging 🐛
• TOIL_DOCKSTORE_TOKEN — when publishing workflow metrics, use this Dockstore authentication token instead of the default public Toil one 🔑
• SINGULARITY_DOCKER_HUB_MIRROR — an http or https URL for the Singularity wrapper inside the Toil Docker container 🪞. It acts as a mirror for Docker Hub, so it's giving main character energy 👑✨
• OMP_NUM_THREADS — bestie, this sets how many cores OpenMP apps get inside the workers 💪 If you don't set it, Toil just uses the number of job threads 🧵✨
• GUNICORN_CMD_ARGS — lets you add extra Gunicorn configurations for the Toil WES server 🎀 The settings docs live at https://docs.gunicorn.org/en/stable/settings.html#settings 📚

---

✨ API REFERENCE ✨

This page contains auto-generated API reference documentation [1] 🤓💖

🧚 toil

Submodules

toil.batchSystems

Submodules

toil.batchSystems.abstractBatchSystem

Attributes

• logger 🪵
• EXIT_STATUS_UNAVAILABLE_VALUE 🚫

Exceptions

• InsufficientSystemResources — the common base class for all non-exit exceptions 💀
• AcquisitionTimeoutException — raised when a resource request times out ⏰

Classes

• BatchJobExitReason — an enum where the members are also (and must be) ints 🔢
• UpdatedBatchJobInfo 📝
• WorkerCleanupInfo 🧹
• AbstractBatchSystem — an abstract base class that represents the interface the batch system has to provide to Toil 🧱
• BatchSystemSupport — a partial implementation of AbstractBatchSystem, with support methods 🛠️
• NodeInfo — the coresUsed attribute is a floating point value between 0 (all cores idle) and 1 (all cores 👀
• AbstractScalableBatchSystem — a batch system that supports a variable number of worker nodes 📈
• ResourcePool — represents an integral amount of a resource, such as memory bytes 🧠
• ResourceSet — represents a collection of distinct resources, such as accelerators ⚡

Module Contents

• toil.batchSystems.abstractBatchSystem.logger 🪵
• toil.batchSystems.abstractBatchSystem.EXIT_STATUS_UNAVAILABLE_VALUE = 255 🚫

class toil.batchSystems.abstractBatchSystem.BatchJobExitReason

Bases: enum.IntEnum

Enum where members are also (and must be) ints. This is giving very strict vibes 😤

• FINISHED = 1 — Successfully finished. ✅
• FAILED = 2 — Job finished, but failed. 💔
• LOST = 3 — Preemptable failure (job's executing host went away). 👻
• KILLED = 4 — Job killed before finishing. 🔪
• ERROR = 5 — Internal error. 🚨
• MEMLIMIT = 6 — Job hit batch system imposed memory limit. 🧠💥
• MISSING = 7 — Job disappeared from the scheduler without actually stopping, so Toil killed it. 🫥
• MAXJOBDURATION = 8 — Job ran longer than --maxJobDuration, so Toil killed it. ⌛
• PARTITION = 9 — Job was not able to talk to the leader via the job store, so Toil declared it failed. 📵

classmethod to_string(value)

Convert to human-readable string. Given an int that may or may not be equal to a value from the enum, it produces the string value of its matching enum entry, or a stringified int 🪄

Parameters: value (int)

Return type: str

class toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo

• jobID: int — the Toil batch system ID of the job 🆔
• exitStatus: int = 255 — the exit status (integer value) of the job. 0 means successful 🎉 EXIT_STATUS_UNAVAILABLE_VALUE is used when the exit status isn't available (e.g. the job is lost, or it died but the actual exit code was never reported) 🫠
• exitReason: BatchJobExitReason | None = None 🚪
• wallTime: float | int | None = None ⏱️
• backing_id: str | None = None — the identifier for the job in the backing scheduler, if it's available 🔖

class toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo

Bases: NamedTuple

• work_dir: str | None — the work directory path (where the cache would go) if the user specified one 📁
• coordination_dir: str | None — the coordination directory path (where lock files would go) if the user specified one 🔒
• workflow_id: str — used to identify files specific to this workflow 🪪
• clean_work_dir: str — when to clean up the work and coordination directories for a job ('always', 'onSuccess', 'onError', 'never') 🧽

class toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

Bases: abc.ABC

An abstract base class to represent the interface the batch system must provide to Toil. It's the blueprint, bestie 📐

classmethod supportsAutoDeployment()

Abstractmethod 🧩

Return type: bool

Whether this batch system supports auto-deployment of the user script itself 🚀

If it does, the setUserScript() can be invoked to set the resource object representing the user script. 💅 Note to implementors: if your implementation returns True here, it should also override classmethod supportsWorkerCleanup() 🫶

🦋 classmethod supportsWorkerCleanup()

Abstractmethod 💀

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

Okay so this one answers whether the batch system supports worker cleanup. ✨ It tells you whether the batch system calls BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. 🧹 Quick vibe check: the word worker means an entire node here, not just one worker process. 🖥️ A worker process can run more than one job, one after another, and several concurrent worker processes can exist on the same worker node for the same workflow. 🫠 The batch system is said to shut down after the last worker process terminates. 👑

🦋 abstractmethod setUserScript(userScript)

Set the user script for this workflow. 🎀

This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True. Otherwise it will raise an exception. 🚨

Parameters:

• userScript (toil.resource.Resource) — the resource object representing the user script or module, plus all the modules it depends on. 🧚

Return type: None 🍓

🦋 set_message_bus(message_bus)

This gives the batch system a chance to connect directly to the message bus. 📣 That way it can send informational messages about the jobs it's running to other Toil components. 💜

Parameters:

• message_bus (toil.bus.MessageBus)

Return type: None 🌸

🦋 abstractmethod issueBatchJob(command, job_desc, job_environment=None)

This one issues a job with the specified command to the batch system and returns a unique job ID number. 🔥 It's giving "send it and get a receipt" energy. 🧾

Parameters:

• command (str) — the command to execute somewhere in order to run the Toil worker process. 🏃‍♀️
• job_desc (toil.job.JobDescription) — the JobDescription for the job being run. 🎀
• job_environment (Optional[dict[str, str]]) — a collection of job-specific environment variables to be set on the worker. 🌈

Returns: a unique job ID number that can be used to reference the newly issued job. 🆔

Return type: int (/usr/share/doc/python3/html/library/functions.html#int) ✨

🦋 abstractmethod killBatchJobs(jobIDs)

This kills the given job IDs. 💀 After it returns, the killed jobs won't show up in the results of getRunningBatchJobIDs, and they won't be returned from getUpdatedBatchJob either. 🙈 Gone, reduced to nothing, iykyk.

Parameters:

• jobIDs (list[int]) — list of IDs of the jobs to kill. ⭐

Return type: None 🫶

🦋 abstractmethod getIssuedBatchJobIDs()

This gets all the currently issued jobs. 📋

Returns: a list of jobs (as job ID numbers) that are currently issued. They may be running, or they may be waiting to be run. 🧃 Even though the result is a list, please don't depend on the ordering. 🙅‍♀️

Return type: list[int] 🌷

🦋 abstractmethod getRunningBatchJobIDs()

This gets a map of jobs, by job ID number, that are currently running (not just waiting), along with how long each one has been running in seconds. ⏱️ The main character of this method is the timer. ⏳

Returns: a dictionary with the currently running job ID numbers as keys, and how many seconds each has been running as the values. 📖

Return type: dict[int, float] 🌙

🦋 abstractmethod getUpdatedBatchJob(maxWait)

This returns info about a job that has updated its status. 🎉 That means it stopped running, either successfully or with an error. 💔 Each job gets returned exactly once, so no double-dipping. 🙅‍♀️

Heads up, bestie: this does not return info for jobs killed by killBatchJobs. 😭 Those can cause None to be returned earlier than maxWait, though, so don't panic when that happens. 🫠

Parameters:

• maxWait (int) — the number of seconds to block while waiting for a result. ⏰

Returns: if a result is available, it returns UpdatedBatchJobInfo. Otherwise it returns None. 🎀 The wallTime is the number of seconds (a strictly positive float) of wall-clock time the job ran for. If this batch system doesn't support tracking wall time, it's None instead. 🕰️

Return type: Optional[UpdatedBatchJobInfo] 🦋

🦋 getSchedulingStatusMessage()

This gets a log message fragment for the user about anything that might be going wrong in the batch system, if there's one available. 🚦 If there's no useful message, it returns None. 🙃

It's perfect for reporting which resource is the limiting factor when scheduling jobs. 💡 If the leader thinks the workflow is stuck, the message can be shown to the user so they can figure out why it might be stuck. 🕵️‍♀️

Returns: a user-directed message about the scheduling state. 💬

Return type: Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str) 🍓

🦋 abstractmethod shutdown()

This is called at the completion of a Toil invocation. 🏁 It should cleanly terminate all worker threads, no messy leftovers. 🧹✨

Return type: None 👋

🦋 abstractmethod setEnv(name, value=None)

This sets an environment variable for the worker process before it's launched. 🌱

The worker process will typically inherit the environment of the machine it's running on. This method lets you override specific variables in that inherited environment before the worker launches. 💅 Note that this mechanism is different from the one the worker uses internally to set up the environment of a job. A call to this method affects all jobs issued after the method returns.

Note to implementors: this means you'd typically need to copy the variables before enqueuing a job. 🫶

If no value is provided, it's looked up from the current environment. 🔍

Parameters:

• name (str) (/usr/share/doc/python3/html/library/stdtypes.html#str)
• value (Optional[str]) (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None 🌸

🦋 classmethod add_options(parser)

If this batch system provides any command line options, this adds them to the given parser. 🛠️

Parameters:

• parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup])

Return type: None 🎀

🦋 classmethod setOptions(setOption)

This processes command line or configuration options relevant to this batch system. ⚙️

Parameters:

• setOption (toil.batchSystems.options.OptionSetter) — a function with the signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) that returns nothing. It's used to update the run configuration as a side effect. ✨

Return type: None 💜

🦋 getWorkerContexts()

This gets a list of picklable context manager objects to wrap worker work in, in order. 🎁

It can be used to ask the Toil worker to do things in-process, like configuring environment variables, hot-deploying user scripts, or cleaning up a node. Those things would otherwise need a wrapping "executor" process, and nobody wants that extra drama. 🙄🔥

Return type: list[ContextManager[Any]] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🌈

Okay so bestie, this next bit is all about the BatchSystemSupport class 💅✨ It's a partial implementation of AbstractBatchSystem that's basically the helper crew, and they are giving main character energy 🎀

class toil.batchSystems.abstractBatchSystem.BatchSystemSupport(config, maxCores, maxMemory, maxDisk)

Bases: AbstractBatchSystem 🦋

Parameters

• config (toil.common.Config) 💖
• maxCores (float) 🌸
• maxMemory (float) 🌸
• maxDisk (int) 🌸

Attributes

• config
• maxCores
• maxMemory
• maxDisk
• environment: dict[str, str]

check_resource_request(requirer)

So this one checks that a resource request isn't bigger than what's available or allowed. No overreaching allowed 🙅‍♀️

Parameters

• requirer (toil.job.Requirer) — the object whose requirements are being checked 👀
• job_name (str) — the name of the job being checked, so you get a useful error report 🧾
• detail (str) — a batch-system-specific message to include in the error 💬

Raises

InsufficientSystemResources — raised when a resource is requested in an amount greater than allowed 💀

Return type

None

setEnv(name, value=None)

This sets an environment variable for the worker process before it launches. Workers usually inherit the environment of whatever machine they're on, but this lets you override specific variables in that inherited environment before the worker starts 🧚 Lowkey important: this is a different mechanism from the one the worker uses internally to set up a job's environment. Any call to this method affects every job issued after it returns. To the implementors out there, this means you'll typically need to copy the variables before enqueuing a job 🫶

If no value is given, it gets looked up from the current environment.

Parameters

• name (str) — the environment variable to be set on the worker 🌈
• value (Optional[str]) — if given, the environment variable named by name gets set to this value. If None, the variable's current value gets used as the value on the worker 💜

Raises

RuntimeError — raised if value is None and the name can't be found in the environment 😭

Return type

None

set_message_bus(message_bus)

This gives the batch system a chance to connect straight to the message bus, so it can send informational messages about its jobs to other Toil components. It's giving collab energy 🤝

Parameters

• message_bus (toil.bus.MessageBus)

Return type

None

get_batch_logs_dir()

This gets the directory where the backing batch system should save its logs 📂

Only really makes sense if the backing batch system actually saves logs to a filesystem. Kubernetes, for example, does not 🙈 It ought to be a directory shared between the leader and the workers, if the backing batch system writes logs onto the worker's view of the filesystem, like many HPC schedulers do.

Return type

str

format_std_out_err_path(toil_job_id, cluster_job_id, std)

This formats the path for the batch system's standard output and error, plus any other files the batch system generates itself 🧾

Files get written to the batch logs directory (--batchLogsDir, which defaults to the Toil work directory) with names that include both the Toil and batch system job IDs, so debugging failed jobs is so much easier 🔍

Parameters

• toil_job_id (int) — the unique ID that Toil gives a job 🆔
• cluster_job_id (str) — what the cluster uses as its internal job ID, for example GridEngine 🖥️
• std (str) — the provenance of the stream, for example 'err' for stderr or 'out' for stdout 🌊

Return type

string — the formatted filename. However, if self.config.noStdOutErr is true, it returns '/dev/null' or the equivalent 🕳️

format_std_out_err_glob(toil_job_id)

This gives you a glob string that matches all the file paths format_std_out_err_path generates for a job. Ultimate pattern-matching queen behavior 👑

Parameters

• toil_job_id (int)

Return type

str

workerCleanup(info) (static)

This cleans up the worker node when the batch system shuts down ✨ Also see supportsWorkerCleanup(), bestie.

Parameters

• info (WorkerCleanupInfo) — a named tuple with all the relevant info for cleaning up the worker 🧹

Return type

None

---

NodeInfo

class toil.batchSystems.abstractBatchSystem.NodeInfo(coresUsed, memoryUsed, coresTotal, memoryTotal, requestedCores, requestedMemory, workers)

Okay, so NodeInfo is all the stats on a node, and each attribute has its own vibe 📊

coresUsed is a floating point value between 0 (all cores idle) and 1 (all cores busy), reflecting the CPU load of the node 🔥

memoryUsed is a floating point value between 0 (no memory used) and 1 (all memory used), reflecting the memory pressure on the node 🧠

coresTotal and memoryTotal are the node's resources, not just the used ones. The full stash 💎

requestedCores and requestedMemory are all the resources that Toil Jobs have reserved on the node, regardless of whether the Jobs are actually using them. Reserved is not the same as used, iykyk 🙃

workers is an integer reflecting the number of currently active workers on the node 🐝

Parameters

• coresUsed (float)
• memoryUsed (float)
• coresTotal (float)
• memoryTotal (int)
• requestedCores (float)
• requestedMemory (int)
• workers (int)

Attributes

• coresUsed
• memoryUsed
• coresTotal
• memoryTotal
• requestedCores
• requestedMemory
• workers

---

AbstractScalableBatchSystem

class toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem

Bases: AbstractBatchSystem 🦋

This is a batch system that can handle a variable number of worker nodes. It's the scaling queen 👑 It's used by toil.provisioners.clusterScaler.ClusterScaler to scale the number of worker nodes in the cluster up or down, depending on overall load 📈📉

getNodes(preemptible=None, timeout=600) (abstract method)

This returns a dictionary mapping node identifiers of preemptible or non-preemptible nodes to NodeInfo objects, one for each node 🗂️

Parameters

• preemptible (Optional[bool]) — if True (or False), only the (non-)preemptible nodes get returned. If None, all nodes get returned 🎯
• timeout (int) ⏳

Return type

dict[str, NodeInfo]

nodeInUse(nodeIP) (abstract method)

This can be used to figure out if a worker node is running any tasks. If the node doesn't exist, this function should just return False 🫠

• nodeIP (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the worker node's private IP address 🌸
• Returns — True if the worker node has been issued any tasks, else False 💖
• Return type — bool (/usr/share/doc/python3/html/library/functions.html#bool) 💅

---

abstract method ignoreNode(nodeAddress) 🛑

Okay so this one tells the batch system to stop sending jobs to a node. It's used in autoscaling when the autoscaler is ready to terminate a node but jobs are still running, so the node gets terminated only after the current jobs have finished 🧹

• nodeAddress (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the IP address of the node to ignore 🫶
• Return type — None 🌈

---

abstract method unignoreNode(nodeAddress) 🦋

This one stops ignoring an address, presumably after a node with that address has been terminated. The vibe is that a brand new node is allowed to show up with the same address as an old terminated one 👑

• nodeAddress (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🌸
• Return type — None ✨

---

💥 exception toil.batchSystems.abstractBatchSystem.InsufficientSystemResources

toil.batchSystems.abstractBatchSystem.InsufficientSystemResources(requirer, resource, available=None, batch_system=None, source=None, details=[])

• Bases — Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception) 🧚

It's the common base class for all non-exit exceptions 💜

• requirer (toil.job.Requirer <#toil.job.Requirer>) 🎀
• resource (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓
• available (Optional[toil.job.ParsedRequirement]) ⭐
• batch_system (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str) 🌸
• source (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str) 🦋
• details (list[str], /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str) 💅

Attributes, bestie:

• job_name — job_name: str | None = '' (/usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/constants.html#None) 🙈
• resource 💖
• requested 💖
• available = None 💖
• batch_system = None 💖
• source = None 💖
• details = [] 💖

__str__() 🔥 Explains the exception.

• Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str) 💀

---

💥 exception toil.batchSystems.abstractBatchSystem.AcquisitionTimeoutException

toil.batchSystems.abstractBatchSystem.AcquisitionTimeoutException(resource, requested, available)

• Bases — Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception) 🧚

This one gets raised when a resource request times out 😭

• resource (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓
• requested (Union[int, float, set[int]], with /usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/functions.html#float, and /usr/share/doc/python3/html/library/stdtypes.html#set plus /usr/share/doc/python3/html/library/functions.html#int) 🌈
• available (Union[int, float, set[int]], with /usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/functions.html#float, and /usr/share/doc/python3/html/library/stdtypes.html#set plus /usr/share/doc/python3/html/library/functions.html#int) 🌈

Attributes:

• requested ✨
• available ✨
• resource ✨

---

🫧 class toil.batchSystems.abstractBatchSystem.ResourcePool

toil.batchSystems.abstractBatchSystem.ResourcePool(initial_value, resource_type, timeout=5)

This is an integral amount of a resource (like memory bytes) 🧠 You can acquire amounts immediately or with a timeout, and you can release them too. It also gives you a context manager so you can do something with an amount of resource you've acquired 🫶

• initial_value (int, /usr/share/doc/python3/html/library/functions.html#int) 🌸
• resource_type (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🌸
• timeout (float, /usr/share/doc/python3/html/library/functions.html#float) 🌸

Attributes:

• condition 💜
• value 💜
• resource_type 💜
• timeout = 5 💜

acquireNow(amount) ⏰ Reserves the given amount of the given resource. It returns True if that worked and False if it's not possible immediately.

• amount (int, /usr/share/doc/python3/html/library/functions.html#int) 🍓
• Return type — bool (/usr/share/doc/python3/html/library/functions.html#bool) 💅

acquire(amount) ⏳ Reserves the given amount of the given resource. If that isn't possible in under self.timeout time, it raises AcquisitionTimeoutException.

• amount (int, /usr/share/doc/python3/html/library/functions.html#int) 🍓
• Return type — None 🌈

release(amount) 🦋

• amount (int, /usr/share/doc/python3/html/library/functions.html#int) 🍓
• Return type — None 🌈

__str__() 🔥

• Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str) 💀

__repr__() 🔥

• Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str) 💀

acquisitionOf(amount) 🎀

• amount (int, /usr/share/doc/python3/html/library/functions.html#int) 🍓
• Return type — collections.abc.Iterator[None] (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator) 👑

---

🫧 class toil.batchSystems.abstractBatchSystem.ResourceSet

toil.batchSystems.abstractBatchSystem.ResourceSet(initial_value, resource_type, timeout=5)

This one represents a collection of distinct resources (like accelerators) 🧚 Subsets can be acquired immediately or with a timeout, and released. There's also a context manager so you can do something with a set of resources you've acquired ✨

• initial_value (set[int], /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/functions.html#int) 🌸
• resource_type (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 🌸
• timeout (float, /usr/share/doc/python3/html/library/functions.html#float) 🌸

Attributes:

• condition 💜
• value 💜
• resource_type 💜
• timeout = 5 💜

acquireNow(subset) ⏰ Reserves the given amount of the given resource. It returns True if that worked and False if it's not possible immediately.

• subset (set[int], /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/functions.html#int) 🍓
• Return type — bool (/usr/share/doc/python3/html/library/functions.html#bool) 💅

acquire(subset) ⏳ Reserves the given amount of the given resource. If that isn't possible in under self.timeout time, it raises AcquisitionTimeoutException.

• subset (set[int], /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/functions.html#int) 🍓
• Return type — None 🌈

release(subset) 🦋

• subset (set[int], /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/functions.html#int) 🍓
• Return type — None 🌈

get_free_snapshot() 📸 Gets a snapshot of what items are free right now. It may be stale as soon as you get it, but you'll need some kind of hint to try and do an acquire.

• Return type — set[int] (/usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/functions.html#int) 💅

__str__() 🔥

• Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str) 💀

__repr__() 🔥

• Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str) 💀

acquisitionOf(subset) 🎀

• subset (set[int], /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/functions.html#int) 🍓
• Return type — collections.abc.Iterator[None] (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator) 👑

---

🌈 toil.batchSystems.abstractGridEngineBatchSystem

Attributes, the whole vibe:

• logger 🧚
• JobTuple 🧚

✨ Exceptions ✨

Okay so the exceptions section is short and sweet 🙈 there's just one here, and it's giving main character energy:

• ExceededRetryAttempts 💀 is the common base class for all non-exit exceptions. It's a subclass of Exception.

🦋 Classes 🦋

Now the classes! 💅 The summary table cuts off mid-sentence, but the full description is below:

• AbstractGridEngineBatchSystem 🌸 a partial implementation of BatchSystemSupport for batch systems run on a

🧚 Module Contents 🧚

Here's everything that lives in the module, bestie 💖

• toil.batchSystems.abstractGridEngineBatchSystem.logger
• toil.batchSystems.abstractGridEngineBatchSystem.JobTuple

🎀 exception toil.batchSystems.abstractGridEngineBatchSystem.ExceededRetryAttempts 🎀

Bases: Exception

Common base class for all non-exit exceptions. It's the one I keep mentioning, so you know it's important 💜

👑 class AbstractGridEngineBatchSystem 👑

class toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport

A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default, auto-deployment is not implemented 🙈

Parameters

• config (toil.common.Config)
• maxCores (float)
• maxMemory (float)
• maxDisk (float)

exception GridEngineThreadException 🍓 Bases: Exception. Common base class for all non-exit exceptions, same energy as the one above ✨

class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) 🧚

Bases: threading.Thread

A class that represents a thread of control. It can be safely subclassed in a limited fashion, and there are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.

Parameters

• newJobsQueue (queue.Queue)
• updatedJobsQueue (queue.Queue)
• killQueue (queue.Queue)
• killedJobsQueue (queue.Queue)
• boss (AbstractGridEngineBatchSystem)

Attributes 🦋

• boss
• newJobsQueue
• updatedJobsQueue
• killQueue
• killedJobsQueue
• waitingJobs: list[JobTuple] = []
• runningJobs
• runningJobsLock
• batchJobIDs: dict[int, str]
• exception = None

Methods 💅

• getBatchSystemID(jobID) ✨ gets the batch system-specific job ID. It's lowkey the only consistent way to cleanly grab that ID right now.
  • Parameters: jobID (int) -- Toil BatchSystem numerical job ID
  • Return type: str
• forgetJob(jobID) 🧹 removes the jobID you pass in.
  • Parameters: jobID (int) -- toil job ID
  • Return type: None
• createJobs(newJob) 🌈 creates a new job with the given attributes. It's implementation-specific and gets called by GridEngineThread.run().
  • Parameters: newJob (JobTuple)
  • Return type: bool
• killJobs() 💀 kills any running jobs within the thread.
• checkOnJobs() 🔍 checks and updates the status of all running jobs. It respects statePollingWait and will return cached results if it isn't time to talk to the scheduler yet.
• run() 🚀 runs any new jobs.
• coalesce_job_exit_codes(batch_job_id_list) 🎯 returns exit codes and possibly exit reasons for a list of jobs, or None if they're still running. It gets called by GridEngineThread.checkOnJobs(). The default version falls back on self.getJobExitCode and polls each job one by one.
  • Parameters: batch_job_id_list (string) -- List of batch system job ID
  • Return type: list[Union[int, tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]]
• abstractmethod prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None) 🎀 is the prep work for putting together a command-line string for submitting to the batch system (via submitJob()).
  • Param int cpu
  • Param int memory
  • Param int jobID: Toil job ID
  • Param string subLine: the command line string to be called
  • Param string jobName: the name of the Toil job, to provide metadata to batch systems if they want it
  • Param dict job_environment: the environment variables to be set on the worker
  • Return type: List[str]
  • Parameters: cpu (int), memory (int), jobID (int), command (str), jobName (str), job_environment (Optional[dict[str, str]]), gpus (Optional[int])
• abstractmethod submitJob(subLine) 🚢 is a wrapper routine that submits the actual command-line call, then processes the output to get the batch system job ID.
  • Param string subLine: the literal command line string to be called
  • Return type: string, the batch system job ID, which gets stored internally
• abstractmethod getRunningJobIDs() 👀 gets a list of running job IDs. It's implementation-specific and gets called by the boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs().
  • Return type: list

abstractmethod killJob(jobID)

okay bestie, this one is the kill switch 💀 It takes out one specific job using its Toil job ID. Every implementation handles the actual killing its own way, and it gets called by GridEngineThread.killJobs() 🔥

• jobID (string) — the Toil job ID

abstractmethod getJobExitCode(batchJobID)

this one gives you the job's exit code, and sometimes an instance of abstractBatchSystem.BatchJobExitReason tagging along ✨

• It returns None if the job is still running 🏃‍♀️
• If the job already stopped but the exit code isn't available, you get EXIT_STATUS_UNAVAILABLE_VALUE 🙈
• Implementation-specific, and it's called by GridEngineThread.checkOnJobs() 💅
• The exit code is only 0 if the job affirmatively succeeded. No vibes-based success here, periodt 🌸

Parameters

• batchjobID (string) — batch system job ID

Return type

Union[int, tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

Doc links for the types in there:

• int → /usr/share/doc/python3/html/library/functions.html#int
• tuple → /usr/share/doc/python3/html/library/stdtypes.html#tuple
• BatchJobExitReason → toil.batchSystems.abstractBatchSystem.BatchJobExitReason

Here are the attributes this class is carrying around 🧚:

• config
• currentJobs
• newJobsQueue
• updatedJobsQueue
• killQueue
• killedJobsQueue
• background_thread

classmethod supportsAutoDeployment()

this checks whether the batch system can auto-deploy the user script itself. If it can, setUserScript() can be called to set the resource object that represents the user script 📦

Note to implementors: if your implementation returns True here, it should also override

---

count_needed_gpus(job_desc)

counts how many cluster-allocatable GPUs you want to grab for the given job 🎀

• job_desc (toil.job.JobDescription)

issueBatchJob(command, job_desc, job_environment=None)

this is the send-it moment 🚀 It issues a job with the command you give it to the batch system, then hands back a unique job ID number.

• command (str) — the command to run somewhere to launch the Toil worker process
• job_desc (toil.job.JobDescription) — the JobDescription for the job being run
• job_environment (Optional[dict[str, str]]) — a bundle of job-specific environment variables to set on the worker 🌈

Returns

a unique job ID number you can use to reference the newly issued job 💖

Doc links for those types:

• str → /usr/share/doc/python3/html/library/stdtypes.html#str
• dict → /usr/share/doc/python3/html/library/stdtypes.html#dict

killBatchJobs(jobIDs)

kills the given jobs (represented as Job ids), then double-checks they're actually dead by making sure they're no longer in the list of issued jobs 💀✨

getIssuedBatchJobIDs()

grabs the list of issued jobs 📋

getRunningBatchJobIDs()

retrieves the running job IDs from both the local machine and the batch scheduler 🏃‍♀️ It respects statePollingWait, so if it isn't time to talk to the scheduler yet, you get cached results instead. Main character patience 👑

getUpdatedBatchJob(maxWait)

returns info about a job that has updated its status, meaning it stopped running either successfully or with an error. Each job only shows up exactly once 🦋

Jobs killed by killBatchJobs won't show up here. They might make this return None sooner than maxWait, though.

• maxWait — the number of seconds to block while waiting for a result

Returns

If a result is available, you get an UpdatedBatchJobInfo. If not, you get None. wallTime is how long the job ran in wall-clock time, as a strictly positive float in seconds, or None if this batch system can't track wall time ⏱️

shutdown()

signals the thread to shut down (using a sentinel), then cleanly joins the thread 🫶

Return type

None

setEnv(name, value=None)

sets an environment variable for the worker process before it launches 🌸 The worker usually inherits the environment of the machine it runs on, but this lets you override specific variables in that inherited environment. This is different from the mechanism the worker uses internally to set up a job's environment, so don't mix them up 🙈 A call to this affects every job issued after the method returns.

Note to implementors: this means you'll typically need to copy the variables before enqueuing a job.

If you don't give a value, it gets looked up from the current environment.

Parameters

• name — the environment variable to set on the worker
• value — if given, the variable named by name is set to this value. If None, the variable's current value is used as the value on the worker

Raises RuntimeError (docs: /usr/share/doc/python3/html/library/exceptions.html#RuntimeError) — if value is None and the name can't be found in the environment 😭

classmethod getWaitDuration()

sleepSeconds(sleeptime=1)

a little helper that every state-querying function leans on so nobody over-queries 🛌

with_retries(operation, *args, **kwargs)

calls operation with args and kwargs. If one of the calls to a command fails, it sleeps and tries again 🔁

---

toil.batchSystems.awsBatch

this is the batch system for running Toil workflows on AWS Batch 🧡 It's especially good with the AWS job store.

AWS Batch can't schedule based on disk usage. The backing machines need to have "enough" disk, and the other constraints need to make sure the disk never fills up. Otherwise it's a whole disaster 💀

It assumes you've already got an AWS Batch Queue name or ARN ready to go. It also handles creating and destroying a JobDefinition for the workflow run 🧁

Additional containers should be launched with Singularity, not Docker 🐳

Attributes

• logger
• STATE_TO_EXIT_REASON
• MAX_POLL_COUNT
• MIN_REQUESTABLE_MIB
• MIN_REQUESTABLE_CORES

Classes

• AWSBatchBatchSystem, which adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler ✨

Module Contents

• toil.batchSystems.awsBatch.logger
• toil.batchSystems.awsBatch.STATE_TO_EXIT_REASON: dict[str, toil.batchSystems.abstractBatchSystem.BatchJobExitReason]
• toil.batchSystems.awsBatch.MAX_POLL_COUNT = 100
• toil.batchSystems.awsBatch.MIN_REQUESTABLE_MIB = 4
• toil.batchSystems.awsBatch.MIN_REQUESTABLE_CORES = 1

class toil.batchSystems.awsBatch.AWSBatchBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport 🧸

This one adds cleanup support when the last running job leaves a node, for batch systems that can't pull that off using the backing scheduler 🧹

Parameters

• config (toil.common.Config)
• maxCores (float) — docs: /usr/share/doc/python3/html/library/functions.html#float
• maxMemory (float) — docs: /usr/share/doc/python3/html/library/functions.html#float
• maxDisk (int) — docs: /usr/share/doc/python3/html/library/functions.html#int

classmethod supportsAutoDeployment()

same deal as before: this checks whether the batch system supports auto-deployment of the user script itself 📦

Okay so, the batch system is about to get its moment 💅 If it does support auto-deployment, you can call setUserScript() to hand over the resource object for the user script 🫶

📝 Note to implementors: If your implementation returns True here, it should also override

Return type — bool 🌸

---

✨ ATTRIBUTES ✨

Here's the full list of attributes on this batch system, no skipping 🦋

• region
• client
• queue
• job_role_arn
• owner_tag
• worker_work_dir
• user_script: toil.resource.Resource | None = None
• docker_image
• job_definition: str | None = None
• bs_id_to_aws_id: dict[int, str]
• aws_id_to_bs_id: dict[str, int]
• killed_job_aws_ids: set[str]

💖 setUserScript(user_script)

Sets the user script for this workflow. Lowkey a must-do: this has to be called before the first job is issued to this batch system, and it only works if supportsAutoDeployment() returns True. Otherwise it's going to raise an exception 😭

Parameters

• userScript — the resource object representing the user script or module, plus any modules it depends on
• user_script (toil.resource.Resource)

Return type — None

🚀 issueBatchJob(command, job_desc, job_environment=None)

Issues a job with the specified command to the batch system and gives you back a unique job ID number. Iconic 👑

Parameters

• command (str) — the command to execute somewhere to run the Toil worker process
• job_desc (toil.job.JobDescription) — the JobDescription for the job being run
• job_environment (Optional[dict[str, str]]) — a collection of job-specific environment variables to be set on the worker

Returns — a unique job ID number that you can use to reference the newly issued job

Return type — int

🔥 getUpdatedBatchJob(maxWait)

Returns info about a job that has updated its status, meaning it stopped running, either successfully or with an error. Each job only shows up here exactly once, so don't expect a repeat 🙈

Jobs killed by killBatchJobs won't be returned here, although they might make this return None earlier than maxWait.

Parameters

• maxWait (int) — the number of seconds to block, waiting for a result

Returns — If a result is available, it returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) of wall-clock time the job ran for, or None if this batch system doesn't support tracking wall time.

Return type — Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo]

🌈 shutdown()

Called at the end of a toil invocation. It should cleanly terminate all worker threads, no drama 🧚

Return type — None

🎀 getIssuedBatchJobIDs()

Gets all currently issued jobs.

Returns — A list of jobs (as job ID numbers) that are currently issued, meaning they might be running or waiting to be run. The result is a list, but don't depend on the ordering 💀

Return type — list[int]

🏃‍♀️ getRunningBatchJobIDs()

Gets a map of job ID numbers for jobs that are currently running (not just waiting), along with how long each one has been running, in seconds ⭐

Returns — a dictionary with currently running job ID number keys, and how many seconds they've been running as the value

Return type — dict[int, float]

🛑 killBatchJobs(job_ids)

Kills the given job IDs. After it returns, the killed jobs won't show up in getRunningBatchJobIDs, and they won't be returned from getUpdatedBatchJob either 🍓

Parameters

• jobIDs — list of IDs of jobs to kill
• job_ids (list[int])

Return type — None

🧩 classmethod add_options(parser)

If this batch system has any command line options, this adds them to the given parser 🦋

Parameters

• parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup])

Return type — None

🌸 classmethod setOptions(setOption)

Processes command line or configuration options that are relevant to this batch system 💜

Parameters

• setOption (toil.batchSystems.options.OptionSetter) — a function with the signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) that returns nothing. It's used to update the run configuration as a side effect ✨

Return type — None

---

✨ toil.batchSystems.cleanup_support ✨

Okay, new module, same energy 🫶

🧁 Attributes

• logger

🎀 Classes

• BatchSystemCleanupSupport — adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler 🌈
• WorkerCleanupContext — a context manager used by BatchSystemCleanupSupport to implement

💅 Module Contents

• toil.batchSystems.cleanup_support.logger

🦋 class toil.batchSystems.cleanup_support.BatchSystemCleanupSupport(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.local_support.BatchSystemLocalSupport

Adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler 🔥

Parameters

• config (toil.common.Config)
• maxCores (float)
• maxMemory (float)
• maxDisk (int)

💖 classmethod supportsWorkerCleanup()

Whether this batch system supports worker cleanup. Main character energy 👑

It tells you whether this batch system calls BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Quick clarity moment: the word worker means an entire node here, not just a worker process. A worker process can run more than one job sequentially, and more than one concurrent worker process can exist on a worker node for the same workflow. The batch system is said to shut down after the last worker process terminates 😭

Okay so the return type here is bool (/usr/share/doc/python3/html/library/functions.html#bool) 💅 yes or no, bestie, nothing in between ✨

🌸 getWorkerContexts() 🌸

Okay this one is giving main character energy 👑. It hands back a list of picklable context manager objects that wrap the worker's work, and they run in order 🧚. It's perfect for asking the Toil worker to do stuff in-process, like configuring environment variables, hot-deploying user scripts, or cleaning up a node, stuff that would otherwise need a wrapping "executor" process 🫶.

Return type: list[ContextManager[Any]] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🎀

---

🦋 class toil.batchSystems.cleanup_support.WorkerCleanupContext(workerCleanupInfo) 🦋

Okay bestie, this is the context manager that BatchSystemCleanupSupport uses to clean up a node once the last worker is done working 🧹. It gets wrapped around the worker's work, so it's always there for the clean-up crew 💖.

Parameters

• workerCleanupInfo (toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo) 🌈

Attribute: workerCleanupInfo ✨

__enter__()

Return type None 💅

__exit__(type, value, traceback)

Parameters:

• type (Optional[WorkerCleanupContext.__exit__.type[BaseException]], with BaseException at /usr/share/doc/python3/html/library/exceptions.html#BaseException) 🙈
• value (Optional[BaseException], with BaseException at /usr/share/doc/python3/html/library/exceptions.html#BaseException) 🙈
• traceback (Optional[types.TracebackType], with types.TracebackType at /usr/share/doc/python3/html/library/types.html#types.TracebackType) 🙈

Return type None 💜

---

🧚 toil.batchSystems.contained_executor 🧚

Okay this is the executor for running inside a container 🐳. It's super useful for the Kubernetes batch system and the TES batch system plugin, so you know it's important 💖

✨ Attributes ✨

• logger 📝

🌈 Functions 🌈

• pack_job(command[, user_script, environment]) — Creates a command that runs the given command in an environment 🫧
• executor() — The main function of the _toil_contained_executor entrypoint 🔥

💜 Module Contents 💜

toil.batchSystems.contained_executor.logger 📝

toil.batchSystems.contained_executor.pack_job(command, user_script=None, environment=None)

Okay so this one builds the command that runs your job inside an environment 🌸 It returns a command to run the job, as an argument list that can be run inside the Toil appliance container 🎁

Parameters:

• command (str, with str at /usr/share/doc/python3/html/library/stdtypes.html#str) — the worker command to run to run the job 💅
• user_script (Optional[toil.resource.Resource]) — the user script that will be loaded before the job is run 📜
• environment (Optional[dict[str, str]], with dict at /usr/share/doc/python3/html/library/stdtypes.html#dict and str at /usr/share/doc/python3/html/library/stdtypes.html#str) — the environment variable dict that will be applied before the job is run 🌿

Return type: list[str] (/usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀

toil.batchSystems.contained_executor.executor()

Okay this is the main function of the _toil_contained_executor entrypoint 🧸 It runs inside the Toil container and is responsible for setting up the user script and running the command for the job. That command may in turn invoke the Toil worker entrypoint, so it's like a whole chain of slay 🔗

Return type: None 💖

---

🍓 toil.batchSystems.gridengine 🍓

✨ Attributes ✨

• logger 📝

🌷 Classes 🌷

• GridEngineBatchSystem — A partial implementation of BatchSystemSupport for batch systems run on a (the table cuts off right there, bestie 😭)

💜 Module Contents 💜

toil.batchSystems.gridengine.logger 📝

🦋 class toil.batchSystems.gridengine.GridEngineBatchSystem(config, maxCores, maxMemory, maxDisk) 🦋

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem 🌸

Okay this is a partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster 🖥️ By default, auto-deployment is not implemented, so keep that in mind 👀

Parameters:

• config (toil.common.Config) ⚙️
• maxCores (float, with float at /usr/share/doc/python3/html/library/functions.html#float) 🔢
• maxMemory (float, with float at /usr/share/doc/python3/html/library/functions.html#float) 🧠
• maxDisk (float, with float at /usr/share/doc/python3/html/library/functions.html#float) 💾

🫧 class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) 🫧

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread ✨

Okay this is the Grid Engine-specific AbstractGridEngineWorker methods, so it's where the Grid Engine magic lives 🪄

Parameters:

• newJobsQueue (queue.Queue, with queue.Queue at /usr/share/doc/python3/html/library/queue.html#queue.Queue) 📥
• updatedJobsQueue (queue.Queue, with queue.Queue at /usr/share/doc/python3/html/library/queue.html#queue.Queue) 🔄
• killQueue (queue.Queue, with queue.Queue at /usr/share/doc/python3/html/library/queue.html#queue.Queue) 🛑
• killedJobsQueue (queue.Queue, with queue.Queue at /usr/share/doc/python3/html/library/queue.html#queue.Queue) 💀
• boss (AbstractGridEngineBatchSystem) 👑

getRunningJobIDs()

Get a list of running job IDs 🏃‍♀️ This one is implementation-specific and gets called by the boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs() 🫶

Return type: list (/usr/share/doc/python3/html/library/stdtypes.html#list) 🎀

killJob(jobID)

Kill a specific job with the Toil job ID 🔪 This one is implementation-specific and gets called by GridEngineThread.killJobs() 💅

Parameters:

• jobID (string) — the Toil job ID 🆔

prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)

Okay so this is the prep work for putting together a command-line string that gets submitted to the batch system (via submitJob()) 📦 It's the behind-the-scenes glow-up before the job goes out 🌟

Param int cpu 🧮

Param int memory 🧠

Param int jobID — the Toil job ID 🆔

Param string subLine — the command line string to be called 📟

Param string jobName — the name of the Toil job, to provide metadata to batch systems if desired 🏷️

Param dict job_environment — the environment variables to be set on the worker 🌿

Return type: List[str] (/usr/share/doc/python3/html/library/stdtypes.html#str) 💖

Parameters:

• cpu (int, with int at /usr/share/doc/python3/html/library/functions.html#int) 🧮
• memory (int, with int at /usr/share/doc/python3/html/library/functions.html#int) 🧠
• jobID (int, with int at /usr/share/doc/python3/html/library/functions.html#int) 🆔
• command (str, with str at /usr/share/doc/python3/html/library/stdtypes.html#str) 📟
• jobName (str, with str at /usr/share/doc/python3/html/library/stdtypes.html#str) 🏷️
• job_environment (Optional[dict[str, str]], with dict at /usr/share/doc/python3/html/library/stdtypes.html#dict and str at /usr/share/doc/python3/html/library/stdtypes.html#str) 🌿
• gpus (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) 🎮 an optional int, and that's the whole vibe.

🚀 submitJob(subLine)

This is the wrapper that actually fires off your command-line call, then digs through the output to grab the batch system job ID. 🎯

• Param string subLine — the literal command line string that gets called 📞
• Return type string — the batch system job ID, which gets stashed internally 🗂️

🔍 getJobExitCode(sgeJobID)

Grabs the job exit code by checking both qstat and qacct. 🕵️‍♀️ If the job is still running, it returns None. Your higher-level code should retry on CalledProcessErrorStderr, because if the job just finished, the qacct result might be stale and you don't want to panic over nothing. 😭

🧁 prepareQsub(cpu, mem, jobID, job_environment=None)

Parameters

• cpu (int </usr/share/doc/python3/html/library/functions.html#int>)
• mem (int </usr/share/doc/python3/html/library/functions.html#int>)
• jobID (int </usr/share/doc/python3/html/library/functions.html#int>)
• job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

Return type

list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

💤 classmethod getWaitDuration()

---

✨ toil.batchSystems.htcondor ✨

🧩 Attributes

• logger
• JobTuple
• schedd_lock

🏛️ Classes

• HTCondorBatchSystem — the partial BatchSystemSupport implementation, which we'll get into below 💖

📦 Module Contents

• toil.batchSystems.htcondor.logger
• toil.batchSystems.htcondor.JobTuple
• toil.batchSystems.htcondor.schedd_lock

🐝 class toil.batchSystems.htcondor.HTCondorBatchSystem(config, maxCores, maxMemory, maxDisk)

• Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>

This is a partial implementation of BatchSystemSupport for batch systems that run on a standard HPC cluster. 🖥️ By default, auto-deployment is not implemented, so don't count on it being there. 🙈

Parameters

• config (toil.common.Config <#toil.common.Config>)
• maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)
• maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)
• maxDisk (float </usr/share/doc/python3/html/library/functions.html#float>)

🧵 class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) (nested inside HTCondorBatchSystem)

• Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread>

This is a thread-of-control class, so basically a little worker thread. 🧵 You can subclass it safely, but only in a limited way. There are two ways to tell it what to do: pass a callable object to the constructor, or override the run() method in your subclass. 💅

Parameters

• newJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)
• updatedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)
• killQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)
• killedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)
• boss (AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>)

🪄 createJobs(newJob)

Creates a brand-new job from the attributes you give it. ✨ It's implementation-specific and gets called by GridEngineThread.run().

• Parameters newJob (JobTuple)
• Return type bool </usr/share/doc/python3/html/library/functions.html#bool>

🧾 prepareSubmission(cpu, memory, disk, jobID, jobName, command, environment)

This is the prep work for putting together the command-line string you'll submit to the batch system (via submitJob()). 📝

• Param int cpu
• Param int memory
• Param int jobID — the Toil job ID 🆔
• Param string subLine — the command line string to be called
• Param string jobName — the name of the Toil job, handy for giving batch systems metadata if they want it 🏷️
• Param dict job_environment — the environment variables to be set on the worker 🌿
• Return type List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

Parameters

• cpu (int </usr/share/doc/python3/html/library/functions.html#int>)
• memory (int </usr/share/doc/python3/html/library/functions.html#int>)
• disk (int </usr/share/doc/python3/html/library/functions.html#int>)
• jobID (int </usr/share/doc/python3/html/library/functions.html#int>)
• jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• environment (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

🚀 submitJob(submitObj)

Wrapper that fires off the actual command-line call, then processes the output to get the batch system job ID. 🎯

• Param string subLine — the literal command line string to be called 📞
• Return type string — the batch system job ID, which gets stored internally 🗂️

📋 getRunningJobIDs()

Gets you a list of the running job IDs. 👀 It's implementation-specific, and the boss AbstractGridEngineBatchSystem implementation calls it via AbstractGridEngineBatchSystem.getRunningBatchJobIDs().

• Return type list </usr/share/doc/python3/html/library/stdtypes.html#list>

🛑 killJob(jobID)

Kills one specific job, identified by its Toil job ID. 💀 It's implementation-specific and gets called by GridEngineThread.killJobs().

• Parameters jobID (string) — the Toil job ID

🏁 getJobExitCode(batchJobID)

Returns the job exit code, and possibly an instance of abstractBatchSystem.BatchJobExitReason. 🎀

• If the job is still running, it returns None. ⏳
• If the job isn't running but the exit code isn't available, you'll get EXIT_STATUS_UNAVAILABLE_VALUE. 🤷‍♀️
• It's implementation-specific and gets called by GridEngineThread.checkOnJobs().
• > The exit code will only be 0 if the job affirmatively succeeded. 🥹

Parameters

• batchjobID (string) — the batch system job ID

🔌 connectSchedd()

Connects to the HTCondor Schedd and yields a Schedd object. 🔌 You can only use it inside the context. It handles locking so only one thread is trying to do this at a time, so no tug-of-war. 🫶

✂️ duplicate_quotes(value)

Escapes a string by doubling up every single and double quote in it. 💜

This one's for arguments we pass to HTCondor that have to be wrapped in both double and single quote enclosures. Double-wrapped and iconic 💅✨

Parameters

• value (str, ref /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀

Return type

• str (ref /usr/share/doc/python3/html/library/stdtypes.html#str)

---

✨ getEnvString(overrides) ✨

Okay so this one builds an environment string that an HTCondor Submit object can use 🧚 If you want examples of valid strings, check out http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html#man-condor-submit-environment 🔗

Parameters

• overrides (dict[str, str]) 🌸
  • dict ref /usr/share/doc/python3/html/library/stdtypes.html#dict
  • str ref /usr/share/doc/python3/html/library/stdtypes.html#str

Return type

• str (ref /usr/share/doc/python3/html/library/stdtypes.html#str)

---

✨ issueBatchJob(command, jobNode, job_environment=None) ✨

issueBatchJob(command, jobNode, job_environment=None)

Bestie, this one issues a job with the specified command to the batch system and gives you back a unique job ID number 🦋

Parameters

• command (str, ref /usr/share/doc/python3/html/library/stdtypes.html#str) 💖
  • The command to execute somewhere to run the Toil worker process.
• job_desc 🫶
  • The JobDescription for the job being run.
• job_environment (Optional[dict[str, str]], refs /usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/stdtypes.html#str) 🌈
  • A collection of job-specific environment variables to be set on the worker.

Returns

• A unique job ID number that can be used to reference the newly issued job 👑

---

✨ toil.batchSystems.kubernetes ✨

Batch system for running Toil workflows on Kubernetes ☁️💜

Only useful with network-based job stores, like AWSJobStore. 🙈

Within non-privileged Kubernetes containers, additional Docker containers cannot yet be launched. That functionality will need to wait for user-mode Docker 💀

Attributes

Here's the squad of attributes 🍓

• logger
• retryable_kubernetes_errors
• KeyValuesList

Classes

• KubernetesBatchSystem, which adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler 🎀

Functions

• is_retryable_kubernetes_error(e), a function that determines whether or not Toil should retry or stop given exceptions thrown by Kubernetes 💅

Module Contents

• toil.batchSystems.kubernetes.logger
• toil.batchSystems.kubernetes.retryable_kubernetes_errors: list[type[Exception] | toil.lib.retry.ErrorCondition]
  • list ref /usr/share/doc/python3/html/library/stdtypes.html#list
  • type ref /usr/share/doc/python3/html/library/functions.html#type
  • Exception ref /usr/share/doc/python3/html/library/exceptions.html#Exception
  • toil.lib.retry.ErrorCondition ref #toil.lib.retry.ErrorCondition
• toil.batchSystems.kubernetes.is_retryable_kubernetes_error(e)

A function that determines whether or not Toil should retry or stop given exceptions thrown by Kubernetes 🦋

Parameters

• e (Exception, ref /usr/share/doc/python3/html/library/exceptions.html#Exception)

Return type

• bool (ref /usr/share/doc/python3/html/library/functions.html#bool)

• toil.batchSystems.kubernetes.KeyValuesList
• class toil.batchSystems.kubernetes.KubernetesBatchSystem(config, maxCores, maxMemory, maxDisk)

class toil.batchSystems.kubernetes.KubernetesBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport (ref #toil.batchSystems.cleanup_support.BatchSystemCleanupSupport)

Adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler 🧹

Parameters

• config (toil.common.Config, ref #toil.common.Config)
• maxCores (int, ref /usr/share/doc/python3/html/library/functions.html#int)
• maxMemory (float, ref /usr/share/doc/python3/html/library/functions.html#float)
• maxDisk (int, ref /usr/share/doc/python3/html/library/functions.html#int)
• classmethod supportsAutoDeployment() 🚀
  • Whether this batch system supports auto-deployment of the user script itself.
  • If it does, setUserScript() can be invoked to set the resource object representing the user script.
  • > Note to implementors: If your implementation returns True here, it should also override

Return type

• bool (ref /usr/share/doc/python3/html/library/functions.html#bool)

• credential_time: datetime.datetime | None = None 🕰️
  • Refs /usr/share/doc/python3/html/library/datetime.html#datetime.datetime and /usr/share/doc/python3/html/library/constants.html#None
• namespace: str 🌈
  • Ref /usr/share/doc/python3/html/library/stdtypes.html#str
• host_path: str | None 🌸
  • Refs /usr/share/doc/python3/html/library/stdtypes.html#str and /usr/share/doc/python3/html/library/constants.html#None
• service_account: str | None 🧚
  • Refs /usr/share/doc/python3/html/library/stdtypes.html#str and /usr/share/doc/python3/html/library/constants.html#None
• pod_timeout: float ⏱️
  • Ref /usr/share/doc/python3/html/library/functions.html#float
• unique_id 🔑
• job_prefix: str = 'Uninferable-toil-Instance of uuid.UUID-' 🍓
  • Ref /usr/share/doc/python3/html/library/stdtypes.html#str
• finished_job_ttl: int = 3600 ⭐
  • Ref /usr/share/doc/python3/html/library/functions.html#int
• user_script: toil.resource.Resource | None = None 💜
  • Refs #toil.resource.Resource and /usr/share/doc/python3/html/library/constants.html#None
• docker_image: str 🐳
  • Ref /usr/share/doc/python3/html/library/stdtypes.html#str
• worker_work_dir: str 📂
  • Ref /usr/share/doc/python3/html/library/stdtypes.html#str
• aws_secret_name: str | None 🔐
  • Refs /usr/share/doc/python3/html/library/stdtypes.html#str and /usr/share/doc/python3/html/library/constants.html#None
• enable_watching: bool 👀
  • Ref /usr/share/doc/python3/html/library/functions.html#bool
• run_id: str = 'toil-Instance of uuid.UUID' 🦄
  • Ref /usr/share/doc/python3/html/library/stdtypes.html#str
• resource_sources: list[toil.batchSystems.abstractBatchSystem.ResourcePool] 🎁
  • Refs /usr/share/doc/python3/html/library/stdtypes.html#list and #toil.batchSystems.abstractBatchSystem.ResourcePool
• schedulingThread: threading.Thread 🧵
  • Ref /usr/share/doc/python3/html/library/threading.html#threading.Thread
• class DecoratorWrapper(to_wrap, decorator) 🎀
  • Class to wrap an object so all its methods are decorated.

Parameters

• to_wrap (Any)
• decorator (Callable[[Callable[P, Any]], Callable[P, Any]])
• P
• __getattr__(name) 💅
  • Get a member as if we are actually the wrapped object. If it looks callable, we will decorate it.

Parameters

• name (str, ref /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

• Any

• ItemT
• CovItemT
• P
• R
• setUserScript(userScript) 📜
  • Set the user script for this workflow.
  • This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True, otherwise it will raise an exception. 🚨

Parameters

• userScript (toil.resource.Resource, ref #toil.resource.Resource) — the resource object representing the user script or module and the modules it depends on.

Return type

• None

• class Placement 🧩
  • Internal format for pod placement constraints and preferences.
  • required_labels: KeyValuesList = [] 🔒
    • Labels which are required to be present (with these values).
  • desired_labels: KeyValuesList = [] 🌟
    • Labels which are optional, but preferred to be present (with these values).

• prohibited_labels: KeyValuesList = []: 🚫 these are the labels that are NOT allowed to be on a node, but only when they show up with these exact values. Total no-go vibes 💀
• tolerated_taints: KeyValuesList = []: 🧪 these are the taints that are totally allowed to be on a node (again, only with these values). Pods can hang there, no drama ✨

set_preemptible(preemptible) 🦋

Okay so this one sets the constraints for whether a job is preemptible or not. Lowkey the most important part of the whole thing 🔥

Preemptible jobs can run on preemptible or non-preemptible nodes, and they'll prefer the preemptible nodes if any are available 💨

Non-preemptible jobs are not allowed on nodes marked as preemptible. No cap, that's the rule 🙅‍♀️

It understands the labeling scheme that EKS uses and the taint scheme that GCE uses. The Toil-managed Kubernetes setup mimics at least one of these, so you're covered 🫶

Parameters:

• preemptible (bool, /usr/share/doc/python3/html/library/functions.html#bool)

Return type: None

apply(pod_spec) 💅

This one sets the affinity and/or tolerations fields on pod_spec so the pod lands on the right kind of nodes for the constraints we've got. It's giving organized 🌸

Parameters:

• pod_spec (kubernetes.client.V1PodSpec)

Return type: None

class FakeResponse 🎀

Just one field, bestie:

• data: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

T 🧚

A solo T with no extra notes. Simple, iconic.

issueBatchJob(command, job_desc, job_environment=None) 🚀

Issues a job with the specified command to the batch system and gives you back a unique job ID number 🔢

Parameters:

• command (str, /usr/share/doc/python3/html/library/stdtypes.html#str): the command to execute somewhere to run the Toil worker process
• job_desc (toil.job.JobDescription, #toil.job.JobDescription): the JobDescription for the job being run
• job_environment (Optional[dict[str, str]], /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str): a collection of job-specific environment variables to be set on the worker 🌈

Returns: a unique job ID number that you can use to reference the newly issued job

Return type: int (/usr/share/doc/python3/html/library/functions.html#int)

getUpdatedBatchJob(maxWait) ⏰

Returns info about a job that has updated its status, meaning it stopped running either successfully or with an error. Each job only shows up exactly once, so no repeats 👀

Jobs killed by killBatchJobs won't get returned here, although they may cause None to come back earlier than maxWait.

Parameters:

• maxWait (float, /usr/share/doc/python3/html/library/functions.html#float): the number of seconds to block, waiting for a result

Returns: if a result is available, it returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) of wall-clock time the job ran for, or None if this batch system doesn't support tracking wall time ⏱️

Return type: Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo] (#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo)

shutdown() 🛑

Called at the completion of a toil invocation. It should cleanly terminate all worker threads, so it's giving tidy exit energy ✨

Return type: None

getIssuedBatchJobIDs() 📋

Gets all currently issued jobs.

Returns: a list of jobs (as job ID numbers) that are currently issued. They might be running, or they might be waiting to run. Even though the result is a list, don't depend on the ordering 🙈

Return type: list[int] (/usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/functions.html#int)

getRunningBatchJobIDs() 🏃‍♀️

Gets a map of job ID numbers for jobs that are currently running (not just waiting), along with how long each one has been running, in seconds 🕰️

Returns: a dictionary with currently running job ID number keys, and how many seconds they've been running as the value

Return type: dict[int, float] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/functions.html#float)

killBatchJobs(jobIDs) 💀

Kills the given job IDs. After it returns, the killed jobs won't appear in the results of getRunningBatchJobIDs, and they won't be returned from getUpdatedBatchJob either. Bye bye 👋

Parameters:

• jobIDs (list[int], /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/functions.html#int): list of IDs of jobs to kill

Return type: None

classmethod get_default_kubernetes_owner() 🧑‍💻

Gets the default Kubernetes-acceptable username string to tack onto jobs.

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

class KubernetesConfig 🌈

Bases: Protocol

This is a type-enforcing protocol for Toil configs that have the extra Kubernetes batch system fields. Main character energy 👑

TODO: Until MyPY lets protocols inherit from non-protocols, we'll have to manually enforce that this also has to be a Config. Not ideal, but we're working on it 🫠

Fields:

• kubernetes_host_path: str | None (/usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/constants.html#None)
• kubernetes_owner: str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• kubernetes_service_account: str | None (/usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/constants.html#None)
• kubernetes_pod_timeout: float (/usr/share/doc/python3/html/library/functions.html#float)

classmethod add_options(parser) 🧁

If this batch system provides any command line options, this adds them to the given parser.

Parameters:

• parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup], /usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser)

Return type: None

OptionType 🫶

Just the name, no description here.

classmethod setOptions(setOption) ⚙️

Processes command line or configuration options relevant to this batch system.

Parameters:

• setOption (toil.batchSystems.options.OptionSetter, #toil.batchSystems.options.OptionSetter): a function with the signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) that returns nothing. It updates the run configuration as a side effect 🪄

Return type: None

---

toil.batchSystems.local_support 🌸

Attributes

• logger 🪵

Classes

• BatchSystemLocalSupport: adds a local queue for helper jobs, useful for CWL & others 💖

Module Contents

toil.batchSystems.local_support.logger

class toil.batchSystems.local_support.BatchSystemLocalSupport(config, maxCores, maxMemory, maxDisk) ✨

Bases: toil.batchSystems.abstractBatchSystem.BatchSystemSupport (#toil.batchSystems.abstractBatchSystem.BatchSystemSupport)

Adds a local queue for helper jobs, useful for CWL & others. It's the helper-job bestie 🧚

Parameters:

• config (toil.common.Config, #toil.common.Config)
• maxCores (float, /usr/share/doc/python3/html/library/functions.html#float)
• maxMemory (float, /usr/share/doc/python3/html/library/functions.html#float)
• maxDisk (int, /usr/share/doc/python3/html/library/functions.html#int)

Attribute localBatch: typed as toil.batchSystems.singleMachine.SingleMachineBatchSystem (#toil.batchSystems.singleMachine.SingleMachineBatchSystem) 💜

handleLocalJob(command, jobDesc) 🦋

To be called by issueBatchJob. Ngl, it's the handoff that keeps things moving 🔥

Returns the jobID if the jobDesc got submitted to the local queue, and None if it didn't 💖✨

Parameters

• command (str) 🎀
• jobDesc (toil.job.JobDescription) 🦋

Return type: Optional[int]

💅 killLocalJobs(jobIDs)

Will kill all local jobs that match the provided jobIDs. It gets called by killBatchJobs, so you won't usually be calling it yourself 🫶

Parameters

• jobIDs (list[int]) 🌸

Return type: None

🧚 getIssuedLocalJobIDs()

Called by getIssuedBatchJobIDs. It's the local side of that lookup, so it's giving teamwork 🤝

Return type: list[int]

🌈 getRunningLocalJobIDs()

Called by getRunningBatchJobIDs(). Okay so this one hands back a dict mapping each running job's int ID to a float 👑

Return type: dict[int, float]

✨ getUpdatedLocalJob(maxWait)

Called by getUpdatedBatchJob(). Literally the "what's new with my jobs?" check 👀

Parameters

• maxWait (int) ⭐

Return type: Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo]

🔥 getNextJobID()

Must be used to get job IDs so that the local and batch jobs do not conflict. Do not make up your own IDs, bestie, it will get messy 💀

Return type: int

🦋 shutdownLocal()

Called from shutdown(). Time to clean up and log off 🌙

Return type: None

---

🌸 toil.batchSystems.lsf 🌸

Attributes

• logger 🪄

Classes

• LSFBatchSystem — a partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster 💜

Module Contents

toil.batchSystems.lsf.logger

✨ class toil.batchSystems.lsf.LSFBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

This is a partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default, auto-deployment is not implemented, so keep that in mind 🙈

Parameters

• config (toil.common.Config) 🎀
• maxCores (float) 🎀
• maxMemory (float) 🎀
• maxDisk (float) 🎀

💖 class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread

These are the LSF-specific GridEngineThread methods, so this is where the LSF magic happens ✨

Parameters

• newJobsQueue (queue.Queue) 🦋
• updatedJobsQueue (queue.Queue) 🦋
• killQueue (queue.Queue) 🦋
• killedJobsQueue (queue.Queue) 🦋
• boss (AbstractGridEngineBatchSystem) 👑

🌟 getRunningJobIDs()

Gets a list of running job IDs. This one is implementation-specific, and the boss AbstractGridEngineBatchSystem implementation calls it through AbstractGridEngineBatchSystem.getRunningBatchJobIDs().

Return type: list

🫧 fallbackRunningJobIDs(currentjobs)

🗡️ killJob(jobID)

Kills one specific job using its Toil job ID. This is implementation-specific, and GridEngineThread.killJobs() calls it 💀

Parameters

• jobID (string) — the Toil job ID 🍓

🪩 prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)

This is the prep work. It puts together the command-line string that gets handed to the batch system (via submitJob()) 🧁

• int cpu 🌸
• int memory 🌸
• int jobID — the Toil job ID 🌸
• string subLine — the command-line string to be called 🌸
• string jobName — the name of the Toil job, which gives batch systems metadata if they want it 🌸
• dict job_environment — the environment variables to set on the worker 🌸

Parameters

• cpu (int) 🎀
• memory (int) 🎀
• jobID (int) 🎀
• command (str) 🎀
• jobName (str) 🎀
• job_environment (Optional[dict[str, str]]) 🎀
• gpus (Optional[int]) 🎀

Return type: List[str]

🚀 submitJob(subLine)

This is the wrapper that submits the actual command-line call, then processes the output to get the batch system job ID 📮

Parameters

• subLine (string) — the literal command-line string to be called 💜

Return type: string — the batch system job ID, which gets stored internally 🔒

🌙 coalesce_job_exit_codes(batch_job_id_list)

Returns exit codes and possibly exit reasons for a list of jobs, or None if they're still running 🏃‍♀️ It's called by GridEngineThread.checkOnJobs().

The default version falls back on self.getJobExitCode and polls each job one by one.

Parameters

• batch_job_id_list (string) — the list of batch system job IDs 🌷

Return type: list

🏁 getJobExitCode(lsfJobID)

Returns the job's exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason.

• It returns None if the job is still running ⏳
• If the job isn't running but the exit code isn't available, you'll get EXIT_STATUS_UNAVAILABLE_VALUE 🫠
• The exit code will only be 0 if the job affirmatively succeeded 🎉

This is implementation-specific, and GridEngineThread.checkOnJobs() calls it.

Parameters

• batchjobID (string) — the batch system job ID 🍒

Okay so this next batch is all about the LSF helpers, and bestie, the return types are giving union energy 💅🔥

Return type (this one is a whole mood ✨)

Union[int, tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

🧚 parse_bjobs_record(bjobs_record, job)

These are helper functions for getJobExitCode and to parse the bjobs status record, so they're doing the behind-the-scenes work 🫶

Parameters

• bjobs_record (dict) 💜
• job (int) 🌸

Return type

Union[int, tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

🦋 getJobExitCodeBACCT(job)

Return type

Union[int, tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

🌈 fallbackGetJobExitCode(job)

This is the backup plan, and it's giving safety net 🧷

Return type

Union[int, tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

🎀 prepareBsub(cpu, mem, jobID)

This one makes a bsub commandline for you to execute, and it's literally so pretty ✨

params:

• cpu: number of cores needed 💖
• mem: number of bytes of memory needed 🧠
• jobID: ID number of the job 🪪

Parameters

• cpu (int)
• mem (int)
• jobID (int)

Return type

list[str]

🍓 parseBjobs(bjobs_output_str)

This one parses records out of bjobs json type output, no cap 🧃

Params

• bjobs_output_str: stdout of bjobs json type output 🖨️

💅 parseMaxMem(jobID)

This one parses the maximum memory from the job, so it's your memory detective 🕵️‍♀️

Parameters

• jobID: ID number of the job 🔢

⭐ getWaitDuration()

Lowkey the sweetest one: we give LSF a second to catch its breath, in seconds 🥹🫧

---

👑 toil.batchSystems.lsfHelper

Attributes

This module has a bunch of attributes, and the page lists just their names with no values next to them, so here they are, iykyk 💜

• LSB_PARAMS_FILENAME 📁
• LSF_CONF_FILENAME 📁
• LSF_CONF_ENV 🌿
• DEFAULT_LSF_UNITS 📏
• DEFAULT_RESOURCE_UNITS 📏
• LSF_JSON_OUTPUT_MIN_VERSION 🔖
• logger 📝

✨ Functions ✨

okay so this table is the cheat sheet 💅 here's every function, bestie:

• find(basedir, string) — walks basedir and returns all files matching string 🔍
• find_first_match(basedir, string) — returns the first file that matches string, starting from basedir 🦋
• get_conf_file(filename, env) — no blurb for this one 🙈
• apply_conf_file(fn, conf_filename) — no blurb for this one either
• per_core_reserve_from_stream(stream) — no blurb, it just exists 🌸
• get_lsf_units_from_stream(stream) — also no blurb 🫶
• tokenize_conf_stream(conf_handle) — converts the key=val pairs in an LSF config stream into tuples of tokens 🧚
• apply_bparams(fn) — applies fn to each line of bparams and returns the result ✨
• apply_lsadmin(fn) — applies fn to each line of lsadmin and returns the result ✨
• get_lsf_units([resource]) — checks if we can find LSF_UNITS_FOR_LIMITS in lsadmin and lsf.conf 💜
• parse_mem_and_cmd_from_output(output) — uses regex to find "MAX MEM" and "Command" inside an output 💀
• get_lsf_version() — gets the current LSF version 🌈
• check_lsf_json_output_supported() — checks if the current LSF system supports bjobs json output 👑
• parse_memory(mem) — parses the memory parameter 🍓
• per_core_reservation() — returns True if the cluster is configured for reservations to be per core, 🎀

---

✨ Module Contents ✨

okay the module contents are where it's at, we're getting the full details 🔥 these constants are the main character energy of this module:

toil.batchSystems.lsfHelper.LSB_PARAMS_FILENAME = 'lsb.params'
toil.batchSystems.lsfHelper.LSF_CONF_FILENAME = 'lsf.conf'
toil.batchSystems.lsfHelper.LSF_CONF_ENV = ['LSF_CONFDIR', 'LSF_ENVDIR']
toil.batchSystems.lsfHelper.DEFAULT_LSF_UNITS = 'KB'
toil.batchSystems.lsfHelper.DEFAULT_RESOURCE_UNITS = 'MB'
toil.batchSystems.lsfHelper.LSF_JSON_OUTPUT_MIN_VERSION = '10.1.0.2'
toil.batchSystems.lsfHelper.logger

Now the functions, one by one 💖

toil.batchSystems.lsfHelper.find(basedir, string)

walk basedir and return all files matching string 🔍

toil.batchSystems.lsfHelper.find_first_match(basedir, string)

return the first file that matches string starting from basedir 🦋

toil.batchSystems.lsfHelper.get_conf_file(filename, env)

toil.batchSystems.lsfHelper.apply_conf_file(fn, conf_filename)

toil.batchSystems.lsfHelper.per_core_reserve_from_stream(stream)

toil.batchSystems.lsfHelper.get_lsf_units_from_stream(stream)

toil.batchSystems.lsfHelper.tokenize_conf_stream(conf_handle)

convert the key=val pairs in a LSF config stream to tuples of tokens 🧚

toil.batchSystems.lsfHelper.apply_bparams(fn)

apply fn to each line of bparams, returning the result ✨

toil.batchSystems.lsfHelper.apply_lsadmin(fn)

apply fn to each line of lsadmin, returning the result ✨

toil.batchSystems.lsfHelper.get_lsf_units(resource=False)

okay this one's a whole vibe 💜 it checks if we can find LSF_UNITS_FOR_LIMITS in lsadmin and lsf.conf files, and it prefers the value in bparams, then lsadmin, then the lsf.conf file.

Parameters

• resource (bool, see /usr/share/doc/python3/html/library/functions.html#bool)

Return type

• str, see /usr/share/doc/python3/html/library/stdtypes.html#str

toil.batchSystems.lsfHelper.parse_mem_and_cmd_from_output(output)

Use regex to find "MAX MEM" and "Command" inside of an output. 💀

Parameters

• output (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.batchSystems.lsfHelper.get_lsf_version()

Get current LSF version 🌈

toil.batchSystems.lsfHelper.check_lsf_json_output_supported()

Check if the current LSF system supports bjobs json output. 👑

toil.batchSystems.lsfHelper.parse_memory(mem)

Parse memory parameter. 🍓

Parameters

• mem (float, see /usr/share/doc/python3/html/library/functions.html#float)

Return type

• str, see /usr/share/doc/python3/html/library/stdtypes.html#str

toil.batchSystems.lsfHelper.per_core_reservation()

returns True if the cluster is configured for reservations to be per core, False if it is per job 🎀

---

✨ toil.batchSystems.mesos ✨

okay we're moving on to the mesos bit 🦋

Submodules

• toil.batchSystems.mesos.batchSystem

✨ toil.batchSystems.mesos.batchSystem ✨

Attributes

• log — no value listed for this one 🙈

Classes

• MesosBatchSystem — A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos 🌸

Module Contents

The attribute first, bestie:

toil.batchSystems.mesos.batchSystem.log

Now the class, and it's giving full main character 👑

class toil.batchSystems.mesos.batchSystem.MesosBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.local_support.BatchSystemLocalSupport, toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem, pymesos.Scheduler

A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos tasks over a cluster of agent nodes. A Mesos framework has a scheduler and an executor. This class is the scheduler, and it's typically run on the master node that also runs the Mesos master process, which it talks to through a driver component. The executor is a separate class that runs on each agent node and talks to the Mesos agent process through its own driver object. The scheduler can also run on a separate node from the master, which is somewhat ambiguously called the driver node 🫶

Parameters

• config (toil.common.Config)
• maxCores (float, see /usr/share/doc/python3/html/library/functions.html#float)
• maxMemory (float, /usr/share/doc/python3/html/library/functions.html#float) 🧠 this one's the memory cap for a job
• maxDisk (int, /usr/share/doc/python3/html/library/functions.html#int) 💾 and this one's the disk cap

💅 classmethod supportsAutoDeployment()

Okay so this tells you whether the batch system can auto-deploy the user script itself 🚀 If it can, you're allowed to call setUserScript() to hand over the resource object for the user script ✨

Note to implementors: If your implementation returns True here, it should also override

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

🦋 classmethod supportsWorkerCleanup()

This one checks whether the batch system calls BatchSystemSupport.workerCleanup() after the last job for a workflow run finishes 🧹 Quick vibe check on terms: a worker means a whole node, not just a worker process. A worker process can run several jobs one after another, and more than one worker process can be running on the same node for the same workflow 🌸 The batch system is said to shut down once the last worker process terminates.

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

🧚 class ExecutorInfo(nodeAddress, agentId, nodeInfo, lastSeen)

Parameters:

• nodeAddress (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• agentId (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• nodeInfo (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• lastSeen (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Attributes: nodeAddress, agentId, nodeInfo, lastSeen 💜

🌈 userScript = None

Type: toil.resource.Resource

🍓 Class attributes

These are all just sitting there on the class, no descriptions, just vibes:

• jobQueues
• mesos_endpoint
• mesos_name
• mesos_timeout = 60
• killedJobIds
• killJobIds
• intendedKill
• hostToJobIDs
• nodeFilter = []
• runningJobMap
• taskResources
• updatedJobsQueue
• driver = None
• frameworkId = None
• executors
• agentsByID
• nonPreemptibleNodes
• executor
• lastTimeOfferLogged = 0
• logPeriod = 30
• ignoredNodes

🎀 setUserScript(userScript)

Set the user script for this workflow ✨ This has to be called before the first job is issued to this batch system, and it only works if supportsAutoDeployment() returns True. Otherwise it'll raise an exception 💀

Parameters:

• userScript (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the resource object representing the user script or module, plus the modules it depends on

Return type: None

🛑 ignoreNode(nodeAddress)

This tells the batch system to stop sending jobs to a node 🙅‍♀️ It's used in autoscaling, when the autoscaler is ready to kill a node but jobs are still running on it. Ignoring it lets the node get terminated once its current jobs wrap up.

Parameters:

• nodeAddress (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — IP address of the node to ignore

Return type: None

✅ unignoreNode(nodeAddress)

Stop ignoring this address 🫶 This is presumably what happens after a node with this address has been terminated. It leaves room for a brand-new node to pop up with the same address as one that got terminated.

Parameters:

• nodeAddress (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

🚀 issueBatchJob(command, jobNode, job_environment=None)

This issues the command you give it and returns a unique jobID 🎯 command is the string to run. The memory argument is an int for how many bytes the job needs, and cores is how many CPUs the job needs. The error-file is the path where any std-err/std-out gets stashed 🗂️

Parameters:

• command (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• jobNode (toil.job.JobDescription)
• job_environment (Optional[dict[str, str]], with /usr/share/doc/python3/html/library/stdtypes.html#dict for the dict and /usr/share/doc/python3/html/library/stdtypes.html#str for both str parts)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

☠️ killBatchJobs(jobIDs)

Kills the given job IDs, no mercy 💅 Once it returns, the killed jobs won't show up in getRunningBatchJobIDs anymore, and they won't come back from getUpdatedBatchJob either.

Parameters:

• jobIDs — list of IDs of the jobs to kill

📋 getIssuedBatchJobIDs()

Gets all the jobs that are currently issued 📝

Returns: a list of jobs (as job ID numbers) that are currently issued. They might be running, or just waiting to run. Even though it's a list, don't depend on the ordering, bestie 🙈

🏃 getRunningBatchJobIDs()

Gets a map of the job ID numbers that are currently running (not just waiting) along with how long each one has been running, in seconds ⏱️

Returns: a dictionary with the currently running job ID numbers as keys, and how many seconds they've been running as the values.

🎁 getUpdatedBatchJob(maxWait)

Returns info about a job that has updated its status, meaning it stopped running either successfully or with an error 🏁 Each job only gets returned exactly once, so no double-dipping.

It doesn't return info for jobs killed by killBatchJobs, though those might cause None to come back sooner than maxWait.

Parameters:

• maxWait — the number of seconds to block while waiting for a result

Returns: if a result is available, you get an UpdatedBatchJobInfo back. Otherwise you get None. The wallTime is the number of seconds (a strictly positive float) of wall-clock time the job ran for, or None if this batch system doesn't support tracking wall time ⏳

🔍 nodeInUse(nodeIP)

This checks whether a worker node is running any tasks 👀 If the node doesn't exist, it should simply return False.

Parameters:

• nodeIP (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the worker node's private IP address

Returns: True if the worker node has been issued any tasks, otherwise False.

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

⏲️ getWaitDuration()

Gets the wait period (a floating point number, in seconds) between checks for missing or overlong jobs 🕰️

🌙 shutdown()

Called at the end of a toil invocation 🌙 It should cleanly terminate all worker threads, no messy leftovers.

Return type: None

🤝 registered(driver, frameworkId, masterInfo)

Invoked when the scheduler successfully registers with a Mesos master 🎉

🚨 error(driver, message)

Invoked when Mesos reports an unrecoverable error 😭

🎟️ resourceOffers(driver, offers)

Invoked when resources have been offered to this framework 🎁

📬 statusUpdate(driver, update)

Invoked when the status of a task changes. For example, an agent gets lost and takes its task with it, or a task finishes and an executor sends a status update about it, etc. ✨

Important: returning from this callback acknowledges receipt of the status update! If the scheduler aborts during this callback (or the process exits), another status update will be delivered. Just know that this is currently not true if the agent sending the status update gets lost or fails during that time 💀

💌 frameworkMessage(driver, executorId, agentId, message)

Invoked when an executor sends a message 💬

🔎 getNodes(preemptible=None, timeout=None)

Returns all nodes that match:

• preemptible status (None includes all)
• timeout period (seen within the last # seconds, or None for all)

Parameters:

• preemptible (Optional[bool], /usr/share/doc/python3/html/library/functions.html#bool)
• timeout (Optional[int], /usr/share/doc/python3/html/library/functions.html#int)

💖 reregistered(driver, masterInfo)

This one fires when the scheduler reconnects to a newly elected Mesos master, so it's basically Toil going "wait, we're back?" 👑

🌸 executorLost(driver, executorId, agentId, status)

Invoked when an executor has exited or terminated abnormally. Honestly it's giving "something went very wrong" 😭

🦋 get_default_mesos_endpoint()

Classmethod. This grabs the default IP/hostname and port that Toil will look for Mesos at 🧚

Return type: str

✨ add_options(parser)

Classmethod. If this batch system has any command line options, it tosses them into the parser you give it 🎀

Parameters:

• parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup])

Return type: None

💅 setOptions(setOption)

Classmethod. Processes the command line or configuration options that matter to this batch system 💜

Parameters:

• setOption (toil.batchSystems.options.OptionSetter) — basically a helper function with the signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None). It returns nothing and quietly updates the run configuration as a side effect 🔥

---

🧚 toil.batchSystems.mesos.conftest

Attributes

• collect_ignore

Module Contents

toil.batchSystems.mesos.conftest.collect_ignore = []

---

🌈 toil.batchSystems.mesos.executor

Attributes

• log

Classes

• MesosExecutor — Part of Toil's Mesos framework. It runs on a Mesos agent, and a Toil job gets passed to it through the task.data field and launched via call(toil.command) ✨

Functions

• main()

Module Contents

toil.batchSystems.mesos.executor.log

class toil.batchSystems.mesos.executor.MesosExecutor

Bases: pymesos.Executor

This is the star of the show: Toil's Mesos framework piece that runs on a Mesos agent. A Toil job is handed over via the task.data field, then launched via call(toil.command) 👑

Attributes:

• popenLock
• runningTasks
• workerCleanupInfo = None
• address = None
• id = None

Methods:

🦋 registered(driver, executorInfo, frameworkInfo, agentInfo)

Invoked once the executor driver has successfully connected with Mesos. We're in! 🎉

🌸 reregistered(driver, agentInfo)

Invoked when the executor re-registers with a restarted agent 🫶

💖 disconnected(driver)

Invoked when the executor becomes "disconnected" from the agent. This happens when the agent is restarting, like during an upgrade 😭

💅 killTask(driver, taskId)

Kills the parent task process along with all of its spawned children. Bye bye, everything 💀

🔥 shutdown(driver)

🚨 error(driver, message)

Invoked when a fatal error has occurred with the executor and/or the executor driver. Not it 💀

🧚 launchTask(driver, task)

Invoked by SchedulerDriver when a Mesos task should be launched by this executor 🌈

🎀 frameworkMessage(driver, message)

Invoked when a framework message has arrived for this executor 💜

toil.batchSystems.mesos.executor.main()

---

🦋 toil.batchSystems.mesos.test

Attributes

• log

Classes

• MesosTestSupport — A mixin for test cases that need a running Mesos master and agent on the local host 🍓

Package Contents

toil.batchSystems.mesos.test.log

class toil.batchSystems.mesos.test.MesosTestSupport

A mixin for test cases that need a running Mesos master and agent on the local host. It's the setup crew for your tests 🎬

💫 wait_for_master()

🌟 Nested class: MesosThread(numCores)

Bases: toil.lib.threading.ExceptionalThread

This is a thread whose join() method re-raises exceptions that were raised during run(). join() is idempotent, but the exception only gets re-raised on the first call to join() that successfully joins the thread. If join() times out, no exception gets re-raised, even if one might already have happened inside run(). 😭

When you subclass this thread, override tryRun() instead of run(). Don't say I didn't warn you 🙈

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Attributes:

• lock
• numCores

Methods:

🥺 mesosCommand()

Abstract method. Subclasses have to fill this in 🫠

🧠 tryRun()

🔍 findMesosBinary(names)

🌟 Nested class: MesosMasterThread(numCores)

Bases: MesosThread

This is a thread whose join() method re-raises exceptions that were raised during run(). join() is idempotent, but the exception only gets re-raised on the first call to join() that successfully joins the thread. If join() times out, no exception gets re-raised, even if one might already have happened inside run(). 😭

When you subclass this thread, override tryRun() instead of run(). Don't say I didn't warn you 🙈

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Methods:

👑 mesosCommand()

🌟 Nested class: MesosAgentThread(numCores)

Bases: MesosThread

This is a thread whose join() method re-raises exceptions that were raised during run(). join() is idempotent, but the exception only gets re-raised on the first call to join() that successfully joins the thread. If join() times out, no exception gets re-raised, even if one might already have happened inside run(). 😭

When you subclass this thread, override tryRun() instead of run(). Don't say I didn't warn you 🙈

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Methods:

🫶 mesosCommand()

🌸 toil.batchSystems.mesos

🦋 Classes

okay so the mesos module has a whole squad of classes 💅 (the chart is a bit messy, so here's the lineup):

• TaskData: a tuple-based class for task info 🎀
• JobQueue: the job line-up, sorted by type 🫶
• MesosShape: represents a job or a node's "shape", aka how much memory, cores, disk and wall-time it needs or gets 🔥
• ToilJob: another tuple-based class, for the job itself 🧚

✨ Package Contents ✨

🧁 class toil.batchSystems.mesos.TaskData

Bases: tuple (see /usr/share/doc/python3/html/library/stdtypes.html#tuple)

Its attributes are:

• startTime
• agentID
• agentIP
• executorID
• memory
• cores

🍓 class toil.batchSystems.mesos.JobQueue

Its attributes are:

• queues
• sortedTypes = []
• jobLock

And its methods are:

• insertJob(job, jobType)
• jobIDs()
• nextJobOfType(jobType)
• typeEmpty(jobType)

💜 class toil.batchSystems.mesos.MesosShape

class toil.batchSystems.mesos.MesosShape(wallTime, memory, cores, disk,
preemptible)

Bases: toil.provisioners.abstractProvisioner.Shape (see #toil.provisioners.abstractProvisioner.Shape)

Represents a job or a node's "shape", in terms of the dimensions of memory, cores, disk and wall-time allocation. Main character energy 👑

The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs? 🙈

The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively. 💖

Parameters

• wallTime (Union[int, float])
• memory (int)
• cores (Union[int, float])
• disk (int)
• preemptible (bool)

🫧 __gt__(other)

This one is inverted 🙃 It returns True if self is less than other, else it returns False.

The reason: jobTypes are sorted in decreasing order, and that was done on purpose so that expensive jobs get priority. Iconic. 👑

🌈 class toil.batchSystems.mesos.ToilJob

Bases: tuple (see /usr/share/doc/python3/html/library/stdtypes.html#tuple)

Its attributes are:

• jobID
• name
• resources
• command
• userScript
• environment
• workerCleanupInfo

---

🧚 toil.batchSystems.options

🫧 Attributes

• logger

🎀 Classes

• OptionSetter: a Protocol for the setOption function. It's how we get to set up CLI options for each batch system 💅 The actual functionality lives in the Config class.

🌷 Functions

• set_batchsystem_options(batch_system, set_option): calls set_option for all the options for the given named batch system, or for all batch systems if no name is provided 🦋
• add_all_batchsystem_options(parser): adds the options to the parser 🫶

✨ Module Contents ✨

💖 toil.batchSystems.options.logger

🔥 class toil.batchSystems.options.OptionSetter

Bases: Protocol

This is the Protocol for the setOption function we get to let us set up CLI options for each batch system. Actual functionality is defined in the Config class. ✨

Its attribute is:

• OptionType

Its method is:

__call__(option_name, parsing_function=None, check_function=None,
default=None, env=None, old_names=None)

Parameters

• option_name (str)
• parsing_function (Optional[Callable[[Any], OptionType]])
• check_function (Optional[Callable[[OptionType], Union[None, bool]]])
• default (Optional[OptionType])
• env (Optional[list[str]])
• old_names (Optional[list[str]])

Return type

bool

🌸 toil.batchSystems.options.set_batchsystem_options

toil.batchSystems.options.set_batchsystem_options(batch_system,
set_option)

Calls set_option for all the options for the given named batch system, or for all batch systems if no name is provided. 🌈

Parameters

• batch_system (Optional[str])
• set_option (OptionSetter)

Return type

None

🧁 toil.batchSystems.options.add_all_batchsystem_options

toil.batchSystems.options.add_all_batchsystem_options(parser)

Parameters

• parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup])

Return type

None

---

🍓 toil.batchSystems.registry

🫶 Attributes

• logger
• DEFAULT_BATCH_SYSTEM

✨ Functions ✨

okay so this is the quick cheat sheet for the batch system registry 💅 these are the functions and factories you can call, no cap

• add_batch_system_factory(key, class_factory) — adds a batch system to the registry, whether it's a workflow one or a plugin-supplied one 🧚
• get_batch_systems() — gets the names of every batch system that's available 🌸
• get_batch_system(key) — grabs a batch system class by its name ✨
• aws_batch_batch_system_factory() 🦋
• gridengine_batch_system_factory() 🦋
• lsf_batch_system_factory() 🦋
• single_machine_batch_system_factory() 🦋
• mesos_batch_system_factory() 🦋
• slurm_batch_system_factory() 🦋
• torque_batch_system_factory() 🦋
• htcondor_batch_system_factory() 🦋
• kubernetes_batch_system_factory() 🦋

---

🌈 Module Contents 🌈

okay the module contents are where it's at, bestie 🫶

• toil.batchSystems.registry.logger

💖 toil.batchSystems.registry.add_batch_system_factory(key, class_factory) 💖

this is the one that adds a batch system to the registry for workflow or plugin-supplied batch systems 🔥

• Parameters
  • class_factory (Callable[[], type[toil.batchSystems.abstractBatchSystem.AbstractBatchSystem]], see /usr/share/doc/python3/html/library/functions.html#type) — a function that returns a batch system class (NOT an instance), and that class implements toil.batchSystems.abstractBatchSystem.AbstractBatchSystem 💅
  • key (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) 🌸

🌸 toil.batchSystems.registry.get_batch_systems() 🌸

gets you the names of all the available batch systems, ngl so useful 🧚

• Return type
  • collections.abc.Sequence[str] (see /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Sequence and /usr/share/doc/python3/html/library/stdtypes.html#str)

🦋 toil.batchSystems.registry.get_batch_system(key) 🦋

grabs a batch system class by name 👑

• Raises KeyError if the key isn't the name of a batch system, and ImportError if the batch system's class can't be loaded 💀
• Parameters
  • key (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) 🫶
• Return type
  • type[toil.batchSystems.abstractBatchSystem.AbstractBatchSystem] (see /usr/share/doc/python3/html/library/functions.html#type) ✨

🍓 toil.batchSystems.registry.DEFAULT_BATCH_SYSTEM 🍓

the default is 'single_machine', so if you don't pick one, that's the one you get 🌈

• toil.batchSystems.registry.DEFAULT_BATCH_SYSTEM = 'single_machine'

🧚 Factory functions 🧚

all of these are just factory functions for their own batch system, and none of them have a description in the original page, so here they are, serving face 💅

• toil.batchSystems.registry.aws_batch_batch_system_factory()
• toil.batchSystems.registry.gridengine_batch_system_factory()
• toil.batchSystems.registry.lsf_batch_system_factory()
• toil.batchSystems.registry.single_machine_batch_system_factory()
• toil.batchSystems.registry.mesos_batch_system_factory()
• toil.batchSystems.registry.slurm_batch_system_factory()
• toil.batchSystems.registry.torque_batch_system_factory()
• toil.batchSystems.registry.htcondor_batch_system_factory()
• toil.batchSystems.registry.kubernetes_batch_system_factory()

---

🦋 toil.batchSystems.singleMachine 🦋

this is the single machine batch system module, the main character of this section, periodt 👑

🌸 Attributes 🌸

• logger (no description given, just vibes 🫶)

💜 Classes 💜

• SingleMachineBatchSystem — the interface for running jobs on a single machine. it runs all the jobs you give it, but the description just cuts off right there at "runs all the jobs you" 🙈
• Info — a record for a running job 🦋

🌈 Module Contents 🌈

• toil.batchSystems.singleMachine.logger

💖 class toil.batchSystems.singleMachine.SingleMachineBatchSystem(config, maxCores, maxMemory, maxDisk, max_jobs=None) 💖

Bases: toil.batchSystems.abstractBatchSystem.BatchSystemSupport (see /usr/share/doc/python3/html/library/functions.html#float and friends for the type links)

the interface for running jobs on a single machine, which runs all the jobs you give it as they come in, but in parallel. it's giving multitasking 🔥

it uses a single "daddy" thread to manage a fleet of child processes 🧚

communication with the daddy thread happens through two queues: one queue of jobs waiting to be run (the input queue), and one queue of jobs that are finished or stopped and need to be returned by getUpdatedBatchJob (the output queue) 📬

when the batch system is shut down, the daddy thread is stopped 🛑

if it's running in debug-worker mode, jobs run immediately as they're sent to the batch system, in the sending thread, and the daddy thread doesn't run at all. the queues are still used though 🙈

• Parameters
  • config (toil.common.Config) 🫶
  • maxCores (float, see /usr/share/doc/python3/html/library/functions.html#float) 💅
  • maxMemory (float, see /usr/share/doc/python3/html/library/functions.html#float) 💅
  • maxDisk (int, see /usr/share/doc/python3/html/library/functions.html#int) 💅
  • max_jobs (Optional[int], see /usr/share/doc/python3/html/library/functions.html#int) 💅

🦋 classmethod supportsAutoDeployment() 🦋

tells you whether this batch system can auto-deploy the user script itself 🚀

if it can, setUserScript() can be invoked to set the resource object that represents the user script ✨

Note to implementors: if your implementation returns True here, it should also override (the note cuts off right there, so keep that in mind 💀)

🌸 classmethod supportsWorkerCleanup() 🌸

tells you whether this batch system supports worker cleanup 🧹

it shows whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. quick reminder that "worker" here means an entire node, not just a worker process. a worker process can run more than one job sequentially, and more than one concurrent worker process can exist on a worker node for the same workflow. the batch system is said to shut down after the last worker process terminates 👑

💜 Attributes 💜

• numCores 💅
• minCores = 0.1 — the minimal fractional CPU. tasks asking for fewer cores get rounded up to this value 🌈
• physicalMemory 💅
• config 💅
• physicalDisk 💅
• scale 💅
• debugWorker 💅
• jobIndex = 1 💅
• jobIndexLock 💅
• jobs: dict[int, toil.job.JobDescription] (see /usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/functions.html#int) 📦
• inputQueue 💅
• outputQueue 💅
• runningJobs: dict[int, Info] 🌸 this is the VIP list of jobs that are running right now, keyed by job ID. Each value is an Info record. Type docs: dict lives at /usr/share/doc/python3/html/library/stdtypes.html#dict, int at /usr/share/doc/python3/html/library/functions.html#int, and Info at #toil.batchSystems.singleMachine.Info.
• children: dict[int, subprocess.Popen] 👶 the child processes, keyed by int. Type docs: dict at /usr/share/doc/python3/html/library/stdtypes.html#dict, int at /usr/share/doc/python3/html/library/functions.html#int, and subprocess.Popen at /usr/share/doc/python3/html/library/subprocess.html#subprocess.Popen.
• childToJob: dict[int, str] 🦋 maps each int to a str. Type docs: dict at /usr/share/doc/python3/html/library/stdtypes.html#dict, int at /usr/share/doc/python3/html/library/functions.html#int, and str at /usr/share/doc/python3/html/library/stdtypes.html#str.
• accelerator_identities ✨ just the name, no type or default given on the page.
• resource_sources 💅 just the name, no type or default given on the page.
• schedulingStatusMessage = None 🔥 starts out as None.
• shuttingDown 🙈 just the name, no type or default given on the page.
• daddyThread = None 👑 the daddy thread lives here, and it starts as None.
• daddyException: Exception | None = None 💀 can be an Exception or None, and it starts as None. Type docs: Exception at /usr/share/doc/python3/html/library/exceptions.html#Exception and None at /usr/share/doc/python3/html/library/constants.html#None.

---

✨ daddy()

Be the "daddy" thread. 👑 Our job is to look at jobs from the input queue.

If a job fits in the available resources, we allocate resources for it and kick off a child process. 🚀

We also check on our children. 👀

When a child finishes, we reap it, release its resources, and put its information in the output queue. 🫶

---

💖 getSchedulingStatusMessage()

Get a log message fragment for the user about anything that might be going wrong in the batch system, if one is available. 🔍

If no useful message is available, return None. 🙈

This can be used to report what resource is the limiting factor when scheduling jobs, for example. 🧐 If the leader thinks the workflow is stuck, the message can be displayed to the user so they can diagnose why it might be stuck. 🌸

Returns

User-directed message about scheduling state.

---

🦋 check_resource_request(requirer)

Check that a resource request is not greater than what's available or allowed. 🛡️

Parameters

• requirer (toil.job.Requirer, see #toil.job.Requirer) — the object whose requirements are being checked
• job_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the job being checked, used to generate a useful error report
• detail (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — a batch-system-specific message to include in the error

Raises

InsufficientSystemResources (see #toil.batchSystems.abstractBatchSystem.InsufficientSystemResources) — raised when a resource is requested in an amount greater than allowed 💀

Return type

None

---

🧚 issueBatchJob(command, job_desc, job_environment=None)

Adds the command and resources to a queue so it can be run. 🏃‍♀️

Parameters

• command (str)
• job_desc (toil.job.JobDescription, see #toil.job.JobDescription)
• job_environment (Optional[dict[str, str]])

Return type

int

---

🍓 killBatchJobs(jobIDs)

Kills jobs by ID. ☠️

Parameters

jobIDs (list[int])

Return type

None

---

🌈 getIssuedBatchJobIDs()

Just returns all the jobs that have been run but not yet returned as updated. 📦

Return type

list[int]

---

⭐ getRunningBatchJobIDs()

Gets a map of job ID numbers for jobs that are currently running (not just waiting), along with how long each has been running, in seconds. ⏱️

Returns

A dictionary with currently running job ID number keys and how many seconds they have been running as the value.

Return type

dict[int, float]

---

🛑 shutdown()

Terminate cleanly and join the daddy thread. 🫡

Return type

None

---

💜 getUpdatedBatchJob(maxWait)

Returns a tuple of a no-longer-running job, the return value of its process, and its runtime, or None. 🎀

Parameters

maxWait (int)

Return type

Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo] (see #toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo)

---

🌸 classmethod add_options(parser)

If this batch system provides any command line options, add them to the given parser. 💅

Parameters

parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup])

Return type

None

---

🔥 classmethod setOptions(setOption)

Process command line or configuration options relevant to this batch system. ⚙️

Parameters

setOption (toil.batchSystems.options.OptionSetter, see #toil.batchSystems.options.OptionSetter) — a function with the signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) that returns nothing and is used to update the run configuration as a side effect. 🧁

---

🦋 toil.batchSystems.singleMachine.Info(startTime, popen, resources, killIntended)

Record for a running job. 📝

It stores the start time of the job, the Popen object representing its child (or None), the tuple of (coreFractions, memory, disk) it is using (or None), and whether the job is supposed to be being killed. 🌙

• time 🕰️
• popen 🍭
• resources 🎁
• killIntended 🛑

---

✨ toil.batchSystems.slurm

🌸 Attributes

• logger
• TERMINAL_STATES
• NONTERMINAL_STATES

🧁 Classes

• SlurmBatchSystem — a partial implementation of BatchSystemSupport for batch systems run on a

🫧 Functions

• parse_slurm_time(slurm_time) — Parse a Slurm-style time duration like 7-00:00:00 to a number of seconds. ⏰
• option_detector(long[, short]) — get a function that returns true if it sees the long or short
• any_option_detector(options) — get a function that returns true if it sees any of the long

💫 Module Contents

• toil.batchSystems.slurm.logger
• toil.batchSystems.slurm.TERMINAL_STATES: dict[str, toil.batchSystems.abstractBatchSystem.BatchJobExitReason] (see #toil.batchSystems.abstractBatchSystem.BatchJobExitReason, plus dict at /usr/share/doc/python3/html/library/stdtypes.html#dict and str at /usr/share/doc/python3/html/library/stdtypes.html#str)
• toil.batchSystems.slurm.NONTERMINAL_STATES: set[str] (set at /usr/share/doc/python3/html/library/stdtypes.html#set and str at /usr/share/doc/python3/html/library/stdtypes.html#str)
• toil.batchSystems.slurm.parse_slurm_time(slurm_time) — Parse a Slurm-style time duration like 7-00:00:00 to a number of seconds. Raises ValueError if not parseable. 🚨

Okay so the only input for this one is slurm_time (str) 🕰️, and it hands back an int 💅

Parameters

• slurm_time (str)

Return type

int

toil.batchSystems.slurm.option_detector(long, short=None) 🔍

Bestie, this one builds you a little checker function ✨. That checker returns True if it spots the long option or the short one.

Parameters

• long (str)
• short (str | None)

Return type

Callable[[str], bool]

toil.batchSystems.slurm.any_option_detector(options) 🦋

This is the upgraded version 👑. It gives you back a checker that returns True if it sees any of the long options, or any long or short option pair.

Parameters

• options (list[str | tuple[str, str]])

Return type

Callable[[str], bool]

class toil.batchSystems.slurm.SlurmBatchSystem(config, maxCores, maxMemory, maxDisk) 🖥️

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

This is a partial implementation of BatchSystemSupport for batch systems that run on a standard HPC cluster 🌸. By default, auto-deployment is not implemented, so keep that in mind 🙈.

Parameters

• config (toil.common.Config)
• maxCores (float)
• maxMemory (float)
• maxDisk (float)

class PartitionInfo 🧁

Bases: NamedTuple

This is the main character of the partition world. Each one has these fields:

• partition_name: str
• gres: bool
• time_limit: float
• priority: int
• cpus: str
• memory: str

class PartitionSet 🍓

This is the set of available partitions that Toil detects on the Slurm batch system 👀.

• default_gpu_partition: SlurmBatchSystem | None
• all_partitions: list[SlurmBatchSystem]
• gpu_partitions: set[str]

get_partition(time_limit) ⏱️

Gets the partition name to use for a job with the given time limit. It's giving precision 💖.

Parameters

• time_limit (float | None) — the time limit in seconds.

Return type

str | None

class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) 🧵

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread

This is a class that represents a thread of control 🧵💜. It can be safely subclassed, but only in a limited fashion, so don't go wild 😭. There are two ways to set up the activity: pass a callable object to the constructor, or override the run() method in a subclass.

Parameters

• newJobsQueue (queue.Queue)
• updatedJobsQueue (queue.Queue)
• killQueue (queue.Queue)
• killedJobsQueue (queue.Queue)
• boss (AbstractGridEngineBatchSystem)

boss: SlurmBatchSystem

getRunningJobIDs() 🏃

Gets a list of running job IDs. This is implementation-specific, and it gets called by the boss AbstractGridEngineBatchSystem implementation through AbstractGridEngineBatchSystem.getRunningBatchJobIDs() 🫶.

Return type

list

killJob(jobID) 💀

Kills a specific job using the Toil job ID. This is implementation-specific too, and it gets called by GridEngineThread.killJobs() 🔥.

Parameters

• jobID (string) — the Toil job ID.

Return type

None

prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None) 🎀

This is the prep work 💅 for putting together a command-line string that gets submitted to the batch system (through submitJob()).

• int cpu
• int memory
• int jobID — the Toil job ID.
• string subLine — the command line string to be called.
• string jobName — the name of the Toil job, which provides metadata to batch systems if they want it 🧚.
• dict job_environment — the environment variables to be set on the worker.

Return type

List[str]

Parameters

• cpu (int)
• memory (int)
• jobID (int)
• command (str)
• jobName (str)
• job_environment (dict[str, str] | None)
• gpus (int | None)

submitJob(subLine) 🚀

This is a wrapper routine. It submits the actual command-line call, then processes the output to get the batch system job ID 🎀.

• string subLine — the literal command line string to be called.

Return type

string: the batch system job ID, which will be stored internally 🌈.

Parameters

• subLine (list[str])

coalesce_job_exit_codes(batch_job_id_list) 👑

This one collects all the job exit codes in a single call. It's so efficient, I'm obsessed 😭✨.

Parameters

• batch_job_id_list (list[str]) — a list of Job ID strings, where each string has the form <job>[.<task>].

Returns

A list of job exit codes, or exit code and exit reason pairs, associated with the list of job IDs.

Raises

• CalledProcessErrorStderr — if communicating with Slurm went wrong. 💔
• OSError — 😭 this one pops up when the job details just aren't available because a Slurm command couldn't even start. Rough day for Slurm 💀

Return type 🧾 list[int | tuple[int, toil.batchSystems.abstractBatchSystem.BatchJobExitReason | None] | None] | None

✨ getJobExitCode(batchJobID)

Okay so this one is the exit code getter 💅 You hand it a batch job ID and it gives you back the job's exit code as an integer. ✨

Parameters

• batchJobID (str) — the ID, which is a string shaped like "<job>[.<task>]" 🎀

Return type 🧾 int | tuple[int, toil.batchSystems.abstractBatchSystem.BatchJobExitReason | None] | None

✨ prepareSbatch(cpu, mem, jobID, jobName, job_environment, gpus)

This one builds the sbatch command line you run to queue up the job 🌸 Basically the prep work before the job gets sent off.

Parameters

• cpu (int) 💖
• mem (int) 💖
• jobID (int) 💖
• jobName (str) 💖
• job_environment (dict[str, str] | None) 💖
• gpus (int | None) 💖

Return type 🧾 list[str]

🧩 Attributes

• partitions 🦋
• start_time 🦋

✨ issueBatchJob(command, job_desc, job_environment=None)

This is the main event 🔥 It sends a job with your chosen command to the batch system and hands back a unique job ID number so you can keep track of it. Iykyk.

Parameters

• command (str) — the command to run somewhere in order to start the Toil worker process 🌈
• job_desc (toil.job.JobDescription) — the JobDescription for the job that's being run 🍓
• job_environment (dict[str, str] | None) — a bundle of job-specific environment variables that get set on the worker 🧚

Returns — a unique job ID number that you can use to reference the newly issued job 👑

Return type 🧾 int

✨ add_options(parser) (classmethod)

If this batch system comes with its own command line options, this is where they get added to the parser you hand over 🎀

Parameters

• parser (argparse.ArgumentParser | argparse._ArgumentGroup) 💜

Return type 🧾 None

🧩 Attributes

• OptionType 🦋

✨ setOptions(setOption) (classmethod)

This one processes the command line or config options that matter for this batch system. Main character energy 👑

Parameters

• setOption (toil.batchSystems.options.OptionSetter) — a function with the signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None). It returns nothing and quietly updates the run configuration as a side effect 🫶

Return type 🧾 None

---

🐘 toil.batchSystems.torque

🧩 Attributes

• logger 🌸

🎀 Classes

• TorqueBatchSystem — a partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster 💅

🧩 Module Contents

• toil.batchSystems.torque.logger ✨

✨ class TorqueBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

This is a partial implementation of BatchSystemSupport for batch systems that run on a standard HPC cluster. Heads up though: auto-deployment is not implemented by default 🙈

Parameters

• config (toil.common.Config) 💖
• maxCores (float) 💖
• maxMemory (float) 💖
• maxDisk (float) 💖

✨ class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread

This is a class that represents a thread of control 🧵 It's safe to subclass, but only in a limited way. You can set the activity in two ways: pass a callable object to the constructor, or override the run() method in a subclass.

getRunningJobIDs()

Gets a list of the running job IDs 📋 This is implementation-specific, and the boss AbstractGridEngineBatchSystem implementation calls it through AbstractGridEngineBatchSystem.getRunningBatchJobIDs().

Return type 🧾 list

getUpdatedBatchJob(maxWait)

killJob(jobID)

This kills one specific job using its Toil job ID 💀 It's implementation-specific, and GridEngineThread.killJobs() is the one who calls it.

Parameters

• jobID (string) — the Toil job ID 🎀

prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)

This is the prep work for putting together a command-line string that gets submitted to the batch system (via submitJob()) ✨

Parameters

• cpu (int) 💖
• memory (int) 💖
• jobID (int) — the Toil job ID 🦋
• command (str) 💖
• jobName (str) — the name of the Toil job, which gives metadata to batch systems if they want it 🌈

Param 🫶 int cpu

Param 🫶 int memory

Param 🫶 int jobID: Toil job ID

Param 🫶 string subLine: the command line string to be called

Param 🫶 string jobName: the name of the Toil job, to provide metadata to batch systems if desired

Param 🫶 dict job_environment: the environment variables to be set on the worker

Return type 🧾 List[str]

Parameters

• cpu (int) 💖
• memory (int) 💖
• jobID (int) 💖
• command (str) 💖
• jobName (str) 💖
• job_environment (Optional[dict[str, str]])
• gpus (Optional[int])
• submitJob(subLine) — the wrapper that sends off the actual command-line call, then digs through the output to snag the batch system job ID 🎀
  • Param string subLine — the literal command line string to be called 💌
  • Return type string — the batch system job ID, which gets stored internally ✨
• getJobExitCode(torqueJobID) — hands back the job's exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason 👑
  • Returns None if the job is still running 🫶
  • If the job isn't running but the exit code isn't available, you get EXIT_STATUS_UNAVAILABLE_VALUE 💀
  • This one's implementation-specific, and it's called by GridEngineThread.checkOnJobs() 🔥
  • The exit code will only be 0 if the job affirmatively succeeded. 😭
  • Parameters
    • batchjobID (string) — the batch system job ID 🌸
• prepareQsub(cpu, mem, jobID, job_environment) — bestie, this one just wants its parameters and return type 💅
  • Parameters
    • cpu (int)
    • mem (int)
    • jobID (int)
    • job_environment (Optional[dict[str, str]])
  • Return type list[str]
• generateTorqueWrapper(command, jobID) — a super simple script generator that just wraps the command you give it, and for now it goes to the default tempdir 🧚

🎀 Exceptions

• DeadlockException — thrown by the Leader or BatchSystem when a deadlock pops up because there aren't enough resources to run the workflow 💀

🌸 Package Contents

Here's the exception class, bestie 💖

exception toil.batchSystems.DeadlockException(msg)

• Bases: Exception
• Thrown by the Leader or BatchSystem when a deadlock is encountered due to insufficient resources to run the workflow 🌈
• Default message:

msg = 'Deadlock encountered: Uninferable'

• __str__() — stringifies the exception, message and all 🧃

🦋 toil.bus

Okay so toil.bus is all about message types and a message bus for coordinating the leader's components 💜

Historically, the Toil Leader was all functions calling other functions to "handle" stuff, and it got SO brittle. The handler functions had to run in exactly the right order, or it would get confused and do the wrong thing 😭

The MessageBus fixes that by loosely coupling everything. Components talk by sending messages instead of calling each other directly 💅

Say a job comes back from the batch system with a failed exit status. That gets turned into a message and sent to the bus. Every component that needs to react to it (like decrementing the retry count) listens for the messages it cares about. Adding a brand new component? Just plug it in, and it can receive and react to messages without messing with the existing components' ability to react to the same ones 🔥

Eventually, the different parts of the Leader could become their own separate objects 🌸

By default, messages stay entirely inside the Toil leader process. They aren't saved anywhere, not even in the JobStore 🙈

The MessageBus also doubles as an extension point! The leader can serialize its messages to a file (check out the --writeMessages option), and you can decode them with MessageBus.scan_bus_messages(), which is exactly what the Toil WES server backend does. Replay the messages and track their effects on job state, and you get an up-to-date picture of every job in a workflow: whether they're issued or running, and which ones have completely finished. That's info the JobStore doesn't persist, so it's giving main character energy 👑

The leader process's MessageBus instance is owned by the Toil leader, but the BatchSystem gets a chance to connect to it and send or listen for messages. Right now it doesn't have to send or receive anything. The Leader polls it through the BatchSystem API and generates the events itself. But a BatchSystem implementation may send extra events, like JobAnnotationMessage 🎀

Right now the MessageBus is built on pypubsub, so messages are always handled on one thread: the Toil leader's main loop thread. If other components send events, they get shipped over to that thread inside the MessageBus. Processes can talk to each other using MessageBus.connect_output_file() and MessageBus.scan_bus_messages() 🦋

💅 Attributes

• logger
• MessageType

✨ Classes ✨

okay bestie, here's the whole message bus crew 🫶 these classes basically tell toil what's happening to each job 💅

heads up: a few of these descriptions just trail off mid-sentence in the original docs, so I've kept them exactly as they are 😭

• Names — stores literally every kind of name a job can have 🎀
• JobIssuedMessage — dropped when a job gets issued to run on the batch system 🚀
• JobUpdatedMessage — dropped when a job is "updated" and ready for something to happen to it ✨
• JobCompletedMessage — dropped when a job is completed, whether it went well or not 🏁
• JobFailedMessage — dropped when a job has completely failed and won't be retried again 💀
• JobMissingMessage — dropped when a job goes missing: it should be in the batch system but isn't 🙈
• JobAnnotationMessage — dropped when extra info shows up, like an AWS Batch job ID from the 🧾
• ExternalBatchIdMessage — dropped when you're using a batch system, and it links the toil assigned batch ID to 🔗
• QueueSizeMessage — dropped to describe the size of the queue of jobs that are issued but not yet 📏
• ClusterSizeMessage — Produced by the Toil-integrated autoscaler describe the number of 📊
• ClusterDesiredSizeMessage — Produced by the Toil-integrated autoscaler to describe the number of 🎯
• MessageBus — holds messages that should cause jobs to change their scheduling states 🚦
• MessageBusClient — the base class for clients (inboxes and outboxes) of a message bus. Handles 💜
• MessageInbox — a buffered connection to a message bus that lets us receive messages 📬
• MessageOutbox — a connection to a message bus that lets us publish messages 📤
• MessageBusConnection — a two-way connection to a message bus. Buffers incoming messages until you 🔁
• JobStatus — records the status of a job 📝

---

✨ Functions ✨

okay now the little helper functions, and they're iconic 👑

• get_job_kind(names) — returns an identifying string for the job 🪪
• message_to_bytes(message) — converts a plain-old-data named tuple into a byte string 🫧
• bytes_to_message(message_type, data) — converts bytes from message_to_bytes back into a message of the given type 🔄
• replay_message_bus(path) — replays all the messages and works out what they mean for jobs 🎬
• gen_message_bus_path([tmpdir]) — returns a file path in tmp to store the message bus at 🗂️

---

✨ Module Contents ✨

the module itself is giving main character energy 🌟

• toil.bus.logger

class toil.bus.Names

Bases: NamedTuple

Stores all the kinds of name a job can have 🎀 each of these is a str:

• job_name: str
• unit_name: str
• display_name: str
• stats_name: str
• job_store_id: str

toil.bus.get_job_kind(names)

Return an identifying string for the job 🪪

The result may contain spaces.

Returns: Either the unit name, job name, or display name, which identifies the kind of job it is to toil. Otherwise "Unknown Job" in case no identifier is available.

Parameters:

• names (Names)

Return type: str

class toil.bus.JobIssuedMessage

Bases: NamedTuple

Produced when a job is issued to run on the batch system 🚀 fields:

• job_type: str
• job_id: str
• toil_batch_id: int

class toil.bus.JobUpdatedMessage

Bases: NamedTuple

Produced when a job is "updated" and ready to have something happen to it ✨ fields:

• job_id: str
• result_status: int

class toil.bus.JobCompletedMessage

Bases: NamedTuple

Okay so this one is for when a job wraps up, whether it slayed or flopped 💅✨ It gets sent out no matter what the outcome was:

• job_type: str
• job_id: str
• exit_code: int

💌 toil.bus.JobFailedMessage

Bases: NamedTuple 💀

This one is for when a job is completely cooked and won't be retried again 😭 Fields:

• job_type: str
• job_id: str

🫥 toil.bus.JobMissingMessage

Bases: NamedTuple 🙈

Sent when a job goes missing. It's supposed to be in the batch system, but it's just not there, bestie 👻 Fields:

• job_id: str

🏷️ toil.bus.JobAnnotationMessage

Bases: NamedTuple 🎀

This is extra info that goes along with a job, like an AWS Batch job ID from the AWSBatchBatchSystem. Fields:

• job_id: str
• annotation_name: str
• annotation_value: str

🔗 toil.bus.ExternalBatchIdMessage

Bases: NamedTuple 🦋

When you're using a batch system, this one links the batch ID Toil assigned to the batch system's own ID. That ID is whatever the local implementation returns, like a PID, a batch ID, etc. Fields:

• toil_batch_id: int
• external_batch_id: str
• batch_system: str

📏 toil.bus.QueueSizeMessage

Bases: NamedTuple 🌸

This tells you how big the queue is for jobs that have been issued but aren't finished yet. Theoretically, you can recover this from other messages, so it's giving redundant but useful. Fields:

• queue_size: int

🐑 toil.bus.ClusterSizeMessage

Bases: NamedTuple 🌈

The Toil-integrated autoscaler sends this one to describe how many instances of a certain type are in a cluster. Fields:

• instance_type: str
• current_size: int

🔮 toil.bus.ClusterDesiredSizeMessage

Bases: NamedTuple 🧚

Also from the Toil-integrated autoscaler! This one says how many instances of a certain type it thinks it's going to need. Fields:

• instance_type: str
• desired_size: int

---

🧳 toil.bus.message_to_bytes(message)

This takes a plain-old-data named tuple and turns it into a byte string. Easy peasy 🍓

Parameters

• message (NamedTuple)

Return type

• bytes

🏷️ toil.bus.MessageType

Just a bare entry with nothing else attached. It's a type placeholder, so there's no drama here 💅

🔁 toil.bus.bytes_to_message(message_type, data)

This is the undo button for message_to_bytes. It converts bytes back into a message of the type you give it ✨

Parameters

• message_type (type[MessageType])
• data (bytes)

Return type

• MessageType

🚌 class toil.bus.MessageBus

Bases: none. This is the main hub for messages that make jobs change their scheduling states 🚌💖 Messages get put in and buffered, and you can take them out and handle them in batches whenever it's convenient. Every message is a NamedTuple object of some subtype. Message order is guaranteed to be preserved within a type, so you can trust it 🫶

publish(message) Puts a message onto the bus. You can call this from any thread, no cap 🧵

• Parameters: message (Any)
• Return type: None

check() If we're in the owning thread, this delivers any messages sitting in the queue for us. You need to call it every once in a while in the main thread, and it can happen through inbox objects too 👀

• Return type: None

MessageType Another bare placeholder. Nothing to see here, babe 🌷

subscribe(message_type, handler) Registers the callable you give it so it gets called whenever messages of the given type are sent. It only gets messages sent after the subscription is created. This returns a subscription object, and when that object gets garbage collected, the subscription ends 🗑️

• Parameters:
  • message_type (type[MessageType])
  • handler (Callable[[MessageType], Any])
• Return type: pubsub.core.listener.Listener

connect(wanted_types) Gets you a connection object that works as an inbox for messages of the given types. Those messages pile up in the inbox until it's destroyed, and you can check it whenever you want 📥

• Parameters: wanted_types (list[type])
• Return type: MessageBusConnection

outbox() Gets you a connection object that's only for sending messages. Send-only energy 📤

• Return type: MessageOutbox

connect_output_file(file_path) Sends copies of all messages to the given output file 📁 It returns connection data that has to be kept alive for the connection to persist. That data is opaque, so you're not supposed to look at it, touch it, or do anything with it except store it somewhere or delete it.

⚠️ Hands off the connection data, bestie. Just store it or delete it. Looking inside or poking at it is not the vibe.

• Parameters: file_path (toil.lib.misc.FileDescriptorOrPath)
• Return type: Any

classmethod scan_bus_messages(stream, message_types) Gives you an iterator over all the messages of the given types in the given log stream, in order. Any trailing partial messages get discarded 🗂️

• Parameters:
  • stream (IO[bytes])
  • message_types (list[type[NamedTuple]])
• Return type: collections.abc.Iterator[Any]

🤝 class toil.bus.MessageBusClient

Bases: none. This is the base class for clients, which means inboxes and outboxes. It handles keeping a reference to the message bus so you don't have to 🫶

📥 class toil.bus.MessageInbox

Bases: MessageBusClient 📥

This is a buffered connection to a message bus that lets you receive messages. It holds incoming messages until you're ready for them. Heads up, though: it does not preserve ordering between messages of different types 🙃

count(message_type) Gets the number of pending messages of the given type 🔢

• Parameters: message_type (type)
• Return type: int

empty() Returns True if no messages are pending, and False otherwise 🫙

• Return type: bool

MessageType Bare placeholder again, we love a consistent girl 💅

for_each(message_type) Loops over all messages that are currently pending of the given type. Each one that gets handled without raising an exception will be removed 🧹

👀 Messages sent while this function is running won't be yielded by the current call. They'll have to wait for the next one.

• Parameters: message_type (type[MessageType])
• Return type: collections.abc.Iterator[MessageType]

📤 class toil.bus.MessageOutbox

Bases: MessageBusClient 📤

This is a connection to a message bus that lets you publish messages 📣

publish(message) Publishes the given message to the connected message bus. This exists so you don't have to store both the bus and your connection, which is honestly very thoughtful of it 💖

• Parameters: message (Any)
• Return type: None

🔄 class toil.bus.MessageBusConnection

Bases: MessageInbox, MessageOutbox 🔄

This is a two-way connection to a message bus. It buffers incoming messages until you're ready for them, and it lets you send messages too. Full receive-and-send main character energy ✨

📊 class toil.bus.JobStatus

This one records the status of a job 📊 When exit_code is -1, that means the job is either not observed or currently running. Fields:

• job_store_id: str
• name: str
• exit_code: int
• annotations: dict[str, str]
• toil_batch_id — an int, the Toil batch id 🎀
• external_batch_id — a str, the external batch id 🌸
• batch_system — a str, the batch system the job is running on 🦋
• __repr__() 💅
  • Return type: str
• is_running() 🏃‍♀️
  • Return type: bool

✨ toil.bus.replay_message_bus(path)

Okay so this one replays every message and figures out what each one means for jobs 🔁✨. We keep track of each job's state and name by its ID here 👀. We'd normally use a list of two items, but MyPy can't understand a list with items of multiple types, so we made a whole new class instead 🙄💅.

It returns a dictionary keyed by job_id, where each value is a dataclass called JobStatus 📦. A JobStatus holds everything we pulled off the message bus about a job 💖, including the job store id, the job's name, its exit code, any associated annotations, the Toil batch id, the external batch id, and the batch system the job is running on 🧚.

• path (toil.lib.misc.FileDescriptorOrPath)

Return type: dict[str, JobStatus]

✨ toil.bus.gen_message_bus_path(tmpdir=None)

This one gives you a file path in tmp to store the message bus at 🌈. Heads up bestie: whoever calls it is responsible for cleaning up the generated file 🧹⚠️.

The tmpdir argument lets you override which directory the message bus gets made in 🗂️. If you don't pass it, the standard tempfile order is used 🫶.

• tmpdir (Optional[str])

Return type: str

---

🧺 toil.common

Attributes

• UUID_LENGTH 🔢
• logger 📝

Exceptions

• ToilRestartException — Common base class for all non-exit exceptions. 🔁
• ToilContextManagerException — Common base class for all non-exit exceptions. 🙈

Classes

• Config — Class to represent configuration operations for a Toil workflow run ⚙️.
• Toil — A context manager that represents a Toil workflow 👑.
• ToilMetrics 📊

Functions

• get_default_config_path() — Get the default path where the Toil configuration file lives 📍.
• ensure_config(filepath) — If the config file at the filepath does not exist, create it 🛠️.
• generate_config(filepath) — Write a Toil config file to the given path ✍️.
• update_config(filepath, key, new_value) — Set the given top-level key to the given value in the given YAML config 🧾.
• parser_with_common_options([provisioner_options, ...]) — Get a command-line option parser for a Toil subcommand 💻.
• addOptions(parser[, jobstore_as_flag, cwl, wdl]) — Add all Toil command line options to a parser 🎀.
• getNodeID() — Return unique ID of the current node (host). The resulting string will be convertible to a uuid.UUID 🆔.
• cacheDirName(workflowID) 🗃️
• getDirSizeRecursively(dirPath) — This method will return the cumulative number of bytes occupied by the files 📏.
• getFileSystemSize(dirPath) — Return the free space, and total size of the file system hosting dirPath 💾.
• safeUnpickleFromStream(stream) 🥒

Module Contents

• toil.common.UUID_LENGTH = 32 🔢
• toil.common.logger 📝

✨ toil.common.get_default_config_path()

Get the default path where the Toil configuration file lives 📍. The file at that path won't necessarily exist, so don't panic if it's missing 🙈.

Return type: str

✨ class toil.common.Config

Class to represent configuration operations for a Toil workflow run ⚙️.

Here are the attributes, bestie 💅:

• logFile: str | None
• logRotating: bool
• cleanWorkDir: str
• max_jobs: int
• max_local_jobs: int
• manualMemArgs: bool
• run_local_jobs_on_workers: bool
• coalesceStatusCalls: bool
• mesos_endpoint — 💖 str | None
• mesos_framework_id — ✨ str | None
• mesos_role — 🎀 str | None
• mesos_name — 🦋 str
• kubernetes_host_path — 🌸 str | None
• kubernetes_owner — 💅 str | None
• kubernetes_service_account — 🫶 str | None
• kubernetes_pod_timeout — ⏱️ float
• kubernetes_privileged — 🔥 bool
• kubernetes_pod_security_context — 🔐 str | None
• kubernetes_security_context — 🔐 str | None
• tes_endpoint — 🌈 str
• tes_user — 🧚 str
• tes_password — 🙈 str
• tes_bearer_token — 👑 str
• aws_batch_region — ☁️ str | None
• aws_batch_queue — 🍓 str | None
• aws_batch_job_role_arn — 🎭 str | None
• scale — ⭐ float
• batchSystem — 🧃 str
• batch_logs_dir — 📂 str | None The backing scheduler gets asked (if it's able to) to save its logs into this directory, so the leader can go read them. Lowkey so helpful 🫶
• statePollingWait — 💜 float
• state_polling_timeout — ⌛ int
• disableAutoDeployment — 🚫 bool
• workflowID — 🆔 str | None This one is the main character 👑 It uniquely identifies the job store, and therefore the whole workflow. It's what lets you tell apart two back-to-back workflows that share the same self.jobStore, like when someone reuses a job store name after a previous run finished successfully and its job store got cleaned up. Iykyk 💅
• workflowAttemptNumber — 🔁 int
• jobStore — 🗄️ str
• logLevel — 📣 str
• colored_logs — 🌈 bool
• workDir — 📁 str | None
• coordination_dir — 🤝 str | None
• noStdOutErr — 🤐 bool
• stats — 📊 bool
• clean — 🧹 str | None
• clusterStats — 📈 str
• restart — 🔄 bool
• caching — 💾 bool | None
• symlinkImports — 🔗 bool
• moveOutputs — 🚚 bool
• symlink_job_store_reads — 🔗 bool
• provisioner — 🏗️ str | None
• nodeTypes — 🖥️ list[tuple[set[str], float | None]]
• minNodes — 📉 list[int]
• maxNodes — 📈 list[int]
• targetTime — 🎯 float
• betaInertia — 🌀 float
• scaleInterval — ⏲️ int
• preemptibleCompensation — ⚖️ float
• nodeStorage — 💽 int
• nodeStorageOverrides — 🗂️ list[str]
• metrics — 📏 bool
• assume_zero_overhead — 🫧 bool
• maxPreemptibleServiceJobs — 🦺 int
• maxServiceJobs — 🛠️ int
• deadlockWait — 🧊 float | int
• deadlockCheckInterval — 🔍 float | int
• defaultMemory — 🧠 int
• defaultCores — 🧮 float | int
• defaultDisk — 💿 int
• defaultPreemptible — 🫠 bool
• defaultAccelerators — 🚀 list[toil.job.AcceleratorRequirement]
• maxCores — 🧮 int
• maxMemory — 🧠 int
• maxDisk — 💿 int
• retryCount — 🔂 int
• stop_on_first_failure — 🛑 bool
• enableUnlimitedPreemptibleRetries — ♾️ bool
• doubleMem — 🥞 bool
• maxJobDuration — ⏳ int
• rescueJobsFrequency — 🚑 int
• job_store_timeout — ⏰ float
• maxLogFileSize — 📄 int
• writeLogs — ✍️ str
• writeLogsGzip — 🗜️ str
• writeLogsFromAllJobs — 📚 bool
• write_messages — 💌 str | None
• realTimeLogging — ⚡ bool
• publish_workflow_metrics — 📡 Literal['all'] | Literal['current'] | Literal['no'] | None
• environment — 🌍 dict[str, str]
• disableChaining — ⛓️ bool
• disableJobStoreChecksumVerification: bool 💖 is a true-or-false switch.
• sseKey: str | None 🔐 is some text, or nothing at all.
• servicePollingInterval: int 🧚 is a whole number.
• useAsync: bool ⭐ is a true-or-false switch.
• forceDockerAppliance: bool 🐳 is a true-or-false switch.
• statusWait: int ⏳ is a whole number.
• disableProgress: bool 🌈 is a true-or-false switch.
• readGlobalFileMutableByDefault: bool 🦋 is a true-or-false switch.
• debugWorker: bool 🙈 is a true-or-false switch.
• disableWorkerOutputCapture: bool 🎀 is a true-or-false switch.
• badWorker: float 🍓 is a decimal number.
• badWorkerFailInterval: float 🍓 is a decimal number.
• kill_polling_interval: int 💀 is a whole number.
• cwl: bool 🌸 is a true-or-false switch.
• memory_is_product: bool 👑 is a true-or-false switch.

set_from_default_config() 🌸

Okay so this one doesn't come with a description, just a return type.

• Return type: None

prepare_start() ✨

After the options are set, this gets everything ready for the very first start of a workflow. Main character setup, bestie 💅

• Return type: None

prepare_restart() 🔁

Before the restart options are set, this gets things ready for restarting a workflow. It sets up any execution-specific parameters and clears out any stale ones 🧹

• Return type: None

setOptions(options) 🎀

This one makes a config object out of the options object. Literally the whole point 🫶

• Parameters
  • options (argparse.Namespace)
• Return type: None

check_configuration_consistency() 🙈

Old checks that couldn't be squeezed into an action class for argparse. Vintage, but still doing the job 🕰️

• Return type: None

__eq__(other) 💜

• Parameters
  • other (object)
• Return type: bool

__hash__() 🔢

• Return type: int

---

toil.common.ensure_config(filepath) 🧚

If the config file at the filepath doesn't exist yet, this creates it. The parent directory should already exist before you call this, bestie 📁

It raises an error if the config file can't be created. It's safe to run at the same time in multiple processes. If this process is the one running the function, it will always see the config file existing with parseable contents, even if other processes are racing to create it. No process will ever see a new empty or partially-written config file 🫧

The caller should still check to make sure there isn't a preexisting empty file here, iykyk 👀

• filepath (str) — path to config file
• Returns: None
• Return type: None

toil.common.generate_config(filepath) 📝

This writes a Toil config file to the given path. It's safe to run simultaneously in multiple processes, and no process will see an empty or partially-written file at that path. Obsessed 😭

Set include to "cwl" or "wdl" to include cwl options and wdl options, respectively. Choose your fighter 💅

• filepath (str)
• Return type: None

toil.common.update_config(filepath, key, new_value) 🔧

This sets the given top-level key to the given value in the given YAML config file. It doesn't dramatically mess with comments or formatting, and it doesn't make a partially-written file visible to anyone. Gentle giant energy 🦋

• key (str) — the setting you want to change
• filepath (str)
• new_value (Union[str, bool, int, float])
• Return type: None

Heads up, bestie! The key has to be the command-line option name, NOT the destination variable name. Mixing those up is a whole mood 💀

toil.common.parser_with_common_options(provisioner_options=False, jobstore_option=True, prog=None, default_log_level=None) 🍓

This gets you a command-line option parser for a Toil subcommand. The parser only comes with the basic stuff (like version reporting and logging) that every Toil subcommand uses.

Toil Python workflows should use toil.job.Job.Runner.getDefaultArgumentParser() instead! That one makes sure you get all the important options for actually running a workflow. Don't skip it 🙅‍♀️

• provisioner_options (bool)
• jobstore_option (bool)
• prog (Optional[str])
• default_log_level (Optional[int])
• Return type: configargparse.ArgParser

toil.common.addOptions(parser, jobstore_as_flag=False, cwl=False, wdl=False) 🎀

This adds all the Toil command line options to a parser. It also gives you support for config files when you're using configargparse, and it checks and sets up the default config file while it's at it. Such a multitasker ✨

• jobstore_as_flag (bool) — makes the job store option a --jobStore flag instead of a required jobStore positional argument
• cwl (bool) — whether CWL options are expected. If they are, CWL options won't get suppressed
• wdl (bool) — whether WDL options are expected. If they are, WDL options won't get suppressed
• parser (argparse.ArgumentParser)
• Return type: None

toil.common.getNodeID() 🆔

This returns the unique ID of the current node (host). The resulting string can be converted into a uuid.UUID. Very precise, very iconic 💖

It tries several methods until one succeeds. The returned ID should be identical across calls from different processes on the same node, at least until the next OS reboot.

The last-resort method is uuid.getnode(), which in some rare OS configurations may return a random ID every time it's called. Honestly, this should never be reached on a Linux system, because reading from /proc/sys/kernel/random/boot_id gets tried first. If uuid.getnode() does get reached, it gets called twice, and an exception is raised if the two values aren't identical. Yikes 😭

• Return type: str

class toil.common.Toil(options, workflow_name=None, trs_spec=None) 👑

Bases: ContextManager[Toil]

This is a context manager that represents a Toil workflow. Specifically, it covers the batch system, the job store, and its configuration. It's giving main character 💅

• options (argparse.Namespace)
• workflow_name (Optional[str])
• trs_spec (Optional[str])

Attributes:

• config: Config
• options

__enter__() 🚪

This derives the configuration from the command line options. Then it loads the job store and, on a restart, consolidates the derived configuration with the one from the previous invocation of the workflow.

• Return type: Toil

__exit__(exc_type, exc_val, exc_tb) 🧹

This cleans up after a workflow invocation. Depending on the configuration, it deletes the job store. Clean girl energy, no cap 🫧

• exc_type (Optional[type[BaseException]])
• exc_val (Optional[BaseException])
• exc_tb (Optional[types.TracebackType])
• Return type: Literal[False]

start(rootJob) 🚀

Invoke a Toil workflow with the given job as the root for an initial run. Let's gooo 🔥

This one has to be called inside a with Toil(...) as toil: block, so no sneaking it in anywhere else 🙅‍♀️ And it shouldn't be called more than once for a workflow that hasn't finished yet, bestie, or things get messy 💀

Parameters

• rootJob (toil.job.Job) — the root job of the workflow 🌸

Returns

• The root job's return value ✨

Return type

• Any

---

restart()

Okay, this is your comeback move 🔄 If a workflow got interrupted, restart() brings it back to life.

Returns

• The root job's return value ✨

Return type

• Any

---

JOB_STORE_TYPES

This is the lineup of job store types you can work with 🧚

JOB_STORE_TYPES = ['file', 'aws', 'google']

---

classmethod getJobStore(locator)

This one is giving matchmaker energy 💘 You hand it a locator and it creates the right concrete job store implementation to match.

• locator (str) — the location of the job store that the new instance will represent 📍

Returns

• an instance of a concrete subclass of AbstractJobStore 💖

Return type

• toil.jobStores.abstractJobStore.AbstractJobStore

---

static parseLocator(locator)

Parses a job store locator into a type string and the data that implementation needs to connect 🔍 Heads up: it does not check whether the job store type is one of the valid ones, so it's not the bouncer 🙈

If the locator isn't written in the right syntax, this one raises RuntimeError 💀 Double-check your locator before you pass it in.

Parameters

• locator (str)

Return type

• tuple[str, str]

---

static buildLocator(name, rest)

The opposite vibe of parseLocator. This one builds a locator string out of its pieces 🧩

Parameters

• name (str)
• rest (str)

Return type

• str

---

classmethod canonical_locator(locator)

Turns a job store locator into one that works from any directory 🗺️ It also always spells out the explicit type of job store, so no guessing games 🫶

Parameters

• locator (str)

Return type

• str

---

classmethod resumeJobStore(locator)

Gets you back into an existing job store 🦋

Parameters

• locator (str)

Return type

• toil.jobStores.abstractJobStore.AbstractJobStore

---

static createBatchSystem(config)

Creates the batch system that your config asked for ⚙️ It's basically the setup crew for the job runner 👷‍♀️

• config (Config) — the current configuration

Returns

• an instance of a concrete subclass of AbstractBatchSystem 🔥

Return type

• toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

---

url_exists(src_uri)

Checks whether a URI actually exists 👀

Parameters

• src_uri (str)

Return type

• bool

---

importFile and import_file

These bring a file from a URL into the job store 📥 Here are all the signatures, the overloads included, so you can see what each version hands back 💅

importFile(srcUrl: str, sharedFileName: str, symlink: bool = True) -> None
importFile(srcUrl: str, sharedFileName: None = None, symlink: bool = True) -> toil.fileStores.FileID
import_file(src_uri: str, shared_file_name: str, symlink: bool = True, check_existence: bool = True) -> None
import_file(src_uri: str, shared_file_name: None = None, symlink: bool = True, check_existence: Literal[True] = True) -> toil.fileStores.FileID
import_file(src_uri: str, shared_file_name: None = None, symlink: bool = True, check_existence: bool = True) -> toil.fileStores.FileID | None

Imports the file at the given URL into the job store 🌈 By default, it returns None if the file does not exist.

• check_existence — if true, it raises FileNotFoundError when the file doesn't exist. If false, it returns None when the file doesn't exist 🙃

For the full tea, see toil.jobStores.abstractJobStore.AbstractJobStore.importFile() 📚

---

exportFile(jobStoreFileID, dstUrl)

The reverse of importing: this sends a file out of the job store 📤

Parameters

• jobStoreFileID (toil.fileStores.FileID)
• dstUrl (str)

Return type

• None

---

export_file(file_id, dst_uri)

Exports a file to the destination pointed at by the destination URL 🚀

For a full description, see toil.jobStores.abstractJobStore.AbstractJobStore.exportFile() 💜

Parameters

• file_id (toil.fileStores.FileID)
• dst_uri (str)

Return type

• None

---

static normalize_uri(uri, check_existence=False, dir_path=None)

Given a URI with no scheme, this makes it a properly quoted file: URI 🎀

Parameters

• check_existence (bool) — if set, it raises FileNotFoundError when a URI points to a local file that doesn't exist 👻
• dir_path (Optional[str]) — if specified, relative paths get interpreted relative to this directory instead of the current one 🗂️
• uri (str)

Return type

• str

---

static getToilWorkDir(configWorkDir=None)

Returns a path to a writable directory that holds the per-workflow directories 📁

This directory is always required to exist on a machine, even if the Toil worker hasn't run yet 🫠 If your workers and leader have different temp directories, you may need to set TOIL_WORKDIR.

Parameters

• configWorkDir (Optional[str]) — the value passed to the program using the --workDir flag 🍓

Returns

• Path to the Toil work directory, constant across all machines 🌍

Return type

• str

---

classmethod get_toil_coordination_dir(config_work_dir, config_coordination_dir)

Returns a path to a writable directory, which will live in memory if that's convenient 🧠 This is the one to use for file locking and coordination 🔒

Parameters

• config_work_dir (Optional[str]) — the value passed to the program using the --workDir flag ✨
• config_coordination_dir (Optional[str], linked to /usr/share/doc/python3/html/library/stdtypes.html#str) — 💖 whatever value got passed to the program with the --coordinationDir flag.
• workflow_id — 🧚 used when a tmpdir_prefix exists, so it can build full directory paths that are unique to each workflow.

Returns 🌸 the path to the Toil coordination directory. It should sit on a POSIX filesystem that lets you delete directories even when they still contain open files.

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🦋 static get_workflow_path_component(workflow_id)

Gets a filesystem-safe path component for a workflow. It stays consistent for all processes on a given machine, and it's different for all processes on different machines. Very reliable, very iconic. 👑

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the current Toil workflow.

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

💅 classmethod getLocalWorkflowDir(workflowID, configWorkDir=None)

Returns the directory where worker directories and the cache will live for this workflow on this machine. Main character energy. ✨

• configWorkDir (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str) — the value passed to the program with the --workDir flag.
• workflowID (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Returns 🌈 the path to the local workflow directory on this machine.

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🫶 classmethod get_local_workflow_coordination_dir(workflow_id, config_work_dir, config_coordination_dir)

Returns the directory where coordination files should go for this workflow on this machine. These include Toil's internal databases and the lock files for the machine. 🔒

If an in-memory filesystem is available, it gets used. Otherwise, it falls back to the local workflow directory, which might be on a shared network filesystem.

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the unique ID of the current workflow.
• config_work_dir (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str) — the value used for the work directory in the current Toil Config.
• config_coordination_dir (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str) — the value used for the coordination directory in the current Toil Config.

Returns 🍓 the path to the local workflow coordination directory on this machine.

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🎀 exception toil.common.ToilRestartException(message)

Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)

The common base class for all non-exit exceptions. 🙈

• message (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🎀 exception toil.common.ToilContextManagerException

Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)

Also the common base class for all non-exit exceptions. Same iconic energy. 🌸

---

🔥 class toil.common.ToilMetrics(bus, provisioner=None)

• bus (toil.bus.MessageBus, <#toil.bus.MessageBus>)
• provisioner (Optional[toil.provisioners.abstractProvisioner.AbstractProvisioner], <#toil.provisioners.abstractProvisioner.AbstractProvisioner>)

Attributes 💜

mtailImage = 'quay.io/ucsc_cgl/toil-mtail:9.1.2-355b72ba0e425619c0367562caf2a337078dba65-py3.13'
grafanaImage = 'quay.io/ucsc_cgl/toil-grafana:9.1.2-355b72ba0e425619c0367562caf2a337078dba65-py3.13'
prometheusImage = 'quay.io/ucsc_cgl/toil-prometheus:9.1.2-355b72ba0e425619c0367562caf2a337078dba65-py3.13'

• nodeExporterProc: subprocess.Popen[bytes] | None = None (subprocess.Popen at /usr/share/doc/python3/html/library/subprocess.html#subprocess.Popen, bytes at /usr/share/doc/python3/html/library/stdtypes.html#bytes, None at /usr/share/doc/python3/html/library/constants.html#None)

Methods 🧚

• startDashboard(clusterName, zone) — starts the dashboard. ✨
  • clusterName (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
  • zone (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• add_prometheus_data_source() — adds the Prometheus data source. 🦋
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• log(message) — logs a message. 💬
  • message (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• logClusterSize(m) — logs the cluster size. 📏
  • m (toil.bus.ClusterSizeMessage, <#toil.bus.ClusterSizeMessage>)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• logClusterDesiredSize(m) — logs the desired cluster size. 🎯
  • m (toil.bus.ClusterDesiredSizeMessage, <#toil.bus.ClusterDesiredSizeMessage>)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• logQueueSize(m) — logs the queue size. 📬
  • m (toil.bus.QueueSizeMessage, <#toil.bus.QueueSizeMessage>)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• logMissingJob(m) — logs a missing job. 🫥
  • m (toil.bus.JobMissingMessage, <#toil.bus.JobMissingMessage>)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• logIssuedJob(m) — logs an issued job. 🚀
  • m (toil.bus.JobIssuedMessage, <#toil.bus.JobIssuedMessage>)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• logFailedJob(m) — logs a failed job. 💀
  • m (toil.bus.JobFailedMessage, <#toil.bus.JobFailedMessage>)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• logCompletedJob(m) — logs a completed job. 🎉
  • m (toil.bus.JobCompletedMessage, <#toil.bus.JobCompletedMessage>)
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)
• shutdown() — shuts everything down. 🌙
  • Return type: None (/usr/share/doc/python3/html/library/constants.html#None)

---

🌷 toil.common.cacheDirName(workflowID)

Returns 🫧 the name of the cache directory.

• workflowID (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

📦 toil.common.getDirSizeRecursively(dirPath)

This one adds up the number of bytes taken up by files on disk in the directory and all of its subdirectories. 🧮

If it can't access a file or directory, whether because of insufficient permissions or because the file or directory was removed while the function was walking through it, the error is handled internally. You'll get back a lower bound on the size of the directory, which could even be 0. 🙈

• dirPath (toil.lib.misc.StrPath, <#toil.lib.misc.StrPath>) — a valid path to a directory or file.

Returns 🍓 the total size, in bytes, of the file or directory at dirPath.

Return type: int (/usr/share/doc/python3/html/library/functions.html#int)

---

🌊 toil.common.getFileSystemSize(dirPath)

Returns the free space and the total size of the filesystem that hosts dirPath. 💾

• dirPath (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — a valid path to a directory.

Returns 🫶 the free space and total size of the file system, as a pair.

Return type: tuple[int, int] (/usr/share/doc/python3/html/library/stdtypes.html#tuple, with int at /usr/share/doc/python3/html/library/functions.html#int)

---

🥞 toil.common.safeUnpickleFromStream(stream)

• stream (IO[Any])

Return type: Any

---

🌈 toil.cwl

Submodules 🧚

• toil.cwl.conftest

💖 toil.cwl.conftest

Attributes

• collect_ignore — empty right now. 🫧

Module Contents

toil.cwl.conftest.collect_ignore = []

🦋 toil.cwl.cwltoil

Implements support for the Common Workflow Language (CWL) in Toil. 🌸

✨ Attributes ✨

okay so this next bit is the attributes list, bestie 💅 the page just names them without giving any extra description, so here's the full lineup 🫶

• logger 🪵
• DEFAULT_TMPDIR 📂
• DEFAULT_TMPDIR_PREFIX 🌸
• MISSING_FILE 🙈
• V ⭐
• ProcessType 🔥
• T 🦋
• usage_message 💜

✨ Exceptions ✨

and now the exceptions, which are giving main character energy 👑

• NoAvailableJobStoreException 💀 — this one gets thrown when no job store name is available, so if you see it, check that you actually gave Toil a job store name 🎀

✨ Classes ✨

okay bestie, here's the whole class lineup 💅 (tables don't render in the terminal, so it's bullets for us 🫶)

• UnresolvedDict — a tag that says "hey, this dict is still full of promises that need resolving" 🎀
• SkipNull — an internal sentinel object, basically a secret handshake only Toil knows 🙈
• Conditional — holds a conditional expression on ice until we're actually ready to evaluate it 🧊
• ResolveSource — applies the linkMerge and pickValue operators to values coming into a port 🔗
• StepValueFrom — a workflow step input with a valueFrom expression attached 💫
• DefaultWithSource — a workflow step input that has both a source and a default value 🌸
• JustAValue — a simple value dressed up as a 'resolve'-able object, lowkey iconic 💁‍♀️
• ToilPathMapper — keeps track of files the Toil way 🗂️
• ToilSingleJobExecutor — a SingleJobExecutor that doesn't assume it's at the top level of the workflow 🧚
• ToilTool — a mixin that hooks Toil into a cwltool tool type 🔌
• ToilCommandLineTool — subclasses the cwltool command line tool to provide the custom ToilPathMapper 🛠️
• ToilExpressionTool — subclasses the cwltool expression tool to provide the custom ToilPathMapper 🧮
• ToilFsAccess — custom filesystem access class that handles Toil filestore references 📂
• VisitFunc — no description given in the original, so this one's a mystery 🕵️
• CWLNamedJob — base class for all CWL jobs that do user work, giving them useful names 🏷️
• ResolveIndirect — a helper Job 🤝
• CWLJobWrapper — wraps a CWL job that uses the dynamic resources requirement 🎁
• CWLJob — executes a CWL tool using cwltool.executors.SingleJobExecutor 🏃‍♀️
• CWLScatter — implements the workflow scatter step 🌈
• CWLGather — follows on from a scatter Job, the clean-up crew 🧹
• SelfJob — a fake job object that makes CWLWorkflow.run() easier to implement 🎭
• CWLWorkflow — a Toil Job that converts a CWL workflow graph into a Toil job graph 🧠
• CWLInstallImportsJob — a class that represents a unit of work in toil 📦
• CWLImportWrapper — a Job that organizes importing files on workers instead of the leader 🚚. It's responsible for extracting filenames and metadata,
• CWLStartJob — the Job responsible for starting the CWL workflow 🚀

---

✨ Functions ✨

• cwltoil_was_removed() — yells at you that this entrypoint is deprecated 😤
• filter_skip_null(name, value) — recursively filters out SkipNull objects from value 🧼
• ensure_no_collisions(directory[, dir_description]) — makes sure no items in the given CWL Directory share the same name 👯
• try_prepull(cwl_tool_uri, runtime_context, batchsystem) — Give it a whirl at prepulling every container in a CWL workflow, using either Singularity or Docker. 🐳✨
• resolve_dict_w_promises(dict_w_promises[, file_store]) — Takes a dictionary full of promises, evaluates the expressions inside, and spits out the actual values. Iconic glow-up 💅
• simplify_list(maybe_list) — Turns a length one list, the kind cwltool loves to hand you, into a plain scalar. Less clutter, more vibes 🫶
• toil_make_tool(toolpath_object, loadingContext) — Emits custom ToilCommandLineTools. Custom and proud 🦋
• toil_get_file(file_store, index, existing, uri[, ...]) — Sets up the given file or directory from the Toil jobstore at a file URI. Pull it up, bestie 📂
• convert_file_uri_to_toil_uri(applyFunc, index, ...) — Takes a file URI and converts it into a toil file URI. The applyFunc argument is the one doing the actual conversion work 🔄
• path_to_loc(obj) — Turns a path into a location. Literally that's it, it's giving simple 📍
• extract_file_uri_once(fileindex, existing, file_metadata) — Pulls the filename out of a CWL file record. Once. Just once 🎀
• visit_files(func, fs_access, fileindex, existing, ...) — Preps all the files and directories. Everybody gets prepared, no exceptions 🌸
• upload_directory(directory_metadata, directory_contents) — Uploads a Directory object. Up it goes ☁️
• extract_and_convert_file_to_toil_uri(convertfunc, ...) — Pulls the file URI out of a file object and converts it into a Toil URI. Two-step glow-up 💖
• writeGlobalFileWrapper(file_store, fileuri) — Wraps writeGlobalFile so it can accept <file://> URIs. Now it understands the assignment 🧚
• remove_empty_listings(rec) — No description given 🙈
• toilStageFiles(toil, cwljob, outdir[, destBucket, ...]) — Copies input files out of the global file store and updates their location and path. Moving day, but make it organised 🚚
• get_container_engine(runtime_context) — No description given 🙈
• makeRootJob(tool, jobobj, runtime_context, ...) — Creates the Toil root Job object for the CWL tool. It's the same as makeJob(), except this one also handles import logic. Main character energy, literally 👑
• makeJob(tool, jobobj, runtime_context, parent_name, ...) — Creates the correct Toil Job object for the CWL tool. Right job, right time ⭐
• remove_pickle_problems(obj) — Heads up! Doc_loader does not pickle correctly, which causes Toil errors, so this one removes the troublemakers from objects. Pickle drama, be gone 💀
• extract_workflow_inputs(options, ...) — Collects all the workflow input files so they can be imported later. A little hoarding, but make it cute 🧺
• import_workflow_inputs(jobstore, options, ...[, log_level]) — Imports all workflow inputs on the leader. Leader gets all the credit 👑
• visitSteps(cmdline_tool, op) — Iterates over a CWL Process object and runs the op on each tool description. Every step, every time 💜
• rm_unprocessed_secondary_files(job_params) — No description given 🙈
• filtered_secondary_files(unfiltered_secondary_files) — Removes unprocessed secondary files. Bye bye, not today 🍓
• scan_for_unsupported_requirements(tool[, ...]): sweeps through the given CWL tool and flags any optional features Toil doesn't support yet 🙈
• determine_load_listing(tool): figures out what's going on with the directory.listing feature in CWL 📂
• generate_default_job_store(batch_system_name, ...): picks a default job store that fits the requested batch system and
• get_options(args): parses your args and neatly tucks any non-Toil arguments into the cwljob of the Namespace 🎀
• main([args, stdout]): runs the main loop for toil-cwl-runner 👑 the whole show
• find_default_container(args, builder): finds the default constructor by checking in with a Toil.options object 🧚

✨ Module Contents ✨

the module contents are giving full catalogue energy 💅

toil.cwl.cwltoil.logger

toil.cwl.cwltoil.DEFAULT_TMPDIR

toil.cwl.cwltoil.DEFAULT_TMPDIR_PREFIX

toil.cwl.cwltoil.cwltoil_was_removed()

This one is a deprecation scolding for the old entrypoint 🙄

Return type: None

class toil.cwl.cwltoil.UnresolvedDict

Bases: dict[Any, Any]

A tag that says a dict still holds promises that need resolving 🫶

class toil.cwl.cwltoil.SkipNull

Internal sentinel object, so this one is strictly behind the scenes 🤫 It marks a null value produced by each port of a skipped conditional step. The CWL 1.2 spec says to treat it exactly the same as a null value 💀

toil.cwl.cwltoil.filter_skip_null(name, value)

Recursively filters out SkipNull objects from value 🧹

• Parameters
  • name (str): the name of the port that produced this value. It's only used when we find an unhandled null from a conditional step and print a warning. The name helps the user figure out which step/port caused the unhandled null 🕵️
  • value (Any): port output value object

Return type: Any

toil.cwl.cwltoil.ensure_no_collisions(directory, dir_description=None)

Makes sure no items in the given CWL Directory share the same name. If any do, it raises a WorkflowException about a "File staging conflict" 🚨 It does not recurse into subdirectories.

• Parameters
  • directory (cwltool.utils.DirectoryType)
  • dir_description (Optional[str])

Return type: None

toil.cwl.cwltoil.try_prepull(cwl_tool_uri, runtime_context, batchsystem)

Tries to prepull all containers in a CWL workflow using Singularity or Docker 🐳 It will not prepull the default container specified on the command line.

• Parameters
  • cwl_tool_uri (str): CWL workflow URL. Fragments are accepted too.
  • runtime_context (cwltool.context.RuntimeContext): runtime context of cwltool
  • batchsystem (str): type of Toil batchsystem

Return type: None

class toil.cwl.cwltoil.Conditional(expression=None, outputs=None, requirements=None, container_engine='docker')

Holds a conditional expression until we're ready to evaluate it 🤔 Evaluation happens before the enclosing step's inputs get type-checked.

• Parameters
  • expression (Optional[str])
  • outputs (Union[dict[str, cwltool.utils.CWLOutputType], None])
  • requirements (Optional[list[cwltool.utils.CWLObjectType]])
  • container_engine (str)

Class attributes

• expression = None
• outputs
• requirements = []
• container_engine = 'docker'
• is_false(job): checks whether the expression comes out to False given the completed step inputs.
  • job (cwltool.utils.CWLObjectType): job output object
  • Returns: bool
  • Return type: bool
• skipped_outputs(): generates a dict of SkipNull objects that match the output structure 🫧
  • Return type: dict[str, SkipNull]

class toil.cwl.cwltoil.ResolveSource(name, input, source_key, promises)

Applies the linkMerge and pickValue operators to values coming into a port 🔀

• Parameters
  • name (str)
  • input (dict[str, cwltool.utils.CWLObjectType])
  • source_key (str)
  • promises (dict[str, toil.job.Job])

Attribute

• promise_tuples: list[tuple[str, toil.job.Promise]] | tuple[str, toil.job.Promise]
• __repr__(): allows for debug printing 🐛
  • Return type: str
• resolve(): applies linkMerge first, then pickValue, if either one is present
  • Return type: Any
• link_merge(values): applies the linkMerge operator to the values object
  • values (cwltool.utils.CWLObjectType): result of step
  • Return type: Union[list[cwltool.utils.CWLOutputType], cwltool.utils.CWLOutputType]
• pick_value(values): applies the pickValue operator to the values object
  • values (Union[list[Union[str, SkipNull]], Any]): intended to be a list, but other types get returned without modification
  • Returns
  • Return type: Any

class toil.cwl.cwltoil.StepValueFrom(expr, source, req, container_engine)

A workflow step input with a valueFrom expression attached to it 🎀

The valueFrom expression gets evaluated to spit out the actual input object for the step 💅✨

🎀 Parameters

• expr (str)
• source (Any)
• req (list[cwltool.utils.CWLObjectType])
• container_engine (str)

The fields on this one are expr, source, context = None, req, and container_engine. 🦋

💖 __repr__()

Allow for debug printing. 🌸

Return type: str

🫶 eval_prep(step_inputs, file_store)

Resolve the contents of any file in a set of inputs. 📂

The inputs must be associated with the StepValueFrom object's self.source. Bestie, this one is important 😭

This gets called when loadContents is specified. ✨

🎀 Parameters

• step_inputs (cwltool.utils.CWLObjectType) — the workflow step inputs 🍓
• file_store (toil.fileStores.abstractFileStore.AbstractFileStore) — a Toil file store. You need it to resolve toilfile:// paths, so don't skip it 💜

Return type: None

👑 resolve()

Resolve the promise in the valueFrom expression's context. It's giving promise-keeping 🔥

Returns: an object that will serve as the expression context

Return type: Any

💅 do_eval(inputs)

Evaluate the valueFrom expression using the given input object. ⭐

Parameters:

• inputs (cwltool.utils.CWLObjectType)

Returns: an object

Return type: Any

---

🧚 class toil.cwl.cwltoil.DefaultWithSource(default, source)

A workflow step input that has both a source and a default value. Two options, one input, zero drama 🌈

🎀 Parameters

• default (Any)
• source (Any)

The attributes are default and source. 🦋

💖 __repr__()

Allow for debug printing. 🌸

Return type: str

👑 resolve()

Figure out the final input value when the time is right, meaning once the source can actually be resolved. Patience is a virtue, babe 🙈

Returns: dict

Return type: Any

---

🧚 class toil.cwl.cwltoil.JustAValue(val)

A simple value dressed up as a resolve-able object. It's giving imposter, but make it cute 🎀

🎀 Parameters

• val (Any)

The attribute here is val. 💜

💖 __repr__()

Allow for debug printing. 🌸

Return type: str

👑 resolve()

Return the value. That's it, that's the whole method 💅

Return type: Any

---

🌈 toil.cwl.cwltoil.resolve_dict_w_promises(dict_w_promises, file_store=None)

Resolve a dictionary of promises and evaluate expressions to produce the actual values. Main character energy for this function ✨

🎀 Parameters

• dict_w_promises (Union[UnresolvedDict, cwltool.utils.CWLObjectType, dict[str, Union[str, StepValueFrom]]]) — the input dict for these values 🍓
• file_store (Optional[toil.fileStores.abstractFileStore.AbstractFileStore])

Returns: a dictionary of actual values

Return type: cwltool.utils.CWLObjectType

---

🌸 toil.cwl.cwltoil.simplify_list(maybe_list)

Turn a length-one list loaded by cwltool into a scalar. Anything else gets passed through as-is, by reference. Simple, clean, iconic 🦋

Parameters:

• maybe_list (Any)

Return type: Any

---

🧚 class toil.cwl.cwltoil.ToilPathMapper(referenced_files, basedir, stagedir, separateDirs=True, get_file=None, stage_listing=False, streaming_allowed=True)

Bases: cwltool.pathmapper.PathMapper

Okay so this class keeps track of files the Toil way 🗂️✨ It maps between three things: the symbolic identifier of a file (the Toil FileID), its local path on the host (the value returned by readGlobalFile), and where the file lives inside the software container. Three identities, one file, very iykyk 💅

🎀 Parameters

• referenced_files (list[cwltool.utils.CWLObjectType])
• basedir (str)
• stagedir (str)
• separateDirs (bool)
• get_file (Union[Any, None])
• stage_listing (bool)
• streaming_allowed (bool)

The attributes are get_file = None, stage_listing = False, and streaming_allowed = True. 💖

🫶 visit(obj, stagedir, basedir, copy=False, staged=False)

Iterate over a CWL object and resolve File and Directory path references. This gets called on every File or Directory CWL object. Those all have location fields: for Files they come from upload_file(), and for Directories they come from upload_directory() or cwltool internally. With upload_directory(), the Directories and their children get locations by listing the Directories through ToilFsAccess. With cwltool, locations get set as absolute paths. Lots of moving parts, but it all works out 🔥

🎀 Parameters

• obj (cwltool.utils.CWLObjectType) — the CWL File or Directory to process 🍓
• stagedir (str) — the base path for target paths to be generated under, except when a File or Directory has an overriding parent directory in dirname
• basedir (str) — the directory relative paths get resolved from. It's used as the base directory for the StdFsAccess that generated the listing being processed 🌿
• copy (bool) — if set, use writable types for Files and Directories ✍️
• staged (bool) — this starts as True at the top of the recursion. It gets set to False when entering a directory that we can actually download, so we don't stage files and subdirectories separately from the directory as a whole. It controls the staged flag on generated mappings, which decides whether files and directories actually get placed at their mapped-to target locations. If stage_listing is True, it stays True the whole way through and everything gets staged 🫧

Return type: None

Each unique location for a File or Directory gets exactly one MapperEnt. Think of these as instructions for cwltool's stage_files function. You can check it out here: https://github.com/common-workflow-language/cwltool/blob/a3e3a5720f7b0131fa4f9c0b3f73b62a347278a6/cwltool/process.py#L254 🦋

A MapperEnt has these fields:

• resolved: an absolute local path anywhere on the filesystem where the file or directory can be found. It can also be the contents of a file to populate it with, if the type is CreateWritableFile or CreateFile. Or it can be a URI understood by the StdFsAccess in use, like toilfile:.
• target: an absolute path under stagedir where cwltool will then place the file or directory. The exception is when a File or Directory has a dirname field giving its parent path. Then that gets used instead 📍
• type: one of the following:
  • File: cwltool will copy or link the file from resolved to target, if possible.
  • CreateFile: cwltool will create the file at target, treating resolved as the contents.
  • WritableFile: cwltool will copy the file from resolved to target, making it writable.
  • CreateWritableFile: cwltool will create the file at target, treating resolved as the contents, and make it writable.
  • Directory: cwltool will copy or link the directory from resolved to target, if possible. Otherwise, if resolved starts with "_:", cwltool will make the directory at target. Otherwise it does nothing. 🌷
• WritableDirectory: okay so if it's possible, cwltool copies the directory from resolved over to target 📁. If it can't do that and resolved starts with "_:", cwltool makes a fresh directory at target 🌸. Otherwise it does absolutely nothing 🙈.
• staged: if you set this to False, cwltool won't make or copy anything for that entry 🚫. Main character decision 💅.

🦋 class toil.cwl.cwltoil.ToilSingleJobExecutor

Bases: cwltool.executors.SingleJobExecutor (docs at /usr/share/doc/cwltool/html/autoapi/cwltool/executors/index.html#cwltool.executors.SingleJobExecutor)

A SingleJobExecutor that does not assume it's at the top level of the workflow ✨. Why does this matter? Otherwise every job thinks it's top level and tries to discover secondary files. Those files might exist even when they haven't actually been passed at the top level, so they aren't supposed to be visible 👀. Iykyk.

💖 run_jobs

run_jobs(process, job_order_object, logger, runtime_context)

This is run_jobs from SingleJobExecutor, just not in a top level runtime context 🧚.

Parameters:

• process (cwltool.process.Process, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)
• job_order_object (cwltool.utils.CWLObjectType)
• logger (logging.Logger, docs at /usr/share/doc/python3/html/library/logging.html#logging.Logger)
• runtime_context (cwltool.context.RuntimeContext, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext)

Return type: None 🫶

🌈 class toil.cwl.cwltoil.ToilTool

class toil.cwl.cwltoil.ToilTool(*args, **kwargs)

This is a mixin that hooks Toil into a cwltool tool type 🔥.

Parameters:

• args (Any)
• kwargs (Any)

🎀 connect_toil_job

connect_toil_job(job)

Attaches the Toil tool to the Toil job that's executing it 🔗. This lets it use the Toil job to stop at certain points if debugging flags are set 🐛.

Parameters:

• job (toil.job.Job, anchor #toil.job.Job)

Return type: None

🍓 make_path_mapper

make_path_mapper(reffiles, stagedir, runtimeContext, separateDirs)

Creates the appropriate PathMapper for the situation 🗺️.

Parameters:

• reffiles (list[Any], docs at /usr/share/doc/python3/html/library/stdtypes.html#list)
• stagedir (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)
• runtimeContext (cwltool.context.RuntimeContext, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext)
• separateDirs (bool, docs at /usr/share/doc/python3/html/library/functions.html#bool)

Return type: cwltool.pathmapper.PathMapper (docs at /usr/share/doc/cwltool/html/autoapi/cwltool/pathmapper/index.html#cwltool.pathmapper.PathMapper)

💜 __str__

__str__()

Returns the string representation of this tool type 🎤.

Return type: str (docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

👑 class toil.cwl.cwltoil.ToilCommandLineTool

class toil.cwl.cwltoil.ToilCommandLineTool(*args, **kwargs)

Bases: ToilTool, cwltool.command_line_tool.CommandLineTool (docs at /usr/share/doc/cwltool/html/autoapi/cwltool/command_line_tool/index.html#cwltool.command_line_tool.CommandLineTool)

This subclasses the cwltool command line tool so it can use the custom ToilPathMapper 🛠️.

Parameters:

• args (Any)
• kwargs (Any)

🌸 class toil.cwl.cwltoil.ToilExpressionTool

class toil.cwl.cwltoil.ToilExpressionTool(*args, **kwargs)

Bases: ToilTool, cwltool.command_line_tool.ExpressionTool (docs at /usr/share/doc/cwltool/html/autoapi/cwltool/command_line_tool/index.html#cwltool.command_line_tool.ExpressionTool)

This subclasses the cwltool expression tool so it can use the custom ToilPathMapper 🧮.

Parameters:

• args (Any)
• kwargs (Any)

🧚 toil.cwl.cwltoil.toil_make_tool

toil.cwl.cwltoil.toil_make_tool(toolpath_object, loadingContext)

This is a factory function that emits custom ToilCommandLineTools 🏭. It's meant to be passed to cwltool.load_tool() 💅.

Parameters:

• toolpath_object (ruamel.yaml.comments.CommentedMap)
• loadingContext (cwltool.context.LoadingContext, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.LoadingContext)

Return type: cwltool.process.Process (docs at /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)

🙈 MISSING_FILE

toil.cwl.cwltoil.MISSING_FILE = 'missing://'

🦋 class toil.cwl.cwltoil.ToilFsAccess

class toil.cwl.cwltoil.ToilFsAccess(basedir, file_store=None)

Bases: cwltool.stdfsaccess.StdFsAccess (docs at /usr/share/doc/cwltool/html/autoapi/cwltool/stdfsaccess/index.html#cwltool.stdfsaccess.StdFsAccess)

This is a custom filesystem access class that handles Toil filestore references 🗃️. Normal file paths get resolved relative to basedir 📂. But toilfile: and toildir: URIs get fulfilled from the Toil file store ✨.

It also supports URLs that are supported by Toil job store implementations 🌐.

Parameters:

• basedir (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)
• file_store (Optional[toil.fileStores.abstractFileStore.AbstractFileStore], anchor #toil.fileStores.abstractFileStore.AbstractFileStore)

🌷 Attributes

• file_store: None
• dir_to_download: dict[str, str] (docs at /usr/share/doc/python3/html/library/stdtypes.html#dict)

🍓 glob

glob(pattern)

Returns a possibly empty list of absolute URI paths that match pathname 🔍.

Parameters:

• pattern (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: list[str] (docs at /usr/share/doc/python3/html/library/stdtypes.html#list)

🎀 open

open(fn, mode)

Parameters:

• fn (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)
• mode (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: IO[Any]

💅 exists

exists(path)

Tests for file existence 👀.

Parameters:

• path (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: bool (docs at /usr/share/doc/python3/html/library/functions.html#bool)

🫶 size

size(path)

Parameters:

• path (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: int (docs at /usr/share/doc/python3/html/library/functions.html#int)

🌈 isfile

isfile(fn)

Parameters:

• fn (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: bool (docs at /usr/share/doc/python3/html/library/functions.html#bool)

🧚 isdir

isdir(fn)

Parameters:

• fn (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: bool (docs at /usr/share/doc/python3/html/library/functions.html#bool)

🔥 listdir

listdir(fn)

Returns a list containing the absolute path URLs of the entries in the directory given by path 📋.

Parameters:

• fn (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: list[str] (docs at /usr/share/doc/python3/html/library/stdtypes.html#list)

⭐ join

join(path, *paths)

Joins one or more path segments intelligently 🧩.

Parameters:

• path (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)
• paths (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

🍓 realpath

realpath(fn)

Parameters:

• fn (str, docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

💖 toil.cwl.cwltoil.toil_get_file

toil.cwl.cwltoil.toil_get_file(file_store, index, existing, uri, streamable=False, streaming_allowed=True, pipe_threads=None)

This sets up the given file or directory from the Toil jobstore at a file URI where it can be accessed locally 📥. It runs as part of tool setup, inside jobs on the workers 🏗️. It's also used when reorganizing files so they get uploaded at the end of a tool 📤.

Parameters:

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore, anchor #toil.fileStores.abstractFileStore.AbstractFileStore): the Toil file store to download from 💾.
• index (dict[str, str], docs at /usr/share/doc/python3/html/library/stdtypes.html#dict): maps from the downloaded file path back to the input Toil URI 🗺️.
• existing (dict[str, str]) — it maps each URI to the path where that file got downloaded 🗺️
• uri (str) — the URI of the file you want to download 🌸
• streamable (bool) — tells you whether the file has the 'streamable' flag set 💅
• streaming_allowed (bool) — whether streaming is allowed ✨
• pipe_threads (Optional[list[tuple[threading.Thread, int]]]) — a list of the threads doing the streaming, each paired with the open file descriptors for those files. You're on the hook for closing the file descriptors (that's how the pipes get broken) and joining the threads 🧵

Return type str

---

🦋 toil.cwl.cwltoil.convert_file_uri_to_toil_uri

toil.cwl.cwltoil.convert_file_uri_to_toil_uri(applyFunc, index, existing, file_uri)

Okay so this one takes a file URI and turns it into a toil file URI ✨ It uses applyFunc to do the actual conversion. It runs once on every unique file URI, so no doubles 💅

existing is a set of files that came in as inputs from toil_get_file. That's what makes sure they get mapped back to the same name if they get passed through 🫶

It returns a toil URI path to the object 🌈

Parameters

• applyFunc (Callable[[str], toil.fileStores.FileID])
• index (dict[str, str])
• existing (dict[str, str])
• file_uri (str)

Return type str

---

🧚 toil.cwl.cwltoil.path_to_loc

toil.cwl.cwltoil.path_to_loc(obj)

This one turns a path into a location 🍓 It's for CWL objects that have a "path" but not a "location".

Parameters

• obj (cwltool.utils.CWLObjectType)

Return type None

---

🌸 toil.cwl.cwltoil.extract_file_uri_once

toil.cwl.cwltoil.extract_file_uri_once(fileindex, existing, file_metadata, mark_broken=False, skip_remote=False)

This one pulls the filename out of a CWL file record 📂 It matches the predefined function signature in visit_files, which is how it ends up getting called on every file inside a CWL object 🔥

It makes sure no duplicate files come back, going by fileindex. If a file hasn't been resolved yet (and had <file://> prepended), it resolves symlinks. Here's the tea on the params:

• fileindex: the forward mapping of filename
• existing: the reverse mapping of filename, which this function does not use 🙈
• file_metadata: the CWL file record
• mark_broken: whether files should be marked as missing
• skip_remote: whether to skip remote files
• :return: has no description in the docs

Parameters

• fileindex (dict[str, str])
• existing (dict[str, str])
• file_metadata (cwltool.utils.CWLObjectType)
• mark_broken (bool)
• skip_remote (bool)

Return type Optional[str]

---

💜 toil.cwl.cwltoil.V

Just a type variable named V, nothing else documented for this one 🙈

---

🎀 class toil.cwl.cwltoil.VisitFunc

Bases: Protocol[V]

__call__(fileindex, existing, file_metadata, mark_broken, skip_remote)

Parameters

• fileindex (dict[str, str])
• existing (dict[str, str])
• file_metadata (cwltool.utils.CWLObjectType)
• mark_broken (bool)
• skip_remote (bool)

Return type V

---

🌈 toil.cwl.cwltoil.visit_files

toil.cwl.cwltoil.visit_files(func, fs_access, fileindex, existing, cwl_object, mark_broken=False, skip_remote=False, bypass_file_store=False)

Okay this one prepares all files and directories ✨ It runs from the leader or a worker, in the context of the given CWL tool, order, or output object, so the workers can use everything. It makes sure their sizes are set and imports all the files 📦

It recurses inside directories using fs_access to find files to upload and the subdirectory structure to encode, even if their listings aren't set or aren't recursive. Any listing fields get preserved 🫶

If a file can't be found (like an optional secondary file that doesn't exist), it fails. The exception is when mark_broken is set, then it applies a sentinel location instead 😭 It also does some miscellaneous normalization 🧹

Parameters

• import_function — the function used to upload a URI and get a Toil FileID for it
• fs_access (cwltool.stdfsaccess.StdFsAccess) — the CWL FS access object we use to get at the filesystem and find files to import. It needs to support the URI schemes used.
• fileindex (dict[str, str]) — the forward map to fill in from file URI to Toil storage location, which write_file uses to dedupe writes
• existing (dict[str, str]) — the reverse map to fill in from Toil storage location to file URI. It's never read from.
• cwl_object (Optional[cwltool.utils.CWLObjectType]) — the CWL tool (or workflow order) we're importing files for
• mark_broken (bool) — if True, files that can't be imported (because they e.g. don't exist) get their locations set to MISSING_FILE instead of failing with an error
• skp_remote — if True, remote URIs get left in place instead of imported
• bypass_file_store (bool) — if True, <file://> URIs get left in place instead of importing files and directories
• log_level — log imported files at the given level
• func (VisitFunc[V])
• skip_remote (bool)

Return type list[V]

---

🍓 toil.cwl.cwltoil.upload_directory

toil.cwl.cwltoil.upload_directory(directory_metadata, directory_contents, mark_broken=False)

This one uploads a Directory object 📁 It ignores the listing (which may not be recursive, and isn't safe or efficient to touch) and uses directory_contents instead. That's a recursive dict structure going from filename to file URI, or to a subdirectory contents dict 🌳

It makes sure the directory actually exists and rewrites its location so it's something we can use on another machine 🦋

If mark_broken is set, it ignores missing directories and replaces them with directories holding the given (possibly empty) contents 🫧

We can't rely on the directory's listing as the next tool sees it being a complete recursive description of the files we'll need to show the tool 💀. Some tools need it cleared or single-level, but still expect to see its contents in the filesystem 🙈

Parameters

• directory_metadata (cwltool.utils.CWLObjectType)
• directory_contents (toil.lib.directory.DirectoryContents)
• mark_broken (bool) 🙈

Return type: None

🦋 toil.cwl.cwltoil.extract_and_convert_file_to_toil_uri

toil.cwl.cwltoil.extract_and_convert_file_to_toil_uri(convertfunc, fileindex, existing, file_metadata, mark_broken=False, skip_remote=False)

Okay so this one is a whole vibe 💅 It pulls the file URI out of a file object and converts it into a Toil URI ✨

It runs convertfunc on the file URI to handle the conversion, and it's what gets used when files are imported into the jobstore 🧚

If a file doesn't exist, it fails with an error… unless mark_broken is set, in which case the missing file gets a special sentinel location 🙈

Unless skip_remote is set, it also runs on remote files and sets their locations to Toil URIs too 🌈

Parameters

• convertfunc (Callable[[str], toil.fileStores.FileID])
• fileindex (dict[str, str])
• existing (dict[str, str])
• file_metadata (cwltool.utils.CWLObjectType)
• mark_broken (bool)
• skip_remote (bool)

Return type: None

🎀 toil.cwl.cwltoil.writeGlobalFileWrapper

toil.cwl.cwltoil.writeGlobalFileWrapper(file_store, fileuri)

This is basically writeGlobalFile with a glow-up: it accepts file:// URIs now 💖

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)
• fileuri (str)

Return type: toil.fileStores.FileID

🌸 toil.cwl.cwltoil.remove_empty_listings

toil.cwl.cwltoil.remove_empty_listings(rec)

Parameters

• rec (cwltool.utils.CWLObjectType)

Return type: None

👑 class toil.cwl.cwltoil.CWLNamedJob

class toil.cwl.cwltoil.CWLNamedJob(cores=1, memory='1GiB', disk='1MiB', accelerators=None, preemptible=None, tool_id=None, parent_name=None, subjob_name=None, local=None)

Bases: toil.job.Job

Base class for all CWL jobs that do actual user work, so they get useful names 🏷️✨

Parameters

• cores (Union[float, None])
• memory (Union[int, str, None])
• disk (Union[int, str, None])
• accelerators (Optional[list[toil.job.AcceleratorRequirement]])
• preemptible (Optional[bool])
• tool_id (Optional[str])
• parent_name (Optional[str])
• subjob_name (Optional[str])
• local (Optional[bool])

🧚 class toil.cwl.cwltoil.ResolveIndirect

class toil.cwl.cwltoil.ResolveIndirect(cwljob, parent_name=None)

Bases: CWLNamedJob

A helper Job 🫶 It takes an unresolved dict (the kind full of promises) and hands back a dictionary of actual values 💜

Parameters

• cwljob (toil.job.Promised[cwltool.utils.CWLObjectType])
• parent_name (Optional[str])

Attributes

• cwljob

run(file_store)

Evaluates the promises and returns their values 🔥

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type: cwltool.utils.CWLObjectType

🍓 toil.cwl.cwltoil.toilStageFiles

toil.cwl.cwltoil.toilStageFiles(toil, cwljob, outdir, destBucket=None, log_level=logging.DEBUG)

This one copies input files out of the global file store and updates their location and path 📦

Parameters

• toil (toil.common.Toil)
• cwljob (Union[cwltool.utils.CWLObjectType, list[cwltool.utils.CWLObjectType]])
• outdir (str)
• destBucket (Union[str, None]) — If set, it exports to this base URL instead of the local filesystem 🌍
• log_level (int) — Logs each file transferred at the given level 📝

Return type: None

🦋 class toil.cwl.cwltoil.CWLJobWrapper

class toil.cwl.cwltoil.CWLJobWrapper(tool, cwljob, runtime_context, parent_name, conditional=None)

Bases: CWLNamedJob

Wraps a CWL job that uses a dynamic resources requirement 💅 When it runs, it creates a new child job with the correct resource requirement set ✨

Parameters

• tool (cwltool.process.Process)
• cwljob (cwltool.utils.CWLObjectType)
• runtime_context (cwltool.context.RuntimeContext)
• parent_name (Optional[str])
• conditional (Union[Conditional, None])

Attributes

• cwltool
• cwljob
• runtime_context
• conditional
• parent_name

run(file_store)

Creates a child job with the correct resource requirements set 🎀

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type: Any

💖 class toil.cwl.cwltoil.CWLJob

class toil.cwl.cwltoil.CWLJob(tool, cwljob, runtime_context, parent_name=None, conditional=None)

Bases: CWLNamedJob

This is the one that actually runs a CWL tool, using cwltool.executors.SingleJobExecutor 🔥👑

Parameters

• tool (cwltool.process.Process)
• cwljob (cwltool.utils.CWLObjectType)
• runtime_context (cwltool.context.RuntimeContext)
• parent_name (Optional[str])
• conditional (Union[Conditional, None])

Attributes

• cwltool
• conditional
• cwljob
• runtime_context
• step_inputs
• workdir: str

required_env_vars(cwljob)

Yields environment variables from EnvVarRequirement 🌈

Parameters

• cwljob (Any)

Return type: Iterator[tuple[str, str]]

populate_env_vars(cwljob)

Prepares the environment variables the job needs at runtime 🫶

Env vars specified in the CWL "requirements" section should already be loaded in self.cwltool.requirements. However, the ones specified with "EnvVarRequirement" take precedence and only get populated here. So this doesn't just return a dictionary with all the evaluated "EnvVarRequirement" env vars 💅 It also checks self.cwltool.requirements for any env vars with the same name and swaps their value for the one found in the "EnvVarRequirement" env var, if there is one ✨

Parameters

• cwljob (cwltool.utils.CWLObjectType)

Return type: dict[str, str]

run(file_store)

Executes the CWL document 🚀

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type: Any

🌈 toil.cwl.cwltoil.get_container_engine

toil.cwl.cwltoil.get_container_engine(runtime_context)

Parameters

• runtime_context — the cwltool.context.RuntimeContext (/usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext) 🌸

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str), a plain string, bestie ✨

🌳 toil.cwl.cwltoil.makeRootJob

toil.cwl.cwltoil.makeRootJob(tool, jobobj, runtime_context, initialized_job_order, options, toil)

okay so this one builds the root Job for your CWL tool 💅 It's basically the same as makeJob(), but it also handles the import logic, so it's the one you want for the very start of things 🫶

It can actually make a little subgraph of two jobs. The second one might be the follow on of the first. If it only makes one job, you get that same job back twice, which is giving consistency 🧚

Returns: nothing is listed here, it's blank 🙈

Parameters

• tool — a cwltool.process.Process (/usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process) 🎀
• jobobj — a cwltool.utils.CWLObjectType 🎀
• runtime_context — the cwltool.context.RuntimeContext (/usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext) 🎀
• initialized_job_order — a cwltool.utils.CWLObjectType 🎀
• options — a configargparse.Namespace 🎀
• toil — a toil.common.Toil (<#toil.common.Toil>) 🎀

Return type: CWLNamedJob (<#toil.cwl.cwltoil.CWLNamedJob>) 💖

---

🦋 toil.cwl.cwltoil.makeJob

toil.cwl.cwltoil.makeJob(tool, jobobj, runtime_context, parent_name, conditional)

this is the one that creates the correct Toil Job object for your CWL tool, depending on what kind of tool it is 🔥 Same vibe as makeRootJob: it can make a subgraph of two jobs, where the second might be the follow on of the first, and if only one job gets made, it's returned twice 👑

It's used for three types: workflow, job, or job wrapper for dynamic resource requirements 🌈

Returns: "wfjob, followOn" if the input tool is a workflow, and "job, job" otherwise ✨

Parameters

• tool — a cwltool.process.Process (/usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process) 🎀
• jobobj — a cwltool.utils.CWLObjectType 🎀
• runtime_context — the cwltool.context.RuntimeContext (/usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext) 🎀
• parent_name — an Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str) 🎀
• conditional — a Union[Conditional] (<#toil.cwl.cwltoil.Conditional>), or None 🎀

Return type: a big union of tuples, so here it is verbatim 🧠

Union[tuple[CWLWorkflow <#toil.cwl.cwltoil.CWLWorkflow>, ResolveIndirect <#toil.cwl.cwltoil.ResolveIndirect>],
tuple[CWLJob <#toil.cwl.cwltoil.CWLJob>, CWLJob <#toil.cwl.cwltoil.CWLJob>],
tuple[CWLJobWrapper <#toil.cwl.cwltoil.CWLJobWrapper>, CWLJobWrapper <#toil.cwl.cwltoil.CWLJobWrapper>]]

---

🌸 toil.cwl.cwltoil.CWLScatter

class toil.cwl.cwltoil.CWLScatter(step, cwljob, runtime_context, parent_name, conditional)

Bases: toil.job.Job (<#toil.job.Job>) 💜

This is the workflow scatter step 🦋 When it runs, it makes a child job for every parameterization of the scatter. Main character energy, honestly 👑

Parameters

• step — a cwltool.workflow.WorkflowStep (/usr/share/doc/cwltool/html/autoapi/cwltool/workflow/index.html#cwltool.workflow.WorkflowStep) 🎀
• cwljob — a cwltool.utils.CWLObjectType 🎀
• runtime_context — the cwltool.context.RuntimeContext (/usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext) 🎀
• parent_name — an Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str) 🎀
• conditional — a Union[Conditional] (<#toil.cwl.cwltoil.Conditional>), or None 🎀

Attributes: step, cwljob, runtime_context, conditional, parent_name 💅

💖 flat_crossproduct_scatter(joborder, scatter_keys, outputs, postScatterEval)

Cartesian product of the inputs, then flattened 🌈

• joborder — a cwltool.utils.CWLObjectType 🎀
• scatter_keys — a list[str] (/usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀
• outputs — a list[toil.job.Promised[cwltool.utils.CWLObjectType]] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🎀
• postScatterEval — a Callable[[cwltool.utils.CWLObjectType], cwltool.utils.CWLObjectType] 🎀

Return type: None 🫶

🧚 nested_crossproduct_scatter(joborder, scatter_keys, postScatterEval)

Cartesian product of the inputs 💫

• joborder — a cwltool.utils.CWLObjectType 🎀
• scatter_keys — a list[str] (/usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀
• postScatterEval — a Callable[[cwltool.utils.CWLObjectType], cwltool.utils.CWLObjectType] 🎀

Return type: list[toil.job.Promised[cwltool.utils.CWLObjectType]] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🍓

🔥 run(file_store)

Generates the follow on scatter jobs 🏃‍♀️

• file_store — an toil.fileStores.abstractFileStore.AbstractFileStore (<#toil.fileStores.abstractFileStore.AbstractFileStore>) 🎀

Return type: list[toil.job.Promised[cwltool.utils.CWLObjectType]] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🍓

---

🌈 toil.cwl.cwltoil.CWLGather

class toil.cwl.cwltoil.CWLGather(step, outputs)

Bases: toil.job.Job (<#toil.job.Job>) 💜

This one follows on from a scatter Job 🦋 It gathers the outputs of each job in the scatter into an array, one array for each output parameter. So the scattered pieces all come back together, iykyk 🫶

Parameters

• step — a cwltool.workflow.WorkflowStep (/usr/share/doc/cwltool/html/autoapi/cwltool/workflow/index.html#cwltool.workflow.WorkflowStep) 🎀
• outputs — a toil.job.Promised[Union[cwltool.utils.CWLObjectType, list[cwltool.utils.CWLObjectType]]] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🎀

Attributes: step, outputs 💅

🫧 extract(obj, k) (static)

Pulls the given key out of the obj 🔑 If the object is a list, it grabs that key from every member of the list ✨

• obj — a Union[cwltool.utils.CWLObjectType, list[cwltool.utils.CWLObjectType]] 🎀
• k — a str (/usr/share/doc/python3/html/library/stdtypes.html#str) 🎀

Return type: Union[cwltool.utils.CWLOutputType, list[cwltool.utils.CWLObjectType]] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🍓

🌸 run(file_store)

Gathers all the outputs of the scatter 🎁

• file_store — an toil.fileStores.abstractFileStore.AbstractFileStore (<#toil.fileStores.abstractFileStore.AbstractFileStore>) 🎀

Return type: dict[str, Any] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓

---

🧚 toil.cwl.cwltoil.SelfJob

class toil.cwl.cwltoil.SelfJob(j, v)

Bases: toil.job.Job (<#toil.job.Job>) 💜

This is a fake job object 🙈 It exists to make CWLWorkflow.run() easier to implement, and that's the whole tea ☕

Parameters

• j — a CWLWorkflow (<#toil.cwl.cwltoil.CWLWorkflow>) 🎀
• v — a cwltool.utils.CWLObjectType 🎀

Attributes: j, v 💅

🦋 rv(*path)

Returns our properties dictionary 📖

• path — an Any 🎀

Return type: Any 🍓

💖 addChild(c)

Adds a child to our workflow 🌱

• c — a toil.job.Job (<#toil.job.Job>) 🎀

Return type: Any 🍓

🫶 hasChild(c)

Checks if the given child is in our workflow 👀

• c — a toil.job.Job (<#toil.job.Job>) 🎀

Return type: Any 🍓

---

💜 toil.cwl.cwltoil.ProcessType

Just the name on this one, no description, bestie 🫠

---

🫧 toil.cwl.cwltoil.remove_pickle_problems(obj)

Doc_loader doesn't pickle correctly, which causes Toil errors, so this function removes it from objects 🧹

See github issue: <https://github.com/mypyc/mypyc/issues/804> 🔗

• obj — a ProcessType 🎀

Return type: ProcessType 🍓

---

👑 toil.cwl.cwltoil.CWLWorkflow

class toil.cwl.cwltoil.CWLWorkflow(cwlwf, cwljob, runtime_context, parent_name=None, conditional=None)

Bases: CWLNamedJob 💜

This is the Toil Job that converts a CWL workflow graph into a Toil job graph ✨ The Toil job graph includes the appropriate dependencies, so everything is connected the right way 🫶

Parameters

• cwlwf — a cwltool.workflow.Workflow (/usr/share/doc/cwltool/html/autoapi/cwltool/workflow/index.html#cwltool.workflow.Workflow) 🎀
• cwljob — a cwltool.utils.CWLObjectType 🎀
• runtime_context — the cwltool.context.RuntimeContext (/usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext) 🎀

okay bestie, the list keeps going 💖

• parent_name — Optional[str] (see /usr/share/doc/python3/html/library/stdtypes.html#str) 🌸
• conditional — Union[Conditional, None] (see toil.cwl.cwltoil.Conditional) ✨

Attributes we get to peek at are cwlwf, cwljob, runtime_context, and conditional 🦋

🧚 run(file_store)

Okay so this one turns a CWL Workflow graph into a Toil job graph, and it is always run on the leader, because the batch system already knows to schedule it as a local job 👑

Parameters 🎀

• file_store — toil.fileStores.abstractFileStore.AbstractFileStore

Return type 💅

• Union[UnresolvedDict, dict[str, SkipNull]], where the dict is /usr/share/doc/python3/html/library/stdtypes.html#dict, the str is /usr/share/doc/python3/html/library/stdtypes.html#str, UnresolvedDict is toil.cwl.cwltoil.UnresolvedDict, and SkipNull is toil.cwl.cwltoil.SkipNull 🫶

---

🌈 toil.cwl.cwltoil.CWLInstallImportsJob

class toil.cwl.cwltoil.CWLInstallImportsJob(initialized_job_order, tool,
basedir, skip_remote, bypass_file_store, import_data, **kwargs)

Bases: toil.job.Job. This class is a unit of work in toil, so it's giving job energy ✨

Parameters 🎀

• initialized_job_order — toil.job.Promised[cwltool.utils.CWLObjectType]
• tool — toil.job.Promised[cwltool.process.Process] (see /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)
• basedir — str (see /usr/share/doc/python3/html/library/stdtypes.html#str)
• skip_remote — bool (see /usr/share/doc/python3/html/library/functions.html#bool)
• bypass_file_store — bool (see /usr/share/doc/python3/html/library/functions.html#bool)
• import_data — list[toil.job.Promised[dict[str, toil.fileStores.FileID]]], where the list is /usr/share/doc/python3/html/library/stdtypes.html#list, the dict is /usr/share/doc/python3/html/library/stdtypes.html#dict, the str is /usr/share/doc/python3/html/library/stdtypes.html#str, and FileID is toil.fileStores.FileID 🌸
• kwargs — Any

Attributes 🦋 initialized_job_order, tool, basedir, skip_remote, bypass_file_store, import_data

🔥 fill_in_files(initialized_job_order, tool, candidate_to_fileid, basedir, skip_remote, bypass_file_store)

This is a static method, and it's lowkey iconic. Give it a mapping of filenames to Toil file IDs and it swaps every filename for its file ID all through the CWL object 💖

Parameters 🎀

• initialized_job_order — cwltool.utils.CWLObjectType
• tool — cwltool.process.Process (see /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)
• candidate_to_fileid — dict[str, toil.fileStores.FileID], where the dict is /usr/share/doc/python3/html/library/stdtypes.html#dict, the str is /usr/share/doc/python3/html/library/stdtypes.html#str, and FileID is toil.fileStores.FileID
• basedir — str (see /usr/share/doc/python3/html/library/stdtypes.html#str)
• skip_remote — bool (see /usr/share/doc/python3/html/library/functions.html#bool)
• bypass_file_store — bool (see /usr/share/doc/python3/html/library/functions.html#bool)

Return type 💅

• tuple[cwltool.process.Process, cwltool.utils.CWLObjectType], where the tuple is /usr/share/doc/python3/html/library/stdtypes.html#tuple, and the Process is /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process

🧚 run(file_store)

Okay so this one converts the filenames in the workflow inputs into URIs. It returns a Promise of the transformed workflow inputs, which is a tuple holding the job order and the process ⭐

Parameters 🎀

• file_store — toil.fileStores.abstractFileStore.AbstractFileStore

Return type 💅

• Tuple[cwltool.process.Process, cwltool.utils.CWLObjectType] (cwltool.process.Process is /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)

---

🦋 toil.cwl.cwltoil.CWLImportWrapper

class toil.cwl.cwltoil.CWLImportWrapper(initialized_job_order, tool,
runtime_context, file_to_data, imported_files, options)

Bases: CWLNamedJob

Bestie, this job is the importer! It organizes importing files on the workers instead of the leader. It extracts the filenames and metadata, calls ImportsJob, applies the imports to the job objects, and schedules the start workflow job 🫶

This class is only used when runImportsOnWorkers is enabled. Don't go poking at it otherwise 💀

Parameters 🎀

• initialized_job_order — cwltool.utils.CWLObjectType
• tool — cwltool.process.Process (see /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)
• runtime_context — cwltool.context.RuntimeContext (see /usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext)
• file_to_data — dict[str, toil.job.FileMetadata], where the dict is /usr/share/doc/python3/html/library/stdtypes.html#dict, the str is /usr/share/doc/python3/html/library/stdtypes.html#str, and FileMetadata is toil.job.FileMetadata
• imported_files — dict[str, toil.fileStores.FileID], where the dict is /usr/share/doc/python3/html/library/stdtypes.html#dict, the str is /usr/share/doc/python3/html/library/stdtypes.html#str, and FileID is toil.fileStores.FileID
• options — configargparse.Namespace

Attributes 🌸 initialized_job_order, tool, runtime_context, file_to_data, imported_files, options

🧚 run(file_store)

Override this function to do the work and dynamically create successor jobs. It's giving customization energy 🌈

Parameters 🎀

• fileStore — used to create local and globally sharable temporary files and to send log messages to the leader process.
• file_store — toil.fileStores.abstractFileStore.AbstractFileStore

Returns ✨ The return value can be passed to other jobs using toil.job.Job.rv().

Return type 💅

• Any

---

💜 toil.cwl.cwltoil.CWLStartJob

class toil.cwl.cwltoil.CWLStartJob(tool, initialized_job_order,
runtime_context, **kwargs)

Bases: CWLNamedJob

This is the job that starts the CWL workflow. It takes in the workflow or tool and the inputs once every file has been imported, then creates the jobs that run those workflows 🎀

Parameters 🎀

• tool — toil.job.Promised[cwltool.process.Process] (see /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)
• initialized_job_order — toil.job.Promised[cwltool.utils.CWLObjectType]
• runtime_context — cwltool.context.RuntimeContext (see /usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext)
• kwargs — Any

Attributes 🦋 tool, initialized_job_order, runtime_context

🧚 run(file_store)

Override this function to do the work and dynamically create successor jobs, same vibe as before 🔥

Parameters 🎀

• fileStore — used to create local and globally sharable temporary files and to send log messages to the leader process.
• file_store — toil.fileStores.abstractFileStore.AbstractFileStore

Returns ✨ The return value can be passed to other jobs using toil.job.Job.rv().

Return type 💅

• Any

---

🍓 toil.cwl.cwltoil.extract_workflow_inputs

toil.cwl.cwltoil.extract_workflow_inputs(options, initialized_job_order,
tool)

Okay so this one collects all the workflow input files so they can be imported later 🧚 It takes options as a namespace, initialized_job_order as a CWL object, and tool as a tool object 💖

Parameters 🎀

• options — configargparse.Namespace
• initialized_job_order — cwltool.utils.CWLObjectType
• tool — cwltool.process.Process (see /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)

Return type 💅

• list[str], where the list is /usr/share/doc/python3/html/library/stdtypes.html#list and the str is /usr/share/doc/python3/html/library/stdtypes.html#str

---

🌸 toil.cwl.cwltoil.import_workflow_inputs

toil.cwl.cwltoil.import_workflow_inputs(jobstore, options,
initialized_job_order, tool, log_level=logging.DEBUG)

Bestie, this one imports all the workflow inputs on the leader. It runs when you're not importing on workers 🙈 It also takes a tool and a log_level, which defaults to logging.DEBUG, and it hands back nothing listed here, so no cap, that's the whole deal ✨

Parameters 🎀

• jobstore — toil.jobStores.abstractJobStore.AbstractJobStore (the Toil jobstore)
• options — configargparse.Namespace (a namespace)
• initialized_job_order — cwltool.utils.CWLObjectType (the CWL object)
• tool 🎀 (type cwltool.process.Process, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)
• log_level 🌸 (type int, docs at /usr/share/doc/python3/html/library/functions.html#int)

Return type 💅 None

✨ toil.cwl.cwltoil.T

No details for this one, bestie. It's just a name sitting there 🫠

🦋 toil.cwl.cwltoil.visitSteps(cmdline_tool, op)

okay so this one iterates over a CWL Process object and runs op on each tool description CWL object 🔁

Parameters 🎀

• cmdline_tool (type cwltool.process.Process, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process)
• op (type Callable[[ruamel.yaml.comments.CommentedMap], list[T]], docs for list at /usr/share/doc/python3/html/library/stdtypes.html#list)

Return type 💖 list[T] (docs for list at /usr/share/doc/python3/html/library/stdtypes.html#list)

🌈 toil.cwl.cwltoil.rm_unprocessed_secondary_files(job_params)

Parameters 🎀

• job_params (type Any)

Return type None

🫶 toil.cwl.cwltoil.filtered_secondary_files(unfiltered_secondary_files)

Removes unprocessed secondary files 🗑️

Interpolated strings and optional inputs in secondary files only showed up in CWL 1.1 🙈.

The CWL libraries we call totally resolve those interpolated strings ✅, but they also toss the resolved fields into the unresolved list, so we clean them out here after the fact.

We keep any secondary file whose location is anything other than MISSING_FILE 📁. The 'required' logic seems to live deeper in cwltool.builder.Builder(), and that's what correctly decides which files get imported 💅. So we drop the files here. If one is SUPPOSED to exist, you'll still get the appropriate file does not exist error, just a bit further down the track 🧚.

Parameters 🎀

• unfiltered_secondary_files (type cwltool.utils.CWLObjectType)

Return type 💖 list[cwltool.utils.CWLObjectType] (docs for list at /usr/share/doc/python3/html/library/stdtypes.html#list)

💀 toil.cwl.cwltoil.scan_for_unsupported_requirements(tool, bypass_file_store=False)

This one scans the given CWL tool for any unsupported optional features 🔍. If it has them, it raises an informative UnsupportedRequirement 🚨.

Parameters 🎀

• tool (type cwltool.process.Process, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process) — the CWL tool to check for unsupported requirements.
• bypass_file_store (type bool, docs at /usr/share/doc/python3/html/library/functions.html#bool) — True if the Toil file store is not being used to transport files between nodes, and raw origin node <file://> URIs are exposed to tools instead.

Return type None

🎀 toil.cwl.cwltoil.determine_load_listing(tool)

In CWL, any input directory can have a DIRECTORY_NAME.listing (where DIRECTORY_NAME is any variable name) set to one of three options 📂:

1. no_listing 🙅 — DIRECTORY_NAME.listing will be undefined. e.g.

inputs.DIRECTORY_NAME.listing == unspecified

2. shallow_listing 🌷 — DIRECTORY_NAME.listing will return a list one level deep of DIRECTORY_NAME's contents. e.g.

inputs.DIRECTORY_NAME.listing == [items in directory]
inputs.DIRECTORY_NAME.listing[0].listing == undefined
inputs.DIRECTORY_NAME.listing.length == # of items in directory

3. deep_listing 🌊 — DIRECTORY_NAME.listing will return a list of the entire contents of DIRECTORY_NAME. e.g.

inputs.DIRECTORY_NAME.listing == [items in directory]
inputs.DIRECTORY_NAME.listing[0].listing == [items in subdirectory if it exists and is the first item listed]
inputs.DIRECTORY_NAME.listing.length == # of items in directory

Check these out 👀: https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingRequirement and https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingEnum

Here's the priority order, periodt 💅: DIRECTORY_NAME.listing should be determined first from loadListing. If that's not specified, it comes from LoadListingRequirement. Else, it defaults to "no_listing" if unspecified.

Parameters 🎀

• tool (type cwltool.process.Process, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process) — ToilCommandLineTool

Return str 💖 One of 'no_listing', 'shallow_listing', or 'deep_listing'.

Return type Literal['no_listing', 'shallow_listing', 'deep_listing']

🌸 exception toil.cwl.cwltoil.NoAvailableJobStoreException

Bases: Exception (docs at /usr/share/doc/python3/html/library/exceptions.html#Exception)

Basically means there's no job store name available 💔.

🧚 toil.cwl.cwltoil.generate_default_job_store(batch_system_name, provisioner_name, local_directory)

This picks a default job store that fits the requested batch system and provisioner, plus whatever modules are installed 🧠. If there's no good default and you have to choose manually, it raises an error 🚫.

Parameters 🎀

• batch_system_name (type Optional[str], docs for str at /usr/share/doc/python3/html/library/stdtypes.html#str) — the registry name of the batch system the user requested, if any. If no name was requested, it should be None.
• provisioner_name (type Optional[str], docs for str at /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the provisioner the user requested, if any. Recognized provisioners include 'aws' and 'gce'. None means no provisioner is in use.
• local_directory (type str, docs for str at /usr/share/doc/python3/html/library/stdtypes.html#str) — path to a nonexistent local directory suitable for use as a file job store.

Return str 💖 The job store specifier for a usable job store ✨.

Return type str (docs at /usr/share/doc/python3/html/library/stdtypes.html#str)

💜 toil.cwl.cwltoil.usage_message = Multiline-String

A multiline string that shows up as empty in the docs 🫧

"""

"""

🌈 toil.cwl.cwltoil.get_options(args)

This one parses the given args and properly adds non-Toil arguments into the cwljob of the Namespace 🧃.

Parameters 🎀

• args (type list[str], docs for list at /usr/share/doc/python3/html/library/stdtypes.html#list and for str at /usr/share/doc/python3/html/library/stdtypes.html#str) — :param args: is the list of args from the command line.

Return type configargparse.Namespace — :return: gives you the options namespace 💫.

🔥 toil.cwl.cwltoil.main(args=None, stdout=sys.stdout)

This runs the main loop for toil-cwl-runner 🏃‍♀️.

Parameters 🎀

• args (type Optional[list[str]], docs for list at /usr/share/doc/python3/html/library/stdtypes.html#list and for str at /usr/share/doc/python3/html/library/stdtypes.html#str)
• stdout (type TextIO)

Return type 💖 int (docs at /usr/share/doc/python3/html/library/functions.html#int)

🫧 toil.cwl.cwltoil.find_default_container(args, builder)

Finds the default constructor by consulting a Toil.options object 🔎.

Parameters 🎀

• args (type configargparse.Namespace)
• builder (type cwltool.builder.Builder, docs at /usr/share/doc/cwltool/html/autoapi/cwltool/builder/index.html#cwltool.builder.Builder)

Return type Optional[str] (docs for str at /usr/share/doc/python3/html/library/stdtypes.html#str)

---

💜 toil.cwl.utils

Utility functions used for Toil's CWL interpreter 🧰.

🎀 Attributes

• logger
• CWL_UNSUPPORTED_REQUIREMENT_EXIT_CODE
• CWL_UNSUPPORTED_REQUIREMENT_EXCEPTION
• DownReturnType
• UpReturnType
• DirectoryStructure

No descriptions were given for any of these, so we're just vibing with the names 🤷

🌷 Exceptions

• CWLUnsupportedException — Fallback exception.

✨ Functions ✨

okay so these are the helper functions living in toil.cwl.utils 💅 bestie, they're the little workhorses behind the CWL stuff 🫶

• visit_top_cwl_class(rec, classes, op): runs op on every top-level CWL object whose class is one of the named CWL classes. It's like cwltool's visit_class, but it won't peek inside anything it visits 👀
• visit_cwl_class_and_reduce(rec, classes, op_down, op_up): runs op_down top-down and op_up bottom-up on every CWL object with the named classes. op_up gets op_down's result plus a list of all the child op_up results, flattened across lists, with nodes of non-matching classes collapsed 🌸
• get_from_structure(dir_dict, path): given a relative path, it follows that path through the directory structure 🗺️
• download_structure(file_store, index, existing, ...): downloads a nested dictionary from the Toil file store into a local path 📥
• trim_mounts_op_down(file_or_directory): a no-op function for mount-point trimming. It literally does nothing, and that's the vibe 🫥
• sniff_location(file_or_directory): gets the local bare path for a CWL file or directory, or None if there isn't one 🔍
• trim_mounts_op_up(file_or_directory, op_down_ret, ...): removes subtrees of the CWL file or directory object tree that only have redundant stuff in them. Declutter era 🧹
• remove_redundant_mounts(cwljob): removes any redundant mount points from the listing. It modifies the CWL object in place, so no copies, just vibes ✏️

---

✨ Module Contents ✨

okay the module contents are next and they're giving organized 🎀

• toil.cwl.utils.logger

toil.cwl.utils.CWL_UNSUPPORTED_REQUIREMENT_EXIT_CODE

toil.cwl.utils.CWL_UNSUPPORTED_REQUIREMENT_EXIT_CODE = 33

exception toil.cwl.utils.CWLUnsupportedException

Bases: Exception

This is the fallback exception, so it's the one you get when nothing more specific fits 🛟

toil.cwl.utils.CWL_UNSUPPORTED_REQUIREMENT_EXCEPTION

This one is a type, and it's either cwltool's UnsupportedRequirement or our CWLUnsupportedException 🧐

toil.cwl.utils.CWL_UNSUPPORTED_REQUIREMENT_EXCEPTION: type[cwltool.errors.UnsupportedRequirement] | type[CWLUnsupportedException]

toil.cwl.utils.visit_top_cwl_class(rec, classes, op)

Applies the given operation to all top-level CWL objects with the given named CWL class. Like cwltool's visit_class, but it doesn't look inside any object it visits 👀

Parameters

• rec (Any)
• classes (collections.abc.Iterable[str])
• op (Callable[[Any], Any])

Return type: None

toil.cwl.utils.DownReturnType

toil.cwl.utils.UpReturnType

toil.cwl.utils.visit_cwl_class_and_reduce(rec, classes, op_down, op_up)

Applies the given operations to all CWL objects with the given named CWL class ✨ op_down runs top-down and op_up runs bottom-up. The up operation gets the down operation's result and a list of the up results for every child key. That list gets flattened across lists, and nodes of non-matching classes get collapsed 🌈

Returns: the flattened list of up operation results from all calls.

Parameters

• rec (Any)
• classes (collections.abc.Iterable[str])
• op_down (Callable[[Any], DownReturnType])
• op_up (Callable[[Any, DownReturnType, list[UpReturnType]], UpReturnType])

Return type: list[UpReturnType]

toil.cwl.utils.DirectoryStructure

toil.cwl.utils.get_from_structure(dir_dict, path)

Given a relative path, follows it through the given directory structure 🗺️

It returns the string URI for files, the directory dict for subdirectories, or None for things that don't exist 🫥

Parameters

• dir_dict (DirectoryStructure)
• path (str)

Return type: Union[str, DirectoryStructure, None]

toil.cwl.utils.download_structure(file_store, index, existing, dir_dict, into_dir)

Downloads a nested dictionary from the Toil file store to a local path 📥

This one is guaranteed to fill the structure with real files, not symlinks pointing out of it to somewhere else. File URIs can be toilfile: URIs or any other URI that Toil's job store system can read 🔒

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore): the Toil file store to download from.
• index (dict[str, str]): maps from the downloaded file path back to the input URI.
• existing (dict[str, str]): maps from a file_store_id URI to the downloaded file path.
• dir_dict (DirectoryStructure): a dict from string to string (for files) or dict (for subdirectories) describing a directory structure.
• into_dir (str): the directory to download the top-level dict's files into.

Return type: None

toil.cwl.utils.trim_mounts_op_down(file_or_directory)

A no-op function for mount-point trimming. Literally does nothing, very chill 🫥

Parameters

• file_or_directory (cwltool.utils.CWLObjectType)

Return type: None

toil.cwl.utils.sniff_location(file_or_directory)

Gets the local bare path for a CWL file or directory, or None 🔍

It returns None if there's no local path or file URI to be found.

Returns: None if we don't have a local path or file URI.

Parameters

• file_or_directory (cwltool.utils.CWLObjectType)

Return type: Optional[str]

toil.cwl.utils.trim_mounts_op_up(file_or_directory, op_down_ret, child_results)

Removes subtrees of the CWL file or directory object tree that only have redundant stuff in them 🧹 Here's how "nonredundant" works, bestie:

• For something in a directory, nonredundant means its path or location is not within the parent directory, or it doesn't match its basename.
• For something in a secondary file, nonredundant means its path or location is not adjacent to the primary file, or it doesn't match its basename.

If it's a File: it returns True if anything in its secondary files is nonredundant, or if it has nonredundant children. Otherwise it returns False.

If it's a Directory: it returns True if anything in its top-level listing is nonredundant, or if it has nonredundant children. Otherwise it returns False. If something in the listing is redundant and all of its children are redundant too, it gets deleted 💀

The docstring says the result is a boolean 🎀

Parameters

• file_or_directory (cwltool.utils.CWLObjectType)
• op_down_ret (None)
• child_results (list[bool])

Return type: bool (see /usr/share/doc/python3/html/library/functions.html#bool) 💖

toil.cwl.utils.remove_redundant_mounts(cwljob)

Okay bestie, this one is a cleanup queen 🧹✨ It goes through the listing and removes any redundant mount points. Heads up: it modifies the CWL object in place, so no fresh copy comes back. It just edits the thing you handed it 💅

Parameters

• cwljob (cwltool.utils.CWLObjectType)

Return type: None 🫶

✨ Functions ✨

Okay so this section has one function in its little table 🎀

• check_cwltool_version() — checks if the installed cwltool version matches Toil's expected version 🧐

✨ Package Contents ✨

toil.cwl.check_cwltool_version()

Checks if the installed cwltool version matches Toil's expected version 👀 If the versions don't match, a warning gets printed to standard error. Toil doesn't assume logging is set up yet, so it doesn't rely on that. It's safe to call this as many times as you want, but only one warning will ever show up 🌸

Return type: None

✨ toil.deferred ✨

Attributes

• logger

Classes

• DeferredFunction
• DeferredFunctionManager — Implements a deferred function system. Each Toil worker will have

Module Contents

• toil.deferred.logger

toil.deferred.DeferredFunction

Bases: namedtuple('DeferredFunction', 'function args kwargs name module') 🧚

Here's a little doctest vibe check ✨

>>> from collections import defaultdict
>>> df = DeferredFunction.create(defaultdict, None, {'x':1}, y=2)
>>> df
DeferredFunction(defaultdict, ...)
>>> df.invoke() == defaultdict(None, x=1, y=2)
True

classmethod create(function, *args, **kwargs)

Captures the given callable and arguments as an instance of this class 🦋

Parameters

• function (callable) — the deferred action to take, in the form of a function
• args (tuple, see /usr/share/doc/python3/html/library/stdtypes.html#tuple) — non-keyword arguments to the function
• kwargs (dict, see /usr/share/doc/python3/html/library/stdtypes.html#dict) — keyword arguments to the function

invoke()

Invokes the captured function with the captured arguments 🔥

__str__()

Returns str(self) 💜

__repr__

Returns repr(self) 💜

toil.deferred.DeferredFunctionManager(stateDirBase)

Okay this one is the main character 👑 It implements a deferred function system. Each Toil worker gets its own instance of this class. When a job runs, it happens inside a context manager from this class. If the job registers any "deferred" functions, they get executed when the context manager exits.

If the Python process dies before properly exiting the context manager and running the deferred functions, then one of two things picks them up. Either some other worker process enters or exits the per-job context manager of this class later on, or the DeferredFunctionManager gets shut down on the worker. Either way, the earlier job's deferred functions get picked up and run 🫶

Deferred function cleanup is on a best-effort basis, and deferred functions may end up getting executed multiple times. Not a vibe, but a warning 💀

Under the hood, deferred functions are serialized into files in the given directory, and those files are locked by the owning process. If that process dies, other processes can detect that the files are able to be locked, and they'll take them over 🌈

Parameters

• stateDirBase (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

Constants

• STATE_DIR_STEM = 'deferred'
• PREFIX = 'func'
• WIP_SUFFIX = '.tmp'

Attributes

• stateDir
• stateFileName
• stateFileOut
• stateFileIn

Methods

• __del__() — cleans up our state on disk. It assumes the deferred functions we manage have all been executed, and none are currently recorded 🧽
• open() — yields a single-argument function that lets you register deferred functions of type toil.DeferredFunction. This design means deferred functions can only be registered inside this context manager 🎀

Not thread safe. Keep it away from your threads 😬

• classmethod cleanupWorker(stateDirBase) — called by the batch system when it shuts down the node, after all workers are done, if the batch system supports worker cleanup. It checks once more for orphaned deferred functions and runs them 🧹

Parameters

• stateDirBase (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

✨ toil.exceptions ✨

Neutral place for exceptions, so import cycles don't get tangled up 🧶

Attributes

• logger

Exceptions

• FailedJobsException — common base class for all non-exit exceptions 🙈

Module Contents

• toil.exceptions.logger

exception toil.exceptions.FailedJobsException(job_store, failed_jobs, exit_code=1)

Bases: Exception (see /usr/share/doc/python3/html/library/exceptions.html#Exception) 🌸

Common base class for all non-exit exceptions.

Parameters

• job_store (toil.jobStores.abstractJobStore.AbstractJobStore, see <#toil.jobStores.abstractJobStore.AbstractJobStore>)
• failed_jobs (list, see /usr/share/doc/python3/html/library/stdtypes.html#list, of toil.job.JobDescription, see <#toil.job.JobDescription>)
• exit_code (int, see /usr/share/doc/python3/html/library/functions.html#int)

Attributes

• msg
• exit_code = 1
• jobStoreLocator
• numberOfFailedJobs

__str__()

Stringifies the exception, including the message 💅

Return type: str (see /usr/share/doc/python3/html/library/stdtypes.html#str)

✨ toil.fileStores ✨

Submodules

• toil.fileStores.abstractFileStore

✨ toil.fileStores.abstractFileStore ✨

Attributes

• logger

Classes

• AbstractFileStore — interface used to allow user code run by Toil to read and write files 🦋

Module Contents

• toil.fileStores.abstractFileStore.logger

class toil.fileStores.abstractFileStore.AbstractFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)

Bases: abc.ABC (see /usr/share/doc/python3/html/library/abc.html#abc.ABC) 🧚

This is the interface that lets user code run by Toil read and write files, and it's the one-stop shop for other Toil facilities that user code needs. That includes:

• normal (non-real-time) logging
• finding the correct temporary directory for scratch work
• importing and exporting files into and out of the workflow

It stores user files in the jobStore, but keeps them separate from actual jobs. It may implement caching. It's passed as an argument to the toil.job.Job.run() (see <#toil.job.Job.run>) method 💖

Access to files is only permitted inside the context manager provided by toil.fileStores.abstractFileStore.AbstractFileStore.open(). Stay inside that context manager or you're not getting in 🙅

It's also responsible for committing completed jobs back to the job store with an update operation, and for allowing that commit operation to be waited for ⏳

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore, see <#toil.jobStores.abstractJobStore.AbstractJobStore>)
• jobDesc (toil.job.JobDescription, see <#toil.job.JobDescription>)
• file_store_dir (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• waitForPreviousCommit (Callable[[], Any])

Attributes

• jobStore
• jobDesc
• localTempDir: str = b'.' (type link: /usr/share/doc/python3/html/library/stdtypes.html#str)
• workflow_dir: str (type link: /usr/share/doc/python3/html/library/stdtypes.html#str)
• coordination_dir: str (type link: /usr/share/doc/python3/html/library/stdtypes.html#str)

okay so the FileStore class has some attributes that are lowkey the backbone of the whole thing 💖

jobName: str = ''

waitForPreviousCommit

logging_messages: list[dict[str, int | str]] = []

logging_user_streams: list[dict[str, str]] = []

filesToDelete: set[str]

jobName defaults to ''. logging_messages and logging_user_streams both start as empty lists 🧚. filesToDelete is a set[str] with no default shown, and waitForPreviousCommit has no type or default listed 🌸.

---

✨ createFileStore

static createFileStore(jobStore, jobDesc, file_store_dir,
waitForPreviousCommit, caching)

Okay, this one makes a concrete FileStore 🎀

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)
• jobDesc (toil.job.JobDescription)
• file_store_dir (str)
• waitForPreviousCommit (Callable[[], Any])
• caching (Optional[bool])

Return type

Union[toil.fileStores.nonCachingFileStore.NonCachingFileStore, toil.fileStores.cachingFileStore.CachingFileStore]

---

🧹 shutdownFileStore

static shutdownFileStore(workflowID, config_work_dir,
config_coordination_dir)

This one does the cleanup, and it's giving serious vibes 💀 It carries out any filestore-specific cleanup it needs to.

This is a destructive operation, so it is important to ensure that there are no other running processes on the system that are modifying or using the file store for this workflow. 😭

This is meant to be the last call to the file store in a Toil run. The batch system cleanup function calls it when the batch system shuts down 🦋

Parameters

• workflowID (str) — the workflow ID for this run of the workflow 🍓
• config_work_dir (Optional[str]) — the path to the work directory in the Toil Config
• config_coordination_dir (Optional[str]) — the path to the coordination directory in the Toil Config

Return type

None

---

🚪 open

open(job)

This creates the context manager that wraps the tasks that run before and after a job. File operations are only allowed inside the context manager, so keep that in mind bestie 🫶

Implementations must only yield from within with super().open(job):.

Parameters

• job (toil.job.Job) — the job instance of the toil job to run

Return type

collections.abc.Generator[None, None, None]

---

📏 get_disk_usage

get_disk_usage()

This gets the number of bytes of disk used by the last job that ran under open() ✨ Disk usage is measured at the end of the job.

TODO: Sample periodically and record peak usage. 🙈

Return type

Optional[int]

---

📁 getLocalTempDir

getLocalTempDir()

This gets you a brand new local temporary directory to write files into 🌈 It only sticks around for the duration of the job.

Returns

The absolute path to a new local temporary directory. This directory exists for the duration of the job only, and it is guaranteed to be deleted once the job terminates, removing all the files it contains recursively 💅

Return type

str

---

📝 getLocalTempFile

getLocalTempFile(suffix=None, prefix=None)

This gets a new local temporary file that will persist for the duration of the job 🦋

Parameters

• suffix (Optional[str]) — if not None, the file name ends with this string. Otherwise the default value ".tmp" is used
• prefix (Optional[str]) — if not None, the file name starts with this string. Otherwise the default value "tmp" is used

Returns

The absolute path to a local temporary file. This file exists for the duration of the job only, and it is guaranteed to be deleted once the job terminates 🫧

Return type

str

---

🔤 getLocalTempFileName

getLocalTempFileName(suffix=None, prefix=None)

This gets a valid name for a new local file, but it does NOT actually create a file at that path 🙅‍♀️

Parameters

• suffix (Optional[str]) — if not None, the file name ends with this string. Otherwise the default value ".tmp" is used
• prefix (Optional[str]) — if not None, the file name starts with this string. Otherwise the default value "tmp" is used

Returns

Path to a valid file 🍓

Return type

str

---

⬆️ writeGlobalFile

abstractmethod writeGlobalFile(localFileName, cleanup=False)

This is an abstract method, so it's basically a promise that the subclass fills in the details 💜 It uploads a file (given as a path) to the job store.

If the file lives in a FileStore-managed temporary directory (aka it came from toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir()), it becomes a local copy of the file. That copy is eligible for deletion by toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile() 🧚

If an executable file on the local filesystem gets uploaded, its executability is preserved when it is downloaded again ⭐

Parameters

• localFileName (str) — the path to the local file to upload. The last path component (the basename of the file) stays associated with the file in the file store, if the backing JobStore supports it, so the file can be searched for by name or name glob 🔍
• cleanup (bool) — if True, the copy of the global file is deleted once the job and all its successors have finished running. If not, the global file must be deleted manually 🧼

Returns

An ID that can be used to retrieve the file 🪪

Return type

toil.fileStores.FileID

---

🌊 writeGlobalFileStream

writeGlobalFileStream(cleanup=False, basename=None,
encoding=None, errors=None)

This is like writeGlobalFile, but it lets you write a stream to the job store 🌸 You don't need to close the yielded file handle explicitly, and you should not do it either 🙅‍♀️

Parameters

• encoding (Optional[str]) — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which means binary mode
• errors (Optional[str]) — how encoding errors are handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified
• cleanup (bool) — works the same as in toil.fileStores.abstractFileStore.AbstractFileStore.writeGlobalFile()

---

📎 Cross-reference targets

/usr/share/doc/python3/html/library/stdtypes.html#str
/usr/share/doc/python3/html/library/stdtypes.html#list
/usr/share/doc/python3/html/library/stdtypes.html#dict
/usr/share/doc/python3/html/library/stdtypes.html#set
/usr/share/doc/python3/html/library/functions.html#int
/usr/share/doc/python3/html/library/functions.html#bool
/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator

• basename (Optional[str]) — okay so if the backing JobStore supports it 💖, pass in a file basename. Then when someone searches the job store with a query matching that basename, the file will totally get detected 🔍

Returns

You get back a context manager that yields a tuple of 1) a file handle you can write to ✍️ and 2) the toil.fileStores.FileID of the resulting file in the job store 🎀

Return type

collections.abc.Iterator[tuple[toil.lib.io.WriteWatchingStream, toil.fileStores.FileID]]

logAccess(fileStoreID, destination=None)

This is how the job records that a file was read 👀 (it gets announced if the job fails 💀).

If destination is not None, it gives the path the file was downloaded to. Otherwise, it's assumed the file was streamed 🌊

It must be called by readGlobalFile() and readGlobalFileStream() implementations, no skipping 🫶

Parameters

• fileStoreID (Union[toil.fileStores.FileID, str])
• destination (Union[str, None])

Return type None

readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False)

This one's marked abstractmethod 🧩 and its whole job is to make the file tied to fileStoreID available locally 📂

readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False)

If mutable is True, a copy of the file gets created locally, so the original isn't modified and doesn't change the file for other jobs 🙅‍♀️. If mutable is False, a link can be created to the file instead, which saves disk resources 💾. The downloaded file will be executable if and only if it was originally uploaded from an executable file on the local filesystem ⚡

If a userPath is specified, it's used as the destination. If not, the file gets stored in the local temp directory with an encoded name 🗂️

The destination file must not be deleted by the user 💀 it can only be deleted through deleteLocalFile.

Implementations must call logAccess() to report the download 📣

Parameters

• fileStoreID (str) — job store id for the file 🆔
• userPath (Optional[str]) — a path to the name of the file the global file gets copied or hard-linked to (see below) 📝
• cache (bool) — Described in toil.fileStores.CachingFileStore.readGlobalFile()
• mutable (bool) — Described in toil.fileStores.CachingFileStore.readGlobalFile()
• symlink (bool) — True if the caller can accept a symlink, False if the caller can only accept a normal file or hardlink 🔗

Returns an absolute path to a local, temporary copy of the file keyed by fileStoreID 📍

Return type str

readGlobalFileStream(...)

readGlobalFileStream(fileStoreID: str, encoding: Literal[None] = None, errors: str | None = None) -> ContextManager[IO[bytes]]
readGlobalFileStream(fileStoreID: str, encoding: str, errors: str | None = None) -> ContextManager[IO[str]]

Read a stream from the job store, kind of like readGlobalFile 🌊 The yielded file handle does not need to be closed explicitly, and it should not be closed explicitly either 🙈

Parameters

• encoding — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which represents binary mode 🔤
• errors — an optional string that specifies how encoding errors are handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified 🚨

Implementations must call logAccess() to report the download 📣

Returns a context manager yielding a file handle which can be read from.

getGlobalFileSize(fileStoreID)

Get the size of the file pointed to by the given ID, in bytes 📏

If it's a FileID or something else with a non-None 'size' field, that value is what gets used ✨. Otherwise, the job store is asked to poll the file's size 🕵️

Note that the job store may overestimate the file's size, for example if it's encrypted and had to be augmented with an IV or other encryption framing 🔐

Parameters

• fileStoreID (Union[toil.fileStores.FileID, str]) — File ID for the file

Returns the file's size in bytes, as stored in the job store

Return type int

deleteLocalFile(fileStoreID)

This one's an abstractmethod too 🧩 Delete local copies of files associated with the provided job store ID 🗑️

It raises an OSError with an errno of errno.ENOENT if no such local copies exist. That means it cannot be called multiple times in succession 💀 one and done, bestie.

The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job's file-store-provided temp directory, plus the file that was written to create the given file ID, if that file was written by the current job from the job's file-store-provided temp directory 🧹

Parameters

• fileStoreID (Union[toil.fileStores.FileID, str]) — File Store ID of the file to be deleted.

Return type None

deleteGlobalFile(fileStoreID)

Also an abstractmethod 🧩 Delete local files, and then permanently delete them from the job store 💀

To make sure the job can be restarted if necessary, the delete won't happen until after the job's run method has completed ⏳

Parameters

• fileStoreID (Union[toil.fileStores.FileID, str]) — the File Store ID of the file to be deleted.

Return type None

importFile(srcUrl, sharedFileName=None)

Parameters

• srcUrl (str)
• sharedFileName (Optional[str])

Return type Optional[toil.fileStores.FileID]

import_file(src_uri, shared_file_name=None)

Parameters

• src_uri (str)
• shared_file_name (Optional[str])

Return type Optional[toil.fileStores.FileID]

exportFile(jobStoreFileID, dstUrl)

Parameters

• jobStoreFileID (toil.fileStores.FileID)
• dstUrl (str)

Return type None

export_file(file_id, dst_uri)

This one's an abstractmethod as well 🧩

Parameters

• file_id (toil.fileStores.FileID)
• dst_uri (str)

Return type: None

log_to_leader(text, level=logging.INFO)

Sends a logging message to the leader, bestie 💌 The worker logs the same message at the same level too. ✨

This doesn't touch the commit system, so it's totally safe to use during an asynchronous commit or without a commit afterward. 🫶

Parameters

• text (str, see </usr/share/doc/python3/html/library/stdtypes.html#str>) — the string to log. 📝
• level (int, see </usr/share/doc/python3/html/library/functions.html#int>) — the logging level. 📊

Return type: None

logToMaster(text, level=logging.INFO)

This one's the older name for the same thing. 🕰️

Parameters

• text (str, see </usr/share/doc/python3/html/library/stdtypes.html#str>)
• level (int, see </usr/share/doc/python3/html/library/functions.html#int>)

Return type: None

log_user_stream(name, stream)

Sends a stream of UTF-8 text to the leader as a named log stream. 🌊

This is super useful for things like the error logs of Docker containers. 🐳 The leader will show it to the user or organize it nicely as user-level log info. 💅

Parameters

• name (str, see </usr/share/doc/python3/html/library/stdtypes.html#str>) — a hierarchical string delimited by ..
• stream (IO[bytes, see </usr/share/doc/python3/html/library/stdtypes.html#bytes>]) — a stream of encoded text. Encoding errors will be tolerated. 🫧

Return type: None

startCommit(jobState=False) (abstractmethod)

Updates the status of the job on the disk. 💾

It might bump the job's version number, and it might kick off an asynchronous process. 🏃‍♀️ Call waitForCommit() to wait on that process. You must call waitForCommit() before committing any more updates to the job. 🚨

While that async process is running, it's totally fine to modify the job. Those changes just won't get committed until the next call. 🦋

Parameters

• jobState (bool, see </usr/share/doc/python3/html/library/functions.html#bool>) — if True, commit the state of the FileStore's job and the file deletes. Otherwise, only commit file creates and updates. 🎀

Return type: None

waitForCommit() (abstractmethod)

Blocks while startCommit is running. ⏳

This gets called by the job's successor, so it doesn't start messing with the job store until this job is done. 🤝

It might also get called when startCommit was never called on that particular instance, in which case it just doesn't block. 🙃

Returns

Always returns True. 💖

Return type: bool, see </usr/share/doc/python3/html/library/functions.html#bool>

shutdown(shutdown_info) (classmethod, abstractmethod)

Parameters

• shutdown_info (Any)

Return type: None

Shuts down the filestore on this node. 🌙

This is meant to be called when the batch system shuts down. 🔌

Parameters

• shutdown_info (Any) — the implementation-specific shutdown info, used to shut down the file store, remove all its state, and delete all job-local temp directories from the node. 🧹

Return type: None

---

✨ toil.fileStores.cachingFileStore ✨

💎 Attributes

• logger
• SQLITE_TIMEOUT_SECS

🚨 Exceptions

• CacheError — error raised if the user tries to add a non-local file to the cache. 🙅‍♀️
• CacheUnbalancedError — raised if the file store can't free up enough space for caching. 😩
• IllegalDeletionCacheError — error raised if the caching code finds a file that represents a reference to a cached file that has gone missing. 👻
• InvalidSourceCacheError — error raised if the user tries to add a non-local file to the cache. 🙅‍♀️

🎀 Classes

• CachingFileStore — a cache-enabled file store. 🧠

📚 Module Contents

toil.fileStores.cachingFileStore.logger

toil.fileStores.cachingFileStore.SQLITE_TIMEOUT_SECS = 60.0

exception toil.fileStores.cachingFileStore.CacheError(message)

Bases: Exception, see </usr/share/doc/python3/html/library/exceptions.html#Exception>

Error raised if the user tries to add a non-local file to the cache. 🙅‍♀️

exception toil.fileStores.cachingFileStore.CacheUnbalancedError

Bases: CacheError

Raised if the file store can't free up enough space for caching. 😭

Its message is 'Unable unable to free enough space for caching. This error frequently arises due to jobs using...'. 💀

exception toil.fileStores.cachingFileStore.IllegalDeletionCacheError(deletedFile)

Bases: CacheError

Error raised if the caching code finds a file that represents a reference to a cached file that has gone missing. 👻

This can be a big problem if a hard link is moved, because then the cache won't be able to evict the file it links to. 😱

Remember that files read with readGlobalFile may not be deleted by the user and need to be deleted with deleteLocalFile. 🫵

exception toil.fileStores.cachingFileStore.InvalidSourceCacheError(message)

Bases: CacheError

Error raised if the user tries to add a non-local file to the cache. 🙅‍♀️

class toil.fileStores.cachingFileStore.CachingFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)

Bases: toil.fileStores.abstractFileStore.AbstractFileStore

A cache-enabled file store. 🧠✨

It hands out files as symlinks or hard links into a cache directory for the node, as long as the workflow allows it. 🔗

It also tries to write files back to the backing JobStore asynchronously, right after quickly taking them into the cache. Writes only have to finish when the job's actual state after running gets committed back to the job store. 🫶

Internally, it manages caching with a database. Each node has its own database, shared between all the workers on that node. 🗄️ The database has several tables:

• files has one entry for each file in the cache. Each entry knows the path to its data on disk, its global file ID, its state, and its owning worker PID. If the owning worker dies, another worker picks it up. It also knows its size. 📂

File states are:

• "cached": happily stored in the cache. Reads can happen immediately. The owner is null. Anyone can adopt it and move it to "deleting", but only if it has no outstanding immutable references. 🥰
• "downloading": in the process of being saved to the cache by a non-null owner. Reads have to wait until the state becomes "cached". If the worker dies, it goes to "deleting", because we don't know whether it was fully downloaded or whether anyone still needs it. No references can be created to a "downloading" file except by the worker responsible for downloading it. 🚧
• "uploadable": stored in the cache and ready to be written to the job store by a non-null owner. It transitions to "uploading" when a (thread of the) owning worker process picks it up and starts uploading it, either to free up cache space or to commit a completed job. If the worker dies, it goes to "cached", because it may still have outstanding immutable references from the dead-but-not-cleaned-up job that was going to write it. 📤
• "uploading": stored in the cache and being written to the job store by a non-null owner. It transitions to "cached" once it's successfully uploaded. If the worker dies, it goes to "cached", because it may still have outstanding immutable references from the dead-but-not-cleaned-up job that was writing it. 🦋
• "deleting" — the file is getting yeeted out of the cache by a non-null owner 🗑️ and will eventually be removed from the database. Bye bestie 👋

The refs table holds one entry for every outstanding reference to a cached file (hard link, symlink, or full copy) 🔗. It's called refs and not references because references is an SQL reserved word 🙅. It remembers which job ID has the reference and the path where that reference lives. References come in three states:

• "immutable" — a hardlink or symlink to a file in the cache 🦋. It dedicates the file's size in bytes from the job's disk requirement to the cache, so it can cache this file or keep other files without references around. It can get upgraded to "copying" if the link can't actually be created.
• "copying" — a file in the cache is mid-copy to a path ✨. It'll eventually be upgraded to a mutable reference.
• "mutable" — a file from the cache was copied to a certain path 💜. These only exist to support the deleteLocalFile API. Only files with nothing but mutable references (or no references at all) are eligible for eviction.

The jobs table has one entry for each job currently running 🏃‍♀️. It keeps track of the job's ID, the worker that's supposed to be running it, the job's disk requirement, and the job's local temp dir path that needs cleaning up 🧹. When workers check for jobs whose workers have died, they null out the old worker and grab ownership of the jobs and their references, cleaning up until the null-worker jobs are gone. Iconic cleanup behavior 💅.

The properties table holds key, value pairs for tracking total space available and whether caching is free for this run 🌸.

---

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)
• jobDesc (toil.job.JobDescription)
• file_store_dir (str)
• waitForPreviousCommit (Callable[[], Any])

Here's the attribute lineup, bestie 🎀:

forceNonFreeCaching = False
forceDownloadDelay = None
contentionBackoff = 15
localCacheDir
jobName: str = ''
jobID
jobDiskBytes: float | None = None
workflowAttemptNumber
dbPath
process_identity_lock
commitThread = None

---

as_process()

Assumes the process's identity so it can act on the caching database 🪪. It yields the process's name in the caching database and holds onto a lock while your thread has it 🔒.

Return type: collections.abc.Generator[str, None, None]

con (property)

Gets the database connection to be used for the current thread 💖.

Return type: sqlite3.Connection

cur (property)

Gets the main cursor to be used for the current thread ✨.

Return type: sqlite3.Cursor

getCacheLimit()

Returns the total number of bytes the cache is limited to 📏. If no limit is available, it raises an error 🙈.

getCacheUsed()

Returns the total number of bytes used in the cache 📦. If no value is available, it raises an error 💀.

getCacheExtraJobSpace()

Returns the total number of bytes of disk space requested by jobs running against this cache but not yet used 🫠. Here's the tea: the jobs on the node can take up all the space, and then they want to write to or read from the cache. So when that happens, we need to debit space from them somehow... 😭 If no value is available, it raises an error.

getCacheAvailable()

Returns the total number of free bytes available for caching 🌈. If the number is negative, it's the total number of bytes of cached files that need to be evicted to free up enough space for all the currently scheduled jobs. If no value is available, it raises an error.

getSpaceUsableForJobs()

Returns the total number of bytes that aren't taken up by job requirements, ignoring files and file usage 🍓. We can never run more jobs than we actually have room for, even with caching. If it's not retrievable, it raises an error.

getCacheUnusedJobRequirement()

Returns the total number of bytes of disk space requested by the current job and not used by the files the job is using in the cache 🧁. Mutable references don't count, but immutable and uploading ones do. If no value is available, it raises an error.

adjustCacheLimit(newTotalBytes)

Adjusts the total cache size limit to the given number of bytes ⚖️.

fileIsCached(fileID)

Returns true if the given file is currently cached, and false otherwise 👀. Be careful though, because this can't really be relied upon. A file may go from cached to deleting right after you look at it 😬. If you need to do something with the file, do it in a transaction.

getFileReaderCount(fileID)

Returns the number of current outstanding reads of the given file 📖. This counts mutable references too.

cachingIsFree()

Returns true if files can be cached for free, without taking up space, and false otherwise 🆓. This is true when you're working with certain job stores in certain configurations, most notably the FileJobStore 🦋.

open(job)

This context manager decorated method lets you do cache-specific operations before and after a job runs in worker.py 🧚.

Parameters: job (toil.job.Job)

Return type: collections.abc.Generator[None, None, None]

writeGlobalFile(localFileName, cleanup=False)

Creates a file in the jobstore and returns a FileID reference 📝.

readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False)

Makes the file associated with fileStoreID available locally 🫶. If mutable is True, a copy of the file gets created locally, so the original isn't modified and doesn't change the file for other jobs. If mutable is False, a link can be created instead, which saves disk resources 💾. The downloaded file will be executable if and only if it was originally uploaded from an executable file on the local filesystem ⚡.

If a userPath is specified, it's used as the destination. If it isn't, the file is stored in the local temp directory with an encoded name 🗂️.

The destination file must not be deleted by you. It can only be deleted through deleteLocalFile 🚫. Implementations must call logAccess() to report the download.

Parameters

• fileStoreID — job store id for the file
• userPath — a path to the name of the file to which the global file will be copied or hard-linked (see below).
• cache — described in toil.fileStores.CachingFileStore.readGlobalFile()
• mutable — described in toil.fileStores.CachingFileStore.readGlobalFile()
• symlink — True if the caller can accept a symlink, False if the caller can only accept a normal file or hardlink

Returns: An absolute path to a local, temporary copy of the file keyed by fileStoreID 🌟.

readGlobalFileStream(fileStoreID, encoding=None, errors=None)

Reads a stream from the job store, similar to readGlobalFile 🌊. The yielded file handle doesn't need to be closed explicitly, and you shouldn't do it either 🙅.

Parameters

• encoding — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which represents binary mode.
• errors — an optional string that specifies how encoding errors are handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

Implementations must call logAccess() to report the download 📣.

Returns: a context manager yielding a file handle that can be read from 💅.

deleteLocalFile(fileStoreID)

Deletes the local copies of files associated with the provided job store ID 🧼.

Okay so, if there are no local copies to delete, this one raises an OSError with an errno of errno.ENOENT 🙈 which means you literally can't call it multiple times in a row, bestie 💀

Here's what actually gets deleted ✨ all the files this job previously read from this file ID with readGlobalFile into the job's file-store-provided temp directory, PLUS the file that was written to create the given file ID, but only if the current job wrote it from that same file-store-provided temp directory 🎀

Parameters

• fileStoreID — the File Store ID of the file to be deleted 🌸

deleteGlobalFile(fileStoreID)

Okay, this one deletes the local files and then permanently deletes them from the job store 😭 Plot twist: it waits. The delete won't happen until after the job's run method has finished, so the job can still be restarted if it needs to 🫶

Parameters

• fileStoreID — the File Store ID of the file to be deleted 🌸

exportFile(jobStoreFileID, dstUrl)

Parameters

• jobStoreFileID (toil.fileStores.FileID)
• dstUrl (str)

Return type

• None

export_file(file_id, dst_uri)

Same energy, different outfit 💅

Parameters

• file_id (toil.fileStores.FileID)
• dst_uri (str)

Return type

• None

waitForCommit()

This one blocks while startCommit is running 🛑 Its job is to make sure the successor job doesn't start messing with the job store until this job is done doing its thing. Bestie protecting the boundaries 🧚

It might get called even when startCommit was never called on that particular instance, and in that case it just doesn't block. No drama 💜

Returns

• Always returns True

Return type

• bool

startCommit(jobState=False)

This updates the status of the job on disk 💖 It may bump the version number of the job, and it may kick off an asynchronous process. If it does, call waitForCommit() to wait on that process 🦋

You must call waitForCommit() before committing any further updates to the job, no exceptions 🚨 While the async process is running, it's totally safe to modify the job. Those modifications just won't get committed until the next call ✨

Parameters

• jobState — If True, commit the state of the FileStore's job, and file deletes. Otherwise, commit only file creates/updates 🍓

startCommitThread(state_to_commit)

This runs in a thread and actually commits the current job 🧵

Parameters

• state_to_commit (Optional[toil.job.JobDescription])

classmethod shutdown(shutdown_info)

Parameters

• shutdown_info (tuple[str, str]) — a tuple of the coordination directory (where the cache database lives) and the cache directory (where the cached data lives) 📦

Return type

• None

Job local temp directories get removed automatically because they show up in the database 🧹

__del__()

Cleanup function that runs when the class instance gets destroyed. It makes sure all the file writing threads exit properly 🌈

---

✨ toil.fileStores.nonCachingFileStore ✨

Attributes

• logger

Classes

• NonCachingFileStore — Interface used to allow user code run by Toil to read and write files 💖

Module Contents

toil.fileStores.nonCachingFileStore.logger: logging.Logger

class toil.fileStores.nonCachingFileStore.NonCachingFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)

Bases: toil.fileStores.abstractFileStore.AbstractFileStore

This is the interface that lets user code run by Toil read and write files 🎀 It also gives user code access to other Toil facilities, including:

• normal (non-real-time) logging 📝
• finding the correct temporary directory for scratch work 🗂️
• importing and exporting files into and out of the workflow 🦋

It stores user files in the jobStore, but keeps them separate from actual jobs. It may implement caching 💅

It gets passed as an argument to the toil.job.Job.run() method. File access is only allowed inside the context manager provided by toil.fileStores.abstractFileStore.AbstractFileStore.open() 🔒

It's also responsible for committing completed jobs back to the job store with an update operation, and it lets you wait on that commit operation too ⏳

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)
• jobDesc (toil.job.JobDescription)
• file_store_dir (str)
• waitForPreviousCommit (Callable[[], Any])

Attributes

• jobStateFile: str | None = None
• localFileMap: DefaultDict[str, list[str]]

static check_for_coordination_corruption(coordination_dir)

Checks that the coordination directory hasn't been deleted unexpectedly 😱 Slurm has been known to delete XDG_RUNTIME_DIR out from under the processes it promised to leave alone, so in certain misconfigured environments the coordination directory and everything in it could vanish. To catch that, this regularly makes sure the things we expect to exist actually exist, and aborts if they don't 🚨

Parameters

• coordination_dir (Optional[str])

Return type

• None

check_for_state_corruption()

Makes sure the state tracking information hasn't been deleted unexpectedly 🕵️‍♀️

Return type

• None

open(job)

This creates the context manager that wraps tasks before and after a job runs 🎀 File operations are only permitted inside this context manager.

Implementations must only yield from within with super().open(job):. Don't skip this or things get messy 💀

Parameters

• job (toil.job.Job) — the job instance of the Toil job to run

Return type

• collections.abc.Generator[None, None, None]

writeGlobalFile(localFileName, cleanup=False)

Uploads a file (as a path) to the job store ☁️

If the file lives in a FileStore-managed temporary directory (i.e. one from toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir()), it becomes a local copy of the file, which makes it eligible for deletion by toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile() 🗑️

If an executable file on the local filesystem gets uploaded, its executability is preserved when it's downloaded again 🏃‍♀️

Parameters

• localFileName (str) — the path to the local file to upload. The last path component (the basename of the file) stays associated with the file in the file store, if the backing JobStore supports it, so the file can be searched for by name or name glob 🔍
• cleanup (bool) — when this is True, the copy of the global file gets deleted once the job and all of its successors are done running 🧹. If it's False, the global file has to be deleted manually, so don't forget, bestie 💅.

You get back an ID 🪪 that you can use to pull the file up later.

Return type: toil.fileStores.FileID

🎀 readGlobalFile 🎀

readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False)

This makes the file tied to fileStoreID available locally 🌸.

If mutable is True, you get your own local copy, so the original stays untouched and doesn't change the file for any other jobs 🛡️. If mutable is False, a link can be made to the file instead, which saves disk space 💾.

The downloaded file is executable if and only if it was originally uploaded from an executable file on the local filesystem ⚡.

If you give a userPath, that's where the file goes. If you don't, the file lands in the local temp directory with an encoded name 🗂️.

⚠️ Don't delete the destination file yourself! The only way to delete it is through deleteLocalFile 🙅‍♀️.

Implementations must call logAccess() to report the download 📣.

Parameters:

• fileStoreID (str) — the job store ID for the file
• userPath (Optional[str]) — a path to the name of the file where the global file gets copied or hard-linked (more on that below) 📁
• cache (bool) — described in toil.fileStores.CachingFileStore.readGlobalFile() 💜
• mutable (bool) — described in toil.fileStores.CachingFileStore.readGlobalFile() 💜
• symlink (bool) — True if the caller can accept a symlink, False if the caller can only accept a normal file or a hardlink 🔗

Returns: an absolute path to a local, temporary copy of the file keyed by fileStoreID 🧚

Return type: str

🦋 readGlobalFileStream 🦋

readGlobalFileStream(fileStoreID: str, encoding: Literal[None] = None, errors: str | None = None) -> ContextManager[IO[bytes]]

readGlobalFileStream(fileStoreID: str, encoding: str, errors: str | None = None) -> ContextManager[IO[str]]

This reads a stream from the job store. It's basically the vibe of readGlobalFile, just streamed ✨.

You do not need to close the yielded file handle, and you actually shouldn't close it explicitly 🙅‍♀️.

Parameters:

• encoding — the name of the encoding used to decode the file. Encodings are the same as the ones for decode(). It defaults to None, which means binary mode 🔢.
• errors — an optional string that says how encoding errors get handled. These are the same as the ones for open(). It defaults to 'strict' when an encoding is specified 🚨.

Implementations must call logAccess() to report the download 📣.

Returns: a context manager that yields a readable file handle 📖.

🌈 exportFile 🌈

exportFile(jobStoreFileID, dstUrl)

Parameters:

• jobStoreFileID (toil.fileStores.FileID)
• dstUrl (str)

Return type: None

🌸 export_file 🌸

export_file(file_id, dst_uri)

Parameters:

• file_id (toil.fileStores.FileID)
• dst_uri (str)

Return type: None

💀 deleteLocalFile 💀

deleteLocalFile(fileStoreID)

This deletes the local copies of files tied to the job store ID you pass in 🗑️.

⚠️ If there are no local copies to delete, it raises an OSError with an errno of errno.ENOENT. That means you can't call it multiple times in a row 😭.

The files it deletes are all the ones this job has previously read from this file ID with readGlobalFile into the job's file-store-provided temp directory 🧺. It also deletes the file that was written to create the given file ID, but only if the current job wrote it from that same file-store-provided temp directory 🫶.

Parameters:

• fileStoreID (str) — the File Store ID of the file to be deleted

Return type: None

🔥 deleteGlobalFile 🔥

deleteGlobalFile(fileStoreID)

This deletes local files first, and then permanently deletes them from the job store 💥.

To keep the job restartable if it needs to be, the delete doesn't happen until after the job's run method has finished 🔁.

Parameters:

• fileStoreID (str) — the File Store ID of the file to be deleted

Return type: None

⭐ waitForCommit ⭐

waitForCommit()

This blocks while startCommit is running ⏳. The job's successor calls it so it doesn't start messing with the job store until this job is totally done 🙅‍♀️.

It might get called when startCommit was never called on that particular instance. In that case, it just doesn't block 🫠.

Returns: always returns True ✅

Return type: bool

👑 startCommit 👑

startCommit(jobState=False)

This updates the status of the job on disk 💾.

It may bump the job's version number 🔢.

It may also start an asynchronous process. Call waitForCommit() to wait on that process. You must call waitForCommit() before committing any further updates to the job 🛑. While the async process is running, it's totally safe to modify the job, but any modifications made after this call won't get committed until the next call 🧠.

Parameters:

• jobState (bool) — if True, it commits the state of the FileStore's job and the file deletes. Otherwise, it only commits the file creates and updates 📝

Return type: None

🧹 __del__ 🧹

__del__()

This is the cleanup function that runs when a class instance is getting destroyed. There's nothing to do here, since there are no async write events 🙃.

Return type: None

🦄 classmethod shutdown 🦄

classmethod shutdown(shutdown_info)

Parameters:

• shutdown_info (str) — the coordination directory 🗂️

Return type: None

---

✨ Classes ✨

• FileID — a small wrapper around Python's builtin string class 🎀

✨ Package Contents ✨

💖 class toil.fileStores.FileID 💖

class toil.fileStores.FileID(fileStoreID, size, executable=False)

Bases: str

FileID is a small wrapper around Python's builtin string class 🧁. It represents a file's ID in the file store, and it has a size attribute that holds the file's size in bytes 📏. This object is what importFile and writeGlobalFile give you back 🎁.

Calls into the file store can use bare strings too. If the size isn't in the ID, it gets queried from the job store 🔍.

Parameters:

• fileStoreID (str)
• size (int)
• executable (bool)

Attributes:

• size
• executable = False
• pack() — packs the FileID into a string so it can be passed through external code 📦

Okay so the return type here is str 💖. The docs for it live at /usr/share/doc/python3/html/library/stdtypes.html#str if you wanna peek ✨

🪄 classmethod forPath(fileStoreID, filePath) 🪄

classmethod forPath(fileStoreID, filePath)

• Parameters
  • fileStoreID (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
  • filePath (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: FileID <#toil.fileStores.FileID> 🦋

🎀 classmethod unpack(packedFileStoreID) 🎀

classmethod unpack(packedFileStoreID)

Unpack the result of pack() into a FileID object. 🫶

• Parameters
  • packedFileStoreID (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: FileID <#toil.fileStores.FileID> 👑

---

🧚 toil.job 🧚

🌈 Attributes 🌈

These are the attributes in toil.job, no cap, and they're all lowkey iconic:

• logger
• REQUIREMENT_NAMES
• ParsedRequirement
• ParseableIndivisibleResource
• ParseableDivisibleResource
• ParseableFlag
• ParseableAcceleratorRequirement
• ParseableRequirement
• T
• Promised

💥 Exceptions 💥

Here's the exception lineup. Some of these are a little dramatic, so read them carefully bestie 😭

• JobPromiseConstraintError — the job was asked to promise its return value, but that value isn't available yet 🙈
• ConflictingPredecessorError — the common base class for all non-exit exceptions 🧱
• DebugStoppingPointReached — raised when a job reaches a point where it was told to stop for debugging 🐛
• FilesDownloadedStoppingPointReached — raised when a job stops because it was asked to download its files, and the files are downloaded ✅
• JobException — the general job exception 💅
• JobGraphDeadlockException — raised if a workflow has an unresolvable dependency, like a cycle. Check out toil.job.Job.checkJobGraphForDeadlocks() for more 💀
• TemporaryID 🦋 is a placeholder for a job ID that hasn't been registered yet, and a JobDescription uses it until registration happens.
• AcceleratorRequirement ✨ is the requirement for one or more computational accelerators, like a GPU or FPGA. 💻
• RequirementsDict 🎀 is typed storage for a job's requirements.
• Requirer 🧚 is the base class that handles the storage and presentation of requirements.
• JobBodyReference 🔗 is a reference from a job description to its body.
• JobDescription 📋 stores everything the Toil Leader could ever need to know about a Job. 👑
• ServiceJobDescription 🌸 is a description of a job that hosts a service.
• CheckpointJobDescription 💾 is a description of a job that is a checkpoint.
• Job 💅 is the class that represents a unit of work in toil. Main character energy, obviously.
• FunctionWrappingJob 🍓 is a job used to wrap a function. When its run method runs, the wrapped function gets called.
• JobFunctionWrappingJob 🌈 is a job function, meaning a function whose first argument is a Job.
• PromisedRequirementFunctionWrappingJob 🦋 handles dynamic resource allocation using toil.job.Promise instances.
• PromisedRequirementJobFunctionWrappingJob 💜 handles dynamic resource allocation for job functions.
• EncapsulatedJob 🫶 is a convenience Job class that makes a job subgraph look like a single job.
• ServiceHostJob ⭐ is a job that runs a service. Toil uses it internally, so bestie, you should subclass Service instead of using this one 🙈.
• FileMetadata 📁 is metadata for a file.
• CombineImportsJob 🔥 combines the outputs of multiple WorkerImportsJobs into one promise.
• WorkerImportJob 🧚 is a job that does file imports on a worker instead of on the leader. It assumes all local and cloud files are accessible.
• ImportsJob 🗂️ organizes and delegates files to individual WorkerImportJobs.
• Promise 🎀 references a return value from a method as a promise before the method itself has run.
• PromisedRequirement 💖 is a class for dynamically allocating job function resource requirements.
• UnfulfilledPromiseSentinel 💀 should be overwritten by a proper promised value. Don't leave it sitting there, periodt.

✨ Functions ✨

okay bestie, here's the lineup of helper functions 💅 each one has a very specific job, so let's go 🎀

• parse_accelerator(spec) — parses an AcceleratorRequirement that someone typed in as user code 🎀
• accelerator_satisfies(candidate, requirement[, ignore]) — checks whether a candidate partially satisfies the given requirement ✨
• accelerators_fully_satisfy(candidates, requirement[, ...]) — figures out whether a whole set of accelerators satisfies a requirement 🔥
• potential_absolute_uris(uri, path[, importer, ...]) — gets the potential absolute URIs to check for an imported file 🦋
• get_file_sizes(filenames, file_source[, search_paths, ...]) — resolves relative-URI files in the given environment and turns them into absolute normalized URIs 🌸 it returns a dictionary of the string values from the WDL file values
• unwrap(p) — makes sure you actually have a promised value, and not just a promise 🫶
• unwrap_all(p) — makes sure you actually have a collection of promised values 💀

---

✨ Module Contents ✨

• toil.job.logger 🧚 — just the logger, listed by name, no extra deets 💅

💜 JobPromiseConstraintError

exception toil.job.JobPromiseConstraintError(promisingJob, recipientJob=None)

• Bases: RuntimeError

This error is for when a job asks to promise its return value, but that value isn't available yet 😭 (it's because the return value hasn't been hit in the topological order of the job graph.)

• Parameters:
  • promisingJob (Job) 🎀
  • recipientJob (Optional[Job]) 🎀
• Attributes:
  • promisingJob
  • recipientJob = None

🌈 ConflictingPredecessorError

exception toil.job.ConflictingPredecessorError(predecessor, successor)

• Bases: Exception

This is the common base class for all non-exit exceptions ✨

• Parameters:
  • predecessor (Job) 🦋
  • successor (Job) 🦋

💀 DebugStoppingPointReached

exception toil.job.DebugStoppingPointReached

• Bases: BaseException

Raised when a job reaches a point where it's been told to stop for debugging 🙈

🍓 FilesDownloadedStoppingPointReached

exception toil.job.FilesDownloadedStoppingPointReached(message, host_and_job_paths=None)

• Bases: DebugStoppingPointReached

Raised when a job stops because it was asked to download its files, and the files have been downloaded 🫶

• Parameters:
  • message (str) 🌸
  • host_and_job_paths (Optional[list[tuple[str, str]]]) 🌸
• Attributes:
  • host_and_job_paths = None

🧚 TemporaryID

class toil.job.TemporaryID

This is a placeholder for an unregistered job ID that a JobDescription uses. Three places need to hold onto it:

• JobDescription objects hold it to record normal relationships 💖
• Jobs hold it to key their connected-component registries, and to record predecessor relationships so an EncapsulatedJob can add itself as a child 🌸
• Services hold it to tie back to their hosting jobs, so the service tree can be built up from Service objects ✨
• Methods:
  • __str__() → str 💅
  • __repr__() → str 💅
  • __hash__() → int 💅
  • __eq__(other) → bool 💅
    • other (Any)
  • __ne__(other) → bool 💅
    • other (Any)

🔥 AcceleratorRequirement

class toil.job.AcceleratorRequirement

• Bases: TypedDict

This is the requirement for one or more computational accelerators, like a GPU or FPGA 🔥 It has these fields:

• count: int — how many of the accelerator are needed to run the job 👑
• kind: str — what kind of accelerator is required. It can be "gpu". Other kinds defined in the future might be "fpga", etc. 💜
• model: typing_extensions.NotRequired[str] — what model of accelerator is needed. The exact set of values depends on what the backing scheduler calls its accelerators, and strings like "nvidia-tesla-k80" might be expected to work. If a specific model isn't required, this should be absent 🙈
• brand: typing_extensions.NotRequired[str] — what brand or manufacturer of accelerator is required. The exact set of values depends on what the backing scheduler calls the brands of its accelerators, and strings like "nvidia" or "amd" might be expected to work. If a specific brand isn't required (for example, because the job can use multiple brands of accelerator that support a given API), this should be absent 🦋
• api: typing_extensions.NotRequired[str] — what API is used to talk to the accelerator. It can be "cuda". Other APIs supported in the future might be "rocm", "opencl", "metal", etc. If the job doesn't need a particular API to talk to the accelerator, this should be absent 🌈

🌸 parse_accelerator

toil.job.parse_accelerator(spec)

This parses an AcceleratorRequirement specified by user code 🎀 It supports formats like these:

>>> parse_accelerator(8)
{'count': 8, 'kind': 'gpu'}

>>> parse_accelerator("1")
{'count': 1, 'kind': 'gpu'}

>>> parse_accelerator("nvidia-tesla-k80")
{'count': 1, 'kind': 'gpu', 'model': 'nvidia-tesla-k80', 'brand': 'nvidia'}

>>> parse_accelerator("nvidia-tesla-k80:2")
{'count': 2, 'kind': 'gpu', 'model': 'nvidia-tesla-k80', 'brand': 'nvidia'}

>>> parse_accelerator("gpu")
{'count': 1, 'kind': 'gpu'}

>>> parse_accelerator("cuda:1")
{'count': 1, 'kind': 'gpu', 'api': 'cuda', 'brand': 'nvidia'}

>>> parse_accelerator({"kind": "gpu"})
{'count': 1, 'kind': 'gpu'}

>>> parse_accelerator({"brand": "nvidia", "count": 5})
{'count': 5, 'kind': 'gpu', 'brand': 'nvidia'}

If nothing is specified, it assumes we're talking about GPUs, and about one of them 🔥 it knows that "gpu" is a kind, "cuda" is an API, and "nvidia" is a brand 💅

• Raises:
  • ValueError — if it gets something it can't parse 😭
  • TypeError — if it gets something it can't parse because it's the wrong type 💀

Parameters 💅

• spec (Union[int, str, dict[str, Union[str, int]]]) — the bestie of this function can be an int, a str, or a dict with str keys whose values are either a str or an int 🎀

Return type 🦋 AcceleratorRequirement

✨ toil.job.accelerator_satisfies(candidate, requirement, ignore=[]) ✨

Okay so this one checks if a candidate at least partially satisfies the given requirement 💖

Returns 🌸 True if the candidate at least partially satisfies the requirement. It checks every field except count, so the number of accelerators doesn't matter here 🙈

Parameters 🧚

• candidate (AcceleratorRequirement)
• requirement (AcceleratorRequirement)
• ignore (list[str])

Return type 👑 bool

✨ toil.job.accelerators_fully_satisfy(candidates, requirement, ignore=[]) ✨

This one is giving full check energy 💅 It figures out if a whole set of accelerators satisfies a requirement, and it ignores any fields listed in ignore 🔥

Returns 🌈 True if the AcceleratorRequirement requirement is fully satisfied by the ones in the list, taken together. This time it checks every field, count included ⭐

Parameters 🍓

• candidates (Optional[list[AcceleratorRequirement]])
• requirement (AcceleratorRequirement)
• ignore (list[str])

Return type 💜 bool

---

🎀 class toil.job.RequirementsDict 🎀

Bases: TypedDict

This is the typed storage for a job's requirements, and it's lowkey iconic because each requirement can have a different type depending on what it is 🦋

• cores — typing_extensions.NotRequired[int | float]
• memory — typing_extensions.NotRequired[int]
• disk — typing_extensions.NotRequired[int]
• accelerators — typing_extensions.NotRequired[list[AcceleratorRequirement]]
• preemptible — typing_extensions.NotRequired[bool]

toil.job.REQUIREMENT_NAMES = ['disk', 'memory', 'cores', 'accelerators', 'preemptible'] 🌸

🧚 Parseable type aliases 🧚

These are the type aliases Toil uses for parsing requirements. There's no extra description for them, so just know they exist 🫶

• toil.job.ParsedRequirement
• toil.job.ParseableIndivisibleResource
• toil.job.ParseableDivisibleResource
• toil.job.ParseableFlag
• toil.job.ParseableAcceleratorRequirement
• toil.job.ParseableRequirement

---

💖 class toil.job.Requirer(requirements) 💖

Base class that handles storing and presenting requirements. It's the main character of the requirements world, and it has cores, memory, disk, and preemptability as properties ✨

Parameters 🌈

• requirements (Mapping[str, ParseableRequirement])

assignConfig(config)

Assigns the given config object so it can provide default values. This has to be called exactly once on a loaded JobDescription before any requirements get queried, so don't forget 💀

Parameters

• config (toil.common.Config) — the config object to query

Return type None

__getstate__()

Returns the dict that becomes the instance's __dict__ when pickling 🔮

Return type dict[str, Any]

__copy__()

Returns a semantically-shallow copy of the object, for copy.copy() 🦋

Return type Requirer

__deepcopy__(memo)

Returns a semantically-deep copy of the object, for copy.deepcopy() 🫧

Parameters

• memo (Any)

Return type Requirer

property requirements

Gets the dict containing all non-None, non-defaulted requirements. Literally the full list of what the job needs 📋

Return type RequirementsDict

property disk

Gets the maximum number of bytes of disk required 💾

Return type int

property memory

Gets the maximum number of bytes of memory required 🧠

Return type int

property cores

Gets the number of CPU cores required 🖥️

Return type Union[int, float]

property preemptible

Says whether a preemptible node is permitted, or whether a nonpreemptible one is required. It's giving yes-or-no drama 🙈

Return type bool

preemptable(val)

Parameters

• val (ParseableFlag)

Return type None

property accelerators

Gets any accelerators that are needed, like GPUs 🎮

Return type list[AcceleratorRequirement]

scale(requirement, factor)

Returns a copy of this object with the given requirement scaled up or down. It only works on requirements where scaling actually makes sense, so no scaling your vibes 😭

Parameters

• requirement (str)
• factor (float)

Return type Requirer

requirements_string()

Gets a nice human-readable string of our requirements 📝

Return type str

---

🍓 class toil.job.JobBodyReference 🍓

Bases: NamedTuple

This is a reference from a job description to its body, basically the job's little pointer to where the actual work lives 🧚

• file_store_id (str) — the File ID of the job's body. For the root job, it's a special shared file name instead 🌸
• module_string (str) — a stringified description of the module needed to load the body 🔮

---

👑 class toil.job.JobDescription(requirements, jobName, unitName='', displayName='', local=None, files=None) 👑

Bases: Requirer

This is the one place that stores everything the Toil Leader ever needs to know about a Job. It's basically the job's whole resume 💅 It includes:

• Resource requirements. 💪
• Which jobs are children, follow-ons, or predecessors of this job. 👯
• A reference to the Job object in the job store. 🗂️

You can get it from an actual, executable Job object, and you can use it to get the Job object back from the JobStore. It never holds other Jobs or JobDescriptions, though. Everything gets referenced by ID, no drama 🫶

It's subclassed into variants for checkpoint jobs and service jobs, which each have their own specific parameters 🦋

Parameters 🌈

• requirements (Mapping[str, Union[int, str, float, bool, list]])
• jobName (str)
• unitName (Optional[str])
• displayName (Optional[str])

Okay so the leftover bits of the list first, bestie 💅

• local (Optional[bool]) 🫶
• files (Optional[set[toil.fileStores.FileID]]) 🎀

Now the actual attributes, and yes they're all giving main character 👑

• local: bool, defaults to False 🦋
• jobName 💖
• unitName, defaults to '' 🌸
• displayName, defaults to '' ✨
• jobStoreID: str or TemporaryID (toil.job.TemporaryID) 🧚
• filesToDelete, defaults to [] 💜
• predecessorNumber, defaults to 0 🍓
• predecessorsFinished 👑
• childIDs: set[str] 🌈
• followOnIDs: set[str] 🔥
• successor_phases: list[set[str]] ⭐
• serviceTree 🦋
• logJobStoreFileID, defaults to None 💀
• files_to_use 🙈

---

get_names()

Okay this one is literally your ID card 🪪 It gets the names and ID of this job, bundled up as a named tuple.

Return type: toil.bus.Names

get_chain()

Ngl this is giving time travel ⏳ It gets all the jobs that ran in this job's chain, in order. Each job comes back as a named tuple with its various names plus its original job store ID. The jobs are listed in execution order. If the job hasn't run yet or it didn't chain, you just get a one-item list 🌸

Return type: list[toil.bus.Names]

serviceHostIDsInBatches()

This one finds all the batches of service host job IDs that can start at the same time 🔥 They come back in the order they need to start in.

Return type: Iterator[list[str]]

successorsAndServiceHosts()

Gets an iterator over all child, follow-on, and service job IDs 💖

Return type: Iterator[str]

allSuccessors()

Gets an iterator over all child, follow-on, and chained, inherited successor job IDs. Follow-ons always come before children 🫶

Return type: Iterator[str]

successors_by_phase()

Gets an iterator over all child, follow-on, and chained inherited successor job IDs, along with their phase number on the stack 🧚 Phases run from higher numbers down to lower numbers.

Return type: Iterator[tuple[int, str]]

services (property)

Gets a collection of the IDs of service host jobs for this job, in arbitrary order ✨ It'll be empty if the job has no unfinished services.

has_body()

Returns True if we have a job body associated, and False otherwise 🎀

Return type: bool

attach_body(file_store_id, user_script)

Attaches a job body to this JobDescription 🦋 You give it the file store ID the body lives at, plus the required user script module. The file store ID can also be "firstJob" for the root job, which is stored as a shared file instead.

Parameters

• file_store_id (str)
• user_script (toil.resource.ModuleDescriptor)

Return type: None

detach_body()

Drops the body reference from a JobDescription 🙈

Return type: None

get_body()

Gets the info you need to load the job body 💅 It returns a file store ID (or the magic shared file name "firstJob") and a user script module.

Return type: tuple[str, toil.resource.ModuleDescriptor]

Heads up bestie: this one fails if no body is attached, so check has_body() first 💀

nextSuccessors()

Returns the collection of job IDs for the successors of this job that are ready to run 🔥 If those jobs have multiple predecessor relationships, they might still be blocked on other jobs.

It returns None when we're at the final phase (all successors are done). It returns an empty collection if there are more phases but they can't be entered yet, for example because we're still waiting on the job itself to run 🌈

Return type: Optional[set[str]]

filterSuccessors(predicate)

Keeps only the successor jobs that the given predicate function approves 👑 The predicate gets called with the job's ID. Every other successor gets treated as complete and forgotten, no cap 💀

Parameters

• predicate (Callable[[str], bool])

Return type: None

filterServiceHosts(predicate)

Keeps only the services that the given predicate approves 🧚 The predicate gets called with the service host job's ID. Every other service gets treated as complete and forgotten 💜

Parameters

• predicate (Callable[[str], bool])

Return type: None

clear_nonexistent_dependents(job_store)

Removes all references to child, follow-on, and associated service jobs that don't exist anymore 🫶 That means every one that's been completed and removed.

Parameters

• job_store (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type: None

clear_dependents()

Removes all references to successor and service jobs. Full clean slate 🧹✨

Return type: None

is_subtree_done()

Checks if the subtree is done 🦋 It returns True if the job appears to be done, and all related child, follow-on, and service jobs appear to be finished and removed.

Return type: bool

replace(other)

Takes on the ID of another JobDescription while keeping our own state and type 🌸 When it gets updated in the JobStore, we'll save over the other JobDescription.

This is super useful for chaining jobs, since the chained-to job can replace the parent job 🔥 It also merges the cleanup state and successors (other than this job) from the job being replaced into this one.

Parameters

• other (JobDescription): the job description to replace

Return type: None

assert_is_not_newer_than(other)

Makes sure this JobDescription isn't newer than a prospective new version of the JobDescription 🙈

Parameters

• other (JobDescription)

Return type: None

is_updated_by(other)

Returns True if the passed JobDescription is a distinct, newer version of this one 💖

Parameters

• other (JobDescription)

Return type: bool

addChild(childID)

Makes the job with the given ID a child of the described job 🧚

Parameters

• childID (str)

Return type: None

addFollowOn(followOnID)

Makes the job with the given ID a follow-on of the described job 🫶

Parameters

• followOnID (str)

Return type: None

addServiceHostJob(serviceID, parentServiceID=None)

Okay so this one is a whole vibe 💅 It makes the ServiceHostJob with the given ID a service of the job we're describing 🦋

If you pass a parent ServiceHostJob ID, that parent service gets started first 🔥 and it has to have already been added. No cutting the line, bestie 🙈

hasChild(childID)

Checks if the job with the given ID is a child of the described job 🌸 It returns True if it is.

Parameters

• childID (str)

Return type: bool

hasFollowOn(followOnID)

Tests if the job with the given ID is a follow-on of the described job 💜

Parameters

• followOnID (str)

Return type: bool

hasServiceHostJob(serviceID)

Tests if the ServiceHostJob is a service of the described job ✨

Return type: bool

renameReferences(renames)

This applies the given dict of ID renames to every reference to jobs 🦋 It does not touch our own ID or the IDs of finished predecessors 🙅‍♀️ Any IDs that aren't in the renames dict stay exactly as they are.

Parameters

• renames (dict[TemporaryID, str]) — the rename operations to apply

Return type: None

addPredecessor()

Lets the JobDescription know that a predecessor has been added to its Job 🎀

Return type: None

onRegistration(jobStore)

Does setup work that needs the JobStore 🧚 The Job saving logic calls this once the JobDescription meets the JobStore and has its ID assigned.

It's overridden to handle setup like hooking up flag files for service jobs, which needs the JobStore to exist first 🌈

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore) — the job store we are being placed into

Return type: None

setupJobAfterFailure(exit_status=None, exit_reason=None)

This is the post-failure glow-up 💖 It reduces remainingTryCount if it's greater than zero, and bumps the memory up to at least the default memory. Why? Running out of memory is super common, so this is a preventative move 🫶

You need a configuration assigned first (see toil.job.Requirer.assignConfig()) or this won't work.

Parameters

• exit_status (Optional[int]) — the exit code from the job
• exit_reason (Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]) — why the job stopped, if the batch system can tell us

Return type: None

getLogFileHandle(jobStore)

Creates a context manager that yields a file handle to the log file 📝 It assumes logJobStoreFileID is already set, so make sure that's happening first 💅

property remainingTryCount

Gets the number of tries left 🔢 This is the try count set on the JobDescription. If none is set, you get the default, which is based on the retry count in the config.

clearRemainingTryCount()

Clears remainingTryCount and resets it to its default value 🧹

It returns True if the JobDescription was actually modified, and False if it wasn't.

Return type: bool

__str__()

Produces a useful logging string that identifies this job 🪪

Return type: str

__repr__()

This one's listed without a description, so there's no extra tea on it here 🍵

reserve_versions(count)

Reserves job version numbers for later, so they can be journaled asynchronously 📚

Parameters

• count (int)

Return type: None

pre_update_hook()

Runs right before a created or updated version of this job gets pickled and saved 🫧 The job store is the one that calls it.

Return type: None

---

class toil.job.ServiceJobDescription(*args, **kwargs)

Bases: JobDescription

This is a description of a job that hosts a service 🏠

terminateJobStoreID: str | None = None
startJobStoreID: str | None = None
errorJobStoreID: str | None = None

onRegistration(jobStore)

Sets up flag files 🚩 When a ServiceJobDescription first meets the JobStore, this is where it gets its flag files set up.

---

class toil.job.CheckpointJobDescription(*args, **kwargs)

Bases: JobDescription

This is a description of a job that is a checkpoint 📍

checkpoint: JobBodyReference | None = None
checkpointFilesToDelete = []

set_checkpoint()

Saves a body checkpoint into self.checkpoint 💾

Return type: str

restore_checkpoint()

Restores the body checkpoint from self.checkpoint ⏪

Return type: None

restartCheckpoint(jobStore)

Restarts a checkpoint after the total failure of the jobs in its subtree 🔁 The changes get written to the jobStore right away. All of the checkpoint's successors are deleted, but its try count does not go down.

It returns a list with the IDs of any successors that were deleted 📋

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type: list[str]

---

class toil.job.Job(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)

This class represents a unit of work in toil 👑 It's the main character of the whole show ✨

Parameters

• memory (Optional[ParseableIndivisibleResource])
• cores (Optional[ParseableDivisibleResource])
• disk (Optional[ParseableIndivisibleResource])
• accelerators (Optional[ParseableAcceleratorRequirement])
• preemptible (Optional[ParseableFlag])
• preemptable (Optional[ParseableFlag])
• unitName (Optional[str])
• checkpoint (Optional[bool])
• displayName (Optional[str])
• descriptionClass (Optional[type])
• local (Optional[bool])
• files (Optional[set[toil.fileStores.FileID]])

userModule: toil.resource.ModuleDescriptor

__str__()

Produces a useful logging string that identifies this Job and sets it apart from its JobDescription 🔍

Return type: str

check_initialized()

Makes sure that Job.__init__() has been called by any subclass __init__() 🛡️ This works because the self._description instance variable should always be set after __init__().

If __init__() hasn't been called, it raises an error 🚨

Return type: None

property jobStoreID: str | TemporaryID

Gets the ID of this Job 🪪

Return type: Union[str, TemporaryID]

• property description: JobDescription: okay bestie, this one exposes the JobDescription that describes this job 💖
  • Return type: JobDescription
• property disk: int: the max number of bytes of disk the job will need to run 💾
  • Return type: int
• property memory: int: the max number of bytes of memory the job will need to run 🧠
  • Return type: int
• property cores: int | float: the number of CPU cores required 🔥
  • Return type: Union[int, float]
• property accelerators: list[AcceleratorRequirement]: any accelerators the job needs, like GPUs 🎮
  • Return type: list[AcceleratorRequirement]
• property preemptible: bool: whether the job can run on a preemptible node 🦋
  • Return type: bool
• preemptable() 🙈
  • Return type: bool
• property checkpoint: bool: determines whether the job is a checkpoint job or not ✨
  • Return type: bool
• property files_to_use: set[toil.fileStores.FileID] 📁
  • Return type: set[toil.fileStores.FileID]
• add_to_files_to_use(val) ➕
  • Parameters: val (toil.fileStores.FileID)
  • Return type: None
• remove_from_files_to_use(val) ➖
  • Parameters: val (toil.fileStores.FileID)
  • Return type: None
• assignConfig(config): assigns the given config object 🎀
  • It gets used by various actions inside the Job class, so it's kind of a big deal.
  • Parameters: config (toil.common.Config) -- the Config object to query
  • Return type: None
• run(fileStore): override this one to do the actual work and dynamically create successor jobs 💅
  • Parameters: fileStore (toil.fileStores.abstractFileStore.AbstractFileStore) -- this is what you use to make local and globally shareable temp files and send log messages to the leader process 📝
  • Returns: whatever the function returns can be passed to other jobs with toil.job.Job.rv() 🎁
  • Return type: Any
• addChild(childJob): adds a childJob to run as a child of this job 👶
  • Child jobs run right after this job's toil.job.Job.run() method finishes.
  • Returns: childJob, for call chaining
  • Parameters: childJob (Job)
  • Return type: Job
• hasChild(childJob): checks if childJob is already a child of this job 👀
  • Returns: True if childJob is a child of the job, otherwise False.
  • Parameters: childJob (Job)
  • Return type: bool
• addFollowOn(followOnJob): adds a follow-on job 🌸
  • Follow-on jobs run after the child jobs and their successors have finished.
  • Returns: followOnJob, for call chaining
  • Parameters: followOnJob (Job)
  • Return type: Job
• hasPredecessor(job): checks if a given job is already a predecessor of this job 🔍
  • Parameters: job (Job)
  • Return type: bool
• hasFollowOn(followOnJob): checks if the given job is already a follow-on of this job 🧐
  • Returns: True if followOnJob is a follow-on of this job, otherwise False.
  • Parameters: followOnJob (Job)
  • Return type: bool
• addService(service, parentService=None): adds a service, and services are iconic 🌈
  • The service's toil.job.Job.Service.start() method gets called after run finishes but before any successors run. Its toil.job.Job.Service.stop() method gets called once the successors have run.
  • Services let things like databases and servers get started and accessed by jobs in a workflow. Literally the main character of workflows 👑
  • > Raises toil.job.JobException if the service has already been made the child of a job or another service.
  • Parameters:
    • service (Job) -- the service to add
    • parentService (Optional[Job]) -- a service that gets started before service is started. This lets you build trees of services. parentService must be a service of this job.
  • Returns: a promise that gets replaced with the return value from toil.job.Job.Service.start() of the service, in any successor of the job ✨
  • Return type: Promise
• hasService(service): returns True if the given Service is a service of this job, and False otherwise 💜
  • Parameters: service (Job)
  • Return type: bool
• addChildFn(fn, *args, **kwargs): adds a function as a child job 🧚
  • Parameters: fn (Callable) -- the function to run as a child job, with *args and **kwargs as its arguments. See toil.job.FunctionWrappingJob for the reserved keyword arguments you use to set resource requirements.
  • Returns: the new child job that wraps fn
  • Return type: FunctionWrappingJob
• addFollowOnFn(fn, *args, **kwargs): adds a function as a follow-on job 🦋
  • Parameters: fn (Callable) -- the function to run as a follow-on job, with *args and **kwargs as its arguments. See toil.job.FunctionWrappingJob for the reserved keyword arguments you use to set resource requirements.
  • Returns: the new follow-on job that wraps fn
  • Return type: FunctionWrappingJob
• addChildJobFn(fn, *args, **kwargs): adds a job function as a child job 🫶
  • See toil.job.JobFunctionWrappingJob for what a job function is.
  • Parameters: fn (Callable) -- the job function to run as a child job, with *args and **kwargs as its arguments. See toil.job.JobFunctionWrappingJob for the reserved keyword arguments you use to set resource requirements.
  • Returns: the new child job that wraps fn
  • Return type: FunctionWrappingJob
• addFollowOnJobFn(fn, *args, **kwargs): adds a follow-on job function 🌟
  • See toil.job.JobFunctionWrappingJob for what a job function is.

Okay so the Parameters for this one are giving main character energy 💖

• fn (Callable) — the job function that runs as a follow-on job, with *args and **kwargs passed along to it. Peep toil.job.JobFunctionWrappingJob for the reserved keyword arguments you use to set your resource requirements 💅

Returns

• The new follow-on job that wraps fn ✨

Return type

• FunctionWrappingJob (toil.job.FunctionWrappingJob) 🦋

---

🌸 property tempDir: str

Okay bestie, this is just a shortcut for calling job.fileStore.getLocalTempDir() 🧚 The temp dir gets created on the very first call, and every call after that gives you the same one. No cap, it's consistent. ✨ :return: Path to tempDir. See job.fileStore.getLocalTempDir

Return type

• str (/usr/share/doc/python3/html/library/stdtypes.html#str) 💜

📣 log(text, level=logging.INFO)

This one logs using fileStore.log_to_leader() 🔥

Parameters

• text (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

• None 💀

🎀 static wrapFn(fn, *args, **kwargs)

Makes a Job out of a function, and it's literally so iconic 😭 It's a convenience function for the constructor of toil.job.FunctionWrappingJob.

Parameters

• fn — the function to run with *args and **kwargs as its arguments. Check out toil.job.JobFunctionWrappingJob for the reserved keyword arguments used to set resource requirements 🍓

Returns

• The new function that wraps fn 🌈

Return type

• FunctionWrappingJob (toil.job.FunctionWrappingJob) ✨

👑 static wrapJobFn(fn, *args, **kwargs)

Makes a Job out of a job function. Also a convenience function, this time for the constructor of toil.job.JobFunctionWrappingJob 💅

Parameters

• fn — the job function to run with *args and **kwargs as its arguments. Same deal as before: check toil.job.JobFunctionWrappingJob for the reserved keyword arguments used to set resource requirements 🫶

Returns

• The new job function that wraps fn 💖

Return type

• JobFunctionWrappingJob (toil.job.JobFunctionWrappingJob) 🦋

🧚 encapsulate(name=None)

This encapsulates the job, so peep toil.job.EncapsulatedJob. It's a convenience function for the constructor of toil.job.EncapsulatedJob ✨

Parameters

• name (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str) — a human-readable name for the encapsulated job 🌸

Returns

• An encapsulated version of this job 🔥

Return type

• EncapsulatedJob (toil.job.EncapsulatedJob) 💜

🔮 rv(*path)

This one creates a promise (toil.job.Promise) 🦋 The promise stands in for a return value of the job's run method, or, if it's a function-wrapping job, the wrapped function's return value. Lowkey the best way to pass results around.

Parameters

• path ((Any)) — an optional path for picking out a component of the promised return value. If it's absent or empty, you get the whole return value. Otherwise, the first element of the path selects one item from the return value. For that to work, the return value has to be a list, a dictionary, or any other type that implements the __getitem__() magic method. If the selected item is another composite value, the second element of the path can select an item from it, and so on and so forth 💅 For example, if the return value is [6,{'a':42}], then .rv(0) selects 6, rv(1) selects {'a':3}, and rv(1,'a') selects 3. To grab a slice from a slicable return value (like a tuple or list), make the path element a slice object. For example, if the return value is [6, 7, 8, 9], then .rv(slice(1, 3)) selects [7, 8]. Note that slicing really only makes sense at the end of the path, so keep that in mind 👀

Returns

• A promise representing the return value of this job's toil.job.Job.run() method 🌈

Return type

• Promise (toil.job.Promise) ✨

🫧 registerPromise(path)

---

💌 prepareForPromiseRegistration(jobStore)

This one sets things up so this job's promises can register themselves. It preps this job (the promisor) so its promises can register with it whenever the jobs they're promised to (the promisees) get serialized. The promisee holds the reference to the promise, usually as part of the job arguments, and when it gets pickled, so do the promises it refers to. Pickling a promise is what triggers it to be registered with the promisor 🍓

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type

• None ✨

🚧 checkJobGraphForDeadlocks()

This ensures a graph of Jobs (one that hasn't been saved to the JobStore yet) doesn't have any pathological relationships between jobs that would cause deadlocks if we tried to run them. Bestie, deadlocks are not it 😭 For more info, see toil.job.Job.checkJobGraphConnected(), toil.job.Job.checkJobGraphAcyclic() and toil.job.Job.checkNewCheckpointsAreLeafVertices().

⚠️ Raises toil.job.JobGraphDeadlockException if the job graph is cyclic, contains multiple roots, or contains checkpoint jobs that aren't leaf vertices when defined (see toil.job.Job.checkNewCheckpointsAreLeaves()). Not a vibe.

🌳 getRootJobs()

This returns the set of root job objects that contain this job. A root job is a job with no predecessors, which means it's not a child, follow-on, or service 🦋

Only deals with jobs created here, rather than jobs loaded from the job store 🙈

Return type

• set[Job] (/usr/share/doc/python3/html/library/stdtypes.html#set, toil.job.Job) 💜

---

🧩 checkJobGraphConnected()

⚠️ Raises toil.job.JobGraphDeadlockException if toil.job.Job.getRootJobs() does not contain exactly one root job. Multiple roots is a whole deadlock situation 💀

Return type

• None 🫶

As execution always starts from one root job, having multiple root jobs will cause a deadlock to occur. Literally the one rule 💅

Only deals with jobs created here, rather than jobs loaded from the job store ✨

🕸️ checkJobGraphAcylic()

⚠️ Raises toil.job.JobGraphDeadlockException if the connected component of jobs containing this job contains any cycles of child/followOn dependencies in the augmented job graph (see below). Cycles like that are NOT allowed in valid job graphs 🙅‍♀️

Return type

• None 🌸

Okay so, a follow-on edge (A, B) between two jobs A and B is equivalent to adding a child edge to B from (1) A, (2) each child of A, and (3) the successors of each child of A. We call each of these an "implied" edge 🔮 The augmented job graph is a job graph that includes all the implied edges. Big brain stuff, but it's giving clarity ✨

For a job graph G = (V, E), the algorithm is O(|V|^2). It's O(|V| + |E|) for a graph with no follow-ons. The former follow-on case could be improved! 🌈

Only deals with jobs created here, rather than jobs loaded from the job store 🧚

🚩 checkNewCheckpointsAreLeafVertices()

A checkpoint job is one that gets restarted if either it fails, or if any of its successors completely fails and exhausts their retries 🔁 A job is a leaf if it has no successors.

A checkpoint job has to be a leaf when it's first added to the job graph. Once its run method is invoked, it can then create direct successors. This restriction is made to simplify the implementation, so it's not you, it's the code 🫶

Only works on connected components of jobs that haven't been added to the JobStore yet 🙈

⚠️ Raises toil.job.JobGraphDeadlockException if there exists a job being added to the graph for which checkpoint=True and which is not a leaf. Not allowed, periodt 💀

Return type

• None 💖

defer

defer(function, *args, **kwargs)

okay so defer is literally the cleanup crew 🧹✨ You register a deferred function (any callable) and Toil calls it once the current attempt at running this job wraps up. An attempt is considered concluded when the job function returns, raises an exception, or the process running it terminates abnormally. For class-based jobs, that means toil.job.Job.run() doing the same thing 💅

The deferred function runs on the node that attempted the job, even if a later attempt happens on a different node 🦋

⚠️ bestie, your deferred function should be idempotent 💀 (safe to run more than once), because it might get called multiple times on the same node or even in the same process.

You can register more than one deferred function per attempt by calling defer again with different arguments. If you register the same function twice, with the same or different arguments, it gets called twice per attempt. So yeah, double registering means double calls 👯

Typical examples are cleanup stuff that lives outside Toil's reach, like Docker containers, files sitting outside the work directory, etc. 🧹🌸

Parameters

• function (callable) — the function that gets called after this job concludes 🎯
• args (list, see /usr/share/doc/python3/html/library/stdtypes.html#list) — the arguments passed to the function
• kwargs (dict, see /usr/share/doc/python3/html/library/stdtypes.html#dict) — the keyword arguments passed to the function

Return type: None

---

class Runner

class Runner

This is the class that sets up and runs a Toil workflow, aka the main character 👑 Its static methods are below.

getDefaultArgumentParser

static getDefaultArgumentParser(jobstore_as_flag=False)

Gets an argument parser with the Toil workflow options already added. This is the Right Way to get an argument parser in a Toil Python workflow 💖

Parameters:

• jobstore_as_flag (bool, see /usr/share/doc/python3/html/library/functions.html#bool) — makes the job store option a --jobStore flag instead of a required jobStore positional argument

Returns: the argument parser used by a Toil workflow, with the Toil options added ✨

Return type: configargparse.ArgParser

getDefaultOptions

static getDefaultOptions(jobStore=None, jobstore_as_flag=False)

Gets the default options for a Toil workflow 🎀

Parameters:

• jobStore (Optional[toil.lib.misc.StrPath]) — a string describing the job store for the workflow
• jobstore_as_flag (bool, see /usr/share/doc/python3/html/library/functions.html#bool) — makes the job store option a --jobStore flag instead of a required jobStore positional argument

Returns: the options used by a Toil workflow

Return type: argparse.Namespace (see /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

addToilOptions

static addToilOptions(parser, jobstore_as_flag=False)

Adds the default Toil options to an optparse (see /usr/share/doc/python3/html/library/optparse.html#module-optparse) or argparse (see /usr/share/doc/python3/html/library/argparse.html#module-argparse) parser object 🧁

💅 Honestly, just use getDefaultArgumentParser() instead. It makes a parser of the correct class so Toil's config file and environment variables actually work. If the parser you pass in here is just an argparse.ArgumentParser and not a configargparse.ArgParser, the Toil config file and environment variables will not be respected 💀

Parameters:

• parser (Union[optparse.OptionParser, argparse.ArgumentParser]) — the options object you want to add Toil options to
• jobstore_as_flag (bool, see /usr/share/doc/python3/html/library/functions.html#bool) — makes the job store option a --jobStore flag instead of a required jobStore positional argument

Return type: None

startToil

static startToil(job, options)

Runs the Toil workflow using the given options 🚀

⚠️ This one is deprecated and has been replaced by toil.common.Toil.start. Sorry bestie, time to upgrade 🥺

(see Job.Runner.getDefaultOptions and Job.Runner.addToilOptions) starting with this job. :param job: root job of the workflow :raises: toil.exceptions.FailedJobsException if, at the end of the function, there are still failed jobs. :return: the return value of the root job's run function.

Parameters:

• job (Job, see <#toil.job.Job>) — the root job of the workflow

Return type: Any

---

class Service

class Service(memory=None, cores=None, disk=None,
accelerators=None, preemptible=None, unitName='')

Bases: Requirer

This is an abstract class that defines the interface for a service 🦋 You should subclass it to define your own services. A service is not executed as a job. It runs inside a ServiceHostJob 🏠

Parameters:

• memory (Optional[ParseableIndivisibleResource])
• cores (Optional[ParseableDivisibleResource])
• disk (Optional[ParseableIndivisibleResource])
• accelerators (Optional[ParseableAcceleratorRequirement])
• preemptible (Optional[ParseableFlag])
• unitName (Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str)

Class attributes

unitName = ''
jobName = 'Service'
hostID = None

start

abstractmethod start(job)

Starts the service 🎬

Parameters:

• job (ServiceHostJob, see <#toil.job.ServiceHostJob>) — the underlying host job the service runs in. You can use it to register deferred functions, or to access the fileStore for making temporary files 🧚

Returns: an object describing how to access the service. It must be pickleable, and jobs use it to access the service (see toil.job.Job.addService()) 💜

Return type: Any

stop

abstractmethod stop(job)

Stops the service. The function can block until it's complete 🛑

Parameters:

• job (ServiceHostJob, see <#toil.job.ServiceHostJob>) — the underlying host job the service runs in. You can use it to register deferred functions, or to access the fileStore for making temporary files

Return type: None

check

check()

Checks that the service is still running 👀

Raises exceptions.RuntimeError: if the service failed, the service job gets labeled as failed 💀

Returns: True if the service is still running, otherwise False. If it returns False, the service job is terminated and considered a success ✅

⚠️ Important point, bestie: if the service job exits because of a failure, it should raise a RuntimeError, not return False!

Return type: bool (see /usr/share/doc/python3/html/library/functions.html#bool)

---

getUserScript

getUserScript()

Return type: toil.resource.ModuleDescriptor (see <#toil.resource.ModuleDescriptor>) 📜

getTopologicalOrderingOfJobs

getTopologicalOrderingOfJobs()

Returns a list of jobs where, for every pair of indices i and j with i < j, the job at index i can run before the job at index j. It's giving order 🧠✨

Return type: list (see /usr/share/doc/python3/html/library/stdtypes.html#list) of Job (see <#toil.job.Job>)

This only considers jobs in this job's subgraph that are newly added, not jobs loaded from the job store. It ignores service jobs 🙈

saveBody(jobStore)

Okay so this one saves the execution data for just this job into the JobStore 💾✨, then fills in the JobDescription with everything needed to fetch it back later 🦋

bestie, the Job's JobDescription must have already been given a real jobStoreID before you call this 🙈

Also, it does not save the JobDescription itself 💀 lowkey a plot twist.

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore) — the job store you're stashing the job body into 🗄️

Return type: None

saveAsRootJob(jobStore)

This one saves the job to the given jobStore as the root job of the whole workflow 👑

Returns: the JobDescription describing this job 💖

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type: JobDescription

classmethod loadJob(job_store, job_description)

Retrieves a toil.job.Job instance from a JobStore. It's giving retrieval queen energy 🧚

Parameters

• job_store (toil.jobStores.abstractJobStore.AbstractJobStore) — the job store 🗄️
• job_description (JobDescription) — the JobDescription of the job you want to pull back out

Returns: the job referenced by the JobDescription 🌸

Return type: Job

set_debug_flag(flag)

Turns on the given debug option for the job 🔥

Parameters

• flag (str)

Return type: None

has_debug_flag(flag)

Returns true if the given debug flag is set. Quick vibe check, basically 🔍

Parameters

• flag (str)

Return type: bool

files_downloaded_hook(host_and_job_paths=None)

This is a function subclasses can call once they've finished downloading their input files 📥✨

heads up: if the "download_only" debug flag is set, this will abort the job 💀 so be careful!

You can hint it a list of file path pairs, one outside and one inside the job container. If you do, the container environment can be reconstructed 🫶

Parameters

• host_and_job_paths (Optional[list[tuple[str, str]]])

Return type: None

---

exception toil.job.JobException(message)

Bases: Exception

This is the general job exception, the catch-all for when a job goes sideways 😭

Parameters

• message (str)

exception toil.job.JobGraphDeadlockException(string)

Bases: JobException

This gets raised when a workflow has an unresolvable dependency, like a cycle 🔁💀 See toil.job.Job.checkJobGraphForDeadlocks().

class toil.job.FunctionWrappingJob(userFunction, *args, **kwargs)

Bases: Job

A job used to wrap a function. In its run method, the wrapped function gets called 🎀

Parameters

• userFunction (Callable[[Ellipsis], Any])
• args (Any)
• kwargs (Any)

Attributes you can peek at:

• userFunctionModule
• userFunctionName = ''

run(fileStore)

Override this to do the actual work and dynamically spin up successor jobs 💅

Parameters

• fileStore (toil.fileStores.abstractFileStore.AbstractFileStore) — used to make local and globally sharable temporary files and to send log messages to the leader process 📝

Returns: the return value of the function can be passed along to other jobs using toil.job.Job.rv() 🦋

Return type: Any

getUserScript()

Return type: str

class toil.job.JobFunctionWrappingJob(userFunction, *args, **kwargs)

Bases: FunctionWrappingJob

A job function is a function whose first argument is a Job instance, the wrapping job for the function. That means it can add successor jobs and use every method the Job class offers 👑

To give the job function access to the toil.fileStores.abstractFileStore.AbstractFileStore instance (see toil.job.Job.run()), it gets made into a variable of the wrapping job called fileStore 🫶

To specify a job's resource requirements, you can pass these default keyword arguments:

• memory
• disk
• cores
• accelerators
• preemptible

For example, to wrap a function into a job, you'd call:

Job.wrapJobFn(myJob, memory='100k', disk='1M', cores=0.1)

Parameters

• userFunction (Callable[[Ellipsis], Any])
• args (Any)
• kwargs (Any)

property fileStore

Return type: toil.fileStores.abstractFileStore.AbstractFileStore ✨

run(fileStore)

Override this to do the actual work and dynamically spin up successor jobs 💅

Parameters

• fileStore (toil.fileStores.abstractFileStore.AbstractFileStore) — used to make local and globally sharable temporary files and to send log messages to the leader process 📝

Returns: the return value of the function can be passed along to other jobs using toil.job.Job.rv() 🦋

Return type: Any

---

class toil.job.PromisedRequirementFunctionWrappingJob(userFunction, *args, **kwargs)

Bases: FunctionWrappingJob

This handles dynamic resource allocation using toil.job.Promise instances 🧠 It spawns a child function using the parent function's parameters and the fulfilled promised resource requirements.

classmethod create(userFunction, *args, **kwargs)

Creates an encapsulated Toil job function with unfulfilled promised resource requirements 🎁 Once the promises get fulfilled, a child job function is created with the updated resource values. The subgraph is encapsulated so this child job function runs before any other children in the workflow. Otherwise, a different child might try to use an unresolved promise return value from the parent 😭

run(fileStore)

Override this to do the actual work and dynamically spin up successor jobs 💅

Parameters

• fileStore — used to make local and globally sharable temporary files and to send log messages to the leader process 📝

Returns: the return value of the function can be passed along to other jobs using toil.job.Job.rv() 🦋

evaluatePromisedRequirements()

class toil.job.PromisedRequirementJobFunctionWrappingJob(userFunction, *args, **kwargs)

Bases: PromisedRequirementFunctionWrappingJob

This handles dynamic resource allocation for job functions. Check out toil.job.JobFunctionWrappingJob for more 💖

run(fileStore)

Override this to do the actual work and dynamically spin up successor jobs 💅

Parameters

• fileStore — used to make local and globally sharable temporary files and to send log messages to the leader process 📝

Returns: the return value of the function can be passed along to other jobs using toil.job.Job.rv() 🦋

---

class toil.job.EncapsulatedJob(job, unitName=None)

Bases: Job

A convenience Job class that makes a job subgraph appear to be a single job. It's giving illusion 🪄

Say A is the root job of a job subgraph, and B is another job you want to run after A and all its successors have finished. For that, you use encapsulate:

#  Job A and subgraph, Job B
A, B = A(), B()
Aprime = A.encapsulate()
Aprime.addChild(B)
#  B will run after A and all its successors have completed, A and its subgraph of
# successors in effect appear to be just one job.

If the job you're encapsulating already has predecessors (aka it isn't the root job), the encapsulated job inherits them automatically 🫶. But bestie, if you add predecessors to the job being encapsulated after you make the encapsulated job, the encapsulating job will NOT inherit those 😭. So double-check that the encapsulated job has the right set of predecessors ✨.

⚠️ Careful! Predecessors added late don't carry over, so make sure the encapsulated job ends up with the full set it needs. 💅

The return value of an encapsulated job (what you get from toil.job.Job.rv()) is the return value of the root job 👑. That means A().encapsulate().rv() and A().rv() resolve to the same value once A or A.encapsulate() has run 💖.

Parameters

• job — type: Optional[Job]
• unitName — type: Optional[str]

---

addChild(childJob)

Add a childJob to run as a child of this job 🌸. Child jobs kick off right after this job's toil.job.Job.run() method finishes.

Returns: childJob, so you can chain calls 🔗

Parameters: childJob (Job)

Return type: Job

---

addService(service, parentService=None)

Add a service 🧚. The toil.job.Job.Service.start() method of the service gets called after the run method finishes, but before any successors run. Then its toil.job.Job.Service.stop() method gets called once the job's successors have all run 🏁.

Services are how things like databases and servers get started up and accessed by jobs in a workflow 🎀.

Raises toil.job.JobException — if the service has already been made the child of a job or another service. Yikes 💀

Parameters

• service — the Service to add.
• parentService — a Service that will be started before service is started. This lets you build trees of services 🌳. parentService must be a service of this job.

Returns: a promise that gets replaced with the return value from toil.job.Job.Service.start() of service, in any successor of the job 🦋.

---

addFollowOn(followOnJob)

Add a follow-on job 💜. Follow-on jobs run after the child jobs and their successors have all been run.

Returns: followOnJob, for call chaining

Parameters: followOnJob (Job)

Return type: Job

---

rv(*path)

Create a promise (toil.job.Promise) ✨. This promise stands in for a return value of the job's run method. If the job is wrapping a function, it stands in for that function's return value instead.

Parameters: path ((Any)) — an optional path for picking out one piece of the promised return value 🍓. If the path is absent or empty, you get the entire return value. Otherwise, the first element of the path picks out an individual item from the return value. For that to work, the return value has to be a list, a dictionary, or some other type that implements the __getitem__() magic method. If the item you picked is another composite value, the second element of the path can pick an item out of that, and so on 🪆.

Some examples, because they're iconic:

• If the return value is [6,{'a':42}], then .rv(0) selects 6.
• rv(1) selects {'a':3}.
• rv(1,'a') selects 3.
• To select a slice from a return value that can be sliced (like a tuple or list), the path element should be a slice object. Say the return value is [6, 7, 8, 9]. Then .rv(slice(1, 3)) selects [7, 8] 🌈.

Heads up: slicing only really makes sense at the end of the path 🙈.

Returns: a promise representing the return value of this job's toil.job.Job.run() method.

Return type: Promise

---

prepareForPromiseRegistration(jobStore)

Set things up so this job's promises can register themselves 💅. You prepare this job (the promisor) so its promises can register with it whenever the jobs they're promised to (the promisees) get serialized.

The promisee holds the reference to the promise, usually as part of the job arguments. When it gets pickled, so do the promises it refers to. Pickling a promise is what triggers it to register with the promisor 🔗.

---

__reduce__()

Called during pickling to define the pickled representation of the job 🥹. We don't want to pickle our internal references to the job we encapsulate, so we leave them out. When it actually runs, we're just a no-op job that can maybe chain ⛓️.

---

getUserScript()

---

class toil.job.ServiceHostJob(service)

Bases: Job

A job that runs a service 🛠️. Toil uses this internally. Users should subclass Service instead of using this one, bestie 🫶.

Attributes:

• serviceModule
• service
• pickledService = None

property fileStore — Returns the file store, which the Service may need 🗂️.

---

addChild(child)

Add a childJob to run as a child of this job 🌸. Child jobs run directly after this job's toil.job.Job.run() method has completed.

Returns: childJob, for call chaining

---

addFollowOn(followOn)

Add a follow-on job 💜. Follow-on jobs run after the child jobs and their successors have been run.

Returns: followOnJob, for call chaining

---

addService(service, parentService=None)

Add a service 🧚. The toil.job.Job.Service.start() method of the service gets called after the run method finishes, but before any successors run. The service's toil.job.Job.Service.stop() method gets called once the job's successors have been run 🏁.

Services let things like databases and servers get started and accessed by jobs in a workflow 🎀.

Raises toil.job.JobException — if the service has already been made the child of a job or another service 💀.

Parameters

• service — the Service to add.
• parentService — a Service that will be started before service is started. This lets you build trees of services 🌳. parentService must be a service of this job.

Returns: a promise that gets replaced with the return value from toil.job.Job.Service.start() of service, in any successor of the job 🦋.

---

saveBody(jobStore)

Serializes the service itself before saving the host job's body 📦.

---

run(fileStore)

Override this to do your work and dynamically create successor jobs 🔥.

Parameters: fileStore — used to create local and globally sharable temporary files, and to send log messages to the leader process 📣.

Returns: the return value of the function can be passed to other jobs using toil.job.Job.rv() ✨.

---

getUserScript()

---

class toil.job.FileMetadata

Bases: NamedTuple

Metadata for a file 🧾. Here's the breakdown:

• source is the URL to grab the file from.
• parent_dir is the parent directory of the source.
• size is the size of the file. It's None if the filesize can't be retrieved 🙈.

Fields:

• source: str
• parent_dir: str
• size: int | None

---

toil.job.potential_absolute_uris(uri, path, importer=None, execution_dir=None)

Get the potential absolute URIs to check for an imported file 🔍.

Given a URI or bare path, this yields, one at a time, every URI (with schemes) where we should actually try to find it. It considers the paths or URIs we want to search under or against, the current directory, and the importing WDL document if there is one 🦋.

Parameters

• uri (str)
• path (list[str])
• importer (Optional[str])
• execution_dir (Optional[str])

Return type: Iterator[str]

✨ toil.job.get_file_sizes ✨

toil.job.get_file_sizes(filenames, file_source, search_paths=None, include_remote_files=True, execution_dir=None)

okay bestie, this one is giving file detective 🕵️‍♀️🔍 It takes relative-URI files in the environment you give it and turns them into absolute normalized URIs ✨ You get back a dictionary that maps the string values from the WDL file values to a tuple of the normalized URI, parent directory ID, and size of the file. The size can be None, which just means the size is unknown 🙈

Parameters

• filenames (List[str]) — the list of filenames to evaluate on 📂
• file_source (toil.jobStores.abstractJobStore.AbstractJobStore) — the context to search for files with 🔎
• task_path — the dotted WDL name of the user-level code doing the importing (probably the workflow name) 💖
• search_paths (Optional[List[str]]) — if you set this, it tries resolving the input location relative to the URLs or directories in this list 🗺️
• include_remote_files (bool) — if set, files get imported from remote locations. If not, they stay as URI references 🌍
• execution_dir (Optional[str]) — no description given, so we're vibing 🤷‍♀️

Return type

Dict[str, FileMetadata]

---

🧩 toil.job.CombineImportsJob 🧩

class toil.job.CombineImportsJob(d, **kwargs)

Bases: Job

this job is the glue queen 👑 It combines the outputs of multiple WorkerImportsJob into one promise ✨

Parameters

• d (Sequence[Promised[Dict[str, toil.fileStores.FileID]]]) 💅

run(file_store) just merges the dicts 🫶

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type

Promised[Dict[str, toil.fileStores.FileID]]

---

🦋 toil.job.WorkerImportJob 🦋

class toil.job.WorkerImportJob(filenames, local=False, **kwargs)

Bases: Job

this one does its file imports on a worker instead of the leader 🏃‍♀️💨 It assumes all local and cloud files are accessible, so make sure they actually are, bestie 😭

For the CWL/WDL runners, this class is only used when runImportsOnWorkers is enabled. So if that's off, this one is just chilling 🧊

Parameters

• filenames (List[str])
• local (bool)
• kwargs (Any)

Attribute: filenames

static import_files(files, file_source)

this method imports a list of files into the jobstore and gives you back a mapping of each filename to its associated FileIDs 🗂️✨

When stream is true but the import is not streamable, the worker will run out of disk space and run a new import job with enough disk space instead. That's a whole plot twist, so watch out 🙈

:param files: the list of files to import :param file_source: AbstractJobStore :return: a dictionary mapping filenames to the associated jobstore FileID

Parameters

• files (List[str]) 📄
• file_source (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type

Dict[str, toil.fileStores.FileID]

run(file_store) imports the workflow inputs, then creates and runs the workflow. It returns a promise of the workflow outputs 🎀

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type

Promised[Dict[str, toil.fileStores.FileID]]

---

🌸 toil.job.ImportsJob 🌸

class toil.job.ImportsJob(file_to_data, max_batch_size, import_worker_disk, **kwargs)

Bases: Job

this job is the organizer 🗂️💜 It sorts out files and hands them off to individual WorkerImportJobs

For the CWL/WDL runners, this is only used when runImportsOnWorkers is enabled. Same energy as before, bestie 💅

Parameters

• file_to_data (Dict[str, FileMetadata])
• max_batch_size (ParseableIndivisibleResource)
• import_worker_disk (ParseableIndivisibleResource)
• kwargs (Any)

run(file_store) imports the workflow inputs, then creates and runs the workflow 🌈

Return: a tuple. The first part is a mapping from the candidate URI to the file ID. The second part is a mapping of the source filenames to their metadata. The candidate URI is a field in the file metadata 🧾

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type

Tuple[Promised[Dict[str, toil.fileStores.FileID]], Dict[str, FileMetadata]]

---

🔮 toil.job.Promise 🔮

class toil.job.Promise(job, path)

this is a placeholder with main character energy ✨ It references a return value from a method as a promise before that method has actually run 🕰️

It references a return value from a toil.job.Job.run() or toil.job.Job.Service.start() method as a promise before the method itself is run.

Here's the tea ☕ Let T be a job. Instances of Promise (termed a promise) are returned by T.rv(), which is how you reference the return value of T's run function. When the promise gets passed to the constructor (or as an argument to a wrapped function) of a different, successor job, the promise gets swapped out for the actual referenced return value. This is how one job's return values can become input arguments for another job before the first job's run function has even executed 🤯

Parameters

• job (Job)
• path (Any)

Attribute: filesToDelete

a set of IDs of files containing promised values, for when we know we won't need them anymore 🗑️

Attributes: job, path 💖

__reduce__()

returns the Promise class and its construction arguments. This gets called during pickling, when a promise (an instance of this class) is about to be pickled 🥒

The construction arguments are evaluated during unpickling. They're the job store coordinates of a file that will hold the promised return value. By the time the promise is about to be unpickled, that file should already be populated, so make sure it is 🫶

---

🧁 toil.job.T 🧁

🧁 toil.job.Promised 🧁

---

🌷 toil.job.unwrap(p) 🌷

toil.job.unwrap(p)

this function makes sure you actually have a promised value and not just a promise 💅 It's mostly useful for keeping the type-checker happy 🙈

The "unwrap" terminology is borrowed from Rust, so if you're a Rustacean, this one's for you 🦀

Parameters

• p (Promised[T])

Return type

T

---

🌷 toil.job.unwrap_all(p) 🌷

toil.job.unwrap_all(p)

same vibe as unwrap, but for a whole collection of promised values. It makes sure none of the promises are left hanging around 🧺 It's mostly useful for type-checking too 💖

Same deal with the name: the "unwrap" terminology is borrowed from Rust 🦀

Parameters

• p (Sequence[Promised[T]])

Return type

Sequence[T]

---

🌟 toil.job.PromisedRequirement 🌟

class toil.job.PromisedRequirement(valueOrCallable, *args)

this class lets you dynamically allocate job function resource requirements 💪 Use it when your resource requirements depend on the return value of a parent function 🧠

(It involves toil.job.Promise instances, bestie.)

You can modify a PromisedRequirement by passing in a function that takes the promise as input, which is very galaxy-brain of you 🌌

For example, let f, g, and h be functions. Then a Toil workflow can be defined like this:

A = Job.wrapFn(f)
B = A.addChildFn(g, cores=PromisedRequirement(A.rv())
C = B.addChildFn(h, cores=PromisedRequirement(lambda x: 2*x, B.rv()))

Parameters

• valueOrCallable (Any)
• args (Any)

getValue()

returns the PromisedRequirement value 💖

Return type

Any

static convertPromises(kwargs)

returns True if a reserved resource keyword is a Promise or PromisedRequirement instance ✨ It also converts a Promise instance into a PromisedRequirement, so it's a whole glow-up 🦋

Parameters

• kwargs (dict[str, Any]) — the function keyword arguments 🎀

Return type

bool

---

💀 toil.job.UnfulfilledPromiseSentinel 💀

class toil.job.UnfulfilledPromiseSentinel(fulfillingJobName, file_id, unpickled)

this is a placeholder that's supposed to get overwritten by a proper promised value 🙅‍♀️ If it ever gets unpickled, it throws an exception, so don't let that happen, bestie 😭

Parameters

• fulfillingJobName (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) 💖
• file_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀
• unpickled (Any) ✨

fulfillingJobName 🦋

file_id 🌸

🌙 static __setstate__(stateDict)

Okay so this one only runs when you're unpickling, bestie 🥹

🚫 It won't get unpickled unless the promise is still unresolved, so we throw an exception. Not cute, but that's the rule 😭

Parameters

• stateDict (dict[str, Any]) (see /usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

• None

---

🗂️ toil.jobStores

Submodules 🧚

• toil.jobStores.abstractJobStore

🗂️ toil.jobStores.abstractJobStore

✨ Attributes ✨

• logger

🚨 Exceptions 🚨

• ProxyConnectionError — Dummy class. 🫶
• LocatorException — Base exception class for all locator exceptions. 💜
• InvalidImportExportUrlException — Common base class for all non-exit exceptions. 🌈
• NoSuchJobException — Indicates that the specified job does not exist. 💀
• ConcurrentFileModificationException — Indicates that the file was attempted to be modified by multiple processes at once. 😭
• NoSuchFileException — Indicates that the specified file does not exist. 🙈
• NoSuchJobStoreException — Indicates that the specified job store does not exist. 🍓
• JobStoreExistsException — Indicates that the specified job store already exists. 👑

🧱 Classes 🧱

• AbstractJobStore — Represents the physical storage for the jobs and files in a Toil workflow. 🔥
• JobStoreSupport — A mostly fake JobStore to access URLs not really associated with real job 🧚

📦 Module Contents 📦

• toil.jobStores.abstractJobStore.logger

exception toil.jobStores.abstractJobStore.ProxyConnectionError 💅

Bases: BaseException (see /usr/share/doc/python3/html/library/exceptions.html#BaseException)

Dummy class.

exception toil.jobStores.abstractJobStore.LocatorException(error_msg, locator, prefix=None) 💖

Bases: Exception (see /usr/share/doc/python3/html/library/exceptions.html#Exception)

Base exception class for all locator exceptions. For example, job store/aws bucket exceptions where they already exist. It's giving organized chaos 🌸

Parameters

• error_msg (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• locator (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• prefix (Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str)

exception toil.jobStores.abstractJobStore.InvalidImportExportUrlException(url) 🦋

Bases: Exception (see /usr/share/doc/python3/html/library/exceptions.html#Exception)

Common base class for all non-exit exceptions.

Parameters

• url (urllib.parse.ParseResult, see /usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult)

exception toil.jobStores.abstractJobStore.NoSuchJobException(jobStoreID) 🧚

Bases: Exception (see /usr/share/doc/python3/html/library/exceptions.html#Exception)

Indicates that the specified job does not exist.

Parameters

• jobStoreID (Union[toil.fileStores.FileID, str])

exception toil.jobStores.abstractJobStore.ConcurrentFileModificationException(jobStoreFileID) 🌈

Bases: Exception (see /usr/share/doc/python3/html/library/exceptions.html#Exception)

Indicates that the file was attempted to be modified by multiple processes at once. Chaotic but make it make sense 💀

Parameters

• jobStoreFileID (Union[toil.fileStores.FileID, str])

exception toil.jobStores.abstractJobStore.NoSuchFileException(jobStoreFileID, customName=None, *extra) 🍓

Bases: Exception (see /usr/share/doc/python3/html/library/exceptions.html#Exception)

Indicates that the specified file does not exist.

Parameters

• jobStoreFileID (Union[toil.fileStores.FileID, str])
• customName (Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str)
• extra (Any)

exception toil.jobStores.abstractJobStore.NoSuchJobStoreException(locator, prefix) 👑

Bases: LocatorException

Indicates that the specified job store does not exist.

Parameters

• locator (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• prefix (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

exception toil.jobStores.abstractJobStore.JobStoreExistsException(locator, prefix) 🔥

Bases: LocatorException

Indicates that the specified job store already exists.

Parameters

• locator (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• prefix (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

class toil.jobStores.abstractJobStore.AbstractJobStore(locator) 🧱

Bases: abc.ABC (see /usr/share/doc/python3/html/library/abc.html#abc.ABC)

Represents the physical storage for the jobs and files in a Toil workflow.

JobStores are responsible for storing toil.job.JobDescription (which relate jobs to each other) and files.

Actual toil.job.Job objects are stored in files, referenced by JobDescriptions. All the non-file CRUD methods the JobStore provides deal in JobDescriptions and not full, executable Jobs.

To actually get ahold of a toil.job.Job, use toil.job.Job.loadJob() with a JobStore and the relevant JobDescription.

Parameters

• locator (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

✨ initialize(config) ✨

Initialize this job store. 🌷

Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store.

Parameters

• config (toil.common.Config) — the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID.

Raises JobStoreExistsException — if the physical storage for this job store already exists 😭

Return type

• None

🫶 writeConfig()

Return type: None 💖

🎀 resume()

Okay so resume() is the method that hooks this instance up to the physical storage it represents 🔌 and loads the Toil configuration into the AbstractJobStore.config attribute. Iconic behavior, honestly 💅

• Raises NoSuchJobStoreException (<#toil.jobStores.abstractJobStore.NoSuchJobStoreException>) — if the physical storage for this job store doesn't exist 💀
• Return type: None

💜 property config: toil.common.Config

This returns the Toil configuration tied to this job store ⚙️✨

• Raises AttributeError (/usr/share/doc/python3/html/library/exceptions.html#AttributeError) — if the config hasn't been assigned yet, which is the case during resume() 🙈
• Return type: toil.common.Config (<#toil.common.Config>)

🦋 property locator: str

This gets the locator that defines the job store, and you can use it to connect to the job store later 🔗 Main character energy.

• Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🌸 rootJobStoreIDFileName = 'rootJobStoreID'

Just a class attribute, but it's giving name-and-tag energy 🏷️

🧚 setRootJob(rootJobStoreID)

This sets the root job of the workflow that's backed by this job store 👑

• Parameters
  • rootJobStoreID (toil.fileStores.FileID, <#toil.fileStores.FileID>)
• Return type: None

🧚 set_root_job(job_id)

Same vibe as setRootJob, just the snake_case version, and it sets the root job of the workflow backed by this job store 🐍💖

• Parameters
  • job_id (toil.fileStores.FileID, <#toil.fileStores.FileID>) — The ID of the job to set as root 🎯
• Return type: None

🍓 loadRootJob()

• Return type: toil.job.JobDescription (<#toil.job.JobDescription>)

🍓 load_root_job()

This loads the JobDescription for the root job in the current job store, so you can grab it 🫶

• Raises toil.job.JobException (<#toil.job.JobException>) — if no root job is set, or if the root job doesn't exist in this job store 😭
• Returns: The root job.
• Return type: toil.job.JobDescription (<#toil.job.JobDescription>)

🌈 createRootJob(desc)

• Parameters
  • desc (toil.job.JobDescription, <#toil.job.JobDescription>)
• Return type: toil.job.JobDescription (<#toil.job.JobDescription>)

🌈 create_root_job(job_description)

This one creates the given JobDescription and sets it as the root job in this job store. Slay 🔥

• Parameters
  • job_description (toil.job.JobDescription, <#toil.job.JobDescription>) — the JobDescription to save and make the root job 📝
• Return type: toil.job.JobDescription (<#toil.job.JobDescription>)

👑 getRootJobReturnValue()

• Return type: Any

👑 get_root_job_return_value()

This parses the return value from the root job. If the root job hasn't fulfilled its promise yet, it raises an exception, so no peeking early 🙈

• Return type: Any

---

🦋 importFile()

There are two overloads of this one, and they're both iconic. The first returns None when you hand it a shared file name:

importFile(srcUrl: str, sharedFileName: str, hardlink: bool = False, symlink: bool = True) -> None

The second returns a FileID when no shared file name is given:

importFile(srcUrl: str, sharedFileName: None = None, hardlink: bool = False, symlink: bool = True) -> toil.fileStores.FileID

📎 Doc links: str (/usr/share/doc/python3/html/library/stdtypes.html#str), bool (/usr/share/doc/python3/html/library/functions.html#bool), None (/usr/share/doc/python3/html/library/constants.html#None), toil.fileStores.FileID (<#toil.fileStores.FileID>)

This imports the file at the given URL into the job store. The ID of the newly imported file is returned. If you give it a shared file name, the file gets imported under that name and None is returned 🥹 If an executable file on the local filesystem gets uploaded, its executability is preserved when it's downloaded again ⚡

Currently supported schemes are:

• 's3' for objects in Amazon S3, e.g. s3://bucket/key ☁️
• 'file' for local files, e.g. file:///local/file/path 📁
• 'http', e.g. http://someurl.com/path 🌐
• 'gs', e.g. gs://bucket/file 🫧

⚠️ Raises FileNotFoundError if the file does not exist. Not me panicking over here 😭

• Parameters
  • src_uri (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — a URL that points to a file or object in the storage mechanism of a supported URL scheme, e.g. a blob in an AWS s3 bucket. It must be a file, not a directory or prefix 📄
  • shared_file_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — an optional name to assign to the imported file within the job store 🏷️
• Returns: The jobStoreFileID of the imported file, or None if shared_file_name was given.
• Return type: toil.fileStores.FileID (<#toil.fileStores.FileID>) or None

🦋 import_file()

This is the snake_case sibling of importFile, and it has two overloads too. Same behavior, same vibes ✨

import_file(src_uri: str, shared_file_name: str, hardlink: bool = False, symlink: bool = True) -> None

import_file(src_uri: str, shared_file_name: None = None, hardlink: bool = False, symlink: bool = True) -> toil.fileStores.FileID

📎 Doc links: the same str, bool, None and toil.fileStores.FileID references as importFile above.

The parameters, returns and supported schemes are the same as importFile, so go check those up there 💅

🌸 exportFile(jobStoreFileID, dstUrl)

• Parameters
  • jobStoreFileID (toil.fileStores.FileID, <#toil.fileStores.FileID>)
  • dstUrl (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: None

🌸 export_file(file_id, dst_uri)

This exports a file to the destination pointed at by the destination URL 📤 The exported file will be executable if and only if it was originally uploaded from an executable file on the local filesystem. Very specific, very serious 🫡

Refer to AbstractJobStore.import_file() for the currently supported URL schemes.

Note that the helper method _exportFile is what actually reads from the source and writes to the destination. If you want to implement any optimizations that get around this, the _exportFile method should be overridden by subclasses of AbstractJobStore. Subclass energy 🧑‍💻

• Parameters
  • file_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the file in the job store that should be exported 📦
  • dst_uri (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — a URL that points to a file or object in the storage mechanism of a supported URL scheme, e.g. a blob in an AWS s3 bucket. May also be a local path 🗂️
• Return type: None

💀 abstractmethod destroy()

This is the inverse of initialize(). It deletes the physical storage represented by this instance 🗑️ It's not atomic, but it is idempotent, which is there to help with eventual consistency issues in the underlying storage. If the method fails and raises an exception, it may (and should be) invoked again 🔁

⚠️ If the underlying storage mechanism is eventually consistent, even a successful call isn't an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow 🙅‍♀️

• Return type: None

🧚 getEnv()

• Return type: dict[str, str] (/usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/stdtypes.html#str)

get_env() 💖

Okay so this one hands you back a dictionary of environment variables that this job store needs set in order to work properly on a worker 🧚 Bestie, it's giving "must have" energy.

Return type: dict[str, str]

---

clean(jobCache=None) 🧹

This is the cleanup crew after a restart ✨ It fixes jobs that might have been partially updated, resets the try counts, and removes jobs that aren't successors of the current root job. Lowkey iconic.

Parameters

• jobCache (Optional[dict[Union[str, toil.job.TemporaryID], toil.job.JobDescription]]) — If you pass a value, it has to be a dict from job ID keys to JobDescription object values. Jobs get loaded from the cache (which can be downloaded from the job store in a batch) instead of one at a time when it recurses into them. 🫶

Return type: toil.job.JobDescription

---

assignID(jobDescription) 🦋

Parameters: jobDescription (toil.job.JobDescription)

Return type: None

---

abstractmethod assign_job_id(job_description) 🌸

This one's an abstract method, so it's the blueprint. It gets a new jobStoreID for the described job and assigns it to the JobDescription. Files tied to that ID get accepted even if the JobDescription was never created or updated. No cap, that's a big deal.

Parameters

• job_description (toil.job.JobDescription) — The JobDescription you want to give an ID to.

Return type: None

---

create(jobDescription) ✨

Parameters: jobDescription (toil.job.JobDescription)

Return type: toil.job.JobDescription

---

abstractmethod create_job(job_description) 🍓

Abstract again, so it's the template. This one writes the given JobDescription to the job store. The job has to already have an ID assigned, so don't skip that step bestie 🙈

Must call jobDescription.pre_update_hook()

Returns: The JobDescription that was passed in.

Return type: toil.job.JobDescription

Parameters

• job_description (toil.job.JobDescription)

---

exists(jobStoreID) 👑

Parameters: jobStoreID (str)

Return type: bool

---

abstractmethod job_exists(job_id) 🔥

Abstract, same as before. This one checks whether a description of the job with the specified jobStoreID exists in the job store. Yes or no, babe.

Return type: bool

Parameters

• job_id (str)

---

publicUrlExpiration 🌈

This one's just listed by name, no description, so that's all the tea on it.

---

getPublicUrl(fileName) 🎀

Parameters: fileName (str)

Return type: str

---

abstractmethod get_public_url(file_name) 🦋

Abstract, so it's the blueprint for this one. It returns a publicly accessible URL to the given file in the job store. The URL may expire as early as 1h after it's been returned. If the file doesn't exist, it throws an exception.

Parameters

• file_name (str) — the jobStoreFileID of the file you want a URL for. 🧚

Raises NoSuchFileException — if the specified file does not exist in this job store.

Return type: str

---

getSharedPublicUrl(sharedFileName) 💜

Parameters: sharedFileName (str)

Return type: str

---

abstractmethod get_shared_public_url(shared_file_name) 🌸

Okay, this one is the sibling of getPublicUrl(), but with a twist. It's for generating URLs for shared files written by writeSharedFileStream(). It returns a publicly accessible URL to the given file in the job store. The returned URL starts with 'http:', 'https:' or 'file:', and it may expire as early as 1h after it's been returned. If the file doesn't exist, it throws an exception.

Parameters

• shared_file_name (str) — The name of the shared file you want a publicly accessible URL for.

Raises NoSuchFileException — raised if the specified file does not exist in the store.

Return type: str

---

load(jobStoreID) 🫶

Parameters: jobStoreID (str)

Return type: toil.job.JobDescription

---

abstractmethod load_job(job_id) 🧚

Abstract, the template again. This loads the description of the job referenced by the given ID, assigns it the job store's config, and hands it back. It might also declare the job to have failed (see toil.job.JobDescription.setupJobAfterFailure()) if there's evidence of a failed update attempt. Plot twist, but it's real 😭

Parameters

• job_id (str) — the ID of the job to load.

Raises NoSuchJobException — if there is no job with the given ID.

Return type: toil.job.JobDescription

---

update(jobDescription) 💅

Parameters: jobDescription (toil.job.JobDescription)

Return type: None

---

batch() 🍓

If the batch system supports it, calls to create() made while this context manager is active get performed in a batch once the context manager is released. Batch goals, honestly ✨

Return type: collections.abc.Iterator[None]

---

abstractmethod update_job(job_description) 💖

Abstract, you know the drill. This one persists changes to the state of the given JobDescription in this store atomically.

Must call jobDescription.pre_update_hook()

Parameters

• job (toil.job.JobDescription) — the job to write to this job store. 🎀
• job_description (toil.job.JobDescription)

Return type: None

---

delete(jobStoreID) 🙈

Parameters: jobStoreID (str)

Return type: None

---

abstractmethod delete_job(job_id) 💀

Abstract. This one removes the JobDescription from the store atomically.

After you delete a job, you may not subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it. Don't go back for seconds, bestie.

This operation is idempotent, which means deleting a job twice, or deleting a non-existent job, will succeed silently. No drama, no error 🌈

Parameters

• job_id (str) — the ID of the job to delete from this job store.

Return type: None

---

abstractmethod jobs() 🔮

Abstract, last one in this group. This is a best effort attempt to return an iterator over the JobDescriptions for all jobs in the store.

The iterator may not return all jobs, and it may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run, construct a more expensive ToilState object instead. Not the move to go in blind, ngl.

Returns: Returns an iterator on jobs in the store. The iterator may or may not contain all jobs, and it may contain invalid jobs. 🫠

Okay so the method above wraps up with a return type of Iterator[toil.job.jobDescription] 🦋, and then we're on to the file-writing crew 💅

💖 writeFile

writeFile(localFilePath, jobStoreID=None, cleanup=False)

Parameters

• localFilePath (str)
• jobStoreID (Optional[str])
• cleanup (bool)

Return type: str

---

✨ write_file

abstractmethod write_file(local_path, job_id=None, cleanup=False)

Bestie, this one takes a file (as a path) and drops it into this job store 🧚 It hands back an ID you can use to grab that file later on 🔑 The write is atomic, which means the file won't show up in the jobStore until the whole write has finished successfully 🌸

Parameters

• local_path (str) — the path to the local file that gets uploaded to the job store 📂. The last path component (the basename of the file) stays attached to the file in the file store, if that's supported, so you can search for it by name or name glob 🔍
• job_id (str) — the id of a job, or None 🎀. If you give one, the may be associated with that job in a job-store-specific way (the docs literally say "the may be," so bear with them 💀). It can even change the returned ID.
• cleanup (bool) — whether to try deleting the file when the job whose jobStoreID was given as jobStoreID gets deleted with jobStore.delete(job) 🗑️. If no jobStoreID was given, it does nothing.

Raises

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file got modified concurrently while this method was running 😭
• NoSuchJobException (toil.jobStores.abstractJobStore.NoSuchJobException) — if the job you specified via jobStoreID doesn't exist 💔

Return type: str

🚨 FIXME: some implementations may not raise this

Returns: an ID referencing the newly created file, which you can use to read the file in the future 🦋

Return type: str

Parameters

• local_path (str)
• job_id (Optional[str])
• cleanup (bool)

---

🌈 writeFileStream

writeFileStream(jobStoreID=None, cleanup=False, basename=None,
encoding=None, errors=None)

Parameters

• jobStoreID (Optional[str])
• cleanup (bool)
• basename (Optional[str])
• encoding (Optional[str])
• errors (Optional[str])

Return type: ContextManager[tuple[IO[bytes], str]]

---

👑 write_file_stream

abstractmethod write_file_stream(job_id=None, cleanup=False,
basename=None, encoding=None, errors=None)

This is basically writeFile's glow-up 💅 Instead of just taking a path, it gives you a context manager that yields a tuple with 1) a file handle you can write to and 2) the ID of the resulting file in the job store 🌷 You don't need to close the yielded file handle yourself, and you shouldn't 🙅‍♀️ Same deal as before: it's atomic, so the file won't appear in the jobStore until the write has successfully completed ✨

Parameters

• job_id (str) — the id of a job, or None 🎀. If you give one, the may be associated with that job in a job-store-specific way (yes, the docs say "the may be" again 🙈). It can influence the returned ID.
• cleanup (bool) — whether to try deleting the file when the job whose jobStoreID was given as jobStoreID gets deleted with jobStore.delete(job) 🗑️. If no jobStoreID was given, it does nothing.
• basename (str) — if the implementation supports it, this uses the given file basename, so when you search the job store with a query matching that basename, the file will be detected 🔎
• encoding (str) — the name of the encoding used to encode the file. Encodings are the same as for encode(). It defaults to None, which means binary mode 🧁
• errors (str) — an optional string that says how encoding errors get handled. Errors work the same as for open(). It defaults to 'strict' when an encoding is specified 🧐

Raises

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file got modified concurrently while this method was running 😭
• NoSuchJobException (toil.jobStores.abstractJobStore.NoSuchJobException) — if the job you specified via jobStoreID doesn't exist 💔

Return type: collections.abc.Iterator[tuple[IO[bytes], str]]

🚨 FIXME: some implementations may not raise this

Returns: a context manager yielding a file handle you can write to, plus an ID that references the newly created file and can be used to read it in the future 🦋

Return type: Iterator[Tuple[IO[bytes], str]]

Parameters

• job_id (Optional[str])
• cleanup (bool)
• basename (Optional[str])
• encoding (Optional[str])
• errors (Optional[str])

---

🧚 getEmptyFileStoreID

getEmptyFileStoreID(jobStoreID=None, cleanup=False,
basename=None)

Parameters

• jobStoreID (Optional[str])
• cleanup (bool)
• basename (Optional[str])

Return type: str

---

🦋 get_empty_file_store_id

abstractmethod get_empty_file_store_id(job_id=None,
cleanup=False, basename=None)

This one is giving "make me a blank canvas" 🎨 It creates an empty file in the job store and returns its ID. Calling fileExists(getEmptyFileStoreID(jobStoreID)) will return True ✅

Parameters

• job_id (Optional[str]) — the ID of a job, or None. If you pass one in, the file may get linked to that job in a job-store-specific way, and that can influence the ID you get back 🍓
• cleanup (bool) — whether to try deleting the file when the job whose jobStoreID was given as jobStoreID gets deleted with jobStore.delete(job). If no jobStoreID was given, this does absolutely nothing 🧹
• basename (Optional[str]) — if the implementation supports it, this uses the basename you give, so searching the job store with a query that matches that basename will find the file 🔍

Returns: a jobStoreFileID that points to the newly created file and can be used to reference it later 💜

Return type: str

---

readFile(jobStoreFileID, localFilePath, symlink=False)

Parameters

• jobStoreFileID (str)
• localFilePath (str)
• symlink (bool)

Return type: None

abstractmethod read_file(file_id, local_path, symlink=False)

okay so this one copies or hard links the file referenced by jobStoreFileID over to the local file path you give it 🦋 The version you get matches the last copy that was written or updated. If the file in the job store gets changed later through updateFile or updateFileStream, whether those writes show up at localFilePath is up to each implementation, so it's implementation-defined. The copy is atomic, which means the file won't show up in your local file system until the copy is totally finished ✨

The file at the given local path may not be modified after this method returns!

Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs.

Parameters

• file_id (str) — the ID of the file you want copied 🌸
• local_path (str) — where to put the file's contents, described as "in the job store" in the original
• symlink (bool) — whether the reader can handle a symlink. If it's set to True, the job store may make a symlink instead of a full copy or a hard link 🔗

Return type: None

readFileStream(jobStoreFileID, encoding=None, errors=None)

Parameters

• jobStoreFileID (str)
• encoding (Optional[str])
• errors (Optional[str])

Return type: Union[ContextManager[IO[bytes]], ContextManager[IO[str]]]

read_file_stream

bestie, there are two overloads of this one, and they're giving different return types depending on whether you pass an encoding 💅

read_file_stream(file_id: toil.fileStores.FileID | str, encoding: Literal[None] = None, errors: str | None = None) -> ContextManager[IO[bytes]]

read_file_stream(file_id: toil.fileStores.FileID | str, encoding: str, errors: str | None = None) -> ContextManager[IO[str]]

This is basically readFile, except you get a context manager that yields a file handle you can read from. You do not need to close the yielded file handle yourself, and you should not close it explicitly 🫶

Parameters

• file_id (str) — the ID of the file you want a readable file handle for 🎀
• encoding (str) — the name of the encoding used to decode the file. Encodings work the same as they do for decode(). It defaults to None, which means binary mode 🧠
• errors (str) — an optional string that decides how encoding errors get handled. Errors work the same as they do for open(). It defaults to 'strict' when an encoding is specified ⭐

Returns: a context manager that yields a file handle you can read from 🦋

Return type: Iterator[Union[IO[bytes], IO[str]]]

deleteFile(jobStoreFileID)

Parameters

• jobStoreFileID (str)

Return type: None

abstractmethod delete_file(file_id)

this deletes the file with the given ID from this job store 🗑️ It's idempotent, which is the iconic part: deleting a file twice, or deleting one that doesn't exist, will just succeed silently 💖

Parameters

• file_id (str) — the ID of the file to delete

Return type: None

fileExists(jobStoreFileID)

this one checks whether a file exists in this job store 👀

Parameters

• jobStoreFileID (str)

Return type: bool

abstractmethod file_exists(file_id)

same energy, it determines whether a file exists in this job store 🔎

Parameters

• file_id (str) — an ID that references the file being checked

Return type: bool

getFileSize(jobStoreFileID)

this gets the size of the given file in bytes 📏

Parameters

• jobStoreFileID (str)

Return type: int

abstractmethod get_file_size(file_id)

this gets the size of the given file in bytes, or 0 if the file doesn't exist when you ask 🙈

Heads up, though: job stores that encrypt files might give you overestimates of the size. That's because the encrypted file may have been padded to the nearest block, or had an initialization vector added, etc. 💀

Parameters

• file_id (str) — an ID that references the file being checked

Return type: int

updateFile(jobStoreFileID, localFilePath)

this replaces the existing version of a file in the job store, no cap 🔥

Parameters

• jobStoreFileID (str)
• localFilePath (str)

Return type: None

abstractmethod update_file(file_id, local_path)

this one also replaces the existing version of a file in the job store, but it's very strict about it: if the file doesn't exist, it throws an exception 😭

Parameters

• file_id (str) — the ID of the file in the job store to be updated

🎀 updateFile (continued) 🎀

• local_path (str) — the local path to the file that's going to overwrite the current version in the job store 💖

Raises 🙈

• ConcurrentFileModificationException — if somebody else changed the file while this method was running. Chaotic, but it happens 😭
• NoSuchFileException — if the file you named just isn't there 💀

Return type: None, so nothing comes back, no cap ✨

🦋 updateFileStream

updateFileStream(jobStoreFileID, encoding=None, errors=None)

Parameters

• jobStoreFileID (str)
• encoding (Optional[str])
• errors (Optional[str])

Return type: ContextManager[IO[Any]]

🌈 update_file_stream

abstractmethod update_file_stream(file_id, encoding=None, errors=None)

Okay so this one replaces the existing version of a file in the job store 💅 It's basically the same vibe as writeFile, except it hands you a context manager that yields a file handle you can write into. ✨

🚨 Don't close the yielded file handle yourself. You don't need to, and you really shouldn't. Let the context manager handle it 🙅‍♀️

Parameters

• file_id (str) — the ID of the file in the job store you're updating 🧚
• encoding (str) — the name of the encoding used to encode the file. It's the same list you get from encode(). Defaults to None, which means binary mode 🫶
• errors (str) — an optional string that decides how encoding errors get handled. The options are the same as for open(). If you set an encoding and leave this out, it defaults to 'strict' 🔥

Raises 🙈

• ConcurrentFileModificationException — if the file got modified concurrently while this method was running 😭
• NoSuchFileException — if the specified file doesn't exist 💀

Return type: collections.abc.Iterator[IO[Any]]

🍓 sharedFileNameRegex

This one's just listed by name. No extra details, bestie 🙈

🌸 writeSharedFileStream

writeSharedFileStream(sharedFileName, isProtected=None, encoding=None, errors=None)

Parameters

• sharedFileName (str)
• isProtected (Optional[bool])
• encoding (Optional[str])
• errors (Optional[str])

Return type: ContextManager[IO[bytes]]

💜 write_shared_file_stream

abstractmethod write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)

This method returns a context manager that yields a writable file handle for the global file with the name you give it. The file gets created in an atomic manner, which is a whole mood 👑

Parameters

• shared_file_name (str) — a file name that matches AbstractJobStore.fileNameRegex and is unique within this job store 🦋
• encrypted (bool) — True if the file has to be encrypted. None if it can be encrypted or not. False if it has to be stored in the clear 🔐
• encoding (str) — the name of the encoding used to encode the file. It's the same list you get from encode(). Defaults to None, which means binary mode 🫶
• errors (str) — an optional string that decides how encoding errors get handled. The options are the same as for open(). If you set an encoding and leave this out, it defaults to 'strict' 🔥

Raises

• ConcurrentFileModificationException — if the file got modified concurrently while this method was running 😭

Returns: a context manager yielding a writable file handle ✨

Return type: Iterator[IO[bytes]]

🧚 readSharedFileStream

readSharedFileStream(sharedFileName, encoding=None, errors=None)

Parameters

• sharedFileName (str)
• encoding (Optional[str])
• errors (Optional[str])

Return type: Union[ContextManager[IO[str]], ContextManager[IO[bytes]]]

🌈 read_shared_file_stream

abstractmethod read_shared_file_stream(shared_file_name: str, encoding: str, errors: str | None = None) -> collections.abc.Iterator[IO[str]]
abstractmethod read_shared_file_stream(shared_file_name: str, encoding: Literal[None] = None, errors: str | None = None) -> collections.abc.Iterator[IO[bytes]]

This one returns a context manager that yields a readable file handle for the global file referenced by the name you pass in 📖

Parameters

• shared_file_name (str) — a file name that matches AbstractJobStore.fileNameRegex and is unique within this job store 🦋
• encoding (str) — the name of the encoding used to decode the file. It's the same list you get from decode(). Defaults to None, which means binary mode 🫶
• errors (str) — an optional string that decides how encoding errors get handled. The options are the same as for open(). If you set an encoding and leave this out, it defaults to 'strict' 🔥

Returns: a context manager yielding a readable file handle ✨

🫧 writeStatsAndLogging

writeStatsAndLogging(statsAndLoggingString)

Parameters

• statsAndLoggingString (str)

Return type: None

💅 write_logs

abstractmethod write_logs(msg)

This one stores a message as a log in the jobstore 📝

Parameters

• msg (str) — the string to be written 💖

Raises

• ConcurrentFileModificationException — if the file got modified concurrently while this method was running 😭

Return type: None

🌟 readStatsAndLogging

readStatsAndLogging(callback, readAll=False)

Parameters

• callback (Callable[Ellipsis, Any])
• readAll (bool)

Return type: int

🎀 read_logs(callback, read_all=False)

Okay so this one is abstract, which means each jobstore has to bring its own version 💅 It goes back and grabs the logs that write_logs() stashed, and for every single log it calls your callback with the message as the argument. Quick heads up bestie: it does not hand the logs back as a return value. You have to pass in a callback that actually processes the log messages 🧚

Only unread logs get read, unless you set read_all.

Parameters

• callback (Callable) — a function to be applied to each of the stats file handles found ✨
• read_all (bool, link target /usr/share/doc/python3/html/library/functions.html#bool) — a true/false switch for whether to also re-read the stats files that were already processed, on top of the unread ones 🌸

Raises ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — yikes 😭 this one pops off if the file got modified concurrently while this method was running.

Returns the number of stats files processed 🔥

Return type int (link target /usr/share/doc/python3/html/library/functions.html#int)

---

💖 write_leader_pid()

This writes the pid of the current process into a file in the job store 🦋

Overwriting the current contents of pid.log is a feature, not a bug 🫶 other methods rely on always having the most current pid, and so far there's no reason to store any old ones.

Return type None

💜 read_leader_pid()

Reads the pid of the leader process from a file in the job store 👑

If the PID file doesn't exist, this raises NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) 💀

Return type int (link target /usr/share/doc/python3/html/library/functions.html#int)

🌈 write_leader_node_id()

Writes the leader node id to the job store. Only the leader should call this, no outside crashers allowed 🧚

Return type None

🍓 read_leader_node_id()

Reads the leader node id that's stored in the job store 🌸

If the node ID file doesn't exist, this raises NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) 🙈

Return type str (link target /usr/share/doc/python3/html/library/stdtypes.html#str)

✨ write_kill_flag(kill=False)

This makes a file inside the job store that acts as a kill flag 🚩 The file starts out containing the characters "NO". Only change it when the user runs the toil kill command, because switching it to "YES" triggers a kill of the leader process. The workers are expected to be cleaned up by the leader, so they get handled too 💅

Parameters

• kill (bool, link target /usr/share/doc/python3/html/library/functions.html#bool)

Return type None

🦋 read_kill_flag()

Reads the kill flag from the job store. It returns True if the leader has been killed and False otherwise 💀

Return type bool (link target /usr/share/doc/python3/html/library/functions.html#bool)

🧚 default_caching()

This is the jobstore's vibe on caching: it either likes caching or doesn't care about it at all 🫧 Some jobstores benefit from caching, but on some local configurations it can be flaky, so tread carefully 😬

see https://github.com/DataBiosphere/toil/issues/4218 🔗

Return type bool (link target /usr/share/doc/python3/html/library/functions.html#bool)

---

🌸 class toil.jobStores.abstractJobStore.JobStoreSupport(locator)

Bases: AbstractJobStore, toil.lib.url.URLAccess

This is a mostly fake JobStore, made for accessing URLs that aren't really tied to real job stores 🧸

Parameters

• locator (str, link target /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🦋 toil.jobStores.aws

Submodules

💅 toil.jobStores.aws.jobStore

This file holds the AWS jobstore, which has its own docstring explaining how to use it. This particular docstring is all about how the file is organized, so pay attention bestie 🎀

All direct AWS boto calls should live in toil.lib.aws, except for creating the session instance and the resource/client, which should only be made ONCE in the jobstore 🔥

Reasons for this

• DRY.
• All retries are on their individual boto functions, instead of here.
• Simple clear functions => simple clear unit tests (ideally).

Variables defining part size, parallelization, and other constants should live in toil.lib.aws.config 💖

✨ Attributes

• DEFAULT_AWS_PART_SIZE
• logger

👑 Classes

• AWSJobStore — The AWS jobstore can be thought of as an AWS s3 bucket, with functions to centralize, store, and track files for the workflow.

🌈 Functions

• parse_jobstore_identifier(jobstore_identifier)

🧚 Module Contents

toil.jobStores.aws.jobStore.DEFAULT_AWS_PART_SIZE = 52428800

toil.jobStores.aws.jobStore.logger

🎀 class toil.jobStores.aws.jobStore.AWSJobStore(locator, partSize=DEFAULT_AWS_PART_SIZE)

Bases: toil.jobStores.abstractJobStore.AbstractJobStore, toil.lib.url.URLAccess

The AWS jobstore can be thought of as an AWS s3 bucket, with functions to centralize, store, and track files for the workflow 🪣

The AWS jobstore stores 4 things:

1. Jobs: These are pickled as files, and contain the information necessary to run a job when unpickled. A job's file is deleted when it finishes, so if it's gone, that job completed 🎉
2. Files: The inputs and outputs of jobs. Each file is written in s3 with the file pattern "files/{uuid4}/{original_filename}", where the file prefix "files/{uuid4}" should only point to one file 📁
3. Logs: The written log files of jobs that have run, plus the log file for the main Toil process 📜
4. Shared Files: Files with human-readable names, used by Toil itself or Python workflows. These include:
   • environment.pickle (environment variables)
   • config.pickle (user options)
   • pid.log (process ID of the workflow; when it finishes, the workflow either succeeded/failed)
   • userScript (hot deployment; this is the job module)
   • rootJobReturnValue (workflow succeeded or not)

📝 NOTES

• The AWS jobstore does not use a database (directly, at least) currently. We can get away with this because:
  1. AWS s3 has strong consistency.
  2. s3's filter/query speed is pretty good.

However, there may be reasons in the future to provide users with a database:

• s3 throttling has limits (3,500/5,000 requests (TODO: per second?); something like dynamodb supports 100,000+ requests).
• Access and filtering would be sped up, though how much faster this would be needs testing.

ALSO NOTE: The caching filestore uses a local (per node) database with a very similar structure that maybe could be synced up with this 🤓

• TODO: Etags are s3's native checksum, so use that for file integrity checking since it's free when fetching object headers from s3. Using an md5sum in addition to this would work well with the current filestore.

WARNING: Etag values differ for the same file when the part size changes, so part size should always be Set In Stone, unless we hit s3's 10,000 part limit, and we need to account for that. 😭

• This class fills in self.config only when initialized/restarted; it is None upon class instantiation. These are the options/config set by the user. When jobs are loaded/unpickled, they must re-incorporate this 🔄
• The config.sseKey field is the single source of truth for bucket encryption status. The key is never stored inside this class; it is always read from the file referenced by the config when needed. Modifying the config at runtime will modify whether encryption is used.

WARNING: Note that files written without encryption (i.e. config.pickle) can't be read when encryption is enabled! 🔐

Okay bestie, we're picking up with a little TODO the Toil devs left for themselves 👀 In general, job stores should log the version of Toil they were initialized with and warn the user if they restart with a different version. That's a whole "check your vibes before you hit run" moment 💅

🎀 Parameters

• locator (str)
• partSize (int)

Here's the stash of class attributes this thing is carrying around 🧚

• s3_resource
• s3_client
• part_size = 52428800
• bucket = None
• job_key_prefix = 'jobs/'
• job_associations_key_prefix = 'job-associations/'
• content_key_prefix = 'files/'
• shared_key_prefix = ''
• logs_key_prefix = 'logs/'

💅 initialize(config)

Called when starting a new jobstore with a non-existent bucket. It creates the bucket and raises if it already exists, then sets options from config ✨

🎀 Parameters

• config (toil.common.Config)

Return type: None

💅 resume()

Called when reusing an old jobstore with an existing bucket 🦋

Raises:

• NoSuchJobStoreException — if the bucket doesn't exist.

Return type: None

💅 destroy()

This is the inverse of initialize(). It deletes the physical storage represented by this instance. It's not atomic, but it is idempotent, which is a lifesaver for dealing with potential eventual consistency issues in the underlying storage mechanisms. If the method fails (raises an exception), it may (and should be) invoked again.

Even a successful invocation isn't an ironclad guarantee that the physical storage vanished completely and immediately if the underlying storage is eventually consistent. A successful invocation only guarantees that the deletion will eventually happen. So it's recommended to not immediately reuse the same job store location for a new Toil workflow 🙈

Return type: None

💅 is_in_bucket(identifier, prefix, bucket=None)

Checks if the key for the given identifier and prefix is in the bucket 🔍

🎀 Parameters

• identifier (str)
• prefix (str)
• bucket (Optional[str])

Return type: bool

💅 write_to_bucket(identifier, prefix, data, bucket=None, encrypted=None)

Writes something directly to a bucket. Use it for small files, because it does not parallelize or use multipart 🍓

🎀 Parameters

• encrypted (Optional[bool]) — can be set to False to disable encryption.
• identifier (str)
• prefix (str)
• data (Optional[Union[bytes, str, Dict[str, Any]]])
• bucket (Optional[str])

Return type: None

💅 read_from_bucket(identifier, prefix, bucket=None)

Reads something directly from a bucket. Same deal as the write: use it for small files, since it does not parallelize or use multipart 📖

Raises:

• NoSuchJobException — if the prefix is the job prefix and the identifier is not found.
• NoSuchFileException — if the prefix is the content prefix and the identifier is not found.
• self.s3_client.exceptions.NoSuchKey — in other cases where the identifier is not found.

🎀 Parameters

• identifier (str)
• prefix (str)
• bucket (Optional[str])

Return type: bytes

💅 assign_job_id(jobDescription)

Gets a new jobStoreID to be used by the described job, and assigns it to the JobDescription 🪪 Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated.

🎀 Parameters

• job_description (toil.job.JobDescription) — the JobDescription to give an ID to.
• jobDescription (toil.job.JobDescription)

Return type: None

💅 create_job(jobDescription)

Pickles a jobDescription object and writes it to the jobstore as a file 🥒 It's responsible for calling toil.job.JobDescription.pre_update_hook() on the job description.

🎀 Parameters

• jobDescription (toil.job.JobDescription)

Return type: toil.job.JobDescription

💅 job_exists(job_id, check=False)

Checks if the job_id is found in s3 🔎

🎀 Parameters

• check (bool) — If True, raise an exception instead of returning false when a job does not exist.
• job_id (str)

Return type: bool

💅 jobs()

Best effort attempt to return an iterator on JobDescriptions for all jobs in the store. ✨ The iterator may not return all jobs, and it may also contain orphaned jobs that have already finished successfully and should not be rerun. If you need to guarantee you get any and all jobs that can be run, construct a more expensive ToilState object instead.

Returns: an iterator on jobs in the store. The iterator may or may not contain all jobs, and it may contain invalid jobs.

Return type: Iterator[toil.job.jobDescription]

💅 load_job(job_id)

Uses a job_id to get a job from the jobstore's s3 bucket, unpickles it, and returns it 📦

🎀 Parameters

• job_id (str)

Return type: toil.job.JobDescription

💅 update_job(jobDescription)

Persists changes to the state of the given JobDescription in this store, atomically 💜

Must call jobDescription.pre_update_hook()

🎀 Parameters

• job (toil.job.JobDescription) — the job to write to this job store.
• jobDescription (toil.job.JobDescription)

Return type: None

💅 delete_job(job_id)

Removes the JobDescription from the store, atomically 🗑️ You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it.

This operation is idempotent, meaning deleting a job twice or deleting a non-existent job will succeed silently 🫶

🎀 Parameters

• job_id (str) — the ID of the job to delete from this job store.

Return type: None

💅 associate_job_with_file(job_id, file_id)

🎀 Parameters

• job_id (str)
• file_id (str)

Return type: None

💅 write_file(local_path, job_id=None, cleanup=False)

Writes a local file into the jobstore and returns a file_id referencing it 📁

🎀 Parameters

• job_id (Optional[str]) — okay so, if job_id AND cleanup are both supplied, this file gets linked to that job 💖. When the job is deleted, the file goes poof too ✨
• cleanup (bool) — same deal as above: if job_id AND cleanup are supplied, the file is tied to that job, and deleting the job deletes the file as well 🧹

🙈 TODO: we don't need cleanup. Remove it and only use job_id. The devs already know, bestie.

• local_path (str) 🌸

Return type toil.fileStores.FileID 🦋

✨ find_s3_key_from_file_id(file_id) ✨

This one is a detective 🕵️‍♀️. It finds an s3 key where file_id is the prefix, and the key already exists. Main character energy.

Parameters

• file_id (str)

Return type str 💅

✨ write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) ✨

Okay, this is giving writeFile but make it fancy 💅. It returns a context manager that yields a tuple of 1) a file handle you can write to and 2) the ID of the new file in the job store. You don't need to close the file handle yourself (and you really shouldn't, ngl). The file gets written in an atomic way, so it won't show up in the jobStore until the write has fully finished. No half-baked files here 🫶

Parameters

• job_id (str) — the id of a job, or None. If you give one, the file may be associated with that job in a job-store-specific way, and that can influence the returned ID 🌈
• cleanup (bool) — whether to try deleting the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID wasn't given, nothing happens 🧚
• basename (str) — if the implementation supports it, this sets a file basename, so searching the job store with a query matching that basename will find the file 🔍
• encoding (str) — the name of the encoding used for the file. Encodings are the same as for encode(). Defaults to None, which means binary mode 🔤
• errors (str) — an optional string that says how encoding errors get handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified ⚠️

Raises

• ConcurrentFileModificationException — if the file was modified concurrently while this method was running 💀
• NoSuchJobException — if the job specified via jobStoreID doesn't exist 😭

Return type Iterator[tuple[IO[bytes], str]]

🙈 FIXME: some implementations may not raise this. Not every store is perfect, bestie.

Returns a context manager yielding a file handle you can write to, plus an ID that points to the newly created file. You can use that ID to read the file later 🎀

Return type Iterator[Tuple[IO[bytes], str]]

Parameters

• job_id (Optional[str])
• cleanup (bool)
• basename (Optional[str])
• encoding (Optional[str])
• errors (Optional[str])

✨ update_file_stream(file_id, encoding=None, errors=None) ✨

This one replaces the existing version of a file in the job store 🔄. It's similar to writeFile, except it returns a context manager yielding a file handle you can write to. Same rule applies: you don't need to close the handle explicitly 💖

Parameters

• file_id (str) — the ID of the file in the job store to update 🪪
• encoding (str) — the encoding name, same as for encode(). Defaults to None, which means binary mode 🔤
• errors (str) — how encoding errors get handled, same as for open(). Defaults to 'strict' when an encoding is specified ⚠️

Raises

• ConcurrentFileModificationException — if the file was modified concurrently during this method call 💀
• NoSuchFileException — if the specified file doesn't exist 🙈

Return type Iterator[IO[Any]]

✨ write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None) ✨

Returns a context manager yielding a writable file handle to the global file referenced by the given name 🌍. The file gets created in an atomic manner, so no sneaky partial files 🦋

Parameters

• shared_file_name (str) — a file name matching AbstractJobStore.fileNameRegex, and it has to be unique within this job store 🏷️
• encrypted (bool) — True if the file must be encrypted, None if it may or may not be encrypted, or False if it must be stored in the clear 🔐
• encoding (str) — the encoding name, same as for encode(). Defaults to None, which means binary mode 🔤
• errors (str) — how encoding errors get handled, same as for open(). Defaults to 'strict' when an encoding is specified ⚠️

Raises ConcurrentFileModificationException — if the file was modified concurrently during an invocation of this method 💀

Returns a context manager yielding a writable file handle 🎀

Return type Iterator[IO[bytes]]

✨ update_file(file_id, local_path) ✨

Replaces the existing version of a file in the job store 🔁. Heads up: it throws an exception if the file doesn't exist, so double check your IDs, bestie 🚨

Parameters

• file_id (str) — the ID of the file in the job store to update 🪪
• local_path (str) — the local path to a file that will overwrite the current version in the job store 📂

Raises

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — 😬 this pops up if the file got modified concurrently during an invocation of this method, so someone else was messing with it at the same time
• NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) — shows up if the specified file does not exist 🙈

Return type — None

✨ file_exists(file_id)

Okay so this one just checks whether a file exists in this job store 👀

• file_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — an ID referencing the file to be checked

Return type — bool (/usr/share/doc/python3/html/library/functions.html#bool) 💖

✨ get_file_size(file_id)

Okay but a dev note in the source literally asks: "Do we need both get_file_size and _get_size???" 🤔 Bestie, same question.

• file_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type — int (/usr/share/doc/python3/html/library/functions.html#int) 🌸

✨ read_file(file_id, local_path, symlink=False)

This one copies or hard links the file referenced by jobStoreFileID to the local file path you give it. The version you get matches the last copy of the file that was written or updated. If the job store file gets changed later through updateFile or updateFileStream, it's implementation-defined whether those writes show up at localFilePath, so don't count on it 🤷‍♀️ The copy is atomic too, which means the file won't appear in your local file system until the copy is totally finished. ✨

The file at the given local path may not be modified after this method returns! 🚫

Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs. 🔑

• file_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the file to be copied
• local_path (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the local path showing where to put the contents of the given file from the job store
• symlink (bool, /usr/share/doc/python3/html/library/functions.html#bool) — whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link. 🦋

Return type — None

✨ read_file_stream(file_id, encoding=None, errors=None)

This is like readFile, except it hands you a context manager that yields a file handle you can read from. You don't need to close the yielded file handle yourself, and you should not 🙅‍♀️

• file_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the file to get a readable file handle for
• encoding (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which means binary mode.
• errors (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — an optional string for how encoding errors get handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

Returns — a context manager yielding a file handle which can be read from

Return type — Iterator[Union[IO[bytes], IO[str]]] (/usr/share/doc/python3/html/library/stdtypes.html#bytes, /usr/share/doc/python3/html/library/stdtypes.html#str) 🌈

✨ read_shared_file_stream

Two overloads, so here they are, exactly as written:

read_shared_file_stream(shared_file_name: str, encoding: str, errors: str | None = None) -> Iterator[IO[str]]
read_shared_file_stream(shared_file_name: str, encoding: Literal[None] = None, errors: str | None = None) -> Iterator[IO[bytes]]

This one gives you a context manager yielding a readable file handle to the global file referenced by the given name. 📂

• shared_file_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — a file name matching AbstractJobStore.fileNameRegex, unique within this job store
• encoding (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which means binary mode.
• errors (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — an optional string for how encoding errors get handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

Returns — a context manager yielding a readable file handle

✨ delete_file(file_id)

Deletes the file with the given ID from this job store. This operation is idempotent, which means deleting a file twice, or deleting a file that doesn't exist, will succeed silently. No drama 💅

• file_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the file to delete

Return type — None

✨ get_public_url(file_id)

Turns s3:// into http:// and puts a public-read ACL on it. 🌐

• file_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

✨ get_shared_public_url(file_id)

Turns s3:// into http:// and puts a public-read ACL on it, same as above 🫶

• file_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

✨ get_empty_file_store_id(job_id=None, cleanup=False, basename=None)

Creates an empty file in S3 and returns a bare string file ID. 🧚

• job_id (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)
• cleanup (bool, /usr/share/doc/python3/html/library/functions.html#bool)
• basename (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

✨ write_logs(log_msg)

Stores a message as a log in the jobstore. 📝

• msg (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the string to be written
• log_msg (Union[bytes, str], /usr/share/doc/python3/html/library/stdtypes.html#bytes, /usr/share/doc/python3/html/library/stdtypes.html#str)

Raises ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — 😬 if the file was modified concurrently during an invocation of this method

Return type — None

✨ read_logs(callback, read_all=False)

This fetches all referenced logs in the database from S3 as readable objects, then runs "callback()" on each one. 📚

• callback (Callable[Ellipsis, Any])
• read_all (bool, /usr/share/doc/python3/html/library/functions.html#bool)

Return type — int (/usr/share/doc/python3/html/library/functions.html#int)

---

✨ toil.jobStores.aws.jobStore.parse_jobstore_identifier(jobstore_identifier)

• jobstore_identifier (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type — Tuple[str, str] (/usr/share/doc/python3/html/library/stdtypes.html#str) 👑

💖 toil.jobStores.aws.utils 💖

okay bestie, this is the toil.jobStores.aws.utils module, a little helper kit for S3 stuff 🧚

✨ Attributes ✨

There are two attributes, and the page doesn't describe either one 💀

• logger
• DIAL_SPECIFIC_REGION_CONFIG

🚨 Exceptions 🚨

• ServerSideCopyProhibitedError: raised when AWS refuses to do a server-side copy between S3 keys. The summary table cuts off at "and," so the full explanation is further down 🙈

🎀 Functions 🎀

• fileSizeAndTime(localFilePath): no description in the summary table
• uploadFromPath(localFilePath, resource, bucketName, fileID): uploads a file to s3, using multipart uploading if applicable
• uploadFile(readable, resource, bucketName, fileID[, ...]): uploads a readable object to s3, using multipart uploading if applicable.
• copyKeyMultipart(resource, srcBucketName, srcKeyName, ...): copies a key from a source key to a destination key in multiple parts. The table stops right after "Note that if the," so the full note is in the body below.
• retryable_ssl_error(e): no description in the summary table

the summary table gets cut off mid-sentence twice, so check the full entries below for the complete info 🫶

📚 Module Contents 📚

toil.jobStores.aws.utils.logger

Just the name, nothing else here. ✨

toil.jobStores.aws.utils.DIAL_SPECIFIC_REGION_CONFIG

Also just the name. 🌸

toil.jobStores.aws.utils.fileSizeAndTime(localFilePath)

toil.jobStores.aws.utils.fileSizeAndTime(localFilePath)

Parameters

• localFilePath (str)

Return type

tuple[int, float]

toil.jobStores.aws.utils.uploadFromPath(...)

toil.jobStores.aws.utils.uploadFromPath(localFilePath, resource, bucketName, fileID, headerArgs=None, partSize=50 << 20)

Uploads a file to s3, using multipart uploading if applicable. Main character behavior 🦋

Parameters

• localFilePath (str): path of the file to upload to s3
• resource (S3.Resource): boto3 resource
• bucketName (str): name of the bucket to upload to
• fileID (str): the name of the file to upload to
• headerArgs (dict): http headers to use when uploading. These are usually for encryption purposes.
• partSize (int): max size of each part in the multipart upload, in bytes

Returns

version of the newly uploaded file

Return type

Optional[str]

toil.jobStores.aws.utils.uploadFile(...)

toil.jobStores.aws.utils.uploadFile(readable, resource, bucketName, fileID, headerArgs=None, partSize=50 << 20)

Upload a readable object to s3, using multipart uploading if applicable. This one is giving flexibility 💅

:param readable: a readable stream or a file path to upload to s3 :param S3.Resource resource: boto3 resource :param str bucketName: name of the bucket to upload to :param str fileID: the name of the file to upload to :param dict headerArgs: http headers to use when uploading. These are usually for encryption purposes. :param int partSize: max size of each part in the multipart upload, in bytes :return: version of the newly uploaded file

Parameters

• readable (IO[bytes])
• resource (mypy_boto3_s3.S3ServiceResource)
• bucketName (str)
• fileID (str)
• headerArgs (Optional[dict[str, Any]])
• partSize (int)

Return type

Optional[str]

toil.jobStores.aws.utils.ServerSideCopyProhibitedError

exception toil.jobStores.aws.utils.ServerSideCopyProhibitedError

Bases: RuntimeError

this is the one AWS throws when it says no to a server-side copy and makes you pay to download and re-upload the data yourself 💀

toil.jobStores.aws.utils.copyKeyMultipart(...)

toil.jobStores.aws.utils.copyKeyMultipart(resource, srcBucketName, srcKeyName, srcKeyVersion, dstBucketName, dstKeyName, sseAlgorithm=None, sseKey=None, copySourceSseAlgorithm=None, copySourceSseKey=None)

Copies a key from a source key to a destination key in multiple parts. Heads up: if the destination key already exists, it gets overwritten implicitly. If it doesn't exist, a new key gets created. If the destination bucket doesn't exist, an error gets raised. 🚨

This function will always do a fast, server-side copy, at least until/unless <<https://github.com/boto/boto3/issues/3270>> is fixed.

Sometimes a fast, server-side copy just isn't possible. For example, say you're inside an AWS VPC with an S3 VPC Endpoint configured, and you try to copy from a bucket in another region into a bucket in your own region. That can't happen server-side. The VPC Endpoint S3 API servers refuse to do server-side copies between regions. The source region's API servers refuse to start the copy and point you to the destination bucket's region's API servers instead. And the VPC routing tables send all access to the current region's S3 API servers over to the S3 Endpoint API servers, so you're stuck in a loop 😭

If a fast server-side copy isn't actually possible, a ServerSideCopyProhibitedError will be raised.

Parameters

• resource (mypy_boto3_s3.S3ServiceResource): boto3 resource
• srcBucketName (str): the name of the bucket to be copied from
• srcKeyName (str): the name of the key to be copied from
• srcKeyVersion (str): the version of the key to be copied from
• dstBucketName (str): the name of the destination bucket for the copy
• dstKeyName (str): the name of the destination key that will be created or overwritten
• sseAlgorithm (str): server-side encryption algorithm for the destination
• sseKey (str): server-side encryption key for the destination
• copySourceSseAlgorithm (str): server-side encryption algorithm for the source
• copySourceSseKey (str): server-side encryption key for the source

Returns

the version of the copied file (or None if versioning is not enabled for dstBucket)

Return type

Optional[str]

toil.jobStores.aws.utils.retryable_ssl_error(e)

toil.jobStores.aws.utils.retryable_ssl_error(e)

Parameters

• e (BaseException)

Return type

bool

💖 toil.jobStores.conftest

Attributes 🎀

• collect_ignore — the value cell is blank in the table, so there's nothing listed here.

Module Contents 🌸

• toil.jobStores.conftest.collect_ignore = [] — it's just an empty list [] ✨

---

🧚 toil.jobStores.fileJobStore

Attributes 💅

• logger — also blank in the table.

Classes 🦋

• FileJobStore — A job store that uses a directory on a locally attached file system. To be compatible with (the table cuts off here, so the rest of that sentence is in the full description below) 👀

Module Contents 🌈

• toil.jobStores.fileJobStore.logger

👑 class toil.jobStores.fileJobStore.FileJobStore(path, fanOut=1000)

Bases: toil.jobStores.abstractJobStore.AbstractJobStore and toil.lib.url.URLAccess. So it's inheriting from both of these, periodt 💅

Okay so this is the store that uses a directory on a locally attached file system. Super iconic, but there's a catch: to be compatible with distributed batch systems, that file system must be shared by all worker nodes. No shared file system, no slay 😭

Parameters 🫶

• path (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓
• fanOut (int, see /usr/share/doc/python3/html/library/functions.html#int) 🍓

Class attributes ✨

• validDirs = 'abcdefghijklmnopqrstuvwxyz0123456789' — the characters allowed in directory names, lowkey the whole alphabet plus digits 🔤
• validDirsSet — (no value shown)
• JOB_DIR_PREFIX = 'instance-'
• JOB_NAME_DIR_PREFIX = 'kind-'
• BUFFER_SIZE = 10485760
• LOG_TEMP_SUFFIX = '.new'
• LOG_PREFIX = 'stats'
• jobStoreDir = b'.'
• jobsDir — (no value shown)
• statsDir — (no value shown)
• stats_inbox — (no value shown)
• stats_archive — (no value shown)
• filesDir — (no value shown)
• jobFilesDir — (no value shown)
• sharedFilesDir — (no value shown)
• fanOut = 1000
• linkImports = None
• moveExports = None
• symlink_job_store_reads = None

🌸 default_caching()

The jobstore's preference on caching: it either likes caching or doesn't care. Some jobstores benefit from it, but on some local setups it can be flaky, so be careful bestie 🙈 See https://github.com/DataBiosphere/toil/issues/4218 for the drama.

Returns bool (see /usr/share/doc/python3/html/library/functions.html#bool).

🦋 __repr__()

Just the string representation of the instance. Nothing too deep here ✨

🧚 initialize(config)

Initializes this job store. It creates the physical storage, allocates a workflow ID, and saves the Toil configuration to the store. Basically it sets the whole thing up 💖

Parameters

• config — the Toil configuration to initialize this job store with. The config gets updated with the newly allocated workflow ID.

Raises JobStoreExistsException (toil.jobStores.abstractJobStore.JobStoreExistsException) — if the physical storage already exists. Not it 💀

🎀 resume()

Connects this instance to the physical storage it represents and loads the Toil configuration into the AbstractJobStore.config attribute. Reconnecting, bestie 🦋

Raises NoSuchJobStoreException (toil.jobStores.abstractJobStore.NoSuchJobStoreException) — if the physical storage doesn't exist.

💀 destroy()

The opposite of initialize(): this deletes the physical storage this instance represents. It's not atomic, but it is idempotent, which is there to help with eventual consistency issues from the underlying storage. So if it fails and raises an exception, you can and should just call it again 🔁

⚠️ Even a successful call isn't a 100% guarantee the storage vanished completely and immediately if the underlying storage is eventually consistent. A successful call only guarantees the deletion will eventually happen. So it's recommended you don't immediately reuse the same job store location for a new Toil workflow.

🌈 assign_job_id(job_description)

Gets a new jobStoreID for the described job and assigns it to the JobDescription. Files tied to that ID get accepted even if the JobDescription was never created or updated.

Parameters

• job_description (toil.job.JobDescription) — the JobDescription to give an ID to.

🧚 create_job(job_description)

Writes the given JobDescription to the job store. The job has to have an ID assigned already, or it's a no-go. Must call jobDescription.pre_update_hook() ✨

Returns the JobDescription that was passed in, type toil.job.JobDescription.

🍓 batch()

If the batch system supports it, any calls to create() made while this context manager is active get performed in a batch once the context manager is released. Efficient vibes 🫶

💖 job_exists(job_id)

Tells you whether a description of the job with the specified jobStoreID exists in the job store. Returns bool.

🔗 get_public_url(jobStoreFileID)

Returns a publicly accessible URL to the given file in the job store. ⚠️ The URL may expire as early as 1h after it's returned. If the file doesn't exist, it throws an exception.

Parameters

• file_name (str) — the jobStoreFileID of the file to generate a URL for. (That's what the docs call it, even though the argument shown in the signature is jobStoreFileID.)

Raises NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) — if the specified file doesn't exist in this job store.

Returns str.

🦋 get_shared_public_url(sharedFileName)

This one is like get_public_url(), except it's for generating URLs for shared files written by writeSharedFileStream(). Returns a publicly accessible URL to the given file in the job store. The returned URL starts with 'http:', 'https:' or 'file:'. ⚠️ It may expire as early as 1h after it's been returned, and if the file doesn't exist it throws an exception.

Parameters

• shared_file_name (str) — the name of the shared file to generate a publicly accessible URL for.

Raises NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) — if the specified file doesn't exist in the store.

Returns str.

💅 load_job(job_id)

Loads the description of the job referenced by the given ID, gives it the job store's config, and returns it. Main character behavior 👑

It might declare the job to have failed (see toil.job.JobDescription.setupJobAfterFailure()) if there's evidence of a failed update attempt. Rough, but honest 💀

Parameters

• job_id — the ID of the job to load.

Raises NoSuchJobException (toil.jobStores.abstractJobStore.NoSuchJobException) — if there's no job with the given ID.

✨ update_job(job)

Persists changes to the state of the given JobDescription in this store, atomically 🔥. Must call jobDescription.pre_update_hook().

Parameters

• job (toil.job.JobDescription) — the job to write to this job store.

🗑️ delete_job(job_id)

Removes the JobDescription from the store, atomically. After that you may not call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it. It's giving "gone for good" 🫠

This one is idempotent, so deleting a job twice, or deleting a job that doesn't exist, will succeed silently. No drama 🦋

Parameters

• job_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the job to delete from this job store.

🔍 jobs()

⚠️ This is a best effort attempt, bestie. The iterator may not return all jobs, and it may also contain orphaned jobs that already finished successfully and shouldn't be rerun. If you need to guarantee you get any and all jobs that can be run, construct a more expensive ToilState object instead.

Returns an iterator on the JobDescriptions for all jobs in the store. It may or may not contain all jobs, and it may contain invalid jobs. Messy but real 💅

Return type: Iterator[toil.job.jobDescription] 💜 that's the tail end of the last method, so you already know the drill ✨

🎀 write_file(local_path, job_id=None, cleanup=False) 🎀

Okay so this one takes a file (you hand it a path) and tucks it into this job store 🧚 It gives you back an ID that you can use later to grab the file again. The best part is that the write is atomic, so the file won't show up in the jobStore until the whole write has finished. No half-written messes, periodt 💅

Parameters

• local_path (str) — the path to the local file you want to upload. The last path component (the basename) stays attached to the file in the file store if that's supported, so you can search for it by name or name glob 🔍
• job_id (str) — the ID of a job, or None. If you pass one, the file may be linked to that job in a job-store-specific way, and that can change the ID you get back 🦋
• cleanup (bool) — whether to try deleting the file when the job whose jobStoreID was given as jobStoreID gets deleted with jobStore.delete(job). If no jobStoreID was given, nothing happens 🧹

Raises

• ConcurrentFileModificationException — if the file got modified concurrently while this method was running 😭
• NoSuchJobException — if the job specified via jobStoreID doesn't exist 💀

FIXME: some implementations may not raise this 🙈

Returns

An ID referencing the newly created file, which you can use to read the file in the future ✨

Return type

str

---

🦋 write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) 🦋

This is basically the cousin of writeFile, except it gives you a context manager instead. Inside it you get a tuple: 1) a file handle you can write to, and 2) the ID of the resulting file in the job store. You don't need to close the handle yourself (and you shouldn't!). It's the same atomic energy, so nothing appears in the jobStore until the write has successfully completed 🌸

Parameters

• job_id (str) — the ID of a job, or None. If you pass one, the file may be linked to that job in a job-store-specific way, and that can change the ID you get back 🦋
• cleanup (bool) — whether to try deleting the file when the job whose jobStoreID was given as jobStoreID gets deleted with jobStore.delete(job). If no jobStoreID was given, nothing happens 🧹
• basename (str) — if the implementation supports it, this basename gets used so that searching the job store with a query matching it will find the file 🔍
• encoding (str) — the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None, which means binary mode 🔤
• errors (str) — an optional string that decides how encoding errors get handled. Errors work the same as for open(). Defaults to 'strict' when an encoding is specified ⚠️

Raises

• ConcurrentFileModificationException — if the file got modified concurrently while this method was running 😭
• NoSuchJobException — if the job specified via jobStoreID doesn't exist 💀

FIXME: some implementations may not raise this 🙈

Returns

A context manager yielding a file handle you can write to, plus an ID that references the newly created file and can be used to read it in the future ✨

Return type

Iterator[Tuple[IO[bytes], str]]

---

🧚 get_empty_file_store_id(jobStoreID=None, cleanup=False, basename=None) 🧚

Creates an empty file in the job store and returns its ID. Literally the vibe is "a blank canvas, but make it stored." After calling this, fileExists(getEmptyFileStoreID(jobStoreID)) will return True 💖

Parameters

• job_id — the ID of a job, or None. If you pass one, the file may be linked to that job in a job-store-specific way, and that can change the ID you get back 🦋
• cleanup — whether to try deleting the file when the job whose jobStoreID was given as jobStoreID gets deleted with jobStore.delete(job). If no jobStoreID was given, nothing happens 🧹
• basename — if the implementation supports it, this basename gets used so that searching the job store with a query matching it will find the file 🔍

Returns

A jobStoreFileID that references the newly created file and can be used to reference it in the future ✨

Return type

str

---

🌈 update_file(file_id, local_path) 🌈

Swaps out the existing version of a file in the job store for a new one. If the file doesn't exist, it throws an exception, so don't try to update ghosts 👻

Parameters

• file_id — the ID of the file in the job store that you want to update 🆔
• local_path — the local path to a file that will overwrite the current version in the job store 📁

Raises

• ConcurrentFileModificationException — if the file got modified concurrently while this method was running 😭
• NoSuchFileException — if the specified file does not exist 💀

---

👑 read_file(file_id, local_path, symlink=False) 👑

This one copies or hard links the file referenced by jobStoreFileID to the local file path you give it. The version you get will match the last copy of the file that was written or updated. If the file in the job store gets modified later via updateFile or updateFileStream, it's implementation-defined whether those writes will be visible at localFilePath. Same atomic vibe here too: the file won't appear on your local file system until the copy has completed 🫶

The file at the given local path may not be modified after this method returns! 🚫

Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs. 🏃

Parameters

• file_id (str) — ID of the file to be copied 🆔
• local_path (str) — the local path showing where to put the contents of the given file from the job store 📁
• symlink (bool) — whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link 🔗

Return type

None

---

💅 delete_file(file_id) 💅

Deletes the file with the given ID from this job store. This operation is idempotent, which just means you can delete a file twice, or delete one that doesn't exist, and it'll succeed silently. No drama, no errors 🌸

Parameters

• file_id (str) — ID of the file to delete 🗑️

---

🔍 file_exists(file_id) 🔍

Figures out whether a file exists in this job store. Simple, iconic, done 💜

• file_id — an ID pointing at the file you wanna check 🔍

get_file_size(file_id)

get_file_size(file_id)

Okay so this one gets the size of the file in bytes 📏. If the file doesn't exist when you ask, you get 0 back, so no crashing, just a sad little zero 🥺.

Heads up, bestie: job stores that encrypt files might give you overestimates 🙈. The encrypted file may have been padded out to the nearest block, plus an initialization vector got added, etc. So the number can run a little big.

Parameters

• file_id (str) — an ID referencing the file to be checked

Return type

• int

---

update_file_stream(file_id, encoding=None, errors=None)

update_file_stream(file_id, encoding=None, errors=None)

This is giving replace the existing version of a file in the job store 💅. It's basically like writeFile, except it hands you a context manager that yields a file handle you can write to. You don't need to close the handle yourself, and you really shouldn't 🫶.

Parameters

• file_id (str) — the ID of the file in the job store to be updated
• encoding (str) — the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None, which means binary mode.
• errors (str) — an optional string that tells you how encoding errors get handled. Errors work the same as for open(). Defaults to 'strict' when an encoding is specified.

Raises

• toil.jobStores.abstractJobStore.ConcurrentFileModificationException — if the file got modified concurrently during a call to this method 😬
• toil.jobStores.abstractJobStore.NoSuchFileException — if the specified file does not exist 💀

---

read_file_stream(file_id, encoding=None, errors=None)

Three flavours of signature here, bestie 🌈:

read_file_stream(file_id: str | toil.fileStores.FileID, encoding: Literal[None] = None, errors: str | None = None) -> collections.abc.Iterator[IO[bytes]]
read_file_stream(file_id: str | toil.fileStores.FileID, encoding: str, errors: str | None = None) -> collections.abc.Iterator[IO[str]]
read_file_stream(file_id: str | toil.fileStores.FileID, encoding: str | None = None, errors: str | None = None) -> collections.abc.Iterator[IO[bytes]] | collections.abc.Iterator[IO[str]]

It's similar to readFile, but instead it returns a context manager that yields a file handle you can read from 📖. The yielded handle doesn't need to be closed explicitly, and again, you shouldn't close it yourself.

Parameters

• file_id (str) — ID of the file to get a readable file handle for
• encoding (str) — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which means binary mode.
• errors (str) — an optional string that tells you how encoding errors get handled. Errors work the same as for open(). Defaults to 'strict' when an encoding is specified.

Returns

• a context manager yielding a file handle which can be read from

Return type

• Iterator[Union[IO[bytes], IO[str]]]

---

write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)

write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)

This one returns a context manager yielding a writable file handle to the global file referenced by the given name ✍️. The file gets created in an atomic manner, which is very responsible of it 🧁.

Parameters

• shared_file_name (str) — a file name matching AbstractJobStore.fileNameRegex, and unique within this job store
• encrypted (bool) — True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear
• encoding (str) — the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None, which means binary mode.
• errors (str) — an optional string that tells you how encoding errors get handled. Errors work the same as for open(). Defaults to 'strict' when an encoding is specified.

Raises

• toil.jobStores.abstractJobStore.ConcurrentFileModificationException — if the file got modified concurrently during a call to this method 😬

Returns

• a context manager yielding a writable file handle

Return type

• Iterator[IO[bytes]]

---

read_shared_file_stream(shared_file_name, encoding=None, errors=None)

Two signatures, the same main idea 💜:

read_shared_file_stream(shared_file_name: str, encoding: str, errors: str | None = None) -> collections.abc.Iterator[IO[str]]
read_shared_file_stream(shared_file_name: str, encoding: Literal[None] = None, errors: str | None = None) -> collections.abc.Iterator[IO[bytes]]

Returns a context manager yielding a readable file handle to the global file referenced by the given name 👀.

Parameters

• shared_file_name (str) — a file name matching AbstractJobStore.fileNameRegex, and unique within this job store
• encoding (str) — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which means binary mode.
• errors (str) — an optional string that tells you how encoding errors get handled. Errors work the same as for open(). Defaults to 'strict' when an encoding is specified.

Returns

• a context manager yielding a readable file handle

---

list_all_file_names(for_job=None)

list_all_file_names(for_job=None)

Gets all the file names (not file IDs) of files stored in the job store 🗂️. This one is used for debugging, so it's a dev-only kinda moment 🔧.

💌 write_logs(msg)

Stores a message as a log in the jobstore. 🧾

Parameters

• msg (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — the string to be written

Raises

• ConcurrentFileModificationException (#toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file was modified concurrently during an invocation of this method

📚 read_logs(callback, read_all=False)

Reads the logs that write_logs() has piled up. ✨ Instead of handing you the logs back directly, this method wants a callback function, which gets called once for each log with the message as its argument. So you bring the processing, it brings the logs. 💅

Only unread logs get read, unless you set the read_all parameter. 👀

Parameters

• callback (Callable) — a function to be applied to each of the stats file handles found
• read_all (bool, see /usr/share/doc/python3/html/library/functions.html#bool) — a boolean that decides whether to also read the already processed stats files, alongside the unread ones

Raises

• ConcurrentFileModificationException (#toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file was modified concurrently during an invocation of this method

Returns

• the number of stats files processed

Return type

• int (/usr/share/doc/python3/html/library/functions.html#int)

---

🌸 toil.jobStores.googleJobStore

🧁 Attributes

• log
• GOOGLE_STORAGE
• MAX_BATCH_SIZE

🧁 Classes

• GoogleJobStore — Represents the physical storage for the jobs and files in a Toil workflow. 🪣

🧁 Functions

• google_retry_predicate(e) — necessary because under heavy load google may throw
• google_retry(f) — This decorator retries the wrapped function if google throws any angry service errors.
• permission_error_reporter(url, notes) — Detect and usefully report permission errors. 🚨

🧁 Module Contents

toil.jobStores.googleJobStore.log

toil.jobStores.googleJobStore.GOOGLE_STORAGE = 'gs'

toil.jobStores.googleJobStore.MAX_BATCH_SIZE = 1000

toil.jobStores.googleJobStore.google_retry_predicate(e)

Necessary because under heavy load google may throw this little gem:

TooManyRequests: 429 The project exceeded the rate limit for creating and deleting buckets.

...or numerous other server errors which need to be retried. Ugh, the worst. 😭

toil.jobStores.googleJobStore.google_retry(f)

This decorator retries the wrapped function if google throws any angry service errors. 😤

It should wrap any function that makes use of the Google Client API. 🫶

toil.jobStores.googleJobStore.permission_error_reporter(url, notes)

Detect and usefully report permission errors. This one is lowkey a hero. 🦸‍♀️

If we fall back to anonymous credentials, but they don't have permission for something, the Google Cloud Storage module will try to refresh them behind the scenes. Then it will complain:

<class 'google.auth.exceptions.InvalidOperation'>: Anonymous credentials cannot be refreshed.

We need to detect this and report that the real problem is that the user has not set up any credentials. When you try to make the client non-anonymously and don't have credentials set up, you get a nice error from Google:

google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

But we swallow that when we fall back to anonymous access. Rude. 🙄

So we take the URL and any notes from client setup here, and if something goes wrong that looks like a permission problem we complain with the notes attached. 📝

Also, if you don't have a project available, you can't use the Python API for storage with authentication at all. gsutil can do it but you can't. TODO: Fall back on command-line gsutil for authenticated reads??? 💀

Parameters

• url (urllib.parse.ParseResult, see /usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult)
• notes (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

• Iterator[None]

class toil.jobStores.googleJobStore.GoogleJobStore(locator)

Bases: toil.jobStores.abstractJobStore.AbstractJobStore (#toil.jobStores.abstractJobStore.AbstractJobStore), toil.lib.url.URLAccess (#toil.lib.url.URLAccess) 🎀

Represents the physical storage for the jobs and files in a Toil workflow. 🪣

JobStores are responsible for storing toil.job.JobDescription (#toil.job.JobDescription), which relate jobs to each other, and files. 🗂️

Actual toil.job.Job (#toil.job.Job) objects are stored in files, referenced by JobDescriptions. All the non-file CRUD methods the JobStore provides deal in JobDescriptions and not full, executable Jobs.

To actually get ahold of a toil.job.Job (#toil.job.Job), use toil.job.Job.loadJob() (#toil.job.Job.loadJob) with a JobStore and the relevant JobDescription. 🔑

Parameters

• locator (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

Attributes

• nodeServiceAccountJson = '/root/service_account.json'
• projectID
• bucketName
• bucket = None
• statsBaseID = 'f16eef0c-b597-4b8b-9b0c-4d605b4f506c'
• statsReadPrefix = '_'
• readStatsBaseID = '_f16eef0c-b597-4b8b-9b0c-4d605b4f506c'
• sseKey = None

classmethod create_client()

Produce a client for Google Storage with the highest level of access we can get. 🔓

Fall back to anonymous access if no project is available, unlike the Google Storage module's behavior.

Warn if GOOGLE_APPLICATION_CREDENTIALS is set but not actually present. ⚠️

Returns

• the client, and any notes about why it might not have permissions

Return type

• tuple (/usr/share/doc/python3/html/library/stdtypes.html#tuple)[google.cloud.storage.Client, str] (/usr/share/doc/python3/html/library/stdtypes.html#str)

initialize(config=None)

Initialize this job store. 🌱

Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store.

Parameters

• config — the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID.

Raises

• JobStoreExistsException (#toil.jobStores.abstractJobStore.JobStoreExistsException) — if the physical storage for this job store already exists

resume()

Connect this instance to the physical storage it represents and load the Toil configuration into the AbstractJobStore.config attribute. 🔌

Raises

• NoSuchJobStoreException (#toil.jobStores.abstractJobStore.NoSuchJobStoreException) — if the physical storage for this job store doesn't exist

destroy() 💥

Okay bestie, destroy() is basically the opposite of initialize() 🧹. It deletes the physical storage that this instance stands for. It's not atomic 🙈, but it is idempotent, which is the glow-up that helps with the eventual consistency quirks some storage backends have. So if it fails and raises an exception, you should call it again. 🔁

Even a successful call isn't an ironclad guarantee that the storage vanished completely and instantly on eventually consistent systems ⏳. All a successful call promises is that the deletion will eventually happen. So it's recommended that you do not immediately reuse the same job store location for a new Toil workflow 🚫✨.

assign_job_id(job_description) 🪪

This one hands out a fresh jobStoreID for the job you describe and attaches it to the JobDescription 🎀. Files tied to that new ID get accepted even if the JobDescription has never been created or updated, which is very giving flexibility 🦋.

• job_description (toil.job.JobDescription) — the JobDescription you want to give an ID to 💖

batch() 📦

If your batch system supports it, any calls to create() made while this context manager is active get bundled up and performed as a batch once the context manager is released 🌸. Batch vibes only.

create_job(job_description) 🌱

This writes the given JobDescription into the job store 📝. The job must already have an ID assigned, so make sure you did that first 🫶.

Must call jobDescription.pre_update_hook() 🚨 This is not optional, bestie.

Returns: the JobDescription you passed in ✨.

Return type: toil.job.JobDescription 💅

job_exists(job_id) 🔍

Tells you whether a description of the job with the specified jobStoreID exists in the job store 👀. Yes or no, no drama.

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool) 💜

get_public_url(fileName) 🌐

This returns a publicly accessible URL for the given file in the job store 🔗. Heads up: the URL may expire as early as 1h after it's returned ⏰. If the file doesn't exist, it throws an exception 💀.

• file_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the jobStoreFileID of the file you want a URL for 🗂️

Raises: NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) — if the specified file does not exist in this job store 😭

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str) 🧁

get_shared_public_url(sharedFileName) 🤝

This one is the cousin of getPublicUrl() 👯. The difference is that it's for generating URLs for shared files written by writeSharedFileStream(). It still returns a publicly accessible URL for the given file in the job store 🌈. The returned URL starts with 'http:', 'https:' or 'file:' 🔗. It may expire as early as 1h after it's returned ⏰, and it throws an exception if the file doesn't exist 💔.

• shared_file_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the shared file you want a publicly accessible URL for 🎟️

Raises: NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException) — raised if the specified file does not exist in the store 🙈

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str) ✨

load_job(job_id) 📂

This loads the description of the job referenced by the given ID, gives it the job store's config, and hands it back to you 🎁.

Heads up: it may declare the job to have failed (see toil.job.JobDescription.setupJobAfterFailure()) if there's evidence of a failed update attempt 😬.

• job_id — the ID of the job to load 🆔

Raises: NoSuchJobException (toil.jobStores.abstractJobStore.NoSuchJobException) — if there is no job with the given ID 💀

update_job(job) 🔄

This persists changes to the state of the given JobDescription in this store, and it does so atomically ⚛️. Lowkey the most important write-up of the bunch.

Must call jobDescription.pre_update_hook() 🚨 Again, no skipping.

• job (toil.job.JobDescription) — the job to write to this job store 💼

delete_job(job_id) 🗑️

This removes the JobDescription from the store atomically 💫. After that, you may not call load(), write(), update(), etc. with the same jobStoreID or with any JobDescription bearing it. Once it's gone, it's gone 🫠.

This operation is idempotent 🔁. That means deleting a job twice, or deleting a job that doesn't exist, will succeed silently 🤫.

• job_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the job to delete from this job store 🎯

get_env() 🌿

Returns a dict of environment variables to send out to the workers, so they can load the job store 🧚.

jobs() 🧺

This is a best effort attempt to return an iterator over JobDescriptions for all jobs in the store 🫧. Real talk: it may not return all jobs, and it may also contain orphaned jobs that already finished successfully and should not be rerun 🙃. If you need to guarantee you get any and all jobs that can be run, construct a more expensive ToilState object instead 💅.

Returns: an iterator on jobs in the store 🔁. It may or may not contain all jobs, and it may contain invalid jobs 😵‍💫.

Return type: Iterator[toil.job.jobDescription] 🌸

write_file(local_path, job_id=None, cleanup=False) 📤

This takes a file (as a path) and places it in this job store 📁. It returns an ID you can use to retrieve the file later 🪄. The file is written in an atomic manner, so it won't appear in the job store until the write has successfully completed 🦋.

• local_path (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the path to the local file that gets uploaded to the job store 📎. The last path component (the basename of the file) stays associated with the file in the file store, if supported, so you can search for it by name or name glob 🔍.
• job_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of a job, or None 🆔. If specified, the file may be associated with that job in a job-store-specific way, and this may influence the returned ID 🌟.
• cleanup (bool, /usr/share/doc/python3/html/library/functions.html#bool) — whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job) 🧹. If jobStoreID was not given, this does nothing 🙃.

Raises:

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file was modified concurrently during an invocation of this method 😱
• NoSuchJobException (toil.jobStores.abstractJobStore.NoSuchJobException) — if the job specified via jobStoreID does not exist 💀

FIXME: some implementations may not raise this 🛠️ (so don't count on it, bestie).

Returns: an ID referencing the newly created file, which can be used to read the file in the future 🎀.

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str) 🧁

write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) 🌊

This is similar to writeFile, but instead it returns a context manager 🎬. That context manager yields a tuple of 1) a file handle you can write to and 2) the ID of the resulting file in the job store 🎁. You don't need to close the yielded file handle explicitly, and you should not 🙅‍♀️. The file is written in an atomic manner too, so it won't appear in the job store until the write has successfully completed 💖.

• job_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of a job, or None 🆔. If specified, the file may be associated with that job in a job-store-specific way, and this may influence the returned ID 🌟.
• cleanup (bool, /usr/share/doc/python3/html/library/functions.html#bool) — whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job) 🧹. If jobStoreID was not given, this does nothing 🙃.
• basename (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — okay bestie, if the implementation supports it, the given file basename gets used so that a job store search matching that basename will find the file 🔍✨
• encoding (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None, which means binary mode 🫧
• errors (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — an optional string for how encoding errors get handled. These are the same as for open(), and it defaults to 'strict' when an encoding is specified 💅

Raises

• ConcurrentFileModificationException (#toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file got modified concurrently during an invocation of this method 😭
• NoSuchJobException (#toil.jobStores.abstractJobStore.NoSuchJobException) — if the job specified via jobStoreID does not exist 💀

FIXME: some implementations may not raise this

Returns

A context manager yielding a file handle that can be written to, plus an ID that references the newly created file and can be used to read it in the future. 🎀

Return type

Iterator[Tuple[IO[bytes], str]]

Doc refs: /usr/share/doc/python3/html/library/stdtypes.html#bytes and /usr/share/doc/python3/html/library/stdtypes.html#str.

---

get_empty_file_store_id(job_id=None, cleanup=False, basename=None)

Creates an empty file in the job store and returns its ID. Calling fileExists(getEmptyFileStoreID(jobStoreID)) will return True, so you can literally check it right away ✨

Parameters

• job_id — the ID of a job, or None. If specified, the file may be associated with that job in a job-store-specific way. This may influence the returned ID 🧚
• cleanup — whether to attempt deleting the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, nothing happens 🧹
• basename — if supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected 🔎

Returns

A jobStoreFileID that references the newly created file and can be used to reference the file in the future 🦋

Return type

str (see /usr/share/doc/python3/html/library/stdtypes.html#str)

---

read_file(file_id, local_path, symlink=False)

Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written or updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner, so it won't appear in the local file system until the copy has completed 🌸

The file at the given local path may not be modified after this method returns!

Note! Implementations of readFile need to respect and provide the executable attribute on FileIDs 🔥

Parameters

• file_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — ID of the file to be copied
• local_path (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — the local path showing where to place the contents of the given file in the job store
• symlink (bool, see /usr/share/doc/python3/html/library/functions.html#bool) — whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link 🔗

---

read_file_stream(file_id, encoding=None, errors=None)

Okay so this one is basically readFile's cooler cousin 😎 It returns a context manager yielding a file handle that you can read from. The yielded file handle does not need to be closed explicitly, and it should not be closed explicitly either 🙅‍♀️

Parameters

• file_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — ID of the file to get a readable file handle for
• encoding (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None, which means binary mode
• errors (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — an optional string for how encoding errors get handled. These are the same as for open(), and it defaults to 'strict' when an encoding is specified

Returns

A context manager yielding a file handle that can be read from 📖

Return type

Iterator[Union[IO[bytes], IO[str]]]

Doc refs: /usr/share/doc/python3/html/library/stdtypes.html#bytes and /usr/share/doc/python3/html/library/stdtypes.html#str.

---

delete_file(file_id)

Deletes the file with the given ID from this job store. This operation is idempotent, which means deleting a file twice, or deleting a non-existent file, will succeed silently. No drama 🫶

Parameters

• file_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — ID of the file to delete 🗑️

---

file_exists(file_id)

Determines whether a file exists in this job store 👀

Parameters

• file_id — an ID referencing the file to be checked

---

get_file_size(file_id)

Gets the size of the given file in bytes, or 0 if it does not exist when queried. 📏

Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc.

Parameters

• file_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — an ID referencing the file to be checked

Return type

int (see /usr/share/doc/python3/html/library/functions.html#int)

---

update_file(file_id, local_path)

Replaces the existing version of a file in the job store. It throws an exception if the file does not exist 🚫

Parameters

• file_id — the ID of the file in the job store to be updated
• local_path — the local path to a file that will overwrite the current version in the job store

Raises

• ConcurrentFileModificationException (#toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file was modified concurrently during an invocation of this method 😬
• NoSuchFileException (#toil.jobStores.abstractJobStore.NoSuchFileException) — if the specified file does not exist 🙈

---

update_file_stream(file_id, encoding=None, errors=None)

Replaces the existing version of a file in the job store. It's similar to writeFile, but it returns a context manager yielding a file handle which can be written to 🪄 The yielded file handle does not need to be closed explicitly, and it should not be closed explicitly either 💜

Parameters

• file_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — the ID of the file in the job store to be updated
• encoding (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None, which means binary mode
• errors (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — an optional string for how encoding errors get handled. These are the same as for open(), and it defaults to 'strict' when an encoding is specified

Raises

• ConcurrentFileModificationException (#toil.jobStores.abstractJobStore.ConcurrentFileModificationException) — if the file was modified concurrently during an invocation of this method 😭
• NoSuchFileException (toil.jobStores.abstractJobStore.NoSuchFileException): bestie, this one gets raised when the file you asked for straight up doesn't exist 💀

✨ write_shared_file_stream(shared_file_name, encrypted=True, encoding=None, errors=None) ✨

okay so this one hands you a context manager that yields a writable file handle to the global file with the name you give it 🦋 the file gets created in an atomic manner, so it's giving all-or-nothing energy 💅

Parameters 🎀

• shared_file_name (str): a file name that matches AbstractJobStore.fileNameRegex and is unique inside this job store 🌸
• encrypted (bool): True if the file has to be encrypted, None if it can be encrypted or maybe not, and False if it must be stored in the clear 🔐
• encoding (str): the name of the encoding used to encode the file. Encodings are the same ones you'd use with encode(). It defaults to None, which means binary mode 🧚
• errors (str): an optional string that tells Python how to handle encoding errors. These work the same as they do for open(). It defaults to 'strict' whenever you specify an encoding 🫶

Raises 😭

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException): if the file got modified by someone else while this method was running 🙈

Returns ✨

a context manager yielding a writable file handle

Return type 💜

Iterator[IO[bytes]]

✨ read_shared_file_stream(shared_file_name, isProtected=True, encoding=None, errors=None) ✨

this is the reading counterpart! 📖 it returns a context manager that yields a readable file handle to the global file that matches the name you pass in 🌈

Parameters 🎀

• shared_file_name (str): a file name that matches AbstractJobStore.fileNameRegex and is unique inside this job store 🌸
• encoding (str): the name of the encoding used to decode the file. Decodings match what you'd get from decode(). It defaults to None, which means binary mode 🧚
• errors (str): an optional string that tells Python how to handle encoding errors, same as open(). It defaults to 'strict' when an encoding is specified 🫶

Returns ✨

a context manager yielding a readable file handle

✨ write_logs(msg) ✨

this one stores a message as a log in the jobstore 📝 very simple, very iconic 💅

Parameters 🎀

• msg (str): the string you want written 🍓

Raises 😭

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException): if the file was modified concurrently during an invocation of this method 🙈

Return type 💜

None

✨ read_logs(callback, read_all=False) ✨

this reads the logs that write_logs() has piled up 📚 ngl, it doesn't hand them back to you directly. instead, it calls the callback function you give it once for each log, with the message as the argument. so your callback is the one doing the processing 🦋

by default, only the unread logs get read. set read_all if you want the already-processed ones too 👑

Parameters 🎀

• callback (Callable): a function to be applied to each of the stats file handles found 🧚
• read_all (bool): a boolean that decides whether to also read the stats files that were already processed, on top of the unread ones 🌸

Raises 😭

• ConcurrentFileModificationException (toil.jobStores.abstractJobStore.ConcurrentFileModificationException): if the file was modified concurrently during an invocation of this method 🙈

Returns ✨

the number of stats files processed

Return type 💜

int

---

🌸 toil.jobStores.utils 🌸

Attributes 🎀

• log: the logger attribute for this module 📝

Exceptions 🫶

• JobStoreUnavailableException: raised when a particular type of job store is requested but can't be used 😤

Functions 🧚

• generate_locator(job_store_type[, local_suggestion, ...]): generates a random locator for a job store of the given type. It raises an error if the job store can't be used 🔮

Module Contents 💅

toil.jobStores.utils.log

exception toil.jobStores.utils.JobStoreUnavailableException

Bases: RuntimeError

this is raised when a particular type of job store is requested but can't be used 🙅‍♀️

toil.jobStores.utils.generate_locator(job_store_type, local_suggestion=None, decoration=None)

generates a random locator for a job store of the given type 🎲 if that job store can't be used, it raises a JobStoreUnavailableException 🚨

Parameters 🎀

• job_store_type (str): the registry name of the job store you want to use 🗂️
• local_suggestion (Optional[str]): a path to a nonexistent local directory that's suitable for use as a file job store 📁
• decoration (Optional[str]): an extra string to add to the job store locator, if it's convenient 🎀

Return str ✨

the job store locator for a usable job store

Return type 💜

str

---

🦋 toil.leader 🦋

the leader script (the leader half of the leader/worker pair) that runs your jobs 👑

Attributes 🎀

• logger: the logger for this module 📝

Classes 💅

• Leader: represents the Toil leader 👑

Module Contents 🌈

toil.leader.logger

class toil.leader.Leader(config, batchSystem, provisioner, jobStore, rootJob, jobCache=None)

represents the Toil leader, and it's the one responsible for figuring out which jobs are ready to be scheduled 🧠 it checks in with the job store and then issues those jobs to the batch system 🚀

Parameters 🎀

• config (toil.common.Config) ⚙️
• batchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem) 🖥️
• provisioner (Optional[toil.provisioners.abstractProvisioner.AbstractProvisioner]) ☁️
• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore) 🗃️
• rootJob (toil.job.JobDescription) 🌳
• jobCache (Optional[dict[Union[str, toil.job.TemporaryID], toil.job.JobDescription]]) 🧺

Attributes 💖

• config
• jobStore
• jobStoreLocator
• toilState
• batchSystem

Okay so the leader is holding a whole bag of attributes, and it's giving organized 💅✨

issued_jobs_by_batch_system_id: dict[int, str]
preemptibleJobsIssued = 0
serviceJobsIssued = 0
serviceJobsToBeIssued: list[str] = []
preemptibleServiceJobsIssued = 0
preemptibleServiceJobsToBeIssued: list[str] = []
timeSinceJobsLastRescued = None
reissueMissingJobs_missingHash: dict[int, int]
provisioner
clusterScaler = None
serviceManager
statsAndLogging
potentialDeadlockedJobs: set[str]
potentialDeadlockTime = 0
toilMetrics: toil.common.ToilMetrics | None = None
debugJobNames = ('CWLJob', 'CWLWorkflow', 'CWLScatter', 'CWLGather', 'ResolveIndirect')
deadlockThrottler
statusThrottler
kill_throttler
progress_overall = None
progress_failed = None
GOOD_COLOR = (0, 60, 108)
BAD_COLOR = (253, 199, 0)
PROGRESS_BAR_FORMAT = '{desc}{desc_pad}{percentage:3.0f}%|{bar}| {count:{len_total}d}/{total:d} ({count_1:d} failures)...
recommended_fail_exit_code = 1

The type references behind those annotations point at these docs, bestie 🧚:

• dict → /usr/share/doc/python3/html/library/stdtypes.html#dict
• int → /usr/share/doc/python3/html/library/functions.html#int
• str → /usr/share/doc/python3/html/library/stdtypes.html#str
• list → /usr/share/doc/python3/html/library/stdtypes.html#list
• set → /usr/share/doc/python3/html/library/stdtypes.html#set
• None → /usr/share/doc/python3/html/library/constants.html#None
• bool → /usr/share/doc/python3/html/library/functions.html#bool
• float → /usr/share/doc/python3/html/library/functions.html#float

---

run() 🏃‍♀️

Okay this one is the main event! It runs the leader process to issue and manage jobs. 👑

• Raises toil.exceptions.FailedJobsException if failed jobs remain after running.
• Returns the return value of the root job's run function.
• Return type Any

create_status_sentinel_file(fail) 🚩

Creates a file in the jobstore that says whether things failed or succeeded. 🎀

• Parameters: fail (bool, see /usr/share/doc/python3/html/library/functions.html#bool)
• Return type None

innerLoop() 🔁

Processes jobs. This is the leader's main loop, so it's literally the heartbeat of the whole thing 💖

checkForDeadlocks() 🔒

Checks if the system is deadlocked while running service jobs. Nobody wants a deadlock, no cap 😭

feed_deadlock_watchdog() 🐕

Notes that progress has been made, so any pending deadlock checks get reset. Good girl watchdog 🫶

• Return type None

issueJob(jobNode) 📥

Adds a job to the queue of jobs that are currently trying to run. Main character energy 🌸

• Parameters: jobNode (toil.job.JobDescription)
• Return type None

issueJobs(jobs) 📦

Adds a whole list of jobs, each one represented as a jobNode object. Batch hours 🍓

issueServiceJob(service_id) 🛎️

Issues a service job. If the maximum number of service jobs to be scheduled has already been reached, it goes on a queue instead, so it just waits its turn 💅

• Parameters: service_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type None

issueQueingServiceJobs() ⏳

Issues any queuing service jobs, up to the limit of the maximum allowed. Queue slay 🧚

getNumberOfJobsIssued(preemptible=None) 🔢

Gets the number of jobs that have been added by issueJob(s) and not removed by removeJob. Iconic counting moment 💜

• Parameters: preemptible (Optional[bool], see /usr/share/doc/python3/html/library/functions.html#bool)
  • If None, it returns all types of jobs.
  • If True, it returns just the number of preemptible jobs.
  • If False, it returns just the number of non-preemptible jobs.
• Return type int, see /usr/share/doc/python3/html/library/functions.html#int

removeJob(jobBatchSystemID) 🗑️

Removes a job from the system by its batch system ID. Bye bye, job 👋

• Returns the job description as it was issued.
• Parameters: jobBatchSystemID (int, see /usr/share/doc/python3/html/library/functions.html#int)
• Return type toil.job.JobDescription

getJobs(preemptible=None) 📋

Gets all the issued jobs. Full list, no skips 🦋

• Parameters: preemptible (Optional[bool], see /usr/share/doc/python3/html/library/functions.html#bool)
  • If specified, it selects only preemptible jobs or only non-preemptible jobs.
• Return type list[toil.job.JobDescription], see /usr/share/doc/python3/html/library/stdtypes.html#list

killJobs(jobsToKill, exit_reason=BatchJobExitReason.KILLED) 💀

Kills the given set of jobs and then sends them off for processing. Harsh but fair 😭

• Returns the jobs that were reissued once they got processed.
• Parameters: exit_reason (toil.batchSystems.abstractBatchSystem.BatchJobExitReason)

reissueOverLongJobs() ⏰

Checks each issued job. If a job has been running for longer than is desirable, it issues a kill instruction. Then it waits for the job to die, and after that the job gets passed to process_finished_job. 🫠

• Return type None

reissueMissingJobs(killAfterNTimesMissing=3) 🕵️‍♀️

Checks that all the current job IDs are in the list of currently issued batch system jobs. If a job is missing, it gets marked as missing. If it stays missing for a number of runs of this function (say 10).. then the leader tries deleting the job (though it's probably lost). It waits after that, then passes the job to process_finished_job. Detective era 🔍

processRemovedJob(issuedJob, result_status) 🤷‍♀️

No description is given for this one, so it's a little mysterious 🙈

process_finished_job(batch_system_id, result_status, wall_time=None, exit_reason=None) 🏁

Processes finished jobs. This gets called when an attempt to run a job finishes, whether it succeeded or not. It takes the job out of the issued state, then works out what to do about the success or failure. ✨

• Returns True if the job is going to run again, and False if the job is fully done or completely failed.
• Return type bool, see /usr/share/doc/python3/html/library/functions.html#bool

process_finished_job_description(finished_job, result_status, wall_time=None, exit_reason=None, batch_system_id=None) 🗂️

Processes a finished JobDescription based on whether it succeeded or failed. Here's the breakdown, bestie:

• If wall-clock time is available, it tells the cluster scaler about the job finishing.
• If the job failed and a batch system ID is available, it checks for and reports batch system logs.
• It checks whether the job succeeded and was removed, or whether it failed and needs to be set up after failure, then dispatches to the appropriate function.
• Returns True if the job is going to run again, and False if the job is fully done or completely failed.
• Parameters:
  • finished_job (toil.job.JobDescription)
  • result_status (int, see /usr/share/doc/python3/html/library/functions.html#int)
  • wall_time (Optional[float], see /usr/share/doc/python3/html/library/functions.html#float)
  • exit_reason (Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason])
  • batch_system_id (Optional[int], see /usr/share/doc/python3/html/library/functions.html#int)
• Return type bool, see /usr/share/doc/python3/html/library/functions.html#bool

getSuccessors(job_id, alreadySeenSuccessors) 🌳

Gets the successors of the given job by walking the job graph recursively. Tree climbing era 🧗‍♀️

• Parameters:
  • alreadySeenSuccessors (set[str], see /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/stdtypes.html#str) — any successor seen here is ignored and not traversed.
  • job_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• Returns the set of successors that were found. This set is added to alreadySeenSuccessors.
• Return type set[str], see /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/stdtypes.html#str

processTotallyFailedJob(job_id) 💔

Processes a totally failed job. Rough, but we process it anyway 🫶

• Parameters: job_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type None

✨ toil.lib ✨

🧩 Submodules

Okay bestie, this is the toil.lib grab bag 🎀 Right now it's got one submodule that I'm obsessed with:

🔥 toil.lib.accelerators

Accelerator (i.e. GPU) utilities for Toil. It's giving hardware realness 💅

🔧 Functions

• have_working_nvidia_smi(): checks whether nvidia's CUDA userspace nvidia-smi binary is there and actually runs 💖
• get_host_accelerator_numbers(): works out what accelerator is what 🦋
• have_working_nvidia_docker_runtime(): checks if Docker can handle an "nvidia" runtime and the "--gpus" option 🐳
• count_nvidia_gpus(): counts the nvidia GPUs nvidia-smi can see, or gives back 0 if it's not working 🌸
• count_amd_gpus(): counts the amd GPUs rocm-smi can see, or gives back 0 if it's not working 🍓
• get_individual_local_accelerators(): finds every local accelerator and reports each one with count 1 ✨
• get_restrictive_environment_for_local_accelerators(...): builds environment variables you can apply to a process so it only uses certain accelerators 🔒

📚 Module Contents

toil.lib.accelerators.have_working_nvidia_smi()

Returns True if the nvidia-smi binary, from nvidia's CUDA userspace utilities, is installed and runs successfully. It's giving confirmed working 🙌

TODO: This isn't quite the same as the check that cwltool uses to decide if it can fulfill a CUDARequirement. 🙈

• Return type: bool (docs: /usr/share/doc/python3/html/library/functions.html#bool)

toil.lib.accelerators.get_host_accelerator_numbers()

Figures out what accelerator is what. For each accelerator we can see, it returns the host-side number, meaning the one outside something like a Slurm job. Usually it matches the apparent number, so no drama 💜

It can be used with Docker's --gpus='"device=#,#,#"' option to forward the right GPUs as seen from a Docker daemon 🐳

• Return type: list[int] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/functions.html#int)

toil.lib.accelerators.have_working_nvidia_docker_runtime()

Returns True if Docker exists and can handle an "nvidia" runtime and the "--gpus" option. Main character energy 👑

• Return type: bool (docs: /usr/share/doc/python3/html/library/functions.html#bool)

toil.lib.accelerators.count_nvidia_gpus()

Returns how many nvidia GPUs nvidia-smi sees, or 0 if it isn't working. Simple, iconic 🌈

• Return type: int (docs: /usr/share/doc/python3/html/library/functions.html#int)

toil.lib.accelerators.count_amd_gpus()

Returns how many amd GPUs rocm-smi sees, or 0 if it isn't working. :return: 🍓

• Return type: int (docs: /usr/share/doc/python3/html/library/functions.html#int)

toil.lib.accelerators.get_individual_local_accelerators()

Finds all the local accelerators available. Each one gets reported with count 1, in the order of the number used to assign them. It's a whole lineup 💖

TODO: How will numbers work with multiple types of accelerator? We need an accelerator assignment API. 🫶

• Return type: list[toil.job.AcceleratorRequirement] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list and #toil.job.AcceleratorRequirement)

toil.lib.accelerators.get_restrictive_environment_for_local_accelerators(accelerator_numbers)

Gets environment variables you can apply to a process to restrict it to using only the given accelerator numbers. Lowkey the best way to keep your processes in their lane 🔒✨

The numbers are in the space of accelerators returned by get_individual_local_accelerators().

• Parameters: accelerator_numbers (Union[set[int], list[int]], with docs at /usr/share/doc/python3/html/library/stdtypes.html#set, /usr/share/doc/python3/html/library/functions.html#int, and /usr/share/doc/python3/html/library/stdtypes.html#list)
• Return type: dict[str, str] (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🌍 toil.lib.aws

🧩 Submodules

• toil.lib.aws.ami: the AMI submodule, where the AWS machine images live 🖼️

🎀 toil.lib.aws.ami

🏷️ Attributes

• logger: a logger attribute, no description needed 🦋

💥 Exceptions

• ReleaseFeedUnavailableError: raised when a Flatcar release can't be located 🙈

🔧 Functions

• get_flatcar_ami(ec2_client[, architecture]): retrieves the Flatcar AMI image to use as the base for all Toil autoscaling instances ✨
• flatcar_release_feed_ami(region[, architecture, source]): yields AMI IDs for the given architecture from the Flatcar release feed 🌸
• feed_flatcar_ami_release(ec2_client[, architecture, ...]): checks a Flatcar release feed for the latest Flatcar AMI 🍓
• aws_marketplace_flatcar_ami_search(ec2_client[, ...]): queries AWS for all AMI names matching Flatcar-stable-* and returns the most recent one 🔎

📚 Module Contents

toil.lib.aws.ami.logger

exception toil.lib.aws.ami.ReleaseFeedUnavailableError

Bases: RuntimeError (docs: /usr/share/doc/python3/html/library/exceptions.html#RuntimeError)

Raised when a Flatcar releases can't be located. That's a whole mood 😭

toil.lib.aws.ami.get_flatcar_ami(ec2_client, architecture='amd64')

Retrieves the Flatcar AMI image to use as the base for all Toil autoscaling instances. It's the foundation, literally 🧚💜

AMI must be available to you on AWS 💅 Attempting to launch one without access gets you a 403, so check that first, bestie 💀

Here's the priority order for picking an AMI 🫶

1. User specified AMI via TOIL_AWS_AMI ✨
2. Official AMI from stable.release.flatcar-linux.net 🦋
3. Search the AWS Marketplace 🛍️

If all three sources fail, it raises ReleaseFeedUnavailableError (#toil.lib.aws.ami.ReleaseFeedUnavailableError) 😭

Parameters

• ec2_client (botocore.client.BaseClient): the Boto3 EC2 Client 🧚
• architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>): the architecture type for the new AWS machine. It can be either amd64 or arm64 💜

Return type

str </usr/share/doc/python3/html/library/stdtypes.html#str>

toil.lib.aws.ami.flatcar_release_feed_ami

toil.lib.aws.ami.flatcar_release_feed_ami(region, architecture='amd64', source='stable')

This one yields AMI IDs for the given architecture from the Flatcar release feed 🌸

Parameters

• source (str </usr/share/doc/python3/html/library/stdtypes.html#str>): can be set to a Flatcar release channel ('stable', 'beta', or 'alpha'), 'archive' to check the Internet Archive for a feed, or 'toil' to check if the Toil project has put up a feed 🔥
• region (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🌈
• architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🎀

Return type

Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

It retries if the release feed can't be fetched. If the release feed has a permanent error, it yields nothing. If some entries in the release feed are unparseable, it still yields the others 🧚

toil.lib.aws.ami.feed_flatcar_ami_release

toil.lib.aws.ami.feed_flatcar_ami_release(ec2_client, architecture='amd64', source='stable')

This checks a Flatcar release feed for the latest flatcar AMI and verifies that it's on AWS ⭐

It does not raise exceptions. 🙈

Parameters

• ec2_client (botocore.client.BaseClient): the Boto3 EC2 Client 🧚
• architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>): the architecture type for the new AWS machine. It can be either amd64 or arm64 💜
• source (str </usr/share/doc/python3/html/library/stdtypes.html#str>): can be set to a Flatcar release channel ('stable', 'beta', or 'alpha'), 'archive' to check the Internet Archive for a feed, or 'toil' to check if the Toil project has put up a feed 🔥

Return type

Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

toil.lib.aws.ami.aws_marketplace_flatcar_ami_search

toil.lib.aws.ami.aws_marketplace_flatcar_ami_search(ec2_client, architecture='amd64')

This queries AWS for all AMI names matching Flatcar-stable-* and returns the most recent one 👑

It does not raise exceptions. 🙈

Returns

An AMI name, or None if no matching AMI was found or we couldn't talk to AWS 🍓

Parameters

• ec2_client (botocore.client.BaseClient) 🧚
• architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 💜

Return type

Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

toil.lib.aws.config

Attributes

• S3_PARALLELIZATION_FACTOR 🦋
• S3_PART_SIZE 🌸
• KiB ✨
• MiB ✨
• AWS_MIN_CHUNK_SIZE 💅
• MULTIPART_THRESHOLD 🎀
• AWS_MAX_MULTIPART_COUNT 👑

Functions

• get_s3_multipart_chunk_size(filesize): returns the chunk size of the S3 multipart object, given a file's size in bytes 🧚

Module Contents

toil.lib.aws.config.S3_PARALLELIZATION_FACTOR = 8
toil.lib.aws.config.S3_PART_SIZE = 16777216
toil.lib.aws.config.KiB = 1024
toil.lib.aws.config.MiB = 1048576
toil.lib.aws.config.AWS_MIN_CHUNK_SIZE = 67108864
toil.lib.aws.config.MULTIPART_THRESHOLD = 67108865
toil.lib.aws.config.AWS_MAX_MULTIPART_COUNT = 10000

toil.lib.aws.config.get_s3_multipart_chunk_size

toil.lib.aws.config.get_s3_multipart_chunk_size(filesize)

This returns the chunk size of the S3 multipart object, given a file's size in bytes 💖

Parameters

• filesize (int </usr/share/doc/python3/html/library/functions.html#int>) 🌈

Return type

int </usr/share/doc/python3/html/library/functions.html#int>

toil.lib.aws.iam

Attributes

• logger 🎀
• CLUSTER_LAUNCHING_PERMISSIONS 🫶
• AllowedActionCollection 🔥

✨ Functions ✨

Okay bestie, here's the whole lineup of IAM helpers 💅 Every function is listed with its signature, and I gave the description the glow-up it needed 🦋

• delete_iam_instance_profile(instance_profile_name[, ...]) 🗑️ (there's no blurb for this one in the table, so it's just the signature)
• delete_iam_role(role_name[, region, quiet]) — Nukes an AWS IAM role 💀 Any separate policies get detached from the role, and any inline policies get deleted.
• create_iam_role(role_name, ...[, region]) — Makes a brand new AWS IAM role ✨ Any separate policies get detached from the role, and any inline policies get deleted.
• init_action_collection() — Sets up a fresh action collection 🎀. An action collection holds allowed Actions and NotActions.
• add_to_action_collection(a, b) — Mashes two action collections together 🫶
• policy_permissions_allow(given_permissions[, ...]) — Checks whether one given set of actions is a subset of another given set of actions. It returns true if they are 🦋
• permission_matches_any(perm, list_perms) — Takes a permission and checks whether it's contained within a list of given permissions 👀
• get_actions_from_policy_document(policy_doc) — Given a policy document, it goes through each statement and creates an AllowedActionCollection representing the… (and the table just stops there 💀)
• allowed_actions_attached(iam, attached_policies) — Goes through all the attached policy documents and creates an AllowedActionCollection representing granted permissions 🌸
• allowed_actions_roles(iam, policy_names, role_name) — Returns a dictionary holding a list of all AWS actions allowed for a given role 🧚
• collect_policy_actions(policy_documents) — Collects all of the actions allowed by the given policy documents into one AllowedActionCollection 💜
• allowed_actions_user(iam, policy_names, user_name) — Gets all allowed actions for a user given by user_name. It returns a dictionary, keyed by resource, (and then it trails off too 🙈)
• allowed_actions_group(iam, policy_names, group_name) — Gets all allowed actions for a group given by group_name. It returns a dictionary, keyed by resource, (same trailing vibe 🙈)
• get_policy_permissions(region) — Returns an action collection containing lists of all permission grant patterns, keyed by resource 🔥
• get_aws_account_num() — Returns the AWS account num 🍓

---

✨ Module Contents ✨

Here's what lives inside the module, bestie 🌈

• toil.lib.aws.iam.logger 📒
• toil.lib.aws.iam.CLUSTER_LAUNCHING_PERMISSIONS 🚀 It's a list of permissions, and it starts like this:

toil.lib.aws.iam.CLUSTER_LAUNCHING_PERMISSIONS = ['iam:CreateRole',
'iam:CreateInstanceProfile', 'iam:TagInstanceProfile',
'iam:DeleteRole',...

• toil.lib.aws.iam.AllowedActionCollection 💖

toil.lib.aws.iam.delete_iam_instance_profile(instance_profile_name, region=None, quiet=True)

Parameters 📝

• instance_profile_name (str, </usr/share/doc/python3/html/library/stdtypes.html#str>)
• region (Optional[str], </usr/share/doc/python3/html/library/stdtypes.html#str>)
• quiet (bool, </usr/share/doc/python3/html/library/functions.html#bool>)

Return type: None 🎀

toil.lib.aws.iam.delete_iam_role(role_name, region=None, quiet=True)

Deletes an AWS IAM role. Any separate policies are detached from the role, and any inline policies are deleted 🗑️

Parameters 📝

• role_name (str, </usr/share/doc/python3/html/library/stdtypes.html#str>) — The name of the AWS IAM role.
• region (Optional[str], </usr/share/doc/python3/html/library/stdtypes.html#str>) — The AWS region that the role_name is in.
• quiet (bool, </usr/share/doc/python3/html/library/functions.html#bool>) — Whether or not to print/log information about the deletion to stdout.

Return type: None ✨

toil.lib.aws.iam.create_iam_role(role_name, assume_role_policy_document, policies, region=None)

Creates an AWS IAM role. Any separate policies are detached from the role, and any inline policies are deleted 🌸

Parameters 📝

toil.lib.aws.iam 🔐

• role_name (str): the name of the AWS IAM role 🎀
• region (Optional[str]): the AWS region that the role_name is in 🌍
• assume_role_policy_document (str): policies to create inline with the role 📜
• policies (dict[str, Any]): global policies to attach to the role 💅

Return type: str

toil.lib.aws.iam.init_action_collection()

okay so this one makes a fresh, empty action collection 🧺. An action collection holds allowed Actions and NotActions sorted by resource, and the entries are patterns with wildcards. An Action explicitly allows whatever it matches, so ec2:* lets through every ec2 permission ✨. A NotAction explicitly allows everything that doesn't match its pattern, so iam:* lets through all the non-iam actions 🙈

Return type: AllowedActionCollection

toil.lib.aws.iam.add_to_action_collection(a, b)

this just mashes two action collections together into one 🫶

Parameters

• a (AllowedActionCollection)
• b (AllowedActionCollection)

Return type: AllowedActionCollection

toil.lib.aws.iam.policy_permissions_allow(given_permissions, required_permissions=[])

this checks whether one set of actions is a subset of another set 👀. It returns true if it is, and false otherwise. If the check fails it also prints a warning, so keep an eye on your console ⚠️

Parameters

• required_permissions (list[str]): dictionary containing the actions required, keyed by resource
• given_permissions (AllowedActionCollection): the set of actions granted to a user or role

Return type: bool

toil.lib.aws.iam.permission_matches_any(perm, list_perms)

checks whether a single permission shows up inside a list of permissions. It returns True if it does and False if it doesn't 💖

Parameters

• perm (str): the permission to check, in string form
• list_perms (list[str]): the permission list to check against

Return type: bool

toil.lib.aws.iam.get_actions_from_policy_document(policy_doc)

give this a policy document and it goes through every statement, then builds an AllowedActionCollection that represents the permissions that document grants 🧠

Parameters

• policy_doc (mypy_boto3_iam.type_defs.PolicyDocumentDictTypeDef): the policy document to examine

Return type: AllowedActionCollection

toil.lib.aws.iam.allowed_actions_attached(iam, attached_policies)

goes through every attached policy document and builds an AllowedActionCollection showing what's granted 🦋

Parameters

• iam (mypy_boto3_iam.IAMClient): the IAM client to use
• attached_policies (list[mypy_boto3_iam.type_defs.AttachedPolicyTypeDef]): the attached policies

Return type: AllowedActionCollection

toil.lib.aws.iam.allowed_actions_roles(iam, policy_names, role_name)

returns a dictionary with a list of every aws action a given role is allowed to do 🌸. The dictionary is keyed by resource, and each key gives you the list of policies allowed on that resource.

Parameters

• iam (mypy_boto3_iam.IAMClient): the IAM client to use
• policy_names (list[str]): the names of the policy documents attached to a role
• role_name (str): the name of the role to look up its policies for

Return type: AllowedActionCollection

toil.lib.aws.iam.collect_policy_actions(policy_documents)

takes a bunch of policy documents and squishes all the allowed actions into one AllowedActionCollection 🧁

Parameters

• policy_documents (list[Union[str, mypy_boto3_iam.type_defs.PolicyDocumentDictTypeDef]]): no description given, sadly 😭

Return type: AllowedActionCollection

toil.lib.aws.iam.allowed_actions_user(iam, policy_names, user_name)

gets every allowed action for the user named user_name 🧑‍💻. It returns a dictionary keyed by resource, with a list of the permissions allowed on each resource.

Parameters

• iam (mypy_boto3_iam.IAMClient): the IAM client to use
• policy_names (list[str]): the names of the policy documents attached to a user
• user_name (str): the name of the user to look up its policies for

Return type: AllowedActionCollection

toil.lib.aws.iam.allowed_actions_group(iam, policy_names, group_name)

same energy as the user version, but for a group named group_name 👯. It returns a dictionary keyed by resource, with the permissions allowed on each one.

Parameters

• iam (mypy_boto3_iam.IAMClient): the IAM client to use
• policy_names (list[str]): the names of the policy documents associated with a user (yes, the original says user here, the vibe is slightly off 🙃)
• group_name (str): the name of the group to look up its policies for

Return type: AllowedActionCollection

toil.lib.aws.iam.get_policy_permissions(region)

returns an action collection full of all the permission grant patterns, keyed by the resource they're allowed on 🗝️. You need AWS credentials tied to a user or an assumed role for this one to work 🔑

Parameters

• zone: the AWS zone to connect to 📍
• region (str)

Return type: AllowedActionCollection

toil.lib.aws.iam.get_aws_account_num()

gives you the AWS account number 🔢

Return type: Optional[str]

---

🧁 toil.lib.aws.s3

Attributes

• logger 🪵
• AWS_MAX_MULTIPART_COUNT 🧩
• AWS_MAX_CHUNK_SIZE 📦
• AWS_MIN_CHUNK_SIZE 🐣
• DEFAULT_AWS_CHUNK_SIZE ✨

Exceptions

• AWSKeyNotFoundError: common base class for all non-exit exceptions 💀
• AWSKeyAlreadyExistsError: common base class for all non-exit exceptions 💀
• AWSBadEncryptionKeyError: common base class for all non-exit exceptions 💀

Classes

• MultiPartPipe: an object-oriented wrapper for os.pipe. Clients should subclass it and implement (the docs cut off right there, bestie 😭)

✨ Functions ✨

okay bestie, here's the whole lineup of S3 helper functions in toil.lib.aws.s3 🪣💖

• create_s3_bucket(s3_resource, bucket_name, region[, ...]) — makes a brand new AWS S3 bucket using the Boto3 S3 session you hand it, with the name and region you want 🎀
• delete_s3_bucket(s3_resource, bucket_name[, quiet]) — deletes the bucket called bucket_name 🗑️
• bucket_exists(s3_resource, bucket)
• head_s3_object(bucket, key, header[, region]) — attempts a HEAD request on an s3 object and hands back its response 👀
• list_multipart_uploads(bucket, region, prefix[, ...])
• copy_s3_to_s3(s3_resource, src_bucket, src_key, ...[, ...])
• copy_local_to_s3(s3_resource, local_file_path, ...[, ...])
• copy_s3_to_local(s3_resource, local_file_path, ...[, ...])
• parse_s3_uri(uri)
• list_s3_items(s3_resource, bucket, prefix[, startafter])
• upload_to_s3(readable, s3_resource, bucket, key[, ...]) — uploads a readable object to s3, using multipart uploading when it applies 🚀
• download_stream(s3_resource, bucket, key[, ...]) — a context manager that gives you a download stream so you can pull the data down 🧚
• download_fileobject(s3_resource, bucket, key, fileobj)
• s3_key_exists(s3_resource, bucket, key[, check, ...]) — returns True if the s3 object exists and False if it doesn't. Heads up: it will error if the encryption args are incorrect 💀
• get_s3_object(s3_resource, bucket, key[, extra_args])
• put_s3_object(s3_resource, bucket, key, body[, extra_args])
• generate_presigned_url(s3_resource, bucket, key_name, ...)
• create_public_url(s3_resource, bucket, key)
• get_s3_bucket_region(s3_resource, bucket)

---

🌈 Module Contents 🌈

• toil.lib.aws.s3.logger
• toil.lib.aws.s3.AWS_MAX_MULTIPART_COUNT = 10000
• toil.lib.aws.s3.AWS_MAX_CHUNK_SIZE = 5000000000000
• toil.lib.aws.s3.AWS_MIN_CHUNK_SIZE = 5000000
• toil.lib.aws.s3.DEFAULT_AWS_CHUNK_SIZE = 134217728

💅 Exceptions

Okay, these three exceptions are all the same vibe. Each one is a subclass of Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception), and the description is "Common base class for all non-exit exceptions." 🙈

• exception toil.lib.aws.s3.AWSKeyNotFoundError — bases: Exception
• exception toil.lib.aws.s3.AWSKeyAlreadyExistsError — bases: Exception
• exception toil.lib.aws.s3.AWSBadEncryptionKeyError — bases: Exception

🫶 Function Details

• toil.lib.aws.s3.create_s3_bucket(s3_resource, bucket_name, region, tags=None, public=True)

Create an AWS S3 bucket using the given Boto3 S3 session, with the given name, in the given region. It supports the us-east-1 region, where bucket creation is special. ALL S3 bucket creation should use this function. 💖

Parameters:

• s3_resource (mypy_boto3_s3.S3ServiceResource)
• bucket_name (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• region (Union[mypy_boto3_s3.literals.BucketLocationConstraintType, Literal['us-east-1']])
• tags (Optional[Dict[str, str]])
• public (bool (/usr/share/doc/python3/html/library/functions.html#bool))

Return type: mypy_boto3_s3.service_resource.Bucket

• toil.lib.aws.s3.delete_s3_bucket(s3_resource, bucket_name, quiet=True)

Delete the bucket with 'bucket_name'. 🗑️

Note: 'quiet' is False when used for a clean up utility script (contrib/admin/cleanup_aws_resources.py) that prints progress rather than logging. Logging should be used for all other internal Toil usage.

Parameters:

• s3_resource (mypy_boto3_s3.S3ServiceResource)
• bucket_name (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• quiet (bool (/usr/share/doc/python3/html/library/functions.html#bool))

Return type: None

• toil.lib.aws.s3.bucket_exists(s3_resource, bucket)

Parameters:

• s3_resource (mypy_boto3_s3.S3ServiceResource)
• bucket (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: Union[bool (/usr/share/doc/python3/html/library/functions.html#bool), mypy_boto3_s3.service_resource.Bucket]

• toil.lib.aws.s3.head_s3_object(bucket, key, header, region=None)

Attempt to HEAD an s3 object and return its response. 👀

Parameters:

• bucket (str (/usr/share/doc/python3/html/library/stdtypes.html#str)) -- AWS bucket name
• key (str (/usr/share/doc/python3/html/library/stdtypes.html#str)) -- AWS Key name for the s3 object

• header (Dict[str, Any]) — the headers to include, mostly for encryption stuff 🔐. There's a boto3 head_object reference page at boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/head_object.html if you want the full tea 🍵
• region (Optional[str]) — the region where we want to look for the bucket 🌸

Return type: mypy_boto3_s3.type_defs.HeadObjectOutputTypeDef ✨

---

✨ toil.lib.aws.s3.list_multipart_uploads ✨

toil.lib.aws.s3.list_multipart_uploads(bucket, region, prefix, max_uploads=1)

• bucket (str) 🪣
• region (str) 🗺️
• prefix (str) 🏷️
• max_uploads (int) — defaults to 1 🔢

Return type: mypy_boto3_s3.type_defs.ListMultipartUploadsOutputTypeDef 💖

---

✨ toil.lib.aws.s3.copy_s3_to_s3 ✨

toil.lib.aws.s3.copy_s3_to_s3(s3_resource, src_bucket, src_key, dst_bucket, dst_key, extra_args=None)

• s3_resource (mypy_boto3_s3.S3ServiceResource) 🧚
• src_bucket (str) 📦
• src_key (str) 🔑
• dst_bucket (str) 📦
• dst_key (str) 🔑
• extra_args (Optional[Dict[Any, Any]]) — defaults to None 🎀

Return type: None 🫶

---

✨ toil.lib.aws.s3.copy_local_to_s3 ✨

toil.lib.aws.s3.copy_local_to_s3(s3_resource, local_file_path, dst_bucket, dst_key, extra_args=None)

• s3_resource (mypy_boto3_s3.S3ServiceResource) 🧚
• local_file_path (str) 💻
• dst_bucket (str) 📦
• dst_key (str) 🔑
• extra_args (Optional[Dict[Any, Any]]) — defaults to None 🎀

Return type: None 🫶

---

✨ toil.lib.aws.s3.copy_s3_to_local ✨

toil.lib.aws.s3.copy_s3_to_local(s3_resource, local_file_path, src_bucket, src_key, extra_args=None)

• s3_resource (mypy_boto3_s3.S3ServiceResource) 🧚
• local_file_path (str) 💻
• src_bucket (str) 📦
• src_key (str) 🔑
• extra_args (Optional[Dict[Any, Any]]) — defaults to None 🎀

Return type: None 🫶

---

✨ class toil.lib.aws.s3.MultiPartPipe ✨

class toil.lib.aws.s3.MultiPartPipe(part_size, s3_resource, bucket_name, file_id, encryption_args, encoding=None, errors=None)

Bases: toil.lib.pipes.WritablePipe 🦋

okay so this class is an object-oriented wrapper for os.pipe and it is so clean 🧼. You subclass it, implement readFrom() to consume the readable end of the pipe, and then instantiate the class as a context manager to get the writable end. Here's the vibe:

>>> import sys, shutil, codecs
>>> class MyPipe(WritablePipe):
...     def readFrom(self, readable):
...         shutil.copyfileobj(codecs.getreader('utf-8')(readable), sys.stdout)
>>> with MyPipe() as writable:
...     _ = writable.write('Hello, world!\n'.encode('utf-8'))
Hello, world!

Each instance creates a thread and invokes the readFrom method in that thread 🧵. The thread gets join()ed upon normal exit from the context manager, which means the body of the with statement finishing up nicely 💅. If an exception occurs, the thread will not be joined, but a well-behaved readFrom() implementation will terminate shortly thereafter because the pipe has been closed 🫠.

Now, exceptions in the reader thread get reraised in the main thread, so you won't be left wondering what went wrong 👀:

>>> class MyPipe(WritablePipe):
...     def readFrom(self, readable):
...         raise RuntimeError('Hello, world!')
>>> with MyPipe() as writable:
...     pass
Traceback (most recent call last):
...
RuntimeError: Hello, world!

More complicated, less illustrative tests (bestie, these are the nerdy ones 🤓):

Same as above, but proving that handles are closed:

>>> x = os.dup(0); os.close(x)
>>> class MyPipe(WritablePipe):
...     def readFrom(self, readable):
...         raise RuntimeError('Hello, world!')
>>> with MyPipe() as writable:
...     pass
Traceback (most recent call last):
...
RuntimeError: Hello, world!
>>> y = os.dup(0); os.close(y); x == y
True

Exceptions in the body of the with statement aren't masked, and handles are closed too 🙌:

>>> x = os.dup(0); os.close(x)
>>> class MyPipe(WritablePipe):
...     def readFrom(self, readable):
...         pass
>>> with MyPipe() as writable:
...     raise RuntimeError('Hello, world!')
Traceback (most recent call last):
...
RuntimeError: Hello, world!
>>> y = os.dup(0); os.close(y); x == y
True

Parameters 🎀

• part_size (int) 📏
• s3_resource (mypy_boto3_s3.S3ServiceResource) 🧚
• bucket_name (str) 📦
• file_id (str) 🆔
• encryption_args (Optional[Dict[Any, Any]]) 🔐
• encoding (Optional[str]) 🔤
• errors (Optional[str]) ⚠️

Class attributes 🌈

• encoding = None
• errors = None
• part_size
• s3_client
• bucket_name
• file_id
• encryption_args

✨ readFrom(readable) ✨

Implement this method to read data from the pipe 📖. It should support both binary and text mode output, so make it work for both bestie 💜.

• readable (file) — the file object representing the readable end of the pipe. Do not explicitly invoke the close() method of the object, because that gets done automatically 🙅‍♀️.

Return type: None 🫶

---

✨ toil.lib.aws.s3.parse_s3_uri ✨

toil.lib.aws.s3.parse_s3_uri(uri)

• uri (str) 🗺️

Return type: Tuple[str, str] 💅

---

✨ toil.lib.aws.s3.list_s3_items ✨

toil.lib.aws.s3.list_s3_items(s3_resource, bucket, prefix, startafter=None)

• s3_resource (mypy_boto3_s3.S3ServiceResource) 🧚
• bucket (str) 📦
• prefix (str) 🏷️
• startafter (Optional[str]) — defaults to None ⏭️

Return type: Iterator[mypy_boto3_s3.type_defs.ObjectTypeDef] 🔁

---

✨ toil.lib.aws.s3.upload_to_s3 ✨

toil.lib.aws.s3.upload_to_s3(readable, s3_resource, bucket, key, extra_args=None)

Upload a readable object to S3, using multipart uploading if applicable 🚀.

• readable (IO[Any]) — a readable stream or a local file path to upload to S3 📤
• resource (S.Resource) — boto3 resource 🧚
• bucket (str) — name of the bucket to upload to 📦
• key (str) — the name of the file to upload to 🔑
• extra_args (dict) — HTTP headers to use when uploading, generally used for encryption purposes 🔐
• partSize (int) — max size of each part in the multipart upload, in bytes 📏
• s3_resource (mypy_boto3_s3.S3ServiceResource) 🧚

Returns: version of the newly uploaded file ✨

Return type: None 🫶

---

✨ toil.lib.aws.s3.download_stream ✨

toil.lib.aws.s3.download_stream(s3_resource, bucket, key, checksum_to_verify=None, extra_args=None, encoding=None, errors=None)

Context manager that gives out a download stream to download data 📥.

• s3_resource (mypy_boto3_s3.S3ServiceResource) 🧚
• bucket (str) 📦
• key (str) 🔑
• checksum_to_verify (Optional[str]) — defaults to None ✅
• extra_args (Optional[Dict[Any, Any]]) — defaults to None 🎀
• encoding (Optional[str]) — defaults to None 🔤
• errors (Optional[str]) — defaults to None ⚠️

Return type: Iterator[IO[Any]] 🔁

---

✨ toil.lib.aws.s3.download_fileobject ✨

toil.lib.aws.s3.download_fileobject(s3_resource, bucket, key, fileobj, extra_args=None)

• s3_resource (mypy_boto3_s3.S3ServiceResource) 🧚
• bucket (mypy_boto3_s3.service_resource.Bucket) 📦
• key (str) 🔑
• fileobj (io.BytesIO </usr/share/doc/python3/html/library/io.html#io.BytesIO>) 🧚
• extra_args (Optional[Dict[Any, Any]]) 🎀

Return type: None 💖

---

🔎 toil.lib.aws.s3.s3_key_exists 🔎

Okay bestie, this one is the "does this thing even exist?" check 👀 It gives back True if the s3 object is there and False if not. Heads up though, it will error if your encryption args are incorrect 💀

toil.lib.aws.s3.s3_key_exists(s3_resource, bucket, key, check=False, extra_args=None)

Parameters

• s3_resource (mypy_boto3_s3.S3ServiceResource) ✨
• bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🪣
• key (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🔑
• check (bool </usr/share/doc/python3/html/library/functions.html#bool>) 🧐
• extra_args (Optional[Dict[Any, Any]]) 🎀

Return type: bool </usr/share/doc/python3/html/library/functions.html#bool> 💅

---

🪄 toil.lib.aws.s3.get_s3_object 🪄

This one is the "go grab me the object" move 🧺

toil.lib.aws.s3.get_s3_object(s3_resource, bucket, key, extra_args=None)

Parameters

• s3_resource (mypy_boto3_s3.S3ServiceResource) ✨
• bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🪣
• key (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🔑
• extra_args (Optional[Dict[Any, Any]]) 🎀

Return type: mypy_boto3_s3.type_defs.GetObjectOutputTypeDef 👑

---

📦 toil.lib.aws.s3.put_s3_object 📦

The "drop it in the bucket" energy 🫶

toil.lib.aws.s3.put_s3_object(s3_resource, bucket, key, body, extra_args=None)

Parameters

• s3_resource (mypy_boto3_s3.S3ServiceResource) ✨
• bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🪣
• key (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🔑
• body (Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]) 🍓
• extra_args (Optional[Dict[Any, Any]]) 🎀

Return type: mypy_boto3_s3.type_defs.PutObjectOutputTypeDef 💜

---

🔗 toil.lib.aws.s3.generate_presigned_url 🔗

toil.lib.aws.s3.generate_presigned_url(s3_resource, bucket, key_name, expiration)

Parameters

• s3_resource (mypy_boto3_s3.S3ServiceResource) ✨
• bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🪣
• key_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🔑
• expiration (int </usr/share/doc/python3/html/library/functions.html#int>) ⏳

Return type: str </usr/share/doc/python3/html/library/stdtypes.html#str> 🦋

---

🌍 toil.lib.aws.s3.create_public_url 🌍

toil.lib.aws.s3.create_public_url(s3_resource, bucket, key)

Parameters

• s3_resource (mypy_boto3_s3.S3ServiceResource) ✨
• bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🪣
• key (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🔑

Return type: str </usr/share/doc/python3/html/library/stdtypes.html#str> 🌈

---

🗺️ toil.lib.aws.s3.get_s3_bucket_region 🗺️

toil.lib.aws.s3.get_s3_bucket_region(s3_resource, bucket)

Parameters

• s3_resource (mypy_boto3_s3.S3ServiceResource) ✨
• bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🪣

Return type: str </usr/share/doc/python3/html/library/stdtypes.html#str> ⭐

---

🌸 toil.lib.aws.session 🌸

🎀 Attributes 🎀

• logger 📝
• ANONYMOUS_CONFIG 🙈

🧁 Classes 🧁

• AWSConnectionManager — a class that represents a connection to AWS and caches Boto 3 and Boto 2 objects 💖

🛠️ Functions 🛠️

• establish_boto3_session([region_name]) — gets a Boto 3 session that the current thread can use 🧚
• client(…) — gets a Boto 3 client for a particular AWS service, usable by the current thread 🦋
• resource(…) — gets a Boto 3 resource for a particular AWS service, usable by the current thread 💅

📚 Module Contents 📚

• toil.lib.aws.session.logger
• toil.lib.aws.session.ANONYMOUS_CONFIG
• class toil.lib.aws.session.AWSConnectionManager

Okay so this class is literally the connection hub 👑 It represents a connection to AWS and caches Boto 3 and Boto 2 objects by region ✨

Anytime you want an item, you go through the method for the thing you want (session, resource, service, Boto2 Context) and then pass in the region you want to work in, plus the type of thing you want if it applies 💅

This class is meant to eventually enable multi-region clusters, where connections to several regions might need to be managed in the same provisioner 🌈

We also support None for a region. In that case no region gets passed to Boto or Boto3, so the caller is responsible for implementing things like TOIL_AWS_REGION support 🫶

Connection objects may not be thread safe, so one is created for each thread that calls the relevant lookup method 🙈 See https://boto3.amazonaws.com/v1/documentation/api/1.14.31/guide/session.html#multithreading-or-multiprocessing-with-sessions for the details.

Attributes of the class

sessions_by_region: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, threading.local </usr/share/doc/python3/html/library/threading.html#threading.local>]

resource_cache: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>], threading.local </usr/share/doc/python3/html/library/threading.html#threading.local>]

client_cache: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>], threading.local </usr/share/doc/python3/html/library/threading.html#threading.local>]

boto2_cache: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, str </usr/share/doc/python3/html/library/stdtypes.html#str>], threading.local </usr/share/doc/python3/html/library/threading.html#threading.local>]

session(region) 🌤️

Gets the Boto3 Session to use for the given region ✨

Parameters

• region (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) 🗺️

Return type: boto3.session.Session 💜

resource(...) 🧩

resource(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['s3'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_s3.S3ServiceResource

✨ API REFERENCE, continued ✨

okay so the resource methods are up next and they're giving whole main character energy 🦋 Each one takes a service_name and hands you back a Boto3 resource object for that service 💅

resource(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['iam'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_iam.IAMServiceResource

resource(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['ec2'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_ec2.EC2ServiceResource

Get the Boto3 Resource to use with the given service (like 'ec2') in the given region. 🌸

Parameters

• endpoint_url — AWS endpoint URL to use for the client. If not specified, a default is used. 🫶

Next up are the client methods, which are the same vibe but hand you a client instead of a resource 💜 Pick your service_name and you're set ✨

client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['ec2'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_ec2.EC2Client

client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['iam'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_iam.IAMClient

client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['s3'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_s3.S3Client

client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['sts'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_sts.STSClient

client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['sdb'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_sdb.SimpleDBClient

client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['autoscaling'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_autoscaling.AutoScalingClient

Get the Boto3 Client to use with the given service (like 'ec2') in the given region. 🎀

Parameters

• endpoint_url — AWS endpoint URL to use for the client. If not specified, a default is used. 🦋
• config — Custom configuration to use for the client. ⭐

---

toil.lib.aws.session

Now we're in the session module, and this one's a whole mood 🌈

establish_boto3_session

toil.lib.aws.session.establish_boto3_session(region_name=None)

Get a Boto 3 session usable by the current thread. Heads up bestie: this function may not always establish a brand-new session, because it can be memoized 🧚

Parameters

• region_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) — the page doesn't describe this one any further 🙈

Return type

• boto3.Session 💖

client

These are the module-level version of the client methods. They're the global alternative to AWSConnectionManager, and they're usable by the current thread 🫶

toil.lib.aws.session.client(service_name: Literal['ec2'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_ec2.EC2Client

toil.lib.aws.session.client(service_name: Literal['iam'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_iam.IAMClient

toil.lib.aws.session.client(service_name: Literal['s3'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_s3.S3Client

toil.lib.aws.session.client(service_name: Literal['sts'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_sts.STSClient

toil.lib.aws.session.client(service_name: Literal['sdb'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_sdb.SimpleDBClient

toil.lib.aws.session.client(service_name: Literal['autoscaling'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_autoscaling.AutoScalingClient

Get a Boto 3 client for a particular AWS service, usable by the current thread. 💅

resource

And the module-level resource version, which is also a global alternative to AWSConnectionManager and usable by the current thread ✨

toil.lib.aws.session.resource(service_name: Literal['s3'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_s3.S3ServiceResource

toil.lib.aws.session.resource(service_name: Literal['iam'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_iam.IAMServiceResource

toil.lib.aws.session.resource(service_name: Literal['ec2'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_ec2.EC2ServiceResource

Get a Boto 3 resource for a particular AWS service, usable by the current thread. 🦋

---

toil.lib.aws.utils

Last stop on this part: the utils module 🌸

Attributes

• logger — the page leaves this one blank, so no details yet 🙈
• THROTTLED_ERROR_CODES — the page leaves this one blank too 💀

💅 Exceptions

• NoBucketLocationError — okay so this is the error that shows up when we literally could not get a location for a bucket 😭

🔥 Functions

• delete_sdb_domain(sdb_domain_name[, region, quiet]) — no description for this one, bestie 🙈
• connection_reset(e) — returns true if an error is a connection reset error 🔌
• connection_error(e) — returns True if an error represents a failure to make a network connection 🌐
• retryable_s3_errors(e) — returns true if this is an S3 error that looks like we ought to retry our request 🔁
• retry_s3([delays, timeout, predicate]) — a retry iterator of context managers, specifically for S3 operations 💜
• delete_s3_bucket(s3_resource, bucket[, quiet]) — deletes the given S3 bucket 🗑️
• create_s3_bucket(s3_resource, bucket_name, region) — creates an AWS S3 bucket using the given Boto3 S3 session, with the... (the docs trail off right there, bestie 💀)
• enable_public_objects(bucket_name) — lets a bucket hold objects that are public 🌍
• get_bucket_region(bucket_name[, endpoint_url, ...]) — gets the AWS region name associated with the given S3 bucket, or raises NoBucketLocationError ✨
• get_bucket_region_if_available(bucket_name[, ...]) — gets the AWS region name associated with the given S3 bucket, or returns None 🎀
• region_to_bucket_location(region) — no description here either, bestie 🙈
• bucket_location_to_region(location) — also no description, the docs are giving mysterious 🕵️
• get_object_for_url(url[, existing, anonymous]) — extracts a key (object) from a given parsed s3:// URL 🧚
• list_objects_for_url(url[, anonymous]) — extracts a key (object) from a given parsed s3:// URL. The URL will be... (and the docs stop mid-sentence, not me being messy 😭)
• flatten_tags(tags) — converts tags from a key-to-value dict into a list of 'Key': xxx, 'Value': xxx dicts 🏷️
• boto3_pager(requestor_callable, result_attribute_name, ...) — yields all the results from calling the given Boto 3 method with the (cut off again, the docs be like "with the" 💀)

---

🌈 Module Contents

• toil.lib.aws.utils.logger — a logger just chilling in the module 🪵
• toil.lib.aws.utils.THROTTLED_ERROR_CODES = ['Throttling', 'ThrottlingException', 'ThrottledException', 'RequestThrottledException',... — the list of "slow down" error codes, lowkey iconic 🐢

🌸 toil.lib.aws.utils.delete_sdb_domain

toil.lib.aws.utils.delete_sdb_domain(sdb_domain_name, region=None, quiet=True)

Parameters

• sdb_domain_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• region (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])
• quiet (bool </usr/share/doc/python3/html/library/functions.html#bool>)

Return type: None

🌸 toil.lib.aws.utils.connection_reset

toil.lib.aws.utils.connection_reset(e)

Returns true if the error is a connection reset error, periodt 🔌

Parameters

• e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

Return type: bool </usr/share/doc/python3/html/library/functions.html#bool>

🌸 toil.lib.aws.utils.connection_error

toil.lib.aws.utils.connection_error(e)

Returns True if the error represents a failure to make a network connection 🌐

Parameters

• e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

Return type: bool </usr/share/doc/python3/html/library/functions.html#bool>

🌸 toil.lib.aws.utils.retryable_s3_errors

toil.lib.aws.utils.retryable_s3_errors(e)

Returns true if this is an S3 error that looks like we ought to retry our request. It's giving "try again, bestie" 🔁

Parameters

• e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

Return type: bool </usr/share/doc/python3/html/library/functions.html#bool>

🌸 toil.lib.aws.utils.retry_s3

toil.lib.aws.utils.retry_s3(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=retryable_s3_errors)

A retry iterator of context managers, specifically for S3 operations 💜

Parameters

• delays (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[float </usr/share/doc/python3/html/library/functions.html#float>])
• timeout (float </usr/share/doc/python3/html/library/functions.html#float>)
• predicate (Callable[[Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>], bool </usr/share/doc/python3/html/library/functions.html#bool>])

Return type: collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[ContextManager[None]]

🌸 toil.lib.aws.utils.delete_s3_bucket

toil.lib.aws.utils.delete_s3_bucket(s3_resource, bucket, quiet=True)

Deletes the given S3 bucket 🗑️

⚠️ bestie, read this part: S3 bucket deletion is only eventually-consistent, and anyone can register any S3 bucket name. For both of these reasons, a bucket name might not be immediately (or ever) available for re-use after this function returns.

Parameters

🪣 toil.lib.aws.utils.create_s3_bucket

toil.lib.aws.utils.create_s3_bucket(s3_resource, bucket_name, region)

Okay so this one makes an AWS S3 bucket 🪣✨ You give it a Boto3 S3 session, a name, and a region, and it creates a bucket with that name in that region. Iconic.

It supports the us-east-1 region, where bucket creation is special 🙈 so bestie, keep that in mind.

ALL S3 bucket creation should use this function. Like, all of it 💅

Parameters

• s3_resource (mypy_boto3_s3.S3ServiceResource)
• bucket_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• region (toil.lib.aws.AWSRegionName)

Return type mypy_boto3_s3.service_resource.Bucket 🎁

---

🔓 toil.lib.aws.utils.enable_public_objects

toil.lib.aws.utils.enable_public_objects(bucket_name)

Lowkey the most dramatic one 🎭 This makes a bucket able to hold objects that are public. It changes the bucket's Public Access Block setting so it doesn't block all public access, and it changes the bucket's Object Ownership setting to one that enables object ACLs 🔓

It does not touch the account's Public Access Block setting, which can also interfere here. That's probably best left to the account administrator 🙅

This configuration used to be the default, and it's what most of Toil's code is written to expect 🫶 But it got changed so new buckets default to the more restrictive setting https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/, with the expectation that people would write IAM policies for the buckets to allow public access if needed. Toil expects to be able to make arbitrary objects in arbitrary places public, and naming them all in an IAM policy would be a very awkward way to do it. So we restore the old behavior 🔁

Parameters

• bucket_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type None

---

💥 exception toil.lib.aws.utils.NoBucketLocationError

exception toil.lib.aws.utils.NoBucketLocationError

Bases Exception (see /usr/share/doc/python3/html/library/exceptions.html#Exception)

This is the error for when we couldn't figure out a location for a bucket 😭

---

🗺️ toil.lib.aws.utils.get_bucket_region

toil.lib.aws.utils.get_bucket_region(bucket_name, endpoint_url=None,
only_strategies=None, anonymous=None)

Gets the AWS region name for the given S3 bucket, or raises NoBucketLocationError 🙃

It doesn't log at info level or above when this doesn't work, since failures are totally expected in some contexts 🤷 It also takes an optional S3 API URL override.

Parameters

• only_strategies (Optional[set[int]], see /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/functions.html#int) — for testing, use only strategies with 1-based numbers in this set 🧪
• bucket_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• endpoint_url (Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str)
• anonymous (Optional[bool], see /usr/share/doc/python3/html/library/functions.html#bool)

Raises

• toil.lib.aws.utils.NoBucketLocationError — if the bucket's region can't be determined (possibly due to lack of permissions) 🚫

Return type str (see /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🗺️ toil.lib.aws.utils.get_bucket_region_if_available

toil.lib.aws.utils.get_bucket_region_if_available(bucket_name,
endpoint_url=None, only_strategies=None, anonymous=None)

Same vibe as the one above, but if it can't find the region it just returns None 🌸

It caches results, so it might not return the location for a bucket that's been created but was previously seen as nonexistent. Main character glitch 👀

Parameters

• only_strategies (Optional[set[int]], see /usr/share/doc/python3/html/library/stdtypes.html#set and /usr/share/doc/python3/html/library/functions.html#int) — for testing, use only strategies with 1-based numbers in this set 🧪
• bucket_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• endpoint_url (Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str)
• anonymous (Optional[bool], see /usr/share/doc/python3/html/library/functions.html#bool)

Return type Optional[str] (see /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🔁 toil.lib.aws.utils.region_to_bucket_location

toil.lib.aws.utils.region_to_bucket_location(region)

Parameters

• region (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type str (see /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🔁 toil.lib.aws.utils.bucket_location_to_region

toil.lib.aws.utils.bucket_location_to_region(location)

Parameters

• location (Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type str (see /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🔗 toil.lib.aws.utils.get_object_for_url

toil.lib.aws.utils.get_object_for_url(url, existing=None,
anonymous=None)

Pulls out a key (object) from a given parsed s3:// URL 🔍

If existing is true and the object doesn't exist, this raises FileNotFoundError 😬

Parameters

• existing (bool, see /usr/share/doc/python3/html/library/functions.html#bool) — if True, the key is expected to exist. If False, the key is expected not to exist, and it'll be created. If None, the key gets created if it doesn't already exist 🧠
• url (urllib.parse.ParseResult, see /usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult)
• anonymous (Optional[bool], see /usr/share/doc/python3/html/library/functions.html#bool)

Raises

• FileNotFoundError (see /usr/share/doc/python3/html/library/exceptions.html#FileNotFoundError) — when existing is True and the object doesn't exist 👻
• RuntimeError (see /usr/share/doc/python3/html/library/exceptions.html#RuntimeError) — when existing is False but the object does exist 💀
• PermissionError (see /usr/share/doc/python3/html/library/exceptions.html#PermissionError) — when we aren't authorized to look at the object 🔒

Return type mypy_boto3_s3.service_resource.Object

---

📋 toil.lib.aws.utils.list_objects_for_url

toil.lib.aws.utils.list_objects_for_url(url, anonymous=None)

Pulls out a key (object) from a given parsed s3:// URL. If the URL is missing a trailing slash, one gets added for you 🪄

Raises

• PermissionError (see /usr/share/doc/python3/html/library/exceptions.html#PermissionError) — when we aren't authorized to do the list operation 🙅

Parameters

• url (urllib.parse.ParseResult, see /usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult)
• anonymous (Optional[bool], see /usr/share/doc/python3/html/library/functions.html#bool)

Return type list[str] (see /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🏷️ toil.lib.aws.utils.flatten_tags

toil.lib.aws.utils.flatten_tags(tags)

Converts tags from a key-to-value dict into a list of 'Key': xxx, 'Value': xxx dicts 🏷️✨

Parameters

• tags (dict[str, str], see /usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type list[dict[Union[Literal['Key'], Literal['Value']], str]] (see /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#dict)

---

🔄 toil.lib.aws.utils.boto3_pager

toil.lib.aws.utils.boto3_pager(requestor_callable,
result_attribute_name, **kwargs)

This one yields all the results from calling the given Boto3 method with the given keyword arguments 🌊 It pages through the results using the Marker or NextToken, then fetches out and loops over the list in the response with the given attribute name. So you never miss a single one 💖

Parameters

• requestor_callable (Callable[Ellipsis, Any])
• result_attribute_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• kwargs (Any)

Return type collections.abc.Iterable[Any] (see /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable)

---

✨ Attributes ✨

Here's the lineup 👑

• AWSRegionName
• AWSServerErrors
• logger

✨ Functions ✨

okay bestie, here's the whole AWS zone squad at a glance 💅

• get_current_aws_region() — returns the AWS region that the currently configured AWS zone (see get_current_aws_zone()) is in 💖
• get_aws_zone_from_environment() — grabs the AWS zone from TOIL_AWS_ZONE if it's set 🌸
• get_aws_zone_from_metadata() — gets the zone from instance metadata if you're on EC2 and the boto module is available. Otherwise it checks ECS task metadata, if you're on ECS ✨
• get_aws_zone_from_boto() — gets the zone from the Boto3 config file or from AWS_DEFAULT_REGION, if it's configured and the boto3 module is available 🎀
• get_aws_zone_from_environment_region() — picks an AWS zone in the region defined by TOIL_AWS_REGION, if it's set 🧚
• get_current_aws_zone() — gets the currently configured or occupied AWS zone to use 👑
• zone_to_region(zone) — gets a region (e.g. us-west-2) from a zone (e.g. us-west-1c) 🦋
• running_on_ec2() — returns True if we're running on EC2, and false otherwise 🌈
• running_on_ecs() — returns True if we're running on Amazon ECS, and false otherwise 💜
• build_tag_dict_from_env([environment]) — no summary given for this one, so the details below are your best friend 🙈

---

✨ Package Contents ✨

the goods, all in toil.lib.aws:

• toil.lib.aws.AWSRegionName
• toil.lib.aws.AWSServerErrors
• toil.lib.aws.logger

toil.lib.aws.get_current_aws_region()

Return the AWS region that the currently configured AWS zone (see get_current_aws_zone()) is in. It's giving "find my region" energy 💖

• Return type: Optional[str] (see /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.aws.get_aws_zone_from_environment()

Get the AWS zone from TOIL_AWS_ZONE if set. Simple and iconic 🌸

• Return type: Optional[str] (see /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.aws.get_aws_zone_from_metadata()

Get the AWS zone from instance metadata, if on EC2 and the boto module is available. Otherwise, gets the AWS zone from ECS task metadata, if on ECS. Multi-platform queen behavior ✨

• Return type: Optional[str] (see /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.aws.get_aws_zone_from_boto()

Get the AWS zone from the Boto3 config file or from AWS_DEFAULT_REGION, if it is configured and the boto3 module is available 🎀

• Return type: Optional[str] (see /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.aws.get_aws_zone_from_environment_region()

Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set 🧚

• Return type: Optional[str] (see /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.aws.get_current_aws_zone()

Get the currently configured or occupied AWS zone to use. This is the main character of the whole group 👑

Reports the TOIL_AWS_ZONE environment variable if set.

Otherwise, if we have boto and are running on EC2, or if we are on ECS, reports the zone we are running in.

Otherwise, if we have the TOIL_AWS_REGION variable set, chooses a zone in that region.

Finally, if we have boto2, and a default region is configured in Boto 2, chooses a zone in that region.

Returns 'us-east-1a' if no method can produce a zone to use. Lowkey the fallback era 💀

• Return type: Optional[str] (see /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.aws.zone_to_region(zone)

Get a region (e.g. us-west-2) from a zone (e.g. us-west-1c). Zone in, region out 🦋

• Parameters: zone (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: AWSRegionName

toil.lib.aws.running_on_ec2()

Return True if we are currently running on EC2, and false otherwise. Yes or no, no drama 🌈

• Return type: bool (see /usr/share/doc/python3/html/library/functions.html#bool)

toil.lib.aws.running_on_ecs()

Return True if we are currently running on Amazon ECS, and false otherwise. Same energy, different cloud 💜

• Return type: bool (see /usr/share/doc/python3/html/library/functions.html#bool)

toil.lib.aws.build_tag_dict_from_env(environment=os.environ)

• Parameters: environment (collections.abc.MutableMapping[str, str], see /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.MutableMapping and /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: dict[str, str] (see /usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🦋 toil.lib.bioio 🦋

✨ Functions ✨

• system(command) — a convenience wrapper around subprocess.check_call that logs the command before passing it along 🌸
• getLogLevelString([logger]) — no summary given, just vibes 🙈
• setLoggingFromOptions(options) — no summary given, just vibes 🙈
• getTempFile([suffix, rootDir]) — no summary given, just vibes 🙈

✨ Module Contents ✨

toil.lib.bioio.system(command)

A convenience wrapper around subprocess.check_call that logs the command before passing it on. The command can be either a string or a sequence of strings. If it is a string shell=True will be passed to subprocess.check_call. Iconic wrapper behavior 💖

:type command: str | sequence[string]

• Parameters: command (list[str], see /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: None

toil.lib.bioio.getLogLevelString(logger=None)

toil.lib.bioio.setLoggingFromOptions(options)

toil.lib.bioio.getTempFile(suffix='', rootDir=None)

✨ toil.lib.checksum ✨

okay bestie, this module is all about hashing and checking that your files are exactly what they're supposed to be 🔍💜 let's go 💅

🪞 Attributes 🪞

• logger 🪵 — one of the module's attributes, just vibing
• hashers 🧁 — the other attribute, also just vibing

🚨 Exceptions 🚨

• ChecksumError 🙅‍♀️ — raised when a download does not contain the correct data. Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)

💅 Classes 💅

• Etag 🧚 — a hasher for s3 etags

🛠️ Functions 🛠️

• compute_checksum_for_file(local_file_path[, algorithm]) 📁 — no description given, but it's the one that works on a local file
• compute_checksum_for_content(fh[, algorithm]) 🌊 — Note: chunk size matters for s3 etags, and it must be the same to get the same hash from the same object. Therefore this buffer is not modifiable throughout Toil. Do not mess with this, it's not changeable 🙅‍♀️

📖 Module Contents 📖

toil.lib.checksum.logger

exception toil.lib.checksum.ChecksumError

Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)

Raised when a download does not contain the correct data. Sus download alert 🚨

class toil.lib.checksum.Etag(chunk_size)

A hasher for s3 etags.

Parameters: chunk_size (int, /usr/share/doc/python3/html/library/functions.html#int)

Attributes:

etag_bytes: int = 0
etag_parts: List[bytes] = []
etag_hasher: hashlib._Hash
chunk_size: int

Methods:

• update(chunk) — takes chunk (bytes, /usr/share/doc/python3/html/library/stdtypes.html#bytes) and returns None 🫧
• hexdigest() — returns a str (/usr/share/doc/python3/html/library/stdtypes.html#str) 🔥

toil.lib.checksum.hashers

toil.lib.checksum.compute_checksum_for_file(local_file_path, algorithm='sha1')

Parameters:

• local_file_path (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• algorithm (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.checksum.compute_checksum_for_content(fh, algorithm='sha1')

Note: chunk size matters for s3 etags, and must be the same to get the same hash from the same object. Therefore this buffer is not modifiable throughout Toil. 🔒

Parameters:

• fh (Union[BinaryIO, io.BytesIO], /usr/share/doc/python3/html/library/io.html#io.BytesIO)
• algorithm (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

✨ toil.lib.compatibility ✨

this one is the glue that keeps old and new Python friends 🦋

🧩 Functions 🧩

• deprecated(new_function_name) 🗑️ — no description given
• compat_bytes(s) 🫧 — no description given
• compat_bytes_recursive(data) 🌳 — convert a tree of objects over bytes to objects over strings

📖 Module Contents 📖

toil.lib.compatibility.deprecated(new_function_name)

Parameters: new_function_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Callable[Ellipsis, Any]

toil.lib.compatibility.compat_bytes(s)

Parameters: s (Union[bytes, str], /usr/share/doc/python3/html/library/stdtypes.html#bytes, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.compatibility.compat_bytes_recursive(data)

Convert a tree of objects over bytes to objects over strings. 🌲✨

Parameters: data (Any)

Return type: Any

---

✨ toil.lib.conversions ✨

Conversion utilities for mapping memory, disk, and core declarations from strings to numbers and vice versa 🧮. There are also general conversion functions in here too, so it's a whole vibe 💖

🎀 Attributes 🎀

• KIB 🎀
• MIB 🎀
• GIB 🎀
• TIB 🎀
• PIB 🎀
• EIB 🎀
• KB 🌸
• MB 🌸
• GB 🌸
• TB 🌸
• PB 🌸
• EB 🌸
• BINARY_PREFIXES 🦋 — the binary prefix set
• DECIMAL_PREFIXES 🍓 — the decimal prefix set
• VALID_PREFIXES 👑 — the valid prefixes

💖 Functions 💖

okay bestie, here's the whole function lineup 🎀 the table cuts off on a few of these, so I'm keeping them exactly as they're written 😭

• bytes_in_unit([unit]) — the table left this one's description blank 🙈
• convert_units(num, src_unit[, dst_unit]) — returns a float with the converted input in dst_units ✨
• parse_memory_string(string) — takes a string like '1024 Mib' and returns the 🫶
• human2bytes(string) — same vibe as above, takes a string like '1024 Mib' and returns the 💅
• bytes2human(n) — turns a binary value into a human readable string with units 🌸
• b_to_mib(n) — converts a number from bytes to mibibytes 🦋
• mib_to_b(n) — converts a number from mibibytes to bytes 🦋
• hms_duration_to_seconds(hms) — parses a time string in hours:minutes:seconds ⏰
• strtobool(val) — makes a human-readable string into a bool 🔮
• opt_strtobool(b) — converts an optional string representation of a bool to None or a bool 💜
• modify_url(url, remove) — given a valid URL string, it splits out the params and removes any offending ones 🌈

---

🌈 Module Contents 🌈

🎀 Size constants

• toil.lib.conversions.KIB = 1024
• toil.lib.conversions.MIB = 1048576
• toil.lib.conversions.GIB = 1073741824
• toil.lib.conversions.TIB = 1099511627776
• toil.lib.conversions.PIB = 1125899906842624
• toil.lib.conversions.EIB = 1152921504606846976
• toil.lib.conversions.KB = 1000
• toil.lib.conversions.MB = 1000000
• toil.lib.conversions.GB = 1000000000
• toil.lib.conversions.TB = 1000000000000
• toil.lib.conversions.PB = 1000000000000000
• toil.lib.conversions.EB = 1000000000000000000

🧚 Prefix lists

• toil.lib.conversions.BINARY_PREFIXES = ['ki', 'mi', 'gi', 'ti', 'pi', 'ei', 'kib', 'mib', 'gib', 'tib', 'pib', 'eib']
• toil.lib.conversions.DECIMAL_PREFIXES = ['b', 'k', 'm', 'g', 't', 'p', 'e', 'kb', 'mb', 'gb', 'tb', 'pb', 'eb']
• toil.lib.conversions.VALID_PREFIXES = ['ki', 'mi', 'gi', 'ti', 'pi', 'ei', 'kib', 'mib', 'gib', 'tib', 'pib', 'eib', 'b', 'k', 'm',...]

🦋 bytes_in_unit

toil.lib.conversions.bytes_in_unit(unit='B')

Parameters

• unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

int </usr/share/doc/python3/html/library/functions.html#int>

✨ convert_units

toil.lib.conversions.convert_units(num, src_unit, dst_unit='B')

Returns a float representing the converted input in dst_units. 💖

Parameters

• num (float </usr/share/doc/python3/html/library/functions.html#float>)
• src_unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• dst_unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

float </usr/share/doc/python3/html/library/functions.html#float>

🫶 parse_memory_string

toil.lib.conversions.parse_memory_string(string)

Given a string representation of some memory (i.e. '1024 Mib'), return the number and unit. 💅

Parameters

• string (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[float </usr/share/doc/python3/html/library/functions.html#float>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

💅 human2bytes

toil.lib.conversions.human2bytes(string)

Given a string representation of some memory (i.e. '1024 Mib'), return the integer number of bytes. 🌸

Parameters

• string (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

int </usr/share/doc/python3/html/library/functions.html#int>

🌸 bytes2human

toil.lib.conversions.bytes2human(n)

Return a binary value as a human readable string with units. 🍓

Parameters

• n (SupportsInt)

Return type

str </usr/share/doc/python3/html/library/stdtypes.html#str>

🦋 b_to_mib

toil.lib.conversions.b_to_mib(n)

Convert a number from bytes to mibibytes. 🦋

Parameters

• n (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

Return type

float </usr/share/doc/python3/html/library/functions.html#float>

🦋 mib_to_b

toil.lib.conversions.mib_to_b(n)

Convert a number from mibibytes to bytes. 🦋

Parameters

• n (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

Return type

float </usr/share/doc/python3/html/library/functions.html#float>

⏰ hms_duration_to_seconds

toil.lib.conversions.hms_duration_to_seconds(hms)

Parses a given time string in hours:minutes:seconds, returns an equivalent total seconds value. ⏰

Parameters

• hms (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

float </usr/share/doc/python3/html/library/functions.html#float>

🔮 strtobool

toil.lib.conversions.strtobool(val)

Make a human-readable string into a bool. 🔮

Convert a string along the lines of "y", "1", "ON", "TrUe", or "Yes" to True, and the corresponding false-ish values to False. 🙈

Parameters

• val (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

bool </usr/share/doc/python3/html/library/functions.html#bool>

💜 opt_strtobool

toil.lib.conversions.opt_strtobool(b)

Convert an optional string representation of bool to None or bool. 💜

Parameters

• b (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

Return type

Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]

🌈 modify_url

toil.lib.conversions.modify_url(url, remove)

Given a valid URL string, split out the params, remove any offending params in 'remove', and return the cleaned URL. 🌈

Parameters

• url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• remove (List[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

Return type

str </usr/share/doc/python3/html/library/stdtypes.html#str>

---

🦋 toil.lib.directory

👑 Attributes

• TOIL_DIR_URI_SCHEME — listed with no value 🫶
• DirectoryContents — listed with no value 🫶

✨ Functions ✨

okay bestie, here's the whole lineup for the directory helpers 🎀 they all live in toil.lib.directory and they're basically all about toildir: paths 🦋

• check_directory_dict_invariants(contents) — makes sure a directory structure dict actually makes sense 💅 it throws an error if it doesn't
• decode_directory(dir_path) — decodes a directory from a "toildir:" path into a directory (or a file inside it) 🌸
• encode_directory(contents[, name, source]) — encodes a directory from a "toildir:" path into a directory (or a file inside it) ✨
• directory_item_exists(dir_path) — checks that a URL to a Toil directory, or something inside it, is real 🫶
• get_directory_contents_item(contents, remaining_path) — grabs a subdirectory or file from a decoded directory plus the leftover path 💜
• get_directory_item(dir_path) — grabs a subdirectory or file from a URL that points to, or into, a toildir: directory 🧚
• directory_contents_items(contents) — yields every file or directory under the given contents, including itself 🍓
• directory_items(dir_path) — yields every file or directory under the given path, including itself 🌈

---

✨ Module Contents ✨

bestie, quick heads up before we get into the details 🙈 the type links in this part go to the Python docs at /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#tuple, and /usr/share/doc/python3/html/library/functions.html#bool.

toil.lib.directory.TOIL_DIR_URI_SCHEME = 'toildir:'

this is the constant that holds the toildir: URI scheme 💖

toil.lib.directory.DirectoryContents

this one's just listed as a name in the module, no extra details given ✨

toil.lib.directory.check_directory_dict_invariants(contents)

okay so this one is the bouncer 💅 it makes sure a directory structure dict makes sense, and if it doesn't, it throws an error.

Right now it only checks one thing: that no empty-string keys exist 🥺

Parameters

• contents (DirectoryContents)

Return type

• None

toil.lib.directory.decode_directory(dir_path)

this is giving decoder energy 🔥 it takes a "toildir:" path and turns it into a directory, or into a file sitting inside that directory.

Returns

• the decoded directory dict
• the remaining part of the path, which may be None
• an identifier string for the directory, which is the stored name URI if one was provided
• the name URI and source task info

Parameters

• dir_path (str)

Return type

• tuple[DirectoryContents, Optional[str], str, Optional[str], Optional[str]]

toil.lib.directory.encode_directory(contents, name=None, source=None)

encode is giving main character energy 👑 it takes a directory from a "toildir:" path into a directory, or a file in it.

Parameters

• contents (DirectoryContents) — the directory dict, which maps a name to a URI for a file, or to another dict for a subdirectory
• name (Optional[str]) — the path or URI the directory belongs at, including its basename. It can't be empty if it's set 🌸
• source (Optional[str]) — the name of the workflow component that uploaded the directory. It can't be empty if it's set 🦋

Return type

• str

toil.lib.directory.directory_item_exists(dir_path)

this one is a vibe check ✅ it checks that a URL to a Toil directory, or a thing inside it, actually exists.

It assumes all the pointed-to URLs exist already. It only checks that the thing is in the encoded directory structure 🫶

Parameters

• dir_path (str)

Return type

• bool

toil.lib.directory.get_directory_contents_item(contents, remaining_path)

this one grabs a subdirectory or file from a decoded directory, using the remaining path 💜

Parameters

• contents (DirectoryContents)
• remaining_path (Optional[str])

Return type

• Union[DirectoryContents, str]

toil.lib.directory.get_directory_item(dir_path)

okay this is the one that takes a URL pointing to, or into, a toildir: directory and pulls out a subdirectory or file 🧚

Parameters

• dir_path (str)

Return type

• Union[DirectoryContents, str]

toil.lib.directory.directory_contents_items(contents)

this is a generator that yields each file or directory under the given contents, including itself 🍓 it's iconic for listing everything.

Parents always come out before their children 🌈 each item is yielded as a str path from the root (which can be empty), paired with either a str value for files or a None for directories.

The path never has trailing slashes ✨

Parameters

• contents (DirectoryContents)

Return type

• Iterator[tuple[str, Union[str, None]]]

toil.lib.directory.directory_items(dir_path)

same energy as the one above, but this one starts from a path instead of contents 🎀 it yields each file or directory under the given path, including itself.

Parents still come before children 🌸 each item is a str path from the root (possibly empty), paired with a str value for files or a None for directories.

The path won't have trailing slashes 💖

Parameters

• dir_path (str)

Return type

• Iterator[tuple[str, Union[str, None]]]

---

toil.lib.docker

✨ Attributes ✨

bestie, this module has four attributes 🐳 the docs don't describe any of them, they're just the names.

• logger
• FORGO
• STOP
• RM

✨ Functions ✨

okay so these are the docker helpers toil hands us 💅 the table is a little wonky in the terminal, so here's the full tea:

• dockerCheckOutput(*args, **kwargs): no blurb on this one bestie, it just vibes there 🫠
• dockerCall(*args, **kwargs): also no blurb, same energy 🌸
• subprocessDockerCall(*args, **kwargs): again, nothing written, but it's giving main character 🦋
• apiDockerCall(…): a toil wrapper for the python docker API ✨
• dockerKill(container_name[, gentleKill, remove, timeout]): immediately kills a container 💀 it's equivalent to "docker kill":
• dockerStop(container_name[, remove]): gracefully kills a container 🫶 it's equivalent to "docker stop":
• containerIsRunning(container_name[, timeout]): checks whether the container is running or not 👀
• getContainerName(job): creates a random string including the job name, and returns it. Name will 🙈 (the page cuts off right there, iykyk)

---

✨ Module Contents ✨

bestie, the module is literally full of goodies 🎀

toil.lib.docker.logger is hanging out in there 📝

Here are the constants, and they're iconic:

• toil.lib.docker.FORGO = 0
• toil.lib.docker.STOP = 1
• toil.lib.docker.RM = 2

toil.lib.docker.dockerCheckOutput(*args, **kwargs)

• Parameters
  • args (Any)
  • kwargs (Any)
• Return type: NoReturn

toil.lib.docker.dockerCall(*args, **kwargs)

• Parameters
  • args (Any)
  • kwargs (Any)
• Return type: NoReturn

toil.lib.docker.subprocessDockerCall(*args, **kwargs)

• Parameters
  • args (Any)
  • kwargs (Any)
• Return type: NoReturn

toil.lib.docker.apiDockerCall(...)

ngl this one is a whole mood 😭 the full signature is below, and every single default is included so you can see exactly what you're working with:

toil.lib.docker.apiDockerCall(job: toil.job.Job <#toil.job.Job>, image: str </usr/share/doc/python3/html/library/stdtypes.html#str>, parameters: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, deferParam: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None, volumes: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, working_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, containerName: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, entrypoint: str </usr/share/doc/python3/html/library/stdtypes.html#str> | list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, detach: Literal[False] = False, log_config: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, auto_remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = False, user: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, environment: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, stdout: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, stderr: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, stream: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, demux: Literal[False] = False, streamfile: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, timeout: int </usr/share/doc/python3/html/library/functions.html#int> = 365 * 24 * 60 * 60, **kwargs: Any) -> str </usr/share/doc/python3/html/library/stdtypes.html#str>

Okay bestie, apiDockerCall comes in three looks 💅 The return type changes depending on how you set stream and detach ✨

toil.lib.docker.apiDockerCall(job: toil.job.Job, image: str, parameters: list[str] | list[list[str]] | None = None, deferParam: int | None = None, volumes: dict[str, dict[str, str]] | None = None, working_dir: str | None = None, containerName: str | None = None, entrypoint: str | list[str] | None = None, detach: Literal[False] = False, log_config: dict[str, str] | None = None, auto_remove: bool | None = None, remove: bool | None = False, user: str | None = None, environment: dict[str, str] | None = None, stdout: bool | None = None, stderr: bool = False, stream: Literal[True] = True, demux: Literal[True] = True, streamfile: str | None = None, accelerators: list[int] | None = None, timeout: int = 365 * 24 * 60 * 60, **kwargs: Any) -> collections.abc.Generator[tuple[str, str]]

Here's the first one 🎀 With stream and demux both set to True, it hands back a generator that yields tuple[str, str] pairs 🌸

toil.lib.docker.apiDockerCall(job: toil.job.Job, image: str, parameters: list[str] | list[list[str]] | None = None, deferParam: int | None = None, volumes: dict[str, dict[str, str]] | None = None, working_dir: str | None = None, containerName: str | None = None, entrypoint: str | list[str] | None = None, detach: Literal[False] = False, log_config: dict[str, str] | None = None, auto_remove: bool | None = None, remove: bool | None = False, user: str | None = None, environment: dict[str, str] | None = None, stdout: bool | None = None, stderr: bool = False, stream: Literal[False] = False, demux: Literal[True] = True, streamfile: str | None = None, accelerators: list[int] | None = None, timeout: int = 365 * 24 * 60 * 60, **kwargs: Any) -> tuple[str, str]

Look at this one, it's the non-streaming version 🦋 Same detach of False, but stream is Literal[False], so you get a single tuple[str, str] back instead of a generator 🫶

toil.lib.docker.apiDockerCall(job: toil.job.Job, image: str, parameters: list[str] | list[list[str]] | None = None, deferParam: int | None = None, volumes: dict[str, dict[str, str]] | None = None, working_dir: str | None = None, containerName: str | None = None, entrypoint: str | list[str] | None = None, detach: Literal[True] = True, log_config: dict[str, str] | None = None, auto_remove: bool | None = None, remove: bool | None = False, user: str | None = None, environment: dict[str, str] | None = None, stdout: bool | None = None, stderr: bool = False, stream: bool = False, demux: bool = False, streamfile: str | None = None, accelerators: list[int] | None = None, timeout: int = 365 * 24 * 60 * 60, **kwargs: Any) -> docker.models.containers.Container

And the final main-character version 👑 With detach set to True, it returns a docker.models.containers.Container object, and stream and demux both default to False here 🌈

toil.lib.docker.apiDockerCall(job: toil.job.Job, image: str, parameters: list[str] | list[list[str]] | None = None, deferParam: int | None = None, volumes: dict[str, dict[str, str]] | None = None, working_dir: str | None = None, containerName: str | None = None, entrypoint: str | list[str] | None = None, detach: bool = False, log_config: dict[str, str] | None = None, auto_remove: bool | None = None, remove: bool | None = False, user: str | None = None, environment: dict[str, str] | None = None, stdout: bool | None = None, stderr: bool = False, stream: bool = False, demux: bool = False, streamfile: str | None = None, accelerators: list[int] | None = None, timeout: int = 365 * 24 * 60 * 60, **kwargs: Any) -> str | docker.models.containers.Container | collections.abc.Generator[tuple[str, str]] | collections.abc.Generator[str] | tuple[str, str]

okay bestie, this one is the main character 💅✨ apiDockerCall is toil's own wrapper for the python docker API 🐳💖. It's docker's python API running inside toil, so every call gets run as a job 🎀. The whole point is that if a docker job fails or gets orphaned, toil can handle it properly 🫶.

• Docker API Docs — https://docker-py.readthedocs.io/en/stable/index.html
• Docker API Code — https://github.com/docker/docker-py

Example 🧚

Here's how you'd use dockerCall in toil to index a FASTA file with SAMtools 🌸:

def toil_job(job):
    working_dir = job.fileStore.getLocalTempDir()
    path = job.fileStore.readGlobalFile(ref_id,
                                      os.path.join(working_dir, 'ref.fasta')
    parameters = ['faidx', path]
    apiDockerCall(job,
                  image='quay.io/ucgc_cgl/samtools:latest',
                  working_dir=working_dir,
                  parameters=parameters)

⚠️ Heads up: when this runs with detach=False, or with detach=True and stdout=True or stderr=True, it's a blocking call. With detach=True and no output capture, the container gets started and returned without waiting for it to finish.

Parameters 💖

• job — the Job instance for the function calling this one 💁
• image — name of the Docker image to use, like 'quay.io/ucsc_cgl/samtools:latest'
• parameters — a list of string elements. If there are multiple elements, they get joined with spaces. This keeps things backwards compatible with older versions that called docker with subprocess.check_call(). If you pass a list of lists (list[list[str]]), each inner list is treated as a successive command, chained with a pipe 🔗
• deferParam — what happens to the container when the job wraps up 🏁:
  • FORGO (0) leaves the container untouched and running
  • STOP (1) sends SIGTERM, then SIGKILL if necessary, to the container
  • RM (2) immediately sends SIGKILL to the container
  • If deferParam is None, RM is the default behavior
• volumes — a dictionary of volume locations to volume options 📦
• working_dir — the working directory 📂
• containerName — the name or ID of the container 🏷️
• entrypoint — prepends commands sent to the container. See https://docker-py.readthedocs.io/en/stable/containers.html
• detach — runs the container in detached mode (equivalent to -d) 🦋
• log_config — specifies which logs to return from the container. See https://docker-py.readthedocs.io/en/stable/containers.html
• remove — removes the container on exit, or doesn't 🗑️
• user — the privileges the container runs with. This can be an actual name, like 'root' or 'lifeisaboutfishtacos', or the uid or gid ('0' is root, '1000' is an example of a less privileged uid or gid), or a complement of the uid:gid (RECOMMENDED), like '0:0' (root user : root group) or '1000:1000' (some other user : some other user group) 👑
• environment — lets you set environment variables inside the container 🌈
• stdout — returns logs from STDOUT when detach=False (default: True). When detach=True, it blocks and captures stdout to a file (default: False). Output capture defaults to output.log, and you can change that with the "streamfile" kwarg
• stderr — returns logs from STDERR when detach=False (default: False). When detach=True, it blocks and captures stderr to a file (default: False). Output capture defaults to output.log, and you can change that with the "streamfile" kwarg
• stream — if True and detach=False, you get a log generator instead of a string. Ignored if detach=True (default: False)
• demux — similar to demux in container.exec_run(). If True and detach=False, you get back a tuple of (stdout, stderr). If stream=True, you get a log generator with tuples of (stdout, stderr). Ignored if detach=True (default: False)
• streamfile — where container output gets collected if detach=True and stderr and/or stdout are True. Defaults to "output.log" 📝
• accelerators — Toil accelerator numbers (usually GPUs) to forward to the container 🔥. These are interpreted in the current Python process's environment. Check toil.lib.accelerators.get_individual_local_accelerators() for the menu of available accelerators
• timeout — the timeout in seconds for interactions with the Docker daemon ⏳. Defaults to 1 year (i.e. wait essentially indefinitely)

🚨 Caveat: the underlying docker module isn't always able to abort ongoing reads and writes in order to respect the timeout.

• kwargs — extra keyword arguments supplied to the docker API's run command. The list is 75 keywords total. For examples and full documentation, see https://docker-py.readthedocs.io/en/stable/containers.html 🧾

Returns 🎀

When detach=False, you get the standard output/standard error text back, as requested. When detach=True, you get the underlying docker.models.containers.Container object from the Docker API.

---

toil.lib.docker.dockerKill 💀

toil.lib.docker.dockerKill(container_name, gentleKill=False, remove=False, timeout=365 * 24 * 60 * 60)

This one is giving "immediate and no crying about it" 😭. It kills a container right away, and it's equivalent to docker kill. Check out https://docs.docker.com/engine/reference/commandline/kill/ if you want the deets 🦋.

Parameters

• container_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the container being killed 💀
• gentleKill (bool, /usr/share/doc/python3/html/library/functions.html#bool) — if this is True, it triggers a graceful shutdown ✨ the polite way to say goodbye 🫶
• remove (bool, /usr/share/doc/python3/html/library/functions.html#bool) — if True, the container gets removed after it exits 🧹
• timeout (int, /usr/share/doc/python3/html/library/functions.html#int) — the number of seconds to wait for interactions with the Docker daemon. Heads up, bestie: the underlying docker module isn't always able to abort ongoing reads and writes to actually respect the timeout 🥺 It defaults to 1 year (i.e. wait essentially indefinitely) ⏳

Return type — None 💖

toil.lib.docker.dockerStop(container_name, remove=False)

Gracefully kills a container. It's equivalent to docker stop, see https://docs.docker.com/engine/reference/commandline/stop/ 🛑

Parameters

• container_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the container being stopped 🫧
• remove (bool, /usr/share/doc/python3/html/library/functions.html#bool) — if True, the container gets removed after it exits 🧹

Return type — None 💖

toil.lib.docker.containerIsRunning(container_name, timeout=365 * 24 * 60 * 60)

Checks whether the container is running or not. Main character check 🔍

Parameters

• container_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the container being checked 👀
• timeout (int, /usr/share/doc/python3/html/library/functions.html#int) — the number of seconds to wait for interactions with the Docker daemon. Same tea as before: the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. It defaults to 1 year (i.e. wait essentially indefinitely) ⏳

Returns

True if the status is 'running', False if the status is anything else, and None if the container does not exist 🫥

Return type — Optional[bool] (/usr/share/doc/python3/html/library/functions.html#bool) 💜

toil.lib.docker.getContainerName(job)

Creates a random string that includes the job name, then returns it. The name will match [a-zA-Z0-9][a-zA-Z0-9_.-] 🎀

Parameters

• job (toil.job.Job, #toil.job.Job) 🧚

Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str) 💖

---

✨ toil.lib.dockstore ✨

Okay so this module is all about integrating Toil with UCSC Dockstore, for reporting metrics 📊 It's giving data girl energy 💅

For basic TRS functionality for fetching workflows, see trs.py 🦋

✨ Attributes ✨

• logger
• DEFAULT_DOCKSTORE_TOKEN
• DOCKSTORE_TOKEN
• DOCKSTORE_PLATFORM
• ExecutionStatus

✨ Classes ✨

• Cost — representation of the cost of running something 💸
• RunExecution — Dockstore metrics data for a workflow or task run 🏃‍♀️
• TaskExecutions — Dockstore metrics data for all the tasks in a workflow 🌸

✨ Functions ✨

• ensure_valid_id(execution_id) — make sure the given execution ID is in Dockstore format and will be accepted by Dockstore 🪪
• pack_workflow_metrics(execution_id, start_time, ...[, ...]) — pack up per-workflow metrics into a format that can be submitted to Dockstore 📦
• pack_single_task_metrics(execution_id, start_time, ...) — pack up metrics for a single task execution in a format that can be used in a Dockstore submission 🎁
• pack_workflow_task_set_metrics(execution_id, ...) — pack up metrics for all the tasks in a workflow execution into a format that can be submitted to Dockstore 🧃
• send_metrics(trs_workflow_id, trs_version, ...) — send packed workflow and/or task metrics to Dockstore 📨
• get_metrics_url(trs_workflow_id, trs_version, execution_id) — get the URL where a workflow metrics object (for a workflow, or for a set of tasks) can be fetched back from 🔗

✨ Module Contents ✨

• toil.lib.dockstore.logger
• toil.lib.dockstore.DEFAULT_DOCKSTORE_TOKEN = '2bff46294daddef6df185452b04db6143ea8a59f52ee3c325d3e1df418511b7d'
• toil.lib.dockstore.DOCKSTORE_TOKEN
• toil.lib.dockstore.DOCKSTORE_PLATFORM = 'TOIL'
• toil.lib.dockstore.ExecutionStatus

✨ class toil.lib.dockstore.Cost ✨

Bases: TypedDict 🌈

Representation of the cost of running something.

• value (float, /usr/share/doc/python3/html/library/functions.html#float) — the cost in US Dollars 💵

✨ class toil.lib.dockstore.RunExecution ✨

Bases: TypedDict 🌈

Dockstore metrics data for a workflow or task run. Okay, this is the big one, so pay attention 👀

• executionId (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the executor-generated unique execution ID for this workflow or task 🆔
• dateExecuted (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — an ISO 8601 UTC timestamp for when the execution happened ⏰
• executionStatus (ExecutionStatus) — did the execution work? That's the whole question 🤞
• executionTime (typing_extensions.NotRequired[str], /usr/share/doc/python3/html/library/stdtypes.html#str) — the total time of the run in ISO 8601 duration format ⏱️
• memoryRequirementsGB (typing_extensions.NotRequired[float], /usr/share/doc/python3/html/library/functions.html#float) — the memory required for the execution in gigabytes (not GiB, so don't mix them up!) 🧠
• cpuRequirements (typing_extensions.NotRequired[int], /usr/share/doc/python3/html/library/functions.html#int) — the number of CPUs required 🖥️
• cost (typing_extensions.NotRequired[Cost], #toil.lib.dockstore.Cost) — how much the execution cost to run 💰
• region (typing_extensions.NotRequired[str], /usr/share/doc/python3/html/library/stdtypes.html#str) — the (cloud) region the workflow was executed in ☁️

additionalProperties: typing_extensions.NotRequired[dict[str, Any]]

Got extra stuff to send? This is where it goes 🎁 Any additional properties you want to ship off to Dockstore. Dockstore is lowkey so accommodating that it takes any JSON-able structured data 💅

Type docs: dict at /usr/share/doc/python3/html/library/stdtypes.html#dict, and str at /usr/share/doc/python3/html/library/stdtypes.html#str.

class toil.lib.dockstore.TaskExecutions

Bases: TypedDict

Dockstore metrics for every task in a workflow, all bundled up in one place 🧚

executionId: str

The unique ID the executor generated for this run 🆔 Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.

dateExecuted: str

When it happened, written as an ISO 8601 UTC timestamp ⏰ Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.

taskExecutions: list[RunExecution]

Each individual task execution in the workflow, one by one 🌸 Type docs: /usr/share/doc/python3/html/library/stdtypes.html#list for list, and RunExecution is defined as toil.lib.dockstore.RunExecution.

additionalProperties: typing_extensions.NotRequired[dict[str, Any]]

Any additional properties to send, again, because it's giving extra 🎀 Dockstore takes any JSON-able structured data. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/stdtypes.html#str.

toil.lib.dockstore.ensure_valid_id(execution_id)

toil.lib.dockstore.ensure_valid_id(execution_id)

Make sure the execution ID is in Dockstore format so Dockstore doesn't side-eye it 👀 It has to be accepted, or else 💀

The ID must be alphanumeric, though internal underscores are allowed 🫶 It also has to be <100 characters long, so keep it short and sweet.

Raises ValueError if the ID isn't in the right format.

Parameters

• execution_id (str) — the execution ID you want to check. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.

Return type: None

toil.lib.dockstore.pack_workflow_metrics(...)

toil.lib.dockstore.pack_workflow_metrics(execution_id, start_time, runtime, succeeded, job_store_type=None, batch_system=None, caching=None, toil_version=None, python_version=None, platform_system=None, platform_machine=None)

Okay, this one packs up per-workflow metrics into a format Dockstore will actually accept 🔥

Parameters

• execution_id (str) — unique ID for the workflow execution. Has to be in Dockstore format. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• start_time (float) — when the execution started, in seconds since the Unix epoch. Type docs: /usr/share/doc/python3/html/library/functions.html#float.
• rutime — how long the execution ran, in seconds.
• jobstore_type — the kind of job store used, like "file" or "aws".
• batch_system (Optional[str]) — the Python class name of the batch system that got used. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• caching (Optional[bool]) — whether Toil filestore-level caching was on. Type docs: /usr/share/doc/python3/html/library/functions.html#bool.
• toil_version (Optional[str]) — the Toil version, without any Git hash 🦋 Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• python_version (Optional[str]) — the Python version used. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• platform_system (Optional[str]) — the operating system type, like "Darwin" or "Linux". Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• platform_machine (Optional[str]) — the machine architecture of the leader, like "AMD64". Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• runtime (float) — type docs: /usr/share/doc/python3/html/library/functions.html#float.
• succeeded (bool) — type docs: /usr/share/doc/python3/html/library/functions.html#bool.
• job_store_type (Optional[str]) — type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.

Return type: RunExecution, which is toil.lib.dockstore.RunExecution.

toil.lib.dockstore.pack_single_task_metrics(...)

toil.lib.dockstore.pack_single_task_metrics(execution_id, start_time, runtime, succeeded, job_name=None, cores=None, cpu_seconds=None, memory_bytes=None, disk_bytes=None)

This one packs up metrics for a single task execution so it's ready for a Dockstore submission 💖

Parameters

• execution_id (str) — unique ID for the workflow execution. Has to be in Dockstore format. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• start_time (float) — when the execution started, in seconds since the Unix epoch. Type docs: /usr/share/doc/python3/html/library/functions.html#float.
• rutime — how long the execution ran, in seconds.
• succeeded (bool) — whether the execution succeeded. Type docs: /usr/share/doc/python3/html/library/functions.html#bool.
• job_name (Optional[str]) — the name of the job within the workflow. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• cores (Optional[float]) — CPU cores allocated to the job. Type docs: /usr/share/doc/python3/html/library/functions.html#float.
• cpu_seconds (Optional[float]) — CPU seconds the job consumed. Type docs: /usr/share/doc/python3/html/library/functions.html#float.
• memory_bytes (Optional[int]) — memory the job consumed, in bytes. Type docs: /usr/share/doc/python3/html/library/functions.html#int.
• disk_bytes (Optional[int]) — disk space the job consumed, in bytes. Type docs: /usr/share/doc/python3/html/library/functions.html#int.
• runtime (float) — type docs: /usr/share/doc/python3/html/library/functions.html#float.

Return type: RunExecution, which is toil.lib.dockstore.RunExecution.

toil.lib.dockstore.pack_workflow_task_set_metrics(...)

toil.lib.dockstore.pack_workflow_task_set_metrics(execution_id, start_time, tasks)

Packs up metrics for every task in a workflow execution into something Dockstore can take 🌈

Parameters

• execution_id (str) — unique ID for the workflow execution. Has to be in Dockstore format. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• start_time (float) — when the overall workflow execution started, in seconds since the Unix epoch. Type docs: /usr/share/doc/python3/html/library/functions.html#float.
• tasks (list[RunExecution]) — packed tasks from pack_single_task_metrics(). Type docs: /usr/share/doc/python3/html/library/stdtypes.html#list and toil.lib.dockstore.RunExecution.

Return type: TaskExecutions, which is toil.lib.dockstore.TaskExecutions.

toil.lib.dockstore.send_metrics(...)

toil.lib.dockstore.send_metrics(trs_workflow_id, trs_version, workflow_runs, workflow_task_sets)

Sends your packed workflow and/or task metrics off to Dockstore 🚀

Parameters

• workflow_runs (list[RunExecution]) — a list of packed metrics objects, one for each workflow. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#list and toil.lib.dockstore.RunExecution.
• workflow_task_sets (list[TaskExecutions]) — a list of packed metrics objects for the tasks in each workflow. Each workflow should have one entry with all of its tasks. It doesn't have to match the same order or set of workflows as workflow_runs. Type docs: /usr/share/doc/python3/html/library/stdtypes.html#list and toil.lib.dockstore.TaskExecutions.
• trs_workflow_id (str) — type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• trs_version (str) — type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.

Heads up: if Dockstore doesn't accept the metrics, this raises requests.HTTPError 😭

Return type: None

toil.lib.dockstore.get_metrics_url(...)

toil.lib.dockstore.get_metrics_url(trs_workflow_id, trs_version, execution_id)

Gets the URL where a workflow metrics object can be fetched back from, whether it's for a whole workflow or a set of tasks 🔗

Parameters

• trs_workflow_id (str) — type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• trs_version (str) — type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.
• execution_id (str) — type docs: /usr/share/doc/python3/html/library/stdtypes.html#str.

Return type: str, with type docs at /usr/share/doc/python3/html/library/stdtypes.html#str.

toil.lib.ec2

Attributes

The attributes are below. The table doesn't give descriptions for any of them, so I'm not going to guess 🙈

• a_short_time
• a_long_time
• logger
• INCONSISTENCY_ERRORS
• iam_client

✨ Exceptions ✨

okay bestie, these are the error bffs of this module 🫶 they're giving "something went wrong and we're gonna tell you about it" 💀

• UserError — Unspecified run-time error. 🙈
• UnexpectedResourceState — the common base class for all non-exit exceptions 💅

---

✨ Functions ✨

here's the full guest list 🎀 some of these descriptions get cut off mid-sentence in the docs (😭 not my fault), so I'm giving you exactly what's there and nothing more

• is_base64(value) — Return True if value is base64-decodeable, and False otherwise. ✨
• not_found(e) — no blurb for this one, it's giving mystery 🔮
• inconsistencies_detected(e) — no blurb either, very mysterious 🧚
• retry_ec2([t, retry_for, retry_while]) — no blurb 🙈
• wait_transition(boto3_ec2, resource, from_states, to_state) — waits until the specified EC2 resource (instance, image, volume, ...) transitions from any (the description stops right there 😭)
• wait_instances_running(boto3_ec2, instances) — waits until no instance in the given iterable is 'pending'. Yields every instance that (also cut off, babe 💀)
• wait_spot_requests_active(boto3_ec2, requests[, ...]) — waits until no spot request in the given iterator is in the 'open' state or, optionally, (cut off again 🥲)
• create_spot_instances(boto3_ec2, price, image_id, spec) — creates instances on the spot market. 🌸
• create_ondemand_instances(boto3_ec2, image_id, spec[, ...]) — requests the RunInstances EC2 API call but accounts for the race between recently created (cut off, the drama continues 🎭)
• increase_instance_hop_limit(boto3_ec2, boto_instance_list) — increases the default HTTP hop limit, as we are running Toil and Kubernetes inside a Docker container, so the default (and stops there 😭)
• prune(bushy) — prunes entries in the given dict with false-y values. 🍓
• wait_until_instance_profile_arn_exists(...) — no blurb 👀
• create_instances(ec2_resource, image_id, key_name, ...) — replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts. 👑
• create_launch_template(ec2_client, template_name, ...) — creates a launch template with the given name for launching instances with the given parameters. 🚀
• create_auto_scaling_group(autoscaling_client, ...[, ...]) — creates a new Auto Scaling Group with the given name (which is also its (cut off, babe 💀)

---

✨ Module Contents ✨

okay the module contents are where the receipts are 🧾 let's go

Constants and names first:

toil.lib.ec2.a_short_time = 5

toil.lib.ec2.a_long_time = 3600

toil.lib.ec2.logger

Now is_base64. It's giving return True or False vibes 🦋

toil.lib.ec2.is_base64(value)

Return True if value is base64-decodeable, and False otherwise.

• Parameters: value (str)
• Return type: bool

Next up is UserError, which is a RuntimeError underneath 🫶

exception toil.lib.ec2.UserError(message=None, cause=None)

Bases: RuntimeError. Unspecified run-time error. 🙈

A few more names:

toil.lib.ec2.not_found(e)

toil.lib.ec2.inconsistencies_detected(e)

toil.lib.ec2.INCONSISTENCY_ERRORS

toil.lib.ec2.retry_ec2(t=a_short_time, retry_for=10 * a_short_time, retry_while=not_found)

Then UnexpectedResourceState, the base class that's basically everyone's mom 💅

exception toil.lib.ec2.UnexpectedResourceState(resource, to_state, state)

Bases: Exception. Common base class for all non-exit exceptions. ✨

Now wait_transition, which is a whole vibe 🌈

toil.lib.ec2.wait_transition(boto3_ec2, resource, from_states, to_state, state_getter=lambda x: ...)

Waits until the specified EC2 resource (instance, image, volume, ...) transitions from any of the given 'from' states to the specified 'to' state. If the instance is found in a state other that the to state or any of the from states, an exception will be thrown. 😬

• Parameters:
  • resource (mypy_boto3_ec2.type_defs.InstanceTypeDef) -- the resource to monitor
  • from_states (collections.abc.Iterable[str]) -- a set of states that the resource is expected to be in before the transition occurs
  • to_state (str) -- the state of the resource when this method returns
  • boto3_ec2 (mypy_boto3_ec2.client.EC2Client)
  • state_getter (Callable[[mypy_boto3_ec2.type_defs.InstanceTypeDef], str])

And last but not least, wait_instances_running 💖

toil.lib.ec2.wait_instances_running(boto3_ec2, instances)

Waits until no instance in the given iterable is 'pending'. Yields every instance that entered the running state as soon as it does. 🦋

• Parameters:
  • boto3_ec2 (mypy_boto3_ec2.client.EC2Client) -- the EC2 connection to use for making requests
• instances (collections.abc.Iterable[mypy_boto3_ec2.type_defs.InstanceTypeDef], docs: /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable) — the instances we're waiting on 🫶

Return type: collections.abc.Generator[mypy_boto3_ec2.type_defs.InstanceTypeDef, None, None] (docs: /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator) ✨

🦋 toil.lib.ec2.wait_spot_requests_active

toil.lib.ec2.wait_spot_requests_active(boto3_ec2, requests,
timeout=None, tentative=False)

Okay so this one waits until no spot request in the iterator is still sitting in the 'open' state, or until a timeout hits, whichever comes first. 💅 It also yields spot requests the second they leave 'open', so you're not stuck waiting for the whole batch. Main character energy 👑

Parameters

• boto3_ec2 (mypy_boto3_ec2.client.EC2Client) — the ec2 client 💖
• requests (collections.abc.Iterable[mypy_boto3_ec2.type_defs.SpotInstanceRequestTypeDef], docs: /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable) — the requests we're waiting on 🌸
• timeout (float, docs: /usr/share/doc/python3/html/library/functions.html#float) — the max number of seconds to wait, or None if you're down to wait forever 🧚 If the timeout goes off, any requests still open get cancelled 💀
• tentative (bool, docs: /usr/share/doc/python3/html/library/functions.html#bool) — if True, give up on a spot request the moment it looks like it won't be fulfilled right away 🙈

Return type: collections.abc.Iterable[list[mypy_boto3_ec2.type_defs.SpotInstanceRequestTypeDef]] (docs: /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable, /usr/share/doc/python3/html/library/stdtypes.html#list) 🎀

🌈 toil.lib.ec2.create_spot_instances

toil.lib.ec2.create_spot_instances(boto3_ec2, price, image_id, spec,
num_instances=1, timeout=None, tentative=False, tags=None)

This one makes instances on the spot market. 🔥

🚨 Heads up bestie: the "UserData" field inside "LaunchSpecification" in spec MUST ALREADY BE base64-encoded. It will NOT be encoded for you, so don't skip this or you'll be crying later 😭

Parameters

• tags (dict[str, str], docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str) — a dict of tag key to tag value, applied to the request 🏷️
• boto3_ec2 (mypy_boto3_ec2.client.EC2Client) 💜
• spec (dict[Literal['LaunchSpecification'], dict[str, Any]], docs: /usr/share/doc/python3/html/library/stdtypes.html#dict) 🍓

Return type: collections.abc.Generator[mypy_boto3_ec2.type_defs.DescribeInstancesResultTypeDef, None, None] (docs: /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator) ✨

🫧 toil.lib.ec2.create_ondemand_instances

toil.lib.ec2.create_ondemand_instances(boto3_ec2, image_id, spec,
num_instances=1)

This one calls the RunInstances EC2 API, but it's extra careful about a race condition 🙌 Recently created instance profiles and IAM roles can still be settling when an instance tries to use them, so this handles that for you. Iconic behaviour 💅

The "UserData" field in spec MUST NOT be base64-encoded. boto3 does that automatically 🩵 For more, check https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2/client/run_instances.html.

Replaced by create_instances 🦋 so maybe use that one instead, bestie.

Parameters

• boto3_ec2 (mypy_boto3_ec2.client.EC2Client) 🌷
• image_id (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 📸
• spec (collections.abc.Mapping[str, Any], docs: /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Mapping, /usr/share/doc/python3/html/library/stdtypes.html#str) 🧁
• num_instances (int, docs: /usr/share/doc/python3/html/library/functions.html#int) 🔢

Return type: list[mypy_boto3_ec2.type_defs.InstanceTypeDef] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list) 💖

🧚 toil.lib.ec2.increase_instance_hop_limit

toil.lib.ec2.increase_instance_hop_limit(boto3_ec2, boto_instance_list)

This one bumps up the default HTTP hop limit. ✨ Why? Because Toil and Kubernetes are running inside a Docker container, and the default hop limit of 1 just isn't enough when you're grabbing metadata with ec2_metadata 🐳 It has to be called after the instances are guaranteed to be running, so timing matters, babe ⏰

Parameters

• boto_instance_list (list[mypy_boto3_ec2.type_defs.InstanceTypeDef], docs: /usr/share/doc/python3/html/library/stdtypes.html#list) — the list of boto instances to modify 🎯
• boto3_ec2 (mypy_boto3_ec2.client.EC2Client) 🫶

Returns: (the docs leave this one blank)

Return type: None 🫧

🌸 toil.lib.ec2.prune

toil.lib.ec2.prune(bushy)

This one clears out any entries in the given dict that have false-y values. 🧹 Boto3 gets cranky about None and would rather the key just not exist at all, so this helps with that 🙄

Parameters

• bushy (dict, docs: /usr/share/doc/python3/html/library/stdtypes.html#dict) 🍀

Return type: dict (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict) 💜

🎀 toil.lib.ec2.iam_client

toil.lib.ec2.iam_client

No description for this one, bestie. Just vibes 🤷‍♀️✨

🌷 toil.lib.ec2.wait_until_instance_profile_arn_exists

toil.lib.ec2.wait_until_instance_profile_arn_exists(instance_profile_arn)

Parameters

• instance_profile_arn (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🌈

🚀 toil.lib.ec2.create_instances

toil.lib.ec2.create_instances(ec2_resource, image_id, key_name,
instance_type, num_instances=1, security_group_ids=None, user_data=None,
block_device_map=None, instance_profile_arn=None, placement_az=None,
subnet_id=None, tags=None)

This is the replacement for create_ondemand_instances 🫶 It uses boto3 and gives back a list of Boto3 instance dicts. Lowkey the one you want 💅

See "create_instances" (this one gives back a list of ec2.Instance objects): https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances

👀 Don't mix this up with "run_instances". It takes the same input args but returns a dictionary instead: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.run_instances

Parameters

• user_data (Optional[Union[str, bytes]], docs: /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#bytes) — the user data that controls instance startup, not base64-encoded 🧃
• tags (Optional[dict[str, str]], docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str) — if you give these, they get applied to the instances and all volumes 🏷️
• ec2_resource (mypy_boto3_ec2.service_resource.EC2ServiceResource) 🌊
• image_id (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 📸
• key_name (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🔑
• instance_type (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🐣
• num_instances (int, docs: /usr/share/doc/python3/html/library/functions.html#int) 🔢
• security_group_ids (Optional[list], docs: /usr/share/doc/python3/html/library/stdtypes.html#list) 🛡️
• block_device_map (Optional[list[dict]], docs: /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#dict) 💾
• instance_profile_arn (Optional[str], docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🪪
• placement_az (Optional[str], docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🗺️
• subnet_id (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🕸️

Return type: list[mypy_boto3_ec2.service_resource.Instance] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list) ✨

🪄 toil.lib.ec2.create_launch_template

toil.lib.ec2.create_launch_template(ec2_client, template_name, image_id,
key_name, instance_type, security_group_ids=None, user_data=None,
block_device_map=None, instance_profile_arn=None, placement_az=None,
subnet_id=None, tags=None)

This one makes a launch template with the name you give it, so you can launch instances with the parameters you pass in 🎀

🌟 Just so you know, we only ever use the default version of any launch template 💅

Under the hood, this calls https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html?highlight=create_launch_template#EC2.Client.create_launch_template 🫶

Parameters

• tags (Optional[dict[str, str]], docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str) — if given, the tags get applied to the template itself, all instances, and all volumes 🏷️
• user_data (Optional[Union[str, bytes]], docs: /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#bytes) — user data to pass to the instances, not base64-encoded 🧃
• ec2_client (mypy_boto3_ec2.client.EC2Client) 💜
• template_name (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 📛
• image_id (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 📸
• key_name (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🔑
• instance_type (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🐣
• security_group_ids — Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>] 🛡️
• block_device_map — Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>]] 💾
• instance_profile_arn — Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>] 🪪
• placement_az — Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>] 📍
• subnet_id — Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>] 🧚

Returns: the ID of the launch template. ✨

Return type: str </usr/share/doc/python3/html/library/stdtypes.html#str> 💖

---

toil.lib.ec2.create_auto_scaling_group(autoscaling_client, asg_name, launch_template_ids, vpc_subnets, min_size, max_size, instance_types=None, spot_bid=None, spot_cheapest=False, tags=None)

toil.lib.ec2.create_auto_scaling_group(autoscaling_client, asg_name, launch_template_ids, vpc_subnets, min_size, max_size, instance_types=None, spot_bid=None, spot_cheapest=False, tags=None)

okay so this one makes a brand new Auto Scaling Group, and the name you give it is also its unique identifier 🦋 no duplicates allowed, bestie.

Parameters

• autoscaling_client (mypy_boto3_autoscaling.client.AutoScalingClient) — your Boto3 client for autoscaling 🔥
• asg_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) — the unique name for the autoscaling group 🌸
• launch_template_ids (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) — the ID of the launch template to make instances from, one for each instance type 🎀
• vpc_subnets (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) — one or more subnet IDs to place the group's instances into. These decide which availability zone(s) the instances launch into 🌈
• min_size (int </usr/share/doc/python3/html/library/functions.html#int>) — the minimum number of instances the group keeps at all times 🫶
• max_size (int </usr/share/doc/python3/html/library/functions.html#int>) — the maximum number of instances the group is allowed to have at any one time 🙈
• instance_types (Optional[collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) — this uses a pool over the instance types you list, instead of the type in the launch template. For on-demand groups it's a prioritized list. For spot groups, AWS balances things according to spot_strategy. It must be 20 types or fewer. 🧠
• spot_bid (Optional[float </usr/share/doc/python3/html/library/functions.html#float>]) — if you set this, the ASG becomes a spot market ASG. The bid is in dollars per instance hour, and every instance type in the group gets bid on equivalently 💸
• spot_cheapest (bool </usr/share/doc/python3/html/library/functions.html#bool>) — if true, it grabs the cheapest spot instances available out of instance_types, instead of the ones that minimize eviction probability 🦋
• tags (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) — tags applied to the ASG only. If you want tags on the instances, add them to the launch template instead 🏷️

Return type: None 🫧

The default version of the launch template is used. 💅

---

toil.lib.ec2nodes

Attributes

• logger 📝
• manager 🧑‍💼
• dirname 📂
• region_json_dirname 🗂️
• EC2Regions 🌍

Classes

• InstanceType 👑

Functions

• is_number(s) — determines if a unicode string (that may include commas) is a number. 🔢
• parse_storage(storage_info) — parses the EC2 JSON storage param string into a number. 💽
• parse_memory(mem_info) — returns the EC2 'memory' string as a float. 🧠
• download_region_json(filename[, region]) — downloads and writes the AWS Billing JSON to a file, using the AWS pricing API. 💰
• reduce_region_json_size(filename) — deletes the info in the JSON file that we don't need, then rewrites it. This makes the file smaller. ✂️
• updateStaticEC2Instances() — generates a new Python file of fetchable EC2 Instances by region, with current prices and specs. ⭐

Module Contents

• toil.lib.ec2nodes.logger 📝
• toil.lib.ec2nodes.manager 🧑‍💼
• toil.lib.ec2nodes.dirname 📂
• toil.lib.ec2nodes.region_json_dirname 🗂️
• toil.lib.ec2nodes.EC2Regions 🌍

class toil.lib.ec2nodes.InstanceType(name, cores, memory, disks, disk_capacity, architecture)

class toil.lib.ec2nodes.InstanceType(name, cores, memory, disks, disk_capacity, architecture)

Parameters

• name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🌷
• cores (int </usr/share/doc/python3/html/library/functions.html#int>) 🧩
• memory (float </usr/share/doc/python3/html/library/functions.html#float>) 🧠
• disks (float </usr/share/doc/python3/html/library/functions.html#float>) 💿
• disk_capacity (float </usr/share/doc/python3/html/library/functions.html#float>) 📦
• architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 🏗️

__slots__ = ('name', 'cores', 'memory', 'disks', 'disk_capacity', 'architecture')

Attributes: name, cores, memory, disks, disk_capacity, architecture 🦋

• __str__() — Return type: str </usr/share/doc/python3/html/library/stdtypes.html#str> 🫶
• __eq__(other) — Parameters: other (object </usr/share/doc/python3/html/library/functions.html#object>); Return type: bool </usr/share/doc/python3/html/library/functions.html#bool> 🔥

---

toil.lib.ec2nodes.is_number(s)

toil.lib.ec2nodes.is_number(s)

Determines if a unicode string (that may include commas) is a number. 🔢

Parameters

• s (str </usr/share/doc/python3/html/library/stdtypes.html#str>) — any unicode string 💖

Returns: True if s represents a number, False otherwise. ✅

Return type: bool </usr/share/doc/python3/html/library/functions.html#bool> 🌈

---

toil.lib.ec2nodes.parse_storage(storage_info)

toil.lib.ec2nodes.parse_storage(storage_info)

Parses the EC2 JSON storage param string into a number. 💽

Examples:

"2 x 160 SSD" "3 x 2000 HDD" "EBS only" "1 x 410" "8 x 1.9 NVMe SSD" "900 GB NVMe SSD"

Parameters 💖

• storage_info (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the EC2 JSON storage param string. That's the whole input, bestie 💾

Returns ✨

• Two floats representing: (# of disks), and (disk_capacity in GiB of each disk). So you get the disk count and the size of each disk 🫶

Return type 🎀

Union[list[int], tuple[Union[int, float], float]]

Doc refs for the types above: list (/usr/share/doc/python3/html/library/stdtypes.html#list), int (/usr/share/doc/python3/html/library/functions.html#int), tuple (/usr/share/doc/python3/html/library/stdtypes.html#tuple), float (/usr/share/doc/python3/html/library/functions.html#float).

---

✨ toil.lib.ec2nodes.parse_memory(mem_info) ✨

okay so this one turns EC2's memory string into a float 🧠 The format should always be # GiB, like '244 GiB' or '1,952 GiB'. Amazon really loves putting commas in their numbers, so this function has to handle that 😭 And if the syntax ever changes, it's going to raise. No silent guessing, which is honestly iconic 💅

Parameters 🌸

• mem_info (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the EC2 JSON memory param string.

Returns 💜

• A float representing memory in GiB.

Return type 🦋

• float (/usr/share/doc/python3/html/library/functions.html#float)

---

✨ toil.lib.ec2nodes.download_region_json(filename, region='us-east-1') ✨

this one downloads and writes the AWS Billing JSON to a file using the AWS pricing API 🛒 The API reference is here: https://aws.amazon.com/blogs/aws/new-aws-price-list-api/ 🔗

Returns 🍓

• A dict of InstanceType objects. The key is the string AWS instance name (example: 't2.micro'), and the value is an InstanceType object representing that AWS instance name. Main character energy 👑

Parameters 🧚

• filename (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• region (str, /usr/share/doc/python3/html/library/stdtypes.html#str), with the default 'us-east-1'

Return type 🌈

• None

---

✨ toil.lib.ec2nodes.reduce_region_json_size(filename) ✨

this one deletes the info in the JSON file that we don't need, then rewrites it so the file is smaller 🪄 The whole reason is the history 😭 We used to download the unified AWS Bulk API JSON, and it eventually crept up to 5.6Gb. Loading that was impossible on a 32Gb RAM machine 💀 Now we download each region JSON individually with AWS's new Query API. Those can still grow ridiculously large one day, so we do what we can to keep the file sizes down (and the amount loaded into memory too). That keeps this script working for longer 🫶

Parameters 💅

• filename (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type ✨

• list (/usr/share/doc/python3/html/library/stdtypes.html#list)[dict (/usr/share/doc/python3/html/library/stdtypes.html#dict)[str (/usr/share/doc/python3/html/library/stdtypes.html#str), Any]]

---

✨ toil.lib.ec2nodes.updateStaticEC2Instances() ✨

this one generates a brand new Python file of fetchable EC2 instances, sorted by region, with current prices and specs 🤑 Heads up, it takes a few (~3+) minutes to run, so you'll need decent internet 🌐

Returns 🔥

• Nothing. It writes a new 'generatedEC2Lists.py' file. 📝

Return type 💜

• None

---

✨ toil.lib.encryption ✨

Submodules 🎀

• toil.lib.encryption.conftest

Attributes 🌸

• collect_ignore

Module Contents 🦋

toil.lib.encryption.conftest.collect_ignore = []

---

✨ toil.lib.exceptions ✨

Exceptions 🙈

• UnimplementedURLException — Unspecified run-time error.

Classes 🍓

• panic — The Python idiom for reraising a primary exception fails when the except block raises a

Functions 🧚

• raise_(exc_type, exc_value, traceback)

Module Contents 💜

okay so panic is a class, and it's a contextmanager 🫠 The Python idiom for reraising a primary exception fails when the except block raises a secondary exception, like while trying to clean up. Then the original exception gets lost and the secondary one is what gets reraised. The fix is to save the primary exception info from sys.exc_info() and reraise that instead ✨

It's meant to be used like this:

try:   # do something that can fail

except:

       with panic( log ):
              # do cleanup that can also fail

If a logging logger is passed to panic(), any secondary Exception raised within the with block gets logged. Otherwise those exceptions are swallowed 👀 At the end of the with block, the primary exception gets reraised.

Parameters 🌈

• log (Optional[logging.Logger], /usr/share/doc/python3/html/library/logging.html#logging.Logger)

Class attributes and methods:

• log = None
• exc_info = None
• __enter__() — Return type: None
• __exit__(*exc_info) — Return type: None

toil.lib.exceptions.raise_(exc_type, exc_value, traceback) takes these:

• exc_type (Optional[type[BaseException]], /usr/share/doc/python3/html/library/functions.html#type and /usr/share/doc/python3/html/library/exceptions.html#BaseException)
• exc_value (Optional[BaseException], /usr/share/doc/python3/html/library/exceptions.html#BaseException)
• traceback (Optional[types.TracebackType], /usr/share/doc/python3/html/library/types.html#types.TracebackType)

Return type 💖

• None

exception toil.lib.exceptions.UnimplementedURLException(url, operation) 🚨

Bases: RuntimeError (/usr/share/doc/python3/html/library/exceptions.html#RuntimeError). Unspecified run-time error, bestie 💀

Parameters 🫶

• url (urllib.parse.ParseResult, /usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult)
• operation (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

---

✨ toil.lib.expando ✨

Classes 💅

• Expando — Pass initial attributes to the constructor:
• MagicExpando — Use MagicExpando for chained attribute access.

Module Contents ✨

class toil.lib.expando.Expando(*args, **kwargs) is so cute 🥹

Bases: dict (/usr/share/doc/python3/html/library/stdtypes.html#dict)

Pass initial attributes to the constructor:

>>> o = Expando(foo=42)
>>> o.foo
42

Dynamically create new attributes:

>>> o.bar = 'hi'
>>> o.bar
'hi'

Expando is a dictionary:

>>> isinstance(o,dict)
True
>>> o['foo']
42

Works great with JSON:

>>> import json
>>> s='{"foo":42}'
>>> o = json.loads(s,object_hook=Expando)
>>> o.foo
42
>>> o.bar = 'hi'
>>> o.bar
'hi'

And since Expando is a dict, it serializes back to JSON just fine:

>>> json.dumps(o, sort_keys=True)
'{"bar": "hi", "foo": 42}'

Attributes can be deleted, too:

>>> o = Expando(foo=42)
>>> o.foo
42
>>> del o.foo
>>> o.foo
Traceback (most recent call last):
...
AttributeError: 'Expando' object has no attribute 'foo'
>>> o['foo']
Traceback (most recent call last):
...
KeyError: 'foo'

>>> del o.foo
Traceback (most recent call last):
...
AttributeError: 'Expando' object has no attribute 'foo'

And copied:

>>> o = Expando(foo=42)
>>> p = o.copy()
>>> isinstance(p,Expando)
True
>>> o == p
True
>>> o is p
False

Same with MagicExpando ...

>>> o = MagicExpando()
>>> o.foo.bar = 42
>>> p = o.copy()
>>> isinstance(p,MagicExpando)
True
>>> o == p
True
>>> o is p
False

... but the copy is shallow:

>>> o.foo is p.foo
True

Class attributes and methods:

• __slots__ = None
• __dict__
• copy() — Return a shallow copy of the dict.

class toil.lib.expando.MagicExpando(*args, **kwargs) 🪄

Bases: Expando

MagicExpando is giving chained attribute access, bestie 💅 Use it when you want to dig through nested stuff without it being a whole thing 🦋

The first time you poke at a missing attribute, it gets auto-filled with a brand new child MagicExpando ✨ Watch:

>>> o=MagicExpando()
>>> o.foo = 42
>>> o
{'foo': 42}
>>> o.bar.hello = 'hi'
>>> o.bar
{'hello': 'hi'}

__getattribute__(name) 🌸 Literally just returns getattr(self, name), no cap 💖

Parameters

• name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

---

💖 toil.lib.ftp_utils 💖

🎀 Attributes

• logger

🎀 Classes

• FtpFsAccess — FTP access with upload 📤

🎀 Module Contents

toil.lib.ftp_utils.logger

class toil.lib.ftp_utils.FtpFsAccess(cache=None) 🦋 FTP access with upload. Taken and modified from <https://github.com/ohsu-comp-bio/cwl-tes/blob/03f0096f9fae8acd527687d3460a726e09190c3a/cwl_tes/ftp.py#L37-L251> 🫶

Parameters

• cache (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Any, ftplib.FTP </usr/share/doc/python3/html/library/ftplib.html#ftplib.FTP>]])

Class attributes 🎀

• cache
• netrc = None

exists(fn) 🔍 Checks if a file or directory exists over an FTP server. You pass in an FTP url as fn, and you get back True or False depending on whether the object exists on the server 💅

Parameters

• fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

• bool </usr/share/doc/python3/html/library/functions.html#bool>

isfile(fn) 📄 Checks if the FTP url points to a file. True if it's a file, otherwise False 🙌

Parameters

• fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

• bool </usr/share/doc/python3/html/library/functions.html#bool>

isdir(fn) 📁 Checks if the FTP url points to a directory. True if it's a directory, otherwise False 💖

Parameters

• fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

• bool </usr/share/doc/python3/html/library/functions.html#bool>

open(fn, mode) 📖 Opens an FTP url. Heads up, this one is read-only, so no write support here 🙅‍♀️

Read-only, bestie. There is no write support for this method.

Parameters

• fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>), the FTP url
• mode (str </usr/share/doc/python3/html/library/stdtypes.html#str>), the mode to open the FTP url in

Return type

• IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]

size(fn) 📏 Gets the size of an FTP object. You pass in the FTP url as fn and get back the size of the object 💅

Parameters

• fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type

• Optional[int </usr/share/doc/python3/html/library/functions.html#int>]

---

💜 toil.lib.generatedEC2Lists 💜

🎀 Attributes

• E2Instances
• regionDict
• ec2InstancesByRegion

🎀 Module Contents

• toil.lib.generatedEC2Lists.E2Instances
• toil.lib.generatedEC2Lists.regionDict
• toil.lib.generatedEC2Lists.ec2InstancesByRegion

---

🧚 toil.lib.history 🧚

Contains tools for tracking history 🗂️

🎀 Attributes

• logger
• RT

🎀 Exceptions

• HistoryDatabaseSchemaTooNewError — Raised when we would write to the history database, but its schema is too

🎀 Classes

• WorkflowSummary — Data class holding summary information for a workflow ✨
• WorkflowAttemptSummary — Data class holding summary information for a workflow attempt ✨
• JobAttemptSummary — Data class holding summary information for a job attempt within a known
• HistoryManager — Class responsible for managing the history of Toil runs 👑

🎀 Functions

• db_retry(function) — Decorates a function with the appropriate retries for accessing the database 🔁

🎀 Module Contents

toil.lib.history.logger

exception toil.lib.history.HistoryDatabaseSchemaTooNewError 🙈 Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

Raised when we would write to the history database, but its schema is too new for us to understand 😭

class toil.lib.history.WorkflowSummary 🌈 Data class holding summary information for a workflow. It represents all the attempts to execute one run of a workflow 💖

Fields

• id: str </usr/share/doc/python3/html/library/stdtypes.html#str>
• name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>
• job_store: str </usr/share/doc/python3/html/library/stdtypes.html#str>
• total_attempts: int </usr/share/doc/python3/html/library/functions.html#int>
• total_job_attempts: int </usr/share/doc/python3/html/library/functions.html#int>
• succeeded: bool </usr/share/doc/python3/html/library/functions.html#bool>
• start_time: float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>
  • Time when the first workflow attempt started, in seconds since epoch ⏰
  • None if there are no attempts recorded
• runtime: float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>
  • Time from the first workflow attempt's start to the last one's end, in seconds ⏱️
  • None if there are no attempts recorded
• trs_spec: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

class toil.lib.history.WorkflowAttemptSummary 🦋 Data class holding summary information for a workflow attempt. It helpfully includes the workflow metadata for Dockstore 🫶

Fields

• workflow_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>
• attempt_number: int </usr/share/doc/python3/html/library/functions.html#int>
• succeeded: bool </usr/share/doc/python3/html/library/functions.html#bool>
• start_time: float </usr/share/doc/python3/html/library/functions.html#float>
• runtime: float </usr/share/doc/python3/html/library/functions.html#float>
• submitted_to_dockstore: bool </usr/share/doc/python3/html/library/functions.html#bool>
• batch_system 🌸 is a str (/usr/share/doc/python3/html/library/stdtypes.html#str) or None (/usr/share/doc/python3/html/library/constants.html#None), so str | None.
• caching 🦋 is a bool (/usr/share/doc/python3/html/library/functions.html#bool) or None, so bool | None.
• toil_version 💜 is a str or None, so str | None.
• python_version 🐍 is a str or None, so str | None.
• platform_system 💻 is a str or None, so str | None.
• platform_machine 🖥️ is a str or None, so str | None.
• workflow_job_store 🗂️ is a str (no None option here), so str.
• workflow_trs_spec 🧾 is a str or None, so str | None.

---

🎀 class toil.lib.history.JobAttemptSummary

okay so this one is a data class 💅 that holds the summary info for one job attempt inside a workflow attempt we already know about ✨

• id 🆔 — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• job_name 🏷️ — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• succeeded 🏆 — bool (/usr/share/doc/python3/html/library/functions.html#bool)
• start_time ⏰ — float (/usr/share/doc/python3/html/library/functions.html#float)
• runtime ⏱️ — float (/usr/share/doc/python3/html/library/functions.html#float)
• submitted_to_dockstore ☁️ — bool (/usr/share/doc/python3/html/library/functions.html#bool)
• cores 🧠 — float (/usr/share/doc/python3/html/library/functions.html#float) or None (/usr/share/doc/python3/html/library/constants.html#None)
• cpu_seconds 🔥 — float (/usr/share/doc/python3/html/library/functions.html#float) or None (/usr/share/doc/python3/html/library/constants.html#None)
• memory_bytes 💾 — int (/usr/share/doc/python3/html/library/functions.html#int) or None (/usr/share/doc/python3/html/library/constants.html#None)
• disk_bytes 💿 — int (/usr/share/doc/python3/html/library/functions.html#int) or None (/usr/share/doc/python3/html/library/constants.html#None)

---

🧚 toil.lib.history.RT

this one is just a type alias, no extra frills 💜

🔁 toil.lib.history.db_retry(function)

Decorate a function with the appropriate retries for accessing the database. 🫶

Parameters:

• function — Callable[Ellipsis, RT]

Return type: Callable[Ellipsis, RT]

---

👑 class toil.lib.history.HistoryManager

Class responsible for managing the history of Toil runs. It's the main character of this whole module 💅

classmethod enabled()

Return True if history should be read from and written to the database. 🔥

If False, no access at all should be made to the database. 🙈

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

classmethod enabled_job()

Return True if job history should be read from and written to the database. 🌈

Always returns False if enabled() returns False.

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

database_path_override

A str (/usr/share/doc/python3/html/library/stdtypes.html#str) or None (/usr/share/doc/python3/html/library/constants.html#None), so str | None, with a default of None ✨

classmethod database_path()

Get the path at which the database we store history in lives. 🗺️

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

classmethod connection()

Connect to the history database. 🍓

⚠️ Caller must not actually use the connection without using ensure_tables() to protect reads and updates.

⚠️ Must be called from inside a top-level method marked @db_retry.

The connection will be in DEFERRED isolation_level, with autocommit off on Python versions that support it. 🧃 If you want to run any commands outside of a transaction, use the no_transaction context manager.

Return type: sqlite3.Connection (/usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Connection)

classmethod no_transaction(con)

Temporarily disable the constant active transaction on the database connection, on Python versions where it exists. 🧁

Commits the current transaction. 💌

Parameters:

• con — sqlite3.Connection (/usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Connection)

Return type: Iterator[None]

classmethod ensure_tables(con, cur)

Ensure that tables exist in the database and the schema is migrated to the current version. 🧹

Leaves the cursor in a transaction where the schema version is known to be correct. 🔒

⚠️ Must be called from inside a top-level method marked @db_retry.

🚨 Raises HistoryDatabaseSchemaTooNewError (#toil.lib.history.HistoryDatabaseSchemaTooNewError) -- if the schema is newer than the current version. 💀

Parameters:

• con — sqlite3.Connection (/usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Connection)
• cur — sqlite3.Cursor (/usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Cursor)

Return type: None

classmethod record_workflow_creation(workflow_id, job_store_spec)

Record that a workflow is being run. 🦋

Takes the Toil config's workflow ID and the location of the job store. 🗂️

Should only be called on the first attempt on a job store, not on a restart. 🙅‍♀️

A workflow may have multiple attempts to run it, some of which succeed and others of which fail. Probably only the last one should succeed. 🤞

Parameters:

• job_store_spec — str (/usr/share/doc/python3/html/library/stdtypes.html#str) -- The job store specifier for the workflow. It should be canonical and always start with the type and a colon. If the job store is later moved by the user, the location will not be updated. 📍
• workflow_id — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

classmethod record_workflow_metadata(workflow_id, workflow_name, trs_spec=None)

Associate a name and optionally a TRS ID and version with a workflow run. 🏷️💖

Parameters:

• workflow_id — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• workflow_name — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• trs_spec — Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/constants.html#None)

Return type: None

classmethod record_job_attempt(workflow_id, workflow_attempt_number, job_name, succeeded, start_time, runtime, cores=None, cpu_seconds=None, memory_bytes=None, disk_bytes=None)

Record that a job ran in a workflow. 🏃‍♀️💨

Doesn't expect the info you give it to uniquely identify the job attempt; it assigns the job attempt its own unique ID. 🆔

Thread safe! 🌈

Parameters:

• job_name — str (/usr/share/doc/python3/html/library/stdtypes.html#str) -- A human-readable name for the job. It's not expected to be a job store ID or to necessarily uniquely identify the job within the workflow. 🙈
• start_time — float (/usr/share/doc/python3/html/library/functions.html#float) -- Job execution start time in seconds since epoch. ⏰
• runtime — float (/usr/share/doc/python3/html/library/functions.html#float) -- Job execution duration in seconds. ⏱️
• cores — Optional[float] (/usr/share/doc/python3/html/library/functions.html#float, /usr/share/doc/python3/html/library/constants.html#None) -- Number of CPU cores the job was scheduled on. 🧠
• cpu_seconds — Optional[float] (/usr/share/doc/python3/html/library/functions.html#float, /usr/share/doc/python3/html/library/constants.html#None) -- CPU core-seconds actually consumed. 🔥
• memory_bytes — Optional[int] (/usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/constants.html#None) -- Peak observed job memory usage. 💾
• disk_bytes — Optional[int] (/usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/constants.html#None) -- Observed job disk usage. 💿
• workflow_id — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• workflow_attempt_number — int (/usr/share/doc/python3/html/library/functions.html#int)
• succeeded — bool (/usr/share/doc/python3/html/library/functions.html#bool)

Return type: None

✨ record_workflow_attempt ✨

classmethod record_workflow_attempt(workflow_id, workflow_attempt_number, succeeded, start_time, runtime, batch_system=None, caching=None, toil_version=None, python_version=None, platform_system=None, platform_machine=None)

Okay bestie, this one records a workflow attempt (a start or a restart) once it has finished or failed. 💅

Parameters

• batch_system (Optional[str]) — the Python type name of the batch system implementation that got used 🚌
• caching (Optional[bool]) — whether Toil filestore-level caching was turned on 💾
• toil_version (Optional[str]) — the version of Toil that ran the workflow 🧚
• python_version (Optional[str]) — the version of Python that ran the workflow 🐍
• platform_system (Optional[str]) — the OS that ran the workflow, like "Darwin", "Linux", etc. 🍎
• platform_machine (Optional[str]) — the CPU type that ran the workflow leader, like "AMD64", etc. 🖥️
• workflow_id (str)
• workflow_attempt_number (int)
• succeeded (bool)
• start_time (float)
• runtime (float)

Return type: None

---

✨ summarize_workflows ✨

classmethod summarize_workflows()

This one is giving "show me everything" 👀. It lists all known workflows along with their summary stats. 📊

Return type: list[WorkflowSummary]

---

✨ get_submittable_workflow_attempts ✨

classmethod get_submittable_workflow_attempts(limit=sys.maxsize)

Lowkey a to-do list 📝. It lists every workflow attempt that hasn't been submitted to Dockstore yet.

Parameters

• limit (int) — get no more than this many 🎯

Return type: list[WorkflowAttemptSummary]

---

✨ get_workflow_attempts_with_submittable_job_attempts ✨

classmethod get_workflow_attempts_with_submittable_job_attempts(limit=sys.maxsize)

This one gets all the workflow attempts that have job attempts still waiting to be submitted to Dockstore 🦋. The workflow attempts themselves will have already finished and been recorded, and they'll have TRS IDs.

Parameters

• limit (int) — get no more than this many 🎯

Return type: list[WorkflowAttemptSummary]

---

✨ get_workflow_attempt ✨

classmethod get_workflow_attempt(workflow_id, attempt_number)

Main character energy 👑. This gets a single workflow attempt summary, if it's there. It doesn't have to be unsubmitted, and it doesn't have to have a TRS ID, so it's pretty flexible 🫶.

Parameters

• workflow_id (str)
• attempt_number (int)

Return type: Optional[WorkflowAttemptSummary]

---

✨ get_unsubmitted_job_attempts ✨

classmethod get_unsubmitted_job_attempts(workflow_id, attempt_number)

This lists all the job attempts in the given workflow attempt that haven't been submitted to Dockstore yet 🌸. Heads up: it doesn't check whether the workflow actually has a TRS ID 🙈.

Parameters

• workflow_id (str)
• attempt_number (int)

Return type: list[JobAttemptSummary]

---

✨ mark_workflow_attempt_submitted ✨

classmethod mark_workflow_attempt_submitted(workflow_id, attempt_number)

This one marks a workflow attempt as successfully submitted to Dockstore ✅. It does not mark that attempt's job attempts as submitted, so those still need their own update 🙃.

Parameters

• workflow_id (str)
• attempt_number (int)

Return type: None

---

✨ mark_job_attempts_submitted ✨

classmethod mark_job_attempts_submitted(job_attempt_ids)

Bulk update energy 🔥. This marks a whole collection of job attempts as submitted to Dockstore in a single transaction.

Parameters

• job_attempt_ids (list[str])

Return type: None

---

✨ count_workflows ✨

classmethod count_workflows()

Counts the workflows in the database 🔢.

Return type: int

---

✨ count_workflow_attempts ✨

classmethod count_workflow_attempts()

Counts the workflow attempts in the database 🔢.

Return type: int

---

✨ count_job_attempts ✨

classmethod count_job_attempts()

Counts the job attempts in the database 🔢.

Return type: int

---

✨ get_fully_submitted_workflow_ids ✨

classmethod get_fully_submitted_workflow_ids(limit=sys.maxsize)

This one finds the workflows that have a successful attempt and no unsubmitted attempts or job attempts. Fully done and dusted 💖.

Parameters

• limit (int)

Return type: list[str]

---

✨ get_oldest_workflow_ids ✨

classmethod get_oldest_workflow_ids(limit=sys.maxsize)

Gets the workflows that are old 🧓. Ngl, the name says it all.

Parameters

• limit (int)

Return type: list[str]

---

✨ delete_workflow ✨

classmethod delete_workflow(workflow_id)

⚠️ Bestie, this one is destructive. It deletes a workflow and all of its attempts and job attempts, so it's gone for good 💀.

It succeeds if the workflow does not exist, so calling it on something that's already gone won't throw a fit 🫶.

Parameters

• workflow_id (str)

Return type: None

---

✨ get_database_byte_size ✨

classmethod get_database_byte_size()

Gets the total number of bytes the database is using 📏.

Return type: int

---

✨ compact_database ✨

classmethod compact_database()

Declutter arc 🧹. This shrinks the database to remove unused space 🌈.

Return type: None

---

✨ enforce_byte_size_limit ✨

classmethod enforce_byte_size_limit(limit=100 * 1024 * 1024)

This is the glow-up you didn't know you needed ✨. It shrinks the database until it's smaller than the given limit, or until it's empty, by throwing away workflows. The data gets tossed in a sensible order, from least important to most important 🗑️.

Parameters

• limit (int)

Return type: None

---

✨ database_dump_lines ✨

classmethod database_dump_lines()

This yields lines from the database dump 🧾. It's meant for debugging tests, so it's a dev-only kind of moment 🙈.

Return type: Iterable[str]

---

✨ toil.lib.history_submission ✨

Contains logic for generating Toil usage history reports to send back to Toil HQ (via Dockstore) 💜, and for working out if the user actually wants to send them 🫶.

✨ Attributes ✨

okay so the attributes are literally just names on the page, no descriptions, so we're vibing off the vibes 🙈

• logger
• KeyType
• Decision
• DIALOG_TITLE
• DIALOG_TEXT
• DIALOG_OPTIONS
• DIALOG_TIMEOUT

---

Classes

• Submission — the class holding a package of information to submit to Dockstore, and the (the docs just trail off there 💀)

---

Functions

bestie, this is the lineup of functions. the big ones get their own deep dive below 🫶

• workflow_execution_id(workflow_attempt) — get the execution ID for a workflow attempt.
• workflow_task_set_execution_id(workflow_attempt) — get the execution ID for a workflow attempt's collection of tasks.
• job_execution_id(job_attempt) — get the execution ID for a job attempt.
• get_parsed_trs_spec(workflow_attempt) — get the TRS ID and version of the workflow, or raise an error.
• create_history_submission([batch_size, desired_tasks]) — make a package of data about recent workflow runs to send in.
• create_current_submission(workflow_id, attempt_number) — make a package of data about the current workflow attempt to send in.
• dialog_tkinter(title, text, options, timeout) — display a dialog with tkinter.
• display_dialog_tkinter(title, text, options, timeout, ...) — display a dialog with tkinter in the current process.
• dialog_tui(title, text, options, timeout) — display a dialog in the terminal.
• ask_user_about_publishing_metrics() — ask the user to set standing workflow submission consent.

---

Module Contents

toil.lib.history_submission.logger

toil.lib.history_submission.workflow_execution_id(workflow_attempt)

Get the execution ID for a workflow attempt. ✨

Result will follow Dockstore's rules.

Deterministic. (no randomness, no surprises, same input gives the same output 🔮)

Parameters

• workflow_attempt (toil.lib.history.WorkflowAttemptSummary)

Return type

• str

toil.lib.history_submission.workflow_task_set_execution_id(workflow_attempt)

Get the execution ID for a workflow attempt's collection of tasks. 🧚

Result will follow Dockstore's rules.

Deterministic.

Parameters

• workflow_attempt (toil.lib.history.WorkflowAttemptSummary)

Return type

• str

toil.lib.history_submission.job_execution_id(job_attempt)

Get the execution ID for a job attempt. 💖

Result will follow Dockstore's rules.

Deterministic.

Parameters

• job_attempt (toil.lib.history.JobAttemptSummary)

Return type

• str

toil.lib.history_submission.get_parsed_trs_spec(workflow_attempt)

Get the TRS ID and version of the workflow, or raise an error. 👑

Returns

The TRS ID and the TRS version of the workflow run.

Raises ValueError if the workflow does not have a TRS spec or if the spec does not contain a version. Not it, bestie 😭

Parameters

• workflow_attempt (toil.lib.history.WorkflowAttemptSummary)

Return type

• tuple[str, str]

class toil.lib.history_submission.Submission

Class holding a package of information to submit to Dockstore, and the information needed to mark the workflows/tasks as submitted if it is accepted.

Acceptance is at the TRS id/version combination level within the Submission, since we need one request per TRS ID/version combination. So it's one request per combo, no shortcuts 🦋

data:

dict[tuple[str, str], tuple[list[tuple[toil.lib.dockstore.RunExecution, str, int]], list[tuple[toil.lib.dockstore.TaskExecutions, list[str]]]]]

add_workflow_attempt(workflow_attempt)

Add a workflow attempt to the submission. 🌸

May raise an exception if the workflow attempt is not well-formed.

Parameters

• workflow_attempt (toil.lib.history.WorkflowAttemptSummary)

Return type

• None

add_job_attempts(workflow_attempt, job_attempts)

Add the job attempts for a workflow attempt to the submission. 🎀

Parameters

• workflow_attempt (toil.lib.history.WorkflowAttemptSummary)
• job_attempts (list[toil.lib.history.JobAttemptSummary]) 💖

Return type: None 🌸

empty()

Okay so this one is the vibe check 🫶 It returns True if there is literally nothing to submit. Zero tasks, zero workflows, it's giving "nothing to see here, bestie" 💀

Return type: bool

report()

Need a recap before you hit send? ✨ report() composes a multi-line, human-readable string about everything that's going to get submitted. Main character recap energy 💅

Return type: str

submit()

The big moment 🔥 submit() sends workflow and task information over to Dockstore. It takes for granted that you already approved this, so make sure you did 🙈

If the submission succeeds for a TRS id/version combination, it gets recorded in the history database 📝 No ghosting, it's logged.

Errors get handled internally, so it will not raise if the submission doesn't go through 🫶

It returns False if there was any error, and True if the submission was accepted and recorded locally ✨

Return type: bool

toil.lib.history_submission.create_history_submission(batch_size=None, desired_tasks=None)

Okay this one packages up a whole bundle of data about your recent workflow runs so it can be sent in 🎀 It hands back a Submission object that remembers how to update the history database to mark when it's successfully submitted. So organized, so iconic 👑

Parameters

• batch_size (Optional[int]) — how many workflows to try and submit in one request 🍓
• desired_tasks (Optional[int]) — how many tasks to try and squeeze into a task submission batch. Use 0 if you don't want any task information sent at all 🙅

Return type: Submission

toil.lib.history_submission.create_current_submission(workflow_id, attempt_number)

Same energy, smaller footprint 🦋 This one bundles up data about only the current workflow attempt. It's the move when you just want to publish metrics for this one time 💜

Parameters

• workflow_id (str)
• attempt_number (int)

Return type: Submission

toil.lib.history_submission.KeyType

Just a bare name in the docs, no extra details 🧚

toil.lib.history_submission.dialog_tkinter(title, text, options, timeout)

Okay this dialog is so cute 🎀 It pops up a tkinter dialog with the title, text, and options you give it. It runs in its own separate Python process, so a Mac dock icon doesn't hang around for as long as Toil is running (nobody wants a zombie dock icon 💀).

Parameters

• options (dict[KeyType, str]) — a dict mapping each machine-readable option key to its button text 🌈
• title (str)
• text (str)
• timeout (float)

Returns

The key of the option that got picked, or None if the user declined to pick one.

Raises an exception if the dialog can't be displayed 😭

Return type: Optional[KeyType]

toil.lib.history_submission.display_dialog_tkinter(title, text, options, timeout, result_queue)

Same tkinter dialog, but this one runs right in the current process 💅 Its whole job is to send back either the key of the chosen option or an exception through the result queue 📬

Parameters

• options (dict[KeyType, str]) — a dict mapping each machine-readable option key to its button text
• title (str)
• text (str)
• timeout (float)
• result_queue (multiprocessing.Queue[Union[Exception, Optional[KeyType]]])

Return type: None

toil.lib.history_submission.dialog_tui(title, text, options, timeout)

The terminal version of the dialog 🖥️ It shows the given title, text, and options right in your terminal, no pop-up window needed ✨

Parameters

• options (dict[KeyType, str]) — a dict mapping each machine-readable option key to its button text
• title (str)
• text (str)
• timeout (float)

Returns

The key of the option that got picked, or None if the user declined to pick one.

Raises an exception if the dialog can't be displayed 😭

Return type: Optional[KeyType]

toil.lib.history_submission.Decision

Another bare name with no extra details, bestie 🧚

toil.lib.history_submission constants

• toil.lib.history_submission.DIALOG_TITLE = 'Publish Workflow Metrics on Dockstore.org?' 🫶
• toil.lib.history_submission.DIALOG_TEXT = '' (an empty string, so no body text by default)
• toil.lib.history_submission.DIALOG_OPTIONS: dict[Decision, str] 💜
• toil.lib.history_submission.DIALOG_TIMEOUT = 120 ⏰

toil.lib.history_submission.ask_user_about_publishing_metrics()

This one asks the user to set their standing consent for workflow submissions 🙋 If they make a persistent choice, either always or never, it gets saved to the default Toil config file 💾

Returns the user's decision about when to publish metrics.

Return type: Union[Literal['all'], Literal['current'], Literal['no']]

---

toil.lib.humanize

Attributes

• logger 🪵

Functions

• bytes2human(n) — turns n bytes into a human readable string 🍓
• human2bytes(s) — guesses the string format using the default symbols set, then gives you back the matching bytes as an integer 🧮

Module Contents

• toil.lib.humanize.logger

toil.lib.humanize.bytes2human(n)

Converts n bytes into a human readable string 🌸

Parameters

• n (SupportsInt)

Return type: str

toil.lib.humanize.human2bytes(s)

Okay this one is lowkey clever 🧠 It attempts to guess the string format based on the default symbols set, then returns the corresponding bytes as an integer. If it can't recognize the format, a ValueError gets raised 😭

Parameters

• s (str)

Return type: int

---

toil.lib.io

Attributes

• logger
• TOIL_URI_SCHEME
• STANDARD_SCHEMES
• REMOTE_SCHEMES
• ALL_SCHEMES

None of these have a description listed, so you'll just have to vibe with the names 🫧

✨ Classes ✨

okay so this section is only two classes, but they're both iconic 💅

• WriteWatchingStream — a stream wrapper 🌸 bestie, it quietly calls whatever functions you pass to onWrite(), and each time something gets written it tells them how many bytes were written that time 🫶
• ReadableFileObj — a Protocol 🎀 that's way more specific than what file_digest is willing to accept as an argument 👑

---

✨ Functions ✨

the function list is huge, so here's the vibe check for each one 🦋

• get_toil_home() — gets the Toil home directory, where configuration and global state live 🏡
• is_standard_url(filename) — returns True if the given URL is a non-Toil, non-file: URL 🌐
• is_remote_url(filename) — decides if a filename is a known, non-file kind of URL 🛰️
• is_any_url(filename) — decides if a string is a URI like <http://> or <file://> 🔗
• is_url_with_scheme(filename, schemes) — returns True if filename is a URL with any of the given schemes, and False otherwise 🎯
• is_toil_url(filename) — returns True if a URL is a toilfile: or toildir: URL 💜
• is_toil_file_url(filename) — returns True if a URL is a toilfile: URL 📄
• is_toil_dir_url(filename) — returns True if a URL is a toildir: URL 📁
• is_file_url(filename) — returns True if a URL is a file: URL 🫶
• is_directory_url(filename) — returns True if a URL points to a directory 🗂️
• mkdtemp([suffix, prefix, dir]) — makes a temporary directory just like tempfile.mkdtemp, but with relaxed permissions 🧚
• robust_rmtree(path) — robustly tries to delete paths 🧹
• atomic_tmp_file(final_path) — returns a tmp file name to use with atomic_install. This will be in the… (the docs literally just trail off right there 💀)
• atomic_install(tmp_path, final_path) — atomically installs tmp_path as final_path ⚡
• AtomicFileCreate(final_path[, keep]) — a context manager for creating a temporary file. Entering returns the path to… (and the docs cut off again, no cap 🙈)
• atomic_copy(src_path, dest_path[, executable]) — copies a file using posix atomic creations semantics 📋
• atomic_copyobj(src_fh, dest_path[, length, executable]) — copies an open file using posix atomic creations semantics 📋
• make_public_dir(in_directory[, suggested_name]) — makes a publicly-accessible directory inside the given directory 📣
• try_path(path[, min_size]) — tries to use the given path. It returns it if it exists or can be made, 💭 (and the sentence just kinda hangs there with a comma)
• file_digest(f, alg_name) — a polyfilled hashlib.file_digest that works on Python <3.11 ⏪

---

💖 Module Contents 💖

okay the main event 🌈 here's the full module reference, bestie

toil.lib.io.logger

✨ toil.lib.io.get_toil_home() ✨

Get the Toil home directory for storing configuration and global state.

Raises an error if it does not exist and cannot be created. Safe to run simultaneously in multiple processes. 🧵

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🎀 Constants 🎀

these are the scheme lists Toil uses, and they're literally the goods 🧃

toil.lib.io.TOIL_URI_SCHEME = 'toilfile:'
toil.lib.io.STANDARD_SCHEMES = ['http:', 'https:', 's3:', 'gs:', 'ftp:']
toil.lib.io.REMOTE_SCHEMES = ['http:', 'https:', 's3:', 'gs:', 'ftp:', 'toilfile:', 'toildir:']
toil.lib.io.ALL_SCHEMES = ['http:', 'https:', 's3:', 'gs:', 'ftp:', 'toilfile:', 'toildir:', 'file:']

🌐 toil.lib.io.is_standard_url(filename) 🌐

Return True if the given URL is a non-Toil, non-file: URL.

Parameters: filename (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

🛰️ toil.lib.io.is_remote_url(filename) 🛰️

Decide if a filename is a known, non-file kind of URL.

Parameters: filename (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

🔗 toil.lib.io.is_any_url(filename) 🔗

Decide if a string is a URI like <http://> or <file://>.

Otherwise it might be a bare path. 🪵

Parameters: filename (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

🎀 toil.lib.io.is_url_with_scheme 🎀

toil.lib.io.is_url_with_scheme(filename, schemes)

Okay bestie, this one is the URL bouncer 💅 It returns True if filename is a URL with any of the given schemes, and False otherwise ✨

Parameters

• filename — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• schemes — list[str] (/usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

🦋 toil.lib.io.is_toil_url 🦋

toil.lib.io.is_toil_url(filename)

Checks if a URL is a toilfile: or toildir: URL. Main character energy 👑 Returns True if it is, False if not.

Parameters

• filename — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

🌸 toil.lib.io.is_toil_file_url 🌸

toil.lib.io.is_toil_file_url(filename)

This one is stricter. It only returns True for toilfile: URLs 🫶

Parameters

• filename — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

🧚 toil.lib.io.is_toil_dir_url 🧚

toil.lib.io.is_toil_dir_url(filename)

Returns True if a URL is a toildir: URL. ngl this is a little sneaky, so note that it might point to either a directory or a leaf file 👀

Parameters

• filename — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

💜 toil.lib.io.is_file_url 💜

toil.lib.io.is_file_url(filename)

Returns True if a URL is a file: URL.

Bare paths get a False, even if they're real. No cap, that's a real trap 🙈

Parameters

• filename — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

🍓 toil.lib.io.is_directory_url 🍓

toil.lib.io.is_directory_url(filename)

Returns True if a URL points to a directory. It covers toildir: URLs and determines whether they point to directories or leaf files. It also covers other supported remote URL schemes, so it's pretty inclusive 🌈

Parameters

• filename — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

🔥 toil.lib.io.mkdtemp 🔥

toil.lib.io.mkdtemp(suffix=None, prefix=None, dir=None)

This is basically tempfile.mkdtemp, but with relaxed permissions 🦋 The directory gets 711 instead of 700, so the group and all other users can traverse it.

Why does this matter? If the directory is on NFS and the Docker daemon wants to mount it (or a file inside it) into a container, NFS makes even the Docker daemon act bound by the file permissions. Lowkey wild 😭

See <https://github.com/DataBiosphere/toil/issues/4644>, and <https://stackoverflow.com/a/67928880> which talks about a similar problem, but in the context of user namespaces 📚

Parameters

• suffix — Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)
• prefix — Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)
• dir — Optional[toil.lib.misc.StrPath]

Return type

str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🧹 toil.lib.io.robust_rmtree 🧹

toil.lib.io.robust_rmtree(path)

Robustly tries to delete paths. It's the cleanup queen 👑 It continues silently if the path is already gone, or if it goes away while the function is running.

It may still raise an error if a path changes between file and directory while the function is executing, or if a permission error is encountered. Not invincible, bestie 💀

Parameters

• path — Union[str, bytes] (/usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#bytes)

Return type

None (/usr/share/doc/python3/html/library/constants.html#None)

---

🌈 toil.lib.io.atomic_tmp_file 🌈

toil.lib.io.atomic_tmp_file(final_path)

Gives you a tmp file name to use with atomic_install. It lives in the same directory as final_path, and it keeps the same extension as finalPath ✨

Special case alert 🚨 If the final path is in /dev (like /dev/null or /dev/stdout), it gets returned unchanged, and atomic_tmp_install will do nothing.

Parameters

• final_path — toil.lib.misc.StrPath

Return type

str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

💖 toil.lib.io.atomic_install 💖

toil.lib.io.atomic_install(tmp_path, final_path)

Atomically installs tmp_path as final_path. Clean and iconic 🫶

Parameters

• tmp_path — toil.lib.misc.StrPath
• final_path — toil.lib.misc.StrPath

Return type

None (/usr/share/doc/python3/html/library/constants.html#None)

---

🦋 toil.lib.io.AtomicFileCreate 🦋

toil.lib.io.AtomicFileCreate(final_path, keep=False)

This is a context manager that creates a temporary file ✨ When you enter it, you get the path to the temp file, which sits in the same directory as finalPath.

If the code inside the context succeeds, the file gets renamed to its actual name. If an error happens, the file doesn't get installed and gets removed, unless keep is specified. Lowkey the best safety net 🧸

Parameters

• final_path — toil.lib.misc.StrPath
• keep — bool (/usr/share/doc/python3/html/library/functions.html#bool)

Return type

collections.abc.Iterator[str] (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator, /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🎀 toil.lib.io.atomic_copy 🎀

toil.lib.io.atomic_copy(src_path, dest_path, executable=None)

Copies a file using POSIX atomic creation semantics. Super reliable energy 💅

Parameters

• src_path — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• dest_path — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• executable — Optional[bool] (/usr/share/doc/python3/html/library/functions.html#bool)

Return type

None (/usr/share/doc/python3/html/library/constants.html#None)

---

🌸 toil.lib.io.atomic_copyobj 🌸

toil.lib.io.atomic_copyobj(src_fh, dest_path, length=16384, executable=False)

Same vibe as atomic_copy, but this time you hand it an open file and it does POSIX atomic creation semantics on it 🧚

Parameters

• src_fh — io.BytesIO (/usr/share/doc/python3/html/library/io.html#io.BytesIO)
• dest_path — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• length — int (/usr/share/doc/python3/html/library/functions.html#int)
• executable — bool (/usr/share/doc/python3/html/library/functions.html#bool)

Return type

None (/usr/share/doc/python3/html/library/constants.html#None)

---

🍓 toil.lib.io.make_public_dir 🍓

toil.lib.io.make_public_dir(in_directory, suggested_name=None)

Makes a publicly-accessible directory inside the directory you give it 🫶

Parameters

• suggested_name — Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str) -- Use this directory name first if possible.
• in_directory — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

str (/usr/share/doc/python3/html/library/stdtypes.html#str)

The naming process is giving strategic 🧠 It tries to make a random directory name with length 4 that doesn't exist yet, using the given prefix. If that's taken, it tries length 5, then 6, and so on, up to a max of 32 (the length of a uuid4 with the dashes replaced).

This is mostly here so you don't end up with super long file names when generating directories. And if somehow all of that fails (which should be incredibly unlikely), it defaults to a normal uuid4, which was our old default 🥹

---

💅 toil.lib.io.try_path 💅

toil.lib.io.try_path(path, min_size=100 * 1024 * 1024)

Tries to use the given path. It returns the path if it exists (or can be made) and you can make things inside it. Otherwise, it returns None 🙈

Parameters

• min_size — int (/usr/share/doc/python3/html/library/functions.html#int) -- Rejects paths on filesystems smaller than this many bytes.
• path — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🧸 class toil.lib.io.WriteWatchingStream 🧸

class toil.lib.io.WriteWatchingStream(backingStream)

A stream wrapper that calls any functions you pass to onWrite() with the number of bytes written on every single write 📊

It's not seekable, so don't go looking for a rewind button 🙃

Parameters

• backingStream — IO[Any]

Attributes

• backingStream
• writeListeners — list[Callable[[int], None]] = [] (/usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/constants.html#None)

Methods

onWrite(listener)

Calls the given listener with the number of bytes written on every write 🎀

Parameters

• listener — Callable[[int], None] (/usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/constants.html#None)

Return type None (/usr/share/doc/python3/html/library/constants.html#None)

write(data)

Writes the given data to the file 💖

Parameters

• data — bytes (/usr/share/doc/python3/html/library/stdtypes.html#bytes)

Return type None (/usr/share/doc/python3/html/library/constants.html#None)

writelines(datas)

Writes each string from the given iterable, without newlines. You're in charge of the line breaks 🙌

Parameters

• datas — collections.abc.Iterable[bytes] (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable, /usr/share/doc/python3/html/library/stdtypes.html#bytes)

Return type None (/usr/share/doc/python3/html/library/constants.html#None)

flush()

Flushes the backing stream 🌊

Return type None (/usr/share/doc/python3/html/library/constants.html#None)

close()

Closes the backing stream 🔒

Return type None (/usr/share/doc/python3/html/library/constants.html#None)

---

👑 class toil.lib.io.ReadableFileObj 👑

Bases: Protocol (/usr/share/doc/python3/html/library/typing.html#typing.Protocol)

okay so this little protocol is giving very specific 💅 it's a stricter version of what file_digest takes as an argument, and it guarantees a read() method 🫶 ngl the plan was to extend the protocol from Typeshed for hashlib, but those only get declared for 3.11+, so no dice for older Pythons 😭

readinto(buf, /) 🌸

• Parameters
  • buf (bytearray)
• Return type
  • int

readable() 🦋

• Return type
  • bool

read(number) ✨

• Parameters
  • number (int)
• Return type
  • bytes

---

toil.lib.io.file_digest(f, alg_name) 🧚

Polyfilled hashlib.file_digest that works on Python <3.11. It's the glow-up you didn't know you needed 💖

• Parameters
  • f (ReadableFileObj)
  • alg_name (str)
• Return type
  • hashlib._Hash

---

✨ toil.lib.memoize ✨

💅 Attributes

• memoize — Memoizes a function result based on its parameters using this decorator.
• MAT — no description, it's a mystery 🙈
• MRT — no description, it's a mystery 🙈

🎀 Functions

• sync_memoize(f) — Like memoize, but guarantees the decorated function only gets called once, even when multiple threads are calling it with multiple parameters 👑
• parse_iso_utc(s) — Parses an RFC 3339 ISO 8601 time in the UTC timezone 🕰️
• strict_bool(s) — A variant of bool() that only accepts two possible string values 🫶

🦋 Module Contents

toil.lib.memoize.memoize

Memoize a function result based on its parameters using this decorator, bestie. It's giving lazy initialization vibes ✨ If the decorating function gets invoked by multiple threads, the decorated function might get called more than once with the same arguments, so keep that in mind 👀

toil.lib.memoize.MAT

nothing else to say about this one 🙈

toil.lib.memoize.MRT

nothing else to say about this one 🙈

toil.lib.memoize.sync_memoize(f)

Like memoize, but guarantees that the decorated function is only called once, even when multiple threads are calling the decorating function with multiple parameters. Main character energy 👑

• Parameters
  • f (Callable[[MAT], MRT])
• Return type
  • Callable[[MAT], MRT]

toil.lib.memoize.parse_iso_utc(s)

Parses an RFC 3339 ISO 8601 time in the UTC timezone. Other timezones are not supported, so don't even try 🙅‍♀️ It returns a timezone-aware UTC datetime object.

• Parameters
  • s (str) — the ISO-formatted time
• Returns
  • A timezone-aware UTC datetime object
• Raises ValueError — if the string isn't in the correct format or isn't in the UTC timezone. No notes, it'll yell at you 💀
• Return type
  • datetime.datetime

Here's how it behaves in action 🦋

>>> parse_iso_utc('2016-04-27T00:28:04.000Z')
datetime.datetime(2016, 4, 27, 0, 28, 4, tzinfo=datetime.timezone.utc)
>>> parse_iso_utc('2016-04-27T00:28:04Z')
datetime.datetime(2016, 4, 27, 0, 28, 4, tzinfo=datetime.timezone.utc)
>>> parse_iso_utc('2016-04-27T00:28:04X')
Traceback (most recent call last):
...
ValueError: Not a valid ISO datetime in UTC: 2016-04-27T00:28:04X

toil.lib.memoize.strict_bool(s)

A variant of bool() that only accepts two possible string values. Strict, but iconic 💅

• Parameters
  • s (str)
• Return type
  • bool

---

✨ toil.lib.misc ✨

💅 Attributes

• logger — 🙈
• StrPath — 🙈
• FileDescriptorOrPath — 🙈

🌈 Exceptions

• CalledProcessErrorStderr — a version of CalledProcessError that includes stderr in the error message if it's set 😭🫶

✨ Functions ✨

Okay bestie, here's the whole lineup of helpers in toil.lib.misc 💅 The table cuts off mid-sentence at slow_down, so the full details for that one are down below 🙈

• get_public_ip() — grabs the IP this machine uses to reach the internet 🌐
• get_user_name() — gets the current user name, or a suitable stand-in string if the user name isn't available 🫶
• utc_now() — gives you a datetime in the UTC timezone for right now ⏰
• unix_now_ms() — the current time in milliseconds since the Unix epoch ⏱️
• unix_seconds_to_timestamp(timestamp) — turns seconds since the Unix epoch into an ISO 8601 string 📅
• unix_seconds_to_local_time(timestamp) — gives you a local time matching a number of seconds since the Unix epoch 🗺️
• seconds_to_duration(time_difference) — turns a time difference in seconds into an ISO 8601 duration string ⌛
• slow_down(seconds) — finished Toil jobs can't have taken 0 seconds, but the table cuts off right there 🫠
• printq(msg, quiet[, log]) — for functions that get used in both Toil proper and the admin scripts 🧚
• truncExpBackoff() — the table has no description for this one, total mystery energy 🌫️
• call_command(cmd, *args[, input, timeout, useCLocale, ...]) — a simplified way to run external commands 🚀

---

✨ Module Contents ✨

Next up is the module contents, starting with the logger 📝

• toil.lib.misc.logger

Now the type aliases, which are basically nicknames for "this can be one of these things" 💜

type toil.lib.misc.StrPath = str </usr/share/doc/python3/html/library/stdtypes.html#str> | os.PathLike </usr/share/doc/python3/html/library/os.html#os.PathLike>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

type toil.lib.misc.FileDescriptorOrPath = int </usr/share/doc/python3/html/library/functions.html#int> | bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes> | os.PathLike </usr/share/doc/python3/html/library/os.html#os.PathLike>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>] | StrPath

---

✨ toil.lib.misc.get_public_ip() ✨

Okay so this grabs the IP this machine uses to contact the internet 🌐 If you're behind a NAT, you'll still get this computer's IP and not the router's 🦋

Return type: str — /usr/share/doc/python3/html/library/stdtypes.html#str

✨ toil.lib.misc.get_user_name() ✨

Gets the current user name, or a suitable substitute string if the user name is not available 🫶

Return type: str — /usr/share/doc/python3/html/library/stdtypes.html#str

✨ toil.lib.misc.utc_now() ✨

Gives you a datetime in the UTC timezone that matches right now ⏰

Return type: datetime.datetime — /usr/share/doc/python3/html/library/datetime.html#datetime.datetime

✨ toil.lib.misc.unix_now_ms() ✨

Returns the current time in milliseconds since the Unix epoch, no cap ⏱️

Return type: float — /usr/share/doc/python3/html/library/functions.html#float

✨ toil.lib.misc.unix_seconds_to_timestamp(timestamp) ✨

Converts a time in seconds since the Unix epoch into an ISO 8601 string 📅

Parameters

• timestamp (float — /usr/share/doc/python3/html/library/functions.html#float)

Return type: str — /usr/share/doc/python3/html/library/stdtypes.html#str

✨ toil.lib.misc.unix_seconds_to_local_time(timestamp) ✨

Returns a local time that lines up with the given number of seconds since the Unix epoch 🗺️

Parameters

• timestamp (float — /usr/share/doc/python3/html/library/functions.html#float)

Return type: datetime.datetime — /usr/share/doc/python3/html/library/datetime.html#datetime.datetime

✨ toil.lib.misc.seconds_to_duration(time_difference) ✨

Turns a time difference in seconds into an ISO 8601 duration string ⌛

Parameters

• time_difference (float — /usr/share/doc/python3/html/library/functions.html#float)

Return type: str — /usr/share/doc/python3/html/library/stdtypes.html#str

✨ toil.lib.misc.slow_down(seconds) ✨

Finished Toil jobs are not allowed to have taken 0 seconds 🚫⏱️ The tricky part is that Kubernetes timestamps round things to the nearest second, so in some batch systems a pod can end up with identical start and end timestamps 🙈

So this function takes a possibly 0 job length in seconds and enforces a minimum length so Toil stays happy 💖 It returns the seconds, or a small positive number if seconds is 0.

Parameters

• seconds (float — /usr/share/doc/python3/html/library/functions.html#float) — the timestamp difference

Returns: seconds, or a small positive number if seconds is 0

Return type: float — /usr/share/doc/python3/html/library/functions.html#float

✨ toil.lib.misc.printq(msg, quiet, log=False) ✨

This one is for functions used simultaneously in Toil proper and in the admin scripts 🧚

Okay so the admin scripts "print" to stdout, while Toil proper uses logging 🪵 Say a script cleans up IAM, EC2, etc. cruft that got left behind after failed CI runs. It can call an AWS delete IAM role function, and that prints or logs progress (unless quiet is True), depending on whether the function is called in, say, the jobstore or a script 💅

Parameters

• msg (str — /usr/share/doc/python3/html/library/stdtypes.html#str) — the string to print or log to stdout
• quiet (bool — /usr/share/doc/python3/html/library/functions.html#bool) — silences output to stdout
• log (bool — /usr/share/doc/python3/html/library/functions.html#bool) — use logging instead of "print" to the screen

Return type: None

✨ toil.lib.misc.truncExpBackoff() ✨

No description here, bestie 🌫️

Return type: collections.abc.Iterator (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator) of float (/usr/share/doc/python3/html/library/functions.html#float)

---

✨ exception toil.lib.misc.CalledProcessErrorStderr ✨

exception toil.lib.misc.CalledProcessErrorStderr(returncode, cmd, output=None, stderr=None)

Bases: subprocess.CalledProcessError — /usr/share/doc/python3/html/library/subprocess.html#subprocess.CalledProcessError

This is a version of CalledProcessError that includes stderr in the error message if it is set 🧾

__str__() — returns str(self) 🫧

Return type: str — /usr/share/doc/python3/html/library/stdtypes.html#str

✨ toil.lib.misc.call_command(...) ✨

toil.lib.misc.call_command(cmd, *args, input=None, timeout=None, useCLocale=True, env=None, quiet=False)

Simplified calling of external commands 🚀

If the process fails, CalledProcessErrorStderr gets raised 💥

The captured stderr is always printed, even if an exception happens, so it can be logged 🪵

It always logs the command at debug log level 🐛

Parameters

✨ toil.lib.objects ✨

• quiet (Optional[bool], see /usr/share/doc/python3/html/library/functions.html#bool) — bestie, True means shh 🤫, don't log the command output at all. False is the default, and it means the output gets logged at the debug log level 🔍.
• useCLocale (bool, see /usr/share/doc/python3/html/library/functions.html#bool) — True forces the C locale 🌸. This stops the failures that can happen on some batch systems when they're using a UTF-8 locale 😭.
• cmd (list[str], see /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str) 💅
• args (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) 🧚
• input (Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓
• timeout (Optional[float], see /usr/share/doc/python3/html/library/functions.html#float) ⏰
• env (Optional[dict[str, str]], see /usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/stdtypes.html#str) 🌈

Returns

You get back the command's standard output, decoded as utf-8 ✨.

Return type

str (see /usr/share/doc/python3/html/library/stdtypes.html#str) 🦋

---

✨ toil.lib.objects ✨

Classes 💖

┌────────────┬────────────────────┐
│ InnerClass │ Note that this is  │
│            │ EXPERIMENTAL code. │
└────────────┴────────────────────┘

Module Contents 🎀

class toil.lib.objects.InnerClass(inner_class)

Okay so, this is EXPERIMENTAL code 🧪, so be a little careful with it, bestie.

So a nested class (the inner class) decorated with this gets an extra attribute called outer. That attribute points at the instance of the nesting class (the outer class) that was used to make the inner class. You don't have to pass the outer instance into the inner class's constructor, because it gets set magically ✨. This was shamelessly stolen from http://stackoverflow.com/questions/2278426/inner-classes-how-can-i-get-the-outer-class-object-at-construction-time#answer-2278595.

The names got made more descriptive (I hope 🙈), and caching of the BoundInner classes got added too.

Caveat: Within the inner class, self.__class__ will not be the inner class. It'll be a dynamically created subclass of it 🌀. That subclass has the same name as the inner class, but its __module__ will be different. There's one dynamic subclass per inner class and per outer class instance, but only if that outer instance actually created any instances of the inner class.

Here's the vibe in code:

>>> class Outer(object):
...     def new_inner(self):
...         # self is an instance of the outer class
...         inner = self.Inner()
...         # the inner instance's 'outer' attribute is set to the outer instance
...         assert inner.outer is self
...         return inner
...     @InnerClass
...     class Inner(object):
...         def get_outer(self):
...             return self.outer
...         @classmethod
...         def new_inner(cls):
...             return cls()
>>> o = Outer()
>>> i = o.new_inner()
>>> i
<toil.lib.objects.Inner...> bound to <toil.lib.objects.Outer object at ...>

>>> i.get_outer()
<toil.lib.objects.Outer object at ...>

Now with inheritance for both the inner and outer classes 👑:

>>> class DerivedOuter(Outer):
...     def new_inner(self):
...         return self.DerivedInner()
...     @InnerClass
...     class DerivedInner(Outer.Inner):
...         def get_outer(self):
...             assert super( DerivedOuter.DerivedInner, self ).get_outer() == self.outer
...             return self.outer
>>> derived_outer = DerivedOuter()
>>> derived_inner = derived_outer.new_inner()
>>> derived_inner
<toil.lib.objects...> bound to <toil.lib.objects.DerivedOuter object at ...>

>>> derived_inner.get_outer()
<toil.lib.objects.DerivedOuter object at ...>

Test a static references:

Test a static references: >>> Outer.Inner # doctest: +ELLIPSIS
<class 'toil.lib.objects...Inner'> >>> DerivedOuter.Inner #
doctest: +ELLIPSIS <class 'toil.lib.objects...Inner'> >>>
DerivedOuter.DerivedInner #doctest: +ELLIPSIS <class
'toil.lib.objects...DerivedInner'>

Can't decorate top-level classes 🚫. Unfortunately, this only gets detected when the instance is created, not when the class is defined 😩.

Can't decorate top-level classes. Unfortunately, this is detected
when the instance is created, not when the class is defined.  >>>
@InnerClass ... class Foo(object): ...    pass >>> Foo()
Traceback (most recent call last): ...  RuntimeError: Inner
classes must be nested in another class.

All inner instances should refer to a single outer instance:

All inner instances should refer to a single outer instance: >>>
o = Outer() >>> o.new_inner().outer == o == o.new_inner().outer
True

All inner instances should be of the same class ...

All inner instances should be of the same class ...  >>>
o.new_inner().__class__ == o.new_inner().__class__ True

... but that class isn't the inner class ...

... but that class isn't the inner class ...  >>>
o.new_inner().__class__ != Outer.Inner True

... but a subclass of the inner class.

... but a subclass of the inner class.  >>> isinstance(
o.new_inner(), Outer.Inner ) True

Static and class methods, e.g. should work, too ✨:

>>> o.Inner.new_inner().outer == o
True

Members:

• inner_class
• __get__(instance, owner)
• __call__(**kwargs)

---

✨ toil.lib.pipes ✨

Attributes 🌸

┌─────┬───┐
│ log │   │
└─────┴───┘

Classes 🦋

┌──────────────────────────┬────────────────────────────┐
│ WritablePipe             │ An object-oriented wrapper │
│                          │ for os.pipe. Clients       │
│                          │ should subclass it,        │
│                          │ implement                  │
├──────────────────────────┼────────────────────────────┤
│ ReadablePipe             │ An object-oriented wrapper │
│                          │ for os.pipe. Clients       │
│                          │ should subclass it,        │
│                          │ implement                  │
├──────────────────────────┼────────────────────────────┤
│ ReadableTransformingPipe │ A pipe which is            │
│                          │ constructed around a       │
│                          │ readable stream, and which │
│                          │ provides a                 │
├──────────────────────────┼────────────────────────────┤
│ HashingPipe              │ Class which checksums all  │
│                          │ the data read through it.  │
│                          │ If it                      │
└──────────────────────────┴────────────────────────────┘

Module Contents 🎀

toil.lib.pipes.log

class toil.lib.pipes.WritablePipe(encoding=None, errors=None)

Bases: abc.ABC (see /usr/share/doc/python3/html/library/abc.html#abc.ABC) 🍓

So this is an object-oriented wrapper for os.pipe 💖. You subclass it, implement readFrom() to consume the readable end of the pipe, and then use the class as a context manager to get the writable end. Here's the example 👇.

>>> import sys, shutil, codecs
>>> class MyPipe(WritablePipe):
...     def readFrom(self, readable):
...         shutil.copyfileobj(codecs.getreader('utf-8')(readable), sys.stdout)
>>> with MyPipe() as writable:
...     _ = writable.write('Hello, world!\n'.encode('utf-8'))
Hello, world!

Each instance of this class creates a thread and invokes the readFrom method in that thread 🧵. The thread gets join()ed on a normal exit from the context manager, which means the body of the with statement. If an exception happens, the thread won't be joined, but a well-behaved readFrom() implementation should wrap up shortly after, because the pipe got closed 🔒.

Now, exceptions in the reader thread get reraised in the main thread 😱:

>>> class MyPipe(WritablePipe):
...     def readFrom(self, readable):
...         raise RuntimeError('Hello, world!')
>>> with MyPipe() as writable:
...     pass
Traceback (most recent call last):
...
RuntimeError: Hello, world!

More complicated, less illustrative tests 🙈:

Same as above, but proving that handles are closed:

>>> x = os.dup(0); os.close(x)
>>> class MyPipe(WritablePipe):
...     def readFrom(self, readable):
...         raise RuntimeError('Hello, world!')
>>> with MyPipe() as writable:
...     pass
Traceback (most recent call last):
...
RuntimeError: Hello, world!
>>> y = os.dup(0); os.close(y); x == y
True

Exceptions in the body of the with statement aren't masked, and handles are closed 🫶:

>>> x = os.dup(0); os.close(x)
>>> class MyPipe(WritablePipe):
...     def readFrom(self, readable):
...         pass
>>> with MyPipe() as writable:
...     raise RuntimeError('Hello, world!')
Traceback (most recent call last):
...
RuntimeError: Hello, world!
>>> y = os.dup(0); os.close(y); x == y
True

Parameters

• encoding (Optional[str]) — the text encoding you want, if any 💖
• errors (Optional[str]) — what to do when encoding goes wrong, if you want to set it 🫶

encoding: str | None = None
errors: str | None = None
readable_fh: int | None = None
writable: IO[Any] | None = None
thread: toil.lib.threading.ExceptionalThread | None = None
reader_done: bool = False

__enter__() ✨

Return type: IO[Any] 🦋

__exit__(exc_type, exc_val, exc_tb) 💅

Parameters:

• exc_type (Optional[str]) 🌸
• exc_val (Optional[str]) 🌸
• exc_tb (Optional[str]) 🌸

Return type: None 🙈

abstractmethod readFrom(readable) 🔥

okay so this one is an abstract method, which means you have to write it yourself if you subclass 😭 Its whole job is to pull data out of the pipe. It needs to handle both binary and text mode output, so no being picky about it 💜

Parameters:

• readable (file) — this is the file object for the readable end of the pipe. Do not call close() on it yourself 🙅 It gets closed automatically, so let it go 🫶

Return type: None ✨

---

✨ class toil.lib.pipes.ReadablePipe(encoding=None, errors=None) ✨

Bases: abc.ABC

Okay bestie, ReadablePipe is an object-oriented wrapper around os.pipe 🌈 The vibe is: you subclass it, write a writeTo() method that puts data into the writable end of the pipe, and then use the class as a context manager to get the writable end back 💅 Here's a lil example so you can see it in action 🦋

>>> import sys, shutil, codecs
>>> class MyPipe(ReadablePipe):
...     def writeTo(self, writable: IO[Any]) -> None:
...         writable.write('Hello, world!\n'.encode('utf-8'))
>>> with MyPipe() as readable:
...     shutil.copyfileobj(codecs.getreader('utf-8')(readable), sys.stdout)
Hello, world!

Every instance spins up its own thread and runs writeTo() inside it 🧵 On a normal exit from the context manager (aka the body of the with statement), that thread gets join()ed, so it's all tidy 💖 If an exception happens instead, the thread does not get joined, but a well-behaved writeTo() should wrap up pretty quickly anyway, since the pipe has been closed 🙈

Now, the main thread will get any exceptions that happen in the reader thread and re-raise them. Literally iconic 👑

>>> class MyPipe(ReadablePipe):
...     def writeTo(self, writable):
...         raise RuntimeError('Hello, world!')
>>> with MyPipe() as readable:
...     pass
Traceback (most recent call last):
...
RuntimeError: Hello, world!

Next up are some more complicated, less illustrative tests 🧐 Same as the one above, but this one proves that the handles get closed 🔒

>>> x = os.dup(0); os.close(x)
>>> class MyPipe(ReadablePipe):
...     def writeTo(self, writable: IO[Any]) -> None:
...         raise RuntimeError('Hello, world!')
>>> with MyPipe() as readable:
...     pass
Traceback (most recent call last):
...
RuntimeError: Hello, world!
>>> y = os.dup(0); os.close(y); x == y
True

Exceptions raised in the body of the with statement aren't masked, and the handles still get closed 🫶

>>> x = os.dup(0); os.close(x)
>>> class MyPipe(ReadablePipe):
...     def writeTo(self, writable):
...         pass
>>> with MyPipe() as readable:
...     raise RuntimeError('Hello, world!')
Traceback (most recent call last):
...
RuntimeError: Hello, world!
>>> y = os.dup(0); os.close(y); x == y
True

Parameters:

• encoding (Optional[str]) 🍓
• errors (Optional[str]) 🍓

abstractmethod writeTo(writable) 💜

This one is also abstract, so you gotta implement it 🔥 Its job is to write data into the pipe, and it needs to support both binary and text mode input 🦋

Parameters:

• writable (file) — the file object for the writable end of the pipe. Do not explicitly call close() on it, because that happens automatically 🙅

Return type: None 💅

encoding: str | None = None
errors: str | None = None
writable_fh: int | None = None
readable: IO[Any] | None = None
thread: toil.lib.threading.ExceptionalThread | None = None

__enter__() 🌸

Return type: IO[Any] ✨

__exit__(exc_type, exc_val, exc_tb) 🌈

Parameters:

• exc_type (Optional[str]) 🌸
• exc_val (Optional[str]) 🌸
• exc_tb (Optional[str]) 🌸

Return type: None 🙈

---

✨ class toil.lib.pipes.ReadableTransformingPipe(source, encoding=None, errors=None) ✨

Bases: ReadablePipe

This is a pipe that gets built around a readable stream, and it gives you a context manager that hands back a readable stream 🦋 It's super useful as a base class for pipes that need to transform or visit the bytes flowing through them, instead of just consuming or producing data 💖

Clients should subclass it and implement transform(), like this:

>>> import sys, shutil, codecs
>>> class MyPipe(ReadableTransformingPipe):
...     def transform(self, readable, writable):
...         writable.write(readable.read().decode('utf-8').upper().encode('utf-8'))
>>> class SourcePipe(ReadablePipe):
...     def writeTo(self, writable):
...         writable.write('Hello, world!\n'.encode('utf-8'))
>>> with SourcePipe() as source:
...     with MyPipe(source) as transformed:
...         shutil.copyfileobj(codecs.getreader('utf-8')(transformed), sys.stdout)
HELLO, WORLD!

The transform() method runs in its own thread 🧵 It should move data chunk by chunk instead of grabbing everything all at once, bestie 🙏 It should finish normally if it hits either an EOF on the readable side, or a BrokenPipeError on the writable side 💅

Make sure you actually catch a BrokenPipeError when you're writing! If you don't, your transform() won't wrap up cleanly 😭

See also: toil.lib.misc.WriteWatchingStream. 👀

Parameters:

• source (IO[Any]) 🌷
• encoding (Optional[str]) 🍓
• errors (Optional[str]) 🍓

source 🫧

abstractmethod transform(readable, writable) 🔥

You have to implement this one to actually ship data through the pipe 🚢

Parameters:

• readable (file) — the input stream file object you want to transform 🌊
• writable (file) — the file object for the writable end of the pipe. Do not explicitly call close() on it, since that happens automatically 🙅

Return type: None 💜

writeTo(writable) 🦋

Implement this method to write data from the pipe 💅 It should support both binary and text mode input.

Parameters:

• writable (file) — the file object for the writable end of the pipe. Do not explicitly call close() on it, because that happens automatically 🙅

Return type: None ✨

---

✨ class toil.lib.pipes.HashingPipe(source, encoding=None, errors=None, checksum_to_verify=None) ✨

Bases: ReadableTransformingPipe

This class checksums all the data that gets read through it 🔐 If it reaches EOF and the checksum isn't correct, it raises ChecksumError 😬 Heads up: it assumes info actually has a checksum 🫶

Parameters:

• source (IO[Any]) 🌷
• encoding (Optional[str]) 🍓
• errors (Optional[str]) 🍓
• checksum_to_verify (Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str) 🧾 this one can be None, so you don't have to bring a checksum if you don't want to 🫶

checksum_to_verify = None

transform(readable, writable)

Okay so this is the method YOU write, bestie 🚰✨ Implement it to ship data through the pipe, and it's giving main character energy.

• readable (file) — the input stream file object you're transforming 📥
• writable (file) — the file object for the writable end of the pipe 🖊️

🙅 Do not explicitly invoke the close() method of the object. It gets closed automatically, so don't even try 💀

Return type: None 💖

---

toil.lib.plugins

Generic plugin system for Toil plugins 🧚 Plugins come in Python packages named like this:

toil_{PLUGIN_TYPE}_{WHATEVER}

When Toil goes looking for plugins, it lists every Python package with the right name prefix for the given type of plugin, then loads them. The plugin modules then get the chance to import register_plugin() and register themselves 🦋

Attributes

• PluginType 🎀
• plugin_types 🎀

Functions

• register_plugin(plugin_type, plugin_name, ...) — adds a plugin to the registry for the given type of plugin 📝
• remove_plugin(plugin_type, plugin_name) — removes a plugin from the registry for the given type of plugin 🗑️
• get_plugin_names(plugin_type) — gets the names of all the available plugins of the given type 📋
• get_plugin(plugin_type, plugin_name) — gets a plugin class factory function by name 🔍

Module Contents

• toil.lib.plugins.PluginType 🎀

The list of plugin types is literally this, so iconic 👑

toil.lib.plugins.plugin_types: list[PluginType] = ['batch_system', 'url_access']

Here list refers to the one in /usr/share/doc/python3/html/library/stdtypes.html#list 📚

toil.lib.plugins.register_plugin(plugin_type, plugin_name, plugin_being_registered)

Adds a plugin to the registry for the given type of plugin 💅

Parameters:

• plugin_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — For batch systems, this is the string the user types to pick the batch system on the command line with --batchSystem. For URL access plugins, this is the URL scheme the plugin implements 🌈
• plugin_being_registered (Any) — a function that, when called, imports and returns a plugin-provided class type. For batch systems, that type must extend toil.batchSystems.abstractBatchSystem.AbstractBatchSystem. For URL access plugins, it must extend toil.lib.url.URLAccess. Note that this function should return the class itself. It should NOT construct an instance of the class 🙈
• plugin_type (PluginType) 🎀

Return type: None ✨

toil.lib.plugins.remove_plugin(plugin_type, plugin_name)

Removes a plugin from the registry for the given type of plugin 🧹

Parameters:

• plugin_type (PluginType) 🎀
• plugin_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀

Return type: None 💜

toil.lib.plugins.get_plugin_names(plugin_type)

Gets the names of all the available plugins of the given type 📋

Parameters:

• plugin_type (PluginType) 🎀

Return type: list[str], where list is in /usr/share/doc/python3/html/library/stdtypes.html#list and str is in /usr/share/doc/python3/html/library/stdtypes.html#str 🧾

toil.lib.plugins.get_plugin(plugin_type, plugin_name)

Gets a plugin class factory function by name 🔍

🚨 Raises KeyError if plugin_name isn't the name of a plugin of the given type. Careful with your spelling, bestie 😭

Parameters:

• plugin_type (PluginType) 🎀
• plugin_name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀

Return type: Any 🦋

---

toil.lib.resources

Classes

• ResourceMonitor — global resource monitoring widget 📊

Functions

• glob(glob_pattern, directoryname) — walks through a directory and its subdirectories looking for files matching 🔎

Module Contents

class toil.lib.resources.ResourceMonitor

Global resource monitoring widget 📊 It gives you class methods to check the resource usage of this process and its child processes. It also has other class methods to tweak the stats, so they can count resources used inside containers, or anything else that should be billable to the current process 💖

classmethod record_extra_memory(peak_ki)

Become responsible for the given peak memory usage, in kibibytes 🧠 The memory gets treated as if a child process used it at the same time our real child processes were using their peak memory.

Parameters:

• peak_ki (int, see /usr/share/doc/python3/html/library/functions.html#int) 🔢

Return type: None 💅

classmethod record_extra_cpu(seconds)

Become responsible for the given CPU time ⏱️ The CPU time gets treated as if it had been used by a child process.

Parameters:

• seconds (float, see /usr/share/doc/python3/html/library/functions.html#float) ⏱️

Return type: None 🌸

classmethod get_total_cpu_time_and_memory_usage()

Gives you the total CPU time of itself and all its children, plus the maximum RSS memory usage of itself and its single largest child (in kibibytes) 📈

Return type: tuple[float, int], where tuple is in /usr/share/doc/python3/html/library/stdtypes.html#tuple, float is in /usr/share/doc/python3/html/library/functions.html#float, and int is in /usr/share/doc/python3/html/library/functions.html#int 🧾

classmethod get_total_cpu_time()

Gives you the total CPU time, children included ⏰

Return type: float, see /usr/share/doc/python3/html/library/functions.html#float 💜

toil.lib.resources.glob(glob_pattern, directoryname)

Walks through a directory and its subdirectories looking for files that match the glob_pattern, then returns a list=[] 🗂️

Parameters:

• directoryname (toil.lib.misc.StrPath) — any accessible folder name on the filesystem 📁
• glob_pattern (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — a string like *.txt, which would find all the text files 📝

Returns: a list=[] of absolute filepaths matching the glob pattern 🌟

Return type: list[str], where list is in /usr/share/doc/python3/html/library/stdtypes.html#list and str is in /usr/share/doc/python3/html/library/stdtypes.html#str ✨

---

toil.lib.retry

This file holds the retry() decorator function and the RetryCondition object 🔁

retry() can decorate any function based on the list of errors you want it to retry on. That list can hold normal Exception objects, and/or RetryCondition objects that wrap Exceptions to add extra conditions 🎯

For example, retrying on one Exception (HTTPError):

from toil.lib.web import web_session
from requests.exceptions import HTTPError

@retry(errors=[HTTPError])
def update_my_wallpaper():
    return web_session.get('https://www.deviantart.com/')

Or:

from toil.lib.web import web_session
from requests.exceptions import HTTPError

@retry(errors=[HTTPError, ValueError])
def update_my_wallpaper():
    return web_session.get('https://www.deviantart.com/')

The examples above will retry for the default interval on any errors you list in the "errors=" arg list 🌈

okay so, to retry on specifically 500/502/503/504 errors, you can swap in an ErrorCondition object instead, and it's giving precision 💅

from toil.lib.web import web_session
from requests.exceptions import HTTPError

@retry(errors=[
    ErrorCondition(
               error=HTTPError,
               error_codes=[500, 502, 503, 504]
           )])
def update_my_wallpaper():
    return web_session.get('https://www.deviantart.com/')

bestie, if you only want to retry on errors containing the phrase "NotFound" 🙈

from toil.lib.web import web_session
from requests.exceptions import HTTPError

@retry(errors=[
    ErrorCondition(
        error=HTTPError,
        error_message_must_include="NotFound"
    )])
def update_my_wallpaper():
    return web_session.get('https://www.deviantart.com/')

and for retrying on ALL HTTPError errors EXCEPT the ones containing "NotFound" 🚫, this is the one 👑

from toil.lib.web import web_session
from requests.exceptions import HTTPError

@retry(errors=[
    HTTPError,
    ErrorCondition(
               error=HTTPError,
               error_message_must_include="NotFound",
               retry_on_this_condition=False
           )])
def update_my_wallpaper():
    return web_session.get('https://www.deviantart.com/')

and if you want to retry on boto3's specific status errors ☁️ it looks something like this:

import boto3
from botocore.exceptions import ClientError

@retry(errors=[
    ErrorCondition(
               error=ClientError,
               boto_error_codes=["BucketNotFound"]
           )])
def boto_bucket(bucket_name):
    boto_session = boto3.session.Session()
    s3_resource = boto_session.resource('s3')
    return s3_resource.Bucket(bucket_name)

Mix and match all you want 🌈, just make sure each code is paired with the correct exception, bc a ValueError is never gonna hand you a 404, for example 💀

the retry function as a decorator makes retrying functions way easier and clearer 🦋 It also pushes you toward smaller independent functions, instead of lumping a bunch of stuff that needs different retry conditions into one giant function 🧩

the ErrorCondition object does the heavy lifting of writing specific retry conditions and boils it all down to an API that covers every common use-case, so you never have to write a bespoke function 🫶

Use-cases covered rn 📋

1. Retrying on a normal error, like a KeyError.
2. Retrying on HTTP error codes (use ErrorCondition).
3. Retrying on boto 3's specific status errors, like "BucketNotFound" (use ErrorCondition).
4. Retrying when an error message contains a certain phrase (use ErrorCondition).
5. Explicitly NOT retrying on a condition (use ErrorCondition).

If you need new functionality, it's currently best practice in Toil to add it to the ErrorCondition itself rather than making a brand new custom retry method 💖

---

✨ Attributes ✨

• SUPPORTED_HTTP_ERRORS
• kubernetes
• botocore
• logger
• RT
• DEFAULT_DELAYS
• DEFAULT_TIMEOUT
• E
• retry_flaky_test

---

🧚 Classes 🧚

• ErrorCondition — a wrapper describing an error condition 🎀

---

🌸 Functions 🌸

• retry([intervals, infinite_retries, errors, ...]) — retry a function if it fails with any Exception defined in errors 🔁
• return_status_code(e)
• get_error_code(e) — get the error code name from a Boto 2 or 3 error, or compatible types 🔍
• get_error_message(e) — get the error message string from a Boto 2 or 3 error, or compatible types 💬
• get_error_status(e) — get the HTTP status code from a compatible source 🔢
• get_error_body(e) — get the body from a Boto 2 or 3 error, or compatible types 📦
• meets_error_message_condition(e, error_message)
• meets_error_code_condition(e, error_codes) — these are expected to be normal HTTP error codes, like 404 or 500 🌡️
• meets_boto_error_code_condition(e, boto_error_codes) — these are expected to be AWS's custom error aliases, like 'BucketNotFound' or 'AccessDenied' ☁️
• error_meets_conditions(e, error_conditions)
• old_retry([delays, timeout, predicate]) — deprecated 💀

---

🦋 Module Contents 🦋

• toil.lib.retry.SUPPORTED_HTTP_ERRORS
• toil.lib.retry.kubernetes = None
• toil.lib.retry.botocore = None
• toil.lib.retry.logger

class toil.lib.retry.ErrorCondition(error=None, error_codes=None, boto_error_codes=None, error_message_must_include=None, retry_on_this_condition=True)

A wrapper describing an error condition.

ErrorCondition events may be used to define errors in more detail, so you can decide whether to retry 🌈

🎀 Parameters 🎀

• error (Optional[type </usr/share/doc/python3/html/library/functions.html#type>[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>]])
• error_codes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>])
• boto_error_codes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])
• error_message_must_include (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• retry_on_this_condition (bool, /usr/share/doc/python3/html/library/functions.html#bool)

Okay and the defaults for ErrorCondition are giving 💅. Everything starts as None except the big one, retry_on_this_condition, which is True by default 🫶:

error = None
error_codes = None
boto_error_codes = None
error_message_must_include = None
retry_on_this_condition = True

toil.lib.retry.RT

This one is just listed with no extra notes 🦋.

toil.lib.retry.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)

Okay this one is the main character 👑. It retries a function if it fails with any Exception defined in errors. It does this every x seconds, where x comes from a list of numbers (ints or floats) in intervals. It also accepts ErrorCondition events for more detailed retry attempts 💖.

Parameters 🎀

• intervals (Optional[list], /usr/share/doc/python3/html/library/stdtypes.html#list) — A list of times in seconds where we keep retrying until we return a failure. By default it uses exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s 🍓
• infinite_retries (bool, /usr/share/doc/python3/html/library/functions.html#bool) — If this is True, the intervals reset once they run out. Defaults to False 🌸
• errors (Optional[collections.abc.Sequence[Union[ErrorCondition, type[Exception]]]], /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Sequence, /usr/share/doc/python3/html/library/functions.html#type, /usr/share/doc/python3/html/library/exceptions.html#Exception) — A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies:
  • Exception (required)
  • Error codes that must match to be retried (optional; defaults to not checking)
  • A string that must be in the error message to be retried (optional; defaults to not checking)
  • A bool that can be set to False to always error on this condition.

If not specified, this will default to a generic Exception 🧚

• log_message (Optional[tuple[Callable, str]], /usr/share/doc/python3/html/library/stdtypes.html#tuple, /usr/share/doc/python3/html/library/stdtypes.html#str) — Optional tuple of ("log/print function()", "message string") that will print before each attempt 📝
• prepare (Optional[list[Callable]], /usr/share/doc/python3/html/library/stdtypes.html#list) — Optional list of functions to call, with the function's arguments, between retries to reset state 🔄

Returns

The result of the wrapped function, or it raises ✨

Return type

Callable[[Callable[Ellipsis, RT]], Callable[Ellipsis, RT]]

toil.lib.retry.return_status_code(e)

Just the name and signature here, no extra deets 🌸.

toil.lib.retry.get_error_code(e)

Okay so this one grabs the error code name from a Boto 2 or 3 error, or compatible types 🔍. For any other errors, it gives back an empty string 🫥.

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)

Return type

str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.retry.get_error_message(e)

This one pulls the error message string from a Boto 2 or 3 error, or compatible types 💬.

Heads up, bestie: error message conditions check more than this. This function does not fall back to the traceback for incompatible types 🙈

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)

Return type

str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.retry.get_error_status(e)

This one gets the HTTP status code from a compatible source 🌐. That includes things like a Boto 2 or 3 error, kubernetes.client.rest.ApiException, http.client.HTTPException, urllib3.exceptions.HTTPError, requests.exceptions.HTTPError, urllib.error.HTTPError, or a compatible type 💜. For other errors, it returns 0 0️⃣.

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)

Return type

int (/usr/share/doc/python3/html/library/functions.html#int)

toil.lib.retry.get_error_body(e)

Grabs the body from a Boto 2 or 3 error, or compatible types 📦. If the error doesn't have a body, it returns the code and message instead 🫶.

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)

Return type

str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.retry.meets_error_message_condition(e, error_message)

No description here, just the parameters 🌷.

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)
• error_message (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.retry.meets_error_code_condition(e, error_codes)

These are expected to be normal HTTP error codes, like 404 or 500 🔢.

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)
• error_codes (Optional[list[int]], /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/functions.html#int)

toil.lib.retry.meets_boto_error_code_condition(e, boto_error_codes)

These are expected to be AWS's custom error aliases, like 'BucketNotFound' or 'AccessDenied' ☁️.

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)
• boto_error_codes (Optional[list[str]], /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.lib.retry.error_meets_conditions(e, error_conditions)

Just the signature, no notes 🦋.

toil.lib.retry.DEFAULT_DELAYS = (0, 1, 1, 4, 16, 64)

A constant with the default delays, it's cute 🌈.

toil.lib.retry.DEFAULT_TIMEOUT = 300

A constant with the default timeout, 300 🕒.

toil.lib.retry.E

Just listed here, no extra notes ✨.

toil.lib.retry.old_retry(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=lambda e: ...)

Deprecated. This one's on its way out, so be careful using it 💀

Okay so this retries an operation while the failure matches a given predicate. It keeps going until a given timeout expires, waiting a set amount of time between attempts. It's a generator that yields context managers. The doctests below show example usage 🎀.

Parameters

• delays (Iterable[float], /usr/share/doc/python3/html/library/functions.html#float) — An iterable yielding the time in seconds to wait before each retried attempt. The last element of the iterable will be repeated 🔁
• timeout (float, /usr/share/doc/python3/html/library/functions.html#float) — An overall timeout that should not be exceeded for all attempts together. This is a best-effort mechanism only. It won't abort an ongoing attempt, even if the timeout expires during that attempt ⏳
• predicate (Callable[[Exception],bool], /usr/share/doc/python3/html/library/exceptions.html#Exception, /usr/share/doc/python3/html/library/functions.html#bool) — A unary callable that returns True if another attempt should be made to recover from the given exception. The default value for this parameter will prevent any retries! 🚫

Returns

A generator yielding context managers, one per attempt ✨

Return type

Iterator

Retry for a limited amount of time 🕰️:

>>> true = lambda _:True
>>> false = lambda _:False
>>> i = 0
>>> for attempt in old_retry( delays=[0], timeout=.1, predicate=true ):
...     with attempt:
...         i += 1
...         raise RuntimeError('foo')
Traceback (most recent call last):
...
RuntimeError: foo
>>> i > 1
True

If timeout is 0, you get exactly one attempt 1️⃣:

>>> i = 0
>>> for attempt in old_retry( timeout=0 ):
...     with attempt:
...         i += 1
...         raise RuntimeError( 'foo' )
Traceback (most recent call last):
...
RuntimeError: foo
>>> i
1

Don't retry on success 🙅‍♀️:

>>> i = 0
>>> for attempt in old_retry( delays=[0], timeout=.1, predicate=true ):
...     with attempt:
...         i += 1
>>> i
1

Don't retry unless the predicate returns True 💅:

>>> i = 0
>>> for attempt in old_retry( delays=[0], timeout=.1, predicate=false):
...     with attempt:
...         i += 1
...         raise RuntimeError( 'foo' )
Traceback (most recent call last):
...
RuntimeError: foo
>>> i
1

toil.lib.retry.retry_flaky_test

Just listed here, no extra notes 🦋.

---

toil.lib.threading

Attributes

• logger 📣
• current_process_name_lock 🔒
• current_process_name_for 🧵

✨ Classes ✨

okay so this module has two classes and both are main character material 🦋

• ExceptionalThread — a thread whose join() method re-raises exceptions raised during run(). While join() is 🌸
• LastProcessStandingArena — a class that lets a whole crew of processes detect each other and elect a last process standing 👑

---

✨ Functions ✨

bestie these are the helper functions and they're giving organized 💅

• ensure_filesystem_lockable(path[, timeout, hint]) — checks that the filesystem at the given path is one where locks are actually safe to use 🔒
• safe_lock(fd[, block, shared]) — grabs an fcntl lock and keeps retrying on IO errors so it doesn't give up on you 💖
• safe_unlock_and_close(fd) — releases an fcntl lock and closes the file descriptor, handling fcntl IO errors while it's at it 🧹
• cpu_count() — gets the rounded-up integer number of whole CPUs available ⭐
• collect_process_name_garbage() — deletes all the process names that point to files that don't exist anymore 🗑️
• destroy_all_process_names() — deletes all of our process name files because our process is about to leave the chat 🫶
• get_process_name(base_dir) — returns the name of the current process. Like a PID, but visible between 🧚
• process_name_exists(base_dir, name) — returns true if the process named by the given name (from process_name) exists, and false otherwise 🔍
• global_mutex(base_dir, mutex) — a context manager that locks a mutex. The mutex is identified by the given 🎀

---

✨ Module Contents ✨

okay the full tea is here, let's go 🍓

• toil.lib.threading.logger — the logger for this module 🌈

🔒 ensure_filesystem_lockable

toil.lib.threading.ensure_filesystem_lockable(path, timeout=30, hint=None)

bestie, this one is the gatekeeper 🚪 It makes sure the filesystem at the given path is one where locks are safe to use.

File locks are not safe on Ceph. 💀 See https://github.com/DataBiosphere/toil/issues/4972.

It raises an exception if the filesystem is detected as one where using locks is known to trigger bugs in the filesystem implementation. It also raises if the given path does not exist, or if figuring out the filesystem type takes longer than the timeout (in seconds). 😭

If the filesystem type can't be determined, it does nothing. no drama 🌸

Parameters

• hint (Optional[str], </usr/share/doc/python3/html/library/stdtypes.html#str>) — extra text to include in an error, if one is raised, telling the user how to change the offending path 💌
• path (toil.lib.misc.StrPath) 🛤️
• timeout (float, </usr/share/doc/python3/html/library/functions.html#float>) ⏱️

Return type: None

🔒 safe_lock

toil.lib.threading.safe_lock(fd, block=True, shared=False)

Gets an fcntl lock and retries on IO errors, so it's super persistent 💪

It raises OSError with EACCES or EAGAIN when a nonblocking lock is not immediately available. 🚨

Parameters

• fd (int, </usr/share/doc/python3/html/library/functions.html#int>) 🔢
• block (bool, </usr/share/doc/python3/html/library/functions.html#bool>) 🚧
• shared (bool, </usr/share/doc/python3/html/library/functions.html#bool>) 🤝

Return type: None

🔓 safe_unlock_and_close

toil.lib.threading.safe_unlock_and_close(fd)

Releases an fcntl lock and closes the file descriptor, and handles fcntl IO errors while doing it. clean exit only ✨

Parameters

• fd (int, </usr/share/doc/python3/html/library/functions.html#int>) 🔢

Return type: None

🧵 class ExceptionalThread

class toil.lib.threading.ExceptionalThread(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None, context=None)

Bases: threading.Thread (</usr/share/doc/python3/html/library/threading.html#threading.Thread>)

okay this is the one I'm obsessed with 😭 A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception only comes up during the first call to join() that successfully joins the thread. If join() times out, no exception gets re-raised, even if one already happened inside run(). 💅

When you subclass this thread, override tryRun() instead of run(). ✨

Here's the example in the docs, verbatim:

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Attributes

• exc_info = None 🧾

Methods

• run() — "Method representing the thread's activity." You can override this in a subclass. The standard run() invokes the callable object passed to the object's constructor as the target argument (if any), with sequential (positional) and keyword arguments taken from the args and kwargs arguments, respectively. 🏃‍♀️
  • Return type: None
• tryRun() 🫧
  • Return type: None
• join(*args, **kwargs) — waits until the thread terminates. This blocks the calling thread until the thread whose join() is called terminates, either normally, through an unhandled exception, or until the optional timeout occurs. ⏳
  • When the timeout argument is present and not None, it should be a floating-point number of seconds (or fractions of one). Since join() always returns None, you must call is_alive() after join() to tell whether a timeout happened. If the thread is still alive, the join() call timed out. 🫠
  • When timeout is missing or None, it blocks until the thread terminates. 🧱
  • A thread can be join()ed many times. 🔁
  • join() raises a RuntimeError if you try to join the current thread, because that would cause a deadlock. 💀 It's also an error to join() a thread before it has been started, and trying that raises the same exception. 🚫
  • Parameters
    • args (Optional[float], </usr/share/doc/python3/html/library/functions.html#float>) 🎈
    • kwargs (Optional[float], </usr/share/doc/python3/html/library/functions.html#float>) 🎈
  • Return type: None

🧮 cpu_count

toil.lib.threading.cpu_count()

Gets the rounded-up integer number of whole CPUs available. ⭐

Hyperthreads count as CPUs here, no cap 🧠

It uses the system's actual CPU count, or the current v1 cgroup's quota per period if the quota is set. 🎀

Okay so this one ignores the cgroup's cpu shares value 🙈 because it's extremely difficult to interpret (same, bestie). If you want the drama, see https://github.com/kubernetes/kubernetes/issues/81021 🌸

It also stashes the result so it's not redoing the work every time 🧠✨

Returns: an integer count of available CPUs, minimum 1 💖

Return type: int

---

🔒 toil.lib.threading.current_process_name_lock

Just a name on the list, no extra notes here 🫶

📇 toil.lib.threading.current_process_name_for

Type: dict[str, str]

🗑️ toil.lib.threading.collect_process_name_garbage()

This one deletes every process name that points to a file that doesn't exist anymore, bc the work directory was temporary and got cleaned up 🧹. It's known to happen during the tests, which get their own temp directories, so it's very much a real thing 💅

🚨 Caller must hold current_process_name_lock.

Return type: None

👋 toil.lib.threading.destroy_all_process_names()

Okay this one deletes all our process name files because our process is going away 😭. We let all our FDs get closed by the process death, which is honestly so iconic of it. We assume there is nobody else using the system during exit to race with 🏃‍♀️💨

Return type: None

🧚 toil.lib.threading.get_process_name(base_dir)

Returns the name of the current process. It's like a PID, but it's visible between containers on what Toil thinks is a node 👑

Parameters

• base_dir (str) — Base directory to work in. Defines the shared namespace.

Returns: Process's assigned name

Return type: str

👀 toil.lib.threading.process_name_exists(base_dir, name)

Returns True if the process named by the given name (from process_name) exists, and False otherwise. It can see across container boundaries using the given node workflow directory, which is so main character energy 🎀

Parameters

• base_dir (str) — Base directory to work in. Defines the shared namespace.
• name (str) — Process's name to poll.

Returns: True if the named process is still alive, and False otherwise.

Return type: bool

🔐 toil.lib.threading.global_mutex(base_dir, mutex)

A context manager that locks a mutex 🔐. The mutex is identified by the given name and scoped to the given directory. It works across all containers that have access to the given directory, and mutexes held by dead processes are automatically released 🫠

⚠️ Only works between processes, NOT between threads.

Parameters

• base_dir (toil.lib.misc.StrPath) — Base directory to work in. Defines the shared namespace.
• mutex (str) — Mutex to lock. Must be a permissible path component.

Return type: collections.abc.Iterator[None]

🏆 class toil.lib.threading.LastProcessStandingArena(base_dir, name)

This class lets a bunch of processes detect and elect a last process standing 👑. Processes enter and leave, sometimes due to sudden existence failure 💀. We guarantee that the last process to leave, if it leaves properly, gets a chance to do some cleanup. If new processes try to enter during the cleanup, they get delayed until after the cleanup has happened and the previous "last" process has finished leaving. Very patient of them 🫶

🚨 The user is responsible for making sure you always leave if you enter! Consider using a try/finally; this class is not a context manager.

Parameters

• base_dir (toil.lib.misc.StrPath)
• name (str)

Attributes

• base_dir = ''
• mutex
• lockfileDir
• lockfileFD = None
• lockfileName = None

Methods

enter()

This process is entering the arena. If cleanup is in progress, it blocks until the cleanup is finished 🚪. You may not enter the arena again before leaving it.

Return type: None

leave()

This process is leaving the arena. If this process happens to be the last process standing, it yields something, and other processes are blocked from joining the arena until the loop body completes and the process has finished leaving. Otherwise, it doesn't yield anything 🌈

It should be used in a loop, like this:

for _ in arena.leave():
    # If we get here, we were the last process. Do the cleanup pass

Return type: collections.abc.Iterator[bool]

---

🦋 toil.lib.throttle

Classes

• LocalThrottle — A thread-safe rate limiter that throttles each thread independently. Can be used as a
• throttle — A context manager for ensuring that the execution of its body takes at least a given amount

Module Contents

🐢 class toil.lib.throttle.LocalThrottle(min_interval)

A thread-safe rate limiter that throttles each thread independently 🧵. It can be used as a function or method decorator, or as a simple object via its .throttle() method ✨

🚨 The use as a decorator is deprecated in favor of throttle().

Parameters

• min_interval (int)

Attributes

• min_interval
• per_thread

Methods

throttle(wait=True)

If the wait parameter is True, this method returns True after suspending the current thread as necessary to ensure that no less than the configured minimum interval has passed since the last invocation of this method in the current thread returned True 🕰️.

If the wait parameter is False, this method immediately returns True (if at least the configured minimum interval has passed since the last time this method returned True in the current thread) or False otherwise.

Parameters

• wait (bool)

Return type: bool

__call__(function)

🦋 class toil.lib.throttle.throttle(min_interval)

A context manager for ensuring that the execution of its body takes at least a given amount of time, sleeping if necessary 😴. It's a simpler version of LocalThrottle if used as a decorator 💅

It ensures that the body takes at least the given amount of time.

>>> start = time.time()
>>> with throttle(1):
...     pass
>>> 1 <= time.time() - start <= 1.1
True

Ditto when used as a decorator.

>>> @throttle(1)
... def f():
...     pass
>>> start = time.time()
>>> f()
>>> 1 <= time.time() - start <= 1.1
True

If the body takes longer by itself, don't throttle 🙅‍♀️.

>>> start = time.time()
>>> with throttle(1):
...     time.sleep(2)
>>> 2 <= time.time() - start <= 2.1
True

Ditto when used as a decorator.

>>> @throttle(1)
... def f():
...     time.sleep(2)
>>> start = time.time()
>>> f()
>>> 2 <= time.time() - start <= 2.1
True

If an exception occurs, don't throttle 💀.

>>> start = time.time()
>>> try:
...     with throttle(1):
...         raise ValueError('foo')
... except ValueError:
...     end = time.time()
...     raise
Traceback (most recent call last):
...
ValueError: foo
>>> 0 <= end - start <= 0.1
True

Ditto when used as a decorator.

>>> @throttle(1)
... def f():
...     raise ValueError('foo')
>>> start = time.time()
>>> try:
...     f()
... except ValueError:
...     end = time.time()
...     raise
Traceback (most recent call last):
...
ValueError: foo
>>> 0 <= end - start <= 0.1
True

Parameters

• min_interval (Union[int, float])

Attributes

• min_interval

Methods

__enter__()

__exit__(exc_type, exc_val, exc_tb)

__call__(function)

✨ toil.lib.trs ✨

okay bestie, this module is all about TRS 🦋 It's how Toil hooks into GA4GH Tool Registry Service servers so you can go fetch workflows straight from them. literally main character energy 💅

💖 Attributes

• logger
• TRS_ROOT

🧚 Functions

• is_trs_workflow(workflow) — checks if a workflow string smells like TRS 👃
• extract_trs_spec(workflow) — turns a Dockstore workflow URL or TSR ID into a string that is definitely a TRS ID 🎀
• parse_trs_spec(trs_spec) — splits a TRS ID into a workflow and an optional version 🌸
• compose_trs_spec(trs_workflow_id, trs_version) — builds a TRS ID from a workflow ID and a version ID 🫶
• find_workflow(workflow[, supported_languages]) — given a Dockstore URL or TRS identifier, gets the root WDL or CWL URL for the workflow, along with the TRS workflow ID and version 🔥
• fetch_workflow(trs_workflow_id, trs_version, language) — returns a URL or local path to a workflow's primary descriptor file 👑
• resolve_workflow(workflow[, supported_languages]) — finds the real workflow URL or filename from a command line argument ⭐

🌈 Module Contents

toil.lib.trs.logger

toil.lib.trs.TRS_ROOT = 'https://dockstore.org'

toil.lib.trs.is_trs_workflow(workflow)

Returns True if a workflow string smells like TRS. It detects Dockstore page URLs and strings that could be Dockstore TRS IDs 👀

• Parameters
  • workflow (str)
• Return type
  • bool

toil.lib.trs.extract_trs_spec(workflow)

Parses a Dockstore workflow URL or TSR ID into a string that is definitely a TRS ID ✨

• Parameters
  • workflow (str)
• Return type
  • str

toil.lib.trs.parse_trs_spec(trs_spec)

Parses a TRS ID into a workflow and an optional version 🌷

• Parameters
  • trs_spec (str)
• Return type
  • tuple[str, Optional[str]]

toil.lib.trs.compose_trs_spec(trs_workflow_id, trs_version)

Composes a TRS ID from a workflow ID and a version ID 🧩

• Parameters
  • trs_workflow_id (str)
  • trs_version (str)
• Return type
  • str

toil.lib.trs.find_workflow(workflow, supported_languages=None)

Given a Dockstore URL or TRS identifier, this gets the root WDL or CWL URL for the workflow, plus the TRS workflow ID and version 🕵️‍♀️

Accepts inputs like:

• https://dockstore.org/workflows/github.com/dockstore-testing/md5sum-checker:master?tab=info
• #workflow/github.com/dockstore-testing/md5sum-checker

It assumes the input is actually one of the supported formats. Check out is_trs_workflow() for that 🫶

📝 TODO: Needs to handle multi-workflow files if Dockstore can.

• Raises
  • FileNotFoundError: if the workflow or version doesn't exist.
  • ValueError: if the version is not specified but cannot be automatically determined.
• Parameters
  • workflow (str)
  • supported_languages (Optional[set[str]])
• Return type
  • tuple[str, str, str]

toil.lib.trs.fetch_workflow(trs_workflow_id, trs_version, language)

Returns a URL or local path to a workflow's primary descriptor file. The file comes with its required files, so it can actually run 🏃‍♀️

• Raises
  • FileNotFoundError: if the workflow or version doesn't exist.
• Parameters
  • trs_workflow_id (str)
  • trs_version (str)
  • language (str)
• Return type
  • str

toil.lib.trs.resolve_workflow(workflow, supported_languages=None)

Finds the real workflow URL or filename from a command line argument. It transforms a workflow URL or path (which might secretly be a Dockstore page URL or TRS specifier) into an actual URL or path to a workflow document, plus an optional TRS specifier 💫

Accepts inputs like:

• https://dockstore.org/workflows/github.com/dockstore-testing/md5sum-checker:master?tab=info
• #workflow/github.com/dockstore-testing/md5sum-checker
• ./local.cwl
• https://example.com/~myuser/workflow/main.cwl
• Raises
  • FileNotFoundError: if the workflow or version should be in Dockstore but doesn't seem to exist.
• Parameters
  • workflow (str)
  • supported_languages (Optional[set[str]])
• Return type
  • tuple[str, Optional[str]]

---

✨ toil.lib.url ✨

🌸 Attributes

• logger

💥 Exceptions

• ProxyConnectionError — a dummy class 🫥

🦋 Classes

• URLAccess — a widget for accessing external storage (URLs) 🔌

✨ Functions ✨

okay bestie, these four factory functions are all part of the toil.lib.url era 💅

• file_job_store_factory()
• google_job_store_factory()
• aws_job_store_factory()
• job_store_support_factory()

🧚 Module Contents 🧚

• exception toil.lib.url.ProxyConnectionError: bases BaseException (/usr/share/doc/python3/html/library/exceptions.html#BaseException). It's a dummy class, so it's basically a placeholder rn 🙈
• toil.lib.url.logger 🪵
• class toil.lib.url.URLAccess: a little widget for accessing external storage (URLs) 🔥
  • classmethod url_exists(src_uri): returns True if the file at the given URI exists and False otherwise. ngl it might raise an error if existence can't be figured out 💀
    • Parameters: src_uri (str, /usr/share/doc/python3/html/library/stdtypes.html#str): URL pointing to a file or object in the storage mechanism of a supported URL scheme, e.g. a blob in an AWS s3 bucket.
    • Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)
  • classmethod get_size(src_uri): gets the size in bytes of the file at the given URL, or None if it can't be obtained 📏
    • Parameters: src_uri (str, /usr/share/doc/python3/html/library/stdtypes.html#str): URL pointing to a file or object in the storage mechanism of a supported URL scheme, e.g. a blob in an AWS s3 bucket.
    • Return type: Optional[int] (/usr/share/doc/python3/html/library/functions.html#int)
  • classmethod get_is_directory(src_uri): returns True if the thing at the given URL is a directory and False if it's a file. The URL may or may not end in '/'.
    • Parameters: src_uri (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
    • Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)
  • classmethod list_url(src_uri): lists the directory at the given URL 📂. Returned path components can be joined with '/' onto the passed URL to form new URLs, and the ones that end in '/' are directories. The provided URL may or may not end with '/'.
    • Currently supported schemes are:
      • 's3' for objects in Amazon S3, e.g. s3://bucket/prefix/
      • 'file' for local files, e.g. file:///local/dir/path/
    • Parameters: src_uri (str, /usr/share/doc/python3/html/library/stdtypes.html#str): URL pointing to a directory or prefix in the storage mechanism of a supported URL scheme, e.g. a prefix in an AWS s3 bucket.
    • Returns: a list of URL components in the given directory, already URL-encoded.
    • Return type: list[str] (/usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str)
  • classmethod read_from_url(src_uri, writable): reads the given URL and writes its content into the given writable stream 📝. Raises FileNotFoundError if the URL doesn't exist.
    • Returns: the size of the file in bytes and whether the executable permission bit is set.
    • Parameters:
      • src_uri (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
      • writable (IO[bytes], /usr/share/doc/python3/html/library/stdtypes.html#bytes)
    • Return type: tuple[int, bool] (/usr/share/doc/python3/html/library/stdtypes.html#tuple, /usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/functions.html#bool)
  • classmethod open_url(src_uri): reads from the given URI 📖. Raises FileNotFoundError if the URL doesn't exist. It has a readable stream interface, unlike read_from_url(), which takes a writable stream.
    • Parameters: src_uri (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
    • Return type: IO[bytes] (/usr/share/doc/python3/html/library/stdtypes.html#bytes)
• toil.lib.url.file_job_store_factory(): return type type[URLAccess <#toil.lib.url.URLAccess>] (/usr/share/doc/python3/html/library/functions.html#type)
• toil.lib.url.google_job_store_factory(): return type type[URLAccess <#toil.lib.url.URLAccess>] (/usr/share/doc/python3/html/library/functions.html#type)
• toil.lib.url.aws_job_store_factory(): return type type[URLAccess <#toil.lib.url.URLAccess>] (/usr/share/doc/python3/html/library/functions.html#type)
• toil.lib.url.job_store_support_factory(): return type type[URLAccess <#toil.lib.url.URLAccess>] (/usr/share/doc/python3/html/library/functions.html#type)

---

🌈 toil.lib.web 🌈

Contains functions for making web requests with Toil 🌐. All web requests should go through this module, so they all use the right user agent. Here's the example:

>>> httpserver = getfixture("httpserver")
>>> handler = httpserver.expect_request("/path").respond_with_json({})
>>> from toil.lib.web import web_session
>>> web_session.get(httpserver.url_for("/path"))
<Response [200]>

✨ Attributes ✨

• web_session

🧚 Module Contents 🧚

• toil.lib.web.web_session

---

🦋 toil.options 🦋

🌸 Submodules 🌸

• toil.options.common

💜 toil.options.common: Attributes 💜

• logger
• defaultTargetTime
• SYS_MAX_SIZE
• JOBSTORE_HELP

✨ Functions ✨

okay so this is the cute little cheat sheet for every helper in toil.options.common 🫶 here's the lineup:

• parse_set_env(l) — parses a list of strings like "NAME=VALUE" (or just "NAME") into a dictionary 💖
• parse_str_list(s) — no description written for this one 🙈
• parse_int_list(s) — no description written for this one either 🌸
• iC(min_value[, max_value]) — hands you back a function that checks if an int is in the half-open interval 🎀
• fC(minValue[, maxValue]) — same energy as iC, but for floats ✨
• parse_accelerator_list(specs) — parses a string description of one or more accelerator requirements 🔥
• parseBool(val) — no description written for this one 💅
• make_open_interval_action(min[, max]) — gives you an argparse action class that checks the input is within the half-open interval 🦋
• parse_jobstore(jobstore_uri) — turns the jobstore string into its matching URI 🧚
• add_base_toil_options(parser[, jobstore_as_flag, cwl]) — adds the base Toil command line options to your parser 👑

---

✨ Module Contents ✨

toil.options.common.logger

No description given for this one 🪵

toil.options.common.defaultTargetTime

toil.options.common.defaultTargetTime = 1800

toil.options.common.SYS_MAX_SIZE

toil.options.common.SYS_MAX_SIZE = 9223372036854775807

toil.options.common.parse_set_env(l)

okay this one is literally so useful 💖 it takes a list of strings shaped like NAME=VALUE (or just NAME) and turns them into a dictionary. names that show up with no = at all get a value of None, so they don't just vanish 🙈

>>> parse_set_env([])
{}
>>> parse_set_env(['a'])
{'a': None}
>>> parse_set_env(['a='])
{'a': ''}
>>> parse_set_env(['a=b'])
{'a': 'b'}
>>> parse_set_env(['a=a', 'a=b'])
{'a': 'b'}
>>> parse_set_env(['a=b', 'c=d'])
{'a': 'b', 'c': 'd'}
>>> parse_set_env(['a=b=c'])
{'a': 'b=c'}
>>> parse_set_env([''])
Traceback (most recent call last):
...
ValueError: Empty name
>>> parse_set_env(['=1'])
Traceback (most recent call last):
...
ValueError: Empty name

Parameters

• l (list[str])

Return type

dict[str, Optional[str]]

toil.options.common.parse_str_list(s)

Parameters

• s (str)

Return type

list[str]

toil.options.common.parse_int_list(s)

Parameters

• s (str)

Return type

list[int]

toil.options.common.iC(min_value, max_value=None)

returns a function that checks if a given int is in the given half-open interval 🎀

Parameters

• min_value (int)
• max_value (Optional[int])

Return type

Callable[[int], bool]

toil.options.common.fC(minValue, maxValue=None)

same vibe as iC but for floats ✨ returns a function that checks if a given float is in the given half-open interval

Parameters

• minValue (float)
• maxValue (Optional[float])

Return type

Callable[[float], bool]

toil.options.common.parse_accelerator_list(specs)

parses a string description of one or more accelerator requirements 🔥

Parameters

• specs (Optional[str])

Return type

list[toil.job.AcceleratorRequirement]

toil.options.common.parseBool(val)

Parameters

• val (str)

Return type

bool

toil.options.common.make_open_interval_action(min, max=None)

returns an argparse action class to check if the input is within the given half-open interval 🦋 for example, the value given to argparse has to be within the interval [min, max). the types of min and max must match (and max can be None) 💅

Parameters

• min (Union[int, float]) — float/int
• max (Optional[Union[int, float]]) — optional float/int

Returns

argparse action class

Return type

type[argparse.Action]

toil.options.common.parse_jobstore(jobstore_uri)

turns the jobstore string into its corresponding URI 🧚 for example, /path/to/jobstore -> <file:/path/to/jobstore>

if the jobstore string is already a URI, it gets returned as-is. for example, aws:/path/to/jobstore -> aws:/path/to/jobstore

:param jobstore_uri: string of the jobstore :return: URI of the jobstore

Parameters

• jobstore_uri (str)

Return type

str

toil.options.common.JOBSTORE_HELP

Multiline-String 💜 this is the help text for the job store location, and it's a whole thing:

The location of the job store for the workflow. A job store holds persistent information about the jobs, stats, and files in a workflow. If the workflow is run with a distributed batch system, the job store must be accessible by all worker nodes. Depending on the desired job store implementation, the location should be formatted according to one of the following schemes:

• file:<path> where <path> points to a directory on the file system
• aws:<region>:<prefix> where <region> is the name of an AWS region like us-west-2 and <prefix> will be prepended to the names of any top-level AWS resources in use by job store, e.g. S3 buckets.
• google:<project_id>:<prefix> TODO: explain

For backwards compatibility, you may also specify ./foo (equivalent to file:./foo or just file:foo) or /bar (equivalent to file:/bar).

toil.options.common.add_base_toil_options(parser,
jobstore_as_flag=False, cwl=False)

okay bestie, this one is the base options helper 💖 It slaps the basic Toil command line flags onto your parser ✨ Basically add_base_toil_options is the one that makes sure every Toil command has the same core setup. no cap, it's giving consistency 💅

• parser is the argparse.ArgumentParser you want the options added to (docs: /usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser) 🧚
• jobstore_as_flag is a bool (docs: /usr/share/doc/python3/html/library/functions.html#bool). If it's True, the job store option becomes a --jobStore flag instead of a required jobStore positional argument. Lowkey a big deal, so pay attention 👀
• cwl is a bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) that controls whether CWL should be included or not 🌈

It returns None. Nothing to see there, just vibes 🫶

---

toil.options.cwl

Functions

• add_cwl_options(parser[, suppress]) — adds CWL options to the parser. It only adds nonpositional CWL arguments ✨

Module Contents

toil.options.cwl.add_cwl_options(parser, suppress=True)

Okay so this one is the CWL version of the same idea 🦋 It adds CWL options to the parser, but only the nonpositional ones, so no positional arguments sneaking in 🙈

• parser is an argparse.ArgumentParser (docs: /usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser) and it's the parser you want the options added to 🎀
• suppress is a bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) that suppresses the help output. Default is True 💜

It returns None. Same energy as before 💅

---

toil.options.runner

Functions

• add_runner_options(parser[, cwl, wdl]) — adds options to the WDL or CWL runners that are shared or the same between runners. The main character of the shared-options world 👑

Module Contents

toil.options.runner.add_runner_options(parser, cwl=False, wdl=False)

This is the one that handles the stuff both runners have in common, so you don't have to copy-paste it twice (iconic, honestly) 🌸 It adds to the WDL or CWL runners options that are shared or the same between runners.

• parser is the parser you want the arguments added to 🧚
• cwl is a bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) ⭐
• wdl is a bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) 🍓

It returns None 🫶

---

toil.options.wdl

Functions

• add_wdl_options(parser[, suppress]) — adds WDL options to a parser. This only adds nonpositional WDL arguments. WDL's own little lane, periodt ✨

Module Contents

toil.options.wdl.add_wdl_options(parser, suppress=True)

Same vibe as the CWL one, just for WDL 🦋 It adds WDL options to a parser, and again only the nonpositional arguments.

• parser is the parser you want the options added to 🎀
• suppress is a bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) that suppresses the help output. Default is True 💜

It returns None 💅

---

toil.provisioners

Submodules

• toil.provisioners.abstractProvisioner 🌈

---

toil.provisioners.abstractProvisioner

Attributes

• a_short_time
• logger

Exceptions

• ManagedNodesNotSupportedException — raised when attempting to add managed nodes (which autoscale up and down by themselves, without the provisioner doing the work) to a provisioner that does not support them 🙈

Classes

• Shape — represents a job or a node's "shape", in terms of the dimensions of memory, cores, disk and wall-time allocation 🧚
• AbstractProvisioner — the interface for provisioning worker nodes to use in a Toil cluster 👑

Module Contents

toil.provisioners.abstractProvisioner.a_short_time = 5

toil.provisioners.abstractProvisioner.logger

exception toil.provisioners.abstractProvisioner.ManagedNodesNotSupportedException 💀

Bases: RuntimeError (docs: /usr/share/doc/python3/html/library/exceptions.html#RuntimeError)

Okay, this is the error you get when you try to add managed nodes to a provisioner that can't handle them. Managed nodes autoscale up and down by themselves, without the provisioner doing the work for them, so it's a whole different situation 😭

Polling with this and try/except is the Right Way to check if managed nodes are available from a provisioner. Use try/except, not vibes 💅

class toil.provisioners.abstractProvisioner.Shape(wallTime, memory, cores, disk, preemptible) 🦋

Bases: none listed. This is the class that describes a job's or a node's "shape" 🌸 It's all about the dimensions: memory, cores, disk and wall-time allocation.

The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs? (yep, the original page is still wondering about that one 🙈)

The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively 💜

Parameters

• wallTime (Union[int, float])
• memory (int)
• cores (Union[int, float])
• disk (int)
• preemptible (bool, docs: /usr/share/doc/python3/html/library/functions.html#bool)

Attributes

• wallTime
• memory
• cores
• disk
• preemptible

Methods

• __eq__(other) — takes other (Any) and returns bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) 🫶
• greater_than(other) — takes other (Any) and returns bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) 🔥
• __gt__(other) — takes other (Any) and returns bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) ⭐
• __repr__() — returns str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🎀
• __str__() — returns str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🌈
• __hash__() — returns int (docs: /usr/share/doc/python3/html/library/functions.html#int) 🍓

class toil.provisioners.abstractProvisioner.AbstractProvisioner(clusterName=None, clusterType='mesos', zone=None, nodeStorage=50, nodeStorageOverrides=None, enable_fuse=False) ✨

Bases: abc.ABC (docs: /usr/share/doc/python3/html/library/abc.html#abc.ABC)

This is the interface for provisioning worker nodes to use in a Toil cluster. It's the blueprint, bestie, so anything that wants to be a provisioner has to follow it 👑

Parameters

• clusterName (Optional[str], docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• clusterType (Optional[str], docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• zone (Optional[str], docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• nodeStorage (int, docs: /usr/share/doc/python3/html/library/functions.html#int)
• nodeStorageOverrides (Optional[list[str]]) — an optional list of strings, so you can skip it if you don't need it 📦
• enable_fuse (bool) — a true-or-false switch ✨

These are the class-level defaults, so you get these vibes unless you override them 🌸

LEADER_HOME_DIR = '/root/'
cloud: str = None
clusterName = None
clusterType = 'mesos'
enable_fuse = False

💅 abstractmethod supportedClusterTypes()

Get all the cluster types this provisioner implementation supports. Think of it as the menu of what's on offer 🍓

Return type — set[str]

✨ abstractmethod createClusterSettings()

Initialize the class for a brand-new cluster that's about to get deployed, when you're running outside the cloud 🌈

👑 abstractmethod readClusterSettings()

Initialize the class from an existing cluster. This one assumes the instance you're on is the leader, so it's giving leader energy 👑. Implementations must call _setLeaderWorkerAuthentication(). Not optional, periodt 💀.

🌸 setAutoscaledNodeTypes(nodeTypes)

Set the node types, shapes and spot bids for Toil-managed autoscaling 📈

Parameters

• nodeTypes (list[tuple[set[str], Optional[float]]]) — a list of node types, as parsed with parse_node_types.

🧚 hasAutoscaledNodeTypes()

Checks whether node types have already been configured on the provisioner, via setAutoscaledNodeTypes. Quick vibe check ✅

Returns — True if node types are configured for autoscaling, and false otherwise.

Return type — bool

🍓 getAutoscaledInstanceShapes()

Gets all the node shapes and their named instance types that the Toil autoscaler should manage 🍓

Return type — dict[Shape, str]

🙈 static retryPredicate(e)

Returns true if the exception e should be retried by the cluster scaler. For example, it should return true if the exception came from hitting an API rate limit. The error gets retried with exponential backoff ⏳

Parameters

• e — the exception raised during execution of setNodeCount.

Returns — a boolean indicating whether the exception e should be retried.

🫶 abstractmethod launchCluster(*args, **kwargs)

Initialize a cluster and create a leader node 🚀. Implementations must call _setLeaderWorkerAuthentication() with the leader so that workers can actually get launched.

Parameters

• leaderNodeType — the leader instance.
• leaderStorage — the amount of disk to allocate to the leader, in gigabytes.
• owner — tag identifying the owner of the instances.

---

🔥 abstractmethod addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

Used to add worker nodes to the cluster 🔥

Parameters

• numNodes (int) — the number of nodes to add.
• preemptible (bool) — whether or not the nodes will be preemptible.
• spotBid (Optional[float]) — the bid for preemptible nodes if applicable (this can be set in config, also).
• nodeTypes (set[str])

Returns — the number of nodes successfully added.

Return type — int

🌈 addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)

Add a group of managed nodes of the given type, up to the given maximum. These nodes launch and terminate automatically depending on cluster load, so they're basically self-managing 💖

Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can't have managed nodes. Know your limits, bestie 🙅‍♀️

Parameters

• minNodes — the minimum number of nodes to scale to.
• maxNodes — the maximum number of nodes to scale to.
• preemptible — whether or not the nodes will be preemptible.
• spotBid — the bid for preemptible nodes if applicable (this can be set in config, also).
• nodeTypes (set[str])

Return type — None

💜 abstractmethod terminateNodes(nodes)

Terminate the nodes represented by the given Node objects. Bye bye 👋

Parameters

• nodes (list[toil.provisioners.node.Node]) — list of Node objects.

Return type — None

👑 abstractmethod getLeader()

Returns — the leader node. The main character 👑

✨ abstractmethod getProvisionedWorkers(instance_type=None, preemptible=None)

Get all nodes from the provisioner, optionally filtered by instance type or preemptability. This includes both static and autoscaled nodes 🧾

Parameters

• preemptible (Optional[bool]) — a boolean to restrict results to preemptible nodes or non-preemptible nodes.
• instance_type (Optional[str])

Returns — a list of Node objects.

Return type — list[toil.provisioners.node.Node]

🦋 abstractmethod getNodeShape(instance_type, preemptible=False)

The shape of a preemptible or non-preemptible node managed by this provisioner. The shape defines the key properties of a machine, like its number of cores or the time between billing intervals ⭐

Parameters

• instance_type (str) — the instance type name to return the shape of.

Return type — Shape

💀 abstractmethod destroyCluster()

Terminates all nodes in the specified cluster and cleans up every resource tied to that cluster. Total clean-up era 🧹

Parameters

• clusterName — identifier of the cluster to terminate.

Return type — None

🎀 class InstanceConfiguration

Lets you define the initial setup for an instance, then turn it into an Ignition configuration for instance user data 🎀

files = []
units = []
sshPublicKeys = []

🧁 addFile(path, filesystem='root', mode='0755', contents='', append=False)

Make a file on the instance with the given filesystem, mode, and contents 📁. For the storage.files section, check out https://github.com/kinvolk/ignition/blob/flatcar-master/doc/configuration-v2_2.md

Parameters

• path (str)
• filesystem (str)
• mode (Union[str, int])
• contents (str)
• append (bool)

💖 addUnit 💖

addUnit(name, enabled=True, contents='')

okay bestie, this one makes a systemd unit on the instance 🧚 you give it a name (with the .service on the end, it's literally required 😭), plus the contents. units are enabled by default so you don't have to do anything extra 🫶

if you need to check a unit's logs, try this:

systemctl status whatever.service

or this:

journalctl -xe

Parameters

• name (str) — the unit's name, including .service
• enabled (bool) — whether the unit is enabled, defaulting to True
• contents (str) — the unit's contents, defaulting to ''

🌸 addSSHRSAKey(keyData) 🌸

addSSHRSAKey(keyData)

this one authorizes a bare, encoded RSA key 🔑 just make sure you leave off the "ssh-rsa" part, bestie 🙅‍♀️

Parameters

• keyData (str)

🦋 toIgnitionConfig() 🦋

toIgnitionConfig()

this returns an Ignition configuration that describes the config you want ✨

Return type

• str

🌈 getBaseInstanceConfiguration() 🌈

getBaseInstanceConfiguration()

this gets the base configuration for both leader and worker instances, for all cluster types 👑

Return type

• InstanceConfiguration

💅 addVolumesService(config) 💅

addVolumesService(config)

this adds a service that preps and mounts local scratch volumes 💾

Parameters

• config (InstanceConfiguration)

🍓 addNodeExporterService(config) 🍓

addNodeExporterService(config)

this adds the node exporter service for Prometheus to an instance configuration 📈

Parameters

• config (InstanceConfiguration)

🧚 toil_service_env_options() 🧚

toil_service_env_options()

Return type

• str

👑 add_toil_service(config, role, keyPath=None, preemptible=False) 👑

add_toil_service(config, role, keyPath=None, preemptible=False)

this adds the Toil leader or worker service to an instance configuration 🔥

btw it'll run the Mesos master or agent as needed in Mesos clusters. in Kubernetes clusters it just sleeps, which gives you a spot to shell into on the leader, and it shouldn't run on the worker 💀

add_toil_service is picky: role needs to be 'leader' or 'worker', and a 'worker' won't work until the leader credentials have been collected.

Parameters

• role (str) — should be 'leader' or 'worker'. won't work for 'worker' until leader credentials have been collected
• keyPath (str) — the path on the node to a server-side encryption key. the key gets added to the node after it starts, and the service waits until the key is there before it starts
• preemptible (bool) — whether a worker identifies itself to the scheduler as preemptible
• config (InstanceConfiguration)

🌸 getKubernetesValues(architecture='amd64') 🌸

getKubernetesValues(architecture='amd64')

this returns a dict of Kubernetes component versions and paths, ready to format into Kubernetes-related templates 🎀

Parameters

• architecture (str)

🦋 addKubernetesServices(config, architecture='amd64') 🦋

addKubernetesServices(config, architecture='amd64')

this adds installing Kubernetes and Kubeadm, plus setting up the Kubelet to run when configured, to an instance configuration ✨ leaders and workers go through the same process, so no drama there 💖

Parameters

• config (InstanceConfiguration)
• architecture (str)

💜 getKubernetesAutoscalerSetupCommands(values) 💜

abstractmethod getKubernetesAutoscalerSetupCommands(values)

this is an abstract method, so it returns Bash commands that set up the Kubernetes cluster autoscaler for provisioning in the environment this provisioner supports 🌈

only implement this if you're supporting Kubernetes clusters, bestie 🙈

Parameters

• values (dict[str, str]) — contains definitions of cluster variables, like AUTOSCALER_VERSION and CLUSTER_NAME

Returns

• a Bash snippet

Return type

• str

⭐ getKubernetesCloudProvider() ⭐

getKubernetesCloudProvider()

this returns the Kubernetes cloud provider (for example, 'aws'), which gets passed to the kubelets in a Kubernetes cluster provisioned by this provisioner ☁️

it defaults to None if you don't override it, and in that case no cloud provider integration gets used 🫶

Returns

• the cloud provider name, or None

Return type

• Optional[str]

🎀 addKubernetesLeader(config) 🎀

addKubernetesLeader(config)

this adds the services that configure a Kubernetes leader, but only if Kubernetes is already set to be installed 👑

Parameters

• config (InstanceConfiguration)

🌸 addKubernetesWorker(config, authVars, preemptible=False) 🌸

addKubernetesWorker(config, authVars, preemptible=False)

this adds the services that configure a Kubernetes worker, but only if Kubernetes is already set to be installed 🔥 the worker authenticates back to the leader using JOIN_TOKEN, JOIN_CERT_HASH, and JOIN_ENDPOINT, which all live in the authentication data dict you pass in 🔐

Parameters

• config (InstanceConfiguration) — the configuration to add services to
• authVars (dict[str, str]) — a dict with authentication info
• preemptible (bool) — whether the worker should be labeled as preemptible

---

🦋 toil.provisioners.aws 🦋

🌸 Submodules 🌸

• toil.provisioners.aws.awsProvisioner

✨ toil.provisioners.aws.awsProvisioner ✨

💖 Attributes 💖

• logger — no description given, just vibes 🫶
• F — no description given either 🙈

🔥 Exceptions 🔥

• InvalidClusterStateException — the common base class for all non-exit exceptions 💀

👑 Classes 👑

• AWSProvisioner — an interface for provisioning worker nodes to use in a Toil cluster 🌈

✨ Functions ✨

okay so this is the lineup of helper functions in the AWS provisioner, bestie 💅 the table version is kinda messy, so here's the tea:

• awsRetryPredicate(e) — 🎀 no blurb in the table, it's giving mystery 🙈
• expectedShutdownErrors(e) — matches the errors we expect during shutdown, which indicate ✨ (the table cuts off right there, no cap)
• awsRetry(f) — 🔁 a decorator that retries the wrapped function if aws throws unexpected errors
• awsFilterImpairedNodes(nodes, boto3_ec2) — 🧹 no blurb here either
• collapse_tags(instance_tags) — 🏷️ collapses tags from boto3 format into node format

---

✨ Module Contents ✨

okay, the full deets, bestie 🫶

toil.provisioners.aws.awsProvisioner.logger 🪵

just the logger! that's the whole thing 💖

toil.provisioners.aws.awsProvisioner.awsRetryPredicate(e)

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

toil.provisioners.aws.awsProvisioner.expectedShutdownErrors(e)

Matches errors that we expect to occur during shutdown, and which indicate that we need to wait or try again. ⏳

It should not match any errors that mean an operation is impossible or unnecessary, like errors from a thing not existing to be deleted 🙅‍♀️

Parameters

• e (Exception, /usr/share/doc/python3/html/library/exceptions.html#Exception)

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

toil.provisioners.aws.awsProvisioner.F

just F by itself, no description, a little mysterious 🌸

toil.provisioners.aws.awsProvisioner.awsRetry(f)

This decorator retries the wrapped function if aws throws unexpected errors. 🔁

It should wrap any function that makes use of boto 🧸

Parameters

• f (Callable[Ellipsis, F])

Return type: Callable[Ellipsis, F]

toil.provisioners.aws.awsProvisioner.awsFilterImpairedNodes(nodes, boto3_ec2)

Parameters

• nodes (list[mypy_boto3_ec2.type_defs.InstanceTypeDef], /usr/share/doc/python3/html/library/stdtypes.html#list)
• boto3_ec2 (mypy_boto3_ec2.client.EC2Client)

Return type: list[mypy_boto3_ec2.type_defs.InstanceTypeDef] (/usr/share/doc/python3/html/library/stdtypes.html#list)

exception toil.provisioners.aws.awsProvisioner.InvalidClusterStateException

Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)

Common base class for all non-exit exceptions. 🚫

toil.provisioners.aws.awsProvisioner.collapse_tags(instance_tags)

Collapse tags from boto3 format to node format. :param instance_tags: tags as a list, :return: Dict of tags 🏷️

Parameters

• instance_tags (list[mypy_boto3_ec2.type_defs.TagTypeDef], /usr/share/doc/python3/html/library/stdtypes.html#list)

Return type: dict[str, str] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)

class toil.provisioners.aws.awsProvisioner.AWSProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey, enable_fuse)

Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner (#toil.provisioners.abstractProvisioner.AbstractProvisioner)

Interface for provisioning worker nodes to use in a Toil cluster. 🚢

Parameters

• clusterName (str | None, /usr/share/doc/python3/html/library/stdtypes.html#str)
• clusterType (str | None, /usr/share/doc/python3/html/library/stdtypes.html#str)
• zone (str | None, /usr/share/doc/python3/html/library/stdtypes.html#str)
• nodeStorage (int, /usr/share/doc/python3/html/library/functions.html#int)
• nodeStorageOverrides (list[str] | None, /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str)
• sseKey (str | None, /usr/share/doc/python3/html/library/stdtypes.html#str)
• enable_fuse (bool, /usr/share/doc/python3/html/library/functions.html#bool)

Attributes

• cloud = 'aws' ☁️
• aws
• s3_bucket_name = 'default-toil-cluster-name--internal' 🪣

supportedClusterTypes()

Get all the cluster types that this provisioner implementation supports. 📋

Return type: set[str] (/usr/share/doc/python3/html/library/stdtypes.html#set, /usr/share/doc/python3/html/library/stdtypes.html#str)

createClusterSettings()

Create a new set of cluster settings for a cluster to be deployed into AWS. ⚙️

Return type: None

readClusterSettings()

Reads the cluster settings from the instance metadata, which assumes the instance is the leader. 👑

Return type: None

launchCluster(leaderNodeType, leaderStorage, owner, keyName, botoPath, userTags, vpcSubnet, awsEc2ProfileArn, awsEc2ExtraSecurityGroupIds, **kwargs)

Starts a single leader node and populates this class with the leader's metadata. 🚀

Parameters

• leaderNodeType (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — An AWS instance type, like "t2.medium", for example.
• leaderStorage (int, /usr/share/doc/python3/html/library/functions.html#int) — An integer number of gigabytes to provide the leader instance with.
• owner (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — Resources will be tagged with this owner string.
• keyName (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — The ssh key to use to access the leader node.
• botoPath (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — The path to the boto credentials directory.
• userTags (dict[str, str] | None, /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str) — Optionally provided user tags to put on the cluster.
• vpcSubnet (str | None, /usr/share/doc/python3/html/library/stdtypes.html#str) — Optionally specify the VPC subnet for the leader.
• awsEc2ProfileArn (str | None, /usr/share/doc/python3/html/library/stdtypes.html#str) — Optionally provide the profile ARN.
• awsEc2ExtraSecurityGroupIds (list[str] | None, /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str) — Optionally provide additional security group IDs.
• kwargs (dict[str, Any], /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)

Returns: None

Return type: None

toil_service_env_options()

Set AWS tags in user docker container 🐳

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

getKubernetesAutoscalerSetupCommands(values)

Get the Bash commands necessary to configure the Kubernetes Cluster Autoscaler for AWS. 🐚

Parameters

• values (dict[str, str], /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

getKubernetesCloudProvider()

Use the "aws" Kubernetes cloud provider when setting up Kubernetes ☸️

Return type: str | None (/usr/share/doc/python3/html/library/stdtypes.html#str)

🎀 getNodeShape(instance_type, preemptible=False)

okay so this one hands you back the Shape for whatever instance type you feed it, like 't2.medium' 🦋

Parameters

• instance_type (str)
• preemptible (bool)

Return type: toil.provisioners.abstractProvisioner.Shape

💅 static retryPredicate(e)

this one is giving "should I try again?" 🔁 It returns true if the exception e should get retried by the cluster scaler, like when you've hit an API rate limit. Retries happen with exponential backoff, so it backs off a little more each time ✨

Parameters

• e (Exception) — the exception raised during execution of setNodeCount

Returns: a boolean saying whether the exception e should be retried

Return type: bool

🌸 destroyCluster()

the nuclear option 💀 It terminates the instances and deletes the profile and security group.

Return type: None

🧚 terminateNodes(nodes)

bye bye, nodes 👋 This terminates the nodes represented by the given Node objects.

Parameters

• nodes (list[toil.provisioners.node.Node]) — list of Node objects

Return type: None

🍓 addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

this is how you add worker nodes to the cluster 🫶

Parameters

• numNodes (int) — the number of nodes to add
• preemptible (bool) — whether or not the nodes will be preemptible
• spotBid (float | None) — the bid for preemptible nodes, if applicable (you can set this in config too 🌈)
• nodeTypes (set[str])

Returns: the number of nodes successfully added

Return type: int

👑 addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)

main character energy 👑 This adds a group of managed nodes of the given type, up to the given maximum. The nodes get launched and terminated automatically depending on cluster load, so you don't have to babysit them 💜

Heads up: this raises ManagedNodesNotSupportedException if the provisioner implementation or the cluster configuration can't have managed nodes.

Parameters

• minNodes (int) — the minimum number of nodes to scale to
• maxNodes (int) — the maximum number of nodes to scale to
• preemptible (bool) — whether or not the nodes will be preemptible
• spotBid (float | None) — the bid for preemptible nodes, if applicable (this can be set in config, also)
• nodeTypes (set[str])

Return type: None

✨ getProvisionedWorkers(instance_type=None, preemptible=None)

pulls up all the nodes from the provisioner, optionally filtered by instance type or preemptability 🔍 It includes both static and autoscaled nodes, so nothing gets left out 🥰

Parameters

• preemptible (bool | None) — a boolean that restricts results to preemptible nodes or non-preemptible nodes
• instance_type (str | None)

Returns: a list of Node objects

Return type: list[toil.provisioners.node.Node]

🌈 getLeader(wait=False)

finding the leader 👀 This gets the leader for the cluster as a Toil Node object.

Parameters

• wait (bool)

Return type: toil.provisioners.node.Node

💜 full_policy(resource)

this one builds a dict that describes the JSON form of a full-access-granting AWS IAM policy for the service with the given name, like 's3' 🪄

Parameters

• resource (str)

Return type: dict[str, Any]

🫧 kubernetes_policy()

okay this one is a little extra 🙈 It gets the Kubernetes policy grants that aren't already covered by the full grants on EC2 and IAM. For the details, check out https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#leader-policy and https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#worker-policy.

These are mostly needed to support Kubernetes' AWS CloudProvider, and some are for the Kubernetes Cluster Autoscaler's AWS integration.

Some of these are really only needed on the leader.

Return type: dict[str, Any]

---

💅 Attributes

• logger

🎀 Classes

• ZoneTuple

🦋 Functions

• get_aws_zone_from_spot_market(spotBid, nodeType, ...) — if a spot bid, node type, and Boto2 EC2 connection are specified, it picks a zone 🎯
• get_best_aws_zone([spotBid, nodeType, boto3_ec2, ...]) — gets the right AWS zone to use 🌍
• choose_spot_zone(zones, bid, spot_history) — returns the zone to put the spot request in, based on the following, in order of priority:
• optimize_spot_bid(boto3_ec2, instance_type, spot_bid, ...) — checks whether the bid is in line with history and makes an effort to place 🌸

🌸 Package Contents

toil.provisioners.aws.logger

class toil.provisioners.aws.ZoneTuple

Bases: tuple

• name
• price_deviation

toil.provisioners.aws.get_aws_zone_from_spot_market(spotBid, nodeType, boto3_ec2, zone_options)

okay so this one picks a zone where instances are easy to buy from, out of the zones in the region of the Boto2 connection, as long as a spot bid, node type, and Boto2 EC2 connection are all specified 🫶 These parameters always have to be specified together, or not at all ⚠️

In this case, zone_options can be used to restrict the choice to a subset of the zones in the region 💖

Parameters

• spotBid (Optional[float])
• nodeType — Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str 💖
• boto3_ec2 — Optional[botocore.client.BaseClient] 🧚
• zone_options — Optional[list[str]], see /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str 🎀

Return type: Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str ✨

---

✨ toil.provisioners.aws.get_best_aws_zone ✨

toil.provisioners.aws.get_best_aws_zone(spotBid=None, nodeType=None,
boto3_ec2=None, zone_options=None)

Okay, this is the zone picker 🗺️ It figures out the right AWS zone for you, and it checks a bunch of places in a specific order, so pay attention bestie 💅

Here's the order it goes in:

1. If the TOIL_AWS_ZONE environment variable is set, it just reports that zone. Done and dusted ✨
2. Otherwise, if we're running on EC2 or ECS, it reports the zone we're already sitting in 🏠
3. Otherwise, if a spot bid, a node type, and a Boto2 EC2 connection are all specified, it picks a zone where instances are easy to buy from the zones in the region of that Boto2 connection 🛒. These three must always be passed together, or not at all. And zone_options can narrow things down to a subset of the zones in that region 🎯
4. Otherwise, if the TOIL_AWS_REGION variable is set, it chooses a zone in that region 🌸
5. Finally, if Boto 2 has a default region configured, it chooses a zone in that region 🦋

If none of these methods can come up with a zone, it returns None. No zone, no vibe 💀

Parameters

• spotBid — Optional[float], see /usr/share/doc/python3/html/library/functions.html#float 💜
• nodeType — Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str 🍓
• boto3_ec2 — Optional[botocore.client.BaseClient] 🫶
• zone_options — Optional[list[str]], see /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str 🌈

Return type: Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str 👑

---

✨ toil.provisioners.aws.choose_spot_zone ✨

toil.provisioners.aws.choose_spot_zone(zones, bid, spot_history)

This one picks the zone for your spot request, and it goes by priority, periodt 💅 Here's the ranking:

1. Zones with prices currently under the bid 🔥
2. Zones with the most stable price 🧘‍♀️

Returns: the name of the selected zone 🎀

Parameters

• zones — list[str], see /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str 🌸
• bid — float, see /usr/share/doc/python3/html/library/functions.html#float 💖
• spot_history — list[mypy_boto3_ec2.type_defs.SpotPriceTypeDef], see /usr/share/doc/python3/html/library/stdtypes.html#list 📈

Return type: str, see /usr/share/doc/python3/html/library/stdtypes.html#str ✨

Here are some examples, so you can see the vibe in action 👀

>>> zones = ['us-west-2a', 'us-west-2b']
>>> FauxHistory = lambda p, z: {"SpotPrice": p, "AvailabilityZone": z}
>>> spot_history = [FauxHistory("0.1", 'us-west-2a'),
                    FauxHistory("0.2", 'us-west-2a'),
                    FauxHistory("0.3", 'us-west-2b'),
                    FauxHistory("0.6", 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2a'

>>> spot_history=[FauxHistory("0.3", 'us-west-2a'),
                  FauxHistory("0.2", 'us-west-2a'),
                  FauxHistory("0.1", 'us-west-2b'),
                  FauxHistory("0.6", 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2b'

>>> spot_history=[FauxHistory("0.1", 'us-west-2a'),
                  FauxHistory("0.7", 'us-west-2a'),
                  FauxHistory("0.1", 'us-west-2b'),
                  FauxHistory("0.6", 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2b'

---

✨ toil.provisioners.aws.optimize_spot_bid ✨

toil.provisioners.aws.optimize_spot_bid(boto3_ec2, instance_type,
spot_bid, zone_options)

This checks whether your bid lines up with the history, and it makes an effort to put the instance in a sensible zone 🧠✨

Parameters

• zone_options — list[str], see /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str. These are the allowed zones to consider, within the region tied to the Boto2 connection 🎯
• boto3_ec2 — botocore.client.BaseClient 🧚
• instance_type — str, see /usr/share/doc/python3/html/library/stdtypes.html#str 🍓
• spot_bid — float, see /usr/share/doc/python3/html/library/functions.html#float 💜

Return type: str, see /usr/share/doc/python3/html/library/stdtypes.html#str 👑

---

✨ toil.provisioners.clusterScaler ✨

🌈 Attributes 🌈

• logger 🪵
• EVICTION_THRESHOLD 🌸
• RESERVE_SMALL_LIMIT 🦋
• RESERVE_SMALL_AMOUNT 💖
• RESERVE_BREAKPOINTS ⭐
• RESERVE_FRACTIONS 🫶
• OS_SIZE 🧚
• FailedConstraint 💀

💅 Exceptions 💅

• JobTooBigError — Raised in the scaler thread when a job cannot fit in any available node 🙈

🎀 Classes 🎀

• BinPackedFit — If jobShapes is a set of tasks with run requirements (mem/disk/cpu), and nodeShapes is a sorted 🍓
• NodeReservation — The amount of resources that we expect to be available on a given node at each point in time ⏰
• ClusterScaler 🌊
• ScalerThread — A thread that automatically scales the number of either preemptible or non-preemptible worker 🧵
• ClusterStats 📊

🦋 Functions 🦋

• adjustEndingReservationForJob(reservation, jobShape, ...) — Add a job to an ending reservation that ends at wallTime 🌅
• split(nodeShape, jobShape, wallTime) — Partition a node allocation into two to fit the job ✂️
• binPacking(nodeShapes, jobShapes, goalTime) — Using the given node shape bins, pack the given job shapes into nodes to 📦

🌸 Module Contents 🌸

• toil.provisioners.clusterScaler.logger 🪵
• toil.provisioners.clusterScaler.EVICTION_THRESHOLD 🌸
• toil.provisioners.clusterScaler.RESERVE_SMALL_LIMIT 🦋
• toil.provisioners.clusterScaler.RESERVE_SMALL_AMOUNT 💖

Okay so RESERVE_BREAKPOINTS is a list of int or float values 🎀 Its type is list[int | float], with list pointing to /usr/share/doc/python3/html/library/stdtypes.html#list, int to /usr/share/doc/python3/html/library/functions.html#int, and float to /usr/share/doc/python3/html/library/functions.html#float.

toil.provisioners.clusterScaler.RESERVE_FRACTIONS = [0.25, 0.2, 0.1, 0.06, 0.02] is the list of fractions, and it's giving very specific energy 💅

toil.provisioners.clusterScaler.OS_SIZE and toil.provisioners.clusterScaler.FailedConstraint are just names in the docs with no description attached 🤷‍♀️

✨ class toil.provisioners.clusterScaler.BinPackedFit(nodeShapes, targetTime=defaultTargetTime) ✨

Bestie, this one is the main character 🦋 If jobShapes is a set of tasks with run requirements (mem/disk/cpu) and nodeShapes is a sorted list of available computers to run them on, this function tries to return a dictionary showing the minimum set of computerNode computers needed to run the tasks in jobShapes.

It uses a first fit decreasing (FFD) style bin packing approach to get an approximate minimum number of nodes that will fit the list of jobs. BinPackingFit expects the ordered list nodeShapes to already be sorted for "node preference" outside of BinPackingFit. So when it virtually "creates" nodes, the first node in nodeShapes that fits the job is the one that gets added 🧩

Parameters

• nodeShapes (list, /usr/share/doc/python3/html/library/stdtypes.html#list) — the properties of an atomic node allocation: wall-time, memory, cores, disk, and whether it's preemptible or not 💖
• targetTime (float, /usr/share/doc/python3/html/library/functions.html#float) — the time before which all jobs should at least be started ⏰

Returns

The minimum number of minimal node allocations estimated to be needed to run all the jobs in jobShapes ✨

Attributes

• nodeReservations: dict[toil.provisioners.abstractProvisioner.Shape, list[NodeReservation]] (with dict at /usr/share/doc/python3/html/library/stdtypes.html#dict and list at /usr/share/doc/python3/html/library/stdtypes.html#list)
• nodeShapes
• targetTime = 1800

Methods

binPack(jobShapes)

Packs a list of jobShapes into the fewest nodes reasonable 🍓 You can run it multiple times.

It returns any distinct Shapes that didn't fit, each mapped to the reasons they didn't fit.

Parameters

• jobShapes (list[toil.provisioners.abstractProvisioner.Shape])

Return type

dict[toil.provisioners.abstractProvisioner.Shape, list[FailedConstraint]]

addJobShape(jobShape)

Adds the job to the first node reservation where it will fit. This is the bin-packing part, so it's lowkey the heart of the whole thing 💜

It returns the job shape again, plus a list of failed constraints, if it didn't fit.

Parameters

• jobShape (toil.provisioners.abstractProvisioner.Shape)

Return type

Optional[tuple[toil.provisioners.abstractProvisioner.Shape, list[FailedConstraint]]]

getRequiredNodes()

Returns a dict from node shape to the number of nodes needed to run the packed jobs 🌈

Return type

dict[toil.provisioners.abstractProvisioner.Shape, int]

✨ class toil.provisioners.clusterScaler.NodeReservation(shape) ✨

This is the amount of resources we expect to be free on a given node at each point in time 🧚 To represent what's available in a reservation, it's stored as a linked list of NodeReservations, where each one gives the resources free within a single timeslice. Iconic design honestly 👑

Parameters

• shape (toil.provisioners.abstractProvisioner.Shape)

Attributes

• shape
• nReservation: NodeReservation | None = None

Methods

__str__()

Return type

str (with /usr/share/doc/python3/html/library/stdtypes.html#str)

get_failed_constraints(job_shape)

Checks whether a job shape's resource requirements will fit within this allocation. If the job doesn't fit, it returns the failing constraints: the resources that can't be accommodated, and the limits that were hit 💀

If the job does fit, it returns an empty list.

This one must always agree with fits()! This codepath is slower and is used for diagnosis, so the two have to match. Don't let them disagree, bestie 🙈

Parameters

• job_shape (toil.provisioners.abstractProvisioner.Shape)

Return type

list[FailedConstraint]

fits(jobShape)

Checks whether a job shape's resource requirements will fit within this allocation ✅

Parameters

• jobShape (toil.provisioners.abstractProvisioner.Shape)

Return type

bool (with /usr/share/doc/python3/html/library/functions.html#bool)

shapes()

Gets all time-slice shapes, in order, from this reservation onward 🦋

Return type

list[toil.provisioners.abstractProvisioner.Shape]

subtract(jobShape)

Subtracts the resources needed to run a jobShape from the reservation 🫶

Parameters

• jobShape (toil.provisioners.abstractProvisioner.Shape)

Return type

None

attemptToAddJob(jobShape, nodeShape, targetTime)

Tries to pack a job into this reservation timeslice and/or the reservations after it. jobShape is the Shape of the job requirements, nodeShape is the Shape of the node this is a reservation for, and targetTime is the maximum time to wait before starting this job ⏳

Parameters

• jobShape (toil.provisioners.abstractProvisioner.Shape)
• nodeShape (toil.provisioners.abstractProvisioner.Shape)
• targetTime (float, /usr/share/doc/python3/html/library/functions.html#float)

Return type

bool (with /usr/share/doc/python3/html/library/functions.html#bool)

✨ toil.provisioners.clusterScaler.adjustEndingReservationForJob(reservation, jobShape, wallTime) ✨

Adds a job to an ending reservation that ends at wallTime, splitting the reservation if the job doesn't fill the whole timeslice 🔥

Parameters

• reservation (NodeReservation)
• jobShape (toil.provisioners.abstractProvisioner.Shape)
• wallTime (float, /usr/share/doc/python3/html/library/functions.html#float)

Return type

None

✨ toil.provisioners.clusterScaler.split(nodeShape, jobShape, wallTime) ✨

Partitions a node allocation into two to fit the job. It returns the modified shape of the node and a new node reservation for the extra time the job didn't fill 🧁

Parameters

• nodeShape (toil.provisioners.abstractProvisioner.Shape)
• jobShape (toil.provisioners.abstractProvisioner.Shape)
• wallTime (float, /usr/share/doc/python3/html/library/functions.html#float)

Return type

tuple[toil.provisioners.abstractProvisioner.Shape, NodeReservation] (with tuple at /usr/share/doc/python3/html/library/stdtypes.html#tuple)

✨ toil.provisioners.clusterScaler.binPacking(nodeShapes, jobShapes, goalTime) ✨

Using the given node shape bins, this packs the given job shapes into nodes so they get done in the given amount of time 🌸

It returns a dict saying how many of each node will be needed, and a dict from job shapes that couldn't fit to the reasons why 💅

Parameters

• nodeShapes — list[toil.provisioners.abstractProvisioner.Shape] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🦋
• jobShapes — list[toil.provisioners.abstractProvisioner.Shape] (/usr/share/doc/python3/html/library/stdtypes.html#list) 🌸
• goalTime — float (/usr/share/doc/python3/html/library/functions.html#float) ⭐

🦋 Return type

tuple[dict[toil.provisioners.abstractProvisioner.Shape, int], dict[toil.provisioners.abstractProvisioner.Shape, list[FailedConstraint]]]

Type docs, in case you want to peek: tuple (/usr/share/doc/python3/html/library/stdtypes.html#tuple), dict (/usr/share/doc/python3/html/library/stdtypes.html#dict), int (/usr/share/doc/python3/html/library/functions.html#int), list (/usr/share/doc/python3/html/library/stdtypes.html#list).

---

🌸 class toil.provisioners.clusterScaler.ClusterScaler(provisioner, leader, config)

🎀 Parameters

• provisioner — toil.provisioners.abstractProvisioner.AbstractProvisioner 💅
• leader — toil.leader.Leader 🫶
• config — toil.common.Config ✨

💖 Attributes

These are the bits the scaler keeps track of for you, bestie:

• provisioner
• leader
• config
• static — dict[bool, dict[str, toil.provisioners.node.Node]] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/functions.html#bool, /usr/share/doc/python3/html/library/stdtypes.html#str)
• on_too_big — list[Callable[[toil.provisioners.abstractProvisioner.Shape, list[toil.provisioners.abstractProvisioner.Shape]], Any]] = [] (/usr/share/doc/python3/html/library/stdtypes.html#list) 👑
• jobNameToAvgRuntime — dict[str, float] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/functions.html#float) 🔥
• jobNameToNumCompleted — dict[str, int] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/functions.html#int) 🧚
• totalAvgRuntime = 0.0 🦋
• totalJobsCompleted = 0 🍓
• targetTime — float (/usr/share/doc/python3/html/library/functions.html#float) ⭐
• betaInertia 🌈
• nodeShapeToType 💜
• instance_types 🫶
• nodeShapes 🌸
• ignoredNodes — set[str] (/usr/share/doc/python3/html/library/stdtypes.html#set, /usr/share/doc/python3/html/library/stdtypes.html#str) 🙈
• preemptibleNodeDeficit 🦋
• previousWeightedEstimate ✨
• minNodes 🎀
• maxNodes 🎀
• node_shapes_after_overhead 💅
• without_overhead 💖

🌈 getAverageRuntime(jobName, service=False)

Gets the average runtime for a job name. Lowkey the main thing to know here is how the arguments work.

Parameters

• jobName — str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• service — bool (/usr/share/doc/python3/html/library/functions.html#bool)

Return type

float (/usr/share/doc/python3/html/library/functions.html#float)

💅 addCompletedJob(job, wallTime)

Okay so this one adds the shape of a completed job to the queue, which lets the scaler look at the last N completed jobs when it figures out how many nodes the cluster needs. It's giving smart planning 🧠✨

:param toil.job.JobDescription job: The description of the completed job

:param int wallTime: The wall-time taken to complete the job in seconds.

Parameters

• job — toil.job.JobDescription
• wallTime — int (/usr/share/doc/python3/html/library/functions.html#int)

Return type

None

🫶 setStaticNodes(nodes, preemptible)

This is used to track statically provisioned nodes, and it has to run before any auto-scaled nodes get provisioned. No cap, the order matters here 🔥

These nodes are treated differently from auto-scaled nodes. They should not be automatically terminated, so don't expect the scaler to clean them up for you 💀

Parameters

• nodes — list[toil.provisioners.node.Node] (/usr/share/doc/python3/html/library/stdtypes.html#list) -- a list of Node objects
• preemptible — bool (/usr/share/doc/python3/html/library/functions.html#bool)

Return type

None

🧚 getStaticNodes(preemptible)

Gets back the nodes you set with setStaticNodes(). Iconic recall energy 🦋

Parameters

• preemptible — bool (/usr/share/doc/python3/html/library/functions.html#bool)

Returns

Statically provisioned nodes.

Return type

dict[str, toil.provisioners.node.Node] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)

🌸 smoothEstimate(nodeShape, estimatedNodeCount)

This smooths out the wobbles in the estimate for a node compared to previous runs, so the cluster doesn't freak out over every little spike 🌊 It returns an integer.

Parameters

• nodeShape — toil.provisioners.abstractProvisioner.Shape
• estimatedNodeCount — int (/usr/share/doc/python3/html/library/functions.html#int)

Return type

int (/usr/share/doc/python3/html/library/functions.html#int)

👑 getEstimatedNodeCounts(queuedJobShapes, currentNodeCounts)

Given the resource needs of the queued jobs and how big the cluster is right now, this gives you two things. First, a dict mapping each node shape to the number of nodes we want in the cluster right now. Second, a dict from job shapes that are too big to run on any node to the reasons why. The reasons are the receipts 🧾💅

Parameters

• queuedJobShapes — list[toil.provisioners.abstractProvisioner.Shape] (/usr/share/doc/python3/html/library/stdtypes.html#list)
• currentNodeCounts — dict[toil.provisioners.abstractProvisioner.Shape, int] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/functions.html#int)

Return type

tuple[dict[toil.provisioners.abstractProvisioner.Shape, int], dict[toil.provisioners.abstractProvisioner.Shape, list[FailedConstraint]]]

Type docs: tuple (/usr/share/doc/python3/html/library/stdtypes.html#tuple), dict (/usr/share/doc/python3/html/library/stdtypes.html#dict), int (/usr/share/doc/python3/html/library/functions.html#int), list (/usr/share/doc/python3/html/library/stdtypes.html#list).

✨ updateClusterSize(estimatedNodeCounts)

This one attempts to launch or remove instances so the cluster lands on the desired size. It also tries to remove any ignored nodes that were marked for graceful removal, so nobody gets left behind 🫶 It returns the new size of the cluster.

Parameters

• estimatedNodeCounts — dict[toil.provisioners.abstractProvisioner.Shape, int] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/functions.html#int)

Return type

dict[toil.provisioners.abstractProvisioner.Shape, int] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/functions.html#int)

🔥 setNodeCount(instance_type, numNodes, preemptible=False, force=False)

This tries to grow or shrink the number of preemptible or non-preemptible worker nodes to the value you give it, or as close as it can get. After it makes the additions or removals, it returns the resulting number of preemptible or non-preemptible nodes currently in the cluster. It's basically a glow-up or a glow-down for your cluster 💅✨

Parameters

• instance_type — str (/usr/share/doc/python3/html/library/stdtypes.html#str) -- The instance type to add or remove.
• numNodes (int) — the desired size of the cluster 🎯 basically the number of nodes you're aiming for ✨
• preemptible (bool) — whether the added nodes will be preemptible 🦋. That means the underlying platform can remove them spontaneously at any time, no warning 💀
• force (bool) — If False, the provisioner is allowed to deviate from the number you asked for. When downsizing, for example, it might leave nodes running if they still have active jobs on them 🫶

Returns

The number of worker nodes in the cluster after the necessary adjustments are made 💅 This should be, but is not guaranteed to be, close or equal to numNodes. It's the closest possible approximation of the actual cluster size at the moment this method returns ✨

Return type

int

filter_out_static_nodes(nodes, preemptible=False)

Parameters

• nodes (dict[toil.provisioners.node.Node, toil.batchSystems.abstractBatchSystem.NodeInfo]) — a dict of Node keys paired with NodeInfo values 🗂️
• preemptible (bool) 🦋

Return type

list[tuple[toil.provisioners.node.Node, toil.batchSystems.abstractBatchSystem.NodeInfo]]

getNodes(preemptible=None)

This method returns a dictionary mapping node identifiers of preemptible or non-preemptible nodes to NodeInfo objects, one for each node 🫧

This method is the definitive source on nodes in the cluster 👑 It's responsible for consolidating cluster state between the provisioner and the batch system. Iconic behavior, honestly.

Parameters

• preemptible (bool) — If True (or False), only the preemptible (or non-preemptible) nodes will be returned. If None, all nodes come back 🌈

Return type

dict[toil.provisioners.node.Node, toil.batchSystems.abstractBatchSystem.NodeInfo]

shutDown()

Return type

None

exception toil.provisioners.clusterScaler.JobTooBigError(job=None, shape=None, constraints=None)

Bases: Exception

Raised in the scaler thread when a job cannot fit in any available node type and is likely to lock up the workflow. Yikes 💀 this one is a red flag 🚩

Parameters

• job (Optional[toil.job.JobDescription])
• shape (Optional[toil.provisioners.abstractProvisioner.Shape])
• constraints (Optional[list[FailedConstraint]])

Attributes

• job = None
• shape = None
• constraints
• msg = ''

__str__()

Stringifies the exception, including the message 🎀

Return type

str

class toil.provisioners.clusterScaler.ScalerThread(provisioner, leader, config, stop_on_exception=False)

Bases: toil.lib.threading.ExceptionalThread

A thread that automatically scales the number of either preemptible or non-preemptible worker nodes according to the resource requirements of the queued jobs 📈 it's giving auto-scaling main character energy 💖

The scaling calculation is essentially as follows: start with 0 estimated worker nodes. For each queued job, check if we expect it can be scheduled into a worker node before a certain time (currently one hour). Otherwise, attempt to add a single new node of the smallest type that can fit that job 🧚

At each scaling decision point, a comparison between the current, C, and newly estimated number of nodes is made. If the absolute difference is less than beta * C, then no change is made. Otherwise, the size of the cluster is adapted. The beta factor is an inertia parameter that prevents continual fluctuations in the number of nodes 🦋 no yo-yo-ing allowed 🙅‍♀️

Parameters

• provisioner (toil.provisioners.abstractProvisioner.AbstractProvisioner)
• leader (toil.leader.Leader)
• config (toil.common.Config)
• stop_on_exception (bool)

Attributes

• scaler
• stop = False
• stop_on_exception = False
• stats = None

check()

Attempts to join any existing scaler threads that may have died or finished 🧹 This ensures any exceptions raised in the threads are propagated in a timely fashion ⏰

Return type

None

shutdown()

Shuts down the cluster 🛑

Return type

None

addCompletedJob(job, wallTime)

Parameters

• job (toil.job.JobDescription)
• wallTime (int)

Return type

None

tryRun()

Return type

None

class toil.provisioners.clusterScaler.ClusterStats(path, batchSystem, clusterName)

Parameters

• path (str)
• batchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem)
• clusterName (Optional[str])

Attributes

• stats: dict[str, dict[str, list[dict[str, Any]]]]
• statsThreads: list[toil.lib.threading.ExceptionalThread] = []
• statsPath
• stop = False
• clusterName
• batchSystem
• scalable

shutDownStats()

Return type

None

startStats(preemptible)

Parameters

• preemptible (bool)

Return type

None

checkStats()

Return type

None

toil.provisioners.gceProvisioner

✨ Attributes ✨

• logger

✨ Classes ✨

• GCEProvisioner — Implements a Google Compute Engine Provisioner using libcloud ☁️

✨ Module Contents ✨

toil.provisioners.gceProvisioner.logger

class toil.provisioners.gceProvisioner.GCEProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey, enable_fuse)

Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner

Implements a Google Compute Engine Provisioner using libcloud 💅 the GCE glow-up is real 🌸

Attributes

• NODE_BOTO_PATH = '/root/.boto'
• SOURCE_IMAGE = b'projects/kinvolk-public/global/images/family/flatcar-stable'
• cloud = 'gce'

supportedClusterTypes()

Gets all the cluster types that this provisioner implementation supports 🧾

createClusterSettings()

Initializes the class for a new cluster, to be deployed when running outside the cloud ☁️✨

🌸 readClusterSettings()

readClusterSettings()

Okay so this one reads the cluster settings straight off the instance, and that instance has to be the leader 👑. If you want the deets on reading metadata, check out https://cloud.google.com/compute/docs/storing-retrieving-metadata 💅

✨ launchCluster(leaderNodeType, leaderStorage, owner, **kwargs)

launchCluster(leaderNodeType, leaderStorage, owner, **kwargs)

Bestie, this one gets all the usual stuff from abstractProvisioner 🫶 and then adds its own Google-only extras. Here's what it takes on top:

• keyName — the key used to talk to the instances 🔑
• botoPath — boto credentials for reading an AWS jobStore (optional) ☁️
• network — a network (optional) 🌐
• vpcSubnet — a subnet (optional) 🧚
• use_private_ip — even though a public IP exists, ignore it (optional) 🙈

🦋 getNodeShape(instance_type, preemptible=False)

getNodeShape(instance_type, preemptible=False)

This gives you the shape of a preemptible or non-preemptible node managed by this provisioner. The shape is the tea on the machine 🍵: stuff like how many cores it has, or how long between billing intervals.

Parameters

• instance_type (str) — the instance type name you want the shape of ✨

Return type

toil.provisioners.abstractProvisioner.Shape

💀 static retryPredicate(e)

static retryPredicate(e)

Not used by GCE. Not me, not this method 🙅‍♀️

🔥 destroyCluster()

destroyCluster()

This one tries a few times to terminate all of the instances in the group. Persistence is a whole vibe 💪

Return type

None

🍓 terminateNodes(nodes)

terminateNodes(nodes)

Terminates the nodes represented by the given Node objects. Bye bye 👋

Parameters

• nodes — a list of Node objects 🌸

💜 addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

This is how you add worker nodes to the cluster. It's giving growth arc 🌱

Parameters

• numNodes — how many nodes to add 🔢
• preemptible — whether or not the nodes will be preemptible ⚡
• spotBid — the bid for preemptible nodes, if it applies (this can be set in config, also) 💰
• nodeTypes (set[str]) — the node types 🧾

Returns

The number of nodes successfully added ✅

Return type

int

🧚 getProvisionedWorkers(instance_type=None, preemptible=None)

getProvisionedWorkers(instance_type=None, preemptible=None)

This gets all the nodes from the provisioner, optionally filtered by instance type or preemptability. It includes both static and autoscaled nodes, so nobody gets left out 🫶

Parameters

• preemptible (Optional[bool]) — a boolean to narrow it down to preemptible nodes or non-preemptible nodes 🎯
• instance_type (Optional[str]) — the instance type to filter by 🔍

Returns

A list of Node objects 📋

👑 getLeader()

getLeader()

Returns the leader node. The main character 👑

Returns

The leader node.

🌈 DEFAULT_TASK_COMPLETION_TIMEOUT = 180

DEFAULT_TASK_COMPLETION_TIMEOUT = 180

💅 ex_create_multiple_nodes(...)

ex_create_multiple_nodes(base_name, size, image, number,
location=None, ex_network='default', ex_subnetwork=None,
ex_tags=None, ex_metadata=None, ignore_errors=True,
use_existing_disk=True, poll_interval=2, external_ip='ephemeral',
ex_disk_type='pd-standard', ex_disk_auto_delete=True,
ex_service_accounts=None,
timeout=DEFAULT_TASK_COMPLETION_TIMEOUT, description=None,
ex_can_ip_forward=None, ex_disks_gce_struct=None,
ex_nic_gce_struct=None, ex_on_host_maintenance=None,
ex_automatic_restart=None, ex_image_family=None,
ex_preemptible=None)

Okay so this is a monkey patch to gce.py in libcloud 🐒 that lets you specify disks and images. The name got switched over to a uuid too. The prefix wp marks preemptible nodes and wn marks non-preemptible nodes, so you can always tell who's who 🦋

---

✨ toil.provisioners.node ✨

✨ Attributes ✨

┌──────────────┬───┐
│ a_short_time │   │
├──────────────┼───┤
│ logger       │   │
└──────────────┴───┘

✨ Classes ✨

┌──────┬───┐
│ Node │   │
└──────┴───┘

✨ Module Contents ✨

toil.provisioners.node.a_short_time = 5

toil.provisioners.node.logger

🌸 Node(publicIP, privateIP, name, launchTime, nodeType, preemptible, tags=None, use_private_ip=None)

class toil.provisioners.node.Node(publicIP, privateIP, name, launchTime,
nodeType, preemptible, tags=None, use_private_ip=None)

Parameters

• publicIP (str) 🌍
• privateIP (str) 🏠
• name (str) 🏷️
• launchTime (Union[datetime.datetime, str]) ⏰
• nodeType (Optional[str]) 🧩
• preemptible (bool) ⚡
• tags (Optional[dict[str, str]]) 🎀
• use_private_ip (Optional[bool]) 🔒

✨ Attributes ✨

• maxWaitTime = 420
• publicIP
• privateIP
• name
• nodeType
• preemptible
• tags = None

✨ Methods ✨

• __str__()
• __repr__()
• __hash__()

🦋 remainingBillingInterval()

remainingBillingInterval()

This returns a float from 0 to 1.0 showing how much time is left in the current billing cycle for the instance. If it returns .25, you're three quarters into the cycle, with one quarter left before you get charged again 💸. It assumes a billing cycle of one hour ⏳.

Returns

A float from 1.0 down to 0.0 showing the fraction of pre-paid time left in the cycle.

Return type

float

Here's how it behaves in practice 👀:

>>> node = Node("127.0.0.1", "127.0.0.1", "localhost",
...             datetime.datetime.utcnow(), None, False)
>>> node.remainingBillingInterval() >= 0
True
>>> node.remainingBillingInterval() <= 1.0
True
>>> node.remainingBillingInterval() > 0.5
True
>>> interval1 = node.remainingBillingInterval()
>>> time.sleep(1)
>>> interval2 = node.remainingBillingInterval()
>>> interval2 < interval1
True

>>> node = Node("127.0.0.1", "127.0.0.1", "localhost",
...             datetime.datetime.now(datetime.timezone.utc) -
...             datetime.timedelta(minutes=5), None, False)
>>> node.remainingBillingInterval() < 0.99
True
>>> node.remainingBillingInterval() > 0.9
True

🌷 waitForNode(role, keyName='core')

waitForNode(role, keyName='core')

Parameters

• role (str) 🎭
• keyName (str) 🔑

Return type

None

🫧 copySshKeys(keyName)

copySshKeys(keyName)

Copies the authorized_keys file to the core user from the keyName user 🗝️

🌊 injectFile(fromFile, toFile, role)

injectFile(fromFile, toFile, role)

rysnc a file to the container with the given role 📦

🌊 extractFile(fromFile, toFile, role)

extractFile(fromFile, toFile, role)

rysnc a file from the container with the given role 📤

🐳 sshAppliance(*args, **kwargs)

sshAppliance(*args, **kwargs)

Parameters

• args — arguments to execute in the appliance ⚙️
• kwargs — tty=bool tells docker whether or not to create a TTY shell for interactive SSHing. The default value is False. Input=string is passed as input to the Popen call 💻

🖥️ sshInstance(*args, **kwargs)

sshInstance(*args, **kwargs)

Runs a command on the instance and returns the binary output of the command 🧃

🔐 coreSSH(*args, **kwargs)

coreSSH(*args, **kwargs)

If strict=False, strict host key checking gets temporarily disabled. This is provided as a convenience for internal and automated functions, but it ought to be set to True whenever feasible, or whenever a user is directly interacting with a resource (e.g. rsync-cluster or ssh-cluster). It's assumed to be False by default 🙏.

kwargs: input, tty, appliance, collectStdout, sshOptions, strict

Parameters

• input (bytes) — UTF-8 encoded input bytes to send to the command 📨

🐝 coreRsync(args, applianceName='toil_leader', **kwargs)

coreRsync(args, applianceName='toil_leader', **kwargs)

Parameters

• args (list[str]) — a list of strings 🎀
• applianceName (str) — a string naming the appliance 🧚
• kwargs (Any) — anything goes here, any type 🌈

Return type: int 🔢

✨ Attributes ✨

• logger 🪵 — no description given, just vibes

💅 Exceptions 💅

Okay, these are the error classes you'll meet, and they are so specific 🙈

• NoSuchClusterException 💀 — Indicates that the specified cluster does not exist.
• NoSuchZoneException 🌍 — Indicates that a valid zone could not be found.
• ClusterTypeNotSupportedException 🚫 — Indicates that a provisioner does not support a given cluster type.
• ClusterCombinationNotSupportedException 🙅 — Indicates that a provisioner does not support making a given type of cluster with a given architecture.

🌸 Functions 🌸

Here's the lineup of helper functions, bestie 🫶

• cluster_factory(provisioner[, clusterName, ...]) 🏭 — Finds and spins up the right provisioner instance to make clusters in the given cloud. ✨
• add_provisioner_options(parser) 🛠️ — no description, but it's giving setup energy
• parse_node_types(node_type_specs) 🧩 — Parses a specification for zero or more node types.
• check_valid_node_types(provisioner, node_types) 🔍 — Raises if an invalid nodeType is specified for aws or gce. 😭

🧁 Package Contents 🧁

The package is giving a whole family of goodies 🍓

• toil.provisioners.logger 🪵

🚀 toil.provisioners.cluster_factory

toil.provisioners.cluster_factory(provisioner, clusterName=None,
clusterType='mesos', zone=None, nodeStorage=50,
nodeStorageOverrides=None, sseKey=None, enable_fuse=False)

This is the main event, bestie 👑 It finds and instantiates the appropriate provisioner instance to make clusters in the given cloud.

Heads up! It raises ClusterTypeNotSupportedException if the given provisioner does not implement clusters of the given type. 🚨

Parameters

• provisioner (str) — the cloud type of the cluster ☁️
• clusterName (Optional[str]) — the name of the cluster 🏷️
• clusterType (str) — the type of cluster: 'mesos' or 'kubernetes' 🐳
• zone (Optional[str]) — the cloud zone 📍
• nodeStorage (int) — no description, but it's a number 🔢
• nodeStorageOverrides (Optional[list[str]]) — no description, just a list of strings 📋
• sseKey (Optional[str]) — no description, a string 🔐
• enable_fuse (bool) — no description, a yes-or-no switch 🎚️

Returns

A cluster object for the cloud type 🎉

Return type

Union[aws.awsProvisioner.AWSProvisioner, gceProvisioner.GCEProvisioner] 🦋

🧰 toil.provisioners.add_provisioner_options(parser)

Parameters

• parser (argparse.ArgumentParser) 🧾

Return type

None 🫥

🧩 toil.provisioners.parse_node_types(node_type_specs)

Parses a specification for zero or more node types, and it's way more fun than it sounds 💖

Takes a comma-separated list of node types. Each node type is a slash-separated list of at least one instance type name (like 'm5a.large' for AWS), plus an optional bid in dollars after a colon. 💸

Heads up! It raises ValueError if a node type cannot be parsed. 😵

Inputs should look something like this, iykyk 👀

>>> types = parse_node_types('c5.4xlarge/c5a.4xlarge:0.42,t2.large')
>>> sorted(types[0][0]), types[0][1]
(['c5.4xlarge', 'c5a.4xlarge'], 0.42)
>>> sorted(types[1][0]), types[1][1]
(['t2.large'], None)

Parameters

• node_type_specs (Optional[str]) — a string defining node types 📝

Returns

A list of node types, where each type is the set of instance types plus the float bid, or None. 🧺

Return type

list[tuple[set[str], Optional[float]]] 🌈

🔍 toil.provisioners.check_valid_node_types(provisioner, node_types)

Raises if an invalid nodeType is specified for aws or gce. It's the bouncer of the group, bestie 💅

Parameters

• provisioner (str) — 'aws' or 'gce', to specify which cloud provisioner is used ☁️
• node_types (list[tuple[set[str], Optional[float]]]) — a list of node types. Example: [({'t2.micro'}, None), ({'t2.medium'}, 0.5)] 📋

Returns

Nothing. It raises if any instance type in the node type isn't real. 🙅‍♀️

💥 toil.provisioners.NoSuchClusterException(cluster_name)

Inherits from Exception. 🧬

Indicates that the specified cluster does not exist. 💀

Parameters

• cluster_name (str) 🏷️

💥 toil.provisioners.NoSuchZoneException

Inherits from Exception. 🧬

Indicates that a valid zone could not be found. 🌍

💥 toil.provisioners.ClusterTypeNotSupportedException(provisioner_class, cluster_type)

Inherits from Exception. 🧬

Indicates that a provisioner does not support a given cluster type. 🚫

💥 toil.provisioners.ClusterCombinationNotSupportedException(provisioner_class, cluster_type, architecture, reason=None)

Inherits from Exception. 🧬

Indicates that a provisioner does not support making a given type of cluster with a given architecture. 🙅

Parameters

• provisioner_class (type) 🧱
• cluster_type (str) 🐳
• architecture (str) 🖥️
• reason (type Optional[str], see /usr/share/doc/python3/html/library/stdtypes.html#str) 💅

---

✨ toil.realtimeLogger ✨

Okay bestie, this module is giving real-time logging vibes 📡🎀. It's a UDP-based logging system so your user scripts can debug live while the jobs are running 💖.

🦋 Attributes

• logger, which shows up in the attributes list with nothing next to it 🙈

🌸 Classes

• LoggingDatagramHandler: receives logging messages from the jobs and displays them on the leader 📥
• JSONDatagramHandler: sends logging records over UDP, serialized as JSON 📤
• RealtimeLoggerMetaclass: the metaclass for RealtimeLogger that lets you add logging methods 🧚
• RealtimeLogger: provides a logger that logs over UDP to the leader 👑

💜 Module Contents

toil.realtimeLogger.logger

class toil.realtimeLogger.LoggingDatagramHandler(request, client_address, server)

Bases: socketserver.BaseRequestHandler (see /usr/share/doc/python3/html/library/socketserver.html#socketserver.BaseRequestHandler)

This one receives logging messages from the jobs and displays them on the leader. It uses bare JSON message encoding, so no fancy stuff, just vibes 🫶.

handle(): handles a single message. SocketServer takes care of splitting out the messages for you ✨. The messages are JSON-encoded logging module records.

• Return type: None

class toil.realtimeLogger.JSONDatagramHandler(host, port)

Bases: logging.handlers.DatagramHandler (see /usr/share/doc/python3/html/library/logging.handlers.html#logging.handlers.DatagramHandler)

This one sends logging records over UDP, serialized as JSON. ⚠️ Each record has to fit in a single UDP datagram, so don't try to log more than 64kb at once 😭.

makePickle(record): actually, it encodes the record as bare JSON instead. No pickles here, bestie 🥒🚫.

• Parameters:
  • record (logging.LogRecord, see /usr/share/doc/python3/html/library/logging.html#logging.LogRecord)
• Return type: bytes (see /usr/share/doc/python3/html/library/stdtypes.html#bytes)

class toil.realtimeLogger.RealtimeLoggerMetaclass

Bases: type (see /usr/share/doc/python3/html/library/functions.html#type)

This is the metaclass for RealtimeLogger that lets you add logging methods. It's giving the same energy as RealtimeLogger.warning(), RealtimeLogger.info(), and so on 🌈.

__getattr__(name): falls back to attributes on the logger 🔁.

• Parameters:
  • name (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: Any

class toil.realtimeLogger.RealtimeLogger(batchSystem, level=defaultLevel)

Bases: no base class is listed here.

This is the main event 👑✨. It provides a logger that logs over UDP to the leader. To use it in a Toil job, you just do this:

>>> from toil.realtimeLogger import RealtimeLogger
>>> RealtimeLogger.info("This logging message goes straight to the leader")

That's literally all a user of Toil needs to do 💅. On the leader, Job.Runner.startToil() automatically starts the UDP server by using an instance of this class as a context manager 🎀.

• Parameters:
  • batchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem, see <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)
  • level (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

Here are the class attributes, bestie 🧚:

• envPrefix = 'TOIL_RT_LOGGING_'
• defaultLevel = 'INFO'
• lock
• loggingServer = None
• serverThread = None
• initialized = 0
• logger = None

classmethod getLogger(): gets the logger that logs real-time to the leader 📡.

Heads up, bestie 🙈: if you use the returned logger on the leader, you'll see the message twice, because it still goes to the normal log handlers too.

• Return type: logging.Logger (see /usr/share/doc/python3/html/library/logging.html#logging.Logger)

__enter__():

• Return type: None

__exit__(exc_type, exc_val, exc_tb):

• Parameters:
  • exc_type (Optional[type[BaseException]], see /usr/share/doc/python3/html/library/functions.html#type and /usr/share/doc/python3/html/library/exceptions.html#BaseException)
  • exc_val (Optional[BaseException], see /usr/share/doc/python3/html/library/exceptions.html#BaseException)
  • exc_tb (Optional[types.TracebackType], see /usr/share/doc/python3/html/library/types.html#types.TracebackType)
• Return type: None

---

🧸 toil.resource 🧸

🦋 Attributes

• logger, which shows up in the attributes list with nothing next to it 🙈

💖 Exceptions

• ResourceException: the common base class for all non-exit exceptions 🚨

🌸 Classes

• Resource: represents a file or directory that gets deployed to each node before any jobs in the user script are invoked 🚀
• FileResource: a resource read from a file on the leader 📄
• DirectoryResource: a resource read from a directory on the leader 📁
• VirtualEnvResource: a resource read from a virtualenv on the leader 🐍
• ModuleDescriptor: a path to a Python module, decomposed into a namedtuple of three elements 🧩

💜 Module Contents

toil.resource.logger

class toil.resource.Resource

Bases: namedtuple('Resource', ('name', 'pathHash', 'url', 'contentHash'))

This represents a file or directory that gets deployed to each node before any jobs in the user script are invoked 🚀. Each instance is a namedtuple with the following elements:

• pathHash: the MD5 (in hexdigest form) of the path to the resource on the leader node. The path, and therefore its hash, is unique within a job store 🔐.
• url: a "file:" or "http:" URL where the resource can be obtained 🔗.
• contentHash: an MD5 checksum of the resource, which allows for validation and caching of resources ✅.

If the resource is a regular file, the type attribute will be 'file' 📄. If the resource is a directory, the type attribute will be 'dir' and the URL will point at a ZIP archive of that directory 🗜️.

Here are the env name prefixes, bestie 🌈:

• resourceEnvNamePrefix = 'JTRES_'
• rootDirPathEnvName = 'JTRES_ROOT'

classmethod create(jobStore, leaderPath): this one saves the content of the file or directory at the given path to the given job store. Then it returns a resource object representing that content, so you can grab it again at a generic, public URL 💅. This method should be invoked on the leader node 👑.

• Parameters:
  • jobStore (toil.jobStores.abstractJobStore.AbstractJobStore, see <#toil.jobStores.abstractJobStore.AbstractJobStore>)
  • leaderPath (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Resource (toil.resource.Resource) ✨ this is the object you get back, bestie 💖

🌸 classmethod prepareSystem()

Okay so this gets your system ready to download and look up resources 🧚 It should only be called on a worker node 🫶 It's idempotent (calling it again changes nothing) but it is not thread-safe, so don't let multiple threads fight over it 💀

• Return type: None

🧹 classmethod cleanSystem()

This one is the clean-up crew 🧽 It removes all the downloaded, localized resources. Iconic behavior, honestly.

• Return type: None

📌 register()

Register this resource so it can be fetched later with lookup(), even from a child process 🦋

• Return type: None

🔍 classmethod lookup(leaderPath)

This gives you back a resource object for a file or directory at the given path on the leader 👑 Call it from the worker. The path does not have to actually exist on the worker, since it's just an ID for the resource inside a Toil instance. If there's no resource for that path, it returns None 😭

• Parameters: leaderPath (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: Optional[Resource] (toil.resource.Resource)

⬇️ download(callback=None)

Downloads this resource from its URL into a file on the local system 💅 Only call it on a worker node, and only after the node has been set up for resource access with prepareSystem().

• Parameters: callback (Optional[Callable[[str], None]], where str is /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: None

📍 property localPath: str

Abstractmethod 🙈

Gets the path to the resource on the worker. The file or directory at this path might or might not exist yet, but calling download() will make sure it does 🌈

• Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

📂 property localDirPath: str

This is the path to the directory that holds the resource on the worker 🍓

• Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🥒 pickle()

• Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🥒 classmethod unpickle(s)

• Parameters: s (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: Resource (toil.resource.Resource)

---

🗂️ class toil.resource.FileResource

Bases: Resource 🎀

A resource that gets read from a file on the leader 📄

📍 property localPath: str

Gets the path to the resource on the worker. The file or directory at that path may or may not exist yet, and download() is what guarantees it does 🌸

• Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

📦 class toil.resource.DirectoryResource

Bases: Resource 🧚

A resource that gets read from a directory on the leader 🗃️ The URL points to a ZIP archive of that directory. Every file in the directory, plus any subdirectories, gets included. The directory can be a package, but it doesn't have to be 💜

📍 property localPath: str

Gets the path to the resource on the worker. The file or directory at that path may or may not exist yet, and calling download() will make sure it does 🦋

• Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🐍 class toil.resource.VirtualEnvResource

Bases: DirectoryResource 🌈

A resource that gets read from a virtualenv on the leader. Every module and package found in the virtualenv's site-packages directory gets included ✨

🧩 class toil.resource.ModuleDescriptor

Bases: namedtuple('ModuleDescriptor', ('dirPath', 'name', 'fromVirtualEnv')) 🫶

Okay this one is so cute. It's a path to a Python module broken down into a namedtuple with three elements:

• dirPath, the path to the directory that should be added to sys.path before importing the module,
• moduleName, the fully qualified name of the module, with leading package names separated by dot and

Here's how it looks in action 👑

>>> import toil.resource
>>> ModuleDescriptor.forModule('toil.resource')
ModuleDescriptor(dirPath='/.../src', name='toil.resource', fromVirtualEnv=False)

>>> import subprocess, tempfile, os
>>> dirPath = tempfile.mkdtemp()
>>> path = os.path.join( dirPath, 'foo.py' )
>>> with open(path,'w') as f:
...     _ = f.write('from toil.resource import ModuleDescriptor\n'
...                 'print(ModuleDescriptor.forModule(__name__))')
>>> subprocess.check_output([ sys.executable, path ])
b"ModuleDescriptor(dirPath='...', name='foo', fromVirtualEnv=False)\n"

>>> from shutil import rmtree
>>> rmtree( dirPath )

Now we test a collision 💀 'collections' is part of the standard library in both Python 2 and 3, so this one is the drama:

>>> dirPath = tempfile.mkdtemp()
>>> path = os.path.join( dirPath, 'collections.py' )
>>> with open(path,'w') as f:
...     _ = f.write('from toil.resource import ModuleDescriptorn'
...                 'ModuleDescriptor.forModule(__name__)')

This should fail and return exit status 1 because it collides with the built-in module 😭

>>> subprocess.call([ sys.executable, path ])
1

Clean up 🧹

>>> rmtree( dirPath )

• Attribute: dirPath: str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• Attribute: name: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🔮 classmethod forModule(name)

Returns an instance of this class that represents the module with the given name ✨ If the name is "__main__", it gets translated into the actual file name of the top-level script, without the .py or .pyc extension. This method assumes the module with that name has already been loaded.

• Parameters: name (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: ModuleDescriptor (toil.resource.ModuleDescriptor)

🏠 property belongsToToil: bool

True if this module is part of the Toil distribution 💖

• Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

💾 saveAsResourceTo(jobStore)

Stores the file containing this module as a resource in the given job store and returns the matching resource object. Sometimes it stores the whole Python package directory hierarchy that contains that file, too 📦 Only call this on a leader node 👑

• Parameters: jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <toil.jobStores.abstractJobStore.AbstractJobStore>)
• Return type: Resource (toil.resource.Resource)

🔎 localize()

Checks whether this module was saved as a resource. If it was, you get back a new module descriptor that points to a local copy of that resource 🫶 Only call it on a worker node. On the leader, this just returns this resource, i.e. self.

• Return type: ModuleDescriptor (toil.resource.ModuleDescriptor)

🔁 globalize()

Undoes what localize() did 🔄

• Return type: ModuleDescriptor (toil.resource.ModuleDescriptor)

🖥️ toCommand()

• Return type: collections.abc.Sequence[str] (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Sequence of str, /usr/share/doc/python3/html/library/stdtypes.html#str)

🖥️ classmethod fromCommand(command)

• Parameters: command (collections.abc.Sequence[str], /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Sequence of str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type: ModuleDescriptor (toil.resource.ModuleDescriptor)

🧃 makeLoadable()

• Return type: ModuleDescriptor (toil.resource.ModuleDescriptor)

📥 load()

• Return type: Optional[types.ModuleType] (/usr/share/doc/python3/html/library/types.html#types.ModuleType)

---

🚨 exception toil.resource.ResourceException

Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception) 😭

This is the common base class for all non-exit exceptions. Catch this one and you've got the whole squad 💅

🌸 toil.server 🌸

bestie, this is the toil.server package, and it's got two submodules that are lowkey the whole vibe ✨

💅 Submodules

• toil.server.api_spec
• toil.server.app

---

🦋 toil.server.app 🦋

🎀 Attributes

• logger

✨ Functions

• parser_with_server_options()
• create_app(args) — okay so this one makes a "connexion.FlaskApp" instance and fills it up with Toil server configurations 🎀
• start_server(args) — gets a Toil server up and running 🔥

💜 Module Contents

toil.server.app.logger

toil.server.app.parser_with_server_options()

• Return type: argparse.ArgumentParser (</usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>)

toil.server.app.create_app(args)

Makes a "connexion.FlaskApp" instance with Toil server configurations baked in 🎀

• Parameters:
  • args (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)
• Return type: connexion.FlaskApp

toil.server.app.start_server(args)

Starts a Toil server 🔥

• Parameters:
  • args (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)
• Return type: None

---

🧚 toil.server.celery_app 🧚

🌈 Attributes

• celery

🌸 Functions

• create_celery_app()

💖 Module Contents

toil.server.celery_app.create_celery_app()

• Return type: celery.Celery

toil.server.celery_app.celery

---

🍓 toil.server.cli 🍓

💜 Submodules

• toil.server.cli.wes_cwl_runner

---

🦋 toil.server.cli.wes_cwl_runner 🦋

🎀 Attributes

• logger

👑 Classes

• WESClientWithWorkflowEngineParameters — a modified version of the WESClient from the wes-service package that…

✨ Functions

• generate_attachment_path_names(paths) — takes a list of path names and hands back a list of names with the common path name stripped out, keeping the original order. no relative paths that traverse up, periodt 💅
• get_deps_from_cwltool(cwl_file[, input_file]) — returns a list of dependencies for the given workflow, pulled from cwltool 🧩
• submit_run(client, cwl_file[, input_file, engine_options]) — given a CWL file, its input files, and an optional list of engine options,…
• poll_run(client, run_id) — returns True if the given workflow run is in a finished state ✅
• print_logs_and_exit(client, run_id) — fetches the workflow logs from the WES server, prints the results, then exits 🚪
• main()

💖 Module Contents

toil.server.cli.wes_cwl_runner.logger

toil.server.cli.wes_cwl_runner.generate_attachment_path_names(paths)

Okay this one is so clever 🧠 It takes a list of path names and returns a list of names with the common path name stripped out, while keeping the input order. That guarantees there are no relative paths that traverse up.

Say we have a CWL workflow where hello.yaml references a file called message.txt:

~/toil/workflows/hello.cwl ~/toil/input_files/hello.yaml
~/toil/input_files/message.txt

You can run it with:

toil-wes-cwl-runner hello.cwl ../input_files/hello.yaml

Then message.txt gets resolved to ../input_files/message.txt. We'd send the workflow file as workflows/hello.cwl, and send the inputs as input_files/hello.yaml and input_files/message.txt 🫶

• Parameters:
  • paths (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) — a list of absolute or relative path names. Relative paths are interpreted as relative to the current working directory.
• Returns: the common path name and a list of minimal path names.
• Return type: tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

class toil.server.cli.wes_cwl_runner.WESClientWithWorkflowEngineParameters(endpoint, auth=None)

• Bases: wes_client.util.WESClient

A modified version of the WESClient from the wes-service package that includes workflow_engine_parameters support 💖

TODO: Propose a PR in wes-service to include workflow_engine_params.

• Parameters:
  • endpoint (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • auth (Optional[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

get_version(extension, workflow_file)

Figures out the version of a .py, .wdl, or .cwl file 🔍

• Parameters:
  • extension (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • workflow_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: str </usr/share/doc/python3/html/library/stdtypes.html#str>

parse_params(workflow_params_file)

Parses the CWL input file into a dictionary, which then gets attached to the body of the WES run request 📦

• Parameters:
  • workflow_params_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>) — the URL or path to the CWL input file.
• Return type: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

modify_param_paths(base_dir, workflow_params)

Makes the file paths in the input workflow parameters relative to base_dir 🗺️

• Parameters:
  • base_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) — the base directory to make the file paths relative to. This should be the common ancestor of all attached files, which will become the root of the execution folder.
• workflow_params (dict[str, Any]) — a dict holding all the workflow parameters 🎀 (the whole vibe of the run lives here, bestie)

Return type — None 💀 it literally gives back nothing

---

💌 build_wes_request

build_wes_request(workflow_file, workflow_params_file, attachments, workflow_engine_parameters=None)

okay so this one is the request builder ✨ it puts together the whole workflow run request that gets sent off to WES 🦋

Parameters

• workflow_file (str) — the path or URL to the CWL workflow document. heads up, only file:// URLs are supported right now 🙈
• workflow_params_file (Optional[str]) — the path or URL to the CWL input file
• attachments (Optional[list[str]]) — a list of local paths to files that get uploaded to the server 📎
• workflow_engine_parameters (Optional[list[str]]) — a list of engine parameters to set along with this workflow run ⚙️

Returns — you get back a dictionary of parameters to use as the request body, plus an iterable of filename and file contents pairs ready to upload to the server 🫶

Return type — tuple[dict[str, str], collections.abc.Iterable[tuple[str, tuple[str, io.BytesIO]]]]

---

🚀 run_with_engine_options

run_with_engine_options(workflow_file, workflow_params_file, attachments, workflow_engine_parameters)

this one whips up a POST request and tells the WES server to start running a workflow 🚀

Parameters

• workflow_file (str) — the path to the CWL workflow document
• workflow_params_file (Optional[str]) — the path to the CWL input file
• attachments (Optional[list[str]]) — a list of local paths to files that get uploaded to the server
• workflow_engine_parameters (Optional[list[str]]) — a list of engine parameters to set along with this workflow run

Returns — the body of the POST result, as a dictionary 💖

Return type — dict[str, Any]

---

🧠 toil.server.cli.wes_cwl_runner.get_deps_from_cwltool

toil.server.cli.wes_cwl_runner.get_deps_from_cwltool(cwl_file, input_file=None)

this one asks cwltool for the list of dependencies the workflow needs and hands it back 🌸

Parameters

• cwl_file (str) — the CWL file
• input_file (Optional[str]) — leave it out to get the dependencies from the CWL file. If you set it, you get the dependencies from the input file instead 🎯

Return type — list[str]

---

🔍 toil.server.cli.wes_cwl_runner.submit_run

toil.server.cli.wes_cwl_runner.submit_run(client, cwl_file, input_file=None, engine_options=None)

okay bestie, so you hand it a CWL file, its input files, and an optional list of engine options, and it submits the CWL workflow to the WES server through the WES client 💅

this function also tries to dig up the attachments from the CWL workflow and its input file, then attaches them to the WES run request 📎✨

Parameters

• client (WESClientWithWorkflowEngineParameters) — the WES client
• cwl_file (str) — the path to the CWL workflow document
• input_file (Optional[str]) — the path to the CWL input file
• engine_options (Optional[list[str]]) — a list of engine parameters to set along with this workflow run

Return type — str

---

👀 toil.server.cli.wes_cwl_runner.poll_run

toil.server.cli.wes_cwl_runner.poll_run(client, run_id)

this one checks in on a run and tells you if it's in a finished state (True) or not 👀

Parameters

• client (WESClientWithWorkflowEngineParameters)
• run_id (str)

Return type — bool

---

📜 toil.server.cli.wes_cwl_runner.print_logs_and_exit

toil.server.cli.wes_cwl_runner.print_logs_and_exit(client, run_id)

this one fetches the workflow logs from the WES server and prints them out 🖨️ then it exits the program with the same exit code as the workflow run 🏁

Parameters

• client (WESClientWithWorkflowEngineParameters) — the WES client
• run_id (str) — the run_id of the target workflow

Return type — None

---

🏁 toil.server.cli.wes_cwl_runner.main

toil.server.cli.wes_cwl_runner.main()

the entry point, the main character of the whole file 👑

Return type — None

---

🌈 toil.server.utils

Attributes

• HAVE_S3
• logger
• state_store_cache
• TERMINAL_STATES
• MAX_CANCELING_SECONDS

🎀 Classes 🎀

okay so these are the state-store classes, and they're giving organized 💅

• MemoryStateCache: an in-memory spot to stash workflow state 🧚
• AbstractStateStore: the place where the WES server keeps its state, meaning the set of workflows that 🫶
• MemoryStateStore: an in-memory spot for workflow state, but it's just for testing 🧪
• FileStateStore: stores workflow state on a POSIX-compatible file system 📁
• S3StateStore: stores workflow state in an S3-compatible object store ☁️
• WorkflowStateStore: a slice of a state store that holds the state for one particular workflow 🍰
• WorkflowStateMachine: the class that manages the WES workflow state machine 🌸

🦋 Functions 🦋

the function lineup is so iconic, bestie 🔥

• get_iso_time(): gives you the current time in ISO 8601 format ⏰
• link_file(src, dest): makes a link to a file, going from src to dest 🔗
• download_file_from_internet(src, dest[, content_type]): grabs a file from the Internet and writes it to dest 🌐
• download_file_from_s3(src, dest[, content_type]): grabs a file from Amazon S3 and writes it to dest ☁️
• get_file_class(path): tells you the type of the file as a human readable string 📝
• safe_read_file(file): reads a file safely by grabbing a shared lock, so no other process can write to it while you're reading 🔒
• safe_write_file(file, s): writes to a file safely by grabbing an exclusive lock, so no other process can read or write to it while you're writing 🔐
• connect_to_state_store(url): connects you to a place that stores workflow state, defined by a URL 🔌
• connect_to_workflow_state_store(url, workflow_id): connects you to a place that stores state for the given workflow, in the state 🧩

---

🌈 Module Contents 🌈

okay this is where the real details live, so pay attention bestie 👀

toil.server.utils.HAVE_S3 = True

toil.server.utils.logger

✨ toil.server.utils.get_iso_time() ✨

Gives you the current time in ISO 8601 format ⏰

• Return type: str (</usr/share/doc/python3/html/library/stdtypes.html#str>)

✨ toil.server.utils.link_file(src, dest) ✨

Makes a link to a file, going from src to dest 🔗

• Parameters:
  • src (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • dest (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: None

✨ toil.server.utils.download_file_from_internet(src, dest, content_type=None) ✨

Grabs a file from the Internet and writes it to dest 🌐

• Parameters:
  • src (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • dest (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • content_type (Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: None

✨ toil.server.utils.download_file_from_s3(src, dest, content_type=None) ✨

Grabs a file from Amazon S3 and writes it to dest ☁️

• Parameters:
  • src (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • dest (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • content_type (Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: None

✨ toil.server.utils.get_file_class(path) ✨

Tells you the type of the file as a human readable string 📝

• Parameters:
  • path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: str (</usr/share/doc/python3/html/library/stdtypes.html#str>)

✨ toil.server.utils.safe_read_file(file) ✨

Reads a file safely by grabbing a shared lock, so no other process can write to it while you're reading 🔒

• Parameters:
  • file (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: Optional[str] (</usr/share/doc/python3/html/library/stdtypes.html#str>)

✨ toil.server.utils.safe_write_file(file, s) ✨

Writes to a file safely by grabbing an exclusive lock, so no other process can read or write to it while you're writing 🔐

• Parameters:
  • file (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: None

✨ class toil.server.utils.MemoryStateCache ✨

An in-memory spot to store workflow state 🧚

✨ get(workflow_id, key) ✨

Gets a key value from memory 🔑

• Parameters:
  • workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: Optional[str] (</usr/share/doc/python3/html/library/stdtypes.html#str>)

✨ set(workflow_id, key, value) ✨

Sets or clears a key value in memory 🧹

• Parameters:
  • workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • value (Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>)
• Return type: None

🌸 class toil.server.utils.AbstractStateStore

This is the WES server's memory bank 🧠✨. It keeps track of which workflows exist and whether they're done or not. It's a key-value store with keys namespaced by workflow ID, and concurrent access from multiple threads or processes is safe and globally consistent 🔒💜.

Keys and workflow IDs are restricted to [-a-zA-Z0-9_], because backends may use them as path or URL components 🙅‍♀️.

Key values are either a string, or None if the key is not set.

Workflow existence isn't a thing 💀. Nonexistent workflows just have None for all keys.

⚠️ Note that we don't have a cleanup operation yet. Things are stored permanently, and even clearing all the keys may leave data behind 😭.

This store also handles storage for a local cache, with a separate key namespace. It is not a read/write-through cache 🦋.

TODO: Can we replace this with just using a JobStore eventually, when AWSJobStore no longer needs SimpleDB? 👀

abstractmethod get(workflow_id, key)

Get the value of the given key for the given workflow, or None if the key is not set for the workflow 🫶.

Parameters

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)

abstractmethod set(workflow_id, key, value)

Set the value of the given key for the given workflow. If the value is None, clear the key 🧹.

Parameters

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• value (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

read_cache(workflow_id, key)

Read a value from a local cache, without checking the actual backend 👀.

Parameters

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)

write_cache(workflow_id, key, value)

Write a value to a local cache, without modifying the actual backend 🌈.

Parameters

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• value (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

---

🦋 class toil.server.utils.MemoryStateStore

Bases: MemoryStateCache, AbstractStateStore

An in-memory place to store workflow state, meant for testing 🧪. It inherits from MemoryStateCache first to provide implementations for AbstractStateStore.

---

🍓 class toil.server.utils.FileStateStore(url)

Bases: AbstractStateStore

A place to store workflow state that uses a POSIX-compatible file system 📁.

Parameters

• url (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

get(workflow_id, key)

Get a key value from the filesystem 🗂️.

Parameters

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)

set(workflow_id, key, value)

Set or clear a key value on the filesystem 🧼.

Parameters

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• value (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

---

☁️ class toil.server.utils.S3StateStore(url)

Bases: AbstractStateStore

A place to store workflow state that uses an S3-compatible object store 🪣💖.

Parameters

• url (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

get(workflow_id, key)

Get a key value from S3 ☁️.

Parameters

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)

set(workflow_id, key, value)

Set or clear a key value on S3 🫧.

Parameters

• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• value (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

---

🧚 toil.server.utils.state_store_cache

toil.server.utils.state_store_cache: dict[str, AbstractStateStore]

This is a dict (/usr/share/doc/python3/html/library/stdtypes.html#dict) keyed by str (/usr/share/doc/python3/html/library/stdtypes.html#str), with AbstractStateStore <#toil.server.utils.AbstractStateStore> values. It's the cache of state stores we've already connected to 🗃️.

---

🔗 toil.server.utils.connect_to_state_store(url)

Connect to a place to store state for workflows, defined by a URL 🔌. The URL may be a local file path or URL, or an S3 URL 🌐.

Parameters

• url (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: AbstractStateStore <#toil.server.utils.AbstractStateStore>

---

💅 class toil.server.utils.WorkflowStateStore(state_store, workflow_id)

This is a slice of a state store that holds the state for one particular workflow 🎀.

Parameters

• state_store (AbstractStateStore <#toil.server.utils.AbstractStateStore>)
• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

get(key)

Get the given item of workflow state 👑.

Parameters

• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)

set(key, value)

Set the given item of workflow state ✍️.

Parameters

• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• value (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

read_cache(key)

Read a value from a local cache, without checking the actual backend 👀.

Parameters

• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str)

write_cache(key, value)

Write a value to a local cache, without modifying the actual backend 🌸.

Parameters

• key (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• value (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

---

🔌 toil.server.utils.connect_to_workflow_state_store(url, workflow_id)

Connect to a place to store state for the given workflow, in the state store defined by the given URL 🫶.

Parameters

• url (str, /usr/share/doc/python3/html/library/stdtypes.html#str): a URL that can be used for connect_to_state_store()
• workflow_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: WorkflowStateStore <#toil.server.utils.WorkflowStateStore>

---

🏁 toil.server.utils.TERMINAL_STATES

This is the set of states where a workflow is finished for good 🏁.

🕰️ toil.server.utils.MAX_CANCELING_SECONDS = 30

toil.server.utils.MAX_CANCELING_SECONDS = 30

---

🚦 class toil.server.utils.WorkflowStateMachine(store)

This is the class for managing the WES workflow state machine 🚦. It's the authority on the WES "state" of a workflow, so you need one to read or change the state 👑.

Guaranteeing that only certain transitions can be observed is possible, but not worth it 🙅‍♀️. Instead, we just let updates clobber each other, and we grab and cache the first terminal state we see, forever 🔒. If it ever becomes important that clients never see things like CANCELED -> COMPLETE or COMPLETE -> SYSTEM_ERROR, we can implement a real distributed state machine here 🛠️.

We do handle making sure that tasks don't get stuck in CANCELING 🫧.

Parameters

• store (AbstractStateStore <#toil.server.utils.AbstractStateStore>)

The state can be any of these:

• "UNKNOWN"
• "QUEUED"
• "INITIALIZING"
• "RUNNING"
• "PAUSED"
• "COMPLETE"
• "EXECUTOR_ERROR"
• "SYSTEM_ERROR"
• "CANCELED"
• "CANCELING"

Uses the state store's local cache to prevent needing to read things we've already seen 🧠💖.

Parameters: store (WorkflowStateStore, #toil.server.utils.WorkflowStateStore) 💖

Okay so these are the state-changing messages, bestie 💅 Each one nudges the workflow from one state to the next, and every send_* method returns None 🙈

• send_enqueue(): Sends an enqueue message that moves the workflow from UNKNOWN to QUEUED. Lowkey the very first step 🌸
  • Return type: None
• send_initialize(): Sends an initialize message that moves the workflow from QUEUED to INITIALIZING 🧚
  • Return type: None
• send_run(): Sends a run message that moves the workflow from INITIALIZING to RUNNING 🔥
  • Return type: None
• send_cancel(): Sends a cancel message that moves the workflow to CANCELING from any non-terminal state. It's the "nope, stop" button 🛑
  • Return type: None
• send_canceled(): Sends a canceled message that moves the workflow to CANCELED from CANCELLING 🦋
  • Return type: None
• send_complete(): Sends a complete message that moves the workflow from RUNNING to COMPLETE 👑
  • Return type: None
• send_executor_error(): Sends an executor_error message that moves the workflow from QUEUED, INITIALIZING, or RUNNING to EXECUTOR_ERROR 💀
  • Return type: None
• send_system_error(): Sends a system_error message that moves the workflow from QUEUED, INITIALIZING, or RUNNING to SYSTEM_ERROR 😭
  • Return type: None
• get_current_state(): Gets the current state of the workflow. Main character energy 👑
  • Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🧚 toil.server.wes

Submodules

• toil.server.wes.abstract_backend

toil.server.wes.abstract_backend

Attributes

• logger: No description is given for this one 🤷
• TaskLog: No description is given for this one either 🌈

Exceptions

• VersionNotImplementedException: Raised when the requested workflow version is not implemented. ✨
• MalformedRequestException: Raised when the request is malformed. 🙈
• WorkflowNotFoundException: Raised when the requested run ID is not found. 🔍
• WorkflowConflictException: Raised when the requested workflow is not in the expected state. 💅
• OperationForbidden: Raised when the request is forbidden. 🚫
• WorkflowExecutionException: Raised when an internal error occurred during the execution of the workflow. 💔

Classes

• WESBackend: A class to represent a GA4GH Workflow Execution Service (WES) API backend. 🎀

Functions

• handle_errors(func): A decorator that catches errors from the wrapped function and returns a JSON formatted error message. The full story is in Module Contents below 💜

Module Contents

• toil.server.wes.abstract_backend.logger: No description is given for this one 🤷
• toil.server.wes.abstract_backend.TaskLog: No description is given for this one either 🌈
• exception toil.server.wes.abstract_backend.VersionNotImplementedException(wf_type, version=None, supported_versions=None)
  • Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)
  • Raised when the requested workflow version is not implemented. ✨
  • Parameters:
    • wf_type (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
    • version (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)
    • supported_versions (Optional[list[str]], /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str)
• exception toil.server.wes.abstract_backend.MalformedRequestException(message)
  • Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)
  • Raised when the request is malformed. 🙈
  • Parameters:
    • message (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• exception toil.server.wes.abstract_backend.WorkflowNotFoundException
  • Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)
  • Raised when the requested run ID is not found. 🔍
• exception toil.server.wes.abstract_backend.WorkflowConflictException(run_id)
  • Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)
  • Raised when the requested workflow is not in the expected state. 💅
  • Parameters:
    • run_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• exception toil.server.wes.abstract_backend.OperationForbidden(message)
  • Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)
  • Raised when the request is forbidden. 🚫
  • Parameters:
    • message (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• exception toil.server.wes.abstract_backend.WorkflowExecutionException(message)
  • Bases: Exception (/usr/share/doc/python3/html/library/exceptions.html#Exception)
  • Raised when an internal error occurred during the execution of the workflow. 💔
  • Parameters:
    • message (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• toil.server.wes.abstract_backend.handle_errors(func)
  • This decorator catches errors from the wrapped function and returns a JSON formatted error message with the appropriate status code defined by the GA4GH WES spec. It's giving error handling with a bow on top 🎀
  • Parameters:
    • func (Callable[Ellipsis, Any])
  • Return type: Callable[Ellipsis, Any]
• class toil.server.wes.abstract_backend.WESBackend(options)
  • A class to represent a GA4GH Workflow Execution Service (WES) API backend. Intended to be inherited. Subclasses should implement all abstract methods to handle user requests when they hit different endpoints. 🎀
  • Parameters:
    • options (list[str], /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str)
  • options (attribute, no description given)
  • resolve_operation_id(operation_id): Maps an operationId defined in the OpenAPI or swagger yaml file to a function. Iykyk 🗝️
    • Parameters:
      • operation_id (str, /usr/share/doc/python3/html/library/stdtypes.html#str): The operation ID defined in the specification.
    • Returns: A function that should be called when the given endpoint is reached.
    • Return type: Any
  • abstractmethod get_service_info(): Gets information about the Workflow Execution Service. GET /service-info 🧚
    • Return type: dict[str, Any] (/usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)
  • abstractmethod list_runs(page_size=None, page_token=None): Lists the workflow runs. GET /runs 🍓
    • Parameters:
      • page_size (Optional[int], /usr/share/doc/python3/html/library/functions.html#int)
      • page_token (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)

The stdtypes links point at the Python docs in /usr/share/doc/python3/html/library/stdtypes.html, with anchors #dict, #str, #tuple, #list, and #bytes. 📚💅

---

🎀 abstractmethod run_workflow()

Spin up a brand-new workflow run! ✨ This endpoint creates the run and hands back a RunId so you can keep tabs on how it's doing. 🚀

• Route: POST /runs
• Return type: dict[str, str]

---

🎀 abstractmethod get_run_log(run_id)

Get the full tea on a workflow run. 🍵 It's the detailed info, bestie.

• Route: GET /runs/{run_id}
• Parameters
  • run_id (str)
• Return type: dict[str, Any]

---

🎀 abstractmethod cancel_run(run_id)

Cancel a running workflow. 🛑 Hit the brakes, babe.

• Route: POST /runs/{run_id}/cancel
• Parameters
  • run_id (str)
• Return type: dict[str, str]

---

🎀 abstractmethod get_run_status(run_id)

Need a quick vibe check? 🔍 This one gives you the overall state of a workflow run in a simple result, no deep dive required.

• Route: GET /runs/{run_id}/status
• Parameters
  • run_id (str)
• Return type: dict[str, str]

---

🎀 static log_for_run(run_id, message)

Logs a message for a run. 📝

• Parameters
  • run_id (Optional[str])
  • message (str)
• Return type: None

---

🎀 static secure_path(path)

Takes a path and gives you back a secured version of it. 🔐

• Parameters
  • path (str)
• Return type: str

---

🎀 collect_attachments(args, run_id, temp_dir)

Collect the attachments from the current request 📎 by staging the uploaded files into temp_dir. It hands back the temp_dir and the parsed body of the request, so you get both. 🫶

• Parameters
  • run_id (Optional[str]), the run ID used for logging.
  • temp_dir (Optional[str]), the directory where uploaded files get staged. If it's None, a temporary directory is created for you. 🗂️
  • args (dict[str, Any])
• Return type: tuple[str, dict[str, Any]]

---

🧚 toil.server.wes.amazon_wes_utils

💖 Attributes

• logger
• NOTICE

🌈 Classes

• WorkflowPlan: these functions pass around dicts of a certain type, with data and files keys. 🧁
• DataDict: under data, there can be:
• FilesDict: under files, there can be:

✨ Functions

• parse_workflow_zip_file(file, workflow_type): processes a workflow zip bundle 📦
• parse_workflow_manifest_file(manifest_file): reads a MANIFEST.json file for a workflow zip bundle 📜
• workflow_manifest_url_to_path(url[, parent_dir]): interprets a possibly-relative parsed URL, relative to the given parent directory. 🧭
• task_filter(task, job_status): AGC requires task names to be annotated with an AWS Batch job ID that they… (the rest of this one, bestie) 🙈

---

🔥 Module Contents

• toil.server.wes.amazon_wes_utils.logger
• toil.server.wes.amazon_wes_utils.NOTICE = Multiline-String

"""
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
"""

👑 class toil.server.wes.amazon_wes_utils.WorkflowPlan

Bases: TypedDict

These functions pass around dicts of a certain type, with data and files keys. 💜

• data: DataDict (anchor #toil.server.wes.amazon_wes_utils.DataDict)
• files: FilesDict (anchor #toil.server.wes.amazon_wes_utils.FilesDict)

🌸 class toil.server.wes.amazon_wes_utils.DataDict

Bases: TypedDict

Under data, there can be:

• workflowUrl (required if no workflowSource): URL to main workflow code. 🔗
• workflowUrl: str

🦋 class toil.server.wes.amazon_wes_utils.FilesDict

Bases: TypedDict

Under files, there can be:

• workflowSource (required if no workflowUrl): open binary-mode file for the main workflow code. 📄
• workflowInputFiles: list of open binary-mode files for input files. Expected to be JSONs. 🧾
• workflowOptions: open binary-mode file for a JSON of options sent along with the workflow. ⚙️
• workflowDependencies: open binary-mode file for the zip the workflow came in, if any. 🗜️
• workflowSource: IO[bytes]
• workflowInputFiles: list[IO[bytes]]
• workflowOptions: IO[bytes]
• workflowDependencies: IO[bytes]

🍓 toil.server.wes.amazon_wes_utils.parse_workflow_zip_file(file, workflow_type)

Processes a workflow zip bundle. 🎁

• Parameters
  • file (str): a string or Path-like path to a workflow.zip file. 📂
  • workflow_type (str): a string, the extension of the workflow to expect (e.g. "wdl"). 🏷️
• Return type: dict of data and files

If the zip only contains a single file, that file is set as workflowSource. 🎯

If the zip contains multiple files with a MANIFEST.json file, the MANIFEST is used to determine the appropriate data and file arguments. (See: parse_workflow_manifest_file()) 🗺️

If the zip contains multiple files without a MANIFEST.json file:

• a main workflow file with an extension matching the workflow_type is expected and will be set as workflowSource. 💅
• 🎀 optional bonus: if inputs*.json files are chilling in the root level of the zip, they get set as workflowInputs(_d)*, in whatever order they're found 📦
• ⚙️ optional settings: if an options.json file is in the root level of the zip, it becomes workflowOptions 🧚

If the zip has multiple files, the original zip gets set as workflowDependencies 🦋

✨ toil.server.wes.amazon_wes_utils.parse_workflow_manifest_file(manifest_file) ✨

okay so this one reads a MANIFEST.json file for a workflow zip bundle 📖

• manifest_file (str (/usr/share/doc/python3/html/library/stdtypes.html#str)) — a string or Path-like path to a MANIFEST.json file
• Return type — dict (/usr/share/doc/python3/html/library/stdtypes.html#dict) of data and files

Parameters 💅

MANIFEST.json is expected to look like this 👀

{
    "mainWorkflowURL": "relpath/to/workflow",
    "inputFileURLs": [
        "relpath/to/input-file-1",
        "relpath/to/input-file-2",
        "relpath/to/input-file-3"
    ],
    "optionsFileURL": "relpath/to/option-file"
}

• 🌟 mainWorkflowURL gives a relative file path in the zip to the workflow file, and that becomes workflowSource
• 📚 inputFileURLs is optional and gives a list of relative file paths in the zip to input.json files. The list is assumed to be in the order the inputs should be applied, so a higher list index means higher priority. If it's there, it gets used to set workflowInputs(_d) arguments.
• 🧁 optionsFileURL is optional and gives a relative file path in the zip to an options.json file. If it's there, it gets used to set workflowOptions.

✨ toil.server.wes.amazon_wes_utils.workflow_manifest_url_to_path(url, parent_dir=None) ✨

Interpret a possibly-relative parsed URL, relative to the given parent directory. It's giving relative path energy 🗺️

• url (urllib.parse.ParseResult (/usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult))
• parent_dir (Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str))
• Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

✨ toil.server.wes.amazon_wes_utils.task_filter(task, job_status) ✨

ngl this one is a little dramatic 😭 AGC requires task names to be annotated with the AWS Batch job ID they were run under. If it finds an un-annotated task name, it crashes. See https://github.com/aws/amazon-genomics-cli/issues/494 💀

This encodes the AWSBatchJobID annotation, from the AmazonBatchBatchSystem, into the task name of the given task, then returns the modified task. If there's no annotation available, the task gets censored and None is returned 🙈

• task (toil.server.wes.abstract_backend.TaskLog)
• job_status (toil.bus.JobStatus (#toil.bus.JobStatus))
• Return type — Optional[toil.server.wes.abstract_backend.TaskLog]

---

🌸 toil.server.wes.tasks

💜 Attributes

• logger
• WAIT_FOR_DEATH_TIMEOUT
• run_wes

🦋 Classes

• ToilWorkflowRunner — a class to represent a workflow runner to run the requested workflow.
• TaskRunner — abstraction over the Celery API. It runs our run_wes task and lets you cancel it. 🛑
• MultiprocessingTaskRunner — a version of TaskRunner that just runs tasks with Multiprocessing. 🧚

🔥 Functions

• run_wes_task(base_scratch_dir, state_store_url, ...) — run a requested workflow.
• cancel_run(task_id) — send a SIGTERM signal to the process that's running task_id.

🌈 Module Contents

• toil.server.wes.tasks.logger
• toil.server.wes.tasks.WAIT_FOR_DEATH_TIMEOUT = 20

🎀 class toil.server.wes.tasks.ToilWorkflowRunner(base_scratch_dir, state_store_url, workflow_id, request, engine_options)

A class to represent a workflow runner to run the requested workflow. It's the main character of this whole module 👑

It's responsible for parsing the user request into a shell command, executing that command, and collecting the outputs of the resulting workflow run.

Parameters 💖

• base_scratch_dir (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• state_store_url (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• workflow_id (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• request (dict (/usr/share/doc/python3/html/library/stdtypes.html#dict)[str (/usr/share/doc/python3/html/library/stdtypes.html#str), Any])
• engine_options (list (/usr/share/doc/python3/html/library/stdtypes.html#list)[str (/usr/share/doc/python3/html/library/stdtypes.html#str)])

Attributes 🧁

• scratch_dir
• store
• state_machine
• request
• engine_options
• wf_type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• version: str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• exec_dir
• out_dir
• default_job_store
• job_store

✨ write_scratch_file(filename, contents)

Write a file to the scratch directory. Lowkey the simplest method in the whole class 🌷

• filename (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• contents (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• Return type — None

✨ get_state()

• Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

✨ write_workflow(src_url)

Fetch the workflow file from its source and write it to a destination file. It's giving fetch-and-save vibes 📥

• src_url (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

✨ sort_options(workflow_engine_parameters=None)

Sort the command line arguments into the order the workflow execution engine can recognize. Order matters, bestie 🎀

• workflow_engine_parameters (Optional[dict[str, Optional[str]]]) — the user-specified parameters for this particular workflow. Keys are command-line options, and values are option arguments, or None for options that are flags. 🚩
• Return type — list[str]

✨ initialize_run()

Write the workflow and input files, then construct a list of shell commands to be executed. It returns that list of shell commands that should be run in order to complete this workflow run 🏁

• Return type — list[str]

✨ call_cmd(cmd, cwd)

Calls a command with Popen. It writes stdout, stderr, and the command to separate files 📝

Parameters

• cmd 💖 — this one's a Union[list[str], str], so you can pass a list of strings or one plain string. Doc refs: /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str
• cwd 🌸 — a str. Doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type 🦋 — subprocess.Popen[bytes]. Doc refs: /usr/share/doc/python3/html/library/subprocess.html#subprocess.Popen and /usr/share/doc/python3/html/library/stdtypes.html#bytes

run() ✨ — okay so this one builds a command to run the requested workflow with the options, runs it, and drops the outputs into the output directory. 💅

Return type — None

write_output_files() 🧚 — fetches every file this workflow generated and writes info about each one out to outputs.json. 🍓

Return type — None

---

toil.server.wes.tasks.run_wes_task(base_scratch_dir, state_store_url, workflow_id, request, engine_options)

Run a requested workflow. It's giving main character energy. 👑

Parameters

• base_scratch_dir (str) 🌈 — the directory where the workflow's scratch dir lives, under the workflow's ID. Doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str
• state_store_url (str) 🦋 — the URL or path where the server and the Celery task talk about workflow state. Doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str
• workflow_id (str) 🫶 — the ID of the workflow run. Doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str
• request (dict[str, Any]) 💜 — no description given, just the type. Doc refs: /usr/share/doc/python3/html/library/stdtypes.html#dict and /usr/share/doc/python3/html/library/stdtypes.html#str
• engine_options (list[str]) 🔥 — no description given, just the type. Doc refs: /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str

Returns — the state of the workflow run. Return type — str. Doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

toil.server.wes.tasks.run_wes 🙈 — this one's just a name with no details listed under it, so that's all the tea I've got. 🫠

---

toil.server.wes.tasks.cancel_run(task_id)

Send a SIGTERM signal to the process that is running task_id. ⭐

Parameters

• task_id (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type — None

---

class toil.server.wes.tasks.TaskRunner

Okay bestie, this is the abstraction over the Celery API. It runs our run_wes task and lets you cancel it. 🎀 We can swap it out in the server to allow testing without Celery, which is so smart. 🧠✨

🦋 static run(args, task_id)

Run the given task args with the given ID on Celery.

Parameters

• args (tuple[str, str, str, dict[str, Any], list[str]]) — doc refs: /usr/share/doc/python3/html/library/stdtypes.html#tuple, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#list
• task_id (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type — None

🌸 static cancel(task_id)

Cancel the task with the given ID on Celery. 💅

Parameters

• task_id (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type — None

🧚 static is_ok(task_id)

Make sure the task running system is working for the given task. If the task system has detected an internal failure, it returns False. 🚨

Parameters

• task_id (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type — bool. Doc ref: /usr/share/doc/python3/html/library/functions.html#bool

---

class toil.server.wes.tasks.MultiprocessingTaskRunner

Bases: TaskRunner 🔗

This is the version of TaskRunner that just runs tasks with multiprocessing. 🌈

Can't use threading here, bestie. There's no way to send a cancel signal or exception to a Python thread if loops in the task (i.e. ToilWorkflowRunner) don't poll for it. 💀

🎀 static set_up_and_run_task(output_path, args)

Set up logging for the process into the given file, and then call run_wes_task with the given arguments. 🎀

If the process finishes successfully, it cleans up the log. But if the process crashes, the caller has to clean up the log. 🫶

Parameters

• output_path (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str
• args (tuple[str, str, str, dict[str, Any], list[str]]) — doc refs: /usr/share/doc/python3/html/library/stdtypes.html#tuple, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#list

Return type — None

🦋 classmethod run(args, task_id)

Run the given task args with the given ID. 🔥

Parameters

• args (tuple[str, str, str, dict[str, Any], list[str]]) — doc refs: /usr/share/doc/python3/html/library/stdtypes.html#tuple, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#list
• task_id (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type — None

🌈 classmethod cancel(task_id)

Cancel the task with the given ID. 🛑

Parameters

• task_id (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type — None

💜 classmethod is_ok(task_id)

Make sure the task running system is working for the given task. If the task system has detected an internal failure, it returns False. 🚨

Parameters

• task_id (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type — bool. Doc ref: /usr/share/doc/python3/html/library/functions.html#bool

---

✨ toil.server.wes.toil_backend ✨

🎀 Attributes

• logger 🪄

🦋 Classes

• ToilWorkflow 🌸
• ToilBackend 💖 — WES backend implemented for Toil to run CWL, WDL, or Toil workflows. This

🧚 Module Contents

toil.server.wes.toil_backend.logger — the logger attribute. ✨

---

class toil.server.wes.toil_backend.ToilWorkflow(base_work_dir, state_store_url, run_id)

Parameters

• base_work_dir (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str
• state_store_url (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str
• run_id (str) — doc ref: /usr/share/doc/python3/html/library/stdtypes.html#str

Attributes 🌷 — these are all just names, so take a peek:

• base_scratch_dir
• state_store_url
• run_id
• scratch_dir
• exec_dir
• store
• state_machine

💅 fetch_state()

There are two signatures, bestie:

fetch_state(key: str, default: str) -> str
fetch_state(key: str, default: None = None) -> str | None

Doc refs: /usr/share/doc/python3/html/library/stdtypes.html#str and /usr/share/doc/python3/html/library/constants.html#None

Return the contents of the given key in the workflow's state store. 🔑 If the key does not exist, the default value is returned. 🫶

fetch_scratch(filename) 📂

okay so this one is giving file retrieval 💅 It hands you a context manager for a stream of the given file from the workflow's scratch directory. If the file isn't there, you get None instead.

• filename (str) — the file you want

Return type: collections.abc.Generator[Optional[TextIO], None, None]

exists() ✨

Returns True if the workflow run exists. Simple, but iconic 🙌

Return type: bool

get_state() 🌸

Returns the state of the current run. Check the vibe of your run 👀

Return type: str

check_on_run(task_runner) 🚨

bestie, this is the vibe check 🫶 It makes sure nothing has gone wrong in the task runner for this workflow. If something has, it logs the problem and fails the workflow with an error. Not the best moment, but at least you know what happened 💀

• task_runner (type[toil.server.wes.tasks.TaskRunner])

Return type: None

set_up_run() 🧚

Sets up the directories this run needs. Think of it as getting the stage ready before the show 🎀

Return type: None

clean_up() 🧹

Cleans up the directory and files related to the run. Tidy girl era activated ✨

Return type: None

queue_run(task_runner, request, options) 🦋

This workflow is ready to run, so you hand it off to the task system. Let's gooo 🔥

• task_runner (type[toil.server.wes.tasks.TaskRunner])
• request (dict[str, Any])
• options (list[str])

Return type: None

get_output_files() 🍓

Returns a collection of the output files this workflow generated. The receipts 🧾

Return type: Any

get_stdout_path() 📜

Returns the path to the standard output log, relative to the run's scratch_dir, or None if it doesn't exist.

Return type: Optional[str]

get_stderr_path() 🙈

Returns the path to the standard output log, relative to the run's scratch_dir, or None if it doesn't exist.

⚠️ Heads up: the original docs literally say standard output log for this method too, even though it's the stderr one. I'm keeping the wording exactly as written, so just be aware of that mismatch 🫶

Return type: Optional[str]

get_messages_path() 💌

Returns the path to the bus message log, relative to the run's scratch_dir, or None if it doesn't exist. Those bus messages are giving secret diary 🤫

Return type: Optional[str]

get_task_logs(filter_function=None) 👑

Returns all the task log objects for the individual tasks in the workflow. Lowkey the main character list 💜

The task names come from the job_type values in issued, completed, and failed messages. If JobAnnotationMessage messages are available, their annotations get added to the names too.

• filter_function (Optional[Callable[[toil.server.wes.abstract_backend.TaskLog, toil.bus.JobStatus], Optional[toil.server.wes.abstract_backend.TaskLog]]]) — optional. If you set it, it gets called with each task log and its job annotations. It should return a modified copy of the task log you want reported, or None if that task log should be left out. Filter with intention 🎀

Return type: list[dict[str, Union[str, int, None]]]

---

🦋 class toil.server.wes.toil_backend.ToilBackend(work_dir, state_store, options, dest_bucket_base, bypass_celery=False, wes_dialect='standard') 🧚

Bases: toil.server.wes.abstract_backend.WESBackend

This is the WES backend built for Toil, and it can run CWL, WDL, or Toil workflows. It's the class that validates and executes the workflows you submit. The whole operation, honestly 🔥

• work_dir (str)
• state_store (Optional[str])
• options (list[str])
• dest_bucket_base (Optional[str])
• bypass_celery (bool)
• wes_dialect (str)

Class attributes 💖

• run_id_prefix = 'run-'
• task_runner
• wes_dialect = 'standard'
• dest_bucket_base
• work_dir = b'.'
• supported_versions

get_runs() 🌈

This is a generator that yields a list of run IDs and their states. Scroll through the whole feed 📱

Return type: collections.abc.Generator[tuple[str, str], None, None]

get_state(run_id) 🌸

Returns the state of the workflow run with the given run ID. Heads up that it may raise an error if the workflow doesn't exist 💀

• run_id (str)

Return type: str

get_service_info() ✨

Gets information about the Workflow Execution Service. The service's bio 💅

Return type: dict[str, Any]

list_runs(page_size=None, page_token=None) 🍓

Lists the workflow runs, with pagination options 📃

• page_size (Optional[int])
• page_token (Optional[str])

Return type: dict[str, Any]

run_workflow(**args) 🔥

Runs a workflow. Hit play and let it cook 🍳

• args (Any)

Return type: dict[str, str]

get_run_log(run_id) 📜

Gets detailed info about a workflow run. The full tea ☕

• run_id (str)

Return type: dict[str, Any]

cancel_run(run_id) 🛑

Cancels a running workflow. Sometimes you gotta close the tab 🫠

• run_id (str)

Return type: dict[str, str]

get_run_status(run_id) 🦋

Gets quick status info about a workflow run. You get a simple result with the overall state of the run, no essay required ⭐

• run_id (str)

Return type: dict[str, str]

get_stdout(run_id) 🧁

Gets the stdout of a workflow run as a static file. Literally the printout of everything it said 📄

Parameters: run_id (str) 🍓 Return type: Any

📄 get_stderr(run_id)

okay so this one grabs the stderr of a workflow run and hands it back as a static file 💅

Parameters: run_id (str) Return type: Any

🫶 get_health()

bestie this is the vibe check 💖 it returns successfully if the server is healthy

Return type: werkzeug.wrappers.response.Response

🏠 get_homepage()

gives / a real answer instead of a 404 🌸 main character energy

Return type: werkzeug.wrappers.response.Response

---

🌸 toil.server.wsgi_app

Classes

• GunicornApplication: an entry point to integrate a Gunicorn WSGI server in Python. To start a (the summary gets cut off right there, bestie 💀)

Functions

• run_app(app[, options]): runs a Gunicorn WSGI server ✨

Module Contents

🎀 class toil.server.wsgi_app.GunicornApplication(app, options=None)

Bases: gunicorn.app.base.BaseApplication

An entry point to integrate a Gunicorn WSGI server in Python, and it's lowkey iconic. To start a WSGI application with callable app, run the following code:

WSGIApplication(app, options={
    ...
}).run()

For more details, see: https://docs.gunicorn.org/en/latest/custom.html

Parameters

• app (object)
• options (Optional[dict[str, Any]])

Members

• options
• application

init(*args)

Parameters: args (Any) Return type: None

load_config()

Return type: None

load()

Return type: object

🦋 toil.server.wsgi_app.run_app(app, options=None)

Runs a Gunicorn WSGI server, no cap 🔥

Parameters

• app (object)
• options (Optional[dict[str, Any]])

Return type: None

---

🦋 toil.serviceManager

Attributes

• logger 🌈 (no description given)

Classes

• ServiceManager: manages the scheduling of services, literally the queen of the group chat 👑

Module Contents

• toil.serviceManager.logger

💜 class toil.serviceManager.ServiceManager(job_store, toil_state)

Manages the scheduling of services. She's the organizer and she is so good at it 💅

Parameters

• job_store (toil.jobStores.abstractJobStore.AbstractJobStore)
• toil_state (toil.toilState.ToilState)

services_are_starting(job_id)

Checks whether the services for a job are currently getting started 🌸

Returns: True if the services for the given job are currently being started, and False otherwise.

Parameters: job_id (str) Return type: bool

get_job_count()

Gets the total number of jobs we're working on ✨ this counts services and their parent non-service jobs.

Return type: int

start()

Starts the service scheduling thread. Let's get this party started 🎉

Return type: None

put_client(client_id)

Schedules the services of a job asynchronously 🧚 once the job's services are running, its ID comes back out of toil.leader.ServiceManager.get_ready_client.

Parameters: client_id (str), which is the ID of the job with services to schedule Return type: None

get_ready_client(maxWait)

Fetches a ready client and waits as long as it needs to 🫶

Parameters: maxWait (float), the time in seconds to wait for a JobDescription before returning

Returns: the ID of a client whose services are running, or None if no such job is available.

Return type: Optional[str]

get_unservable_client(maxWait)

Fetches a client whose services failed to start 💀 rough day for that one.

Parameters: maxWait (float), the time in seconds to wait for a JobDescription before returning

Returns: the ID of a client whose services failed to start, or None if no such job is available.

Return type: Optional[str]

get_startable_service(maxWait)

Fetches a service job that's ready to start 🚀

Parameters: maxWait (float), the time in seconds to wait for a job before returning

Returns: the ID of a service job that the leader can start, or None if no such job exists.

Return type: Optional[str]

kill_services(service_ids, error=False)

Stops all the given service jobs 🥀

Parameters

• services: service jobStoreIDs to kill
• error (bool): whether to signal that the service failed with an error when stopping it
• service_ids (collections.abc.Iterable[str])

Return type: None

is_active(service_id)

Returns true if the service job has not been told to terminate 🌈

Parameters: service_id (str), the service to check on Return type: bool

is_running(service_id)

Returns true if the service job has started and is active 🎀

Parameters

• service: the service to check on
• service_id (str)

Return type: bool

check()

Checks on the service manager thread 🔍

Raises RuntimeError: if the underlying thread has quit. Yikes, that's not it, chief 😭

Return type: None

shutdown()

Terminates worker threads cleanly 🫧 this starts and kills all service threads, and it will block until all services are started and blocked.

Return type: None

💖 toil.statsAndLogging 💖

okay bestie, this module is where all the stats and logging lives, so you can actually see what your workflow is doing 👀✨ it's giving organized and it's giving pretty 💅

---

🦋 Attributes 🦋

okay so these are the module-level bits you can grab:

• logger
• root_logger
• toil_logger
• DEFAULT_LOGLEVEL
• TRACE

---

🎀 Classes 🎀

• StatsAndLogging — this is a thread that gathers up stats and log messages and collates them, so nothing gets lost in the shuffle 🧵✨

---

🌸 Functions 🌸

• set_log_level(level[, set_logger]) — sets the root logger's level to a string level, like "INFO" 📢
• install_log_color([set_logger]) — makes your logs colored 🌈
• add_logging_options(parser[, default_level]) — adds logging options so you can set the global log level 🎛️
• configure_root_logger() — sets up the root logger with handlers and formatting 🧁
• log_to_file(log_file, log_rotation)
• set_logging_from_options(options)
• suppress_exotic_logging(local_logger) — tries to quiet the loggers of every non-Toil package by setting them to CRITICAL 🔇💀

---

💅 Module Contents 💅

here's the full lineup of module-level names and values:

toil.statsAndLogging.logger

toil.statsAndLogging.root_logger

toil.statsAndLogging.toil_logger

toil.statsAndLogging.DEFAULT_LOGLEVEL = 20

toil.statsAndLogging.TRACE = 5

🎀 class toil.statsAndLogging.StatsAndLogging(jobStore, config)

okay this is the big one. it's a thread that aggregates stats and logging so the whole run stays tidy 🧵💖

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)
• config (toil.common.Config)

✨ start()

kicks off the stats and logging thread 🚀

Return type: None

🌈 classmethod formatLogStream(stream, stream_name)

okay so, you hand this a stream of text or bytes, plus the job name, the job itself, or any other optional identity info that can be turned into a string. it gives you back one big formatted job log that's ready to show the user 📜

the vibe here is that we don't stick a prefix on every single line of the job's log. if we did, the prefixes would get wider than any reasonable terminal and longer than the messages themselves, and nobody wants that 🙈

Parameters

• stream (Union[IO[str], IO[bytes]]) — the stream of text or bytes to print for the user
• stream_name (str)

Return type: str

🦋 classmethod logWithFormatting(stream_name, jobLogs, method=logger.debug, message=None)

Parameters

• stream_name (str)
• jobLogs (Union[IO[str], IO[bytes]])
• method (Callable[[str], None])
• message (Optional[str])

Return type: None

🫶 classmethod writeLogFiles(jobNames, jobLogList, config, failed=False)

Parameters

• jobNames (list[str])
• jobLogList (list[str])
• config (toil.common.Config)
• failed (bool)

Return type: None

🔥 classmethod statsAndLoggingAggregator(jobStore, stop, config)

this one is the collator. it pulls stats and reporting log messages from the workers. it runs inside of a thread and keeps collating for as long as the stop flag is not True 💅

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)
• stop (threading.Event)
• config (toil.common.Config)

Return type: None

👀 check()

checks in on the stats and logging aggregator. if the underlying thread has quit, it raises a RuntimeError 😭

Return type: None

🧹 shutdown()

wraps up the stats and logging aggregation thread, so everything gets finished neatly 🧹✨

Return type: None

🌸 toil.statsAndLogging.set_log_level(level, set_logger=None)

sets the root logger level to a given string level, like "INFO" 📢

Parameters

• level (str)
• set_logger (Optional[logging.Logger])

Return type: None

🌈 toil.statsAndLogging.install_log_color(set_logger=None)

makes logs colored, obsessed 🌈💖

Parameters

• set_logger (Optional[logging.Logger])

Return type: None

🎛️ toil.statsAndLogging.add_logging_options(parser, default_level=None)

adds logging options so you can set the global log level 🎛️

Parameters

• default_level (Optional[int]) — a logging level, like logging.INFO, to use as the default
• parser (argparse.ArgumentParser)

Return type: None

🧁 toil.statsAndLogging.configure_root_logger()

sets up the root logger with handlers and formatting 🧁

this needs to be called before any entry point tries to log anything. that way the formatting stays consistent across the whole thing, no exceptions 🫶

Return type: None

📁 toil.statsAndLogging.log_to_file(log_file, log_rotation)

Parameters

• log_file (Optional[str])
• log_rotation (bool)

Return type: None

⚙️ toil.statsAndLogging.set_logging_from_options(options)

toil.statsAndLogging.suppress_exotic_logging(local_logger)

Okay so this one is lowkey a hero 🦸 It tries to mute every non-Toil package's logger by setting it to CRITICAL, so the chatty libraries stop clogging up your terminal. Periodt 🔇✨

The noisy ones it goes after include 'requests_oauthlib', 'google', 'boto', 'websocket', 'oauthlib', etc. 🙈

Heads up though: it only mutes loggers that already exist and that you can actually see in the environment 👀 The exception is anything in the list declared in "always_suppress", which gets handled no matter what 💅

Why does this matter? Some packages, especially boto3, aren't always loaded yet when this runs. So we create the logger and set the level preemptively 🧚 That way it's already handled before the package shows up and starts yapping 💀

Parameters 🎀

• local_logger (str) 🍓

Return type — None 💜

---

toil.test 🧪

Base testing class for Toil. This is the foundation everything test-related is built on, bestie 🫶

Submodules 🌈

• toil.test.batchSystems

Submodules 🌈

• toil.test.batchSystems.batchSystemTest

Attributes 🔥

These are the attributes that live in there:

• logger
• numCores
• preemptible
• defaultRequirements

Classes 👑

Here's the whole lineup of test classes, iykyk 💅

• BatchSystemPluginTest — the class that tests batch system plugin functionality ✨
• hidden — hides an abstract base class from unittest's test case loader 🙈
• AbstractBatchSystemJobTest — an abstract base class for batch system tests that use a full Toil workflow rather 🦋
• KubernetesBatchSystemTest — tests against the Kubernetes batch system ☁️
• KubernetesBatchSystemBenchTest — Kubernetes batch system unit tests that don't need to actually talk to a cluster 🧁
• AWSBatchBatchSystemTest — tests against the AWS Batch batch system 🌸
• MesosBatchSystemTest — tests against the Mesos batch system 🍓
• SingleMachineBatchSystemTest — tests against the single-machine batch system 💜
• MaxCoresSingleMachineBatchSystemTest — this test makes sure the single machine batch system doesn't go past the configured number 🫡
• Service — an abstract class used to define the interface to a service 🧚
• GridEngineBatchSystemTest — tests against the GridEngine batch system 🔥
• SlurmBatchSystemTest — tests against the Slurm batch system ⭐
• LSFBatchSystemTest — tests against the LSF batch system 🦋
• TorqueBatchSystemTest — tests against the Torque batch system 🌈
• HTCondorBatchSystemTest — tests against the HTCondor batch system 👑
• TestSingleMachineBatchSystemJob — tests a Toil workflow against the SingleMachine batch system 💖
• TestMesosBatchSystemJob — tests a Toil workflow against the Mesos batch system 🫶

✨ Functions ✨

Okay bestie, here's the helper squad 💅 these are the functions living in this test module:

• write_temp_file(s, temp_dir) — dumps a string into a temp file and hands you back its path 📄
• parentJob(job, cmd)
• childJob(job, cmd)
• grandChildJob(job, cmd)
• greatGrandChild(cmd)
• measureConcurrency(filepath[, sleep_time]) — runs stuff in parallel to figure out how many concurrent tasks can go at once 🏃‍♀️
• count(delta, file_path) — increments the counter file and returns the max number of times the file (the page literally trails off right there, no cap 💀)
• getCounters(path)
• resetCounters(path)
• get_omp_threads()

✨ Module Contents ✨

Here's what's sitting in the module namespace 🎀

• toil.test.batchSystems.batchSystemTest.logger
• toil.test.batchSystems.batchSystemTest.numCores = 2
• toil.test.batchSystems.batchSystemTest.preemptible = False
• toil.test.batchSystems.batchSystemTest.defaultRequirements

---

🦋 class toil.test.batchSystems.batchSystemTest.BatchSystemPluginTest(methodName='runTest')

Bases: toil.test.ToilTest — the plugin testing queen 👑 Its whole job is testing batch system plugin functionality.

• tearDown() — the cleanup hook that takes the test fixture apart once testing is done 🧹
  • Return type: None
• test_add_batch_system_factory()
  • Return type: None

---

🌸 class toil.test.batchSystems.batchSystemTest.hidden

This one's giving hide-and-seek 🙈 It hides the abstract base class from unittest's test case loader, so it doesn't get picked up as a real test. The reference for the trick is http://stackoverflow.com/questions/1323455/python-unit-test-with-base-and-sub-class#answer-25695512.

• class AbstractBatchSystemTest(methodName='runTest')
  • Bases: toil.test.ToilTest
  • A base test case full of generic tests that every batch system should pass 💖 It can't assume the batch system actually runs commands on the local machine or filesystem, so keep that in mind bestie.
  • abstractmethod createBatchSystem()
    • Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem
  • supportsWallTime()
    • Return type: bool
  • classmethod createConfig() — returns a dummy config for the batch system tests. We need a workflowID set up since we're running tests without setting up a jobstore. This is the class version, for when an instance isn't available 🔧
    • Return type: toil.common.Config
  • classmethod setUpClass() — the hook that sets up the class fixture before any tests in the class run 🎬
    • Return type: None
  • setUp() — the hook that sets up the test fixture before we put it to work ✨
    • Return type: None
  • tearDown() — the hook that takes the test fixture apart after testing it 🧹
    • Return type: None
  • get_max_startup_seconds() — gets the number of seconds this test should wait for the first job to run. Some batch systems need time to scale up, so this one's a lowkey lifesaver ⏳
    • Return type: int
  • test_available_cores()
    • Return type: None
  • test_run_jobs()
    • Return type: None
  • test_set_env()
    • Return type: None
  • test_set_job_env() — tests the mechanism for setting per-job environment variables on batch system jobs 🌈
    • Return type: None
  • testCheckResourceRequest()
    • Return type: None
  • testScalableBatchSystem()
    • Return type: None
• class AbstractGridEngineBatchSystemTest(methodName='runTest')
  • Bases: AbstractBatchSystemTest
  • An abstract class that cuts down on repeated code between Grid Engine, Slurm, and other similar batch systems 🫶

---

💜 class toil.test.batchSystems.batchSystemTest.AbstractBatchSystemJobTest

An abstract base class for batch system tests that run a full Toil workflow, instead of talking to the batch system directly. It's the bigger, glam version 💅

• cpuCount
• allocatedCores
• sleepTime = 30
• abstractmethod getBatchSystemName()
  • Return type: (str, AbstractBatchSystem)
• getOptions(tempDir) — configures options for the Toil workflow and makes the job store 🗂️
  • Parameters: tempDir (pathlib.Path), the path to the test directory
  • Return type: argparse.Namespace
  • Returns: the Toil options object
• testJobConcurrency(tmp_path) — checks that the batch system is allocating core resources properly for concurrent tasks 🧮
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• test_omp_threads(tmp_path) — checks that the OMP_NUM_THREADS env var is set correctly based on jobs.cores 🧵
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None

---

🍓 class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemTest(methodName='runTest')

Bases: hidden. The Kubernetes batch system gets its own tests here ☸️

• supportsWallTime()
  • Return type: bool
• createBatchSystem()
  • Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

---

🧚 class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemBenchTest(methodName='runTest')

Bases: toil.test.ToilTest. These are Kubernetes batch system unit tests that don't need to actually talk to a cluster, so they're low-key chill 🧘‍♀️

• test_preemptability_constraints() — makes sure we generate the right preemptability constraints 🛡️
  • Return type: None
• test_label_constraints() — makes sure we generate the right preemptability constraints (yes, the docstring says preemptability here too, iykyk 🙃)
  • Return type: None

---

🎀 class toil.test.batchSystems.batchSystemTest.AWSBatchBatchSystemTest(methodName='runTest')

Bases: hidden. Tests against the AWS Batch batch system, slay ☁️

• supportsWallTime()

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool) 💅

createBatchSystem() 🧪

Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

get_max_startup_seconds() ⏳

Gets the number of seconds this test should wait for the first job to run, because some batch systems need time to scale up 🌄

Return type: int (/usr/share/doc/python3/html/library/functions.html#int)

---

class toil.test.batchSystems.batchSystemTest.MesosBatchSystemTest(methodName='runTest') 🌸

Bases: hidden, toil.batchSystems.mesos.test.MesosTestSupport

Tests against the Mesos batch system ✨

Attribute: batchSystem: toil.batchSystems.mesos.batchSystem.MesosBatchSystem

classmethod createConfig() 🎀

Needs to set mesos_endpoint to localhost for testing, since the default is now the private IP address 🙈

Return type: toil.common.Config

supportsWallTime() ⏰

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

createBatchSystem() 🧪

Return type: toil.batchSystems.mesos.batchSystem.MesosBatchSystem

tearDown() 🧹

Hook method for deconstructing the test fixture after testing it, aka cleaning up after the vibes are over 💜

Return type: None

testIgnoreNode() 🙅‍♀️

Return type: None

---

toil.test.batchSystems.batchSystemTest.write_temp_file(s, temp_dir) 📝

Dumps a string into a temp file and returns its path, no cap 🦋

Parameters:

• s (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• temp_dir (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemTest(methodName='runTest') 🍓

Bases: hidden

Tests against the single-machine batch system 💅

supportsWallTime() ⏰

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

createBatchSystem() 🧪

Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

testProcessEscape(hide=False) 🏃‍♀️

Tests to make sure child processes and their descendants go away when the Toil workflow stops.

If hide is true, it will try to hide the child processes to make them hard to stop 🫣

Parameters:

• hide (bool (/usr/share/doc/python3/html/library/functions.html#bool))

Return type: None

testHidingProcessEscape() 🙈

Tests to make sure child processes and their descendants go away when the Toil workflow stops, even if the job process stops and leaves children behind. That's a whole plot twist 😭

Return type: None

---

class toil.test.batchSystems.batchSystemTest.MaxCoresSingleMachineBatchSystemTest(methodName='runTest') 👑

Bases: toil.test.ToilTest

This test makes sure the single machine batch system doesn't go over the configured number of cores. Main character energy, no exceptions 🔥

classmethod setUpClass() 🌈

Hook method for setting up the class fixture before running tests in the class ✨

Return type: None

setUp() 🌷

Hook method for setting up the test fixture before exercising it.

Return type: None

tearDown() 🧹

Hook method for deconstructing the test fixture after testing it.

Return type: None

scriptCommand() 📜

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

test() 🧪

Return type: None

testServices() 🛎️

Return type: None

---

toil.test.batchSystems.batchSystemTest.parentJob(job, cmd) 👩‍👧

Parameters:

• job (toil.job.Job)
• cmd (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: None

toil.test.batchSystems.batchSystemTest.childJob(job, cmd) 🧒

Parameters:

• job (toil.job.Job)
• cmd (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: None

toil.test.batchSystems.batchSystemTest.grandChildJob(job, cmd) 👶

Parameters:

• job (toil.job.Job)
• cmd (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: None

toil.test.batchSystems.batchSystemTest.greatGrandChild(cmd) 🍼

Parameters:

• cmd (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: None

---

class toil.test.batchSystems.batchSystemTest.Service(cmd) 🧚

Bases: toil.job.Job.Service

Abstract class used to define the interface to a service. You subclass it to make your own services 🫶

It is not executed as a job. It runs inside a ServiceHostJob 🏠

Parameters:

• cmd (str (/usr/share/doc/python3/html/library/stdtypes.html#str))

Attribute: cmd

start(job) 🚀

Starts the service.

Parameters:

• job (toil.job.ServiceHostJob): the underlying host job the service runs in. You can use it to register deferred functions or to reach the fileStore for making temporary files 🗂️

Returns: an object describing how to access the service. It has to be pickleable, and jobs use it to access the service (see toil.job.Job.addService()) 💌

Return type: None

check() 🩺

Checks that the service is still running.

Raises exceptions.RuntimeError: if the service failed, the service job gets labeled failed. 💀

Returns: True if the service is still running, otherwise False. If it returns False, the service job gets terminated and counted as a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False! 🚨

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

stop(job) 🛑

Stops the service. This function can block until it's complete ⏳

Parameters:

• job (toil.job.ServiceHostJob): the underlying host job the service runs in. You can use it to register deferred functions or to reach the fileStore for making temporary files 🗂️

Return type: None

---

class toil.test.batchSystems.batchSystemTest.GridEngineBatchSystemTest(methodName='runTest') 🌿

Bases: hidden

Tests against the GridEngine batch system 🌽

createBatchSystem() 🧪

Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown() 🧹

Hook method for deconstructing the test fixture after testing it.

Return type: None

---

class toil.test.batchSystems.batchSystemTest.SlurmBatchSystemTest(methodName='runTest') 🐌

Bases: hidden

Tests against the Slurm batch system 🔥

createBatchSystem() 🧪

Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown() 🧹

Hook method for deconstructing the test fixture after testing it.

Return type: None

---

class toil.test.batchSystems.batchSystemTest.LSFBatchSystemTest(methodName='runTest') 🦩

Bases: hidden

Tests against the LSF batch system 🫧

createBatchSystem() 🧪

Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

---

class toil.test.batchSystems.batchSystemTest.TorqueBatchSystemTest(methodName='runTest') 🎶

Bases: hidden

Tests against the Torque batch system 💜

createBatchSystem() 🧪

Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

Okay so the tearDown() hook is where you clean up the test fixture after a test is done running 🧹✨ It's a hook method for deconstructing the test fixture after testing it, and it returns None. 💅

🐣 HTCondorBatchSystemTest

class toil.test.batchSystems.batchSystemTest.HTCondorBatchSystemTest(methodName='runTest')

Bases: hidden 🙈

This one's the HTCondor test squad, and it runs tests against the HTCondor batch system 🦋

• createBatchSystem() — builds the batch system for these tests. Return type: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem 🔥
• tearDown() — the same cleanup hook as before, now for the HTCondor tests. It's a hook method for deconstructing the test fixture after testing it. Return type: None 🧹

🌈 TestSingleMachineBatchSystemJob

class toil.test.batchSystems.batchSystemTest.TestSingleMachineBatchSystemJob

Bases: AbstractBatchSystemJobTest ✨

This class is the main character for testing Toil workflows against the SingleMachine batch system 👑

• getBatchSystemName() — returns the name of the batch system under test. Return type: (str, AbstractBatchSystem), where the AbstractBatchSystem is toil.batchSystems.abstractBatchSystem.AbstractBatchSystem 💖

testConcurrencyWithDisk(tmp_path)

• testConcurrencyWithDisk(tmp_path) — checks that the batch system is allocating disk resources properly. It takes:
  • tmp_path (pathlib.Path) 🍓
  • Return type: None

testNestedResourcesDoNotBlock(tmp_path)

• testNestedResourcesDoNotBlock(tmp_path) — resources get requested in the order Memory > Cpu > Disk. This test makes sure that when there aren't enough CPUs for one scheduled job, it doesn't block another job that can run. No waiting around for the bestie 🫶 It takes:
  • tmp_path (pathlib.Path) 🍓
  • Return type: None

🦋 TestMesosBatchSystemJob

class toil.test.batchSystems.batchSystemTest.TestMesosBatchSystemJob

Bases: AbstractBatchSystemJobTest, toil.batchSystems.mesos.test.MesosTestSupport 🔥

This class tests Toil workflows against the Mesos batch system, and it's giving full test squad energy 💜

• mesos_support() — the Mesos setup helper. Return type: collections.abc.Generator[None] 🌸
• getOptions(tempDir) — configures the options for the Toil workflow and makes the job store. It's documented as taking a str path to the test directory and returning the Toil options object. It takes:
  • tempDir (pathlib.Path) 🍓
  • Return type: argparse.Namespace 💅
• getBatchSystemName() — returns the name of the batch system under test. Return type: (str, AbstractBatchSystem), where the AbstractBatchSystem is toil.batchSystems.abstractBatchSystem.AbstractBatchSystem ✨

---

🧁 measureConcurrency

toil.test.batchSystems.batchSystemTest.measureConcurrency(filepath, sleep_time=10)

Okay this one runs in parallel to figure out how many tasks are running at the same time 👯‍♀️ The code was copied from toil.batchSystemTestMaxCoresSingleMachineBatchSystemTest. It takes:

• filepath (toil.lib.misc.StrPath) — the path to the counter file 📂
• sleep_time (int) — the number of seconds to sleep before counting down ⏰

It returns the max concurrency value as an int 💖

🍭 count

toil.test.batchSystems.batchSystemTest.count(delta, file_path)

This one increments the counter file and returns the max number of times the file has been modified 📈 The counter data has to look like this: concurrent tasks, max concurrent tasks. The counter should be initialized to 0,0 🫶 It takes:

• delta (int) — the increment value ➕
• file_path (toil.lib.misc.StrPath) — the path to the shared counter file 📂

It returns the max concurrent tasks as an int 👑

🔢 getCounters

toil.test.batchSystems.batchSystemTest.getCounters(path)

It takes path (toil.lib.misc.StrPath) and returns a tuple[int, int] 💅

🧹 resetCounters

toil.test.batchSystems.batchSystemTest.resetCounters(path)

It takes path (toil.lib.misc.StrPath) and returns None 🧹

🧵 get_omp_threads

toil.test.batchSystems.batchSystemTest.get_omp_threads()

It returns a str 🌈

---

🔌 toil.test.batchSystems.batch_system_plugin_test

✨ Attributes

• logger — the module's logger 📝

👯‍♀️ Classes

• FakeBatchSystem — adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler 🫧
• BatchSystemPluginTest — the legacy common base class for Toil tests 🕰️

📦 Module Contents

toil.test.batchSystems.batch_system_plugin_test.logger 📝

🎀 FakeBatchSystem

class toil.test.batchSystems.batch_system_plugin_test.FakeBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport 🦋

This is the fake batch system with cleanup support. It adds cleanup when the last running job leaves a node, for batch systems that can't do it using the backing scheduler 🧚 It takes:

• config (toil.common.Config) ⚙️
• maxCores (float) 🍓
• maxMemory (float) 🍓
• maxDisk (int) 🍓

classmethod supportsAutoDeployment()

• supportsAutoDeployment() — tells you whether this batch system supports auto-deployment of the user script itself 🚀

If it does, setUserScript() can be invoked to set the resource object that represents the user script. Return type: bool ✨

Note to implementors: if your implementation returns True here, it should also override 🙈 (That's where the page trails off, so that's all the info we get on that one, bestie.)

issueBatchJob(command, job_desc, job_environment=None)

• issueBatchJob(command, job_desc, job_environment=None) — issues a job with the specified command to the batch system and returns a unique job ID number 🎫 It takes:
  • command (str) — the command to execute somewhere to run the Toil worker process 🏃‍♀️
  • job_desc (toil.job.JobDescription) — the JobDescription for the job being run 📋
  • job_environment (Optional[dict[str, str]]) — a collection of job-specific environment variables to be set on the worker 🌿
• It returns a unique job ID number (int) that can be used to reference the newly issued job 🔖

killBatchJobs(jobIDs)

• killBatchJobs(jobIDs) — kills the given job IDs ☠️ After it returns, the killed jobs won't show up in the results of getRunningBatchJobIDs, and they won't be returned from getUpdatedBatchJob either. It takes:
  • jobIDs (list[int]) — the list of IDs of jobs to kill 🗑️
  • Return type: None

getIssuedBatchJobIDs()

• getIssuedBatchJobIDs() — gets all currently issued jobs 📬 It returns a list of jobs (as job ID numbers) that are currently issued, which means they might be running or waiting to be run. Even though the result is a list, don't depend on the ordering 🙅‍♀️ Return type: list[int]

getUpdatedBatchJob(maxWait)

• getRunningBatchJobIDs() — Hmm okay so this one hands you a dict of job ID numbers that are actively running right now (not just sitting in the queue waiting 💅), with how many seconds each one has been going as the value.

Returns

• A dictionary with running job ID number keys, and how many seconds each has been running as the values 🦋

Return type

• dict[int, float]

Okay so getUpdatedBatchJob(maxWait) is giving "tell me what just finished" energy 🫶 It returns info about a job that has updated its status (aka it stopped running, either successfully or with an error). Each job gets returned exactly once, so no double-counting bestie 💖

Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait. ngl that's a plot twist 🙈

Parameters

• maxWait (int) — the number of seconds to block, waiting for a result ⏳

Returns

• If a result is available, it returns UpdatedBatchJobInfo. Otherwise it returns None. The wallTime is the number of seconds (a strictly positive float) of wall-clock time the job ran for, or None if this batch system doesn't support tracking wall time. Main character energy for the timing info 🧚

Return type

• Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo]

---

✨ shutdown() ✨

Called at the completion of a toil invocation. It should cleanly terminate all worker threads, so no loose ends 🌸

Return type

• None

✨ classmethod add_options(parser) ✨

If this batch system provides any command line options, it adds them to the given parser. Iconic behaviour 👑

Parameters

• parser (configargparse.ArgumentParser)

Return type

• None

✨ classmethod setOptions(setOption) ✨

This one processes command line or configuration options that are relevant to this batch system. It's giving config-wrangling queen 💅

Parameters

• setOption (toil.batchSystems.options.OptionSetter) — A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) that returns nothing. It's used to update the run configuration as a side effect 🎀

Return type

• None

---

✨ toil.test.batchSystems.batch_system_plugin_test.BatchSystemPluginTest(methodName='runTest') ✨

Bases: toil.test.ToilTest

Legacy common base class for Toil tests. New tests should be made in the pytest style and not use this class, bestie 🙈

When running tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files placed. The directory gets created if it doesn't exist. The path may be relative, in which case it's assumed to be relative to the project root. If TOIL_TEST_TEMP isn't defined, temporary files and directories get created in the system's default location for such files, and any temporary files or directories left over from tests will be removed automatically during tear down. Otherwise, left-over files will not be removed. 🧹

✨ tearDown() ✨

Hook method for deconstructing the test fixture after testing it.

Return type

• None

✨ test_batchsystem_plugin_installable() ✨

Tests that installing a batch system plugin works. 🦋 :return:

---

✨ toil.test.batchSystems.test_gridengine ✨

Classes

• FakeBatchSystem — A class that implements a minimal Batch System, needed to create a Worker (see below). 🧚
• GridEngineTest — Class for unit-testing GridEngineBatchSystem 💖

Functions

• call_qstat_or_qacct(args, **_)

---

✨ toil.test.batchSystems.test_gridengine.FakeBatchSystem ✨

Class that implements a minimal Batch System, needed to create a Worker (see below).

config

getWaitDuration()

✨ with_retries(operation, *args, **kwargs) ✨

The grid engine batch system needs a with_retries function when running the GridEngineThread, so we fake one 🫶

---

✨ toil.test.batchSystems.test_gridengine.call_qstat_or_qacct(args, **_) ✨

---

✨ toil.test.batchSystems.test_gridengine.GridEngineTest(methodName='runTest') ✨

Bases: toil.test.ToilTest

Class for unit-testing GridEngineBatchSystem 🌈

✨ setUp() ✨

Hook method for setting up the test fixture before exercising it. 💜

test_coalesce_job_exit_codes_one_exists()

test_coalesce_job_exit_codes_one_still_running()

test_coalesce_job_exit_codes_many_all_exist()

---

✨ toil.test.batchSystems.test_lsf_helper ✨

lsfHelper.py shouldn't need a batch system, so the unit tests here should aim to run on any system. 🍓

Classes

• LSFHelperTest — Legacy common base class for Toil tests. 🎀

---

✨ toil.test.batchSystems.test_lsf_helper.LSFHelperTest(methodName='runTest') ✨

Bases: toil.test.ToilTest

Legacy common base class for Toil tests. New tests should be made in the pytest style and not use this class 💅

When running tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files placed. The directory gets created if it doesn't exist. The path may be relative, in which case it's assumed to be relative to the project root. If TOIL_TEST_TEMP isn't defined, temporary files and directories get created in the system's default location for such files, and any temporary files or directories left over from tests will be removed automatically during tear down. Otherwise, left-over files will not be removed. 🫶

✨ test_parse_mem_and_cmd_from_output() ✨

---

✨ toil.test.batchSystems.test_slurm ✨

Attributes

• logger
• JOB_BASE_TIME

Classes

• FakeBatchSystem — A class that implements a minimal Batch System, needed to create a Worker (see below). ✨
• SlurmTest — Class for unit-testing SlurmBatchSystem 👑

✨ Functions ✨

Okay bestie, this little table is the lineup of fake helpers that pretend to be Slurm 🧚

• call_either(args, **_): pretends to call either sacct or scontrol, whichever one fits 💅
• call_sacct(args, **_): shows the arguments call_command gets when it runs sacct. The full example is down in Module Contents 🦋
• call_scontrol(args, **_): same energy, but for when call_command runs scontrol 💜
• call_sacct_raises(*_): fakes a failing sacct by raising a CalledProcessErrorStderr 💀
• call_sinfo(*_): simulates asking Slurm for partition info 🌸

---

✨ Module Contents ✨

Okay so here are the goods, starting with the module-level bits 🎀

• toil.test.batchSystems.test_slurm.logger
• toil.test.batchSystems.test_slurm.JOB_BASE_TIME

toil.test.batchSystems.test_slurm.call_either(args, **_)

Pretends to call either sacct or scontrol, whichever one is the right pick ✨

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.test.batchSystems.test_slurm.call_sacct(args, **_)

Okay, when call_command runs sacct, the arguments look something like this: ['sacct', '-n', '-j', '<comma-separated list of job-ids>', '--format', 'JobIDRaw,State,ExitCode', '-P', '-S', '1970-01-01']. The multi-line output is something like this 👀

1234|COMPLETED|0:0
1234.batch|COMPLETED|0:0
1235|PENDING|0:0
1236|FAILED|0:2
1236.extern|COMPLETED|0:0

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.test.batchSystems.test_slurm.call_scontrol(args, **_)

When call_command runs scontrol, the arguments are either ['scontrol', 'show', 'job'] or ['scontrol', 'show', 'job', '<job-id>'] 🍓

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.test.batchSystems.test_slurm.call_sacct_raises(*_)

Fakes the sacct command failing by raising a CalledProcessErrorStderr 💀

toil.test.batchSystems.test_slurm.call_sinfo(*_)

Simulates asking Slurm for partition info 🫶

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

class toil.test.batchSystems.test_slurm.FakeBatchSystem

Bases: toil.batchSystems.abstractBatchSystem.BatchSystemSupport (#toil.batchSystems.abstractBatchSystem.BatchSystemSupport)

A bare-bones batch system class, just enough to build a Worker (see below) 🧚

• start_time
• getWaitDuration()

class toil.test.batchSystems.test_slurm.SlurmTest(methodName='runTest')

Bases: toil.test.ToilTest (#toil.test.ToilTest)

The unit-test class for SlurmBatchSystem 🧪

• setUp(): the hook method that sets up the test fixture before it gets exercised
• test_getJobDetailsFromSacct_one_exists()
• test_getJobDetailsFromSacct_one_not_exists()
• test_getJobDetailsFromSacct_many_all_exist()
• test_getJobDetailsFromSacct_many_some_exist()
• test_getJobDetailsFromSacct_many_none_exist()
• test_getJobDetailsFromSacct_argument_list_too_big()
• test_getJobDetailsFromScontrol_one_exists()
• test_getJobDetailsFromScontrol_one_not_exists(): asking for the job details of a single job that scontrol doesn't know about should raise an exception 🙈
• test_getJobDetailsFromScontrol_many_all_exist()
• test_getJobDetailsFromScontrol_many_some_exist()
• test_getJobDetailsFromScontrol_many_none_exist()
• test_getJobExitCode_job_exists()
• test_getJobExitCode_job_not_exists()
• test_getJobExitCode_sacct_raises_job_exists(): this one forces scontrol to be used for job info, by letting sacct raise an exception
• test_getJobExitCode_sacct_raises_job_not_exists(): this one forces scontrol to be used for job info, by letting sacct raise an exception. Then scontrol should also raise, because it doesn't know the job
• test_coalesce_job_exit_codes_one_exists()
• test_coalesce_job_exit_codes_one_not_exists()
• test_coalesce_job_exit_codes_many_all_exist()
• test_coalesce_job_exit_codes_mix_sacct_scontrol()
• test_coalesce_job_exit_codes_some_exists()
• test_coalesce_job_exit_codes_sacct_raises_job_exists(): forces scontrol to be used for job info, by letting sacct raise an exception
• test_coalesce_job_exit_codes_sacct_raises_job_not_exists(): forces scontrol to be used for job info, by letting sacct raise an exception. Then scontrol should also raise, because it doesn't know the job
• test_PartitionSet_get_partition()
• test_PartitionSet_default_gpu_partition()
• test_prepareSbatch_partition()
• test_prepareSbatch_time()
• test_prepareSbatch_export()
• test_option_detector()
• test_any_option_detector()

---

✨ toil.test.cactus ✨

Submodules

toil.test.cactus.test_cactus_integration

Classes

Here's the one class in this module, the Cactus Integration test running on a Kubernetes AWS cluster 🌈

• CactusIntegrationTest: runs the Cactus Integration test on a Kubernetes AWS cluster

Module Contents

class toil.test.cactus.test_cactus_integration.CactusIntegrationTest(methodName)

Bases: toil.test.provisioners.clusterTest.AbstractClusterTest (#toil.test.provisioners.clusterTest.AbstractClusterTest)

Runs the Cactus Integration test on a Kubernetes AWS cluster 💖

• clusterName = 'cactus-test-'
• leaderNodeType = 't2.medium'
• clusterType = 'kubernetes'
• setUp(): sets up for the test. It has to be overridden to call this method and set self.jobStore
• test_cactus_integration()

---

✨ toil.test.conftest ✨

Functions

• rootpath(request): records the rootpath at the class level, so you can use it on a unittest.TestCase 🌸

Module Contents

toil.test.conftest.rootpath(request)

Records the rootpath at the class level, so you can use it on a unittest.TestCase 🌸

Parameters: request (pytest.FixtureRequest)

Return type: None

---

✨ toil.test.cwl ✨

Submodules

toil.test.cwl.conftest

Attributes

• logger
• collect_ignore

Functions

• pytest_cwl_execute_test(config, processfile, jobfile): uses Toil to run CWL tests, the same way toil-cwl-runner would 🔥

Module Contents

• toil.test.cwl.conftest.logger
• toil.test.cwl.conftest.collect_ignore = ['spec']

toil.test.cwl.conftest.pytest_cwl_execute_test(config, processfile, jobfile)

Uses Toil to execute CWL tests, the same way running toil-cwl-runner would 👑

Parameters

• config (cwltest.utils.CWLTestConfig)
• processfile (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• jobfile (Optional[str] (/usr/share/doc/python3/html/library/stdtypes.html#str))

Return type: Tuple[int, Optional[Dict[str, Any]]] (/usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/stdtypes.html#str)

💖 toil.test.cwl.cwlTest 💖

🎀 Attributes

• pkg_root — no description given, but it's one of the module's attributes 🌸
• log — no description given, just sitting there being an attribute ✨
• CONFORMANCE_TEST_TIMEOUT — no description given, bestie, but the name says it all ⏱️
• TesterFuncType — no description given 🦋

👑 Classes

• TestCWLWorkflow — the CWL tests that ship with Toil and don't involve the whole CWL conformance suite 💅
• TestCWLv10Conformance — runs the CWL 1.0 conformance tests in various environments 🌈
• TestCWLv11Conformance — runs the CWL 1.1 conformance tests in various environments 🌈
• TestCWLv12Conformance — runs the CWL 1.2 conformance tests in various environments 🌈
• ImportWorkersMessageHandler — checks whether any WorkerImportJob jobs ran during a workflow 🔥

💅 Functions

• run_conformance_tests(workDir, yml[, runner, caching, ...]) — this is the one that actually runs the CWL conformance tests, so it's giving main character 👑
• cwl_v1_0_spec(tmp_path) — no description given 📜
• cwl_v1_1_spec(tmp_path) — no description given 📜
• cwl_v1_2_spec(tmp_path) — no description given 📜
• test_workflow_echo_string_scatter_stderr_log_dir(tmp_path) — no description given 🧚
• test_log_dir_echo_no_output(tmp_path) — no description given 🧚
• test_log_dir_echo_stderr(tmp_path) — no description given 🧚
• test_filename_conflict_resolution(tmp_path) — no description given 🧚
• test_filename_conflict_resolution_3_or_more(tmp_path) — no description given 🧚
• test_filename_conflict_detection(tmp_path) — makes sure we don't just stage files over each other when using a container 🐳
• test_filename_conflict_detection_at_root(tmp_path) — makes sure we don't just stage files over each other 🫶
• test_pick_value_with_one_null_value(caplog, tmp_path) — makes sure toil-cwl-runner does not false log a warning when pickValue is (the original description just trails off right there, bestie 💀)
• test_workflow_echo_string(tmp_path) — no description given 🧚
• test_workflow_echo_string_scatter_capture_stdout(tmp_path) — no description given 🧚
• test_visit_top_cwl_class() — no description given 🧚
• test_visit_cwl_class_and_reduce() — no description given 🧚
• test_trim_mounts_op_nonredundant() — makes sure we don't remove all non-duplicate listings 🙈
• test_trim_mounts_op_redundant() — makes sure we remove all duplicate listings 🧹
• test_trim_mounts_op_partially_redundant() — makes sure we remove only the redundant listings in the CWL object and leave nonredundant listings intact ✨
• test_trim_mounts_op_mixed_urls_and_paths(): main character energy check that we clean up redundant listings in certain edge cases 💅
• test_trim_mounts_op_decodable_paths() 🦋
• test_trim_mounts_op_multiple_encodings() 🌸
• test_download_structure(tmp_path): makes sure download_structure makes the right calls to what it thinks is the file store 📦
• test_import_on_workers() 🧚
• test_missing_tmpdir_and_tmp_outdir(tmp_path): tmpdir_prefix and tmp_outdir_prefix don't need to exist before the workflow runs 🎀

---

✨ Module Contents ✨

• toil.test.cwl.cwlTest.pkg_root = b'.' 🌈
• toil.test.cwl.cwlTest.log 📝
• toil.test.cwl.cwlTest.CONFORMANCE_TEST_TIMEOUT = 10000 ⏰

🏃‍♀️ toil.test.cwl.cwlTest.run_conformance_tests

toil.test.cwl.cwlTest.run_conformance_tests(workDir, yml, runner=None,
caching=False, batchSystem=None, selected_tests=None,
selected_tags=None, skipped_tests=None, extra_args=None,
must_support_all_features=False, junit_file=None)

Okay so this one runs the whole CWL conformance test suite, bestie 💖

Parameters 💌

• workDir (str </usr/share/doc/python3/html/library/stdtypes.html#str>): the folder the tests run in 📁
• yml (str </usr/share/doc/python3/html/library/stdtypes.html#str>): the CWL test list YML that tells us which tests to pull from
• runner (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]): if you set this, it uses that cwl runner instead of the default toil-cwl-runner
• caching (bool </usr/share/doc/python3/html/library/functions.html#bool>): if True, Toil file store caching gets turned on 🧠
• batchSystem (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]): if set, uses this batch system instead of the default single_machine
• selected_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]): if set, a description of which test numbers to run (comma-separated numbers or ranges) 🔢
• selected_tags (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]): an alternative to selected_tests, runs the tests that have the given tags 🏷️
• skipped_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]): comma-separated string labels of the tests to skip 🙈
• extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]): extra arguments handed to the runner for each test
• must_support_all_features (bool </usr/share/doc/python3/html/library/functions.html#bool>): if set, the run fails when some optional CWL features aren't supported 💀
• junit_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]): the JUnit XML file the test info gets written to 📄

Return type: None

🌟 toil.test.cwl.cwlTest.TesterFuncType

• toil.test.cwl.cwlTest.TesterFuncType 🎭

🎀 class toil.test.cwl.cwlTest.TestCWLWorkflow

CWL tests that live in Toil and don't involve the whole CWL conformance suite. These check Toil-specific stuff, like which URL types are supported for inputs 🔗

💅 test_cwl_cmdline_input()

Checks that a CWL workflow with inputs specified on the command line runs and passes 🏆

Return type: None

🦋 revsort(cwl_filename, tester_fn, out_dir)

Parameters 💌

• cwl_filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• tester_fn (TesterFuncType)
• out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

🌸 revsort_no_checksum(cwl_filename, tester_fn, out_dir)

Parameters 💌

• cwl_filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• tester_fn (TesterFuncType)
• out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

📦 download(inputs, tester_fn, out_dir)

Parameters 💌

• inputs (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• tester_fn (TesterFuncType)
• out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

🧚 load_contents(inputs, tester_fn, out_dir)

Parameters 💌

• inputs (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• tester_fn (TesterFuncType)
• out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

📁 download_directory(inputs, tester_fn, out_dir)

Parameters 💌

• inputs (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• tester_fn (TesterFuncType)
• out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

🗂️ download_subdirectory(inputs, tester_fn, out_dir)

Parameters 💌

• inputs (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• tester_fn (TesterFuncType)
• out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

🌈 test_mpi(tmp_path)

Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

☁️ test_s3_as_secondary_file(tmp_path)

Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

🍓 test_run_revsort(tmp_path)

Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

🧁 test_run_revsort_nochecksum(tmp_path)

Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

🔥 test_run_revsort_no_container(tmp_path)

Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

⭐ test_run_revsort2(tmp_path)

Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

🫶 test_run_revsort_debug_worker(tmp_path)

Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type: None

👑 test_run_colon_output(tmp_path)

we're so close to the end of this section, bestie 🌈

test_run_dockstore_trs(tmp_path) ✨

• Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• Return type 💅: None

• test_glob_dir_bypass_file_store(tmp_path) 🦋
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_required_input_condition_protection(tmp_path) 🛡️
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_slurm_node_memory(tmp_path) 🧠
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_s3(tmp_path) ☁️
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_http(tmp_path) 🌸
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_https(tmp_path) 🔒
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_https_reference(tmp_path) 🔗
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_file(tmp_path) 📁
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_directory_s3(tmp_path) 🗂️
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_directory_s3_reference(tmp_path) 🗂️
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_directory_file(tmp_path) 🗂️
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_subdirectory_s3(tmp_path) 📂
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_download_subdirectory_file(tmp_path) 📂
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_load_contents_s3(tmp_path) 📦
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_load_contents_http(tmp_path) 📦
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_load_contents_https(tmp_path) 📦
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_load_contents_file(tmp_path) 📦
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_bioconda(tmp_path) 🧬
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_default_args(tmp_path) 🎯
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_biocontainers(tmp_path) 🐳
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_cuda(tmp_path) 🎮
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_restart(tmp_path) 🔁
  • Enable restarts with toil-cwl-runner: run the failing test, then re-run the correct test. Only implemented for single machine 🙈
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_caching(tmp_path) 💾
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_streamable(tmp_path, extra_args=None) 🧜
  • Checks that a file with 'streamable'=True is a named pipe. This is a CWL1.2 feature ✨
  • Parameters 🎀:
    • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
    • extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])
  • Return type 💅: None
• test_streamable_reference(tmp_path) 🌊
  • Checks that a streamable file is still a stream even when it gets passed around by URI.
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_preemptible(tmp_path) 💜
  • Makes sure the http://arvados.org/cwl#UsePreemptible extension is supported.
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_preemptible_expression(tmp_path) 💜
  • Makes sure the http://arvados.org/cwl#UsePreemptible extension gets validated.
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_missing_import(tmp_path) 🙈
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_optional_secondary_files_exists(tmp_path) 📎
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• test_optional_secondary_files_missing(tmp_path) 📎
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: None
• toil.test.cwl.cwlTest.cwl_v1_0_spec(tmp_path) 📜
  • Parameters 🎀: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • Return type 💅: collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]
• class toil.test.cwl.cwlTest.TestCWLv10Conformance 👑
  • Runs the CWL 1.0 conformance tests in various environments 🌈
  • test_run_conformance_with_caching(cwl_v1_0_spec) 💾
    • Parameters 🎀: cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
    • Return type 💅: None
  • test_run_conformance(cwl_v1_0_spec, batchSystem=None, caching=False, selected_tests=None, skipped_tests=None, extra_args=None) 🏃
    • Parameters 🎀:
      • cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
      • batchSystem (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])
      • caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)
• selected_tests — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str> 🌸
• skipped_tests — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str> 🙈
• extra_args — Optional[list[str]] </usr/share/doc/python3/html/library/stdtypes.html#list> </usr/share/doc/python3/html/library/stdtypes.html#str> ✨

Return type: None 💖

✨ test_lsf_cwl_conformance(cwl_v1_0_spec, caching=False)

Parameters 🎀

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>

Return type: None 💅

✨ test_slurm_cwl_conformance(cwl_v1_0_spec, caching=False)

Parameters 🎀

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>

Return type: None 💅

✨ test_torque_cwl_conformance(cwl_v1_0_spec, caching=False)

Parameters 🎀

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>

Return type: None 💅

✨ test_gridengine_cwl_conformance(cwl_v1_0_spec, caching=False)

Parameters 🎀

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>

Return type: None 💅

✨ test_mesos_cwl_conformance(cwl_v1_0_spec, caching=False)

Parameters 🎀

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>

Return type: None 💅

✨ test_kubernetes_cwl_conformance(cwl_v1_0_spec, caching=False)

Parameters 🎀

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>

Return type: None 💅

🦋 test_lsf_cwl_conformance_with_caching(cwl_v1_0_spec)

Parameters

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

🦋 test_slurm_cwl_conformance_with_caching(cwl_v1_0_spec)

Parameters

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

🦋 test_torque_cwl_conformance_with_caching(cwl_v1_0_spec)

Parameters

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

🦋 test_gridengine_cwl_conformance_with_caching(cwl_v1_0_spec)

Parameters

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

🦋 test_mesos_cwl_conformance_with_caching(cwl_v1_0_spec)

Parameters

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

🦋 test_kubernetes_cwl_conformance_with_caching(cwl_v1_0_spec)

Parameters

• cwl_v1_0_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

---

✨ toil.test.cwl.cwlTest.cwl_v1_1_spec(tmp_path)

Parameters 🎀

• tmp_path — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type 🌈: collections.abc.Generator[pathlib.Path] </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator> </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

👑 class toil.test.cwl.cwlTest.TestCWLv11Conformance

Okay, this class runs the CWL 1.1 conformance tests across different environments 🌈

💖 test_run_conformance(cwl_v1_1_spec, caching=False, batchSystem=None, skipped_tests=None, extra_args=None)

Parameters 🎀

• cwl_v1_1_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>
• batchSystem — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>
• skipped_tests — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>
• extra_args — Optional[list[str]] </usr/share/doc/python3/html/library/stdtypes.html#list> </usr/share/doc/python3/html/library/stdtypes.html#str>

Return type: None 💅

🦋 test_run_conformance_with_caching(cwl_v1_1_spec)

Parameters

• cwl_v1_1_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

🦋 test_kubernetes_cwl_conformance(cwl_v1_1_spec, caching=False)

Parameters

• cwl_v1_1_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>

Return type: None 💅

🦋 test_kubernetes_cwl_conformance_with_caching(cwl_v1_1_spec)

Parameters

• cwl_v1_1_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

---

✨ toil.test.cwl.cwlTest.cwl_v1_2_spec(tmp_path)

Parameters 🎀

• tmp_path — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type 🌈: collections.abc.Generator[pathlib.Path] </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator> </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

👑 class toil.test.cwl.cwlTest.TestCWLv12Conformance

Okay, this one runs the CWL 1.2 conformance tests across different environments 🌈

💖 test_run_conformance(cwl_v1_2_spec, runner=None, caching=False, batchSystem=None, selected_tests=None, skipped_tests=None, extra_args=None, must_support_all_features=False, junit_file=None)

Parameters 🎀

• cwl_v1_2_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• runner — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>
• caching — bool </usr/share/doc/python3/html/library/functions.html#bool>
• batchSystem — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>
• selected_tests — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>
• skipped_tests — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>
• extra_args — Optional[list[str]] </usr/share/doc/python3/html/library/stdtypes.html#list> </usr/share/doc/python3/html/library/stdtypes.html#str>
• must_support_all_features — bool </usr/share/doc/python3/html/library/functions.html#bool>
• junit_file — Optional[str] </usr/share/doc/python3/html/library/stdtypes.html#str>

Return type: None 💅

🦋 test_run_conformance_with_caching(cwl_v1_2_spec)

Parameters

• cwl_v1_2_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

🦋 test_run_conformance_with_task_caching(cwl_v1_2_spec, tmp_path)

Parameters

• cwl_v1_2_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>
• tmp_path — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

🦋 test_run_conformance_with_in_place_update(cwl_v1_2_spec)

The main thing here: with --bypass-file-store, in-place update has to work properly on a single node, and it can't break any other features 💀🫶

Parameters

• cwl_v1_2_spec — pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

Return type: None 💅

Return type: None 🫶

• test_kubernetes_cwl_conformance_with_caching(cwl_v1_2_spec) — 🦋 the kubernetes CWL conformance test, but the caching version
  • Parameters: cwl_v1_2_spec (pathlib.Path)
  • Return type: None
• test_wes_server_cwl_conformance(cwl_v1_2_spec) — 🌈 runs the CWL conformance tests through WES. You must set TOIL_WES_ENDPOINT or it won't work bestie. If your WES server wants auth, set TOIL_WES_USER and TOIL_WES_PASSWORD too 🔐

To run it manually, use this:

TOIL_WES_ENDPOINT=http://localhost:8080 TOIL_WES_USER=test TOIL_WES_PASSWORD=password python -m pytest src/toil/test/cwl/cwlTest.py::TestCWLv12Conformance::test_wes_server_cwl_conformance -vv --log-level INFO --log-cli-level INFO

• Parameters: cwl_v1_2_spec (pathlib.Path)
• Return type: None

• toil.test.cwl.cwlTest.test_workflow_echo_string_scatter_stderr_log_dir(tmp_path) 🍓
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_log_dir_echo_no_output(tmp_path) 🙈
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_log_dir_echo_stderr(tmp_path) 📣
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_filename_conflict_resolution(tmp_path) 🧩
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_filename_conflict_resolution_3_or_more(tmp_path) 🧩
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_filename_conflict_detection(tmp_path) 👀 Make sure we don't just stage files over each other when using a container.
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_filename_conflict_detection_at_root(tmp_path) 👀 Again, make sure we don't stage files over each other, specifically when using a container and the files sit at the root of the work dir. The vibe check is strict here 💅
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_pick_value_with_one_null_value(caplog, tmp_path) 🪄 Make sure toil-cwl-runner does not falsely log a warning when pickValue is used but outputSource only has one null value. See: #3991.
  • Parameters:
    • caplog (pytest.LogCaptureFixture)
    • tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_workflow_echo_string(tmp_path) 🌸
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_workflow_echo_string_scatter_capture_stdout(tmp_path) 🌸
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_visit_top_cwl_class() 🦋
  • Return type: None
• toil.test.cwl.cwlTest.test_visit_cwl_class_and_reduce() 🦋
  • Return type: None
• toil.test.cwl.cwlTest.test_trim_mounts_op_nonredundant() ✂️ Make sure we don't remove all the non-duplicate listings.
  • Return type: None
• toil.test.cwl.cwlTest.test_trim_mounts_op_redundant() ✂️ Make sure we remove all the duplicate listings.
  • Return type: None
• toil.test.cwl.cwlTest.test_trim_mounts_op_partially_redundant() ✂️ Make sure we remove only the redundant listings in the CWL object and leave the nonredundant listings intact.
  • Return type: None
• toil.test.cwl.cwlTest.test_trim_mounts_op_mixed_urls_and_paths() ✂️ Ensure we remove redundant listings in certain edge cases.
  • Return type: None
• toil.test.cwl.cwlTest.test_trim_mounts_op_decodable_paths() ✂️
  • Return type: None
• toil.test.cwl.cwlTest.test_trim_mounts_op_multiple_encodings() ✂️
  • Return type: None
• toil.test.cwl.cwlTest.test_download_structure(tmp_path) 📂 Make sure download_structure makes the right calls to what it thinks is the file store.
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• toil.test.cwl.cwlTest.test_import_on_workers() 🚚
  • Return type: None
• toil.test.cwl.cwlTest.test_missing_tmpdir_and_tmp_outdir(tmp_path) 🫧 tmpdir_prefix and tmp_outdir_prefix do not need to exist before you run the workflow. Literally no setup needed, bestie 💖
  • Parameters: tmp_path (pathlib.Path)
  • Return type: None
• class toil.test.cwl.cwlTest.ImportWorkersMessageHandler 🕵️‍♀️ This one is a detective 🔍
  • Bases: _stream_handler
  • Detects whether any WorkerImportJob jobs ran during a workflow.
  • Attribute: detected = False
  • Method: emit(record). Emits a record 📨
    • If a formatter is set, it gets used to format the record. Then the record gets written to the stream with a trailing newline.
    • If there's exception info, it's formatted with traceback.print_exception and tacked onto the stream.
    • If the stream has an 'encoding' attribute, that's what decides how the output gets written to the stream.
    • Parameters: record (logging.LogRecord)
    • Return type: None

✨ toil.test.docs ✨

Submodules

• toil.test.docs.scriptsTest 📜

Attributes

• pkg_root (no value listed here, just the name 🌿)

Classes

• toil.test.docs.scriptsTest.ToilDocumentationTest 🎀 Tests for scripts in the Toil tutorials.

Module Contents

• toil.test.docs.scriptsTest.pkg_root = b'.' 🌿
• class toil.test.docs.scriptsTest.ToilDocumentationTest(methodName='runTest') 💜
  • Bases: toil.test.ToilTest
  • Tests for scripts in the Toil tutorials.
  • classmethod setUpClass() — 🧁 Hook method for setting up the class fixture before the tests in the class run.
  • tearDown() — 🧹 Hook method for deconstructing the test fixture after testing it.
    • Return type: None
  • checkExitCode(script, extra_args=[]) 🚦
    • Parameters: extra_args (list[str])
  • checkExpectedOut(script, expectedOutput) 📤
  • checkExpectedPattern(script, expectedPattern) 🔎
  • testStats() 📊
  • testDynamic() 🌀
  • testEncapsulation() 🫧
  • testEncapsulation2() 🫧
  • testHelloworld() 👋
  • testInvokeworkflow() 🚀
  • testInvokeworkflow2() 🚀
  • testJobFunctions() 🧠
  • testManaging() 🗂️
  • testManaging2() 🗂️
  • testMultiplejobs() 🧵
  • testMultiplejobs2() 🧵
  • testMultiplejobs3() 🧵
  • testPromises2() 🤞
  • testQuickstart() 🏁
  • testRequirements() 📋
  • testArguments() 🎛️
  • testDocker() 🐳
  • testPromises() 🤞
  • testServices() 🛎️
  • testStaging() 📦

✨ toil.test.jobStores ✨

Submodules

• toil.test.jobStores.jobStoreTest 🗄️

Attributes

• logger (no value listed here, just the name 🪵)

✨ Classes ✨

okay so here's the whole test class lineup, bestie 💅

• AbstractJobStoreTest — hides this abstract base class from unittest's test case loader so it doesn't get picked up as a real test 🙈
• AbstractEncryptedJobStoreTest — no blurb on this one 🤷
• FileJobStoreTest — legacy common base class for Toil tests 🗂️
• GoogleJobStoreTest — legacy common base class for Toil tests ☁️
• AWSJobStoreTest — legacy common base class for Toil tests 🟧
• InvalidAWSJobStoreTest — legacy common base class for Toil tests 💀
• EncryptedAWSJobStoreTest — legacy common base class for Toil tests 🔐
• StubHttpRequestHandler — a simple HTTP request handler that only speaks GET and HEAD 🌐

✨ Functions ✨

• google_retry(x) — no blurb listed 🫶
• tearDownModule() — no blurb listed 🧹

✨ Module Contents ✨

here's everything that lives in the module, all in one spot 🧁

• toil.test.jobStores.jobStoreTest.google_retry(x)
• toil.test.jobStores.jobStoreTest.logger
• toil.test.jobStores.jobStoreTest.tearDownModule()

🎀 class toil.test.jobStores.jobStoreTest.AbstractJobStoreTest

Hide abstract base class from unittest's test case loader 🙈 this is literally the whole point of this one.

Here's the Stack Overflow thread this trick comes from 🧵: http://stackoverflow.com/questions/1323455/python-unit-test-with-base-and-sub-class#answer-25695512

• class Test(methodName='runTest') 🌸
  • Bases: toil.test.ToilTest
  • Legacy common base class for Toil tests. New tests should be made in the 'pytest' style and not use this class.
  • When running tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files placed. The directory gets created if it doesn't exist. The path can be relative, in which case it's treated as relative to the project root. If TOIL_TEST_TEMP isn't defined, temporary files and directories go in the system's default location for such files, and any leftovers from tests get automatically removed during tear down. Otherwise, left-over files won't be removed.
  • classmethod setUpClass() — hook method for setting up the class fixture before running tests in the class 🏗️
  • setUp() — hook method for setting up the test fixture before exercising it 🧪
  • tearDown() — hook method for deconstructing the test fixture after testing it 🧼
  • testInitialState() — ensures proper handling of nonexistent files 👻
  • testJobCreation() — tests creation of a job. Does the job exist in the jobstore it's supposed to be in? Are its attributes what we expect? 🧾
  • testConfigEquality() — ensures the command line configurations get loaded and stored successfully. In setUp(), self.jobstore1 is created and initialized. In this test, after creating newJobStore, .resume() looks for a previously instantiated job store and loads its config options. These are expected to be equal but not the same object ⚖️
  • testJobLoadEquality() — tests that a job created via one JobStore instance can be loaded from another 🔁
  • testChildLoadingEquality() — tests that loading a child job operates as expected 👶
  • testPersistantFilesToDelete() — makes sure updating a job persists filesToDelete. The pattern goes like this: files to be deleted atomically with a job update are referenced in the "filesToDelete" array, which gets persisted to disk first. If things go wrong during the update, this list is used to make sure the updated job and the files are never both visible at the same time 🗑️
  • testUpdateBehavior() — tests the proper behavior when updating jobs 🔄
  • testJobDeletions() — tests the consequences of deleting jobs 💥
  • testSharedFiles() — tests the sharing of files 📂
  • testReadWriteSharedFilesTextMode() — checks if text mode is compatible with shared file streams 📝
  • testReadWriteFileStreamTextMode() — checks if text mode is compatible with file streams 📄
  • testPerJobFiles() — tests the behavior of files on jobs 🗃️
  • testStatsAndLogging() — tests the behavior of reading and writing stats and logging 📊
  • testWriteLogFiles() — tests writing log files 🪵
  • testBatchCreate() — tests creation of many jobs at once 🏭
  • testGrowingAndShrinkingJob() — makes sure jobs update correctly when they grow or shrink 📈
  • externalStoreCache — attribute, no blurb listed 🗄️
  • classmethod cleanUpExternalStores() — cleans up external stores 🧽
  • mpTestPartSize = 5242880 — the multipart test part size attribute 📏
  • classmethod makeImportExportTests() — makes the import and export tests 📦
  • testImportHttpFile() — tests importing a file over HTTP 🌍
  • testImportFtpFile() — tests importing a file over FTP 📡
  • testFileDeletion() — intended to cover batch deletion of items in the AWSJobStore, but it doesn't hurt to run it on the other job stores too 🧨
  • testMultipartUploads() — meant to cover multi-part uploads in the AWSJobStore, but it doesn't hurt to run it against the other job stores as well 🧩
  • testZeroLengthFiles() — tests reading and writing of empty files 🫥
  • testLargeFile() — tests reading and writing of large files 🐘
  • fetch_url(url) — fetches the given URL. Throws an error if it can't be fetched in a reasonable number of attempts 🛰️
    • Parameters: url (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
    • Return type: None
  • assertUrl(url) — no blurb listed 🔗
  • testCleanCache() — no blurb listed 🧹
  • testPartialReadFromStream() — tests whether readFileStream will deadlock on a partial read 🔒
  • testDestructionOfCorruptedJobStore() — no blurb listed 🧟
  • testDestructionIdempotence() — no blurb listed ♻️
  • testEmptyFileStoreIDIsReadable() — creates an empty fileStoreID and attempts to read from it 🫧

🎀 class toil.test.jobStores.jobStoreTest.AbstractEncryptedJobStoreTest

• class Test(methodName='runTest') 🔐
  • Bases: AbstractJobStoreTest
  • A test of job stores that use encryption 🛡️
  • setUp() — hook method for setting up the test fixture before exercising it 🧪
  • tearDown() — hook method for deconstructing the test fixture after testing it 🧼
  • testEncrypted() — creates an encrypted file and reads it in encrypted mode. Then it tries again with encryption off, to make sure that fails ❌

⚠️ testEncrypted() is the one that proves the encryption actually bites. If reading with encryption off doesn't fail, something is very wrong, so don't ignore it 😭

🎀 class toil.test.jobStores.jobStoreTest.FileJobStoreTest(methodName='runTest')

Bases: AbstractJobStoreTest 🗂️

Legacy common base class for Toil tests. Same vibe as before, bestie 💜

New tests should be made in the 'pytest' style, so bestie, don't reach for this class for anything new 🙈 It's giving legacy energy and we're not doing that anymore 💅

New tests should be made in the 'pytest' style and not use this class.

When you're running tests, you can optionally set the TOIL_TEST_TEMP environment variable 🌸 to the path of a directory where you want your temporary test files to live. The directory gets created if it doesn't exist yet. The path can be relative, and if it is, it's relative to the project root 🗺️. If TOIL_TEST_TEMP isn't defined, temporary files and directories get made in the system's default location for that sort of thing, and any temp files or directories left over from tests get automatically removed during tear down 🧹. But if TOIL_TEST_TEMP is defined, left-over files will not be removed.

• testPreserveFileName() — Checks that the fileID ends with the file name you gave it ✨
• test_jobstore_init_preserves_symlink_path() — Makes sure that if you hand a fileJobStore a symlink to a directory, it doesn't de-reference it 🔗
• test_jobstore_does_not_leak_symlinks() — If we link imports into the FileJobStore, you can't get hardlinks to symlinks. No leaks, periodt 🙅‍♀️
• test_file_link_imports() — Checks that imported files get symlinked when expected 🦋
• test_symlink_read_control() — Checks that files are read by symlink when expected 👀

---

🎀 class toil.test.jobStores.jobStoreTest.GoogleJobStoreTest(methodName='runTest')

Bases: AbstractJobStoreTest

This is a legacy common base class for Toil tests, so it's not the move for new stuff. New tests should be made in the 'pytest' style and not use this class.

When running tests, you can optionally set TOIL_TEST_TEMP to the path of a directory where your temp test files should go. The directory gets created if it doesn't exist. Relative paths are relative to the project root. If TOIL_TEST_TEMP isn't defined, temp files and directories go in the system's default location, and any leftovers from tests get automatically removed during tear down. If TOIL_TEST_TEMP is defined, left-over files will not be removed.

• projectID
• headers

---

🎀 class toil.test.jobStores.jobStoreTest.AWSJobStoreTest(methodName='runTest')

Bases: AbstractJobStoreTest

Legacy common base class for Toil tests, vibes only for old code. New tests should be made in the 'pytest' style and not use this class.

When running tests, you can optionally set TOIL_TEST_TEMP to a directory path for your temp test files. It gets created if it doesn't exist. Relative paths are relative to the project root. If TOIL_TEST_TEMP isn't defined, temp files and directories go in the system's default location, and leftovers get automatically removed during tear down. If TOIL_TEST_TEMP is defined, left-over files will not be removed.

• testOverlargeJob()
• testMultiThreadImportFile() — Tests that importFile is thread-safe 🧵

Return type: None

---

🎀 class toil.test.jobStores.jobStoreTest.InvalidAWSJobStoreTest(methodName='runTest')

Bases: toil.test.ToilTest

This is a legacy common base class for Toil tests. New tests should be made in the 'pytest' style and not use this class.

When running tests, you can optionally set TOIL_TEST_TEMP to the path of a directory where you want temporary test files placed. It gets created if it doesn't exist. Relative paths are relative to the project root. If TOIL_TEST_TEMP isn't defined, temp files and directories get created in the system's default location, and leftovers from tests get automatically removed during tear down. If TOIL_TEST_TEMP is defined, left-over files will not be removed.

• testInvalidJobStoreName() 🙅‍♀️

---

🎀 class toil.test.jobStores.jobStoreTest.EncryptedAWSJobStoreTest(methodName='runTest')

Bases: AWSJobStoreTest, AbstractEncryptedJobStoreTest

Legacy common base class for Toil tests. New tests should be made in the 'pytest' style and not use this class. 🫶

When running tests, you can optionally set TOIL_TEST_TEMP to the path of a directory for temporary test files. It gets created if it doesn't exist. Relative paths are relative to the project root. If TOIL_TEST_TEMP isn't defined, temp files and directories go to the system's default location, and leftovers get automatically removed during tear down. If TOIL_TEST_TEMP is defined, left-over files will not be removed.

---

🎀 class toil.test.jobStores.jobStoreTest.StubHttpRequestHandler(*args, directory=None, **kwargs)

Bases: http.server.SimpleHTTPRequestHandler (see /usr/share/doc/python3/html/library/http.server.html#http.server.SimpleHTTPRequestHandler)

Okay so this is a simple HTTP request handler with GET and HEAD commands 📡. It serves files from the current directory and any of its subdirectories. The MIME type for files is determined by calling the .guess_type() method.

The GET and HEAD requests are identical, except that the HEAD request omits the actual contents of the file.

fileContents = 'A good programmer looks both ways before crossing a one-way street'

• do_GET() — Serves a GET request 🍓

---

🌈 toil.test.lib

Submodules

• toil.test.lib.aws

---

🌈 toil.test.lib.aws

Submodules

• toil.test.lib.aws.test_iam
• toil.test.lib.aws.test_s3
• toil.test.lib.aws.test_utils

---

🧚 toil.test.lib.aws.test_iam

Attributes

• logger

Classes

• IAMTest — Checks that given permissions and associated functions perform correctly 💜

Module Contents

• toil.test.lib.aws.test_iam.logger

🎀 class toil.test.lib.aws.test_iam.IAMTest(methodName='runTest')

Bases: toil.test.ToilTest

Checks that given permissions and associated functions perform correctly 🔐

• test_permissions_iam()

Return type: None

• test_negative_permissions_iam()

Return type: None

• test_wildcard_handling()

Return type: None

• test_get_policy_permissions()

Return type: None

• test_create_delete_iam_role()

Return type: None

---

🧚 toil.test.lib.aws.test_s3

Attributes

• logger

Classes

• S3Test — Confirms the workarounds for us-east-1 🇺🇸

Module Contents

• toil.test.lib.aws.test_s3.logger

🎀 class toil.test.lib.aws.test_s3.S3Test(methodName='runTest')

Bases: toil.test.ToilTest

Confirms the workarounds for us-east-1, no cap 💅

• s3_resource: mypy_boto3_s3.S3ServiceResource | None (the None points to the Python constants docs at /usr/share/doc/python3/html/library/constants.html#None)
• setUpClass() (classmethod) — Hook method for setting up the class fixture before running tests in the class.

Return type: None

• test_create_bucket() — Tests bucket creation for us-east-1 🪣

Return type: None

• test_get_bucket_location_public_bucket() — Tests getting the bucket location for a bucket we don't own 👀

Return type: None

---

🧚 toil.test.lib.aws.test_utils

Attributes

• logger

Classes

• TagGenerationTest — Tests tag generation from environment variables 🏷️

Module Contents

• toil.test.lib.aws.test_utils.logger

🎀 class toil.test.lib.aws.test_utils.TagGenerationTest(methodName='runTest')

Bases: toil.test.ToilTest

Tests tag generation from environment variables ✨

• test_build_tag()

Return type: None

• test_empty_aws_tags()

Return type: None

• test_incorrect_json_object()

Return type: None

• test_incorrect_json_emoji()

Return type: None

• test_build_tag_with_tags()

Return type: None

✨ toil.test.lib.dockerTest ✨

🏷️ Attributes

• logger — a module attribute 📝 with no description attached

🧁 Classes

• TestDocker — the one class here. It tests dockerCall and makes sure no containers are left around 👻

📦 Module Contents

• toil.test.lib.dockerTest.logger 📝
• class toil.test.lib.dockerTest.TestDocker — tests dockerCall and ensures no containers are left around 🐳

🐳 TestDocker

okay so TestDocker is the whole main character 💅 It tests dockerCall and makes sure no containers are left around.

• dockerTestLogLevel = 'INFO' — a class attribute set to 'INFO' ✨

🧪 Methods

Heads up, bestie: the type names in these parameter lists are links in the original page. They point into the Python 3 docs at /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path, /usr/share/doc/python3/html/library/functions.html#bool and /usr/share/doc/python3/html/library/functions.html#int.

• testDockerClean(tmp_path, caching=False, detached=True, rm=True, deferParam=None) — this one is seriously iconic 🔥 It runs the test container, which creates a file in the work dir and then sleeps for 5 minutes. It makes sure the calling job gets SIGKILLed after a minute, leaving behind the spooky/ghost/zombie container 👻 It also checks that the container gets killed on batch system shutdown (through the deferParam mechanism) 💀
  • Parameters:
    • tmp_path (pathlib.Path)
    • caching (bool)
    • detached (bool)
    • rm (bool)
    • deferParam (Optional[int])
  • Return type: None
• testDockerClean_CRx_FORGO(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_CRx_STOP(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_CRx_RM(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_CRx_None(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_CxD_FORGO(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_CxD_STOP(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_CxD_RM(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_CxD_None(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_Cxx_FORGO(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_Cxx_STOP(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_Cxx_RM(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_Cxx_None(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xRx_FORGO(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xRx_STOP(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xRx_RM(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xRx_None(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xxD_FORGO(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xxD_STOP(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xxD_RM(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xxD_None(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xxx_FORGO(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xxx_STOP(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xxx_RM(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerClean_xxx_None(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerPipeChain(tmp_path, caching=False) — this checks the piping API for dockerCall() 🔗 The API kicks in when you pass a list of argument lists as parameters, and the commands get piped together into a chain. For example, parameters=[ ['printf', 'x\n y\n'], ['wc', '-l'] ] should run printf 'x\n y\n' | wc -l 🧚
  • Parameters:
    • tmp_path (pathlib.Path)
    • caching (bool)
  • Return type: None
• testDockerPipeChainErrorDetection(tmp_path, caching=False) — by default, running cmd1 | cmd2 | ... | cmdN only throws an error if cmdN fails 🙈 That means all manner of errors can get silently missed. This test makes sure the piping API for dockerCall() throws an exception if any non-last command in the chain fails 🚨
  • Parameters:
    • tmp_path (pathlib.Path)
    • caching (bool)
  • Return type: None
• testNonCachingDockerChain(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testNonCachingDockerChainErrorDetection(tmp_path) — no description, just vibes 🤷‍♀️ Takes tmp_path (pathlib.Path) and returns None
• testDockerLogs(tmp_path, stream=False, demux=False) — tests the different log outputs when deatch=False 📜 Only tmp_path (pathlib.Path) shows up in the parameter list here, and it returns None. The stream and demux arguments aren't described on this page.
• stream (bool, see /usr/share/doc/python3/html/library/functions.html#bool) 💖
• demux (bool, see /usr/share/doc/python3/html/library/functions.html#bool) ✨

Return type: None 💅

• testDockerLogs_Stream(tmp_path) 🌊
  • Parameters
    • tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • Return type: None
• testDockerLogs_Demux(tmp_path) 🧃
  • Parameters
    • tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • Return type: None
• testDockerLogs_Demux_Stream(tmp_path) 🦋
  • Parameters
    • tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • Return type: None

---

🌸 toil.test.lib.test_conversions

🫧 Attributes

• logger — no description given, it just vibes 🙈

🎀 Classes

• ConversionTest — Legacy common base class for Toil tests.

💜 Module Contents

• toil.test.lib.test_conversions.logger
• class toil.test.lib.test_conversions.ConversionTest(methodName='runTest')

Bases: toil.test.ToilTest

This is the legacy common base class for Toil tests 🕰️ Real talk, new tests should be made in the pytest style and not use this class, so don't you dare start something new with it 🙅‍♀️

Okay, here's the tea on TOIL_TEST_TEMP 🍵 When you run tests, you can optionally set it to the path of a directory where you want temporary test files to go. The directory gets created if it doesn't exist yet. The path can be relative, in which case it's assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temp files and directories get made in the system's default location for that stuff, and any leftovers from the tests get removed automatically during tear down. Otherwise, left-over files will not be removed 🧹

• test_convert()
• test_human2bytes()
• test_hms_duration_to_seconds()

---

🦢 toil.test.lib.test_ec2

🫧 Attributes

• logger — no description given, it just vibes 🙈

🎀 Classes

• FlatcarFeedTest — Test accessing the Flatcar AMI release feed, independent of the AWS API
• AMITest — Legacy common base class for Toil tests.

💜 Module Contents

• toil.test.lib.test_ec2.logger
• class toil.test.lib.test_ec2.FlatcarFeedTest(methodName='runTest')

Bases: toil.test.ToilTest

Okay so this one checks the Flatcar AMI release feed, and it's giving independent of the AWS API 🛰️

• test_parse_archive_feed() — Make sure we can get a Flatcar release from the Internet Archive.
• test_parse_beta_feed() — Make sure we can get a Flatcar release from the beta channel.
• test_parse_stable_feed() — Make sure we can get a Flatcar release from the stable channel.
• class toil.test.lib.test_ec2.AMITest(methodName='runTest')

Bases: toil.test.ToilTest

Another legacy common base class for Toil tests 🕰️ Same deal as before, new tests should be made in the pytest style and not use this class 🙅‍♀️

Same TOIL_TEST_TEMP energy here 🍵 You can optionally set it to the path of a directory where you want temporary test files placed. The directory gets created if it doesn't exist. The path may be relative, in which case it's assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories get created in the system's default location, and any temporary files or directories left over from tests will be removed automatically during tear down. Otherwise, left-over files will not be removed.

• classmethod setUpClass() — Hook method for setting up class fixture before running tests in the class.
• test_fetch_flatcar()
• test_fetch_arm_flatcar() — Test flatcar AMI finder architecture parameter.

---

📜 toil.test.lib.test_history

🫧 Attributes

• logger — no description given, it just vibes 🙈

🎀 Classes

• TestHistory — Tests for Toil history tracking.

💜 Module Contents

• toil.test.lib.test_history.logger
• class toil.test.lib.test_history.TestHistory

Tests for Toil history tracking. Each test gets its own history database 🗂️

• private_history_manager(tmp_path, monkeypatch)
  • Parameters
    • tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • monkeypatch (pytest.MonkeyPatch)
  • Return type: collections.abc.Generator[None], see /usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator
• make_fake_workflow(workflow_id)
  • Parameters
    • workflow_id (str, see /usr/share/doc/python3/html/library/stdtypes.html#str)
  • Return type: None
• test_history_submittable_detection() 🔍 Make sure that a submittable workflow shows up as such before submission and doesn't afterward. Return type: None
• test_history_deletion() 🗑️ Return type: None
• test_history_size_limit() 📏 Make sure the database size can be controlled. Return type: None

---

👑 toil.test.lib.test_misc

🫧 Attributes

• logger — no description given, it just vibes 🙈

🎀 Classes

• UserNameAvailableTest — Make sure we can get user names when they are available.
• UserNameUnvailableTest — Make sure we can get something for a user name when user names are not (the table cuts off right there, bestie, but the full version is down below 👇)
• UserNameVeryBrokenTest — Make sure we can get something for a user name when user name fetching is (cut off here too, full version below 👇)

💜 Module Contents

• toil.test.lib.test_misc.logger
• class toil.test.lib.test_misc.UserNameAvailableTest(methodName='runTest')

Bases: toil.test.ToilTest

Make sure we can get user names when they are available 🙌

• test_get_user_name()
• class toil.test.lib.test_misc.UserNameUnvailableTest(methodName='runTest')

Bases: toil.test.ToilTest

Make sure we can get something for a user name when user names are not available 🫥

• setUp() — Hook method for setting up the test fixture before exercising it.
• tearDown() — Hook method for deconstructing the test fixture after testing it.
• test_get_user_name()
• class toil.test.lib.test_misc.UserNameVeryBrokenTest(methodName='runTest')

Bases: toil.test.ToilTest

Make sure we can get something for a user name when user name fetching is broken in ways we did not expect 💀

• setUp() — Hook method for setting up the test fixture before exercising it.
• tearDown() — Hook method for deconstructing the test fixture after testing it.
• test_get_user_name()

🌸 toil.test.lib.test_trs 🌸

okay bestie, this one is all about testing that we can look up workflows on Dockstore 🧚

✨ Attributes ✨

• logger — the only attribute here, with its value left blank in this listing 💅

💅 Classes 💅

• DockstoreLookupTest — make sure we can look up workflows on Dockstore. ✨

🎀 Module Contents 🎀

• toil.test.lib.test_trs.logger

🦋 DockstoreLookupTest 🦋

class toil.test.lib.test_trs.DockstoreLookupTest(methodName='runTest')

Bases: toil.test.ToilTest

Same vibe as the table up top: this class makes sure we can look up workflows on Dockstore 💖

📖 read_result(url_or_path) 📖

Reads a file or a URL, in binary mode so we can test binary file support too 🫶

This is the move because it lets us check that we have the right workflow contents without caring how they're being shown to us. no cap, very clean 🔥

Parameters: url_or_path (str, which points to /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: IO[bytes] (with bytes pointing to /usr/share/doc/python3/html/library/stdtypes.html#bytes)

🌈 test_lookup_from_page_url() 🌈

Return type: None 💀

🌈 test_lookup_from_trs_with_version() 🌈

Return type: None 💀

🌈 test_lookup_from_trs_no_version() 🌈

Return type: None 💀

🌈 test_get() 🌈

Return type: None 💀

🌈 test_get_from_trs_cached() 🌈

Return type: None 💀

🌈 test_lookup_from_trs_nonexistent_workflow() 🌈

Return type: None 💀

🌈 test_lookup_from_trs_nonexistent_workflow_bad_format() 🌈

Return type: None 💀

🌈 test_lookup_from_trs_nonexistent_version() 🌈

Return type: None 💀

🌈 test_get_nonexistent_workflow() 🌈

Return type: None 💀

🌈 test_get_nonexistent_version() 🌈

Return type: None 💀

🌈 test_get_nonexistent_workflow_bad_format() 🌈

Return type: None 💀

---

🧚 toil.test.lib.test_url 🧚

this module is where the URL access tests live, and they're giving thorough 🦋

✨ Attributes ✨

• logger — the only attribute listed, value left blank 💅

💅 Classes 💅

• TestURLAccess — test URLAccess class handling read, list, and checking the size/existence of resources at given URL 🫶

🎀 Module Contents 🎀

• toil.test.lib.test_url.logger

🦋 TestURLAccess 🦋

class toil.test.lib.test_url.TestURLAccess

Test URLAccess class handling read, list, and checking the size/existence of resources at given URL. the whole main character energy of URL testing 👑

📖 test_get_url_access(httpserver) 📖

Parameters: httpserver (pytest_httpserver.HTTPServer)

Return type: None 💀

🌈 test_get_size() 🌈

Return type: None 💀

🌈 test_get_is_directory() 🌈

Return type: None 💀

🌈 test_list_url() 🌈

Return type: None 💀

🌈 test_read_from_url() 🌈

Return type: None 💀

🌈 test_open_url() 🌈

Return type: None 💀

---

🌈 toil.test.lib.url_plugin_test 🌈

this one has a fake plugin AND a legacy test class, so pay attention 👀

✨ Attributes ✨

• logger — the only attribute here, value left blank 💅

💅 Classes 💅

• FakeURLPlugin — widget for accessing external storage (URLs) 🔥
• TestURLAccess — legacy common base class for Toil tests 🫶

🎀 Module Contents 🎀

• toil.test.lib.url_plugin_test.logger

🦋 FakeURLPlugin 🦋

class toil.test.lib.url_plugin_test.FakeURLPlugin

Bases: toil.lib.url.URLAccess

Widget for accessing external storage (URLs). a little fake, a lot of vibes 💖

🦋 TestURLAccess 🦋

class toil.test.lib.url_plugin_test.TestURLAccess(methodName='runTest')

Bases: toil.test.ToilTest

Legacy common base class for Toil tests. 🧚

⚠️ bestie, new tests should be written in the pytest style and should not use this class. it's giving old school, so keep it for the vintage stuff only 💀

When running tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files placed 🌸 The directory gets created if it doesn't exist. The path can be relative, and if it is, it's treated as relative to the project root 🗺️

If TOIL_TEST_TEMP isn't defined, temporary files and directories go in the system's default location for that kind of thing. Any temporary files or directories left over from tests get removed automatically during tear down ✨ But if you do set TOIL_TEST_TEMP, the left-over files stick around and won't be removed 🫠

📖 setUp() 📖

Hook method for setting up the test fixture before exercising it. 🫶

Return type: None 💀

📖 tearDown() 📖

Hook method for deconstructing the test fixture after testing it. the clean-up crew 🧹

Return type: None 💀

🌈 test_url_exists() 🌈

Return type: None 💀

🌈 test_get_size() 🌈

Return type: None 💀

🌈 test_get_is_directory() 🌈

Return type: None 💀

🌈 test_list_url() 🌈

Return type: None 💀

🌈 test_read_from_url() 🌈

Return type: None 💀

🌈 test_open_url() 🌈

Return type: None 💀

---

🍓 toil.test.mesos 🍓

the mesos tests are next, and this section is a whole mood 🧚

✨ Submodules ✨

• toil.test.mesos.MesosDataStructuresTest

💅 Classes 💅

• DataStructuresTest — legacy common base class for Toil tests 🫶

🎀 Module Contents 🎀

class toil.test.mesos.MesosDataStructuresTest.DataStructuresTest(methodName='runTest')

Bases: toil.test.ToilTest

Legacy common base class for Toil tests. 🧚

⚠️ same deal as before: new tests should be written in the pytest style and not use this class 💀

When running tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files placed 🌸 The directory gets created if it doesn't exist. The path can be relative, and if it is, it's treated as relative to the project root 🗺️

If TOIL_TEST_TEMP isn't defined, temporary files and directories go in the system's default location for that kind of thing. Any temporary files or directories left over from tests get removed automatically during tear down ✨ But if you do set TOIL_TEST_TEMP, the left-over files stick around and won't be removed 🫠

📖 testJobQueue(testJobs=1000) 📖

The mesos JobQueue sorts MesosShape objects by requirement, and this test checks that the sorting is what we expect: non-preemptible job groups go first, with priority given to large jobs 👑 no thoughts, just clean sorting 🔥

---

🌸 toil.test.mesos.helloWorld 🌸

A simple user script for Toil. the OG hello world, iconic 🧚

✨ Attributes ✨

• childMessage — value shown under Module Contents 💖
• parentMessage — value shown under Module Contents 💖

💅 Functions 💅

• hello_world(job) 🦋
• hello_world_child(job, hw) 🦋
• main() 🦋

🎀 Module Contents 🎀

🦋 Attributes 🦋

toil.test.mesos.helloWorld.childMessage = 'The child job is now running!'

toil.test.mesos.helloWorld.parentMessage = 'The parent job is now running!'

🌈 Functions 🌈

toil.test.mesos.helloWorld.hello_world(job)

toil.test.mesos.helloWorld.hello_world_child(job, hw)

toil.test.mesos.helloWorld.main()

🦋 toil.test.mesos.stress 🦋

okay bestie, this is the stress module and it's giving organized 💅 here's everything in it

Classes

• LongTestJob — a class that represents a unit of work in toil 🎀
• LongTestFollowOn — also a class that represents a unit of work in toil 🌸
• HelloWorldJob — yep, another unit of work in toil, the classic 🧚
• HelloWorldFollowOn — the follow-on version, still a unit of work in toil 💜

Functions

• touchFile(fileStore)
• main(numJobs)

Module Contents

toil.test.mesos.stress.touchFile(fileStore)

toil.test.mesos.stress.touchFile(fileStore)

class toil.test.mesos.stress.LongTestJob(numJobs)

Bases: toil.job.Job

This one's a unit of work in toil, obviously 💅

Attribute:

• numJobs

run(fileStore) is where you override to do the actual work and make successor jobs on the fly ✨

Parameters

• fileStore — this is your tool for making local and globally sharable temporary files, and for sending log messages to the leader process 📬

Returns

The return value can be handed to other jobs through toil.job.Job.rv() 🔥

class toil.test.mesos.stress.LongTestFollowOn

Bases: toil.job.Job

Also a unit of work in toil, no notes 💖

run(fileStore) is where you override to do the work and dynamically create successor jobs 🦋

Parameters

• fileStore — used to create local and globally sharable temporary files and to send log messages to the leader process 📬

Returns

The return value can be passed to other jobs by means of toil.job.Job.rv() 🌈

class toil.test.mesos.stress.HelloWorldJob(i)

Bases: toil.job.Job

Classic unit of work in toil, hi world 🧚

Attribute:

• i

run(fileStore) is where you override to do the work and dynamically create successor jobs 🎀

Parameters

• fileStore — used to create local and globally sharable temporary files and to send log messages to the leader process 📬

Returns

The return value can be passed to other jobs by means of toil.job.Job.rv() 💜

class toil.test.mesos.stress.HelloWorldFollowOn(i)

Bases: toil.job.Job

The follow-on hello world, still a unit of work in toil 🌸

Attribute:

• i

run(fileStore) is where you override to do the work and dynamically create successor jobs ✨

Parameters

• fileStore — used to create local and globally sharable temporary files and to send log messages to the leader process 📬

Returns

The return value can be passed to other jobs by means of toil.job.Job.rv() 👑

toil.test.mesos.stress.main(numJobs)

toil.test.mesos.stress.main(numJobs)

---

🌈 toil.test.options 🌈

Submodules

• toil.test.options.options

toil.test.options.options

Classes

• OptionsTest — a class to test functionality of all Toil options 🧪

Module Contents

class toil.test.options.options.OptionsTest(methodName='runTest')

Bases: toil.test.ToilTest

This class tests the functionality of all Toil options, so it's basically the options main character 👑

test_default_caching_slurm() — makes sure caching gets set to false when running on Slurm 🌸 :return:

test_caching_option_priority() — makes sure the --caching option beats the default_caching() return value 💅 :return:

---

💖 toil.test.provisioners 💖

Submodules

• toil.test.provisioners.aws

toil.test.provisioners.aws

Submodules

• toil.test.provisioners.aws.awsProvisionerTest

toil.test.provisioners.aws.awsProvisionerTest

Attributes

• log

Classes

• TestAWSProvisionerBenchTest — tests for the AWS provisioner that don't actually spin up any instances 🫶
• AbstractAWSAutoscaleTest — a legacy common base class for Toil tests 🦖
• AWSAutoscaleTest — a legacy common base class for Toil tests 🦖
• AWSStaticAutoscaleTest — runs the tests on a statically provisioned cluster with autoscaling enabled 🔥
• AWSManagedAutoscaleTest — runs the tests on a self-scaling Kubernetes cluster ☁️
• AWSAutoscaleTestMultipleNodeTypes — a legacy common base class for Toil tests 🦖
• AWSRestartTest — this test makes sure autoscaling still works on a restarted Toil run 🔁
• PreemptibleDeficitCompensationTest — a legacy common base class for Toil tests 🦖

Functions

• aws_zone() — supplies an appropriate AWS zone for tests that need one 🌍
• aws_region(aws_zone) — supplies an appropriate AWS region for tests that need one 🗺️
• ec2_client(aws_region) — supplies an AWS EC2 client for tests that need one 🖥️

Module Contents

toil.test.provisioners.aws.awsProvisionerTest.log

🌸 toil.test.provisioners.aws.awsProvisionerTest.aws_zone()

okay so this one is the zone fairy 🧚 it hands an appropriate AWS zone to any test that needs one. no setup drama, just vibes ✨

💜 toil.test.provisioners.aws.awsProvisionerTest.aws_region(aws_zone)

this one takes that zone and gives back an appropriate AWS region for any test that needs one 🦋

🧚 toil.test.provisioners.aws.awsProvisionerTest.ec2_client(aws_region)

and this is your AWS EC2 client for the tests that need one. iconic supporting role 💅

🔥 class toil.test.provisioners.aws.awsProvisionerTest.TestAWSProvisionerBenchTest

bestie, these are the tests for the AWS provisioner that don't actually provision instances 🙈 no real cloud spending, just checking the logic slays 💖

• test_AMI_finding()
• test_read_write_global_files(aws_zone) — makes sure _write_file_to_cloud() and _read_file_from_cloud() in the AWS provisioner work the way they're supposed to ✨
• test_get_spot_history(ec2_client) — checks that we can download spot price history from AWS 📈
  • Return type: None
• test_get_aws_zone_from_spot_market(ec2_client) — checks that we can process spot price history to pick a zone 🎯
  • Return type: None

🎀 class toil.test.provisioners.aws.awsProvisionerTest.AbstractAWSAutoscaleTest(methodName)

Bases: toil.test.provisioners.clusterTest.AbstractClusterTest

⚠️ this is a legacy common base class for Toil tests. new tests should be written in the pytest style and not use this class, no cap 💀

okay so when you run tests you can optionally set the TOIL_TEST_TEMP environment variable to a directory where you want the temp test files to go 🌈 the directory gets created if it doesn't exist yet. the path can be relative, in which case it's treated as relative to the project root. if TOIL_TEST_TEMP isn't defined, temp files and directories go in the system's default spot for that stuff, and any leftovers get automatically removed during tear down 🧹. but if you do set it, the leftover files are not removed.

✨ Attributes

• instanceTypes = ['m5a.large']
• clusterName = 'aws-provisioner-test-'
• numWorkers = ['2']
• numSamples = 2
• spotBid = 0.15
• scriptDir = '/tmp/t'
• venvDir = '/tmp/venv'
• dataDir = '/tmp'
• scriptName = 'test_script.py'

💅 Methods

• script() — returns the full path to the user script on the leader 🫶
• data(filename) — returns the full path to the data file with the given name on the leader 📁
• rsyncUtil(src, dest)
  • Parameters:
    • src (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
    • dest (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
  • Return type: None
• getRootVolID()
  • Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• putScript(content) — a helper for _getScript that injects a script file at the configured script path, made from text 📝
  • Parameters:
    • content (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

---

💖 class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTest(name)

Bases: AbstractAWSAutoscaleTest

⚠️ legacy common base class for Toil tests, again! new tests should be made in the pytest style and not use this class 🙈

same TOIL_TEST_TEMP energy as before 🌸: set it optionally to a directory for temp test files, it gets created if missing, relative paths count from the project root, and with it undefined the temp stuff lives in the system default spot and gets cleaned up on tear down. set it, and leftovers stay put.

✨ Attributes

• clusterName = 'provisioner-test-'
• requestedLeaderStorage = 80
• scriptName = 'sort.py'

💅 Methods

• setUp() — set up for the test. it must be overridden to call this method and set self.jobStore ⚠️
• launchCluster()
• getRootVolID() — adds a check that the EBS volume is built with an adequate size. otherwise it's functionally the same as the parent 🧚
  • Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
  • Returns: volumeID
• testAutoScale()
• testSpotAutoScale()
• testSpotAutoScaleBalancingTypes()

---

🌈 class toil.test.provisioners.aws.awsProvisionerTest.AWSStaticAutoscaleTest(name)

Bases: AWSAutoscaleTest

this one runs the tests on a statically provisioned cluster with autoscaling turned on 👑

• requestedNodeStorage = 20
• launchCluster()

---

🌈 class toil.test.provisioners.aws.awsProvisionerTest.AWSManagedAutoscaleTest(name)

Bases: AWSAutoscaleTest

and this one runs the tests on a self-scaling Kubernetes cluster 🧚 it's giving managed, it's giving effortless ✨

• requestedNodeStorage = 20
• launchCluster()

---

🔥 class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTestMultipleNodeTypes(name)

Bases: AbstractAWSAutoscaleTest

⚠️ legacy common base class for Toil tests, yes it's the same warning, the pytest style is the move 💅

and the TOIL_TEST_TEMP rules are the same too 🦋: optional, created if missing, relative to the project root, default system location when undefined with automatic cleanup on tear down, and no cleanup for leftovers when it's set.

✨ Attributes

• clusterName = 'provisioner-test-'

💅 Methods

• setUp() — set up for the test. must be overridden to call this method and set self.jobStore ⚠️
• testAutoScale()

---

🫶 class toil.test.provisioners.aws.awsProvisionerTest.AWSRestartTest(name)

Bases: AbstractAWSAutoscaleTest

this test makes sure autoscaling still works on a restarted Toil run 🔄 main character restart energy 💖

• clusterName = 'restart-test-'
• scriptName = 'restartScript.py'
• setUp() — set up for the test. must be overridden to call this method and set self.jobStore ⚠️
• testAutoScaledCluster()

---

🎀 class toil.test.provisioners.aws.awsProvisionerTest.PreemptibleDeficitCompensationTest(name)

Bases: AbstractAWSAutoscaleTest

⚠️ legacy common base class for Toil tests. use the pytest style for new tests, not this class 🙈

same deal with TOIL_TEST_TEMP 🌸: set it optionally to a temp directory (created if missing, relative paths start from the project root). left unset, temp files live in the system default spot and leftovers get removed on tear down. set it, and leftovers are kept.

✨ Attributes

• clusterName = 'deficit-test-'
• scriptName = 'userScript.py'

💅 Methods

• setUp() — set up for the test. must be overridden to call this method and set self.jobStore ⚠️
• test()

---

💜 toil.test.provisioners.clusterScalerTest

✨ Attributes

• logger
• c4_8xlarge_preemptible
• c4_8xlarge
• r3_8xlarge
• r5_2xlarge
• r5_4xlarge
• t2_micro

🎀 Classes 🎀

Four classes live here, bestie 💅

• BinPackingTest: Legacy common base class for Toil tests 🕰️
• ClusterScalerTest: Legacy common base class for Toil tests 🕰️
• ScalerThreadTest: Legacy common base class for Toil tests 🕰️
• MockBatchSystemAndProvisioner: Mimics a leader, a job batcher, a provisioner and a scalable batch system, all in one 👑 multitasking queen energy

💖 Module Contents 💖

Here's the lineup of module-level names 🧚

• toil.test.provisioners.clusterScalerTest.logger
• toil.test.provisioners.clusterScalerTest.c4_8xlarge_preemptible
• toil.test.provisioners.clusterScalerTest.c4_8xlarge
• toil.test.provisioners.clusterScalerTest.r3_8xlarge
• toil.test.provisioners.clusterScalerTest.r5_2xlarge
• toil.test.provisioners.clusterScalerTest.r5_4xlarge
• toil.test.provisioners.clusterScalerTest.t2_micro

🦋 BinPackingTest

class toil.test.provisioners.clusterScalerTest.BinPackingTest(methodName='runTest')

Bases: toil.test.ToilTest

Legacy common base class for Toil tests 🕰️ New tests should be made in the 'pytest' style and not use this class, bestie 🙅‍♀️

When you run tests, you can optionally set the TOIL_TEST_TEMP environment variable to a directory where you want temp test files to go 🗂️ It'll get created if it doesn't exist yet. The path can be relative, in which case it's relative to the project root. If TOIL_TEST_TEMP isn't set, temp files and directories go to the system's default temp location, and any leftovers from tests get automatically removed during tear down. If you DO set it, leftover files won't be removed.

setUp()

Hook method for setting up the test fixture before exercising it 🧁

testPackingOneShape()

Pack one shape and check that the resulting reservations look sane ✨

testSorting()

Test that sorting is correct: preemptible, then memory, then cores, then disk, then wallTime 📋

testAddingInitialNode()

Pack one shape when no nodes are available and confirm that we fit one node properly 🌱

testLowTargetTime()

Test that a low targetTime (0) parallelizes jobs aggressively (1000 queued jobs require 1000 nodes) 🚀

Ideally, low targetTime means: start quickly and maximize parallelization after the cpu/disk/mem have been packed.

Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can only run one job at a time with its resources.

Each job is parametrized to take 300 seconds, so (the minimum of) 1 of them should fit into each node's 0 second window, so we expect 1000 nodes.

testHighTargetTime()

Test that a high targetTime (3600 seconds) maximizes packing within the targetTime 🧩

Ideally, high targetTime means: maximize packing within the targetTime after the cpu/disk/mem have been packed.

Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can only run one job at a time with its resources.

Each job is parametrized to take 300 seconds, so 12 of them should fit into each node's 3600 second window. 1000/12 = 83.33, so we expect 84 nodes.

testZeroResourceJobs()

Test that jobs requiring zero cpu/disk/mem pack first, regardless of targetTime 🫧

Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can run a seemingly infinite number of jobs with its resources.

Since all jobs should pack cpu/disk/mem-wise on a t2.micro, we expect only one t2.micro to be provisioned. If we raise this, as in testLowTargetTime, it will launch 1000 t2.micros.

testLongRunningJobs()

Test that jobs with long run times (especially service jobs) are aggressively parallelized 🏃‍♀️

This is important, because services are one case where the degree of parallelization really, really matters. If you have multiple services, they may all need to be running simultaneously before any real work can be done.

Despite setting globalTargetTime=3600, this should launch 1000 t2.micros because each job's estimated runtime (30000 seconds) extends well beyond 3600 seconds.

run1000JobsOnMicros(jobCores, jobMem, jobDisk, jobTime, globalTargetTime)

Test packing 1000 jobs on t2.micros. Depending on the targetTime and resources, these should pack differently 📦

testPathologicalCase()

Test a pathological case where only one node can be requested to fit months' worth of jobs 😵‍💫

If the reservation is extended to fit a long job, and the bin-packer naively searches through all the reservation slices to find the first slice that fits, it will happily assign the first slot that fits the job, even if that slot occurs days in the future.

testJobTooLargeForAllNodes()

If a job is too large for all node types, the scaler should print a warning, but definitely not crash 🙏 warning, not a crash

🦋 ClusterScalerTest

class toil.test.provisioners.clusterScalerTest.ClusterScalerTest(methodName='runTest')

Bases: toil.test.ToilTest

Legacy common base class for Toil tests 🕰️ New tests should be made in the 'pytest' style and not use this class, bestie 🙅‍♀️

When you run tests, you can optionally set the TOIL_TEST_TEMP environment variable to a directory where you want temp test files to go 🗂️ It'll get created if it doesn't exist yet. The path can be relative, in which case it's relative to the project root. If TOIL_TEST_TEMP isn't set, temp files and directories go to the system's default temp location, and any leftovers from tests get automatically removed during tear down. If you DO set it, leftover files won't be removed.

setUp()

Hook method for setting up the test fixture before exercising it 🧁

testRounding()

Test to make sure the ClusterScaler's rounding rounds properly 🔢

testMaxNodes()

Set the scaler to be very aggressive, give it a ton of jobs, and make sure it doesn't go over maxNodes 🛑

testMinNodes()

Without any jobs queued, the scaler should still estimate "minNodes" nodes 😌

testPreemptibleDeficitResponse()

When a preemptible deficit was detected by a previous run of the loop, the scaler should add non-preemptible nodes to compensate in proportion to preemptibleCompensation ⚖️

testPreemptibleDeficitIsSet()

Make sure that updateClusterSize sets the preemptible deficit if it can't launch preemptible nodes properly. That way, the deficit can be communicated to the next run of estimateNodeCount 📨

testNoLaunchingIfDeltaAlreadyMet()

Check that the scaler doesn't try to launch "0" more instances if the delta was able to be met by unignoring nodes 🙅‍♀️ no need to launch zero more

testBetaInertia()

No description for this one, it's a mystery 🙈

test_overhead_accounting_large()

If a node has a certain raw memory or disk capacity, that won't all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.) 🧮

Make sure this overhead is accounted for for large nodes 🐘

test_overhead_accounting_small()

If a node has a certain raw memory or disk capacity, that won't all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.) 🧮

Make sure this overhead is accounted for for small nodes 🐭

test_overhead_accounting_observed()

If a node has a certain raw memory or disk capacity, that won't all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.) 🧮

Make sure this overhead is accounted for so that real-world observed failures cannot happen again 🔒

🦋 ScalerThreadTest

class toil.test.provisioners.clusterScalerTest.ScalerThreadTest(methodName='runTest')

Bases: toil.test.ToilTest

Legacy common base class for Toil tests 🕰️ New tests should be made in the 'pytest' style and not use this class, bestie 🙅‍♀️

okay bestie, so when you run tests you can optionally set the TOIL_TEST_TEMP environment variable 💅 to a path where your temporary test files should live. if that directory doesn't exist yet, it gets created for you 🧚. the path can be relative, and if it is, it's measured from the project root 🌸. if TOIL_TEST_TEMP isn't defined, temp files and dirs go to the system's default spot for that stuff, and any leftover temp files or directories from tests get automatically removed during tear down ✨. but if you do set it, leftover files stick around and don't get removed 😭

• testClusterScaling() — tests scaling for a batch of non-preemptible jobs and no preemptible jobs (makes debugging easier 🐛)
• testClusterScalingMultipleNodeTypes() 🦋
• testClusterScalingWithPreemptibleJobs() — tests scaling at the same time for a batch of preemptible and non-preemptible jobs 👑

class toil.test.provisioners.clusterScalerTest.MockBatchSystemAndProvisioner(config, secondsPerJob)

Bases: toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem, toil.provisioners.abstractProvisioner.AbstractProvisioner

this class is giving multitasking 💜 it mimics a leader, job batcher, provisioner and scalable batch system all at once, and it's honestly iconic for it 🔥

Attributes:

• config
• secondsPerJob
• provisioner
• batchSystem
• jobQueue
• updatedJobsQueue
• jobBatchSystemIDToIssuedJob
• totalJobs = 0
• totalWorkerTime = 0.0
• toilMetrics = None
• nodesToWorker
• workers
• maxWorkers
• running = False
• leaderThread
• toilState

Methods:

• start() 🌈
• shutDown() 🥀
• nodeInUse(nodeIP) — ngl this one is so useful. it can tell you if a worker node is running any tasks. if the node doesn't exist, it just returns False 🙈
  • Parameters: nodeIP — the worker node's private IP address
  • Returns: True if the worker node has been issued any tasks, else False
• ignoreNode(nodeAddress) — stop sending jobs to this node. this is used in autoscaling when the autoscaler is ready to terminate a node but jobs are still running. it lets the node get terminated once the current jobs finish 🫶
  • Parameters: nodeAddress — IP address of node to ignore
• unignoreNode(nodeAddress) — stop ignoring this address, presumably after a node with this address got terminated. this way a brand new node can have the same address as a terminated one, which is kinda wild 🌸
• supportedClusterTypes() — get all the cluster types this provisioner implementation supports ✨
• createClusterSettings() — initialize the class for a new cluster that's going to be deployed when you're running outside the cloud 🌍
• readClusterSettings() — initialize the class from an existing cluster. this method assumes the instance we're running on is the leader 👑. implementations must call _setLeaderWorkerAuthentication()
• setAutoscaledNodeTypes(node_types) — sets node types, shapes and spot bids for Toil-managed autoscaling 💅 :param nodeTypes: A list of node types, as parsed with parse_node_types.
  • Parameters: node_types (list[tuple[set[toil.provisioners.abstractProvisioner.Shape], Optional[float]]])
• getProvisionedWorkers(instance_type=None, preemptible=None) — returns a list of Node objects, each one representing a worker node in the cluster 🧚
  • Parameters: preemptible — if True, only return preemptible nodes, else return non-preemptible nodes
  • Returns: list of Node
• terminateNodes(nodes) — terminates the nodes represented by the given Node objects 💀
  • Parameters: nodes — list of Node objects
• remainingBillingInterval(node) ⏳
• addJob(jobShape, preemptible=False) — adds a job to the job queue 🍓
• getNumberOfJobsIssued(preemptible=None) 🔢
• getJobs() 📋
• getNodes(preemptible=False, timeout=600) — returns a dictionary mapping node identifiers of preemptible or non-preemptible nodes to NodeInfo objects, one for each node 🦋
  • Parameters:
    • preemptible (Optional[bool]) — if True (False), only (non-)preemptible nodes will be returned. if None, all nodes will be returned
    • timeout (int)
• addNodes(nodeTypes, numNodes, preemptible) — used to add worker nodes to the cluster 🚀
  • Parameters:
    • numNodes — the number of nodes to add
    • preemptible — whether or not the nodes will be preemptible
    • spotBid — the bid for preemptible nodes if applicable (this can be set in config, also)
    • nodeTypes (set[str])
  • Returns: number of nodes successfully added
  • Return type: int
• getNodeShape(nodeType, preemptible=False) — the shape of a preemptible or non-preemptible node managed by this provisioner. the node shape defines key properties of a machine, like its number of cores or the time between billing intervals 🧠
  • Parameters: instance_type (str) — instance type name to return the shape of
• getWorkersInCluster(nodeShape) 👯
• launchCluster(leaderNodeType, keyName, userTags=None, vpcSubnet=None, leaderStorage=50, nodeStorage=50, botoPath=None, **kwargs) — initializes a cluster and creates a leader node 🎀. implementations must call _setLeaderWorkerAuthentication() with the leader so workers can be launched
  • Parameters:
    • leaderNodeType — the leader instance
    • leaderStorage — the amount of disk to allocate to the leader in gigabytes
    • owner — tag identifying the owner of the instances
• destroyCluster() — terminates all nodes in the specified cluster and cleans up every resource associated with it 🧹 :param clusterName: identifier of the cluster to terminate.
  • Return type: None
• getLeader() 👑
  • Returns: the leader node
• getNumberOfNodes(nodeType=None, preemptible=None) 🔢

---

toil.test.provisioners.clusterTest

Attributes

                                     ┌─────┬───┐
                                     │ log │   │
                                     └─────┴───┘

Classes

                 ┌─────────────────────┬──────────────────────────┐
                 │ AbstractClusterTest │ Legacy common base class │
                 │                     │ for Toil tests.          │
                 ├─────────────────────┼──────────────────────────┤
                 │ CWLOnARMTest        │ Run the CWL 1.2          │
                 │                     │ conformance tests on ARM │
                 │                     │ specifically.            │
                 └─────────────────────┴──────────────────────────┘

Module Contents

• toil.test.provisioners.clusterTest.log 📝

class toil.test.provisioners.clusterTest.AbstractClusterTest(methodName)

Bases: toil.test.ToilTest

this is the legacy common base class for Toil tests 🕰️ new tests should be written in the 'pytest' style and not use this class, bestie 🙅‍♀️

when running tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files placed. the directory gets created if it doesn't exist. the path may be relative, in which case it's assumed to be relative to the project root 🌸. if TOIL_TEST_TEMP isn't defined, temp files and directories get created in the system's default location, and any temporary files or directories left over from tests get automatically removed during tear down. otherwise, left-over files will not be removed 🧼

Parameters:

• methodName (str)

Attributes:

• keyName
• clusterName = 'aws-provisioner-test-Instance of uuid.UUID' 🦄
• leaderNodeType = 't2.medium' 💻
• clusterType = 'mesos' 🌵
• zone
• region 🌍

aws 🦋 okay so we're still in the AWS testing stuff, bestie, and this is where the good bits are 💅

• venvDir = '/tmp/venv' — the venv lives in /tmp/venv 🧚

python()

Grab the full path to the venv Python on the leader 🐍✨

Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

pip()

Same energy, but this time you get the full path to the venv pip on the leader 🌸

Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

destroyCluster()

Destroys the cluster we built, if it exists 💀 And if it's already gone? It still counts as a success, so no drama 🫶

Return type: None

setUp()

Sets up the test. You must override this and call this method, and you also have to set self.jobStore or it's not it 🔥

Return type: None

tearDown()

Hook method for cleaning up the test fixture after testing it. Clean up after yourself, besties 🧹

Return type: None

sshUtil(command)

Runs the command you give it on the cluster. If it fails, you'll get a subprocess.CalledProcessError raised at you 😭

Parameters:

• command (list[str], docs: /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

rsync_util(from_file, to_file)

Transfers a file to or from the cluster 📦 Heads up: the cluster-side path needs a : in front of it, so don't forget it, bestie 🙈

Parameters:

• from_file (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• to_file (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

createClusterUtil(args=None)

Parameters:

• args (Optional[list[str]], docs: /usr/share/doc/python3/html/library/stdtypes.html#list and /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: None

launchCluster()

Launches the cluster 🚀 It's giving "ready to go."

Return type: None

---

class toil.test.provisioners.clusterTest.CWLOnARMTest(methodName)

Bases: AbstractClusterTest 👑

Runs the CWL 1.2 conformance tests on ARM specifically. The ARM check is the whole main character moment 🦋

Parameters:

• methodName (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

Attributes:

• clusterName = 'cwl-test-Instance of uuid.UUID'
• leaderNodeType = 't4g.2xlarge'
• clusterType = 'kubernetes'
• cwl_test_dir = '/tmp/toil/cwlTests'

setUp()

Sets up the test. Like before, you must override it, call this method, and set self.jobStore 🫶

Return type: None

test_cwl_on_arm()

Return type: None

---

toil.test.provisioners.gceProvisionerTest

Attributes

• log ✨

Classes

• AbstractGCEAutoscaleTest — legacy common base class for Toil tests 🧁
• GCEAutoscaleTest — legacy common base class for Toil tests 🧁
• GCEStaticAutoscaleTest — runs the tests on a statically provisioned cluster with autoscaling enabled ⚡
• GCEAutoscaleTestMultipleNodeTypes — legacy common base class for Toil tests 🧁
• GCERestartTest — this test makes sure autoscaling works on a restarted Toil run 🔁

Module Contents

• toil.test.provisioners.gceProvisionerTest.log

class toil.test.provisioners.gceProvisionerTest.AbstractGCEAutoscaleTest(methodName)

Bases: toil.test.ToilTest 💜

Legacy common base class for Toil tests. Okay, real talk: new tests should be made in the pytest style and not use this class, so don't build on it 🙅‍♀️

Now, the TOIL_TEST_TEMP environment variable is optional, and you can set it to the path of a directory where you want your temporary test files to go 🌷 If the directory doesn't exist, it gets created for you. The path can be relative, in which case it's relative to the project root. If TOIL_TEST_TEMP isn't defined, temporary files and directories go in the system's default location, and anything left over from the tests gets removed automatically during tear down 🧹 If you did set it, though, leftover files stay put and won't be removed.

Attributes:

• projectID
• sshUtil(command)
• rsyncUtil(src, dest)
• destroyClusterUtil()
• createClusterUtil(args=None)
• cleanJobStoreUtil()
• keyName
• botoDir
• googleZone = 'us-west1-a'
• leaderInstanceType = 'n1-standard-1'
• instanceTypes = ['n1-standard-2']
• numWorkers = ['2']
• numSamples = 2
• spotBid = 0.15
• setUp() — hook method for setting up the test fixture before exercising it 🌈
• tearDown() — hook method for deconstructing the test fixture after testing it 🧼
• launchCluster() 🚀

class toil.test.provisioners.gceProvisionerTest.GCEAutoscaleTest(name)

Bases: AbstractGCEAutoscaleTest 💖

Legacy common base class for Toil tests. Same rule as before: new tests should be made in the pytest style and not use this class 🙅‍♀️

Same TOIL_TEST_TEMP tea as above: it's optional, it can point at a directory you want for temp test files, and the directory gets created if it's missing. Relative paths count as relative to the project root. If TOIL_TEST_TEMP isn't defined, temp files and directories go to the system's default spot and get removed automatically during tear down. If you set it, left-over files will not be removed.

Attributes:

• clusterName = 'provisioner-test-'
• requestedLeaderStorage = 80
• setUp() — hook method for setting up the test fixture before exercising it 🌸
• launchCluster() 🚀
• testAutoScale() ⭐
• testSpotAutoScale() ⭐

class toil.test.provisioners.gceProvisionerTest.GCEStaticAutoscaleTest(name)

Bases: GCEAutoscaleTest 👑

Runs the tests on a statically provisioned cluster with autoscaling enabled. It's giving static but make it autoscale 😤

Attributes:

• requestedNodeStorage = 20
• launchCluster() 🚀

class toil.test.provisioners.gceProvisionerTest.GCEAutoscaleTestMultipleNodeTypes(name)

Bases: AbstractGCEAutoscaleTest 🧁

Legacy common base class for Toil tests. Reminder, bestie: new tests should be made in the pytest style and not use this class 🙅‍♀️

The TOIL_TEST_TEMP variable is optional. Point it at a directory for temp test files and it gets created if needed. Relative paths are relative to the project root. If you leave TOIL_TEST_TEMP undefined, temp files and directories go to the system's default location and get removed automatically during tear down. If you set it, left-over files won't be removed 🫧

Attributes:

• clusterName = 'provisioner-test-'
• setUp() — hook method for setting up the test fixture before exercising it 🌈
• testAutoScale() ⭐

class toil.test.provisioners.gceProvisionerTest.GCERestartTest(name)

Bases: AbstractGCEAutoscaleTest 🔁

This test makes sure autoscaling works on a restarted Toil run. Restarts are a whole mood, and this one checks that it still scales 🙌

Attributes:

• clusterName = 'restart-test-'
• setUp() — hook method for setting up the test fixture before exercising it 🌸
• testAutoScaledCluster() ⭐

---

toil.test.provisioners.provisionerTest

Attributes

• log 🦋

Okay so the ProvisionerTest class is a legacy common base class for Toil tests 🧚 ngl, it's giving old school. New tests should be made in the pytest style and not use this class 🙈

✨ Module Contents ✨

• toil.test.provisioners.provisionerTest.log 🦋

💖 ProvisionerTest

class toil.test.provisioners.provisionerTest.ProvisionerTest(methodName='runTest')

Bases: toil.test.ToilTest 🌈

Legacy common base class for Toil tests. Bestie, reminder: new tests go in the pytest style, not this class 💅

When you run tests you can optionally set the TOIL_TEST_TEMP environment variable to a directory path where you want temporary test files to live 🫶 The directory gets created if it doesn't exist yet. The path can be relative, and if it is, it's assumed to be relative to the project root. If TOIL_TEST_TEMP isn't defined, temporary files and directories get created in the system's default location for that stuff, and any leftovers from tests get removed automatically during tear down. Otherwise, left-over files will not be removed 😭

• test_node_type_parsing() 🍓
  • Return type: None

---

🧚 toil.test.provisioners.restartScript

🌸 Attributes

• parser (no description attached) 🦋

🌸 Functions

• f0(job) (no description attached) 🦋

✨ Module Contents ✨

• toil.test.provisioners.restartScript.f0(job)
• toil.test.provisioners.restartScript.parser

---

🌈 toil.test.server

🎀 Submodules

• toil.test.server.serverTest

🌈 toil.test.server.serverTest

💜 Attributes

• logger 🦋

💜 Classes

• ToilServerUtilsTest — tests for the utility functions used by the Toil server 🔥
• hidden — no description attached, but it holds some nested test classes, see below 🙈
• FileStateStoreTest — test file-based state storage 💖
• FileStateStoreURLTest — test file-based state storage using URLs instead of local paths 🌸
• BucketUsingTest — base class for tests that need a bucket 🪣
• AWSStateStoreTest — test AWS-based state storage ☁️
• TrueDict — dict that is truthy even when empty 👑
• AbstractToilWESServerTest — class for server tests that provides a self.app in testing mode ✨
• ToilWESServerBenchTest — tests for Toil's Workflow Execution Service API that don't run workflows ⭐
• ToilWESServerWorkflowTest — tests of the WES server running workflows 🏃‍♀️
• ToilWESServerCeleryWorkflowTest — end-to-end workflow-running tests against Celery 🥬
• ToilWESServerCeleryS3StateWorkflowTest — test the server with Celery and state stored in S3 🧁

💜 Module Contents

• toil.test.server.serverTest.logger 🦋

💖 ToilServerUtilsTest

class toil.test.server.serverTest.ToilServerUtilsTest(methodName='runTest')

Bases: toil.test.ToilTest 🌈

Tests for the utility functions used by the Toil server 🔥

• test_workflow_canceling_recovery() 🍓
  • Make sure that a workflow in CANCELING state will be recovered to a terminal state eventually, even if the workflow runner Celery task goes away without flipping the state. No cap, this one is important 😭

💖 hidden

This one's a container, bestie. It holds the classes below 🧚

🦋 AbstractStateStoreTest

class AbstractStateStoreTest(methodName='runTest')

Bases: toil.test.ToilTest 🌈

Basic tests for state stores 💅

• get_state_store() (abstract method) 🌸
  • Make a state store to test, on a single fixed URL.
  • Return type: AbstractStateStore (toil.server.utils.AbstractStateStore)
• test_state_store() 🌸
  • Make sure that the state store under test can store and load keys.
  • Return type: None

💖 FileStateStoreTest

class toil.test.server.serverTest.FileStateStoreTest(methodName='runTest')

Bases: hidden 🌈

Test file-based state storage 💖

• setUp() 🫶
  • Hook method for setting up the test fixture before exercising it.
  • Return type: None
• get_state_store() 🫶
  • Make a state store to test, on a single fixed local path.
  • Return type: AbstractStateStore (toil.server.utils.AbstractStateStore)

💖 FileStateStoreURLTest

class toil.test.server.serverTest.FileStateStoreURLTest(methodName='runTest')

Bases: hidden 🌈

Test file-based state storage using URLs instead of local paths 🌸

• setUp() 🫶
  • Hook method for setting up the test fixture before exercising it.
  • Return type: None
• get_state_store() 🫶
  • Make a state store to test, on a single fixed URL.
  • Return type: AbstractStateStore (toil.server.utils.AbstractStateStore)

💖 BucketUsingTest

class toil.test.server.serverTest.BucketUsingTest(methodName='runTest')

Bases: toil.test.ToilTest 🌈

Base class for tests that need a bucket 🪣

• region (str | None) 🦋
• s3_resource (mypy_boto3_s3.S3ServiceResource | None) 🦋
• bucket (mypy_boto3_s3.service_resource.Bucket | None) 🦋
• bucket_name (str | None) 🦋
• setUpClass() (classmethod) 🔥
  • Set up the class with a single pre-existing AWS bucket for all tests.
  • Return type: None
• tearDownClass() (classmethod) 🔥
  • Hook method for deconstructing the class fixture after running all tests in the class.
  • Return type: None

🎀 class toil.test.server.serverTest.AWSStateStoreTest(methodName='runTest')

Bases: hidden, BucketUsingTest 💅

Okay so this is the test class for AWS-based state storage ☁️✨. It's literally checking that the state store works on AWS, no cap.

The class attribute is bucket_path = 'prefix/of/keys' 🌸. That's the little prefix all the keys hang out under.

get_state_store()

Makes a state store to test, on a single fixed URL 🔗.

Return type: AbstractStateStore (#toil.server.utils.AbstractStateStore) 💖

test_state_store_paths()

Makes sure the S3 state store puts things in the right places 📦. We don't really care about the exact internal structure, but we do care that everything is actually under the path we're supposed to use 🫶.

Return type: None

---

🦋 class toil.test.server.serverTest.TrueDict

Bases: dict (/usr/share/doc/python3/html/library/stdtypes.html#dict)

This is a dict that is truthy even when empty 👑. It's a workaround to set httpx post request encoding, as recommended in https://github.com/encode/httpx/discussions/2399#discussioncomment-3814186 🧚.

__bool__()

Always says the object is truthy. Iconic behaviour 💅.

Return type: bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

🌈 class toil.test.server.serverTest.AbstractToilWESServerTest(*args, **kwargs)

Bases: toil.test.ToilTest (#toil.test.ToilTest)

This is the base class for server tests, and it gives you a self.app in testing mode 🎀.

setUp()

Hook method for setting up the test fixture before exercising it ✨.

Return type: None

tearDown()

Hook method for deconstructing the test fixture after testing it 🧹.

Return type: None

---

⭐ class toil.test.server.serverTest.ToilWESServerBenchTest(*args, **kwargs)

Bases: AbstractToilWESServerTest

These are tests for Toil's Workflow Execution Service API that don't run workflows 🍃. Low-key a chill vibe.

test_home()

Tests the homepage endpoint 🏠.

Return type: None

test_health()

Tests the health check endpoint 🫀.

Return type: None

test_get_service_info()

Tests the GET /service-info endpoint 💁‍♀️.

Return type: None

---

🍓 class toil.test.server.serverTest.ToilWESServerWorkflowTest(*args, **kwargs)

Bases: AbstractToilWESServerTest

These are the tests of the WES server actually running workflows 🔥.

run_zip_workflow(zip_path, include_message=True, include_params=True)

We have several zip file tests, so this helper submits a zip file and makes sure it ran OK 📦✅.

If include_message is set to False, don't send a "message" argument in workflow_params. If include_params is also set to False, don't send workflow_params at all 🙈.

Parameters:

• zip_path (str (/usr/share/doc/python3/html/library/stdtypes.html#str))
• include_message (bool (/usr/share/doc/python3/html/library/functions.html#bool))
• include_params (bool (/usr/share/doc/python3/html/library/functions.html#bool))

Return type: None

test_run_workflow_relative_url_no_attachments_fails()

Runs the example CWL workflow from a relative workflow URL, but with no attachments. This one is supposed to fail, bestie 💀.

Return type: None

test_run_workflow_relative_url()

Runs the example CWL workflow from a relative workflow URL 🔗.

Return type: None

test_run_workflow_https_url()

Runs the example CWL workflow from the Internet 🌐.

Return type: None

test_run_workflow_single_file_zip()

Runs the example CWL workflow from a single-file ZIP 🤏.

Return type: None

test_run_workflow_multi_file_zip()

Runs the example CWL workflow from a multi-file ZIP 🗂️.

Return type: None

test_run_workflow_manifest_zip()

Runs the example CWL workflow from a ZIP with a manifest 📜.

Return type: None

test_run_workflow_inputs_zip()

Runs the example CWL workflow from a ZIP without a manifest, but with inputs 🎁.

Return type: None

test_run_workflow_manifest_and_inputs_zip()

Runs the example CWL workflow from a ZIP with a manifest and inputs 👑.

Return type: None

test_run_workflow_no_params_zip()

Runs the example CWL workflow from a ZIP without workflow_params 🫥.

Return type: None

test_run_and_cancel_workflows()

Runs two workflows, cancels one of them, and makes sure they all exist 🚫✨.

Return type: None

---

🧚 class toil.test.server.serverTest.ToilWESServerCeleryWorkflowTest(*args, **kwargs)

Bases: ToilWESServerWorkflowTest

These are end-to-end workflow-running tests against Celery 🥕⚡.

---

💜 class toil.test.server.serverTest.ToilWESServerCeleryS3StateWorkflowTest(*args, **kwargs)

Bases: ToilWESServerWorkflowTest, BucketUsingTest

This tests the server with Celery and its state stored in S3 ☁️💜.

setUp()

Hook method for setting up the test fixture before exercising it 🌷.

Return type: None

---

🌸 toil.test.sort

🧁 Submodules

🫧 toil.test.sort.restart_sort

A demonstration of toil! It sorts the lines of a file into ascending order by doing a parallel merge sort 🔀.

bestie, this one is intentionally buggy 💀. It doesn't include restart() on purpose, for testing purposes.

🎀 Attributes

• defaultLines
• defaultLineLen
• sortMemory

🛠️ Functions

• setup(job, inputFile, N, downCheckpoints, options) — Sets up the sort 🏗️.
• down(job, inputFileStoreID, N, path, downCheckpoints, ...) — Input is a file, a subdivision size N, and a path in the hierarchy of jobs 🪜.
• up(job, inputFileID1, inputFileID2, path, options[, ...]) — Merges the two files and places them in the output 🫶.
• sort(file) — Sorts the given file 🔤.
• merge(fileHandle1, fileHandle2, outputFileHandle) — Merges together two files while keeping sorted order 📚.
• copySubRangeOfFile(inputFile, fileStart, fileEnd) — Copies the range (in bytes) between fileStart and fileEnd to the given (the description got cut off in the source, bestie 🙈).
• getMidPoint(file, fileStart, fileEnd) — Finds the point in the file to split ✂️.
• makeFileToSort(fileName[, lines, lineLen]) — No description given 🤷‍♀️.
• main([options]) — No description given 🤷‍♀️.

📦 Module Contents

toil.test.sort.restart_sort.defaultLines = 1000
toil.test.sort.restart_sort.defaultLineLen = 50
toil.test.sort.restart_sort.sortMemory = '600M'

toil.test.sort.restart_sort.setup(job, inputFile, N, downCheckpoints, options)

Sets up the sort. Returns the FileID of the sorted file 🎉.

• toil.test.sort.restart_sort.down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory) — okay bestie, this one takes a file, a subdivision size N, and a path through the hierarchy of jobs 🌳. If the range is bigger than the threshold N, it gets split recursively and a follow-on job is created to merge everything back together. Otherwise it just sorts the file and drops it into the output 💅
• toil.test.sort.restart_sort.up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory) — merges the two files and places the result in the output 🫶
• toil.test.sort.restart_sort.sort(file) — sorts the given file 🧹
• toil.test.sort.restart_sort.merge(fileHandle1, fileHandle2, outputFileHandle) — squishes two files together while keeping them in sorted order 🎀

⚠️ Bestie, every handle here has to be a text-mode stream. No exceptions. 🙅

• toil.test.sort.restart_sort.copySubRangeOfFile(inputFile, fileStart, fileEnd) — copies the bytes between fileStart and fileEnd into the output file handle you give it 📋
• toil.test.sort.restart_sort.getMidPoint(file, fileStart, fileEnd) — finds the spot in the file where it should get split ✂️ It hands back an int called i where fileStart <= i < fileEnd ✨
• toil.test.sort.restart_sort.makeFileToSort(fileName, lines=defaultLines, lineLen=defaultLineLen) — no description given for this one 🙈
• toil.test.sort.restart_sort.main(options=None) — no description given for this one 🙈

---

toil.test.sort.sort

A demonstration of toil 🎀 It sorts the lines of a file into ascending order by doing a parallel merge sort 💜

Attributes

• defaultLines — no notes on this one 🤷
• defaultLineLen — no notes on this one 🤷
• sortMemory — no notes on this one 🤷

Functions

• setup(job, inputFile, N, downCheckpoints, options) — sets up the sort 🌸
• down(job, inputFileStoreID, N, path, downCheckpoints, ...) — takes a file, a subdivision size N, and a path in the hierarchy of jobs 🌳 If the range is larger than N, it divides recursively and creates a follow-on job to merge the results back. Otherwise it sorts the file and places it in the output 💅
• up(job, inputFileID1, inputFileID2, path, options[, ...]) — merges the two files and places them in the output 🫶
• sort(file) — sorts the given file 🧹
• merge(fileHandle1, fileHandle2, outputFileHandle) — merges two files together while keeping sorted order 🎀
• copySubRangeOfFile(inputFile, fileStart, fileEnd) — copies the bytes between fileStart and fileEnd to the given output file handle 📋
• getMidPoint(file, fileStart, fileEnd) — finds the point in the file to split ✂️
• makeFileToSort(fileName[, lines, lineLen]) — no description given 🙈
• main([options]) — no description given 🙈

Module Contents

• toil.test.sort.sort.defaultLines = 1000 🔢
• toil.test.sort.sort.defaultLineLen = 50 🔢
• toil.test.sort.sort.sortMemory = '600M' 💾
• toil.test.sort.sort.setup(job, inputFile, N, downCheckpoints, options) — gets the sort set up and hands back the FileID of the sorted file 🎉
• toil.test.sort.sort.down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory) — takes a file, a subdivision size N, and a path in the hierarchy of jobs 🌳 If the range is larger than the threshold N, it gets divided recursively and a follow-on job is created to merge the results back. Otherwise the file is sorted and placed in the output 💅
• toil.test.sort.sort.up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory) — merges the two files and places them in the output 🫶
• toil.test.sort.sort.sort(file) — sorts the given file 🧹
• toil.test.sort.sort.merge(fileHandle1, fileHandle2, outputFileHandle) — merges two files together while keeping sorted order 🎀

⚠️ All handles must be text-mode streams. 🙅

• toil.test.sort.sort.copySubRangeOfFile(inputFile, fileStart, fileEnd) — copies the bytes between fileStart and fileEnd into the given output file handle 📋
• toil.test.sort.sort.getMidPoint(file, fileStart, fileEnd) — finds the point in the file to split ✂️ It returns an int i where fileStart <= i < fileEnd ✨
• toil.test.sort.sort.makeFileToSort(fileName, lines=defaultLines, lineLen=defaultLineLen) 📝

Parameters

• fileName (toil.lib.misc.StrPath)
• lines (int)
• lineLen (int)

Return type: None

• toil.test.sort.sort.main(options=None) — no description given 🙈

---

toil.test.sortTest

Attributes

• logger — no notes on this one 🤷
• defaultLineLen — no notes on this one 🤷
• defaultLines — no notes on this one 🤷
• defaultN — no notes on this one 🤷

Classes

• SortTest — tests Toil by sorting a file in parallel across various combinations of job stores and batch systems 🧪

Functions

• runMain(options) — makes sure the output file gets deleted every single time main is run 🧹

Module Contents

• toil.test.sort.sortTest.logger 📣
• toil.test.sort.sortTest.defaultLineLen 🔢
• toil.test.sort.sortTest.defaultLines 🔢
• toil.test.sort.sortTest.defaultN 🔢
• toil.test.sort.sortTest.runMain(options) — makes sure the output file is deleted every time main is run 🧹
• class toil.test.sort.sortTest.SortTest(methodName='runTest') 💅

Bases: toil.test.ToilTest, toil.batchSystems.mesos.test.MesosTestSupport

Tests Toil by sorting a file in parallel on various combinations of job stores and batch systems 🧪

• setUp() — hook method for setting up the test fixture before exercising it 🛠️
• tearDown() — hook method for deconstructing the test fixture after testing it 🧼
• testAwsSingle() 🦋
• testAwsMesos() 🦋
• testFileMesos() 🦋
• testGoogleSingle() 🦋
• testGoogleMesos() 🦋
• testFileSingle() 🦋
• testFileSingleNonCaching() 🦋
• testFileSingleCheckpoints() 🦋
• testFileSingle10000() 🦋
• testFileGridEngine() 🦋
• testFileTorqueEngine() 🦋
• testNo = 5 🔢
• testSort() 🦋
• testMerge() 🦋
• testCopySubRangeOfFile() 🦋
• testGetMidPoint() 🦋

---

toil.test.src

Submodules

toil.test.src.autoDeploymentTest

Attributes

• logger — no notes on this one 🤷

💖 Classes 💖

• AutoDeploymentTest — bestie, this one is the queen of auto-deployment tests 👑 It runs a bunch of different scenarios using the appliance, i.e. a docker container 🐳

✨ Module Contents ✨

• toil.test.src.autoDeploymentTest.logger — the logger for this module 📝

class toil.test.src.autoDeploymentTest.AutoDeploymentTest(methodName='runTest')

Bases: toil.test.ApplianceTestSupport

okay so this class tests all the auto-deployment scenarios 💅 Using the appliance (a docker container) lets worker processes run on the same node as the leader process, while keeping their file systems separate from each other and from the leader 🫶 Separate file systems are crucial here, because that's how we prove auto-deployment is actually doing its job 🔥

• setUp() — the hook method that sets up the test fixture before we exercise it 🧚
• sitePackages — set to 'venv/lib/python3.13/site-packages' 📦

sitePackages = 'venv/lib/python3.13/site-packages'

• testRestart() — checks whether auto-deployment still works on restart 🔁
• testSplitRootPackages() — checks whether auto-deployment works with a virtualenv where jobs are defined in completely separate branches of the package hierarchy 🌳 Back in the day, auto-deployment did deploy the entire virtualenv, but jobs could only be defined in one branch of the package hierarchy 😭 A branch is the maximum set of fully qualified package paths that share the same first component. IOW, a.b and a.c are in the same branch, while a.b and d.c are not.
• testUserTypesInJobFunctionArgs() — tests encapsulated, function-wrapping jobs where the function arguments reference user-defined types 💜 This was mainly written to cover https://github.com/BD2KGenomics/toil/issues/1259, but it also revealed https://github.com/BD2KGenomics/toil/issues/1278 🙈
• testDeferralWithConcurrentEncapsulation() — ensures that the following DAG succeeds 🌈

                 ┌───────────┐
                 │ Root (W1) │
                 └───────────┘
                       │
            ┌──────────┴─────────┐
            ▼                    ▼
   ┌────────────────┐ ┌────────────────────┐
   │ Deferring (W2) │ │ Encapsulating (W3) │═══════════════╗
   └────────────────┘ └────────────────────┘               ║
                                 │                         ║
                                 ▼                         ▼
                       ┌───────────────────┐      ┌────────────────┐
                       │ Encapsulated (W3) │      │ Follow-on (W6) │
                       └───────────────────┘      └────────────────┘
                                 │                         │
                         ┌───────┴────────┐                │
                         ▼                ▼                ▼
                 ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
                 │ Dummy 1 (W4) │ │ Dummy 2 (W5) │ │  Last (W6)   │
                 └──────────────┘ └──────────────┘ └──────────────┘

The Wn numbers show which worker process a particular job runs in 🦋 Deferring adds a deferred function and then runs for a long time. That deferred function stays in the cache state for the whole duration of Deferred. Follow-on is the generic Job instance that gets added when you encapsulate a job. It runs on the same worker node, but in a separate worker process, as the first job in that worker. Because…

1. it is the first job in its worker process (the user script has not been put on the sys.path by a previous job in that worker) and
2. it shares the cache state with the Deferring job and
3. it is an instance of Job (and so does not introduce the user script to sys.path itself),

…it might cause problems with deserializing a deferred function defined in the user script 😬

Encapsulated has two children so that Follow-on runs in a separate worker 🧚

• testDeferralWithFailureAndEncapsulation() — ensures that the following DAG succeeds 💀

                 ┌───────────┐
                 │ Root (W1) │
                 └───────────┘
                       │
            ┌──────────┴─────────┐
            ▼                    ▼
   ┌────────────────┐ ┌────────────────────┐
   │ Deferring (W2) │ │ Encapsulating (W3) │═══════════════════════╗
   └────────────────┘ └────────────────────┘                       ║
                                 │                                 ║
                                 ▼                                 ▼
                       ┌───────────────────┐              ┌────────────────┐
                       │ Encapsulated (W3) │════════════╗ │ Follow-on (W7) │
                       └───────────────────┘            ║ └────────────────┘
                                 │                      ║
                          ┌──────┴──────┐               ║
                          ▼             ▼               ▼
                   ┌────────────┐┌────────────┐ ┌──────────────┐
                   │ Dummy (W4) ││ Dummy (W5) │ │ Trigger (W6) │
                   └────────────┘└────────────┘ └──────────────┘

Trigger causes Deferring to crash 💥 Follow-on runs next, detects Deferring's left-overs and runs the deferred function. Follow-on is an instance of Job and the first job in its worker process. This test makes sure that, even in these circumstances, the user script is loaded before the deferred functions defined in it get run 🔥

Encapsulated has two children so that Follow-on runs in a new worker. That's the only way to guarantee the user script hasn't been loaded yet, which would make the test pass by accident 🙈 We want to test that auto-deploying and loading the user script are done properly before deferred functions run, and before any jobs have been executed by that worker 👑

✨ toil.test.src.busTest ✨

🦋 Attributes 🦋

• logger

🌸 Classes 🌸

• TestMessageBus

💅 Functions 💅

• failing_job_fn(job) — this function is guaranteed to fail 💀

✨ Module Contents ✨

• toil.test.src.busTest.logger
• class toil.test.src.busTest.TestMessageBus
  • test_enum_ints_in_file(tmp_path) — makes sure writing bus messages to files works with enums 🍓

Parameters: tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

Return type: None

• test_cross_thread_messaging() — makes sure the message bus works across threads 🧵

Return type: None

• test_restart_without_bus_path(tmp_path) — tests the ability to restart a workflow when the message bus path used by the previous attempt is gone 🫣

Parameters: tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

Return type: None

• toil.test.src.busTest.failing_job_fn(job) — this is the same guaranteed-to-fail function from above, now in its module-contents home 🌈

Parameters: job (toil.job.Job)

✨ toil.test.src.checkpointTest ✨

Return type: NoReturn 💅

💖 Classes

• CheckpointTest — Legacy common base class for Toil tests. 🎀
• CheckRetryCount — Fail N times, succeed on the next try. 🔥
• AlwaysFail — Class represents a unit of work in toil. ⭐
• CheckpointFailsFirstTime — Class represents a unit of work in toil. 🦋
• FailOnce — Fail the first time the workflow is run, but succeed thereafter. 🌸

🧚 Module Contents

toil.test.src.checkpointTest.CheckpointTest(methodName='runTest')

Bases: toil.test.ToilTest 🫶

Legacy common base class for Toil tests. Okay bestie, heads up: new tests should be made in the 'pytest' style and not use this class. 🙈

When you run tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files to go. The directory gets created if it doesn't exist yet. The path can be relative, in which case it's assumed to be relative to the project root. If TOIL_TEST_TEMP isn't defined, temporary files and directories get created in the system's default location for that stuff, and any leftover temp files or directories from tests get removed automatically during tear down. Otherwise, leftover files will not be removed. ✨

• testCheckpointNotRetried() — A checkpoint job should not be retried if the workflow has a retryCount of 0. 🙅‍♀️
• testCheckpointRetriedOnce() — A checkpoint job should be retried exactly once if the workflow has a retryCount of 1. 🔁
• testCheckpointedRestartSucceeds() — A checkpointed job should succeed on restart of a failed run if its child job succeeds. 👑

class toil.test.src.checkpointTest.CheckRetryCount(numFailuresBeforeSuccess)

Bases: toil.job.Job 💜

Fail N times, succeed on the next try. It's giving stubborn but iconic 💅

Attribute: numFailuresBeforeSuccess 🍓

• getNumRetries(fileStore) — Marks a retry in the fileStore, then returns the number of retries so far. 📈
• run(fileStore) — Override this function to perform work and dynamically create successor jobs. 🛠️
  • Parameters:
    • fileStore — Used to create local and globally sharable temporary files and to send log messages to the leader process. 📬
  • Returns: The return value of the function can be passed to other jobs by means of toil.job.Job.rv(). 🎁

class toil.test.src.checkpointTest.AlwaysFail(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)

Bases: toil.job.Job 🌈

Class represents a unit of work in toil. Main character energy, no notes 👑

class toil.test.src.checkpointTest.AlwaysFail(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)

Parameters:

• memory (Optional[ParseableIndivisibleResource])
• cores (Optional[ParseableDivisibleResource])
• disk (Optional[ParseableIndivisibleResource])
• accelerators (Optional[ParseableAcceleratorRequirement])
• preemptible (Optional[ParseableFlag])
• preemptable (Optional[ParseableFlag])
• unitName (Optional[str])
• checkpoint (Optional[bool])
• displayName (Optional[str])
• descriptionClass (Optional[type])
• local (Optional[bool])
• files (Optional[set[toil.fileStores.FileID]])
• run(fileStore) — Override this function to perform work and dynamically create successor jobs. 🛠️
  • Parameters:
    • fileStore — Used to create local and globally sharable temporary files and to send log messages to the leader process. 📬
  • Returns: The return value of the function can be passed to other jobs by means of toil.job.Job.rv(). 🎁

class toil.test.src.checkpointTest.CheckpointFailsFirstTime

Bases: toil.job.Job 🦋

Class represents a unit of work in toil. ✨

• run(fileStore) — Override this function to perform work and dynamically create successor jobs. 🛠️
  • Parameters:
    • fileStore — Used to create local and globally sharable temporary files and to send log messages to the leader process. 📬
  • Returns: The return value of the function can be passed to other jobs by means of toil.job.Job.rv(). 🎁

class toil.test.src.checkpointTest.FailOnce(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)

Bases: toil.job.Job 🍓

Fail the first time the workflow is run, but succeed thereafter. First run = oops, every run after = slay 💀✨

class toil.test.src.checkpointTest.FailOnce(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)

Parameters:

• memory (Optional[ParseableIndivisibleResource])
• cores (Optional[ParseableDivisibleResource])
• disk (Optional[ParseableIndivisibleResource])
• accelerators (Optional[ParseableAcceleratorRequirement])
• preemptible (Optional[ParseableFlag])
• preemptable (Optional[ParseableFlag])
• unitName (Optional[str])
• checkpoint (Optional[bool])
• displayName (Optional[str])
• descriptionClass (Optional[type])
• local (Optional[bool])
• files (Optional[set[toil.fileStores.FileID]])
• run(fileStore) — Override this function to perform work and dynamically create successor jobs. 🛠️
  • Parameters:
    • fileStore — Used to create local and globally sharable temporary files and to send log messages to the leader process. 📬
  • Returns: The return value of the function can be passed to other jobs by means of toil.job.Job.rv(). 🎁

---

✨ toil.test.src.deferredFunctionTest ✨

🌸 Attributes

• logger 📝

💅 Classes

• TestDeferredFunction — Test the deferred function system. 🧪

🦋 Functions

• options(tmp_path) 🔧

🧚 Module Contents

toil.test.src.deferredFunctionTest.logger

Attribute, no description attached. Just vibes 🫧

toil.test.src.deferredFunctionTest.options(tmp_path)

Parameters:

• tmp_path (pathlib.Path)

Return type: collections.abc.Generator[argparse.Namespace] 🌈

class toil.test.src.deferredFunctionTest.TestDeferredFunction

Test the deferred function system. The whole deferred function system, tested. Iconic behaviour 👑🔥

testDeferredFunctionRunsWithMethod(tmp_path, options)

Literally just check the docstring for _testDeferredFunctionRuns 💅 This version tests with a Method 🦋

Parameters

• tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• options (argparse.Namespace, /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type: None

testDeferredFunctionRunsWithClassMethod(tmp_path, options)

Same deal, go read _testDeferredFunctionRuns 🌸 This one tests with a Class Method ✨

Parameters

• tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• options (argparse.Namespace, /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type: None

testDeferredFunctionRunsWithLambda(tmp_path, options)

Refer to the docstring in _testDeferredFunctionRuns again, bestie 🫶 This version tests with a Lambda 🍓

Parameters

• tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• options (argparse.Namespace, /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type: None

testDeferredFunctionRunsWithFailures(options, tmp_path)

Okay this one is a whole saga 😭 It creates 2 non local files to use as flags 🚩 Then it makes a job that registers a function that deletes one non local file. If that file exists, the job SIGKILLs itself 💀 If it doesn't exist, the job registers a second deferred function that deletes the second non local file, and then it exits normally.

Initially the first file exists, so the job should SIGKILL itself and neither deferred function will run (in fact, the second should not even be registered) 🙈 On the restart, the first deferred function should run and the first file should not exist, but the second one should. We assert the presence of the second, then register the second deferred function and exit normally. At the end of the test, neither file should exist ✨

Incidentally, this also tests for multiple registered deferred functions, and the case where a deferred function fails (since the first file doesn't exist on the retry) 👑

Parameters

• options (argparse.Namespace, /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)
• tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

Return type: None

testNewJobsCanHandleOtherJobDeaths(options, tmp_path)

Main character energy 🦋 We start by making 2 non-local files, then 2 jobs. The first job registers a deferred job to delete the second non-local file, deletes the first non-local file, and then kills itself 💀

The second job waits for the first file to be deleted, then sleeps for a few seconds, then spawns a child. The child of the second does nothing. However, starting it should handle the untimely demise of the first job and run the registered deferred function that deletes the first file 😭 We assert the absence of the two files at the end of the run 🫶

Parameters

• options (argparse.Namespace, /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)
• tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

Return type: None

testBatchSystemCleanupCanHandleWorkerDeaths(options, tmp_path)

Okay so we make some non-local files 🧚 Then a job registers a deferred function to delete the file, and then kills its worker 💀

We assert that the file is missing after the pipeline fails, because we're using a single-machine batch system, and the leader's batch system cleanup will find and run the deferred function. Iykyk 🔥

Parameters

• options (argparse.Namespace, /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)
• tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

Return type: None

---

🧪 toil.test.src.dockerCheckTest

🎀 Classes 🎀

• DockerCheckTest — Tests checking whether a docker image exists or not. 🐳

🌈 Module Contents 🌈

class toil.test.src.dockerCheckTest.DockerCheckTest(methodName='runTest')

Bases: toil.test.ToilTest

Tests checking whether a docker image exists or not. 💖

• testOfficialUbuntuRepo() — Image exists, so this should pass ✅
• testBroadDockerRepo() — Image exists, so this should pass ✅
• testBroadDockerRepoBadTag() — Bad tag, so this should raise 💀
• testNonexistentRepo() — Bad image, so this should raise 💀
• testToilQuayRepo() — Image exists, should pass ✅
• testBadQuayRepoNTag() — Bad repo and tag, this should raise 💀
• testBadQuayRepo() — Bad repo, this should raise 💀
• testBadQuayTag() — Bad tag, this should raise 💀
• testGoogleRepo() — Image exists, should pass ✅
• testBadGoogleRepo() — Bad repo and tag, this should raise 💀
• testApplianceParser() — Tests that a specified appliance is parsed correctly 🧁

---

🧪 toil.test.src.environmentTest

💜 Attributes 💜

• logger

🦋 Classes 🦋

• EnvironmentTest — Test to make sure that Toil's environment variable save and restore system (environment.pickle) works. 🌷

🌸 Functions 🌸

• signal_leader(job) — Make a file in the file store that the leader can see 👀
• check_environment(job, try_name) — Fail if the test environment is wrong 🚨
• wait_a_bit(job) — Toil job that waits ⏳
• check_environment_repeatedly(job) — Toil job that checks the environment, waits, and checks it again, as
• main([options]) — Run the actual workflow with the given options 🚀

✨ Module Contents ✨

class toil.test.src.environmentTest.logger

class toil.test.src.environmentTest.EnvironmentTest(methodName='runTest')

Bases: toil.test.ToilTest

Test to make sure that Toil's environment variable save and restore system (environment.pickle) works 💅

The environment should be captured once at the start of the workflow, and should be sent through based on that, not based on the leader's current environment when the job is launched. Bestie, that's the whole point 🫶

• test_environment()

toil.test.src.environmentTest.signal_leader(job)

Make a file in the file store that the leader can see 👀

toil.test.src.environmentTest.check_environment(job, try_name)

Fail if the test environment is wrong 🚨

Parameters

• try_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.test.src.environmentTest.wait_a_bit(job)

Toil job that waits ⏳

✨ toil.test.src.environmentTest.check_environment_repeatedly(job) ✨

Okay so this is a Toil job 💅 It checks the environment, waits, and then checks it again, as separate invocations. Main character energy 🦋

✨ toil.test.src.environmentTest.main(options=None) ✨

bestie, this one runs the actual workflow with whatever options you hand it 🎀

Parameters

• options (Optional[argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>]) 🌸

---

🌸 toil.test.src.fileStoreTest 🌸

Attributes

• testingIsAutomatic 🦋
• logger 🦋

Classes

• hidden — hiding the abstract test classes from the Unittest loader so they can be inherited in different test suites for the different job stores 🙈
• NonCachingFileStoreTestWithFileJobStore — abstract tests for the general functions in the non-caching file store 💜
• CachingFileStoreTestWithFileJobStore — abstract tests for the cache-related functions 💜
• NonCachingFileStoreTestWithAwsJobStore — abstract tests for the general functions in the non-caching file store 💜
• CachingFileStoreTestWithAwsJobStore — abstract tests for the cache-related functions 💜
• NonCachingFileStoreTestWithGoogleJobStore — abstract tests for the general functions in the non-caching file store 💜
• CachingFileStoreTestWithGoogleJobStore — abstract tests for the cache-related functions 💜

Module Contents

• toil.test.src.fileStoreTest.testingIsAutomatic = True ✨
• toil.test.src.fileStoreTest.logger ✨
• class toil.test.src.fileStoreTest.hidden 🙈 Hiding the abstract test classes from the Unittest loader so it can be inherited in different test suites for the different job stores.
  • class AbstractFileStoreTest(methodName='runTest') 🧚 Bases: toil.test.ToilTest <#toil.test.ToilTest>

An abstract base class for testing the various general functions described in :class:toil.fileStores.abstractFileStore.AbstractFileStore 💅

• jobStoreType = None
• setUp(): hook method for setting up the test fixture before exercising it 🎀
• options(): mints a fresh options object with default settings and a fresh jobstore 🍓
• create_file(content, executable=False)
• testToilIsNotBroken(): runs a simple DAG to test whether any features other than caching were broken 🔥
• testFileStoreLogging(): writes a couple of files to the jobstore, deletes a couple of them, then reads back the written and locally deleted files 🌈
• testFileStoreOperations(): writes a couple of files to the jobstore, deletes a couple of them, then reads back the written and locally deleted files 🌈
• testWriteReadGlobalFilePermissions(): makes sure uploaded files keep their file permissions when they get downloaded again. This checks that a written executable file stays executable after being read 🫶
• testWriteExportFileCompatibility(): makes sure files created in a job keep their executable permissions when they're exported from the leader 🫶
• testImportReadFileCompatibility(): makes sure files imported to the leader keep their executable permissions when they're read by the fileStore 🫶
• testReadWriteFileStreamTextMode(): checks whether text mode is compatible with file streams 💖

• class AbstractNonCachingFileStoreTest(methodName='runTest') 🧚 Bases: AbstractFileStoreTest

Abstract tests for the general functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore. 👑

• options(): mints a fresh options object with default settings and a fresh jobstore 🍓

• class AbstractCachingFileStoreTest(methodName='runTest') 🧚 Bases: AbstractFileStoreTest

Abstract tests for the cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore. 👑

• options(): mints a fresh options object with default settings and a fresh jobstore 🍓
• testExtremeCacheSetup(): tries to create the cache with a bad worker active, then has 10 child jobs try to run in the chain. This tests whether the cache gets created properly even when the job crashes randomly 💀
• testCacheEvictionPartialEvict(): makes sure cache eviction happens as expected. Two files (20MB and 30MB) get written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A third job requests 10MB of disk, which requires evicting the 1st file. Checks that the behavior is as expected 🔥
• testCacheEvictionTotalEvict(): makes sure cache eviction happens as expected. Two files (20MB and 30MB) get written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A third job requests 10MB of disk, which requires evicting the 1st file. Checks that the behavior is as expected 🔥
• testCacheEvictionFailCase(): makes sure cache eviction happens as expected. Two files (20MB and 30MB) get written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A third job requests 10MB of disk, which requires evicting the 1st file. Checks that the behavior is as expected 🔥
• testAsyncWriteWithCaching(): makes sure async writing of files happens as expected. The first job forcefully modifies the cache size to 1GB. The second job asks for 1GB of disk and writes a 900MB file into the cache, then rewrites it to the job store. That triggers an async write, since the two unique jobstore IDs point to the same local file. The second write also isn't cached, because the first one already went into the cache and there "isn't enough space" for another. It immediately asserts that the second write isn't cached and is being asynchronously written to the job store. Getting the file from the jobstore afterwards should not fail 🚀
• testWriteNonLocalFileToJobStore(): writes a file that isn't in localTempDir to the job store. A file like that shouldn't be cached, so this checks that it isn't 🙈
• testWriteLocalFileToJobStore(): writes a file from localTempDir to the job store. Files like this get cached by default, so this checks that it is cached 🧁

⚠️ Nothing on this page changes the facts. The fine print is still all here, bestie 💖

• testReadCacheMissFileFromJobStoreWithoutCachingReadFile(): reads a file from the file store that has no cached copy, doesn't cache it, and checks that the link count on the file is right 🔗
• testReadCacheMissFileFromJobStoreWithCachingReadFile(): same cache miss, but this time the read file does get cached. The link count still has to check out 💅
• testReadCachHitFileFromJobStore(): reads a file that already has a cached copy and makes sure the link count is correct 🌸
• testMultipleJobsReadSameCacheHitGlobalFile(): writes a local file to the job store (which adds a copy to the cache), then has 10 jobs read it. It asserts the cached file size never goes up, and that unused job-required disk space is always (a multiple of job reqs) - (number of current file readers * filesize). At the very end, the cache should show unused job-required space = 0 🔥
• testMultipleJobsReadSameCacheMissGlobalFile(): writes a non-local file to the job store (so there's no cached copy), then has 10 jobs read it. Same vibes as the test above: cached file size never goes up, unused job-required disk space is always (a multiple of job reqs) - (number of current file readers * filesize), and at the end the cache shows unused job-required space = 0 💜
• testFileStoreExportFile()
• testReturnFileSizes(): writes a couple of files, deletes a couple, then reads back the written and locally deleted files. The cache has to be in a valid state after every single step 🧚
• testReturnFileSizesWithBadWorker(): the same routine as testReturnFileSizes(), but with a bad worker in the mix. Cache has to stay valid after every step 😭
• testControlledFailedWorkerRetry(): does a couple of job store operations, then dies on purpose. It checks that the restarted job is tracking values in the cache state file appropriately 🦋
• testRemoveLocalMutablyReadFile(): if a mutably read file gets deleted by the user, that's ok ✨
• testRemoveLocalImmutablyReadFile(): if an immutably read file gets deleted by the user, that's not ok 🙈
• testDeleteLocalFile(): tests the deletion powers of deleteLocalFile 🍓
• testSimultaneousReadsUncachedStream(): throws a bunch of simultaneous read attempts at a file that was created via a stream straight into the job store 🌈

---

🎀 toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore(methodName='runTest')

• Bases: hidden

Abstract tests for the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These are general enough that you can also use them for :class:toil.fileStores.CachingFileStore. 💖

• jobStoreType = 'file'

🎀 toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore(methodName='runTest')

• Bases: hidden

Abstract tests for the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore. ✨

• jobStoreType = 'file'

🎀 toil.test.src.fileStoreTest.NonCachingFileStoreTestWithAwsJobStore(methodName='runTest')

• Bases: hidden

Abstract tests for the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These are general enough that you can also use them for :class:toil.fileStores.CachingFileStore. 🌸

• jobStoreType = 'aws'

🎀 toil.test.src.fileStoreTest.CachingFileStoreTestWithAwsJobStore(methodName='runTest')

• Bases: hidden

Abstract tests for the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore. 🦋

• jobStoreType = 'aws'

🎀 toil.test.src.fileStoreTest.NonCachingFileStoreTestWithGoogleJobStore(methodName='runTest')

• Bases: hidden

Abstract tests for the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These are general enough that you can also use them for :class:toil.fileStores.CachingFileStore. 💜

• jobStoreType = 'google'

🎀 toil.test.src.fileStoreTest.CachingFileStoreTestWithGoogleJobStore(methodName='runTest')

• Bases: hidden

Abstract tests for the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore. 👑

• jobStoreType = 'google'

---

🦋 toil.test.src.helloWorldTest

💅 Classes

• HelloWorldTest: the legacy common base class for Toil tests 🕰️
• HelloWorld: a class that represents a unit of work in Toil ⭐
• FollowOn: a class that represents a unit of work in Toil 🫶

🧚 Functions

• childFn(job)

🌈 Module Contents

🎀 class toil.test.src.helloWorldTest.HelloWorldTest(methodName='runTest')

• Bases: toil.test.ToilTest

This is the legacy common base class for Toil tests, so it's giving old school 📜 New tests should be made in the pytest style and should not use this class.

Heads up bestie: new tests should be pytest-style, not this class. 💀

When running tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files to go. The directory gets created if it doesn't exist yet. The path can be relative, in which case it's assumed to be relative to the project root. If TOIL_TEST_TEMP isn't defined, temporary files and directories get created in the system's default location for that kind of thing, and any temp files or directories left over from tests get removed automatically during tear down. If you do set TOIL_TEST_TEMP, left-over files won't be removed. 🧹

• testHelloWorld()

🎀 class toil.test.src.helloWorldTest.HelloWorld

• Bases: toil.job.Job

A unit of work in Toil, bestie 🧁

• run(fileStore): override this function to do the work and dynamically create successor jobs ⚡
  • Parameters: fileStore, used to create local and globally sharable temporary files and to send log messages to the leader process 📮
  • Returns: the return value can be passed to other jobs using toil.job.Job.rv() 🎁

🎀 toil.test.src.helloWorldTest.childFn(job)

🎀 class toil.test.src.helloWorldTest.FollowOn(fileId)

• Bases: toil.job.Job

A unit of work in Toil, again 🔁

• fileId
• run(fileStore): override this function to do the work and dynamically create successor jobs 💫
  • Parameters: fileStore, used to create local and globally sharable temporary files and to send log messages to the leader process 📮
  • Returns: the return value can be passed to other jobs using toil.job.Job.rv() 🎁

---

🦋 toil.test.src.importExportFileTest

💅 Classes

• TestImportExportFile
• RestartingJob: a class that represents a unit of work in Toil 🌷

🧚 Functions

• create_file(tmp_path, content[, executable])

🌈 Module Contents

🎀 toil.test.src.importExportFileTest.create_file(tmp_path, content, executable=False)

• Parameters:
  • tmp_path (pathlib.Path)
  • content (str)
  • executable (bool)
• Return type — pathlib.Path 🧚

okay so the return type is pathlib.Path, and then we're into the actual test class bestie 💅

class toil.test.src.importExportFileTest.TestImportExportFile ✨

• message_portion_1 = 'What do you get when you cross a seal and a polar bear?' 🦭 this is the setup line of the joke
• message_portion_2 = ' A polar bear.' 😭 and this is the punchline
• test_import_export_restart_true(tmp_path) 🌸
  • Parameters:
    • tmp_path (pathlib.Path)
  • Return type: None
• test_import_export_restart_false(tmp_path) 🦋
  • Parameters:
    • tmp_path (pathlib.Path)
  • Return type: None
• test_basic_import_export(tmp_path, subtests) 💜
  • Ensures that uploaded files preserve their file permissions when they're downloaded again. This one checks that an imported executable file stays executable after being exported, so no lost vibes 🔥
  • Parameters:
    • tmp_path (pathlib.Path)
    • subtests (pytest.Subtests)
  • Return type: None

class toil.test.src.importExportFileTest.RestartingJob(msg_portion_file_id, trigger_file_id, message_portion_2) 🔥

• Bases: toil.job.Job
• A Job is basically a unit of work in Toil, and this is one of them 💅
• Parameters:
  • msg_portion_file_id (str)
  • trigger_file_id (str)
  • message_portion_2 (str)
• Attributes: msg_portion_file_id, trigger_file_id, message_portion_2 🧚
• run(file_store) 👑
  • Override this one to do the actual work and dynamically create successor jobs. It's giving main character energy 🎀
  • Parameters:
    • fileStore — the thing you use to make local and globally sharable temporary files, and to send log messages to the leader process 💖
    • file_store (toil.fileStores.abstractFileStore.AbstractFileStore)
  • Returns: whatever you return can be handed off to other jobs using toil.job.Job.rv() ✨
  • Return type: str

---

🌸 toil.test.src.jobDescriptionTest

🎀 Classes

• JobDescriptionTest — the legacy common base class for Toil tests 🕰️

💅 Module Contents

• class toil.test.src.jobDescriptionTest.JobDescriptionTest(methodName='runTest') 💖
  • Bases: toil.test.ToilTest
  • This is the legacy common base class for Toil tests. Okay so, new tests should be written in the pytest style and should NOT use this class 🙅‍♀️
  • When you're running tests you can optionally set the TOIL_TEST_TEMP environment variable to a directory where you want the temporary test files to go. It gets created if it doesn't exist yet. The path can be relative, in which case it's assumed to be relative to the project root 🌈
  • If TOIL_TEST_TEMP isn't defined, temporary files and directories get created in the system's default location for that kind of stuff, and any leftover temporary files or directories from tests get removed automatically during tear down. Otherwise, the leftover files won't be removed 🧹
  • setUp() 🌷 — the hook method for setting up the test fixture before it gets exercised
  • tearDown() 🌷 — the hook method for deconstructing the test fixture after testing it
  • testJobDescription() 🧚 — tests the public interface of a JobDescription
  • testJobDescriptionSequencing() ⭐

---

🦋 toil.test.src.jobEncapsulationTest

🎀 Classes

• TestJobEncapsulation — tests testing the EncapsulationJob class 💜

✨ Functions

• noOp() 🫶
• encapsulatedJobFn(job, string, outFile) 🫶

💅 Module Contents

• class toil.test.src.jobEncapsulationTest.TestJobEncapsulation 🔥
  • Tests testing the EncapsulationJob class. Iconic 👑
  • testEncapsulation(tmp_path) 💖
    • Tests the Job.encapsulation method, which uses the EncapsulationJob class ✨
    • Parameters:
      • tmp_path (pathlib.Path)
    • Return type: None
  • testAddChildEncapsulate() 🦋
    • Make sure that the encapsulate child doesn't have two parents with unique roots. Not me having to double check this 💀
    • Return type: None
• toil.test.src.jobEncapsulationTest.noOp() 🌸
  • Return type: None
• toil.test.src.jobEncapsulationTest.encapsulatedJobFn(job, string, outFile) 🧚
  • Parameters:
    • job (toil.job.Job)
    • string (str)
    • outFile (pathlib.Path)
  • Return type: toil.job.Promise

---

🌈 toil.test.src.jobFileStoreTest

💜 Attributes

• logger 🪵
• PREFIX_LENGTH 📏
• fileStoreString 🧵
• streamingFileStoreString 🌊

🎀 Classes

• JobFileStoreTest — tests testing the methods defined in :class:toil.fileStores.abstractFileStore.AbstractFileStore 💅

✨ Functions

• fileTestJob(job, inputFileStoreIDs, testStrings, ...) — a test job that exercises toil.fileStores.abstractFileStore.AbstractFileStore functions 🔥
• simpleFileStoreJob(job) 🌸
• fileStoreChild(job, testID1, testID2) 🦋

💅 Module Contents

• toil.test.src.jobFileStoreTest.logger 🪵
• toil.test.src.jobFileStoreTest.PREFIX_LENGTH = 200 📏
• class toil.test.src.jobFileStoreTest.JobFileStoreTest(methodName='runTest') 💖
  • Bases: toil.test.ToilTest
  • Tests testing the methods defined in :class:toil.fileStores.abstractFileStore.AbstractFileStore 💜
  • testCachingFileStore() ✨
  • testNonCachingFileStore() ✨
  • testJobFileStore() 🧚 — tests the case where about half the files are cached
  • testJobFileStoreWithBadWorker() 😭 — tests the case where about half the files are cached and the worker is randomly failing. Chaotic but make it make sense 🙈

toil.test.src.jobFileStoreTest.fileTestJob(job, inputFileStoreIDs, testStrings, chainLength) Test job exercises toil.fileStores.abstractFileStore.AbstractFileStore functions

toil.test.src.jobFileStoreTest.fileStoreString = 'Testing writeGlobalFile'

toil.test.src.jobFileStoreTest.streamingFileStoreString = 'Testing writeGlobalFileStream'

toil.test.src.jobFileStoreTest.simpleFileStoreJob(job)

toil.test.src.jobFileStoreTest.fileStoreChild(job, testID1, testID2)

toil.test.src.jobServiceTest

Attributes ┌────────┬───┐ │ logger │ │ └────────┴───┘

Classes ┌────────────────────────┬───────────────────────────┐ │ TestJobService │ Tests testing the │ │ │ Job.Service class │ ├────────────────────────┼───────────────────────────┤ │ TestPerfectServicet │ Tests testing the │ │ │ Job.Service class │ ├────────────────────────┼───────────────────────────┤ │ ToyService │ Abstract class used to │ │ │ define the interface to a │ │ │ service. │ ├────────────────────────┼───────────────────────────┤ │ ToySerializableService │ Abstract class used to │ │ │ define the interface to a │ │ │ service. │ └────────────────────────┴───────────────────────────┘

Functions ┌───────────────────────────────────┬────────────────────────────┐ │ serviceTest(job, outFile, │ Creates one service and │ │ messageInt) │ one accessing job, which │ │ │ communicate with two files │ │ │ to establish │ ├───────────────────────────────────┼────────────────────────────┤ │ serviceTestRecursive(job, │ Creates a chain of │ │ outFile, messages) │ services and accessing │ │ │ jobs, each paired │ │ │ together. │ ├───────────────────────────────────┼────────────────────────────┤ │ serviceTestParallelRecursive(job, │ Creates multiple chains of │ │ outFiles, messageBundles) │ services and accessing │ │ │ jobs. │ ├───────────────────────────────────┼────────────────────────────┤ │ serviceAccessor(job, │ Writes a random integer │ │ communicationFiles, outFile, ...) │ iinto the inJobStoreFileID │ │ │ file, then tries 10 times │ │ │ reading │ ├───────────────────────────────────┼────────────────────────────┤ │ fnTest(strings, outputFile) │ Function concatenates the │ │ │ strings together and │ │ │ writes them to the output │ │ │ file │ └───────────────────────────────────┴────────────────────────────┘

Module Contents toil.test.src.jobServiceTest.logger

class toil.test.src.jobServiceTest.TestJobService Tests testing the Job.Service class

testServiceSerialization(tmp_path) Tests that a service can receive a promise without producing a serialization error.

Parameters tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type None

testService(tmp_path, checkpoint=False) Tests the creation of a Job.Service with random failures of the worker.

Parameters

• tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• checkpoint (bool </usr/share/doc/python3/html/library/functions.html#bool>)

Return type None

testServiceDeadlock(tmp_path) Creates a job with more services than maxServices, checks that deadlock is detected.

Parameters tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type None

testServiceWithCheckpoints(tmp_path) Tests the creation of a Job.Service with random failures of the worker, making the root job use checkpointing to restart the subtree.

Parameters tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

Return type None

testServiceRecursive(tmp_path, checkpoint=True) Tests the creation of a Job.Service, creating a chain of services and accessing jobs. Randomly fails the worker.

Parameters

• tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• checkpoint (bool </usr/share/doc/python3/html/library/functions.html#bool>)

Return type None

testServiceParallelRecursive(tmp_path, checkpoint=True) Tests the creation of a Job.Service, creating parallel chains of services and accessing jobs. Randomly fails the worker.

Parameters

• tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• checkpoint (bool </usr/share/doc/python3/html/library/functions.html#bool>)

Return type None

runToil(tmp_path, rootJob, retryCount=1, badWorker=0.5, badWorkedFailInterval=0.1, maxServiceJobs=sys.maxsize, deadlockWait=60, max_attempts=50)

Parameters

• tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• rootJob (toil.job.Job <#toil.job.Job>)
• retryCount (int </usr/share/doc/python3/html/library/functions.html#int>)
• badWorker (float </usr/share/doc/python3/html/library/functions.html#float>)
• badWorkedFailInterval (float </usr/share/doc/python3/html/library/functions.html#float>)
• maxServiceJobs (int </usr/share/doc/python3/html/library/functions.html#int>)
• deadlockWait (int </usr/share/doc/python3/html/library/functions.html#int>)
• max_attempts (int </usr/share/doc/python3/html/library/functions.html#int>)

Return type None

class toil.test.src.jobServiceTest.TestPerfectServicet Bases: TestJobService

Tests testing the Job.Service class

runToil(*args, **kwargs) Let us run all the tests in the other service test class, but without worker failures.

Parameters

• args (Any)
• kwargs (Any)

Return type None

toil.test.src.jobServiceTest.serviceTest(job, outFile, messageInt) Creates one service and one accessing job, which communicate with two files to establish that both run concurrently.

Parameters

• job (toil.job.Job <#toil.job.Job>)
• outFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• messageInt (int </usr/share/doc/python3/html/library/functions.html#int>)

Return type None

toil.test.src.jobServiceTest.serviceTestRecursive(job, outFile, messages) Creates a chain of services and accessing jobs, each paired together.

Parameters

• job (toil.job.Job <#toil.job.Job>)
• outFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• messages (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>])

Return type None

toil.test.src.jobServiceTest.serviceTestParallelRecursive(job, outFiles, messageBundles) Creates multiple chains of services and accessing jobs.

Parameters

• job (toil.job.Job) 💅 The job that gets to hold the outFiles and messageBundles.
• outFiles (list[pathlib.Path]) 📂 A list of paths for the output files.
• messageBundles (list[list[int]]) 🧚 A list of lists, each holding ints.

Return type: None 🫶

---

toil.test.src.jobServiceTest.ToyService

class toil.test.src.jobServiceTest.ToyService(messageInt, *args, **kwargs)

Bases: toil.job.Job.Service ✨

This is the blueprint for a service. You subclass it to make your own services 🎀. It never runs as a regular job, though. It runs inside a ServiceHostJob 🦋

Parameters

• messageInt (int) 🌸
• args (Any) 🌈
• kwargs (Any) 🌈

Attribute: messageInt 💜

🦋 start(job)

Starts the service 🚀

Parameters

• job (toil.job.ServiceHostJob) — The host job the service is running inside. You can use it to register deferred functions, or to grab the fileStore for making temporary files 🧚

Returns

An object that tells jobs how to reach the service. It has to be pickleable, since jobs use it to access the service (see toil.job.Job.addService()) 🔥

Return type: tuple[str, str] 💖

🦋 stop(job)

Stops the service. This one can block until it's totally done, so no rushing 🛑

Parameters

• job (toil.job.Job) — The underlying host job the service is running in. Same deal: use it to register deferred functions, or to access the fileStore for temporary files 🌸

Return type: None 🫶

🦋 check()

Checks that the service is still running 👀

Raises

• exceptions.RuntimeError — If the service failed, the service job gets labeled failed 💀

Returns

True if the service is still running, otherwise False. If it returns False, the service job gets terminated and counts as a success ✨

Important point, bestie: if the service job exits because of a failure, it should raise a RuntimeError. It should NOT return False! 😭🚨

Return type: Literal[True] 💅

🦋 serviceWorker (static)

static serviceWorker(jobStore, terminate, error, inJobStoreID,
outJobStoreID, messageInt)

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore) 🗄️
• terminate (threading.Event) 🛑
• error (threading.Event) ⚠️
• inJobStoreID (str) 🦋
• outJobStoreID (str) 🦋
• messageInt (int) 🌸

Return type: None 🫶

---

toil.test.src.jobServiceTest.serviceAccessor

toil.test.src.jobServiceTest.serviceAccessor(job, communicationFiles,
outFile, to_subtract)

This one writes a random integer into the inJobStoreFileID file. Then it tries 10 times to read from outJobStoreFileID to get a pair of integers. The first one should equal i, and the second one gets written into the outputFile 🧠

Parameters

• job (toil.job.ServiceHostJob) 🧚
• communicationFiles (tuple[str, str]) 📨
• outFile (pathlib.Path) 📂
• to_subtract (int) ➖

Return type: None 🫶

---

toil.test.src.jobServiceTest.ToySerializableService

class toil.test.src.jobServiceTest.ToySerializableService(messageInt,
*args, **kwargs)

Bases: toil.job.Job.Service ✨

Same vibe as ToyService: it's the blueprint for a service you subclass 🎀. It doesn't run as a job either. It runs inside a ServiceHostJob 🦋

Parameters

• messageInt (str) 🌸
• args (Any) 🌈
• kwargs (Any) 🌈

Attribute: messageInt 💜

🦋 start(job)

Starts the service 🚀

Parameters

• job (toil.job.Job) — The underlying host job the service is running in. You can register deferred functions with it, or access the fileStore for temporary files 🧚

Returns

An object that describes how to access the service. It must be pickleable, because jobs use it to reach the service (see toil.job.Job.addService()) 🔥

Return type: str 💖

🦋 stop(job)

Stops the service. This can block until it finishes 🛑

Parameters

• job (toil.job.Job) — The underlying host job the service is running in. Use it to register deferred functions, or to access the fileStore for temporary files 🌸

Return type: None 🫶

🦋 check()

Checks that the service is still running 👀

Raises

• exceptions.RuntimeError — If the service failed, the service job gets labeled failed 💀

Returns

True if the service is still running, otherwise False. If it's False, the service job gets terminated and counts as a success ✨

Important point again, bestie: if the service job exits due to a failure, it should raise a RuntimeError, not return False! 😭🚨

Return type: Literal[True] 💅

---

toil.test.src.jobServiceTest.fnTest

toil.test.src.jobServiceTest.fnTest(strings, outputFile)

This function smooshes the strings together and writes them to the output file 🧁

Parameters

• strings (list[str]) 🧵
• outputFile (pathlib.Path) 📂

Return type: None 🫶

---

toil.test.src.jobTest

Classes 🌈

• TestJob — Tests the job class 🧪
• TrivialService — Abstract class used to define the interface to a service. Basically the same energy as the services above 🦋

More to come, so stay tuned! 💖

✨ FUNCTIONS ✨

okay so bestie, these are the test helper functions 🧚 some of them have a description and some just... don't, so I'll tell you which is which:

• simpleJobFn(job, value): no description on this one 🤷‍♀️
• fn1Test(string, outputFile): this one tacks the next character onto the end of the given string. The description literally cuts off mid-sentence after "in the given" 💀
• fn2Test(pStrings, s, outputFile): this one glues together everything in pStrings, then s, in that exact order, and writes the result out. The description also stops mid-sentence after "writes the result to" 😭
• trivialParent(job): no description 🌸
• parent(job): no description 🌸
• diamond(job): no description 🌸
• child(job): no description 🌸
• errorChild(job): no description 🌸

---

✨ MODULE CONTENTS ✨

🦋 class toil.test.src.jobTest.TestJob

This is where all the job class tests live. Tests the job class, obviously 💅

💜 testStatic(tmp_path)

Makes a DAG of jobs the non-dynamic way, then runs it. Here's the shape of it:

A -> F
\-------
B -> D  \
 \       \
  ------- C -> E

A follow-on is marked by ->.

• Parameters
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• Return type
  • None

💜 testStatic2(tmp_path)

Same vibe as testStatic: a non-dynamic DAG that gets built and run. Same diagram too, so you know the drill 🫶

A -> F
\-------
B -> D  \
 \       \
  ------- C -> E

A follow-on is marked by ->.

• Parameters
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• Return type
  • None

💜 testTrivialDAGConsistency(tmp_path)

No description here, but the params are the same as always 🌈

• Parameters
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• Return type
  • None

💜 testDAGConsistency(tmp_path)

No description on this one either, bestie 🙈

• Parameters
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• Return type
  • None

💜 testSiblingDAGConsistency(tmp_path)

A slightly spicier case 🌶️ The stranded job's predecessors are siblings here, not parent and child.

• Parameters
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• Return type
  • None

💜 testDeadlockDetection()

This one randomly generates job graphs with all kinds of cycles in them and checks that each one throws an exception properly. It also checks that having multiple roots causes a deadlock exception 🔥

• Return type
  • None

💜 testNewCheckpointIsLeafVertexNonRootCase(tmp_path, subtests)

This is the test for issue #1465, where checkpoint jobs that aren't leaf vertices were being identified as leaf vertices incorrectly 😤 It checks that new checkpoint jobs really are leaf vertices, starting from this baseline workflow:

Parent
  |
Child # Checkpoint=True

• Parameters
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • subtests (pytest.Subtests)
• Return type
  • None

💜 testNewCheckpointIsLeafVertexRootCase(tmp_path, subtests)

This is the test for issue #1466, where detecting checkpoint jobs that aren't leaf vertices was skipping the workflow root job 🙃 It checks that a new checkpoint job is a leaf vertex, starting from a baseline workflow that's just one root job:

Root # Checkpoint=True

• Parameters
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • subtests (pytest.Subtests)
• Return type
  • None

💜 runNewCheckpointIsLeafVertexTest(tmp_path, subtests, createWorkflowFn)

This runs the leaf vertex check for a checkpoint job using both valid and invalid cases. Slay the coverage 💖

• Parameters
  • createWorkflowFn (collections.abc.Callable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Callable>[[], tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.job.Job <#toil.job.Job>, toil.job.Job <#toil.job.Job>]]). This is the function that makes a brand new workflow and hands back a tuple of:
    • 0. the workflow root job
    • 1. a checkpoint job to test inside that workflow
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • subtests (pytest.Subtests)
• Return type
  • None

💜 runCheckpointVertexTest(workflowRootJob, checkpointJob, tmp_path, checkpointJobService=None, checkpointJobChild=None, checkpointJobFollowOn=None, expectedException=None)

This tweaks the checkpoint job based on whatever you pass in, then runs the workflow and checks for the expected exception, if there is one ✨

• Parameters
  • workflowRootJob (toil.job.Job <#toil.job.Job>)
  • checkpointJob (toil.job.Job <#toil.job.Job>)
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
  • checkpointJobService (Optional[toil.job.Job.Service <#toil.job.Job.Service>])
  • checkpointJobChild (Optional[toil.job.Job <#toil.job.Job>])
  • checkpointJobFollowOn (Optional[toil.job.Job <#toil.job.Job>])
  • expectedException (Optional[type </usr/share/doc/python3/html/library/functions.html#type>[Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>]])
• Return type
  • None

💜 testEvaluatingRandomDAG(tmp_path)

This randomly generates test input, then checks that the job graph runs successfully. It uses the existence of promises to validate the run 🎀

• Parameters
  • tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• Return type
  • None

💜 static getRandomEdge(nodeNumber)

A static helper that gives you a random edge 🎲

• Parameters
  • nodeNumber (int </usr/share/doc/python3/html/library/functions.html#int>)
• Return type
  • tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]

💜 static makeRandomDAG(nodeNumber)

Makes a random DAG with "nodeNumber" nodes, where every node is connected. The return value is a list of edges, each one shaped like (a, b), where a and b are integers >= 0 and < nodeNumber that refer to nodes, and the edge goes from a to b 🧩

heads up bestie 👀 the words say "list" but the return type below says set. Both are in the original, so I'm flagging the mismatch and not picking one for you.

• Parameters
  • nodeNumber (int </usr/share/doc/python3/html/library/functions.html#int>)
• Return type
  • set </usr/share/doc/python3/html/library/stdtypes.html#set>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]]

💜 static getAdjacencyList(nodeNumber, edges)

Builds an adjacency list representation of the edges 🌸

• Parameters
  • nodeNumber (int </usr/share/doc/python3/html/library/functions.html#int>)
• edges (set[tuple[int, int]]) 🦋 the set of int pairs that link the nodes together

Return type

list[set[int]] 💖

reachable(node, adjacencyList) ✨

Hunts down every node you can reach from this one, and yes, the node itself counts too 💅 It hands back a set of integers 🌸

Parameters

• node (int)
• adjacencyList (list[set[int]])

Return type

set[int]

addRandomFollowOnEdges(childAdjacencyList) 🎀

Adds random follow on edges to the graph, which is stored as an adjacency list 🧚 The follow on edges come back as a set, and their augmented edges get added right into the adjacency list ✨

Parameters

• childAdjacencyList (list[set[int]])

Return type

set[tuple[int, int]]

makeJobGraph(nodeNumber, childEdges, followOnEdges, outPath, addServices=True) 👑

Converts a DAG into a job graph 🌈 childEdges and followOnEdges are the lists of child and followOn edges 🫶

Parameters

• nodeNumber (int)
• childEdges (set[tuple[int, int]])
• followOnEdges (set[tuple[int, int]])
• outPath (Optional[pathlib.Path])
• addServices (bool)

Return type

toil.job.Job

isAcyclic(adjacencyList) 💀

Returns true if there are any cycles in the graph, which is represented as an adjacency list 🔁

Parameters

• adjacencyList (list[set[int]])

Return type

bool

toil.test.src.jobTest.simpleJobFn(job, value) 🧚

Parameters

• job (toil.job.ServiceHostJob)
• value (str)

Return type

None

toil.test.src.jobTest.fn1Test(string, outputFile) 🦋

This function appends the next character after the last character in the given string to the string, writes the string to a file, and returns it. For example, if the string is "AB", we write and return "ABC" 💖

Parameters

• string (str)
• outputFile (toil.lib.misc.FileDescriptorOrPath)

Return type

str

toil.test.src.jobTest.fn2Test(pStrings, s, outputFile) 🫶

This function concatenates the strings in pStrings and s, in that order, and writes the result to the output file. It returns s ✨

Parameters

• pStrings (list[str])
• s (str)
• outputFile (pathlib.Path)

Return type

str

toil.test.src.jobTest.trivialParent(job) 🌸

Parameters

• job (toil.job.Job)

Return type

None

toil.test.src.jobTest.parent(job) 👪

Parameters

• job (toil.job.Job)

Return type

None

toil.test.src.jobTest.diamond(job) 💎

Parameters

• job (toil.job.Job)

Return type

None

toil.test.src.jobTest.child(job) 🍓

Parameters

• job (toil.job.Job)

Return type

None

toil.test.src.jobTest.errorChild(job) 🙈

Parameters

• job (toil.job.Job)

Return type

NoReturn

---

class toil.test.src.jobTest.TrivialService(message, *args, **kwargs) 🌷

Bases: toil.job.Job.Service 👑

This is an abstract class used to define the interface to a service. You should subclass it yourself to define your own services 💅 It is not executed as a job. It runs inside a ServiceHostJob 🏠

Parameters

• message (str)
• args (Any)
• kwargs (Any)

message

start(job) 🚀

Starts the service ✨

Parameters

• job (toil.job.ServiceHostJob): the underlying host job the service runs in. You can use it to register deferred functions, or to get at the fileStore for making temporary files 🧚

Returns

An object describing how to access the service. It has to be pickleable, and jobs will use it to reach the service (see toil.job.Job.addService()) 🦋

Return type

str

stop(job) 🛑

Stops the service. This function can block until it's totally done 💀

Parameters

• job (toil.job.ServiceHostJob): the underlying host job the service runs in. You can use it to register deferred functions, or to get at the fileStore for making temporary files 🧚

Return type

None

check() 🔍

Checks that the service is still running 💖

Raises

• exceptions.RuntimeError: if the service failed, this makes the service job get labeled failed 😭

Returns

True if the service is still running, else False. If it's False, the service job gets terminated and counts as a success 🎀

Important point: if the service job exits because of a failure, it should raise a RuntimeError, not return False! 🚨

Return type

bool

toil.test.src.miscTests 🌈

Attributes ✨

• log (no value given)

✨ Classes ✨

okay so here's the lineup 💖

• TestMisc — the grab bag 🧺 home for miscellaneous tests that are too small to get their own test file ✨
• TestPanic — (no description for this one 🙈)

---

✨ Module Contents ✨

the log file is toil.test.src.miscTests.log 📝

🦋 class toil.test.src.miscTests.TestMisc

bestie, this is the misc class 🎀 it's for tests that are too small for their own file and don't fit neatly into any other test suite, so they live here 💅

testIDStability()

• Return type: None

testGetSizeOfDirectoryWorks(tmp_path)

this one is lowkey important 😭 it checks that toil.common.getDirSizeRecursively never underestimates how much disk space you'll need 💾 allocation is different on every system, so the computed number should always be equal to or slightly bigger than the creation value. the test makes a pile of random directories and randomly sized files, then runs getDirSizeRecursively on them to make sure it's giving the right vibes 🔥

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: None

test_atomic_install(tmp_path)

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: None

test_atomic_install_dev()

• Return type: None

test_atomic_context_ok(tmp_path)

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: None

test_atomic_context_error(tmp_path)

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: None

test_call_command_ok()

• Return type: None

test_call_command_err()

• Return type: None

🧯 class toil.test.src.miscTests.TestPanic

this class has no description, so we're just gonna vibe 🫶

test_panic_by_hand()

• Return type: None

test_panic()

• Return type: None

test_panic_with_secondary()

• Return type: None

test_nested_panic()

• Return type: None

try_and_panic_by_hand()

• Return type: None

try_and_panic()

• Return type: None

try_and_panic_with_secondary()

• Return type: None

try_and_nested_panic_with_secondary()

• Return type: None

---

✨ toil.test.src.promisedRequirementTest ✨

✨ Attributes ✨

• log — (blank, nothing to see here 🙈)

✨ Classes ✨

• AbstractPromisedRequirementsTest — an abstract base class for testing Toil workflows with promised requirements 💖
• TestSingleMachinePromisedRequirements — tests against the SingleMachine batch system 🔥
• TestMesosPromisedRequirements — tests against the Mesos batch system 🌈

✨ Functions ✨

• maxConcurrency(job, cpuCount, filename, coresPerJob) — returns the max number of concurrent tasks when you're using a PromisedRequirement instance 👑
• getOne() — (no description 🙈)
• getThirtyTwoMb() — (no description 🙈)
• logDiskUsage(job, funcName[, sleep]) — logs the job's disk usage to master, then sleeps for however long you specified 😴

✨ Module Contents ✨

the log file is toil.test.src.promisedRequirementTest.log 📝

🌸 class toil.test.src.promisedRequirementTest.AbstractPromisedRequirementsTest

• Bases: toil.test.batchSystems.batchSystemTest.AbstractBatchSystemJobTest <#toil.test.batchSystems.batchSystemTest.AbstractBatchSystemJobTest>

this is the abstract base class for testing Toil workflows with promised requirements, so it's the foundation everything else is built on 🧱

testConcurrencyDynamic(tmp_path)

asserts that promised core resources get allocated properly when you're running a dynamic Toil workflow ✨

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: None

testConcurrencyStatic(tmp_path)

same energy, but this time it asserts that promised core resources get allocated properly using a static DAG 💅

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: None

getOptions(tempDir, caching=True)

configures the options for a Toil workflow and makes the job store 🎀

• Parameter tempDir (str): path to the test directory
• Returns a Toil options object 💖
• Parameters:
  • tempDir (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • caching (bool, /usr/share/doc/python3/html/library/functions.html#bool)
• Return type: argparse.Namespace (/usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

getCounterPath(tempDir)

returns the path to a counter file 🧚

• Parameter tempDir: path to the test directory
• Returns the path to the counter file
• Parameters:
  • tempDir (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: pathlib.Path (/usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

testPromisesWithJobStoreFileObjects(tmp_path, caching=True)

okay this one is giving main character energy 👑 it checks that FileID objects get pickled properly when they're used as return values from functions. then it makes sure lambdas of promised FileID objects can describe the requirements of a later job, which is a super common move in Toil scripts 🔥 returns None.

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • caching (bool, /usr/share/doc/python3/html/library/functions.html#bool)
• Return type: None

testPromisesWithNonCachingFileStore(tmp_path)

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: None

testPromiseRequirementRaceStatic(tmp_path)

checks for a race condition when you're using promised requirements together with child job functions 😭 we do not want race conditions, bestie 🙅‍♀️

• Parameters:
  • tmp_path (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type: None

toil.test.src.promisedRequirementTest.maxConcurrency(job, cpuCount, filename, coresPerJob)

Okay bestie, this one is the MVP 👑 It hands back the max number of concurrent tasks when you're using a PromisedRequirement instance to allocate cores per job ✨

Parameters

• cpuCount (int, /usr/share/doc/python3/html/library/functions.html#int) — number of available cpus 🫶
• filename (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path) — path to counter file 🦋
• coresPerJob (int, /usr/share/doc/python3/html/library/functions.html#int) — number of cores assigned to each job 💜
• job (toil.job.Job) 🌸

Returns — max concurrency value per CPU 💖

Return type — list[toil.job.Promise] (list: /usr/share/doc/python3/html/library/stdtypes.html#list) 🍓

toil.test.src.promisedRequirementTest.getOne()

Iconic, it just returns an int 💅

Return type — int (/usr/share/doc/python3/html/library/functions.html#int)

toil.test.src.promisedRequirementTest.getThirtyTwoMb()

This one gives you a str back, no cap 🧚

Return type — str (/usr/share/doc/python3/html/library/stdtypes.html#str)

toil.test.src.promisedRequirementTest.logDiskUsage(job, funcName, sleep=0)

Okay so this one logs the job's disk usage to master and then naps for whatever amount of time you give it 😴🌈

Returns — job function's disk usage ✨

Parameters

• job (toil.job.JobFunctionWrappingJob) 🎀
• funcName (str, /usr/share/doc/python3/html/library/stdtypes.html#str) 💜
• sleep (int, /usr/share/doc/python3/html/library/functions.html#int) 🙈

Return type — int (/usr/share/doc/python3/html/library/functions.html#int)

class toil.test.src.promisedRequirementTest.TestSingleMachinePromisedRequirements

Bases: AbstractPromisedRequirementsTest 🌸

This class is the test suite that runs against the SingleMachine batch system 💖

• getBatchSystemName() — Return type: (str (/usr/share/doc/python3/html/library/stdtypes.html#str), AbstractBatchSystem) ✨
• tearDown() — Return type: None 🫶

class toil.test.src.promisedRequirementTest.TestMesosPromisedRequirements

Bases: AbstractPromisedRequirementsTest, toil.batchSystems.mesos.test.MesosTestSupport 🔥

This one is the Mesos batch system test suite, and it's giving main character energy 👑

• mesos_support() — Return type: collections.abc.Generator (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator)[None] 🧚
• getOptions(tempDir, caching=True) — Okay this one configures options for the Toil workflow and makes the job store 💅 Here's the docstring, verbatim: :param str tempDir: path to test directory and :return: Toil options object 🌈

Parameters

• tempDir (pathlib.Path, /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path) 🦋
• caching (bool, /usr/share/doc/python3/html/library/functions.html#bool) 💜

Return type — argparse.Namespace (/usr/share/doc/python3/html/library/argparse.html#argparse.Namespace) ✨

• getBatchSystemName() — Return type: (str (/usr/share/doc/python3/html/library/stdtypes.html#str), AbstractBatchSystem) 🍓

---

🌸 toil.test.src.promisesTest 🌸

Classes

• CachedUnpicklingJobStoreTest — Legacy common base class for Toil tests 🕰️
• ChainedIndexedPromisesTest — Legacy common base class for Toil tests 🕰️
• PathIndexingPromiseTest — Test support for indexing promises of arbitrarily nested data structures of lists, dicts and 🦋

Functions

• parent(job) 🧚
• child() 🧚
• a(job) 🧚
• b(job) 🧚
• c() 🧚
• d(job) 🧚
• e() 🧚

Module Contents

• toil.test.src.promisesTest.CachedUnpicklingJobStoreTest(methodName='runTest') — Bases: toil.test.ToilTest 💖

Legacy common base class for Toil tests. Okay so, new tests should be made in the 'pytest' style and not use this class, bestie 🙅‍♀️

When you run tests you can optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files placed. The directory gets created if it doesn't exist. The path can be relative, in which case it's relative to the project root. If TOIL_TEST_TEMP isn't defined, temp files and directories get created in the system's default location for such files, and any temp files or directories left over from tests get removed automatically during tear down. Otherwise, left-over files will not be removed 🫶

• test() — Runs two identical Toil workflows with different job store paths 🔥

• toil.test.src.promisesTest.parent(job) 🌈
• toil.test.src.promisesTest.child() 🌈
• toil.test.src.promisesTest.ChainedIndexedPromisesTest(methodName='runTest') — Bases: toil.test.ToilTest 💅

Legacy common base class for Toil tests. New tests should be made in the 'pytest' style and not use this class 🙅‍♀️

Same vibe with the env var: optionally set TOIL_TEST_TEMP to a directory for temporary test files. The directory gets created if it doesn't exist. The path may be relative, in which case it's assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories get created in the system's default location for such files, and any temporary files or directories left over from tests will be removed automatically during tear down. Otherwise, left-over files will not be removed 🦋

• test() 🎀

• toil.test.src.promisesTest.a(job) 🧚
• toil.test.src.promisesTest.b(job) 🧚
• toil.test.src.promisesTest.c() 🧚
• toil.test.src.promisesTest.PathIndexingPromiseTest(methodName='runTest') — Bases: toil.test.ToilTest 🌸

Test support for indexing promises of arbitrarily nested data structures of lists, dicts and tuples, or any other object supporting the __getitem__() protocol. Bestie, this one is a whole vibe 💜

• test() ✨

• toil.test.src.promisesTest.d(job) 🧚
• toil.test.src.promisesTest.e() 🧚

---

🌈 toil.test.src.realtimeLoggerTest 🌈

Classes

• RealtimeLoggerTest — Legacy common base class for Toil tests 🕰️
• MessageDetector — Detect the secret message and set a flag 🕵️‍♀️
• LogTest — Class represents a unit of work in toil 💖

Module Contents

• toil.test.src.realtimeLoggerTest.RealtimeLoggerTest(methodName='runTest') — Bases: toil.test.ToilTest 💜

Legacy common base class for Toil tests. New tests should be made in the 'pytest' style and not use this class. Periodt 💅

When you run tests, you can optionally set the TOIL_TEST_TEMP environment variable 🧚 to the path of a directory where your temporary test files should go 💖. If that directory doesn't exist yet, it gets created for you ✨. The path can be relative, and if it is, it's counted from the project root 🌸. If TOIL_TEST_TEMP isn't defined, temp files and directories land in your system's default spot for that stuff, and any leftover temp files or directories from tests get cleaned up automatically during tear down 🧹. If you do set it, leftover files stick around and don't get removed 👀.

testRealtimeLogger() 🎀

💌 class toil.test.src.realtimeLoggerTest.MessageDetector

Bases: logging.StreamHandler </usr/share/doc/python3/html/library/logging.handlers.html#logging.StreamHandler> 🦋

This class is all about detecting the secret message 🤫 and setting a flag when it shows up 🚩.

• detected = False
• overLogged = False

💬 emit(record)

Emit a record. 📝

If a formatter is specified, it gets used to format the record. The record is then written to the stream with a trailing newline. If exception information is present, it's formatted with traceback.print_exception and tacked onto the stream. If the stream has an 'encoding' attribute, that's what decides how the output gets written to the stream. Iykyk 💅

💖 class toil.test.src.realtimeLoggerTest.LogTest

Bases: toil.job.Job <#toil.job.Job> ✨

Class represents a unit of work in toil. 🍓

🛠️ run(fileStore)

Override this function to perform work and dynamically create successor jobs. Main character energy 👑

Parameters

• fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process. 📬

Returns: the return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>. 🎁

---

🌈 toil.test.src.regularLogTest

🧾 Attributes

• logger

📚 Classes

• RegularLogTest

🌸 Module Contents

toil.test.src.regularLogTest.logger

🌷 class toil.test.src.regularLogTest.RegularLogTest

• testLogToMaster() — Return type: None 💜
• testWriteLogs(tmp_path) — Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 🧪
• testWriteGzipLogs(tmp_path) — Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 🗜️
• testMultipleLogToMaster() — Return type: None 🔁
• testRegularLog() — Return type: None 🥰

---

🦋 toil.test.src.resourceTest

🧸 Classes

• TestResource — Test module descriptors and resources derived from them. 🎀

🪄 Functions

• tempFileContaining(directory, content[, suffix]) — Write a file with the given contents, and keep it on disk as long as the context is active. 📄

🌸 Module Contents

✨ toil.test.src.resourceTest.tempFileContaining(directory, content, suffix='')

Write a file with the given contents, and keep it on disk as long as the context is active. :param str content: The contents of the file. :param str suffix: The extension to use for the temporary file. 🗂️

Parameters

• directory (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)
• content (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
• suffix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type: str </usr/share/doc/python3/html/library/stdtypes.html#str> 💫

🫧 class toil.test.src.resourceTest.TestResource

Test module descriptors and resources derived from them. 🌷

• testStandAlone(tmp_path) — Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 🧍‍♀️
• testPackage(tmp_path) — Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 📦
• testVirtualEnv(tmp_path) — Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 🐍
• testStandAloneInPackage(tmp_path) — Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 🧩
• testBuiltIn() — Return type: None 🏠
• testNonPyStandAlone(tmp_path) — Asserts that Toil enforces the user script to have a .py or .pyc extension, because that's the only way auto-deployment can re-import the module on a worker. See <https://github.com/BD2KGenomics/toil/issues/631> and <https://github.com/BD2KGenomics/toil/issues/858>. 🚫

Heads up bestie: a user script must end in .py or .pyc or auto-deployment can't re-import it on a worker 😭

Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 💅

---

🍑 toil.test.src.restartDAGTest

📌 Attributes

• logger

🎀 Classes

• TestRestartDAG — Tests that restarted job DAGs don't run children of jobs that failed in the first run till the parent completes successfully in the restart. 🔄

🧁 Functions

• passingFn(job[, file]) — This function is guaranteed to pass, because it does nothing out of the ordinary. ✅
• failingFn(job, failType, file) — This function is guaranteed to fail via a raised assertion, or an os.kill. 💥

🌼 Module Contents

toil.test.src.restartDAGTest.logger

🌺 class toil.test.src.restartDAGTest.TestRestartDAG

Tests that restarted job DAGs don't run children of jobs that failed in the first run till the parent completes successfully in the restart. 🫶

• testRestartedWorkflowSchedulesCorrectJobsOnFailedParent(tmp_path) — Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 🗓️
• testRestartedWorkflowSchedulesCorrectJobsOnKilledParent(tmp_path) — Parameters: tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>). Return type: None 🔪

🌟 toil.test.src.restartDAGTest.passingFn(job, file=None)

This function is guaranteed to pass, because it does nothing out of the ordinary. ✅

If "file" is provided, it will be created. 📁

Parameters

• file (Optional[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]) -- The path of a file that must be created if provided. 🗃️
• job (toil.job.Job <#toil.job.Job>) 🧃

Return type: None 🫧

💥 toil.test.src.restartDAGTest.failingFn(job, failType, file)

This function is guaranteed to fail via a raised assertion, or an os.kill 😭

Parameters

• job (toil.job.Job, see #toil.job.Job) — the Job itself 💅
• failType (str, see /usr/share/doc/python3/html/library/stdtypes.html#str) — 'raise' or 'kill 💥
• file (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path) — the path of a file that must be created 📁

Return type: None 🫶

---

🎀 toil.test.src.resumabilityTest

Classes

• TestResumability — the class that holds all the resumability tests, bestie 💖 Reference: https://github.com/BD2KGenomics/toil/issues/808

Functions

• parent(job) — sets up a bunch of dummy child jobs, plus a bad job that needs to be restarted as the follow-on 🌸
• chaining_parent(job) — sets up a failing job for something to chain to 🦋
• goodChild(job) — does nothing. Literally nothing. Iconic 😴
• badChild(job) — fails the first time it's run, then succeeds the second time 🙈

Module Contents

• toil.test.src.resumabilityTest.TestResumability — the class again, same issue link: https://github.com/BD2KGenomics/toil/issues/808 ✨
  • test(tmp_path) — checks that a toil workflow which fails once can be resumed without a NoSuchJobException 🧪
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
  • test_chaining(tmp_path) — checks that a job which is chained to and fails can resume and succeed 🔗
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
• toil.test.src.resumabilityTest.parent(job) — sets up a bunch of dummy child jobs, and a bad job that needs to be restarted as the follow-on 🌟
  • Parameters: job (toil.job.Job, see #toil.job.Job)
  • Return type: None
• toil.test.src.resumabilityTest.chaining_parent(job) — sets up a failing job to chain to 🦋
  • Parameters: job (toil.job.Job, see #toil.job.Job)
  • Return type: None
• toil.test.src.resumabilityTest.goodChild(job) — does nothing 😴
  • Parameters: job (toil.job.Job, see #toil.job.Job)
  • Return type: None
• toil.test.src.resumabilityTest.badChild(job) — fails the first time it's run, succeeds the second time 🙈
  • Parameters: job (toil.job.JobFunctionWrappingJob, see #toil.job.JobFunctionWrappingJob)
  • Return type: None

---

🧚 toil.test.src.retainTempDirTest

Classes

• TestCleanWorkDir — tests testing :class:toil.fileStores.abstractFileStore.AbstractFileStore 🧹

Functions

• tempFileTestJob(job) 📝
• tempFileTestErrorJob(job) 📝

Module Contents

• toil.test.src.retainTempDirTest.TestCleanWorkDir — tests testing :class:toil.fileStores.abstractFileStore.AbstractFileStore 💅
  • testNever(tmp_path)
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
  • testAlways(tmp_path)
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
  • testOnErrorWithError(tmp_path)
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
  • testOnErrorWithNoError(tmp_path)
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
  • testOnSuccessWithError(tmp_path)
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
  • testOnSuccessWithSuccess(tmp_path)
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
• toil.test.src.retainTempDirTest.tempFileTestJob(job) 📝
  • Parameters: job (toil.job.JobFunctionWrappingJob, see #toil.job.JobFunctionWrappingJob)
  • Return type: None
• toil.test.src.retainTempDirTest.tempFileTestErrorJob(job) 📝
  • Parameters: job (toil.job.JobFunctionWrappingJob, see #toil.job.JobFunctionWrappingJob)
  • Return type: None

---

🌈 toil.test.src.systemTest

Classes

• TestSystem — tests various assumptions about the operating system's behavior 🖥️

Module Contents

• toil.test.src.systemTest.TestSystem — tests various assumptions about the operating system's behavior 🧠
  • testAtomicityOfNonEmptyDirectoryRenames(tmp_path) — checks how directory renames behave when the directory isn't empty 📂
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None

---

🦋 toil.test.src.threadingTest

Attributes

• log 📜

Classes

• TestThreading — tests Toil's threading and synchronization tools 🔒

Module Contents

• toil.test.src.threadingTest.log — the log attribute 📜
• toil.test.src.threadingTest.TestThreading — tests Toil's threading and synchronization tools 🔒
  • testGlobalMutexOrdering(tmp_path) — the global mutex ordering test, no cap 🔐
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
  • testLastProcessStanding(tmp_path) — the last-process-standing test 🏆
    • Parameters: tmp_path (pathlib.Path, see /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None

🦋 toil.test.src.toilContextManagerTest 🦋

🍓 Classes

               ┌────────────────────────┬────────────────────────────┐
               │ ToilContextManagerTest │ Legacy common base class   │
               │                        │ for Toil tests.            │
               ├────────────────────────┼────────────────────────────┤
               │ HelloWorld             │ Class represents a unit of │
               │                        │ work in toil.              │
               ├────────────────────────┼────────────────────────────┤
               │ FollowOn               │ Class represents a unit of │
               │                        │ work in toil.              │
               └────────────────────────┴────────────────────────────┘

✨ Functions ✨

                                ┌──────────────┬───┐
                                │ childFn(job) │   │
                                └──────────────┴───┘

💅 Module Contents 💅

💖 class toil.test.src.toilContextManagerTest.ToilContextManagerTest(methodName='runTest')

Bases: toil.test.ToilTest <#toil.test.ToilTest>

okay so this is the legacy common base class for Toil tests 🙈 It's giving old-school, so new tests should be made in the pytest style and not use this class, bestie 💜

When running tests you can optionally set the TOIL_TEST_TEMP environment variable to a path where you want your temp test files to live 🌸 The directory gets created if it doesn't exist yet. The path can be relative, and if it is, it's measured from the project root. If TOIL_TEST_TEMP isn't set, temp files and directories go into the system's default spot for that stuff, and any leftovers get cleaned up automatically during tear down ✨ If you do set it, left-over files stick around and won't be removed.

• setUp() — Hook method for setting up the test fixture before exercising it.
• tearDown() — Hook method for deconstructing the test fixture after testing it.
• testContextManger()
• testNoContextManger()
• testExportAfterFailedExport()

🧚 class toil.test.src.toilContextManagerTest.HelloWorld

Bases: toil.job.Job <#toil.job.Job>

Class represents a unit of work in toil. It's your little job bestie 🫶

• run(fileStore) — Override this function to perform work and dynamically create successor jobs.
  • Parameters: fileStore — used to create local and globally sharable temporary files and to send log messages to the leader process 📝
  • Returns: The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>. 🦋

✨ toil.test.src.toilContextManagerTest.childFn(job)

🌈 class toil.test.src.toilContextManagerTest.FollowOn(fileId)

Bases: toil.job.Job <#toil.job.Job>

Class represents a unit of work in toil. Another job, another vibe 💅

• fileId
• run(fileStore) — Override this function to perform work and dynamically create successor jobs.
  • Parameters: fileStore — used to create local and globally sharable temporary files and to send log messages to the leader process 📝
  • Returns: The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>. 🦋

---

🌸 toil.test.src.userDefinedJobArgTypeTest 🌸

🍓 Classes

             ┌───────────────────────────┬────────────────────────────┐
             │ UserDefinedJobArgTypeTest │ Test for issue #423 (Toil  │
             │                           │ can't unpickle classes     │
             │                           │ defined in user scripts)   │
             │                           │ and variants               │
             ├───────────────────────────┼────────────────────────────┤
             │ JobClass                  │ Class represents a unit of │
             │                           │ work in toil.              │
             ├───────────────────────────┼────────────────────────────┤
             │ Foo                       │                            │
             └───────────────────────────┴────────────────────────────┘

✨ Functions ✨

                           ┌─────────────────────────┬───┐
                           │ jobFunction(job, level, │   │
                           │ foo)                    │   │
                           ├─────────────────────────┼───┤
                           │ main()                  │   │
                           └─────────────────────────┴───┘

💅 Module Contents 💅

💜 class toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest(methodName='runTest')

Bases: toil.test.ToilTest <#toil.test.ToilTest>

This is the test for issue #423 (Toil can't unpickle classes defined in user scripts) and all the variants of that drama 😭 The issue lives here: https://github.com/BD2KGenomics/toil/issues/423

• setUp() — Hook method for setting up the test fixture before exercising it.
• testJobFunction() — Test with first job being a function.
• testJobClass() — Test with first job being an instance of a class.
• testJobFunctionFromMain() — Test with first job being a function defined in __main__.
• testJobClassFromMain() — Test with first job being an instance of a class defined in __main__.

🦋 class toil.test.src.userDefinedJobArgTypeTest.JobClass(level, foo)

Bases: toil.job.Job <#toil.job.Job>

Class represents a unit of work in toil. Main character job 👑

• level
• foo
• run(fileStore) — Override this function to perform work and dynamically create successor jobs.
  • Parameters: fileStore — used to create local and globally sharable temporary files and to send log messages to the leader process 📝
  • Returns: The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>. 🦋

✨ toil.test.src.userDefinedJobArgTypeTest.jobFunction(job, level, foo)

🌈 class toil.test.src.userDefinedJobArgTypeTest.Foo

• original_id
• assertIsCopy()

💖 toil.test.src.userDefinedJobArgTypeTest.main()

---

🧚 toil.test.src.workerTest 🧚

🍓 Classes

                     ┌─────────────┬──────────────────────────┐
                     │ WorkerTests │ Test miscellaneous units │
                     │             │ of the worker.           │
                     └─────────────┴──────────────────────────┘

💅 Module Contents 💅

🌸 class toil.test.src.workerTest.WorkerTests(methodName='runTest')

Bases: toil.test.ToilTest <#toil.test.ToilTest>

Tests miscellaneous units of the worker. Small but mighty 🔥

• setUp() — Hook method for setting up the test fixture before exercising it.
• testNextChainable() — Make sure chainable and non-chainable jobs are identified correctly. ✅

---

🌈 toil.test.utils 🌈

💅 Submodules 💅

🦋 toil.test.utils.toilDebugTest 🦋

✨ Attributes ✨

                                   ┌────────┬───┐
                                   │ logger │   │
                                   └────────┴───┘

🍓 Classes

                     ┌──────────────┬─────────────────────────┐
                     │ TestDebugJob │ Test the toil debug-job │
                     │              │ command.                │
                     └──────────────┴─────────────────────────┘

✨ Functions ✨

okay bestie, here's the squad of helper functions in this module 💅

• workflow_debug_jobstore(tmp_path)
• testJobStoreContents(tmp_path) — a test that checks toilDebugFile.printContentsOfJobStore() 🫶
• fetchFiles(symLink, jobStoreDir, outputDir) — the fn that powers testFetchJobStoreFiles() and testFetchJobStoreFilesWSymlinks() ✨
• testFetchJobStoreFiles(tmp_path) — checks toilDebugFile.fetchJobStoreFiles() with symlinks 🦋

✨ Module Contents ✨

🌸 toil.test.utils.toilDebugTest.logger

🧚 toil.test.utils.toilDebugTest.workflow_debug_jobstore(tmp_path)

• Parameters
  • tmp_path (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type
  • pathlib.Path (/usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

💜 toil.test.utils.toilDebugTest.testJobStoreContents(tmp_path)

Tests toilDebugFile.printContentsOfJobStore(), and bestie, the vibe is so specific 🙈 It runs a workflow that imports 'B.txt' and 'mkFile.py' into the jobStore. Then 'A.txt', 'C.txt', and 'ABC.txt' get created. The test makes sure all of these contents are actually found in the jobStore and printed. Main character energy ✨

• Parameters
  • tmp_path (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type
  • None

🌈 toil.test.utils.toilDebugTest.fetchFiles(symLink, jobStoreDir, outputDir)

This is the helper fn for testFetchJobStoreFiles() and testFetchJobStoreFilesWSymlinks() 🫶 It runs a workflow that imports 'B.txt' and 'mkFile.py' into the jobStore, then creates 'A.txt', 'C.txt', and 'ABC.txt'. After that, it grabs a list of those files, copies them from the jobStore into the output directory, confirms they're all there, and then deletes them. Iconic behaviour, no notes 💅

• Parameters
  • symLink (bool: /usr/share/doc/python3/html/library/functions.html#bool)
  • jobStoreDir (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • outputDir (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type
  • None

🦋 toil.test.utils.toilDebugTest.testFetchJobStoreFiles(tmp_path)

Tests toilDebugFile.fetchJobStoreFiles() symlinks, and it's giving organised ✨

• Parameters
  • tmp_path (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• Return type
  • None

👑 class toil.test.utils.toilDebugTest.TestDebugJob

Tests for the toil debug-job command 🔥

• test_run_job(tmp_path)
  • Makes sure we can use toil debug-job to try running a job in-process. Lowkey the main event 🎀
  • Parameters: tmp_path (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • Return type: None
• test_print_job_info(tmp_path)
  • Makes sure we can use --printJobInfo to pull up info on a job from a job store 💖
  • Parameters: tmp_path (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • Return type: None
• test_retrieve_task_directory(tmp_path)
  • Makes sure we can use --retrieveTaskDirectory to grab the input files for a job 🧚
  • Parameters: tmp_path (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • Return type: None

---

✨ toil.test.utils.toilKillTest ✨

🎀 Attributes

• logger
• pkg_root

🌷 Classes

• TestToilKill — a set of test cases for "toil kill" 💅
• TestToilKillWithAWSJobStore — a set of test cases for "toil kill" using the AWS job store ☁️

🫶 Module Contents

• toil.test.utils.toilKillTest.logger
• toil.test.utils.toilKillTest.pkg_root = b'.'
• class toil.test.utils.toilKillTest.TestToilKill
  • Bases: _ToilKillTest
  • test_cwl_toil_kill(tmp_path)
    • Tests "toil kill" on a CWL workflow with a 100 second sleep. Bestie, that's a whole vibe 😭
    • Parameters: tmp_path (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
    • Return type: None
• class toil.test.utils.toilKillTest.TestToilKillWithAWSJobStore
  • Bases: _ToilKillTest
  • test_cwl_toil_kill()
    • Tests "toil kill" on a CWL workflow with a 100 second sleep, but this time with the AWS job store 🌈
    • Return type: None

---

✨ toil.test.utils.utilsTest ✨

🌸 Attributes

• logger

🧚 Classes

• TestUtils — tests the utilities Toil ships with, like stats and status, working together with restart functionality 💖
• RunTwoJobsPerWorker — runs a child job with the same resources as itself, so the jobs get chained onto the same worker 🔥

🦋 Functions

• unsortedFile(tmp_path)
• correctSort(unsortedFile)
• printUnicodeCharacter()

💜 Module Contents

• toil.test.utils.utilsTest.logger
• unsortedFile(tmp_path)
  • Parameters: tmp_path (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • Return type: collections.abc.Generator (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator) of pathlib.Path (/usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
• correctSort(unsortedFile)
  • Parameters: unsortedFile (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)
  • Return type: list (/usr/share/doc/python3/html/library/stdtypes.html#list) of str (/usr/share/doc/python3/html/library/stdtypes.html#str)
• class toil.test.utils.utilsTest.TestUtils
  • Tests the utilities Toil ships with, like stats and status, in conjunction with restart functionality. The vibes are immaculate 🌈
  • N: int = 1000 (/usr/share/doc/python3/html/library/functions.html#int)
  • property toilMain: str (/usr/share/doc/python3/html/library/stdtypes.html#str)
    • Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)
  • cleanCommand(jobstore)
    • Parameters: jobstore (pathlib.Path: /usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

Okay, the function right above this one hands back a list[str], so you get a list of strings, bestie 💅

statsCommand(jobstore) 📊

This one only needs a single thing:

• jobstore (pathlib.Path)

And it gives you back a list[str]. Simple, slay 💖

statusCommand(jobstore, failIfNotComplete=False) 🦋

Two params here, and one of them has a default:

• jobstore (pathlib.Path)
• failIfNotComplete (bool)

It also returns a list[str] 🌸

test_config_functionality(tmp_path) 🧪

This test is lowkey a vibe check on the config file. It makes sure that creating the config file and reading it back actually works 🙈

• tmp_path (pathlib.Path)

Return type: None

testAWSProvisionerUtils() ☁️

This one is a whole main character moment 👑 It runs a bunch of the cluster utilities back to back, in this order:

1. Launches a cluster with custom tags.
2. Verifies the tags exist.
3. ssh's into the cluster.
4. Does some weird string comparisons 🫠
5. Makes certain that TOIL_WORKDIR is set as expected in the ssh'ed cluster.
6. Rsyncs a file and verifies it exists on the leader.
7. Destroys the cluster.

Returns: nothing documented here, just the return type below.

Return type: None

testUtilsSort(tmp_path, unsortedFile, correctSort) 🔥

This one tests the status and stats commands of the toil command line utility, using the sort example with the --restart flag. Iconic 🍓

• tmp_path (pathlib.Path)
• unsortedFile (pathlib.Path)
• correctSort (list[str])

Return type: None

testUtilsStatsSort(tmp_path, unsortedFile, correctSort) 📈

This one checks the stats commands on a complete run of the stats test. Very thorough, very on brand 💜

• tmp_path (pathlib.Path)
• unsortedFile (pathlib.Path)
• correctSort (list[str])

Return type: None

testUnicodeSupport(tmp_path) 🌈

• tmp_path (pathlib.Path)

Return type: None

testMultipleJobsPerWorkerStats(tmp_path) 🧚

This one covers the case where multiple jobs run on 1 worker. The goal is to make sure that every single job reports its data back, no one gets left on read 📬

• tmp_path (pathlib.Path)

Return type: None

check_status(jobstore, status, status_fn, process=None, seconds=20) ⏱️

Here's the full parameter lineup:

• jobstore (pathlib.Path)
• status (str)
• status_fn (collections.abc.Callable[[str], str])
• process (Optional[subprocess.Popen[Any]])
• seconds (int), which defaults to 20

Return type: None

testGetPIDStatus(tmp_path, unsortedFile) 🆔

This test makes sure that ToilStatus.getPIDStatus() behaves the way it's supposed to 🫶

• tmp_path (pathlib.Path)
• unsortedFile (pathlib.Path)

Return type: None

testGetStatusFailedToilWF(tmp_path, unsortedFile) 💀

This one checks that ToilStatus.getStatus() acts right with a failing Toil workflow. Now, you could totally call this workflow by importing it and calling its main function. But if you did that, you'd lose the chance to test the 'RUNNING' functionality of getStatus(), and that's the whole point 😤

• tmp_path (pathlib.Path)
• unsortedFile (pathlib.Path)

Return type: None

testGetStatusFailedCWLWF(tmp_path) 🔥

Same energy as the one above, but this time it checks that ToilStatus.getStatus() behaves with a failing CWL workflow.

• tmp_path (pathlib.Path)

Return type: None

testGetStatusSuccessfulCWLWF(tmp_path) ✨

The happy path! This checks that ToilStatus.getStatus() behaves as expected with a successful CWL workflow 🎀

• tmp_path (pathlib.Path)

Return type: None

testPrintJobLog(tmp_path, monkeypatch) 📜

This one tests that ToilStatus.printJobLog() can read the log from a failed command without throwing a fit 💅

• tmp_path (pathlib.Path)
• monkeypatch (pytest.MonkeyPatch)

Return type: None

testRestartAttribute(tmp_path) 🦋

Okay, this one is the most dramatic test in the bunch 😭 The job store should only get destroyed once we actually see a successful workflow run. So the test simulates a workflow that fails and then tries to resume without restart(). In that situation, the job store has to stick around until restart() is called. No early deletions, no cap.

• tmp_path (pathlib.Path)

Return type: None

toil.test.utils.utilsTest.printUnicodeCharacter() 🌸

This function is all about the unicode vibes ✨

Return type: None

class toil.test.utils.utilsTest.RunTwoJobsPerWorker 🧃

class toil.test.utils.utilsTest.RunTwoJobsPerWorker(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)

Bases: toil.job.Job

This class runs a child job with the same resources as itself, in an attempt to chain the jobs onto the same worker. It's giving teamwork 🫶

Here's everything it takes in:

• memory (Optional[ParseableIndivisibleResource])
• cores (Optional[ParseableDivisibleResource])
• disk (Optional[ParseableIndivisibleResource])
• accelerators (Optional[ParseableAcceleratorRequirement])
• preemptible (Optional[ParseableFlag])
• preemptable (Optional[ParseableFlag])
• unitName (Optional[str])
• checkpoint (Optional[bool])
• displayName (Optional[str])
• descriptionClass (Optional[type])
• local (Optional[bool])
• files (Optional[set[toil.fileStores.FileID]])

run(fileStore) 🌟

This is the method you override when you want to do the actual work and dynamically spin up successor jobs. Bestie, this is where the magic happens ✨

• fileStore (toil.fileStores.abstractFileStore.AbstractFileStore) — this is what you use to create local and globally sharable temporary files, and to send log messages to the leader process 📬

Okay, that's the whole vibe for this part! 💜

• Returns — okay so the return value of the function is giving handoff energy 🎀. You can pass it along to other jobs with toil.job.Job.rv() ✨, no cap.
• Return type — None 🙈

---

🌸 toil.test.wdl

🧚 Submodules

• toil.test.wdl.wdltoil_test

---

🌸 toil.test.wdl.wdltoil_test

🔥 Attributes

• logger
• WDL_CONFORMANCE_TEST_REPO
• WDL_CONFORMANCE_TEST_COMMIT
• WDL_CONFORMANCE_TESTS_UNSUPPORTED_BY_TOIL
• WDL_UNIT_TESTS_UNSUPPORTED_BY_TOIL

👑 Classes

• TestWDLConformance — WDL conformance tests for Toil. 🫶
• TestWDL — Tests for Toil's MiniWDL-based implementation. 💅
• TestWDLToilBench — Tests for Toil's MiniWDL-based implementation that don't run workflows. 🧪

💜 Functions

• wdl_conformance_test_repo(tmp_path)

🌈 Module Contents

The logger, just vibing:

toil.test.wdl.wdltoil_test.logger

The constants are the main characters here, so here they are exactly as written:

toil.test.wdl.wdltoil_test.WDL_CONFORMANCE_TEST_REPO = 'https://github.com/DataBiosphere/wdl-conformance-tests.git'
toil.test.wdl.wdltoil_test.WDL_CONFORMANCE_TEST_COMMIT = '46b5f85ee38ec60d0b8b9c35928b5104a2af83d5'
toil.test.wdl.wdltoil_test.WDL_CONFORMANCE_TESTS_UNSUPPORTED_BY_TOIL = [16, 21, 64, 77]
toil.test.wdl.wdltoil_test.WDL_UNIT_TESTS_UNSUPPORTED_BY_TOIL = [14, 19, 52, 58, 59, 66, 67, 68, 69, 87, 97, 105, 107, 108, 109, 110, 120, 131, 134, 144]

Now the helper function. It's giving fixture energy 🧚:

• toil.test.wdl.wdltoil_test.wdl_conformance_test_repo(tmp_path) — yep, a fixture 🌸
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • collections.abc.Generator[pathlib.Path]

💖 class toil.test.wdl.wdltoil_test.TestWDLConformance

WDL conformance tests for Toil. The whole class is a total slay 💅

• check(p) — make sure a call finished, or explain why it didn't 🙈
  • Parameters
    • p (subprocess.CompletedProcess[bytes])
  • Return type
    • None
• test_unit_tests_v11(wdl_conformance_test_repo) 🦋
  • Parameters
    • wdl_conformance_test_repo (pathlib.Path)
  • Return type
    • None
• test_conformance_tests_v10(wdl_conformance_test_repo) 🦋
  • Parameters
    • wdl_conformance_test_repo (pathlib.Path)
  • Return type
    • None
• test_conformance_tests_v11(wdl_conformance_test_repo) 🦋
  • Parameters
    • wdl_conformance_test_repo (pathlib.Path)
  • Return type
    • None
• test_conformance_tests_development(wdl_conformance_test_repo) 🦋
  • Parameters
    • wdl_conformance_test_repo (pathlib.Path)
  • Return type
    • None
• test_conformance_tests_integration(wdl_conformance_test_repo) 🦋
  • Parameters
    • wdl_conformance_test_repo (pathlib.Path)
  • Return type
    • None

💜 class toil.test.wdl.wdltoil_test.TestWDL

Tests for Toil's MiniWDL-based implementation. Main character behavior, bestie 👑

• base_command — the attribute, no extra notes needed 🍓
• test_MD5sum(tmp_path) — checks that Toil gives the same outputs as the known good ones from WDL's GATK tutorial #1 🧚
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_url_to_file(tmp_path) — checks that web URL strings can be turned into usable Files 🌈
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_string_file_coercion(tmp_path) — checks that input Files can be turned into strings and back again 🫶
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_gather(tmp_path) — checks files that share a name but come from different scatter tasks 🦋
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_wait(tmp_path) — checks whether Bash wait works inside WDL scripts ⏳
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_restart(tmp_path) — checks that a WDL workflow can be restarted and still finish successfully 🔁
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_workflow_file_deletion(tmp_path) — checks whether Toil can delete non-output outputs at the very end of a workflow 🗑️
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_all_call_outputs(tmp_path) — checks whether Toil can collect all call outputs from a workflow that doesn't expose them 📦
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_croo_detection(tmp_path) — checks whether Toil can detect Cromwell Output Organizer workflows and do something sensible with them 🧠
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_caching(tmp_path) — checks whether Toil can cache task runs 💾
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None
• test_url_to_optional_file(tmp_path, httpserver) — checks that missing and error-producing URLs are handled correctly for optional File? values 🙈
  • Parameters
    • tmp_path (pathlib.Path)
    • httpserver (pytest_httpserver.HTTPServer)
  • Return type
    • None
• test_missing_output_directory(tmp_path) — checks whether Toil can run a WDL workflow into a new directory 🌸
  • Parameters
    • tmp_path (pathlib.Path)
  • Return type
    • None

✨ test_miniwdl_self_test(tmp_path, extra_args=None) ✨

okay so this one checks that the MiniWDL self test actually runs and spits out the output it's supposed to 🧪💖

• Parameters
  • tmp_path (pathlib.Path)
  • extra_args (Optional[list[str]])
• Return type: None

✨ test_miniwdl_self_test_by_reference(tmp_path) ✨

bestie, this one makes sure the MiniWDL self test still works when the input files come in as URL references instead of local copies 🔗🦋

• Parameters
  • tmp_path (pathlib.Path)
• Return type: None

✨ test_dockstore_trs(tmp_path, extra_args=None) ✨

this one runs the Dockstore TRS check 🌈

• Parameters
  • tmp_path (pathlib.Path)
  • extra_args (Optional[list[str]])
• Return type: None

✨ test_dockstore_metrics_publication(tmp_path, extra_args=None) ✨

and this one checks that Dockstore metrics publication works 📊🔥

• Parameters
  • tmp_path (pathlib.Path)
  • extra_args (Optional[list[str]])
• Return type: None

✨ test_giraffe_deepvariant(tmp_path) ✨

this is the big one, so bring snacks 🍿😭 It checks that Giraffe and GPU DeepVariant both run. Heads up, this could take 25 minutes ⏰

• Parameters
  • tmp_path (pathlib.Path)
• Return type: None

✨ test_giraffe(tmp_path) ✨

checks that Giraffe runs 🦒 Fair warning, this could take 12 minutes. We scale it down, but it still needs lots of memory, so it's giving memory-hungry 🧠💀

• Parameters
  • tmp_path (pathlib.Path)
• Return type: None

✨ test_gs_uri(tmp_path) ✨

this one makes sure Toil can actually reach Google Storage URIs ☁️💜

• Parameters
  • tmp_path (pathlib.Path)
• Return type: None

✨ test_check(tmp_path) ✨

and last but not least, this one checks that Toil's lint check works 🧹✨

• Parameters
  • tmp_path (pathlib.Path)
• Return type: None

---

✨ class toil.test.wdl.wdltoil_test.TestWDLToilBench(methodName='runTest') ✨

Bases: unittest.TestCase

this class is the MiniWDL-based implementation tests that don't actually run workflows 🎀 Basically the no-workflow-needed squad 💅

✨ test_coalesce() ✨

checks whether WDLSectionJob can coalesce WDL decls 🧩

This is a white box test, so it'll need to be changed or removed if the WDL interpreter changes. Heads up, bestie 👀

• Return type: None

✨ make_string_expr(to_parse, expr_type=WDL.Expr.String) ✨

parses pseudo-WDL so we can test whitespace removal 🫧

• Parameters
  • to_parse (str)
  • expr_type (type[WDL.Expr.String])
• Return type: WDL.Expr.String

✨ test_choose_human_readable_directory() ✨

makes sure we pick sensible directories for files, and that they don't collide with each other 📁🌸

• Return type: None

✨ test_uri_packing() ✨

makes sure Toil's URI packing carries through all the info it needs 🎁

• Return type: None

✨ test_disk_parse() ✨

makes sure the disk parsing is correct 💾✨

• Return type: None

---

toil.test.wdl.wdltoil_test_kubernetes

Attributes

• logger

Classes

• WDLKubernetesClusterTest — ensures WDL works on the Kubernetes batchsystem 🧚

Module Contents

toil.test.wdl.wdltoil_test_kubernetes.logger

✨ class toil.test.wdl.wdltoil_test_kubernetes.WDLKubernetesClusterTest(name) ✨

Bases: toil.test.provisioners.clusterTest.AbstractClusterTest

this is the Kubernetes batchsystem check, and it makes sure WDL works there 👑

• Parameters
  • name (str)

Here are the class attributes:

• clusterName = 'wdl-integration-test-'
• leaderNodeType = 't2.medium'
• instanceTypes = ['t2.medium']
• clusterType = 'kubernetes'

setUp()

set-up for the test 🛠️ You have to override this and call this method, and you also have to set self.jobStore, or it won't work 🙈

• Return type: None

launchCluster()

launches the cluster 🚀

• Return type: None

test_wdl_kubernetes_cluster()

checks that a WDL workflow works on a Kubernetes cluster 🎀 It launches a cluster with 1 worker, then runs a WDL workflow that does an image pull on that worker 🐳 :return:

• Return type: None

---

Attributes

• logger 📝
• MT 🧁
• pneeds_rsync3 🔄
• pneeds_online 🌐
• pneeds_aws_s3 🪣
• pneeds_aws_ec2 💻
• pneeds_gridengine 🍓
• pneeds_torque 🌪️
• pneeds_kubernetes ☸️
• pneeds_mesos 🌿
• pneeds_slurm 🐌
• pneeds_lsf 🧃
• pneeds_docker 🐳
• pneeds_local_cuda 🟢
• pneeds_docker_cuda 🟩
• pneeds_cwl 📜
• pneeds_wdl 🧬
• pneeds_wes_server 🖥️
• pintegrative 🫶
• pslow 🐢
• methodNamePartRegex 🔍

✨ Exceptions ✨

okay so this is where the error classes live 💀 bestie, these are the ones you'll want to know about when things go sideways 🫶

• ProxyConnectionError — 🚨 the common base class for all exceptions, so it's the big-picture one that everything else in the family is built on 👑

🎀 Classes 🎀

these are the test classes, and they're giving old school meets new school 🦋

• ToilTest — 🕰️ the legacy common base class for Toil tests, so it's the OG that a lot of testing stuff grew out of 💜
• ApplianceTestSupport — 🐳 a Toil test that runs a user script on a minimal cluster of appliance containers, which is lowkey iconic for testing a whole workflow in a tiny mini-cluster 🧚

✨ Functions ✨

okay so this is the little toolbox of test helpers 🧰 bestie, and these are the ones you slap on your tests to decide when they're allowed to run 💅

• get_data(filename) — gives you back an absolute path 🗺️ to a file that lives inside this package
• get_temp_file([suffix, rootDir]) — hands you a string representing a temporary file 🫶 but ngl, YOU have to delete it yourself because it won't clean up after you 💀

The needs_* ones are the gatekeepers 🚪✨ each one goes above test classes or methods and only lets them run when their requirement is there.

• needs_env_var(var_name[, comment]) — use it as a decorator 🌸 so the test only runs if the given …
• needs_rsync3(test_item) — decorate any classes or methods that depend on features from rsync version 3.0.0+ 🦋
• needs_online(test_item) — use it as a decorator 💜 to run only if we're meant to talk to the Internet 🌈
• needs_aws_s3(test_item) — use it as a decorator 🪣 to run only if AWS S3 is usable
• needs_aws_ec2(test_item) — use it as a decorator ☁️ to run only if AWS EC2 is usable
• needs_aws_batch(test_item) — use it as a decorator 🍓 to run only if AWS Batch…
• needs_google_storage(test_item) — use it as a decorator 🫧 to run only if Google…
• needs_google_project(test_item) — use it as a decorator 💖 to run only if we have a Google Cloud project set
• needs_gridengine(test_item) — use it as a decorator 🧚 to run only if GridEngine is installed
• needs_torque(test_item) — use it as a decorator 🌸 to run only if PBS/Torque is installed
• needs_kubernetes_installed(test_item) — use it as a decorator ⭐ to run only if Kubernetes is installed
• needs_kubernetes(test_item) — use it as a decorator 👑 to run only if Kubernetes is installed and configured
• needs_mesos(test_item) — use it as a decorator 🔥 to run only if Mesos is installed
• needs_slurm(test_item) — use it as a decorator ✨ to run only if Slurm is installed
• needs_htcondor(test_item) — use a decorator 🦋 before test classes or methods to run only if the HTCondor is installed
• needs_lsf(test_item) — use it as a decorator 🎀 to only run them if LSF is installed
• needs_java(test_item) — use it as a test decorator ☕ to run only if java is installed
• needs_docker(test_item) — use it as a decorator 🐳 to only run them if…
• needs_singularity(test_item) — use it as a decorator 🫶 to only run them if…
• needs_singularity_or_docker(test_item) — use it as a decorator 💅 to only run them if…
• needs_local_cuda(test_item) — use it as a decorator 🙈 before test classes or methods to…

heads up bestie 🥺 the original page cuts off mid-sentence on a bunch of these (needs_env_var, needs_aws_batch, needs_google_storage, needs_docker, needs_singularity, needs_singularity_or_docker, needs_local_cuda), so I kept them exactly as they end. no guessing, no cap 🫶

…okay so the row above wraps up with test classes or methods to only run them if 👀

• needs_docker_cuda(test_item) — slap this decorator on test classes or methods and they'll only run them if 🎮 (the original cuts off right there, bestie 💀)
• needs_encryption(test_item) — decorator that only lets tests run if PyNaCl is installed 🔐
• needs_cwl(test_item) — decorator for test classes or methods that only runs them if CWLTool is installed 🧩
• needs_wdl(test_item) — decorator for test classes or methods that only runs them if miniwdl is installed 🌊
• needs_server(test_item) — decorator for test classes or methods that only runs them if Connexion is installed. 🛰️
• needs_celery_broker(test_item) — decorator for test classes or methods that runs them only if RabbitMQ is set up to take Celery jobs 🐇
• needs_wes_server(test_item) — decorator for test classes or methods that runs them only if a WES (the page cuts off here too 😭)
• needs_local_appliance(test_item) — decorator for test classes or methods that only runs them if (and yep, it ends there 🫠)
• needs_fetchable_appliance(test_item) — decorator for test classes or methods that only runs them if (same story, cut off 🫠)
• integrative(test_item) — use this to decorate integration tests so they get skipped during regular builds 🙈
• slow(test_item) — use this decorator to flag tests that are slow and not critical 🐢
• timeLimit(seconds) — use this to cap how long a function is allowed to run ⏱️
• make_tests(generalMethod, targetClass, **kwargs) — this one dynamically generates test methods, using generalMethod as a template. Each (and that's where the original stops 💅)

---

📦 Package Contents

💥 exception toil.test.ProxyConnectionError

• Bases: BaseException (/usr/share/doc/python3/html/library/exceptions.html#BaseException)
• It's the common base class for all exceptions, so it's giving foundation vibes 🧱

📝 toil.test.logger

• Just the name, no extra details, bestie 🤷‍♀️

🗂️ toil.test.get_data(filename)

• Returns an absolute path for a file from this package 🗺️
• Parameters
  • filename (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type
  • contextlib.AbstractContextManager (/usr/share/doc/python3/html/library/contextlib.html#contextlib.AbstractContextManager) wrapping pathlib.Path (/usr/share/doc/python3/html/library/pathlib.html#pathlib.Path)

🧪 class toil.test.ToilTest(methodName='runTest')

• Bases: unittest.TestCase (/usr/share/doc/python3/html/library/unittest.html#unittest.TestCase)
• This is the legacy common base class for Toil tests. New tests should be written in the 'pytest' style and not use this class. 🙅‍♀️
• When running tests you can optionally set the TOIL_TEST_TEMP environment variable to a directory where you want temporary test files to go 📂. The directory gets created if it doesn't exist. A relative path is treated as relative to the project root. If TOIL_TEST_TEMP isn't defined, temp files and directories land in the system's default location, and any leftovers from tests get removed automatically during tear down 🧹. If you did set it, leftover files will not be removed.

setup_method(method)

• Parameters
  • method (Any)
• Return type
  • None

classmethod setUpClass()

• Hook method for setting up the class fixture before running the tests in the class 🛠️
• Return type
  • None

classmethod tearDownClass()

• Hook method for deconstructing the class fixture after running all the tests in the class 🧽
• Return type
  • None

setUp()

• Hook method for setting up the test fixture before exercising it 🎬
• Return type
  • None

tearDown()

• Hook method for deconstructing the test fixture after testing it 🧼
• Return type
  • None

classmethod awsRegion()

• Picks an appropriate AWS region. It uses us-west-2 unless you're running on EC2, in which case it uses the region where the instance is located 🌎
• Return type
  • str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🌀 toil.test.MT

• Just the name again, nothing else to see here 🫶

🎟️ toil.test.get_temp_file(suffix='', rootDir=None)

• Returns a string representing a temporary file. You have to delete it manually 🗑️
• Parameters
  • suffix (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
  • rootDir (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type
  • str (/usr/share/doc/python3/html/library/stdtypes.html#str)

🌱 toil.test.needs_env_var(var_name, comment=None)

• Use as a decorator before test classes or methods to run them only if the given environment variable is set. You can add a comment saying what the variable should be set to 💡
• Parameters
  • var_name (str, /usr/share/doc/python3/html/library/stdtypes.html#str)
  • comment (Optional[str], /usr/share/doc/python3/html/library/stdtypes.html#str)
• Return type
  • Callable[[MT], MT]

🦋 toil.test.needs_rsync3(test_item)

• Decorate classes or methods that depend on any features from rsync version 3.0.0+ 🐢✨
• It's necessary because utilsTest.testAWSProvisionerUtils() uses the option --protect-args, which is only available in rsync 3 🫠

Okay so bestie, this is the part where we slap decorators on our tests and they only run when the right stuff is actually installed 💅🧚

toil.test.needs_rsync3(test_item)

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_rsync3 🌸 (listed on its own, no description given)

toil.test.needs_online(test_item)

Use this as a decorator above test classes or methods and they'll only run if we're meant to talk to the Internet 🌐✨

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_aws_s3(test_item)

Decorator alert 🚨 Put this on your tests and they'll only run if AWS S3 is usable. It's giving storage vibes 🪣

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_aws_s3 🦋 (listed on its own, no description given)

toil.test.needs_aws_ec2(test_item)

Same energy as S3, but make it EC2. These tests only run if AWS EC2 is usable 🖥️💜

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_aws_ec2 🦋 (listed on its own, no description given)

toil.test.needs_aws_batch(test_item)

Only runs if AWS Batch is usable. Batch-processing iconic behavior 👑

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_google_storage(test_item)

This one's a whole mood 🌈 It only runs if Google Cloud is installed AND we ought to be able to reach public Google Storage URIs. Lowkey the most specific check ever 😭

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_google_project(test_item)

Only runs if we have a Google Cloud project set. No project, no party 🎀

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_gridengine(test_item)

Only runs if GridEngine is installed. Literally the vibe check for schedulers 🔥

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_gridengine 🌸 (listed on its own, no description given)

toil.test.needs_torque(test_item)

Only runs if PBS/Torque is installed. Ate, left no crumbs 💅

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_torque 🌸 (listed on its own, no description given)

toil.test.needs_kubernetes_installed(test_item)

Only runs if Kubernetes is installed. That's it, that's the check 🧚

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_kubernetes(test_item)

A step up from the last one: these only run if Kubernetes is installed AND configured. Main character energy 👑

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_kubernetes 🌸 (listed on its own, no description given)

toil.test.needs_mesos(test_item)

Only runs if Mesos is installed. Okay, iconic 🌸

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_mesos 🌸 (listed on its own, no description given)

toil.test.needs_slurm(test_item)

Only runs if Slurm is installed. Another HPC queen showing up 👸💜

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_slurm 🌸 (listed on its own, no description given)

toil.test.needs_htcondor(test_item)

Use a decorator here (yes, just "a" decorator 🙈) and the tests only run if HTCondor is installed.

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_lsf(test_item)

Put this on classes or methods and they'll only run if LSF is installed 🦋

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_lsf 🌸 (listed on its own, no description given)

toil.test.needs_java(test_item)

Use this as a test decorator and it runs only if java is installed. Simple, no drama 🫶

• Parameters: test_item (MT)
• Return type: MT

---

toil.test.needs_docker(test_item)

Only runs if docker is installed AND docker-based tests are enabled. Both boxes gotta be checked, bestie 🐳

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_docker 🌸 (listed on its own, no description given)

toil.test.needs_singularity(test_item)

Only runs if singularity is installed. Container energy 📦✨

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_singularity_or_docker(test_item)

Lowkey the flexible one 🦋 These run if docker is installed AND docker-based tests are enabled, OR if Singularity is installed. Either one is giving enough.

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_local_cuda(test_item)

Only runs if there's a CUDA setup that cwltool can read (i.e. it's providing userspace nvidia-smi). If it's there, we're good 💚

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_local_cuda 🌸 (listed on its own, no description given)

toil.test.needs_docker_cuda(test_item)

Only runs if a CUDA setup is available through Docker. GPU slay through containers 🎮

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_docker_cuda 🌸 (listed on its own, no description given)

toil.test.needs_encryption(test_item)

Only runs if PyNaCl is installed and configured. Locked and loaded 🔐

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_cwl(test_item)

Only runs if CWLTool is installed and configured. CWL besties, this one's for you 🫶

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_cwl 🌸 (listed on its own, no description given)

toil.test.needs_wdl(test_item)

Only runs if miniwdl is installed and configured. WDL gang, you're up 🧚

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_wdl 🌸 (listed on its own, no description given)

toil.test.needs_server(test_item)

Only runs if Connexion is installed. Server stuff, sorted ✨

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_celery_broker(test_item)

Use this to run only if RabbitMQ is set up to take Celery jobs. Bunny and celery, a power couple 🐰🥬

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_wes_server(test_item)

Only runs if a WES server is available to run against. No server, no test, periodt 🙅‍♀️

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pneeds_wes_server 🌸 (listed on its own, no description given)

toil.test.needs_local_appliance(test_item)

Only runs if the Toil appliance Docker image is downloaded. We love a local copy 💖

• Parameters: test_item (MT)
• Return type: MT

toil.test.needs_fetchable_appliance(test_item)

Only runs if the Toil appliance Docker image is able to be downloaded from the Internet. Not downloaded yet? Doesn't matter as long as it's fetchable 🌐

• Parameters: test_item (MT)
• Return type: MT

---

toil.test.integrative(test_item)

Use this to decorate integration tests so they get skipped during regular builds 🙈 Lowkey the best way to keep builds snappy.

We call something an integration test if A) it involves other, non-Toil software components that we develop and/or B) it has a higher cost, whether that's time or money 💸

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pintegrative 🌸 (listed on its own, no description given)

toil.test.slow(test_item)

Use this decorator to flag tests that are slow and not critical. These get skipped if TOIL_TEST_QUICK is true 🐢💜

• Parameters: test_item (MT)
• Return type: MT
• toil.test.pslow 🌸 (listed on its own, no description given)
• toil.test.methodNamePartRegex 🌸 (listed on its own, no description given)

toil.test.timeLimit(seconds)

Use this to limit how long a function is allowed to run ⏳ If the function takes more than the specified amount of time, it raises an exception. Credit where it's due: the approach is from http://stackoverflow.com/a/601168.

• Parameters: seconds (int) (/usr/share/doc/python3/html/library/functions.html#int) — maximum allowable time, in seconds
• Return type: collections.abc.Generator[None, None, None] (/usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator)

Watch this 👀

>>> import time
>>> with timeLimit(2):
...    time.sleep(1)
>>> import time
>>> with timeLimit(1):
...    time.sleep(2)
Traceback (most recent call last):
    ...
RuntimeError: Timed out

The first one finishes in time, so it's all good ✨ The second one takes too long and gets a RuntimeError: Timed out 💀

toil.test.make_tests(generalMethod, targetClass, **kwargs)

This one's a test-generating machine 🏭 It builds test methods dynamically, using generalMethod as the template. Every generated function is one unique combo of parameters applied to generalMethod, and each parameter gets a matching string that names the method. The generated functions follow the naming scheme test_[generalMethodName]___[firstParamaterName]_[someValueName]__[secondParamaterName]_...

The arguments after generalMethodName should be one or more dictionaries shaped like {str : type, ...}, where the key is the name of the value. Those names show up in the method name to represent which values got passed for each parameter in generalMethod.

The generated method names list parameters in lexicographic order by parameter name, so they come out tidy 🧹💅

• Parameters:
  • generalMethod (Callable[[Any], Any]) — A method that will be parameterized with values passed as kwargs. Note that the generalMethod must be a regular method 🫶
• targetClass (Optional[Callable[[Any], Any]]) — 🎯 This is the class the generated test methods get bound to. If you don't pass one, the class of the generalMethod is assumed to be the target.
• kwargs (Any) — a series of dictionaries that define values and their names, where each keyword is the name of a parameter in generalMethod. 💖

Return type: None 💅

>>> class Foo:
...     def has(self, num, letter):
...         return num, letter
...
...     def hasOne(self, num):
...         return num

>>> class Bar(Foo):
...     pass

>>> make_tests(Foo.has, Bar, num={'one':1, 'two':2}, letter={'a':'a', 'b':'b'})

>>> b = Bar()

Okay so, num comes lexicographically before letter, which is why it shows up first in the generated method names. 👀

>>> assert b.test_has__letter_a__num_one() == b.has(1, 'a')

>>> assert b.test_has__letter_b__num_one() == b.has(1, 'b')

>>> assert b.test_has__letter_a__num_two() == b.has(2, 'a')

>>> assert b.test_has__letter_b__num_two() == b.has(2, 'b')

>>> f = Foo()

>>> hasattr(f, 'test_has__num_one__letter_a')  # should be false because Foo has no test methods
False

---

🦋 class toil.test.ApplianceTestSupport(methodName='runTest')

Bases: ToilTest 🌸

A Toil test that runs a user script on a minimal cluster of appliance containers. i.e. one leader container and one worker container. ✨

🧵 class Appliance(outer, mounts, cleanMounts=False)

Bases: toil.lib.threading.ExceptionalThread 🫶

This is a thread whose join() method re-raises exceptions that happened during run(). join() is idempotent, but the exception only gets raised during the first join() call that actually manages to join the thread. If join() times out, no exception gets re-raised, even if one might already have happened in run(). 😭

When you subclass this thread, override tryRun() instead of run(). 💜

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Parameters 🎀

• outer (ApplianceTestSupport)
• mounts (dict[str, str])
• cleanMounts (bool)

Attributes 🧚

• lock
• outer
• mounts
• cleanMounts = False
• containerName = ''
• popen: subprocess.Popen[bytes] | None = None

Methods 💅

• __enter__() — Return type: typing_extensions.Self.
• __exit__(exc_type, exc_val, exc_tb) — Parameters:
  • exc_type (type[BaseException])
  • exc_val (Exception)
  • exc_tb (Any)

Return type: Literal[False].

• tryRun() — Return type: None. This is the one you override when subclassing. 🔥
• runOnAppliance(*args, **kwargs) — Parameters:
  • args (str)
  • kwargs (Any)

Return type: None.

• writeToAppliance(path, contents) — Parameters:
  • path (str)
  • contents (Any)

Return type: None.

• deployScript(path, packagePath, script) — Deploys a Python module on the appliance. ✨ Parameters:
  • path (str) — the path (absolute, or relative to the WORDIR of the appliance container) to the root of the package hierarchy where the given module should be placed. The given directory should be on the Python path.
  • packagePath (str) — the desired fully qualified module name (dotted form) of the module.
  • script (str|callable) — the contents of the Python module. If a callable is given, its source code will be extracted. This is a convenience that lets you embed user scripts into test code as nested functions. 🧸

Return type: None.

🌈 class LeaderThread(outer, mounts, cleanMounts=False)

Bases: Appliance 👑

This is a thread whose join() method re-raises exceptions that happened during run(). join() is idempotent, but the exception only gets raised during the first join() call that actually manages to join the thread. If join() times out, no exception gets re-raised, even if one might already have happened in run(). 😭

When you subclass this thread, override tryRun() instead of run(). 💜

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Parameters 🎀

• outer (ApplianceTestSupport)
• mounts (dict[str, str])
• cleanMounts (bool)

🍓 class WorkerThread(outer, mounts, numCores)

Bases: Appliance 🌸

This is a thread whose join() method re-raises exceptions that happened during run(). join() is idempotent, but the exception only gets raised during the first join() call that actually manages to join the thread. If join() times out, no exception gets re-raised, even if one might already have happened in run(). 😭

When you subclass this thread, override tryRun() instead of run(). 💜

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Parameters 🎀

• outer (ApplianceTestSupport)
• mounts (dict[str, str])
• numCores (int)

Attributes 🧚

• numCores

💖 toil.toilState

✨ Attributes ✨

• logger 🪵 (the one attribute listed here)

🎀 Classes 🎀

• ToilState — holds the leader's scheduling information 💅

🦋 Module Contents 🦋

toil.toilState.logger

class toil.toilState.ToilState(jobStore)

Okay so ToilState is the leader's scheduling brain 🧠, but it only keeps the stuff that does not need to be persisted back to the JobStore (like info on completed and outstanding predecessors) 🫶

It holds the true single copies of all the JobDescription objects that the Leader and ServiceManager use. The leader and service manager shouldn't do their own load() and update() calls on the JobStore. They go through this class, bestie 💅

Everything in the leader references JobDescriptions by ID 🆔. It only holds JobDescription objects, not Job objects, and those JobDescription objects only exist in single copies ✨

Parameters:

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Attributes:

• bus — listed with no type or description, so it's a total mystery 🕵️‍♀️
• successor_to_predecessors — dict[str, set[str]] 🌸
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#set
• successorCounts — dict[str, int] 🔢
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/functions.html#int
• service_to_client — dict[str, str] 🧚
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str
• servicesIssued — dict[str, set[str]] 🎀
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/stdtypes.html#set
• jobs_issued — set[str] 🦋
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#set, /usr/share/doc/python3/html/library/stdtypes.html#str
• totalFailedJobs — set[str] 💀
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#set, /usr/share/doc/python3/html/library/stdtypes.html#str
• hasFailedSuccessors — set[str] 🙈
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#set, /usr/share/doc/python3/html/library/stdtypes.html#str
• failedSuccessors — set[str] 😭
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#set, /usr/share/doc/python3/html/library/stdtypes.html#str
• jobsToBeScheduledWithMultiplePredecessors — set[str] 🔥
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#set, /usr/share/doc/python3/html/library/stdtypes.html#str

Methods:

load_workflow(rootJob, jobCache=None)

load_workflow(rootJob, jobCache=None)

Loads the whole workflow rooted at the given job 🌳. If any jobs get loaded that have updated and need the leader to deal with them, JobUpdatedMessage messages get sent to the message bus 📨

The jobCache is a map from jobStoreID to JobDescription or None. It's used to speed up building the state when loading initially from the JobStore, and it is not preserved afterwards 🏃‍♀️

• rootJob (toil.job.JobDescription) — the description for the root job of the workflow being run
• jobCache (Optional[dict[str, toil.job.JobDescription]]) — a dict to cache downloaded job descriptions in, keyed by ID
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str

Return type: None

job_exists(job_id)

job_exists(job_id)

Checks if the given job exists right now 👀. It returns True if the job exists right now, and False if it hasn't been created or it has been deleted elsewhere.

Heads up: it doesn't guarantee the job will or won't be gettable, especially if you're racing another process or if it's still cached 🙈

• job_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type: bool

• docs: /usr/share/doc/python3/html/library/functions.html#bool

get_job(job_id)

get_job(job_id)

Gets the one true copy of the JobDescription with the given ID 👑

• job_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type: toil.job.JobDescription

commit_job(job_id)

commit_job(job_id)

Saves back any modifications made to a JobDescription (one retrieved from get_job()) 💾

• job_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type: None

delete_job(job_id)

delete_job(job_id)

Destroys a JobDescription 💥. This may raise an exception if the job could not be cleaned up (i.e. files belonging to it failed to delete).

• job_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type: None

reset_job(job_id)

reset_job(job_id)

Discards any local modifications to a JobDescription 🧹. This will make modifications from other hosts visible.

• job_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type: None

reset_job_expecting_change(job_id, timeout)

reset_job_expecting_change(job_id, timeout)

Discards any local modifications to a JobDescription 🧹, and makes modifications from other hosts visible. It will wait for up to timeout seconds for a modification (or deletion) from another host to actually become visible ⏳

It always replaces the JobDescription with what is stored in the job store, even if no modification ends up being visible. It returns True if an update was detected in time, and False otherwise.

• job_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str
• timeout (float)
  • docs: /usr/share/doc/python3/html/library/functions.html#float

Return type: bool

• docs: /usr/share/doc/python3/html/library/functions.html#bool

successors_pending(predecessor_id, count)

successors_pending(predecessor_id, count)

Remembers that the given job has the given number more pending successors 📈 (ones that have not yet succeeded or failed).

• predecessor_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str
• count (int)
  • docs: /usr/share/doc/python3/html/library/functions.html#int

Return type: None

successor_returned(predecessor_id)

successor_returned(predecessor_id)

Remembers that the given job has one fewer pending successor 📉, because one has either succeeded or failed.

• predecessor_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type: None

count_pending_successors(predecessor_id)

count_pending_successors(predecessor_id)

Counts the number of pending successors of the given job 🔢. Pending successors are the ones that have not yet succeeded or failed.

• predecessor_id (str)
  • docs: /usr/share/doc/python3/html/library/stdtypes.html#str

Return type: int

• docs: /usr/share/doc/python3/html/library/functions.html#int

---

🌸 toil.utils

🧚 Submodules 🧚

🌈 toil.utils.toilClean

Delete a job store used by a previous Toil workflow invocation 🧹

✨ Attributes ✨

• logger 🪵

🎀 Functions 🎀

• main()

🦋 Module Contents 🦋

toil.utils.toilClean.logger

toil.utils.toilClean.main()

Return type: None

---

🌈 toil.utils.toilConfig

Create a config file with all default Toil options ⚙️

✨ Attributes ✨

• logger 🪵

🎀 Functions 🎀

• main()

🦋 Module Contents 🦋

toil.utils.toilConfig.logger

toil.utils.toilConfig.main()

Return type: None

---

🌈 toil.utils.toilDebugFile

Debug tool for copying files contained in a toil jobStore 🐛🫶

✨ Attributes ✨

okay so the first thing on the list is the logger attribute 💅 that's the whole entry, no extra details, bestie 🫶

🌸 Functions 🌸

• fetchJobStoreFiles(jobStore, options) — Takes a list of file names as glob patterns, searches for these within a
• printContentsOfJobStore(job_store[, job_id]) — Fetch a list of all files contained in the job store if nameOfJob is not
• main() — no description given 🙈

🎀 Module Contents 🎀

• toil.utils.toilDebugFile.logger ✨

🦋 toil.utils.toilDebugFile.fetchJobStoreFiles(jobStore, options)

Okay this one is giving file-hunter energy 🔥 It takes a list of file names written as glob patterns, searches for them within a given directory, and tries to grab every file it finds and copy it into options.localFilePath.

Parameters

• jobStore (toil.jobStores.fileJobStore.FileJobStore, link target #toil.jobStores.fileJobStore.FileJobStore) — a fileJobStore object 💖
• options.fetch — the list of file glob patterns to look for in the jobStore and copy into options.localFilePath 🧚
• options.localFilePath — the local directory you're copying files into 📁
• options.jobStore — the path to the jobStore directory 🗂️
• options (argparse.Namespace, docs: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type

None 🤍

🦋 toil.utils.toilDebugFile.printContentsOfJobStore(job_store, job_id=None)

Okay so this one has two modes, and it's lowkey iconic. If nameOfJob isn't declared, it fetches a list of all files in the job store. If it is declared, it only prints out the names of files for that specific job that it can find a match for. It also creates a log file of these file names in the current directory 📝

Parameters

• job_store (toil.jobStores.fileJobStore.FileJobStore, link target #toil.jobStores.fileJobStore.FileJobStore) — the job store you're asking for files from 🌈
• job_id (Optional[str], docs: /usr/share/doc/python3/html/library/stdtypes.html#str) — defaults to None, which prints out all files in the jobStore. If you specify one, it prints every jobStore file that job has written to the jobStore 🫶

Return type

None 🤍

🦋 toil.utils.toilDebugFile.main()

Return type

None 🤍

---

🧚 toil.utils.toilDebugJob 🧚

Debug tool for running a toil job locally. Super helpful when you need to see what's happening without the whole cluster drama 🫠

✨ Attributes ✨

• logger ✨

💅 Functions 💅

• main() — no description given

🎀 Module Contents 🎀

• toil.utils.toilDebugJob.logger ✨

🦋 toil.utils.toilDebugJob.main()

Return type

None 🤍

---

💀 toil.utils.toilDestroyCluster 💀

Okay this one is literally the cleanup crew 🧹 It terminates the specified cluster and its associated resources. Bye bye, cluster 👋

✨ Attributes ✨

• logger ✨

💅 Functions 💅

• main() — no description given

🎀 Module Contents 🎀

• toil.utils.toilDestroyCluster.logger ✨

🦋 toil.utils.toilDestroyCluster.main()

Return type

None 🤍

---

🔥 toil.utils.toilKill 🔥

Kills rogue toil processes. Main character energy for villain-arc processes 💀

✨ Attributes ✨

• logger ✨

💅 Functions 💅

• main() — no description given

🎀 Module Contents 🎀

• toil.utils.toilKill.logger ✨

🦋 toil.utils.toilKill.main()

Return type

None 🤍

---

🚀 toil.utils.toilLaunchCluster 🚀

Launches a toil leader instance with the specified provisioner. It's giving "let's get this party started" 🎉

✨ Attributes ✨

• build_tag_dict_from_env 🏷️
• logger ✨

💅 Functions 💅

• create_tags_dict(tags) — no description given
• main() — no description given

🎀 Module Contents 🎀

🌈 toil.utils.toilLaunchCluster.build_tag_dict_from_env

Type: dict[str, str] (docs for dict: /usr/share/doc/python3/html/library/stdtypes.html#dict; docs for str: /usr/share/doc/python3/html/library/stdtypes.html#str) 💜

• toil.utils.toilLaunchCluster.logger ✨

🦋 toil.utils.toilLaunchCluster.create_tags_dict(tags)

Parameters

• tags (list[str], docs for list: /usr/share/doc/python3/html/library/stdtypes.html#list; docs for str: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

dict[str, str] (docs for dict: /usr/share/doc/python3/html/library/stdtypes.html#dict; docs for str: /usr/share/doc/python3/html/library/stdtypes.html#str) 🤍

🦋 toil.utils.toilLaunchCluster.main()

Return type

None 🤍

---

🛠️ toil.utils.toilMain 🛠️

💅 Functions 💅

• main() — no description given
• get_or_die(module, name) — Get an object from a module or complain that it is missing. 🙅‍♀️
• loadModules() — no description given
• printHelp(modules) — no description given
• printVersion() — no description given

🎀 Module Contents 🎀

🦋 toil.utils.toilMain.main()

Return type

None 🤍

🦋 toil.utils.toilMain.get_or_die(module, name)

Get an object from a module or complain that it is missing. Literally the bouncer of imports 🧐

Parameters

• module (types.ModuleType, docs: /usr/share/doc/python3/html/library/types.html#types.ModuleType)
• name (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type

Any 🤍

🦋 toil.utils.toilMain.loadModules()

Return type

dict[str, types.ModuleType] (docs for dict: /usr/share/doc/python3/html/library/stdtypes.html#dict; docs for str: /usr/share/doc/python3/html/library/stdtypes.html#str; docs for types.ModuleType: /usr/share/doc/python3/html/library/types.html#types.ModuleType) 🤍

🦋 toil.utils.toilMain.printHelp(modules)

Parameters

• modules (dict[str, types.ModuleType], docs for dict: /usr/share/doc/python3/html/library/stdtypes.html#dict; docs for str: /usr/share/doc/python3/html/library/stdtypes.html#str; docs for types.ModuleType: /usr/share/doc/python3/html/library/types.html#types.ModuleType)

Return type

None 🤍

🦋 toil.utils.toilMain.printVersion()

Return type

None 🤍

---

💖 toil.utils.toilRsyncCluster 💖

Rsyncs into the toil appliance container running on the leader of the cluster. It's basically sliding into the leader's DMs 📬

✨ Attributes ✨

• logger ✨

💅 Functions 💅

• main() — no description given

🎀 Module Contents 🎀

• toil.utils.toilRsyncCluster.logger ✨

🦋 toil.utils.toilRsyncCluster.main()

Return type

None 🤍

✨ toil.utils.toilServer ✨

okay so this is the CLI entry point for the Toil servers 💖 it's giving main character energy in the utils folder 👑

✨ Attributes ✨

• logger — the page leaves the description blank 🙈

🧚 Functions 🧚

• main() — no description given 🙈

🎀 Module Contents 🎀

• toil.utils.toilServer.logger
• toil.utils.toilServer.main()
  • Return type None

---

✨ toil.utils.toilSshCluster ✨

this one lets you SSH into the toil appliance container running on the leader of your cluster 🦋 bestie this is so useful ngl 🫶

✨ Attributes ✨

• logger — no description given 🙈

🧚 Functions 🧚

• have_ipv6() — gives back True if the IPv6 loopback interface is useable 🌈
• main() — no description given 🙈

🎀 Module Contents 🎀

• toil.utils.toilSshCluster.logger
• toil.utils.toilSshCluster.have_ipv6()
  • gives back True if the IPv6 loopback interface is useable 🌈
  • Return type bool </usr/share/doc/python3/html/library/functions.html# bool>
• toil.utils.toilSshCluster.main()
  • Return type None

---

✨ toil.utils.toilStats ✨

this one crunches the stats for a given Toil workflow and reports them back 📊 the numbers are literally everything 💅

✨ Attributes ✨

• logger — no description given 🙈
• CATEGORIES — no description given 🙈
• CATEGORY_UNITS — no description given 🙈
• TITLES — no description given 🙈
• TIME_CATEGORIES — no description given 🙈
• SPACE_CATEGORIES — no description given 🙈
• COMPUTED_CATEGORIES — no description given 🙈
• LONG_FORMS — no description given 🙈
• sort_category_choices — no description given 🙈
• sort_field_choices — no description given 🙈

🌸 Classes 🌸

• ColumnWidths — a convenience object that stores the width of columns for printing 🎀 it helps make things pretty ✨

✨ Functions ✨

okay bestie, this is the big table of helper functions 🎀 they all exist to make the stats printout look gorgeous, and ngl that's a whole vibe 💅

• pad_str(s[, field]) — pads the start of a string with spaces, but only if it actually needs it 🌸
• pretty_space(k[, field, alone]) — takes k as kibibytes and hands back a nicely formatted string ✨
• pretty_time(t[, field, unit, alone]) — takes t as seconds and gives you a cute formatted string back 🦋
• report_unit(unit) — formats a unit name so it's display-ready 💖
• report_time(t, options[, field, unit, alone]) — given t seconds, reports back the correct format as a string ⏱️
• report_space(k, options[, field, unit, alone]) — given k kibibytes, reports back the correct format as a string 💾
• report_number(n[, field, nan_value]) — given a number, reports back the correct format as a string 🔢
• report(v, category, options[, field, alone]) — reports a value of the given category, formatted as a string 🎀
• sprint_tag(key, tag, options[, columnWidths]) — generates a pretty-print-ready string from a JTTag() 🧚
• decorate_title(category, title, options) — adds extra bits to the category titles 👑
• decorate_subheader(category, columnWidths, options) — slaps a marker on the correct field if the TITLE is the one being sorted on 🔥
• get(tree, name) — returns a float value attribute NAME from TREE 🌈
• sort_jobs(jobTypes, options) — returns jobTypes, all sorted and ready to go ⭐
• report_pretty_data(root, worker, job, job_types, options) — prints out the important bits 🫶
• compute_column_widths(job_types, worker, job, options) — returns a ColumnWidths() object with the correct max widths 📏
• update_column_widths(tag, cw, options) — updates the column width attributes for this tag's fields 💅
• build_element(element, items, item_name, defaults) — creates an element for output 🍓
• create_summary(element, containingItems, ...) — figures out how many jobs (or contained items) ran on each worker (or containing item) 🧮
• get_stats(jobStore) — sums together all the stats information in the job store 💜
• process_data(config, stats) — collates the stats and reports them 📊
• report_data(tree, options) — no description given here, so I'm not guessing 🙈
• add_stats_options(parser) — also no description given, mystery energy 🙈
• main() — reports stats on the workflow, use it with the --stats option to toil 👑

---

✨ Module Contents ✨

okay, the module contents are giving organised chaos 💜

• toil.utils.toilStats.logger
• toil.utils.toilStats.CATEGORIES = ['time', 'clock', 'wait', 'memory', 'disk']
• toil.utils.toilStats.CATEGORY_UNITS
• toil.utils.toilStats.TITLES
• toil.utils.toilStats.TIME_CATEGORIES
• toil.utils.toilStats.SPACE_CATEGORIES
• toil.utils.toilStats.COMPUTED_CATEGORIES
• toil.utils.toilStats.LONG_FORMS

🎀 class toil.utils.toilStats.ColumnWidths

this is the convenience object that stores the width of columns for printing, so everything looks pretty 🌸

• categories = ['time', 'clock', 'wait', 'memory', 'disk']
• fields_count = ['count', 'min', 'med', 'ave', 'max', 'total']
• fields = ['min', 'med', 'ave', 'max', 'total']
• data: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>]
• title(category) — returns the total printed length of this category item ✨
  • Parameters
    • category (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • Return type
    • int </usr/share/doc/python3/html/library/functions.html#int>
• get_width(category, field) 🦋
  • Parameters
    • category (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
    • field (str </usr/share/doc/python3/html/library/stdtypes.html#str>)
  • Return type
    • int </usr/share/doc/python3/html/library/functions.html#int>
• set_width(category, field, width) 💖
  • Parameters

Parameters

• category (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• field (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• width (int: /usr/share/doc/python3/html/library/functions.html#int)

Return type: None

report()

Return type: None

---

🌸 toil.utils.toilStats.pad_str(s, field=None)

okay so this one just pads the front of a string with spaces, but only if it actually needs them 💅

Parameters

• s (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• field (Optional[int]: /usr/share/doc/python3/html/library/functions.html#int)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🦋 toil.utils.toilStats.pretty_space(k, field=None, alone=False)

Given k as kibibytes, this gives you back a nicely formatted string. the vibe is immaculate ✨

Parameters

• k (float: /usr/share/doc/python3/html/library/functions.html#float)
• field (Optional[int]: /usr/share/doc/python3/html/library/functions.html#int)
• alone (bool: /usr/share/doc/python3/html/library/functions.html#bool)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🧚 toil.utils.toilStats.pretty_time(t, field=None, unit='s', alone=False)

this one takes t in seconds and serves it back looking gorgeous 💖 the unit defaults to 's' so you're set unless you say otherwise

Parameters

• t (float: /usr/share/doc/python3/html/library/functions.html#float)
• field (Optional[int]: /usr/share/doc/python3/html/library/functions.html#int)
• unit (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• alone (bool: /usr/share/doc/python3/html/library/functions.html#bool)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🎀 toil.utils.toilStats.report_unit(unit)

formats a unit name so it looks presentable on screen 🫶

Parameters

• unit (str: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

⭐ toil.utils.toilStats.report_time(t, options, field=None, unit='s', alone=False)

Given t seconds, this reports back the correct formatted string. it's the main character of time formatting, no cap 👑

Parameters

• t (float: /usr/share/doc/python3/html/library/functions.html#float)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)
• field (Optional[int]: /usr/share/doc/python3/html/library/functions.html#int)
• unit (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• alone (bool: /usr/share/doc/python3/html/library/functions.html#bool)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

💜 toil.utils.toilStats.report_space(k, options, field=None, unit='KiB', alone=False)

Given k kibibytes, this reports back the correct formatted string. the unit defaults to 'KiB' 🍓

If unit is set to B, it converts to KiB first. so that's one extra step it takes for you, bestie 🙈

Parameters

• k (float: /usr/share/doc/python3/html/library/functions.html#float)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)
• field (Optional[int]: /usr/share/doc/python3/html/library/functions.html#int)
• unit (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• alone (bool: /usr/share/doc/python3/html/library/functions.html#bool)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🌈 toil.utils.toilStats.report_number(n, field=None, nan_value='NaN')

given a number, this hands back a nicely formatted string 🔥

if n is a NaN or None, it uses nan_value to represent it instead. the default is 'NaN', so no drama 💀

Parameters

• n (Union[int, float, None]: /usr/share/doc/python3/html/library/functions.html#int, /usr/share/doc/python3/html/library/functions.html#float)
• field (Optional[int]: /usr/share/doc/python3/html/library/functions.html#int)
• nan_value (str: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

💅 toil.utils.toilStats.report(v, category, options, field=None, alone=False)

Reports a value of the given category, formatted as a string. it's giving organized 🎀

it uses the given field width if one is set. if alone is set, the field is being formatted outside a table and might need a unit 🦋

Parameters

• v (float: /usr/share/doc/python3/html/library/functions.html#float)
• category (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)
• field (Optional[int]: /usr/share/doc/python3/html/library/functions.html#int)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🌸 toil.utils.toilStats.sprint_tag(key, tag, options, columnWidths=None)

this one generates a pretty-print ready string from a JTTag() ✨

Parameters

• key (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• tag (toil.lib.expando.Expando: #toil.lib.expando.Expando)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)
• columnWidths (Optional[ColumnWidths]: #toil.utils.toilStats.ColumnWidths)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🧚 toil.utils.toilStats.decorate_title(category, title, options)

this adds extra bits to the category titles so everything reads cleanly 🫶

units get added to the title if they won't show up in the formatted values. and if the title is the one being sorted on, it gets a marker so you can spot it instantly 👑

Parameters

• category (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• title (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🦋 toil.utils.toilStats.decorate_subheader(category, columnWidths, options)

Adds a marker to the correct field if the title is sorted on. it's all about being precise bestie 💖

Parameters

• category (str: /usr/share/doc/python3/html/library/stdtypes.html#str)
• columnWidths (ColumnWidths: #toil.utils.toilStats.ColumnWidths)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🍓 toil.utils.toilStats.get(tree, name)

returns a float value for the attribute name from tree 💅

Parameters

• tree (toil.lib.expando.Expando: #toil.lib.expando.Expando)
• name (str: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: float (/usr/share/doc/python3/html/library/functions.html#float)

---

👑 toil.utils.toilStats.sort_jobs(jobTypes, options)

returns jobTypes all sorted, iconic behaviour 🔥

Parameters

• jobTypes (list[Any]: /usr/share/doc/python3/html/library/stdtypes.html#list)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type: list[Any] (/usr/share/doc/python3/html/library/stdtypes.html#list)

---

💖 toil.utils.toilStats.report_pretty_data(root, worker, job, job_types, options)

this one prints out the important bits 🌈 it's the whole highlight reel

Parameters

• root (toil.lib.expando.Expando: #toil.lib.expando.Expando)
• worker (toil.lib.expando.Expando: #toil.lib.expando.Expando)
• job (toil.lib.expando.Expando: #toil.lib.expando.Expando)
• job_types (list[Any]: /usr/share/doc/python3/html/library/stdtypes.html#list)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type: str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

✨ toil.utils.toilStats.compute_column_widths(job_types, worker, job, options)

Returns a ColumnWidths() object with the correct max widths. the widths are locked in and ready to slay 💅

Parameters

• job_types (list[Any]: /usr/share/doc/python3/html/library/stdtypes.html#list)
• worker (toil.lib.expando.Expando: #toil.lib.expando.Expando)
• job (toil.lib.expando.Expando: #toil.lib.expando.Expando)
• options (argparse.Namespace: /usr/share/doc/python3/html/library/argparse.html#argparse.Namespace)

Return type: ColumnWidths (#toil.utils.toilStats.ColumnWidths)

---

🫶 toil.utils.toilStats.update_column_widths(tag, cw, options)

updates the column width attributes for this tag's fields, so every column stays in its lane 🎀

Parameters

• tag (toil.lib.expando.Expando) 💖
• cw (ColumnWidths) ✨
• options (argparse.Namespace) 🌸

Return type: None 🎀

toil.utils.toilStats.build_element(element, items, item_name, defaults)

Okay so this one makes a brand new element for the output. It's literally the blank canvas for your stats 🎨💅

Parameters

• element (toil.lib.expando.Expando)
• items (list[toil.job.Job])
• item_name (str)
• defaults (dict[str, float])

Return type: toil.lib.expando.Expando 👑

toil.utils.toilStats.create_summary(element, containingItems, containingItemName, count_contained)

This is the one that figures out how many jobs (or contained items) ran on each worker (or containing item). It's giving math-brain energy 🧠🦋

It also sticks a bunch of xxx_number_per_xxx stats into element to describe all of that. Iconic ✨

Parameters

• count_contained (Callable[[toil.lib.expando.Expando], int]) — the function that maps from a containing item to the number of contained items 🔢
• element (toil.lib.expando.Expando)
• containingItems (list[toil.lib.expando.Expando])
• containingItemName (str)

Return type: None 💖

toil.utils.toilStats.get_stats(jobStore)

Bestie, this one adds together all the stats info sitting in the job store. 📊

What you get back is one single object holding lists of the values from every summed object. Main character energy 👑

Parameters

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type: toil.lib.expando.Expando 🌸

toil.utils.toilStats.process_data(config, stats)

This one collates the stats and reports them. Literally the glow-up moment 💫

Parameters

• config (toil.common.Config)
• stats (toil.lib.expando.Expando)

Return type: toil.lib.expando.Expando 🦋

toil.utils.toilStats.report_data(tree, options)

Okay this one takes your tree and your options and reports on them. Quick and clean 🧚

Parameters

• tree (toil.lib.expando.Expando)
• options (argparse.Namespace)

Return type: None 🫶

toil.utils.toilStats.sort_category_choices

The valid sort categories are ['time', 'clock', 'wait', 'memory', 'disk', 'alpha', 'count']. Choose wisely, babe 💅

toil.utils.toilStats.sort_field_choices

The valid sort fields are ['min', 'med', 'ave', 'max', 'total']. Pick your fave 🍓

toil.utils.toilStats.add_stats_options(parser)

This one adds the stats options onto your parser. Super simple, no notes 🎀

Parameters

• parser (argparse.ArgumentParser)

Return type: None ✨

toil.utils.toilStats.main()

Reports stats on the workflow. Use it with the --stats option to toil, and you're set 📈🔥

Return type: None 💜

---

🦋 toil.utils.toilStatus

Tool for reporting on job status. It's the one who keeps you in the loop 👀

Attributes

• logger

Classes

• ToilStatus — Tool for reporting on job status. 🧚

Functions

• main() — Reports the state of a Toil workflow. 🌈

Module Contents

toil.utils.toilStatus.logger 🌸

class toil.utils.toilStatus.ToilStatus(jobStoreName, specifiedJobs=None)

Tool for reporting on job status. This is literally the whole reason we're here 💖

Parameters

• jobStoreName (str)
• specifiedJobs (Optional[list[str]])

Attributes

• jobStoreName
• jobStore
• message_bus_path

print_dot_chart()

Prints a dot output graph representing the workflow. Very visual, very aesthetic 🎨

Return type: None

printJobLog()

Takes a list of jobs, finds their log files, and prints them to the terminal. Log dive 🕵️‍♀️

Return type: None

printJobChildren()

Takes a list of jobs and prints their successors. Family tree vibes 🌳

Return type: None

printAggregateJobStats(properties, childNumber)

Prints each job's ID, log file, remaining tries, and other properties. It's a whole vibe check 🔍

Parameters

• properties (list[set[str]]) — a set of string flag names for each job in self.jobsToReport 🏷️
• childNumber (list[int]) — a list of child counts for each job in self.jobsToReport 🔢

Return type: None

report_on_jobs()

Gathers information about jobs, like their child jobs and status. Lowkey the most useful one 📋

Returns jobStats: a dict containing some lists of jobs by category, plus some lists of job properties for each job in self.jobsToReport 💜

Return type: dict[str, Any]

static getPIDStatus(jobStoreName)

Determines the status of a process with a particular local pid. It checks to see if a process exists or not. Simple and iconic 🧠

Returns a string indicating the status of the PID of the workflow as stored in the jobstore 🌟

Return type: str

Parameters

• jobStoreName (str)

static getStatus(jobStoreName)

Determines the status of a workflow. If the jobstore doesn't exist yet, this returns 'QUEUED', assuming it hasn't been created yet. ⏳

It checks for the existence of files created in toil.Leader.run(). In toil.Leader.run(), if a workflow completes with failed jobs, 'failed.log' is created. Otherwise, 'succeeded.log' is written. If neither of these exist, the leader is still running jobs 🏃‍♀️💨

Heads up: the status comes back as one of ['COMPLETED', 'RUNNING', 'ERROR', 'QUEUED']. That's the whole list, no extras 💀

Returns a string indicating the status of the workflow.

Return type: str

Parameters

• jobStoreName (str)

print_running_jobs()

Prints a list of the currently running jobs. Who's up? 🏃‍♀️

Return type: None

fetchRootJob()

Fetches the root job from the jobStore, the one that gives context for all the other jobs 👑

It's exactly the same as the jobStore.loadRootJob() function. The only difference is the exit message if the root job isn't found. That message means the workflow finished successfully, so some stats can't really be gathered from it, like which jobs are left to run 🎉

Raises JobException — if the root job doesn't exist ❌

Return type: toil.job.JobDescription

fetchUserJobs(jobs)

Takes a user-input array of jobs, verifies that they're in the jobStore, and returns the array of jobsToReport. 🫶

Parameters

• jobs (list) — a list of jobs to be verified ✅

Returns jobsToReport: a list of jobs that are verified to be in the jobStore 💖

Return type: list[toil.job.JobDescription] 🎀 that's the one from before, bestie 💖

✨ traverseJobGraph(rootJob, jobsToReport=None, foundJobStoreIDs=None) ✨

Okay so this one goes hunting through the jobStore for every current job and hands them all back as an Array 🔍💅

Parameters

• rootJob (toil.job.JobDescription) — the root job of the workflow, aka where the whole story starts 🌱
• jobsToReport (list) — a list of jobNodes that gets added to and then returned 🧾
• foundJobStoreIDs (set) — a set of jobStoreIDs that keeps track of every jobStoreID it's already spotted during the traversal 👀

Returns — jobsToReport: the list of jobs currently in the job graph ✨

Return type — list[toil.job.JobDescription] 🎀

✨ toil.utils.toilStatus.main() ✨

Reports the state of a Toil workflow 📊 basically the vibe check for your run 🫶

Return type — None 💀

✨ toil.utils.toilUpdateEC2Instances ✨

Updates Toil's internal list of EC2 instance types 🦋 so it stays up to date with the newest drops 💜

🎀 Attributes

• logger

🎀 Functions

• internet_connection() — Returns True if there is an internet connection present, and False otherwise 🌐
• main()

🎀 Module Contents

toil.utils.toilUpdateEC2Instances.logger

toil.utils.toilUpdateEC2Instances.internet_connection() — Returns True if there is an internet connection present, and False otherwise 🌐

Return type — bool 🫶

toil.utils.toilUpdateEC2Instances.main()

Return type — None 💀

✨ toil.version ✨

🎀 Attributes

• baseVersion
• cgcloudVersion
• version
• cacheTag
• mainCacheTag
• distVersion
• exactPython
• python
• dockerTag
• currentCommit
• dockerRegistry
• dockerName
• dirty
• cwltool_version

🎀 Module Contents

These are all the version details, bestie, so you can see exactly what build you're running 🧚💜

toil.version.baseVersion = '9.1.2'

toil.version.cgcloudVersion = '1.6.0a1.dev393'

toil.version.version = '9.1.2-355b72ba0e425619c0367562caf2a337078dba65'

toil.version.cacheTag = 'cache-local-py3.13'

toil.version.mainCacheTag = 'cache-master-py3.13'

toil.version.distVersion = '9.1.2'

toil.version.exactPython = 'python3.13'

toil.version.python = 'python3.13'

toil.version.dockerTag = '9.1.2-355b72ba0e425619c0367562caf2a337078dba65-py3.13'

toil.version.currentCommit = '355b72ba0e425619c0367562caf2a337078dba65'

toil.version.dockerRegistry = 'quay.io/ucsc_cgl'

toil.version.dockerName = 'toil'

toil.version.dirty = False

toil.version.cwltool_version = '3.1.20250715140722'

✨ toil.wdl ✨

🎀 Submodules

• toil.wdl.utils

🎀 Functions

• get_version(iterable) — Get the version of the WDL document 📜

🎀 Module Contents

toil.wdl.utils.get_version(iterable) — Get the version of the WDL document 📜

Parameters

• iterable (collections.abc.Iterable[str]) — an iterable that contains the lines of a WDL document 🧾

Returns — The WDL version used in the workflow 🌸

Return type — str 🎀

✨ toil.wdl.wdltoil ✨

🎀 Attributes

• logger
• WDLINode
• AnyINode
• file_digest
• WDLContext
• F
• WDLBindings
• SHARED_PATH_ATTR

🎀 Exceptions

• InsufficientMountDiskSpace — Common base class for all non-exit exceptions 🚨

✨ Classes ✨

okay bestie, here's the whole WDL class lineup 💅🎀 every one of these has its own vibe, so let's go:

• INodeTransform — a type for a function that takes a File or Directory and hands back a modified copy, or None 🦋
• ReadableFileObj — a protocol that's more specific than what file_digest takes as an argument 🌸
• FileDigester — a protocol for the features we need from hashlib.file_digest 🔥
• NonDownloadingSize — a WDL size() implementation that avoids downloading files at all 🧚
• ToilWDLStdLibBase — the standard library for WDL as run on Toil 💖
• ToilWDLStdLibWorkflow — the standard library for workflow scope ✨
• ToilWDLStdLibTaskCommand — the standard library to use inside a WDL task command evaluation 🫶
• ToilWDLStdLibTaskOutputs — the standard library for WDL as run on Toil, with additional 🍓
• WDLBaseJob — the base job class for all WDL-related jobs, the OG of the group 👑
• WDLTaskWrapperJob — a job that figures out the resources needed to run a WDL job 💜
• WDLTaskJob — a job that runs a WDL task 🌈
• WDLWorkflowNodeJob — a job that evaluates a WDL workflow node 🎀
• WDLWorkflowNodeListJob — a job that evaluates a list of WDL workflow nodes, which are in the same 🦋
• WDLCombineBindingsJob — a job that collects the results from WDL workflow nodes and combines their 💅
• WDLWorkflowGraph — represents a graph of WDL WorkflowNodes 🌸
• WDLSectionJob — a job that can create more graph for a section of the workflow 🧚
• WDLScatterJob — a job that evaluates a scatter in a WDL workflow. Runs the body for each 🔥
• WDLArrayBindingsJob — a job that takes all new bindings created in an array of input environments, 💖
• WDLConditionalJob — a job that evaluates a conditional in a WDL workflow ✨
• WDLWorkflowJob — a job that evaluates an entire WDL workflow 🫶
• WDLOutputsJob — a job which evaluates an outputs section for a workflow 🍓
• WDLStartJob — a job that evaluates an entire WDL workflow, and returns the workflow outputs 👑
• WDLInstallImportsJob — a class that represents a unit of work in toil 💜
• WDLImportWrapper — a job that organizes importing files on workers instead of the leader. Responsible for extracting filenames and metadata, 🌈

---

✨ Functions ✨

the helper functions are small but mighty 🥹💅

• is_inode(value) — determines if a WDL value is either a File or Directory 🎀
• wdl_error_reporter(task[, exit, log]) — runs code in a context where WDL errors get reported with pretty formatting 🦋
• report_wdl_errors(task[, exit, log]): okay so this one is a decorator that slaps the given task message onto any WDL errors, periodt 🎀
• toil_read_source(uri, path, importer): a MiniWDL read_source implementation that can use any 🧚 (the description literally cuts off right there, bestie 💀)
• virtualized_equal(value1, value2): checks whether two WDL values are equal once file virtualization is taken into account 🦋
• combine_bindings(all_bindings): merges variable bindings from multiple predecessor tasks into one set for 🌸 (this one also trails off, no cap)
• log_bindings(log_function, message, all_bindings): logs bindings to the console, even if some of them are still promises 📝
• get_supertype(types): finds the supertype that can hold values of all the given types 👑
• for_each_node(root): iterates over every WDL workflow node inside the given node, including inputs, 🫶 (cut off again, we get it 😭)
• recursive_dependencies(root): gets the combined workflow_node_dependencies of root and everything under 💜 (you can feel the sentence getting away from it)
• parse_disks(spec, disks_spec): parses a WDL disk spec into a disk mount specification 💾
• pack_toil_uri(file_id, task_path, parent, file_basename): encodes a Toil file ID plus metadata about who wrote it as a URI 🔥
• unpack_toil_uri(toil_uri): unpacks a URI made by make_toil_uri 🍓
• clone_metadata(old_inode, new_inode): copies all Toil metadata from one WDL File/Directory to another ✨
• make_inode(example_inode, value, expr): makes a new File or Directory of the same type as the example, using the given arguments 🌈
• set_inode_value(inode, new_value): returns a copy of a WDL File/Directory with the value changed 💅
• set_inode_nonexistent(inode, nonexistent): returns a copy of a WDL File/Directory with the nonexistent flag changed 🙈
• get_inode_nonexistent(inode): returns the nonexistent flag for a File/Directory 💖
• set_inode_virtualized_value(inode, virtualized_value): returns a copy of a WDL File/Directory with the virtualized_value attribute set 🦋
• get_inode_virtualized_value(inode): gets the virtualized storage location for a File/Directory 🌸
• get_shared_fs_path(inode): if a File/Directory has a shared filesystem path, this grabs that path ⭐
• set_shared_fs_path(inode, path): returns a copy of the given File/Directory with a shared filesystem path attached 🫶
• view_shared_fs_paths(bindings) — 🗂️ Takes your WDL bindings and gives you back a copy where every file has its shared filesystem path as its value 💖
• poll_execution_cache(node, bindings) — 🔍 Checks the cache and returns the cached result of calling this workflow or task, plus its key 🔑
• fill_execution_cache(cache_key, output_bindings, ...) — 💾 Stores the result of calling a workflow or task in the cache so it can be reused later ✨
• choose_human_readable_directory(root_dir, ...) — 📁 Picks a nice, readable directory to save files from a task and source directory in 🌸
• evaluate_decls_to_bindings(decls, all_bindings, ...[, ...]) — 🧮 Evaluates decls using a given bindings environment and standard library. Main character energy 👑
• extract_inode_values(environment) — 📦 Grabs a list of every File or Directory object value in the given bindings 🦋
• extract_inode_virtualized_values(environment) — 🪄 Grabs a list of all File/Directory object virtualized values in the bindings 🧚
• extract_toil_file_uris(environment) — 🔗 Pulls out the toilfile: URIs from the given bindings 💜
• virtualize_inodes_in_bindings(environment, file_to_id, ...) — 🫶 Fills in the virtualized_value fields for File/Directory objects 🌈
• convert_remote_files(environment, file_source, task_path) — 🌍 Resolves relative-URI files in the given environment and imports all of them. No cap, it's a whole glow-up 🔥
• evaluate_named_expression(context, name, ...) — 🏷️ Evaluates an expression when we already know its name 💅
• evaluate_decl(node, environment, stdlib) — 🧠 Evaluates the expression of a declaration node, or raises an error if it can't 💀
• evaluate_call_inputs(context, expressions, ...[, ...]) — 🎀 Evaluates a bunch of named expressions and turns them into a fresh set of bindings. inputs_dict is a mapping of (the original cuts off right there, bestie 😭)
• evaluate_defaultable_decl(node, environment, stdlib) — 🍓 If the declaration's name is already defined in the environment, you get its value back. Otherwise, you get the evaluated expression ✨
• devirtualize_inodes(environment, stdlib) — 🦋 Makes sure all the File values embedded in the given bindings point to files 🌸
• virtualize_inodes(environment, stdlib[, enforce_existence]) — 🪞 Makes sure all the File/Directory values embedded in the given bindings point to files 💖
• delete_dead_files(internal_bindings, ...) — 🗑️ Deletes any files that are in the given bindings but not in the live list. Bye bye, files 👋
• all_parents(path) — ⬆️ Yields all parents of the given path, all the way up to the filesystem root 🌳
• add_paths(task_container, host_paths) — 🫧 Based off of WDL.runtime.task_container.add_paths from miniwdl. Iykyk 💜

💅 Function Table (keep scrolling, bestie) 💅

• drop_if_missing(inode, standard_library) — Returns None if a File/Directory doesn't exist, or its path if it does 🦋
• missing_inode_dropper(standard_library) — Gets you a function that nulls out missing File/Directory values 🧚
• drop_missing_files(environment, standard_library) — Makes sure every File value embedded in the given bindings actually points to a file 📁
• get_paths_in_bindings(environment) — Grabs the paths of all Files and Directories in the bindings 🌸
• map_over_inodes_in_bindings(environment, transform) — Runs all File values embedded in the given bindings through the given transform ✨
• map_over_inodes_in_binding(binding, transform) — Runs all File values' types and values embedded in the given binding's value through the given transform 💜
• remove_expr_from_value(value) — Removes the expression from a WDL value, bye bye 👋
• map_over_typed_value(value, transform) — Applies a transform to a WDL value and every WDL value it contains 🍓
• map_over_typed_inodes_in_value(value, transform) — Runs all File values embedded in the given value through the given transform 🎀
• ensure_null_inodes_are_nullable(value, original_value, ...) — Walks through all the nested values inside the given value and double-checks that the null values are valid 🙈
• make_root_job(target, inputs, inputs_search_path, ...) — (No description was given for this one, so we're leaving it there, babe.) 🫶
• main() — The main event: a Toil workflow that interprets WDL input files 👑

---

✨ Module Contents ✨

okay so this next bit is the actual inventory of the module, and it's giving "everything you need, all in one place" 💖

• toil.wdl.wdltoil.logger
• toil.wdl.wdltoil.WDLINode
• toil.wdl.wdltoil.AnyINode

🦋 class toil.wdl.wdltoil.INodeTransform

Bases: Protocol

This is a type for a function that takes a File or Directory and hands back a modified copy, or None.

ngl the typing here is a whole thing 💀 If you write Callable[[AnyINode], AnyINode] as an argument type, your function becomes generic over the type variable, but that does not mean you're accepting a function that's generic over it too. So the module defines a more complicated type for functions that turn inodes into the same kind of inode. Main character energy 👑

Its __call__ comes in two flavors:

__call__(__file: WDL.Value.File) -> WDL.Value.File | None
__call__(__directory: WDL.Value.Directory) -> WDL.Value.Directory | None

🌈 toil.wdl.wdltoil.is_inode(value)

This one checks whether a WDL value is a File or a Directory. It's a MyPy type guard, so if you wrap code in an if statement that calls it, MyPy gets convinced it's safe to treat whatever you passed in as a File-or-Directory. Iykyk 🧚

• Parameters
  • value (WDL.Value.Base)
• Return type
  • typing_extensions.TypeGuard[WDLINode]

🎀 class toil.wdl.wdltoil.ReadableFileObj

Bases: Protocol

A protocol that's more specific than what file_digest takes as an argument, and it guarantees you get a read() method. Lowkey it's the bestie of the hashing world 🫶

It was going to extend the Typeshed protocol for hashlib, but those only get declared for 3.11+, so we're not doing that. Rip 😭

• readinto(buf, /)
  • Parameters
    • buf (bytearray)
  • Return type
    • int
• readable()
  • Return type
    • bool
• read(number)
  • Parameters
    • number (int)
  • Return type
    • bytes

🍓 class toil.wdl.wdltoil.FileDigester

Bases: Protocol

A protocol for just the features we need from hashlib.file_digest. Only the essentials, no notes 💅

• __call__(__f, __alg_name)
  • Parameters
    • __f (ReadableFileObj)
    • __alg_name (str)
  • Return type
    • hashlib._Hash

🔥 toil.wdl.wdltoil.file_digest: FileDigester

This is the file digester itself, typed as a FileDigester. Slay 🔥

💜 toil.wdl.wdltoil.WDLContext

The WDL context. Just sitting there being important 🌸

⚠️ exception toil.wdl.wdltoil.InsufficientMountDiskSpace(mount_targets, desired_bytes, available_bytes)

Bases: Exception

This is the common base class for all non-exit exceptions. So when your mount doesn't have enough space, this is the drama 😭

• Parameters
  • mount_targets (list[str])
  • desired_bytes (int)
  • available_bytes (int)

🧚 toil.wdl.wdltoil.wdl_error_reporter(task, exit=False, log=logger.critical)

This lets you run code in a context where WDL errors get reported with pretty formatting. Chef's kiss ✨

• Parameters
  • task (str)
  • exit (bool)
  • log (Callable[[str], None])
• Return type
  • Generator[None]

🌈 toil.wdl.wdltoil.F

Just the F name, living its best life 🦋

🌸 toil.wdl.wdltoil.report_wdl_errors(task, exit=False, log=logger.critical)

This creates a decorator that reports WDL errors with the task message you give it. You slap the decorator on a function, and if a WDL error happens, it says that it could not {task}. Super iconic, no notes 💖

• Parameters
  • task (str)
  • exit (bool)
  • log (Callable[[str], None])
• Return type
  • Callable[[F], F]

💖 toil.wdl.wdltoil.toil_read_source(uri, path, importer)

okay so this one is a MiniWDL read_source function that can use any filename or URL Toil supports 🌸 it's giving versatile ✨ and it has to be async because MiniWDL is going to await its result, so don't skip that part bestie 💅

Parameters

• uri (str) 🧚
• path (list[str]) 🧚
• importer (WDL.Tree.Document | None) 🧚

Return type — WDL.Tree.ReadSourceResult 🎀

---

💜 toil.wdl.wdltoil.virtualized_equal(value1, value2)

this one checks if two WDL values are equal while accounting for file virtualization 🦋 main character energy for real. it treats virtualized and non-virtualized Files and Directories that point at the same underlying thing as equal, so no drama over that 🫶

Parameters

• value1 (WDL.Value.Base) — a WDL value 💖
• value2 (WDL.Value.Base) — a WDL value 💖

Returns — whether the two values are equal with file and directory virtualization accounted for ✨

Return type — bool 💅

---

🌈 toil.wdl.wdltoil.WDLBindings

just vibing here with no description at all 🙈 it's listed by itself, so consider it the mysterious one in the group chat 👑

---

🔥 toil.wdl.wdltoil.combine_bindings(all_bindings)

this one takes variable bindings from multiple predecessor tasks and smooshes them into one set for the current task 🫶 it's like merging everyone's notes before the group project is due 🍓

Parameters

• all_bindings (Sequence[WDLBindings]) 🧚

Return type — WDLBindings ✨

---

🦋 toil.wdl.wdltoil.log_bindings(log_function, message, all_bindings)

this one logs bindings to the console, even if some of them are still promises 🔮 yes, we love a little chaos, but it all still gets logged 💅

Parameters

• log_function (Callable[Ellipsis, None]) — the function you want to log with, like logger.info 📣
• message (str) — a message to log right before the bindings 💬
• all_bindings (Sequence[toil.job.Promised[WDLBindings]]) — a list of bindings or promises for bindings that you want logged 🌸

Return type — None 🫧

---

🧚 toil.wdl.wdltoil.get_supertype(types)

this one finds the supertype that can hold values of all the given types 👑 it's the one type that's big enough to fit everybody, no cap 🍓

Parameters

• types (Sequence[WDL.Type.Base]) 🧚

Return type — WDL.Type.Base 💖

---

🌸 toil.wdl.wdltoil.for_each_node(root)

okay so this one is an iterator that goes through all WDL workflow nodes inside the given node 🦋 that includes inputs, internal nodes of conditionals and scatters, and gather nodes. literally everything, nobody gets left out 🫶

Parameters

• root (WDL.Tree.WorkflowNode) 💖

Return type — Iterator[WDL.Tree.WorkflowNode] ✨

---

💜 toil.wdl.wdltoil.recursive_dependencies(root)

this one gathers up the combined workflow_node_dependencies of root and everything under it, but only the ones that aren't satisfied by something inside that same subtree 🔥 it's useful because section nodes can have internal nodes with dependencies that don't show up in the section node's own list, so this one catches them 🫶

Parameters

• root (WDL.Tree.WorkflowNode) 💖

Return type — set[str] 🎀

---

🎀 toil.wdl.wdltoil.parse_disks(spec, disks_spec)

this one takes a WDL disk spec and turns it into a disk mount specification 💅 it hands back three things: the mount point (None if it was omitted or is local-disk), the number of units, and the size of each unit (like GB).

Parameters

• spec (str) — the disks spec you want to parse 💖
• disks_spec (list[WDL.Value.String] | str) — the full disks spec as written in the WDL file. it's only there for better error messages, so it's not really doing the parsing 🙈

Return type — tuple[str | None, float, str] 🦋

---

🫧 toil.wdl.wdltoil.pack_toil_uri(file_id, task_path, parent, file_basename)

this one packs a Toil file ID plus info about who wrote the file into a URI 🔮 the URI always starts with the scheme in TOIL_URI_SCHEME, so you know what you're looking at ✨

Parameters

• parent (str) — the bare path or URI to the parent of the file. only one unique value can be used for a given parent location 🚨 it has to match the name parameter of toil.lib.directory.encode_directory(). it can be absolute or relative, but to avoid collisions it should only be relative for worker temp storage 🌸
• file_id (toil.fileStores.FileID) 🧚
• task_path (str) 🧚
• file_basename (str) 🧚

Return type — str 💖

---

🌈 toil.wdl.wdltoil.unpack_toil_uri(toil_uri)

the opposite of packing! this one unpacks a URI made by make_toil_uri 🙈 yes, that's what the docs say, iykyk 💀

Returns — you get back the FileID, source task, source parent path or URI, and basename, all in one go 🎁

Parameters

• toil_uri (str) 🧚

Return type — tuple[toil.fileStores.FileID, str, str, str] ✨

---

⭐ toil.wdl.wdltoil.SHARED_PATH_ATTR

a little constant that's set to '_shared_fs_path' 🦋 it's the attribute name we use for the shared filesystem path, so it's worth knowing 🧚

---

💅 toil.wdl.wdltoil.clone_metadata(old_inode, new_inode)

this one copies all Toil metadata from one WDL File/Directory over to another 🫶 total glow-up, nothing gets left behind 💖

Parameters

• old_inode (AnyINode) 🧚
• new_inode (AnyINode) 🧚

Return type — None 🫧

---

🦋 toil.wdl.wdltoil.make_inode(example_inode, value, expr)

this one makes a brand new File or Directory that's the same type as the example, using the arguments you give it 🌸 fun fact: we use this because MyPy can't tell that type(a)(args) has the same type as a when a is typed with a TypeVar. MyPy really be out here not understanding the assignment 💀

Parameters

• example_inode (AnyINode) 🧚
• value (str) 🧚
• expr (Optional[WDL.Expr.Base]) 🧚

Return type — AnyINode ✨

---

💖 toil.wdl.wdltoil.set_inode_value(inode, new_value)

this one returns a copy of a WDL File/Directory with the value changed 🔥 and it keeps all the Toil metadata intact, so nothing gets lost in the shuffle 🫶

Parameters

• inode (AnyINode) 🧚
• new_value (str) 🧚

Return type — AnyINode 💅

---

🎀 toil.wdl.wdltoil.set_inode_nonexistent(inode, nonexistent)

this one returns a copy of a WDL File/Directory with the nonexistent flag changed 🌈 all Toil metadata gets preserved here too, we love consistency 💜

Parameters

• inode (AnyINode) 🧚
• nonexistent (bool) 🧚

Return type — AnyINode 🫧

---

🧚 toil.wdl.wdltoil.get_inode_nonexistent(inode)

this one gets the nonexistent flag for a File/Direcotry (yes, that's how it's spelled in the docs 🙈) 🌸

Parameters

• inode (WDLINode) 🧚

Return type — bool ✨

---

🔮 toil.wdl.wdltoil.set_inode_virtualized_value(inode, virtualized_value)

this one returns a copy of a WDL File/Directory with the virtualized_value attribute set 🦋 all Toil metadata stays put, obviously 💖

Parameters

• inode (AnyINode) 🧚
• virtualized_value (str) 🧚

Return type — AnyINode 🫶

---

🌸 toil.wdl.wdltoil.get_inode_virtualized_value(inode)

this one gets the virtualized storage location for a File/Directory 💅 it's the place it's actually stored behind the scenes 🫧

Parameters

• inode (WDLINode) 🧚

Return type — Optional[str] 🎀

---

💜 toil.wdl.wdltoil.get_shared_fs_path(inode)

if a File/Directory has a shared filesystem path, this one gets that path for you 🌈 it'll be the path the File was first imported from, or the path it has in the call cache, so you know where it came from 🫶

Parameters

• inode (WDLINode) 🧚

Return type — Optional[str] 💖

---

🔥 toil.wdl.wdltoil.set_shared_fs_path(inode, path)

this one returns a copy of the given File/Directory with a shared filesystem path set 🦋 ideally this should be the path it was originally imported from, or the path it has in the call cache 🌸

Parameters

• inode (AnyINode) 🧚
• path (str) 🧚

Return type — AnyINode ✨

---

🫧 toil.wdl.wdltoil.view_shared_fs_paths(bindings)

given some WDL bindings, this one returns a copy where every file's value is its shared filesystem path 🔮 it's like a little filter that shows you the real addresses, periodt 💅

Parameters

• bindings (WDLBindings) 🧚

Return type — WDLBindings 🎀

---

🍓 toil.wdl.wdltoil.poll_execution_cache(node, bindings)

this one checks the execution cache and returns the cached result of calling this workflow or task, along with its key 🧚 if the cache has no result for us, it returns None and the key, so we know to run it fresh 💖

this one deals in un-namespaced bindings 🙈

Parameters

• node (Union[WDL.Tree.Workflow, WDL.Tree.Task]) 🧚
• bindings (WDLBindings) 🧚

Return type — tuple[WDLBindings | None, str] 💅 so you get bindings (or None) and a string, bestie 🎀

---

✨ toil.wdl.wdltoil.fill_execution_cache ✨

toil.wdl.wdltoil.fill_execution_cache(cache_key, output_bindings, file_store, wdl_options, miniwdl_logger=None, miniwdl_config=None)

Okay so this one caches the result of calling a workflow or task 💾🔥 It deals in un-namespaced bindings, so keep that in your back pocket 🫶

Parameters

• cache_key (str) 🔑
• output_bindings (WDLBindings) 📦
• file_store (toil.fileStores.abstractFileStore.AbstractFileStore) 🗄️
• wdl_options (WDLContext) ⚙️
• miniwdl_logger (Optional[logging.Logger]) 📝
• miniwdl_config (Optional[WDL.runtime.config.Loader]) 🧩

Returns — possibly modified bindings to keep going with, which might reference the cache ✨

Return type — WDLBindings

---

✨ toil.wdl.wdltoil.choose_human_readable_directory ✨

toil.wdl.wdltoil.choose_human_readable_directory(root_dir, source_task_path, parent)

This one picks a good, human-friendly directory to save a task's files into, based on the source directory 🌸 The directories involved might not exist yet, so don't panic if they're missing 🙈

Parameters

• root_dir (str) — the directory the path will live under 🏡
• source_task_path (str) — the dotted WDL name of whatever made the file. We assume it's a valid filename component 🧾
• parent (str) — the directory path or parent URI the file came from. If it's a path, it can be absolute (on the worker or leader filesystem) or relative 🗺️

Return type — str

---

✨ toil.wdl.wdltoil.evaluate_decls_to_bindings ✨

toil.wdl.wdltoil.evaluate_decls_to_bindings(decls, all_bindings, standard_library, include_previous=False, drop_missing_files=False, expressions_are_defaults=False)

This one evaluates decls using a given bindings environment and standard library 🧠 It builds a brand-new bindings object that only holds the bindings from the decls you gave it. It also guarantees each decl can see the variables defined by the ones before it 🫶 Main character sequencing, no cap 👑

Parameters

• all_bindings (WDLBindings) — the environment to use when evaluating the decls 🌈
• decls (list[WDL.Tree.Decl]) — the decls to evaluate 📜
• standard_library (ToilWDLStdLibBase) — the standard library 📚
• include_previous (bool) — whether to carry the existing environment into the new one. This is False for outputs, where only the defined decls should show up 🎯
• drop_missing_files (bool) — whether to coerce nonexistent files to null. The coerced elements get checked to make sure the transformation is valid. Right now this should only be turned on in output sections, see https://github.com/openwdl/wdl/issues/673#issuecomment-2248828116 🚨
• expressions_are_defaults (bool) — if True, value expressions in decls are treated as default values, and any values already in the incoming environment can take precedence. If False, each decl is a fresh definition, and expressions are always evaluated and used 💅

Returns — a new bindings object ✨

Return type — WDLBindings

---

✨ toil.wdl.wdltoil.NonDownloadingSize ✨

class toil.wdl.wdltoil.NonDownloadingSize

Bases: WDL.StdLib._Size

This is a WDL size() implementation that avoids downloading files 🙅‍♀️📉 Miniwdl's default one downloads the entire file just to get its size, which is not it for code running on the leader, since there might not be room to fit the whole file. So this class overrides the fancy class that does the work, letting it handle sizes for FileIDs using the size info Toil already stored 🦋

---

✨ toil.wdl.wdltoil.extract_inode_values ✨

toil.wdl.wdltoil.extract_inode_values(environment)

This one grabs a list of every File or Directory object value in the given bindings 📂

Parameters

• environment (WDLBindings) 🌍

Return type — list[str]

---

✨ toil.wdl.wdltoil.extract_inode_virtualized_values ✨

toil.wdl.wdltoil.extract_inode_virtualized_values(environment)

Similar vibe, but this one gets the virtualized values of File/Directory objects in the bindings 🪄 If a value hasn't been virtualized, it just won't show up in the list 🙈

Parameters

• environment (WDLBindings) 🌍

Return type — list[str]

---

✨ toil.wdl.wdltoil.extract_toil_file_uris ✨

toil.wdl.wdltoil.extract_toil_file_uris(environment)

This one gets the toilfile: URIs from the given bindings 🔗 It checks every File in the bindings, and every file sitting inside a Directory in the bindings 🗂️

Parameters

• environment (WDLBindings) 🌍

Return type — Iterable[str]

---

✨ toil.wdl.wdltoil.virtualize_inodes_in_bindings ✨

toil.wdl.wdltoil.virtualize_inodes_in_bindings(environment, file_to_id, file_to_metadata, task_path)

This one fills in the virtualized_value fields for File/Directory objects 🪞✨

Parameters

• environment (WDLBindings) — the bindings to evaluate on. This one is not modified 🛡️
• file_to_id (Dict[str, toil.fileStores.FileID]) — maps an imported URI to the Toil FileID holding the data 🗝️
• file_to_metadata (Dict[str, toil.job.FileMetadata]) — maps a WDL-level file value to metadata about the file, including the URI that would have been imported 🏷️
• task_path (str) 🧭

Returns — a new bindings object with the annotated File objects inside it 💖

Return type — WDLBindings

---

✨ toil.wdl.wdltoil.convert_remote_files ✨

toil.wdl.wdltoil.convert_remote_files(environment, file_source, task_path, search_paths=None, import_remote_files=True, execution_dir=None)

This one resolves relative-URI files in the given environment and imports all the files 🌐✈️

Returns an environment where:

• each File's value is set to the URI it was found at 📍
• its virtualized value is set to whatever it was loaded into the filestore as (if applicable) 📦
• its shared filesystem path is set if it came from the local filesystem 🏠

Parameters

• environment (WDLBindings) — the bindings to evaluate on 🌍
• file_source (toil.jobStores.abstractJobStore.AbstractJobStore) — the context to search for files with 🔍
• task_path (str) — the dotted WDL name of the user-level code doing the importing (probably the workflow name) 👑
• search_paths (Optional[list[str]]) — if set, tries resolving the input location relative to the URLs or directories in this list 🧭
• import_remote_files (bool) — if set, imports files from remote locations. Otherwise, the files stay as URI references 📎
• execution_dir (Optional[str]) 📁

Return type — WDLBindings

---

✨ toil.wdl.wdltoil.ToilWDLStdLibBase ✨

class toil.wdl.wdltoil.ToilWDLStdLibBase(file_store, wdl_options, share_files_with=None)

Bases: WDL.StdLib.Base

The standard library implementation for WDL as run on Toil 📚🩷

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore) 🗄️
• wdl_options (WDLContext) ⚙️
• share_files_with (ToilWDLStdLibBase | None) 🤝

Attributes and properties

• size — the size attribute 📏
• property execution_dir: str — Return type str 📂
• property task_path: str — Return type str 🧭

get_local_paths() — gets all the local paths of files devirtualized (or virtualized) through the stdlib 🗺️ Return type list[str] 🌸

classmethod devirtualize_to(filename, dest_dir, file_source,
wdl_options, devirtualized_to_virtualized,
virtualized_to_devirtualized, export=None)

Okay bestie, this one takes a WDL virtualized filename/URL and either downloads it or exports it into the given directory. 📦✨

⚠️ The dest_dir has to already exist before you call this! And no other devirtualize_to call can be writing into it at the same time, not even another workflow writing the same task to the same spot in the call cache. That's a hard no 💀

It keeps sibling files as siblings and stops files with the same name from clobbering each other 🫶. It also keeps Files or Directories inside Directories in their proper place in the hierarchy. It gets called from inside this class for tasks, and statically at the end of the workflow for outputs. ✨

It returns the local path to the file. If the file is already a local path, or it already has an entry in virtualized_to_devirtualized, that path gets reused instead of making a fresh copy in dest_dir. 🦋

The input filename might already be devirtualized, and if so, it should not be added to the cache. Main character restraint 💅

📋 Parameters

• wdl_options (WDLContext) — WDL options to carry through.
• export (bool | None) — Always creates exported copies of files instead of views that a FileStore might clean up. (docs: /usr/share/doc/python3/html/library/functions.html#bool)
• filename (str) (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• dest_dir (str) (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• file_source (toil.fileStores.abstractFileStore.AbstractFileStore | toil.common.Toil)
• devirtualized_to_virtualized (dict[str, str]) (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)
• virtualized_to_devirtualized (dict[str, str]) (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🌸 class toil.wdl.wdltoil.ToilWDLStdLibWorkflow

class toil.wdl.wdltoil.ToilWDLStdLibWorkflow(*args, **kwargs)

Bases: ToilWDLStdLibBase

This is the standard library implementation for workflow scope. 🌈

It handles deduplicating files made by write_* calls at workflow scope with copies that are already in the call cache, so tasks that depend on them can also be fulfilled straight from the cache. Big brain energy 🧠💖

📋 Parameters

• args (Any)
• kwargs (Any)

---

🧚 class toil.wdl.wdltoil.ToilWDLStdLibTaskCommand

class toil.wdl.wdltoil.ToilWDLStdLibTaskCommand(file_store, container,
wdl_options)

Bases: ToilWDLStdLibBase

This is the standard library implementation you use inside a WDL task command evaluation. 🎀

Heads up: it expects every filename in variable bindings to be a container-side path. These are the "virtualized" filenames, while the "devirtualized" filenames are host-side paths. Keep those two straight or you'll be cooked 💀

📋 Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)
• container (WDL.runtime.task_container.TaskContainer)
• wdl_options (WDLContext)

Member: container

---

👑 class toil.wdl.wdltoil.ToilWDLStdLibTaskOutputs

class toil.wdl.wdltoil.ToilWDLStdLibTaskOutputs(file_store, stdout_path,
stderr_path, file_to_mountpoint, wdl_options, share_files_with=None)

Bases: ToilWDLStdLibBase, WDL.StdLib.TaskOutputs

This is the standard library implementation for WDL running on Toil, with extra functions that are only allowed in task output sections. 🔥

📋 Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)
• stdout_path (str) (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• stderr_path (str) (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• file_to_mountpoint (dict[str, str]) (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)
• wdl_options (WDLContext)
• share_files_with (ToilWDLStdLibBase | None)

✨ stdout_used()

Returns True if the WDL read the standard output. 📤

Return type: bool (docs: /usr/share/doc/python3/html/library/functions.html#bool)

✨ stderr_used()

Returns True if the WDL read the standard error. 📥

Return type: bool (docs: /usr/share/doc/python3/html/library/functions.html#bool)

---

🦋 toil.wdl.wdltoil.evaluate_named_expression

toil.wdl.wdltoil.evaluate_named_expression(context, name, expected_type,
expression, environment, stdlib)

Evaluates an expression when you already know its name. Simple and iconic 🌷

📋 Parameters

• context (WDL.Error.SourceNode | WDL.Error.SourcePosition)
• name (str) (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)
• expected_type (WDL.Type.Base | None)
• expression (WDL.Expr.Base | None)
• environment (WDLBindings)
• stdlib (WDL.StdLib.Base)

Return type: WDL.Value.Base

---

🍓 toil.wdl.wdltoil.evaluate_decl

toil.wdl.wdltoil.evaluate_decl(node, environment, stdlib)

Evaluates the expression of a declaration node, or raises an error if it can't. 🚫

📋 Parameters

• node (WDL.Tree.Decl)
• environment (WDLBindings)
• stdlib (WDL.StdLib.Base)

Return type: WDL.Value.Base

---

💜 toil.wdl.wdltoil.evaluate_call_inputs

toil.wdl.wdltoil.evaluate_call_inputs(context, expressions, environment,
stdlib, inputs_dict=None)

Evaluates a whole batch of named expressions and turns them into a fresh set of bindings. 🧁

inputs_dict is a mapping of variable names to their expected type for the input decls in a task. Basically the cheat sheet for types 📝

📋 Parameters

• context (WDL.Error.SourceNode | WDL.Error.SourcePosition)
• expressions (dict[str, WDL.Expr.Base]) (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)
• environment (WDLBindings)
• stdlib (WDL.StdLib.Base)
• inputs_dict (dict[str, WDL.Type.Base] | None) (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: WDLBindings

---

🌈 toil.wdl.wdltoil.evaluate_defaultable_decl

toil.wdl.wdltoil.evaluate_defaultable_decl(node, environment, stdlib)

If the declaration's name is already defined in the environment, this returns its value. Otherwise, it returns the evaluated expression. It's giving "use what you've got" 🫶

📋 Parameters

• node (WDL.Tree.Decl)
• environment (WDLBindings)
• stdlib (WDL.StdLib.Base)

Return type: WDL.Value.Base

---

⭐ toil.wdl.wdltoil.devirtualize_inodes

toil.wdl.wdltoil.devirtualize_inodes(environment, stdlib)

This makes sure every File value embedded in the given bindings points to a file that command line commands can actually reach. 🗝️ The same virtual file always maps to the same devirtualized filename, even when there are duplicates, so no mix-ups 🙈

📋 Parameters

• environment (WDLBindings)
• stdlib (ToilWDLStdLibBase)

Return type: WDLBindings

---

🫧 toil.wdl.wdltoil.virtualize_inodes

toil.wdl.wdltoil.virtualize_inodes(environment, stdlib,
enforce_existence=True)

This makes sure every File/Directory value embedded in the given bindings points to a file that other machines can use. Multi-machine serve 🌍✨

📋 Parameters

• environment (WDLBindings)
• stdlib (ToilWDLStdLibBase)
• enforce_existence (bool) (docs: /usr/share/doc/python3/html/library/functions.html#bool)

Return type: WDLBindings

---

🗑️ toil.wdl.wdltoil.delete_dead_files

toil.wdl.wdltoil.delete_dead_files(internal_bindings,
live_bindings_list, file_store)

Deletes any files that are in the given bindings but not in the live list. Bye bye, clutter 👋💅

It scans the virtualized values of File and Directory objects anywhere in the bindings. It only tries to delete leaf files, never whole directories, so your folders are safe 🫶

📋 Parameters

• internal_bindings (WDLBindings)
• live_bindings_list (list[WDLBindings]) (docs: /usr/share/doc/python3/html/library/stdtypes.html#list)
• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type: None

---

🌊 toil.wdl.wdltoil.all_parents

toil.wdl.wdltoil.all_parents(path)

Yields all parents of the given path, all the way up to the filesystem root. 🌳

All yielded parents end in "/". If the path is "/", it yields the path itself. Otherwise, if the path ends in "/", it does not yield the path itself. Tiny rules, big payoff 💖

📋 Parameters

• path (str) (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: Iterable[str] (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

💖 toil.wdl.wdltoil.add_paths(task_container, host_paths)

okay so this one is the sorting hat for paths 🧚 It's based off WDL.runtime.task_container.add_paths from miniwdl, but it levels up. For every host path it comes up with a container path, then fills in input_path_map and input_path_map_rev on the TaskContainer so you can go host → container and container → host. ✨

It also makes sure directories always end with a trailing slash, bestie. 📁

Here's the part that matters: File and Directory sibling constraints are a whole thing, so anything that's a child of something on the host has to stay a child of that same thing in the container. MiniWDL's add_paths didn't do this, so Toil fixes it. 🫶 Directories sitting at the top of the included hierarchy also have to stay siblings if they were siblings originally. No drama allowed 💅

📝 TODO: Maybe dedupe this with the CWL mount dedup code that's built around "nonredundant mounts"? But unlike that code, this one has to list every File or Directory mentioned in the input, even if a mount is redundant. Probably. The reason is that Toil isn't sure yet whether those mappings get used for reverse lookups. 🤷‍♀️

Parameters

• task_container (WDL.runtime.task_container.TaskContainer)
• host_paths (Iterable[str])

Return type

None

---

🌸 toil.wdl.wdltoil.drop_if_missing(inode, standard_library)

This one's a bouncer 🚪 If the File or Directory doesn't exist, it returns None. If it does exist, it returns its path. Simple, iconic. 🙈

Parameters

• inode (WDLINode)
• standard_library (ToilWDLStdLibBase)

Return type

WDLINode | None

---

🦋 toil.wdl.wdltoil.missing_inode_dropper(standard_library)

This one hands you a function that nulls out any missing File or Directory values 🫧 The function needs a standard library so it knows which current directory to use when it turns strings into paths. Very thoughtful of it, ngl. 💜

Parameters

• standard_library (ToilWDLStdLibBase)

Return type

INodeTransform

---

🍓 toil.wdl.wdltoil.drop_missing_files(environment, standard_library)

Okay so this one makes sure every File value stuffed inside the bindings either points to a file that actually exists, or is null. Zero ghost files allowed 👻 Heads up: the Files here must not be virtualized. ⚠️

Parameters

• environment (WDLBindings)
• standard_library (ToilWDLStdLibBase)

Return type

WDLBindings

---

🧚 toil.wdl.wdltoil.get_paths_in_bindings(environment)

This one collects the paths of every File and Directory in the bindings and gives you back a list with no duplicates. Clean girl energy ✨

📝 TODO: This is kinda duplicative with WDL.runtime.task._fspaths. Noted, bestie. 🫶

Parameters

• environment (WDLBindings)

Return type

list[str]

---

🎀 toil.wdl.wdltoil.map_over_inodes_in_bindings(environment, transform)

Runs every File value in the bindings through your transformation function 💅 The transform is not allowed to mutate the original File, so don't you dare 😤

📝 TODO: Swap this out for WDL.Value.rewrite_env_paths or WDL.Value.rewrite_files eventually. 🌈

Parameters

• environment (WDLBindings)
• transform (INodeTransform)

Return type

WDLBindings

---

💜 toil.wdl.wdltoil.map_over_inodes_in_binding(binding, transform)

Same energy as the one above, but for a single binding 🧸 It runs all File values' types and values inside that binding's value through the transform. Again, the transform must not mutate the original File. Iykyk. 🙃

Parameters

• binding (WDL.Env.Binding[WDL.Value.Base])
• transform (INodeTransform)

Return type

WDL.Env.Binding[WDL.Value.Base]

---

🫧 toil.wdl.wdltoil.remove_expr_from_value(value)

Takes a WDL value and gives you back a brand new one with the expr field stripped out 🪄 Think of it as a glow-up with no expression attached. 💖

Parameters

• value (WDL.Value.Base): the original WDL value

Return type

WDL.Value.Base: a new WDL value without the expr field

---

🌷 toil.wdl.wdltoil.map_over_typed_value(value, transform)

This one's the main character 👑 It applies a transform to a WDL value and every WDL value it contains. The transform takes a WDL value and hands back a new one. Recursive slay 🔥

Parameters

• value (WDL.Value.Base): the WDL value to transform
• transform (Callable[[WDL.Value.Base], WDL.Value.Base]): takes a WDL value and returns a new WDL value

Return type

WDL.Value.Base: the new transformed WDL value

---

🍒 toil.wdl.wdltoil.map_over_typed_inodes_in_value(value, transform)

Runs all File values embedded in the given value through the transformation function 🎬 Same golden rule: the transform must not mutate the original File. 🚫

Here's the tea ☕ If the transform returns None, that file value gets changed to Null. The transform also gets access to type info, so it knows whether it's allowed to return None (which depends on whether the value is optional or not).

⚠️ The transform may only return None if the mapping result won't actually get used. This is what lets scans work. That means any error checking has to live inside the transform itself. Be careful out there, babe 🫶

Parameters

• value (WDL.Value.Base)
• transform (INodeTransform)

Return type

WDL.Value.Base

---

🌙 toil.wdl.wdltoil.ensure_null_inodes_are_nullable(value, original_value, expected_type)

This is the bodyguard for null values 🛡️ It walks through every nested value inside value and checks that each null is valid. If it finds a null that doesn't have a matching expected_type that's allowed to be null, it raises an error. 😭

Fun side quest: this function is currently only used to check that nulls coming from File/Directory coercion sit in places with a nullable type. If someone uses it somewhere else, the error message needs updating so it describes the right types instead of just talking about files. Noted, bestie 📝

For example: if one of the nested values is null but the matching nested expected_type isn't optional, a FileNotFoundError gets raised. 🚨

Parameters

• value (WDL.Value.Base): the WDL base value to check. This is the transformed value that has the null elements.
• original_value (WDL.Value.Base): the original WDL base value from before the transformation. Only used for error messages.
• expected_type (WDL.Type.Base): the WDL type of the value

Return type

None

---

🧸 class toil.wdl.wdltoil.WDLBaseJob(wdl_options, **kwargs)

Bases: toil.job.Job

This is the base class for every WDL-related job, so it's giving foundation 🏗️ It handles post-processing of returned bindings, like adding in null values for things a section didn't define. You can queue up post-processing on any job before it gets saved, and it'll be applied as long as the job's run method calls postprocess(). It also remembers the Toil WDL configuration keys and values. 💅

Parameters

• wdl_options (WDLContext)
• kwargs (Any)

---

💕 run(file_store)

Runs a WDL-related job. ✨ If you're writing a non-trivial override, remember to decorate it with report_wdl_errors(). Future-you will thank present-you 🫶

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type

Any

---

🌈 then_underlay(underlay)

Applies an underlay of backup bindings to the result. Think of it as a safety net 🕸️

Parameters

• underlay (toil.job.Promised[WDLBindings])

Return type

None

---

🗑️ then_remove(remove)

Removes the given bindings from the result. Byebye 👋

Parameters

• remove (toil.job.Promised[WDLBindings])

Return type

None

---

🏷️ then_namespace(namespace)

Puts the result bindings into a namespace. Organized girlies only 📂

Parameters

• namespace (str)

Return type

None

---

🎀 then_overlay(overlay)

Overlays the given bindings on top of the result, which might already be namespaced. Layering is an art 🎨

Parameters

• overlay (toil.job.Promised[WDLBindings])

Return type

None

---

🫧 remove_expr_from_bindings(bindings)

This one doesn't come with a description, so here are just the facts 🤍

Parameters

• bindings (WDLBindings)

Return type

WDLBindings

---

🔮 postprocess(bindings)

Applies any queued changes to the bindings 🪄 Subclasses' run() implementations should call this on their return values.

Parameters

• bindings (WDLBindings)

Return type

WDLBindings

---

🦄 defer_postprocessing(other)

Hands our postprocessing steps off to a different job. Use this when you're returning a promise for bindings from the job that issues the promise. Passing the baton 🏃‍♀️💨

Parameters

• other (WDLBaseJob)

Return type

None

---

🌟 class toil.wdl.wdltoil.WDLTaskWrapperJob(task, prev_node_results, enclosing_bindings, task_id, wdl_options, **kwargs)

Bases: WDLBaseJob

This job figures out the resources a WDL job needs to run. Resource planning, but make it aesthetic 📋💖

Okay so this one is the bouncer of the whole operation 💅 It checks the input declarations for any inputs nobody specified 🦋, evaluates the runtime section 🧚, and then either schedules the real WDL job or chains over to it ✨ Quick reminder: all bindings are in terms of task-internal names 🎀

Parameters:

• task — WDL.Tree.Task 🌸
• prev_node_results — Sequence[toil.job.Promised[WDLBindings]] 🌈
• enclosing_bindings — WDLBindings 💜
• task_id — list[str] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓
• wdl_options — WDLContext ✨
• kwargs — Any 🫶

💖 run(file_store)

Evaluate inputs and runtime and schedule the task 🔥

Parameters:

• file_store — toil.fileStores.abstractFileStore.AbstractFileStore (anchor: #toil.fileStores.abstractFileStore.AbstractFileStore) 🦋

Return type: toil.job.Promised[WDLBindings] 💅

---

👑 class WDLTaskJob

class toil.wdl.wdltoil.WDLTaskJob(task, task_internal_bindings,
runtime_bindings, enclosing_bindings, task_id, mount_spec, wdl_options,
cache_key=None, **kwargs)

Bases: WDLBaseJob 🌸

Job that runs a WDL task. 💖

This one is the main character 👑 It's responsible for re-evaluating input declarations for unspecified inputs, evaluating the runtime section, re-scheduling if resources are not available, running any command, and evaluating the outputs 🔥 Same deal as before: all bindings are in terms of task-internal names 🎀

Parameters:

• task — WDL.Tree.Task 🌸
• task_internal_bindings — toil.job.Promised[WDLBindings] 🌈
• runtime_bindings — toil.job.Promised[WDLBindings] ✨
• enclosing_bindings — WDLBindings 💜
• task_id — list[str] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list, /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓
• mount_spec — dict[str | None, int] (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/functions.html#int) 🦋
• wdl_options — WDLContext ✨
• cache_key — str | None (docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🧚
• kwargs — Any 🫶

Class attribute:

INJECTED_MESSAGE_DIR = '.toil_wdl_runtime'

🌸 add_injections(command_string, task_container)

Inject extra Bash code from the Toil WDL runtime into the command for the container. Currently doesn't implement the MiniWDL plugin system, but does add resource usage monitoring to Docker containers 🐳

Parameters:

• command_string — str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 💖
• task_container — WDL.runtime.task_container.TaskContainer 🌈

Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str) ✨

🧚 handle_injection_messages(outputs_library)

Handle any data received from injected runtime code in the container 💅

Parameters:

• outputs_library — ToilWDLStdLibTaskOutputs (anchor: #toil.wdl.wdltoil.ToilWDLStdLibTaskOutputs) 🎀

Return type: None 🌙

🦋 handle_message_file(file_path)

Handle a message file received from in-container injected code. Takes the host-side path of the file 🫶

Parameters:

• file_path — str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🍓

Return type: None ✨

🔥 can_fake_root()

Determine if --fakeroot is likely to work for Singularity 🙈

Return type: bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) 💜

💀 can_mount_proc()

Determine if --containall will work for Singularity. On Kubernetes, this will result in operation not permitted 😭

See: https://github.com/apptainer/singularity/issues/5857

So if Kubernetes is detected, return False :return: bool 🙈

Return type: bool (docs: /usr/share/doc/python3/html/library/functions.html#bool) ✨

🌈 ensure_mount_point(file_store, mount_spec)

Ensure the mount point sources are available 🦋 It will check if the mount point source has the requested amount of space available 🌸

Note: We are depending on Toil's job scheduling backend to error when the sum of multiple mount points disk requests is greater than the total available. For example, if a task has two mount points request 100 GB each but there is only 100 GB available, the df check may pass but Toil should fail to schedule the jobs internally 😭

Parameters:

• mount_spec — dict[str | None, int] (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str, /usr/share/doc/python3/html/library/functions.html#int) — this is the mount specification from the disks attribute in the WDL task. It's a dict where the key is the mount point target and the value is the size 💖
• file_store — toil.fileStores.abstractFileStore.AbstractFileStore (anchor: #toil.fileStores.abstractFileStore.AbstractFileStore) — the file store used to create a tmp directory for the mount point source 🧚

Returns: a dict mapping mount point target to mount point source 🎀

Return type: dict[str, str] (docs: /usr/share/doc/python3/html/library/stdtypes.html#dict, /usr/share/doc/python3/html/library/stdtypes.html#str) 👑

🫶 run(file_store)

Actually run the task 🔥

Parameters:

• file_store — toil.fileStores.abstractFileStore.AbstractFileStore (anchor: #toil.fileStores.abstractFileStore.AbstractFileStore) 🍓

Return type: toil.job.Promised[WDLBindings] ✨

---

🌷 class WDLWorkflowNodeJob

class toil.wdl.wdltoil.WDLWorkflowNodeJob(node, prev_node_results,
wdl_options, **kwargs)

Bases: WDLBaseJob 💅

Job that evaluates a WDL workflow node. 🌸

Parameters:

• node — WDL.Tree.WorkflowNode 🦋
• prev_node_results — Sequence[toil.job.Promised[WDLBindings]] 🌈
• wdl_options — WDLContext ✨
• kwargs — Any 🫶

💜 run(file_store)

Actually execute the workflow node 🔥

Parameters:

• file_store — toil.fileStores.abstractFileStore.AbstractFileStore (anchor: #toil.fileStores.abstractFileStore.AbstractFileStore) 🧚

Return type: toil.job.Promised[WDLBindings] 💖

---

🍓 class WDLWorkflowNodeListJob

class toil.wdl.wdltoil.WDLWorkflowNodeListJob(nodes, prev_node_results,
wdl_options, **kwargs)

Bases: WDLBaseJob 🎀

Job that evaluates a list of WDL workflow nodes, which are in the same scope and in a topological dependency order, and which do not call out to any other workflows or tasks or sections. ✨

Parameters:

• nodes — list[WDL.Tree.WorkflowNode] (docs: /usr/share/doc/python3/html/library/stdtypes.html#list) 💅
• prev_node_results — Sequence[toil.job.Promised[WDLBindings]] 🌈
• wdl_options — WDLContext ✨
• kwargs — Any 🫶

🦋 run(file_store)

Actually execute the workflow nodes 🔥

Parameters:

• file_store — toil.fileStores.abstractFileStore.AbstractFileStore (anchor: #toil.fileStores.abstractFileStore.AbstractFileStore) 🌸

Return type: toil.job.Promised[WDLBindings] 👑

---

🌈 class WDLCombineBindingsJob

class toil.wdl.wdltoil.WDLCombineBindingsJob(prev_node_results,
**kwargs)

Bases: WDLBaseJob 💜

Job that collects the results from WDL workflow nodes and combines their environment changes. 🧚

Parameters:

• prev_node_results — Sequence[toil.job.Promised[WDLBindings]] 🌈
• kwargs — Any 🫶

✨ run(file_store)

Aggregate incoming results 💅

Parameters:

• file_store — toil.fileStores.abstractFileStore.AbstractFileStore (anchor: #toil.fileStores.abstractFileStore.AbstractFileStore) 🍓

Return type: WDLBindings 🎀

---

🦋 class WDLWorkflowGraph

class toil.wdl.wdltoil.WDLWorkflowGraph(nodes)

Represents a graph of WDL WorkflowNodes. 🌸

Operates at a certain level of instantiation (i.e. sub-sections are represented by single nodes) 🔥

Assumes all relevant nodes are provided. Dependencies outside the provided nodes are assumed to be satisfied already 💖

Parameters:

• nodes — Sequence[WDL.Tree.WorkflowNode] 🌈

🫶 real_id(node_id)

Map multiple IDs for what we consider the same node to one ID. This elides/resolves gathers 🦋

Parameters:

• node_id — str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str) 🧚

Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str) ✨

💀 is_decl(node_id)

Return True if a node represents a WDL declaration, and false otherwise 🙈

Parameters

• node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) 💖

Return type bool </usr/share/doc/python3/html/library/functions.html#bool> ✨

🌸 get(node_id)

Okay so this one is the "gimme that node" move 🍓 It grabs a node by its ID.

Parameters

• node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type WDL.Tree.WorkflowNode

🦋 get_dependencies(node_id)

Get all the nodes that a node depends on, recursively (into the node if it has a body) but not transitively. It digs into the node if there's a body 🧠 but it doesn't go down the whole rabbit hole, bestie 🙈

It only hands back dependencies after gathers and internal-to-section dependencies are resolved, and only for nodes that are also in this graph ✨

Parameters

• node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

🔮 get_transitive_dependencies(node_id)

Get all the nodes that a node depends on, transitively. This is the full chain, no cap 👑

Parameters

• node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

Return type set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

🧭 topological_order()

Get a topological order of the nodes, based on their dependencies. It's giving "who goes first" energy 🎀

Return type list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

🍃 leaves()

Get all the workflow node IDs that have no dependents in the graph. These are the end-of-the-line nodes, the ones nothing else is waiting on 🥹

Return type list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

---

🧚 class toil.wdl.wdltoil.WDLSectionJob(wdl_options, **kwargs)

Bases: WDLBaseJob

A job that can create more graph for a section of the workflow. Main character energy for sections 💅

Parameters

• wdl_options (WDLContext)
• kwargs (Any)

✨ static coalesce_nodes(order, section_graph)

Given a topological order of WDL workflow node IDs, this produces a list of lists of IDs. Everything stays in topological order, and each inner list can run under a single Toil job, so it's basically bundling the tasks up for the group chat 🫶

Parameters

• order (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])
• section_graph (WDLWorkflowGraph <#toil.wdl.wdltoil.WDLWorkflowGraph>)

Return type list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

🌈 create_subgraph(nodes, gather_nodes, environment, local_environment=None, subscript=None)

Make a Toil job to evaluate a subgraph inside a workflow or workflow section. It's giving full glam 💖

Returns

A child Job that will return the aggregated environment after running all the things in the section.

Parameters

• gather_nodes (Sequence[WDL.Tree.Gather]) — Names exposed by these will always be defined with something, even if the code that defines them does not actually run. Never left on read 👀
• environment (WDLBindings) — Bindings in this environment will be used to evaluate the subgraph and will be passed through.
• local_environment (WDLBindings | None) — Bindings in this environment will be used to evaluate the subgraph but will go out of scope at the end of the section. Temporary, but still valid 🦋
• subscript (int </usr/share/doc/python3/html/library/functions.html#int> | None) — If the subgraph is being evaluated multiple times, this should be a disambiguating integer for logging. Helps tell the copies apart in the logs 🔥
• nodes (Sequence[WDL.Tree.WorkflowNode])

Return type WDLBaseJob <#toil.wdl.wdltoil.WDLBaseJob>

🎀 make_gather_bindings(gathers, undefined)

Given a collection of Gathers, this creates bindings from every identifier gathered to the given "undefined" placeholder. That placeholder would be Null for a single execution of the body, or an empty array for a completely unexecuted scatter 🧸

These bindings can be overlaid with bindings from the actual execution. That way, references to names defined in unexecuted code get a proper default undefined value instead of a KeyError at runtime. Nobody gets crashed on, periodt 💅

The info for all this comes from MiniWDL's "gathers" system: <<https://miniwdl.readthedocs.io/en/latest/WDL.html#WDL.Tree.WorkflowSection.gathers>>

⚠️ TODO: This approach will scale O(n^2) when run on n nested conditionals, because generating these bindings for the outer conditional will visit all the bindings from the inner ones. Not the vibe, so it's on the to-do list 😭

Parameters

• gathers (Sequence[WDL.Tree.Gather])
• undefined (WDL.Value.Base)

Return type WDLBindings

---

🎀 class toil.wdl.wdltoil.WDLScatterJob(scatter, prev_node_results, wdl_options, **kwargs)

Bases: WDLSectionJob

A job that evaluates a scatter in a WDL workflow. It runs the body for each value in an array, then makes arrays of the new bindings created in each instance of the body. If an instance of the body doesn't create a binding, it gets a null value in the matching array spot 🍓

Parameters

• scatter (WDL.Tree.Scatter)
• prev_node_results (Sequence[toil.job.Promised[WDLBindings]])
• wdl_options (WDLContext)
• kwargs (Any)

🚀 run(file_store)

Run the scatter. Let's gooo 🔥

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

Return type toil.job.Promised[WDLBindings]

---

🦋 class toil.wdl.wdltoil.WDLArrayBindingsJob(input_bindings, base_bindings, **kwargs)

Bases: WDLBaseJob

A job that takes all the new bindings created in an array of input environments, relative to a base environment, and produces bindings where each new binding name is bound to an array of the values from all the input environments. Ngl this is the scatter-results glow-up 🌸

It's useful for producing the results of a scatter 💜

Parameters

• input_bindings (Sequence[toil.job.Promised[WDLBindings]])
• base_bindings (WDLBindings)
• kwargs (Any)

🧁 run(file_store)

Actually produce the array-ified bindings now that the promised values are available. Finally, the reveal ✨

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

Return type WDLBindings

---

🌷 class toil.wdl.wdltoil.WDLConditionalJob(conditional, prev_node_results, wdl_options, **kwargs)

Bases: WDLSectionJob

A job that evaluates a conditional in a WDL workflow. It's the "if this, then that" queen 👑

Parameters

• conditional (WDL.Tree.Conditional)
• prev_node_results (Sequence[toil.job.Promised[WDLBindings]])
• wdl_options (WDLContext)
• kwargs (Any)

🎈 run(file_store)

Run the conditional. Bestie, it's giving decision time 🫶

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

Return type toil.job.Promised[WDLBindings]

---

🌟 class toil.wdl.wdltoil.WDLWorkflowJob(workflow, prev_node_results, enclosing_bindings, workflow_id, wdl_options, **kwargs)

Bases: WDLSectionJob

A job that evaluates an entire WDL workflow. The whole production, start to finish 🎬

Parameters

• workflow (WDL.Tree.Workflow)
• prev_node_results (Sequence[toil.job.Promised[WDLBindings]])
• enclosing_bindings (WDLBindings)
• workflow_id (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])
• wdl_options (WDLContext)
• kwargs (Any)

🏁 run(file_store)

Run the workflow, then return the result of the workflow. Served, every time 💅

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

Return type toil.job.Promised[WDLBindings]

---

🧸 class toil.wdl.wdltoil.WDLOutputsJob(workflow, bindings, enclosing_bindings, wdl_options, cache_key=None, **kwargs)

Bases: WDLBaseJob

A job that evaluates an outputs section for a workflow. This is where the final results get packaged up and handed back, so it's the big finish 🎁✨

Returns an environment with just the outputs bound, in no namespace. 💅✨ It's the outputs-only env, no namespace drama 🎀

Parameters

• workflow (WDL.Tree.Workflow)
• bindings (toil.job.Promised[WDLBindings])
• enclosing_bindings (WDLBindings)
• wdl_options (WDLContext)
• cache_key (str | None)
• kwargs (Any)

run(file_store) 🔥 Make bindings for the outputs.

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type: WDLBindings

---

💖 class toil.wdl.wdltoil.WDLStartJob(target, inputs, wdl_options, **kwargs)

Bases: WDLSectionJob

Okay bestie, this one is the main event 👑 It's a job that evaluates an entire WDL workflow and returns the workflow outputs, namespaced with the workflow name. Inputs can be namespaced or not, and both forms are accepted 🫶

Parameters

• target (WDL.Tree.Workflow | WDL.Tree.Task)
• inputs (toil.job.Promised[WDLBindings])
• wdl_options (WDLContext)
• kwargs (Any)

run(file_store) ✨ Actually build the subgraph.

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type: toil.job.Promised[WDLBindings]

---

🦋 class toil.wdl.wdltoil.WDLInstallImportsJob(task_path, inputs, import_data, **kwargs)

Bases: toil.job.Job

A unit of work in toil, obviously 🧚

Parameters

• task_path (str)
• inputs (WDLBindings)
• import_data (toil.job.Promised[Tuple[Dict[str, toil.fileStores.FileID], Dict[str, toil.job.FileMetadata]]])
• kwargs (Any)

run(file_store) 🌸 Convert the filenames in the workflow inputs into URIs.

:return: Promise of transformed workflow inputs

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type: toil.job.Promised[WDLBindings]

---

🎀 class toil.wdl.wdltoil.WDLImportWrapper(target, inputs, wdl_options, inputs_search_path, import_remote_files, import_workers_batchsize, import_workers_disk, **kwargs)

Bases: WDLSectionJob

This job is the one that organises file importing onto workers instead of the leader 🙌 It's in charge of pulling out filenames and metadata, calling ImportsJob, applying the imports to the input bindings, and scheduling the start workflow job. Iconic multitasking 💅

This class is only used when runImportsOnWorkers is enabled.

Parameters

• target (Union[WDL.Tree.Workflow, WDL.Tree.Task])
• inputs (WDLBindings)
• wdl_options (WDLContext)
• inputs_search_path (list[str])
• import_remote_files (bool)
• import_workers_batchsize (toil.job.ParseableIndivisibleResource)
• import_workers_disk (toil.job.ParseableIndivisibleResource)
• kwargs (Any)

run(file_store) 🍓 Run a WDL-related job.

Remember to decorate non-trivial overrides with report_wdl_errors(). Don't skip this, babe 🙈

Parameters

• file_store (toil.fileStores.abstractFileStore.AbstractFileStore)

Return type: toil.job.Promised[WDLBindings]

---

🌈 toil.wdl.wdltoil.make_root_job(target, inputs, inputs_search_path, toil, wdl_options, options)

Parameters

• target (WDL.Tree.Workflow | WDL.Tree.Task)
• inputs (WDLBindings)
• inputs_search_path (list[str])
• toil (toil.common.Toil)
• wdl_options (WDLContext)
• options (configargparse.Namespace)

Return type: WDLSectionJob

---

⭐ toil.wdl.wdltoil.main()

A Toil workflow to interpret WDL input files. Main character energy 👑

Return type: None

---

✨ toil.worker ✨

💜 Attributes

• logger

🌸 Classes

• StatsDict — Subclass of MagicExpando for type-checking purposes.

🧚 Functions

• nextChainable(predecessor, job_store, config) — Returns the next chainable job's JobDescription after the given predecessor.
• unstick_worker([interval, timeout]) — Thread function that tries to prevent the process from getting stuck. 🫶
• workerScript(job_store, config, job_name, job_store_id) — Worker process script, runs a job.
• parse_args(args) — Parse command-line arguments to the worker.
• in_contexts(contexts) — Unpickle and enter all the pickled, base64-encoded context managers in the…
• main([argv])

🔥 Module Contents

toil.worker.logger

class toil.worker.StatsDict(*args, **kwargs)

Bases: toil.lib.expando.MagicExpando

Subclass of MagicExpando for type-checking purposes. Literally just vibes 💅

jobs: list[toil.lib.expando.MagicExpando]

toil.worker.nextChainable(predecessor, job_store, config)

Returns the next chainable job's JobDescription after the given predecessor JobDescription, if one exists, or None if the chain must terminate. 🦋

Parameters

• predecessor (toil.job.JobDescription) — The job to chain from
• job_store (toil.jobStores.abstractJobStore.AbstractJobStore) — The JobStore to fetch JobDescriptions from.
• config (toil.common.Config) — The configuration for the current run.

Return type: Optional[toil.job.JobDescription]

toil.worker.unstick_worker(interval=120, timeout=120)

Thread function that tries to prevent the process from getting stuck. 🙈

Meant to be used as a daemon thread: it has no shutdown signal, but it cleans up on exit. ✨

Parameters

• interval (float) — Try to unstick the process at intervals of this many seconds.
• timeout (float) — Stop child processes that take longer than this many seconds to finish.

Return type: None

toil.worker.workerScript(job_store, config, job_name, job_store_id, redirect_output_to_log_file=True, local_worker_temp_dir=None, debug_flags=None)

Worker process script, runs a job. 🫶

Parameters

• job_store (toil.jobStores.abstractJobStore.AbstractJobStore) — this is the JobStore we fetch the JobDescriptions from 🗄️
• config (toil.common.Config) — the configuration for the current run ⚙️
• job_name (str) — the "job name" 🏷️, which is the user-friendly name of the job that's about to run 🌸
• job_store_id (str) — the job store ID of the job to be run 🆔
• redirect_output_to_log_file (bool) — if this is False, logs go straight to the console instead of capturing the job's output 📺
• local_worker_temp_dir (Optional[str]) — the directory the worker gets to work in 📁

heads up bestie 💀 this directory may be recursively removed after the job runs, so don't store anything precious in there.

• debug_flags (Optional[set[str]]) — flags that get set on each job before it runs 🐛

Return int — 1 if a job failed, or 0 if all jobs succeeded 🎉

Return type — int 💜

---

toil.worker.parse_args(args) ✨

Okay so this one parses the command-line arguments for the worker 🧚

Parameters

• args (list[str])

Return type — Any 🌈

---

toil.worker.in_contexts(contexts) 🦋

This one is giving unpack-and-enter energy. It unpickles and enters all the pickled, base64-encoded context managers in the given list, then runs the body, then leaves them all 💅

Parameters

• contexts (list[str])

Return type — collections.abc.Iterator[None] 👑

---

toil.worker.main(argv=None) 💖

Parameters

• argv (Optional[list[str]])

Return type — None 🫶

---

✨ Attributes ✨

• log 🪵
• KNOWN_EXTANT_IMAGES 🌸

---

✨ Exceptions ✨

• ApplianceImageNotFound — error raised when using TOIL_APPLIANCE_SELF results in an HTTP error 😭

✨ Functions ✨

okay bestie, this table is the whole lineup of helpers in toil 💅 here's the tea:

• which(cmd[, mode, path]) — returns the path that conforms to the given mode on the Path 🧭
• toilPackageDirPath() — gives you the absolute path of the directory for the top-level toil package 📂
• inVirtualEnv() — checks if you're inside a virtualenv or Conda virtual environment 🐍
• resolveEntryPoint(entryPoint) — finds the path to the entry point that should work on a worker 👷
• physicalMemory() — calculates the total physical memory, in bytes 🧠
• physicalDisk(directory) — no blurb on this one bestie 🙈
• applianceSelf([forceDockerAppliance]) — returns the fully qualified name of the Docker image that Toil appliance containers start from 🐳
• customDockerInitCmd() — returns the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable 🌸
• customInitCmd() — returns the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable 🌸
• lookupEnvVar(name, envName, defaultValue) — looks up the environment variables that control Toil and logs what it found 🔍
• checkDockerImageExists(appliance) — attempts to check a url registryName for a docker image with a given tag 🔎
• parseDockerAppliance(appliance) — pulls out the parsed registry, image reference, and tag from a docker image string 🧩
• checkDockerSchema(appliance) — no blurb for this one either 💀
• requestCheckRegularDocker(origAppliance, registryName, ...) — checks if an image exists using the requests library 📡
• requestCheckDockerIo(origAppliance, imageName, tag) — checks docker.io to see if an image exists using the requests library 📡
• logProcessContext(config) — no blurb here either, sorry bestie 💀

---

✨ Package Contents ✨

okay so this is where we get into the actual goods 🎀

• toil.log — listed as a module, and that's all it says 🪵

toil.which(cmd, mode=os.F_OK | os.X_OK, path=None)

this one was copy-pasted straight from python3.6's shutil.which() 📋 it returns the path that conforms to the given mode on the Path.

mode defaults to os.F_OK | os.X_OK. path defaults to the result of os.environ.get("PATH"), but you can override it with your own custom search path 🛤️

Returns: the path found, or None.

Return type: Optional[str] (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.toilPackageDirPath()

Returns the absolute path of the directory that corresponds to the top-level toil package 📂

the return value is guaranteed to end in '/toil', so you can count on that one 💖

Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.inVirtualEnv()

Tests if we're inside a virtualenv or Conda virtual environment 🐍

Return type: bool (docs: /usr/share/doc/python3/html/library/functions.html#bool)

toil.resolveEntryPoint(entryPoint)

Finds the path to the given entry point that should work on a worker 👷

Returns: the path found, which may be an absolute or a relative path.

Parameters: entryPoint (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

toil.physicalMemory()

Calculates the total amount of physical memory, in bytes 🧠 the vibes check out in this example:

>>> n = physicalMemory()
>>> n > 0
True
>>> n == physicalMemory()
True

Return type: int (docs: /usr/share/doc/python3/html/library/functions.html#int)

toil.physicalDisk(directory)

No blurb on this one, bestie 🙈

Parameters: directory (str, docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

Return type: int (docs: /usr/share/doc/python3/html/library/functions.html#int)

toil.applianceSelf(forceDockerAppliance=False)

Returns the fully qualified name of the Docker image that Toil appliance containers start from 🐳 the answer depends on the current version of Toil and three environment variables: TOIL_DOCKER_REGISTRY, TOIL_DOCKER_NAME and TOIL_APPLIANCE_SELF. here's how they work:

• TOIL_DOCKER_REGISTRY — picks an account on a publicly hosted docker registry like Quay or Docker Hub 🏪 the default is UCSC's CGL account on Quay.io, where the Toil team publishes the official appliance images
• TOIL_DOCKER_NAME — the base name of the image 🏷️ the default of toil is adequate in most cases
• TOIL_APPLIANCE_SELF — fully qualifies the appliance image, complete with registry, image name and version tag 👑 it overrides both TOIL_DOCKER_NAME and TOIL_DOCKER_REGISTRY, as well as the version tag of the image

setting TOIL_APPLIANCE_SELF won't be necessary in most cases, so don't stress about it 🫶

Parameters: forceDockerAppliance (bool, docs: /usr/share/doc/python3/html/library/functions.html#bool)

Return type: str (docs: /usr/share/doc/python3/html/library/stdtypes.html#str)

💖 toil.customDockerInitCmd()

okay so this one just hands back whatever you stashed in the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable 🧚 it's giving custom setup energy ✨

The custom docker command runs before the workers and/or the primary node's services start up 🔥

It's lowkey so useful for custom initialization on instances, like logging into private docker registries 🔐 Any single quotes get escaped for you, but the command can't include blacklisted chars (newline or tab) 🙅

Returns The custom command, or an empty string if the environment variable isn't set 🍓

Return type str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🌸 toil.customInitCmd()

this one gives you back the command set in TOIL_CUSTOM_INIT_COMMAND 💜

The custom init command runs before the Toil appliance itself in workers and/or the primary node. Basically it goes one stage before TOIL_CUSTOM_DOCKER_INIT_COMMAND, so the order is giving main character energy 👑

Same vibes as the docker one: super handy for custom initialization on instances (like authenticating to private docker registries 🫶). Single quotes are escaped, and the command can't have blacklisted chars (newline or tab) 🚫

Returns The custom command, or an empty string if the environment variable isn't set 🧁

Return type str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🧚 toil.lookupEnvVar(name, envName, defaultValue)

this one is the env var detective 🕵️‍♀️ it looks up the environment variables that control Toil and logs whatever it finds 📝

Parameters

• name (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the human readable name of the variable 🌷
• envName (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the name of the environment variable to look up 🔎
• defaultValue (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the fall-back value 🛟

Returns the value of the environment variable, or the default value if the variable is not set 💅

Return type str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🐳 toil.checkDockerImageExists(appliance)

bestie, this one checks a registry URL for a docker image with a given tag 👀

Parameters

• appliance (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the url of a docker image's registry (with a tag), in one of these shapes: 'quay.io/<repo_path>:<tag>' or '<repo_path>:<tag>' 🍒
  • Examples: 'quay.io/ucsc_cgl/toil:latest', 'ubuntu:latest', or 'broadinstitute/genomes-in-the-cloud:2.0.0' ✨

Returns This one raises an exception if the docker image can't be found or is invalid 💀 otherwise it returns the appliance string 💖

Return type str (/usr/share/doc/python3/html/library/stdtypes.html#str)

---

🍓 toil.parseDockerAppliance(appliance)

this one is the image splitter 🔪 it takes a docker image string and derives the parsed registry, image reference, and tag 🧩

Example: "quay.io/ucsc_cgl/toil:latest" should return "quay.io", "ucsc_cgl/toil", "latest" 🌈

Defaults, because we love a backup plan 🫶:

• If a registry isn't defined, the default is "docker.io" 🐋
• If a tag isn't defined, the default is "latest" 🏷️

Parameters

• appliance (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the full url of the docker image originally specified by the user (or the default), e.g. "quay.io/ucsc_cgl/toil:latest" 🎀

Returns registryName, imageName, tag 🧁

Return type tuple[str, str, str] (/usr/share/doc/python3/html/library/stdtypes.html#tuple, /usr/share/doc/python3/html/library/stdtypes.html#str)

---

🦄 toil.checkDockerSchema(appliance)

this one is just a bare signature with no description, so there's nothing extra to say here 🙈

---

🚨 exception toil.ApplianceImageNotFound(origAppliance, url, statusCode)

Bases: docker.errors.ImageNotFound

this is the error you get when using TOIL_APPLIANCE_SELF results in an HTTP error 😭

Parameters

• origAppliance (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the full url of the docker image originally specified by the user (or the default), e.g. "quay.io/ucsc_cgl/toil:latest" 🍓
• url (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the URL where the image's manifest is supposed to appear 🔗
• statusCode (int, /usr/share/doc/python3/html/library/functions.html#int) — the failing HTTP status code returned by the URL 🙅‍♀️

---

🌈 toil.KNOWN_EXTANT_IMAGES

another bare one with no description, so we're just gonna vibe past it 💅

---

🔍 toil.requestCheckRegularDocker(origAppliance, registryName, imageName, tag)

this one checks if an image exists using the requests library 🧠

The URL is based on the docker v2 schema (https://docs.docker.com/registry/spec/manifest-v2-2/) and has this format: https://{websitehostname}.io/v2/{repo}/manifests/{tag} 🦋

Heads up, bestie: this does not work with the official docker.io site, because they require an OAuth token. That's why docker.io images get their own separate check 🚨

Parameters

• origAppliance (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the full url of the docker image originally specified by the user (or the default), e.g. quay.io/ucsc_cgl/toil:latest 💖
• registryName (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the url of a docker image's registry, e.g. quay.io 🌷
• imageName (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the image, including path and excluding the tag, e.g. ucsc_cgl/toil 🧁
• tag (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the tag used at that docker image's registry, e.g. latest 🏷️

Raises ApplianceImageNotFound if no match is found 😭

Returns Returns True if a match is found ✨

Return type bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

🐋 toil.requestCheckDockerIo(origAppliance, imageName, tag)

this one checks docker.io to see if an image exists, also with the requests library 🫧

The URL is based on the docker v2 schema, but there's a catch: you have to fetch an access token first 🔑

Parameters

• origAppliance (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the full url of the docker image originally specified by the user (or the default), e.g. ubuntu:latest 🍒
• imageName (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the image, including path and excluding the tag, e.g. ubuntu 🌸
• tag (str, /usr/share/doc/python3/html/library/stdtypes.html#str) — the tag used at that docker image's registry, e.g. latest 💜

Raises ApplianceImageNotFound if no match is found 😭

Returns Returns True if a match is found 🦄

Return type bool (/usr/share/doc/python3/html/library/functions.html#bool)

---

📝 toil.logProcessContext(config)

this one logs the process context for a given config 🧾

Parameters

• config (common.Config, #toil.common.Config) ⚙️

Return type None

---

[1] Created with sphinx-autoapi https://github.com/readthedocs/sphinx-autoapi 🛠️

• Index
• Search Page

---

✨ AUTHOR ✨

UCSC Computational Genomics Lab 🎓

---

💜 COPYRIGHT 💜

2015 – 2026 UCSC Computational Genomics Lab 🌈

---

Bestie, that's the whole page 🫶 Toil 9.1.2, dropped Jan 08, 2026, and we're officially done. Stay iconic 🦋💅✨