weaver(1)

💖 NAME 💖

weaver — a whole workflow engine for executing distributed workflows 🦋✨ (and yes bestie, this is the weaver page from the Cooperative Computing Tools manual, section 1, we're fancy like that 💅)

---

🎀 SYNOPSIS 🎀

weaver [options] <weaverfile>

---

🌸 DESCRIPTION 🌸

okay so. Weaver is a high level interface to makeflow, and it is literally the moment. A weaver input file is written in python 🐍, and in it you define functions to be applied on sets of files. weaver reads that input file and generates a workflow specification that can be executed by makeflow. and the way this lets you implement different workflow execution patterns in a straightforward way — like MapReduce and AllPairs — is giving main character energy, no cap 🔥

                                         /--------                        +-+ Python |
                                       | ---+----/
               +---------------------------------+ |     | Generate DAG
               |          Weaver          +-+     v
               +---------------------------------+   /-------|        Makeflow         +---+  DAG  |
               +--------+-----------+-----+------+   -------/
               | Condor | WorkQueue | SGE | Unix +-+     | Dispatch Jobs
               +--------+-----------+-----+------+ |     v
                                       | /-------                    +-+ Jobs  |
                                         -------/

---

✨ OPTIONS ✨

by default, running weaver on a <weaverfile> generates an input file for makeflow, <Makeflow>, plus a directory, <_Stash>, where all the intermediate files get stashed 🫶

💫 General options

• -h — give help information 🆘
• -W — stop on warnings. like, we're not ignoring the red flags bestie 💀
• -g — include debugging symbols in DAG 🐛
• -I — do not automatically import built-ins 🙈
• -N — do not normalize paths 🛤️
• -b <options> — set batch job options (cpu, memory, disk, batch, local, collect) ⚙️
• -d <subsystem> — enable debugging for <subsystem> 🔍
• -o <log_path> — set log path (default: stderr) 📝
• -O <directory> — set stash output directory (default <_Stash>) 📦

🧚 Optimization Options

• -a — automatically nest abstractions 🪆
• -t <group_size> — inline tasks based on <group_size> 📏

👑 Engine Options

• -x — execute DAG using workflow engine after compiling 🏃‍♀️
• -e <arguments> — set arguments to workflow engine when executing 🎯
• -wwrapper <> — set workflow engine wrapper 🎁

---

🚦 EXIT STATUS 🚦

• On success, returns zero ✅
• On failure, returns non-zero ❌

---

🌈 EXAMPLES 🌈

Weaver expresses common workflow patterns succinctly ✨ for example, with only the following three lines of code we can express a map pattern, in which we convert some images to the jpeg format:

convert = ParseFunction('convert {IN} {OUT}')
dataset = Glob('/usr/share/pixmaps/*.xpm')
jpgs    = Map(convert, dataset, '{basename_woext}.jpg')

please refer to cctools/doc/weaver_examples for further information 📚

The Cooperative Computing Tools are Copyright (C) 2005-2019 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details. 🫶

---

and that's the whole page bestie, ily, go slay some workflows 💅💖 this was the weaver(1) page from CCTools 8.0.0 DEVELOPMENT — mwah, see u next manpage 🎀✨