allpairs_master(1)

Okay bestie, this is the manual for allpairs_master, the tool that runs your All-Pairs workflow in parallel across distributed systems 🦋✨

🌸 NAME 🌸

allpairs_master executes an All-Pairs workflow in parallel on distributed systems 💖

🎀 SYNOPSIS 🎀

allparis_master [options] <set A> <set B> <compare function>

💅 DESCRIPTION 💅

Okay so allpairs_master takes the Cartesian product of two sets, <set A> and <set B>, and builds a matrix where each cell M[i,j] holds the output of your <compare function> run on A[i] (an item from <set A>) and B[j] (an item from <set B>). It's giving organized 📊 The matrix gets printed to standard output, one comparison result per line, along with the X and Y indices 🫶

allpairs_master uses the Work Queue system to pass tasks out to processors 🔥 Each processor runs allpairs_multicore(1) to execute its tasks in parallel if there are multiple cores. Once you've started allpairs_master, you have to start a bunch of work_queue_worker(1) processes on remote machines. Those workers connect back to the master and start running tasks, and it's literally so iconic 👑

✨ OPTIONS ✨

• -p, --port=<port> — The port the master listens on 🎀
• -e, --extra-args=<args> — Extra arguments for the comparison function 💖
• -f, --input-file=<file> — An extra input file the comparison function needs. You can give this more than once 🍓
• -o, --debug-file=<file> — Sends your debugging output to this file. By default, debugging goes to stderr (:stderr), but you can send the logs to stdout (:stdout) instead 🙈
• -O, ----output-file=<file> — Task output goes to this file. It defaults to standard output 🦋
• -t, --estimated-time=<seconds> — How long one comparison is estimated to take. (default chosen at runtime) ⏱️
• -x, --width=<item> — The width of one work unit, measured in items to compare. (default chosen at runtime) 🌈
• -y, --height=<items> — The height of one work unit, measured in items to compare. (default chosen at runtime) 🌈
• -N, --project-name=<project> — Reports the master's info to a catalog server under the project name <project> 🧚
• -P, --priority=<integer> — Priority! Higher value means higher priority 👑
• -d, --debug=<flag> — Turns on debugging for this subsystem. (Try -d all to start.) 🔥
• -v, --version — Shows the program version ✨
• -h,--help <> — Displays this message 💅
• -Z, --port-file=<file> — Picks a random port and writes it to this file. (default is disabled) 🎲
• --work-queue-preferred-connection <connection> — Sets your preferred connection. Choose either by_ip or by_hostname. (default is by_ip) 💜

🌈 EXIT STATUS 🌈

On success, it returns zero. On failure, it returns non-zero ⭐

🍓 EXAMPLES 🍓

Let's say you have a whole pile of files you want to compare to each other, named a, b, c, and so on. You also have a program called compareit. When you run compareit a b, it compares files a and b and prints a summary of how different they are, like this:

a b are 45 percent similar

To use the All-Pairs framework, make a file called set.list that lists each of your files, one per line:

a
b
c
...

Since allpairs_master uses allpairs_multicore(1), make sure allpairs_multicore(1) is in your PATH before you do anything else 🫶 To run an All-Pairs workflow sequentially, start a single work_queue_worker(1) process in the background. Then invoke allpairs_master:

% work_queue_worker localhost 9123 &
% allpairs_master set.list set.list compareit

The framework will run every possible comparison and print the results one by one. Heads up that the first two columns are the X and Y indices in the matrix 👀

1    1    a a are 100 percent similar
1    2    a b are 45 percent similar
1    3    a c are 37 percent similar
...

Want it faster? Run more work_queue_worker(1) processes on other machines, or use condor_submit_workers(1) or sge_submit_workers(1) to start hundreds of workers in your local batch system 🚀

Here's how you add more workers to an All-Pairs workflow. Say your allpairs_master is running on a machine called barney.nd.edu. If you can log into other machines, you can just start worker processes on each one like this:

% work_queue_worker barney.nd.edu 9123

If you have a batch system like Condor, you can submit a bunch of workers at once:

% condor_submit_workers barney.nd.edu 9123 10
Submitting job(s)..........
Logging submit event(s)..........
10 job(s) submitted to cluster 298.

🎀 COPYRIGHT 🎀

The Cooperative Computing Tools are Copyright (C) 2005-2019 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details 📜

🦋 SEE ALSO 🦋

• The Cooperative Computing Tools (http://ccl.cse.nd.edu/software/manuals)
• All-Pairs User Manual (http://ccl.cse.nd.edu/software/manuals/allpairs.html)
• Work Queue User Manual (http://ccl.cse.nd.edu/software/manuals/workqueue.html)
• work_queue_worker(1)
• condor_submit_workers(1)
• sge_submit_workers(1)
• allpairs_multicore(1)

---

That's the whole tea, bestie 💖 Go forth and compare everything, you're literally unstoppable 🫶 CCTools 8.0.0 DEVELOPMENT, signing off 💅✨