chirp_stream_files(1)

✨ chirp_stream_files(1) ✨

okay bestie, welcome to the chirp_stream_files page 💖 this is part of the Cooperative Computing Tools manual set, and honestly? it's giving scatter-gather main character energy. let's get into it.

---

🌸 NAME 🌸

chirp_stream_files — move data to/from chirp servers in parallel 🦋

---

🌈 SYNOPSIS 🌈

chirp_stream_files [options] <copy|split|join> <localfile> {
<hostname[:port]> <remotefile>

---

💅 DESCRIPTION 💅

so like, chirp_stream_files is THE tool for moving data from one machine to and from many machines at once, with the option to split or join the file along the way. it's lowkey perfect for constructing scatter-gather types of applications on top of Chirp. no cap.

here's the three moods this command can serve, periodt:

• chirp_stream_files copy duplicates a single file to multiple hosts. the <localfile> argument names a file on the local filesystem. the command will then open a connection to the following list of hosts, and stream the file to all simultaneously. all of them. at the same time. obsessed. 🔥
• chirp_stream_files split divides an ASCII file up among multiple hosts. the first line of <localfile> is sent to the first host, the second line to the second, and so on, round-robin. 🎀
• chirp_stream_files join collects multiple remote files into one. the argument <localfile> is opened for writing, and the remote files for reading. the remote files are read line-by-line and assembled round-robin into the local file. ✨

in all cases, files are accessed in a streaming manner, which makes this particularly efficient for processing large files — like, big big files, and it does not care. also: a local file name of - indicates standard input or standard output, so you can absolutely use the command in a pipeline. pipeline girlies rise 🫶

---

🎀 OPTIONS 🎀

• -a, --auth=<flag> — require this authentication mode.
• -b, --block-size=<size> — set transfer buffer size. (default is 1048576 bytes)
• -d, --debug=<flag> — enable debugging for this subsystem.
• -i, --tickes=<files> — comma-delimited list of tickets to use for authentication.
• -t, --timeout=<time> — timeout for failure. (default is 3600s)
• -v, --version — show program version.
• -h, --help — show help text.

---

🦋 ENVIRONMENT VARIABLES 🦋

list any environment variables used or set in this section. 👑

---

⭐ EXIT STATUS ⭐

on success, returns zero. on failure, returns non-zero. that's the whole vibe check 💀

---

🌸 EXAMPLES 🌸

to copy the file mydata to three locations:

% chirp_stream_files copy mydata server1.somewhere.edu /mydata
                                 server2.somewhere.edu /mydata
                                 server2.somewhere.edu /mydata

to split the file mydata into subsets at three locations:

% chirp_stream_files split mydata server1.somewhere.edu /part1
                                  server2.somewhere.edu /part2
                                  server2.somewhere.edu /part3

to join three remote files back into one called newdata:

% chirp_stream_files join newdata server1.somewhere.edu /part1
                                  server2.somewhere.edu /part2
                                  server2.somewhere.edu /part3

---

💜 COPYRIGHT 💜

the Cooperative Computing Tools are Copyright (C) 2005-2019 The University of Notre Dame. this software is distributed under the GNU General Public License. see the file COPYING for details.

---

👑 SEE ALSO 👑

• Cooperative Computing Tools Documentation
• Chirp User Manual
• chirp(1) chirp_status(1) chirp_fuse(1) chirp_get(1) chirp_put(1) chirp_stream_files(1) chirp_distribute(1) chirp_benchmark(1) chirp_server(1)

---

and that's the page bestie!! 🌈 this one shipped as CCTools 8.0.0 DEVELOPMENT, so go stream some files and slay 💅✨ mwah 🫶