DOLLY(1)

📦 version 0.64.2, in opensuse-leap 16.0

🌸 DOLLY(1) — clone disks / partitions / data, bestie 🌸

A program to clone disks / partitions / data. Take same amount of time to copy data to one node or to X nodes.

✨ SYNOPSIS ✨

okay so the shapes are literally:

dolly [-f config]

dolly [-I infile] [-O outfile|-] [-H hostlist]

dolly -v -r 40 -S SERVERIP -H IPNODE1,IPNODE2,IPNODE3 -I /dev/vdd -O
/dev/vdd

🎀 DESCRIPTION 🎀

Dolly is used to clone the installation of one machine to (possibly many) other machines. It can distribute image-files (even gnu-zipped), partitions or whole hard disk drives to other partitions or whole hard disk drives. As it forms a “virtual TCP ring” to distribute data, it works best with fast switched networks.

And lowkey: as dolly clones whole partitions block-wise.

💅 OPTIONS 💅

🌟 if you’re not using a config file 🌟

okay so if you’re not using a configuration file, these three command-line options must be set:

  • -I FILE — FILE is used as input file.
  • -O FILE|- — FILE will be used as output file; if - is used as FILE, the output will be printed to stdout.
  • -H HOSTLIST — a comma separated hostlist, where the first host of the list is used as firstclient and the last host as lastclient, like in the configuration file.

💖 the rest of the options 💖

  • -h — prints a short help and exits.
  • -V — prints the version number as well as the date of that version and exits.
  • -v — switches to verbose mode in which dolly prints out a little bit more information. This option is recommended if you want to know what’s going on during cloning and it might be helpful during debugging.
  • -s — this option specifies the server machine and should only be used on the master. Dolly will warn you if the config file specifies another master than the machine on which this option is set. This option must be specified before the -f option!
  • -S — same as -s, but dolly will not warn you if the server’s hostname and the name specified in the config file do not match.
  • -R — resolve the hostnames to ipv4 addresses.
  • -6 — resolve the hostnames to ipv6 addresses.
  • -q — usually dolly will print a warning when the select() system call is interrupted by a signal. This option suppresses these warnings.
  • -c — with this option it is possible to specify the uncompressed size of a compressed file. It’s only needed for performance statistics at the end of a cloning process and not important if you are not interested in the statistics.
  • -d — connect to systemd socket on clients nodes to start the dolly service (port 9996). This option only available on command line.
Warning !! By default if you use the dolly.socket, the dolly.service start as root user, which means that you can delete all your nodes data easily while pushing into the ring data in the wrong place. There is NO AUTH process, which means that all nodes with a dolly socket open and listed as a target from the server will get the data.
  • -t — when in dummy mode, this option allows to specify how long the testrun should approximately take. Since the dummy mode is mostly used for benchmarking purposes and single runs might result in different speeds (especially with many machines and bad switches or with small TCP segment sizes), it’s more convenient to specify the run-lenght in seconds, as the benchmark-time becomes more predictable.
  • -f — this option is used to select the config file for this cloning process. This option makes only sense on the master machine and the configuration file must exist on the master.
  • -o — this option specifies the logfile. Dolly will write some statistical information into the logfile. It is mostly used when benchmarking switches. The format of the lines in the logfile is as follows:
Trans.  data Segsize Clients Time Dataflow Agg.  dataflow [MB] [Byte] [#] [s] [MB/s] [MB/s]
  • -a — sometimes it might be useful if Dolly would terminate instead of waiting indefinitely in case something goes wrong. This option lets you specify this timeout. If dolly could not transfer any data after seconds, then it will simply print an error message and terminate. This feature might be especially useful for scripted and automatic installations (such as “CloneSys”), where you don’t want to have dolly-processes hang around if a machine hangs.
  • -n — do not sync() before exit. Thus, dolly will exit sooner, but data may not make it to disk if power fails soon after dolly exits.
  • -u — specify the size of buffers for TCP sockets. Should be a multiple of 4K.
  • -b — option to specify the TRANSFER_BLOCK_SIZE. Should be a multiple of the size of buffers for TCP sockets.
  • -r — Retry to connect to mode times

🎀 CONFIGURATION FILE 🎀

okay so bestie, you've got two ways to feed Dolly its cloning deets: either you use the commandline options (-i, -o and -H), or you hand it a configuration file. Pick your fighter 💅 The config file format is strict — but like, easy strict, not scary strict. It has these entries, and heads up: the order of the entries is fix (yes, fixed, don't come for me). The text after Syntax: explains the syntax of the entry, and the lines following EG: are example lines 🌸

  1. infile / compressed infile — the file/partition you want to clone, preceded by the keywords infile or compressed infile in case of a compressed image. This file or partition only needs to be available on the master. Dolly will warn you if you try to use a compressed infile that does not end with .gz. The compressed keyword is genuinely important — it's how the master informs the clients that they have to use gunzip before writing a file. The optional keyword split after the filename instructs Dolly to read all files with the given name and an appended number, separated by an underscore. 🦋

Syntax: [compressed] infile [split]

EG: infile /dev/sda10 — will just send the partition /dev/sda10 to all clients.

EG: compressed infile /images/cloneimages/sda10_WinNTRes.gz — will send the given file compressed to all the clients, instructing them to uncompress the image before writing it.

EG: infile /images/cloneimages/sda split — will send all files of the form /images/cloneimages/sda_ in order to the clients.

EG: compressed infile /images/cloneimages/sda.gz split — will send all files of the form /images/cloneimages/sda.gz_ in order to the clients, instructing them to decompress the incoming stream before writing it.

  1. outfile — the file or partition you want to write (usually it's a partition, but bestie you can also write to a file), after the keyword outfile. This file only needs to be available on the clients. The optional keyword compressed instructs the server to compress the data before sending it, so the client will store the data compressed. The optional keyword split after the filename, followed by a number and a multiplier, instructs the client to write the data in junks of no more than the given size. This is useful if the file system on your client does not allow files greater than a certain size. The files will be stored with the given namen and an appended _. 👑

Syntax: [compressed] outfile [split (k|M|G|T)]

EG: outfile /dev/sda10 — will store the incoming data stream to the partition sda10.

EG: compressed outfile /images/cloneimages/sda10_SuSE81.gz — will store the compressed data stream in the given file.

EG: compressed outfile /images/cloneimages/sda_all.gz split 2G — will store the incoming compressed data stream in the directory /images/cloneimages/ in files sda_all.gz_0, sda_all.gz_1 and so on.

  • dummy — instead of the first two entries (infile and outfile) you can also use the single line dummy [], where the blank is the number of Megabytes to transfer in dummy mode. If it's set to 0, then the clients will just terminate. This is useful when benchmarking with different options, so the clients can run all the time ✨ To finally terminate them on all clients, just set dummy to 0.
NOTE: it is probably better to use the newer -t switch on the server to specify the number of seconds the benchmarks should run. In that case you can leave the blank empty.

Syntax: dummy []

EG: dummy 128

  • segsize — mostly used to benchmark switches 🔥 It specifies the maximal size of TCP segments during the network transfer. Usually you don't need to specify this option at all.

Syntax: segsize

EG: segsize 128

  • add — with this optional keyword it's possible to add more interfaces to use. The network traffic is then evenly distributed across the interfaces. This option is useful if you have, for example, two fast ethernet interfaces in your machines: one for administrative purposes and one for your main application on the cluster. It's not so useful if you have multiple interfaces with different bandwidths — in that case just use the fastest available. You have to specify the number of additional interfaces and the suffixes of thouse interfaces. For example, in a cluster where the machines are named slave0..slave15 on their default interfaces and all the machines have a second interface named slave0-fast..slave15-fast, you should use the line specified below (EG) 💜

Syntax: add <num>:<suffix>{:<suffix>}

EG: add 1:-fast

  • fanout — mostly used during performance tests of different network topologies. You barely need it in practice. Fanout specifies the number of outlinks from the server and the following machines (except the leafes). A fanout of 1 is a linear list (the default behaviour of Dolly and usually the fastest), 2 is a binary tree, 3 is a ternary tree, etc. Dolly automatically connects all the specified clients with the desired topology. 🌈

Syntax: fanout

EG: fannout 1

  • hypennormal — this optional keyword instructs Dolly to treat the - character in hostnames as any other character. By default the hyphen is used to separate the base hostnames from the names of the different interface (e.g. node12-giga). You might use this paramater if your hostnames include a hypen (like e.g. node-12). 🫶

Syntax: hyphennormal

EG: hyphennormal

  1. server — after this keyword follows the hostname of the server (or master). This is required for the last machine in the ring to be able to send the end-acknowledge back to the server. ⭐

Syntax: server

EG: server cluster-master

  1. firstclient — this entry has the keyword firstclient followed by the hostname of the first client in the ring. You should use the hostname of the machine here, not the name of the interface where you want to connect. 💖

Syntax: firstclient

EG: firstclient cluster-1

  1. lastclient — this entry has the keyword lastclient followed by the hostname of the last client in the ring. Again: use the hostname of the machine here, not the name of the interface where you want to connect. 💖

Syntax: lastclient

EG: lastclient cluster-9

  1. clients — this entry specifies how many clients are in the ring. The keyword is clients followed by the actual number of clients. This number does not include the master. 🧚

Syntax: clients

EG: clients 9

  1. The interface names — the following lines contain the interface-names of the client machines. The number of machines must match the above number of clients (see 6.). You should use the name of the interface on which the machines will receive the data. 🎀

Syntax: <name of client 1> <name of client 2> [...]

EG: cluster-1-giga cluster-2-giga [...] cluster-9-giga

  1. endconfig — the last entry in the config file consists of the keyword endconfig and marks the end of the configuration file. Bye bestie 👋

Syntax: endconfig

EG: endconfig

🦋 NOTE on NODES HOSTNAMES 🦋

So on some machines (e.g. with very small maintenance installations), gethostbyname() does not return the hostname (and like, honestly? I don't know why either 😭). If you have that problem, you should make sure that the environment variables MYNODENAME or HOST are set accordingly. Dolly first tries to get the environment variable MYNODENAME, then HOST, then it tries gethostbyname(). This feature was introduced in dolly version 0.58 ✨

👑 HOW IT WORKS 👑

Setting up or upgrading a cluster of PCs typically leads to the problem that many machines need the exact same files. There are different approaches to distribute the setup of one "master" machine to all the other machines in the cluster. Our approach is not sophisticated, but simple and fast (at least for fast switched networks). We send the data around in a "virtual TCP ring" from the server to all the clients which store tha received data on their local disks 💅

One machine is the master and distributes the data to the others. The master can be a machine of the cluster or some other machine (in the current version of dolly it should be the same architecture though). It stores the image of the partition or disk to be cloned or has the partition on a local disk. The server should be on a fast switched network (as all the other machines too) for fast cloning.

All other machines are clients. They receive the data from the ring, store it to the local disk and send it to the next machine in the ring. It is important to note that all of this happens at the same time ⚡

The cloning process is depicted in the following two figures. Usually there are more than two clients, but you get the idea:

                +========+  +==========+ +==========+
                | Master |  | Client 1 | | Client 2 |
                +====+===+  +===|======+ +====+=====+
                      \         |            /
                       \    +===+====+      /
                        +===+ Switch |=====+
                            +========+

Cloning process, physical network

               +========+  Data   +==========+  Data  +==========+
               | Master |========>| Client 1 |=======>| Client 2 |
               +========+         +==========+        +==========+
                   ^                   |                   |
                   | Data              | Data              | Data
                   |                   V                   V
                +======+            +======+            +======+
                | Disk |            | Disk |            | Disk |
                +======+            +======+            +======+

Cloning process, virtual network with TCP connections

We choose this method instead of a multicast scheme because it is simple to implement, doesn't require the need to write a reliable multicast protocol and works quite well with existing technologies. One could also use the master as an NFS server and copy the data to each client, but this puts quite a high load on the server and makes it the bottleneck. Furthermore, it would not be possible to directly clone partitions from one machine to some others without any filesystem in the partition 🌸

💖 DIFFERENT CLONING POSSIBILITIES

okay so there are a few different ways to clone your master machine, and they’re all valid depending on what setup you’ve got:

  • You already have an image of the partition you wanna clone on your master — raw or compressed, either one. In this case you need Linux on your master (some other UNIX might also work, but they haven’t tested that yet) and a Linux on each client.
  • You wanna clone a partition that’s on a local disk of your master. In this case you need Linux (or probably another UNIX, they haven’t tried that) on your master and on all the clients. You can use any Linux installation as long as it’s not the one you want to clone — i.e. you can not clone the Linux you’re currently running in. See the warning below, bestie.
  • You wanna clone a whole disk including all the partitions. In this case you either need a second disk on all machines where your Linux used for the cloning process runs on (not the one you want to clone), or you need a small one-floppy-disk-Linux which you boot on all machines. In the later case you also need dolly on all machines (copy it to your floppy disk or mount it with NFS) and the config-file on the master.
WARNING: You can NOT clone an OS which is currently in use. That is why they have a small second Linux installation on all of their machines (or a small system that can be booted over the network by PXE), which they can boot to clone their regular Linux partition. 💀

🔄 CHANGES

See the CHANGELOG file. that’s literally it, go peek 👀

🚧 TODO

Add an AUTH method to validate client nodes from server. that’s the whole todo list, bestie ✨

🌈 EXAMPLE

In this example we assume a cluster of 16 machines, named node0..node15. We want to clone the partition sda5 from node0 to all other nodes. The configuration file (let’s name it dollytab.cfg) should then look as follows:

infile /dev/sda5
outfile /dev/sda5
server node0
firstclient node1
lastclient node15
clients 15
node1
node2
node3
node4
node5
node6
node7
node8
node9
node10
node11
node12
node13
node14
node15
endconfig

Next, we start Dolly on all the clients. No options are required for the clients (but you might want to add the -v option for verbose progress reports). Finally, Dolly is started on the server as follows: dolly -v -s -f dollytab.cfg That’s all. 🎀

📚 Bibliography

Felix Rauch, Christian Kurmann, Thomas M. Stricker: Optimizing the distribution of large data sets in theory and practice. Concurrency and Computation: Practice and Experience, volume 14, issue 3, pages 165-181, april 2002. (c) John Wiley & Sons, Ltd.

Maintained by Felix Rauch. http://www.cs.inf.ethz.ch/~rauch/ Felix Rauch rauch@inf.ethz.ch

👑 AUTHORS / CONTRIBUTORS

  • Felix Rauch rauch@inf.ethz.ch
  • Antoine Ginies aginies@suse.com
  • Christian Goll cgoll@suse.com

and that’s the whole page, bestie — the footer at the bottom is just vibing with the source/version and date, so go peek at those 💅✨🫶

📦 where this page comes from