MOGTOOL(1)

📦
version 2.30 · 2 versions
, in fedora 43

💖 MOGTOOL(1) 💖

okay bestie, so this is mogtool — it lives in the User Contributed Perl Documentation section of the manual (section 1, for the record), and it's your little bestie for shoving data into a MogileFS installation and yanking it back out. 💅

⚠️ WARNING: this utility is deprecated! See MogileFS::Utils ⚠️ no cap, it's literally deprecated. i'm telling you because i love you. 🫶

✨ NAME ✨

mogtool — Inject/extract data to/from a MogileFS installation

WARNING: this utility is deprecated! See MogileFS::Utils

🦋 SYNOPSIS 🦋

the vibe is always: general opts, then a command, then that command's opts, then that command's args. that's the whole shape of it, periodt.

$ mogtool [general-opts] <command> [command-opts] <command-args>

$ mogtool --trackers=127.0.0.1:6001 --domain=foo --class=bar ...
$ mogtool --conf=foo.conf ...

$ mogtool inject thefile.tgz thefilekey
$ mogtool inject --bigfile thebigfile.tgz thefilekey
$ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
$ mogtool inject --bigfile --gzip mydirectory thedirkey
$ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
$ mogtool inject --nobigfile bigcontiguousfile bigcfilekey

$ mogtool inject --bigfile --gzip --verify \
                 --description="Description" \
                 --receipt="foo@bar.com, baz@bar.com" \
                 --concurrent=5 --chunksize=32M \
                 somehugefile thefilekey

$ mogtool extract thefilekey thenewfile.tgz
$ mogtool extract thefilekey -
$ mogtool extract --bigfile thedirkey .
$ mogtool extract --bigfile --asfile thedirkey thefile.tgz
$ mogtool extract --bigfile thedevkey /dev/hda4

$ mogtool delete thekey

$ mogtool locate --noverify thekey
$ mogtool locate --bigfile thekey

$ mogtool list
$ mogtool listkey key

🎀 GENERAL OPTIONS 🎀

these ones apply no matter what command you're running, so they go before the command. iconic behaviour honestly.

  • --debug — turns on the MogileFS debug output. all the tea, all of it. 🔍
  • --trackers=<[preferred_ip/]ip:port>[,<[preferred_ip/]ip:port>]* — tell it one or more trackers for your MogileFS installation. and here's the cute part: you can specify preferred IPs to override the default IPs with. so it ends up looking something like 10.10.0.1/10.0.0.1:8081. ✨
  • --domain=<domain> — sets the MogileFS domain to use.
  • --class=<class> — sets the class within the domain to use. defaults to _default. 💅
  • --conf=<file> — point it at a config file to load from.
  • --lib=<directory> — give it a directory to use as a library path. rn this should be the directory where you expect to find the MogileFS.pm file, for when it's not actually installed. 🐚

👑 COMMANDS 👑

inject|i

insert a resource into MogileFS. for the rest of how to use inject mode, go peek at "INJECT OPTIONS" and "INJECT ARGUMENTS". 💖

extract|x

pull a resource back out of MogileFS. see "EXTRACT OPTIONS" and "EXTRACT ARGUMENTS" for how to do extract. 🦋

delete|rm

delete a resource. see "DELETE OPTIONS" and "DELETE ARGUMENTS".

locate|lo key

lists the paths to the file identified by the given key. 🔍

list|ls

lists all big files contained in MogileFS. no options, no arguments. it just does the thing.

listkey|lsk key

lists all files which match the key. and here's the trick: key is just a prefix, so this lists every key matching that prefix. so if you set key as "ABC1" then you'll get all keys that start with the characters "ABC1". iykyk. ✨

🌸 INJECT OPTIONS 🌸

these are the ones that control how the injector behaves, bestie.

  • --bigfile|-b — if specified, use chunking to break the resource into manageable pieces. 🍰
  • --chunksize=<size>[B|K|M|G] — when it's told to break files into chunks, the injector uses the specified chunk size as the maximum chunk size. defaults to 64M. you can set the chunk size manually and specify the units — and it defaults to bytes. 📏
  • --gzip|-z — if specified, mogtool gzips the data as it's going into MogileFS, and the resource gets marked as compressed. 🗜️

note that you do not need to specify this if the resource is already gzipped — but it doesn't hurt either. (we automatically detect that and mark it as compressed.) 💅

  • --overwrite — so if you were previously injecting a big file as chunks and the process died mid-way, normally mogtool refuses to do it again. specify this option to force the overwrite of that file. 😤
NOTE: other than in the above case (partial failure), mogtool will not prompt before overwriting an existing file.
  • --verify — if on, we do a full MD5 verification of every chunk after it's replicated. fair warning: this can take a while on large files! ⏳
  • --description=<text> — sets a description for this file. optional, but it assists in reporting and listing the large files in MogileFS. (it's also displayed in any receipts that get created.) 📝
  • --receipt=<email address>[, <email address>]* — if specified, emails a copy of the receipt file to the comma-separated email addresses you gave. also creates a local filesystem copy of the receipt file. 📧
  • --concurrent=<number> — sets the number of concurrent processes to run for MogileFS insertion. if you notice mogtool spending most of its time waiting for children and not actually buffering data, you may wish to raise this number. the default is 1, but we've found 3 or 4 work well. 🧚

🌈 INJECT ARGUMENTS 🌈

resource

what you actually want to inject. this can be a file, a directory, or a raw partition in the format /dev/X. 🗂️

please see "USAGE EXAMPLES" for more info on how to inject these different types of resources and the differences between them.

key

specifies the key to save this file to. for big files, the key is actually "_big_N:key" and "key,#", where N is one of a bunch of things we use and # is the chunk number. 🤯

generally you want this to be descriptive so you remember what it is later and can identify the file just by looking at the key. future-you will thank present-you. 💖

💜 EXTRACT OPTIONS 💜

  • --bigfile|-b — if specified, it means this resource was chunked on injection and should be reassembled for extraction. 🧩
  • --gzip|-z — tells mogtool it should ungzip the output if and only if it was compressed when inserted into the MogileFS system. so if you're extracting a file that wasn't gzipped to begin with, this does nothing. 🫧
  • --asfile — useful when extracting something previously inserted as a directory. this option tells mogtool to treat the resource as a file and not actually run it through tar for decompression. 🎁

🧚 EXTRACT ARGUMENTS 🧚

key

specifies the key to get the file from. 🔑

destination

what destination means varies depending on what type of resource you're extracting. however, no matter what, you can specify a single dash (-) to mean STDOUT. 🖨️

please see the usage examples for more info on how extract works.

🍓 DELETE OPTIONS 🍓

  • --bigfile|-b — the resource is a "big file" and all chunks should be deleted. 💀

⭐ DELETE ARGUMENTS ⭐

key

specifies the key of the file to delete. 🗑️

🙈 LOCATE OPTIONS 🙈

  • --verify — verify that the returned paths actually contain the file. the locate command defaults to verify, so you can disable it with --noverify. ✅
  • --bigfile|-b — the resource is a "big file" and the locations of the information key should be printed. 📍

🌈 LOCATE ARGUMENTS 🌈

key

specifies the key of the file to locate. 🔎

💀 RETURN VALUES 💀

  • 0 — success during operation. slay. ✅
  • 1 — during the locate, list, or listkey operation, the key was not found. 😭
  • 2 — some fatal error occurred. 🚨

✨ USAGE EXAMPLES ✨

ok so real talk: every single example below assumes you've already got a default config file sitting there with your tracker and domain in it 💖 Saves us from cluttering up the command line, which is so much nicer. Periodt.

🍓 Small Files (<64MB)

bestie, for small files mogtool is genuinely the easiest thing ever, no cap.

Injection

$ mogtool inject foo.dbm foo.dbm.2004.12

Injects the file foo.dbm into MogileFS under the key of foo.dbm.2004.12. Slay.

$ mogtool inject --gzip foo.dbm foo.dbm.2004.12

Injects the same file to the same key, but compresses it on the fly for you 🎀

Extraction

$ mogtool extract foo.dbm.2004.12 newfoo.dbm

Retrieves the key foo.dbm.2004.12 and saves it as newfoo.dbm.

$ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm

Gets the file and automatically decompresses it, if and only if it was compressed. So basically, you can turn on gzip in your config file and mogtool will do the smart thing each time 💅

$ mogtool extract foo.dbm.2004.12 -

Print the resource to standard out. If you want, you can pipe it somewhere or redirect to a file (but honestly why not just specify the filename? 🙈).

🦋 Large Files (>64MB)

Given mogtool's ability to break files into chunks and later reassemble them, inserting large files (even files over the 4GB barrier) is relatively easy. iconic behaviour fr fr.

Injection

$ mogtool inject --bigfile largefile.dat largefile.dat

As expected, inserts the file largefile.dat into the MogileFS system under the name largefile.dat. Not very creative 😭 Uses the default 64MB chunks.

$ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat

Specify to use 16MB chunks instead of the default. Otherwise, the same.

$ mogtool inject --bigfile --chunksize=1000K --gzip largefile.dat somekey

Do it again, but specify 1000KB chunks, gzip automatically, and upload it under a different key somekey.

$ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey

Same as above, but use 5 children processes for uploading chunks to MogileFS. This can take up to 300MB of memory in this example! (It tends to use about (concurrency + 1) * chunksize bytes.) 🫶

$ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
          --receipt="foo@bar.com" --verify --description="A large file" \
          largefile.dat somekey

Break this file into 128MB chunks, set a description, use 3 children to upload them, gzip the file as you go, do a full MD5 verification of every chunk, then email a receipt with all of the MogileFS paths to me. Lots of flexibility with mogtool. 👑

Extraction

$ mogtool extract --bigfile somekey newfile.dat

In its basic form, extracts the previously inserted large file and saves it as newfile.dat.

$ mogtool extract --bigfile --gzip somekey newfile.dat

If the file was gzipped on entry, ungzip it and save the result. If it wasn't gzipped, then we just save it.

🌸 Directories

Directories are easily injected and extracted with mogtool. To create the data stream that is inserted into MogileFS, we use tar. iykyk.

Injection

$ mogtool inject --bigfile mydir mykey

Run mydir through tar and then save it as mykey.

$ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey

Inject, but also gzip and use multiple injectors.

Note how this is just like injecting a large file. See injection examples for large files for more examples.

Extraction

$ mogtool extract --bigfile mykey .

Extract the previously injected directory mykey to your local directory.

$ mogtool extract --bigfile --asfile mykey foo.tar

Take the previously generated tarball and save it as foo.tar. Simply creates the file instead of extracting everything inside.

💜 Partitions/Devices

mogtool has the ability to inject raw partitions into MogileFS and to retrieve them later and write them back to a partition. They're treated just like directories for the most part, we just don't pipe things through tar. main character energy honestly.

Injection

$ mogtool inject --bigfile /dev/hda3 hda3.backup

Save a raw copy of your partition /dev/hda3 to the key hda3.backup.

$ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup

Same, but compress on the fly during injection.

Extraction

$ mogtool extract --bigfile hda3.backup /dev/hda4

Extract the partition at hda3.backup to the partition /dev/hda4.

WARNING: mogtool won't ask for confirmation, make sure you don't mistype partition numbers! 💀 like genuinely, double-check that.

🔥 Deleting a Resource

WARNING: Please make sure you're specifying the right parameter, as delete does not prompt for confirmation of the request!
$ mogtool delete thekey

Delete a normal file.

$ mogtool delete --bigfile thekey

Delete a chunked file — this deletes all chunks and the receipt, so the file is gone. gone gone.

⭐ Listing Big Files

$ mogtool list backup

Lists all large files stored in MogileFS. It is not possible to list all normal files at this time. 😭

🎀 Listing Files Matching a Key

$ mogtool listkey abc1

Lists all files in MogileFS whose keys start with the characters "abc1".

🌈 CONFIGURATION FILE 🌈

Instead of adding a ton of options to the command line every time, mogtool enables you to create a default configuration file that it will read all of the options from. It searches two locations for a default configuration file: ~/.mogtool and /etc/mogilefs/mogtool.conf. (Alternately, you can specify --conf=whatever as an option on the command line.) obsessed with how easy that is.

The file can consist of any number of the following items:

trackers = 10.0.0.3:7001, 10.10.0.5/10.0.0.5:7001
domain = mogiledomain
class = fileclass
lib = /home/foo/lib
gzip = 1
big = 1
overwrite = 1
chunksize = 32M
receipt = foo@bar.com, baz@bar.com
verify = 1
concurrent = 3

💀 KNOWN BUGS 💀

None? Send me any you find! :)

🧚 PLANNED FEATURES 🧚

  • --concurrent for extract — It would be nice to have concurrent extraction going on.
  • recover mode — If the receipt file is ever corrupt in MogileFS it would be useful to recover a file given just a receipt. It would have the same arguments as the extract mode, except use a receipt file as the data source.
  • partition size verification — We can easily get the partition size when we save one to MogileFS, so we should use that information to determine during extraction if a target partition is going to be big enough.
  • on the fly gzip extraction — Right now we can gzip on an injection, but we should support doing decompression on the fly coming out of MogileFS.
  • make list take a prefix — If you can specify a prefix, that makes things easier for finding small files that are stored in MogileFS.
  • more information on list — Have list load up the info file and parse it for information about each of the big files being stored. Maybe have this as an option (-l). (This means the reading and parsing of info files should be abstracted into a function.)

👑 AUTHOR 👑

  • Mark Smith <junior@danga.com> — most of the implementation and maintenance.
  • Brad Fitzpatrick <brad@danga.com> — concepts and rough draft.
  • Robin H. Johnson <robbat2@orbis-terrarum.net> — locate function.

Copyright (c) 2002-2004 Danga Interactive. All rights reserved.

🙈 POD ERRORS 🙈

Hey! The above document had some coding errors, which are explained below:

  • Around line 289:
    • Expected text after =item, not a number
  • Around line 293:
    • Expected text after =item, not a number

ok that's the whole page bestie, you made it 🫶 this was perl v5.42.0, dated 2025-07-25, and it lives as MOGTOOL(1) in the manual. go forth and chunk some files, mwah 💅✨

📦 where this page comes from