S3BACKER(1)

📦
version 2.1.3 · 3 versions
, in opensuse-leap 16.0

💅 S3BACKER(1) 💅

General Commands Manual but make it group-chat energy. okay so bestie, this is s3backer: the FUSE-based single file backing store via Amazon S3. No cap, it’s giving “one file, but make it a whole filesystem.” 💖

✨ NAME ✨

s3backer — FUSE-based single file backing store via Amazon S3.

🌸 SYNOPSIS 🌸

s3backer [options] bucket[/subdir] /mount/point

s3backer --nbd [options] bucket[/subdir] /dev/nbdX

s3backer --test [options] dir /mount/point

s3backer --erase [options] bucket[/subdir]

s3backer --reset-mounted-flag [options] bucket[/subdir]

🦋 DESCRIPTION 🦋

Okay so s3backer is a filesystem that contains a single file backed by the Amazon Simple Storage Service (Amazon S3). As a filesystem, it is very simple: it provides a single normal file having a fixed size. Underneath, the file is divided up into blocks, and the content of each block is stored in a unique Amazon S3 object. In other words, what s3backer provides is really more like an S3-backed virtual hard disk device, rather than a filesystem. periodt.

In typical usage, a normal filesystem is mounted on top of the file exported by the s3backer filesystem using a loopback mount (or disk image mount on Mac OS X).

This arrangement has several benefits compared to more complete S3 filesystem implementations:

  • By not attempting to implement a complete filesystem — which is a complex undertaking and difficult to get right — s3backer can stay very lightweight and simple. All of the experience and knowledge about how to properly implement filesystems that already exists can be reused. 💅
  • By utilizing existing filesystems, you get full UNIX filesystem semantics. Subtle bugs or missing functionality relating to hard links, extended attributes, POSIX locking, etc. are avoided. 🫶
  • The gap between normal filesystem semantics and Amazon S3 eventual consistency is more easily and simply solved when one can interpret S3 objects as simple device blocks rather than filesystem objects (see below).
  • When storing your data on Amazon S3 servers, which are not under your control, the ability to encrypt and authenticate data becomes a critical issue. s3backer supports secure encryption and authentication. Alternately, the encryption capability built into the Linux loopback device can be used. 🔥
  • Since S3 data is accessed over the network, local caching is also very important for performance reasons. Since s3backer presents the equivalent of a virtual hard disk to the kernel, most of the filesystem caching can be done where it should be: in the kernel, via the kernel's page cache. However s3backer also includes its own internal block cache for increased performance, using asynchronous worker threads to take advantage of the parallelism inherent in the network. ✨

💖 Consistency Guarantees 💖

Previously, Amazon S3 made relatively weak guarantees relating to the timing and consistency of reads vs. writes (collectively known as eventual consistency). s3backer includes logic and configuration parameters to work around these limitations, allowing the user to guarantee consistency to whatever level desired, up to and including 100% detection and avoidance of incorrect data. These are:

  1. s3backer enforces a minimum delay between consecutive PUT or DELETE operations on the same block. This ensures that Amazon S3 doesn't apply these operations out of order.
  2. s3backer maintains an internal block MD5 checksum cache, which enables automatic detection and rejection of stale blocks returned by GET operations.

This logic is configured by the following command line options: --md5CacheSize, --md5CacheTime, and --minWriteDelay.

In December of 2020, Amazon announced that S3 is now fully consistent. Therefore, the MD5 cache is disabled by default in s3backer versions after 1.6.3. Other S3-compatible services may still require it however.

To guarantee consistency, --md5CacheTime and --minWriteDelay should be greater than the maximum S3 convergence time.

🧚 Zeroed Block Optimization 🧚

As a simple optimization, s3backer does not store blocks containing all zeroes; instead, they are simply deleted. Conversely, reads of non-existent blocks will contain all zeroes. In other words, the backed file is always maximally sparse. 🦋

As a result, blocks do not need to be created before being used and no special initialization is necessary when creating a new filesystem.

s3backer also uses a simple bitmap to track blocks that are known to be zero on an ongoing basis. When the --listBlocks flag is given (recommended), s3backer will list all existing blocks at startup to initialize this bitmap. This especially affects performance with operations that read or write a large number of zero blocks, such as filesystem initialization, fstrim(8), bulk copies, etc.

🎀 File and Block Size Auto-Detection 🎀

As a convenience, whenever the first block of the backed file is written, s3backer includes as meta-data (in the x-amz-meta-s3backer-filesize header) the total size of the file. Along with the size of the block itself, this value can be checked and/or auto-detected later when the filesystem is remounted, eliminating the need for the --blockSize or --size flags to be explicitly provided and avoiding accidental mis-interpretation of an existing filesystem. 💖

🌈 Block Cache 🌈

s3backer includes support for an internal block cache to increase performance. The block cache is completely separate from the MD5 cache, which only stores MD5 checksums transiently and whose sole purpose is to mitigate eventual consistency. The block cache is a traditional cache containing cached data blocks. When full, clean blocks are evicted as necessary in LRU order.

Reads of cached blocks will return immediately with no network traffic. Writes to the cache also return immediately and trigger an asynchronous write operation to the network via a separate worker thread. Because the kernel typically writes blocks through FUSE filesystems one at a time, performing writes asynchronously allows s3backer to take advantage of the parallelism inherent in the network, vastly improving write performance.

When the cache is full, least recently accessed blocks are evicted first (but see also the --blockCacheNumProtected flag).

The block cache can be configured to store the cached data in a local file instead of in memory. This permits larger cache sizes and allows s3backer to reload cached data after a restart. Reloaded data is verified via MD5 checksum with Amazon S3 before reuse.

Having said all that, Linux users may want to consider instead using the kernel bcache mechanism for local caching of blocks.

The block cache is configured by the following command line options: --blockCacheFile, --blockCacheMaxDirty, --blockCacheNoVerify, --blockCacheNumProtected, --blockCacheSize, --blockCacheSync, --blockCacheThreads, --blockCacheTimeout, --blockCacheWriteDelay, and --blockCacheRecoverDirtyBlocks.

⭐ Read Ahead ⭐

s3backer implements a simple read-ahead algorithm in the block cache. When a configurable number of blocks are read in order, block cache worker threads are awoken to begin reading subsequent blocks into the block cache. Read ahead continues as long as the kernel continues reading blocks sequentially. The kernel typically requests blocks one at a time, so having multiple worker threads already reading the next few blocks improves read performance by taking advantage of the parallelism inherent in the network.

Note that the kernel implements a read ahead algorithm as well; its behavior should be taken into consideration. By default, s3backer passes the -o max_readahead=0 option to FUSE.

Read ahead is configured by the --readAhead and --readAheadTrigger command line options.

🔥 Encryption and Authentication 🔥

s3backer supports encryption via the --encrypt, --password, and --passwordFile flags. When encryption is enabled, SHA1 HMAC authentication is also automatically enabled, and s3backer rejects any blocks that are not properly encrypted and signed.

Encrypting at the s3backer layer is preferable to encrypting at an upper layer (e.g., at the loopback device layer), because if the data s3backer sees is already encrypted it can't optimize away zeroed blocks or do meaningful compression.

🍓 Compression 🍓

s3backer supports block-level compression, which minimizes transfer time and storage costs.

Compression is configured via the --compress and --compress-level flags. Compression is automatically enabled when encryption is enabled.

👑 Server Side Encryption 👑

s3backer supports server side encryption via the --sse flag.

💜 Read-Only Access 💜

An Amazon S3 account is not required in order to use s3backer. The filesystem must already exist and have S3 objects with ACL's configured for public read access (see --accessType below); users should perform the looback mount with the read-only flag (see mount(8)) and provide the --readOnly flag to s3backer. This mode of operation facilitates the creation of public, read-only filesystems.

🙈 Simultaneous Mounts 🙈

Although it functions over the network, the s3backer filesystem is not a distributed filesystem and does not support simultaneous read/write mounts. (This is not something you would normally do with a hard-disk partition either.) As a safety measure, s3backer attempts to detect this situation using an 'already mounted' flag in the data store, and will fail to start if it does.

This detection may produce a false positive if a former s3backer process was not shutdown cleanly; if so, the --reset-mounted-flag flag can be used to reset the 'already mounted' flag. But see also BUGS below.

💖 Statistics File 💖

s3backer populates the filesystem with a human-readable statistics file. Use --statsFilename to change the name of this file (default stats). The statistics can be reset to zero by attempting to remove the file.

🌸 NBD Plugin 🌸

On platforms with ndbkit(1), s3backer is also installed as an nbdkit(1) plugin.

When using the NBD plugin, command line flags like --blockSize=SIZE should instead be specified as plugin parameters like blockSize=SIZE; for boolean flags, specify a value of true. The bucket name (with optional subdirectory) must be specified with the bucket parameter.

To avoid conflicts with other nbdkit(1) flags, any s3backer flag can be specified with a s3b_ prefix, for example s3b_force=true.

As an alternative to launching nbdkit(1) directly, the --nbd flag causes s3backer to function as a wrapper process for managing an NBD session; see below.

💅 Logging 💅

In normal operation s3backer will log via syslog(3). When run with the -d or -f flags, s3backer will log to standard error.

✨ OPTIONS ✨

Okay so bestie, here's the tea on how flags work 🫖 every single command line flag has two forms, like --accessFile=FILE and -o accessFile=FILE. Only the first form is shown below, but honestly either one is valid — they're literally equivalent, no cap. The second form exists so you can drop mount options straight into /etc/fstab and have them passed seamlessly to s3backer by FUSE. Slay.

  • --accessFile=FILE — points to a file holding accessID:accessKey pairs, one per-line. Blank lines and lines beginning with a # get ignored (yes, comments, we love to see it 💅). If no --accessKey or --accessKeyEnv is specified, this file gets searched for the entry matching the access ID you gave via --accessId; and if you didn't specify an --accessId at all, the first entry in this file is the one that gets used. Default value is $HOME/.s3backer_passwd.
  • --accessId=ID — your Amazon S3 access ID goes here. Pass an empty string to force no access ID. If no access ID is specified (and none is found in the access file), s3backer will still function, but lowkey only reads of publicly available filesystems will work. 🙈
  • --accessKey=KEY — your Amazon S3 access key goes here.
Bestie, real talk: to avoid publicizing this secret via the command line, use --accessKeyEnv or --accessFile instead of this flag. Please. For me. 💀
  • --accessKeyEnv=VARNAME — name an environment variable that contains the Amazon S3 access key. Much safer vibe ✨
  • --accessType=TYPE — sets the Amazon S3 access privilege ACL type for newly written blocks. The value must be one of private, public-read, public-read-write, or authenticated-read. Default is private.
  • --accessEC2IAM=ROLE — downloads access credentials and security token in JSON document form from http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE every five minutes.

This option lets S3 credentials be provided automatically via the specified IAM role to s3backer when it's running on an Amazon EC2 instance. Set it and forget it, iconic behavior 👑

  • --authVersion=TYPE — specifies how to authenticate requests. There are two supported authentication methods: aws2 is the original AWS authentication scheme, and aws4 is the newer, recommended authentication scheme.

aws4 is the default setting starting in version 1.4, and is required for certain non-US regions, while aws2 may still be required by some non-Amazon S3 providers. iykyk.

Use this flag to specify FIPS and dual-stack endpoints.

Note: the region name is used in authentication, so if you include a region name you probably also need to specify it via --region. Don't forget or it's giving auth error 😭
  • --blockCacheFile=FILE — specifies a file in which to store cached data blocks. Without this flag, the block cache lives entirely in process memory and the cached data disappears when s3backer is stopped. The file will be created if it doesn't exist.

Cache files that have been created by previous invocations of s3backer are reusable as long as they were created with the same configured block size (if not, startup will fail). This is true even if s3backer was stopped abruptly, e.g., due to a system crash; however, this guarantee rests on the assumption that the filesystem containing the cache file will not reorder writes across calls to fsync(2).

If an existing cache is used but was created with a different size, s3backer will automatically expand or shrink the file at startup. When shrinking, blocks that don't fit in the new, smaller cache are discarded. This process also compacts the cache file to the extent possible.

By default, only clean cache blocks are recoverable after a restart. This means a system crash will cause dirty blocks in the cache to be lost (of course, that is the case with an in-memory cache as well). 💀

With the newer cache file format introduced in release 1.5.0, you can recover these dirty blocks by specifying the --blockCacheRecoverDirtyBlocks option. This will cause any dirty blocks in the cache file to be made writable again on startup. If your cache file was created with a prior release of s3backer or you do not specify this option, dirty blocks in the cache file are discarded on startup. The window of this data loss can be limited by --blockCacheWriteDelay.

By default, when having reloaded the cache from a cache file, s3backer will verify the MD5 checksum of each reloaded block with Amazon S3 before its first use. This verify operation does not require actually reading the block's data, and therefore is relatively quick. This guards against the cached data having unknowingly gotten out of sync since the cache file was last used — a situation that is otherwise impossible for s3backer to detect. So yeah, it's doing the most so you don't have to 🫶

  • --blockCacheMaxDirty=NUM — puts a limit on the number of dirty blocks in the block cache. When this limit is reached, subsequent write attempts will block until an existing dirty block is successfully written (and therefore becomes no longer dirty). This flag limits the amount of inconsistency there can be with respect to the underlying S3 data store.

The default value is zero, which means no limit. Live a little, bestie ✨

  • --blockCacheNoVerify — disables the MD5 verification of blocks loaded from a cache file specified via --blockCacheFile.
Using this flag is dangerous; use only when you are sure the cached file is uncorrupted and the data it contains is up to date. I'm not gonna stop you but I am gonna look at you like this 🙈
  • --blockCacheSize=SIZE — the block cache size, in number of blocks. Each entry in the cache will consume approximately block size plus 20 bytes. A value of zero disables the block cache. Default value is 1000.
  • --blockCacheThreads=NUM — sets the size of the thread pool associated with the block cache (if enabled). This bounds the number of simultaneous writes that can occur to the network. Default value is 20. Parallelism, we stan 🌈
  • --blockCacheTimeout=MILLIS — the maximum time a clean entry can remain in the block cache before it will be forcibly evicted and its associated memory freed. A value of zero means there is no timeout; in this case, the number of entries in the block cache will never decrease, eventually reaching the maximum size configured by --blockCacheSize and staying there. Configure a non-zero value if the memory usage of the block cache is a concern. Default value is zero (no timeout). So default is "let it balloon", essentially 😭
  • --blockCacheWriteDelay=MILLIS — the maximum time a dirty block can remain in the block cache before it must be written out to the network. Blocks may be written sooner when there is cache pressure. A value of zero configures a "write-through" policy; greater values configure a "write-back" policy. Larger values increase performance when a small number of blocks are accessed repeatedly, at the cost of greater inconsistency with the underlying S3 data store. Default value is 250 milliseconds.
  • --blockCacheSync — forces synchronous writes in the block cache layer. Instead of returning immediately and scheduling the actual write to operation happen later, write requests will not return until the write has completed. This flag is a stricter requirement than --blockCacheWriteDelay=0, which merely causes the writes to be initiated as soon as possible (but still after the write request returns).

This flag requires --blockCacheWriteDelay to be zero.

Using this flag is likely to drastically reduce write performance. It's giving patience. It's giving waiting room. 🧘‍♀️
  • --blockCacheRecoverDirtyBlocks — an unclean dismount may leave dirty blocks (blocks written to the local cache file, but not yet flushed to S3) in the cache file.

If this option is set, s3backer will recover any such dirty blocks and eventually write them back to S3. If this option is not specified, all dirty data in the cache file are discarded on startup.

If the filesystem has been mounted since the cache file was last used, s3backer will refuse to mount. This is verified by checking a unique 32-bit mount token in the cache file against the 'already mounted' flag in the data store.

This flag requires --blockCacheFile to be set. No file, no recovery, periodt 💅

  • --blockCacheNumProtected=NUM — preferentially retain the first NUM blocks in the block cache.

Some upper filesystems store highly active data (e.g., write journal) at the beginning of the filesystem. This option can be used to improve performance by reducing network reads for these regions of the file. With this option enabled, blocks after the first NUM blocks will be evicted before any protected blocks. Protect the VIPs 👑

  • --blockCacheFileAdvise — immediately after being read or written by the kernel, data in the block cache file can end up being cached twice by the kernel: once in the cache for the block cache file, and again in the cache for the s3backer file.

When this flag is given, posix_fadvise(2) is used whenever data is read or written to/from the block cache file to tell the kernel not to cache that data on behalf of the block cache file. One cache is enough, bestie, we're not made of RAM 😭

This flag is ignored if --blockCacheFile is not specified.

  • --blockHashPrefix — prepend random prefixes (generated deterministically from the block number) to block object names. This spreads requests more evenly across the namespace, and prevents heavy access to a narrow range of blocks from all being directed to the same backend server. Load balancing, we love her ✨
As with --prefix or subdir, this flag must be used consistently once a disk image is established. Pick a lane and stay in it 🎀
  • --blockSize=SIZE — the block size. This must be a power of two and should be a multiple of the kernel's native page size. The size may have an optional suffix 'K' for kilobytes, 'M' for megabytes, etc. The total number of blocks must fit into 32 bits. Math is math, babe 🧮

...so like, real talk: s3backer supports partial block operations, but that means it has to read before every write 😭. Using the block cache and properly aligning the s3backer block size with whatever you're actually doing (e.g., the block size of the upper filesystem) helps keep those extra reads to a minimum. And ngl, even when filesystems are configured for big block sizes, the kernel will often still go ahead and write page-sized blocks. That's just how it is 💅

s3backer will try to auto-detect the block size by reading block number zero at startup ✨. If you don't pass this option, the auto-detected value wins. If you do pass it and it disagrees with the auto-detected value, s3backer exits with an error unless --force is also given 🫶. And if auto-detection fails because block number zero doesn't exist and you didn't specify this option, then the default value of 4K (4096) is used. Periodt.

  • --cacert=FILE — Points at the SSL certificate file used to verify the remote server's identity when you're operating over SSL connections 💖. Equivalent to the --cacert flag documented in curl(1).
  • --compress[=ALGORITHM] — Compresses blocks with the chosen ALGORITHM before yeeting them over the network 🦋. Net result: less network traffic (both directions) and lower storage costs. Slay.

The ALGORITHM part is optional; currently the default value is deflate. zstd is also supported 👑

This flag only turns on compression for newly written blocks — decompression is always enabled and gets applied whenever appropriate. So it's totally safe to flip this flag on or off between different invocations of s3backer on the same filesystem 🙌

This flag is implied when --encrypt is used.

Heads up: if you're using an encrypted upper layer filesystem, this flag adds literally no value, because the data won't be compressible 💀
  • --compress-level=LEVEL — Sets the compression level for the algorithm you picked, which you must specify via --compress.

How LEVEL gets interpreted depends on the algorithm; for deflate it has to be an integer between 1 (maximum speed) and 9 (maximum compression), inclusive.

If you omit this flag, the default compression level for the selected algorithm is used.

  • --configFile=FILE — Reads command line flags and arguments from the specified file and inserts them right in place of this flag 🎀.

Leading and trailing whitespace gets trimmed from each line, and blank lines and lines starting with # are ignored. Each line contains exactly one command line argument (including internal whitespace, if any).

Nested --configFile flags are allowed 👀

This option may also be specified as -F FILE.

  • --directIO — Disables kernel caching of the backed file. This forces the kernel to always pass reads and writes directly to s3backer. It may reduce performance, but it also eliminates one source of inconsistency 🧚
  • --debug — Turns on logging of debug messages, including logging HTTP error response payloads 🔥
Note that this flag is different from -d, which is a flag to FUSE — however, the -d FUSE flag implies this flag.
  • --debug-http — Configures libcurl(3) to print debugging information (e.g., HTTP request and response headers) to standard error 🌸

With this flag you'd normally also specify -f so s3backer runs in the foreground and standard error is actually visible 👑

  • --defaultContentEncoding=VALUE — Use this to work around S3 backends that fail to send back the Content-Encoding header that s3backer sent them. If a block read response contains no Content-Encoding header, this value gets substituted in 🩹

If you get errors complaining that the content was expected to be encrypted, try setting this to deflate,encrypt-AES-128-CBC.

  • --encrypt[=CIPHER] — Enables encryption and authentication of block data 🔐. Check your OpenSSL documentation for the list of supported ciphers; the default if no cipher is specified is AES-128 CBC.

Currently, only block ciphers are supported.

The encryption password may be supplied via one of --password or --passwordFile. If neither flag is given, s3backer will ask for the password at startup 🙈

Note: the actual key used is derived by hashing the password, the bucket name, the prefix name (if any), and the block number. So encrypted data cannot be ported to different buckets or prefixes. No cap.

This flag implies --compress.

  • --erase — Completely erases the file system by deleting all non-zero blocks, clears the 'already mounted' flag, and then exits 💥. User confirmation is required unless the --force flag is also given. Note that no simultaneous mount detection is performed in this case.
This operation bypasses the caching layers, so any leftover cache file must be manually deleted. Don't forget, bestie 😭
  • --filename=NAME — Sets the name of the backed file that appears in the s3backer filesystem. Default is file 🎀
  • --fileMode=MODE — Sets the UNIX permission bits for the backed file that appears in the s3backer filesystem. Default is 0600, unless --readOnly is specified, in which case the default is 0400 🔒
  • --force — Proceeds even if the value specified by --blockSize or --size disagrees with the auto-detected value, or if s3backer detects that another s3backer instance is still mounted on top of the same S3 bucket (and prefix). In any of these cases, proceeding will lead to corrupted data, so the --force flag should be avoided for normal use 💀
The simultaneous mount detection can produce a false positive when a previous s3backer instance was not shut down cleanly. In this case, don't use --force — instead run s3backer once with the --reset-mounted-flag flag. Trust.

If --erase is given, --force causes s3backer to proceed without user confirmation.

  • -h, --help — Prints a help message and exits 🫶
  • --http11 — Restricts to HTTP version 1.1. There have been reports of errors and/or reduced performance with some S3-compatible backends when HTTP/2 is used, so this flag can be used to prevent HTTP/2 negotiation 👑
  • --initialRetryPause=MILLIS — Sets the initial pause time in milliseconds before the first retry attempt after failed HTTP operations ⏱️. Failures include network failures and timeouts, HTTP errors, and reads of stale data (i.e., MD5 mismatch); s3backer will make multiple retry attempts using an exponential backoff algorithm, starting with this initial retry pause time. Default value is 200ms. See also --maxRetryPause.
  • --insecure — Does not verify the remote server's identity when operating over SSL connections 🙈. Equivalent to the --insecure flag documented in curl(1).
  • --keyLength — Overrides the length of the generated block encryption key 🔑

Versions of s3backer prior to 1.3.6 contained a bug where the length of the generated encryption key was fixed but system-dependent, causing it to be possibly incompatible on different systems for some ciphers. In version 1.3.6 this bug was corrected; however, in some cases this changed the generated key length, making the encryption no longer compatible with previously written data 😭. This flag can be used to force the older, fixed key length. The value you want to use is whatever is defined for EVP_MAX_KEY_LENGTH on your system, typically 64.

It is an error to specify a value smaller than the cipher's natural key length; however, a value of zero is allowed and is equivalent to not specifying anything ✨

  • --listBlocks — After starting, kicks off a background query to figure out which blocks already exist 🔍. This enables optimizations whereby, for each block that doesn't exist yet, reads return zeroes and zeroed writes are omitted — thereby eliminating any network access 💅

This flag is useful when creating a new backed file, or any time you expect a large number of zeroed blocks to be read or written, like when initializing a new filesystem.

In general, use of this flag is recommended — but it does create additional network traffic during startup in proportion to the number of blocks that already exist. Tradeoffs, bestie 🌈

  • --listBlocksThreads=NUM — To minimize startup delay, the initial block enumeration of --listBlocks is performed in parallel using multiple threads. This flag configures the number of threads used 🧵

Default value is 16.

  • --maxUploadSpeed=BITSPERSEC
  • --maxDownloadSpeed=BITSPERSEC — These flags set a limit on the bandwidth utilized for individual block uploads and downloads (i.e., the setting applies on a per-thread basis) 🚦. The limits only apply to HTTP payload data and do not include any additional overhead from HTTP or TCP headers, etc.

The value is measured in bits per second, and abbreviations like 256k, 1m, etc. may be used. By default, there is no fixed limit.

Use of these flags may also require setting the --timeout flag to a higher value.

  • --maxRetryPause=MILLIS — Sets the total amount of time in milliseconds s3backer should pause when retrying failed HTTP operations before giving up 😤. Failures include network failures and timeouts, HTTP errors, and reads of stale data (i.e., MD5 mismatch); s3backer will make multiple retry attempts using an exponential backoff algorithm, up to this maximum total retry pause time. This value does not include the time it takes to perform the HTTP operations themselves (use --timeout for that). Default value is 30000 (30 seconds). See also --initialRetryPause.
  • --minWriteDelay=MILLIS — okay so this one sets a minimum time window in milliseconds between one write succeeding and the next write to the same block even being allowed to start 🫶. Why? So S3 doesn't get your writes out of order — it's literally just queue etiquette for the cloud. Important: this value must be set to zero when --md5CacheSize is set to zero (aka MD5 cache disabled). Default value is 500ms.
  • --md5CacheSize=SIZE — sets how big the MD5 checksum cache is, measured in number of blocks 💅. Heads up: if the cache is full when a new block gets written, that write will literally block until there's room. So configure --md5CacheTime and --md5CacheSize to match how often the filesystem gets written overall, and how often the same block gets written repeatedly. Alternatively — a value equal to the number of blocks in the filesystem nukes this problem entirely, but it snacks the most memory when full (each entry in the cache is approximately 40 bytes). A value of zero disables the MD5 cache. Default value is zero (disabled).
  • --md5CacheTime=MILLIS — how long, in milliseconds, after a block has been successfully written that its MD5 checksum should stay cached, so we can detect stale data on later reads 🌸. A value of zero means infinite and gives you a straight-up guarantee against reading stale data — but only do this when --md5CacheSize is configured equal to the number of blocks; otherwise deadlock will (eventually) occur 😭. This value must be at least as big as --minWriteDelay. This value must be set to zero when --md5CacheSize is set to zero (MD5 cache disabled). Default value is 10 seconds.

The MD5 checksum cache is not persisted across restarts. So to keep the same eventual-consistency protection while s3backer isn't running, you must delay at least --md5CacheTime milliseconds between stopping and restarting s3backer. No skipping that cooldown bestie ⏳.

  • --nbd — make a Network Block Device (NBD) instead of a FUSE filesystem ✨. Honestly this is probably a more logical and direct way to do things on systems that support NBD.

s3backer will start up appropriate instances of nbdkit(1) and nbd-client(1) in order to attach the S3 volume to the specified device.

When you invoke it in this mode, only --doubleDash style flags are supported, with the exception of -f and -d; any other flags (including any FUSE-specific flags) are disallowed 🚫. And the --nbd flag may not appear in a --configFile file.

To detach the S3 volume, just kill the s3backer process — it'll automatically disconnect the NBD client and shut down the NBD server. Clean exit, no mess, we love to see it 💅.

  • --nbd-flag=string
  • --nbd-param=string — add string as an nbdkit(1) command line parameter, either before or after the plugin name (respectively) 🎀.
  • --noAutoDetect — disables block and file size auto-detection at startup 🙈. If you give this flag, then the block size defaults to 4096 and the --size flag is required.
  • --noCurlCache — disables caching of cURL handles. This flag will reduce performance; it exists to help in certain debugging situations 🔍.
  • --password=PASSWORD — supply the password for encryption and authentication as a command-line parameter 🔐.
  • --passwordFile=FILE — read the password for encryption and authentication from (the first line of) the specified file 📄.
  • --prefix=STRING — sets a prefix to prepend to the resource names within the bucket that identify each block 🦋. Use different prefixes and multiple independent s3backer disks can live in the same S3 bucket. Iconic, honestly.

Instead of using this flag, it's recommended to specify a prefix implicitly by appending a slash and a subdirectory name to the bucket name (the S3 bucket namespace is flat, so the notion of a subdirectory is not real). The configured prefix then becomes the given subdirectory name, followed by slash. The AWS web console will then display the virtual disk's blocks grouped into a subfolder 😌.

For example, specifying bucket mybucket/foo/bar is equivalent to specifying bucket mybucket along with the flag --prefix=foo/bar/ (note the additional trailing slash ⚠️).

Extraneous slash characters in a subdirectory name are disallowed.

This flag (or subdir) must be used consistently once a disk image is established 🫡.

The default prefix is the empty string.

  • --quiet — suppress progress output during initial startup 🤫.
  • --readAhead=NUM — configure the number of blocks of read ahead. This determines how many blocks will be read into the block cache ahead of the last block read by the kernel when read ahead is active 🏃‍♀️. This option has no effect if the block cache is disabled. Default value is 4 in FUSE mode, zero in NBD mode.
  • --readAheadTrigger=NUM — configure the number of blocks that must be read consecutively before the read ahead algorithm is triggered ⚡. Once triggered, read ahead will continue as long as the kernel continues reading blocks sequentially. This option has no effect if the block cache is disabled. Default value is 2 in FUSE mode, zero in NBD mode.
  • --readOnly — assume the filesystem is going to be mounted read-only, and return EROFS in response to any attempt to write 🔒. This flag also changes the default mode of the backed file from 0600 to 0400 and disables the MD5 checksum cache.
  • --region=REGION — specify an AWS region 🌍. This flag changes the default base URL to include the region name and automatically sets the --vhost flag, unless the --no-vhost flag is used.

The default region is us-east-1.

  • --reset-mounted-flag — reset the 'already mounted' flag on the underlying S3 data store 🔄.

s3backer detects simultaneous mounts by checking a special flag. If a previous invocation of s3backer was not shut down cleanly, the flag may not have been cleared. Running s3backer --erase will clear it manually. But see also BUGS below 👀.

  • --size=SIZE — specify the size (in bytes) of the backed file to be exported by the filesystem 📏. The size may have an optional suffix K for kilobytes, M for megabytes, G for gigabytes, T for terabytes, E for exabytes, Z for zettabytes (if supported), or Y for yottabytes (if supported). The total number of blocks must fit into 32 bits. Suffixes represent powers of two.

s3backer will attempt to auto-detect the size of the backed file by reading block number zero. If this option is not specified, the auto-detected value will be used. If this option is specified but disagrees with the auto-detected value, s3backer will exit with an error unless --force is also given 💀.

  • --sse=TYPE — enable server side encryption 🔐. This adds the x-amz-server-side-encryption header to all PUT requests.

The value must be either AES256 or aws:kms. In the latter case, a customer-supplied key is used for encryption, and the --sse-key-id flag is required.

  • --sse-key-id=ID — specify the ID of the customer-supplied key for server-side encryption 🗝️. This flag is required if --sse is given, ignored otherwise.
  • --ssl — equivalent to --baseURL https://s3.amazonaws.com/ 🔒✨
  • --statsFilename=NAME — specify the name of the human-readable statistics file that appears in the s3backer filesystem 📊. A value of empty string disables the appearance of this file. Default is stats.
  • --storageClass=TYPE — specify storage class 🗄️.

Valid values are: STANDARD, STANDARD_IA, ONEZONE_IA, REDUCED_REDUNDANCY, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, and OUTPOSTS.

The default is STANDARD.

  • --test — operate in local test mode 🧪. Filesystem blocks are stored as regular files in the directory dir. No network traffic occurs.

Note if dir is a relative pathname (and -f is not given) it will be resolved relative to the root directory 🙃.

  • --test-delays — in test mode, introduce random I/O delays 🐢.
  • --test-discard — in test mode, discard all data written, return zero for all data blocks read, and perform no I/O operations 🕳️. This mode is useful for isolating the FUSE and s3backer performance overhead.
  • --test-errors — in test mode, introduce random I/O errors 💥.
  • --timeout=SECONDS — specify a time limit in seconds for one HTTP operation attempt ⏱️. This limits the entire operation including connection time (if not already connected) and data transfer time. The default is 30 seconds; this value may need to be adjusted upwards to avoid premature timeouts on slower links and/or when using a large number of block cache worker threads.

See also --maxRetryPause.

This flag is required when S3 buckets have been created with location constraints (for example EU buckets). Put another way, this flag is required for buckets defined outside of the US region. This flag is automatically set when the --region flag is used, unless the --no-vhost flag is used.

  • --no-vhost — disable virtual hosted style requests (the default) 🚫🌐.

In addition, s3backer accepts all of the generic FUSE options as well 🎀. Here is a partial list:

  • -o uid=UID — override the user ID of the backed file, which defaults to the current user ID 🫶.
  • -o gid=GID — override the group ID of the backed file, which defaults to the current group ID 🫶.
  • -o sync_read — do synchronous reads 😴.
  • -o max_readahead=NUM — set maximum read-ahead (in bytes) 📚.
  • -f — run in the foreground (do not fork) 🧍‍♀️. Causes logging to be sent to standard error.
  • -d — enable FUSE debug mode 🐛. Implies -f.
  • -s — run in single-threaded mode 🧵.

...and on TOP of all that, s3backer quietly hands these flags to FUSE for you 💅 (unless you override them yourself on the command line — your call, bestie):

-o kernel_cache
-o fsname=<baseURL><bucket>/<prefix>
-o subtype=s3backer
-o use_ino
-o entry_timeout=31536000
-o negative_timeout=31536000
-o max_readahead=0
-o attr_timeout=0
-o default_permissions
-o allow_other
-o nodev
-o nosuid

Like, that's the whole optimized-for-s3backer starter pack 🎀. Just vibes that make everything faster.

🗂️ FILES 🗂️

  • $HOME/.s3backer_passwd — this is where your Amazon S3 accessID:accessKey pairs live. Your little secret stash, no cap 🔐.
  • /run/s3backer-nbd — holds the UNIX socket files that nbd-client and nbdkit(1) use when you're running in --nbd mode. And listen — for security reasons this directory is only readable by root, periodt. Don't try it. 🙅‍♀️

👀 SEE ALSO 👀

💀 BUGS 💀

Okay so, brace yourself, because some of these are rough 😭

The FUSE unmount flaw (fr fr). Due to a design flaw in FUSE, an unmount of the s3backer filesystem will complete successfully before s3backer has actually finished writing back all its dirty blocks. So when you're using the block cache, trying to remount the same bucket and prefix may fail with an 'already mounted' error while the former s3backer process is still out there flushing its cache. Before you assume it's a false positive and go reaching for --reset-mounted-flag, make sure any previous s3backer process attached to that same bucket and prefix has actually exited. See issue #40 for the details. Don't skip this step bestie, I'm serious 💅
The 32-bit block index thing. For cache space efficiency, s3backer uses 32 bit values to index individual blocks. Which means: when you're creating a very large filesystem (greater than 16 terabytes), you must bump the block size up beyond the default 4K. No way around it 🔥
Mac OS X timeout situation. On Mac OS X, the kernel imposes its own timeout of 600 seconds on FUSE operations, and it will automatically unmount the filesystem once that limit is hit 😬. This can happen when a combination of --maxRetryPause and/or --timeout settings lets HTTP retries take longer than that value. A warning gets emitted on startup in this case, so pay attention to it.
  • Filesystem size is limited by the maximum allowable size of a single file. That's just how it is 🌸
  • The default block size of 4k is non-optimal from a compression and cost perspective. Typically, users will want a larger value to maximize compression and minimize transaction costs — e.g., 1m. Highly recommend, ngl 💖
  • When running under NBD, the stats file is not available. Soz 🫶

👑 AUTHOR 👑

Archie L. Cobbs <archie.cobbs@gmail.com> — the icon who made all this happen, we stan 🫡✨

And that's the whole page, bestie! This little masterpiece comes to you from Debian, dated September 7, 2009, and it's the S3BACKER(1) manual page. Go mount something iconic 💅🦋🌸

📦 where this page comes from