SYNCTHING-BEP(7)

πŸ¦‹ syncthing-bep β€” Block Exchange Protocol v1

okay bestie, gather round, because this is the syncthing-bep(7) page and it's genuinely iconic. we're talking about the actual Block Exchange Protocol, v1, the whole thing. grab a snack πŸ“


✨ INTRODUCTION AND DEFINITIONS ✨

so. the Block Exchange Protocol (BEP) is what two or more devices speak to each other to form a cluster. that's the vibe. 🌈

each device has one or more folders of files, and those are described by the local model, which contains metadata and block hashes. the local model gets sent to the other devices in the cluster. now β€” the union of all files in the local models, with files selected for highest change version, forms the global model. and every device is out here striving to get its folders in sync with the global model by requesting missing or outdated blocks from the other devices in the cluster. main character energy, honestly.

file data is described and transferred in units of blocks, and each block is from 128 KiB (131072 bytes) to 16 MiB in size, in steps of powers of two. the block size may vary between files but is constant in any given file β€” except for the last block, which may be smaller. the way that last block just does its own thing πŸ’…

the key words β€œMUST”, β€œMUST NOT”, β€œREQUIRED”, β€œSHALL”, β€œSHALL NOT”, β€œSHOULD”, β€œSHOULD NOT”, β€œRECOMMENDED”, β€œMAY”, and β€œOPTIONAL” in this document are to be interpreted as described in RFC 2119 β€” https://datatracker.ietf.org/doc/html/rfc2119.html. no cap, that's literally the rule.


πŸŽ€ TRANSPORT AND AUTHENTICATION πŸŽ€

BEP is deployed as the highest level in a protocol stack, with the lower level protocols providing encryption and authentication. she sits on top. queen behavior πŸ‘‘

   +-----------------------------+
   |   Block Exchange Protocol   |
   |-----------------------------|
   | Encryption & Auth (TLS 1.3) |
   |-----------------------------|
   |      Reliable Transport     |
   |-----------------------------|
   v             ...             v

the encryption and authentication layer SHALL use TLS 1.3 or a higher revision. periodt.

the exact nature of the authentication is up to the application, BUT it SHALL be based on the TLS certificate presented at the start of the connection. possibilities include:

  • certificates signed by a common trusted CA
  • preshared certificates
  • preshared certificate fingerprints
  • certificate pinning combined with some out of band first verification

the reference implementation uses preshared certificate fingerprints (SHA-256) referred to as β€œDevice IDs”. iykyk πŸ”₯

okay important: there is no required order or synchronization among BEP messages except as noted per message type β€” any message type may be sent at any time and the sender need not await a response to one message before sending another. chaotic? yes. valid? also yes. πŸ’€

the underlying transport protocol MUST guarantee reliable packet delivery. non-negotiable.

now, in this document, in diagrams and text, β€œbit 0” refers to the most significant bit of a word. so β€œbit 15” is the least significant bit of a 16 bit word (int16), and β€œbit 31” is the least significant bit of a 32 bit word (int32). non protocol buffer integers are always represented in network byte order (i.e., big endian) and are signed unless stated otherwise β€” but when describing message lengths negative values do not make sense and the most significant bit MUST be zero. πŸ™ˆ

the protocol buffer schemas in this document are in proto3 syntax. this means, among other things, that all fields are optional and will assume their default value when missing. this does not necessarily mean that a message is valid with all fields empty β€” for example, an index entry for a file that does not have a name is not useful and MAY be rejected by the implementation. however! the folder label is for human consumption only, so an empty label should be accepted β€” the implementation will have to choose some way to represent the folder, perhaps by using the ID in its place or automatically generating a label. slay, adaptive. πŸ’–


🌸 PRE-AUTHENTICATION MESSAGES 🌸

AFTER establishing a connection, but BEFORE performing any authentication, devices MUST exchange Hello messages. it's giving knocking on the door before you're allowed in 🫢

Hello messages are used to carry additional information about the peer, which might be of interest to the user even if the peer is not permitted to communicate due to failing authentication. note that the certificate based authentication may be considered part of the TLS handshake that precedes the Hello message exchange β€” but even in the case that a connection is rejected, a Hello message must be sent before the connection is terminated. polite and strict. we love her.

Hello messages MUST be prefixed with an int32 containing the magic number 0x2EA7D90B, followed by an int16 representing the size of the message, followed by the contents of the Hello message itself.

   0                   1
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Magic             |
  |           (32 bits)           |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  /                               /
  \             Hello             \
  /                               /
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

the Hello message itself is in protocol buffer format with the following schema:

message Hello {
    string device_name    = 1;
    string client_name    = 2;
    string client_version = 3;
}

Fields (Hello message)

the device_name is a human readable (configured or auto detected) device name or host name, for the remote device. 🏷️

the client_name and client_version identifies the implementation. the values SHOULD be simple strings identifying the implementation name, as a user would expect to see it, and the version string in the same manner. an example client name is β€œsyncthing” and an example client version is β€œv0.7.2”. the client_version field SHOULD follow the patterns laid out in the Semantic Versioning standard β€” https://semver.org/ ✨

immediately after exchanging Hello messages, the connection MUST be dropped if the remote device does not pass authentication. no vibes pass = no entry. πŸšͺ

⭐ POST-AUTHENTICATION MESSAGES ⭐

every message post authentication is made up of several parts:

  • a header length word
  • a Header
  • a message length word
  • a Message
   0                   1
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |         Header Length         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  /                               /
  \            Header             \
  /                               /
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |         Message Length        |
  |           (32 bits)           |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  /                               /
  \            Message            \
  /                               /
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

the header length word is 16 bits. it indicates the length of the following Header message. the Header is in protocol buffer format. the Header describes the type and compression status of the following message. πŸ’…

the message is preceded by the 32 bit message length word and is one of the concrete BEP messages described below, identified by the type field of the Header.

as always, the length words are in network byte order (big endian).

message Header {
    MessageType        type        = 1;
    MessageCompression compression = 2;
}

enum MessageType {
    CLUSTER_CONFIG    = 0;
    INDEX             = 1;
    INDEX_UPDATE      = 2;
    REQUEST           = 3;
    RESPONSE          = 4;
    DOWNLOAD_PROGRESS = 5;
    PING              = 6;
    CLOSE             = 7;
}

enum MessageCompression {
    NONE = 0;
    LZ4  = 1;
}

when the compression field is NONE, the message is directly in protocol buffer format.

when the compression field is LZ4, the message consists of a 32 bit integer describing the uncompressed message length followed by a single LZ4 block. after decompressing the LZ4 block it should be interpreted as a protocol buffer message just as in the uncompressed case. compressed and still understood? that's the dream 🧚


πŸ‘‘ MESSAGE SUBTYPES πŸ‘‘

Cluster Config

this informational message provides information about the cluster configuration as it pertains to the current connection. a Cluster Config message MUST be the first post authentication message sent on a BEP connection. ✨ additional Cluster Config messages MAY be sent after the initial exchange to change the connection in place with the update configuration. if an implementation does not support that, it SHOULD close the connection. know your limits, bestie πŸ’œ

Protocol Buffer Schema

message ClusterConfig {
    repeated Folder folders = 1;
}

message Folder {
    string id                    = 1;
    string label                 = 2;
    FolderType type              = 3;
    FolderStopReason stop_reason = 7;
    reserved 4 to 6;

    repeated Device devices = 16;
}

message Device {
    bytes           id                         = 1;
    string          name                       = 2;
    repeated string addresses                  = 3;
    Compression     compression                = 4;
    string          cert_name                  = 5;
    int64           max_sequence               = 6;
    bool            introducer                 = 7;
    uint64          index_id                   = 8;
    bool            skip_introduction_removals = 9;
    bytes           encryption_password_token  = 10;
}

enum Compression {
    METADATA = 0;
    NEVER    = 1;
    ALWAYS   = 2;
}

enum FolderType {
    FOLDER_TYPE_SEND_RECEIVE      = 0;
    FOLDER_TYPE_SEND_ONLY         = 1;
    FOLDER_TYPE_RECEIVE_ONLY      = 2;
    FOLDER_TYPE_RECEIVE_ENCRYPTED = 3;
}

enum FolderStopReason {
    FOLDER_STOP_REASON_RUNNING = 0;
    FOLDER_STOP_REASON_PAUSED  = 1;
}

Fields (Cluster Config Message)

the folders field contains the list of folders that will be synchronized over the current connection. πŸ“

Fields (Folder Message)

the id field contains the folder ID, which is the unique identifier of the folder.

the label field contains the folder label, the human readable name of the folder. 🏷️

the folder_type field contains the type of the folder as defined by the FolderType enumeration.

the stop_reason field is set for folders that are currently stopped for any reason. the zero means that the folder is not stopped; further reasons are defined by the FolderStopReason enumeration.

the devices field is a list of devices participating in sharing this folder. 🫢

Fields (Device Message)

the device id field is a 32 byte number that uniquely identifies the device. for instance, the reference implementation uses the SHA-256 of the device X.509 certificate. πŸ”

  • name β€” the human-readable name the sending device picked for the device it’s describing. It MAY be empty, and honestly it doesn’t even need to be unique. πŸ’…
  • the list of addresses β€” this is the list of addresses the sending device uses to connect to the described device. 🫢
  • compression β€” tells you which compression mode is in use for this device and folder. Valid values are:
    • 0 β€” Compress metadata. This enables compression of metadata messages such as Index.
    • 1 β€” Compression disabled. No compression is used on any message.
    • 2 β€” Compress always. Metadata messages as well as Response messages are compressed.
  • cert_name β€” the expected certificate name for this device. It’s commonly blank, and blank means β€œuse the implementation default.” πŸŽ€
  • max_sequence β€” contains the highest sequence number of the files in the index. See Delta Index Exchange for how this field is used. πŸ”₯
  • introducer β€” set for devices that are trusted as cluster introducers. πŸ‘‘
  • index_id β€” contains the unique identifier for the current set of index data. See Delta Index Exchange for how this field is used. πŸ¦‹
  • skip_introduction_removals β€” says whether the remote device has opted to ignore introduction removals for the given device. This setting gets copied across when we’re being introduced to a new device. πŸ™ˆ
  • encryption_password_token β€” a token derived from the password, used to encrypt data sent to this device. If the device is the same as the device sending the message, it means the device itself has encrypted data that was encrypted with the given token. It’s empty or missing if there’s no encryption. See Untrusted Device Encryption for the details of the encryption scheme. πŸ’–

✨ Index and Index Update ✨

The Index and Index Update messages define the contents of the sender’s folder. An Index message represents the full contents of the folder, so it supersedes any previous index. An Index Update amends an existing index with new information and does not affect any entries that aren’t included in the message. An Index Update MAY NOT be sent unless it’s preceded by an Index β€” unless a non-zero Max Sequence has been announced for the given folder by the peer device. πŸ‘€

The Index and Index Update messages are currently identical in format, although this is not guaranteed to be the case in the future. ⭐

πŸ“¦ Protocol Buffer Schema

message Index {
    string            folder = 1;
    repeated FileInfo files  = 2;
}

message IndexUpdate {
    string            folder = 1;
    repeated FileInfo files  = 2;
}

message FileInfo {
    string       name           = 1;
    FileInfoType type           = 2;
    int64        size           = 3;
    uint32       permissions    = 4;
    int64        modified_s     = 5;
    int32        modified_ns    = 11;
    uint64       modified_by    = 12;
    bool         deleted        = 6;
    bool         invalid        = 7;
    bool         no_permissions = 8;
    Vector       version        = 9;
    int64        sequence       = 10;
    int32        block_size     = 13;

    repeated BlockInfo Blocks         = 16;
    string             symlink_target = 17;
}

enum FileInfoType {
    FILE              = 0;
    DIRECTORY         = 1;
    SYMLINK_FILE      = 2 [deprecated = true];
    SYMLINK_DIRECTORY = 3 [deprecated = true];
    SYMLINK           = 4;
}

message BlockInfo {
    int64 offset = 1;
    int32 size   = 2;
    bytes hash   = 3;
    reserved 4;
}

message Vector {
    repeated Counter counters = 1;
}

message Counter {
    uint64 id    = 1;
    uint64 value = 2;
}

πŸŽ€ Fields (Index Message)

  • folder β€” identifies the folder that the index message pertains to.
  • files β€” a list of files making up the index information.

πŸ’… Fields (FileInfo Message)

  • name β€” the file name path relative to the folder root. Like all strings in BEP, the Name is always in UTF-8 NFC, regardless of operating system or file system specific conventions. The name field uses the slash character "/" as path separator, regardless of the implementation’s operating system conventions. The combination of folder and name uniquely identifies each file in a cluster.
  • type β€” contains the type of the described item. The type is one of file (0), directory (1), or symlink (4).
  • size β€” contains the size of the file, in bytes. For directories and symlinks, the size is zero.
  • permissions β€” holds the common Unix permission bits. An implementation MAY ignore or interpret these as is suitable on the host operating system.
  • modified_s β€” the time expressed as the number of seconds since the Unix Epoch (1970-01-01 00:00:00 UTC). The modified_ns field holds the nanosecond part of the modification time.
  • modified_by β€” holds the short id of the client that last made any modification to the file, whether add, change or delete. This will be overwritten every time a change is made to the file by the last client to do so, so it does not hold history.
  • deleted β€” set when the file has been deleted. The block list SHALL be of length zero, and the modification time indicates the time of deletion or, if the time of deletion is not reliably determinable, the last known modification time.
  • invalid β€” set when the file is invalid and unavailable for synchronization. A peer MAY set this bit to indicate that it can temporarily not serve data for the file.
  • no_permissions β€” set when there is no permission information for the file. This is the case when it originates on a file system which does not support permissions. Changes to only permission bits SHOULD be disregarded on files with this bit set. The permissions bits MUST be set to the octal value 0666.
  • version β€” a version vector describing the updates performed to a file by all members in the cluster. Each counter in the version vector is an ID-Value tuple. The ID is the first 64 bits of the device ID. The Value is a simple incrementing counter, starting at zero. The combination of Folder, Name and Version uniquely identifies the contents of a file at a given point in time.
  • sequence β€” the value of a device local monotonic clock at the time of the last local database update to a file. The clock ticks on every local database update, thus forming a sequence number over database updates.
  • block_size β€” the size, in bytes, of each individual block in the block list (except, possibly, the last block). If this field is missing or zero, the block size is assumed to be 128 KiB (131072 bytes). Valid values of this field are the powers of two from 128 KiB through 16 MiB. See also Selection of Block Size.
  • the blocks list (Blocks) β€” contains the size and hash for each block in the file. Each block represents a block_size-sized slice of the file, except for the last block which may represent a smaller amount of data. The block list is empty for directories and symlinks.
  • symlink_target β€” contains the symlink target, for entries of symlink type. It is empty for all other entry types.

πŸ¦‹ Request πŸ¦‹

The Request message expresses the desire to receive a data block corresponding to a part of a certain file in the peer’s folder. 🫢

πŸ“¦ Protocol Buffer Schema

message Request {
    int32  id             = 1;
    string folder         = 2;
    string name           = 3;
    int64  offset         = 4;
    int32  size           = 5;
    bytes  hash           = 6;
    bool   from_temporary = 7;
    int32 block_no        = 9;
    reserved 8;
}

πŸŽ€ Fields

  • id β€” the request identifier. It will be matched in the corresponding Response message. Each outstanding request must have a unique ID.
  • folder and name β€” as documented for the Index message. The offset and size fields specify the region of the file to be transferred. This SHOULD equate to exactly one block as seen in an Index message.
  • hash β€” MAY be set to the expected hash value of the block. If set, the other device SHOULD ensure that the transmitted block matches the requested hash. The other device MAY reuse a block from a different file and offset having the same size and hash, if one exists.
  • from_temporary β€” set to indicate that the read should be performed from the temporary file (converting the name to its temporary form) and falling back to the non temporary file if any error occurs. Knowledge of contents of temporary files comes from DownloadProgress messages.

🌸 Response 🌸

The Response message is sent in response to a Request message. πŸ’Œ

πŸ“¦ Protocol Buffer Schema

message Response {
    int32     id   = 1;
    bytes     data = 2;
    ErrorCode code = 3;
}

enum ErrorCode {
    NO_ERROR     = 0;
    GENERIC      = 1;
    NO_SUCH_FILE = 2;
    INVALID_FILE = 3;
}

πŸŽ€ Fields

  • id β€” the request identifier. It must match the ID of the Request that is being responded to.
  • data β€” contains either the requested data block, or is empty if the requested block is not available.
  • code β€” contains an error code describing the reason a Request could not be fulfilled, in the case where zero length data was returned. The following values are defined:
    • 0 β€” No Error (data should be present)
    • 1 β€” Generic Error
    • 2 β€” No Such File (the requested file does not exist, or the offset is outside the acceptable range for the file)
    • 3 β€” Invalid (file exists but has invalid bit set or is otherwise unavailable)

πŸ“ DownloadProgress πŸ“

The DownloadProgress message is used to notify remote devices about partial availability of files. By default, these messages are sent every 5 seconds, and only in the cases where progress or state changes have been detected. Each DownloadProgress message is addressed to a specific folder and MUST contain zero or more FileDownloadProgressUpdate messages. ✨

πŸ“¦ Protocol Buffer Schema

message DownloadProgress {
    string                              folder  = 1;
    repeated FileDownloadProgressUpdate updates = 2;
}

message FileDownloadProgressUpdate {
    FileDownloadProgressUpdateType update_type   = 1;
    string                         name          = 2;
    Vector                         version       = 3;
    repeated int32                 block_indexes = 4;
}
enum FileDownloadProgressUpdateType {
    APPEND = 0;
    FORGET = 1;
}

πŸ’… Fields (DownloadProgress Message)

  • folder β€” the ID of the folder that this update is for. That's it, that's the field πŸŽ€
  • updates β€” a whole list of progress update messages. A lil stack of them ✨

🌸 Fields (FileDownloadProgressUpdate Message)

  • update_type β€” tells you which flavor of update this is: append (0), meaning new blocks just became available, or forget (1), meaning the file transfer has either completed or failed πŸ’€
  • name β€” the file name, straight from the global index, for the file this update is about
  • version β€” the version of the file this update is about
  • block_indexes β€” a list of positive integers, where each integer is the index into that file's Blocks array in the FileInfo message of a block that's now available for download

So like, an integer with the value 3 means the data in the fourth BlockInfo message of that file's FileInfo message is now available. Counting from zero, bestie, obviously πŸ™ˆ And please note: you match on name AND version. Both. Not one, both.

Also every update you receive is incremental. Example time: the first update might have indexes 0, 1, 2, and then an update 5 seconds later might have 3, 4, 5 which you append to the original list β€” which means blocks 0-5 are currently available. Slay, you're tracking it 🫢

Block indexes MAY be added in any order. An implementation MUST NOT assume block indexes are added in any specific order. No assumptions, no cap 🚫

If the forget field is set, that implies the previously advertised file is no longer available, so the list of block indexes should be truncated. Gone. Bye πŸ’…

Messages with the forget field set MUST NOT have any block indexes. Like, zero. None.

Any update message being sent for a different version of the same file name must be preceded by an update message for the old version of that file with the forget field set. Do the cleanup first, then the new thing, that's the order πŸ”₯

And here's the receiving-side safeguard: if the value of version changes between update messages, that implies the file changed and that any indexes previously advertised are no longer available. So the list of available block indexes MUST be replaced (not appended!!) with the indexes specified in this message. Replace, don't stack. Iykyk 😭

πŸ‘‘ Ping

The Ping message is how you check that a connection is still alive, and how you keep connections alive through state-tracking network elements like firewalls and NAT gateways. A Ping message is sent every 90 seconds, but only if no other message has been sent in the preceding 90 seconds. She only speaks up when it's been quiet ✨

πŸŽ€ Protocol Buffer Schema

message Ping {
}

πŸ’€ Close

The Close message MAY be sent to say "hey, this connection is being torn down because of an error condition." A Close message MUST NOT be followed by further messages. It's the last word, periodt.

πŸŽ€ Protocol Buffer Schema

message Close {
    string reason = 1;
}

🌸 Fields

  • reason β€” a human readable description of what went wrong 😭

πŸ”₯ SHARING MODES πŸ”₯

🫢 Trusted

Trusted mode is the default sharing mode. Updates flow in both directions, it's giving mutual, it's giving reciprocal πŸ’–

   +------------+     Updates      /---------\
   |            |  ----------->   /           \
   |   Device   |                 |  Cluster  |
   |            |  <-----------   \           /
   +------------+     Updates      \---------/

🌈 Send Only

In send only mode, the device does not apply any updates from the cluster β€” but it does publish changes of its local folder to the cluster as usual. She talks, she doesn't listen πŸ’…

   +------------+     Updates      /---------\
   |            |  ----------->   /           \
   |   Device   |                 |  Cluster  |
   |            |                 \           /
   +------------+                  \---------/

πŸ¦‹ Receive Only

In receive only mode, the device does not send any updates to the cluster β€” but it accepts changes to its local folder from the cluster as usual. She listens, she doesn't talk πŸŽ€

   +------------+     Updates      /---------\
   |            |  <-----------   /           \
   |   Device   |                 |  Cluster  |
   |            |                 \           /
   +------------+                  \---------/

✨ DELTA INDEX EXCHANGE ✨

Index data must be exchanged whenever two devices connect, so that one knows which files are available on the other. In the most basic case this happens by sending an Index message followed by one or more Index Update messages. Any previous index data known for a remote device gets removed and replaced with the new index data received in an Index message, while the contents of an Index Update message are simply added to the existing index data. Replace vs. append, keep track bestie 🫢

For situations with large indexes or frequent reconnects, that can be quite inefficient. So there's a mechanism you can use to retain index data between connections and only transmit the changes since that data when a connection starts. This is called "delta indexesβ€œ. To enable it, the sequence and index ID fields are used.

  • Sequence: Each index item (i.e., file, directory or symlink) has a sequence number field. It contains the value of a counter at the time the index item was updated. The counter increments by one for each change. So as files are scanned and added to the index they get assigned sequence numbers 1, 2, 3 and so on. The next file to be changed or detected gets sequence number 4, and future updates continue in the same fashion. It's a lil ticket number system, obsessed 🎟️
  • Index ID: Each folder has an Index ID. This is a 64 bit random identifier set at index creation time.

Given all that, we know the tuple {index ID, maximum sequence number} uniquely identifies a point in time of a given index. Any further changes will increase the sequence number of some item, and thus the maximum sequence number for the index itself. Should the index be reset or removed (i.e., the sequence number reset to zero), a new index ID must be generated.

By letting a device know the {index ID, maximum sequence number} we have for their index data, that device can arrange to only transmit Index Update messages for items with a higher sequence number. That's the delta index mechanism, and honestly? iconic πŸ‘‘

The index ID and maximum sequence number known for each device is transmitted in the Cluster Config message at connection start.

For this mechanism to be reliable it's essential that outgoing index information is ordered by increasing sequence number. Devices announcing a non-zero index ID in the Cluster Config message MUST send all index data ordered by increasing sequence number. Devices not intending to participate in delta index exchange MUST send a zero index ID or, equivalently, not send the index_id attribute at all. Pick one, don't be weird about it 😀


🌸 MESSAGE LIMITS 🌸

An implementation MAY impose reasonable limits on the length of messages and message fields, to aid robustness in the face of corruption or broken implementations. And an implementation should strive to keep messages short and to the point β€” favouring more and smaller messages over fewer and larger ones. Like, for example: favour a smaller Index message followed by one or more Index Update messages rather than sending one absolutely gigantic Index message. Small and many > big and few, no cap πŸ’…

The Syncthing implementation imposes a hard limit of 500,000,000 bytes on all messages. Attempting to send or receive a larger message will result in a connection close πŸ’€ This size was chosen to accommodate Index messages containing a large block list. It's intended that the limit may be further reduced in a future protocol update supporting variable block sizes (and thus shorter block lists for large files). So like, don't get attached to the number 🫢


πŸŽ€ SELECTION OF BLOCK SIZE πŸŽ€

The desired block size for any given file is the smallest block size that results in fewer than 2000 blocks, or the maximum block size for larger files. That rule gives you this lil table of block sizes per file size:

  • 0 - 250 MiB β†’ 128 KiB
  • 250 MiB - 500 MiB β†’ 256 KiB
  • 500 MiB - 1 GiB β†’ 512 KiB
  • 1 GiB - 2 GiB β†’ 1 MiB
  • 2 GiB - 4 GiB β†’ 2 MiB
  • 4 GiB - 8 GiB β†’ 4 MiB
  • 8 GiB - 16 GiB β†’ 8 MiB
  • 16 GiB - up β†’ 16 MiB

An implementation MAY deviate from the block size rule when there's a good reason to do so. For example, if a file has been indexed at a certain block size and then grows beyond 2000 blocks, it may be retained at the current block size for practical reasons. But when there's no overriding reason to the contrary β€” like when you're indexing a new file for the first time β€” the block size rule above SHOULD be followed. Bestie, follow the rule 🌟

An implementation MUST therefore accept files with a block size differing from the above rule. This does not mean arbitrary block sizes are allowed. The block size used MUST be exactly one of the power-of-two block sizes listed in the table above. No random numbers, no vibes-based sizing πŸ’€

🌈 EXAMPLE EXCHANGE 🌈

This table is the whole lore, bestie β€” read it top to bottom and watch who sends what to whom. πŸ‘€βœ¨

              β”Œβ”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
              β”‚ #  β”‚ A                      β”‚ B                      β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 1  β”‚ ClusterConfiguration-> β”‚ <-ClusterConfiguration β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 2  β”‚ Index->                β”‚ <-Index                β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 3  β”‚ IndexUpdate->          β”‚ <-IndexUpdate          β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 4  β”‚ IndexUpdate->          β”‚                        β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 5  β”‚ Request->              β”‚                        β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 6  β”‚ Request->              β”‚                        β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 7  β”‚ Request->              β”‚                        β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 8  β”‚ Request->              β”‚                        β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 9  β”‚                        β”‚ <-Response             β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 10 β”‚                        β”‚ <-Response             β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 11 β”‚                        β”‚ <-Response             β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 12 β”‚                        β”‚ <-Response             β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 13 β”‚ Index Update->         β”‚                        β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ …  β”‚                        β”‚                        β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 14 β”‚                        β”‚ <-Ping                 β”‚
              β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
              β”‚ 15 β”‚ Ping->                 β”‚                        β”‚
              β””β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

So the connection gets established, and at 1 both peers send ClusterConfiguration messages, then Index records. Those Index records are received, and both peers recompute their knowledge of the data in the cluster. In this example, peer A has four missing or outdated blocks β€” yep, four. At 5 through 8, peer A sends requests for those blocks. The requests are received by peer B, who retrieves the data from the folder and transmits Response records (9 through 12). Device A updates their folder contents and transmits an Index Update message (13). Both peers enter idle state after 13. At some later time 14, the ping timer on device B expires and a Ping message is sent. The same process occurs for device A at 15.


🫢 AUTHOR 🫢

The Syncthing Authors πŸ’œ

2014-2026, The Syncthing Authors


That’s the whole page, bestie β€” SYNCTHING-BEP(7), version v2.1.0, dated Oct 07, 2026. Now go sync something iconic. πŸ’…βœ¨πŸŽ€

πŸ“¦ where this page comes from