LEIL-CHUNKSERVER.CFG(5)

📦 version 5.12.0, in nixpkgs unstable

🦋💅 LEIL-CHUNKSERVER.CFG(5) 💅🦋

✨ NAME ✨

  • leil-chunkserver.cfg — the main configuration file for leil-chunkserver 🫶

🌸 DESCRIPTION 🌸

Okay so bestie, leil-chunkserver.cfg is the file that contains the configuration of the LeilFS chunkserver process. That’s the whole vibe. 💖

🎀 SYNTAX 🎀

The syntax is literally this:

OPTION = VALUE

Lines starting with # character are ignored. Yes, queen, comments are free. 🎀

💅 OPTIONS 💅

Configuration options, coming at you one by one, no skips:

  • DATA_PATH — where to store files with usage statistics and the daemon lock file 🗂️
  • LABEL — the label of this chunkserver, for tiering 🏷️
  • WORKING_USER — user to run daemon as 👤
  • WORKING_GROUP — group to run daemon as. It’s optional — if empty, then the default user group will be used 🫶
  • SYSLOG_IDENT — name of process to place in syslog messages. Default is leil-chunkserver 📝
  • LOCK_MEMORY — whether to perform mlockall() to avoid swapping out the leil-chunkserver process. Default is 0, i.e. no 🔒
  • LIMIT_GLIBC_MALLOC_ARENAS — Linux only: limit glibc malloc arenas to the given value — prevents using a huge amount of virtual memory. This can influence performance by reducing memory fragmentation and improving cache locality, but it may also lead to contention and reduced parallelism in multi-threaded applications. Use it in constrained memory environments; recommended values are 4 or 8. Default is 0: disabled or let glibc decide 🧠
  • MALLOC_TRIM_INTERVAL — interval in seconds at which the chunkserver will attempt to release unused memory back to the OS using malloc_trim(). This can help reduce the process’s memory footprint by returning free memory to the system more frequently. Set to 0 to disable memory trimming entirely. Default is 0, 0 = disabled 🧹
  • NICE_LEVEL — nice level to run daemon with. Default is -19 if possible; note: process must be started as root to increase priority ⚡
  • MASTER_HOST — address of LeilFS master host to connect with. Default is leil-master 🌐
  • MASTER_PORT — number of LeilFS master port to connect with. Default is 9420 🔌
  • CLUSTER_ID — only connect to metadata servers with this CLUSTER_ID. Prevents accidental connections to wrong metadata servers. Default is default 🛡️
  • MASTER_RECONNECTION_DELAY — delay in seconds before trying to reconnect to the master server after disconnection. Default is 5 ⏳
  • MASTER_TIMEOUT — timeout, in seconds, for the master server connection. Default is 60, minimum is 0.01 ⏱️
  • MASTER_NR_OF_WORKERS — number of threads that the connection to master may use to process operations on chunks, like create, duplicate, replicate and truncate. Default is 10, minimum is 2 🧵
  • MASTER_REPLICATION_NR_OF_WORKERS — number of threads that the connection to master may use to process replications operations on chunks. Default is 5, minimum in 1 🔁
  • BIND_HOST — local address to use for connecting with the master server. Default is *, i.e. default local address 🏠
  • CSSERV_LISTEN_HOST — IP address to listen on for client (mount) connections. * means any 👂
  • CSSERV_LISTEN_PORT — port to listen on for client (mount) connections. Default is 9422 🚪
  • HDD_CONF_FILENAME — alternative name of leil-hdd.cfg file 📁
  • HDD_LEAVE_SPACE_DEFAULT — free space threshold to set volume as 100% utisaued when there is less than the given amount of free space left. Default is "4GiB". This number is always added to the used disk space reported by chunkserver 💾
  • HDD_TEST_FREQ — chunk test period in seconds. Default is 10 🩺
  • HDD_CHECK_CRC_WHEN_READING — whether to check the CRC on every read operation. Default is 1 📖
  • HDD_CHECK_CRC_WHEN_WRITING — whether to check the CRC on every write operation. The safest and recommended value is 1; disable it only to test maximum throughput. Default is 1 ✍️
  • HDD_ADVISE_NO_CACHE — whether to remove each chunk from page when closing it to reduce cache pressure generated by chunkserver. Default is 0, i.e. no 🧊
  • HDD_CHUNK_BULK_SIZE — ADVANCED:: number of chunks to process in a single bulk operation when reporting chunks to the master (e.g., when registering chunks, reporting new chunks, or reporting damaged or lost chunks). Reloadable. Default is 1000, minimum is 1, maximum is 100000 📦
  • HDD_PUNCH_HOLES — if enabled then chunkserver detects zero values in chunk data and frees corresponding file blocks, decreasing file system usage. This option works only on Linux with file systems supporting punching holes (XFS, ext4, Btrfs, tmpfs) 🕳️
  • ENABLE_LOAD_FACTOR — if enabled, chunkserver will send periodical reports of its I/O load to master, which will be taken into consideration when picking chunkservers for I/O operations 📊
  • REPLICATION_BANDWIDTH_LIMIT_KBPS — limit how many kilobytes can be replicated from other chunkservers to this chunkserver in every second. By default undefined, i.e. no limits 🚦
  • NR_OF_NETWORK_WORKERS — number of threads which handle, in a round-robin manner, connections with clients. Default is 4; these threads are responsible for reading from sockets and coping data from internal buffers to sockets 🧑‍💻
  • NR_OF_HDD_WORKERS_PER_NETWORK_WORKER — number of threads that each network worker may use to do disk operations like opening chunks, reading or writing them. Default is 16 💽
  • BGJOBSCNT_PER_NETWORK_WORKER — maximum number of jobs that each network worker may use for disk operations. Default is 4000 🧰
  • POLL_TIMEOUT_MS — maximum amount of time in milliseconds that the polling operation will wait for events. In the chunkservers, the same value is applied for the events loop and for the network worker threads. Smaller values could reduce latency at the cost of CPU usage. Default: 50 ⏲️
  • MAX_BLOCKS_PER_HDD_WRITE_JOB — maximum number of blocks written to the device in a single write job. If possible, will try to make a single pwrite call. Default is 16, minimum is 1, maximum is 64 🧱
  • MAX_BLOCKS_PER_HDD_READ_JOB — maximum number of blocks read from the device in a single read job. If possible, will try to make a single pread call. Default is 16, minimum is 1 📚
  • MAX_PARALLEL_HDD_READ_JOBS_PER_CS_ENTRY — maximum number of read jobs processed in parallel per client connection. Default is 1 🚀
  • MAX_BUFFERS_POOL_SIZE_MB — maximum size of the buffers pool used for read, write and replicate operations, in megabytes. Consider decreasing it if the system is running out of memory, or increasing it to improve performance if there is enough memory available. Unused buffers are regularly freed. Default is 512 🫧
  • READ_AHEAD_KB — additional number of kilobytes which should be passed to posix_fadvise(POSIX_FADV_WILLNEED) before reading data from a chunk. Default is 0, i.e. use posix_fadvise only with the amount of data that is really needed; the value is aligned down to 64 KiB 👀
  • MAX_READ_BEHIND_KB — try to fix out-of-order read requests; the value tells how much of skipped data to read if an offset of some read operation is greater than the offset where the previous operation finished. Default is 0, i.e. don’t read any skipped data; the value is aligned down to 64 KiB 🔄
  • PERFORM_FSYNC — call fsync() after a chunk is modified. In the case of individual write jobs it does not apply, but when closing the chunks. Default is 1, i.e. enabled 💽
  • STAT_CHUNKS_AT_DISK_SCAN — verify that chunk metadata and data parts exists at scan time. If set to 1, the data part is also checked to be multiple of SFSBLOCKSIZE. Disabling this option accelerates the scan process, but does not check the chunk files. Warning: use it with caution, only in scenarios where a fast chunkserver restart is needed. Default is 1, i.e. enabled 🔍
⚠️ Warning: STAT_CHUNKS_AT_DISK_SCAN should be used with caution, only in scenarios where a fast chunkserver restart is needed. It accelerates the scan process but does not check the chunk files. Do not just vibe-check this one blindly, bestie. 💀
  • REPLICATION_TOTAL_TIMEOUT_MS — total timeout for single replication operation. Replications that take longer than that are considered failed and are immediately aborted. Default: 60000 ⏰
  • REPLICATION_CONNECTION_TIMEOUT_MS — connection for single replication operation. If connecting to another chunkserver takes longer than this timeout, the operation is considered failed and is immediately aborted. Default: 1000 📡
  • REPLICATION_WAVE_TIMEOUT_MS — ADVANCED:: timeout for single wave in replication. After this timeout, next wave of read requests is sent to other chunkservers. Default: 500 🌊
  • PLUGINS_DIR — directory to search for Chunkserver plugins, as an absolute path. When set, the plugin manager will search this directory first. If not set, then it will search in usual plugin locations, like PLUGINS_PATH/chunkserver (PLUGINS_PATH is defined at build time), /usr/local/lib/saunafs/plugins/chunkserver, and /usr/lib/saunafs/plugins/chunkserver. Default is empty string, i.e. disabled 🔌
  • CHUNK_TRASH_ENABLED — enables or disables the chunk trash feature. When enabled, deleted chunks are moved to a trash directory instead of being immediately removed. Default: 0 🗑️
  • CHUNK_TRASH_EXPIRATION_SECONDS — specifies the timeout in seconds for chunks to remain in the trash before being permanently deleted. Default: 259200 ⏳
  • CHUNK_TRASH_FREE_SPACE_THRESHOLD_GB — sets the available space threshold in gigabytes. If the available space on the disk falls below this threshold, the system will start deleting older chunks from the trash to free up space. A suggested value is about the 15% of the capacity of the smaller configured disk. Default: 0 💽
  • CHUNK_TRASH_GC_BATCH_SIZE — defines the maximum bulk size per disk for the garbage collector when processing chunks in the trash. This value is used when no I/O pressure is detected and is dynamically scaled down based on current disk load. Default: 500 ♻️
  • CHUNK_TRASH_GC_SPACE_RECOVERY_BATCH_SIZE — [ADVANCED] the number of files to remove from the trash in a single GC cycle, in case the disk is full and space needs to be recovered. Default: 100 🧹
  • CHUNK_TRASH_GC_PURGE_WORKERS_NR — the number of workers purging files in parallel. Default: 5 👯
  • LOG_LEVEL — setup logging. Uses the environment variable SAUNAFS_LOG_LEVEL or config value LOG_LEVEL to determine logging level. Valid log levels are:
    • trace
    • debug
    • info
    • warn or warning
    • err or error
    • crit or critical
    • off

💅 IO_PRIORITY_MODE (EXPERIMENTAL) 💅

okay so this one defines the prioritization scheme for IO jobs, i.e. reads and writes from drives. the available modes are:

  • FIFO — prioritizes the IO jobs that were enqueued first. first come, first served energy ✨
  • SWITCH — alternates read and write jobs (interleaving) so one class of IO doesn't starve the other. Not reloadable.

Default: FIFO 🎀 (that's the default, bestie)

🫧 WRITE_BUFFERING_SIZE_MB (EXPERIMENTAL) 🫧

Caps the amount of data in MiB the chunkserver "buffers" — like, it fast-replies to the client as if it was synced to the drives, but it was only read from the network and stored in memory. Similar to page cache. This feature has the following constraints:

  • does not work on standard (replica) chunks.
  • must have chunkserver side chunk locking enabled (see USE_CHUNKSERVER_SIDE_CHUNK_LOCK in master config).
  • it is also suggested to set IO_PRIORITY_MODE to SWITCH.

The PERFORM_FSYNC continues working as intended, as it affects the time spent closing of the written chunk. Can provide better results for writes in general, but may significantly improve performance for small files. Reloadable. (Default: 0)

🔐 TLS_CERT_FILE (EXPERIMENTAL) 🔐

Path to the TLS certificate file the chunk server will use for TLS connections. There is no default value. ngl, you gotta hand it over.

🔑 TLS_KEY_FILE (EXPERIMENTAL) 🔑

Path to the TLS private key file the chunk server will use for TLS connections. There is no default value. again, no default — bring your own key 💅

🛡️ TLS_CA_CERT_FILE (EXPERIMENTAL) 🛡️

Path to the trusted CA certificate which is used to authenticate through TLS to the master server. There is no default value. yep, you provide this too ✨

🐛 REPORTING BUGS 🐛

Report bugs to the GitHub repository https://github.com/leil-io/leilfs as an issue. periodt. 🫶

Copyright 2008-2009 Gemius SA

Copyright 2013-2019 Skytechnology sp. z o.o.

Copyright 2023-2024 Leil Storage OÜ

LeilFS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

LeilFS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with LeilFS. If not, see http://www.gnu.org/licenses/.

👑 SEE ALSO 👑

and that's the whole manual vibe for LEIL-CHUNKSERVER.CFG(5), dated 1980-01-01 — stay slaying, bestie 💅✨🌈

📦 where this page comes from