LEIL-MASTER.CFG(5)

📦 version 5.12.0, in nixpkgs unstable

✨ NAME ✨

leil-master.cfg(5) — okay bestie, this is the main configuration file for leil-master. 💖

🎀 DESCRIPTION 🎀

The file leil-master.cfg contains the configuration of the LeilFS metadata server process. It’s literally the main character of your metadata config stack. ✨

🦋 SYNTAX 🦋

Syntax is:

OPTION = VALUE

Lines starting with # character are ignored.

👑 OPTIONS 👑

Configuration options:

  • PERSONALITY — Current personality of this instance of metadata server. Valid values are master, shadow and ha-cluster-managed. If installation is managed by an HA cluster the only valid value is ha-cluster-managed; otherwise the only valid values are master and shadow, in which case only one metadata server in LeilFS shall have master personality. master means that this instance of metadata server acts as main metadata server governing all file system metadata modifications. shadow means that this instance of metadata server acts as backup metadata server ready for immediate deployment as new master in case of current master failure. Metadata server personality can be changed at any moment as long as one changes personality from shadow to master; changing personality the other way around is forbidden. ha-cluster-managed means that this instance is managed by HA cluster, server runs in shadow mode as long as it’s not remotely promoted to master.
  • CLUSTER_ID — Uniquely identifies this namespace. Prevents accidental connections to wrong metadata or chunkservers. (default is default)
  • DATA_PATH — where to store metadata files and lock file
  • WORKING_USER — user to run daemon as
  • WORKING_GROUP — group to run daemon as (optional — if empty then default user group will be used)
  • SYSLOG_IDENT — name of process to place in syslog messages (default is leil-master)
  • LOCK_MEMORY — whether to perform mlockall() to avoid swapping out leil-master process (default is 0, i.e. no)
  • LIMIT_GLIBC_MALLOC_ARENAS — Linux only: limit glibc malloc arenas to given value — prevents from using huge amount of virtual memory. This can influence performance by reducing memory fragmentation and improving cache locality, but it may also lead to contention and reduced parallelism in multi-threaded applications. Use it in constrained memory environments, recommended values are 4 or 8. (default is 0: disabled or let glibc decide)
  • NICE_LEVEL — nice level to run daemon with (default is -19 if possible; note: process must be started as root to increase priority)
  • EXPORTS_FILENAME — alternative name of leil-exports.cfg file
  • TOPOLOGY_FILENAME — alternative name of leil-topology.cfg file
  • CUSTOM_GOALS_FILENAME — alternative name of leil-goals.cfg file
  • PREFER_LOCAL_CHUNKSERVER — If a client mountpoint has a local chunkserver, and a given chunk happens to reside locally, then leil-master will list the local chunkserver first. However, when the local client mount is issuing many read(s)/write(s), to many local chunks, these requests can overload the local chunkserver and disk subsystem. Setting this to 0 (the default is 1) means that remote chunkservers will be considered as equivalent to the local chunkserver. This is useful when the network is faster than the disk, and when there is high-IO load on the client mountpoints.
  • BACK_LOGS — number of metadata change log files (default is 50)
  • BACK_META_KEEP_PREVIOUS — number of previous metadata files to be kept (default is 1)
  • AUTO_RECOVERY — when this option is set (equals 1) master will try to recover metadata from changelog when it is being started after a crash; otherwise it will refuse to start and leil-metarestore should be used to recover the metadata (default is 0)
  • OPERATIONS_DELAY_INIT — initial delay in seconds before starting chunk operations (default is 300)
  • OPERATIONS_DELAY_DISCONNECT — chunk operations delay in seconds after chunkserver disconnection (default is 3600)
  • MATOML_LISTEN_HOST — IP address to listen on for metalogger connections (* means any)
  • MATOML_LISTEN_PORT — port to listen on for metalogger connections (default is 9419)
  • MATOML_LOG_PRESERVE_SECONDS — how many seconds of change logs have to be preserved in memory (default is 600; note: logs are stored in blocks of 5k lines, so sometimes real number of seconds may be little bigger; zero disables extra logs storage)
  • MATOCS_LISTEN_HOST — IP address to listen on for chunkserver connections (* means any)
  • MATOCS_LISTEN_PORT — port to listen on for chunkserver connections (default is 9420)
  • MATOCL_LISTEN_HOST — IP address to listen on for client (mount) connections (* means any)
  • MATOCL_LISTEN_PORT — port to listen on for client (mount) connections (default is 9421)
  • MATOTS_LISTEN_HOST — IP address to listen on for tapeserver connections (* means any)
  • MATOTS_LISTEN_PORT — Port to listen on for tapeserver connections (default is 9424)
  • CHUNKS_LOOP_MAX_CPS — Chunks loop shouldn’t check more chunks per seconds than given number (default is 100000)
  • CHUNKS_LOOP_MIN_TIME — Chunks loop will check all chunks in specified time (default is 300) unless CHUNKS_LOOP_MAX_CPS will force slower execution.
  • CHUNKS_LOOP_PERIOD — Time in milliseconds between chunks loop execution (default is 1000).
  • CHUNKS_LOOP_MAX_CPU — Hard limit on CPU usage by chunks loop (percentage value, default is 60).
  • CHUNKS_SOFT_DEL_LIMIT — Soft maximum number of chunks to delete on one chunkserver (default is 10)
  • CHUNKS_HARD_DEL_LIMIT — Hard maximum number of chunks to delete on one chunkserver (default is 25)
  • CHUNKS_WRITE_REP_LIMIT — Maximum number of chunks to replicate to one chunkserver (default is 2)
  • CHUNKS_READ_REP_LIMIT — Maximum number of chunks to replicate from one chunkserver (default is 10)
  • ENDANGERED_CHUNKS_PRIORITY — Percentage of endangered chunks that should be replicated with high priority. Example: when set to 0.2, up to 20% of chunks served in one turn would be extracted from endangered priority queue. When set to 1 (max), no other chunks would be processed as long as there are any endangered chunks in the queue (not advised) (default is 0, i.e. there is no overhead for prioritizing endangered chunks).
  • ENDANGERED_CHUNKS_MAX_CAPACITY — Max capacity of endangered chunks queue. This value can limit memory usage of master server if there are lots of endangered chunks in the system. This value is ignored if ENDANGERED_CHUNKS_PRIORITY is set to 0. (default is 1Mi, i.e. no more than 1Mi chunks will be kept in a queue).
  • ACCEPTABLE_DIFFERENCE — A maximum difference between disk usage on chunkservers that doesn’t trigger chunk rebalancing (default is 0.1, i.e. 10%).
  • CHUNKS_REBALANCING_BETWEEN_LABELS — When balancing disk usage, allow moving chunks between servers with different labels (default is 0, i.e. chunks will be moved only between servers with the same label).
  • GLOBALIOLIMITS_FILENAME — Configuration of global I/O limits (default is no I/O limiting)
  • GLOBALIOLIMITS_RENEGOTIATION_PERIOD_SECONDS — How often mountpoints will request bandwidth allocations under constant, predictable load (default is 0.1)
  • GLOBALIOLIMITS_ACCUMULATE_MS — After inactivity, no waiting is required to transfer the amount of data equivalent to normal data flow over the period of that many milliseconds (default is 250)
  • METADATA_CHECKSUM_INTERVAL — how often metadata checksum shall be sent to backup servers (default is: every 50 metadata updates)
  • METADATA_CHECKSUM_RECALCULATION_SPEED — how fast should metadata be recalculated in background (default: 100 objects per function call)
  • DISABLE_METADATA_CHECKSUM_VERIFICATION — should checksum verification be disabled while applying changelog
  • NO_ATIME — when this option is set to 1 inode access time is not updated on every access, otherwise (when set to 0) it is updated (default is 0)
  • METADATA_SAVE_REQUEST_MIN_PERIOD — minimal time in seconds between metadata dumps caused by requests from shadow masters (default is 1800)
  • SESSION_SUSTAIN_TIME — Time in seconds for which client session data (e.g. list of open files) should be sustained in the master server after connection with the client was lost. Values between 60 and 604800 (one week) are accepted. (default is 86400)
  • USE_BDB_FOR_NAME_STORAGE — When this option is set to 1 Berkeley DB is used for storing file/directory names in file (DATA_PATH/name_storage.db). By default all strings are kept in system memory. (default is 0)
  • BDB_NAME_STORAGE_CACHE_SIZE — Size of memory cache (in MB) for file/directory names used by Berkeley DB storage. (default is 10)
  • AVOID_SAME_IP_CHUNKSERVERS — When this option is set to 1, process of selecting chunkservers for chunks will try to avoid using those that share the same ip. (default is 0)
  • REDUNDANCY_LEVEL — minimum number of required redundant chunk parts that can be lost before chunk becomes endangered (default is 0)
  • SNAPSHOT_INITIAL_BATCH_SIZE — This option can be used to specify initial number of snapshotted nodes that will be atomically cloned before enqueuing the task for execution in fixed-sized batches. (default is 1000)
  • SNAPSHOT_INITIAL_BATCH_SIZE_LIMIT — This option specifies the maximum initial batch size set for snapshot request. (default is 10000)
  • FILE_TEST_LOOP_MIN_TIME — Test files loop will try to check all files in specified time in seconds (default is 3600). It’s possible for the loop to take more time if the master server is busy or the machine doesn’t have enough processing power to make all the needed calculations.

Options below are mandatory for all Shadow instances:

  • MASTER_HOST — address of the host running LeilFS metadata server that currently acts as master
  • MASTER_PORT — port number where LeilFS metadata server currently running as master listens for connections from shadows and metaloggers (default is 9420)
  • MASTER_RECONNECTION_DELAY — delay in seconds before trying to reconnect to metadata server after disconnection (default is 1)
  • MASTER_TIMEOUT — timeout (in seconds) for metadata server connections (default is 60)
  • LOAD_FACTOR_PENALTY — set this and a percentage of the load gets added on top of each chunkserver's disk usage when the master is picking the most fitting chunkserver 💅 so the heavy-loaded ones get chosen for operations less often. No cap, but fr fr it's <pink>default 0 </pink> and the correct values live in the range from 0 to 0.5 🎀
  • PRIORITIZE_DATA_PARTS — when this is on, the master server prioritizes data parts in EC goals so they land on the chunkservers that have the higher percentage of available space ✨ lowkey could cause parities to always land in the same chunkservers if the cluster isn't well balanced, so keep that in mind 👀 <pink>default: 1 </pink>
  • POLL_TIMEOUT_MS — the maximum amount of time in milliseconds that the polling operation will wait for events 🕐 this value gets applied to the polling in the events loop. Smaller values could reduce latency at the cost of CPU usage, so it's giving tradeoff city 😭 <pink>default: 50 </pink>
  • ENABLE_PROMETHEUS (EXPERIMENTAL) — whether to enable Prometheus support and metric collection 📊 heads up bestie, this requires compiling with Prometheus support. Set it to either 1 to enable, or 0 to disable <pink>(default is 0)</pink>
  • PROMETHEUS_HOST (EXPERIMENTAL) — the host address where Prometheus metric data can be collected 🌈 it must be in the format of HOST:PORT <pink>(default 0.0.0.0:9499)</pink>
  • CREATE_EMPTY_FOLDERS_WHEN_SPACE_DEPLETED (EXPERIMENTAL) — when enabled, this option allows the system to create metadata for empty folders when storage space is depleted 🫶 this helps preserving the intended file and directory structure. When it's disabled, no metadata for folders will be created if there's insufficient storage space <pink>(default: 1)</pink>
  • LOG_LEVEL — setup logging, periodt 📝 it uses the environment variable SAUNAFS_LOG_LEVEL or the config value LOG_LEVEL to determine logging level. The valid log levels are:
    • trace
    • debug
    • info
    • warn or warning
    • err or error
    • crit or critical
    • off
  • TLS_CERT_FILE (EXPERIMENTAL) — path to the TLS certificate file the master/shadow server will use for TLS connections 🔐 there is no default value, so you gotta supply it yourself 💅
  • TLS_KEY_FILE (EXPERIMENTAL) — path to the TLS private key file the master/shadow server will use for TLS connections 🔑 there is no default value either, ngl you're on your own here 🫶
  • TLS_CA_CERT_FILE (EXPERIMENTAL) — path to the trusted CA certificate which is used to authenticate the TLS connection 🛡️ there is no default value ✨
  • USE_CHUNKSERVER_SIDE_CHUNK_LOCK (EXPERIMENTAL) — when set to 1, it enables sending chunk part lock messages to the chunkservers 🔒 this can be useful to track down which chunk parts are currently being written. Reloadable <pink>(default: 0)</pink>
  • EMPTY_RESERVED_FILES_PERIOD_MSECONDS (EXPERIMENTAL) — interval for periodic cleaning of reserved files, in milliseconds 🧹 if set to 0, the reserved files deletion is disabled <pink>(Default: 0)</pink>
⚠️ Warning ⚠️ Administrator-only option, bestie — do not play with this casually. Enabling periodic deletion may remove master-side references to reserved files that are still held by other applications or client instances that have not released them yet 😭 this can disrupt ongoing operations and lead to unexpected errors. Fr fr, it's giving chaos. Recommended approach: do not enable automatic deletion unless you fully understand the impact on running workloads 🫶 prefer manually freeing reserved files by locating each reference or process using them and stopping it before releasing the reservation. Iconic behaviour only, please 👑

🌸 NOTES 🌸

Chunks in master are tested in a loop 🔁 speed (or frequency) is regulated by two options: CHUNKS_LOOP_MIN_TIME and CHUNKS_LOOP_MAX_CPS. The first one defines the minimal time of the loop, and the second one defines the maximal number of chunk tests per second. Typically at the beginning, when the number of chunks is small, the time is constant and regulated by CHUNK_LOOP_MIN_TIME, but when the number of chunks becomes bigger, then the time of the loop can increase according to CHUNKS_LOOP_MAX_CPS ✨ slay, it scales.

Deletion limits are defined as a soft and a hard limit 💀 when the number of chunks to delete increases from loop to loop, then the current limit can be temporarily increased above the soft limit — but never above the hard limit 🙅‍♀️ periodt.

🦋 REPORTING BUGS 🦋

Report bugs to the GitHub repository https://github.com/leil-io/leilfs as an issue 🐛 ty bestie ily.

Copyright 2008-2009 Gemius SA

Copyright 2013-2019 Skytechnology sp. z o.o.

Copyright 2023-2024 Leil Storage OÜ

LeilFS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 💜

LeilFS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE 🔥 see the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with LeilFS. If not, see http://www.gnu.org/licenses/ 🎀

🌈 SEE ALSO 🌈

okay that's the whole page bestie, you made it to the end 🥳 this version is dated 1980-01-01 and it's the LEIL-MASTER.CFG(5) page, so go be the main character of your own cluster 💅✨🫶

📦 where this page comes from