LLAMA-IDLE(1)

📦
version 10438 · 2 versions
, in ubuntu stonking

✨ NAME ✨

llama-idle — and bestie, the official description of this command is literally llama-idle - llama-idle. That's it. That's the whole vibe. Iconic of them honestly 💅

🎀 DESCRIPTION 🎀

Okay so this is a BIG one, so buckle up, grab a strawberry smoothie 🍓, and let's go. This page is split into two chunks of params: the sampling ones (aka how the model picks its next word) and the example-specific ones (aka how the actual binary runs). Part 1, here we gooo 💖

🌈 ----- sampling params ----- 🌈

  • --samplers SAMPLERS — the samplers that are gonna be used for generation, in the order you list them, separated by ; (default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature)
  • -s, --seed SEED — RNG seed (default: -1, and -1 means "use a random seed", so surprise me ✨)
  • --sampler-seq, --sampling-seq SEQUENCE — the simplified sequence for the samplers that get used (default: edskypmxt)
  • --ignore-eos — ignore the end of stream token and just keep generating, bestie doesn't know when to stop 😭 (this implies --logit-bias EOS-inf)
  • --temp, --temperature N — temperature (default: 0.80)
  • --top-k N — top-k sampling (default: 40, 0 = disabled) (env: LLAMA_ARG_TOP_K)
  • --top-p N — top-p sampling (default: 0.95, 1.0 = disabled)
  • --min-p N — min-p sampling (default: 0.05, 0.0 = disabled)
  • --top-nsigma, --top-n-sigma N — top-n-sigma sampling (default: -1.00, -1.0 = disabled)
  • --xtc-probability N — xtc probability (default: 0.00, 0.0 = disabled)
  • --xtc-threshold N — xtc threshold (default: 0.10, 1.0 = disabled)
  • --typical, --typical-p N — locally typical sampling, parameter p (default: 1.00, 1.0 = disabled)
  • --repeat-last-n N — the last n tokens to consider for penalizing (default: 64, 0 = disabled)
  • --repeat-penalty N — penalize that repeat sequence of tokens, we are NOT doing the same sentence twice 🚫 (default: 1.00, 1.0 = disabled)
  • --presence-penalty N — repeat alpha presence penalty (default: 0.00, 0.0 = disabled)
  • --frequency-penalty N — repeat alpha frequency penalty (default: 0.00, 0.0 = disabled)
  • --dry-multiplier N — set the DRY sampling multiplier (default: 0.00, 0.0 = disabled)
  • --dry-base N — set the DRY sampling base value (default: 1.75)
  • --dry-allowed-length N — set the allowed length for DRY sampling (default: 2)
  • --dry-penalty-last-n N — set the DRY penalty for the last n tokens (default: 64, 0 = disable)
  • --dry-sequence-breaker STRING — add a sequence breaker for DRY sampling, and heads up, this clears out the default breakers ('\n', ':', '"', '*') in the process; use "none" if you don't wanna use any sequence breakers at all 🙈
  • --adaptive-target N — adaptive-p: pick tokens near this probability (valid range 0.0 to 1.0; negative = disabled) (default: -1.00) — more info lives at https://github.com/ggml-org/llama.cpp/pull/17927
  • --adaptive-decay N — adaptive-p: the decay rate for target adaptation over time. lower values = more reactive, higher values = more stable. (valid range 0.0 to 0.99) (default: 0.90)
  • --dynatemp-range N — dynamic temperature range (default: 0.00, 0.0 = disabled)
  • --dynatemp-exp N — dynamic temperature exponent (default: 1.00)
  • --mirostat N — use Mirostat sampling. ⚠️ Top K, Nucleus and Locally Typical samplers get ignored if you use this, so pick your fighter (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0)
  • --mirostat-lr N — Mirostat learning rate, parameter eta (default: 0.10)
  • --mirostat-ent N — Mirostat target entropy, parameter tau (default: 5.00)
  • -l, --logit-bias TOKEN_ID(+/-)BIAS — modifies the likelihood of a token appearing in the completion, like, say, --logit-bias 15043+1 to increase the likelihood of the token ' Hello', or --logit-bias 15043-1 to decrease the likelihood of the token ' Hello' 🔥
  • --grammar GRAMMAR — BNF-like grammar to constrain generations (peep the samples over in the grammars/ dir)
  • --grammar-file FNAME — file to read the grammar from
  • -j, --json-schema SCHEMA — JSON schema to constrain generations (https://json-schema.org/), e.g. {} for any JSON object. For schemas w/ external $refs, use --grammar + example/json_schema_to_grammar.py instead
  • -jf, --json-schema-file FILE — File containing a JSON schema to constrain generations (https://json-schema.org/), e.g. {} for any JSON object. For schemas w/ external $refs, use --grammar + example/json_schema_to_grammar.py instead
  • -bs, --backend-sampling — enable backend sampling (experimental) (default: disabled) (env: LLAMA_ARG_BACKEND_SAMPLING)

🦋 ----- example-specific params ----- 🦋

  • -h, --help, --usage — print usage and dip ✌️
  • --version — show version and build info
  • -cl, --cache-list — show list of models in the cache
  • --completion-bash — print a source-able bash completion script for llama.cpp (so slay of it)
  • -t, --threads N — number of CPU threads to use during generation (default: <num_cpus>) (env: LLAMA_ARG_THREADS)
  • -tb, --threads-batch N — number of threads to use during batch and prompt processing (default: same as --threads)
  • -C, --cpu-mask M — CPU affinity mask: arbitrarily long hex. Complements cpu-range (default: "")
  • -Cr, --cpu-range lo-hi — range of CPUs for affinity. Complements --cpu-mask
  • --cpu-strict <0|1> — use strict CPU placement (default: 0)
  • --prio N — set process/thread priority: low(-1), normal(0), medium(1), high(2), realtime(3) (default: 0)
  • --poll <0...100> — use polling level to wait for work (0 - no polling, default: 50)
  • -Cb, --cpu-mask-batch M — CPU affinity mask: arbitrarily long hex. Complements cpu-range-batch (default: same as --cpu-mask)
  • -Crb, --cpu-range-batch lo-hi — ranges of CPUs for affinity. Complements --cpu-mask-batch
  • --cpu-strict-batch <0|1> — use strict CPU placement (default: same as --cpu-strict)
  • --prio-batch N — set process/thread priority: 0-normal, 1-medium, 2-high, 3-realtime (default: 0)
  • --poll-batch <0|1> — use polling to wait for work (default: same as --poll)
  • -c, --ctx-size N — size of the prompt context (default: 0, 0 = loaded from model) (env: LLAMA_ARG_CTX_SIZE)
  • -n, --predict, --n-predict N — number of tokens to predict (default: -1, -1 = infinity, so it's gonna yap forever 💀) (env: LLAMA_ARG_N_PREDICT)
  • -b, --batch-size N — logical maximum batch size (default: 2048) (env: LLAMA_ARG_BATCH)
  • -ub, --ubatch-size N — physical maximum batch size (default: 512) (env: LLAMA_ARG_UBATCH)
  • --keep N — number of tokens to keep from the initial prompt (default: 0, -1 = all)
  • --swa-full — use full-size SWA cache (default: false) — more info at https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055 (env: LLAMA_ARG_SWA_FULL)
  • -fa, --flash-attn [on|off|auto] — set Flash Attention use ('on', 'off', or 'auto', default: 'auto') (env: LLAMA_ARG_FLASH_ATTN)
  • -p, --prompt PROMPT — prompt to start generation with; for a system message, use -sys
  • --perf, --no-perf — whether to enable internal libllama performance timings (default: false) (env: LLAMA_ARG_PERF)
  • -f, --file FNAME — a file containing the prompt (default: none)
  • -bf, --binary-file FNAME — binary file containing the prompt (default: none)
  • -e, --escape, --no-escape — whether to process escapes sequences (\n, \r, \t, \', \", \\) (default: true)
  • --rope-scaling {none,linear,yarn} — RoPE frequency scaling method, defaults to linear unless the model itself says otherwise (env: LLAMA_ARG_ROPE_SCALING_TYPE)
  • --rope-scale N — RoPE context scaling factor, expands the context by a factor of N (env: LLAMA_ARG_ROPE_SCALE)
  • --rope-freq-base N — RoPE base frequency, used by NTK-aware scaling (default: loaded from model) (env: LLAMA_ARG_ROPE_FREQ_BASE)
  • --rope-freq-scale N — RoPE frequency scaling factor, expands context by a factor of 1/N (env: LLAMA_ARG_ROPE_FREQ_SCALE)
  • --yarn-orig-ctx N — YaRN: original context size of the model (default: 0 = model training context size) (env: LLAMA_ARG_YARN_ORIG_CTX)
  • --yarn-ext-factor N — YaRN: extrapolation mix factor (default: -1.00, 0.0 = full interpolation) (env: LLAMA_ARG_YARN_EXT_FACTOR)
  • --yarn-attn-factor N — YaRN: scale sqrt(t) or attention magnitude (default: -1.00) (env: LLAMA_ARG_YARN_ATTN_FACTOR)
  • --yarn-beta-slow N — YaRN: high correction dim or alpha (default: -1.00) (env: LLAMA_ARG_YARN_BETA_SLOW)
  • --yarn-beta-fast N — YaRN: low correction dim or beta (default: -1.00) (env: LLAMA_ARG_YARN_BETA_FAST)
  • -kvo, --kv-offload, -nkvo, --no-kv-offload — whether to enable KV cache offloading (default: enabled) (env: LLAMA_ARG_KV_OFFLOAD)
  • --repack, -nr, --no-repack — whether to enable weight repacking (default: enabled) (env: LLAMA_ARG_REPACK)
  • --no-host — bypass host buffer allowing extra buffers to be used (env: LLAMA_ARG_NO_HOST)
  • -ctk, --cache-type-k TYPE — KV cache data type for K
    • allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_K)
  • -ctv, --cache-type-v TYPE — KV cache data type for V
    • allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_V)
  • -dt, --defrag-thold N — KV cache defragmentation threshold (DEPRECATED) (env: LLAMA_ARG_DEFRAG_THOLD)
  • -np, --parallel N — number of parallel sequences to decode (default: 1) (env: LLAMA_ARG_N_PARALLEL)
  • --mlock — DEPRECATED in favor of --load-mode: force the system to keep the model in RAM rather than swapping or compressing (env: LLAMA_ARG_MLOCK)
  • --mmap, --no-mmap — DEPRECATED in favor of --load-mode: whether to memory-map the model. (if mmap is disabled, slower load but it may reduce pageouts if you're not using mlock) (env: LLAMA_ARG_MMAP)
  • -dio, --direct-io, -ndio, --no-direct-io — DEPRECATED in favor of --load-mode: use DirectIO if available (env: LLAMA_ARG_DIO)
  • -lm, --load-mode MODE — model loading mode (default: auto)
⚠️ bestie, see all those DEPRECATED tags? Yeah. They're old news. The glow-up replacement is --load-mode, so maybe don't build your whole personality around --mlock, --mmap, -dio or -dt 🫶
  • auto — mmap, unless a device doesn't support it
  • none — no special loading mode 💤
  • mmap — memory-map model (if mmap is disabled, you get a slower load, but it may reduce pageouts if you're not using mlock)
  • mlock — force system to keep model in RAM rather than swapping or compressing 🔒
  • mmap+mlock — mmap + force system to keep model in RAM rather than swapping or compressing
  • dio — use DirectIO if available

and the env var for that whole vibe is LLAMA_ARG_LOAD_MODE 🫶

  • --numa TYPE — attempt optimizations that help on some NUMA systems
    • distribute: spread execution evenly over all nodes
    • isolate: only spawn threads on CPUs on the node that execution started on
    • numactl: use the CPU map provided by numactl

ngl, if you've been running without this before, it is recommended that you drop the system page cache before you start using it. see https://github.com/ggml-org/llama.cpp/issues/1437 (env: LLAMA_ARG_NUMA)

  • -dev, --device <dev1,dev2,..> — comma-separated list of devices to use for offloading (none = don't offload). use --list-devices to see a list of available devices (env: LLAMA_ARG_DEVICE) 🎮
  • --list-devices — print list of available devices and exit. that's it, that's the whole job, queen shit 👑
  • -ot, --override-tensor <tensor name pattern>=<buffer type>,... — override tensor buffer type (env: LLAMA_ARG_OVERRIDE_TENSOR)
  • -cmoe, --cpu-moe — keep all Mixture of Experts (MoE) weights in the CPU (env: LLAMA_ARG_CPU_MOE)
  • -ncmoe, --n-cpu-moe N — keep the Mixture of Experts (MoE) weights of the first N layers in the CPU (env: LLAMA_ARG_N_CPU_MOE)
  • -ngl, --gpu-layers, --n-gpu-layers N — max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS) 🔥
  • -sm, --split-mode {none,layer,row,tensor} — how to split the model across multiple GPUs, one of:
    • none: use one GPU only
    • layer (default): split layers and KV across GPUs (pipelined)
    • row: split weight across GPUs by rows (parallelized)
    • tensor: split weights and KV across GPUs (parallelized, EXPERIMENTAL)

(env: LLAMA_ARG_SPLIT_MODE)

  • -ts, --tensor-split N0,N1,N2,... — fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1 (env: LLAMA_ARG_TENSOR_SPLIT)
  • -mg, --main-gpu INDEX — the GPU to use for the model (with split-mode = none), or for intermediate results and KV (with split-mode = row) (default: 0) (env: LLAMA_ARG_MAIN_GPU)
  • -fit, --fit [on|off] — whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on') (env: LLAMA_ARG_FIT)
  • -fitt, --fit-target MiB0,MiB1,MiB2,... — target margin per device for --fit, comma-separated list of values, a single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET)
  • -fitc, --fit-ctx N — minimum ctx size that can be set by the --fit option, default: 4096 (env: LLAMA_ARG_FIT_CTX)
  • --check-tensors — check model tensor data for invalid values (default: false) 🕵️
  • --override-kv KEY=TYPE:VALUE,... — advanced option to override model metadata by key. to specify multiple overrides, either use comma-separated values. types: int, float, bool, str. example: --override-kv tokenizer.ggml.add_bos_token=bool:false,tokenizer.ggml.add_eos_token=bool:false
  • --op-offload, --no-op-offload — whether to offload host tensor operations to device (default: true)
  • --lora FNAME — path to LoRA adapter (use comma-separated values to load multiple adapters)
  • --lora-scaled FNAME:SCALE,... — path to LoRA adapter with user defined scaling (format: FNAME:SCALE,...) note: use comma-separated values
  • --control-vector FNAME — add a control vector. note: use comma-separated values to add multiple control vectors
  • --control-vector-scaled FNAME:SCALE,... — add a control vector with user defined scaling SCALE. note: use comma-separated values (format: FNAME:SCALE,...)
  • --control-vector-layer-range START END — layer range to apply the control vector(s) to, start and end inclusive
  • -m, --model FNAME — model path to load (env: LLAMA_ARG_MODEL) 💖
  • -mu, --model-url MODEL_URL — model download url (default: unused) (env: LLAMA_ARG_MODEL_URL)
  • -dr, --docker-repo [<repo>/]<model>[:quant] — Docker Hub model repository. repo is optional, defaults to ai/. quant is optional, defaults to :latest. example: gemma3 (default: unused) (env: LLAMA_ARG_DOCKER_REPO)
  • -hf, -hfr, --hf-repo <user>/<model>[:quant] — Hugging Face model repository; quant is optional, case-insensitive, defaults to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist. mmproj is also downloaded automatically if available. to disable, add --no-mmproj. example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M (default: unused) (env: LLAMA_ARG_HF_REPO)
  • -hff, --hf-file FILE — Hugging Face model file. If specified, it will override the quant in --hf-repo (default: unused) (env: LLAMA_ARG_HF_FILE)
  • -hft, --hf-token TOKEN — Hugging Face access token (default: value from HF_TOKEN environment variable) (env: HF_TOKEN) 🔑
  • --log-disable — Log disable
  • --log-file FNAME — Log to file (env: LLAMA_ARG_LOG_FILE)
  • --log-colors [on|off|auto] — Set colored logging ('on', 'off', or 'auto', default: 'auto'). 'auto' enables colors when output is to a terminal (env: LLAMA_ARG_LOG_COLORS) 🌈
  • -v, --verbose, --log-verbose — Set verbosity level to infinity (i.e. log all messages, useful for debugging)
  • --offline — Offline mode: forces use of cache, prevents network access (env: LLAMA_ARG_OFFLINE)
  • -lv, --verbosity, --log-verbosity N — Set the verbosity threshold. Messages with a higher verbosity will be ignored. Values:
    • 0: generic output
    • 1: error
    • 2: warning
    • 3: info
    • 4: trace (more info)
    • 5: debug

(default: 3) (env: LLAMA_ARG_LOG_VERBOSITY)

  • --log-prefix, --no-log-prefix — Enable prefix in log messages (env: LLAMA_ARG_LOG_PREFIX)
  • --log-timestamps, --no-log-timestamps — Enable timestamps in log messages (env: LLAMA_ARG_LOG_TIMESTAMPS)
  • --spec-draft-type-k, -ctkd, --cache-type-k-draft TYPE — KV cache data type for K for the draft model. allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K)
  • --spec-draft-type-v, -ctvd, --cache-type-v-draft TYPE — KV cache data type for V for the draft model. allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V)

example usage:

llama-idle -m model.gguf [-ngl n_gpu_layers]

okay bestie that's the whole flag buffet, go forth and offload responsibly 💅

and that's a wrap!! 🌸✨ this page came from the debian source, it's the September 2026 edition, and its doc reference is LLAMA-IDLE(1) — stan the manual, love the manual, man the manual 🎀💖🦋

📦 where this page comes from