NGRAMSHRINK(1)

📦
version 1.3.17 · 3 versions
, in fedora 44

✨ NGRAMSHRINK ✨

okay bestie, gather round, because this is the manual page for ngramshrink — a User Commands legend, version 1.3.17, and it is lowkey doing the most for your language models 💅

💖 NAME 💖

ngramshrink — the manual page for ngramshrink 1.3.17, no cap.

🦋 DESCRIPTION 🦋

She shrinks your n-gram model. You hand her an input model file, she hands you back a smaller one. That's the whole vibe. Iconic, honestly.

Here's how you call her:

Usage: ngramshrink [--options] [in.fst [out.fst]]

So like: some options, then optionally in.fst, then optionally out.fst. Simple as. 🌸

👑 PROGRAM FLAGS 👑

These are the flags that are hers, the main character ones:

  • --backoff_label — type = int64_t, default = 0. The backoff label. That's it, that's the whole description, she said what she said.
  • --check_consistency — type = bool, default = false. Checks whether the model is consistent. Vibe check for your model, fr fr.
  • --context_pattern — type = std::string, default = "". The pattern of contexts to prune.
  • --count_pattern — type = std::string, default = "". The pattern of counts to prune.
  • --list_file — type = std::string, default = "". A file holding the list of n-grams to prune. It's giving "I made a hit list and I'm sticking to it" 📋
  • --method — type = std::string, default = "seymore". Pick your fighter — one of: "context_prune", "count_prune", "relative_entropy", "seymore", "list_prune". Default is "seymore" because she's that girl.
  • --min_order_to_prune — type = int32_t, default = 2. The minimum n-gram order that's allowed to get pruned. Don't come for the little ones, bestie.
  • --norm_eps — type = double, default = 0.001. The epsilon used for the normalization check. Tiny little tolerance, so precise ✨
  • --retry_downcase — type = bool, default = false. If a pruned symbol isn't found in the FST, she automatically tries the lower-cased variant of that symbol. ⚠️ Heads up: this is only useful in list_prune mode. Don't be out here using it elsewhere and expecting magic.
  • --shrink_opt — type = int32_t, default = 0. Optimization level, and it's a range: 0 (fastest) all the way to 2 (most accurate). Pick your speed/accuracy tradeoff, queen.
  • --target_number_of_ngrams — type = int64_t, default = -1. The maximum number of ngrams you want left in the model after pruning. And listen: a value less than zero means there's no target number at all — she just uses theta. 🎯
  • --theta — type = double, default = 0. The pruning threshold theta. The sacred number.
  • --total_unigram_count — type = double, default = -1. The total unigram count.

🧚 LIBRARY FLAGS 🧚

Now the library side of the family. These come in little friend groups (by source file), and I'm presenting them exactly as they come, no skips.

🍓 Flags from: flags.cc

  • --help — type = bool, default = false. Shows usage information. Your bestie when you're lost 😭
  • --helpshort — type = bool, default = false. Shows brief usage information. The tl;dr version.
  • --tmpdir — type = std::string, default = "/tmp". The temporary directory.
  • --v — type = int32_t, default = 0. Verbosity level. Crank it up if you want her to talk more 💬

💜 Flags from: fst.cc

  • --fst_align — type = bool, default = false. Writes FST data aligned where appropriate. Alignment matters, periodt.
  • --fst_default_cache_gc — type = bool, default = true. Enables garbage collection of the cache. Taking out the trash 🗑️✨
  • --fst_default_cache_gc_limit — type = int64_t, default = 1048576. The cache byte size that triggers garbage collection. Hit that number and GC wakes up.
  • --fst_read_mode — type = std::string, default = "read". The default file reading mode for mappable files.
  • --fst_verify_properties — type = bool, default = false. Verifies FST properties queried by TestProperties.
  • --save_relabel_ipairs — type = std::string, default = "". Saves the input relabel pairs to a file.
  • --save_relabel_opairs — type = std::string, default = "". Saves the output relabel pairs to a file.

🌈 Flags from: ngram-output.cc

  • --end_symbol — type = std::string, default = "</s>". The class label for sentence end.
  • --start_symbol — type = std::string, default = "<s>". The class label for sentence start. Matching set, we love symmetry 🫶

🎀 Flags from: symbol-table.cc

  • --fst_compat_symbols — type = bool, default = true. Requires symbol tables to match when appropriate. No mismatched tables on her watch.
  • --fst_field_separator — type = std::string, default = " ". The set of characters used as a separator between printed fields. (Yes bestie, that default is a whole run of spaces. It's literally just spaces. Iconic.)

💀 Flags from: util.cc

  • --fst_error_fatal — type = bool, default = true. FST errors are fatal; otherwise she returns objects flagged as bad: e.g., FSTs get the kError property set, and FST weights are not a Member().
  • --ngram_error_fatal — type = bool, default = true. NGram errors are fatal if true; otherwise she returns objects flagged as bad: e.g., NGramModel::Error() is true.

⭐ Flags from: weight.cc

  • --fst_weight_parentheses — type = std::string, default = "". Characters enclosing the first weight of a printed composite weight (e.g., pair weight, tuple weight and derived classes), so that nested composite weights do proper I/O. ⚠️ It must have size 0 (meaning none) or 2 (meaning open and close parenthesis). No in-between, she's strict about it.
  • --fst_weight_separator — type = std::string, default = ",". The character separator between printed composite weights. ⚠️ And it must be a single character. One. Not two.
🫶 Reminder bestie: this page is literally just the manual for ngramshrink 1.3.17 from the User Commands section. Everything above is the whole tea. No more, no less, no invented lore. 💅

and that's a wrap!! 🎀 this little masterpiece shipped as ngramshrink 1.3.17 back in February 2022, living in the NGRAMSHRINK(1) section of the manual — go shrink some n-grams, you absolute legend 👑✨

📦 where this page comes from