SLMBUILD(1)

💖 NAME 💖

okay so bestie, this is SLMBUILD(1), from the User Contributed Perl Documentation 💅 and it’s literally the page that teaches you how to build a language model from an idngram file. slmbuild generates a back-off smoothing language model from a given idngram_file. Generally, that idngram_file is created by ids2ngram. periodt ✨

🦋 SYNOPSIS 🦋

slmbuild [option]... idngram_file...

🌸 DESCRIPTION 🌸

slmbuild generates a back-off smoothing language model from a given idngram file. Generally, the idngram_file is created by ids2ngram. that’s the whole vibe 🫶

💅 OPTIONS 💅

bestie, all the following options are mandatory. no cap, you need them. 💀

• -n,--NMax N — 1 for unigram, 2 for bigram, 3 for trigram. any number not in the range 1..3 is not valid.
• -o, --out output-file — specify the output xfilei name.
• -l, --log — using -log(pr); by default, pr is used directly.
• -w, --wordcount N — Lexican size, number of different words.
• -b, --brk id... — set the ids which should be treated as breaker.
• -e, --e id... — set the ids which should not be put into LM.
• -c, --cut c... — k-grams whose freq <= c[k] are dropped.
• -d, --discount method, param... — the k-th -d parm specifies the discount method.

for a k-gram, the possible method/param values are:

• GT,R,dis : GT discount for r <= R; r is the freq of an ngram. Linear discount for those r > R, i.e. r'=r*dis. 0 << dis < 1.0, for example 0.999.
• ABS,[dis] : Absolute discount r'=r-dis. And dis is optional. 0 << dis < cut[k]+1.0, normally dis < 1.0.
• LIN,[dis] : Linear discount r'=r*dis. And dis is optional. 0 < dis < 1.0.

🔥 NOTE 🔥

-n must be given before -c -b. And -c must give right number of cut-off, also -ds must appear exactly N times specifying the discounts for 1-gram, 2-gram..., respectively. 👑

BREAKER-IDs could be SentenceTokens or ParagraphTokens. Conceptually, these ids have no meaning when they appeared in the middle of n-gram.

EXCLUDE-IDs could be ambiguious-ids. Conceptually, n-grams which contain those ids are meaningless.

We can not erase ngrams according to BREAKER-IDS and EXCLUDE-IDs directly from IDNGRAM file, because some low-level information is still useful in it. so yeah, don’t try to skip that step 😭

🌈 EXAMPLE 🌈

the following example reads all.id3gram and writes trigram model all.slm.

At 1-gram level, use Good-Turing discount with cut-off 0, R=8, dis=0.9995. At 2-gram level, use Absolute discount with cut-off 3, dis auto-calc. At 3-gram level, use Absolute discount with cut-off 2, dis auto-calc. Word id 10,11,12 are breakers (sentence/para/paper breaker, etc). Exclude-ID is 9. Lexicon contains 200000 words. The result languagme model uses -log(pr).

slmbuild -l -n 3 -o all.slm -w 200000 -c 0,3,2 -d GT,8,0.9995 -d ABS -d ABS -b 10,11,12 -e 9 all.id3gram

🎀 AUTHOR 🎀

Originally written by Phill.Zhang <phill.zhang@sun.com>. Currently maintained by Kov.Chai <tchaikov@gmail.com>.

🦋 SEE ALSO 🦋

ids2ngram(1), slmprune(1).

---

and that’s the page, bestie 💖 it’s from perl v5.36.0, dated 2023-08-08 — iconic source/version energy, no notes ✨