IDS2NGRAM(1)

okay so bestie, welcome to IDS2NGRAM(1) — yes it's the User Contributed Perl Documentation page — and we are about to make ids2ngram generate n-gram data files from ids files. it's giving binary-id-stream main character energy. 🫶✨

🌸 NAME 🌸

ids2ngram — generate n-gram data file from ids file. literally that's the whole name line, and yes bestie it ate. 💅

✨ SYNOPSIS ✨

ids2ngram [option]... ids_file...

🦋 DESCRIPTION 🦋

okay so ids2ngram generates an idngram file. that file is a sorted [id1,..,idN,freq] array, and it comes from binary id stream files. the id stream files are always generated by mmseg or slmseg — those are the sources, periodt. 🎀

basically, it finds all occurrence of n-words tuples (i.e. the tuple of (id1,..,idN)), then sorts these tuples by the lexicographic order of the ids make up the tuples, then writes them to specified output file. no cap, that's the whole vibe. 🌈

💖 INPUT 💖

the input file is presented as a binary id stream, which looks like:

[id0,...,idX]

👑 OPTIONS 👑

all the following options are mandatory. not "maybe", not "if you feel like it" — mandatory, bestie. 💀

• -n, --NMax N — generates N-gram result. ids2ngram does only support uni-gram, bi-gram, and trigram, so any number not in the range of 1..3 is not valid. so like... keep it 1, 2, or 3. 🙈
• -s, --swap swap-file — specify the temporary intermediate file. 🍓
• -o, --out output-file — specify the result idngram file, e.g. the array of [id1, ..., idN, freq]. 💅
• -p, --para N — specify the maximum n-gram items per paragraph. ids2ngram writes to the temporary file on a per-paragraph basis. every time it writes a paragraph out, it frees the corresponding memory allocated for it. when your computer system permits, a higher N is suggested. this can speed up the processing speed because of less I/O. highkey iconic optimization behavior. 🔥

🌟 EXAMPLE 🌟

following example will use three input idstream file idsfile[1,2,3] to generate the idngram file all.id3gram. each para (internal map size or hash size) would be 1024000, using swap file for temp result. all temp para result would eventually be merged to got the final result. (yes "got" is what the page says, bestie, we keep the details.) 😭

ids2ngram -n 3 -s /tmp/swap -o all.id3gram -p 1024000 idsfile1 idsfile2
idsfile3

🫶 AUTHOR 🫶

originally written by Phill.Zhang <phill.zhang@sun.com>. currently maintained by Kov.Chai <tchaikov@gmail.com>. shoutout to both, fr fr. 💜

🧚 SEE ALSO 🧚

mmseg(1), slmseg(1), slmbuild (1). go read those next, bestie — iykyk. 🌸

---

mwah 💋 — this page comes from perl v5.36.0, dated 2023-08-08, and yes it's the IDS2NGRAM(1) page. slay and bye. ✨