FINALFRONTIER-DEPS(1)

📦 version 2022-01, in nixos 26.05, nixpkgs unstable

🦋 finalfrontier-deps — the manual, but make it make sense 🦋

okay so this is FINALFRONTIER-DEPS(1) — the dep-model girlies' manual page. grab a strawberry drink 🍓 and let's go.

✨ NAME ✨

finalfrontier deps — trains dependency-based word embeddings with subword representations. yes bestie, that's the whole vibe. 💅

🎀 SYNOPSIS 🎀

finalfrontier deps [options] corpus output

💖 DESCRIPTION 💖

so what this subcommand does: it trains dependency based word embeddings (that's Levy and Goldberg, 2014 — iconic paper, iykyk 🔥) using data from a corpus that's in CONLL-U format.

your corpus has sentences separated by empty lines. and lowkey the important part: every single sentence needs to be annotated with a dependency graph. no graph, no training, that's the deal 🙈.

after training finishes, the embeddings get written to output in the finalfusion format. periodt.

🌸 OPTIONS 🌸

okay this is the long part, but honestly every one of these slaps in its own way. read them all, don't skip 👑

  • --buckets EXP — the bucket exponent. finalfrontier uses 2^EXP buckets to store subword representations. each subword representation (n-gram) gets hashed, and that hash decides which bucket it lands in. more buckets = fewer bucket collisions between subword representations, but at the cost of memory use. default bucket exponent is 21 (that's approximately 2 million buckets, bestie 😭).
  • --context-discard THRESHOLD — the context discard threshold. it influences how often frequent contexts get discarded during training. default is 1e-4.
  • --context-mincount FREQ — the minimum count, which controls discarding of infrequent contexts. contexts occurring fewer than FREQ times are simply not considered during training. default minimum count is 5.
  • --context-target-size SIZE — the target size for the context vocabulary. at most SIZE contexts are included for training. only contexts appearing more frequently than the context at SIZE are included. 💫
  • --dependency-depth DEPTH — dependency contexts up to DEPTH distance from the focus word in the dependency graph will be used to learn the representation of the focus word. default depth is 1.
  • --dims DIMS — the dimensionality of the trained word embeddings. default dimensionality is 300.
  • --discard THRESHOLD — the discard threshold, which influences how often frequent focus words get discarded from training. default is 1e-4.
  • --epochs N — the number of training epochs. the number of necessary training epochs typically decreases as the corpus size goes up (big corpus energy 🌈). default number of epochs is 15.
  • --hash-indexer INDEXER — the indexer to use when bucket-based subwords are used (peep --subwords). possible values are finalfusion or fasttext. default: finalfusion.
    • finalfusion uses the FNV-1a hasher, whereas fasttext emulates the (broken) implementation of FNV-1a in fastText. use of finalfusion is recommended, unless the resulting embeddings should be compatible with fastText.
  • -f, --format FORMAT — the output format. must be one of fasttext, finalfusion, word2vec, text, and textdims.
    • literally all formats except finalfusion result in a loss of information: word2vec, text, and textdims do not store subword embeddings, nor hyperparameters. the fastText format does not store all hyperparameters. 💀
    • the fasttext format can only be used in conjunction with --subwords buckets and --hash-indexer fasttext.
  • --lr LEARNING_RATE — the learning rate determines what fraction of a gradient is used for parameter updates. default initial learning rate is 0.05, and the learning rate decreases monotonically during training. it's giving "steady decline, but make it math" 📉
  • --maxn LEN — the maximum n-gram length for subword representations. default: 6
  • --mincount FREQ — the minimum count controls discarding of infrequent focus words. focus words occurring fewer than FREQ times are not considered during training. default minimum count is 5.
  • --minn LEN — the minimum n-gram length for subword representations. default: 3
  • --ngram-mincount FREQ — the minimum n-gram frequency. n-grams occurring fewer than FREQ times are excluded from training. this option is only applicable with the ngrams argument of the subwords option. just a lil heads up 🫶
  • --ngram-target-size SIZE — the target size for the n-gram vocabulary. at most SIZE n-grams are included for training. only n-grams appearing more frequently than the n-gram at SIZE are included. this option is only applicable with the ngrams argument of the subwords option.
  • --normalize-contexts — normalize the attached form in the dependency contexts. ✨
  • --ns FREQ — the number of negatives to sample per positive example. default: 5
  • --projectivize — projectivize dependency graphs before training embeddings. do it before, not after, bestie.
  • --threads N — the number of threads to use during training for parallelization. the default is to use half of the logical CPUs of the machine, capped at 20 threads. increasing the number of threads increases the probability of update collisions, which means you'll need more epochs to reach the same loss 😭 (parallelism has a price, tragic).
  • --subwords SUBWORDS — the type of subword embeddings to train. possible types are buckets, ngrams, and none. subword embeddings are used to compute embeddings for unknown words by summing embeddings of n-grams within unknown words. 🧚
    • the none type does not use subwords. the resulting model will not be able to assign an embedding to unknown words. (she simply doesn't know them, and that's that.)
    • the ngrams type stores subword n-grams explicitly. the included n-gram lengths are specified using the minn and maxn options. the frequency threshold for n-grams is configured with the ngram-mincount option.
    • the buckets type maps n-grams to buckets using the FNV1 hash. the considered n-gram lengths are specified using the minn and maxn options. the number of buckets is controlled with the buckets option.
  • --target-size SIZE — the target size for the token vocabulary. at most SIZE tokens are included for training. only tokens appearing more frequently than the token at SIZE are included.
  • --untyped-deps — only use the word of the attached token in the dependency relation as contexts to train the representation of the focus word. lowkey minimal, highkey useful 💅
  • --use-root — include the abstract root node in the dependency graph as contexts during training. root node gets to sit with us now 🌳
  • --zipf EXP — exponent s used in the Zipf distribution p(k) = 1 / (k^s H_N) for negative sampling. default: 0.5

🌈 EXAMPLES 🌈

train embeddings on dewiki.txt using the dependency model with default parameters — chef's kiss, no notes:

finalfrontier deps dewiki.conll dewiki-deps.bin

now train embeddings with dimensionality 200 on dewiki.conll using the dependency model from contexts with depth up to 2:

finalfrontier deps --depth 2 --normalize --dims 200 \
  dewiki.conll dewiki-deps.bin
ngl those examples are written exactly like the original says — copy them character for character, don't go "fixing" the flag names, bestie 🫶

👑 SEE ALSO 👑

finalfrontier(1), finalfrontier-skipgram(1) — go read those too, they're the other main characters.

and that's the whole page! this iconic piece of documentation is version 3, written by Daniel de Kok and Sebastian Pütz, dated Apr 6, 2019. absolute legends, we stan 💜🎀✨

📦 where this page comes from