FINALFRONTIER-SKIPGRAM(1)

okay so bestie, you found finalfrontier-skipgram(1) and we’re gonna make it make sense 💅✨ this is the finalfrontier skipgram subcommand — the one that trains word embeddings with subword representations — and I promise it’s giving legendary, not scary.

🎀 NAME 🎀

finalfrontier skipgram — train word embeddings with subword representations 🦋

✨ SYNOPSIS ✨

finalfrontier skipgram [options] corpus output

🌸 DESCRIPTION 🌸

The finalfrontier skipgram subcommand trains word embeddings using data from a corpus. The corpus should have tokens separated by spaces and sentences separated by newlines. After training, the embeddings are written to output in the finalfusion format. slay. 💖

💅 OPTIONS 💅

• --buckets EXP — the bucket exponent. finalfrontier uses 2^EXP buckets to store subword representations. Each subword representation (n-gram) is hashed and mapped to a bucket based on this hash. using more buckets means fewer bucket collisions between subword representations, at the cost of memory use. the default bucket exponent is 21 (approximately 2 million buckets). ✨
• --context CONTEXT_SIZE — words within the CONTEXT_SIZE of a focus word will be used to learn the representation of the focus word. the default context size is 10.
• --dims DIMENSIONS — the dimensionality of the trained word embeddings. the default dimensionality is 300.
• --discard THRESHOLD — the discard threshold influences how often frequent words are discarded from training. the default discard threshold is 1e-4.
• --epochs N — the number of training epochs. the number of necessary training epochs typically decreases with the corpus size. the default number of epochs is 15.
• -f, --format FORMAT — the output format. this must be one of fasttext, finalfusion, word2vec, text, and textdims.
  • all formats, except finalfusion, result in a loss of information: word2vec, text, and textdims do not store subword embeddings, nor hyperparameters. the fastText format does not store all hyperparameters.
  • the fasttext format can only be used in conjunction with --subwords buckets and --hash-indexer fasttext.
• --hash-indexer INDEXER — the indexer to use when bucket-based subwords are used (see --subwords). the possible values are finalfusion or fasttext. default: finalfusion.
  • finalfusion uses the FNV-1a hasher, whereas fasttext emulates the (broken) implementation of FNV-1a in fastText. use of finalfusion is recommended, unless the resulting embeddings should be compatible with fastText. 💅
• --lr LEARNING_RATE — the learning rate determines what fraction of a gradient is used for parameter updates. the default initial learning rate is 0.05; the learning rate decreases monotonically during training.
• --maxn LEN — the maximum n-gram length for subword representations. default: 6.
• --mincount FREQ — the minimum count controls discarding of infrequent words. words occurring fewer than FREQ times are not considered during training. the default minimum count is 5.
• --minn LEN — the minimum n-gram length for subword representations. default: 3.
• --model MODEL — the model to use for training word embeddings. the choices here are: dirgram for the directional skip-gram model (Song et al., 2018), skipgram for the skip-gram model (Mikolov et al., 2013), and structgram for the structured skip-gram model (Ling et al. 2015).
  • the structured skip-gram model takes the position of a context word into account and results in embeddings that are typically better suited for syntax-oriented tasks. 👑
  • the dependency embeddings model is supported by the separate finalfrontier deps(1) subcommand.
  • the default model is skipgram.
• --ngram-mincount FREQ — the minimum n-gram frequency. n-grams occurring fewer than FREQ times are excluded from training. this option is only applicable with the ngrams argument of the subwords option.
• --ngram-target-size SIZE — the target size for the n-gram vocabulary. at most SIZE n-grams are included for training. only n-grams appearing more frequently than the n-gram at SIZE are included. this option is only applicable with the ngrams argument of the subwords option.
• --ns FREQ — the number of negatives to sample per positive example. default: 5.
• --subwords SUBWORDS — the type of subword embeddings to train. the possible types are buckets, ngrams, and none. subword embeddings are used to compute embeddings for unknown words by summing embeddings of n-grams within unknown words. 🦋
  • the none type does not use subwords. the resulting model will not be able to assign an embedding to unknown words.
  • the ngrams type stores subword n-grams explicitly. the included n-gram lengths are specified using the --minn and --maxn options. the frequency threshold for n-grams is configured with the --ngram-mincount option.
  • the buckets type maps n-grams to buckets using the FNV1 hash. the considered n-gram lengths are specified using the --minn and --maxn options. the number of buckets is controlled with the --buckets option.
• --target-size SIZE — the target size for the token vocabulary. at most SIZE tokens are included for training. only tokens appearing more frequently than the token at SIZE are included.
• --threads N — the number of threads to use during training for parallelization. the default is to use half of the logical CPUs of the machine, capped at 20 threads. increasing the number of threads increases the probability of update collisions, requiring more epochs to reach the same loss. 💀
• --zipf EXP — exponent s used in the Zipf distribution p(k) = 1 / (k^s H_N) for negative sampling. default: 0.5.

🦋 EXAMPLES 🦋

train embeddings on dewiki.txt using the skip-gram model:

finalfrontier skipgram dewiki.txt dewiki-skipgram.bin

train embeddings with dimensionality 200 on dewiki.txt using the structured skip-gram model with a context window of 5 tokens:

finalfrontier skipgram --model structgram --context 5 --dims 200 \
  dewiki.txt dewiki-structgram.bin

🔥 SEE ALSO 🔥

finalfrontier(1), finalfrontier-deps(1)

👑 AUTHORS 👑

Daniel de Kok.

and that’s the whole page, bestie 🫶 this is from the FINALFRONTIER-SKIPGRAM(1) source/version and it’s dated Sep 8, 2018 — periodt 💅✨