HFST-TOKENIZE(1)

📦
version 3.17.1 · 2 versions
  • 3.17.1 · debian bookworm, debian trixie, nixos 26.05, opensuse-leap 16.0, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking (latest)
  • 3.17.3 · nixpkgs unstable
, in debian bookworm, debian trixie, nixos 26.05, opensuse-leap 16.0, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

okay so</p> you just pulled up hfst-tokenize, a User Commands manual page from the HFST girlies, and honestly? you're about to have the best time doing matching/lookup on text streams. welcome to the group chat 💅✨

✨ NAME ✨

hfst-tokenize — perform matching/lookup on text streams 🫶

🌸 SYNOPSIS 🌸

hfst-tokenize [--segment | --xerox | --cg | --giella-cg] [OPTIONS...]
RULESET

🦋 DESCRIPTION 🦋

perform matching/lookup on text streams. that's the whole vibe. 🎀

💖 Common options: 💖

  • -h, --help — Print help message. your lil "what do I do" button.
  • -V, --version — Print version info.
  • -v, --verbose — Print verbosely while processing. she talks a LOT.
  • -q, --quiet — Only print fatal errors and requested output.
  • -s, --silent — Alias of --quiet. same girl, different name 🫶
  • -n, --newline — Newline as input separator (default is blank line). so default is blank-line, and -n says "nah, newlines".
  • -a, --print-all — Print nonmatching text. nobody gets left out ✨
  • -w, --print-weight — Print weights (overrides earlier -W option).
  • -W, --no-weights — Don't print weights (default; overrides earlier -w, or -w implied by -g, options).
  • -m, --tokenize-multichar — Tokenize multicharacter symbols (by default only one utf-8 character is tokenized at a time regardless of what is present in the alphabet).
  • -b, --beam=B — Output only analyses whose weight is within B from best result.
  • -tS, --time-cutoff=S — Limit search after having used S seconds per input.
  • -lN, --weight-classes=N — Output no more than N best weight classes (where analyses with equal weight constitute a class).
  • -u, --unique — Remove duplicate analyses. de-dupe, slay.
  • -z, --segment — Segmenting / tokenization mode (default).
  • -i, --space-separated — Tokenization with one sentence per line, space-separated tokens.
  • -x, --xerox — Xerox output.
  • -c, --cg — Constraint Grammar output.
  • -S, --superblanks — Ignore contents of unescaped [] (cf. apertium-destxt); flush on NUL.
  • -g, --giella-cg — CG format used in Giella infrastructure (implies -w and -l2, treats @PMATCH_INPUT_MARK@ as subreading separator, expects tags to be Multichar_symbols, flush on NUL).
  • -C, --conllu — CoNLL-U format.
  • -f, --finnpos — FinnPos output.
  • -L, --visl — VISL input and output (implies -W, handles <s> as blocks and <STYLE> inline).

Use standard streams for input and output (for now). 🌀

😭 REPORTING BUGS 😭

Report bugs to <hfst-bugs@helsinki.fi> or directly to our bug tracker at: <https://github.com/hfst/hfst/issues>

hfst-tokenize home page: <https://github.com/hfst/hfst/wiki/HfstTokenize> General help using HFST software: <https://github.com/hfst/hfst/wiki>

bestie if it's broken, tell them. they literally want to know. fr fr 🙈

Copyright © 2017 University of Helsinki, License GPLv3: GNU GPL version 3 <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

and that's the whole page 💜 this beauty is part of the HFST collection and the version in front of you is dated August 2018 — so yes, she's an icon, she's been an icon, and she'll stay an icon. periodt. 🎀🦋💖

📦 where this page comes from