ucto(1)

🎀 NAME 🎀

ucto — the Unicode Tokenizer 💖✨ (yes bestie, we're in the General Commands Manual, section 1, and this lil icon is about to become your fave)

---

🌸 SYNOPSIS 🌸

ucto [[options]] [input‐file] [[output‐file]]

---

🦋 DESCRIPTION 🦋

okay so picture this: you got a text file and it's just… a wall of vibes. ucto fixes that. It tokenizes text files, which is a fancy way of saying it:

• separates words from punctuation ✂️
• splits sentences (and optionally paragraphs, if you're feeling extra)
• finds paired quotes 👯

And the best part? Ucto comes preconfigured with tokenisation rules for several languages. No cap.

Those rules? They're provided by uctodata. Thank u uctodata, you ate. 🙏

---

💅 OPTIONS 💅

so many options. we love options. let's get into it, one by one, no skipping, periodt.

• -c configfile — read your settings from a 'configfile'. main character config energy.
• -B — run in batch mode. It processes all inputfiles to an output directory specified with -O.
• -d value — set debug mode to 'value'. 🔍
• -e value — set input encoding. (default is UTF8) 🫶
• -I value — set the input directory to 'value'. (batch mode only)
• -O value — set the output directory to 'value'. (Required for batch mode — like, actually required, don't come crying to me)
• -N value — set UTF8 output normalization. (default NFC)
• --filter=[YES|NO] — disable filtering of special characters, (default YES). These special characters can be specified in the [FILTER] block of the configuration file. 🧚
• -L language — Automatically selects a configuration file by language code. The language code is generally a three-letter iso-639-3 code. For example, 'fra' will select the file tokconfig‐fra from the installation directory. so smart of her honestly.
• --detectlanguages=<lang1,lang2,..langn> — try to detect all the specified languages. The default language will be 'lang1'. (only useful for FoLiA output).
  • All values must be iso-639-3 codes.
  • You can also use the special language code und. This ensures there is NO default language, and any language that is NOT in the list will remain unanalyzed.

⚠️ Warning: To be able to handle utterances of mixed language, Ucto uses a simple sentence splitter based on the markers '.' '?' and '!'. This may occasionally lead to surprising results. 💀 not me getting jumpscared by a sentence split

• -l — Convert output text to all lowercase. whisper mode 🥺
• -u — Convert all input text to all uppercase. SCREAM MODE 📣
• -n — Emit one sentence per line on output. clean. tidy. iconic.
• -m — Assume one sentence per line on input. trust the input, bestie.
• --normalize=class1,class2,..,classn — map all occurrences of tokens with class1,...class to their generic names. e.g. --normalize=DATE will map all dates to the word {{DATE}}. Very useful to normalize tokens like URL's, DATE's, E-mail addresses and so on. ✨
• -T value or --textredundancy=value — set text redundancy level for text nodes in FoLiA output:
  • 'full' — add text to all levels: <p> <s> <w> etc.
  • 'minimal' — don't introduce text on higher levels, but retain what is already there.
  • 'none' — only introduce text on <w>, AND remove all text from higher levels.
• --allow-word-correction — Allow ucto to tokenize inside FoLiA Word elements, creating FoLiA Corrections. glow up 🫶
• --ignore-tag-hints — Skip all tag=token hints from the FoLiA input. These hints can be used to signal text markup like subscript and superscript.
• --add-tokens="file" — Add additional tokens to the [TOKENS] block of the default language. The file should contain one TOKEN per line. 📝
• --passthru — Don't tokenize, but perform input decoding and simple token role detection. just vibing, no thoughts.
• --filterpunct — remove most of the punctuation from the output. (not from abreviations and embedded punctuation like John's). bye bye commas 👋
• -P — Disable Paragraph Detection.
• -Q — Enable Quote Detection. (this is experimental and may lead to unexpected results) 🙈 she's trying her best okay
• -s <string> — Set End‐of‐sentence marker. (Default <utt>)
• -V or -- version — Show version information.
• -v — set Verbose mode. she will NOT stop talking 💬
• -F — The input file(s) are assumed to be FoLiA XML. Text in the correct 'inputclass' will be tokenized. For files with an '.xml' extension, -F is the default.
  • In batch mode, this forces to only select files with the '.xml' extension from the input directory.
• --inputclass="cls" — When tokenizing a FoLiA XML document, search for text nodes of class 'cls'. The default is "current".
• --outputclass="cls" — When tokenizing a FoLiA XML document, output the tokenized text in text nodes with 'cls'. The default is "current". It is recommended to have different classes for input and output.
• --textclass="cls" (obsolete) — use 'cls' for input and output of text from FoLiA. Equivalent to both --inputclass='cls' and --outputclass='cls'.

🚨 This option is obsolete and NOT recommended. Please use the separate --inputclass= and --outputclass options. do not use the old one bestie i'm begging

• --copyclass — when ucto is used on FoLiA with fully tokenized text in inputclass='inputclass', no text in textclass 'outputclass' is produced. (A warning will be given). To circumvent this. Add the --copyclass option. Which assures that text will be emitted in that class. 🩹 the little fixer upper
• -X — All output will be FoLiA XML. Document id's are autogenerated.
  • Works in batch mode too.
• --id <DocId> — Use the specified Document ID for the FoLiA XML. (not allowed in batch mode) When not provided, a document is is generated based on the nema of the input file.

---

💀 BUGS 💀

likely

(that's it. that's the whole bugs section. she's honest. she's humble. she's real for that. 😭)

---

👑 AUTHORS 👑

• Maarten van Gompel
• Ko van der Sloot

e-mail: lamasoftware@science.ru.nl

---

okay that's the whole page bestie, you're officially a tokenizing genius now 🌈⭐ mwah 💋

anyway this lil doc? it's from the General Commands Manual, it's ucto(1), and it's dated 2024 apr 11. love that for her. bye pookie 🎀💖