txt(3)

💖 NAME 💖

txt — the standard text processing module. yep, that's the whole name. it's txt(3) from the Library Functions Manual and it's lowkey a whole universe. 🦋

---

🧚 STANDARD TEXT PROCESSING MODULE 🧚

okay so the Standard Text Processing module is an original implementation of an object collection dedicated to text processing — not a remix, an original, we don't do copy-paste here. 💅

Text scanning is the operation everybody's doing in the text-processing field rn, but that is not all it does. The module also hands you specialized objects to store and index text data. Text sorting and transliteration? Also part of this module. it's giving full package, no crumbs. ✨

🔍 Scanning concepts 🔍

Text scanning is the ability to pull lexical elements (a.k.a. lexemes) out of a stream. The main character doing this job is the scanner, a.k.a. the lexical analyzer. You build a scanner by adding special objects that act as pattern matchers. And when a pattern gets matched, a special object called a lexeme comes back. that's the whole loop bestie. 🎀

🎯 Pattern object 🎯

A Pattern object is that special object acting as the model for the string you want to match. There are several ways to build a pattern. The simplest? Build it with a regular expression. Another type is the balanced pattern. In its first form, a pattern object can be created with a regular expression object.

# create a pattern object
const pat (afnix:txt:Pattern "$d+")

In this example the pattern object is built to detect integer objects. slay.

pat:check "123" # true
pat:match "123" # 123

So check returns true if the input string matches the pattern, and match returns the string that matches the pattern. And since a pattern object can also operate with a stream object, the match method is the right one when you want to match a particular string. The pattern object is, as usual, available with the appropriate predicate.

afnix:txt:pattern-p pat # true

Now — another form of pattern object is the balanced pattern. A balanced pattern is determined by a starting string and an ending string. There are two types of balanced pattern: one is the single balanced pattern, the other is the recursive balanced pattern. The single balanced pattern is appropriate for those lexical elements defined by a character. For example, the classical C-string is a single balanced pattern with the double quote character. iconic behavior tbh.

# create a balanced pattern
const pat (afnix:txt:Pattern "ELEMENT" "<" ">")
pat:check "<xml>" # true
pat:match "<xml>" # xml

In the case of the C-string, the pattern might be more appropriately defined with an additional escape character. That character is used by the pattern matcher to grab characters that might actually be part of the pattern definition. iykyk.

# create a balanced pattern
const pat (afnix:txt:Pattern "STRING" "'" '\')
pat:check "'hello'" # true
pat:match "'hello'" # "hello"

In this form you get a balanced pattern with an escape character. The same string is used for both the starting and the ending string. There's another constructor that takes two strings, which you'd use if the starting and ending strings are different. And the last pattern form is the balanced recursive form. Here a starting and ending string delimit the pattern, but in this mode a recursive use of the starting and ending strings is allowed. In order to get an exact match, the number of starting strings must equal the number of ending strings. For example, the C-comment pattern can be viewed as a recursive balanced pattern. 🤯

# create a c-comment pattern
const pat (afnix:txt:Pattern "STRING" "/*" "*/" )

🏷️ Lexeme object 🏷️

The Lexeme object is the object built by a scanner that holds the matched string. So a lexeme is basically a tagged string. On top of that, a lexeme can carry extra info like a source name and an index.

# create an empty lexeme
const lexm (afnix:txt:Lexeme)
afnix:txt:lexeme-p lexm # true

The default lexeme is created with any value. A value can be set with the set-value method and retrieved with the get-value methods.

lexm:set-value "hello"
lexm:get-value # hello

Similar vibes with set-tag and get-tag, which operate with an integer. The source name and index get defined with the same style of methods too.

# check for the source
lexm:set-source "world"
lexm:get-source # world
# check for the source index
lexm:set-index 2000
lexm:get-index # 2000

📖 Text scanning 📖

Text scanning is the ability to extract lexical elements (lexemes) from an input stream. Generally the lexemes are the results of a matching operation defined by a pattern object. So the definition of a scanner object is the object itself plus one or several pattern objects. simple as that. 🫶

🛠️ Scanner construction 🛠️

By default a scanner is created without pattern objects. The length method returns the number of pattern objects. As usual, a predicate is associated with the scanner object.

# the default scanner
const  scan (afnix:txt:Scanner)
afnix:txt:scanner-p scan # true
# the length method
scan:length # 0

The scanner construction proceeds by adding pattern objects. Each pattern can be created independently and added to the scanner later. For example, a scanner that reads real, integer and string can be defined as follow:

# create the scanner pattern
const REAL    (
  afnix:txt:Pattern "REAL"    [$d+.$d*])
const STRING  (
  afnix:txt:Pattern "STRING"  """ '\')
const INTEGER (
  afnix:txt:Pattern "INTEGER" [$d+|"0x"$x+])
# add the pattern to the scanner
scanner:add INTEGER REAL STRING

The order of pattern integration defines the priority at which a token gets recognized. The symbol name for each pattern is optional, since functional programming lets you create patterns directly. And honestly this writing style makes the scanner definition easier to read. 👑

🚀 Using the scanner 🚀

Once constructed, the scanner can be used as is. A stream is generally the best way to operate. If the scanner reaches the end-of-stream, or it cannot recognize a lexeme, the nil object is returned. With a loop it's easy to grab all the lexemes.

while (trans valid (is:valid-p)) {
  # try to get the lexeme
  trans lexm (scanner:scan is)
  # check for nil lexeme and print the value
  if (not (nil-p lexm)) (println (lexm:get-value))
  # update the valid flag
  valid:= (and (is:valid-p) (not (nil-p lexm)))
}

In this loop it's necessary to check for the end of the stream first. That's done with the help of the special loop construct that initializes the valid symbol. And as soon as the lexeme is built, you can use it — the lexeme holds the value as well as its tag. ✨

🔤 Text sorting 🔤

Sorting is one of the primary functions implemented inside the text processing module. There are three sorting functions available in the module. three! a trio! 💅

⬆️⬇️ Ascending and descending order sorting ⬆️⬇️

The sort-ascent function operates with a vector object and sorts the elements in ascending order. Any kind of objects can be sorted as long as they support a comparison method. The elements are sorted in place by using a quick sort algorithm.

# create an unsorted vector
const v-i (Vector 7 5 3 4 1 8 0 9 2 6)
# sort the vector in place
afnix:txt:sort-ascent v-i
# print the vector
for (e) (v) (println e)

The sort-descent function is similar to the sort-ascent function, except the objects get sorted in descending order. flippin' the script, literally. 🙈

🅰️ Lexical sorting 🅰️

The sort-lexical function operates with a vector object and sorts the elements in ascending order using a lexicographic ordering relation. Objects in the vector must be literal objects, or an exception is raised. so like, don't test it. 😭

🌈 Transliteration 🌈

Transliteration is the process of changing characters by mapping one to another one. The transliteration process operates with a character source and produces a target character with the help of a mapping table. And heads up: the transliteration process is not necessarily reversible, as is often indicated in the literature. not every glow-up is a round trip bestie. 💀

🧚 Literate object 🧚

The Literate object is a transliteration object that is bound by default with the identity function mapping. As usual, a predicate is associated with the object.

# create a transliterate object
const tl (afnix:txt:Literate)
# check the object
afnix:txt:literate-p tl # true

The transliteration process can also operate with an escape character, in order to map a double character sequence into a single one — as usually found inside programming languages.

# create a transliterate object by escape
const tl (afnix:txt:Literate '\')

⚙️ Transliteration configuration ⚙️

The set-map configures the transliteration mapping table, while the set-escape-map configures the escape mapping table. The mapping is done by setting the source character and the target character. For instance, if one wants to map the tabulation character to a white space, the mapping table is set as follow:

tl:set-map '' ' '

The escape mapping table operates the same way. And it should be noted that the mapping algorithm translates the input character first, eventually yielding an escape character, and then the escape mapping takes place. Note also that the set-escape method can be used to set the escape character. order matters, periodt. 💅

tl:set-map '' ' '

🔁 Transliteration process 🔁

The transliteration process is done either with a string or an input stream. In the first case, the translate method operates with a string and returns a translated string. On the other hand, the read method returns a character when operating with a stream. two methods, two moods. 🦋

# set the mapping characters
tl:set-map 'w'
tl:set-map '\' 'o'
tl:set-map 'r'
tl:set-map 'd'
# translate a string
tl:translate "helo" # word

✨ STANDARD TEXT PROCESSING REFERENCE ✨

Okay bestie, this is the part where we get into the actual classes and functions that make the text processing magic happen. Buckle up. 🎀

🌸 Pattern 🌸

The Pattern class is a pattern matching class. It's based on either a regular expression or a balanced string. In regex mode, the pattern is defined with a regex, and a match happens when a regex match is achieved. In balanced string mode, the pattern is defined with a start pattern and end pattern strings. The balanced mode can be single or recursive. Also, an escape character can be associated with the class. A name and a tag are also bound to the pattern object to make integration within a scanner easier. 💅

Predicate

• pattern-p

Inheritance

• Object

Constructors

• Pattern (none) — creates an empty pattern. It's giving blank canvas. 🎨
• Pattern (String|Regex) — creates a pattern object associated with a regular expression. The argument can be either a string or a regular expression object. If the argument is a string, it gets converted into a regular expression object. Slay. ✨
• Pattern (String String) — creates a balanced pattern. First argument is the start pattern string. Second argument is the end balanced string. Balanced queen. ⚖️
• Pattern (String String Character) — creates a balanced pattern with an escape character. First argument is the start pattern string. Second argument is the end balanced string. Third character is the escape character. Escape room vibes. 🚪
• Pattern (String String Boolean) — creates a recursive balanced pattern. First argument is the start pattern string. Second argument is the end balanced string. Recursion? We love to see it. 🔁

Constants

• REGEX — this constant indicates that the pattern is a regular expression. Regex era. 💖
• BALANCED — this constant indicates that the pattern is a balanced pattern. Balanced baddie. 🧘
• RECURSIVE — this constant indicates that the pattern is a recursive balanced pattern. It goes deeper. 🌀

Methods

• check -> Boolean (String) — checks the pattern against the input string. If verification is successful, returns true, otherwise false. Yes/no energy. ✅
• match -> String (String|InputStream) — attempts to match an input string or an input stream. If matching occurs, the matching string is returned. If input is a string, the end of string is used as an end condition. If input stream is used, the end of stream is used as an end condition. Match made in heaven. 💘
• set-tag -> none (Integer) — sets the pattern tag. The tag can be used inside a scanner. Tag it. 🏷️
• get-tag -> Integer (none) — returns the pattern tag. Gimme the tag. 🏷️
• set-name -> none (String) — sets the pattern name. The name is a symbol identifier for that pattern. Name it, bestie. 📛
• get-name -> String (none) — returns the pattern name. What's their name? 🗣️
• set-regex -> none (String|Regex) — sets the pattern regex either with a string or with a regex object. If successfully completed, the pattern type is switched to the REGEX type. Glow up to regex. ✨
• set-escape -> none (Character) — sets the pattern escape character. The escape character is used only in balanced mode. Escape only when balanced. 🦋
• get-escape -> Character (none) — returns the escape character. Who's the escape artist? 🎭
• set-balanced -> none (String| String String) — sets the pattern balanced string. With one argument, the same balanced string is used for starting and ending. With two arguments, the first argument is the starting string and the second is the ending string. Balanced and booked. 📚

🌼 Lexeme 🌼

The Lexeme class is a literal object designed to hold a matching pattern. A lexeme consists of a string (the lexeme value), a tag, and eventually a source name (file name) and a source index (line number). It's the receipt of a match. 🧾

Predicate

• lexeme-p

Inheritance

• Literal

Constructors

• Lexeme (none) — creates an empty lexeme. Empty but full of potential. 🌱
• Lexeme (String) — creates a lexeme by value. The string argument is the lexeme value. Value check. 💎

Methods

• set-tag -> none (Integer) — sets the lexeme tag. The tag can be used inside a scanner. Tag it again. 🏷️
• get-tag -> Integer (none) — returns the lexeme tag. Tag retrieval. 🔍
• set-value -> none (String) — sets the lexeme value. The lexeme value is generally the result of a matching operation. Set the value, get the glory. 🌟
• get-value -> String (none) — returns the lexeme value. What's the value? 💰
• set-index -> none (Integer) — sets the lexeme source index. The lexeme source index can be for instance the source line number. Index it. 📇
• get-index -> Integer (none) — returns the lexeme source index. Where in the line? 📍
• set-source -> none (String) — sets the lexeme source name. The lexeme source name can be for instance the source file name. Source reveal. 📁
• get-source -> String (none) — returns the lexeme source name. Name drop. 🗂️

🦋 Scanner 🦋

The Scanner class is a text scanner or lexical analyzer that operates on an input stream and permits matching one or several patterns. The scanner is built by adding patterns to the scanner object. With an input stream, the scanner object attempts to build a buffer that matches at least one pattern. When such matching occurs, a lexeme is built. When building a lexeme, the pattern tag is used to mark the lexeme. It's the main character of tokenization. 👑

Predicate

• scanner-p

Inheritance

• Object

Constructors

• Scanner (none) — creates an empty scanner. Empty scanner, full potential. 🫧

Methods

• add -> none (Pattern*) — adds 0 or more pattern objects to the scanner. The priority of the pattern is determined by the order in which the patterns are added. Order matters, bestie. 🔢
• length -> Integer (none) — returns the number of pattern objects in this scanner. Count them. 🧮
• get -> Pattern (Integer) — returns a pattern object by index. Get that pattern. 🎯
• check -> Lexeme (String) — checks that a string is matched by the scanner and returns the associated lexeme. Check yes. ✅
• scan -> Lexeme (InputStream) — scans an input stream until a pattern is matched. When a matching occurs, the associated lexeme is returned. Scan until you slay. 🔍

🧚 Literate 🧚

The Literate class is a transliteration mapping class. Transliteration is the process of changing characters by mapping one to another one. The transliteration process operates with a character source and produces a target character with the help of a mapping table. This transliteration object can also operate with an escape table. In the presence of an escape character, an escape mapping table is used instead of the regular one. It's giving character transformation. 🪄

Predicate

• literate-p

Inheritance

• Object

Constructors

• Literate (none) — creates a default transliteration object. Default but cute. 🌸
• Literate (Character) — creates a default transliteration object with an escape character. The argument is the escape character. Escape artist. 🎨

Methods

• read -> Character (InputStream) — reads a character from the input stream and translates it with the help of the mapping table. A second character might be consumed from the stream if the first character is an escape character. Read and translate. 📖
• getu -> Character (InputStream) — reads a Unicode character from the input stream and translates it with the help of the mapping table. A second character might be consumed from the stream if the first character is an escape character. Unicode baddie. 🦄
• reset -> none (none) — resets all the mapping table and installs a default identity one. Reset button. 🔄
• set-map -> none (Character Character) — sets the mapping table by using a source and target character. The first character is the source character. The second character is the target character. Map it. 🗺️
• get-map -> Character (Character) — returns the mapping character by character. The source character is the argument. Get the map. 🧭
• translate -> String (String) — translates a string by transliteration and returns a new string. New string, who dis? ✨
• set-escape -> none (Character) — sets the escape character. Escape set. 🚪
• get-escape -> Character (none) — returns the escape character. Escape reveal. 🎭
• set-escape-map -> none (Character Character) — sets the escape mapping table by using a source and target character. The first character is the source character. The second character is the target character. Escape map unlocked. 🗝️
• get-escape-map -> Character (Character) — returns the escape mapping character by character. The source character is the argument. Escape map retrieval. 🔎

⭐ Functions ⭐

• sort-ascent -> none (Vector) — sorts in ascending order the vector argument. The vector is sorted in place. Ascend, queen. ⬆️
• sort-descent -> none (Vector) — sorts in descending order the vector argument. The vector is sorted in place. Descend, king. ⬇️
• sort-lexical -> none (Vector) — sorts in lexicographic order the vector argument. The vector is sorted in place. Lexicographic realness. 🔤

---

That's the whole text processing reference, bestie. This has been your AFNIX Module, AFNIX, txt(3) moment — stay iconic, stay matched, and don't forget to close your streams. 💖✨🎀