DTSEARCH(5)

🌸 DtSearch — the special file that runs the whole search vibe 🌸

okay bestie, so this page is DtSearch(5) — that's the manual section she lives in — and she is literally the whole text search and retrieval system. not a drill. 💅

---

✨ DESCRIPTION ✨

so DtSearch is a general purpose text search and retrieval system, and she serves as the text search engine for the DtInfo browser in the Common Desktop Environment (CDE). main character energy fr fr. 🎀

she uses a full text inverted index of natural language words and stems. both queries and documents have been internationalized for CDE single- and multi-byte languages, with provision for the definition of custom languages. so yes, she speaks everyone's language. multilingual icon. 🌈

queries are just simple text strings, but they can optionally include full boolean specifications with a simple intuitive syntax — so you can go full nerd if you want. and results of searches can be ranked statistically. 👑

document retrievals can include information for highlighting query words in retrieved documents, which is honestly so helpful of her.

---

DtSearch is made of two major functional areas. let's break it down:

💾 the offline build tools

this first area is a set of offline build tools that:

• create searchable databases.
• index the user's text files and load the resultant search information into the databases.
• maintain the databases.

🔍 the online search API

the second functional area is an online search API. it provides a simple interface to the search engine to facilitate user-written search and retrieval programs. the API is a set of functions compiled into the library libDtSearch, with function prototypes, constant definitions, and data structures defined in Search.h. 🦋

also DtSearch includes a sample browser source program, dtsrtest.c, to demonstrate API usage — a little demo she brought along to show off. 👑

information and error messages in both functional areas — including those appended to the online API MessageList — are generated from a single DtSearch Message catalog, dtsearch.cat. the source for this catalog is dtsearch.msg.

every DtSearch database is associated with a single full text inverted index. plus (and this is the cute part) each database can be partitioned into logical subsets of documents called "keytypes" by a naming convention of the database keys. the search engine can open multiple databases, and users can specify any combination of databases and keytypes for each query — that gives you a two tier search capability. 🌟

users can further qualify searches by restricting the search return list by date ranges and maximum number of documents returned. so you can be like "only the fresh stuff, only this many, thank you." 🙈

DtSearch is written in ANSI Standard and POSIX compliant C.

⚠️ okay important: the DtSearch online search API is not reentrant (not "thread-safe") and must therefore be directly linked into the user-written search program. no cap, that's a hard requirement. 💀

the DtSearch API will increase the size of a browser search program from 100K to 200K bytes depending on which functions are called. she's thicc but worth it. 😭

---

🧚 GENERAL SPECIFICATIONS AND CONVENTIONS 🧚

🗂️ Database Names

databases consist of a set of binary and ASCII files whose names are:

• the 1- to 8-character ASCII database name specified to the dtsrcreate command,
• a period (.),
• and a 1- to 3-character ASCII file name suffix.

executing dtsrcreate will create and initialize these files. after creation, databases are always identified by the 1- to 8-character name string used in dtsrcreate. consistency is key bestie. 💅

🚫 the database names dtsearch and austext are reserved and may not be specified. those names are taken, respectfully.

🌍 DtSearch Languages

each database is associated with a single natural language. unlike conventional locales, a DtSearch language includes code set presumptions and — most importantly — linguistic parsing and stemming rules to identify indexable terms in a text stream. she's not just doing vibes, she's doing linguistics. 🎀

a DtSearch language is specified when a database is created. developers can also define custom languages with special code sets and linguistic rules. see "Language Parsing and Stemming" in this man page below for details. ✨

🏷️ Database Types

the API can be used simply as a search engine, referring to documents only through the inverted indexes. alternatively, a database can be configured to store actual document text in compressed format in a repository efficiently accessible to the engine. 💖

the configuration options that indicate these alternatives are referred to as database types and are specified to dtsrcreate at database creation time. so like, choose your fighter at the start. 🦋

📝 Abstracts

a field called the "abstract" is included in the fzk file for each document loaded into a database, and is included on the Results list for each document in a successful search. she shows up in the results and in the file. overachiever. 👑

when documents are not stored in a repository, the abstract typically specifies a file name, URL, or other reference useful to the browser. it can also include summary information viewable by users to help them select documents for retrieval and display. so helpful honestly. 🌸

🛠️ Offline Build Tools

dtsrcreate creates and initializes new databases or reinitializes preexisting databases. textual data is loaded into databases by the execution of two programs:

• dtsrload creates a database object record for each text document.
• dtsrindex creates the full text inverted index of words and stems for each object record.

based on unique database keys for each object, dtsrload and dtsrindex can also serve as update programs for preexisting databases. glow-ups are allowed. 💅

the input to the load and index programs is a canonical text file with a .fzk file name suffix. the format of fzk files is sufficiently simple that they can be generated manually — yes, by hand, bestie, it's giving DIY. ✨

in addition, DtSearch includes a utility program, dtsrhan, which can generate a correctly formatted fzk file for some kinds of text documents.

several other utilities provided in the distribution package are suitable for extracting summary database information, including dtsrdbrec and dtsrkdump. 🎀

⌨️ Argument Conventions

optional command line arguments are specified with a dash (-) and typically a single character argument identifier. some required arguments also use the dash convention. unless specifically indicated otherwise, dash arguments may be specified in any order. 🌀

where values are used with dash arguments, they must be directly appended to the argument without white space. like, no space, no gap, no nothing. periodt. 💀

optional arguments precede required arguments. non-dash required arguments must usually be specified in the order indicated by the usage statement. order matters, bestie, she's not flexible about everything. 🫶

---

okay so that's the whole foundation — the specs, the conventions, the whole vibe check of how DtSearch operates. and there's still so much more coming, so don't go anywhere. 🌈

🌸 LANGUAGE PARSING AND STEMMING 🌸

✨ Parsing and Stemming ✨

Okay so bestie, word parsing is literally fundamental to how DtSearch works at both index time and query time. Linguistic parsing algorithms filter incoming text strings into sequences of word tokens for each natural language. Depending on the language, word tokens may also be processed into stem tokens. At index time, each linguistic token — or term — in a document is stored in the inverted index. At search time, queries are parsed for linguistic terms and used to access the documents that contain them. Iconic behavior, no cap. 💅

Each database gets its own DtSearch language, identified by a language number at database creation time. That language number determines the parsing and stemming algorithms applied to the database’s text and queries. Internal DtSearch algorithms are supplied for supported languages, including several European languages and Japanese. Plus there’s a user exit mechanism that lets developers provide their own custom language algorithms for a database. Main character energy for the developers, honestly. 🫶

🎀 Language Files 🎀

Language algorithms often use various word lists. Typically these lists are stored in language files for easy maintenance, with the type of list identified by the file name extension. Language files are opened and read into internal tables when the offline programs initialize, or when the DtSearchInit online API function is called. Some language files are required, and initialization will return fatal errors if they’re missing — dramatic, but fair. Some language files are optional, and associated algorithms will be silently bypassed if they’re missing. Files for supported languages may be edited to provide database-specific enhancements. At open time, database-specific files supersede generic language files. Slay. ✨

🌍 General European Parsing Rules 🌍

The currently supported European languages are:

• 0 — English, ASCII character set
• 1 — English, ISO Latin-1 character set
• 2 — Spanish, ISO Latin-1 character set
• 3 — French, ISO Latin-1 character set
• 4 — Italian, ISO Latin-1 character set
• 5 — German, ISO Latin-1 character set

If not otherwise specified, dtscreate will initialize databases with language number 0. Note that all supported European languages use a single-byte encoding method, with the ASCII code set as a proper subset. Cute and efficient. 🎀

Parsed text — including both queries and indexed text in documents — is case insensitive in supported European languages.

In supported European languages, parsing is accomplished with the Teskey algorithm. It partitions a character set into characters that are always parts of words (concordable), characters that are never parts of words (nonconcordable), and characters that may be parts of words depending on context (optionally concordable). Typically, alphanumeric characters are concordable. Whitespace and most punctuation is nonconcordable. Slashes are examples of characters that may or may not separate words depending on context. The essence of the parsing algorithm is “optionally concordable characters preceding concordable characters are concordable; otherwise, they are nonconcordable”. For example, UNIX directory names of the form /usr/local/bin would be considered just one word, but slashes in isolation would be discarded as nonconcordable. The way that works is lowkey elegant, ngl. 💖

The parsing algorithm does a table lookup to determine the concordability of characters. The tables are arrays of the characters for each code page supported by the algorithm. Currently 7-bit ASCII and ISO Latin-1 are supported.

🚫 Words Not Indexed 🚫

Several additional parsing rules are applied to prevent indexing meaningless terms. These terms include common prepositions, indefinite articles, and nonlinguistic text strings such as formatting tags, sequences of hexadecimal dump characters, list identifiers, etc. Not everything deserves main character status, bestie. 💀

Tokens whose lengths are less than a minimum word size or greater than a maximum word size are discarded. The default minimum and maximum word sizes can be overridden in dtsrcreate.

Similarly, words found in the “stop list” file for the database are discarded. Stop lists are external, editable language files. Each supported European language is provided with a default stop list.

Words found in an “include list” file are forcibly indexed even if they would otherwise be discarded. Include list database files are optional; no defaults are provided. So if you want extra words to get VIP treatment, that’s your moment. 🎟️

🌱 Stemming 🌱

When specified for a language, individual parsed words will be “conflated” or mapped into their “stem” form — a new word that represents the etymological root of the original word. A default null stemming algorithm is used for languages that are not otherwise provided with a supported stemmer. The null stemmer returns the original word as its own stem. Both words and stems are stored in the inverted index. API searches can be specified for either words or stems, but the two search methods are distinguished only when real stems have been stored in the inverted index. Important distinction, bestie. 🫡

In the supported European languages, stemming can be accomplished heuristically or by dictionary lookup. The heuristic algorithms typically remove affixes in a language-dependent way. Affix lists are usually stored in language files. Currently stemming is supported for English languages 0 and 1, Spanish language 2, French language 3, Italian language 4, and German language 5. We love a multilingual queen. 🌈

🗾 Japanese 🗾

Two Japanese DtSearch languages — numbers 6 and 7 — are supported. Both use the same packed, Extended UNIX Code (EUC) character set. The two languages differ only in the technique used to parse compound kanji words. All validly encoded text for supported Japanese languages incorporates ASCII encoding as a proper, single-byte subset. The supported Japanese languages use the null stemmer.

🈴 Kanji Compounds 🈴

Individual kanji characters are parsed as single words. In addition:

• For language number 6, all possible kanji substrings — pairs, triplets, etc. — found in any contiguous string of kanjis will be parsed as compound kanji words, up to a maximum word size of 6 kanji characters.
• For language number 7, only kanji substrings listed in the jpn.knj language file may be treated as compound kanji words.

At offline index time, all possible individual kanjis and kanji compounds for a language are stored in the inverted index. At online search time, kanji substrings in the query are treated as single query terms and are not compounded further. Periodt. 🔥

🔤 Japanese Code Sets 🔤

The supported packed EUC character set consists of four separate multibyte Code Sets.

• Code Set 0 can be either 7-bit ASCII or 7-bit JIS-Roman. The first and only byte of a character in Code Set 0 is less than 0x80. Substrings of Code Set 0 in supported Japanese text are parsed into individual words with the European language parser described above. Minimum and maximum word sizes, stop lists, and include lists will be used as in European languages if provided with a Japanese database.
• Code Set 1 is JIS X 0208-1990. The two-byte characters in Code Set 1 always begin with a byte greater than 0xA0 and less than 0xFF. Symbols and line drawing elements are not indexed. Hirigana strings are discarded as equivalent to stop list words. Contiguous substrings of katakana, Roman, Greek, or cyrillic are parsed as single words. Individual kanji characters are treated as single words with additional kanji compounding depending on language number, as described above. Characters from unassigned kuten rows are treated as user-defined kanji.
• Code Set 2 is halfwidth katakana. The two-byte characters in Code Set 2 always begin with the unique byte 0x8E. Contiguous strings are parsed as single words.
• Code Set 3 is JIS X 0212-1990. The three-byte characters in Code Set 3 always begin with the unique byte 0x8F. Parsing is similar to Code Set 1: discard symbols, etc., contiguous strings of related foreign characters equal words, and individual kanjis and unassigned characters equal single words, with additional kanji compounding depending on language. Kuten row 5 is treated as katakana; undefined rows are treated as kanji.

🛠 Custom Languages 🛠

All language-dependent data structures and functions are referenced by fields in the main internal DtSearch structure for databases (DBLK). The same structure is used for offline build programs as well as online API search functions. Language processing is initialized database by database by an internal language loader function which stores values in DBLK fields. A database whose language number is not supported is presumed to be associated with a custom language. A special function, load_custom_language, is called to initialize language fields for custom languages. The default load_custom_language merely returns an error code. However, developers can link in their own load_custom_language function, which will be called to initialize the DBLK fields needed to parse and stem one or more custom languages. Values required for the language fields of a DBLK are specified in DtSrAPI(3). Build-your-own-language era, honestly iconic. 🧚

---

👀 SEE ALSO 👀

• dtsrcreate(1)
• dtsrdbrec(1)
• dtsrhan(1)
• dtsrindex(1)
• dtsrload(1)
• dtsrkdump(1)
• huffcode(1)
• DtSrAPI(3)
• dtsrfzkfiles(4)
• dtsrocffile(4)
• dtsrhanfile(4)
• dtsrlangfiles(4)
• dtsrdbfiles(4)

xoxo, your Unix bestie 💅 — page source/version: DtSearch(special file), and the date is whatever the footer says on the real page (timestamp lives at the bottom, bestie 🎀).