swish++.index(5)

💖 swish++.index — the SWISH++ index file format, explained by your fav Unix bestie 💖

okay so. bestie. gather round. this is the manual page for swish++.index(5) — it lives in the File Formats Manual section of your man pages, and it's literally just "here's what a SWISH++ index file looks like on the inside." no cap. it's giving "backstage pass to the index." 🎀

---

✨ NAME ✨

swish++.index — the SWISH++ index file format. that's it. that's the whole name. iconic behavior honestly. 💅

---

🦋 SYNOPSIS 🦋

here's the layout, verbatim, don't touch a single character:

long      num_words;
off_t     word_offset[ num_words ];
long      num_stop_words;
off_t     stop_word_offset[ num_stop_words ];
long      num_directories;
off_t     directory_offset[ num_directories ];
long      num_files;
off_t     file_offset[ num_files ];
long      num_meta_names;
off_t     meta_name_offset[ num_meta_names ];
          word index
          stop-word index
          directory index++
          file index
          meta-name index

look at her. so structured. so organized. main character energy fr fr. 👑

---

🌸 DESCRIPTION 🌸

so the SWISH++ index file format is exactly what you see up there. let me walk you through it because it is a lot and you deserve to know.

every word_offset is an offset into the word index that points at the first character of a word entry. same vibe for stop_word_offset — it's an offset into the stop-word index pointing at the first character of a stop-word entry. directory_offset? offset into the directory index++, pointing at the first character of a directory entry. file_offset is an offset into the file index, pointing at the first byte of a file entry (yes, byte, not character, she's different like that). and finally meta_name_offset is an offset into the meta-name index pointing at the first character of a meta-name entry.

the index file is written this way on purpose so it can be mapped into memory via the mmap(2) Unix system call, which enables ``instantaneous'' access. instantaneous!! the way that she just... loads. we love a memory-mapped queen. 🧚

---

🔢 Encoded Integers 🔢

okay this part is so cool actually. all integers in an index file are stored in an encoded format for compactness. an integer is encoded to use a varying number of bytes. for a given byte, only the lower 7 bits are used for data; the high bit, if set, is used to indicate whether the integer continues into the next byte. the encoded integer is in big-endian byte order.

for example, the integers 0-127 are encoded as single bytes of \x00-\x7F, respectively; the integer 128 is encoded as the two bytes of \x8100.

note that the byte \x80 will never be the first byte of an encoded integer (although it can be any other byte); therefore, it can be used as a marker to embed extra information into an encoded integer byte sequence. sneaky. clever. we stan a reserved marker. 😭

---

📜 Encoded Integer Lists 📜

because the high bit of the last byte of an encoded integer is always clear, lists of encoded integers can be stored one right after the other with no separators. no commas, no spaces, no nothing. just vibes and bytes back to back.

for example, the byte sequence \x010203 encodes the three separate integers 1, 2, and 3; the byte sequence \x81002A8101 encodes the three integers 128, 42, and 129. slay. 💅

---

🔤 Word Entries 🔤

every word entry in the word index is of the form:

word0{data}...

that is: a null-terminated word followed by one or more data entries, where a data entry is:

{F}{O}{R}[{list}...]{marker}

that is: a file-index (F) followed by the number of occurrences in the file (O) followed by a rank (R) followed by zero or more lists of integers followed by a marker byte.

the file-index is an index into the file_offset table. the marker byte is one of:

00  Another data entry follows.
80  No more data entries follow.

a list is:

{type}{I}...x80

that is: a type followed by one or more integers (I) followed by an \x80 byte, where type defines the type of list, i.e., what the integers mean. currently, there is only one type of list:

• 01 — Meta-ID list. Meta-IDs are unique integers identifying which meta name(s) a word is associated with in the meta-name index.
• 02 — Word-position list. every word in a file has a position: the first word is 1, the second word is 2, etc. each word position is stored as a delta from the previous position for compactness.

---

🚫 Stop-Word Entries 🚫

every stop-word entry in the stop-word index is of the form:

stop-word0

that is: every word is null-terminated. short, sweet, does not elaborate. periodt. 💀

---

📁 Directory Entries 📁

every directory entry in the directory index++ is of the form:

directory-path0

that is: a null-terminated full pathname of a directory (not including the trailing slash). the pathnames are relative to where the indexing was performed (unless absolute paths were used). so like... know where you were standing when you built this, bestie. 📍

---

📄 File Entries 📄

every file entry in the file index is of the form:

{D}file-name0{S}{W}file-title0

that is: the file's directory index++ (D) followed by a null-terminated file name followed by the file's size in bytes (S) followed by the number of words in the file (W) followed by the file's null-terminated title.

now here's the tea on where the title comes from, depending on the file type:

• for an HTML or XHTML file, the title is what's between <TITLE> ... </TITLE> pairs
• for an MP3 file, the title is the value of title field
• for a mail or news file, the title is the value of the Subject header
• for a LaTeX file, the title is the argument of the \title command
• for a Unix manual page file, the title is the contents of the first line within the NAME section

and if a file is not one of those types of files, or is but doesn't have a title, the title is simply the file (not path) name. fallback queen. 🫶

---

🏷️ Meta-Name Entries 🏷️

every meta-name entry in the meta-name index is of the form:

meta-name0{I}

that is: a null-terminated meta-name followed by the ID (I). boom. done. ✨

---

⚠️ CAVEATS ⚠️

generated index files are machine-dependent (size of data types and byte-order). so like, do NOT just copy your index file to a different machine and expect it to work, bestie. that's not a vibe check you're gonna pass. 😭

---

👀 SEE ALSO 👀

• index++(1)
• search++(1)

---

💜 AUTHOR 💜

Paul J. Lucas <pauljlucas@mac.com> — the actual legend behind all of this. give him his flowers. 🌸

---

and that's the whole page, bestie!! this came to you from SWISH++, dated March 29, 2004 — yes she's vintage, yes she still eats. now go map something into memory and be free. 🦋💖✨