index++(1)

okay bestie, this is the index++(1) page from the General Commands Manual and it's giving absolute Unix royalty. ๐Ÿ’– buckle up, we're indexing everything.

โœจ NAME โœจ

  • index++ โ€” SWISH++ indexer. that's the whole vibe. ๐Ÿ’…

โœจ SYNOPSIS โœจ

index++ [ options ] directory...  file...

โœจ DESCRIPTION โœจ

okay so index++ is the SWISH++ file indexer. it indexes the files you specify and the files inside the directories you specify. by default, files in subdirectories of those specified directories also get indexed โ€” unless you pass either the -r or --no-recurse option, or the RecurseSubdirs variable is set. ๐Ÿ“โœจ

files are indexed either only if their filename matches one of the patterns in the set you gave with either the -e or --pattern option or the IncludeFile variable (unless standard input is used โ€” next paragraph, bestie) or if the file is not in the set you gave with either the -E or --no-pattern option or the ExcludeFile variable. yes, it's giving "either this or that" energy. ๐Ÿซถ

if there's a single filename of -, then the list of directories and files to index is instead taken from standard input, one per line. in this case, filename patterns of files to index don't need to be specified explicitly: all files, regardless of whether they match a pattern (unless they're in the set not to index specified with either the -E or --no-pattern option or the ExcludeFile variable), are indexed. i.e., index++ assumes you know what you're doing when you specify filenames this way. periodt. ๐Ÿ’…

in any case, care must be taken not to specify files or subdirectories inside directories that are also specified. since directories are recursively indexed by default (unless either the -r or --no-recurse option or the RecurseSubdirs variable is given), explicitly specifying a subdirectory or file in a directory that is also specified will result in those files being indexed more than once. not the duplicate era we want. ๐Ÿ˜ญ

๐Ÿ’– Character Mapping ๐Ÿ’–

characters in the ISO 8859-1 (Latin 1) character set are mapped to their closest ASCII equivalent before further examination and indexing. (individual indexing modules may also do their own character mapping.) it's giving glow-up before the main event. โœจ

๐Ÿ’– Word Determination ๐Ÿ’–

stop words โ€” words that occur too frequently or have no information content โ€” are not indexed. (there's a default built-in set of a few hundred such English words.) additionally, several heuristics are used to determine which words should not be indexed. ๐Ÿง 

first, a word is checked to see if it looks like an acronym. a word is considered an acronym only if it starts with a capital letter and is composed exclusively of capital letters, digits, and punctuation symbols, e.g., AT&T. if a word looks like an acronym, it is indexed and no further checks are done. iconic behavior. ๐Ÿ‘‘

second, there are several other checks that are applied. a word is not indexed if it:

  1. is less than Word_Min_Size letters. (default is 4.)
  2. contains less than Word_Min_Vowels vowels. (default is 1.)
  3. contains more than Word_Max_Consec_Same of the same character consecutively (not including digits). (default is 2.)
  4. contains more than Word_Max_Consec_Consonants consecutive consonants. (default is 5.)
  5. contains more than Word_Max_Consec_Vowels consecutive vowels. (default is 4.)
  6. contains more than Word_Max_Consec_Puncts consecutive punctuation characters. (default is 1.)

๐Ÿ’– Filters ๐Ÿ’–

via the FilterFile configuration file variable, files matching particular patterns can be filtered prior to indexing. via the FilterAttachment configuration file variable, e-mail attachments whose MIME types match particular patterns can be filtered prior to indexing. (see FILTERS in swish++.conf(5).) filtered before the index even sees them โ€” lowkey protective. ๐Ÿ›ก๏ธ

๐Ÿ’– Incremental Indexing ๐Ÿ’–

in order to add words from new documents to an existing index++, either the entire set of documents can be reindexed or the new documents alone can be incrementally indexed. in many cases, reindexing everything is sufficient since index++ is really fast. for a very large document set, however, this may use too many resources. ๐Ÿ˜ฎโ€๐Ÿ’จ

however, there is a pitfall for incremental indexing: if any of the -f, --word-files, -p, or --word-percent options or WordFilesMax or WordPercentMax variables are used, then words that are too frequent are discarded. if new documents are added containing very few of those words, then they could no longer be too frequent. however, there is no way to get them back since they were discarded. ๐Ÿ’€

the way around this problem is not to discard any words by specifying 101%. however, because no words are discarded, the size of the index file will be larger, perhaps significantly so. bigger index, bigger feelings. ๐Ÿซ 

it is possible that, in practice, the loss of words may not be that important โ€” especially if new documents are very similar to old documents and that words that were too frequent in the old set would also be too frequent in new set.

another way around this problem is to do periodic full indexing. self-care, but for your index. ๐Ÿง–โ€โ™€๏ธ

โœจ INDEXING MODULES โœจ

index++ is written in a modular fashion where different types of files have different indexing modules. currently, there are 7 modules: Text (plain text), HTML (HTML and XHTML), ID3 (ID3 tags found in MP3 files), LaTeX, Mail (RFC 822 and Usenet News), Manual (Unix manual pages in nroff(1) with man(7) macros), and RTF (Rich Text Format). squad goals. ๐Ÿ‘ฏโ€โ™€๏ธ

๐Ÿ’– Text Module ๐Ÿ’–

this module simply indexes plain text files performing character mapping and word determination as has already been described. simple. clean. no drama. โœจ

๐Ÿ’– HTML and XHTML Module ๐Ÿ’–

additional processing is done for HTML and XHTML files. the additional processing is:

  1. character and numeric (decimal and hexadecimal) entity references are converted to their ASCII character equivalents before further examination and indexing. for example, résumé becomes resume before indexing. glow-up complete. ๐Ÿ’…
  2. if a matched set of <TITLE> ... </TITLE> tags is found within the first TitleLines lines of the file (default is 12), then the text between the tags is stored in the generated index file as the file's title rather than the file's name. (every non-space whitespace character in the title is converted to a space; leading and trailing spaces are removed.) title drop, but make it searchable. ๐ŸŽค
  3. if an HTML or XHTML element contains a CLASS attribute whose value is among the set of class names specified as those not to index (via one or more of either the -C or --no-class option or the ExcludeClass variable), then all the text up to the tag that ends the element will not be indexed. for an element that has an optional end tag, "the tag that ends the element" is either the element's end tag or a tag of another element that implicitly ends it; for an element that does not have an end tag, "the tag that ends the element" is the element's start tag. (see the EXAMPLES.) all elements from the HTML 4.0 specification (including deprecated elements), Ruby elements, plus common, browser-specific elements are recognized; unrecognized elements are ignored. (see the -H or --dump-html option.)
  4. if an HTML or XHTML element contains a TITLE attribute, then the words specified as the value of the TITLE attribute are indexed. attribute said "index me" and we did. ๐Ÿซถ
  5. if an AREA, IMG, or INPUT element contains an ALT attribute, then the words specified as the value of the ALT attribute are indexed. alt text really said "i'm the moment." ๐Ÿ“ธ
  6. if a META element contains both a NAME and CONTENT attribute, then the words specified as the value of the CONTENT attribute are indexed associated with the meta name specified as the value of the NAME attribute. (however, if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words specified as the value of the CONTENT attribute are still indexed, but not associated with the meta name.) (see also the -m, --meta, -M, and --no-meta options or the IncludeMeta or ExcludeMeta variables.) meta names can later be queried against specifically using search++(1). meta, but make it fashion. ๐Ÿ’…
  7. if a TABLE element contains a SUMMARY attribute, then the words specified as the value of the SUMMARY attribute are indexed. table summary said "don't skip me." ๐Ÿ“Š
  8. if an OBJECT element contains a STANDBY attribute, then the words specified as the value of the STANDBY attribute are indexed. standby text gets its moment. โœจ
  9. all other HTML or XHTML tags and comments (anything between < and > characters) are discarded. bye bestie. ๐Ÿ‘‹

in compliance with the HTML specification, any one of no quotes, single quotes, or double quotes may be used to contain attribute values and attributes can appear in any order. values containing whitespace, however, must be quoted. the specification is vague as to whether whitespace surrounding the = is legal, but index++ allows it. flexible queen behavior. ๐Ÿ‘‘

๐Ÿ’– ID3 Module ๐Ÿ’–

ID3 tags are used to store audio meta information for MP3 files (generally). since audio files contain mostly binary information, only the ID3 tag text fields are indexed. ID3 tag versions 1.x and 2.x (through 2.4) are supported (except for encrypted frames). if a file contains both 1.x and 2.x tags, only the 2.x tag is indexed. the processing done for files containing an ID3 tag is:

  1. if a title field is found, then the value of the title is stored in the generated index file as the file's title rather than the file's name. (every non-space whitespace character in the title is converted to a space; leading and trailing spaces are removed.) title field said main character energy. ๐ŸŽถ
  2. words that are the value of fields are indexed associated with the field name as a meta name. (however, if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words specified as the value of the field are still indexed, but not associated with the field.) (see also the -m, --meta, -M, and --no-meta options or the IncludeMeta or ExcludeMeta variables.) meta names can later be queried against specifically using search++(1).

for ID3v1.x, the recommended fields to be indexed are: album, artist, comments, genre, and title.

for ID3v2.2, the recommended text fields (with reassignments) to be indexed are: com=comments, tal=album, tcm=composer, tco=genre, tcr=copyright, ten=encoder, txt=lyricist, tt1=content, tt2=title, tt3=subtitle, ipl=musicians, tot=original-title, tol=original-lyricist, toa=original-artist, tp1=artist, tp2=performers, tp3=conductor, tpb=publisher, txx=user, slt=lyrics, and ult=lyrics.

for ID3v2.4, the recommended text fields (with reassignments) to be indexed are: comm=comments, talb=album, tcom=composer, tcon=genre, tcop=copyright, tenc=encoder, text=lyricist, tipl=people, tit1=content, tit2=title, tit3=subtitle, tmcl=musicians, tmoo=mood, toal=original-title, toly=original-lyricist, tope=original-artist, town=owner, tpe1=artist, tpe2=performers, tpe3=conductor, tpub=publisher, tsst=set-subtitle, txxx=user, user=terms, sylt=lyrics, and uslt=lyrics.

ID3v2.3 is the same as 2.4 except replace tmcl=musicians with ipls=musicians.

  1. For ID3v2.x, any text fields that are compressed get uncompressed before indexing. we're not indexing the zip file bestie, we're indexing the actual words ๐Ÿ’…
  2. Also for ID3v2.x: Unicode text encoded in either UTF-8 or UTF-16 (big-endian or little-endian, both valid, we don't play favourites) gets decoded before indexing. no cursed garbled bytes in your index, periodt โœจ

๐ŸŽ€ LaTeX Module ๐ŸŽ€

LaTeX files get extra special treatment on top of the usual stuff. The tea is:

  • If a \title command is found within the first TitleLines lines of the file (default is 12), then the value of that title is what gets stored in the generated index file as the file's title โ€” not the file's name. ๐Ÿซถ (Every non-space whitespace character in the title gets converted to a space, and leading and trailing spaces get removed.)

๐Ÿ’Œ Mail Module ๐Ÿ’Œ

Mail and news files also get extra processing. Here's the whole rundown, bestie, no skipping:

  1. If a Subject header is found within the first TitleLines lines of the file (default is 12), then the value of that subject is what gets stored in the generated index file as the file's title, instead of the file's name. ๐Ÿฆ‹ (Every non-space whitespace character in the title is converted to a space; leading and trailing spaces are removed.)
  2. Words that are the value of a header get indexed associated with the header name as a meta name. So like, the words live under the header's name, iykyk.
โš ๏ธ BUT โ€” if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words specified as the value of the header are still indexed, they're just not associated with the header. still in the index, just no longer holding hands with its header.

(See also the -m, --meta, -M, and --no-meta options or the IncludeMeta or ExcludeMeta variables.) Meta names can later be queried against specifically using search++(1). ๐Ÿ‘‘

The recommended headers to be indexed are: Bcc, Cc, Comments, Content-Description, From, Keywords, Newsgroups, Resent-To, Subject, and To.

  1. MIME attachments get indexed. yes, the attachments. all of them. ๐Ÿ’€
  2. Text that's in the text/enriched content type gets converted to plain text before indexing.
  3. Text that's encoded as either quoted-printable or base-64 gets decoded before indexing.
  4. Unicode text encoded in either the UTF-7 or UTF-8 character set gets decoded before indexing.
  5. Text in vCards gets indexed in a way where the values of types (fields) are associated with those types as meta names. ๐Ÿชช
โš ๏ธ Same caveat as before: if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words specified as the value of types are still indexed, but not associated with the types.

The recommended vCard types (with reassignments) to be indexed are: adr=address, categories, class, label=address, fn=name, nickname, note, org, role, and title.


Okay so real talk ๐Ÿซถ โ€” indexing mail and news files is most effective only when there's exactly one message per file. Usenet news files are usually like that, but mail files? not so much. Mail files, e.g., mailboxes, are usually made up of multiple messages. ๐Ÿ˜ญ Such files would need to be split up into files of individual messages prior to indexing, because there's literally no point in indexing a single mailbox: every single search result would return a rank of 100 for the same file. like, every. single. one. 100, 100, 100. That's why the splitmail++(1) utility is included in the SWISH++ distribution. ๐Ÿงš

๐Ÿ“– Manual Module ๐Ÿ“–

Unix manual page files get extra processing too. Here's what goes down:

  1. If a NAME section heading macro (.SH) is found within the first TitleLines lines of the file (default is 12), then the contents of the next line get stored in the generated index file as the file's title โ€” instead of the file's name. ๐ŸŒธ (Every non-space whitespace character in the title is converted to a space; leading and trailing spaces, plus backslash sequences such as \f2, are removed.)
  2. Words that are in a section get indexed associated with the name of that section as a meta name.
โš ๏ธ You already know what's coming: if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words in a section are still indexed, but not associated with the section heading.

Spaces in multi-word section headings get converted to dashes โ€” for example, see also'' becomes see-also'' as a meta name. ๐ŸŽ€ (See also the -m, --meta, -M, and --no-meta options or the IncludeMeta or ExcludeMeta variables.) Meta names can later be queried against specifically using search++(1).

The recommended sections to be indexed are: AUTHOR, BUGS, CAVEATS, DESCRIPTION, DIAGNOSTICS, ENVIRONMENT, ERRORS, EXAMPLES, EXIT-STATUS, FILES, HISTORY, NAME, NOTES, OPTIONS, RETURN-VALUE, SEE-ALSO, SYNOPSIS, and WARNINGS. ๐Ÿ‘‘

๐ŸŒˆ RTF Module ๐ŸŒˆ

This module is the low-effort icon of the group: it simply indexes rich text format files without all the formatting commands. no \rtf nonsense in your search results, just the words. slay. ๐Ÿ’…

๐Ÿ’… OPTIONS ๐Ÿ’…

okay bestie so options are giving two flavors: short ones start with a single - and long ones start with a --. and yes, a lone - or -- all by itself is the "we're done with options now" signal โ€” explicit vibes only. you can mix short and long however you want.

long option names can be abbreviated as long as the abbreviation is unambiguous โ€” like, no two options fighting over the same prefix, iykyk.

short option arguments (the rules) ๐ŸŽ€

for a short option that takes an argument, the argument is either the rest of the characters in that same option (if there are any), or, if not, it comes from the next option โ€” unless that next thing starts with a -.

short options with no arguments can be grouped together (but the last one in the group is allowed to take an argument), so like -lrv4 is literally the same as -l -r -v4. iconic.

long option arguments (the rules) ๐ŸŒธ

for a long option that takes an argument, the argument is either the characters after a =, if there are any, or, if not, it comes from the next option โ€” unless that next thing starts with a -.


  • -?, --help โ€” prints the usage (a.k.a. the "help" message) and then exits. your lil emergency exit ๐Ÿ’–
  • -A, --no-assoc-meta โ€” do not associate words with meta names while indexing, and don't store those associations in the generated index file either. so you're trading meta names for lower memory usage and a smaller index file. bestie, you're giving up meta names, no cap.
  • -cf, --config-file=f โ€” the name of the configuration file, f, to use. (default is swish++.conf in the current directory.) here's the tea: a config file is not required โ€” if you don't specify one and the default doesn't exist, none gets used. BUT if you do specify one and it doesn't exist, that's an error. ๐Ÿ’€
  • -Cc, --no-class=c โ€” for HTML or XHTML files only: a class name, c, of an HTML or XHTML element whose text should not be indexed. you can stack multiple -C or --no-class options.
  • -em:p[,p...], --pattern=m:p[,p...] โ€” a module name, m, and a filename pattern (or a set of patterns separated by commas), p, of files to index. case doesn't matter for the module name, but it totally matters for the patterns. multiple -e or --pattern options can be specified.
  • -Ep[,p...], --no-pattern=p[,p...] โ€” a filename pattern (or a set of patterns separated by commas), p, of files not to index. case is significant here. multiple -E or --no-pattern options can be specified.
  • -fn, --word-files=n โ€” the maximum number of files, n, that a word may occur in before it gets discarded for being too frequent. (default is infinity, so like, no limit by default โœจ)
  • -Fn, --files-reserve=n โ€” reserve space for this number of files, n, to start with. more space gets allocated as needed, but with a slight performance penalty. (default is 1000.)
  • -gn, --files-grow=n โ€” grow the space for the reserved number of files, n, when incrementally indexing. the number can be either an absolute number of files or a percentage (when followed by a percent sign %). just like with the -F option, more space gets allocated as necessary, but with a slight performance penalty. (default is 100.)
  • -H, --dump-html โ€” dumps the built-in set of recognized HTML and XHTML elements to standard output and exits. show me the lore ๐Ÿ‘‘
  • -if, --index-file=f โ€” the name of the generated index file, f (for new indexes; default is swish++.index in the current directory), or the old index file when you're doing incremental indexing.
  • -I, --incremental โ€” incrementally add the indexed files and words to an existing index++. the existing index++ is not touched; instead a new index is created that has the same pathname as the existing index++ with .new appended. no takebacks on the original ๐Ÿซถ
  • -l, --follow-links โ€” follow symbolic links during indexing. (default is not to follow them.) heads up: this option is not available under Microsoft Windows since it doesn't support symbolic links.
  • -mm[=n], --meta=m[=n] โ€” the value of a meta name, m, for which words are to be associated when indexed. case is irrelevant. multiple -m or --meta options can be specified.

a meta name can be reassigned when it's followed by a new name, n โ€” meaning that the name n and not m is stored in the generated index file, so queries would use the new name rather than the original.

by default, words associated with all meta names are indexed. specifying at least one meta name via this option changes that: now only the words associated with a member of the set of meta names explicitly specified via one or more -m or --meta options get indexed. choose your fighters carefully ๐Ÿ’…

  • -Mm, --no-meta=m โ€” the value of a meta name, m, for which words are not to be indexed. case is irrelevant. multiple -M or --no-meta options can be specified.
  • -pn, --word-percent=n โ€” the maximum percentage, n, of files a word may occur in before it gets discarded for being too frequent. (default is 100.) and if you want to keep all words regardless? specify 101. periodt.
  • -P, --no-pos-data โ€” do not store word positions in memory during indexing, nor in the generated index file needed to do near searches later during searching. so you're sacrificing near searching for lower memory usage and a smaller index file (approximately 50%). bestie, that's a big trade ๐Ÿ˜ญ
  • -r, --no-recurse โ€” do not recursively index the files in subdirectories. meaning: when a directory is encountered, all the files in that directory get indexed (modulo the filename patterns specified via either the -e, --pattern, -E, or --no-pattern options or the IncludeFile or ExcludeFile variables), but subdirectories encountered are ignored, and therefore the files inside them are not indexed. this option is most useful when you're specifying the directories and files to index via standard input. (default is to index the files in subdirectories recursively.)
  • -sf, --stop-file=f โ€” the name of a file, f, containing the set of stop-words to use instead of the built-in set. whitespace, including blank lines, and characters starting with # and continuing to the end of the line (comments) are ignored.
  • -S, --dump-stop โ€” dumps the built-in set of stop-words to standard output and exits. stop-word lore drop ๐ŸŒˆ
  • -tn, --title-lines=n โ€” the maximum number of lines, n, into a file to look at for a file's title. (default is 12.) bigger numbers slow indexing down.
  • -Td, --temp-dir=d โ€” the path of the directory, d, to use for temporary files. the directory must exist. (default is /tmp for Unix or /temp for Windows.)
โš ๏ธ if your OS mounts swap space on /tmp, then as indexing progresses and more files get created in /tmp, you'll have less swap space, indexing will get slower, and you may run out of memory. if that's your situation, specify a directory on a real filesystem, i.e., one on a physical disk. no cap, save yourself.
  • -vn, --verbosity=n โ€” the verbosity level, n, for printing additional information to standard output during indexing. the verbosity levels, 0โ€“4, are:
    • 0 โ€” no output is generated except for errors. (this is the default.)
    • 1 โ€” only run statistics (elapsed time, number of files, word count) are printed.
    • 2 โ€” directories are printed as indexing progresses.
    • 3 โ€” directories and files are printed with a word-count for each file.
    • 4 โ€” same as 3 but also prints all files that are not indexed and why. the full tea โ˜•
  • -V, --version โ€” prints the version number of SWISH++ to standard output and exits. slay.
  • -Wn, --word-threshold=n โ€” the word count past which partial indices are generated and merged, since all the words are too big to fit into memory at the same time. if you index++ and your machine begins to swap like mad, lower this value. only the super-user can specify a value larger than the compiled-in default. ๐Ÿ‘‘

๐Ÿ“ CONFIGURATION FILE ๐Ÿ“

the following variables can be set in a configuration file. variables and command-line options can be mixed, and the command-line options take priority. (yes, the CLI wins. main character energy.)

  • AssociateMeta โ€” same as -A or --no-assoc-meta
  • ExcludeClass โ€” same as -C or --no-class
  • ExcludeFile โ€” same as -E or --no-pattern
  • ExcludeMeta โ€” same as -M or --no-meta
  • FilesGrow โ€” same as -g or --files-grow
  • FilesReserve โ€” same as -F or --files-reserve
  • FilterAttachment โ€” (see FILTERS in swish++.conf(5).)
  • FilterFile โ€” (see FILTERS in swish++.conf(5).)
  • FollowLinks โ€” same as -l or --follow-links
  • IncludeFile โ€” same as -e or --pattern
  • IncludeMeta โ€” same as -m or --meta
  • Incremental โ€” same as -I or --incremental
  • IndexFile โ€” same as -i or --index-file
  • RecurseSubdirs โ€” same as -r or --no-recurse
  • StopWordFile โ€” same as -s or --stop-file
  • StoreWordPositions โ€” same as -P or --no-pos-data
  • TempDirectory โ€” same as -T or --temp-dir
  • TitleLines โ€” same as -t or --title-lines
  • Verbosity โ€” same as -v or --verbosity
  • WordFilesMax โ€” same as -f or --word-files
  • WordPercentMax โ€” same as -p or --word-percent
  • WordsNear โ€” same as -n or --near
  • WordThreshold โ€” same as -W or --word-threshold

โœจ EXAMPLES โœจ

These are the recipes bestie, and they're all assuming you already cd'd into your web server's document root before you start indexing. Don't skip that part, it's giving prerequisite energy ๐Ÿ’…


๐Ÿ–ฅ๏ธ Unix Command-Lines

Wanna index every HTML and text file on a web server? Say less:

index++ -v3 -e 'html:*.*htm*' -e 'text:*.txt' .

Wanna index everything that's not hiding inside a directory called CVS? Pipe it in, queen:

find . -name CVS -prune -o -type f -a -print | index++ -e 'html:*.*htm*' -

๐ŸชŸ Windows Command-Lines

Okay so when you're in the Windows command interpreter, single quotes around filename patterns simply do not work. You must use double quotes:

index++ -v3 -e "html:*.*htm*" -e "text:*.txt" .

And just so we're clear: this is a Windows problem, not an SWISH++ problem. Not SWISH++'s fault, she's innocent ๐Ÿ™ˆ (Also, double quotes work fine under Unix too, so if you want one blessed syntax forever, there's your answer.)


๐ŸŽ€ Using CLASS Attributes to Index HTML Selectively

Sometimes an HTML or XHTML doc has chunks you truly do not want indexed. Classic example: every page on a site has a nav menu like this:

<SELECT NAME="menu">
  <OPTION>Home
  <OPTION>Automotive
  <OPTION>Clothing
  <OPTION>Hardware
</SELECT>

โ€ฆor a shared header and footer. Ordinarily those words get indexed on every single page, which means they'd end up so frequent they'd just get thrown out anyway. Waste of energy tbh.

But here's the slay: you can name one or more class names using either the -C / --no-class option or the ExcludeClass variable, and then any HTML or XHTML element belonging to one of those classes will have its text left alone โ€” nothing gets indexed up to the tag that ends it.

So give a class the name no_index, and that menu above becomes:

<SELECT NAME="menu" CLASS="no_index">

and boom โ€” everything up to the </SELECT> tag is not indexed. Periodt.

Now for an HTML element that has an optional end tag (hi, <P> element ๐Ÿ’–), the text up to whatever tag ends it is not indexed โ€” and that "whatever tag" can be the element's own end tag or a tag from some other element that implicitly ends it. Watch:

<P CLASS="no_index">
This was the poem that Alice read:
<BLOCKQUOTE>
  <B>Jabberwocky</B><BR>
  `Twas brillig, and the slithy toves<BR>
  Did gyre and gimble in the wabe;<BR>
  All mimsy were the borogoves,<BR>
  And the mome raths outgrabe.
</BLOCKQUOTE>

The <BLOCKQUOTE> tag implicitly ends the <P> element โ€” as do all block-level elements, just so you know โ€” so the only text above that is not indexed is: ``This was the poem that Alice read.''

And for an HTML or XHTML element that has no end tag at all? Only the text inside the start tag itself stays unindexed. Behold:

<IMG SRC="home.gif" ALT="Home" CLASS="no_index">

The word ``Home'' will not be indexed โ€” even though it absolutely would have been if that CLASS attribute weren't sitting there. Iconic behavior, honestly.


๐Ÿงš Filters

(Go look at Filters under EXAMPLES in swish++.conf(5) โ€” they live over there.)


๐Ÿ’€ EXIT STATUS ๐Ÿ’€

It bails out with one of these bad boys:

  • 0 โ€” Success. We ate, we left no crumbs ๐ŸŽ‰
  • 1 โ€” Error in configuration file.
  • 2 โ€” Error in command-line options.
  • 10 โ€” Unable to open temporary file.
  • 11 โ€” Unable to write index file.
  • 12 โ€” Unable to write temporary file.
  • 13 โ€” Root-only operation attempted.
  • 30 โ€” Unable to read stop-word file.
  • 40 โ€” Unable to read index file.
  • 127 โ€” Internal error. ๐Ÿ˜ญ

โš ๏ธ CAVEATS โš ๏ธ

  1. Generated index files are machine-dependent โ€” that means the size of data types and byte order matter. No porting these around like they're nothing, bestie.
  2. The word-determination heuristics are heavily geared for English. Using SWISH++ as-is to index and search++ files in non-English languages is not recommended. Fr fr, don't.
  3. Unless it says otherwise somewhere up above, the character encoding always used is ISO 8859-1 (Latin 1). Any character encodings specified inside HTML or XHTML files are ignored. Ignored!! ๐Ÿ™ˆ
  4. An e-mail message can have both an encoding and a non-ASCII or non-ISO-8859-1 charset at the same time โ€” like base64-encoded UTF-8. (In practice this specific case should never happen, since UTF-7 should be used instead; but you get the idea.)

But handling an encoding and such a charset simultaneously is problematic, which is why an e-mail message or attachment can have either an encoding or a non-ASCII or a non-ISO-8859-1 character set โ€” but not both. If it somehow does, the encoding takes precedence. Encoding wins, always, no negotiation.


๐Ÿ“ FILES ๐Ÿ“

  • swish++.conf โ€” default configuration file name
  • swish++.index โ€” default index file name

๐ŸŒˆ ENVIRONMENT ๐ŸŒˆ

  • TMPDIR โ€” If it's set, this is the default path of the directory used for temporary files. The directory must exist. This gets superseded by either the -T / --temp-dir option or the TempDirectory variable.

๐Ÿ‘‘ SEE ALSO ๐Ÿ‘‘

And now the reading list, because this page is nothing if not well-cited ๐ŸŽ“

  • Tim Berners-Lee. ``The text/enriched MIME Content-type,'' Request for Comments 1563, Network Working Group of the Internet Engineering Task Force, January 1994.
  • David H. Crocker. ``Standard for the Format of ARPA Internet Text Messages,'' Request for Comments 822, Department of Electrical Engineering, University of Delaware, August 1982.
  • Frank Dawson and Tim Howes. ``vCard MIME Directory Profile,'' Request for Comments 2426, Network Working Group of the Internet Engineering Task Force, September 1998.
  • Ned Freed and Nathaniel S. Borenstein. ``Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies,'' Request for Comments 2045, RFC 822 Extensions Working Group of the Internet Engineering Task Force, November 1996.
  • David Goldsmith and Mark Davis. ``UTF-7, a mail-safe transformation format of Unicode,'' Request for Comments 2152, Network Working Group of the Internet Engineering Task Force, May 1997.
  • International Standards Organization. ISO 8859-1: Information Processing -- 8-bit single-byte coded graphic character sets -- Part 1: Latin alphabet No. 1, 1987.
  • --. ISO 8879: Information Processing -- Text and Office Systems -- Standard Generalized Markup Language (SGML), 1986.
  • --. ISO/IEC 9945-2: Information Technology -- Portable Operating System Interface (POSIX) -- Part 2: Shell and Utilities, 1993.
  • Leslie Lamport. LaTeX: A Document Preparation System, 2nd ed., Addison-Wesley, Reading, MA, 1994.
  • Martin Nilsson. ID3 tag version 2, March 1998.
  • --. ID3 tag version 2.3.0, February 1999.
  • --. ID3 tag version 2.4.0 - Main Structure, November 2002.
  • --. ID3 tag version 2.4.0 - Native Frames, November 2002.
  • Steven Pemberton, et al. XHTML 1.0: The Extensible HyperText Markup Language, World Wide Web Consortium, January 2000.
  • Dave Raggett, Arnaud Le Hors, and Ian Jacobs. ``On SGML and HTML: SGML constructs used in HTML: Entities,'' HTML 4.0 Specification, ยง3.2.3, World Wide Web Consortium, April 1998.
  • --. ``The global structure of an HTML document: The document head: The title attribute,'' HTML 4.0 Specification, ยง7.4.3, World Wide Web Consortium, April 1998.
  • --. ``The global structure of an HTML document: The document head: Meta data,'' HTML 4.0 Specification, ยง7.4.4, World Wide Web Consortium, April 1998.
  • --. ``The global structure of an HTML document: The document body: Element identifiers: the id and class attributes,'' HTML 4.0 Specification, ยง7.5.2, World Wide Web Consortium, April 1998.
  • --. ``Tables: Elements for constructing tables: The TABLE element,'' HTML 4.0 Specification, ยง11.2.1, World Wide Web Consortium, April 1998.
  • --. ``Objects, Images, and Applets: Generic inclusion: the OBJECT element,'' HTML 4.0 Specification, ยง13.3, World Wide Web Consortium, April 1998.
  • --. ``Objects, Images, and Applets: How to specify alternate text,'' HTML 4.0 Specification, ยง13.8, World Wide Web Consortium, April 1998.
  • --. ``Index of Elements,'' HTML 4.0 Specification, World Wide Web Consortium, April 1998.
  • Marcin Sawicki, et al. Ruby Annotation, World Wide Web Consortium, April 2001.
  • The Unicode Consortium. ``Encoding Forms,'' The Unicode Standard 3.0, ยง2.3, Addison-Wesley, 2000.
  • Francois Yergeau. ``UTF-8, a transformation format of ISO 10646,'' Request for Comments 2279, Network Working Group of the Internet Engineering Task Force, January 1998.

๐Ÿ’œ AUTHOR ๐Ÿ’œ

Paul J. Lucas <pauljlucas@mac.com> โ€” the actual main character, no notes ๐Ÿซถ


Okay and that's the whole page!! You made it to the end, I'm so proud of you ๐Ÿฅน This little masterpiece is from SWISH++, dated March 25, 2004 โ€” and that's index++(1), signed, sealed, delivered. Go index something, bestie ๐Ÿ’…โœจ
๐Ÿ“ฆ where this page comes from