INDEXER(1)

✨ INDEXER(1) ✨

okay bestie, gather round 🎀 this is the manual page for indexer, from the Sphinxsearch crew. It's a whole lot, so we're doing it in parts — this is part 1 and honestly? It's giving main character energy. Let's go. 💅

---

🌸 NAME 🌸

indexer — the Sphinxsearch fulltext index generator. She makes the indexes. That's the job. Periodt. 👑

---

🦋 SYNOPSIS 🦋

indexer [--config CONFIGFILE] [--rotate] [--noprogress | --quiet] [--all
        | INDEX | ...]

indexer --buildstops OUTPUTFILE COUNT [--config CONFIGFILE]
        [--noprogress | --quiet] [--all | INDEX | ...]

indexer --merge MAIN_INDEX DELTA_INDEX [--config CONFIGFILE] [--rotate]
        [--noprogress | --quiet]

---

💖 DESCRIPTION 💖

Sphinx is a whole collection of programs whose entire goal in life is providing high quality fulltext search. Iconic behaviour, no notes. ✨

indexer is the first of the two principle tools that make up Sphinx. You can call her straight from the command line, or tuck her inside a bigger script — either way, she is solely responsible for gathering up the data that's gonna be searchable. She's the one doing the actual work, fr fr. 😤

The calling syntax for indexer goes like this:

$ indexer [OPTIONS] [indexname1 [indexname2 [...]]]

So basically, you list out the different possible indexes (the ones you're gonna make available to search later) inside sphinx.conf. Then when you call indexer, the absolute minimum you gotta do is tell her which index (or indexes) you want indexed. Communication is key, bestie. 🫶

Say your sphinx.conf had details on 2 indexes, mybigindex and mysmallindex — you could totally do this:

$ indexer mybigindex
$ indexer mysmallindex mybigindex

Inside the config file, sphinx.conf, you specify one or more indexes for your data. You might call indexer to reindex just one of them, ad-hoc style, or you can tell her to process all the indexes. You're not limited to just one, or all-at-once — you can always pick some lil combination of the available indexes. Mix and match, it's your world. 🌈

---

✨ OPTIONS ✨

Most of indexer's options live in the configuration file, but there are some you might need to drop on the command line too, because they can affect how the indexing operation actually goes down. Here they are:

• --all — Tells indexer to update every single index listed in sphinx.conf, instead of you listing them out one by one. Super useful in small configurations, or for cron-type / maintenance jobs where the entire index set gets rebuilt each day, or week, or whatever period is best for you. 🗓️

Example usage:

$ indexer --config /home/myuser/sphinx.conf --all

• --buildstops outfile.txt NUM — Reviews the index source, as if it were indexing the data, and produces a list of the terms that are being indexed. In other words, it gives you a list of all the searchable terms that are becoming part of the index. Note: it does not update the index in question — it simply processes the data 'as if' it were indexing, including running queries defined with sql_query_pre or sql_query_post. 📝 outputfile.txt will contain the list of words, one per line, sorted by frequency with most frequent first. NUM specifies the maximum number of words that will be listed; if it's sufficiently large to encompass every word in the index, only that many words will be returned. Such a dictionary list could be used for client application features around "Did you mean..." functionality, usually in conjunction with --buildfreqs, below. 💅

Example:

$ indexer myindex --buildstops word_freq.txt 1000

This would produce a document in the current directory, word_freq.txt, with the 1,000 most common words in myindex, ordered by most common first. Note that the file will pertain to the last index indexed when specified with multiple indexes or --all (i.e. the last one listed in the configuration file). 🫠

• --buildfreqs — Used in pair with --buildstops (and it gets ignored if --buildstops isn't specified). Since --buildstops provides the list of words used within the index, --buildfreqs adds the quantity present in the index — which is useful for establishing whether certain words should be considered stopwords if they're too prevalent. It'll also help with developing "Did you mean..." features, where you can see how much more common a given word is compared to another, similar one. 📊

Example:

$ indexer myindex --buildstops word_freq.txt 1000 --buildfreqs

This would produce the word_freq.txt as above, except after each word would be the number of times it occurred in the index in question. Slay.

• --config CONFIGRILE, -c CONFIGFILE — Use the given file as configuration. Normally she'll look for sphinx.conf in the installation directory (e.g. /usr/local/sphinx/etc/sphinx.conf if installed into /usr/local/sphinx), followed by the current directory you're in when calling indexer from the shell. This is most of use in shared environments where the binary files are installed somewhere like /usr/local/sphinx/ but you want to give users the ability to make their own custom Sphinx set-ups — or if you want to run multiple instances on a single server. In cases like those you could let them create their own sphinx.conf files and pass them to indexer with this option. 🗂️

For example:

$ indexer --config /home/myuser/sphinx.conf myindex

• --dump-rows FILE — Dumps rows fetched by SQL source(s) into the specified file, in a MySQL compatible syntax. The resulting dumps are the exact representation of data as received by indexer, and they help to repeat indexing-time issues. 🐛 (Reproducing bugs? We love a debugging queen.)
• --merge DST-INDEX SRC-INDEX — Physically merge together two indexes. For example, if you've got a main+delta scheme, where the main index rarely changes but the delta index is rebuilt frequently, --merge would be used to combine the two. The operation moves from right to left — the contents of SRC-INDEX get examined and physically combined with the contents of DST-INDEX, and the result is left in DST-INDEX. In pseudo-code, it might be expressed as: DST-INDEX += SRC-INDEX ➕

An example:

$ indexer --merge main delta --rotate

In the above example, where the main is the master, rarely modified index, and delta is the less frequently modified one, you might use the above to call indexer to combine the contents of the delta into the main index and rotate the indexes. 💫

• --merge-dst-range ATTR MIN MAX — Run the filter range given upon merging. Specifically, as the merge is applied to the destination index (as part of --merge, and it's ignored if --merge isn't specified), indexer will also filter the documents ending up in the destination index — and only documents that pass through the filter given will end up in the final index. This could be used for example in an index where there's a 'deleted' attribute, where 0 means 'not deleted'. Such an index could be merged with:

$ indexer --merge main delta --merge-dst-range deleted 0 0

Any documents marked as deleted (value 1) would be removed from the newly-merged destination index. 🗑️ It can be added several times to the command line, to add successive filters to the merge — and all of them must be met in order for a document to become part of the final index. No shortcuts, bestie.

• --merge-killlists, --merge-klists — Used in pair with --merge. Usually when merging, indexer uses the kill-list of the source index (i.e., the one which is merged into) as the filter to wipe out the matching docs from the destination index. At the same time, the kill-list of the destination itself isn't touched at all. When using --merge-killlists (or its shorter form --merge-klists), the indexer will not filter the dst-index docs with the src-index killlist — instead it will merge their kill-lists together, so the final result index will have the kill-list containing the merged source kill-lists. 💀🔪
• --noprogress — Don't display progress details as they occur; instead, the final status details (such as documents indexed, speed of indexing and so on) are only reported at completion of indexing. In instances where the script is not being run on a console (or 'tty'), this will be on by default. 🖥️

Example usage:

$ indexer --rotate --all --noprogress

• --print-queries — Prints out SQL queries that indexer sends to the database, along with SQL connection and disconnection events. That's useful for diagnosing and fixing problems with SQL sources. 🔍
• --quiet — Tells indexer not to output anything, unless there's an error. Again, most used for cron-type or other script jobs where the output is irrelevant or unnecessary — except in the event of some kind of error. 🤫

Example usage:

$ indexer --rotate --all --quiet

• --rotate — Used for rotating indexes. Unless you're in the situation where you can take the search function offline without troubling users, you will almost certainly need to keep search running whilst indexing new documents. --rotate creates a second index, parallel to the first (in the same place, simply including .new in the filenames). Once complete, indexer notifies searchd by sending the SIGHUP signal, and searchd will attempt to rename the indexes (renaming the existing ones to include .old and renaming the .new to replace them), and then start serving from the newer files. Depending on the setting of seamless_rotate, there may be a slight delay in being able to search the newer indexes. ⏳

Example usage:

$ indexer --rotate --all

• --sighup-each — Useful when you're rebuilding many big indexes and want each one rotated into searchd as soon as possible. With --sighup-each, indexer will send a SIGHUP signal to searchd after successfully completing the work on each index. (The default behavior is to send a single SIGHUP after all the indexes were built.) 📣
• --verbose — Guarantees that every single row that caused problems indexing (duplicate, zero, or missing document ID; or file field IO issues; etc) will be reported. By default this option is off, and problem summaries may be reported instead. 🔎

---

💜 AUTHOR 💜

Andrey Aksenoff (shodan@sphinxsearch.com). This manual page is written by Alexey Vinogradov (klirichek@sphinxsearch.com), using the one written by Christian Hofstaedtler ch+debian-packages@zeha.at for the Debian system (but it may be used by others). Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 2 any later version published by the Free Software Foundation. 📜

On Debian systems, the complete text of the GNU General Public License can be found in /usr/share/common-licenses/GPL.

---

✨ SEE ALSO ✨

okay so bestie, when you're done here and you want the whole ✨cinematic universe✨, go read these too:

• searchd(1) — the daemon, the main character 👑
• search(1) — the actual searching bit 🔍
• indextool(1) — for when your indexes need a lil checkup 🩺
• spelldump(1) — spelling things out, literally ✨

---

Sphinx and it's programs are documented fully by the Sphinx reference manual available in /usr/share/doc/sphinxsearch.

like, everything, fully, no cap — that's where the deep lore lives 📚💜

---

and that's a wrap on INDEXER(1), bestie!! 🎀 this lil icon is 2.2.11-release, dated 07/19/2016 — so if you're reading this in the future, yeah, she's vintage, and she still ate 💅✨ go index something gorgeous, ily, bye!! 🦋💖