SWISH-FAQ(1)

📦

version 2.4.7 · 2 versions

2.4.7 · debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking (latest)
2.4.7 · alpine 3.22, alpine 3.23, alpine 3.24

, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

🎀 SWISH-FAQ(1) — SWISH-E Documentation 🎀

✨ NAME ✨

SWISH-FAQ — the Swish-e FAQ. Answers to common questions, bestie 🫶

✨ OVERVIEW ✨

okay so this is the list of questions everyone keeps asking, and the answers 💅 Please read through it before you post anything on the Swish-e discussion list. Trust me, your future self will thank you 🌸

🌈 General Questions 🌈

🦋 What is Swish-e?

Swish-e is Simple Web Indexing System for Humans - Enhanced ✨ With it, you can quickly and easily index directories of files or remote web sites, then search the generated indexes for words and phrases. Main character energy 💖

💅 So, is Swish-e a search engine?

Well, yes 💁‍♀️ Probably the most common use of Swish-e is to provide a search engine for web sites. The Swish-e distribution includes CGI scripts that can be used with it to add a search engine for your web site. The CGI scripts can be found in the example directory of the distribution package. See the README file for information about the scripts.

But Swish-e can also index all sorts of data, like email messages, data stored in a relational database management system, XML documents, or documents such as Word and PDF documents -- or any combination of those sources at the same time 🔥 Searches can be limited to fields or MetaNames within a document, or limited to areas within an HTML document (e.g. body, title). Programs other than CGI applications can use Swish-e, as well.

🍓 Should I upgrade if I'm already running a previous version of Swish-e?

Okay, the answer is basically yes 😭 A large number of bug fixes, feature additions, and logic corrections were made in version 2.2. Indexing speed also got drastically improved (reports of indexing times changing from four hours to 5 minutes!!), and major parts of the indexing and search parsers were rewritten. There's better debugging options, enhanced output formats, more document meta data (e.g. last modified date, document summary), options for indexing from external data sources, and faster spidering, just to name a few changes. (See the CHANGES file for more information.)

Since so much effort went into version 2.2, support for previous versions will probably be limited 🥺

🌸 Are there binary distributions available for Swish-e on platform foo?

Foo? Well, yes, there are some binary distributions available 🧚 Please see the Swish-e web site for a list at http://swish-e.org/.

In general, it is recommended that you build Swish-e from source, if possible 💜

⭐ Do I need to reindex my site each time I upgrade to a new Swish-e version?

At times it might not strictly be necessary, but since you don't really know if anything in the index has changed, it is a good rule to reindex 🫶

🔥 What's the advantage of using the libxml2 library for parsing HTML?

Swish-e may be linked with libxml2, a library for working with HTML and XML documents. Swish-e can use libxml2 for parsing HTML and XML documents.

The libxml2 parser is a better parser than Swish-e's built-in HTML parser. It offers more features, and it does a much better job at extracting the text out of a web page 😤 In addition, you can use the ParserWarningLevel configuration setting to find structural errors in your documents that could (and would, with Swish-e's HTML parser) cause documents to be indexed incorrectly.

Libxml2 is not required, but it is strongly recommended for parsing HTML documents. It's also recommended for parsing XML, since it offers many more features than the internal Expat xml.c parser.

The internal HTML parser has limited support and does have a number of bugs 💀 For example, HTML entities may not always be correctly converted, and properties do not have entities converted. The internal parser tends to get confused when invalid HTML is parsed, where the libxml2 parser doesn't get confused as often. The structure is better detected with the libxml2 parser.

If you are using the Perl module (the C interface to the Swish-e library), you may wish to build two versions of Swish-e: one with the libxml2 library linked into the binary, and one without. Then build the Perl module against the library without the libxml2 code. This is to save space in the library. Hopefully, the library will someday soon be split into indexing and searching code (volunteers welcome 🙈).

🧚 Does Swish-e include a CGI interface?

Yes. Kind of. 💅

There are two example CGI scripts included, swish.cgi and search.cgi. Both are installed at $prefix/lib/swish-e.

Both require a bit of work to set up and use. swish.cgi is probably what most people will want to use, since it contains more features. search.cgi is for those who want to start with a small script and customize it to fit their needs 🌈

An example of using swish.cgi is given in the INSTALL man page, and in the swish.cgi documentation. Like, it's often the case that it will be easier to use if you read the documentation first 📖

Please be careful with CGI scripts found on the Internet for use with Swish-e. Some are not secure 🚨

The included example CGI scripts were designed with security in mind. Regardless, you are encouraged to have your local Perl expert review them (and all other CGI scripts you use) before putting them into production. This is just a good policy to follow 🫶

🛡️ How secure is Swish-e?

We know of no security issues with using Swish-e. Careful attention has been paid to common security problems, such as buffer overruns, when programming Swish-e 💖

The most likely security issue with Swish-e is when it's run via a poorly written CGI interface. This is not limited to CGI scripts written in Perl, since it's just as easy to write an insecure CGI script in C, Java, PHP, or Python. A good source of information is included with the Perl distribution. Type perldoc perlsec at your local prompt for more information. Another must-read document is located at http://www.w3.org/Security/faq/wwwsf4.html.

Note that there are many free yet insecure and poorly written CGI scripts available -- even some designed for use with Swish-e. Please carefully review any CGI script you use. Free is not such a good price when you get your server hacked... 💀

🚫 Should I run Swish-e as the superuser (root)?

No. Never. 🙅‍♀️

📁 What files does Swish-e write?

Swish writes the index file, of course 🍓 This is specified with the IndexFile configuration directive or by the -f command line switch.

The index file is actually a collection of files, but they all start with the file name specified with the IndexFile directive or the -f command line switch.

For example, the file ending in .prop contains the document properties.

When creating the index files, Swish-e appends the extension .temp to the index file names. When indexing is complete, Swish-e renames the .temp files to the index files specified by IndexFile or -f. This is done so that existing indexes remain untouched until it finishes indexing. Protecting your old index? Iconic 👑

Swish-e also writes temporary files in some cases during indexing (e.g. -s http, -s prog with filters), when merging, and when using -e). Temporary files are created with the mkstemp(3) function (with 0600 permission on unix-like operating systems).

The temporary files are created in the directory specified by the environment variables TMPDIR and TMP, in that order. If those are not set, then swish uses the TmpDir configuration setting. Otherwise, the temporary file will be located in the current directory 🦋

📄 Can I index PDF and MS-Word documents?

Yes! You can use a Filter to convert documents while indexing, or you can use a program that "feeds" documents to Swish-e that have already been converted. See "Indexing" below 💅

🌐 Can I index documents on a web server?

Yes, Swish-e gives you two ways to index (spider) documents on a web server. See "Spidering" below 🕸️

Swish-e can retrieve documents from a file system or from a remote web server. It can also execute a program that returns documents back to it. That program can retrieve documents from a database, filter compressed document files, convert PDF files, extract data from mail archives, or spider remote web sites. The possibilities are literally endless ✨

🏷️ Can I implement keywords in my documents?

Yes, Swish-e can associate words with MetaNames while indexing, and you can limit your searches to these MetaNames while searching 💜

In your HTML files, you can put keywords in HTML META tags or in XML blocks.

META tags can have two formats in your source documents:

<META NAME="DC.subject" CONTENT="digital libraries">

And in XML format (which can also be used in HTML documents when using libxml2):

<meta2>
    Some Content
</meta2>

Then, to tell Swish-e about the existence of the meta name in your documents, edit the line in your configuration file:

MetaNames DC.subject meta1 meta2

When searching, you can now limit some or all search terms to that MetaName. For example, to look for documents that contain the word apple and also have either fruit or cooking in the DC.subject meta tag 🍎

🧾 What are document properties?

A document property is typically data that describes the document. For example, properties might include a document's path name, its last modified date, its title, or its size. Swish-e stores a document's properties in the index file, and they can be reported back in search results 📋

Swish-e also uses properties for sorting. You may sort your results by one or more properties, in ascending or descending order 🔢

Properties can also be defined within your documents. HTML and XML files can specify tags (see previous question) as properties. The contents of these tags can then be returned with search results. These user-defined properties can also be used for sorting search results 🌸

For example, if you had the following in your documents:

<meta name="creator" content="accounting department">

and creator is defined as a property (see PropertyNames in SWISH-CONFIG), Swish-e can return "accounting department" with the result for that document 🎀

swish-e -w foo -p creator

Or for sorting:

swish-e -w foo -s creator

🥊 What's the difference between MetaNames and PropertyNames?

MetaNames allow keyword searches in your documents. That is, you can use MetaNames to restrict searches to just parts of your documents 🔍

PropertyNames, on the other hand, define text that can be returned with results, and can be used for sorting 📦

Both use meta tags found in your documents (as shown in the above two questions) to define the text you wish to use as a property or meta name. Lowkey the same vibe, but different jobs 🫶

You may define a tag as both a property and a meta name. For example:

<meta name="creator" content="accounting department">

placed in your documents, and then using configuration settings of:

PropertyNames creator
MetaNames creator

will allow you to limit your searches to documents created by accounting:

swish-e -w 'foo and creator=(accounting)'

okay bestie so this one finds all documents with the word "foo" that ALSO have a creator meta tag containing "accounting" 💅 this is using MetaNames, so it's giving precision ✨

And you can also say:

swish-e -w foo -p creator

which returns all documents with the word "foo", but the results will also include the contents of the "creator" meta tag along with the results 🦋 this is using properties, and it's lowkey so helpful.

You can use properties and meta names at the same time too 🫶:

swish-e -w creator=(accounting or marketing) -p creator -s creator

this one searches only in the "creator" meta name for either "accounting" or "marketing", prints out the contents of the "creator" property, and sorts the results by the "creator" property name 🔥 main character energy, no cap.

(See also the "-x" output format switch in SWISH-RUN. 🌸)

🌈 Can Swish-e index multi-byte characters?

No. 💀 This would take a lot of work to change, sorry bestie.

BUT, Swish-e works with eight-bit characters, so lots of character sets can be used ✨. Heads up though: it does call the ANSI-C tolower() function, which depends on the current locale setting. See locale(7) for more info 🧚.

✨ Indexing ✨

🎀 How do I pass Swish-e a list of files to index?

Currently there's no configuration directive to include a file that contains a list of files to index 😭. But there IS a directive to include another configuration file, so that's the move:

IncludeConfigFile /path/to/other/config

And in "/path/to/other/config" you can say:

IndexDir file1 file2 file3 file4 file5 ...
IndexDir file20 file21 file22

You may also specify more than one configuration file on the command line 💖:

./swish-e -c config_one config_two config_three

Another option is to create a directory with symbolic links to the files you want indexed, and then index just that directory 🦋.

🌸 How does Swish-e know which parser to use?

Swish can parse HTML, XML, and text documents 💜. The parser is set by associating a file extension with a parser using the "IndexContents" directive. You may set the default parser with the "DefaultContents" directive. If a document isn't assigned a parser, it defaults to the HTML parser (HTML2 if built with libxml2) 🫶.

You may use Filters or an external program to convert documents to HTML, XML, or text ✨.

🔥 Can I reindex and search at the same time?

Yes, bestie! 👑 Starting with version 2.2, Swish-e indexes to temporary files and then renames them when indexing is complete. On most systems renames are atomic. BUT, since Swish-e also generates more than one file during indexing, there's a very short window between renaming the various files where the index is out of sync 😬.

Settings in src/config.h control some options related to temporary files and how they're used during indexing 🧚.

💅 Can I index phrases?

Phrases are indexed automatically 🌈. To search for one, just put double quotes around the phrase.

For example:

swish-e -w 'free and "fast search engine"'

🙈 How can I prevent phrases from matching across sentences?

Use the BumpPositionCounterCharacters configuration directive 🍓.

🫠 Swish-e isn't indexing a certain word or phrase.

Okay, this one is a whole thing 😭. There are a number of configuration parameters that control what Swish-e considers a "word", and it has a debugging feature to help pinpoint any indexing problems 🔍.

The configuration file directives (SWISH-CONFIG) "WordCharacters", "BeginCharacters", "EndCharacters", "IgnoreFirstChar", and "IgnoreLastChar" are the main settings Swish-e uses to define a "word". See SWISH-CONFIG and SWISH-RUN for details 💜.

Swish-e also uses compile-time defaults for many settings. These are located in the src/config.h file 🌸.

Use the command line arguments -k, -v, and -T when debugging these problems. Using -T INDEXED_WORDS while indexing will display each word as it gets indexed. You should specify one file when using this feature, since it can generate a LOT of output 💀.

./swish-e -c my.conf -i problem.file -T INDEXED_WORDS

You may also wish to index a single file that contains words that are, or are not, indexing the way you expect, and use -T to output debugging information about the index. A useful command might be:

./swish-e -f index.swish-e -T INDEX_FULL

Once you see how Swish-e is parsing and indexing your words, you can adjust the configuration settings mentioned above to control what words get indexed ✨.

Another useful command might be:

./swish-e -c my.conf -i problem.file -T PARSED_WORDS INDEXED_WORDS

This shows white-spaced words parsed from the document (PARSED_WORDS), and how those words are split up into separate words for indexing (INDEXED_WORDS) 🦋.

🍓 How do I keep Swish-e from indexing numbers?

Swish-e indexes words as defined by the "WordCharacters" setting, as described above. So to avoid indexing numbers, you simply remove digits from the "WordCharacters" setting 🔢.

There are also some settings in src/config.h that control what "words" get indexed. You can configure swish to never index words that are all digits, vowels, or consonants, or that contain more than some consecutive number of digits, vowels, or consonants. In general, you won't need to change these settings 🙅‍♀️.

Also, there's an experimental feature called "IgnoreNumberChars" which lets you define a set of characters that describe a number. If a word is made up of only those characters, it will not be indexed 🧪.

💀 Swish-e crashes and burns on a certain file. What can I do?

This shouldn't happen 😭. If it does, please post the details to the Swish-e discussion list so the developers can reproduce it.

In the meantime, you can use a "FileRules" directive to exclude the particular file name, pathname, or its title 🚫. If there are serious problems indexing certain types of files, they may not have valid text in them (they may be binary files, for instance). You can use NoContents to exclude that type of file 🙈.

Swish-e will issue a warning if an embedded null character is found in a document. This warning is an indication that you are trying to index binary data. If you need to index binary files, try to find a program that will extract the text (e.g. strings(1), catdoc(1), pdftotext(1)).

🙅 How to I prevent indexing of some documents?

When using the file system to index your files, you can use the "FileRules" directive. Other than "FileRules title", "FileRules" only works with the file system (-S fs) indexing method, NOT with -S prog or -S http 🗂️.

If you are spidering a site you have control over, use a robots.txt file in your document root. This is a standard way to exclude files from search engines, and it's fully supported by Swish-e. See http://www.robotstxt.org/ 🤖.

If spidering a website with the included spider.pl program, add any necessary tests to the spider's configuration file. Type <perldoc spider.pl> in the "prog-bin" directory for details, or see the spider documentation on the Swish-e website. Look for the section on callback functions 🕷️.

If you're using the libxml2 library for parsing HTML (which you probably are), you may also use the Meta Robots Exclusion in your documents:

<meta name="robots" content="noindex">

See the obeyRobotsNoIndex directive 🛑.

🌷 How do I prevent indexing parts of a document?

To stop Swish-e from indexing a common header, footer, or navigation bar, AND you're using libxml2 for parsing HTML, you can wrap the text you want ignored in a fake HTML tag and use the "IgnoreMetaTags" directive 🫣. This will generate an error message if "ParserWarningLevel" is set, since it's invalid HTML 🚨.

"IgnoreMetaTags" works with XML documents (and HTML documents when using libxml2 as the parser), but NOT with documents parsed by the text (TXT) parser ❌.

If you're using the libxml2 parser (HTML2 and XML2), you can use these comments in your documents to prevent indexing:

<!-- SwishCommand noindex -->
<!-- SwishCommand index -->

and/or these may be used also:

<!-- noindex -->
<!-- index -->

🛣️ How do I modify the path or URL of the indexed documents.

Use the "ReplaceRules" configuration directive to rewrite path names and URLs 🔁. If you're using the -S prog input method, you may set the path to any string 🪄.

🗄️ How can I index data from a database?

Use the "prog" document source method of indexing 💾. Write a program to extract the data from your database, and format it as XML, HTML, or text. See the examples in the "prog-bin" directory, and the next question 🧚.

📄 How do I index my PDF, Word, and compressed documents?

Swish-e can internally only parse HTML, XML, and TXT (text) files by default, but it can make use of filters that convert other types of files, such as MS Word documents, PDF, or gzipped files, into one of the file types Swish-e understands 📚.

Please see SWISH-CONFIG and the examples in the filters and filter-bin directories for more information 🔍.

See the next question to learn about the filtering options with Swish-e 👉.

🎞️ How do I filter documents?

The term "filter" in Swish-e means converting a document of one type (one that swish-e cannot index directly) into a type that Swish-e can index, namely HTML, plain text, or XML. To add to the confusion, there are a number of ways to accomplish this in Swish-e, so here's a bit of background 🌈.

The FileFilter directive was added to swish first 🕰️. This feature lets you specify a program to run for documents that match a given file extension. For example, to filter PDF files (files that end in .pdf), you can specify the configuration setting of:

FileFilter .pdf pdftotext   "'%p' -"

This says to run the program "pdftotext", passing it the pathname of the file (%p) and a dash (which tells pdftotext to output to stdout). Then, for each .pdf file, Swish-e runs this program and reads in the filtered document from the filter program's output 📥.

This has the advantage of being easy to set up -- a single line in the config file is all it takes to add the filter into Swish-e 💖. But it also has a number of problems 😩. For example, if you use a Perl script to do your filtering, it can be very slow, since the filter script must be run (and thus compiled) for each processed document. This is made worse when using the -S http method, since -S http also uses a Perl script that runs for every URL fetched. Also, when using the -S prog method of input (reading input from a program), using FileFilter means that Swish-e must first read the file in from the external program, and then write the file out to a temporary file before running the filter 🐌.

With -S prog it makes way more sense to filter the document in the program that's fetching it, instead of having swish-e read the file into memory, write it to a temporary file, and then run an external program on that. ✨

The Swish-e distribution comes with a couple of example -S prog programs. spider.pl is a reasonably full-featured web spider with way more options than the -S http method, and it's much faster than running -S http too 🏎️💨

The spider has a Perl configuration file, which is so iconic because you can slide programming logic right into it without ever editing the spider program itself 💅 One bit of logic already in there is a "call-back" function that lets you filter the content. Before the spider hands a fetched web document over to swish for indexing, it can call a simple subroutine in the configuration file and pass along the document and its content type. That subroutine checks the content type and decides whether the document needs filtering.

For example, when processing a document of type application/msword, the call-back subroutine might call the doc2txt.pm Perl module, and a document of type appliation/pdf could use the pdf2html.pm module. The prog-bin/SwishSpiderConfig.pl file shows this usage 🌸

This system works reasonably well, but it also means more work to set up the filters 🥲 First, you have to explicitly check for specific content types and then call the right Perl module. Second, you have to know how each module needs to be called and how each one returns the possibly modified content.

In comes SWISH::Filter. 🦋

To make things easier, the SWISH::Filter Perl module was created. The idea is that there's one interface for filtering all types of documents. Instead of checking for specific types of content, you just pass the content type and the document to the SWISH::Filter module, and it returns a new content type and document if it was filtered. The filters that do the actual work follow a standard interface and work like filter "plug-ins". Adding a new filter just means downloading it to a directory, and no changes are needed to the spider's configuration file. Download a filter for Postscript, and the next time you run indexing, your Postscript files will be indexed 🔥

Since the filters are standardized, hopefully when you need to filter documents of a specific type, there'll already be a filter ready for you to use 🎀

Now, the Perl modules may or may not actually do the conversion themselves. For example, the PDF conversion module calls the pdfinfo and pdftotext programs. Those programs (part of the Xpdf package) must be installed separately from the filters 🧚

The SwishSpiderConfig.pl example spider configuration file shows how to use the SWISH::Filter module for filtering. This file is installed at $prefix/share/doc/swish-e/examples/prog-bin, where $prefix is normally /usr/local on Unix-type machines 🌈

The SWISH::Filter method of filtering can also be used with the -S http method of indexing. By default, the swishspider program (the Perl helper script that fetches documents from the web) will attempt to use the SWISH::Filter module if it can be found in Perl's library path. That path is set automatically for spider.pl, but not for swishspider, because it would slow down a method that's already slow, and spider.pl is recommended over the -S http method 🙈

So all that's required to use this with -S http is setting the @INC array to point to the filter directory. For example, if the swish-e distribution was unpacked into ~/swish-e:

PERL5LIB=~/swish-e/filters swish-e -c conf -S http

That will allow the -S http method to make use of the SWISH::Filter module 💖

Note that if you are not using the SWISH::Filter module, you may wish to edit the swishspider program and disable it using this setting:

    use constant USE_FILTERS  => 0;  # disable SWISH::Filter

This stops the program from trying to use the SWISH::Filter module for every non-text URL that gets fetched. Of course, if you're concerned about indexing speed, you should be using the -S prog method with spider.pl instead of -S http 🏎️

If you're not spidering but still want to use the SWISH::Filter module for filtering, you can use the DirTree.pl program (in $prefix/lib/swish-e). It's a simple program that traverses the file system and uses SWISH::Filter for filtering 📂

Here are two examples of how to run a filter program: one using Swish-e's FileFilter directive, and another using a prog input method program. See the SwishSpiderConfig.pl file for an example of using the SWISH::Filter module.

These filters simply use the program /bin/cat as a filter and only index .html files.

First, using the FileFilter method, here's the entire configuration file (swish.conf):

IndexDir .
IndexOnly .html
FileFilter .html "/bin/cat"   "'%p'"

And index with the command:

swish-e -c swish.conf -v 1

Now, the same thing using the -S prog document source input method and a Perl program called catfilter.pl. You can see it's way more work than the FileFilter method above, but it gives you a place to do additional processing 🧠 In this example, the prog method is only slightly faster. But if you needed a Perl script to run as a FileFilter, then prog will be significantly faster 👑

#!/usr/local/bin/perl -w
use strict;
use File::Find;  # for recursing a directory tree

$/ = undef;
find(
    { wanted => \&wanted, no_chdir => 1, },
    '.',
);

sub wanted {
    return if -d;
    return unless /\.html$/;

    my $mtime  = (stat)[9];

    my $child = open( FH, '-⎪' );
    die "Failed to fork $!" unless defined $child;
    exec '/bin/cat', $_ unless $child;

    my $content = <FH>;
    my $size = length $content;

    print <<EOF;
Content-Length: $size
Last-Mtime: $mtime
Path-Name: $_

EOF

    print <FH>;
}

And index with the command:

swish-e -S prog -i ./catfilter.pl -v 1

This example will probably not work under Windows due to the '-⎪' open. A simple piped open may work just as well 🫶

That is, replace:

my $child = open( FH, '-⎪' );
die "Failed to fork $!" unless defined $child;
exec '/bin/cat', $_ unless $child;

with this:

open( FH, "/bin/cat $_ ⎪" ) or die $!;

Perl will try to avoid running the command through the shell if meta characters aren't passed to the open. See perldoc -f open for more information 🤓

Eh, but I just want to know how to index PDF documents! 😭

See the examples in the conf directory and the comments in the SwishSpiderConfig.pl file. Also see the previous question for the details on filtering. The method you pick will depend on how fast you want to index and how comfy you are with Perl modules 💅

Regardless of the filtering method you use, you'll need to install the Xpdf packages available from http://www.foolabs.com/xpdf/ 📦

I'm using Windows and can't get Filters or the prog input method to work! 💀

Both the -S prog input method and filters use the popen() system call to run the external program. If your external program is, for example, a Perl script, you have to tell Swish-e to run Perl instead of the script. Swish-e will convert forward slashes to backslashes when running under Windows.

For example, you would need to specify the path to Perl like this (assuming this is where Perl lives on your system):

IndexDir e:/perl/bin/perl.exe

Or run a filter like:

FileFilter .foo e:/perl/bin/perl.exe 'myscript.pl "%p"'

It's often easier to just install Linux 🐧

How do I index non-English words? 🌍

Swish-e indexes 8-bit characters only. This is the ISO 8859-1 Latin-1 character set, and it includes many non-English letters (and symbols). As long as they're listed in WordCharacters, they will be indexed.

Actually, you probably can index any 8-bit character set, as long as you don't mix character sets in the same index and don't use libxml2 for parsing (see below) 🧐

The TranslateCharacters directive (SWISH-CONFIG) can translate characters while indexing and searching. You can specify the mapping of one character to another with the TranslateCharacters directive.

TranslateCharacters :ascii7: is a predefined set of characters that will translate eight-bit characters to ascii7 characters. Using the :ascii7: rule will, for example, translate "Ääç" to "aac". This means searching "Çelik", "çelik" or "celik" will all match the same word ✨

Note: When using libxml2 for parsing, parsed documents are converted internally (within libxml2) to UTF-8. This is converted to ISO 8859-1 Latin-1 when indexing. In cases where a string cannot be converted from UTF-8 to ISO 8859-1 (because it contains non 8859-1 characters), the string will be sent to Swish-e in UTF-8 encoding. This will result in some words being indexed incorrectly. Setting ParserWarningLevel to 1 or more will display warnings when UTF-8 to 8859-1 conversion fails ⚠️

Can I add/remove files from an index? 🗂️

Try building swish-e with the --enable-incremental option.

The rest of this FAQ applies to the default swish-e format.

Swish-e currently has no way to add or remove items from its index. But Swish-e indexes so quickly that it's often possible to reindex the entire document set whenever a file needs to be added, modified or removed. If you're spidering a remote site, consider caching documents locally, compressed 🗜️

Incremental additions can be handled a couple of ways, depending on your situation. It's probably easiest to create one main index every night (or every week), then create an index of just the new files between main indexing jobs, and use the -f option to pass both indexes to Swish-e while searching 🫧

You can merge the indexes into one index (instead of using -f), but it's not clear that this has any advantage over searching multiple indexes.

How does one create the incremental index? 🤔

One method is by using the -N switch to pass a file path to Swish-e when indexing. It will only index files that have a last modification date "newer" than the file supplied with the -N switch 🦋

This option has a bit of a glow-up problem 😭 Swish-e has to process every single file in every directory like it's about to index them. The -N test runs dead last, right before the file contents get indexed and after all the other tests on the file are done. And that's all just to find a few new files 💀

Also, if you pass the Swish-e index file as the file given to -N, some files might get added after indexing started but before the index file was written. Then a file could end up not being added to the index at all 🙈

Another option is to keep a parallel directory tree full of symlinks that point to the main files. When a new file is added (or changed) in the main directory tree, you create a symlink to the real file in the parallel tree. Then you just index the symlink directory to generate the incremental index 🦋

This one has the downside that you need a central program that creates the new files and can also create the symlinks. BUT indexing is super fast, since Swish-e only has to look at the files that need indexing 🔥 When you run full indexing, you just unlink (delete) all the symlinks ✨

Both of these methods have issues where files could end up in both indexes, or get left out of an index entirely 😬 Using file locks while indexing, and hash lookups during searches, can help prevent these problems 🫶

💀 I run out of memory trying to index my files.

Okay, indexing can take up a lot of memory, and that's real! Swish-e is extremely fast at indexing, but that speed comes at the cost of memory 🥲

The best answer is to install more memory 💅

Another option is to use the -e switch. This needs less memory, but indexing takes longer because not all the data gets stored in memory while indexing. How much less memory and how much more time you get depends on the documents you're indexing and the hardware you're using ⏳

Here's an example of indexing all .html files in /usr/doc on Linux. This first run is without -e and used about 84M of memory:

270279 unique words indexed.
23841 files indexed.  177640166 total bytes.
Elapsed time: 00:04:45 CPU time: 00:03:19

This one is with -e, and used about 26M of memory:

270279 unique words indexed.
23841 files indexed.  177640166 total bytes.
Elapsed time: 00:06:43 CPU time: 00:04:12

You can also build a bunch of smaller indexes and then merge them together with -M. Using -e while merging will save memory 🧚

Finally, if you do build a bunch of smaller indexes, you can specify more than one index when searching by using the -f switch. Sorting large result sets by a property will be slower when you specify multiple index files while searching 🐢

🗂️ "too many open files" when indexing with `-e` option

Some platforms report "too many open files" when you use the -e economy option. The -e feature uses lots of temporary files (something like 377) plus the index files, and that can go over your system's limits 😵‍💫

Depending on your platform, you may need to set ulimit or unlimit.

For example, under the Linux bash shell:

$ ulimit -n 1024

Or under an old Sparc:

% unlimit openfiles

😤 My system admin says Swish-e uses too much of the CPU!

That's a good thing! That expensive CPU is supposed to be busy 💪

Indexing takes a lot of work. To make indexing fast, most of the work gets done in memory, which cuts down the time Swish-e spends waiting on I/O. But there are two things you can try 🤓

The -e option runs Swish-e in economy mode, which uses the disk to store data while indexing. This makes Swish-e run somewhat slower, but it also uses less memory. Since it writes to disk more often, it spends more time waiting on I/O and less time in CPU. Maybe 🤷‍♀️

The other thing is to simply lower the priority of the job using the nice(1) command:

/bin/nice -15 swish-e -c search.conf

If you're worried about search time, make sure you're using the -b and -m switches to only return a page at a time. If you know your result sets will be large, that you want to return results one page at a time, and that many pages of the same query often get requested, you might be smart to request all the documents on the first request and then cache the results to a temporary file. The perl module File::Cache makes this super simple to set up 🧠

🕷️ Spidering 🕸️

🌐 How can I index documents on a web server?

If possible, use the file system method -S fs to index documents in your web area of the file system. This skips the overhead of spidering a web server and is way faster 🏎️ (-S fs is the default method if -S isn't specified.)

If that's impossible (the web server isn't local, or the documents are dynamically generated), Swish-e gives you two spidering methods. First, there's the http method of indexing, -S http. A bunch of special configuration directives control spidering (see "Directives for the HTTP Access Method Only" in SWISH-CONFIG). A perl helper script (swishspider) is included in the src directory to help with spidering web servers, and there are example configurations for spidering in the conf directory 📚

As of Swish-e 2.2, there's also a general purpose "prog" document source, where a program can feed documents to Swish-e for indexing. A number of example programs live in the "prog-bin" directory, including a program that spiders web servers. The provided spider.pl program is full-featured and easy to customize 💖

The advantage of the "prog" document source over the "http" method is that the program only runs one time. The swishspider.pl program used by the "http" method runs once for every document read from the web server. Forking Swish-e and compiling the perl script can be really expensive, time-wise 😩

The other advantage of the "spider.pl" program is that it's simple and efficient to add filtering (like for PDF or MS Word docs) right into spider.pl's configuration. It also includes features like MD5 checks to prevent duplicate indexing, options to avoid spidering some files, or to index a file but avoid spidering it. And since it's a perl program, there's no limit on the features you can add 👑

🤔 Why does swish report "./swishspider: not found"?

Does the file swishspider exist where the error message shows up? If not, either set the configuration option SpiderDirectory to point to the directory where the swishspider program lives, or place the swishspider program in the current directory when you run swish-e 📂

If you're running Windows, make sure "perl" is in your path. Try typing perl at a command prompt 🪟

If you're not running Windows, make sure the shebang line (the first line of the swishspider program that starts with #!) points to the correct location of perl. Typically this is /usr/bin/perl or /usr/local/bin/perl. Also make sure you have execute and read permissions on swishspider 🔐

The swishspider perl script is only used with the -S http method of indexing.

🐛 I'm using the spider.pl program to spider my web site, but some large files are not indexed.

The "spider.pl" program has a default limit of 5MB file size. You can change this with the max_size parameter setting. See "perldoc spider.pl" for more information 📏

🕵️ I still don't think all my web pages are being indexed.

The spider.pl program has a number of debugging switches and can be pretty verbose about what's happening and why. See "perldoc spider.pl" for instructions 🔎

🙅‍♀️ Swish is not spidering Javascript links!

Swish can't follow links generated by Javascript, since they're generated by the browser and aren't part of the document 🙃

🔀 How do I spider other websites and combine it with my own (filesystem) index?

You can either merge two indexes into a single index with -M, or use -f to specify more than one index while searching.

You'll get better results with the -f method 🏆

🔍 Searching 🔍

🗃️ How do I limit searches to just parts of the index?

If you can identify "parts" of your index by path name, you have two options 🧭

The first option is to index the document path. Add this to your configuration:

MetaNames swishdocpath

Now you can search for words or phrases in the path name:

swish-e -w 'foo AND swishdocpath=(sales)'

That will only find documents with the word "foo" where the file's path contains "sales". It might not work as well as you'd like, though, because both of these paths will match:

/web/sales/products/index.html
/web/accounting/private/sales_we_messed_up.html

You can solve this by searching with a phrase (assuming "/" is not a WordCharacter):

swish-e -w 'foo AND swishdocpath=("/web/sales/")'
swish-e -w 'foo AND swishdocpath=("web sales")'  (same thing)

The second option is a bit more powerful. With the ExtractPath directive, you can use a regular expression to pull out a subset of the path and save it as a separate meta name:

MetaNames department
ExtractPath department regex !^/web/([^/]+).+$!$1/

This says: match a path that starts with "/web/", extract everything after that up to, but not including, the next "/", and save it in variable $1. Then match everything from the "/" onward, and replace the entire matches string with $1. That gets indexed as meta name "department" 🎀

Now you can search like this:

swish-e -w 'foo AND department=sales'

and be sure you'll only match documents in the /www/sales/* path. Note that you can map completely different areas of your file system to the same metaname:

# flag the marketing specific pages
ExtractPath department regex !^/web/(marketing⎪sales)/.+$!marketing/
ExtractPath department regex !^/internal/marketing/.+$!marketing/

# flag the technical departments pages
ExtractPath department regex !^/web/(tech⎪bugs)/.+$!tech/

Finally, if you have something more complicated, use -S prog and write a perl program, or use a filter, to set a meta tag while processing each file 🛠️

🏅 How is ranking calculated?

The swishrank property value is calculated based on which Ranking Scheme (or algorithm) you've selected. In this discussion, any time the word fancy is used, you should consult the actual code for more details. It is open source, after all 🤓

Things you can do to affect ranking 💅

`MetaNamesRank`

You can configure your index to bias certain metaname values more or less than others. See the "MetaNamesRank" configuration option in SWISH-CONFIG.

`IgnoreTotalWordCountWhenRanking`

Set this to 1 (default) or 0 in your config file. See SWISH-CONFIG.

NOTE: You must set this to 0 to use the IDF Ranking Scheme.

✨ structure ✨

Okay bestie, so every term's position in each HTML document gets a structure value based on where the word is hiding 👀. That value artificially inflates the term's frequency in that document 💅. These are defined in config.h:

#define RANK_TITLE             7
#define RANK_HEADER            5
#define RANK_META              3
#define RANK_COMMENTS          1
#define RANK_EMPHASIZED        0

For example, if "foo" shows up in a document's title, the Scheme treats that doc like "foo" appeared 7 extra times. Main character energy 🎀

Every single Scheme shares these vibes 🫶

AND searches

With AND searches, the rank value gets averaged across all the AND'd terms 🧮. Anything inside a set of parentheses () gets averaged as one single term. Ngl this is a known weakness and it's on the TODO list 💀

OR searches

With OR searches, the rank value gets summed, then doubled for each pair of OR'd terms ✨. So docs with multiple OR'd terms end up with higher ranks. Iconic.

scaled rank

Once a document's raw rank score is calculated, a final rank score gets calculated using a fancy log() function 🤓. Then every document is scaled against a base score of 1000. The top-ranked doc will always have a swishrank value of 1000 👑

Here is a brief overview of how the different Schemes work.

Here's a quick tour of the Schemes 🗺️ The number in parentheses after each name is the value you use to invoke that scheme with swish-e -R or RankScheme().

Default (0)

The default ranking scheme looks at how many times a term shows up in a document (frequency), the MetaNamesRank, and the structure value 🍓 The rank can be summarized like this:

DocRank = Sum of ( structure + metabias )

Here's what you'd see with the DEBUG_RANK variable set at compile time:

Ranking Scheme: 0
Word entry 0 at position 6 has struct 7
Word entry 1 at position 64 has struct 41
Word entry 2 at position 71 has struct 9
Word entry 3 at position 132 has struct 9
Word entry 4 at position 154 has struct 9
Word entry 5 at position 423 has struct 73
Word entry 6 at position 541 has struct 73
Word entry 7 at position 662 has struct 73
File num: 1104.  Raw Rank: 21.  Frequency: 8 scaled rank: 30445
 Structure tally:
 struct 0x7 = count of 1 ( HEAD TITLE FILE ) x rank map of 8 = 8

 struct 0x9 = count of 3 ( BODY FILE ) x rank map of 1 = 3

 struct 0x29 = count of 1 ( HEADING BODY FILE ) x rank map of 6 = 6

 struct 0x49 = count of 3 ( EM BODY FILE ) x rank map of 1 = 3

Every word instance starts with a base score of 1. Then, for each instance of your word, a running sum is taken of the structural value of that word position plus any bias you've configured 🧠 In the example above, the raw rank is 1 + 8 + 3 + 6 + 3 = 21.

Check out this line:

struct 0x7 = count of 1 ( HEAD TITLE FILE ) x rank map of 8 = 8

That means there was one instance of our word in the title of the file 📄 Its context was in the <head> tagset, inside the <title>. The <title> is the most specific structure, so it gets the RANK_TITLE score of 7. The base rank of 1 plus the structure score of 7 equals 8. If there had been two instances of this word in the title, the score would've been 8 + 8 = 16.

IDF (1)

IDF stands for Inverse Document Frequency 🤓 That's fancy ranking lingo for taking the total frequency of a term across the entire index into account, on top of the term's frequency in a single document. IDF ranking also uses the relative density of a word in a document to judge how relevant it is. Words that show up more often in a doc make that doc's rank higher, and longer docs don't get weighted higher than shorter ones 📚

The IDF Scheme can be summarized as:

DocRank = Sum of ( density * idf * ( structure + metabias ) )

Here's some output from DEBUG_RANK:

Ranking Scheme: 1
File num: 1104  Word Score: 1  Frequency: 8  Total files: 1451
Total word freq: 108   IDF: 2564
Total words: 1145877   Indexed words in this doc: 562
Average words: 789   Density: 1120    Word Weight: 28716
Word entry 0 at position 6 has struct 7
Word entry 1 at position 64 has struct 41
Word entry 2 at position 71 has struct 9
Word entry 3 at position 132 has struct 9
Word entry 4 at position 154 has struct 9
Word entry 5 at position 423 has struct 73
Word entry 6 at position 541 has struct 73
Word entry 7 at position 662 has struct 73
Rank after IDF weighting: 574321
scaled rank: 132609
 Structure tally:
 struct 0x7 = count of  1 ( HEAD TITLE FILE ) x rank map of 8 = 8

 struct 0x9 = count of  3 ( BODY FILE ) x rank map of 1 = 3

 struct 0x29 = count of  1 ( HEADING BODY FILE ) x rank map of 6 = 6

 struct 0x49 = count of  3 ( EM BODY FILE ) x rank map of 1 = 3

It's pretty similar to the default Scheme, but notice how the total number of files in the index and the total word frequency (as opposed to the document frequency) are both part of the math now 🧮✨

Ranking is a complicated subject 🫠 SWISH-E lets you develop and experiment with more Ranking Schemes using the -R option (from the swish-e command) and the RankScheme (see the API documentation). Experiment and share your findings on the discussion list 🦋

How can I limit searches to the title, body, or comment?

Use the -t switch 🎯

I can't limit searches to title/body/comment.

Or, I can't search with meta names, because all the names are indexed as "plain" 😭

Check the config.h file to see if #define INDEXTAGS is set to 1. If it is, change it to 0, recompile, and index again. When INDEXTAGS is 1, ALL the tags get indexed as plain text. So you index "title", "h1", and so on, AND they lose their indexing meaning 💀 If INDEXTAGS is set to 0, you'll still index meta tags and comments, unless you've said otherwise in the user config file with the IndexComments directive.

Also, check for the UndefinedMetaTags setting in your configuration file 🔍

I've tried running the included CGI script and I get a "Internal Server Error"

Debugging CGI scripts is beyond the scope of this document 🙈 "Internal Server Error" basically means "check the web server's log for an error message," because it could be a bad shebang (#!) line, a missing perl module, an FTP transfer error, or just an error in the program. The CGI script swish.cgi in the example directory has some debugging suggestions. Type perldoc swish.cgi for info.

There are also many, many CGI FAQs out on the Internet. A quick web search should help. As a last resort, you could ask your webadmin for help... 🫠

When I try to view the swish.cgi page I see the contents of the Perl program.

Your web server isn't configured to run the program as a CGI script. This problem is described in perldoc swish.cgi 🌸

How do I make Swish-e highlight words in search results?

Short answer: Use the supplied swish.cgi or search.cgi scripts located in the example directory ✨

Long answer: Swish-e can't do it because it doesn't have access to the source documents when returning results, of course 🙃 But a front-end program of your creation can highlight terms. Your program can open up the source documents and use regular expressions to swap search terms for highlighted or bolded words.

But that'll fail with all but the most simple source documents. For HTML documents, for example, you have to parse the document into words, tags, and comments. A word you want to highlight might span multiple HTML tags, or be a word inside a URL where you want to highlight the entire link text 😩

Perl modules like HTML::Parser and XML::Parser make word extraction possible. Then you need to consider that Swish-e uses settings like WordCharacters, BeginCharacters, EndCharacters, IgnoreFirstChar, and IgnoreLast, char to define a "word." So you can't just assume a string of characters with white space on each side is a word 🧐

Then things like TranslateCharacters and HTML Entities may transform a source word into something else, as far as Swish-e is concerned. Finally, searches can be limited by metanames, so you may need to limit your highlighting to only parts of the source document. Throw phrase searches and stopwords into the mix and you'll see it's not a trivial problem to solve 💀

All hope is not lost though, because Swish-e does give some help 🫶 Using the -H option, it will return the current index (or indexes) settings for WordCharacters (and others) in the headers. Those are what it needs to parse your source documents the same way it did during indexing. It will also return a Parsed Words: header that shows how it parsed the query internally. If you use fuzzy indexing (word stemming, soundex, or metaphone), you'll also need to stem each word in your document before comparing it with the Parsed Words: returned by Swish-e.

The Swish-e stemming code is available by using the Swish-e Perl module (SWISH::API) or the C library (included with the swish-e distribution), or by using the SWISH::Stemmer module on CPAN. Also on CPAN is Text::DoubleMetaphone. Using SWISH::API probably gives you the best stemming support 👑

Do filters effect the performance during search?

No 💖 Filters (FileFilter or via the "prog" method) are only used for building the search index database. During search requests, there are no filter calls.

I have read the FAQ but I still have questions about using Swish-e.

The Swish-e discussion list is the place to go: http://swish-e.org/. Please don't email developers directly. The list is the best place to ask questions 🦋

Before you post, read QUESTIONS AND TROUBLESHOOTING in the INSTALL page. You should also search the Swish-e discussion list archive, which you can find on the swish-e web site 🔍

In short, make sure you include the following when asking for help 🎀

The swish-e version (./swish-e -V)
What you are indexing (and maybe a sample), and the number of files
Your Swish-e configuration file
Any error messages that Swish-e is reporting

📜 Document Info

$Id: SWISH-FAQ.pod 2147 2008-07-21 02:48:55Z karpet $

.

Bestie, that's the whole FAQ 🎀 This page is from version 2.4.7, dated 2009-04-04, and it's SWISH-FAQ(1) forever 💜🌈

📦 where this page comes from

debian bookworm: swish-e 2.4.7-6.1+b2
debian trixie: swish-e 2.4.7-6.3
ubuntu jammy: swish-e 2.4.7-6.1build1
ubuntu noble: swish-e 2.4.7-6.2build3
ubuntu questing: swish-e 2.4.7-7
ubuntu resolute: swish-e 2.4.7-7.1
ubuntu stonking: swish-e 2.4.7-7.2build1
💖 girlified by claude-haiku-5-5 on 2026-10-12: a language model's rewrite of the original, under its terms