RA-INDEX(1)

📦 version 2.12, in ubuntu jammy

💖 RA-INDEX(1) — the glow-up guide 💖

okay bestie, so this is a page from the General Commands Manual, and it's all about ra-index — the indexing half of the Savant search engine, the thing that makes the Remembrance Agent actually remember stuff. grab a boba, we're doing this 🧋✨

✨ NAME ✨

ra-index — index files for use with remembrance agent software 🦋

🌸 SYNOPSIS 🌸

ra-index [--version] [-v] [-d] [-s] <base-dir> <source1> [<source2>]
[...]  [-e <excludee1> [<excludee2>] [...]]

👑 DESCRIPTION 👑

ra-index and ra-retrieve together make up the Savant search engine — an information retrieval engine designed as a back-end for the Remembrance Agent (RA). so basically: you give it a whole pile of your accumulated email, usenet news articles, papers, saved HTML files and other text notes, and the RA goes hunting for the documents that are most relevant to whatever you're doing right now. like, it searches that whole collection for the documents with the highest word-for-word similarity to the text you're currently editing, hoping they'll also have high conceptual similarity and therefore be genuinely useful to your current work. and with the Emacs front-end, those suggestions just keep showing up in a little buffer at the bottom of your window. if one looks good? full text retrieved with a single command. iconic behaviour, no cap. 💅

The Remembrance Agent works in two stages. first, your collection of text documents gets indexed into a database saved in a vector format. once that database exists, the other stage of the RA runs from emacs — it periodically takes a sample of text from the working buffer and finds those documents from the collection that are most similar. it summarizes the top documents in a small emacs window and lets you retrieve the entire text of any one with a keystroke. (see the README file for info on using the Emacs front-end 🫶)

At its core, Savant is a text-retrieval search-engine using a standard TF/iDF algorithm, but it also uses a template system to recognize different kinds of documents and extract various field information. for example, ra-index can recognize subject lines and address information from email files and file that info separately. it can also pull apart file archives into separate documents — e.g. RMAIL files get indexed as separate email documents. and finally, there are filters defined for many document types to strip out extraneous info like HTML tags that might otherwise cause problems in retrieval. all of this is precompiled in a template structure. it is not currently well documented, though if anyone wants to play with it, it's all defined in the source file templates/conftemplates.c. 🙈

The RA is primarily designed as a proactive information provider that continually gives you information that might be relevant to your current environment — but Savant can also be used as a standard text and information retrieval search engine. versatile queen 👑

🌈 USAGE 🌈

To index, you must have a set of source text-files, and a directory Savant can put database files into. the <source> arguments may be files or directories. if a directory is in the list, Savant will use all its contents, recursing into all subdirectories. Non-text files and backup files (those appended with ~ or prepended with #) are ignored. it also ignores dot-files (those starting with .) and symbolic links. any files or directories specified after the optional -e flag will be excluded. Savant will use any files it finds to create a database in the specified base directory, which must already exist. the optional -v argument (verbose) will direct Savant to keep you updated on its progress. so for example,

ra-index -v ~/RA-indexes/mail ~/RMAIL ~/Rmail-files -e ~/Rmail-files/Old-files

will build a database in the ~/RA-indexes/mail directory, made up of emails from my RMAIL file plus all files and subdirectories of ~/Rmail-files, excluding files and directories in ~/Rmail-files/Old-files. see, not scary at all 💖

ra-index can build databases in any directory you like, but the emacs interface for the Remembrance Agent expects a particular structure. for each database you want to make, you should create a directory, and all these directories should live in the same parent directory. for example, for my own use I have a directory ~/RA-indexes/, and within that are the directories ~/RA-indexes/mail/, ~/RA-indexes/papers/, etc. which actually contain the database files. ✨

🎀 OPTIONS 🎀

  • -v — Verbose mode. print useful information. 🗣️
  • -d — Debug mode. print <em>not-so-useful</em> information. 💀
  • -e — Exclude all filenames and directories which follow
  • -s — Follow symbolic links when indexing 🔗
  • --version — Print version information. 📜

🔥 SEE ALSO 🔥

ra-retrieve(1)

💜 AUTHOR 💜

Bradley Rhodes, MIT Media Lab. please send comments and questions to ra-bugs@media.mit.edu. new versions and updates can be found at http://www.media.mit.edu/~rhodes/RA/ 🦋

All code included in versions up to and including 2.09:

Copyright (C) 2001 Massachusetts Institute of Technology.

All modifications subsequent to version 2.09 are copyright Bradley Rhodes or their respective authors.

Developed by Bradley Rhodes at the Media Laboratory, MIT, Cambridge, Massachusetts, with support from British Telecom and Merrill Lynch.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. for commercial licensing under other terms, please consult the MIT Technology Licensing Office.

This program may be subject to the following US and/or foreign patents (pending): "Method and Apparatus for Automated, Context-Dependent Retrieval of Information," MIT Case No. 7870TS. If any of these patents are granted, royalty-free license to use this and derivative programs under the GNU General Public License are hereby granted.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

😭 BUGS 😭

Dates are not currently indexed, so anything trying to do a date query gets <em>no suggestion back</em>. sad. 😔

Requires GNU make to compile.

The template structure isn't documented. 💀

and that's the whole page, bestie!! this one came from the MIT Media Lab, it's part of the REMEMBRANCE AGENT family, and it's filed under RA-INDEX(1). go forth and index things ✨🦋💖

📦 where this page comes from