ESTCMD(1)

📦 version 1.4.13, in fedora 43, fedora 44

✨ NAME ✨

estcmd — the command line interface of the core API. that's it, that's the whole name section, bestie 💅

🦋 SYNOPSIS 🦋

estcmd create [-tr] [-apn|-acc] [-xs|-xl|-xh|-xh2|-xh3] [-sv|-si|-sa]
[-attr name type] db

estcmd put [-tr] [-cl] [-ws] [-apn|-acc] [-xs|-xl|-xh||-xh2|-xh3]
[-sv|-si|-sa] db [file]

estcmd out [-cl] [-pc enc] db expr

estcmd edit [-pc enc] db expr name [value]

estcmd get [-nl|-nb] [-pidx path] [-pc enc] db expr [attr]

estcmd list [-nl|-nb] [-lp] db

estcmd uriid [-nl|-nb] [-pidx path] [-pc enc] db expr

estcmd meta db [name [value]]

estcmd inform [-nl|-nb] db

estcmd optimize [-onp] [-ond] db

estcmd merge [-cl] db target

estcmd repair [-rst|-rsh] db

estcmd search [-nl|-nb] [-pidx path] [-ic enc]
[-vu|-va|-vf|-vs|-vh|-vx|-dd] [-sn wnum hnum anum] [-kn num] [-um] [-ec
rn] [-gs|-gf|-ga] [-cd] [-ni] [-sf|-sfr|-sfu|-sfi] [-hs] [-attr expr]
[-ord expr] [-max num] [-sk num] [-aux num] [-dis name] [-sim id] db
[phrase]

estcmd gather [-tr] [-cl] [-ws] [-no] [-fe|-ft|-fh|-fm] [-fx sufs cmd]
[-fz] [-fo] [-rm sufs] [-ic enc] [-il lang] [-bc] [-lt num] [-lf num]
[-pc enc] [-px name] [-aa name value] [-apn|-acc]
[-xs|-xl|-xh|-xh2|-xh3] [-sv|-si|-sa] [-ss name] [-sd] [-cm] [-cs num]
[-ncm] [-kn num] [-um] db [file|dir]

estcmd purge [-cl] [-no] [-fc] [-pc enc] [-attr expr] db [prefix]

estcmd extkeys [-no] [-fc] [-dfdb file] [-ncm] [-ni] [-kn num] [-um]
[-attr expr] db [prefix]

estcmd words [-nl|-nb] [-dfdb file] [-kw|-kt] db

estcmd draft [-ft|-fh|-fm] [-ic enc] [-il lang] [-bc] [-lt num] [-kn
num] [-um] [file]

estcmd break [-ic enc] [-il lang] [-apn|-acc] [-wt] [file]

estcmd iconv [-ic enc] [-il lang] [-oc enc] [file]

estcmd regex [-inv] [-repl str] expr [file]

estcmd scandir [-tf|-td] [-pa|-pu] [dir]

estcmd multi [-db db] [-nl|-nb] [-ic enc] [-gs|-gf|-ga] [-cd] [-ni]
[-sf|-sfr|-sfu|-sfi] [-hs] [-hu] [-attr expr] [-ord expr] [-max num]
[-sk num] [-aux num] [-dis name] [phrase]

estcmd randput [-ren|-rla|-reu|-ror|-rjp|-rch] [-cs num] db dnum

estcmd wicked db dnum

estcmd regression db

estcmd version

🌸 DESCRIPTION 🌸

okay so here's the tea: estcmd is literally an aggregation of sub commands, fr fr. the name of a sub command is specified by the first argument, and then every other argument gets parsed according to whichever sub command you picked. the argument db specifies the path of an index. 🫶

💖 estcmd create 💖

estcmd create [-tr] [-apn|-acc] [-xs|-xl|-xh|-xh2|-xh3] [-sv|-si|-sa]
[-attr name type] db

Create an index. iconic behaviour honestly 👑

  • -tr — a brand new index gets created no matter what, even if one already exists. no questions asked.
  • -apn — N-gram analysis is performed against European text also.
  • -acc — character category analysis is performed instead of N-gram analysis.
  • -xs — the index gets tuned to register less than 50000 documents.
  • -xl — the index gets tuned to register more than 300000 documents.
  • -xh — the index gets tuned to register more than 1000000 documents.
  • -xh2 — the index gets tuned to register more than 5000000 documents.
  • -xh3 — the index gets tuned to register more than 10000000 documents.
  • -sv — scores are stored as void.
  • -si — scores are stored as 32-bit integer.
  • -sa — scores are stored as-is and marked not to be tuned when search.
  • -attr name type — specifies an attribute index and its data type. you can drop this option multiple times, it's not a one-and-done situation. 💅

🎀 estcmd put 🎀

estcmd put [-tr] [-cl] [-ws] [-apn|-acc] [-xs|-xl|-xh||-xh2|-xh3]
[-sv|-si|-sa] db [file]

Register a document of document draft to an index. yes bestie, we're adding stuff ✨

file specifies a target file. if it's omitted, the standard input is read.

  • -tr — a new index is created regardless if one exists.
  • -cl — regions of a overwritten document are cleaned up.
  • -ws — scores are weighted statically with score weighting attribute.
  • -apn — N-gram analysis is performed against European text also.
  • -acc — character category analysis is performed instead of N-gram analysis.
  • -xs — the index is tuned to register less than 50000 documents.
  • -xl — the index is tuned to register more than 300000 documents.
  • -xh — the index is tuned to register more than 1000000 documents.
  • -xh2 — the index is tuned to register more than 5000000 documents.
  • -xh3 — the index is tuned to register more than 10000000 documents.
  • -sv — scores are stored as void.
  • -si — scores are stored as 32-bit integer.
  • -sa — scores are stored as-is and marked not to be tuned when search.

🧚 estcmd out 🧚

estcmd out [-cl] [-pc enc] db expr

Remove information of a document from an index. bye boo 👋

expr specifies the ID number, the URI, or the local path of a document.

  • -cl — regions of the document are cleaned up.
  • -pc enc — specifies the encoding of file paths. by default, it's ISO-8859-1.

🍓 estcmd edit 🍓

estcmd edit [-pc enc] db expr name [value]

Edit an attribute of a document in an index. little glow-up moment ✨

expr specifies the ID number, the URI, or the local path of a document.

name specifies the name of an attribute.

value specifies the value of the attribute. if it's omitted, the attribute is removed.

  • -pc enc — specifies the encoding of the file path and the attribute value. by default, it's ISO-8859-1.

💜 estcmd get 💜

estcmd get [-nl|-nb] [-pidx path] [-pc enc] db expr [attr]

Output document draft of a document in an index. show me what you got 💅

expr specifies the ID number, the URI, or the local path of a document.

if attr is specified, only the value of the attribute is output.

  • -nl — the index is opened without file locking.
  • -nb — file locking is performed without blocking.
  • -pidx path — specifies the path of a pseudo index. this option can be specified multiple times.
  • -pc enc — specifies the encoding of file paths. by default, it's ISO-8859-1.

🌈 estcmd list 🌈

estcmd list [-nl|-nb] [-lp] db

Output a list of all document in an index. the whole roster, no cap 📋

  • -nl — the index is opened without file locking.
  • -nb — file locking is performed without blocking.
  • -lp — local path equivalent to URL of "file://" is output.

⭐ estcmd uriid ⭐

estcmd uriid [-nl|-nb] [-pidx path] [-pc enc] db expr

Output the ID number of a document specified by URI. name to number, iconic 🔢

expr specifies the URI or the local path of a document.

  • -nl — the index is opened without file locking.
  • -nb — file locking is performed without blocking.
  • -pidx path — specifies the path of a pseudo index. this option can be specified multiple times.
  • -pc enc — specifies the encoding of file paths. by default, it's ISO-8859-1.

🦋 estcmd meta 🦋

estcmd meta db [name [value]]

Handle meta data. the little details that matter 🎀

name specifies the name of a piece of meta data. if it's omitted, a list of all names is output.

value specifies the value of the meta data to be recorded. if it's omitted, the current value is output. if it's an empty string, the meta data is removed.

🔥 estcmd inform 🔥

estcmd inform [-nl|-nb] db

Output the number of documents and the number of unique words in an index. stats check ✨

  • -nl — the index is opened without file locking.
  • -nb — file locking is performed without blocking.

👑 estcmd optimize 👑

estcmd optimize [-onp] [-ond] db

Optimize an index and clean up dispensable regions. self-care for your database 💅

  • -onp — it is omitted to clean up dispensable regions.
  • -ond — it is omitted to optimize the database files.

🌸 estcmd merge 🌸

estcmd merge [-cl] db target

Merge another index. two become one, romantic honestly 🫶

target specifies the path of another index.

  • -cl — regions of overwritten documents are cleaned up.

🩹 estcmd repair 🩹

estcmd repair [-rst|-rsh] db

Repair a broken index. we can fix her 😭

  • -rst — strict consistency check is performed.
  • -rsh — consistency check is omitted.

🎀 OPTIONS 🎀

Okay bestie, here's the search vibe, in all its glory:

estcmd search [-nl|-nb] [-pidx path] [-ic enc]
[-vu|-va|-vf|-vs|-vh|-vx|-dd] [-sn wnum hnum anum] [-kn num] [-um] [-ec
rn] [-gs|-gf|-ga] [-cd] [-ni] [-sf|-sfr|-sfu|-sfi] [-hs] [-attr expr]
[-ord expr] [-max num] [-sk num] [-aux num] [-dis name] [-sim id] db
[phrase]

This one searches an index for documents, which is literally the whole point of having an index, right? 💅

phrase is the thing you're searching for — it specifies the search phrase. That's it, that's the tea.

  • -nl — the index gets opened with no file locking at all. Living dangerously, no cap.
  • -nb — file locking does happen, but it won't block while doing it.
  • -pidx path — tells it the path of a pseudo index. You can drop this option as many times as you want, it stacks. ✨
  • -ic enc — the input encoding. Default is UTF-8, periodt.
  • -vu — outputs TSV of ID number and URI. Clean, minimal, iconic.
  • -va — outputs multipart format including attributes.
  • -vf — outputs multipart format including the document draft.
  • -vs — multipart format with attributes and snippets. The full meal. 🍓
  • -vh — human readable format with attributes and snippets, for when you want to actually read it with your eyeballs.
  • -vx — XML including attributes and snippets.
  • -dd — document draft data gets dumped and saved into separated files.
  • -sn wnum hnum anum — sets three widths, in order: the whole width of the snippet, the width of strings picked up from the beginning of the text, and the width of strings picked up around each highlighted word.
  • -kn num — the number of keywords to be extracted. By default? Keyword extraction just does not happen, so you gotta ask for it.
  • -um — morphological analyzers get used for keyword extraction. Fancy. 🧚
  • -ec rn — sets the lower limit of similarity eclipse.
  • -gs — every single key of N-gram gets checked. By default it's done alternately instead.
  • -gf — keys of N-gram get checked every three.
  • -ga — keys of N-gram get checked every four.
  • -cd — checks whether documents match the search phrase definitely. Certainty check, slay.
  • -ni — omits TF-IDF tuning.
  • -sf — the phrase is treated as a simplified form.
  • -sfr — the phrase is treated as a rough form.
  • -sfu — the phrase is treated as a union form.
  • -sfi — the phrase is treated as an intersection form.
  • -hs — score information gets output as an attribute.
  • -attr expr — specifies an attribute search condition. Stack it as many times as you like. 💖
  • -ord expr — the order expression. By default it's descending by score.
  • -max num — the maximum number of shown documents. A negative number means unlimited. Default is 10.
  • -sk num — how many documents get skipped. Default is 0.
  • -aux num — permission to adopt the result of the auxiliary index. If it's not more than 0, the auxiliary index is not used at all. Default is 32.
  • -dis name — the name of the distinct attribute.
  • -sim id — the ID number of the seed document for a similarity search. 🌈

🌸 estcmd gather 🌸

The collector. It goes out into your filesystem and brings back documents. 🫶

estcmd gather [-tr] [-cl] [-ws] [-no] [-fe|-ft|-fh|-fm] [-fx sufs cmd]
[-fz] [-fo] [-rm sufs] [-ic enc] [-il lang] [-bc] [-lt num] [-lf num]
[-pc enc] [-px name] [-aa name value] [-apn|-acc]
[-xs|-xl|-xh|-xh2|-xh3] [-sv|-si|-sa] [-ss name] [-sd] [-cm] [-cs num]
[-ncm] [-kn num] [-um] db [file|dir]

This one scans the local file system and registers documents into an index. It's giving very main character energy. ✨

So about that third argument:

  • If the third argument is the name of a file, then a list of paths of target documents gets read from it. And if it's -, then standard input is what you're feeding it.
  • If the third argument is the name of a directory, then all files under that directory are treated as target documents. Everything. The whole folder. Obsessed.

Now the options:

  • -tr — a new index is created regardless of whether one already exists. Fresh start, we love that for you.
  • -cl — regions of overwritten documents get cleaned up.
  • -ws — scores get weighted statically using the score weighting attribute.
  • -no — operations are printed but not actually executed. A dress rehearsal, bestie. Nothing real happens.
  • -fe — target files are treated as document draft. By default the format is detected by the suffix of each document.
  • -ft — target files are treated as plain text.
  • -fh — target files are treated as HTML.
  • -fm — target files are treated as MIME.
  • -fx sufs cmd — target files with the specified suffixes get processed by the specified outer command. A "*" matches any file. Now listen closely, this part is important:
    • If the command is leaded by "T@", the output of the command is treated as plain text.
    • If the command is leaded by "H@", the output is treated as HTML.
    • If the command is leaded by "M@", the output is treated as MIME.
    • Else, the output is treated as document draft.
    • You can specify this option multiple times. 💖
  • -fz — documents that don't correspond to the condition of -fx get ignored.
  • -fo — target files are not read. Sounds weird but it's genuinely useful for efficient processing of the outer command. iykyk.
  • -rm sufs — target files with the specified suffixes get removed. A "*" matches any file. Multiple times, stackable. 💀
  • -ic enc — the input encoding. By default it's detected automatically.
  • -il lang — the preferred input language. By default, English is preferred.
  • -bc — binary files get detected and ignored.
  • -lt num — the text size limitation, in kilo bytes. Default is 128KB. If it's negative, the size is unlimited.
  • -lf num — the file size limitation, in mega bytes. Default is 32MB. If it's negative, the size is unlimited.
  • -pc enc — the encoding of file paths. Default is ISO-8859-1.
  • -px name — the name of an attribute read from the list of paths. Since that list can be in TSV format: the first field is treated as the path of a target document, and the second field plus the ones after it are definitions of attribute values. So -px names each of the values of the second field and the followers. You can specify it multiple times. ✨
  • -aa name value — the name and the value of an additional attribute. Stackable, multiple times. 💅
  • -apn — N-gram analysis is performed against European text also.
  • -acc — character category analysis is performed instead of N-gram analysis.
  • -xs — the index is tuned to register less than 50000 documents.
  • -xl — the index is tuned to register more than 300000 documents.
  • -xh — the index is tuned to register more than 1000000 documents.
  • -xh2 — the index is tuned to register more than 5000000 documents.
  • -xh3 — the index is tuned to register more than 10000000 documents. Absolute unit. 👑
  • -sv — scores are stored as void.
  • -si — scores are stored as 32-bit integer.
  • -sa — scores are stored as-is and marked not to be tuned when searching.
  • -ss name — the name of an attribute for the substitute score.
  • -sd — the modification date of each file gets recorded as an attribute.
  • -cm — documents whose modification date hasn't changed get ignored. Efficiency queen. 🎀
  • -cs num — the size of cache memory, in mega bytes. Default is 64MB.
  • -ncm — checking availability of the virtual memory is omitted.
  • -kn num — the number of keywords to be extracted. By default, keyword extraction is not performed.
  • -um — morphological analyzers get used for keyword extraction. 🧚

💜 estcmd purge 💜

estcmd purge [-cl] [-no] [-fc] [-pc enc] [-attr expr] db [prefix]

Purges information of documents which do not exist on the file system. Basically it's the ghostbuster for your index. 👻

If prefix is specified, then only documents whose URIs begin with it get touched. And plot twist: it can be specified by the local path of a directory.

  • -cl — regions of the deleted documents get cleaned up.
  • -no — operations are printed but not actually executed. Look, don't touch.
  • -fc — information of all target documents gets deleted. All of them. No survivors. 💀
  • -pc enc — the encoding of file paths. Default is ISO-8859-1.
  • -attr expr — an attribute search condition. You can specify this multiple times. 💖

⭐ estcmd extkeys ⭐

estcmd extkeys [-no] [-fc] [-dfdb file] [-ncm] [-ni] [-kn num] [-um]
[-attr expr] db [prefix]

Creates a database of keywords extracted from documents. Keyword girlboss behavior, honestly. 👑

If prefix is specified, only documents whose URIs begin with it are in the mix.

  • -no — operations are printed but not actually executed. A little preview moment.
  • -fc — all target documents get processed, whichever they have existing records or not. No skipping.
  • -dfdb file — specifies another database of document frequency. By default, document frequency is calculated dynamically according to the index.
  • -ncm — checking availability of the virtual memory is omitted.
  • -ni — omits TF-IDF tuning.
  • -kn num — the number of keywords to be extracted. Default here is 32.
  • -um — morphological analyzers get used for keyword extraction. ✨
  • -attr expr — an attribute search condition. Stackable, multiple times. 🫶
  • estcmd words [-nl|-nb] [-dfdb file] [-kw|-kt] db — bestie this one spits out a list of every single unique word plus each record size, and that size is what gets treated as the document frequency. 💖
    • If -nl is specified, the index is opened without file locking (living dangerously, no cap).
    • If -nb is specified, file locking happens but without blocking — so it doesn't sit there waiting like an ex who left you on read. 💀
    • -dfdb names an outer database where the result gets stored. By default the result goes to standard output as TSV. And listen — if that outer database already exists, the value of each record is incremented. ✨
    • If -kw is specified, keywords and the numbers of their corresponding documents get output.
    • If -kt is specified, keywords and their related terms get output.
  • estcmd draft [-ft|-fh|-fm] [-ic enc] [-il lang] [-bc] [-lt num] [-kn num] [-um] [file] — for test and debug. 🧪
  • estcmd break [-ic enc] [-il lang] [-apn|-acc] [-wt] [file] — for test and debug. 🧪
  • estcmd iconv [-ic enc] [-il lang] [-oc enc] [file] — for test and debug. 🧪
  • estcmd regex [-inv] [-repl str] expr [file] — for test and debug. 🧪
  • estcmd scandir [-tf|-td] [-pa|-pu] [dir] — for test and debug. 🧪
  • estcmd multi [-db db] [-nl|-nb] [-ic enc] [-gs|-gf|-ga] [-cd] [-ni] [-sf|-sfr|-sfu|-sfi] [-hs] [-hu] [-attr expr] [-ord expr] [-max num] [-sk num] [-aux num] [-dis name] [phrase] — for test and debug. 🧪
  • estcmd randput [-ren|-rla|-reu|-ror|-rjp|-rch] [-cs num] db dnum — for test and debug. 🧪
  • estcmd wicked db dnum — for test and debug. 🧪
  • estcmd regression db — for test and debug. 🧪
  • estcmd version — shows the version information. 👑

Okay so here's the ✨ general lore ✨ that applies across the board:

Every single sub command returns 0 if the operation is a success, and otherwise returns 1. Slay, simple.

But as for put, out, gather, purge, randput, wicked, and regression — those ones finish up by closing the database whenever they catch signal 1 (SIGHUP), 2 (SIGINT), 3 (SIGQUIT), 13 (SIGPIPE), or 15 (SIGTERM). So they're polite about it, they tidy up before they go. 🫶

The data type of the attribute indexes you specify with the -attr option of the create sub command should be "seq" for sequencial type, "str" for string type, or "num" for number type. Pick your fighter. 💅

Each pseudo index you specify with the -pidx option of the search sub command and so on is literally just a directory containing files of document draft. And if you search a main index with pseudo indexes, a meta search of the main index and the pseudo indexes gets performed. That's the vibe. 🦋

The encoding name you pass to the -ic option should be a name registered to IETF, like UTF-8, ISO-8859-1, and so on. The language name you pass to the -il option should be one of "en" (English), "ja" (Japanese), "zh" (Chinese), or "ko" (Korean). 🌈

The outer command specified by the -fx option of gather receives the path of the target document as its first argument and the path for output as its second argument. And the original path of the target document is handed over as the value of the environment variable ESTORIGFILE. 🎀

⚠️ Okay important heads up bestie: note that similarity search is very slow, by default. Like, tragically slow. 😭 To improve the performance of similarity search, running estcmd extkeys beforehand is strongly recommended. Do it. Trust. 💀

👀 SEE ALSO 👀

Please see http://hyperestraier.sourceforge.net/uguide-en.html for detail. It's giving ✨ required reading ✨ fr fr. 📚

And that's the whole page, bestie — you ate that. 💅 This lil' masterpiece is ESTCMD(1), from the Man Page era, dated 2007-03-06. Iconic behavior, honestly. Go forth and index. 👑💖

📦 where this page comes from