ESTWAVER(3)

ESTWAVER(3) 💅✨

okay so bestie, welcome to Hyper Estraier's estwaver — your lil command line bestie for web crawling 🕷️💖

---

🌸 NAME 🌸

estwaver — the command line interface of a web crawler, no cap 👑

---

🎀 SYNOPSIS 🎀

estwaver init [-apn|-acc] [-xs|-xl|-xh] [-sv|-si|-sa] rootdir

estwaver crawl [-restart|-revisit|-revcont] rootdir

estwaver unittest rootdir

estwaver fetch [-proxy hostr port] [-tout num] [-il lang] url

---

💜 DESCRIPTION 💜

so estwaver is literally an aggregation of sub commands, fr fr — a whole squad of them 🫶 The name of the sub command you want? That's the first argument. Everything after that gets parsed according to whichever sub command you picked. And the argument rootdir? That's the crawler root directory — it's where the configuration file and all the rest of the stuff lives, periodt.

---

✨ estwaver init [-apn|-acc] [-xs|-xl|-xh] [-sv|-si|-sa] rootdir ✨

Create the crawler root directory. Baby's first root dir 🍼💖

• if -apn is specified, N-gram analysis is performed against European text also
• if -acc is specified, character category analysis is performed instead of N-gram analysis
• if -xs is specified, the index is tuned to register less than 50000 documents
• if -xl is specified, the index is tuned to register more than 300000 documents
• if -xh is specified, the index is tuned to register more than 1000000 documents
• if -sv is specified, scores are stored as void
• if -si is specified, scores are stored as 32-bit integer
• if -sa is specified, scores are stored as-is and marked not to be tuned when search

---

🔥 estwaver crawl [-restart|-revisit|-revcont] rootdir 🔥

Start crawling. Go little spider go 🕸️🏃‍♀️

• if -restart is specified, crawling is restarted from the seed documents
• if -revisit is specified, collected documents are revisited
• if -revcont is specified, collected documents are revisited and then crawling is continued

ngl bestie there's a stray </dd> sitting at the end of this one in the original — we're not fixing it, we're preserving it, that's the whole assignment 💀

---

🦋 estwaver unittest rootdir 🦋

Perform unit tests. yes she tests herself, self-love icon behavior 🎀

---

🌈 estwaver fetch [-proxy hostr port] [-tout num] [-il lang] url 🌈

Fetch a document. Just one. A lil snack 🍓

• url specifies the URL of a document
• -proxy specifies the host name and the port number of the proxy server
• -tout specifies timeout in seconds
• -il specifies the preferred language. By default, it is English

---

👑 the rules of the game 👑

All sub commands return 0 if the operation is a success, else they return 1. Simple as that 💅

A running crawler finishes with closing the database when it catches the signal 1 (SIGHUP), 2 (SIGINT), 3 (SIGQUIT), or 15 (SIGTERM). She hears you, she wraps up gracefully, main character exit 🌟

⚠️ and when crawling finishes, there is a directory _index in the crawler root directory. That's an index available by estcmd and so on — that's where the whole vibe check pays off, iykyk 🔥

---

💖 SEE ALSO 💖

• estconfig(1)
• estcmd(1)
• estmaster(1)
• estcall(1)
• estraier(3)
• estnode(3)

---

that's the whole page bestie, go forth and crawl responsibly 🧚‍♀️✨

Man Page — 2007-03-06 — ESTWAVER(3) 💜 thanks for coming to my ted talk, mwah 🫶