AUTOSEARCH(1)

📦
version 2.519 · 2 versions
, in fedora 43

🦋 AUTOSEARCH(1) 🦋

Part 1 of 2 💖 Okay bestie, this is the User Contributed Perl Documentation manual page, and it's LONG, so we're splitting it up. Here's part one ✨

✨ NAME ✨

AutoSearch is a web-search tracking application 🔥

🎀 SYNOPSIS 🎀

AutoSearch [--stats] [--verbose] -n "Query Name" -s "query string"
--engine engine [--mail you@where.com] [--options "opt=val"]...
[--filter "filter"] [--host host] [--port port] [--userid bbunny
--password c4rr0t5] [--ignore_channels KABC,KCBS,KNBC] qid

AutoSearch --VERSION AutoSearch --help AutoSearch --man

Okay that's a lot of flags 😭 but it's all in the lineup above, so no stress 💅

🌈 DESCRIPTION 🌈

AutoSearch runs a web-based search and dumps the results into qid/index.html 💅 Later runs (that's the second form up in the SYNOPSIS) figure out what changed since the last time, and those changes get recorded in qid/YYYYMMDD.html ✨

AutoSearch is cron-job friendly because all the input parameters get saved in the web pages 🧚 So it basically works as an automated query agent for one particular search. The output files are a set of web pages you can view in a browser, so your results look cute 🍓

🌸 Example

AutoSearch -n 'LSAM Replication'
    -s '"lsam replication"'
    -e AltaVista
    replication_query

Heads up, that whole thing should be on one line 🙈 It makes a replication_query directory and fills it with AltaVista's results for "lsam replication", with pages titled "LSAM Replication" 🦋 Quick quoting tea: the single quotes around '"lsam replication"' are for the shell, and the double quotes tell AltaVista to search for the whole phrase instead of the separate words. Iconic 💜

💅 A more complicated example

AutoSearch -n 'External Links to LSAM'
    -s '(link:www.isi.edu/lsam or link:www.isi.edu/~lsam) -url:isi.edu'
    -e AltaVista::AdvancedWeb
    -o coolness=hot

This one does an advanced AltaVista search and passes along the (hypothetical) coolness option to the search engine. Main character energy 👑

✨ OPTIONS ✨

  • qid — this is the query identifier, and it's the directory where every file for this query and its results lives 🗂️ It can be an absolute path or a relative path from cwd. If the directory doesn't exist yet, it gets created and a brand new search starts 🌱
  • --stats — shows the search stats: the query string, number of hits, number of filtered hits, filter string, number of suspended (deleted) hits, previous set size, current set size, and more 📊
  • -v, --verbose — extra chatty mode 🗣️ You get additional messages and warnings
  • -n, --qn, --queryname — sets the query name 🏷️ It's used as the heading on your web pages, so make it a nice-looking version of your query string ✨
  • -s, --qs, --querystring — sets the query string 🔍 This is the exact text sent to the search engine, and you can add special characters to group or qualify your search 💖
  • -e, --engine — picks the search engine 🎯 Your query string gets submitted to the one you name. Lots of engines have specialized versions, like AltaVista::AdvancedWeb and AltaVista::News, which unlock more powerful searches and Usenet searches 🧠 For the deets on those variations, check out AltaVista or the man page for your search engine
  • --listnewurls — along with all the normal file maintenance, prints every new URL to STDOUT, one per line 🦋
  • -o, --options — sends query options to the search engine alongside the query string 🎛️ This lets you tweak the query for a specific engine or option, and you can use more than one. For example, -o what=news makes AltaVista search Usenet. It works, but the preferred move is -e AltaVista::News or -e AltaVista::AdvancedNews. Options are meant for internal or expert use, bestie 🧑‍💻
  • -f, --uf, --urlfilter — a regular expression that gets compared against the URLs of your results 🚫 If one matches (case-insensitive), it's removed from the hit set. For example, -f '.*\.isi\.edu' avoids all of ISI's web pages 🙅‍♀️
  • --cleanup i — deletes all traces of query results from more than i days ago 🧹 If --cleanup is given, every other option except the qid gets ignored 💀
  • --cmdline — rebuilds the complete command line (AutoSearch and all its arguments) that was used to create the query results 🧾 The command line gets shown on STDERR. If --cmdline is given, every other option except the qid gets ignored 🙈
  • --mail user@address, -m user@address — after the search finishes, emails that user the NEW results in HTML format 💌 This requires the Email::Send and related modules. If you send through an SMTP server, set the environment variable SMTPSERVER to your server name or IP address 🌐 If your SMTP server needs a password, set the environment variables SMTPUSERNAME and SMTPPASSWORD 🔐 If you send via sendmail, set the environment variable SENDMAIL when the sendmail executable isn't in the path 📮
  • --emailfrom user@address — if your outgoing mail server is being picky and rejects email from certain users, use this to set the From: header 💅
  • --userid bbunny — if the search engine needs a login and password (like Ebay::Completed), use this 🪪
  • --password Carr0t5 — if the search engine needs a login and password (like Ebay::Mature), use this 🔑

✨ CHANGING THE LOOK OF THE PAGES ✨

okay bestie, the basic format of these two pages is super simple AND customizable 💅 the one big rule is that the basic structure has to stay exactly the same 😭 HTML comments are how it finds each section of the doc. You can change almost everything, except the strings that mark where sections start and end 🫶

Here are the noteworthy tags and what they do 🎀

  • <!--Top-->.*<!--/Top--> — whatever text lives inside this tag gets placed at the very top of the output page 🔝 If the text includes the magic phrase AutoSearch WEB Searching, the query name swaps in to replace it. If there's no magic phrase and it's the very first search ever, AutoSearch will ask the user for a query name 🙋‍♀️
  • <!--Query{.*}/Query--> — the text between the braces is the query string 🔍 This is how AutoSearch keeps track of it. You can change it, but ONLY in qid/index.html 🚫 The text ask user is special and forces AutoSearch to request the search string from the user 🎤
  • <!--SearchEngine{.*}/SearchEngine--> — the text between the braces is the search engine 🌐 The other engines it supports are HotBot and Lycos 🌸 You can edit it to change the engine, but ONLY in qid/index.html. The text ask user is special here too, and forces AutoSearch to request the search string from the user 🙋
  • <!--QueryOptions{.*}/QueryOptions--> — the text between the braces specifies a query option ✨ You can use this tag multiple times to set multiple options 🫧
  • <!--URLFilter{.*}/URLFilter--> — the text between the braces is the URL filter 🧹 This is how AutoSearch maintains the filter. You can edit it, but ONLY in qid/index.html 🚫 The text ask user is special and forces AutoSearch to ask the user (STDIN) for the query string 🎤 When setting up the very first search, you have to edit first_index.html, NOT qid/index.html 🛑 The URL filter is a standard perl5 regular expression 🧠 URLs that don't match get KEPT 💖
  • <!--Bottom-->.*<!--/Bottom--> — the text inside this tag goes at the very bottom of the output page 👇 It's a great spot for navigation, page owner info, etc. 🧭

The rest of the tags come in a trio: ~Heading, ~Template, and ~, where ~ is Summary, Weekly, Appended, or Suspended 🦋 The sub-sections show up in the order given. To make a section, AutoSearch outputs the heading, then the template, then the section itself, then n copies of the formatted data, then a /section closer. Here's what each tag does:

  • ~Heading — identifies the heading for a section of the output file. So SummaryHeading is for the summary part, and so on. A section can be empty (like Suspended), and in that case no heading gets output 🫥
  • ~Template — defines how each item gets formatted. It's simple text replacement that turns the template into the real output text. The text to be replaced is written in ALLCAPS 🔠
  • ~ — locates the section (Summary, Weekly, etc.). This is where the actual n items of data live 📦

You can edit these values in the qid/index.html page of an existing search 💻 The file first_index.html (in the directory above qid) is used as a default template for new queries 🌱

Examples of these files can be seen under http://www.isi.edu/lsam/tools/autosearch/, or in the output generated by a brand-new AutoSearch 👀

📁 FILES 📁

  • first_index.html — optional file that sets the default format of the index.html file for a new query 🌷
  • first_date.html — optional file that sets the default format of the YYYYMMDD.html file for a new query 🌷
  • qid/index.html — (automatically created) the latest search results, plus a reverse chronological list of periodic searches 📜
  • qid/date.html — file used as a template for the YYYYMMDD.html files 🧁
  • qid/YYYYMMDD.html — (automatically created) a summary of changes for a particular date, AKA the 'Weekly' file 📅

The optional files first_index.html and first_date.html are used for the initial search as templates for qid/index.html and date.html, respectively. If either of these files doesn't exist, a default-default template is stored within the AutoSearch source 🫠 The whole point is to let a user set up a framework for a group of search sets that share a common format 💜 By leaving the default query name and query string alone, they'll be overridden by command line inputs 🖥️

👯 SEE ALSO 👯

For the library, see WWW::Search. For the perl regular expressions, see perlre. 🧠

✍️ AUTHORS ✍️

Wm. L. Scheding 🖊️

AutoSearch is a re-implementation of an earlier version written by Kedar Jog 🫶

Copyright (C) 1996-1997 University of Southern California. All rights reserved. 🏛️

Redistribution and use in source and binary forms are permitted, as long as the above copyright notice and this paragraph are duplicated in all such forms. Any documentation, advertising materials, and other materials related to that distribution and use have to acknowledge that the software was developed by the University of Southern California, Information Sciences Institute. The name of the University may not be used to endorse or promote products derived from this software without specific prior written permission 📜

THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 🚨

💡 DESIRED FEATURES 💡

These are good ideas that people have suggested 🧠

URL validation.

Validate the status of each URL (with HTTP HEAD requests) and indicate that status in the output 🔍

It should be possible to merge the results of searches from two search engines. If this merger were done as a new search engine, the operation would be transparent to AutoSearch 🦋

🐛 BUGS 🐛

None known at this time 🙈 If any crop up, please let the maintainer know at mthurn@cpan.org 💌

That's the whole page, bestie 🌈 This one comes from perl v5.42.0, dated 2025-07-25, and it's AUTOSEARCH(1) 💖 slay forever 💅✨

📦 where this page comes from