VISITORS(1)

📦 version 0.7, in arch rolling

🦋 VISITORS(1) — General Commands Manual 🦋

okay bestie, buckle up, because this is the manpage for visitors, and it's genuinely one of the most underrated little tools out there. it's a fast web server log analyzer and it's about to make your logs make sense. no cap. 💅

✨ NAME ✨

visitors — a fast web server log analyzer 🚀

✨ SYNOPSIS ✨

visitors [options] <filename> [<filename> ...]

✨ DESCRIPTION ✨

so basically, visitors generates access statistics from whatever web log files you hand it. like, you point it at your logs and it goes off and produces the whole report card. 📊

the resulting reports contain a bunch of genuinely useful info and statistics, and here's the full list because we do not skip anything in this house:

  • Requested pages 📄
  • Requested images 🖼️
  • Referers by number of visits and age 🔗
  • Unique visitors in each day 👥
  • Page views per visit 👀
  • Pages accessed by the Google crawler (plus the date of google's last access on every page) 🤖
  • Pages accessed by the AdSense crawler (plus the date of adsense's last access on every page) 💰
  • Percentage of visits originated from Google searches for every day 🔍
  • User navigation patterns (web trails) 🐾
  • Keyphrases used in Google searches ⌨️
  • Human languages used in google searches 🌍
  • User agents 🕵️
  • Weekdays and Hours distributions of accesses 📅
  • Weekdays/Hours combined bidimensional map 🗺️
  • Month/Day combined bidimensional map 📆
  • Visual path analysis with Graphviz 🎨
  • Operating systems, browsers and domains popularity 👑
  • Visitors screen resolution and color depth 🖥️
  • 404 errors 💀

now, the web log files don't need to follow some strict rigid format, which is honestly so slay of them. except — and this part is non-negotiable, bestie:

  • the date MUST be included between [ and ] chars
  • the client hostname MUST be the first entry in the log
  • referers and requests MUST be included between double quote chars

out of the box Apache log files will just work, no problems, no drama. 🫶

it's also possible to use Visitors with IIS log files — you just convert them first using the iis2apache.pl utility distributed with Visitors. (that utility is the same one you can find at http://www.jammed.com/~jwa/hacks/ and it's distributed under the GPL license, fyi.)

📌 note that logfile can be a - character to use the standard input. yes, that's a real feature, and yes, it's giving piping energy.

✨ Available options: ✨

  • -A, --all — activates all the optional reports. this option is equivalent to -GKUWRDOB. but lowkey important: --trails is not implicitly included in this option because it also requires --prefix. see the --trails option documentation for the details, bestie. 👑
  • -T, --trails — enables the Web Trails feature. the report shows what the more frequent moves between pages of your site are (like, the actual paths people take). this option requires the --prefix option to work. 🐾
  • -G, --google — activates two reports about pages accessed by the Google and Adsense web crawlers. pages are shown ordered according to the last time the Google web crawler requested the page. the first page shown is the latest that was accessed. 🤖
  • -K, --google-keyphrases — activates a report showing common search keyphrases used to found your web site from Google. 🔍
  • -Z, --google-keyphrases-age — activates a report showing common the latest keyphrases used to found your site from Google. ⏳
  • -H, --google-human-language — activates a report showing common human languages used to search from Google. this feature uses the 'hl' variable of the Google referer URL. 🌍
  • -U, --user-agents — show information about common user agents. 🕵️
  • -W, --weekday-hour-map — activates the generation of a combined weekdays/hours bidimensional map, showing info about traffic in every 168 different hours of a 7 days week. brighter colors mean higher traffic. this is ideal for figuring out the best moment on a week for a maintenance downtime, what the target of the site is, whether people are accessing it from work or from home, and so on. the map is generated as pure html inside the report. 🗺️
  • -M, --month-day-map — activates the generation of a combined month/day bidimensional map, showing info about traffic in every 365 different days of the year. brighter colors mean higher traffic. this is useful to figure out, with a quick look, traffic trends and days with particularly high or low traffic. the map is generated as pure html inside the report. 📆
  • -R, --referers-age — shows referers ordered by age. the 'age' of a referer is the date it appeared the first time. in the report, newer referers are on top. this report is useful to check for new external links. 🔗
  • -D, --domains — activates the generation of info about Top Level Domains popularity. this info may be useful to guess the amount of visits from different countries. note that Visitors will not resolve numerical IP addresses if they're not already resolved in the log file — all the unresolved IP addresses will be shown in this report under the entry Unresolved IP. 🌐
  • -O, --operating-systems — activates the report about Operating Systems popularity, sorted by number of accesses. all the common operating systems are listed in the report, while unknown operating systems get summed into the unknown entry. 🐧
  • -B, --browsers — activates the report about Browsers popularity, sorted by number of accesses. all the common browsers are listed, while unknown browsers get summed into the unknown entry. browsers are listed by family (for example Internet Explorer, Opera, and so on), and not by specific version. 🌈
  • -X, --error404 — activates the generation of the missing documents (404 error) report. this shows files requested but missing, ordered by number of requests. it's useful to discover if, for some mistake, there's some file missing in the web site — but honestly you'll often see bizarre requests performed by users or internet worms and security scans. 💀
  • -Y, --pageviews — activates the generation of a report showing (an approximation of) the percentage of pages viewed per unique visit. the goal here is to understand the usage pattern of the site and the level of interest of the visitors. for example, in a site that provides a number of pages with interesting contents, the percentage of visitors performing a single page view per visit is probably searching for something else. 👀
  • -S, --robots — activates the generation of a report showing user agents of clients requesting the file robots.txt, with the exception of the MSIE Crawler requests. the result is a list of web robots and spieders that accessed your web site, ordered by number of requests of robots.txt. 🕷️
  • --screen-info — activates the screen resolution and color depth reports. ⚠️ note that for this report to work you have to insert on your HTML pages the javascript code you can find in the README file in the visitors tarball. 🖥️
  • --stream — enables the Stream Mode (see the STREAM MODE DETAILS section for more info). shortly: when in stream mode Visitors will process all the log files specified (possibly none — that's valid in this mode) as usual, producing the report. then the stream mode is entered and Visitors starts to read from standard input for a continuous stream of web logs, updating the statistics incrementally as new data is available. a new report is produced periodically if new data arrived, according to the --update-every option (default is to update the statistics every ten minutes). it's possible to ask Visitors to reset the statistics after some period of time using the --reset-every option — this allows you to have a snapshot of what's going on in the last five minutes, hour, day or week. 🌊
🚨 note that --stream requires --output-file because Visitors needs to overwrite the report for every update, so it can't output to standard output as usual. if you plan to use the stream mode, also check the --tail option.
  • --update-every seconds — by default in Stream Mode statistics are updated every 10 minutes. this option specifies a different period in seconds. ⏱️
  • --reset-every seconds — by default in Stream Mode statistics are never reset, but continuously updated incrementally. this option specifies to reset statistics after the given amount of time in seconds. useful to have a snapshot of the web site usage. 🔄
  • -f, --output-file file — write output to file instead of stdout. 📝
  • -m, --max-lines number — set the max number of entries that should be shown in reports like referers, keyphrases and so on. this option sets all the reports max number of entries for all the reports at once. ✨
  • -r, --max-referers number — set the max number of entries in the referer report. 🔗
  • -p, --max-pages number — set the max number of entries in the accessed pages report. 📄
  • -i, --max-images number — set the max number of entries in the accessed images report. 🖼️
  • -x, --max-error404 number — set the max number of entries in the missing documents report. 💀
  • -u, --max-useragents number — set the max number of entries in the user agents report. 🕵️
  • -t, --max-trails number — set the max number of entries in the web trails report. 🐾
  • -g, --max-googled number — set the max number of entries in the crawled pages report (google bot). 🤖
  • --max-adsensed number — set the max number of entries in the crawled pages report (adsense bot). 💰
  • -k, --max-google-keyphrases number — set the max number of entries in the Google keyphrases report. ⌨️
  • -a, --max-referers-age number — set the max number of entries in the referers by date report. ⏳
  • -d, --max-domains number — set the max number of entries in the domains report. 🌐
  • -P, --prefix string — prefixes specify to visitors how a link should look like to be classified as internal to your site. this option is required for --trails, and will also have the nice effect of avoiding internal links being shown in the referers report. if you're analyzing statistics for http://www.your.site.com/, just use:
--prefix http://www.your.site.com

if your site is reachable using more hostnames you should specify all these, like in the following example:

--prefix http://www.your.site.com --prefix http://your.site.com

🔗💖

  • -o, --output html|text — output module. you can use text or html. the default is html. 🎀

(( part 2 is coming, don't go anywhere bestie )) 🌸

  • -V, --graphviz — bestie, this turns on Graphviz mode ✨ Visitors analyzes the log file and builds a graph describing your website’s access patterns. The info used is the same as the web trails report (which you can turn on with --trails), but as a graph it can be way more readable for nontrivial sites. Example of how to use it:
% visitors access.log --prefix http://www.hping.org \
  --graphviz > graph.dot

% dot /tmp/graph.dot -Tpng > graph.png

On Debian systems, the dot command is included in the graphviz package. The generated graph has edges of different colors, from blue to red, to show a low-to-high level of popularity for a given movement from one page to another of the website. This option requires one or more --prefix options in order to work, just like the --trails option. 💅

  • -V, --graphviz-ignorenode-google — don’t put the google node on the generated graph. Only useful with --trails.
  • -V, --graphviz-ignorenode-external — don’t put the external referer node on the generated graph. Only useful with --trails.
  • -V, --graphviz-ignorenode-noreferer — don’t put the node indicating requests without referer on the generated graph. Only useful with --trails.
  • --tail — when this option is specified, Visitors emulates the Unix command tail -f --max-unchanged-stats=1 -q. You can specify the log file names to monitor for changes; once new data is appended in any of the specified files, Visitors outputs the new data to standard output. This option is useful together with Stream Mode (--stream). Files can be log-rotated because Visitors in Tail Mode will always try to reopen the file to check for changes. 🦋
  • --time-delta delta — if your web server is in a different timezone than most of your visitors or yourself, you’ll notice a shift in the reports regarding time and days of week. By default, Visitors generates output using the host’s locale. You can use --time-delta to adjust the output. Positive values shift right (toward the future) by the given number of hours; negative values shift left (toward the past). In the future this option may support directly specifying the output timezone.
  • --filter-spam — filters referer spam using a keyword-based filter (see blacklist.h for more information on keywords). If you don’t know what referer spam is, check this Wikipedia page: http://en.wikipedia.org/wiki/Referer_spam.
  • --ignore-404 — when this option is turned on, log lines with 404 errors are just used to generate the 404 errors report and not used for other reports.
  • --grep pattern — process only log lines matching the specified pattern. Patterns are matched using glob-style matching (the one used by the Unix shell):
    • * — matches any sequence of characters in string, including a null string.
    • ? — matches any single character in string.
    • [chars] — matches any character in the set given by chars. If a sequence of the form x-y appears in chars, then any character between x and y, inclusive, will match.
    • \x — matches the single character x. This provides a way of avoiding the special interpretation of the characters *?[]\ in pattern.

By default, matching is performed in a case-sensitive way, but case-insensitive matching may be forced by prefixing the pattern with the string cs:, so for example the pattern cs:firefox will match all the log lines containing the string firefox, FireFox, FIREFOX and so on. 🎀

  • --exclude pattern — works exactly like --grep, but only lines NOT matching the specified pattern are processed. Note that --grep and --exclude can be used multiple times, and are processed sequentially. For example, visitors --grep firefox --exclude download will process only lines including the string firefox but not including the string download.
  • --debug — show additional information on errors. For example, invalid lines are printed on the standard error if found. Mainly useful for developers and error reporting.
  • -h, --help — show usage and copyright information.
  • -v, --version — show program version.

💖 EXAMPLES 💖

The simplest usage — perfect for checking a web log interactively (for example over ssh on your web server) — is just:

% visitors access.log | less

That produces a human-readable output in text only. To generate HTML web stats with much more information, you may use this instead:

% visitors --output text -A -m 30 access.log -o html > report.html

If you want information on the usage patterns for your site, you must provide the URL prefix of your website and specify the --trails option. The next example produces an HTML report with usage patterns information.

% visitors -A -m 30 access.log --trails \
  --prefix http://www.hping.org > report.html

Note that it’s okay to specify multiple file names, or to provide the input using standard input, like in the following two examples:

% visitors /var/log/apache/access.log.*
% zcat access.log.*.gz | visitors -

🌈 STREAM MODE DETAILS 🌈

The usual way to run Visitors is to specify some option to control the report generation, and the name of log files. For example, to generate a report from two Apache access log files you can write:

% visitors -A access.log.1 access.log.2 > report.html

Visitors analyzes the log files and outputs the report. Sometimes it can be more interesting to have web statistics updated continuously, almost in real time, as new data becomes available. To provide this feature, Visitors implements a mode called Stream Mode that reads a stream of logs from standard input. The following command line shows how to use it (but check the --stream option documentation for more information).

% tail -f /var/log/apache/access.log | \
  visitors --stream -A --update-every 60 \
  --output-file /tmp/report.html

Visitors incrementally updates the statistics as new logs become available and updates the HTML report every 60 seconds. As you can see, in this mode you’re required to specify the report file name using the --output-file option because Visitors needs to overwrite the report to update it. Note that instead of the tail command in the above example, it’s possible to use Visitors in Tail Mode instead (an emulation for the tail program):

% visitors --tail /var/log/apache/access.log | \
  visitors --stream -A --update-every 60 \
  --output-file /tmp/report.html

It’s possible to generate real-time statistics about the last N seconds of web traffic, where N is configurable and can be from a few seconds to one week or more, using the --reset-every option. The following example generates statistics updated every 30 seconds about the last hour of traffic:

% visitors --tail /var/log/apache/access.log | \
  visitors --stream -A --update-every 30 --reset-every 3600 \
  --output-file /tmp/report.html

👑 AUTHORS 👑

Visitors was written by Salvatore Sanfilippo <antirez@invece.org>.

🧚 COPYING 🧚

Copyright (C) 2004,2005 Salvatore Sanfilippo <antirez@invece.org>.

Visitors is distributed under the GNU General Public License.

This manual page was written (based on the original HTML documentation) by Romain Francoise <rfrancoise@debian.org> for the Debian GNU/Linux system, but may be used by others. Salvatore Sanfilippo updated this man page starting from Visitors 0.5; this manual page is now part of the Visitors tarball.

okay bestie, that’s the whole vibe check ✅ this page is Visitors 0.7, dated April 2005, and it’s page VISITORS(1) — iconic, fr fr 💜

📦 where this page comes from