WWW::RobotRules(3)

💅 WWW::RobotRules — like, the manual page, but make it make sense

okay so bestie, you're looking at the User Contributed Perl Documentation for WWW::RobotRules(3) and honestly? it's giving "polite internet bouncer" energy and I'm obsessed. let's get into it ✨

---

🌸 NAME 🌸

WWW::RobotRules — a whole database of permissions that came straight out of robots.txt. no cap, that's literally what it is 🫶

---

✨ SYNOPSIS ✨

here's the vibe in code form, copy it exactly bestie:

use WWW::RobotRules;
my $rules = WWW::RobotRules->new('MOMspider/1.0');

use LWP::Simple qw(get);

{
  my $url = "http://some.place/robots.txt";
  my $robots_txt = get $url;
  $rules->parse($url, $robots_txt) if defined $robots_txt;
}

{
  my $url = "http://some.other.place/robots.txt";
  my $robots_txt = get $url;
  $rules->parse($url, $robots_txt) if defined $robots_txt;
}

# Now we can check if a URL is valid for those servers
# whose "robots.txt" files we've gotten and parsed:
if($rules->allowed($url)) {
    $c = get $url;
    ...
}

---

🦋 DESCRIPTION 🦋

so this module parses /robots.txt files exactly as specified in "A Standard for Robot Exclusion", which lives at <http://www.robotstxt.org/wc/norobots.html>. webmasters use that /robots.txt file to tell well-behaved robots "hey, pls don't touch these parts of my site" 💅

the parsed files get kept inside a WWW::RobotRules object (main character energy honestly), and that object hands you methods to check whether access to a given URL is prohibited. and here's the slay part: the same WWW::RobotRules object works for one or more parsed /robots.txt files across any number of hosts. multitasking icon 👑

here are the methods, bestie:

• $rules = WWW::RobotRules->new($robot_name) — the constructor for WWW::RobotRules objects. the first argument you hand to new() is the name of the robot, periodt.
• $rules->parse($robot_txt_url, $content, $fresh_until) — the parse() method takes as arguments the URL that was used to retrieve the /robots.txt file, and the contents of the file. (yep, that's the args, fr fr.)
• $rules->allowed($uri) — returns TRUE if this robot is allowed to retrieve this URL. simple as that 💖
• $rules->agent([$name]) — get/set the agent name.

⚠️ NOTE: changing the agent name will clear the robots.txt rules and expire times out of the cache. like, fully wipes them. don't do it casually, bestie 😭

---

🎀 ROBOTS.TXT 🎀

the format and semantics of the /robots.txt file go like this — and heads up, this is an edited abstract of <http://www.robotstxt.org/wc/norobots.html>:

the file consists of one or more records separated by one or more blank lines. each record contains lines shaped like this:

<field-name>: <value>

the field name is case insensitive. any text after the # character on a line gets ignored during parsing — that's what comments are for, obviously. and these are the <field-names> you can use:

🌈 User-Agent

the value of this field is the name of the robot that the record is describing an access policy for. if more than one User-Agent field is present, the record describes an identical access policy for more than one robot. at least one field needs to be present per record. if the value is '*', the record describes the default access policy for any robot that hasn't matched any of the other records.

the User-Agent fields must occur before the Disallow fields. if a record contains a User-Agent field after a Disallow field, that constitutes a malformed record 💀. this parser will assume that a blank line should have been placed before that User-Agent field, and will break the record into two. all the fields before the User-Agent field will constitute a record, and the User-Agent field will be the first field in a new record. so like, it self-heals, king behavior 🫶

💜 Disallow

the value of this field specifies a partial URL that is not to be visited. this can be a full path, or a partial path — any URL that starts with this value will not be retrieved. nope. not today. 🙈

unrecognized records are ignored. (we simply do not perceive them ✨)

---

⭐ ROBOTS.TXT EXAMPLES ⭐

🍓 example one

this /robots.txt file says no robots should visit any URL starting with /cyberworld/map/ or /tmp/:

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear

🧚 example two

this one says no robots should visit any URL starting with /cyberworld/map/ — except the robot called "cybermapper":

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space

# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:

💖 example three

this one says no robots should visit this site any further, like, at all:

# go away
User-agent: *
Disallow: /

💀 example four — the malformed one

yep, this is an example of a malformed robots.txt file. iconic of ancientcastle.example.com tbh:

# robots.txt for ancientcastle.example.com
# I've locked myself away.
User-agent: *
Disallow: /
# The castle is your home now, so you can go anywhere you like.
User-agent: Belle
Disallow: /west-wing/ # except the west wing!
# It's good to be the Prince...
User-agent: Beast
Disallow:

this file is missing the required blank lines between records. however, the intention is clear. (we stan a messy but understandable file 😌)

---

👑 SEE ALSO 👑

• LWP::RobotUA
• WWW::RobotRules::AnyDBM_File

---

💅 COPYRIGHT 💅

Copyright 1995-2009, Gisle Aas
Copyright 1995, Martijn Koster

this library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. 🫶

---

and that's the whole page, bestie — brought to you by perl v5.42.0, dated 2012-02-20, in the WWW::RobotRules(3) section. go be a well-behaved robot now, mwah 💋🌸