📦 perl-www-robotrules

3 pages

W

WWW::RobotRules(3pm)database of robots.txt-derived permissions✨
WWW::RobotRules::AnyDBM_File(3pm)Persistent RobotRules✨
WWW::RobotRules::DB_File(3pm)Parse robots.txt files using a disk cache✨