parsero(1)

💖 parsero(1) — Audit tool for robots.txt of a site 💖

okay bestie, sit down, hydrate, and let me tell you about parsero — the lil audit tool for robots.txt of a site. she's tiny, she's free, she's here to spill the tea on what a website thinks it's hiding. 🕵️♀️✨

🌸 SYNOPSIS 🌸

parsero [-h] [-u URL] [-o] [-sb] [-f FILE]

that's it. that's the whole vibe. five options, maximum slay. 💅

🦋 DESCRIPTION 🦋

so Parsero is a free script written in Python, and what she does is iconic: she reaches out over the network, reads a web server's Robots.txt file, and goes straight for the Disallow entries. 👀

and like, what ARE Disallow entries? glad you asked. they're the lines where a website tells search engines which directories or files hosted on that web server mustn't be indexed. fr fr.

for example, "Disallow: /portal/login" means the content at www.example.com/portal/login is not allowed to be indexed by crawlers like Google, Bing, Yahoo... 🫣

so basically that's the way the administrator has to not share sensitive or private information with the search engines. like, it's the "pls don't look at this" note. and parsero reads it out loud. 😭

✨ OPTIONS ✨

• -h, --help — show the help message and dip. the classic. 🙈
• -u URL — type the URL which will be analyzed. this is your main character input. 🎀
• -o — show only the "HTTP 200" status code. filtered, curated, minimal. ✨
• -sb — search in Bing indexed Disallows. going straight to the source, no cap. 🔍
• -f FILE — scan a list of domains from a list. bulk mode, she's efficient. 👑

🔥 EXAMPLE 🔥

common usage, the everyday serve:

$ parsero -u www.example.com

and using a list of domains from a list:

$ parsero -f /tmp/list-of-domains.txt

see? easy. no cap, you got this. 🫶

🌈 SEE ALSO 🌈

• linkchecker(1)
• proxychains4(1)

👑 AUTHOR 👑

parsero was written by Javier Nieto <javier.nieto@behindthefirewalls.com>.

this manual page was written by Thiago Andrade Marques <andrade@debian.org> for the Debian project (but may be used by others). so generous of them honestly. 💜

and that's the whole page bestie — parsero-0.0+git20140929.e5b585a, dated 18 Aug 2020. old but gold, she ate and left no crumbs. periodt. 💅✨🦋