SCRAPY(1)

✨ NAME ✨

scrapy — the Scrapy command-line tool 💖 (and yes bestie, this is the General Commands Manual entry for it, so you're in exactly the right place)

🌸 SYNOPSIS 🌸

scrapy [command] [OPTIONS] ...

🦋 DESCRIPTION 🦋

Okay so the way you actually talk to Scrapy is through the scrapy command-line tool. That's the whole vibe. The script hands you a bunch of different commands, each one built for a different purpose.

And here's the thing you gotta internalise: every single command has its own syntax. Like, its own personality, its own rules. In other words — each command supports a different set of arguments and options. No universal cheat code, sorry bestie 😭

💅 OPTIONS 💅

🍓 fetch [OPTION] URL

Go grab a URL using the Scrapy downloader. Fetch is fetch, she just gets it. 💅

• --headers — print the response HTTP headers instead of the body. Just the receipts, not the whole meal.

🎀 runspider [OPTION] spiderfile

Run a spider. That's it, that's the job. 🕷️✨

• --output=FILE — store scraped items to FILE in XML format.

👑 settings [OPTION]

Query Scrapy settings. Like a little interrogation but for your config. 💜

• --get=SETTING — print the raw setting value.
• --getbool=SETTING — print the setting value, interpreted as a boolean.
• --getint=SETTING — print the setting value, interpreted as an integer.
• --getfloat=SETTING — print the setting value, interpreted as a float.
• --getlist=SETTING — print the setting value, interpreted as a float. (no cap, that's literally what the page says — I'm just the messenger 🫶)
• --init — print the initial setting value (before loading extensions and spiders).

🧚 shell URL | file

Launch the interactive scraping console. Main character energy, but for debugging. 🌈

⭐ startproject projectname

Create a new project with an initial project template. Fresh start, fresh project, slay. 🎀

🌸 The general ones (they go with everything) 🌸

• --help, -h — print command help and options.
• --logfile=FILE — log file. If it's omitted, stderr will be used.
• --loglevel=LEVEL, -L LEVEL — log level (default: None).
• --nolog — disable logging completely. Silence. Blessed silence. 🤫
• --spider=SPIDER — always use this spider when the arguments are urls.
• --profile=FILE — write python cProfile stats to FILE.
• --lsprof=FILE — write lsprof profiling stats to FILE.
• --pidfile=FILE — write process ID to FILE.
• --set=NAME=VALUE, -s NAME=VALUE — set/override a setting (may be repeated 👀 so go off).

💖 AUTHOR 💖

Scrapy was written by the Scrapy Developers. Iconic behaviour, honestly. 👑

This manual page was written by Ignace Mouzannar mouzannar@gmail.com, for the Debian project (but may be used by others). We love a generous king 🫶

And that's the whole page, bestie — sourced from the General Commands Manual, dated October 17, 2009. Go forth and scrape responsibly. Periodt. 💅✨