waymore(1)

📦
version 3.7 · 2 versions
, in ubuntu resolute, ubuntu stonking

waymore(1) 💖

okay bestie, sit DOWN, because waymore is about to become your new hyperfixation ✨ it's an ✨ iconic ✨ little tool from the waymore manual, and I'm gonna walk you through every single flag like we're gossiping at lunch 🫶

✨ Name ✨

waymore — the tool that goes and finds extensive data from online archives. yes, THAT kind of extensive. the way it just. keeps. finding. things. 💅

🌸 SYNOPSIS 🌸

waymore [-h] [-i INPUT] [-n] [-mode {U,R,B}] [-oU OUTPUT_URLS] [-oR OUTPUT_RESPONSES] [-f] [-fc FC] [-mc MC] [-l <signed integer>] [-from <yyyyMMddhhmmss>] [-to <yyyyMMddhhmmss>]
        [-ci {h,d,m,none}] [-ra REGEX_AFTER] [-url-filename] [-xwm] [-xcc] [-xav] [-xus] [-xvt] [-lcc LCC] [-lcy LCY] [-t <seconds>] [-p <integer>] [-r RETRIES] [-m <integer>]
        [-ko [KEYWORDS_ONLY]] [-lr LIMIT_REQUESTS] [-ow] [-nlf] [-c CONFIG] [-wrlr WAYBACK_RATE_LIMIT_RETRY] [-urlr URLSCAN_RATE_LIMIT_RETRY] [-co] [-nd] [-v] [--version]

that's a whole lot of options and honestly? she's worth every one of them. 😌

🦋 DESCRIPTION 🦋

so like. waymore is a versatile tool built to extract comprehensive information from a whole lil friend group of sources, and those sources are:

  • the Wayback Machine
  • Common Crawl
  • Alien Vault OTX
  • URLScan
  • VirusTotal

whether you're out here searching for historical web data or doing a whole security-threat analysis moment, waymore gives you a seamless experience with its intuitive interface and its extensive features. no cap. 💅✨

💅 OPTIONS 💅

buckle up, this is the main character section 👑

  • -h, --help — displays the command usage and options. it's your quick-access bestie for comprehensive assistance, including detailed explanations of all the available options 🫶
  • -i INPUT, --input INPUT — the target domain (or a file of domains) to find links for. this can be a domain only, OR a domain with a specific path. important: if it's a domain only and you want everything for that domain, do not prefix it with "www." 🙈
  • -n, --no-subs — don't include subdomains of the target domain. (only used if the input is not a domain with a specific path) 🚫
  • -mode {U,R,B} — the mode to run: U (retrieve URLs only), R (download Responses only) or B (Both). pick your fighter ✨
  • -oU OUTPUT_URLS, --output-urls OUTPUT_URLS — the file to save the Links output to, including path if necessary. if the "-oR" argument is not passed, a "results" directory gets created in the path specified by the DEFAULT_OUTPUT_DIR key in the config.yml file (typically defaults to "~/.config/waymore/"). within that, a directory gets created with the target domain (or domain with path) passed with "-i" — or, for each line of a file passed with "-i" 📁
  • -oR OUTPUT_RESPONSES, --output-responses OUTPUT_RESPONSES — the directory to save the response output files to, including path if necessary. if the argument is not passed, a "results" directory gets created in the path specified by the DEFAULT_OUTPUT_DIR key in the config.yml file (typically defaults to "~/.config/waymore/"). within that, a directory gets created with the target domain (or domain with path) passed with "-i" — or, for each line of a file passed with "-i" 🗂️
  • -f, --filter-responses-only — the initial links from Wayback Machine will not be filtered (MIME Type and Response Code), only the responses that are downloaded. e.g. it might be useful to still see all available paths from the links even if you don't want to check the content 👀
  • -fc FC — filter HTTP status codes for retrieved URLs and responses. it's a comma separated list of codes (default: the FILTER_CODE values from config.yml). passing this argument overrides the value from config.yml 🔥
  • -mc MC — only Match HTTP status codes for retrieved URLs and responses. comma separated list of codes. passing this argument overrides the config FILTER_CODE and -fc 🎯
  • -l <signed integer>, --limit <signed integer> — how many responses will be saved (if -mode is R or B). a positive value gets the first N results, a negative value gets the last N results. a value of 0 gets ALL responses (default: 5000) 🔢
  • -from <yyyyMMddhhmmss>, --from-date <yyyyMMddhhmmss> — what date to get responses from. if not specified, it gets from the earliest possible results. a partial value can be passed, e.g. 2016, 201805, etc. ⏳
  • -to <yyyyMMddhhmmss>, --to-date <yyyyMMddhhmmss> — what date to get responses to. if not specified, it gets to the latest possible results. a partial value can be passed, e.g. 2016, 201805, etc. 🕰️
  • -ci {h,d,m,none}, --capture-interval {h,d,m,none} — filters the search on Wayback Machine (archive.org) to only get at most 1 capture per hour (h), day (d) or month (m). this filter is used for responses only. the default is 'd' but it can also be set to 'none' to not filter anything and get all responses 🌈
  • -ra REGEX_AFTER, --regex-after REGEX_AFTER — RegEx for filtering purposes against links found across all sources of URLs AND responses downloaded. only positive matches will be output ✨
  • -url-filename — sets the file name of downloaded responses to the URL that generated the response; otherwise it gets set to the hash value of the response. using the hash value means multiple URLs that generated the same response will only result in one file being saved for that response 💾
  • -xwm — exclude checks for links from Wayback Machine (archive.org) 🚫
  • -xcc — exclude checks for links from commoncrawl.org 🚫
  • -xav — exclude checks for links from alienvault.com 🚫
  • -xus — exclude checks for links from urlscan.io 🚫
  • -xvt — exclude checks for links from virustotal.com 🚫
  • -lcc LCC — limit the number of Common Crawl index collections searched, e.g. '-lcc 10' will just search the latest 10 collections (default: 3). as of July 2023 there are currently 95 collections. setting to 0 (default) will search ALL collections. if you don't want to search Common Crawl at all, use the -xcc option 🗃️
  • -lcy LCY — limit the number of Common Crawl index collections searched by the year of the index data. the earliest index has data from 2008. setting to 0 (default) will search collections of any year (but in conjunction with -lcc). for example, if you're only interested in data from 2015 and after, pass -lcy 2015. if you don't want to search Common Crawl at all, use the -xcc option 📅
  • -t <seconds>, --timeout <seconds> — this is for archived responses only! how many seconds to wait for the server to send data before giving up (default: 30 seconds) ⏱️
  • -p <integer>, --processes <integer> — basic multithreading is done when getting requests for a file of URLs. this argument determines the number of processes (threads) used (default: 1) 🧵
  • -r RETRIES, --retries RETRIES — the number of retries for requests that get a connection error or get rate limited (default: 1) 🔁
  • -m <integer>, --memory-threshold <integer> — the memory threshold percentage. if the machine's memory goes above the threshold, the program will be stopped and ended gracefully before running out of memory (default: 95) 🧠
  • -ko [KEYWORDS_ONLY], --keywords-only [KEYWORDS_ONLY] — only return links and responses that contain keywords you're interested in. this can reduce the time it takes to get results. if you provide the flag with no value, keywords are taken from the comma separated list in the "config.yml" file with the "FILTER_KEYWORDS" key; otherwise you can pass a specific Regex value to use, e.g. -ko "admin" to only get links containing the word admin, or -ko ".js(?|$)" to only get JS files. the Regex check is NOT case sensitive. 🔍
  • -lr LIMIT_REQUESTS, --limit-requests LIMIT_REQUESTS — limit the number of requests that will be made when getting links from a source (this doesn't apply to Common Crawl). some targets can return a huge amount of requests needed that are just not feasible to get, so this can be used to manage that situation. this defaults to 0 (Zero) which means there is no limit 🚦
  • -ow, --output-overwrite — if the URL output file (default waymore.txt) already exists, it will be overwritten instead of being appended to 💥
  • -nlf, --new-links-file — if this argument is passed, a .new file will also be written that will contain links for the latest run 🆕
  • -c CONFIG, --config CONFIG — path to the YML config file. if not passed, it looks for the file 'config.yml' in the same directory as the runtime file 'waymore.py' ⚙️
  • -wrlr WAYBACK_RATE_LIMIT_RETRY, --wayback-rate-limit-retry WAYBACK_RATE_LIMIT_RETRY — the number of minutes the user wants to wait for a rate limit pause on Watback Machine (archive.org) instead of stopping with a 429 error (default: 3) 😮‍💨
  • -urlr URLSCAN_RATE_LIMIT_RETRY, --urlscan-rate-limit-retry URLSCAN_RATE_LIMIT_RETRY — the number of minutes the user wants to wait for a rate limit pause on URLScan.io instead of stopping with a 429 error (default: 1) 😮‍💨
  • -co, --check-only — this will make a few minimal requests to show you how many requests, and roughly how long it could take, to get URLs from the sources and downloaded responses from Wayback Machine 📊
  • -nd, --notify-discord — whether to send a notification to Discord when waymore completes. it requires WEBHOOK_DISCORD to be provided in the config.yml file 🔔
  • -v, --verbose — verbose output 🗣️
  • --version — show version number 🔢
part 1 done bestie, don't go anywhere 💖 more of this page is coming right up 🫶

✨ EXAMPLES ✨

Okay bestie, here's the tea on how people actually use this thing 💅

Common usage:

  • Example 1:

Just get the URLs from all sources for redbull.com (-mode U is just for URLs, so no responses get downloaded, we're being chill about bandwidth):

$ waymore -i redbull.com -mode U

The URLs are saved in the same path as config.yml (typically ~/.config/waymore) under results/redbull.com/waymore.txt — so it's giving organized, it's giving findable. 🎀

  • Example 2:

Get ALL the URLs from Wayback for redbull.com (heads up: no filters are applied in mode U with -f, and no URLs are retrieved from Common Crawl, Alien Vault, URLScan and Virus Total, because -xcc, -xav, -xus, -xvt are passed respectively — yes, that's four whole exclusions, she's picky 💅). Save the FIRST 200 responses that are found starting from 2022 (-l 200 -from 2022):

$ waymore -i redbull.com -f -xcc -xav -xus -xvt -l 200 -from 2022
  • Example 3:

You can pipe waymore to other tools — the way that this just works is so iconic 🔥. Any errors are sent to stderr and any links found are sent to stdout. The output file is still created in addition to the links being piped to the next program. However — and this is the important bit, don't skip it bestie — archived responses are not piped to the next program, but they are still written to files. For example:

$ waymore -i redbull.com -mode U | unfurl keys | sort -u

You can also pass the input through stdin instead of -i:

$ cat redbull_subs.txt | waymore
  • Example 4:

Sometimes you may just want to check how many requests, and how long waymore is likely to take if you ran it for a particular domain. You can do a quick check by using the -co/--check-only argument — it's literally a vibe check for your scan, no cap 🦋. For example:

$ waymore -i redbull.com --check-only

👑 Author

Aquila Macedo <aquilamacedo@riseup.net>

Expat

And that's a wrap on waymore(1), bestie 💖 this page is from the waymore project, dated 2024-03-22, and you can pull up the whole thing yourself with man waymore whenever you need a little refresher. Go forth and fetch those URLs, you absolute icon 🌸✨

📦 where this page comes from