URLWATCH-COOKBOOK(7)

📦
version 2.29 · 3 versions
, in arch rolling, debian trixie, fedora 43, fedora 44, nixos 26.05, nixpkgs unstable, ubuntu questing, ubuntu resolute, ubuntu stonking

✨ URLWATCH-COOKBOOK(7) ✨

okay bestie, this is the urlwatch-cookbook page from the urlwatch documentation, and it's literally the advanced topics and recipes situation 💅 it's long so it comes in two parts, this is part one, we're not done yet 🫶

🌸 NAME 🌸

urlwatch-cookbook — advanced topics and recipes for urlwatch 💖

🎀 ADDING URLS FROM THE COMMAND LINE 🎀

you can just... add a new URL to the job list straight from the command line, no cap:

urlwatch --add url=http://example.org,name=Example

so easy, so slay ✨

🦋 USING WORD-BASED DIFFERENCES 🦋

so you can also plug in an external diff-style tool — and by that I mean a tool that takes two filenames (old, new) as parameters and returns the difference of the files on its standard output 👑

like, say you want word-based differences instead of line-based ones, you'd use wdiff(1). or if you want markdown differences, there's pandiff at https://github.com/davidar/pandiff:

url: https://example.com/
diff_tool: wdiff
ngl important: diff_tool specifies an external command-line tool, so that tool has to be installed separately (think apt install wdiff on Debian, or brew install wdiff on macOS). syntax highlighting is supported for wdiff-style output, but potentially not for other diff tools 😭

🧚 IGNORING WHITESPACE CHANGES 🧚

if you want to ignore whitespace changes so you're not getting notifications for the most trivial little differences ever, diff_tool is your girl. like:

diff_tool: "diff --ignore-all-space --unified"

and when you're using another external diff-like tool, make sure it returns unified output format so you keep that syntax highlighting 💅

🌈 ONLY SHOW ADDED OR REMOVED LINES 🌈

the diff_filter feature lets you filter the diff output text with the exact same tools (see Filters) you use for filtering web pages. iconic.

to show only diff lines with added lines:

url: http://example.com/things-get-added.html
diff_filter:
  - grep: '^[@+]'

this only keeps diff lines starting with @ or +. and similarly, to only keep removed lines:

url: http://example.com/things-get-removed.html
diff_filter:
  - grep: '^[@-]'

if you want more sophisticated diff filtering, it's possible by combining existing filters, writing a new filter, or using shellpipe to delegate the filtering/processing of the diff output to an external tool 💖

read the next section if you wanna disable empty notifications 👀

💜 DISABLE EMPTY NOTIFICATIONS 💜

as an extension of the previous example, let's say you only want to get notified with all lines added, but receive no notifications at all if lines are removed. vibe check passed.

a diff usually looks like this:

--- @       Fri, 04 Mar 2022 19:58:14 +0100
+++ @       Fri, 04 Mar 2022 19:58:22 +0100
@@ -1,3 +1,3 @@
 someline
-someotherlines
+someotherline
 anotherline

we want to filter all lines starting with "+" only, but because of the headers we also want to filter lines that start with "+++", which can be accomplished like so:

url: http://example.com/only-added.html
diff_filter:
  - grep: '^[+]'      # Include all lines starting with "+"
  - grepi: '^[+]{3}'  # Exclude the line starting with "+++"

this deals with all diff lines now, but since urlwatch reports "changed" pages even when the diff_filter returns an empty string (which might be useful in some cases), you have to explicitly opt out by using urlwatch --edit-config and setting the empty-diff option to false in the display category:

display:
  empty-diff: false

🍓 PASS DIFF OUTPUT TO A CUSTOM SCRIPT 🍓

in some situations it might be useful to run a script with the diff as input when changes were detected (e.g. to start an update or process something). this is doable by combining diff_filter with the shellpipe filter, which can be any custom script you want 💅

heads up bestie: the output of the custom script will then be the diff result as reported by urlwatch, so if it outputs any status, the CHANGED notification that urlwatch does will contain the output of the custom script, not the original diff 😭

this can even have a "normal" filter attached to only watch links (the css: a part of the filter definitions):

url: http://example.org/downloadlist.html
filter:
  - css: a
diff_filter:
  - shellpipe: /usr/local/bin/process_new_links.sh

👑 COMPARING WEB PAGES VISUALLY 👑

to compare the visual contents of web pages, Nicolai wrote pyvisualcompare at https://github.com/nspo/pyvisualcompare as a frontend (with GUI) to urlwatch 🔥 the tool can be used to select a region of a web page. it then generates a configuration for urlwatch to run pyvisualcompare and generate a hash for the screen contents. main character tool fr fr.

🩷 IGNORING CONNECTION ERRORS 🩷

in some cases it might be useful to ignore (temporary) network errors so notifications don't get sent. while there is a display.error config option (defaulting to true) to control reporting of errors globally, if you want to ignore network errors for specific jobs only, you can use the ignore_connection_errors key in the job list configuration file:

url: https://example.com/
ignore_connection_errors: true

similarly, you might wanna ignore some (temporary) HTTP errors on the server side:

url: https://example.com/
ignore_http_error_codes: 408, 429, 500, 502, 503, 504

or ignore all HTTP errors if you like:

url: https://example.com/
ignore_http_error_codes: 4xx, 5xx

you can also ignore incomplete reads:

url: "https://example.com/"
ignore_incomplete_reads: true

🌸 OVERRIDING THE CONTENT ENCODING 🌸

for web pages with misconfigured HTTP headers or rare encodings, it may be useful to explicitly specify an encoding from Python's Standard Encodings (https://docs.python.org/3/library/codecs.html#standard-encodings) 💖

url: https://example.com/
encoding: utf-8

⏰ CHANGING THE DEFAULT TIMEOUT ⏰

by default, url jobs timeout after 60 seconds. if you want a different timeout period, use the timeout key to specify it in number of seconds, or set it to 0 to never timeout:

url: https://example.com/
timeout: 300

no cap, zero means forever 👑

it is possible to add cookies to HTTP requests for pages that need it. the YAML syntax for this is:

url: http://example.com/
cookies:
    Key: ValueForKey
    OtherKey: OtherValue

snack time 🍪✨

🦋 COMPARING WITH SEVERAL LATEST SNAPSHOTS 🦋

if a webpage frequently changes between several known stable states, it may be desirable to have changes reported only if the webpage changes into a new unknown state. you can use compared_versions to do this:

url: https://example.com/
compared_versions: 3

in this example, changes are only reported if the webpage becomes different from the latest three distinct states. the differences are shown relative to the closest match 💅

💌 RECEIVING A REPORT EVERY TIME URLWATCH RUNS 💌

if you're watching pages that change seldomly, but you still want to be notified daily if urlwatch still works, you can watch the output of the date command, for example:

name: "urlwatch watchdog"
command: "date"

since the output of date changes every second, this job should produce a report every time urlwatch is run 🔥

🌈 USING REDIS AS A CACHE BACKEND 🌈

if you want to use Redis as a cache backend over the default SQLite3 file:

urlwatch --cache=redis://localhost:6379/
no migration path from the SQLite3 format, bestie — the cache will be empty the first time Redis is used 😭

🧅 WATCHING CHANGES ON .ONION (TOR) PAGES 🧅

since pages on the Tor Network (https://www.torproject.org) are not accessible via public DNS and TCP, you need to either configure a Tor client as HTTP/HTTPS proxy, or use the torify(1) tool from the tor package (apt install tor on Debian, brew install tor on macOS). setting up Tor is out of scope for this document. on a properly set up Tor installation, one can just prefix the urlwatch command with the torify wrapper to access .onion pages:

torify urlwatch

lowkey magical 🧚

📘 WATCHING FACEBOOK PAGE EVENTS 📘

if you want to be notified of new events on a public Facebook page, you can use the following job pattern — replace PAGE with the name of the page (can be found by navigating to the events page on your browser):

url: http://m.facebook.com/PAGE/pages/permalink/?view_type=tab_events
filter:
  - css:
      selector: div#objects_container
      exclude: 'div.x, #m_more_friends_who_like_this, img'
  - re.sub:
      pattern: '(/events/\d*)[^"]*'
      repl: '\1'
  - html2text: pyhtml2text

🎀 SETTING THE CONTENT WIDTH FOR HTML2TEXT (LYNX METHOD) 🎀

when using the lynx method in the html2text filter, it uses a default width that will cause additional line breaks to be inserted 😭

to set the lynx output width to 400 characters, use this filter setup:

url: http://example.com/longlines.html
filter:
  - html2text:
      method: lynx
      width: 400

💅 CONFIGURING HOW LONG BROWSER JOBS WAIT FOR PAGES TO LOAD 💅

for browser jobs, you can configure how long the headless browser will wait before a page is considered loaded by using the wait_until option.

it can take one of four values (see the wait_until docs at https://playwright.dev/python/docs/api/class-page#page-goto-option-wait-until of Playwright):

  • load — consider operation to be finished when the load event is fired
  • domcontentloaded — consider operation to be finished when the DOMContentLoaded event is fired
  • networkidle — discouraged — consider operation to be finished when there are no network connections for at least 500 ms. Don't use this method for testing, rely on web assertions to assess readiness instead
  • commit — consider operation to be finished when network response is received and the document started loading

🔥 TREATING NEW JOBS AS CHANGED 🔥

in some cases (e.g. when the diff_tool or diff_filter executes some external command as a side effect that should also run for the initial page state), you can set treat_new_as_changed to true, which will make the job report as CHANGED instead of NEW the first time it is retrieved (and the diff will be reported, too) 👑

url: http://example.com/initialpage.html
treat_new_as_changed: true

this option will also change the behavior of --test-diff-filter, and allow testing the diff filter if only a single version of the page has been retrieved. periodt.

💖 MONITORING THE SAME URL IN MULTIPLE JOBS 💖

because urlwatch uses the url/navigate (for URL/Browser jobs) and/or the command (for Shell jobs) key as unique identifier, each URL can only appear in a single job. if you want to monitor the same URL multiple times, you can append #1, #2, ... (or anything that makes them unique) to the URLs, like this:

name: "Looking for Thing A"
url: http://example.com/#1
filter:
  - grep: "Thing A"
---
name: "Looking for Thing B"
url: http://example.com/#2
filter:
  - grep: "Thing B"

two jobs, one URL, zero drama 💅

💖 UPDATING A URL AND KEEPING PAST HISTORY

okay so here's the tea bestie: job history is stored based on the value of the url parameter. so if you go and change a job's URL in your config file urls.yaml, urlwatch is gonna be like "new phone who dis" and create a brand new job with no history. 💀

but!! you can keep that history with --change-location:

urlwatch --change-location http://example.org#old http://example.org#new

and this slays for Browser and Shell jobs too — it changes navigate and command respectively. periodt. ✨

🎀 RUNNING A SUBSET OF JOBS

not tryna run every single job? valid. to run one or more specific jobs instead of all known jobs, just hand urlwatch the job index numbers. for example, to run jobs with index 2, 4, and 7:

urlwatch 2 4 7

so efficient. so main character. 👑

📮 SENDING HTML FORM DATA USING POST

wanna simulate submitting a HTML form using the POST method? you literally just pass the form fields in the data field of the job description, like:

name: "My POST Job"
url: http://example.com/foo
data:
  username: "foo"
  password: "bar"
  submit: "Send query"

by default, the request will use the HTTP POST method, and the Content-type will be set to application/x-www-form-urlencoded. it's giving form submission but make it automated. 💅

🚀 SENDING ARBITRARY DATA USING HTTP PUT

you can also customize the HTTP method and the Content-type header, which lets you send arbitrary requests to the server:

name: "My PUT Request"
url: http://example.com/item/new
method: PUT
headers:
  Content-type: application/json
data: '{"foo": true}'

custom method, custom header, custom payload — the full fit. 🔥

🔍 SEE ALSO

🪟 UTF-8 SUPPORT ON WINDOWS

okay so on Windows, the default file encoding might be locale-specific, and that means it won't work correctly if your files are saved using the (recommended) UTF-8 encoding. 🙈

if you're having problems loading UTF-8-encoded files on Windows, you might see an issue like this one when urlwatch parses your config files:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 214: character maps to <undefined>

to work around this issue, Python 3.7 and newer have a new UTF-8 Mode <https://peps.python.org/pep-0540/> that you can enable by setting the environment variable PYTHONUTF8 to 1:

set PYTHONUTF8=1
urlwatch

you can also add this environment variable to your user environment or system environment to apply the UTF-8 Mode to all Python programs on your machine. glow up your whole setup, honestly. ✨

2024 Thomas Perl

and that's the whole cookbook, bestie — you ate. 🫶 this page comes from the urlwatch manual, dated Oct 28, 2024, and you can find it as URLWATCH-COOKBOOK(7). go forth and watch those URLs, icon. 💖🦋🌸

📦 where this page comes from