URLWATCH-FILTERS(5)

💖 URLWATCH-FILTERS 💖

okay bestie, welcome to URLWATCH-FILTERS(5) from the urlwatch documentation, and buckle up because this one is long — it's split into parts, and this is part 1. we're talking about filtering the output and diff data of your urlwatch jobs. it's giving ✨data glow-up✨. let's get into it 🫶

---

🌸 NAME 🌸

urlwatch-filters — filtering output and diff data of urlwatch jobs. yes, that's the whole vibe. 🎀

---

✨ SYNOPSIS ✨

urlwatch --edit

that's it. that's the entry point. open the config, start cooking. 💅

---

🦋 DESCRIPTION 🦋

so every single job can have two filter stages configured, and inside each stage you can chain one or more filters that get processed one after the other:

• applied to the downloaded page before diffing the changes — this one is called filter
• applied to the diff result before reporting the changes — this one is called diff_filter

while you're building out your filter pipeline, you're probably gonna wanna peek at what the filtered output actually looks like, right? valid. you can do that by first setting up your job, then running urlwatch with the --test-filter command, passing in either the index (from --list) or the URL/location of the job you wanna test:

urlwatch --test-filter 1   # Test the first job in the list
urlwatch --test-filter https://example.net/  # Test the job with the given URL

what you get back is the filtered plaintext of the job — and that's literally the exact thing that would (in a real urlwatch run) be fed into the diff algorithm. main character input energy. 👑

heads up though, this part matters: the filter only gets applied to new content. the old content was already filtered back when it was retrieved. which means changes you make to the filter are not visible when reporting unchanged contents (go check Display for the deets), and your diff output is gonna be between (old content with the filter as it was at the time the old content was retrieved) and (new content with the current filter). so don't come for me when the diff looks weird after you tweak a filter, okay? it's not a bug, it's the timeline. 😭

once urlwatch has collected at least 2 historic snapshots of a job (i.e. two different states of a webpage), you can use the command-line option --test-diff-filter to test out your diff_filter settings — and it'll use historic data cached locally. slay. 💖

---

💅 BUILT-IN FILTERS 💅

wanna see the whole menu of built-in filters? run this:

urlwatch --features

at the moment, these are the built-ins, and there are a lot of them, so let's go:

• beautify — beautify HTML ✨
• css — filter XML/HTML using CSS selectors
• csv2text — convert CSV to plaintext
• element-by-class — get all HTML elements by class
• element-by-id — get an HTML element by its ID
• element-by-style — get all HTML elements by style
• element-by-tag — get an HTML element by its tag
• format-json — convert to formatted json
• grep — filter only lines matching a regular expression
• grepi — remove lines matching a regular expression
• hexdump — convert binary data to hex dump format
• html2text — convert HTML to plaintext
• pdf2text — convert PDF to plaintext
• pretty-xml — pretty-print XML
• ical2text — convert iCalendar (https://en.wikipedia.org/wiki/ICalendar) to plaintext
• ocr — convert text in images to plaintext using Tesseract OCR
• re.sub — replace text with regular expressions using Python's re.sub
• re.findall — find all non-overlapping matches using Python's re.findall
• reverse — reverse input items
• sha1sum — calculate the SHA-1 checksum of the content
• shellpipe — filter using a shell command
• sort — sort input items
• remove-duplicate-lines — remove duplicate lines (case sensitive)
• strip — strip leading and trailing whitespace
• striplines — strip leading and trailing whitespace in each line
• xpath — filter XML/HTML using XPath expressions
• jq — filter, transform and extract values from JSON

iconic lineup fr fr 🌈

---

🎀 PICKING OUT ELEMENTS FROM A WEBPAGE 🎀

you can pull out just one specific HTML element with a built-in filter. say you wanna extract <div id="something">.../<div> from a page — here's what you drop into your urls.yaml:

url: http://example.org/idtest.html
filter:
  - element-by-id: something

and bestie, you can chain these. so you can run html2text on whatever comes out the other side:

url: http://example.net/id2text.html
filter:
  - element-by-id: something
  - html2text

pipeline behavior. we love to see it. 🦋

---

💜 CHAINING MULTIPLE FILTERS 💜

the example urls.yaml file also shows off how the built-in filters work together. here, 3 filters get used: html2text, line-grep and whitespace removal, all so you can grab just one certain info field off a webpage:

url: https://example.net/version.html
filter:
  - html2text
  - grep: "Current.*version"
  - strip

---

🌸 EXTRACTING ONLY THE <BODY> TAG OF A PAGE 🌸

if you only want the body tag, this filter is your girl:

url: https://example.org/bodytag.html
filter:
  - element-by-tag: body

---

⭐ FILTERING BASED ON AN XPATH EXPRESSION ⭐

to filter using an XPath expression (https://www.w3.org/TR/1999/REC-xpath-19991116/), you use the xpath filter like this:

url: https://example.net/xpath.html
filter:
  - xpath: /html/body/marquee

this grabs only the <marquee> elements sitting directly below the <body> element — and that <body> in turn has to be below the <html> element of the document. everything else gets stripped out. bye. 💀

there's also Microsoft's XPath Examples page (https://msdn.microsoft.com/en-us/library/ms256086(v=vs.110).aspx) if you want more examples. and hey, fun trick: you can also grab the XPath of an <html> node straight from the Chromium/Google Chrome developer tools — just right click on the node and select copy XPath. no cap, that's the cheat code. 👑

---

✨ FILTERING BASED ON CSS SELECTORS ✨

to filter based on a CSS selector (https://www.w3.org/TR/2011/REC-css3-selectors-20110929/), you use the css filter like so:

url: https://example.net/css.html
filter:
  - css: ul#groceries > li.unchecked

this would filter only <li class="unchecked"> tags directly below <ul id="groceries"> elements. precise. 💅

there are some limitations and extensions though, and cssselect's documentation explains all of them: https://cssselect.readthedocs.io/en/latest/#supported-selectors

---

🫶 USING XPATH AND CSS FILTERS WITH XML AND EXCLUSIONS 🫶

by default, XPath and CSS filters are set up for HTML documents. but you can absolutely use them for XML documents too — check it, these examples parse an RSS feed and filter only the titles and publication dates:

url: https://example.com/blog/xpath-index.rss
filter:
  - xpath:
      path: '//item/title/text()|//item/pubDate/text()'
      method: xml

url: http://example.com/blog/css-index.rss
filter:
  - css:
      selector: 'item > title, item > pubDate'
      method: xml
  - html2text: re

now, to match an element that lives in an XML namespace (https://www.w3.org/TR/xml-names/), you put a namespace prefix in front of the tag name. use a : to separate the namespace prefix and the tag name in an XPath expression, and use a | in a CSS selector:

url: https://example.net/feed/xpath-namespace.xml
filter:
  - xpath:
      path: '//item/media:keywords/text()'
      method: xml
      namespaces:
        media: http://search.yahoo.com/mrss/

url: http://example.org/feed/css-namespace.xml
filter:
  - css:
      selector: 'item > media|keywords'
      method: xml
      namespaces:
        media: http://search.yahoo.com/mrss/
  - html2text

alternatively — and this is the sneaky shortcut — you can use the XPath expression //*[name()='<tag_name>'] to bypass the namespace entirely. 😌

another genuinely useful option with XPath and CSS filters is exclude. any elements selected by this exclude expression get removed from the final result. for example, this job will end up with no <a> tag at all in its results:

url: https://example.org/css-exclude.html
filter:
  - css:
      selector: body
      exclude: a

---

🌈 LIMITING THE RETURNED ITEMS FROM A CSS SELECTOR OR XPATH 🌈

if you only want a subset of the items that a CSS selector or XPath filter hands back, there are two extra subfilters for that:

• skip — how many elements to skip from the beginning (default: 0)
• maxitems — how many elements to return at most (default: no limit)

so like, if a page has multiple elements but you only want the second and third matching element (skip the first, return at most two), you'd use this filter:

url: https://example.net/css-skip-maxitems.html
filter:
  - css:
      selector: div.cpu
      skip: 1
      maxitems: 2

🍓 Dealing with duplicated results 🍓

if you're getting multiple results on one page but you only expected one — say, because the page contains both a mobile and a desktop version in the same HTML document and just shows/hides one via CSS depending on the viewport size — you can use maxitems: 1 to only return the first item. problem solved, bestie. 🙈

---

🧚 FIXING LIST REORDERINGS WITH CSS SELECTOR OR XPATH FILTERS 🧚

sometimes the ordering of items on a webpage changes around on the regular without the actual content changing at all. and by default, that shows up in the diff output as an element being removed from one part of the page and inserted into another part of the page. so annoying. 😭

but when the order of items genuinely doesn't matter, you can sort the matched items lexicographically to dodge those spurious reports that only happen because the ordering changed on the page.

the subfilter for css and xpath filters is sort, and it can be true or false (the default):

url: https://example.org/items-random-order.html
filter:
  - css:
      selector: span.item
      sort: true

---

💖 FILTERING PDF DOCUMENTS 💖

to monitor the text inside a PDF file, you use the pdf2text filter. it needs the pdftotext library installed (https://github.com/jalan/pdftotext/blob/master/README.md#pdftotext), plus any of its OS-specific dependencies (https://github.com/jalan/pdftotext/blob/master/README.md#os-dependencies).

real talk: this filter must be the first filter in a chain of filters, because it consumes binary data and outputs text data. no putting it in the middle, okay? 💅

url: https://example.net/pdf-test.pdf
filter:
  - pdf2text
  - strip

and if the PDF is password protected, you can hand over its password like this:

url: https://example.net/pdf-test-password.pdf
filter:
  - pdf2text:
      password: urlwatchsecret
  - strip

---

🎀 DEALING WITH CSV INPUT 🎀

the csv2text filter turns CSV data into a prettier textual representation. you do this by supplying a format_string, which is a python format string (https://docs.python.org/3/library/string.html#format-string-syntax).

if your CSV has a header, the format string should use the header names lowercased. so let's say we've got a CSV file with data like this:

Name;Company
Smith;Initech
Doe;Initech

a possible format string for that CSV (and note the lowercase keys, bestie):

Mr {name} works at {company}

if there's no header row, you'll need to use the numeric array notation instead:

Mr {0} works at {1}

and you can force the use of numeric indices with the flag ignore_header.

the key has_header can be used to force using the first line as a header, or to force ignoring the first line as a header. otherwise, csv.Sniffer (https://docs.python.org/3/library/csv.html#csv.Sniffer) gets used. that's the auto-detect vibe. ✨

🔄 SORTING OF WEBPAGE CONTENT ✨

okay so sometimes a webpage is out here being chaotic — same data every time, but it shows up in a random order and it's giving absolutely unhinged. bestie, if that's happening, you can just sort before the comparison and restore the peace 💅

url: https://example.net/sorting.txt
filter:
  - sort

the sort filter takes an optional separator parameter that defines the item separator (ngl, by default sorting is line-based), like for example to sort text paragraphs — text separated by an empty line, iykyk:

url: http://example.org/paragraphs.txt
filter:
  - sort:
      separator: "\n\n"

and you can lowkey combine this with a boolean reverse option, which is iconic for sorting and reversing with the same separator (using % as separator, this would turn 3%2%4%1 into 4%3%2%1, no cap):

url: http://example.org/sort-reverse-percent.txt
filter:
  - sort:
      separator: '%'
      reverse: true

---

🔁 REVERSING OF LINES OR SEPARATED ITEMS 🦋

to reverse the order of items without sorting, the reverse filter is your main character. by default it reverses lines:

url: http://example.com/reverse-lines.txt
filter:
  - reverse

this behavior can be changed by using an optional separator string argument (e.g. items separated by a pipe (|) symbol, as in 1|4|2|3, which would be reversed to 3|2|4|1 — ate that fr fr):

url: http://example.net/reverse-separator.txt
filter:
  - reverse: '|'

alternatively, the filter can be specified more verbose with a dict 💖. in this example "\n\n" is used to separate paragraphs (items that are separated by an empty line):

url: http://example.org/reverse-paragraphs.txt
filter:
  - reverse:
      separator: "\n\n"

---

👀 WATCHING GITHUB RELEASES AND GITLAB TAGS 👑

this is an example how to watch the GitHub "releases" page for a given project for the latest release version, so you get notified of new releases — literally the dream setup:

url: https://github.com/tulir/gomuks/releases
filter:
  - xpath:
      path: //*[@class="Link--primary Link"]
      maxitems: 1
  - html2text:

and this is the corresponding version for Github tags:

url: https://github.com/thp/urlwatch/tags
filter:
  - xpath:
      path: //*[@class="Link--primary Link"]
      maxitems: 1
  - html2text:

and for Gitlab tags:

url: https://gitlab.com/chinstrap/gammastep/-/tags
filter:
  - xpath: (//a[contains(@class,"item-title ref-name")])[1]
  - html2text

alternatively, jq can be used for filtering — she's versatile like that:

url: https://api.github.com/repos/voxpupuli/puppet-rundeck/tags
filter:
  - jq: '.[0].name'

---

🧚 FIND, REMOVE OR REPLACE TEXT USING REGULAR EXPRESSIONS 💅

you can use re.sub and re.findall to apply regular expressions ✨

re.sub can be used to remove or replace all non-overlapping instances of matched text. the following example applies the filter 3 times:

1. Just specifying a string as the value will replace the matches with the empty string.
2. Simple patterns can be replaced with another string using "pattern" as the expression and "repl" as the replacement.
3. You can use groups (()) and back-reference them with \1 (etc..) to put groups into the replacement string.

repl defaults to the empty string, which will remove matched strings.

url: https://example.com/regex-substitute.html
filter:
    - re.sub: '\s*href="[^"]*"'
    - re.sub:
        pattern: '<h1>'
        repl: 'HEADING 1: '
    - re.sub:
        pattern: '</([^>]*)>'
        repl: '<END OF TAG \1>'

re.findall can be used to find all non-overlapping matches of a regular expression. each match is output on its own line. the following example applies the filter twice:

1. It uses a group (()) and back-reference (\1) to extract a date from the input string.
2. It breaks the numbers in the date out into separate lines.

if repl is not specified, the full match will be included in the output.

url: https://example.com/regex-findall.html
filter:
    - re.findall:
        pattern: 'The next draw is on (\d{4}-\d{2}-\d{2}).'
        repl: '\1'
    - re.findall: '\d+'

note bestie: when using HTML or XML, it is usually better to use CSS selectors or XPATH expressions. HTML and XML cannot be parsed https://stackoverflow.com/a/1732454/1047040 properly using regular expressions 😭. if the CSS selector or XPATH cannot provide the targeted selection required, using an html2text filter first then using re.findall can be a good pattern.

if you want to enable flags (e.g. re.MULTILINE) in re.sub or re.findall filters, use an "inline flag" — here are some examples:

• re.MULTILINE: (?m) (Makes ^ match start-of-line and $ match end-of-line)
• re.DOTALL: (?s) (Makes . also match a newline)
• re.IGNORECASE: (?i) (Perform case-insensitive matching)

this allows you, for example, to remove all leading spaces (only space character and tab) — obsessed with the efficiency:

url: http://example.com/leading-spaces.txt
filter:
  - re.sub: '(?m)^[ \t]*'

---

🐚 USING A SHELL SCRIPT AS A FILTER 💀

while the built-in filters are powerful for processing markup such as HTML and XML, in some cases you might already know how you would filter your content using a shell command or shell script. the shellpipe filter allows you to start a shell and run custom commands to filter the content. she's the moment fr.

the text data to be filtered will be written to the standard input (stdin) of the shell process and the filter output will be taken from the shell's standard output (stdout).

for example, if you want to use grep tool with the case insensitive matching option (-i) and printing only the matching part of the line (-o), you can specify this as shellpipe filter:

url: https://example.net/shellpipe-grep.txt
filter:
  - shellpipe: "grep -i -o 'price: <span>.*</span>'"

this feature also allows you to use sed(1), awk(1) and perl(1) one-liners for text processing (of course, any text tool that works in a shell can be used, no gatekeeping here 🫶). for example, this awk(1) one-liner prepends the line number to each line:

url: https://example.net/shellpipe-awk-oneliner.txt
filter:
  - shellpipe: awk '{ print FNR " " $0 }'

you can also use a multi-line command for a more sophisticated shell script (| in YAML denotes the start of a text block — iconic YAML behaviour):

url: https://example.org/shellpipe-multiline.txt
filter:
  - shellpipe: |
      FILENAME=`mktemp`
      # Copy the input to a temporary file, then pipe through awk
      tee $FILENAME | awk '/The numbers for (.*) are:/,/The next draw is on (.*)./'
      # Analyze the input file in some other way
      echo "Input lines: $(wc -l $FILENAME | awk '{ print $1 }')"
      rm -f $FILENAME

within the shellpipe script, two environment variables will be set for further customization (this can be useful if you have an external shell script file that is used as filter for multiple jobs, but needs to treat each job in a slightly different way — very main character energy of you):

• $URLWATCH_JOB_NAME — The name of the job (name key in jobs YAML)
• $URLWATCH_JOB_LOCATION — The URL of the job, or command line (for shell jobs)

---

🔍 CONVERTING TEXT IN IMAGES TO PLAINTEXT 🌸

the ocr filter uses the Tesseract OCR engine https://github.com/tesseract-ocr to convert text in images to plain text. it requires two Python modules to be installed: pytesseract https://github.com/madmaze/pytesseract and Pillow https://python-pillow.org. any file formats supported by Pillow (PIL) are supported. slay.

⚠️ this filter must be the first filter in a chain of filters, since it consumes binary data and outputs text data. do not rearrange her.

url: https://example.net/ocr-test.png
filter:
  - ocr:
      timeout: 5
      language: eng
  - strip

the subfilters timeout and language are optional:

• timeout: Timeout for the recognition, in seconds (default: 10 seconds)
• language: Text language (e.g. fra or eng+fra, default: eng)

---

🧃 FILTERING JSON RESPONSE DATA USING JQ SELECTORS ⭐

the jq filter uses the Python bindings for jq https://stedolan.github.io/jq/, a lightweight JSON processor. use of this filter requires the optional jq Python module https://github.com/mwilliamson/jq.py to be installed.

url: https://example.net/jobs.json
filter:
   - jq:
      query: '.[].title'

the subfilter query is optional:

• query: A valid jq filter string.

supports aggregations, selections, and the built-in operators like length. for more information on the operations permitted, see the jq Manual https://stedolan.github.io/jq/manual/. go read it, bestie, it's lowkey fun.

---

📁 FILES 🎀

• $XDG_CONFIG_HOME/urlwatch/urls.yaml

---

👯 SEE ALSO 💖

urlwatch(1), urlwatch-intro(5), urlwatch-jobs(5)

---

©️ COPYRIGHT 🌈

2024 Thomas Perl

---

and that's a wrap on the whole thing, bestie 🫶 this was urlwatch, page URLWATCH-FILTERS(5), dated Oct 28, 2024 — go forth and filter everything, you absolute icon 👑💅✨