XIDEL(1)

📦 version 2022-11, in nixos 26.05, nixpkgs unstable

✨ XIDEL ✨

okay bestie, welcome to Xidel — your new fave command line tool 💅 (from the User Manuals section, no big deal, she's iconic 👑)

🌸 NAME 🌸

xidel — it's a command line tool to download and extract data from HTML/XML pages. literally that's her whole job and she ate 💖

🦋 SYNOPSIS 🦋

the baby-steps trivial usage: extract an expression from a webpage, like this 🎀

xidel http://www.example.org --extract //title

the next important option is --follow, which follows links on a page 🫶 so this example right here prints the titles of all the pages that are linked from http://www.example.org:

xidel http://www.example.org --follow //a --extract //title

⭐ DESCRIPTION ⭐

Xidel supports... a LOT. get comfy 🍓

Extract expressions — there are a few different kinds of extract expressions:

  • CSS 3 Selectors — for extracting simple elements ✨
  • XPath 3 — for extracting values and calculating things with them 🔥
  • XQuery 3 — for creating new documents from the extracted values 💜
  • JSONiq — for working with JSON apis
  • Templates — for extracting several expressions in an easy way, using an annotated version of the page for pattern-matching 🎀
  • Multipage templates — i.e. a file that contains templates for several pages
  • XPath 2 / XQuery 1 — for legacy queries (respect the elders 🙏)

Following:

  • HTTP Codes — redirections like 30x are automatically followed, while keeping things like cookies 🍪
  • Links — it can follow all links on a page as well as some extracted values

Output formats:

  • Adhoc — just prints the data in a human readable format
  • XML — encodes the data in XML
  • HTML — encodes the data in HTML
  • JSON — encodes the data in a JSON
  • bash/cmd — exports the data as shell variables 🐚

Connections: HTTP / HTTPS as well as local files or stdin 💫

Systems: Windows (using wininet), Linux (using synapse+openssl), Mac (with newest synapse) 🌈

💖 OPTIONS 💖

  • Call xidel --help to see a list of support command line options 👀
  • Call xidel --usage for a full reference 📖

🧚 EXAMPLES - Basics 🧚

  1. Print all urls found by a google search.
xidel http://www.google.de/search?q=test --extract "//a/extract(@href, 'url[?]q=([^&]+)&', 1)[. != '']"
  1. Print the title of all pages found by a google search and download them:
xidel http://www.google.de/search?q=test --follow "//a/extract(@href, 'url[?]q=([^&]+)&', 1)[. != '']" --extract //title --download '{$host}/'

Generally follow all links on a page and print the titles of the linked pages:

  1. Another template example:

If you have an example.xml file like <x><foo>ood</foo><bar>IMPORTANT!</bar></x>, you can read the important part like:

xidel example.xml -e "<x><foo>ood</foo><bar>{.}</bar></x>"

(and this will also check, if the part with the ood is there, and fail otherwise 😭)

  1. Calculate something with XPath:
xidel -e "(1 + 2 + 3) * 1000000000000 + 4 + 5 + 6 + 7.000000008"
yes, the list jumps from 3 to 5, no I don't make the rules, I just report them 💀
  1. Print all newest stackoverflow questions with title and url:
xidel http://stackoverflow.com -e "<A class='question-hyperlink'>{title:=text(), url:=@href}</A>*"
  1. Print all reddit comments of an user, with html and url:
xidel "http://www.reddit.com/user/username/" --extract "<t:loop><div class='usertext-body'><div>{outer-xml(.)}</div></div><ul class='flat-list buttons'><a><t:s>link:=@href</t:s>permalink</a></ul></div></div></t:loop>" --follow "<a rel='nofollow next'>{.}</a>?"
  1. Check if your reddit letter is red:

Webscraping, combining CSS, XPath, JSONiq and automatically form evaluation:

xidel http://reddit.com -f "form(css('form.login-form')[1], {'user': '$your_username', 'passwd': '$your_password'})" -e "css('#mail')/@title"

Using the Reddit API:

xidel -d "user=$your_username&passwd=$your_password&api_type=json" https://ssl.reddit.com/api/login --method GET 'http://www.reddit.com/api/me.json' -e '($json).data.has_mail'
  1. Use XQuery, to create a html table of odd and even numbers:
xidel --xquery '<table>{for $i in 1 to 1000 return <tr><td>{$i}</td><td>{if ($i mod 2 = 0) then "even" else "odd"}</td></tr>}</table>' --output-format xml
  1. Export variables to bash
eval "$(xidel http://site -e 'title:=//title' -e 'links:=//a/@href' --output-format bash)"

This sets the bash variable $title to the title of the page and $links becomes an array of all links there 🫶

🌈 ENVIRONMENT 🌈

Use XIDEL_OPTIONS to set global command line options. that's the whole section, she's short and sweet 💜

👑 AUTHOR 👑

Benito van der Zander, <benito_NOSPAM_benibela.de>, http://www.benibela.de

Download link: http://sourceforge.net/projects/videlibri/files/Xidel/

You can test it online on http://www.videlibri.de/cgi-bin/xidelcgi or directly by sending a request to the cgi service like:

http://www.videlibri.de/cgi-bin/xidelcgi?data=<html><title>foobar</title></html>&extract=//title&raw=true

💀 SEE ALSO 💀

wget(1), curl(1)

and that's the whole manual bestie, slay 🎀 this page is coming at you from Linux, dated JUNE 2016 — vintage, but like, in a main character way ⭐💅

📦 where this page comes from