ratt(5)

💅 ratt(5) — lua config files syntax 💅

okay so welcome to ratt(5), bestie — it’s from the File Formats Manual and it’s literally the whole vibe guide for ratt’s Lua config files. no cap, this page is about how the config syntax works so ratt can pull data from a website and build the rss/atom feed. let’s get into it 🫶

✨ NAME ✨

ratt — lua config files syntax. that’s the name section, short and iconic.

🌸 SYNOPSIS 🌸

ratt reads config files in the current directory, then searches in ~/.config/ratt/*.lua, and then searches the embedded configs. yes bestie, that’s the lookup order — current dir first, then your config dir, then the embedded ones.

🦋 SYNTAX 🦋

Config files are lua scripts. There are some defined attributes that ratt will interpret and use to extract data from the website and build the rss/atom feed.

The main function is ratt.add(), which takes two arguments. The first is the regex for matching the website url and the second is the selectors table.

💫 regex 💫

Config files are matched with the url provided and the regex in the config file. The regular expression has the syntax of go regexp described at https://golang.org/s/re2syntax.

regex: https://www.youtube.com/results.*

🎀 CSS SELECTORS ATTRIBUTES 🎀

feed and item attributes can be either css selectors (strings) or lua functions. highkey flexible, lowkey powerful.

🪩 selectors 🪩

is a whole object that defines how the data will be extracted. selectors object has the following attributes: httpsettings, feed, item, nextpage, nextpageattr, nextpagecount, sort.

📡 httpsettings 📡

(optional) describes the http request details, which ratt will send to obtain the website data.

Every css selector has an accompanying attribute with the suffix 'attr'. If this attribute isn’t empty, ratt will not get the data of the selected element, but the data from the element’s attribute. e.g.:

HTML:

<span class="mydate" data-date="1970-01-01">the beginning</span>

Config:

...
item = {
     created = "span.mydate",
     createdattr = "data-date",
...

This will extract "1970-01-01" to the item.created, and not the element’s data "the beginning". the way that attribute pull is doing exactly what it’s supposed to 💅

🍓 feed 🍓

object which defines how ratt will obtain the data for the feed details.

Example:

feed = {
    title = "h1",
    description = "p.description",
    authorname = "span.author",
    authoremail = "",

feed.title

feed title (can be css selector or lua function)

feed.titleattr

attribute name of the selected element for the feed.title

feed.description

feed description (can be css selector or lua function)

feed.descriptionattr

attribute name of the selected element for the feed.description

feed.authorname

feed author’s name (can be css selector or lua function)

feed.authornameattr

attribute name of the selected element for the feed.authorname

feed.authoremail

feed author’s email (can be css selector or lua function)

feed.authoremailattr

attribute name of the selected element for the feed.authoremail

🧚 item 🧚

object which defines how ratt will obtain the data of the feed items. item data are obtained in parallel. parallel extraction is giving main character energy, fr fr.

item.container

individual items are selected with this attribute. This selector extracts the html elements that contain all the data that the item needs. Later item attributes run against this element. So if the items are all in a div e.g.:

<div class="article-box">
     <a href="https://article/url"><h3>example article title</h3></a>
     <p>some article description</p>
</div>

The selector will be:

item = {
     container = ".article-box",

Container can be also just a number or a lua function. In that case it is required, that the lua function will return a integer, with the number of items.

item.title

item title (can be css selector or lua function)

item.titleattr

attribute name of the selected element for the item.title

item.link

item link (can be css selector or lua function)

item.linkattr

attribute name of the selected element for the item.link

item.created

item created date (can be css selector or lua function)

item.createdattr

attribute name of the selected element for the item.created. The date data which must be then parsed with the createdformat.

item.createdformat

the date format of item.created; ratt uses the go time format https://pkg.go.dev/time#pkg-constants e.g.:

item.createdformat = "01/02 03:04:05PM '06 -0700"
item.createdformat = "Mon Jan 2 15:04:05 MST 2006"

item.description

item description (can be css selector or lua function)

item.descriptionattr

attribute name of the selected element for the item.description

item.content

item content (can be css selector or lua function)

item.contentattr

attribute name of the selected element for the item.content

item.image

item image url (can be css selector or lua function)

item.imageattr

attribute name of the selected element for the item.image

🔁 nextpage 🔁

if you want to extract data from more then one page, you can specify the selector for a link to the next page. The link will be requested and above selectors will be applied recursively.

🪢 nextpageattr 🪢

attribute name of the selected element for the nextpage

🔢 nextpagecount 🔢

an integer on how deep to follow the nextpage

🎚️ sort 🎚️

is a enum for sorting the extracted feed items.

Possible values are: REVERSE, CREATED_ASD, CREATED_DESC, TITLE_ASD, TITLE_DESC

🚨 ERRORS 🚨

Returning errors is done with the error() function.

if err ~= nil then
     error("some error: " ... err)
end

💜 LUA FUNCTION ARGUMENTS 💜

Every lua function has two arguments by default:

sel

the local goquery.Selection; in feed attributes it is the entire html document, in item attributes it is the item container element.

index

if the item.container is lua code and returns the number of items, the index variable is available in the item attributes and is the index the processed item.

feed = {
     title = function(sel, index)
          local title = sel:find("h1"):text()
          title:sub(index+2) --just a weird example
     ...
item = {
     ...
     title = function(sel, _)
          local itemTitle = sel:find(".title"):text()
          return mytitle .. ": " .. itemTitle

🌈 LUA GLOBAL VARIABLES 🌈

url

a url object, with the parsed value of the url with which ratt was called. It has methods: scheme, host, path, query, url.

item = {
     container = function(sel, _)
          --do some magic
          return numberOfItems
     end,
     link = function(sel, index)
          return "https://example.com/article" .. index
     end,

🧚 LUA LIBRARIES 🧚

Libraries are imported by default, no need for require(...).

🦋 goquery 🦋

it’s a subset of the great goquery library (https://github.com/PuerkitoBio/goquery).

goquery.newDoc(STRING)

takes a string of the html document and parsed it to goquery.Document

goquery.newDocFromURL(URL)

parses a document from a url. Returns goquery.Document.

document.find(SELECTOR)

find gets the descendants of each element in the current set of matched elements, filtered by a selector. It returns a new Selection object containing these matched elements.

selection

represents a collection of nodes matching some criteria. The initial Selection can be created by using Document.Find, and then manipulated using the jQuery-like chainable syntax and methods.

selection.find(SELECTOR)

finds the descendants filtered by a selector

selection.each(function(index, selection))

each iterates over a Selection object, executing a function for each matched element. It returns the current Selection object. The function f is called for each element in the selection with the index of the element in that selection starting at 0, and a Selection that contains only that element. example:

sel:find(".something"):each(function(i, s)
     value = s:text()
end)

selection.attr

attr gets the specified attribute’s value for the first element in the Selection. To get the value for each element individually, use a looping construct such as Each or Map method. selection.first

selection.first

the original page name-drops selection.first right after the attr note, with no extra description, so that’s literally all we have bestie 🙈

selection.parent

parent gets the parent of each element in the Selection. It returns a new Selection object containing the matched elements.

selection.text

text gets the combined text contents of each element in the set of matched elements, including their descendants.

selection.html

Html gets the HTML contents of the first element in the set of matched elements. It includes text and comment nodes.

🍋 gojq 🍋

it’s a subset of the gojq library (https://github.com/itchyny/gojq) which parses a query and extracts data from json.

gojq.parse(STRING)

parse parses and returns a gojq.Query

query.runMap(MAP)

run the query on a map object

query.runArray(ARRAY)

run the query on a array object

--extract title from json
query, err = gojq.parse(".title")
title, err = query:runMap(sel:text())
if err ~= nil then
     error("gojq runMap:" .. err)
end

🔥 EXAMPLES 🔥

Full example of extracting feed from apnews.com:

ratt.add(
     "https://apnews.com/hub/*",
     {
       feed = {
          title = "head title",
       },
       item = {
          container = ".FeedCard:not(.HubPeek)",
          title = "h3.-cardHeading",
          link = function(sel, _)
            local l = sel:find("a[data-key=card-headline]")
            return "https://apnews.com" .. l:attr("href")
          end,
          linkattr = "href",
          created = function(sel, _)
            ts = sel:find("span.Timestamp"):attr("data-source")
            return ts
          end,
          createdformat = "2006-01-02T15:04:05Z",
          description = "div.content p",
          imageattr = "src"
       }
     }
)

Some explanation:

• item.container — css selector which gets all elements that have the class FeedCard but not the class HubPeek
• item.title — h3 elements with -cardHeading class
• item.link — lua code that gets all <a> elements with attr data-key=card-headline. the links on the website are local, so we must prepend the host "https://apnews.com"
• item.created — get the span element with the "Timestamp" class and then get it’s "data-source" attribute
• item.createdformat — the format to parse the item.created value

👑 SEE ALSO 👑

ratt(1)

🌟 AUTHORS 🌟

Okay so this whole thing is held down by Vladimír Magyar <vladimir@mgyar.me> — he's the maintainer, the main character, the one keeping ratt alive fr fr 👑. And no cap, he is not doing it alone: he's assisted by other open source contributors, aka the group chat of legends who show up and help 💖.

For more info about ratt development, the spot to be is https://sr.ht/~ghost08/ratt — go look, it's giving ✨.

And listen, if you find a bug or you've got a patch, don't just sit there — bugs and patches can be submitted by email to ~ghost08/ratt@todo.sr.ht 🫶. That's the whole vibe, iykyk.

---

💅 lowkey obsessed with you for reading all the way to the authors section, bestie. that's main character energy.

slay, that's the whole page. go read some manual pages, ily 🎀

🌷 ratt(5) — this page, dated 1980-01-01. Yeah the date is goofy, I know 😭 but that's literally what it says, so we're keeping it exactly like that. Byeee 💜🦋