FEED2EXEC(1)

📦
version 0.23.0 · 5 versions
, in ubuntu stonking

💖 FEED2EXEC(1) 💖

okay bestie, gather round, because this is the manual page for feed2exec in the feed2exec manual and it is literally the most useful thing you'll read today. grab a snack 🍓

✨ NAME ✨

feed2exec — the programmable feed reader 👑

🌈 SYNOPSIS 🌈

feed2exec {add,ls,rm,fetch,import,export}

💅 DESCRIPTION 💅

so here's the whole vibe: you set up a bunch of feeds, and this command fires off a specific plugin for every single new item it finds in those feeds. that's it. that's the whole thing. feed goes in, plugin goes brrr. 🔥

🎀 OPTIONS 🎀

  • --version — shows you the version and then dips. bye! 👋
  • --loglevel — pick your exact log level, bestie. [default: WARNING]
  • -v, --verbose — tells you what's actually happening (loglevel: VERBOSE) 👀
  • -d, --debug — full detective mode, all the debugging info (loglevel: DEBUG) 🔍
  • --syslog LEVEL — sends LEVEL logs to syslog 📮
  • --config TEXT — use a different configuration file 🗂️
  • --database DB — use a different database 💾
  • -h, --help — shows this message and exits. cute, right?

🦋 EXAMPLES 🦋

a simple run with no side effects (aka nothing gets touched, we're just vibing):

feed2exec parse https://www.nasa.gov/rss/dyn/breaking_news.rss --output echo --args '{item.title}'

saving feed items to a Maildir folder 📬:

feed2exec add "NASA breaking news" https://www.nasa.gov/rss/dyn/breaking_news.rss --folder nasa
feed2exec fetch

this creates the equivalent of this configuration file in ~/.config/feed2exec.ini:

[DEFAULT]
output = feed2exec.plugins.maildir
mailbox = ~/Maildir

[NASA breaking news]
folder = nasa
url = https://www.nasa.gov/rss/dyn/breaking_news.rss

send new feed items to Transmission 🌀:

feed2exec add "Example torrent list" http://example.com/torrents/feed --output transmission --folder /srv/incoming

send new feed items to Mastodon, using the toot commandline client from https://github.com/ihabunek/toot/ 🐘:

feed2exec add "My site" http://example.com/blog/feed --output exec --args 'toot post "{item.title} {item.link}"'

send new feed items to Twitter, using the tweet commandline client from python-twitter at https://github.com/bear/python-twitter 🐦:

feed2exec add "My site on twitter" http://example.com/blog/feed --output exec --args 'tweet "{item.title:.40s} {item.link:.100s}"'

show feed contents 👀:

feed2exec add "NASA breaking news" https://www.nasa.gov/rss/dyn/breaking_news.rss --output echo --args "{item.title} {item.link}"
feed2exec fetch

👑 COMMANDS 👑

the whole squad, in order:

  • parse
  • fetch
  • add
  • ls
  • rm
  • import
  • export

🧚 parse 🧚

usage bestie:

parse URL
    [--output PLUGIN [--args ARG [ARG [...]]]
    [--filter PLUGIN] [--filter_args ARG [ARG [...]]]
    [--mailbox PATH] [--folder PATH]

the parse command loads and parses a single feed, and it does not touch the database. not even a little. it's basically like calling add then fetch on one single feed, except the feed doesn't get kept in your configuration. it's designed for doing quick lil tests with a new feed ✨. the arguments are the exact same as the add command, so no surprises there.

🔥 fetch 🔥

usage:

fetch [--parallel | -p | --jobs N | -j N] [--force | -f] [--pattern pattern]

the fetch command goes through all your configured feeds — or just the ones matching the pattern substring if you gave it one.

options:

  • --pattern TEXT — only fetch feeds matching the name or URL 🔎
  • --parallel — parses feeds in the background to improve performance ⚡
  • -j,--jobs N — starts N jobs in parallel. this implies --parallel, and it defaults to the number of CPUs detected on the machine 🖥️
  • -f, --force — skips reading and writing the cache entirely and will consider all entries as new 😳
  • -n, --catchup — tells output plugins to simulate their actions 🙈

💜 add 💜

usage:

add NAME URL
    [--output PLUGIN [--args ARG [ARG [...]]]
    [--filter PLUGIN] [--filter_args ARG [ARG [...]]]
    [--mailbox PATH] [--folder PATH]

the add command adds the given feed NAME that's gonna be fetched from the provided URL.

options:

  • --output PLUGIN — uses PLUGIN as the output module. it defaults to maildir, so it stores into a mailbox. use null or echo if you just wanna fetch the feed and not do anything else with it. modules get searched in the feed2exec.plugins package, unless the name contains a dot — in that case the whole Python search path gets used 🐍
  • --args ARGS — passes arguments ARGS to the output plugin. it supports interpolation of feed parameters using, for example, {title} ✨
  • --filter PLUGIN — filters feed items through the PLUGIN filter plugin 🧽
  • --filter_args ARGS — arguments passed to the filter plugin
  • --mailbox PATH — folder to store email into, defaults to ~/Maildir. 📬
  • --folder PATH — subfolder to store the email into 📁

those parameters are documented way more extensively in their equivalent settings over in the configuration file, see below. iykyk 💅

🎀 ls 🎀

the ls command lists all your configured feeds as JSON packets. all of them. in JSON. iconic 💖

💀 rm 💀

usage:

rm NAME

removes the feed named NAME from the configuration. gone. bye bestie 👋

🌸 import 🌸

usage:

import PATH

imports feeds from the file named PATH. that file is expected to have outline elements, and only the title and xmlUrl elements get imported — as the NAME and URL parameters respectively 📥

🫶 export 🫶

usage:

export PATH

exports feeds into the file named PATH. that file will use the feed NAME elements as title and the URL as xmlUrl 📤

📁 FILES 📁

🎀 Configuration file 🎀

ok so bestie, the config file gets loaded from — and written back to, when you run add — either ~/.config/feed2exec.ini or $XDG_CONFIG_HOME/feed2exec.ini. You can also point at a different one with the --config commandline parameter. it's giving very flexible, very demure 💅

here's a lil baby example config snippet:

[NASA breaking news]
url = https://www.nasa.gov/rss/dyn/breaking_news.rss
output = feed2exec.plugins.echo
args = {title} {link}

obvi you can just change all those settings directly in the config file whenever, no gatekeeping. and here's the cute part — there's a [DEFAULT] section that applies settings to all feeds. like, this makes every feed dump new items into a maildir subfolder:

[DEFAULT]
output = feed2exec.plugins.maildir
folder = feeds

so individual feeds don't need to be configured one by one, which is the whole point. slay 🫶

Note: feed2exec does not take care of adding the folder to "subscriptions" in the mailbox. it is assumed that folders are auto-susbcribed or the user ignores subscription. if that is a problem, you should subscribe to the folder by hand in your email client when you add a new config. you can also subscribe to a folder (say feeds above) directly using the doveadm mailbox subscribe feeds command in Dovecot, for example.

so here's every configuration parameter it supports, no crumbs left behind:

  • name — human readable name for the feed. Equivalent to the NAME argument in the add command.
  • url — address to fetch the feed from. Can be HTTP or HTTPS, but also file:// resources for test purposes.
  • output — output plugin to use. Equivalent to the --output option in the add command.
  • args — arguments to pass to the output plugin. Equivalent to the --args option in the add command.
  • filter — filter plugin to use. Equivalent to the --filter option in the add command.
  • mailbox — store emails in that mailbox prefix. Defaults to ~/Maildir.
  • folder — subfolder to use when writing to a mailbox. By default, a slugified version of the feed name (where spaces and special character are replaced by -) is used. For example, the feed named "NASA breaking news" would be stored in ~/Maildir/nasa-breaking-news/. Note that the mailbox prefix is used only if the folder path is relative.
  • catchup — skip to the latest feed items. The feed is still read and parsed, and new feed items are added to the database, but output plugins are never called.
  • pause — completely skip feed during fetch or parse. Similar to catchup, but doesn't fetch the feed at all and doesn't touch the cache.

and now the main event, a more complete example configuration with ALL the settings used. i am obsessed with this one fr fr 🔥

# this section will apply to all feeds
[DEFAULT]
# special folder location for maildir. I use this when I have multiple
# accounts synchronized with Offlineimap
mailbox = ~/Maildir/Remote/

# a feed to store NASA breaking news entry in a "nasa" subfolder
# this also demonstrates the droptitle filter
[NASA breaking news]
url = https://www.nasa.gov/rss/dyn/breaking_news.rss
folder = nasa
filter = feed2exec.plugins.droptitle
filter_args = trump

# some maildir storage require dots to get subfolders. for example,
# this will store messages in INBOX/feeds/images/ on Dovecot
[NASA image of the day]
url = https://www.nasa.gov/rss/dyn/lg_image_of_the_day.rss
folder = .feeds.images

# same feed, but save to wayback machine
[NASA IOTD wayback]
url = https://www.nasa.gov/rss/dyn/lg_image_of_the_day.rss
output = feed2exec.plugins.wayback

# this demonstrates the emptysummary filter, which fixes GitHub feeds
# that lack a proper summary
[restic]
url = https://github.com/restic/restic/tags.atom
filter = feed2exec.plugins.emptysummary

# saving to a mbox folder, one file per feed instead of one file per item
[International Space Station Reports]
url = http://blogs.nasa.gov/stationreport/feed/
mailbox = ~/Mail/
folder = stationreport.mbx

# simple generic exec call example: check for broken links using linkchecker
[NASA linkchecker]
url = https://www.nasa.gov/rss/dyn/breaking_news.rss
output = feed2exec.plugins.exec
args = linkchecker --check-extern --no-robots --recursion-level 1 --quiet '{item.link}'

# same, but with a Ikiwiki RSS feed, which needs fixing
[Ikiwiki linkchecker]
url = http://ikiwiki.info/recentchanges/index.rss
output = feed2exec.plugins.exec
filter = feed2exec.plugins.ikiwiki_recentchanges
args = linkchecker --check-extern --no-robots --recursion-level 1 --quiet '{item.link}'

# retweet hurricane news
[NASA Hurricane breaking news]
url = https://www.nasa.gov/rss/dyn/hurricaneupdate.rss
output = feed2exec.plugins.exec
args = tweet "{item.title:.40s} {item.link:.100s}"

# same, but on the mastodon network
#
# we can have multiple entries with the same URL without problems, as
# long as the feed name is different. it does mean that the feed will
# be fetched and parsed multiple times, unfortunately.
#
# this could be improved to include the '{item.summary}' and extra markup,
# for example.
[NASA Hurricane breaking news - Mastodon]
url = https://www.nasa.gov/rss/dyn/hurricaneupdate.rss
output = feed2exec.plugins.exec
# unfortunately, this will noisily report the URL of the posted link,
# which you may not want. to avoid that, encourage upstream to do the
# right thing: https://github.com/ihabunek/toot/issues/46 ... or use
# another tool listed here:
# https://github.com/tootsuite/documentation/blob/master/Using-Mastodon/Apps.md
args = toot post "{item.title} {item.link}"
# output is disabled here. feed will be fetched and parsed, but no
# toot will be sent
catchup = True

# same, but on the Pump.io network
[NASA Hurricane breaking news - Pump]
url = https://www.nasa.gov/rss/dyn/hurricaneupdate.rss
output = feed2exec.plugins.exec
args = p post note "{item.title} {item.link}"

# crude podcast client
[NASA What up?]
url = https://www.nasa.gov/rss/dyn/whats_up.rss
output = feed2exec.plugins.exec
# XXX: this doesn't handle errors properly: if there is a feed without
# enclosures, the whole thing will crash.
args = wget -P /srv/podcasts/nasa/ "{item.enclosures[0].href}"
# feed is paused here. feed will not be fetched and parsed at all and
# no post will be sent.
pause = True

# download torrents linked from a RSS feed
[torrents]
url = http://example.com/torrents.rss
output = feed2exec.plugins.exec
args = transmission-remote -a '{item.link}' -w '/srv/incoming'

# same thing with an actual plugin
[torrents]
url = http://example.com/torrents.rss
output = feed2exec.plugins.transmission
args = seedbox.example.com
folder = /srv/incoming

💾 Cache database 💾

the feeds cache lives in a feed2exec.db file. it's a SQLite database and you can poke at it with standard sqlite tools, no secret handshake required 🦋 it exists to keep track of which feed and items have been processed.

want to clear the cache? literally just remove the file and the program will process all feeds items from scratch again. but ok hear me out: if you do that, you should use the --catchup argument to avoid duplicate processing. or you can use the null output plugin for the same effect. two paths, same destination, main character energy ✨

💀 LIMITATIONS 💀

ngl bestie, feed support is only as good as the feedparser library, which isn't as solid as expected. specifically, there were issues with feeds without dates (https://github.com/kurtmckee/feedparser/issues/113) and without guid (https://github.com/kurtmckee/feedparser/issues/112). yeah 😭

unit test coverage is incomplete, but still pretty decent — above 90%. we take those 🎀

the exec plugin itself is not well tested and may have serious security issues. so like, be careful with that one, fr fr.

the API, commandline interface, configuration file syntax and database format can all be changed until the 1.0 release is published, at which point normal Semantic Versioning semantics apply (http://semver.org). so don't get toooo attached yet.

the program is written mainly targeting Python 3.5 and 3.7, but should support later releases as well. see the setup.py classification for an authoritative reference. Python 2.7 is not supported anymore. rip to 2.7, not sorry 💅

the SQL storage layer is badly written and is known to trigger locking issues with SQLite when doing multiprocessing. the global LOCK object could be used to work around this issue but that could mean pretty bad coupling. a good inspiration may be the beets story about this problem (http://beets.io/blog/sqlite-nightmare.html). and of course, another alternative would be to considering something like SQLalchemy instead of rolling our own ORM. there has, however, been some improvements to the locking recently, although that has been done with thread instead of process-specific locks. growth 💖

older feed items are not purged from the database when they disappear from the feed, which may lead to database bloat in the long term. similarly, there is no way for plugins to remove old entry that expire from the feed. so it just. accumulates. 🙈

👀 SEE ALSO 👀

🌸 Author 🌸

Antoine Beaupré

Copyright (C) 2016-2019 Antoine Beaupré

ok that's the whole page bestie!! you just read feed2exec version 0.23.0, dated Sep 16, 2026, and honestly? iconic. go forth and syndicate 💜🦋✨

📦 where this page comes from