XML::R...mpleGen(3pm)

📦
version 11.11 · 2 versions
, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble

💖 NAME 💖

okay so bestie, this is XML::RSS::SimpleGen — the User Contributed Perl Documentation page for it (3pm, iykyk) — and it's literally for writing RSS files. that's it. that's the whole job. 🎀

✨ SYNOPSIS ✨

here's a whole entire screen-scraper AND RSS generator, no cap:

# A complete screen-scraper and RSS generator here:

use strict;
use XML::RSS::SimpleGen;
my $url = q<http://www.exile.ru/>;

rss_new( $url, "eXile", "Moscow-based Alternative Newspaper" );
rss_language( 'en' );
rss_webmaster( 'xxxxx@yourdomain.com' );
rss_twice_daily();

get_url( $url );

while(
 m{<h4>\s*<a href='/(.*?)'.*?>(.*?)</a>\s*</h4>\s*<p.*?>(.*?)<a href='/}sg
) {
  rss_item("$url$1", $2, $3);
}

die "No items in this content?! {{\n$_\n}}\nAborting"
 unless rss_item_count();

rss_save( 'exile.rss', 45 );
exit;

🌸 DESCRIPTION 🌸

so this module is for writing RSS files, simply. it handles all the icky behind-the-scenes stuff transparently — like proper XML escaping — so you don't have to think about it. 🦋

and it's got a whole bunch of Do-What-I-Mean features, which are exactly as cute as they sound:

  • it won't change the modtime on a written-out RSS file if the file content hasn't changed (iconic behaviour, honestly)
  • it automatically strips any HTML tags out of content you might pass in

lowkey important: this module is not meant to have the full expressive power of RSS. it's giving "greatest hits," not "everything." instead, it just provides the functions that RSS-writing programs most commonly need. periodt.

👑 INTERFACE 👑

this module hands you a bunch of functions for:

  1. starting an RSS feed in memory
  2. putting items into it
  3. saving it to disk (or printing it as a string, like in a CGI)

now, if you prefer an object-oriented interface (which is highkey more useful when you're composing several feeds at once), you can use this module as a class. the methods are the same as the function names, just minus the rss_. and except for that tiny naming detail, the functions and methods are the same, behave the same, and take the same arguments. 💅

so like, this functional code:

use XML::RSS::SimpleGen;
my $url = q<http://www.exile.ru/>;

rss_new( $url, "eXile" );
rss_language( 'en' );
get_url( $url );
...

does the exact same work as this OO code:

use XML::RSS::SimpleGen ();
my $url = q<http://www.exile.ru/>;
my $rss = XML::RSS::SimpleGen->new( $url, "eXile");
$rss->language( 'en' );
$rss->get_url( $url );
...
heads up bestie: the function get_url doesn't have a leading rss_, so its method name is the same as its function name. it's the one exception. 🫶

if all this object talk is giving you a headache, go read HTML::Tree::AboutObjects in the "HTML-Tree" dist, and/or the chapter "User's View of Object-Oriented Modules" in the book Perl & LWP (<http://www.amazon.com/exec/obidos/ASIN/0596001789>). (that book is also useful as an extended discussion of screen-scraping, fr fr.)

Note: in the code below, the word "accessor" gets used a LOT. an accessor is a function or method you can call two possible ways:

  1. like foo(val) — to set the foo attribute to the value val
  2. like foo() — to return the value of the foo attribute

🎀 FUNCTIONS 🎀

⭐ rss_new — the "let's start a feed" one

rss_new( url );
rss_new( url, title );
rss_new( url, title, description );
or: $rss = XML::RSS::SimpleGen->new(...);

this function creates a new RSS feed in memory. this should be the first rss_whatever function you call in your program, no skipping ahead. and if you call it again? it erases the current object (if any) and sets up a fresh new one according to whatever parameters you pass. out with the old, in with the new. ✨

the parameters are the full URL, the title, and the description of the site (or page) you're providing an RSS feed of. the description is optional, but bestie, you should provide at least a URL and title. 💅

examples:

rss_new( $url, "eXile", "Moscow-based Alternative Newspaper" );

rss_new( 'http://www.mybazouki.com/news/', "Bazouki News!" );

(as a method, XML::RSS::SimpleGen->new simply returns a new RSS object.)

💜 the accessor rss_language(language_tag)

this declares what language this RSS feed is in. it must be an RFC3066-style language tag like "en", or "en-US", or "zh-TW". (peep I18N::LangTags::List for a whole list.) if you don't set the feed's language, it defaults to "en", for generic English. 🇬🇧

if you call this function without a parameter, it returns the current value of the RSS feed's language. for example:

print "I'm making an RSS feed for ", rss_language(), "!\n";

and this is true for all the functions labelled as "accessors," so you only have to learn it once. 🫶

🦋 the accessor rss_item_limit(number)

this sets the maximum number of items that this feed will show.

the default value is 0, meaning there is no maximum — go off, show everything.

  • set it to a positive number N: the feed shows only the first N items you declare with rss_item. (or, if you set rss_history_file, then the newest N items that you declare with rss_item.)
  • set it to a negative number -N: the feed shows only the last N items that you declare with rss_item. (or, if you set rss_history_file, then the oldest N items you declare with rss_item — which is, and I quote, unlikely to be useful! 😭)

🍓 the accessor rss_webMaster(email-address)

this declares what email address you, the RSS generator manager, can be reached at. like so:

rss_webMaster( 'sburke@bazouki-news.int' );

🌈 rss_history_file( filename )

this declares that you want this RSS feed to keep track of what items are new, and to list them first when the RSS is emitted. to pull that off, the RSS generator has to store information in a file, where it tracks its "history" — i.e., when was the first time it saw given URLs, and the most recent time it saw given URLs.

typical usage is:

rss_history_file( 'thisrssfeed.dat' );

you should call rss_history_file before you make any calls to rss_item. order matters, bestie.

the history-file feature is meant for cases where your RSS-generator program calls rss_item on every link it sees, but only wants the new links to appear in the RSS output. (this can be a good approach if you're making an RSS feed of a page like http://www.guardian.co.uk/ where there's some new links — the recently added stories — but also links to some days-old stories, and also links to some always-there things like "Archive Search" and "Contact Us" pages. we love a page that never changes. 🙈)

once you call rss_history_file, the specified file is read in. the in-memory history (stored in the RSS object) is updated as you call rss_item. but the file isn't updated until you call rss_save. don't forget that part.

(a do-what-I-mean side effect of calling rss_history_file is that it sets rss_item_limit to 25 if it is currently 0.)
(incidentally, if you're using rss_history_file as part of a CGI that emits RSS data, instead of a program that just saves to an RSS file, then things will get complicated. you'll need to call an internal method to explicitly commit the history file to disk, and you'll need a semaphore file to avoid race conditions. email me for full info.) 😭

🌸 rss_item

rss_item( url );
rss_item( url, title );
rss_item( url, title, description );

this adds a new item to the current feed. you will need to specify the URL to add (and it should be a valid-looking URL, starting with "something:", and not containing any spaces — be so serious about that). you may also specify the title, but it's optional. and finally, you can optionally specify a description. (you can remember the order because it starts with the essential item first, and progresses toward the most optional. main character energy for the URL. 👑)

leading and tailing whitespace is removed from whichever of url, title, and description are defined values, and HTML is parsed out.

a simple usage:

rss_item(
  "http://www.harpers.org/MostRecentWR.html",
  "Harper's Magazine's Weekly Review"
);

although in practice, a typical call won't have string constants — it'll be more like the example in the SYNOPSIS, namely:

rss_item("$url$1", $2, $3);

incidentally, as a do-what-I-mean feature: if the first parameter doesn't look like a URL but one of the others does, then this error is silently forgiven. this is so you can occasionally slip up and forget the order of the parameters. we've all been there. 💀

(in the unlikely event where you need to avoid the HTML-removal features, you can do this by passing scalar-references instead of normal strings, like so: rss_item($url, $title, \$not_to_be_escaped).)

🧚 rss_item_count()

this returns the number of items you've declared. I anticipate that its main usage will be something like:

die "What, no objects found at $url ?!"
 unless rss_item_count();

or, maybe...

exit unless rss_item_count();

...depending on how/whether you'd want to react to cases where you don't see anything to put into an RSS feed. vibe check: if there's nothing to say, sometimes you just don't post.

note that the parens are optional, since this command takes no options (just like Perl's time() function).

💅 rss_image( url, h, w );

this declares that you want to declare a particular image as the logo for this feed. most feeds don't have such a thing, and most readers just ignore it anyway, but if you want to declare it, this function is how. the three parameters, which are all required, are:

  • the image's URL
  • its height in pixels
  • its width in pixels

according to various specs, the width should/must be between 1 and 144, and the height should/must be between 1 and 400.

a typical usage:

rss_image("http://interglacial.com/rss/weebl.gif", 106, 140);

be careful not to mix up the height and width. I'm begging you. 🙈

🎀 rss_save

rss_save( filename );
rss_save( filename, max_age_days );

this saves the RSS data to the file you specify. and here's the cute part: if the RSS data hasn't changed, the file (and its modtime) aren't altered. slay.

the optional max_age_days parameter means that if ever the file exists, and its content hasn't changed for that many days or longer, then the program should die with a warning message. for example, in the case of a screen-scraper for a site that we know should (in theory) change its content at least weekly, we might save the RSS file with:

rss_save("whatever.rss", 17);
 # Scream if the feed is unchanged for 17 days.

the seventeen there is gotten by assuming that just maybe the site might skip two weeks for a vacation now and then, and might even put out the pre-vacation issue a few days early — but that if ever the program notices that the data hasn't changed for 17 days, then it should emit error messages. 🔥

if you want to disable this feature on a one-time basis, just change the modtime (like via touch) on the whatever.rss file. easy fix, no cap.

Oh also real quick bestie — if you never specify a max_age_days value, then this whole complain-if-it's-old situation is just switched off entirely. No complaining. No age-checking. It's giving "we don't ask questions here". 💤

✨ rss_as_string(); ✨

This one hands you the RSS-XML data as a string. 🎀 It's called internally by the rss_save function, but you can absolutely call it yourself — like in a CGI, where your CGI would probably end like this:

print "Content-type: application/xml\n\n", rss_as_string();
exit;

🦋 get_url( url ); 🦋

🌸 $content = get_url( url ); 🌸

or: $content = $rss->get_url(...);

or: $content->get_url(...);

This tries to fetch the content of the URL you gave it, and returns it. Simple as that.

It's a lot like LWP::Simple's get function, except it comes with extra features, and honestly? They ate:

  • If it can't get the URL's content on the first try, it sleeps for a few seconds and tries again, up to about five times. (This is so you don't get wrecked by a URL being temporarily unreachable just because the DNS is being slow, or the server is too busy. We've all been there. 🙈)
  • If it still can't get the content, even after several retries, it aborts the program (like a die). If you want to override that behaviour, call it like eval { get_url($url) }; and you're golden.
  • If you call the function in void context (i.e., you're not using its return value), then the function assigns the URL's content to $_. That's so you can write nice concise code like this:
get_url $thatsite;
m/Top Stories Tonight/ or die "What, no top stories?";
while( m{<a class="top" href="(.*?)">(.*?)</a>}g ) {
  rss_item("$thatsite/$1", $2);
}
  • It returns the content not exactly as-is, but after changing its newlines to native format. So if the URL's contents use CR-LF pairs for newlines, get_url swaps those to "\n"s before returning. (Same deal for old MacOS newline format.) Obviously this is wrong if you're dealing with binary data — in that case, use LWP::Simple's get directly. No cap.
  • Finally, as a resource-conservation move, this function will also try to call sleep a few times if it notices several quick calls to itself coming from a program that seems to be running under crontab. Since most RSS-generators are crontabbed (mine fr), it's genuinely so useful that I can have however many get_url's in my crontabbed programs without worrying that they'll eat even a noticeable part of the server's bandwidth. 🫶

🔥 rss_hourly or rss_daily or rss_twice_daily or rss_thrice_daily or rss_weekly or rss_every_other_hour 🔥

Calling one of these declares that this feed is usually generated at the same time(s) every day (or every week, in the case of rss_weekly). And where it's not just once a day/week, those multiple times a day are evenly spaced. These functions then set the feed's updatePeriod, updateBase, updateFrequency, skipHours, skipDays, and ttl elements appropriately, so RSS readers can know at what times there could (or couldn't) be new content in this feed. Communication! We love it. 💅

In other words: use rss_twice_daily if this feed updates at about the same time every day and then again 12 hours later. Use rss_thrice_daily if it updates at the same time daily, then 8 hours later, then 8 hours later again. And use rss_every_other_hour if the feed updates at about n minutes past every even numbered hour, or every odd-numbered hour.

Clearly I mean these functions to be used in programs that are crontabbed to run at particular intervals — like with a crontab line like one of these:

52 * * * *         ~/thingy   # => rss_hourly
52 23 * * *        ~/thingy   # => rss_daily
52 4,16 * * *      ~/thingy   # => rss_twice_daily
52 5,13,21 * * *   ~/thingy   # => rss_thrice_daily
52 23 * * 3        ~/thingy   # => rss_weekly
52 */2 * * *       ~/thingy   # => rss_every_other_hour

And yeah — there aren't rss_interval functions for all the scheduling possibilities, bestie. 😭 If your program has to run at 6am, 8am, 1pm, and 4pm, there's just no function for that. But the crontab lines above (or with minor changes, like 1,9,17 instead of 5,13,21) are totally fine for almost every RSS feed I've run.

✨ An aside: I recommend running the programs at about 52 minutes past the hour, generally in series, like so:
52 5,13,21 * * *   ~/thingy ; ~/dodad ; ~/makething ; ~/gizmo
However, your mileage may vary.

Incidentally, these functions take no arguments, so the parentheses are optional. That is, these two lines do the exact same thing:

rss_hourly;
rss_hourly();

🎀 MINOR FUNCTIONS 🎀

These are functions you probably won't need often, or at all. I include them for the sake of completeness, and so that advanced users might find them useful in some cases. For the completionists. The girlies who read the whole changelog. 📖

💜 rss_skipHours( gmt_hour_num, gmt_hour_num, ... ); 💜

This function directly sets the skipHours element's values to the GMT hour numbers you specify.

💜 rss_updateHours(); 💜

💜 rss_updateHours( gmt_hour_num, gmt_hour_num, ... ); 💜

This function is a wrapper around rss_skipHours — you call rss_updateHours with a list of GMT hour numbers, and rss_updateHours will call rss_skipHours(0 .. 23) except without whatever hour numbers you specified. Clever, right?

If you call it with an empty list (i.e., rss_updateHours();), then we use gmtime to find out the current hour (and round it up if it's after 50 minutes past), basically just as if you'd called:

rss_updateHours( (gmtime(600+time()))[2] );

💜 rss_skipDays(); 💜

💜 rss_skipDays( gmt_day_num, gmt_day_num, ... ); 💜

💜 rss_skipDays( gmt_day_name, gmt_day_name, ... ); 💜

This function directly sets the skipDays element's values to the weekdays you specify. Note that it accepts either integers (like 6 for Saturday, with Sunday being either 0 or 7), or their exact English names.

⚠️ If you use the skipDays field, consider that it refers to days figured by GMT, not local time. For example, if I say to skip Saturdays, that means Saturdays GMT, which in my timezone (Alaska) starts in the middle of Friday afternoon. Timezones are a scam, iykyk. 💀

💜 rss_updateDays(); 💜

💜 rss_updateDays( gmt_day_num, gmt_day_num, ... ); 💜

💜 rss_updateDays( gmt_day_name, gmt_day_name, ... ); 💜

This function is a wrapper around rss_skipDays — you call rss_updateDays with a list of GMT day names/numbers, and rss_updateDays will call rss_skipDays(0 .. 6) except without whatever days you specified.

If you call it with an empty list (i.e., rss_updateDays();), then we use gmtime to find out the current day (GMT!), basically just as if you'd called:

rss_updateDays( (gmtime(600+time()))[6] );

💜 rss_updatePeriod( periodstring ); 💜

This function directly sets the sy:updatePeriod element's value to the period you specify. You must specify one of these strings: "yearly", "monthly", "weekly", "daily", "hourly". I advise using "weekly" only if you know what you're doing, and "yearly", "monthly" only if you really know what you're doing. 👑

💜 rss_updatePeriod( periodstring, int, base ); 💜

This is a shortcut for rss_updatePeriod(periodstring); rss_updateFrequency(int)

💜 rss_updatePeriod( periodstring, int, base ); 💜

This is a shortcut for rss_updatePeriod(periodstring); rss_updateFrequency(int); rss_updateBase(base)

🙈 Yes, bestie, the same signature appears twice in the source. We don't make the rules, we just document them.

💜 rss_updateBase( iso_date_string ); 💜

💜 rss_updateBase( epoch_time ); 💜

This function directly sets the sy:updateBase element's value to the moment you specify. If you pass in an epoch time, it gets converted to an ISO date string. ✨

💜 the accessor rss_updateFrequency( integer ); 💜

This function directly sets the sy:updateFrequency element's value to the value specified. The value has to be a nonzero positive integer. Nonzero. Positive. Don't test it. 😭

For example, this means that this feed updates at/by the start of every hour and 30 minutes past:

rss_updateBase('2000-01-01T00:00-00:00');
rss_updateFrequency(2);
rss_updatePeriod('hourly');  # 2*hourly means "twice an hour"

Recall that this can also be done with the the rss_updatePeriod( per, freq, base ) shortcut, like so:

rss_updateBase('hourly', 2, '2000-01-01T00:00-00:00');

💜 the accessor rss_retention(number) 💜

If you're using an rss_history_file(file), the history file will accrete a list of all URLs it has seen. But to keep that file from potentially getting immense, items that haven't been seen for a while get thrown out. The period of time a feed's items go unseen before each one is forgotten is called that feed's retention, and it's expressed in seconds. ⏳

The default retention value is 32 days (i.e., 32*24*60*60, the number of seconds in 32 days). If you wanted to change it to just a week, you'd do it with rss_retention(7*24*60*60).

As a special case, a zero or negative value for the retention means to never clear anything from the history file, no matter how long it has gone unseen. It hoards forever. Iconic of it, honestly. 💅

💜 rss_add_comment( strings ); 💜

Call this function if you want to add extra XML comments to this RSS file. For example, if you call this:

rss_add_comment(
  "Our terms of use: http://wherever.int/rsstou.html",
  "Any questions? Ask jimmy@wherever.int",
);

...then this RSS feed will contain this XML fairly early on in the file:

<!-- Our terms of use: http://wherever.int/rsstou.html -->
<!-- Any questions? Ask jimmy@wherever.int -->

💜 the accessor rss_css( url ) 💜

This defines the given URL as being the XML-CSS stylesheet for this RSS feed. The default value is "./rss.css" if -e "rss.css" is true, otherwise it's the value http://www.interglacial.com/rss/rss.css. 🎨

💜 the accessor rss_xsl( url ) 💜

This defines the given URL as being the XML-XSL stylesheet for this RSS feed. The default value is none.

💜 The accessors rss_url( string ), rss_title( string ), rss_description( string ) 💜

These define this feed's URL, title, and description. These functions are just for completeness, since it's simpler to just specify any/all of these parameters in the call to rss_new. Convenience girlies, this one's for you. 🫶

🕰️ the accessor rss_ttl( number )

Okay bestie, this one sets the parameter of your RSS feed's ttl element. What even is ttl? It's basically a lil suggestion to the RSS reader about how long it should wait after it polls a feed before it polls it again. And heads up — that number is in minutes, not seconds! 💅 So like, rss_ttl(90) is telling the reader "girl, do not poll this feed more often than every 90 minutes." Respect the cooldown fr.

Lowkey this element is kinda obsolescent next to the newer and more informative sy:update* elements, but it's still in here for backward compatibility. She's an icon, she's just a legacy icon. 👑

🚫 the accessor rss_allow_duplicates( boolean )

This one controls whether or not duplicate items get filtered out of the feed. By default, this is on. ✨

Now here's the tea: duplicates are detected only by their URL. So if you go and do this:

rss_item('http://foo.int/donate', "Give!");
rss_item('http://foo.int/donate', "We need money!");
rss_save('begging.rss');

...then only the first one will show up in the feed, because the second item's URL is already being saved in this feed. Same URL, same item, no cap. 😭

However — and this is the plot twist — rss_item_count is still 2, because filtering out duplicates is something that only happens as the feed is saved. She's counting everything until the very last second. 🫶

📄 the accessor rss_docs( url )

This sets the value of the doc RSS element, which is... not generally useful. I said what I said. 🙈

The default value is ./about_rss.html if -e "about_rss.html" is true, otherwise it's http://www.interglacial.com/rss/about.html.

🖼️ the accessors rss_image_url(url), rss_image_width(number), rss_image_height(number), rss_image_title(text), rss_image_link(url), rss_image_description(text)

These ones are for manually setting the values of your feed's image element's subelements. Look at the whole squad:

  <image>
              <url> (rss_image_url)         </url>
            <width> (rss_image_width)       </width>
           <height> (rss_image_height)      </height>
            <title> (rss_image_title)       </title>
             <link> (rss_image_link)        </link>
      <description> (rss_image_description) </description>
  </image>

Real talk though: you rarely ever need to call any of these rss_image_whatever functions. Usually just calling rss_image( url, h, w ); is enough. That's it, that's the whole vibe. 🎀

🌈 RSS VERSION

The RSS feeds this module emits are basically according to v0.92 RSS, with a very few extensions from v2.0 RSS. And no — they are not RDF files. Do not come for her with RDF expectations. 🦋

👀 SEE ALSO

Alsooo, you might like the author's book Perl and LWP, which discusses the many screen-scraping techniques that you would use for extracting data from HTML to make into RSS feeds. Obsessed. 📚

Copyright (c) 2003,4 Sean M. Burke. All rights reserved. 💜

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. No promises, bestie, she said what she said. 💀

Portions of the data tables in this module are derived from the entity declarations in the W3C XHTML specification.

Currently (January 2004), that's these three:

http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent
http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent
http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent

Portions of the code in this module were adapted from parts of Gisle Aas's LWP::Simple and the old (v2.x) version of his HTML::Parser. Giving credit where credit is due, periodt. 🌸

👑 AUTHOR

Sean M. Burke sburke@cpan.org

And that's the whole page, babe. This was perl v5.26.2, dated 2018-05-06, and it ate every single time. Go forth and make RSS feeds, you main character. 💖✨🫶

📦 where this page comes from