tribut/html2rss: 📰 Build RSS 2.0 feeds from websites (and JSON APIs) with a few CSS selectors. · GitHub

This Ruby gem builds RSS 2.0 feeds from a feed config.

With the feed config containing the URL to scrape and CSS selectors for information extraction (like title, URL, ...) your RSS builds. Extractors and chain-able post processors make information extraction, processing and sanitizing a breeze. Scraping JSON responses and setting HTTP request headers is supported, too.

Searching for a ready to use app which serves generated feeds via HTTP? Head over to html2rss-web!

To support the development, feel free to donate. Thank you! 💓

You can also install it as a dependency in your Ruby project:

Create a file called my_config_file.yml with this example content:

channel:
  url: https://stackoverflow.com/questions
selectors:
  items:
    selector: "#hot-network-questions > ul > li"
  title:
    selector: a
  link:
    selector: a
    extractor: href

Build the RSS with: html2rss feed ./my_config_file.yml.

Here's a minimal working example within Ruby:

require 'html2rss'

rss =
  Html2rss.feed(
    channel: { url: 'https://stackoverflow.com/questions' },
    selectors: {
      items: { selector: '#hot-network-questions > ul > li' },
      title: { selector: 'a' },
      link: { selector: 'a', extractor: 'href' }
    }
  )

puts rss

A feed config consists of a channel and a selectors Hash. The contents of both hashes are explained in the chapters below.

Good to know:

• You'll find extensive example feed configs at spec/*.test.yml.
• See html2rss-configs for ready-made feed configs!
• If you've already created feed configs, you're invited to send a PR to html2rss-configs to make your config available to the general public.

Alright, let's move on.

Sometimes there are structurally equal pages with different URLs. In such a case you can add dynamic parameters to the channel's attributes.

Example of a dynamic id parameter in the channel URLs:

channel:
  url: "http://domainname.tld/whatever/%<id>s.html"

Command line usage example:

bundle exec html2rss feed the_feed_config.yml id=42

config = Html2rss::Config.new({ channel: { url: 'http://domainname.tld/whatever/%<id>s.html' } }, {}, { id: 42 })
Html2rss.feed(config)

See the more complex formatting of the sprintf method for formatting options.

First, you must give an items selector hash which contains a CSS selector. The selector selects a collection of HTML tags from which the RSS feed items are build. Except the items selector, all other keys are scoped to each item of the collection.

Then, to build a valid RSS 2.0 item, you need to have at least a title or a description. You can have both.

Having an items and a title selector is already enough to build a simple feed.

Your selectors Hash can contain arbitrary named selectors, but only a few will make it into the RSS feed (This due to the RSS 2.0 specification):

Every named selector in your selectors hash can have these attributes:

Extractors help with extracting the information from the selected HTML tag.

• The default extractor is text, which returns the tag's inner text.
• The html extractor returns the tag's outer HTML.
• The href extractor returns a URL from the tag's href attribute and corrects relative ones to absolute ones.
• The attribute extractor returns the value of that tag's attribute.
• The static extractor returns the configured static value (it doesn't extract anything).
• See file list of extractors.

Extractors might need extra attributes on the selector hash.
👉 Read their docs for usage examples.

Html2rss.feed(
  channel: {}, selectors: { link: { selector: 'a', extractor: 'href' } }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  link:
    selector: 'a'
    extractor: 'href'

Extracted information can be further manipulated with post processors.

⚠️ Always make use of the sanitize_html post processor for HTML content. Never trust the internet! ⚠️

• See file list of post processors.

👉 Read their docs for usage examples.

Html2rss.feed(
  channel: {},
  selectors: {
    description: {
      selector: '.content', post_process: { name: 'sanitize_html' }
    }
  }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  description:
    selector: '.content'
    post_process:
      - name: sanitize_html

Pass an array to post_process to chain the post processors.

channel:
  # ... omitted
selectors:
  # ... omitted
  price:
    selector: '.price'
  description:
    selector: '.section'
    post_process:
      - name: template
        string: |
          # %{self}

          Price: %{price}
      - name: markdown_to_html

Note the use of | for a multi-line String in YAML.

The categories selector takes an array of selector names. Each value of those selectors will become a <category> on the RSS item.

Html2rss.feed(
  channel: {},
  selectors: {
    genre: {
      # ... omitted
      selector: '.genre'
    },
    branch: { selector: '.branch' },
    categories: %i[genre branch]
  }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  genre:
    selector: ".genre"
  branch:
    selector: ".branch"
  categories:
    - genre
    - branch

An enclosure can be any file, e.g. a image, audio or video.

The enclosure selector needs to return a URL of the content to enclose. If the extracted URL is relative, it will be converted to an absolute one using the channel's URL as base.

Since html2rss does no further inspection of the enclosure, its support comes with trade-offs:

1. The content-type is guessed from the file extension of the URL.
2. If the content-type guessing fails, it will default to application/octet-stream.
3. The content-length will always be undetermined and therefore stated as 0 bytes.

Read the RSS 2.0 spec for further information on enclosing content.

Html2rss.feed(
  channel: {},
  selectors: {
    enclosure: { selector: 'img', extractor: 'attribute', attribute: 'src' }
  }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  enclosure:
    selector: "img"
    extractor: "attribute"
    attribute: "src"

Although this gem's name is html​2rss, it's possible to scrape and process JSON.

Adding json: true to the channel config will convert the JSON response to XML.

Html2rss.feed(
  channel: {
    url: 'https://example.com', json: true
  },
  selectors: {} # ... omitted
)

channel:
  url: https://example.com
  json: true
selectors:
  # ... omitted

This JSON object:

{
  "data": [{ "title": "Headline", "url": "https://example.com" }]
}

converts to:

<hash>
  <data>
    <datum>
      <title>Headline</title>
      <url>https://example.com</url>
    </datum>
  </data>
</hash>

Your items selector would be data > datum, the item's link selector would be url.

Find further information in ActiveSupport's Hash.to_xml documentation.

This JSON array:

[{ "title": "Headline", "url": "https://example.com" }]

converts to:

<objects>
  <object>
    <title>Headline</title>
    <url>https://example.com</url>
  </object>
</objects>

Your items selector would be objects > object, the item's link selector would be url.

Find further information in ActiveSupport's Array.to_xml documentation.

You can add any HTTP headers to the request to the channel URL. Use this to e.g. have Cookie or Authorization information sent or to spoof the User-Agent.

Html2rss.feed(
  channel: {
    url: 'https://example.com',
    headers: {
      "User-Agent": "html2rss-request",
      "X-Something": "Foobar",
      "Authorization": "Token deadbea7",
      "Cookie": "monster=MeWantCookie"
    }
  },
  selectors: {}
)

channel:
  url: https://example.com
  headers:
    "User-Agent": "html2rss-request"
    "X-Something": "Foobar"
    "Authorization": "Token deadbea7"
    "Cookie": "monster=MeWantCookie"
selectors:
  # ...

The headers provided by the channel are merged into the global headers.

By default, html2rss keeps the order of the collection returned from the items selector. The items selector hash can optionally contain an order attribute. If its value is reverse, the order of items in the RSS will reverse.

channel:
  # ... omitted
selectors:
  items:
    selector: 'ul > li'
    order: 'reverse'
  # ... omitted

Note that the order of items, according to the RSS 2.0 spec, should not matter to the feed-consuming client.

This step is not required to work with this gem. If you're using html2rss-web and want to create your private feed configs, keep on reading!

First, create a YAML file, e.g. feeds.yml. This file will contain your global config and multiple feed configs under the key feeds.

Example:

headers:
  "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1"
feeds:
  myfeed:
    channel:
    selectors:
  myotherfeed:
    channel:
    selectors:

Your feed configs go below feeds. Everything else is part of the global config.

Find a full example of a feeds.yml at spec/feeds.test.yml.

Now you can build your feeds like this:

require 'html2rss'

myfeed = Html2rss.feed_from_yaml_config('feeds.yml', 'myfeed')
myotherfeed = Html2rss.feed_from_yaml_config('feeds.yml', 'myotherfeed')

$ html2rss feed feeds.yml myfeed
$ html2rss feed feeds.yml myotherfeed

To display RSS feeds nicely in a web browser, you can:

• add a plain old CSS stylesheet, or
• use XSLT (eXtensible Stylesheet Language Transformations).

A web browser will apply these stylesheets and show the contents as described.

In a CSS stylesheet, you'd use element selectors to apply styles.

If you want to do more, then you need to create a XSLT. XSLT allows you to use a HTML template and to freely design the information of the RSS, including using JavaScript and external resources.

You can add as many stylesheets and types as you like. Just add them to your global configuration.

  config =  Html2rss::Config.new(
    { channel: {}, selectors: {} }, # omitted
    {
      stylesheets: [
        {
          href: '/relative/base/path/to/style.xls',
          media: :all,
          type: 'text/xsl'
        },
        {
          href: 'http://example.com/rss.css',
          media: :all,
          type: 'text/css'
        }
      ]
    }
  )

Html2rss.feed(config)

stylesheets:
  - href: "/relative/base/path/to/style.xls"
    media: "all"
    type: "text/xsl"
  - href: "http://example.com/rss.css"
    media: "all"
    type: "text/css"
feeds:
  # ... omitted

Recommended further readings:

• How to format RSS with CSS on lifewire.com
• XSLT: Extensible Stylesheet Language Transformations on MDN
• The XSLT used by html2rss-web

• Check that the channel URL does not redirect to a mobile page with a different markup structure.
• Do not rely on your web browser's developer console. html2rss does not execute JavaScript.
• Fiddling with curl and pup to find the selectors seems efficient (curl URL | pup).
• CSS selectors are versatile. Here's an overview.

After checking out the repository, run bin/setup to install dependencies. Then, run bundle exec rspec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

1. git pull
2. increase version in lib/html2rss/version.rb
3. bundle
4. git add Gemfile.lock lib/html2rss/version.rb
5. VERSION=$(ruby -e 'require "./lib/html2rss/version.rb"; puts Html2rss::VERSION')
6. git commit -m "chore: release $VERSION"
7. git tag v$VERSION
8. standard-changelog -f
9. git add CHANGELOG.md && git commit --amend
10. git tag v$VERSION -f
11. git push && git push --tags

Bug reports and pull requests are welcome on GitHub at https://github.com/html2rss/html2rss.