How to scrape JavaScript sites into RSS (without guesswork)

Some pages look fine in a browser and arrive empty in a scraper. The usual reason is JavaScript: the server sends a skeleton, and the actual items appear only after the browser runs the site's code.

This guide covers how to detect that, how to handle it reliably, and when to let a managed service do the work.

First: confirm it is a JavaScript problem

Fetch the page with curl and search for the item titles you see in the browser:

curl -s "https://example.com/news" | grep -i "headline you expected"

If the title is missing, the content is rendered client-side. You may still find the data in one of these places:

• A JSON endpoint the page calls (open DevTools → Network → fetch/XHR).
• Embedded state such as __NEXT_DATA__, window.__INITIAL_STATE__, or a JSON-LD block.
• A sitemap listing article URLs.

These are preferable to browser automation when available: they are smaller, faster, and less likely to break.

Option 1: Use the site's own API or embedded state

Many modern sites expose more than they intend. A quick check of the network panel often reveals an endpoint like:

/api/articles?page=1&limit=20

If the endpoint is stable, poll it and map the fields to RSS. Always set a realistic User-Agent and throttle your requests.

Option 2: Headless browser rendering

When there is no usable endpoint, render the page in a real browser. Playwright is the most common choice:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/news", wait_until="networkidle")
    items = page.query_selector_all("div.article-card")
    for item in items:
        title = item.query_selector("h2").inner_text()
        link = item.query_selector("h2 a").get_attribute("href")
        print(title, link)
    browser.close()

Practical notes:

• Wait for the right condition. networkidle is a start; a selector wait is often more reliable.
• Block images, fonts, and media. It speeds up rendering dramatically.
• Reuse a browser instance. Launching one per item does not scale.
• Expect maintenance. Sites change scripts and selectors.

Option 3: A scraping service with rendering

A managed service handles the browser fleet, retries, and anti-bot changes. The trade-off is cost and less control. This is the right choice when scraping is not your core business.

RSSFlux takes this approach: it tries a normal fetch and native feed discovery first, then uses browser rendering when the page needs it. You paste a URL and get a feed without running Chromium yourself.

Handling Cloudflare and bot protection

Some sites challenge non-browser clients. Options, in order of preference:

1. Use the site's feed or API if one exists.
2. Slow down and use a realistic user agent.
3. Use a rendering service that handles challenges.
4. Use a proxy service when the block is network-level.

Do not build a captcha-solving pipeline unless you have a very good reason and a legal review. It is fragile and often unwelcome.

Turn the result into a feed

Once you can extract items, the RSS part is straightforward:

• Title, link, publish date, description.
• Resolve relative URLs.
• Deduplicate by canonical link.
• Sort newest-first.
• Emit RSS 2.0 or Atom.

The hard part is staying working: selectors break, pages move, and a feed that silently returns zero items is worse than no feed.

A checklist for JavaScript sites

• Check for an existing feed first.
• Inspect the network panel for a JSON endpoint.
• Search the HTML for embedded state or JSON-LD.
• If rendering is needed, wait for a specific selector, not just a timeout.
• Block unnecessary assets to speed up rendering.
• Deduplicate by URL and keep items newest-first.
• Monitor the item count; alert on zero-item runs.

When to use RSSFlux instead

If you want feeds without maintaining a browser fleet, RSSFlux handles JavaScript rendering, Cloudflare bypass on eligible plans, selector overrides, and dead-link filtering. The free plan includes three feeds; Developer and Business plans add faster refreshes and premium proxies.

Related: How to create an RSS feed for a website that doesn't have one and Monitor any website for changes.