1 min readRSSFlux Research

Scraping JavaScript-rendered pages with Python and Playwright

When a page returns an empty shell, Playwright renders it first. Here is how to detect the problem and write a scraper that survives it.

pythonweb-scrapingplaywrightlessons

A page can look perfect in your browser and arrive as an empty shell in your scraper. The reason: the server sends a skeleton, and JavaScript builds the content in the browser. Plain requests never runs that JavaScript.

Detect the problem first

Fetch and search for a phrase you can see in your browser:

import requests

html = requests.get("https://example.com/app", timeout=20).text
print("shell only:", "expected headline text" not in html)

Before reaching for a browser, check the page's network traffic for a JSON API — endpoints like /api/articles?page=1 are faster and more stable than rendering. Only use Playwright when no API exists.

Render with Playwright

Install playwright and run playwright install chromium once. Then:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.route("**/*", lambda route: route.abort()
        if route.request.resource_type in ("image", "media", "font")
        else route.continue_())
    page.goto("https://example.com/app", wait_until="domcontentloaded")
    page.wait_for_selector("article.teaser", timeout=10000)
    cards = page.query_selector_all("article.teaser")
    for card in cards:
        title = card.query_selector("h2").inner_text()
        link = card.query_selector("h2 a").get_attribute("href")
        print(title, link)
    browser.close()

The details that make this reliable:

  • Block images, media, and fonts — pages render much faster without them.
  • Wait for a selector, not networkidle — networkidle waits for endless trackers.
  • Reuse the browser across URLs; launching Chromium per page does not scale.
  • Set a timeout so a broken page fails loudly instead of hanging.

When the browser changes behavior

JavaScript sites change more often than static ones. Keep the selectors in one place, log the URL and the selector on failure, and re-check your rules whenever a run returns zero items. A scraper that silently returns nothing is worse than one that crashes — at least the crash wakes you up.

Use a service instead of your own browser fleet

Maintaining Chromium, retries, and anti-bot handling is real work. If scraping is not your core business, a managed fallback is worth it: RSSFlux fetches pages directly first and automatically escalates to browser rendering when the page needs it — you paste a URL instead of running Playwright yourself.

For the no-code view of the same problem, read How to scrape JavaScript sites into RSS. If you are still on plain fetch-and-parse, go back to the basics lesson first.

Frequently asked questions

How do I know a page needs a browser?
Fetch it with requests and search the HTML for text you can see in your browser. If the text is missing, the page is rendered client-side and needs Playwright or a rendering service.
Is Playwright slow?
Slower than a plain request by design. Mitigate it by blocking images, fonts, and media, waiting for a specific selector instead of a fixed sleep, and reusing a browser instance.
Can a scraping service replace writing this myself?
Yes. Services render pages for you and handle retries and anti-bot changes. RSSFlux does exactly this fallback: plain fetch first, browser rendering when the page needs it.

Do this without writing code

RSSFlux turns any public web page into a feed, keeps it fresh, and routes updates to Slack, Discord, Telegram, email, or your own API. Start free.

Related reading