2 min readRSSFlux Research

Web scraping with Python: fetch a page and extract what matters

A practical introduction to Python web scraping with requests and BeautifulSoup — with the etiquette, error handling, and cleanup that keep scrapers alive.

pythonweb-scrapinglessons

Web scraping is just asking a computer to read a page for you. The recipe is the same at every scale: fetch, parse, extract, then store only what changed. This lesson covers that loop in Python and the details that keep a scraper working for months instead of days.

The two-minute scraper

Fetch the HTML with requests, then walk it with BeautifulSoup:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "https://example.com/news"
html = requests.get(BASE, timeout=20, headers={"User-Agent": "LessonBot/1.0"})
html.raise_for_status()

soup = BeautifulSoup(html.text, "html.parser")
for card in soup.select("article.teaser"):
    title = card.select_one("h2 a").get_text(strip=True)
    link = urljoin(BASE, card.select_one("h2 a")["href"])
    print(title, link)

Three things in that example matter more than they look:

  • timeout=20 — without it a hung server freezes your job forever.
  • A user agent — some sites block requests that look headless.
  • urljoin — relative links are the most common silent bug in scrapers.

Respect the site

Before you scale up, check robots.txt, keep your request rate low, and cache pages you have already fetched. A scraper that re-downloads the same listing every minute is a burden on the site and gets blocked.

Practical defaults: one request per second per domain, retry only on server errors (5xx), and never hammer a page just because your loop allows it.

What breaks first

Selectors drift, encodings surprise you, and dates show up as prose.

  • Wrap parsing in a try/except and log which URL failed instead of crashing.
  • Prefer semantic selectors (article, h2 a) over brittle paths like div:nth-child(7).
  • Normalize dates with dateutil.parser so "3 days ago" becomes a real timestamp.
  • Deduplicate by canonical URL, not by title — titles repeat.

If a page needs JavaScript to render, a plain fetch returns an empty shell. That is the subject of the Playwright lesson.

Turn the output into something useful

Raw prints are only useful for five minutes. Write to a CSV, a database, or — better — publish the results as an RSS feed so any reader can subscribe to what your code found.

If you would rather not maintain a scraper at all, RSSFlux does this loop for you: point it at a listing page and it produces a feed with titles, links, dates, and thumbnails, refreshed on a schedule. The free plan covers three feeds.

Checklist before you ship

  • Timeout and a user agent on every request
  • Politeness delay + respect for robots.txt
  • Relative links resolved with urljoin
  • Dates parsed into real timestamps
  • Errors logged, not printed
  • Deduplication by canonical link

Start with one site and run the scraper daily for a week. The problems you hit will teach you more than any tutorial — and they are all covered in the other lessons on this blog.

Frequently asked questions

Is web scraping legal?
Scraping public pages is generally acceptable, but you must respect robots.txt, a site's terms, and copyright law. Scrape politely, cache results, and never hit login-protected content without permission.
What Python libraries should I use?
Start with requests for fetching and BeautifulSoup for parsing. Add httpx for timeouts/retries, and Playwright only when the page renders content with JavaScript.
How do I keep a scraper from breaking?
Pin selectors loosely, resolve relative URLs with urljoin, parse dates into real timestamps, deduplicate by canonical link, and log failures instead of printing.

Do this without writing code

RSSFlux turns any public web page into a feed, keeps it fresh, and routes updates to Slack, Discord, Telegram, email, or your own API. Start free.

Related reading