Python tips for cleaner scripts: dataclasses, type hints, retries, and logging
Four small Python habits that turn throwaway scripts into tools other people — including future you — can actually run.
Scripts start clean and drift into mud. Four habits stop most of the drift without adding ceremony.
1. Dataclasses for records
Dictionaries are fine until a typo writes titel next to title and nobody notices for a week. A dataclass makes the record explicit:
from dataclasses import dataclass
@dataclass
class Article:
title: str
link: str
published: str = ""
article = Article(title="Hello", link="https://example.com")
print(article.titel) # AttributeError at development time, not next month
For scraped items this is the single highest-value change: one definition of what an "item" is, checked everywhere it is used.
2. Type hints as documentation
def fetch_items(base_url: str, limit: int = 20) -> list[Article]:
...
Editors now autocomplete, mypy catches mismatches, and the signature reads like a sentence. Hints cost nothing at runtime.
3. Retries only when they help
Retry on transient failures — timeouts and 5xx — never on 4xx, which mean "stop asking":
import time
def fetch(url: str, retries: int = 3) -> str:
for attempt in range(retries):
try:
response = requests.get(url, timeout=10)
if response.status_code < 500:
return response.text # 4xx surfaces immediately
except requests.RequestException as exc:
last_error = exc
time.sleep(2**attempt) # 2s, 4s, 8s backoff
raise RuntimeError(f"fetch failed after {retries} attempts: {last_error}")
Exponential backoff with a cap protects both you and the site you are scraping.
4. Logging instead of printing
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("monitor")
log.info("fetched %d items", len(items))
log.warning("zero items found — selector drift?")
The last line is the important one: a scraper that found nothing should make noise. Log files also survive cron, unlike terminal output.
Putting it together
@dataclass
class Article:
title: str
link: str
published: str = ""
def run(base_url: str) -> list[Article]:
log.info("scraping %s", base_url)
html = fetch(base_url)
articles = parse(html)
if not articles:
log.warning("no articles parsed from %s", base_url)
return articles
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("url")
args = parser.parse_args()
for article in run(args.url):
print(article.title, article.link)
None of these habits require a framework — they are just decisions made early. Apply them to the code in the scraping basics lesson and your scripts survive their own success.
When your scripts graduate to a product, the same discipline carries over to the frontend — see Next.js vs Vite vs Astro for how we choose the UI layer.
Frequently asked questions
- Do type hints slow Python down?
- No. They are erased at runtime and only help tools like editors, mypy, and readers understand your code. You get documentation for free.
- When should I use dataclasses instead of dicts?
- As soon as you have a fixed set of fields. Dataclasses catch typos at construction, show up in tooltips, and make refactors safe.
- Why logging instead of print?
- print goes to stdout and mixes with program output; logging carries levels, timestamps, and can be routed to files or sentinels. In a scraper, a warning level is the difference between noticing a failure and a silent zero-item run.