Choose Your Web Scraping Stack: Beautiful Soup, Scrapy, or Playwright

The wrong scraping stack feels like fighting the website. The right one feels boring — fetch, parse, store, repeat. Here is how to choose between the three setups most Python developers actually use.

The decision in one minute

Situation	Reach for
Static or server-rendered HTML, a few pages	requests + Beautiful Soup
Many URLs, pipelines, retries, sitemap crawls	Scrapy
Content appears only after JavaScript runs	Playwright (or Puppeteer)
Site offers a stable JSON API	Call the API — skip scraping

New to the idea itself? Start with Getting Started with Web Scraping and What is Web Scraping?.

requests + Beautiful Soup

Best for: tutorials, one-off scripts, internal tools, pages where “View Source” already shows your data.

Why it works: You control every line. requests handles HTTP; Beautiful Soup turns HTML into something you can query. See the Beautiful Soup cheatsheet.

Limits: You build retries, concurrency, caching, and crawl rules yourself. JS-heavy pages return empty shells.

import requests
from bs4 import BeautifulSoup

html = requests.get(url, timeout=30).text
soup = BeautifulSoup(html, "lxml")
price = soup.select_one(".price").get_text(strip=True)

Scrapy

Best for: crawls across thousands of URLs, teams that want structure, and jobs that need queues, pipelines, and polite defaults.

Why it works: Scrapy gives you spiders, item pipelines, auto-throttling, and export formats out of the box. You spend less time reinventing crawl infrastructure.

Limits: Steeper learning curve than a 20-line script. For heavy client-side rendering you still bolt on a browser (Playwright/Splash) or find the underlying XHR API.

Use Scrapy when the hard part is orchestration, not “can I find this one div?”

Playwright

Best for: React/Vue/SPA pages, infinite scroll, “Load more” buttons, and anything where the first HTML response is a loading spinner.

Why it works: It is a real browser. JavaScript runs; the DOM matches what users see. See What is a Headless Browser? and the Playwright scraping cheatsheet.

Limits: Heavier CPU and memory. Slower than pure HTTP. Easier to over-engineer a job that only needed requests.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(url, wait_until="networkidle")
    titles = page.locator("h2.card-title").all_text_contents()
    browser.close()

Deep dive: Scrape JavaScript-Heavy Sites with Playwright.

A simple chooser flowchart

1. Is there an official API? Use it.
2. Does curl or requests already return the data? Beautiful Soup (or Scrapy if the crawl is large).
3. Is the data only in the rendered page or in XHR responses you can call directly? Prefer calling the JSON endpoint if it is public and stable; otherwise Playwright.
4. Do you need scheduling, retries, and multi-spider pipelines? Scrapy (optionally with Playwright for the JS bits).

Hybrid setups are normal

Production systems often combine tools:

• Scrapy for scheduling + Playwright only for the JS routes
• Playwright to capture HTML, Beautiful Soup to parse it
• HTTP for 90% of pages, browser for the awkward 10%

Do not start hybrid on day one. Start with the smallest stack that returns correct data, then add structure when volume demands it.

Ethics still beat tooling

No stack makes aggressive crawling polite. Respect robots.txt, throttle, and stay on public data. Tooling choices and legal/ethical choices are separate layers — read Is Web Scraping Legal? before you industrialize a crawler.

Bottom line

• Learn with Beautiful Soup
• Scale crawls with Scrapy
• Render with Playwright when HTML alone lies to you

Pick for the page you have, not the blog post you bookmarked.