How to Scrape JavaScript-Heavy Sites with Playwright

If your scraper downloads a page and the important content is missing, you are probably looking at a JavaScript-heavy site. The server sent a shell; the browser was supposed to finish the job. Playwright lets your code be that browser.

Background: What is a Headless Browser?. Quick reference: Playwright for Scraping Cheatsheet.

Confirm you need a browser

Before you launch Chromium, check:

1. View Source (not Inspect) — is the data in the raw HTML?
2. Network tab — is there a JSON XHR/fetch you can call directly?
3. requests.get(url).text — same emptiness?

If a clean JSON endpoint returns what you need, call that instead of driving a full browser. It is faster and more stable.

Install Playwright (Python)

pip install playwright
playwright install chromium

Minimal working example

from playwright.sync_api import sync_playwright

url = "https://example.com"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        user_agent="ResearchBot/1.0 (+https://plainenglish.io)"
    )
    page = context.new_page()
    page.goto(url, wait_until="domcontentloaded")
    page.wait_for_selector("h1")
    title = page.locator("h1").inner_text()
    print(title)
    context.close()
    browser.close()

Waiting is the whole game

Most flaky scrapers lose a race. Prefer condition waits over fixed sleeps:

page.goto(url, wait_until="networkidle")
page.wait_for_selector(".product-card")
page.locator(".product-card").first.wait_for()

Use wait_for_timeout only as a last resort while debugging — it hides timing bugs and slows every run.

Extracting data

Locators (Playwright-native):

cards = page.locator(".product-card")
count = cards.count()
rows = []
for i in range(count):
    card = cards.nth(i)
    rows.append({
        "name": card.locator("h2").inner_text().strip(),
        "price": card.locator(".price").inner_text().strip(),
    })

HTML + Beautiful Soup (when you prefer CSS soup APIs):

from bs4 import BeautifulSoup

soup = BeautifulSoup(page.content(), "lxml")

Pagination

Classic next-button pattern:

while True:
    # ... extract current page ...
    next_btn = page.locator("a.next")
    if next_btn.count() == 0 or not next_btn.is_enabled():
        break
    next_btn.click()
    page.wait_for_selector(".product-card")

Infinite scroll

previous_height = 0
while True:
    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    page.wait_for_timeout(800)
    height = page.evaluate("document.body.scrollHeight")
    if height == previous_height:
        break
    previous_height = height

Tighten this for production: wait for a specific new node count instead of a blind timeout.

Common pitfalls

1. Scraping the loading state — you did not wait for the selector that means “data is here.”
2. Ignoring lazy images/text — content exists only after scroll or intersection observers fire.
3. Hard-coding sleeps — works on your laptop, fails in CI or under load.
4. Opening a browser per URL with no pooling — burns memory; reuse a browser/context carefully.
5. Fighting anti-bot with volume — if you are blocked, first slow down, respect robots.txt, and confirm you are on public pages. See also How to Build Scrapers That Survive Anti-Bot Updates.

Playwright vs. Beautiful Soup vs. Scrapy

• Static HTML → Beautiful Soup (stack guide)
• Large crawl graph → Scrapy (± Playwright for JS routes)
• Client-rendered UI → Playwright

Keep it ethical

Headless does not mean invisible or exempt from rules. Throttle, identify your bot, prefer APIs, and stay on public data. Legal overview: Is Web Scraping Legal?.

Wrap-up

Playwright is the right tool when the browser is part of the data path. Wait for real conditions, extract with locators or parsed HTML, handle pagination deliberately, and only then optimize for speed. Correct and polite beats clever and blocked.