Playwright for Scraping Cheatsheet

Playwright automates Chromium, Firefox, and WebKit. For scraping, use it when a plain HTTP response is missing the data users see after JavaScript runs.

Install & Launch

Open a headless browser and a page.

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="domcontentloaded")
    print(page.title())
    browser.close()

Wait for Content

Avoid racing the page.

page.goto(url, wait_until="networkidle")
page.wait_for_selector(".product-card")
page.locator("text=Load more").click()
page.wait_for_timeout(500)  # last resort; prefer selectors

Extract Data

Locators and rendered HTML.

titles = page.locator("h2.title").all_text_contents()
href = page.locator("a.product").first.get_attribute("href")
html = page.content()  # pass to Beautiful Soup if you want

Pagination & Scroll

Move through lists and infinite scroll.

while True:
    page.locator("text=Next").click()
    page.wait_for_selector(".item")
    # break when Next is disabled or missing

page.evaluate("window.scrollTo(0, document.body.scrollHeight)")

Headers, Timeouts & Cleanup

Be polite and always close browsers.

context = browser.new_context(
    user_agent="MyBot/1.0 (+https://example.com/bot)",
    viewport={"width": 1280, "height": 720},
)
page = context.new_page()
page.set_default_timeout(30_000)
# ... scrape ...
context.close()
browser.close()

---

Prefer Playwright for dynamic UIs; prefer requests + Beautiful Soup for static HTML. Keep concurrency low, throttle requests, and respect robots.txt.

For full documentation, see https://playwright.dev/python/docs/intro