What is a Headless Browser?

A headless browser is a full browser engine (Chromium, Firefox, WebKit) that runs without showing a UI window. It still downloads resources, executes JavaScript, builds the DOM, and can take screenshots — you just drive it with code instead of a mouse.

For scrapers, that matters because many modern sites ship almost empty HTML and fill the page with JavaScript. A simple HTTP GET never sees the content users see.

How Scrapers Use Them:

1. Launch Chromium (or another browser) in headless mode
2. Navigate to a URL
3. Wait for selectors, network idle, or specific text
4. Read the rendered DOM or intercept JSON responses
5. Extract data, then close the page

Popular Tools:

• Playwright: Multi-browser, strong waiting APIs, great for scraping and tests
• Puppeteer: Chromium-focused automation from the Chrome DevTools Protocol
• Selenium: Older, still widely used across languages

Trade-offs vs. Plain HTTP:

• Pro: Sees the same page a user sees after JS runs
• Pro: Can click, scroll, and handle light interaction
• Con: Heavier CPU/RAM and slower than requests + Beautiful Soup
• Con: Easier to fingerprint as automation if you are careless

When You Need One:

• Infinite scroll or "Load more" buttons
• Content injected after XHR/fetch calls
• Client-side routing (many React/Vue apps)
• Pages that break without cookies or a real viewport

When You Do Not:

• Static HTML or server-rendered pages
• A clean JSON API you can call directly
• Simple sites where curl already returns the data

FAQ

Is headless the same as invisible to the site?

No. Sites can still detect automation via TLS fingerprints, JS checks, and behavior. Headless only means "no window" — not "undetectable."

Should every scraper use Playwright?

No. Start with HTTP + a parser. Reach for a headless browser when the HTML response is missing the data you need.