Why Concurrent Puppeteer Scraping Is Slow in Node.js (And How to Get a 3.3x Speedup)

In my previous benchmark testing Effect 4 against Effect 3 on a Node.js web scraping pipeline, I ran into a bottleneck: 50 concurrent Puppeteer pages on a single browser capped throughput at 3.29 tasks/s, even when Effect 4's underlying Node.js scheduler could run 724,585 tasks/s. Obviously, this was not Effect's fault, but I wanted to know where those 15+ seconds were actually going — and whether the standard advice you see on Stack Overflow (waitUntil: "domcontentloaded" and blocking images/CSS with page.setRequestInterception(true)) actually fixed it.

Just like my previous pipeline, the production setup here uses residential proxies, Web Unlocker, and Scraping Browser from Bright Data to handle IP rotation, CAPTCHA solving, and remote browser scaling once we ungate local concurrency.

TL;DR: The usual fixes DON'T work (and request interception actually makes scraping slower). Because the real culprit wasn't renderer CPU or bandwidth at all — it was a hidden HttpCache lock inside Chromium's shared BrowserContext! Breaking that lock with browser.createBrowserContext() gives an instant 3.3x speedup, while pairing isolated contexts with Bright Data stops IP bans and scales past local Chromium limits.

Benchmark Methodology: Profiling 6 Concurrency Modes in Node.js and Puppeteer

If you ask most TypeScript or Node.js developers why N concurrent Puppeteer tabs take ages to finish, they'll blame one of three things:

1. waitUntil: "networkidle2" waiting 500 ms for network quiet on every tab.
2. Heavy static resources (CSS, fonts, images) eating bandwidth and Blink layout CPU.
3. V8 JavaScript execution and DOM serialization (page.content()) saturating the renderer thread.

Why guess? I tested six scraping configurations across N = 1, 10, and 50 concurrent tabs (each run across 3 fresh node --expose-gc processes), wrapping every Puppeteer phase in performance.now() timestamps and pulling two internal telemetry sources from every tab before closing it:

• W3C PerformanceNavigationTiming to measure stalledMs (requestStart - fetchStart — how long a tab sits queued inside Chromium before a single byte leaves the browser) and ttfbMs (responseStart - requestStart).
• CDP page.metrics() to measure how much time the tab's renderer thread actually spent running JavaScript and layout (TaskDuration).

Because the static HTML on quotes.toscrape.com/js/ actually lists zero quotes until client-side JavaScript runs, a scrape should only count as valid if Cheerio extracts all 10 rendered quotes and authors.

Setup: Windows 10 x64, Node v24.19.0, puppeteer@22.10.0 (Headless Chrome/125.0.6422.78), cheerio@1.0.0-rc.12, target URL https://quotes.toscrape.com/js/. All numbers below are the median of 3 isolated runs.

Puppeteer Concurrency Benchmark Results (N = 50 Concurrent Pages)

At 50 concurrent tabs, the standard advice falls flat. Switching from networkidle2 to domcontentloaded barely helps throughput (2.82 vs. 2.98 tasks/s), and adding request interception actually makes it slower (2.77 tasks/s). Only isolating each tab in its own BrowserContext (isolated-contexts-dcl) breaks the queue — cutting down stall time 11.3x and boosting throughput 3.3x to 9.23 tasks/s:

Benchmark Mode (N = 50)	Wait & Browser Configuration	Throughput (tasks/s)	Total Wall Time (ms)	Latency p50 / p95 (ms)	Browser Queue Stalled Median (Min–Max ms)	Renderer TaskDuration (ms/tab / Sum ms)	Valid Quotes (10/10 Quotes)
1. networkidle2 (Baseline)	1 shared context (browser.newPage()), waitUntil: "networkidle2"	2.82	17,709	10,902 / 17,686	8,556 (2,180–14,883)	53.2 / 2,691	50 / 50
2. domcontentloaded	1 shared context, waitUntil: "domcontentloaded"	2.98	16,766	10,213 / 16,244	8,899 (2,349–15,211)	18.6 / 1,074	50 / 50
3. intercept-networkidle2	Shared context + block CSS/fonts/images + networkidle2	2.68	18,633	12,508 / 18,605	9,276 (1,959–16,199)	30.4 / 1,547	50 / 50
4. intercept-dcl	Shared context + block CSS/fonts/images + domcontentloaded	2.77	18,046	10,580 / 17,440	8,836 (1,757–15,800)	29.6 / 1,519	50 / 50
5. isolated-contexts-dcl	50 BrowserContexts (createBrowserContext) + DCL	9.23	5,415	4,977 / 5,367	786 (627–841)	76.0 / 3,727	50 / 50
6. http-fetch	Native Node.js fetch() + Cheerio (no local browser)	48.92	1,022	973 / 1,000	0	0.0 / 0	0 / 50 (no JS execution)

Look at the Renderer TaskDuration column first. Even though the 50-tab networkidle2 run took 17.7 seconds, each tab's renderer thread spent only 53.2 ms doing actual V8 and Blink work (18.6 ms on domcontentloaded). Renderer CPU was never the bottleneck! The browser wasn't sweating over JavaScript or DOM layout at all — the tabs were just sitting around waiting.

Where were those 17.7 seconds actually going? Look at how the bottleneck completely flips when we break each tab's lifecycle into four phases:

Puppeteer Lifecycle Phase (N = 50, per-tab median)	networkidle2 Median (ms)	domcontentloaded Median (ms)	intercept-dcl Median (ms)	isolated-contexts-dcl Median (ms)	Share of Tab Time (Shared Baseline)	Share of Tab Time (Isolated dcl)
1. Context + newPage()	910.6 ms	894.4 ms	880.8 ms	2,314.9 ms	7.87%	46.56%
2. page.goto()	10,712.7 ms	9,501.8 ms	9,706.1 ms	2,450.2 ms	92.05%	48.85%
3. page.content() + Cheerio	3.8 ms	4.6 ms	18.7 ms	64.1 ms	0.04%	1.53%
4. page.close() + context.close()	2.0 ms	2.1 ms	1.9 ms	110.0 ms	0.03%	2.55%

This phase table tells a wild two-part story:

1. In a shared BrowserContext: Opening and parsing a tab is dirt cheap, while 92% of every tab's life is trapped inside page.goto() (~10 seconds of dead air).
2. In isolated BrowserContexts: page.goto() drops nearly 4x (to 2.45 seconds), but context and tab creation explodes to 46.6% of the entire lifecycle (2.31 seconds per tab)!

Let's look at why page.goto() stalls so badly in a shared context first — and why the standard Stack Overflow fixes don't touch it.

Takeaway 1: Why domcontentloaded and setRequestInterception Fail to Speed Up Concurrent Puppeteer

With a single tab (N = 1), switching to "domcontentloaded" and blocking images/CSS with page.setRequestInterception(true) works as advertised — shaving 27% off the scrape time. At 50 concurrent tabs, however, those exact same tweaks do nothing at all:

Concurrency Scale	networkidle2 (tasks/s / Wall)	domcontentloaded (tasks/s / Wall)	intercept-dcl (tasks/s / Wall)	isolated-contexts-dcl (tasks/s / Wall)	Speedup vs. networkidle2 Baseline
N = 1 (Single Tab)	0.19/s (5,137 ms)	0.25/s (3,954 ms)	0.27/s (3,736 ms)	0.54/s (1,861 ms)	2.76x faster
N = 10 (10 Tabs)	1.47/s (6,792 ms)	1.60/s (6,257 ms)	1.71/s (5,848 ms)	4.29/s (2,330 ms)	2.92x faster
N = 50 (50 Tabs)	2.82/s (17,709 ms)	2.98/s (16,766 ms)	2.77/s (18,046 ms)	9.23/s (5,415 ms)	3.27x faster

Here is the interception and timing code I ran:

const BLOCKED_RESOURCE_TYPES = new Set(["image", "stylesheet", "font", "media"]);

if (isIntercept) {
  await page.setRequestInterception(true);
  page.on("request", (req) => {
    if (BLOCKED_RESOURCE_TYPES.has(req.resourceType())) {
      void req.abort();
    } else {
      void req.continue();
    }
  });
}

await page.goto(TARGET_URL, {
  waitUntil: "domcontentloaded",
  timeout: 60_000,
});

But these results are a little confusing. Why did intercept-dcl actually run slower than your garden variety domcontentloaded at N = 50?

1. CDP round-trip tax: Turning on setRequestInterception(true) pauses every network request inside Chromium, fires an event over the DevTools pipe to Node.js, and waits for Node.js to reply with req.abort() or req.continue(). Across 250 requests (50 tabs $\times$ 5 requests), all this back-and-forth IPC chatter costs more time than blocking a tiny stylesheet saves.
2. You CAN'T actually block the scripts that actually render the page: To get the quotes on quotes.toscrape.com/js/, we have to allow the HTML document and <script src="/static/jquery.js">. And those exact allowed requests are what get stuck for ~9 seconds thanks to Chromium's cache lock.

How Chromium's HttpCache Lock Serializes Same-Resource Fetches in browser.newPage()

When you call browser.newPage() 50 times in Puppeteer, all 50 tabs share the default BrowserContext — and the same underlying HttpCache. To keep concurrent tabs from corrupting cache entries, Chromium enforces an exclusive single-writer lock per cache key (HttpCache::Transaction). Whenever 50 tabs fetch the same URL or a shared script like <script src="/static/jquery.js">, only one tab can create or validate that cache entry at a time while the other 49 wait in line.

You can see that single-file line in the stalledMs percentiles across all 50 concurrent tabs:

Tab Queue Percentile (N = 50)	Shared networkidle2 stalledMs	Shared domcontentloaded stalledMs	Shared intercept-dcl stalledMs	Isolated createBrowserContext() stalledMs
Tab 1 (Min)	2,180.1 ms	2,348.9 ms	1,756.5 ms	626.9 ms
Tab 5 (p10)	3,243.8 ms	3,630.5 ms	2,880.5 ms	712.8 ms
Tab 13 (p25)	5,326.0 ms	5,691.8 ms	5,215.2 ms	762.5 ms
Tab 25–26 (p50 Median)	8,556.3 ms	8,899.1 ms	8,835.6 ms	786.4 ms
Tab 38 (p75)	11,789.9 ms	12,114.3 ms	12,441.8 ms	808.0 ms
Tab 48 (p95)	14,366.0 ms	14,691.6 ms	15,232.5 ms	826.4 ms
Tab 50 (Max)	14,883.0 ms	15,210.7 ms	15,799.9 ms	841.0 ms

Here is what makes that staircase so brutal in a shared BrowserContext:

• The remote server was never slow! Actual server response time (ttfbMs) was a snappy 258 ms median across all 50 tabs.
• Tabs were stuck waiting in Chromium's own lobby: Queue stall climbed in a straight line from 2.3 seconds on Tab 1 to 15.2 seconds on Tab 50 (~262 ms per tab step) before a single byte even went over the wire.
• Even scraping 50 different URLs on the same domain won't save you: Every page still loads <script src="/static/jquery.js">. Because DOMContentLoaded cannot fire until synchronous <script> tags finish executing, all 50 tabs still pile up single-file behind the cache validation lock for jquery.js!

Takeaway 2: createBrowserContext() Fixes It (3.3x Faster) But Moves the Bottleneck to Context Provisioning

To bypass Chromium's shared HttpCache lock and connection pool contention without launching 50 heavy OS browser processes, create an isolated BrowserContext per task:

const runIsolatedTask = async (browser: Browser, url: string) => {
  // Creates an isolated in-memory cache partition and socket pool inside the same Chromium process
  const context = await browser.createBrowserContext();
  const page = await context.newPage();

  try {
    await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30_000 });
    return await page.content();
  } finally {
    await page.close();
    await context.close();
  }
};

💡 Puppeteer v22.0.0 replaced browser.createIncognitoBrowserContext() with browser.createBrowserContext(). On Puppeteer < v22, call browser.createIncognitoBrowserContext().

Giving each tab its own BrowserContext (isolated-contexts-dcl) breaks the shared cache lock immediately — slashing median stall time 11.3x (to 786 ms) and boosting 50-page throughput 3.27x to 9.23 tasks/s (5,415 ms total wall time).

But Why Did Isolated Throughput Stop at 9.23 tasks/s?

Hold on — if page.goto() dropped to 2.45 seconds and network stall dropped under 800 ms, why did our 50-tab run still take 5.4 seconds instead of ~2.5 seconds?

That's because we traded a network-service cache lock, for a browser-process provisioning bottleneck:

• Spawning 50 Isolated Contexts at Once Chokes the Browser Process: Creating an isolated context and tab took just 72 ms at N = 1, and 412 ms at N = 10. At N = 50, forcing Chromium's main browser process to initialize 50 independent StoragePartitions at the same time pushed createBrowserContext() + newPage() to 2.31 seconds per tab — eating nearly half of every tab's life! (Actually, destroying them with close() also skyrocketed 52x, from 2 ms to 110 ms.)
• Renderer CPU Finally Wakes Up: When tabs were trickling through an 18-second cache queue, renderer CPU was asleep (6.4% of wall time). Once all 50 tabs actually loaded in parallel inside a 5.4-second window, all 50 V8 and Blink threads fired at once — spiking the total renderer work up to 73.1% of wall time.

Pro Tip (Context Pooling): If you run local Puppeteer in production, don't create and destroy a BrowserContext on every single URL. Pre-allocate a warm pool of $K$ isolated BrowserContexts at startup and reuse them (or just call await page.setCacheEnabled(false) if you don't need cookie/storage isolation between scrapes).

Takeaway 3: Unblocking Local Concurrency Triggers Immediate IP Bans (Bright Data Solves It)

Isolated BrowserContexts means your single local Chromium instance no longer reuses cached assets or a single connection pool — instead, it fires N independent TLS handshakes and 5*N uncached HTTP requests concurrently from a single IP address. Against virtually any production target protected by Cloudflare, DataDome, or rate limiters, a burst like this triggers immediate HTTP 429 rate limits and HTTP 403 CAPTCHA blocks -- unless each context routes through Bright Data Residential Proxies.**

Big deal, so we'll just slap a single proxy on the browser, right? Wrong.

A proxy is still on the network, so it cannot fix a shared BrowserContext stall like this. Once createBrowserContext() ungates all 50 tabs simultaneously, you need per-context IP rotation so those 50 parallel sessions don't all exit from one IP.

Here's how to wire both into runIsolatedTask.

How to Configure Bright Data Per-Session Proxy Rotation and Retries in Puppeteer

When you use createBrowserContext(), pass --proxy-server=brd.superproxy.io:33335 at browser launch and call page.authenticate() inside runIsolatedTask with a unique -session-<id> credential from your Bright Data proxy zone. If a target throws an HTTP 403 CAPTCHA or HTTP 429 rate limit, retrying with a new sessionId forces Bright Data's superproxy to assign a fresh residential exit IP:

const browser = await puppeteer.launch({
  headless: true,
  args: ["--no-sandbox", "--proxy-server=brd.superproxy.io:33335"],
});

// Inside runIsolatedTask(browser, url, attempt) before page.goto():
const sessionId = `${taskId}_attempt_${attempt}`;
await page.authenticate({
  username: `brd-customer-${CUSTOMER_ID}-zone-${ZONE}-session-${sessionId}`,
  password: PASSWORD,
});

const response = await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30_000 });
if (response && (response.status() === 403 || response.status() === 429)) {
  throw new Error(`Retryable anti-bot status ${response.status()} on session ${sessionId}`);
}

How To Choose the Right Node.js Scraping Architecture for High Concurrency

If you take one thing away from this benchmark, let it be this: never fan out browser.newPage() inside a single shared BrowserContext. Isolating tabs with createBrowserContext() (ideally from a warm pool) breaks Chromium's cache lock for an instant 3.3x local speedup, and pairing each context with a rotating residential proxy keeps that burst from getting your IP banned.

Still, a single machine can only spin up so many browser contexts, and plain fetch() — while 17x faster — is useless on JavaScript-rendered pages. When building high-throughput Node.js and TypeScript web data pipelines for RAG or LLM training, you can keep fetch() speed without giving up full JS rendering by offloading Chromium to Bright Data:

• Bright Data Web Unlocker (Keep Mode 6 fetch() Speed): Send standard HTTP requests from Node.js while Web Unlocker handles headless JS rendering, browser fingerprinting, CAPTCHA solving, and residential IP rotation at the edge.
• Bright Data Scraping Browser (Remote Browser Cluster over WebSocket): For multi-step Puppeteer workflows (page.click(), form fills, infinite scroll), replace puppeteer.launch() with puppeteer.connect() so each concurrent worker runs on its own dedicated cloud browser:

// 1. Bright Data Scraping Browser: scale multi-step Puppeteer past 50 local contexts
export const connectRemoteScrapingBrowser = () =>
  puppeteer.connect({
    browserWSEndpoint: `wss://brd-customer-${CUSTOMER_ID}-zone-${ZONE}:${PASSWORD}@brd.superproxy.io:9222`,
  });

// 2. Bright Data Web Unlocker: get rendered HTML + automatic CAPTCHA solving at fetch() speed
export const fetchWithWebUnlocker = async (targetUrl: string) => {
  const res = await fetch("https://api.brightdata.com/request", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${BRIGHT_DATA_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ zone: WEB_UNLOCKER_ZONE, url: targetUrl, format: "raw" }),
  });
  return res.text();
};

That's everything! I'll sign off with this quick summary you can look up.

Scraping Architecture	Measured / Target Concurrency (N = 50)	Browser Queue Stalled Median	Primary Bottleneck	Captures Dynamic JS Quotes (10/10)?	Anti-Bot, CAPTCHA & IP Rotation	Best Use Case
Puppeteer Shared Context (networkidle2)	2.82 tasks/s (17,709 ms)	8,556 ms	Shared HttpCache lock (92.1% in goto)	Yes (50/50)	None	Single-page scripts (N = 1) only
Puppeteer Shared Context (intercept-dcl)	2.77 tasks/s (18,046 ms)	8,836 ms	Shared HttpCache lock + CDP IPC	Yes (50/50)	None	Single-page bandwidth reduction
Puppeteer Isolated Contexts + Bright Data Residential Proxies	9.23 tasks/s (5,415 ms)	786 ms	Context creation (46.6% in newPage)	Yes (50/50)	Per-context IP rotation via brd.superproxy.io:33335 (-session-<id>)	Medium-scale (10–50 concurrent pages) JS scraping on 1 server
Raw Node.js fetch() + Cheerio (No Proxy)	48.92 tasks/s (1,022 ms)	0 ms	Network TTFB only	No (0/50)	None (immediate IP blocks)	Static SSR HTML pages with no anti-bot protection
Puppeteer + Bright Data Scraping Browser	100–1,000+ remote sessions	~0 ms local	Remote network latency	Yes (50/50)	Built-in remote browser fingerprinting, IP rotation & CAPTCHA solver	High-concurrency interactive Puppeteer workflows
Node.js HTTP + Bright Data Web Unlocker	1,000+ concurrent requests	0 ms local	Edge response time	Yes (Edge-rendered)	Automated JS rendering, CAPTCHA solving & residential IP rotation at proxy edge	High-throughput AI/RAG data pipelines & scale scraping