Effect 3 vs. Effect 4: Puppeteer & Node.js Scraping Benchmarks

Effect V4 came out recently and promises 6.4x concurrent task throughput and 86% less memory per fiber. Given that I already had an Effect V3 pipeline from January built around Effect's concurrency and resource management, it felt like a no-brainer to test V4 and see if those runtime gains pan out in real-world testing. (Spoiler alert: They mostly do! Turns out I'm still in love with Effect nearly a year later. 💖)

Just like that original project, the pipeline uses these residential proxies from Bright Data to automatically handle the usual issues of fingerprinting, CAPTCHA solving, IP rotation, and geo-targeting.

TL;DR — I could reproduce or exceed Effect 4's runtime claims (6.4x throughput, 86% less memory per fiber) in my benchmarks: 10x (724,585 tasks/s) and 71% (1,487 B per fiber) — while offloading local Chromium bottlenecks (3.29 tasks/s at 50 pages) to Bright Data Web Unlocker or Scraping Browser gets you that concurrency in production.

How to Migrate a Node.js Web Scraping Pipeline from Effect 3 to Effect 4

To quickly recap, my project uses a Puppeteer browser for a single page, with typed errors, acquireUseRelease for the browser and page, exponential retry on transient errors, a 100 ms rate-limit sleep, and Cheerio to parse.

Helpfully, Effect includes their own migration guide for going from V3 to V4. If you have a subscription to a top-of-the-line coding agent, just hand it that and you'll probably one-shot the port as their blog post suggests.

Anyway, the actual API surface that changed in my code was quite small:

Effect 3 (effect@3.19.14)	Effect 4 (effect@4.0.2)	Migration Impact on Scraping Pipelines
Effect.catchAll	Effect.catch	Simpler top-level error recovery across all ScrapingError tags
Context.GenericTag<T>("X")	Context.Service<T>("X")	Unified dependency injection for proxy and browser services
Schedule.exponential(…).pipe(Schedule.intersect(Schedule.recurs(3)))	Schedule.exponential(…).pipe(Schedule.upTo({ times: 3 }))	Cleaner bounded retry schedules for transient network/proxy errors
Effect.fork	Effect.forkChild	Explicit parent-bound fiber supervision that prevents memory leaks
Effect.yieldNow()	Effect.yieldNowWith()	Updated function signature for manual scheduler yielding
moduleResolution: "node"	moduleResolution: "bundler"	Required in tsconfig.json so TypeScript resolves effect v4 exports

Note how Effect.fork has transformed into its much less chill cousin Effect.forkChild. If you're a TypeScript dev who's used to fire-and-forget promises, this is big -- you can no longer casually write Effect.fork without explicitly stating the child fiber's supervision and lifetime relationship. Yes. Every spawned task is explicitly bound to its parent fiber's lifecycle and supervision scope (unless you deliberately opt out with forkDaemon I guess, but let's not go down that rabbit hole in this article).

No more orphan background tasks that leak memory in your bog standard async/await code.

How to Implement Exponential Backoff and Handle HTTP 403 / 429 Errors in Effect 4

In Effect 4, replace Schedule.intersect(Schedule.recurs(3)) with Schedule.upTo({ times: 3 }) and use while: isRetryableError inside Effect.retry to retry transient HTTP 429 rate limits, HTTP 403 blocks, and CAPTCHA errors across rotating proxy sessions.

There are two not-quite-gotchas I found while updating the retry logic:

• Schedule.intersect has been removed in v4 — its replacement for bounding a schedule is Schedule.upTo({ times: 3 }).
• While auditing Effect.retry, I also caught a bug in my original v3 code: I had until: isRetryable when I actually wanted while: isRetryable (both options exist in v3 and v4; until stops when the predicate returns true, whereas while continues while true).

Effect 3:

const retryPolicy = pipe(
  Schedule.exponential(Duration.seconds(1)),
  Schedule.intersect(Schedule.recurs(3))
);

const retryIfRetryable = <A, E extends ScrapingError, R>(
  effect: Effect.Effect<A, E, R>
): Effect.Effect<A, E, R> =>
  Effect.retry(effect, {
    schedule: retryPolicy,
    until: (error) => isRetryableError(error),
  });

Effect 4 (migrated to upTo and fixed to while):

const retryableTags: ReadonlyArray<ScrapingError["_tag"]> = [
  "NetworkError",
  "TimeoutError",
  "RateLimitError", // HTTP 429
  "IPBlockError",   // HTTP 403 / CAPTCHA block
  "BrowserError",
];

const retryPolicy = Schedule.exponential(Duration.seconds(1)).pipe(
  Schedule.upTo({ times: 3 }),
);

const isRetryableError = (error: ScrapingError): boolean =>
  retryableTags.includes(error._tag);

const retryIfRetryable = <A, E extends ScrapingError, R>(
  effect: Effect.Effect<A, E, R>,
): Effect.Effect<A, E, R> =>
  Effect.retry(effect, {
    schedule: retryPolicy,
    while: (error) => isRetryableError(error),
  });

You'll also need moduleResolution: "bundler" in tsconfig.json (instead of "node") or effect will not resolve.

Everything else I was worried about — Data.TaggedError, Effect.tryPromise, Effect.acquireUseRelease, Effect.sleep — seem to have ported over without changes.

Benchmark Methodology: Testing Effect 3 vs. Effect 4 Across Three Scraping Tiers

For my project, a one-shot scrape cannot test the Effect V4 claims. Effect's V4 announcement highlights three big improvements: a 5x smaller bundle (via tree-shaking), 6.4x task throughput, and 86% less memory per fiber.

We can skip the bundle size claim right away. First off, my code is a server-side Node.js script launching a local Chromium instance, not a browser app or size-constrained edge worker; secondly, dependencies like Cheerio (and its DOM stack) will obviously completely dwarf Effect in the module graph anyway. A reduction in a minimal tree-shaken bundle is irrelevant for me.

That leaves the fiber runtime claims. My scraping use case for Effect is mostly for the DX. It's just puppeteer.launch, page.goto, and page.content(). Effect is not going to be a big part of that wall time.

So I'm just gonna split the benchmark into three tiers and report them separately like this:

1. A single scrape — the actual pipeline, run as-is. This should measure wall time, Node peak RSS and heap (and naturally, make sure that both versions parse the same result.)
2. Concurrent scrapes — Scrape N pages using a shared browser, with concurrency: "unbounded". N = 10 and N = 50. This will measure tasks/s, p50, p95, and peak memory.
3. Fiber microbench — Leave Puppeteer and the network out of it. This runs 200k concurrent yieldNow tasks for throughput, and 50k parked Effect.never fibers for heap. I'll take the median of 9 fresh Node processes, matching Effect's stated method.

Did the app get faster because of Effect v4? #1 and #2 will answer that. And #3 will answer how much better the runtime got, and how Effect's own claims stack up in isolation.

I'm doing this with Windows 10 x64, Node v24.19.0, effect@3.19.14 vs effect@4.0.2, Puppeteer 20.x on both, same target URL (quotes.toscrape.com/js/). Each worker is a separate node --expose-gc process.

Note that these memory numbers are Node's process only.

Effect 3 vs. Effect 4 Web Scraping Pipeline Benchmarks (Summary Table)

Benchmark Tier	Workload	Effect 3 (3.19.14)	Effect 4 (4.0.2)	Ratio	Primary Bottleneck
Tier 1: Single Scrape	1 Puppeteer page + Cheerio (median of 3)	3,381 ms (167.9 MB RSS)	3,363 ms (151.2 MB RSS)	1.01x speed (1.11x lower RSS)	Local Chromium & network I/O
Tier 2: Concurrent (N = 10)	10 concurrent pages on 1 shared browser	1.97 tasks/s (p95: 4,921 ms)	1.93 tasks/s (p95: 5,020 ms)	0.98x speed (1.03x lower RSS)	Local Chromium renderer
Tier 2: Concurrent (N = 50)	50 concurrent pages on 1 shared browser	3.33 tasks/s (185.4 MB RSS)	3.29 tasks/s (160.8 MB RSS)	0.99x speed (1.15x lower RSS)	Local Chromium networkidle2 & DOM
Tier 3: Fiber Throughput	200k concurrent yieldNow tasks (median of 9)	2,760 ms (72,473 tasks/s)	276 ms (724,585 tasks/s)	10.0x faster (1.45x lower heap)	Effect fiber scheduler
Tier 3: Fiber Memory	50k parked Effect.never fibers (after GC)	248.1 MB (5,203 B / fiber)	70.9 MB (1,487 B / fiber)	71% less memory (3.5x smaller)	Per-fiber heap footprint

Tier 1 Benchmark Results: Single-Page Puppeteer Scrape in Effect 3 vs. Effect 4

metric	Effect 3	Effect 4	ratio
one-shot wall (median of 3)	3,381 ms	3,363 ms	1.01x
peak RSS	167.9 MB	151.2 MB	1.11x lower
peak heap	50.0 MB	71.3 MB	0.70x (v4 higher)

Eighteen milliseconds! That is the real-world speedup of migrating a one-page Puppeteer scrape to Effect 4. (I can't be sure -- but the 21 MB heap jump is probably just V8 GC timing variance on a short three-second run.)

Tier 2 Benchmark Results: Why Local Puppeteer Bottlenecks High-Concurrency Scraping at 50 Pages

Fifty concurrent pages on one browser is the kind of fan-out where a faster scheduler should matter. We'll make both versions run a single browser instance with unbounded concurrency over N pages. The only difference here is that Effect 3 uses Effect.catchAll while Effect 4 uses Effect.catch.

const program = Effect.acquireUseRelease(
  launchBrowser(proxyConfig),
  (browser) =>
    Effect.forEach(
      Array.from({ length: n }, (_, i) => i),
      () => {
        const start = performance.now();
        return navigatePageAndGetContent(browser, TARGET_URL, proxyConfig, 30000).pipe(
          Effect.flatMap(parseHtml),
          Effect.tap(() =>
            Effect.sync(() => {
              latenciesMs.push(performance.now() - start);
            }),
          ),
          Effect.catch(() =>
            Effect.sync(() => {
              errorCount += 1;
              latenciesMs.push(performance.now() - start);
            }),
          ),
        );
      },
      { concurrency: "unbounded" },
    ),
  releaseBrowser,
);

In the full pipeline, proxyConfig holds the Bright Data credentials you get when you sign up and set up a proxy to use here..

N = 10	Effect 3	Effect 4	ratio
tasks/s	1.97	1.93	0.98x
p50	3,678 ms	3,787 ms	0.97x
p95	4,921 ms	5,020 ms	0.98x
peak RSS	168.1 MB	163.1 MB	1.03x lower

N = 50	Effect 3	Effect 4	ratio
tasks/s	3.33	3.29	0.99x
p50	8,680 ms	8,814 ms	0.98x
p95	14,338 ms	14,335 ms	1.00x
peak RSS	185.4 MB	160.8 MB	1.15x lower

Within 2% on every latency metric, mostly a 50/50 success on both. The only consistent v4 advantage is RSS, which is 25 MB lower at N = 50.

Why Concurrent Puppeteer Scraping Is Slow in Node.js (And When to Use Bright Data)

Concurrent Puppeteer scraping in Node.js bottlenecks at 3.29 tasks/s (50 pages, 160.8 MB RSS) because a single local Chromium process serializes networkidle2 waits and DOM parsing. Upgrading from Effect 3 to Effect 4 changes local 50-page Puppeteer throughput by less than 2% (3.33 vs. 3.29 tasks/s) because Chromium -- and not the Effect fiber scheduler -- is the bottleneck.

Why? Because fifty pages through one browser process means fifty networkidle2 waits and fifty DOM serializations, all gated on one renderer. Clearly, Effect 3's scheduler was never the thing making this slow. Swapping it for a 10x faster can't fix a bottleneck that isn't there.

To actually get to Effect 4's ballpark of 724,585 tasks/s fiber throughput in a TypeScript web scraping pipeline, we have to get local Chromium out of the hot path. Use one of these:

• Bright Data Web Unlocker (HTTP API): Offloads fingerprinting, CAPTCHA solving, residential IP rotation, and JavaScript rendering to an HTTP proxy endpoint so Effect 4 only ever runs lightweight HTTP fibers (for me, 1,487 B per fiber).
• Bright Data Scraping Browser (Remote Browser Cluster): Runs concurrent Puppeteer or Playwright sessions over WebSocket (wss://brd.superproxy.io:9222) across distributed remote browsers on Bright Data servers instead of locally.

Connecting Puppeteer to Bright Data Scraping Browser in TypeScript

If you want to scale Puppeteer concurrency past, say, 50 pages in Node.js without exhausting local CPU and RAM, replace puppeteer.launch() with puppeteer.connect() using a Bright Data Scraping Browser WebSocket endpoint (wss://brd.superproxy.io:9222):

// 1. Local Chromium: Bottlenecked at 3.29 tasks/s across 50 pages on 1 CPU
const launchLocalBrowser = Effect.tryPromise({
  try: () => puppeteer.launch({ headless: true }),
  catch: (cause) => new BrowserError({ message: "Local launch failed", cause }),
});

// 2. Remote Bright Data Scraping Browser: Scales concurrent sessions over WebSocket
const connectScrapingBrowser = (cfg: typeof BRIGHT_DATA_CONFIG) =>
  Effect.tryPromise({
    try: () =>
      puppeteer.connect({
        browserWSEndpoint: `wss://brd-customer-${cfg.customerId}-zone-${cfg.zone}:${cfg.password}@brd.superproxy.io:9222`,
      }),
    catch: (cause) =>
      new BrowserError({ message: "Scraping Browser connection failed", cause }),
  });

You might be asking, "Hey, won't firing thousands of concurrent requests with concurrency: "unbounded" mean you are immediately getting IP-banned and rate-limited by the target site?"

Excellent point! So turns out, that's the exact reason you pair Effect's retry scheduler with a rotating proxy pool instead of raw-dogging requests from a single server IP.

How to Configure Fault-Tolerant Retries and Bright Data Proxies in Effect 4

In Effect 4, Effect.acquireUseRelease prevents memory leaks and zombie Chromium processes by guaranteeing browser.close() and page.close() run on fiber interruption, timeout, or failure. Pairing Effect.acquireUseRelease with Bright Data Residential Proxies (brd.superproxy.io:33335) and page.authenticate() rotates exit IPs when Effect.retry catches a RateLimitError (HTTP 429) or IPBlockError (HTTP 403):

const BRIGHT_DATA_CONFIG = {
  customerId: process.env.BRIGHT_DATA_CUSTOMER_ID,
  zone: process.env.BRIGHT_DATA_ZONE,
  password: process.env.BRIGHT_DATA_PASSWORD,
  proxyHost: "brd.superproxy.io",
  proxyPort: 33335,
};

export const buildProxyConfig = (sessionId?: number): ProxyConfig => ({
  host: BRIGHT_DATA_CONFIG.proxyHost,
  port: BRIGHT_DATA_CONFIG.proxyPort,
  // Append -session-<id> to rotate or pin the residential exit IP per fiber/retry
  username: sessionId
    ? `brd-customer-${BRIGHT_DATA_CONFIG.customerId}-zone-${BRIGHT_DATA_CONFIG.zone}-session-${sessionId}`
    : `brd-customer-${BRIGHT_DATA_CONFIG.customerId}-zone-${BRIGHT_DATA_CONFIG.zone}`,
  password: BRIGHT_DATA_CONFIG.password!,
});

const launchBrowser = (proxyConfig: ProxyConfig) =>
  Effect.tryPromise({
    try: () =>
      puppeteer.launch({
        headless: true,
        args: [`--proxy-server=${proxyConfig.host}:${proxyConfig.port}`],
      }),
    catch: (cause) => new BrowserError({ message: "Failed to launch browser", cause }),
  });

const acquireAuthenticatedPage = (browser: Browser, proxyConfig: ProxyConfig) =>
  Effect.acquireUseRelease(
    Effect.tryPromise({
      try: async () => {
        const page = await browser.newPage();
        await page.authenticate({
          username: proxyConfig.username,
          password: proxyConfig.password,
        });
        return page;
      },
      catch: (cause) => new BrowserError({ message: "Failed to authenticate proxy page", cause }),
    }),
    (page) => Effect.tryPromise({ try: () => page.goto(TARGET_URL, { waitUntil: "networkidle2" }), catch: (cause) => new NetworkError({ message: "Navigation failed", url: TARGET_URL, cause }) }),
    (page) => Effect.promise(() => page.close()),
  );

Tier 3 Benchmark Results: Effect 4 Delivers 10x Fiber Throughput and 71% Less Memory

Now, let's take Puppeteer out entirely.

To measure throughput, we'll run 200,000 tasks, each a single yieldNow, fanned out with Effect.forEach. Then take the median of 9 processes.

// Effect 3
await Effect.runPromise(
  Effect.forEach(indices, () => Effect.yieldNow(), {
    concurrency: "unbounded",
    discard: true,
  }),
);

// Effect 4
await Effect.runPromise(
  Effect.forEach(indices, () => Effect.yieldNowWith(), {
    concurrency: "unbounded",
    discard: true,
  }),
);

Note that Effect 4's function form is yieldNowWith()

	Effect 3	Effect 4	ratio
wall for 200k tasks (median)	2,760 ms	276 ms	10.0x
tasks/s	72,473	724,585	10.0x
peak heap during run	460.5 MB	318.4 MB	1.45x lower

That's a 10x boost — even higher than Effect's 6.4x claim!

Why Effect 4 Is 10x Faster on High-Concurrency Effect.forEach Fan-Outs

How curious. Now why would this happen?

A quick look at the Effect v3 I was using before (effect@3.19.14) specifically its forEachConcurrentDiscard implementation in fiberRuntime.ts shows why a 200,000-item unbounded fan-out hits v3 so hard:

joinOrder.push(fiber)
fibers.delete(fiber)
pushResult(exit, index)
if (results.length === target) {
  resume(core.succeed(Option.getOrElse(
    core.exitCollectAll(collectExits(), { parallel: true }),
    () => core.exitVoid
  )))
}
// ...
onSuccess: () => core.forEachSequential(joinOrder, (f) => f.inheritAll)

See the issue?

Even with discard: true, Effect 3's forEachConcurrentDiscard tracks every fiber in startOrder and joinOrder arrays, pushes 200,000 { index, exit } wrapper objects into results, scans all 200,000 results at completion, and then runs core.forEachSequential(joinOrder, (f) => f.inheritAll) across all 200,000 finished fibers to merge child FiberRef state back into the parent.

In Effect 4, FiberRef was replaced by Context.Reference (This is big: so there's no child-to-parent inheritAll pass at all), and forEach with discard: true skips allocating or scanning a results array (out = undefined) — which is why a 200,000-fiber burst penalizes Effect v3 way harder than a general benchmark harness does.

Fiber Memory Benchmark: 50,000 Parked Fibers in Effect 3 vs. Effect 4

With that mystery solved, let's measure memory usage per fiber next. We'll fork into 50,000 Effect.never fibers, force GC, and measure heap delta.

Note that fork in Effect 3 is forkChild in Effect 4, while discard: true keeps the resultant array out of the measurement:

// Effect 3
yield* Effect.forEach(
  Array.from({ length: fibers }, (_, i) => i),
  () => Effect.fork(Effect.never),
  { concurrency: "unbounded", discard: true },
);
gc();

// Effect 4
yield* Effect.forEach(
  Array.from({ length: fibers }, (_, i) => i),
  () => Effect.forkChild(Effect.never),
  { concurrency: "unbounded", discard: true },
);
gc();

	Effect 3	Effect 4	ratio
heap delta for 50k fibers	248.1 MB	70.9 MB	3.5x
bytes per parked fiber	5,203 B	1,487 B	71% less

Effect claims 86% memory reduction, I got 71% for my pipeline. That's close enough I'd say. Massive improvement, still.

I'm not sure of their methodology, but I think my Effect.never + forkChild inside Effect.gen is a far heavier fiber than whatever their harness spawns, and doing all this via V8 on Windows doesn't help. As far as I know, their fiber harness isn't public, so this is as far as I can get.

Should You Migrate Your TypeScript Pipeline to Effect 4?

I think if you can migrate, do so. Aside from how the mental model changes it asked of me directly made me a better programmer, I fell in love with Effect a year ago specifically because of its DX -- and that's gotten even better!

The API cleanup they've done is awesome (catch instead of catchAll, Context.Service instead of three different tag constructors), the core package has zero runtime dependencies now, there's a three-year LTS commitment, and the agent-assisted migration took me one sitting for a 300-line pipeline even with a local model that was by no means frontier level.

The performance boost is also most welcome. You'll definitely see a massive difference if your project is a large fan-out of lightweight fibers — e.g. HTTP calls against an API, stream processing, queue consumers.

But you might not see these gains if your work is a small number of heavy tasks. The 10x and the 71% gain is in the fiber runtime a.k.a. tens of thousands of short tasks, where scheduling and the bookkeeping per fiber are a BIG share of the cost.

Some examples off the top of my head would be if you had:

• a long wait on an external system,
• a payload large enough to dwarf a few kilobytes of fiber state,
• some kind of shared limit (a pool, quota, or lock) that caps how much can run no matter how fast the scheduler is.

In that case, a faster runtime won't really help much.

---

How to Build a High-Throughput Web Data Ingestion Pipeline for AI and LLMs in TypeScript

For high-volume AI and RAG data ingestion in TypeScript, combining Effect 4 (effect@4.0.2) with Bright Data Web Unlocker supports 10,000+ concurrent fibers in ~14.9 MB of fiber heap (1,487 B per fiber), compared to >2 GB of Chromium RAM for local Puppeteer.

Scraping Architecture	Concurrency Ceiling per Node.js Worker	Node.js Memory for 10,000 In-Flight Tasks	Anti-Bot, CAPTCHA & IP Rotation Handling	Best Use Case
Local Puppeteer (No Proxy)	~10–50 pages (~3.3 tasks/s)	>2 GB (dominated by Chromium processes)	None (immediate IP blocks / HTTP 403)	Local testing & low-volume static targets
Local Puppeteer + Bright Data Residential Proxies	~20–50 pages per browser (~3.3 tasks/s)	~160 MB Node RSS + Chromium RAM	Automatic residential IP rotation & geo-targeting via brd.superproxy.io:33335	Complex multi-step browser interactions
Effect 4 + Bright Data Scraping Browser	100–1,000+ remote browsers (WebSocket-bound)	~1.5 MB fiber heap (1,487 B x 1k)	Remote Chromium cluster with built-in fingerprinting & CAPTCHA solver	High-concurrency interactive DOM scraping
Effect 4 HTTP Fibers + Bright Data Web Unlocker	10,000+ concurrent fibers (network-bound)	~14.9 MB fiber heap (1,487 B x 10k) + Cheerio buffers	Automated fingerprinting, CAPTCHA solving, JS rendering & retries at the proxy edge	High-throughput AI/RAG data ingestion & scale scraping