How to Clean Scraped Web HTML to Reduce LLM Token Costs by 90% in Node.js

If you're feeding raw scraped HTML directly into an LLM or a RAG embedding pipeline, 90.6% of the tokens you're paying for are complete garbage. A personal HTML-to-Markdown project led me down a js-tiktoken (o200k_base) rabbit hole across 30 real production web pages (10 e-commerce product pages, 10 developer docs pages, and 10 engineering/release news reports) -- and while the raw HTML clocked in at 66,785 tokens on average (207.5 KB), only 6,268 tokens (9.4%) were actual readable content. To see which extraction method actually strips this bloat without mangling structure, I benchmarked 8 different approaches for Node.js HTML-to-Markdown and DOM-pruning head-to-head.

For protected sites, I also paired local DOM pruning with a few friends -- Web Unlocker, residential proxies, and Scraping Browser from Bright Data -- so I wouldn't have to deal with anti-bot challenges.

Here are the 8 approaches tested:

1. Raw HTML (Baseline): Unmodified HTML passed straight to js-tiktoken.
2. Regex tag stripping: Native V8 regex (html.replace(/<[^>]+>/g, " ")).
3. cheerio: Very basic DOM selector pruning + plain-text .text() extraction.
4. turndown + turndown-plugin-gfm: Basic HTML-to-Markdown on unpruned HTML.
5. node-html-markdown: Different library, same approach. Basic HTML-to-Markdown (NodeHtmlMarkdown.translate()) on unpruned HTML.
6. jsdom + @mozilla/readability + turndown: Full DOM emulation + Readability article extraction + Turndown.
7. linkedom + @mozilla/readability + node-html-markdown: Lightweight DOM + Readability + node-html-markdown.
8. cheerio + node-html-markdown: Cheerio DOM/attribute pruning + node-html-markdown.

Why go to such lengths? Because I actually tried two "quick fixes" that an AI coding agent recommended, and...neither worked:

1. Calling $("body").text() or nuking tags with a regex DOES cut tokens by 93%–94%, but destroys your Markdown tables, fenced code blocks, and heading hierarchy — turning everything into unreadable single-line soup.

2. Dumping raw HTML straight into turndown or node-html-markdown DOES preserve tables and code blocks, but leaks ~9,500 to 10,000 tokens per page (up to 15,037 tokens on docs pages — nearly 2x token bloat!) because both converters dutifully translate header mega-menus, doc sidebars, and footer links into Markdown.

TL;DR: Neither .text() nor raw Markdown conversion works on its own. To cut LLM token costs by 90.6%–92.0% on standard pages (and up to 99.5% on 1.88 MB script-heavy storefronts) without destroying content, you have to prune boilerplate DOM subtrees before converting to Markdown. The best combo I found for the job was cheerio + node-html-markdown, which runs 5.8x faster than jsdom + @mozilla/readability + turndown while retaining almost all content. Pairing it with Bright Data Web Unlocker also stops JS/Cloudflare challenge pages from poisoning your RAG chunks -- and can even offload Markdown conversion directly to the proxy edge.

Why Does Raw Scraped HTML Waste 90%+ of LLM Tokens?

First, let's look at what inside a modern web page actually eats context window. Using js-tiktoken and cheerio, I peeled back five layers of HTML across 30 live URLs:

1. 10 E-Commerce Product Pages (Avg 42,643 tokens): Live hardware and storefront product pages from Adafruit, System76, PINE64, and ScrapingCourse.
2. 10 API Docs Pages (Avg 81,634 tokens): Live technical docs from MDN, GitHub REST API, Python 3 Docs, Redis, Kubernetes, TypeScript, and Vite.
3. 10 Tech & Release News Pages (Avg 76,078 tokens): Live engineering and release reports from Cloudflare, Node.js, Go, V8, Python, WebKit, and React.

Across all 30 pages:

HTML Layer / Token Category	E-Commerce Avg	Developer Docs Avg	Tech News Avg	30-URL Average	Share of Context Window
1. Inline <style>, <svg> Sprites & <head> Meta	11,744 tok	18,559 tok	25,513 tok	18,605 tok	27.9%
2. HTML Tag & Attribute Syntax Inside <main>	17,529 tok	19,720 tok	8,584 tok	15,278 tok	22.9%
3. <header> Mega-Menu, <nav>, <footer> & Modals	3,548 tok	31,061 tok	7,657 tok	14,089 tok	21.1%
4. <script> & Hydration JSON (__NEXT_DATA__, GTM)	6,650 tok	4,911 tok	26,533 tok	12,698 tok	19.0%
5. Core Content Signal (Clean Markdown)	3,174 tok	7,745 tok	7,886 tok	6,268 tok	9.4%
Total Raw HTML Payload	42,643 tok	81,634 tok	76,078 tok	66,785 tok	100.0%

Also, 90.6% of raw HTML tokens have zero value to an LLM, and each category wastes tokens differently:

• Inline CSS and SVG sprites (27.9%) are the #1 overall token sink. V8's Sparkplug post alone embeds 96,618 tokens of inline SVG diagrams and styles (96% of the entire document!).
• HTML wrapper tags and attributes inside <main> (22.9%) are #2. Storefronts and docs portals wrap every table cell, code token <span>, and price badge in nested <div> trees with class, style, and data-* attributes — eating 41.1% of all tokens on e-commerce pages.
• Navigation, doc sidebars, and footers (21.1%) dominate documentation sites (38.0%). Kubernetes's Namespaces doc packs 123,027 tokens of sidebar links (91.4% of the page!) around a 1,646-token article. This is the exact layer that trips up off-the-shelf Markdown converters.
• Hydration JSON and scripts (19.0%) explode on Next.js/React sites specifically: Node.js's v22 release post embeds 132,125 tokens of serialized Next.js hydration data (64.7% of the page).

---

Benchmark Methodology: Testing 8 HTML Cleaning Pipelines in Node.js

Measuring token reduction by itself is a trap — return "" gives you 100% token reduction and infinite throughput. 💀 To make sure no pipeline gamed the benchmark by deleting real content, every output across the 8 approaches above was scored against two automated checks across all 30 URLs:

• Factual Recall: Checks 120 ground-truth facts (4 per URL — SKUs, prices, API endpoints, CLI flags, and release specs).
• Structural Fidelity: Checks whether headings (#), Markdown pipe tables (| --- |), and code blocks (```) survived on pages that had them.

Why js-tiktoken (o200k_base)? It's the exact 200k BPE tokenizer still used in GPT-5, runs offline without skewing CPU/heap profiling, and shares the same BPE behavior on HTML/URLs as Claude, Llama 4, DeepSeek, and Gemini tokenizers.

---

Benchmark Results: Token Reduction, Speed, Structure & LLM Cost (30 Live URLs)

Pruning the DOM with cheerio before converting to Markdown with node-html-markdown cuts LLM token costs by 90.6% while retaining 98.3% of ground-truth facts and 100% of tables and code blocks — running 5.8x faster than jsdom + @mozilla/readability + turndown and using 245 MB less RAM.

Pipeline Combination	Avg Output Tokens	Token Reduction	Throughput (docs/s)	Median Latency	Fact Recall	Structure Score	Peak RSS	Cost / 1M Docs (@ $2.50/1M tok)
1. Raw HTML (Baseline)	66,785	0.0%	699,300.7	0.00 ms	100.0%	100.0%	420.7 MB	$166,963
2. Regex tag strip	4,609	93.1%	1,270.2	0.69 ms	97.5%	9.7%	414.2 MB	$11,523
3. cheerio (.text())	3,777	94.3%	80.2	10.36 ms	98.3%	16.7%	394.8 MB	$9,443
4. turndown + turndown-plugin-gfm (raw)	9,983	85.1%	40.0	18.90 ms	98.3%	100.0%	413.0 MB	$24,957
5. node-html-markdown (raw)	9,483	85.8%	24.0	11.80 ms	98.3%	98.7%	437.9 MB	$23,707
6. jsdom + @mozilla/readability + turndown	5,517	91.7%	5.7	144.94 ms	92.5%	92.4%	660.4 MB	$13,793
7. linkedom + @mozilla/readability + node-html-markdown	5,373	92.0%	15.2	37.00 ms	92.5%	92.4%	492.4 MB	$13,432
8. cheerio + node-html-markdown	6,268	90.6%	32.8	20.24 ms	98.3%	100.0%	415.3 MB	$15,670

Testing against 30 live URLs exposed four real-world traps.

Takeaway 1: Why cheerio .text() and Regex Stripping Are a Trap for RAG (9.7%–16.7% Structure Score)

If you only look at token count and speed, cheerio (.text()) (3,777 tokens) and Regex tag stripping (4,609 tokens) look tempting. Why bother with Markdown converters when $("main").text() gets you down to 3,777 tokens?

Because .text() and regex stripping destroyed document structure across all 30 URLs — flattening 100% of HTML <table> matrices, <pre> code blocks, and headings into a single run-on string:

What cheerio + node-html-markdown outputs (GFM Markdown — 100% Structure Score):

| HTTP Status | Error Code | Retryable | Resolution Strategy |
| --- | --- | --- | --- |
| 200 OK | NONE | No | Batch committed to WAL; returns upsertedCount |
| 400 Bad Request | ERR_DIMENSION_MISMATCH | No | Verify embedding array length matches index dimension (1536) |
| 429 Too Many Requests | ERR_RATE_LIMIT_EXCEEDED | Yes | Respect Retry-After header (default base 500ms backoff) |

What cheerio (.text()) / Regex stripping outputs:

HTTP Status Error Code Retryable Resolution Strategy 200 OK NONE No Batch committed to WAL; returns upsertedCount 400 Bad Request ERR_DIMENSION_MISMATCH No Verify embedding array length matches index dimension (1536)

Good luck getting an embedding model or a chunker to figure out whether No belongs to 200 OK or 400 Bad Request when the pipes and newlines are gone.

---

Takeaway 2: Why Unpruned turndown and node-html-markdown Nearly Double Token Costs on Docs

What happens if we hand the raw HTML string straight to turndown or node-html-markdown without pruning the DOM first?

Both libraries strip <script>, <style>, and <svg> tags out of the box. Yet across our 30 live URLs, unpruned turndown still outputs 9,983 tokens per page on average — and 15,037 tokens per page on typical docs sites.

Why? A Markdown converter's job is to translate HTML into Markdown, not to judge your site's layout. Without DOM pruning first, both libraries translate Kubernetes's 1,000-link left sidebar, GitHub's REST navigation tree, and Adafruit's header menus into hundreds of Markdown links. On docsites, skipping DOM pruning nearly doubles your LLM bill (15,037 vs 7,745 tokens) and fills half your vector DB chunks with...sidebar links.

Probably not what you want.

---

Takeaway 3: Why @mozilla/readability Deletes Markdown Tables and jsdom Is 5.8x Slower

This brings us to @mozilla/readability paired with jsdom and turndown.

This combo has two major flaws:

1. @mozilla/readability deletes product specs and pricing tables on E-Commerce sites: On our 10 E-Commerce URLs, Readability's Fact Recall dropped to 82.5% (33/40 facts) and its Structure Score dropped to 77.1%. Because Readability is tuned for prose blog posts (Firefox Reader View), it misclassified Adafruit and System76 pricing tiers and short spec tables inside <div> cards as "low-density boilerplate" and deleted them. cheerio + node-html-markdown kept 100% of E-Commerce facts and tables by explicitly removing noise selectors (nav, footer, aside, script, style) while preserving the entire <main> product layout.
2. jsdom is a 5.8x throughput and 7.2x latency bottleneck: Calling new JSDOM(html) emulates a full browser Window and Document for every node in a big, chonky page just so Readability can throw 90% of them away, then serializes the HTML back to a string for turndown to parse a second time. Across our 30 URLs, jsdom + @mozilla/readability + turndown was painful at 5.7 docs/s (144.94 ms median, 586.8 ms p95) and hit 660.4 MB Peak RSS.

How to Convert HTML to Markdown in Node.js Without Losing Tables or Code (cheerio + node-html-markdown vs linkedom)

If you're scraping modern pages that use semantic containers (<main>, <article>, [role="main"]), you don't need jsdom or @mozilla/readability.

Using cheerio to strip noise selectors and dead attributes (class, style, data-*) before passing the stripped HTML to node-html-markdown runs 5.8x faster in throughput (32.8 vs 5.7 docs/s) and 7.2x faster in median latency (20.24 ms vs 144.94 ms) while saving 245 MB of RAM.

Here is the ~30-line TypeScript implementation:

import * as cheerio from "cheerio";
import { NodeHtmlMarkdown } from "node-html-markdown";

const sharedNhm = new NodeHtmlMarkdown({
  keepDataImages: false,
  useLinkReferenceDefinitions: false,
  useInlineLinks: true,
});

const NOISE_SELECTORS = [
  "script", "style", "noscript", "svg", "iframe", "canvas", "link", "meta",
  "header.site-header", "nav", "footer", "aside",
  "[role='banner']", "[role='navigation']", "[role='contentinfo']",
  "[role='dialog']", "[data-nosnippet]", "[aria-hidden='true']",
  ".cookie-consent-modal", ".global-footer", ".global-megamenu", ".svg-icon-sheet",
].join(", ");

const ALLOWED_ATTRS = new Set(["href", "src", "alt", "title", "colspan", "rowspan"]);

export function cleanHtmlWithCheerioNhm(html: string): string {
  const $ = cheerio.load(html);
  // 1. Nuke scripts, styles, SVGs, navbars, footers, and cookie modals
  $(NOISE_SELECTORS).remove();

  // 2. Scope to semantic main/article container if present
  const root = $("main, article, [role='main']").first();
  const target = root.length ? root : $("body");

  // 3. Strip data-*, class, and style attributes before Markdown conversion (usually bulky)
  target.find("*").each((_idx, el) => {
    if (el.type !== "tag" || !el.attribs) return;
    for (const attr of Object.keys(el.attribs)) {
      if (!ALLOWED_ATTRS.has(attr)) {
        delete el.attribs[attr];
      }
    }
  });

  // 4. Convert pruned HTML fragment to GFM Markdown (tables + code blocks intact)
  return sharedNhm.translate(target.html() ?? "").trim();
}

💡 What if you're crawling legacy blogs that don't use <main> or <article> tags? Mozilla's @mozilla/readability still shines on unstructured blog layouts, trimming an extra ~900 tokens on average (5,373 vs 6,268). Just swap jsdom for linkedom + node-html-markdown, which cuts median latency 3.9x (37.00 ms vs 144.94 ms) and saves 168 MB of RAM:

import { Readability } from "@mozilla/readability";
import { parseHTML } from "linkedom";
import { NodeHtmlMarkdown } from "node-html-markdown";

const sharedNhm = new NodeHtmlMarkdown();

export function cleanHtmlWithLinkedomReadability(html: string): string {
  const { document } = parseHTML(html);
  const reader = new Readability(document as unknown as Document);
  const parsed = reader.parse();
  const contentHtml = parsed?.content ?? document.body?.innerHTML ?? html;
  const titlePrefix = parsed?.title ? `# ${parsed.title}\n\n` : "";
  return `${titlePrefix}${sharedNhm.translate(contentHtml)}`.trim();
}

---

How Does cheerio + node-html-markdown Perform Across the 3 Categories?

Here is how cheerio + node-html-markdown performed across each 10-URL vertical:

Page Category (10 Live URLs Each)	Avg Raw Tokens	Avg Cleaned Tokens	Token Reduction	Median Latency	Fact Recall	Structure Score
E-Commerce Products (10 URLs)	42,643 tok	3,174 tok	92.6% (13.4x)	12.93 ms	100.0%	100.0%
Developer API Docs (10 URLs)	81,634 tok	7,745 tok	90.5% (10.5x)	22.36 ms	95.0%	100.0%
Tech & Release News (10 URLs)	76,078 tok	7,886 tok	89.6% (9.6x)	13.20 ms	100.0%	100.0%
30-URL Corpus Average	66,785 tok	6,268 tok	90.6% (10.7x)	20.24 ms	98.3%	100.0%

On pages with heavy sidebar or CSS/SVG bloat, savings jump even higher — e.g., Kubernetes Namespaces Doc goes from 134,653 to only 1,646 tokens (-98.8%), and ScrapingCourse Hoodie goes from 50,664 to 698 tokens (-98.6%).

---

How to Scrape Anti-Bot Sites into Clean Markdown for RAG (Bright Data Web Unlocker)

Our 30-URL benchmark above used public URLs that allowed a direct fetch().

Out in the wild, if you run plain Node.js fetch() against protected e-commerce, travel, or media sites guarded by Cloudflare, AWS WAF, or DataDome, you will almost certainly get an HTTP 403 block or a sneaky HTTP 202 challenge shell.

To see what happens when we hit the anti-bot wall, here's a second benchmark against TripAdvisor, Booking.com, and IMDb, comparing normal HTTP fetch() against Bright Data Web Unlocker:

Live Protected Target	Direct fetch() Cleaned Output	Web Unlocker Raw HTML	Unlocker + jsdom + Readability + turndown	Unlocker + cheerio + node-html-markdown	Unlocker Edge data_format: "markdown" + Trimmer
TripAdvisor (The Plaza NYC)	HTTP 403 → 8 tok ("Please enable JS...", 0% recall)	1.05 MB / 612,557 tok	690.7 ms (100% recall)	78.6 ms (9,940 tok / 100% recall)	11,242 tok (-98.2%) / 0 ms DOM CPU
Booking.com (The Plaza NYC)	HTTP 202 → 0 tok (empty WAF shell, 0% recall)	1.88 MB / 731,031 tok	899.5 ms (50% recall)	144.9 ms (4,809 tok / 75% recall)	4,673 tok (-99.4%) / 0 ms DOM CPU
IMDb (The Dark Knight)	HTTP 202 → 0 tok (empty WAF shell, 0% recall)	565.5 KB / 209,239 tok	212.6 ms (50% recall)	31.2 ms (7,294 tok / 100% recall)	1,128 tok (-99.5%) / 0 ms DOM CPU

This is very useful information. It gives us 3 things to work off of:

1. Direct fetch() on most real-world sites will probably poison your RAG database: Booking.com and IMDb both returned HTTP 202 (which passes a naive res.status < 400 check!) containing an empty JS challenge shell (0 tokens, 0% recall), while TripAdvisor returned an 8-token ad-blocker error string. Without an unlocker layer, your vector DB indexes bot-block strings instead of content.

2. On 1.88 MB unlocked pages (731,031 tokens), jsdom freezes the event loop for 899.5 ms (and @mozilla/readability deletes cast/hotel specs): Just like on our 10 E-Commerce URLs, @mozilla/readability deleted Booking.com's property specs and IMDb's cast list (Christian Bale), dropping recall to 50%. On the other hand, cheerio + node-html-markdown finished 6.2x faster (144.9 ms) and kept 100% of IMDb's cast while cutting tokens by 96.5%–99.3%. Easy win.

3. Offloading Markdown conversion to the proxy edge (data_format: "markdown") drops local DOM CPU to 0 ms: Passing data_format: "markdown" to Bright Data Web Unlocker converts the unlocked DOM at the proxy edge itself — cutting Booking.com from 731,031 tokens down to 4,673 tokens (-99.4%) after a <0.5 ms link-density trim, with zero local DOM parsing.

How to Use Bright Data Web Unlocker in Your Pipeline

Once you set up a Web Unlocker zone in your Bright Data dashboard, you can either prune raw unlocked HTML locally (Option A) or offload Markdown conversion to the proxy edge (Option B — or swap in Bright Data Scraping Browser if you need multi-step browser interactions):

const pruneEdgeMarkdown = (md: string): string =>
  md
    .split(/\r?\n/)
    .filter((line) => !(/^[-*+]\s+\[[^\]]*\]\([^)]+\)\s*$/.test(line.trim()) && line.trim().length > 90))
    .join("\n");

export async function fetchWithUnlocker(
  targetUrl: string,
  edgeMarkdown = false,
): Promise<string> {
  const res = await fetch("https://api.brightdata.com/request", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.BRIGHT_DATA_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      zone: process.env.BRIGHT_DATA_UNLOCKER_ZONE, // e.g. "web_unlocker2"
      url: targetUrl,
      format: "raw",
      ...(edgeMarkdown ? { data_format: "markdown" } : {}),
    }),
  });
  const body = await res.text();
  // Option A: Local Cheerio + NHM pruning | Option B: 0ms DOM CPU edge Markdown trimmer
  return edgeMarkdown ? pruneEdgeMarkdown(body) : cleanHtmlWithCheerioNhm(body);
}

---

That's everything, thanks for reading! Here is a quick cheat sheet to pick the right library combo for your Node.js or TypeScript pipeline:

Which HTML-to-Markdown Library Should You Use for RAG in Node.js?

Pipeline Combination	Output Tokens (Avg)	Token Saved	Speed / Local CPU	Preserves Tables & Code?	Best Use Case
1. cheerio + node-html-markdown	6,268 tok	90.6% (up to 98.8%)	32.8 docs/s (20.2 ms)	Yes (100%)	Best local cleaner for production RAG & AI agents: 5.8x faster than jsdom, 98.3% fact recall (100% on e-commerce), and 245 MB lower RSS.
2. Bright Data Web Unlocker + cheerio + node-html-markdown	4,809–9,940 tok (from 209k–731k)	96.5%–99.3%	31–145 ms local	Yes (100%)	Best for anti-bot protected e-commerce & directories: Bypasses 403/202 WAF blocks and preserves cast/spec lists that Readability deletes.
3. Bright Data Web Unlocker (data_format: "markdown")	1,128–11,242 tok (from 209k–731k)	98.2%–99.5%	0 ms DOM CPU	Yes (100%)	Best for zero-DOM-overhead scale: Offloads anti-bot unblocking, JS rendering, and Markdown conversion entirely to Bright Data's proxy edge.
4. linkedom + @mozilla/readability + node-html-markdown	5,373 tok	92.0%	15.2 docs/s (37.0 ms)	Partial (82.5% ecom recall)	Best for unstructured blog/news articles: Gives you Mozilla Readability's prose scorer with 3.9x lower latency than jsdom, but avoid on e-commerce specs.
5. jsdom + @mozilla/readability + turndown	5,517 tok	91.7%	5.7 docs/s (144.9 ms)	Partial (82.5% ecom recall)	Small one-off scripts only: jsdom's 660 MB RSS and 587–899 ms stalls on large pages choke concurrent workers.
6. cheerio (.text())	3,777 tok	94.3%	80.2 docs/s (10.4 ms)	No (16.7%)	Plain-prose classification / sentiment analysis: Fast and compact, but flattens all HTML tables, headings, and multi-line code blocks.
7. turndown / node-html-markdown (unpruned)	9,483–9,983 tok	85.1%–85.8%	24.0–40.0 docs/s	Yes (99%–100%)	Pre-sanitized HTML fragments only: Never run on full web pages, or doc sidebars and header mega-menus will nearly double your LLM token bill.
8. Regex tag stripping (html.replace(/<[^>]+>/g, " "))	4,609 tok	93.1%	1,270.2 docs/s (0.7 ms)	No (9.7%)	Quick keyword grep only: Blazing fast, but leaves nav/footer text behind and destroys all tables, code blocks, and headings.

FAQ: Cleaning Scraped HTML for LLMs, RAG, and Token Cost Reduction

How do you clean scraped HTML to reduce LLM token costs in Node.js?

Prune boilerplate DOM nodes (script, style, svg, nav, footer, aside) with cheerio, scope to <main> or <article>, then convert the remaining fragment to Markdown with node-html-markdown. Across 30 live URLs, that pipeline reduced raw HTML from 66,785 tokens to 6,268 tokens (90.6%) while retaining 98.3% of ground-truth facts and 100% of tables and code blocks.

What is the best HTML-to-Markdown library for RAG pipelines in Node.js?

For semantic HTML (<main>, <article>), cheerio + node-html-markdown is the best local combo: 32.8 docs/s, 20.24 ms median latency, 415.3 MB Peak RSS, and 100% table/code retention. Use linkedom + @mozilla/readability + node-html-markdown only for unstructured blogs that lack <main>/<article> tags. Do not use jsdom + @mozilla/readability in production workers (5.7 docs/s, 660.4 MB Peak RSS, 82.5% e-commerce fact recall).

Does @mozilla/readability preserve HTML tables and product specs?

No. On 10 live e-commerce product pages, @mozilla/readability scored 82.5% fact recall and 77.1% structure score because it is tuned for prose blogs and deletes short pricing tables and SKU/spec <div> cards. cheerio + node-html-markdown kept 100% of e-commerce facts and tables.

Why does turndown or node-html-markdown still cost so many tokens on documentation sites?

Both converters translate every remaining <a> tag into Markdown, including documentation sidebars and header mega-menus. Unpruned turndown averaged 9,983 tokens per page overall and 15,037 tokens on developer docs — nearly double the 7,745 tokens produced after cheerio pruning.

Can Cloudflare, AWS WAF, or DataDome poison a RAG vector database?

Yes. Direct Node.js fetch() against TripAdvisor, Booking.com, and IMDb returned HTTP 403/HTTP 202 challenge shells that cleaned to 0–8 tokens and 0% fact recall. HTTP 202 still passes a naive res.status < 400 check, so empty bot-block strings get embedded unless you fetch through Bright Data Web Unlocker first.

How do you scrape anti-bot websites directly to Markdown for LLMs?

POST the URL to Bright Data Web Unlocker with data_format: "markdown". That converts the unlocked DOM at the proxy edge with 0 ms local DOM CPU, cutting Booking.com from 731,031 tokens to 4,673 tokens (-99.4%) after a sub-millisecond link-density trim.