Why Google, Common Crawl, and ChatGPT See Different Versions of the Same URL

A URL identifies a resource, but it does not guarantee one canonical representation of that resource. Go visit the same URL with a browser, an HTTP client, Common Crawl / the Wayback Machine, or ask an LLM to analyze it (people increasingly do this) — and you’ll get back a different version or a different slice of the same content. We tend to treat all of these as interchangeable, but the distinction matters because if a RAG pipeline’s retrieval step comes back empty, stale, or partial, the bug might not be retrieval logic or the model, but that you simply looked in the wrong place.

Exactly how can this go wrong? I used MDN’s guide to HTTP caching as the test subject, and looked at six different observers of that same URL. For the acquisition side, I used Bright Data’s Web Unlocker and Browser APIs, mostly because they gave me a convenient way to fetch without babysitting a local Chromium install myself. Googlebot’s render came from Google’s own Rich Results Test — the public way to see the HTML their Web Rendering Service produces.

So what did I actually get? The differences weren’t as subtle as you’d think.

• HTTP response. The most complete body of the five, but also already 46 minutes old by the time it reached me, answered out of CDN. Even fetch won’t get you the page as it stands, right now.
• Rendered DOM. Same page — but surprisingly, all the code samples were gone. MDN’s client-side scripts had moved 37 of them into custom elements that a naive Playwright extract would never see.
• Googlebot (WRS). Looks like the best of both worlds, but not quite: it’s 20% bigger for the same article, and the code it kept was chopped into highlighter spans that grep could no longer match.
• Common Crawl. Captured after the July 14 edit, yet the payload was still the July 2 edition — the crawler must’ve gotten an edge-cached build. Also, 14 links were removed vs. the HTTP response, yet word overlap with the live HTTP body was 99.8%. If you were only going by that, you’d miss the missing links completely.
• Wayback Archive Snapshot. The article text was byte-identical — but with 11,913 extra bytes of injected scripts and rewritten URLs, added by the archive itself. If you were using Archive.org for your RAG, you’d blow up your context window.
• Three LLMs with tool use on. gpt-5.6-sol and sonnet-5 were mostly fine — but gemini-3.6-flash actually fetched a truncated page that was cut off a third of the way through, while confidently declaring it the full page.

None of these six observations is wrong, per se. They’re each just answering a different question. Each one can be seen as a delayed, partial, filtered, transformed, or re-encoded view of something richer, and it’s easy to query the wrong observer entirely and end up with non-obvious bugs.

The disagreement here wasn’t only about dates, either. When I diffed the four layers that actually return article content, block by block, the layer that lost the most actual content turned out to be the one most people trust the most: the real browser, via Playwright.

So before doing anything else, ask yourself: which observation do I actually need? Let’s answer that question.

Why can one URL produce different content?

When you’ve decided to fetch a URL, you’ve already made a decision you didn’t know you were making. Because you’re actually asking at least one of the following:

• Do you want the HTTP status, headers, and body returned right now?
• Do you want the DOM after a browser runs the scripts?
• Do you want the HTML Google’s renderer produces after JavaScript?
• Do you want a fixed snapshot some crawler collected weeks ago?
• Do you want a historical replay recorded by Archive.org?
• Do you want what an AI model can say about the page?

These are not routes to the same answer. They’re different questions that just so happen to share a source URL.

                             ┌── Common Crawl WARC record  
                             │  
                             ├── HTTP representation  
                             │  
Served web resource ─────────┼── Rendered browser DOM  
                             │  
                             ├── Googlebot WRS HTML  
                             │  
                             └── Web archive snapshot  
  
                             (also: search index / cached snippet,  
                              CMS export, object store — matter for SEO tasks)
                           

Each one is an independent observer that keeps its own metadata, applies its own filter, and has its own idea of what’s worth retaining.

I suppose none of them is strictly “closer to the truth” than the others — for a policy dispute you want the origin/CMS export, for an audit you want a WARC with a digest, for a UX bug you want an instrumented browser layer. The point isn’t that truth is relative; it’s that the ground truth for a given task is rarely the layer people default to — it’s not always intuitive.

You can absolutely make a fetch more live if you want to. It just won’t necessarily be more correct. It’s a different tradeoff, is all. Nothing is strictly an upgrade over another layer.

💡 Where does an LLM fit into this? A model with tools doesn’t observe the origin directly — it calls one of these acquisition layers and then reports back what it thinks it saw. That makes it a consumer of the other observation layers. An observer of another observer, if you’re feeling poetic.

How the experiment compared the observation layers

I picked MDN’s caching page specifically because it’s public, and because a page that’s already in Common Crawl can safely be assumed to show up across the acquisition layers — including whatever a model’s browsing tool would fetch.

1. The HTTP Representation.

For this, I used asimple GET request to the URL through Web Unlocker's native proxy to make sure I didn’t get HTTP 429'd out— I'm using no cookies, English preference headers, identity encoding, and cache-bypass request directives.

plain-http.ts:

const agent = new ProxyAgent({  
  uri: `http://${auth}@brd.superproxy.io:44445`,  
  requestTls: { rejectUnauthorized: false },  
  proxyTls: { rejectUnauthorized: false },  
});  
  
const resp = await proxyFetch(targetUrl, {  
  method: "GET",  
  dispatcher: agent,  
  headers: {  
    "Accept-Language": "en",  
    "Accept-Encoding": "identity",  
    "Cache-Control": "no-cache",  
    Pragma: "no-cache",  
  },  
});

This gives us the following metadata:

• Status 200, body 260,543 bytes
• Last-Modified: Mon, 10 Aug 2026 00:53:23 GMT
• Cache-Control: public, max-age=3600
• ETag: "03a7ddbde75eb66379571489a2c54b42"
• Via: 1.1 google, 1.1 varnish, 1.1 varnish, 1.1 varnish
• Age: 2751
• X-Cache: MISS, HIT, MISS
• Server: Google Frontend

If you’re using this, you’ll need your Web Unlocker API username and password formatted as that ${auth} string. Sign up here to get them. New pay-as-you-go accounts get 5,000 free credits per month, no credit card required, with a hard stop when you run out.

That Via header is already telling — this response passed through four intermediaries before it ever reached me. X-Cache: MISS, HIT, MISS says one of those intermediaries answered out of its own store instead of going back to origin, and Age: 2751 (this is in seconds) says the copy I got handed was already 46 minutes old by the time I saw it.

💡 To make sure Web Unlocker itself didn’t invent some CDN picture, I also ran a direct no-proxy fetch immediately after. So same URL, same request headers, just no Bright Data hop. I had to fight through some 429 errors as a result, but eventually I got the exact same results.

Now look at the date mismatch: Last-Modified said August 10. The editorial date printed on the page itself said July 14. These two were keeping two different clocks, and conflating them is one of the easiest mistakes you could make. One is the freshness of the deployed representation sitting behind a CDN. The other is when a human last edited the content. RFC 9110 and RFC 9111 define an HTTP response as contingent on the request that produced it — neither RFC treats that header as the document's editorial timestamp, and neither should you.

Oh, and by the way — this body carries all 39 <pre> blocks of the article's code samples. Why am I making such a big deal out of an obvious fact? As you'll see shortly, a naive browser extract of the same page loses 37 of them. The plain GET keeps every sample.

2. The Rendered DOM Itself.

For this layer, I used Playwright over CDP with JavaScript enabled, waiting for document readiness, fonts, and network idle, via the Browser API (I didn’t want to babysit my own Chromium.)

dom.ts:

const endpointURL = `wss://${auth}@brd.superproxy.io:9222`;  
const browser = await playwright.chromium.connectOverCDP(endpointURL);  
const page = await browser.newPage();  
await page.goto(targetUrl, { waitUntil: "networkidle", timeout: 120_000 });  
  
const stats = await page.evaluate(() => {  
  const main = document.querySelector("main") || document.body;  
  return {  
    title: document.title,  
    h1: document.querySelector("h1")?.textContent?.trim(),  
    editorial: document.querySelector("time")?.getAttribute("datetime"),  
    mainChars: (main?.innerText || "").length,  
    headings: document.querySelectorAll("h1,h2,h3,h4,h5,h6").length,  
    links: document.querySelectorAll("a").length,  
    resources: performance.getEntriesByType("resource").length,  
    domElements: document.querySelectorAll("*").length,  
  };  
});

As before, if you're using the Browser API, you'd need its credentials in your .env file.

That gives us:

• Title: HTTP caching - HTTP | MDN
• H1: HTTP caching
• Editorial date: 2026-07-14T00:35:10Z
• Main text: (29,185 characters worth of content here)
• Headings: 34 · Links: 575 · Resources: 76 · DOM elements: 2,402

As Chrome’s own documentation explains, the DOM is not the response body. The browser parses the HTML, runs the scripts, and JavaScript is free to change what you actually get back — that’s the entire point of running a browser instead of just fetching the raw bytes, after all.

But on this particular page, it changes something you’d never predict. For some reason, MDN’s client-side JavaScript takes every server-rendered code sample — a plain <div class="code-example"><pre><code>Cache-Control: private</code></pre></div> in the raw HTML — and swaps it for an empty custom element:

<mdn-code-example class="brush: http notranslate"></mdn-code-example>

The actual code gets moved into that element’s shadow root. This means that this content lives outside the light DOM, making it invisible to the two things a naïve (or something an AI agent one shots) Playwright or Puppeteer script looks for by default:

// this returns 37 empty custom elements, and 3,711 characters of code that are simply not here  
const html = await page.content();  
const text = await page.evaluate(() => document.querySelector("main").innerText);  
  
// the code blocks are only reachable by walking each shadow root explicitly  
const code = await page.evaluate(() =>  
  [...document.querySelectorAll("mdn-code-example")].map(  
    (el) => el.shadowRoot?.querySelector("pre")?.textContent ?? "",  
  ),  
);

Relative to the raw HTTP body, the browser removed 37 code samples from everything a normal extraction pass can see — 3,711 code characters were recovered by walking the shadow root.

This complicates the situation. The plain GET, the July crawl, and the archived replay all had every code sample because none of them run a light-DOM-only extractor against post-hydration content — they read the server-rendered HTML directly, before MDN's client-side script ever swaps it into a custom element. Googlebot does run the scripts. It still keeps the samples, because it flattens.

So when using a real browser — be it your own instance or a remote one over WSS — explicitly probe for custom elements and shadow DOM before you trust that innerText or page.content() got everything, just in case — this particular failure mode doesn't announce itself, and is real easy to miss.

3. Googlebot’s rendered HTML.

That same page, run through Google’s renderer, does not lose the samples. I pasted the MDN URL into the Rich Results Test — Google’s public tool for viewing the HTML their Web Rendering Service produces after it executes JavaScript. (Their docs say this is how you inspect rendered source for a page you do not own. URL Inspection in Search Console is the owner-only equivalent.)

The crawl succeeded on 13 August 2026. The HTML tab returned 311,939 bytes. Against the Playwright extract above:

• <mdn-code-example> elements: 37 — same count Playwright saw
• empty custom elements: 0 (Playwright: 37)
• <pre> blocks: 39 — identical to the raw HTTP body
• the three sample strings from the shadow-root probe are all in the flattened tree

Google’s JavaScript SEO documentation says this directly: “When Google renders a page, it flattens the shadow DOM and light DOM content.” Flattening is why the code samples are in the WRS HTML even though they live in a shadow root. Playwright’s default page.content() / innerText do not flatten. So on the one metric that broke the browser layer, Googlebot beats it outright.

What that flattened markup actually looks like: the empty custom element from the Playwright extract is, in Google’s HTML, a host with the <pre> serialized inside it:

<mdn-code-example class="brush: http notranslate"><!---->
  <div class="code-example">
    <div class="example-header">
      <span class="language-name"><!--?lit$639302200$-->http</span>
      <mdn-copy-button variant="secondary">…<slot>Copy</slot>…</mdn-copy-button>
    </div>
    <pre class="brush: http notranslate"><code><span class="token header"><span
      class="token header-name keyword">Cache-Control</span><span
      class="token punctuation">:</span> <span
      class="token header-value">private</span></span></code></pre>
  </div>
</mdn-code-example>

Look at that for a second before you decide it’s an upgrade. The sample survived — wrapped in a copy button, a language label, Lit’s bookkeeping comments, and four syntax-highlighting spans.

So do not read this as “Googlebot is the layer you should be using.” This is the exact trap the whole article is about, just pointed at me instead of at you. Googlebot won this round on one axis — it didn’t drop the code samples — and it is very tempting to promote it to “the good observer” on the strength of that. It isn’t a better observer. It ran the same headless Chromium against the same page and reached the same tree Playwright did. It just ships a better default serializer. The flattening it does for free is a DOM.getFlattenedDocument call, or the same shadow-root walk from the previous section, that I could have written myself in ten lines. What I actually measured wasn’t Google seeing more than a browser can. It was my extractor being wrong, with Google as the control group.

And once you look past that one axis, this layer is a worse RAG source than the plain GET it agrees with:

• 51,396 bytes of noise for zero extra content. The WRS HTML is 311,939 bytes against the raw body’s 260,543 — about 20% larger, carrying the same article. The extra is hydration debris: Lit’s comment markers (<!--?lit$639302200$-->, <!--lit-part-->, <!--lit-node 0-->), generated ids like aria-labelledby="label-x3gteze3fzc", and an injected mdn-copy-button / mdn-button subtree with a “Copy” label on every single sample. That’s 37 fake “Copy” strings your chunker will happily embed. Same failure category as the Wayback envelope — smaller, and from a different source.
• The code samples stop being greppable. In the raw body, Cache-Control: no-store is literal text. In Google’s HTML the highlighter has split it into token spans — <span class="token header-name keyword">Cache-Control</span><span class="token punctuation">:</span> <span class="token header-value">no-store</span> — so a plain substring search fails against the WRS HTML and succeeds against the raw body, even though both contain the sample. Strip tags first and they match again. Any assertion you wrote against the fetch will silently stop firing here.
• You don’t control the observation. No wait condition, no viewport, no locale, no request headers, and no Age / Last-Modified for Google’s own fetch unless you go digging in the More Info tab. Google even warns that repeat runs can differ when page resources fail to load. Compare that to the Playwright run, where networkidle and the resource count were mine to set and to record.
• It doesn’t scale, and it isn’t yours. One URL at a time, through a captcha-protected Google UI, on Google’s schedule. It is a diagnostic, not a pipeline component.
• Finally, it answers a narrower question than the tab implies. Same shape as the CDX index later: a live WRS render tells you what Googlebot can see now. It does not tell you what Google stored, and it says nothing about ranking or snippets — cache: was retired, and Search Console’s indexed-HTML view is owner-only.

So the correct use of this layer is as a reference implementation to check your own extractor against, not as the thing you ingest. Run it once, diff it against what your scraper produced, and if Google has content you don’t, the bug is in your extraction — go fix that and keep acquiring the page yourself.

4. The Common Crawl Record.

Found in CC-MAIN-2026-30 via a CDX lookup, then pulled with a byte-range WARC fetch.

common-crawl.ts:

const indexUrl =  
  `https://index.commoncrawl.org/${collection}-index` +  
  `?url=${encodeURIComponent(targetUrl)}&output=json`;  
const hits = (await (await fetch(indexUrl)).text())  
  .trim().split("\n").map((line) => JSON.parse(line));  
const hit = hits[0];  
  
const warcResp = await fetch(`https://data.commoncrawl.org/${hit.filename}`, {  
  headers: {  
    Range: `bytes=${hit.offset}-${Number(hit.offset) + Number(hit.length) - 1}`,  
  },  
});  
const record = await new WARCParser(warcResp.body).parse();

You can also use their toolkit if you’d rather do this via CLI and not code.

Regardless, I got the following metadata:

• Captured 2026-07-14T01:00:56Z
• Record ID "019f5e24-2b31-7ebd-a31f-8d9015b0619f"
• WARC digest "LFGTECNLKGCTUNOIVSVPFESH7OKODFZT"

That capture timestamp is 26 minutes after the July 14 edit, so I assumed — reasonably — that the record contained the July 14 version. It actually doesn’t. The HTML inside the WARC says the page was last modified on July 2, and the record’s own HTTP headers explain exactly why:

• Date: Tue, 14 Jul 2026 01:00:56 GMT // this was when the crawler asked for the page
• Last-Modified: Mon, 13 Jul 2026 01:13:23 GMT // the build it was given
• Age: 2842 // welp. already 47 minutes old at the edge.

But what about the data we got back? The drift between the Common Crawl record and today’s page is tiny — but can be dangerous: 99.8% of the words were identical between the two…but one list item got added to the “See also” section, the footer date line changed, and 14 links now exist that the crawl doesn’t have.

The crawler was handed a cached representation that had been built before the edit went out. You can’t catch changes like that with a plaintext diff.

Key Takeaway

If you date your corpus by capture time alone, you will confidently mis-date content, and the mistake will stay invisible until someone actually reads the payload.

Don’t rely on just the CDX index. That is only a locator — it can tell you a capture exists, full stop. The actual observation lives in the WARC file, not the index entry pointing at it.

5. The Wayback Machine Snapshot

I requested 2026-08-09T12:22:35Z and recorded the resolved timestamp, the redirect chain (if any), and a hash of the body.

wayback.ts:

const requested = https://web.archive.org/web/20260809122235/${targetUrl}; // follow redirects manually - record every hop const resp = await fetch(url, { redirect: "manual" }); // …until a non-3xx response; keep requested vs resolved stamps + body hash

A perfect result. We got a status 200, with no redirect, and Memento-Datetime: Sun, 09 Aug 2026 12:22:35 GMT — exactly the capture I asked for.

This does not actually prove the content is current or safe to ingest as-is.

Why? Because that timestamp does not say when the article text was last looked over by a real human for correctness. Just like Common Crawl, the footer date (July 14) is actually in the payload. You must open the body to see it. An acquisition tool that only checks the response never reads that far — that is an observation-layer mismatch.

So if you’re using Archive.org/The Wayback Machine’s historical replays, you actually have to track three separate pieces of info here:

• requested timestamp
• resolved timestamp
• body hash

Also, another critical thing to keep in mind — the Wayback Machine replay you get back always adds extra scripts and rewritten URLs. The article text might match the live HTTP representation character for character, but the stuff it adds as envelope is way larger. If you feed the raw replay into your RAG, those extras become tokens and link edges the model did not ask for — noise that bloats your context window.

6. The LLM layer

Finally, I sent an identical prompt to gpt-5.6-sol, sonnet-5, and gemini-3.6-flash via their APIs, asking each to open the URL and report back a structured set of facts about it: title, heading list, code sample count, see-also items, last-modified line, and a handful of verbatim quotes.

Please open and analyze this page:  
https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching  
Base your answer only on what you can retrieve now. If you cannot open it, say so.  
  
Return:  
  
A short analysis (what the page is about, who it is for).  
Then a JSON object with exactly these fields:  
{  
  "title": "",  
  "h1": "",  
  "editorial_date": "",  
  "heading_list": [],  
  "code_sample_count": null,  
  "code_samples": [],  
  "see_also_items": [],  
  "last_modified_line": "",  
  "verbatim_quotes": [],  
  "sources_used": [{"url": "", "note": ""}]  
}  
  
Rules:  
  
Prefer verbatim values from the page.  
If unknown, use null or [].  
Put 3–8 short verbatim quotes in verbatim_quotes.  
List every URL/tool result you relied on in sources_used.

This wasn’t a typical question + answer “what do you know about caching” prompt. Each model had tool access and was expected to actually retrieve the page. So the interesting question isn’t whether they knew the topic — of course they did — it’s whether what came back matched any real, identifiable layer of the five above, and whether the model told me when it hadn’t.

• gpt-5.6-sol: code_sample_count: 39 · last_modified: Jul 14, 2026 · 32/34 headings
• sonnet-5: code_sample_count: 27 · last_modified: Jul 14, 2026 · 31/34 headings
• gemini-3.6-flash: code_sample_count: 10 · last_modified: null · 10/34 headings

gpt-5.6-sol and sonnet-5 were fine for this task — a few small heading/count nits to pick, but nothing that would steer you wrong. Gemini-3.6-flash was the real failure: it stopped seeing any headings at all after "Validation" (24 of 34 missed), missed the see-also section, had no date, and only included ten code samples — basically, it only saw roughly the first third of the page, confidently reported it as the whole thing, with no warning.

Here’s why that matters. Nobody pastes a URL into ChatGPT or Claude and asks for a JSON schema. They ask something like “does this page say I need to set Cache-Control: private here?" or "summarize what changed on this page" or "is this doc still accurate?" — and they take the answer at face value, because the model sounds exactly as confident when it's right as when it's missing two-thirds of the source.

So what would happen if we did that here?

Nothing in Gemini’s response ever said it had fetched a partial page.

So the model could only give you summary/analysis of the first third — cache types, freshness, validation — with no mention that force-revalidation, cache busting, or the entire “common caching patterns” section exists.

If you’d put that answer into a wiki page, a Slack reply, or a code review comment without checking it against the page directly, you’d be shipping an incomplete answer with full confidence attached.

That’s the actual cost of this layer. It’s a different cost than a slow API or an empty search result. A failed fetch tells you it failed, but a truncated LLM read tells you it succeeded — with the same formatting, same tone, same confidence — and the only way to catch it is to already know the answer.

How to Choose The Right Observer

An observation-layer mismatch happens when a system uses a representation whose freshness, completeness, filtering, personalization, or reproducibility doesn’t match what the task actually needs.

On this exact page, the 27-day-old WARC record contained every code sample, and so did Googlebot’s flattened render. The freshly rendered browser DOM — the more expensive observation — contained none of them in a naive extract, because of the shadow-DOM extraction gap described above. You can make a fetch more “live”, but you would not be necessarily making it more “correct.”

The right move is to match the observer to the question. From experience, here are my suggested guidelines for when each layer tends to make sense.

• Use an HTTP representation if you need the current server-returned status, headers, and body.
• Use a rendered browser DOM if JavaScript, layout, or interaction actually matter for the content you want — and explicitly probe for custom elements or shadow roots before you assume innerText got everything, since a componentized page can hide content from a naive extractor even though a human sees it fine.
• Use Googlebot’s rendered HTML (Rich Results Test, or URL Inspection if you own the property) to audit your own extractor, or to ask whether Google can see JS-rendered content — not as an ingestion source, since it costs you ~20% more bytes and tokenizes code blocks. A live WRS test is not the stored index.
• Use a Common Crawl WARC if you need a stable, citable large-scale crawl record.
• Use a Wayback snapshot if the historical public state is the subject.
• Use a tools-off model observation if you specifically want the model’s unaided expectation — not current page facts.
• Use a tools-on model observation only if you also verify its specific claims against one of the acquisition layers above — because as we’ve seen above, confidently wrong looks identical to confidently right until you check, and this layer is a consumer of the others rather than an independent source.
• Use a scheduled HTTP or browser acquisition if you’re feeding an agent or RAG system that needs fresh facts

Whichever you pick, my point is that with only a few trivial checks of the observation layer you’re using, and your extract assertions, you can save yourself a TON of issues downstream. Fix these early; then you can dig into retrieval logic if you have to.