A Study of the Public URLs Cited by the IANA Time Zone Database

Fig 1: What tzdata is for. The map shows the time actually observed in each place, legality aside. Its zone data comes from the IANA database. Source: Heitordp, Wikimedia Commons. Public domain (CIA World Factbook).

The IANA Time Zone Database contains the rules that determine how local time changes in different parts of the world, but they also contain something less obvious: comments explaining where those rules came from. You can find it here: https://github.com/eggert/tz

I became interested in the sources themselves. In particular, I wanted to know how many of the citations that used a public URL were still accessible right now. So I tried to find every one, and recover them if necessary — first using the Web Unlocker to get past common anti-bot blocks, and then the Wayback API (Archive.org) for truly dead links.

I’ve put my code up on GitHub, including the census data I found, and the retrieval + recovery pipeline used for the 1,327 URLs. Feel free to check it out.

GitHub - sixthextinction/tzdata-citation-census: Census of 1,327 URLs cited by the IANA Time Zone Database, with direct retrieval, Wayback recovery, and Web Unlocker

I went down this rabbit hole when I encountered a 2008 comment by a Canadian volunteer named Patrice Scattolin. He was trying to determine exactly when a Moroccan DST decree took effect. He was working from a news article and had to resolve an ambiguity in the French-to-English translation, and documented the reasoning in a code comment:

The decision was taken by decree 2–08–224 but I can’t find the decree published on the web.

This is a fairly ordinary kind of note for a historical dataset. There are many such comments from maintainers — from other databases in other domains, in fact — each contain decree numbers, government publications, newspaper articles, URLs, dates, and notes from individual maintainers explaining why they preferred one source over another.

But it raises a bigger question for me: how many public URLs cited in tzdata comments (and other such historical datasets) are still accessible, particularly when so many of them date back many years?

What Is tzdata, Actually?

The IANA Time Zone Database is a collection of text files maintained by a group of contributors. Operating systems and software libraries use it to determine the local time corresponding to a particular time zone.

These files are not something most people encounter directly or are even aware of, but they use them, regardless. Linux distributions ship them, as does pretty much every operating system and programming environment. When a country changes its clocks (because of Daylight Savings or whatever else) this database needs to be updated so that all software continues to produce the correct local time.

Press enter or click to view image in full size

Fig 2: Where daylight saving time is observed. These are the clock changes tzdata has to encode. Source: Heitordp, Wikimedia Commons. Public domain (CC0).

They even have a short how-to for people who want to send in a correction.

The actual “rules” can be quite simple. For example, the 2008 Moroccan Daylight Savings change I mentioned earlier is just two lines of plaintext in the africa document:

# Rule NAME FROM TO - IN ON AT SAVE LETTER/S  
Rule Morocco 2008 only - Jun 1 0:00 1:00 -  
Rule Morocco 2008 only - Sep 1 0:00 0 -

These rules specify the clock changes on June 1 and September 1 that every machine in the world syncs to.

Press enter or click to view image in full size

Fig 3: Time zones in Africa, including Morocco. Darker colours mark where daylight saving time is used. Source: Cyfraw, Wikimedia Commons. CC BY-SA 4.0.

It’s the comments surrounding the rules are more informative if you are interested in how the database is constructed. Maintainers often record the evidence they used: a decree number, a government gazette, a ministry press release, a newspaper article, or the name of a person who supplied the information. They also generally record when the information was added.

To my surprise, I found there were even cases where the sources themselves disagree! The maintainers sometimes record that disagreement and explain which source they decided to use. For example, Paul Eggert in 2006, choosing the Austrian Federal Office of Metrology and Surveying over an atlas:

# From Paul Eggert (2006-03-22): Shanks & Pottenger give 1918-06-16 and  
# 1945-11-18, but the Austrian Federal Office of Metrology and  
# Surveying (BEV) gives 1918-09-16 and for Vienna gives the "alleged"  
# date of 1945-04-12 with no time. For the 1980-04-06 transition  
# Shanks & Pottenger give 02:00, the BEV 00:00. Go with the BEV,  
# and guess 02:00 for 1945-04-12.

I don’t know whether this approach to sourcing was an explicit design decision when the database was established, or whether it developed gradually as different people contributed to it.

Either way, it provides a useful record of how individual decisions were made. A shared database that has been maintained for more than thirty years obviously needs some way for contributors to explain where information came from, particularly when the original evidence is uncertain or conflicting.

So What Happens When You Actually Go Check the Citations?

I extracted every URL from the comment blocks in the nine tzdata source files. This produced 1,609 comment blocks, 1,352 URL occurrences, and 1,327 unique URLs across 623 different hosts. Many comment blocks cite a book, a decree, or a mail message with no URL, and those were out of scope for this project.

I then made a GET request to each URL using a normal browser user-agent and a timeout.

The first pass successfully retrieved 663 URLs.

The remaining 664 URLs could not be retrieved directly, but they did not all fail because of the same reason. I separated these into several categories because a page returning 404 is different from a page that is still online but refuses an automated request.

Of those URLs:

• 433 could not be retrieved directly, but worked using a proxy. 178 had refused the direct request (mostly HTTP 403s, some 429s). The other 255 were unreachable from a home connection — DNS failure, timeout, or TLS error — not an anti-bot block. I retried both groups through Bright Data’s Web Unlocker, and they were recovered this way.
• 123 were actually gone. These returned HTTP 404 or 410, so the source itself was indicating that the resource was no longer available at that URL. I checked these against the Internet Archive’s Wayback Machine. Thankfully, all but two could be recovered, with 121 producing a full saved copy of the page.
• 108 remained unresolved (57 had no archive copy, 39 still fail even using Web Unlocker, and 12 return HTTP 500). There was no accessible live copy, no usable archive copy from the pass I ran, or the proxy still failed. Eighteen of these, all on Mexico’s dof.gob.mx gazette, gave me a certificate error instead of a refusal – likely a misconfiguration on the Mexican government’s end rather than a missing page, but I had no means to chase that up with them.

A site that has disappeared is a preservation problem, but a site that is still online but refuses an automated request is an access problem. They both prevent a simple GET from succeeding, but they tell us different things about whether the underlying source still exists.

Why Would a Citation Be Blocked in the First Place?

Most websites now treat automated requests differently from requests made by a normal browser. Depending on the site, a request that does not execute JavaScript or otherwise behave like a browser may be rate-limited, rejected, or presented with a challenge page.

That creates a problem when trying to bulk-check old citations via code, like this. A URL that worked normally when a tzdata maintainer added it may still be live years later, but a simple automated request to the same URL may no longer be accepted.

It becomes even more important when you realize government gazettes and legal databases frequently show up in the data — dre.pt, the Portuguese official gazette; resmigazete.gov.tr, Turkey’s official gazette; and nevo.co.il, an Israeli legal database, and more. These are important sources in the corpus because maintainers used them repeatedly, and even then some of their pages could not be retrieved directly.

This means that an unsuccessful request does not by itself tell us that a citation has disappeared. It may indicate that the resource has been removed, but it may also indicate that the resource is still available and the particular method used to retrieve it is no longer accepted.

For this reason, I treated the two cases separately in the recovery process.

Is This Actually a tzdata Problem?

No. My experiment is more directly a test of how well the external sources recorded by a long-lived historical dataset such as this one have survived.

Here’s the limitation. What this historical dataset cannot do is preserve the external material it cites. The rule itself is stored in a version-controlled repository and is copied into software distributions around the world, sure, but the source material it cites remains on external websites, government systems, newspaper sites, and other services that are maintained independently of tzdata.

There is at least one example in the corpus where this problem was already known at the time the citation was written. A maintainer sourcing a 2014 Egyptian daylight-saving decision noted that the Egyptian cabinet’s own page describing the decision “[was] not accessible outside Egypt.”

In other words, access problems are not necessarily a recent discovery. The maintainers themselves sometimes encountered them while doing the original research.

What Did the Recovery Pipeline Actually Look Like?

I kept the recovery process in separate passes because I wanted to distinguish between different kinds of failure.

One possible approach would have been to try every method against every URL: make a direct request, then try a proxy, then check the archive, and use whichever method happened to produce an answer. That would give a larger recovery number, but it would make the resulting categories less meaningful.

A page that returns 404 is different from a page that is temporarily inaccessible. Likewise, an archived copy is not necessarily identical to the version of the page that a maintainer originally consulted. It is a copy captured at some particular point in time.

So my first pass used only direct requests and recorded the exact HTTP status. For URLs returning 404 or 410, I then made a separate, rate-limited pass against the Internet Archive’s availability API. I did not send the timed-out or blocked URLs to the archive at this stage.

So something like this:

async function checkCitation(url) {  
  const direct = await get(url); // one plain GET, no tricks  
  if (isOk(direct)) return { status: "live", via: "direct" };  
  
  if (direct.status === 404 || direct.status === 410) {  
    const archived = await wayback(url); // only for confirmed-dead  
    if (archived.hit) return { status: "recovered", via: "wayback" };  
  }  
  
  return { status: "unresolved" };  
}

The proxy pass was separate again. It was applied to URLs that the direct pass could not retrieve because they were blocked or otherwise unreachable, rather than to URLs that had explicitly returned 404 or 410.

For those blocked URLs I retried the same GET through Bright Data’s Web Unlocker, as a native HTTPS proxy.

import { ProxyAgent, fetch as proxyFetch } from "undici";  
  
const AUTH = process.env.BRIGHT_DATA_UNLOCKER_AUTH; // format like USER:PASS  
  
const dispatcher = new ProxyAgent({  
  uri: `@brd.superproxy.io:44445`">http://${AUTH}@brd.superproxy.io:44445`,  
  requestTls: { rejectUnauthorized: false },  
  proxyTls: { rejectUnauthorized: false },  
});  
  
const res = await proxyFetch(url, {  
  dispatcher,  
  headers: { "User-Agent": UA, Accept: "*/*" },  
  redirect: "follow",  
});

Note that I’m not using the browser-rendering version of this service. The request was still an ordinary HTTP fetch; it was just being sent through the Web Unlocker proxy.

If you want to do the same, you need your Unlocker username and password formatted as that ${AUTH} string. Sign up here to get them. I really like this service primarily because new pay-as-you-go accounts don’t need a credit card to get started, and have a hard stop when you run out of free credits (you get 5000 to start you off).

Put your credentials in .env like so:

BRIGHT_DATA_UNLOCKER_AUTH=brd-customer-XXXXX-zone-web_unlocker:PASSWORD  
BRIGHT_DATA_UNLOCKER_PROXY_HOST=brd.superproxy.io  
BRIGHT_DATA_UNLOCKER_PROXY_PORT=44445

Keeping these passes separate allowed the final results to distinguish between direct retrieval, recovery through a proxy, recovery from an archive, and citations that remained unresolved.

What Could You Do With the Sources Once You Have Them?

Actually, there is another step I did not try here. The question in this study was whether a cited URL could still be retrieved, and, if not, whether the document could be recovered elsewhere.

But for a historical dataset, one could instead preserve some of the information in the recovered document itself. For a government gazette or legal database, for example, this might mean extracting the decree number, date, title, and relevant text and storing that alongside the original citation.

Why do this? Well, the advantage would be that the evidence is then preserved independently of the URL — it wouldn’t need a pass to grab the URLs first.

This can also be automated for sources that occur repeatedly in a dataset. I already use the Bright Data Web Unlocker. The next step could be using their Scraper Studio as a way to define the fields to collect and run the collector on the same unblocking infrastructure used here. A Code Worker is suitable for ordinary HTTP requests, while a Browser Worker can handle pages that require JavaScript.

So What’s Actually Left When the Archive Doesn’t Have It?

There are 108 URLs as citations that I could not resolve.

That is 108 out of 1,327 unique URLs, or a little over 8 percent. The percentage is relatively small, but each citation represents a source that was used at some point to establish a particular time-zone rule.

The rules themselves are still present in tzdata. What is missing in these cases is some or all of the external page at that URL.

Of the 108, 57 returned 404 or 410 and the Wayback Availability API had no record of them. 39 still failed through the proxy (18 of those are the Mexico certificate errors). The remaining 12 produced other HTTP errors that I did not retry and did not send to Wayback.

I could have spent more time trying to resolve those final cases, but I decided that the census needed a defined stopping point. The objective was to measure the current state of the citations using a consistent procedure, rather than continue indefinitely with increasingly specialized recovery techniques for a small remainder.

The Limits of Citation as Preservation

There’s an interesting asymmetry here — these historical datasets and the sources they cite have very different preservation properties.

A time-zone rule is stored in a version-controlled file and distributed through many software systems. Once the rule has been incorporated into the database, it can continue to be available even if the source that originally supported it disappears.

But the cited material has no equivalent guarantee. It may exist on one website, under one URL, and depend on an organization continuing to operate that site. If the organization moves the material, removes the page, changes its access policy, or stops maintaining the service, the citation may become difficult or impossible to retrieve.

This kind of failure has a name — we call it link rot.

The tzdata maintainers have been recording their sources for decades. In many cases they also recorded uncertainty about those sources. That makes the citations considerably more useful for later investigation than an unsupported assertion would have been.

But careful citation does not by itself preserve the cited material. The Tz database maintainers do not have a responsibility to save and archive everything they cite.

The results of this project are therefore less a criticism of tzdata than a demonstration of the limits of relying on external URLs as historical references. Most of the citations were still recoverable in some form…but a smaller yet significant number were not.

It was a very interesting project to work on — I learnt a ton about the IANA and historical datasets, in general. The TZ database continues to contain the information that those sources were used to establish. In 108 cases, however, this does not recover the URL.