Building A Search Feature? Stop Relying on AI. Learn The Algorithm First.

A search feature is the easiest thing to screw up because it’s so plug-and-play these days, you barely even think about it. You’ll probably pick a BM25 library (used by Elasticsearch, Lucene, and others) and it will give you a top-10 of plausible-looking results in some order — but whether that’s actually correct depends on the little details that were decided by library authors, in defaults you probably never read, and had no way of noticing when they went wrong. Naturally, that goes double if your code was one-shot by your AI agent — which is what most do these days.

I tested this on two corpora: 9,740 Google News results over six years, quarterly, that I collected myself with Bright Data’s SERP API, and 2,284 documents from ClusTREC-COVID — a free HuggingFace dataset of COVID-19 research papers, each labeled with a topic/category.

I first saw this while searching ClusTREC-COVID for covid mrna with the rank_bm25 library. Two of my top-10 results were a chest X-ray image collection and a study on vitamin D and mortality. Neither contained mrna even once. A paper on the SARS coronavirus ORF8 protein, which did mention mrna and was exactly what the query was for, should have been at rank 9. Instead, it was at rank 412.

Same issue with the Google News corpus. Searching for ai labour, two spots in the top-10 were 3 Ways Predictive AI Delivers More Value Than Generative AI and AI Ethics And AI Law Clarifying What In Fact Is Trustworthy AI. Neither containing the word labour at all.

There were no errors, because this is not a programmatic ‘error’. If I didn’t know what a “good” result even looked like, I never would have dug deeper.

So what was my search feature even doing? What should it have done instead?

I love writing blog posts like these because they’re such great teaching moments. Let’s dive right in.

What is BM25? How Does It Work?

The BM25 algorithm is used to answer a simple question:

Given this query, how much evidence is there that this document is relevant to the user’s interests?

It gets there using three measurements:

• How often the query term appears in it (with diminishing returns for repetition.)
• The document’s length (so long-ish documents don’t win simply because they contain more words.)
• And finally, how rare the query term is across the collection. This is IDF, or inverse document frequency.

It’s based on the assumption that the presence of a rare query term is useful evidence. If only 1% of your documents contain a word, finding it in a document would be significant. On the flipside, if ~70% of your corpus contains a certain term, then seeing it in a document doesn’t tell you much — most documents in your collection contain it anyway, it’s not special.

The standard BM25 IDF formula (from Robertson-Spärck Jones) turns that into a singular value you can use:

idf(term) = log((N — n + 0.5) / (n + 0.5))

Where N is the number of documents in the collection and n is the number of documents containing the term.

So if a term appears in more than half the collection (i.e. n greater than N/2) the number inside the logarithm falls below 1, and so the IDF value becomes negative.

This single number is intuitive, even if you’re not a math major. BM25 is telling you that this term is so common that its presence is actually LESS informative than average.

And that’s where any search implementation has to make a choice. The BM25 formula CAN give you a negative IDF, but it doesn’t prescribe what a search library should do with it. There are three obvious options:

• Leave the IDF negative. Matching the common term then lowers the score.
• Clip it to zero. The common term neither helps nor hurts.
• Replace it with a positive value. Matching the common term boosts the score.

None of these is universally “correct.” It boils down to whether the choice matches what you’re trying to retrieve. The rank_bm25 library I use chooses the third behavior by default, it's a perfectly legitimate implementation strategy. What surprised me was how much it changed the results on a corpus where one topic term appears in most documents.

Before going any further, I should talk a little bit about the nature of the two datasets because they both have something in common that turns out to be quite important later on.

How do you collect publicly available data efficiently?

I used a SERP API for the Google News results, and downloaded the public dataset from HuggingFace. Here’s how to do both.

Dataset 1: Custom-built Google News Results

This is a JSON of 9,740 Google News results, collected quarterly over six years against eleven AI-related search queries. I actually used this same corpus in an earlier project to answer a different question.

These are the search terms I used for the SERP API call.

artificial intelligence  
AI technology  
machine learning  
AGI  
AI safety  
AI ethics  
generative AI  
AI regulation  
AI slop  
AI jobs  
AI layoffs

Bright Data’s SERP API returns these fields visible on the search page if you send Google a News search (tbm=nws) and ask for parsed JSON (brd_json=1):

{  
  "headline": "...",  
  "date": "Sep 24, 2025",  
  "source": "prnewswire.com",  
  "url": "https://...",  
  "query": "\"AI\" \"technology\"",  
  "quarter": "Q3 2025",  
  "description": "<truncated Google News snippet>"  
}

I chose to include the query field for each row to record provenance. I'll use that field and also the headline/description together for something more ordinary: ranking.

Fetching the data was just a Python script, something like this:

import os  
from urllib.parse import urlencode  
import requests  
  
API_KEY = os.environ["BRIGHT_DATA_API_KEY"]  
ZONE = os.environ["BRIGHT_DATA_SERP_ZONE"]  
  
def news_url(query: str, start: int) -> str:  
    return "https://www.google.com/search?" + urlencode({  
        "q": query,  
        "tbm": "nws",  
        "hl": "en",  
        "gl": "us",  
        "start": start,  
        "brd_json": 1,  
    })  
  
def fetch_page(query: str, start: int) -> dict:  
    resp = requests.post(  
        "https://api.brightdata.com/request",  
        headers={  
            "Authorization": f"Bearer {API_KEY}",  
            "Content-Type": "application/json",  
        },  
        json={  
            "zone": ZONE,  
            "url": news_url(query, start),  
            "format": "raw",  
        },  
        timeout=60,  
    )  
    resp.raise_for_status()  
    return resp.json()  
  
def collect_slice(query: str, quarter: str) -> list[dict]:  
    rows = []  
    for start in (0, 10, 20, 30, 40):  
        page = fetch_page(query, start)  
        for item in page.get("news") or []:  
            rows.append({  
                "headline": item.get("title"),  
                "date": item.get("date"),  
                "source": item.get("source"),  
                "url": item.get("link"),  
                "query": query,  
                "quarter": quarter,  
                "description": item.get("description"),  
            })  
    return rows

This gave me 9,740 distinct URLs. If you want to do the same, you'll need to sign up here and get your credentials from the dashboard. I really like this service primarily because new pay-as-you-go accounts don’t need a credit card to get started, and have a hard stop when you run out of your 5000 free credits. You can see current pricing here.

Dataset 2: ClusTREC-COVID from HuggingFace

ClusTREC-COVID is a labelled slice of TREC-COVID, which is itself drawn from CORD-19. It's 2,284 rows (JSONL) of title plus abstract, each tagged with one of fifty topics.

Each row is a record like this:

{"sentences": "Diversity of Coronaviruses in Bats: Insights Into Origin of SARS Coronavirus", "labels": "coronavirus origin", "doc_id": "6foz003n"}

You can download the file yourself here (it's about 3MB) or if you'd rather not open a browser, use this script:

import requests  
  
URL = (  
    "https://huggingface.co/datasets/Uri-ka/ClusTREC-Covid/resolve/main/"  
    "title_and_abstract/clustrec_covid_abstract_and_title.jsonl"  
)  
  
resp = requests.get(URL, timeout=60)  
resp.raise_for_status()  
  
with open("clustrec_covid_abstract_and_title.jsonl", "wb") as f:  
    f.write(resp.content)

What Do Our Datasets Have in Common?

Both were built by searching for a specific topic, which means both have the same unusual property: a term that represents the subject of the dataset appears in most of its documents, and therefore will produce a negative raw IDF.

	AI news	ClusTREC-COVID
rows	9,740 headlines + snippets	2,284 titles + abstracts
average length	33.96 tokens	186.93 tokens
saturated term	ai, in 7,090 rows (72.8%)	covid, in 1,764 rows (77.2%)
raw IDF of that term	-0.9840	-1.2208

Now let’s see what rank_bm25 does about it.

The Problem Caused By The rank_bm25 Library Defaults

If you search for covid mrna in the ClusTREC-COVID dataset using BM25 -- covid is a saturated term, occurring in 1,764 of 2,284 documents. By the formula given earlier, raw IDF for this search will yield -1.2208.

A person typing in that query wants to see papers about COVID-19 mRNA. What the default ranking with this library shows them(slots 9 and 10) are documents that never mention **mrna** at all:

Document	covid Occurrences	mrna Occurrences	Rank
COVID-19-CT-CXR chest X-ray/CT collection	18	0	9
Vitamin D and COVID-19 in UK Biobank	12	0	10

Turns out, the rank_bm25 library does the opposite. Its default replaces every negative IDF with a positive value, with no mention in the project README, and no easy way to override this behavior and restore the original IDF value. Repeating a saturated term then becomes a positive outcome, and that is what brought those two otherwise unrelated documents to the first page.

Worse — something had to be displaced to make room. In this case it was Severe Acute Respiratory Syndrome (SARS) Coronavirus ORF8 Protein…, which did contain mrna, was a genuine, relevant document, and would be at rank 9 had it not been for the default behavior of this library. Under the default, it was pushed down to rank 412.

Fig 1: The true IDF value for a term (red) turns negative once it’s in more than half the documents, like ai and covid here. But rank_bm25 won’t store that by default. Instead, it swaps in a positive number (blue) that turns matching a common term into a reward instead of a penalty.

Here’s what’s happening behind the scenes in rank_bm25 to produce that ranking (accurate as of v0.2.2):

ef _calc_idf(self, nd):  
    idf_sum = 0  
    negative_idfs = []  
  
    for word, freq in nd.items():  
        idf = math.log(self.corpus_size - freq + 0.5) \  
            - math.log(freq + 0.5)  
  
        self.idf[word] = idf  
        idf_sum += idf  
  
        if idf < 0:  
            negative_idfs.append(word)  
  
    self.average_idf = idf_sum / len(self.idf)  
  
    eps = self.epsilon * self.average_idf  
  
    for word in negative_idfs:  
        self.idf[word] = eps

This library first calculates the ordinary IDF. If it is negative, it remembers the term. It then takes the average IDF over the vocabulary and replaces the negative value with:

epsilon * average_idf

The default epsilon is 0.25 but you can pass in a different one:

BM25Okapi(tokenized, epsilon=0)

But that's the lowest you can go. Setting epsilon=0 only clips negatives to zero, it would not retain the actual raw IDF value in the negatives, which you want here.

For covid in this corpus, the values become:

raw IDF:     -1.2208  
stored IDF:  +1.5954

So this behavior has flipped what the presence of covid does to the ranking. Presence of a term which should, mathematically speaking, be less informative than average, is now scored as if it were useful evidence -- and term frequency will obviously amplify it (remember the formula).

This happens when the index is first built, not when you run a query. You will not see it in the results list. You will only see it if you inspect the stored IDF yourself like this:

import math  
  
N = bm25.corpus_size  
n = sum(1 for d in bm25.doc_freqs if "covid" in d)  
  
raw = math.log(N - n + 0.5) - math.log(n + 0.5)  
stored = bm25.idf["covid"]  
  
print(raw)      # -1.2208  
print(stored)   # +1.5954

This Will Also Happen For Sparse Searches.

This also happens when the second query term is scarce enough that there aren’t enough genuine matches to fill the first page. This will, of course, depend on how many results you’re asking for in your “Top X”. I was going for the Top 10.

Let’s search for covid ivermectin on ClusTREC-COVID. Only four documents contain the token ivermectin. The default puts all four first -- then fills ranks 5–10 with papers that never mention it:

Document	covid	ivermectin	Default rank	Rank (negative IDF → 0)	Rank (negative IDF allowed)
COVID-19-CT-CXR chest X-ray/CT collection	18	0	5	1,548	2,284
Vitamin D and COVID-19 in UK Biobank	12	0	6	1,898	2,283

The Google News corpus shows the same pattern. Searching for ai labour -- the token labour appears in only eight documents. Ranks 1–8 contain those, then ranks 9–10 do not:

Document	ai	labour	Default rank	Rank (negative IDF → 0)	Rank (negative IDF allowed)
3 Ways Predictive AI Delivers More Value Than Generative AI	6	0	9	6,477	9,740
AI Ethics And AI Law Clarifying What In Fact Is Trustworthy AI	7	0	10	3,773	9,738

A person searching ai labour did not ask for an ethics explainer or a generative-AI comparison. Clipping ai to zero sends those rows into the thousands. Leaving the IDF negative actually sends them to the bottom. Only the default behaviour would put them on page one.

Fig 2: Two documents that don’t contain the rare query term at all. The default setting still keeps them on page one (top 10). Either fix pushes them far down the list, where they belong

---

That’s why the rank_bm25 default isn't just a small numerical adjustment. It can completely change what gets shown to the user when the query contains one very common term and one very rare one.

But Is Negative IDF Always the Right Call?

No. There will always be instances where that same negative-IDF behavior is the exact opposite of what you want.

Take the same AI news corpus again, this time searching for AI layoffs.

Under negative IDF, the top results will be stories about layoffs that never mention AI at all*:

• Ranked: America’s 20 Biggest Tech Layoffs This Decade
• Layoffs: Surviving the new-age workplace!
• Apple layoffs hit more than 600 employees on its car projects

It’s possible that some of these stories will mention AI in passing in their article bodies, after all, every one of those rows was stored because Google News returned it for the search AI layoffs. But that's not what we're measuring -- and not what a reasonable user would want returned with that search.

None of them contain the token ai. For a user-facing search, I would argue that is a failed ranking.

But suppose I’m auditing an unknown or untrusted “AI news” collection and trying to find records that got swept in even though they don’t actually mention AI. Then this is exactly the list I want.

The default floor would bury those documents at ranks 197, 231, and 262. Negative IDF puts them on page one.

These are all different problems, different use-cases. Negative IDF is useful for the second one precisely because the absence of a common term is now informative. There can be no simple rule-of-thumb, you have to understand your data and the algorithm, measure both approaches, and make the call yourself.

How To Override The BM25 Library Default

Unfortunately, rank_bm25 doesn't expose a setting that keeps the raw negative IDF as an epsilon setting. If you actually want that behavior, the hacky-but-effective approach I use is to manually restore the raw values after construction like this:

import math  
from rank_bm25 import BM25Okapi  
bm25 = BM25Okapi(tokenized) # default  

# and then, reconstruct one by one  
for term in bm25.idf:  
    n = sum(1 for doc in bm25.doc_freqs if term in doc)  
    bm25.idf[term] = math.log(  
        bm25.corpus_size - n + 0.5  
    ) - math.log(  
        n + 0.5  
    )

This replaces the values in place, so using get_scores() later will use those new reconstructed values because it reads directly from bm25.idf.

You can also inspect what happened to a particular term like this:

term = "covid"  
  
n = sum(1 for doc in bm25.doc_freqs if term in doc)  
N = bm25.corpus_size  
  
raw = math.log(N - n + 0.5) - math.log(n + 0.5)  
stored = bm25.idf[term]  
  
print("raw:", raw)  
print("stored:", stored)

What Did I Learn?

Algorithmic choices are contextual. A library can’t know what your users mean by a “good” result, and neither can an AI coding agent.

The code I ran was entirely correct. rank_bm25 implements its chosen BM25 variant exactly as intended, and an AI agent could easily produce the same implementation without anything being technically wrong with the code.

The problem was that I hadn’t stopped to ask what its IDF policy meant for MY corpus and MY query.

This is the trap of not understanding the code you ship, and it’s something that worries me greatly in this AI era. They say the cost of generating code has cratered. That’s true — but I would say understanding the behavior and assumptions of the code has only become more important as a result. Correct, maintainable, testable code needs someone who understands what it’s supposed to do, and that person has to exist before you can even write the test that would have caught this.

For this use case — lexical search — you don’t need to implement BM25 from scratch yourself. You just need to know enough to check:

• Which terms appear in most of your documents? In a topic-focused collection there’s almost always at least one.
• What does your library actually store for them, versus what the formula says?
• Do the rankings under different policies (default, clipped to zero, raw negative) match what your users are trying to do? Look at ranks 5 through 10, where this will most likely show up.

An agent can’t catch that for you. It doesn’t know your corpus is built around a single topic, it doesn’t know what a good top-10 looks like for your users, and it has no reason to inspect what was stored at index time.

That’s the point of learning the core concepts, even when you’re not the one writing the code.