Check Your Metric's Ceiling Before You Optimize Against It

A five-minute check I have never seen in anyone's evaluation pipeline, including my own.

Here is the check. Before you tune a retrieval system, hand your metric a perfect ranking — every relevant item first, the best ordering the metric is capable of expressing — and print what it says.

If it does not report 100%, stop. The problem is in the arithmetic, not the model.

I run this now on every scoring change. It takes minutes, it costs nothing, and the one time it mattered it told me something I had no other way to learn.

What it told me

I was building the retrieval layer of a knowledge base — an LLM agent that answers questions about your own notes. Three retrieval paths over three stores, and my first benchmark report read:

12.3% is an alarming number, and it triggered the standard reflex — which I can recite accurately because I was reciting it at the time. A larger embedding model. Tune chunk overlap and re-index. Add a cross-encoder reranker. That is a legitimate backlog. Every item on it is defensible. I had the tickets half-written.

Then I ran the oracle check:

Oracle Notebook Recall@5:  43.0%
Oracle Semantic Recall@5:  12.3%
Oracle Episodic Recall@5:  34.2%

Identical to what the live system scored. To the digit.

Perfect retrieval and my actual retrieval were indistinguishable to this metric. Not "close" — byte-identical. Which means every experiment on that backlog would have returned the same three numbers, and I would have read that as embeddings don't help here, and drawn a conclusion about my retrieval from an instrument that was incapable of responding to it.

What I assumed:
  current (12.3%) ──[ weeks of vector tuning ]──> ceiling (100%)

What was true:
  current (12.3%) == this metric's own ceiling

The point I want to make is not that I saved time. I can't prove that — I never ran the three weeks. The point is narrower and I think more useful: I did not know which direction my own metric could move in, and there was no way to find out except to ask it. Everything else I could have done — reading the scoring code, reasoning about the corpus, adding more queries — I had already done, and none of it had raised the question.

Why the ceiling was 12.3%

Eight lines in the scoring module.

Recall is retrieved-relevant over available-relevant. My pipeline built the candidate pool as the union of what all three paths surfaced, then had a human judge relevance across that whole union. That part is right — it is standard pooling, and it is how you build a judged retrieval set without labelling an entire corpus.

The defect was downstream. The denominator was built from the whole union, with no filter for which path could even reach a given item. The numerator was filtered to the single path being scored.

The three paths search disjoint spaces: note chunks, facts, conversation episodes. So on one query, semantic search retrieved every relevant fact available to it — a perfect result — and was then divided by forty items, most of which live in namespaces it cannot search. It was being graded on documents it is structurally incapable of returning.

There had been a signal, and I want to flag that I missed it. The paths were coupled through that shared denominator: flipping one notebook candidate from not-relevant to relevant raised the notebook score by two permille and lowered the episodic score by three — a path whose candidate list and ranking had not changed at all. A score that moves when its own subject doesn't is a ruler problem announcing itself out loud. I looked straight at it and filed it as noise. The oracle run is what made me hear it.

Two repairs that are one small edit away, and wrong

Both are what a reasonable engineer reaches for first, so both are worth naming.

Restrict the denominator to what the path itself surfaced. It is a one-line mirror of the filter already on the numerator, and it produces plausible numbers immediately. It is also self-referential: a path that surfaces exactly one candidate, judged relevant, scores 100%. Retrieving less raises the score. And it destroys pooling — an item no path surfaced belongs in no path's denominator, which is precisely the most valuable finding a judged set can produce.

Score one fused ranking across the three paths. Tempting, because a benchmark table wants one headline number. But this product implements no fusion; every candidate is surfaced by exactly one path. A fused number means inventing a ranking rule that exists nowhere in the system and then publishing its quality as the system's own.

I wrote both up as rejected options with reasons, because both are what a future reader — me, in six months — will reach for.

How hard the real repair turned out to be

I want to be exact here, because "and then I fixed it" would be the satisfying ending and it is not what happened.

The repair I chose was to divide each path by the partition of the corpus it can structurally reach. It landed exactly as specified. Its tests passed. It did not achieve its goal — a candidate's namespace turns out to be minted from the step type that surfaced it, so filtering by namespace and filtering by surfacing path select the same set on every input the pool builder can produce. Two denominators I believed were different are identical, measured 304 out of 304 candidates.

So the self-referential property I had rejected is still live, and my benchmark document says so, in words, directly above the numbers: read these as a ranking quality over what was pooled, never as recall over the corpus, and do not compare the three paths to each other. A real fix means judging every chunk, fact and episode against every query whether or not a path surfaced it — a labelling programme that scales with corpus times queries, and the judging is my own attention, so it does not parallelise. It is accepted, not scheduled.

That is a worse ending than "I fixed it," and it is the honest one. It also sharpens the original point rather than weakening it: a denominator defect can survive a correct diagnosis, a specified repair, and a green test suite. The oracle check is cheap because the thing it detects is expensive to remove.

The defect the broken metric was hiding

One more finding, because it is the reason any of this mattered past arithmetic.

The same judged set had already caught a genuine retrieval failure, unrelated to the denominator. A specific memory sits in the fixture corpus. Two queries name it in their own stated intent. It appeared in zero results for both. No path ever surfaced it.

The cause is multiplicative. Episodic relevance is similarity times an exponential recency decay, 90-day half-life. Under the evaluation profile's frozen clock that memory was 297 days old — a decay multiplier of about 0.10. Its competitors were episodes the evaluation run had written itself, stamped at the frozen instant, decay exactly 1.0. It needed roughly four times their similarity just to survive the top-k cut. It never did. The confirming symptom: that user's episodic top five was the same five records for nine of eleven queries, independent of query text. A clock ranking, not a relevance ranking.

That one generalises, and it is the finding I would carry to another system: any memory older than the decay window is unreachable when it competes against same-session writes. Product behaviour, not test artifact.

The general shape

The union-pool defect is not peculiar to my system. Any agent that retrieves across disjoint namespaces — tickets, chat logs, documents, code — and divides one path's hits by the union of every path's relevant items is measuring corpus composition, not retrieval quality. The paths will also silently move each other's scores.

But the reusable part is one step up from retrieval. A metric is a measuring instrument, and an instrument that has never been fed a known input is not yet an instrument. It is a number. The oracle run is the cheapest known-input test I have found: you already know what a perfect ranking should score, so any other answer is the instrument telling you about itself.

Feed it a perfect ranking. See what it says. Then decide what to optimize.

---

One thread from a longer set of engineering notes on building taidle — a knowledge base with an LLM agent over it, where every reply renders the route the system actually took. The full version of this story is essay 04; the decision record, including the amendment recording why the repair missed, is ADR-0011.