How to find out what your LLM filter is throwing away

Sample what it rejected, and have a person rate those rows against the standard the filter claims to apply. Precision on what a filter kept tells you almost nothing. We care because the pipeline behind Backthread decides which reasoning from an agent session reaches a team, and a filter that quietly over-rejects costs a codebase its record.

The procedure below is four hours of work. That is four hours against a failure mode which, from the side everyone measures, leaves no trace at all.

Why the keep side cannot tell you

A filter produces two populations: kept and dropped. Every standard metric a team puts on a judge — agreement with a rubric, spot checks, a pass rate holding steady week to week — is computed on the kept population. The dropped population is never looked at again, because looking at it is exactly as expensive as not having the filter.

So the failure is structurally silent. A judge that gets strict scores better on everything you are watching: the items it keeps are the obvious ones, agreement goes up, the kill rate looks stable. Meanwhile the material you built the pipeline for is in the bin. There is no number on the keep side that moves when this happens, which is why it runs for months.

The audit

Four hours, one person, no new tooling.

1. Define the population that matters, not the whole reject pile. Auditing everything the filter dropped is mostly a waste, because most of it was dropped for good reason. Audit the rows that something upstream had already passed — in ours, items the salience gate scored above threshold and the model call then culled. That subset is where a disagreement between two stages is doing real damage. Call it the cull-override set.
2. Pull a random sample, and write the number down before you look. Ninety rows was enough for us to act on. Sample randomly within the subset; do not sample the ones a colleague complained about, which is how teams accidentally audit their own anecdotes.
3. Rate them blind, against the filter's own written standard. Not against your taste. If the rubric says "an item whose answer is visible in the diff is not worth asking", that is the question you answer per row. One rater is enough to find a disaster; two are needed to make a fine-grained claim. Report which you did.
4. Compute lift, not accuracy. The number that decides anything is how the stage performs relative to whatever sits beneath it. If a threshold, score or heuristic already does the first cut, that is the baseline your judge has to beat. Accuracy without a baseline is a number you cannot act on.
5. Run the same items in isolation and in a batch. This is the step people skip, and it is the one that caught us out. Take a handful of rows a human already accepted, feed them to the judge alone, then feed them embedded in a batch of other candidates. If the verdicts differ, your harness is part of the model's input and every result you have is conditional on it.

The decision rule at the end is simple arithmetic: simulate the fix you were planning against the same sample. If a rewrite recovers a handful of rows out of ninety, you do not have a prompt problem.

What it found in ours

The stage was a model call on top of a numeric gate — close to the default architecture people reach for, which is why the result is worth publishing.

Lift bounded below 1.0 across the interval means the stage was performing at or beneath the thing it was supposed to improve on. We deleted it rather than tuning it; the full argument for that decision is here, including what the numbers do not show.

The harness detail that changed the verdict

Step five is the part worth copying even if you skip the rest.

Judging candidates in a batch and judging them one at a time produced an identical kill rate — 25 of 25 either way. By the metric we were watching, the evaluation setup did not matter at all. But judged alone, the same code false-rejected two of three items a human had already accepted.

Sit with that. The headline number was perfectly stable while the behaviour underneath it inverted. A stable number is not evidence of a stable system; it is often the thing that stops people looking. Batch composition is part of your judge's input whether you designed it that way or not, and an item that survives among nine others may not survive alone.

The corollary is about fixtures. Ours were certified against two of them, three rows and seven rows, and both passed. Three rows cannot disagree with you. If your fixture has never failed, you do not know whether it can.

What to do with the result

• Wrongly-hidden rate under about 10 percent, lift comfortably above 1 — the stage earns its place. Re-run the audit when the prompt or the model changes, because both reset it.
• High wrongly-hidden rate, lift above 1 — a calibration problem. Tune, then re-audit the cull-override set, not the keep side.
• Lift at or below 1, whatever the rate — a design problem. The stage sits in the wrong position, and a prompt cannot fix a position. Delete it, or move the model call to replace the gate rather than sit above it.
• Verdicts that move between isolation and batch — fix the harness before you believe any of the above.

One warning on scope. This is one pipeline, one domain, one period, ninety rows from our own ingest. Nothing here says your judge has no lift. It says you have probably not measured whether it does, because the measurement requires looking at the population your architecture is designed to never look at again.

The same reflex that produces an unmeasured filter stage produces unmeasured changes in a codebase: add the model call, tune it when it underperforms, never audit what it discarded. Our own deletions are on the record for that reason, and keeping that record is what Backthread does for the code agents write — including the parts where we were wrong. The audit above is the version you can run this week with a spreadsheet, and the reading order for an unfamiliar area is the same instinct applied to code rather than rows.

Connect one repo and the reasoning behind each merged change starts accumulating per area of your system, the reversals included; the trial runs fourteen days.