Why your feature works locally and does nothing in production

Because something on the path took a soft branch and could not say so. A fail-open guard, an empty read, a flag that could not be evaluated: each keeps the system up, and each can swallow a feature whole while reporting health. Agents write more of these branches than anyone reads, which is why Backthread keeps the reasoning behind each merged change next to the area it touched.

What follows is the order to work in. It is deliberately not "read the diff again", which is where most of these investigations start and stall.

What a soft branch is, and why this bug survives a deploy

A soft branch is any place the code chooses to continue rather than fail: a default value, a skipped enrichment step, a cache miss treated as absence, an allowlist that did not contain the new thing, a flag evaluation that fell back. Each one is usually a correct design choice, and most of them were added for a good reason by someone who was right.

The defect is never the branch. The defect is a branch that cannot report which one it took, because at that point two very different situations produce byte-identical telemetry: the feature is doing nothing and the feature was never deployed. Three properties keep it alive through a release:

• It reports health. No error, no exception, no failed check. The dashboards are green, which is the strongest possible argument against anyone looking.
• It is environment-shaped. Local has the row, the table grant, the flag, the seeded config. Production has one of them missing. Nothing in the code differs, so nothing in the code review finds it.
• It is often account-shaped. The branch fires only for the request whose data trips it, which means the one user who notices is the unusual one, and the engineer reproducing it is not.

What it looked like on our own deploy

Ours shipped, and production kept showing the old behaviour while local was clean and the tests were green.

The cause was two layers below the feature. A read proxy in front of the data layer was allowlisted by table, and the new table was not on the list. The read came back with nothing, the code did what it was written to do with nothing — took the soft path, served the previous behaviour — and the feature silently reverted for exactly one account: the one that browses every repository, which is a shape nobody else in the system had. Everyone else saw the feature working.

What made this a one-line diagnosis rather than a week of bisecting was a decision made earlier, by a reviewer, over mild objection. They had insisted that the soft path distinguish a real error from an absent row in what it wrote out. Without that distinction both branches say "no data" and the log is worthless. With it, the log said which, and the allowlist was the obvious next place to look.

That is the whole lesson, and it is cheaper to adopt than to re-derive: a fail-soft path that cannot name the branch it took is indistinguishable from the feature not existing.

The five steps, in this order

1. List the soft branches on the path before you look at anything else. Grep the request path for fallbacks, defaults, catch blocks that continue, ?? , allowlists, and anything that treats absence as a non-event. Write them down. This list is the suspect pool, and it is usually shorter than people expect and longer than they guessed.
2. Reproduce with production's shape, not with yours. Your account is clean, small and recently seeded. Find the account, tenant or repository with the unusual shape — the largest, the oldest, the one with every permission, the one with none — and run it there. If the bug is account-shaped, your own account is the least informative place to look.
3. Make each branch answer which one it took. Add the field, deploy, replay. If no branch on the path can answer that question, you have found the bug class rather than the bug, and the next step is instrumentation, not investigation.
4. Diff the configuration, not the code. Allowlists, table and column grants, proxy rules, flag defaults, environment variables, row-level policies, cache TTLs. The code is identical across environments by construction; the configuration is where they actually differ, and this bug lives in the difference.
5. Only now read the diff. If steps one to four came up empty, the change itself is a reasonable suspect. Not before — reading the diff first is how a config difference gets diagnosed as a logic error and then "fixed" twice.

Make each soft branch report its reason

The fix is one structured field per branch, and the vocabulary already exists. OpenTelemetry's semantic conventions define feature_flag.result.reason with the values static, targeting_match, error and default, recorded as attributes on the event emitted through the logger, plus error.type which is required when an evaluation error occurs. The convention is written for flags; the shape generalises to every soft branch you have.

Two rules for the field. It names the branch, not the outcome — reason=absent and reason=error must never collapse into one message, which is the whole point. And it goes on the hot path, at info level, not behind a debug toggle somebody will have to enable in production during an incident.

Then alert on the mix, not on errors

An error rate cannot see this class of bug, because there is no error to count. What moves is the proportion of branches taken.

• Count soft-branch hits as a metric, labelled by reason. One counter, one label. This is the smallest possible version and it is enough.
• Alert on a sharp change in a reason's share after a deploy. Going from two percent default to eighty percent default is a release note nobody wrote.
• Treat any reason that fires one hundred percent of the time in one environment as a configuration difference. A soft branch that always fires is not a fallback, it is the only path.
• Keep the per-account view. An aggregate hides the account-shaped version of this bug completely, which is how ours survived.

The honest cost: more log volume, one more field to maintain per branch, and a review habit that is tedious to enforce. Against that, this bug class reports perfect health while a feature is absent for a subset of your users, and the subset is usually the one that pays you the most attention. Our record of what we got wrong is kept on the same principle we apply to auditing what a filter discards: the population your architecture is designed to never look at again is where the expensive failures accumulate.

One more thing, specific to code agents write. These branches multiply faster than anyone can read them, because a fallback is the obvious thing to write when the requirements are underspecified, and an agent writes the obvious thing. The reasoning behind each one — which absence was expected, which was a guess — is in the session that produced it and nowhere else, which is what Backthread keeps against each area of a system as changes merge.

Connect one repo and the next merge records why each fallback is there, against the part of the system it belongs to; the trial runs fourteen days.