A check passed. We looked at why it passed, and it turned out something entirely unrelated had answered it — not the thing we were testing, which was never consulted at all.
That happened six times in one day. Not six failures: six successes, each of which meant nothing. Every one had been written deliberately, by someone thinking carefully about the failure they wanted to catch, and every one was returning green for a reason its author would not have recognised.
This is about that gap. Not about checks that are wrong — those are easy, because they fail and you go and look. About checks that are right about the wrong question, which never fail, and so never ask you to look at all.
Note
Nothing here describes our own systems, which is a deliberate rule rather than an omission. All of it is generic — firewalls, HTTP headers, greps, unit tests, container images. The interesting part of a mistake is almost never the machine it happened on.
The shape of the problem
When a test passes, two things could be true. Either the system does what you wanted, or the test never exercised the thing you wanted. From the outside these are identical: one green tick, no output, move on.
Failure does not have this property. A failing check demands attention and usually explains itself. A passing check is silent, and its silence is indistinguishable from the silence of a check that was never capable of speaking.
So the useful question is not did it pass but what would have had to be true for it to fail. If you cannot answer that quickly, you do not yet know what the check measures — and the six below are six different ways that answer turns out to be "nothing could have made it fail".
1. Someone else answered first
We wanted to know whether a service would accept a forged identity header — the kind of header a proxy sets and an application trusts. So we sent a request with a forged one and checked it was refused. It was refused. Green.
The refusal came from the content delivery network in front of the service, which rejects requests that try to spoof its own headers. Our application was never reached. Its own defence — the thing under test — was never consulted, and could have been missing entirely.
Worse, the test was measuring a path the real traffic does not take. The traffic we cared about arrives on a route that deliberately bypasses that network, precisely where nothing would strip the header. So the check was green on a path that is safe and silent about the path that is not.
The tell: look at who sent the refusal, not just that one arrived. Status codes have
fingerprints — a vendor's error page, a header, an error code that is not in your application's
vocabulary. Ours said error code 1000, which is not a phrase our code has ever produced.
The fix was not a better test. It was removing the dependency: the application now reads only a value written by infrastructure we control, and ignores the forgeable header entirely. A test you cannot trust is sometimes telling you the design has a single point of failure in somebody else's configuration file.
2. The documentation answered
A rule said never pass a whole request body straight into a database update — the classic way a user sets a field they should not be able to set. To enforce it, we wrote a check that searched the source for the pattern.
It reported a violation immediately, in the file most carefully written to avoid one. The match was the comment explaining why you must never do this.
Then it happened again the same day, in something entirely unrelated: a check for whether a package manager had a particular source configured searched a configuration file for the keyword. The keyword was present — in a comment saying "append this keyword to enable…". The setting was not enabled. The check said it was.
This is not bad luck; it is structural. Prose about a rule uses exactly the vocabulary of a breach of it. The better-documented the code, the more likely a rule-checking search fires on the documentation before the mistake. The most careful file in the repository is the one most likely to be flagged.
Two fixes, and the second is better. Strip comments before matching — cheap, works. Or stop searching text and ask the tool: instead of grepping a configuration file, run the command that fails when the setting is absent. Text can describe a state; only the system can be in one.
3. Nothing available to you could have failed
A firewall rule allowed connections from two addresses and dropped everything else. To verify it, we connected from a machine and checked it worked; then we wanted the other half — a connection from somewhere unauthorised, which must fail.
Every machine we had was on the allow-list. All of them shared one outbound address. The check could be run as many times as we liked and could only ever produce the passing half, and the symmetry made it look complete: connect from A — works. Connect from B — works. Two data points, one meaning.
It took a phone on a mobile network to produce the other half. Thirty seconds, and no machine in the estate could substitute for it.
The tell is a population question: does the set of things you can test from contain an example of the case you want to fail? Often it does not, and the reason is systemic rather than accidental — everything you own is configured the same way, which is usually a virtue.
⭐ And when the failure did arrive, its exact form mattered. The connection timed out rather than being refused. Both look like a pass. But "refused" would have meant the machine sent a reply saying the door is shut — announcing its existence to anyone who asks. A timeout means the packets were dropped and the machine said nothing at all. If that check ever starts returning "refused", something has been rebuilt in a subtly weaker way, and it still passes.
4. The tool you tested is not the tool that ships
A small administrative script worked perfectly. It had been run, it did the right thing, its output was checked.
It could not run in production. The production image is built by a bundler that copies named directories, and the directory holding that script is not one of them. Neither are the dependencies it needs. So it worked everywhere it was tested and could not exist where it was needed — and nothing would have revealed that until someone was following written instructions, under pressure, on the day it mattered.
This one is invisible to testing by construction, because the tool is correct. The defect is in the relationship between the tool and the environment, and no amount of exercising the tool will surface it.
The check that works is a different question: not does it run, but does it run there. Read the packaging. Ask what the deployed artefact actually contains. A build that copies an allow-list of directories silently omits everything you did not think to name — and it fails asymmetrically in the worst way, because the parts that were present at build time keep working.
5. Two values that look identical, and are not
A service compared the address a request came from against a stored address. Same address, written the same way. It failed.
The incoming one arrived as ::ffff:203.0.113.10 — an IPv4 address expressed in IPv6 notation, a
representation that appears or does not depending on how the connection was made and what sits in
front of it. A string comparison cannot see past that prefix, and the refusal message named
neither value, so the log and the configuration showed the same address with a rejection between
them.
This is the cruellest kind, because the evidence actively misleads. The obvious hypotheses — wrong address, stale configuration, firewall — are all disproved by looking, and looking is what convinces you the problem is elsewhere.
⚠️ And it is environment-dependent, which means the check passes in development and fails in production. Not sometimes: reliably, in exactly the place where nobody is watching a test suite.
The fix is normalisation at the boundary — one named function, applied to both sides, that strips the representational differences before anything is compared. The check that it is working is a test asserting the mapped form is accepted and a genuinely different address in the same form is still rejected; the second half matters, or "normalisation" just means "matches everything".
6. One field answering two questions
A record held a timestamp meaning "when we last saw this". A comment above it said the value never moves backwards. The code beneath the comment moved it backwards.
Neither was wrong, exactly. Two different questions were being asked of one field: what does the system currently report — which must be allowed to go backwards, because a restarted service genuinely knows an earlier value — and when did we ever last see this — which must not, or a restart makes an active thing look untouched.
The test caught the disagreement between the code and its own comment, which is the only reason anyone noticed there were two questions.
The fix is not to choose. It is to notice that one field was doing two jobs and give the second job its own field. ⭐ A comment that contradicts the code beneath it is not a documentation problem. It is usually a sign that the code is doing something its author did not intend, and the comment records the intention.
What actually works
Six mechanisms, one remedy, and it is not "write more tests".
Break the check on purpose and watch it fail. Every one of the six was settled the same way: we deliberately introduced the fault the check claims to catch, and confirmed it went red. Where it did not, we learned something immediately. It takes a minute and it is the only direct evidence a check can produce about itself.
This is the same idea as a control in an experiment, applied to your instruments rather than your subject. Some specific forms worth having:
- For any negative assertion — "this thing is absent", "this is refused", "nothing matches" — add a positive twin that must be present. An empty result is what a broken query looks like.
- For any comparison, test that a genuinely different value is still rejected. Otherwise a normalisation, a trim or a case-fold can quietly become "always equal".
- For any check that passes, ask what would have had to be true for it to fail — and whether anything in your environment could have made that true.
- Prefer asking the system to reading about it. A command that fails when a condition is false beats a search for words that describe the condition.
- When something refuses you, ask who refused. A vendor's error page, a proxy's status code and your application's own response are different evidence, and only one of them is about your code.
None of that is sophisticated. The hard part is remembering that a green tick is a claim about the check, not about the system — and that the check is the thing nobody is checking.