---
title: "Who Answered the Check? Six Tests That Passed for the Wrong Reason"
description: "A refusal that came from a content delivery network rather than the application, a rule-checking search that matched the comment explaining the rule, a control that no available machine could have failed, a tool absent from the image it shipped in, an address that differed only by a prefix, and one field answering two questions."
canonical_url: "https://www.truthpromoters.com/help/who-answered-the-check"
type: "help"
section: "Help Center"
keywords: "test passes for the wrong reason, check that cannot fail, grep matches its own comment, ipv4-mapped ipv6 comparison, standalone build missing directory, control group for instruments, connection refused versus timeout, false green"
---

# Who Answered the Check? Six Tests That Passed for the Wrong Reason

A check passed. We looked at *why* it passed, and it turned out something entirely
unrelated had answered it — not the thing we were testing, which was never consulted at all.

That happened six times in one day. Not six failures: six **successes**, each of which meant
nothing. Every one had been written deliberately, by someone thinking carefully about the failure
they wanted to catch, and every one was returning green for a reason its author would not have
recognised.

This is about that gap. Not about checks that are wrong — those are easy, because they fail and
you go and look. About checks that are *right about the wrong question*, which never fail, and so
never ask you to look at all.

> [!NOTE]
> Nothing here describes our own systems, which is a deliberate rule rather than an omission. All
> of it is generic — firewalls, HTTP headers, greps, unit tests, container images. The interesting
> part of a mistake is almost never the machine it happened on.

## The shape of the problem

When a test passes, two things could be true. Either the system does what you wanted, or **the
test never exercised the thing you wanted**. From the outside these are identical: one green tick,
no output, move on.

Failure does not have this property. A failing check demands attention and usually explains
itself. A passing check is silent, and its silence is indistinguishable from the silence of a
check that was never capable of speaking.

So the useful question is not *did it pass* but **what would have had to be true for it to fail**.
If you cannot answer that quickly, you do not yet know what the check measures — and the six
below are six different ways that answer turns out to be "nothing could have made it fail".

## 1. Someone else answered first

We wanted to know whether a service would accept a forged identity header — the kind of header a
proxy sets and an application trusts. So we sent a request with a forged one and checked it was
refused. It was refused. Green.

The refusal came from the content delivery network in front of the service, which rejects requests
that try to spoof its own headers. Our application was never reached. Its own defence — the thing
under test — was never consulted, and could have been missing entirely.

Worse, the test was **measuring a path the real traffic does not take**. The traffic we cared
about arrives on a route that deliberately bypasses that network, precisely where nothing would
strip the header. So the check was green on a path that is safe and silent about the path that is
not.

**The tell:** look at *who* sent the refusal, not just that one arrived. Status codes have
fingerprints — a vendor's error page, a header, an error code that is not in your application's
vocabulary. Ours said `error code 1000`, which is not a phrase our code has ever produced.

**The fix was not a better test.** It was removing the dependency: the application now reads only
a value written by infrastructure we control, and ignores the forgeable header entirely. A test
you cannot trust is sometimes telling you the design has a single point of failure in somebody
else's configuration file.

## 2. The documentation answered

A rule said *never pass a whole request body straight into a database update* — the classic way a
user sets a field they should not be able to set. To enforce it, we wrote a check that searched
the source for the pattern.

It reported a violation immediately, in the file most carefully written to avoid one. The match
was the **comment explaining why you must never do this**.

Then it happened again the same day, in something entirely unrelated: a check for whether a
package manager had a particular source configured searched a configuration file for the keyword.
The keyword was present — in a comment saying *"append this keyword to enable…"*. The setting was
not enabled. The check said it was.

**This is not bad luck; it is structural.** Prose about a rule uses exactly the vocabulary of a
breach of it. The better-documented the code, the more likely a rule-checking search fires on the
documentation before the mistake. The most careful file in the repository is the one most likely
to be flagged.

**Two fixes, and the second is better.** Strip comments before matching — cheap, works. Or stop
searching text and **ask the tool**: instead of grepping a configuration file, run the command
that fails when the setting is absent. Text can describe a state; only the system can be in one.

## 3. Nothing available to you could have failed

A firewall rule allowed connections from two addresses and dropped everything else. To verify it,
we connected from a machine and checked it worked; then we wanted the other half — a connection
from somewhere unauthorised, which must fail.

Every machine we had was on the allow-list. All of them shared one outbound address. **The check
could be run as many times as we liked and could only ever produce the passing half**, and the
symmetry made it look complete: connect from A — works. Connect from B — works. Two data points,
one meaning.

It took a phone on a mobile network to produce the other half. Thirty seconds, and no machine in
the estate could substitute for it.

**The tell is a population question:** does the set of things you can test from *contain* an
example of the case you want to fail? Often it does not, and the reason is systemic rather than
accidental — everything you own is configured the same way, which is usually a virtue.

⭐ **And when the failure did arrive, its exact form mattered.** The connection *timed out* rather
than being *refused*. Both look like a pass. But "refused" would have meant the machine sent a
reply saying the door is shut — announcing its existence to anyone who asks. A timeout means the
packets were dropped and the machine said nothing at all. If that check ever starts returning
"refused", something has been rebuilt in a subtly weaker way, **and it still passes**.

## 4. The tool you tested is not the tool that ships

A small administrative script worked perfectly. It had been run, it did the right thing, its
output was checked.

It could not run in production. The production image is built by a bundler that copies named
directories, and the directory holding that script is not one of them. Neither are the
dependencies it needs. So it worked everywhere it was tested and could not exist where it was
needed — and nothing would have revealed that until someone was following written instructions,
under pressure, on the day it mattered.

**This one is invisible to testing by construction**, because the tool is correct. The defect is
in the relationship between the tool and the environment, and no amount of exercising the tool
will surface it.

**The check that works is a different question:** not *does it run*, but *does it run **there***.
Read the packaging. Ask what the deployed artefact actually contains. A build that copies an
allow-list of directories silently omits everything you did not think to name — and it fails
asymmetrically in the worst way, because the parts that were present at build time keep working.

## 5. Two values that look identical, and are not

A service compared the address a request came from against a stored address. Same address, written
the same way. It failed.

The incoming one arrived as `::ffff:203.0.113.10` — an IPv4 address expressed in IPv6 notation, a
representation that appears or does not depending on how the connection was made and what sits in
front of it. A string comparison cannot see past that prefix, and the refusal message named
neither value, so the log and the configuration showed *the same address* with a rejection between
them.

**This is the cruellest kind, because the evidence actively misleads.** The obvious hypotheses —
wrong address, stale configuration, firewall — are all disproved by looking, and looking is what
convinces you the problem is elsewhere.

⚠️ **And it is environment-dependent**, which means the check passes in development and fails in
production. Not sometimes: reliably, in exactly the place where nobody is watching a test suite.

**The fix is normalisation at the boundary** — one named function, applied to both sides, that
strips the representational differences before anything is compared. The check that it is working
is a test asserting the mapped form is *accepted* and a genuinely different address in the same
form is still *rejected*; the second half matters, or "normalisation" just means "matches
everything".

## 6. One field answering two questions

A record held a timestamp meaning "when we last saw this". A comment above it said the value never
moves backwards. The code beneath the comment moved it backwards.

Neither was wrong, exactly. Two different questions were being asked of one field: *what does the
system currently report* — which must be allowed to go backwards, because a restarted service
genuinely knows an earlier value — and *when did we ever last see this* — which must not, or a
restart makes an active thing look untouched.

**The test caught the disagreement between the code and its own comment**, which is the only
reason anyone noticed there were two questions.

**The fix is not to choose.** It is to notice that one field was doing two jobs and give the second
job its own field. ⭐ **A comment that contradicts the code beneath it is not a documentation
problem.** It is usually a sign that the code is doing something its author did not intend, and the
comment records the intention.

## What actually works

Six mechanisms, one remedy, and it is not "write more tests".

**Break the check on purpose and watch it fail.** Every one of the six was settled the same way: we
deliberately introduced the fault the check claims to catch, and confirmed it went red. Where it
did not, we learned something immediately. It takes a minute and it is the only direct evidence a
check can produce about itself.

This is the same idea as a control in an experiment, applied to your instruments rather than your
subject. Some specific forms worth having:

- **For any negative assertion** — "this thing is absent", "this is refused", "nothing matches" —
  add a positive twin that must be present. An empty result is what a broken query looks like.
- **For any comparison**, test that a genuinely different value is still rejected. Otherwise a
  normalisation, a trim or a case-fold can quietly become "always equal".
- **For any check that passes**, ask what would have had to be true for it to fail — and whether
  anything in your environment could have made that true.
- **Prefer asking the system to reading about it.** A command that fails when a condition is false
  beats a search for words that describe the condition.
- **When something refuses you, ask who refused.** A vendor's error page, a proxy's status code and
  your application's own response are different evidence, and only one of them is about your code.

None of that is sophisticated. The hard part is remembering that a green tick is a claim about the
check, not about the system — and that the check is the thing nobody is checking.

## Related topics

- [Every Check Was Green: Five Faults Nobody Could See](https://www.truthpromoters.com/help/every-check-was-green) — Five faults in one small mail system over two days — and every check that existed passed throughout, because each was measuring something next to the problem
- [The Thing You Didn't Change: Why Every Check Needs a Control](https://www.truthpromoters.com/help/the-thing-you-didnt-change) — A machine broke the moment we changed it — except it had been broken for days, and the only thing that proved it was a machine we had deliberately left alone
- [The Server Was Fine: Nine Wrong Readings, and the Accidents That Caught Them](https://www.truthpromoters.com/help/nine-wrong-readings) — Rebuilding one server produced nine broken instruments and no broken system — and every one of them failed toward alarm
- [Technical Writings](https://www.truthpromoters.com/help/technical-writings) — Notes on the systems behind this site — what broke, what we learned, and what misled us