---
title: "The Thing You Didn't Change: Why Every Check Needs a Control"
description: "A change and a coincidence look identical from one observation. Four shapes of control that tell them apart, three times we skipped one and what it cost, and why an empty result never validates itself."
canonical_url: "https://www.truthpromoters.com/help/the-thing-you-didnt-change"
type: "help"
section: "Help Center"
keywords: "control group debugging, x-forwarded-for first or last entry, test fixture does not match real output, dual stack IPv4 IPv6 allowlist, negative check passes when unasked, assert the neighbouring case is refused, grep first match generalised to file"
---

# The Thing You Didn't Change: Why Every Check Needs a Control

We changed a setting on one machine. Within a minute, that machine could no longer be reached
from another one.

The conclusion was sitting right there, fully formed and entirely reasonable: *the change broke
it*. Roll it back, work out what went wrong, try again. Nobody would have questioned that. The
timing was perfect, the mechanism was plausible, and the machine had been fine an hour earlier.

It was wrong. The change was innocent. The fault had been there for days, and the only reason we
found that out — rather than spending an afternoon fixing something that was never broken — was a
machine we had deliberately **not** touched, which failed in exactly the same way.

This is about that: the second thing you check, the one that makes the first thing mean something.
It has a name in experimental work — a **control** — and almost nobody uses one when debugging,
because debugging feels like looking rather than measuring. It isn't. Every check you run is a
measurement, and a measurement without a control is a story.

> [!NOTE]
> Nothing here describes our own systems, which is a deliberate rule rather than an omission. All
> of it is generic: certificates, shell scripts, HTTP headers, unit tests. That constraint costs
> nothing, because the interesting part of a mistake is almost never the machine it happened on.

## Why one data point cannot tell you anything

Here is the structural problem, and it is worth having in your head before the examples.

**A change and a coincidence look identical from a single observation.** You altered something,
then something failed. That is consistent with "the change broke it" — and equally consistent with
"it was already broken and you just became the first person to look". One data point cannot
separate those, no matter how carefully you stare at it.

What makes this dangerous rather than merely limiting is that the two explanations are not equally
*available*. The one involving your change comes with a mechanism attached — you know what you
did, so you can immediately imagine how it might cause this. The other explanation has no
mechanism, because you haven't investigated it yet. So the mind reaches for the one it can
complete, and the completeness feels like evidence.

In our case, the untouched machine took about ninety seconds to check. It cost less than the first
paragraph of the explanation we were about to write.

## Four kinds of control, and what each one buys

A control isn't one technique. There are several distinct shapes, and they answer different
questions. Once you can name them you start noticing which one is missing.

### 1. The subject you didn't change

The one above. If you altered A and A is now broken, check B — as similar to A as you can find,
and untouched.

Three outcomes, all informative:

- **B is fine.** Your change is a live suspect. Now go and investigate it properly.
- **B is broken too.** Your change is almost certainly innocent, and the real fault is older and
  wider than you thought. This is the outcome that saves the most time, and you would never have
  reached it.
- **B doesn't exist.** You have no control, and you should say so out loud rather than proceeding
  as though you had one.

⭐ The version of this that matters most: when you find a fault right after making a change,
**look for it in a place your change never reached.** If it's there too, you have learned something
much more valuable than what you set out to learn.

### 2. The attack you perform yourself

We built a check that decided whether a request had come from inside the network. It rested
entirely on one assumption: that a particular piece of information attached to each request was
written by our own infrastructure and could not be set by whoever was asking.

That assumption is either true or the check is worthless. There is no middle.

So we sent the request ourselves with that value forged, from outside, and watched what arrived.
It had been overwritten — our infrastructure had replaced it, exactly as assumed. The check was
sound, and now we knew rather than hoped.

⚠️ **This is the control people skip most often, because performing the attack feels like a
detour.** But a security check is a claim that something *cannot* happen, and the only way to test
a negative claim is to try to make it happen. Reading the documentation tells you what should
occur. Doing it tells you what does.

There is a related trap here worth naming. When we first reached for a value to trust, the obvious
one was a header listing everyone who had handled the request. It is a list, and lists have two
ends. Most code takes the **first** entry, because that's nominally the original sender — which is
correct when you want to know *who the user is*, and catastrophic when you want to know *whether to
trust them*, because the first entry is the one the sender wrote. Each subsequent handler appends,
so the trustworthy end is the one nearest you. Same data, opposite ends, depending on the question.

### 3. The near-miss that must fail

We had to check whether an address fell inside a permitted range. The obvious test is: does the
address we expect get in? It did. Green.

That test is nearly worthless on its own, because it passes just as happily if the range check is
broken and lets *everything* through. A permissive bug and a correct implementation produce
identical results when you only ever try things that should succeed.

So the test that carries the weight is the **neighbour**: an address one bit outside the range,
asserted to be refused. If that passes too, the comparison is genuinely happening. If it fails,
you've found a check that was never checking.

⭐ **Whenever you write a test that something is allowed, write the one next to it that must be
refused.** The gap between them is the only thing that proves your boundary exists.

### 4. The case that must pass

The mirror image, and just as necessary.

We wrote a guard that refuses to run unless a credentials file has restrictive permissions. The
test asserts it refuses a badly-permissioned file. Good — except that a guard which refuses
*everything* passes that test perfectly, and would break the tool completely while looking
correct.

So there's a second case: a correctly-permissioned file, asserted to be **accepted**. It looks
redundant. It is the only thing standing between you and a guard that has quietly stopped
distinguishing.

## Three times we didn't have one, and what it cost

The failures are more instructive than the successes, because in each case the missing control was
obvious in hindsight and invisible at the time.

### The fixture we typed ourselves

A script needed to count entries in a block of output from a standard tool. We wrote the counter,
and wrote a test with a small sample of that output typed by hand.

The test passed. The counter was wrong — it over-counted by exactly two, every time, because the
real tool indents its section *labels* the same way it indents the *values* underneath them, and
our hand-typed sample didn't. We had tested our idea of the format, not the format.

It surfaced on real data, where it failed safely and refused to proceed. Lucky.

⭐ **A fixture you wrote tests your understanding of the input; a fixture captured from the real
thing tests the input.** When the format comes from a tool, capture the tool's actual output —
even once, even pasted in — rather than reproducing it from memory.

### The protocol we didn't test

We built an access check that permitted a known network address, tested it from the command line,
and watched it work.

Then it refused a browser, from the same building, at the same moment.

Modern networks carry two kinds of address, and browsers generally prefer the newer one. Our
command-line tool had chosen the older one. The check was correct; the list of permitted addresses
only covered one of the two ways a machine can arrive.

Worse — and this is the part that would have hurt later — half of the newer kind of address
regenerates itself periodically for privacy reasons. Listing one exactly would have worked on the
day it was written and stopped working weeks later, with no visible connection to anything anyone
had done. The fix was to permit a *range* rather than an address, which is a different design and
not a bigger one.

⚠️ **Testing with a tool is not testing with the thing that will actually connect.** Where a real
client exists, use it: a browser for something a browser will open. The tool and the client are
different clients, and they make different choices on your behalf.

### The search result we generalised

Looking for permission gaps, we searched a set of files for a particular pattern and reported what
came back. The search returned the *first* match in each file. We described the finding as though
it applied to the whole file.

It didn't. The first match was one part of each file; other parts were already correct. The report
was more alarming than the truth by a wide margin, and the correction — made before it reached
anyone — was narrower and more useful than the original.

⭐ **A search tells you where to look. It does not tell you what is there.** Before writing up a
finding, open the file. This is cheap, it is boring, and it has killed more wrong conclusions than
any other habit in this account.

## The asymmetry that makes all of this necessary

There is a reason controls matter more than they seem to, and it is not symmetric.

**A positive result largely validates itself.** If a check finds a real problem, and you fix the
problem, and the check goes quiet — the machinery demonstrably worked. You watched it work.

**A negative result validates nothing.** "No problems found" is what a working check returns when
everything is fine, *and* what a broken check returns when it never ran, never matched, never
resolved a name, or was never wired up at all. Emptiness is what success looks like, and emptiness
is also what absence looks like, and nothing in the output distinguishes them.

That is why the four shapes above are all, underneath, the same move: **manufacture a result you
know the answer to, and check you get it.** The untouched machine, the forged request, the
neighbouring address, the file that must be accepted — each one is a case where you already know
what should happen, so the answer tells you about the instrument rather than about the world.

## What to actually do

Not a methodology. Four questions, each of which takes under a minute:

1. **It broke right after I changed it — is it also broken somewhere I didn't touch?**
2. **This check assumes something cannot be faked — have I tried faking it?**
3. **I've proved the right thing is allowed — have I proved the wrong thing isn't?**
4. **My result was empty — would this check have been empty if it were broken?**

The last one is the one that catches the most, and it is the least natural to ask, because an empty
result feels like an answer. It usually is. But "usually" is doing a great deal of work in that
sentence, and the cost of finding out which case you're in is one more command.

The machine we hadn't changed took ninety seconds. It was the most useful ninety seconds of the
day, and we very nearly skipped it.

## Related topics

- [The Server Was Fine: Nine Wrong Readings, and the Accidents That Caught Them](https://www.truthpromoters.com/help/nine-wrong-readings) — Rebuilding one server produced nine broken instruments and no broken system — and every one of them failed toward alarm
- [Proving a Backup Restores, and Six Tools That Told Us the Wrong Thing](https://www.truthpromoters.com/help/proving-a-backup-restores) — We had backups of every machine and had never restored one, so we erased a server and put it back
- [Technical Writings](https://www.truthpromoters.com/help/technical-writings) — Notes on the systems behind this site — what broke, what we learned, and what misled us