The Thing You Didn't Change: Why Every Check Needs a Control

We changed a setting on one machine. Within a minute, that machine could no longer be reached from another one.

The conclusion was sitting right there, fully formed and entirely reasonable: the change broke it. Roll it back, work out what went wrong, try again. Nobody would have questioned that. The timing was perfect, the mechanism was plausible, and the machine had been fine an hour earlier.

It was wrong. The change was innocent. The fault had been there for days, and the only reason we found that out — rather than spending an afternoon fixing something that was never broken — was a machine we had deliberately not touched, which failed in exactly the same way.

This is about that: the second thing you check, the one that makes the first thing mean something. It has a name in experimental work — a control — and almost nobody uses one when debugging, because debugging feels like looking rather than measuring. It isn't. Every check you run is a measurement, and a measurement without a control is a story.

Why one data point cannot tell you anything

Here is the structural problem, and it is worth having in your head before the examples.

A change and a coincidence look identical from a single observation. You altered something, then something failed. That is consistent with "the change broke it" — and equally consistent with "it was already broken and you just became the first person to look". One data point cannot separate those, no matter how carefully you stare at it.

What makes this dangerous rather than merely limiting is that the two explanations are not equally available. The one involving your change comes with a mechanism attached — you know what you did, so you can immediately imagine how it might cause this. The other explanation has no mechanism, because you haven't investigated it yet. So the mind reaches for the one it can complete, and the completeness feels like evidence.

In our case, the untouched machine took about ninety seconds to check. It cost less than the first paragraph of the explanation we were about to write.

Four kinds of control, and what each one buys

A control isn't one technique. There are several distinct shapes, and they answer different questions. Once you can name them you start noticing which one is missing.

1. The subject you didn't change

The one above. If you altered A and A is now broken, check B — as similar to A as you can find, and untouched.

Three outcomes, all informative:

• B is fine. Your change is a live suspect. Now go and investigate it properly.
• B is broken too. Your change is almost certainly innocent, and the real fault is older and wider than you thought. This is the outcome that saves the most time, and you would never have reached it.
• B doesn't exist. You have no control, and you should say so out loud rather than proceeding as though you had one.

⭐ The version of this that matters most: when you find a fault right after making a change, look for it in a place your change never reached. If it's there too, you have learned something much more valuable than what you set out to learn.

2. The attack you perform yourself

We built a check that decided whether a request had come from inside the network. It rested entirely on one assumption: that a particular piece of information attached to each request was written by our own infrastructure and could not be set by whoever was asking.

That assumption is either true or the check is worthless. There is no middle.

So we sent the request ourselves with that value forged, from outside, and watched what arrived. It had been overwritten — our infrastructure had replaced it, exactly as assumed. The check was sound, and now we knew rather than hoped.

⚠️ This is the control people skip most often, because performing the attack feels like a detour. But a security check is a claim that something cannot happen, and the only way to test a negative claim is to try to make it happen. Reading the documentation tells you what should occur. Doing it tells you what does.

There is a related trap here worth naming. When we first reached for a value to trust, the obvious one was a header listing everyone who had handled the request. It is a list, and lists have two ends. Most code takes the first entry, because that's nominally the original sender — which is correct when you want to know who the user is, and catastrophic when you want to know whether to trust them, because the first entry is the one the sender wrote. Each subsequent handler appends, so the trustworthy end is the one nearest you. Same data, opposite ends, depending on the question.

3. The near-miss that must fail

We had to check whether an address fell inside a permitted range. The obvious test is: does the address we expect get in? It did. Green.

That test is nearly worthless on its own, because it passes just as happily if the range check is broken and lets everything through. A permissive bug and a correct implementation produce identical results when you only ever try things that should succeed.

So the test that carries the weight is the neighbour: an address one bit outside the range, asserted to be refused. If that passes too, the comparison is genuinely happening. If it fails, you've found a check that was never checking.

⭐ Whenever you write a test that something is allowed, write the one next to it that must be refused. The gap between them is the only thing that proves your boundary exists.

4. The case that must pass

The mirror image, and just as necessary.

We wrote a guard that refuses to run unless a credentials file has restrictive permissions. The test asserts it refuses a badly-permissioned file. Good — except that a guard which refuses everything passes that test perfectly, and would break the tool completely while looking correct.

So there's a second case: a correctly-permissioned file, asserted to be accepted. It looks redundant. It is the only thing standing between you and a guard that has quietly stopped distinguishing.

Three times we didn't have one, and what it cost

The failures are more instructive than the successes, because in each case the missing control was obvious in hindsight and invisible at the time.

The fixture we typed ourselves

A script needed to count entries in a block of output from a standard tool. We wrote the counter, and wrote a test with a small sample of that output typed by hand.

The test passed. The counter was wrong — it over-counted by exactly two, every time, because the real tool indents its section labels the same way it indents the values underneath them, and our hand-typed sample didn't. We had tested our idea of the format, not the format.

It surfaced on real data, where it failed safely and refused to proceed. Lucky.

⭐ A fixture you wrote tests your understanding of the input; a fixture captured from the real thing tests the input. When the format comes from a tool, capture the tool's actual output — even once, even pasted in — rather than reproducing it from memory.

The protocol we didn't test

We built an access check that permitted a known network address, tested it from the command line, and watched it work.

Then it refused a browser, from the same building, at the same moment.

Modern networks carry two kinds of address, and browsers generally prefer the newer one. Our command-line tool had chosen the older one. The check was correct; the list of permitted addresses only covered one of the two ways a machine can arrive.

Worse — and this is the part that would have hurt later — half of the newer kind of address regenerates itself periodically for privacy reasons. Listing one exactly would have worked on the day it was written and stopped working weeks later, with no visible connection to anything anyone had done. The fix was to permit a range rather than an address, which is a different design and not a bigger one.

⚠️ Testing with a tool is not testing with the thing that will actually connect. Where a real client exists, use it: a browser for something a browser will open. The tool and the client are different clients, and they make different choices on your behalf.

The search result we generalised

Looking for permission gaps, we searched a set of files for a particular pattern and reported what came back. The search returned the first match in each file. We described the finding as though it applied to the whole file.

It didn't. The first match was one part of each file; other parts were already correct. The report was more alarming than the truth by a wide margin, and the correction — made before it reached anyone — was narrower and more useful than the original.

⭐ A search tells you where to look. It does not tell you what is there. Before writing up a finding, open the file. This is cheap, it is boring, and it has killed more wrong conclusions than any other habit in this account.

The asymmetry that makes all of this necessary

There is a reason controls matter more than they seem to, and it is not symmetric.

A positive result largely validates itself. If a check finds a real problem, and you fix the problem, and the check goes quiet — the machinery demonstrably worked. You watched it work.

A negative result validates nothing. "No problems found" is what a working check returns when everything is fine, and what a broken check returns when it never ran, never matched, never resolved a name, or was never wired up at all. Emptiness is what success looks like, and emptiness is also what absence looks like, and nothing in the output distinguishes them.

That is why the four shapes above are all, underneath, the same move: manufacture a result you know the answer to, and check you get it. The untouched machine, the forged request, the neighbouring address, the file that must be accepted — each one is a case where you already know what should happen, so the answer tells you about the instrument rather than about the world.

What to actually do

Not a methodology. Four questions, each of which takes under a minute:

1. It broke right after I changed it — is it also broken somewhere I didn't touch?
2. This check assumes something cannot be faked — have I tried faking it?
3. I've proved the right thing is allowed — have I proved the wrong thing isn't?
4. My result was empty — would this check have been empty if it were broken?

The last one is the one that catches the most, and it is the least natural to ask, because an empty result feels like an answer. It usually is. But "usually" is doing a great deal of work in that sentence, and the cost of finding out which case you're in is one more command.

The machine we hadn't changed took ninety seconds. It was the most useful ninety seconds of the day, and we very nearly skipped it.