We spent a day rebuilding a server — taking it out of service, erasing it, installing a fresh operating system, and putting its work back. By the end we had a list of nine things that had gone wrong.
Not one of them was the server.
All nine were instruments: a command, a check, a counter, a line in a script, each reporting a problem that did not exist. Every one of them was working exactly as designed. And every one of them, without a single exception, failed in the same direction — toward alarm.
This is an account of all nine, and of the more useful question underneath them: what actually caught them. It was not care, and it was not experience. In every case it was a second source disagreeing with the first — and three of the most valuable second sources were accidents nobody had designed.
Note
This page assumes you have used a Unix shell. Everything in it is generic — grep, zsh,
dpkg, pgrep, systemd, SMART counters — and none of it describes our own systems. That is a
deliberate rule here rather than an omission, and it costs nothing: the interesting part is almost
never specific to us.
Why "toward alarm" is structural, not bad luck
Nine out of nine in one direction is not a coincidence, and the reason is worth having in your head before the examples.
A working instrument returns an answer. A broken one usually returns nothing — an empty list, a zero, no output at all, a non-zero exit status. And in almost every convention anyone checks against, nothing means bad. No matching process means the service is down. No packages found means the package is missing. An empty result means the search failed.
So the default failure mode of a measurement is indistinguishable from a negative finding about the thing being measured. The tool falling over and the world being broken produce the same output.
That gives you one genuinely useful asymmetry, which is the most portable idea on this page:
A positive result validates itself. A negative result does not.
If a check finds something, the check ran — you have learned two things for the price of one. If it finds nothing, you have learned either "there is nothing" or "the check did not run", and the output is identical in both cases.
Which is why every one of the nine below cost time, and why the fix is almost always the same shape: give the failing check a control — something you already know it should find. If the control comes back empty too, the instrument is broken and the alarming reading means nothing.
The nine, in five families
Nine separate mistakes is a list. The five mechanisms underneath them are worth more, because they are the ones you will meet again in a tool none of us has used yet.
One · The measurement changed the thing it was measuring
We check the health of a new disk before trusting it with anything. One reported 55 entries in its error log — a number that would ordinarily disqualify a drive.
Every entry was the same benign code, meaning roughly "you asked me something I do not support". And the act of reading the log had produced the most recent ones. Reading it again made it 56.
So a nearly-new, perfectly healthy drive was being penalised for having been inspected, and the more carefully you inspected it, the worse it looked.
The count was never a health signal. The contents are. A check that counts events without reading them will eventually count its own.
Two · Success changed the shape the check was matching on
To stop a graphics driver upgrading out from under us, its packages are pinned. The verification
listed the installed packages and matched lines beginning ii — desired: install, status:
installed.
It reported every pin as a phantom.
dpkg's status is two characters, and holding a package changes the first one:
ii → desired = install, status = installed ← before the pin
hi → desired = HOLD, status = installed ← after
The check went red as a direct consequence of the operation succeeding. Anchor on the property
you actually care about — ^.i , meaning any intent, definitely installed — not on the
representation it happened to have when you wrote the line.
Three · The command never ran at all
Two of the nine, and they are the ones with no fingerprints. Both are zsh, which most modern
systems use as the interactive default and which is not bash in two places people rarely think
about.
A glob that matches nothing aborts the whole command. In bash the pattern is passed through
unexpanded and the command runs anyway; in zsh you get no matches found and the command is never
executed. We ran a sweep for references to a setting across a machine, and it printed nothing. That
reads as a clean, confident "no references anywhere". It was not a result at all.
This is the one that needs a control most, and the control is free: search for something you know is there, in the same breath. Ours came back with dozens of matches, which is the only reason we knew the empty answer was empty for the wrong reason.
And zsh does not split unquoted expansions into words. This is idiomatic bash and broken zsh:
for row in "alpha 10.0.0.1" "beta 10.0.0.2"; do
set -- $row # bash: $1=alpha, $2=10.0.0.1
# zsh: $1="alpha 10.0.0.1", $2 is EMPTY
ssh "$2" uptime
done
In a loop checking that a set of machines was reachable, that produced:
ssh: connect to host port 22: Connection refused
Which reads as "the machine is down", not "my loop is wrong" — and the machine it was reporting on was a machine we were in the middle of worrying about.
Four · The tool answered a slightly different question
pgrep silently matches only the first 15 characters of a process name. It is not a bug; it is
the width of the kernel's comm field, and the manual says so. But a process with a name longer
than that matches nothing, and nothing means not running. We spent time on a service that was
running perfectly.
Then, having widened the search with pgrep -f to match the full command line, it matched its own
invocation — because the sweep's command line contains the pattern it is searching for. So the
same tool gave us a false negative and then a false positive, in that order, within a minute.
pgrep -fmatching one thing is very often matching itself. Exclude your own PID, or count on finding exactly one more than you expect.
And grep -c counts matching lines, not matching things. An API answered with its entire
payload on a single line, as JSON APIs generally do, so a count of "how many items came back"
returned 1 for a response containing many, and 0 for one containing none — which at least half
looked right, and is the worst possible amount of correct.
Five · The plumbing swallowed the verdict
Three of the nine, all in the shell rather than in any tool.
A pipeline's exit status is its LAST command's.
some-command | grep ERROR | head -5 || echo "nothing to report"
The fallback is unreachable. head exits 0 whether or not grep matched anything, so the ||
branch can never run. Capture the output first, then test it.
grep -c fires both branches of a ||.
count=$(grep -c pattern file || echo 0)
When there are no matches, grep -c prints 0 and exits 1 — so the fallback runs too, and
count becomes the two-line string 0\n0. Every arithmetic comparison after that behaves
strangely, and nothing errors.
And a subshell made a self-test's green case impossible to fail. We have a rule here that a check is not trustworthy until you have watched it go red, so our checks carry self-tests with deliberately broken inputs. In one of them the passing case was written as:
out=$(run_the_judgement) # a subshell
An exit 1 inside the judgement ends the subshell, not the test — so that case reported success
no matter what happened inside it. A test whose whole purpose is proving a check can fail, which
itself could not.
What actually caught them
Here is the part we did not expect, and it is why this page exists rather than being a list of shell gotchas.
Almost nothing on that list was found by looking harder at the thing reporting the problem. Every one was found because two sources disagreed — and in three cases, the second source was not something anyone had built for the purpose. It was already there, by accident, and the whole skill was noticing it.
The USB key that had already worked
A machine would not boot the operating system installer. It got most of the way and stopped, with nothing on screen and no way in.
The obvious first suspect is the installation media. It is also the most expensive suspect, because verifying and rewriting it takes a while and proves nothing if it was fine.
We did not have to. The same key had installed other machines earlier that evening. That made it a proven instrument, so the machine was the only variable, and the search narrowed immediately to firmware and boot parameters — which is where the answer was.
Nobody prepared that key as a control. It was simply the key. The habit worth building is asking, before you test a suspect, whether something has already tested it for you.
The one name that disagreed with its identical siblings
After moving a service from one machine to another, a group of aliases pointed at it. Following the move, most of them still resolved to the old machine — which reads as "the change did not take", and sends you to check the change.
One of them resolved to the new machine. Identical alias, identical configuration, different answer.
The difference was that the odd one out is not queried by any routine health check, so nothing had put a cached answer in front of it. It resolved from scratch and got the truth. Its siblings were serving cached answers that had not expired yet.
That single disagreement proved, in one command, that the change had taken and the problem was downstream caching — a completely different fix from the one we had been about to attempt. An inconsistency between things that should be identical is worth more than any amount of agreement.
The revert nobody wanted to do
While hunting the boot failure we changed several firmware settings, as you do. Then it booted.
The instinct at that point is universal and it is wrong: it works, stop touching it.
We put the suspected settings back. It still booted. So none of them had been the fix, and the two boot parameters we had also added were.
Important
That is a stronger elimination than the reasoning we had already done. Checking that other machines run happily with those settings is inference — "it works elsewhere, so it is probably not the cause here." Re-enabling it on this machine and watching it boot is direct evidence.
And it had a second payoff nobody was thinking about: had we left the settings as they were, that machine would have been permanently, silently different from every other one — a divergence introduced by debugging rather than by any decision.
When a fix lands after several changes, put the ones you suspect back. The cost of not doing it is a permanent uncertainty about which change mattered, plus however many unnecessary deviations you have just baked in.
The one control we should have had, and did not
Not every second source can be an accident. One was missing entirely and had been for a while.
Our disk-health check is built out of assertions that a counter did not move — no new errors, no new bad blocks, nothing retired. Sound reasoning, and structurally blind in one way: a drive whose reporting had frozen altogether would sail through every one of them with a perfect record.
Nothing asserted that the counters could move. So we added the cheapest possible positive control: write a known amount of data, then confirm the "bytes written" counter went up. If it does not, the drive is not reporting, and every reassuring zero above it is worthless.
A check made entirely of negative assertions cannot tell a healthy subject from a silent one. It needs at least one thing it expects to find.
The thing we would tell anyone else
Not "be careful". Everyone is already careful, and careful is what produced all nine of these.
When a check tells you something has failed, suspect the check first. Not out of optimism — for the structural reason at the top of this page: a broken instrument and a real failure produce the same output, and the broken instrument is more common, because there are far more ways for a one-line check to be subtly wrong than for a working system to break on the day you happen to look.
Then give the failing check a control, which almost always costs one extra command. Something you already know it should find. If the control comes back empty as well, you have learned the instrument is blind and the alarming reading means nothing at all.
And keep an eye out for the controls you did not build. The three that saved us the most time that day were a USB key that had already worked, an alias nothing bothers to look up, and the willingness to undo something that appeared to be helping. None of them was designed. Each was simply a second opinion that happened to be lying around, and the entire skill was recognising one when it turned up.