---
title: "The Server Was Fine: Nine Wrong Readings, and the Accidents That Caught Them"
description: "Nine checks that reported a fault that did not exist, grouped by the five mechanisms behind them — and the three accidental controls that caught them, none of which anyone had designed."
canonical_url: "https://www.truthpromoters.com/help/nine-wrong-readings"
type: "help"
section: "Help Center"
keywords: "pgrep not matching process name, zsh no matches found, dpkg hi instead of ii, grep -c exits 1 when no match, pipeline exit status head, zsh set -- word splitting, false negative check"
---

# The Server Was Fine: Nine Wrong Readings, and the Accidents That Caught Them

We spent a day rebuilding a server — taking it out of service, erasing it, installing a fresh
operating system, and putting its work back. By the end we had a list of nine things that had gone
wrong.

Not one of them was the server.

All nine were **instruments**: a command, a check, a counter, a line in a script, each reporting a
problem that did not exist. Every one of them was working exactly as designed. And every one of
them, without a single exception, failed in the same direction — **toward alarm**.

This is an account of all nine, and of the more useful question underneath them: what actually
caught them. It was not care, and it was not experience. In every case it was **a second source
disagreeing with the first** — and three of the most valuable second sources were accidents nobody
had designed.

> [!NOTE]
> This page assumes you have used a Unix shell. Everything in it is generic — `grep`, `zsh`,
> `dpkg`, `pgrep`, `systemd`, SMART counters — and none of it describes our own systems. That is a
> deliberate rule here rather than an omission, and it costs nothing: the interesting part is almost
> never specific to us.

## Why "toward alarm" is structural, not bad luck

Nine out of nine in one direction is not a coincidence, and the reason is worth having in your head
before the examples.

**A working instrument returns an answer. A broken one usually returns nothing** — an empty list, a
zero, no output at all, a non-zero exit status. And in almost every convention anyone checks
against, *nothing* means *bad*. No matching process means the service is down. No packages found
means the package is missing. An empty result means the search failed.

So the default failure mode of a measurement is **indistinguishable from a negative finding about
the thing being measured**. The tool falling over and the world being broken produce the same
output.

That gives you one genuinely useful asymmetry, which is the most portable idea on this page:

> **A positive result validates itself. A negative result does not.**
>
> If a check finds something, the check ran — you have learned two things for the price of one. If
> it finds nothing, you have learned either *"there is nothing"* or *"the check did not run"*, and
> the output is identical in both cases.

Which is why every one of the nine below cost time, and why the fix is almost always the same shape:
give the failing check a **control** — something you already know it should find. If the control
comes back empty too, the instrument is broken and the alarming reading means nothing.

## The nine, in five families

Nine separate mistakes is a list. The five mechanisms underneath them are worth more, because they
are the ones you will meet again in a tool none of us has used yet.

### One · The measurement changed the thing it was measuring

We check the health of a new disk before trusting it with anything. One reported **55 entries in its
error log** — a number that would ordinarily disqualify a drive.

Every entry was the same benign code, meaning roughly *"you asked me something I do not support"*.
And the act of reading the log had produced the most recent ones. Reading it again made it 56.

So a nearly-new, perfectly healthy drive was being penalised **for having been inspected**, and the
more carefully you inspected it, the worse it looked.

> The count was never a health signal. The *contents* are. A check that counts events without
> reading them will eventually count its own.

### Two · Success changed the shape the check was matching on

To stop a graphics driver upgrading out from under us, its packages are pinned. The verification
listed the installed packages and matched lines beginning `ii` — *desired: install, status:
installed*.

It reported every pin as a phantom.

`dpkg`'s status is **two characters**, and holding a package changes the first one:

```
ii  →  desired = install,  status = installed     ← before the pin
hi  →  desired = HOLD,     status = installed     ← after
```

The check went red **as a direct consequence of the operation succeeding.** Anchor on the property
you actually care about — `^.i `, meaning *any intent, definitely installed* — not on the
representation it happened to have when you wrote the line.

### Three · The command never ran at all

Two of the nine, and they are the ones with no fingerprints. Both are `zsh`, which most modern
systems use as the interactive default and which is **not** bash in two places people rarely think
about.

**A glob that matches nothing aborts the whole command.** In bash the pattern is passed through
unexpanded and the command runs anyway; in zsh you get `no matches found` and the command is never
executed. We ran a sweep for references to a setting across a machine, and it printed nothing. That
reads as a clean, confident *"no references anywhere"*. It was not a result at all.

> This is the one that needs a control most, and the control is free: **search for something you
> know is there, in the same breath.** Ours came back with dozens of matches, which is the only
> reason we knew the empty answer was empty for the wrong reason.

**And zsh does not split unquoted expansions into words.** This is idiomatic bash and broken zsh:

```bash
for row in "alpha 10.0.0.1" "beta 10.0.0.2"; do
  set -- $row          # bash: $1=alpha, $2=10.0.0.1
                       # zsh:  $1="alpha 10.0.0.1", $2 is EMPTY
  ssh "$2" uptime
done
```

In a loop checking that a set of machines was reachable, that produced:

```
ssh: connect to host  port 22: Connection refused
```

Which reads as *"the machine is down"*, not *"my loop is wrong"* — and the machine it was reporting
on was a machine we were in the middle of worrying about.

### Four · The tool answered a slightly different question

**`pgrep` silently matches only the first 15 characters of a process name.** It is not a bug; it is
the width of the kernel's `comm` field, and the manual says so. But a process with a name longer
than that matches nothing, and *nothing* means *not running*. We spent time on a service that was
running perfectly.

Then, having widened the search with `pgrep -f` to match the full command line, it matched **its own
invocation** — because the sweep's command line contains the pattern it is searching for. So the
same tool gave us a false negative and then a false positive, in that order, within a minute.

> `pgrep -f` matching one thing is very often matching itself. Exclude your own PID, or count on
> finding exactly one more than you expect.

**And `grep -c` counts matching lines, not matching things.** An API answered with its entire
payload on a single line, as JSON APIs generally do, so a count of "how many items came back"
returned `1` for a response containing many, and `0` for one containing none — which at least half
looked right, and is the worst possible amount of correct.

### Five · The plumbing swallowed the verdict

Three of the nine, all in the shell rather than in any tool.

**A pipeline's exit status is its LAST command's.**

```bash
some-command | grep ERROR | head -5 || echo "nothing to report"
```

The fallback is unreachable. `head` exits 0 whether or not `grep` matched anything, so the `||`
branch can never run. Capture the output first, then test it.

**`grep -c` fires both branches of a `||`.**

```bash
count=$(grep -c pattern file || echo 0)
```

When there are no matches, `grep -c` prints `0` **and** exits 1 — so the fallback runs too, and
`count` becomes the two-line string `0\n0`. Every arithmetic comparison after that behaves
strangely, and nothing errors.

**And a subshell made a self-test's green case impossible to fail.** We have a rule here that a
check is not trustworthy until you have watched it go red, so our checks carry self-tests with
deliberately broken inputs. In one of them the passing case was written as:

```bash
out=$(run_the_judgement)     # a subshell
```

An `exit 1` inside the judgement ends the **subshell**, not the test — so that case reported success
no matter what happened inside it. A test whose whole purpose is proving a check can fail, which
itself could not.

## What actually caught them

Here is the part we did not expect, and it is why this page exists rather than being a list of shell
gotchas.

Almost nothing on that list was found by looking harder at the thing reporting the problem. Every
one was found because **two sources disagreed** — and in three cases, the second source was not
something anyone had built for the purpose. It was already there, by accident, and the whole skill
was noticing it.

### The USB key that had already worked

A machine would not boot the operating system installer. It got most of the way and stopped, with
nothing on screen and no way in.

The obvious first suspect is the installation media. It is also the most expensive suspect, because
verifying and rewriting it takes a while and proves nothing if it was fine.

**We did not have to.** The same key had installed other machines earlier that evening. That made it
a *proven instrument*, so the machine was the only variable, and the search narrowed immediately to
firmware and boot parameters — which is where the answer was.

Nobody prepared that key as a control. It was simply the key. **The habit worth building is asking,
before you test a suspect, whether something has already tested it for you.**

### The one name that disagreed with its identical siblings

After moving a service from one machine to another, a group of aliases pointed at it. Following the
move, most of them still resolved to the old machine — which reads as *"the change did not take"*,
and sends you to check the change.

**One of them resolved to the new machine.** Identical alias, identical configuration, different
answer.

The difference was that the odd one out is not queried by any routine health check, so nothing had
put a cached answer in front of it. It resolved from scratch and got the truth. Its siblings were
serving cached answers that had not expired yet.

That single disagreement proved, in one command, that the change *had* taken and the problem was
downstream caching — a completely different fix from the one we had been about to attempt. **An
inconsistency between things that should be identical is worth more than any amount of agreement.**

### The revert nobody wanted to do

While hunting the boot failure we changed several firmware settings, as you do. Then it booted.

The instinct at that point is universal and it is wrong: *it works, stop touching it.*

**We put the suspected settings back.** It still booted. So none of them had been the fix, and the
two boot parameters we had also added were.

> [!IMPORTANT]
> That is a stronger elimination than the reasoning we had already done. Checking that other
> machines run happily with those settings is *inference* — "it works elsewhere, so it is probably
> not the cause here." Re-enabling it on **this** machine and watching it boot is direct evidence.
>
> And it had a second payoff nobody was thinking about: had we left the settings as they were, that
> machine would have been permanently, silently different from every other one — a divergence
> introduced by debugging rather than by any decision.

**When a fix lands after several changes, put the ones you suspect back.** The cost of not doing it
is a permanent uncertainty about which change mattered, plus however many unnecessary deviations you
have just baked in.

## The one control we should have had, and did not

Not every second source can be an accident. One was missing entirely and had been for a while.

Our disk-health check is built out of assertions that a counter did **not** move — no new errors, no
new bad blocks, nothing retired. Sound reasoning, and structurally blind in one way: a drive whose
reporting had frozen altogether would sail through every one of them with a perfect record.

**Nothing asserted that the counters *could* move.** So we added the cheapest possible positive
control: write a known amount of data, then confirm the "bytes written" counter went up. If it does
not, the drive is not reporting, and every reassuring zero above it is worthless.

> A check made entirely of negative assertions cannot tell a healthy subject from a silent one. It
> needs at least one thing it expects to find.

## The thing we would tell anyone else

Not *"be careful"*. Everyone is already careful, and careful is what produced all nine of these.

**When a check tells you something has failed, suspect the check first.** Not out of optimism — for
the structural reason at the top of this page: a broken instrument and a real failure produce the
same output, and the broken instrument is more common, because there are far more ways for a
one-line check to be subtly wrong than for a working system to break on the day you happen to look.

**Then give the failing check a control**, which almost always costs one extra command. Something
you already know it should find. If the control comes back empty as well, you have learned the
instrument is blind and the alarming reading means nothing at all.

And **keep an eye out for the controls you did not build.** The three that saved us the most time
that day were a USB key that had already worked, an alias nothing bothers to look up, and the
willingness to undo something that appeared to be helping. None of them was designed. Each was
simply a second opinion that happened to be lying around, and the entire skill was recognising one
when it turned up.

## Related topics

- [Proving a Backup Restores, and Six Tools That Told Us the Wrong Thing](https://www.truthpromoters.com/help/proving-a-backup-restores) — We had backups of every machine and had never restored one, so we erased a server and put it back
- [Technical Writings](https://www.truthpromoters.com/help/technical-writings) — Notes on the systems behind this site — what broke, what we learned, and what misled us