---
title: "Every Check Was Green: Five Faults Nobody Could See"
description: "A relay advertising a capability it lacked, a setting validated at startup and never read, the same mistake repeated two hours later, a health check watching the wrong process, and a system that had never delivered anything while four DNS probes said it was fine."
canonical_url: "https://www.truthpromoters.com/help/every-check-was-green"
type: "help"
section: "Help Center"
keywords: "health check passes while service broken, STARTTLS advertised without certificate, envsubst empty variable not exported, docker compose environment not passed to container, postfix chroot resolv.conf missing, bind mount hides directory contents, adjacent measurement false green, control that must fail"
---

# Every Check Was Green: Five Faults Nobody Could See

Over two days we found five separate faults in one small mail system. Not one of them showed up as
a failure. Every check that existed was green, every service reported itself healthy, and the whole
time the system could not do the thing it was for.

The faults were ordinary. The interesting part is *why they were invisible*, because it was the
same reason five times, wearing five different costumes — and the pattern is not about mail at all.

> [!NOTE]
> This page assumes you have configured a server before. Everything in it is generic — SMTP,
> Postfix, DNS, Docker, shell — and none of it describes our own systems. That is a deliberate rule
> here rather than an omission, and it costs nothing: the useful part is never the hostname.

## The shape, stated first

Each fault sat behind a check that was **measuring something next to the problem**.

Not a broken check. Not a missing one. A working check, correctly reporting a true fact about
something *adjacent* to the thing that mattered. That is much harder to notice than a check that
errors, because a green result from a real instrument is the most convincing thing a system can
produce.

> **A check that cannot fail is worse than no check, because it is consulted and trusted.**

Here are the five.

## 1 · The relay advertised something it could not do

A mail server can offer to encrypt a connection. Ours announced that it could, in the standard way,
and had no certificate to do it with. Every client that took up the offer — which is every real
client — was refused mid-handshake and gave up. No mail left the building.

The check that existed opened a connection, said hello, named a sender and a recipient, and hung
up. It passed, every time, for hours.

It was a real conversation with the real server. It proved the relay accepts mail **in plaintext**.
It never once asked for encryption, so it never met the failure — and the capability that was
broken was precisely the one it did not exercise.

> **A probe that does not speak like the real client is measuring a different resource.**

The replacement completes an actual encrypted handshake. That sounds obvious in retrospect. It was
not obvious at the time, because the existing check was not lazy or sloppy: it was a careful,
deliberate, working probe of the wrong thing.

## 2 · A setting that was validated at startup and never read

The system had a configuration value that decides where delivery failures are reported. It was
documented at length, it was **required** — the service refused to start without it — and there was
a paragraph in the code explaining precisely why it mattered.

Nothing read it.

The library underneath had a default, the default was silently wrong, and the carefully-guarded
setting sat in memory being correct and unused. Failures were reported to an address that could not
receive them, so a message that never arrived looked exactly like one that did.

This is the most valuable idea on this page, so it gets stated plainly:

> **Declaring a value and delivering it are different claims, and only the second is worth
> anything.** A start-up assertion answers *"is it set?"* It never answers *"does anything read
> it?"* — and an assertion on a value nothing consumes passes forever, while reading as coverage.

## 3 · The same mistake again, two hours later

Having fixed that, we added a new setting. It was defined in the template, given a default, listed
in the substitution, and written into the example configuration.

It never reached the running process.

Two independent reasons, either sufficient on its own. Setting a shell variable does not *export*
it, and the tool that performs the substitution is a separate process that only sees exported
variables. And the container's environment is exactly the list its configuration names — a value in
a file used to *build* that configuration is not thereby passed *into* the container.

The result was an empty setting, which the mail server rejected outright, killing the process that
handles every incoming connection.

Two things are worth saying about this one.

The first is that it happened **two hours after we had fixed the identical shape and written up why
it happens.** Knowing the pattern, and having just named it, was not protection. If you take one
practical habit from this page, take this: on any configuration change, go and read the line that
*consumes* the value, and watch it arrive. Reading the line that sets it proves nothing.

The second is that the analogy is what hides it. Two neighbouring settings in the same file had
never had this problem, because something else exported them for us. The new one sat one line
below two that worked.

## 4 · The health check was watching the wrong process

While that was broken, the container reported itself **healthy** throughout.

Its health check asked the master daemon whether it was running. The master daemon was running
perfectly. The process that dies when a client connects is a *different* process — a short-lived
one, spawned per connection — and no check anywhere looked at it.

So the port was open, the service was up, the health probe was green, and not one client could
send.

The fix was to make the check do the smallest thing that a broken system cannot fake: **open a
connection and read the greeting**. That greeting is produced by the very process that was dying.

> The question to ask of any health check is not *"is it passing?"* but **"what would this say if
> the thing it guards were broken in the obvious way?"** If the answer is *"the same"*, it is
> decoration.

## 5 · Nothing had ever been delivered, and four instruments said DNS was fine

The last one was the largest, and it had been true since the system was built.

Mail servers commonly run their delivery process in a restricted view of the filesystem — it can
see a small subtree and nothing else. That subtree needs its own copy of the machine's DNS
configuration, which the software packaging provides.

Our storage directory was mounted over that subtree from the host, for a good and separate reason.
Mounting a directory over a path **hides what was underneath it**. The delivery process had no
resolver at all. It could not look up where to send anything, and every message was deferred with
an error that reads exactly like a DNS problem somewhere else.

We checked DNS four ways. All four said it was fine, and all four were right:

- a query against the network's resolver — correct answers
- a query through the machine's own configuration — correct answers
- a lookup from inside the container — correct answers
- a lookup from a throwaway container **sharing the container's own network** — correct answers

That last one felt definitive. It was not, and the reason is worth keeping: the restriction was a
**filesystem** boundary, and we had spent four measurements proving things about the **network**.
Every instrument was sound. Every instrument was pointed one layer away from the fault.

There was also a check specifically for outbound delivery. It opened a connection to a well-known
mail server and confirmed it was reachable — **from the host**, resolving outside the restricted
view. It had been green for the entire life of the system, during which the system had delivered
nothing.

## What actually caught them

Not care, and not expertise. In every case it was the same two things.

**A control that must fail.** Before believing that a probe succeeded, run it against something
that cannot possibly work — a port with nothing behind it, an address known to be absent. If that
also succeeds, the probe is measuring something else and its verdict is worthless. This costs one
extra line and it is the single highest-value habit here.

**Proving a check by breaking the subject.** Not by admiring the check, but by deliberately
reintroducing the fault and watching it go red. Twice in these two days we wrote a check, watched
it pass, deliberately broke the thing it guarded — and found the check still passing. A green
result is not evidence. A green result from a control you have proven can go red is.

## A postscript, and the sharpest one

When the mail server first met the encryption failure in section 1, it did what it was designed to
do: it composed a notification to the administrator describing the fault, with a full transcript of
the failed session.

That notification could not be delivered. It sat in the queue for ten hours.

We found it two days later, while fixing something else — and it was a perfect, timestamped
description of the original outage, written at the moment it happened, by the system itself.

**The monitoring worked. The monitoring could not report.** Everything above is a variation on that
one sentence.

## Three questions worth carrying away

1. **What would this check say if the thing it guards were broken in the obvious way?** If the
   answer is "the same", it is not a check.
2. **Does anything actually read this value?** Not *is it set*, not *is it validated* — is it
   *read*, on the path that matters, by the code that does the work.
3. **What layer is my instrument measuring, and is it the layer the fault lives in?** Network,
   filesystem, process, protocol. Four correct answers about the wrong layer are still four wrong
   answers.

None of that is about mail. Mail is simply an unusually good teacher, because it has many layers,
each of which can fail while looking healthy from the one above.

## Related topics

- [The Server Was Fine: Nine Wrong Readings, and the Accidents That Caught Them](https://www.truthpromoters.com/help/nine-wrong-readings) — Rebuilding one server produced nine broken instruments and no broken system — and every one of them failed toward alarm
- [The Thing You Didn't Change: Why Every Check Needs a Control](https://www.truthpromoters.com/help/the-thing-you-didnt-change) — A machine broke the moment we changed it — except it had been broken for days, and the only thing that proved it was a machine we had deliberately left alone
- [Technical Writings](https://www.truthpromoters.com/help/technical-writings) — Notes on the systems behind this site — what broke, what we learned, and what misled us