Over two days we found five separate faults in one small mail system. Not one of them showed up as a failure. Every check that existed was green, every service reported itself healthy, and the whole time the system could not do the thing it was for.
The faults were ordinary. The interesting part is why they were invisible, because it was the same reason five times, wearing five different costumes — and the pattern is not about mail at all.
Note
This page assumes you have configured a server before. Everything in it is generic — SMTP, Postfix, DNS, Docker, shell — and none of it describes our own systems. That is a deliberate rule here rather than an omission, and it costs nothing: the useful part is never the hostname.
The shape, stated first
Each fault sat behind a check that was measuring something next to the problem.
Not a broken check. Not a missing one. A working check, correctly reporting a true fact about something adjacent to the thing that mattered. That is much harder to notice than a check that errors, because a green result from a real instrument is the most convincing thing a system can produce.
A check that cannot fail is worse than no check, because it is consulted and trusted.
Here are the five.
1 · The relay advertised something it could not do
A mail server can offer to encrypt a connection. Ours announced that it could, in the standard way, and had no certificate to do it with. Every client that took up the offer — which is every real client — was refused mid-handshake and gave up. No mail left the building.
The check that existed opened a connection, said hello, named a sender and a recipient, and hung up. It passed, every time, for hours.
It was a real conversation with the real server. It proved the relay accepts mail in plaintext. It never once asked for encryption, so it never met the failure — and the capability that was broken was precisely the one it did not exercise.
A probe that does not speak like the real client is measuring a different resource.
The replacement completes an actual encrypted handshake. That sounds obvious in retrospect. It was not obvious at the time, because the existing check was not lazy or sloppy: it was a careful, deliberate, working probe of the wrong thing.
2 · A setting that was validated at startup and never read
The system had a configuration value that decides where delivery failures are reported. It was documented at length, it was required — the service refused to start without it — and there was a paragraph in the code explaining precisely why it mattered.
Nothing read it.
The library underneath had a default, the default was silently wrong, and the carefully-guarded setting sat in memory being correct and unused. Failures were reported to an address that could not receive them, so a message that never arrived looked exactly like one that did.
This is the most valuable idea on this page, so it gets stated plainly:
Declaring a value and delivering it are different claims, and only the second is worth anything. A start-up assertion answers "is it set?" It never answers "does anything read it?" — and an assertion on a value nothing consumes passes forever, while reading as coverage.
3 · The same mistake again, two hours later
Having fixed that, we added a new setting. It was defined in the template, given a default, listed in the substitution, and written into the example configuration.
It never reached the running process.
Two independent reasons, either sufficient on its own. Setting a shell variable does not export it, and the tool that performs the substitution is a separate process that only sees exported variables. And the container's environment is exactly the list its configuration names — a value in a file used to build that configuration is not thereby passed into the container.
The result was an empty setting, which the mail server rejected outright, killing the process that handles every incoming connection.
Two things are worth saying about this one.
The first is that it happened two hours after we had fixed the identical shape and written up why it happens. Knowing the pattern, and having just named it, was not protection. If you take one practical habit from this page, take this: on any configuration change, go and read the line that consumes the value, and watch it arrive. Reading the line that sets it proves nothing.
The second is that the analogy is what hides it. Two neighbouring settings in the same file had never had this problem, because something else exported them for us. The new one sat one line below two that worked.
4 · The health check was watching the wrong process
While that was broken, the container reported itself healthy throughout.
Its health check asked the master daemon whether it was running. The master daemon was running perfectly. The process that dies when a client connects is a different process — a short-lived one, spawned per connection — and no check anywhere looked at it.
So the port was open, the service was up, the health probe was green, and not one client could send.
The fix was to make the check do the smallest thing that a broken system cannot fake: open a connection and read the greeting. That greeting is produced by the very process that was dying.
The question to ask of any health check is not "is it passing?" but "what would this say if the thing it guards were broken in the obvious way?" If the answer is "the same", it is decoration.
5 · Nothing had ever been delivered, and four instruments said DNS was fine
The last one was the largest, and it had been true since the system was built.
Mail servers commonly run their delivery process in a restricted view of the filesystem — it can see a small subtree and nothing else. That subtree needs its own copy of the machine's DNS configuration, which the software packaging provides.
Our storage directory was mounted over that subtree from the host, for a good and separate reason. Mounting a directory over a path hides what was underneath it. The delivery process had no resolver at all. It could not look up where to send anything, and every message was deferred with an error that reads exactly like a DNS problem somewhere else.
We checked DNS four ways. All four said it was fine, and all four were right:
- a query against the network's resolver — correct answers
- a query through the machine's own configuration — correct answers
- a lookup from inside the container — correct answers
- a lookup from a throwaway container sharing the container's own network — correct answers
That last one felt definitive. It was not, and the reason is worth keeping: the restriction was a filesystem boundary, and we had spent four measurements proving things about the network. Every instrument was sound. Every instrument was pointed one layer away from the fault.
There was also a check specifically for outbound delivery. It opened a connection to a well-known mail server and confirmed it was reachable — from the host, resolving outside the restricted view. It had been green for the entire life of the system, during which the system had delivered nothing.
What actually caught them
Not care, and not expertise. In every case it was the same two things.
A control that must fail. Before believing that a probe succeeded, run it against something that cannot possibly work — a port with nothing behind it, an address known to be absent. If that also succeeds, the probe is measuring something else and its verdict is worthless. This costs one extra line and it is the single highest-value habit here.
Proving a check by breaking the subject. Not by admiring the check, but by deliberately reintroducing the fault and watching it go red. Twice in these two days we wrote a check, watched it pass, deliberately broke the thing it guarded — and found the check still passing. A green result is not evidence. A green result from a control you have proven can go red is.
A postscript, and the sharpest one
When the mail server first met the encryption failure in section 1, it did what it was designed to do: it composed a notification to the administrator describing the fault, with a full transcript of the failed session.
That notification could not be delivered. It sat in the queue for ten hours.
We found it two days later, while fixing something else — and it was a perfect, timestamped description of the original outage, written at the moment it happened, by the system itself.
The monitoring worked. The monitoring could not report. Everything above is a variation on that one sentence.
Three questions worth carrying away
- What would this check say if the thing it guards were broken in the obvious way? If the answer is "the same", it is not a check.
- Does anything actually read this value? Not is it set, not is it validated — is it read, on the path that matters, by the code that does the work.
- What layer is my instrument measuring, and is it the layer the fault lives in? Network, filesystem, process, protocol. Four correct answers about the wrong layer are still four wrong answers.
None of that is about mail. Mail is simply an unusually good teacher, because it has many layers, each of which can fail while looking healthy from the one above.