The Certificate Authority That Merely Answers

There is a moment, a week or so into standing up an internal certificate authority, when everything looks finished. The service is running. It answers. You have issued a certificate and it verified. The five-command tutorial you followed ended here, with a green tick.

Almost everything that will later go wrong is already wrong at this point, and none of it is visible.

That is the specific problem with a certificate authority: its failures are quiet by design. A CA that has stopped being able to revoke anything still issues. A certificate with no permissions attached still authenticates. A revocation list that covers nothing still downloads, still parses, still returns a valid signature. You do not get an error. You get a slightly weaker system and no notification that it happened.

This is a collection of things that looked fine and were not, and — more usefully — the checks that told them apart. The commands will age. The discriminators will not.

Most of it is written against step-ca, which is what we run, but only a couple of items are specific to it — the rest are properties of sshd, of RFC 5280, or of how system trust stores work, and they will find you whichever CA you pick.

The habit that catches most of it: verify by issuing

If you take one thing, take this. Reading configuration back tells you what a file contains. It does not tell you what the running service will do.

Those diverge more often than you would expect. Configuration gets cached at startup. A setting gets overridden by an environment variable. A file gets read from a different location than you think. A feature gets disabled and a stale copy keeps answering. In every case the file says one thing and the service does another, and the file is the thing you naturally check.

So: don't check the file. Ask the CA for a certificate and look at what comes back. Every claim in this piece was established that way, and the ones that turned out false were all things that had been read rather than exercised.

It costs about thirty seconds and it is the difference between "configured" and "working".

The lifetime you did not choose

Ask someone how long an internally-issued certificate lasts and they will usually say ninety days. That is Let's Encrypt's number, it has become the ambient assumption, and it is not what you are running.

A self-hosted ACME provisioner will happily default to something far longer. step-ca's default leaf lifetime is a year unless you set claims.maxTLSCertDuration and friends on the provisioner. Nothing warns you. The certificate is valid, the chain verifies, everything works. You simply have a year-long credential where you assumed you had a ninety-day one.

⚠️ A convention is not a measurement. Read notAfter off a certificate you actually issued.

And the length matters for a reason that is not the obvious one. The obvious reason is exposure — a leaked key is useful for as long as the certificate is valid. The better reason is this:

🔴 A one-year certificate means your renewal path is exercised once a year, which is to say never.

Renewal is the part that breaks. It depends on the CA being reachable, the client being scheduled, the credential still being valid, the clock being right, and the service picking up the new file. A year is long enough for every one of those to drift, for the person who set it up to leave, and for the failure to arrive as an outage rather than a warning. Ninety days is not primarily about exposure. It is about finding out.

⭐ Tie the lifetime to the renewal window your client actually uses. Most renew at some fraction of the remaining life — a third, say. If your client renews at a third and the certificate lasts a year, you have four months of silence between attempts, and the first attempt after any change is also the discovery that it is broken.

The revocation you almost certainly do not have

This is the most under-documented thing about SSH certificate authorities, and it deserves stating plainly:

🔴 An SSH certificate cannot be revoked. There is no live check. sshd never contacts your CA.

When a server trusts a CA, it does so by holding the CA's public key and verifying signatures locally. That is the whole mechanism, and it is why SSH certificates scale so well — no lookup, no network dependency, no latency. It is also why revocation does not exist unless you build it.

The trap is that revoking does something, so it looks like it worked. Revoke a certificate at the CA and the CA will refuse to renew it. That is real and useful. It is also not revocation: the certificate already in someone's hands keeps working, on every machine, until it expires.

There is a mechanism — sshd reads a RevokedKeys file — but it is a file you have to write, and distribute, to every host, on every revocation. If you have not built that, you have not got revocation.

⭐ So: whatever lifetime you chose IS your exposure window. That is the number to look at when someone asks what happens if a laptop is stolen. If your user certificates last thirty days, the honest answer is "up to thirty days" — and if that is unacceptable, the fix is a shorter lifetime or a RevokedKeys distribution mechanism, not a revocation command that does not do what its name suggests.

⚠️ Check which you have before you need it, because the moment you need it is a bad moment to discover the answer.

The revocation list that covers nothing

X.509 has revocation lists, and they have a subtler failure.

A certificate can point at where its revocation list lives, and the list itself can declare which certificates it covers. Those two things have to agree. When they do not, a validator does not report an error — it reports that there is no revocation information available, which is a different thing and looks much less alarming.

The way this happens in practice is mundane. step-ca derives that pointer — the crlDistributionPoints extension — automatically from the first entry in its dnsNames, and omits a non-standard port when it builds the URL. So you end up with a certificate saying "my revocation list is at https://ca.example.internal/crl" while the list is actually served on a different port — and the URL in the certificate now points at something else entirely, or at nothing.

You then have two components that are each individually correct and are decoration when combined. The list is signed, current and valid. The certificate's pointer is well-formed. Nothing errors.

⭐ The discriminator is to fetch the URL that is actually written in the certificate, rather than the one you configured, and check that what comes back both parses as a revocation list and claims to cover the certificate you started from. Two steps, and skipping the second is the usual mistake.

The provisioner that vanished from the API and not from disk

Here is one that will waste an afternoon.

step-ca supports two ways of managing itself: a remote administration API, and plain configuration in ca.json. Turning the API off — enableAdmin: false — is a reasonable hardening step, since it removes a network-reachable path to your most sensitive component.

⚠️ It also changes where the service reads its list of provisioners from. With the admin API enabled, that list lives in an internal database; with it disabled, it comes from ca.json. Those two are not automatically the same, and nothing reconciles them.

The result is a provisioner that exists, that has been working for months, that is still present in the configuration you can read — and that has silently disappeared from what the running service will accept. Requests against it fail with an error that suggests it was never there.

⭐ After any change to how a CA is administered, list the provisioners from the running service and compare the list to what you expect. Not the config file — the service.

🔴 And a related caution about writing this sort of thing down. When we hit this, the first correction recorded was that the provisioner "never existed" — written confidently, at the moment its own author had caused it to disappear. It read as the careful, self-critical thing to do, so nobody re-checked it, and it quietly discredited a correct measurement someone else had taken two days earlier. State what is true now. Do not invent a history to go with it.

The permissions that were silently dropped

SSH certificates carry a list of principals — the identities the certificate is allowed to authenticate as. It is the field that makes the certificate mean anything.

Some issuance paths will silently drop the principals you asked for. Issuing with step ssh certificate --sign does exactly this: you pass --principal, the command succeeds, the certificate is issued, and it comes back with none. There is no warning.

A certificate with no principals then behaves in a way that is genuinely confusing, and — being honest — we measured the behaviour without establishing the mechanism, so this piece will not invent one. What matters is the check:

⭐ Read the principals back off every certificate you issue. Not out of the command you ran, not out of the request — out of the certificate itself, with whatever inspection command your tooling provides. It is one line, and it is the difference between a credential that grants what you meant and one that grants something else.

The second listener, and how to find out what it exposes

step-ca can run a plaintext listener alongside the TLS one — insecureAddress — and you may well need it, because some things a CA serves are meant to be fetched without a certificate. A revocation list is the obvious case: requiring a valid certificate to fetch the list of invalid certificates is a circularity you do not want.

⚠️ The question is what else that listener answers, and the documentation is thinner than you would like. The honest answer is to find out rather than to read about it.

⭐ The method is worth generalising beyond CAs: point a throwaway container at it and enumerate, with the secure listener as the control. Ask both listeners the same set of requests from a machine with no credentials and no trust configuration, and diff the two sets of answers. What the plaintext one serves and the secure one does not is your exposure; what both serve is fine; and the control is what tells you your enumeration was actually reaching the service rather than failing for some unrelated reason and looking clean.

That last clause is the point. An empty result from a probe you have not proved can produce a non-empty one is not evidence of anything.

The template that replaced everything

If your CA supports certificate templates, it is worth knowing what "template" means here.

⚠️ A template does not add fields to the default profile. It replaces the profile. Everything the default would have set and your template does not mention is simply gone — key usages, extended key usages, constraints. The certificate issues fine and validates for whatever it does still contain.

⭐ Diff a templated certificate against an untemplated one before trusting it. Issue one of each and compare the full text. Whatever appears in the second and not the first is what you removed, and you should be able to say why for each.

The trust store that ignored your file

Distributing your root certificate to the machines that must trust it is the bulk of the work. It is also where the least interesting failures live.

⚠️ System trust stores can be particular about file extensions, and they do not tell you when they are not. Put a certificate in the right directory with the wrong extension and the update command will scan the directory, ignore your file, report success and exit zero. The trust store is unchanged and every signal says it worked.

⭐ Assert with a real TLS connection to something signed by that root, from the machine you just configured. Never with a file listing, and never with the exit code of the update command. Those answer "is the file where I put it", which was never the question.

And half-distributing is worse than not starting

The last one is not technical.

Rolling out an internal CA means touching every machine, every container image, every language runtime with its own certificate bundle, and every developer's laptop. It takes a while, and there is a strong temptation to enable strict verification as soon as most things trust the root.

🔴 Do not. A certificate warning that appears sometimes is worse than one that appears always, because people learn to click through it. Once they have learned that, the warning has stopped being a signal — and it stops being a signal for the real failure too, which is the one you built the whole system to catch.

⭐ Either the root is everywhere and you enforce, or it is not and you do not. The intermediate state feels like progress and is the only state in which the system actively trains people to ignore it.

What all of these have in common

Read them together and the pattern is hard to miss. Every one is a component that answers correctly while doing nothing useful: a CA that issues but cannot revoke, a revocation list that covers no certificates, a provisioner that is configured and not loaded, a certificate with no permissions, a trust store that ignored the file you gave it.

None of them produce an error. All of them produce a green tick.

So the closing thought is the opening one. Do not ask whether it is configured. Ask what it does when you use it — issue the certificate, read it back, fetch the URL that is written in it, make a real connection from the machine you just changed. Each of those takes under a minute, and each one is the only thing that distinguishes a working certificate authority from one that merely answers.