Changelog — 2.253–2.261

Releases 2.253.0 through 2.261.0. Current releases are on the main changelog, and every band is listed in the release archive.

Loading audio...

Version 2.261.0

September 2, 2026

A pass that lasts sixteen hours, renewed once a day, is expired for eight of them

  • Nothing on the public site was affected. This is about our own monitoring, and it is written up because the mistake is pure arithmetic and anyone could repeat it.
  • Our machines identify themselves to each other with a pass that expires after sixteen hours. A job renewed those passes once a day. Sixteen is less than twenty-four, so for the eight hours in between there was no valid pass at all — every day, at the same time, in the small hours.
  • It hid because the gap closed before the working day started. Anything anyone tried by hand, at a sensible hour, worked perfectly. Only something scheduled inside the gap ever met it.
  • What met it was the check that watches our backups and our cluster. It ran in the gap, could not log in anywhere, and reported that no machine had answered and it could not tell whether the cluster was healthy. The cluster was completely healthy the entire time.
  • The check threw away the reason, which is the part we actually fixed. It knew only “I got nothing back”, and the sentence it printed named the cluster — the one thing it had learned nothing about. A refused login and a genuinely dead cluster produced identical output. It now keeps the reason and says it, and it distinguishes “I could not look” from “I looked, and here is what is wrong”.
  • That is the second time in two days we have found the same shape — a catch-all handler that produces a confident, specific message about something it never actually checked. It is now written down as a thing to look for rather than a thing to rediscover.
  • The passes are now renewed every six hours instead of daily, which leaves no gap and still works if two renewals in a row fail. And the monitoring check now runs four times a day rather than once, because a cluster can lose quorum at any moment and a once-daily reading can be nearly a day out of date.
  • One near-miss worth recording. The obvious six-hourly schedule — midnight, six, noon, six — is exactly when our backups run, a clash the original schedule had been chosen to avoid. We caught it only because whoever chose the original time had written down why. A schedule is a decision with reasons behind it, and inheriting the timing without the reasons is how you quietly undo them.
  • Nothing to do at your end.

Version 2.260.0

September 1, 2026

Our voice tool blamed a service that was working perfectly and had never been asked anything

  • Nothing on the public site was affected. This is one of our internal editorial tools — the one that turns written text into spoken audio — and it had become effectively unusable.
  • The message it gave was confidently wrong. It said the voice service had rejected the request. The voice service was healthy, had been working all day, and had never been contacted at all. The work stops one step earlier than that, and the error named the wrong step. Hours went into checking equipment that was fine.
  • What actually fails is the pronunciation step. Before anything is spoken aloud, Arabic and other non-Latin passages are rewritten into letters an English-speaking voice can pronounce. On one long passage of scripture, the system doing that rewriting kept slipping back into the original script partway through — about two-thirds of the time. We check for exactly that and refuse to read out the result, which is correct: the alternative is a recording that mangles scripture.
  • The check was right and looked like it was lying, and that is the part worth reading. When it refused, it showed you the first line of what it had received — which was perfect. The problem was several sentences further in, out of view. So a correct warning read as an obvious falsehood, and was disbelieved. It now shows you the actual offending part.
  • We had a backup arrangement for this, and it could never once have run. If one system cannot answer, the work moves to another. But that only triggered when a system failed to reply — and here it replied promptly, politely, with the wrong thing. A wrong answer is not a failure to answer, and nothing in the design had noticed the difference. That is now fixed, and it was a general fault rather than one confined to this tool.
  • One thing you will notice: when it happens, generating takes longer. The passage is handed to a second system, and that one is slower — a couple of seconds becomes something closer to a minute. It only happens on the passages that fail, and we would rather you waited than got a recording that mispronounces scripture. In testing, the second system got it right every single time.
  • We also found we had been recording these as successes. The reply arrived, so it was counted as having worked. That meant a problem happening most of the time was invisible on the screens that exist to show us such things. It is now counted honestly, which will make one of our internal figures look worse and be more truthful.
  • The check built to catch this reported “all clear” while unable to see anything at all. Three of its own routes for saying “I could not look” failed to say it. On the evening this broke, the one instrument for the job would have said everything was fine. We proved the repair by breaking it deliberately — and the first two attempts at that proof were themselves worthless, because the test failed earlier for an unrelated reason and never reached the part being tested.
  • Two sliders in that tool did nothing whatsoever. They were saved with each recording and shown back to you afterwards, so moving one looked like it had taken effect. The engine has no such setting and never did. A control that cannot do anything, presented as one that can, is worse than no control — it is believed. They are gone.
  • Nothing to do at your end.

Version 2.259.0

September 1, 2026

A checklist can be skipped; a check that goes red on its own cannot

  • Nothing on the site has changed — except this changelog page itself, which is the last point below.
  • We wrote down how a new machine gets set up, and then immediately admitted the document is the half that gets skipped. Every recent mistake of ours in this area happened with correct instructions sitting there available.
  • So the part that does the work is a check that runs on its own — and it asks the question backwards. The obvious version walks down a list and asks whether each thing on it is set up properly. But a list cannot report what it does not mention. We have lost track of equipment in exactly that way before: something existed for months, was written down nowhere, and nobody noticed until it failed. This one starts from everything that exists anywhere and asks whether the other records know about it.
  • It expects different things of different kinds of machine, which sounds like a detail and is actually what decides whether anyone keeps using it. A check that cries wolf on its first run is ignored by its second.
  • When it cannot read one of the things it consults, it says so — rather than reporting that everything is missing. Those two look identical on screen, and only one of them is an emergency.
  • It found a bug in itself within the hour, and the embarrassing part is that the same mistake was already guarded against a few lines away. A comparison matched a shorter number inside a longer one and reported two unrelated things as the same thing. The lesson had been learned in that very file and had not travelled.
  • This changelog page has been split again. It had grown past the size where it stays quick to load, so older releases have moved onto their own page, listed in the release archive. Any link you have ever shared to an older release still works — it forwards you automatically. We checked that every single release came through the move exactly once, rather than assuming it.
  • Nothing to do at your end.

Version 2.258.0

September 1, 2026

A tool told us something was misconfigured when it was simply switched off

  • Nothing you can see has changed. This is behind-the-scenes work, written up because the mistake it uncovered is one anybody can make.
  • One of our setup tools reported a very specific problem — a particular setting missing from a particular file — on a machine that was, at that moment, unplugged and being carried across a room. It could not have read that file. It had not read that file.
  • The cause was three answers being squeezed into two. The tool asked a question that could come back “yes” or “no”, and never considered that it might come back with nothing at all. Nothing isn’t “yes”, so it was treated as “no”.
  • That is worse than an error message, not better. Silence makes you look again; a confident, specific, wrong answer sends you off to fix something that was never broken. It now says plainly that it could not reach the machine.
  • What caught it was free, and is the part worth keeping: we ran the same check against a machine that was definitely off. Anything that still reports a finding under those conditions isn’t reading anything.
  • Nothing to do at your end.

Version 2.257.3

September 1, 2026

Something was quietly doing half the work it could, and everything said it was fine

  • Nothing you can see has changed — this is about the machinery that turns our code into the running site.
  • Two machines share the job of building the site. One of them had been sitting idle for weeks, because of a small mismatch in how it described what kinds of work it could accept. Every check we had said it was healthy, and every check was right: it was healthy. It simply was never being asked.
  • That is the interesting part. “Is it switched on” was green the whole time, and a count of what it was doing would have read zero — which looks exactly like a quiet afternoon. Neither question could tell the difference between resting and unreachable.
  • Both machines now share the work, which roughly doubles how much can be built at once, and it needed no new hardware — only the mismatch corrected.
  • Our internal status page now watches for the same thing happening again, by checking that the two machines still agree with each other rather than merely that each one answers.
  • Nothing to do at your end.

Version 2.257.2

September 1, 2026

We made the error message readable, and it was still wrong

  • Nothing you can see has changed. This continues yesterday’s fix to an internal tool, and the follow-up is more interesting than the original.
  • The tool sends its findings to a second copy of the site as a backup. It was building that address by gluing its own path onto the end of an address that already had a different one — so it had been calling somewhere that has never existed, every single time.
  • The message it printed then blamed the wrong thing entirely. It reported the address from its settings rather than the one it actually called, so it accused a perfectly healthy system of missing a feature it has had for weeks.
  • That is the sharper lesson, and it is why this is written up. Yesterday we made a failure legible. Legible is not the same as true — and a clear message pointing at the wrong system costs more than a messy one pointing at the right system, because people believe it.
  • An error message must name what was actually attempted, never what was configured. Four new self-checks now hold that.
  • Nothing to do at your end.

Version 2.257.1

September 1, 2026

A tool we built to make failures readable produced an unreadable failure

  • Nothing you can see has changed. This is a small fix to one of our own internal tools, written up because the mistake is a nicely ironic one.
  • Yesterday we built something to notice when backups stop happening. The first time it ran for real it could not reach the thing it reports to — correctly, because that part had not been rolled out yet.
  • But instead of saying so, it printed the entire web page it got back into our system log — about forty kilobytes of markup, burying the single useful line beneath a rendered copy of the whole site. Which, with some irony, contained the answer.
  • It now says the one sentence that helps, and no longer prints a response without trimming it first. A log line that cannot be read at three in the morning is not a log line — and this tool exists to be read at three in the morning.
  • Its own self-check then caught a mistake in the fix, which is exactly what a self-check is for.
  • Nothing to do at your end.

Version 2.257.0

September 1, 2026

Our internal status board was organised by history rather than by anything useful

  • This is an admin-only page, so nothing public has changed. But the people using our editing tools are users of the site too, and it had been quietly getting in their way.
  • It had grown a section per idea rather than per system. Several sections held a single item, and one service sat by itself in its own category despite running on the very same machine as two others it plainly belongs with.
  • Two entries were named after things that no longer exist — a service retired months ago, and a whole category named after a machine decommissioned in August. Both had entirely correct addresses behind them; only the labels were wrong, which is the kind of mistake that survives every casual reading because everything around it works.
  • Now grouped into eight sections that match how the systems actually relate, with every entry accounted for by counting them rather than by eye.
  • And a correction to ourselves worth recording: three entries had names that looked like placeholders, and we nearly “fixed” them. They turned out to be exactly right — deliberate service names that let a machine move without anything else needing to change. Renaming them would have broken the page at the next migration. A name that looks wrong is not evidence that it is.
  • We also added the machinery underneath — the servers, the backups, the name servers and the thing that issues our internal certificates. That was the original request, and we nearly deferred it.
  • Nearly deferred it because the check it would have relied on was broken, and broken in the worst possible direction. One of our reachability checks reported a service as healthy when the machine could not be found at all, and again when the door was firmly shut. Only a complete silence counted as a failure. It had been quietly doing that for some time.
  • So we fixed that first, and proved the fix by breaking things on purpose — a machine that does not exist, a closed port, an address that goes nowhere, and one that genuinely works. All four now report what actually happened.
  • Each new tile says in plain terms what it does and does not prove. One of them confirms the backup service is answering; it does not confirm backups are running, because we learned in August that those are very different claims. Saying so on the tile is the point.
  • And we added two that answer the harder questions — whether the cluster actually agrees it is a cluster, and whether every machine that should be backing itself up recently did. Neither can be answered by asking whether a door is open, so a small job on a machine that already has the keys works them out and posts the answer in. The website itself holds no keys to anything, deliberately.
  • The first version of that job reported everything healthy while most of the machines were missing entirely. It counted what was there rather than asking what ought to be, so anything absent simply was not counted — and absence read as success. It now starts from the list of machines that are supposed to report, so a missing one is loudly missing.
  • Those tiles have four states, and three of them are not green. A report that has never arrived is not the same as a healthy one, and an old report is not the same as a current one. Getting that wrong would have reproduced, on the very page built to notice silent failure, exactly the kind of silent failure it exists to notice.
  • Nothing to do at your end.

Version 2.256.0

September 1, 2026

The backups that stopped, the drive that was dying, and the checks that said everything was fine

  • Nothing you can see has changed, and no data was lost. This is our own housekeeping, written up because the failures are more interesting than the fixes.
  • Yesterday we found that some of our backups had silently stopped days earlier, after a security improvement replaced a certificate and one group of machines was never told. They failed every night, on schedule, without a word.
  • Today we found something else: a drive holding those backups was dying. Not slow — failing, and failing in a way that made everything around it look slow instead, so the obvious suspects were all innocent. We only found it because unrelated work made us measure how fast things were.
  • Neither problem was actually invisible. Every affected machine was reporting failure correctly, in its own logs, every single night. What was missing was anything that gathered those reports somewhere a person would look. A report nobody reads is not a report.
  • We rebuilt the backup storage from scratch rather than copying the old backups over. That turned out to be both the safer choice and, by a wide margin, the faster one — copying would have taken days to produce something worse.
  • And we fixed the certificate problem properly instead of patching it. The quick patch would have worked that day and failed again, silently, in about three months. We knew that because we had written down the same mistake the last time we made it.
  • The lesson we are keeping, because it caught us four separate times in one afternoon: a check that cannot ask its question returns an empty answer, and an empty answer looks exactly like good news. Twice that day a machine with days of failed backups would have been reported perfectly healthy — not because anything was hidden, but because the tool doing the checking lacked permission to see. Ask questions that can only be answered by success.
  • Nothing to do at your end.

Version 2.255.0

August 31, 2026

Some of our backups stopped three nights ago, and nothing anywhere told us

  • Nothing you can see has changed, and no data has been lost. This is about our own housekeeping, written up because the failure is more interesting than the fix.
  • A security improvement in late August replaced a security certificate on the machine that keeps our backups. Everything that talks to it had to be told. One group of machines was missed, and their nightly backups began failing immediately.
  • The part that broke visibly was fixed within minutes, because a person was watching it. The part that broke quietly had nothing watching it at all — so it failed every night for three nights, and looked exactly like success from every screen we have.
  • We found it by accident, while planning an unrelated piece of storage work. That is the uncomfortable part, and it is why this entry exists.
  • We are fixing the cause properly rather than patching it. The quick patch would have worked today and failed again, silently, in about three months — we have that exact mistake written down from a previous occasion, which is the only reason we did not make it again.
  • And we are building the alarm that should have existed. A backup that stops now gets noticed the next morning instead of the next time somebody happens to look.
  • The lesson we are keeping: our own notes already said “do not trust this list, go and check” — and gave the exact command to run. Nobody ran it. A check that depends on someone remembering to run it is not a check.
  • One more, because it nearly bit us twice: the first version of the script we wrote to detect this returned a wrong answer that happened to look tidy. It was obvious enough to spot. A slightly luckier version of the same mistake would have reported everything healthy.
  • Nothing to do at your end.

Version 2.254.0

August 30, 2026

Two of our backups quietly belong to two different machines — we cannot fix the cause, so we made it visible instead

  • Nothing you can see has changed, and no backup has been lost. This is about how our backups are labelled.
  • Backups are filed under a number rather than a name, and when a machine is retired its number goes straight back into the pool and is handed to the next one within days. The new machine’s backups then land in the same folder as the old one’s.
  • We cannot change that — it is how the backup system is built. So rather than pretend otherwise, we wrote a check that finds and names every folder holding two different machines.
  • The cost of not knowing is a restore during an emergency, which is exactly when nobody reads carefully. The list would offer two unrelated machines with nothing but a date to tell them apart, because the name is tucked away where no restore screen shows it.
  • The first version of the check reported five times too many problems, and that is the part worth keeping. Some backups carry no label at all, and counting “no label” as if it were a machine’s name turned two real cases into ten. Missing information is not a value. The over-reporting version is now kept as a test, precisely because it is what a first attempt produces — and the alarming number is the one somebody acts on.
  • It also has to be told which set of records to look in. The same number can have two entirely separate histories in two different places, one of which no screen anywhere will show you.
  • And when it finds something, it says explicitly not to tidy it up by deleting. A retired machine’s backups are sometimes the only copy left, and we have come close to losing one that way before.
  • We have shipped it with the honest caveat that it has been proven against test data and not yet run for real. Saying so is the point — a tool described as working before anyone has used it is how a gap stays open while looking closed.
  • Nothing to do at your end.

Version 2.253.0

August 30, 2026

A tool that could only describe half of what it was describing — and the fix was to make it refuse rather than warn

  • Nothing you can see has changed. This is about an internal screen that helps us set up how addresses reach this site.
  • Typing truthpromoters.com without the www sends you to the www version, and always has. Our own records had no way of saying that. They could only describe “pass the visitor through to a machine”, so that redirect was written down as if it were one — with a note beside it correctly saying it was a redirect. The note and the record had contradicted each other since January.
  • That mattered because the screen generates instructions to paste into the live setup. Following them would have quietly removed the redirect — and the tool that checks such instructions for mistakes would still have said they were fine, because they were valid. They were just not what we meant.
  • The bigger change is a refusal. The same screen builds the file listing who may connect to our gateway. If any entry is missing its key, that entry is silently left out — and applying the result would disconnect them. It used to print the file with a warning at the top. It now prints no file at all.
  • A warning is advice, and advice can be scrolled past. The danger was never that the file was unlabelled; it was that a file which looked usable existed. There is now nothing to paste, which is not a thing you can overlook.
  • Writing tests for it turned up a case we had not considered. If the list of connections comes back empty, that is far more likely to mean the question failed than that nobody is connected — and an empty file would have disconnected everyone at once. It refuses on that too.
  • And a counter on that screen had been reassuring us with the wrong number. It showed how many entries someone had ticked as done, not how many were actually usable — so during a real fault it displayed zero problems. Both numbers are now shown, because hiding the disagreement is what made it hard to see.
  • Nothing to do at your end.