Changelog — 2.227–2.235

Releases 2.227.0 through 2.235.1. Current releases are on the main changelog, and every band is listed in the release archive.

Loading audio...

Version 2.235.1

August 27, 2026

Our own setup instructions had been quietly creating a security file that anyone using that computer could read — and we only found it while tidying up after moving house

  • Nothing you can see has changed, and nothing about the site was affected. This is about a file on one of our own development machines.
  • What happened. Our written instructions for setting up a secure connection on a Mac told you to create a credential file — and never told you to restrict who could read it. The instructions for doing the same thing on our other machines did, and always had. Only the Mac version was missing that step.
  • So the file sat readable by anything running on that machine for about seven months, in a folder that syncs to cloud storage by default. To be clear about the scale: that credential had already been retired hours earlier and grants access to nothing, and it was only ever reachable by someone already using that computer or that personal cloud account. It was untidiness rather than an incident — but it is the sort of untidiness that is only untidy until the day it isn’t.
  • It was the instructions’ fault, not the person following them. That distinction matters, because the fix for a careless person is a telling-off and the fix for a bad instruction is a better instruction. We changed the instruction.
  • And we changed it so the file can never exist unprotected, rather than protecting it a moment later. The obvious fix is “create it, then lock it down” — but that leaves a gap, however brief, where the file is readable, and in a syncing folder a brief gap is long enough for a copy to leave. It is now created locked from the outset. Prefer the fix that makes the bad state impossible over the one that corrects it.
  • The uncomfortable part: we had checked that very document hours earlier and corrected six errors in it — and this was not one of them. That review asked “does this describe our systems accurately?” It never asked “are these instructions safe to follow?” A review finds the shape of thing it went looking for, and two different questions of the same document, on the same day, found two entirely different sets of problems.
  • We found it by auditing the removal rather than the setup. Nobody would have thought to re-read a seven-month-old instruction. Checking that something was properly taken apart turned out to be the thing that revealed how it had been put together.
  • Nothing to do at your end.

Version 2.235.0

August 27, 2026

We moved development off the laptop that also runs the live majalis — and the safety net we built for the risky step had quietly stopped working before we needed it

  • Nothing you can see has changed. This is about where we build the site, not about the site.
  • The laptop that runs the cameras and encoding for Wednesday and Saturday majalis was also running the whole development environment. Development has moved onto a machine of its own, and the heavy software has come off the laptop entirely. It now does one job on a majlis night instead of two.
  • We arranged it so nothing had to be repointed one address at a time. The new machine took over the old one’s network identity, so everything that pointed at development followed by itself. The alternative would have been a long list of individual changes, each its own chance to get one wrong.
  • The new machine generated its own credentials rather than being handed a copy of the old ones. That sounds like a detail and is the whole safety argument: the laptop’s access stopped working the instant we switched over. Had we copied the credentials instead, the laptop would have re-connected by itself after any restart and the two machines would have fought over the same identity — intermittently, which costs far more to diagnose than an outright failure.
  • We proved the move by showing the old route was dead first. Checking that development works while the old machine is still able to answer proves nothing at all. We also tagged our test request so we could confirm which machine actually replied, rather than trusting that it must have been the new one.
  • We gave development a ceiling on how much memory it may consume — and the obvious ceiling was the wrong one. The obvious choice is the most it has ever actually used. But the software is deliberately permitted to grow well beyond that, so a ceiling at the observed high point would have killed it abruptly the first time it legitimately needed the room — replacing a clear, explanatory error with a silent death. The measurement was the less important of the two things that decided the number.
  • We threw away our first two measurements, because both flattered us. One measured a job that had already been done, so it looked far quicker than it is. The other tried to clear the previous result first, silently failed to, and reported an even better figure — which read as good news and was simply a leftover. A measurement that comes back better than expected deserves more suspicion than one that comes back worse.
  • The safety net for the dangerous step stopped working twenty minutes after we tested it. Before making a change that could have locked us out of the equipment entirely, we set up a spare way in and confirmed it worked. Shortly afterwards it had silently stopped working, while every other sign from that machine looked completely normal. A safety net verified once is not a safety net — so we made the change carry its own automatic undo, which would have reversed it unattended if we had not confirmed success in time.
  • And our own written instructions were wrong in three places, all of them written recently and confidently by people looking straight at the thing they described. Each is corrected, along with a new section on the one step in the process that has no undo.
  • Nothing to do at your end.

Version 2.234.0

August 27, 2026

We built ourselves an admin screen for our security certificates — and deliberately made it unable to do the most useful thing on it

  • Nothing you can see has changed. This is internal tooling, for the certificates our own machines use to prove their identity to one another.
  • Those certificates expire, quietly, about once a month. Nothing goes wrong until the day one lapses, and then several things stop working at once. Knowing in advance is the whole value, and until now that meant someone remembering to go and look.
  • So there is now a screen showing what is enrolled, what expires when, and whether the authority that issues them is healthy.
  • The interesting decision was what to leave OUT. The obvious version of this screen also does the work — press a button, the machine is set up. We deliberately did not build that. It would mean the website holding the keys to every machine we own, sitting behind a login. A website is a much bigger target than a laptop, and a flaw anywhere else on the site would then sit in front of all of it.
  • Instead it writes out the exact commands and a person runs them. The computer does the fiddly part — working out the right names and options, which is where mistakes actually happen — and a human still reads it before anything happens. That also leaves a better record: someone saw it and ran it, rather than a log entry claiming it happened.
  • And we removed a secret we were about to introduce. The original plan had the website create a private key and send it to a browser for download. Moving that step onto the machine that actually needs the key meant the website no longer needs to hold any secret at all. Fewer features, less to steal — and the safer version turned out to be the simpler one.
  • The screen tells you how old its own information is, and says “unknown” rather than guessing. It cannot ask the authority what it has issued — there is no way to ask — so it shows a recording made by a check that runs elsewhere. If that check stops running, the page says so instead of quietly showing last month’s answer as though it were today’s. Old information that looks current is worse than no information.
  • It refuses reports it cannot authenticate — including when nobody has set up how. The tempting shortcut is “nothing configured, so accept everything”, which is exactly how something ends up open in production with nothing erroring. “Not configured” must never mean “open”. We tested every way in, including a made-up name containing a delete-everything command, and each was turned away.
  • Two of our own tests were wrong before the code was. One insisted a command should refuse an empty input, when accepting it — meaning “do all of them” — was correct. The other banned a phrase from a warning message that the warning message needed to contain. A test going red is not proof the code is broken, and a test going green is not proof it is checking the right thing.
  • The two screens that only work from inside our own network are now grouped together and labelled as such. Before, the only way to find out was to click one from elsewhere and get an unexplained refusal — which reads as a broken page rather than a deliberate one, and a protection that surprises the people it protects is a protection that gets switched off. We tried a badge on each row first; it made the longest one wrap onto two lines and look broken, so the grouping says it once instead.
  • Nothing to do at your end.

Version 2.233.0

August 27, 2026

One of our admin pages would have handed someone instructions that took the site off the internet — while every indicator on that page said everything was fine

  • Nothing you can see has changed. This is an internal tool, used by us, to set up the equipment that carries traffic to the site.
  • The page writes out a configuration for a human to copy and apply by hand. We had never once compared what it writes against what is actually running. Doing that for the first time was uncomfortable reading.
  • It would have cut off most of our own connections. It silently leaves out any connection whose identifying key we hadn’t recorded — and we hadn’t recorded most of them. Following the page’s own numbered steps would have disconnected the live site, the development environment and several others.
  • And the page said everything was set up, because a person had ticked a box saying so. That tick was never checked against anything. A status somebody typed is not a status anything verified — and it read reassuringly at precisely the moment the page was at its most dangerous.
  • It would also have quietly undone a redirect that matters for search engines — the one sending the short form of our address to the full one. The page can only describe one kind of rule, and that redirect is a different kind. It wasn’t getting it wrong so much as being unable to say the right thing, which is harder to notice.
  • We found unused code that would have deleted a security certificate. Nothing called it. It was still one keystroke away for anyone who went looking for it, so we removed it.
  • The fix isn’t better wording — it’s a check that runs again. There is now a tool that compares what the page believes against what the equipment is really doing, in both directions, and fails loudly when they disagree. The instructions on the page now begin and end by running it.
  • Two of our own measuring tools misled us while we were building that check, both in the flattering direction. One counted a two-item list as one item. The other quietly did a third of its job and reported success each time, so it looked finished. A tool that fails by making things look better is far more expensive than one that crashes, and we now have a test that would catch each of them.
  • Our written instructions had been wrong for seven months, and were listed as trustworthy. They described addresses as free that had been in use for months, so anyone following them would have collided with something live. We deleted that list rather than correcting it — the software already works it out correctly, and a hand-written copy was only ever going to drift again.
  • Then the same mistake turned up again one layer down. Having fixed the written list, we found the underlying records had the same gap — and the next task on our list was the exact operation that would have hit it. Fixing a wrong document doesn’t fix a wrong record. Only comparing against the real equipment found the second one.
  • Nothing to do at your end.

Version 2.232.0

August 27, 2026

We finished the security change we started yesterday — and on the way found two of our own machines had quietly stopped being able to talk to each other

  • Nothing you can see has changed. This is all internal: how our machines prove their identity to each other, and who is allowed to look at the settings that control it.
  • Yesterday we built it; today we rolled it out everywhere. Each machine now presents a signed proof of identity rather than relying on everyone having memorised its fingerprint. We checked every single one by actually connecting to it, rather than by asking it whether it thought it was fine.
  • Doing it at scale found a fault that had been there for a while. Two of our machines could not connect to each other at all. Both had been rebuilt recently, and the others were never told their new identities. Nothing alerted anyone, because nothing was trying — it only surfaced because we went and looked at all of them at once. This is exactly the problem the change was meant to prevent, sitting there in advance.
  • And we nearly blamed ourselves for it. The first machine we checked failed right after we changed it, which looks damning. But a machine we had not touched failed in exactly the same way, and two others we had not touched were fine. Without that comparison the obvious conclusion was confident, plausible and wrong.
  • We found a genuine permissions gap and closed it. A section of the admin tools that describes our internal network could be read by accounts that should not have had it — not changed, only read, and the most sensitive parts were already protected. It is now restricted properly, and the page checks permission on the server before it sends anything, rather than after.
  • That area is now also restricted by where you are, not just who you are. Even the right account cannot open it from the open internet.
  • The obvious way to build that check would have been wrong in both directions at once. It would have shut out people who are genuinely on site, while letting in anyone who simply claimed to be — because the piece of information it relies on can be typed by whoever is asking. We tested the forgery ourselves before trusting the alternative.
  • Then the check turned us away, and it was right to — our list was written wrong. Modern networks have two kinds of address, and browsers quietly prefer the newer one. We had listed only the older kind, so it worked when we tested it from a terminal and failed from an actual browser. Worse, half of the newer kind of address changes by itself every so often — so listing one exactly would have worked that day and broken later for no visible reason. It now matches a range instead, and we check that a neighbouring range is still refused, so we know the matching is really happening.
  • These certificates expire after a month, quietly. Renewing them needed a password nobody would be awake to type. So we arranged the renewal during the setup, while a person was already there — and gave it permission to do exactly one narrowly-defined thing and nothing else. We then tried to abuse that permission in three different ways and confirmed it refused each time.
  • We also corrected something we had written down and believed. Our own notes said a capability was already switched on. It was not, and we had begun planning around it. It went unchallenged because nothing we had built so far actually needed it.
  • And we wrote plain instructions. There is now a straightforward set of steps for doing all of this, separate from the notes explaining why it works that way — because the thing you read calmly and the thing you follow under pressure are not the same document.
  • And there is a new technical write-up to read, if you like that sort of thing. The thing you didn’t change — about why something breaking the moment you touch it is such convincing evidence, and so often wrong. It is in the help centre.
  • Nothing to do at your end.

Version 2.231.0

August 27, 2026

We found a lost password had made part of our own security system unusable — and fixed it without rebuilding anything

  • Nothing you can see has changed. We changed how our own machines verify who is allowed to connect to them. It is entirely internal.
  • The old way doesn’t scale, and the new way is genuinely elegant. Previously every machine kept its own list of who may connect — so adding a person meant editing that list on every single machine, and removing them meant editing every list again and hoping none were missed. Now each machine trusts one authority, and asks it. Adding a person touches no machines at all.
  • We discovered we already owned the thing that does this. It had been quietly running since January doing a related job; the capability we needed was simply switched off. That is the second time in as many days that the answer turned out to be something already paid for and half-used.
  • Then we hit a wall: a password nobody had. It had been generated automatically on the day the system was first set up, printed once into a log, and that log had long since been discarded. There was never a moment where anyone was asked to save it.
  • We didn’t rebuild — we moved the settings somewhere readable. The password guarded settings stored in a sealed internal database. Moving those same settings into an ordinary text file made the missing password irrelevant rather than fatal, and everything already issued kept working. Settings hidden behind a generated password fail invisibly. Settings in a file you can open fail visibly.
  • Two things then went wrong in a way that looked like success. A tool wasn’t installed where we were working, so we used a different route to the same result. It worked — but the thing it produced was missing a field that limits what it may be used for. The test passed, because the test asked “did it work?” and not “on what terms?”
  • And we wrote down something we could not explain. That missing field behaved differently from what the documentation predicts. We recorded exactly what we observed and explicitly declined to invent a reason for it. A confident wrong explanation is worse than an admitted gap, because someone builds on it later.
  • Nothing to do at your end.

Version 2.230.0

August 27, 2026

We asked whether one of our own rules was being followed, and the check we wrote to answer it broke the same rule

  • Nothing you can see has changed. We keep a local copy of the software libraries our systems are built from, so builds don’t depend on the public internet being available or unchanged. The question was simply: is everything actually using it?
  • The check we wrote to find out gave four wrong answers. It searched for the phrasing our own documentation recommends — and several systems were doing the right thing in a different way, so they were reported as broken when they weren’t. We had searched for the words rather than the meaning, which is a mistake we have a written rule about.
  • Underneath that, one thing really was wrong, and it was invisible by design. One build tried the local copy and, if it couldn’t reach it, quietly used the public internet instead — reporting success either way. So a successful build told you nothing about where its ingredients came from. We kept the fallback, because a local outage shouldn’t stop work; we made it announce itself, and write down which route it took inside the result, so the question is answerable afterwards.
  • Two others weren’t using the local copy at all. One of them is fixed in a slightly unusual way, for a reason worth stating: we used an explicit instruction rather than a setting. If the instruction isn’t understood, the build stops and tells us. A setting that isn’t understood is simply ignored — and everything looks fine.
  • There is now an automatic check, and it refuses to pass on nothing. If it ever fails to find any files to examine — a renamed folder, a moved file — it reports a problem rather than quietly reporting success. That is the same trap we wrote a standing rule about this week: an inspection that can’t run looks exactly like an inspection that found nothing wrong.
  • Nothing to do at your end.

Version 2.229.0

August 27, 2026

Our setup checklist met a real machine for the first time and immediately found things we’d have missed — including one that would never have announced itself

  • Nothing you can see has changed. The setup tool written yesterday was run against the new development machine for the first time. It found nine missing pieces, four of which a list written from memory would have skipped entirely.
  • One gap was the interesting kind: quiet. Our project states a minimum version for one of the tools it’s built with. The standard installation provides an older one. Nothing fails, nothing warns — the machine simply doesn’t meet a requirement we set ourselves, and carries on.
  • So we made the setup tool check its own answer. It now reads the minimum straight from the project rather than having it typed in twice, and its self-test deliberately feeds it the old version to confirm it would object. A check that has never objected to anything isn’t yet a check.
  • We added the assistant tooling, and chose the installation method carefully. Installed the “administrator” way it would be frozen at whatever version we first put on, because it wouldn’t have permission to update itself. Installed for the person using the machine, it keeps itself current. Small choice, and the wrong one only becomes visible months later.
  • And we corrected the tool’s note about itself. Every command it runs has now been run by hand and verified. The automated version has still never been run start to finish. Those are two different claims, and it would have been easy — and wrong — to let the first one stand in for the second. The tool now says which is which, in its own opening lines.
  • We decided against putting a password manager on the new machine, for a reason that surprised us. The obvious move is to install one so the machine can hold its own keys. But our own maintenance scripts deliberately ignore that kind of key store — so a key held that way is the one thing they cannot use. And a password manager on a machine means that machine holds the key to everything in it, which is a larger secret than the handful it would fetch.
  • There is a way this fails that looks like something else entirely. A machine set up the convenient way answers a person perfectly and is completely invisible to our automated checks — which then report it as unreachable, i.e. “that machine is down”. We hit exactly that in August, on a machine someone was logged into at the time. The setup tool now checks for it directly.
  • And the tool’s own tests caught a broken test. Three new checks failed while the thing they were checking was perfectly fine — the test was looking in the wrong place and finding nothing. It failed loudly, which is the only reason we noticed. Had we phrased it the other way round, the same broken test would have passed and checked nothing at all. That is the same mistake we wrote a standing rule about yesterday, appearing inside the tool written to prevent it.
  • Nothing to do at your end.

Version 2.228.0

August 27, 2026

Building this site is moving off the machine that records the majalis — and a missing tool turns out to make a safety check pass by never asking

  • Nothing you can see has changed. The work of building this website has been happening on the same computer that records and streams the Wednesday and Saturday majalis. We are separating the two.
  • The recording is the thing that cannot be redone, so it gets the machine to itself. Every majlis becomes a sermon, a write-up and a transcription. Sharing one computer between recording and development meant the two competed for it, and only one of them can be repeated afterwards.
  • Setting up the replacement needed a written list, and that surprised us. Several of the small tools our safety checks depend on aren’t part of a standard installation. A list written from memory would have missed at least two.
  • And that matters far more than tidiness. Many of our checks ask “is anything wrong?” — so if the tool that asks the question isn’t installed, the check finds nothing and reports all clear. We have been caught by exactly this shape more than once. The setup tool’s main job is therefore the opposite one: prove each tool is present and actually runnable, which a half-built machine cannot fake.
  • We wrote down, in the tool itself, that half of it has never been run. It was written before there was a machine to run it on. Saying so plainly is the point — a note written in the past tense reads as finished, and then nobody goes back to check.
  • Three settings that were right on the old machine are wrong on the new one. One made the computer constantly re-check thousands of files instead of simply being told when they change. Another was quietly protecting us from a runaway process, by accident rather than design — and the new arrangement removes that protection unless we ask for it deliberately.
  • We also found a number that has now meant three different things. Machines get an internal identifier, and when one is retired the number goes straight back into the pool and gets reused within days. Anything written down that refers to a machine by number rather than by name is quietly pointing somewhere else.
  • Nothing to do at your end.

Version 2.227.0

August 27, 2026

Our own notes about our own equipment had a gap in them — and a missing entry turns out to be more dangerous than a wrong one

  • Nothing you can see has changed. A new machine was added to our internal setup, and the notes we keep about our own equipment hadn’t caught up with it.
  • One detail was missing altogether, rather than being wrong. That sounds like the milder problem. It isn’t. A wrong note is something you can read, doubt, and check. A missing note leaves whoever needs it to guess — and the guess comes from whatever similar thing they happen to remember.
  • The guess landed on a different machine. Work began on the wrong one and got a fair way in before a detail failed to match and gave it away. Nothing was altered and no harm was done — but only because the checking happened to come before the changing, rather than after.
  • A second problem looked like one thing and was another. A machine appeared to have no usable storage. The obvious remedy was a disk operation. The actual remedy was adding its name to a one-line list. Both are equally believable from the symptom, and only one of them can destroy data — which is exactly why we now write down the reason rather than only the fix.
  • Two of our measuring tools appeared to disagree, and didn’t. One reported free space one way and the other another; the figures looked different and were identical, because one counts in thousands and the other in 1024s. The instinct is to distrust one of the instruments. The right move was to convert.
  • We deleted two counts rather than correcting them. Both had already gone stale once before, and one had now been wrong in two opposite directions within a fortnight. A number written into a heading is precisely the thing nobody goes back and re-checks. Where a figure can be worked out on demand, we now do that instead of writing it down.
  • Nothing to do at your end.