Changelog — 2.265–2.281

We published an announcements page and forgot to tell search engines it existed

• Dispatches can now be found by searching. Every page on this site is listed in a file we hand to search engines so they know what to look at. When we added Dispatches yesterday we did not add it to that list, so the page worked perfectly for anyone we sent the link to and was invisible to everybody else.
• Nothing was broken, which is the awkward part. There is no error to notice and nothing that goes red. A page missing from that list looks exactly like a page nobody has searched for yet, and the only symptom is an announcement that quietly finds no readers.
• We had written the warning and still walked into it. The line immediately below the one we added is a note-to-self about the last time this happened, on a different page, a few months ago. A comment only reaches the person already reading that file — never the one who did not know it was there. We are building a check that asks the question for us instead.
• The tests now cover it. Our checks on that file looked at the pages generated automatically and said nothing at all about the ones typed by hand, so a passing test run was perfectly compatible with the mistake.

Nothing to do at your end.

A place for our announcements, and a tidier menu to reach it from

• Dispatches. There is now a page for announcements about our work — what we have built, what we are testing, and what we are asking for help with. You will find it in the menu, and the most recent one appears at the top of the home page. Like our other written pages, it can be read aloud to you.
• Why not “News”.Because the word makes a promise. A news page with a handful of posts a year looks neglected; the same posts under “Dispatches” read as deliberate. We would rather write something worth sending than fill a page on a schedule.
• Reflections and Vignettes now share a menu entry. They are the same sort of thing — something to read or watch whenever suits you — and they are only going to get closer.
• Events deliberately stayed where it was.We tried grouping it with the other two and it was wrong: everything else is “whenever you like”, events are “be somewhere at a time”. Hiding that behind an extra tap is how people miss things.

Nothing to do at your end.

Our internal health board now watches something it was not watching

• Nothing you can see has changed. This is about the page we use to check that everything is running.
• A new tool of ours was not on it. It had been quietly left off, which means the board could have shown everything green while that tool was down.
• It is checked the same way a visitor would reach it — from the outside, through everything in between — rather than by asking the machine whether it thinks it is fine. Those two questions have different answers more often than you would like.
• The trade-off is deliberate: a fault anywhere along that path will now show as the tool being down. From where someone using it stands, it is.
• Nothing to do at your end.

A safety check quietly skipped anything that did not fit one of its two questions

• Nothing you can see has changed. This is about the tooling we use to look after our own systems.
• One of our checks makes sure everything that needs a particular credential actually has one. It works that list out by asking two questions — and it only ever considered things that answered one of them.
• Something arrived that answered neither, so the check never visited it. Not an error, not a warning: it simply was not on the list, and a list you cannot see the end of looks complete.
• It would have surfaced at the worst possible moment — part-way through an update, when the thing being updated turns out to be unable to fetch what it needs.
• The check now asks a third question, and we wrote down why both obvious shortcuts were wrong. One would have made a different tool do something actively harmful; the other would have meant recording something untrue in order to make the numbers work.
• Nothing to do at your end.

Our release checklist could leave out a step without ever saying so

• Nothing you can see has changed. This is about our own release tooling.
• The tool that walks us through a release decides what needs attention by looking things up in two lists — and it only ever checked one. Anything recorded in the other produced no instruction at all, which on screen is indistinguishable from “nothing to do here”.
• That is the whole problem: the silence looked like an all-clear. There was nothing to notice, so nobody noticed. We found it while working through a release by hand, not because anything went wrong.
• It now checks both lists, and when something is in neither it says so loudly instead of quietly moving on. We fixed the opposite fault at the same time, where one item could be handed two different sets of instructions in a single release.
• One of our existing tests had been passing for the wrong reason. It claimed to check that a particular case stays quiet, but the quiet came from somewhere else entirely — so it would have gone on passing even with the feature it named completely absent. It now checks both directions.
• Separately, groundwork for a research project that has not started: a worked template for the kind of carefully reviewed teaching material such a project would need, together with checks that refuse anything we have not verified ourselves. Nothing on the site uses it.
• Nothing to do at your end.

A whole family of our internal checks would report “found it” as “didn’t find it” — but only on large inputs

• Nothing you can see has changed. This is our own tooling.
• A common way of asking “does this output contain X?” gets the answer backwards when the output is big. The search stops as soon as it finds a match, and the thing feeding it is then cut off mid-sentence — which our scripts were treating as a failure. So finding the thing looked exactly like not finding it.
• It only misbehaves at size, which is why it survived for years. Every small test passes. Only real data fails, and it fails quietly.
• The assumed threshold turned out to be wrong, and that mattered. We had expected a simple size limit, which would have let us declare most places safe. Measuring showed the real trigger is how quickly the search can decide — so a smaller input with many short lines fails every time while a larger single-line one is fine. There was no safe way to rule places out by size, so we removed the pattern everywhere instead.
• One line covered sixty-five of them, because they all went through a single shared helper.
• The worst instance was a check scanning logs for errors — where getting the answer backwards means reporting all-clear on a genuine fault.
• There is now an automated rule preventing it returning, and we broke it twice on purpose to confirm it actually fires.
• Nothing to do at your end.

The site now builds from a local copy of its ingredients — and we measured the benefit rather than assuming it

• Nothing you can see has changed. This is about how the site is assembled.
• Assembling the site pulls in several hundred pieces of third-party software. Those now come from a copy held a few feet away rather than from across the world — but only when that copy answers first. If it does not, the build quietly uses the original source exactly as before. It costs five seconds to find out and never fails because of it.
• We measured the difference instead of assuming it: about 11% faster on the kind of build where this step actually costs anything. Modest, and honestly smaller than we expected — most builds already skip this work entirely because of other caching.
• The number is trustworthy for a specific reason: the slowest run using the local copy still beat the fastest run using the original. When the two overlap you are measuring noise, and earlier this week we were.
• The real reason to do it is not speed. It is that the outside service does go down sometimes, and when it does we can still build and deploy. That argument does not need a stopwatch.
• Nothing to do at your end.

Older release notes have moved to their own page, before this one got too big rather than after

• Release notes from 2.262 to 2.264 now live on their own page, linked from the archive index. Older links still work — they forward to the right place automatically.
• We did this while the page was still comfortably within its limit. Last time we waited, it grew to nearly twice what it should have been and had to be split into two pages at once, under pressure. Doing it early costs a few minutes; doing it late costs an afternoon.
• The split point is the version currently running in production, so nothing gets filed away before you have had a chance to see it on this page.
• And copying the previous page turned up a mistake in it. One archive page had been carrying another page’s address in two places since it was created — which quietly pointed its audio version at the wrong file and told search engines it was a duplicate. Neither of those complains; they just do the wrong thing.
• There is now a test that would have caught it, and we proved it works by putting the mistake back and watching it fail.
• Nothing to do at your end.

We built a local store for the building blocks our software is made of — then measured it and found the benefit was somewhere else entirely

• Nothing you can see has changed. This is about how the site gets assembled.
• We now keep a local copy of the software components the site is built from, so that assembling it does not depend on a service on the other side of the world being available at that moment.
• Before relying on it, we checked whether it would actually be used — and that check mattered. The build instructions name the original source explicitly, so it was entirely possible for everything to look right while every component was still fetched from the internet and the local copy sat there doing nothing. We proved otherwise by watching the local store grow as a real install ran through it.
• Then we measured the benefit and it was not the one we expected. We assumed this step was one of the slowest parts of a build. It is not — it already costs essentially nothing, because two separate layers of caching handle it. The real gain is on the unusual builds where those caches miss, and on the day the outside service is unavailable.
• Saying that rather than claiming the speed-up, because a benefit measured wrongly is how a thing gets built again next year for the same wrong reason.
• Nothing to do at your end.

Our internal status board could have shown you yesterday’s answer with today’s clock on it

• Nothing you can see has changed. This is the dashboard we watch our own systems on.
• Nothing stopped those health checks being answered from a cache. A check served from a cache tells you a service was working, and looks exactly like one telling you it is — which is the same uselessness as a check that can never report a problem.
• The page’s own answer could be cached too, and that was the worse half: every reading could be taken correctly and still be shown from hours ago, carrying a “checked at” time that looked current because that had been cached along with everything else.
• We found these while chasing a different problem, and they turned out not to be its cause. Saying so rather than claiming the win, because a fix credited to the wrong fault leaves the real one in place.
• Nothing to do at your end.

We wrote a health indicator that was incapable of ever reporting a problem, and caught it two hours later by reading it

• Nothing you can see has changed. Internal plumbing for the machines that build the site.
• The check asked a question whose answer was always the same. It queried a list that, for this particular kind of service, is empty by design and stays empty no matter how well or badly the thing is working. We confirmed that by running the real software and caching something in it — the list still came back empty, and the check still reported everything fine. It could not have failed.
• It now asks for something a real build actually needs, and we tested it four ways: working, broken, and with the outside world deliberately cut off both before and after the cache had anything in it. It correctly stays healthy through an outage once warmed, and correctly refuses when it genuinely has nothing to offer.
• Then the test suite for that same tool turned out to be missing half of what it claimed to check. We deleted one of its two safety checks on purpose and the suite still passed. Two checks were being tested as if they were one. Fixed, and re-proved by deleting each in turn and watching the suite fail.
• A new section on our internal dashboard shows the three stores a build depends on. One of those tiles would have reported our image store permanently broken — not because it is, but because asking a question the wrong way gets you a “not found” that looks identical to a real fault. Measuring it first is the only reason we noticed.
• Nothing to do at your end.

A stored password stopped existing, rather than being changed to a new one

• Nothing you can see has changed. This is housekeeping on the machinery that assembles and publishes the site.
• Two machines doing the same job were set up in two different ways, which meant a check that worked on one could quietly do nothing on the other — and that is most of the reason one of them sat idle for weeks earlier this month without anyone noticing. They are now set up identically.
• Doing that removed a stored password instead of replacing it. The old arrangement had to keep a joining credential on disk permanently, in case it ever needed to re-introduce itself. The new one never needs to, so there is simply nothing to keep, rotate, or leak. That is a better outcome than a fresh secret in a safer place.
• And we kept the work it had already cached. Left alone it would have started from empty and looked perfectly healthy while quietly rebuilding several gigabytes of things it already had.
• Nothing to do at your end.

Our backup monitor announced a disaster when what had actually happened was that it could not read

• Nothing you can see has changed. This is about the internal dashboard we watch our own systems on.
• A check that failed to read its source reported the failure as a finding. When we deliberately pointed it at somewhere with nothing in it, it announced that every single machine we back up had fallen behind — and finished reporting success. It could not tell “I looked and the news is bad” from “I could not look”, and it stated the first with total confidence. It now refuses to answer when it cannot see, and says which of the two happened.
• The readings were also allowed to be a day and a half old while presenting as current. The allowance had been written down years’ worth of habit ago, justified by a note saying how often the check ran — and that note had quietly stopped being true when the schedule changed. The two lived in different places, so nothing could notice they disagreed.
• So the schedule and the allowance are now tied together, and a test fails if they drift apart. We proved that test works by deliberately breaking it and watching it go red, twice, rather than trusting that it passed.
• The check now runs every ten minutes instead of every six hours, so what you see on that dashboard is from this hour rather than possibly from yesterday.
• And it used to tell us an observation was “0h ago”. Harmless when nothing was ever that fresh; meaningless now that most readings are minutes old. It says minutes when it means minutes, and rounds down, so it never claims to be more up to date than it is.
• Nothing to do at your end.

A store of software we rely on could not start without the very thing it exists to replace

• Nothing you can see has changed. Behind-the-scenes work on the machines that build the site.
• We keep a local store of the building blocks our software is assembled from, so builds do not fetch them across the world every time. It turned out that store fetched itself from the internet each time it started up — so if the service it was meant to insulate us from were unavailable, our replacement for it could not start either. It now carries everything it needs from the moment it is built.
• Its health indicator had never once worked, in nine months. It was written to use a tool that has never existed inside that particular container, so it reported a problem continuously while the service was answering in a twentieth of a second. Because it was always red, everyone learned to ignore it — which is just as useless as an indicator that is always green, and arguably worse.
• Its configuration also existed in exactly one place, on one machine, with no copy anywhere. It is now in our repository with everything else, so it can be rebuilt rather than remembered.
• And a new machine joined the estate — brought in through the full checklist rather than just plugged in. Doing that turned up two flaws in the checklist itself, including one where a security key we install goes stale within hours, which had gone unnoticed because nobody had added a machine at the wrong moment before.
• Nothing to do at your end.

A shortcut that makes our builds faster is designed to fail quietly — so we built something that notices when it has

• Nothing you can see has changed. More behind-the-scenes work on how the site is built.
• Rebuilding the site reuses work from the previous build wherever it can. That shortcut is deliberately set up so that if it breaks, the build carries on without it — which is right, because a broken shortcut should never stop us publishing.
• The cost of that is silence. One of our two build machines was getting no benefit from it at all, for as long as that machine had existed, and nothing anywhere said so. It looked healthy the whole time. So we now check three separate things: that every build asks for the shortcut, that each machine is genuinely storing anything, and that the store can be reached from somewhere other than itself — which is what was actually broken.
• The new check had two bugs of its own, and we found them by reading its output rather than trusting that it passed. One made it quietly report on one machine instead of two while looking perfectly happy — the exact fault it was written to catch. The other made it complain about a problem that did not exist. A check that cries wolf is as useless as one that never barks.
• We also corrected the note that started this work. Two of the things it claimed turned out not to be true when measured — including the main task, which had already been done before the note was written. The wrong wording is struck through rather than deleted, because “why did we think that?” is a question only the original answers.
• Nothing to do at your end.

We now keep in-house copies of everything we build, not just the website itself

• Nothing you can see has changed. This is behind-the-scenes work on how the site and its supporting pieces are built and stored.
• A few days ago we started keeping our own copy of the finished website package rather than relying solely on a service out on the internet. That now applies to everything else we build too, so publishing and restoring no longer depend on a round trip across the world.
• We checked what it costs before rolling it out, rather than after. Copying adds around two percent to a build. We had the timings already recorded from the first few runs, so the question was answered from existing data instead of a guess.
• It is built so it can never hold up a release. If the in-house copy is unreachable, the build says so plainly and carries on — because the only thing worse than not having a spare copy is not being able to publish at all.
• And a small embarrassment found on the way: the step counted how many copies it had made, and those counts could never have worked — a quirk of how the loop ran meant the totals were discarded the moment they were added up. We removed them rather than copy the mistake ten more times. A summary that quietly always says zero is worse than no summary.
• Nothing to do at your end.

Text on four of our green panels was too faint to read comfortably, and we had told ourselves it could not be fixed

• You may notice this one. The green “Join Our Community” panel, the news bar across the top of the home page, and the closing panel on the writing page now use dark text on green instead of white. It is a small change and it makes them noticeably easier to read.
• White on that particular green was below the accepted standard for readable text. The usual measure asks for a certain amount of separation between text and its background; these four were at about three quarters of it. That is the difference between comfortable and squinting, and it matters most for anyone reading in bright light or with reduced vision.
• The awkward part: a note in our own code said two of the four could not be fixed without a design decision. That note was wrong, and we only found out by measuring rather than re-reading it. The fix was a single adjustment to how faint one small label is. A confidently written note is one nobody re-checks.
• It had also been unmeasurable for three weeks. The tool that checks colour contrast was one of several that could not start at all — the problem we fixed earlier today. This is the first thing that repair paid for, within the hour.
• Nothing to do at your end.

Several of our own safety checks had quietly stopped working — and the check meant to notice reported that everything was fine

• Nothing you can see has changed. This is behind-the-scenes work on the tools we use to check our own work.
• We keep a set of scripts whose only job is to prove things still work — that a fix really fixed something, that a safeguard can still catch the mistake it was written to catch. Several of them had stopped being able to run at all. They asked for a piece of software we had since replaced with a newer version, so they stopped before they had started, and said nothing about what they were meant to be checking.
• The uncomfortable part is that we had a readiness check for exactly this, and it passed. It confirmed the newer tool was installed — correctly — and never asked what the scripts were actually calling for. So it reported everything ready while a good number of our checks could not start. A green tick against the wrong question is worse than no tick, because it stops anyone looking.
• We had also already found this once, written it down, and not fixed it. The note was still sitting in our own documentation, and the figure in it had since drifted out of date. Writing something down is not the same as doing it.
• The fix now lives in one place instead of being repeated everywhere, and an automatic check refuses the outdated form from now on. More usefully, the readiness check runs each script rather than reading a list of what it thinks they need — so the next tool that goes missing is caught by the thing that was supposed to catch this one. We proved that by breaking it on purpose first.
• Pulling that thread found the same shape twice more, and we fixed both rather than filing them. The routine that is meant to prove each of our automated rules can still catch its own mistake was quietly testing four fifths of them, while reporting that it had tested all. And a check on how text is drawn into pictures had been unable to start for weeks on end — it failed at the very first step, then carried on anyway and blamed the picture-drawing library, which had nothing to do with it. A confidently worded wrong answer is the most expensive kind, because it sends whoever reads it somewhere healthy. That one now stops and says plainly that it could not run.
• Nothing to do at your end.

We now keep our own copies of the software we build, and several machines had quietly lost the ability to fetch it

• Nothing you can see has changed. This is behind-the-scenes work on how the site gets built and published.
• The software that runs the site is assembled into packages, and until now every machine fetched those from a service on the public internet. There is now a copy kept in-house as well, so publishing and updating no longer depend on a round trip across the world — the same reasoning as the last few notes, applied to a different journey.
• The secure part went right first time, which is the point. Setting the same thing up a few days ago took an afternoon and two failed attempts to understand why it kept being refused. We wrote down what we learned; this time it worked in thirty seconds. That is the whole return on writing things down rather than remembering them.
• On the way we found five machines that could no longer fetch anything at all. Their stored credentials had expired. Three of them do real work every day.
• The interesting part is how that failure looked. A machine with no credential quietly carries on doing the things that need no permission. A machine with an expired one presents it, gets the whole request refused, and can no longer fetch even the freely available things. So the broken case fails far more widely than the missing case — and looks like a network problem rather than a login problem. All five are fixed, and one command now checks every machine at once.
• Nothing to do at your end.