Changelog

The two links we added yesterday were the wrong colour

• Small and entirely cosmetic. Yesterday’s entry was the first on this page ever to contain a link, and we gave those links a colour picked out of the air rather than the one the rest of the site uses. They now match every other link here.
• Why it was worth fixing rather than leaving. The right colour is not a fixed value, it is a setting that follows whichever theme you are viewing in. The wrong one was pinned, so it would have stayed put through any future change to the site’s colours while everything around it moved.
• Spotted by a reader, which is the only way this sort of thing gets found. Nothing to do at your end.

A new technical writing: nine ways a security system can look finished and be doing nothing

• A fourth piece in Technical Writings. The certificate authority that merely answers — about the system that issues the digital identities machines use to prove who they are to each other.
• Why it is worth reading even if you never run one. The subject is a particular kind of failure: something that answers every question correctly while accomplishing nothing. A revoke command that does not revoke. A certificate that expires in a year when you believed three months. A list of cancelled credentials that is perfectly valid and covers none of them. Not one of these produces an error message.
• The honest bit. Every one of the nine is something we got wrong first and found afterwards, and the piece says so. That includes an awkward one about writing rather than about software — a correction we published that was right about the present and wrong about the past, and which quietly discredited a colleague’s correct measurement in the process.
• Nothing in it describes our systems. That is a rule we apply mechanically rather than case by case: no machine name, address or count of ours appears, and we checked by searching for them rather than by reading it over. The findings survive that completely, because each one is a property of the software rather than of where it happened.
• Nothing to do at your end — it is simply there if it is useful to you.

Our development database was sitting on the same server as the live one, and now it is not

• Nothing on this website changed. This is about how we build the site, not about the site itself. We write these up anyway.
• What was wrong. The database we experiment against lived on the same machine as the real one that holds everything you see. Nothing had gone wrong, but the arrangement meant a mistake in the wrong window could have reached somewhere it should never be able to — and it also meant two people working at once could quietly overwrite each other’s work-in-progress.
• What we did. Each developer now gets their own private copy, running on their own machine, seeded from a backup. The live database is no longer something a development mistake can touch at all.
• How we checked the copy was faithful. Not by whether the command succeeded — by comparing every one of the 73 tables, one at a time. Seventy-one matched exactly. Two did not, and rather than shrug those off we proved why: they were short by precisely the six and seven records added in the hours between the backup being taken and it being restored. Six and seven, not roughly six and seven.
• A small thing we got wrong and are recording anyway. One of the safety checks we wrote to guard this had a deliberately-broken version alongside it, to prove the check could actually detect a fault. It could not — the “broken” version we chose turned out to be perfectly correct, so it demonstrated nothing. A test that passes when it was supposed to fail is not reassurance; it is the same test written twice. We replaced it with one that genuinely fails.
• Nothing to do at your end.

We rebuilt the foundations under the system that stores this site’s code — and the written plan for doing it turned out to be wrong

• Nothing on this website changed. This is behind-the-scenes work on the system our developers use to store and review changes to the site. We are writing it up because we write everything up, not because you need to do anything.
• What was wrong. That system was using a simple file-based database, which only lets one thing write at a time. As the amount of work going through it grew, that started producing intermittent failures — roughly fifty in the preceding week — whenever several things happened at once. It had been logged as a nice-to-have improvement. It was not; it was already failing.
• The plan we had written for the job was wrong. It named a specific command to perform the conversion. That command does something else entirely — it converts between two other kinds of database. Following our own written plan would have failed partway through, with everything already shut down. We only found out because we rehearsed the whole thing first against a disposable copy.
• The rehearsal was the point. Every number in the real run had already been measured on the copy, so the maintenance window was a repeat performance rather than a first attempt. Total time offline: three minutes and ten seconds.
• How we checked it worked. Counting records is not enough — a database can read back perfectly and still fail the moment something new is written. So we wrote: created an entry, commented on it, closed it, cloned the code, and ran a real build. Reading proves nothing that writing does not prove better.
• And one claim we had to correct about ourselves. Before starting we said rolling back would mean undoing seven lines of configuration. It would not have: the software rewrites its own configuration file on startup, so a rollback would also have needed that file restored. We found this out afterwards, from a backup we had taken anyway. The plan was wrong; the habit of taking the backup was what covered it.
• Nothing to do at your end.

We stopped sending your IP address to an outside company, and started listening when your device tells us where you are

• Your address is no longer sent anywhere outside. When your browser would not tell us your location, three parts of this site quietly asked an external company to look your internet address up for us — over an unencrypted connection. That has stopped completely. The same job is now done by a machine we run ourselves, on our own network. Nothing about you leaves.
• We were also throwing away your actual location. If you had granted location permission, we asked your browser for it and gave it ten seconds — then gave up on it after five, and if the real answer arrived at six seconds we discarded it and used the rough guess instead. On a desktop computer, which has no GPS chip, working out where you are means asking the network and often takes longer than five seconds. So the more patient your device was being, the more likely we were to ignore it.
• Why that mattered. A visitor in Leicester was being shown the location of their internet provider in Rochdale, 160 km north. For prayer times that is nine minutes out on Isha and four on Dhuhr and Asr. Not a cosmetic problem.
• What happens now. The quick approximate answer still appears straight away so nothing sits blank — but if your device produces a proper fix a few seconds later, we now take it, and we recalculate the prayer times to match rather than just changing the name at the top.
• Something no amount of better data could fix. Looking up an internet address tells you where that address is registered, which is often your provider’s offices rather than your home. That is why setting your location yourself remains the only way to get times that are exactly right — and why we would now rather ask you than confidently show you somewhere you are not.
• And two things we broke yesterday and fixed today. The location service we deployed on Wednesday sat reporting itself unavailable for five minutes after its data had actually arrived, because it only ever looked once at startup. And its own self-check assumed a program that is not installed on that machine — it happened to fail loudly rather than quietly, which was luck rather than design, and it now says plainly that it cannot run instead.
• Nothing to do at your end. If you have already set your location by hand, none of this changes anything for you.

Our location lookup thought a lot of British visitors were in Detroit — and we had written down the reason a month ago

• What was wrong. When your browser will not tell us where you are — you declined, or it timed out — we fall back to working out an approximate location so that prayer times still show something sensible. That fallback was using a reference dataset that had quietly gone years out of date. Checked against real visits to this site, it put thirteen of thirty-six people in the wrong city, including several British visitors in Detroit, Madrid and Vietnam.
• Why nobody noticed. Being wrong about a location looks exactly like being right about one. There is no error, nothing turns red, and the answer is a real place name. The only way to tell is to already know where the person was.
• The uncomfortable part. Our own notes had warned about this, in writing, a month earlier: the data ages silently and nothing reports it. That warning was correct, it was read, and it changed nothing — because a warning has no way to fail. It has been replaced with a check that does, and which now goes red on its own if the data ever stops being refreshed.
• What we changed. The dataset is now updated automatically every day rather than whenever somebody happened to rebuild the service. More usefully, it now also tells us how confident it is: some addresses can only be narrowed to “somewhere in this country, give or take a thousand kilometres”. Previously that was indistinguishable from a precise answer. Now we can tell, and refuse it.
• What that means for you. If you have set your location yourself, nothing changes at all — your choice has always taken priority and still does, and it remains the only way to get times that are exactly right. If you have not, the guess should now be better, and where it is genuinely too vague to trust we would rather ask you than quietly show you the wrong times.
• A caution we would rather state than bury. The first comparison we ran suggested the new data was barely any better. It was measured against well-known internet infrastructure addresses, which every such dataset handles badly and which no visitor actually uses. Measured against real visits it was a large improvement. We nearly published the flattering-sounding conclusion from the wrong sample, and it would have been wrong in both directions.
• Nothing to do at your end.

We finally removed a security warning our own team had been trained to click through — and briefly broke the backups doing it

• Nothing you can see has changed. This is internal: the administration screens our team uses have been showing a browser security warning for years, and they no longer do.
• The warning was the problem, not the symptom. A warning that appears every single day is one people learn to dismiss without reading — at which point it can no longer tell them about a real one. That is the whole reason for the work.
• Our first measurement was wrong, and it nearly sent us down the wrong path. We checked whether our machines could reach the certificate service, and the answer looked like “most of them can”. But the checking tool had been told to skip the very security check we were asking about. Asked properly, the answer was none of them could. A test that quietly disables the thing it is testing will tell you what you hoped to hear.
• Then we took the backups offline for about four minutes. Giving the backup server a new certificate broke the connection to it — because the machines that back up to it had memorised the old certificate exactly, and refused anything else. That is the security check working correctly; we simply had not asked who was relying on it before changing it.
• The obvious fix would have failed again in November, silently. We could have told them to memorise the new certificate — but certificates now renew themselves every three months, so it would have broken at the next renewal, with nothing to announce it. We changed it to verify properly instead of memorising, which keeps working through every future renewal.
• And we proved it by actually taking a backup, not by reading a status screen. The status said “working” in both the broken and the fixed configuration, so it could not tell the two apart.
• One address deliberately got worse before it got better. A shared address covering the whole cluster briefly stopped working entirely, because the old certificate we replaced was the only thing covering it. We had planned to fix that later; postponing it did not leave it unchanged, it made it worse. Fixed the same afternoon.
• Nothing to do at your end.

The job that renews our security certificates could fail in four ways, and two of them told nobody anything

• Nothing you can see has changed. This is internal housekeeping — the automatic job that keeps our own machines able to prove who they are to each other.
• Our notes claimed a failure would always show up on an admin screen. It would have, twice out of four times. If the job failed early enough it wrote nothing at all, and the screen went on showing yesterday’s perfectly reassuring picture. Six different early failures behaved that way.
• So the job now reports on itself, and an alert appears on the dashboard rather than waiting to be looked for. The machine that runs it is a single-purpose box nobody logs into, so “it’s on a screen” only helps if something tells you to open that screen.
• The important part is that it also fires when the job does not run at all. A machine that is switched off cannot report that it is switched off — so an alarm that waits to be told about a failure will never hear about the worst one. It now treats prolonged silence as a problem in its own right.
• And “silence” is measured from the last time it actually WORKED, not the last time it ran. A job failing every hour looks perfectly busy; checking only for recent activity would report everything healthy right through a total outage.
• While adding this we found one of our own safety checks had quietly stopped checking anything. It was written to catch a specific mistake, and it still reported success after we deliberately made that exact mistake — it was looking for the problem on a single line, and real code spreads across several. A test can stop enforcing its rule without ever turning red, which makes it worse than having no test, because everyone believes it.
• Fixing it went wrong twice more, and both times looked like success. The check reads a section of the file, and our first two ways of finding that section quietly selected the wrong part — once selecting nothing at all, and once selecting the check’s own text so that it counted itself. Both produced confident, wrong numbers. We then deliberately broke things three ways to confirm it now goes red.
• The alert is not dismissible, on purpose. An unread message can be dismissed because reading it is the whole point; a failing renewal clears when the next attempt succeeds and by nothing else. Anything you can wave away without fixing is something that gets waved away.
• Nothing to do at your end.

We switched the cancellation list on for real, and checked the certificate itself rather than the settings file that was supposed to produce it

• Nothing you can see has changed. Internal security housekeeping again — finishing the job described in the previous two entries.
• The change we described last time is now actually applied. Certificates issued from today carry a pointer telling anything that checks where to find the list of cancelled ones, and that pointer now leads to the list rather than to an unrelated page.
• We confirmed it by issuing a real certificate and reading it, not by re-reading the setting we had just typed. That distinction has caught us out before: a setting can be present in a file and absent from what the system actually produces, which is exactly what a half-applied change looks like.
• The check we were most glad we ran was the one that seemed unnecessary. The mechanism that adds the new pointer replaces the certificate’s whole template rather than adding to it — so it silently owns everything else on there too. We had already tested that nothing was lost, but on a practice copy. Confirming it on the real one is what tells us a different feature will not quietly stop working months from now, long after anyone would connect the two.
• And we corrected our own instructions in the same breath. They still described this as work to be done. A guide that describes finished work in the future tense is worse than no guide, because someone will follow it and undo something.
• Nothing to do at your end.

We built a cancellation list nobody was reading, then found the list was pointing at a page that was not it

• Nothing you can see has changed. This is internal security housekeeping — the system our own machines use to prove who they are to each other.
• Last time we published a list of cancelled passes and said plainly that nothing was reading it. This time we gave it a reader. There is now a check that fetches the list, confirms it is current, and compares it against what our machines are actually presenting — and it can fail, which is the entire point.
• The list said, in its own small print, which address it covered — and that address served an ordinary web page instead. A cancellation list is only trusted for the exact address written inside it, and ours had been generated with a detail missing. So anything that went looking would have arrived somewhere unrelated, received a perfectly normal-looking response, and learned nothing. It would not have reported an error; it would have concluded that nothing had been cancelled.
• The counting is deliberately arranged so that “nothing wrong” cannot hide “nothing checked”. Very few of our machines use these passes yet, so a check asking only “is anything cancelled?” would answer no forever and look reassuring. Ours reports how many it actually examined first, and treats zero as a finding rather than a pass.
• We had recorded that none of our main machines were using our own passes. Two of them had been, since January. We went and looked rather than trusting the note. Worse, they had been set up by hand in a way that will not renew itself and that a routine maintenance command quietly undoes — so it would have expired unnoticed in the new year. Nothing is broken today, and it is now written down properly and scheduled to be corrected deliberately rather than discovered during an outage.
• Before opening a new door, we measured exactly how wide it was instead of reasoning about it. Publishing the list needed a plainly-reachable address, which sounded like a meaningful loosening. We built a throwaway copy of the system to find out, and confirmed it exposes the cancellation list and refuses everything else — checking each refusal against the same request on the normal route, so that a mistyped test could not be mistaken for a locked door.
• Our administrators can now see all of this on one screen, alongside the equivalent view we already had for machine logins: what every service is presenting, when each expires, and what has been cancelled — with the honest caveat, on the page itself, that publishing a cancellation list is not the same as anything consulting one. The three-month expiry we introduced last time is still the thing doing the real work.
• Nothing to do at your end.

We planned a day’s work to add something we had switched off ourselves an hour earlier — and the note explaining it had never existed was written by the people who removed it

• Nothing you can see has changed. This is all internal certificate work — the system that lets our own machines prove who they are to each other.
• We had written down that a capability “never existed”. It had existed since the day the system was built. We turned it off ourselves as a side effect of an unrelated fix, checked an hour later, found it absent, and recorded that it had never been there. The evidence was still on disk the whole time, and a colleague’s note from three days earlier had described it working — that note was filed as a mistake. It was not.
• The lesson is uncomfortable and worth stating plainly: a correction is trusted more than an ordinary claim, and it should not be. Admitting a mistake reads as the careful, honest thing to do, so nobody goes back and re-checks the reasoning of someone owning up. A correction can state what is true now without also inventing a history. “This does not exist” was complete, checkable and enough. The four extra words — “and it never did” — cost nothing to type and were the only part that was wrong.
• The fix we had feared was a restoration. It took minutes rather than the day it had been scoped at, and we proved it by actually issuing a certificate — because a system answering “yes I am here” is not the same as a system that can do its job.
• Separately, a safety check on the machine that renews our certificates could never have passed. It was written for one kind of computer, and the machines we now work on behave differently — so the check quietly refused everything. The dangerous part is when it would have surfaced. That check is only consulted when a certificate genuinely needs renewing, and none did yet, so every nightly run reported everything healthy. It would have failed on the first day it actually mattered, several weeks from now, with nothing before then to suggest a problem.
• Its own self-test was completely green on a completely broken check, for two separate reasons, and neither could ever have failed. One test had accidentally switched off the very safeguard it was testing, so nothing could refuse. The other looked like it passed because the safeguard was refusing correctly — when in fact it was refusing everything. Both now check why something was refused rather than merely that it was, and we deliberately reintroduced the fault to confirm they go red.
• We had been assuming an industry convention was our own setting. Certificates elsewhere are usually short-lived; ours lasted a year, and nobody had measured it. That matters because a renewal that only happens once a year is a renewal nobody has ever watched work. They now last three months — so the automatic renewal is exercised four times a year instead of never, and we confirmed it by issuing a real one rather than by reading the setting back.
• We also checked whether we can actually cancel a certificate, rather than assuming. We can now publish a proper cancellation list, and we proved it works by issuing a certificate, cancelling it, and watching it appear — having first confirmed it was not there, because a test that cannot fail proves nothing. But almost nothing out there actually checks such a list, and ours does not yet tell anything where to look. So the honest description is “published, and unread”, and that is what our own screens say. A safeguard that is believed and does not fire is worse than one everybody knows is missing — and the three-month expiry above is the thing genuinely doing the work in the meantime.
• Moving that powerful credential to its own machine turned up a failure that looked like the whole estate being down. The new machine created its passes perfectly, confirmed they said the right thing, and then reported that every single one of our machines was unreachable. Every one of them was fine. It had made the credentials and never offered them — a step missing from instructions written an hour earlier. What stopped that becoming an alarm was a guard we had built the day before, which refuses to file a report claiming everything is broken, on the grounds that it is far more likely to be the reporting machine at fault. It was right, but it was guessing, so the tool now asks directly and says exactly what is wrong.
• Nothing to do at your end.

Our machines’ security certificates now renew themselves — and the harder question was not when to run it, but where anyone would see it fail

• Nothing you can see has changed. This is the internal certificate work from the previous two releases, finally put on a schedule.
• These certificates last about a month, and every one of ours expires on the same day, because they were all set up in one sitting. That turns something you would notice gradually into a single event, and nothing was watching for it.
• It could not simply be “run the renewal on a timer”. Renewing requires logging in, and the pass needed to log in lasts sixteen hours — so the timer has to issue itself a fresh one every time before it can do anything at all. It then checks that the pass it received actually says what it asked for, because we have already seen this system hand back one that was subtly empty and still worked.
• The real design question was where a failure would be seen. A renewal that fails quietly is worth less than the calendar reminder it replaces, and we have a standing example of a warning light nobody reads. So it reports to the screen we built for this: if renewal fails, the screen shows the certificates marching towards their expiry date; if the whole thing fails, the screen says its information has gone stale. Neither depends on anyone remembering to check a log.
• It runs daily rather than weekly, and that is not caution — it is arithmetic. The renewal only acts in the last ten days before expiry, so running daily gives ten chances to succeed and running weekly gives one. A single missed week would be a lapse.
• We also fixed a check that was guarding the wrong thing. It refused to file an empty report — a situation that essentially never happens — while happily filing one saying every machine we own is unreachable, which is what a failure of the reporting machine looks like. Those two are indistinguishable on screen, and only one of them is an emergency. It now declines to file such a report at all, because a screen admitting its information is old is honest, and a screen announcing a fault that is not occurring is not.
• The new tool’s own self-check caught two mistakes in itself before it caught anything else, both the same one: a check searching for a phrase found the phrase inside its own instructions and reported a problem that did not exist. A tool that inspects a file it is part of will find itself, and we now say so in the code rather than leaving the next person to rediscover it.
• And the credential this needs is a genuinely powerful one, so we have written down honestly what holding it costs and raised the work to move it onto a machine that does nothing else. File permissions protect the original and do nothing about the copies in our backups — the answer to those is changing the credential periodically, which is now recorded as a task rather than an intention.
• Nothing to do at your end.

The security screen we built last time was empty — and the reason looked so exactly like a missing key that fixing it the obvious way would have undone the thing it was protecting

• Nothing you can see has changed. This is the internal screen from the previous release, which had nothing on it because three separate things stood between it and its own data. Only one of them was the one we expected.
• The first was simply that nobody had given it the password it needs to accept information. It was refusing everything, loudly and correctly, and saying so. We checked that it still refuses a wrong password before we believed it accepting the right one — something starting to work is not evidence that the lock works, and that is the wrong order to find out in.
• The second is the interesting one. Our pass was refused by a number of machines, with an error that means “I have never heard of you”. The natural reading is that those machines are missing our key, and the natural fix is to go and install it on each of them — which is exactly the permanent, never-expiring access that this whole system was built to get rid of. In fact nothing was missing at all. A pass is issued in a particular name, and we were presenting one issued in the wrong name for those machines. The fix is a second pass, which expires by itself, rather than a key on a shelf that does not.
• An error message that suggests a plausible wrong fix is worse than an obscure one, because nobody stops to check a diagnosis that already makes sense. It is written down now, next to the instruction it would otherwise derail.
• The third would have been reported to us as a disaster that had not happened. A tool only offers a pass it has been told exists, and on our new development machine it had not been told — so our estate-wide check reached almost nothing. That result is not an error; it is a complete, well-formed report saying every machine is unreachable. It would have posted successfully and shown everything as broken, at a moment when nothing was.
• The check refuses a report containing nothing. It did not refuse one containing nothing but bad news. Those are the same failure to a reader and very different to a guard, and the realistic failure is the second one. The breakdown of the machine doing the reporting looks identical to the breakdown of everything it reports on — so we now confirm the check can reach things before trusting what it says about them.
• A helper tool of ours refused to run because it was looking for a file in a Mac-only location, and development moved onto a different kind of machine last week. That tool exists specifically because the obvious alternative silently mangles the thing it edits — so refusing to work left only the damaging route available. It now looks in the right place for whichever machine it is on.
• That is the second Mac-shaped assumption to surface in two days, after the one in the previous release. Two is not a coincidence — it is what moving development to a new machine is for, and we are expecting more.
• We found that none of these certificates renew automatically, and they all expire on the same day. They were all set up in one sitting, so they all lapse within about an hour of each other — which turns a slow trickle anyone would notice into a single event. There is now a ticket to schedule it, and it records the awkward part honestly rather than quietly solving it: anything that renews these unattended has to hold a password that would let it do considerably more than renew them. That is a decision to take deliberately, not one to arrive at by writing a timer.
• And a settings file on one of our machines was readable by anyone with an account on it, while holding several of the site’s passwords. It predates this work and we found it while adding one more. It is now restricted.
• Nothing to do at your end.

Earlier releases

This page carries the newest releases. Everything before 2.236 is organised into version bands, listed in the release archive. Links shared to any earlier release still work; they are forwarded automatically.