--- title: "Proving a Backup Restores, and Six Tools That Told Us the Wrong Thing" description: "A bare-metal restore from Proxmox Backup Server, start to finish: why the LVM UUIDs matter more than the filesystem ones, and six tools that reported failure while succeeding." canonical_url: "https://www.truthpromoters.com/help/proving-a-backup-restores" type: "help" section: "Help Center" keywords: "bare metal restore, Proxmox Backup Server restore, vgcfgrestore, dm-uuid-LVM fstab, proxmox-backup-client restore, swapon it appears to have holes" --- # Proving a Backup Restores, and Six Tools That Told Us the Wrong Thing We had backups of every machine behind this site. What we did not have was a single machine anyone had ever *restored*. Those are two different claims, and we had been writing them down as one. So we took a server that was doing no work, erased its disk completely, and put it back from the backup alone. It came back — first time, with nothing needing repair. On the way, **six of our own tools told us the wrong thing**, and every one of them was working correctly. This is an account of all of it, including the parts we got wrong. > [!NOTE] > The first half of this page needs no technical background. From *"What a file-level backup does > not contain"* onward it assumes you have administered a Linux server. The tool is > [Proxmox Backup Server](https://pbs.proxmox.com); the machine was Ubuntu on LVM. **All device > names, volume names and addresses here are generic examples**, not ours. ## Why this was worth a day A backup that has never been restored is the textbook example of a check that cannot fail. It succeeds every night. It reports success in a log nobody reads. And the single occasion on which its success actually matters is the one occasion nobody has rehearsed. We had also just been caught by a related failure: a note of ours described a machine as protected by a piece of software that had never been installed. The plan had been written in the past tense before it was carried out, and then the plan changed for good reasons — leaving behind a sentence that read like a completed verification. **An intention recorded in the past tense is indistinguishable from a measurement**, so nobody went back to check. That is what pushed us from *"we should test a restore some time"* to doing it that evening. > [!IMPORTANT] > **What this proves, and what it does not.** We restored one machine, from a disk failure, with its > firmware settings intact, using an LVM layout with a separate `/boot`. We did **not** test a > motherboard replacement, where the firmware's own boot entry is gone too. We did not test the > simpler partition layout, though the harder case implies it. And one thing did not come back > working — see the end. Stating the boundary is the whole point; a restore procedure that claims > more than it has done is the problem we were trying to solve. ## What a file-level backup does not contain This is the part that decides everything else, and it is easy to miss. A file-level backup contains files. It does not contain the partition table, the LVM metadata, or the filesystem UUIDs. What it *does* contain is `/etc/fstab` — **which refers to all three.** So those identifiers are not documentation. They are **inputs** to the rebuild, and they are readable only while the disk still exists. The obvious plan is to recreate each filesystem with its original UUID, so the restored `fstab` is correct without editing. That plan is right for `/boot` and the EFI partition, and **wrong for the root filesystem**, which caught us: ``` /dev/disk/by-id/dm-uuid-LVM- / ext4 defaults 0 1 /dev/disk/by-uuid/ /boot ext4 defaults 0 1 ``` The root line does not name the filesystem's UUID at all. It names the **volume group and logical volume UUIDs**, concatenated with their dashes stripped. The root filesystem's own ext4 UUID appears in `fstab` nowhere. Recreate the volume group with the right *names* and fresh UUIDs and you get a machine that cannot find its own root filesystem. The fix is LVM's own disaster-recovery path, and it only works from a dump taken **before** the disk is erased: ```bash vgcfgbackup -f /root/vg-backup.lvm some-vg # BEFORE the wipe # ... later, on the rebuilt disk: pvcreate --uuid --restorefile /root/vg-backup.lvm /dev/nvme0n1p3 vgcfgrestore -f /root/vg-backup.lvm some-vg ``` There is a matching trick at the other end. A `sfdisk -d` dump carries the disk GUID and every partition UUID — and a UEFI boot entry refers to the EFI partition **by partition UUID**. Restore the table from that dump and the firmware's existing boot entry stays valid, so the machine boots with no bootloader work at all. That is exactly what happened. ## Capture what you cannot read from a wiped disk Before erasing anything: ```bash sfdisk -d /dev/nvme0n1 > partition-table.sfdisk # disk GUID + every partition UUID vgcfgbackup -f vg-backup.lvm some-vg # PV, VG and LV UUIDs, and the extent map cp /etc/fstab . blkid > blkid.txt # filesystem UUIDs efibootmgr -v > efibootmgr.txt ``` > [!WARNING] > Run every one of these as root. `pvs`, `vgs` and `lvs` as an unprivileged user print **nothing** > and exit successfully — which is indistinguishable from a machine that has no LVM at all. That is > the first of many instruments here that fail toward a confident, wrong answer. Write these into the filesystem you are about to back up, so the recovery notes end up inside the archive as well. Then take a copy off the machine, because you need them *before* you can extract anything. ## The restore With the disk erased, in order: 1. **Rebuild the partition table** from the dump — `sfdisk /dev/nvme0n1 < partition-table.sfdisk`. Verify by dumping it again and diffing against the original; that checks the disk GUID and every partition UUID in one command, which is better than reading hex off a screen. 2. **Rebuild LVM** with `pvcreate --uuid --restorefile`, then `vgcfgrestore`, then `vgchange -ay`. Confirm the logical volume's UUID matches the capture — if it does not, stop. 3. **Create the filesystems with their original UUIDs.** `mkfs.ext4 -U `, and for the EFI partition `mkfs.vfat -i ` — note the volume id is written **without** the dash that `fstab` and `blkid` show. That single character produces a machine that boots and then cannot mount its EFI partition. 4. **Mount the target tree, parents first, *before* extracting.** If the backup was taken with the boot filesystems explicitly included, they live inside the archive at those paths. Extract into an unmounted target and they land on the root volume, where the later mount hides them — leaving an empty `/boot`, no kernel, and every command reporting success. 5. **Extract**, with the client told that existing directories are expected. The mount points *must* pre-exist for step 4, and without that flag the extractor stops at the first one. 6. **Boot it before repairing anything.** That last step is worth insisting on even when you are fairly sure it will fail, because the two failure modes tell you different things. *"No bootable device"* means the firmware → partition UUID → EFI partition chain broke. Dropping to an `initramfs` prompt means firmware and the bootloader worked and the **root** path failed, which points at the LVM UUIDs. Reinstalling the bootloader first hides which of those you had. We ran it expecting to need the repair step. It booted straight to a login prompt. ## Six tools that told us the wrong thing None of these are bugs. Each tool answered a question slightly different from the one we were asking, and in every case the check that told the truth was already available and cost nothing. ### A timer that reported a run that never happened `systemctl list-timers` showed a `LAST` value three hours old for our backup job. The job had never run once. `systemd` writes a timestamp file when a persistent timer is **enabled** — that is precisely what stops it firing for occurrences it "missed" before it existed — and `list-timers` derives its `LAST` column from that file. An enable is indistinguishable from a trigger. **It fails toward the reassuring answer.** The line reads as *"backed up three hours ago"*, while `PASSED: -`, the honest column, looks like missing data. > The discriminator is `journalctl -u `. A timer's own reporting cannot answer a question > about the service it triggers. ### "0 logical volumes now active", while something still held the disk `vgchange -an` reported success. `blkdiscard` then refused with `Device or resource busy`, which reads like a protected device rather than a held one. The device-mapper node was still there, most likely because LVM's own auto-activation had raced the manual deactivation on a live installer. > `vgchange`'s message describes LVM's model. `dmsetup ls`, and > `/sys/block///holders/`, describe the kernel. Trust the second. ### The wipe worked, and `lsblk` said it hadn't After a successful full-device discard, `lsblk` still showed the old partition table and the old logical volume. It reads the kernel's **cached** table; discarding blocks does not ask the kernel to re-read it. Anyone checking their wipe with `lsblk` concludes it failed and runs it again — or, worse, decides the disk was never touched. > `pvscan` and `blkid` read the media. `pvscan` finding no physical volume is also the check that > proves the LVM restore has real work to do, rather than quietly finding the old metadata still > sitting there and appearing to succeed. ### `partprobe` told us to reboot, and it was wrong `partprobe` could not remove a partition that the device-mapper node was still holding, and said *"You should reboot now before making further changes."* The actual fix is `dmsetup remove` first, then `partprobe`. No reboot. A tool confidently recommending something slower and less correct than the real answer is a category worth watching for. ### A warning that means the command is working `pvcreate --uuid --restorefile ` prints: ``` WARNING: Couldn't find device with uuid ... Physical volume "/dev/nvme0n1p3" successfully created. ``` The warning is true, and it is the entire reason you are running the command: no device currently carries that identifier, because you are about to write it. It would be *worrying* if the warning were absent. > The success line is also proof the identifier was entered correctly — a typo fails outright with > `Can't find uuid ... in backup file`, not with a warning. ### The one that would have cost us: an error that is word-for-word a documented success This is the dangerous one. Our backup server's per-machine credentials are deliberately scoped so that each machine can reach only its own backups. Our own notes document the *proof* of that scoping: point the credential at somewhere it should not reach, and confirm it is refused. The refusal message is recorded as the sign that everything is set up correctly. Deleting a credential also silently removes its permissions, and recreating it does not bring them back. The error that produces is **character-for-character the documented success message, except for the final word** — the name of the thing being refused. So the failure mode is: your machine dies, you recreate its credential, you try to restore, you get an error, you check the runbook, you find that exact error described as the expected healthy result, and you conclude the scoping is working while being completely locked out of your own backups. At three in the morning. > **A failure whose signature is a documented success signal is worse than one nobody has written > about at all.** We now recreate credentials in three steps rather than two, and the third — > reapplying the permissions — is marked as not optional. ## What we changed afterwards **We turned the procedure into a program.** Copying long identifiers off a screen, under pressure, where one wrong character produces something that looks fine and will not start, is a defect waiting to happen rather than a task to be careful about. The program reads them out of the capture files itself. It also refuses to extract into a target that is not empty, because a failed extraction leaves the tree half-written — and since the archive is walked alphabetically, the retry then fails on *its own leftovers*, at a point with no relation to the original cause. Chasing that leads you through one plausible flag after another, none of which is the problem. **Its own self-test found two real faults in it before it was used for anything.** Both would have produced a confidently wrong restore. **We corrected two of our own time estimates**, which were wrong the same way — we had timed a slow phase of a long job and assumed the rest would proceed at that rate. One job we projected at most of a day finished in about two hours. Our own notes warn against exactly that error a few paragraphs above where we made it. ## The one thing that did not come back The restored machine had exactly one failed service: swap. ``` $ ls -ls /swap.img 4 -rw------- 1 root root 8589934592 ... $ swapon /swap.img swapon: /swap.img: skipping - it appears to have holes. ``` That first column is **allocated blocks** — four kilobytes of them, against an apparent size of eight gigabytes. The archive had restored the swapfile as a hole, and the kernel will not swap to a file with holes in it. Nothing was broken. The file's metadata came back perfectly, timestamp and all; a swapfile that has never been written to is entirely zeros, the backup stored those zeros as the absence of data — which is exactly what a backup should do — and the restore faithfully reproduced an absence. **And it is not a quirk of an untouched swapfile.** On a machine that has actually swapped, the pages that were used come back as real data and the pages that were not still come back as holes. You get a file that still has holes, and it still fails. > [!WARNING] > The tempting conclusion — *exclude the swapfile from the backup* — is the appealing half of the fix > and does not solve the problem. The restored `fstab` still refers to the file, so the boot still > fails; you have changed the error from *"it appears to have holes"* to *"no such file"*. **Recreating > the swapfile after a restore is what fixes it, and it is needed whether you exclude it or not.** > Excluding it is a separate and reasonable decision about not writing the contents of a machine's > memory into backup storage. Two problems that look like one. ```bash swapoff -a; rm -f /swap.img fallocate -l 8G /swap.img && chmod 600 /swap.img && mkswap /swap.img && swapon /swap.img swapon --show # check the outcome, not the exit code ``` > [!IMPORTANT] > **That works because this `fstab` names the swapfile by path**, so `mkswap` minting a brand-new > identifier changes nothing. Every other line in it pins its filesystem by identifier — which is the > entire reason the rest of this restore was delicate. **If yours pins swap by `UUID=` instead, this > recipe leaves you with a machine that still will not swap**, and you want `mkswap -U ` > with the value from the `blkid` you captured before wiping. > > We found that one command after writing the recipe down. It is a small illustration of the thing > this whole page is about: **a procedure that worked once is not the same as a procedure that > works**, and the difference is usually a condition nobody wrote down because on their machine it > happened to hold. ## The thing we would tell anyone else Not *"test your restores"*, which everybody already knows and nobody schedules. **Separate the two claims in writing.** *"We have a backup"* and *"we have restored from it"* are different sentences with different evidence behind them, and collapsing them into "we're covered" is what lets years pass. Write them on separate lines, and let the second one sit there marked *unproven* until it is not. It is uncomfortable to look at, which is the point. And when a check tells you something has failed, **check the instrument before you believe it.** Six times in one evening, the tool was right and the reading was wrong — and every single time, the measurement that told the truth was one command away and free. ## Related topics - [Technical Writings](https://www.truthpromoters.com/help/technical-writings) — Notes on the systems behind this site — what broke, what we learned, and what misled us