We had backups of every machine behind this site. What we did not have was a single machine anyone had ever restored. Those are two different claims, and we had been writing them down as one.
So we took a server that was doing no work, erased its disk completely, and put it back from the backup alone. It came back — first time, with nothing needing repair. On the way, six of our own tools told us the wrong thing, and every one of them was working correctly. This is an account of all of it, including the parts we got wrong.
Note
The first half of this page needs no technical background. From "What a file-level backup does not contain" onward it assumes you have administered a Linux server. The tool is Proxmox Backup Server; the machine was Ubuntu on LVM. All device names, volume names and addresses here are generic examples, not ours.
Why this was worth a day
A backup that has never been restored is the textbook example of a check that cannot fail. It succeeds every night. It reports success in a log nobody reads. And the single occasion on which its success actually matters is the one occasion nobody has rehearsed.
We had also just been caught by a related failure: a note of ours described a machine as protected by a piece of software that had never been installed. The plan had been written in the past tense before it was carried out, and then the plan changed for good reasons — leaving behind a sentence that read like a completed verification. An intention recorded in the past tense is indistinguishable from a measurement, so nobody went back to check.
That is what pushed us from "we should test a restore some time" to doing it that evening.
Important
What this proves, and what it does not. We restored one machine, from a disk failure, with its
firmware settings intact, using an LVM layout with a separate /boot. We did not test a
motherboard replacement, where the firmware's own boot entry is gone too. We did not test the
simpler partition layout, though the harder case implies it. And one thing did not come back
working — see the end. Stating the boundary is the whole point; a restore procedure that claims
more than it has done is the problem we were trying to solve.
What a file-level backup does not contain
This is the part that decides everything else, and it is easy to miss.
A file-level backup contains files. It does not contain the partition table, the LVM metadata, or the
filesystem UUIDs. What it does contain is /etc/fstab — which refers to all three.
So those identifiers are not documentation. They are inputs to the rebuild, and they are readable only while the disk still exists.
The obvious plan is to recreate each filesystem with its original UUID, so the restored fstab is
correct without editing. That plan is right for /boot and the EFI partition, and wrong for the
root filesystem, which caught us:
/dev/disk/by-id/dm-uuid-LVM-<VG_UUID><LV_UUID> / ext4 defaults 0 1
/dev/disk/by-uuid/<uuid> /boot ext4 defaults 0 1
The root line does not name the filesystem's UUID at all. It names the volume group and logical
volume UUIDs, concatenated with their dashes stripped. The root filesystem's own ext4 UUID appears
in fstab nowhere.
Recreate the volume group with the right names and fresh UUIDs and you get a machine that cannot find its own root filesystem. The fix is LVM's own disaster-recovery path, and it only works from a dump taken before the disk is erased:
vgcfgbackup -f /root/vg-backup.lvm some-vg # BEFORE the wipe
# ... later, on the rebuilt disk:
pvcreate --uuid <PV_UUID> --restorefile /root/vg-backup.lvm /dev/nvme0n1p3
vgcfgrestore -f /root/vg-backup.lvm some-vg
There is a matching trick at the other end. A sfdisk -d dump carries the disk GUID and every
partition UUID — and a UEFI boot entry refers to the EFI partition by partition UUID. Restore the
table from that dump and the firmware's existing boot entry stays valid, so the machine boots with no
bootloader work at all. That is exactly what happened.
Capture what you cannot read from a wiped disk
Before erasing anything:
sfdisk -d /dev/nvme0n1 > partition-table.sfdisk # disk GUID + every partition UUID
vgcfgbackup -f vg-backup.lvm some-vg # PV, VG and LV UUIDs, and the extent map
cp /etc/fstab .
blkid > blkid.txt # filesystem UUIDs
efibootmgr -v > efibootmgr.txt
Important
Run every one of these as root. pvs, vgs and lvs as an unprivileged user print nothing
and exit successfully — which is indistinguishable from a machine that has no LVM at all. That is
the first of many instruments here that fail toward a confident, wrong answer.
Write these into the filesystem you are about to back up, so the recovery notes end up inside the archive as well. Then take a copy off the machine, because you need them before you can extract anything.
The restore
With the disk erased, in order:
- Rebuild the partition table from the dump —
sfdisk /dev/nvme0n1 < partition-table.sfdisk. Verify by dumping it again and diffing against the original; that checks the disk GUID and every partition UUID in one command, which is better than reading hex off a screen. - Rebuild LVM with
pvcreate --uuid --restorefile, thenvgcfgrestore, thenvgchange -ay. Confirm the logical volume's UUID matches the capture — if it does not, stop. - Create the filesystems with their original UUIDs.
mkfs.ext4 -U <uuid>, and for the EFI partitionmkfs.vfat -i <volume-id>— note the volume id is written without the dash thatfstabandblkidshow. That single character produces a machine that boots and then cannot mount its EFI partition. - Mount the target tree, parents first, before extracting. If the backup was taken with the
boot filesystems explicitly included, they live inside the archive at those paths. Extract into an
unmounted target and they land on the root volume, where the later mount hides them — leaving an
empty
/boot, no kernel, and every command reporting success. - Extract, with the client told that existing directories are expected. The mount points must pre-exist for step 4, and without that flag the extractor stops at the first one.
- Boot it before repairing anything.
That last step is worth insisting on even when you are fairly sure it will fail, because the two
failure modes tell you different things. "No bootable device" means the firmware → partition UUID →
EFI partition chain broke. Dropping to an initramfs prompt means firmware and the bootloader worked
and the root path failed, which points at the LVM UUIDs. Reinstalling the bootloader first hides
which of those you had.
We ran it expecting to need the repair step. It booted straight to a login prompt.
Six tools that told us the wrong thing
None of these are bugs. Each tool answered a question slightly different from the one we were asking, and in every case the check that told the truth was already available and cost nothing.
A timer that reported a run that never happened
systemctl list-timers showed a LAST value three hours old for our backup job. The job had never
run once.
systemd writes a timestamp file when a persistent timer is enabled — that is precisely what
stops it firing for occurrences it "missed" before it existed — and list-timers derives its LAST
column from that file. An enable is indistinguishable from a trigger.
It fails toward the reassuring answer. The line reads as "backed up three hours ago", while
PASSED: -, the honest column, looks like missing data.
The discriminator is
journalctl -u <service>. A timer's own reporting cannot answer a question about the service it triggers.
"0 logical volumes now active", while something still held the disk
vgchange -an reported success. blkdiscard then refused with Device or resource busy, which
reads like a protected device rather than a held one.
The device-mapper node was still there, most likely because LVM's own auto-activation had raced the manual deactivation on a live installer.
vgchange's message describes LVM's model.dmsetup ls, and/sys/block/<disk>/<partition>/holders/, describe the kernel. Trust the second.
The wipe worked, and lsblk said it hadn't
After a successful full-device discard, lsblk still showed the old partition table and the old
logical volume. It reads the kernel's cached table; discarding blocks does not ask the kernel to
re-read it.
Anyone checking their wipe with lsblk concludes it failed and runs it again — or, worse, decides the
disk was never touched.
pvscanandblkidread the media.pvscanfinding no physical volume is also the check that proves the LVM restore has real work to do, rather than quietly finding the old metadata still sitting there and appearing to succeed.
partprobe told us to reboot, and it was wrong
partprobe could not remove a partition that the device-mapper node was still holding, and said
"You should reboot now before making further changes."
The actual fix is dmsetup remove first, then partprobe. No reboot. A tool confidently recommending
something slower and less correct than the real answer is a category worth watching for.
A warning that means the command is working
pvcreate --uuid <uuid> --restorefile <file> prints:
WARNING: Couldn't find device with uuid ...
Physical volume "/dev/nvme0n1p3" successfully created.
The warning is true, and it is the entire reason you are running the command: no device currently carries that identifier, because you are about to write it. It would be worrying if the warning were absent.
The success line is also proof the identifier was entered correctly — a typo fails outright with
Can't find uuid ... in backup file, not with a warning.
The one that would have cost us: an error that is word-for-word a documented success
This is the dangerous one.
Our backup server's per-machine credentials are deliberately scoped so that each machine can reach only its own backups. Our own notes document the proof of that scoping: point the credential at somewhere it should not reach, and confirm it is refused. The refusal message is recorded as the sign that everything is set up correctly.
Deleting a credential also silently removes its permissions, and recreating it does not bring them back. The error that produces is character-for-character the documented success message, except for the final word — the name of the thing being refused.
So the failure mode is: your machine dies, you recreate its credential, you try to restore, you get an error, you check the runbook, you find that exact error described as the expected healthy result, and you conclude the scoping is working while being completely locked out of your own backups. At three in the morning.
A failure whose signature is a documented success signal is worse than one nobody has written about at all. We now recreate credentials in three steps rather than two, and the third — reapplying the permissions — is marked as not optional.
What we changed afterwards
We turned the procedure into a program. Copying long identifiers off a screen, under pressure, where one wrong character produces something that looks fine and will not start, is a defect waiting to happen rather than a task to be careful about. The program reads them out of the capture files itself. It also refuses to extract into a target that is not empty, because a failed extraction leaves the tree half-written — and since the archive is walked alphabetically, the retry then fails on its own leftovers, at a point with no relation to the original cause. Chasing that leads you through one plausible flag after another, none of which is the problem.
Its own self-test found two real faults in it before it was used for anything. Both would have produced a confidently wrong restore.
We corrected two of our own time estimates, which were wrong the same way — we had timed a slow phase of a long job and assumed the rest would proceed at that rate. One job we projected at most of a day finished in about two hours. Our own notes warn against exactly that error a few paragraphs above where we made it.
The one thing that did not come back
The restored machine had exactly one failed service: swap.
$ ls -ls /swap.img
4 -rw------- 1 root root 8589934592 ...
$ swapon /swap.img
swapon: /swap.img: skipping - it appears to have holes.
That first column is allocated blocks — four kilobytes of them, against an apparent size of eight gigabytes. The archive had restored the swapfile as a hole, and the kernel will not swap to a file with holes in it.
Nothing was broken. The file's metadata came back perfectly, timestamp and all; a swapfile that has never been written to is entirely zeros, the backup stored those zeros as the absence of data — which is exactly what a backup should do — and the restore faithfully reproduced an absence.
And it is not a quirk of an untouched swapfile. On a machine that has actually swapped, the pages that were used come back as real data and the pages that were not still come back as holes. You get a file that still has holes, and it still fails.
Important
The tempting conclusion — exclude the swapfile from the backup — is the appealing half of the fix
and does not solve the problem. The restored fstab still refers to the file, so the boot still
fails; you have changed the error from "it appears to have holes" to "no such file". Recreating
the swapfile after a restore is what fixes it, and it is needed whether you exclude it or not.
Excluding it is a separate and reasonable decision about not writing the contents of a machine's
memory into backup storage. Two problems that look like one.
swapoff -a; rm -f /swap.img
fallocate -l 8G /swap.img && chmod 600 /swap.img && mkswap /swap.img && swapon /swap.img
swapon --show # check the outcome, not the exit code
Important
That works because this fstab names the swapfile by path, so mkswap minting a brand-new
identifier changes nothing. Every other line in it pins its filesystem by identifier — which is the
entire reason the rest of this restore was delicate. If yours pins swap by UUID= instead, this
recipe leaves you with a machine that still will not swap, and you want mkswap -U <original>
with the value from the blkid you captured before wiping.
We found that one command after writing the recipe down. It is a small illustration of the thing this whole page is about: a procedure that worked once is not the same as a procedure that works, and the difference is usually a condition nobody wrote down because on their machine it happened to hold.
The thing we would tell anyone else
Not "test your restores", which everybody already knows and nobody schedules.
Separate the two claims in writing. "We have a backup" and "we have restored from it" are different sentences with different evidence behind them, and collapsing them into "we're covered" is what lets years pass. Write them on separate lines, and let the second one sit there marked unproven until it is not. It is uncomfortable to look at, which is the point.
And when a check tells you something has failed, check the instrument before you believe it. Six times in one evening, the tool was right and the reading was wrong — and every single time, the measurement that told the truth was one command away and free.