There is an uncomfortable line that circulates among sysadmins: a backup that has never been restored is not a backup, it is a hope. It sounds like an exaggeration until you have to find out on the bad day.
Why it fails exactly when you need it
The problem is not that backups do not run. They almost always run. The problem is that taking the backup and being able to recover it are two different things, and only the first one verifies itself.
The failures that only surface on restore are always the same: the backup completed but the file was corrupt; the database dump was taken with the machine running and came out inconsistent; the system was copying a folder that was no longer the data folder because the application changed paths months ago; the backup was fine but nobody knew the procedure and six hours were lost improvising.
None of these problems shows up in a successful backup report. All of them appear the first time you restore.
What a real test must prove
A useful restore test is not opening the backup and seeing files inside. It has to answer four questions with data:
- Does the full system come back? Booted, not just decompressed.
- How long does it take? Timed. That number is your real recovery time, and it is for setting expectations before a crisis, not during one.
- Is the data consistent? Databases above all. That they start and answer a query.
- Is the procedure written down? Because on the day it is needed, whoever knows it by heart may not be available.
How to do it without risk
The golden rule: always restore to a new destination, never over the original. A different identifier, and with the network interface disconnected.
That last detail matters more than it seems. A restore keeps the original network configuration, so if you bring it up connected you will have two machines fighting over the same IP and you will break production while running a safety test. Bring it up with the network down, verify from the console, and destroy it when done.
How often
Quarterly is a good starting point for most. And always, without exception, after a relevant change: a migration, a major upgrade, a change in data structure or in the backup policy. Changes are precisely what breaks backups without warning.
An example with numbers
Our latest documented test, run on a real server rather than a lab case: full restore from our primary data centre in 33 seconds, and from the second data centre in 58 seconds, both times onto a physical node different from the original.
What is interesting about those numbers is not that they are low. It is that they exist: when a customer asks how long a recovery takes, the answer is not an estimate, it is a measurement.
What you should ask your provider
If you have backups with someone, the question is direct: when was the last real restore you performed, and how long did it take? A provider who tests has the figure at hand. One who does not improvises a generic answer.
How we do it
In our backup services we document restore tests with date, source, destination and measured times. And on the managed tier, we run the periodic test for you and hand you the report. All inside our two data centres in Andorra.