At one place I remember that we had the usual scheduled backups for our MySQL databases. We also wanted a quicker way to bring systems back with less data loss, so we also had a lagged replica that purposely applied changes delayed by I think it was about 4 hours. The other handy thing was that swapping a replica for the write leader was a common operation that we did whenever we upgraded hardware or software versions so it was a well exercised procedure.

Lagged replication is truly a great feature. Nowadays, on most systems we run one instant replica and one with 24 hours delay.

W. Curtis Preston, author of the O'Reilly book "Backup and Recovery" has said:

> Backups are useless. Restores are priceless.

I also like:

> The state of a backup is unknown and unknowable, until such time as it has been tested with a restore.

Sounds legit. This goes double on the cloud. In my early days of using AWS, it was so easy to create and destroy EC2s, I created some EC2s for testing - then terminated them... then realized I terminated the wrong ones.

Whoops

human error is here. creating a backup and forgetting the password for example or mistyping it

so true, had a zfs nas in 2009 with raidz, snaphots, everything, yet I did delete personnal stuff by mistake.