Data backup on Web Hosting Servers

[upback]

When considering a backup strategy for your servers, there are quite a few options, from the control panel integrated tools to third-party applications. However the most convenient way is using the snapshots provided by the hypervisor (if your server is not a physical box of course).
Wondering about your choice and why.

 

[Greenhost.cloud]

Snapshots are useful for creating short-term restore points, but they may not be sufficient if there is a hardware failure on the server. In addition to using snapshots, I also regularly backup server data to other servers for long-term storage. This ensures that in the event of a hardware issue on any server, the data is still accessible on another server.

 

[Saranya Subramaniam]

Hi..
Use Hypervisor Snapshots for quick recovery and Third-Party Backup Solutions for comprehensive, long-term backups.

 

[SharedGrid]

There is no right or wrong answer here, you go for the solution that fits your business (or personal) needs.

Saying that, as a provider, we use per-website backups for Shared and Reseller hosting customers so we can restore individual websites or individual files. A block based 'backup' such as snapshots (A snapshot should not be used as a backup, by the way) doesn't make restoring individual websites or files more difficult.

We take block based backups for VPS customers, since we don't have access to their operating system to do anything else and these are for DR purposes rather than for general restoration.

 

[bubblehost]

The way I do it is to download everything automatically and put it in one big zip folder, ready to be imported if something fails.
I take backups of both MySQL databases and files for websites, and they are safely secured in a backup server.
Nice question!

 

[OffshoreRacks]

Whatever strategy it's taken make sure the backup is in a remote location, not within the same rack or data center, and then test this backup in a test environment, by restoring them occasionally. We usually take both 1 daily local for fast restores at 10Gb/s and then remote where it's not taken daily. You can also add replication which is not a backup but a way to be back faster than a restore in case of disaster.

 

[systeminplace]

My backup strategy has changed over the years and I will share my horror story so you can see why.

Back in 2020 I thought I had a great backup solution. I setup a home server which had two separate raid cards with two separate JBOD enclosures.

I used JBOD one for my primary data and I mirrored that to JBOD 2. I had a backup rsync running over the network each night to another servers JBOD.

Of course, my rack was installed in the worst possible place and positioning so that at the time I could not fully access the back of the rack.

I was using a VRTX enclosure, which for those who are not familiar, runs (4) blades and has multiple PCIe slots for passthrough. This enclosure has a feature where PCIe slots can be turned off so they can be removed or added without taking down the other running systems.

My first raid card was throwing errors so I slid the chassis out to replace it. Reaching behind it, I thought I was unplugging the SFF cables from it. Suddenly my phone started chiming that raid 2 was offline. I look further to see that one of the SFF cables I unplugged was from the wrong raid card. In a panic, I shutdown the second blade and re-plugged the SFF cable and rebooted the machine.

That's when things took a horrible turn. Raid 2 was now offline and degraded. All data was unreadable and two drives were now offline. Unfortunately they were from the same mirror.

With raid 1 down to replace the raid card, I assured myself that everything would be ok. I installed the new raid card and booted the server. On boot, two drives would not spin up. My panic quickly turned to horror as raid 1 was now offline and degraded. I go into the controller again to see... yes, both drives from the same mirrored pair. Now realizing I just lost both of my raids I log into my third backup only to find I had not updated the keys and it was backing up 0KB archives...

My panic quickly turned to horror as I realized I just lost 4 years of my kids photos. Multiple TB of data gone in an instant but the only thing that could not be rebuilt or replaced was gone..... Anything not backed up to google photos was gone forever.

Learn from my lesson and plan appropriately.

Now, my strategy is this:

Snapshots are great but I use them as a quick oops I need a quick restore to fix something. I do not use them as a full backup because too many things can go wrong.

The old backup strategy 3-2-1 is great, but becoming more outdated. I would now consider this the BARE minimum of a good backup plan. (3) copies of your data, (2) onsite and (1) offsite. I would go further saying it needs to be (2) on site copies on FULLY SEPERATE HARDWARE. My current backup plan is 3-3-2-1. (3) Backups, (3) Separate machines, (2) Locations, (1) local copy/snapshot on host machine for quick restore. Also, RAID is not a backup, its just one way to bring fault tolerance to a copy of your data.

NEVER consider a backup on the same hardware as an actual backup. When your raid card dies, or your array holding your primary information degrades or you accidentally unplug the wrong cable... everything in that server goes poof, its all gone. I now consider a copy of your data on the same server as a convenience, not a backup (in my eyes).

If you do not have the ability to do this then I suggest:

1. Create a copy of your data locally on your webhost.
2. Use a cheap NUC to backup daily to your home, use a USB drive for space if needed.
3. Create a cheap VM at a low cost host to store minimum 3-7 days backup offsite.

This ensures that even if you forget and one of the 3 backups stop working, you still have 2 backups.

Also another thing, periodically check your backups. Nothing is worse than needing your backups only to find your archives are 0KB and have been failing to login for the last month because your changed your password or your keys. Write a script to check the file sizes of the backups and have it email you if sizes change by more than the set amount. I have mine set to email me if a backup is now 5% smaller or larger in size than the previous backup, or if that backup is 0KB in size.

In the end your backup plan should be equal to the value of your data.

Can you afford to live without that data? Can you rebuild that data cheaper than the system to back it up costs? Are you backing up files which can easily be found and redownloaded from the internet?

Backup, Backup, Backup.