Categories
Home Lab Synology

I lost three years of data

Whelp, today I’m enjoying a nice slice of humble pie after losing three years worth of backup versions. While admitting failure is hard, I believe in owning mistakes and learning from them.

I began receiving notifications from DSM saying that a backup task was continually failing due to a lack of space a few days ago. The task was configured using Smart Recycle, so I thought, no biggie – I’ll just reduce the number of retained versions and move along. What I didn’t know was that the damage was already done…

When navigating into the Task Settings, Rotation was grayed out. That wasn’t going to bail me out.

Then when navigating into the Version List, there’s no “X” to manually delete versions. Not getting bailed out there, either.

Trying to discard the task only results in getting the error “Failed to discard backup task”. As a hail-mary, I tired rebooting DSM to see if a fresh boot would cleanup the suspended job. Nope.

So, I was faced with accepting the inevitable: The task and data needed to be deleted entirely… Luckily, I still have a remote backup, and two local offline cold backups, so this was more of a blow to the ego versus a true data loss. Moving forward:

Never let it happen again.

I’ve made two adjustments to keep myself out of this situation in the future.

The first step was to adjust the “Low capacity of USB disk” notification threshold from 5% to 20% for External storage, allowing for more reaction time.

The second step was reducing the maximum number of backup versions.

I left the version count set to the default of 256 when originally configuring the backup task, attempting to retain backups for as long as possible. Thus beginning to brew the perfect storm. The disk was just large enough to store 256 versions of the data set and was continually on the bubble. I thought that I was managing the ratio of free drive space to versions retained perfectly, but was unknowingly playing with fire. The moment that I added just enough data, the canoe tipped over.

There’s no perfect answer for how many versions are appropriate, but I’ve restarted by cutting it half to 128 and will keep a closer eye on things over the coming months. I’d love to see Synology update Hyper Backup to better handle these situations.

All in all, while this pie tastes bitter, my data is still totally safe thanks to a diversified strategy and I’m very happy with how the Synology system has been functioning.

Categories
Home Lab Synology

Backup vSphere virtual machines with Synology DSM

As I’ve slowly built out my home lab, I realized that while none of it is mission-critical, I’ve invested a lot of time educating myself and getting everything configured. Therefore – it should be backed up.

I began searching r/homelab for solutions and came across plenty of notable options (GhettoVCB, Veeam, Vertical Backup) until one, almost too obvious, recommendation slapped me in the face: Synology Active Backup for Business. I’ve become quite the fan of Synology products over the past few years. I’ve successfully deployed a fully-automated Synology powered backup solution for my personal and small business use. Admittedly, I’m a borderline fanboy at this point.

Active Backup for Business essentially allows you to backup data that isn’t already on your NAS: PCs, physical servers, file servers, and virtual machines. I’m going walk through backing up virtual machines to avoid loosing immeasurable amounts of time.

As of writing this post, Active Backup for Business supports:

  • VMware vSphere versions: 5.0, 5.1, 5.5, 6.0, 6.5, 6.7
  • VMware editions: 
    • VMware free ESXi
    • VMware vSphere Essentials, VMware vSphere Essentials Plus
    • VMware vSphere Standard, VMware vSphere Advanced
    • VMware vSphere Enterprise, VMware vSphere Enterprise Plus

See the official documentation for an exhaustive list of supported platforms and system requirements.

Before beginning, decide what volume and Share to store virtual machine backups on. I’ve created a new share aptly named “VM Backups.”

Install the Active Backup for Business package.

A window will appear asking to activate using a Synology Account, click Activate. A new window will open, login and complete the process.

Open Active Backup for Business, choose Virtual Machine, on the VMware vSphere Hypervisor tab, click Manage Hypervisor. The Manage Hypervisor window open, click Add. The Add Hypervisor window will open, fill in a vCenter IP address or FDQN, administrator account and password, click Apply.

For best security practices, create a dedicated vSphere role and account with the minimally required permissions to backup and restore a virtual machine. Reference the official Synology documentation here.

If vCenter is configured with a self-signed certificate, click Yes to trust the certificate. Close the Manage Hypervisor window. vSphere virtual machines will now be listed. I’m going to create a backup task for my lab Domain Controller, DC1.

On the VMware vSphere page, click Create Task. The next prompt will be to choose a shared folder to use as a backup destination. Choose a share to send backups to, click Next.

One or more multiple virtual machines can be chosen on the next screen, I’m sticking with one for the sake of this example. Click Next.

The next screen offers a ton of options to configure for the backup task. Read through them and choose the appropriate options for the scenario. The default option leaves Changed Block Tracking enabled, which will keep backup sizes reduced to deltas, saving space. Dig into Advanced Settings to run scripts on the virtual machine pre-freeze or post-thaw.

Click Next when finished.

Schedule the task as appropriate, click Next.

Adjust the retention policy as necessary; bear in mind that more retention means more disk space used. I dialed back the defaults a little, keeping a backup for up to 3 months.

When finished, click Next.

Choose which users have privileges to restore the virtual machine(s), then click Next.

Review all of the settings, click Apply.

Finally, click Yes to backup now or No to wait until the schedule fires the first backup task.

Checking the task log after completion shows a verbose log of the actions taken. Most notably, 14.3 GB, in 22 minutes, averaging 11 MB/s on my gigabit network.

Now to back up the rest and to see how the backup task performs over time. In the future, I’ll be working testing restores and shipping virtual machine backups to a remote Synology RackStation.

Full documentation on Active Backup for Business can be found on Synology’s website.