Overly Complex Backup

I’m never fully happy with my backups. I’ve gone through many iterations, from burning CD-Rs then DVD-Rs to USB hard drives to finally (mostly) settling on Time Machine, which does an admirable job of getting nearly everything right. Time Machine satisfies the two main requirements of backups: through the use of HFS+ hard links to directories it makes the most efficient use of available storage space, and it does so automatically, reliably, and effortlessly. No one likes to backup any more than one likes to walk around everywhere carrying an umbrella. But when you need it, you’d do anything to have it.

So, after some tweaking, my entire system backup is at a point where I feel safe. I have protection from stupid goofs, from drive failure, and even some off-site redundancy.

First, a bit about my setup. I operate primarily on two computers. There’s a MacBook Pro where I do most of my reading, writing, development, entertainment, and real work. It’s my main computer. Then there’s a midsize scratch-built server running Slackware Linux. I have no real attachment to Slackware or any other distribution in particular, it’s just the simplest for me to install, get it up and running, and offers what I think is the best mix of package management and get-the-hell-out-of-my-way-and-let-me-admin-my-own-box-ness. Popular distributions that use apt or rpm tend to annoy me.

THE SETUP
The MacBook Pro has a 250GB hard drive. The server has a 60GB OCZ Solid State Disk, and 4 1TB drives. The OS, installed software, configuration files, and just about anything that I would not call “my data” is on the SSD and takes up just under 10GB. 3 of those 1TB disks are part of a ZFS pool. The 4th 1TB disk is separately mounted and allocated solely for backup purposes.

FIRST LINE OF DEFENSE
Most of the time, when I need to recover data, it is personal data — the sort of thing in a home directory — on the Mac. For this, I use Time Machine. From a Unix hacker’s point of view, Time Machine really is fantastic, and I wish that directory hard links were possible on most Linux filesystems. I would love to see a feature-complete rip-off clone of TM for Linux.

My Time Machine volume is not an external USB or FireWire hard drive. It’s actually a volume that resides on the isolated 1TB disk in the server. I achieved this by installing the Netatalk package for Linux and running an Apple Filing Protocol server daemon (afpd) included with it. Netatalk’s afpd is as simple as pointing it to a directory on your server, irrespective of the filesystem, and then adding a service entry to a multicast server like avahi so that OS X’s Bonjour will detect it automatically. But that shared volume isn’t exactly where the data is kept.

I had to create a sparsebundle type volume from OS X (using either Disk Utility.app or hdiutil) that sits on that AFP share. Then you tell Time Machine to use the AFP share, not the sparsebundle, for backups, and somehow it works. It’s still using the secondary volume-within-a-volume for Time Machine, but it does so without ever first mounting the AFP volume. To be honest I don’t know how that works, but it does work great. Every hour, when Time Machine has new data to sync, it mounts the ‘deep’ sparsebundle volume, and it alone, to do its work. Then it unmounts it when finished.

I set the size of that sparsebundle to 500GB, which leaves another 500GB on that isolated 1TB disk. I’ll get back to that later.

MANUAL RSYNC to ZFS
ZFS is a wonderful filesystem and if you don’t know about it, you really should go read on it. It’s a shame that it looks like Apple will never include support for it in OS X, and in Linux you have to use it through the FUSE driver. That hurts performance a bit, but for my purposes it doesn’t matter. I have 3TB of ZFS spread over 3 drives. Now, the main thing sitting on this data pool is my $HOME from the Mac. In fact, /home/gregday on the server is almost a direct clone of /Users/gregday on the Mac. I sync ~/Pictures to ~/Pictures, ~/Music to ~/Music, and so on. The only exception is that on the server, I have a ~/Backup that holds macbookpro: ~/Applications, ~/Library, and all of the Unix “dot files” in server: ~/Backup/dotfiles

For some reason it just bothers me that Applications and Library are practically useless in Linux and I think they should be set aside and marked as being there for backup purposes. But the rest of the media, music, pictures, is just data so it can be there as if that’s it’s normal place.

I keep this data in sync using an rsync script I wrote, but I have not yet created a launchd entry for it. I occasionally just run it manually.

Should something happen to the Mac, and Time Machine breaks too, I can rely on the ZFS pool to at least contain a full copy of my personal data. This is my second line of defense.

WHEN ALL ELSE FAILS
In case the Mac hard drive dies, Time Machine takes a dump, and ZFS decides to never mount again, all is not lost. I haven’t forgotten about the data on my Solid State Disk either. That’s the installation and configuration of my server, and a lot of that has taken a very long time to get just right. To protect it, I have a daily cron job that copies the contents of the SSD to a place on the remaining 500GB of the isolated 1TB. I’m currently keeping a full bare metal copy of the past 10 days, but there’s room for expansion if I get more paranoid. And since there’s room, my daily backup job also rsyncs the backup of my home on ZFS to a second backup on the 1TB. But due to space, only 1 copy. After all, this is a backup of a backup, which is also backed up on Time Machine too.

This is all a bit confusing by now, so let’s recap clearly:
MacBook Pro:
- All contents Time Machine’d wirelessly to server on single drive.
- Home periodically manually rsynced to server on ZFS pool.

Server:
- OS backed up daily with 10-day rotation to single drive.
- Most recent instance of ZFS pool contents rsynced to single drive.

This is working remarkably well, and it’s surprisingly hands-off given the level of apparent complexity. And just to go that extra mile, some very critical data is also copied to my USB flash drive keychain and also copied to an off-site server co-located at my employer.