Mindblowing stuff. Content-aware resizing. If you are unfamiliar with the idea, this video from SIGGRAPH 2007 lays it out plainly. And now it’s possible in a $2 iPhone app.
Liquid Scale by Savoy Software.
Frasier Spiers details conventions found in the iPhone OS from public information and demos.
http://www.flickr.com/photos/fraserspeirs/sets/72157623224262135/
http://vladalexa.com/apps/osx/magicprefs/
Nice little Preference Pane to push the Magic Mouse to the fullest ability of its potential as a multi-touch device.
http://github.com/millenomi/swapkit
Art thrives on its limitations. Brilliant new open source framework for iPhone OS that lets apps use the RAM as semi-persistent storage.
http://www.youtube.com/watch?v=7GM4Lt5k24s
If there’s any truth to this, Ballmer is as much the unstable psychopath many suspect he is.
Couple of confirmed points:
Anything else is pure speculation.
http://www.theregister.co.uk/2009/12/19/verizon_snuffs_google_for_bing/
The move is part of the five-year search and advertising deal Verizon signed with Microsoft in January for a rumored $500m.
Utterly despicable.
As soon as I posted my last article, I realized how completely insane my backup system is, and sought to simplify it. Let’s start with the biggest problem: 4×1TB disks in a single pool with no mirroring. This means 4x as likely to have a drive failure, and when I did, it would have been catastrophic. Furthermore, it was obvious that I didn’t place much confidence in ZFS, since all the data in that pool was being copied to a single bare drive. Insanity.
So I have gone back and refactored the whole shebang. Those 4×1TB disks are part of a 2TB ZFS pool, with the other 2TB as a full mirror. If any disk fails, replacing it is simple and no data is lost. Over time, I hope to be able to replace each of them with a 2TB disk (4×2TB = 8TB total, 4TB mirrored available).
The ZFS pool is divided into a few filesystems. First, the largest portion is dedicated to /home which is still kept rsynced with the Mac. But there’s also a local /mnt/zpool/backup that the rest of the Linux system is backed up to with 30 days of rotation. And 500GB set aside for an AFP share for Time Machine to use over the wireless.
That’s all. I trust the ZFS way more than I trust the disks not to fail, and when they do, I won’t be panicking. Time Machine works flawlessly, and whether I login as a local user on the Mac or in Linux, the same set of user data is available in ~.
I’m never fully happy with my backups. I’ve gone through many iterations, from burning CD-Rs then DVD-Rs to USB hard drives to finally (mostly) settling on Time Machine, which does an admirable job of getting nearly everything right. Time Machine satisfies the two main requirements of backups: through the use of HFS+ hard links to directories it makes the most efficient use of available storage space, and it does so automatically, reliably, and effortlessly. No one likes to backup any more than one likes to walk around everywhere carrying an umbrella. But when you need it, you’d do anything to have it.
So, after some tweaking, my entire system backup is at a point where I feel safe. I have protection from stupid goofs, from drive failure, and even some off-site redundancy.
First, a bit about my setup. I operate primarily on two computers. There’s a MacBook Pro where I do most of my reading, writing, development, entertainment, and real work. It’s my main computer. Then there’s a midsize scratch-built server running Slackware Linux. I have no real attachment to Slackware or any other distribution in particular, it’s just the simplest for me to install, get it up and running, and offers what I think is the best mix of package management and get-the-hell-out-of-my-way-and-let-me-admin-my-own-box-ness. Popular distributions that use apt or rpm tend to annoy me.
THE SETUP
The MacBook Pro has a 250GB hard drive. The server has a 60GB OCZ Solid State Disk, and 4 1TB drives. The OS, installed software, configuration files, and just about anything that I would not call “my data” is on the SSD and takes up just under 10GB. 3 of those 1TB disks are part of a ZFS pool. The 4th 1TB disk is separately mounted and allocated solely for backup purposes.
FIRST LINE OF DEFENSE
Most of the time, when I need to recover data, it is personal data — the sort of thing in a home directory — on the Mac. For this, I use Time Machine. From a Unix hacker’s point of view, Time Machine really is fantastic, and I wish that directory hard links were possible on most Linux filesystems. I would love to see a feature-complete rip-off clone of TM for Linux.
My Time Machine volume is not an external USB or FireWire hard drive. It’s actually a volume that resides on the isolated 1TB disk in the server. I achieved this by installing the Netatalk package for Linux and running an Apple Filing Protocol server daemon (afpd) included with it. Netatalk’s afpd is as simple as pointing it to a directory on your server, irrespective of the filesystem, and then adding a service entry to a multicast server like avahi so that OS X’s Bonjour will detect it automatically. But that shared volume isn’t exactly where the data is kept.
I had to create a sparsebundle type volume from OS X (using either Disk Utility.app or hdiutil) that sits on that AFP share. Then you tell Time Machine to use the AFP share, not the sparsebundle, for backups, and somehow it works. It’s still using the secondary volume-within-a-volume for Time Machine, but it does so without ever first mounting the AFP volume. To be honest I don’t know how that works, but it does work great. Every hour, when Time Machine has new data to sync, it mounts the ‘deep’ sparsebundle volume, and it alone, to do its work. Then it unmounts it when finished.
I set the size of that sparsebundle to 500GB, which leaves another 500GB on that isolated 1TB disk. I’ll get back to that later.
MANUAL RSYNC to ZFS
ZFS is a wonderful filesystem and if you don’t know about it, you really should go read on it. It’s a shame that it looks like Apple will never include support for it in OS X, and in Linux you have to use it through the FUSE driver. That hurts performance a bit, but for my purposes it doesn’t matter. I have 3TB of ZFS spread over 3 drives. Now, the main thing sitting on this data pool is my $HOME from the Mac. In fact, /home/gregday on the server is almost a direct clone of /Users/gregday on the Mac. I sync ~/Pictures to ~/Pictures, ~/Music to ~/Music, and so on. The only exception is that on the server, I have a ~/Backup that holds macbookpro: ~/Applications, ~/Library, and all of the Unix “dot files” in server: ~/Backup/dotfiles
For some reason it just bothers me that Applications and Library are practically useless in Linux and I think they should be set aside and marked as being there for backup purposes. But the rest of the media, music, pictures, is just data so it can be there as if that’s it’s normal place.
I keep this data in sync using an rsync script I wrote, but I have not yet created a launchd entry for it. I occasionally just run it manually.
Should something happen to the Mac, and Time Machine breaks too, I can rely on the ZFS pool to at least contain a full copy of my personal data. This is my second line of defense.
WHEN ALL ELSE FAILS
In case the Mac hard drive dies, Time Machine takes a dump, and ZFS decides to never mount again, all is not lost. I haven’t forgotten about the data on my Solid State Disk either. That’s the installation and configuration of my server, and a lot of that has taken a very long time to get just right. To protect it, I have a daily cron job that copies the contents of the SSD to a place on the remaining 500GB of the isolated 1TB. I’m currently keeping a full bare metal copy of the past 10 days, but there’s room for expansion if I get more paranoid. And since there’s room, my daily backup job also rsyncs the backup of my home on ZFS to a second backup on the 1TB. But due to space, only 1 copy. After all, this is a backup of a backup, which is also backed up on Time Machine too.
This is all a bit confusing by now, so let’s recap clearly:
MacBook Pro:
- All contents Time Machine’d wirelessly to server on single drive.
- Home periodically manually rsynced to server on ZFS pool.
Server:
- OS backed up daily with 10-day rotation to single drive.
- Most recent instance of ZFS pool contents rsynced to single drive.
This is working remarkably well, and it’s surprisingly hands-off given the level of apparent complexity. And just to go that extra mile, some very critical data is also copied to my USB flash drive keychain and also copied to an off-site server co-located at my employer.
For the past 2 weeks I’ve been working on a script that uses the free software project apcupsd to regularly poll an APC UPS, collect statistics (Line Voltage, Battery Load, Time Remaining) and then create graphs from this data using gnuplot.
If you have a typical Linux system with all the standard tools (bash, grep, awk, sed, tac) and gnuplot installed with libpng support (you probably do) then it should work for you. First, make sure you are running apcupsd and that the “apcaccess” program works for communicating with your UPS. If it does, all you need to do is tweak a few options in the script and setup a cron job to run it at whatever interval of time you’d like your graphs to cover. I currently keep 4032 entries for each data point, run at 5 minute cron intervals which comes out to 1 week’s worth of data.
The graphs are PNG images, intended to be displayed with a web server, but if you just want the images, comment out the HTML portions of the script. I have tried to make it easy to understand and modify.
To see a live example of my UPS data, go to airwaterunix.org/ups.
And here’s the script: http://airwaterunix.org/makegraphs.sh