Lucid Nonsense


Time Machine, Drobo and Sparse Disk Images

Tuesday, 28 April 2009

In my post about the DroboPro I mentioned that due to the way that the Drobo lies about it’s storage space there are a couple of tricks that are worth doing to prevent things from going “odd”. This applies to both the Drobo and DroboPro as it’s a result of the company’s BeyondRAID system which is used by both devices.

Drobos utilise a technique called thin provisioning to allow for the one of the Drobo’s headline features: being able to easily expand storage with little hassle. The immediately visible side of this is that the Drobo’s Dashboard utility will present the Drobo to the OS as a drive with a certain size, by default 2TB, even if the unit doesn’t contain that actual amount of physical storage. The reason for this is pretty simple even if the technique is fairly unusual in consumer drives. If you install two 500GB drives in your Drobo the data protection techniques the Drobo uses leaves you with 500GB of available storage. If you insert a third 500GB drive the drobo automatically increases the available storage to 1TB. If the disk size had been initially set to 500GB, a new volume would have to be created, or the old volume changed manually, for that space to become available. By setting the volume size to a larger capacity, the Drobo can increase and decrease the capacity available without the user or system having to be aware of it and without the drive having to be taken offline. Mac OS X expects drives to stay the same size, so by lying to the OS the Drobo can change the amount of storage elastically and automatically. This is quite neat although you need to get your head around it as it is different from most consumer drives on the market and even most high-end RAIDs.

This does lead to a problem. With a default Drobo setup the OS thinks the Drobo is a 2TB drive. The Drobo may only have 1TB available, so what happens if you try to copy 1.5TB to the Drobo? The OS will let you do this because, as far as it’s concerned there’s enough space on the drive, even if there isn’t. The Drobo gets around this by slowing down data transfers to a crawl once the storage used is over 95% of the real capacity, the capacity that the Drobo has installed. It then flashes it’s LEDs and pops a warning up via the Drobo Dashboard saying, effectively, “give me more drives”. Throw a new SATA drive into the Drobo, it does its stuff and as there is now more real storage available it removes the speed restriction.

This is, in general, an acceptable compromise. People normally fill a drive up reasonably slowly, and when they hit the 95% mark it becomes painfully obvious that something is “up”. The Drobo really does get that slow in order to get your attention. What’s more, a lot of people will never fill a hard drive to 95% when they’re manually copying files to it. It takes a long time to fill even a 500GB drive with general use and you generally have a rough idea of how much you’re putting on there and Drobo Dashboard gives you a handy menubar based idea of the real capacity remaining. Time Machine causes a problem here though because it is designed to operate without user interaction, and it’s designed to completely fill a drive, at which point Time Machine “recycles” space by replacing older backups. The Drobo gets in the way of this because Time Machine will happily start a backup that it can’t complete because as far as the OS is concerned there is enough space on the volume to finish. Time Machine doesn’t know it should replace older backups because it still thinks there’s plenty of room left to carry on. The Drobo will simply slow the backups down until something goes wrong or the user notices and sticks a bigger drive in. This can lead to backups going on for days and never completing and because of the length of time that the backup can go on for the chance of corruption increases. With the design of Time Machine aiming to reduce user interaction you are also far less likely to notice that the backups are slowing down. If you have the Drobo either under the desk, or connected somewhere on the network to an Airport Extreme or Mac, you may also not see the flashing LEDs and blue capacity lights. Drobo Dashboard only works when the Drobo is directly connected to a Mac as well so that’s potentially another warning you may not see. There’s nothing much more annoying than to have a backup system that you think is working fine yet to suddenly discover an important file has gone forever because the backup last Saturday is still going on, just incredibly slowly, and missed that file. Or even worse that the backups were going on for a fortnight and became corrupted just before your hard drive died.

So, it’s worth taking a few precautions if you want to use a Drobo as a Time Machine destination. There are a couple of options.

  1. Use the same amount of physical storage as the drive size set by Drobo Dashboard. For example with the default 2TB size, use three 1TB drives which gives you 2TB of usable space. The OS sees 2TB and the Drobo has 2TB, so no problems there. You do lose a bit of flexibility with that though; what happens if you want to add more storage? You would have to set your Drobo to use the larger virtual drive size (you can set it at up to 16TB), copying your data off and back on in the process. Without doing this the Drobo would give you access to the additional storage by creating another 2TB “drive”.

  2. Manually partition the Drobo so that one volume is less than the total real capacity of the Drobo. For instance assuming the Drobo has 1.5TB of storage, and has been set up as default to show 2TB to the OS, the Drobo Dashboard will, by default, configure that as a single 2TB virtual drive (although as far as the OS concerned this is just a normal external disk). You can happily go into Disk Utility and partition this into two 1TB volumes. If you use one of those for Time Machine, it will initially be used without causing any problems as Time Machine will only ever use 1TB, or 75%, of the available capacity. This doesn’t entirely solve the problem of course, if you manually fill the other partition to 1TB, your Time Machine partition will only have 500GB of real storage available even though the OS will think that there is 1TB free. Back to square one, although at least this way you are more likely to see the Drobo slowing down, and can solve the problem more easily by removing some data from your non-Time Machine volume. This method does mean that you are stuck with the initial size of the partitions you set up because Drobo do not support re-sizing partitions that contain data. Leopard will let you do this and for most drives it’s not a problem, but according to Drobo, doing this will cause data loss.

  3. Create a sparse image. This method is the best option because it lets you limit the amount of data your Time Machine backups use but gives you the option of changing that size in the future if you add more storage to the Drobo. Instructions for this method are below…

Sparse Image Backups

The easiest way to use a sparseimage file for Time Machine is to use Time Tamer. That will create a sparseimage file on your Drobo that is limited to twice the size of your Mac’s internal hard drive. It’s a quick and simple solution and it works pretty well.

For a bit more flexibility, you can manually set the size of the sparse image. The first thing you need is the MAC address of the Mac you’re using, so open the Terminal and enter:

ifconfig en0 | grep ether

That will give you the MAC address which will be 12 hex characters in pairs separated by colons. Write that down omitting the colons, so you should have something that looks like 002332aabbcc. This is the MAC address of your primary ethernet port, this is required even if you primarily use Airport or plan to use the backups over a different network port. The only MAC this will be different for is the MacBook Air, in that case you can use the same command, but the MAC address returned will be for the Air’s Airport card.

Once you’ve got that address you can add it to this command to create the sparseimage:

hdiutil create -size 1000g -fs HFS+J -volname "Backup of ComputerName" 
/Volumes/Drobo/ComputerName_002332aabbcc.sparsebundle

The above command should be entered as a single command on one line. That command will create a sparseimage file restricted to 1000GB, on the Drobo. Just replace the ComputerName and MAC address with your versions. Then simply point Time Machine, via System Preferences, to use that volume for backups. It will automatically use the sparseimage. The big advantage of this method is that you can later resize the image with the command:

hdiutil resize -size 2000g /Volumes/Drobo/ComputerName_002332aabbcc.sparsebundle

Which would, for instance, double the backup space available to Time Machine. You just need to make sure that you keep the size of the sparseimage below the 95% real capacity of the Drobo, and don’t fill the rest of the space up to go above that limit. Using this method at least you won’t automatically run out of space and it does make it easier to keep an eye on things.

Not Just for Drobos

This method also lets you limit the amount of space used by backups on any drive. Drobos are particularly problematic because of the often inaccurate storage size reported to the OS, but the above process is just as useful if you simply want to limit the amount of storage that Time Machine is using on your LaCie, Maxtor, Seagate etc. For other drives partitioning is more of an option as it’s possible to resize partitions on “normal” drives but the sparseimage method is effective and very flexible. Furthermore the same method will happily work on Time Capsule and Airport Extreme connected disks, or even the Time Capsule’s internal drive. All network based Time Machine backups already use sparseimages, so there is one less step involved in setting this up. It’s a useful trick where you don’t want to solely use the storage for Time Machine backups and would like a bit of space left over for other uses.

The easiest way to limit these backups is to let the first backup run over the network as normal, once that’s done mount the network volume on your Mac so you can see the sparsebundle file for your machine, it will be labelled with the computer name and MAC address. Then just use the same command as before to set your size limit, changing the names and size as appropriate.

hdiutil resize -size 2000g /Volumes/TimeCapsule/Name_002332aabbcc.sparsebundle

Possible Problems

There are, as ever, a few things that can go a bit wrong with using sparseimage backups. In general they’re very reliable and being able to resize the storage easily is a big advantage. Under 10.5.5 resizing sparsebundles fails, so make sure you’re running 10.5.6 as a minimum. Secondly under 10.5.6 depending on the name of the computer (set in System Preferences > Sharing), the Time Machine GUI may show no backups. The backups will have been successful, and you can manually browse them in the Finder, but you won’t be able to use the shiny space-themed recovery method. This is annoying but not critical and doesn’t happen with every machine. Generally if you keep your computer’s name short and without any odd or unusual characters that seems to avoid that problem. If you want to use this method over a network make sure that you have the latest firmware for the Airport Extreme or Time Capsule and are again running 10.5.6 on your Macs. With the newer versions of OS X Apple have gradually added a lot of tricks to make network backups work reliably. The latest software versions in particular do a very good job of making sure that backups are always in a consistent state, even with laptops that may be put to sleep, or leave the network, in the middle of a backup. If you’re backing up to a share hosted on OS X Server make sure both the server and clients are running 10.5.6.

There is a further problem with Time Machine backups of Mac OS X Server itself. Time Machine is fine for backing up data stored on the server, it’s actually quite good, but it will not make a 100% usable backup of the server itself and it’s best to use Time Machine with an alternative such as a periodic clone of the system. I’ll write something about that later, this entry is already long enough and at risk of drifting even further off course.


Previous Entry: "Patience"

Next Entry: "Apple's Little Hobby"