Matt Connolly's Blog

my brain dumps here…

Tag Archives: os x

ZFS = Data integrity

So, for a while now, I’ve been experiencing crappy performance of a Western Digital Green drive (WD15EARS) I have an a zfs mirror storing my time machine backups (using OpenIndiana and Netatalk).

Yesterday, the drive started reporting errors. Unfortunately, the system hung – that’s not so cool – ZFS is supposed to keep working when a drive fails… Aside from that, when I rebooted, the system automatically started a scrub to verify data integrity, and after about 10 minutes:

  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Thu Mar 10 10:19:42 2011
    1.68G scanned out of 1.14T at 107M/s, 3h5m to go
    146K resilvered, 0.14% done
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            c8t1d0s0  DEGRADED     0     0    24  too many errors  (resilvering)
            c8t0d0s0  ONLINE       0     0     0
        cache
          c12d0s0     ONLINE       0     0     0

errors: No known data errors

Check it out. It’s found 24 errors on the Western Digital Drive, but so far no data errors have been found, because they were correct on the other drive.

That’s obvious, right? But what other operating systems can tell the difference between the right and wrong data when they’re both there??? Most raid systems only detect a total drive failure, but don’t deal with incorrect data coming off the drive !!

Sure backing up to a network (Time Machine’s sparse image stuff) is *way* slower than a directly connected firewire drive, but in my opinion, it’s well worth doing it this way for the data integrity that you don’t get on a single USB or Firewire drive.

Thank you ZFS for keeping my data safe. B*gger off Western Digital for making crappy drives. I’m off to get a replacement today… what will it be? Samsung or Seagate?

Advertisements

ZFS for Mac Coming soon…

A little birdy told me, that there might be a new version of ZFS ported to Mac OS X coming up soon…

It seems the guys at Tens Compliment are working on a port of ZFS at a much more recent version than what was left behind by apple and forked as a Google code project: http://code.google.com/p/maczfs/

On my mac, I have installed the Mac-ZFS which can be found at the Google Code project. (I don’t have any ZFS volumes, it’s installed because I wanted to know what version it was up to.)

bash-3.2# uname -prs
Darwin 10.6.0 i386
bash-3.2# zpool upgrade
This system is currently running ZFS pool version 8.

All pools are formatted using this version.

My backup server at home is running OpenIndiana oi-148:

root@vault:~# uname -prs
SunOS 5.11 i386
root@vault:~# zpool upgrade
This system is currently running ZFS pool version 28.

All pools are formatted using this version.

Pretty exciting that we can get the same zpool version as the latest OpenIndiana… think of the backup/restore possibilities sending a snapshot over to a remote machine.

Using Git on Mac OS X

There’s a mac gui client called GitX, which lives at: http://gitx.frim.nl/

However, it’s a little out of date, in that it’s been forked on github many times, and brotherbard has released a version which is *way* more advanced in features.

I tried this one out: GitX Sep-20-2010.2.zip from here http://github.com/brotherbard/gitx/downloads – way cool.

Redmine: Ruby on Rails without Virtual Hosts

I’ve been playing around with Redmine, a Rails app, on Mac OS X 10.6. It’s been quite a pain to get set up. It seems that the only easy way to get passenger to work is with virtual hosts. It’s not really obvious or intuitive how virtual hosts work. After a lot of RTFMing the apache docs, and passenger google group, I got it working. But Virtual hosts are still a pain, because you have to set them up in your /etc/hosts file more for every machine on the network, and there’s still a high likelihood of breaking other things (like php apps)

So, is it possible to install redmine (or any ruby on rails app) in a directory without using a separate virtual host just for the app, AND without taking down other php apps (drupal, phpmyadmin, etc) which are running on the same apache server?

Yes. After lots of trial and error, the solution is fairly simple, but has to be done in a specific way.

Step 1. Configure httpd.conf files

Add this to your httpd.conf file, (or an included file) before your users/other conf files are loaded:

LoadModule passenger_module /Library/Ruby/Gems/1.8/gems/passenger-2.2.15/ext/apache2/mod_passenger.so
PassengerRoot /Library/Ruby/Gems/1.8/gems/passenger-2.2.15
PassengerRuby /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
PassengerEnabled Off

Follow the passenger install instructions to make sure that you’ve got the right paths and versions in there. “PassengerEnabled Off” in the global scope disables it by default. We’ll enable it just for where we want it.

Step 2. Setup a .conf file for the rails app

In my case, I’ve added a “redmine.conf” file which is imported by httpd.conf. In this file, I have:

<Location "/redmine">
    Options Indexes ExecCGI FollowSymLinks -MultiViews
    Order allow,deny
    Allow from all
    AllowOverride all

    PassengerEnabled On
    RailsBaseURI /redmine
    RailsEnv production
</Location>

Here we enable Passenger, set the rails environment and the rails base URI.

Step 3. Setup a symlink

I had tried this with real directories and apache “Alias” commands to no luck. The passenger module docs does have instructions, but only for use with virtual hosts, if you ignore the virtual hosts bit, it works. On the mac, I have it set up like this:

$ # /Library/WebServer/Documents -- document root
$ # /Library/WebServer/Sites -- for rails and other apps.
$ cd /Library/WebServer/Documents
$ ln -s ../Sites/redmine/public redmine

Finally, a solution that other network machines can access without Virtual Host nightmare.

ZFS performance networked from a Mac

Before I go ahead and do a full time machine backup to my OpenSolaris machine with a ZFS mirror, I thought I’d try and test out what performance hit there might be when using compression. I also figured, that I’d test out the impact of changing the recordsize too. Optimising this for the data record size seems to be best practices for databases, and since Time Machine stores data in a Mac Disk Image (SparseBundle) it probably writes data in 4k chunks matching the allocation size of the HFS filesystem in the disk image.

There were three copy tasks done:

  1. Copy a single large video file (1.57GB) to the Netatalk AFP share,
  2. Copy a single large video file (1.57GB) to a locally (mac) mounted disk image stored on the Netatalk AFP share,
  3. Copy a folder with 2752 files (117.3MB) to a locally (mac) mounted disk image stored on the Netatalk AFP share.

Here’s the results:

To Netatalk AFP share To Disk Image stored on AFP share To Disk Image stored on AFP share
ZFS Recordsize and compression 1 video file, 1.57GB 1 video file, 1.57GB 2752 files, 117.3MB
128k, off 0m29.826s

53.9MB/s

2m5.889s

12.7MB/s

1m45.809s

1.1MB/s

128k, on 0m52.179s

30.9MB/s

1m36.084s

16.7MB/s

1m34.367s

1.24MB/s

128k, gzip 0m31.290s

51.4MB/s

2m32.485s

10.5MB/s

2m29.141s

0.79MB/s

4k, off 0m27.131s

59.3MB/s

2m16.138s

11.8MB/s

2m47.718s

0.70MB/s

4k, on 0m25.651s

62.7MB/s

1m59.459s

13.5MB/s

1m41.551s

1.2MB/s

4k, gzip 0m30.348s

53.0MB/s

5m16.195s

5.08MB/s

4m48.378s

0.41MB/s

I think there was something else happening on the server for the 128k compression=on test, impacting on its data rate.

Conclusion:

The clear winner is the default compression and default record size. It must be that even my low powered Atom processor machine can compress the data faster than it can be written to disk resulting in less bandwidth to disc and therefore increasing performance at the same time as saving space. Well done ZFS!

Mac File sharing from OpenSolaris

I’ve just played around with 3 different ways of sharing files from OpenSolaris to my Mac:

  1. Using ZFS built in SMB sharing
  2. Using ZFS built in iSCSI sharing (with globalSAN iSCSI initiator for mac)
  3. Using AFP from netatalk 2.1

Using ZFS built in SMB sharing

This is by far the easiest, it requires no special software to install on either machine after the OS itself.

Using ZFS built in iSCSI sharing

Setting up the iSCSI share is just as easy as the SMB, however Mac doesn’t have an iSCSI client built in. You need to download and install the globalSAN iSCSI initiator for Mac.

This method should be good for Time Machine because the iSCSI device appears as a real hard drive, which you then format as Mac OS Extended and Time Machine’s funky little linked files and things should all work perfectly. No need to worry about users and accounts on the server, etc. In theory , this should deliver the best results, but it’s actually the worst performing of the lot.

Using AFP from netatalk 2.1

A little bit of work is required to install Netatalk 2.1. See my previous post and thanks again to the original posters where I learned how to do this.

This one should also be a very good candidate since it appears as a Mac server on the network and you should be able to access this shared Time Machine directly from the OS X install disc – an important consideration if the objective is to use it as a Time Machine backup (which it is for me).

Additionally, this one proved to have the best performance:

Performance

I tested copying 3GB files to each of the above shares and then reading it back again. Here’s the results:

Writing 3GB over gigabit ethernet:

iSCSI: 44m01s – 1.185MB/s

SMB share: 4m27 – 11.73MB/s

AFP share: 2m49 – 18.52MB/s

Reading 3GB over gigabit ethernet:

iSCSI: 4m36 – 11.34MB/s

SMB share: 1m45 – 29.81MB/s

AFP share: 1m16s – 41.19MB/s

The iSCSI was by far the slowest. Ten times slower than SMB – yikes! I have no idea if that’s due to the globalSAN initiator software or something else. Whatever the case, it’s not acceptable for now.

And Netatalk comes up trumps. Congratulations to everyone involved in Netatalk – great work indeed!

OpenSolaris screen sharing with Mac

I found two ways of connecting my Mac to my OpenSolaris box remotely:

1. Running a gnome-session over SSH.

$ ssh -X username@opensolaris.local gnome-session

And up pops a X11 app on the Mac and you can see the desktop. It’s slow and clunky, but it works.

2. Using Mac Snow Leopard’s built in Screen Sharing client.

This requires more configuration on the OpenSolaris side of this – apparently the Mac OS will only connect to a server that requires authentication in its expected method. This article showed me how to do it: Share your OpenSolaris 2008.11 screen to Mac Os X.

The second method is much prettier, doesn’t have windows that disappear under the Gnome application bar at the top, neatly puts everything in a window to the remote machine, and to boot it is actually way faster too.

Oh, and here’s a trick. Screen Sharing application for some reason doesn’t give you a nice interface to connect to a remote machine manually (as opposed to clicking the Share button in a Finder window). This also works, open your favourite web browser and type vnc://opensolaris.local/ to launch screen sharing on the machine “opensolaris.local” (also works with ip addresses).

Enjoy.

Finally I can hide Gmail’s All Mail folder in IMAP

I don’t know how long this Google Labs feature has been around, but I’m thrilled that now I can hide the “All Mail” folder from Gmail’s IMAP so Mac Mail doesn’t download messages twice!

Some people may not like that you cannot access archived messages from Mac Mail, but I’m happy to go online to do that. Messages I want to keep I move into folders anyway.

Once you enable the Advanced IMAP labs feature, you see this in the Labels settings page:

The Show in Imap checkbox appears when you enable the Google Labs Advanced IMAP feature.

The Show in Imap checkbox appears when you enable the Google Labs Advanced IMAP feature.

It also hides it on my iPhone = yay!

Time Machine backup very slow – Finder interference

I recently bought a brand new 1TB FW drive, formatted it for Mac (Mac OS Extended Journaled), and set it up to do a fresh TM back, about 180GB.

After 18 hours, it had backed up only 60GB, averaging just under 1MB/sec. Abysmally slow for a brand new drive, which bench tests over 30MB/sec sustained reads and writes.

I was watching the Console logs for anything unusual, and also had the Activity Monitor open watching the Disk Activity. I noticed that the Finder was using a lot of CPU – don’t know why.

So, using Activity Monitor, I quit the Finder.

Suddenly the disk activity to the FW drive jumped from less than 2MB/s to over 30MB/s. The rest of the backup completed in under 2 hours.

It seems that whatever the Finder was doing was interfering massively with Time Machine’s performance.

Using Mac OS X 10.6.1 Snow Leopard and a 1TB Western Digital My Book Home drive.

Time machine wants to do another full backup

My MacBook Pro had a weird problem where the hard-drive suddenly became full (when there should have been about 80GB free) and subsequently failed to boot.

No problems, out comes the install disc. Erase the drive and restore from last time machine backup.

Everything was perfect for a few days, then suddenly, Time Machine wants to do a full backup – I don’t know why – and has deleted all the older backups to make space for a new full backup. There isn’t space, so it’s now stuck in a Failed state.

Is there a way to reset the link between the computer and the TM drive so it can do an incremental backup again?

Or do I have to erase the computer and do a full restore?
Or erase the TM drive to do a full backup? (which is better in this case)

Thanks,
Matt