Matt Connolly's Blog

my brain dumps here…

Monthly Archives: June 2010

Western Digital Green Lemon

I have an OpenSolaris backup machine with 2 x 1.5 TB drives mirrored. One is a Samsung Silencer, the other is a Western Digital Green drive. The silencer is, ironically, the noisier of the two, but way outperforms the WD drive.

I’ve done some failure tests on the mirror by unplugging one drive while copying files to/from the backup server from my laptop.

First, I was copying from the server, onto a single FW drive, writing at a solid 30MB/s. I disconnected the Samsung drive while it was running and the file copy proceeded without fault at about 25MB/s of the single WD drive.

`zpool status` showed the drive was UNAVAIL and that the pool would continue to work in a degraded state. When I reconnected the drive, `cfgadm` showed it as connected by unconfigured. When I reconfigured the Samsung drive, the pool automatically resilvered any missing data. (wasn’t much because I was reading from the network) in a matter of seconds.

Failure test #2 was to remove the WD drive. I copied data to the server from the laptop, and the progress was intermittent… bursts of 30MB/s, then nothing for quite a few seconds, etc…. I disconnected the WD drive, and hey presto, the transfer rate instantly jumped up to a solid 20MB/s. This samsung drive definitely writes a whole stack faster than the WD drive. (A mirror is as fast as the slowest writing drive).

And here’s the lemon part. When I reconnected the WD drive, it showed up as disconnected. The samsung was connected, but unconfigured. To my frustration, I couldn’t reconnect the drive:

$ cfgadm
Ap_Id                          Type         Receptacle   Occupant     Condition
sata1/0                        sata-port    disconnected unconfigured failed
$ cfgadm -c connect sata1/0
cfgadm: Insufficient condition
I did a bit of searching and found this page: SolarisZfsReplaceDrive : use the -f force option:
$ pfexec cfgadm -f -c connect sata1/0
Activate the port: /devices/pci@0,0/pci8086,4f4d@1f,2:0
This operation will enable activity on the SATA port
Continue (yes/no)? yes
$ cfgadm
Ap_Id                          Type         Receptacle   Occupant     Condition
sata1/0                        disk         connected    unconfigured unknown
sata1/1::dsk/c8t1d0            disk         connected    configured   ok

So, now OpenSolaris sees the drive as connected, let’s configure it and zpool should see it straight away…

$ pfexec cfgadm -c configure sata1/0
$ cfgadm
Ap_Id                          Type         Receptacle   Occupant     Condition
sata1/0::dsk/c8t0d0            disk         connected    configured   ok
sata1/1::dsk/c8t1d0            disk         connected    configured   ok
$ zpool status -x
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: resilver in progress for 0h0m, 0.00% done, 465h28m to go
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c8t0d0s0  ONLINE       0 1.14K     0  544K resilvered
            c8t1d0s0  ONLINE       0     0     0

Oh man… I have to resilver the whole drive. Why!!??! The other drive remembered it was a part of the pool and intelligently went about resilvering the differences. This drive looks like it was to resilver the whole damn thing.

After a while:

$ zpool status
  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h23m, 5.05% done, 7h20m to go
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c8t0d0s0  ONLINE       0     0     0  12.3G resilvered
            c8t1d0s0  ONLINE       0     0     0

And here’s another interesting bit… the performance of the WD drive (c8t0d0) on my machine is really poor:

$ iostat -x 5

                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0   61.2    0.0 1056.2  0.0  9.1    0.0  148.1   0 100 c8t0d0
   79.0    0.0  978.7    0.0  0.0  0.0    0.0    0.6   0   3 c8t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c9t0d0
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0   72.0    0.0  178.8  0.0  7.2    0.0   99.6   0 100 c8t0d0
  111.8    0.0  361.3    0.0  0.0  0.0    0.0    0.3   0   1 c8t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c9t0d0
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0   51.6    0.0  120.4  0.0  7.5    0.0  145.9   0 100 c8t0d0
   79.4    0.0  143.7    0.0  0.0  0.0    0.0    0.2   0   1 c8t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c9t0d0
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0   62.2    0.0 1968.5  0.0  8.3    0.0  133.7   0 100 c8t0d0
   81.8    0.0 2616.7    0.0  0.0  0.3    0.0    3.2   0   8 c8t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c9t0d0
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0   34.6    0.0 1880.2  0.0  7.1    0.0  204.9   0  79 c8t0d0
   28.4   11.6 1413.5   41.7  0.0  0.1    0.0    3.1   0   7 c8t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c9t0d0

Check it out. 100% busy use of the drive, and it’s writing less than 2MB/s. Compare that to the %b busy for the Samsun (on c8t1d0) for reading the same amount of data. And check out the average service time (asvc_t) – that’s bad like a cd-rom!! Yikes.

It doesn’t get reconnect to the system, its service time is way slow and its write performance stinks. This WD drive is a total lemon!

Advertisements

My first real Time Machine backup on a ZFS mirror

So following my last post about the impact of compression on ZFS, I’ve created a ZFS file system with Compression ON and am sharing it via Netatalk to my MacBook Pro.

I connected the Mac via gigabit ethernet for the original backup, and it backed up 629252 items (193.0 GB) in 7 hours, 23 minutes, 4.000 seconds, according the backup log. That’s an average of 7.4MB/sec. Nowhere near the maximum transfer rates that I’ve seen to the ZFS share, but acceptable nonetheless.

`zfs list` reports that the compression ratio is 1.11x. I would have expected more, but oh well.

And now my incremental backups are also working well over the wireless connection. Excellent.

ZFS performance networked from a Mac

Before I go ahead and do a full time machine backup to my OpenSolaris machine with a ZFS mirror, I thought I’d try and test out what performance hit there might be when using compression. I also figured, that I’d test out the impact of changing the recordsize too. Optimising this for the data record size seems to be best practices for databases, and since Time Machine stores data in a Mac Disk Image (SparseBundle) it probably writes data in 4k chunks matching the allocation size of the HFS filesystem in the disk image.

There were three copy tasks done:

  1. Copy a single large video file (1.57GB) to the Netatalk AFP share,
  2. Copy a single large video file (1.57GB) to a locally (mac) mounted disk image stored on the Netatalk AFP share,
  3. Copy a folder with 2752 files (117.3MB) to a locally (mac) mounted disk image stored on the Netatalk AFP share.

Here’s the results:

To Netatalk AFP share To Disk Image stored on AFP share To Disk Image stored on AFP share
ZFS Recordsize and compression 1 video file, 1.57GB 1 video file, 1.57GB 2752 files, 117.3MB
128k, off 0m29.826s

53.9MB/s

2m5.889s

12.7MB/s

1m45.809s

1.1MB/s

128k, on 0m52.179s

30.9MB/s

1m36.084s

16.7MB/s

1m34.367s

1.24MB/s

128k, gzip 0m31.290s

51.4MB/s

2m32.485s

10.5MB/s

2m29.141s

0.79MB/s

4k, off 0m27.131s

59.3MB/s

2m16.138s

11.8MB/s

2m47.718s

0.70MB/s

4k, on 0m25.651s

62.7MB/s

1m59.459s

13.5MB/s

1m41.551s

1.2MB/s

4k, gzip 0m30.348s

53.0MB/s

5m16.195s

5.08MB/s

4m48.378s

0.41MB/s

I think there was something else happening on the server for the 128k compression=on test, impacting on its data rate.

Conclusion:

The clear winner is the default compression and default record size. It must be that even my low powered Atom processor machine can compress the data faster than it can be written to disk resulting in less bandwidth to disc and therefore increasing performance at the same time as saving space. Well done ZFS!