Matt Connolly's Blog

my brain dumps here…

Tag Archives: apple

Happy Holidays

This is a quick happy holidays and thank you to all the people and companies that have done great things in 2013. In no particular order:

Podcasters:

I’ve enjoyed many a podcast episode this year. My favourites are the Edge Cases featuring Wolf Rentzsch and Andrew Pontious, Accidental Tech Podcast featuring John Siracusa, Casey Liss and Marco Arment and Rails Casts by Ryan Bates.
Thank you all for your hard work putting your respective shows together. Your efforts are greatly appreciated, and I hope you are getting enough out of it so that it’s worthwhile continuing in 2014!

Companies:

JetBrains, makers of Rubymine. These guys pump out great work. If you’re keen to get involved in the early access program you can get nightly or weekly builds. Twice this year I’ve submitted a bug and within a week had it verified by JetBrains, fixed, in a build and in my hands. Their CI system even updates the bug with the build number including the fix. Seriously impressive. They set the bar so high, I challenge any company (including myself) to match their effective communication rapid turn around on issues.

Joyent for actually innovating in the cloud, and your contributions to open source projects such as NodeJS and SmartOS! Pretty impressive community engagement, not only in open source code, but community events too… What a shame I don’t live in San Francisco to attend and thank you guys in person.

Github for helping open source software and providing an awesome platform for collaboration. So many projects benefit from being on Github.

Apple, thanks for making great computers and devices. Well done on 64 bit ARM. The technology improvements in iOS 7 are great, however, my new iPhone 5S doesn’t feel a single bit faster than my previous iPhone 5 due to excessive use of ease out animations which have no place in a User Interface. Too many of my bug reports have been closed as “works as intended”, when the problem is in the design not the implementation. Oh well.

Products / Services:

Strava has helped me improve in my cycling and fitness. The website and iPhone apps are shining examples of a great user experience: works well, easy to use, functional and good looking. Thanks for a great product.

Reveal App is a great way to break down the UI of an iOS app. Awesome stuff.

Twitter has been good, mostly because of how people use it. I suppose it’s more thanks to the people on Twitter who I follow.

Black Star Coffee, it’s how I start my day! Great coffee.

Technologies:

ZeroMQ: This is awesome. Reading the ZeroMQ guide was simply fantastic. This has changed my approach to communications in computing. Say goodbye to mutexes and locks and hello to messages and event driven applications. Special thanks to Pieter Hintjens for his attention to the ZeroMQ mailing lists, and to all of the contributors to a great project.

SmartOS: Totally the best way to run a hypervisor stack. The web page says it all: ZFS + DTrace + Zones + KVM. Get into it. Use ZFS. You need a file system that can verify your data. Hard drives cannot be trusted. I repeat, use ZFS.

Advertisements

Time Machine Backups and silent data corruptions

I’ve recently heard many folk talking about Time Machine backup strategies. To do it well, you really do need to backup your backup, as Time Machine can “eat itself”, especially doing network backups.

Regardless of whether your Time Machine backup is to a locally attached disk or a network drive, when you make a backup of your backup, you want to make sure it’s valid, otherwise you’re propagating a corrupt backup.

So how do you know if your backup is corrupt? You could read it from beginning to end. But this would only protect you from data corruptions that can be detected by the drive itself. Disk verify, fsck, and others go further and validate the file system structures, but still not your actual data.

There are “silent corruptions”, which is where the data you wrote to the disk comes back corrupted (different data, not a read error). “That never happens”, you might say, but how would you know?

I have two servers running SmartOS using data stored on ZFS. I ran a data scrub on them, and both reported checksum errors. This is exactly the silent data corruption scenario.

ZFS features full checksumming of all data when stored, and if your data is in a RAIDZ or mirror configuration, it will also self-heal. This means that instead of returning an error, ZFS will go fetch the data from a good drive and also make another clean copy of that block so that its durability matches your setup.

Here’s the specifics of my corruptions:

On a XEON system with ECC RAM, the affected drive is a Seagate 1TB Barracuda 7200rpm, ST31000524AS, approximately 1 year old.

  pool: zones
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
   
  scan: resilvered 72.4M in 0h48m with 0 errors on Mon Nov 18 13:28:16 2013
config:

        NAME          STATE     READ WRITE CKSUM
        zones         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c1t1d0s0  ONLINE       0     0     0
            c1t0d0s0  ONLINE   2.61K  366k   635
            c1t4d0s1  ONLINE       0     0     0
        logs
          c1t2d0s0    ONLINE       0     0     0
        cache
          c1t2d0s1    ONLINE       0     0     0

errors: No known data errors

On a Celeron system with non-ECC RAM, the affected drive is a Samsung 2TB low power drive, approximately 2 years old.

  pool: zones
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 8K in 12h51m with 0 errors on Thu Nov 21 00:44:25 2013
config:

        NAME          STATE     READ WRITE CKSUM
        zones         ONLINE       0     0     0
          raidz1-0    ONLINE       0     0     0
            c0t1d0    ONLINE       0     0     0
            c0t3d0    ONLINE       0     0     0
            c0t2d0p2  ONLINE       0     0     2
        logs
          c0t0d0s0    ONLINE       0     0     0
        cache
          c0t0d0s1    ONLINE       0     0     0

errors: No known data errors

Any errors are scary, but the checksum errors even more so.

I had previously seen thousands of checksum errors on a Western Digital Green drive. I stopped using it and threw it in the bin.

I have other drives that are HFS formatted. I have no way of knowing if they have any corrupted blocks.

So unless your data is being checksummed, you are not protected from data corruption, and making a backup of a backup could easily be propagating data corruptions.

I dream of a day when we can have ZFS natively on Mac. And if it can’t be done for whatever ‘reasons’, at least give us the features from ZFS that we can use to protect our data.

Xcode testing AFNetwork Operation callback blocks

Just recently, I was writing some tests in Xcode for some HTTP requests using the AFNetwork library. Previously I’ve used the ASIHTTPRequest library, but in this particular project, I’ve chosen to use AFNetworking for its JSON support.

Since the requests run asynchronously we need a way to wait for the operation to complete. This is easy:

- (void)testRequest
{
    MyHTTPClient* api = [MyHTTPClient sharedInstance]; // subclass of AFHTTPClient
    NSDictionary* parameters = [NSDictionary dictionary]; // add query parameters to this dict.
    __block int status = 0;
    AFJSONRequestOperation* request = [api getPath:@"path/to/test"
                                        parameters:parameters
                                           success:^(AFHTTPRequestOperation *operation, id responseObject) {
                                               // success code
                                               status = 1;
                                               NSLog(@"succeeded");
                                           } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
                                               // failure
                                               status = 2;
                                               NSLog(@"failed");
                                           }];
    [api enqueueHTTPRequestOperation:request];
    [api.operationQueue waitUntilAllOperationsAreFinished];

    STAssertTrue([request isFinished], @"request finished");
    STAssertEquals(request.response.statusCode, 200, @"request returned 200 OK");
    STAssertEquals(status, 1, @"success block was executed");
}

This is great for testing that the request completes, and verifying its status. But if we need to test anything in the success or failure callbacks, the last test will fail with `status == 0`.

This is because AFNetwork processes its response in a background thread, and the final success or failure block callback is dispatched asynchronously from there to a specific queue, which unless provided is the main queue. This means that the block won’t get called until *AFTER* the test code has completed.

Putting in some kind of a lock causes a deadlock, since the test is running on the main thread, and the block callback never gets an opportunity to run. The solution is to manually run the main threads runloop until the callbacks have been processed.

Here’s my solution:

- (void)testRequest
{
    MyHTTPClient* api = [MyHTTPClient sharedInstance]; // subclass of AFHTTPClient
    NSDictionary* parameters = [NSDictionary dictionary]; // add query parameters to this dict.
    __block int status = 0;
    AFJSONRequestOperation* request = [api getPath:@"path/to/test"
                                        parameters:parameters
                                           success:^(AFHTTPRequestOperation *operation, id responseObject) {
                                               // success code
                                               status = 1;
                                               NSLog(@"succeeded");
                                           } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
                                               // failure
                                               status = 2;
                                               NSLog(@"failed");
                                           }];
    [api enqueueHTTPRequestOperation:request];
    [api.operationQueue waitUntilAllOperationsAreFinished];

    while (status == 0)
    {
        // run runloop so that async dispatch can be handled on main thread AFTER the operation has 
        // been marked as finished (even though the call backs haven't finished yet).
        [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode
                                 beforeDate:[NSDate date]];
    }

    STAssertTrue([request isFinished], @"request finished");
    STAssertEquals(request.response.statusCode, 200, @"request returned 200 OK");
    STAssertEquals(status, 1, @"success block was executed");
}

This addition will continually pump that run loop which allows AFNetwork’s async dispatch of the block to the main queue to execute, and hey presto! We now have a test that can also verify code in the success (or failure) completion blocks of an AFNetwork request operation.

ZFS = Data integrity

So, for a while now, I’ve been experiencing crappy performance of a Western Digital Green drive (WD15EARS) I have an a zfs mirror storing my time machine backups (using OpenIndiana and Netatalk).

Yesterday, the drive started reporting errors. Unfortunately, the system hung – that’s not so cool – ZFS is supposed to keep working when a drive fails… Aside from that, when I rebooted, the system automatically started a scrub to verify data integrity, and after about 10 minutes:

  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Thu Mar 10 10:19:42 2011
    1.68G scanned out of 1.14T at 107M/s, 3h5m to go
    146K resilvered, 0.14% done
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            c8t1d0s0  DEGRADED     0     0    24  too many errors  (resilvering)
            c8t0d0s0  ONLINE       0     0     0
        cache
          c12d0s0     ONLINE       0     0     0

errors: No known data errors

Check it out. It’s found 24 errors on the Western Digital Drive, but so far no data errors have been found, because they were correct on the other drive.

That’s obvious, right? But what other operating systems can tell the difference between the right and wrong data when they’re both there??? Most raid systems only detect a total drive failure, but don’t deal with incorrect data coming off the drive !!

Sure backing up to a network (Time Machine’s sparse image stuff) is *way* slower than a directly connected firewire drive, but in my opinion, it’s well worth doing it this way for the data integrity that you don’t get on a single USB or Firewire drive.

Thank you ZFS for keeping my data safe. B*gger off Western Digital for making crappy drives. I’m off to get a replacement today… what will it be? Samsung or Seagate?

ZFS for Mac Coming soon…

A little birdy told me, that there might be a new version of ZFS ported to Mac OS X coming up soon…

It seems the guys at Tens Compliment are working on a port of ZFS at a much more recent version than what was left behind by apple and forked as a Google code project: http://code.google.com/p/maczfs/

On my mac, I have installed the Mac-ZFS which can be found at the Google Code project. (I don’t have any ZFS volumes, it’s installed because I wanted to know what version it was up to.)

bash-3.2# uname -prs
Darwin 10.6.0 i386
bash-3.2# zpool upgrade
This system is currently running ZFS pool version 8.

All pools are formatted using this version.

My backup server at home is running OpenIndiana oi-148:

root@vault:~# uname -prs
SunOS 5.11 i386
root@vault:~# zpool upgrade
This system is currently running ZFS pool version 28.

All pools are formatted using this version.

Pretty exciting that we can get the same zpool version as the latest OpenIndiana… think of the backup/restore possibilities sending a snapshot over to a remote machine.

ZFS saved my Time Machine backup

For a while now, I’ve been using Time Machine to backup to an AFP share hosted by netatalk on an OpenIndiana low powered home server.

Last night, Time Machine stopped, with an error message: “Time Machine completed a verification of your backups. To improve reliability, Time Machine must create a new backup for you“.

Periodically I create ZFS snapshots of the volume containing my Time Machine backup. I haven’t enabled any automatic snapshots yet (like OpenIndiana/Solaris’s Time Slider service), so I just do it manually every now and then.

So, I shutdown netatalk, rolled back the snapshot, checked the netatalk database, restarted netatalk, and was then back in business.

# /etc/init.d/netatalk stop
# zfs rollback rpool/MacBackup/TimeMachine@20100130
# /usr/local/bin/dbd -r /MacBackup/TimeMachine
# /etc/init.d/netatalk start

Lost only a day or two’s incremental backups, which was much more palatable than having to do another complete backup of >250GB.

ZFS is certainly proving to be useful, even in a low powered home backup scenario.

ZFS performance networked from a Mac

Before I go ahead and do a full time machine backup to my OpenSolaris machine with a ZFS mirror, I thought I’d try and test out what performance hit there might be when using compression. I also figured, that I’d test out the impact of changing the recordsize too. Optimising this for the data record size seems to be best practices for databases, and since Time Machine stores data in a Mac Disk Image (SparseBundle) it probably writes data in 4k chunks matching the allocation size of the HFS filesystem in the disk image.

There were three copy tasks done:

  1. Copy a single large video file (1.57GB) to the Netatalk AFP share,
  2. Copy a single large video file (1.57GB) to a locally (mac) mounted disk image stored on the Netatalk AFP share,
  3. Copy a folder with 2752 files (117.3MB) to a locally (mac) mounted disk image stored on the Netatalk AFP share.

Here’s the results:

To Netatalk AFP share To Disk Image stored on AFP share To Disk Image stored on AFP share
ZFS Recordsize and compression 1 video file, 1.57GB 1 video file, 1.57GB 2752 files, 117.3MB
128k, off 0m29.826s

53.9MB/s

2m5.889s

12.7MB/s

1m45.809s

1.1MB/s

128k, on 0m52.179s

30.9MB/s

1m36.084s

16.7MB/s

1m34.367s

1.24MB/s

128k, gzip 0m31.290s

51.4MB/s

2m32.485s

10.5MB/s

2m29.141s

0.79MB/s

4k, off 0m27.131s

59.3MB/s

2m16.138s

11.8MB/s

2m47.718s

0.70MB/s

4k, on 0m25.651s

62.7MB/s

1m59.459s

13.5MB/s

1m41.551s

1.2MB/s

4k, gzip 0m30.348s

53.0MB/s

5m16.195s

5.08MB/s

4m48.378s

0.41MB/s

I think there was something else happening on the server for the 128k compression=on test, impacting on its data rate.

Conclusion:

The clear winner is the default compression and default record size. It must be that even my low powered Atom processor machine can compress the data faster than it can be written to disk resulting in less bandwidth to disc and therefore increasing performance at the same time as saving space. Well done ZFS!

Mac File sharing from OpenSolaris

I’ve just played around with 3 different ways of sharing files from OpenSolaris to my Mac:

  1. Using ZFS built in SMB sharing
  2. Using ZFS built in iSCSI sharing (with globalSAN iSCSI initiator for mac)
  3. Using AFP from netatalk 2.1

Using ZFS built in SMB sharing

This is by far the easiest, it requires no special software to install on either machine after the OS itself.

Using ZFS built in iSCSI sharing

Setting up the iSCSI share is just as easy as the SMB, however Mac doesn’t have an iSCSI client built in. You need to download and install the globalSAN iSCSI initiator for Mac.

This method should be good for Time Machine because the iSCSI device appears as a real hard drive, which you then format as Mac OS Extended and Time Machine’s funky little linked files and things should all work perfectly. No need to worry about users and accounts on the server, etc. In theory , this should deliver the best results, but it’s actually the worst performing of the lot.

Using AFP from netatalk 2.1

A little bit of work is required to install Netatalk 2.1. See my previous post and thanks again to the original posters where I learned how to do this.

This one should also be a very good candidate since it appears as a Mac server on the network and you should be able to access this shared Time Machine directly from the OS X install disc – an important consideration if the objective is to use it as a Time Machine backup (which it is for me).

Additionally, this one proved to have the best performance:

Performance

I tested copying 3GB files to each of the above shares and then reading it back again. Here’s the results:

Writing 3GB over gigabit ethernet:

iSCSI: 44m01s – 1.185MB/s

SMB share: 4m27 – 11.73MB/s

AFP share: 2m49 – 18.52MB/s

Reading 3GB over gigabit ethernet:

iSCSI: 4m36 – 11.34MB/s

SMB share: 1m45 – 29.81MB/s

AFP share: 1m16s – 41.19MB/s

The iSCSI was by far the slowest. Ten times slower than SMB – yikes! I have no idea if that’s due to the globalSAN initiator software or something else. Whatever the case, it’s not acceptable for now.

And Netatalk comes up trumps. Congratulations to everyone involved in Netatalk – great work indeed!

OpenSolaris + TimeMachine backup + network discovery

I found several tutorials on blogs about how to build “netatalk” on OpenSolaris to enable file sharing using the AFP protocol for better use by TimeMachine backups on the mac. Here are a few:

As far as making the AFP service discoverable, most people point to this article:

However, I found a better way to make the service discoverable, and that is by using the avahi-bridge. You’ll need to make sure that you have multicast dns running and the avahi bridge

1. Install netatalk according to the above instructions.

2. Install multicast dns and avahi bridge (I’m using snv_134 development branch, note that the package names have changed since 2009.06 release)
# pkg install system/network/avahi
# pkg install service/network/dns/mdns

3. enable both services (why is one called mdns and the next multicast?)

# svcadm enable network/dns/multicast:default
# svcadm enable system/avahi-bridge-dsd:default

4. setup a service xml file for avahi-bridge:

# cat > /etc/avahi/services/afp.service
<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_device-info._tcp</type>
<port>548</port>
<txt-record>model=RackMac</txt-record>
</service>
<service>
<type>_afpovertcp._tcp</type>
<port>548</port>
</service>
</service-group>

If you’re interested in advertising other services so that they are discoverable by the Mac, this is a great article (it’s for linux, but the avahi side of things is the same using avahi-bridge): Benjamin Sherman » Advertising Linux Services via Avahi/Bonjour.

Now I have a server in my Finder side bars just like any other mac server.

Finally I can hide Gmail’s All Mail folder in IMAP

I don’t know how long this Google Labs feature has been around, but I’m thrilled that now I can hide the “All Mail” folder from Gmail’s IMAP so Mac Mail doesn’t download messages twice!

Some people may not like that you cannot access archived messages from Mac Mail, but I’m happy to go online to do that. Messages I want to keep I move into folders anyway.

Once you enable the Advanced IMAP labs feature, you see this in the Labels settings page:

The Show in Imap checkbox appears when you enable the Google Labs Advanced IMAP feature.

The Show in Imap checkbox appears when you enable the Google Labs Advanced IMAP feature.

It also hides it on my iPhone = yay!