Building netatalk in SmartOS

I’m looking at switching my home backup server from OpenIndiana to SmartOS. (there’s a few reasons, and that’s another post).

One of the main functions of my box is to be a Time Machine backup for my macs (my laptop and my wife’s iMac). I found this excellent post about building netatalk 3.0.1 in SmartOS, but it skipped a few of the dependencies, and did the patch after configure, which means if you change you reconfigure netatalk, then you need to reapply the patch.

Based on that article, I came up with a patch for netatalk, and here’s a gist of it: https://gist.github.com/mattconnolly/5230461

Prerequisites:

SmartOS already has most of the useful bits installed, but these are the ones I needed to install to allow netatalk to build:

$ sudo pkgin install gcc47 gmake libgcrypt

Build netatalk:

Download the latest stable netatalk. The netatalk home page has a handy link on the left.

$ cd netatalk-3.0.2
$ curl 'https://gist.github.com/mattconnolly/5230461/raw/27c02a276e7c2ec851766025a706b24e8e3db377/netatalk-3.0.2-smartos.patch' > netatalk-smartos.patch
$ patch -p1 < netatalk-smartos.patch
$ ./configure --with-bdb=/opt/local --with-init-style=solaris --with-init-dir=/var/svc/manifest/network/ --prefix=/opt/local
$ make
$ sudo make install

With the prefix of ‘/opt/local’ netatalk’s configuration file will be at ‘/opt/local/etc/afp.conf’

Enjoy.

[UPDATE]

There is a very recent commit in the netatalk source for an `init-dir` option to configure which means that in the future this patch won’t be necessary, and adding `--with-init-dir=/var/svc/manifest/network/` will do the job. Thanks HAT!

[UPDATE 2]

Netatalk 3.0.3 was just released, which includes the –init-dir option, so the patch is no longer necessary. Code above is updated.

OpenIndiana – running openvpn as a service

Here’s a gist for the XML manifest to run openvpn as a service:

https://gist.github.com/2484917

It expects that there is an openvpn config file at /etc/openvpn/config which, you’ll need to configure with your settings, certificates, etc.

If you configure it to run a tap interface then bonjour advertising will work over the link, which is great if you want time machine or other bonjour services to work to an OpenIndiana server from a mac connecting from anywhere with openvpn.

Passenger apache module for OpenIndiana

I did a bit of hunting and made some patches to the ‘passenger’ gem so that it’s apache module would compile for OpenIndiana. Changes are in my github fork:

https://github.com/mattconnolly/passenger

And I just noticed that one of the fixes was in a patch in Joyent’s SmartOS instructions for using passenger.

I tested this also on a VM guest installation of Solaris 11 Express, and it worked too. I’d be interested to hear if it works for others on OpenIndiana, Solaris or SmartOS.

So with updates to rvm, latest version of ruby and with this patched version of passenger, I’m finally good to go to deploy rails apps on OpenIndiana. Woot!

Installing mysql2 gem on OpenIndiana

Quick one, with mysql 5.1 installed from standard OpenIndiana package repository, the ruby mysql2 gem can be installed with this command:

$ gem install mysql2 -v '0.2.18' -- --with-mysql-dir=/usr/mysql/5.1 --with-mysql-include=/usr/mysql/5.1/include/mysql

This one requires /usr/gnu/bin in the front(ish) of your path, so you may need an `export PATH=/usr/gnu/bin:$PATH` before you do this.

Enjoy.

 

[UPDATE]

This also works for the latest version ‘0.3.11’:

$ gem install mysql2 -v '0.3.11' -- --with-mysql-dir=/usr/mysql/5.1 --with-mysql-include=/usr/mysql/5.1/include/mysql

[UPDATE 2 – ruby-2.0.0]

Ruby 2.0.0 compiles as a 64-bit executable, which means another bit needs to be added to the command line options:

$ gem install mysql2 -v '0.3.11'-- --with-mysql-dir=/usr/mysql/5.1/ --with-mysql-include=/usr/mysql/5.1/include/mysql --with-mysql-lib=/usr/mysql/5.1/lib/amd64/mysql

TTCP, in Ruby

I’ve used the TTCP tcp test program from time to time, and am at present looking at some networking in Ruby. So why not have a look at porting that to ruby? So I did.

This has been built as a gem, which an executable ‘ttcp’ that will install in your gem’s bin folder. You can get the gem from here: http://rubygems.org/gems/ttcp

Or type: `gem install ttcp` at your terminal.

Source code is released under MIT license, and available on github: https://github.com/mattconnolly/ttcp

So far, I’ve tested it out on Mac and OpenIndiana in ruby 1.8.7, 1.9.3 and JRuby 1.6.5. I can’t seem to run the tests in JRuby, but it appears to work anyway.

Enjoy.

Using RVM with Jenkins

I’ve recently set up RVM running with Jenkins, and finally have it all working just the way I want.

I’m running on OpenIndiana and have Jenkins running as its own user. I ssh in as that user and install rvm for that user and verified that I can run ruby tests with rvm from the terminal.

There’s a “RVM plugin” (version 0.1) for Jenkins which has great promise – I like the idea of running all build steps with RVM setup. However, I couldn’t get it to find my rvm installation no matter how much stuffing around with environment variables I did. So I’m not using it for now.

The trick was to use a “Execute shell” build step and use a script like so:

#!/bin/bash -e
export TERM=xterm
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"
# Use the correct ruby
rvm use 1.9.2@gemset --create
# Do any setup
# e.g. possibly do 'rake db:migrate db:test:prepare' here
bundle install
# Finally, run your tests
rake test
# and build the gem
rake build
# and install the gem locally
rake install

This was for a ruby gem project which installs that gem in the named rvm gemset. Other projects can use this gem simply by using that same gemset.

Note also the line: “export TERM=xterm” – rvm is designed to run in a terminal where the TERM environment variable is set. Apparently there was a change to gracefully ignore its absence or be able to use TERM=dumb for no colour, (github issue 698), but that doesn’t appear to be working for me in rvm 1.10.2.

Good luck.

sudo with askpass

Normally I connect to a remote server with ssh then at the command prompt execute a command using sudo if I require the additional privileges. Just today, I had occasion to run sudo over ssh (something broken with the login prompt). Without having preconfigured anything, I got this to work:

ssh (server) SUDO_ASKPASS=/usr/lib/ssh/ssh-askpass sudo -A (command)

In my case, I had an appropriate command at ‘/usr/lib/ssh/ssh-askpass’. Your system might be different.

Using rvm on OpenIndiana

OpenIndiana is a fork of Sun’s OpenSolaris before it was killed by Oracle. There’s quite a few commands in there that are different from linux, and there’s been a bit of discussion on the openindiana mailing list about whether commands should be more linux like or not.

In any case, there’s the built in commands in /usr/bin and the linux-like versions in /usr/gnu/bin. This is what you need in your path to use the linux versions, which I have, but I have it at the end of my path.

Unfortunately that’s no good for rvm. Well at least for the rvm upgrading part.

You can put /usr/gnu/bin at the beginning of your PATH. Or if you don’t want to do that, you can:

$ # installing:
$ PATH=/usr/gnu/bin:$PATH bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
$ # upgrading
$ PATH=/usr/gnu/bin rvm get latest

ZFS = Data integrity

So, for a while now, I’ve been experiencing crappy performance of a Western Digital Green drive (WD15EARS) I have an a zfs mirror storing my time machine backups (using OpenIndiana and Netatalk).

Yesterday, the drive started reporting errors. Unfortunately, the system hung – that’s not so cool – ZFS is supposed to keep working when a drive fails… Aside from that, when I rebooted, the system automatically started a scrub to verify data integrity, and after about 10 minutes:

  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Thu Mar 10 10:19:42 2011
    1.68G scanned out of 1.14T at 107M/s, 3h5m to go
    146K resilvered, 0.14% done
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            c8t1d0s0  DEGRADED     0     0    24  too many errors  (resilvering)
            c8t0d0s0  ONLINE       0     0     0
        cache
          c12d0s0     ONLINE       0     0     0

errors: No known data errors

Check it out. It’s found 24 errors on the Western Digital Drive, but so far no data errors have been found, because they were correct on the other drive.

That’s obvious, right? But what other operating systems can tell the difference between the right and wrong data when they’re both there??? Most raid systems only detect a total drive failure, but don’t deal with incorrect data coming off the drive !!

Sure backing up to a network (Time Machine’s sparse image stuff) is *way* slower than a directly connected firewire drive, but in my opinion, it’s well worth doing it this way for the data integrity that you don’t get on a single USB or Firewire drive.

Thank you ZFS for keeping my data safe. B*gger off Western Digital for making crappy drives. I’m off to get a replacement today… what will it be? Samsung or Seagate?

Building Netatalk from source on OpenIndiana

On my OpenIndiana backup server, I’ve built quite a few packages from source over the last year. Today I went to try building netatalk. It required a few more things to get it going:

# pkg install developer/build/libtool
# pkg install developer/build/automake-110

Now, for some reason, the automake package installs `aclocal-1.10` but not `aclocal` which the netatalk bootstrap script looks for.

# ln -s /usr/bin/aclocal-1.10 /usr/bin/aclocal
# ln -s /usr/bin/automake-1.10 /usr/bin/automake

Then I can proceed with the normal build process:
$ ./bootstrap
$ ./configure
$ make
$ sudo make install