Matt Connolly's Blog

my brain dumps here…

Monthly Archives: August 2013

Network latency in SmartOS virtual machines

Today I decided to explore network latency in SmartOS virtual machines. Using the rbczmq ruby gem for ZeroMQ, I made two very simple scripts: a server that replies “hello” and a benchmark script that times how long it takes to send and receive 5000 messages after establishing the connection.

The server code is:

require 'rbczmq'
ctx =
sock = ctx.socket ZMQ::REP
loop do
  sock.send "reply"

The benchmark code is:

require 'rbczmq'
require 'benchmark'

ctx =
sock = ctx.socket ZMQ::REQ

# establish the connection
sock.send "hello"

# run 5000 cycles of send request, receive reply.
puts Benchmark.measure {
  5000.times {
    sock.send "hello"

The test machines are:

* Mac laptop – server & benchmarking
* SmartOS1 (SmartOS virtual machine/zone) server & benchmarking
* SmartOS2 (SmartOS virtual machine/zone) benchmarking
* Linux1 (Ubuntu Linux in KVM virtual machine) server & benchmarking
* Linux2 (Ubuntu Linux in KVM virtual machine) benchmarking

The results are:

Source      Dest        Connection      Time          Req-Rep/Sec
------      ----        ----------      ----          --------
Mac         Linux1      1Gig Ethernet   5.038577      992.3
Mac         SmartOS1    1Gig Ethernet   4.972102      1005.6
Linux2      Linux1      Virtual         1.696516      2947.2
SmartOS2    Linux1      Virtual         1.153557      4334.4
Linux2      SmartOS1    Virtual         0.952066      5251.8
Linux1      Linux1      localhost       0.836955      5974.0
Mac         Mac         localhost       0.781815      6395.4
SmartOS2    SmartOS1    Virtual         0.470290      10631.7
SmartOS1    SmartOS1    localhost       0.374373      13355.7

localhost tests use

SmartOS has an impressive network stack. Request-reply times from one SmartOS machine to another are over 3 times faster than when using Linux under KVM (on the same host). This mightn’t make much of a difference to web requests coming from slow mobile device connections, but if your web server is making many requests to internal services (database, cache, etc) this could make a noticeable difference.

Installing Jenkins-CI in SmartOS

I found a very helpful gist with a script and SMF service manifest for installing Jenkins CI on SmartOS. A few tweaks later, I have my own set up. Here’s my gist:

Now I have rvm, ruby 2.0 and the rbczmq gem all building and running tests and emailing me if any fail. I’m polling git for changes hourly from the projects github repositories.

Screen Shot 2013-08-21 at 9.04.18 am

Jenkins has a plethora of plugins available and integrates with git nicely. The only thing I found unobvious was that build scripts run from Jenkins don’t inherit a normal shell environment, so you may need to set up environment variables such as TERM, PATH and CA_BUN for curl.

Here’s my build script for rvm, which installs it locally and verifies it installed.

# setup:
export HOME=/home/admin
export rvm_path=$HOME/.rvm

echo "Destroying any current installation of RVM in /home/admin/.rvm ..."
rm -rf $rvm_path

echo "running install script:"
bash ./install

if test -x $rvm_path/scripts/rvm
source $rvm_path/scripts/rvm
echo "Failed to install rvm"
exit 1

rvm version

One nice thing I set up is dependent builds. I have “install rvm” => “install ruby 2.0” => “build & test rbczmq” as dependent builds, so if a dependency changes, I re-run the dependent projects to make sure there’s no negative downstream side effects. Might be overkill, but if it picks up a change, I’ll be glad to know about it.

Next, I’d like to have a nice “Build passes tests in SmartOS” badge that I could stick in the repositories read me pages just like travis-ci has.

Even better would be for Travis to run builds in SmartOS directly given their awesome integration with github!