Matt Connolly's Blog

my brain dumps here…

Tag Archives: ZeroMQ

A year with ZeroMQ + Ruby

It was a year ago since my first commit to rbczmq, a Ruby bindings gem for ZeroMQ, which wraps the higher level CZMQ library.

I don’t recall exactly how I first heard about ZeroMQ, but needless to say, I got interested and started reading up on it. At the time, I discovered there were three gems for using ZeroMQ in ruby:

  • zmq – the original official zmq library. This is old. It tracks ZeroMQ version 2 (currently version 4) and has not been updated for nearly 3 years.
  • ffi-rzmq – a binding using FFI, which is compatible with Ruby, Rubinius and JRuby.
  • rbczmq – a native extension (no JRuby support) binding using the czmq library.

I didn’t spend a lot of time with the zmq gem since it was so old. The ffi-rzmq gem worked well, but didn’t feel very ruby like in its interface. For example, when receiving a message with a C method you would pass a buffer as a parameter to receive the contents of the message and the returned value would be an error code. This is quite un-ruby-like: I would expect receive to return the received data or raise an exception for the error code in keeping with built in ruby socket/file i/o calls.

So I started to explore rbczmq. Initially, I wasn’t so interested in the CZMQ wrapping part, I just wanted something that was more ruby-like to use. And it was. And it was faster. And the CZMQ part actually helps too.

In ZeroMQ each message part, or “frame” is considered a message. So when you read multi-frame messages in ZeroMQ you need to check the “more” flag, and read the next part. CZMQ wraps this as a single message with a number of “frames”. rbczmq neatly exposes these as Ruby classes: ZMQ::Message and ZMQ::Frame. You can still send and receive raw frames (as strings), but the class is a nice wrapper.

And to boot, it turns out that it was way faster than the ffi gem. I seem to have lost track of the comparison I did, but I recall it was convincing.

What’s changed?

During this year, rbczmq has received a number of updates and new features, major ones including:

  • Upgrade to ZeroMQ 4
  • Upgrade to CZMQ 2
  • Support for SmartOS platform
  • Fixes to memory management

Major things still to do:

  • Add support for new authentication interface.
  • Ship binary gems (like libv8) to save compilation time on deploy / install.

Hard bits

The hardest bit of work I contributed to this project was fixing bugs in the memory management. In particular, CZMQ has specific rules about ownership of memory. Ruby is a garbage collected environment, which also has its own set of rules about ownership of memory. The two do not match.

Most calls to ZeroMQ are done outside of the Ruby “GVL” (global lock) which allows the ruby VM to continue processing ruby code in other threads while one is doing a synchronous/blocking read on a socket, for example. When you combine this with Ruby threads, things can get hairy. The solution was two-fold:

  1. Use an ownership flag. When ownership was known to be transferred to ZeroMQ, mark the ruby object as no longer owned by Ruby. This meant that Ruby garbage collection callbacks would know if they were ultimately responsible for freeing memory used by an object. There was also some tricky interplay between contexts and sockets, since a socket is owned by a context, and destroying a context also destroys the sockets, so a socket is only owned by ruby if it has not been closed and the context has not been destroy.
  2. A socket closing mutex: Socket closing and context closing are asynchronous. If a socket is still open when a context is destroyed, then all sockets belonging to that socket will be closed. This happens outside the Ruby GVL, which means that a race condition exists where the Ruby garbage collector may collect the socket while it is still closing. ZeroMQ socket close is not threadsafe, so a mutex was the only solution to make this safe.

Using a mutex for socket close may result in a performance hit for an application which opens and closes sockets rapidly, but from what I understand, that is a bad thing to do anyway.

Looking forward

I have a few projects in the wild now using the rbczmq gem, and am very happy with its stability and performance. I haven’t used all of the APIs in Anger (such as Loops or Beacons), but I’m sure the time will come. I look forward to another year of contributions to this project to keep it up to date with what’s happening in the ZeroMQ and CZMQ projects.

I’d love to hear from other people using this gem, so give me a shout!

Advertisements

Network latency in SmartOS virtual machines

Today I decided to explore network latency in SmartOS virtual machines. Using the rbczmq ruby gem for ZeroMQ, I made two very simple scripts: a server that replies “hello” and a benchmark script that times how long it takes to send and receive 5000 messages after establishing the connection.

The server code is:

require 'rbczmq'
ctx = ZMQ::Context.new
sock = ctx.socket ZMQ::REP
sock.bind("tcp://0.0.0.0:5555")
loop do
  sock.recv
  sock.send "reply"
end

The benchmark code is:

require 'rbczmq'
require 'benchmark'

ctx = ZMQ::Context.new
sock = ctx.socket ZMQ::REQ
sock.connect(ARGV[0])

# establish the connection
sock.send "hello"
sock.recv

# run 5000 cycles of send request, receive reply.
puts Benchmark.measure {
  5000.times {
    sock.send "hello"
    sock.recv
  }
}

The test machines are:

* Mac laptop – server & benchmarking
* SmartOS1 (SmartOS virtual machine/zone) server & benchmarking
* SmartOS2 (SmartOS virtual machine/zone) benchmarking
* Linux1 (Ubuntu Linux in KVM virtual machine) server & benchmarking
* Linux2 (Ubuntu Linux in KVM virtual machine) benchmarking

The results are:

Source      Dest        Connection      Time          Req-Rep/Sec
------      ----        ----------      ----          --------
Mac         Linux1      1Gig Ethernet   5.038577      992.3
Mac         SmartOS1    1Gig Ethernet   4.972102      1005.6
Linux2      Linux1      Virtual         1.696516      2947.2
SmartOS2    Linux1      Virtual         1.153557      4334.4
Linux2      SmartOS1    Virtual         0.952066      5251.8
Linux1      Linux1      localhost       0.836955      5974.0
Mac         Mac         localhost       0.781815      6395.4
SmartOS2    SmartOS1    Virtual         0.470290      10631.7
SmartOS1    SmartOS1    localhost       0.374373      13355.7

localhost tests use 127.0.0.1

SmartOS has an impressive network stack. Request-reply times from one SmartOS machine to another are over 3 times faster than when using Linux under KVM (on the same host). This mightn’t make much of a difference to web requests coming from slow mobile device connections, but if your web server is making many requests to internal services (database, cache, etc) this could make a noticeable difference.

ZeroMQ logging for ruby apps

I’ve been thinking for a while about using ZeroMQ for logging. This is especially useful with trends towards micro-services and scaling apps to multiple cloud server instances.

So I put thoughts into action and added a logger class to the rbczmq gem that logs to a ZeroMQ socket from an object that looks just like a normal ruby logger: https://github.com/mattconnolly/rbczmq/blob/master/lib/zmq/logger.rb

There’s not much to it, because, well, there’s not much to it. Here’s a simple app that writes log messages:

Log Writer:

require 'rbczmq'
require_relative './logger'
require 'benchmark'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PUSH)
socket.connect('tcp://localhost:7777')
logger = ZMQ::Logger.new(socket)
puts Benchmark.measure {
 10000.times do |x|
 logger.debug "Hello world, #{x}"
 end
}

With benchmark results such as:

  0.400000   0.220000   0.620000 (  0.418493)

Log Reader:

And reading is even easier:

require 'rbczmq'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PULL)
socket.bind('tcp://*:7777')
loop do
 msg = socket.recv
 puts msg
end

Voila. Multiple apps can connect to the same log reader. Log messages will be “fair queued” between the sources. In a test run on my 2010 MacBook Pro, I can send about 13000 log messages a second. I needed to run three of the log writers above in parallel before I maxed out the 4 cores and it slowed down. Each process used about 12 MB RAM. Lightweight and fast.

Log Broadcasting:

If we then need to broadcast these log messages for multiple readers, we could easily do this:

require 'rbczmq'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PULL)
socket.bind('tcp://*:7777')
publish = ctx.socket(ZMQ::PUB)
publish.bind('tcp://*:7778')
loop do
 msg = socket.recv
 publish.send(msg)
end

Then we have many log sources connected to many log readers. And the log readers can also subscribe to a filtered stream of messages, so one reader could do something special with error messages, for example.