Matt Connolly's Blog

my brain dumps here…

A year with ZeroMQ + Ruby

It was a year ago since my first commit to rbczmq, a Ruby bindings gem for ZeroMQ, which wraps the higher level CZMQ library.

I don’t recall exactly how I first heard about ZeroMQ, but needless to say, I got interested and started reading up on it. At the time, I discovered there were three gems for using ZeroMQ in ruby:

  • zmq – the original official zmq library. This is old. It tracks ZeroMQ version 2 (currently version 4) and has not been updated for nearly 3 years.
  • ffi-rzmq – a binding using FFI, which is compatible with Ruby, Rubinius and JRuby.
  • rbczmq – a native extension (no JRuby support) binding using the czmq library.

I didn’t spend a lot of time with the zmq gem since it was so old. The ffi-rzmq gem worked well, but didn’t feel very ruby like in its interface. For example, when receiving a message with a C method you would pass a buffer as a parameter to receive the contents of the message and the returned value would be an error code. This is quite un-ruby-like: I would expect receive to return the received data or raise an exception for the error code in keeping with built in ruby socket/file i/o calls.

So I started to explore rbczmq. Initially, I wasn’t so interested in the CZMQ wrapping part, I just wanted something that was more ruby-like to use. And it was. And it was faster. And the CZMQ part actually helps too.

In ZeroMQ each message part, or “frame” is considered a message. So when you read multi-frame messages in ZeroMQ you need to check the “more” flag, and read the next part. CZMQ wraps this as a single message with a number of “frames”. rbczmq neatly exposes these as Ruby classes: ZMQ::Message and ZMQ::Frame. You can still send and receive raw frames (as strings), but the class is a nice wrapper.

And to boot, it turns out that it was way faster than the ffi gem. I seem to have lost track of the comparison I did, but I recall it was convincing.

What’s changed?

During this year, rbczmq has received a number of updates and new features, major ones including:

  • Upgrade to ZeroMQ 4
  • Upgrade to CZMQ 2
  • Support for SmartOS platform
  • Fixes to memory management

Major things still to do:

  • Add support for new authentication interface.
  • Ship binary gems (like libv8) to save compilation time on deploy / install.

Hard bits

The hardest bit of work I contributed to this project was fixing bugs in the memory management. In particular, CZMQ has specific rules about ownership of memory. Ruby is a garbage collected environment, which also has its own set of rules about ownership of memory. The two do not match.

Most calls to ZeroMQ are done outside of the Ruby “GVL” (global lock) which allows the ruby VM to continue processing ruby code in other threads while one is doing a synchronous/blocking read on a socket, for example. When you combine this with Ruby threads, things can get hairy. The solution was two-fold:

  1. Use an ownership flag. When ownership was known to be transferred to ZeroMQ, mark the ruby object as no longer owned by Ruby. This meant that Ruby garbage collection callbacks would know if they were ultimately responsible for freeing memory used by an object. There was also some tricky interplay between contexts and sockets, since a socket is owned by a context, and destroying a context also destroys the sockets, so a socket is only owned by ruby if it has not been closed and the context has not been destroy.
  2. A socket closing mutex: Socket closing and context closing are asynchronous. If a socket is still open when a context is destroyed, then all sockets belonging to that socket will be closed. This happens outside the Ruby GVL, which means that a race condition exists where the Ruby garbage collector may collect the socket while it is still closing. ZeroMQ socket close is not threadsafe, so a mutex was the only solution to make this safe.

Using a mutex for socket close may result in a performance hit for an application which opens and closes sockets rapidly, but from what I understand, that is a bad thing to do anyway.

Looking forward

I have a few projects in the wild now using the rbczmq gem, and am very happy with its stability and performance. I haven’t used all of the APIs in Anger (such as Loops or Beacons), but I’m sure the time will come. I look forward to another year of contributions to this project to keep it up to date with what’s happening in the ZeroMQ and CZMQ projects.

I’d love to hear from other people using this gem, so give me a shout!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: