Matt Connolly's Blog

my brain dumps here…

Monthly Archives: April 2014

A year with ZeroMQ + Ruby

It was a year ago since my first commit to rbczmq, a Ruby bindings gem for ZeroMQ, which wraps the higher level CZMQ library.

I don’t recall exactly how I first heard about ZeroMQ, but needless to say, I got interested and started reading up on it. At the time, I discovered there were three gems for using ZeroMQ in ruby:

  • zmq – the original official zmq library. This is old. It tracks ZeroMQ version 2 (currently version 4) and has not been updated for nearly 3 years.
  • ffi-rzmq – a binding using FFI, which is compatible with Ruby, Rubinius and JRuby.
  • rbczmq – a native extension (no JRuby support) binding using the czmq library.

I didn’t spend a lot of time with the zmq gem since it was so old. The ffi-rzmq gem worked well, but didn’t feel very ruby like in its interface. For example, when receiving a message with a C method you would pass a buffer as a parameter to receive the contents of the message and the returned value would be an error code. This is quite un-ruby-like: I would expect receive to return the received data or raise an exception for the error code in keeping with built in ruby socket/file i/o calls.

So I started to explore rbczmq. Initially, I wasn’t so interested in the CZMQ wrapping part, I just wanted something that was more ruby-like to use. And it was. And it was faster. And the CZMQ part actually helps too.

In ZeroMQ each message part, or “frame” is considered a message. So when you read multi-frame messages in ZeroMQ you need to check the “more” flag, and read the next part. CZMQ wraps this as a single message with a number of “frames”. rbczmq neatly exposes these as Ruby classes: ZMQ::Message and ZMQ::Frame. You can still send and receive raw frames (as strings), but the class is a nice wrapper.

And to boot, it turns out that it was way faster than the ffi gem. I seem to have lost track of the comparison I did, but I recall it was convincing.

What’s changed?

During this year, rbczmq has received a number of updates and new features, major ones including:

  • Upgrade to ZeroMQ 4
  • Upgrade to CZMQ 2
  • Support for SmartOS platform
  • Fixes to memory management

Major things still to do:

  • Add support for new authentication interface.
  • Ship binary gems (like libv8) to save compilation time on deploy / install.

Hard bits

The hardest bit of work I contributed to this project was fixing bugs in the memory management. In particular, CZMQ has specific rules about ownership of memory. Ruby is a garbage collected environment, which also has its own set of rules about ownership of memory. The two do not match.

Most calls to ZeroMQ are done outside of the Ruby “GVL” (global lock) which allows the ruby VM to continue processing ruby code in other threads while one is doing a synchronous/blocking read on a socket, for example. When you combine this with Ruby threads, things can get hairy. The solution was two-fold:

  1. Use an ownership flag. When ownership was known to be transferred to ZeroMQ, mark the ruby object as no longer owned by Ruby. This meant that Ruby garbage collection callbacks would know if they were ultimately responsible for freeing memory used by an object. There was also some tricky interplay between contexts and sockets, since a socket is owned by a context, and destroying a context also destroys the sockets, so a socket is only owned by ruby if it has not been closed and the context has not been destroy.
  2. A socket closing mutex: Socket closing and context closing are asynchronous. If a socket is still open when a context is destroyed, then all sockets belonging to that socket will be closed. This happens outside the Ruby GVL, which means that a race condition exists where the Ruby garbage collector may collect the socket while it is still closing. ZeroMQ socket close is not threadsafe, so a mutex was the only solution to make this safe.

Using a mutex for socket close may result in a performance hit for an application which opens and closes sockets rapidly, but from what I understand, that is a bad thing to do anyway.

Looking forward

I have a few projects in the wild now using the rbczmq gem, and am very happy with its stability and performance. I haven’t used all of the APIs in Anger (such as Loops or Beacons), but I’m sure the time will come. I look forward to another year of contributions to this project to keep it up to date with what’s happening in the ZeroMQ and CZMQ projects.

I’d love to hear from other people using this gem, so give me a shout!

Ruby Tuples (and file scanning)

I enjoyed Andrew Pontious’s recent episode of Edge Cases podcast talking about tuples. I’m doing a lot of Ruby these days, so I thought I’d add my two cents worth about using tuples in Ruby.

It’s true that there is no separate tuple class, but Ruby arrays can do everything that tuples in Python can do.

To assign two variables, you can do:

a, b = 1, 2

Which is equivalent to:

a, b = [1, 2]

Which is equivalent to:

a = 1
b = 2

Elements not present are treated as nil, so a, b = 1 assigns the value 1 into a and nil into b.

Functions can return arrays like so:

def f(x)
  [1, 2]
end

def g(x)
  return 1, 2
end  

The Ruby way to iterate a list of items is with the each method that takes a block:

[1,2,3].each { |x| puts x }

Calls the block 3 times with x having the values 1, 2 and 3 from the list. If these items are themselves arrays, then the items in those sub-arrays can be expanded out into the block variables, like so:

[[1,2], [3,4], [5,6]].each { |a, b| puts "a = #{a}, b = #{b}" }
# outputs:
# a = 1, b = 2
# a = 3, b = 4
# a = 5, b = 6

Hashes can also be enumerated this way, where each key value pair is represented as an array with 2 items:

{a: 1, b: 2, c: 3}.each { |key, value| puts "#{key} => #{value}"}
# outputs:
# a => 1
# b => 2
# c => 3

Python’s list comprehension is really great. Where in python you might write the following to select only items from a list given some condition determined by the function g(x), and return the value f(x) for those values:

results = [f(x) for x in source_list if g(x)]

Ruby achieves the same with select and map methods, which can be composed in either order according to your needs. The Ruby equivalent would be:

results = source_list.select { |x| g(x) }.map { |x| f(x) }

Python’s list comprehension can only do these two things, in that order. By making the select step and the map steps separate in Ruby, they can be composed in any order. To reverse the map and select order in Ruby:

results = source_list.map { |x| f(x) }.select { |x| g(x) }

This is not so easy in python:

results = [y for y in [f(x) for x in source_list] if g(y)]

Ruby also contains many more useful operations that can be done on any enumerable sequence (for example readlines from a file), just take a look at the Enumerable module docs: http://www.ruby-doc.org/core-2.1.0/Enumerable.html

So I’ve got a bit off the tuple track, so I’ll finish with yet another tangent relating to the podcast episode: Deep searching a file hierarchy for files matching an extension. Try this out for concise:

Dir.glob("**/*.json")

To return an array of all the .json files anywhere under the current directory. Ruby is full of little treasures like this.

I used to do quite a bit of scripting in Python until I learnt Ruby. I’ve never looked back.