I’ve been thinking for a while about using ZeroMQ for logging. This is especially useful with trends towards micro-services and scaling apps to multiple cloud server instances.
So I put thoughts into action and added a logger class to the rbczmq gem that logs to a ZeroMQ socket from an object that looks just like a normal ruby logger: https://github.com/mattconnolly/rbczmq/blob/master/lib/zmq/logger.rb
There’s not much to it, because, well, there’s not much to it. Here’s a simple app that writes log messages:
Log Writer:
require 'rbczmq'
require_relative './logger'
require 'benchmark'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PUSH)
socket.connect('tcp://localhost:7777')
logger = ZMQ::Logger.new(socket)
puts Benchmark.measure {
10000.times do |x|
logger.debug "Hello world, #{x}"
end
}
With benchmark results such as:
0.400000 0.220000 0.620000 ( 0.418493)
Log Reader:
And reading is even easier:
require 'rbczmq'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PULL)
socket.bind('tcp://*:7777')
loop do
msg = socket.recv
puts msg
end
Voila. Multiple apps can connect to the same log reader. Log messages will be “fair queued” between the sources. In a test run on my 2010 MacBook Pro, I can send about 13000 log messages a second. I needed to run three of the log writers above in parallel before I maxed out the 4 cores and it slowed down. Each process used about 12 MB RAM. Lightweight and fast.
Log Broadcasting:
If we then need to broadcast these log messages for multiple readers, we could easily do this:
require 'rbczmq'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PULL)
socket.bind('tcp://*:7777')
publish = ctx.socket(ZMQ::PUB)
publish.bind('tcp://*:7778')
loop do
msg = socket.recv
publish.send(msg)
end
Then we have many log sources connected to many log readers. And the log readers can also subscribe to a filtered stream of messages, so one reader could do something special with error messages, for example.
