Matt Connolly's Blog

my brain dumps here…

Category Archives: Development

ZeroMQ logging for ruby apps

I’ve been thinking for a while about using ZeroMQ for logging. This is especially useful with trends towards micro-services and scaling apps to multiple cloud server instances.

So I put thoughts into action and added a logger class to the rbczmq gem that logs to a ZeroMQ socket from an object that looks just like a normal ruby logger: https://github.com/mattconnolly/rbczmq/blob/master/lib/zmq/logger.rb

There’s not much to it, because, well, there’s not much to it. Here’s a simple app that writes log messages:

Log Writer:

require 'rbczmq'
require_relative './logger'
require 'benchmark'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PUSH)
socket.connect('tcp://localhost:7777')
logger = ZMQ::Logger.new(socket)
puts Benchmark.measure {
 10000.times do |x|
 logger.debug "Hello world, #{x}"
 end
}

With benchmark results such as:

  0.400000   0.220000   0.620000 (  0.418493)

Log Reader:

And reading is even easier:

require 'rbczmq'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PULL)
socket.bind('tcp://*:7777')
loop do
 msg = socket.recv
 puts msg
end

Voila. Multiple apps can connect to the same log reader. Log messages will be “fair queued” between the sources. In a test run on my 2010 MacBook Pro, I can send about 13000 log messages a second. I needed to run three of the log writers above in parallel before I maxed out the 4 cores and it slowed down. Each process used about 12 MB RAM. Lightweight and fast.

Log Broadcasting:

If we then need to broadcast these log messages for multiple readers, we could easily do this:

require 'rbczmq'
ctx = ZMQ::Context.new
socket = ctx.socket(ZMQ::PULL)
socket.bind('tcp://*:7777')
publish = ctx.socket(ZMQ::PUB)
publish.bind('tcp://*:7778')
loop do
 msg = socket.recv
 publish.send(msg)
end

Then we have many log sources connected to many log readers. And the log readers can also subscribe to a filtered stream of messages, so one reader could do something special with error messages, for example.

Building Ruby 2 in SmartOS

Installing Ruby 2 in SmartOS!

Ruby 2 has been out for a while, so let’s get it going in SmartOS!!

I’m working with the SmartMachine (base64 1.9.0).

Part 1: Requirements.

The SmartMachine has a whole bunch of useful packages already installed. Here are the additional packages needed to build Ruby 2.0.0 in SmartOS:

# pkgin install build-essential gcc47 gcc47-libs libyaml 

Part 2: Configure hacks

There are a few issues with Ruby 2.0.0, but fortunately all of them have a command line workaround.

1. Ruby bug #5384 https://bugs.ruby-lang.org/issues/5384

Actually an upstream bug in (Open)Solaris/Illumos/OpenIndiana/SmartOS. Workaround is to add `ac_cv_func_dl_iterate_phdr=no` to the configure line.

See also: https://www.illumos.org/issues/1587

2. Ruby bug #8071 https://bugs.ruby-lang.org/issues/8071

Non-portable code in ruby’s configure scipt. Easy workaround, prepend the configure command with a shell that can handle the current state of the configure script, ie: `bash`. (A fix has already been submitted, ans should be in the next ruby patch.)

3. Ruby bug #8268 https://bugs.ruby-lang.org/issues/8268

“-ggdb3″ C flags issue (logged with Ruby, but not necessarily actually a bug in ruby. Please help if you can on this one!)

Workaround 1: Add `debugflags=”-ggdb”` to the configure line. Caveat: It appears this will add gdb debug info to built ruby binaries, which may not be desired.

Workaround 2: Add `CFLAGS=”-R -fPIC”`. This introduces a make error. Missing function ‘signbit’. Boo.

Workaround 3: Add `CFLAGS=”-R -fPIC” rb_cv_have_signbit=no`. Works!

So with these three taken care of, the following command line will configure ruby-2.0.0-p0 to compile on SmartOS (in 64 bit):

$ bash ./configure --prefix=$PREFIX --with-opt-dir=/opt/local --enable-shared ac_cv_func_dl_iterate_phdr=no CFLAGS="-R -fPIC" rb_cv_have_signbit=no
$ make && make install

And there we have it. Ruby 2.0.0-p0 building in SmartOS. Next challenge, making use of those built in DTrace probes…

iOS Core Data: Group by and count results

I looked in the usual places and couldn’t find any decent examples of using core data to do a group by and count. So I’m making one here.

Let’s suppose you have a bunch of Records, and each Record has a Status attribute among others, and you want a break down of how many have each Status. The SQL would be:

SELECT `Status`, COUNT(*) FROM `Records` GROUP BY `Status`

How do we do this with Core Data?

start with a fetch request:

NSFetchRequest* fetch = [NSFetchRequest fetchRequestWithEntityName:@"Record"];

set up an attribute description and expression description for the two values we want in the results:

NSEntityDescription* entity = [NSEntityDescription entityForName:@"Record"
                                          inManagedObjectContext:myManagedObjectContext];
NSAttributeDescription* statusDesc = [entity.attributesByName objectForKey:@"status"];
NSExpression *keyPathExpression = [NSExpression expressionForKeyPath: @"url"]; // Does not really matter
NSExpression *countExpression = [NSExpression expressionForFunction: @"count:"
                                                          arguments: [NSArray arrayWithObject:keyPathExpression]];
NSExpressionDescription *expressionDescription = [[NSExpressionDescription alloc] init];
[expressionDescription setName: @"count"];
[expressionDescription setExpression: countExpression];
[expressionDescription setExpressionResultType: NSInteger32AttributeType];

tell the fetch request to only fetch these, and group by the status attribute description

[fetch setPropertiesToFetch:[NSArray arrayWithObjects:statusDesc, expressionDescription, nil]];
[fetch setPropertiesToGroupBy:[NSArray arrayWithObject:statusDesc]];
[fetch setResultType:NSDictionaryResultType];
NSError* error = nil;
NSArray *results = [myManagedObjectContext executeFetchRequest:fetch
                                                         error:&error];

And voila. The result is an array of dictionaries, each with “status” and “count” keys and corresponding values. Even though the group by guarantees the grouped values be unique, there could be more than one grouped column, so it does make sense.
If you think that’s a lot of code, it is. Let’s compare that with the Rails solution:

Record.count(:group => :status)

And that returns a dictionary (Hash), keyed by status, with the counts in the values

Rubymine theme convert into Xcode theme

Rubymine has lots of great themes. Xcode doesn’t have so many. So I made a gem to to convert them.

Install like so:

$ gem install rubymine2xcode-theme

Save or find your rubymine theme, and run the gem like so:

$ rubymine2xcode theme.xml

Where ‘theme.xml’ is a path to the rubymine theme. The resulting Xcode theme file will be saved in the user’s Library where Xcode can find it. Relaunch Xcode and it should appear in the theme list!

TTCP, in Ruby

I’ve used the TTCP tcp test program from time to time, and am at present looking at some networking in Ruby. So why not have a look at porting that to ruby? So I did.

This has been built as a gem, which an executable ‘ttcp’ that will install in your gem’s bin folder. You can get the gem from here: http://rubygems.org/gems/ttcp

Or type: `gem install ttcp` at your terminal.

Source code is released under MIT license, and available on github: https://github.com/mattconnolly/ttcp

So far, I’ve tested it out on Mac and OpenIndiana in ruby 1.8.7, 1.9.3 and JRuby 1.6.5. I can’t seem to run the tests in JRuby, but it appears to work anyway.

Enjoy.

Using RVM with Jenkins

I’ve recently set up RVM running with Jenkins, and finally have it all working just the way I want.

I’m running on OpenIndiana and have Jenkins running as its own user. I ssh in as that user and install rvm for that user and verified that I can run ruby tests with rvm from the terminal.

There’s a “RVM plugin” (version 0.1) for Jenkins which has great promise – I like the idea of running all build steps with RVM setup. However, I couldn’t get it to find my rvm installation no matter how much stuffing around with environment variables I did. So I’m not using it for now.

The trick was to use a “Execute shell” build step and use a script like so:

#!/bin/bash -e
export TERM=xterm
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"
# Use the correct ruby
rvm use 1.9.2@gemset --create
# Do any setup
# e.g. possibly do 'rake db:migrate db:test:prepare' here
bundle install
# Finally, run your tests
rake test
# and build the gem
rake build
# and install the gem locally
rake install

This was for a ruby gem project which installs that gem in the named rvm gemset. Other projects can use this gem simply by using that same gemset.

Note also the line: “export TERM=xterm” – rvm is designed to run in a terminal where the TERM environment variable is set. Apparently there was a change to gracefully ignore its absence or be able to use TERM=dumb for no colour, (github issue 698), but that doesn’t appear to be working for me in rvm 1.10.2.

Good luck.

Using rvm on OpenIndiana

OpenIndiana is a fork of Sun’s OpenSolaris before it was killed by Oracle. There’s quite a few commands in there that are different from linux, and there’s been a bit of discussion on the openindiana mailing list about whether commands should be more linux like or not.

In any case, there’s the built in commands in /usr/bin and the linux-like versions in /usr/gnu/bin. This is what you need in your path to use the linux versions, which I have, but I have it at the end of my path.

Unfortunately that’s no good for rvm. Well at least for the rvm upgrading part.

You can put /usr/gnu/bin at the beginning of your PATH. Or if you don’t want to do that, you can:

$ # installing:
$ PATH=/usr/gnu/bin:$PATH bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
$ # upgrading
$ PATH=/usr/gnu/bin rvm get latest

Rails data backups, independent of database

With my continuing my interest in redmine / ChiliProject, I’m really wanting a way of backing up my data that is database independent. I did a bunch of searching around, and the best solution I found was here. However, it appears a little out dated, and didn’t work in Rails 2.3 for a few reasons:

  • There isn’t always a 1:1 mapping between tables and models. (Example: many-many relationships create extra tables).
  • The excluded tables list was outdated (probably from an earlier Rails)[1]

There were also some comments about the export being quite slow. This could have been because every record of every table was loaded up in an ActiveRecord model instance before being dumped to the YML files.

I addressed these issues and made a new version. Can’t attach a file to this blog, so it’s on pastie for now: http://pastie.org/1877530

I’ve tested this getting data out of mysql and into sqlite3 and that works just fine. This is really handy for getting the data into a test environment quickly without having to set up more mysql databases, users, etc.

[1]: I’m testing this for Rails 2.3.11 as Redmine and ChiliProject are Rails 2.3.x apps.

If this saves you some time, or could be improved, please let me know.

Using git to move from redmine to chiliproject

I originally got onto Redmine because of its great out of the box support for git as well as remote subversion repositories. I’m keeping an eye on the “chiliproject” fork of “redmine” at the moment.

I have a redmine installation which includes a few custom modifications. So I have my “local” branch, which forked from redmine’s 1.1-stable some time ago, but when there’s a new version to do, I merge the new redmine version into my local, giving me, for example, redmine 1.1.3 + my changes.

So in order to evaluate chiliproject, and to see if my custom changes would go across, I figured I’d just add the chiliproject git remote and merge it across. Easy right? Well, kind of.

Simply doing this, didn’t work:

$ git remote add chiliproject git://github.com/chiliproject/chiliproject.git
$ git fetch chiliproject
$ git merge chiliproject/stable

This resulted in a *whole bunch* of merge conflicts; many of which actually looked to me like they should have been resolved automatically.

It appears those two repositories have a lot in common, but the main branches I’m interested in (redmine/1.1-stable and chiliproject/stable) have quite a distant common ancestor, which is what git will use to do the merge.

So, check this out:

$ # how many changes have I made from `redmine/1.1-stable`... out of curiosity?
$ git diff redmine/1.1-stable | wc
    3340   11917  126173
$ git branch chili-stable chiliproject/stable
$ git checkout chili-stable
$ git merge -s ours redmine/1.1-stable
$ git branch local_chili local_redmine
$ git checkout local_chili
$ git merge chili-stable
<--snip-->
$ # merge succeeded, how many changes do I now have from `chiliproject/stable`?
$ git diff chiliproject/stable | wc
    3340   11917  126173

And Voila! All of my changes since redmine/1.1-stable can be applied to chiliproject/stable without any conflict. So, when two git repositories are related by difficult to merge, let `git merge -s ours` come to the rescue!

Mac OS X SnowLeopard upgrading RubyGems

I looked around for a while on how to upgrade ruby gems to >= 1.3.6 so I could install rails 3 on Mac OS X 10.6 Snow Leopard.

It’s actually really easy:

$ sudo gem install rubygems-update
Successfully installed rubygems-update-1.6.2
Successfully installed rubygems-update-1.6.2
1 gem installed
Installing ri documentation for rubygems-update-1.6.2...
Installing RDoc documentation for rubygems-update-1.6.2...
$ sudo gem update --system
Updating RubyGems
Updating rubygems-update
Successfully installed rubygems-update-1.6.2
Updating RubyGems to 1.6.2
Installing RubyGems 1.6.2
RubyGems 1.6.2 installed

=== 1.6.2 / 2011-03-08

Bug Fixes:

* require of an activated gem could cause activation conflicts.  Fixes
  Bug #29056 by Dave Verwer.
* `gem outdated` now works with up-to-date prerelease gems.

------------------------------------------------------------------------------

RubyGems installed the following executables:
	/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/gem

$ gem -v
1.6.2

Done. Easy as.