Matt Connolly's Blog

my brain dumps here…

Comparing Amazon EC2 to Joyent SmartOS

Recently, I’ve been using Amazon web services (EC2, especially) quite a bit more at work. At home, I still use OpenIndiana, so I’ve been really interested in comparing Joyent’s offerings against Amazons first hand. In particular, my tasks I have in Amazon’s cloud always feel CPU bound, so I’ve decided to do a comparison of just CPU performance, giving some context to Amazon’s jargon ECU (Elastic Compute Unit) by comparing it with a Joyent SmartOS instance, as well as my MacBook Pro, iMac and OpenIndiana server.

So I spun up a Joyent Micro SmartOS instance and an Amazon EC2 linux Micro and small instances.

Joyent startup is impressive. The workflow is simple and easy to understand. I chose the smartosplus64 machine just because it was near the top of the list.

Amazon startup is about what I’ve learned to expect. Many more pages of settings later we’re up and running.

Installing ruby 1.9.3 with RVM

Ubuntu linux has fantastic community support, and many packages just work out of the box. Following the RVM instructions was easy to get it installed.

SmartOS, like OpenIndiana often requires a bit more work.

I made this patch to get ruby to compile: https://gist.github.com/4104287
Thanks to this article: http://www.hiawatha-webserver.org/forum/topic/1177

A Simple Benchmark

Here’s a really quick ruby benchmark, that will sort 5 million random numbers in a single thread:

require 'benchmark'

array = (1..5000000).map { rand }
Benchmark.bmbm do |x|
  x.report("sort!") { array.dup.sort! }
  x.report("sort") { array.dup.sort }
end

I also tested my MacBook Pro, my iMac and my Xeon E3 OpenIndiana server to get some perspective.

Here’s the results:

Machine Benchmark (sec)
MacBook Pro 2.66gHz core i7 (2010) 86.99
iMac 24″ 2.5GHz core i5 (2012) 19.30
Xeon E3-1230 3.2GHz OpenIndiana server 35.57
Joyent EXTRA SMALL SmartOS 64-bit 55.10
Amazon MICRO Ubuntu 64-bit 361.42
Amazon SMALL Ubuntu 64-bit 123.69

Snap. Amazon is *SLOW*! And iMac the surprise winner!

And so what is this Elastic Compute Unit (ECU) jargon that Amazon have created? Since the Amazon Small instance is 1 ECU, we can reverse measure the others into compute units. And by converting their hourly price to a monthly price (* 24 hours * 365.25 days / 12 months), we can also determine the price per ECU:

Machine Benchmark (sec) $/hour ECUs $/month/ECU
MacBook Pro 2.66gHz core i7 (2010) 86.99 1.422
iMac 24″ 2.5GHz core i5 (2012) 19.30 6.409
Xeon E3-1230 3.2GHz OpenIndiana server 35.57 3.477
Joyent EXTRA SMALL SmartOS 64-bit ruby 55.10 $0.03 2.245 $9.76
Amazon MICRO Ubuntu 64-bit 361.42 $0.02 0.342 $42.69
Amazon SMALL Ubuntu 64-bit 123.69 $0.07 1.000 $47.48

Snap. Amazon is *EXPENSIVE*!

My laptop with 4 threads could do the CPU work of 5.7 small amazon EC2 instances, worth $270/month. And my Xeon box with 8 threads could do the work of 27.8 small instances, worth $1320/month. (I built the whole machine for $1200!!). Mind you, these comparisons are on the native operating system, but if you’re running a machine in house this is an option, so might be worth consideration.

I’ve read that comparing SmartOS to Linux in a virtual machine isn’t a fair comparison because you’re not comparing apples with apples; one is operating system level virtualisation (Solaris Zones), the other is a full virtual machine (Xen Hypervisor). Well tough. All I need to do is install tools and my code and get work done. And if I can do that faster then that is a fair comparison.

Conclusion

Joyent CPU comes in more than 4 times cheaper than Amazon EC2.

Amazon need to lift their game in terms of CPU performance. They offer a great service that obviously extends far beyond a simple CPU benchmark. But when you can get the same work done in Joyent significantly faster for the comparable price, you’ll get far more mileage per instance, which is ultimately going to save the dollars.

 

EDIT: 19/11/12: Joyent’s machine is called “Extra Small”, not Micro as I originally had it.

About these ads

17 responses to “Comparing Amazon EC2 to Joyent SmartOS

  1. Magnus Hedemark 19 November, 2012 at 03:32

    I’m looking at Joyent’s public cloud offering and don’t see a “micro” sized instance being offered. Is that the actual branding of the instance size you used? If so, was it hidden somewhere special on Joyent’s site?

  2. Brendan Gregg (@brendangregg) 19 November, 2012 at 07:34

    Thanks for sharing the results. Things should get even more interesting when you test multiple instances in parallel, as would happen for a busy ruby-based web server. For example, on my 1GB SmartOS instance:

    [root@smartos1 ~]# time ruby bench.rb
    […]
    real 0m58.313s

    Now two in parallel:

    [root@smartos1 ~]# time ruby bench.rb & time ruby bench.rb &
    […]
    real 0m57.810s
    […]
    real 1m0.415s

    The runtime (MAX) only increased from 58s to 60s. Try that on AWS. :-) I imagine a multi-bench.rb could also be written to use Thread.new() to run these in parallel, and also try higher counts (3, 4, …), to test CPU scalability (not just single-CPU speed).

    • mattconnolly 19 November, 2012 at 09:10

      Cheers. I actually deliberately wanted to focus on a single thread with max CPU since that’s indicative of the time to do one unit of work (rendering a html response for example). I’ll try some parallel stuff soon.

      The standard ruby implementation of threads is cooperative on a single CPU thread (pretty lame, I think). Hence the plethora of ruby hosting tools to help process requests in parallel (Passenger, Thin, Puma, Unicorn, to name a few). The JRuby implementation does ruby threads as real system threads.

      Oh, and I’m also super impressed with the analytics, I’ll do a write up about that soon too!

  3. Pingback: Comparing Amazon EC2 to Joyent SmartOS | My Daily Feeds

  4. tracker1 19 November, 2012 at 17:14

    Going to be expanding into using Node.js and MongoDB at work, and was considering Amazon’s cloud, despite Joyent being behind Node.js … rethinking this strategy now.

  5. Antony Brooke-Wood 19 November, 2012 at 19:40

    While I can appreciate the work you have done, this is an incredibly naive comparison. You have performed a single test at a single point in time and used this to draw some fairly strong conclusions. If you want to do something more comprehensive, you should have done multiple tests, multiple iterations of each of the tests and performed them at different times (cloud resources are not always consistent in their performance). Have a look at something like this:

    http://blog.scalyr.com/2012/10/16/a-systematic-look-at-ec2-io/

    Also, I am not sure that comparing your iMac to any cloud provider is really that useful: would you host a website from your house running on an iMac? A fairer comparison would be looking at the cost of some Dell or HP boxes instead – i.e. some actual enterprise servers.

    Anyway, nice to see someone else running OpenIndiana ;)

    • mattconnolly 19 November, 2012 at 21:12

      Hi Antony, thanks for your comments. Wow, you guys have put together an excellent investigation of I/O in various instances within EC2. You must have spent a lot of time as well as money on doing that one.

      I’d like to think simple rather naive. Frankly, I’d like to do more, but my time is limited – I’m not getting paid to do this.

      I’ve been using Amazon EC2 for a year now and I always find it feels slow and CPU bound. So I decided to do a simple CPU test. What exactly is an ECU? How does it relate to something that I understand (like my MacBook Pro or iMac machines that I use for development, so I have some reference for what will be slow or fast in production vs in development.) And given my interest in OpenIndiana, how does Joyent’s offering compare?

      I’d love to repeat this test over a longer period of time, and report more detailed results. Any tools/assistance you can suggest/offer would be greatly appreciated. And I encourage other readers to do their own tests and share.

      • Antony Brooke-Wood 19 November, 2012 at 21:16

        I wish I could claim credit for that last work, but it had nothing to do with me!

        For IO statistics, IOZone is pretty popular and easy.

        I haven’t used Joyent yet, but they look really interesting and it is good to see them contributing to OpenIndiana.

        Anyway, apologies if I came off rude – just wanted to warn of the consequences of drawing too many conclusions from a single test.

        I think the benefits of the cloud (regardless of the platform) are less to do with the cost (or even the performance) and more to do with agility, scalability etc.

  6. Ksec 19 November, 2012 at 21:53

    I have been asking, and wondering why would anyone want to use AWS. It is insanely expensive. It is Slow on CPU, Slow on I/O, Low on Bandwidth, Low on memory, and Zero Support. Compared to many other alternatives that gives you 3 – 4 times the on all the above plus better support for the same price. Even their High CPU and Memory plan is still on the expensive side, only the High I/O SSD offers some value. AWS works great when you are in the mid range of expanding, they you can calculate your usage and book their server in advance using Reserved Instances. Instagram and Netflix as example. But that is hassle you have to put though, there are alternatives that are priced and perform just as well without using Reserved Instances. And once you got past the mid range, a managed dedicated server and some instant cloud severs should be much more cost effective.

    Apart from being forced by VC ( because they are heavily invested in Amazon ) i could not understand why startup are choosing AWS at all.

    • Antony Brooke-Wood 20 November, 2012 at 08:56

      Well, I did a fairly comprehensive cost analysis and most cloud vendors are NOT that expensive when you compare them to hosting things internally. You need to factor in a whole range of things that people normally exclude: server purchase price; server maintenance/support; rack power; rack costs; Internet/network costs; cross-connects (cables between racks); and the cost of people to manage all of that.

      Also, don’t forget that servers don’t run by themselves … so you need to add switch capex; switch opex; router capex; router opex; firewall capex; firewall opex … and of course some network staff to run all of it. Now, if you were starting out fresh, would you want to buy al of that stuff, or just rent a VM from a cloud vendor? Especially when you could turn off the VM when not in use …

      Also, setting up something like regional/data centre redundancy is insanely easy with most cloud vendors. Doing it yourself involves replicating all of the equipment and staff I mentioned above and putting it in another location – it’s expensive, difficult and time consuming.

      I’m not saying AWS is the best or only vendor – it is simply the one that has been around the longest and the one I have the most experience with. Regardless, there are cases where the cloud may not make any sense – big databases, IO intensive work, even large file access (where offline storage like AWS Glacier is too slow). You need to do your homework and the answer is never simple or even straight forward to calculate.

  7. marianobe 20 November, 2012 at 04:46

    On a 512 MB Rackspace instance (64-bit, single cpu):

    real 1m12.716s

  8. Lewis Cowles (@LewisCowles1) 20 November, 2012 at 09:01

    In at $50 per month (greater than the figure you shared so there is margin) over 24-month Amazon would have only cost $1200 (50*24) so really is it just about speed? Your iMac equipment probably cost at least $999 each ( so we’ll round that up to $1000 each as an average, if you didn’t buy it new and haven’t invested in the whole iSuite). At $999 the apple devices don’t even come with tech support, not to mention a dedicated machine hard-line to the internet, or guaranteed updates, notifications etc, so the TCO (total-cost of ownership) for your Mac is much higher, think then on how much power you are using as well. I’d say Amazon looks pretty rosy then doesn’t it lol. It’s all relative, I am happy for you to feel that your tools are more successful, but I just wanted to point out there are many ways to judge the price-point of a product.

    This is like when people come to me and ask how their web-designer friends can offer $300-$400 websites, it’s a false economy, they don’t work and most people that are any good won’t even try to offer them, if websites or servers etc are rock-bottom cheap it’s ALWAYS because there is a catch (built by script kiddie with no marketing background, offline and useless to the rest of the world, painfully slow (in the case of amazon micro and small servers). Just try a re-do with XL high-mem servers for me, see how well Amazon does then). P.S. Amazon is billed in hours, unless you sign-up for a lengthier service anyway, so shut it down once your done testing and post the results, I’m sure you’ll change your mind.

  9. Antony Brooke-Wood 20 November, 2012 at 09:07

    There are a few calculators you can use to start looking at this. One of them is here:

    Here is an example cost calculation on cloud versus internal hosting that I did quickly:

    http://tcoservice.2ndwatch.com/AwsTcoServices/pdfData/956f1cc4-4e6b-c07e-13ed-957df89b75ba_1353365986956_pdf.pdf

    Obviously these vendors have a vested interest in making cloud hosting look cheaper, but they do at least cover all of the major areas (Servers, Storage, Network, Environment & Administration) and justify their reasoning. Again, the point isn’t to say that one is always better than another, just to reinforce that the calculation and cost comparison is not as simple as was represented above.

  10. Pingback: SmartOS News: Nov 21, 2012 – SmartOS.org

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 64 other followers

%d bloggers like this: