d: statsd, metrics, and measuring what you do

A long time ago in web years was written a blog post “Measure Anything, Measure Everything” by the devs at Etsy. It got me thinking about this issue, and it’s been really critical to me ever since. Years ago Seth Godin advised me that you have to measure in order to understand your effectiveness. And I’ve since found that whether that’s big or small, measuring removes anxiety. You know. You don’t have to guess. It’s either working or it’s not. And… it’s amazing how often people guess wrong. Also, people don’t believe that it is possible to measure a lot of things. But I’ve seen some great examples of making it possible to measure even really difficult things (like how you’re feeling) and reflect that data in a way that makes it clear.

A stat example

In this specific case… I loved that fact that UDP keeps things loosely coupled, and that statsd rolls up measurements so that the raw data isn’t overwhelmingly large.

And last, I liked the fact that the folks at Metrics were friendly and helpful, even though they’re running pretty fast to get Metrics where they want it to be.

This is a brief guide to the steps I took to get a Rails 3 app sending notifications to statsd running on Ubuntu in the EC2 cloud, and pointed it at Metrics. I borrowed knowledge from here for the first steps, but I didn’t want some of the stuff that follows, and the recipes have grown out of date a bit, so realize that you’ll have some digging to do to get it to run. Generally, I didn’t find all that much on the topic of how to set this up so, maybe this will save another busy developer some time.

Note that this fires up a small EC2 instance, so you’ll have to pay for it…

download and install the EC2 API tools (e.g., into ~/tools)
generate a X.509 certificate (both public cert-…pem and private pk-…pem files) on the AWS Security Credentials page
generate a keypair (statsd_setup.pem) in the AWS Management Console
I stored the key files in ~/.ec2/ locally.

$ sudo chmod 600 ~/.ec2/statsd_setup.pem # set appropriate permissions on private key file
The EC2 API tools require certain environment variables to be set. I put the following in a local file (e.g., ~/bash/statsd_setup.sh):

export EC2_HOME=~/tools/ec2-api-tools-
export EC2_PRIVATE_KEY=~/.ec2/pk-statsd_setup.pem
export EC2_CERT=~/.ec2/cert-statsd_setup.pem
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
export PATH=$PATH:$EC2_HOME/bin

$ source ~/bash/statsd_setup.sh

Finally, start the instances and modify some basic security settings:

$ ec2-run-instances ami-06ad526f -k statsd_setup # start a new instance with a recent Ubuntu 11 image
$ ec2-authorize default -p 22 # permit SSH
$ ec2-authorize default -p 80 # permit HTTP
$ ec2-authorize default -p 8125 -P udp # statsd will listen here

Then I logged into the instance once it was running: ssh -i ~/.ec2/statsd_setup.pem ubuntu@your_instance_here.amazonaws.com

Next, node and npm.

Install the dependencies:

sudo apt-get install g++ curl libssl-dev apache2-utils
sudo apt-get install git-core
sudo apt-get install make

When all done running…

mkdir dev
cd dev
git clone git://github.com/ry/node.git
cd node
sudo make install

If all went well node is installed. I smoke tested by simply typing the command “node”, and sure enough I got the prompt.

Since this is a test box, I took over /usr/local

sudo chown -R `whoami` /usr/local

and then installed npm in classic fashion: curl http://npmjs.org/install.sh | sh

next I moved back out to dev (cd ../)

And cloned statsd. Since I’m using Librato’s fork: git clone git://github.com/librato/statsd.git

I made a copy of the config example, and copied the settings I wanted in there.

I ran (as usual) npm install to get the dependencies installed, and then I fired it up: node stats.js myconfig.js


Locally, I fire up irb, where the gem statsd-ruby is already installed. I require ‘statsd’ (be warned, there’s more than one, and the namespaces collide), and fire off a few stats statsd = Statsd.new(your_host_here), and then statsd.increment ‘test’.

I already had my Metrics account, so I logged in and voilà, stats.

Next step wiring this into a Rails app. I create an initializer called notifications.rb. Inside is the code below cribbed mostly from here:


Which fires off all sorts of great info to statsd when you fire up the app and hit a few pages. Also, $statsd is available all over the place so you can measure pretty much anything going on in your app. I was surprised that there isn’t a simpler setup for this, but as I write this I tried Heroku, and Engineyard Cloud beta for node, and asked over at no.de and nodejitsu, and none of them support the standard port, or the UDP routing, or several of these requirements. It seems like this is an atypical scenario for now. Too bad, because it’s a great way to start measuring everything. A few months might change all of this, but for now that’s the state of things, so off to EC2 I went.

Published by Daniel

In the 4th grade Daniel was asked to step in into the hall because he was coaching fellow students on a similarity assignment he provided the class. Since then, Daniel’s passion for helping people has propelled him to be a creative developer of software and systems, and a team coach who excels at successfully delivering business value. Daniel founded QuietlyHelping in 2009 in order to have a direct hand in helping families with the struggles of debilitating disease. QuietlyHelping is an organization of compassion and empathy that brings ease, comfort, smiles, laughter, healing, and love to people battling cancer and other diseases. Spare time activities include cycling, occasional work as a musician, woodworking, photography, and reading books he never got around to in school. Unsurprisingly, Daniel is able to cook minute rice in just under thirty seconds.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s