Workling, BackgroundJob, and some config file and UTC caution.

One of the features of the software I’m building is the ability to upload images and other binary files. Like so many other Rails apps we’re using attachment_fu. So far so good. We expected the “client” could pull these assets when they needed them and store them wherever they desired. And the developers rejoiced.

A follow up story required that we move those assets over to the website via SFTP. Since we use Capistrano I wrapped Net::SFTP (a Capistrano dependency). We wired it into the Model via after_save, and after_destroy, and again the developers rejoiced.

I decided to move the “hard coded” user, pass, etc. stuff to a config file, so created a YAML file and added the aforementioned sftp stuff, and some other application specific that was crufting up the place. I created an initializer to load the file, and again the… yeah, ok, you got it. Soon after there was some stuff that was environment specific so I created the 3 basic environment sections (development, production, and test) and moved along. I figured I could test for them in just one spot (for now) and all would be good. I’ve since had a change of heart, but no matter for now.

A short while later I accidently demo’ed the new transfer code with a very large image file, and was less than impressed by the way it went. Since Rails is single threaded (still, for the moment, although about to change as I write this) the file transfer absorbed copious resources on this somewhat long lived transfer. Genius that I am, I said we should run the transfer as a background process, and since I know we have other similar requirements coming up Real Soon Now(tm), decided to implement a queue as well. After looking around the landscape I decided on Workling and BackgroundJob. It looked like they would play nicely, and keep thing clean and straightforward.

In go the plugins, get things setup and migrated, create a worker class, and fire off the MyWorker.asynch_test_method. Cool! I see stuff in the database… hmmm nothing happened. I’m going to skip through the process at this point, and just discuss what we (I was pairing with Evan during most of this) found.

One of the recent changes in Rails (we started with 2.1 and have since moved to 2.1.2 as I write this) includes control over Time Zones. We had talked and played around with this a bit and decided that for our purposes UTC was just fine, and so left the default setting. Stuff in the database gets stored as UTC and when everything works as expected is converted to a local time setting, if you set one. Here’s the first gotcha. Since this is fairly new, folks have liberally sprinkled libraries with calls to Time.now(). Which is fine for what it is, but at least with the version of Ruby we’re using (1.8.6 as I write this) returns local time… but the database is living over in UTC land. I wouldn’t notice if my time zone was Greenwich Mean, but over here on the East Coast of the US there’s a five hour difference. So we hunted down (not for the first time mind you) the use of Time.now in the BJ runnner.rb an changed it to Time.now.utc. One bug down.

The next one was bug in the bj_invoker.rb that is installed by Workling in the script folder. Stuff happens (and a patch is being submitted) but there’s a bit of meta-programming going on, and it was hard to find the places where STDOUT was being nullified, and because it’s trying to act as background or async process, exceptions are being swallowed. So no significant logging, and no exceptions. In the end we patched the runner code in BJ to display STDERR and STDOUT, and we wrapped a method call in a begin/rescue in Workling to enable some exception logging at least in this common case.

Having done that we found our error, fixed our bugs, and watched everything work. And the developers rejoiced.

The lessons so far: Developers do not pay enough attention to time zones so watch for gotchas like how time is being stored in your database vs. how you test for time stuff. Secondly, only the main path will be well worn. All others will have bumps. Workling is commonly used with Starling and so using with other, even supported libs may be a somewhat more bug laden experience than one would hope for. This wasn’t too bad, but the process was methodically annoying and to a degree, orthogonal to our goals.

So… next we have the config file caution. So there’s this nice thing known as the Rails environment. It’s a beautiful thing as it makes it easy to separate development from testing and production. You can point each environment to a unique database. Excellent! You can change key settings about caching, and reloading of classes that makes development smoother and production faster. Yay! So what’s the caution? While those three are standard folks make up others. In addition to those three we have two others which are not common. They serve they’re purpose, but they’re not conventional.

Today I had a deploy fail. Hmm, that’s odd, they’re usually so smooth. Migration failed… nothing strange in there. Hmm there’s a couple of lines about config files… so what it turned out to be were two plugins that each rely on a config file. In one case the developers did the right thing and when they could not find a config file section that matched the environment, shut themselves down and wrote about it in the log. Excellent! The second was seriously hard wired to the config file, and so tossed an unhandled exception when it couldn’t find the matching environment section. Worse, the config file isn’t necessary for the setup we used, and the docs say it can be removed. (It can’t, for the same reason). So an unnecessary config file, missing an unnecessary environment hung my deploy. Sad. But it made me realize that I was missing those sections in my own little config file, so I added them in there as well, and have a card to rewrite the loading code to ensure that we always do the right thing if the environment section is not there.

A coupe more… it’s worthwhile in your Capistrano deploy stuff to be able to run migrations with –trace turned on. You should have matching environments on your development machine as you do elsewhere… I know I do… now. Lastly, as much as possible have a match to your production environment where you can make a mess without consequence. We use VMWare to slice up a hefty machine, and it’s a great tool for the purpose. It doesn’t hurt that we use Engineyard as our host and so can use the Express vm for testing.

Just one more layer… one more layer.

Leaves begin to change color along the Swift River in the White Mountain National Forest in Albany, N.H., Wednesday, Oct. 1, 2008.(AP Photo/Jim Cole)So it’s looking a lot like this (an AP photo by Jim Cole) around here these days. The days are bright with autumn sunshine, the air is crisp and clean. I can’t say enough about this time of year… except when I get up to ride at 5:30am. This morning it was 32 degrees F… also known as freezing. And a bit breezy. The problem is that after a few hours it can be significantly warmer out, and you need to figure the layers and compressibility, so that you might be able to shed something later, and stick in a pocket.

This morning I guessed wrong. And I payed the price. We had a nice route going, where there is this nice steady climb. Especially this time of year, it kicks my butt. But that’s what I’m out there for, so that was fine. Sure, Gerry felt the need to inquire of my plans for succession at the top of a long climb, but fear not. I’ve been there before, just not quite under dressed for the weather like today.

nov_2_gerry_route.pngThe kicker is when I dress too lightly and then climb, it costs me. First, I’m cold starting the climb. I feel weak and small. Then I warm up, and start to sweat. All fine until you reach the usually joyful downhill. Now it’s just freezes that sweat to your body, and makes you feel exceptionally cold. In more serious weather it’s a great way to get hypothermic. Today, it was just annoying, but made me cut my intended ride short. I did find a balance again when we took a short break and I stood in the sun for a few minutes. The rest, the sun, and then a nice pace brought me back to comfortable enough, but the ride had been long enough and uncomfortable enough that I didn’t feel like continuing. 24 uncomfortable miles. Here’s the thing. Despite all the whining and all about my screwing up a ride, it was a glorious fall day for ride. The light is amazing, the leaves are spectacular, the neighborhoods we ride through are well kept and quaint. The folks I ride with all the time always have interesting topics and trivia to discuss, and never seem to have an agenda about well, just about anything on the rides. Just one more layer… one more layer.

(Two other mistakes I made today… thinking it would be warmer since it was “an hour later”, and “I should be wide awake” since it was “an hour later”. Not so much on either count.)