One of the features of the software I’m building is the ability to upload images and other binary files. Like so many other Rails apps we’re using attachment_fu. So far so good. We expected the “client” could pull these assets when they needed them and store them wherever they desired. And the developers rejoiced.

A follow up story required that we move those assets over to the website via SFTP. Since we use Capistrano I wrapped Net::SFTP (a Capistrano dependency). We wired it into the Model via after_save, and after_destroy, and again the developers rejoiced.

I decided to move the “hard coded” user, pass, etc. stuff to a config file, so created a YAML file and added the aforementioned sftp stuff, and some other application specific that was crufting up the place. I created an initializer to load the file, and again the… yeah, ok, you got it. Soon after there was some stuff that was environment specific so I created the 3 basic environment sections (development, production, and test) and moved along. I figured I could test for them in just one spot (for now) and all would be good. I’ve since had a change of heart, but no matter for now.

A short while later I accidently demo’ed the new transfer code with a very large image file, and was less than impressed by the way it went. Since Rails is single threaded (still, for the moment, although about to change as I write this) the file transfer absorbed copious resources on this somewhat long lived transfer. Genius that I am, I said we should run the transfer as a background process, and since I know we have other similar requirements coming up Real Soon Now(tm), decided to implement a queue as well. After looking around the landscape I decided on Workling and BackgroundJob. It looked like they would play nicely, and keep thing clean and straightforward.

In go the plugins, get things setup and migrated, create a worker class, and fire off the MyWorker.asynch_test_method. Cool! I see stuff in the database… hmmm nothing happened. I’m going to skip through the process at this point, and just discuss what we (I was pairing with Evan during most of this) found.

One of the recent changes in Rails (we started with 2.1 and have since moved to 2.1.2 as I write this) includes control over Time Zones. We had talked and played around with this a bit and decided that for our purposes UTC was just fine, and so left the default setting. Stuff in the database gets stored as UTC and when everything works as expected is converted to a local time setting, if you set one. Here’s the first gotcha. Since this is fairly new, folks have liberally sprinkled libraries with calls to Time.now(). Which is fine for what it is, but at least with the version of Ruby we’re using (1.8.6 as I write this) returns local time… but the database is living over in UTC land. I wouldn’t notice if my time zone was Greenwich Mean, but over here on the East Coast of the US there’s a five hour difference. So we hunted down (not for the first time mind you) the use of Time.now in the BJ runnner.rb an changed it to Time.now.utc. One bug down.

The next one was bug in the bj_invoker.rb that is installed by Workling in the script folder. Stuff happens (and a patch is being submitted) but there’s a bit of meta-programming going on, and it was hard to find the places where STDOUT was being nullified, and because it’s trying to act as background or async process, exceptions are being swallowed. So no significant logging, and no exceptions. In the end we patched the runner code in BJ to display STDERR and STDOUT, and we wrapped a method call in a begin/rescue in Workling to enable some exception logging at least in this common case.

Having done that we found our error, fixed our bugs, and watched everything work. And the developers rejoiced.

The lessons so far: Developers do not pay enough attention to time zones so watch for gotchas like how time is being stored in your database vs. how you test for time stuff. Secondly, only the main path will be well worn. All others will have bumps. Workling is commonly used with Starling and so using with other, even supported libs may be a somewhat more bug laden experience than one would hope for. This wasn’t too bad, but the process was methodically annoying and to a degree, orthogonal to our goals.

So… next we have the config file caution. So there’s this nice thing known as the Rails environment. It’s a beautiful thing as it makes it easy to separate development from testing and production. You can point each environment to a unique database. Excellent! You can change key settings about caching, and reloading of classes that makes development smoother and production faster. Yay! So what’s the caution? While those three are standard folks make up others. In addition to those three we have two others which are not common. They serve they’re purpose, but they’re not conventional.

Today I had a deploy fail. Hmm, that’s odd, they’re usually so smooth. Migration failed… nothing strange in there. Hmm there’s a couple of lines about config files… so what it turned out to be were two plugins that each rely on a config file. In one case the developers did the right thing and when they could not find a config file section that matched the environment, shut themselves down and wrote about it in the log. Excellent! The second was seriously hard wired to the config file, and so tossed an unhandled exception when it couldn’t find the matching environment section. Worse, the config file isn’t necessary for the setup we used, and the docs say it can be removed. (It can’t, for the same reason). So an unnecessary config file, missing an unnecessary environment hung my deploy. Sad. But it made me realize that I was missing those sections in my own little config file, so I added them in there as well, and have a card to rewrite the loading code to ensure that we always do the right thing if the environment section is not there.

A coupe more… it’s worthwhile in your Capistrano deploy stuff to be able to run migrations with –trace turned on. You should have matching environments on your development machine as you do elsewhere… I know I do… now. Lastly, as much as possible have a match to your production environment where you can make a mess without consequence. We use VMWare to slice up a hefty machine, and it’s a great tool for the purpose. It doesn’t hurt that we use Engineyard as our host and so can use the Express vm for testing.