Thursday, March 15, 2012

Make scheduled jobs in heroku safer for your wallet.

I recently checked my current usage on Heroku to find that I had used a much higher than expected number of dyno-hours. The project CheckTheSock.com is a RC aviation website that provides some weather data. The forecast data is pulled every hour via FTP by a rake task that is run with a scheduled job in Heroku. Normally this takes around a minute to run and uses about 13 or 14 dyno-hours in a month. Less than half way through this month I found that it had already burned through over 200 dyno-hours.

After digging into the problem I found that the FTP connection was hanging for some reason. After a little research I came up with a solution that works pretty nicely and for more than just hanging FTP connections.

It turns out that ruby's standard library includes a library named Timeout which provides a way to terminate the execution of a thread. Using this library it is very simple to set a time limit for a block of code in a rake test.

task :task_that_takes_a_long_time_sometimes do
  Timeout.timeout(30) do
    #The code that takes a long time to run sometimes
  end
end

That will raise a Timeout::Error if the code takes longer than 30 seconds to run. The error will cause the rake task to exit, preventing unexpected dyno-hour usage.

Wednesday, February 29, 2012

Rails A/B testing with Split on Heroku

I recently decided to start doing some A/B testing on www.checkthesock.com. The last time I tried A/B testing I used Google website optimizer, it worked fairly well but I did not like how difficult it was to setup a test. This time I looked around for A/B testing tools for rails. I ended up finding split and so far I have been very happy with the results. The documentation is fairly good but it did take a little fiddling to get it working on Heroku.

First off add the Redis to Go add-on in Heroku, the Nano plan is free and has been sufficient for my needs so far. Install redis on the dev server. I use a mac so that was as simple as "brew install redis" and then I followed the brew instructions for getting redis to start on login.

To install the split gem add the following to the Gemfile

 gem 'split', '0.3.3', :require => 'split/dashboard'

To set the development environment up in a way that is compatible to how Redis to Go is configured on Heroku add the following to config/environments/development.rb

ENV['REDISTOGO_URL'] = 'redis://localhost:6379'
Then create an initializer for redis in config/initializers/redis.rb and put the following in that file.
uri = URI.parse(ENV['REDISTOGO_URL'])
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password, :username => uri.user)
To configure split to use the applications redis configuration create config/initializers/split.rb and add the following.
Split.redis = REDIS

To setup the split dashboard add the following to config/routes.rb

mount Split::Dashboard, :at => 'split'
To add authentication to the dashboard add the following to config/initializers/split.rb
Split::Dashboard.use Rack::Auth::Basic do |username, password|
  username == 'username' && password == 'password'
end

Setting up an A/B test is super easy just follow the instructions in the split README.

Resources:

Note: These instructions are for rails 3.x, the listed resources do have instructions for earlier versions of rails.

Friday, February 3, 2012

Testing to ensure code gets cleaned up.

The other day I was working on a change on www.checkthesock.com that involved getting rid of some pages. It is generally a good idea to redirect those URL's to a relevant location so that search engines and people can get to a useful location. Eventually no one uses the old URL anymore and whatever code is redirecting it is just sitting around gathering dust.

I wanted to make sure I eventually cleaned up the code I had doing the redirection. In the past I have tried several different ways of remembering to do this; try to remember, put a note somewhere, or put a reminder in my calendar. None of those have ever ended up working out. None of those methods involve anything that is part of my normal working process.

The solution I came up with was to write a test that checks to see if the code is still around at some future date. I run my tests all the time as part of my normal workflow and I never release code without fixing a broken test. Now at some future point in time the test will break and I will not forget to clean up my unused code. Seems like a potentially useful pattern. In a few months, when my tests start breaking to remind me to clean stuff up, I will discover if it is useful or just really annoying.

Monday, January 23, 2012

Rails 3.2 development environment asset issue

I just tracked down and dealt with an issue I was having after upgrading www.checkthesock.com to rails 3.2. After the upgrade assets (css, javascript) were not updating unless I restarted the server process. Took me a while to track down the issue but I finally tracked it down to asset fingerprinting. Asset fingerprinting is off by default in the development environment but I had turned it on to debug an issue a long time ago.

If you are having similar problems make sure that you have the following line in "config/environments/development.rb"

  config.assets.digest = false

Friday, January 20, 2012

Rails 3.2 on Heroku asset:precompile issue

I just upgraded www.checkthesock.com to Rails 3.2 and found that the asset precompile step of the release was not working properly.

Here is the output I was getting:

-----> Preparing app for Rails asset pipeline
       Running: rake assets:precompile
       rake aborted!
       could not connect to server: Connection refused
       Is the server running on host "127.0.0.1" and accepting
       TCP/IP connections on port 5432?
       
       Tasks: TOP => environment
       (See full trace by running task with --trace)
       Precompiling assets failed, enabling runtime asset compilation
       Injecting rails31_enable_runtime_asset_compilation

The issue was rails was trying to connect to the database during the precompile process, and heroku is setup not to allow that. I am guessing that this issue is due to a change in the rails initialization setup in rails 3.2.

It turns out that the fix for this is very simple as long as you don't actually need to connect to the database during asset precompiling. All you need to do is add the following line to 'config/application.rb'

config.assets.initialize_on_precompile = false

Additional Resources

Monday, November 21, 2011

rescue_from temporary fix for rails 3.0 and 3.1

In rails 3.0 and 3.1 rescue_from will not successfully rescue a ActionController::RoutingError (Bug Report). Based on comments in the bug report here is the solution I came up with. A way that handles this bug temporarily and once it is fixed all that needs to be cleaned up is a single line in the routes.

First set up a simple errors controller:
class ErrorsController < ApplicationController
  def missing_page
   render :status => :not_found
  end
end

The view for missing_page is intended to be shown to visitors as the 404 page, it uses the application layout as set up here but you can set :layout to false or whatever other layout you might wish to use.
In the application controller add the following lines:
rescue_from ActiveRecord::RecordNotFound, :with => :missing_page
rescue_from ActionController::UnknownAction, :with => :missing_page
rescue_from ActionController::RoutingError, :with => :missing_page
The first two lines work as expected, the line with ActionController::RoutingError does not work correctly though, it will start working once the bug is fixed.

The temporary fix for routing errors not being handled properly is to add the following line as the last route in the route file.
match '*path', :to => 'errors#missing_page'

Once the bug is fixed all you need to do to clean this up is remove the line that was placed at the end of the route file.

Thursday, September 8, 2011

Rails Code QA Update

I just released an updated version of Rails Code QA which now supports rails 3.0 and 3.1 as well as ruby 1.8.x and ruby 1.9.x


Installation
If you are using ruby 1.8.x all you need to do to install rails_code_qa in your rails 3.x app is add the following line to your Gemfile.
gem "rails_code_qa"


If you are using ruby 1.9.x you will also need to add the following to your test/test_helper.rb file
require 'simplecov'
SimpleCov.start 'rails'


Code Coverage
Rails Code QA takes care of picking which code coverage tool to use based on the version of ruby you are using. If you are using ruby 1.8.x code coverage is calculated using rcov and if you are using ruby 1.9.x coverage is calculated using simplecov.

Static Code Analysis
Rails Code QA also runs flog and flay to help you find other issues in your apps code.


Resources

Monday, March 28, 2011

Sitemap testing with doublecheck

I hate releasing a new version of a site and finding out a few days later that I broke some of the pages. So I wrote the Doublecheck gem to help with post release testing.

Doublecheck

Doublecheck pulls a sitemap file and checks each URL listed, outputing a list of URL's for each HTTP status code that is encountered.

Installation

$ gem install doublecheck

Usage

$ doublecheck http://example.com/sitemap.xml

Notes

Get the code: github.com/nathanhumbert/doublecheck

Doublecheck is pretty simple at this point, it does one request at a time, so large sites may take a substantial amount of time to process.

Google's webmaster tools will eventually let you know if one of your pages has a problem, but it isn't a particularly fast way to find issues on your site.

Tuesday, February 15, 2011

Using Sinatra with Bundler to deploy on Heroku

I decided to try using sinatra for a small project I started the other day. I normally work in rails and host my personal projects on heroku. Since rails 3 came out I have also gotten hooked on bundler. I could not find any single source that documented how to use all of these together nicely. Here is how I got it to work.

Gemfile

The gemfile is pretty basic, set a source for gems and include the sinatra gem.
source :rubygems
gem 'sinatra'

config.ru

This is the rackup config file which sets the app up to run in rack.
require 'rubygems'
require 'bundler'

Bundler.require

require './myapp'

run MyApp

myapp.rb

This is the sinatra file. Note that I defined a class (MyApp). This is very important because our config.ru needs the class as a handle for the 'run MyApp' call. An alternative would be to use "run Sinatra::Application" in your config.ru file.
class MyApp < Sinatra::Base
  get '/' do
    'Hello world!'
  end
end

Running locally

First make sure rack is installed, then run the following commands in the root folder and the app should be up and running locally.
bundle install
rackup

Deploying to Heroku

Just run the following commands in the root folder and the app should be up and running on Heroku.
git init
git add .
git commit -m 'Initial version of MyApp'
heroku create
git push heroku master

Additional resources

Sunday, January 23, 2011

Easy data caching in Rails 3

Three common forms of caching

There are 3 types of caching that are very commonly talked about in the rails community; page caching, action caching, and fragment caching. All of these are well documented and are great ways to cache varying amounts of content for your pages.

Is that all the caching I need?

Caching at the view level is great but it isn't necessarily the best fit for every situation. In some cases it may be easier to cache data at a lower level. Perhaps you want to cache the results of an external data request for a given period of time. None of the view caching methods would be a really great fit.

Lower level data caching

Rails 3 actually has a great method for caching data at a lower level than the view. It even uses the same caching store as action and fragment caching. ActiveSupport::Cache::Store allows you to store any serializable Ruby object. Rails even provides an already initialized cache store via Rails.cache.

Example usage and code

I used this on www.checkthesock.com to cache METAR queries. I was using current_metar a gem I built that pulls metar data from ADDS. I made a model called CachedMetar to return cached results to the rest of my application.


Code walk through

This model basically returns the result of the block argument to Rails.cache.fetch the first time it is called. Any time it is called within the next 5 minutes it returns the value that was cached the first time. Everywhere I had previously been calling CurrentMetar::Metar.get_metar("ICAO code") I switched to call CachedMetar.metar("ICAO code") and that was all it took.

Testing

Testing is actually surprisingly easy. I used shoulda and mocha and here is what I ended up with.


Reference

Friday, January 21, 2011

SEO friendly maps with jQuery

In my free time I am working on a social networking site for RC pilots. I have information pages for various RC airfields and I was looking for a good way to display a listing of the fields. Putting markers on an embedded google map is a great way to display this type of data to users. The method I have seen recommended for this is to make separate list and map views. I don't really think that is a great user experience though.

The main reason making two pages is suggested is that maps require many lines of code and JavaScript functions that will dilute your content. The solution I arrived at was to use as little javascript as possible to generate my map. I ended up accomplishing this by using javascript to parse the content of the page to generate the markers.

To be able to easily parse out the data I wanted to display on the map I ended up making my own microformat. I put a class of field on the div that contained each field I was showing on the page. In that div I put the content that I wanted to display in the information popup for each marker. Then in that div I put two spans that contained the latitude and longitude and gave them each a class. The results ended up looking like this:


Then I wrote some javascript to parse this microformat and generate markers for my map. I used jQuery selectors to select each field in the html. For each field the latitude, longitude, and html contents are selected. Then the add_marker method is called to setup the marker with that data. After the marker is created I hide the field div.


Users with javascript will see a map with a bunch of markers that have an information bubble when clicked on. Users that do not have javascript enabled will see a list of field information. The textual content is the same for both users, the users with javascript just get an improved experience.

Go to www.checkthesock.com/fields to see the end result.

Monday, September 20, 2010

Getting over assert_difference

For quite a while now I have been using assert_difference or its close cousin from shoulda should_change. When I recently updated to a recent version of shoulda I noticed that they had deprecated should_change. The note in the shoulda documentation says the following: "This macro was deprecated because these tests aren’t as valuable as alternative tests that explicitly test the final state."

Ok, now what to do? assert_change is super easy to use, but does it really ensure that the desired action is actually happening? For example:

This example test asserts that the number of User objects in the database has increased by 1. No real guarantee that it was actually the User object we were interested in creating.

Lately I have been trying to be a little more explicit about what is actually happening. It is taking a little while to get used to doing things this way but the results have been satisfying. Here is an example:

This example test asserts that a User object is getting assigned to the @user instance variable and that User object is no longer a new record.

Wednesday, September 8, 2010

Code Readability: Quick, Millions or Billions?

3193490646

It is really easy to get the order of magnitude wrong on a number in code. I had settled into the habit of carefully counting my digits to make sure it was right until one of my co-workers pointed out a neat feature of ruby.

Ruby allows underscores to be inserted into numbers between any two digits. The underscores are purely cosmetic and do not affect the value of the number. This feature allows a developer to use underscores as an accountant might use commas.

3193490646      # standard representation of a number in ruby
3_193_490_646   # same value, much more readable

One less excuse for order of magnitude errors when using ruby.

A little looking around did not find any similar functionality in other languages, I checked into C/C++, Smalltalk, Python, Perl, PHP, Java, Javascript, Lisp, and Erlang. I am very curious to know if anyone knows of another language or particular implementation of one of the listed languages that supports similar functionality.


Thanks to Dallas Reedy for introducing me to this awesome feature of Ruby

Sunday, April 4, 2010

Rails Code Stats gem

When rails 2.2 came out it added some syntactic sugar by implementing a declarative block syntax for tests. The syntax for this looks like this:

test "make sure something really critical happens" do
  #code that tests for something really critical
end

While this syntax update is great some of the tools that come with rails still haven't been updated to handle it nicely. The stats rake task which outputs basic code statistics such as lines of code and methods per class still doesn't count these tests. In the past I just copied the rake tasks code and made a new rake task that could handle the new syntax as well as the format shoulda uses. After I started playing around with rails 3 and figured out how to put rake tasks in a gem I decided to write a gem that would generate an improved stats output.

Introducing Rails Code Stats

For this gem I didn't just copy the code and slightly improve the output. I refactored the code to clean things up as well as making it easier to extend functionality. One of the main changes was separating the code that calculates the stats from the code that displays the stats. Currently it still just outputs to the command line, but I intend to add the ability to have it output a HTML version as well. In addition to the new test syntax introduced in rails 2.2, I also added support for shoulda's test syntax and improved the code so that it should be quite trivial to add support for another test method syntax. Installing this plugin is as easy as adding the following to your rails 3 applications Gemfile.

gem 'rails_code_stats'

I am contemplating suggesting to the rails core team as an alternative to the current rails stats task. It seems that it fits with where rails 3 is trying to go, if you can easily swap out DB persistance layers and javascript engines why not modularize the code statistics tool as well. Please comment if you have any suggestions on improving this gem.

Monday, March 15, 2010

Rails 3: Code coverage with rcov

UPDATE: Added support for ruby 1.9 and rails 3.1

I use rcov to make sure I have 100% test coverage on my rails projects. In earlier versions of rails I used the rails_rcov plugin to accomplish this task. Since rails 3 allows rake tasks to be imported by gems, I decided to write a gem with a rake task that reports on code coverage.

rails_code_qa
Currently rails_code_qa runs the unit, functional, and integration tests for a Rails 3 app. It generates 2 different coverage reports. Unit test coverage is calculated on models, helpers, and lib; functional test coverage is calculated on the controllers. Coverage is not calculated on integration tests. In my experience I have found this to be a good way to calculate coverage because it encourages you to write unit tests that cover your core code very well. 

Install
All you need to do to install rails_code_qa in your rails 3 app is add the following line to your Gemfile.
gem "rails_code_qa"
Then update the bundle for the rails app with the following command.
bundle install
Bundler automatically pulls from rubygems.org and installs any dependencies (rcov) that rails_code_qa needs.

Usage
rails_code_qa adds a few useful rake tasks to your rails app.
rake rcqa              #all tests with coverage reports  
rake rcqa:units        #unit tests with coverage
rake rcqa:functionals  #functional tests with coverage

Future
I plan to add other code quality tools to this gem in the future. Likely candidates include flog, flay, and roodi. Feel free to leave a comment if you know of another code quality tool that you think I should include.

Sunday, March 7, 2010

Rails 3: Heroku

Last week I was pleased to discover that Heroku now supports the rails 3 beta. To use it you have to switch the application stack. The main Heroku stack (Argent Aspen) runs ruby 1.8.6 which does not work with Rails 3. Heroku enabled a new stack (Badious Bamboo) which runs either Ruby Enterprise Edition 1.8.7  or MRI 1.9.1 which both support Rails 3. 


To get your Rails 3 app running on Heroku you need switch to your stack by following the Heroku stacks documentation. After you have the correct stack set up you need to make sure you have all of your gems set up in your Gemfile. I ran into issues because I use SQLite during development and I didn't have the required pg gem set up in my Gemfile.


That is all there is to getting up and running. I tried out a few of the add-ons, none of them seemed to break my app, but they did not all work quite right. The New Relic add-on reported on deployments but it still has not registered a single request. 

Update: Rails 3 beta2 breaks on heroku when the New Relic add-on is added.

Tuesday, February 16, 2010

Rails 3: Loading rake tasks from a gem

Several times in the last year I have wished that a gem could have rake tasks that would be loaded into a Rails app. I looked into it several times but kept running into dead ends with Rails 2. Since the Rails 3 beta came out I decided to try again, this time with much more success.

Basically you are trying to include a rake task in your rails app, the tricky part is you are trying to include the rake task from the gem it is inside of, which itself is being included in the rails app.

Gem file structure
my_plugin.gemspec
lib/my_plugin.rb
lib/my_plugin/railtie.rb
tasks/my_plugin.rake

lib/my_plugin.rb
module MyPlugin
  require 'my_plugin/railtie' if defined?(Rails)
end
This file is automatically included as part of including the gem using bundler. All we need to do in this file is require our railtie.

lib/my_plugin/railtie.rb
require 'my_plugin'
require 'rails'
module MyPlugin
  class Railtie < Rails::Railtie
    railtie_name :my_plugin

    rake_tasks do
      load "tasks/my_plugin.rake"
    end
  end
end
In this file we create a railtie class inside of our module, our new class has to be a descendant of the Rails::Railtie class. The 'rake_tasks' method is defined in the Rails::Railtie class, it takes a block that it then runs during the initialization of the rake environment in your app.


lib/tasks/my_plugin.rake
desc 'my plugins rake task'
task :do_something do
  puts "the rake task did something"
end
Just a really simple rake task.


Using your new rake task
After you have built your gem and included it in your rails app using bundler the task is integrated just like any of the default rails tasks.
$ rake -T
# ...
# rake doc:rerails    # Force a rebuild of the RDOC files
# rake do_something   # my plugins rake task
# ...

$ rake do_something
# the rake task did something

Resources
http://weblog.rubyonrails.org/2010/2/9/plugin-authors-toward-a-better-future
https://gist.github.com/af7e572c2dc973add221

Sunday, February 7, 2010

Rails 3 beta: First impressions on a new app

Initial application creation
Initial application creation is as easy as it has always been. The output is formatted a little better and you can create an app with a path instead of just a subdirectory name. A .gitignore file is created automatically as well as a few .gitkeep files on directories that don't start out with files in them, a really nice touch for those of us who use git.


script/* replaced by script/rails
This takes a little getting used to if you are used to the old script commands. Stuff like "script/console" doesn't work any more, it has been replaced by "rails console" or "rails c" if you prefer the shorthand version. "rails --help" will be your friend until you get used to the new commands.


Generators
The new generator system seems to be very well thought out and has all kinds of great new features if you want to modify templates or write your own generators. Even if you are just running the default generators things have improved a bit. The default templates have improved, for example div tags instead of p tags and use of a form partial instead of duplicating code in the edit and new views.

Testing
Testing out of the box does not seem to have changed much. The main thing I noticed is that most of the major testing tools are not yet working with the rails 3 beta. I tried out shoulda, rspec, and cucumber all without success.

Monday, December 28, 2009

Interesting Rails Methods: update_attribute

Method: update_attribute
Module: ActiveRecord::Base
Documentation: Updates a single attribute and saves the record without going through the normal validation procedure. This is especially useful for boolean flags on existing records.

Notes:
This method actually does far more than just update an attribute. It updates the whole object without validations. This is fine if you just retrieved the object from the database but it can be a disaster if you have modified other attributes on the object.

Given:

Article.find(1)
#<Article id:1 title:"title" body:"first article">

Safe usage of update attribute:

article = Article.find(1)
article.update_attribute(:title, "new title")
Article.find(1)
#<Article id:1 title:"new title" body:"first article">

Unsafe usage of update attribute:

article = Article.find(1)
article.body = ""
article.update_attribute(:title, "new title")
Article.find(1)
#<Article id:1 title:"new title" body:"">

Wednesday, October 7, 2009

4 Ruby gems that will improve your rails app

These four gems have helped my team improve the quality of our code. We weren't writing bad code before but these tools made it possible for us to take our code to the next level.


Flog
Flog helps you analyze the complexity of your methods. It looks at the number of assignments, branches, and calls in a method and creates a complexity score for each method. The lower the score the less complex the method is, generally making it easier to read, test, and refactor.


Flay
Flay analyzes code for duplication. It looks for structural similarities so things like variable names and whitespace won't keep it from detecting duplication in your code.


Roodi
Roodi parses your code and gives warnings on several different code design issues. Roodi includes a few different measures of code complexity such as cyclomatic complexity, line count, and parameter count as well as some useful lint style checks.


Rcov
Rcov is a code coverage tool for ruby, it shows you what code was run during your tests. While that doesn't guarantee that you tested all of your boundary conditions it does go a long ways in giving you confidence in your test coverage. My teams threshold for code coverage is 100%, anything less is treated the same as a failing test. If you are planning on using rcov with a rails app I would suggest the rails_rcov plugin.


Bonus FREE rails plugin
When my team runs tests on our apps we use rails_quality a plugin I wrote that runs all four of these gems during the testing process. Any output that is outside of the thresholds we have set is treated the same as a failing test. I would recommend this practice to anyone. It will make your code better and it pays off way earlier in the development cycle than you expect.