Running YETI tests automatically with Watchr

YETI (YUI Easy Testing Interface) provides an easy, automated way to run YUI 3 tests. Watchr provides an easy way to run arbitrary Ruby based on file system events. Putting the two together, we get an easy way to run YETI when a YUI 3 script is saved.


  • Mac 10.6.4.  What follows may work elsewhere, but I haven’t tried it yet

Set up:

  1. Install Watchr. Please refer to the readme file in Watchr’s github repository for installation instructions. I wrote a post the other day about getting started with Watcher on Mac 10.6.4.
  2. Install Node.js.  YETI requires Node.js.  Please refer to the Node.js documentation for downloading and building Node
  3. Install npm.  YETI can be installed easily with npm.  Please refer to the readme file in npm’s github repository for installation instructions.  The Joyent blog also has an informative post on Installing Node and npm.
  4. Install YETI: npm install yeti
  5. Create the following directories: test and lib.  These directory names are completely arbitrary, but they match the watchr script introduced below.  If you want to use different names, please update the watchr script accordingly
  6. Create a file called autotest.watchr and put the following ruby into it:
  7. Create a file called test_example.html in the test directory and put the following html in it:
  8. Create one last file called example.js in the lib directory and put the following javascript in it:

You should now have a file structure like this:

Let’s run this rig:

  1. In your terminal, launch Watchr: watchr autotest.watchr
  2. Edit /lib/example.js so Y.example is no longer set to “foo”, e.g., Y.example = “bar”;
  3. Save /lib/example.js and view your terminal.  You should see YETI’s output of the failing test results
    Screen shot of YUI test failure
  4. Edit /lib/example.js resetting Y.example to “foo”, save, and note YETI’s output of the successful test results
    Screen shot of YETI output showing YUI tests passing
  5. Kill watchr (when you’re ready): Ctrl+C

Going forward:

Using the autotest.watchr script above, any file named /test/test_{lib name}.html will be run when /lib/{lib name}.js is edited.  The test file will also be run when it is edited.  If you add a new lib, but do not define a corresponding test file, watchr will fail silently.  Likewise, if you add a test file, but don’t put YUI tests in it.  In short, add libs and YUI tests in pairs, and you’re all good.

In closing, here’s one of my favorite songs from Drive Like Jehu:

Getting started with Watchr (and trying again to install Node.js on Mac 10.6.4)

I recently started exploring testing options for Node.js. Yesterday, I wrote about my experiences with nodeunit. Today, I found Christian Johansen’s blog post Unit testing node.js apps. (Thanks for the write-up, Christian!) Although I was looking for unit testing options, what really got me excited was his mention of Watchr.

Watchr provides a way to run tests automatically in response to system events, e.g., when a file is saved, much like Autotest. I had fallen in love with Autotest’s functionality after learning about it in Micheal Hartl’s nice Ruby on Rails tutorial. According to Watchr’s docs, Autotest leaves something to be desired, but in any case I very much would like my tests to run without my having to think about it.

Git-ting (ha!) Watchr was easy enough, but to run Node tests on my Mac, which for some reason is an idea I’m hung up on, I need Node, and to date I haven’t been able to build Node on my Mac (10.6.4) successfully, so this is my challenge. After searching here and there, I found an archived thread from the Node mailing list that seemed promising. It mentions that MacPorts can break if I upgrade to Snow Leopard without upgrading MacPorts, which I had, and that this can prevent Node from compiling. After clicking through to the MacPorts migration docs, I followed the steps outlined there and I was able to build Node like this:

  1. I had tried and failed to build Node multiple times, so I blew away the build directory: rm -rf build
  2. ./configure
  3. Clean things up to be thorough: make clean
  4. make
  5. Run tests just in case: make test
  6. sudo make install

Ok, on to the testing. Here’s my folder structure:

    – autotest.watchr
    – lib/
      – example.js
    – test/
       – test_example.js

My autotest.watchr file is a blend of the one on Christian’s blog, and Watchr’s tests.watchr prepackaged script. It contains

watch( 'test/test_.*\.js' )  {|md| system("node #{md[0]}") }
watch( 'lib/(.*)\.js' )      {|md| system("node test/test_#{md[1]}.js") }

# --------------------------------------------------
# Signal Handling
# --------------------------------------------------
# Ctrl-\
Signal.trap('QUIT') do
  puts " --- Running all tests ---\n\n"

# Ctrl-C
Signal.trap('INT') { abort("\n") }

example.js contains = 'bar';

test_example.js contains

var assert = require("assert");
var example = require('../lib/example');

assert.strictEqual(, 'bar', 'var foo should be "bar"');

I fire up watchr like this: watchr autotest.watchr

Watchr then captures the terminal until I enter Ctrl+C. Saving either example.js or test_example.js causes test_example.js to run. At this point the tests are crude, so my output is nothing if the test passes, or an assertion error, e.g., “AssertionError: var foo should be “bar””, if the test fails.

I think this is a good start. Time to listen to some Bonobo and call it a day.

First experiences with Rack::Test

I love test-driven development, and I love Rack apps, so I was delighted to discover the Rack::Test toolset.  But, I wasn’t able to get it working immediately using the documentation I could find, so I’m taking notes here along my journey to discovery.

I. Install Rack::Test

The docs on the Rack::Test site didn’t do me wrong.  Rack::Test installed cleanly with: 
sudo gem install rack-test

II. Define some test code to get started

I grabbed the sample code from the Rack::Test site and saved it into a file called test.rb.

  require "rack/test"

  class HomepageTest < Test::Unit::TestCase
    include Rack::Test::Methods

    def app

    def test_redirect_logged_in_users_to_dashboard
      authorize "bryan", "secret"
      get "/"

      assert_equal "", last_request.url
      assert last_response.ok?


III. Run the code

Now, this is where I stumbled. How do we run this?

I tried rackup test.rb and ruby test.rb, but both complained of an “uninitialized constant Test”, so I guess there’s a prerequisite.

I checked out the Rack::Test Gemfile and installed rspec and upgraded rack to no avail.

I’m on Mac 10.6, btw.  I’ve got rack 1.2.1 and rack-test 0.5.6.

Aha!  As per a stackoverflow thread, I learned I need to add require “test/unit” to my code, so it looks like

require "rack/test"
require "test/unit"

Now, when I run ruby test.rb, it throws, `require’: no such file to load — rack/test, but this is easily solved by requiring rubygems:

require "rubygems"
require "rack/test"
require "test/unit"

Dah! NameError: uninitialized constant HomepageTest::MyApp

I’ll just use Sinatra, that’s my end goal anyways.

require "rubygems"
require "sinatra"
require "rack/test"
require "test/unit"

class HomepageTest < Test::Unit::TestCase
  include Rack::Test::Methods

  def app

There we go:

Loaded suite test.rb
Finished in 0.007586 seconds.

1) Error:
Rack::Test::Error: Last response was not a redirect. Cannot follow_redirect!

Now it’s time for bed.  Sweet dreams

Leaning Palm HDR, "A stretch of beach along the Blue Lagoon on the atoll of Rangiroa."
Credit: vgm8383

Helpful links

Ruby utility for YQL storage


  • I want to be able to interact w/ YQL storage as easily as I can w/ SQLite on my own machine.  Ideally, I’d just like to be able to say storage.use(‘table’).set(‘foo’, ‘bar’) and forget about it.


  • This class is based on the SQLite utility class, YQL utility function, and simple key/val layer for YQL storage I recently posted.  The methods available are use(), set(), and get().  The use() method accepts the select, update, and execute addresses of a YQL storage record.  Calling get() or set() fires off a request to read or write, respectively, a key using the YQL key/val table.


  • An “installation” (it’s just a couple static files on your server) of the YQL key/val table mentioned above
  • All other requirements are the same as for the SQLite class & YQL fn mentioned above


class YqlStorage
  def use(settings={})
    @settings = settings
    return self
  def get(key)
    response = yql( %{
      use 'http://{your domain}/kv.xml' as kv;
      select * from kv where key = '%s' and select = '%s'
    } % [ key, @settings[:select] ] )
    if response['error']
      raise 'error: %s ' % response['error']['description']
    elsif !response['query']['results']
      return nil
    return response['query']['results']['result']
  def set(key, val)

    response = yql( %{
      insert into kv (key, val) values ('%s', '%s')
    } % [ key, val ], { 'env' => 'http://{your domain}/kv.env' } )
    if response['error']
      raise 'YQL error: %s ' % response['error']['description']
    return response


  1. Save the code below into a file
  2. Edit the file to change all occurrences of ‘{your domain}’ to your domain
  3. Use (ha!) the YQL storage addresses defined in the key/val table setup for the use() settings
  4. here’s some example code
    store = {
      # get these from YQL:
      :execute => 'store://h5Y4iRockdwzZdEHvGbBkCe',
      :select => 'store://deGTN05aNePaper04EOL30W',
      :update => 'store://qG4Scissors8917SHDjv88Wb'
    } )
    store.set( 'foo', 'bar' )
    p store.get( 'foo' )

Please let me know if you’ve got any suggestions/questions.  

And now, to lighten the mood, here’s a picture of a squirrel yawning:
Squirrel yawning
Photo credit: _temaki_

A simple Ruby Rack router


  • I want to be able to handle http requests.  I like the simplicity of Rack, but I want more control over the initial mapping.  I like Sinatra, but I want to easily handle request methods other than GET, PUT, POST, DELETE, and HEAD. So, the Router class below maps regular expressions to handler functions.


  • Mac os x 10.5.8
  • Ruby 1.8.6 (2008-08-11 patchlevel 287) [universal-darwin9.0]
  • Rack gem version 1.1.0


class Router
  def initialize(routes)    
    @routes = routes
  def default
    [ 404, {'Content-Type' => 'text/plain'}, 'file not found' ]
  def call(env)
    @routes.each do |route|
      match = env['REQUEST_PATH'].match(route[:pattern])
      if match
        return route[:controller].call( env, match )


# assumes router code is in router.rb
require 'router'

use Rack::CommonLogger
use Rack::ShowExceptions
use Rack::Lint
use Rack::Static, :urls => ["/static"]

    :pattern => %r{^/page1$}, 
    :controller => lambda do |env, match|
      [ 200, {'Content-Type' => 'text/html'}, 'page 1' ]

    :pattern => %r{^/}, 
    :controller => lambda do |env, match|
      [ 200, {'Content-Type' => 'text/html'}, 'index!' ]

Run it on the command line using Rack’s native Rackup ($ rackup or via Ryan Tomayko’s shotgun, which conveniently auto-reloads.


Update Mar 5, 2011

  • because Ruby < 1.9 doesn't preserve ordering in hashes, I’ve updated the code to use an array of routes. It’s more verbose, but now the route matching correctly runs top to bottom.
  • I also added a method for overriding the default response from the router

Update Mar 9, 2011

updated Ruby utility for simple SQLite3 key/val storage


  • modify my previous sqlite utility to allow for table name definition & auto table creation


  • the requirements and environment are the same as in my previous post


require 'rubygems'
require 'sqlite3'

class Storage
  def initialize
    @db = 'sqlite' )
  def create(table)
    @db.execute( %{
      CREATE TABLE #{table}
      (key varchar(100) PRIMARY KEY,
      value varchar(1000))
    } )
    rescue SQLite3::SQLException => details
      # puts details
  def use(table)
    @table = table
    self.create( table )
    return self
  def get(key)
    results = @db.get_first_row( %{
      SELECT value FROM #{@table} WHERE key='#{key}'
    } )
    if results
      return results[0]
  def set(key, val)
    result = @db.execute( %{
      (key, value)
      VALUES ('%s', '%s')
    } % [@table, key, val ] )


require 'rubygems'
require 'sqlite3'
require 'json' 'user' ).set( 'user123', { :str => 'orale!' }.to_json )

json = 'user' ).get( 'user123' )

p JSON.parse json


  • DataMapper looks like it might provide much of this functionality. If my script’s complexity continues to increase, I’m inclined to incorporate a more sophisticated solution to run it. Thoughts?

a simple SQLite key/val store in Ruby


  • I just like having a place to store blobs of data on my local machine


  • Mac os x 10.5.8
  • ruby 1.8.6 (2008-08-11 patchlevel 287) [universal-darwin9.0]
  • SQLite3
  • sqlite3-ruby (1.2.1) gem


require 'rubygems'
require 'sqlite3'
class Storage
  def self.init
    db = 'sqlite' )
    db.execute( %{
      CREATE TABLE foo 
      (key varchar(100) PRIMARY KEY, 
      value varchar(1000), 
      modified timestamp(20))
    } )
  def self.get(key)
    db = 'sqlite' )
    db.get_first_row( %{
      SELECT * FROM foo WHERE key='#{key}'
    } )
  def self.set(key, val)
    db = 'sqlite' )
    result = db.execute( %{
      REPLACE INTO foo
      (key, value, modified) 
      VALUES ('%s', '%s', %d)
    } % [key, val,] )


require 'rubygems'
require 'sqlite3'
require 'json'
# init table once (anyone have a table detection query?)
# put data in
value = {:foo => 'bar'}
Storage.set( '123', value.to_json )
# get data out
row = Storage.get( '123' )

Ruby YQL utility function example

I can’t tell whether it’s YQL, Ruby, or my lil’ YQL utility function, but I’m having fun. Here’s an example of usage:

json = yql(%{
  use '' as github;
  select * from github where id = 'yql' and repo = 'yql-tables'

Dig the multiline string syntax (inspiration: benschwarz’s Smoke gem). YQL allows POST requests for select statements (to work around URL-length limits), so I can continue to use POST for everything 🙂

1st attempt at a Ruby YQL utility function

I use YQL a lot and I find myself writing query = “select * from …”, passing the query to the YQL webservice, and then JSON-parsing the response, repeat … I searched (briefly) for a Ruby gem for YQL, but couldn’t find one, so I made (a very basic) function to perform the actions listed above.

Drop the code below in a file and run it


require 'net/http'
require 'rubygems'
require 'json'

def yql(query)
  uri = ""

  # everything's requested via POST, which is all I needed when I wrote this
  # likewise, everything coming back is json encoded
  response = Net::HTTP.post_form( URI.parse( uri ), {
    'q' => query,
    'format' => 'json'
  } )

  json = JSON.parse( response.body )
  return json

hadoop summit 09 > applications track > lightning talks

– hadoop is for performance, not speed
– use activerecord or hibernate for rapid, iterative web dev
– few businesses write map reduce jobs –> use cascading instead
– emi is a ruby shop
– I2P
— feed + pipe script + processing node
— written in a ruby dsl
— can run on a single node or in a cluster
— all data is pushed into S3, which is great cause it’s super cheap
— stack: aws > ec2 + s3 > conductor + processing node + processing center > spring + hadoop > admin + cascading > ruby-based dsl > zookeeper > jms > rest
— deployment via chef
— simple ui (built by engineers, no designer involved)
– cascading supports dsls
– “i helpig ciomputers learn languages
– higher accuracy can be achieved using a dependency syntax tree, but this is expensive to produce
– the expectation-maximum algorithm is a cheaper alternative
– easy to parallelize, but not a natural fit for map-reduce
— map-reduce overhead can become a bottleneck
– 15x speed-up using hadoop on 50 processors
– allowing 5% of data to be dropped results in a 22x speed-up w/ no loss in accuracy
– a more complex algorithm, not more data, resulted in better accuracy
– bayesian estimation w/ bilingual pairs, a more complex algo, with 8000 only sentences results in 62% accuracy (after a week of calculation!)