Running YETI tests automatically with Watchr

YETI (YUI Easy Testing Interface) provides an easy, automated way to run YUI 3 tests. Watchr provides an easy way to run arbitrary Ruby based on file system events. Putting the two together, we get an easy way to run YETI when a YUI 3 script is saved.

Prerequisites:

  • Mac 10.6.4.  What follows may work elsewhere, but I haven’t tried it yet

Set up:

  1. Install Watchr. Please refer to the readme file in Watchr’s github repository for installation instructions. I wrote a post the other day about getting started with Watcher on Mac 10.6.4.
  2. Install Node.js.  YETI requires Node.js.  Please refer to the Node.js documentation for downloading and building Node
  3. Install npm.  YETI can be installed easily with npm.  Please refer to the readme file in npm’s github repository for installation instructions.  The Joyent blog also has an informative post on Installing Node and npm.
  4. Install YETI: npm install yeti
  5. Create the following directories: test and lib.  These directory names are completely arbitrary, but they match the watchr script introduced below.  If you want to use different names, please update the watchr script accordingly
  6. Create a file called autotest.watchr and put the following ruby into it: https://gist.github.com/738737#file_autotest.watchr
  7. Create a file called test_example.html in the test directory and put the following html in it: https://gist.github.com/738737#file_test_example.html
  8. Create one last file called example.js in the lib directory and put the following javascript in it: https://gist.github.com/738737#file_example.js

You should now have a file structure like this:
/autotest.watchr
/lib/example.js
/test/test_example.html

Let’s run this rig:

  1. In your terminal, launch Watchr: watchr autotest.watchr
  2. Edit /lib/example.js so Y.example is no longer set to “foo”, e.g., Y.example = “bar”;
  3. Save /lib/example.js and view your terminal.  You should see YETI’s output of the failing test results
    Screen shot of YUI test failure
  4. Edit /lib/example.js resetting Y.example to “foo”, save, and note YETI’s output of the successful test results
    Screen shot of YETI output showing YUI tests passing
  5. Kill watchr (when you’re ready): Ctrl+C

Going forward:

Using the autotest.watchr script above, any file named /test/test_{lib name}.html will be run when /lib/{lib name}.js is edited.  The test file will also be run when it is edited.  If you add a new lib, but do not define a corresponding test file, watchr will fail silently.  Likewise, if you add a test file, but don’t put YUI tests in it.  In short, add libs and YUI tests in pairs, and you’re all good.

In closing, here’s one of my favorite songs from Drive Like Jehu:

Getting started with Watchr (and trying again to install Node.js on Mac 10.6.4)

I recently started exploring testing options for Node.js. Yesterday, I wrote about my experiences with nodeunit. Today, I found Christian Johansen’s blog post Unit testing node.js apps. (Thanks for the write-up, Christian!) Although I was looking for unit testing options, what really got me excited was his mention of Watchr.

Watchr provides a way to run tests automatically in response to system events, e.g., when a file is saved, much like Autotest. I had fallen in love with Autotest’s functionality after learning about it in Micheal Hartl’s nice Ruby on Rails tutorial. According to Watchr’s docs, Autotest leaves something to be desired, but in any case I very much would like my tests to run without my having to think about it.

Git-ting (ha!) Watchr was easy enough, but to run Node tests on my Mac, which for some reason is an idea I’m hung up on, I need Node, and to date I haven’t been able to build Node on my Mac (10.6.4) successfully, so this is my challenge. After searching here and there, I found an archived thread from the Node mailing list that seemed promising. It mentions that MacPorts can break if I upgrade to Snow Leopard without upgrading MacPorts, which I had, and that this can prevent Node from compiling. After clicking through to the MacPorts migration docs, I followed the steps outlined there and I was able to build Node like this:

  1. I had tried and failed to build Node multiple times, so I blew away the build directory: rm -rf build
  2. ./configure
  3. Clean things up to be thorough: make clean
  4. make
  5. Run tests just in case: make test
  6. sudo make install

Ok, on to the testing. Here’s my folder structure:

project/
    – autotest.watchr
    – lib/
      – example.js
    – test/
       – test_example.js

My autotest.watchr file is a blend of the one on Christian’s blog, and Watchr’s tests.watchr prepackaged script. It contains

watch( 'test/test_.*\.js' )  {|md| system("node #{md[0]}") }
watch( 'lib/(.*)\.js' )      {|md| system("node test/test_#{md[1]}.js") }

# --------------------------------------------------
# Signal Handling
# --------------------------------------------------
# Ctrl-\
Signal.trap('QUIT') do
  puts " --- Running all tests ---\n\n"
  run_all_tests
end

# Ctrl-C
Signal.trap('INT') { abort("\n") }

example.js contains

exports.foo = 'bar';

test_example.js contains

var assert = require("assert");
var example = require('../lib/example');

assert.strictEqual(example.foo, 'bar', 'var foo should be "bar"');

I fire up watchr like this: watchr autotest.watchr

Watchr then captures the terminal until I enter Ctrl+C. Saving either example.js or test_example.js causes test_example.js to run. At this point the tests are crude, so my output is nothing if the test passes, or an assertion error, e.g., “AssertionError: var foo should be “bar””, if the test fails.

I think this is a good start. Time to listen to some Bonobo and call it a day.

Ruby utility for YQL storage

motivation:

  • I want to be able to interact w/ YQL storage as easily as I can w/ SQLite on my own machine.  Ideally, I’d just like to be able to say storage.use(‘table’).set(‘foo’, ‘bar’) and forget about it.

overview:

  • This class is based on the SQLite utility class, YQL utility function, and simple key/val layer for YQL storage I recently posted.  The methods available are use(), set(), and get().  The use() method accepts the select, update, and execute addresses of a YQL storage record.  Calling get() or set() fires off a request to read or write, respectively, a key using the YQL key/val table.

requirements/environment:

  • An “installation” (it’s just a couple static files on your server) of the YQL key/val table mentioned above
  • All other requirements are the same as for the SQLite class & YQL fn mentioned above

code:

class YqlStorage
  def use(settings={})
    @settings = settings
    return self
  end
  def get(key)
    
    response = yql( %{
      use 'http://{your domain}/kv.xml' as kv;
      select * from kv where key = '%s' and select = '%s'
    } % [ key, @settings[:select] ] )
    
    if response['error']
      raise 'error: %s ' % response['error']['description']
    elsif !response['query']['results']
      return nil
    end
    
    return response['query']['results']['result']
  end
  
  def set(key, val)

    response = yql( %{
      insert into kv (key, val) values ('%s', '%s')
    } % [ key, val ], { 'env' => 'http://{your domain}/kv.env' } )
    
    if response['error']
      raise 'YQL error: %s ' % response['error']['description']
    end
    
    return response
  end
end

usage:

  1. Save the code below into a file
  2. Edit the file to change all occurrences of ‘{your domain}’ to your domain
  3. Use (ha!) the YQL storage addresses defined in the key/val table setup for the use() settings
  4. here’s some example code
    store = YqlStorage.new.use( {
      # get these from YQL: http://developer.yahoo.com/yql/console/#h=desc%20yql.storage.admin
      :execute => 'store://h5Y4iRockdwzZdEHvGbBkCe',
      :select => 'store://deGTN05aNePaper04EOL30W',
      :update => 'store://qG4Scissors8917SHDjv88Wb'
    } )
    store.set( 'foo', 'bar' )
    p store.get( 'foo' )
    

Please let me know if you’ve got any suggestions/questions.  

And now, to lighten the mood, here’s a picture of a squirrel yawning:
Squirrel yawning
Photo credit: _temaki_

A simple Ruby Rack router

Motivation

  • I want to be able to handle http requests.  I like the simplicity of Rack, but I want more control over the initial mapping.  I like Sinatra, but I want to easily handle request methods other than GET, PUT, POST, DELETE, and HEAD. So, the Router class below maps regular expressions to handler functions.

Requirements/Environment

  • Mac os x 10.5.8
  • Ruby 1.8.6 (2008-08-11 patchlevel 287) [universal-darwin9.0]
  • Rack gem version 1.1.0

Code

class Router
  def initialize(routes)    
    @routes = routes
  end
  def default
    [ 404, {'Content-Type' => 'text/plain'}, 'file not found' ]
  end
  def call(env)
    @routes.each do |route|
      match = env['REQUEST_PATH'].match(route[:pattern])
      if match
        return route[:controller].call( env, match )
      end
    end
    default
  end
end

Usage

# assumes router code is in router.rb
require 'router'

use Rack::CommonLogger
use Rack::ShowExceptions
use Rack::Lint
use Rack::Static, :urls => ["/static"]

run Router.new([
  {
    :pattern => %r{^/page1$}, 
    :controller => lambda do |env, match|
    
      [ 200, {'Content-Type' => 'text/html'}, 'page 1' ]

    end
  },
  {
    :pattern => %r{^/}, 
    :controller => lambda do |env, match|
    
      [ 200, {'Content-Type' => 'text/html'}, 'index!' ]
  
    end
  }
  
]);

Run it on the command line using Rack’s native Rackup ($ rackup config.ru) or via Ryan Tomayko’s shotgun, which conveniently auto-reloads.

Reference

Update Mar 5, 2011

  • because Ruby < 1.9 doesn't preserve ordering in hashes, I’ve updated the code to use an array of routes. It’s more verbose, but now the route matching correctly runs top to bottom.
  • I also added a method for overriding the default response from the router

Update Mar 9, 2011

updated Ruby utility for simple SQLite3 key/val storage

motivation:

  • modify my previous sqlite utility to allow for table name definition & auto table creation

requirements/environment:

  • the requirements and environment are the same as in my previous post

code:

require 'rubygems'
require 'sqlite3'

class Storage
  def initialize
    @db = SQLite3::Database.new( 'sqlite' )
  end
  def create(table)
    begin
    @db.execute( %{
      CREATE TABLE #{table}
      (key varchar(100) PRIMARY KEY,
      value varchar(1000))
    } )
    rescue SQLite3::SQLException => details
      # puts details
    end
  end
  def use(table)
    @table = table
    self.create( table )
    return self
  end
  def get(key)
    results = @db.get_first_row( %{
      SELECT value FROM #{@table} WHERE key='#{key}'
    } )
    if results
      return results[0]
    end
  end
  def set(key, val)
    result = @db.execute( %{
      REPLACE INTO %s
      (key, value)
      VALUES ('%s', '%s')
    } % [@table, key, val ] )
  end
end

usage:

require 'rubygems'
require 'sqlite3'
require 'json'

Storage.new.use( 'user' ).set( 'user123', { :str => 'orale!' }.to_json )

json = Storage.new.use( 'user' ).get( 'user123' )

p JSON.parse json

notes:

  • DataMapper looks like it might provide much of this functionality. If my script’s complexity continues to increase, I’m inclined to incorporate a more sophisticated solution to run it. Thoughts?

a simple SQLite key/val store in Ruby

motivation:

  • I just like having a place to store blobs of data on my local machine

environment/requirements:

  • Mac os x 10.5.8
  • ruby 1.8.6 (2008-08-11 patchlevel 287) [universal-darwin9.0]
  • SQLite3
  • sqlite3-ruby (1.2.1) gem

code:

require 'rubygems'
require 'sqlite3'
class Storage
  def self.init
    db = SQLite3::Database.new( 'sqlite' )
    db.execute( %{
      CREATE TABLE foo 
      (key varchar(100) PRIMARY KEY, 
      value varchar(1000), 
      modified timestamp(20))
    } )
  end
  def self.get(key)
    db = SQLite3::Database.new( 'sqlite' )
    db.get_first_row( %{
      SELECT * FROM foo WHERE key='#{key}'
    } )
  end
  def self.set(key, val)
    db = SQLite3::Database.new( 'sqlite' )
    result = db.execute( %{
      REPLACE INTO foo
      (key, value, modified) 
      VALUES ('%s', '%s', %d)
    } % [key, val, Time.now.to_i] )
  end
end

usage:

require 'rubygems'
require 'sqlite3'
require 'json'
...
# init table once (anyone have a table detection query?)
Storage.init
...
# put data in
value = {:foo => 'bar'}
Storage.set( '123', value.to_json )
...
# get data out
row = Storage.get( '123' )
...

Ruby YQL utility function example

I can’t tell whether it’s YQL, Ruby, or my lil’ YQL utility function, but I’m having fun. Here’s an example of usage:

json = yql(%{
  use 'http://www.datatables.org/github/github.repo.xml' as github;
  select * from github where id = 'yql' and repo = 'yql-tables'
})

Dig the multiline string syntax (inspiration: benschwarz’s Smoke gem). YQL allows POST requests for select statements (to work around URL-length limits), so I can continue to use POST for everything 🙂

1st attempt at a Ruby YQL utility function

Motivation:
I use YQL a lot and I find myself writing query = “select * from …”, passing the query to the YQL webservice, and then JSON-parsing the response, repeat … I searched (briefly) for a Ruby gem for YQL, but couldn’t find one, so I made (a very basic) function to perform the actions listed above.

Usage:
Drop the code below in a file and run it

Code:

require 'net/http'
require 'rubygems'
require 'json'

def yql(query)
  uri = "http://query.yahooapis.com/v1/public/yql"

  # everything's requested via POST, which is all I needed when I wrote this
  # likewise, everything coming back is json encoded
  response = Net::HTTP.post_form( URI.parse( uri ), {
    'q' => query,
    'format' => 'json'
  } )

  json = JSON.parse( response.body )
  return json
end

hadoop summit 09 > applications track > lightning talks

emi
– hadoop is for performance, not speed
– use activerecord or hibernate for rapid, iterative web dev
– few businesses write map reduce jobs –> use cascading instead
– emi is a ruby shop
– I2P
— feed + pipe script + processing node
— written in a ruby dsl
— can run on a single node or in a cluster
— all data is pushed into S3, which is great cause it’s super cheap
— stack: aws > ec2 + s3 > conductor + processing node + processing center > spring + hadoop > admin + cascading > ruby-based dsl > zookeeper > jms > rest
— deployment via chef
— simple ui (built by engineers, no designer involved)
– cascading supports dsls
– “i helpig ciomputers learn languages
– higher accuracy can be achieved using a dependency syntax tree, but this is expensive to produce
– the expectation-maximum algorithm is a cheaper alternative
– easy to parallelize, but not a natural fit for map-reduce
— map-reduce overhead can become a bottleneck
– 15x speed-up using hadoop on 50 processors
– allowing 5% of data to be dropped results in a 22x speed-up w/ no loss in accuracy
– a more complex algorithm, not more data, resulted in better accuracy
– bayesian estimation w/ bilingual pairs, a more complex algo, with 8000 only sentences results in 62% accuracy (after a week of calculation!)