Software engineering notes

Posts Tagged ‘yahoo

Ruby utility for YQL storage

with one comment

motivation:

  • I want to be able to interact w/ YQL storage as easily as I can w/ SQLite on my own machine.  Ideally, I’d just like to be able to say storage.use(‘table’).set(‘foo’, ‘bar’) and forget about it.

overview:

  • This class is based on the SQLite utility class, YQL utility function, and simple key/val layer for YQL storage I recently posted.  The methods available are use(), set(), and get().  The use() method accepts the select, update, and execute addresses of a YQL storage record.  Calling get() or set() fires off a request to read or write, respectively, a key using the YQL key/val table.

requirements/environment:

  • An “installation” (it’s just a couple static files on your server) of the YQL key/val table mentioned above
  • All other requirements are the same as for the SQLite class & YQL fn mentioned above

code:

class YqlStorage
  def use(settings={})
    @settings = settings
    return self
  end
  def get(key)
    
    response = yql( %{
      use 'http://{your domain}/kv.xml' as kv;
      select * from kv where key = '%s' and select = '%s'
    } % [ key, @settings[:select] ] )
    
    if response['error']
      raise 'error: %s ' % response['error']['description']
    elsif !response['query']['results']
      return nil
    end
    
    return response['query']['results']['result']
  end
  
  def set(key, val)

    response = yql( %{
      insert into kv (key, val) values ('%s', '%s')
    } % [ key, val ], { 'env' => 'http://{your domain}/kv.env' } )
    
    if response['error']
      raise 'YQL error: %s ' % response['error']['description']
    end
    
    return response
  end
end

usage:

  1. Save the code below into a file
  2. Edit the file to change all occurrences of ‘{your domain}’ to your domain
  3. Use (ha!) the YQL storage addresses defined in the key/val table setup for the use() settings
  4. here’s some example code
    store = YqlStorage.new.use( {
      # get these from YQL: http://developer.yahoo.com/yql/console/#h=desc%20yql.storage.admin
      :execute => 'store://h5Y4iRockdwzZdEHvGbBkCe',
      :select => 'store://deGTN05aNePaper04EOL30W',
      :update => 'store://qG4Scissors8917SHDjv88Wb'
    } )
    store.set( 'foo', 'bar' )
    p store.get( 'foo' )
    

Please let me know if you’ve got any suggestions/questions.  

And now, to lighten the mood, here’s a picture of a squirrel yawning:
Squirrel yawning
Photo credit: _temaki_

Written by Erik

February 21, 2010 at 9:05 pm

Posted in code

Tagged with , , ,

initial attempt at simple key/val layer on YQL storage

with one comment

motivation:

  • YQL storage is very nice.  It’s globally distributed, performant, and free(!), but if you want to say “get me value where key = ‘foo'”, you need to apply a layer to map our keys to YQL’s keys.  Here’s my first attempt at this.

overview:

  • A single YQL storage record is used to store a JavaScript hash, JSON encoded.  When a value is requested, the table extracts the JSON from storage, decodes it, and applies the key passed w/ the request (we’ll let JS manage the mapping of keys to values ;).  If the request is to modify or delete the value, the updated hash is re-encoded to JSON and saved back to storage.

requirements/environement:

usage:

  1. Save the code below to a file on your server, eg kv.xml
  2. Edit the file to change all occurrences of ‘{your domain}’ to your domain
  3. Create a file called kv.env and define your table url and YQL storage addresses into it, as described in the YQL documentation
  4. For select queries, just pass the select address in the request.  For all other queries, set env={your domain}/kv.env in the request URL

code:

<?xml version="1.0"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
    <meta>
        <author>Erik Eldridge</author>
        <documentationURL></documentationURL>
        <sampleQuery>use 'http://{your domain}/kv.xml' as kv; select * from kv where key='foo' and select='store://{select store id}'</sampleQuery>
    </meta>
    <bindings>
        <select itemPath="" produces="XML">
            <urls><url></url></urls>
            <inputs>
                <key id="key" type="xs:string" paramType="variable" required="true"/>
                <key id="select" type="xs:string" paramType="variable" required="true"/>
            </inputs>
            <execute>                
                // http://www.JSON.org/json2.js w/ alert removed
                y.include('http://{your domain}/json2.js');
                
                // supplant fn (credit: http://www.crockford.com/javascript/remedial.html)
                if(!String.prototype.supplant){String.prototype.supplant=function(o){return this.replace(/{([^{}]*)}/g,function(a,b){var r=o[b];return typeof r==='string'||typeof r==='number'?r:a;});};}
                
                // response
                response.object = function () {
                    var queries = [],
                        results = [];
                
                    queries[0] = "select * from yql.storage where name = '{select}'"
                        .supplant( { 'select' : select } );
                    results[0] = y.xmlToJson( y.query(queries[0]).results );
                    
                    if ( results[0].results.result.value ) {
                        return results[0].results.result.value[key];
                    }
                }();
            </execute>
        </select>
        <insert>
            <!-- sample query: use 'http://{your domain}/kv.xml' as kv; insert into kv (key, val) values ('foo', 'bar')
            <!-- note: use env file to define table url, select, & update -->
            <urls><url></url></urls>
            <inputs>
                <key id="select" type="xs:string" paramType="variable" required="true"/>
                <key id="update" type="xs:string" paramType="variable" required="true"/>
                <value id="key" type="xs:string" paramType="variable" required="true"/>
                <value id="val" type="xs:string" paramType="variable" required="true"/>
            </inputs>
            <execute>
                // http://www.JSON.org/json2.js w/ alert removed
                y.include('http://{your domain}/json2.js');
                
                // supplant fn (credit: http://www.crockford.com/javascript/remedial.html)
                if(!String.prototype.supplant){String.prototype.supplant=function(o){return this.replace(/{([^{}]*)}/g,function(a,b){var r=o[b];return typeof r==='string'||typeof r==='number'?r:a;});};}
                
                // response
                response.object = function () {
                    var queries = [],
                        results = [];
                    
                    queries[0] = "select * from yql.storage where name = '{select}'"
                        .supplant( { 'select' : select } );
                    results[0] = y.xmlToJson( y.query(queries[0]).results );
                    
                    results[0].results.result.value[key] = val;
                    
                    queries[1] = "update yql.storage set value = '{value}' where name = '{update}'"
                        .supplant( { 
                            'value' : JSON.stringify( results[0].results.result.value ), 
                            'update' : update
                        } );
                    results[1] = y.xmlToJson( y.query( queries[1] ).results );
                    
                    return results[1].results
                }();
            </execute>
        </insert>
        <update>
            <!-- note: use env file to define table url, select, & update -->
            <urls><url></url></urls>
            <inputs>
                <key id="select" type="xs:string" paramType="variable" required="true"/>
                <key id="update" type="xs:string" paramType="variable" required="true"/>
                <key id="key" type="xs:string" paramType="variable" required="true"/>
                <value id="val" type="xs:string" paramType="variable" required="true"/>
            </inputs>
            <execute>
                // http://www.JSON.org/json2.js w/ alert removed
                y.include('http://{your domain}/json2.js');
                
                // supplant fn (credit: http://www.crockford.com/javascript/remedial.html)
                if(!String.prototype.supplant){String.prototype.supplant=function(o){return this.replace(/{([^{}]*)}/g,function(a,b){var r=o[b];return typeof r==='string'||typeof r==='number'?r:a;});};}
                
                // response
                response.object = function () {
                    var queries = [],
                        results = [];
                    
                    queries[0] = "select * from yql.storage where name = '{select}'"
                        .supplant( { 'select' : select } );
                    results[0] = y.xmlToJson( y.query(queries[0]).results );
                    
                    if ( !results[0].results.result.value || !results[0].results.result.value[key] ) {
                        return {
                            error : 'key not found'
                        }
                    }
                    
                    results[0].results.result.value[key] = val;
                    
                    queries[1] = "update yql.storage set value='{value}' where name='{update}'"
                        .supplant( { 
                            'value' : JSON.stringify( results[0].results.result.value ), 
                            'update' : update
                        } );
                    results[1] = y.xmlToJson( y.query( queries[1] ).results );
                    
                    return results[1].results
                }();
            </execute>
        </update>
        <delete>
            <!-- sample query: use 'http://{your domain}/kv.xml' as kv; delete from kv where key='foo' -->
            <!-- note: use env file to define table url, select, & update -->
            <urls><url></url></urls>
            <inputs>
                <key id="select" type="xs:string" paramType="variable" required="true"/>
                <key id="update" type="xs:string" paramType="variable" required="true"/>
                <key id="key" type="xs:string" paramType="variable" required="true"/>
            </inputs>
            <execute>
                // http://www.JSON.org/json2.js w/ alert removed
                y.include('http://{your domain}/json2.js');
                
                // supplant fn (credit: http://www.crockford.com/javascript/remedial.html)
                if(!String.prototype.supplant){String.prototype.supplant=function(o){return this.replace(/{([^{}]*)}/g,function(a,b){var r=o[b];return typeof r==='string'||typeof r==='number'?r:a;});};}
                
                // response
                response.object = function () {
                    var queries = [],
                        results = [];
                    
                    queries[0] = "select * from yql.storage where name = '{select}'"
                        .supplant( { 'select' : select } );
                    results[0] = y.xmlToJson( y.query(queries[0]).results );
                    
                    if ( !results[0].results.result.value || !results[0].results.result.value[key] ) {
                        return {
                            error : 'key not found'
                        }
                    }
                    
                    delete results[0].results.result.value[key];
                    
                    queries[1] = "update yql.storage set value='{value}' where name='{update}'"
                        .supplant( { 
                            'value' : JSON.stringify( results[0].results.result.value ), 
                            'update' : update
                        } );
                    results[1] = y.xmlToJson( y.query( queries[1] ).results );
                    
                    return results[1].results
                }();
            </execute>
        </delete>
    </bindings>
</table>

Please post back w/ suggestions/questions.  Here’s a duck for good luck:

Flickr picture of a mallard

Photo credit: foxypar4

Written by Erik

February 21, 2010 at 6:43 pm

Posted in code

Tagged with ,

notes from Cloud Expo 2009: Surendra Reddy’s presentation on “Walking Through Cloud Serving at Yahoo!”

leave a comment »

open cloud access protocol (opencap)
– definition, deployment, and life cycle mgmt of cloiud resources
– aloc, provisioning, and metering of clourd resources
– metasdat,/registry for cloud resources
– virtual infrastructure
– why ietf?  they are brilliant folks.  we’re not attaching any vendor-specific details
– open source implentation planned
– structure
— resource model
— all infastructure
— nodes, networks, etc
— resource properties
— modeled as json objects
— standard catalog of attributes, extensible
— resource operations
— operation, control, etc.
— management ntification services

– smtp is simple.  protocols must be simple
– traffic server has bindings built in (or vice versa?)
– open cloud consortium
— a national testbed to bring clouds together

Written by Erik

November 3, 2009 at 3:11 pm

Posted in notes

Tagged with , ,

Notes from Cloud Expo 2009: Raghu Ramakrishnan’s talk on the Yahoo! cloud: “key challenges in cloud computing … and the yahoo! approach”

leave a comment »

raghu ramakrishnan
– a triumphant preso
– “key chalengeds in cloud comoputing .. and the y! approach”

this is a watershed time.  we’ve spent lots of time building packabged software now wer’re moving to the cloud

key challenges
– elastic scaling
– availabiolity
— if the cloud goes down, everyone is hosed.  consistency or performance myst be traded for availoability.
– handliong failures
— if things go wrong, what can the developer count on when things come up?
– operational efficiency
— cloud managers are db admins for 1000s of clients
– the right abstractions

yahoo’s cloud
– the cloud is an ecosystem.  it’s bigger than a single componenet.  all the pueces must work together seamlessly.

data management in the cloud
– how to make sense of the many options
– what are you trying todo?
– oltp vs olap
– oltp
— random access to a few records
— read-heavy vs write-heavy
– olap
— scan access to a large number of records
— by rows vs columns vs unstructired
– storage
— common features
— managed service. rest apis
— replication
— global footprint
— sherpa
— mopbstor

y! storage problem
– small records, 100kb or less
– structured records, lots of fields
– extreme data scale

typical applications
– user logins and profiles
— single=-record transactions suffice
– events
— alerts, social network activity
— ad clicks
app-specific data
– postings to messsage boards
– uploaded photos and tags

vsld data serving stores
– scale based on partitioning data accross machines
– range selections
— requests span machines
– availability
– replication
– durability
— is it required?
– how is data stored on a single machine?

the cap theorem
– consistency vs availability vs partition tolerance
– consistency => serializability

approaches to cap
– use a single version of a db w/ defered reconciliation
– defer transaction commit
– eventual consistency eg dynamo
– restrict transatctions eg sharded mysql
– object timelines, eg sherpa
– ref: julianbrowne.cim/artice/viewer/brewers-cap-theorem

single slide hadoop primer
– hadoop is wrte optimized, not ideal for serving

out there in the world
– oltp
— oracle, mysql,
— write optimized: cassandra
— main-mem; memchached

ways of using hadoop
– data workloads -> olap -> pig for row ops, zebra for column ops, map reduce for others

hadoop based apps
– we own the terasort benchmark

sherpa`
– parallel db
– geo replication
– structured, flexible schemas
– hashed and ordered tables
– components
— req -> routers -> (record looked up, if necessary) -> lookup cached -> individual machine
– raghu is awesome (“And then!”, sprinting through dense slides)
– write-ahead
– asynch replication
— why? we’re doing geo replication due to the physics involved
— supposing an eearthquake hits and ca falls in th ocean, two users can continue to update their profiles
– consistency model
— acid requiores synch updates
— eventual consistency works
— is there any middle ground?
— sherpa follows a timeline of changes achieved through a standard per-record primary copy protocol

operability
– cloud allows us to apperate at scale
– tablet splitting and balancing
– automatic transfer of mastership

comparing systems
– main point: all of this needs to be thought through and handled automatically

example
– sherpa, oracle, mysql work well for oltp

banchmark tiers
– cluster performance
– replication
– scale out
– availability
– we’d like to do this a group effort, in keeping w/ our philosophy

the integrated cloud
– big idea: declrative lang for specifying structure of service
– key insight: multi-env
– central mechanism: the integrated cloud
– surrendra will talk about htis

foundation componenets
– how to describe app
– desc for resources, entrypoijts, bindings, etc

yst hadled 16.4 million uniques for mj death news

acm socc
– acm symposium on cloud computing

Written by Erik

November 3, 2009 at 10:25 am

Posted in notes

Tagged with , ,

notes from YUIConf 2009: “Building YUI 3 Custom Modules”, by Caridy Patino

leave a comment »

what is a module in yui 3?
– modules are not plugins, but there is a plugin module
– module names are passed into a sandbox w/ the ‘use’ method
– prefer YUI().use instead of var Y = new YUI(); Y.use …
– you can have multiple use() calls to defer loading
– community modules vs basic yui core team modules

custom modules
– registration
— by seed YUI().use
— seed will import
— by inclusion
— manually add script include and then YUI.use
— YUI(config)
— most performant
— this takes advantage of onload handling
— reduces number of http requests req’d in ‘by inclusion’
— organization
—- use YUI_config global var to manage registration
—- you can have multiple config options

building custom modules
– YUI.add(‘foo’, fn(Y){mod code}, version, requirement list);
– naming convention: utilities are all lowercase, classes are camelcase w/ uppercase leading char
– plugins extend host modules
– stack: utilities –> classes –> plugins –> mashups

how to use and build plugins
– plugins allow us to extend an existing class at runtime
– the def of a plugin looks much like that for a module class
– instead of extending y.base, we extend y.plugin.base

mashups and legacy code
– using multiple modules, including external dependencies, enhancing dom, defining event listeners
– use case: using a pre-existing yui2-based object in yui3
– check out zakas’ talkon scalable app arch
– cool: organize app as module repo
– conclusions
— define apps at a granular level
— modular apps are easier to test
— share code thru yui3 gallery
— use yui custom modules to integrate pre-existing code

q/a
– differences btwn yui2 and yui3 lazy loading?
— yui3 will load everything as a single item, if module requirements are defined using config option
— yui3 will load items in the order they are specified
– reusing modules across multiple sandbox
— yes, if defined as such in config

github/caridy
twitter/caridy
caridy.name

Written by Erik

October 29, 2009 at 10:21 am

Posted in notes

Tagged with , , ,

notes: Chi-Hua Chien (Kleiner Perkins Caufiled & Byers) Keynote @ iphonedevcamp ’09

leave a comment »

Chi-Hua Chien (Kleiner Perkins Caufiled & Byers) Keynote @ iphonedevcamp ‘09

Chi-Hua Chien (Kleiner Perkins Caufiled & Byers) Keynote @ iphonedevcamp ‘09

iphonedevcamp 09 friday keynote intro
– BT is the music guest
— did the Monster soundtrack.
— He’s a legend in the electronic scene.
— He has an app (“sonify”) in the appstore.  It performs a stutter edit when shaken

Chi-Hua Chien (Kleiner Perkins Caufiled & Byers) Keynote
– $100M ifund
– KPCB runs it
– # users of mobile internet is 4x desktop users, so it’s a huge opportunity
– how did the iphone change things?
— high speed internest
— consumer-oriented biz behind it
— appstore
— unified platform
– how is the fund doing?
— iphone sales dwarf ipod sales
— news, games, music usage on iphone far exceeds that on desktops
— 1.5B downloads from appstore in 165 days
— 4000 proposals reviewed
— 250 companies met
— actively engaged w/ 5 companes
— $45M deployed; will probably exceed $100M cap
— ifund companies: pelago, gogii, icontrol, ngmoco, booyah, mobshop, +1 unannounced
– ideas to consider
— intro
— #1 app in appstore (100k downloads/day): “do not push the red button”.  Does nothing!
— current top apps are a bit disappointing, non-innovative
— general thoughts
— innovation is key
— small, hacky apps are not long-term
— $0.06 ecp currently for ad space on apps
— categories
— mobile commerce
—- hypothesis: will be 3-4x size of commerce on internet ($100B), ie close to $1T
—- not too long ago, we were willing to pay $1 for info on 411
—- phones are great for spontaneous purchases
—- what will be the amazon, google of the mobile world
— realtime everywhere & anywhere
—- the internet was a really good newspaper; after you walk away from the computer, the internet is over.
—- the mobile internet knows a lot more about you
—- the mobile internet is always on
— local search
—- local search is completely different from desktop search, ie social, geo are bigger factors
— healthcare
—- mobile device can transmit monitoring data from embedded device
—- m. device can also analyze data
—- us gov is going to spend a lot to digitize med. data
— augmented reality
—- mobile device overlays data on top of real world
— real-world gaming
—- online games have spent a lot to build massive virtual worlds.  what if the planet was the world you played in?  what would ou play?
— QA
— size of funding & size of teams?
—- pelago $18M, 3 ppl
—- gogii $25M(?), 3
—- ngmoco $15M, 3 ppl
—- booyah $5M, 3 ppl
—- mobshop $5M, 1 guy, worked out of KPCB
—- icontrol 25-30 ppl
— min size of team?
—- 1 person
— can u form a company w/ 1 person?
—- yes, the laywers can work that out
— how can a vc firm, fund these companies, given the “capriciousness” of the apple review process?
—- KPCB has a very good relationship w/ apple
—- don’t try to launch a mobile operating sys.
— eductaional apps?
—- mobie is a componenet of ed
—- huge market
— top 5 things that matter when selecting a company to fund?
—- market, team (do you know what you’re doing), tech (is it defensible), prod (compelling?), biz. model (how are you going to make money)
— 4k proposals, $100M avail, are we on track?
—- ballpark, this is a really fast pace
—- KP has 3 ractices: life, info, green
—- 10k biz plans fund 15 comapnies/year
— signal to noise?
—- comperable to other biz areas
—- 300 in 1st 24/hrs
—- former web host has published early proposals
— level of detail in projections?
—- google’s orig. biz plan didn’t account for $1B in first years
—- no way to get it right
—- most important is how you think about it
— do you encourage fin. projections? yes
— preferred method of receiving proposals? email
— do you need a full biz plan?
—- 35 pages used to be the norm
—- now a deck that covers the 5 main areas is sufficient
— given current apps are disapponting, how could apple improve this?
—- apple selects apps that push the platform
—- fb app store was a freeforall
—- all the great apps we could have built around fb never happened because fb had no editorial
—- we’re going thru a market education process
—- the media hasn’t given enough notice to apps that are changing the world
— are these slides avail somewhere?
—- they’ll be on the ipdc website
— how important are barriers to entry?  how many have patents?
—- very imp., and almost all have patents
—- very difficult tech. is more imp. than patents
— hardware, software, accessories?
—- hardware takes a lot of capital, so it’s noty a vc-backed
//who’s bored?  ok, i’ll take 2 more
— to what extent has the pace of dev. in the iphone space affected KP biz. practice?
—- they impl. salesforce behind the site to hadnle the scale of submissions
—- this is still a very personal, high-touch biz
— I have been awarded a project (online sales to iphone), can we get funding?
—- no, KP is not investing in services
—- KP is looking for ppl developing IP
— what’s the dominant revenue model?
—- user-purchase, either virtual goods or real
—- thru apple’s payments platform
— are there tools deleopers can use to promote beyond the appstore, ie how to promote?
—- mobclix, adworld, tapjoy, etc run ad campaigns outside appstore
—- apple has sold 45M iphones/ipod touch, vs billions of handhelds sold overall
— make a lot from a few users, or a little from a lot?
—- depends on market: medical apps cost a lot; tons of free apps that are ad supported.
— would it help or hinder to have multiple disparate projects?
—- focus is best
—- pick the best one
— any segmentation among the funded projects?
—- not at the outset, but now we can see some in the data
—- gogii is loved by teens because their parents won’t get them a texting plan
— iterative dev?
—- yes, the app store looks more like internet dev, than packaged goods
— size of exit from 5M invested?
—- material, pub. companies, are coming.  this is the goal of KP
cchien@kpcb.com

Written by Erik

July 31, 2009 at 6:31 pm