Notes from Cloud Expo 2009: Raghu Ramakrishnan’s talk on the Yahoo! cloud: “key challenges in cloud computing … and the yahoo! approach”

raghu ramakrishnan
– a triumphant preso
– “key chalengeds in cloud comoputing .. and the y! approach”

this is a watershed time.  we’ve spent lots of time building packabged software now wer’re moving to the cloud

key challenges
– elastic scaling
– availabiolity
— if the cloud goes down, everyone is hosed.  consistency or performance myst be traded for availoability.
– handliong failures
— if things go wrong, what can the developer count on when things come up?
– operational efficiency
— cloud managers are db admins for 1000s of clients
– the right abstractions

yahoo’s cloud
– the cloud is an ecosystem.  it’s bigger than a single componenet.  all the pueces must work together seamlessly.

data management in the cloud
– how to make sense of the many options
– what are you trying todo?
– oltp vs olap
– oltp
— random access to a few records
— read-heavy vs write-heavy
– olap
— scan access to a large number of records
— by rows vs columns vs unstructired
– storage
— common features
— managed service. rest apis
— replication
— global footprint
— sherpa
— mopbstor

y! storage problem
– small records, 100kb or less
– structured records, lots of fields
– extreme data scale

typical applications
– user logins and profiles
— single=-record transactions suffice
– events
— alerts, social network activity
— ad clicks
app-specific data
– postings to messsage boards
– uploaded photos and tags

vsld data serving stores
– scale based on partitioning data accross machines
– range selections
— requests span machines
– availability
– replication
– durability
— is it required?
– how is data stored on a single machine?

the cap theorem
– consistency vs availability vs partition tolerance
– consistency => serializability

approaches to cap
– use a single version of a db w/ defered reconciliation
– defer transaction commit
– eventual consistency eg dynamo
– restrict transatctions eg sharded mysql
– object timelines, eg sherpa
– ref: julianbrowne.cim/artice/viewer/brewers-cap-theorem

single slide hadoop primer
– hadoop is wrte optimized, not ideal for serving

out there in the world
– oltp
— oracle, mysql,
— write optimized: cassandra
— main-mem; memchached

ways of using hadoop
– data workloads -> olap -> pig for row ops, zebra for column ops, map reduce for others

hadoop based apps
– we own the terasort benchmark

– parallel db
– geo replication
– structured, flexible schemas
– hashed and ordered tables
– components
— req -> routers -> (record looked up, if necessary) -> lookup cached -> individual machine
– raghu is awesome (“And then!”, sprinting through dense slides)
– write-ahead
– asynch replication
— why? we’re doing geo replication due to the physics involved
— supposing an eearthquake hits and ca falls in th ocean, two users can continue to update their profiles
– consistency model
— acid requiores synch updates
— eventual consistency works
— is there any middle ground?
— sherpa follows a timeline of changes achieved through a standard per-record primary copy protocol

– cloud allows us to apperate at scale
– tablet splitting and balancing
– automatic transfer of mastership

comparing systems
– main point: all of this needs to be thought through and handled automatically

– sherpa, oracle, mysql work well for oltp

banchmark tiers
– cluster performance
– replication
– scale out
– availability
– we’d like to do this a group effort, in keeping w/ our philosophy

the integrated cloud
– big idea: declrative lang for specifying structure of service
– key insight: multi-env
– central mechanism: the integrated cloud
– surrendra will talk about htis

foundation componenets
– how to describe app
– desc for resources, entrypoijts, bindings, etc

yst hadled 16.4 million uniques for mj death news

acm socc
– acm symposium on cloud computing

quote: U.S. Government’s Cloud Computing Requirements

“… cloud service level agreements must provide for at least 99.95% availability, vendors have to take steps to secure their services, and trouble tickets and order management need to be able to be done via API. Virtual machine services must allow live migration of workloads from one VM to another, while Web hosting services require both Windows and Linux options….”

From InformationWeek article “GSA Outlines U.S. Government’s Cloud Computing Requirements”

sdforum: cloud services SIG 2 (cloud wizard)

– meta

— opensource cross-cloud scripting

— written in python

— floats over amazon, mosso, gogrid, etc.

– why python?

— java’s written by committee

— 3000+ packages already written

– design

— cloud object

— authentication

— lists services available

— config files

— service obj

— list to see the types

— device obj

— reboots and deletes

— runs file services

— shell access

– features

— storage: s3, nirvanix

— DB: SDB, Bigtable, etc

— will also run dreamhost pseudo server

– contribute

— bsd license, which won’t go away like a gpl license

– QA

— advantages of python? shell-like

— choices for management tools?

— rightscale, cohesiveft, ylastic, scalar, cavoo, coolparty, elastra

sdforum: cloud services SIG 2 (cloudfoundry)

– meta

— chris richardson

— automated, outsourced, data center management for java apps on amazon ec2

— sdforums does cloud camps?

— wrote “pojos in action”

– why java?

— 22% market

— grails?

— spring, hibernate, terracotta, scala

— java’s community is growing

– cloud

— 18 months experience

— learning curve

— ephemeral storage

— elastic ips

— cloudtools?

— provide amis preconfiged w/ java stack: apache/tomcat/mysql

— cloudfoundry

— a hosted service

— targeted at jvm community

— monitoring ans automated management

— support

— features

— you provide war files + sql scripts

— written in jvm language

— portable

— run your apps on your amazon instance

— you can use your favorite framework

— demo

— define a name

— upload a .war file

— .war file is stored in s3

— example

— grails app

— short-term

— fluctuating load: sat/sun 4 servers, mon-fri 1 server

— a perfect candidate for amazon ec2

— app deployed by the developers

— conclusion

— contribute to cloudtools

— tiers

— db

— terracotta caching

— ap server

— ?

— QA

— what if my app needs to run solo? not yet

— ejb/j2e apps?

—- you shouldn’t be using it.

—- not yet

—- ibm will build AMIs for pop software