Building TweetReach with Sinatra, Tokyo Cabinet and Grackle: Austin on Rails 2009-03-24

Off the Reservation with TweetReach Hayes Davis Co-Founder, Appozite [email_address] @hayesdavis

So what's this all about?
TweetReach is a simple Twitter app that helps you see how many people have seen a particular “message” you've sent
A few requirements
Show people useful stats
Be simple(ish)
Be reasonably fast for a resonable number of users
Fit within Twitter API limits

What to use?
Web app framework: Sinatra
Persistence layer: Tokyo Cabinet + Tokyo Tyrant + Memcache-client
Twitter API: Grackle

What is Sinatra?
Framework for simple web applications
Handles just the view and controller part
Controller is a simple file with a DSL for defining routes and the blocks that handle them
Views can use ERB (and others)
Can use ActiveRecord or another model and/or persistence mechanism (as you'll see)

Sinatra Example set :port, 3000 get '/' do erb :index end get '/reach' do @query = params[:q] tr = TweetReach.new(username,pass) @results = tr.measure_reach(@query) erb :reach_results end

What is Tokyo Cabinet?
Persistent (and fast) key-value store from Mikio Hirabayashi at mixi (large Japanese social network)
Stats: 2.5M inserts/second, 3M queries/second, Store 8 exabytes
Has a server called Tokyo Tyrant

More Tokyo Cabinet
Offers multiple database engines:
Hash: simple key-value store
B-tree: functionally the same as the hash DB but with ordered keys based on a user-defined function
Fixed-length: basically a giant array which you index into by offset keys
Table: similar to a relational DB except no predefined schema (ala CouchDB). Can index columns and query them

Tokyo Tyrant
Server for Tokyo Cabinet databases
Provides replication, failover, etc
Speaks memcached protocol, so it looks just like memcached with a couple exceptions
Data is persistent
Does not provide expiration

Why Tokyo Cabinet/Tyrant?
Relevant Twitter information is easily indexed by either a Twitter screen name or id
Reach calculations require O(n) looping and retrievals – much faster with a dedicated key-value store vs. a relational DB
Need a read-through cache so I don't have to hit Twitter so often
Can store stuff for relatively long periods that I may not want to keep solely in memory

What is Grackle?
Grackle is a simple Twitter REST and Search API library designed not to break when the Twitter API changes (or breaks)
I wrote it based on my experience building CheapTweet.com
Dynamically makes requests to the APIs via a generic syntax that maps to Twitter URIs
Builds OpenStructs of returned JSON or XML dynamically

Grackle Example client = Grackle::Client( :username=>'some_user', :password=>'secret' ) #GET http://twitter.com/users/show.json?id=hayesdavis client.users.show? :id=>'hayesdavis' #POST http://twitter.com/statuses/update.json client.statuses.update! :status=>'howdy world' #GET http://search.twitter.com/search.json?q=AoR client[:search].search? :q=>'AoR'

Building TweetReach

Basic Structure TweetReach Calculator Read-Through Cache Sinatra

Using Sinatra
Install the sinatra gem
Sinatra apps are a single ruby file (I called it app.rb)
Just require rubygems and sinatra
Run “ruby app.rb”

Using Tokyo Cabinet/Tyrant
Get the source for Tokyo Cabinet and Tokyo Tyrant and build it
Run “ttserver -port <port> filename.<ext>”
The type of database comes from the filename extension. I used “tch” which is the Hash engine
Get Mike Perham's memcache-client gem

Using Grackle
Make sure you've added gems.github.com to your list of gem sources
Install the hayesdavis-grackle gem
Just require “grackle”

Lessons Learned

Sinatra Lessons
Very little built-in anything:
No page or action caching
No helpers for easy formatting (stole things from Rails)
No validation against models
Make sure you really only want something extremely simple
You'll most likely need to roll one or two of your own things

Tokyo Cabinet Lessons
Lack of auto-expiration when using as mostly a key-value cache is annoying
Would definitely use it again for this type of task

Grackle
Twitter is Twitter so make sure you've got decent error handling in place – things will go wrong or not respond, etc
Still a very new library so I'm sure there are places that need cleaning up

Handy Stuff
http://www.sinatrarb.com/
http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/
http://www.scribd.com/doc/12016121/Tokyo-Cabinet-and-Tokyo-Tyrant-Presentation
http://github.com/hayesdavis/grackle

http://hayesdavis.net	906
http://www.slideshare.net	28
http://damon.tumblr.com	5
http://www.mefeedia.com	3
http://translate.googleusercontent.com	3
http://safe.tumblr.com	1
http://courselink.desire2learn.com	1

Building TweetReach with Sinatra, Tokyo Cabinet and Grackle: Austin on Rails 2009-03-24

by hayesdavis

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 947

Statistics

Building TweetReach with Sinatra, Tokyo Cabinet and Grackle: Austin on Rails 2009-03-24 Presentation Transcript