Building TweetReach with Sinatra, Tokyo Cabinet and Grackle: Austin on Rails 2009-03-24
Upcoming SlideShare
Loading in...5
×
 

Building TweetReach with Sinatra, Tokyo Cabinet and Grackle: Austin on Rails 2009-03-24

on

  • 11,490 views

Slides from a presentation I gave to Austin on Rails on March 24, 2009. It describes building an application called TweetReach that uses Sinatra, Tokyo Cabinet/Tyrant and Grackle.

Slides from a presentation I gave to Austin on Rails on March 24, 2009. It describes building an application called TweetReach that uses Sinatra, Tokyo Cabinet/Tyrant and Grackle.

Statistics

Views

Total Views
11,490
Views on SlideShare
10,306
Embed Views
1,184

Actions

Likes
15
Downloads
83
Comments
2

8 Embeds 1,184

http://hayesdavis.net 1136
http://www.slideshare.net 28
http://damon.tumblr.com 7
http://translate.googleusercontent.com 7
http://www.mefeedia.com 3
http://safe.tumblr.com 1
http://courselink.desire2learn.com 1
http://webcache.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Building TweetReach with Sinatra, Tokyo Cabinet and Grackle: Austin on Rails 2009-03-24 Building TweetReach with Sinatra, Tokyo Cabinet and Grackle: Austin on Rails 2009-03-24 Presentation Transcript

  • Off the Reservation with TweetReach Hayes Davis Co-Founder, Appozite [email_address] @hayesdavis
  • So what's this all about?
    • TweetReach is a simple Twitter app that helps you see how many people have seen a particular “message” you've sent
    • A few requirements
      • Show people useful stats
      • Be simple(ish)
      • Be reasonably fast for a resonable number of users
      • Fit within Twitter API limits
  • What to use?
    • Web app framework: Sinatra
    • Persistence layer: Tokyo Cabinet + Tokyo Tyrant + Memcache-client
    • Twitter API: Grackle
  • What is Sinatra?
    • Framework for simple web applications
    • Handles just the view and controller part
      • Controller is a simple file with a DSL for defining routes and the blocks that handle them
      • Views can use ERB (and others)
    • Can use ActiveRecord or another model and/or persistence mechanism (as you'll see)
  • Sinatra Example set :port, 3000 get '/' do erb :index end get '/reach' do @query = params[:q] tr = TweetReach.new(username,pass) @results = tr.measure_reach(@query) erb :reach_results end
  • What is Tokyo Cabinet?
    • Persistent (and fast) key-value store from Mikio Hirabayashi at mixi (large Japanese social network)
    • Stats: 2.5M inserts/second, 3M queries/second, Store 8 exabytes
    • Has a server called Tokyo Tyrant
  • More Tokyo Cabinet
    • Offers multiple database engines:
      • Hash: simple key-value store
      • B-tree: functionally the same as the hash DB but with ordered keys based on a user-defined function
      • Fixed-length: basically a giant array which you index into by offset keys
      • Table: similar to a relational DB except no predefined schema (ala CouchDB). Can index columns and query them
  • Tokyo Tyrant
    • Server for Tokyo Cabinet databases
    • Provides replication, failover, etc
    • Speaks memcached protocol, so it looks just like memcached with a couple exceptions
      • Data is persistent
      • Does not provide expiration
  • Why Tokyo Cabinet/Tyrant?
    • Relevant Twitter information is easily indexed by either a Twitter screen name or id
    • Reach calculations require O(n) looping and retrievals – much faster with a dedicated key-value store vs. a relational DB
    • Need a read-through cache so I don't have to hit Twitter so often
    • Can store stuff for relatively long periods that I may not want to keep solely in memory
  • What is Grackle?
    • Grackle is a simple Twitter REST and Search API library designed not to break when the Twitter API changes (or breaks)
    • I wrote it based on my experience building CheapTweet.com
    • Dynamically makes requests to the APIs via a generic syntax that maps to Twitter URIs
    • Builds OpenStructs of returned JSON or XML dynamically
  • Grackle Example client = Grackle::Client( :username=>'some_user', :password=>'secret' ) #GET http://twitter.com/users/show.json?id=hayesdavis client.users.show? :id=>'hayesdavis' #POST http://twitter.com/statuses/update.json client.statuses.update! :status=>'howdy world' #GET http://search.twitter.com/search.json?q=AoR client[:search].search? :q=>'AoR'
  • Building TweetReach
  • Basic Structure TweetReach Calculator Read-Through Cache Sinatra
  • Using Sinatra
    • Install the sinatra gem
    • Sinatra apps are a single ruby file (I called it app.rb)
    • Just require rubygems and sinatra
    • Run “ruby app.rb”
  • Using Tokyo Cabinet/Tyrant
    • Get the source for Tokyo Cabinet and Tokyo Tyrant and build it
    • Run “ttserver -port <port> filename.<ext>”
    • The type of database comes from the filename extension. I used “tch” which is the Hash engine
    • Get Mike Perham's memcache-client gem
  • Using Grackle
    • Make sure you've added gems.github.com to your list of gem sources
    • Install the hayesdavis-grackle gem
    • Just require “grackle”
  • Lessons Learned
  • Sinatra Lessons
    • Very little built-in anything:
      • No page or action caching
      • No helpers for easy formatting (stole things from Rails)
      • No validation against models
    • Make sure you really only want something extremely simple
    • You'll most likely need to roll one or two of your own things
  • Tokyo Cabinet Lessons
    • Lack of auto-expiration when using as mostly a key-value cache is annoying
    • Would definitely use it again for this type of task
  • Grackle
    • Twitter is Twitter so make sure you've got decent error handling in place – things will go wrong or not respond, etc
    • Still a very new library so I'm sure there are places that need cleaning up
  • Handy Stuff
    • http://www.sinatrarb.com/
    • http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/
    • http://www.scribd.com/doc/12016121/Tokyo-Cabinet-and-Tokyo-Tyrant-Presentation
    • http://github.com/hayesdavis/grackle