Hot Potato - OSCON 2011
Upcoming SlideShare
Loading in...5
×
 

Hot Potato - OSCON 2011

on

  • 6,784 views

More information can be found here: https://bit.ly/oscon-hp

More information can be found here: https://bit.ly/oscon-hp

Statistics

Views

Total Views
6,784
Views on SlideShare
6,185
Embed Views
599

Actions

Likes
3
Downloads
55
Comments
1

10 Embeds 599

http://perfeccionate.urp.edu.pe 320
http://www.darianshimy.com 242
http://lanyrd.com 13
http://www.weebly.com 10
http://www.hanrss.com 5
http://shimy.weebly.com 3
http://localhost 3
http://embed.ly 1
http://shaw-devel.weebly.com 1
http://www.juventudempresa.org 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • nice..
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Hot Potato - OSCON 2011 Hot Potato - OSCON 2011 Presentation Transcript

  • Hot PotatoA Real-Time Processing Framework Darian Shimy, OSCON 2011
  • @dshimy 1998 2000 2002 2004 2006 2008 2010 Darian Shimy http://profiles.google.com/dshimyHot
Potato h(p://bit.ly/oscon‐hp
  • Hot
Potato • Real-Time Processing Framework • Written in Ruby (1.9) • Designed for Processing the Twitter Firehose • MIT License • ...and it’s Web Scale!Hot
Potato h(p://bit.ly/oscon‐hp
  • What
can
you
use
it
for? • Click Stream Analysis • Social Media Analysis • Fraud Prevention • SPAM Detection • Synchronized CommunicationHot
Potato h(p://bit.ly/oscon‐hp
  • What
do
I
use
it
for?Hot
Potato h(p://bit.ly/oscon‐hp
  • Talking
Points • Statistics • Process Management • Administration Interface • Centralized Logging • Distributed Management • Generators • Library Management via Bundler • EnvironmentsHot
Potato h(p://bit.ly/oscon‐hp
  • Vernacular • AppTasks • Controllers of the framework • Types: Faucets, Workers, and Sinks • Supervisor • One instance per machine • Starts AppTasksHot
Potato h(p://bit.ly/oscon‐hp
  • Architecture host01 host02 db01Hot
Potato h(p://bit.ly/oscon‐hp
  • Architecture host01 host02 db01 RedisHot
Potato h(p://bit.ly/oscon‐hp
  • Architecture host01 host02 db01 Sample App Sample App RedisHot
Potato h(p://bit.ly/oscon‐hp
  • Architecture host01 host02 db01 Sample App Sample App Redis Supervisor SupervisorHot
Potato h(p://bit.ly/oscon‐hp
  • Architecture host01 host02 db01 Sample App Sample App Redis Supervisor Supervisor AppTask AppTask AppTask AppTask AppTask AppTask AppTaskHot
Potato h(p://bit.ly/oscon‐hp
  • Faucets Faucet • Inject data into the system • Implement perform (with a loop) • Call send_message • Examples: • Twitter Reader • SMTP • Tail Log FileHot
Potato h(p://bit.ly/oscon‐hp
  • Faucet Faucet class TwitterFaucet < HotPotato::Faucet def perform TweetStream::Client.new("user", "secret").sample do |s| message = {} message["username"] = s.user.screen_name message["text"] = s.text send_message message end end endHot
Potato h(p://bit.ly/oscon‐hp
  • Workers Worker • Manipulate data • Implement perform(message) • Call send_message • Examples: • Calculate Scores • Merge Data • Filter DataHot
Potato h(p://bit.ly/oscon‐hp
  • Workers Worker class Influencer < HotPotato::Worker def perform(message) message["influence"] = rand(100) send_message message end endHot
Potato h(p://bit.ly/oscon‐hp
  • Sinks Sink • Send data out of the system • Implement perform(message) • Examples: • WebSocket • Database (Data Warehouse) • File WriterHot
Potato h(p://bit.ly/oscon‐hp
  • Sinks Sink class LogWriter < HotPotato::Sink def perform(message) log.debug "#{message["username"]}:#{message["influence"]}" end endHot
Potato h(p://bit.ly/oscon‐hp
  • Routes • Ruby DSL • Defines AppTasks (Faucets, Workers, Sinks) • Defines processing chain for AppTasks • Restrict AppTasks to a host group • Limit number of instancesHot
Potato h(p://bit.ly/oscon‐hp
  • Routes HotPotato::Route.build do # config/routes.rb endHot
Potato h(p://bit.ly/oscon‐hp
  • Routes HotPotato::Route.build do faucet :twitter_faucet worker :influencer, :source => :twitter_faucet sink :log_writer, :source => :influencer endHot
Potato h(p://bit.ly/oscon‐hp
  • Routes Twitter Influencer LogWriter FirehoseHot
Potato h(p://bit.ly/oscon‐hp
  • Routes HotPotato::Route.build do faucet :twitter_faucet faucet :beacon worker :influencer, :source => [:twitter_faucet, :beacon] sink :log_writer, :source => :influencer endHot
Potato h(p://bit.ly/oscon‐hp
  • Routes Twitter Influencer LogWriter Firehose BeaconHot
Potato h(p://bit.ly/oscon‐hp
  • Routes
‐
Instances HotPotato::Route.build do faucet :twitter_faucet worker :influencer, :source => :twitter_faucet, :instances => 2 sink :log_writer, :source => :influencer endHot
Potato h(p://bit.ly/oscon‐hp
  • Routes
‐
Host
Groups development: redis_hostname: localhost redis_port: 6379 servers: - hostname: worker01 group: incoming max_app_tasks: 15 - hostname: worker02 group: worker max_app_tasks: 15Hot
Potato h(p://bit.ly/oscon‐hp
  • Routes
‐
Host
Groups HotPotato::Route.build do faucet :twitter_faucet, :group => :incoming worker :influencer, :source => :twitter_faucet sink :log_writer, :source => :influencer endHot
Potato h(p://bit.ly/oscon‐hp
  • !Simple
Example PubSub Twitter Influencer Firehose LogWriter Sentiment MongoDB Language Detector Writer WebSocket Matcher ServerHot
Potato h(p://bit.ly/oscon‐hp
  • !Simple
Example HotPotato::Route.build do faucet :twitter_firehose sink :log_writer, :source => :twitter_firehose faucet :pub_sub worker :influencer, :source => [:twitter_faucet, :pub_sub] worker :sentiment, :source => :influencer worker :lid, :source => :sentiment worker :matcher, :source => :lid sink :mongo_writer, :source => :matcher sink :ws_server, :source => :matcher endHot
Potato h(p://bit.ly/oscon‐hp
  • Supervisor • Process that runs on each machine that participates in the cluster • Read the routes file • Connect to the Redis server and get the appTask process ID table • Acquire the global lock • If a process is needed, fork a new process for AppTask • Release the global lock • Rinse and RepeatHot
Potato h(p://bit.ly/oscon‐hp
  • Simple
Example Twitter Influencer LogWriter FirehoseHot
Potato h(p://bit.ly/oscon‐hp
  • Live
ExampleHot
Potato h(p://bit.ly/oscon‐hp
  • What’s
next? • Conditional Routes • WebSocket Interface • Improved Admin Interface • Filters • Plugins • AMQP SupportHot
Potato h(p://bit.ly/oscon‐hp
  • If you don’t have any questions, you were not paying attention!Hot
Potato ? h(p://bit.ly/oscon‐hp