Hot Potato - OSCON 2011
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Hot Potato - OSCON 2011

on

  • 6,876 views

More information can be found here: https://bit.ly/oscon-hp

More information can be found here: https://bit.ly/oscon-hp

Statistics

Views

Total Views
6,876
Views on SlideShare
6,271
Embed Views
605

Actions

Likes
3
Downloads
55
Comments
1

10 Embeds 605

http://perfeccionate.urp.edu.pe 320
http://www.darianshimy.com 248
http://lanyrd.com 13
http://www.weebly.com 10
http://www.hanrss.com 5
http://shimy.weebly.com 3
http://localhost 3
http://embed.ly 1
http://shaw-devel.weebly.com 1
http://www.juventudempresa.org 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • nice..
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Hot Potato - OSCON 2011 Presentation Transcript

  • 1. Hot PotatoA Real-Time Processing Framework Darian Shimy, OSCON 2011
  • 2. @dshimy 1998 2000 2002 2004 2006 2008 2010 Darian Shimy http://profiles.google.com/dshimyHot
Potato h(p://bit.ly/oscon‐hp
  • 3. Hot
Potato • Real-Time Processing Framework • Written in Ruby (1.9) • Designed for Processing the Twitter Firehose • MIT License • ...and it’s Web Scale!Hot
Potato h(p://bit.ly/oscon‐hp
  • 4. What
can
you
use
it
for? • Click Stream Analysis • Social Media Analysis • Fraud Prevention • SPAM Detection • Synchronized CommunicationHot
Potato h(p://bit.ly/oscon‐hp
  • 5. What
do
I
use
it
for?Hot
Potato h(p://bit.ly/oscon‐hp
  • 6. Talking
Points • Statistics • Process Management • Administration Interface • Centralized Logging • Distributed Management • Generators • Library Management via Bundler • EnvironmentsHot
Potato h(p://bit.ly/oscon‐hp
  • 7. Vernacular • AppTasks • Controllers of the framework • Types: Faucets, Workers, and Sinks • Supervisor • One instance per machine • Starts AppTasksHot
Potato h(p://bit.ly/oscon‐hp
  • 8. Architecture host01 host02 db01Hot
Potato h(p://bit.ly/oscon‐hp
  • 9. Architecture host01 host02 db01 RedisHot
Potato h(p://bit.ly/oscon‐hp
  • 10. Architecture host01 host02 db01 Sample App Sample App RedisHot
Potato h(p://bit.ly/oscon‐hp
  • 11. Architecture host01 host02 db01 Sample App Sample App Redis Supervisor SupervisorHot
Potato h(p://bit.ly/oscon‐hp
  • 12. Architecture host01 host02 db01 Sample App Sample App Redis Supervisor Supervisor AppTask AppTask AppTask AppTask AppTask AppTask AppTaskHot
Potato h(p://bit.ly/oscon‐hp
  • 13. Faucets Faucet • Inject data into the system • Implement perform (with a loop) • Call send_message • Examples: • Twitter Reader • SMTP • Tail Log FileHot
Potato h(p://bit.ly/oscon‐hp
  • 14. Faucet Faucet class TwitterFaucet < HotPotato::Faucet def perform TweetStream::Client.new("user", "secret").sample do |s| message = {} message["username"] = s.user.screen_name message["text"] = s.text send_message message end end endHot
Potato h(p://bit.ly/oscon‐hp
  • 15. Workers Worker • Manipulate data • Implement perform(message) • Call send_message • Examples: • Calculate Scores • Merge Data • Filter DataHot
Potato h(p://bit.ly/oscon‐hp
  • 16. Workers Worker class Influencer < HotPotato::Worker def perform(message) message["influence"] = rand(100) send_message message end endHot
Potato h(p://bit.ly/oscon‐hp
  • 17. Sinks Sink • Send data out of the system • Implement perform(message) • Examples: • WebSocket • Database (Data Warehouse) • File WriterHot
Potato h(p://bit.ly/oscon‐hp
  • 18. Sinks Sink class LogWriter < HotPotato::Sink def perform(message) log.debug "#{message["username"]}:#{message["influence"]}" end endHot
Potato h(p://bit.ly/oscon‐hp
  • 19. Routes • Ruby DSL • Defines AppTasks (Faucets, Workers, Sinks) • Defines processing chain for AppTasks • Restrict AppTasks to a host group • Limit number of instancesHot
Potato h(p://bit.ly/oscon‐hp
  • 20. Routes HotPotato::Route.build do # config/routes.rb endHot
Potato h(p://bit.ly/oscon‐hp
  • 21. Routes HotPotato::Route.build do faucet :twitter_faucet worker :influencer, :source => :twitter_faucet sink :log_writer, :source => :influencer endHot
Potato h(p://bit.ly/oscon‐hp
  • 22. Routes Twitter Influencer LogWriter FirehoseHot
Potato h(p://bit.ly/oscon‐hp
  • 23. Routes HotPotato::Route.build do faucet :twitter_faucet faucet :beacon worker :influencer, :source => [:twitter_faucet, :beacon] sink :log_writer, :source => :influencer endHot
Potato h(p://bit.ly/oscon‐hp
  • 24. Routes Twitter Influencer LogWriter Firehose BeaconHot
Potato h(p://bit.ly/oscon‐hp
  • 25. Routes
‐
Instances HotPotato::Route.build do faucet :twitter_faucet worker :influencer, :source => :twitter_faucet, :instances => 2 sink :log_writer, :source => :influencer endHot
Potato h(p://bit.ly/oscon‐hp
  • 26. Routes
‐
Host
Groups development: redis_hostname: localhost redis_port: 6379 servers: - hostname: worker01 group: incoming max_app_tasks: 15 - hostname: worker02 group: worker max_app_tasks: 15Hot
Potato h(p://bit.ly/oscon‐hp
  • 27. Routes
‐
Host
Groups HotPotato::Route.build do faucet :twitter_faucet, :group => :incoming worker :influencer, :source => :twitter_faucet sink :log_writer, :source => :influencer endHot
Potato h(p://bit.ly/oscon‐hp
  • 28. !Simple
Example PubSub Twitter Influencer Firehose LogWriter Sentiment MongoDB Language Detector Writer WebSocket Matcher ServerHot
Potato h(p://bit.ly/oscon‐hp
  • 29. !Simple
Example HotPotato::Route.build do faucet :twitter_firehose sink :log_writer, :source => :twitter_firehose faucet :pub_sub worker :influencer, :source => [:twitter_faucet, :pub_sub] worker :sentiment, :source => :influencer worker :lid, :source => :sentiment worker :matcher, :source => :lid sink :mongo_writer, :source => :matcher sink :ws_server, :source => :matcher endHot
Potato h(p://bit.ly/oscon‐hp
  • 30. Supervisor • Process that runs on each machine that participates in the cluster • Read the routes file • Connect to the Redis server and get the appTask process ID table • Acquire the global lock • If a process is needed, fork a new process for AppTask • Release the global lock • Rinse and RepeatHot
Potato h(p://bit.ly/oscon‐hp
  • 31. Simple
Example Twitter Influencer LogWriter FirehoseHot
Potato h(p://bit.ly/oscon‐hp
  • 32. Live
ExampleHot
Potato h(p://bit.ly/oscon‐hp
  • 33. What’s
next? • Conditional Routes • WebSocket Interface • Improved Admin Interface • Filters • Plugins • AMQP SupportHot
Potato h(p://bit.ly/oscon‐hp
  • 34. If you don’t have any questions, you were not paying attention!Hot
Potato ? h(p://bit.ly/oscon‐hp