Hot Potato - OSCON 2011

Hot PotatoA Real-Time Processing Framework Darian Shimy, OSCON 2011

@dshimy 1998 2000 2002 2004 2006 2008 2010 Darian Shimy http://proﬁles.google.com/dshimyHot Potato h(p://bit.ly/oscon‐hp

Hot Potato • Real-Time Processing Framework • Written in Ruby (1.9) • Designed for Processing the Twitter Firehose • MIT License • ...and it’s Web Scale!Hot Potato h(p://bit.ly/oscon‐hp

What can you use it for? • Click Stream Analysis • Social Media Analysis • Fraud Prevention • SPAM Detection • Synchronized CommunicationHot Potato h(p://bit.ly/oscon‐hp

What do I use it for?Hot Potato h(p://bit.ly/oscon‐hp

Talking Points • Statistics • Process Management • Administration Interface • Centralized Logging • Distributed Management • Generators • Library Management via Bundler • EnvironmentsHot Potato h(p://bit.ly/oscon‐hp

Vernacular • AppTasks • Controllers of the framework • Types: Faucets, Workers, and Sinks • Supervisor • One instance per machine • Starts AppTasksHot Potato h(p://bit.ly/oscon‐hp

Architecture host01 host02 db01Hot Potato h(p://bit.ly/oscon‐hp

Architecture host01 host02 db01 RedisHot Potato h(p://bit.ly/oscon‐hp

Architecture host01 host02 db01 Sample App Sample App RedisHot Potato h(p://bit.ly/oscon‐hp

Architecture host01 host02 db01 Sample App Sample App Redis Supervisor SupervisorHot Potato h(p://bit.ly/oscon‐hp

Architecture host01 host02 db01 Sample App Sample App Redis Supervisor Supervisor AppTask AppTask AppTask AppTask AppTask AppTask AppTaskHot Potato h(p://bit.ly/oscon‐hp

Faucets Faucet • Inject data into the system • Implement perform (with a loop) • Call send_message • Examples: • Twitter Reader • SMTP • Tail Log FileHot Potato h(p://bit.ly/oscon‐hp

Faucet Faucet class TwitterFaucet < HotPotato::Faucet def perform TweetStream::Client.new("user", "secret").sample do |s| message = {} message["username"] = s.user.screen_name message["text"] = s.text send_message message end end endHot Potato h(p://bit.ly/oscon‐hp

Workers Worker • Manipulate data • Implement perform(message) • Call send_message • Examples: • Calculate Scores • Merge Data • Filter DataHot Potato h(p://bit.ly/oscon‐hp

Workers Worker class Inﬂuencer < HotPotato::Worker def perform(message) message["inﬂuence"] = rand(100) send_message message end endHot Potato h(p://bit.ly/oscon‐hp

Sinks Sink • Send data out of the system • Implement perform(message) • Examples: • WebSocket • Database (Data Warehouse) • File WriterHot Potato h(p://bit.ly/oscon‐hp

Sinks Sink class LogWriter < HotPotato::Sink def perform(message) log.debug "#{message["username"]}:#{message["inﬂuence"]}" end endHot Potato h(p://bit.ly/oscon‐hp

Routes • Ruby DSL • Deﬁnes AppTasks (Faucets, Workers, Sinks) • Deﬁnes processing chain for AppTasks • Restrict AppTasks to a host group • Limit number of instancesHot Potato h(p://bit.ly/oscon‐hp

Routes HotPotato::Route.build do # conﬁg/routes.rb endHot Potato h(p://bit.ly/oscon‐hp

Routes HotPotato::Route.build do faucet :twitter_faucet worker :inﬂuencer, :source => :twitter_faucet sink :log_writer, :source => :inﬂuencer endHot Potato h(p://bit.ly/oscon‐hp

Routes Twitter Inﬂuencer LogWriter FirehoseHot Potato h(p://bit.ly/oscon‐hp

Routes HotPotato::Route.build do faucet :twitter_faucet faucet :beacon worker :inﬂuencer, :source => [:twitter_faucet, :beacon] sink :log_writer, :source => :inﬂuencer endHot Potato h(p://bit.ly/oscon‐hp

Routes Twitter Inﬂuencer LogWriter Firehose BeaconHot Potato h(p://bit.ly/oscon‐hp

Routes ‐ Instances HotPotato::Route.build do faucet :twitter_faucet worker :inﬂuencer, :source => :twitter_faucet, :instances => 2 sink :log_writer, :source => :inﬂuencer endHot Potato h(p://bit.ly/oscon‐hp

Routes ‐ Host Groups development: redis_hostname: localhost redis_port: 6379 servers: - hostname: worker01 group: incoming max_app_tasks: 15 - hostname: worker02 group: worker max_app_tasks: 15Hot Potato h(p://bit.ly/oscon‐hp

Routes ‐ Host Groups HotPotato::Route.build do faucet :twitter_faucet, :group => :incoming worker :inﬂuencer, :source => :twitter_faucet sink :log_writer, :source => :inﬂuencer endHot Potato h(p://bit.ly/oscon‐hp

!Simple Example PubSub Twitter Inﬂuencer Firehose LogWriter Sentiment MongoDB Language Detector Writer WebSocket Matcher ServerHot Potato h(p://bit.ly/oscon‐hp

!Simple Example HotPotato::Route.build do faucet :twitter_firehose sink :log_writer, :source => :twitter_firehose faucet :pub_sub worker :influencer, :source => [:twitter_faucet, :pub_sub] worker :sentiment, :source => :influencer worker :lid, :source => :sentiment worker :matcher, :source => :lid sink :mongo_writer, :source => :matcher sink :ws_server, :source => :matcher endHot Potato h(p://bit.ly/oscon‐hp

Supervisor • Process that runs on each machine that participates in the cluster • Read the routes ﬁle • Connect to the Redis server and get the appTask process ID table • Acquire the global lock • If a process is needed, fork a new process for AppTask • Release the global lock • Rinse and RepeatHot Potato h(p://bit.ly/oscon‐hp

Simple Example Twitter Inﬂuencer LogWriter FirehoseHot Potato h(p://bit.ly/oscon‐hp

Live ExampleHot Potato h(p://bit.ly/oscon‐hp

What’s next? • Conditional Routes • WebSocket Interface • Improved Admin Interface • Filters • Plugins • AMQP SupportHot Potato h(p://bit.ly/oscon‐hp

If you don’t have any questions, you were not paying attention!Hot Potato ? h(p://bit.ly/oscon‐hp

http://perfeccionate.urp.edu.pe	320
http://www.darianshimy.com	66
http://lanyrd.com	13
http://www.weebly.com	10
http://www.hanrss.com	5
http://shimy.weebly.com	3
http://localhost	3
http://embed.ly	1
http://shaw-devel.weebly.com	1
http://www.juventudempresa.org	1

Hot Potato - OSCON 2011

by Darian Shimy

Accessibility

Categories

Upload Details

Usage Rights

10 Embeds 423

Statistics

Hot Potato - OSCON 2011 Presentation Transcript