Ruby C10K: High Performance Networkinga case study with EM-ProxyIlya Grigorik@igrigorik

postrank.com/topic/rubyTwitterMy blog

C10K EM-Proxy+ ExamplesBenchmarks + MiscEventMachine

Proxy Love

“Rails, Django, Seaside, Grails…” cant scale.Myth: Slow Frameworks

The Proxy Solution

The “More” Proxy Solution

Transparent Scalability

Load BalancerReverse ProxyApp ServerMySQL ProxyArchitecturemiddleware ftw!Shard 1Shard 2

C10K Problem + Rubywhy do we care?

Bottleneck: ~100 req / sComplexity, Time, and Moneycirca 1995-2000

ReceiveVerifyDispatchAggregateHandle errorsRenderSendApplication BottlenecksI/O + KernelBottlenecks Kernel + I/O Bottlenecks

C10K Challenge: 10,000 Concurrent Connections

No concurrencyBlockingOk resource utilizationrequire 'rubygems'require 'socket'server = TCPServer.new(80)loop do session = server.acceptsession.print"HTTP/1.1 200 OK
done"session.closeendFork!Synchronous + Blocking IO

Fork LatencyLinux 2.6: ~200 microseconds

Socket.accept_nonblockBusy-wait CPU cycles Poll for each socketselect( […], nil, nil )1024 FD limit by defaultNon linear performanceNon-Blocking IO + Pollconcurrency without threads

Busy-wait CPU cycles

Poll for each socket

1024 FD limit by default

Non linear performance

Epoll + Kqueue Benchmarks

while (1) {intnfds = epoll_wait(fd, arr, 3, timeout);if (nfds < 0) die("Error in epoll_wait!");for(inti = 0; i < nfds; i++) {intfd = events[i].data.fd;handle_io_on_socket(fd); }}and in Ruby…EPoll & KQueueconcurrency without threadsrequire 'eventmachine'EM.epollEM.run { # ...}

while (1) {intnfds = epoll_wait(fd, arr, 3, timeout);if (nfds < 0) die("Error in epoll_wait!");for(inti = 0; i < nfds; i++) {intfd = events[i].data.fd;handle_io_on_socket(fd); }}and in Ruby…EPoll & KQueueconcurrency without threadsrequire 'eventmachine'EM.epollEM.run { # ...}

EventMachine: Speed + Conveniencebuilding high performance network apps in Ruby

p "Starting"EM.run do p "Running in EM reactor"endputs "Almost done"whiletruedo timersnetwork_ioother_ioendEventMachine Reactorconcurrency without threads

p "Starting"EM.rundo p "Running in EM reactor"endputs "Almost done"whiletruedotimersnetwork_ioother_ioendEventMachine Reactorconcurrency without threads

C++ core Easy concurrency without threadingEventMachine Reactorconcurrency without threads

http = EM::HttpRequest.new('http://site.com/').gethttp.callback { p http.response } # ... do other work, until callback fires. Event = IO event + block or lambda callEventMachine Reactorconcurrency without threads

http=EM::HttpRequest.new('http://site.com/').gethttp.callback{phttp.response}# ... do other work, until callback fires.Screencast: http://bit.ly/hPr3j Event = IO event + block or lambda callEventMachine Reactorconcurrency without threads

EM.rundoEM.add_timer(1) { p "1 second later" }EM.add_periodic_timer(5) { p "every 5 seconds"}EM.defer { long_running_task() }endclass Server < EM::Connection def receive_data(data)send_data("Pong; #{data}") end def unbind p [:connection_completed] endendEM.run doEM.start_server "0.0.0.0", 3000, Serverend

EM.run doEM.add_timer(1) { p "1 second later" }EM.add_periodic_timer(5) { p "every 5 seconds"}EM.defer { long_running_task() }endclass Server < EM::Connection def receive_data(data)send_data("Pong; #{data}") end def unbind p [:connection_completed] endendEM.rundoEM.start_server"0.0.0.0", 3000, ServerendStart Reactor

EM.run doEM.add_timer(1) { p "1 second later" }EM.add_periodic_timer(5) { p "every 5 seconds"}EM.defer { long_running_task() }endclass Server < EM::Connectiondefreceive_data(data)send_data("Pong; #{data}")enddef unbind p [:connection_completed]endendEM.rundoEM.start_server"0.0.0.0", 3000, ServerendConnection HandlerStart Reactor

http://bit.ly/aiderss-eventmachineby Dan Sinclair (Twitter: @dj2sincl)

Profile of queries changes FailLoad on production changes FailParallel environment FailSlower release cycle FailProblem: Staging Environment Fail

Proxies for Monitoring, Performance and Scalewelcome tothe wonderful world of… (C10K proof)…

Duplex Ruby Proxy, FTW!Real (production) trafficBenchmarking Proxyflash of the obvious

github.com/igrigorik/em-proxyProxy DSL: EM + EPoll

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:name, :host => "127.0.0.1", :port => 81conn.on_data do |data| # ... endconn.on_response do |server, resp| # ... endconn.on_finish do # ... endendRelay ServerEM-Proxywww.github.com/igrigorik/em-proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:name, :host => "127.0.0.1", :port => 81conn.on_datado |data|# ...endconn.on_response do |server, resp| # ... endconn.on_finish do # ... endendProcess incoming dataEM-Proxywww.github.com/igrigorik/em-proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:name, :host => "127.0.0.1", :port => 81conn.on_datado |data|# ...endconn.on_responsedo |server, resp|# ...endconn.on_finish do # ... endendProcess response dataEM-Proxywww.github.com/igrigorik/em-proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:name, :host => "127.0.0.1", :port => 81conn.on_datado |data|# ...endconn.on_responsedo |server, resp|# ...endconn.on_finishdo# ...endendPost-processing stepEM-Proxywww.github.com/igrigorik/em-proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81 # modify / process request streamconn.on_data do |data| p [:on_data, data] data end # modify / process response streamconn.on_response do |server, resp| p [:on_response, server, resp]resp end endExample: Port-Forwardingtransparent proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81# modify / process request streamconn.on_datado |data| p [:on_data, data] dataend# modify / process response streamconn.on_response do |server, resp| p [:on_response, server, resp]resp end endExample: Port-Forwardingtransparent proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81# modify / process request streamconn.on_datado |data| p [:on_data, data] dataend# modify / process response streamconn.on_responsedo |server, resp| p [:on_response, server, resp]respendendNo data modificationsExample: Port-Forwardingtransparent proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81conn.on_datado |data| dataendconn.on_response do |backend, resp|resp.gsub(/hello/, 'good bye') endendExample: Port-Forwarding + Altertransparent proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81conn.on_datado |data| dataendconn.on_responsedo |backend, resp|resp.gsub(/hello/, 'good bye')endendAlter responseExample: Port-Forwarding + Altertransparent proxy

Duplicating HTTP Trafficfor benchmarking & monitoring

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn|@start = Time.now@data = Hash.new("")conn.server:prod, :host => "127.0.0.1", :port => 81conn.server:test, :host => "127.0.0.1", :port => 82 conn.on_data do |data|data.gsub(/User-Agent: .*?
/, 'User-Agent: em-proxy
') endconn.on_response do |server, resp| @data[server] += respresp if server == :prod endconn.on_finish do p [:on_finish, Time.now - @start] p @data endendProd + TestDuplex HTTP: BenchmarkingIntercepting proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn| @start = Time.now @data = Hash.new("")conn.server :prod, :host => "127.0.0.1", :port => 81conn.server :test, :host => "127.0.0.1", :port => 82 conn.on_datado |data|data.gsub(/User-Agent: .*?
/, 'User-Agent: em-proxy
')endconn.on_responsedo |server, resp|@data[server] += resprespif server == :prodendconn.on_finish do p [:on_finish, Time.now - @start] p @data endendRespond from productionDuplex HTTP: BenchmarkingIntercepting proxy

Proxy.start(:host => "0.0.0.0", :port => 80) do |conn| @start = Time.now @data = Hash.new("")conn.server :prod, :host => "127.0.0.1", :port => 81conn.server :test, :host => "127.0.0.1", :port => 82 conn.on_data do |data|data.gsub(/User-Agent: .*?
/, 'User-Agent: em-proxy
') endconn.on_response do |server, resp| @data[server] += respresp if server == :prod endconn.on_finishdo p [:on_finish, Time.now - @start] p @dataendendRun post-processingDuplex HTTP: BenchmarkingIntercepting proxy

[ilya@igvita] >ruby examples/appserver.rb 81[ilya@igvita] >ruby examples/appserver.rb 82[ilya@igvita] >ruby examples/line_interceptor.rb[ilya@igvita] >curl localhost>> [:on_finish, 1.008561]>> {:prod=>"HTTP/1.1 200 OK
Connection: close
Date: Fri, 01 May 2009 04:20:00 GMT
Content-Type: text/plain
hello world: 0", :test=>"HTTP/1.1 200 OK
Connection: close
Date: Fri, 01 May 2009 04:20:00 GMT
Content-Type: text/plain
hello world: 1"}Duplex HTTP: BenchmarkingIntercepting proxy

[ilya@igvita] >ruby examples/appserver.rb 81[ilya@igvita] >ruby examples/appserver.rb 82[ilya@igvita] >ruby examples/line_interceptor.rb[ilya@igvita] >curl localhostSTDOUT[:on_finish, 1.008561]{:prod=>"HTTP/1.1 200 OK
Connection: close
Date: Fri, 01 May 2009 04:20:00 GMT
Content-Type: text/plain
hello world: 0",:test=>"HTTP/1.1 200 OK
Connection: close
Date: Fri, 01 May 2009 04:20:00 GMT
Content-Type: text/plain
hello world: 1"}Duplex HTTP: BenchmarkingIntercepting proxy

Same response, different turnaround timeDifferent response body!

Woops!Validating Proxyeasy, real-time diagnostics

Hacking SMTP: Whitelistingfor fun and profit

Proxy.start(:host => "0.0.0.0", :port => 2524) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 2525# RCPT TO:<name@address.com>
RCPT_CMD = /RCPT TO:<(.*)?>
/conn.on_data do |data| if rcpt = data.match(RCPT_CMD) if rcpt[1] != "ilya@igvita.com"conn.send_data "550 No such user here
" data = nil end end data endconn.on_responsedo |backend, resp|respendendIntercept AddresseeDefeating SMTP WildcardsIntercepting proxy

Proxy.start(:host => "0.0.0.0", :port => 2524) do |conn|conn.server :srv, :host => "127.0.0.1", :port => 2525 # RCPT TO:<name@address.com>
RCPT_CMD = /RCPT TO:<(.*)?>
/conn.on_datado |data|if rcpt = data.match(RCPT_CMD)if rcpt[1] != "ilya@igvita.com"conn.send_data"550 No such user here
" data = nilendend dataendconn.on_response do |backend, resp|resp endendAllow: ilya@igvita.com550 Error otherwiseDefeating SMTP WildcardsIntercepting proxy

[ilya@igvita] >mailtrap run –p 2525 –f /tmp/mailtrap.log[ilya@igvita] >ruby examples/smtp_whitelist.rb> require 'net/smtp‘> smtp = Net::SMTP.start("localhost", 2524)> smtp.send_message "Hello World!", "ilya@aiderss.com", "ilya@igvita.com" => #<Net::SMTP::Response:0xb7dcff5c @status="250", @string="250 OK
">> smtp.finish => #<Net::SMTP::Response:0xb7dcc8d4 @status="221", @string="221 Seeya
">> smtp.send_message "Hello World!", "ilya@aiderss.com", “missing_user@igvita.com"=> Net::SMTPFatalError: 550 No such user hereDuplex HTTP: BenchmarkingIntercepting proxy

[ilya@igvita] >mailtrap run –p 2525 –f /tmp/mailtrap.log[ilya@igvita] >ruby examples/smtp_whitelist.rbTo: ilya@igvita.com> require 'net/smtp‘> smtp = Net::SMTP.start("localhost", 2524)> smtp.send_message"Hello World!", "ilya@aiderss.com", "ilya@igvita.com" => #<Net::SMTP::Response:0xb7dcff5c @status="250", @string="250 OK
">> smtp.finish => #<Net::SMTP::Response:0xb7dcc8d4 @status="221", @string="221 Seeya
">> smtp.send_message"Hello World!", "ilya@aiderss.com", “missing_user@igvita.com"=> Net::SMTPFatalError: 550 No such user hereDenied!Duplex HTTP: BenchmarkingIntercepting proxy

: Beanstalkd + EM-Proxy because RAM is still expensive

~ 93 Bytes of overhead per job~300 Bytes of data / job x 80,000,000 jobs in memory ~ 30 GB of RAM = 2 X-Large EC2 instancesOi, expensive!BeanstalkdMath

Observations: 1. Each job is rescheduled several times 2. > 95% are scheduled for > 3 hours into the future 3. Beanstalkd does not have overflow page-to-diskMemory is wasted…Extending BeanstalkdWe’ll add it ourselves!

1 “Medium” EC2 InstanceIntercepting Proxy@PostRank: “Chronos Scheduler”

Proxy.start(:host => "0.0.0.0", :port => 11300) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 11301 PUT_CMD = /put (d+) (d+) (d+) (d+)
/conn.on_data do |data| if put = data.match(PUT_CMD) if put[2].to_i > 600 p [:put, :archive] # INSERT INTO ....conn.send_data "INSERTED 9999
" data = nil end end data endconn.on_responsedo |backend, resp|respendendIntercept PUT command

Proxy.start(:host => "0.0.0.0", :port => 11300) do |conn|conn.server :srv, :host => "127.0.0.1", :port => 11301 PUT_CMD = /put (d+) (d+) (d+) (d+)
/conn.on_datado |data|if put = data.match(PUT_CMD)if put[2].to_i > 600 p [:put, :archive]# INSERT INTO ....conn.send_data"INSERTED 9999
" data = nilendend dataendconn.on_response do |backend, resp|resp endendIf over 10 minutes…Archive & Reply

Overload the protocol PUTput job, 900 RESERVE, PUT, …@PostRank: “Chronos Scheduler”

~79,000,000 jobs, 4GB RAM400% cheaper + extensible! PUTUpcoming jobs: ~ 1MRESERVE, PUT, …@PostRank: “Chronos Scheduler”

… x 2,5001 process / 1 core~ 5,000 open sockets~ 1200 req/sEM-ProxyBeanstalkdMySQL2x EM-Proxy (dual core)C10K Success!Performance: Beanstalk + EM-Proxyis it “C10K proof”?

C10K: http://www.kegel.com/c10k.html Code: http://github.com/igrigorik/em-proxyTwitter: @igrigorikThanks. Questions?TwitterMy blog