High performance app in ruby                                        It works                                    alexander....
Email MarketingWednesday, October 26, 2011
Email Template               • Custom FieldsWednesday, October 26, 2011
Email is a personalized template               • Custom Fields               • Open Tracking               • Click Trackin...
Schema                                                 Message Transfer                                                   ...
Satisfaction drops down                              Email   Customer SatisfactionWednesday, October 26, 2011
Satisfaction drops down                                 Email       Customer Satisfaction                              800...
Business driven solution                • Home made Java tool to compose emails                • Proprietary Message Trans...
Changes                                              Message Transfer                                   Email          Age...
Changes                                  Problem                                              Message Transfer            ...
Road to hell                • Unable to handle peak load                • Unable to scale because it kills database       ...
It doesn’t scaleWednesday, October 26, 2011
Business needs stability                              Email   Customer Satisfaction   DesiredWednesday, October 26, 2011
Requirements                •      one million emails per hour                •      horizontal scaling                •  ...
Ruby                      ruby 1.8.7 REE               • achieve parallelism by forks               • daemon-kit gem      ...
Queues               • RabbitMQ                 - clients                   - Bunny                   - AMQP              ...
Queues                  Dispatcher                               Tasks                                                Reci...
Sent emails statistics               • 1 email produce 2 updates               • 1_000_000 emails produce 2_000_000 update...
Aggregate counters                              Increment key:         "#{email_campaign_id}_#{email_queue_id}"Wednesday, ...
Use aggregated counts                                 Extract key:         "#{email_campaign_id}_#{email_queue_id}"       ...
Less updates                              1_000_000 emails produce 8_000 updates                                     ~250x...
Key Value store               Tokyo Tyrant               as persistent storage for precachingWednesday, October 26, 2011
Coding              2 weeks of pair programmingWednesday, October 26, 2011
And the show begins...Wednesday, October 26, 2011
Test experiment               Benchmark:                    • Test throughput              Stress test:                   ...
Count Size(Kbytes)              Test data                                                                     151 | 15    ...
Testing                              throughput                   high load handlingWednesday, October 26, 2011
Issues               • Tokyo Tyrant fails under concurrent key updates               • RabbitMQ fails on 500_000 messages ...
Solution               • Redis instead of Tokyo Tyrant               • RabbitMQ                 - Queue limited to 300_000...
Testing                              throughput                   high load handlingWednesday, October 26, 2011
Optimize Code                String        gsub   =>   gsub!                String        sub    =>   sub!                ...
Optimize Code               extra.each do |field, value|          body.gsub!(/__(.*?)__/) do |match|                 body....
Testing                              throughput                   high load handlingWednesday, October 26, 2011
You should see the light               at the end of tunnelWednesday, October 26, 2011
SMTP is slow               Synchronous execution               S: 220 smtp.example.com ESMTP Postfix               C: HELO...
ECStream                 Internal protocol of our Momentum MTA                 “Just send C structure to the socket”      ...
Testing                              throughput                   high load handlingWednesday, October 26, 2011
Launch Planning                • Delivery                • Switching                • MonitoringWednesday, October 26, 2011
Parallel systems                                       Email                                        Java tool             ...
Production delivery steps                • Blackhole sending                • A few customers switch                • Full...
Got issuesWednesday, October 26, 2011
Got issues               We got claims about broken emails              http://eimages.ratepoint.com => http://eimagesrate...
back to slow SMTP               CPU is bottle neckWednesday, October 26, 2011
back to slow SMTP               CPU is bottle neck               Optimize moreWednesday, October 26, 2011
back to slow SMTP               CPU is bottle neck               Optimize more               StopWednesday, October 26, 2011
back to slow SMTP               CPU is bottle neck               Optimize more               Stop               Change poi...
back to slow SMTP               CPU is bottle neck               Optimize more               Stop               Change poi...
back to slow SMTP               CPU is bottle neck               Optimize more               Stop               Change poi...
Testing                              throughput                   high load handlingWednesday, October 26, 2011
Switch to new systemWednesday, October 26, 2011
Monitoring               • Scout app                   - process usage plugin                   - redis monitoring        ...
SummaryWednesday, October 26, 2011
Estimation                              Engineering               week1                          week8          Analyzing ...
Hosting                                    2 Quad-core           3 Quad-core                                    Xeon 2.83G...
Email Handling System                • Horizontally scalable                  • 1 million emails per/hour (one server)    ...
Distribution                                                               Sent emails                              600000...
So that ...Wednesday, October 26, 2011
1_000_000_000                                emails in 2010Wednesday, October 26, 2011
QuestionsWednesday, October 26, 2011
Upcoming SlideShare
Loading in...5
×

High performance app in ruby

1,000

Published on

The story of how we build high performance app in ruby

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,000
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

High performance app in ruby

  1. 1. High performance app in ruby It works alexander.mishyn@railsware.com @omishynWednesday, October 26, 2011
  2. 2. Email MarketingWednesday, October 26, 2011
  3. 3. Email Template • Custom FieldsWednesday, October 26, 2011
  4. 4. Email is a personalized template • Custom Fields • Open Tracking • Click TrackingWednesday, October 26, 2011
  5. 5. Schema Message Transfer Agent Email (SMTP server) Database ActionMailer PostfixWednesday, October 26, 2011
  6. 6. Satisfaction drops down Email Customer SatisfactionWednesday, October 26, 2011
  7. 7. Satisfaction drops down Email Customer Satisfaction 80000 emails per hour - maxWednesday, October 26, 2011
  8. 8. Business driven solution • Home made Java tool to compose emails • Proprietary Message Transfer Agent(MTA) • MomentumWednesday, October 26, 2011
  9. 9. Changes Message Transfer Email Agent Database Java tool Momentum Message SystemsWednesday, October 26, 2011
  10. 10. Changes Problem Message Transfer Email Agent Database Java tool Momentum Message SystemsWednesday, October 26, 2011
  11. 11. Road to hell • Unable to handle peak load • Unable to scale because it kills database • Hard to supportWednesday, October 26, 2011
  12. 12. It doesn’t scaleWednesday, October 26, 2011
  13. 13. Business needs stability Email Customer Satisfaction DesiredWednesday, October 26, 2011
  14. 14. Requirements • one million emails per hour • horizontal scaling • ability to handle peak load • durabilityWednesday, October 26, 2011
  15. 15. Ruby ruby 1.8.7 REE • achieve parallelism by forks • daemon-kit gem ruby 1.9.2 was unreleased yetWednesday, October 26, 2011
  16. 16. Queues • RabbitMQ - clients - Bunny - AMQP - uses EventMachineWednesday, October 26, 2011
  17. 17. Queues Dispatcher Tasks Recipients Poller daemons Sender daemonsWednesday, October 26, 2011
  18. 18. Sent emails statistics • 1 email produce 2 updates • 1_000_000 emails produce 2_000_000 updates • Definitely kills DatabaseWednesday, October 26, 2011
  19. 19. Aggregate counters Increment key: "#{email_campaign_id}_#{email_queue_id}"Wednesday, October 26, 2011
  20. 20. Use aggregated counts Extract key: "#{email_campaign_id}_#{email_queue_id}" Update counters for: email_campaign_id email_queue_idWednesday, October 26, 2011
  21. 21. Less updates 1_000_000 emails produce 8_000 updates ~250x less updatesWednesday, October 26, 2011
  22. 22. Key Value store Tokyo Tyrant as persistent storage for precachingWednesday, October 26, 2011
  23. 23. Coding 2 weeks of pair programmingWednesday, October 26, 2011
  24. 24. And the show begins...Wednesday, October 26, 2011
  25. 25. Test experiment Benchmark: • Test throughput Stress test: •Test high load handlingWednesday, October 26, 2011
  26. 26. Count Size(Kbytes) Test data 151 | 15 120 | 68 109 | 30 13 | 61 8 |116 Select proper size of email 68Kb to have representative 30Kb results 61Kb 15Kb 116Kb Average email length is 30Kb Email length clustering Selected email length is 60KbWednesday, October 26, 2011
  27. 27. Testing throughput high load handlingWednesday, October 26, 2011
  28. 28. Issues • Tokyo Tyrant fails under concurrent key updates • RabbitMQ fails on 500_000 messages in queueWednesday, October 26, 2011
  29. 29. Solution • Redis instead of Tokyo Tyrant • RabbitMQ - Queue limited to 300_000 messages - AMQP - Disable routing from brokerWednesday, October 26, 2011
  30. 30. Testing throughput high load handlingWednesday, October 26, 2011
  31. 31. Optimize Code String gsub => gsub! String sub => sub! String += => << Array map => map! ... C’mon you know all this, right?Wednesday, October 26, 2011
  32. 32. Optimize Code extra.each do |field, value| body.gsub!(/__(.*?)__/) do |match| body.gsub!("__#{field}__", value) extra[$1.to_sym]|| end end body.gsub!(/__([0-9a-z_-]+)__/, ) 100000 messages @ 11.8556471347809 100000 messages @ 4.43633253574371 2.6 times fasterWednesday, October 26, 2011
  33. 33. Testing throughput high load handlingWednesday, October 26, 2011
  34. 34. You should see the light at the end of tunnelWednesday, October 26, 2011
  35. 35. SMTP is slow Synchronous execution S: 220 smtp.example.com ESMTP Postfix C: HELO relay.example.org S: 250 Hello relay.example.org, I am glad to meet you C: MAIL FROM:<bob@example.org> S: 250 Ok C: RCPT TO:<alice@example.com> S: 250 Ok C: DATA S: 354 End data with <CR><LF>.<CR><LF> C: From: "Bob Example" <bob@example.org> C: To: "Alice Example" <alice@example.com> C: Cc: theboss@example.com C: Date: Tue, 15 Jan 2008 16:02:43 -0500 C: Subject: Test message C: C: Hello Alice. C: This is a test message with 5 header fields and 4 lines in the message body. C: Your friend, C: Bob C: . S: 250 Ok: queued as 12345 C: QUIT S: 221 Bye {The server closes the connection}Wednesday, October 26, 2011
  36. 36. ECStream Internal protocol of our Momentum MTA “Just send C structure to the socket” Wrote C native extension using FFI 5-10x faster than SMTPWednesday, October 26, 2011
  37. 37. Testing throughput high load handlingWednesday, October 26, 2011
  38. 38. Launch Planning • Delivery • Switching • MonitoringWednesday, October 26, 2011
  39. 39. Parallel systems Email Java tool Message Transfer Agent Email Database Momentum Email Handling SystemWednesday, October 26, 2011
  40. 40. Production delivery steps • Blackhole sending • A few customers switch • Full switchWednesday, October 26, 2011
  41. 41. Got issuesWednesday, October 26, 2011
  42. 42. Got issues We got claims about broken emails http://eimages.ratepoint.com => http://eimagesratepoint.com •Switch to previous system •Figure out the issue •The problem is at the Email ProvidersWednesday, October 26, 2011
  43. 43. back to slow SMTP CPU is bottle neckWednesday, October 26, 2011
  44. 44. back to slow SMTP CPU is bottle neck Optimize moreWednesday, October 26, 2011
  45. 45. back to slow SMTP CPU is bottle neck Optimize more StopWednesday, October 26, 2011
  46. 46. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of viewWednesday, October 26, 2011
  47. 47. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of view Change servers to have more CPU and less RAMWednesday, October 26, 2011
  48. 48. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of view Change servers to have more CPU and less RAMWednesday, October 26, 2011
  49. 49. Testing throughput high load handlingWednesday, October 26, 2011
  50. 50. Switch to new systemWednesday, October 26, 2011
  51. 51. Monitoring • Scout app - process usage plugin - redis monitoring - server overview more at: https://github.com/railsware/scout-app-plugins • Home made realtime monitorWednesday, October 26, 2011
  52. 52. SummaryWednesday, October 26, 2011
  53. 53. Estimation Engineering week1 week8 Analyzing LaunchingWednesday, October 26, 2011
  54. 54. Hosting 2 Quad-core 3 Quad-core Xeon 2.83GHz Xeon 2.00GHz Message Transfer Email Agent Database Email Handling system Momentum Message SystemsWednesday, October 26, 2011
  55. 55. Email Handling System • Horizontally scalable • 1 million emails per/hour (one server) • RabbitMQ (Clustered) • Redis • 17 ruby daemonsWednesday, October 26, 2011
  56. 56. Distribution Sent emails 6000000 4500000 3000000 1500000 0 Sunday Monday Tuesday Wednesday Thursday Friday SaturdayWednesday, October 26, 2011
  57. 57. So that ...Wednesday, October 26, 2011
  58. 58. 1_000_000_000 emails in 2010Wednesday, October 26, 2011
  59. 59. QuestionsWednesday, October 26, 2011

×