High performance app in ruby

1,198 views
1,121 views

Published on

The story of how we build high performance app in ruby

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,198
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

High performance app in ruby

  1. 1. High performance app in ruby It works alexander.mishyn@railsware.com @omishynWednesday, October 26, 2011
  2. 2. Email MarketingWednesday, October 26, 2011
  3. 3. Email Template • Custom FieldsWednesday, October 26, 2011
  4. 4. Email is a personalized template • Custom Fields • Open Tracking • Click TrackingWednesday, October 26, 2011
  5. 5. Schema Message Transfer Agent Email (SMTP server) Database ActionMailer PostfixWednesday, October 26, 2011
  6. 6. Satisfaction drops down Email Customer SatisfactionWednesday, October 26, 2011
  7. 7. Satisfaction drops down Email Customer Satisfaction 80000 emails per hour - maxWednesday, October 26, 2011
  8. 8. Business driven solution • Home made Java tool to compose emails • Proprietary Message Transfer Agent(MTA) • MomentumWednesday, October 26, 2011
  9. 9. Changes Message Transfer Email Agent Database Java tool Momentum Message SystemsWednesday, October 26, 2011
  10. 10. Changes Problem Message Transfer Email Agent Database Java tool Momentum Message SystemsWednesday, October 26, 2011
  11. 11. Road to hell • Unable to handle peak load • Unable to scale because it kills database • Hard to supportWednesday, October 26, 2011
  12. 12. It doesn’t scaleWednesday, October 26, 2011
  13. 13. Business needs stability Email Customer Satisfaction DesiredWednesday, October 26, 2011
  14. 14. Requirements • one million emails per hour • horizontal scaling • ability to handle peak load • durabilityWednesday, October 26, 2011
  15. 15. Ruby ruby 1.8.7 REE • achieve parallelism by forks • daemon-kit gem ruby 1.9.2 was unreleased yetWednesday, October 26, 2011
  16. 16. Queues • RabbitMQ - clients - Bunny - AMQP - uses EventMachineWednesday, October 26, 2011
  17. 17. Queues Dispatcher Tasks Recipients Poller daemons Sender daemonsWednesday, October 26, 2011
  18. 18. Sent emails statistics • 1 email produce 2 updates • 1_000_000 emails produce 2_000_000 updates • Definitely kills DatabaseWednesday, October 26, 2011
  19. 19. Aggregate counters Increment key: "#{email_campaign_id}_#{email_queue_id}"Wednesday, October 26, 2011
  20. 20. Use aggregated counts Extract key: "#{email_campaign_id}_#{email_queue_id}" Update counters for: email_campaign_id email_queue_idWednesday, October 26, 2011
  21. 21. Less updates 1_000_000 emails produce 8_000 updates ~250x less updatesWednesday, October 26, 2011
  22. 22. Key Value store Tokyo Tyrant as persistent storage for precachingWednesday, October 26, 2011
  23. 23. Coding 2 weeks of pair programmingWednesday, October 26, 2011
  24. 24. And the show begins...Wednesday, October 26, 2011
  25. 25. Test experiment Benchmark: • Test throughput Stress test: •Test high load handlingWednesday, October 26, 2011
  26. 26. Count Size(Kbytes) Test data 151 | 15 120 | 68 109 | 30 13 | 61 8 |116 Select proper size of email 68Kb to have representative 30Kb results 61Kb 15Kb 116Kb Average email length is 30Kb Email length clustering Selected email length is 60KbWednesday, October 26, 2011
  27. 27. Testing throughput high load handlingWednesday, October 26, 2011
  28. 28. Issues • Tokyo Tyrant fails under concurrent key updates • RabbitMQ fails on 500_000 messages in queueWednesday, October 26, 2011
  29. 29. Solution • Redis instead of Tokyo Tyrant • RabbitMQ - Queue limited to 300_000 messages - AMQP - Disable routing from brokerWednesday, October 26, 2011
  30. 30. Testing throughput high load handlingWednesday, October 26, 2011
  31. 31. Optimize Code String gsub => gsub! String sub => sub! String += => << Array map => map! ... C’mon you know all this, right?Wednesday, October 26, 2011
  32. 32. Optimize Code extra.each do |field, value| body.gsub!(/__(.*?)__/) do |match| body.gsub!("__#{field}__", value) extra[$1.to_sym]|| end end body.gsub!(/__([0-9a-z_-]+)__/, ) 100000 messages @ 11.8556471347809 100000 messages @ 4.43633253574371 2.6 times fasterWednesday, October 26, 2011
  33. 33. Testing throughput high load handlingWednesday, October 26, 2011
  34. 34. You should see the light at the end of tunnelWednesday, October 26, 2011
  35. 35. SMTP is slow Synchronous execution S: 220 smtp.example.com ESMTP Postfix C: HELO relay.example.org S: 250 Hello relay.example.org, I am glad to meet you C: MAIL FROM:<bob@example.org> S: 250 Ok C: RCPT TO:<alice@example.com> S: 250 Ok C: DATA S: 354 End data with <CR><LF>.<CR><LF> C: From: "Bob Example" <bob@example.org> C: To: "Alice Example" <alice@example.com> C: Cc: theboss@example.com C: Date: Tue, 15 Jan 2008 16:02:43 -0500 C: Subject: Test message C: C: Hello Alice. C: This is a test message with 5 header fields and 4 lines in the message body. C: Your friend, C: Bob C: . S: 250 Ok: queued as 12345 C: QUIT S: 221 Bye {The server closes the connection}Wednesday, October 26, 2011
  36. 36. ECStream Internal protocol of our Momentum MTA “Just send C structure to the socket” Wrote C native extension using FFI 5-10x faster than SMTPWednesday, October 26, 2011
  37. 37. Testing throughput high load handlingWednesday, October 26, 2011
  38. 38. Launch Planning • Delivery • Switching • MonitoringWednesday, October 26, 2011
  39. 39. Parallel systems Email Java tool Message Transfer Agent Email Database Momentum Email Handling SystemWednesday, October 26, 2011
  40. 40. Production delivery steps • Blackhole sending • A few customers switch • Full switchWednesday, October 26, 2011
  41. 41. Got issuesWednesday, October 26, 2011
  42. 42. Got issues We got claims about broken emails http://eimages.ratepoint.com => http://eimagesratepoint.com •Switch to previous system •Figure out the issue •The problem is at the Email ProvidersWednesday, October 26, 2011
  43. 43. back to slow SMTP CPU is bottle neckWednesday, October 26, 2011
  44. 44. back to slow SMTP CPU is bottle neck Optimize moreWednesday, October 26, 2011
  45. 45. back to slow SMTP CPU is bottle neck Optimize more StopWednesday, October 26, 2011
  46. 46. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of viewWednesday, October 26, 2011
  47. 47. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of view Change servers to have more CPU and less RAMWednesday, October 26, 2011
  48. 48. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of view Change servers to have more CPU and less RAMWednesday, October 26, 2011
  49. 49. Testing throughput high load handlingWednesday, October 26, 2011
  50. 50. Switch to new systemWednesday, October 26, 2011
  51. 51. Monitoring • Scout app - process usage plugin - redis monitoring - server overview more at: https://github.com/railsware/scout-app-plugins • Home made realtime monitorWednesday, October 26, 2011
  52. 52. SummaryWednesday, October 26, 2011
  53. 53. Estimation Engineering week1 week8 Analyzing LaunchingWednesday, October 26, 2011
  54. 54. Hosting 2 Quad-core 3 Quad-core Xeon 2.83GHz Xeon 2.00GHz Message Transfer Email Agent Database Email Handling system Momentum Message SystemsWednesday, October 26, 2011
  55. 55. Email Handling System • Horizontally scalable • 1 million emails per/hour (one server) • RabbitMQ (Clustered) • Redis • 17 ruby daemonsWednesday, October 26, 2011
  56. 56. Distribution Sent emails 6000000 4500000 3000000 1500000 0 Sunday Monday Tuesday Wednesday Thursday Friday SaturdayWednesday, October 26, 2011
  57. 57. So that ...Wednesday, October 26, 2011
  58. 58. 1_000_000_000 emails in 2010Wednesday, October 26, 2011
  59. 59. QuestionsWednesday, October 26, 2011

×