High performance app in ruby

  • 918 views
Uploaded on

The story of how we build high performance app in ruby

The story of how we build high performance app in ruby

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
918
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. High performance app in ruby It works alexander.mishyn@railsware.com @omishynWednesday, October 26, 2011
  • 2. Email MarketingWednesday, October 26, 2011
  • 3. Email Template • Custom FieldsWednesday, October 26, 2011
  • 4. Email is a personalized template • Custom Fields • Open Tracking • Click TrackingWednesday, October 26, 2011
  • 5. Schema Message Transfer Agent Email (SMTP server) Database ActionMailer PostfixWednesday, October 26, 2011
  • 6. Satisfaction drops down Email Customer SatisfactionWednesday, October 26, 2011
  • 7. Satisfaction drops down Email Customer Satisfaction 80000 emails per hour - maxWednesday, October 26, 2011
  • 8. Business driven solution • Home made Java tool to compose emails • Proprietary Message Transfer Agent(MTA) • MomentumWednesday, October 26, 2011
  • 9. Changes Message Transfer Email Agent Database Java tool Momentum Message SystemsWednesday, October 26, 2011
  • 10. Changes Problem Message Transfer Email Agent Database Java tool Momentum Message SystemsWednesday, October 26, 2011
  • 11. Road to hell • Unable to handle peak load • Unable to scale because it kills database • Hard to supportWednesday, October 26, 2011
  • 12. It doesn’t scaleWednesday, October 26, 2011
  • 13. Business needs stability Email Customer Satisfaction DesiredWednesday, October 26, 2011
  • 14. Requirements • one million emails per hour • horizontal scaling • ability to handle peak load • durabilityWednesday, October 26, 2011
  • 15. Ruby ruby 1.8.7 REE • achieve parallelism by forks • daemon-kit gem ruby 1.9.2 was unreleased yetWednesday, October 26, 2011
  • 16. Queues • RabbitMQ - clients - Bunny - AMQP - uses EventMachineWednesday, October 26, 2011
  • 17. Queues Dispatcher Tasks Recipients Poller daemons Sender daemonsWednesday, October 26, 2011
  • 18. Sent emails statistics • 1 email produce 2 updates • 1_000_000 emails produce 2_000_000 updates • Definitely kills DatabaseWednesday, October 26, 2011
  • 19. Aggregate counters Increment key: "#{email_campaign_id}_#{email_queue_id}"Wednesday, October 26, 2011
  • 20. Use aggregated counts Extract key: "#{email_campaign_id}_#{email_queue_id}" Update counters for: email_campaign_id email_queue_idWednesday, October 26, 2011
  • 21. Less updates 1_000_000 emails produce 8_000 updates ~250x less updatesWednesday, October 26, 2011
  • 22. Key Value store Tokyo Tyrant as persistent storage for precachingWednesday, October 26, 2011
  • 23. Coding 2 weeks of pair programmingWednesday, October 26, 2011
  • 24. And the show begins...Wednesday, October 26, 2011
  • 25. Test experiment Benchmark: • Test throughput Stress test: •Test high load handlingWednesday, October 26, 2011
  • 26. Count Size(Kbytes) Test data 151 | 15 120 | 68 109 | 30 13 | 61 8 |116 Select proper size of email 68Kb to have representative 30Kb results 61Kb 15Kb 116Kb Average email length is 30Kb Email length clustering Selected email length is 60KbWednesday, October 26, 2011
  • 27. Testing throughput high load handlingWednesday, October 26, 2011
  • 28. Issues • Tokyo Tyrant fails under concurrent key updates • RabbitMQ fails on 500_000 messages in queueWednesday, October 26, 2011
  • 29. Solution • Redis instead of Tokyo Tyrant • RabbitMQ - Queue limited to 300_000 messages - AMQP - Disable routing from brokerWednesday, October 26, 2011
  • 30. Testing throughput high load handlingWednesday, October 26, 2011
  • 31. Optimize Code String gsub => gsub! String sub => sub! String += => << Array map => map! ... C’mon you know all this, right?Wednesday, October 26, 2011
  • 32. Optimize Code extra.each do |field, value| body.gsub!(/__(.*?)__/) do |match| body.gsub!("__#{field}__", value) extra[$1.to_sym]|| end end body.gsub!(/__([0-9a-z_-]+)__/, ) 100000 messages @ 11.8556471347809 100000 messages @ 4.43633253574371 2.6 times fasterWednesday, October 26, 2011
  • 33. Testing throughput high load handlingWednesday, October 26, 2011
  • 34. You should see the light at the end of tunnelWednesday, October 26, 2011
  • 35. SMTP is slow Synchronous execution S: 220 smtp.example.com ESMTP Postfix C: HELO relay.example.org S: 250 Hello relay.example.org, I am glad to meet you C: MAIL FROM:<bob@example.org> S: 250 Ok C: RCPT TO:<alice@example.com> S: 250 Ok C: DATA S: 354 End data with <CR><LF>.<CR><LF> C: From: "Bob Example" <bob@example.org> C: To: "Alice Example" <alice@example.com> C: Cc: theboss@example.com C: Date: Tue, 15 Jan 2008 16:02:43 -0500 C: Subject: Test message C: C: Hello Alice. C: This is a test message with 5 header fields and 4 lines in the message body. C: Your friend, C: Bob C: . S: 250 Ok: queued as 12345 C: QUIT S: 221 Bye {The server closes the connection}Wednesday, October 26, 2011
  • 36. ECStream Internal protocol of our Momentum MTA “Just send C structure to the socket” Wrote C native extension using FFI 5-10x faster than SMTPWednesday, October 26, 2011
  • 37. Testing throughput high load handlingWednesday, October 26, 2011
  • 38. Launch Planning • Delivery • Switching • MonitoringWednesday, October 26, 2011
  • 39. Parallel systems Email Java tool Message Transfer Agent Email Database Momentum Email Handling SystemWednesday, October 26, 2011
  • 40. Production delivery steps • Blackhole sending • A few customers switch • Full switchWednesday, October 26, 2011
  • 41. Got issuesWednesday, October 26, 2011
  • 42. Got issues We got claims about broken emails http://eimages.ratepoint.com => http://eimagesratepoint.com •Switch to previous system •Figure out the issue •The problem is at the Email ProvidersWednesday, October 26, 2011
  • 43. back to slow SMTP CPU is bottle neckWednesday, October 26, 2011
  • 44. back to slow SMTP CPU is bottle neck Optimize moreWednesday, October 26, 2011
  • 45. back to slow SMTP CPU is bottle neck Optimize more StopWednesday, October 26, 2011
  • 46. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of viewWednesday, October 26, 2011
  • 47. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of view Change servers to have more CPU and less RAMWednesday, October 26, 2011
  • 48. back to slow SMTP CPU is bottle neck Optimize more Stop Change point of view Change servers to have more CPU and less RAMWednesday, October 26, 2011
  • 49. Testing throughput high load handlingWednesday, October 26, 2011
  • 50. Switch to new systemWednesday, October 26, 2011
  • 51. Monitoring • Scout app - process usage plugin - redis monitoring - server overview more at: https://github.com/railsware/scout-app-plugins • Home made realtime monitorWednesday, October 26, 2011
  • 52. SummaryWednesday, October 26, 2011
  • 53. Estimation Engineering week1 week8 Analyzing LaunchingWednesday, October 26, 2011
  • 54. Hosting 2 Quad-core 3 Quad-core Xeon 2.83GHz Xeon 2.00GHz Message Transfer Email Agent Database Email Handling system Momentum Message SystemsWednesday, October 26, 2011
  • 55. Email Handling System • Horizontally scalable • 1 million emails per/hour (one server) • RabbitMQ (Clustered) • Redis • 17 ruby daemonsWednesday, October 26, 2011
  • 56. Distribution Sent emails 6000000 4500000 3000000 1500000 0 Sunday Monday Tuesday Wednesday Thursday Friday SaturdayWednesday, October 26, 2011
  • 57. So that ...Wednesday, October 26, 2011
  • 58. 1_000_000_000 emails in 2010Wednesday, October 26, 2011
  • 59. QuestionsWednesday, October 26, 2011