High performance app in ruby

High performance app in ruby
It works

alexander.mishyn@railsware.com
@omishyn
Wednesday, October 26, 2011

Email Marketing


Email Template
• Custom Fields


Email is a personalized template
• Custom Fields
• Open Tracking
• Click Tracking


Schema

Message Transfer
Agent
Email
(SMTP server)

Database ActionMailer Postﬁx


Satisfaction drops down
Email Customer Satisfaction


Satisfaction drops down
Email Customer Satisfaction

80000 emails per hour - max

Business driven solution
• Home made Java tool to compose emails
• Proprietary Message Transfer Agent(MTA)
• Momentum


Changes

Message Transfer
Email Agent

Database Java tool Momentum
Message Systems


Changes
Problem

Message Transfer
Email Agent

Database Java tool Momentum
Message Systems


Road to hell
• Unable to handle peak load
• Unable to scale because it kills database
• Hard to support


It doesn’t scale


Business needs stability
Email Customer Satisfaction Desired


Requirements
• one million emails per hour
• horizontal scaling
• ability to handle peak load
• durability


Ruby
ruby 1.8.7 REE
• achieve parallelism by forks
• daemon-kit gem

ruby 1.9.2 was unreleased yet

Queues
• RabbitMQ
- clients
- Bunny
- AMQP
- uses EventMachine


Queues
Dispatcher

Tasks

Recipients
Poller daemons

Sender daemons

Sent emails statistics
• 1 email produce 2 updates
• 1_000_000 emails produce 2_000_000 updates
• Deﬁnitely kills Database


Aggregate counters

Increment key:

"#{email_campaign_id}_#{email_queue_id}"


Use aggregated counts

Extract key:

"#{email_campaign_id}_#{email_queue_id}"

Update counters for:
email_campaign_id
email_queue_id


Less updates

1_000_000 emails produce 8_000 updates

~250x less updates


Key Value store
Tokyo Tyrant
as persistent storage for precaching


Coding
2 weeks of pair programming


And the show begins...


Test experiment
Benchmark:
• Test throughput

Stress test:
•Test high load handling


Count Size(Kbytes)

Test data 151 | 15
120 | 68
109 | 30
13 | 61
8 |116

Select proper size of email 68Kb
to have representative 30Kb
results
61Kb

15Kb
116Kb

Average email length is 30Kb Email length clustering

Selected email length is 60Kb

Testing

throughput
high load handling


Issues
• Tokyo Tyrant fails under concurrent key updates
• RabbitMQ fails on 500_000 messages in queue


Solution
• Redis instead of Tokyo Tyrant
• RabbitMQ
- Queue limited to 300_000 messages
- AMQP
- Disable routing from broker


Optimize Code
String gsub => gsub!
String sub => sub!
String += => <<
Array map => map!
...

C’mon you know all this, right?

Optimize Code

extra.each do |field, value| body.gsub!(/__(.*?)__/) do |match|
body.gsub!("__#{field}__", value) extra[$1.to_sym]||''
end end
body.gsub!(/__([0-9a-z_-]+)__/, '')

100000 messages @ 11.8556471347809 100000 messages @ 4.43633253574371

2.6 times faster


You should see the light
at the end of tunnel


SMTP is slow
Synchronous execution
S: 220 smtp.example.com ESMTP Postfix
C: HELO relay.example.org
S: 250 Hello relay.example.org, I am glad to meet you

C: MAIL FROM:<bob@example.org>
S: 250 Ok
C: RCPT TO:<alice@example.com>
S: 250 Ok
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: From: "Bob Example" <bob@example.org>
C: To: "Alice Example" <alice@example.com>
C: Cc: theboss@example.com
C: Date: Tue, 15 Jan 2008 16:02:43 -0500
C: Subject: Test message
C:
C: Hello Alice.
C: This is a test message with 5 header fields and 4 lines in the message body.
C: Your friend,
C: Bob
C: .
S: 250 Ok: queued as 12345
C: QUIT
S: 221 Bye
{The server closes the connection}


ECStream
Internal protocol of our Momentum MTA
“Just send C structure to the socket”

Wrote C native extension using FFI

5-10x faster than SMTP


Launch Planning
• Delivery
• Switching
• Monitoring


Parallel systems

Email

Java tool
Message Transfer
Agent

Email
Database Momentum
Email Handling System

Production delivery steps
• Blackhole sending
• A few customers switch
• Full switch


Got issues


Got issues
We got claims about broken emails

http://eimages.ratepoint.com => http://eimagesratepoint.com

•Switch to previous system
•Figure out the issue
•The problem is at the Email Providers

back to slow SMTP
CPU is bottle neck


back to slow SMTP
CPU is bottle neck
Optimize more


back to slow SMTP
CPU is bottle neck
Optimize more
Stop


back to slow SMTP
CPU is bottle neck
Optimize more
Stop

Change point of view


back to slow SMTP
CPU is bottle neck
Optimize more
Stop

Change point of view
Change servers to have
more CPU and less RAM

Switch to new system


Monitoring
• Scout app
- process usage plugin
- redis monitoring
- server overview
more at: https://github.com/railsware/scout-app-plugins
• Home made realtime monitor


Summary


Estimation

Engineering
week1 week8

Analyzing Launching


Hosting
2 Quad-core 3 Quad-core
Xeon 2.83GHz Xeon 2.00GHz

Message Transfer
Email Agent

Database Email Handling system Momentum
Message Systems


Email Handling System
• Horizontally scalable
• 1 million emails per/hour (one server)
• RabbitMQ (Clustered)
• Redis
• 17 ruby daemons


Distribution
Sent emails

6000000

4500000

3000000

1500000

0
Sunday Monday Tuesday Wednesday Thursday Friday Saturday


So that ...


1_000_000_000
emails in 2010


Questions


High performance app in ruby

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

High performance app in ruby