It’s not always sunny in the clouds
Me,
teetering
Text

Mike Subelsky
@subelsky

Mike Subelsky
@subelsky

1 / 1 / 08
08:00:00 ET

“Everything needs
to be automated”

Autoscaling is the easiest
part

Think carefully about
credential management

You really could use internal
DNS

It's maybe not that cheap

Launching servers is not
that fast

You will become dependent
on “glue” services

You will become dependent
on “glue” services

You will depend on a distant
faceless provider

Use DVCS

You will spend a lot of time
on monitoring

Your logs will
runneth over

Your logs will
runneth over

Write lots of
“in-process tests”

Write lots of
“in-process tests”
m = Message.find_last_slug_message
secs = m.created_at - m.date_sent
if secs > 600
ErrorNotifier.fatal(\"Missed slug delivery\",secs,m)
end

Snapshots are slow

Snapshots are slow

Rails will be the
least of your worries

Cloud services involve
subtle-yet-massive
tradeos

SQS guarantees delivery at
least once

SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id => smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
end

SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id => smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
rows_updated = raw_update(\"UPDATE application_message_logs
SET last_attempt_at = UTC_TIMESTAMP(),
WHERE smtp_message_id = #{smtp_message_id} AND
last_attempt_at < DATE_SUB(UTC_TIMESTAMP(),
INTERVAL 900 SECOND)\")
end

SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id => smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
rows_updated = raw_update(\"UPDATE application_message_logs
SET last_attempt_at = UTC_TIMESTAMP(),
WHERE smtp_message_id = #{smtp_message_id} AND
last_attempt_at < DATE_SUB(UTC_TIMESTAMP(),
INTERVAL 900 SECOND)\")
if rows_updated < 1
logger.warn(\"Preventing duplicate msg delivery #{smtp_message_id}\")
return false
else
return true
end
end

Queue lengths inaccurate
for < 1000 items

SQS not
necessarily FIFO

So you may not
want a cloud queue

SimpleDB optimized
for writes, not reads

You must code defensively

There are no good \"cloud
sandboxes\"

Pay attention to
MySQL timeouts

\"User account management
is
-not- ideal.\"
-Deacon
Bradley

You are locked-in to your
provider

Relational DB may not
be the best choice

Is there a benefit?

Changes the way you write
code

You can start
right away

Pretty awesome redundancy

Time for an example?

US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
X XX
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

X
US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
X
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

US-EAST-1C
X
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
X
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

X
US-EAST-1C
SMTP Cloud
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
X
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

Me,recovering

Questions?
mike@oib.com
@subelsky
subelsky.com