The document discusses various challenges and tradeoffs of moving systems and services to the cloud, noting that autoscaling may be easier than expected but other aspects like monitoring, logging, and managing credentials can become more complex. It also touches on dependencies on cloud providers and "glue" services that connect cloud components as well as best practices for testing, defensive coding, and database choices when building cloud-based systems.
25. SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id = smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
end
26. SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id = smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
rows_updated = raw_update(quot;UPDATE application_message_logs
SET last_attempt_at = UTC_TIMESTAMP(),
WHERE smtp_message_id = #{smtp_message_id} AND
last_attempt_at DATE_SUB(UTC_TIMESTAMP(),
INTERVAL 900 SECOND)quot;)
end
27. SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id = smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
rows_updated = raw_update(quot;UPDATE application_message_logs
SET last_attempt_at = UTC_TIMESTAMP(),
WHERE smtp_message_id = #{smtp_message_id} AND
last_attempt_at DATE_SUB(UTC_TIMESTAMP(),
INTERVAL 900 SECOND)quot;)
if rows_updated 1
logger.warn(quot;Preventing duplicate msg delivery #{smtp_message_id}quot;)
return false
else
return true
end
end
43. US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
44. US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
X XX
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
45. X
US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
X
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
46. US-EAST-1C
X
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
X
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
47. X
US-EAST-1C
SMTP Cloud
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
X
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
     - so you either have to pay for RightScale, EY, Heroku, Scalr, etc.
     - or you have to roll your own
     - or you can install an open source project
     - either way, it's going to cost you time or treasure
Nine different sets of creds to manage
Maybe YAML not so much
some people, like Gartner, think AWS still charges too much
- general AWS launch time
- JRuby JVM
- For us, SQS, S3, EC2
You can get AWS SLAs, premium support
otherwise you’re too dependent on the VCS host
- Because nodes are ephemeral
- We use Alertra & custom scripts & rightscale
- we've written a ton of custom monitoring scrips
one log file turns out to be better than many
40+ boxes, ephemeral so don’t save logs
have not found a palatable alternative to splunk
hoptoad FTW
- use to test \"can we hit SQS a lot\", now we test \"can a message get delivered in a timely fashion end-to-end\"
- show snippet
but can be a lifesaver
- show idempotency code example
- show idempotency code example
- show idempotency code example
- esp if you have a task that HAS to be performed immediately
- we use this for email delivery and generally is fast enough
you miss a lot of awesome aggregation that RDBMs does for you
- Cloud services give a lot of expected and unexpected errors
- Write lots of exponential retry code
- Testing is going to be difficult
- Need something like Paypal test gateway
- Fakeweb shows promise
I’m working on something
- connections for killed-off servers may hang around
- lowering the timeout caused lots of issues for seemingly-unrelated pieces
\"A system like Kerberos which is ideal for multi-server multi-role environments requires too much overhead to setup”
- cloudkick may change this
- it's bound to change
- forces you to loosely couple components
- affords you trade offs - you can throw more servers at a problem