0
It’s not always sunny in the clouds
             Me,
             teetering




                 Text
Mike Subelsky




 @subelsky
Mike Subelsky




 @subelsky
1 / 1 / 08
08:00:00 ET
“Everything needs
to be automated”
Autoscaling is the easiest
          part
Think carefully about
credential management
You really could use internal
            DNS
It's maybe not that cheap
Launching servers is not
       that fast
You will become dependent
    on “glue” services
You will become dependent
    on “glue” services
You will depend on a distant
     faceless provider
Use DVCS
You will spend a lot of time
      on monitoring
Your logs will
runneth over
Your logs will
runneth over
Write lots of
“in-process tests”
Write lots of
          “in-process tests”

m = Message.find_last_slug_message
secs = m.created_at - m.date_sent

if secs >...
Snapshots are slow
Snapshots are slow
Rails will be the
least of your worries
Cloud services involve
 subtle-yet-massive
      tradeos
SQS guarantees delivery at
       least once
SQS guarantees delivery at
          least once
def self.guard_against_repeat_delivery(smtp_message_id)
 begin
  create!(:...
SQS guarantees delivery at
         least once
def self.guard_against_repeat_delivery(smtp_message_id)
 begin
  create!(:s...
SQS guarantees delivery at
         least once
def self.guard_against_repeat_delivery(smtp_message_id)
 begin
  create!(:s...
Queue lengths inaccurate
   for  1000 items
SQS not
necessarily FIFO
So you may not
want a cloud queue
SimpleDB optimized
for writes, not reads
You must code defensively
There are no good quot;cloud
      sandboxesquot;
Pay attention to
MySQL timeouts
quot;User account management
            is
       -not- ideal.quot;
             -Deacon
              Bradley
You are locked-in to your
         provider
Relational DB may not
 be the best choice
Is there a benefit?
Changes the way you write
          code
You can start
 right away
Pretty awesome redundancy
Time for an example?
US-EAST-1C
                            o
                       t                           Ta
                    me     ...
US-EAST-1C
                            o
                       t                           Ta
                    me     ...
X
US-EAST-1C
                            o
                       t                           Ta
                    me   ...
US-EAST-1C




                                                          X
                            o
                 ...
X
US-EAST-1C

                             SMTP Cloud
                            o
                       t              ...
Me,recovering
Questions?


mike@oib.com
 @subelsky

subelsky.com
It's Not Always Sunny in the Clouds
Upcoming SlideShare
Loading in...5
×

It's Not Always Sunny in the Clouds

1,756

Published on

I discuss why I became disillusioned with cloud computing and why I ultimately came to embrace it, using the case study of OtherInbox.com

Published in: Technology, Education
2 Comments
2 Likes
Statistics
Notes
No Downloads
Views
Total Views
1,756
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
63
Comments
2
Likes
2
Embeds 0
No embeds

No notes for slide



  •      - so you either have to pay for RightScale, EY, Heroku, Scalr, etc.
         - or you have to roll your own
         - or you can install an open source project
         - either way, it's going to cost you time or treasure


  • Nine different sets of creds to manage
    Maybe YAML not so much

  • some people, like Gartner, think AWS still charges too much
  • - general AWS launch time
    - JRuby JVM

  • - For us, SQS, S3, EC2

  • You can get AWS SLAs, premium support
  • otherwise you’re too dependent on the VCS host
  • - Because nodes are ephemeral
    - We use Alertra & custom scripts & rightscale
    - we've written a ton of custom monitoring scrips

  • one log file turns out to be better than many
    40+ boxes, ephemeral so don’t save logs
    have not found a palatable alternative to splunk
    hoptoad FTW

  • - use to test \"can we hit SQS a lot\", now we test \"can a message get delivered in a timely fashion end-to-end\"
    - show snippet

  • but can be a lifesaver


  • - show idempotency code example
  • - show idempotency code example
  • - show idempotency code example


  • - esp if you have a task that HAS to be performed immediately
    - we use this for email delivery and generally is fast enough
  • you miss a lot of awesome aggregation that RDBMs does for you

  • - Cloud services give a lot of expected and unexpected errors
    - Write lots of exponential retry code

  • - Testing is going to be difficult
    - Need something like Paypal test gateway
    - Fakeweb shows promise
    I’m working on something
  • - connections for killed-off servers may hang around
    - lowering the timeout caused lots of issues for seemingly-unrelated pieces
  • \"A system like Kerberos which is ideal for multi-server multi-role environments requires too much overhead to setup”
  • - cloudkick may change this
    - it's bound to change



  • - forces you to loosely couple components
    - affords you trade offs - you can throw more servers at a problem

  • - very low activation energy

  • - very low activation energy








  • Transcript of "It's Not Always Sunny in the Clouds"

    1. 1. It’s not always sunny in the clouds Me, teetering Text
    2. 2. Mike Subelsky @subelsky
    3. 3. Mike Subelsky @subelsky
    4. 4. 1 / 1 / 08 08:00:00 ET
    5. 5. “Everything needs to be automated”
    6. 6. Autoscaling is the easiest part
    7. 7. Think carefully about credential management
    8. 8. You really could use internal DNS
    9. 9. It's maybe not that cheap
    10. 10. Launching servers is not that fast
    11. 11. You will become dependent on “glue” services
    12. 12. You will become dependent on “glue” services
    13. 13. You will depend on a distant faceless provider
    14. 14. Use DVCS
    15. 15. You will spend a lot of time on monitoring
    16. 16. Your logs will runneth over
    17. 17. Your logs will runneth over
    18. 18. Write lots of “in-process tests”
    19. 19. Write lots of “in-process tests” m = Message.find_last_slug_message secs = m.created_at - m.date_sent if secs > 600 ErrorNotifier.fatal(quot;Missed slug deliveryquot;,secs,m) end
    20. 20. Snapshots are slow
    21. 21. Snapshots are slow
    22. 22. Rails will be the least of your worries
    23. 23. Cloud services involve subtle-yet-massive tradeos
    24. 24. SQS guarantees delivery at least once
    25. 25. SQS guarantees delivery at least once def self.guard_against_repeat_delivery(smtp_message_id) begin create!(:smtp_message_id = smtp_message_id) return true rescue ActiveRecord::StatementInvalid end
    26. 26. SQS guarantees delivery at least once def self.guard_against_repeat_delivery(smtp_message_id) begin create!(:smtp_message_id = smtp_message_id) return true rescue ActiveRecord::StatementInvalid rows_updated = raw_update(quot;UPDATE application_message_logs SET last_attempt_at = UTC_TIMESTAMP(), WHERE smtp_message_id = #{smtp_message_id} AND last_attempt_at DATE_SUB(UTC_TIMESTAMP(), INTERVAL 900 SECOND)quot;) end
    27. 27. SQS guarantees delivery at least once def self.guard_against_repeat_delivery(smtp_message_id) begin create!(:smtp_message_id = smtp_message_id) return true rescue ActiveRecord::StatementInvalid rows_updated = raw_update(quot;UPDATE application_message_logs SET last_attempt_at = UTC_TIMESTAMP(), WHERE smtp_message_id = #{smtp_message_id} AND last_attempt_at DATE_SUB(UTC_TIMESTAMP(), INTERVAL 900 SECOND)quot;) if rows_updated 1 logger.warn(quot;Preventing duplicate msg delivery #{smtp_message_id}quot;) return false else return true end end
    28. 28. Queue lengths inaccurate for 1000 items
    29. 29. SQS not necessarily FIFO
    30. 30. So you may not want a cloud queue
    31. 31. SimpleDB optimized for writes, not reads
    32. 32. You must code defensively
    33. 33. There are no good quot;cloud sandboxesquot;
    34. 34. Pay attention to MySQL timeouts
    35. 35. quot;User account management is -not- ideal.quot; -Deacon Bradley
    36. 36. You are locked-in to your provider
    37. 37. Relational DB may not be the best choice
    38. 38. Is there a benefit?
    39. 39. Changes the way you write code
    40. 40. You can start right away
    41. 41. Pretty awesome redundancy
    42. 42. Time for an example?
    43. 43. US-EAST-1C o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 US-EAST-1B to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
    44. 44. US-EAST-1C o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 X XX US-EAST-1B to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
    45. 45. X US-EAST-1C o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 X US-EAST-1B to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
    46. 46. US-EAST-1C X o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 US-EAST-1B X to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
    47. 47. X US-EAST-1C SMTP Cloud o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 US-EAST-1B X to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
    48. 48. Me,recovering
    49. 49. Questions? mike@oib.com @subelsky subelsky.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×