It's Not Always Sunny in the Clouds

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

2 comments

Comments 1 - 2 of 2 previous next Post a comment

  • + subelsky subelsky 6 months ago
    thanks! unfortunately they didn’t video anything. Maybe I can record a special edition for slideshare!
  • + jboutelle Jonathan Boutelle 6 months ago
    Awesome presentation! Would love to hear the audio for this...
Post a comment
Embed Video
Edit your comment Cancel

Notes on slide 1




     - so you either have to pay for RightScale, EY, Heroku, Scalr, etc.
     - or you have to roll your own
     - or you can install an open source project
     - either way, it's going to cost you time or treasure


Nine different sets of creds to manage
Maybe YAML not so much


some people, like Gartner, think AWS still charges too much

- general AWS launch time
- JRuby JVM

- For us, SQS, S3, EC2

You can get AWS SLAs, premium support

otherwise you’re too dependent on the VCS host

- Because nodes are ephemeral
- We use Alertra & custom scripts & rightscale
- we've written a ton of custom monitoring scrips

one log file turns out to be better than many
40+ boxes, ephemeral so don’t save logs
have not found a palatable alternative to splunk
hoptoad FTW

- use to test \"can we hit SQS a lot\", now we test \"can a message get delivered in a timely fashion end-to-end\"
- show snippet

but can be a lifesaver



- show idempotency code example

- show idempotency code example

- show idempotency code example



- esp if you have a task that HAS to be performed immediately
- we use this for email delivery and generally is fast enough

you miss a lot of awesome aggregation that RDBMs does for you

- Cloud services give a lot of expected and unexpected errors
- Write lots of exponential retry code

- Testing is going to be difficult
- Need something like Paypal test gateway
- Fakeweb shows promise
I’m working on something

- connections for killed-off servers may hang around
- lowering the timeout caused lots of issues for seemingly-unrelated pieces

\"A system like Kerberos which is ideal for multi-server multi-role environments requires too much overhead to setupā€

- cloudkick may change this
- it's bound to change



- forces you to loosely couple components
- affords you trade offs - you can throw more servers at a problem

- very low activation energy

- very low activation energy








2 Favorites

It's Not Always Sunny in the Clouds - Presentation Transcript

  1. It’s not always sunny in the clouds Me, teetering Text
  2. Mike Subelsky @subelsky
  3. Mike Subelsky @subelsky
  4. 1 / 1 / 08 08:00:00 ET
  5. ā€œEverything needs to be automatedā€
  6. Autoscaling is the easiest part
  7. Think carefully about credential management
  8. You really could use internal DNS
  9. It's maybe not that cheap
  10. Launching servers is not that fast
  11. You will become dependent on ā€œglueā€ services
  12. You will become dependent on ā€œglueā€ services
  13. You will depend on a distant faceless provider
  14. Use DVCS
  15. You will spend a lot of time on monitoring
  16. Your logs will runneth over
  17. Your logs will runneth over
  18. Write lots of ā€œin-process testsā€
  19. Write lots of ā€œin-process testsā€ m = Message.find_last_slug_message secs = m.created_at - m.date_sent if secs > 600 ErrorNotifier.fatal(\"Missed slug delivery\",secs,m) end
  20. Snapshots are slow
  21. Snapshots are slow
  22. Rails will be the least of your worries
  23. Cloud services involve subtle-yet-massive tradeos
  24. SQS guarantees delivery at least once
  25. SQS guarantees delivery at least once def self.guard_against_repeat_delivery(smtp_message_id) begin create!(:smtp_message_id => smtp_message_id) return true rescue ActiveRecord::StatementInvalid end
  26. SQS guarantees delivery at least once def self.guard_against_repeat_delivery(smtp_message_id) begin create!(:smtp_message_id => smtp_message_id) return true rescue ActiveRecord::StatementInvalid rows_updated = raw_update(\"UPDATE application_message_logs SET last_attempt_at = UTC_TIMESTAMP(), WHERE smtp_message_id = #{smtp_message_id} AND last_attempt_at < DATE_SUB(UTC_TIMESTAMP(), INTERVAL 900 SECOND)\") end
  27. SQS guarantees delivery at least once def self.guard_against_repeat_delivery(smtp_message_id) begin create!(:smtp_message_id => smtp_message_id) return true rescue ActiveRecord::StatementInvalid rows_updated = raw_update(\"UPDATE application_message_logs SET last_attempt_at = UTC_TIMESTAMP(), WHERE smtp_message_id = #{smtp_message_id} AND last_attempt_at < DATE_SUB(UTC_TIMESTAMP(), INTERVAL 900 SECOND)\") if rows_updated < 1 logger.warn(\"Preventing duplicate msg delivery #{smtp_message_id}\") return false else return true end end
  28. Queue lengths inaccurate for < 1000 items
  29. SQS not necessarily FIFO
  30. So you may not want a cloud queue
  31. SimpleDB optimized for writes, not reads
  32. You must code defensively
  33. There are no good \"cloud sandboxes\"
  34. Pay attention to MySQL timeouts
  35. \"User account management is -not- ideal.\" -Deacon Bradley
  36. You are locked-in to your provider
  37. Relational DB may not be the best choice
  38. Is there a benefit?
  39. Changes the way you write code
  40. You can start right away
  41. Pretty awesome redundancy
  42. Time for an example?
  43. US-EAST-1C o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 US-EAST-1B to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
  44. US-EAST-1C o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 X XX US-EAST-1B to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
  45. X US-EAST-1C o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 X US-EAST-1B to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
  46. US-EAST-1C X o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 US-EAST-1B X to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
  47. X US-EAST-1C SMTP Cloud o t Ta me SQS s a kin en e Fil ars g 3P S Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3 US-EAST-1B X to Ta e SQS am skin en e Fil ars g S3 P Postfix MailSender MailReceiver MailParser Master DB le Ra Fi w Mb x bo ox M Fil w e Ra S3
  48. Me,recovering
  49. Questions? mike@oib.com @subelsky subelsky.com

+ subelskysubelsky, 6 months ago

custom

1492 views, 2 favs, 6 embeds more stats

I discuss why I became disillusioned with cloud com more

More info about this document

© All Rights Reserved

Go to text version

  • Total Views 1492
    • 1418 on SlideShare
    • 74 from embeds
  • Comments 2
  • Favorites 2
  • Downloads 56
Most viewed embeds
  • 60 views on http://www.subelsky.com
  • 8 views on http://subelsky.com
  • 3 views on http://damon.tumblr.com
  • 1 views on http://draft.blogger.com
  • 1 views on http://www.slideshare.net

more

All embeds
  • 60 views on http://www.subelsky.com
  • 8 views on http://subelsky.com
  • 3 views on http://damon.tumblr.com
  • 1 views on http://draft.blogger.com
  • 1 views on http://www.slideshare.net
  • 1 views on http://www.subelsky.net

less

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

Cancel
File a copyright complaint
Having problems? Go to our helpdesk?

Categories