SlideShare a Scribd company logo
1 of 50
Download to read offline
It’s not always sunny in the clouds
             Me,
             teetering




                 Text
Mike Subelsky




 @subelsky
Mike Subelsky




 @subelsky
1 / 1 / 08
08:00:00 ET
“Everything needs
to be automated”
Autoscaling is the easiest
          part
Think carefully about
credential management
You really could use internal
            DNS
It's maybe not that cheap
Launching servers is not
       that fast
You will become dependent
    on “glue” services
You will become dependent
    on “glue” services
You will depend on a distant
     faceless provider
Use DVCS
You will spend a lot of time
      on monitoring
Your logs will
runneth over
Your logs will
runneth over
Write lots of
“in-process tests”
Write lots of
          “in-process tests”

m = Message.find_last_slug_message
secs = m.created_at - m.date_sent

if secs > 600
  ErrorNotifier.fatal(quot;Missed slug deliveryquot;,secs,m)
end
Snapshots are slow
Snapshots are slow
Rails will be the
least of your worries
Cloud services involve
 subtle-yet-massive
      tradeos
SQS guarantees delivery at
       least once
SQS guarantees delivery at
          least once
def self.guard_against_repeat_delivery(smtp_message_id)
 begin
  create!(:smtp_message_id = smtp_message_id)
  return true
 rescue ActiveRecord::StatementInvalid




 end
SQS guarantees delivery at
         least once
def self.guard_against_repeat_delivery(smtp_message_id)
 begin
  create!(:smtp_message_id = smtp_message_id)
  return true
 rescue ActiveRecord::StatementInvalid
  rows_updated = raw_update(quot;UPDATE application_message_logs
                    SET last_attempt_at = UTC_TIMESTAMP(),
                    WHERE smtp_message_id = #{smtp_message_id} AND
                    last_attempt_at  DATE_SUB(UTC_TIMESTAMP(),
                             INTERVAL 900 SECOND)quot;)




 end
SQS guarantees delivery at
         least once
def self.guard_against_repeat_delivery(smtp_message_id)
 begin
  create!(:smtp_message_id = smtp_message_id)
  return true
 rescue ActiveRecord::StatementInvalid
  rows_updated = raw_update(quot;UPDATE application_message_logs
                    SET last_attempt_at = UTC_TIMESTAMP(),
                    WHERE smtp_message_id = #{smtp_message_id} AND
                    last_attempt_at  DATE_SUB(UTC_TIMESTAMP(),
                             INTERVAL 900 SECOND)quot;)

  if rows_updated  1
    logger.warn(quot;Preventing duplicate msg delivery #{smtp_message_id}quot;)
    return false
  else
    return true
  end
 end
Queue lengths inaccurate
   for  1000 items
SQS not
necessarily FIFO
So you may not
want a cloud queue
SimpleDB optimized
for writes, not reads
You must code defensively
There are no good quot;cloud
      sandboxesquot;
Pay attention to
MySQL timeouts
quot;User account management
            is
       -not- ideal.quot;
             -Deacon
              Bradley
You are locked-in to your
         provider
Relational DB may not
 be the best choice
Is there a benefit?
Changes the way you write
          code
You can start
 right away
Pretty awesome redundancy
Time for an example?
US-EAST-1C
                            o
                       t                           Ta
                    me                  SQS           s
                   a                                   kin
                 en e
              Fil ars                                      g
              3P
             S


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB




                                                            le
                 Ra




                                                          Fi
                   w
                       Mb




                                                       x
                                                    bo
                           ox




                                                   M
                                 Fil




                                               w
                                    e




                                              Ra
                                        S3




US-EAST-1B
                            to                     Ta
                       e                SQS
                    am                                skin
                  en e
               Fil ars                                     g
             S3 P


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB
                                                           le
                 Ra
                                                          Fi

                   w
                       Mb
                                                       x
                                                     bo




                           ox
                                                   M




                                 Fil
                                                 w




                                    e
                                              Ra




                                        S3
US-EAST-1C
                            o
                       t                           Ta
                    me                  SQS           s
                   a                                   kin
                 en e
              Fil ars                                      g
              3P
             S


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB




                                                            le
                 Ra




                                                          Fi
                   w
                       Mb




                                                       x
                                                    bo
                           ox




                                                   M
                                 Fil




                                               w
                                    e




                                              Ra
                                        S3




X XX
US-EAST-1B
                            to                     Ta
                       e                SQS
                    am                                skin
                  en e
               Fil ars                                     g
             S3 P


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB
                                                           le
                 Ra
                                                          Fi

                   w
                       Mb
                                                       x
                                                     bo




                           ox
                                                   M




                                 Fil
                                                 w




                                    e
                                              Ra




                                        S3
X
US-EAST-1C
                            o
                       t                           Ta
                    me                  SQS           s
                   a                                   kin
                 en e
              Fil ars                                      g
              3P
             S


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB




                                                            le
                 Ra




                                                          Fi
                   w
                       Mb




                                                       x
                                                    bo
                           ox




                                                   M
                                 Fil




                                               w
                                    e




                                              Ra
                                        S3




                                                                                         X
US-EAST-1B
                            to                     Ta
                       e                SQS
                    am                                skin
                  en e
               Fil ars                                     g
             S3 P


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB
                                                           le
                 Ra
                                                          Fi

                   w
                       Mb
                                                       x
                                                     bo




                           ox
                                                   M




                                 Fil
                                                 w




                                    e
                                              Ra




                                        S3
US-EAST-1C




                                                          X
                            o
                       t                           Ta
                    me                  SQS           s
                   a                                   kin
                 en e
              Fil ars                                      g
              3P
             S


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB




                                                            le
                 Ra




                                                          Fi
                   w
                       Mb




                                                       x
                                                    bo
                           ox




                                                   M
                                 Fil




                                               w
                                    e




                                              Ra
                                        S3




US-EAST-1B




                                                          X
                            to                     Ta
                       e                SQS
                    am                                skin
                  en e
               Fil ars                                     g
             S3 P


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB
                                                           le
                 Ra
                                                          Fi

                   w
                       Mb
                                                       x
                                                     bo




                           ox
                                                   M




                                 Fil
                                                 w




                                    e
                                              Ra




                                        S3
X
US-EAST-1C

                             SMTP Cloud
                            o
                       t                           Ta
                    me                  SQS           s
                   a                                   kin
                 en e
              Fil ars                                      g
              3P
             S


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB




                                                            le
                 Ra




                                                          Fi
                   w
                       Mb




                                                       x
                                                    bo
                           ox




                                                   M
                                 Fil




                                               w
                                    e




                                              Ra
                                        S3




US-EAST-1B




                                  X
                            to                     Ta
                       e                SQS
                    am                                skin
                  en e
               Fil ars                                     g
             S3 P


    Postfix   MailSender                                      MailReceiver   MailParser   Master DB
                                                           le


                 Ra
                                                          Fi



                   w
                       Mb
                                                       x
                                                     bo




                           ox
                                                   M




                                 Fil
                                                 w




                                    e
                                              Ra




                                        S3
It's Not Always Sunny in the Clouds
Me,recovering
Questions?


mike@oib.com
 @subelsky

subelsky.com

More Related Content

Recently uploaded

Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 

Recently uploaded (20)

Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 

It's Not Always Sunny in the Clouds

Editor's Notes

  1.      - so you either have to pay for RightScale, EY, Heroku, Scalr, etc.      - or you have to roll your own      - or you can install an open source project      - either way, it's going to cost you time or treasure
  2. Nine different sets of creds to manage Maybe YAML not so much
  3. some people, like Gartner, think AWS still charges too much
  4. - general AWS launch time - JRuby JVM
  5. - For us, SQS, S3, EC2
  6. You can get AWS SLAs, premium support
  7. otherwise you’re too dependent on the VCS host
  8. - Because nodes are ephemeral - We use Alertra & custom scripts & rightscale - we've written a ton of custom monitoring scrips
  9. one log file turns out to be better than many 40+ boxes, ephemeral so don’t save logs have not found a palatable alternative to splunk hoptoad FTW
  10. - use to test \"can we hit SQS a lot\", now we test \"can a message get delivered in a timely fashion end-to-end\" - show snippet
  11. but can be a lifesaver
  12. - show idempotency code example
  13. - show idempotency code example
  14. - show idempotency code example
  15. - esp if you have a task that HAS to be performed immediately - we use this for email delivery and generally is fast enough
  16. you miss a lot of awesome aggregation that RDBMs does for you
  17. - Cloud services give a lot of expected and unexpected errors - Write lots of exponential retry code
  18. - Testing is going to be difficult - Need something like Paypal test gateway - Fakeweb shows promise I’m working on something
  19. - connections for killed-off servers may hang around - lowering the timeout caused lots of issues for seemingly-unrelated pieces
  20. \"A system like Kerberos which is ideal for multi-server multi-role environments requires too much overhead to setup”
  21. - cloudkick may change this - it's bound to change
  22. - forces you to loosely couple components - affords you trade offs - you can throw more servers at a problem
  23. - very low activation energy
  24. - very low activation energy