Bootstrapping - Session 1 - Your First Week with Amazon EC2

4,681 views

Published on

Amazon Elastic Compute Cloud (Amazon EC2) provides resizable compute capacity in the cloud and is often the starting point for your first week using AWS. This presentation will introduce some essential getting started tips and walk through the journey into AWS, the basic technologies you need to understand and why you should use them. You'll hear real a customer's first year journey and benefit from what they would tell you to do in your first week as they impart the lessons learned, challenges faced and opportunities presented.

Ryan Shuttleworth, Technical Evangelist, AWS
Andrew Dunn, CTO and Nick Hills, Ops Manager, Compliant Phones

Published in: Technology, Economy & Finance

Bootstrapping - Session 1 - Your First Week with Amazon EC2

  1. 1. Ryan Shuttleworth, Technical EvangelistYour first week with EC2(and other AWS things…)
  2. 2. things you should know/think about up frontsome best practices for getting startedessential technologies to dive into and get familiar witharchitectural principles you should immerse yourself inWhat are we going to cover?Your first 5 days with EC2…
  3. 3. hear a ‘looking back at our first year’ customer storycompressed into 5 daysWhat are we going to cover?Your first 5 days with EC2…
  4. 4. organise yourhouse1DAY
  5. 5. Users & RolesStart as you mean to go onSecure your console with IAMrolesA little time spent now will saveheadaches later
  6. 6. Users & RolesStart as you mean to go onSecure your console with IAMrolesA little time spent now will saveheadaches laterAccounts & BillingCreate a structure that makes senseDev & Test accounts vs productionConsolidated billingResource tagging
  7. 7. Master Accountaws.invoices@mycompany.com
  8. 8. Division Badmin@divisionB.comUser2Dev2Admin2IAMMaster Accountaws.invoices@mycompany.comconsolidated billing information
  9. 9. Division Badmin@divisionB.comUser2Dev2Admin2IAMTags:Own=DivProj=PTags:Own=DivProj=QTags:Own=DivProj=RMaster Accountaws.invoices@mycompany.comconsolidated billing information Tags: (key-value)e.g Own=DivProj=R
  10. 10. Operating Co. Aadmin@opcoa.comUser1Dev1Admin1IAMTags:Own=OpCoProj=ATags:Own=OpCoProj=BTags:Own=OpCoProj=CDivision Badmin@divisionB.comUser2Dev2Admin2IAMTags:Own=DivProj=PTags:Own=DivProj=QTags:Own=DivProj=RBusiness Unit Cadmin@busUnitC.comUser3Dev3Admin3IAMTags:Own=BusCProj=XTags:Own=BusCProj=YTags:Own=BusCProj=ZMaster Accountaws.invoices@mycompany.comconsolidated billing information
  11. 11. Operating Co. Aadmin@opcoa.comUser1Dev1Admin1IAMTags:Own=OpCoProj=ATags:Own=OpCoProj=BTags:Own=OpCoProj=CDivision Badmin@divisionB.comUser2Dev2Admin2IAMTags:Own=DivProj=PTags:Own=DivProj=QTags:Own=DivProj=RBusiness Unit Cadmin@busUnitC.comUser3Dev3Admin3IAMTags:Own=BusCProj=XTags:Own=BusCProj=YTags:Own=BusCProj=ZMaster Accountaws.invoices@mycompany.comconsolidated billing information
  12. 12. Master Accountaws.invoices@mycompany.comconsolidated billing informationProgrammatic billing accessS3 CSVOperating Co. Aadmin@opcoa.comUser1Dev1Admin1IAMTags:Own=OpCoProj=ATags:Own=OpCoProj=BTags:Own=OpCoProj=CDivision Badmin@divisionB.comUser2Dev2Admin2IAMTags:Own=DivProj=PTags:Own=DivProj=QTags:Own=DivProj=RBusiness Unit Cadmin@busUnitC.comUser3Dev3Admin3IAMTags:Own=BusCProj=XTags:Own=BusCProj=YTags:Own=BusCProj=Z
  13. 13. Operating Co. Aadmin@opcoa.comUser1Dev1Admin1IAMTags:Own=OpCoProj=ATags:Own=OpCoProj=BTags:Own=OpCoProj=CDivision Badmin@divisionB.comUser2Dev2Admin2IAMTags:Own=DivProj=PTags:Own=DivProj=QTags:Own=DivProj=RBusiness Unit Cadmin@busUnitC.comUser3Dev3Admin3IAMTags:Own=BusCProj=XTags:Own=BusCProj=YTags:Own=BusCProj=ZMaster Accountaws.invoices@mycompany.comconsolidated billing informationProgrammatic billing accessS3 CSV
  14. 14. Secrets & Keys
  15. 15. Secrets & KeysYour front door keys
  16. 16. Secrets & KeysYour front door keysControl access toyour instancesKey managementstrategy
  17. 17. Secrets & KeysControl access toyour APIsYour front door keysUse IAM Roles todistribute to instancesControl access toyour instancesKey managementstrategy
  18. 18. learn thebasics2DAY
  19. 19. What is EC2?Elastic capacity FlexibleComplete controlReliableInexpensiveSecure
  20. 20. Disposable compute
  21. 21. Instance
  22. 22. Instance Unit of scaleUnit of resilienceUnit of control
  23. 23. Instance Unit of scaleUnit of resilienceUnit of controlYour stack
  24. 24. InstanceInstanceInstanceInstanceUnit of scaleUnit of resilienceUnit of controlScaleout
  25. 25. InstanceInstanceInstanceInstanceUnit of scaleUnit of resilienceUnit of control
  26. 26. InstanceInstanceInstanceUnit of scaleUnit of resilienceUnit of control
  27. 27. InstanceInstanceInstanceUnit of scaleUnit of resilienceUnit of controlInstance
  28. 28. Think differentlyCompute is transient
  29. 29. ProgrammaticresourcesTreat your datacentreresources like code
  30. 30. DistributedsystemsProgrammaticresourcesDesign for decoupledsystems up frontTreat your datacentreresources like code
  31. 31. Late bindingDistributedsystemsProgrammaticresourcesDecide what it willrun on at point ofdeploymentDesign for decoupledsystems up frontTreat your datacentreresources like code
  32. 32. go wild withtools3DAY
  33. 33. ComputeStorageSecurity ScalingDatabaseNetworkingMonitoringMessagingWorkflowDNSLoad BalancingBackupCDNAccess everythingvia CLI, API orConsoleAchieve the highest levelsof automationsophistication with easeEverything is programmable
  34. 34. ec2-run-instances ami-54cf5c3d--instance-count 2--group webservers--key mykey--instance-type m1.small$>
  35. 35. >>> import boto.ec2>>> conn = boto.ec2.connect_to_region("us-east-1")>>> conn.run_instances(ami-54cf5c3d,key_name=mykey,instance_type=m1.small,security_groups=[webservers])
  36. 36. Resources created programmatically
  37. 37. Resources created programmaticallyConfigure automatically
  38. 38. Bake an AMIStart an instanceConfigure theinstanceCreate an AMI fromyour instanceStart new ones fromthe AMIBootstrapping
  39. 39. Bake an AMIStart an instanceConfigure theinstanceCreate an AMI fromyour instanceStart new ones fromthe AMIBootstrappingec2-run-instances<your ami-id>$>
  40. 40. Bake an AMI Configure dynamicallyStart an instanceConfigure theinstanceCreate an AMI fromyour instanceStart new ones fromthe AMIBootstrappingLaunch an instanceUse metadata serviceand cloud-init toperform actions oninstance when itlaunchesvsBootstrapping
  41. 41. Bake an AMIBuild your baseimages and setupcustom initialisationscriptsMaintain your ‘golden’baseBootstrappingUse bootstrapping topass custominformation in andperform post launchtasks like pullingcode from SVN+BootstrappingConfigure dynamically
  42. 42. Bake an AMIBootstrappingTime consuming configuration(e.g startup time)Static configurations(e.g less change management)BootstrappingConfigure dynamically
  43. 43. Bake an AMIBootstrappingContinuous deployment(e.g latest code)Environment specific(e.g dev-test-prod)BootstrappingConfigure dynamically
  44. 44. Goal is bring an instance up in auseful stateThe balance will vary depending upon yourapplication
  45. 45. InstancerequestUserdata
  46. 46. InstancerequestUserdataMeta-dataservice
  47. 47. InstancerequestUserdataInstanceMeta-dataservice
  48. 48. #!/bin/shyum -y install httpd php mysql php-mysqlchkconfig httpd on/etc/init.d/httpd startShell script in user-data will be executed on launch:
  49. 49. Amazon Windows EC2Config Service executes user-data on launch:<script>dir > c:test.log</script><powershell>any command that you can run</powershell><powershell>Read-S3Object -BucketName myS3Bucket-Key myFolder/myFile.zip-File c:destinationFile.zip</powershell>AWS Powershell Tools
  50. 50. Why do this?AutomationLess fingers, less mistakesAvailabilityDrive higheravailability withself-healingSecurityInstanceslocked down bydefaultFlexibleShell,Powershell,CloudFormation,Chef, Puppet,OpsWorksScaleManage large scaledeployments and driveautoscalingEfficiencyAudit and manageyour estate withless time & effort
  51. 51. try somethingnew4DAY
  52. 52. UnconstrainedEC2 resources
  53. 53. UnconstrainedComplimentary services
  54. 54. My little instance(created programmatically)
  55. 55. A bit of S3 code(pulled down automatically from S3)>>> from boto.s3.key import Key>>> k = Key(bucket)>>> k.key = foobar>>> k.set_contents_from_string(This is a test of S3)My little instance(created programmatically)
  56. 56. A bit of S3 code(installed automatically)>>> from boto.s3.key import Key>>> k = Key(bucket)>>> k.key = foobar>>> k.set_contents_from_string(This is a test of S3)My little instance(created programmatically)UNLIMITED storagefrom my little instance
  57. 57. Services instead of softwareRemoves undifferentiated heavy lifting
  58. 58. Services instead of softwareRemoves undifferentiated heavy liftingS3 for object storageSQS for queuesRDS for databasesCloudWatch for monitoring
  59. 59. put somethingtogether5DAY
  60. 60. 1. Use multipleavailability zones
  61. 61. 2. Use RDS with replicasand slaves
  62. 62. 3. Use auto-scalinggroups
  63. 63. 4. Use Elastic LoadBalancing
  64. 64. 5. Use Route53 to hostDNS zones
  65. 65. Find out more about model architectures in:Building Web Scale Applications (bootstrapping track)Architecting for High Availability (advanced track)
  66. 66. listen to Nickand Andy’sstory6DAY
  67. 67. Amazon Web ServicesOne Year, in 5 days…
  68. 68. What do we do➔ Record peoples mobile and fixed communications– Voice, sms etc..– Highly sensitive data– Real time analytics and alerting– Over 120 Financial Services organisations and other verticals
  69. 69. What is Important➔ Security➔ Scalability➔ Flexibility➔ Global Platform
  70. 70. Day Zero➔ Recently Funded➔ Full Pipeline of customers➔ 6 Weeks to ‘go live’➔ Web Console Service & Storage needed asap for demos
  71. 71. Day One – I found us a Server!One “slightly firedamaged” servergoing cheap..
  72. 72. Day One – Getting Started
  73. 73. Day One – Lessons Learnt➔ I’m dangerous on my own – I need a SysOps Guru➔ Elastic IPs – Elastic Load Balancers & Route53➔ RDS Monitoring – avoid that Sunday ‘brown alert’moment, use cloud watch➔ Single Instances… No– Stateless and multiple instances➔ Decouple stuff, SQS, SWF, SNS
  74. 74. Day Two
  75. 75. Day Two – Lessons Learnt➔ Larger Estate:– Autoscaling– More Logs (evidence)– More Metrics– More code, more services, more features.➔ Devolve everything (where possible) to be HTTP –easier to scale➔ Make things stateless – accept failure as routine
  76. 76. Day ThreeAvailability Zone #CAvailability Zone #BAvailability Zone #AConsole ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AStorage Frontend ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AStorage Backend ASGSplunk ASG
  77. 77. Day Three – Lessons Learnt➔ No human access to live➔ Centralized Logging– We use Splunk, considered an employee in its own right– Consumes CloudWatch, S3 Logs, Application logs– Hand built alerts and filter by what is relevant– Evidence is key to diagnosis and resolution of any issues➔ Amazon Support – cool story! Invaluable➔ Think Big, you have a potential global platform at yourfingertips.
  78. 78. Availability Zone #CAvailability Zone #BAvailability Zone #AConsole ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AStorage Frontend ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AStorage Backend ASGSplunk ASGAvailability Zone #CAvailability Zone #BAvailability Zone #ACommunications ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AAPI Services ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AAPI Services ASGSplunk ASGAvailability Zone #CAvailability Zone #BZone #AAudit & Reporting ASGAmazonCloudSearchAmazon SNSQueueQueueQueueQueueVPN GatewayCustomerGatewayAWS DirectConnectAmazon Route 53Availability Zone #CAvailability Zone #BAvailability Zone #AConsole ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AStorage Frontend ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AStorage Backend ASGSplunk ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AAPI Services ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AAPI Services ASGSplunk ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AAudit & Reporting ASGAmazonCloudSearchQueueQueueQueueQueueAvailability Zone #CAvailability Zone #BAvailability Zone #AConsole ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AStorage Frontend ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AStorage Backend ASGSplunk ASGAvailability Zone #CAvailability Zone #BAvailability Zone #ACommunications ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AAPI Services ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AAPI Services ASGSplunk ASGAvailability Zone #CAvailability Zone #BAvailability Zone #AAudit & Reporting ASGAmazonCloudSearchAmazon SNSQueueQueueQueueQueueAmazon Route 53DayFour
  79. 79. Day Four– Lessons Learnt➔ Check your growth – Easy to run more than you need.➔ Document and diagram your system – roguse instances.➔ Use Consolidated Billing– Separate accounts for Live & Test.– Get it off the credit card – Invoiced Billing➔ Use as many of the tools as you can. - Don’t re-inventthe wheel, exploit the full ecosystem.➔ Read the blogs, announcements, examples, and talk tothe AWS SA’s
  80. 80. Day Five➔ Check out trusted advisor– Whats your score?➔ Leave work early!– You have a Global fault tolerant self healing system.– It notifies you if there is a problem, and then resolves itself.– You can keep an eye on your logs from the Pub.➔ On a beer mat, do your reserved instance calculations– Buy reserved instances on Monday!
  81. 81. Lessons Learnt➔ DevOps – tightly coupled development and systemsteams = rapid evolution➔ Building it right (evolution) allowed us to take time outand not be worried by failure➔ Failure doesnt have to be all bad, if you expect it.➔ Automation of testing, release and the deploymentprocess, removes the risk from human mistake.
  82. 82. www.compliantphones.com
  83. 83. Monitoring, auto-scaling, de-couplingStateless, expect failureLock-down live, AWS support rocks, think big!Consolidated billing, follow the blogReserved instancesSummaryLessons learned…aws.typepad.com

×