Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Taking advantage of the Amazon Web Services (AWS) Family

6,808 views

Published on

Presentation on Amazon Web Services which I presented at DDD North on Saturday 8th October 2011

Published in: Technology
  • Be the first to comment

Taking advantage of the Amazon Web Services (AWS) Family

  1. 1. Taking advantage of the Amazon Web Services Family<br />@Ben_Hall<br />Ben@BenHall.me.uk<br />Blog.BenHall.me.uk<br />MaydayHQ.com<br />
  2. 2. Agenda<br />S3<br />EC2<br />Structuring applications<br />Others<br />
  3. 3. What is AWS?<br />The way most startups survive<br />Collection of services each solving one small part of the overall problem<br />Platform / Foundation<br />
  4. 4.
  5. 5. http://www.cargolaw.com/images/disaster2007.Ital.Florida7.GIF<br />Amazon S3Amazon Simple Storage Service<br />
  6. 6. Amazon Product ImagesTwitter Profile ImagesSmugMug – 10tb each monthDropboxCurrently has 449 billion files<br />
  7. 7. AppHarbour’s (or any PaaS) storage recommendation<br />
  8. 8.
  9. 9.
  10. 10. Designed to provide 99.99% durability and 99.99% availability <br />Designed to provide 99.999999999% durability and 99.99% availability<br />Also versioning allows you to preserve, retrieve, and restore <br />every version of every object stored in your bucket<br />
  11. 11.
  12. 12. How to access files?<br />
  13. 13. Bucket Name<br />https://s3.amazonaws.com/DDDNorth_Demo_1/<br />ddd_north_s3.html<br />File Name<br />
  14. 14.
  15. 15.
  16. 16. REST API<br />GET<br />DELETE<br />PUT<br />http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html<br />
  17. 17. GET / HTTP/1.1<br />Host: BucketName.s3.amazonaws.com<br />Authorization: AWS 15B4D3461F177624206A:xQE0diMbLRepdf3YB+FIEXAMPLE=<br /><?xml version="1.0" encoding="UTF-8"?><br /><ListBucketResultxmlns="http://s3.amazonaws.com/doc/2006-03-01"><br /> <Name>bucket</Name><br /> <Prefix/><br /> <Marker/><br /> <MaxKeys>1000</MaxKeys><br /> <IsTruncated>false</IsTruncated><br /> <Contents><br /><Key>my-image.jpg</Key><br /> <LastModified>2009-10-12T17:50:30.000Z</LastModified><br /> <ETag>&quot;fba9dede5f27731c9771645a39863328&quot;</ETag><br /> <Size>434234</Size><br /> <StorageClass>STANDARD</StorageClass><br /> <Owner><br /> <ID>8a6925ce4a7f21c32aa379004fef</ID><br /> <DisplayName>mtd@amazon.com</DisplayName><br /> </Owner><br /> </Contents><br /></ListBucketResult><br />
  18. 18. PUT /my-image.jpg HTTP/1.1<br />Host: myBucket.s3.amazonaws.com<br />Authorization: AWS 15B4D3461F177624206A:xQE0diMbLRepdf3YB+FIEXAMPLE=<br />Content-Type: text/plain<br />Content-Length: 11434<br />Expect: 100-continue<br />HTTP/1.1 100 Continue<br />HTTP/1.1 200 OK<br />x-amz-id-2: LriYPLdmOdAiIfgSm/F1YsViT1LW94/xUQxMsF7xiEb1a0wiIOIxl+zbwZ163pt7<br />x-amz-request-id: 0A49CE4060975EAC<br />x-amz-version-id: 43jfkodU8493jnFJD9fjj3HHNVfdsQUIFDNsidf038jfdsjGFDSIRp<br />Date: Wed, 12 Oct 2009 17:50:00 GMT<br />ETag: "fbacf535f27731c9771645a39863328"<br />Content-Length: 0<br />Connection: close<br />Server: AmazonS3<br />
  19. 19. .NET SDK<br />using (AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretKey))<br />{<br />MemoryStreamms = new MemoryStream();<br />PutObjectRequest request = new PutObjectRequest();<br />request.WithBucketName("Demo_1")<br /> .WithCannedACL(S3CannedACL.PublicRead)<br /> .WithKey("image.jpg").InputStream = file.InputStream;<br /> S3Response response = client.PutObject(request);<br />}<br />http://aws.amazon.com/sdkfornet/<br />
  20. 20. .NET SDK<br />ListObjectsRequest request = new ListObjectsRequest();<br />request.BucketName = bucketName;<br />using (ListObjectsResponse response = client.ListObjects(request))<br />{<br />foreach (S3Object entry in response.S3Objects)<br /> {<br />Console.WriteLine("key = {0} size = {1}", entry.Key, entry.Size);<br /> }<br />}<br />
  21. 21. https://s3.amazonaws.com/DDDNorth_Demo_1/<br />ddd_north_s3.html?torrent<br />S3 Torrent Support<br />
  22. 22.
  23. 23.
  24. 24. CloudFront<br />Video Streaming<br />
  25. 25. Index Page<br />
  26. 26. Bucket Name<br />
  27. 27.
  28. 28. date = Time.now.strftime("%a, %d %b %Y %H:%M:%S %Z")<br />digest = Base64.encode64(OpenSSL::HMAC.digest(OpenSSL::Digest::Digest.new('sha1'), aws_secret, date)).strip<br />uri = URI.parse("https://cloudfront.amazonaws.com/2010-08-01/distribution/#{distribution}/invalidation")<br />req = Net::HTTP::Post.new(uri.path)<br />req.initialize_http_header({<br /> 'x-amz-date' => date,<br /> 'Content-Type' => 'text/xml',<br /> 'Authorization' => "AWS %s:%s" % [aws_account, digest]<br />})<br />req.body = "<InvalidationBatch><Path>" + path + "</Path><CallerReference>ref_#{Time.now.utc.to_i}</CallerReference></InvalidationBatch>"<br />http = Net::HTTP.new(uri.host, uri.port)<br />http.use_ssl = true<br />http.verify_mode = OpenSSL::SSL::VERIFY_NONE<br />res = http.request(req)<br />
  29. 29. WARNING<br />HTTPS and CNAMEs are not supported<br />S3 bucket name must only contain lower case alphanumeric characters, periods, or hyphens<br />Bucket names must be unique across entire AWS<br />You can’t rename – Delete + Recreate (which costs)<br />
  30. 30. Cost<br />Data transfer from EC2 machines in same region are free<br />AppHarbour runs out of US East Region, meaning transfers to US Standard are free.<br />
  31. 31. 1gb stored, 10gb downloaded via 1,000,000 requests == $2.22 per month<br />1gb stored, 1tb downloaded via 10,000,000 requests == $132.90 per month<br />CloudFront has additional pricing. 10gb = $2<br />http://calculator.s3.amazonaws.com/calc5.html<br />
  32. 32.
  33. 33. Amazon EC2Amazon Elastic compute cloud<br />http://www.flickr.com/photos/clarkk/38817024/<br />
  34. 34. Heroku, AppHarbouretcZynga > 12,000 nodesAmazon SilkAmazon<br />
  35. 35. Built on top of XEN virtualisationYes, you could run a private EC2<br />
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41. Be safe<br />
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49. Instance Types<br />On demand<br />Reserved<br />Spot<br />
  50. 50. Spot Instance Bidding<br />Think Differently.<br />Design a system capable of supporting it<br />Save!<br />
  51. 51.
  52. 52.
  53. 53. Instance Sizes<br />32bit – micro, small, medium<br />64bit – Extra-large ><br />Super computer – GPU<br />1 ECU (biggest == 20) == 2007 Xeon processor / 1.0-1.2 GHz 2007 Opteron<br />
  54. 54. Availability Zones<br />Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zone<br />2-3 per region.<br />
  55. 55. Regions<br />US East (Northern Virginia)<br />US West (Northern California)<br />EU (Ireland)<br />Asia Pacific (Singapore)<br />Asia Pacific (Tokyo)<br />
  56. 56. Business / Technical Decisions<br />Geo-location<br />It’s a data centre, it can die. Scale across multiple regions for maximum up-time.<br />
  57. 57. EBS & Snapshots<br />EBS == Hard drive.<br />Can increase volume size, but means taking machine down. <br />Snapshots == Backup. Can attach to new / different AMI<br />
  58. 58.
  59. 59.
  60. 60. Keypairs<br />BE CAREFUL – DON’T LOSE THE FILE<br />Used to generate password / SSH auth<br />What do you do if you lose the key pair?<br />Storing in AMI? Encrypt with key passed in via user-data<br />
  61. 61. User Data<br />Install-lamp<br />#!/bin/bash<br />set -e -x<br />export DEBIAN_FRONTEND=noninteractive<br />apt-get update && apt-get upgrade -y<br />tasksel install lamp-server<br />echo "Please remember to set the MySQL root password!”<br />ec2-run-instances --key KEYPAIR --user-data-file install-lamp ami-bf5eb9d6<br />
  62. 62. AMI<br />Amazon Machine Image<br />Official base snapshots or private<br />AMI images allow you to spin up multiple machines. Great when combined with Spot Pricing.<br />Before creating AMI for Windows:<br />Disable SysPrep (BundleConfig.xml)<br />Disable machine renaming (Config.xml)<br />
  63. 63.
  64. 64.
  65. 65.
  66. 66.
  67. 67. CloudWatch Alerts<br />
  68. 68. Auto-scaling<br />Spin up / down nodes<br />
  69. 69.
  70. 70. Starting machines via cmd<br />
  71. 71. ec2-describe-images -o amazon<br />IMAGE ami-23b6534a ec2-public-images/fedora-core4-apache.manifest.xml<br />IMAGE ami-25b6534c ec2-public-images/fedora-core4-apache-mysql.manifest.xml<br />IMAGE ami-26b6534f ec2-public-images/developer-image.manifest.xml<br />IMAGE ami-2bb65342 ec2-public-images/getting-started.manifest.xml<br />ec2-run-instances ami-23b6534a -k windows<br />RESERVATION r-xxxxxxxxxxxxxxxxxxxx default<br />INSTANCE i-xxxxxxxx ami-23b6534a pending windows<br />ec2-describe-instances<br />RESERVATION r-xxxxxxxxxxxxxxxxxxxx default<br />INSTANCE i-xxxxxxxx ami-23b6534a ec2-xx-xxx-xx-xx.compute-1.amazonaws.com<br />ec2-terminate-instances i-xxxxxxxx<br />
  72. 72. Via .NET SDK<br />
  73. 73. public void LaunchEc2ImageWithSqlServerInstalled()<br />{<br />RunningInstancetargetMachineToStart = null;<br /> string ami = "ami-e0916389";<br />varrunInstancesRequest = new RunInstancesRequest();<br />runInstancesRequest.ImageId = ami;<br />runInstancesRequest.MinCount = 1;<br />runInstancesRequest.MaxCount = 1;<br />runInstancesRequest.KeyName = "aws";<br />runInstancesRequest.InstanceType = "t1.micro";<br />runInstancesRequest.Placement = new Placement { AvailabilityZone = "us-east-1b" };<br />var response = _ec2.RunInstances(runInstancesRequest);<br />foreach (var instance in response.RunInstancesResult.Reservation.RunningInstance)<br /> { <br /> if(instance.InstanceState.Name == "pending") //NEED TO WAIT FOR "Running”<br />targetMachineToStart = WaitUntilInstanceStateIs(instance.InstanceId, "running");<br /> }<br />}<br />
  74. 74. DescribeInstancesRequest request = new DescribeInstancesRequest();<br />varresponse = _ec2.DescribeInstances(request);<br />varcount = response.DescribeInstancesResult.Reservation.Count;<br />foreach(var reservation in response.DescribeInstancesResult.Reservation)<br />{<br />var instance = reservation.RunningInstance[0];<br />}<br />
  75. 75. public string GetPassword(string instanceId)<br />{<br /> string rsaPrivateKey;<br /> using (StreamReader reader = new StreamReader(pathToPem))<br /> {<br />rsaPrivateKey = reader.ReadToEnd();<br /> }<br />GetPasswordDataResult result = _ec2.GetPasswordData(new GetPasswordDataRequest()<br /> .WithInstanceId(instanceId)).GetPasswordDataResult;<br />var pass = result.GetDecryptedPassword(rsaPrivateKey);<br />Console.WriteLine(pass);<br /> return pass;<br />}<br />
  76. 76. WinRM<br />string path = “C:WindowsSystem32winrs.exe”<br />string args = string.Format("-r:{0} -u:Administrator -p:{1} "{2}"", machine.Url, GetPassword(machine.InstanceId), cmd);<br />Process.start(path, args);<br />Remotely connects to EC2 machine, executes command.<br />
  77. 77. List<String> commands = new List<string>();<br />commands.Add(@"New-Item c:tmp -type directory -ErrorActionSilentlyContinue");<br />commands.Add(@"$memcached_url = ""http://www.meerkatalyst.com/beta/agent.zip""");<br />commands.Add(@"$memcached_path = ""c:tmpmeerkatalyst_agent.zip""");<br />commands.Add(@"$zip_url = ""https://s3.amazonaws.com/meerkatalyst-us/utilities/unzip.exe""");<br />commands.Add(@"$zip_path = ""c:tmpunzip.exe""");<br />commands.Add("$client = new-object System.Net.WebClient");<br />commands.Add("$client.DownloadFile( $memcached_url, $memcached_path )");<br />commands.Add("$client.DownloadFile( $zip_url, $zip_path )");<br />commands.Add(@"invoke-expression ""C:tmpunzip.exe -o C:tmpmeerkatalyst_agent.zip -d C:tmpagent""");<br />StringBuilder builder = new StringBuilder();<br />foreach (var command in commands)<br />builder.AppendFormat("echo {0} >> script.ps1 && ", command);<br />cmd = “string.Format("{0} powershell -File script.ps1 && del script.ps1 && {1} &", builder, pathToAgent)”<br />var output = ExecuteRemoteCommand(cmd, machine);<br />//ask me for the code<br />
  78. 78. WARNINGS<br />http://www.flickr.com/photos/gagilas/2659695352/<br />
  79. 79. Temp Data<br />It will get deleted on reboots.<br />RabbitMQ + Hbase both used temp locations as default storage. <br />Reboot == lost work. <br />Takes a long time to debug.<br />
  80. 80. Sharepoint / SQL Server + Rename<br />Microsoft products generally don’t like a machine to be renamed.<br />Reboots == rename as it’s based on IP.<br />Return it off as described earlier.<br />
  81. 81. Regions, Zones.<br />They can go down.<br />
  82. 82.
  83. 83. Unofficial AMIs<br />They are configured by someone else…<br />Who knows what they have done.<br />
  84. 84.
  85. 85.
  86. 86.
  87. 87. Double prices if you want SQL Server<br />More of less anyway.<br />Large $0.48 per hour => $1.08 per hour<br />Extra Large $0.96 per hour => $1.56 per hour<br />
  88. 88.
  89. 89. Architecture & AWSApplication considerations for AWS <br />http://www.flickr.com/photos/wouterpostma/695850212/<br />
  90. 90. Configuration<br />Self discovery<br />Use DNS + Load Balancers to reduce config changes <br />Internal DNS on EC2?<br />HAProxy<br />Store in external system which can update / replicate to other nodes. <br />
  91. 91. Hybrid Cloud Providers<br />One won’t fit all<br />Optimize for use-case<br />Best of breed<br />CAN BE REALLY EXPENSIVE + COMPLEX! <br />
  92. 92. Think!<br />At some point and scale, the cloud doesn’t make sense. <br />Zynga has around 12,000 EC2 nodes<br />Once game is proven, moved to private cloud. EC2 for experiments and additional load.<br />
  93. 93. Small, isolated components<br />Design system for scale<br />Design system for failure<br />Message Queues are great! Use them.<br />Keep everything async<br />
  94. 94. User’s Browser<br />Javascript File<br /> via Lighttpd<br />ELB<br />Single EC2 Machine<br />Easily add additional nodes to ELB<br />User’s Browser<br />Javascript File<br /> via S3<br />CloudFront<br />
  95. 95. Heroku + NodeJS<br />Data Collection<br />Ruby Data Processor<br />RabbitMQ Queue<br />HBase Data Storage<br />Heroku + Rails<br />UI<br />Single EC2 Machine<br />Three separate components, designed to scale each individually when required.<br />Deployment, Scale, Cost<br />
  96. 96. Chaos Monkey<br />Kill random machines. See what breaks, fix it.<br />Amazon will do it without telling you, be prepared and plan ahead!<br />http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html<br />
  97. 97. Other AWS ServicesElasticCache, MapReduce, SQS, SNS, etc<br />http://www.flickr.com/photos/kky/704056791/<br />
  98. 98.
  99. 99.
  100. 100. Elastic Beanstalk – Easily deploy Java based applications + Stack<br />VPC – virtual network topology<br />Elastic MapReduce – Hosted Hadoop<br />CloudFormation – Similar to Chef / Puppet. Start group of machines<br />RDS – Hosted MySQL / Oracle<br />ElastiCache – Hosted Memcached<br />SNS - Simple Notification Service. Similar to a queue<br />IAM – Sharing access to AWS with a team<br />
  101. 101. Amazon Mechanical Turk<br />EC2 for Humans!<br />Crowdsourcing work<br />Grockit used it to verify content - http://mechanicalturk.typepad.com/blog/2011/10/editors-note-ari-bader-natal-is-the-chief-learning-architect-at-grockit-the-fast-growing-online-social-learning-startup-bac.html<br />“Usability Test on www.somewebsite.com #1148” - $10<br />“Click on google +1 for my website - $0.01”<br />
  102. 102. http://awsdocs.s3.amazonaws.com/toolkit-visualstudio/latest/aws-tkv-ug.pdf<br />AWS Toolkit for Visual Studio<br />http://aws.amazon.com/visualstudio/<br />
  103. 103.
  104. 104.
  105. 105.
  106. 106.
  107. 107.
  108. 108.
  109. 109.
  110. 110. Summary<br />http://www.flickr.com/photos/leon_homan/2856628778/<br />
  111. 111. Amazon Web Services<br />Fast<br />Cheap<br />Flexible<br />http://highscalability.com/<br />
  112. 112. THANK YOU!<br />@Ben_Hall<br />Ben@BenHall.me.uk<br />Blog.BenHall.me.uk<br />

×