Uploaded on

AWS Summit 2014 Brisbane - Breakout 6 …

AWS Summit 2014 Brisbane - Breakout 6

Technical deep dive in to 10 AWS Cloud best practices with in-depth look at the tips and tricks of architecting on the AWS platform.

Presenter: Dean Samuels, Solutions Architect, Amazon Web Services

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • 'minimise blast radius in event of issues', good way to say it.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,092
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
0
Comments
1
Likes
25

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. Black Belt Tips on AWS Dean Samuels, Solutions Architect, Amazon Web Services
  • 2. AWS Rapid Pace of Innovation! 2009! Amazon RDS! Amazon VPC! Auto Scaling! Elastic Load! ! Balancing! +4 8! 2010! Amazon SNS! AWS Identity ! ! & Access ! ! Management! Amazon Route 53! +6 1! 2011! Amazon ! ! ElastiCache! Amazon SES! AWS ! ! CloudFormation! AWS Direct ! ! Connect! AWS Elastic ! ! Beanstalk! GovCloud! +82! Amazon ! ! CloudTrail! Amazon ! ! CloudHSM! Amazon ! ! WorkSpaces! Amazon Kinesis! Amazon Elastic! ! Transcoder! Amazon ! ! AppStream! AWS OpsWorks! +280! 2013! Amazon SWF! Amazon Redshift! Amazon Glacier! Amazon !! Dynamo DB! Amazon ! ! CloudSearch! AWS Storage! ! Gateway! AWS Data ! ! Pipeline! +159! 2012! Since inception AWS has:! ! •  Released 927 new services and features ! •  Introduced over 35 major new services! •  Announced 45 price reductions! ! ! 2008! +24! Amazon EBS! Amazon! ! CloudFront! +270! 2014! Amazon Cognito! Amazon Zocalo! Amazon Mobile! ! Analytics! *as of July 31, 2014
  • 3. Ninja Tips •  Compute and Networking •  Storage & Content Delivery •  Deployment & Management •  Security •  Big Data & App Services……maybe!
  • 4. •  Black Belt Tip –  Route53 & Elastic Load Balancing •  Cross-Zone Load Balancing •  Application Failover via DNSMeet Steve Challenges •  Use of AWS is starting to grow •  Focus on end user experience •  Minimise blast radius in event of issues •  Prefers compartmentalization •  Hitting AWS account limits
  • 5. •  Route 53 DNS Failover ELB & Route53 •  Cross-Zone Load Balancing
  • 6. Meet Steve •  Black Belt Tip –  Route53 & Elastic Load Balancing •  Cross-Zone Load Balancing •  Application Failover via DNS •  Ninja Tip –  VPC Peering •  Trust thy neighbour! –  VPC peering within an account –  VPC peering between accounts Challenges •  Use of AWS is starting to grow •  Focus on end user experience •  Minimise blast radius in event of issues •  Prefers compartmentalization •  Hitting AWS account limits
  • 7. VPC Peering Steve’s Shared Services VPC 10.1.0.0/16 Steve’s Workspaces VPC 192.168.0.0/20 Steve’s Enterprise Apps VPC 172.16.0.0/16 Steve’s Web Apps VPC 10.11.0.0/16 Steve’s Proxy VPC 10.20.10.0/24 Internet Dean’s WAF VPC 10.100.0.0/16 George’s Test/Dev VPC 10.10.0.0/16
  • 8. •  Black Belt Tip –  Storage Gateway File Shares •  S3 Backed NAS –  Large volume file shares, no upfront cost –  On-premise or in the AWS Cloud This is Gwen Challenges •  Leverages multiple storage tiers on AWS •  EBS for persistent block storage •  S3 for backups and serving web & media •  Glacier for archiving data •  But storage is starting to become costly… even on AWS •  Favours the pay for what you use model with S3 rather than what you provision •  Requires high performance block storage
  • 9. Next Generation Storage File Servers Corporate Data center AWS Cloud Internet or WAN SSL On-Premise AWS Storage Gateway Cache & Upload Buffer Storage Direct Attached or Storage Area Network Disks iSCSI Cached-Volumes Multi-Terabyte AWS Storage Gateway Service “Block” Volumes @ S3 Prices “Block” Volumes @ S3 Prices Encrypted & Compressed Volume Snapshots EC2 File Servers iSCSI Cached-Volumes Multi-Terabyte CIFS/ NFS Clients CIFS/ NFS EC2 Clients Third-Party options too: •  Riverbed SteelStore •  SoftNAS •  Maginatics EC2 AWS Cached Storage Gateway Cache & Upload Buffer EBS PIOPS
  • 10. •  Black Belt Tip –  Storage Gateway File Shares •  S3 Backed NAS –  Large volume file shares, no upfront cost –  On-premise or in the AWS Cloud •  Ninja Tip – Instance Storage •  Normally ephemeral storage –  Using replication = durable storage –  EBS PIOPs, General Purpose SSDs and Enhanced Networking This is Gwen Challenges •  Leverages multiple storage tiers on AWS •  EBS for persistent block storage •  S3 for backups and serving web & media •  Glacier for archiving data •  But storage is starting to become costly… even on AWS •  Favours the pay for what you use model with S3 rather than what you provision •  Requires high performance block storage
  • 11. High Speed* & High Density* Instance storage for durable data Instance Storage with sync to EBS Instance Storage to Instance Storage to EBS *I2 and C3 Instances: - Multiple 10s & 100’s GB SSD-based instance storage - Enhanced Networking = Higher PPS and lower jitter & latency EBS Optimized MDADM RAID 0 array DRBD protocol A (asynchronous) Up to 50,000 IOPs = 800MBs General Network Traffic EBS PIOPS or GP2 SSD Backed Data Store EC2 Instance MDADM RAID 0 or 1+0 array HDD or SSD (100,000s IOPS) Enhanced Networking*
  • 12. •  Black Belt Tip –  AWS = Programmable Resources •  AWS Support is an API •  Use Resource Tags for management •  Centralised logging and notification Say Hi to Felix Challenges •  Still very manual deployment and configuration processes of AWS resources •  Lots of human interaction •  Starting to get resource sprawl – harder to manage •  Not everything is supported by CloudFormation
  • 13. Everything is an API •  Monitoring Your Service Limits –  Via Service API •  aws iam get-account-summary •  aws autoscaling describe-account-limits •  aws ec2 describe-account-attributes •  aws ses get-send-quota –  Via Trusted Advisor •  aws support describe-trusted-advisor-check-result --check-id eW7HH0l7J9 --language en •  Accessing Support via API –  Integrate with your own management/monitoring systems –  Automatically log tickets via CloudFormation
  • 14. Resource Management with Tags #!/usr/bin/ruby require 'aws-sdk' AWS.regions.sort_by(&:name).each do |region|   puts region.name   region.ec2.instances.each do |instance|     if instance.status == :stopped and instance.tags.to_h.has_key?('DevProjectA')       instance.start puts "t#{instance.id} starting"     end   end end for region in $(aws ec2 describe-regions --query 'Regions[*].RegionName' --output text) do echo ${region} aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters "Name=instance-state-name,Values=running" "Name=tag-key, Values=Uptime, Name=tag-value, Values=BusinessHoursOnly" --output text --region ${region} | xargs aws ec2 stop-instances -- instance-ids --region ${region} 2> /dev/null done Ruby SDK AWS CLI
  • 15. Centralised Log Collection •  CloudTrail –  Get log files of API calls made on your AWS account •  CloudWatch Logs –  Store and Monitor OS & Application Log Files with Amazon CloudWatch •  Service Logs –  RDS, ELB, S3, CloudFront, EMR •  Detailed Billing Reports –  Cost Allocation For Customer Bills All stored in S3
  • 16. •  Black Belt Tip –  AWS = Programmable Resources •  AWS Support is an API •  Use Resource Tags for management •  Centralised logging and notification •  Ninja Tip –  CloudFormation •  Taking it to the next level! –  Custom Resources Say Hi to Felix Challenges •  Still very manual deployment and configuration processes of AWS resources •  Lots of human interaction •  Starting to get resource sprawl – harder to manage •  Not everything is supported by CloudFormation
  • 17. CloudFormation Custom Resources Region SQS Queue AWS CloudFormation Custom Resource Topic Auto scaling Group Custom Resource Implementation •  Add New Resources –  Including AWS resources not currently supported by CFN •  Interact with the CloudFormation Workflow •  Inject dynamic data into a stack •  Extend the capabilities of existing resources •  Data management via CloudFormation •  It’s really simple if you use aws-cfn-resource-bridge –  Install or fork from https://github.com/aws/aws-cfn-resource-bridge Create Parameter1:Value1 Parameter2:Value2 …. Parametern:Valuen Data Export Data Import DynamoDB S3Datapipeline 1 2 3 4 5 6 Output Parameter1:Value1 Parameter2:Value2 …. Parametern:Valuen
  • 18. CloudFormation Custom Resources Region SQS Queue AWS CloudFormation Custom Resource Topic Auto scaling Group Custom Resource Implementation •  Add New Resources –  Including AWS resources not currently supported by CFN •  Interact with the CloudFormation Workflow •  Inject dynamic data into a stack •  Extend the capabilities of existing resources •  Data management via CloudFormation •  It’s really simple if you use aws-cfn-resource-bridge –  Install or fork from https://github.com/aws/aws-cfn-resource-bridge Delete Parameter1:Value1 Parameter2:Value2 …. Parametern:Valuen Data Import Data Export DynamoDB S3Datapipeline 1 2 3 4 5 Output Parameter1:Value1 Parameter2:Value2 …. Parametern:Valuen 6
  • 19. What’s up Alex? •  Black Belt Tip – IAM Roles with EC2 •  Don’t leave home without it! Challenges •  Admin users with no MFA •  Users leaving credentials in software •  Users not rotating their credentials •  Users not using strong password policies •  Finds it hard to keep track of individual IAM identifies for users
  • 20. IAM Roles for EC2 Instances AWS Cloud Amazon S3 Amazon DynamoDB Your Application AWS IAM Your Application Your Application Your Application Auto Scaling Your Application Auto Scaling Role: RW access to objects, items and instances •  Eliminates use of long-term credentials •  Automatic credential rotation •  Less coding – AWS SDK does all the work •  Easier and more Secure! Amazon EC2
  • 21. What’s up Alex? •  Black Belt Tip – IAM Roles with EC2 •  Don’t leave home without it! •  Ninja Tip – Limit number of IAM Users •  Use IAM Roles instead –  Cross-Account IAM Access –  Identity Federation Challenges •  Admin users with no MFA •  Users leaving credentials in software •  Users not rotating their credentials •  Users not using strong password policies •  Finds it hard to keep track of individual IAM identifies for users
  • 22. dsamuel@amazon.com Acct ID: 111122223333 ec2-role {  "Statement":  [      {          "Action":  [              "ec2:StartInstances",              "ec2:StopInstances"          ],          "Effect":  "Allow",          "Resource":  "*"      }   ]  }   squigg@amazon.com Acct ID: 123456789012 Authenticate with squigg access keys Optionally also with MFA Get temporary security credentials for ec2-role Call AWS APIs using temporary security credentials of ec2-role {  "Statement":  [      {        "Effect":  "Allow",        "Action":  "sts:AssumeRole",        "Resource":      "arn:aws:iam::111122223333:role/ec2-­‐role"      }   ]  }   {  "Statement":  [      {        "Effect":"Allow",        "Principal":{"AWS":"123456789012"},        "Action":"sts:AssumeRole"      }   ]  }   Cross-account API access ec2-role trusts IAM users from the AWS account squigg@amazon.com (123456789012) Permissions assigned to squigg granting him permission to assume ec2-role in dsamuel@amazon.com account IAM user: squigg Permissions assigned to ec2-role STS Amazon EC2
  • 23. How to Keep Up to Date •  AWS Podcast –  http://aws.amazon.com/podcasts/aws-podcast/ •  Amazon Web Services Blog –  http://aws.amazon.com/blogs/aws •  What’s New from AWS –  http://aws.amazon.com/new •  Social Media –  @awscloud, /amazonwebservices, /amazonwebservices •  Your Friendly Solution Architect Team –  Speak to the team today at the SA booth
  • 24. Expand your skills with AWS Certification aws.amazon.com/certification Exams Validate your proven technical expertise with the AWS platform On-Demand Resources aws.amazon.com/training/ self-paced-labs Videos & Labs Get hands-on practice working with AWS technologies in a live environment aws.amazon.com/training Instructor-Led Courses Training Classes Expand your technical expertise to design, deploy, and operate scalable, efficient applications on AWS