Black Belt Tips on AWS 
Dean Samuels, 
Solutions Architect, Amazon Web Services 
© 2014 Amazon.com, Inc. and its affiliate...
AWS Rapid Pace of Innovation! 
+48! 
E!lastic Load! 
Balancing! 
Auto Scaling! 
Amazon VPC! 
Amazon RDS! 
2009! 
+61! 
Ama...
Ninja Tips 
• Compute and Networking 
• Storage & Content Delivery 
• Deployment & Management 
• Security 
• Big Data & Ap...
• Black Belt Tip 
– Route53 & Elastic Load Balancing 
• Cross-Zone Load Balancing 
• Application Meet Steve Failover via D...
• Route 53 DNS Failover 
ELB & Route53 
• Cross-Zone Load Balancing
Meet Steve 
• Black Belt Tip 
– Route53 & Elastic Load Balancing 
• Cross-Zone Load Balancing 
• Application Failover via ...
VPC Peering 
Steve’s Shared Services VPC 
10.1.0.0/16 
Steve’s Workspaces VPC 
192.168.0.0/20 
Steve’s Enterprise Apps VPC...
• Black Belt Tip 
– Storage Gateway File Shares 
• S3 Backed NAS 
– Large volume file shares, no upfront cost 
– On-premis...
Next Generation Storage 
Corporate Data center 
File Servers 
AWS Cloud 
Internet 
or 
WAN 
SSL 
On-Premise AWS 
Storage G...
• Black Belt Tip 
– Storage Gateway File Shares 
• S3 Backed NAS 
– Large volume file shares, no upfront cost 
– On-premis...
High Speed* & High Density* 
Instance storage for durable data 
Instance Storage with sync to EBS Instance Storage to Inst...
• Black Belt Tip 
– AWS = Programmable Resources 
• AWS Support is an API 
• Use Resource Tags for management 
• Centralis...
Everything is an API 
• Monitoring Your Service Limits 
– Via Service API 
• aws iam get-account-summary 
• aws autoscalin...
Resource Management with Tags 
#!/usr/bin/ruby 
require 'aws-sdk' 
AWS.regions.sort_by(&:name).each do |region| 
puts regi...
Centralised Log Collection 
• CloudTrail 
– Get log files of API calls made on your AWS account 
• CloudWatch Logs 
– Stor...
• Black Belt Tip 
– AWS = Programmable Resources 
• AWS Support is an API 
• Use Resource Tags for management 
• Centralis...
CloudFormation Custom Resources 
2 3 
Custom Resource 
Implementation 
Region 
SQS Queue 
AWS 
CloudFormation 
Custom Reso...
CloudFormation Custom Resources 
1 2 
Custom Resource 
Implementation 
4 
Region 
SQS Queue 
AWS 
CloudFormation 
Custom R...
What’s up Alex? 
• Black Belt Tip 
– IAM Roles with EC2 
• Don’t leave home without it! 
Challenges 
• Admin users with no...
IAM Roles for EC2 Instances 
AWS Cloud 
Amazon 
S3 
Amazon 
DynamoDB 
Your 
Application 
AWS IAM 
Your 
Application 
Your ...
What’s up Alex? 
• Black Belt Tip 
– IAM Roles with EC2 
• Don’t leave home without it! 
• Ninja Tip 
– Limit number of IA...
Cross-account API access 
dsamuel@amazon.com 
Acct ID: 111122223333 
ec2-role 
{""Statement":"[" 
""{" 
"""""Action":"[" 
...
How to Keep Up to Date 
• AWS Podcast 
– http://aws.amazon.com/podcasts/aws-podcast/ 
• Amazon Web Services Blog 
– http:/...
Expand your skills with AWS 
Certification 
Exams 
Validate your proven 
technical expertise with 
the AWS platform 
aws.a...
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or i...
Upcoming SlideShare
Loading in...5
×

AWS Black Belt Tips

590

Published on

AWS Summit 2014 Perth - Breakout 3

Technical deep dive in to 10 AWS Cloud best practices with in-depth look at the tips and tricks of architecting on the AWS platform.

Presenter: Dean Samuels, Solutions Architect, Amazon Web Services

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
590
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

AWS Black Belt Tips

  1. 1. Black Belt Tips on AWS Dean Samuels, Solutions Architect, Amazon Web Services © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. AWS Rapid Pace of Innovation! +48! E!lastic Load! Balancing! Auto Scaling! Amazon VPC! Amazon RDS! 2009! +61! Amazon SNS! !AWS Identity ! & Access ! ! Management! Amazon Route 53! 2010! +82! Amazon SES! !AWS Elastic ! Beanstalk! !AWS ! CloudFormation! !Amazon ! ElastiCache! !AWS Direct ! Connect! GovCloud! 2011! +280! !Amazon Elastic! Transcoder! AWS OpsWorks! !Amazon ! CloudHSM! !Amazon ! AppStream! !Amazon ! CloudTrail! !Amazon ! WorkSpaces! Amazon Kinesis! 2013! +159! AWS S!torage! Gateway! !Amazon ! Dynamo DB! !Amazon ! CloudSearch! Amazon SWF! Amazon Glacier! Amazon Redshift! AWS Data ! ! Pipeline! 2012! Since inception AWS has:! ! • Released 927 new services and features ! • Introduced over 35 major new services! • Announced 45 price reductions! ! ! +24! Amazon EBS! Amazon! ! CloudFront! 2008! +270! Amazon Cognito! !Amazon Mobile! Analytics! Amazon Zocalo! 2014! *as of July 31, 2014
  3. 3. Ninja Tips • Compute and Networking • Storage & Content Delivery • Deployment & Management • Security • Big Data & App Services……maybe!
  4. 4. • Black Belt Tip – Route53 & Elastic Load Balancing • Cross-Zone Load Balancing • Application Meet Steve Failover via DNS Challenges • Use of AWS is starting to grow • Focus on end user experience • Minimise blast radius in event of issues • Prefers compartmentalization • Hitting AWS account limits
  5. 5. • Route 53 DNS Failover ELB & Route53 • Cross-Zone Load Balancing
  6. 6. Meet Steve • Black Belt Tip – Route53 & Elastic Load Balancing • Cross-Zone Load Balancing • Application Failover via DNS • Ninja Tip – VPC Peering • Trust thy neighbour! – VPC peering within an account – VPC peering between accounts Challenges • Use of AWS is starting to grow • Focus on end user experience • Minimise blast radius in event of issues • Prefers compartmentalization • Hitting AWS account limits
  7. 7. VPC Peering Steve’s Shared Services VPC 10.1.0.0/16 Steve’s Workspaces VPC 192.168.0.0/20 Steve’s Enterprise Apps VPC 172.16.0.0/16 Steve’s Web Apps VPC 10.11.0.0/16 Steve’s Proxy VPC 10.20.10.0/24 Internet Dean’s WAF VPC 10.100.0.0/16 George’s Test/Dev VPC 10.10.0.0/16
  8. 8. • Black Belt Tip – Storage Gateway File Shares • S3 Backed NAS – Large volume file shares, no upfront cost – On-premise or in the AWS Cloud This is Gwen Challenges • Leverages multiple storage tiers on AWS • EBS for persistent block storage • S3 for backups and serving web & media • Glacier for archiving data • But storage is starting to become costly… even on AWS • Favours the pay for what you use model with S3 rather than what you provision • Requires high performance block storage
  9. 9. Next Generation Storage Corporate Data center File Servers AWS Cloud Internet or WAN SSL On-Premise AWS Storage Gateway Cache & Upload Buffer Storage Direct Attached or Storage Area Network Disks iSCSI Cached-Volumes Multi-Terabyte AWS Storage Gateway Service “Block” Volumes @ S3 Prices Encrypted & Compressed Volume Snapshots “Block” Volumes @ S3 Prices EC2 File Servers iSCSI Cached-Volumes Multi-Terabyte CIFS/ NFS Clients CIFS/ NFS EC2 Clients Third-Party options too: • Riverbed SteelStore • SoftNAS • Maginatics EC2 AWS Cached Storage Gateway Cache & Upload Buffer EBS PIOPS
  10. 10. • Black Belt Tip – Storage Gateway File Shares • S3 Backed NAS – Large volume file shares, no upfront cost – On-premise or in the AWS Cloud • Ninja Tip – Instance Storage • Normally ephemeral storage – Using replication = durable storage – EBS PIOPs, General Purpose SSDs and Enhanced Networking This is Gwen Challenges • Leverages multiple storage tiers on AWS • EBS for persistent block storage • S3 for backups and serving web & media • Glacier for archiving data • But storage is starting to become costly… even on AWS • Favours the pay for what you use model with S3 rather than what you provision • Requires high performance block storage
  11. 11. High Speed* & High Density* Instance storage for durable data Instance Storage with sync to EBS Instance Storage to Instance Storage to EBS General Network Traffic EBS Optimized MDADM RAID 0 array *I2 and C3 Instances: - Multiple 10s & 100’s GB SSD-based instance storage - Enhanced Networking = Higher PPS and lower jitter & latency DRBD protocol A (asynchronous) Up to 50,000 IOPs = 800MBs Data Store EC2 Instance EBS PIOPS or GP2 SSD Backed MDADM RAID 0 or 1+0 array HDD or SSD (100,000s IOPS) Enhanced Networking*
  12. 12. • Black Belt Tip – AWS = Programmable Resources • AWS Support is an API • Use Resource Tags for management • Centralised logging and notification Say Hi to Felix Challenges • Still very manual deployment and configuration processes of AWS resources • Lots of human interaction • Starting to get resource sprawl – harder to manage • Not everything is supported by CloudFormation
  13. 13. Everything is an API • Monitoring Your Service Limits – Via Service API • aws iam get-account-summary • aws autoscaling describe-account-limits • aws ec2 describe-account-attributes • aws ses get-send-quota – Via Trusted Advisor • aws support describe-trusted-advisor-check-result --check-id eW7HH0l7J9 --language en • Accessing Support via API – Integrate with your own management/monitoring systems – Automatically log tickets via CloudFormation
  14. 14. Resource Management with Tags #!/usr/bin/ruby require 'aws-sdk' AWS.regions.sort_by(&:name).each do |region| puts region.name region.ec2.instances.each do |instance| if instance.status == :stopped and instance.tags.to_h.has_key?('DevProjectA') instance.start puts "t#{instance.id} starting" end end end for region in $(aws ec2 describe-regions --query 'Regions[*].RegionName' --output text) do Ruby SDK AWS CLI echo ${region} aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters "Name=instance-state-name,Values=running" "Name=tag-key, Values=Uptime, Name=tag-value, Values=BusinessHoursOnly" --output text --region ${region} | xargs aws ec2 stop-instances -- instance-ids --region ${region} 2> /dev/null done
  15. 15. Centralised Log Collection • CloudTrail – Get log files of API calls made on your AWS account • CloudWatch Logs – Store and Monitor OS & Application Log Files with Amazon CloudWatch • Service Logs – RDS, ELB, S3, CloudFront, EMR • Detailed Billing Reports – Cost Allocation For Customer Bills All stored in S3
  16. 16. • Black Belt Tip – AWS = Programmable Resources • AWS Support is an API • Use Resource Tags for management • Centralised logging and notification • Ninja Tip – CloudFormation • Taking it to the next level! – Custom Resources Say Hi to Felix Challenges • Still very manual deployment and configuration processes of AWS resources • Lots of human interaction • Starting to get resource sprawl – harder to manage • Not everything is supported by CloudFormation
  17. 17. CloudFormation Custom Resources 2 3 Custom Resource Implementation Region SQS Queue AWS CloudFormation Custom Resource Topic Auto scaling Group • Add New Resources – Including AWS resources not currently supported by CFN • Interact with the CloudFormation Workflow • Inject dynamic data into a stack • Extend the capabilities of existing resources • Data management via CloudFormation • It’s really simple if you use aws-cfn-resource-bridge – Install or fork from https://github.com/aws/aws-cfn-resource-bridge Create Parameter1:Value1 Parameter2:Value2 …. Parametern:Valuen Data Export Data Import DynamoDB Datapipeline S3 1 4 5 6 Output Parameter1:Value1 Parameter2:Value2 …. Parametern:Valuen
  18. 18. CloudFormation Custom Resources 1 2 Custom Resource Implementation 4 Region SQS Queue AWS CloudFormation Custom Resource Topic Auto scaling Group • Add New Resources – Including AWS resources not currently supported by CFN • Interact with the CloudFormation Workflow • Inject dynamic data into a stack • Extend the capabilities of existing resources • Data management via CloudFormation • It’s really simple if you use aws-cfn-resource-bridge – Install or fork from https://github.com/aws/aws-cfn-resource-bridge Delete Parameter1:Value1 Parameter2:Value2 …. Parametern:Valuen Data Import Data Export 3 DynamoDB Datapipeline S3 5 Output Parameter1:Value1 Parameter2:Value2 …. Parametern:Valuen 6
  19. 19. What’s up Alex? • Black Belt Tip – IAM Roles with EC2 • Don’t leave home without it! Challenges • Admin users with no MFA • Users leaving credentials in software • Users not rotating their credentials • Users not using strong password policies • Finds it hard to keep track of individual IAM identifies for users
  20. 20. IAM Roles for EC2 Instances AWS Cloud Amazon S3 Amazon DynamoDB Your Application AWS IAM Your Application Your Application Your Application Auto Scaling Your Application Auto Scaling Role: RW access to objects, items and instances • Eliminates use of long-term credentials • Automatic credential rotation • Less coding – AWS SDK does all the work • Easier and more Secure! Amazon EC2
  21. 21. What’s up Alex? • Black Belt Tip – IAM Roles with EC2 • Don’t leave home without it! • Ninja Tip – Limit number of IAM Users • Use IAM Roles instead – Cross-Account IAM Access – Identity Federation Challenges • Admin users with no MFA • Users leaving credentials in software • Users not rotating their credentials • Users not using strong password policies • Finds it hard to keep track of individual IAM identifies for users
  22. 22. Cross-account API access dsamuel@amazon.com Acct ID: 111122223333 ec2-role {""Statement":"[" ""{" """""Action":"[" """""""ec2:StartInstances"," """""""ec2:StopInstances"" """"]," """""Effect":""Allow"," """""Resource":""*"" ""}" ]"}" squigg@amazon.com Acct ID: 123456789012 Authenticate with squigg access keys Optionally also with MFA Get temporary security credentials for ec2-role Call AWS APIs using temporary security credentials of ec2-role IAM user: squigg {""Statement":"[" ""{" """"Effect":""Allow"," """"Action":""sts:AssumeRole"," """"Resource":"" ""arn:aws:iam::111122223333:role/ec2Brole"" ""}" ]"}" {""Statement":"[" ""{" """"Effect":"Allow"," """"Principal":{"AWS":"123456789012"}," """"Action":"sts:AssumeRole"" ""}" ]"}" ec2-role trusts IAM users from the AWS account squigg@amazon.com (123456789012) Permissions assigned to squigg granting him permission to assume ec2-role in dsamuel@amazon.com account Permissions assigned to ec2-role STS Amazon EC2
  23. 23. How to Keep Up to Date • AWS Podcast – http://aws.amazon.com/podcasts/aws-podcast/ • Amazon Web Services Blog – http://aws.amazon.com/blogs/aws • What’s New from AWS – http://aws.amazon.com/new • Social Media – @awscloud, /amazonwebservices, /amazonwebservices • Your Friendly Solution Architect Team – Speak to the team today at the SA booth
  24. 24. Expand your skills with AWS Certification Exams Validate your proven technical expertise with the AWS platform aws.amazon.com/certification On-Demand Resources Videos & Labs Get hands-on practice working with AWS technologies in a live environment aws.amazon.com/training/ self-paced-labs Instructor-Led Courses Training Classes Expand your technical expertise to design, deploy, and operate scalable, efficient applications on AWS aws.amazon.com/training
  25. 25. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

×