AWS Black Belt Tips

Black Belt Tips on AWS
Dean Samuels,
Solutions Architect, Amazon Web Services
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

AWS Rapid Pace of Innovation!
+48!
E!lastic Load!
Balancing!
Auto Scaling!
Amazon VPC!
Amazon RDS!
2009!
+61!
Amazon SNS!
!AWS Identity !
& Access !
!
Management!
Amazon Route 53!
2010!
+82!
Amazon SES!
!AWS Elastic !
Beanstalk!
!AWS !
CloudFormation!
!Amazon !
ElastiCache!
!AWS Direct !
Connect!
GovCloud!
2011!
+280!
!Amazon Elastic!
Transcoder!
AWS OpsWorks!
!Amazon !
CloudHSM!
!Amazon !
AppStream!
!Amazon !
CloudTrail!
!Amazon !
WorkSpaces!
Amazon Kinesis!
2013!
+159!
AWS S!torage!
Gateway!
!Amazon !
Dynamo DB!
!Amazon !
CloudSearch!
Amazon SWF!
Amazon Glacier!
Amazon Redshift!
AWS Data !
!
Pipeline!
2012!
Since inception AWS has:!
!
• Released 927 new services and features !
• Introduced over 35 major new services!
• Announced 45 price reductions!
!
!
+24!
Amazon EBS!
Amazon!
!
CloudFront!
2008!
+270!
Amazon Cognito!
!Amazon Mobile!
Analytics!
Amazon Zocalo!
2014!
*as of July 31, 2014

Ninja Tips
• Compute and Networking
• Storage & Content Delivery
• Deployment & Management
• Security
• Big Data & App Services……maybe!

• Black Belt Tip
– Route53 & Elastic Load Balancing
• Cross-Zone Load Balancing
• Application Meet Steve Failover via DNS
Challenges
• Use of AWS is starting to grow
• Focus on end user experience
• Minimise blast radius in event of issues
• Prefers compartmentalization
• Hitting AWS account limits

• Route 53 DNS Failover
ELB & Route53

Meet Steve
• Black Belt Tip
– Route53 & Elastic Load Balancing
• Application Failover via DNS
• Ninja Tip
– VPC Peering
• Trust thy neighbour!
– VPC peering within an account
– VPC peering between accounts
Challenges
• Use of AWS is starting to grow
• Focus on end user experience
• Minimise blast radius in event of issues
• Prefers compartmentalization
• Hitting AWS account limits

VPC Peering
Steve’s Shared Services VPC
10.1.0.0/16
Steve’s Workspaces VPC
192.168.0.0/20
Steve’s Enterprise Apps VPC
172.16.0.0/16
Steve’s Web Apps VPC
10.11.0.0/16
Steve’s Proxy VPC
10.20.10.0/24
Internet
Dean’s WAF VPC
10.100.0.0/16
George’s Test/Dev VPC
10.10.0.0/16

• Black Belt Tip
– Storage Gateway File Shares
• S3 Backed NAS
– Large volume file shares, no upfront cost
– On-premise or in the AWS Cloud
This is Gwen
Challenges
• Leverages multiple storage tiers on AWS
• EBS for persistent block storage
• S3 for backups and serving web & media
• Glacier for archiving data
• But storage is starting to become costly…
even on AWS
• Favours the pay for what you use model
with S3 rather than what you provision
• Requires high performance block storage

Next Generation Storage
Corporate Data center
File Servers
AWS Cloud
Internet
or
WAN
SSL
On-Premise AWS
Storage Gateway
Cache & Upload Buffer Storage
Direct Attached or Storage Area
Network Disks
iSCSI
Cached-Volumes
Multi-Terabyte
AWS Storage
Gateway Service
“Block” Volumes
@ S3 Prices
Encrypted &
Compressed
Volume
Snapshots
“Block” Volumes
@ S3 Prices
EC2
File Servers
iSCSI
Cached-Volumes
Multi-Terabyte
CIFS/
NFS
Clients
CIFS/
NFS
EC2 Clients
Third-Party options too:
• Riverbed SteelStore
• SoftNAS
• Maginatics
EC2 AWS Cached
Storage Gateway
Cache &
Upload Buffer
EBS PIOPS

• Black Belt Tip
– Storage Gateway File Shares
• S3 Backed NAS
– Large volume file shares, no upfront cost
– On-premise or in the AWS Cloud
• Ninja Tip
– Instance Storage
• Normally ephemeral storage
– Using replication = durable storage
– EBS PIOPs, General Purpose SSDs
and Enhanced Networking
This is Gwen
Challenges
• Leverages multiple storage tiers on AWS
• EBS for persistent block storage
• S3 for backups and serving web & media
• Glacier for archiving data
• But storage is starting to become costly…
even on AWS
• Favours the pay for what you use model
with S3 rather than what you provision
• Requires high performance block storage

High Speed* & High Density*
Instance storage for durable data
Instance Storage with sync to EBS Instance Storage to Instance Storage to EBS
General Network
Traffic
EBS Optimized
MDADM
RAID 0
array
*I2 and C3 Instances:
- Multiple 10s & 100’s GB SSD-based instance storage
- Enhanced Networking = Higher PPS and lower jitter & latency
DRBD
protocol A
(asynchronous)
Up to 50,000 IOPs = 800MBs
Data Store
EC2 Instance
EBS PIOPS or GP2
SSD Backed
MDADM
RAID 0 or 1+0
array
HDD
or
SSD (100,000s
IOPS) Enhanced
Networking*

• Black Belt Tip
– AWS = Programmable Resources
• AWS Support is an API
• Use Resource Tags for management
• Centralised logging and notification
Say Hi to Felix
Challenges
• Still very manual deployment and
configuration processes of AWS resources
• Lots of human interaction
• Starting to get resource sprawl – harder to
manage
• Not everything is supported by
CloudFormation

Everything is an API
• Monitoring Your Service Limits
– Via Service API
• aws iam get-account-summary
• aws autoscaling describe-account-limits
• aws ec2 describe-account-attributes
• aws ses get-send-quota
– Via Trusted Advisor
• aws support describe-trusted-advisor-check-result --check-id eW7HH0l7J9
--language en
• Accessing Support via API
– Integrate with your own management/monitoring systems
– Automatically log tickets via CloudFormation

Resource Management with Tags
#!/usr/bin/ruby
require 'aws-sdk'
AWS.regions.sort_by(&:name).each do |region|
puts region.name
region.ec2.instances.each do |instance|
if instance.status == :stopped and instance.tags.to_h.has_key?('DevProjectA')
instance.start
puts "t#{instance.id} starting"
end
end
end
for region in $(aws ec2 describe-regions --query 'Regions[*].RegionName' --output text)
do
Ruby SDK
AWS CLI
echo ${region}
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters
"Name=instance-state-name,Values=running" "Name=tag-key, Values=Uptime, Name=tag-value,
Values=BusinessHoursOnly" --output text --region ${region} | xargs aws ec2 stop-instances --
instance-ids --region ${region} 2> /dev/null
done

Centralised Log Collection
• CloudTrail
– Get log files of API calls made on your AWS account
• CloudWatch Logs
– Store and Monitor OS & Application Log Files with Amazon CloudWatch
• Service Logs
– RDS, ELB, S3, CloudFront, EMR
• Detailed Billing Reports
– Cost Allocation For Customer Bills
All stored in S3

• Black Belt Tip
– AWS = Programmable Resources
• AWS Support is an API
• Use Resource Tags for management
• Centralised logging and notification
• Ninja Tip
– CloudFormation
• Taking it to the next level!
– Custom Resources
Say Hi to Felix
Challenges
• Still very manual deployment and
configuration processes of AWS resources
• Lots of human interaction
• Starting to get resource sprawl – harder to
manage
• Not everything is supported by
CloudFormation

CloudFormation Custom Resources
2 3
Custom Resource
Implementation
Region
SQS Queue
AWS
CloudFormation
Custom Resource
Topic
Auto scaling Group
• Add New Resources
– Including AWS resources not currently
supported by CFN
• Interact with the CloudFormation
Workflow
• Inject dynamic data into a stack
• Extend the capabilities of existing
resources
• Data management via
CloudFormation
• It’s really simple if you use
aws-cfn-resource-bridge
– Install or fork from
https://github.com/aws/aws-cfn-resource-bridge
Create
Parameter1:Value1
Parameter2:Value2
….
Parametern:Valuen
Data
Export
Data
Import
DynamoDB Datapipeline S3
1
4
5
6
Output
Parameter1:Value1
Parameter2:Value2
….
Parametern:Valuen

CloudFormation Custom Resources
1 2
Custom Resource
Implementation
4
Region
SQS Queue
AWS
CloudFormation
Custom Resource
Topic
Auto scaling Group
• Add New Resources
– Including AWS resources not currently
supported by CFN
• Interact with the CloudFormation
Workflow
• Inject dynamic data into a stack
• Extend the capabilities of existing
resources
• Data management via
CloudFormation
• It’s really simple if you use
aws-cfn-resource-bridge
– Install or fork from
https://github.com/aws/aws-cfn-resource-bridge
Delete
Parameter1:Value1
Parameter2:Value2
….
Parametern:Valuen
Data
Import
Data
Export
3
DynamoDB Datapipeline S3
5
Output
Parameter1:Value1
Parameter2:Value2
….
Parametern:Valuen
6

What’s up Alex?
• Black Belt Tip
– IAM Roles with EC2
• Don’t leave home without it!
Challenges
• Admin users with no MFA
• Users leaving credentials in software
• Users not rotating their credentials
• Users not using strong password
policies
• Finds it hard to keep track of
individual IAM identifies for users

IAM Roles for EC2 Instances
AWS Cloud
Amazon
S3
Amazon
DynamoDB
Your
Application
AWS IAM
Your
Application
Your
Application
Your
Application
Auto
Scaling
Your
Application
Auto
Scaling
Role: RW access to
objects, items and
instances
• Eliminates use of long-term credentials
• Automatic credential rotation
• Less coding – AWS SDK does all the work
• Easier and more Secure!
Amazon
EC2

What’s up Alex?
• Black Belt Tip
– IAM Roles with EC2
• Don’t leave home without it!
• Ninja Tip
– Limit number of IAM Users
• Use IAM Roles instead
– Cross-Account IAM Access
– Identity Federation
Challenges
• Admin users with no MFA
• Users leaving credentials in software
• Users not rotating their credentials
• Users not using strong password
policies
• Finds it hard to keep track of
individual IAM identifies for users

Cross-account API access
dsamuel@amazon.com
Acct ID: 111122223333
ec2-role
{
"Statement":
[
{
"Action":
[
"ec2:StartInstances",
"ec2:StopInstances"
],
"Effect":
"Allow",
"Resource":
"*"
}
]
}
squigg@amazon.com
Acct ID: 123456789012
Authenticate with
squigg access keys
Optionally also with MFA
Get temporary
security credentials
for ec2-role
Call AWS APIs
using temporary
security credentials
of ec2-role
{
"Statement":
[
{
IAM user: squigg
"Effect":
"Allow",
"Action":
"sts:AssumeRole",
"Resource":
"arn:aws:iam::111122223333:role/ec2-‐role"
}
]
}
{
"Statement":
[
{
"Effect":"Allow",
"Principal":{"AWS":"123456789012"},
"Action":"sts:AssumeRole"
}
]
}
ec2-role trusts IAM users from the AWS account
squigg@amazon.com (123456789012)
Permissions assigned to squigg granting him permission
to assume ec2-role in dsamuel@amazon.com account
Permissions assigned
to ec2-role
STS
Amazon EC2

How to Keep Up to Date
• AWS Podcast
– http://aws.amazon.com/podcasts/aws-podcast/
• Amazon Web Services Blog
– http://aws.amazon.com/blogs/aws
• What’s New from AWS
– http://aws.amazon.com/new
• Social Media
– @awscloud, /amazonwebservices, /amazonwebservices
• Your Friendly Solution Architect Team
– Speak to the team today at the SA booth

Expand your skills with AWS
Certification
Exams
Validate your proven
technical expertise with
the AWS platform
aws.amazon.com/certification
On-Demand
Resources
Videos & Labs
Get hands-on practice
working with AWS
technologies in a live
environment
aws.amazon.com/training/
self-paced-labs
Instructor-Led
Courses
Training Classes
Expand your technical
expertise to design, deploy,
and operate scalable,
efficient applications on AWS
aws.amazon.com/training

AWS Black Belt Tips

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to AWS Black Belt Tips

Similar to AWS Black Belt Tips (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Recently uploaded

Recently uploaded (20)

AWS Black Belt Tips