Lessons Learned Managing Large AWS Environments

Lessons learned
managing large
AWS Environments
Ronald Bradford
http://ronaldbradford.com
@RonaldBradford
2013.06

SCOPE
Consulting experiences with AWS
Several different clients
Largest - 500+ servers
Some 40-50+ servers
Some 2-5 servers

LAMP/RoR/RDS/Windows
EffectiveMySQL.com - Performance, Scalability & Business Continuity

ABOUT MySELF
Ronald BRADFORD

Enterprise Data Architecture
24 years with RDBMS - 13 years with MySQL
Using AWS 4+ years
Published author - 4 books
Accomplished presenter - 8 years
Work at Independent MySQL Consultant

Covering
1. Products

5. Instrumentation

2. Cost

6. Failure

3. Web Scale
4. Security


1

AWS Products &
Ecosystem


ABOUT AWS
AMAZON WEB SERVICES
30+

Many, many products and
features

EC2, S3, EBS, ELB, RDS,
EMR,VPC, CDN, SWF,
SQS, SES, SNS, IAM, ...
Mechanical Turk
Flexible Payments Service (FPS)

AWS CONSOLE

May 2013

Aug 2012


Announcements
Product Announcements
http://aws.amazon.com/about-aws/newsletters/
Pricing Changes
New instance types
New features (e.g. IOPS)
New Products (e.g. Redshift/ OpsWorks)


Announcements
Product Announcements
Examples in presentation

http://aws.amazon.com/about-aws/newsletters/
Pricing Changes
New instance types
New features (e.g. IOPS)
New Products (e.g. Redshift/ OpsWorks)


ECOSYSTEM
AWS Marketplace

Over 800

https://aws.amazon.com/marketplace/


Product growth
When I started
No RDS, In-memory Cache,
DynamoDB, Glacier
No Elastic Beanstalk, OpsWorks
No management console


2

AWS Costs


operating cost
Are you monitoring your costs?
Daily
Hourly


Operating Cost
$ ec2_cost.sh

https://github.com/ronaldbradford/aws

Operating Cost
$29,000 p.m.

$ ec2_cost.sh

https://github.com/ronaldbradford/aws

Your Money
What is AWS costing you?
Instance types/sizes
http://aws.amazon.com/ec2/instance-types

Cost options
http://aws.amazon.com/ec2/pricing


Instance Types
General-purpose
Compute-optimized
Memory-optimized
Storage-optimized
GPU


$

Instance Prices
Large Instance (m1.large)


$

Instance Prices
On Demand

$0.24

Reserved

$0.136 *

+ Annual contract ( +$ 0.043)

Spot

$0.03+ *

Can be terminated (budget)

Per hour investment


$

Instance Prices

Was $0.32 til 11/19/2012
Was $0.26 til 1/16/2013

On Demand

$0.24

Reserved

$0.136 *

+ Annual contract ( +$ 0.043)

Spot

$0.03+ *

Can be terminated (budget)

Light/Medium/Heavy utilization

Per hour investment

40% saving

up to 80+% saving


SPOT EXAMPLE
One hour (24 cents)
1 x Large - Reserved
7.5G, 4 CPUs, 850G
8 x Large - Spot

or
1 x Eight Extra Large - Spot (cc2.8xlarge)
60G, 88 CPUs, 3.4T,10Gb NIC

SPOT EXAMPLE
One hour (24 cents)
1 x Large - Reserved
7.5G, 4 CPUs, 850G
8 x Large - Spot

price has changed 3 times
in 8 months

or
1 x Eight Extra Large - Spot (cc2.8xlarge)
60G, 88 CPUs, 3.4T,10Gb NIC

SPOT HISTORY
3c to 10c Zone A
3c to 40c Zone B

2013

$ ec2-describe-spot-price-history -t m1.large -d Linux/UNIX
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE

0.030000
0.100000
0.030000
0.100000
0.030000
0.030000
0.100000
0.060000
0.030000
0.240000
0.030000
0.030000
0.045000
0.050000
0.400000
0.260000
0.030000
0.030000
0.026000

2013-05-28T17:20:41-0500
2013-05-28T17:07:02-0500
2013-05-28T16:37:51-0500
2013-05-28T16:31:03-0500
2013-05-28T16:24:48-0500
2013-05-28T16:24:48-0500
2013-05-28T16:15:03-0500
2013-05-28T16:08:34-0500
2013-05-28T16:01:59-0500
2013-05-28T15:55:12-0500
2013-05-28T15:48:32-0500
2013-05-28T15:42:07-0500
2013-05-28T15:35:47-0500
2013-05-28T15:35:47-0500
2013-05-28T15:29:15-0500
2013-05-28T15:22:47-0500
2013-05-28T15:16:01-0500
2013-05-28T15:16:01-0500
2013-05-28T15:09:30-0500

m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large

Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX

us-east-1a
us-east-1a
us-east-1a
us-east-1a
us-east-1d
us-east-1a
us-east-1a
us-east-1d
us-east-1b
us-east-1b
us-east-1b
us-east-1a
us-east-1a
us-east-1b
us-east-1b
us-east-1b
us-east-1d
us-east-1a
us-east-1a


SPOT HISTORY
2.6c to 17c (1/2 of 34c)
One AZ only

2012

$ ec2-describe-spot-price-history -t m1.large -d Linux/UNIX
0.0260
0.0260
0.0290
0.0370
0.0600
0.1700
0.1600
0.0600
0.0900
0.0260
0.0260

2012-09-27T09:45:46-0800
2012-09-27T09:45:46-0800
2012-09-27T09:38:37-0800
2012-09-27T09:38:37-0800
2012-09-27T09:31:29-0800
2012-09-27T09:31:29-0800
2012-09-27T09:24:20-0800
2012-09-27T09:17:11-0800
2012-09-27T09:17:11-0800
2012-09-27T09:09:55-0800
2012-09-27T09:09:55-0800

m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large

Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX

us-east-1b
us-east-1d
us-east-1b
us-east-1d
us-east-1b
us-east-1d
us-east-1d
us-east-1b
us-east-1d
us-east-1c
us-east-1b


Using SPOTS
Is your volume predicable?
Splitting on-demand/spot instances
Can work be done asynchronously?
i.e. can be queued
Is work restartable?


Using SPOTS
Is your volume predicable?
Splitting on-demand/spot instances
Can work be done asynchronously?
i.e. can be queued
Is work restartable?

WARNING: Not for
general workloads


Instance sizes
Evaluating the right instance size
What is your bottleneck?


Instance sizes
Evaluating the right instance size
What is your bottleneck?

Developing a tool to
recommend savings


TRUSTED ADVISOR
AWS now offers Trusted Advisor
Recommendations to save money
Improve performance
Close security problems

http://aws.amazon.com/premiumsupport/trustedadvisor/

COST SAVINGS
Other players
http://www.newvem.com/
http://www.cloudyn.com/


OTHER COST SAvings
CDN - Cloudfront
Bandwidth

http://aws.amazon.com/cloudfront/

Reduce response size (e.g. 10%)
Storage
old EBS snapshots
Remove unused instances

NEW: Announced 1/9/2103
CloudWatch Alarm Actions


3

Web Scale
(hint: no humans)


ABOUT WEB SCALE
GUI = #FAIL
CLI is necessary
Manual CLI use is slow
Automation in crucial
Parallel


AWS CLI’s
Different for EC2, ELB, RDS etc
Updated frequently (i.e. monthly)
$ git clone https://github.com/ronaldbradford/aws.git
$ cd aws/scripts
$ ./aws_cli_configure.sh


AWS CLI’s
Different for EC2, ELB, RDS etc
Updated frequently (i.e. monthly)
Simple helper
$ git clone https://github.com/ronaldbradford/aws.git
$ cd aws/scripts
$ ./aws_cli_configure.sh


RTFM

http://aws.amazon.com/archives/Amazon-EC2

Identifiers
Access Key ID
Private Access Key
X.509 Certiﬁcates (2 of)
Private

(*)

& Public

AWS Account ID
Canonical User ID
https://portal.aws.amazon.com/gp/aws/securityCredentials

CLI Examples
Launch Script
Demand/Spot or switch between
Verify SSH
Verify MySQL
Verify replication in sync
Add to ELB

CLI Examples
Audit Script
Consolidates information
Parallel operations
Unused EC2/EBS etc
Feeds reporting
ELB/EC2 usage

CLI EXAMPLES
Others
Cost Measurement
Cloning (optimizes scale-up)
Move servers between load balancers
Spot History graphing
Spot History email alerts

4

AWS Security


SECURITY
Do not give away the front door keys
Do not open all the windows


SECURITY OPTIONS
Keypairs

Learn the different beneﬁts

Security groups
Virtual Private Cloud (VPC)
Identity and Access Management (IAM)
Multi-factor authentication
http://aws.amazon.com/mfa/

SECURITY TIPS
Restrict open access to port 80/443
Jump box
Restrict IP Access
Additional authentication
Per user SSH authentication
Do not use keypair

products
Many Others (AWS Summit 2013)
Cloudaware

Dome9

Enstratius

SafeNet

AlertLogic


5

Instrumentation


Instrumentation


Instrumentation
What is important to you?


Instrumentation
All server stats


Instrumentation
All server stats
Sampling issues


Instrumentation
All server stats
Sampling issues
Deceiving averages (frequency)


REQUESTS PER SEC

5 second averages, not 1 minute sample
https://github.com/ronaldbradford/reqstat

REQUESTS PER SEC

-1,500 RPS

5 second averages, not 1 minute sample
https://github.com/ronaldbradford/reqstat

outliers


outliers
I care about these


TESTING
End to end testing critical
Network latency
ELB performance


products
AWS Cloudwatch
Datadog

AppDynamics

Boundary
CopperEgg


products
AWS Cloudwatch

What features matter?

Datadog

AppDynamics

Boundary
CopperEgg


6

Failure


FAILURE


FAILURE
Instances fail


FAILURE
Instances fail
Outages occur
AWS scheduled reboots


FAILURE
Instances fail
Outages occur
AWS scheduled reboots
Be prepared
Chaos Monkey
http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html


CONCLUSION


CONCLUSION
Cost Management (saving money)


CONCLUSION
CLI automation


CONCLUSION
CLI automation
Instrumentation (inc business metrics)


CONCLUSION
CLI automation
Distribute your application & data


CONCLUSION
CLI automation
Distribute your application & data
Disaster is inevitable


AWS for FREE
http://aws.amazon.com/free/
Free EC2 t1.micro for a year
Free RDS t1.micro for a year
S3, DynamoDB, SimpleDB, +++


Ronald Bradford
http://effectiveMySQL.com

Lessons Learned Managing Large AWS Environments

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Lessons Learned Managing Large AWS Environments

Similar to Lessons Learned Managing Large AWS Environments (20)

More from Ronald Bradford

More from Ronald Bradford (20)

Recently uploaded

Recently uploaded (20)

Lessons Learned Managing Large AWS Environments