We will review why cost optimisation at scale is challenging, and then focus on a particularly important area, right sizing. This session will explain exactly what EC2 rightsizing is, how to identify an EC2 instance that is oversized and how to calculate your potential savings. Atlassian will share a solution they use which allows them to perform EC2 rightsizing efficiently at scale. Speaker: Evan Crawford, Customer Support Manager, Amazon Web Services Featured Customer - Atlassian

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evan Crawford
Account Manager, Amazon Web Services
Mike Fuller
Principal Engineer, Atlassian
Cost Optimisation at Scale
Business

2. Recap of Last Year’s Session
• 3 Key Optimisation Levers
• Automation is Key
• Build a Lean Culture
• Cost Optimisation is Everyone’s Responsibility

3. Thank You!

4. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evan Crawford – Account Manager, Amazon Web Services
Mike Fuller – Principal Engineer, Atlassian
April 2016
Cost Optimisation at Scale – 2.0
Business

5. Cost Optimisation Metrics
1. % Instances Turned Off Daily
2. % Instances Right-­sized
3. % Always On Resources Covered by RI
4. % RI Utilization
All weighted by ECU

6. Elastic Compute Unit (ECU)
A Consistent Measure of CPU
Processing Power

7. Cost Optimisation Metrics
1. % Instances turned off daily
2. % Instances right-­sized
3. % Always On Resources Covered by RI
4. % RI Utilization
All weighted by ECU

8. Traditional Approach
“The
Business”
Finance
Dev
Team
Ops
Team
Tin App
“IT”
Customers

9. Traditional Approach: Problems
• Majority of Expense before Customer Feedback
• Separation of:
• Cost
• Revenue
• Ability to Cost Optimise
• Scale adds Complexity
• Right Sizing is Complex

10. “The
Business”
DevOps
Team
A More Modern Approach
Finance
Dev
Team
Ops
Team
Tin App
“IT”
Customers
Cloud
The
Team
Cloud
App
YOU
ARE
HERE

11. Some Problems Solved
• Majority of Expense before Customer Feedback ✔
• Separation of:
• Cost ✔
• Revenue ✔
• Ability to Cost Optimise ✘
• Scale adds Complexity ✘
• Right Sizing is Complex ✘

12. Atlassian’s approach
The
Business
DevOps
Team
Cloud
App
Customers
Cloud
Engineering

13. Problems Solved
• Majority of Expense before Customer Feedback ✔
• Separation of:
• Cost ✔
• Revenue ✔
• Ability to Cost Optimise ✔
• Scale is an Advantage ✔
• Right Sizing is Easy ✔

16. ISM402 Recap
Cloud
Engineering
Team
Tagging
Policy
Allocating
Costs
Reporting
Costs and
Usages

17. ISM402 Recap
Volume
Discounts
Elastic
Load
Reserved
Instances

18. EC2 Rightsizing

19. What is Rightsizing What is the Goal What data do we
need
How does Atlassian
do Rightsizing

20. Pay For What You Use.

21. Pay For What You Use.
Pay For What You Provision.

22. What is Rightsizing?

23. Rightsizing Part of Being Elastic

24. Cost
Usage
Time (Months)

25. Cost
Usage
Time (Months)

26. Avoided
Costs
Time (Months)
Cost
Usage

27. Why is Scale an Issue?

28. < 10 Instances 1
Operations
Team
Scale

29. 100+ Instances 1 – 10
Operations
Teams
Scale

30. 1000s / 10,000s
Instances
10s
Operations
Teams
Scale

32. Goals
Single definition of
what is
Underutilised
Remove duplicated
effort
Single Report Include all EC2
Instances by Default

33. TEAM

38. What About Existing Products

39. Before We Begin
Consolidated Billing Tagging Policy

42. All Servers Included by Default

43. CloudWatch

44. CloudWatch
• CPU Utilisation
• Network bytes/packets in and out
• Instance Store read write and iops

45. Max vs Average

48. What does Atlassian define as Underutilised?
MAX CPU
< 30%
MAX Disk
i/o < 3000
IOPS

49. CloudWatch
Data kept for 14
days
Data available
almost immediately
Data is stored per
account per region
Gathering many
datapoints is difficult

50. Gathering Data from CloudWatch
EC2 IAM Role
IAM Role
IAM Role
IAM Role
STS
STS
STS
CloudWatch
CloudWatch
CloudWatch

56. AWS Cost and Usage Reports

57. AWS Cost and Usage Reports
• Instance ID
• Availability Zone
• Instance Size
• Instance Rate
• OS / Software / License
• Memory / vCPU / Network Performance
• Tags

58. Allocate your Tags

59. Enable Cost and Usage Reports

60. Manifest
JSON File containing the schema
of the report
Cost and Usage Report
Report Files
Report files zipped csv

61. Cost and Usage Report Line Example
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,

62. Cost and Usage Report Line Example
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,
Operation

63. Cost and Usage Report Line Example
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,
Account ID

64. Cost and Usage Report Line Example
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,
Availability Zone

65. Cost and Usage Report Line Example
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068
per On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,
Instance ID

66. Cost and Usage Report Line Example
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,
Instance Size

67. Cost and Usage Report Line Example
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,
Tag Values

68. File Delivery

69. Cost and Usage Report
Watch out for duplication

70. Cost and Usage Report
The schema changes!

73. AWS Usage Analytics Pipeline

74. S3 Bucket
Cost and
Usage
Reports
SNS Lambda Data Pipeline
EC2
EMR
S3 Bucket
Analytics
Results
AWS Usage Analytics Pipeline

75. How Often do we Run the Analytics?

76. Lambda Function Breakdown
Calculate the day to
Analyse
Gather Input
Schema
Cost and Usage
Report
Create a new
DataPipeline
And Activate

77. DataPipeline Breakdown
Generate an EC2
instance list
Gather CloudWatch
metrics from our
AWS Accounts
Join the Cost and
Usage data with the
CloudWatch Data
Concatenate our
analytics data into
our output location

78. Generate an EC2 instance list
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,
09876543211,us-­west-­1c,i-­ca99dca1

79. Gather CloudWatch Metrics
09876543211,us-­west-­1c,i-­ca99dca1
STS
09876543211,i-­ca99dca1,2016-­01-­04T07:00:00Z,
0.68,1.07766666667,1.36,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,42.0,130.266666667,754.0

80. Join the Cost and Usage and CW Data
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,
09876543211,i-­ca99dca1,2016-­01-­04T07:00:00Z,
0.68,1.07766666667,1.36,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,42.0,130.266666667,7
54.0

81. Join the Cost and Usage and CW Data
3rehiql2lwldcgsvzsf7oowbigz7btk563jty6qtv7e7nk66thuq,2016-­01-­04T07:00:00Z/2016-­01-­
04T08:00:00Z,,AWS,Anniversary,12345678900,2016-­01-­01T00:00:00Z,2016-­02-­
01T00:00:00Z,09876543211,Usage,2016-­01-­04T07:00:00Z,2016-­01-­
04T08:00:00Z,AmazonEC2,USW1-­BoxUsage:t2.medium,RunInstances,us-­west-­1c,i-­
6a99dca1,1.00000000,USD,0.0680000000,0.06800000,0.0680000000,0.06800000,$0.068 per
On Demand Linux t2.medium Instance Hour,,Amazon Elastic Compute Cloud,,Up to 3.3
GHz,,,,,,,,,,,,,,,,,,,,General purpose,t2.medium,,No License required,US West (N.
California),AWS Region,,,,,4 GiB,,Low to Moderate,Linux,RunInstances,,Intel Xeon
Family,NA,32-­bit or 64-­bit,Intel AVX;; Intel Turbo,Compute
Instance,,,,,,,,AmazonEC2,7T9BSUQBURKAGP2T,EBS only,,,Shared,,,,,USW1-­
BoxUsage:t2.medium,2,,,,OnDemand,,,,,,tag-­value1,,,my-­
deptment,,prod,,,username,servicename,09876543211,i-­ca99dca1,2016-­01-­04T07:00:00Z,
0.68,1.07766666667,1.36,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,42.0,130.266666667,7
54.0

83. AWS Price List API
https://pricing.us-­east-­1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.csv
"XKREJETCK4Q363EE","JRTCKXETXF","XKREJETCK4Q363EE.JRTC
KXETXF.6YS6EN2CT7","OnDemand","$1.378 per On Demand
Windows with SQL Std r3.xlarge Instance Hour","2016-­01-­
01","0","Inf","Hrs","1.3780000000","USD",,,"Compute
Instance","AmazonEC2","US East (N. Virginia)","AWS
Region","r3.xlarge","Yes","Memory optimized",“4","Intel Xeon E5-­2670
v2 (Ivy Bridge)","2.5 GHz","30.5 GiB","1 x 80 SSD","Moderate","64-­
bit",,,,,,,,"Shared",,"Windows","License
Included",,,,,,,,"BoxUsage:r3.xlarge","RunInstances:0006",,"Yes",,,,,,,,,,"S
QL Std","Intel AVX;; Intel Turbo",

84. We have the data, how do we
make the report

85. BI Tool

86. Report Layout
Search / Filter Top Offenders
Graphs
Report Content Report Item
Detailed View

87. Search / Filter

88. Top Offenders Graphs

89. Report Content

90. Report Item Detailed View

91. Report Item Detailed View
Instance Current Details

92. Report Item Detailed View
All Recommended Configurations

93. Finding the Underutilised Instances
Run in the last 7
days
Run for 24 hours+ All MAX CPU
values under 30%
All MAX Disk i/o
under 3000
In the Analytics Data we find instances that

94. Finding the right size
Location Operating System
Pre-­Installed
Software
License Model Tenancy
What must stay the same:

95. Finding the right size
Memory >= Current Cost < Current vCPU > current
usage x current
vCPU

96. Report Item Detailed View

97. Report Item Detailed View

98. Report Item Detailed View

99. Report Item Detailed View

100. Report Item Detailed View

101. What if the Instance must be this Size
• We are only looking at Disk I/O and CPU
• Business Reasons
• Warm DR nodes
• Cluster master nodes
• Other reasons we cant be aware of from the metrics

102. The Report is Great, How do
we Make it Awesome?

103. Investigated
View
Report
Open JIRA Ticket Review EC2
instance
Tag Instance
with JIRA
Ticket

104. Only Show Active Instances
Edda

106. Pay For What You Use.
Pay For What You Provision.
Pay For What You Need.

107. Where To From Here
Open Source
Contact Atlassian: awsrightsizing@atlassian.com

108. Rightsize Everything

109. Where to from here
EBS RDS DynamoDB Lambda
Add more resources to the DataPipeline

111. Tools
Partners
Implementation

112. Recap of This Year’s Session
• Know your Key Levers and Metrics
• Automation is Key
• Build a Lean Culture
• Empower Your Team. Make Them Responsible.

113. AWS Training & Certification
Intro Videos & Labs
Free videos and labs to
help you learn to work
with 30+ AWS services
– in minutes!
Training Classes
In-­person and online
courses to build
technical skills –
taught by accredited
AWS instructors
Online Labs
Practice working with
AWS services in live
environment –
Learn how related
services work
together
AWS Certification
Validate technical
skills and expertise –
identify qualified IT
talent or show you
are AWS cloud ready
Learn more: aws.amazon.com/training

114. Your Training Next Steps:
ü Visit the AWS Training & Certification pod to discuss your
training plan & AWS Summit training offer
ü Register & attend AWS instructor led training
ü Get Certified
AWS Certified? Visit the AWS Summit Certification Lounge to pick up your swag
Learn more: aws.amazon.com/training

115. Thank You! (2.0)