AWS Melbourne Cost Mgt. and Opti. Meetup - 20181109 - v2.2

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Cost Management &
Optimisation Interest Group
Melbourne Usergroup Meetup
Nov 2018

Agenda
• 2:00pm - Setup
• 2:10pm - Kick off, welcome, and intro
• 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your
AWS usage
• 2:50pm - Discussion and Q&A
• 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that
provides high speed to insight
• 4:30pm - Discussion, Q&A, and networking over drinks and snacks
• 5:30pm - Event Concludes

Learning &
Accelerating
Asking
questions
Sharing stories
Voice to target
product asks
Contribute &
open source
Respect NDAs Taking off the
sales hat
Fully realise
the benefits
of AWS

Why should we care about Cost Optimisation?
Example non-prod workload checklist
Can this run on CentOS/Linux?
$1000
Turn off outside of work hours?
Right size down by 1 size? $118
$236
$787
Can this run on EC2 Spot? $30
Starting non-prod workload

Cloud Cost Optimization
HOW TO GET THE MOST OUT OF YOUR AWS USAGE

History
• About Post
• About Me
• My Journey with Cloud Services
• Cost Optimisation principles
• Where to from here?

Beginning the journey
Communicate
• Constant two-way communication between business, IT and the billing
team is vital
Educate
• An understanding of the intricacies of AWS billing
Empower
• Give control to users of the platform to manage their own costs

Cost Allocation
ASSIGNING COSTS TO THOSE RESPONSIBLE

• Tagging (keys and values)
• Policy/standards
• Tagging (mapping)
• Transition
• Data sources
• Did we mention tagging?

Cost Avoidance
BETTER USAGE OF YOUR PRODUCTS

• Cost control
• Know your products
• Usage
• Multi vendor

Cost Accountability
THE TOOLS AND WHO SHOULD USE THEM

• Spot instances
• Rightsizing
• Reserved instances
• Billing tools

Cost Transparency
COMMUNICATION

• Showback and Chargeback
• Monitoring and reporting
• Analysis and trending
• Collaboration and communication

Where to from here?
• Automated tagging
• Product/team account based strategy
• Review of services consumed
• Tagging (integration and central management)

Contents
• Why should I build my own dashboard?
• AWS Data Sources
• Data Pipelines into Athena
• Gaining Speed to Insight in Quicksight
(incl. visualization tips and KPIs)

Speed, scale, complexity, and value at stake drives
the need for visibility and speed to insight

Pick the tool that provides the cost visibility and
speed to insight that you need
Simple, Static, Small
environment
Complex, Dynamic,
Large environment
1. Monthly AWS Invoice
2. AWS Billing
console
3. AWS Cost Explorer
and AWS Budgets
4. AWS Billing File Analysis,
DIY dashboards, and
3rd party tools

100% Agility
Engineering bias
100% Control
Finance bias
Balanced approach
Speed to Insight will help you take a balanced
approach

AWS Data Sources
• AWS Cost and Usage Report (CUR)
• Hourly billing data for each service
+ more info such as RI usage
• Can have a very large number of rows and columns
• AWS CloudWatch data
• Resource Utilization data
• DIY budget and revenue data
• Flat .csv file of how much you’ve budgeted to spend +
revenue generated associated with AWS spend
• AWS CloudTrail

Step 1.1: Generate the AWS Cost and Usage
Report (CUR) for your account @ Payer acct. level
• CUR is the data source of
Cost Explorer
• Enable the CUR (5 minute
exercise)
• https://docs.aws.amazon.com/
awsaccountbilling/latest/about
v2/billing-reports-
gettingstarted-
turnonreports.html

Step 1.2: Create some DIY budget and revenue data
• Create CSV file with 4 columns:
• Account id
• Month
• Budget (example business constraint)
• Revenue (example business metric)
Other examples:
• minutes on website
• number of devs

Step 1.3: Save CloudWatch data across relevant
accounts to S3
• Amazon CloudWatch is a monitoring and management
service that collects and reports resource metrics
• Metrics that indicate effective EC2 use (for many workloads)
includes:
• CPU % utilisation
• Memory % utilisation
• Network IO (to internet and to EBS)
• For the purposes of today’s exercise we’ll use only CPU %

Step 1.3: Save CloudWatch data across relevant
accounts to S3
• Open source multi-account example of Cost Optimization:
EC2 Right Sizing solution which collects CloudWatch data
across multiple accounts
https://github.com/saltysoup/cost-optimization-multi

What’s a data pipeline and what’s Athena?
• Data pipelines get your data in a format and to a location
where you want it to be, typically in an automated way. Also
known as ETL (Extract, Transform and Load)
• The “Load” portion of this will get our data into Amazon
Athena, our serverless interactive query service that can
query data directly from S3 (Simple Storage Service).
• $5 per TB of data scanned. For the typical $100k p.m. biller
analysing billing data via Athena should cost approx. $5 per
month. However it’s always smart to set an AWS Budget
warning to catch rogue scripts.

Step 2.1: Create an automated way to get CUR
billing data into Athena
• Option 1: Use an open source tool
https://bitbucket.org/atlassian/squeegee/wiki/Home
• Option 2: To be updated in next version of slides

Step 2.2: Get your DIY Budget and Revenue data
into Athena
• Option 1: Manual via Athena SQL script
• Option 2: Automated method 1:
S3 Event -> call Lambda -> run Athena SQL
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-
example.html
• Option 3: Automated method 2: AWS Glue

Step 2.2: Get your DIY Budget and Revenue data
into Athena – Example SQL
CREATE EXTERNAL TABLE IF NOT EXISTS dbname.budget_and_rev_data (
`accountid` string,
`month_` string,
`budget` string,
`revenue` string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
'separatorChar' = ',',
'quoteChar' = '"',
'escapeChar' = ''
)
LOCATION 's3://bucketname/budgetandrevenue/'
TBLPROPERTIES ('has_encrypted_data'='false', "skip.header.line.count"="1")

Step 2.3: Get your CloudWatch data into Athena
• Option 1: Manual via Athena SQL script
• Option 2: Automated method 1:
S3 Event -> call Lambda -> run Athena SQL
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-
example.html
• Option 3: Automated method 2: AWS Glue

Step 2.3: Get your CloudWatch data into Athena –
Example SQL
CREATE EXTERNAL TABLE IF NOT EXISTS
dbname.cw_data (
`humanReadableTimestamp` string,
`timestamp` string,
àccountId` string,
àz` string,
ìnstanceId` string,
ìnstanceType` string,
ìnstanceTags` string,
èbsBacked` string,
`volumeIds` string,
ìnstanceLaunchTime` string,
`humanReadableInstanceLaunchTime` string,
`CPUUtilization` string,
`NetworkIn` string,
`NetworkOut` string,
`DiskReadOps` string,
`DiskWriteOps` string
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSe
rde'
WITH SERDEPROPERTIES (
'separatorChar' = ',',
'quoteChar' = '"',
'escapeChar' = ''
)
LOCATION 's3://bucketname/cw/'
TBLPROPERTIES
('has_encrypted_data'='false',
"skip.header.line.count"="1")

What’s QuickSight?
• Amazon QuickSight is a fast, cloud-powered BI service that
makes it easy to build visualizations, perform ad-hoc
analysis, and quickly get business insights from your data.
• Accessed from any browser or mobile device.
• First BI service to offer pay-per-session pricing so no upfront
costs, no annual commitments, and no charges for inactive
users!

How much does QuickSight cost?

A part of Speed to Insight is getting data to where you are
familiar with consuming it from (e.g. email).
QuickSight can send dashboards to you via email.
https://aws.amazon.com/blogs/big-data/amazon-quicksight-now-supports-
email-reports-and-data-labels/

Step 3.1: Set up a new Athena data source in
QuickSight
• S3 permissions
• Data types
• Date format
• Decimal format

QuickSight – Example SQL for date formatting
SELECT
substring(cast(from_iso8601_timestamp(bill_billingperiodstartdate) AS varchar), 1, 19)
AS billingperiodstartdate,
substring(cast(from_iso8601_timestamp(bill_billingperiodenddate) AS varchar), 1, 19)
AS billingperiodenddate,
substring(cast(from_iso8601_timestamp(lineitem_usagestartdate) AS varchar), 1, 19)
AS usagestartdate,
substring(cast(from_iso8601_timestamp(lineitem_usageenddate) AS varchar), 1, 19)
AS usageenddate,
*
FROM dbname.cost_usage_report

QuickSight – Example Date format syntax for
QuickSight
yyyy-MM-dd HH:mm:ss

Step 3.2: Visualize spend by account and month
• This helps us see
• Largest spends
• Changes in spend
• But what do we mean by spend?

• What do we mean by spend?
Cost View CUR Column Description
Blended • lineItem/UnblendedCost Cost based on a common rate across
all accounts
Unblended • lineItem/UnblendedCost Cost based on a common rate across
all accounts
Amortized • reservation/AmortizedUpfront
FeeForBillingPeriod
• reservation/UnusedAmortized
UpfrontFeeForBillingPeriod
Amortised value of upfront RI spend
Public On
Demand Cost
• pricing/
publicOnDemandCost
True reflection of usage.
Price that would have been paid if run
on-demand and if no-free tier

• What would give better insight?
• Spend by account by week
• Having account names instead of IDs
• Having a granular account structure
• Tagging for visibility into apps, teams, cost centers

Step 3.3: Now lets visualize:
“what is my % spend against budget?”
• Create a view in Athena that joins budget and billing data
• Visualise the view in QuickSight
A “view” in database-world is a saved query that does not
store any data but runs that query each time you ask.
This allows the query to be designed to always retrieve the
latest data. The query can source data from one or more
data sources

Step 3.4: Why is account X over budget?
Lets see spend by service/product for that account
• Create spend by product for all accounts
• Add a parameterised filter for linked account
• Which service/product significantly increased during April?

Step 3.5: Great to see which service, but which
team and app drove this change?
• Which team significantly increased spend during April?
• Which app significantly increased spend during April?

Step 3.6: Search for optimization opportunity in
resource sizing via EC2 instance utilization
• Join CloudWatch data with billing data in Athena
(showing peak CPU over 14 days by instance and tag)
and visualize in QuickSight
• Create a join view in Athena

Step 3.6: Search for optimization opportunity in
resource sizing via EC2 instance utilization - SQL
SELECT
lineitem_resourceid
, month
, lineitem_usageaccountid
, instancetags
, max_cpu
, sum(CAST(unblendedcost_withoutri AS DOUBLE)) AS unblendedcost_withoutri
, sum(CAST(lineitem_unblendedcost AS DOUBLE)) AS lineitem_unblendedcost
FROM "dbname"."cost_usage_report"
INNER JOIN
(SELECT instanceId, instancetags, max(cpuutilization) AS max_cpu FROM "dbname"."cw_data" GROUP BY instanceId,
instancetags) cw
ON lineitem_resourceid = cw.instanceId
GROUP BY
lineitem_resourceid
, month
, instancetags
, max_cpu

Step 3.7: Establish our first KPI
• What is the cost per revenue change month on month
for the account with a business value metric?
• Use the budget and revenue view created earlier
via the KPI visual type

Step 3.7: Establish our first KPI – Example SQL
SELECT
lineitem_resourceid
, month
, instancetags
, max_cpu
, sum(CAST(unblendedcost_withoutri AS DOUBLE)) AS unblendedcost_withoutri
, sum(CAST(lineitem_unblendedcost AS DOUBLE)) AS lineitem_unblendedcost
FROM "dbname"."cost_usage_report"
INNER JOIN
(SELECT instanceId, instancetags, max(cpuutilization) AS max_cpu FROM "dbname"."cw_data"
GROUP BY instanceId, instancetags) cw
ON lineitem_resourceid = cw.instanceId
GROUP BY
lineitem_resourceid
, month
, instancetags
, max_cpu

REAGroup has driven cost governance and good cost
behaviour through Finance working with Engineering
A talk about their story is here:
http://bit.ly/FinOpsAtREA

DevOps drives agility, applying FinOps principles
enhances control whilst maintaining agility

Next step options (if useful)
• Have a chat with your TAM about this
• If you need help we have AWS ProServe who can help,
Let your account manager know and ask to CC Peter
• Try this yourself

Agenda
• 2:00pm - Setup
• 2:10pm - Kick off, welcome, and intro
• 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your
AWS usage
• 2:50pm - Discussion and Q&A
• 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that
provides high speed to insight
• 4:30pm - Survey, Discussion, Q&A, and networking over drinks and snacks
• 5:30pm - Event Concludes

AWS Melbourne Cost Mgt. and Opti. Meetup - 20181109 - v2.2

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AWS Melbourne Cost Mgt. and Opti. Meetup - 20181109 - v2.2

Similar to AWS Melbourne Cost Mgt. and Opti. Meetup - 20181109 - v2.2 (20)

Recently uploaded

Recently uploaded (20)

AWS Melbourne Cost Mgt. and Opti. Meetup - 20181109 - v2.2