Optimizing Costs in Amazon S3 Creating Cost Efficiencies w/ Amazon S3 Storage Classes & Introducing S3 Intelligent-Tiering (STG398-R) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Optimizing cost in Amazon Simple Storage Service
(Amazon S3): Creating Cost Efficiencies and Introducing
S3 Intelligent-Tiering
Christoph Bartenstein
Principal Product Manager
Amazon S3
S T G 3 9 8 - R

Agenda
Amazon S3 overview
Pillars of cost optimization
Introducing S3 Intelligent-Tiering
Cost optimization patterns
Special guest – Pinterest
Putting it all together

Broad portfolio of storage services
Amazon Elastic
Block Store
Amazon S3 Amazon Glacier
Data movement
OnlineOffline
AWS Snow family
AWS Storage
Gateway Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon Elastic
File System
NEW! STORAGE CLASS
Object storage
S3 Standard
S3 Standard-IA
S3 One Zone-IA
S3 Glacier

Pillars of cost optimization
Application
Requirements
Data
Organization
Right
Sizing
Monitor,
Analyze,
Optimize

Define application requirements
Media
master files
Big Data
File
Sharing
Content
Distribution
Archive
Data
Analytics
Backup &
Restore
Dynamic
Websites
Mobile sync
& backup
Disaster
Recover
Re-creatable
data

Organize data with Object Tags and Prefixes
CustomerID =
3a24xxyz24
Department =
Finance
Project =
FinancialAnalysis
Classification =
Confidential
Environment =
Test
Control access, analyze usage,
lifecycle and replicate objects
Up to 10 mutable metadata tags
(key value pair) per object
Completely customizable (Dept.,
Project, Environment, etc.)
Tag objects when created, later, or
both

Choose the storage class that fits best
≥ 3 AZs 1 AZ
99.99% 99.5%
Milliseconds Hours
Hours YearsFrequent Infrequent
0 Bytes 5 Terabytes
Reduce storage cost > 80% by choosing the
storage class option that best fits your use case
2 Regions
99.9%

Monitor, Analyze, and Tier …
Monitor with S3 Inventory,
Amazon CloudWatch, AWS
CloudTrail
Tier and expire storage
with S3 Lifecycle Policy
… or just let S3 Intelligent-Tiering do the work and you save on storage costs automatically
Understand access patterns with
S3 Storage Class Analysis

AWS pricing principles
No upfront
investment
Pay-as-you-go
approach
Pay less by
using more
Pay less as AWS
grows
$

Decreasing prices and more storage options
1 2
Decreasing storage prices
S3 Standard
(2006)
Glacier
(2012)
S-IA
(2015)
Z-IA
(H1-2018)
INT
(this week)
Accelerating
innovation
2006 2018

Your choice of Amazon S3 storage classes
Access FrequencyFrequent Infrequent
• Active, frequently
accessed data
• Milliseconds access
• > 3 AZ
• $0.0210/GB
• Data with changing
access patterns
• > 3 AZ
• $0.0210 to $0.0125/GB
• Monitoring fee per Obj.
• Min storage duration
• Infrequently accessed
data
• > 3 AZ
• $0.0125/GB
• Retrieval fee per GB
• Min object size
S3 Standard S3 S-IA S3 Z-IA Amazon Glacier
• Re-creatable, less
accessed data
• 1 AZ
• $0.0100/GB
• Min object size
• Archive data
• Select minutes or
hours
• > 3 AZ
• $0.0040/GB
• Min object size
S3 INT

S3 Storage Class Analysis and S3 Lifecycle Policy
Use S3 Storage Cl ass Anal ysi s to
i d enti fy stor ag e ag e g r oups that are
l ess freq uentl y accessed
Set S3 L i fecycl e Pol i cy to tier storag e
to lower cost stor ag e classes and expi re
storage based on age of obj ect
G reat for pr edictable wor kloads (obj ect
age i nd i cates access freq uency)
F i ne tune anal ysi s by bucket, prefi x, or
obj ect tag

S3 Intelligent-Tiering automates cost savings

The story behind S3 Intelligent-Tiering
New cloud storage class that automates cost savings for customers
Heavy Lifting
Fragmented applications,
constraints on resources
and experience
Unmatched experience
>1M S3 customers,
Trillions of objects,
Millions of requests per second
Amazon Machine Learning
predict future access patterns,
inform storage of objects in most cost-
effective way
+ +

S3 Intelligent-Tiering storage class
Automatically optimizes storage costs for
data with changing access patterns
Moves objects between two storage tiers:
• Frequent access tier optimized for
frequent use of data
• Lower cost infrequent access tier
optimized for less accessed data
Monitors access patterns and auto-tiers
on granular object level
No performance impact, no operational
overhead
Milliseconds access, > 3 AZs, Monitoring
fee per Object, Minimum storage duration

Ideal use cases for S3 Intelligent-Tiering
Dynamic cost optimization with no performance impact and no operational overhead
Big Data, Data Lakes
Storage with changing access
patterns used by multiple
applications
Enterprises
Storage accessed by fragmented
applications from various
organizations
Startups
Constraint on resources and
experience to optimize storage
themselves
Amazon S3

The power of S3 storage classes
Optimize your storage cost by utilizing all S3 storage classes
Industry-leading performance,
scalability, availability, and
durability
Unmatched security, compliance, and
audit capabilities
Cost optimization on a granular
object-level
Storage classes that fit your workload
Set of analytics services such as S3
Select, Athena, Redshift, and EMR
S-IAZ-IA
Amazon
Glacier
Standard
INT
Amazon S3

Workload pattern 1 – frequently accessed data
Workl oad characteri sti cs :
• F r e q u e n t l y a c c e s s e d s t o r a g e
( > 1 0 0 % o f s t o r a g e r e t r i e v e d )
• S o m e t i m e s s m a l l o b j e c t s ( a v g .
o b j e c t s i z e ~ K B )
• S t o r a g e d u r a t i o n s o m e t i m e s
s h o r t
Common use cases:
• B i g d a t a a n a l y t i c s , d y n a m i c
w e b s i t e h o s t i n g , I o T s e n s o r
d a t a , D N A s e q u e n c e s , f i n a n c i a l
s i m u l a t i o n s , o r i g i n s t o r a g e f o r
C D N
Storage cl asses:
• S 3 S t a n d a r d , m a y b e S 3 I N T

Workload pattern 2 – infrequently accessed data
• O v e r t i m e i n f r e q u e n t l y a c c e s s e d
s t o r a g e ( < 1 0 0 % o f s t o r a g e
r e t r i e v e d a f t e r 9 0 d a y s )
• L a r g e o b j e c t s ( a v g . o b j e c t s i z e
~ M B )
• S t o r a g e d u r a t i o n l o n g t e r m
Common use cases:
• M o b i l e s y n c & b a c k u p , d a t a l o g s ,
m e d i a a s s e t s f o r g a m i n g ,
c u s t o m e r g e n e r a t e d c o n t e n t ,
d a t a s t o r e d f o r d i s a s t e r
r e c o v e r y
Storage cl asses:
• L i f e c y c l e f r o m S 3 S t a n d a r d t o S -
I A o r Z - I A f o r r e - c r e a t a b l e d a t a
• U s e S 3 I N T f o r a u t o m a t e d
t i e r i n g
• U s e G l a c i e r f o r A r c h i v e

Workload pattern 3 – data with changing access
• D a t a w i t h c h a n g i n g o r
u n p r e d i c t a b l e a c c e s s p a t t e r n s
• M i x o f o b j e c t s i z e s ( a v g . o b j e c t
s i z e ~ M B )
• S t o r a g e d u r a t i o n l o n g t e r m
Common use cases:
• M a c h i n e L e a r n i n g t r a i n i n g
d a t a , S a t e l l i t e a n d G e o s p a t i a l
i m a g e r y , F i n a n c i a l T r a n s a c t i o n
R e c o r d s , A u t o n o m o u s v e h i c l e
d a t a , d a t a l a k e s
Storage cl asses:
• S 3 I N T

Workload pattern 4 – unknown access patterns
• U n k n o w n w o r k l o a d
• Y o u o n l y k n o w t h a t o b j e c t s a r e
l a r g e ( ~ M B ) a n d s t o r a g e
d u r a t i o n i s l o n g ( ~ m o n t h s )
 S 3 I N T
Workl oad characteri sti cs:
• U n k n o w n w o r k l o a d
• U n k n o w n o b j e c t s i z e a n d s h o r t
l i v e d o b j e c t s ( < m o n t h s )
 S t a r t w i t h S 3 S t a n d a r d a n d
a f t e r s o m e t i m e l i f e c y c l e l a r g e
o b j e c t s i n t o S 3 I N T

Pinterest storage efficiency strategy
David Chaiken
Chief Architect
Pinterest
Chunyan Wang
Engineering Manager
Pinterest

Help people
discover and do
what they love

+80%
+50% of users are outside
the U.S.
On mobile
+175BPins
People
on Pinterest
each month
+3B Boards

© 2018, Pinterest. All rights reserved.
Pinterest S3 use cases
Operational Data Store on AWS S3
Business Intelligence: Reporting and Ad Hoc Analysis
Product Analytics and Experimentation
Machine Learning
...and all of those fun Pinterest images
...and more standard use cases like online storage backups

Storage usage growth
Standard
IA
Glacier

Storage efficiency strategy
Tooling & Visibility
Systematic Optimization
AWS Collaboration

Tooling & Visibility - Server Access Logging

Tooling & Visibility - inventory & access insights
S3 Server Access Logs (45 day retention)
S3 Inventory Tables (infinite retention)
Storage size by prefix and "dataset" (mostly table partitions)
S3 Access Insights Tables (infinite retention)
Storage size + access type, principal, timeframe
by prefix and "dataset" (mostly table partitions)

Tooling & Visibility - S3 insights dataset

Tooling & Visibility - deep analysis

Tooling & Visibility - alerting

Systematic optimization
Garbage collection
Compression, file format change
Lifecycle policy to IA (Infrequent Access), Glacier

Case Study: Lifecycle Policy

AWS collaboration
Performance Optimization for Operational Data Store
Advice on Pinterest's Challenges, Analytics, and Solutions
Ongoing Investigation of New Tiers of Storage:
Standard, Infrequent Access, One Zone, Auto Tiering, Glacier

Future work
Analyze Components of Storage Growth
Self-Serve Storage Budgeting and Management inside Pinterest
Granular Access Controls to Data Stored in S3

Putting it all together
Understand your application requirements
Use tags and prefixes to organize your data
Optimize across all storage classes
Cost optimize on an object level (or tag, prefix, bucket)
S3 Intelligent-Tiering for automated cost savings
INT
S-
IA
Z-IA
Glacier
Std
✔
✔
✔
✔
✔
AWS Cloud enables you to be more innovative, agile, and cost effective

Taking it to the next level
Thursday, 11/29
STG213-L: AWS Storage Leadership Session: What's New in Amazon S3, EFS, EBS, and more
13:45 | Mirage, Event Center B
Thursday, 11/29
STG404-R1: Performance & Cost Optimization at Scale with Amazon S3 & Amazon Glacier
16:00 | Aria West, Starvine 10
Friday, 11/30
STG203-R2: Best Practices for Amazon S3 and Amazon Glacier
11:30 | Mirage, Montego D

Thank you!
Christoph Bartenstein
Amazon S3
David Chaiken
chaiken@pinterest.com
Chunyan Wang
chunyan@pinterest.com

Optimizing Costs in Amazon S3 Creating Cost Efficiencies w/ Amazon S3 Storage Classes & Introducing S3 Intelligent-Tiering (STG398-R) - AWS re:Invent 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimizing Costs in Amazon S3 Creating Cost Efficiencies w/ Amazon S3 Storage Classes & Introducing S3 Intelligent-Tiering (STG398-R) - AWS re:Invent 2018

Similar to Optimizing Costs in Amazon S3 Creating Cost Efficiencies w/ Amazon S3 Storage Classes & Introducing S3 Intelligent-Tiering (STG398-R) - AWS re:Invent 2018 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Optimizing Costs in Amazon S3 Creating Cost Efficiencies w/ Amazon S3 Storage Classes & Introducing S3 Intelligent-Tiering (STG398-R) - AWS re:Invent 2018