Cost-effective Data Management with S3 Storage Classes and Batch Operations

P U B L I C S E C T O R
S U M M I T
WASHINGTON D.C.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Cost-effective Data Management with S3
Batch Operations and the S3 Storage
Classes
Rob Wilson
Product Manager
Amazon Web Services
3 2 0 9 8 7

S U M M I T
Agenda
Pillars of cost optimization
Data placement
Introducing Amazon S3 Intelligent-Tiering
Optimizing for specific workloads
Amazon S3 Batch Operations overview and demo

S U M M I T
Pillars of cost optimization

S U M M I T
Pillars of Cost Optimization
Application
Requirements
Data
Organization
Right
Sizing
Monitor,
Optimize, Repeat

S U M M I T
Benefits of Amazon Simple Storage Service (Amazon S3)
Enterprise
applications
Analytics
Archiving
Backup & restore
Origin storage for
CDN
Website hosting
Mobile sync
and storage

S U M M I T
Your choice of Amazon S3 storage classes
Access frequencyFrequent Infrequent
• Active, frequently
accessed data
• Data with changing
access patterns
• Infrequently accessed
data
• Re-creatable, less
accessed data
• Archive data
S3 Standard S3 Standard-IA S3 One Zone-IA S3 Glacier
S3 Intelligent-
Tiering
S3 Glacier
Deep Archive
• Archive data

S U M M I T
accessed data
• Milliseconds access
access patterns
data
accessed data
• Archive data
• Minutes or hours access
S3 Intelligent-
Tiering
S3 Glacier
Deep Archive
• Archive data
• Hours to access

S U M M I T
accessed data
• > 3 AZ
• $0.0210/GB
access patterns
• > 3 AZ
• $0.0210 to
$0.0125/GB
data
• > 3 AZ
• $0.0125/GB
accessed data
• 1 AZ
• $0.0100/GB
• Archive data
• > 3 AZ
• $0.0040/GB
S3 Intelligent-
Tiering
S3 Glacier
Deep Archive
• Archive data
• Hours to access
• > 3 AZ
• $0.00099/GB

S U M M I T
accessed data
• > 3 AZ
• $0.0210/GB
access patterns
• > 3 AZ
• $0.0210 to
$0.0125/GB
• Monitoring fee
• Min storage duration
data
• > 3 AZ
• $0.0125/GB
• Retrieval fee
• Min object size
accessed data
• 1 AZ
• $0.0100/GB
• Retrieval fee
• Min object size
• Archive data
• > 3 AZ
• $0.0040/GB
• Retrieval fee
• Min object size
S3 Intelligent-
Tiering
S3 Glacier
Deep Archive
• Archive data
• Hours to access
• > 3 AZ
• $0.00099/GB
• Retrieval fee
• Min object size

S U M M I T
Data placement

S U M M I T
Customers save millions of dollars annually with
Storage Class Analysis and
Lifecycle Management

S U M M I T
Amazon S3 Storage Class Analysis
provides lifecycle recommendations
Monitors access patterns
Classifies data as
frequently or infrequently
accessed
Great for predictable
workloads
Can be filtered by bucket,
prefix, or object tag

S U M M I T
Set Amazon S3 Lifecycle Policies to Tier and Expire
storage
Amazon S3 lifecycle policies tier
objects to lower cost storage
classes and expire storage
Amazon S3 Storage Class Analysis
results help you set lifecycle
policies
Policies are based on the object’s
age and can be set by bucket,
prefix, or object tag
S3 Standard S3 S-IA S3 Glacier

S U M M I T
Lifecycle Management Example Policies
Lifecycle rules take action based on object age:
1. Move all objects older than 60 days to Amazon S3 S-IA
2. Move all objects older than 180 days to Amazon S3 Glacier
S3 Standard S3 S-IA S3 Glacier

S U M M I T
S3 Intelligent-
Tiering
S3 Glacier

S U M M I T
2. Move all objects older than 365 days to Amazon S3 Glacier Deep Archive
S3 Standard S3 Glacier
S3 Glacier Deep
Archive

S U M M I T
Introducing
Amazon S3 Intelligent-Tiering

S U M M I T
Amazon S3 Intelligent-Tiering automates cost savings

S U M M I T
Amazon S3 Intelligent-Tiering storage class
• Automatically optimizes storage costs
for data with changing access patterns
• Monitors access patterns and auto -
tiers on granular object level
• No performance impact, no operational
overhead
• Milliseconds access, > 3 AZs
• Monitoring fee per object, minimum
storage duration

S U M M I T
Ideal use cases for Amazon S3 Intelligent-Tiering
Dynamic cost optimization with no performance impact and no operational overhead
Big Data, Data Lakes
Storage with changing access
patterns used by multiple
applications
Enterprises
Storage accessed by fragmented
applications from various
organizations
Startups
Constraint on resources and
experience to optimize storage
themselves

S U M M I T
Workload pattern 1 – frequently accessed data
Workl oad characteri sti cs:
• F r e q u e n t l y a c c e s s e d s t o r a g e
( > 1 0 0 % o f s t o r a g e r e t r i e v e d )
Common use cases:
• B i g d a t a a n a l y t i c s , d y n a m i c
w e b s i t e h o s t i n g , I o T s e n s o r
d a t a , D N A s e q u e n c i n g ,
f i n a n c i a l s i m u l a t i o n s , o r i g i n
s t o r a g e f o r C D N
Storage cl asses:
• A m a z o n S 3 S t a n d a r d , m a y b e S 3
I n t e l l i g e n t - T i e r i n g

S U M M I T
Workload pattern 2 – infrequently accessed data
• O v e r t i m e i n f r e q u e n t l y a c c e s s e d
s t o r a g e ( < 1 0 0 % o f s t o r a g e
r e t r i e v e d a f t e r 9 0 d a y s )
Common use cases:
• M o b i l e s y n c & b a c k u p , d a t a
l o g s , m e d i a a s s e t s f o r g a m i n g ,
c u s t o m e r g e n e r a t e d c o n t e n t ,
d a t a s t o r e d f o r d i s a s t e r
r e c o v e r y
Amazon S3 storage cl asses:
• L i f e c y c l e f r o m A m a z o n S 3
S t a n d a r d t o S 3 S t a n d a r d - I A o r
S 3 O n e Z o n e - I A f o r r e - c r e a t a b l e
d a t a
• U s e A m a z o n S 3 I n t e l l i g e n t -
T i e r i n g f o r a u t o m a t e d t i e r i n g
• U s e A m a z o n S 3 G l a c i e r f o r
a r c h i v e

S U M M I T
Workload pattern 3 – data with changing access
• D a t a w i t h c h a n g i n g o r
u n p r e d i c t a b l e a c c e s s p a t t e r n s
• M i x o f o b j e c t s i z e s ( a v g . o b j e c t
s i z e ~ M B )
Common use cases:
• M a c h i n e L e a r n i n g t r a i n i n g
d a t a , S a t e l l i t e a n d G e o s p a t i a l
i m a g e r y , F i n a n c i a l T r a n s a c t i o n
R e c o r d s , A u t o n o m o u s v e h i c l e
d a t a , d a t a l a k e s
Storage cl asses:
• S 3 I n t e l l i g e n t - T i e r i n g

S U M M I T
Workload pattern 4 – unknown access patterns
• U n k n o w n w o r k l o a d
• Y o u o n l y k n o w t h a t o b j e c t s a r e
l a r g e ( ~ M B ) a n d s t o r a g e
d u r a t i o n i s l o n g ( ~ m o n t h s )
 S 3 I n t e l l i g e n t - T i e r i n g
• U n k n o w n w o r k l o a d
• U n k n o w n o b j e c t s i z e a n d
s h o r t - l i v e d o b j e c t s ( < m o n t h s )
 S t a r t w i t h A m a z o n S 3
S t a n d a r d a n d a f t e r s o m e t i m e
l i f e c y c l e l a r g e o b j e c t s i n t o S 3
I n t e l l i g e n t - T i e r i n g
?

S U M M I T
Amazon S3 Batch Operations

S U M M I T
Save on operations and development time by managing
billions of objects with a single request

S U M M I T
S3 Batch Operations: How it works
Choose
Objects
Select an
Operation
View
Progress
75%

S U M M I T
• Amazon S3
Inventory Report
• CSV List
Choose
Objects
Select an
Operation
View
Progress

S U M M I T
• Amazon S3
Inventory Report
• CSV List
• Copy
• Restore from Amazon S3
Glacier
• Put Access Control List (ACL)
• Replace Object Tag Sets
• Run AWS Lambda Functions
Choose
Objects
Select an
Operation
View
Progress

S U M M I T
• Amazon S3
Inventory Report
• CSV List
• Copy
• Restore from Amazon S3
Glacier
• Put Access Control List (ACL)
• Replace Object Tag Sets
• Run Lambda Functions
• Object Level Progress
• Job Notifications
• Completion Report
Choose
Objects
Select an
Operation
View
Progress

S U M M I T
Amazon S3 Inventory
Regularly generates a list of
objects for
analytics and auditing.
 Storage class
 Creation date
 Encryption status
 Replication status
 Object size, and more

S U M M I T
Console view

S U M M I T
Amazon S3 Batch Operations is a managed
solution
• Automatic retries
• Progress visibility
• Management controls
• Notifications
• Auditing
No need to build and maintain an
application to call APIs in bulk

S U M M I T

S U M M I T
Demo Overview
• Images stored in an Amazon S3 bucket
• Amazon S3 Batch Operations invokes an AWS
Lambda function that calls Amazon
Rekognition
• Results stored in Amazon Elasticsearch Service
• A Kibana dashboard shows the output

S U M M I T
Demo Overview
S3 Batch
Operations
Lambda function
Create job
Images
Labels Data visualization

S U M M I T
Common uses for Amazon S3 Batch Operations

S U M M I T
Encrypt your existing objects
Use Amazon S3 Select or Amazon Athena to filter the bucket’s contents
and identify only the unencrypted objects
Copy objects to the same bucket and specify the desired type
of encryption

S U M M I T
Copy objects to a new location
Works within regions, across regions, and across accounts

S U M M I T
Restore objects from Amazon S3 Glacier
Easily restore and then copy objects by using the same
manifest for both operations
Use Amazon S3 Glacier restore notifications to identify when objects are
available in Amazon S3

S U M M I T
Putting it all together

S U M M I T
Putting it all together
Understand your application requirements
Use tags and prefixes to organize your data
Optimize across all storage classes
Use Amazon S3 Batch Operations to simplify common tasks
Amazon S3 Intelligent-Tiering for automated cost savings
AWS Cloud enables you to be more innovative, agile, and cost effective

Thank you!
S U M M I T
Rob Wilson
Product Manager, Amazon S3

Cost-effective Data Management with S3 Storage Classes and Batch Operations

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Cost-effective Data Management with S3 Storage Classes and Batch Operations

Similar to Cost-effective Data Management with S3 Storage Classes and Batch Operations (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Cost-effective Data Management with S3 Storage Classes and Batch Operations