Deep Dive on Amazon Glacier - STG303 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep Dive on Amazon
Glacier
M a s K u b o , S e n i o r P r o d u c t M a n a g e r , A m a z o n G l a c i e r
N o v e m b e r 3 0 , 2 0 1 7
S T G 3 0 3

Session Summary
 AWS storage portfolio
 Amazon Glacier overview
 Deep Dive on Amazon Glacier features
 Pricing/TCO
 Durability/security
 Data ingress/egress
 New features
 Pro tips
 Customer examples

Complete set of data building blocks
Amazon
EFS
Amazon
EBS
Amazon
S3
Amazon
Glacier

Data movement
OnlineOfflin
eComplete set of data building blocks
Amazon
EFS
Amazon
EBS
AWS Snow family
AWS Storage Gateway
Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier

Data movement
OnlineOffline
Data security
and management
Amazon
EFS
Amazon
EBS
AWS Snow family
AWS Storage Gateway
Family
AWS Direct Connect
Amazon S3
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
AWS QuickSight

Benefits of Amazon S3 & Amazon Glacier
Durable, Available, & Scalable

Durable, Available, & Scalable Security & Compliance

Durable, Available, & Scalable Security & Compliance Query In Place

Flexible Management

Flexible Management Ecosystem

Global Availability

Glacier Customers

Just Say No
X No capital Investment
X No commitment
X No capacity planning
X No idle capacity
X No onerous media handling
X No complex technology refreshes
X No undifferentiated heavy lifting

1.
COST
OPTIMIZATION
 Ultra-low pricing
 Automated lifecycle
 S3 Analytics
2.
DURABILITY
& SECURITY
 11 9’s of durability
 Encrypted at rest
 Vault lock
 Access policies
 CloudTrail
3.
DATA INGRESS /
EGRESS
 Flexible retrieval options
 Provisioned capacity
 Snowball/SGW/3P
 Multipart upload
 Glacier Select

Cost Optimization

Ultra-low Pricing
Amazon Glacier S3 Standard S3-IA
$ / GB / month $0.004 $0.021 $0.0125
$ / PB / month $4,194 $22,020 $13,107
Savings per month -$17,826 -$8,913

Glacier Historical Pricing

Customer Case Study
Media supply chain and delivery solution
25PB growing to 40PB
Challenges
• Complex/expensive/aging on-prem legacy infrastructure
• Decrease time-to-market for their customer’s finished content
• Improve agility and security while lowering costs
Use of AWS
• EC2 content processing and SWF, SQS, SNS for workflow automation
• S3 for storage, Glacier for content archive
• CloudFront for content distribution for OTT
Business Benefits
• Workflow pipelines can be run in a highly parallelized fashion through AWS
elastic scalability
• Significantly shorten content delivery SLA with a new AWS enabled target of 1-hr
• Fully migrating away from on-prem infrastructure.

Customer Case Study

Lifecycle Policies
Lifecycle rules take action based on object age
Example policy:
• Move all objects older than 30 days to Standard–Infrequent Access
• Move all objects older than 90 days to Amazon Glacier
Create rules to automatically Transition or Expire your storage

Lifecycle Demo

Lifecycle
Available
S3: 99.99%
S3-IA: 99.9%
Performant
Low Latency
High
Throughput
Durable
99.999999999%
Scalable
Elastic capacity
No preset limits
$0.004/GB per month
$0.0125/GB per month
“Hot” Data
Active and/or
Temporary Data
“Warm” Data
Infrequently
Accessed Data
“Cold” Data
Archive and
Compliance Data
$0.021/GB per month
$0.01/GB retrieval
Lifecycle Policies
S3-IA
Glacier
S3
Starting $0.0025/GB
retrieval
Global
Availability
Query in
Place

Transition to S3-IA when:
 Access frequency < 1 per month
 Retention > 1 month
Pro Tip #1: Choosing a Storage Class
TCO EXAMPLE
Scenario: 1GB for 1 year + 6 gets
S3: $0.252 + $0.00 = $0.252
S3-IA: $0.150 + $0.06 = $0.210
Savings: $0.042 (-17%)

Daily Storage
Class Analysis
Data-driven storage management and cost optimization for Amazon S3
Export Storage Class
Analysis to your S3 Bucket
Filter by Bucket,
Prefix, or Object
Tags
• Monitors access patterns to understand your storage usage
• After 30 days, recommends when to move objects to Standard–Infrequent Access
• Export file includes a daily report of storage, retrieved bytes, and GETs by object
age
Storage Class Analysis

Storage Class Analysis

Pro Tip #2: Zero Day Lifecycle Policies
Use tagging to move individual objects to
Amazon Glacier.

Pro Tip #3: Choosing a Storage Class
Transition to Glacier when:
 Millisecond access time not required
 Retention > 3 months
 Access Frequency
 Expedited (1-5 min): < 3 per year
 Standard (3-5 hrs): < 1 per month
 Bulk (5-12 hrs): < 4 per month
TCO EXAMPLE
Scenario: 1GB for 1 year + GETs
Expedited Standard Bulk
# of GETs in yr 3 12 50
S3 TCO $0.25 $0.25 $0.25
SIA TCO $0.18 $0.27 $0.65
Glacier TCO $0.17 $0.17 $0.17
Savings vs S3 -33% -33% -31%
Savings vs SIA -7% -38% -73%

Pro Tip #4: Cost Optimizations
1.Target average object size of at least 1MB
2.Target long-lived data

Data Durability/Security

Traditional Redundancy
Primary
Four to five 9’s of durability
RAID, periodic fixity check
No geographic redundancy
Vulnerable to concurrent failures
and catastrophic events

Traditional Redundancy
DRPrimary
Five to six 9’s of durability
Double the cost and effort
Async redundancy
Migration and technology refresh
Minimal geographic distance

AZ1
AZ2
AZ3
Eleven 9’s of durability
Objects automatically and
synchronously spread across a
minimum of three Availability Zones
AZs separated in power, network,
flood domains
Automated periodic data integrity
checks
Tolerant of concurrent failures
Optional cross-region, cross-account
replication

Glacier’s Built-in Durability
AZ1
AZ2
AZ3
Amazon Glacier
flood domains
checks
replication

Glacier’s Built-in Durability
AZ1
AZ2
AZ3
Amazon Glacier
flood domains
checks
replication
CALL TO ACTION:
James Hamilton re:invent
1026 (YouTube)

Glacier’s Built-in Security Features
 At-rest Encryption
 Server Side Encryption
 Vault Access policies
 CloudTrail Integration
 Vault Lock

Vault Lock Demo

Use case
 Glacier VaultLock: WORM storage for retention, legal hold
 Industries: FinSvcs (SEC 17a-4), Health care (patient records)
 Enterprise: E-mail, document archiving w/ retention
Value proposition
 Low cost, durable, compliant archive
PFPT in AWS
Policy engine MySQL/C*/SolrSocial
Amazon Glacier &
Vault Lock
Customer Case Study

Data Ingress/Egress

Data movement
OnlineOfflin
e
Data security
and management
Amazon
EFS
Amazon
EBS
AWS Snow family
AWS Storage Gateway
Family
AWS Direct Connect
Amazon S3
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
AWS QuickSight

AWS Snow Family
Snowball
Petabyte-scale data migration
Snowball Edge
Compute and storage for
hybrid/edge workloads
Snowmobile
Exabyte-scale data migration
• 50TB and 80TB device size
• 10G network interfaces (RJ45
and SFP+, fiber/copper)
• Starting at $200 per job
• All data encrypted end-to-
end
• S3-compatible endpoint
• 100TB local storage
• Local compute equivalent to
an Amazon EC2 m4.4xlarge
instance
• 10GBase-T, 10/25Gb SFP28,
and 40Gb QSFP+ networking
• $300 per job
• S3 and NFS endpoints
• 10 to 100PB capacity
• Interfaces up to 1Tb/s (100PB
in a few weeks)
• NFS endpoint
• $0.005/GB/mo during job

Customer Case Study
Active archive transport and archival for digital content provider
Challenges
• Need storage platform to manage active archive
content
• Existing content repository too large to migrate via
available network-based ingest methods
Use of AWS
• Amazon S3 and Snowball for massively scalable ingest
• Amazon S3 for storage, Amazon Glacier for content
archive
• Snowball to securely transport existing media content
from on-premises storage and tape vault
Business benefits
• Petabyte-scale data transport without increased
network costs
• Massive scalability and elasticity
• Lower TCO for active archive storage

Storage Gateway
Customer
premises
Storage
Gateway
Amazon EBS
snapshots
Amazon S3
Amazon Glacier
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
Enterprise
storage
Devices
Application
servers
Modes:
iSCSI = Volume Gateway
NFS = File Gateway
VTL = Tape Gateway
iSCSI
NFS
VTL
Enables using standard storage protocols to access AWS storage services

Storage Partner Solutions
Technology Solutions with AWS Storage Competency
Backup and
Recovery
Archive BCDR Primary
Storage/Tiering

Pro Tip #5: Multipart Uploads
Improving throughput and reliability with idempotency
Archive
Parallel Uploads
Parts
InitiateMultipartUpload(partSize) → uploadId
UploadPart(uploadId, data)
CompleteMultipartUpload(uploadId) → archiveId
Amazon Glacier

Flexible Data Retrieval Options
A l l o f y o u r A m a z o n G l a c i e r d a t a i s a c c e s s i b l e w i t h a n y o f
t h r e e r e t r i e v a l o p t i o n s
Standard
• Current model
• 3-5 hours
• $0.01/GB
Bulk
• Batch/Bulk access
• 5-12 hours
• $0.0025/GB
Expedited
• Rare urgent access
• 1-5 minutes
• $0.03/GB
On-site tape replacement Off-site tape replacement

Expedited Retrieval Demo

Customer Case Study
Challenges
• Use the right analytic tools for the job—seamless evolve as requirements
change, with no data migration
• Quickly ingest data from many sources, and store it efficiently in one location
• Decouple storage and compute—scale optimally and cost efficiently
Use of AWS
• Analyzes & stores 75 billion events per day with Amazon S3, Amazon EMR,
and Amazon Redshift
• Securely stores 5PB of historical data on Amazon S3 for analytics
• Multiple analytics/processing tools (e.g., Amazon Athena, Amazon EMR,
Amazon Rekognition, Amazon Redshift) to speed time to value
• Multiple on-premises data warehouses, Hadoop and big data clusters
migrated to AWS
Business benefit
• Increased agility and speed to results
• Estimated savings of $10M-20M per year over previous on-premises solution
• Eliminate silos—centrally manage, govern and access all data

Pro Tip #6: Retrieval Performance
Expedited (1-5 minutes)
• Up to 250MB objects
• Available the vast majority of the time
• Use provisioned capacity units to guarantee availability of expedited retrievals
Standard (3-5 hours)
• Regularly retrieve multi-TB archives in under 5 hours
Bulk (5-12 hours)
• Used today to cost-effectively restoring hundreds of TB in less than 12 hours
• Supports petabyte-scale workloads

Provisioned Capacity Units
• Ensures 3 Expedited requests every 5 minutes and
up to 150 MB/s of aggregate retrieval throughput
• $100 per month per unit
• No change required for expedited restore request
• Does not accelerate Expedited retrievals
• Same access time of 1 – 5 minutes for <250MB
objects

Retrieval Cost Example
Amazon Glacier Example Use Case
• Extremely cold archive store
• Compliance, digital preservation
• 5PB of storage

• Cold archive store
• Research archive, log repository
• 5PB of storage
• 100MB average object size
• 178k standard requests, 50TB (5% of
data)

• Mass content distribution, batch
workloads
• 5PB of storage
• 3.5MM bulk requests, 1PB (20% of
data)

• Active use case
• Media library, news broadcasting
• 5PB of storage
data)
• 10k expedited requests, 3TB

• Extreme active use case
• 5PB of storage
data)
• 2.6MM bulk requests, 750TB (15% of
data)
• 10k expedited requests, 3TB
• 2 provisioned capacity units

Introducing Glacier Select
Run queries directly on your Glacier objects
Simple SQL Expression
SELECT and WHERE
Familiar semantics
Work and scales like
GET requests
Integrated
AWS SDK and CLI

Glacier Select Use cases
Pattern Matching Auditing Big Data Integration

Using Glacier Select
<RestoreRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Type>SELECT</Type>
<Tier>Expedited</Tier>
<Description>this is a description</Description>
<SelectParameters>
<InputSerialization>
<Expression>select * from object</Expression>
<OutputSerialization>
<OutputLocation> <BucketName>example-output-bucket</BucketName>
<Prefix>test-s3</Prefix>
<AccessControlList>
<UserMetadata>
Example POST request

Using Glacier Select
Select everything
SELECT * FROM Object
Specify columns using positional header
SELECT s._1, s._2 FROM archive s WHERE s._3 > 100
Specify headers
SELECT s.Id, s.FirstName, s.SSN FROM archive s
Example SQL Expressions

How Glacier Select Works
App Amazon Glacier Amazon S3
Initiate-job(ArchiveId,
SQL, Tier,
S3 bucket to write output)
200 OK
Read data and
Perform
filtering
Write output
to S3Notify app using SNS, that
output ready

Glacier Select Pricing
Expedited Standard Bulk
Access time 1 - 5 minutes 3 - 5 hours 5 - 12 hours
Data Scanned $0.02 per GB $0.008 per GB $0.001 per GB
Data Returned $0.03 per GB $0.01 per GB $0.0025 per GB
Request $0.01 per request
$0.05 per
1,000 requests
$0.025 per
1,000 requests

Thank you!

STG312 – Best Practices for Building a Data Lake in Amazon S3 & Amazon
Glacier – Thurs, 3:15 PM
Learn more…

For Enterprise Storage Engineers
• Learn how to architect and
manage highly available
solutions on AWS storage
services
• Advance toward AWS
certifications
• Help your organization migrate
to the cloud faster
Online at www.aws.training
• Access 100+ new digital
training courses including
advanced training on storage
• Deep Dives on Amazon S3,
Amazon EFS, and Amazon EBS
• Migrating and Tiering Storage
to AWS (Hybrid Solutions)
At re:Invent
• Visit Hands-on Labs at the
Venetian
• Attend a proctored
“Introduction to EFS” Spotlight
Lab on Thursday at 3pm at the
Venetian
• Meet Storage experts at the Ask
the Experts in Hands-on Labs
room at the Venetian
New Storage Training

Q&A
Amazon Glacier

Deep Dive on Amazon Glacier - STG303 - re:Invent 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Deep Dive on Amazon Glacier - STG303 - re:Invent 2017

Similar to Deep Dive on Amazon Glacier - STG303 - re:Invent 2017 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Deep Dive on Amazon Glacier - STG303 - re:Invent 2017