Choosing the Right Data Storage Solution
Joe Lyons – Sr. Manager Global Storage
November 13, 2013

© 2013 Amazon.com, Inc....
AWS Storage Options
•
•
•
•
•

Scalable Storage
Inexpensive Archive Storage
Persistent Direct Attached Storage
Turn-key ga...
We are constantly producing more data
From all types of industries
#

1

Object Storage
●○○○○
AMAZON S3

SIMPLE STORAGE SERVICE
99.999999999%
Durability
Trillions
Of Unique Customer Objects
1.5 Million+
Peak Transactions Per Second
Storage Tiers: Buckets + Unlimited Objects
Reduced Redundancy Option
99.99% saves ~20%
Spotify adds over 20,000 tracks a day - RRS
Amazon S3 Website: Static Content
Amazon S3 continuous cost reduction
• 16 Price reductions since launch
• TCO: On-premises vs. Amazon S3
– Can be challengi...
RAW Storage at On perm vs. Cloud
Storage
1PB Raw Storage
800TB Usable Storage
600TB Allocated Storage

Amazon S3 Only actu...
Use Amazon S3 When You Need
•
•
•
•

Unlimited storage capacity
High durability
Storage for backups
Single Origin Store wi...
AMAZON GLACIER

LOW-COST ARCHIVING SERVICE
1c
Per GB / Month
$120
Per TB / Year
99.999999999%
Durability
3-5 Hours
Data Retrieval
STORAGE COSTS
VS

RETRIEVAL COSTS
Use Amazon Glacier When You Need
•
•
•
•
•

Inexpensive/Long-term archiving
Unlimited storage capacity
Eliminated Tape Mus...
Amazon S3 / Amazon Glacier Integration
POLICY-BASED ARCHIVING SERVICE
Lifecycle Rule
Archive Recovery Process with Tape

+ Days or Weeks
Archive Recovery Process with AWS &
Amazon Glacier
Glacier

S3

EC2
/HPC

CloudFront

Generating
Business
Value

$$
Hours
Use Amazon S3 and Amazon Glacier When You Need

•
•

•

HSM in the cloud

Archive data from Amazon S3/RRS to Amazon Glacie...
#

2

Block Storage
●●○○○
#

3

SYNC VOLUMES
●●●○○
AMAZON EBS

ELASTIC BLOCK STORAGE
Ephemeral Storage
10 GB

1TB

Provisioned

IOPS

4000
IOPS
Amazon EBS
Snapshots
Use Amazon EBS When You Need
•
•
•
•
•

Long-term persistent storage
Data changes frequently
Block storage for your databa...
AWS STORAGE GATEWAY
AWS Storage Gateway
What is AWS Storage Gateway?
Integrates on-prem IT environments with Cloud storage for
departmental and remote office back...
Solution Overview
1. Offsite Backup
2. DR

3. Data Mirroring

AWS Storage Gateway works with your existing backup
applicat...
IT’S ALL ABOUT

CHOICE
PERFORMANCE-ORIENTED
COST-ORIENTED
Understanding AWS Storage Options
Tamara Monson, iMemories, Inc.
November 13, 2013

© 2013 Amazon.com, Inc. and its affili...
Our Story: History
iMemories was founded in 2006
Cloud was in its infancy
–

Handling physical assets:
–
–
–

Using on-pre...
Our Story: Success

1.5PB
0.75PB

2013

2012

2010

8TB

2008

•

Growing infrastructure
Reliability not as high as desire...
Our Story: Need
Use Case
• Video and photo
converted and digital
source files
• Short-term local copy
• Access infrequentl...
Evaluating Options: Calculating TCO
• Operations
– Percentage FTE
– Facilities and power
– Maintenance

• Hardware
– Life ...
Evaluating Options: Comparisons

Amazon S3

Amazon Glacier

Cloud Files

Cloud
iMemories Evolution
Architecture 1.0
• All storage and processing
on-premises (circa 2006)
• Centered on physical
assets, ...
Storage 1.0
• On-premises
• ~1.5 PB capacity
• Limitations of scale:
– Add new hardware
– Space and power

Storage Volume
...
Storage 2.0: Amazon Glacier
Amazon SNS

Glacier
Manager

Amazon SQS

Vaults
Regions

AWS Cloud

Storage Volume

Digitizati...
Amazon Glacier
Manager
-

Interfaces with Amazon SQS
Tracks restore/submit request
state
Processes Submit requests in
real...
2.x: Move to AWS
v-ec2-east1.imemories.com
v-ec2-west2.imemories.com

S3 Manager
Streaming
Server
Elastic Load
Balancing

...
2.x: Move to AWS

Upload
Web App

Video
Files

Video
Processing
Group

u-ec2.imemories.com

API
Elastic Load
Balancing

Fi...
In Closing
•

iMemories Story
–
–
–

•

Evaluating Storage Options
–
–
–

•

History
Success
Need
Define use case and solu...
Enabling families to share memories in endless ways.
http://www.imemories.com
tamara.monson@imemories.com
Please give us your feedback on this
presentation

STG101
As a thank you, we will select prize
winners daily for completed...
Understanding AWS Storage Options (STG101) | AWS re:Invent 2013
Understanding AWS Storage Options (STG101) | AWS re:Invent 2013
Understanding AWS Storage Options (STG101) | AWS re:Invent 2013
Understanding AWS Storage Options (STG101) | AWS re:Invent 2013
Understanding AWS Storage Options (STG101) | AWS re:Invent 2013
Understanding AWS Storage Options (STG101) | AWS re:Invent 2013
Understanding AWS Storage Options (STG101) | AWS re:Invent 2013
Upcoming SlideShare
Loading in …5
×

Understanding AWS Storage Options (STG101) | AWS re:Invent 2013

2,167 views

Published on

With AWS you can choose the right storage service for the right use case. Given the myriad of choices, from object storage to block storage, this session will profile details and examples of some of the choices available to you, with details on real world deployments from customers using Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), Amazon Glacier and AWS Storage Gateway. In addition, this session will also cover all the new AWS storage features introduced in the last 12 months.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,167
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
150
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Understanding AWS Storage Options (STG101) | AWS re:Invent 2013

  1. 1. Choosing the Right Data Storage Solution Joe Lyons – Sr. Manager Global Storage November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. AWS Storage Options • • • • • Scalable Storage Inexpensive Archive Storage Persistent Direct Attached Storage Turn-key gateway solution Customer Presentation: iMemories
  3. 3. We are constantly producing more data
  4. 4. From all types of industries
  5. 5. # 1 Object Storage ●○○○○
  6. 6. AMAZON S3 SIMPLE STORAGE SERVICE
  7. 7. 99.999999999% Durability
  8. 8. Trillions Of Unique Customer Objects
  9. 9. 1.5 Million+ Peak Transactions Per Second
  10. 10. Storage Tiers: Buckets + Unlimited Objects
  11. 11. Reduced Redundancy Option 99.99% saves ~20%
  12. 12. Spotify adds over 20,000 tracks a day - RRS
  13. 13. Amazon S3 Website: Static Content
  14. 14. Amazon S3 continuous cost reduction • 16 Price reductions since launch • TCO: On-premises vs. Amazon S3 – Can be challenging for some customers – We can help!
  15. 15. RAW Storage at On perm vs. Cloud Storage 1PB Raw Storage 800TB Usable Storage 600TB Allocated Storage Amazon S3 Only actual usage is charged 400TB Written Application Storage
  16. 16. Use Amazon S3 When You Need • • • • Unlimited storage capacity High durability Storage for backups Single Origin Store with Delivery via Amazon CloudFront
  17. 17. AMAZON GLACIER LOW-COST ARCHIVING SERVICE
  18. 18. 1c Per GB / Month
  19. 19. $120 Per TB / Year
  20. 20. 99.999999999% Durability
  21. 21. 3-5 Hours Data Retrieval
  22. 22. STORAGE COSTS VS RETRIEVAL COSTS
  23. 23. Use Amazon Glacier When You Need • • • • • Inexpensive/Long-term archiving Unlimited storage capacity Eliminated Tape Museums Eliminate Tech Refresh High durability
  24. 24. Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE
  25. 25. Lifecycle Rule
  26. 26. Archive Recovery Process with Tape + Days or Weeks
  27. 27. Archive Recovery Process with AWS & Amazon Glacier Glacier S3 EC2 /HPC CloudFront Generating Business Value $$ Hours
  28. 28. Use Amazon S3 and Amazon Glacier When You Need • • • HSM in the cloud Archive data from Amazon S3/RRS to Amazon Glacier by policy Delete data from Amazon Glacier by policy
  29. 29. # 2 Block Storage ●●○○○
  30. 30. # 3 SYNC VOLUMES ●●●○○
  31. 31. AMAZON EBS ELASTIC BLOCK STORAGE
  32. 32. Ephemeral Storage
  33. 33. 10 GB 1TB 
  34. 34. Provisioned IOPS 4000 IOPS
  35. 35. Amazon EBS Snapshots
  36. 36. Use Amazon EBS When You Need • • • • • Long-term persistent storage Data changes frequently Block storage for your databases – Provisioned IOPS volumes Filesystem for an instance NTFS, ExtFS, RAID, LVM… Access to raw, unformatted block-level storage
  37. 37. AWS STORAGE GATEWAY
  38. 38. AWS Storage Gateway
  39. 39. What is AWS Storage Gateway? Integrates on-prem IT environments with Cloud storage for departmental and remote office backup and DR Utilizes a virtual appliance that sits in customer datacenter Exposes compatible iSCSI interface on front end Stores primary data on-AWS in Amazon S3 or on-premise with data backed-up to Amazon S3 as Amazon EBS snapshots Amazon Confidential 45
  40. 40. Solution Overview 1. Offsite Backup 2. DR 3. Data Mirroring AWS Storage Gateway works with your existing backup application and moves your data into Amazon S3 as EBS snapshots Run AWS Storage Gateway in Amazon EC2 and access snapshots up to 32 TB in size Run apps in the cloud using your uploaded data – HPC/Hadoop/Analytics 4. Dept. File Share AWS Storage Gateway – Cached Volumes will help you create Storage Volumes in cloud and keep most recent accessed data locally [Reduce SAN footprint for File Shares] 5. Archive/Glacier VTL version of AWS Storage Gateway would help customer move data in the form of Virtual Tapes from on premise to Amazon S3 (VTL – Virtual Tape Library) and then to Amazon Glacier (VTS – Virtual Tape Shelf)
  41. 41. IT’S ALL ABOUT CHOICE PERFORMANCE-ORIENTED COST-ORIENTED
  42. 42. Understanding AWS Storage Options Tamara Monson, iMemories, Inc. November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  43. 43. Our Story: History iMemories was founded in 2006 Cloud was in its infancy – Handling physical assets: – – – Using on-premises equipment: Focus was digitization for web/online viewing, DVD creation, archiving 2013 • VCRs, decks Film projectors Photo and slide scanners 2012 – – – 2010 • VHS, Betamax, MiniDV tapes 16mm, 8mm film reels Photos, slides 2008 • AWS officially launched in 2006: Amazon S3 2006 • • iMemories' Growth
  44. 44. Our Story: Success 1.5PB 0.75PB 2013 2012 2010 8TB 2008 • Growing infrastructure Reliability not as high as desired Less-than-ideal customer experience Tripled full-time resources to manage Costs for static, infrequently accessed assets much higher than for morefrequently accessed assets Rate of growth driving need for infrastructure elasticity 2006 • • • • • iMemories' Growth
  45. 45. Our Story: Need Use Case • Video and photo converted and digital source files • Short-term local copy • Access infrequently – Disc creation – Additional encodings Requirements • Durable, redundant, secure • Keep forever • No or low operational costs • No new hardware • Off-premises
  46. 46. Evaluating Options: Calculating TCO • Operations – Percentage FTE – Facilities and power – Maintenance • Hardware – Life span – Number of copies – Raw GB vs. usable GB vs. consumed GB 1.2 1 0.8 0.6 0.4 0.2 0 Time Raw Usable Consumed
  47. 47. Evaluating Options: Comparisons Amazon S3 Amazon Glacier Cloud Files Cloud
  48. 48. iMemories Evolution Architecture 1.0 • All storage and processing on-premises (circa 2006) • Centered on physical assets, devices, servers • Expensive: scale by adding more • Non-ideal customer experience Architecture 2.x • Solve scale challenges • Reduce costs • Best possible customer experience
  49. 49. Storage 1.0 • On-premises • ~1.5 PB capacity • Limitations of scale: – Add new hardware – Space and power Storage Volume Digitization, Encoding Servers iMemories Scottsdale Data center File Mgmt
  50. 50. Storage 2.0: Amazon Glacier Amazon SNS Glacier Manager Amazon SQS Vaults Regions AWS Cloud Storage Volume Digitization, Encoding Servers File Mgmt iMemories Scottsdale Data Center
  51. 51. Amazon Glacier Manager - Interfaces with Amazon SQS Tracks restore/submit request state Processes Submit requests in real-time Processes Restore requests once/hour - - Uses 10 MB chunks Performs audits 3x/week - - 5% of consumed / 720 hours  peak hourly rate Distribute request across 1-hr windows One discrepancy Zero lost files on Glacier (!!!) Generates internal activity reports File Management Glacier Manager Digitization, Encoding Servers
  52. 52. 2.x: Move to AWS v-ec2-east1.imemories.com v-ec2-west2.imemories.com S3 Manager Streaming Server Elastic Load Balancing Streaming Servers Video Files Availability Zones Regions AWS Cloud Storage Volume Digitization, Encoding Servers File Mgmt iMemories Scottsdale Data Center
  53. 53. 2.x: Move to AWS Upload Web App Video Files Video Processing Group u-ec2.imemories.com API Elastic Load Balancing File Mgmt Amazon SWF Region AWS Cloud iMemories Scottsdale Data Center
  54. 54. In Closing • iMemories Story – – – • Evaluating Storage Options – – – • History Success Need Define use case and solution requirements Understand TCO Design and build for Amazon Glacier’s 4-hr download SLA, 5% restore cap iMemories Evolution – – – Amazon Glacier is first full production integration Migrating more functionality to cloud Will always have an on-premises component
  55. 55. Enabling families to share memories in endless ways. http://www.imemories.com tamara.monson@imemories.com
  56. 56. Please give us your feedback on this presentation STG101 As a thank you, we will select prize winners daily for completed surveys!

×