Understanding AWS Storage Options (STG101) | AWS re:Invent 2013
 

Like this? Share it with your network

Share

Understanding AWS Storage Options (STG101) | AWS re:Invent 2013

on

  • 1,484 views

With AWS you can choose the right storage service for the right use case. Given the myriad of choices, from object storage to block storage, this session will profile details and examples of some of ...

With AWS you can choose the right storage service for the right use case. Given the myriad of choices, from object storage to block storage, this session will profile details and examples of some of the choices available to you, with details on real world deployments from customers using Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), Amazon Glacier and AWS Storage Gateway. In addition, this session will also cover all the new AWS storage features introduced in the last 12 months.

Statistics

Views

Total Views
1,484
Views on SlideShare
1,484
Embed Views
0

Actions

Likes
1
Downloads
106
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Understanding AWS Storage Options (STG101) | AWS re:Invent 2013 Presentation Transcript

  • 1. Choosing the Right Data Storage Solution Joe Lyons – Sr. Manager Global Storage November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • 2. AWS Storage Options • • • • • Scalable Storage Inexpensive Archive Storage Persistent Direct Attached Storage Turn-key gateway solution Customer Presentation: iMemories
  • 3. We are constantly producing more data
  • 4. From all types of industries
  • 5. # 1 Object Storage ●○○○○
  • 6. AMAZON S3 SIMPLE STORAGE SERVICE
  • 7. 99.999999999% Durability
  • 8. Trillions Of Unique Customer Objects
  • 9. 1.5 Million+ Peak Transactions Per Second
  • 10. Storage Tiers: Buckets + Unlimited Objects
  • 11. Reduced Redundancy Option 99.99% saves ~20%
  • 12. Spotify adds over 20,000 tracks a day - RRS
  • 13. Amazon S3 Website: Static Content
  • 14. Amazon S3 continuous cost reduction • 16 Price reductions since launch • TCO: On-premises vs. Amazon S3 – Can be challenging for some customers – We can help!
  • 15. RAW Storage at On perm vs. Cloud Storage 1PB Raw Storage 800TB Usable Storage 600TB Allocated Storage Amazon S3 Only actual usage is charged 400TB Written Application Storage
  • 16. Use Amazon S3 When You Need • • • • Unlimited storage capacity High durability Storage for backups Single Origin Store with Delivery via Amazon CloudFront
  • 17. AMAZON GLACIER LOW-COST ARCHIVING SERVICE
  • 18. 1c Per GB / Month
  • 19. $120 Per TB / Year
  • 20. 99.999999999% Durability
  • 21. 3-5 Hours Data Retrieval
  • 22. STORAGE COSTS VS RETRIEVAL COSTS
  • 23. Use Amazon Glacier When You Need • • • • • Inexpensive/Long-term archiving Unlimited storage capacity Eliminated Tape Museums Eliminate Tech Refresh High durability
  • 24. Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE
  • 25. Lifecycle Rule
  • 26. Archive Recovery Process with Tape + Days or Weeks
  • 27. Archive Recovery Process with AWS & Amazon Glacier Glacier S3 EC2 /HPC CloudFront Generating Business Value $$ Hours
  • 28. Use Amazon S3 and Amazon Glacier When You Need • • • HSM in the cloud Archive data from Amazon S3/RRS to Amazon Glacier by policy Delete data from Amazon Glacier by policy
  • 29. # 2 Block Storage ●●○○○
  • 30. # 3 SYNC VOLUMES ●●●○○
  • 31. AMAZON EBS ELASTIC BLOCK STORAGE
  • 32. Ephemeral Storage
  • 33. 10 GB 1TB 
  • 34. Provisioned IOPS 4000 IOPS
  • 35. Amazon EBS Snapshots
  • 36. Use Amazon EBS When You Need • • • • • Long-term persistent storage Data changes frequently Block storage for your databases – Provisioned IOPS volumes Filesystem for an instance NTFS, ExtFS, RAID, LVM… Access to raw, unformatted block-level storage
  • 37. AWS STORAGE GATEWAY
  • 38. AWS Storage Gateway
  • 39. What is AWS Storage Gateway? Integrates on-prem IT environments with Cloud storage for departmental and remote office backup and DR Utilizes a virtual appliance that sits in customer datacenter Exposes compatible iSCSI interface on front end Stores primary data on-AWS in Amazon S3 or on-premise with data backed-up to Amazon S3 as Amazon EBS snapshots Amazon Confidential 45
  • 40. Solution Overview 1. Offsite Backup 2. DR 3. Data Mirroring AWS Storage Gateway works with your existing backup application and moves your data into Amazon S3 as EBS snapshots Run AWS Storage Gateway in Amazon EC2 and access snapshots up to 32 TB in size Run apps in the cloud using your uploaded data – HPC/Hadoop/Analytics 4. Dept. File Share AWS Storage Gateway – Cached Volumes will help you create Storage Volumes in cloud and keep most recent accessed data locally [Reduce SAN footprint for File Shares] 5. Archive/Glacier VTL version of AWS Storage Gateway would help customer move data in the form of Virtual Tapes from on premise to Amazon S3 (VTL – Virtual Tape Library) and then to Amazon Glacier (VTS – Virtual Tape Shelf)
  • 41. IT’S ALL ABOUT CHOICE PERFORMANCE-ORIENTED COST-ORIENTED
  • 42. Understanding AWS Storage Options Tamara Monson, iMemories, Inc. November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • 43. Our Story: History iMemories was founded in 2006 Cloud was in its infancy – Handling physical assets: – – – Using on-premises equipment: Focus was digitization for web/online viewing, DVD creation, archiving 2013 • VCRs, decks Film projectors Photo and slide scanners 2012 – – – 2010 • VHS, Betamax, MiniDV tapes 16mm, 8mm film reels Photos, slides 2008 • AWS officially launched in 2006: Amazon S3 2006 • • iMemories' Growth
  • 44. Our Story: Success 1.5PB 0.75PB 2013 2012 2010 8TB 2008 • Growing infrastructure Reliability not as high as desired Less-than-ideal customer experience Tripled full-time resources to manage Costs for static, infrequently accessed assets much higher than for morefrequently accessed assets Rate of growth driving need for infrastructure elasticity 2006 • • • • • iMemories' Growth
  • 45. Our Story: Need Use Case • Video and photo converted and digital source files • Short-term local copy • Access infrequently – Disc creation – Additional encodings Requirements • Durable, redundant, secure • Keep forever • No or low operational costs • No new hardware • Off-premises
  • 46. Evaluating Options: Calculating TCO • Operations – Percentage FTE – Facilities and power – Maintenance • Hardware – Life span – Number of copies – Raw GB vs. usable GB vs. consumed GB 1.2 1 0.8 0.6 0.4 0.2 0 Time Raw Usable Consumed
  • 47. Evaluating Options: Comparisons Amazon S3 Amazon Glacier Cloud Files Cloud
  • 48. iMemories Evolution Architecture 1.0 • All storage and processing on-premises (circa 2006) • Centered on physical assets, devices, servers • Expensive: scale by adding more • Non-ideal customer experience Architecture 2.x • Solve scale challenges • Reduce costs • Best possible customer experience
  • 49. Storage 1.0 • On-premises • ~1.5 PB capacity • Limitations of scale: – Add new hardware – Space and power Storage Volume Digitization, Encoding Servers iMemories Scottsdale Data center File Mgmt
  • 50. Storage 2.0: Amazon Glacier Amazon SNS Glacier Manager Amazon SQS Vaults Regions AWS Cloud Storage Volume Digitization, Encoding Servers File Mgmt iMemories Scottsdale Data Center
  • 51. Amazon Glacier Manager - Interfaces with Amazon SQS Tracks restore/submit request state Processes Submit requests in real-time Processes Restore requests once/hour - - Uses 10 MB chunks Performs audits 3x/week - - 5% of consumed / 720 hours  peak hourly rate Distribute request across 1-hr windows One discrepancy Zero lost files on Glacier (!!!) Generates internal activity reports File Management Glacier Manager Digitization, Encoding Servers
  • 52. 2.x: Move to AWS v-ec2-east1.imemories.com v-ec2-west2.imemories.com S3 Manager Streaming Server Elastic Load Balancing Streaming Servers Video Files Availability Zones Regions AWS Cloud Storage Volume Digitization, Encoding Servers File Mgmt iMemories Scottsdale Data Center
  • 53. 2.x: Move to AWS Upload Web App Video Files Video Processing Group u-ec2.imemories.com API Elastic Load Balancing File Mgmt Amazon SWF Region AWS Cloud iMemories Scottsdale Data Center
  • 54. In Closing • iMemories Story – – – • Evaluating Storage Options – – – • History Success Need Define use case and solution requirements Understand TCO Design and build for Amazon Glacier’s 4-hr download SLA, 5% restore cap iMemories Evolution – – – Amazon Glacier is first full production integration Migrating more functionality to cloud Will always have an on-premises component
  • 55. Enabling families to share memories in endless ways. http://www.imemories.com tamara.monson@imemories.com
  • 56. Please give us your feedback on this presentation STG101 As a thank you, we will select prize winners daily for completed surveys!