Understanding AWS Storage Options (STG101) | AWS re:Invent 2013

1,940 views
1,684 views

Published on

With AWS you can choose the right storage service for the right use case. Given the myriad of choices, from object storage to block storage, this session will profile details and examples of some of the choices available to you, with details on real world deployments from customers using Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), Amazon Glacier and AWS Storage Gateway. In addition, this session will also cover all the new AWS storage features introduced in the last 12 months.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,940
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
144
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Understanding AWS Storage Options (STG101) | AWS re:Invent 2013

  1. 1. Choosing the Right Data Storage Solution Joe Lyons – Sr. Manager Global Storage November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. AWS Storage Options • • • • • Scalable Storage Inexpensive Archive Storage Persistent Direct Attached Storage Turn-key gateway solution Customer Presentation: iMemories
  3. 3. We are constantly producing more data
  4. 4. From all types of industries
  5. 5. # 1 Object Storage ●○○○○
  6. 6. AMAZON S3 SIMPLE STORAGE SERVICE
  7. 7. 99.999999999% Durability
  8. 8. Trillions Of Unique Customer Objects
  9. 9. 1.5 Million+ Peak Transactions Per Second
  10. 10. Storage Tiers: Buckets + Unlimited Objects
  11. 11. Reduced Redundancy Option 99.99% saves ~20%
  12. 12. Spotify adds over 20,000 tracks a day - RRS
  13. 13. Amazon S3 Website: Static Content
  14. 14. Amazon S3 continuous cost reduction • 16 Price reductions since launch • TCO: On-premises vs. Amazon S3 – Can be challenging for some customers – We can help!
  15. 15. RAW Storage at On perm vs. Cloud Storage 1PB Raw Storage 800TB Usable Storage 600TB Allocated Storage Amazon S3 Only actual usage is charged 400TB Written Application Storage
  16. 16. Use Amazon S3 When You Need • • • • Unlimited storage capacity High durability Storage for backups Single Origin Store with Delivery via Amazon CloudFront
  17. 17. AMAZON GLACIER LOW-COST ARCHIVING SERVICE
  18. 18. 1c Per GB / Month
  19. 19. $120 Per TB / Year
  20. 20. 99.999999999% Durability
  21. 21. 3-5 Hours Data Retrieval
  22. 22. STORAGE COSTS VS RETRIEVAL COSTS
  23. 23. Use Amazon Glacier When You Need • • • • • Inexpensive/Long-term archiving Unlimited storage capacity Eliminated Tape Museums Eliminate Tech Refresh High durability
  24. 24. Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE
  25. 25. Lifecycle Rule
  26. 26. Archive Recovery Process with Tape + Days or Weeks
  27. 27. Archive Recovery Process with AWS & Amazon Glacier Glacier S3 EC2 /HPC CloudFront Generating Business Value $$ Hours
  28. 28. Use Amazon S3 and Amazon Glacier When You Need • • • HSM in the cloud Archive data from Amazon S3/RRS to Amazon Glacier by policy Delete data from Amazon Glacier by policy
  29. 29. # 2 Block Storage ●●○○○
  30. 30. # 3 SYNC VOLUMES ●●●○○
  31. 31. AMAZON EBS ELASTIC BLOCK STORAGE
  32. 32. Ephemeral Storage
  33. 33. 10 GB 1TB 
  34. 34. Provisioned IOPS 4000 IOPS
  35. 35. Amazon EBS Snapshots
  36. 36. Use Amazon EBS When You Need • • • • • Long-term persistent storage Data changes frequently Block storage for your databases – Provisioned IOPS volumes Filesystem for an instance NTFS, ExtFS, RAID, LVM… Access to raw, unformatted block-level storage
  37. 37. AWS STORAGE GATEWAY
  38. 38. AWS Storage Gateway
  39. 39. What is AWS Storage Gateway? Integrates on-prem IT environments with Cloud storage for departmental and remote office backup and DR Utilizes a virtual appliance that sits in customer datacenter Exposes compatible iSCSI interface on front end Stores primary data on-AWS in Amazon S3 or on-premise with data backed-up to Amazon S3 as Amazon EBS snapshots Amazon Confidential 45
  40. 40. Solution Overview 1. Offsite Backup 2. DR 3. Data Mirroring AWS Storage Gateway works with your existing backup application and moves your data into Amazon S3 as EBS snapshots Run AWS Storage Gateway in Amazon EC2 and access snapshots up to 32 TB in size Run apps in the cloud using your uploaded data – HPC/Hadoop/Analytics 4. Dept. File Share AWS Storage Gateway – Cached Volumes will help you create Storage Volumes in cloud and keep most recent accessed data locally [Reduce SAN footprint for File Shares] 5. Archive/Glacier VTL version of AWS Storage Gateway would help customer move data in the form of Virtual Tapes from on premise to Amazon S3 (VTL – Virtual Tape Library) and then to Amazon Glacier (VTS – Virtual Tape Shelf)
  41. 41. IT’S ALL ABOUT CHOICE PERFORMANCE-ORIENTED COST-ORIENTED
  42. 42. Understanding AWS Storage Options Tamara Monson, iMemories, Inc. November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  43. 43. Our Story: History iMemories was founded in 2006 Cloud was in its infancy – Handling physical assets: – – – Using on-premises equipment: Focus was digitization for web/online viewing, DVD creation, archiving 2013 • VCRs, decks Film projectors Photo and slide scanners 2012 – – – 2010 • VHS, Betamax, MiniDV tapes 16mm, 8mm film reels Photos, slides 2008 • AWS officially launched in 2006: Amazon S3 2006 • • iMemories' Growth
  44. 44. Our Story: Success 1.5PB 0.75PB 2013 2012 2010 8TB 2008 • Growing infrastructure Reliability not as high as desired Less-than-ideal customer experience Tripled full-time resources to manage Costs for static, infrequently accessed assets much higher than for morefrequently accessed assets Rate of growth driving need for infrastructure elasticity 2006 • • • • • iMemories' Growth
  45. 45. Our Story: Need Use Case • Video and photo converted and digital source files • Short-term local copy • Access infrequently – Disc creation – Additional encodings Requirements • Durable, redundant, secure • Keep forever • No or low operational costs • No new hardware • Off-premises
  46. 46. Evaluating Options: Calculating TCO • Operations – Percentage FTE – Facilities and power – Maintenance • Hardware – Life span – Number of copies – Raw GB vs. usable GB vs. consumed GB 1.2 1 0.8 0.6 0.4 0.2 0 Time Raw Usable Consumed
  47. 47. Evaluating Options: Comparisons Amazon S3 Amazon Glacier Cloud Files Cloud
  48. 48. iMemories Evolution Architecture 1.0 • All storage and processing on-premises (circa 2006) • Centered on physical assets, devices, servers • Expensive: scale by adding more • Non-ideal customer experience Architecture 2.x • Solve scale challenges • Reduce costs • Best possible customer experience
  49. 49. Storage 1.0 • On-premises • ~1.5 PB capacity • Limitations of scale: – Add new hardware – Space and power Storage Volume Digitization, Encoding Servers iMemories Scottsdale Data center File Mgmt
  50. 50. Storage 2.0: Amazon Glacier Amazon SNS Glacier Manager Amazon SQS Vaults Regions AWS Cloud Storage Volume Digitization, Encoding Servers File Mgmt iMemories Scottsdale Data Center
  51. 51. Amazon Glacier Manager - Interfaces with Amazon SQS Tracks restore/submit request state Processes Submit requests in real-time Processes Restore requests once/hour - - Uses 10 MB chunks Performs audits 3x/week - - 5% of consumed / 720 hours  peak hourly rate Distribute request across 1-hr windows One discrepancy Zero lost files on Glacier (!!!) Generates internal activity reports File Management Glacier Manager Digitization, Encoding Servers
  52. 52. 2.x: Move to AWS v-ec2-east1.imemories.com v-ec2-west2.imemories.com S3 Manager Streaming Server Elastic Load Balancing Streaming Servers Video Files Availability Zones Regions AWS Cloud Storage Volume Digitization, Encoding Servers File Mgmt iMemories Scottsdale Data Center
  53. 53. 2.x: Move to AWS Upload Web App Video Files Video Processing Group u-ec2.imemories.com API Elastic Load Balancing File Mgmt Amazon SWF Region AWS Cloud iMemories Scottsdale Data Center
  54. 54. In Closing • iMemories Story – – – • Evaluating Storage Options – – – • History Success Need Define use case and solution requirements Understand TCO Design and build for Amazon Glacier’s 4-hr download SLA, 5% restore cap iMemories Evolution – – – Amazon Glacier is first full production integration Migrating more functionality to cloud Will always have an on-premises component
  55. 55. Enabling families to share memories in endless ways. http://www.imemories.com tamara.monson@imemories.com
  56. 56. Please give us your feedback on this presentation STG101 As a thank you, we will select prize winners daily for completed surveys!

×