Understanding AWS Storage OptionsJeff Putt, APAC Business Development Manager, AWS
AWS Storage OptionsObject Storage Block StorageConnect AWS Storage with On-Premises EnvironmentsFor NoSQL and Relational D...
We are sincerely eager to hearyour feedback on thispresentation and on re:Invent.Please fill out an evaluation formwhen yo...
We are sincerely eager to hearyour feedback on thispresentation and on re:Invent.Please fill out an evaluation formwhen yo...
#1Object Storage
Amazon S3Simple Storage Service
99.999999999%Durability
Over 1 TrillionUnique Customer Objects
750,000+Peak Transactions Per Second
1. High Durability2. Unlimited Capacity3. Low latency4. High Volume Static Websites5. Security & Compliance6. Delivery via...
Amazon GlacierLow-Cost Archiving Service
1¢Per GB / Month
$120Per TB / Year
99.999999999%Durability
3−5 HoursData Retrieval
Storage Costsvs.Retrieval Costs
1. High Durability2. Low cost3. 3-5 hour latencyGlacier gives you So you can• Archive files• Build Hierarchal StorageManag...
Amazon S3 and Glacier can help you build…GamingCommerceDigital MediaDistributionWebsites Social MediaPhoto/VideoSharing an...
Store unlimited objects in Amazon S3 and Amazon GlacierS3 Glacier
Access both via SDKs and via third-party softwareS3 GlacierInternet WebServices APIHTTP(S)Via SDK Via 3rd parties
Or send S3 files directly to GlacierS3 Glacier
S3 storage costs as low as 4.7 cents per GB
Serve static content at high volumes
Deliver content with lower latency via Amazon CloudFront
Smugmug is home to millions of customers sharingBillions of photos
Adds over 20,000 tracks of music a day
Nasdaq FinQloud is a WORM compliant archive forBroker Dealer financial data
Paul PetersonVice President of Engineering
nearmap Aerial Imagerynearmap routinely captures hi-res aerial imagescovering 75% of Australia’s populationnearmap provide...
Clear30
CurrentImages are delivered within7-14 days, rather than 3-6months after flying6moTraditionalnearmapProcessing Time3mo 9mo...
Bundaberg Flood32
33Bundaberg Flood
34Bundaberg Flood : Before
35Bundaberg Flood : After
36Bundaberg Flood : Before
37Bundaberg Flood : After
38Bundaberg Flood : Before
39Bundaberg Flood : After
nearmap ToolsThe photomap iscorrectly scaled foraccuratemeasurementsObjects are shown intheir correct real-world position44
Why Use nearmap• Less time driving• Less time on site• Reduce fleet costs and environmental impact• Eliminates existing sp...
nearmap and AWSQ. What does nearmap use AWS for?A. Everything• Production• Development and QA• Corporate systems• Image ar...
Live on AWS October-2012Transition went very smoothlyLocal AWS team provided fantastic supportAWS systems continue to work...
AWS Features UsedEC2 - both Reserved and Spot InstancesS3GlacierRDSRoute53CloudFormationCloudWatchIAMSESDirect Connect7 Ma...
Image Data Storage6 Terabytes of new image data per monthWe have 4+ years of existing image dataS3: 150 Terabytes and coun...
50
#2Block Storage
Amazon EBSElastic Block Storage
Ephemeral storage is on every EC2 instance
EBS volumes attach to EC2 instances
EBS volumes persist and can be attached to other instances
Multiple drives on one Amazon EC2 Instance
10GB1TB
IOPSProvisioned
Workloads on EBS Provisioned IOPSP-IOPS is designed to run transactional applications that require high andconsistent IO s...
EBSSnapshots
Take snapshots of EBS volumes and keep them in S3
Use Amazon EBS When You Need• Filesystem for an instance NTFS, ExtFS, RAID, LVM• Long-term persistent storage• Ability to ...
#3Connect AWS Storage with On-PremisesEnvironments
Access both via SDKs and via third-party softwareS3 GlacierInternet WebServices APIHTTP(S)Via SDK Via 3rd parties
Common Data Storage Challenges and the AWS OpportunityInternet WebServices APIHTTP(S)AWS DirectConnectBlock FileCustomerDa...
Common Data Storage Challenges and the AWS OpportunityPrimaryBlockStoragePrimaryFileStorageArchivalStorageDisk-BasedBackup...
Common Data Storage Challenges and the AWS OpportunityS3 & Glacier - Designed to work with other AWS Services
Storage Gateway – Connect On-Prem with the AWS Cloud1. Local, low-latency access to themost frequently used files whilesto...
Use AWS Storage Gateway When You Need To• Backup your data• Synchronize data• Export data for migration• Set up a departme...
AWS Storage OptionsObject Storage Block StorageConnect AWS Storage with On-Premises Environments+ +
Performance-OrientedCost-OrientedIT’S ALL ABOUTCHOICE
Related ContentWeb resources:• http://aws.amazon.com/s3/• http://aws.amazon.com/ebs• http://aws.amazon.com/glacier/• http:...
Thank You.
Shares customer submitted photos
Shazam hit 500,000 writes per secondduring the Super Bowl
Foursquare streams hundreds of millions of application logsinto Amazon S3 for analytics
Forced to choose between saving their source data or savingtheir results. With Amazon Glacier, they can keep both
Real Clear Politics served up 20x normal demand withS3 Websites
Understanding AWS Storage Options
AWS Sydney Summit 2013 - Understanding your AWS Storage Options
AWS Sydney Summit 2013 - Understanding your AWS Storage Options
AWS Sydney Summit 2013 - Understanding your AWS Storage Options
AWS Sydney Summit 2013 - Understanding your AWS Storage Options
Upcoming SlideShare
Loading in...5
×

AWS Sydney Summit 2013 - Understanding your AWS Storage Options

621

Published on

Session 1, Presentation 6 from the AWS Sydney Summit

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
621
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Why are we going through the different storage solutions? Data comes in many shapes and sizes – this gives you different scale issues when you are looking at volumes of dataData also comes in a different rates – this is you a scale problem of how you ingest the data.It is important to understand what technology will meet your data needs, giving the right performance at the right price
  • One of the most common workloads in the mobile sector is multi-media.With the proliferation of smartphones, the amount of data that can be sent in simultaneously from a globally distributed network of phones can be huge. Both in total volume and ingest
  • It is not just the consumer segment that is producing masses of data. Oil and gas companies, such as our customer Shell for example, have embedded sensors all the way down the drill heads – this is producing huge amounts of data that needs to be ingested and stored
  • We are durable: One of its key value propositions is durability, 11 9sYour data is stored across three separate facilities giving you geo-redundancy and we can sustain data loss in two facilities simultaneously and your data is still safe.
  • We are BIG. We manage this level of durability across over 1 trilion unique customer objects. This does not include all the objects that AWS stores like snapshots.
  • We heard the feedback around S3, that it was a great product, but when you looked at infrequently accessed data that was to be archived for a long time, you wanted a cheaper solution
  • Glacier is 1c per GB per month…
  • That equates to $120 per TB/Year.
  • Whilst keeping the 11 x 9s of durability so your data is safe.Important to remember that Glacier is designed for the long term – if you store data on tape for 50 years, you will have to refresh the tapes every 5-10 years as otherwise the hardware will end of life.With Glacier we manage that for you.If you store data today, and retrieve it in 50 years, the chances are every part of hardware except for the concrete floor will have been swapped out
  • In contrast to S3, the data in Glacier needs 3-5 hoursSend in a request to retrieve data,We then notify you when it is there.Then you can download
  • Important to manage the retrieval costs. If your data is too hot, it will be more economic to store in S3You have 5% of your data per month (prorated to the day) available to retrieve.
  • Fractal pic under creative commons: http://www.flickr.com/photos/jayakody2000lk/7216399294/
  • TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
  • TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
  • TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
  • Different pricing around the globe.Also note RRS.This gives lower durability, but at a lower cost
  • Can also serve HTML and CSSS3 can be your web server for static content
  • If you have common content that is delivered to a high number of peopleCloudfront is our CDN. This can cache content locally, and the when you ask for the content you will be directed to the nearest POP
  • https://na3.salesforce.com/a0750000008gz0tAAA
  • One of our customers is Spotify, an on demand music service who adds over 20,000 music tracks a day and delivers that music to its customersCall out RRS in description
  • File systems have been the bedrock of computing for decades.
  • For more persistent data you need EBS
  • Each server comes with Ephemeral storage,However this cannot be transferred between instances, and is lost when the instance is disposed of.Great for temp data etc
  • You can therefore attach the drive to the instance
  • However with EBS, you can then reattach that EBS volume to a new instance. Your data is now persistent
  • You can also add multiple drives. Maybe you need it to segment your data – different DBs on different drives.Or you can get the OS to create a RAID drive and therefore see it as a single volume, which gives a larger max volume size, and can increase performance.
  • Allows you to create a volume of up to 1TB, that you can either use as a block device, or you can format for a file system
  • EBS is great for spiky loads, it will allow you to flex up. However for more consistent loads – ie databases you should look at PIOPs that allow you to configure drives of up to 2000IOPS
  • What is really important is to get point in time snapshots – this is your backup
  • You can take snapshots of your EBS drives to allow you to roll back to an older version if needed -all kept in S3
  • 00:47:00
  • Important requirement is for people to be able to extend to the cloud and sync data between the twoTalk about Storage Gateway and RMAN
  • TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
  • Storage GatewayDownload a VM image that runs on your environmentGive it some local storage to use as a cache and upload buffer.I can now sync the data into the cloud as EBS snapshots.Which means that you can either download the snapshot and recover form that, or attach it to an EC2 instanceAlso have a cached volume version to enable thin provisioning or departmental / remote office file-sharingRemove some workloads from expensive file systems to simple/scalable gateway
  • 00:47:00
  • Why are we going through the different storage solutions? Data comes in many shapes and sizes – this gives you different scale issues when you are looking at volumes of dataData also comes in a different rates – this is you a scale problem of how you ingest the data.It is important to understand what technology will meet your data needs, giving the right performance at the right price
  • It is all about choice.Pick the technology that delivers the right performance at the right price.
  • https://na3.salesforce.com/a0750000006xf87AAA
  • https://na3.salesforce.com/a0750000006xf9iAAA
  • Shazam at superbowl during the adverts managed to hit 500,000 write a second – maintaining that sub 10ms write time. They actually called us 3 days prior to the SuperBowl with the goal to make the impossible, possible and did so with great success
  • https://na3.salesforce.com/a0750000008fnIQAAY
  • https://na3.salesforce.com/a0750000008fnIQAAY
  • https://na3.salesforce.com/a0750000008hOUM
  • AWS Sydney Summit 2013 - Understanding your AWS Storage Options

    1. 1. Understanding AWS Storage OptionsJeff Putt, APAC Business Development Manager, AWS
    2. 2. AWS Storage OptionsObject Storage Block StorageConnect AWS Storage with On-Premises EnvironmentsFor NoSQL and Relational Database storage, catch the dedicated sessions+ +
    3. 3. We are sincerely eager to hearyour feedback on thispresentation and on re:Invent.Please fill out an evaluation formwhen you have a chance.We are constantly producing more data
    4. 4. We are sincerely eager to hearyour feedback on thispresentation and on re:Invent.Please fill out an evaluation formwhen you have a chance.From all types of industries
    5. 5. #1Object Storage
    6. 6. Amazon S3Simple Storage Service
    7. 7. 99.999999999%Durability
    8. 8. Over 1 TrillionUnique Customer Objects
    9. 9. 750,000+Peak Transactions Per Second
    10. 10. 1. High Durability2. Unlimited Capacity3. Low latency4. High Volume Static Websites5. Security & Compliance6. Delivery via CDNS3 gives you So you can buildMobile AppsMusic AppsVideo networksVideo AppsBackup AppsPhoto AppsStatic WebsitesAd networksAnything else you’d like!
    11. 11. Amazon GlacierLow-Cost Archiving Service
    12. 12. 1¢Per GB / Month
    13. 13. $120Per TB / Year
    14. 14. 99.999999999%Durability
    15. 15. 3−5 HoursData Retrieval
    16. 16. Storage Costsvs.Retrieval Costs
    17. 17. 1. High Durability2. Low cost3. 3-5 hour latencyGlacier gives you So you can• Archive files• Build Hierarchal StorageManagement in the cloud
    18. 18. Amazon S3 and Glacier can help you build…GamingCommerceDigital MediaDistributionWebsites Social MediaPhoto/VideoSharing andBackupBig Data
    19. 19. Store unlimited objects in Amazon S3 and Amazon GlacierS3 Glacier
    20. 20. Access both via SDKs and via third-party softwareS3 GlacierInternet WebServices APIHTTP(S)Via SDK Via 3rd parties
    21. 21. Or send S3 files directly to GlacierS3 Glacier
    22. 22. S3 storage costs as low as 4.7 cents per GB
    23. 23. Serve static content at high volumes
    24. 24. Deliver content with lower latency via Amazon CloudFront
    25. 25. Smugmug is home to millions of customers sharingBillions of photos
    26. 26. Adds over 20,000 tracks of music a day
    27. 27. Nasdaq FinQloud is a WORM compliant archive forBroker Dealer financial data
    28. 28. Paul PetersonVice President of Engineering
    29. 29. nearmap Aerial Imagerynearmap routinely captures hi-res aerial imagescovering 75% of Australia’s populationnearmap provides browser-based access to up-to-date and historical aerial imagery to thousands ofcompanies, large and small29
    30. 30. Clear30
    31. 31. CurrentImages are delivered within7-14 days, rather than 3-6months after flying6moTraditionalnearmapProcessing Time3mo 9mo 1yrBrisbane floods – nearmap was the only company to capture theflooded areas and publish within days31
    32. 32. Bundaberg Flood32
    33. 33. 33Bundaberg Flood
    34. 34. 34Bundaberg Flood : Before
    35. 35. 35Bundaberg Flood : After
    36. 36. 36Bundaberg Flood : Before
    37. 37. 37Bundaberg Flood : After
    38. 38. 38Bundaberg Flood : Before
    39. 39. 39Bundaberg Flood : After
    40. 40. nearmap ToolsThe photomap iscorrectly scaled foraccuratemeasurementsObjects are shown intheir correct real-world position44
    41. 41. Why Use nearmap• Less time driving• Less time on site• Reduce fleet costs and environmental impact• Eliminates existing spend on imagery• Integrates with enterprise systems45
    42. 42. nearmap and AWSQ. What does nearmap use AWS for?A. Everything• Production• Development and QA• Corporate systems• Image archival46
    43. 43. Live on AWS October-2012Transition went very smoothlyLocal AWS team provided fantastic supportAWS systems continue to work flawlessly47
    44. 44. AWS Features UsedEC2 - both Reserved and Spot InstancesS3GlacierRDSRoute53CloudFormationCloudWatchIAMSESDirect Connect7 May, 2013
    45. 45. Image Data Storage6 Terabytes of new image data per monthWe have 4+ years of existing image dataS3: 150 Terabytes and countingGlacier: 300 Terabytes and counting7 May, 2013
    46. 46. 50
    47. 47. #2Block Storage
    48. 48. Amazon EBSElastic Block Storage
    49. 49. Ephemeral storage is on every EC2 instance
    50. 50. EBS volumes attach to EC2 instances
    51. 51. EBS volumes persist and can be attached to other instances
    52. 52. Multiple drives on one Amazon EC2 Instance
    53. 53. 10GB1TB
    54. 54. IOPSProvisioned
    55. 55. Workloads on EBS Provisioned IOPSP-IOPS is designed to run transactional applications that require high andconsistent IO such as:• Relational databases• NoSQL databases (e.g. MongoDB)• High performance file systems• Productivity applications (e.g. Microsoft Exchange)• Enterprise applications (e.g. CRM, ERP)
    56. 56. EBSSnapshots
    57. 57. Take snapshots of EBS volumes and keep them in S3
    58. 58. Use Amazon EBS When You Need• Filesystem for an instance NTFS, ExtFS, RAID, LVM• Long-term persistent storage• Ability to change data frequently• Access to raw, unformatted block-level storage
    59. 59. #3Connect AWS Storage with On-PremisesEnvironments
    60. 60. Access both via SDKs and via third-party softwareS3 GlacierInternet WebServices APIHTTP(S)Via SDK Via 3rd parties
    61. 61. Common Data Storage Challenges and the AWS OpportunityInternet WebServices APIHTTP(S)AWS DirectConnectBlock FileCustomerDataCenterStorageUse CasesArchive BackupDisasterRecoveryNextGenerationEnterpriseStorageS3 Glacier
    62. 62. Common Data Storage Challenges and the AWS OpportunityPrimaryBlockStoragePrimaryFileStorageArchivalStorageDisk-BasedBackupStorageTapeInfrastructure&ManagementReplicatedStorage forDisasterRecoveryOffsiteLocationsGeo-ResilienceNext Generation: Gateway & Cloud StorageReduceSANFootprintReduceNASFootprintEliminateArchitectureHardware& SoftwareEliminateBackupHardware& SoftwareEliminateTapeRecover inCloud orany DataCenterEliminateSecondaryDataCentersReduce Complexity. Reduce Cost.NextGenerationEnterpriseStorage-Benefits
    63. 63. Common Data Storage Challenges and the AWS OpportunityS3 & Glacier - Designed to work with other AWS Services
    64. 64. Storage Gateway – Connect On-Prem with the AWS Cloud1. Local, low-latency access to themost frequently used files whilestoring all data in Amazon S3(Cached-Volumes)Or2. Scheduled off-site backups toAmazon S3 for on-premises data(Stored-Volumes)
    65. 65. Use AWS Storage Gateway When You Need To• Backup your data• Synchronize data• Export data for migration• Set up a departmental fileshare• Conduct simple disaster recovery
    66. 66. AWS Storage OptionsObject Storage Block StorageConnect AWS Storage with On-Premises Environments+ +
    67. 67. Performance-OrientedCost-OrientedIT’S ALL ABOUTCHOICE
    68. 68. Related ContentWeb resources:• http://aws.amazon.com/s3/• http://aws.amazon.com/ebs• http://aws.amazon.com/glacier/• http://aws.amazon.com/digital-media/72
    69. 69. Thank You.
    70. 70. Shares customer submitted photos
    71. 71. Shazam hit 500,000 writes per secondduring the Super Bowl
    72. 72. Foursquare streams hundreds of millions of application logsinto Amazon S3 for analytics
    73. 73. Forced to choose between saving their source data or savingtheir results. With Amazon Glacier, they can keep both
    74. 74. Real Clear Politics served up 20x normal demand withS3 Websites
    75. 75. Understanding AWS Storage Options

    ×