Amazon Web Services Update | LondonNovember 2012
Getting to Glacier…Why AWS for storage & archive?AWS fundamental servicesStorage & archive – examples &patternsAmazon Glac...
Storage & Archive                          AWS is used in a variety of ways…Powers applications that allows customers to a...
Business & technical drivers                      You might be able to:       Reduce costs                     Reduce on-p...
Business & technical drivers                       You might be able to:        Reduce costs                     Reduce on...
Business & technical drivers                       You might be able to:        Reduce costs                     Reduce on...
Business & technical drivers                       You might be able to:        Reduce costs                     Reduce on...
Business & technical drivers                       You might be able to:        Reduce costs                     Reduce on...
AWS fundamental services
Fundamental Storage Options                                        Elastic Block Store, S3 and Glacier    Elastic Block St...
Fundamental Storage Options                                        Elastic Block Store, S3 and Glacier    Elastic Block St...
Fundamental Storage Options                                        Elastic Block Store, S3 and Glacier    Elastic Block St...
Fundamental Storage Options                                        Elastic Block Store, S3 and Glacier    Elastic Block St...
Use case journey On-premise       On-instance   Object level   Long term    Locallyaccessible file   systems Workloadswith...
Use case journey On-premise       On-instance   Object level   Long term    Locallyaccessible file   systems             A...
Use case journey On-premise       On-instance     Object level    Long term    Locally        EC2 based          Data     ...
Use case journey On-premise         On-instance          Object level      Long term    Locally           EC2 based       ...
Use case journey On-premise         On-instance          Object level      Long term    Locally           EC2 based       ...
Use case journey On-premise             On-instance          Object level      Long term    Locally       1       EC2 base...
Getting data into the cloud                         Direct connect, import/export and storage gatewayAWS Direct Connect   ...
Use case journey On-premise             On-instance          Object level      Long term    Locally       1       EC2 base...
Use case journey On-premise             On-instance         Object level       Long term    Locally       1      EC2 based...
Curiosity
Curiosity            The mars.jpl.nasa.gov website             is based on the open-source            Content Management S...
Use case journey On-premise             On-instance         Object level       Long term    Locally       1      EC2 based...
Use case journey On-premise                On-instance         Object level       Long term    Locally       1         EC2...
Database services                            RDS, SimpleDB, DynamoDBRelational Database                SimpleDB           ...
Use case journey On-premise                On-instance         Object level       Long term    Locally       1         EC2...
Use case journey On-premise                On-instance         Object level       Long term    Locally       1         EC2...
Web accessible S3 storage…
You put in it S3AWS stores with 99.999999999% durability
Highly scalable web                         access to objects    You put in it S3AWS stores with 99.999999999% durability ...
“Spotify needed a storage solution that  could scale very quickly without incurringlong lead times for upgrades. This led ...
Use case journey On-premise                On-instance         Object level       Long term    Locally       1         EC2...
Use case journey On-premise                On-instance         Object level        Long term    Locally       1         EC...
What we heard from youYou love Amazon S3  for its simplicity, security, durability, and performance.
What we heard from youYou love Amazon S3           You wanted a highly  for its simplicity,         secure, extremely secu...
The need…    Reliable and cheap storage of data Data with long       Multi-PB, infrequentlyretention periods      accessed...
spectrumdata.com.au
Our goals with Glacier…Redefine data archiving           Replace physical media for    and backup:                        ...
The solution…Reliable and cheap storage of data
The solution…      Reliable and cheap storage of data    Same storagedurability mechanisms         as S3
The solution…      Reliable and cheap storage of data    Same storage        Trade-off on retrievaldurability mechanisms  ...
The solution…      Reliable and cheap storage of data    Same storage        Trade-off on retrievaldurability mechanisms  ...
3-5 hour retrieval time We assume you won’t access often
Glacier allows you to cost-effectively and securely storeOffsite archive   enterprise data offsite, making it simple, inex...
Glacier allows you to cost-effectively and securely storeOffsite archive        enterprise data offsite, making it simple,...
Glacier allows you to cost-effectively and securely storeOffsite archive        enterprise data offsite, making it simple,...
Good reasons to replace off-site tape archives 100% restore success rate – no broken or missing tapes No lost tapes and im...
S3      GlacierBucket     VaultObject    Archive
Create vault supported via console
What is an archive?Any object, such as a photo, video, document or             compressed collection It is a base unit of ...
API credentials            Glacier client             (keys)                                                            Re...
Transfer manager                                             Region endpointvar manager = newArchiveTransferManager(Amazon...
Retrieval   S3                GlacierSynchronous      Asynchronous Immediate           3-5 hours
Retrieval1. Initiate a retrieval job2. After the job completes, download the   bytes
Initiate job      JobParameters jobParameters = new JobParameters()Glacier   .withArchiveId("*** provide an archive id ***...
Track job   After 3-5 hours:1. SNS topic notification2. Call describeJob                    Using JobID
API credentials          Download job        Glacier client         (keys)                                                ...
Download jobvar manager = new ArchiveTransferManager(Amazon.RegionEndpoint.USEast1);var options = new DownloadOptions();op...
“Every day our genome sequencers produceterabytes of data. As our company moves into             the clinical space, we fa...
“An organization like ours thinks in centuries   when it comes to content retention, and long   term preservation of our M...
Desktop clientsS3 integration coming soon    Use Glacier through S3 APIs    Policy based tiered storage
Pricing
Storage            Retrievals           Data In   Data OutFrom $0.1 per GB   Free up to 5% of     Free      Tiered (1st GB...
Storage             Retrievals              Data In        Data OutFrom $0.1 per GB    Free up to 5% of        Free       ...
Benefits of Amazon Glacier          Low cost                                               SecureAs little as $0.01/GB/mon...
http://aws.amazon.com/gla            cier/
AWS Update | London - Amazon Glacier
AWS Update | London - Amazon Glacier
AWS Update | London - Amazon Glacier
AWS Update | London - Amazon Glacier
AWS Update | London - Amazon Glacier
Upcoming SlideShare
Loading in...5
×

AWS Update | London - Amazon Glacier

1,541

Published on

We are excited to announce Amazon Glacier, a fully-managed archive service in the cloud that allows customers to store data in 'cold storage' at an extremely competitive price point. Built to support the same 11 9s durability as S3, we'll take you through Glacier, how it works, where it sits with the storage spectrum and our planned integration with S3.

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,541
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
48
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

AWS Update | London - Amazon Glacier

  1. 1. Amazon Web Services Update | LondonNovember 2012
  2. 2. Getting to Glacier…Why AWS for storage & archive?AWS fundamental servicesStorage & archive – examples &patternsAmazon Glacier
  3. 3. Storage & Archive AWS is used in a variety of ways…Powers applications that allows customers to access historical Store its vast repository of music to stock price information feed to over 15 million active users Estimates it has saved $500,000 Digital assets and usage data behind in storage expenditures and cut publication sites and mobileits disk storage array costs in half applications
  4. 4. Business & technical drivers You might be able to: Reduce costs Reduce on-premiseSlash storage & archive budgets Eliminate on premise equipment to manage archives Change processes Remove aging technologiesRemove the need to do capacity Eliminate tape for backup and archive planning
  5. 5. Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramaticallySlash storage & archive budgets by Eliminate on premise equipment to increasing scalability up to 50% manage archivesEliminate the need for secondary sites Change processes Remove aging technologies Remove the need to do capacity Eliminate tape for backup and archive planning
  6. 6. Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate tape for backup and archive planning
  7. 7. Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate capacity planning Eliminate tape for backup and archive planning Eliminate provisioning for peak demand
  8. 8. Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate capacity planning Eliminate tape for backup and planning Remove tape archives Eliminate provisioning for peak Cycle out aging disk arrays demand
  9. 9. AWS fundamental services
  10. 10. Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities
  11. 11. Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities Very fast Fast web object Slow, rare access ‘instance’ disks storage
  12. 12. Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities
  13. 13. Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Archive Backup Simple Storage Service DR GlacierHigh performance block storage device Highly scalable object storage Long term object archive Data1TB in size 1GB to accessed Snapshots 1 byte to 5TB in size Extremely low cost per gigabyte Rapid RTO ~>10% / monthAmazo as drives to instances with Mount 99.999999999% durability 99.999999999% durability Shorter term data nsnapshot/cloning functionalities backup with rapid S3 Expiration policies 11 9s durability RTOAmazo Lower cost when 11 n S3 9s not required Lower cost Lower cost RRS Long termAmazo archiving Use policies to Retain “write once - move cold backup read never” copy in n Infrequent data data for long term case of worst caseGlacier access (~<10% retention scenario data/month)
  14. 14. Use case journey On-premise On-instance Object level Long term Locallyaccessible file systems Workloadswith local data
  15. 15. Use case journey On-premise On-instance Object level Long term Locallyaccessible file systems AWS Workloadswith local data
  16. 16. Use case journey On-premise On-instance Object level Long term Locally EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data
  17. 17. Use case journey On-premise On-instance Object level Long term Locally EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  18. 18. Use case journey On-premise On-instance Object level Long term Locally EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  19. 19. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  20. 20. Getting data into the cloud Direct connect, import/export and storage gatewayAWS Direct Connect AWS Import/Export Amazon Storage GatewayDedicated bandwidth between you Physical transfer of media into and Shrink-wrapped gateway for volume site and AWS out of AWS synchronization
  21. 21. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  22. 22. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  23. 23. Curiosity
  24. 24. Curiosity The mars.jpl.nasa.gov website is based on the open-source Content Management System (CMS) Railo, running on Amazon EC2 Shared storage for Railo is provided by Amazon EC2 instances running Gluster on a pool of Amazon Elastic Block Store (EBS) volumes for consistently high performance disk I/O.
  25. 25. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  26. 26. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance as a service 3 Scalability
  27. 27. Database services RDS, SimpleDB, DynamoDBRelational Database SimpleDB DynamoDB Schemaless NoSQL, Schemaless, Service Smaller datasets Provisioned throughput Fully managed database database (MySQL, Oracle, MSSQL)
  28. 28. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance as a service 3 Scalability
  29. 29. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage
  30. 30. Web accessible S3 storage…
  31. 31. You put in it S3AWS stores with 99.999999999% durability
  32. 32. Highly scalable web access to objects You put in it S3AWS stores with 99.999999999% durability Multiple redundant copies in a region
  33. 33. “Spotify needed a storage solution that could scale very quickly without incurringlong lead times for upgrades. This led us to cloud storage, and in that market, Amazon Simple Storage Service (Amazon S3) is the most mature large-scale product. Amazon S3 gives us confidence in ourability to expand storage quickly while also providing high data durability.” Emil Fredriksson, Operations Director
  34. 34. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage
  35. 35. Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data Cold System imagesaccessible file applications and distribution storage & Database Disks systems data DR Durable mediaarchiving backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 5 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage
  36. 36. What we heard from youYou love Amazon S3 for its simplicity, security, durability, and performance.
  37. 37. What we heard from youYou love Amazon S3 You wanted a highly for its simplicity, secure, extremely security, durability, durable, and extremely and performance. cost effective option for archiving data for years
  38. 38. The need… Reliable and cheap storage of data Data with long Multi-PB, infrequentlyretention periods accessed data sets
  39. 39. spectrumdata.com.au
  40. 40. Our goals with Glacier…Redefine data archiving Replace physical media for and backup: archiving: no upfront payments an easy to use storage service that is a very low price for storage infinitely scalableability to scale up and down as a secure service for important data needed assets designed for an annual average 99.999999999% durability per saved object for as little as $0.01 per gigabyte per month
  41. 41. The solution…Reliable and cheap storage of data
  42. 42. The solution… Reliable and cheap storage of data Same storagedurability mechanisms as S3
  43. 43. The solution… Reliable and cheap storage of data Same storage Trade-off on retrievaldurability mechanisms time as S3
  44. 44. The solution… Reliable and cheap storage of data Same storage Trade-off on retrievaldurability mechanisms time as S3
  45. 45. 3-5 hour retrieval time We assume you won’t access often
  46. 46. Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data
  47. 47. Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data Libraries, historical societies, non-profit organizations andDigital preservation governments are increasing their efforts to preserve valuable but aging digital content such as websites, software source code, video games, user-generated content and other digital artifacts
  48. 48. Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data Libraries, historical societies, non-profit organizations andDigital preservation governments are increasing their efforts to preserve valuable but aging digital content such as websites, software source code, video games, user-generated content and other digital artifacts Amazon Glacier is cost competitive, even at scale, andTape replacement eliminates pain points like capacity planning, capital budgeting and investments, media formats, hardware refreshes, and off-site storage costs, shipping and retrieving
  49. 49. Good reasons to replace off-site tape archives 100% restore success rate – no broken or missing tapes No lost tapes and improved security posture No device or media admin or handling No capacity planning Pay as you go No need for recurrent and risky data migrations
  50. 50. S3 GlacierBucket VaultObject Archive
  51. 51. Create vault supported via console
  52. 52. What is an archive?Any object, such as a photo, video, document or compressed collection It is a base unit of storage in Amazon Glacier Upload an archive in a single request For large archives use multipart upload API
  53. 53. API credentials Glacier client (keys) Region endpointclient = new AmazonGlacierClient(credentials);client.setEndpoint("https://glacier.us-east-1.amazonaws.com/");ArchiveTransferManager atm = new ArchiveTransferManager(client, credentials);UploadResult result = atm.upload(vaultName, ”MyArc “, new File(archiveToUpload)); Transfer manager File to upload Vault & archive name Java
  54. 54. Transfer manager Region endpointvar manager = newArchiveTransferManager(Amazon.RegionEndpoint.USEast1);string archiveId = manager.Upload(vaultName, ”MyArchive",archiveToUpload).ArchiveId; Vault & archive File to upload name .net
  55. 55. Retrieval S3 GlacierSynchronous Asynchronous Immediate 3-5 hours
  56. 56. Retrieval1. Initiate a retrieval job2. After the job completes, download the bytes
  57. 57. Initiate job JobParameters jobParameters = new JobParameters()Glacier .withArchiveId("*** provide an archive id ***") client .withDescription("archive retrieval") .withType("archive-retrieval"); InitiateJobResult initiateJobResult = client.initiateJob(new InitiateJobRequest() .withJobParameters(jobParameters) .withVaultName(vaultName)); String jobId = initiateJobResult.getJobId(); JobID to track Java
  58. 58. Track job After 3-5 hours:1. SNS topic notification2. Call describeJob Using JobID
  59. 59. API credentials Download job Glacier client (keys) Region endpoint client = new AmazonGlacierClient(credentials); client.setEndpoint("https://glacier.us-east-1.amazonaws.com/"); ArchiveTransferManager atm = new ArchiveTransferManager(client, credentials); atm.download(vaultName, archiveId, new File(downloadFilePath));Transfer manager Download path Vault name & archive id Java
  60. 60. Download jobvar manager = new ArchiveTransferManager(Amazon.RegionEndpoint.USEast1);var options = new DownloadOptions();options.StreamTransferProgress += ArchiveDownloadHighLevel.progress;manager.Download(vaultName, archiveId, downloadFilePath, options); static int currentPercentage = -1; static void progress(object sender, StreamTransferProgressArgs args) { if (args.PercentDone != currentPercentage) { currentPercentage = args.PercentDone; Console.WriteLine("Downloaded {0}%", args.PercentDone); } } .net
  61. 61. “Every day our genome sequencers produceterabytes of data. As our company moves into the clinical space, we face a legal requirement to archive patient data for years that would drastically raise the cost of storage. Thanks to Amazon Glacier’s secure and scalable solution, we will be able to provide cost-effective, long-term storage and therebyeliminate a barrier to providing whole genome sequencing for medical treatment of cancer and other genetic diseases.”
  62. 62. “An organization like ours thinks in centuries when it comes to content retention, and long term preservation of our Master Archives is a critical part our mission here at NYPR. Storing these core assets on traditional mediasuch as local disk and off-site tape exposes us to corruption and even outright-loss of data. We are excited to move our archives to Amazon Glacier, which will be a better long-term solution.” Steve Shultis, CTO, New York Public Radio
  63. 63. Desktop clientsS3 integration coming soon Use Glacier through S3 APIs Policy based tiered storage
  64. 64. Pricing
  65. 65. Storage Retrievals Data In Data OutFrom $0.1 per GB Free up to 5% of Free Tiered (1st GB free) average monthly storage the tiered fees
  66. 66. Storage Retrievals Data In Data OutFrom $0.1 per GB Free up to 5% of Free Tiered (1st GB free) average monthly storage the tiered fees Anticipation is archives will be accessed infrequently Storage is cheap, trade-off on retrieval pricing
  67. 67. Benefits of Amazon Glacier Low cost SecureAs little as $0.01/GB/month with no up-front capital Secure and durable technology platform withcommitments. industry-recognized certifications and audits. Durable SimpleAverage annual durability of 99.999999999% per Eliminate hardware, software, and capacityarchive. planning. Flexible Use multiple servicesAdd any amount of data, quickly. Easily expire and Easily leverage other AWS services once your data isdelete without handling media. in the AWS cloud.
  68. 68. http://aws.amazon.com/gla cier/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×