AWS Update | London - Amazon Glacier

Amazon Web Services Update | LondonNovember 2012

Getting to Glacier…Why AWS for storage & archive?AWS fundamental servicesStorage & archive – examples &patternsAmazon Glacier

Storage & Archive AWS is used in a variety of ways…Powers applications that allows customers to access historical Store its vast repository of music to stock price information feed to over 15 million active users Estimates it has saved $500,000 Digital assets and usage data behind in storage expenditures and cut publication sites and mobileits disk storage array costs in half applications

Business & technical drivers You might be able to: Reduce costs Reduce on-premiseSlash storage & archive budgets Eliminate on premise equipment to manage archives Change processes Remove aging technologiesRemove the need to do capacity Eliminate tape for backup and archive planning

Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramaticallySlash storage & archive budgets by Eliminate on premise equipment to increasing scalability up to 50% manage archivesEliminate the need for secondary sites Change processes Remove aging technologies Remove the need to do capacity Eliminate tape for backup and archive planning

Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate tape for backup and archive planning

Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate capacity planning Eliminate tape for backup and archive planning Eliminate provisioning for peak demand

Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate capacity planning Eliminate tape for backup and planning Remove tape archives Eliminate provisioning for peak Cycle out aging disk arrays demand

AWS fundamental services

Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities

Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities Very fast Fast web object Slow, rare access ‘instance’ disks storage

Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities

Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Archive Backup Simple Storage Service DR GlacierHigh performance block storage device Highly scalable object storage Long term object archive Data1TB in size 1GB to accessed Snapshots 1 byte to 5TB in size Extremely low cost per gigabyte Rapid RTO ~>10% / monthAmazo as drives to instances with Mount 99.999999999% durability 99.999999999% durability Shorter term data nsnapshot/cloning functionalities backup with rapid S3 Expiration policies 11 9s durability RTOAmazo Lower cost when 11 n S3 9s not required Lower cost Lower cost RRS Long termAmazo archiving Use policies to Retain “write once - move cold backup read never” copy in n Infrequent data data for long term case of worst caseGlacier access (~<10% retention scenario data/month)

Use case journey On-premise On-instance Object level Long term Locallyaccessible file systems Workloadswith local data

Use case journey On-premise On-instance Object level Long term Locallyaccessible file systems AWS Workloadswith local data

Use case journey On-premise On-instance Object level Long term Locally EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data

Use case journey On-premise On-instance Object level Long term Locally EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability

Getting data into the cloud Direct connect, import/export and storage gatewayAWS Direct Connect AWS Import/Export Amazon Storage GatewayDedicated bandwidth between you Physical transfer of media into and Shrink-wrapped gateway for volume site and AWS out of AWS synchronization

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability

Curiosity

Curiosity The mars.jpl.nasa.gov website is based on the open-source Content Management System (CMS) Railo, running on Amazon EC2 Shared storage for Railo is provided by Amazon EC2 instances running Gluster on a pool of Amazon Elastic Block Store (EBS) volumes for consistently high performance disk I/O.

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance as a service 3 Scalability

Database services RDS, SimpleDB, DynamoDBRelational Database SimpleDB DynamoDB Schemaless NoSQL, Schemaless, Service Smaller datasets Provisioned throughput Fully managed database database (MySQL, Oracle, MSSQL)

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance as a service 3 Scalability

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage

Web accessible S3 storage…

You put in it S3AWS stores with 99.999999999% durability

Highly scalable web access to objects You put in it S3AWS stores with 99.999999999% durability Multiple redundant copies in a region

“Spotify needed a storage solution that could scale very quickly without incurringlong lead times for upgrades. This led us to cloud storage, and in that market, Amazon Simple Storage Service (Amazon S3) is the most mature large-scale product. Amazon S3 gives us confidence in ourability to expand storage quickly while also providing high data durability.” Emil Fredriksson, Operations Director

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage

Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data Cold System imagesaccessible file applications and distribution storage & Database Disks systems data DR Durable mediaarchiving backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 5 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage

What we heard from youYou love Amazon S3 for its simplicity, security, durability, and performance.

What we heard from youYou love Amazon S3 You wanted a highly for its simplicity, secure, extremely security, durability, durable, and extremely and performance. cost effective option for archiving data for years

The need… Reliable and cheap storage of data Data with long Multi-PB, infrequentlyretention periods accessed data sets

spectrumdata.com.au

Our goals with Glacier…Redefine data archiving Replace physical media for and backup: archiving: no upfront payments an easy to use storage service that is a very low price for storage infinitely scalableability to scale up and down as a secure service for important data needed assets designed for an annual average 99.999999999% durability per saved object for as little as $0.01 per gigabyte per month

The solution…Reliable and cheap storage of data

The solution… Reliable and cheap storage of data Same storagedurability mechanisms as S3

The solution… Reliable and cheap storage of data Same storage Trade-off on retrievaldurability mechanisms time as S3

3-5 hour retrieval time We assume you won’t access often

Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data

Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data Libraries, historical societies, non-profit organizations andDigital preservation governments are increasing their efforts to preserve valuable but aging digital content such as websites, software source code, video games, user-generated content and other digital artifacts

Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data Libraries, historical societies, non-profit organizations andDigital preservation governments are increasing their efforts to preserve valuable but aging digital content such as websites, software source code, video games, user-generated content and other digital artifacts Amazon Glacier is cost competitive, even at scale, andTape replacement eliminates pain points like capacity planning, capital budgeting and investments, media formats, hardware refreshes, and off-site storage costs, shipping and retrieving

Good reasons to replace off-site tape archives 100% restore success rate – no broken or missing tapes No lost tapes and improved security posture No device or media admin or handling No capacity planning Pay as you go No need for recurrent and risky data migrations

S3 GlacierBucket VaultObject Archive

Create vault supported via console

What is an archive?Any object, such as a photo, video, document or compressed collection It is a base unit of storage in Amazon Glacier Upload an archive in a single request For large archives use multipart upload API

API credentials Glacier client (keys) Region endpointclient = new AmazonGlacierClient(credentials);client.setEndpoint("https://glacier.us-east-1.amazonaws.com/");ArchiveTransferManager atm = new ArchiveTransferManager(client, credentials);UploadResult result = atm.upload(vaultName, ”MyArc “, new File(archiveToUpload)); Transfer manager File to upload Vault & archive name Java

Transfer manager Region endpointvar manager = newArchiveTransferManager(Amazon.RegionEndpoint.USEast1);string archiveId = manager.Upload(vaultName, ”MyArchive",archiveToUpload).ArchiveId; Vault & archive File to upload name .net

Retrieval S3 GlacierSynchronous Asynchronous Immediate 3-5 hours

Retrieval1. Initiate a retrieval job2. After the job completes, download the bytes

Initiate job JobParameters jobParameters = new JobParameters()Glacier .withArchiveId("*** provide an archive id ***") client .withDescription("archive retrieval") .withType("archive-retrieval"); InitiateJobResult initiateJobResult = client.initiateJob(new InitiateJobRequest() .withJobParameters(jobParameters) .withVaultName(vaultName)); String jobId = initiateJobResult.getJobId(); JobID to track Java

Track job After 3-5 hours:1. SNS topic notification2. Call describeJob Using JobID

API credentials Download job Glacier client (keys) Region endpoint client = new AmazonGlacierClient(credentials); client.setEndpoint("https://glacier.us-east-1.amazonaws.com/"); ArchiveTransferManager atm = new ArchiveTransferManager(client, credentials); atm.download(vaultName, archiveId, new File(downloadFilePath));Transfer manager Download path Vault name & archive id Java

Download jobvar manager = new ArchiveTransferManager(Amazon.RegionEndpoint.USEast1);var options = new DownloadOptions();options.StreamTransferProgress += ArchiveDownloadHighLevel.progress;manager.Download(vaultName, archiveId, downloadFilePath, options); static int currentPercentage = -1; static void progress(object sender, StreamTransferProgressArgs args) { if (args.PercentDone != currentPercentage) { currentPercentage = args.PercentDone; Console.WriteLine("Downloaded {0}%", args.PercentDone); } } .net

“Every day our genome sequencers produceterabytes of data. As our company moves into the clinical space, we face a legal requirement to archive patient data for years that would drastically raise the cost of storage. Thanks to Amazon Glacier’s secure and scalable solution, we will be able to provide cost-effective, long-term storage and therebyeliminate a barrier to providing whole genome sequencing for medical treatment of cancer and other genetic diseases.”

“An organization like ours thinks in centuries when it comes to content retention, and long term preservation of our Master Archives is a critical part our mission here at NYPR. Storing these core assets on traditional mediasuch as local disk and off-site tape exposes us to corruption and even outright-loss of data. We are excited to move our archives to Amazon Glacier, which will be a better long-term solution.” Steve Shultis, CTO, New York Public Radio

Desktop clientsS3 integration coming soon Use Glacier through S3 APIs Policy based tiered storage

Pricing

Storage Retrievals Data In Data OutFrom $0.1 per GB Free up to 5% of Free Tiered (1st GB free) average monthly storage the tiered fees

Storage Retrievals Data In Data OutFrom $0.1 per GB Free up to 5% of Free Tiered (1st GB free) average monthly storage the tiered fees Anticipation is archives will be accessed infrequently Storage is cheap, trade-off on retrieval pricing

Benefits of Amazon Glacier Low cost SecureAs little as $0.01/GB/month with no up-front capital Secure and durable technology platform withcommitments. industry-recognized certifications and audits. Durable SimpleAverage annual durability of 99.999999999% per Eliminate hardware, software, and capacityarchive. planning. Flexible Use multiple servicesAdd any amount of data, quickly. Easily expire and Easily leverage other AWS services once your data isdelete without handling media. in the AWS cloud.

http://aws.amazon.com/gla cier/

AWS Update | London - Amazon Glacier

by Amazon Web Services on Nov 08, 2012

Accessibility

Upload Details

Usage Rights

2 Embeds 190

Statistics

AWS Update | London - Amazon Glacier Presentation Transcript

http://www.scoop.it	189
http://webcache.googleusercontent.com	1