• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
AWS Sydney Summit 2013 - Understanding your AWS Storage Options
 

AWS Sydney Summit 2013 - Understanding your AWS Storage Options

on

  • 748 views

Session 1, Presentation 6 from the AWS Sydney Summit

Session 1, Presentation 6 from the AWS Sydney Summit

Statistics

Views

Total Views
748
Views on SlideShare
748
Embed Views
0

Actions

Likes
1
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Why are we going through the different storage solutions? Data comes in many shapes and sizes – this gives you different scale issues when you are looking at volumes of dataData also comes in a different rates – this is you a scale problem of how you ingest the data.It is important to understand what technology will meet your data needs, giving the right performance at the right price
  • One of the most common workloads in the mobile sector is multi-media.With the proliferation of smartphones, the amount of data that can be sent in simultaneously from a globally distributed network of phones can be huge. Both in total volume and ingest
  • It is not just the consumer segment that is producing masses of data. Oil and gas companies, such as our customer Shell for example, have embedded sensors all the way down the drill heads – this is producing huge amounts of data that needs to be ingested and stored
  • We are durable: One of its key value propositions is durability, 11 9sYour data is stored across three separate facilities giving you geo-redundancy and we can sustain data loss in two facilities simultaneously and your data is still safe.
  • We are BIG. We manage this level of durability across over 1 trilion unique customer objects. This does not include all the objects that AWS stores like snapshots.
  • We heard the feedback around S3, that it was a great product, but when you looked at infrequently accessed data that was to be archived for a long time, you wanted a cheaper solution
  • Glacier is 1c per GB per month…
  • That equates to $120 per TB/Year.
  • Whilst keeping the 11 x 9s of durability so your data is safe.Important to remember that Glacier is designed for the long term – if you store data on tape for 50 years, you will have to refresh the tapes every 5-10 years as otherwise the hardware will end of life.With Glacier we manage that for you.If you store data today, and retrieve it in 50 years, the chances are every part of hardware except for the concrete floor will have been swapped out
  • In contrast to S3, the data in Glacier needs 3-5 hoursSend in a request to retrieve data,We then notify you when it is there.Then you can download
  • Important to manage the retrieval costs. If your data is too hot, it will be more economic to store in S3You have 5% of your data per month (prorated to the day) available to retrieve.
  • Fractal pic under creative commons: http://www.flickr.com/photos/jayakody2000lk/7216399294/
  • TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
  • TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
  • TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
  • Different pricing around the globe.Also note RRS.This gives lower durability, but at a lower cost
  • Can also serve HTML and CSSS3 can be your web server for static content
  • If you have common content that is delivered to a high number of peopleCloudfront is our CDN. This can cache content locally, and the when you ask for the content you will be directed to the nearest POP
  • https://na3.salesforce.com/a0750000008gz0tAAA
  • One of our customers is Spotify, an on demand music service who adds over 20,000 music tracks a day and delivers that music to its customersCall out RRS in description
  • File systems have been the bedrock of computing for decades.
  • For more persistent data you need EBS
  • Each server comes with Ephemeral storage,However this cannot be transferred between instances, and is lost when the instance is disposed of.Great for temp data etc
  • You can therefore attach the drive to the instance
  • However with EBS, you can then reattach that EBS volume to a new instance. Your data is now persistent
  • You can also add multiple drives. Maybe you need it to segment your data – different DBs on different drives.Or you can get the OS to create a RAID drive and therefore see it as a single volume, which gives a larger max volume size, and can increase performance.
  • Allows you to create a volume of up to 1TB, that you can either use as a block device, or you can format for a file system
  • EBS is great for spiky loads, it will allow you to flex up. However for more consistent loads – ie databases you should look at PIOPs that allow you to configure drives of up to 2000IOPS
  • What is really important is to get point in time snapshots – this is your backup
  • You can take snapshots of your EBS drives to allow you to roll back to an older version if needed -all kept in S3
  • 00:47:00
  • Important requirement is for people to be able to extend to the cloud and sync data between the twoTalk about Storage Gateway and RMAN
  • TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
  • Storage GatewayDownload a VM image that runs on your environmentGive it some local storage to use as a cache and upload buffer.I can now sync the data into the cloud as EBS snapshots.Which means that you can either download the snapshot and recover form that, or attach it to an EC2 instanceAlso have a cached volume version to enable thin provisioning or departmental / remote office file-sharingRemove some workloads from expensive file systems to simple/scalable gateway
  • 00:47:00
  • Why are we going through the different storage solutions? Data comes in many shapes and sizes – this gives you different scale issues when you are looking at volumes of dataData also comes in a different rates – this is you a scale problem of how you ingest the data.It is important to understand what technology will meet your data needs, giving the right performance at the right price
  • It is all about choice.Pick the technology that delivers the right performance at the right price.
  • https://na3.salesforce.com/a0750000006xf87AAA
  • https://na3.salesforce.com/a0750000006xf9iAAA
  • Shazam at superbowl during the adverts managed to hit 500,000 write a second – maintaining that sub 10ms write time. They actually called us 3 days prior to the SuperBowl with the goal to make the impossible, possible and did so with great success
  • https://na3.salesforce.com/a0750000008fnIQAAY
  • https://na3.salesforce.com/a0750000008fnIQAAY
  • https://na3.salesforce.com/a0750000008hOUM

AWS Sydney Summit 2013 - Understanding your AWS Storage Options AWS Sydney Summit 2013 - Understanding your AWS Storage Options Presentation Transcript

  • Understanding AWS Storage OptionsJeff Putt, APAC Business Development Manager, AWS
  • AWS Storage OptionsObject Storage Block StorageConnect AWS Storage with On-Premises EnvironmentsFor NoSQL and Relational Database storage, catch the dedicated sessions+ +
  • We are sincerely eager to hearyour feedback on thispresentation and on re:Invent.Please fill out an evaluation formwhen you have a chance.We are constantly producing more data
  • We are sincerely eager to hearyour feedback on thispresentation and on re:Invent.Please fill out an evaluation formwhen you have a chance.From all types of industries
  • #1Object Storage
  • Amazon S3Simple Storage Service
  • 99.999999999%Durability
  • Over 1 TrillionUnique Customer Objects
  • 750,000+Peak Transactions Per Second
  • 1. High Durability2. Unlimited Capacity3. Low latency4. High Volume Static Websites5. Security & Compliance6. Delivery via CDNS3 gives you So you can buildMobile AppsMusic AppsVideo networksVideo AppsBackup AppsPhoto AppsStatic WebsitesAd networksAnything else you’d like!
  • Amazon GlacierLow-Cost Archiving Service
  • 1¢Per GB / Month
  • $120Per TB / Year
  • 99.999999999%Durability
  • 3−5 HoursData Retrieval
  • Storage Costsvs.Retrieval Costs
  • 1. High Durability2. Low cost3. 3-5 hour latencyGlacier gives you So you can• Archive files• Build Hierarchal StorageManagement in the cloud
  • Amazon S3 and Glacier can help you build…GamingCommerceDigital MediaDistributionWebsites Social MediaPhoto/VideoSharing andBackupBig Data
  • Store unlimited objects in Amazon S3 and Amazon GlacierS3 Glacier
  • Access both via SDKs and via third-party softwareS3 GlacierInternet WebServices APIHTTP(S)Via SDK Via 3rd parties
  • Or send S3 files directly to GlacierS3 Glacier
  • S3 storage costs as low as 4.7 cents per GB
  • Serve static content at high volumes
  • Deliver content with lower latency via Amazon CloudFront
  • Smugmug is home to millions of customers sharingBillions of photos
  • Adds over 20,000 tracks of music a day
  • Nasdaq FinQloud is a WORM compliant archive forBroker Dealer financial data
  • Paul PetersonVice President of Engineering
  • nearmap Aerial Imagerynearmap routinely captures hi-res aerial imagescovering 75% of Australia’s populationnearmap provides browser-based access to up-to-date and historical aerial imagery to thousands ofcompanies, large and small29
  • Clear30
  • CurrentImages are delivered within7-14 days, rather than 3-6months after flying6moTraditionalnearmapProcessing Time3mo 9mo 1yrBrisbane floods – nearmap was the only company to capture theflooded areas and publish within days31
  • Bundaberg Flood32
  • 33Bundaberg Flood
  • 34Bundaberg Flood : Before
  • 35Bundaberg Flood : After
  • 36Bundaberg Flood : Before
  • 37Bundaberg Flood : After
  • 38Bundaberg Flood : Before
  • 39Bundaberg Flood : After
  • nearmap ToolsThe photomap iscorrectly scaled foraccuratemeasurementsObjects are shown intheir correct real-world position44
  • Why Use nearmap• Less time driving• Less time on site• Reduce fleet costs and environmental impact• Eliminates existing spend on imagery• Integrates with enterprise systems45
  • nearmap and AWSQ. What does nearmap use AWS for?A. Everything• Production• Development and QA• Corporate systems• Image archival46
  • Live on AWS October-2012Transition went very smoothlyLocal AWS team provided fantastic supportAWS systems continue to work flawlessly47
  • AWS Features UsedEC2 - both Reserved and Spot InstancesS3GlacierRDSRoute53CloudFormationCloudWatchIAMSESDirect Connect7 May, 2013
  • Image Data Storage6 Terabytes of new image data per monthWe have 4+ years of existing image dataS3: 150 Terabytes and countingGlacier: 300 Terabytes and counting7 May, 2013
  • 50
  • #2Block Storage
  • Amazon EBSElastic Block Storage
  • Ephemeral storage is on every EC2 instance
  • EBS volumes attach to EC2 instances
  • EBS volumes persist and can be attached to other instances
  • Multiple drives on one Amazon EC2 Instance
  • 10GB1TB
  • IOPSProvisioned
  • Workloads on EBS Provisioned IOPSP-IOPS is designed to run transactional applications that require high andconsistent IO such as:• Relational databases• NoSQL databases (e.g. MongoDB)• High performance file systems• Productivity applications (e.g. Microsoft Exchange)• Enterprise applications (e.g. CRM, ERP)
  • EBSSnapshots
  • Take snapshots of EBS volumes and keep them in S3
  • Use Amazon EBS When You Need• Filesystem for an instance NTFS, ExtFS, RAID, LVM• Long-term persistent storage• Ability to change data frequently• Access to raw, unformatted block-level storage
  • #3Connect AWS Storage with On-PremisesEnvironments
  • Access both via SDKs and via third-party softwareS3 GlacierInternet WebServices APIHTTP(S)Via SDK Via 3rd parties
  • Common Data Storage Challenges and the AWS OpportunityInternet WebServices APIHTTP(S)AWS DirectConnectBlock FileCustomerDataCenterStorageUse CasesArchive BackupDisasterRecoveryNextGenerationEnterpriseStorageS3 Glacier
  • Common Data Storage Challenges and the AWS OpportunityPrimaryBlockStoragePrimaryFileStorageArchivalStorageDisk-BasedBackupStorageTapeInfrastructure&ManagementReplicatedStorage forDisasterRecoveryOffsiteLocationsGeo-ResilienceNext Generation: Gateway & Cloud StorageReduceSANFootprintReduceNASFootprintEliminateArchitectureHardware& SoftwareEliminateBackupHardware& SoftwareEliminateTapeRecover inCloud orany DataCenterEliminateSecondaryDataCentersReduce Complexity. Reduce Cost.NextGenerationEnterpriseStorage-Benefits
  • Common Data Storage Challenges and the AWS OpportunityS3 & Glacier - Designed to work with other AWS Services
  • Storage Gateway – Connect On-Prem with the AWS Cloud1. Local, low-latency access to themost frequently used files whilestoring all data in Amazon S3(Cached-Volumes)Or2. Scheduled off-site backups toAmazon S3 for on-premises data(Stored-Volumes)
  • Use AWS Storage Gateway When You Need To• Backup your data• Synchronize data• Export data for migration• Set up a departmental fileshare• Conduct simple disaster recovery
  • AWS Storage OptionsObject Storage Block StorageConnect AWS Storage with On-Premises Environments+ +
  • Performance-OrientedCost-OrientedIT’S ALL ABOUTCHOICE
  • Related ContentWeb resources:• http://aws.amazon.com/s3/• http://aws.amazon.com/ebs• http://aws.amazon.com/glacier/• http://aws.amazon.com/digital-media/72
  • Thank You.
  • Shares customer submitted photos
  • Shazam hit 500,000 writes per secondduring the Super Bowl
  • Foursquare streams hundreds of millions of application logsinto Amazon S3 for analytics
  • Forced to choose between saving their source data or savingtheir results. With Amazon Glacier, they can keep both
  • Real Clear Politics served up 20x normal demand withS3 Websites
  • Understanding AWS Storage Options