Deep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech Talks

2. Cloud Data Migration Direct Connect Snow* data transport family 3rd Party Connectors Transfer Acceleration Storage Gateway Kinesis Firehose AWS Storage Platform and SolutionsThe AWS Storage Portfolio Object Amazon GlacierAmazon S3 Block Amazon EBS (persistent) Amazon EC2 Instance Store (ephemeral) File Amazon EFS

3. Hybrid storage use cases and architectures Enabling cloud workloads Move data to AWS storage for Big Data, cloud bursting, or migration Tiered cloud storage Easily add AWS storage to your on-premises environment Backup, archive, and disaster recovery Cost effective storage in AWS with local or cloud restore

4. Storage Gateway hybrid storage solutions Enables using standard storage protocols to access AWS storage services AWS Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Files Volumes Tapes

5. Storage Gateway – Files, volumes, and tapes File gateway NFS (v3 and v4.1) interface On-premises file storage backed by Amazon S3 objects Tape gateway iSCSI virtual tape library interface Virtual tape storage in Amazon S3 and Glacier with VTL management Volume gateway iSCSI block interface On-premises block storage backed by S3 with EBS snapshots

6. Storage Gateway – Common capabilities Standard storage protocols integrate with on-premises applications Local caching for low-latency access to frequently used data Efficient data transfer with buffering and bandwidth management Native data storage in AWS Stateless virtual appliance for resiliency Integrated with AWS management and security

7. File gateway On-premises file storage maintained as objects in Amazon S3 Customer Premises File Gateway Data stored and retrieved from your S3 buckets One-to-one mapping from files-to-objects File metadata stored in object metadata Bucket access through IAM role you own and manage Use S3 Lifecycle Policies, versioning, or CRR to manage data GlacierS3 Standard S3 Standard - Infrequent Access HTTPS NFS v3 / v4.1 Application Server

8. Enabling cloud workloads Move data to AWS storage for Big Data, cloud bursting, or migration “Storage Gateway has the promise to transform the way we move data into the cloud. The NFS interface lets us easily integrate data files from analytical instruments, and the transparent S3 storage lets us easily connect our cloud-based applications and leverage the powerful storage capabilities of S3. With Storage Gateway, we can now unleash the full power of AWS on our instrument data.”

9. S3 Bucket Storage Gateway S3 Bucket Read-only NFS client Cross region replication NFS clientAWS Snowball S3 Bucket AWS SnowballS3 cross-region replication Storage Gateway Read-only NFS client Job GETs/PUTs objects RefreshCache Amazon EMR In-cloud workload S3 Bucket Storage Gateway RefreshCache RefreshCache File Gateway – CacheRefresh

10. Application Server Volume Gateway On-premises volume storage backed by Amazon S3 with EBS snapshots Block storage in S3 accessed via the volume gateway Data compressed in-transit and at-rest Backup on-premises volumes to EBS snapshots Create on-premises volumes from EBS snapshots Up to 1PB of total volume storage per gateway Amazon EBS snapshots Storage Gateway bucket in Amazon S3 Customer Premises Volume Gateway iSCSI HTTPS

11. Customer data center AWS Storage Gateway VM Volume Gateway – Stored Primary data stored on-premises Asynchronous upload to AWS Point-in-time backups stored as Amazon EBS snapshots Up to 32 volumes, up to 16 TB each, for up to 512 TB per gateway Amazon EBS snapshots Application server INITIATOR TARGET Upload Buffer Volume Storage AWS Storage Gateway service

12. Customer data center AWS Storage Gateway VM Volume Gateway – Cached Primary data stored in AWS Frequently accessed data cached on-premises Point-in-time backups stored as Amazon EBS snapshots Up to 32 volumes, up to 32 TB each, for up to 1 PB per gateway Amazon EBS snapshots Application server INITIATOR TARGET Upload Buffer Cache Storage Volume storage backed by Amazon S3 AWS Storage Gateway service

13. Tiered storage to S3, Glacier, and EBS Easily add AWS storage to your on-premises environment “Storage Gateway is at the core of our disaster recovery and business continuity (BCM) processes, handling our co-lo'd OLTP and OLAP off site data backups, as well as our in-office BCM. It works transparently, in a lights out way, archiving off to a separate AWS account with a simple grandfather-father- son snapshot plan in place”.

14. Tape gateway Virtual tape storage in Amazon S3 and Glacier with VTL management Virtual tape storage in S3 and Glacier accessed via tape gateway Data compressed in-transit and at-rest Up to 1 PB total tape storage per gateway, unlimited archive capacity Supports leading backup applications: Archived Tapes stored in Amazon Glacier MEDIA CHANGER TAPE DRIVE Customer Premises Tape Gateway Virtual Tapes stored in Amazon S3 Backup Server HTTPSiSCSI

15. Backup, archive, and disaster recovery Cost effective storage in AWS with local or cloud restore “Tapes are a headache, prone with hardware failures, offsite storage costs, and constant maintenance needs. Storage Gateway provided the most cost-effective and simple alternative. We even got disaster recovery by using a bi-coastal data center”.

16. Storage Gateway – Key Benefits Seamless integration across standard storage protocols Low-latency access Durability, cost, and elasticity of AWS Storage services Efficient data transfer Data encryption Integrated with AWS monitoring, management, and security

17. Storage Gateway – Pricing All gateway types $0.01 per GB of data written to AWS* File Files stored and billed by S3 Volume $0.023 per GB-month of volume data stored Snapshots stored and billed by EBS Tape $0.023 per GB-month of tape data stored $0.004 per GB-month of tape data archived $0.01 per GB of data retrieved from archive * Up to a maximum of $125/month. First 100GB per gateway free.

18. Under the hood

19. File Gateway – File system metadata File system metadata persisted in object user-metadata, eg. Configurable defaults for objects that don’t have this metadata • E.g. objects that were already in the bucket Changing file metadata copies the object x-amz-meta-file-permissions: 0666 x-amz-meta-file-user-class: 4321 x-amz-meta-file-group-class: 42 x-amz-meta-file-created: 2016-10-05T20:08:45+00:00 x-amz-meta-file-last-modified: 2016-10-05T20:08:45+00:00

20. How do I monitor cache performance? CloudWatch metrics metrics for gateway CachePercentUsed CacheHitPercent CachePercentDirty High CachePercentUsed is good Once 100% will start to impact CacheHitPercent May indicate the cache is too small for the working set Writes increase CachePercentDirty Reduces size of working cache, and can reduce CacheHitPercent May indicate data not uploading to AWS quickly enough

21. How does Storage Gateway transfer data to AWS? Dirty data asynchronously uploaded to AWS Byte-level parallel upload and download Compression for tape and volume Uploaded data committed to storage resource periodically File uses multi-part PUT Tape and volume create periodic recovery points (internal snapshots) Rate of writes by application determine periodicity of commit High write rates will commit more frequently Low write rates will commit based on timer

22. How do I monitor data transfer? CloudWatch metrics for gateway and per storage resource For volumes TimeSinceLastRecoveryPoint indicates how long since the last commit Application server Storage Gateway WriteBytes ReadBytes CloudBytesUploaded CloudBytesDownloaded

23. Monitoring and Security extended to on- premises gateways IAM Integration for File Gateway rolesCloudTrail Logging of API Calls

24. Getting Started AWS Storage Gateway home page: aws.amazon.com/storagegateway AWS Storage Gateway documentation: aws.amazon.com/documentation/storage-gateway/ Ready to get started? AWS Storage Gateway console: console.aws.amazon.com/storagegateway/home Storage Gateway

25. Thanks! Simple, Secure, Cost-effective Hybrid Storage in AWS