Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deep Dive on AWS Cloud Data Migration Services

1,350 views

Published on

When evaluating and planning migrating your data from on premises to the Cloud, you might encounter physical limitations. Amazon offers a suite of tools to help you surmount these limitations by moving data using networks, roads, and technology partners. In this session, we discuss how to move large amounts of data into and out of the Cloud in batches, increments, and streams.

Published in: Technology
  • Be the first to comment

Deep Dive on AWS Cloud Data Migration Services

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Data Transfer Services Data Ingest Strategies into the AWS Cloud Ken Chan Product Business Development Manager Greater China kenchan@amazon.com
  2. 2. Storage is the gravity for cloud applications
  3. 3. Customers have Data Centers Depl oyDeploy
  4. 4. Batches and Streams Direct Connect Snowball, Snowball Edge, Snowmobile 3rd Party Connectors Transfer Acceleration Storage Gateway Kinesis Firehose File Amazon EFS Block Amazon EBS (persistent) Object Amazon GlacierAmazon S3 Amazon EC2 Instance Store (ephemeral) Internet/VPN CloudFront
  5. 5. Long Term Archive All tiers accessible through a single API Oldest content trickles down to glacier automatically to save cost Amazon S3 S3 In-frequent Access Amazon Glacier LifeCyclePolicies Onsite Storage Frequently Accessed New S3 Life Cycle Policy Use case for Media
  6. 6. Internet/VPN ingest
  7. 7. What is Internet/VPN? Globally available Default method of ingesting content into Amazon S3 Simple standards-based (HTTP) connection Use your existing internet connection Available in a VPC for VPN connectivity Acceleration through multipart upload Data transfer into AWS is free VPN connections using VPC virtual private gateway •$0.05 per VPN connection-hour •$0.048 per VPN connection-hour for connections to the Tokyo region
  8. 8. How does Internet/VPN ingest work? Accelerate data transfer using multipart upload Ingest data directly into S3 buckets with existing internet connectivity S3 bucket AWS Region and through the console or API customer gateway endpoints VPN connection Internet Internet through VPN + VPC
  9. 9. Amazon S3 Transfer Acceleration
  10. 10. What is Transfer Acceleration? Network- and protocol-based data transfer service Acceleration of data ingress/egress with S3 buckets Typically 50% to 300% faster Feature of S3 enabled at the bucket level Available in all S3 regions worldwide No client/server software required No code changes to your application No firewall exceptions Simple pricing model
  11. 11. Ingest & egress with Transfer Acceleration S3 bucket AWS edge location Uploader Optimized throughput! Uses AWS 59 global edge locations AWS determines best edge location Data transfer optimized between edge and customer, and edge and S3 Data is not stored on the edge cache
  12. 12. Customers: Frame.io, Hudl, Viocorp Problem Statement: • Needed to accelerate customer content ingest into their respective applications running on AWS • Existing ingest options were proprietary and too expensive Use of AWS: • S3 and S3 transfer acceleration for massively scalable ingest • S3 for storage, CloudFront and S3 transfer acceleration for ingest Business Benefits: • Global highly distributed data transport available on demand • Massive scalability and elasticity • Lower TCO for storage and data transport infrastructure Accelerating media content uploads to their platforms S3 Bucket AWS Edge Location Uploader Optimized Throughput!
  13. 13. Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from these edge locations to a bucket in Singapore Public internet How fast is S3 Transfer Acceleration? S3 transfer acceleration
  14. 14. AWS Direct Connect
  15. 15. What is AWS Direct Connect? Dedicated, 1 or 10 GE private pipes into AWS Create private (VPC) or public virtual interfaces to AWS Reduced data-out rates (data-in still free) Consistent network performance At least 1 location to each AWS region Option for redundant connections Uses BGP to exchange routing information over a VLAN
  16. 16. Getting Started with DX
  17. 17. Create Connection to issue LOA
  18. 18. Pass this LOA to our DX partner to get cross connection setup
  19. 19. At the Direct Connect location CORP AWS Direct Connect Routers Customer Router Colocation DX Location Customer network ` AWS backbone network Cross- connect Customer router Customer’s network Demarcation
  20. 20. Dedicated port through Direct Connect partner CORP AWS Direct Connect Routers Colocation DX Location Partner network AWS backbone network Cross- connect Customer router Partner network Access circuit Demarcation Partner equipment
  21. 21. Hybrid cloud storage expansion: Amazon EFS through Direct Connect “Bursting” File Workloads Data Migration into EFS Amazon EFSOn-Premises AWS Direct Connect
  22. 22. AWS Storage Gateway
  23. 23. What is AWS Storage Gateway? Works with your existing applications Secure and durable storage in AWS Low latency for frequently used data Scalable and cost-effective on-premises storage - $.01/GB written to AWS + S3/Amazon Glacier storage fees Service connecting an on-premises software appliance with cloud-based storage
  24. 24. Hybrid storage use cases and architectures for AWS Storage Gateway Enabling cloud workloads Move data to AWS storage for Big Data, cloud bursting, or migration Tiered cloud storage Easily add AWS storage to your on-premises environment Backup, archive, and disaster recovery Cost effective storage in AWS with local or cloud restore
  25. 25. Storage Gateway hybrid storage solutions Enables using standard storage protocols to access AWS storage services Customer Premises Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Enterprise storage Devices Application servers
  26. 26. Storage gateway – Files, volumes, and tapes File gateway NFS (v3 and v4.1) interface **NEW!** On-premises file storage backed by Amazon S3 objects Volume gateway iSCSI block interface On-premises block storage backed by Amazon S3 with EBS snapshots Tape gateway iSCSI virtual tape library (VTL) interface Virtual tape storage in Amazon S3 and Glacier with VTL management
  27. 27. Detail: AWS File Gateway for S3 NFS Interface ElasticityAmazon S3 Bucket Easy Integration Cloud ScaleCloud Access
  28. 28. AWS Snowball
  29. 29. What is AWS Snowball? Petabyte-scale data transport E-ink shipping label Ruggedized case “8.5G impact” All data encrypted end-to-end Rain- and dust- resistant Tamper-resistant case and electronics 80 TB 10 GE network
  30. 30. AWS storage migration expansion: AWS Snowball Transfer Capacity Integration Regional Availability 80TB model HDFS support 3rd party API HIPAA support Continue to expand
  31. 31. How it works
  32. 32. How fast is Snowball? • Less than 1 day to transfer 200TB via 3x10G connections with 3 Snowballs, less than 1 week including shipping • Number of days to transfer 200TB via the Internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 71 141 236 471 50% 36 71 118 236 75% 24 47 225 157
  33. 33. Customer: Scripps Networks Interactive Problem Statement: • Need storage platform to manage active archive content • Existing content repository too large to migrate via available network-based ingest methods Use of AWS: • S3 and Snowball for massively scalable ingest • S3 for storage, Glacier for content archive • Snowball to securely transport existing media content from on- premises storage and tape vault Business Benefits: • Petabyte-scale data transport without increased network costs • Massive scalability and elasticity • Lower TCO for active archive storage Active archive transport and archival for digital content provider
  34. 34. AWS storage migration expansion: AWS Snowmobile
  35. 35. Hybrid cloud storage expansion: AWS Snowball Edge On-premises Capacity On-premises Integration On-premises Compute Clustered local storage 100TB capacity NFS and S3-compatible endpoint AWS Lambda support for local transformation
  36. 36. AWS Snowball Edge Integrated Storage and Compute Applications STG214
  37. 37. Storage Ecosystem Partners
  38. 38. Backup to AWS approaches Amazon S3 Amazon Glacier AWS Direct Connect Internet Amazon S3-IA Application servers Cloud gateway Local disk Media server Cloud gateway Application servers Backup SW cloud connector Local disk Media server with cloud connector
  39. 39. Hybrid cloud storage ecosystem Backup AWS Storage Gateway VTL Direct to Amazon S3 File Systems Object Storage
  40. 40. Remember to complete your evaluations!
  41. 41. Thank you!

×