Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Migration Using AWS Snowball, Snowball Edge & Snowmobile

5,493 views

Published on

by Rashim Gupta, Principal Product Manager, Storage Service, AWS

  • Gym guru exposes fat burning secret! I'm going to reveal to you the secret method that allows you to get the equivalent exercise of 45 minutes at the gym, in just a few minutes per day! Just watch this short video explaining how it works, JUST CLICK HERE ▲▲▲ http://ishbv.com/1minweight/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Have you ever heard of taking paid surveys on the internet before? We have one right now that pays $50, and takes less than 10 minutes! If you want to take it, here is your personal link ▲▲▲ http://ishbv.com/surveys6/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Data Migration Using AWS Snowball, Snowball Edge & Snowmobile

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Rashim Gupta, Principal Product Manager, Storage Services March 2017 Data migration using AWS Snowball, Snowball Edge & Snowmobile
  2. 2. Data Migration, Batch and Stream Processing Direct Connect Snowball, Snowball Edge, Snowmobile 3rd Party Connectors Transfer Acceleration Storage Gateway Kinesis Firehose File Amazon EFS Block Amazon EBS (persistent) Object Amazon GlacierAmazon S3 Amazon EC2 Instance Store (ephemeral)
  3. 3. Why are we here? Large Customer: “We want to move 250 PBs into AWS, but it would take years to do that over the network. How can we upload our data faster into AWS?”
  4. 4. Why are we here? Data is growing at a tremendous pace! 4k/8k Genomics Seismic Financial Logs IoT 100TB over the Internet: 2 years 1PB over the internet: 22 years 100PB over T3: 609 years
  5. 5. AWS Snow Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Compute & Storage for Hybrid/Edge workloads Exabyte-scale data migration
  6. 6. AWS Snow Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Compute & Storage for Hybrid/Edge workloads Exabyte-scale data migration
  7. 7. The AWS data transport solution AWS Snowball—petabyte scale data transport • Rugged 8.5G impact case • Rain and dust resistant • Data encryption end-to-end • 50 TB capacity/10G network • E-ink label E-ink shipping label RE:INVENT 2015 LAUNCH
  8. 8. Snowball features released post re:Invent 2015 • Data export • 80 TB Snowball • Snowball SDK • S3 Adaptor • HDFS import • HIPAA eligibility
  9. 9. How is my data transported securely? • 256-bit encryption • Data is encrypted by the Snowball client before it is written to the Snowball appliance • Keys are managed by AWS and are never written to the Snowball
  10. 10. How is my data transported securely? • Strong chain of custody • Tamper-resistant case • Tamper-resistant electronics (TPM) • Each Snowball is erased according to NIST 800-88 media sanitization guidelines between every job
  11. 11. How Snowball moves data into and out of AWS Create a job Connect the Snowball Copy data to the Snowball Your data moved to Amazon S3 In transit to you Delivered to you Delivered to AWS At AWS Job created Job completed
  12. 12. Snowball to date SPEED COSTLOGISTICS SECURITY • Circled the earth 100x • 63 total launches • 10 AWS Regions 100x
  13. 13. When to use AWS Snowball Cloud Migration Disaster Recovery Datacenter Decommission Content Distribution
  14. 14. Customer use case: S3 as a data lake Amazon EMR Amazon Kinesis Amazon Redshift Amazon DynamoDB Amazon RDS AWS Data Pipeline Spark StreamingCassandra Storm Amazon S3
  15. 15. Case Study: Oregon State University Use-case: • Collect and analyze oceanic and coastal images • 60TB of data per week • Environmental and ocean ecosystem research Architecture before Snowball: • Transferred data with many small hard drives • Used to take weeks to months to upload data • $4MM+ in infrastructure investment • Expensive and inefficient Snowball lets OSU migrate TBs of data in days at a fraction of the cost
  16. 16. What does Snowball cost? • Per Job shipping - $250 / Job plus shipping (80TB) - $200 / Job plus shipping (50TB) • Includes 10 days to fill the device at your site • $15/day after the tenth day on site • Standard S3 charges apply • $0.00/GB to transfer data in
  17. 17. AWS Snow Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Compute & Storage for Hybrid/Edge workloads Exabyte-scale data migration
  18. 18. AWS Snowball Edge Petabyte-scale hybrid device with onboard compute and storage • 100 TB local storage • Local compute equivalent to an Amazon EC2 m4.4xlarge instance • 10GBase-T, 10/25Gb SFP28, and 40Gb QSFP+ networking • Ruggedized and rack-mountable RE:INVENT 2016 LAUNCH
  19. 19. Snowball Edge key features S3-compatible endpoint File interface (NFS) Clustering Run AWS Lambda functions Faster data transfer Encryption
  20. 20. Hybrid capabilities beyond data migrationMIGRATION Create job Copy data, Ship to AWS Moved to S3 Collect & Process data PROCESSING Create job Ship to AWS Moved to S3
  21. 21. Snowball Edge can… Extend of your data center Process data Expedite move Encrypted, secure, and embedded compute Write data directly as data is generated Offers a fast and cost effective way to ensure data can be quickly transferred to and from the cloud Simplify data transfer Uses standard and familiar tools for the data transfer process
  22. 22. Snowball Edge applicability • Incorporate Snowball Edge into existing applications, i.e. • Backup destination • File/object store • Data cluster • … • Build new, end to end applications with Snowball Edge • In the manufacturing plant • In the hospital • In automobiles • …
  23. 23. Snowball Edge use cases Offline Staging Local Tiering and Compute IoT Local Transformation
  24. 24. Aviation •Use Cases: - Planes equipped with cameras - Fly over regions to build 3D models - Run algorithms on content for data mining •Current solution: - Cameras stream data to massive HDDs which are shipped to DC •How Snowball helps? - Migrate directly to AWS •How Snowball Edge helps? - Identify operational failures in trips for immediate action
  25. 25. Healthcare Industry  Use Cases: - Typical hospital's data repository can exceed 100s of TBs - Expensive (time and $$$) to transfer data to AWS - How to integrate with solutions like Philips HeathSuite to provide healthcare professionals with data quickly  How Snowball helps? - Can transfer data in days instead of weeks - Can easily scale to hundreds of hospitals - Meets HIPAA requirements  How Snowball Edge helps? - While data is being stored, can we detect anomalies to instantly raise alerts?
  26. 26. Oil & Gas Ben Wilson (CTO of GE Oil and Gas): “PIGs and Snowballs—a match made in heaven! AWS Snowball 25 TB of pipeline PIG data to be managed at AWS. That is our GE PIG we pulled some of the data from. It’s always fun to try new AWS features and try to break them!” •Use Cases: - PIG is a device to inspect cracks in pipelines using sensors - Device fits inside pipeline and sent through to measure quality •How Snowball helps? - Use Snowball to migrate data to AWS
  27. 27. Security • Use Cases: - Biometric system uses face detection for security systems - Replacing security guards with automated security systems - Dozens of video cameras recording 8-10 hours/day - Tens of thousands of videos a week • How Snowball helps? - When video is recorded, goes to local storage and then shipped to AWS • How Snowball Edge helps? - Do lot of image processing; Can use Lambda to do on premise processing
  28. 28. Demo
  29. 29. Getting started with Snowball Edge Step 1: Create a Job and wait for Snowball to ship to you
  30. 30. Getting started with Snowball Edge Step 2: Power it on and connect to network Step 3: Install the Snowball Edge Client Software Step 4: Note the IP address on the Kindle Fire of device Step 5: Get the Manifest file and Unlock code from AWS Console
  31. 31. Getting started with Snowball Edge Step 6: Unlock the device snowballEdge unlock -i 192.168.1.141 -m C:sfloft.bin -u 906d9- 960bf-3e3f2-99aa2-79bf1 Step 7: Check the status of the Snowball snowballEdge status -i 192.168.1.141 -m C:sfloft.bin -u 906d9- 960bf-3e3f2-99aa2-79bf1 Step 8: Get the S3 credentials snowballEdge credentials -i 192.168.1.141 -m C:sfloft.bin -u 906d9-960bf-3e3f2-99aa2-79bf1
  32. 32. S3 operations You can use AWS CLI for S3 commands: • Store the S3 credentials under .aws/credentials • aws s3 ls s3://rashimgcmh --endpoint http://192.168.1.141:8080 • aws s3 cp c:bar.txt s3://rashimgcmh --endpoint http://192.168.1.141:8080
  33. 33. Using CloudBerry with Snowball
  34. 34. Third-party tools and gateways •Consumer grade: less than $50 • Example: Cloudberry, Arq (Haystack Software) •Small / medium business: $500 - $1,000 • Example: Synology, Veeam, QNap •Enterprise grade gateway (price varies) • Example: NetApp AltaVault, Commvault, TSM, EMC Networker, BackupExec
  35. 35. Solve Seeding Headaches with NetApp ® AltaVaultTM and AWS Snowball Primary Storage Cloud-Integrated Storage AltaVault™ 3rd Party Backup and Recovery Solution Encryption De-duplication Compression NetApp® AltaVault TM AWS Snowball
  36. 36. What does Snowball Edge cost? • Per Job shipping - $300 / Job plus shipping • Includes 10 days to fill the device at your site • $30/day after the tenth day on site • Standard S3 charges apply • $0.00/GB to transfer data in
  37. 37. AWS Snow Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Compute & Storage for Hybrid/Edge workloads Exabyte-scale data migration
  38. 38. Storage migration for exabytes: AWS Snowmobile
  39. 39. What • 10-100PB in a 45 foot-long ruggedized container truck Where & When • Can be made available in all AWS regions How • Data transferred via multiple 40Gbps interfaces up to 1Tb/s (100PB in a few weeks) • Appears as HTTP end point • End-to-end secure (256-bit encryption) • Customer orders a Snowmobile, we dispatch it to their site, they hook it up and fill it, it returns How much does it cost • $0.005/GB/mo based on provisioned capacity (from site departure to AWS ingestion completion)
  40. 40. Snowmobile key features Massively Scalable Customized for your needs Faster data transfer Strong Encryption Rugged, Durable, and Secure
  41. 41. Snowmobile Case Study: DigitalGlobe Use-case: Seeing a better world • DigitalGlobe takes Satellite imagery of the Earth • 100PB image library = 6 billion square kilometers • 1PB new image every year Architecture before Snowmobile: • Stored data in their own data center • Needed elastic compute power to retrieve and analyze images • Wanted to move data to the cloud, but no feasible solution Snowmobile lets DigitalGlobe migrate 100PB of data to the cloud
  42. 42. Summary Typical scenario Data Migration Data Migration or Local compute Data Migration Storage 50TB, 80TB 100TB 100PB Compute N/A Comparable to m4.4xlarge N/A AWS Services available S3 Lambda, File Gateway, S3 N/A NFS Endpoint N/A Yes Yes Encryption Yes Yes Yes Clustering for local use N/A Available N/A Typical job lifetime Days - weeks Data Migration – Days-Weeks Local compute – Weeks - Years Weeks – months HIPAA Yes Roadmap No Snowball SnowmobileSnowball Edge
  43. 43. Questions? Thank you

×