Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

(STG202) AWS Import/Export Snowball: Large-Scale Data Ingest into AWS

3,994 views

Published on

Moving terabyte and petabyte volumes of data into the cloud can be a challenge for many businesses. Come learn how you can use Snowball, a new AWS feature, to move large-scale (terabyte and petabyte) data to AWS storage services.

Published in: Technology
  • Be the first to comment

(STG202) AWS Import/Export Snowball: Large-Scale Data Ingest into AWS

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Colin Lazier, General Manager - AWS Import/Export & Amazon Glacier October 2015 Introduction to AWS Import/Export Snowball Accelerating Large-Scale Data Ingest into AWS Storage Services STG202
  2. 2. What to expect from the session • What is AWS Import/Export Snowball? • How does it work? • What are the two most commonly asked Snowball questions? • How fast is that truck full of drives? • How is my data transported securely? • Where can it go? • What does it cost? • Demo
  3. 3. What is Snowball? Petabyte scale data transport E-ink shipping label Ruggedized case “8.5G Impact” All data encrypted end-to-end 50 TB 10G network Rain & dust resistant Tamper-resistant case & electronics
  4. 4. How does it work?
  5. 5. Question 1:
  6. 6. Question 1: You have heard of networks, right?
  7. 7. Question 1: You have heard of networks, right? • Yes
  8. 8. Question 1: You have heard of networks, right? • Yes • However, for certain use cases networks are: Too slow Inelastic Too expensive
  9. 9. Question 1: You have heard of networks, right? • Yes • However, for certain use cases networks are: Too slow Inelastic Too expensive Bandwidth is growing more slowly than storage volumes
  10. 10. Question 1: You have heard of networks, right? • Snowball Fast Elastic Cost effective 10G network 50TB capacity
  11. 11. Question 1: You have heard of networks, right? • Snowball Fast Elastic Cost effective 10G network 50TB capacity
  12. 12. Question 1: You have heard of networks, right? • Snowball • Parallelize as needed 5X Snowballs 50G & 250TB
  13. 13. Question 2:
  14. 14. Question 2: Can I drop it?
  15. 15. Question 2: Can I drop it? • No (please don’t)
  16. 16. Question 2: Can I drop it? • No (please don’t) • Snowball is its own box • Has had many drop tests already • Can handle 8.5G impacts • Designed for shipping
  17. 17. Ok, let’s drop it
  18. 18. How fast is that truck full of drives? • Less than 1 day to transfer 50TB via a 10G connection with Snowball, less than 1 week including shipping • Number of days to transfer 50TB via the internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 19 38 63 126 50% 9 19 32 63 75% 6 13 21 42
  19. 19. How fast is that truck full of drives? • Less than 1 day to transfer 250TB via 5x10G connections with 5 Snowballs, less than 1 week including shipping • Number of days to transfer 250TB via the Internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 95 190 316 632 50% 47 95 158 316 75% 32 63 105 211
  20. 20. How is my data transported securely? • 256-bit encryption • Data is encrypted by the Snowball client before it reaches the Snowball appliance • Keys are managed by KMS and are never sent to the Snowball
  21. 21. How is my data transported securely? • Strong chain of custody • Tamper-resistant case • Tamper-resistant electronics (TPM) • Each Snowball is erased according to NIST 800-88 media sanitization guidelines between every job
  22. 22. From Where can it go? Import to Amazon S3, Export coming soon Amazon EMR Amazon Redshift Amazon S3 Amazon Glacier Amazon DynamoDBAmazon RDS Amazon EBS US-EAST-1 (Virginia) US-WEST-2 (Oregon)
  23. 23. What does it cost? • $200 / job plus shipping • Includes 10 days to fill the device at your site • $15/day after the tenth day on site • Standard Amazon S3 charges apply • $0.03/GB to transfer data out • $0.00/GB to transfer data in
  24. 24. What does it cost? Example 1: • 40TB loaded on to 1 Snowball • 2 days at your site • $200 plus shipping Example 2: • 30TB loaded on to 1 Snowball • 12 days at your site • $200 + 2*$15/day = $230 plus shipping
  25. 25. What does it cost? Example 3: • 140TB loaded on to 3 Snowballs • 8 days at your site • 3 * $200 = $600 plus shipping Example 4: • 30TB exported on to 1 Snowball • 8 days at your site • $200 + 30TB * $0.03/GB = $1,121.60 plus shipping
  26. 26. Demo
  27. 27. Overview
  28. 28. Step 1: Create a job
  29. 29. Step 1: Create a job
  30. 30. Step 1: Create a job
  31. 31. Step 1: Create a job
  32. 32. Step 2: Install client & await arrival
  33. 33. Step 3: Plug in your Snowball appliance
  34. 34. Step 4: Get your credentials
  35. 35. Step 5: Connect client to the Snowball appliance From a terminal:  snowball start -i 192.0.2.0 -m /Downloads/JID2EXAMPLE-0c40-49a7-9f53- 916aEXAMPLE81-manifest.bin -u 12345-abcde-12345- ABCDE-123 snowball start -i [Snowball IP Address] -m [Path/to/manifest/file] -u [29 character unlock code]
  36. 36. Step 6: Copy your data From terminal:  snowball cp --recursive /Logs/April s3://MyBucket/Logs/April snowball cp [options] [path/to/data/source] s3://[path/to/data/destination]
  37. 37. Step 7: Unplug your Snowball appliance & notify UPS
  38. 38. Step 7: Wait for import to complete
  39. 39. Thank you!
  40. 40. Remember to complete your evaluations!

×