This document discusses various options for migrating large data sets to AWS, including AWS Snowball, Snowball Edge, Snowmobile, Storage Gateway, and S3 Transfer Acceleration. It provides an overview of each solution's key capabilities and features. Example use cases are given to illustrate how customers can use these tools to migrate data to AWS cost effectively and simplify ongoing data transfer processes.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tobias Börjeson, AWS Solution Architecture
Guy Kfir, AWS Marketplace
Migrating Large Scale Data Sets to the Cloud
with AWS Marketplace

2. Storage is the gravity
for cloud applications

3. Cloud Data Migration
Direct
Connect
Snow* data
transport
family
3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
The AWS Storage Portfolio
Object
Amazon GlacierAmazon S3
Block
Amazon EBS
(persistent)
Amazon EC2
Instance Store
(ephemeral)
File
Amazon EFS

4. Snow*

5. AWS Snow Family
Snowball Snowball Edge Snowmobile
Petabyte-scale
data migration
Compute & Storage for
Hybrid/Edge workloads
Exabyte-scale data
migration

6. Collect data Create job Copy data Moved to S3
Hybrid capabilities beyond data migrationMIGRATIONCOLLECTION
Create job Copy data Moved to S3

7. Snowball Edge key features
S3-compatible endpoint
File interface (NFS)
Clustering
Run AWS Lambda functions
Faster data transfer
Encryption

8. Snowball Edge can…
Extend of your
data center
Process data Expedite
move
Encrypted, secure,
and embedded
compute
Write data directly
as data is
generated
Offers a fast and
cost effective way
to ensure data
can be quickly
transferred to and
from the cloud
Simplify
data transfer
Uses standard
and familiar tools
for the data
transfer process

9. Snowball Edge: Remote Data Collection
Customer Example: Oregon State University
Collect and analyze oceanic and coastal images (60TB/week)
• Environmental and ocean ecosystem research
• Prior to Snowball, used many small HDDs – took weeks to months to upload
• $4MM+ in infrastructure investment -- Expensive and inefficient
“Snowball lets us migrate TBs of data in days, at a fraction of the cost”

10. S3 Transfer Acceleration

11. Amazon S3 transfer acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Change your endpoint, not your code
Leverages 70 global edge locations
Optimized protocols
No firewall exceptions
No client software required

12. Amazon
Route 53
Resolve
b1.s3-accelerate.amazonaws.com
HTTPS PUT/POST
upload_files.zip
HTTP/S PUT/POST
“upload_files.zip”
Service traffic flow
Client to S3 Bucket example
S3 Bucket
b1.s3-accelerate.amazonaws.com
EC2 Proxy
AWS Region
AWS Edge Location
Customer Client
1
2
3
4

13. Rio De
Janeiro
Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los
Angeles
Seattle Tokyo Singapore
Time[hrs]
500 GB upload from these edge locations to a bucket in Singapore
Public Internet
How fast is S3 Transfer Acceleration?
S3 Transfer Acceleration
On average, we have seen 171%
improvement over regular S3 when
uploading over long distances

14. Use Case: Media Upload
“S3 transfer acceleration reduces the average time it takes for
us to ingest videos from our global user base by almost half.
This gives our customers the ability to edit and share videos
sooner where speed is a critical factor. - Brian Kaiser, CTO
• Growing 30% per year
• Typical Friday during football season – over 35 hours of
video every minute is uploaded
Data in: User > S3-TA > S3 > Transcode > Redshift
Data out: S3 > CloudFront > User
S3-TA = >20% increase in upload and encoding speeds

15. Storage Gateway

16. Storage Gateway hybrid storage solutions
Enables using standard storage protocols to access AWS storage services
AWS Storage
Gateway
Amazon EBS
snapshots
Amazon S3
Amazon Glacier
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
Files
Volumes
Tapes

17. File gateway
On-premises file storage maintained as objects in Amazon S3
Customer Premises
File
Gateway
Data stored and retrieved from your S3 buckets
One-to-one mapping from files-to-objects
File metadata stored in object metadata
Bucket access managed by IAM role you own and manage
Use S3 Lifecycle Policies, versioning, or CRR to manage data
GlacierS3
Standard
S3
Standard -
Infrequent
Access
HTTPS
NFS
v3 / v4.1
Application
Server

18. Enabling cloud workloads
Move data to AWS storage for Big Data, cloud bursting, or migration
“Storage Gateway has the promise to transform the way we move
data into the cloud. The NFS interface lets us easily integrate data
files from analytical instruments, and the transparent S3 storage
lets us easily connect our cloud-based applications and leverage the
powerful storage capabilities of S3.
With Storage Gateway, we can now unleash the full power of AWS
on our instrument data.”

19. Application
Server
Volume gateway
On-premises volume storage backed by Amazon S3 with EBS snapshots
Block storage in S3 accessed via the volume gateway
Compression of data in-transit and at-rest
Backup on-premises volumes to EBS snapshots
Create on-premises volumes from EBS snapshots
Up to 1PB of total volume storage per gateway
Amazon
EBS
snapshots
Storage Gateway
bucket in
Amazon S3
Customer Premises
Volume
Gateway
iSCSI HTTPS

20. Tiered storage to S3, Glacier, and EBS
Easily add AWS storage to your on-premises environment
“Storage Gateway is at the core of our disaster
recovery and business continuity (BCM) processes,
handling our Co-Lo'd OLTP and OLAP off site data
backups, as well as our in-office BCM. It works
transparently, in a lights out way, archiving off to a
separate AWS account with a simple grandfather-father-
son snapshot plan in place”.

21. Tape gateway
Virtual tape storage in Amazon S3 and Glacier with VTL management
Virtual tape storage in S3 and Glacier accessed via tape gateway
Data compressed in-transit and at-rest
Up to 1 PB total tape storage per gateway, unlimited archive capacity
Supports leading backup applications:
Archived Tapes
stored in
Amazon Glacier
MEDIA
CHANGER
TAPE
DRIVE
Customer Premises
Tape
Gateway
Virtual Tapes
stored in
Amazon S3
Backup
Server
HTTPSiSCSI

22. Backup, archive, and disaster recovery
Cost effective storage in AWS with local or cloud restore
“Tapes are a headache, prone with hardware
failures, offsite storage costs, and constant
maintenance needs. Storage Gateway
provided the most cost-effective and simple
alternative. We even got disaster recovery by
using a bi-coastal data center”.

23. Partner Solutions

24. Storage Partner Solutions
Technology Solutions vetted by the AWS Storage Competency Program
aws.amazon.com/backup-recovery/partner-solutions/ Note: Represents a sample of storage partners
Note: Dell-EMC, IBM and Veritas have solutions and are working towards competency requirements
Backup and RecoveryPrimary Storage Archive BCDR
Solutions that leverage file, block, object,
and streamed data formats as an
extension to on-premises storage
Solutions that leverage Amazon S3 for
durable data backup
Solutions that leverage Amazon Glacier
for durable and cost-effective long-term
data backup
Solutions that utilize AWS to enable
recovery strategies focused on RTO
and RPO requirements

25. Development
& Test
New
Applications
Digital & CX
Analytics
Mobile
DC Migration
Mission
Critical Apps
All In
This journey includes migrating existing
enterprise software to the cloud
Enterprise Software and the Journey to the
Cloud

26. Gartner Report "Market Insight: Cloud Computing's Drive to Digital Business Creates Opportunities for Providers.“ May 24, 2016
Jeffrey Mann, David Mitchell Smith, Ed Anderson, Jay Heiser, Yefim V. Natis, Thomas Bitman, Neville Cannon
“Cloud will increasingly be
the default option for
software deployment.”
Gartner
The Enterprise Software Portfolio Is In Transition
Software catalog starting position
• 10-40% in house developed applications
• 30% open source/free software
• 70% COTS software
• ~200+ software product vendors
• Large EA/ELAs for top 50 vendors
• Upgrade cascade over 1-3 years – too slow
• Windows workloads moving quickly

27. What We Hear from Customers
Challenges
• Software entitlement & deployment models
• Complex agreement management
• Constant renewal & replacement
• Out of date procurement mechanisms
• No single approved catalog of software i
Customers want to
• Rapidly innovate by buying and deploying software
solutions on-demand
• Simplify and streamline purchasing, license
management and invoicing
• Upgrade on demand
• Reduce cost while picking new standards
Source: AWS Marketplace Survey, Q1 2016
50% of apps migrate*
20% get abandoned*
30% are replace*

28. Submit
Publish
Preview
Test
• Ongoing catalog additions
• Version updates*
• Vendor supply portal
• Centralize digital assets
*Vendor dependent
• Sunset older versions
• Add instance type/region
• Actively monitor
health and expansion
• 7/24 scanning
• Vulnerability notifications
and removal/work-rounds
• MCO Team
AWS Marketplace – A Public Managed Catalog

29. Use third-party software to accelerate workload adoption on AWS
Procuring 3rd Party Software from AWS Marketplace
Customer Benefits
• Deploy software on demand
• Curated software from
trusted vendors
• Rapidly innovate
• Familiar technologies
• Simplified procurement
and deployment
• Billed through AWS
account
• Retire EDP
commitments
• Lower TCO
Platform Highlights
• Over 100K active customers
• 3,600+ listings
• Open-source and
commercial software
• Metered Billing
and BYOL
• Subscriptions for SaaS
• Private Pricing
for large deals
• Vendor-based support
• Available in
all new AWS regions

30. Security BIStorage MediaDatabaseNetworking DevOps
Categories Most Often Deployed
Sample Vendors in Category
Operating
Systems

31. • New APIs for software vendors to call
• Elastic pricing on dimensions tied
most closely to usage and value
• Single pane of glass for billing
and monitoring on AWS or in
AWS/hybrid scenarios
• All software charges shown on AWS Bill
Marketplace is enabling NEW flexible metering
On-premises Cloud
AWS invoice

32. SaaS and API Subscriptions
Now available in AWS Marketplace
NEW
Nov 16th

33. SaaS CONTRACTS
Now available in AWS Marketplace
NEW
April 18th

34. AWS Marketplace Usage and Pricing
• No SW costs for use of these products but customers are
still responsible for infrastructure costs
• Dominated by operating systems
• Buy perpetual SW licenses direct from the vendor and
deploy via Marketplace
• Customers can often transfer licenses from on-premises
(Migrations) and negotiate discounted pricing with ISVs
• Metered by time, users, bandwidth, hosts or data
• Pay for only what you use – costs are tied to usage and
can lower TCO for workloads with spiky traffic
• Free Trials available for many products
• Service completely managed and fulfilled by SaaS vendor
• Customer does not manage the infrastructure
• Available as subscriptions and contracts
Free /
Open
Source
Bring Your
Own
License
SaaS &
Contracts
Paid
Software
Free Trial
• Get Started on AWS Marketplace
with free trial & no commitment
• Converts to Metered after trial period
• Good for initial evaluation
Metered
• Pay for software and compute
capacity by the hour, users, data, etc.
• No long-term commitments
• Ideal for dev / test or spiky workloads
Annual
• Make a low, one-time payment and
receive a significant pricing discount
• Ideal for periodic and regular
workloads with constant traffic levels
Private Pricing
• Negotiated pricing between customer
& ISV & fulfilled on AWS Marketplace
• Customers able to get better terms
for large & long-term purchases

35. 1. Audit or build your Software library as you move to the cloud
2. Standardize
3. Modernize or abandon
4. Compare choices to AWS Marketplace & find your preferred suppliers
5. Use BYOL and upgrade to newer versions
6. Shift to subscription: Pay-as-you-go or Duration Contracts
7. Consider AWS Marketplace as your public managed catalog
8. Manage private choices & republish and control for compliance on AWS Service
Catalog – your private catalog
9. Plug-in a Spend Management tool & track spend to asset
10. Innovate faster, apply launch controls, and reduce costs
What to do next? Ten options and suggestions

36. Bursting
Migration
Tiering
Accessing EFS
via Direct Connect
Storage
Gateway
Snowball
Edge
3rd Party
Ecosystem
AWS hybrid cloud storage choices

37. Thank You!