SFTP is used for the exchange of data across many industries, including financial services, healthcare, and retail. In this session, we will introduce you to AWS Transfer for SFTP, a service that helps you easily migrate file transfer workflows to AWS, without needing to modify applications or manage SFTP servers. We will demonstrate the product and talk about how to migrate your users so they continue to use their existing SFTP clients and credentials, while the data they access is stored in S3. You will also learn how FINRA is using this new service in conjunction with their Data Lake on AWS.N/A

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Introducing AWS DataSync: simplify, automate,
and accelerate your online data transfer
K E M - 3 5
Paul Reed
Principal Product Manager
Amazon Web Services
David Green
Enterprise Solution Architect
Amazon Web Services
Dan Boisvert
Lead Software Engineer
Cox Automotive

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is AWS DataSync?

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
As more and more critical workloads move to the cloud…
…you need to
move
increasingly
large datasets
along with them
AWS

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What use cases need online data transfer?
Migration of active
application data
Online data transfer
Transfers for timely
in-cloud processing
Replication for
data protection
and recovery

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you solve this today?
Solution #1: Build DIY from open source tools
But you have to worry about ...
… it’s complicated and time consuming

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you solve this today?
Solution #2 License commercial software
But these are not managed or AWS-integrated...
… it’s complicated and costly

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS
Transfer for
SFTP
Managed file
transfers into
Amazon S3
NEW
AWS data transfer & hybrid storage
Online
data transfer
Hybrid
storage
Offline
data transfer
Load
streaming
data into
Amazon S3
Ship static
data into and
out of
Amazon S3
Access AWS
storage from
on-premises
Edge locations
for S3 enabled
applications
Online
transfer of
active data
Storage and
compute in
disconnected
environments
AWS
DataSync
NEW

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is AWS DataSync?
Online transfer service that simplifies, automates, and
accelerates moving data between on-premises storage and AWS
Fast data
transfer
Cost-
effective
Combines the speed and reliability of network acceleration
software with the cost-effectiveness of open source tools
Easy to use Secure and
reliable
Cloud
integrated
AWS

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared
file systems
NFS TLS
How AWS DataSync works
On-Premise
Amazon S3
bucket
AWS storage resources
AWS
DataSync
Deploy on-
premises agent
for fast access to
local storage
Region
Amazon EFS
file system
AWS DataSync
agent
Data transfer
over the WAN
using purpose-
built protocol
Managed from
AWS Console or
Command Line
Interface (CLI)
Service in AWS
writes or reads
data from AWS
storage services

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transfers at speeds up to 10 Gbps per agent
Purpose-built transfer protocol
Multithreaded design scales across multiple agents
Data reduction through incremental transfers, inline
compression, and sparse file detection
Optimized read and write to Amazon S3 and
Amazon EFS
Configurable throughput limits
Fast data
transfer

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simplifies management of data movement
Easy to use
Setup and manage in the AWS Console,
CLI, or SDK
No infrastructure in AWS to deploy or manage
Agent updates and patches are fully-managed
Metadata preserved between storage systems
and services

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transfer your data with confidence
Secure and
reliable
Encryption in-transit with TLS1.2
Supports AWS KMS encryption at-rest for
AWS services
Accesses AWS services within your VPCs
Data validation in-transit and at-rest
Automatic recovery from I/O errors
or transmission failures

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS management, identity, and compliance
Monitor with Amazon CloudWatch
Detailed CloudWatch Logs track data movement
Control and audit usage with AWS Identity and
Access Management (IAM) and AWS CloudTrail
PCI-DSS compliant and HIPAA eligible
Cloud
integrated
AWS

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Spend less money
Cost-
effective
Pay $0.04 per-GB of data copied, with no
minimum commitments or upfront fees
E.g., transferring 100 GB into EFS costs $4
AWS infrastructure is fully managed
and scaled based on usage
On-premises agents automatically updated
and patched

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
As simple as 1-2-3 ...
On-premises NFS file
systems, S3 buckets,
or EFS file systems
Create a task by configuring
storage locations and options
Monitor in the
console or through
Amazon CloudWatch
Start the task and
monitor progress
VMware ESXi
8 vCPU
32 GB memory
Deploy on-premises agent and
connect to your AWS account
1 2 3

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deploy the agent and connect to your AWS account
console.aws.amazon.com/datasync
1 2 3

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create a task by configuring storage locations and options
1 2 3

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Start and monitor the task
1 2 3

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Task execution status
Computing
which files need
to be transferred
Transferring data
to/from AWS
Initializing the
task execution
Performing full
data and
metadata
integrity
verification
If full verification not configured
Start your task again

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Task options
File-level validation of source and
destination after execution
Configured with the task, but can be overridden each execution
Which file metadata is copied and used to
determine changes
Mirror files deleted in source to destination
Set bandwidth limits when sharing network

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrate active application data
AWS
Migrate active data to AWS confidently with efficient
incremental transfers and end-to-end data validation
Existing on-premise data
Large data in a constant state of flux, with no natural break
or stopping point for a one-time transfer.
Mirrored to S3 objects or an EFS file system
Phased migration enables testing and application testing and
validation before cutover

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transfer data for timely in-cloud processing
Data processed in-cloud
Amazon CloudWatch Events trigger analysis on
data stored natively in AWS storage services
Data generated on-premises
e.g., DNA sequencers, video
production, GIS or seismographic data
AWS
Automated and accelerated data transfer, securely and reliably
copies data to Amazon S3 or Amazon EFS for timely analysis

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replicate for data protection and recovery
AWS
Periodic incremental copies reduce bandwidth and
minimize loss in the event of an on-premises failure

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reduce infrastructure with AWS DataSync and AWS Storage Gateway
DataSync migrates to Amazon S3. Incremental copies keeps Amazon S3 up to date1
2 Latency-sensitive applications can access data in S3 through File Gateway
Can reduce on-premises storage
NFS
server
Amazon S3
bucket
On-Premise Region
Applications 2
AWS
DataSync
AWS DataSync
agent
1
File Gateway

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DataSync and File Gateway working together
AWS DataSync
Copy on-premises files to
Amazon S3 or EFS
File Gateway
On-premises access to data
in Amazon S3AWS
DataSync
• NFS or SMB server
• Local caching
• Backed by Amazon S3
• Client to existing NFS
server
• Accelerated data transfer
• Copies into Amazon S3 or
Amazon EFS

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Dan Boisvert
Lead Software Engineer
Cox Automotive

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OUR VISION
TRANSFORM THE WAY THE WORLD
BUYS, SELLS, OWNS AND USES CARS

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
T R A N S F O R M I N G T H E W A Y T H E W O R L D B U Y S , S E L L S , O W N S A N D U S E S C C A R S

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How We Work

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current Guidance
Move everything to AWS
Go faster
Use PaaS wherever possible
Build what provides value to our customers

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Use Case
Pictures…

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Use Case
Pictures…
…lots of them

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Old vs new
S3 buckets in multiple regionsStack of NetApps
Image manipulation with CDN toolingHome-grown image transformation service
Web service accepts writesWrites via NFS
Access from multiple locationsMany readers via NFS
The new wayThe old way
Simple cache hierarchyOverly complex cache hierarchy

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges
• Finding all the producers &
consumers
• Sequencing
• Migration within SLAs
• Getting other teams to
update things

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How Do We Get There?
AWS
DataSync

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Near-Term Way
Get backing from the business
Find all the writers, collect perf requirements
Test File Gateway config against requirements
Update the readers to use HTTP endpoint
Final DataSync + cutover
Initial DataSync Incremental DataSyncs

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Near-Term Way
S3 buckets in multiple regions
Image manipulation with CDN tooling
File Gateway accepts writes
All reads through web endpoint
Simple cache hierarchy
File Gateway accepts writes

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Technical Considerations
DataSync is FAST!
• Your storage will likely be your
limiting factor—use throttling
• Your network people will chat
you
• 95th percentile circuit billing
could be your friend
File Gateway is good at mixed
workloads now
• Size your caches to suit your
workload
• Test, and use CloudWatch to
confirm
Remember your SLAs
• File Gateway restarts for updates; can your apps handle that?
• Mounting interruptible on readers and non- on writers can help
• Can you meet your storage SLA with a VM in your datacenter? You may want the hardware
appliance, or two VMs

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Outcomes
Prettier pictures on websites
Reduced management burden
Better sleep
Cost savings
Less cursing in 10 years when somebody else
has to work on this thing!

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
David Green
Enterprise Solution Architect
Amazon Web Services

45. jpg files

46. Gigabytes

47. average size per file

48. files per second

49. total duration

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
architecture and
requirements
• NAS Server
• Directory1 to S3 (us-west-2)
• Directory2 to EFS (us-west-2)
• Replicate S3 data to another
region for archive and partner
access with Storage Gateway
• 2 x 10Gb DirectConnect
• AWS DataSync Agent
AWS DataSync

57. Amazon
EFS
Demo architecture AWS Cloud
us-west-2
Oregon
On-premises architecture
NFS Server /
NAS:/data
Los Angeles
1. AWS DataSync moves files from NAS to Amazon S3 us-west-1
2. AWS DataSync writes data to Amazon S3 and Amazon Elastic File
System (Amazon EFS)
3. S3 Cross Region Replication (CRR) moves data to another region
within another account
4. S3 Lifecycle Policy writes data to Amazon Glacier
5. File access provided by AWS Storage Gateway (file gateway)
ap-northeast-2
Seoul
NFS Server /
NAS:/home/users
AWS
DataSync
Agent
AWS
DataSync

58. Demo architecture―DX
On-premises architecture
Los Angeles
group DirectConnect-Public-VIF {
type external;
peer-as 7224;
multipath;
neighbor 1.2.3.1 {
description "pubic-vif-01 dxvif-fg3qmoaq";
local-address 1.2.3.2;
authentication-key ""; ## SECRET-DATA
export EXPORT-PREFIXES;
}
neighbor 2.3.4.1 {
description "pubic-vif-02 dxvif-fgafmipq";
local-address 2.3.4.2;
authentication-key ""; ## SECRET-DATA
export EXPORT-PREFIXES;
}
}

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DataSync
Simplifies, automates, and accelerates your online data transfer
Transfers
up to 10 Gbps
per agent
Pay as
you go
Simple data
movement to
S3 or EFS
Secure and
reliable
transfers
Combines the speed and reliability of network acceleration
software with the cost-effectiveness of open source tools
AWS
integrated
AWS
Migrate active
application data
Transfer data for
timely processing
Replicate for data
protection and recovery

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Available in 10 AWS regions
• US East (N. Virginia)
• US East (Ohio)
• US West (N. California)
• US West (Oregon)
• EU (Frankfurt)
• EU (Ireland)
• Asia Pacific (Seoul)
• Asia Pacific (Singapore)
• Asia Pacific (Sydney)
• Asia Pacific (Tokyo)

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Get started at
http://aws.amazon.com/datasync

63. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Paul Reed
Principal Product Manager, AWS
paulreed@amazon.com

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.