There are a myriad of ways to get your file-based data into Amazon S3 and Amazon EFS, from the simple AWS command line tools and similar script-based approaches to proprietary commercial tools built for media workflows. Attend this session to learn about an architecture and associated tips and tricks that enable you to improve data transfer performance—scaling out with parallel streams—without the manual labor of extensive scripting or the cost of third-party licensed software.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Move Data to AWS Faster for Migrations, DR,
& Bidirectional Workflows
S T G 3 8 2
Olga Kogan
Sr. Product Manager
Amazon Web Services
David Green
Enterprise Solutions Architect
Amazon Web Services
Lance Smith
Associate Director
Celgene

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Celgene case study

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is AWS DataSync?
Online transfer service that simplifies, automates, and
accelerates moving data between on-premises storage and AWS
Fast data
transfer
Cost-
effective
Combines the speed and reliability of network acceleration
software with the cost-effectiveness of open source tools
Easy to use Secure and
reliable
Cloud
integrated
AWS

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What’s the problem?
As more and more critical workloads move to the cloud …
… you need to
move increasingly
large datasets
along with them
AWS

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common use patterns for AWS DataSync
Migration of
active
application data
to AWS
Transferring
data for time
sensitive in-
cloud analysis
Replication of
data to AWS for
business
continuity

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared
file system
NFS TLS
How AWS DataSync works
On-Premise
Amazon S3
bucket
AWS storage resources
AWS
DataSync
Agent deployed
on-premises for
fast access to
local storage
Region
Amazon EFS
file system
AWS DataSync
agent
Data transfer
over the WAN via
efficient purpose-
built protocol
Managed from
AWS Console or
AWS Command
Line Interface
(AWS CLI)
Service in AWS
writes or reads
data from AWS
storage services

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Celgene
Celgene is building a preeminent global
biopharmaceutical company focused on the
discovery, development, and commercialization of
innovative therapies for patients with cancer,
immune-inflammatory, and other unmet medical
needs

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Biotech lab environment challenges
• Multiple lab systems
• Different standards
• Distributed sites
• Various lab processes
• Diverse storage &
networking

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Moving data to the cloud
Use cases
• Rein in ”hidden” files
stored on local lab disk
• Data archive
• Data transfer to compute
• Centralize data for machine
learning & analytics
Challenges
• Impossible to change
applications in near-term
• Very hard to change people
and processes
• Data transfers & latency
• Offline data movement
possible, but lengthy
Previous options
• Amazon Simple Storage Service
(Amazon S3) upload clients, but
resistance to change by users
• Multiple backup tools
• Custom scripted utilizing cron
• One backup software cost $25K
for one instrument; we have 1K
instruments …

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Moving data to the cloud
Use cases
• Rein in ”hidden” files
stored on local lab disk
• Data archive
• Data transfer to compute
• Centralize data for machine
learning & analytics
Challenges
• Impossible to change
applications in near-term
• Very hard to change people
and processes
• Data transfers & latency
• Offline data movement
possible, but lengthy
Previous options
• Amazon S3 upload clients, but
resistance to change by users
• Multiple backup tools
• Custom scripted utilizing cron
• One backup software cost $25K
for one instrument; we have 1K
instruments …

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Moving data to the cloud
Use cases
• Rein in ”hidden” files
stored on local lab disk
• Data archive
• Data transfer to compute
• Centralize data for machine
learning & analytics
Challenges
• Impossible to change
applications in near-term
• Very hard to change people
and processes
• Data transfers & latency
• Offline data movement
possible, but lengthy
Previous options
• Amazon S3 upload clients, but
resistance to change by users
• Multiple backup tools
• Custom scripted utilizing cron
• One backup software cost $25K
for one instrument; we have 1K
instruments …

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our hybrid architecture AWS Cloud
Region
VPC
On-premises architecture
NFS Server/
NAS
Lab PCResearch
Scientists
Data Acquisition
Instrumentation
Scientific
Experiment
CambridgeSeattleSan FranciscoSevilleSan Diego
1. Instruments and users write to PCs and then NAS
2. AWS DataSync pulls or reads data from NAS, and …
3. Sends it over the internet, or through our colo and AWS Direct
Connect, based on data type and size, and validates data integrity
4. Once the data lands in Amazon S3, archiving, analytics, machine
learning, and processing actions occur
Internet
Colocation
facilities
Or
AWS
DataSync
AWS
DataSync
Agent

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. Terabytes

18. files

19. size per file

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. AWS EFS
Demo architecture AWS Cloud
us-west-2
Oregon
On-premises architecture
NFS Server /
NAS:/data
Los Angeles
1. AWS DataSync moves files from NAS to Amazon S3 us-west-1
2. AWS DataSync writes data to Amazon S3 and Amazon Elastic File
System (Amazon EFS)
3. S3 Cross Region Replication (CRR) moves data to another region
within another account
4. S3 Lifecycle Policy writes data to Amazon Glacier
5. File access provided by AWS Storage Gateway (file gateway)
ap-northeast-2
Seoul
NFS Server /
NAS:/home/users
AWS
DataSync
Agent
AWS
DataSync

22. Demo architecture―DX
On-premises architecture
Los Angeles
group DirectConnect-Public-VIF {
type external;
peer-as 7224;
multipath;
neighbor 1.2.3.1 {
description "pubic-vif-01 dxvif-fg3qmoaq";
local-address 1.2.3.2;
authentication-key ""; ## SECRET-DATA
export EXPORT-PREFIXES;
}
neighbor 2.3.4.1 {
description "pubic-vif-02 dxvif-fgafmipq";
local-address 2.3.4.2;
authentication-key ""; ## SECRET-DATA
export EXPORT-PREFIXES;
}
}

23. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.