Petabytes of Data and No Servers: Corteva Scales DNA Analysis to Meet Increasing Business Demand - At #reInvent, join Corteva and Sogeti as they share their journey from petabytes of data & no servers to success in meeting increasing business demand.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Petabytes of Data & NoServers:
CortevaScales DNAAnalysis to Meet
Increasing Business Demand
Ryan Smith
Software Development Leader –
Bioinformatics
Corteva Agrisciences
E N T 2 1 8 - S
Scott Warren
Cloud Architect
Sogeti USA

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
WhereWeStarted
• Every 6 hours Corteva produces as much Genetic data as
existed in the entire public sphere in 2008
• On premise compute and storage demands were
becoming unsustainable
• 35 node Hadoop cluster with 2PB of storage
• Significant increase in future demand

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DataUses
• Assembly level variation
screening
• Transformation assay
• Quality control
• Whole genome assembly

9. 9
OurApplications
SNPFinder
• Whole genome alignment of short reads
• Looking for single nucleotide differences
• Surrounding area contains certain properties
Vector Quality Control (VQC)
• Synthesize a DNA fragment to create a
transgenic event
• Synthesis needs to be quality controlled
• Regulatory requirements

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ProjectTheseus

11. 11
Guiding Principles
User Patterns
• Time Sensitive Workloads
• Small User Base
Technical
• Serverless
• Immutable Infrastructure
• Automate Everything

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Short ReadSequencing

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Difference inDesign
• Both applications use the same input
data
• Type of processing, outputs and
technical requirements are very
different

14. 14
User Interaction
SNPFinder:
• Data is transformed into a query-able state
• Analysis is done ad-hoc through a user interface
or API layer
VQC
• All processing is completed when data
enters the application
• Users are viewing these results to inform
decision making

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon API Gateway
Amazon EMR
Pipeline
Amazon EMR
Query
AWS Lambda
SNPFinder-API
AWS LambdaAmazon SQS
Amazon RDS
Amazon S3
Snpfinder-data
Amazon EFS
Write job parameters
Job ID
Job ID
Get Job ID Start Cluster
Job Tracking
Write Intermediate Files
Job Tracking
Get Job Parameters
Write Output Parquet Files
Amazon SNS
Job Tracking
Complete|Failed
Amazon CloudWatch
Event every 2 minutes
Lexicon
Get ID
PSR
Get Sequencing Metadata
Amazon S3
Ngsa-data
Get Sequencing Data
Amazon S3
Snpfinder-intermediate
Write intermediate data
GAIA
Reference Files
GAIA Importer
Get Reference Files
PubSub
Power user
SNPFinder
Architecture

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User Interface

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VQC Architecture

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sideby side – difference betweenthetwo
Working on producing this picture:
Will show beginning and end of processes very similar. Middle/compute –
very different.

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EMRvs.Batch
• Both applications use the same input data
• SNPFinder has a small number of larger jobs
• EMR gave the ability to parallelize these jobs
• VQC runs a large number of much smaller workloads
• AWS Batch allowed for better scheduling, containerization and a more
natural programming paradigm

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWSGlue
• Introduced later in the project
• Using for data cleanup
• Move data without having to fully
reprocess

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results

27. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ryan Smith
ryand.smith@pioneer.com
Corteva Agrisciences
Scott Warren
scott.warren@us.sogeti.co
m
Sogeti USA

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.