Genomics on aws-webinar-april2018

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Angel Pizarro
AWS Research & Technical Computing
April 24, 2018
High Throughput Genomics on AWS
Containers and serverless technology for science

Agenda
• Background and introduction
• Deep dive on application packaging and AWS Batch
• Demo - packaging samtools using Docker and submitting a Job
• Encoding and executing full scientific workflows with AWS Lambda and
AWS Step Functions
• Demo - running a full workflow

The problem

Genomics data processing
Typical workflow in genomics analysis

Serial steps

Parallel steps

Retry logic

Problem 1: Application packaging
Need to package a application with its dependencies

Problem 2: Application execution
Need to provide inputs, runtime arguments, and collect output
input
output

Problem 2: Orchestration of execution
Need to define a dependency graph of applications and data
input
output output

A reference architecture for genomics workflows
Amazon ECR
Amazon S3
Applications
Data
Application Layer

Amazon ECR
Amazon S3
AWS Batch
Execution Layer
Job
Execution
Application Layer

Lambda
functions
Amazon ECR
Amazon S3
AWS Batch AWS Step FunctionsAWS Lambda
Orchestration
Application Layer Execution Layer Orchestration Layer

Lambda
functions
Amazon ECR
Amazon S3
Orchestration LayerApplication Layer Execution Layer

The Application Layer
Lambda
functions
Amazon ECR
Amazon S3

Bioinformatics application stacks
* Image courtesy of The Broad Institute - https://www.broadinstitute.org/gatk/img/BP_workflow.png

Virtualization of whole pipelines
Pros:
• Easy application publishing
• Clean dependency bundling
Cons:
• Large OS images
• Duplication of basic services
• Long start time
GATK v4.0
Bins/Libs
OS
GATK v4.0.1
Bins/Libs
OS

Packaging applications using Docker containers
GATK v4.0
Bins/Libs
OS
GATK v4.0.1
Pros:
• Easy application publishing
• Clean dependency bundling
• Shared dependencies
• Shared OS services
• Small images
Cons:
• Some cross container
networking issues

FROM ubuntu:16.04
RUN apt-get install -y python-pip python-dev
RUN pip install PIL
FROM python:2.7
RUN pip install numpy pandas
Docker Dockerfile and the build process
961f9d3583
c6d01316e4
a408d3cfe23
python:2.7ubuntu:precise
e3fc50a88d0
961f9d3583
c6d01316e4

Docker container source repositories
Community containers Custom developed
• Control specific version and build
features
• Support for S3 download and check
pointing data
• Scratch space management
• Container metadata management
• Full control on the software stack
• Licensing
• Monitoring
• Security and compliance adherence
https://dockstore.org/
http://biocontainers.pro/

Considerations for genomics applications on AWS
Data Staging
Use Amazon S3 to store reference and input data, store
results
Multi-tenancy
Have processes work with temporary directories
Storage cost/efficiency
Each Job cleans up after itself before returning

Demo 1 - Application packaging
using Docker

The Execution Layer
Lambda
functions
Amazon ECR
Amazon S3

Introducing AWS Batch
Fully Managed
Task Execution
No software to install or
servers to manage. AWS
Batch provisions and
scales your infrastructure
Integrated with AWS
AWS Batch jobs can easily
and securely interact with
services such as Amazon S3,
DynamoDB, and Rekognition
Cost-Efficient
AWS Batch launches compute
resources tailored to your jobs
and can provision Amazon EC2
and EC2 Spot instances

AWS Batch Concepts
Compute Environments
• The EC2 resources that do the work
Scheduler
• The resource scheduler, looks for submitted jobs and their
dependencies
Job Queue
• The resource to submit jobs to
Job Definition
• Defines the application, the minimal resources (CPUs, RAM)
and application arguments
Jobs
• The runtime instance of a Job Definition

Example AWS Batch Job Architecture
IAM Role for
Batch Job
Amazon S3
Input Files
Queue of
Runnable Jobs
Events Trigger
Lambda Function
Submits Batch Job
AWS Batch
Compute Environments
AWS Batch Job
Output
Job Definition
Job Resource Requirements
and other parameters
AWS Batch Execution
Application
Image
AWS Batch
Scheduler

A Visual Representation of AWS Batch

Executing Job(s)
Specify Docker run parameters as container overrides
Specify Job Queue
Submit Dependencies
aws batch submit-job --job-name testsamtools_stats
--job-queue ${JOB_QUEUE}
--job-definition ${JOB_DEFINITION}
--container-overrides vcpus=4,memory=6
# STDERR return should resemble the following
{ "jobName": "testsamtools_stats", "jobId": "f92b20d3-cdcd-4b92-aa0c-
6bfd98a65ac6" }

Demo 2 - Executing samtools stats
with AWS Batch

The Orchestration Layer
Lambda
functions
Amazon ECR
Amazon S3

Orchestration of workflows
Initiate Actions and Transitions

input
output

Workflow orchestration using
Serverless technologies

Owning servers means dealing with ...
Scaling
Availability and fault tolerance
Operations and management
Provisioning and utilization

No servers to provision
or manage
Scales with usage
Never pay for idle Availability and fault
tolerance built in
Serverless means…

SERVICES (ANYTHING)
Changes in
data state
Requests to
endpoints
New data
available
EVENT SOURCE FUNCTION
Node.js
Python
Java
C#
Go
AWS Lambda provides Functions as a Service

input
output
AWS Lambda

Using AWS Lambda
Bring your own code
• Node.js, Java, Python,
C#, Go
• Bring your own libraries
(even native ones)
Simple resource model
• Select power rating from
128 MB to 3 GB
• CPU and network
allocated proportionately
Flexible use
• Synchronous or
asynchronous
• Integrated with other
AWS services
Flexible authorization
• Securely grant access to
resources and VPCs
• Fine-grained control for
invoking your functions

Anatomy of a Lambda function
Handler() function
Function to be executed
upon invocation
Event object
Data sent during
Lambda Function
Invocation
Context object
Methods available to
interact with runtime
information (request ID,
log group, etc.)
def handle_request(job, context):
batch_job_response = submitBatchJob(job["name"],
job["queue"],
job["definition"])
return batch_job_response

Keep orchestration out of code.
Sequence Choice Parallel

AWS Step Functions
“Serverless” workflow management with zero
administration:
• Makes it easy to coordinate the components of
distributed applications and microservices using
visual workflows
• Automatically triggers and tracks each step, and
retries when there are errors, so your application
executes in order and as expected
• Logs the state of each step, so when things do go
wrong, you can diagnose and debug problems
quickly

AWS Step Functions

Seven State Types
Task A single unit of work
Choice Adds branching logic
Parallel Fork and join the data across tasks
Wait Delay for a specified time
Fail Stops an execution and marks it as a failure
Succeed Stops an execution successfully
Pass Passes its input to its output

Deployment with Step Functions and Lambda

A Genomics Workflow
Alignment
Variant
Calling
Annotation
QC

Putting it all together
$ aws stepfunctions start-execution
--state-machine-arn <your-
state-machine-arn>
--input
file://input.states.json
AWS Command Line Interface
AWS Batch console
Step Function console
S3 object listing

Demo 3 - Implementing a full
workflow using Lambda and Step
Functions

Alternatives!

AWS Partner Network

Open Source Workflow Orchastration
Cromwell

BioIT World workshop May 15th

Thank you!
We will send a follow up email with more information
on how to get started using AWS Batch for genomics.

Genomics on aws-webinar-april2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Genomics on aws-webinar-april2018

Similar to Genomics on aws-webinar-april2018 (20)

Recently uploaded

Recently uploaded (20)

Genomics on aws-webinar-april2018