CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:Invent
AWS Batch: Easy & Efficient Batch
Computing on Amazon Web Services
C M P 3 2 3
AdRoll: Mikko Juola & Oleg Avdeev
Base2 Genomics: Ryan Layer, Aaron Quinlan, & Brent Pedersen
Autodesk: Dinu Bunduchi
AWS: Jamie Kinney
November 28, 2017

Introductions
Mikko Juola: Sr. Software Engineer, AdRoll
Oleg Avdeev: Staff Engineer, AdRoll
Ryan Layer: Co-Founder, Base2 Genomics
Aaron Quinlan: Co-Founder, Base2 Genomics
Brent Pedersen: Co-Founder, Base2 Genomics
Dinu Bunduchi: Cloud Infrastructure Architect, Autodesk
Jamie Kinney: Principal Product Manager, AWS Batch and HPC

Agenda
- Summary of Recent AWS Batch Launches
- Glimpse Into Our Roadmap
- Real-World Examples:
• AdRoll
• Base2 Genomics
• Autodesk
-Q&A

AWS Batch
Fully Managed
No software to install or
servers to manage. AWS
Batch provisions,
manages, and scales your
infrastructure
Integrated with AWS
Natively integrated with the
AWS Platform, AWS Batch
jobs can easily and securely
interact with services such as
Amazon S3, Amazon
DynamoDB, and Amazon
Rekognition
Cost-optimized
Resource Provisioning
AWS Batch automatically
provisions compute
resources tailored to the
needs of your jobs using
Amazon EC2 and EC2 Spot

2017 Launches

AWS Batch Regional Expansion
AWS Batch is available in the following regions:
• us-east-1 (N. Virginia)
• us-east-2 (Ohio)
• us-west-2 (Oregon)
• eu-west-1 (Ireland)
• eu-west-2 (London)
• eu-central-1 (Frankfurt)
• ap-northeast-1 (Tokyo)
• ap-southeast-2 (Sydney)
• ap-southeast-1 (Singapore)

Improved Managed Compute Environments
Custom AMIs:
• Auto-mount EFS and other shared filesystems
• Larger/faster/encrypted EBS
• GPU/FPGA drivers
New instance families
• G3
• F1
• P3
• C5
Faster scale-up/scale-down and per-second billing
Automated tagging of EC2 Spot instances

Manageability & Performance Improvements
• Automated Job Retries: Easily recover from application
errors, hardware failures, or EC2 Spot terminations
• Scheduling throughput: run jobs as short as 5 seconds with
~90% scheduling efficiency
• Native AWS CloudFormation Support for AWS Batch
Resources
• Amazon CloudWatch Events for Job State Transitions

HIPAA Eligibility
https://aws.amazon.com/compliance/hipaa-eligible-services-reference/

Workflows, Pipelines, and Job Dependencies
Jobs can express a dependency on the successful completion of
other jobs or specific elements of an array job.
You can also use AWS Step Functions or other workflow systems to
submit jobs. Flow-based systems submit jobs serially, while DAG-
based systems submit many jobs at once, identifying inter-job
dependencies.
$ aws batch submit-job –depends-on 606b3ad1-aa31-48d8-92ec-f154bfc8215f ...
Job A Job CJob B

Workflows and Job Dependencies
https://github.com/awslabs/aws-batch-genomics

New! Easily Run Massively Parallel Jobs
Array jobs allow you to run up to 10,000 copies of an application. AWS Batch
creates child jobs for each element in the array.
Array jobs are an efficient way to run:
• Parametric sweeps
• Monte Carlo simulations
• Processing a large collection of objects
Also includes:
• Extended Dependency Model
• Enhancements to Job APIs
Job A Job C
Job B:0
Job B:1
Job B:n
…

Example Array Job Submission
$ aws batch submit-job --job-name BigBatch --job-queue ProdQueue --job-
definition monte-carlo:8 --array-properties “size=10000” ...
{
"jobName": ”BigBatch", "jobId": "350f4655-5d61-40f0-aa0b-03ad787db329”
}
Job Name: BigBatch
Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329
Job Name: BigBatch
Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329:0
Job Name: BigBatch
Job Name: BigBatch
…

Array Job Dependency Models
Job Depends on Array Job
“Job-A”
A:0
…
A:1
A:99
“Job-B”
$ aws batch submit-job –cli-input-json file://./Job-A.json
<Job-A.json>
{
"jobName": ”Job-A",
"jobQueue": "ProdQueue",
"jobDefinition": ”Job-A-Definition:1",
”arrayProperties":
{
“copies”: 100
}
}
$ aws batch submit-job –cli-input-json file://./Job-B.json
<Job-B.json>
{
"jobName": ”Job-B",
"jobDefinition": ”Job-B-Definition:1",
"dependsOn": [
{"jobId": "<job ID for Job A>" }
]
}

Array Job depends on Job
“Job-B”
B:0
…
B:1
B:99
“Job-A”
$ aws batch submit-job --job-name Job-A --job-queue ProdQueue --
job-definition job-A-Definition:1
{
"jobName": "sequential-stress-10",
"jobId": "7a6225f0-a16e-4241-9103-192c0c68124c”
}
<Job-B.json>
{
"jobDefinition": ”Job-A-Definition:1",
”arrayProperties":
{
“copies”: 100
},
"dependsOn": [
{"jobId": "7a6225f0-a16e-4241-9103-192c0c68124c" }
]
}

Two Equally-Sized Array Jobs, a.k.a. “N-to-N”
“Job-A”
A:0
…
A:1
A:2
A:3
A:9999
B:0
B:1
B:2
B:3
B:n
“Job-B”
$ aws batch submit-job --job-name Job-A --job-queue
ProdQueue --job-definition job-A-Definition:1 --array-
properties size=10000
{
"jobId": "7a6225f0-a16e-4241-9103-192c0c68124c”
}
$ aws batch submit-job --job-name Job-B --job-queue
ProdQueue --job-definition job-B-Definition:1 --array-
properties size=10000 --depends-on jobId=7a6225f0-a16e-
4241-9103-192c0c68124c,type=N_TO_N
{
"jobName": ”Job-B”,
"jobId": "7f2b6bfb-75e8-4655-89a5-1e5b233f5c08”
}

Array Job Depends on Self, a.k.a. Sequential Job
“Job-A”
A:0
…
A:1
A:9
A:2
$ aws batch submit-job --job-name Job-A --job-queue ProdQueue
--job-definition job-A-Definition:1 --array-properties
size=10 --depends-on type=SEQUENTIAL
{
"jobId": "7a6225f0-a16e-4241-9103-192c0c68124c”
}

Another Example
C is dependent on A and B
D has an N_TO_N dependency on C
“Job-A”
“Job-C”
C:0
…
C:1
C:2
C:3
C:9999
D:0
D:1
D:2
D:3
D:9999
“Job-D”
“Job-B”

These Models Can be Combined
“Job-A”
“Job-C”
C:0
…
C:1
C:2
C:3
C:9999
D:0
D:1
D:2
D:3
D:9999
“Job-D”“Job-B”
B:0
…
B:1
B:9
B:2
“Job-E”
Heavy
Network I/O
CPU
Intensive
Large
Memory
Setup Cleanup

Roadmap

What Can You Expect in 2018?
• Significant improvements to the AWS Batch console
• Automatically submit AWS Batch jobs in response to
CloudWatch Events
• CloudTrail auditing of AWS Batch API calls
• Consumable resources
• Additional job types
• Further regional expansion

Mikko Juola: Senior Software Engineer, AdRoll
Oleg Avdeev: Staff Engineer, AdRoll

Adoption of AWS Batch at AdRoll
AdRoll started using AWS Batch in June 2017. These numbers were collected between June 2017 and November
2017.
1.2 MILLION
JOBS
SUBMITTED
300K
INSTANCES
CHURNED
600
CPU YEARS
SPENT
2 TEAMS USE
IN
PRODUCTION

Why does AdRoll like AWS Batch?
Docker support is first class. Very flexible!
Workflow is conceptually simple:
• Put code in Docker image
• Submit Docker image + command line arguments
Lots of control over the environment you are running your job in (e.g. huge instances, custom
monitoring software).
Cost-effective (especially with per-second billing and spot instances). AdRoll uses spot instances
exclusively in AWS Batch applications.

On the next few slides, I’ll go through some mechanisms we’ve built on top of AWS Batch.
How is AWS Batch being used by AdRoll?
These can give you some ideas how to structure your batch job workflow on top of this
service.

Monitoring Batch Jobs
When you are running thousands of jobs per
day, being able to monitor them by the bulk
becomes important.
At AdRoll, we built a monitoring tool tailored
to our way of using AWS Batch.
The monitoring on AWS Management Console for
AWS Batch is not bad, but it can be difficult to sift
through tens of thousands of jobs to find
something specific and doing it quickly.

SEARCH
STATUS
JOB QUEUE

Submitting Jobs
from pybatch import run_on_awsbatch
run_on_awsbatch(image=‘ubuntu:16.04',
name=‘Hello Job’,
jobqueue='attribution-managed-spot-staging',
command_line=['echo', 'hello'],
timeout=3600)
We wrote a Python library for the purpose of making submitting jobs as
simple as possible.

AWS Batch has no built-in support for timeouts. There will be jobs that rarely get stuck.
Timeouts
When we submit a job, we set an environment variable BATCH_TIMEOUT with the timestamp
when the job would time out.
A periodically running script inspects all active jobs and kills those whose timeout has expired.

Other quirks
We use a custom AMI that sets up instance store for batch jobs. To give an
example why this is useful: some of our jobs are specifically designed for i3
instances that make use of their highly efficient NVMe SSD storage.
Production and staging pipelines have entirely separate AWS Batch
infrastructure.
Our job queues are organized by the instance type. If I submit a job to a
certain job queue, I can be certain it’ll get a specific kind of instance type.

General tips
Start simple. You don’t have to build anything complex on top of AWS Batch
to get started.
If you decide to scale up your AWS Batch use, think about how you are going
to monitor and debug your jobs. The system will not have a problem running
tens of thousands of containers. But you might have a problem finding
specific jobs and their logs after they have run.
If you can use spot instances, use spot instances. Can save you lots of $$$!

Ryan Layer: Co-Founder, Base2 Genomics

S3 Batch
Large files
Heterogeneous pipeline
Task and data parallelism

Ewing Sarcoma
Raw sequence
1,049 samples
50 GB each
52.5 TB
0 GB / 0 CPU h© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Sequence Alignment: Sample-Parallel
Raw sequence
1,049 samples
50 GB each
52.5 TB

Raw sequence

Raw sequence
0 GB / 0 CPU h

Raw sequence
50GB
alignment

Raw sequence
50GB
alignment
batchit

batchitgithub.com/base2genomics/batchit

batchit
# alignment.sh
bwa mem –t ${cpus}
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
github.com/base2genomics/batchit

batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--image $docker_image
--role $container_role
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--ebs "/mnt/ebs:100:gp2:ext4"
--cpus 16
--mem 30000
--jobname align-SS-12345
alignment.sh

# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
Set the Docker image

# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
Set IAM role

# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
Set Batch queue

# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
Fill variables in script

batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit
Create, format, attach in Docker
Auto cleanup

batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
{
"Parameters": null,
"DependsOn": null,
"JobQueue": "i3-16xl",
"JobName": "i3-test",
"RetryStrategy": null,
"ContainerOverrides": {
"Environment": [ { "Name": "B64GZ", "Value": "H4sIAAAAAAAA/wEAAP//AAAAAAAAAAA=" },
{ "Name": "cpus", "Value": "16" },
{ "Name": "sample", "Value": "SS-12345" },
{ "Name": "reference", "Value": "GRCh37" },
{ "Name": "TMPDIR", "Value": "/mnt/ebs" } ],
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"for i in "$@"; do eval "$i"; done",
"export vid=$(batchit ebsmount -n 1 -m /mnt/ebs -s 100 -v gp2 -t ext4)",
"trap "set +e; umount /mnt/ebs || umount -l /mnt/ebs; batchit ddv $vid;" EXIT",
"export BATCH_SCRIPT=$(mktemp)",
"echo "$B64GZ" | base64 -d | gzip -dc > $BATCH_SCRIPT",
"chmod +x $BATCH_SCRIPT",
"$BATCH_SCRIPT" ],
"Memory": 300000
},
"JobDefinition": "i3-test"
}
POST /v1/submitjob
HTTP/1.1 Content-type: application/json

{
"Parameters": null,
"DependsOn": null,
{ "Name": "cpus", "Value": "2" },
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
"Memory": 300000
},
}
batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
POST /v1/submitjob
{ "Name": "cpus", "Value": "16" },
{ "Name": "cpus", "Value": "16" },

{
"Parameters": null,
"DependsOn": null,
{ "Name": "cpus", "Value": "2" },
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
"Memory": 300000
},
}
batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
POST /v1/submitjob
{ "Name": "cpus", "Value": "2" },
{ "Name": "cpus", "Value": "2" },

{
"Parameters": null,
"DependsOn": null,
{ "Name": "cpus", "Value": "2" },
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
"Memory": 300000
},
}
batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
POST /v1/submitjob
{ "Name": "cpus", "Value": "2" },

{
"Parameters": null,
"DependsOn": null,
{ "Name": "cpus", "Value": "2" },
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
"Memory": 300000
},
}
batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
POST /v1/submitjob
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],

batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=2"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
{
"Parameters": null,
"DependsOn": null,
{ "Name": "cpus", "Value": "2" },
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"trap "set +e; umount /mnt/ebs || umount -l /mnt/ebs; batchit ddv $vid; if [[ $si
"$BATCH_SCRIPT" ],
"Memory": 300000
},
}
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
/bin/bash -c
for i in "$@"; do eval "$i"; done
export vid=$(batchit ebsmount -n 1 -m /mnt/ebs -s 100 -v gp2 -t ext4)
trap "set +e;
umount /mnt/ebs || umount -l /mnt/ebs;
batchit ddv $vid;" EXIT
export BATCH_SCRIPT=$(mktemp)",
echo "$B64GZ" | base64 -d | gzip -dc > $BATCH_SCRIPT",
chmod +x $BATCH_SCRIPT
$BATCH_SCRIPT

batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=2"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
{
"Parameters": null,
"DependsOn": null,
{ "Name": "cpus", "Value": "2" },
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
"Memory": 300000
},
}
"Command": [
"/bin/bash", "-c",
"trap "set +e; umount /mnt/ebs || umount -l /mnt/ebs; batchit ddv $vid;;" EXIT",
"$BATCH_SCRIPT" ],
/bin/bash -c
trap "set +e;
$BATCH_SCRIPT

batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=2"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
{
"Parameters": null,
"DependsOn": null,
{ "Name": "cpus", "Value": "2" },
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
"Memory": 300000
},
}
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
Andrey Kislyuk https://github.com/kislyuk/aegea
/bin/bash -c
trap "set +e;
$BATCH_SCRIPT
{
"ContainerProperties": {
"MountPoints": [ { "SourceVolume": "vol00",
"ContainerPath": "/dev” } ],
"Volumes": [ { "Host": { "SourcePath": "/dev },
"Name": "vol00" } ],
"Privileged": true,
"Ulimits": [ { "SoftLimit": 40000,
"HardLimit": 40000 } ]
}
}

batchit
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam
batchit submit
--queue $queue
--envvars "cpus=2"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
{
"Parameters": null,
"DependsOn": null,
{ "Name": "cpus", "Value": "2" },
"Vcpus": 16,
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
"Memory": 300000
},
}
"Command": [
"/bin/bash", "-c",
"$BATCH_SCRIPT" ],
/bin/bash -c
trap "set +e;
$BATCH_SCRIPT
{
"ContainerProperties": {
"MountPoints": [ { "SourceVolume": "vol00",
"ContainerPath": "/dev" } ],
"Volumes": [ { "Host": { "SourcePath": "/dev" },
"Name": "vol00" } ],
"Privileged": true,
"Ulimits": [ { "SoftLimit": 40000,
"HardLimit": 40000 } ]
}
}
Andrey Kislyuk https://github.com/kislyuk/aegea

batchit submit
--queue $queue
--envvars "cpus=16"
"sample=SS-12345"
"reference=GRCh37"
--cpus 16
--mem 30000
alignment.sh
batchit
Returns job ID
# alignment.sh
${reference}.fa
${sample}.fq.gz
> ${sample}.sam

Raw sequence
50GB
alignment
- 32 vCPUs
- 160 CPU hours
m4.16xlarger4.8xlarge
c4.8xlarger4.16xlarge

i3.16xlarge

r4.16xlarge

Raw sequence
50GB
alignment50GB
100 GB / 6.67 CPU d
Aligned Genome
- 32 vCPUs
- 160 CPU hours

Raw sequence
50GB
alignment50GB
1,049 ✕ 50GB+50GB
1,049 ✕ 160 CPU hours
= 104.9 TB
= 19.2 CPU years
104.9 TB / 19.2 CPU y
Aligned Genome
- 32 vCPUs
- 160 CPU hours

Variant Calling: Sample-Parallel
Aligned Genome
104.9 TB / 19.2 CPU y
Raw sequence
50GB
alignment50GB
1,049 ✕ 50GB+50GB
1,049 ✕ 160 CPU hours
= 104.9 TB
= 19.2 CPU years
- 32 vCPUs
- 160 CPU hours
m4.16xlarg
e
r4.8xlarge

Aligned Genome
104.9 TB / 19.2 CPU y
small_variant
large_variant
- 32 vCPUs
- 128 CPU hours
- 2 vCPUs
- 4-8 CPU hours
m4.large
r4.2xlarge
r4.xlarge
c4.large
c4.xlarge
c4.2xlarge
r4.large
m4.xlarge
m4.2xlarge
50GB
50GB

Aligned Genome
50GB
small_variant
large_variant50GB 16GB
8GB
50GB
74GB
287.4 TB / 35.2 CPU y
Sample
Variants
- 32 vCPUs
- 128 CPU hours
- 2 vCPUs
- 4-8 CPU hours
m4.large
r4.2xlarge
r4.xlarge
c4.large
c4.xlarge
c4.2xlarge
r4.large
m4.xlarge
m4.2xlarge

287.4 TB / 35.2 CPU y
Sample
Variants
Joint Calling: Region-Parallel
Aligned Genome
50GB
small_variant
large_variant50GB 16GB
8GB
50GB
74GB- 32 vCPUs
- 128 CPU hours
- 2 vCPUs
- 4-8 CPU hours
m4.large
r4.2xlarge
r4.xlarge
c4.large
c4.xlarge
c4.2xlarge
r4.large
m4.xlarge
m4.2xlarge

Sample
Variants
287.4 TB / 35.2 CPU y© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

1,049
genomes
Sample
Variants
287.4 TB / 35.2 CPU y© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

joint_calling
Sample
Variants
287.4 TB / 35.2 CPU y

100MB
100MB
100MB
100MB
100MB
100MB
joint_calling
Sample
Variants
287.4 TB / 35.2 CPU y

105GB
joint_calling
Sample
Variants
287.4 TB / 35.2 CPU y

joint_calling105GB 50GB
Sample
Variants
287.4 TB / 35.2 CPU y
m4.16xlarge
- 64 vCPUs
- 100 GB EBS
- 4 CPU hours

1 of 80
Sample
Variants
287.4 TB / 35.2 CPU y
m4.16xlarge
- 64 vCPUs
- 100 GB EBS
- 4 CPU hours

80 ✕ 105GB+50GB
80 ✕ 4 CPU hours
= 12.4TB
= 13.3 CPU days1 of 80
Sample
Variants
299.8 TB / 35.2 CPU y
m4.16xlarge
- 64 vCPUs
- 100 GB EBS
- 4 CPU hours
Wall Time : 3.1 days

Academic Software and Exit Codes
$ bioinformatics-software
$input
> $output
# catastrophic silent ERROR resulting in truncated output
$ echo $?
0
Validate output at each step, create a sentinel file on S3
$ bioinformatics-software
$input
> $output
$ verify-output $output && touch $output.sentinel
$ aws s3 cp $output.sentinel s3://base2-sentinel/

base2mon

base2mon
sample1.fq
sample2.fq
…
sampleN.fq
alignment
small_variant
large_variant
joint_calling
Input file Operations

base2mon
sample1.fq
sample2.fq
…
sampleN.fq
alignment
small_variant
large_variant
joint_calling
sample1.fq
sample1.s.vcf
sample1.s.vcf.sentinel
sample1.l.vcf
sample1.l.vcf.sentinel
small_variant large_variant
sampleN.fq
sampleN.bam
sampleN.bam.sentinel
sampleN.s.vcf
sampleN.s.vcf.sentinel
sampleN.l.vcf
sampleN.l.vcf.sentinel
alignment
samples1_N.vcf
joint_calling
...
sample2.fq
sample2.bam
sample2.bam.sentinel
alignment
...
sample1.bam
sample1.bam.sentinel
alignment
small_variant large_variant

base2mon
sample1.fq
sample2.fq
…
sampleN.fq
alignment
small_variant
large_variant
joint_calling
Batch
t2.micro
S3
Data-dependent job planning, recovery, restart
Modularize and isolate 3rd party software
Reactive resubmit (e.g., more memory)
Per-instance time/cost monitoring

Dinu Bunduchi: Cloud Infrastructure Architect, Autodesk

© 2017 Autodesk
Engineering Outcome
The ultimate goal for any engineering activity is to strike the right
balance between performance and cost to produce for a given design
challenge

© 2017 Autodesk
Generative Design
Designers input design goals into
generative design software, along with
parameters such as materials,
manufacturing methods, and cost
constraints. Then, using cloud
computing, the software explores all the
possible permutations of a solution,
quickly generating design alternatives.

© 2017 Autodesk
 It’s a state machine
 Amazon SWF helps developers build, run, and scale
background jobs that have parallel or sequential steps.
You can think of Amazon SWF as a fully-managed
state tracker and task coordinator in the Cloud
Simple Workflow Service

© 2017 Autodesk
Architecture
API Server (ECS)
SWF Workflow
Job Manager (ECS)
Generate Variants
Solve Variants
start Workflow poll for Decision
submit Batch Job
poll for Activity
submit Batch Job
poll for Activity
task Completed
task Completed

© 2017 Autodesk
 Two managed Batch compute environments
 CPU cluster for generating variants (Instance Type: optimal)
 GPU cluster for variant solvers (Instance Type: p2) – custom AMI
 Two job queues, one per compute environment
AWS Batch

© 2017 Autodesk
{
"containerProperties": {
"command": [ "solver" ],
"image": "_ECR_IMAGE_TAG_",
"vcpus": 4,
"memory": 48000,
"environment": [ { "name": "SOLVER_TIMEOUT", "value": "120" } … ],
"mountPoints": [ {
"containerPath": "/usr/local/nvidia",
"readOnly": false,
"sourceVolume": "nvidia"
} ],
"privileged": true,
"ulimits": [ { "hardLimit": 65535, "name": "nofile", "softLimit": 65535 } ],
"volumes": [ {
"host": { "sourcePath": "/var/lib/nvidia-docker/volumes/nvidia_driver/latest" },
"name": "nvidia"
} ]
},
"jobDefinitionName": "prod-solver-job",
"type": "container"
}
AWS Batch - Solver Job Definition

© 2017 Autodesk
import (
"github.com/aws/aws-sdk-go/aws”
"github.com/aws/aws-sdk-go/aws/awserr”
"github.com/aws/aws-sdk-go/service/batch"
)
input := &batch.SubmitJobInput {
JobDefinition: aws.String( jobDef ),
JobName: aws.String( "variantGen” ),
JobQueue: aws.String( cfg.VariantGenBatchJobQueue ),
ContainerOverrides: &batch.ContainerOverrides {
Command: cmdParams,
Environment: environment,
},
}
b.BatchConn.SubmitJob( input )
AWS Batch - SubmitJob()

© 2017 Autodesk
Autodesk Generative Design Customers
Autodesk’s new Toronto office is the first example of a generatively
designed office space.

Autodesk and the Autodesk logo are registered trademarks or trademarks of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders.
Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document.
© 2017 Autodesk. All rights reserved.

Fully Managed Integrated with AWS Cost-optimized
Resource Provisioning

Important Links
Product Details:
https://aws.amazon.com/batch/details/
Getting Started:
https://aws.amazon.com/batch/getting-started/
Sample code for AWS Batch + Step Functions Integration:
https://github.com/awslabs/aws-batch-genomics
Compute Blog Post Describing How to Use Batch + FPGAs:
https://aws.amazon.com/blogs/compute/accelerating-precision-medicine-at-scale/
Deep Learning on AWS Batch:
https://aws.amazon.com/pt/blogs/compute/deep-learning-on-aws-batch/

Thank you!

CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services

Similar to CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services (20)

More from Amazon Web Services

More from Amazon Web Services (20)

CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services