SlideShare a Scribd company logo
1 of 38
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Andres Silva, Sr. Technical Account Manager, AWS Enterprise Support
Introducing AWS Batch
A highly-efficient, dynamically-scaled, batch computing service
Agenda
• A brief history of batch computing
• AWS Batch overview and concepts
• Use cases
• Let’s take it for a spin!
• Q&A
What is batch computing?
Run jobs asynchronously and automatically across one or
more computers.
Jobs may dependencies, making the sequencing and
scheduling of multiple jobs complex and challenging.
Batch computing has been around for a while…
Images from: history.nasa.gov
Early Batch APIs
CRAY-1: 1976
• First commercial
supercomputer
• 167 millions
calculations/second
• USD$8.86 million
($7.9 million plus
$1 million for disk)
CRAY-1 on display in the hallways of the EPFL in Lausanne. https://commons.wikimedia.org/wiki/File:Cray_1_IMG_9126.jpg
Early Batch on AWS: NY Times TimesMachine
aws.amazon.com/blogs/aws/new-york-times/
In 2007 the New York Times
processed 130 years of archives in
36 hours.
11 million articles & 4TB of data
AWS services used:
Amazon S3, SQS, EC2, and EMR
Total cost (in 2007): $890
$240 compute + $650 storage
http://open.blogs.nytimes.com/2007/11/01/self-service-
prorated-super-computing-fun/
Batch Computing On-Premises
RAM
I/O
CPU CPU CPU RAM CPU
CPU RAM
I/O
CPU CPU RAM
RAM
I/O
CPU
CPU
CPU
I/O
RAM
CPU RAM
I/O
CPU
RAM
I/O
GPU
StorageCPU FPGA
How does this work in the cloud?
RAM
I/O
GPU
StorageCPU FPGA
R4
I3
P2
D2C5 F1
However, batch computing could be easier…
AWS
Components:
• EC2
• Spot Fleet
• Auto-Scaling
• SNS
• SQS
• CloudWatch
• AWS Lambda
• S3
• DynamoDB
• API Gateway
• …
Introducing AWS Batch
Fully Managed
No software to install or
servers to manage. AWS
Batch provisions,
manages, and scales your
infrastructure
Integrated with AWS
Natively integrated with the
AWS Platform, AWS Batch
jobs can easily and securely
interact with services such as
Amazon S3, DynamoDB, and
Rekognition
Cost-optimized
Resource Provisioning
AWS Batch automatically
provisions compute
resources tailored to the
needs of your jobs using
Amazon EC2 and EC2 Spot
Introducing AWS Batch
• Fully-managed batch primitives
• Focus on your applications (shell scripts,
Linux executables, Docker images) and
their resource requirements
• We take care of the rest!
AWS Batch Concepts
• Jobs
• Job Definitions
• Job Queue
• Compute Environments
• Scheduler
Jobs
Jobs are the unit of work executed by AWS Batch as containerized
applications running on Amazon EC2.
Containerized jobs can reference a container image, command, and
parameters or users can simply provide a .zip containing their
application and we will run it on a default Amazon Linux container.
$ aws batch submit-job --job-name variant-calling
--job-definition gatk --job-queue genomics
Easily run massively parallel jobs
Today, users can submit a large number of independent “simple jobs.”
Soon, we will add support for “array jobs” that run many copies of an
application against an array of elements.
Array jobs are an efficient way to run:
• Parametric sweeps
• Monte Carlo simulations
• Processing a large collection of objects
These use-cases are still possibly today. Simply submit more jobs.
Workflows and Job Dependencies
Workflows, Pipelines, and Job Dependencies
Jobs can express a dependency on the successful
completion of other jobs or specific elements of an
array job.
Use your preferred workflow engine and language to
submit jobs. Flow-based systems simply submit jobs
serially, while DAG-based systems submit many jobs
at once, identifying inter-job dependencies.
$ aws batch submit-job –depends-on 606b3ad1-aa31-48d8-92ec-f154bfc8215f ...
Job Definitions
Similar to ECS Task Definitions, AWS Batch Job Definitions specify how
jobs are to be run. While each job must reference a job definition, many
parameters can be overridden.
Some of the attributes specified in a job definition:
• IAM role associated with the job
• vCPU and memory requirements
• Mount points
• Container properties
• Environment variables
$ aws batch register-job-definition --job-definition-name gatk
--container-properties ...
Job Queues
Jobs are submitted to a Job Queue, where they reside until they are
able to be scheduled to a compute resource. Information related to
completed jobs persists in the queue for 24 hours.
$ aws batch create-job-queue --job-queue-name genomics
--priority 500 --compute-environment-order ...
Compute Environments
Job queues are mapped to one or more Compute Environments
containing the EC2 instances used to run containerized batch jobs.
Managed compute environments enable you to describe your business
requirements (instance types, min/max/desired vCPUs, and EC2 Spot bid
as a % of On-Demand) and we launch and scale resources on your behalf.
You can choose specific instance types (e.g. c4.8xlarge), instance families
(e.g. C4, M4, R3), or simply choose “optimal” and AWS Batch will launch
appropriately sized instances from our more-modern instance families.
Compute Environments
Alternatively, you can launch and manage your own resources within an
Unmanaged compute environment. Your instances need to include the
ECS agent and run supported versions of Linux and Docker.
AWS Batch will then create an Amazon ECS cluster which can accept the
instances you launch. Jobs can be scheduled to your Compute
Environment as soon as your instances are healthy and register with the
ECS Agent.
$ aws batch create-compute-environment --compute-
environment-name unmanagedce --type UNMANAGED ...
AWS Batch Concepts
The Scheduler evaluates when, where, and
how to run jobs that have been submitted to
a job queue.
Jobs run in approximately the order in which
they are submitted as long as all
dependencies on other jobs have been met.
Job States
Jobs submitted to a queue can have the following states:
SUBMITTED: Accepted into the queue, but not yet evaluated for execution
PENDING: Your job has dependencies on other jobs which have not yet completed
RUNNABLE: Your job has been evaluated by the scheduler and is ready to run
STARTING: Your job is in the process of being scheduled to a compute resource
RUNNING: Your job is currently running
SUCCEEDED: Your job has finished with exit code 0
FAILED: Your job finished with a non-zero exit code or was cancelled or
terminated.
AWS Batch Actions
Jobs:
SubmitJob
ListJobs
DescribeJobs
CancelJob
TerminateJob
Job Definitions:
RegisterJobDefinition
DescribeJobDefinitions
DeregisterJobDefinition
Job Queues:
CreateJobQueue
DescribeJobQueues
UpdateJobQueue
DeleteJobQueue
Compute Environments:
CreateComputeEnvironment
DescribeComputeEnvironments
UpdateComputeEnvironment
DeleteComputeEnvironment
AWS Batch Actions
CancelJob: Marks jobs that are not yet STARTING as FAILED.
TerminateJob: Cancels jobs that are currently waiting in the
queue. Stops jobs that are in a STARTING or RUNNING state
and transitions them to FAILED.
Requires a “reason” which is viewable via DescribeJobs
$ aws batch cancel-job --reason “Submitted to wrong queue”
--jobId= 8a767ac8-e28a-4c97-875b-e5c0bcf49eb8
AWS Batch Pricing
There is no charge for AWS Batch; you only pay for the
underlying resources that you consume!
AWS Batch Availability
• AWS Batch is GA in the US East (Northern Virginia) Region
• Support for Array Jobs and jobs executed as AWS Lambda
functions coming soon!
Use the Right Tool for the Job
Not all batch workloads are the same…
ETL and Big Data processing/analytics?
Consider EMR, Data Pipeline, Redshift, and related services.
Lots of small Cron jobs? AWS Batch is a great way to execute these jobs, but
you will likely want a workflow or job-scheduling system to orchestrate job
submissions.
Efficiently run lots of big and small compute jobs on heterogeneous
compute resources? That’s why we’re here!
Some ideas for inspiration
DNA Sequencing
Media Encoding/Transcoding
Would you like to see a demo?
Fully Managed Integrated with AWS Cost-optimized
Resource Provisioning
Any questions?

More Related Content

What's hot

Azure Introduction
Azure IntroductionAzure Introduction
Azure Introductionbrunoterkaly
 
SRV308 Deep Dive on Amazon Aurora
SRV308 Deep Dive on Amazon AuroraSRV308 Deep Dive on Amazon Aurora
SRV308 Deep Dive on Amazon AuroraAmazon Web Services
 
Behind the Scenes: Exploring the AWS Global Network (NET305) - AWS re:Invent ...
Behind the Scenes: Exploring the AWS Global Network (NET305) - AWS re:Invent ...Behind the Scenes: Exploring the AWS Global Network (NET305) - AWS re:Invent ...
Behind the Scenes: Exploring the AWS Global Network (NET305) - AWS re:Invent ...Amazon Web Services
 
AZ-204 : Implement Azure security
AZ-204 : Implement Azure securityAZ-204 : Implement Azure security
AZ-204 : Implement Azure securityAzureEzy1
 
What is Microsoft Azure used for?-Microsoft azure
What is Microsoft Azure used for?-Microsoft azure What is Microsoft Azure used for?-Microsoft azure
What is Microsoft Azure used for?-Microsoft azure Zabeel Institute
 
Getting Started with AWS Lambda and Serverless
Getting Started with AWS Lambda and ServerlessGetting Started with AWS Lambda and Serverless
Getting Started with AWS Lambda and ServerlessAmazon Web Services
 
202201 AWS Black Belt Online Seminar Apache Spark Performnace Tuning for AWS ...
202201 AWS Black Belt Online Seminar Apache Spark Performnace Tuning for AWS ...202201 AWS Black Belt Online Seminar Apache Spark Performnace Tuning for AWS ...
202201 AWS Black Belt Online Seminar Apache Spark Performnace Tuning for AWS ...Amazon Web Services Japan
 
AWS Tutorial | AWS Certified Solutions Architect | Amazon AWS | AWS Training ...
AWS Tutorial | AWS Certified Solutions Architect | Amazon AWS | AWS Training ...AWS Tutorial | AWS Certified Solutions Architect | Amazon AWS | AWS Training ...
AWS Tutorial | AWS Certified Solutions Architect | Amazon AWS | AWS Training ...Edureka!
 
Introduce AWS Lambda for newbie and Non-IT
Introduce AWS Lambda for newbie and Non-ITIntroduce AWS Lambda for newbie and Non-IT
Introduce AWS Lambda for newbie and Non-ITChitpong Wuttanan
 
Optimización de costos en migraciones a la nube AWS
Optimización de costos en migraciones a la nube AWSOptimización de costos en migraciones a la nube AWS
Optimización de costos en migraciones a la nube AWSAmazon Web Services LATAM
 
Building a well-engaged and secure AWS account access management - FND207-R ...
 Building a well-engaged and secure AWS account access management - FND207-R ... Building a well-engaged and secure AWS account access management - FND207-R ...
Building a well-engaged and secure AWS account access management - FND207-R ...Amazon Web Services
 
AWS Direct Connect 및 VPN을 이용한 클라우드 아키텍쳐 설계:: Steve Seymour :: AWS Summit Seou...
AWS Direct Connect 및 VPN을 이용한 클라우드 아키텍쳐 설계:: Steve Seymour :: AWS Summit Seou...AWS Direct Connect 및 VPN을 이용한 클라우드 아키텍쳐 설계:: Steve Seymour :: AWS Summit Seou...
AWS Direct Connect 및 VPN을 이용한 클라우드 아키텍쳐 설계:: Steve Seymour :: AWS Summit Seou...Amazon Web Services Korea
 
Amazon EKS - Elastic Container Service for Kubernetes
Amazon EKS - Elastic Container Service for KubernetesAmazon EKS - Elastic Container Service for Kubernetes
Amazon EKS - Elastic Container Service for KubernetesAmazon Web Services
 
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법Amazon Web Services Korea
 
모놀리스에서 마이크로서비스 아키텍처로의 전환 전략::박선용::AWS Summit Seoul 2018
모놀리스에서 마이크로서비스 아키텍처로의 전환 전략::박선용::AWS Summit Seoul 2018모놀리스에서 마이크로서비스 아키텍처로의 전환 전략::박선용::AWS Summit Seoul 2018
모놀리스에서 마이크로서비스 아키텍처로의 전환 전략::박선용::AWS Summit Seoul 2018Amazon Web Services Korea
 

What's hot (20)

Azure App Service Deep Dive
Azure App Service Deep DiveAzure App Service Deep Dive
Azure App Service Deep Dive
 
AWS Deployment Best Practices
AWS Deployment Best PracticesAWS Deployment Best Practices
AWS Deployment Best Practices
 
Introduction to Microsoft Azure 101
Introduction to Microsoft Azure 101Introduction to Microsoft Azure 101
Introduction to Microsoft Azure 101
 
Azure Introduction
Azure IntroductionAzure Introduction
Azure Introduction
 
SRV308 Deep Dive on Amazon Aurora
SRV308 Deep Dive on Amazon AuroraSRV308 Deep Dive on Amazon Aurora
SRV308 Deep Dive on Amazon Aurora
 
Deep Dive on AWS Lambda
Deep Dive on AWS LambdaDeep Dive on AWS Lambda
Deep Dive on AWS Lambda
 
Behind the Scenes: Exploring the AWS Global Network (NET305) - AWS re:Invent ...
Behind the Scenes: Exploring the AWS Global Network (NET305) - AWS re:Invent ...Behind the Scenes: Exploring the AWS Global Network (NET305) - AWS re:Invent ...
Behind the Scenes: Exploring the AWS Global Network (NET305) - AWS re:Invent ...
 
IaC on AWS Cloud
IaC on AWS CloudIaC on AWS Cloud
IaC on AWS Cloud
 
AZ-204 : Implement Azure security
AZ-204 : Implement Azure securityAZ-204 : Implement Azure security
AZ-204 : Implement Azure security
 
What is Microsoft Azure used for?-Microsoft azure
What is Microsoft Azure used for?-Microsoft azure What is Microsoft Azure used for?-Microsoft azure
What is Microsoft Azure used for?-Microsoft azure
 
Getting Started with AWS Lambda and Serverless
Getting Started with AWS Lambda and ServerlessGetting Started with AWS Lambda and Serverless
Getting Started with AWS Lambda and Serverless
 
202201 AWS Black Belt Online Seminar Apache Spark Performnace Tuning for AWS ...
202201 AWS Black Belt Online Seminar Apache Spark Performnace Tuning for AWS ...202201 AWS Black Belt Online Seminar Apache Spark Performnace Tuning for AWS ...
202201 AWS Black Belt Online Seminar Apache Spark Performnace Tuning for AWS ...
 
AWS Tutorial | AWS Certified Solutions Architect | Amazon AWS | AWS Training ...
AWS Tutorial | AWS Certified Solutions Architect | Amazon AWS | AWS Training ...AWS Tutorial | AWS Certified Solutions Architect | Amazon AWS | AWS Training ...
AWS Tutorial | AWS Certified Solutions Architect | Amazon AWS | AWS Training ...
 
Introduce AWS Lambda for newbie and Non-IT
Introduce AWS Lambda for newbie and Non-ITIntroduce AWS Lambda for newbie and Non-IT
Introduce AWS Lambda for newbie and Non-IT
 
Optimización de costos en migraciones a la nube AWS
Optimización de costos en migraciones a la nube AWSOptimización de costos en migraciones a la nube AWS
Optimización de costos en migraciones a la nube AWS
 
Building a well-engaged and secure AWS account access management - FND207-R ...
 Building a well-engaged and secure AWS account access management - FND207-R ... Building a well-engaged and secure AWS account access management - FND207-R ...
Building a well-engaged and secure AWS account access management - FND207-R ...
 
AWS Direct Connect 및 VPN을 이용한 클라우드 아키텍쳐 설계:: Steve Seymour :: AWS Summit Seou...
AWS Direct Connect 및 VPN을 이용한 클라우드 아키텍쳐 설계:: Steve Seymour :: AWS Summit Seou...AWS Direct Connect 및 VPN을 이용한 클라우드 아키텍쳐 설계:: Steve Seymour :: AWS Summit Seou...
AWS Direct Connect 및 VPN을 이용한 클라우드 아키텍쳐 설계:: Steve Seymour :: AWS Summit Seou...
 
Amazon EKS - Elastic Container Service for Kubernetes
Amazon EKS - Elastic Container Service for KubernetesAmazon EKS - Elastic Container Service for Kubernetes
Amazon EKS - Elastic Container Service for Kubernetes
 
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
 
모놀리스에서 마이크로서비스 아키텍처로의 전환 전략::박선용::AWS Summit Seoul 2018
모놀리스에서 마이크로서비스 아키텍처로의 전환 전략::박선용::AWS Summit Seoul 2018모놀리스에서 마이크로서비스 아키텍처로의 전환 전략::박선용::AWS Summit Seoul 2018
모놀리스에서 마이크로서비스 아키텍처로의 전환 전략::박선용::AWS Summit Seoul 2018
 

Similar to Introduction to AWS Batch

Announcing AWS Batch - Run Batch Jobs At Scale - December 2016 Monthly Webina...
Announcing AWS Batch - Run Batch Jobs At Scale - December 2016 Monthly Webina...Announcing AWS Batch - Run Batch Jobs At Scale - December 2016 Monthly Webina...
Announcing AWS Batch - Run Batch Jobs At Scale - December 2016 Monthly Webina...Amazon Web Services
 
Amazon Batch: 實現簡單且有效率的批次運算
Amazon Batch: 實現簡單且有效率的批次運算Amazon Batch: 實現簡單且有效率的批次運算
Amazon Batch: 實現簡單且有效率的批次運算Amazon Web Services
 
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing
 	  NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing 	  NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computingAmazon Web Services
 
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing on Amaz...
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing on Amaz...NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing on Amaz...
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing on Amaz...Amazon Web Services
 
New AWS Services for Bioinformatics
New AWS Services for BioinformaticsNew AWS Services for Bioinformatics
New AWS Services for BioinformaticsLynn Langit
 
AWS Batch: Simplifying Batch Computing in the Cloud
AWS Batch: Simplifying Batch Computing in the CloudAWS Batch: Simplifying Batch Computing in the Cloud
AWS Batch: Simplifying Batch Computing in the CloudAmazon Web Services
 
AWS Batch: Simplifying batch computing in the cloud
AWS Batch: Simplifying batch computing in the cloudAWS Batch: Simplifying batch computing in the cloud
AWS Batch: Simplifying batch computing in the cloudAdrian Hornsby
 
ECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container Day
ECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container DayECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container Day
ECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container DayAmazon Web Services Korea
 
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web ServicesCMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web ServicesAmazon Web Services
 
Nyc big datagenomics-pizarroa-sept2017
Nyc big datagenomics-pizarroa-sept2017Nyc big datagenomics-pizarroa-sept2017
Nyc big datagenomics-pizarroa-sept2017delagoya
 
Getting Started with Amazon EC2 Container Service
Getting Started with Amazon EC2 Container ServiceGetting Started with Amazon EC2 Container Service
Getting Started with Amazon EC2 Container ServiceAmazon Web Services
 
Introduction to Batch Processing on AWS
Introduction to Batch Processing on AWSIntroduction to Batch Processing on AWS
Introduction to Batch Processing on AWSAmazon Web Services
 
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017Amazon Web Services
 
Building Distributed Applications with AWS Step Functions
Building Distributed Applications with AWS Step FunctionsBuilding Distributed Applications with AWS Step Functions
Building Distributed Applications with AWS Step FunctionsAmazon Web Services
 
What is Amazon Web Services & How to Start to deploy your apps ?
What is Amazon Web Services & How to Start to deploy your apps ?What is Amazon Web Services & How to Start to deploy your apps ?
What is Amazon Web Services & How to Start to deploy your apps ?Sébastien ☁ Stormacq
 
Building and scaling your containerized microservices on Amazon ECS
Building and scaling your containerized microservices on Amazon ECSBuilding and scaling your containerized microservices on Amazon ECS
Building and scaling your containerized microservices on Amazon ECSAmazon Web Services
 
CMP209_Getting started with Docker on AWS
CMP209_Getting started with Docker on AWSCMP209_Getting started with Docker on AWS
CMP209_Getting started with Docker on AWSAmazon Web Services
 
Cloud Computing in PHP With the Amazon Web Services
Cloud Computing in PHP With the Amazon Web ServicesCloud Computing in PHP With the Amazon Web Services
Cloud Computing in PHP With the Amazon Web ServicesAmazon Web Services
 
Batch Processing with Containers on AWS - June 2017 AWS Online Tech Talks
Batch Processing with Containers on AWS -  June 2017 AWS Online Tech TalksBatch Processing with Containers on AWS -  June 2017 AWS Online Tech Talks
Batch Processing with Containers on AWS - June 2017 AWS Online Tech TalksAmazon Web Services
 

Similar to Introduction to AWS Batch (20)

Introduction to AWS Batch
Introduction to AWS BatchIntroduction to AWS Batch
Introduction to AWS Batch
 
Announcing AWS Batch - Run Batch Jobs At Scale - December 2016 Monthly Webina...
Announcing AWS Batch - Run Batch Jobs At Scale - December 2016 Monthly Webina...Announcing AWS Batch - Run Batch Jobs At Scale - December 2016 Monthly Webina...
Announcing AWS Batch - Run Batch Jobs At Scale - December 2016 Monthly Webina...
 
Amazon Batch: 實現簡單且有效率的批次運算
Amazon Batch: 實現簡單且有效率的批次運算Amazon Batch: 實現簡單且有效率的批次運算
Amazon Batch: 實現簡單且有效率的批次運算
 
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing
 	  NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing 	  NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing
 
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing on Amaz...
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing on Amaz...NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing on Amaz...
NEW LAUNCH! Introducing AWS Batch: Easy and efficient batch computing on Amaz...
 
New AWS Services for Bioinformatics
New AWS Services for BioinformaticsNew AWS Services for Bioinformatics
New AWS Services for Bioinformatics
 
AWS Batch: Simplifying Batch Computing in the Cloud
AWS Batch: Simplifying Batch Computing in the CloudAWS Batch: Simplifying Batch Computing in the Cloud
AWS Batch: Simplifying Batch Computing in the Cloud
 
AWS Batch: Simplifying batch computing in the cloud
AWS Batch: Simplifying batch computing in the cloudAWS Batch: Simplifying batch computing in the cloud
AWS Batch: Simplifying batch computing in the cloud
 
ECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container Day
ECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container DayECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container Day
ECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container Day
 
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web ServicesCMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
 
Nyc big datagenomics-pizarroa-sept2017
Nyc big datagenomics-pizarroa-sept2017Nyc big datagenomics-pizarroa-sept2017
Nyc big datagenomics-pizarroa-sept2017
 
Getting Started with Amazon EC2 Container Service
Getting Started with Amazon EC2 Container ServiceGetting Started with Amazon EC2 Container Service
Getting Started with Amazon EC2 Container Service
 
Introduction to Batch Processing on AWS
Introduction to Batch Processing on AWSIntroduction to Batch Processing on AWS
Introduction to Batch Processing on AWS
 
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
 
Building Distributed Applications with AWS Step Functions
Building Distributed Applications with AWS Step FunctionsBuilding Distributed Applications with AWS Step Functions
Building Distributed Applications with AWS Step Functions
 
What is Amazon Web Services & How to Start to deploy your apps ?
What is Amazon Web Services & How to Start to deploy your apps ?What is Amazon Web Services & How to Start to deploy your apps ?
What is Amazon Web Services & How to Start to deploy your apps ?
 
Building and scaling your containerized microservices on Amazon ECS
Building and scaling your containerized microservices on Amazon ECSBuilding and scaling your containerized microservices on Amazon ECS
Building and scaling your containerized microservices on Amazon ECS
 
CMP209_Getting started with Docker on AWS
CMP209_Getting started with Docker on AWSCMP209_Getting started with Docker on AWS
CMP209_Getting started with Docker on AWS
 
Cloud Computing in PHP With the Amazon Web Services
Cloud Computing in PHP With the Amazon Web ServicesCloud Computing in PHP With the Amazon Web Services
Cloud Computing in PHP With the Amazon Web Services
 
Batch Processing with Containers on AWS - June 2017 AWS Online Tech Talks
Batch Processing with Containers on AWS -  June 2017 AWS Online Tech TalksBatch Processing with Containers on AWS -  June 2017 AWS Online Tech Talks
Batch Processing with Containers on AWS - June 2017 AWS Online Tech Talks
 

More from Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Recently uploaded

ISO 25964-1Working Group ISO/TC 46/SC 9/WG 8
ISO 25964-1Working Group ISO/TC 46/SC 9/WG 8ISO 25964-1Working Group ISO/TC 46/SC 9/WG 8
ISO 25964-1Working Group ISO/TC 46/SC 9/WG 8Access Innovations, Inc.
 
Machine learning workshop, CZU Prague 2024
Machine learning workshop, CZU Prague 2024Machine learning workshop, CZU Prague 2024
Machine learning workshop, CZU Prague 2024Gokulks007
 
Juan Pablo Sugiura - eCommerce Day Bolivia 2024
Juan Pablo Sugiura - eCommerce Day Bolivia 2024Juan Pablo Sugiura - eCommerce Day Bolivia 2024
Juan Pablo Sugiura - eCommerce Day Bolivia 2024eCommerce Institute
 
Burning Issue presentation of Zhazgul N. , Cycle 54
Burning Issue presentation of Zhazgul N. , Cycle 54Burning Issue presentation of Zhazgul N. , Cycle 54
Burning Issue presentation of Zhazgul N. , Cycle 54ZhazgulNurdinova
 
Communication Accommodation Theory Kaylyn Benton.pptx
Communication Accommodation Theory Kaylyn Benton.pptxCommunication Accommodation Theory Kaylyn Benton.pptx
Communication Accommodation Theory Kaylyn Benton.pptxkb31670
 
Dynamics of Professional Presentationpdf
Dynamics of Professional PresentationpdfDynamics of Professional Presentationpdf
Dynamics of Professional Presentationpdfravleel42
 
Communication Accommodation Theory Kaylyn Benton.pptx
Communication Accommodation Theory Kaylyn Benton.pptxCommunication Accommodation Theory Kaylyn Benton.pptx
Communication Accommodation Theory Kaylyn Benton.pptxkb31670
 
The Real Story Of Project Manager/Scrum Master From Where It Came?!
The Real Story Of Project Manager/Scrum Master From Where It Came?!The Real Story Of Project Manager/Scrum Master From Where It Came?!
The Real Story Of Project Manager/Scrum Master From Where It Came?!Loay Mohamed Ibrahim Aly
 

Recently uploaded (8)

ISO 25964-1Working Group ISO/TC 46/SC 9/WG 8
ISO 25964-1Working Group ISO/TC 46/SC 9/WG 8ISO 25964-1Working Group ISO/TC 46/SC 9/WG 8
ISO 25964-1Working Group ISO/TC 46/SC 9/WG 8
 
Machine learning workshop, CZU Prague 2024
Machine learning workshop, CZU Prague 2024Machine learning workshop, CZU Prague 2024
Machine learning workshop, CZU Prague 2024
 
Juan Pablo Sugiura - eCommerce Day Bolivia 2024
Juan Pablo Sugiura - eCommerce Day Bolivia 2024Juan Pablo Sugiura - eCommerce Day Bolivia 2024
Juan Pablo Sugiura - eCommerce Day Bolivia 2024
 
Burning Issue presentation of Zhazgul N. , Cycle 54
Burning Issue presentation of Zhazgul N. , Cycle 54Burning Issue presentation of Zhazgul N. , Cycle 54
Burning Issue presentation of Zhazgul N. , Cycle 54
 
Communication Accommodation Theory Kaylyn Benton.pptx
Communication Accommodation Theory Kaylyn Benton.pptxCommunication Accommodation Theory Kaylyn Benton.pptx
Communication Accommodation Theory Kaylyn Benton.pptx
 
Dynamics of Professional Presentationpdf
Dynamics of Professional PresentationpdfDynamics of Professional Presentationpdf
Dynamics of Professional Presentationpdf
 
Communication Accommodation Theory Kaylyn Benton.pptx
Communication Accommodation Theory Kaylyn Benton.pptxCommunication Accommodation Theory Kaylyn Benton.pptx
Communication Accommodation Theory Kaylyn Benton.pptx
 
The Real Story Of Project Manager/Scrum Master From Where It Came?!
The Real Story Of Project Manager/Scrum Master From Where It Came?!The Real Story Of Project Manager/Scrum Master From Where It Came?!
The Real Story Of Project Manager/Scrum Master From Where It Came?!
 

Introduction to AWS Batch

  • 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Andres Silva, Sr. Technical Account Manager, AWS Enterprise Support Introducing AWS Batch A highly-efficient, dynamically-scaled, batch computing service
  • 2. Agenda • A brief history of batch computing • AWS Batch overview and concepts • Use cases • Let’s take it for a spin! • Q&A
  • 3. What is batch computing? Run jobs asynchronously and automatically across one or more computers. Jobs may dependencies, making the sequencing and scheduling of multiple jobs complex and challenging.
  • 4. Batch computing has been around for a while… Images from: history.nasa.gov
  • 6. CRAY-1: 1976 • First commercial supercomputer • 167 millions calculations/second • USD$8.86 million ($7.9 million plus $1 million for disk) CRAY-1 on display in the hallways of the EPFL in Lausanne. https://commons.wikimedia.org/wiki/File:Cray_1_IMG_9126.jpg
  • 7. Early Batch on AWS: NY Times TimesMachine aws.amazon.com/blogs/aws/new-york-times/ In 2007 the New York Times processed 130 years of archives in 36 hours. 11 million articles & 4TB of data AWS services used: Amazon S3, SQS, EC2, and EMR Total cost (in 2007): $890 $240 compute + $650 storage http://open.blogs.nytimes.com/2007/11/01/self-service- prorated-super-computing-fun/
  • 9. RAM I/O CPU CPU CPU RAM CPU CPU RAM I/O CPU CPU RAM
  • 12. How does this work in the cloud?
  • 14. However, batch computing could be easier… AWS Components: • EC2 • Spot Fleet • Auto-Scaling • SNS • SQS • CloudWatch • AWS Lambda • S3 • DynamoDB • API Gateway • …
  • 15. Introducing AWS Batch Fully Managed No software to install or servers to manage. AWS Batch provisions, manages, and scales your infrastructure Integrated with AWS Natively integrated with the AWS Platform, AWS Batch jobs can easily and securely interact with services such as Amazon S3, DynamoDB, and Rekognition Cost-optimized Resource Provisioning AWS Batch automatically provisions compute resources tailored to the needs of your jobs using Amazon EC2 and EC2 Spot
  • 16. Introducing AWS Batch • Fully-managed batch primitives • Focus on your applications (shell scripts, Linux executables, Docker images) and their resource requirements • We take care of the rest!
  • 17. AWS Batch Concepts • Jobs • Job Definitions • Job Queue • Compute Environments • Scheduler
  • 18. Jobs Jobs are the unit of work executed by AWS Batch as containerized applications running on Amazon EC2. Containerized jobs can reference a container image, command, and parameters or users can simply provide a .zip containing their application and we will run it on a default Amazon Linux container. $ aws batch submit-job --job-name variant-calling --job-definition gatk --job-queue genomics
  • 19. Easily run massively parallel jobs Today, users can submit a large number of independent “simple jobs.” Soon, we will add support for “array jobs” that run many copies of an application against an array of elements. Array jobs are an efficient way to run: • Parametric sweeps • Monte Carlo simulations • Processing a large collection of objects These use-cases are still possibly today. Simply submit more jobs.
  • 20. Workflows and Job Dependencies
  • 21. Workflows, Pipelines, and Job Dependencies Jobs can express a dependency on the successful completion of other jobs or specific elements of an array job. Use your preferred workflow engine and language to submit jobs. Flow-based systems simply submit jobs serially, while DAG-based systems submit many jobs at once, identifying inter-job dependencies. $ aws batch submit-job –depends-on 606b3ad1-aa31-48d8-92ec-f154bfc8215f ...
  • 22. Job Definitions Similar to ECS Task Definitions, AWS Batch Job Definitions specify how jobs are to be run. While each job must reference a job definition, many parameters can be overridden. Some of the attributes specified in a job definition: • IAM role associated with the job • vCPU and memory requirements • Mount points • Container properties • Environment variables $ aws batch register-job-definition --job-definition-name gatk --container-properties ...
  • 23. Job Queues Jobs are submitted to a Job Queue, where they reside until they are able to be scheduled to a compute resource. Information related to completed jobs persists in the queue for 24 hours. $ aws batch create-job-queue --job-queue-name genomics --priority 500 --compute-environment-order ...
  • 24. Compute Environments Job queues are mapped to one or more Compute Environments containing the EC2 instances used to run containerized batch jobs. Managed compute environments enable you to describe your business requirements (instance types, min/max/desired vCPUs, and EC2 Spot bid as a % of On-Demand) and we launch and scale resources on your behalf. You can choose specific instance types (e.g. c4.8xlarge), instance families (e.g. C4, M4, R3), or simply choose “optimal” and AWS Batch will launch appropriately sized instances from our more-modern instance families.
  • 25. Compute Environments Alternatively, you can launch and manage your own resources within an Unmanaged compute environment. Your instances need to include the ECS agent and run supported versions of Linux and Docker. AWS Batch will then create an Amazon ECS cluster which can accept the instances you launch. Jobs can be scheduled to your Compute Environment as soon as your instances are healthy and register with the ECS Agent. $ aws batch create-compute-environment --compute- environment-name unmanagedce --type UNMANAGED ...
  • 26. AWS Batch Concepts The Scheduler evaluates when, where, and how to run jobs that have been submitted to a job queue. Jobs run in approximately the order in which they are submitted as long as all dependencies on other jobs have been met.
  • 27. Job States Jobs submitted to a queue can have the following states: SUBMITTED: Accepted into the queue, but not yet evaluated for execution PENDING: Your job has dependencies on other jobs which have not yet completed RUNNABLE: Your job has been evaluated by the scheduler and is ready to run STARTING: Your job is in the process of being scheduled to a compute resource RUNNING: Your job is currently running SUCCEEDED: Your job has finished with exit code 0 FAILED: Your job finished with a non-zero exit code or was cancelled or terminated.
  • 28. AWS Batch Actions Jobs: SubmitJob ListJobs DescribeJobs CancelJob TerminateJob Job Definitions: RegisterJobDefinition DescribeJobDefinitions DeregisterJobDefinition Job Queues: CreateJobQueue DescribeJobQueues UpdateJobQueue DeleteJobQueue Compute Environments: CreateComputeEnvironment DescribeComputeEnvironments UpdateComputeEnvironment DeleteComputeEnvironment
  • 29. AWS Batch Actions CancelJob: Marks jobs that are not yet STARTING as FAILED. TerminateJob: Cancels jobs that are currently waiting in the queue. Stops jobs that are in a STARTING or RUNNING state and transitions them to FAILED. Requires a “reason” which is viewable via DescribeJobs $ aws batch cancel-job --reason “Submitted to wrong queue” --jobId= 8a767ac8-e28a-4c97-875b-e5c0bcf49eb8
  • 30. AWS Batch Pricing There is no charge for AWS Batch; you only pay for the underlying resources that you consume!
  • 31. AWS Batch Availability • AWS Batch is GA in the US East (Northern Virginia) Region • Support for Array Jobs and jobs executed as AWS Lambda functions coming soon!
  • 32. Use the Right Tool for the Job Not all batch workloads are the same… ETL and Big Data processing/analytics? Consider EMR, Data Pipeline, Redshift, and related services. Lots of small Cron jobs? AWS Batch is a great way to execute these jobs, but you will likely want a workflow or job-scheduling system to orchestrate job submissions. Efficiently run lots of big and small compute jobs on heterogeneous compute resources? That’s why we’re here!
  • 33. Some ideas for inspiration
  • 36. Would you like to see a demo?
  • 37. Fully Managed Integrated with AWS Cost-optimized Resource Provisioning

Editor's Notes

  1. Today, customers often build batch solutions using a combination of AWS services such as EC2, ECS, ECR, S3, SQS, SNS, AutoScaling, EBS, DynamoDB, and Cloudwatch.
  2. High-Throughput: Can process as many concurrent genomic workflows as needed (>1000 day). Flexible: You define your containers, dependencies, and resource requirements. Batch takes care of the rest. Elastic and Scalable: Treat each workflow like a burst compute. Pay only for what you need when you need it. Cost-Optimized: Runs on spot-fleet to significantly reduce cost of genomic analysis.