Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Batch: Simplifying Batch Computing in the Cloud

522 views

Published on

AWS Batch: Simplifying Batch Computing in the Cloud

  • Be the first to comment

AWS Batch: Simplifying Batch Computing in the Cloud

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ian Robinson, Specialist SA, Analytics, EMEA 10 April 2018 AWS Batch: Simplifying Batch Computing in the Cloud
  2. 2. What is Batch Computing? Run jobs asynchronously and automatically across one or more computers. Jobs may have dependencies, making the sequencing and scheduling of multiple jobs complex and challenging.
  3. 3. Batch Computing Today • In-house compute clusters powered by open source or commercial job schedulers. • Often comprised of a large array of identical, undifferentiated processors, all of the same vintage and built to the same specifications.
  4. 4. AWS Batch • Fully managed batch processing • Enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS • Jobs executed as containerized applications • Dynamically provisions the optimal compute resources • Allows you to focus on analyzing results and solving problems
  5. 5. AWS Batch advantages Reduces operational complexities Saves time Reduces costs
  6. 6. AWS Batch Use Cases High Performance Computing Post-Trade Analytics Fraud Surveillance Drug Screening DNA Sequencing Rendering Transcoding Media Supply Chain
  7. 7. Financial Services: Automate the analysis of the day’s transaction for fraud surveillance.
  8. 8. Life Sciences: Drug Screening for Biopharma Rapidly search libraries of small molecules for drug discovery.
  9. 9. Digital Media: Visual Effects Rendering Automate content rendering workloads and reduce the need for human intervention due to execution dependencies or resource scheduling.
  10. 10. AWS Batch Components • Compute environments • Job queues • Job definitions • Jobs • Job Scheduler
  11. 11. Components relation Batch Compute Environment ** Batch Queue (2) Batch Queue (1) Batch Queue (0) Job Definition 1 Job Definition 2 Job Definition 3 Job Definition n priority Job 1 Job 2 Container Property Compute Resources DependsOn Container Property Container Property Container Property ** regional service
  12. 12. Compute Environment Job queues are mapped to one or more compute environments Managed You describe: • instance types – or choose “optimal” • min/max/desired vCPUs • Spot or On-Demand provisioning Batch launches and scales resources on your behalf Unmanaged Instances must include ECS agent and run supported operating systems and Docker versions You control the instances and scaling
  13. 13. Create Environment aws batch create-compute-environment --cli-input-json file://job_env.json --region us-east-1
  14. 14. Customer Provided AMIs Customer Provided AMIs let you set the AMI that is launched as part of a managed compute environment. Makes it possible to configure Docker settings, mount EBS/EFS volumes, and configure drivers for GPU jobs. AMIs must be Linux-based, HVM and have a working ECS agent installation.
  15. 15. Job Queue Jobs are submitted to a job queue, where they reside until they are able to be scheduled to a compute resource. Information related to completed jobs persists in the queue for 24 hours. Job queues support priorities and multiple queues can schedule work to the same compute environment.
  16. 16. Create Job Queue aws batch create-job-queue --region us-east-1 --cli-input-json file://job_queue.json
  17. 17. Job Definition AWS Batch job definitions specify how jobs are to be run. Some of the attributes specified in a job definition: • IAM role associated with the job • vCPU and memory requirements • Mount points • Container properties • Environment variables • Retry strategy • While each job must reference a job definition, many parameters can be overridden.
  18. 18. Create Job Definition aws batch register-job-definition --region us-east-1 --cli-input-json file://job_def.json
  19. 19. Jobs Jobs are the unit of work executed by AWS Batch as containerized applications running on Amazon EC2. Containerized jobs can reference a container image, command, and parameters. Or, users can fetch a .zip containing their application and run it on a Amazon Linux container.
  20. 20. Submit Job aws batch submit-job --cli-input-json file://submit_job.json --region us-east-1
  21. 21. Submit Job with dependency aws batch submit-job --cli-input-json file://submit_job.json --region us-east-1
  22. 22. Array Jobs Collection of jobs (between 2 and 10,000) that share common parameters, such as the job definition, vCPUs, and memory. Distributed across multiple hosts and may run concurrently SEQUENTIAL dependency First child job must succeed before the next child job starts N_TO_N dependency Each index child of this job must wait for the corresponding index child of each dependency to complete before it can begin
  23. 23. Job States Jobs submitted to a queue can have the following states: SUBMITTED: Accepted into the queue, but not yet evaluated for execution PENDING: Your job has dependencies on other jobs which have not yet completed RUNNABLE: Your job has been evaluated by the scheduler and is ready to run STARTING: Your job is in the process of being scheduled to a compute resource RUNNING: Your job is currently running SUCCEEDED: Your job has finished with exit code 0 FAILED: Your job finished with a non-zero exit code, was cancelled or terminated.
  24. 24. Job Scheduler The scheduler evaluates when, where, and how to run jobs that have been submitted to a job queue. Jobs run in approximately the order in which they are submitted, as long as all dependencies on other jobs have been met.
  25. 25. AWS Batch and CloudWatch Events Event stream – near real-time notifications regarding the current state of jobs: • Monitor the progress of jobs • Build custom workflows with complex dependencies • Generate usage reports or metrics around job execution • Build your own custom dashboards Jobs are available as CloudWatch Events targets: • Match events and submit AWS Batch jobs in response to them
  26. 26. Resource Limits
  27. 27. Demo

×