Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Azure Batch Service Meetup Presentation


Published on

An Intro To Azure Batch Service

  • Be the first to comment

  • Be the first to like this

Azure Batch Service Meetup Presentation

  1. 1. Azure Batch Service George Chrysovalantis Grammatikos Cloud Platform MCSA | Microsoft Azure MVP E-mail: Blog : TechNet Wiki Profile: George Chrysovalantis Grammatikos
  2. 2. What is Azure Batch Service? “ Use Azure Batch to run large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure. Azure Batch creates and manages a pool of compute nodes (virtual machines), installs the applications you want to run, and schedules jobs to run on the nodes. ”
  3. 3. Cloud Basic Principals Resources Self Service Scalability Billing by consumption
  4. 4. Microsoft Azure Virtual Machines Types General Purpose Compute Optimized Memory Optimized Storage Optimized GPU High Performance Compute
  5. 5. Azure Batch Features Resource Management Process Management Resource & Process Monitoring
  6. 6. • Node Management • Auto-Scaling • Low Priority Nodes • Applications Resource Management
  7. 7. Node Management Cloud Service Worker Roles Virtual Machines Third Party Applications
  8. 8. Auto-Scaling Configure the number of nodes in a pool Auto Scale Formula, based on service variables (CPU, Memory, Number of tasks) startingNumberOfVMs = 1; maxNumberofVMs = 25; pendingTaskSamplePercent = $PendingTasks.GetSamplePercent(180 * TimeInterval_Second); pendingTaskSamples = pendingTaskSamplePercent < 70 ? startingNumberOfVMs : avg($PendingTasks.GetSample(180 * TimeInterval_Second)); $TargetDedicatedNodes=min(maxNumberofVMs, pendingTaskSamples);
  9. 9. Low Priority Nodes
  10. 10. Applications • Zip Package containing executables • Deployed to nodes when required • Pools – Deployed to all nodes in pool • Task – Deployed to nodes running the task
  11. 11. Process Management Job and Task Management • Scheduling • Re-try tasks on failure • Max task duration • Tasks may depends from other tasks • Explicitly defined on task creation Task Dependencies Multi-Instance Tasks • Run tasks on multiple compute nodes • Inter-node communication required on pool
  12. 12. Resource & Process Monitoring Azure Portal Batch Explorer (Batch Labs) App Insights Metrics using API • Rest • .NET • Azure CLI • Azure PowerShell
  13. 13. Parallel Job Processing Job Tasks Result
  14. 14. 1. Upload Application, Input Files 2. Create Pool, Jobs, Tasks 3. Download App, Input File & Run Tasks 4. Monitor Tasks 5. Upload Output Files 6. Download Output Files Azure Batch Architecture
  15. 15. Azure Batch Architecture (Based On The Demo)
  16. 16. Azure Batch Processing Architecture (Based On The Demo) Application Jobs Input Files Output Files
  17. 17. Resource Quotas Resource Default Limit Maximum Limit Batch accounts per region per subscription 1-3 50 Dedicated cores per Batch account 10-100 N/A Low-priority cores per Batch account 10-100 N/A Active jobs and job schedules per Batch account 100-300 1000 Pools per Batch account 20-100 500
  18. 18. Pool Size Limits Resource Maximum Limit Compute nodes in inter-node communication enabled pool Batch service pool allocation mode 100 Batch subscription pool allocation mode 80 Compute nodes in Dedicated nodes 2000 Low-priority nodes 1000
  19. 19. NOT Supported VM Sizes Family Unsupported Sizes Basic A-series Basic_A0(A0) A-series Standard_A0 B-series All DC-series All Extreme memory optimized All Hb-series* All Hc-series* All Lsv2-series* All NDv2-series* All NVv2-series* All SAP-HANA All * Not currently supported, but will be supported in the future
  20. 20. Batch Account Credentials Storage Account Credentials Pool And Job Constants Batch .NET Project Basic Constants
  21. 21. Simulation Examples • Education: Monte Carlo Simulation (solve complex math problems) • Computer Software Industry: gaming • Dev / DevOps : (Common Test execution framework used to manage and • schedule tasks) • Motor Racing: F1(Complex simulation and predictions) • Economy: banks, • Animation Rendering (Single Frame is rendered as its own task)
  22. 22. Azure Batch Pricing Azure Batch is a Free service. Charges accrue for the underlying resources that run your Batch workloads. Configure Cloud Services and Virtual Machines to estimate the cost of your Batch pools. TIER: Low priority INSTANCE: A1: 1 cores(s), 1.75 GB RAM, 225 GB Temporary storage, €0.027/hour No Of Instances: 5 No Of Days: 1 Total Cost: € 3.64 Storage: Bandwitdth: € 0.02
  23. 23. Microsoft Azure Compute
  24. 24. Demo Azure Batch .NET File Processing With ffmpeg
  25. 25. Useful Tools & Links Azure Batch Tools  Azure Portal  Azure Batch GitHub  Azure CLI  Azure Storage Explorer  AzCopy Tool  Blobxfer Tool  Azure Batch Explorer Docker Support  Batch Shipyard Feedback & Forums  Feedback Pricing  Cloud Services  Windows Virtual Machines  Linux Virtual Machines
  26. 26. Resources  Azure Batch Documentation  Azure Batch Code Samples Repo  Batch service quotas and limits Jump  VM sizes for compute nodes in an Azure Batch pool Jump  Tutorial: Run a parallel workload with Azure Batch using the .NET API Jump  Azure Batch Videos - Channel 9 Jump   Create an automatic scaling formula for scaling compute nodes in a Batch pool Jump  Batch computing at a fraction of the price  Real-world Applications of Simulations  Simulation Examples
  27. 27. Help us improve and organize even better meetups