Presented by Kevin Hannon, Open Source Developer, G-Research Open Source, at Kubernetes Community Days, Washington DC, September 14, 2022 ● Background in Chemistry and High Performance Computing ● Entrypoint into Kubernetes was focused on enabling running scientific workflows across high performance computing clusters and/or Kubernetes clusters ● Working at G-Research Open Source focused on enabling batch workloads on multiple kubernetes clusters

1. Armada
Batch Workloads Across 10s of Kubernetes Clusters

2. Who am I
● Background in Chemistry and High Performance Computing
● Entrypoint into Kubernetes was focused on enabling running scientific
workflows across high performance computing clusters and/or Kubernetes
clusters
● Working at G-Research Open Source focused on enabling batch workloads
on multiple kubernetes clusters

3. G-Research
● FinTech company based out of London (plus an office in Dallas)
● Builds/Operates a large research cluster for quantitative researchers

4. Batch Introduction
● Finite lifetime computational jobs
● Computational Science
○ Molecular Modeling, Protein Folding, Fluid Dynamics, etc
● Machine Learning
● ETL/ELT data workloads
● Genomics

5. Embarrassingly Parallel
Input 1
Input 2
Input x
Process 1
Process 2
Process x
Gather
Data

6. Parallel Computing
Request 2 nodes
Process 1 Process 2
● Message Passing Interface (MPI)
● GPU to GPU communication

8. SLURM HTCondor

10. Research Platform
● Researchers run large numbers of batch jobs.
● Jobs are being ported from HTCondor on Windows to Kubernetes on Linux
● Need a queue/scheduler to which jobs are submitted to run “somewhere”
● Target:
○ Manage 100k nodes, 1 M cores and many GPUs
○ Handle submitting ~10k jobs over a “short” span
○ Schedule jobs from single-core to multi-node
○ Divide resources fairly between users
○ Provide detailed visibility for users and admins

11. Kubernetes Jobs

12. Batch Processing With Kubernetes
● 10k nodes with Kubernetes is nontrivial
○ https://openai.com/blog/scaling-kubernetes-to-7500-nodes/
● Lack of Queuing system
● Initial support on Job API was for parallel jobs with little communication
● Scheduling is for individual pods

13. Kubernetes Batch Processing Projects
● Volcano
○ Extend Kubernetes via Custom Resource Definitions
○ Single cluster
○ Own scheduler
○ Queue
● Armada
○ Multiple kubernetes clusters
○ Use Kubernetes scheduler
○ Queue

14. Extend Kubernetes for batch

15. Armada Solution
● Use multiple kubernetes clusters
● Kubernetes native scheduler
● Use of pods rather than jobs
● Subscribe to events for a given grouping of jobs

16. Core Concepts
Job bag of K8s resources (e.g., a pod spec + ingress) to be created +
metadata.
Job Set group of jobs managed as a unit.
Queue represents a user or project- resources are divided fairly between queues.
user guide for more details

17. Armada Architecture

18. Queue
● Queue specifies resource
requirements
● RBAC for queue

19. Job Spec
● API takes a pod spec
● Can use common K8
resources, taints, tolerations,
etc

20. Clients
● DotNet client
● Python client
● Go client
● CLI

24. Lookout
● React UI for monitoring progress

25. Lookout

26. Roadmap
● Adding ability to support gang scheduling (all or nothing)
○ Extend the API to allow grouping of jobs
● Preemption
● Better Observability/Admin support
● Future Integrations with open source software

27. Getting Involved
● Kubernetes Working Group Focused on Adding Batch Capabilities
○ https://github.com/kubernetes/community/tree/master/wg-batch
○ Adding Queueing, PodGroups, Enhancing JobApi
● Research User Group
○ CNCF group focusing on enabling research workloads on cloud native architecture
○ https://github.com/cncf/research-user-group
● CNCF Batch Working Group
○ https://github.com/cncf/tag-runtime/blob/master/wg/bsi.md
○ Best practices for batch in K8, etc
● Armada Github
○ https://github.com/G-Research/armada
○ Slack Channel https://cloud-native.slack.com/archives/C03T9CBCEMC

28. Questions?