Recommended
More Related Content
What's hot
What's hot (20)
Similar to Multi-Cloud Federated Kubernetes at CERN
Similar to Multi-Cloud Federated Kubernetes at CERN (20)
Recently uploaded
Recently uploaded (20)
Multi-Cloud Federated Kubernetes at CERN
- 1. Multi-Cloud Federated Kubernetes at CERN Clenimar Filemon clenimar@lsd.ufcg.edu.br Ricardo Rocha ricardo.rocha@cern.ch
- 2. Founded in 1954 What is 96% of the universe made of? Fundamental Science Why isn’t there anti-matter in the universe? What was the state of matter just after the Big Bang?
- 7. ~40 MHz ~ 1PB/sec L1 Trigger ~ 100 kHz HL Trigger Collisions Hardware Filter Software Filter ~ 1 kHz Raw Data ~ 1-10 GB/s Huge Data Still Big Still Big
- 8. 320 000 Cores 3 300 Users 4 300 Projects 10 000 Hypervisors 210 Kubernetes Clusters 250 Petabytes
- 9. 200+ Sites 700 000 Cores ~400 000 Jobs Distributed Computing ~30 GiB/s CERN T1 T2 ... ... ... ... ... ... ... ... Reconstruction Calibration Simulation Analysis
- 10. CMS Higgs Event, May 2012 ATLAS Higgs Analysis, May 2012
- 11. Motivation for Federation Periodic load spikes International Conferences, Reconstruction Campaigns Simplification Monitoring, Lifecycle, Alarms Deployment Uniform API, Replication, Load Balancing Use Cases: CERN Batch System, RECAST Analysis
- 12. Sched Collector Negotiator StartD AcctGroup = "ATLAS" JobPrio = 0 RequestCpus = 2 RequestMemory = 4260 ... CERNEnvironment = “production” Datacenter = “meyrin” HasMPI = true TotalCpus = 8 TotalMemory = 22500 ... Matchmaking with ClassAds Fair Share Preemption Running Virtualized Extensive Experience in HEP External Storage and Networking
- 13. Sched Collector Negotiator StartD AcctGroup = "ATLAS" JobPrio = 0 RequestCpus = 2 RequestMemory = 4260 ... CERNEnvironment = “production” Datacenter = “meyrin” HasMPI = true TotalCpus = 8 TotalMemory = 22500 ... Matchmaking with ClassAds Fair Share Preemption Running Virtualized Extensive Experience in HEP External Storage and Networking
- 14. Sched Negotiator Collector Host kubefed init fed --host-cluster-context=condor-host ...
- 15. kind: DaemonSet ... hostNetwork: true containers: - name: condor-startd image: .../cloud/condor-startd command: ["/usr/sbin/condor_startd", "-f"] securityContext: privileged: true livenessProbe: exec: command: - condor_who Sched Negotiator Collector Host StartD ... StartD ... StartD ... kubefed init fed --host-cluster-context=condor-host ... kubefed join --context fed tsystems --host-cluster-context condor-host --cluster-context tsystems
- 16. REANA / RECAST Reusable Analysis Platform Workflow Engine (Yadage) Each step a Kubernetes Job Integrated Monitoring & Logging Centralized Log Collection https://github.com/reanahubhttps://github.com/recast-hep https://github.com/diana-hep/yadage
- 18. Thank You Great Community, Amazing Tools Credits CERN OpenStack Cloud and Batch teams (Spyros Trigazis and all) Lukas Heinrich, REANA / RECAST Kelsey Hightower
Editor's Notes
- Dark energy + dark matter Quark gluon plasma, moments after the big bang
- Location, lake, alps mont blanc, swiss-french border Complex of accelerators, higher and higher energy. 27km circumference Two beams of protons travelling on different directions, close to speed of light
- Almost 10.000 magnets Kept in the ring thanks to these superconducting magnets Temperature kept at 1.9K (-271 celsius) to keep superconducting properties
- Sibling of ATLAS, with similar goals but different design 14.000 tons 20 meters long, 15x15
- Anti-matter Decelarator, creates anti-atoms to better understand its properties AMS experiment, launched on mission STS-134 (penultime shuttle mission), measures antimatter in cosmic rays
- 2 floors Historical building, 50 years old, from mainframes to racks Internet backbone, biggest in late 80s and early 90s
- Hierachical system, few big T1s and many smaller T2s
- Not a physicist, but learned to look for patterns in plots
- Based on htcondor, which HEP has decades of experience operating Component description, requests and resources published as classads Advanced functionality (fair share, pre-emption) Currently running mostly on virtualized resources Important: htcondor relies on an established storage and net infrastructure, handling compute only (which is what we try to federate)
- StartD is our first containerization goal, deployed at scale
- Host cluster, with the condor control plane One command only to establish federation
- StartD deployment as a daemonset, meaning we get one instance on every host Clusters are added again with one single command CVMFS caching for software distribution speed-up Data access and networking outside the scope of this exercise