Downloaded 21 times
Uploaded byCoreOS
Tectonic Summit 2016: Multi-Cluster Kubernetes: Planning for Unknowns
- Concur is a travel and expense management company with 6500+ employees and offices worldwide. They process over 70 million transactions and $50 billion in travel and expense spend annually. - The presenter is a Principal Architect at Concur who has been working with Kubernetes since 2015. He discusses why Concur chose Kubernetes and CoreOS for container orchestration. - Concur runs multiple Kubernetes clusters across different regions for high availability. A custom tool called kube2cnqr manages load balancing between clusters.
More Related Content
Similar to Tectonic Summit 2016: Multi-Cluster Kubernetes: Planning for Unknowns
Tectonic Summit 2016: Multi-Cluster Kubernetes: Planning for Unknowns
- 2. Agenda • Why k8s? •Why CoreOS? • K8s @ Concur • Q & A
- 3. What is Concur? 6500+employees Offices: Australia China Czech Republic France Germany Holland India Japan Philippines Singapore Sweden UK US HQ: Bellevue 70m transactions 50b+ USD T&E spend annually €59m New cloud bookings Q3 2015 Cloud DNA 99 Established 1993 > 60% Fortune 500
- 4. 2011 2012 20132014 2015 2016 20182017 Expense transactions Bookings growth 2013 2015 55 million 77+ million Provisioned users 35+ million 2015 Registered mobile users 2013 2015 2.1 million 5+ million Concur growth
- 5. A little aboutme Principal Architect • Concur since 1998 in various roles (DBA, Ops Manager & Architect, etc.) • Corking w/ k8s since mid 2015 • contributor to k8s and participant in federation & service catalog SIGs • Email danwilson on github, k8s slack & gmail
- 6. Why Kubernetes? • Largestcommunity support • Gifted to Linux Foundation • Protection from shifts in container technology • Pluggable API • Capable of isolation between namespaces • Declarative syntax for the complete make up of services • Built in logging, service discovery, etc. • Networking design eliminates port conflicts of containers
- 7. Why CoreOS? • Simpleupdates by channel • Designed to run as a cluster • Low overhead • Best documentation for k8s
- 8. Why CoreOS? • Simpleupdates by channel • Designed to run as a cluster • Low overhead • Best documentation for k8s
- 9.
- 10. High Availability • Allthings fail, yes everything! • K8s HA guide: http://kubernetes.io/docs/admin/high-availability/ • cluster etcd – 3, 5 or 7 nodes • replicated + load balanced api servers • many minions
- 11. High Availability If cluster== alwaysOnline { glog.V(1). Infof(“nothing to do here!”) }
- 12. staging prod N clusters CI K8s@ Concur
- 13. Location US EUROPE ZoneA B C A B C Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c K8s @ Concur
- 14.
- 15. K8s @ Concur •kube2cnqr • Golang docker container • Watches the k8s API for updates to services and worker nodes • Calls a Concur internal API to addremove load balancer entries as needed • Handles connecting multiple k8s clusters to a single load balancer endpoint • Extras • External-IP set on service objects • Annotation w/ DNS name set on service objects • Handles multiple load balancer tiers • Allow services to specify DNS name
- 16. Sample project • https://github.com/concur/kubegowatcher •A sample golang program that leverages kubernetes watch endpoints using the client-go library. • Designed to run as a container on k8s • example of setting annotation on the service • example of checking event delay • Apache 2.0 license • Add your own business logic • Contribute back improvements to make it better
- 17. Location US EUROPE ZoneA B C A B C Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c K8s @ Concur – k8s upgrades
- 18. Location US EUROPE ZoneA B C A B C Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c Cluster v.1.3 us13a us13b us13c eur13a eur13b eur13c K8s @ Concur – k8s upgrades
- 19. K8s @ Concur DeploymentChallenges • Stale kubectl scripts • Stale kubectl config • Handling credentials for each ci system • Many commands for each cluster • Handling proxy
- 20.
- 21. Deployment API CI Deploy petshop:v1 to test test14a,b & c K8s API calls for service & deployment
- 22. Deployment API CI Deploy petshop:v1 to prod US14a,b & c EMEA14a, b & c APAC14a, b & c K8s API calls for service & deployment
- 23.
- 24. Setup API curl /setup Outputkubectl config commands for all cluster zones requested | bash
- 25.
- 26. K8s @ Concur Addfeatures to kubernetes federation • v1.6 – deploying to clusters based on label #29887 Other potentials… • Location specific overrides • Global overrides • Feature limitations (for example, don’t allow pods/rc’s without a deployment) Docs: https://github.com/concur/skipper
- 27.
- 28. Q & A Emaildanwilson on github, k8s slack & gmail
Editor's Notes
- #5 Tripit Hipmunk
- #7 Tripit Hipmunk
- #10 Explain hypercar components and how they relate to building a kubernetes cluster. CoreOS makes it easy by already having the pieces tested together (kernel, flannel, etcd, systemd, etc)
- #12 Plan for unknown unknowns Locking thresholds or bad query plans are known unknowns
- #13 kubelet to container which messed up dns settings
- #16 Terraform deployments CoreOS Flannel Custom logstash container
- #17 Updated to use client-go library From centurylink/ca-certs Working w/scratch Gotchas: checking for the delay on events and handling updates made
- #18 Docker golang setup for kubernetes f5 plugin
- #19 Best way to get to v.1.3? Blue/green
- #21 So, we created skipper
- #23 So what does this skipper thing do anyways?
- #24 So what does this skipper thing do anyways?
- #25 So what does this skipper thing do anyways?
- #26 So what does this skipper thing do anyways?