Kubernetes at Concur: Agenda, Overview, Challenges & Solutions
•Download as PPTX, PDF•
1 like•529 views
- Concur is a travel and expense management company with 6500+ employees and offices worldwide. They process over 70 million transactions and $50 billion in travel and expense spend annually. - The presenter is a Principal Architect at Concur who has been working with Kubernetes since 2015. He discusses why Concur chose Kubernetes and CoreOS for container orchestration. - Concur runs multiple Kubernetes clusters across different regions for high availability. A custom tool called kube2cnqr manages load balancing between clusters.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (18)
Similar to Kubernetes at Concur: Agenda, Overview, Challenges & Solutions
Similar to Kubernetes at Concur: Agenda, Overview, Challenges & Solutions (20)
More from CoreOS
More from CoreOS (10)
Recently uploaded
Recently uploaded (20)
Kubernetes at Concur: Agenda, Overview, Challenges & Solutions
- 2. Agenda • Why k8s? • Why CoreOS? • K8s @ Concur • Q & A
- 3. What is Concur? 6500+ employees Offices: Australia China Czech Republic France Germany Holland India Japan Philippines Singapore Sweden UK US HQ: Bellevue 70m transactions 50b+ USD T&E spend annually €59m New cloud bookings Q3 2015 Cloud DNA 99 Established 1993 > 60% Fortune 500
- 4. 2011 2012 2013 2014 2015 2016 20182017 Expense transactions Bookings growth 2013 2015 55 million 77+ million Provisioned users 35+ million 2015 Registered mobile users 2013 2015 2.1 million 5+ million Concur growth
- 5. A little about me Principal Architect • Concur since 1998 in various roles (DBA, Ops Manager & Architect, etc.) • Corking w/ k8s since mid 2015 • contributor to k8s and participant in federation & service catalog SIGs • Email danwilson on github, k8s slack & gmail
- 6. Why Kubernetes? • Largest community support • Gifted to Linux Foundation • Protection from shifts in container technology • Pluggable API • Capable of isolation between namespaces • Declarative syntax for the complete make up of services • Built in logging, service discovery, etc. • Networking design eliminates port conflicts of containers
- 7. Why CoreOS? • Simple updates by channel • Designed to run as a cluster • Low overhead • Best documentation for k8s
- 8. Why CoreOS? • Simple updates by channel • Designed to run as a cluster • Low overhead • Best documentation for k8s
- 9. CoreOS
- 10. High Availability • All things fail, yes everything! • K8s HA guide: http://kubernetes.io/docs/admin/high-availability/ • cluster etcd – 3, 5 or 7 nodes • replicated + load balanced api servers • many minions
- 11. High Availability If cluster == alwaysOnline { glog.V(1). Infof(“nothing to do here!”) }
- 12. staging prod N clusters CI K8s @ Concur
- 13. Location US EUROPE Zone A B C A B C Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c K8s @ Concur
- 15. K8s @ Concur • kube2cnqr • Golang docker container • Watches the k8s API for updates to services and worker nodes • Calls a Concur internal API to addremove load balancer entries as needed • Handles connecting multiple k8s clusters to a single load balancer endpoint • Extras • External-IP set on service objects • Annotation w/ DNS name set on service objects • Handles multiple load balancer tiers • Allow services to specify DNS name
- 16. Sample project • https://github.com/concur/kubegowatcher • A sample golang program that leverages kubernetes watch endpoints using the client-go library. • Designed to run as a container on k8s • example of setting annotation on the service • example of checking event delay • Apache 2.0 license • Add your own business logic • Contribute back improvements to make it better
- 17. Location US EUROPE Zone A B C A B C Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c K8s @ Concur – k8s upgrades
- 18. Location US EUROPE Zone A B C A B C Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c Cluster v.1.3 us13a us13b us13c eur13a eur13b eur13c K8s @ Concur – k8s upgrades
- 19. K8s @ Concur Deployment Challenges • Stale kubectl scripts • Stale kubectl config • Handling credentials for each ci system • Many commands for each cluster • Handling proxy
- 20. K8s @ Concur
- 21. Deployment API CI Deploy petshop:v1 to test test14a, b & c K8s API calls for service & deployment
- 22. Deployment API CI Deploy petshop:v1 to prod US14a, b & c EMEA14a, b & c APAC14a, b & c K8s API calls for service & deployment
- 24. Setup API curl /setup Output kubectl config commands for all cluster zones requested | bash
- 25. K8s @ Concur
- 26. K8s @ Concur Add features to kubernetes federation • v1.6 – deploying to clusters based on label #29887 Other potentials… • Location specific overrides • Global overrides • Feature limitations (for example, don’t allow pods/rc’s without a deployment) Docs: https://github.com/concur/skipper
- 27. K8s @ Concur
- 28. Q & A Email danwilson on github, k8s slack & gmail
Editor's Notes
- Tripit Hipmunk
- Tripit Hipmunk
- Explain hypercar components and how they relate to building a kubernetes cluster. CoreOS makes it easy by already having the pieces tested together (kernel, flannel, etcd, systemd, etc)
- Plan for unknown unknowns Locking thresholds or bad query plans are known unknowns
- kubelet to container which messed up dns settings
- Terraform deployments CoreOS Flannel Custom logstash container
- Updated to use client-go library From centurylink/ca-certs Working w/scratch Gotchas: checking for the delay on events and handling updates made
- Docker golang setup for kubernetes f5 plugin
- Best way to get to v.1.3? Blue/green
- So, we created skipper
- So what does this skipper thing do anyways?
- So what does this skipper thing do anyways?
- So what does this skipper thing do anyways?
- So what does this skipper thing do anyways?