Kubernetes Meetup @ Be-Mobile 2017-01-17 Slides are a bit malformed unfortunately on slideshare...

1. KUBERNETES @ BE-MOBILE
Pieter Callewaert – Teamlead Devops

2. DATA AGGREGATOR
KUBERNETES @ BE-MOBILE 2
Be-Mobile
mobility
database
Floating car data
• GPS data
• Smartphone data
• Telco data
Road sensors
• Cameras
• Radars
• Loops
Crowd sourced
• Drivers
• Social media
• Police
• Emergency vehicles
Other data
• Public transport
• Car & bike sharing
• Vehicles & bike parking, fuel
• Traffic emission & noise zones
• Toll
Sourcesfrom>20countries

3. WHO WE ARE
KUBERNETES @ BE-MOBILE 3
Smart Tolling &
Map Matching
Mobility
as a Service
Mobility Monitoring
& Analysis
Traffic
Management
Smart ParkingMobility Payments
Platform
Connected Vehicle
Platforms
Traveler
Information
Smart
mobility

4. BEFORE
KUBERNETE
S

5. MANUAL DEPLOYMENT
➞ Deployments were done manually on the servers
• Growing amount of countries where Be-Mobile is active
• Each country needs the same basic set of applications to be set up
• Growing amount of developers
• Meaning more applications to be deployed
• But, all deployments done by the DevOps-team
➞ Time to deployment increased due to the growth
➞ Management of the applications was getting complex
➞ Human errors might occur
KUBERNETES @ BE-MOBILE 5

6. PUPPET DEPLOYMENT
Dependencies problem: correct version of NodeJS, Java, Mono,…
• Programming the deployment of the applications by configuration
• Roll-outs of new countries and applications were much faster
• Helped avoiding human errors
KUBERNETES @ BE-MOBILE 6

7. PUPPET+DOCKER DEPLOYMENT
• Isolated environments for an application
• Contains all necessities for the application
• E.g.: website + webserver / Java application + Java framework
• No more need for ‘type-specific servers’
• Different types of applications
can all run on the same server
• Easy to migrate a container or scale
over multiple servers
KUBERNETES @ BE-MOBILE 7

8. HOWEVER… NEW BOTTLENECKS
• Chaining multiple applications
• Time consuming
• Error sensitive
• Crashes
• Manual interventions
• Deployments
• Faster, but still a backlog of deployments
KUBERNETES @ BE-MOBILE 8

9. WELCOME
KUBERNETE
S

10. WHAT IS KUBERNETES
• Google project (Borg) made open-source
• Further developed by a strong community
• Orchestration of containers
• Namespaces
• Self-healing
• Allows us to create deployment chains
KUBERNETES @ BE-MOBILE 10

11. ORCHESTRATION IN KUBERNETES
• Multiple servers are combined into a cluster
• Kubernetes picks a server to deploy an container to
• By checking the container resource needs (CPU/memory)
• By checking the servers resource availability
• Containers can be scaled or made high available very easy
• Different ways to do a deployment
• E.g.: rolling update, recreate
KUBERNETES @ BE-MOBILE 11

12. SELF-HEALING WITH KUBERNETES
• When a container no longer responds
• Health checks detect this (if configured)
• Container gets destroyed and restarted
• When a container crashes
• Container gets restarted
• When a server crashes
• Move containers running on the crashed server to another one
➞ Faster recovery time!
KUBERNETES @ BE-MOBILE 12

13. STARTING WITH KUBERNETES
First experience: The Sockshop demo from the kubernetes.io site (on minikube)
KUBERNETES @ BE-MOBILE 13

14. PRODUCTION DEPLOYMENT
Installing kubernetes on bare metal!
 A lot of deployment guides are focused on cloud environments
Started with kargo (now known as kubespray). Was our first production cluster.
Very alpha, updating between versions of kubernetes or kargo was very hard.
Then we tried with Kismatic
Easy and well documented installation
”Community” support is great, but only people from Apprenda are working on it
Is still our current installation method.
KUBERNETES @ BE-MOBILE 14

15. DISTRIBUTE
D
STORAGE

16. OPTION 1: GLUSTERFS
First experience with distributed storage systems
Installed automatic with Kismatic
Pros:
• Easy to manage/setup
Cons:
• We had to create our Persistent Volumes ourselves
• For some reason, not scalable when creating a lot of volumes
KUBERNETES @ BE-MOBILE 16

17. OPTION 2: CEPH
Deployed with Ansible
Pros:
• It just works.
• And really fast!
• StorageClass support in Kubernetes
Cons:
• Steep learning curve
• More complex architecture (compared to GlusterFS)
KUBERNETES @ BE-MOBILE 17

18. KUBERNETE
S
TODAY

19. WHAT DO WE DEPLOY ON KUBERNETES?
What we do not deploy on Kubernetes :
• Kafka
• Cassandra
• MongoDB
• Ceph
• Elastic Search
• Redis*
What do we deploy on Kubernetes:
• Everything else.
KUBERNETES @ BE-MOBILE 19

20. INSTALLATION SERVERS: ANSIBLE
• Installing one server was OK, installing 10 servers is slow, repetitive and boring.
• Automatic server installations
• Partitioning
• Basic installation of the operating system
• Configuration of firewall rules
• Deployment of core components (Kubernetes/Ceph)
KUBERNETES @ BE-MOBILE 20

21. CLUSTERS TODAY
• More than 1 year of production
• We migrated once a full Kubernetes cluster (switch from Kargo to Kismatic)
• Largest cluster surpassed 100 nodes (6.9 TB memory, 1000+ cores)
• Multiple clusters: test cluster, QA clusters, staging cluster,..
KUBERNETES @ BE-MOBILE 21

22. CONTINOUS DEPLOYMENT
KUBERNETES @ BE-MOBILE 22

23. VMAP DEPLOYMENTS
KUBERNETES @ BE-MOBILE 23

24. PROJECT DEPLOYMENT
KUBERNETES @ BE-MOBILE 24

25. CONTINOUS DEPLOYMENT: OTHER METHODS
KUBERNETES @ BE-MOBILE 25
• QA environments (e.g. Tolling Be-Mobile Acceptance)
• Self deploying applications

26. KUBERNETE
S
TOOLS

27. KUBERNETES PROXY MANGEMENT
KUBERNETES @ BE-MOBILE 27

28. BE-MOBILE LOGVIEWER
KUBERNETES @ BE-MOBILE 28

29. BE-MOBILE KUBETOKEN
• Inspired on Atlassian kubetoken (https://github.com/atlassian/kubetoken)
• Integrates with Hashicorp Vault
• Uses PKI backend to sign certificates, LDAP as auth backend.
KUBERNETES @ BE-MOBILE 29

30. LESSONS
LEARNED

31. NGINX INGRESS CONTROLLER AND WEBSOCKETS
Ingress controller getting unresponsive after a while
A lot of nginx processes still running
Perfect blog that describes the problem:
http://danielfm.me/posts/painless-nginx-ingress.html
KUBERNETES @ BE-MOBILE 31

32. CERTIFICATES EXPIRED
KUBERNETES @ BE-MOBILE 32

33. BE-MOBILE RECRUITING
We’re always looking for passionate talents!
http://www.be-mobile.com/careers/
PRESENTATION TITLE 33

34. info@be-mobile.com ⎹ www.be-mobile.com