Oliver Moser presented on Kubernetes at Telekom Austria Group. They were looking for a way to scale their geo analytics workloads and established a Kubernetes playground in late 2015. Some key requirements they had were to scale workloads in and out based on changing needs, self-heal from failures, load balance workloads, and schedule and manage jobs. They implemented Kubernetes on bare metal servers and used Prometheus for monitoring and logging. They developed some custom tools like Makefiles and Ansible playbooks to help automate common tasks like deploying and managing workloads. They ran into some issues around storage, resource limits, and upgrades but are focusing on testing, high availability, alerting and Helm going forward.

1. 1
Oliver Moser, July 2017
Kubernetes at Telekom Austria Group

2. Who am I
• Working at A1 and TAG for quite a while
• Cloud
• Big Data
• DevOps
• Containers
• “Future Technology” J
2

3. Why Kubernetes?
3

4. What we were looking for
• End of 2015 we established a playground to deal with TV and Geo
analytics
• High level goals/stages:
• Start small
• Proof it works
• Scale out
• Reproduce
• No Hadoop in the beginning J

5. Geo Analytics
5
Ingest Enrich Output
DWH/CRM/Billing Data

6. OpenSource Galore
Prometheus

7. Captain Ahead!

8. Distributed
File StorageData Sources
Container
Workloads
Bare Metal
Workloads
Geo Data Pipeline
Data
Ingestors
ReadWrite
GlusterFS
Mobile
Access
CRM

9. Key Requirements
What Why How
Scale In/Out changing workloads Nodes/Deployments
Self Heal things... just break Deployments/ReplicaSets
Load Balance distribute workloads Services / Deployments
Schedule Jobs import data, update caches Jobs / CronJobs
Per-node
Agents
monitoring DaemonSets
Manage
Passwords
access to external systems
(e.g. Elasticsearch)
Secrets
Manage
Configuration
centralized provisioning of
application config
ConfigMaps

10. Bare Metal(RHEL)
Monitoring and Logging
Prometheus
Server
K8S Node
node_exporter
DaemonSet
Pod
/metrics
push alerts
read
logs

11. Federated Prometheus
Prometheus
Server
K8S Node
Pod
Pod B
/metrics
Prometheus Server
Service
Pod A
/metrics

12. Our Hacks
12

13. Hack 1: Makefiles FTW
> make build
> make push
> make (un)deploy

14. Hack 1: Makefiles FTW
> make show-logs
> make exec-pod

15. Hack 1: Makefiles FTW
> make scale

16. Hack 1: Makefiles FTW
> cat Makefile
> cat enricher-deployment.yaml

17. Hack 1: Makefiles FTW
Many alternatives to sed/make:
• Dedicated templating tool (e.g. Jinja)
• Use configuration management (e.g. Ansible)
• Wait for native template support (http://bit.ly/2t7HHQN)
... but...
• Makefiles are easy to understand (e.g. for Operations)
• make is super fast (yeah Ansible I’m talking to you J)
• make just works

18. Hack 2: kubeadm orchestration
• Cluster bootstrapping via kubeadm
• Automation via Ansible
• Easy to add/reset/remove worker nodes
• East to install/switch plugins (e.g. flanneld, Kube UI, Weave...)

19. Hack 2: kubeadm orchestration
• Playbooks are started with... make :-!

20. Hack 3: CLI helpers
• Shortcuts for most
common tasks
(exec, logs, etc)

21. Hack 3: CLI helpers
• Decoding secrets

22. Pain Points and Outlook
22

23. Issue’s we were/are running into
• DeviceMapper http://bit.ly/2uer3OV
• Blocked Tasks (switching from ext4 to XFS helped)
• <defunct> processes à containers stick around forever
• http://bit.ly/2vtcsMX http://bit.ly/2ckFrt7
• No Job history limit à Many Pods stick around à kube gets slow
• Flanneld/CNI IP address pool exhausted http://bit.ly/2t7HHQN
• Upgrade to 1.6 à new RBAC features à Kaboom!

24. Next Steps
• Tests
• HA
• Alerting
• Helm
• Websocket support for our AAA proxy (for k exec)

25. Key Learning
25

26. 26
Automation is Key
Or: Death by Repetition