Successfully reported this slideshow.
Your SlideShare is downloading. ×

Kubeinvaders & Chaos Engineering practices for Kubernetes-1.pdf


Check these out next

1 of 27 Ad

More Related Content

Similar to Kubeinvaders & Chaos Engineering practices for Kubernetes-1.pdf (20)

Recently uploaded (20)


Kubeinvaders & Chaos Engineering practices for Kubernetes-1.pdf

  1. 1. Kubeinvaders & Chaos Engineering practices for Kubernetes Eugenio Marzo - CKA, Vault Associate FOSDEM 2023
  2. 2. Agenda ● Definition of Chaos Engineering ● k-inv - game and programming mode ● Chaos Programming Console ● Community Links
  3. 3. Definition of Chaos Engineering ● ChaosMonkey ● Litmuschaos ● ChaosMesh ● ChaosToolkit ● Kubedoom Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production ( ● Test the resilience of a distributed system ● Trigger controlled alerts for testing monitoring systems Tools Use cases Definition
  4. 4. Monitoring system exposes tons of metrics. All are important, but what are definitely significant? Use Case - Metrics Selection
  5. 5. Sometime too many metrics and alerts can confuse SysOps teams. There is not clarity of what is important for a first-level monitoring system. Use Case - Metrics Selection
  6. 6. Chaos Engineering is a discipline for stressing systems and see how they are resilient and rock-solid. But, can it help us to solve our problem? Stressing systems reliability == Producing controlled alerts Use Case - Metrics Selection
  7. 7. Openshift 4.10 (3 Master/Workers) Chaosd (physical nodes) wrk1 wrk2 wrk3 Chaos Mesh Pods Prometheus Stack Use Case - Metrics Selection
  8. 8. Running Chaos Mesh we saw some interesting Alerts and related Metrics from Prometheus console: MEM Attack (/usr/local/chaosd-v1.0.0-linux-amd64/tools/stress-ng --vm 2 --vm-bytes 15G) ● etcdMembersDown ● etcdNoLeader ● TargetDown ● KubeClientErrors ● ExtremelyHighIndividualControlPlaneCPU Disk Attack - ./chaosd attack disk fill -s95G -p /var/lib/containers/ ● NodeFilesystemAlmostOutOfSpace CPU Attack - ./chaosd attack stress cpu -w 4 ● etcdMemberCommunicationSlow ● etcdHighCommitDurations ● KubePodNotReady ● HighOverallControlPlaneCPU Network Fault - Delay 3s ● TargetDown ● KubeAPIErrorBudgetBurn
  9. 9. What is k-inv Chaos Engineering tool for Kubernetes. It is composed by a game part (space-invaders imitation for killing pods) and a chaos programming console ● Kill pods randomly and start chaos jobs against worker and master nodes ● Define and run chaos experiments and load testing Features Definition
  10. 10. k-inv - Helm helm repo add kubeinvaders helm repo update kubectl create namespace kubeinvaders helm install kubeinvaders --set-string config.target_namespace="namespace1,namespace2" -n kubeinvaders kubeinvaders/kubeinvaders --set ingress.enabled=true --set --set deployment.image.tag=v1.9.6 Definition
  11. 11. k-inv - Docker docker run -p 8080:8080 --env K8S_TOKEN=<k8s_service_account_token> --env ENDPOINT=localhost:8080 --env INSECURE_ENDPOINT=true --env KUBERNETES_SERVICE_HOST=<k8s_controlplane_host> --env KUBERNETES_SERVICE_PORT_HTTPS=<k8s_controlplane_port> --env NAMESPACE=<comma_separated_namespaces_to_stress> luckysideburn/kubeinvaders:develop Definition
  12. 12. Game Mode
  13. 13. Game Mode
  14. 14. Architecture
  15. 15. Switch between game and programming mode Control Plane Overview
  16. 16. OpenMetrics exporter http://kubeinvaders: 8080/metrics Control Plane Overview
  17. 17. Customizable presets for chaos experiments and load testing Control Plane Overview
  18. 18. Chaos Programming Console - Controls and metrics
  19. 19. Light and Dark Mode
  20. 20. Options and Chaos Container Def.
  21. 21. Chaos Programming Console - Logging
  22. 22. Chaos Programming Mode - Pods status Watch status of pods related to current chaos experiments.
  23. 23. chaos-codename: promethium jobs: cpu-attack-job: additional-labels: chaos-controller: kubeinvaders chaos-type: stress-ng chaos-codename: promethium image: command: "stress-ng" args: - --version mem-attack-job: additional-labels: chaos-controller: kubeinvaders chaos-type: stress-ng chaos-codename: promethium image: command: "stress-ng" args: - --version experiments: - name: cpu-attack-exp job: cpu-attack-job loop: 5 - name: mem-attack-exp job: mem-attack-job loop: 5
  24. 24. Chaos Programming Mode - k-inv language
  25. 25. Chaos Programming Mode - HTTP load test
  26. 26. Community Links ● Awesome_k8s: me-kubernetes ● Kubernetes_blog: /kubeinvaders-gamified-chaos-engine ering-tool-for-kubernetes/ ● Live_session: =k0w-NXt0_hA (repo git) (my blog…) 2/ (linkedin profile) ● eugenio.marzo [at] ● kubeinvaders [at] Other Tools Contacts Links
  27. 27. Thank you for your attention!