The cloud already has a larger CO2 footprint than the worldwide aviation industry. With increasing digitalization and cloudification this trend will continue if we don’t do anything about it. Many Kubernetes installations are greatly oversized in terms of the resources actually required and thus contribute unnecessarily to global warming. What is the energy balance of your cluster and its workloads? In this session we will discuss approaches and technologies that help to make K8s clusters green(er). First of all, transparency is needed: what is the energy balance of the cluster and its workloads? Only then we can start to optimize for better energy efficiency through elasticity and reducing “waste”. And it’s not that difficult, so let’s get started!

1. qaware.de
Blue turns green!
Approaches and technologies for sustainable
Kubernetes Clusters
Mario-Leander Reimer
mario-leander.reimer@qaware.de
@LeanderReimer
https://cisweb.lancaster.ac.uk/

2. 2
Mario-Leander Reimer
Managing Director | CTO
@LeanderReimer
#cloudnativenerd #qaware
#gernperDude

3. 3
Mario-Leander Reimer
Managing Director | CTO
@LeanderReimer
#cloudnativenerd #qaware
#gernperDude

4. 4
IT
Create sustainability impact
in other industries
Show responsibility for own
consumption of resources

5. sustainable
kubernetes

6. cncf/tag-env-sustainability
TAG Environmental Sustainability
🌳🌍♻

7. https://tag-env-sustainability.cncf.io/landscape/

8. Shared Responsibility means it is our responsibility to use the
required cloud resources in a sustainable manner.
8
QAware

9. Sustainability Engineering to optimize the CO2 footprint of
your K8s clusters and workloads.
9
QAware
Monitoring
What is the energy
consumption and
CO2 footprint of
my cluster and
workloads?
Measure,
Measure,
Measure,
Act.
Elasticity
Oversized or
underutilized
resources?
Infrastructure and
workloads should
be proportionate
to actual demand.
Architecture
Where is the
cluster running?
On which
hardware?
Take sustainability
into account when
building the
infrastructure.
Waste
Shutdown or
deplete the
wasted
resources.
Not required,
redundant or
forgotten
environments or
workloads?

10. 10
QAware
ELECTRICITYMAPS.COM

11. Business requirements and sustainability goals need to be
aligned and reconciled.
The choice of region for the cluster and workloads has a significant impact on their KPIs,
including performance, cost, and carbon footprint.
■ Ensure that the region meets the required legal regulations (e.g. GDPR)
■ Ensure that the region supports the required services and features.
– Which hardware classes are supported?
– Databases? Storage? …
■ Calculate costs for the cluster and workloads for the selected region.
■ Test network latency between your end user locations and region!
11
QAware
https://docs.aws.amazon.com/wellarchitected/latest/sustainability-pillar/sus_sus_region_a2.html
AWS Graviton 2 can save up to 65%
on energy consumption
AWS Graviton 2 can save up to 15%
on costs

12. Cluster Rightsizing mit Cluster-Autoscaler und Karpenter
12
QAware
Node 1 Node 2 Node 3
X
Node 4

13. Workload Rightsizing with VPA and Goldilocks
13
QAware
https://github.com/FairwindsOps/goldilocks
apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
name: hamster-vpa
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: hamster
updatePolicy:
# updateMode: "Auto"
updateMode: "Off"
resourcePolicy
:
containerPolicies
:
- containerName: '*'
minAllowed:
cpu: 100m
memory: 50Mi
maxAllowed:
cpu: 1
memory: 500Mi
controlledResources
: ["cpu", "memory"]
defined and tune requests and
limits for CPU and RAM
perform realistic
performance tests
analyse measurements of
resource consumption
https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler

14. kube-green
An operator to reduce CO2 footprint of your clusters
14
QAware
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
name: non-working-hours
spec:
weekdays: "1-5"
sleepAt: "18:00"
wakeUpAt: "08:00"
timeZone: "Europe/Rome"
suspendCronJobs : true
excludeRef:
- apiVersion: "apps/v1"
kind: Deployment
name: no-sleep-deployment
- matchLabels :
kube-green.dev/exclude : "true" https://kube-green.dev/docs/FAQ/#how-many-co2-is-produced-by-pod

15. High-Level View on Scaling in Kubernetes
15
QAware

16. Resource Metrics, Custom Metrics, External Metrics API
16
QAware

17. Sustainable Computing Projects
■ Kepler (Kubernetes-based Efficient Power Level Exporter) uses eBPF to probe energy
related system stats and exports as Prometheus metrics.
■ PEAKS (Power Efficiency Aware Kubernetes Scheduler) uses metrics exported by
Kepler to help Kubernetes schedule to improve energy efficiency by placing Pods on
optimal nodes.
■ CLEVER (Container Level Energy-efficient VPA Recommender) uses metrics exported
by Kepler to recommend Vertical Pod Autoscaler the resource profiles to improve
energy efficiency by running workloads.
17
QAware
https://www.sustainable-computing.io

18. Monitor and Measure the Energy Consumption of Nodes and
Pods using Kepler
18
QAware

19. Carbon-aware Scaling mit KEDA
■ https://keda.sh
■ KEDA ist ein Kubernetes-basierter ereignisgesteuerter
und zweck-orientierter Autoscaler.
■ Unterstützt 50+ Scaler Out-of-the-box, e.g. Cron,
RabbitMQ, …
■ Carbon Aware Scheduling
- Kubernetes Carbon Intensity Exporter
- Carbon Aware KEDA Operator
19
QAware

20. Carbon aware Spatial Shifting with Karmada
20
QAware https://rossfairbanks.com/2023/07/12/carbon-aware-spatial-shifting-with-karmada/
karmada control plane
Cluster A
API Server
Cluster B
API Server
Agent
karmada API server
etcd
karmada
scheduler
karmada API server
Cluster
Controller
Policy
Controller
Binding
Controller
Execution
Controller
W1 W1 W1 W1

21. Carbon aware Spatial Shifting with Karmada
21
QAware https://rossfairbanks.com/2023/07/12/carbon-aware-spatial-shifting-with-karmada/
karmada control plane
Cluster A
API Server
Cluster B
API Server
Agent
karmada API server
etcd
karmada
scheduler
karmada API server
Cluster
Controller
Policy
Controller
Binding
Controller
Execution
Controller
W1 W1 W1 W1

22. lreimer/green-kubernetes

23. qaware.de
QAware GmbH
Aschauer Straße 32
81549 München
Tel. +49 89 232315-0
info@qaware.de
twitter.com/qaware
linkedin.com/company/qaware-gmbh
xing.com/companies/qawaregmbh
slideshare.net/qaware
github.com/qaware