January
Les bonnes résolutions de la sécu dans mon cluster Kubernetes
15/01/2025
Un Grand Merci A
PRESENTATION D’EASY
PARTNER
Experts de la mise
en relation de
profils tech
DÉCOUVRIR NOTRE
OFFRE
PRÉSENTATION
D’EASY PARTNER
Trois piliers, trois offres adaptées
Cabinet de recrutement
Du profil junior au directeur technique,
notre équipe de Tech Recruiters spécialisés
par stack technique vous accompagne sur
vos recherches, forts de nos +3000
recrutements pour plus de 500 clients
Assistance technique
Vous cherchez à lancer un projet rapidement,
apporter une expertise spécifique à vos
équipes, ou encore absorber un pic de
charge, nous vous trouvons, comme nous le
faisons avec nos 300 clients, le talent adapté à
vos enjeux en 48h.
Formation
Bénéficiez de nos 11 ans d’expertise
pour transformer vos équipes en
experts autonomes de la gestion de
talents tech, nous vous transmettons
l’expérience Easy Partner
PRÉSENTATION
D’EASY PARTNER
Experts de la mise en relation de profils tech
Spécialiste de la mise en relation de profils tech
depuis 11 ans
Spécialisation de nos collaborateurs par
stack technique, formations spécifiques par
d’ex-ingénieurs
> 3500 placements réussis : nous plaçons > 1
ingénieur par jour sur un projet (sur un poste en
CDI ou en prestation de service)
96% des candidats valident leur période
d’essai, 68% des missions sont renouvelées
Valider plus rapidement les postes grâce à
nos recruteurs experts de leur marché
technologique
Pour vous c’est l’assurance de…
Réduire le temps passé par recherche
grâce à notre compréhension rapide de
vos besoins et notre pré-sélection
rigoureuse
Candidats qui restent chez poste /
mission, dont les motivations profondes
sont alignées avec les valeurs des clients
Pour vous c’est l’assurance de…
PRÉSENTATION
D’EASY PARTNER
La base de profils tech la plus complète de France
> 100.000 profils tech qualifiés en base
+1.000 nouveaux candidats qualifiés par nos
collaborateurs par mois dans notre base
2/3 des candidats placés en 2022 étaient
issus de notre base de profils
Staffer vos besoins plus rapidement
grâce à notre base activable pré-
qualifiée
Avoir accès aux meilleurs profils du
marché en un temps record
90% des candidats placés en 2022 ont été
débauchés
ZOOM SUR…
Quelques chiffres
96%
30
Tech Recruiter
Passionnés et
expert du métier
des candidats
placés par Easy
Partner ont validé
leur période d’essai
45%
des candidats
rencontrés par nos
clients ont reçu une
offre
ZOOM SUR…
Nos clients qui nous font confiance
Grands Comptes ETI PME et Start-ups
+750 CLIENTS SUR PLUS DE 30
SECTEURS D’ACTIVITÉ
Merci !
Qui sommes-nous ?
Katia Himeur
Henrik Rexed
Agenda
I. Intro
II. Sécuriser votre cluster Kubernetes
III. À quel agent confier vos clusters k8s
IV. Jeu + goodies
V. Apéro / Buffet
2025
DevOps Aix-Marseille
Cloud-Native Aix-Marseille
?
PR Github CNCF https://github.com/cncf/communitygroups/issues/229
⇒
Teasing
● Passage sur conference-hall
○ https://conference-hall.io/team/meetup-devops-aix-marseille/meetup-devops-aix-marseille
○ N’hésitez pas à faire passer le mot ou à poster votre abstract
● Un agenda déjà calé
● Conclusions
○ Le meetup évolue
○ Des infos au cours du début d’année
○ Ne ratez pas le meetup de mars… 😉
Agenda 2025
● Janvier: 2025-01-15 🤦
● Février: 2025-02-06
● Mars: 2025-03-06
● Avril: 2025-04-24
● Mai: 2025-05-22
● Juin: 2025-06-12
● Juillet: 2025-07-03
Sécuriser votre cluster
Kubernetes : la recette pour
réussir, étape par étape !
Securing your Kubernetes cluster:
Your step-by-step guide to success!
janvier 2025
Katia HIMEUR
19
Who am I?
Who am I?
• 🪪 Katia HIMEUR
• 💻 Computer scientist
• ☁️CTO & co-founder at Cockpit io, cloud & DevOps specialist
• 💻
‍
💻‍ SRE/Cloud/DevOps consultant for several years
• Duchess Core Team Member
❤️#Cloud #DevOps #Containers #Serverless #GitOps #IaC #CICD ❤️
Securing your Kubernetes cluster: a step-by-step guide to success! 2
0
janvier 2025
21
Introduction
Kubernetes, the leading container orchestrator
janvier 2025
Source : https://marketsnresearch.com/report/1649/global-kubernetes-market
2
2
Securing your Kubernetes cluster: a step-by-step guide to success!
The new cloud OS
janvier 2025
Source : https://www.dynatrace.com/news/blog/kubernetes-in-the-wild-2023/
2
3
Securing your Kubernetes cluster: a step-by-step guide to success!
janvier 2025
Attackers on the lookout
2
4
Securing your Kubernetes cluster: a step-by-step guide to success!
Safety not always a priority
janvier 2025
2
5
Securing your Kubernetes cluster: a step-by-step guide to success!
26
Why is it important to
secure your cluster?
K8s's popularity increases attacker interest
janvier 2025
2
7
Securing your Kubernetes cluster: a step-by-step guide to success!
Flexibility Complexity Errors
⥤ ⥤
janvier 2025
Configuration errors are a
security risk
Complexity can lead to
configuration errors
Its flexibility leads to
complex configurations
Kubernetes is extremely
flexible
2
8
Securing your Kubernetes cluster: a step-by-step guide to success!
Extended attack surface
janvier 2025
https://kubernetes.io/docs/concepts/overview/components/
2
9
Securing your Kubernetes cluster: a step-by-step guide to success!
Scalability and resilience can be a vector for vulnerability propagation
janvier 2025
3
0
Securing your Kubernetes cluster: a step-by-step guide to success!
Company background
Need :
●Compliance with specific standards and regulations
●Protect sensitive and critical applications and infrastructure
●Secure the sensitive data that may pass through
janvier 2025
3
1
Securing your Kubernetes cluster: a step-by-step guide to success!
32
What will we see during
this talk?
What will we see during this talk?
●Understand the different types of attackers and attack vectors
●A non-exhaustive list of concrete actions you can take to secure your
cluster
●Focus on managed Kubernetes clusters
●The list of tools provided is for information only
janvier 2025
3
3
Securing your Kubernetes cluster: a step-by-step guide to success!
34
Understanding the Threat
Landscape
Attacker types
35 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Attacker types
External
attackers
Compromised
users
Internal
attackers
Compromised
containers
janvier 2025
36
Securing your Kubernetes cluster: a step-by-step guide to success!
Main attack vectors
37 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Main attack vectors
janvier 2025
3
8
Securing your Kubernetes cluster: a step-by-step guide to success!
Insecure Server API
Compromised
containers
Misconfigured access
controls
Exposed dashboards
Incorrect network
configuration
Compromised nodes
Compromised secrets
Supply chain
(dependencies…)
…
39
Rely on community and
ecosystem support
Rely on Community and ecosystem support
●Strong focus on safety
●Continuous improvement and disclosure of vulnerabilities
●Goals :
○Share security best practices
○Encouraging corrective action
○Stay ahead of potential threats
janvier 2025
4
0
Securing your Kubernetes cluster: a step-by-step guide to success!
41
How do you secure your
Kubernetes cluster?
Zero trust architecture
42 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Zero trust architecture
●Approach to designing and implementing IT systems where trust is totally
eliminated
●Never trust, always verify
janvier 2025
43
Securing your Kubernetes cluster: a step-by-step guide to success!
Secure API Server
janvier 2025
44
Securing your Kubernetes cluster: a step-by-step guide to success!
Why API server is critical ?
●Critical cluster component
●External and internal communications gateway
●Manage
○Authentication and authorization
○Data validation and storage in etcd
○Orchestration and resource management
○Scalability and performance
janvier 2025
45
Securing your Kubernetes cluster: a step-by-step guide to success!
Limit API Server exposure?
●Restrict access
●Encrypt all flows with API Server
janvier 2025
46
Private clusters Security groups
Endpoint
access control
Network ACLs
Securing your Kubernetes cluster: a step-by-step guide to success!
Identity and Access
Management
janvier 2025
47
Securing your Kubernetes cluster: a step-by-step guide to success!
Identity and Access Management
●Integration with Cloud IAM
●Use an SSO portal to connect to your clusters (oAuth2, OpenID Connect or
LDAP)
●Enable multifactor authentication
janvier 2025
48
Securing your Kubernetes cluster: a step-by-step guide to success!
Authentication
janvier 2025
49
Securing your Kubernetes cluster: a step-by-step guide to success!
Using RBAC
50 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Using RBAC
●What is it?
○RBAC: Role-Based Access Control
○Resource access control based on user roles and service accounts
○Defines who can do what
5
1
janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Using RBAC
●Good practice
○Apply the principle of least privilege to users and service accounts
52 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Example of an RBAC role
5
3
janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Pod Security Admission
janvier 2025
54
Securing your Kubernetes cluster: a step-by-step guide to success!
Pod Security Admission
●Kubernetes native admission controller to enforce Pod Security standards
policies
●Ensure containerized workloads are secure against known privilege
escalations
●Pod security restrictions are applied at the namespace level
●Enabled by default in the latest versions of Kubernetes
55 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Pod Security Standards Policies
56
Restricted
●Very restrictive policy
●Follows good curing
practices
Privileged
●No restrictions
●Climbing possibilities
Baseline
●Minimum restriction
policy
●Prevents the most
common climbs
janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Apply a safety policy
5
7
janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Alternatives
58 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Using network policies
59 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Network status report
●Default: pods are not isolated, all traffic flows are allowed (ingress and
egress).
●Best practice: Restrict pod-to-pod communication to the strict minimum
●How ? Use network policies
60 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
What are network policies?
●Allows you to define how pods are authorized to communicate with :
○Other pods
○Other namespaces
○IP blocks
61 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Example of a policy that prohibits all outbound flow
62 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Alternatives
63 janvier 2025
Network
policies
Security Network
Observability
Performance
Securing your Kubernetes cluster: a step-by-step guide to success!
Securing your secrets
64 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Secrets... Not very secret 😱
●A Kubernetes secret is an object used to store sensitive data (passwords,
tokens, SSH keys, etc.).
●Default :
○No encryption
○Stored unencrypted
○Base64 encoded value
65 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Securing your secrets
●Use RBAC to secure access
●Rotate secret
●Audit access to secrets
●Regularly review and update access policies
●Prevent secrets from ending up in logs
●Don't hard code secrets
66 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Gitleaks
Turn secrets into volume 👍
Exposing them as environment variables👎
68 janvier 2025
ServiceAccount secrets can
only be mounted on specific
resource types.
Securing your Kubernetes cluster: a step-by-step guide to success!
Turn secrets into volume 👍
Exposing them as environment variables👎
69 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Volume mounting
● Secrets isolated within the file
system
● Not visible in dumps and crash
logs
● Granular permissions possible
● Automatic refresh possible
Environment variables
● Visible in process dumps
● Appear in crash logs
● Inherited by child processes
● Risk of accidental logging
Use external secret management tools
7
0
AWS Secrets Manager
Azure Key Vault Google Cloud Secret
Manager
janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Securing your secrets
●The Secrets Store CSI Driver enables integration of external secrets
managers with Kubernetes
●Integration is via a CSI (Container Storage Interface) volume.
●This driver allows you to mount several secrets, keys, certificates, etc.
stored in these external secret managers.
●Once the volume has been attached, the data is mounted in the
container's file system.
71 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Audit logging
72 janvier 2025
Securing your Kubernetes cluster: a step-by-step guide to success!
Audit logging
●Record every request made to the API server
●Analyze logs to detect suspicious and unusual activity
●Use security tools to analyze and react to logs in real time
●Define audit policies to fine-tune the configuration of events to be
recorded
janvier 2025
73
Securing your Kubernetes cluster: a step-by-step guide to success!
Example of an audit policy
janvier 2025
74
Securing your Kubernetes cluster: a step-by-step guide to success!
Update and patch management
janvier 2025
75
Securing your Kubernetes cluster: a step-by-step guide to success!
Update and patch management
●Apply update and security patches, automatically if possible, and quickly :
○Kubernetes clusters
○all components deployed on clusters
○nodes
●Use tools that scan all Kubernetes cluster components, including images
●Respect best operating system practices
janvier 2025
76
Securing your Kubernetes cluster: a step-by-step guide to success!
Securing containers
janvier 2025
77
Securing your Kubernetes cluster: a step-by-step guide to success!
Securing containers
●Use only trusted images
●Reduce image size to reduce attack surface
●Use immutable containers to avoid runtime changes
janvier 2025
78
Securing your Kubernetes cluster: a step-by-step guide to success!
Securing containers
●Scanning your container images: before and after deployment
●Sign your container images
●Secure and protect your containers using tools that react to the detection
of abnormal events
janvier 2025
79
Securing your Kubernetes cluster: a step-by-step guide to success!
Securing containers
janvier 2025
80
Securing your Kubernetes cluster: a step-by-step guide to success!
Securing containers
janvier 2025
81
Securing your Kubernetes cluster: a step-by-step guide to success!
Admission controllers
janvier 2025
82
Securing your Kubernetes cluster: a step-by-step guide to success!
Admission controllers
●Controllers that intercept requests to the API server before objects are
persisted in the etcd database
●Two types of controller :
○Validating admission controller : Validates or rejects requests
○Mutating admission controller : Modifies the requests it accepts
janvier 2025
83
Securing your Kubernetes cluster: a step-by-step guide to success!
Configuration example
janvier 2025
84
This admission controller
prevents the API server
from being overloaded by
requests applying rate
limiting.
Securing your Kubernetes cluster: a step-by-step guide to success!
85
Service mesh
What is a service mesh?
janvier 2025
86
Securing your Kubernetes cluster: a step-by-step guide to success!
●Dedicated infrastructure layer added to applications
●Designates both the tools and the network domain created
●Allow us to :
○understand traffic
○make decisions based on traffic type or origin
○reduce the complexity of network management in a microservice
context
What is a service mesh?
janvier 2025
87
Securing your Kubernetes cluster: a step-by-step guide to success!
Service Mesh
functionalities
janvier 2025
88
Securing your Kubernetes cluster: a step-by-step guide to success!
Service mesh functionalities
janvier 2025
89
Securing your Kubernetes cluster: a step-by-step guide to success!
●Observability
●Traffic management
●Security
●A/B testing
●Canary deployment
●Rate limiting
●Access control
●Encryption (including mTLS)
●End-to-end authentication
●Service discovery
janvier 2025
90
Securing your Kubernetes cluster: a step-by-step guide to success!
Some services mesh
Using services mesh
janvier 2025
91
Securing your Kubernetes cluster: a step-by-step guide to success!
95
Requests & limits
Requests & limits
janvier 2025
96
Securing your Kubernetes cluster: a step-by-step guide to success!
●Best practices
○Set memory requests lower than or equal to limits
○Limit CPU on sensitive workloads
Requests & limits
janvier 2025
97
Securing your Kubernetes cluster: a step-by-step guide to success!
●Why?
○Restrict resources used by pods on nodes
○Avoid the effects of a denial-of-service attack
105
Real-time protection
Real-time protection
janvier 2025
106
Securing your Kubernetes cluster: a step-by-step guide to success!
●Detect abnormal behavior, security threats and compliance violations
●Be alerted in real time.
●Some tools rely on kernel events, enriched with container and Kubernetes
metadata, to succeed in their protection missions.
Real-time protection
janvier 2025
107
Securing your Kubernetes cluster: a step-by-step guide to success!
108
Further information
Follow best practices and
recommendations
janvier 2025
109
Securing your Kubernetes cluster: a step-by-step guide to success!
Follow best practices and recommendations
janvier 2025
110
Securing your Kubernetes cluster: a step-by-step guide to success!
●Follow recommendations and best practices
○Security Checklist from the Kubernetes community:
https://kubernetes.io/docs/concepts/security/security-checklist/
○CIS (Center for Internet Security (CIS) ) Kubernetes Benchmark
○Recommendations from cloud providers
○Keeping up to date
113
Conclusion
Conclusion
●Securing Kubernetes environments is fundamental to protecting your
company's interests, ensuring compliance and maintaining trust.
●As Kubernetes continues to evolve, so will security strategies, requiring
constant attention and adaptation.
●Today's list is not exhaustive. No list can be, as attackers show their creativity.
●Safety is a matter of continuous improvement and constant attention
janvier 2025
114
Securing your Kubernetes cluster: a step-by-step guide to success!
Thank you
/in/katiahimeur/
🔗 blog.cockpitio.com
🔗 www.cockpitio.com
/company/cockpit-io/
Keep in touch
17/04/2024
115
Securing your Kubernetes cluster: a step-by-step guide to success!
A team of enthusiasts,
guided by DevOps
culture,
to bring you the best of
the Cloud!
À quel agent confier vos
clusters k8s :
Falco, Tetragon ou
KubeAmor ?
What agent to trust with
your k8s:
Henrik Rexed
CloudNative Advocate
v
Cloud Native Advocate
• 15+ years of Performance engineering
• Owner of : IsitObservable
Producer of : Perfbytes
Henrik Rexed
v
▪No Spartans, Eagles or Shields were harmed in the making
of this presentation
▪The intention behind this talk track is not to assign blame to
any CNCF project.
▪This session is made to help the community in choosing
their runtime security agent.
CONFIDENTIAL 120
DISCLAIMER
KubeOlympics 2024
KubeOlympics 2024
The Athletes
Falco
Tetragon
Kubearmor
Tracee
v
• What we expect from a runtime security agent
▪ Compare each solution in :
• Desing Experience
• The Components required
• The Observability
• See various Benchmarking results
• Recommendation on which agents needs to used
under specific conditions
If you stay with me you will ...
The Rules
125
C
Filter
Components
Observability
Performance
Stage 1: Components
127
C
Observability
Performance
Components
Components
Observability
Performance
• A security agent usually relies on ebpf to collect the
kernel or user event of our environment
• In k8S , the security agent has usually at least :
• A daemonset deploying the ebpf probe on each node of our
cluster
v
Components required
▪Falco ▪Tetragon ▪KubeArmor ▪Tracee
Stage 2: Capture & Filter
130
C
Filter
Components
Observability
Performance
• Be able to capture events from our
k8S cluster related to :
• Process launched/ finished
• File Access Read/Write
• Networking details
• Change of privileges
• …etc.
• Get the details attached to the event:
• Path: executable path, file path…
• Process details: pid, process name ,
• User: userid, groupid
• Capabilities
• Container: image , name
• K8s : pod name, namespace, ..etc
C
Observability
Performance
Filter
Components
Observability
Performance
• Filtering means having the option to create a policy
filtering the event on :
• Process properties
• User details
• K8S metada
• File details
• Syscall/ kernel functions or event
• Have Predefined Rules
• Be able to react
v
Falco Capture
▪ Falco is rule engine producing logs if the Falco event is
matching a rule
▪ Falco receive event from :
▪ Falco kernel agent
▪ External plugins
▪ Falco provides a SDK allowing us to build our custom Plugin
▪ Falco agents captures :
▪ Process details
▪ Syscall
▪ Tracepoints
▪ File
Hostname
output
Output
field_fields
Pid
Proc.name
Container.nam
e
Proc.execpath
User.name
K8s.pod.name
…
Tags …
Rule
source
v
134
Falco rule
▪Falco starts by loading the rule files that
would be used to generate the events.
By default, Falco provide a default sets
of rule.
▪Falco provides “fields” helping us to
define our filtering rule based on:
▪Event information
▪Proces
▪File,
▪Syscalls
▪a global rule file allows to reuse filtering
rules or conditions
macro: access_file
condition: evt.type=open
- rule: program_accesses_file
desc: track whenever a set of programs opens a file
condition: (access_file) and proc.name in (cat, ls)
output: a tracked program opened a file (user=%user.name
command=%proc.cmdline file=%fd.name)
priority: INFO
- rule: test_rule
desc: test rule description
condition: evt.type = close
output: user=%user.name command=%proc.cmdline file=
%fd.name
priority: INFO
enabled: false
v
135
Tetragon capture
▪ Tetragon will by default produce events related to :
▪ Process execution
▪ Process exit
▪ If you deploy Policies, it will produce events
matching our policy rule
▪ Policy required to defined a hook point to our
system using:
▪ Kprobe
▪ Tracepoints
▪ Ubprobe
▪ llsm
Process_exe
c
Process
Pid
Uid
Binary
Arguments
Pod
…
Parent
Node name
v
136
Tetragon policy
▪ Building a policy means creating a
TracingPolicy with:
▪ The right hook point
▪ The index argument we would like to extract
▪ The data type of the argument
▪ Filtering in Tetragon means applying a
selector on :
▪ The Args extracted
▪ The process, file ...etc
▪ The user
▪ The k8S metadata
▪ TracingPolicy is also defining how to react
on the event ( block, Audit….etc)
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "k8s-api-calls"
spec:
kprobes:
- call: "tcp_connect"
syscall: false
args:
- index: 0
type: "sock"
selectors:
- matchArgs:
- index: 0
operator: "DAddr"
values:
- "10.43.0.1"
- "10.1.0.0/20"
matchBinaries:
- operator: "NotIn"
values:
- "/usr/bin/rancher"
- "/usr/bin/dumb-init“
v
137
Kubearmor capture
▪ KubeArmor will only produces events matching:
▪ The policiy deployed
▪ Or the K8S objects having the right annotations
▪ KubeArmor will capture :
▪ Process
▪ File
▪ Network
▪ Capabilities
▪ Syscalll
▪ KubeArmor Operator provides annotations defining :
▪ The type of events to report ( file, process, network)
▪ How to react on a given event
Clustername
Hostname
Namesapce
Podname
Labels
ParentProcessNam
e
ProcessName
HostPPID
HostPID
PPID
Source
Operation
Data
…
v
138
KubeArmor Policy
▪KubeArmor policies are defined by
building a :
▪KubeArmorClusterPolicy
▪KubeArmorPolicy
▪KubeArmorHostPolicy
▪The policy defines :
▪The Tags
▪The message of the event
▪The Selector to filter to a specific
namespace or workload
▪The rule for the event type ( process, file,
network..Etc)
▪And the action : Block, Audit, Allow
apiVersion: security.kubearmor.com/v1
kind: KubeArmorClusterPolicy
metadata:
name: ksp-nist-remote-access-and-control
spec:
tags: ["NIST","system","ksp","AC-17(1)"]
message: "warning! someone tried to access and
control"
selector:
matchExpressions:
- key: namespace
operator: NotIn
values:
- kube-system
- istio-system
process:
severity: 4
matchPaths:
- path: /usr/bin/ssh
- path: /etc/ssh
action: Audit
v
139
Tracee capture
▪ Tracee will only produce events matching the policies deployed
▪ Tracee will capture :
▪ Security event
▪ Network details
▪ Anything using a given list of syscalls ( process, file…Etc)
▪ The type of events detected will rely on a set of “signatures” defining
▪ What to capture
▪ What to decode
▪ Tracee provides a “sdk” helping us to build our own signature
Processorid
Processid
Cgroupid
Parentprocessid
Userid
processName
Executable
Hostname
Container
Name
Image
Kubernetes
podName
Namespace
eventName
Matchpolicies
Args
..
v
140
Tracee Policy
▪Tracee policies are defined by
configuring a :
▪Policy
▪Tracee provides a default policy
based on the security signature
▪The policy defines :
▪The scope
▪The set of rules defined by a event
name and filters
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
name: sample-data-filter
annotations:
description: sample data filter
spec:
scope:
- global
rules:
- event: vfs_read
filters:
- data.pathname=/etc/*
- data.pathname=/etc/fstab
- data.pathname=/etc/crontab
- data.pathname=/etc/hosts
- data.pathname=/etc/hosts.allow
v
141
Summary
Capture
Filter
Default Policy
Ability to react
Customizable
Stage 3: Observability
142
C
Observability
Performance
Observability
Components
Observability
Performance
• Extend observability with the events produced by :
• Collecting the logs
• Create a parsing rule
• Limit the number of fields exported
• Limit the events by enabling throttling
• Be able to report health metrics related to :
• The policy deployed
• The various components required to run the runtime agent
v
Falco
▪Falco events are available in the Falco agent logs
▪The event structure highly depends on the rule created
and the syscall used.
▪To simply the log collection we usually rely on
FalcoSidekick that provides a large number of
integrations
▪Falco provides a metric server reporting :
▪Metrics related to the rule
▪Actual health of Falco agent
v
145
Tetragon
▪Tetragon expose all the events
directly in the logs of the agent
▪Tetragon expose 2 Prometheus
exporter:
▪Agent: sharing details on the policies
deployed and the health
▪Operator: Sharing the health of the
agent
v
146
Kubearmor
▪Kubearmor will not produce events in
the logs of the agents
▪To collect events you will need to :
▪Enable the logging option on the Relay
Server
▪Or use the Kubearmor receiver that will
collect the logs from the agent. This receiver
is currently only compatible with v0.96 of the
collector
▪Kubearmor is not providing any
metrics
v
147
Tracee
▪Tracee is producing the event
directly at the logs of the agent
▪Tracee can also push the logs
using :
▪FluentForward protocol
▪Webhook endpoint
▪The Traceee agent expose
Prometheus metrics related to the
events produced and the errors.
v
148
Summary
Collecting
events
Extend
observability
Rule metrics
Heath metrics
Stage 4: Performance
149
v
Kpi that we want to measure
• Measure the
latency added
in our
application
Latency
• Cpu usage
• Memory usage
Resource
usage
v
• Using the default policies
Falco
• k8S api calls
• Service account files
• Sensitive files
• Egress communication
• Install tools
• Network activities
• Process spawned
Tetragon
• Block unauthorize binaries
• Sensitive files
• Audit write in sensitive folder
• Suspicious Network tools
• External access
Kubearmo
r
• Sensitive files
• New containers created
• Process spawned
• Security signature
• Read write access from non
root user
Tracee
151
What type of policies we would apply
v
152
The various tests executed
No Agent
Falco
Tetragon with default events
Tetragon with default events & policies
KubeArmor with policies and no events
KubeArmor with policies and events
KubeArmor no policies but with events
Tracee with default policy
Tracee with default policy and custom policies
v
Falco Architecture
Dynatra
ce
Ingress-
nginx
Otel-
demo
defau
lt
Goat-
app
Ungua
rd
Falc
o
v
Tetragon Architecture
Dynatra
ce
Ingress-
nginx
Otel-
demo
defau
lt
Goat-
app
Ungua
rd
Tetrag
on
v
KubeArmor Architecture
Dynatra
ce
Ingress-
nginx
Otel-
demo
defau
lt
Goat-
app
Ungua
rd
KubeArm
or
v
Tracee Architecture
Dynatra
ce
Ingress-
nginx
Otel-
demo
defau
lt
Goat-
app
Ungua
rd
Trace
e
v
Constant Load with no policies
Falco +10,84ms
Tetragon + 5,03ms
KubeArmor +15,73ms
Tracee +110,79ms
v
Constant Load with policies
Falco + 10,84ms
Tetragon +26,13ms
KubeArmor +176,62ms
Tracee +114,23ms
v
Constant Load with policies & events
Falco + 10,84ms
Tetragon +26,13ms
KubeArmor +477,34ms
Tracee +114,23ms
v
160
Summary
Latency
CPU
memory
v
161
Conclusion
v
162
Conclusion
1
2 3
v
▪Looking for educational content on Observability , Checkout the YouTube
Channel :
Is It Observable
Is it observable
Thank You
cloud-native needs you !
● Speaker(e)s
● Volontaires
Scannez moi
pour devenir
speaker cloud-
native !
meetup.com/Devops-Aix-Marseille
ibd.sh/openbar
conference-hall.io/meetup-devops-aix-
marseille
Quizz !!

meetup devops aix-marseille - 2025-01 --

  • 1.
    January Les bonnes résolutionsde la sécu dans mon cluster Kubernetes 15/01/2025
  • 2.
  • 3.
    PRESENTATION D’EASY PARTNER Experts dela mise en relation de profils tech DÉCOUVRIR NOTRE OFFRE
  • 4.
    PRÉSENTATION D’EASY PARTNER Trois piliers,trois offres adaptées Cabinet de recrutement Du profil junior au directeur technique, notre équipe de Tech Recruiters spécialisés par stack technique vous accompagne sur vos recherches, forts de nos +3000 recrutements pour plus de 500 clients Assistance technique Vous cherchez à lancer un projet rapidement, apporter une expertise spécifique à vos équipes, ou encore absorber un pic de charge, nous vous trouvons, comme nous le faisons avec nos 300 clients, le talent adapté à vos enjeux en 48h. Formation Bénéficiez de nos 11 ans d’expertise pour transformer vos équipes en experts autonomes de la gestion de talents tech, nous vous transmettons l’expérience Easy Partner
  • 5.
    PRÉSENTATION D’EASY PARTNER Experts dela mise en relation de profils tech Spécialiste de la mise en relation de profils tech depuis 11 ans Spécialisation de nos collaborateurs par stack technique, formations spécifiques par d’ex-ingénieurs > 3500 placements réussis : nous plaçons > 1 ingénieur par jour sur un projet (sur un poste en CDI ou en prestation de service) 96% des candidats valident leur période d’essai, 68% des missions sont renouvelées Valider plus rapidement les postes grâce à nos recruteurs experts de leur marché technologique Pour vous c’est l’assurance de… Réduire le temps passé par recherche grâce à notre compréhension rapide de vos besoins et notre pré-sélection rigoureuse Candidats qui restent chez poste / mission, dont les motivations profondes sont alignées avec les valeurs des clients
  • 6.
    Pour vous c’estl’assurance de… PRÉSENTATION D’EASY PARTNER La base de profils tech la plus complète de France > 100.000 profils tech qualifiés en base +1.000 nouveaux candidats qualifiés par nos collaborateurs par mois dans notre base 2/3 des candidats placés en 2022 étaient issus de notre base de profils Staffer vos besoins plus rapidement grâce à notre base activable pré- qualifiée Avoir accès aux meilleurs profils du marché en un temps record 90% des candidats placés en 2022 ont été débauchés
  • 7.
    ZOOM SUR… Quelques chiffres 96% 30 TechRecruiter Passionnés et expert du métier des candidats placés par Easy Partner ont validé leur période d’essai 45% des candidats rencontrés par nos clients ont reçu une offre
  • 8.
    ZOOM SUR… Nos clientsqui nous font confiance Grands Comptes ETI PME et Start-ups +750 CLIENTS SUR PLUS DE 30 SECTEURS D’ACTIVITÉ
  • 9.
  • 10.
    Qui sommes-nous ? KatiaHimeur Henrik Rexed
  • 11.
    Agenda I. Intro II. Sécuriservotre cluster Kubernetes III. À quel agent confier vos clusters k8s IV. Jeu + goodies V. Apéro / Buffet
  • 12.
  • 13.
  • 14.
    PR Github CNCFhttps://github.com/cncf/communitygroups/issues/229 ⇒
  • 15.
    Teasing ● Passage surconference-hall ○ https://conference-hall.io/team/meetup-devops-aix-marseille/meetup-devops-aix-marseille ○ N’hésitez pas à faire passer le mot ou à poster votre abstract ● Un agenda déjà calé ● Conclusions ○ Le meetup évolue ○ Des infos au cours du début d’année ○ Ne ratez pas le meetup de mars… 😉
  • 16.
    Agenda 2025 ● Janvier:2025-01-15 🤦 ● Février: 2025-02-06 ● Mars: 2025-03-06 ● Avril: 2025-04-24 ● Mai: 2025-05-22 ● Juin: 2025-06-12 ● Juillet: 2025-07-03
  • 17.
    Sécuriser votre cluster Kubernetes: la recette pour réussir, étape par étape !
  • 18.
    Securing your Kubernetescluster: Your step-by-step guide to success! janvier 2025 Katia HIMEUR
  • 19.
  • 20.
    Who am I? •🪪 Katia HIMEUR • 💻 Computer scientist • ☁️CTO & co-founder at Cockpit io, cloud & DevOps specialist • 💻 ‍ 💻‍ SRE/Cloud/DevOps consultant for several years • Duchess Core Team Member ❤️#Cloud #DevOps #Containers #Serverless #GitOps #IaC #CICD ❤️ Securing your Kubernetes cluster: a step-by-step guide to success! 2 0 janvier 2025
  • 21.
  • 22.
    Kubernetes, the leadingcontainer orchestrator janvier 2025 Source : https://marketsnresearch.com/report/1649/global-kubernetes-market 2 2 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 23.
    The new cloudOS janvier 2025 Source : https://www.dynatrace.com/news/blog/kubernetes-in-the-wild-2023/ 2 3 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 24.
    janvier 2025 Attackers onthe lookout 2 4 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 25.
    Safety not alwaysa priority janvier 2025 2 5 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 26.
    26 Why is itimportant to secure your cluster?
  • 27.
    K8s's popularity increasesattacker interest janvier 2025 2 7 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 28.
    Flexibility Complexity Errors ⥤⥤ janvier 2025 Configuration errors are a security risk Complexity can lead to configuration errors Its flexibility leads to complex configurations Kubernetes is extremely flexible 2 8 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 29.
    Extended attack surface janvier2025 https://kubernetes.io/docs/concepts/overview/components/ 2 9 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 30.
    Scalability and resiliencecan be a vector for vulnerability propagation janvier 2025 3 0 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 31.
    Company background Need : ●Compliancewith specific standards and regulations ●Protect sensitive and critical applications and infrastructure ●Secure the sensitive data that may pass through janvier 2025 3 1 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 32.
    32 What will wesee during this talk?
  • 33.
    What will wesee during this talk? ●Understand the different types of attackers and attack vectors ●A non-exhaustive list of concrete actions you can take to secure your cluster ●Focus on managed Kubernetes clusters ●The list of tools provided is for information only janvier 2025 3 3 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 34.
  • 35.
    Attacker types 35 janvier2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 36.
  • 37.
    Main attack vectors 37janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 38.
    Main attack vectors janvier2025 3 8 Securing your Kubernetes cluster: a step-by-step guide to success! Insecure Server API Compromised containers Misconfigured access controls Exposed dashboards Incorrect network configuration Compromised nodes Compromised secrets Supply chain (dependencies…) …
  • 39.
    39 Rely on communityand ecosystem support
  • 40.
    Rely on Communityand ecosystem support ●Strong focus on safety ●Continuous improvement and disclosure of vulnerabilities ●Goals : ○Share security best practices ○Encouraging corrective action ○Stay ahead of potential threats janvier 2025 4 0 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 41.
    41 How do yousecure your Kubernetes cluster?
  • 42.
    Zero trust architecture 42janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 43.
    Zero trust architecture ●Approachto designing and implementing IT systems where trust is totally eliminated ●Never trust, always verify janvier 2025 43 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 44.
    Secure API Server janvier2025 44 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 45.
    Why API serveris critical ? ●Critical cluster component ●External and internal communications gateway ●Manage ○Authentication and authorization ○Data validation and storage in etcd ○Orchestration and resource management ○Scalability and performance janvier 2025 45 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 46.
    Limit API Serverexposure? ●Restrict access ●Encrypt all flows with API Server janvier 2025 46 Private clusters Security groups Endpoint access control Network ACLs Securing your Kubernetes cluster: a step-by-step guide to success!
  • 47.
    Identity and Access Management janvier2025 47 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 48.
    Identity and AccessManagement ●Integration with Cloud IAM ●Use an SSO portal to connect to your clusters (oAuth2, OpenID Connect or LDAP) ●Enable multifactor authentication janvier 2025 48 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 49.
    Authentication janvier 2025 49 Securing yourKubernetes cluster: a step-by-step guide to success!
  • 50.
    Using RBAC 50 janvier2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 51.
    Using RBAC ●What isit? ○RBAC: Role-Based Access Control ○Resource access control based on user roles and service accounts ○Defines who can do what 5 1 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 52.
    Using RBAC ●Good practice ○Applythe principle of least privilege to users and service accounts 52 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 53.
    Example of anRBAC role 5 3 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 54.
    Pod Security Admission janvier2025 54 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 55.
    Pod Security Admission ●Kubernetesnative admission controller to enforce Pod Security standards policies ●Ensure containerized workloads are secure against known privilege escalations ●Pod security restrictions are applied at the namespace level ●Enabled by default in the latest versions of Kubernetes 55 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 56.
    Pod Security StandardsPolicies 56 Restricted ●Very restrictive policy ●Follows good curing practices Privileged ●No restrictions ●Climbing possibilities Baseline ●Minimum restriction policy ●Prevents the most common climbs janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 57.
    Apply a safetypolicy 5 7 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 58.
    Alternatives 58 janvier 2025 Securingyour Kubernetes cluster: a step-by-step guide to success!
  • 59.
    Using network policies 59janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 60.
    Network status report ●Default:pods are not isolated, all traffic flows are allowed (ingress and egress). ●Best practice: Restrict pod-to-pod communication to the strict minimum ●How ? Use network policies 60 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 61.
    What are networkpolicies? ●Allows you to define how pods are authorized to communicate with : ○Other pods ○Other namespaces ○IP blocks 61 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 62.
    Example of apolicy that prohibits all outbound flow 62 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 63.
    Alternatives 63 janvier 2025 Network policies SecurityNetwork Observability Performance Securing your Kubernetes cluster: a step-by-step guide to success!
  • 64.
    Securing your secrets 64janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 65.
    Secrets... Not verysecret 😱 ●A Kubernetes secret is an object used to store sensitive data (passwords, tokens, SSH keys, etc.). ●Default : ○No encryption ○Stored unencrypted ○Base64 encoded value 65 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 66.
    Securing your secrets ●UseRBAC to secure access ●Rotate secret ●Audit access to secrets ●Regularly review and update access policies ●Prevent secrets from ending up in logs ●Don't hard code secrets 66 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success! Gitleaks
  • 67.
    Turn secrets intovolume 👍 Exposing them as environment variables👎 68 janvier 2025 ServiceAccount secrets can only be mounted on specific resource types. Securing your Kubernetes cluster: a step-by-step guide to success!
  • 68.
    Turn secrets intovolume 👍 Exposing them as environment variables👎 69 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success! Volume mounting ● Secrets isolated within the file system ● Not visible in dumps and crash logs ● Granular permissions possible ● Automatic refresh possible Environment variables ● Visible in process dumps ● Appear in crash logs ● Inherited by child processes ● Risk of accidental logging
  • 69.
    Use external secretmanagement tools 7 0 AWS Secrets Manager Azure Key Vault Google Cloud Secret Manager janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 70.
    Securing your secrets ●TheSecrets Store CSI Driver enables integration of external secrets managers with Kubernetes ●Integration is via a CSI (Container Storage Interface) volume. ●This driver allows you to mount several secrets, keys, certificates, etc. stored in these external secret managers. ●Once the volume has been attached, the data is mounted in the container's file system. 71 janvier 2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 71.
    Audit logging 72 janvier2025 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 72.
    Audit logging ●Record everyrequest made to the API server ●Analyze logs to detect suspicious and unusual activity ●Use security tools to analyze and react to logs in real time ●Define audit policies to fine-tune the configuration of events to be recorded janvier 2025 73 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 73.
    Example of anaudit policy janvier 2025 74 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 74.
    Update and patchmanagement janvier 2025 75 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 75.
    Update and patchmanagement ●Apply update and security patches, automatically if possible, and quickly : ○Kubernetes clusters ○all components deployed on clusters ○nodes ●Use tools that scan all Kubernetes cluster components, including images ●Respect best operating system practices janvier 2025 76 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 76.
    Securing containers janvier 2025 77 Securingyour Kubernetes cluster: a step-by-step guide to success!
  • 77.
    Securing containers ●Use onlytrusted images ●Reduce image size to reduce attack surface ●Use immutable containers to avoid runtime changes janvier 2025 78 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 78.
    Securing containers ●Scanning yourcontainer images: before and after deployment ●Sign your container images ●Secure and protect your containers using tools that react to the detection of abnormal events janvier 2025 79 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 79.
    Securing containers janvier 2025 80 Securingyour Kubernetes cluster: a step-by-step guide to success!
  • 80.
    Securing containers janvier 2025 81 Securingyour Kubernetes cluster: a step-by-step guide to success!
  • 81.
    Admission controllers janvier 2025 82 Securingyour Kubernetes cluster: a step-by-step guide to success!
  • 82.
    Admission controllers ●Controllers thatintercept requests to the API server before objects are persisted in the etcd database ●Two types of controller : ○Validating admission controller : Validates or rejects requests ○Mutating admission controller : Modifies the requests it accepts janvier 2025 83 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 83.
    Configuration example janvier 2025 84 Thisadmission controller prevents the API server from being overloaded by requests applying rate limiting. Securing your Kubernetes cluster: a step-by-step guide to success!
  • 84.
  • 85.
    What is aservice mesh? janvier 2025 86 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 86.
    ●Dedicated infrastructure layeradded to applications ●Designates both the tools and the network domain created ●Allow us to : ○understand traffic ○make decisions based on traffic type or origin ○reduce the complexity of network management in a microservice context What is a service mesh? janvier 2025 87 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 87.
    Service Mesh functionalities janvier 2025 88 Securingyour Kubernetes cluster: a step-by-step guide to success!
  • 88.
    Service mesh functionalities janvier2025 89 Securing your Kubernetes cluster: a step-by-step guide to success! ●Observability ●Traffic management ●Security ●A/B testing ●Canary deployment ●Rate limiting ●Access control ●Encryption (including mTLS) ●End-to-end authentication ●Service discovery
  • 89.
    janvier 2025 90 Securing yourKubernetes cluster: a step-by-step guide to success! Some services mesh
  • 90.
    Using services mesh janvier2025 91 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 91.
  • 92.
    Requests & limits janvier2025 96 Securing your Kubernetes cluster: a step-by-step guide to success! ●Best practices ○Set memory requests lower than or equal to limits ○Limit CPU on sensitive workloads
  • 93.
    Requests & limits janvier2025 97 Securing your Kubernetes cluster: a step-by-step guide to success! ●Why? ○Restrict resources used by pods on nodes ○Avoid the effects of a denial-of-service attack
  • 94.
  • 95.
    Real-time protection janvier 2025 106 Securingyour Kubernetes cluster: a step-by-step guide to success! ●Detect abnormal behavior, security threats and compliance violations ●Be alerted in real time. ●Some tools rely on kernel events, enriched with container and Kubernetes metadata, to succeed in their protection missions.
  • 96.
    Real-time protection janvier 2025 107 Securingyour Kubernetes cluster: a step-by-step guide to success!
  • 97.
  • 98.
    Follow best practicesand recommendations janvier 2025 109 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 99.
    Follow best practicesand recommendations janvier 2025 110 Securing your Kubernetes cluster: a step-by-step guide to success! ●Follow recommendations and best practices ○Security Checklist from the Kubernetes community: https://kubernetes.io/docs/concepts/security/security-checklist/ ○CIS (Center for Internet Security (CIS) ) Kubernetes Benchmark ○Recommendations from cloud providers ○Keeping up to date
  • 100.
  • 101.
    Conclusion ●Securing Kubernetes environmentsis fundamental to protecting your company's interests, ensuring compliance and maintaining trust. ●As Kubernetes continues to evolve, so will security strategies, requiring constant attention and adaptation. ●Today's list is not exhaustive. No list can be, as attackers show their creativity. ●Safety is a matter of continuous improvement and constant attention janvier 2025 114 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 102.
    Thank you /in/katiahimeur/ 🔗 blog.cockpitio.com 🔗www.cockpitio.com /company/cockpit-io/ Keep in touch 17/04/2024 115 Securing your Kubernetes cluster: a step-by-step guide to success!
  • 103.
    A team ofenthusiasts, guided by DevOps culture, to bring you the best of the Cloud!
  • 104.
    À quel agentconfier vos clusters k8s : Falco, Tetragon ou KubeAmor ?
  • 105.
    What agent totrust with your k8s: Henrik Rexed CloudNative Advocate
  • 106.
    v Cloud Native Advocate •15+ years of Performance engineering • Owner of : IsitObservable Producer of : Perfbytes Henrik Rexed
  • 107.
    v ▪No Spartans, Eaglesor Shields were harmed in the making of this presentation ▪The intention behind this talk track is not to assign blame to any CNCF project. ▪This session is made to help the community in choosing their runtime security agent. CONFIDENTIAL 120 DISCLAIMER
  • 108.
  • 109.
  • 110.
  • 111.
    v • What weexpect from a runtime security agent ▪ Compare each solution in : • Desing Experience • The Components required • The Observability • See various Benchmarking results • Recommendation on which agents needs to used under specific conditions If you stay with me you will ...
  • 112.
  • 113.
  • 114.
  • 115.
    C Observability Performance Components Components Observability Performance • A securityagent usually relies on ebpf to collect the kernel or user event of our environment • In k8S , the security agent has usually at least : • A daemonset deploying the ebpf probe on each node of our cluster
  • 116.
  • 117.
    Stage 2: Capture& Filter 130
  • 118.
    C Filter Components Observability Performance • Be ableto capture events from our k8S cluster related to : • Process launched/ finished • File Access Read/Write • Networking details • Change of privileges • …etc. • Get the details attached to the event: • Path: executable path, file path… • Process details: pid, process name , • User: userid, groupid • Capabilities • Container: image , name • K8s : pod name, namespace, ..etc
  • 119.
    C Observability Performance Filter Components Observability Performance • Filtering meanshaving the option to create a policy filtering the event on : • Process properties • User details • K8S metada • File details • Syscall/ kernel functions or event • Have Predefined Rules • Be able to react
  • 120.
    v Falco Capture ▪ Falcois rule engine producing logs if the Falco event is matching a rule ▪ Falco receive event from : ▪ Falco kernel agent ▪ External plugins ▪ Falco provides a SDK allowing us to build our custom Plugin ▪ Falco agents captures : ▪ Process details ▪ Syscall ▪ Tracepoints ▪ File Hostname output Output field_fields Pid Proc.name Container.nam e Proc.execpath User.name K8s.pod.name … Tags … Rule source
  • 121.
    v 134 Falco rule ▪Falco startsby loading the rule files that would be used to generate the events. By default, Falco provide a default sets of rule. ▪Falco provides “fields” helping us to define our filtering rule based on: ▪Event information ▪Proces ▪File, ▪Syscalls ▪a global rule file allows to reuse filtering rules or conditions macro: access_file condition: evt.type=open - rule: program_accesses_file desc: track whenever a set of programs opens a file condition: (access_file) and proc.name in (cat, ls) output: a tracked program opened a file (user=%user.name command=%proc.cmdline file=%fd.name) priority: INFO - rule: test_rule desc: test rule description condition: evt.type = close output: user=%user.name command=%proc.cmdline file= %fd.name priority: INFO enabled: false
  • 122.
    v 135 Tetragon capture ▪ Tetragonwill by default produce events related to : ▪ Process execution ▪ Process exit ▪ If you deploy Policies, it will produce events matching our policy rule ▪ Policy required to defined a hook point to our system using: ▪ Kprobe ▪ Tracepoints ▪ Ubprobe ▪ llsm Process_exe c Process Pid Uid Binary Arguments Pod … Parent Node name
  • 123.
    v 136 Tetragon policy ▪ Buildinga policy means creating a TracingPolicy with: ▪ The right hook point ▪ The index argument we would like to extract ▪ The data type of the argument ▪ Filtering in Tetragon means applying a selector on : ▪ The Args extracted ▪ The process, file ...etc ▪ The user ▪ The k8S metadata ▪ TracingPolicy is also defining how to react on the event ( block, Audit….etc) apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "k8s-api-calls" spec: kprobes: - call: "tcp_connect" syscall: false args: - index: 0 type: "sock" selectors: - matchArgs: - index: 0 operator: "DAddr" values: - "10.43.0.1" - "10.1.0.0/20" matchBinaries: - operator: "NotIn" values: - "/usr/bin/rancher" - "/usr/bin/dumb-init“
  • 124.
    v 137 Kubearmor capture ▪ KubeArmorwill only produces events matching: ▪ The policiy deployed ▪ Or the K8S objects having the right annotations ▪ KubeArmor will capture : ▪ Process ▪ File ▪ Network ▪ Capabilities ▪ Syscalll ▪ KubeArmor Operator provides annotations defining : ▪ The type of events to report ( file, process, network) ▪ How to react on a given event Clustername Hostname Namesapce Podname Labels ParentProcessNam e ProcessName HostPPID HostPID PPID Source Operation Data …
  • 125.
    v 138 KubeArmor Policy ▪KubeArmor policiesare defined by building a : ▪KubeArmorClusterPolicy ▪KubeArmorPolicy ▪KubeArmorHostPolicy ▪The policy defines : ▪The Tags ▪The message of the event ▪The Selector to filter to a specific namespace or workload ▪The rule for the event type ( process, file, network..Etc) ▪And the action : Block, Audit, Allow apiVersion: security.kubearmor.com/v1 kind: KubeArmorClusterPolicy metadata: name: ksp-nist-remote-access-and-control spec: tags: ["NIST","system","ksp","AC-17(1)"] message: "warning! someone tried to access and control" selector: matchExpressions: - key: namespace operator: NotIn values: - kube-system - istio-system process: severity: 4 matchPaths: - path: /usr/bin/ssh - path: /etc/ssh action: Audit
  • 126.
    v 139 Tracee capture ▪ Traceewill only produce events matching the policies deployed ▪ Tracee will capture : ▪ Security event ▪ Network details ▪ Anything using a given list of syscalls ( process, file…Etc) ▪ The type of events detected will rely on a set of “signatures” defining ▪ What to capture ▪ What to decode ▪ Tracee provides a “sdk” helping us to build our own signature Processorid Processid Cgroupid Parentprocessid Userid processName Executable Hostname Container Name Image Kubernetes podName Namespace eventName Matchpolicies Args ..
  • 127.
    v 140 Tracee Policy ▪Tracee policiesare defined by configuring a : ▪Policy ▪Tracee provides a default policy based on the security signature ▪The policy defines : ▪The scope ▪The set of rules defined by a event name and filters apiVersion: tracee.aquasec.com/v1beta1 kind: Policy metadata: name: sample-data-filter annotations: description: sample data filter spec: scope: - global rules: - event: vfs_read filters: - data.pathname=/etc/* - data.pathname=/etc/fstab - data.pathname=/etc/crontab - data.pathname=/etc/hosts - data.pathname=/etc/hosts.allow
  • 128.
  • 129.
  • 130.
    C Observability Performance Observability Components Observability Performance • Extend observabilitywith the events produced by : • Collecting the logs • Create a parsing rule • Limit the number of fields exported • Limit the events by enabling throttling • Be able to report health metrics related to : • The policy deployed • The various components required to run the runtime agent
  • 131.
    v Falco ▪Falco events areavailable in the Falco agent logs ▪The event structure highly depends on the rule created and the syscall used. ▪To simply the log collection we usually rely on FalcoSidekick that provides a large number of integrations ▪Falco provides a metric server reporting : ▪Metrics related to the rule ▪Actual health of Falco agent
  • 132.
    v 145 Tetragon ▪Tetragon expose allthe events directly in the logs of the agent ▪Tetragon expose 2 Prometheus exporter: ▪Agent: sharing details on the policies deployed and the health ▪Operator: Sharing the health of the agent
  • 133.
    v 146 Kubearmor ▪Kubearmor will notproduce events in the logs of the agents ▪To collect events you will need to : ▪Enable the logging option on the Relay Server ▪Or use the Kubearmor receiver that will collect the logs from the agent. This receiver is currently only compatible with v0.96 of the collector ▪Kubearmor is not providing any metrics
  • 134.
    v 147 Tracee ▪Tracee is producingthe event directly at the logs of the agent ▪Tracee can also push the logs using : ▪FluentForward protocol ▪Webhook endpoint ▪The Traceee agent expose Prometheus metrics related to the events produced and the errors.
  • 135.
  • 136.
  • 137.
    v Kpi that wewant to measure • Measure the latency added in our application Latency • Cpu usage • Memory usage Resource usage
  • 138.
    v • Using thedefault policies Falco • k8S api calls • Service account files • Sensitive files • Egress communication • Install tools • Network activities • Process spawned Tetragon • Block unauthorize binaries • Sensitive files • Audit write in sensitive folder • Suspicious Network tools • External access Kubearmo r • Sensitive files • New containers created • Process spawned • Security signature • Read write access from non root user Tracee 151 What type of policies we would apply
  • 139.
    v 152 The various testsexecuted No Agent Falco Tetragon with default events Tetragon with default events & policies KubeArmor with policies and no events KubeArmor with policies and events KubeArmor no policies but with events Tracee with default policy Tracee with default policy and custom policies
  • 140.
  • 141.
  • 142.
  • 143.
  • 144.
    v Constant Load withno policies Falco +10,84ms Tetragon + 5,03ms KubeArmor +15,73ms Tracee +110,79ms
  • 145.
    v Constant Load withpolicies Falco + 10,84ms Tetragon +26,13ms KubeArmor +176,62ms Tracee +114,23ms
  • 146.
    v Constant Load withpolicies & events Falco + 10,84ms Tetragon +26,13ms KubeArmor +477,34ms Tracee +114,23ms
  • 147.
  • 148.
  • 149.
  • 150.
    v ▪Looking for educationalcontent on Observability , Checkout the YouTube Channel : Is It Observable Is it observable
  • 151.
  • 152.
    cloud-native needs you! ● Speaker(e)s ● Volontaires Scannez moi pour devenir speaker cloud- native !
  • 153.
  • 154.

Editor's Notes

  • #24 Se conformer à certaines normes et réglementation : la sécurité n’est pas un prérequis technique mais légale
  • #27 Se conformer à certaines normes et réglementation : la sécurité n’est pas un prérequis technique mais légale
  • #28 Se conformer à certaines normes et réglementation : la sécurité n’est pas un prérequis technique mais légale
  • #29 Se conformer à certaines normes et réglementation : la sécurité n’est pas un prérequis technique mais légale
  • #30 Se conformer à certaines normes et réglementation : la sécurité n’est pas un prérequis technique mais légale
  • #31 Se conformer à certaines normes et réglementation : la sécurité n’est pas un prérequis technique mais légale
  • #33 Se conformer à certaines normes et réglementation : la sécurité n’est pas un prérequis technique mais légale
  • #36 Se conformer à certaines normes et réglementation : la sécurité n’est pas un prérequis technique mais légale
  • #53 Role read only sur les pods du namespace default
  • #70 Plus robuste