Cilium provides network security and visibility for microservices. It uses eBPF/XDP to provide fast and scalable networking and security controls at layers 3-7. Key features include identity-based firewalling, load balancing, and mutual TLS authentication between services. It integrates with Kubernetes to apply network policies using standard Kubernetes resources and custom CiliumNetworkPolicy resources for finer-grained control.

1. Cilium
L7 Aware Network Security for
Microservices using BPF & XDP

2. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server
App
Yearly
Low
Evolution of Application Design & Delivery Frequency

3. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server
App
Yearly
Low
3-Tier App
Monthly
Moderate
Evolution of Application Design & Delivery Frequency

4. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server
App
Yearly
Low
Distributed
Microservices
10-100 x’s / day
Extreme
3-Tier App
Monthly
Moderate
Evolution of Application Design & Delivery Frequency

5. Network Security
has barely evolved
$ iptables -A INPUT -p tcp
-s 15.15.15.3 --dport 80
-m conntrack --ctstate NEW
-j ACCEPT
The world still runs on iptables
matching IPs and ports:

6. Your HTTP ports be like …

7. Network Security
for Microservices
Gordon the intern
has a brilliant
idea…

8. Gordon wants to build a service
to tweet out all job offerings.
We’re Hiring!
Tweet
Service

9. GET /healthz
GET /jobs/{id}
GET /applicants/{job-id}
POST /jobs
API
GET /jobs/{id}
Jobs API
Service
Tweet
Service
The Jobs API service has all the
data Gordon needs.

10. GET /healthz
GET /jobs/{id}
GET /applicants/{job-id}
POST /jobs
API
GET /jobs/331
GET /jobs/{id}
Jobs API
Service
Tweet
Service
Gordon uses the”GET /jobs/” API

11. GET /healthz
GET /jobs/{id}
GET /applicants/{job-id}
POST /jobs
API
GET /jobs/331
GET /jobs/{id}
TLS Jobs API
Service
Tweet
Service
Good thinking Gordon . Developer etiquette.
Super simple stuff.
Gordon uses mutual TLS Auth

12. L3/L4
GET /healthz
GET /jobs/{id}
GET /applicants/{job-id}
POST /jobs
API
GET /jobs/331
The security team has L3/L4 network
security in place for all services
GET /jobs/{id}
Jobs API
Service
Tweet
Service
TLS
iptables -s 10.1.1.1
-p tcp --dport 80
-j ACCEPT

13. Jobs API
Service
L3/L4
GET /healthz
GET /jobs/{id}
GET /applicants/{job-id}
POST /jobs
API
exposed
exposed
exposed
GET /jobs/331
Large parts of the API are still exposed
unnecessarily
Tweet
Service
GET /jobs/{id}
TLS
iptables -s 10.1.1.1
-p tcp --dport 80
-j ACCEPT

14. Not exactly
least privilege
Security

15. GET /healthz
GET /jobs/{id}
GET /applicants/{job-id}
POST /jobs
API
GET /jobs/331
Back to the drawing board…
GET /jobs/{id}
TLS Jobs API
Service
Tweet
Service

16. L3/L4
GET /healthz
GET /jobs/{id}
GET /applicants/{job-id}
POST /jobs
API
GET /jobs/331
Least privilege security for microservices
GET /jobs/{id}
ALLOW “GET /jobs/.*”
FROM “Tweets”
TLS Jobs API
Service
Tweet
Service

17. Network Security for
Microservices using BPF

18. We demand
a demo

20. BPF - The
Superpowers
inside Linux

21. Powers of BPF: XDP - DDoS mitigation
Metric iptables / ipset XDP
DDoS rate [packets/s] 11.6M 11.6M
Drop rate [packets/s] 7.1M 11.6M
Time to load rules [time] 3 min 20 sec 31 sec
Latency under load [ms] 2.3ms 0.1ms
Throughput under DDoS [Gbit/s] 0.014 6.5
Requests/s under DDoS [kReq/s] 0.28 82.8
Sender: Send 64B packets as fast as possible è Receiver: Drop as fast as possible
Source: Daniel Borkmann’s presentation yesterday:
http://schd.ws/hosted_files/ossna2017/da/BPFandXDP.pdf

22. Facebook published BPF/XDP numbers
for L3/L4 LB at Netdev 2.1
ECMP L7 LBL3/L4 LB App

23. Source: https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf
Facebook published BPF/XDP numbers
for L3/L4 LB at Netdev 2.1
BPF/XDP throughput
IPVS throughput

24. Kernel community declared
iptables obsolete
• Kernel community decided to stop iptables development and
replace it with BPF
• https://lwn.net/Articles/747504/
• https://www.mail-archive.com/netdev@vger.kernel.org/msg217095.html

26. WHAT ABOUT L7?

27. NodeNode
Service
Operating
System
Service
Network
Socket
TCP/IP
Socket
TCP/IP
Socket
TCP/IP
Socket
TCP/IP
Socket
TCP/IP
Socket
TCP/IP
Consequences:
• 3x Socket memory requirement
• 3x TCP/IP stack traversals
• 3x Context switches
• Complexity
Currently using Sidecar/ServiceMesh
Network

28. Can we turn
the sidecar
into a racecar?

29. Service
Operating
System
Service
Network
Socket
TCP/IP
Socket Redirect & In-Kernel L7
In-Kernel L7
Network
Socket
TCP/IP
Socket Redirect
Socket
TCP/IP
In-Kernel L7
kTLS

30. In-Kernel L7 with BPF

31. Cilium Summary
• CNI and libnetwork plugin
• Kubernetes, Docker, Mesos
• Security at L3-L7
• L3-L4: Identity/Label based or CIDR based
• L7: HTTP, Kafka, gRPC
• Load-balancing (inside cluster)
• Say goodbye to iptables
• Minimal dependencies
• Only requires kvstore, written in go

32. @ciliumproject
http://github.com/cilium/cilium
Thank You! Questions?
Tutorial / Getting Started:
http://cilium.io/try

33. Node 2Node 1
Task
Operating
System
Kernel Proxy
Task
Network
Socket
KProxy
with
BPF
TCP/IP
Socket
TCP/IP
KProxy
with
BPF
kTLS kTLS
Sidecar
Proxy
Sidecar
Proxy
Network

34. Kubernetes Integration

35. Kubernetes Integration
NetworkPolicy
Standard Resources
L3, L4 policy (ingress only in k8s 1.7)

36. Kubernetes Integration
NetworkPolicy
Services
Standard Resources
L3, L4 policy (ingress only in k8s 1.7)
ClusterIP, NodePort, LoadBalancer

37. Kubernetes Integration
NetworkPolicy
Services
Standard Resources
L3, L4 policy (ingress only in k8s 1.7)
Pods Pod Labels to specify policy on
ClusterIP, NodePort, LoadBalancer

38. Kubernetes Integration
NetworkPolicy
Services
Standard Resources
L3, L4 policy (ingress only in k8s 1.7)
Nodes
Pods Pod Labels to specify policy on
ClusterIP, NodePort, LoadBalancer
NodeIP to Node CIDR mapping

39. Kubernetes Integration
NetworkPolicy
CiliumNetworkPolicy
Services
Standard Resources
Custom Resource Definitions (CRD)
L3, L4 policy (ingress only in k8s 1.7)
L3 (Labels/CIDR), L4, L7 (ingress & egress)
Nodes
Pods Pod Labels to specify policy on
ClusterIP, NodePort, LoadBalancer
NodeIP to Node CIDR mapping

40. Should I encapsulate or not?
Node 1
Node 2
Node 3
Encap
Encap
Encap
Mode I: Overlay

41. Should I encapsulate or not?
Node 1
Node 2
Node 3
Encap
Encap
Encap
Mode I: Overlay
Name NodeIP Node CIDR
Node 1 192.168.10.1 10.0.1.0/24
Node 2 192.168.10.8 10.0.2.0/24
Node 3 192.168.10.9 10.0.3.0/24
Kubernetes Node resources table:
Installation
Run the kube-controller-
manager with the --allocate-
node-cidrs option

42. Should I encapsulate or not?
Mode I: Overlay Mode II: Native Routing
Node 1
Node 2
Node 3
L3
Network
Use case:
• Run your own routing daemon
• Use the cloud provider’s router
Use case:
• Simple
• “Just works” on Kubernetes
Node 1
Node 2
Node 3
Encap
Encap
Encap

43. L3 Policy (Labels Based)
Metadata
Allow from
pods
Pods the policy
applies to…
From Pod
To Pod

44. L3 Policy (CIDR)
Metadata
Allow to
IP 8.8.8.8/32
Pods the policy
applies to…
To CIDR
From Pod

45. L4 Policy
Metadata
Policy applies
to pods …
Allow incoming
on port 80
Pod
To Port

46. L7 Policy – Only allow “GET /v1/”
L4 Policy
Rule 1:
Allow “GET /v/1”
Rule 2:
Allow PUT
If header is set
Allowed
API
Calls