Service mesh is a powerful pattern for implementing strong zero-trust networking practices, introducing better network observability, and allowing for more fine-grained traffic control. Up until now, the sidecar pattern was used to implement service-mesh capability but as the technology matures, a new pattern has emerged: sidecarless service mesh. Two prominent open-source networking projects, Cilium and Istio, have implemented a sidecar-free approach to service mesh but they both make interesting design decisions and tradeoffs. In this talk we review the architecture of both, focusing on the pros and cons of implementations such as mutual authentication, ingress, and observability.

2. Christian Posta - Global Field CTO, Solo.io
Sidecar-less Service Mesh
Architectures: Cilium and Istio

3. VP, Global Field CTO, Solo.io
@christianposta
christian@solo.io
/in/ceposta
Christian Posta

4. Service Mesh is Networking Infrastructure

5. Business Value of A Service Mesh
Security Observability Traffic Control
Avoid breaches, implement
policy, simplify apps, satisfy
industry compliance
Reduce MTTR (Mean Time
To Recover), measure
changes, improve operations
Improve business
continuity, failover, high
availability, cost control

6. The “First” Service Mesh
(Linkerd 1.x)

7. LinkerD (1.x) Architecture

8. Challenges with LinkerD (1.x)
● JVM-based, difficult to size/constrain
● High tail latencies
● Difficult to require traffic to go through the proxy
● Noisy neighbor problems (unconstrained L7 issues)

9. The Case For the Sidecar
● Per-host proxy resource consumption is unpredictable
● Per-host proxy must ensure fairness and QoS, or the application risks
starvation
● Upgrades, blast radius, etc, affect all workloads on that node (or
worse)
● Per-host proxy must account for the key material for all workloads on
the node, becomes a new attack vector
The case for the sidecar:
https://thenewstack.io/ebpf-or-not-sidecars-are-the-future-of-the-service-mesh/

10. Move Networking Closer to Application

11. Move Networking Closer to Application

12. Benefits of Sidecar Containers
● Transparent *
● Part of the application lifecycle
● Finer grained, can associate workload identity (SPIFFE, etc),
pod-level encryption
● Single-tenant (ie, per workload identity)
● No “noisy neighbor problems”
● Customizable

13. Sidecars were a
“necessary point in time implementation”
to deliver networking value

14. Drawbacks to Service Mesh Sidecars
● Container race conditions
● Security: cert/key material
● Difficult to size / easy to over-provision
● Jobs/CronJobs have issues
● Apps need to be aware
● Can be circumvented
● Upgrades can be challenging

15. Goodbye Sidecar, Hello eBPF?
https://isovalent.com/blog/post/2021-12-08-ebpf-servicemesh/

16. What can eBPF do?
https://www.solo.io/blog/ebpf-for-service-mesh/
https://www.youtube.com/watch?v=heDVglDRDNw

17. TL;DR, You Still Need a Proxy
Separation
of
L7
and
L4

18. Digging into Cilium and Istio Service Mesh
(sidecar-less, service-mesh implementations)

19. Benefits of a Sidecar-less Service Mesh
● Fully transparent, cannot opt-out
● Optimize networking paths/reduce latency in service calls
● Reduce overall resource allocation (Mem/CPU)
● Eliminate in-Pod container race conditions
● Eliminate pod injection
● Remove security credentials from the app
● Implementations vary, may have more benefits

20. Cilium
● eBPF based L3/L4 data plane
● Container networking (Kubernetes needs a CNI)
● Networking flows/observability
● Kubernetes NetworkPolicy (and more advanced
NetworkPolicy)
● KubeProxy replacement
● Lay the foundation for a sidecar-less service mesh

21. Cilium Service Mesh Functionality
● Ingress (Gateway API)
● Mutual Authentication (beta)
● CiliumNetworkPolicy
● Direct Envoy Configuration

22. Istio
● Stable, mature, multi-cluster L4/L7 service mesh
● Diverse, multi-vendor CNCF community, broad industry
adoption
● Based on Envoy Proxy
● Workload identity based on SPIFFE
● Authentication (mTLS) and Authorization
● Observability, tracing, audit logging
● Recently added support for sidecarless (ambient)
● CNI/Kubernetes independent

23. Istio (Ambient Mode)
● Explicitly separate L4 and L7 into composable pieces
● Supports any CNI (works great on Cilium CNI)
● L7 authorization policy, observability, traffic control
● Standards based mTLS mutual authentication
(FIPS, compliance, etc)
● Gateway API support
● Production ready in next Istio release (v1.22)

24. Sidecar-less Service Mesh Architecture
● Control Plane
● Data Plane
● Mutual Authentication / mTLS
● Observability
● Traffic Control

25. Control Plane Architecture
and API

26. Cilium Control Plane Architecture

27. Cilium Control Plane API
● Gateway API
● CiliumNetworkPolicy
● CiliumEnvoyConfig (caution)
● CiliumClusterwideEnvoyConfig (caution)

28. Istio Control Plane Architecture
https://github.com/cncf/xds

29. Istio Control Plane API
● Gateway API
● VirtualService
● DestinationRule
● AuthorizationPolicy
● PeerAuthentication
● RequestAuthentication
● JWTRule

30. Data Plane Architecture

31. Cilium (L4)

32. Cilium (L7)
Separation of L4 and L7

33. Cilium (L7)

34. Cilium (L7)

35. Istio Ambient Mode (L4)

36. Istio Ambient Mode (L4)
https://istio.io/latest/blog/2024/inpod-traffic-redirection-ambient/

37. Istio Ambient Mode (L4)

38. Istio Ambient Mode (L7)

39. Istio Ambient Mode (L7)
Separation of L4 and L7

40. Mutual Authentication / mTLS

41. Cilium (mutual authentication)

42. Cilium (mutual authentication)

43. Cilium (mutual authentication)

44. Cilium (mutual authentication)

45. Cilium (mutual authentication)

46. Cilium (mutual authentication)
https://thenewstack.io/how-ciliums-mutual-authentication-can-compromise-security/
Could network cache-based identity be mistaken?

47. Istio Ambient Mode mTLS

48. Istio Ambient Mode mTLS
● Uses standard mTLS
● Peer-to-peer tunnelling
● mTLS originates directly from Pod network namespace
● Identity model based on SPIFFE
● Standard x509 / expiry / rotation
● No caching, state, or eventual consistency issues
● Can be combined with Cilium CNI

49. Observability

50. Cilium (Observability)
https://github.com/cilium/hubble

51. Istio Ambient Mode Observability

52. Traffic Control / Ingress

53. Cilium (Ingress / Gateway API)

54. Istio (Ingress / Gateway API)

55. Recap

56. Service Mesh Functionality
Separation
of
L7
and
L4

57. Cilium Service Mesh Architecture Recap
Separation
of
L7
and
L4

58. Cilium Service Mesh Architecture Recap
Separation
of
L7
and
L4

59. Cilium Service Mesh Architecture Recap

60. Istio (Ambient Mode)
Separation
of
L7
and
L4

61. Architecture Recap
Separation
of
L7
and
L4

62. Istio (Ambient Mode)

63. More Service Mesh Talks (Friday!)
● “At the intersection of Cilium CNI and Service Mesh - Who has
the right of way” - Christine Kim (Isovalent) Friday 11:00
● Next level security: mTLS in Istio Multi Cluster with Spire” -
Eduardo Bonilla & Samuel Veloso (Solo.io) Friday 16:00

64. The CAKES Stack
An Open Source Modern Cloud Networking Stack

65. Thank you!
Please reach out
with any questions!
VP, Global Field CTO, Solo.io
@christianposta
christian@solo.io
/in/ceposta

66. Thank you!
Please reach out
with any questions!
VP, Global Field CTO, Solo.io
@christianposta
christian@solo.io
/in/ceposta
Slides Online: