Real world Istio - War stories and Gotchas Speakers: Lasse Højgaard & Frederik Mogensen - Trifork In a public health care project requirements for security and data protection are very rigid - can Istio help us solve these requirements? And at what cost? This talk is about using Istio for running a multi-tenant setup, allowing the different tenants to talk to the core services on the platform, but not disrupting each other. Enforcing security between services using the service mesh, by demanding JWT tokens on all requests, adding mutual encryption, locking down egress traffic, disallowing inter-namespace traffic, etc. We share our experiences running Istio, war stories, bugs found, and failures.

1. Real World Istio
War Stories and Gotchas
Frederik Mogensen & Lasse Højgaard

2. ▪ The Project
▪ Istio 101
▪ JWT Policy checking
▪ Mutual TLS
▪ Secure Egress
▪ Gotchas
Agenda

3. The Project

4. FUT - Telemedicine
Platform
▪ Helping chronically ill patients to live at
home
▪ Defining different questionnaires for
each illness
▪ Patients measuring and responding to
questionnaires daily
Photo by Jair Lázaro on Unsplash

5. Platform Focus
▪ Handling healthcare data demands a
high focus on
▪ Stability
▪ Observability
▪ Security
Photo by Philipp Katzenberger on Unsplash

6. Multi Tenant

7. Istio 101
In three minutes or less

8. Service mesh
“A service mesh is a configurable infrastructure layer for
a microservices application. It makes communication
between service instances flexible, reliable, and fast.
The mesh provides service discovery, load balancing,
encryption, authentication and authorization, support
for the circuit breaker pattern, and other capabilities.”
https://www.nginx.com/learn/service-mesh/

9. Istio 101

10. Istio 101
▪ Connect
▪ Ingress, Intelligent Routing, Load-balancing
▪ Secure
▪ AuthN/Z, mTLS, Namespace segregation, Egress
▪ Control
▪ Rate-limiting, White/Black Listing
▪ Observe
▪ Metrics, Telemetry, Access Logs

11. Grafana

12. Jaeger

13. Kiali

14. Istio 101 - Components

15. Use Cases

16. Enforcing JWT validation
on all request
Photo by Blake Guidry on Unsplash

17. JWT validation
https://blog.christianposta.com/how-a-service-mesh-can-help-with-microservices-security/

18. JWT validation - Gotcha
Kubelet
https://blog.christianposta.com/how-a-service-mesh-can-help-with-microservices-security/

19. Mutual encryption
Photo by Markus Spiske on Unsplash

20. Mutual encryption
https://www.infoq.com/articles/istio-security-mtls-jwt/
Citadel

21. mTLS - Gotcha
▪ Health Checks..?
▪ Kubelet does not know about mTLS, and is
therefore no longer allowed to talk to the
apps
Kubelet

22. JWT & mTLS
● JWT checking
● Permissive mTLS

23. Gotcha!
▪ Symptoms
▪ Non-deterministic behaviour for jobs
▪ The Clue
▪ Why are we getting 401’s for excluded endpoints?

24. Gotcha - Policy selected at random
▪ We thought: “these are separate policies, we can deploy them side-by-side”
▪ RTFM!
▪
▪ Solution: Merge the two policies into one

25. Gotcha - Policy selected at random
https://flickr.com/photos/31418530@N02/3635981474

26. Secure Egress
Gnurx @ /r/TechSupportGore

27. Secure Egress
Used to be enabled by default, changed in Istio 1.2

28. Secure Egress

29. Secure Egress - Gotcha!

30. Other Random Gotchas

31. Jobs with sidecars
Photo by David Tostado on Unsplash

32. Gotcha - Jobs with sidecars
▪ Kubernetes enhancement issue #753

33. Gotcha - Jobs with sidecars

34. Gotcha - Jobs with sidecars
▪ Kubernetes enhancement issue #753
▪ Need sidecar ready before main container runs
▪ When the main container completes sidecar continues running
▪ Jobs never complete
▪ PR almost made it into Kubernetes 1.16
▪ Hopefully in 1.17
▪ Workarounds
▪ Wrap in shell-script or https://github.com/monzo/envoy-preflight
▪ Istio 1.3:
▪ Manually POST to http://localhost:15020/quitquitquit

35. Gotcha - Jobs with sidecars - “fix”
▪ Resulting in no mTLS between jobs and deployments

36. Frequent upgrades
Photo by Djim Loic on Unsplash

37. Gotcha - Frequent upgrades
▪ Bleeding edge software 🔪
▪ Sometimes you find errors 🤷
▪ Things quickly gets fixed 👏
🖼 Czar @ WikiMedia Commons

38. Release cadence
▪ LTS release every quarter
▪ Supported 3 months after next LTS

39. Bug: 1.0.6 -> 1.1.6 upgrade

40. Other possibilities

41. The Istio toolbox
▪ Authorization
▪ Retry-policies
▪ Circuit breakers
▪ Fault injection
▪ Rate limiting
▪ Traffic mirroring
▪ Distributed tracing
▪ Metric collection
▪ Service graph
▪ Locality based load balancing
▪ ...
▪
Photo by Barn Images on Unsplash

42. Wrap-up
▪ Extremely powerful tool.
▪ Complex configuration
▪ RTFM

43. Thank You
Lasse Højgaard
lho@trifork.com
Frederik Mogensen
fmo@trifork.com