This document discusses how to make Istio work with applications and provides best practices for long-term success. It introduces service mesh concepts and Istio, discusses when to adopt Istio, describes common issues and solutions when using Istio, and recommends strategies like version controlling Istio configs and providing feedback to the Istio community.

1. How to Make IstioWork withYour App
Karen Bruner

2. 2©2019 StackRox. All rights reserved.
Overview
1. Quick intro to Istio and Service Mesh concepts
2. Deciding to adopt Istio
3. Debugging issues with Istio routing
4. Common problems (and solutions) with Istio
5. Strategies and practices for long-term success

3. 3©2019 StackRox. All rights reserved.
Overview of the Istio Service Mesh
• What is a service mesh?
• Infrastructure layer dedicated to managing service-to-service communications
• Istio
• Background
• Built on Lyft’s Envoy proxy
• Joint project by Google, IBM, and Lyft
• Features and capabilities
• Security: zero-trust model, mTLS, authentication, authorization
• Service routing: load balancing, HTTP request routing, A/B testing, throttling,
deployment canaries, traffic shifting, traffic mirroring, fault injection
• Visibility: detailed connection and request logging, request tracing
• Multi-cluster mesh

4. 4©2019 StackRox. All rights reserved.
Istio Architecture

5. 5©2019 StackRox. All rights reserved.
Do you need Istio?
• Driving factors
• Security
• Microservice visibility
• Intelligent request routing
• Counterarguments
• Steep learning curve
• Operational complexity
• Other considerations
• Istio’s tools and performance keep improving with each release
• Adoption of most individual features and security restrictions can be turned on later

6. 6©2019 StackRox. All rights reserved.
Istio Debugging andTroubleshooting
• Network connection logging
• istio-proxy sidecar container logs
• Log-level can be set globally or on-the-fly per pod
• curl -s -XPOST http://localhost:15000/logging?level=debug
• Mixer telemetry container logs (may change with Istio v1.4+)
• kubectl logs -l app=telemetry -n istio-system -c mixer
• Configuration propagation
• Pilot logs
• kubectl logs -l app=pilot -n istio-system -c discovery
• istioctl
• CLI
• Interface for various Istio Control Plane services
• Growing set of troubleshooting tools

7. 7©2019 StackRox. All rights reserved.
istioctl

8. 8©2019 StackRox. All rights reserved.
Common Application Issues
Use case / Issue Best Solution Workarounds
Proxy start-up
latency
Application should
retry network
connections
Wrap your container command:
...
spec:
template:
spec:
containers:
- name: my-app
args:
- 'while ! wget -q -O -
http://localhost:15000/server_info | grep
''"state": "LIVE"''; do echo Waiting for
proxy; done; echo Istio-proxy ready; exec
/entrypoint'
command:
- /bin/sh
- -c

9. 9©2019 StackRox. All rights reserved.
Use case / Issue Best Solution Workarounds
Kubernetes
liveness/readiness
probes and mTLS
● Use a separate
port on the
application
container for
health checks
● Exec check
● Global (for all Istio-managed services): Pass the --set
values.sidecarInjectorWebhook.rewriteAppHTTP
Probe=true option to helm
● Per deployment: add
sidecar.istio.io/rewriteAppHTTPProbers:
"true" annotation to the pod spec
Mixed-protocol ports Use a different
application container
port for each protocol
type
Add a small proxy to the pod to handle one protocol
Non-TCP protocols Use Kubernetes Network Policies to control service ingress and egress for UDP and
SCTP

10. 10©2019 StackRox. All rights reserved.
Use case / Issue Best Solution Workarounds
Pod Security Policies
or Contexts breaking
istio-init container
Use istio CNI to handle iptables rules outside application pods
Services with their
ownTLS certs
● Use your own CA
in Citadel
● Convert
application to
use Istio auth
Set mTLS mode to PERMISSIVE (could be security issue)
Headless services ● Internal traffic: Add an Istio ServiceEntry resource per
pod in the StatefulSet (cassandra-0, cassandra-1,
etc)
● Traffic from outside mesh: Add Istio VirtualService
entry per pod

11. 11©2019 StackRox. All rights reserved.
Long-term Strategies for Success with Istio
• Keep Istio resource manifests with all other Kubernetes manifests for an application
• Version controlled
• Apply identical configs in pre-production environments as production
• Create internal “library” of Istio resource manifest templates for different application
types
• Leverage the metrics and details Istio collects to show value and drive stack
improvements
• Suggest improvements to Istio
• Open a feature request on GitHub
• Join the Istio Slack channel

12. 12©2019 StackRox. All rights reserved.
Me + More
• More Istio information at https://www.stackrox.com/post/
• Me: Karen Bruner
• Personal tech blog: https://nightmare-before-devops.xyz/
• LinkedIn: https://www.linkedin.com/in/kmbruner
• My cats: