This document provides an overview of observability in distributed computing systems using Kubernetes and Istio. It begins with definitions of key concepts like monitoring, metrics, and observability. It then discusses the differences between traditional "black box" monitoring approaches and modern "white box" approaches enabled by service meshes. The document demonstrates installing Istio on a Kubernetes cluster and deploying the Bookinfo sample application. It concludes with references for further reading.

1. OBSERVABILITY
DISTRIBUTED
COMPUTING
SERVICEMESH
KUBERNETES
ISTIO
ENVOY
EBRU CUCEN @ebrucucen#pwshsummit19

2. WHO AM I?
▸ Possibly the only Turkish, vegan
living in London, ex-.Net
developer, working in DevOps,
at Contino, proud-mum of 6-yo,
and prod-wife
▸ Loves exploring, learning,
sharing, inevitably communities!
▸ Ex-co-orgasinator of London
PowerShell User Group
@ebrucucen#pwshsummit19 02/05/2019

3. AGENDA
▸What? Definitions
▸When? Differences
▸Why? History - Demo I
▸How? Theory
▸How? Practice - Demo II
@ebrucucen#pwshsummit19 02/05/2019

4. MONITORING
▸ Collecting, processing, aggregating, and
displaying real-time quantitative data about a
system, such as query counts and types, error
counts and types, processing times, and
server lifetimes.[1]
@ebrucucen#pwshsummit19 02/05/2019
[1] Monitoring Distributed Systems, by Betsy Beyer, Rob Ewaschuk

5. TRADITIONAL SYSTEMS
Monitoring (Black box)
▸ Structured Log
▸ Well-defined Metrics
▸ Tracing
@ebrucucen#pwshsummit19 02/05/2019

6. TEXT
Audit Logs
Azure Tenant
Metrics
Application
Azure Subscription
Service Health
Activity Logs
Azure Resources
Diagnostic Logs
Monitoring Solutions
Guest OS
Application Insights
Dependency Agent
Log Analytics Agent
Diagnostics Extension
Azure
Custom API
Data Collector API
Non-Azure
Service Configuration
NOT ENOUGH!

7. TRADITIONAL SYSTEMS
Monitoring (Black box)
▸ Structured Log
▸ Well-defined Metrics
▸ Tracing
@ebrucucen#pwshsummit19 02/05/2019

8. TRADITIONAL SYSTEMS
Monitoring (Black box)
▸ Structured Log
▸ Well-defined Metrics
▸ Tracing
@ebrucucen#pwshsummit19 02/05/2019
Not Scalable
Up!=Working
May not be
complete

9. TEXT
▸ Latency
▸ Traffic
▸ Errors
▸ Saturation
FULLY CONNECTED NETWORKIMPROVED METRICS
https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/

10. Nines don’t matter if users aren’t happy
Charity Majors
@ebrucucen#pwshsummit19 02/05/2019

11. MODERN ERA
Black Box Monitor
▸ Checks
▸ Partial
▸ Passive/reactive
▸ No root cause
@ebrucucen#pwshsummit19 02/05/2019

12. MODERN ERA
White Box Monitor
▸ Discovery
▸ Whole
Black Box Monitor
▸ Checks
▸ Partial
▸ Passive/reactive
▸ No root cause
@ebrucucen#pwshsummit19 02/05/2019

13. TAXONOMY
Observability ?= Monitorability
@ebrucucen#pwshsummit19 02/05/2019

14. TAXONOMY
Observability ?= Monitorability
@ebrucucen#pwshsummit19 02/05/2019

15. OBSERVABILITY
▸ In control theory, observability is a
measure of how well internal states of a
system can be inferred from knowledge
of its external outputs[2]
@ebrucucen#pwshsummit19 02/05/2019
[2] Wikipedia, 1960, Rudolf Kalman

16. The goal of an observability team is not to
collect logs, metrics or traces. It is to build
a culture of engineering based on facts
and feedback, and then spread that
culture within the broader organization.
@ebrucucen#pwshsummit19 02/05/2019
Brian Knox (DigitalOcean)
OBSERVABILITY

17. 8 FALLACIES OF DISTRIBUTED COMPUTING[3]
1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous
@ebrucucen#pwshsummit19 02/05/2019
[3] L. Peter Deutsch,1994, James Gosling 1997

18. 8 FALLACIES OF DISTRIBUTED COMPUTING[3]
1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous
@ebrucucen#pwshsummit19 02/05/2019
[3] L. Peter Deutsch,1994, James Gosling 1997
‣ Manual restart
‣ Dropped packet
‣ Bottlenecks
‣ SSL/TLS?
‣ Cattle
‣ Conflicting rules
‣ I/O CPU
‣ Not anymore

19. HISTORY
VIRTUALISATIONTEAM ORCHESTRATION
2025 2026 2027 2028 20292020 2021 2022 2023 2024
Agile
Manifesto
DevOps
Microservices
Thoughtworks
Microservices
Defined
XP
AWS
EC2
NETFLIX
to AWS
Docker
1999
Kubernetes
2001
2006
2009
2010 2012 2013
2014
2014

20. DEMO
PS: INVOLVES BALLOONS, MAYBE SOME EXPLOSIONS…

21. TEXT
▸ Namespace
▸ Pod
▸ Container
▸ Sidecar
CRYSTAL CLEAR?

22. NETFLIX - OSS- JAVAMICROSERVICES BEFORE SERVICE MESH
CONTAINER
RUNTIME C
SERVICE C
Load-balancer
Discovery
Resiliency
Metrics
Tracing
CONTAINER
RUNTIME A
SERVICE A
Load-balancer
Discovery
Resiliency
Metrics
Tracing
CONTAINER
RUNTIME B
SERVICE B
Load-balancer
Discovery
Resiliency
Metrics
Tracing

23. TEXT FULLY CONNECTED NETWORK
SERVICE
C
SERVICE
B
SERVICE
A
SERVICE
D
SERVICE
E
SERVICE
F

24. TEXT
Next logical step after a container orchestration deployment
- insight (observability), uniformly and ubiquitously
- connection
- control
- observability
- security
SERVICE MESH

25. The Enterprise Path to Service Mesh Architectures,Lee Calcote
NETWORK PLANES

26. POD A POD B
TEXT SIDECAR PATTERN
INGRESS EGRESS
SERVICE A SERVICE B
PROXY
SIDECAR
PROXY
SIDECAR
CONTROL PLANE
CONTAINER ORCHESTRATION

27. ISTIO NAMESPACE
TEXT ISTIO
CITADEL PILOT MIXER
SERVICE
FOO
BAR POD
SIDECAR PROXY
FOO CONTAINER
SERVICE
FOO
FOO POD
SIDECAR PROXY
FOO CONTAINER
Discovery & config
tls certs Telemetry Reports Policy Checks

28. TEXT DEMO

29. TEXT GOAL: BOOKINFO APP

30. TEXT BACKUP PLAN
screenshots

31. TEXT
▸ Install helm client
▸ Install helm Tiller server (RBAC)
$ brew install kubernetes-helm
$ kubectl apply -f install/kubernetes/helm/helm-service-account.yaml
$ helm init --service-account tiller
$ kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
coredns-79c89b8f4-2x2bv 1/1 Running 0 18h
coredns-79c89b8f4-jfnzm 1/1 Running 0 37h
coredns-autoscaler-6fcdb7d64-d7f2z 1/1 Running 0 37h
heapster-7677c744b8-s725f 2/2 Running 0 37h
kube-proxy-6gtrf 1/1 Running 0 37h
kube-svc-redirect-2w8tx 2/2 Running 0 37h
kubernetes-dashboard-6dffbcc8b9-hljr7 1/1 Running 2 37h
metrics-server-7b97f9cd9-p9wx8 1/1 Running 0 37h
omsagent-9vr56 1/1 Running 0 37h
omsagent-rs-69ccb9cc4c-n59qp 1/1 Running 1 37h
tiller-deploy-7b65c7bff9-kndx7 1/1 Running 0 18h
tunnelfront-c665446bf-d29vn 1/1 Running 0 37h
TILLER SERVICE ACCOUNT

32. az aks browse --resource-group demo-rg --name democluster

33. TEXT

34. TEXT
▸ Install helm charts istio-init and istio for istio-system
namespace
$ helm install install/kubernetes/helm/istio --name istio --namespace istio-system
--set global.controlPlaneSecurityEnabled=true
--set mixer.adapters.useAdapterCRDs=false
--set grafana.enabled=true --set grafana.security.enabled=true
--set tracing.enabled=true
--set kiali.enabled=true
$ helm install install/kubernetes/helm/istio-init --name istio-init --namespace istio-system
ISTIO-SYSTEM NAMESPACE
$ kubectl get jobs -n istio-system
NAME COMPLETIONS DURATION AGE
istio-init-crd-10 1/1 16s 33h
istio-init-crd-11 1/1 15s 33h

35. TEXT ISTIO-SYSTEM

36. TEXT ISTIO-SYSTEM

37. TEXT ISTIO-SYSTEM

38. TEXT

39. TEXT
8 Fallacies
https://dzone.com/articles/understanding-the-8-fallacies-of-distributed-syste
Kubectl
https://kubernetes.io/docs/reference/kubectl/cheatsheet/#deleting-resources
The Enterprise Path to Service Mesh Architectures,Lee Calcote
https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/
REFERENCES