Cloud-Native Application Debugging with Envoy and Service Mesh

Cloud-native Application
Debugging with Envoy and
Service Mesh
Christian Posta
Field CTO – Solo.io

2 | Copyright © 2020
CHRISTIAN POSTA
Global Field CTO, Solo.io
@christianposta
christian@solo.io
https://blog.christianposta.com
https://slideshare.net/ceposta

01
02
03
04
05
06
Challenges of microservices,
debugging
Introduction to our lab
environment
Distributed tracing with a
service mesh
Debugging microservices
Debugging in production with
record and replay
Proactive debugging with
chaos experimentation
Approximate flow of workshop

Moving to microservices?

Microservices and Kubernetes

Move fast, safely
https://puppet.com/resources/whitepaper/state-of-devops-report

SERVICE MESH JOURNEY
INNOVATION
MODERNIZE TO
MICROSERVICES
SERVICE MESH MANAGEMENT
ANY MESH - ANYWHERE
ADAPTIVE SERVICE
MESH

December 11, 2018
2018 TOP WOMEN ENTREPRENEURS IN CLOUD INNOVATION
Seventh Annual Award Honors Women Founders for Outstanding
Accomplishments in Cloud and Emerging Technologies, Sponsored by
Facebook, Intel, and Google.
Award Winning Innovation
Key Industry Collaborations

9 | Copyright © 20209 | Copyright © 2020
The problem

As we move to services architectures,
on cloud-native deployment platforms,
we increase the complexity between
our services.

Cloud application networking challenges
• Service discovery
• Retries
• Timeouts
• Load balancing
• Rate limiting
• Thread bulk heading
• Circuit breaking

• Edge/DMZ routing
• Surgical / fine / per-request routing
• A/B rollout
• Traffic shaping
• Request racing
• Internal releases / dark launches
• Request shadowing
• Fault injection

• Adaptive, zone-aware routing
• Deadlines
• Health checking
• Stats, metric, collection
• Logging
• Distributed tracing
• Security

How do we begin to understand what’s
happening so we can debug?

How we typically like to solve this problem:

Decentralized, language-independent
observability in the network
Foundational technology to help solve these challenges in a
cloud-native application architecture

Envoy is to Application Networking
what Kubernetes is to Container Deployment
http://envoyproxy.io

Envoy implements:
• zone aware, least request load balancing
• circuit breaking
• outlier detection
• retries, retry policies
• timeout (including budgets)
• traffic shadowing
• request racing
• rate limiting
• access logging, statistics collection
• Many other features!

Envoy to do application networking heavy lifting

Why Envoy?
• C++
• Built ground-up for services environment
• Large, diverse, vibrant community
• Dynamic configuration model
• Highly extensible (in C++  we’ll come back to this)
• Many out of the box L7 filters (HTTP, HTTP2, grpc, redis, mysql,
DynamoDB, thrift, zookeeper, kafka, et. al.)
• Incredible trove of telemetry, tracing out of the box
• Very versatile deployment options (as we’ll see)

Versatility of Envoy: Edge proxy

Versatility of Envoy: Middle proxy

Versatility of Envoy: Service proxy

Control plane for managing mesh of service proxies

Service proxy lives with application instance

Service mesh technologies provide the following:
• Service discovery / Load balancing
• Secure service-to-service communication
• Traffic control / shaping / shifting
• Policy / Intention based access control
• Traffic metric collection
• Service resilience
• API / programmable interface

These application-networking technologies provide a
nice API for programming our network

Setting up the lab environment

http://bit.ly/debug-microservices

Consul Service Mesh

Consul Service Mesh
connect = {
proxy = {
config = {
upstreams = [
{
destination_name = "mysql",
local_bind_port = 8001
}
]
}
}
}

Consul Service Mesh

Tracing with a service mesh

36 | Copyright © 2020 @christianposta
DB S3
DEBUGGING IN PRODUCTION
CLUSTER
POD 1 POD 2 POD 3 POD 4

DB S3
P P P P
CLUSTER

Lab: Distributed Tracing

Debugging

THE PROBLEM

THE PROBLEM
A MONOLITHIC APPLICATION
CONSISTS OF A SINGLE
PROCESS
AN ATTACHED DEBUGGER
ALLOWS VIEWING THE
COMPLETE STATE OF THE
APPLICATION DURING RUNTIME
A MICROSERVICES APPLICATION
CONSISTS OF POTENTIALLY
HUNDREDS OF PROCESSES
IS IT POSSIBLE TO GET A
COMPLETE VIEW OF THE STATE
OF A SUCH APPLICATION?!

Demo: multi-language, distributed
debugging with Squash

SQUASH
DEFAULT MODE SECURE MODE

SQUASH DEFAULT MODE
Node
Namespace: ns-a Namespace: squash
s-dlvc1

-> ls -l /proc/self/ns
total 0
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 net -> net:[4026532009]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 pid -> pid:[4026531836]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 pid_for_children -> pid:[4026531836]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 user -> user:[4026531837]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 uts -> uts:[4026531838]
-> inod of mnt namespace (unique identifier to the container namespace)
via CRI api call ExecSyncRequest
Node
Namespace: ns-a
s-dlv
CRI
c1
We need to translate the pid of the process (application that run in the container) to
the host pid namespace to allow debugger to attach.
Namespace: Squash

SQUASH SECURE MODE
Node
Namespace: ns-a Namespace: squash
s-dlvc1
CRD Intent
squash

DOCS: HTTPS://SQUASH.SOLO.IO
GITHUB: HTTPS://GITHUB.COM/SOLO-IO/ SQUASH
COMMUNITY: HTTPS://SLACK.SOLO.IO

Break: 3:00p – 3:30p
When we come back: Debugging microservices lab
NOTE: Make sure to charge your devices!

Lab: Squash

Debugging in production

DB S3
CLUSTER
POD 1 POD 2
> ONLY HEADER WILL BE SENT
> SAMPLING
POD 3 POD 4

DB S3
P P P P
CLUSTER
> ONLY HEADER WILL BE SENT
> SAMPLING

DB S3
P P P P
CLUSTER

++

Getting traffic into your mesh
Workflow-specific APIs for Envoy Proxy

Versatility of Envoy: Edge proxy

Envoy needs a control plane.

API Gateway built on Envoy
https://github.com/solo-io/gloo

Gloo Data Plane and Control Plane
EXTERNAL AUTH
RATE LIMITING
GLOO FILTERS
ROUTER
UPSTREAM
EXTERNAL AUTH SERVER
RATE LIMITING SERVER
CACHING
DATA LOSS
PREVENTION
LAMBDA
NATS.IO
TRANSFORMATION
WEB APPLICATION
FIREWALL (WAF)

API Gateway built on Envoy
ENVIRONMENT
SECRET
CONFIGURATION
Data Plane
Upstream
gRPC-JSON transcoder
Rate limiting
External AUTH
…
Control Plane
Configure and manage
envoy’s plugins
Router

Gloo API Gateway
• Unify backend APIs running in Kubernetes, VMs, Physical, FaaS, etc
• Decentralized configuration: allow service teams to move fast
• Declarative configuration
• Provides a control plane for Envoy
• Security (Oauth/ODIC, API Key, TLS, SNI, OPA, HMAC, custom)
• Kubernetes native / run outside Kube as well
• Highly pluggable/extensible
• “If you know Kubernetes, you know Gloo”  user quote

Lab: Using Loop with Gloo

DOCS: COMING REAL SOON …
GITHUB: COMING REAL SOON …

Demo: Loop with service mesh

Proactive debugging

CHAOS ENGINEERING
THINK OF A VACCINE OR A FLU SHOT
INJECT YOURSELF WITH SOMETHING HARMFUL
IN ORDER TO PREVENT A FUTURE ISSUE.
CAREFULLY INJECTING THIS HARM INTO YOUR SYSTEMS
TO TEST THE SYSTEM’S ABILITY TO RESPOND TO IT.
“BREAK THINGS ON PURPOSE" IN ORDER TO LEARN
HOW TO BUILD MORE RESILIENT SYSTEMS.

PROBLEMS WITH CHAOS ENGINEERING TODAY?
LANGUAGE SPECIFIC CODE MODIFICATION
1 2

NETWORK ABSTRACTION
EAST-WEST
TRAFFIC
NORTH-SOUTH
TRAFFIC
SERVICE
I
SERVICE
II
SERVICE
III
SERVICE
IV
SERVICE
V

CONTROL EXPERIMENT
⍄ DEFINE EXPERIMENTS (SET OF: MESSAGE DELAYS, NETWORK FAULTS)
⍄ RUN EVERY INTERVAL (E.G. EVERY FRIDAY AT 9PM)
⍄ GATHERED METRICS – COMPARE BASELINE
⍄ STOP EXPERIMENT IF CONDITION REACHED

GLOOSHOT
GLOOSHOT ALLOWS YOU TO PERFORM CHAOS EXPERIMENTS AT THE SERVICE MESH
LEVEL.
DEFINE ERROR CONDITIONS IN TERMS OF SUCH FAILURE MODES:
⍄ MESSAGE DELAYS
⍄ NETWORK FAULTS.
RUN EXPERIMENTS UNTIL A STOP CONDITION IS MET.
GLOOSHOT INTERFACES WITH ALL MAJOR SERVICE MESHES THROUGH SERVICE MESH
INTERFACE (SMI).

Demo: Glooshot

DOCS: HTTPS://GLOOSHOT.SOLO.IO
GITHUB: HTTPS://GITHUB.COM/SOLO-IO/GLOOSHOT

What to watch for
Upcoming improvements for which to keep an eye out

Web Assembly shaking up the data plane

https://github.com/envoyproxy/envoy-wasm

https://webassemblyhub.io

THANK YOU FOR COMING OUT!
@christianposta
christian@solo.io
https://blog.christianposta.com
https://slideshare.net/ceposta

• https://solo.io
• https://slack.solo.io
• https://gloo.solo.io
• https://envoyproxy.io
• https://istio.io
• https://webassemblyhub.io
• https://servicemeshhub.io
• https://blog.christianposta.com

Cloud-Native Application Debugging with Envoy and Service Mesh

More Related Content

What's hot

Similar to Cloud-Native Application Debugging with Envoy and Service Mesh

More from Christian Posta

Recently uploaded

Cloud-Native Application Debugging with Envoy and Service Mesh