This talk was done in Feb 2020. Sergey and I co-presented at CTO Forum on Microservices and Service Mesh (how they relate, requirements, goals, best practices and how DevOps and Agile has had convergence in the set of features for Service Mesh and gateways around observability, feature flags, etc.)

1. CTO Forum
Service Mesh
Draft 2
Microservice Journey
Service Mesh
Architecture Service Mesh
Service Mesh Concerns
Service Mesh Security
Service Mesh Evolution

2. [
2
Author of best-selling agile development book
Early adopter of Microservices, TDD, DevOps, Agile,
Container Orchestration, 12 factor deployments, KPIs/
metric, health checks, tracing, etc.
Successfully ran development organizations
Developed open source software used by millions
• Java Champion 2018
Early adopter and advocate of microservices
• Worked on Vert.x, QBit, Reakt, Groovy, Boon, etc.
• Speaker on microservices at JavaOne
• Designed/implemented microservices-based
systems that scale to 100M users
Wrote App Gateway for streaming music service
Worked with Service Meshes as early as 2015
Worked with Container Orchestration as early as 2016
Senior Director at fortune 100, managing group using
Kubernetes and implementing stream processing
RICK HIGHTOWER
Sergey Sundukovskiy, Ph.D. has over 20 years of
experience serving in capacities of Chief Technology
Officer, Chief Information Officer and Chief Product
Officer. Sergey specializes in implementation of
subscription based high volume SaaS platforms, with
strong emphasis on early stage product development
and market deployment. Specific areas of expertise
include A/B Testing, Big Data, Video Management,
eCommerce, RTB platforms and Cloud Computing.
Sergey often mentors first-time founders and advises
early stage Startups with emphasis on Product
Development, Product Market Testing, Public Relations,
Product Marketing, Team Building, Customer Success
and Organizational Management

3. Lorem Ipsum Dolor
Service Mesh Intro

4. Lorem Ipsum Dolor
Microservices
Without Service Mesh
Difficulty Is Not In Breaking Down the
Monolith
Easy Problems
Service Granularity
Service Boundaries
Service Communication
Service Contract
Service Roles and Responsibilities

5. Distributed System Problems
❖ Unreliable Networks - Nothing Works As Expected
❖ Lack of High Availability - Everything Eventually Fails
❖ Communication Latency - Everything Slows Down
❖ Limited Bandwidth - It Is Never Enough
❖ Zero Trust Environment - It Is Never Safe
❖ Changing Service Topology - Everybody Gets Lost

6. Microservice Components - Service Config
The interesting part is that each of these microservices can have their own
configuration
Such configurations include details like:
❖ Application configuration.
❖ Database configuration.
❖ Communication Channel Configuration - queues and other
infrastructure.
❖ URLs of other microservices to talk to.
Ex. Git, Vault, File System

7. Microservice Components - Service Discovery
Service discovery involves 3 parties: service provider, service consumer and service
registry.
❖ service provider registers itself with service registry when it enters and
deregister itself when it leaves the system
❖ service consumer gets the location of a provider from registry, and then talks to
the provider
❖ service registry maintains the latest location of providers
Ex. Zooker, Consul, Etcd

8. Microservice Components - Service Routing
Service Routing primary responsibilities for API routing, composition and edge functions
❖ authentication – verifying the identity of the client making the request
❖ authorization – verifying that the client is authorized to perform that particular operation
❖ rate limiting – limiting how many requests per second are allowed from either a specific client
and/or from all clients
❖ caching – cache responses to reduce the number of requests made to the services
❖ metrics collection – collect metrics on API usage for billing analytics purposes
Ex. Zuul, NGINX, Spring Cloud Gateway

9. Microservice Observability
Observability is not monitoring
❖ Health Checking
❖ Metrics
❖ Audit Logging
❖ Distributed Tracing
❖ Exception Logging
❖ Service Logging
Ex. Prometheus, Grafana, Jaeger

10. Microservice Patterns
❖ Circuit Breaker
❖ Rate Limiter
❖ Retry
❖ Bulkhead

11. Microservice Patterns - Circuit Breaker
The circuit breaker concept is straightforward. It wraps a function with a
monitor that tracks failures. The circuit breaker has 3 distinct states, Closed,
Open, and Half-Open:
❖ Closed – When everything is normal, the circuit breaker remains in the
closed state and all calls pass through to the services.
❖ Open – The circuit breaker returns an error for calls without executing the
function.
❖ Half-Open – After a timeout period, the circuit switches to a half-open
state to test if the underlying problem still exists.

12. Microservice Patterns - Rate Limiter
Rate Limiting pattern ensures that a service accepts only a defined
maximum number of requests during a window. This ensures that
underline resources are used as per their limits and don't exhaust.

13. Microservice Patterns - Retry
Retry pattern enables an application to handle transient failures while
calling to external services. It ensures retrying operations on external
resources a set number of times. If it doesn't succeed after all the retry
attempts, it should fail and response should be handled gracefully by the
application.

14. Microservice Patterns - Bulkhead
Bulkhead ensures the failure in one part of the system doesn't cause the
whole system down. It controls the number of concurrent calls a
component can take. This way, the number of resources waiting for the
response from that component is limited. There are two types of bulkhead
implementation:
❖ The semaphore isolation approach limits the number of concurrent
requests to the service. It rejects requests immediately once the limit is
hit.
❖ The thread pool isolation approach uses a thread pool to separate the
service from the caller and contain it to a subset of system resources.

15. CTO Forum
Microservice Journey
Webify SOA
Microservices
CI/CD / Agile
DevOps / SRE
Containers
Container Orchestration

16. How we got here
❖ Web pages that were brochures
❖ eCommerce
❖ Legacy integration
❖ Rush to ‘webify’ businesses
❖ SOA: wrap legacy systems as services to use from the web
❖ Virtualization, Virtualization 2.0, Cloud, Containers, and now
Container orchestration
❖ We want faster feedback and leaner more agile delivery

17. Continuous delivery
❖ The ability to deliver
❖ Build quality in
❖ Work in small batches
❖ Automate repetitive tasks including
❖ testing & deployments
❖ Pursue continuous improvement
❖ Ownership
❖ Comprehensive configuration management
❖ Continuous integration
❖ Continuous testing
You can’t skip steps.
There is investment up
front.
Today’s speed up can
be tomorrows painted
yourself
In a corner.

18. Why DevOps, CI/CD and Microservices?
❖ High performers 2x the rate will exceed organizational performance goals as
low performers:
❖ 2x profitability
❖ 2x productivity
❖ 2x market share
❖ 2x number of customers
❖ High performers twice as likely to exceed non-commercial performance goals as
low performers
❖ 2x better quantity of products and services
❖ 2x operating efficiency
❖ 2x customer satisfaction
❖ 2x quality of products/services
❖ 2x achieving organizational/mission goals
❖ 50% increase in market capitalization compared to low performers!
18

19. DevOps Acceleration
❖ Microservices/
Containers
❖ CI/CD
❖ DevOps
❖ 12 Factory Deploys
❖ Observability

20. Convergence
DevOps
Automation is better
CI/CD
Fast Feedback is better
Lean/Agile
Simpler is better
Microservices
Small is better
12 Factor Deploys
KPIs and Health
Service Mesh
• Observability
• Logging
• Tracing
• KPIs
• Dashboards
• Canary Deployments
• Fractional
• Version Labels
• Supports small CI/CD
with Microservice
• Traffic Management

21. Microservices: INCEPTION and Natural Evolution
❖ Now you can run a Java Virtual Machine in a Docker
image
❖ Which is just a process pretending to be an OS
❖ Which is running in an OS that is running in the cloud
❖ Which is running inside of a virtual machine
❖ Which is running in Linux server that you don’t own
that you share with people whom you don’t know
❖ Servers are not giant refrigerator boxes that you order
from Sun and wait three months for (circa 2000)..… Goal
was to run a lot of things on same server
❖ Did you develop code in the 90s with punch cards?
❖ Microservices recognize trend
21

22. [
22
‣ Philosophy behind microservices mirrors Unix
‣ Unix’s inventor, Ken Thompson, defined its philosophy:
• One tool, one job.
‣ Emphasizes building short, simple, clear, modular, and extendable code
• Easily maintained and repurposed by other developers
MICROSERVICES: UNIX PHILOSOPHY
What is microservice arc

23. Microservices
❖ Focus is building small, reusable, scalable services
❖ Adopt the Unix single-purpose utility approach to service development
❖ Small and malleable so they can be released more often
❖ Easier to write
❖ Easier to change
❖ Go hand in hand with continuous integration and continuous delivery
❖ Heavily REST-based and message oriented
❖ Focus on business capability
❖ Refocus on object oriented programming roots
❖ Organize code around business domains.
❖ Data and business rules colocated in the same process or set of processes.
What is microservice architecture?

24. Microservices: Key ingredients
❖ Independently deployable, small, domain-driven services
❖ Own their data (no shared databases)
❖ Communication through a well-defined wire protocol
usually JSON over HTTP (curl-able interfaces)
❖ Well defined interfaces and minimal functionality
❖ Avoiding cascading failures and synchronous calls -
reactive design for failure
❖ Shortly after MicroServices: Containers came out

25. Microservices and Containers
Microservices
Containers

26. MicroServices: Achieving Resilience
❖ Avoid synchronous calls to avoid cascading failures
❖ Circuit breaker frameworks, retries, resiliency, network layer libs
❖ Instead embrace:
❖ Streams, queues,
❖ Actor systems
❖ Event loops
❖ Other async calls.
❖ Spend more time with distributed logging/log aggregation w/MDC
❖ Distributed tracing: A calls B who calls D or E or F who calls X or Y or Z
26

27. MicroServices: Monitoring and KPIs
❖ Customer/User experience KPIs
❖ Debugging (requests per second, # threads, #
connections, failed auth, expired tokens, etc.)
❖ Circuit breaker (monitor health, restarts, act/react based
on KPIs)
❖ Cloud orchestration (monitor load, spin up instances)
❖ Health checks and observable KPIs
27

28. MicroServices: Continuous Deployment
❖ Microservices are continuously deployable services
❖ Focus of microservices is on breaking applications into small (micro),
reusable services that might be useful to other services or other
applications.
❖ ‘micro’ part of microservices comes to denote small
❖ Services can be deployed independently.
❖ Can be tweaked and then redeployed independently.
❖ Microservice vs monolith when deploying
What is microservice a

29. [
29
KUBERNETES

30. –Rick Hightower
“Service Mesh like Istio does the things that the
very best InfoSec, Dev teams, SREs and DevOps
teams would do: mTLS zero trust networking,
automate observability and dashboard creation,
automate tracing, and automate logging
aggregation while enabling continuous
deployment via traffic management and canary
deployments. It takes what we’ve learned in the
DevSecOps community and makes it the default,
out of the box.”

31. –Rick Hightower (Why you might need a Service Mesh like Istio?)
“To maximize shareholder value, companies are
embracing CI/CD and Microservices architecture.
This allows product teams to deliver faster, get
feedback more often and evolve quickly.
This Digital Transformation strategy allows
companies to address nimble upstarts as well as
provide our customers with an intelligent, rich
experience.”

32. CTO Forum
What is Service
Mesh?
Observability and Telemetry
Service discovery
Traffic management
Security
Supports CI/CD and Microservices

34. What is a Service Mesh?
❖ Service mesh is a network of microservices and
interactions between microservices
❖ Service mesh tools scale to help manage size and
complexity of large Service Meshes
❖ Modern service mesh aids understanding and
managing
❖ Helps organizations migrate from monolithic
applications to microservice architecture

35. –Rick Hightower (Why you might need a Service Mesh like Istio?)
“Using a Service Mesh facilitates CI/CD and
Microservices architecture. Service Mesh
automates best practices for DevSecOps needs like
failover, scale-out, scalability, 0 trust networking,
health checks, circuit breakers, rate limiters, KPI
collection, dashboard creation, observability,
avoiding cascading failure, disaster recovery, and
traffic routing”

36. Decorate Network Data Layer
❖ Service Mesh decorates network layer to implement
cross-cutting concerns which are usually NFRs
❖ Service Mesh is to MicroServices as AOP is to DDD
and OOP
❖ Service Mesh is to MicroServices as Servlet Filters
are to Servlets.

37. Service Mesh Features
❖ Networking: Discovery, load balancing, failure recovery (circuit
breaking), rate limiting, etc.
❖ Observability: time series KPIs, log aggregation, alerting and
monitoring, USE and RED Dashboards
❖ CI/CD and frequent releases: canary rollouts, green/blue deploys,
new version rollouts, traffic management
❖ And to gradually release a Microservice and select which
downstream and upstream Microservice that can talk
❖ Security access control, end-to-end authentication (RBAC), service
identity, 0 trust networking - mTLS, etc.

38. Simplifies hard programming
❖ Service Mesh performs many low-level L3/L4 networking tasks
❖ Previously left up to application developers to implement or to
many libs for many platforms/languages
❖ Low level network code is hard to write and maintain
❖ filled with edge cases.
❖ Service Mesh completely abstracted out from the microservices
business logic
❖ Provides level of consistency provides additional operational
predictability for polyglot programming environments

40. Top 3

41. Service Meshes At a glance
❖ Istio
❖ Backed by IBM, Red Hat, Google, and Lyft
❖ Uses Envoy
❖ Supports more than Kubernetes
❖ Linkerd
❖ CNCF
❖ V1: Finagle, Scala, Twitter stack
❖ V2: Conduit merged: Now Rust and Go Lang based
❖ Consul
❖ Hashicorp
❖ Uses Envoy
❖ Supports more than Kubernetes
❖ Nice comparison of Consul, Linkerd and Istio

42. Observability and Telemetry
❖ automate many aspects of observability
❖ log aggregation, telemetry of services, collecting KPIs
and generating
❖ Automates creating USE and RED Dashboards
❖ See service performance trends and dashboards
❖ how long did a service request take?
❖ how often is the service being called?

43. Service Discovery
❖ Service inventory and understand how services
communicate—tracing call graph, amount of calls per span,
etc.
❖ essential for microservices architecture
❖ Allows services to find other dependent services
❖ Helps keep track of services running in infra
❖ essential for microservices architecture
❖ Manage and visualize services and its dependencies
❖ essential for microservices architecture

44. Traffic Management
❖ Segment features through feature flags and limit
consumption of new services with clients that can
handle changes to APIs or wire protocols with gradual
rollouts
❖ Gradual and continuous release instead of a big bang
rollout
❖ Fine grain deployments
❖ Essential for microservices architecture and CI/CD

45. Traffic Mgmt Interoperability
❖ Big Kubernetes issue with cloud interoperability has been ingress and egress
❖ Service Mesh makes great strides to solve interoperability
❖ Standardize ingress/egress and many other networking concerns so routing
rules, RBAC and TLS termination don’t vary with each vendor or cloud provider
❖ Interoperability suffers w/ Kubernetes federation and hybrid clouds
❖ Service Mesh, and Git Ops (Flux, Argo CD, Anthos Config Manager)
❖ Keep copy of Kubernetes objects between clusters
❖ Using Service Meshes to span clouds and clusters
❖ Now possible to create service meshes that span clusters and clouds
❖ standard service registry plugins (consul/kubernetes), Istio gateways, ad
hoc services and networks defined with CIDR addresses.

46. –Rick Hightower (Why you might need a Service Mesh like Istio?)
“Service Mesh aids in avoiding data breaches as
well as limiting their blast radius. Data breaches
can have dire business value consequences.”

47. Security
❖ Identity, Security, RBAC, 0 trust networking
❖ Secure service-to-service communications via 0 trust networking
❖ Key is service identity
❖ Service identity enables automatic mTLS (mutual TLS) for service-to-service communications
❖ Microservices enhanced to automatically communicate securely via mTLS without code
change
❖ Plugin an existing CA certificate
❖ Enforce service-level authentication using either TLS SNI or JSON Web Tokens (JWS) or
headers or networking origination
❖ Enables fine-grained traffic governance
❖ Allows configure role-based access control (RBAC) for each service and limit which other
services have access to key services
❖ Can be configured to block access based on headers or specific URLs or sub-URIs and paths

48. –Rick Hightower (Why you might need a Service Mesh like Istio?)
“(A Service Mesh’s) ability to automate and maintaining
zero trust networks is its most important feature. In the
age of high-profile data breaches, security is paramount.
…avoid major brand issues … (that can) shrink market
capitalization in an instant. (Service Mesh) helps prevent
a breach and limits the blast radius …”

49. Traffic Management Features
❖ Rate limits based on identity or headers or policies
❖ Fail-over rules (via circuit breakers)
❖ Fine-grained traffic management policies and the application code
never changes
❖ Extend policies to connected service meshes
❖ Route rules can be based on locality of the service
❖ prefer local data center,
❖ or local proximity networks over remotes.
❖ Failover rules are location-aware
❖ Routing can take into account the health of services (active and passive)

50. CTO Forum
Microservice
Example
Example

52. Book info App with No Service Mesh

53. Book info App with Service Mesh