The service mesh is becoming the most critical component of the cloud-native stack, with users ranging from small startups to Internet giants and traditional enterprises. While still early in terms of adoption, this new infrastructure layer has massive implications for the way companies build and operate distributed systems. In this session, SignalFx provides an overview of distributed systems and service instrumentation, then examines how a service mesh approach addresses inter-service communication, testing, and other fundamental challenges of adopting microservices architecture. Learn about the considerations for monitoring and observability, as well as the trade-offs, of implementing service mesh. This session is brought to you by AWS partner, SignalFx.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fully Realizing the Microservices Vision
with Service Mesh
Arijit Mukherji
CTO
SignalFx
D E V 3 1 2 - S

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
In a beautiful dream future . . .
• Developers focus on core functionality
• No more reliance on tribal knowledge and out-of-date wiki pages
• Summer intern can run chaos experiments

4. Service mess!

5. Cloud
Containers
OSS
Microservices and cloud environments are complex

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
About me
• Monitoring and observability for 11+ years
• Architected, developed, managed
• Been a customer of monitoring services
• Currently CTO @ SignalFx
• Ring-side view to state of monitoring in enterprises
• Monitoring @ Facebook 2007-2013
• Original developer/member of FB’s ODS – Their metrics platform
11+

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
How did we get here?
Realizing the potential of service mesh
SignalFx and service mesh
Takeaways
Service mesh risks and mitigations

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• An infrastructure layer for service-to-service communication
• Makes communication visible, manageable, controlled
What is a service mesh?

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s say μ1 needs to communicate with μ2
μ1 μ2

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
They could talk directly by name/IP
μ1 μ2

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service discovery typically used to locate services
μ1 μ2

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh adds an L7 proxy with each μ instance
μ1 μ2Proxy 1 Proxy 2

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service discovery handled in the proxy layer
μ1 μ2Proxy 1 Proxy 2

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh data plane
μ1 μ2Proxy 1 Proxy 2
Data plane

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Finally, put a ring on it – Policy layer to control proxies
μ1 μ2Proxy 1 Proxy 2

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh control plane
μ1 μ2Proxy 1 Proxy 2
Control plane

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Another way to view this

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Policy engine = Target for expressing our intent

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common use cases
• What are some common uses of service mesh?

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common use cases
• Automatic retry on errors by protocol-aware proxy
• Circuit breaker – Stop sending calls to impaired/failed microservice instance

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common use cases
• Proxy acts as L7 load balancer

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common use cases
• Application-aware proxy + service discovery allows sophisticated routing
• For example, route requests based on customer, software version, and so on

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common use cases
• Authentication – Who can speak and to whom?
• Encryption – Transparently encrypt inter-microservice communication

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh implementations
Usually coupled with Envoy proxy

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Standardize on a simple architectural foundation
• Solves the problems associated with code libraries
• Binary proxy can be developed, deployed, and operated independently
• APIs instead of libraries – Exact same functionality for callers of all languages
• Provides a target for implementing policy-driven behavior

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Proxy as a microservice and service discovery

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto-generate and deploy policy-driven configs
• User configures intent centrally in policy engine
• Policy engine generates and pushes relevant configs to all proxies

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Policy engine pushes configuration

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Audit and enforce user intent continually
• Dynamically re-configure proxies with feedback based on operational state
• Feedback provided by monitoring/observability system

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Adapt configuration based on feedback

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.Feedback is fantastic!

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The full picture – Feedback-driven closed loop

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Feedback-driven dynamic policy is a key enabler
• Feedback will come from monitoring systems
• Service mesh will control far more than inter-service networking
• It will consolidate multiple different use cases under one umbrella
• Let us discuss a few of them
A. Code deployments B. Run-time behavior optimization
C. Testing D. Monitoring

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fully automated deploymentsA

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workflow for automated blue/green deployment
In future, this workflow can be fully automated end to end
Container orchestrators, CI/CD tools, monitoring systems will work in concert
A

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Runtime behavior optimization
• If using standard protocols (such as HTTP), proxy can handle errors/retries
• Simplifies microservice codebase
• Makes error handling behavior centrally configurable
B

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Runtime behavior optimization
• Proxies maintain statistics on errors and latencies
• This telemetry is used by policy engine to determine healthy/unhealthy instances
• Impaired/failed instances automatically stop receiving new requests
B

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Runtime behavior optimization
• Prefer co-located targets while optimizing for network cost
• Prefer faster instances while optimizing for latency
• Changing behavior on the fly through configuration will be transformational
B

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Testing – Chaos engineering
• Simulate failures or specific error conditions
• Artificially simulate high-latency conditions and other perf issues
• Simulate issues randomly or by host, geography, customer, and so on
C

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Run manual or automated tests and workflows without jeopardizing service health
• Feedback from monitoring system will inform tests and automation
• For example, determine if test passed/failed; stop experiment if service is
affected
Testing – Chaos engineeringC

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh as a platform

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh will evolve into a platform
• Standardization will lead to faster innovation
• Common platform => Faster development of new features
• Consolidation will lead to wider adoption of higher-order features
• Sophisticated functionality for all
• Unified platform => Fewer things to learn
• Higher quality services, more productive engineers

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
D

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh will have a huge impact on monitoring
Metrics
Logs
APM
Most actionable – Visualize, alert, single-pane-of-glass view
Root cause analysis (RCA), forensics, security
Traditional APM giving way to distributed tracing in modern environments

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh’s impact on monitoring
• Fix spotty adoption through auto-instrumentation of all communication
• Unified vendor-agnostic target for all telemetry (metrics, logs, traces)

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Fix uneven quality by standardized collection of all RED metrics, traces, and so on
• RED = Rates (calls/sec), Errors, Durations (call latencies)
Service mesh’s impact on monitoring

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh will consolidate monitoring
ProxyContainer
Application
Host agent

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Standardized, high-quality telemetry will support feedback-based automation
Service mesh’s impact on monitoring

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Modern app & infra monitoring – What does it take?

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Infrastructure monitoring requirements
• Support for diverse environments
• Today’s environments use 10s or 100s of technologies
• Auto-configuration of data collection
• Elastic/ephemeral environments require quick discovery
• High resolution
• Ephemerality, SLA requirements => Measure with fine grain

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application monitoring requirements
• Rich metadata and dimensionality
• Track by service, customer, version, and so on
• Consistent collection of key telemetry
• RED metrics (Rates/Errors/Durations)
• Transaction flows between microservices
• Infrastructure correlation
• Drilldown from application => Container => Host/instance

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring platform requirements
• Scalability
• Handle high-res data from ‘000s of reporters
• Handle metadata churn of ephemeral environments
• Mutable metadata and flexible querying
• Reflect operational state of environment
• Slice/dice/filter/aggregate across any metadata
• Dependency-aware directed triage
• Use service dependencies, data science to help reduce MTTR

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analytics requirements
• High-cardinality analytics across full dataset
• Data silos prevent monitoring of KPIs across services
• Timely and interactive
• Auto-remediation requires real-time feedback to keep SLAs
• Operators need quick analytics during triage process
• Programmability and data science capabilities
• Integrate with infrastructure-as-code; for example, Terraform
• Predictive alerts to catch problems before an outage

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
About SignalFx
• Monitoring and operational intelligence for the cloud
• Only solution built on real-time streaming analytics architecture

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• How best to support/monitor a cloud-native microservices world?
• SignalFx works everywhere but is optimized for modern environments
• Belief in open standards, easy instrumentation, and integrated toolsets
and service mesh

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Member of CNCF
• Integrating with service-mesh ecosystem for monitoring
and service mesh

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
supports telemetry from service mesh

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Distributed tracing is now an integral part of the product
Mesh will lead to wider adoption of distributed tracing, and it will be combined with metrics
and service mesh

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Microservices APM
Granular, Real-
time analytics
Open
instrumentation
• Complete flexibility
• No vendor lock-in
• No production overhead
• Observes every transaction
• Focus on what matter most
• Single pane-of-glass
• Correlated infrastructure and trace visibility
• Machine learning based prescriptive RCA
• Follow unique customer
• Historical trends at the span level
• Path-level insights
• Real-time – In seconds versus minutes
NoSample
architecture
Faster RCA, lower MTTR

65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
infrastructure monitoring
• Support for diverse environments
• 150+ integrations, APIs, libraries
• Auto-configuring data collection
• Smart Agent does auto-discovery, can be a k8s sidecar
• High resolution
• One-second resolution

66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
application monitoring
• Rich metadata and dimensionality
• True multi-dimensional data model. Automatic tag syncing
• Consistent collection of key telemetry
• Integrated with service mesh and open standards
• Infrastructure correlation
• Rich metadata enables correlating apps with infra

67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
monitoring platform
• Large scale
• Horizontally scalable, many large enterprise customers
• Separate metadata DB scales independently
• Mutable metadata and flexible querying
• APIs to add color to dataset – Properties, tags
• Dependency-aware triage for low MTTR
• Transaction-flow-based dependency maps and navigation
• Cross links to other tools

68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SignalFlow analytics
• High-cardinality analytics across full dataset
• Core strength – for example, KPIs, per customer analytics &
alerting
• Timely and interactive
• The only streaming monitoring platform from ground up
• 2-3 second from data arrival => Alert or chart
• Programmability and data science capabilities
• Powerful analytics platform with data science built-in
• Fully programmable through APIs

69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Power of feedback-driven automation
Lower MTTR with integrated metrics and traces

70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service mesh considerations
• Easy configuration => Easy to make big mistakes
• More systematic configuration testing and management will need to be adopted
• Large blast radius of problems involving proxy itself
• Gradual feedback-driven deployment workflows will reduce risk
• Security
• Will improve over time as more engineers get involved and solutions get hardened
• Higher API latency, more failure modes
• Usually not a big deal in the bigger scheme of things. Benefits far outweigh costs

72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
• Addresses real use cases and challenges

74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
• Policy-driven operations
• Simplified development
• Consolidation of multiple functions under one umbrella

75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
• New software projects/companies should seriously consider it
• Nascent field – Plan ahead for yet-to-be-developed functionality

76. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
• What can service mesh do for you?

77. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Booth 1613
@Venetian

78. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Arijit Mukherji
arijit@signalfx.com
Booth 1613
@Venetian

79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Support for diverse environments

81. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Flexible metadata and querying

82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Dependency maps from distributed traces

83. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Powerful analytics

84. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Integrated with external systems