This document discusses observability and building an observability platform. It begins with defining observability and distinguishing it from monitoring. It describes the need for observability to understand system and user behavior based on inputs and outputs. The document outlines challenges with existing observability tools and proposes rethinking observability. It presents considerations for building a platform that enables exploration and quick root cause analysis. Finally, it discusses enabling observability at scale through approaches like messaging systems and querying tools.

1. GLOBAL SOFTWARE CONSULTANCY
- By Balvinder Khurana & Sarang Shinde
1
© 2020 ThoughtWorks

2. Observability
In Real time at Scale
GLOBAL SOFTWARE CONSULTANCY
- By Balvinder Khurana & Sarang Shinde
2
© 2020 ThoughtWorks

3. 3
1. What is Observability
2. How it is different from Monitoring
3. Why do we need Observability
4. Why out of the box tools are not solving all the problems
5. Our approach to Observability
6. What is real time Observability
7. How to handle Scale
8. Design principles and Gotchas
9. Questions
© 2020 ThoughtWorks
Agenda

4. Discovering the information about System and User Behaviour
by looking at its inputs and outputs
that leads to Customer and Business Impact
4
© 2020 ThoughtWorks

5. The KNOWN Zone
The WEIRD Zone
Monitoring
Observability
Anticipated
Unanticipated
Anticipated,
Unexpected
5
© 2020 ThoughtWorks
Monitoring vs Observability

6. Discovering the information about System and User Behaviour
by looking at its inputs and outputs
that leads to Customer and Business Impact
6
© 2020 ThoughtWorks

7. The Evolution
© 2020 ThoughtWorks
7

8. The Evolution
© 2020 ThoughtWorks
8

9. Explosion of Data Sources
© 2020 ThoughtWorks
System
Customer
Support
System
Level
Metrics
User Click
Stream
Market
Data
Social
Media
Other
Sources
Logs
Mobile
Application
Events
9

10. Explosion of Personas & Explosion of Requirements
Deployment status
Load on a service
Downtime for
Service
Developers
Service Availability
Service Traceability
Routing
Security Team
Revenue Generated
Price Sensitivity
Number of Users
Business
Data Scientists
© 2020 ThoughtWorks

11. Observability (as we know) is not a Silver Bullet
● Unified architectures (of consuming platform) are not possible
● Siloed and locked useful data into various tools
● Learn individual tools
● Difficult to Correlate Quickly for Monitoring or finding Business relevance
● Limited scope for exploration
● Who is focussed on Customer happiness
● Do we want 99.99% availability but still pissed off customer
© 2020 ThoughtWorks
11
We need to RETHINK Observability

12. Platform which enables people to use their skills,
extend their senses, support their intuitions.
Make it quick and easy to explore a hypothesis
(business or technical), accept or disprove it, and move
on to find the root cause
12
© 2020 ThoughtWorks
12

13. Make it Self Service
13
© 2020 ThoughtWorks

14. ● Data/Access Patterns
● ETL/EL(T)
● Lambda/Kappa
● Raw layer
● Compaction
● Partitioning
Questions to ask
© 2020 ThoughtWorks

15. 15
Technical Architecture
Application
Data (Kafka)
Infrastructure
(prometheus)
Each Customer
Journey
(Jaeger)
Request
Routing
(ISTIO)
Retries (DLQ) LOGS (EFK)
Clickstream
(e.g. GTM)
Data Sources
Real Time/Interactive Query
Engine
Real Time (Historical Dashboard)
Historical Data Processing
Data Lake
Observability Platform
Raw Layer CDM/Staging Layer
© 2020 ThoughtWorks
Schema
Registry
Data
Encryption
Configurations
Stream data
producer
Real Time
events

16. 16
© 2020 ThoughtWorks
Points to consider while building
● Designing messaging systems
● Schema registry for versioning
● Contract testing for consumers
● Reprocessing of data in failures
● Track all the metadata
● Enforce the retention policy
● Do not make exceptions
● Build it incrementally
● CI/CD
● Alerting

17. Observability in Real Time
© 2020 ThoughtWorks
Dev-Ops
- Monitor
Production
Systems
- Debug Errors
Analysts
- Micro Trends
- Anomalies
End Users
- Fraud
Detection
< 5 min
Stream
Processing
< 1 sec
Database
End Users
- OLTP
Systems
< 1 hour
Incremental
Mini-batch Processing
Business Users
- Run ad hoc reports
- KPI monitoring
- Functionality Validation
Analysts / Data Scientists
- Feature extraction
- Model validations
- Quick hypothesis validation
17
CXO
- Weekly/Monthly report
- Baseline Measures
ML Engineers
- Experimentation
- Training models
Batch Processing

18. Ingestion
How to ingest data at scale ?
Data Lake
Storage and Processing
How to store and process data at scale?
Visualization
How to query and visualise
data at scale ?
Observability at Scale
© 2020 ThoughtWorks
18

19. 19
Messaging Technologies
Streaming Technologies
Querying Tools
Reporting Tools
© 2020 ThoughtWorks

21. 21
© 2020 ThoughtWorks
Making Observability a success
● Upfront capacity planning
● Standardization using events
● Configurable datasource onboarding
● Clarity on KPIs, Metrices
● Not fixating on visualization
● Metadata management - Business, Technical
and Operational
● Collaboration / Promote your work
● Teach Basic Skills
● Measure Usage
● Track Need

22. Thanks !
Questions ?
22
© 2020 ThoughtWorks