Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The Observability Graph; Knowledge Graphs for Automated Infrastructure Observability
Similar to The Observability Graph; Knowledge Graphs for Automated Infrastructure Observability (20)
Recently uploaded
Recently uploaded (20)
The Observability Graph; Knowledge Graphs for Automated Infrastructure Observability
- 1. The Observability Graph: Knowledge Graphs for Automated Infrastructure Observability Monitorama 22 October 2019
- 4. IMAGE So many hosts, so many containers, so many regions... INFRASTRUCTURE
- 5. IMAGE Microservices, part monolith/part micro, serverless, orchestrated, semi-orchestrated SERVICES
- 6. IMAGE High cardinality, lots and lots of tags METRICS
- 7. IMAGE So many requests. Spans coming from everywhere TRACES
- 8. IMAGE So much variety; so many fields. EVENTS
- 9. IMAGE So much detail. So many lines. LOGS
- 12. IDEAL service:A
- 14. IDEAL Service A Service BRequests DB
- 15. IDEAL Service A Service BRequests DB lag_metric{service:B}
- 16. IDEAL Service A Service BRequests DB system.mem.pct_usable{DB, host:h111}
- 19. IMAGE Amazing Observability Ideal Environment Easy Root Cause Analysis
- 20. IMAGE Amazing Observability Ideal Environment Easy Root Cause Analysis (maybe even…)
- 21. IMAGE Amazing Observability Ideal Environment Easy Root Cause Analysis Automatically Surface All Contributing Factors
- 23. REALITY service:A service:A is an intern project with simulated traffic
- 24. REALITY Service ARequests DB lag_metric{lsv-systype:B} Lsv-systype (???) B Service X Service Y Service Z Service F Service G
- 25. REALITY Service A Service BRequests DB proxy
- 26. IMAGE Fruggo
- 29. IMAGE The mirroring hypothesis a.k.a. Conway’s Law a.k.a. You ship your org chart... Colfer, Lyra J., and Carliss Y. Baldwin. "The mirroring hypothesis: theory, evidence, and exceptions." Industrial and Corporate Change 25.5 (2016): 709-738.
- 30. IMAGE Corollary: Observability follows your org chart.
- 31. IMAGE Corollary: Observability follows your org chart. app | storage
- 32. IMAGE Corollary: Observability follows your org chart.
- 33. IMAGE Corollary: Observability follows your org chart.
- 34. IMAGE Gore’s hypothesis a.k.a. The prequel to Dunbar’s Number a.k.a. The thing you heard about from The Tipping Point Zhou, W-X., et al. "Discrete hierarchical organization of social group sizes." Proceedings of the Royal Society B: Biological Sciences 272.1561 (2005): 439-444. Hamel, Gary, and B. Breen. "Building an innovation democracy: WL Gore." The future of management (2007)
- 35. IMAGE Corollary: Observability standards completely fall apart after 150 employees.
- 36. IMAGE Synthesis Without enforcement, observability standards fall apart after 150 employees, but still tend to follow the org chart.
- 37. IMAGE Fruggo
- 40. (in the real world) ROOT CAUSE ANALYSIS
- 41. The Situation Negatives: – Large-scale, messy, inconsistent observability data – Labels are hard to come by Positives: – Domain knowledge – Lots of user-interaction data
- 42. The Situation Negatives: – Large-scale, messy, inconsistent observability data – Labels are hard to come by Positives: – Domain knowledge – Lots of user-interaction data Machine Learning
- 43. The Situation Negatives: – Large-scale, messy, inconsistent observability data – Labels are hard to come by Positives: – Domain knowledge – Lots of user-interaction data Unsupervised (or Semi-Supervised) Machine Learning
- 44. The Situation Negatives: – Large-scale, messy, inconsistent observability data – Labels are hard to come by Positives: – Domain knowledge – Lots of user-interaction data Unsupervised (or Semi-Supervised) Known Entities Machine Learning
- 45. The Situation Negatives: – Large-scale, messy, inconsistent observability data – Labels are hard to come by Positives: – Domain knowledge – Lots of user-interaction data Unsupervised (or Semi-Supervised) Known Entities Relational Data Machine Learning
- 48. Observability Graphs Alert A Metric M Team T Service S
- 49. Observability Graphs Alert A Metric M Team T Alert B Metric N Service S
- 50. Observability Graphs Alert A Metric M Team T Alert B Metric N Service S
- 51. Observability Graphs Alert A Metric M Dashboard DTeam T Alert B Metric N Service S
- 52. Observability Graphs Alert A Metric M Dashboard DTeam T Alert B Metric N Metric OAlert C Service S
- 53. Observability Graphs Alert A Metric M Dashboard DTeam T Service SService R Alert B Metric N Metric OAlert C system.cpu.idle{role:R}“[R] CPU is high on R!”
- 54. Observability Graphs Alert A Metric M Dashboard DTeam T Service SService R Alert B Metric N Metric OAlert C
- 55. Observability Graphs Alert A Metric M Dashboard D Dashboard E Team T Service SService R Alert W Metric P Alert B Metric N Metric OAlert C
- 56. Observability Graphs Alert A Metric M Dashboard D Dashboard E Team T Service SService R Alert W Metric P Alert B Metric N Metric OAlert C Alert X Alert Y Alert Z
- 57. Observability Graphs Alert A Metric M Dashboard D Dashboard E Team T Service SService R Alert W Metric P Alert B Metric N Metric OAlert C Alert X Alert Y Alert Z DB U Service V DB W
- 58. Observability Graphs Alert A Metric M Dashboard D Dashboard E Team T Service SService R Alert W Metric P Alert B Metric N Metric OAlert C Alert X Alert Y Alert Z DB U Service V DB W Metric L
- 59. REALITY Service S Service RRequests DB W system.mem.pct_usable{DB, host:h111}
- 61. IMAGE https://knowyourmeme.com/memes/galaxy-brain CONCLUSIONS Machine Learning and Knowledge Graphs save the day!
- 62. IMAGE Sam Cali CONCLUSIONS Observability data is created by people to be consumed by people. Monitoring tools and data are useless if people can’t make sense of them. By studying how people interact with this data, we can increase the observability of our systems.