Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Stream Processing Inside Librato [Monitorama PDX 2015]


Published on

As modern software engineering tools and techniques continue to evolve, very few engineering teams have been able to meaningfully improve their operational visibility. In this talk Librato's Dave Josephsen makes that case that monitoring has always been a signal processing problem. He explains in practical terms what that means, and how viewing the problem in the context of signal processing necessitates a different sort of toolset.

He then shares Librato's stream processing infrastructure, which is an implementation of a signal-processing telemetry system, and shows how our infrastructure solves many of the data persistence and signal transformation problems that engineering teams who use traditional operations-centric monitoring toolchain have struggled with for years.

Published in: Data & Analytics
  • Be the first to comment

Stream Processing Inside Librato [Monitorama PDX 2015]

  1. 1. hi.
  2. 2. hi. @davejosephsen github: djosephsen
  3. 3. hi. @davejosephsen github: djosephsen
  4. 4. Who put Stream Processing in my Monitoring Conference?
  5. 5. Let’s talk about signal processing
  6. 6. RealTime Audio Processing
  7. 7. WhisperDB WhisperDB WhisperDB WhisperDB
  8. 8. Nagios Zabbix Collectd DAC (transform and normalize the signal) (e.g., translate input to a standard json blob)
  9. 9. Zabbix Filters (de-duplication, noise reduction) (e.g., compute percentiles to filter outliers)
  10. 10. Multiplex (aggregation) (e.g., aggregate http 200’s from multiple hosts)
  11. 11. Effects (ad-hoc external signal processing) ??
  12. 12. Effects (ad hoc external signal processing) ??
  13. 13. Say “Exponential Smoothing” again
  14. 14. Disk Latency Queue Latency Model A Model B
  15. 15. ing MultiVariate Data Science Menu Specials Gaussian Fuzzy Discrete Hybridized Smoothing ExponentialSmoothing Holt-Winters Chicken Fried Exponential Smoothing Naieve Baysian linear regression Markov Chains extreme studentized Kung-Pow distance vector grouping stream decompisiton Organic, free-range Ansecomb Quartet really really smooth Exponential Smoothing xtra seasonality available on request Downsampling Statistical Convergence Lacunary Sequence Hashsum Distance Lambda Convergence Lambda ANYTHING Seasonal Hybridized Whatever HALF OFF! Anything Named after a turn-of-the-century Member of the Russian Intelligencia Spacial Forecasting deviant Test Quantile isolation Probablistic merging Hybridized/Fuzzy Combo platter! Collaborative n-gram cosine matching Robust Sequential Map-Reduce Micro-lot N-variate chi-square Ask about our locally-sourced quantile robustness Roasted Daily!
  16. 16. Scientist gonna science
  17. 17. Scientist gonna science Engineer gonna.. yeah.
  18. 18. Scientist gonna scienceData Engineer gonna ??
  19. 19. Making a model is just the beginning
  20. 20. Nagios
  21. 21. GraphiosNagios
  22. 22. Graphios StatsdNagios
  23. 23. Graphios Statsd GraphiteNagios
  24. 24. raphios Statsd Graphite Grafana
  25. 25. raphios Statsd Graphite Grafana Math thingy ???
  26. 26. Effects (ad hoc external signal processing) ??
  27. 27. Stuff Statsd Graphite Grafana
  28. 28. Stuff Graphite GrafanaHeka
  29. 29. Stuff Graphite GrafanaHeka Model A Model B
  30. 30. Stuff Graphite GrafanaHeka Model A Model B carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay
  31. 31. Stuff Graphite GrafanaHeka Model A Model B carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay Elastisearch Elastisearch
  32. 32. Data in Truth out
  33. 33. X
  34. 34. XX X
  35. 35. XX X X X X X
  36. 36. Kafka
  37. 37. SSA Kafka
  38. 38. SSA Kafka
  39. 39. SSA r60 Kafka
  40. 40. SSA r60 r900 Kafka
  41. 41. SSA
  42. 42. SSA r60 Raw Writer
  43. 43. SSA r60 Raw Writer r900 r3600
  44. 44. SSA r60 Raw Writer Alerts
  45. 45. SSA r60 Raw Writer Alerts
  46. 46. SSA MySQL
  47. 47. SSA Alerts MySQL
  48. 48. SSA Alerts MySQL
  49. 49. SSA Alerts MySQL { {
  50. 50. SSA Alerts MySQL
  51. 51. Storm is “pretty OK”
  52. 52. Lacks visibility into flow, performance, and failures
  53. 53. Netty? JVM Hops? Ser/Der?
  54. 54. Samza? Spark? NIH? ( kafka + zk + mezos + disruptor+? )
  55. 55. Questions?