Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Scalable Online Analytics for Monitoring
LISA15, Nov. 13, 2015, Washington, D.C.
Heinrich Hartmann, PhD, Chief Data Scient...
I’m Heinrich
· From Mainz, Germany
· Studied Math. in Mainz, Bonn, Oxford
· PhD in Algebraic Geometry
· Sensor Data Mining...
Circonus is ...
@HeinrichHartman(n)
· monitoring and telemetry analysis platform
· scalable to millions of incoming metric...
@HeinrichHartman(n)
This talk is about...
I. The Future of Monitoring
II. Patterns of Telemetry Analysis
III. Design of th...
Part I - The Future of Monitoring
Monitor this:
· 100 containers in one fleet
· 200 metrics per-container
· 50 code pushes a day
· For each push all contain...
Line-plots work well for small numbers of nodes
@HeinrichHartman(n)
Node 1
Node 2
Node 3
Node 4
Node 5
Node 6
CPU utilizat...
... but can get polluted easily
@HeinrichHartman(n)
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
CPU utilization for a DB clust...
“Information is not a scarce resource. Attention is.”
Herbert A. Simon
@HeinrichHartman(n) Source: http://www.circonus.com...
Store Histograms and Alert on Percentiles
@HeinrichHartman(n)
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
90% percentile...
@HeinrichHartman(n)
Anomaly Detection for Surfacing relevant data
Mockup of an metric overview page. Source: Circonus
Part II - Patterns of Telemetry Analysis
@HeinrichHartman(n)
Telemetry analysis comes in two forms
Online Analytics
· Anomaly detection
· Percentile alerting
· Sma...
@HeinrichHartman(n)
New Components in Circonus
BunsenBeaker
@HeinrichHartman(n)
Beaker & Bunsen in the Circonus Architecture
metrics Ernie
Alerting Service
Snowth
Metric Store
alerts...
@HeinrichHartman(n)
Pattern 1: Windowing
Examples
· Supervised Machine Learning
· Fourier Transformation
· Anomaly Detecti...
@HeinrichHartman(n)
Pattern 3: Processing Units
processing_unit = {
state = ...
update = function(self, y) ... end
}
Examp...
@HeinrichHartman(n)
local q = 0.9
exponential_smoothing = {
s = 0,
update = function(self, y)
self.s = (y * q) + (self.s *...
Processing units are convenient
· Primitive transformation are readily implemented:
Arithmetic, Smoothing, Forecasting, .....
Circonus Analytics Query Language
· Create your own customized processing unit from primitives
· UNIX-inspired syntax with...
@HeinrichHartman(n)
CAQL: Example 1 - Low frequency AD
Pre-process a metric before feeding into anomaly detection
metric:a...
@HeinrichHartman(n)
CAQL: Example 2 - Histogram Aggregation
Histograms are first-class citizens
metric:average(<uuid>) | w...
Part III - The Design of the Online Analytics
Engine ‘Beaker’
1. Read messages from a message queue
2. Execute a processing units over incoming metrics
3. Publish computed values to me...
Challenge 1: Rollup metrics by the minute
@HeinrichHartman(n)
· Metrics arrive asynchronously on the input queue
· PUs exp...
Consequence: No real-time processing in Beaker
· Rolling up data in 1m periods causes an average delay
of 30sec for data p...
Challenge 2: Multiple input slots
input metric
input metric
input metric
input metric
input metric
processing unit
process...
Challenge 3:
Synchronize roll-ups for multiple inputs is tricky
@HeinrichHartman(n)
Challenge 4: Fault Tolerance
Definition (Birman): A service is fault tolerant if it
tolerates the failure of nodes without...
Beaker v0.5:
· Automated restarts
a. Service Management Facilities (svcadm) in OmniOS
b. systemd or watchdogs in Linux
· B...
Beaker v. 0.5: Use db to rebuild state on startup
@HeinrichHartman(n)
Beaker Service
output
Q
input
Q
Snowth db
Challenge 5: High Availability
Definition (Birman): A service is highly available it
continues to publish valid messages d...
Beaker v. 0.5 -- HA Cluster
Beaker HA Cluster
Beaker Master
output
Q
input
Q
Beaker
Slave
Beaker
Slave On
Failover:
Master...
@HeinrichHartman(n)
Challenge 6: Scalability
Beaker needs to scale in the following dimensions
· Number of processing unit...
Beaker v.0.6 -- Multiple - HA Clusters
Beaker HA Cluster II (3 slaves)
Beaker HA Cluster I (5 slaves)
..
.
..
Beaker HA Cl...
@HeinrichHartman(n)
Done! Great. This works...
Can we do better?
@HeinrichHartman(n)
· Divide Beaker into multiple services
· Avoid master election in processing service, by allowing duplicates
· Upside: Onl...
Scaling the Processing Service is simplified
· No rollup logic in workers
· All workers publish messages
· No failover log...
Benchmark results on prototype
· PU type anomaly_detection
PU count 100
Throughput 15 kHz
· PU type anomaly_detection
PU c...
Conclusion
· Stateful processing units allow implementation of next
generation of monitoring analytics
· Use CAQL to build...
Credits
Joint work with:
· Jonas Kunze
· Theo Schlossnagle
Image Credits:
Example of Kummer Surface, Claudio Rocchini, CC-...
Upcoming SlideShare
Loading in …5
×

Scalable Online Analytics for Monitoring

2,425 views

Published on

Monitoring systems will get smarter in order to keep up with the demands of tomorrow's IT architectures. Features like anomaly detection, root cause analysis, and forecasting tools will be critical components of this next level of monitoring. At the same time, the data that monitoring systems ingest is ever increasing in amount and velocity.

This session covers architectural models for advanced online analytics. We argue that stateful online computations provide a means to realize machine learning on high-velocity data. We show how alerting systems, event engines, stream aggregators, and time-series databases interact to support smart, scalable, and resilient monitoring solutions.

Heinrich Hartmann is the Chief Data Scientist at Circonus. He is driving the development of analytics methods that transform monitoring data into actionable information as part of the Circonus monitoring platform. In his prior life, Heinrich pursued an academic career as a mathematician (PhD in Bonn, Oxford). Later he transitioned into computer science and worked as consultant for a number of different companies and research institutions.

Published in: Technology

Scalable Online Analytics for Monitoring

  1. 1. Scalable Online Analytics for Monitoring LISA15, Nov. 13, 2015, Washington, D.C. Heinrich Hartmann, PhD, Chief Data Scientist, Circonus
  2. 2. I’m Heinrich · From Mainz, Germany · Studied Math. in Mainz, Bonn, Oxford · PhD in Algebraic Geometry · Sensor Data Mining at Univ. Koblenz · Freelance IT consultant · Chief Data Scientist at Circonus Heinrich.Hartmann@Circonus.com @HeinrichHartman(n)
  3. 3. Circonus is ... @HeinrichHartman(n) · monitoring and telemetry analysis platform · scalable to millions of incoming metrics · available as public and private SaaS · built-in histograms, forecasting, anomaly-detection, ...
  4. 4. @HeinrichHartman(n) This talk is about... I. The Future of Monitoring II. Patterns of Telemetry Analysis III. Design of the Online Analytics Engine ‘Beaker’
  5. 5. Part I - The Future of Monitoring
  6. 6. Monitor this: · 100 containers in one fleet · 200 metrics per-container · 50 code pushes a day · For each push all containers are recreated The “cloud monitoring challenge” 2015 @HeinrichHartman(n)
  7. 7. Line-plots work well for small numbers of nodes @HeinrichHartman(n) Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 CPU utilization for a DB cluster. Source: Internal.
  8. 8. ... but can get polluted easily @HeinrichHartman(n) N N N N N N N N N N N N N N N N N N N N CPU utilization for a DB cluster. Source: Internal.
  9. 9. “Information is not a scarce resource. Attention is.” Herbert A. Simon @HeinrichHartman(n) Source: http://www.circonus.com/the-future-of-monitoring-qa-with-john-allspaw/
  10. 10. Store Histograms and Alert on Percentiles @HeinrichHartman(n) N N N N N N N N N N N N N N N N N N N N N N N 90% percentile CPU Utilization of a db service with a variable number of nodes.
  11. 11. @HeinrichHartman(n) Anomaly Detection for Surfacing relevant data Mockup of an metric overview page. Source: Circonus
  12. 12. Part II - Patterns of Telemetry Analysis
  13. 13. @HeinrichHartman(n) Telemetry analysis comes in two forms Online Analytics · Anomaly detection · Percentile alerting · Smart alerting rules · Smart dashboards Offline Analytics · Post mortem analysis · Assisted thresholding Stream Processing ‘Big Data’
  14. 14. @HeinrichHartman(n) New Components in Circonus BunsenBeaker
  15. 15. @HeinrichHartman(n) Beaker & Bunsen in the Circonus Architecture metrics Ernie Alerting Service Snowth Metric Store alerts Web-UI Q Beaker Online AE Bunsen Offline AE
  16. 16. @HeinrichHartman(n) Pattern 1: Windowing Examples · Supervised Machine Learning · Fourier Transformation · Anomaly Detection (etsy) Remarks · Tradeoff: window size rich features vs. memory · Tradeoff: overlap latency vs. CPU windows = window(y_stream) def results(): for w in windows: yield z = method(w)
  17. 17. @HeinrichHartman(n) Pattern 3: Processing Units processing_unit = { state = ... update = function(self, y) ... end } Example · Exponential Smoothing · Holt Winters forecasting · Anomaly Detection (Circonus) Remarks · Fast updates · Fully general · Cost: maintains state Circonus Anomaly Detection
  18. 18. @HeinrichHartman(n) local q = 0.9 exponential_smoothing = { s = 0, update = function(self, y) self.s = (y * q) + (self.s * (1 - q)) return self.s end } For A Processing Unit for Exponential Smoothing Exponential smoothing applied to a dns-duration metric More examples on http://heinrichhartmann.com/.../Generative-Models-for-Time-Series
  19. 19. Processing units are convenient · Primitive transformation are readily implemented: Arithmetic, Smoothing, Forecasting, ... · Fully general. Allow window-based processing as well · Composable. Compose several PUs to get a new one! @HeinrichHartman(n)
  20. 20. Circonus Analytics Query Language · Create your own customized processing unit from primitives · UNIX-inspired syntax with pipes ‘|’ · Native support for histogram metrics @HeinrichHartman(n)
  21. 21. @HeinrichHartman(n) CAQL: Example 1 - Low frequency AD Pre-process a metric before feeding into anomaly detection metric:average(<>) | rolling:mean(30m) | anomaly_detection()
  22. 22. @HeinrichHartman(n) CAQL: Example 2 - Histogram Aggregation Histograms are first-class citizens metric:average(<uuid>) | window:histogram(1h) | histogram:percentile(95)
  23. 23. Part III - The Design of the Online Analytics Engine ‘Beaker’
  24. 24. 1. Read messages from a message queue 2. Execute a processing units over incoming metrics 3. Publish computed values to message queue Beaker v. 0.1: Simple stream processing Beaker Service output Q input Q Beaker: Basic Data Flow
  25. 25. Challenge 1: Rollup metrics by the minute @HeinrichHartman(n) · Metrics arrive asynchronously on the input queue · PUs expect exactly one sample per minute · Rollup logic needs to allow for · late arrival, e.g. when broker is behind · out of order arrival · errors in system time (time-zones, drifts)
  26. 26. Consequence: No real-time processing in Beaker · Rolling up data in 1m periods causes an average delay of 30sec for data processing. · Real-time threshold-based alerting still available. · Approach: Avoid rolling up data for stateless PUs. @HeinrichHartman(n)
  27. 27. Challenge 2: Multiple input slots input metric input metric input metric input metric input metric processing unit processing unit processing unit output metric output metric output metric Beaker: Logical Data Flow @HeinrichHartman(n)
  28. 28. Challenge 3: Synchronize roll-ups for multiple inputs is tricky @HeinrichHartman(n)
  29. 29. Challenge 4: Fault Tolerance Definition (Birman): A service is fault tolerant if it tolerates the failure of nodes without producing wrong output messages. Failed nodes must be able to recover from a crash and rejoin the cluster. The time to recovery should be as low as possible. @HeinrichHartman(n) Source: K. Birman - Reliable Distributed Systems, Springer, 2005
  30. 30. Beaker v0.5: · Automated restarts a. Service Management Facilities (svcadm) in OmniOS b. systemd or watchdogs in Linux · But, recovery can take a long time State has to be rebuilt from input stream. · Need a way to recover faster from errors: a. Persist processing unit state (software updates!) b. Access persisted metric data @HeinrichHartman(n)
  31. 31. Beaker v. 0.5: Use db to rebuild state on startup @HeinrichHartman(n) Beaker Service output Q input Q Snowth db
  32. 32. Challenge 5: High Availability Definition (Birman): A service is highly available it continues to publish valid messages during a node failure after a small reconfiguration period. For Beaker we require a reconfiguration period of less than 1 minute. In this time messages may be delayed (e.g. 30sec) and duplicated messages may be published. @HeinrichHartman(n) Source: K. Birman - Reliable Distributed Systems, Springer, 2005
  33. 33. Beaker v. 0.5 -- HA Cluster Beaker HA Cluster Beaker Master output Q input Q Beaker Slave Beaker Slave On Failover: Master election @HeinrichHartman(n) Snowth db
  34. 34. @HeinrichHartman(n) Challenge 6: Scalability Beaker needs to scale in the following dimensions · Number of processing units up to an unlimited amount. · In the number of incoming metrics up to ~100M metrics.
  35. 35. Beaker v.0.6 -- Multiple - HA Clusters Beaker HA Cluster II (3 slaves) Beaker HA Cluster I (5 slaves) .. . .. Beaker HA Cluster III (1 slave) BI-M BI-S1 BI-S5 BII-M BII-S1 BII-S3 BIII-M BIII-S1 Meta Service Msg Broker Msg Broker BIII-S2 @HeinrichHartman(n) Snowth db ..
  36. 36. @HeinrichHartman(n) Done! Great. This works...
  37. 37. Can we do better? @HeinrichHartman(n)
  38. 38. · Divide Beaker into multiple services · Avoid master election in processing service, by allowing duplicates · Upside: Only the processing service needs to scale out Beaker Service Beaker v. 1.0 -- Divide and Conquer @HeinrichHartman(n) Q Q Processing Service Dedup Service Rollup Service
  39. 39. Scaling the Processing Service is simplified · No rollup logic in workers · All workers publish messages · No failover logic necessary · Replicate each PUs on multiple workers · No crosstalk between nodes ( = 0 in USL). @HeinrichHartman(n) Q Q ... snowth db Worker Worker Worker Meta Service Processing Service: Data Flow
  40. 40. Benchmark results on prototype · PU type anomaly_detection PU count 100 Throughput 15 kHz · PU type anomaly_detection PU count 10k Throughput 4.2 kHz Machine: 6-core Xeon E5 2630 v2 at 2.6 GHz @HeinrichHartman(n)
  41. 41. Conclusion · Stateful processing units allow implementation of next generation of monitoring analytics · Use CAQL to build your own processing units · Service orientation facilitates scaling · Beaker will be out soon @HeinrichHartman(n)
  42. 42. Credits Joint work with: · Jonas Kunze · Theo Schlossnagle Image Credits: Example of Kummer Surface, Claudio Rocchini, CC-BY-SA, https://en.wikipedia.org/wiki/File:Kummer_surface.png Indian truck, by strudelt, CC-BY, https://commons.wikimedia.org/wiki/File:Truck_in_India_-_overloaded.jpg Kafka, Public Domain, https://en.wikipedia.org/wiki/File:Kafka_portrait.jpg Thinker, Public Domain, https://www.flickr.com/photos/mustangjoe/5966894496 @HeinrichHartman(n)

×