Monitoring patterns for mitigating technical risk
•Download as PPTX, PDF•
4 likes•3,070 views
How to define auto healing and alerts for a low latency REST API system. Includes real world riemann code snippets.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (10)
Similar to Monitoring patterns for mitigating technical risk
Similar to Monitoring patterns for mitigating technical risk (20)
Recently uploaded
Recently uploaded (20)
Monitoring patterns for mitigating technical risk
- 1. Monitoring Patterns for Mitigating Technical Risk @Forter
- 2. #1 risk Slow or bad (500) API responses Auto-healing because humans are slow SLA, Failover, Degradation, Throttling Alerting Detect, Filter, Alert, Diagnostics
- 3. SLA Performance Data Loss Business Logic TX Processing Low Latency Nope Best Effort Stream Processing High Throughput Best Effort Best Effort Batch Processing High Volume Nope Reconciliation
- 4. Automatic Failover http fencing (Incapsula) http load balancing (ELB) instance restart (Scaling Group) process restart (upstart) exceptions bubble up and crash
- 5. Graceful Degradation nginx (lua) expressjs (nodejs) storm (java) Stability Code Changes
- 6. Throttling (without back-pressure) request priority reduced when TX/sec > thresh Different priority → Different queue → Different worker lower priority inside queue for test probes
- 7. Detect -> Filter -> Alert -> Manual Diagnostics Alerting
- 8. Detection
- 10. alert
- 11. diagnostics
- 12. redundancy CloudWatch/CollectD - fast, no root cause App events (exceptions) - too noisy, root cause Pingdom probes - low coverage, reliable Internal probes - better coverage, false alarms
- 15. filter tests using a state machine
- 16. filter tests using a state machine (tagged "apisRegression" (pagerduty-test-dispatch "1234567892ed295d91")) (defn pagerduty-test-dispatch [key] (let [pd (pagerduty key)] (changed-state {:init "passed"} (where (state "passed") (:resolve pd)) (where (state "failed") (:trigger pd)))))
- 17. re-open manually resolved alert
- 18. re-open manually resolved alert (tagged "apisRegression" (pagerduty-test-dispatch "1234567892ed295d91")) (defn pagerduty-test-dispatch [key] (let [pd (pagerduty key)] (sdo (changed-state {:init "passed"} (where (state "passed") (:resolve pd))) (where (state "failed") (by [:host :service] (throttle 1 60 (:trigger pd)))))))
- 19. Diagnostics - Storm topology timing
- 20. Diagnostics - Storm timelines
- 22. #2 risk Slowing down merchant's website Alerting Monitor each and every browser Aggregations (per browser type) Notify on thresholds
- 23. Monitoring our javascript snippet Timeouts Exceptions by browser Exception aggregation Monitoring new versions
- 24. Riemann's Index (server monitoring) key (host+service) event TTL 10.0.0.1-redisfree { "metric":"5"} 60 10.0.0.1-probe1 {"state":"failed"} 300 10.0.0.2-probe1 { "state":"passed"} 300
- 25. Riemann's Index key (host+service) event TTL 199.25.1.1-1234 {"state":"loaded"} 300 199.25.2.1-4567 {"state":"downloaded"} 300 199.25.3.1-8901 {"state":"loaded"} 300 For our use case: host=browser-ip, service=cookie
- 26. Riemann's state machine (index) stores last event and creates expired events (TTL) (by [:host :service] stream) creates a new stream for each host/service (by-host-service stream) - forter's fork only also closes stream when TTL expires
- 27. (defn calc-load-time [& children] (by-host-service (changed :state {:pairs? true} (smap (fn [[previous current]] (cond (and (= (:state previous) "downloaded") (= (:state current) "loaded")) (assoc previous :metric (- (:time current) (:time previous))) (and (= (:state previous) "downloaded") (= (:state current) "expired")) (assoc previous :metric (* JS_TIMEOUT 1000)))) children))))
- 28. (defn aggregate-by-browser [& children] (by [:browser] (fixed-time-window 60 (sdo (smap folds/median (tag "median-load-time" children)) (smap folds/count (tag "load-count" children)))))))
- 29. #3 risk Wrong decision (approve/decline) Alerting Anomaly detection
- 30. Motivation Control false alarms mathematically Threshold per customer Threshold seasonality
- 32. Alert me if the probability that we decline more than k out of n transactions given probability p is 1 in a million (t=0.0001%) n number of tx (30 minutes) k number of declined txs (30 minutes) p per customer declined/total (24 hours) t alert threshold
- 34. Binomial Distribution Assumption External events are uncorrelated What happens when a customer retries the same Tx because the first one was declined?
Editor's Notes
- riemann is an event streaming processing server written in clojure tailored for monitoring. It receives input from the infrastructure (OS/queues/DBs) and instrumented applications (all prog languages) processes the events (enrichement, filtering, aggregation) and forwards them for visualization and alerting. explain the pros/cons of infra monitoring (plugins), application monitoring (custom events), and system probes. the need for processing before a machine wakes you up at night (the role of pagerduty in all of this) and the need for visualization when you wake up at night.
- riemann is an event streaming processing server written in clojure tailored for monitoring. It receives input from the infrastructure (OS/queues/DBs) and instrumented applications (all prog languages) processes the events (enrichement, filtering, aggregation) and forwards them for visualization and alerting. explain the pros/cons of infra monitoring (plugins), application monitoring (custom events), and system probes. the need for processing before a machine wakes you up at night (the role of pagerduty in all of this) and the need for visualization when you wake up at night.
- riemann is an event streaming processing server written in clojure tailored for monitoring. It receives input from the infrastructure (OS/queues/DBs) and instrumented applications (all prog languages) processes the events (enrichement, filtering, aggregation) and forwards them for visualization and alerting. explain the pros/cons of infra monitoring (plugins), application monitoring (custom events), and system probes. the need for processing before a machine wakes you up at night (the role of pagerduty in all of this) and the need for visualization when you wake up at night.
- riemann is an event streaming processing server written in clojure tailored for monitoring. It receives input from the infrastructure (OS/queues/DBs) and instrumented applications (all prog languages) processes the events (enrichement, filtering, aggregation) and forwards them for visualization and alerting. explain the pros/cons of infra monitoring (plugins), application monitoring (custom events), and system probes. the need for processing before a machine wakes you up at night (the role of pagerduty in all of this) and the need for visualization when you wake up at night.
- riemann is an event streaming processing server written in clojure tailored for monitoring. It receives input from the infrastructure (OS/queues/DBs) and instrumented applications (all prog languages) processes the events (enrichement, filtering, aggregation) and forwards them for visualization and alerting. explain the pros/cons of infra monitoring (plugins), application monitoring (custom events), and system probes. the need for processing before a machine wakes you up at night (the role of pagerduty in all of this) and the need for visualization when you wake up at night.
- On the left: example for cloudwatch alert for the load balancer detecting REST API error 500 (internal server error) On the right: example for escalation policies. alert for a system test (probe) that failed.
- On the left: example for cloudwatch alert for the load balancer detecting REST API error 500 (internal server error) On the right: example for escalation policies. alert for a system test (probe) that failed.
- PagerDuty has an alerts panel. An alert is either triggered or resolved. PD assigns the alert to whoever is on duty and then notifies him based on predefined rules (for example call if the alert has not been resolved for 15 minutes). However, PagerDuty API has a throttler. We cannot send every test result to PD every time the test runs. In essence we want to replicate the riemann state into PD. Only when the riemann state changes (a test that previously passed now fails or vica versa) we update PD. Riemann makes it easy to define a state machine by storing the last's event per [host+service] combination.
- PagerDuty has an alerts panel. An alert is either triggered or resolved. PD assigns the alert to whoever is on duty and then notifies him based on predefined rules (for example call if the alert has not been resolved for 15 minutes). However, PagerDuty API has a throttler. We cannot send every test result to PD every time the test runs. In essence we want to replicate the riemann state into PD. Only when the riemann state changes (a test that previously passed now fails or vica versa) we update PD. Riemann makes it easy to define a state machine by storing the last's event per [host+service] combination.
- But what happens if we manually resolved the alert on PD and the test keeps failing? You would want the alert to be repoened again. This means that we need to filter passed test events using state machine, but not filter failed tests at all. We just throttle them to no more than 1 per minute per test permutation [host service] combination.
- But what happens if we manually resolved the alert on PD and the test keeps failing? You would want the alert to be repoened again. This means that we need to filter passed test events using state machine, but not filter failed tests at all. We just throttle them to no more than 1 per minute per test permutation [host service] combination.
- filtering widgets that were not been possible without riemann enrichmenent. The quick filters allow fast zoom-in on the branch you are working on, and werether you would like to see test probes or not.