Logging makes perfect - Riemann, Elasticsearch and friends
•
7 likes•2,316 views
Real-world monitoring and visualizations
Recommended
More Related Content
What's hot
What's hot (10)
Similar to Logging makes perfect - Riemann, Elasticsearch and friends
Similar to Logging makes perfect - Riemann, Elasticsearch and friends (20)
Recently uploaded
Recently uploaded (20)
Logging makes perfect - Riemann, Elasticsearch and friends
- 1. Logging Makes Perfect Itamar Syn-‐Hershko h1p://code972.com @synhershko Real-world monitoring and visualizations
- 2. Me? • Itamar Syn-‐Hershko / @synhershko • Blog at h1p://code972.com • Long @me OSS contributor • Lucene.NET PMC and lead commi1er • Elas@csearch consul@ng partner • Building soKware to catch fraudsters @ Forter
- 5. $15K Purchase Amount Washington (high income) Billing Neighborhood Platinum+ Credit Card Type Kibuve, Rwanda IP Location Virginia (Low income) Shipping Neighborhood < $100 Past Purchase Amounts Ivy League School Billing Address Expedited Shipping Shipping type Sean Smith, Macbook Air x 3 DECLINE D
- 6. $2,450 Purchase Amount Washington (high income) Billing Neighborhood Platinum+ Credit Card Type Kibuve, Rwanda IP Location Virginia (Low income) Shipping Neighborhood < $100 Past Purchase Amounts Ivy League School Billing Address (202) 222-3850 Phone Sean is an officer, serving overseas in Rwanda. His shipping address matches the pattern of diplomat mailboxes for US embassies and his billing address is still linked to his old college accommodation. How did we know? Sean Smith, Macbook Air x 3 BEHAVIORAL DATA REAL-TIME APPROVED
- 7. Different systems, different SLAs Performance Data Loss Business Logic TX Processing Low Latency Nope Best Effort Stream Processing High Throughput Best Effort Best Effort Batch Processing High Volume Nope Reconcilia@on
- 8. Forter’s TX Processing API Forter Transac@on details Decision (approve / decline) milliseconds
- 9. Achieving Low Latencies For comparison: • A few network hops between AWS availability zones (~1ms) • JVM young garbage collec@on of a few hundred MBs (~20ms) • Read by primary key from SSD storage (~30ms) • US West Coast to East Coast network latency (~100ms) • JVM full garbage collec@on a few hundred MBs (~500ms-‐1000ms)
- 11. Our stack nginx (lua) expressjs (nodejs) Storm (java) Stability Code Changes
- 14. What do we monitor? • Latencies – Benchmark tools are useless here – Always prefer percen@les, measure the long tail – Dis@nguishing real traffic from synthe@c requests • Availability – No non-‐200 HTTP status codes returned • High / suspicious metrics • Errors, excep@ons • Wrong processing (data loss?)
- 15. Detec@on
- 16. filter & route
- 17. alert
- 18. diagnos@cs
- 19. • Pingdom probes -‐ low coverage, reliable • CloudWatch, collectd -‐ fast, no root cause • Internal probes -‐ be1er coverage, false alarms • libbeat and logstash for monitoring 3rd party applica@ons – some@mes noisy, false alarms • App events (excep@ons) -‐ too noisy, root cause
- 20. Redundancy of detection sources …. is important! • Different concerns • Varying accuracy and verbosity • High availability • Correla@on
- 21. Automatic Latency Reporting: NodeJS georg (named aKer Georg Friedrich Bernhard Riemann) is a nodejs library that acts as an in-‐process riemann agent. The library can send nodejs specific and applica@on specific metrics to riemann. h1ps://github.com/forter/georg
- 24. Automatic Latency Reporting: Apache Storm h1ps://github.com/forter/riemann-‐storm-‐monitor
- 25. The ELK Stack Queue (Redis) “Shippers” “Indexer”
- 26. The ELK Stack + Riemann Queue (Redis) “Shippers”
- 27. • Detect failures • Expect different latencies from different components, dynamically • Tolerate spikes • High / low watermarks • Reshape event streams (split, merge, …) Done via Riemann – an event stream processor
- 28. Riemann
- 29. ; Listen on the local interface over TCP (5555), UDP (5555) & websockets ; (5556) (let [host "127.0.0.1"] (tcp-server {:host host}) (udp-server {:host host}) (ws-server {:host host})) ; Expire old events from the index every 5 seconds. (periodically-expire 5) (let [index (index)] ; Inbound events will be passed to these streams: (streams (default :ttl 60 ; Index all events immediately. index ; Log expired events. (expired (fn [event] (info "expired" event))))))
- 30. Filter tests using a state machine
- 31. (tagged "apisRegression" (pagerduty-test-dispatch "1234567892ed295d91")) (defn pagerduty-test-dispatch [key] (let [pd (pagerduty key)] (changed-state {:init "passed"} (where (state "passed") (:resolve pd)) (where (state "failed") (:trigger pd))))) Filter tests using a state machine
- 32. Re-open manually resolved alert
- 33. Re-open manually resolved alert (tagged "apisRegression" (pagerduty-test-dispatch "1234567892ed295d91")) (defn pagerduty-test-dispatch [key] (let [pd (pagerduty key)] (sdo (changed-state {:init "passed"} (where (state "passed") (:resolve pd))) (where (state "failed") (by [:host :service] (throttle 1 60 (:trigger pd)))))))
- 34. Riemann Events
- 35. Riemann’s Index Key (host + service) Event TTL 10.0.0.1-‐redisfree {"metric":"5"} 60 10.0.0.1-‐probe1 {"state":"failed"} 300 10.0.0.2-‐probe1 {"state":"passed”} 300
- 36. Riemann Ghosts • “Expired events” • Use case: Detect dead services • Use case: Cron stopped working (e.g. backup)
- 37. Riemann Overloaded key (IP + cookie) event TTL 199.25.1.1-‐1234 {"state":"loaded"} 300 199.25.2.1-‐4567 {"state":"downloaded"} 300 199.25.3.1-‐8901 {"state":"loaded"} 300
- 38. Something failed, now what? • Always prefer auto-‐healing – Humans are slow – But careful from a domino effect • Fail fast and automa@c failover – Code throws excep@on, logs it and moves forward – Fatal excep@ons will cause the process to restart (upstart) – Auto scaling groups keep enough machines up – ELB will phase out dead stacks – HTTP fencing (commercial products)
- 39. Alert • When human interven@on is required • Urgent: immediate ac@on – PagerDuty (phone calls, SMS, push no@fica@ons) • Low urgency: events are logged and reviewed the morning aKer – PagerDuty (Phone push no@fica@ons) – Slack – Email
- 40. Diagnostics
- 41. The ELK stack For real-‐@me analy@cs and dashboards
- 42. The ELK Stack + Riemann Queue (Redis) “Shippers”
- 44. Analyzing sleep patterns J * Times are Israel Day Time
- 45. Storm topology visualization & timing
- 46. Storm timelines
- 47. Alerting via anomaly detection
- 48. To conclude… • Fail fast – With back-‐off • Log everything • Aler@ng via state machines • By host, service, @me • Alerts are useless without proper debug tools • Try to self-‐heal when possible.
- 49. Thank you Ques>ons? Itamar Syn-‐Hershko h1p://code972.com @synhershko / @ForterTech