Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 1 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 2 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 3 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 4 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 5 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 6 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 7 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 8 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 9 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 10 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 11 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 12 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 13 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 14 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 15 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 16 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 17 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 18 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 19 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 20 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 21 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 22 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 23 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 24 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 25 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 26 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 27 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 28 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 29 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 30 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 31 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 32 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 33 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 34 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 35 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 36 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 37 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 38 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 39 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 40 Metrics, Logs, Transaction Traces, Anomaly Detection at Scale Slide 41
Upcoming SlideShare
Multi-container Applications on OpenShift with Ansible Service Broker
Next
Download to read offline and view in fullscreen.

2 Likes

Share

Download to read offline

Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Download to read offline

Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

  1. 1. Processing Metrics, Logs & Traces … at Scale Otis Gospodnetić
  2. 2. WHO WHY WHAT HOW
  3. 3. WHO HQ: Brooklyn People: Everywhere
  4. 4. WHO Otis Gospodnetić Sematext founder Apache member Book author ex-Lucene/Solr dev
  5. 5. WHO Services Solr Elasticsearch* Kafka Spark HBase Cassandra... * We’ve got serious Solr & Elasticsearch ninjas on the team!
  6. 6. WHY
  7. 7. WHO Clients want Performance Bottlenecks Tuning Scaling WHY
  8. 8. WHO Before you can fix things need to know what to fix WHY
  9. 9. WHY We need….INSIGHT Performance Metrics! Anomalies! Logs!
  10. 10. WHY i.e. Need Tools! Metrics monitoring Log searching Anomaly alerting
  11. 11. OSS Use the (Open) Source, Luke
  12. 12. OSS OpenTSDB InfluxDB Ganglia Graphite Nagios ELK ...
  13. 13. OSS http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/
  14. 14. OSS “I have an ELK stack that has been suffering as of late. The logstash service will continually crash, the elasticsearch cluster is hardly in the green, and it is taking a constant amount of maintenance.”
  15. 15. WHAT
  16. 16. WHAT SPM → monitoring Logsene → logging On PremisesCloud http://sematext.com/spm http://sematext.com/logsene
  17. 17. WHAT http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/
  18. 18. WHAT SPM Logsene
  19. 19. HOW
  20. 20. WHAT Agent Java Node.js Want Traces? Embed it! Collectd ⇒ SIGAR for OS Flume SpilloverChannel ES API
  21. 21. WHAT Interesting finds Variable Collectd support Collectd ⇒ SIGAR Apache Flume Elasticsearch Stats API Metrics 2nd class citizen
  22. 22. WHAT Transaction Tracing Java Bytecode Instrumentation Bottleneck finder AppMap maker
  23. 23. WHAT Custom Pointcuts <method signature="java.lang.String com.company. example.Service#getUserName(com.company.model. Company company)"/>
  24. 24. Write-agg vs. Read-agg
  25. 25. Anomalies > Thresholds
  26. 26. WHAT Alerts Heartbeats Thresholds Anomalies
  27. 27. WHAT Anomaly Detection ExponentialSTDFromMA KNN ... boolean result = anomalyCount / (notAnomalyCount + anomalyCount) >= 3d / 4d;
  28. 28. WHAT Anomaly issues Warn early / create noise Normal abnormalities Slow change
  29. 29. Scalable Data Stores
  30. 30. http://blog.sematext.com/2015/06/09/docker-monitoring-support/
  31. 31. Logging
  32. 32. Hot vs. Cold HOT COLD
  33. 33. Drop, don’t Delete HOT COLD drop
  34. 34. Pull, don’t Push GET QUEUE pull ES
  35. 35. Beware of Aggregations Circuit Breakers
  36. 36. http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/
  37. 37. Thank you! @otisg otis@sematext.com @sematext http://sematext.com
  • IuliusHutuleac

    Aug. 17, 2016
  • psj1972

    Jun. 12, 2015

Views

Total views

4,658

On Slideshare

0

From embeds

0

Number of embeds

2,501

Actions

Downloads

33

Shares

0

Comments

0

Likes

2

×