This document discusses Sematext's monitoring and logging products and services. It introduces Sematext, which is headquartered in Brooklyn and has employees globally. It then discusses why performance monitoring, log searching, and anomaly alerting are needed capabilities (Why). The document proceeds to describe Sematext's SPM and Logsene products, which provide these capabilities using open source technologies like OpenTSDB, Elasticsearch, and Kafka. It covers how the SPM agent collects metrics and traces and how Logsene ingests and analyzes logs at scale.

1. Processing Metrics,
Logs & Traces
… at Scale
Otis Gospodnetić

2. WHO WHY
WHAT HOW

3. WHO
HQ: Brooklyn
People: Everywhere

4. WHO
Otis Gospodnetić
Sematext founder
Apache member
Book author
ex-Lucene/Solr dev

5. WHO Services
Solr Elasticsearch*
Kafka Spark
HBase Cassandra...
* We’ve got serious Solr & Elasticsearch ninjas on the team!

6. WHY

7. WHO Clients want
Performance
Bottlenecks
Tuning
Scaling
WHY

8. WHO
Before you can fix things
need to know what to fix
WHY

9. WHY We need….INSIGHT
Performance Metrics!
Anomalies!
Logs!

10. WHY i.e. Need Tools!
Metrics monitoring
Log searching
Anomaly alerting

11. OSS
Use the (Open) Source, Luke

12. OSS
OpenTSDB
InfluxDB
Ganglia
Graphite
Nagios
ELK
...

13. OSS
http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/

14. OSS
“I have an ELK stack that has
been suffering as of late. The
logstash service will
continually crash, the
elasticsearch cluster is hardly
in the green, and it is taking a
constant amount of
maintenance.”

15. WHAT

16. WHAT
SPM → monitoring
Logsene → logging
On PremisesCloud
http://sematext.com/spm http://sematext.com/logsene

17. WHAT
http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/

18. WHAT SPM
Logsene

19. HOW

21. WHAT Agent
Java Node.js
Want Traces? Embed it!
Collectd ⇒ SIGAR for OS
Flume SpilloverChannel
ES API

22. WHAT Interesting finds
Variable Collectd support
Collectd ⇒ SIGAR
Apache Flume
Elasticsearch Stats API
Metrics 2nd class citizen

23. WHAT Transaction Tracing
Java Bytecode
Instrumentation
Bottleneck finder
AppMap maker

26. WHAT Custom Pointcuts
<method signature="java.lang.String com.company.
example.Service#getUserName(com.company.model.
Company company)"/>

28. Write-agg vs. Read-agg

29. Anomalies > Thresholds

30. WHAT Alerts
Heartbeats
Thresholds
Anomalies

31. WHAT Anomaly Detection
ExponentialSTDFromMA
KNN ...
boolean result = anomalyCount /
(notAnomalyCount + anomalyCount) >= 3d / 4d;

32. WHAT Anomaly issues
Warn early / create noise
Normal abnormalities
Slow change

33. Scalable Data Stores

34. http://blog.sematext.com/2015/06/09/docker-monitoring-support/

35. Logging

36. Hot vs. Cold
HOT COLD

37. Drop, don’t Delete
HOT COLD
drop

38. Pull, don’t Push
GET QUEUE
pull
ES

39. Beware of Aggregations
Circuit Breakers

40. http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/

41. Thank you!
@otisg
otis@sematext.com
@sematext
http://sematext.com