Logs management using Logstash, ElasticSearch and Kibana, Some extra content about statsd and graphite

1. Logs Management

2. infrastructure

3. logs management

4. logflow

5. logstash
Logstash tool for managing events and logs
Logstash written in JRuby
logstash “all in one” jar file
logstash inputs -> filters -> outputs

6. logstash plugins
example
Inputs
eventlog, file, s3, syslog, tcp, udp, websocket, wmi
Filters
csv, dns, geoip, grok, mutate
Outputs
elasticsearch, email, exec, mongodb, rabbitmq, redis
http://logstash.net/docs/1.2.2/

7. logstash example
with puppet
=
input {
stdin{
type => “example“
}
}

8. logstash example
with puppet
=
output {
redis {
data_type => "list"
host => [’10.0.22.26']
key => "logstash”
}
}

9. logflow

10. redis
Redis open source, advanced key-value store
Redis buffer new logs from any type of slower parsing

11. logflow

12. logflow

13. logstash example
with puppet
=
input {
syslog {
port => 5544
type => "hapr"
}
}

14. log4net example

15. logflow

16. logstash example
with puppet
=
output {
elasticsearch {
cluster => "logs”
embedded => false
index => "%{type}-%{+YYYY.MM.dd}”
node_name => "logstash_output"
}
}

17. logflow

18. elasticsearch
ES distributed restful search and analytics engine
ES build on top of apache lucene
ES distributed, highly available
ES document oriented, schema free
ES restfull api

19. elasticsearch
Data Structure
Index
shards
replica
http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch

20. elasticsearch
Data Structure
Index
shards
replica
http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch

21. elasticsearch
Data Structure
Index
shards
replica
http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch

22. elasticsearch
Demo
http://logs.adform.com:9200/_plugin/bigdesk
http://logs.adform.com:9200/_plugin/paramedic/

23. logflow

24. kibana 3
kibana: HTML + JavaScript
kibana: analytics and search interface to timestamped data sets stored in
ElasticSearch
kibana: browser connects directly to
ElasticSearch

25. kibana 3
http://logs.adform.com/kibana

26. logflow

27. logflow

28. logflow

29. statsD
statsD: NodeJS daemon
statsD: extarcts metrics data and flushes to backend
statsD: counters, timers, gouges
timers example:
450
120
553
994
334
844
675
496
=>
mean_90 496
upper_90 844
sum_90 3472
upper 994
lower 120
count 8
sum 4466
mean 558.2
http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/

30. statsD
clients
.NET 4.0 : https://github.com/robbihun/NStatsD.Client
C#: https://github.com/goncalopereira/statsd-csharp-client
.NET: https://github.com/peschuster/graphite-client
.NET 3.5 – 4.5: http://www.nuget.org/packages/StatsdCsharpClient/
https://github.com/etsy/statsd/wiki

31. statsD
example
NStatsD.Client.Current.Increment("testing.increment");
NStatsD.Client.Current.Decrement("testing.decrement");
NStatsD.Client.Current.Timing("testing.timing", 2345);
NStatsD.Client.Current.Gauge("testing.gauge", 45);
https://github.com/robbihun/NStatsD.Client

32. logflow

33. graphite
graphite: highly scalable real-time graphing system
graphite: good when you don’t know the names metrics

34. graphite
web
DEMO

35. graphite
clients
.NET: https://github.com/peschuster/graphite-client
•
•
•
•
•
WCF
MSBuild
ELMAH
SQL Server
PerfCounterMonitor.exe (Graphite.System)
• Performance counters
• Event log
• IIS Application Pools (with)

36. what's next

37. what's next
LEARN!!!

38. what's next
dashboards
http://techblog.netflix.com/2012/12/hystrix-dashboard-and-turbine.html
http://shopify.github.io/dashing/#widgets
http://fdietz.github.io/team_dashboard/