Your SlideShare is downloading. ×
0
Listen to the sounds of
your application
Maciej Biłas @maciejb softwaremill.com jbison.com

!2
Presented at
J-Day
Lbn.sc

Originally authored by:
Maciej Biłas
Krzysztof Ciesielski
Agenda
•

Monitoring, huh?

•

Introducing Graphite

•

Log analysis – the whys

•

Logstash architecture & use cases

•

...
Monitoring, huh?
source: codeascraft.com
Types of measurements
Network
Machine
Application
source: codeascraft.com
measurement > prediction
measurement >> prediction
Our stack
Server

Server
App

…

App

Yammer Metrics

Graphite

Logstash
http://graphite/render?
target=server.web1.load&height=800&width=600
Graphite
echo "local.random.diceroll 4 `date +%s`" |
nc graphite-server.your.org 2003
successful.login.attempt 1 1384471287

successful.login.attempt 1 1384471297

successful.login.attempt = 1
Yammer metrics
private final Meter successfulLogins =

metrics.meter(name(LoginHandler.class, "successful"));
!
public voi...
Metrics
Types:

Reporters:

•

Gauges

•

STDOUT

•

Counters

•

CSV

•

Meters

•

SLF4J

•

Histograms

•

JMX

•

Time...
Counters, meters…
vs.
Gauges
Aggregation
app00.webservice.requestTime
app01.webservice.requestTime
app02.webservice.requestTime
…
source: codeascraft.com
You can also check out
•

collectd https://collectd.org

•

StatsD https://github.com/etsy/statsd/

•

Riemman http://riem...
Log aggregation?
Event Sourcing
Tracing and storing all the events
Current app state: replaying the stream
Great for data mining and analys...
Log aggregation

•

Less invasive way to build an “event stream”

•

Logs are data with plenty of value
Log aggregation
Log
source

Log
source

Log
source

Central
storage

Query
engine

Web
view
Logstash
•

Open source
•

Written in JRuby

•

Gathers logs from various inputs

•

Parses and extracts metadata

•

Writ...
Architecture
Inputs
collectd

graphite

s3

drupal_dblog

heroku

snmptrap
unix

elasticsearch

imap

sqlite
varnishlog

eventlog

irc
...
Grok
Thin layer on top of a regular expression
2013-11-10 19:41:25.321 [main] INFO
o.a.camel.impl.DefaultCamelContext - Ap...
Grok
2013-11-10 19:41:25.321 [main] INFO
o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1
(CamelContext: camel-1) ...
Outputs
boundary
circonus
cloudwatch
csv
datadog
datadog_metric
s
elasticsearch
elasticsearch_ht
tp
elasticsearch_ri
ver
e...
Kibana
Checking app activity
Spotting anomalies
Regular events
How often is a feature used?
Plotting multiple event types
Other tools
•

Splunk

•

Graylog2

•

Fluentd

•

log.io
References
•

Graphite

http://graphite.readthedocs.org/en/latest/

•

Yammer Metrics

http://metrics.codahale.com/

•

Lo...
Thank you!
Questions?
Poznań JUG: Listening to the sounds of your application
Poznań JUG: Listening to the sounds of your application
Poznań JUG: Listening to the sounds of your application
Upcoming SlideShare
Loading in...5
×

Poznań JUG: Listening to the sounds of your application

839

Published on

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
839
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
12
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "Poznań JUG: Listening to the sounds of your application"

  1. 1. Listen to the sounds of your application
  2. 2. Maciej Biłas @maciejb softwaremill.com jbison.com !2
  3. 3. Presented at J-Day Lbn.sc Originally authored by: Maciej Biłas Krzysztof Ciesielski
  4. 4. Agenda • Monitoring, huh? • Introducing Graphite • Log analysis – the whys • Logstash architecture & use cases • Exploring logs with Kibana
  5. 5. Monitoring, huh?
  6. 6. source: codeascraft.com
  7. 7. Types of measurements Network Machine Application
  8. 8. source: codeascraft.com
  9. 9. measurement > prediction
  10. 10. measurement >> prediction
  11. 11. Our stack Server Server App … App Yammer Metrics Graphite Logstash
  12. 12. http://graphite/render? target=server.web1.load&height=800&width=600
  13. 13. Graphite
  14. 14. echo "local.random.diceroll 4 `date +%s`" | nc graphite-server.your.org 2003
  15. 15. successful.login.attempt 1 1384471287
 successful.login.attempt 1 1384471297 successful.login.attempt = 1
  16. 16. Yammer metrics private final Meter successfulLogins =
 metrics.meter(name(LoginHandler.class, "successful")); ! public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
  17. 17. Metrics Types: Reporters: • Gauges • STDOUT • Counters • CSV • Meters • SLF4J • Histograms • JMX • Timers • Graphite • Health Checks • Ganglia
  18. 18. Counters, meters… vs. Gauges
  19. 19. Aggregation app00.webservice.requestTime app01.webservice.requestTime app02.webservice.requestTime …
  20. 20. source: codeascraft.com
  21. 21. You can also check out • collectd https://collectd.org • StatsD https://github.com/etsy/statsd/ • Riemman http://riemann.io/ • Twitter’s Ostrich https://github.com/twitter/ostrich • Ganglia http://ganglia.sourceforge.net/ • Dashboards:
 http://shopify.github.io/dashing/
 https://github.com/obfuscurity/descartes
 https://github.com/obfuscurity/dusk
  22. 22. Log aggregation?
  23. 23. Event Sourcing Tracing and storing all the events Current app state: replaying the stream Great for data mining and analysis
  24. 24. Log aggregation • Less invasive way to build an “event stream” • Logs are data with plenty of value
  25. 25. Log aggregation Log source Log source Log source Central storage Query engine Web view
  26. 26. Logstash • Open source • Written in JRuby • Gathers logs from various inputs • Parses and extracts metadata • Writes to various outputs
  27. 27. Architecture
  28. 28. Inputs collectd graphite s3 drupal_dblog heroku snmptrap unix elasticsearch imap sqlite varnishlog eventlog irc sqs websocket exec log4j stdin wmi file lumberjack stomp xmpp ganglia pipe syslog zenoss gelf rabbitmq tcp zeromq gemfire redis twitter generator relp udp
  29. 29. Grok Thin layer on top of a regular expression 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds time 2013-11-10 19:41:25.321 thread main loglevel INFO source o.a.camel.impl.DefaultCamelContext message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
  30. 30. Grok 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds grok { patterns_dir => “./some-dir“ match => ["message", "%{TIMESTAMP_ISO8601:time} 
 [%{DATA:thread}] %{LOGLEVEL:loglevel}
 %{DATA:source} - % {MULTILINE_GREEDYDATA:message}"] }
  31. 31. Outputs boundary circonus cloudwatch csv datadog datadog_metric s elasticsearch elasticsearch_ht tp elasticsearch_ri ver email exec http null sqs file irc opentsdb statsd ganglia jira pagerduty stdout gelf juggernaut pipe stomp gemfire librato rabbitmq syslog google_bigquer y loggly redis tcp lumberjack riak udp google_cloud_st orage metriccatcher riemann websocket graphite mongodb s3 xmpp graphtastic nagios sns zabbix hipchat nagios_nsca solr_http zeromq
  32. 32. Kibana
  33. 33. Checking app activity
  34. 34. Spotting anomalies
  35. 35. Regular events
  36. 36. How often is a feature used?
  37. 37. Plotting multiple event types
  38. 38. Other tools • Splunk • Graylog2 • Fluentd • log.io
  39. 39. References • Graphite
 http://graphite.readthedocs.org/en/latest/ • Yammer Metrics
 http://metrics.codahale.com/ • Logstash
 http://logstash.net/ • Surfing the event stream by Sam Newman at Geecon
 http://www.slideshare.net/spnewman/surfing-the-event-stream • Lessons from Building and Scaling LinkedIn by Jay Kreps
 http://www.infoq.com/presentations/linkedin-architecture-stack • Code as Craft
 http://codeascraft.com/
  40. 40. Thank you! Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×