Poznań JUG: Listening to the sounds of your application

1,145 views

Published on

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,145
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
13
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Poznań JUG: Listening to the sounds of your application

  1. 1. Listen to the sounds of your application
  2. 2. Maciej Biłas @maciejb softwaremill.com jbison.com !2
  3. 3. Presented at J-Day Lbn.sc Originally authored by: Maciej Biłas Krzysztof Ciesielski
  4. 4. Agenda • Monitoring, huh? • Introducing Graphite • Log analysis – the whys • Logstash architecture & use cases • Exploring logs with Kibana
  5. 5. Monitoring, huh?
  6. 6. source: codeascraft.com
  7. 7. Types of measurements Network Machine Application
  8. 8. source: codeascraft.com
  9. 9. measurement > prediction
  10. 10. measurement >> prediction
  11. 11. Our stack Server Server App … App Yammer Metrics Graphite Logstash
  12. 12. http://graphite/render? target=server.web1.load&height=800&width=600
  13. 13. Graphite
  14. 14. echo "local.random.diceroll 4 `date +%s`" | nc graphite-server.your.org 2003
  15. 15. successful.login.attempt 1 1384471287
 successful.login.attempt 1 1384471297 successful.login.attempt = 1
  16. 16. Yammer metrics private final Meter successfulLogins =
 metrics.meter(name(LoginHandler.class, "successful")); ! public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
  17. 17. Metrics Types: Reporters: • Gauges • STDOUT • Counters • CSV • Meters • SLF4J • Histograms • JMX • Timers • Graphite • Health Checks • Ganglia
  18. 18. Counters, meters… vs. Gauges
  19. 19. Aggregation app00.webservice.requestTime app01.webservice.requestTime app02.webservice.requestTime …
  20. 20. source: codeascraft.com
  21. 21. You can also check out • collectd https://collectd.org • StatsD https://github.com/etsy/statsd/ • Riemman http://riemann.io/ • Twitter’s Ostrich https://github.com/twitter/ostrich • Ganglia http://ganglia.sourceforge.net/ • Dashboards:
 http://shopify.github.io/dashing/
 https://github.com/obfuscurity/descartes
 https://github.com/obfuscurity/dusk
  22. 22. Log aggregation?
  23. 23. Event Sourcing Tracing and storing all the events Current app state: replaying the stream Great for data mining and analysis
  24. 24. Log aggregation • Less invasive way to build an “event stream” • Logs are data with plenty of value
  25. 25. Log aggregation Log source Log source Log source Central storage Query engine Web view
  26. 26. Logstash • Open source • Written in JRuby • Gathers logs from various inputs • Parses and extracts metadata • Writes to various outputs
  27. 27. Architecture
  28. 28. Inputs collectd graphite s3 drupal_dblog heroku snmptrap unix elasticsearch imap sqlite varnishlog eventlog irc sqs websocket exec log4j stdin wmi file lumberjack stomp xmpp ganglia pipe syslog zenoss gelf rabbitmq tcp zeromq gemfire redis twitter generator relp udp
  29. 29. Grok Thin layer on top of a regular expression 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds time 2013-11-10 19:41:25.321 thread main loglevel INFO source o.a.camel.impl.DefaultCamelContext message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
  30. 30. Grok 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds grok { patterns_dir => “./some-dir“ match => ["message", "%{TIMESTAMP_ISO8601:time} 
 [%{DATA:thread}] %{LOGLEVEL:loglevel}
 %{DATA:source} - % {MULTILINE_GREEDYDATA:message}"] }
  31. 31. Outputs boundary circonus cloudwatch csv datadog datadog_metric s elasticsearch elasticsearch_ht tp elasticsearch_ri ver email exec http null sqs file irc opentsdb statsd ganglia jira pagerduty stdout gelf juggernaut pipe stomp gemfire librato rabbitmq syslog google_bigquer y loggly redis tcp lumberjack riak udp google_cloud_st orage metriccatcher riemann websocket graphite mongodb s3 xmpp graphtastic nagios sns zabbix hipchat nagios_nsca solr_http zeromq
  32. 32. Kibana
  33. 33. Checking app activity
  34. 34. Spotting anomalies
  35. 35. Regular events
  36. 36. How often is a feature used?
  37. 37. Plotting multiple event types
  38. 38. Other tools • Splunk • Graylog2 • Fluentd • log.io
  39. 39. References • Graphite
 http://graphite.readthedocs.org/en/latest/ • Yammer Metrics
 http://metrics.codahale.com/ • Logstash
 http://logstash.net/ • Surfing the event stream by Sam Newman at Geecon
 http://www.slideshare.net/spnewman/surfing-the-event-stream • Lessons from Building and Scaling LinkedIn by Jay Kreps
 http://www.infoq.com/presentations/linkedin-architecture-stack • Code as Craft
 http://codeascraft.com/
  40. 40. Thank you! Questions?

×