Listen to the sounds of your application
Krzysztof Ciesielski
softwaremill.com
@kpciesielski

Lublin Software Craftsmen

!2
Maciej Biłas
softwaremill.com
@maciejb

jbison.com

!3
Agenda
• Monitoring, huh?
• Introducing Graphite
• Log analysis – the whys
• Logstash architecture & use cases
• Exploring...
Monitoring, huh?

!5
source: codeascraft.com
!6
Types of measurements

!7
Types of measurements

Network

!7
Types of measurements

Network
Machine

!7
Types of measurements

Network
Machine
Application

!7
source: codeascraft.com
!8
measurement > prediction

!9
!10
measurement >> prediction

!11
Our stack

!12
Our stack
Server
App

Server

…

App

!12
Our stack
Server
App

Graphite

Server

…

App

Logstash

!12
Our stack
Server
App

Graphite

Server

…

App

Logstash

!12
Our stack
Server
App

Graphite

Server

…

App

Logstash

!12
Our stack
Server
App

Graphite

Server

…

App

Logstash

!12
Our stack
Server
App

Graphite

Server

…

App

Logstash

!12
Our stack
Server
App

Server

…

App

Yammer Metrics

Graphite

Logstash

!12
Introducing Graphite

!13
Graphite

!14
PORT=2003
SERVER=graphite.your.org
echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT}; 

!15
successful.login.attempt 1 1384471287
successful.login.attempt 1 1384471297

!16
successful.login.attempt 1 1384471287
successful.login.attempt 1 1384471297

=> successful.login.attempt = 1

!16
Yammer Metrics

!17
Yammer Metrics

private final Meter successfulLogins =
metrics.meter(name(LoginHandler.class, "successful"));

!

public v...
Metrics

!18
Metrics
Types:

!18
Metrics
Types:
• Gauges

!18
Metrics
Types:
• Gauges
• Counters

!18
Metrics
Types:
• Gauges
• Counters
• Meters

!18
Metrics
Types:
• Gauges
• Counters
• Meters
• Histograms

!18
Metrics
Types:
• Gauges
• Counters
• Meters
• Histograms
• Timers

!18
Metrics
Types:
• Gauges
• Counters
• Meters
• Histograms
• Timers
• Health Checks

!18
Metrics
Types:

Reporters:

• Gauges

• STDOUT

• Counters

• CSV

• Meters

• SLF4J

• Histograms

• JMX

• Timers

• Gra...
Counters, meters…
vs.
Gauges

!19
Aggregation

!20
source: codeascraft.com
!21
You can also check out
• collectd https://collectd.org
• StatsD https://github.com/etsy/statsd/
• Riemman http://riemann.i...
Event Sourcing
Tracing and storing all the events
Current app state: replaying the stream
Great for data mining and analys...
Log aggregation

Less invasive way to build an “event stream”
Logs are data with plenty of value

!24
Log aggregation
Log
source

Log
source

Central
storage

Query
engine

Web
view

Log
source

!25
Logstash
Open Source (written in JRuby)

!26
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs

!26
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata

!26
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata
Writes to various ou...
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata
Writes to various ou...
Logstash
centralized architecture

!27
Logstash
grok
2013-11-10 19:41:25.321 [main]
INFO o.a.camel.impl.DefaultCamelContext

!

- Apache Camel 2.11.1 (CamelConte...
Logstash
grok
2013-11-10 19:41:25.321 [main]
INFO o.a.camel.impl.DefaultCamelContext

!

- Apache Camel 2.11.1 (CamelConte...
Logstash
simple use cases
Grok:
Exceptions
Security issues
Rare and business-meaningful events

Output: e-mail alert

!29
Kibana

!30
Kibana
checking general app activity

!31
Kibana
Spotting anomalies

!32
Kibana
Checking regular events

!33
Kibana
How often is this feature used?

!34
Kibana
Displaying multiple query results
Query: exception
Type: applog

Query: index
Type: mongolog

!35
References
• Graphite http://graphite.readthedocs.org/en/latest/
• Yammer Metrics http://metrics.codahale.com/
• Logstash ...
!37
Thank You

Reach us at:
@maciejb and @kpciesielski
softwaremill.com

!38
Upcoming SlideShare
Loading in …5
×

J-Day Kraków: Listen to the sounds of your application

1,960 views
1,807 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,960
On SlideShare
0
From Embeds
0
Number of Embeds
235
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

J-Day Kraków: Listen to the sounds of your application

  1. 1. Listen to the sounds of your application
  2. 2. Krzysztof Ciesielski softwaremill.com @kpciesielski Lublin Software Craftsmen !2
  3. 3. Maciej Biłas softwaremill.com @maciejb jbison.com !3
  4. 4. Agenda • Monitoring, huh? • Introducing Graphite • Log analysis – the whys • Logstash architecture & use cases • Exploring logs with Kibana !4
  5. 5. Monitoring, huh? !5
  6. 6. source: codeascraft.com !6
  7. 7. Types of measurements !7
  8. 8. Types of measurements Network !7
  9. 9. Types of measurements Network Machine !7
  10. 10. Types of measurements Network Machine Application !7
  11. 11. source: codeascraft.com !8
  12. 12. measurement > prediction !9
  13. 13. !10
  14. 14. measurement >> prediction !11
  15. 15. Our stack !12
  16. 16. Our stack Server App Server … App !12
  17. 17. Our stack Server App Graphite Server … App Logstash !12
  18. 18. Our stack Server App Graphite Server … App Logstash !12
  19. 19. Our stack Server App Graphite Server … App Logstash !12
  20. 20. Our stack Server App Graphite Server … App Logstash !12
  21. 21. Our stack Server App Graphite Server … App Logstash !12
  22. 22. Our stack Server App Server … App Yammer Metrics Graphite Logstash !12
  23. 23. Introducing Graphite !13
  24. 24. Graphite !14
  25. 25. PORT=2003 SERVER=graphite.your.org echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT};  !15
  26. 26. successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297 !16
  27. 27. successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297 => successful.login.attempt = 1 !16
  28. 28. Yammer Metrics !17
  29. 29. Yammer Metrics private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); ! public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } } !17
  30. 30. Metrics !18
  31. 31. Metrics Types: !18
  32. 32. Metrics Types: • Gauges !18
  33. 33. Metrics Types: • Gauges • Counters !18
  34. 34. Metrics Types: • Gauges • Counters • Meters !18
  35. 35. Metrics Types: • Gauges • Counters • Meters • Histograms !18
  36. 36. Metrics Types: • Gauges • Counters • Meters • Histograms • Timers !18
  37. 37. Metrics Types: • Gauges • Counters • Meters • Histograms • Timers • Health Checks !18
  38. 38. Metrics Types: Reporters: • Gauges • STDOUT • Counters • CSV • Meters • SLF4J • Histograms • JMX • Timers • Graphite • Health Checks • Ganglia !18
  39. 39. Counters, meters… vs. Gauges !19
  40. 40. Aggregation !20
  41. 41. source: codeascraft.com !21
  42. 42. You can also check out • collectd https://collectd.org • StatsD https://github.com/etsy/statsd/ • Riemman http://riemann.io/ • Twitter’s Ostrich https://github.com/twitter/ostrich • Ganglia http://ganglia.sourceforge.net/ • Dashboards:
 http://shopify.github.io/dashing/
 https://github.com/obfuscurity/descartes
 https://github.com/obfuscurity/dusk !22
  43. 43. Event Sourcing Tracing and storing all the events Current app state: replaying the stream Great for data mining and analysis !23
  44. 44. Log aggregation Less invasive way to build an “event stream” Logs are data with plenty of value !24
  45. 45. Log aggregation Log source Log source Central storage Query engine Web view Log source !25
  46. 46. Logstash Open Source (written in JRuby) !26
  47. 47. Logstash Open Source (written in JRuby) Gathers logs from various inputs !26
  48. 48. Logstash Open Source (written in JRuby) Gathers logs from various inputs Parses and extracts metadata !26
  49. 49. Logstash Open Source (written in JRuby) Gathers logs from various inputs Parses and extracts metadata Writes to various outputs !26
  50. 50. Logstash Open Source (written in JRuby) Gathers logs from various inputs Parses and extracts metadata Writes to various outputs Similar tools: ! Splunk, Graylog, Fluentd, log.io !26
  51. 51. Logstash centralized architecture !27
  52. 52. Logstash grok 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext ! - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds !28
  53. 53. Logstash grok 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext ! - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds time 2013-11-10 19:41:25.321 thread main loglevel INFO source o.a.camel.impl.DefaultCamelContext message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds !28
  54. 54. Logstash simple use cases Grok: Exceptions Security issues Rare and business-meaningful events Output: e-mail alert !29
  55. 55. Kibana !30
  56. 56. Kibana checking general app activity !31
  57. 57. Kibana Spotting anomalies !32
  58. 58. Kibana Checking regular events !33
  59. 59. Kibana How often is this feature used? !34
  60. 60. Kibana Displaying multiple query results Query: exception Type: applog Query: index Type: mongolog !35
  61. 61. References • Graphite http://graphite.readthedocs.org/en/latest/ • Yammer Metrics http://metrics.codahale.com/ • Logstash http://logstash.net/ • Surfing the event stream by Sam Newman at Geecon
 http://www.slideshare.net/spnewman/surfing-the-event-stream • Lessons from Building and Scaling LinkedIn by Jay Kreps
 http://www.infoq.com/presentations/linkedin-architecture-stack • Code as Craft http://codeascraft.com/ !36
  62. 62. !37
  63. 63. Thank You Reach us at: @maciejb and @kpciesielski softwaremill.com !38

×