1. Listen to the sounds of your application

2. Krzysztof Ciesielski
softwaremill.com
@kpciesielski
Lublin Software Craftsmen
!2

3. Maciej Biłas
softwaremill.com
@maciejb
jbison.com
!3

4. Agenda
• Monitoring, huh?
• Introducing Graphite
• Log analysis – the whys
• Logstash architecture & use cases
• Exploring logs with Kibana
!4

5. Monitoring, huh?
!5

6. source: codeascraft.com
!6

7. Types of measurements
!7

8. Types of measurements
Network
!7

9. Types of measurements
Network
Machine
!7

10. Types of measurements
Network
Machine
Application
!7

11. source: codeascraft.com
!8

12. measurement > prediction
!9

13. !10

14. measurement >> prediction
!11

15. Our stack
!12

16. Our stack
Server
App
Server
…
App
!12

17. Our stack
Server
App
Graphite
Server
…
App
Logstash
!12

18. Our stack
Server
App
Graphite
Server
…
App
Logstash
!12

19. Our stack
Server
App
Graphite
Server
…
App
Logstash
!12

20. Our stack
Server
App
Graphite
Server
…
App
Logstash
!12

21. Our stack
Server
App
Graphite
Server
…
App
Logstash
!12

22. Our stack
Server
App
Server
…
App
Yammer Metrics
Graphite
Logstash
!12

23. Introducing Graphite
!13

24. Graphite
!14

25. PORT=2003
SERVER=graphite.your.org
echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT};
!15

26. successful.login.attempt 1 1384471287
successful.login.attempt 1 1384471297
!16

27. successful.login.attempt 1 1384471287
successful.login.attempt 1 1384471297
=> successful.login.attempt = 1
!16

28. Yammer Metrics
!17

29. Yammer Metrics
private final Meter successfulLogins =
metrics.meter(name(LoginHandler.class, "successful"));
!
public void login(String user, String password) {
if (canLogin(user,password)) {
successfulLogins.mark();
// ...
} else {
// ...
}
}
!17

30. Metrics
!18

31. Metrics
Types:
!18

32. Metrics
Types:
• Gauges
!18

33. Metrics
Types:
• Gauges
• Counters
!18

34. Metrics
Types:
• Gauges
• Counters
• Meters
!18

35. Metrics
Types:
• Gauges
• Counters
• Meters
• Histograms
!18

36. Metrics
Types:
• Gauges
• Counters
• Meters
• Histograms
• Timers
!18

37. Metrics
Types:
• Gauges
• Counters
• Meters
• Histograms
• Timers
• Health Checks
!18

38. Metrics
Types:
Reporters:
• Gauges
• STDOUT
• Counters
• CSV
• Meters
• SLF4J
• Histograms
• JMX
• Timers
• Graphite
• Health Checks
• Ganglia
!18

39. Counters, meters…
vs.
Gauges
!19

40. Aggregation
!20

41. source: codeascraft.com
!21

42. You can also check out
• collectd https://collectd.org
• StatsD https://github.com/etsy/statsd/
• Riemman http://riemann.io/
• Twitter’s Ostrich https://github.com/twitter/ostrich
• Ganglia http://ganglia.sourceforge.net/
• Dashboards:
http://shopify.github.io/dashing/
https://github.com/obfuscurity/descartes
https://github.com/obfuscurity/dusk
!22

43. Event Sourcing
Tracing and storing all the events
Current app state: replaying the stream
Great for data mining and analysis
!23

44. Log aggregation
Less invasive way to build an “event stream”
Logs are data with plenty of value
!24

45. Log aggregation
Log
source
Log
source
Central
storage
Query
engine
Web
view
Log
source
!25

46. Logstash
Open Source (written in JRuby)
!26

47. Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
!26

48. Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata
!26

49. Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata
Writes to various outputs
!26

50. Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata
Writes to various outputs
Similar tools:
!
Splunk, Graylog, Fluentd, log.io
!26

51. Logstash
centralized architecture
!27

52. Logstash
grok
2013-11-10 19:41:25.321 [main]
INFO o.a.camel.impl.DefaultCamelContext
!
- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
!28

53. Logstash
grok
2013-11-10 19:41:25.321 [main]
INFO o.a.camel.impl.DefaultCamelContext
!
- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
time
2013-11-10 19:41:25.321
thread
main
loglevel
INFO
source
o.a.camel.impl.DefaultCamelContext
message
Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
!28

54. Logstash
simple use cases
Grok:
Exceptions
Security issues
Rare and business-meaningful events
Output: e-mail alert
!29

55. Kibana
!30

56. Kibana
checking general app activity
!31

57. Kibana
Spotting anomalies
!32

58. Kibana
Checking regular events
!33

59. Kibana
How often is this feature used?
!34

60. Kibana
Displaying multiple query results
Query: exception
Type: applog
Query: index
Type: mongolog
!35

61. References
• Graphite http://graphite.readthedocs.org/en/latest/
• Yammer Metrics http://metrics.codahale.com/
• Logstash http://logstash.net/
• Surfing the event stream by Sam Newman at Geecon
http://www.slideshare.net/spnewman/surfing-the-event-stream
• Lessons from Building and Scaling LinkedIn by Jay Kreps
http://www.infoq.com/presentations/linkedin-architecture-stack
• Code as Craft http://codeascraft.com/
!36

62. !37

63. Thank You
Reach us at:
@maciejb and @kpciesielski
softwaremill.com
!38