Surfing the event stream
•
7 likes•2,276 views
As presented at GeeCon 2013. We have lots of information available about our systems. CPU, disk IO, orders placed, error rates, users logged in. But typically all these pieces of information are collected, aggregated and stored in very different ways making correlation difficult and increasing the operational overhead of our systems. What if we could treat all of this information as events? What if we could aggregate, store, and report on all of this information as a uniform event stream? This talk will look at emerging trends in the space of log aggregation, monitoring and event streaming to paint a picture for how you too can start to make real use of the information already available to you using nothing more complex than some free, off the shelf Open Source software.
Recommended
Recommended
More Related Content
More from Sam Newman
More from Sam Newman (19)
Recently uploaded
Recently uploaded (20)
Surfing the event stream
- 1. @samnewman#geecon Surfing The Event Stream Sam Newman ThoughtWorks Sunday, 21 July 13
- 8. @samnewman#geecon Operational Data CPU Disk IO Memory Use Threads Sunday, 21 July 13
- 9. @samnewman#geecon Collection & Display • sar • syslog • collectd • syslog-ng • nagios • ganglia Sunday, 21 July 13
- 14. @samnewman#geecon Business Data Sunday, 21 July 13
- 15. @samnewman#geecon Business Data Orders Placed Sunday, 21 July 13
- 17. @samnewman#geecon Business Data Orders Placed Revenue Fraud Cases Sunday, 21 July 13
- 18. @samnewman#geecon Business Data Orders Placed Bounce Rate Revenue Fraud Cases Sunday, 21 July 13
- 19. @samnewman#geecon How did we handle them? • Google Analytics • Data Warehouse Systems • Log files! Sunday, 21 July 13
- 20. @samnewman#geecon Something Happened! Sunday, 21 July 13
- 21. @samnewman#geecon Something Happened! What Should We Do? Sunday, 21 July 13
- 22. @samnewman#geecon Something Happened! What Should We Do? Sunday, 21 July 13
- 23. @samnewman#geecon Something Happened! What Should We Do? Sunday, 21 July 13
- 25. @samnewman#geecon http://blog.jgc.org/2006/05/what-slashdot-effect-looks-like.html Sunday, 21 July 13
- 28. @samnewman#geecon Fast And Easy... Sunday, 21 July 13
- 29. @samnewman#geecon Fast And Easy... At Scale Sunday, 21 July 13
- 30. @samnewman#geecon Aggregation Is Key Sunday, 21 July 13
- 31. @samnewman#geecon Mark McGranaghan: "Logs as Data" http://blip.tv/clojure/mark-mcgranaghan-logs-as-data-5953857 Sunday, 21 July 13
- 32. @samnewman#geecon Paul Ingles: "Users as Data" http://vimeo.com/45136211 Sunday, 21 July 13
- 33. @samnewman#geecon Log Stash + Graylog2 Sunday, 21 July 13
- 34. @samnewman#geecon Log Stash + Graylog2 Sunday, 21 July 13
- 35. @samnewman#geecon Log Stash + Graylog2 Sunday, 21 July 13
- 36. @samnewman#geecon Log Stash + Graylog2 Sunday, 21 July 13
- 41. @samnewman#geecon www01.cpuUsage 42 1286269200 Sunday, 21 July 13
- 57. @samnewman#geecon www01.cpuUsage 42 1286269200 Sunday, 21 July 13
- 58. @samnewman#geecon orderplaced 1 1286269200 Sunday, 21 July 13
- 59. @samnewman#geecon orderplaced 1 1286269200 orderplaced 1 1286269200 Sunday, 21 July 13
- 60. @samnewman#geecon orderplaced 1 1286269200 orderplaced 1 1286269200 orderplaced = 1 Sunday, 21 July 13
- 73. @samnewman#geecon (service "api req") (percentiles 5 [0.5 0.95 0.99] index)) Sunday, 21 July 13
- 74. @samnewman#geecon (service "api req") (percentiles 5 [0.5 0.95 0.99] index)) Sunday, 21 July 13
- 75. @samnewman#geecon (def tell-ops (rollup 5 3600 (email "ops@vonbraun.mil"))) (streams (where (state "critical") tell-ops)) Sunday, 21 July 13
- 76. @samnewman#geecon (let[client (tcp-client :host "aggregator")] (by [:host :service] (changed :state (forward client)))) Sunday, 21 July 13
- 79. @samnewman#geecon Riemann Server Client Client Riemann Server Client Client Riemann Server Sunday, 21 July 13
- 80. @samnewman#geecon So What Do We Have? Sunday, 21 July 13
- 85. @samnewman#geecon Server Server Graphite Graylog 2Dashboard A Dashboard B Dashboard C Server Sunday, 21 July 13
- 86. @samnewman#geecon Server Server StatsD/ Riemann Graylog 2 Graphite Dashboard A Dashboard B Dashboard C Sunday, 21 July 13
- 95. @samnewman#geecon Realtime Aggregator Data is lost! Sunday, 21 July 13
- 96. @samnewman#geecon Realtime Aggregator Data is lost! Sunday, 21 July 13
- 102. @samnewman#geecon Realtime Aggregator Lossless Event Store Hadoop HBase Cassandra Sunday, 21 July 13
- 105. @samnewman#geecon Event Sourcing Sunday, 21 July 13
- 109. @samnewman#geecon Can I have one view? Lossless Event Store Realtime Aggregator Sunday, 21 July 13
- 113. @samnewman#geecon Lossless Event Store Realtime Aggregator Up to date, but only for a small window Sunday, 21 July 13
- 114. @samnewman#geecon Lossless Event Store Realtime Aggregator Consistent, but out of date Up to date, but only for a small window Sunday, 21 July 13
- 115. @samnewman#geecon Lossless Event Store Realtime Aggregator Unified Query Consistent, but out of date Up to date, but only for a small window Sunday, 21 July 13
- 116. @samnewman#geecon Lossless Event Store Realtime Aggregator Lambda Architecture Unified Query Consistent, but out of date Up to date, but only for a small window Sunday, 21 July 13
- 117. @samnewman#geecon The Future? Sunday, 21 July 13
- 119. @samnewman#geecon Server Server Aggregating Relay Graphite Graylog 2 Hadoop Unified Query Sunday, 21 July 13
- 121. @samnewman#geecon AllYour Data Sunday, 21 July 13
- 122. @samnewman#geecon AllYour Data In Realtime Sunday, 21 July 13
- 123. @samnewman#geecon AllYour Data In Realtime Sunday, 21 July 13
- 125. @samnewman#geecon Find and free your data Sunday, 21 July 13
- 126. @samnewman#geecon Find and free your data Start simple Sunday, 21 July 13
- 127. @samnewman#geecon Find and free your data Start simple Create different views for different stakeholders Sunday, 21 July 13
- 128. @samnewman#geecon Find and free your data Start simple Create different views for different stakeholders Don’t be scared of real-time! Sunday, 21 July 13