Download as KEY, PPTX
Uploaded byTomas Doran
London devops logging
Tomas Doran presented on their implementation of Logstash at TIM Group to process over 55 million messages per day. Their applications are all Java/Scala/Clojure and they developed their own library to send structured log events as JSON to Logstash using ZeroMQ for reliability. They index data in Elasticsearch and use it for metrics, alerts and dashboards but face challenges with data growth.
More Related Content
![8th grade founding father project[1]](https://cdn.slidesharecdn.com/ss_thumbnails/8thgradefoundingfatherproject1-111129094456-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
London devops logging
- 1. Practical logstash - beyond the basics. Tomas Doran (t0m) <bobtfish@bobtfish.net>
- 2. Who are you •Sysadmin at TIM Group • t0m on irc.freenode.net • twitter.com/bobtfish • github.com/bobtfish • slideshare.com/bobtfish
- 3.
- 4. Logstash • I hopeyou already know what logstash is?
- 5. Logstash • I hopeyou already know what logstash is? • I’m going to talk about our implementation.
- 6. Logstash • I hopeyou already know what logstash is? • I’m going to talk about our implementation. • Elasticsearch
- 7. Logstash • I hopeyou already know what logstash is? • I’m going to talk about our implementation. • Elasticsearch • Metrics
- 8. Logstash • I hopeyou already know what logstash is? • I’m going to talk about our implementation. • Elasticsearch • Metrics • Nagios
- 9. Logstash • I hopeyou already know what logstash is? • I’m going to talk about our implementation. • Elasticsearch • Metrics • Nagios • Riemann
- 11. > 55 millionmessages a day
- 12. > 55 millionmessages a day • Now ~30Gb of indexed data per day • All our applications • All of syslog • Used by developers and product managers • 2 x DL360s with 8x600Gb discs, also graphite install
- 13.
- 14. About 4 monthsold • Almost all apps onboard to various levels • All of syslog was easy • Still haven’t done apache logs • Haven’t comprehensively done router/ switches • Lots of apps still emit directly to graphite
- 15.
- 16. Java • All ourapps are Java / Scala / Clojure
- 17. Java • All ourapps are Java / Scala / Clojure • https://github.com/tlrx/slf4j-logback-zeromq
- 18. Java • All ourapps are Java / Scala / Clojure • https://github.com/tlrx/slf4j-logback-zeromq • Own layer (x2 1 Java, 1 Scala) for sending structured events as JSON
- 19. Java • All ourapps are Java / Scala / Clojure • https://github.com/tlrx/slf4j-logback-zeromq • Own layer (x2 1 Java, 1 Scala) for sending structured events as JSON • Java developers hate native code
- 20. On host logcollector
- 21. On host logcollector • Need a lightweight log shipper. • VMs with 1Gb of RAM.. • Message::Passing - perl library I wrote. • Small, light, pluggable
- 22. On host logcollector
- 23. On host logcollector • Application to logcollector is ZMQ • Small amount of buffering (1000 messages)
- 24. On host logcollector • Application to logcollector is ZMQ • Small amount of buffering (1000 messages) • logcollector to logstash is ZMQ • Large amount of buffering (disc offload, 100s of thousands of messages)
- 25. ZeroMQ has the correctsemantics
- 26. ZeroMQ has the correct semantics • Pub/Sub sockets
- 27. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking
- 28. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed)
- 29. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed) • Buffer sizes / locations configureable
- 30. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed) • Buffer sizes / locations configureable • Arbitrary message size
- 31. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed) • Buffer sizes / locations configureable • Arbitrary message size • IO done in a background thread (nice in interpreted languages - ruby/perl/python)
- 32.
- 33. What, no AMQP? •Could go logcollector => AMQP => logstash for extra durability
- 34. What, no AMQP? •Could go logcollector => AMQP => logstash for extra durability • ZMQ buffering ‘good enough’
- 35. What, no AMQP? •Could go logcollector => AMQP => logstash for extra durability • ZMQ buffering ‘good enough’ • logstash uses a pure ruby AMQP decoder
- 36. What, no AMQP? •Could go logcollector => AMQP => logstash for extra durability • ZMQ buffering ‘good enough’ • logstash uses a pure ruby AMQP decoder • Slooooowwwwww
- 37.
- 38. Reliability • Multiple Elasticsearchservers (obvious)!
- 39. Reliability • Multiple Elasticsearchservers (obvious)! • Due to ZMQ buffering, you can: • restart logstash, messages just buffer on hosts whilst it’s unavailable • restart logcollector, messages from apps buffer (lose some syslog)
- 40.
- 41. Reliability: TODO • Elasticsearchcluster getting sick happens
- 42. Reliability: TODO • Elasticsearchcluster getting sick happens • In-flight messages in logstash lost :(
- 43. Reliability: TODO • Elasticsearchcluster getting sick happens • In-flight messages in logstash lost :( • Solution - elasticsearch_river output • logstash => durable RabbitMQ queue • ES reads from queue • Also faster - uses bulk API
- 44.
- 45. Redundancy • Add aUUID to each message at emission point.
- 46. Redundancy • Add aUUID to each message at emission point. • Index in elasticsearch by UUID
- 47. Redundancy • Add aUUID to each message at emission point. • Index in elasticsearch by UUID • Emit to two backend logstash instances (TODO)
- 48. Redundancy • Add aUUID to each message at emission point. • Index in elasticsearch by UUID • Emit to two backend logstash instances (TODO) • Index everything twice! (TODO)
- 49. Elasticsearch optimisation • You need a template • compress source • disable _all • discard unwanted fields from source / indexing • tweak shards and replicas • compact your yesterday’s index at end of day!
- 51.
- 52. Elasticsearch size • 87daily indexes
- 53. Elasticsearch size • 87daily indexes • 800Gb of data (per instance)
- 54. Elasticsearch size • 87daily indexes • 800Gb of data (per instance) • Just bumped ES heap to 22G • Just writing data - 2Gb • Query over all indexes - 17Gb!
- 55. Elasticsearch size • 87daily indexes • 800Gb of data (per instance) • Just bumped ES heap to 22G • Just writing data - 2Gb • Query over all indexes - 17Gb! • Hang on - 800/87 does not = 33Gb/day!
- 56. Rate has increased! Text Text We may have problems fitting onto 5 x 600Gb discs!
- 57.
- 58.
- 59.
- 60. TimedWebRequest • Most obviousexample of a standard event • App name • Environment • HTTP status • Page generation time • Request / Response size
- 61. TimedWebRequest • Most obviousexample of a standard event • App name • Environment • HTTP status • Page generation time • Request / Response size • Can derive loads of metrics from this!
- 63.
- 64. statsd • Rolls upcounters and timers into metrics
- 65. statsd • Rolls upcounters and timers into metrics • One bucket per stat, emits values every 10 seconds
- 66. statsd • Rolls upcounters and timers into metrics • One bucket per stat, emits values every 10 seconds • Counters: Request rate, HTTP status rate
- 67. statsd • Rolls upcounters and timers into metrics • One bucket per stat, emits values every 10 seconds • Counters: Request rate, HTTP status rate • Timers: Total page time, mean page time, min/max page times
- 68.
- 69.
- 70.
- 71. JSON everywhere • Legacyshell ftp mirror scripts • gitolite hooks for deployments • keepalived health checks
- 72.
- 73.
- 74. Alerting use cases: •Replaced nsca client with standardised log pipeline • Developers log an event and get (one!) email warning of client side exceptions • Passive health monitoring - ‘did we log something recently’
- 75.
- 76. Riemann • Using forsome simple health checking
- 77. Riemann • Using forsome simple health checking • logcollector health
- 78. Riemann • Using forsome simple health checking • logcollector health • Load balancer instance health
- 79.
- 80. Riemann • Ambitious plansto do more
- 81. Riemann • Ambitious plansto do more • Web pool health (>= n nodes)
- 82. Riemann • Ambitious plansto do more • Web pool health (>= n nodes) • Replace statsd
- 83. Riemann • Ambitious plansto do more • Web pool health (>= n nodes) • Replace statsd • Transit collectd data via logstash and use to emit to graphite
- 84. Riemann • Ambitious plansto do more • Web pool health (>= n nodes) • Replace statsd • Transit collectd data via logstash and use to emit to graphite • disc usage trending / prediction
- 85.
- 86. Metadata • It’s allabout the metadata
- 87. Metadata • It’s allabout the metadata • Structured events are describable
- 88. Metadata • It’s allabout the metadata • Structured events are describable • Common patterns to give standard metrics / alerting for free
- 89. Metadata • It’s allabout the metadata • Structured events are describable • Common patterns to give standard metrics / alerting for free • Dashboards!
- 90.
- 91.
- 92. Dashboard love/hate • Riemannx 2 • Graphite dashboards x 2
- 93. Dashboard love/hate • Riemannx 2 • Graphite dashboards x 2 • Nagios x 3
- 94. Dashboard love/hate • Riemannx 2 • Graphite dashboards x 2 • Nagios x 3 • CI radiator
- 95. Dashboard love/hate • Riemannx 2 • Graphite dashboards x 2 • Nagios x 3 • CI radiator
- 96. Dashboard love/hate • Riemannx 2 • Graphite dashboards x 2 • Nagios x 3 • CI radiator • Information overload!
- 97. Thanks! • Questions? • slideswith more detail about my log collector code: • http://slideshare.net/bobtfish/
Editor's Notes
- #2 \n
- #3 \n
- #4 \n
- #5 \n
- #6 \n
- #7 \n
- #8 \n
- #9 \n
- #10 \n
- #11 \n
- #12 \n
- #13 \n
- #14 \n
- #15 \n
- #16 \n
- #17 \n
- #18 \n
- #19 \n
- #20 The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
- #21 The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
- #22 The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
- #23 The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
- #24 The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
- #25 The last point here is most important - ZMQ networking works entirely in a background thread perl knows nothing about, which means that you can asynchronously ship messages with no changes to your existing codebase.\n
- #26 \n
- #27 \n
- #28 \n
- #29 \n
- #30 \n
- #31 \n
- #32 \n
- #33 \n
- #34 \n
- #35 \n
- #36 \n
- #37 \n
- #38 \n
- #39 \n
- #40 \n
- #41 \n
- #42 \n
- #43 \n
- #44 \n
- #45 \n
- #46 \n
- #47 \n
- #48 \n
- #49 \n
- #50 \n
- #51 \n
- #52 \n
- #53 \n
- #54 \n
- #55 \n
- #56 \n
- #57 \n
- #58 \n
- #59 \n
- #60 \n
- #61 \n
- #62 \n
- #63 \n
- #64 \n
- #65 \n
- #66 \n
- #67 \n
- #68 \n
- #69 \n
- #70 \n
- #71 \n
- #72 \n
- #73 \n
- #74 \n
- #75 \n
- #76 \n
- #77 \n
- #78 \n
- #79 \n
- #80 \n