On Centralizing Logs

14,748 views

Published on

On Centralizing Logs with Syslog, LogStash, Elasticsearch, Kibana. Presentation from Radu Gheorghe from Sematext at Monitorama EU 2013.

Published in: Technology
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
14,748
On SlideShare
0
From Embeds
0
Number of Embeds
11,910
Actions
Shares
0
Downloads
88
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide

On Centralizing Logs

  1. 1. On Centralizing Logs Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext
  2. 2. Hello World! Logsene mlmoneu13cf for -44%
  3. 3. app app app app files files
  4. 4. app app app app files files Elasticsearchlogstash Kibana
  5. 5. Elasticsearch Reason #1: Quick Search No indexing But... =>
  6. 6. ...and other reasons good write speed lots of tools for logging scales easily
  7. 7. Production Tips stability performance
  8. 8. Stability 1/4: Discovery multicast unicast vs cluster name list of nodes + plugins: EC2, GCE
  9. 9. Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1
  10. 10. Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor the requirements SPM for Elasticsearch 20% off with MONEU2013
  11. 11. Stability 4/4: Field Cache can be changed to index.cache.field.type: soft indices.fielddata.cache.size: X%
  12. 12. Performance 1/4: Bulk Processing use Bulk API or Bulk UDP API ...translog.flush_threshold_ops
  13. 13. Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but every 5s +25% indexing* every 30s +70% indexing*
  14. 14. Performance 3/4: Timed Indices
  15. 15. Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
  16. 16. Setting Up Kibana as Frontend servers you
  17. 17. Kibana: Search
  18. 18. Kibana: Visualize
  19. 19. Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
  20. 20. X-ray of a Modern Syslog Daemon read+buffer file /dev/log … parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
  21. 21. 2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world TCP + LF = no year, ms, nor TZ little structure
  22. 22. 2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
  23. 23. Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize 2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
  24. 24. Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS RELP + TLS
  25. 25. Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify) outputs (+codecs) lots of plugins => lots of options
  26. 26. Logstash: Example Lumberjack Logstash Elasticsearch
  27. 27. Logstash: Add Buffer Lumberjack Lumberjack
  28. 28. Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack
  29. 29. Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd
  30. 30. Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene
  31. 31. (More) Alternatives files syslog
  32. 32. Alternatives Can Mix files syslog Logstash Elasticsearch Kibana
  33. 33. Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext
  34. 34. rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance nicer config format omelasticsearch
  35. 35. rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp
  36. 36. rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
  37. 37. rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific queue.dequeuebatchsize="1000" queue.workerthreads="3"
  38. 38. Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext

×