On Centralizing Logs with Syslog, LogStash, Elasticsearch, Kibana. Presentation from Radu Gheorghe from Sematext at Monitorama EU 2013.

1. On Centralizing Logs
Radu Gheorghe
@radu0gheorghe
radu.gheorghe@sematext.com
@sematext

2. Hello World!
Logsene
mlmoneu13cf for -44%

3. app
app
app
app
files
files

4. app
app
app
app
files
files
Elasticsearchlogstash
Kibana

5. Elasticsearch Reason #1: Quick Search
No indexing
But...
=>

6. ...and other reasons
good write speed lots of tools for logging
scales easily

7. Production Tips
stability performance

8. Stability 1/4: Discovery
multicast unicast
vs
cluster name list of nodes
+ plugins: EC2, GCE

9. Stability 2/4: Preventing Split Brain
minimum_master_nodes = N/2 + 1

10. Stability 3/4: No OOMs, pls!
1GB
½ total RAM
Monitor the requirements
SPM for Elasticsearch
20% off with MONEU2013

11. Stability 4/4: Field Cache
can be changed to
index.cache.field.type: soft
indices.fielddata.cache.size: X%

12. Performance 1/4: Bulk Processing
use Bulk API
or Bulk UDP API
...translog.flush_threshold_ops

13. Performance 2/4: Refresh Interval
http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/
default:
every second => but
every 5s
+25% indexing*
every 30s
+70% indexing*

14. Performance 3/4: Timed Indices

15. Performance 4/4: Buffers
...index_buffer_size: 30%
(YMMV)
index.store.type: mmapfs
(on 64-bit machines)
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

16. Setting Up Kibana as Frontend
servers you

17. Kibana: Search

18. Kibana: Visualize

19. Meet Some Syslog Daemons
syslogd
traditional
everywhere
syslog-ng
OSE, PE
documentation++
config format++
rsyslog
OSS only
ES output*
* http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/

20. X-ray of a Modern Syslog Daemon
read+buffer
file
/dev/log
…
parse
syslog formats
JSON
unstructured data
assemble
conditionals
formatting
...
buffer+write
file
syslog
Elasticsearch
...

21. 2001's RFC3164: The Semi-Standard
<10>Oct 11 22:14:15 host program:hello world
TCP + LF =
no year, ms, nor TZ
little structure

22. 2009's RFC5424
<165>1 2003-10-11T22:14:15.003Z host program - - -
[origin ip="192.168.0.1"] hello world
[ structured=data ] octet-count* + LF =
* UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)

23. Teaching Old Dog New Tricks
RSYSLOG_ForwardFormat
(ISO8601 over RFC3164)
$MaxMessageSize 2048k
log_message_size(2097152)
@cee: {"message": "hello world"} @@(o)192.168.0.1
octet-counted framing

24. Reliable Transport? Encryption?
TCP + TLS (RFC5425)
RLTP + TLS RELP + TLS

25. Logstash: The Swiss Army Knife
inputs
(+codecs)
filters
(parse, modify)
outputs
(+codecs)
lots of plugins => lots of options

26. Logstash: Example
Lumberjack
Logstash Elasticsearch

27. Logstash: Add Buffer
Lumberjack
Lumberjack

28. Logstash: Scale Everything
Lumberjack
Lumberjack
Lumberjack
Lumberjack

29. Back to the Beginning
Lumberjack
Lumberjack
Lumberjack
Lumberjack
syslogd

30. Logsene
Lumberjack
Lumberjack
Lumberjack
Lumberjack
syslogd
Logsene
http://sematext.com/logsene

31. (More) Alternatives
files
syslog

32. Alternatives Can Mix
files
syslog
Logstash
Elasticsearch Kibana

33. Thank you!
Radu Gheorghe
@radu0gheorghe
radu.gheorghe@sematext.com
@sematext

35. rsyslog 1/4: Upgrade to 7.x
RPMs or DEBs better performance
nicer config format omelasticsearch

36. rsyslog 2/4: Faster Inputs
UDP
increase TimeRequery
TCP
use imptcp

37. rsyslog 3/4: Main Message Queue
$MainMsgQueueType FixedArray
$MainMsgQueueSize 1000000....
...or LinkedList or Disk
$...DequeueBatchSize 1000 $...WorkerThreads 3

38. rsyslog 4/4: Action Queue
queue.type="linkedlist"
queue.size="1000000"
bulkmode="on" # ES specific
queue.dequeuebatchsize="1000"
queue.workerthreads="3"

39. Thank you!
Radu Gheorghe
@radu0gheorghe
radu.gheorghe@sematext.com
@sematext