The document discusses centralized log management using Logstash and Kibana, addressing various problems and solutions related to log transport formats, architectures, and best practices. Key components include a log sender, router, collector, and a full-text search engine, with details on transport formats like Syslog and JSON. It emphasizes practices such as processing logs on the sender side, maintaining log systems, and utilizing Elasticsearch and Kibana for searching and analyzing logs.

1. Centralized log management based on
Logstash and Kibana - case study
Dariusz Eliasz
20.05.2014 Atmosphere Conference

2. • What’s the problem ?
• Solutons
• Transport format
• Architecture
– Sender
– Log router
– Log collector
– Full text search engine
– GUI
• Use case
Agenda

3. What’s the problem ??

6. Solutons ??

11. Transport format - syslog
• RFC3164 (BSD syslog )
• limited size - 1kB
• format of a syslog message:
– PRIORITY (calculated from severity and facility)
– HEADER (tmestamp + hostname or IP)
– MSG (tag + content)
<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

12. Transport format - json
• JavaScript Object Notaton
• lightweight text-data interchange format
• language independent
• self-describing

13. Transport format - json
{
"LogType": "access_log",
"Vhost": "atmosphere-conference.com",
"HtpsOn": "false",
"Xrealip": "1.2.3.5",
"Clientp": "91.17.13.28",
"UserAgent": "Mozilla/4.0 (compatble; MSIE 6.0; Windows NT 5.1)",
}

14. Architecture

15. Sender

16. Sender
• nxlog (htp://nxlog-ce.sourceforge.net/)
• multple input types:
– tcp socket
– udp socket
– fle input
– unx socket
• multple parser types:
– bsd syslog
– json

17. Sender
Good practce:
• make as much as possible processing on sender site, eg: apache access logs in
json format
• automate confguraton management

18. Log router

19. Log router
nxlog
nxlog logstash
logstash
redis
redissyslog-ng
syslog-ng redis

20. Log router
### JSON PARSER
parser p_json {
json-parser (prefx("_json."));
};
### FILTERS
flter f_someflter {
("${_json.SomeJsonField}” == ”abc.com”)
};
### INPUTS & OUTPUTS
### LOG PATHS
log {
source(s_network_json);
destnaton(d_udp_logstash);
destnaton(d_tcp_hadoop);
#fags(fow-control); # disabled to separate destnatons
};

21. Log router
Good practce:
• good separaton of destnatons
• calculate enough redis size – it’s yor bufer
• batch events writes to redis

22. Log collector

23. Log collector
• Logstash htp://logstash.net/
• collectng, parsing and storing logs tool
• plugins:
Inputs
• fle
• gelf
• tcp
• log4j
• redis
• varnishlog
Codecs
• json
• line
• msgpack
• netlow
• multline
Filters
• grok
• alter
• cidr
• geoip
• grep
• mutate
Outputs
• elastcsearch
• graphite
• jira
• tcp
• zeromq
• zabbix

24. Log collector
Good practce:
• keep up2date version of java & logstash
• use batch & multthread read from redis
• read logs 
• bulk writes to elastcsearch

25. Full text search engine
• Elastcsearch htp://www.elastcsearch.org/
• distributed, real-tme search and analytcs engine
• store documents as a JSON
• high availability
• schema free
• index mult-tenancy
• on top of Lucene

26. Full text search engine
• every index is replicated
• every index sharded
• index parttoning – tme based
• data retenton – tme based

27. Full text search engine
Good practce:
• half memory for ES (<30GB), half for system cache
• bootstrap.mlockall: true
• gateway.recover_afer_nodes
• indices.felddata.cache.size
• authorizaton via proxy
• curator
• Marvel plugin

28. GUI
• Kibana 3 htp://www.elastcsearch.org/overview/kibana/
• search, graph & analyze logs
• JavaScript based (AngularJS)
• only simple htp server needed

30. Q&A

31. Sources
Images:
htp://www.datalife7.com/2014/01/einsteins-secret-to-problem-solving-and.html
htp://www.formengifs.com/victorinox-swiss-army-swiss-champ-multtool-knife/
htp://www.slashgear.com/google-data-center-hd-photos-hit-where-the-internet-lives-gallery-17252451/