The document discusses the importance and utilization of logs within logging systems, specifically focusing on the ELK stack (Elasticsearch, Logstash, and Kibana). It highlights how logs, often seen as mere files, are essential time-oriented streams that provide real-time insights into system operations and issues. The document emphasizes the common difficulties in log management, including log formatting differences and the risks of neglecting log monitoring, while advocating for the use of the ELK stack to streamline log data processing and visualization.

1. ELK Stack
Because logs are not meant to go to /dev/null
Guillermo Carrasco
@guillemch

2. Logging & ELK stack

3. Logging & ELK stack
What are logs?
What are logs for? Theory vs reality
Logstash & Elasticsearch
Kibana

4. What are logs?
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

5. What are logs?
2014-09-19 13:08:37,972 INFO [MiSeqIntegrator] (SequencingIntegrationUtil)
Extracted raw content - [cycleNumber = 308 , runFolder = "D:IlluminaMiSeqTemp
140904_M01320_0130_000000000-A9NE9" , netFolder = "Z:140904_M01320_0130_000000000-A9NE9" ,
Record
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

6. What are logs?
2014-09-19 13:08:37,972 INFO [MiSeqIntegrator] (SequencingIntegrationUtil)
Extracted raw content - [cycleNumber = 308 , runFolder = "D:IlluminaMiSeqTemp
140904_M01320_0130_000000000-A9NE9" , netFolder = "Z:140904_M01320_0130_000000000-A9NE9" ,
Record
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

7. What are logs?
2014-09-19 13:08:37,972 INFO [MiSeqIntegrator] (SequencingIntegrationUtil)
Extracted raw content - [cycleNumber = 308 , runFolder = "D:IlluminaMiSeqTemp
140904_M01320_0130_000000000-A9NE9" , netFolder = "Z:140904_M01320_0130_000000000-A9NE9" ,
Record
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

8. What are logs?
2014-09-19 13:08:37,972 INFO [MiSeqIntegrator] (SequencingIntegrationUtil)
Extracted raw content - [cycleNumber = 308 , runFolder = "D:IlluminaMiSeqTemp
140904_M01320_0130_000000000-A9NE9" , netFolder = "Z:140904_M01320_0130_000000000-A9NE9" ,
Record
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

9. What are logs?
2014-09-19 13:08:37,972 INFO [MiSeqIntegrator] (SequencingIntegrationUtil)
Extracted raw content - [cycleNumber = 308 , runFolder = "D:IlluminaMiSeqTemp
140904_M01320_0130_000000000-A9NE9" , netFolder = "Z:140904_M01320_0130_000000000-A9NE9" ,
Record
MiSeqIntegrator.log
HiSeqIntegrator.log
apache.log
GenStat.log
supervisord.log
…
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

10. What are logs?
2014-09-19 13:08:37,972 INFO [MiSeqIntegrator] (SequencingIntegrationUtil)
Extracted raw content - [cycleNumber = 308 , runFolder = "D:IlluminaMiSeqTemp
140904_M01320_0130_000000000-A9NE9" , netFolder = "Z:140904_M01320_0130_000000000-A9NE9" ,
Record
MiSeqIntegrator.log
HiSeqIntegrator.log
apache.log
GenStat.log
supervisord.log
…
So
logs
are
files?
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

11. What are logs?
2014-09-19 13:08:37,972 INFO [MiSeqIntegrator] (SequencingIntegrationUtil)
Extracted raw content - [cycleNumber = 308 , runFolder = "D:IlluminaMiSeqTemp
140904_M01320_0130_000000000-A9NE9" , netFolder = "Z:140904_M01320_0130_000000000-A9NE9" ,
Record
MiSeqIntegrator.log
HiSeqIntegrator.log
apache.log
GenStat.log
supervisord.log
…
- Logs are time-oriented streams of
records
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

12. What are logs for?
Theory

13. What are logs for?
Theory
Provide real-time and valuable
information about the execution of
a program
Use this information in your benefit:
prevent problems, do analytics, plot
status…

14. Example: Our pipeline

15. Example: Our pipeline
started job X for sample Y
Aligning sample X
Generating report for Project A.Sample_14_09
Cleaning /proj/a2010002/nobackup area

16. Example: Our pipeline
started job X for sample Y
Aligning sample X
Generating report for Project A.Sample_14_09
Cleaning /proj/a2010002/nobackup area
Submitted jobs in the last X mins… Pipeline crashes in the last X days…

17. Example: Illumina logs

18. Example: Illumina logs
- Status of a particular
run
- Failures/Anomalies
- Cycles sequenced
today/this week/etc
- …

19. What are logs for?
Reality

20. What are logs for?
Reality
Something we look at ONLY when
something has already gone wrong…
if we can!

21. On the previous examples…

22. On the previous examples…
- The pipeline logs are dumped to
nextgen_analysis_server.log, in
milou-b, under the functional
account… and rotated!
- The Illumina logs are just never
looked at…

23. Problems

24. Problems
- Logs spread around servers and
accounts
- Rotating logs may disappear
- If you don’t rotate, logs will fill up
disks
- Hardly difficult to do any analytics
(real-time)
- Different applications == different
log formats

25. Problems
Genologics support:
”I took a look at the system. Unfortunately
the logs are filling up in too quick of a
time. I have increased the number of logs
and the size of them. We should have more
that one day of logs now.”

26. Problems
rm -rf <all_the_logs>

27. ELK Stack
!
- Elasticsearch
- Logstash
- Kibana

28. Logstash

29. Logstash
Index log records form
different sources
Re-format log data to be
structured and ”queryable"
Apply filters
Store your structured data
into Elasticsearch (and
other outputs)

30. input {
#Read messages from redis
redis {
host => "localhost"
data_type => "list"
password => "password"
key => "python"
codec => json
}
}
!
#We want to filter multiline events, and we'll suppose that multiline
#events are composed by one event and the following ones starting with
#a sapce (like anexception traceback)
filter {
multiline {
type => "exception"
pattern => "^s"
what => "previous"
add_tag => [ "exception" ]
}
}
!
output {
elasticsearch {
host => "tools.scilifelab.se"
}
}

31. Elasticsearch

32. Elasticsearch
Built on top of Lucene
Store complex data as
structured JSON documents.
All fields are indexed by
default, and all the indices
can be used in a single query.
Schema free (good for logs)
RESTful API

33. Kibana

34. Kibana
No code required
Real-time analysis for
streaming data
Customise and create
dashboards
For freeeee!!!

36. Shippers

37. Shippers
Broker*

38. Shippers
Broker*
Indexer

39. Shippers
Broker*
Indexer
Storage & search

40. Shippers
Broker*
Indexer
Visualization
Storage & search

41. Thank you!