This document discusses using ELK (Elasticsearch, Logstash, Kibana) to gain insights from logs. It describes the components of ELK - Elasticsearch as the database, Kibana as the UI, and Logstash to parse logs. Logstash can use GROK patterns to parse logs into structured data for analysis in Kibana. The document provides examples of using ELK to track web traffic, user activity, and API responses for benefits like reducing costs and monitoring performance. While custom solutions can be built, ELK is beneficial as it requires no software costs and allows full control over collected log data.

1. LOGS
THE MISSING GOLD MINE
by Franjo Žilić

2. WHY TALK ABOUT LOGS?
They are all around us
We use them to debug our software every day
Is that all tere is?

3. CONVENTIONAL WAY

4. CONVENTIONAL WAY

5. CONVENTIONAL WAY
grep -v " 200 " access_log-20160124

6. HOW ABOUT A DIFFERENT VIEW

7. ELK
* not that elk.

8. ELK
Meet the familiy

9. ELK
elasticsearch
logstash*
kibana
* or something else

10. ELASTICSEARCH
The database
document oriented
distributed
sharded
replicated*
timestamp partitioning*
Java & lucene

11. KIBANA
The UI
colorful
small
GRAPHS

12. LOGSTASH
The parser
uses GROK
plugins for nearly everything

13. PARSING LOGS
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

14. PARSING LOGS
^(?#regex designed to parse VyOS kernel log)(?#some global parsing,
like timestamp, fitlter, interfaces, and so on)(?<time>[^ ]* [^ ]* [
^ ]*) (?<host>[^ ]*) (?<vyos_sylog_facility>[^: ]*)?: [(?<vyos_fw_f
ilter_name>[^[]*)] ?IN=( |(?<vyos_in_interface>[^ ]*) )OUT=( |(?<v
yos_out_interface>[^ ]*) )(MAC=( |(?<vyos_mac_addres>[^ ]*) ))?SRC=(
|(?<vyos_source_ip_address>[^ ]*) )DST=( |(?<vyos_dstination_ip_add
ress>[^ ]*) )LEN=( |(?<vyos_len>[^ ]*) )TOS=( |(?<vyos_tos>[^ ]*) )P
REC=( |(?<vyos_prec>[^ ]*) )TTL=( |(?<vyos_ttl>[^ ]*) )ID=( |(?<vyps
_packet_id>[^ ]*) )(?<vyos_packet_flags>[^ |(PROTO)]*)? ?PROTO=( |(?
<vyps_ip_protocol>[^ ]*))(?#here comes the fun part, different parse
r for different interesting packet types, regex if and positive look
behind matching each type of interesting )(?:(?<=(TCP))((?#tcp speci
fic matchers) ?SPT=( |(?<vyos_source_port>[^ ]*) )DPT=( |(?<vyos_des
tination_port>[^ ]*) )WINDOW=( |(?<vyos_tcp_window>[^ ]*) )RES=( |(?
<vyos_res>[^ ]*) )(?<vyps_tcp_state>[^(URGP)]* ).*)|(?:(?<=(UDP))((?
#udp specific matchers) ?SPT=( |(?<vyos_source_port>[^ ]*) )DPT=( |(
?<vyos_destination_port>[^ ]*) ).*)|(?:(?<=(ICMP))((?#icmp specific
matchers) TYPE=( |(?<vyos_icmp_type>[^ ]*) )CODE=( |(?<vyos_icmp_cod
e>[^ ] )).*)|(.*))))$

15. CUSTOM SOLUTION?
Implement custom data collection within the application
Populate data with Servlet filters or Spring AOP
Index data in Elasticsearch

16. BENEFITS
Logging that meets your needs
Ability to extract analytical data
Near real-time event tracking

17. REAL WORLD EXAMPLES
Tracking HTTP traffic
Tracking user activity
Tracking 3rd party API responses

18. HTTP DATA
Bandwith costs money
Web site scrapers are common
Serving non-compressed data is expensive
Identification of scrapers can reduce cost

19. 3RD PARTY API TRACKING
Log all requests and responses
Monitor performance
Monitor availability
Provide extra troubleshooting data

20. REAL WORLD DATA

21. BENEFITS OVER COMMERTIAL
SOLUTIONS
no software cost
on site or cloud install
full control over data
fully customizable

22. DRAWBACKS
additional setup
nothing is predefined
no "enterprise" support

23. CONCLUSIONS
Utilize data that you already have
learn more abiout your users and applications

24. THANK YOU
elastic.co
Fluentd
Apache Flume