Javantura v3 - Logs – the missing gold mine – Franjo Žilić

1. LOGS
THE MISSING GOLD MINE
by Franjo Žilić

2. WHY TALK ABOUT LOGS?
They are all around us
We use them to debug our software every day
Is that all tere is?

3. CONVENTIONAL WAY

4. CONVENTIONAL WAY

5. CONVENTIONAL WAY
grep -v " 200 " access_log-20160124

6. HOW ABOUT A DIFFERENT VIEW

7. ELK
* not that elk.

8. ELK
Meet the familiy

9. ELK
elasticsearch
logstash*
kibana
* or something else

10. ELASTICSEARCH
The database
document oriented
distributed
sharded
replicated*
timestamp partitioning*
Java & lucene

11. KIBANA
The UI
colorful
small
GRAPHS

12. LOGSTASH
The parser
uses GROK
plugins for nearly everything

13. PARSING LOGS
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

14. PARSING LOGS
^(?#regex designed to parse VyOS kernel log)(?#some global parsing,
like timestamp, fitlter, interfaces, and so on)(?<time>[^ ]* [^ ]* [
^ ]*) (?<host>[^ ]*) (?<vyos_sylog_facility>[^: ]*)?: [(?<vyos_fw_f
ilter_name>[^[]*)] ?IN=( |(?<vyos_in_interface>[^ ]*) )OUT=( |(?<v
yos_out_interface>[^ ]*) )(MAC=( |(?<vyos_mac_addres>[^ ]*) ))?SRC=(
|(?<vyos_source_ip_address>[^ ]*) )DST=( |(?<vyos_dstination_ip_add
ress>[^ ]*) )LEN=( |(?<vyos_len>[^ ]*) )TOS=( |(?<vyos_tos>[^ ]*) )P
REC=( |(?<vyos_prec>[^ ]*) )TTL=( |(?<vyos_ttl>[^ ]*) )ID=( |(?<vyps
_packet_id>[^ ]*) )(?<vyos_packet_flags>[^ |(PROTO)]*)? ?PROTO=( |(?
<vyps_ip_protocol>[^ ]*))(?#here comes the fun part, different parse
r for different interesting packet types, regex if and positive look
behind matching each type of interesting )(?:(?<=(TCP))((?#tcp speci
fic matchers) ?SPT=( |(?<vyos_source_port>[^ ]*) )DPT=( |(?<vyos_des
tination_port>[^ ]*) )WINDOW=( |(?<vyos_tcp_window>[^ ]*) )RES=( |(?
<vyos_res>[^ ]*) )(?<vyps_tcp_state>[^(URGP)]* ).*)|(?:(?<=(UDP))((?
#udp specific matchers) ?SPT=( |(?<vyos_source_port>[^ ]*) )DPT=( |(
?<vyos_destination_port>[^ ]*) ).*)|(?:(?<=(ICMP))((?#icmp specific
matchers) TYPE=( |(?<vyos_icmp_type>[^ ]*) )CODE=( |(?<vyos_icmp_cod
e>[^ ] )).*)|(.*))))$

15. CUSTOM SOLUTION?
Implement custom data collection within the application
Populate data with Servlet filters or Spring AOP
Index data in Elasticsearch

16. BENEFITS
Logging that meets your needs
Ability to extract analytical data
Near real-time event tracking

17. REAL WORLD EXAMPLES
Tracking HTTP traffic
Tracking user activity
Tracking 3rd party API responses

18. HTTP DATA
Bandwith costs money
Web site scrapers are common
Serving non-compressed data is expensive
Identification of scrapers can reduce cost

19. 3RD PARTY API TRACKING
Log all requests and responses
Monitor performance
Monitor availability
Provide extra troubleshooting data

20. REAL WORLD DATA

21. BENEFITS OVER COMMERTIAL
SOLUTIONS
no software cost
on site or cloud install
full control over data
fully customizable

22. DRAWBACKS
additional setup
nothing is predefined
no "enterprise" support

23. CONCLUSIONS
Utilize data that you already have
learn more abiout your users and applications

24. THANK YOU
elastic.co
Fluentd
Apache Flume