A mostly non technical presentation on the WHY behind why you should be centralising your logging infrastructure and how to achieve it. This is an area of your infrastructure that is often left by the wayside and is potentially one of your most important development assets. It will cover briefly what tools are out there and point you in the right direction on how to best make your own decisions on what tools to choose to achieve a centralised infrastructure yourself. Finally, the presentation covers how to structure your logs, what content to include and a short explanation on how THE ICONIC has built our infrastructure.

1. CENTRALISED
LOGGINGOliver Brennan
Senior Software Engineer
THE ICONIC
@BrOllie

2. THE
PROBLEM

3. THE
PROBLEM

4. THE
SOLUTION?
> sudo i2cssh -b username@web0{1..4}

5. THE
PROBLEM

6. THE
PROBLEM

7. THE
SOLUTION?

8. THE
SOLUTION?

9. A STORY!
I L O V E S T O R I E S

10. RECEIVE ALERT
A STORY!
I L O V E S T O R I E S

11. YOU WAKE UP
A STORY!
I L O V E S T O R I E S

12. COMPUTER IS SLOW
(GETTING ANGRY NOW)
A STORY!
I L O V E S T O R I E S

13. DO SOME DIGGINGFULL DISK
A STORY!
I L O V E S T O R I E S

14. CLEAR THE LOGS
(BACK TO BED, YAY)
A STORY!
I L O V E S T O R I E S

15. A STORY!
I L O V E S T O R I E S
YOU CLEARED THE LOGS!
Was it a rogue process?
Will it wake you up again in an hour?
How do you know what used all the disk space?
Are you awake enough to be root on live servers at 3am?
rm –rf ./* vs rm –rf /*

16. THE
SOLUTION

17. TOOLS
AVAILABLE
Cloud Hosted
Plus others…

18. HOW TO
CHOOSE?
• EASY TO QUERY DATA
• ADVANCED QUERY FUNCTIONALITY
• MUST SYNC IN REAL TIME
• PROVIDED ‘TAIL –F’ FUNCTIONALITY
• PRICE
• SCALES WITH US
• KEEP LOGS AFTER SERVERS HAVE BEEN
RECYCLED (CLOUD SERVERS)
• NON SELF-MANAGED SOLUTION
• TRIAL THEM ALL
RIGHT NOW

19. HOW TO
CHOOSE?
• EASY TO QUERY DATA
• ADVANCED QUERY FUNCTIONALITY
• MUST SYNC IN REAL TIME
• PROVIDED ‘TAIL –F’ FUNCTIONALITY
• PRICE
• SCALES WITH US
• KEEP LOGS AFTER SERVERS HAVE BEEN
RECYCLED (CLOUD SERVERS)
• NON SELF-MANAGED SOLUTION
• TRIAL THEM ALL
RIGHT NOW GROWINTO
• ALERTS & NOTIFICATIONS
• TRACK APPLICATION EVENTS
• ANNOTATIONS
• DASHBOARDS
• JSON FORMATTED LOGS
• WHO KNOWS?

20. AND THE
WINNER?
?In the interest of fairness, I wont be posting it here

21. THINGS TO
KEEP IN MIND
CONTEXT IS KING. IMAGINE THE FOLLOWING LINE:
29/11/2014 – 10:11:12 – 306654 - /login - 200
29/11/2014 – 10:11:12 – cust_id=306654 – action=/login – status=200
WITH CONTEXT:
*easier to troubleshoot, gain context and understanding of the domain
29/11/2014 – 10:11:12 – cust_id=306654 – action=/login – status=200
– hostname=web07.theiconic.com.au
WITH CONTEXT AND ENV INFO:

22. THINGS TO
KEEP IN MIND
STRUCTURE YOUR LOGS CORRECTLY
• KEY VALUE PAIRS (KVP)
• JSON LOGS
29/11/2014 – 10:11:12 – $330.00 $12.00 304857 20181743 5
29/11/2014 – 10:11:12 – order_val=$330.00 shipping_val=$12.00
cust_id=304857 order_nr20181743 item_count=5
WITH KVP:
*enables advanced querying of data. KVP is a little easier to read though
{"date":"29/11/2014 u2013 10:11:12","order_val":"$330.00",
"shipping_val":"$12.00","cust_id":304857,"order_nr":20181743,"item_count":5}
WITH JSON:

23. LOGGING
STRATEGY
THINK ABOUT WHAT YOU WANT TO LOG. LOGGING EVERYTHING JUST MEANS
WE HAVE MORE DATA TO SIFT THROUGH. THINK ABOUT
• PERFORMANCE AND RESOURCES: Memory, CPU, disk, network – we can show spikes
and performance degradation of our apps
• EXCEPTIONS & WARNINGS: No explanations required here
• USER ACTIONS: Logging access, audit trails, device access, failed access attempts
• EXTERNAL SERVICES: Requests / responses from external services should all be
logged. You can thank me later for this. Integration is hard, make it easier on yourself
- at least in the early days keep logging running
• WEB SERVERS: Server access/error logs (helps when distributed over servers)
TROUBLESHOOTING
SECURITY
AUDITING
MONITORING

24. OUR
SETUP!

25. SOME
STATS!
GB stored per day
(approx)
4510 Million records
stored per day
(approx)
93Log files