1.
How to ditch the debugger
and use logging instead
Matthew Skelton & Rob Thatcher, Skelton Thatcher Consulting
/dev/winter 2015,
Saturday 24th January, Cambridge, UK
#devwinter

2.
Event tracing
Vagrant + ELK
Demo: no debugging

3.
Skelton Thatcher
Introduction and Services Overview

4.
‘Cloud’ changes the way we must
design, deliver, and operate
our software systems

5.
Changes for ‘cloud’

6.
Event tracing

7.
Event type IDs
Transaction tracing
Adjusting logging levels

8.
Event type IDs
How many event types in your
application?

9.
Event type IDs

10.
Event type IDs
enum
Human-readable sets:
unique values, sparse, immutable
C#, Java, Python, node
(Ruby, PHP, …)

11.
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
ItemAddedToBasket = 60001,
ItemRemovedFromBasket = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}

12.
Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
ItemAddedToBasket = 60001,
ItemRemovedFromBasket = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}

13.
ItemAddedToBasket = 60001

14.
ItemAddedToBasket = 60001

15.
ItemAddedToBasket = 60001

16.
Transaction tracing

17.
Monolith to microservices:
debugger does not have the full view
Distributed systems

18.
Even with remote debugger, it’s
boring to attach and detach
Remote debugging?

19.
Transaction tracing
Unique-ish identifier assigned to
web request
Passed through downstream
layers

20.
What about APM?
APM gives us application insight
BUT
How much do we learn?
Is APM available on the Dev box?
It’s not just ‘an Ops problem’!

21.
Minimal viable tracing?
Helps us to understand how the
software really works
Small overhead is worth it

22.
Which log level is right?
DEBUG, INFO, WARNING,
ERROR, CRITICAL

23.
Which log level is right?
Log level should *not* be fixed at
compile or build time!

24.
<?xml version="1.0" encoding="utf-8" ?>
<Application.EventMappings>
<Events>
<Event Type="CacheServiceStarted">
<Severity Type="Information"/>
</Event>
<Event Type="PageCachePurged">
<Severity Type="Debug"/>
<Enabled="false"/>
</Event>
<Event Type="DatabaseConnectionTimeOut">
<Severity Type="Error"/>
</Event>
</Events>
</Application.EventMappings>

25.
Tune log levels

26.
Tune log levels

27.
Tune log levels

28.
Tune log levels
http://bit.ly/TuneLoggingLevels

29.
Dev and Ops collaboration*
* and testers too!

30.
Event tracing
Use enumerations (or closest thing)
Technical and Domain event types
Distributed systems: debuggers less useful
Trace calls with ‘unique-enough’ handles
Tune log levels via config

31.
Vagrant + ELK

32.
Log Aggregation
Elasticsearch, Logstash & Kibana
Vagrant

33.
Log Aggregation

34.
Multiple log sources
Many logs in one place
Searchable
Time Sequenced

35.
What is ELK?
http://www.exploringnature.org/db/detail.php?dbID=43&detID=956

36.
Vagrant

37.
Ruby CLI to deploy Virtual environments
Deploy locally to AWS or Azure etc.

38.
Four Simple Steps
$ vagrant box add ubuntu/trusty64
$ vagrant init trusty64
$ vagrant up
$ vagrant ssh

39.
Rapid
Repeatable
Provisioning via scripts ++

40.
• Listening to:
• Local nginx logs
• Local Syslog
• (Application logs)
http://bit.ly/velk-demo
Vagrant + ELK Demo VM

41.
Elasticsearch, Logstash, Kibana

42.
Recap

43.
Aggregated Logging:
Vagrant + ELK
Centralised logs, time sequenced, searchable
Vagrant :quickly spin up & manage machines
ELK : collect, store, display & search logs
Developer-friendly!

44.
Demo: ditching the debugger

45.
Vagrant on developer laptop
ELK for log aggregation
Web app: PHP (Slim)
Event type IDs
Demo: background

46.
Use VirtualBox
shared folders
Browse to
Kibana UI via
NAT-ed interface

47.
Event type IDs
Log aggregation with ELK
Demo

48.
Demo: summary
Super-easy to hook application logs into ELK
Sequences of events emerge
Compare timings of events: past and present

49.
Recapitulation

50.
How to ditch the debugger
Use Vagrant + ELK on development VM
Use a set of event type IDs (enums)
Log distinct states of execution
Both technical and domain events
Consistent event naming helps with search
Compare previous and current executions
New logging-fu: make friends with Ops!

51.
Links
Tune logging levels in Production without recompiling code:
http://bit.ly/TuneLoggingLevels
VELK (Vagrant + ELK) demo VM:
http://bit.ly/velk-demo
GOOS book (section on logging: p.233-5):
http://www.growing-object-oriented-software.com/

52.
Thank you!
http://skeltonthatcher.com/
enquiries@skeltonthatcher.com
@SkeltonThatcher
+44 (0)20 8242 4103