Given by Tom Wilkie on 11 May 2016 at GOTO Stockholm The rise of microservice-based applications has had many knock-on effects, not least on the complexity of monitoring your application. Order-of-magnitude increase in the number of moving parts and rate of change of the application require us to reassess traditional monitoring techniques. In this talk we will discuss some different approaches to monitoring, visualising and tracing containerised, microservices-based applications. We’ll present different techniques to some of the emergent problems, and try not to rant too much.

1. Workshops and Conference: May 9-11, 2016
2016
Stockholm
Let us know
what you think!
Click“Engage”
to rate a session.
If you rate 12 sessions
you get a cool GOTO prize!

2. Monitoring Microservices
tom@weave.works
@tom_wilkie

4. VisualisationMonitoring Tracing
0
25
50
75
100

5. Monitoring
0
25
50
75
100

6. Traditional 3-tier architecture
Incoming traffic
Load balancers
Application servers
Database & replica

7. Microservice architecture
Public APIWeb UI
NoSQL serversDatabase
Message
Broker
Services

9. Microservices should be
treated like cattle not pets

11. USE Method* - for every
resource, check:
• utilization,
• saturation, and
• errors
RED Method - for every
service, check request:
• rate,
• error (rate), and
• duration (distributions)
* http://www.brendangregg.com/usemethod.html
An alternative view

12. Okay, but how?
var rpcDurations = prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "rpc_durations_histogram_microseconds",
Help: "RPC latency distributions.",
Buckets: prometheus.LinearBuckets(0, 100, 20),
})
func init() {
prometheus.MustRegister(rpcDurations)
}
func handleRequest(w http.ResponseWriter, r *http.Request) {
begin := time.Now()
...
rpcDurations.WithLabelValues(r.Method).Observe(
float64(time.Since(begin).Nanoseconds()))
}

14. There must be a better way…
Kubeproxy
Replicas
incoming traffic from
other services

15. Demo Time

16. https://github.com/weaveworks/flux
Try it out!

17. Monitoring
0
25
50
75
100

18. Visualisation

20. Weave Scope

21. Connection
Tracking
/home/weave # conntrack -E
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41066 dport=80 src=172.17.0.1 dst=172.17.0.10
[DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=36236 dport=32778 src=172.17.0.8 dst=1
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41068 dport=80 src=172.17.0.1 dst=172.17.0.10
[DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=52996 dport=32776 src=172.17.0.6 dst=1
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41070 dport=80 src=172.17.0.1 dst=172.17.0.10
[DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=52998 dport=32776 src=172.17.0.6 dst=1
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41072 dport=80 src=172.17.0.1 dst=172.17.0.10
[DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=57975 dport=32777 src=172.17.0.7 dst=1
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41074 dport=80 src=172.17.0.1 dst=172.17.0.10
/home/weave # cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 16810 1 ffff
1: 0100007F:EB74 0100007F:0FC8 06 00000000:00000000 03:0000016D 00000000 0 0 0 3 ffff8800
2: 0100007F:EB69 0100007F:0FC8 01 00000000:00000000 00:00000000 00000000 0 0 307011 1 fff
3: 0100007F:EB7B 0100007F:0FC8 06 00000000:00000000 03:00000D27 00000000 0 0 0 3 ffff8800
4: 0100007F:EB7C 0100007F:0FC8 06 00000000:00000000 03:0000110E 00000000 0 0 0 3 ffff8800
5: 0100007F:EB67 0100007F:0FC8 01 00000000:00000000 00:00000000 00000000 0 0 306868 1 fff
6: 0100007F:EB76 0100007F:0FC8 06 00000000:00000000 03:00000556 00000000 0 0 0 3 ffff8800
7: 0100007F:EB7F 0100007F:0FC8 06 00000000:00000000 03:000014F7 00000000 0 0 0 3 ffff8800

22. all connections
from
/proc
conntrack
Connection
Tracking
load
balanced

23. Demo Time

24. https://github.com/weaveworks/scope
Try it out!
https://weave-scope-slack.herokuapp.com
v0.13.1
https://www.weave.works/products/weave-scope
https://scope.weave.works

25. Visualisation

26. Tracing

27. Distributed
Tracing

28. Not a new topic
• Lots of literature
• Existing open source
projects
• e.g. Zipkin, originally from
Twitter

29. • Challenge: detecting
causality between
incoming and outgoing
requests
• Existing solutions require
propagation of some
unique ID (dapper, zipkin)
• This requires application-
specific modifications
some service
incoming
request
outgoing
requests
?

30. Can this be done without
application modifications?

31. By intercepting systems calls,
build up a data structure of:
• which threads are reading
to / writing from which FDs
• which FDs are talking to
which IPs & ports
Use this to infer causality
between incoming and
outgoing connections.
some service
kernel
?
System calls

32. Demo Time

33. Try it out!
https://github.com/weaveworks/scope/tree/master/experimental/tracer

34. Tracing

35. VisualisationMonitoring Tracing
0
25
50
75
100

36. We’re hiring!

37. @weaveworks github.com/weaveworks
Questions?
tom@weave.works
@tom_wilkie

38. https://weave-scope-slack.herokuapp.com
https://github.com/weaveworks/scope
https://github.com/weaveworks/flux
https://scope.weave.works
Links
https://github.com/weaveworks/scope/tree/master/experimental/tracer

39. Workshops and Conference: May 9-11, 2016
2016
Stockholm
Please remember to
rate this session
...Thank You!