This document discusses various techniques for monitoring microservices including Circuit Breaker pattern, bulkheads, back pressure, Hystrix, Hystrix Dashboard, thread pools, Dynatrace, and memory analysis. Hystrix is a latency and fault tolerance library used for monitoring, controlling concurrency, and isolating points of access to remote systems. The Hystrix Dashboard provides real-time monitoring. Dynatrace allows deep profiling of applications through memory dumps and other techniques to identify performance issues and root causes. The workshops agenda covers setting up Hystrix commands, the dashboard, and connecting applications to Dynatrace for memory and GC monitoring.

2. Microservice monitoring
Marek Koniew

3. Reactive manifesto
0
http://www.reactivemanifesto.org/
https://github.com/Netflix/Hystrix/wiki
Bulkheads

4. Cascading failures
2
Circuit breaker, bulkhead pattern, cascading failures

5. Back pressure
4
https://github.com/Netflix/Hystrix/wiki
Circuit breaker, back pressure, bulkhead pattern, cascading failures, http 429 too many requests

6. Hystrix
9
- Circuit Breaker
- Request Collapsing
- Request Caching
- Monitoring

7. Hystrix Dashboard
12
Real time information. To store information use Graphite.

8. Hystrix Command Widget
14

9. Hello World
17

10. Migrating a Library to Hystrix
19
- External service call should be wrapped into a HystrixCommand
- Configure thread pools
- Fine grained command names
Circuit breaker, Timeouts, queue size, thread pool size

11. Patterns and anti-patterns: naming convention
22
https://wiki.hybris.com/display/prodandtech/Hystrix+commands+naming+convention+-+draft
docu-repo-v2.deleteType.db
Service name
version Command name
External service

12. Patterns and anti-patterns: thread pools
22
https://github.com/Netflix/Hystrix/wiki/Configuration#threadpool-properties
- Netflix API has 30+ of its threadpools
set at 10, 2 at 20 and 1 at 25
- Queue size affect request times !

13. Hystrix Dashboard Demo
25
DEMO

14. Hystrix Dashboard Demo
26
DEMO

15. Dynatrace
Dynatrace architecture
https://www.youtube.com/watch?v=judLcsVns-s
Memory analysis:
https://www.youtube.com/watch?v=dUn4iBM6Hik
28

16. Dynatrace
31

17. Dynatrace
35
Wiki:
https://wiki.hybris.com/display/INFRA/OnDemand+performance+testing+setup
Configure java agent:
java
–agentpath:libdtagent.so=
name=app,
server=dynatrace.yrd.fra.hybris.com,wait=45,
storage=.

18. Dynatrace – advantages
 Direct access to agent machine
 Memory dumps
 CPU sampling
 Much more
 JVM and system monitoring
 Request tracing across services
 Hot sensor placement – class instrumentation can be changed without restart
38

19. Dynatrace – disadvantages
 Performance overhead
 Sensor configuration affects all agents
 Weak support for java 8 (although dynatrace version 6 is out)
 Steep learning curve
41

20. Dynatrace patterns and anti-patterns
 Do NOT use it for monitoring – utilize its profiling power
 Much better tools available for monitoring (Graphite, hystrix dashboard)
 Applications designed to be monitored does not need dynatrace
 Only very simple alerts available
 Turn off sensors by default to save performance
 Configure separate service instance with extended number of sensors on
demand
 Route part of the traffic through this service (canary testing)
44

21. Dynatrace demo scenario
47
1) Start application with agent.
2) Observe allocated memory
3) Observe GC suspension times
4) Create memory dump
5) Analyze memory dump

22. Dynatrace demo – allocated memory
49

23. Dynatrace demo – GS suspentions
51

24. Dynatrace demo – application crash
52

25. Dynatrace demo – memory dump
53

26. Dynatrace demo – memory dump
54

27. Dynatrace demo – analyze memory
55

28. Dynatrace demo – root cause
57
https://wiki.hybris.com/display/prodandtech/2014/08/26/Your+service+is+leaking+memory

29. Hystrix Dashboard Demo
60
DEMO

30. Questions

31. Workshops agenda
 Create hystrix command:
 Set command name.
 Set command group.
 Configure thread pool.
 Setup hystrix.stream endpoint.
 Setup hystrix dashboard.
 Connect application to dynatrace.
 Create dynatrace graph:
 Memory usage.
 GC suspension.
 Create memory dump.