In pre-production, there’s lots of tools that help optimizing your code: debuggers, CI/CD, load tests, etc. There’s even tools that automatically deploy them into production. Plus, engineers usually have a whole sprint as a time frame. Once you’re in production, things are a bit different and all that convenience is just not there for operators. They need to be able to pinpoint trouble spots within minutes. They have to identify the handful of bad requests out of thousands that allows for reproduction of the problem. And then, they are finally required to hand all that information over to the developers as convenient and as soon as possible. And nothing of that is automated. Performance monitoring, call-tracing and visualization are the concepts any developer should know about to be able to provide as much insight as possible into running systems. This session introduces open-source tools that allow devs and ops to work together much closer. To name just a few: * statsd / collectd * Zipkin * Spring-Cloud Sleuth * and some more For the sake of completeness and to also cover the Enterprise user-space, the main commercial vendors in that space will also be mentioned real quick. After that session, you’ll see new ideas popping up inside your head and already have all the knowledge you need to directly jump into planning and implementation.

1. DEV
OPS
Push the hassle
from production
to developers.
Easily
DevOpsDays Ghent 2016
October 27th
@MartinGoodwell
Dynatrace

2. @MartinGoodwell
About me
Passionate about life,
technology, and the people
behind both of them.
• Started with Commodore 8-bit (VC-20 and C-64)
• Built Null-modem connections for playing Doom and WarCraft
• Built IPX/SPX networks between MS-DOS 5.0 and Windows 3.1
• Did DevOps before they called it that way (mainly Java and Web)
for about 10 years
• Now at Dynatrace Innovation Lab
• Tech Lead for Microsoft Technologies
and Software Architecture
• Find me on Twitter: @MartinGoodwell
@MartinGoodwell

3. @MartinGoodwell
Agenda
• The Rules
• Warm-up
• The Ops dilemma (I call it that)
• The second Ops dilemma
• On Monitoring ...
• ... and Logging
• ... and Call Tracing
• ... and Databases
• Commercial offerings
@MartinGoodwell

4. @MartinGoodwell
The Rules
• Please, ask or interrupt anytime
• But keep ideas for open space discussions
• Or track me down anytime around
@MartinGoodwell

5. @MartinGoodwell
Warm up
• What's your occupation?
• Dev, Ops, BinExec?
• What's your technology stack?
• Node.js
• Go
• Java
• .net
• Who of you does
• Monitoring
• Logging
• Call-Tracing
• Application performance management/monitoring (APM)
@MartinGoodwell

6. @MartinGoodwell
The Ops dilemma (1)
Dev
• Single transaction
• Deal with a specific problem
• No impact on real users and business
• Concentrate on single component
• Deadlines refer to sprints
• Weeks, usually
Ops
• 100s or 1000s of txns
• No idea, what the cause is
• Real user impact
• Lots of moving parts
• Deadlines usually mean SLAs
• Hours, maybe just minutes
@MartinGoodwell

7. @MartinGoodwell@MartinGoodwell

8. @MartinGoodwell
The Ops dilemma (1)
Dev
• Single transaction
• Deal with a specific problem
• No impact on real users and business
• Concentrate on single component
• Deadlines refer to sprints
• Weeks, usually
Ops
• 100s or 1000s of txns
• No idea, what the cause is
• Real user impact
• Lots of moving parts
• Deadlines usually mean SLAs
• Hours, maybe just minutes
@MartinGoodwell

9. @MartinGoodwell
The Ops dilemma (2)
Automation
• Continuous {Integration/Deployment/Delivery} pipeline
• triggering unit tests for fast feedback
• Build servers
• Repositories
• Automatic deployments
• Helps devs getting stuff into production
• Does nothing for the opposite direction
@MartinGoodwell

10. @MartinGoodwell
DevOps is about collaboration.
Collaboration requires documentation.
Automation is implicit documentation.
But there is no automation for
supporting Ops with troubleshooting.
@MartinGoodwell

11. @MartinGoodwell
Monitoring
@MartinGoodwell

12. @MartinGoodwell
Host metrics
• CPU usage
• Memory usage
• Disk I/O
• Network performance
• No insight into app's
problems and performance
@MartinGoodwell

13. @MartinGoodwell
In your code
@MartinGoodwell

14. @MartinGoodwell
Use statsd
@MartinGoodwell

15. @MartinGoodwell
statsd real quick
http://www.slideshare.net/DatadogSlides/dev-opsdays-tokyo2013effectivestatsdmonitoring
@MartinGoodwell

16. @MartinGoodwell
Downsides?
• "Polluting" business logic with monitoring code
• Code introspection (ie AOP) requires advanced skills
• Not using something like statsd leads to cluttered metrics
• Great for single component insight
• what about called 3rd parties?
• what about microservices (ie distributed transactions)?
• what about calls to databases, queues, etc.
@MartinGoodwell

17. @MartinGoodwell
Logging
@MartinGoodwell

18. @MartinGoodwell
http://theburningmonk.com/2015/05/a-consistent-approach-to-track-correlation-ids-through-microservices/
@MartinGoodwell

19. @MartinGoodwell
http://theburningmonk.com/2015/05/a-consistent-approach-to-track-correlation-ids-through-microservices/
@MartinGoodwell

20. @MartinGoodwell
http://theburningmonk.com/2015/05/a-consistent-approach-to-track-correlation-ids-through-microservices/
@MartinGoodwell

21. @MartinGoodwell
Logging learnings
• Use a logging server (eg ELK stack)
• directly log as JSON
• at least store as JSON
• Using logging for monitoring is expensive
• log analysis is a real resource hog
• works great for troubleshooting
• works great with limited problem scope
• for Java, use Logback via SLF4J
• to local logfiles
• to logstash
• to syslog
@MartinGoodwell

22. @MartinGoodwell
Call Tracing
@MartinGoodwell

23. @MartinGoodwell
Google Dapper paper
• The Dapper paper (2010)
http://research.google.com/pubs/archive/36356.pdf
• OpenTracing
http://opentracing.io/documentation/
• OpenZipkin (by Twitter)
• http://zipkin.io/
@MartinGoodwell

24. @MartinGoodwell
Zipkin architecture
http://zipkin.io/pages/architecture.html
@MartinGoodwell

25. @MartinGoodwell
https://github.com/openzipkin/zipkin
@MartinGoodwell

26. @MartinGoodwell
http://zipkin.io/
@MartinGoodwell

27. @MartinGoodwell
https://github.com/ordina-jworks/microservices-dashboard
@MartinGoodwell

28. @MartinGoodwell
https://github.com/spring-cloud/spring-cloud-sleuth
Spring Cloud Sleuth is a distributed tracing solution on top of Spring Cloud
@MartinGoodwell

29. @MartinGoodwell
http://trace.risingstack.com
@MartinGoodwell

30. @MartinGoodwell
Databases
@MartinGoodwell

31. @MartinGoodwell
Getting database insight
• Database automation
• eg. DB Maintain
• https://dbmaintain.github.io/
• Database performance logging
• log4jdbc
• https://github.com/arthurblake/log4jdbc
@MartinGoodwell

32. @MartinGoodwell
The commercial hood
@MartinGoodwell

33. @MartinGoodwell
Broad technology support
@MartinGoodwell

34. @MartinGoodwell
Zero-conf and ready to run dashboards
@MartinGoodwell

35. @MartinGoodwell
Method level insight for code and database
@MartinGoodwell

36. @MartinGoodwell
Host, process and network metrics
@MartinGoodwell

37. @MartinGoodwell
Call-tracing across technologies
@MartinGoodwell

38. @MartinGoodwell
Including log analytics
@MartinGoodwell

39. @MartinGoodwell
Full Docker insight (zero-conf)
@MartinGoodwell

40. @MartinGoodwell
Dedicated support for most important technologies
@MartinGoodwell

41. @MartinGoodwell
Automated baselining, root-cause-analysis, and problem
correlation
@MartinGoodwell

42. @MartinGoodwell
You can't fight in here, Gentlemen.
This is the war room!

43. @MartinGoodwell
Let's talk
• From monolith to microservice
• cloud migration
• performance optimization
• team culture
@MartinGoodwell
martin.goodwell@dynatrace.com
@MartinGoodwell

44. @MartinGoodwell
Thank you!
@MartinGoodwell