Presented at the Bucharest DevTalks event in 2018

1. Monitoring and Automation
From developer to devops

2. Quick questions

3. Quick questions
● Who here is doing operations?
● Who develops applications?
● Who develops infrastructure software?
● Who is doing all above?
● Who manages more than 10 servers?
● Who manages more than 100?
● 1000?

4. Where do we look?

5. Why do we measure?
● Optimize hardware usage
● Locate performance bottlenecks
● Identify anomalous behaviour
● Understand our own operational characteristics
○ How long do we normally take to add a new type of server?
○ What should we automate?
● Find out the costs of a given operation
○ If your operation takes 3 minutes of a server time, how much did we pay for it?
● Metering and billing
● Better understand the user
● Plan for future versions

6. Where do we look?
CPU counters
Revenue
Profit
Strategies Host OSGuest OSPlatformApplicationInternal processes

7. Where do we look?
CPU counters
Revenue
Profit
Strategies Host OSGuest OSPlatformApplicationInternal processes
OperationslandDeveloperland

8. Where do we look?
CPU counters
Revenue
Profit
Strategies Host OSGuest OSPlatformApplicationInternal processes
OperationslandDeveloperland
DevOpsland (according to a very lax definition)

9. Where do we look?
CPU counters
Revenue
Profit
Strategies Host OSGuest OSPlatformApplicationInternal processes
OperationslandDeveloperland
DevOpsland
Billing, HW
usage Performance
What to
automate?

10. How do we look?
● Time resolution (and retention)
● GDPR (don't be evil, or keep what you don't need)
● Push vs. Pull
○ Register clients or let them register themselves
● Is UDP your friend?
○ Fast, cheap and unreliable
○ If you need security, you need to build it
○ You should probably use it in combination with TCP (a subsampled guaranteed channel)
● Standards
○ Prometheus is your friend
● Visualization
○ You'll need to figure it out

11. On Automation

12. How about measuring ourselves?
● Learn what to automate
○ And what not to
● For stateless clusters, deployment is more important
○ Don't fix stuff you can replace
● For stateful ones, you'll want to keep lifecycle
○ You'll hate when your database server that has a disk space problem gets a brand new bigger
volume that also doesn't have your database
● Terraform/Chef/Puppet/Ansible/Salt/Whatever-works-for-you
○ All software sucks in one way or another
● Infrastructure as code
○ Versioned
○ Testable, tested routinely (deployment and life-cycle scenarios)

13. Why?
● Repeatability
○ High-fidelity replicas of the production environment
● Consistency
○ Unless you change something, the result should always be the same
■ Either that, or you have a more serious bug
● Less work == fewer mistakes
● Isn't it handy that the script you use to automate is under version control?
● One side can build upon the tools of the other - and vice-versa
○ And collaborate
■ And, ultimately, becoming a single team, even if you have different roles

14. So?

15. It all boils down to attitude
It's about cooperation, about tearing down walls
Offering and accepting insights from different people with different priorities
Reconciling different priorities around a single mission
A drive to understand your tooling, your systems
(Random) failure is not an option

16. Questions?

17. Thank you
@rbanffy - twitter, github,