The document outlines essential concepts in web operations (webops) for developers, emphasizing the integration of code and infrastructure alongside continuous monitoring and metrics collection. It discusses methodologies for managing alerts, investigating issues, and ensuring reliable service, while highlighting tools and processes integral to effective webops. The focus is on bridging the gap between development and operations for improved efficiency and reduced risks.

1. Just Enough WebOps
for Developers
Alexis Lê-Quôc @alq
http://www.datadoghq.com

2. @alq

3. @alq
Co-founder
DATADOG

4. Datadog is Monitoring that does not
suck... as a Service

5. Datadog is Monitoring that does not
suck... as a Service
“Metrics
made
social”

6. People-friendly
Monitoring

7. Developer-friendly
Monitoring

8. 930,000 350,000
Dev Ops
2010 US figures from BLS

9. The New
Development
Equation

10. The New Development Equation
Code + + AWS =

11. The New Development Equation
Code + + AWS =
3 months

12. The New Development Equation
Code + + AWS =
3 months 5 minutes

14. Web
Operations?

15. The New Development Equation
Code + + AWS =
3 months 5 minutes

16. The New Development Equation
Code + + AWS =
3 months 5 minutes
Web
Operations?

18. Cargo cult
Operations

20. Common vocabulary
between Dev & WebOps?

21. Users
SysAdmin

22. “Come and get it”
“We want root!”

23. Dev
WebOps

24. WebOps
and this is what I do

25. But first an
important
digression

26. Product Service

27. Service = Code + Infrastructure

28. Service = Product + Access

30. Provide access

31. Provide access

32. reliable, fast, cheap
Provide access

33. reliable, fast, cheap
Provide access

34. reliable, fast, cheap
Provide access
24x7
without going crazy

35. 24x7 && !crazy

36. Development
Models

38. Delivery historically
not the focus

39. Agile Cycle Delivery

40. Agile Cycle Delivery

41. Agile Cycle WDeebliOveprsy
Cycle

42. WebOps
and this is what I do

43. Dev Release Measure & Log
Monitor
Change
WebOps Cycle
Investigate Alert
Fix || Escalate

44. (Release)

45. Dev Release
Measure & Log
Monitor
Change
Investigate Alert
Fix || Escalate

46. Purpose
Collect quantitative metrics
Process
Instrument servers
Instrument code
Instrument SaaS deps
Automate collection
Measure
Risks
Imprecise metric
definition
Manual collection
“What does it mean?”
Tools
System (ganglia, collectd, munin, nagios, etc.)
Code (metrics, statsd)
SaaS (Datadog et al.)

47. Dev Release
Measure & Log
Monitor
Change
Investigate Alert
Fix || Escalate

48. Purpose
Collect meaningful, timestamped events
Log
Process
All the time
In one place
Access for everyone
Discipline
Risks
TiB of garbage
Non-uniform
timestamps
Non-uniform formats
Tools
log4j et al.
syslog et al.
logstash, splunk
+ Logging-as-a-Service

49. Dev Release Measure & Log
Change
Investigate Alert
Fix || Escalate
Monitor

50. Purpose
Watch actionable events & metrics
Process
Health of the app?
Which metrics for health?
Compute metrics
Metric domain
Access for everyone
Pretty graphs
Monitor
Risks
Non-actionable metrics
Tools
graphite, cubism et al.
+ services

51. Dev Release Measure & Log
Monitor
Change
Investigate
Alert
Fix || Escalate

52. Purpose
Bring human in the loop
when automated fix does not work
Alert
Process
Alert on vital monitors
Add new alerts with new
monitors
Compute metrics from alerts
Ruthlessly edit
Risks
Too many alerts
Become desensitized
Ignore alerts
App crashes for real
Pendulum swings back
Tools
nagios
+ services

53. Dev Release Measure & Log
Monitor
Change
Investigate Alert
Fix || Escalate

54. Purpose
Fix issue or find someone who can
Process
(fix) capture actions as soon as
possible (while or shortly after)
(fix) runbooks
(fix) automate fixes
(escalation) on-call rotation
(escalation) agree on rules
Fix || Escalate
Risks
Burn out
Tools
PagerDuty
Bug tracker

55. Dev Release Measure & Log
Monitor
Alert
Change
Investigate
Fix || Escalate

56. Purpose
Collect evidence
Reconstruct what happened
Process
Start where/when problem 1st detected
Work your way from there
Capture relevant graphs/logs
Investigate
Risks
Missing the starting point
Lagging events/metrics
Low-level events/metrics
Blame game
Tools
Post-mortems

57. Dev Release Measure & Log
Monitor
Investigate Alert
Fix || Escalate
Change

58. Change
Purpose
Fewer alerts
Better service
Process
Change infrastructure, code
Infrastructure == code
Add/Edit monitors & alerts
Risks
ad-hoc changes
Tools
...

59. WebOps
and this is what I do

60. Dev Release Measure & Log
Monitor
Change
Questions?
Comments?
@alq
Investigate Alert
Fix || Escalate