Bad monitoring, alerting and logging has made Gil Zellner very frustrated in some of his previous positions. It seems that almost nobody gets this exactly right. This will be a talk about the most annoying issues he has come across and advice for how to fix them.

1. Relaxing picture of Yoga

2. PagerDuty Alert

4. hunt through logs for 2
hours

8. How to not burn out your
production team
Gil Zellner
(CloudifyDev at Gigaspaces)
Twitter: @Heathenaspargus

9. Who am I?
Now:
Past:
@Heathenaspargus

10. @Heathenaspargus

11. @Heathenaspargus

12. cost of hiring new employee is 1.5-3x their monthly
salary
@Heathenaspargus

13. @Heathenaspargus

14. @Heathenaspargus

15. Next day
@Heathenaspargus

16. @Heathenaspargus

17. @Heathenaspargus

18. frustration - I am unable to complete my task
@Heathenaspargus

20. Time spent inefficiently
@Heathenaspargus

21. Repetitive tasks
@Heathenaspargus

22. Working Alone
@Heathenaspargus

23. Yak
Shaving
@Heathenaspargus

24. https://www.ergoflex.co.uk/blog/category/sleep-research/sleeponomics-could-sleep-deprivation-be-the-real-reason-politicians-make-bad-decisions
@Heathenaspargus

25. Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus

26. Mandatory Half day-off after night production issue
@Heathenaspargus

27. Allocate weekly time to resolve or automate issues that kept us up at night
@Heathenaspargus

28. Wider rotation (more people do on-call)
@Heathenaspargus

29. Knowledge Matrix
Deploy System Mobile Link Backend
Gil V V
Karen V V
Ari V V
@Heathenaspargus

30. @Heathenaspargus

31. Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus

34. @Heathenaspargus

35. solution: alert only things that meet the following
criteria:
1) Alert on symptoms, not suspected "causes"
2) Actionable
3) Business breaking
@Heathenaspargus

36. Alerte générale!
@Heathenaspargus

37. Solution: direct alerts to relevant parties
@Heathenaspargus

38. @Heathenaspargus

40. @Heathenaspargus

41. @Heathenaspargus

42. @Heathenaspargus

43. What are your KPIs ?
@Heathenaspargus

44. @Heathenaspargus

45. Netflix stream starts per second
@Heathenaspargus

46. Picking how to measure things

48. Diagnosis
@Heathenaspargus

49. Make heal script
@Heathenaspargus

50. @Heathenaspargus

51. @Heathenaspargus

52. Facebook Auto Remediation
https://www.facebook.com/notes/facebook-engineering/making-facebook-self-healing/10150275248698920
@Heathenaspargus

53. Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus

55. @Heathenaspargus

56. Bad artists copy, great artists steal
email:
Gil.Zellner@gmail.com
Twitter:
@Heathenaspargus