The more we are connected and the more others are connected to us, the more important reliability of your sites becomes. Site Reliability Engineering is an engineering discipline devoted to helping an organization sustainably achieve the appropriate level of reliability in their systems, services, and products. But what does this mean, and how do get started with this? In this session I will talk about the concepts of Site Reliability Engineering and use Microsoft Azure to implement some of the concepts and practices

1. WHEN "WE ARE DOWN" IS NOT GOOD ENOUGH
SITE RELIABILITY ENGINEERING (SRE) IN AZURE
RENÉ VAN OSNABRUGGE
@RENEVO

2. @RENEVO

3. LET’S COUNT
TOGETHER @RENEVO

4. OR NOT @RENEVO

5. 99.9%
@RENEVO

6. AVAILABILITY IN HUMAN WORDS
per year
per
quarter
per
month per week per day per hour
90% 36.5 days 9 days 3 days 16.8 hours 2.4 hours 6 minutes
95% 18.25 days 4.5 days 1.5 days 8.4 hours 1.2 hours 3 minutes
99% 3.65 days 21.6 hours 7.2 hours 1.68 hours 14.4 minutes 36 seconds
99.5% 1.83 days 10.8 hours 3.6 hours 50.4 minutes 7.20 minutes 18 seconds
99.9% 8.76 hours 2.16 hours 43.2 minutes 10.1 minutes 1.44 minutes 3.6 seconds
99.95% 4.38 hours 1.08 hours 21.6 minutes 5.04 minutes 43.2 seconds 1.8 seconds
99.99% 52.6 minutes 12.96 minutes 4.32 minutes 60.5 seconds 8.64 seconds 0.36 seconds
99.999% 5.26 minutes 1.30 minutes 25.9 seconds 6.05 seconds 0.87 seconds 0.04 seconds
@RENEVO

7. WHAT DO OTHERS HAVE?
Azure DevOps
99.9%
Azure Policy
----
Gmail
99.9%
ex. planned
Azure AAD
99.9%
Outlook.com
----
Azure Functions
99.95%
Google Maps API
99.9%
ING Banking
----
Slack
99.99%
@RENEVO

8. OPS VERSUS DEV @RENEVO

9. WHAT WE HEAR FROM OPS
“We need to have a launch review!”
“Please let the CAB approve first”
“This is our change management checklist”
“We should validate first in Test and Pre-Prod”
@RENEVO

10. WHAT WE HEAR FROM DEV
“We do not launch big changes, this is just a flag flip”
“This is a hotfix!”
“This is just a UI change... No big thing”
“Only a 20% experiment.”
@RENEVO

11. CONFLICTING KPI’S
DOES NOT WORK
@RENEVO

12. 2 PARALLEL MOVEMENTS
EMERGED
@RENEVO

13. “DEVOPS IS THE UNION OF PEOPLE,
PROCESS AND PRODUCTS TO ENABLE
CONTINUOUS DELIVERY OF VALUE TO
OUR END-USERS”
DONOVAN BROWN
@RENEVO

14. @RENEVO

15. SITE RELIABILITY ENGINEERING IS AN
ENGINEERING DISCIPLINE DEVOTED TO
HELPING AN ORGANIZATION SUSTAINABLY
ACHIEVE THE APPROPRIATE LEVEL OF
RELIABILITY IN THEIR SYSTEMS, SERVICES,
AND PRODUCTS.
@RENEVO

16. public class SRE: DevOps {
…
}
@RENEVO

17. DEVOPS SRE
REDUCE ORGANIZATIONAL SILOS
ACCEPT FAILURE AS NORMAL
IMPLEMENT GRADUAL CHANGE
LEVERAGE TOOLING & AUTOMATION
MEASURE EVERYTHING
SHARE OWNERSHIP
SLO’S & BLAMELESS LEARNING
REVIEWS
REDUCE COST OF FAILURE
AUTOMATION AS 1st CLASS CITIZEN
MEASURE TOIL AND RELIABILITY
@RENEVO

18. SRE focusses on the reliability of your site
Product team focusses on the business value
They share ownership
They are all engineers
@RENEVO

19. THE PRICE OF GREATNESS IS RESPONSIBILITY
WINSTON CHURCHILL
@RENEVO

20. S L *
SLI SLO SLA
@RENEVO

21. S L I
SERVICE LEVEL INDICATOR
@RENEVO

22. SLIS ARE A RATIO/PROPORTION
# of successful HTTP calls/# of HTTP calls
# of operations that completed in < 10ms/# of operations
# of “full quality responses”/# of responses
# of records processed/# of records
Ratio * 100 = % proportion
@RENEVO

23. SLIS ARE A RATIO/PROPORTION & HOW
# of successful HTTP calls/# of HTTP calls
# of operations that completed in < 10 ms/# of operations
…as measured at the client
…as measured at the load balancer
@RENEVO

24. S L O
SERVICE LEVEL OBJECTIVE
@RENEVO

25. BASIC SLO RECIPE
THE THING
HTTP requests
Storage checks
Operations
SLI PROPORTIONS
”Successful 50% of the time”
“Can read the data 99.9% of the time”
“Return in 10ms 90% of the time”
TIME STATEMENT
“In the last ten minute period”
”During last quarter”
“In the previous rolling 30 day period”
90% of HTTP requests
as reported by the
load balancer
succeeded in the last
30 day window.
SERVICE LEVEL OBJECTIVE (SLO)
@RENEVO

26. AGREEMENT BETWEEN THE PRODUCT TEAM AND SRE
@RENEVO

27. SLA
UPTIME
COMMERCIAL AGREEMENT
WITH CUSTOMER
SLOSLA
@RENEVO

28. ERROR BUDGETS
UPTIME
ERROR BUDGET
SLO
@RENEVO

29. ERROR BUDGETS
UPTIME
ERROR BUDGET
SLO
@RENEVO

30. ERROR BUDGETS
UPTIME
NO
ERROR BUDGET
NO
RELEASE
SLO
@RENEVO

31. DEMO – SLI AND SLO IN LOG ANALYTICS
@RENEVO

32. WHAT DOES DOWN ACTUALLY MEAN?
@RENEVO

33. RELIABILITY
AVAILABILITY LATENCY
THROUGHPUT
CORRECTNESS
FRESHNESS
COVERAGE
FIDELITY
DURABILITY
IT MUST BE ABOUT THE CUSTOMER, NOT THE
SOFTWARE COMPONENT @RENEVO

34. WHAT DOES AN SRE DO?
TIME ALLOCATION OF SRE
50%
Operational Work
50%
Project Work
• Incidents
• Tickets
• Operational work
• Project Work in Product Teams
• Add Service Features
• Reduce future toil
@RENEVO

35. TOIL
• Manual
• Repetitive
• Automatable
• Tactical
• Devoid of enduring value
• cales linearly as a service grows
@RENEVO

36. @RENEVO

37. DEMO – USING AZURE BOTS TO REDUCE TOIL
@RENEVO

38. OPERATIONAL WORK
@RENEVO

39. DETECT @RENEVO

40. WHY MONITOR?
Are my apps and infrastructure doing what I expect?
Are my apps and infrastructure doing what others expect?
@RENEVO

41. 3 KINDS OF MONITORING OUTPUT
ALERTS
TICKETS
LOGGING
@RENEVO

42. RESPOND @RENEVO

43. WHO RESPONDS?
SHARED RESPONSIBILITY OF SRE
@RENEVO

44. ALERT FATIGUE
A serious problem
@RENEVO

45. ACTIONABLE ALERTS
• Alerts are not: logs, notifications, heartbeats, normal
• Needs a human to investigate (and ideally resolve)
• Right human(s) (scope)
• Humans not automation
• Crucial details:
• Where the alert is coming from
• What expectation was violated
• Why this is an issue (for the customer)
• Steps to resolve the problem (or at least a specific pointer)
@RENEVO

46. IT IS ABOUT THE USER!
@RENEVO

47. DEMO – ACTIONABLE ALERTS
@RENEVO

48. REMEDIATE @RENEVO

49. MTTR
On average, how long does it take to restore service when a
service incident occurs? @RENEVO

50. CONTINUOUS DELIVERY AS
A MEANS TO AN END
@RENEVO

51. ANALYZE @RENEVO

52. @RENEVO

53. BLAMELESS LEARNING REVIEW
INVOLVE EVERYONE
DO IT ASAP
FOCUS ON WHAT HAPPENED
WHAT DID THEY KNOW
WHEN DID THEY KNOW
HOW DID IT MAKE SENSE
CREATE A TIMELINE
WHAT WAS THE THOUGHT PROCESS?
@RENEVO

54. OUTCOMES
COUNTERMEASURES
LONG TERM
SHORT TERM
INTERNAL SUMMARY
EXTERNAL SUMMARY
@RENEVO

55. DEMO – TROUBLESHOOTING GUIDES
@RENEVO

56. READINESS @RENEVO

57. MISTAKES ARE MEANT FOR LEARNING
NOT FOR REPEATING
@RENEVO

58. The “Paul” attack
Response training (game days)
On-call rotations
Escalation paths
Communication channels
Chaos engineering/testing in production
BE READY
@RENEVO

59. WRAP UP
CONSIDER YOUR SLA CAREFULLY
SRE IS NOT A NEW OPS DEPARTMENT
USE SLO’S AND ERROR BUDGETS
MEASURE EVERYTHING
CLOSE THE INCIDENT LOOP WITH LEARNING
@RENEVO

60. René van Osnabrugge
Xpirit Netherlands
@renevo
rvanosnabrugge@xpirit.com
https://roadtoalm.com
Attributions
Pictures: https://unsplash.com / https://www.flickr.com/photos/wocintechchat
Gifs: https://giphy.com
Music: https://open.spotify.com/user/rvanosnabrugge/playlist/
0BWgsNPM5iwgk8ZGlMHeoY?si=l9-tV8FTR8S1J7AbKBz-KA
Video: https://www.youtube.com/watch?v=SGAnLY46zAk
Thanks: Martijn, Xpirit