There you are… an agile company building cloud native microservices, everything is automated and you are deploying multiple times a day. You think you ticked all the DevOps boxes but when does the money start flowing in? Maybe it's because your users don't like the way your software works, maybe you don't actually have any users. How do you know? Traditional monitoring tools are dead. They can't give you the insights you need to see if your distributed system is still working or not. "Observability driven development" is a way to focus on building observable systems that focus on seeing if your system is actually working and delivering value to your customers. By being able to observe your production environment you are able to learn from it, experiment on it (in production) and in the end improve it. In this session we'll discuss what it means to create observable systems and how you can start adding observability to your own systems which help everyone from developers to product owners to make better decisions in adding value to their software.

1. OBSERVABILITY DRIVEN
DEVELOPMENT
GEERT VAN DER CRUIJSEN
@GEERTVDC

2. GEERT VAN DER CRUIJSEN
@GEERTVDC
CLOUDNATIVEARCHITECT FULLCYCLEDEVELOPERDEVOPSCOACH
CBO (CHIEF BEER OFFICER) AT XPIRIT NETHERLANDS

3. I LOVE BELGIUM
+
+
=
=

4. SINCE WE ALL LOVE BEER
I BROUGHT SOME DUTCH BEERS!!

5. SINCE WE ALL LOVE BEER
I BROUGHT SOME DUTCH BEERS!!
FIND THIS LOGO DURING MY
PRESENTATION, TAKE A PICTURE,
TWEET IT
MENTION/FOLLOW @GEERTVDC
AND WIN BEER!

6. I HAVE TO MAKE
A CONFESSION
@GEERTVDC

7. I HAVE TO MAKE
A CONFESSION
I TEST IN
PRODUCTION
@GEERTVDC

8. I TEST IN
PRODUCTION
I’M NOT LIKE
THIS GUY THOUGH
@GEERTVDC

9. TODAY’S PREACH
YOU SHOULD TEST IN
PRODUCTION TOO
@GEERTVDC

10. YOU SHOULD TEST IN
PRODUCTION TOO
STOP BEING AFRAID
OF PRODUCTION!
@GEERTVDC

11. WHO’S DOING
AGILE OR DEVOPS?
@GEERTVDC

12. WHO’S DOING
AGILE OR DEVOPS?
COMMON
AGILE / DEVOPS
MISTAKES
@GEERTVDC

13. FOCUS ON SPEED?
@GEERTVDC

14. DO YOU WANT FAST WHEN YOU’RE
NOT GOING IN THE RIGHT DIRECTION?
@GEERTVDC

15. TEST IN PRODUCTION
USER BEHAVIOR
A/B TESTING
EXPERIMENTS
@GEERTVDC

16. BEING ABLE TO BRAKE AND STEER
THAT IS WHAT MAKES YOU GO FASTER!
@GEERTVDC

17. DEVOPS IS THE UNION OF PEOPLE, PROCESS,
AND PRODUCTS TO ENABLE CONTINUOUS
DELIVERY OF VALUE TO OUR END USERS.
DONOVAN BROWN
@GEERTVDC

18. DEVOPS IS THE UNION OF PEOPLE, PROCESS,
AND PRODUCTS TO ENABLE CONTINUOUS
DELIVERY OF VALUE TO OUR END USERS.
DONOVAN BROWN
@GEERTVDC

19. VALUE IS ONLY VALUE WHEN
IT’S RUNNING IN PRODUCTION
@GEERTVDC

20. VALUE IS ONLY VALUE WHEN
IT’S RUNNING IN PRODUCTION
@GEERTVDC

21. TEST IN PRODUCTION
CANARY RELEASING
RING BASED DEPLOYMENTS
MULTI REGION
CHAOS ENGINEERING
SHADOW TESTING@GEERTVDC

22. BUT I USE STAGING?@GEERTVDC

23. BUT I USE STAGING?
DOES STAGING HAVE REAL DATA?
DOES STAGING HAVE REAL USERS?
DOES STAGING REPRESENT PRODUCTION ENOUGH?
HOW MUCH TIME DO YOU SPEND ON STAGING?

24. WHAT IS KEY TO TESTING ON PROD?
OBSERVABILITY
@GEERTVDC

25. OBSERVABILITY
“OBSERVABILITY IS A MEASURE OF HOW
WELL INTERNAL STATES OF A SYSTEM CAN
BE INFERRED FROM KNOWLEDGE OF ITS
EXTERNAL OUTPUTS”
CONTROL THEORY
@GEERTVDC

26. WHAT IS THE DIFFERENCE
WITH MONITORING?
@GEERTVDC

27. MONITORING
KNOWN UNKNOWNS
OBSERVABILITY
UNKNOWN UNKNOWNS
@GEERTVDC

28. COMPLEX
APPLICATION LANDSCAPES
DISTRIBUTED SYSTEMS – MICROSERVICES – CLOUD

29. “IN A COMPLEX LANDSCAPE YOUR
APPLICATION IS NEVER FULLY UP”
@GEERTVDC

30. MICROSERVICES
TRADITIONAL MONITORING
TOOLS ARE DEAD
@GEERTVDC

31. MEASURE
USER IMPACT
@GEERTVDC

32. MEASURE
USER IMPACT
https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a
@GEERTVDC

33. RELIABILITY
AVAILABILITY LATENCY
THROUGHPUT
CORRECTNESS
FRESHNESS
COVERAGE
QUALITY
DURABILITY
RELIABILITY
@GEERTVDC

34. FAIL OPEN
PARTIAL FAILURE MODE
@GEERTVDC

35. OBSERVABILITY IS THE
KEY TO SOFTWARE
OWNERSHIP
@GEERTVDC

36. WE’VE TAUGHT OPS TO DEV
SOURCE CONTROL
INFRASTRUCTURE AS CODE
AUTOMATION
SCRIPTING
@GEERTVDC

37. TIME HAS COME
DEVS GET PROD ACCESS
@GEERTVDC

38. TIME HAS COME
DEVS GET PROD ACCESS
DEVS TAKE OWNERSHIP
@GEERTVDC

39. TIME HAS COME
DEVS GET PROD ACCESS
DEVS TAKE OWNERSHIP
DEVS TAKE ON CALL!
@GEERTVDC

40. DEVOPS CYCLE
@GEERTVDC

41. DEVOPS CYCLE
@GEERTVDC

42. BUSINESS + DEV
IT OPERATIONS
@GEERTVDC

43. BUSINESS + DEV
IT OPERATIONS
IMPROVE THE COMPANY
@GEERTVDC

44. OBSERVABILITY
CONNECT DEV TO BUSINESS
OBSERVABILITY
CONNECT DEV TO OPERATIONS
@GEERTVDC

45. 3 PILLARS OF
OBSERVABILITY
@GEERTVDC

46. 3 PILLARS OF
OBSERVABILITY
LOGS METRICS TRACES
@GEERTVDC

47. LOGGING
EXAMPLE: REQUEST DURATION
SERVICE REQUEST X FOR USER Y
TOOK 50 MILLISECONDS
@GEERTVDC

48. LOGGING
EASY TO GENERATE, HARD TO QUERY?
@GEERTVDC

49. STRUCTURED LOGGING
Log.Information(
“Request by {User} took {Duration}",
user,
duration);
Log.Information(“Request by userA took 35ms");
FROM
TO
@GEERTVDC

50. STRUCTURED LOGGING
GENERATE LOGS
SERILOG
APPLICATION INSIGHTS
NLOG
@GEERTVDC

51. STRUCTURED LOGGING
GENERATE LOGS STORE & QUERY LOGS
AZURE
LOG ANALYTICS
SERILOG
APPLICATION INSIGHTS
NLOG
@GEERTVDC

52. LOGGING
SHOULD YOU SAMPLE?
STORAGE == MONEY
AUDIT LOGS DO NOT SAMPLE
OPERATIONAL LOGS DO SAMPLE
DYNAMIC SAMPLING
@GEERTVDC

53. METRICS
AGGREGATE INFORMATION INTO TIME SERIES
CREATE REAL TIME GRAPHS OR HISTOGRAPHS
CHEAPER TO STORE
@GEERTVDC

54. METRICS
EXAMPLE: REQUEST DURATION
50 MILLISECONDS REQUEST IS 15 MILLISECONDS
HIGHER THAN AVERAGE
@GEERTVDC

55. METRICS
EXAMPLE: REQUEST DURATION
50 MILLISECONDS REQUEST IS 15 MILLISECONDS
HIGHER THAN AVERAGE
IN MECHELEN
ON FRIDAYS
PEOPLE WHO LIKE BEER
@GEERTVDC

56. DISTRIBUTED TRACING
EXAMPLE: REQUEST DURATION
WHY DID THIS REQUEST TAKE 50
MILLISECONDS -> IT CALLED DB, OTHER
SERVICES?
@GEERTVDC

57. DISTRIBUTED TRACING
APPLICATION FLOW FROM FRONT TO BACK
USER SESSION
TRANSACTION
AMOUNT OF DATA IN MICROSERVICE
LANDSCAPE?
@GEERTVDC

58. @GEERTVDC
DISTRIBUTED TRACING

59. WHAT TO MEASURE?
USE RED
@GEERTVDC
FOCUS ON YOUR USERS
LOG ALL USER EVENTS

60. USE RED
UTILIZATION
SATURATION
ERROR RATE
RESOURCE SCOPE
@GEERTVDC

61. USE RED
UTILIZATION
SATURATION
ERROR RATE
RATE
ERRORS
DURATION
RESOURCE SCOPE REQUEST SCOPE
@GEERTVDC

62. FEATURE FLAGS
If(_featureFlag.IsEnabled(“NewCheckoutFlow”)
{
log.Information(“NewCheckoutFlow feature used”);
ExecuteNewCheckoutFlow();
}
else
{
log.Information(“LegacyCheckout feature used”);
ExecuteLegacyCheckoutFlow();
}
@GEERTVDC

63. FEATURE FLAGS

64. FEATURE FLAGS
INITIAL DEPLOYMENT

65. FEATURE FLAGS
INITIAL DEPLOYMENT
BUG FOUND

66. FEATURE FLAGS
INITIAL DEPLOYMENT
BUG FOUND
SOLVED THE BUG

67. FEATURE FLAGS
INITIAL DEPLOYMENT
BUG FOUND
SOLVED THE BUG
ROLL OUT TO MORE USERS

68. FEATURE FLAGS
INITIAL DEPLOYMENT
BUG FOUND
SOLVED THE BUG
ROLL OUT TO MORE USERS
REMOVE FEATURE FLAG

69. EXPERIMENT IN PRODUCTION
public bool CanAccess(IUser user)
{
return Scientist.Science<bool>("widget-permissions", experiment =>
{
experiment.Use(() => IsCollaborator(user)); // old way
experiment.Try(() => HasAccess(user)); // new way
}); // returns the control value
}
SCIENTIST.NET
@GEERTVDC https://github.com/scientistproject/Scientist.net

70. FROM OBSERVABILITY
TO OBSERVABILITY DRIVEN DEVELOPMENT
@GEERTVDC

71. TDD WRITE
TESTS
PASS
TESTS
REFACTOR
@GEERTVDC

72. PLAN DESIGN DEVELOP TEST DEPLOY OPERATE
TDD
@GEERTVDC

73. ODDOBSERVABILITY DRIVEN DEVELOPMENT
DEFINE
EXPECTED
OUTCOME
MEASURE
THE
OUTCOME
CHANGE
FEATURE
& KEEP
MEASURING
@GEERTVDC

74. PLAN DESIGN DEVELOP TEST DEPLOY OPERATE
TDD
WHAT IS THE USER IMPACT?
@GEERTVDC
OBSERVABILITY DRIVEN DEVELOPMENT

75. PLAN DESIGN DEVELOP TEST DEPLOY OPERATE
TDD
WHAT IS THE USER IMPACT?
IS THE FEATURE BEHAVING
LIKE WE EXPECTED?@GEERTVDC
OBSERVABILITY DRIVEN DEVELOPMENT

76. PLAN DESIGN DEVELOP TEST DEPLOY OPERATE
OBSERVABILITY DRIVEN DEVELOPMENT
TDD
WHAT IS THE USER IMPACT?
IS THE FEATURE BEHAVING
LIKE WE EXPECTED?
DEPLOYMENT FEEDBACK
@GEERTVDC

77. KNOWING HOW OUR SYSTEM
OPERATES SHOULD BE IN
OUR SYSTEM AS DEVELOPERS
WHAT IS NORMAL?
RELEASE GATES TO NEXT STAGE?
@GEERTVDC

82. SLI
SLO
SLA
@GEERTVDC

83. SLI
SLO
SLA
SERVICE LEVEL INDICATOR
SERVICE LEVEL OBJECTIVE
SERVICE LEVEL AGREEMENT
@GEERTVDC

84. SLI SERVICE LEVEL INDICATOR
QUANTITATIVE MEASURE FOR YOUR SERVICE
AVAILABILITY
ERROR RATE
DURATION
LATENCY
FRESHNESS
@GEERTVDC

85. SLO SERVICE LEVEL OBJECTIVE
TARGET MEASURE FOR A SERVICE
MEASURED BY SLIS
AVAILABILITY OF 99.9% FOR LAST 30 DAYS
@GEERTVDC

86. SLA SERVICE LEVEL AGREEMENT
CONTRACT WITH USERS WITH
CONSEQUENSES WHEN
MISSING YOUR SLO
10% DISCOUNT FOR EACH 0.1%
BELOW AVAILABILITY SLO
@GEERTVDC

87. HOW TO DO THIS IN PRACTICE?
@GEERTVDC

88. HOW TO DO THIS IN PRACTICE?
DEFINE AN SLO
BUILD INDICATORS BY LOGGING / METRICS
BUILD A DASHBOARD – START MEASURING
MAKE CHOICES BASED ON SERVICE LEVEL
LEAVE SLA PART FOR SALES PEOPLE

89. MAKE IT VISIBLE@GEERTVDC

90. MAKE IT VISIBLE
SLO
AVAILABILITY
99.9954%
@GEERTVDC

91. MAKE IT VISIBLE
SLO
AVAILABILITY
99.9954%
RING 0
98%
RING 1
99.91%
RING 2
100%
@GEERTVDC

92. MAKE IT VISIBLE
SLO
AVAILABILITY
99.9954%
RING 0
98%
RING 1
99.91%
RING 2
100%
USER SIGN UP FLOW – 100%
CHECKOUT – 99.91%
SEARCH – 98%
@GEERTVDC

93. MAKE IT VISIBLE
SLO
AVAILABILITY
99.9954%
RING 0
98%
RING 1
99.91%
RING 2
100%
USER SIGN UP FLOW – 100%
CHECKOUT – 99.91%
SEARCH – 98%
CLIENT A - USER SIGN UP FLOW – 100%
CLIENT A - CHECKOUT – 99.91%
CLIENT A - SEARCH – 90%

94. TAKEAWAYS
START SMALL AT KEY AREAS OF YOUR APP
EXPLORE TOOLS
EMBRACE TESTING ON PROD!
FOCUS ON CUSTOMERS
TAKE OWNERSHIP OF CODE
@GEERTVDC

95. @GEERTVDC

96. GEERT VAN DER CRUIJSEN
@GEERTVDC
THANK YOU!
MOBILEFIRSTCLOUDFIRST.NET