Keynote given at QS-Barcamp 2021 You know your own product team needs to improve when it comes to testing and quality, yet you are not sure how to tackle this challenge? Many teams out there feel the need to grow a quality mindset, considering this to be a major stumbling block in their environments. At my current company, tech leadership had a clear mission: improve the testing and quality culture of our product teams by leveling up knowledge, skills and practices. The problem: how to get there? Experiments to the rescue! In a complex environment like ours, no one could tell what would work, so we had to try things out. I designed an experiment to increase transparency, raise awareness, and especially drive experimentation in individual teams to grow a quality culture. Together with a newly formed working group, I supported four pilot teams to design, run and evaluate their own measurable experiments in their own specific context. These teams tried various approaches like exploratory testing, pair and ensemble testing, or documenting architectural decisions. In this talk I will share our lessons learned, the outcome and impact of these experiments, and what influenced the next experiments to try. Get inspired by our continuous journey towards making experiments a part of team culture across the company! Takeaways: - Understand why small, frugal experiments are crucial in complex environments - Learn how to design your own measurable experiments, in your team or globally - Get tips on how to run and evaluate experiments - Gain insights on how to learn from experiment results and decide on next steps - Start a movement towards an experiment-driven quality culture

1. GROWING AN EXPERIMENT-DRIVEN
QUALITY CULTURE LISI HOCKE
@lisihocke

2. THIS IS A STORY OF EXPERIMENTATION
@lisihocke

3. THIS IS A STORY OF EXPERIMENTATION
@lisihocke

4. THIS IS A STORY OF EXPERIMENTATION
@lisihocke

5. THIS IS A STORY OF EXPERIMENTATION
@lisihocke

6. THIS IS A STORY OF EXPERIMENTATION
@lisihocke

7. THIS IS A STORY OF EXPERIMENTATION
@lisihocke

8. WHAT’S IN THE BOX? OUR STARTING
POINT
@lisihocke

9. @lisihocke
THIS IS US AT FLIXMOBILITY

10. @lisihocke
IT‘S TIME TO FOCUS ON QUALITY

11. THE CHALLENGE
@lisihocke

12. THE CHALLENGE
@lisihocke
Quality is value

13. THE CHALLENGE
@lisihocke
Quality is value
Lack of transparency

14. THE CHALLENGE
@lisihocke
Quality is value
Lack of transparency
Knowledge is not common

15. THE CHALLENGE
@lisihocke
Quality is value
Lack of transparency
Knowledge is not common
The need to improve

16. THE CHALLENGE
@lisihocke
Quality is value
Lack of transparency
Knowledge is not common
The need to improve
Let’s tackle it before it tackles us

17. OUR MISSION
@lisihocke
Improve the testing & quality culture of our product
teams by leveling up knowledge, skills and practices

18. LET‘S EXPERIMENT! AN EXPERIMENT
OF EXPERIMENTS
@lisihocke

19. OUR HYPOTHESIS
@lisihocke

20. OUR HYPOTHESIS
With an experimental group of 4 product teams, we believe that
• creating transparency where they currently are regarding testing and
quality as well as identifying problems,
• raising awareness and knowledge of the options at hand to probe the
problems, and
• running first experiments to learn what works in their context to enable
them to rinse and repeat,
will result in increased maturity of each team with regards to testing and
quality.
…
@lisihocke

21. OUR HYPOTHESIS
…
We'll know we have succeeded when each of the 4 teams
• has a clear, shared test strategy in place reflecting the current
situation as well as identified problems,
• has gained further knowledge and skills regarding testing and quality
through a workshop, and
• has adopted 1 new practice in their testing and quality culture that
proved valuable in their context.
@lisihocke

22. EXIT CRITERIA
@lisihocke

23. EXIT CRITERIA
The experiment will stop and be re-evaluated in the following cases,
whichever comes first:
• No problem there! We find out that the chosen product teams
already have transparency, awareness and experiments in place to
improve their testing and quality.
• Yay, success :) The hypothesis success criteria are fulfilled as defined
above.
• Didn't work out - let's rethink it. It is November 1, 2019 (even
though success criteria are not fulfilled).
@lisihocke

24. @lisihocke
THE HIGH LEVEL PLAN

25. @lisihocke
THE HIGH LEVEL PLAN
Find contributors

26. @lisihocke
THE HIGH LEVEL PLAN
Find contributors
Align among contributors

27. @lisihocke
THE HIGH LEVEL PLAN
Find contributors
Align among contributors
Select product teams as pilot teams

28. @lisihocke
THE HIGH LEVEL PLAN
Find contributors
Align among contributors
Select product teams as pilot teams
Work with each team

29. @lisihocke
THE HIGH LEVEL PLAN
Find contributors
Align among contributors
Select product teams as pilot teams
Work with each team
Evaluate the hypothesis success criteria

30. @lisihocke
CALLING FOR CONTRIBUTORS

31. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS

32. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.

33. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.
It's an opportunity to learn.

34. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.
It's an opportunity to learn.
Let's improve together.

35. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.
It's an opportunity to learn.
Let's improve together.
It's about you, the team!

36. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.
It's an opportunity to learn.
Let's improve together.
It's about you, the team!
We support you, custom-tailored to your needs.

37. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.
It's an opportunity to learn.
Let's improve together.
It's about you, the team!
We support you, custom-tailored to your needs.
Senior tech leadership has our back!

38. @lisihocke
SELECT PILOT TEAMS – LET’S TALK!

39. @lisihocke
SELECT PILOT TEAMS – LET’S TALK!
Building quality in

40. @lisihocke
SELECT PILOT TEAMS – LET’S TALK!
Building quality in
What's working

41. @lisihocke
SELECT PILOT TEAMS – LET’S TALK!
Building quality in
What's working
What's worrying

42. @lisihocke
SELECT PILOT TEAMS – LET’S TALK!
Building quality in
What's working
What's worrying
Eagerness

43. @lisihocke
SELECT PILOT TEAMS – LET’S TALK!
Building quality in
What's working
What's worrying
Eagerness
Readiness

44. @lisihocke
READY, SET GO! LET’S WORK WITH THE TEAMS

45. @lisihocke
TRANSPARENCY

46. @lisihocke
TRANSPARENCY
Desired outcome: The team has a clear, shared test strategy in place
reflecting the current situation as well as identified problems.

47. @lisihocke
TRANSPARENCY: PAIN POINTS
Shout-out to Ashley Hunsberger & Lisa Crispin
Visualize current workflow & identify problems

48. @lisihocke
TRANSPARENCY: RISK
Shout-out to Beren van Daele & Marcel Gehlen
RiskStorming to identify context-specific product risks

49. @lisihocke
TRANSPARENCY: STATUS QUO
Document current test strategy

50. @lisihocke
AWARENESS

51. @lisihocke
AWARENESS
Desired outcome: The team has gained further knowledge
and skills regarding testing and quality through a workshop.

52. @lisihocke
TEAM EXPERIMENTS

53. @lisihocke
TEAM EXPERIMENTS
Desired outcome: The team has adopted 1 new practice in their
testing and quality culture that proved valuable in their context.

54. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges

55. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next

56. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions

57. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions
Pick one option to try out

58. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions
Pick one option to try out
Design the experiment

59. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions
Pick one option to try out
Design the experiment
Run the experiment

60. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions
Pick one option to try out
Design the experiment
Run the experiment
Evaluate the experiment

61. @lisihocke
TEAM EXPERIMENTS
Challenge: <short challenge description>
Hypothesis:
We believe that <doing X>
will result in <desired outcome>.
We'll know we have succeeded when <measurable criterion is
achieved>.
Experiment: <concrete details like planned actions, experiment run time,
exit criteria, etc.>

62. TEAM EXPERIMENTS
@lisihocke

63. TIME TO EVALUATE LESSONS
LEARNED
@lisihocke

64. HYPOTHESIS EVALUATION
@lisihocke

65. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
HYPOTHESIS EVALUATION
@lisihocke

66. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
HYPOTHESIS EVALUATION
@lisihocke

67. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
HYPOTHESIS EVALUATION
@lisihocke

68. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
→ YES ☺
HYPOTHESIS EVALUATION
@lisihocke

69. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
All 4 teams have adopted 1 new practice in their testing
and quality culture that proved valuable in their context
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
→ YES ☺
HYPOTHESIS EVALUATION
@lisihocke

70. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
All 4 teams have adopted 1 new practice in their testing
and quality culture that proved valuable in their context
All 4 teams have adopted 1 new practice in their testing
and quality culture that proved valuable in their context
→ Only 2 
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
→ YES ☺
HYPOTHESIS EVALUATION
@lisihocke

71. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE

72. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality

73. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills

74. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
☺ Teams inspired to improve their practices

75. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
☺ Teams inspired to improve their practices
 Not the whole team was on board, silos remained

76. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
☺ Teams inspired to improve their practices
 Not the whole team was on board, silos remained
 Not all people opened up for new concepts, misconceptions prevailed

77. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
☺ Teams inspired to improve their practices
 Not the whole team was on board, silos remained
 Not all people opened up for new concepts, misconceptions prevailed
 Teams struggled with experiments, fell back into everyday business

78. @lisihocke
TOO MUCH EFFORT TO SCALE THIS APPROACH

79. HOW TO SCALE? A LIGHTWEIGHT
APPROACH
@lisihocke

80. @lisihocke
A SECOND EXPERIMENT

81. @lisihocke
A SECOND EXPERIMENT
We believe that if we learn about the testing strategy of product teams
and provide them peer feedback regarding our perspective on their
challenges, then this will
• help the teams build increased self-awareness regarding testing and
quality, and
• help us understand our current testing and quality culture.
We'll know we have proven the hypothesis when the teams as well as we
are able to explain the difference between their approach and our
suggestions and why it might differ from expectations.

82. A SECOND PLAN
@lisihocke

83. A SECOND PLAN
Align within the initiative group on core values and select product teams
@lisihocke

84. A SECOND PLAN
Align within the initiative group on core values and select product teams
Interview, present and provide feedback on the teams’ testing strategy
@lisihocke

85. A SECOND PLAN
Align within the initiative group on core values and select product teams
Interview, present and provide feedback on the teams’ testing strategy
Evaluate what we learned in this experiment and propose the next steps
@lisihocke

86. @lisihocke
CORE VALUES TO BASE OUR FEEDBACK ON

87. @lisihocke
CORE VALUES TO BASE OUR FEEDBACK ON
Quality is Team Responsibility

88. @lisihocke
CORE VALUES TO BASE OUR FEEDBACK ON
Quality is Team Responsibility
Tackle the Unknown, Automate the Known

89. @lisihocke
CORE VALUES TO BASE OUR FEEDBACK ON
Quality is Team Responsibility
Tackle the Unknown, Automate the Known
Invest in Quality and Quality Will Work for You

90. @lisihocke
INTERVIEW TEAM MEMBERS ON THEIR TEAM’S
TESTING STRATEGY

91. @lisihocke
FEEDBACK SESSION & EXPERIMENT EVALUATION

92. @lisihocke
FEEDBACK SESSION & EXPERIMENT EVALUATION

93. @lisihocke
A PROMISING APPROACH!

94. @lisihocke
… YET THEN LIFE HAPPENED

95. GROWING A CULTURE BELIEFS & NEEDS
@lisihocke

96. @lisihocke
PRODUCT TEAMS: WE NEED YOU TO SOLVE OUR
PROBLEMS!

97. @lisihocke
PRODUCT TEAMS: WE NEED YOU TO SOLVE OUR
PROBLEMS!
Or rather: we need support so we become aware of our
options and enablement so we can solve our own problems

98. @lisihocke
→ INSPIRE & BUILD ON PEOPLE’S CURIOSITY

99. @lisihocke
→ INSPIRE & BUILD ON PEOPLE’S CURIOSITY
Experimenting with tech talks and personal conversations
inside a domain, creating a “pull”

100. @lisihocke
TECH LEADS: WE NEED DATA & GLOBAL METRICS!

101. Or rather: we need to know which impact we want to have in
our specific contexts and what information we need to have it
@lisihocke
TECH LEADS: WE NEED DATA & GLOBAL METRICS!

102. @lisihocke
→ UNDERSTAND NEEDS & FIND HELPFUL
INDICATORS

103. @lisihocke
→ UNDERSTAND NEEDS & FIND HELPFUL
INDICATORS
Experimenting together with tech leads on perceived quality
needs and indicators helping them make better decisions

104. @lisihocke
→ UNDERSTAND NEEDS & FIND HELPFUL
INDICATORS
Experimenting together with tech leads on perceived quality
needs and indicators helping them make better decisions
Shout-out to Michael Kutz, João Proença, Toyer Mamoojee,
Emanuil Slavov and Jenny Bramble

105. @lisihocke
CONTRIBUTORS: WE NEED QUALITY TO START
WITHIN THE TEAMS!

106. @lisihocke
CONTRIBUTORS: WE NEED QUALITY TO START
WITHIN THE TEAMS!
Or rather: we need to change the system so behaviors
contributing to quality get rewarded and valued

107. @lisihocke
→ SHAPE THE SYSTEM & REWARD BEHAVIORS
CONTRIBUTING TO QUALITY

108. @lisihocke
→ SHAPE THE SYSTEM & REWARD BEHAVIORS
CONTRIBUTING TO QUALITY
Experimenting on shaping the system through leadership
workshops, building a foundation for quality to emerge

109. @lisihocke
QUALITY IS ON ALL OF US

110. @lisihocke
QUALITY IS ON ALL OF US
Create transparency on the status quo, pain points and needs
→ The teams, the leadership, the system

111. @lisihocke
QUALITY IS ON ALL OF US
Create transparency on the status quo, pain points and needs
→ The teams, the leadership, the system
Raise awareness regarding the options at hand to improve

112. @lisihocke
QUALITY IS ON ALL OF US
Create transparency on the status quo, pain points and needs
→ The teams, the leadership, the system
Raise awareness regarding the options at hand to improve
Experiment in your context to grow a quality culture - together

113. THANK YOU! www.lisihocke.com
@lisihocke

114. THANK YOU! www.lisihocke.com
@lisihocke

115. THANK YOU! www.lisihocke.com
@lisihocke

116. THANK YOU! www.lisihocke.com
@lisihocke

117. THANK YOU! www.lisihocke.com
@lisihocke

118. THANK YOU! www.lisihocke.com
@lisihocke

119. THANK YOU! www.lisihocke.com
@lisihocke