Successfully reported this slideshow.

Experimenting our Way to Team-based Testing (Soft test 2019 final version)



Loading in …3
1 of 100
1 of 100

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Experimenting our Way to Team-based Testing (Soft test 2019 final version)

  1. 1. Experimenting our Way to Team-based Testing Rob Meaney & Conor Fitzgerald @RobMeaney @ConorFi #SoftTest
  2. 2. Are You A Lone Tester? @ConorFi
  3. 3. @RobMeaney@RobMeaney Quality is a Battle @ConorFi
  4. 4. @RobMeaney@RobMeaney@ConorFi Weight On Your Shoulders @ConorFi
  5. 5. @RobMeaney@RobMeaney Overwhelmed @ConorFi
  6. 6. @RobMeaney@RobMeaney Last Line Of Defence @RobMeaney
  7. 7. @RobMeaney@RobMeaney@RobMeaney Constantly Scrambling @RobMeaney
  8. 8. @RobMeaney@RobMeaney@RobMeaney@RobMeaney “Quality Assured” @RobMeaney
  9. 9. @RobMeaney A Different Way @RobMeaney
  10. 10. @RobMeaney Whole Team Quality Focus @RobMeaney
  11. 11. @RobMeaney @RobMeaney A New Hope
  12. 12. @RobMeaney Break The Cycle @RobMeaney
  13. 13. Poppulo - Great Place to Work @ConorFi
  14. 14. Latest Tech @ConorFi
  15. 15. Open to New Ideas @ConorFi
  16. 16. Coaching - Big Bang @ConorFi
  17. 17. 3 Years Of Experiments @ConorFi
  18. 18. @RobMeaney@RobMeaney@RobMeaney We Faced Challenges @RobMeaney
  19. 19. @RobMeaney Our Challenges @RobMeaney
  20. 20. @RobMeaney@RobMeaney We Lost 3 Testers @RobMeaney
  21. 21. @RobMeaney@RobMeaney@RobMeaney Inconsistent Approaches @RobMeaney
  22. 22. @RobMeaney@RobMeaney@RobMeaney@RobMeaney Monolith To Microservice @RobMeaney
  23. 23. @RobMeaney@RobMeaney@RobMeaney@RobMeaney Automation Fixation @RobMeaney
  24. 24. @RobMeaney@RobMeaney No Risky Analysis @RobMeaney
  25. 25. @RobMeaney@RobMeaney Very Little Exploration @RobMeaney
  26. 26. @RobMeaney@RobMeaney@RobMeaney@RobMeaney Testers As GateKeepers @RobMeaney
  27. 27. Our Approach: 1. Visualise the Work 2. Socialise Problems 3. Establish a Quality Culture @RobMeaney
  28. 28. @RobMeaney@RobMeaney@RobMeaney 1.Visualise The Work @RobMeaney
  29. 29. Visualising the Work: 1. Quality Indicator Board 2. Department SoS Board 3. Team Kanban Board @RobMeaney
  30. 30. @RobMeaney@RobMeaney@RobMeaney@RobMeaney Quality Indicator Board @RobMeaney
  31. 31. @RobMeaney@RobMeaney@RobMeaney@RobMeaney 4 Simple Questions @RobMeaney
  32. 32. @RobMeaney@RobMeaney@RobMeaney@RobMeaney @RobMeaney Interesting Conversations
  33. 33. @RobMeaney@RobMeaney@RobMeaney@RobMeaney Changing Habits @RobMeaney
  34. 34. @RobMeaney@RobMeaney@RobMeaney@RobMeaney @RobMeaney Gamification
  35. 35. @RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney Baby Steps @RobMeaney
  36. 36. @RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney @RobMeaney Changed Attitudes
  37. 37. @RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney Department SoS Board @RobMeaney
  38. 38. @RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney Customer Impacting Issues @RobMeaney
  39. 39. @RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney Incident Reviews Lane @RobMeaney
  40. 40. @RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney Improvements Visualised @RobMeaney
  41. 41. @ConorFiThe Impact of Kanban @ConorFi
  42. 42. @ConorFiBillboard @ConorFi
  43. 43. @ConorFi @ConorFi Process Improvement Tool @ConorFi
  44. 44. Three Kanban Principles @ConorFi
  45. 45. 1. Visualise the Workflow @ConorFi
  46. 46. 2. Limit Work In Progress (WIP) @ConorFi
  47. 47. 3. Manage Flow @ConorFi
  48. 48. @ConorFi “Trust [the] quality of what you know, not [the] quantity.” Miyagi Kanban - Quality Focus @ConorFi
  49. 49. @ConorFiBreaking things to smaller parts @ConorFi @ConorFi
  50. 50. Switching Sides @ConorFi
  51. 51. Lone Tester & High WIP @ConorFi
  52. 52. Team Based Testing & Reduced WIP @ConorFi
  53. 53. @RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney@RobMeaney 2. Socialise Problems @ConorFi
  54. 54. Our Approach 1. Daily Stand-ups 2. Learning Reviews 3. Department Retros 4. Team Retros @ConorFi
  55. 55. @ConorFi Daily Stand-up
  56. 56. Incident Learning Reviews @RobMeaney
  57. 57. Safe Learning Environment @RobMeaney
  58. 58. Retrospectives @RobMeaney
  59. 59. Engineering Department Retro @RobMeaney
  60. 60. @RobMeaney Discuss In Small Groups @RobMeaney
  61. 61. @RobMeaney@RobMeaney Present, Vote & Act @RobMeaney
  62. 62. @ConorFiRetro - Team Level @ConorFi@ConorFi @ConorFi
  63. 63. @ConorFiBlameless @ConorFi @ConorFi
  64. 64. @ConorFiEveryone has a Voice @ConorFi
  65. 65. @ConorFiChange It Up @ConorFi@ConorFi @ConorFi
  66. 66. @ConorFi @ConorFi@ConorFi@ConorFi 3. Establishing a Quality Culture @ConorFi
  67. 67. Our Approach 1. Exploratory Testing 2. Pairing & Mobbing 3. Bug Bashes 4. Fresh Perspectives 5. Mentoring & Coaching @ConorFi
  68. 68. @ConorFi Exploratory Testing (ET) @ConorFi
  69. 69. @ConorFi Not just Checking, Learning @ConorFi
  70. 70. @ConorFi Not through Teaching @ConorFi
  71. 71. Bring People Together @ConorFi
  72. 72. @ConorFi Pairing and Mobbing @ConorFi @ConorFi
  73. 73. @ConorFi @ConorFi Pairing @ConorFi
  74. 74. @ConorFi @ConorFi Mobbing @ConorFi
  75. 75. @ConorFiBug Bashes @ConorFi
  76. 76. @ConorFiFind Big Problems Sooner @ConorFi @ConorFi
  77. 77. @ConorFiCommunication & Collaboration @ConorFi @ConorFi
  78. 78. @ConorFiDiversity @ConorFi @ConorFi
  79. 79. @ConorFiFun @ConorFi @ConorFi
  80. 80. @RobMeaney@RobMeaney@RobMeaney Fresh Perspectives @RobMeaney
  81. 81. @RobMeaney@RobMeaney@RobMeaney Doug Sillars @RobMeaney
  82. 82. @RobMeaney@RobMeaney@RobMeaney Dan Billing @RobMeaney
  83. 83. @RobMeaney@RobMeaney@RobMeaney Gwen Diagram @RobMeaney
  84. 84. @RobMeaney@RobMeaney@RobMeaney Learning Together @RobMeaney
  85. 85. @RobMeaney@RobMeaney@RobMeaney New Ideas @RobMeaney
  86. 86. @RobMeaneyTeam Coaching @RobMeaney
  87. 87. @RobMeaney@RobMeaney@RobMeaney Workshops On Testability @RobMeaney
  88. 88. @RobMeaney@RobMeaney@RobMeaney Workshops on Risk @RobMeaney
  89. 89. @RobMeaney@RobMeaney@RobMeaney Architectural Reviews @RobMeaney
  90. 90. @RobMeaney@RobMeaney@RobMeaney Sharing & Discussing Models @RobMeaney
  91. 91. @RobMeaney@RobMeaney@RobMeaney Closing Credits @ConorFi
  92. 92. Our Approach 1. Visualise the Work 2. Socialise Problems 3. Establishing a Quality Culture @ConorFi
  93. 93. @RobMeaney@RobMeaney@RobMeaney@RobMeaney What Worked? @ConorFi
  94. 94. At Team Level: ● Quality Indicator Boards ● Kanban Board ● Retros & Learning Reviews ● Pairing/Mobbing ● Bug Bashes ● Revisit Coaching @ConorFi
  95. 95. At Department Level: 1. Be The Example 2. Pain Is A Teacher 3. Make Problems Visible 4. Give People a Voice @RobMeaney
  96. 96. Final Takeaway @RobMeaney
  97. 97. @RobMeaney@RobMeaney Think Big @RobMeaney
  98. 98. @RobMeaney@RobMeaney Start Small @RobMeaney
  99. 99. @RobMeaney@RobMeaney Learn Fast @RobMeaney
  100. 100. Questions? @ConorFi @RobMeaney

Editor's Notes

  • Are you a lone tester?
  • Does every quality discussion feel like a battle
  • Do you feel like the weight of the world is on your shoulders?
  • Are you overwhelmed with the burden of testing?
  • We’ve all worked on teams as the last line of defense, furiously trying to figure out all the weird and wacky ways the software can break.

  • Scrambling to try to get adequate test coverage before the impending release,
    hacking together automation,
    logging bugs,
    retesting bugs
  • Before you the tester alone has to sign off on the release as “Assured the Quality”??
  • I’d seen enough in my career to suggest that there was
    a different way
    a better way.
  • where quality was a whole team responsibility

  • I dared to dream of a new world where testers and developers worked together towards a common goal

  • Having worked together in the past we both Conor & I decided we wanted to break the cycle of the "lone tester"

  • Poppulo focuses on employee engagement.
    Rob and I both joined poppulo 3 years ago, having worked together in the past.
    It felt that Poppulo could provide the platform for us to try to break the cycle of the lone tester and focus on team based testing.
    Fantastic Workplace with highly engaged colleagues, passionate about their work
    Culture of innovation
    Regularly features in the great place to work awards

  • Fast Growing Company using leading tech
  • BUT Most Importantly it felt that everyone
    seemed to be open to new ideas, change and expermation
  • We initially went for a Big Bang approach and our initial plan was to implement a Coaching Model.
    Whereby every tester would become a quality coach.
    We quickly learnt that sometimes Smaller is better
  • Instead we began to try short experiments ..
    So 3 years later and lots of experiments, we want to share our story
    Successes, failures, and if time permits plans for the future.

    We’ve all worked on teams as the last line of defense, furiously trying to figure out all the weird and wacky ways the software can break.
    Scrambling to try to get adequate test coverage before the impending release, hacking together automation, logging bugs, retesting bugs, fixing the automation before signing off on the release because you’ve “Assured the Quality” of the product, you’ve squeezed every last bit of testing juice out of your brain, you’ve done all you can do with the limited time available but by the time i’d joined Poppulo I’d seen enough elsewhere to suggest that there was a different way, a better way. I was lured to Poppulo by the open minded learning culture and opportunity to work with great people. So over the last 3 years I’ve taken the opportunity to explore better ways of testing software that has resulted in dozens of experiments, successes, failures and lots of learning.
  • We faced some Challenges
  • Rob faced challenges at a strategic department level working within the Engineering management team
    How can we change mindsets and influence teams to focus on quality?

    Conor faced challenges working across two delivery teams including one team that were building a entirely new mobile product from scratch in the Cloud!!!
  • Went from 8 to 5 testers
  • Inconsistent process across teams
    Inconsistent tech stack
    Inconsistent testing approach
  • Teams were moving from Monolith to Microservice so there were lots of new challenges & risks - deal with the worst of both worlds
  • Testers spending lots of time writing, maintaining & debugging automation
    Testers only ones that cared about acceptance queue & automation failures

  • So the testers weren’t analysing risk with the team to uncover issues early and mitigate them together as a team
  • Because of the overwhelming workload around test automation and the acceptance queue testers had almost no time to explore the product and some testers had very little exploratory testing experience.
  • Biggest challenge! - Quality Police
    Testers owning the Acceptance Queues
    Signing off on releases
  • Over the last 3 years we’ve seen a pattern emerge that has worked for us
    Often all 3 of these can overlap

  • Make all the work done in the team visible - often testing tasks aren’t tracked - eg. Investigating Automation failures etc.
    Also lets us see bottlenecks and things that impede our work
    this allows the whole team to see what's really going on
  • I’ll run through 3 examples of how we’ve visualised the work
  • I wanted the teams as a whole(not just the testers) to become more conscious of the reliability of their feedback loops
    To understand some of the impediments to moving to continuous delivery
    But I wanted to do it in a lean, low cost way with minimum impact to the teams

  • Every morning wheeled around my board to each team & asked 4 simple questions
    Any alerts in last 24 hours
    Any build deploy issues
    any e2e test failures
    Any rework

  • Huge benefit of not automating it!
    Started to have really interesting conversations and gaining an insight into the common problems that the teams were experiencing as well as problems that were specific to individual teams.

    Allowed me to gain a deeper understanding of each team

  • You can change a culture by changing people habits

    By walking to the teams everyday they developed a habit of checking the indicators every morning before I came around,
    a habit which even became part of stand up for some of the teams

  • After only a couple days teams began to look at other teams quality indicators and compare to theirs, some teams didn’t like what they were seeing

  • So the teams started to make small improvements and changes without any prompting
    Lesson: When you make problems visible people seem to act
    to so long standing problems like, noisey alerts, flaky e-2-e tests were fixed :)

  • All team members started to get more engaged & interested in the feedback loops.
    It all culminated one day during our Team leads metrcis meeting,
    with 15 mins left the team leads actively asked for the quality indicator board unprompted
    so that they could discuss the findings as a team :)

  • So we have a department meeting everyday, all 65 people in the engineering department attend and we discuss our high-level goals together - progress, impediments etc.

  • Changed our SoS board so that all customer impacting issues HAVE to be placed on the SosS board
    First question every morning was if there were any new customer impacting issues & updates on existing ones
    Highlights that customer impacting issues are more important than standard work!!!!
  • We added an incident reviews lane at the very top so that every customer impacting incident needed to have a learning review with associated actions
  • Actions from Reviews were visualised and added to the board so that learning & improvement is part of how we work

    Example security training and monthly security bug bash
  • At a team level we had quality indicator boards and each day gave our updates at the scrum of scrums
    We also use kanban and I believe it complemented the use of quality boards and scrum of scrums

    We started kanban by having a book club, read the book kanban in action and began to use it.
    As we reading a chapter in each week, we were learning from our kanban board.
  • Kan meaning visual and ban board
    Literally sign board /billboard
    Kanban Visualises both the process(the workflow) and the actual work passing through it.
  • Kanban is essentially a Process Improvement Tool
    Which means it can be used to improve scrum

    Can be used to improved your scrum process
  • 1.
    Kanban board
    Radiates information
    Makes hidden work apparent
  • 2.
    Number of items you have going on
    High WIP vs Low WIP
    Value of reduced context switching and decomposing work in to small tasks

  • 3.

    Monitor, measure and report
    All about eliminating bottlenecks
    Highlights the “Testing Bottleneck” (Theory of Constraints)
  • We learnt that when toyota was using kanban
    They focused on quality not quantity
  • Smaller tasks/ Reduced WIP
    Reduced Complexity
    Easier to Test
    Reduced Risk and Analysis
    Easier to Convince other to test :)
    Less intimidating to test
  • In poppulo in the first team i had fallen in to a gatekeeper role
    Kanban helped me move from being a gatekeeper to focsu on more team based testing
    Where we began to colloboate more and more
  • If we look at the way our kanban board was initially.
    The developers focused on developing and reviewing, and i was tested their outputs.
    We had too much work in progress and little was getting done
  • We made to simple changes
    We reduced our work and began to work together to get things done.
    Developers would take items to test from the acceptance queue
    I would review unit tests with developers
    It began to lead the groundwork for team based testing
  • We had 3 approachs now for visualising work and most importantly quality problems.
    We now needed to solve those problems with others
    What we call this socialising the problems, rather than solving the problme alone, solve it with others
    Actions from Reviews were viualised and added to the board so that improvement visualised and part of the way we work

    Example security training and monthly security bug bash
  • The daily stand provides an opportunity to share problems with others
    Highlight problems from the quality board
    Make Changes to the kanban board
    Raise issues at the scrum of scrums

  • Distill as much learning as possible from each customer impacting issue
  • Everyone involved in an incident can safely talk about their experience at each stage of the incident
    Reinforce good behaviours
    Identify areas for improvement
  • Are a regular meeting that allows a Team to reflect and create a plan for improvement
  • Every Quarter have a Department Retro
    opportunity for everyone to bubble up issues
    Facilitated it for 65 people
  • @RobMeaney
    Split in small, mixed groups - EVERYONE Gets a voice

    Identified, discussed & voted for their top Liked/Lacked/Learned
  • Spokesperson for the team presents their top choices
    And Discuss as a department
    Whole department dot vote to identify priority
    Engineering management team focuses on the top liked/lacked/learned
  • Retrospectives provide a great opportunity to learn and grow as a team.
    We have learnt 3 important lessons on running retros
    Fun Retos
    Have a voice
  • Focus on blameless retros both for those in and outside the team
    Have a voice
  • Like rob spoke about earlier, use post-its to ensure every has a change to get their ideas discussed
    Have a voice
  • We use to try different formats for each retro.
    Have a voice
  • Once we have visuaised the problems and socialised them.
    Then it was time to figure out how to establish a culture of quality
  • We wanted to introduce more ET, as rob alluded to earlier ET was not part of the culture
  • We want to take the focus away from just checking and more to exploring, learning about the software.
    Asking open ended questions
  • Training sessions didn’t seem to be the way forward, so we focused more on bring people together
  • But by bringing people together
  • P
  • On of the teams we had issues around the quality of the Apis.
    BY pairing on the testing togther we created quality APIS
    Questions to ask
    .. when Pair testing with Dev

    What problem are you solving
    What is your solution
    What testing have you performed
    Pause . . Rubber Duck Effect :)
    Then Suggest other Tests

  • Mobbing can be Daunting
    As a tester it can be hard to be writing code with really telented developers
    But when it comes to testing, you can see they find it difficult, so there is empathy
    Safe Environment is important
    And it all Ties back to Kanban
    Pairing and Kanban reduce WIP
  • Definition taken from here:

    Putting People at the heart of Quality
    A bug bash is a company or team-wide event held to discover a large number of bugs within a short time frame.
    Prior to Release or Tight Testing Timelines
    Help spread some of the exploratory testing concepts

  • Efficient at big problems sooner
  • Not just bugs but also the feedback around the user experince
    Sales and support begin to spread the word about new features
  • A whole company can provide so much diversity that a lone tester or evena team in both terms of cultural and deiversity of discpline
  • Make them fun, bring food, give a prize for the best bug
  • A key part of changing our culture was introducing fresh ideas & perspectives into the Org
  • As we were at the start of building out our new mobile platform
    Talked about mobile testing and specifically designing and testing for performance
  • In one of our learning reviews we identified that we had a lack of security testing knowledge in house so I set 2 a 2 day hands-on training session with Dan Billing where we had people from across the organisation learn about and do security testing of our App.

    I set up a monthly Security bug bash too.
  • We brought in Gwen to talk about her experiences in Monzo(Engineering Manager) and run an exploratory testing workshop with participants from UI/UX, Dev and Test
  • So we do weekly show and tells and we took this as an opportunity to talk about great testing & quality work
    In this particular picture we have one of our most senior Development Engineers talking through some really great performance testing
  • Learning from community - going to conferences - networking & learning

    Least diverse team ever, but we’ve changed that since :)
  • So I have used coaching when working with teams to influence quality using workshops and modeling, it’s a hugely valuable tool in your toolkit
  • Run a workshop with teams called the “Team Testing Experience” to talk through the 10 factors that influence Testability so that we can reflect & improve continuously
  • I work with teams to Identify various forms of Risk and how best to mitigate them
  • I do workshops focusing on the architectural design where I ask questions about
  • Having conversations with the whole team, share models and discuss appropriate approaches in their context
  • Key things that Conor thinked worked ..
    Visualise the problem
    Kanban Board
    Quality Board
    Creating a quality culture
    Pairing and Mobbing
    Bug Bash

    I would like to Revisit Coaching

  • Rob - Show people problems don't give them solutions
    Get people having good conversations
    Pain is a great teacher, use incidents to drive change
    Be the example rather than tell people
    Give people the opportunity to talk about problems and solve them together
    There's lots of tools & techniques out there but the most important thing is safe, open, honest conversation
    When you make problems visible people seem to act
  • Key things that Conor thinked worked ..
    Visualise the problem
    Kanban Board
    Quality Board
    Creating a quality culture
    Pairing and Mobbing
    Bug Bash

    I would like to Revisit Coaching

  • Think big, start small, learn fast
  • Think about Quality in a whole team holistic sense
  • Start with
    safe to fail
  • Reflect on those small experiments together and iterate on your learnings :)
  • ×