Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tranformative Culture - The Shift From QA To Engineering Productivity - SeleniumConf Austin 2017

604 views

Published on

In the presentation, Ashley will review how Blackboard revived a dead automated test suite and turned it into a gold standard, transforming the way the company thinks about software quality along the way. Topics covered will include:

- QA is taboo! (we are now Engineering Productivity – but what does that mean?) – establishing goals as a department goes beyond ensuring quality. How did Blackboard build the teams needed to insure success?
- Documentation is like pulling teeth, but so very critical – how the documentation that was created laid the groundwork to improve. She will show examples that have helped teams get on the right track.
- Getting the right environment and pieces in place – and slashing execution times (going from our own PD Lab to the world of Mesos).
- Project Guardrails. Feature by feature, risk analysis, defining/redistributing/removing tests, and getting developers to trust the tests. Pairing our team with scrum teams fix their tests.
It wasn’t all fun and games – what were some of the hiccups that you can learn from Blackboard?

Ashley Hunsberger – Test Automation Architect, Blackboard

Published in: Software
  • Be the first to comment

Tranformative Culture - The Shift From QA To Engineering Productivity - SeleniumConf Austin 2017

  1. 1. Transforma)ve Culture The shi' from QA to Engineering Produc6vity
  2. 2. @aahunsberger /ashleyhunsberger
  3. 3. About a year ago…
  4. 4. 1 hr. 370 Unreliable FUTURE? 33 It wasn’t all bad…
  5. 5. 1 hr. 370 Unreliable FUTURE? 33 It wasn’t all bad…
  6. 6. So what to do?
  7. 7. “Produc6vity is our job; tes6ng and quality are the job of everyone involved in development. This means that developers own tes6ng and developers own quality. The produc6vity team is responsible for enabling development to nail those two things.” - Patrick Copeland, How Google Tests So9ware
  8. 8. QA is Taboo! The shi' to Engineering Produc6vity begins…
  9. 9. Reduce the 6me from concept to deliverable by providing our product development teams with the tools, prac2ces and support to increase their produc6vity while maintaining high quality standards. Mission
  10. 10. Provide an easily maintainable and extensible framework that enables scrum teams to add and remove tests Goal #1
  11. 11. Enable the automa2c and early detec2on of failures within the so'ware under development Goal #2
  12. 12. Prevent the source of detected failures from moving any further downstream Goal #3
  13. 13. Accommodate all of this without impac2ng the feature engineer’s 6me. Goal #4
  14. 14. The Team
  15. 15. Product Owner The Team
  16. 16. Product Owner Scrum Master The Team
  17. 17. Product Owner Scrum Master SETs The Team
  18. 18. Product Owner Scrum Master SETs Engineer The Team
  19. 19. Product Owner Scrum Master SETs Engineer The Team CI/CD
  20. 20. Documenta)on
  21. 21. All code delivered during the project to provide beZer tooling, prac6ces, and support has been reviewed, tested, documented and released. All documenta6on created to beZer support development teams has been reviewed, tested, communicated and published in a convenient loca6on. Defini)on of Done
  22. 22. Goals Trigger Gate Requirements Test Suite Defini)on
  23. 23. Goals Trigger Gate Requirements Test Suite Defini)on
  24. 24. Goals Trigger Gate Requirements Test Suite Defini)on
  25. 25. Goals Trigger Gate Requirements Test Suite Defini)on
  26. 26. Goals Trigger Gate Requirements Test Suite Defini)on
  27. 27. Environments Insert picture of environment here.
  28. 28. Single Slave (VM)
  29. 29. Jenkins Pipeline (Build Per Branch)
  30. 30. Project Guardrails
  31. 31. Guardrails •  Three phases – (1) Quality review; (2) Clean up exis6ng tests (3) add or remove other tests. •  Defini6on of Done •  Risk Analysis •  Impact •  Likelihood •  Risk priority number •  Sedng a threshold •  At Bb – our ‘general guidance’ is 6. •  hZps://docs.google.com/spreadsheets/d/1j8RV0wJsAgAVZqOZyXfFBugtNEuTzfIVE7bA3s_G5bg/edit?usp=sharing hZps://docs.google.com/spreadsheets/d/1j8RV0wJsAgAVZqOZyXfFBugtNEuTzfIVE7bA3s_G5bg/edit?usp=sharing •  Quaran6ne Strategy •  Point is to further ins6ll confidence in the tests. We want reproducible failures. •  Figuring out where to focus energy… allows us to focus on flaky tests while not impeding dev workflow in the pipeline. •  E2E Retry strategy – reruns test if it fails up to three 6mes; if it passes, we do not fail the build. Test gets inves6gated. Make call to take test out and into quaran6ne. •  We are consultants now! Representa6ve from our team that works with the scrum teams (risk analysis AND code review); •  One of the first 6mes I’ve seen devs and QA really working together on what to test.
  32. 32. Risk Analysis Guidelines Stability Gate Test Suite Defini)on
  33. 33. # Quality Risk Likelihood Impact Risk Priority # Extent of Tes2ng Tracing Instructor can grade a test 2 1 2 0 0 0
  34. 34. Don’t repeat our mistakes…
  35. 35. Lessons Learned •  Have a quaran6ne strategy in place! (keeping things reliable and building trust with developers) •  Checking failures is hard work! No one likes to do it. •  Everyone has a different idea of what’s cri6cal. •  Risk Analysis got the FULL team talking, and became easier to decide as a team as they set their threshold for a feature. •  Not everything can have impact or likelihood of 1 •  Everyone wants to test everything – but you can’t. •  Regardless of threshold, cannot test everything in your list or you will go bankrupt. •  Scalable environment is key to our success •  Dockerize or containerize repeatable processes (like our E2E cache – all the dependent node modules)
  36. 36. 1 hr. 370 Unreliable FUTURE? 33 It wasn’t all bad…
  37. 37. 30 min Quaran6ne More… 165
  38. 38. Thank you!

×