Download to read offline
Uploaded byAsh Winter
NWEWT_Slides_Ash_Winter_04_2016
This document discusses regression testing for a project to stabilize and upgrade the underlying technology of a system while maintaining normal service. It raises questions about how to test that nothing has changed when everything is changing, and whether changes to responsiveness and capacity would still be noticeable to customers. The regression testing involved risk modeling sessions with stakeholders to understand the system, exploring the system to determine what to test, and testing from the user interface down to the unit level. The results showed some issues were found and fixed, while other changes like increased speed and capacity were noticed by customers, raising questions about how to prevent customers noticing any differences after changes. It concludes that taking broad statements literally can be problematic, and that preventing all changes from being noticed may be
NWEWT_Slides_Ash_Winter_04_2016
- 1. Ash Winter Short stuffhere: @northern_tester Long stuff here:
- 2. “Don’t make it anyworse.” A tale of regression testing, misguided expectations and logical fallacies…
- 3. Instability – it’sa thing • Projects to stabilise • Update/upgrade underlying technology • While maintaining “normal” service
- 4. Context… • Wobbly • PCI •POC as a £££ revenue stream • New for some, old for others • New database/ binaries/ hardware.
- 5. Two statements… • “Don’tmake it any worse” • “Customers shouldn’t be able to tell the difference”
- 6. Immediate Questions… • “Todetermine if something is worse, I’ll need to know how bad it is?” • “What happens if its good for some too?” • “For customers to notice, I’ll need to know what they pay attention to…”
- 7. Deeper Questions… • “We’regoing to change everything, how can I test that nothing has changed?” • “If you change the responsiveness and capacity of a system, that’s still noticeable?”
- 8. Orders of Ignorance… •Little codified knowledge. • We wouldn’t know how to know if something had changed…
- 9. #1 Risk Modelling •FIBLOTS again! • 4 people & whiteboard x n • People were the primary oracle • With a bit of existing system…
- 10. #2 Light thepath… • Map the system • Explore the thing • Decide what sheds light
- 11. #3 Pyramid Scheme •FilthyUI Checks •Less grimy API checks
- 12. #4 Check yourself •Filled in the unit level • Big? definitely • Clever? Hmm
- 13. Results – Rumsfeldian Problems “Don’ttouch that, its borked in a known way to us, but unknown to our clients.”
- 14. Results - Everything changesbut you… Client A - “This thingy responds much faster than before…” Client B – “Before we only do X jobs a minute, now its increased to Y jobs”
- 15. Results - Moveany mountain? • When does one stop exposing a previously unseen mountain of knowledge?
- 16. Conclusions? #1 • Shallow statements, taken literal •People fix stuff, sub/un/consc iously
- 17. Conclusions #2 •Deterministic > random… •Software> Hardware… •Mission impossible?
- 18.