Recommended
More Related Content
What's hot
What's hot (20)
Similar to Is test automation really the solution to your quality problem
Similar to Is test automation really the solution to your quality problem (20)
Recently uploaded
Recently uploaded (20)
Is test automation really the solution to your quality problem
- 1. Reuben Korngold May 2018 Is test automation really the solution to your quality problem?
- 2. Commercial in Confidence 1. Are you automating to save time/money or improve quality? 2. Have you over-promised automation benefits? 3. Are you trying to check or test? 4. Can you replicate a human? Are there humans left to think about quality? 5. With convenience of automation: a) Are your tests telling you the truth? b) Are the same checks passing every time providing confidence of quality? c) How will you address quality fade? Big Test Automation Questions.
- 3. Commercial in Confidence Hey, Got Milk? ???
- 4. Commercial in Confidence Will quality bring all the boys to the yard?
- 5. Commercial in Confidence Standing on the shoulders of our quality veterans W Edwards Deming 1900-1993 “… is accomplished by better management of design, engineering, testing and by improvement of processes”. ”The customer’s definition of quality is the only one that matters” Phil Crosby 1926-2001 “Has to be the fabric of the organisation, not part of the fabric”. “Has to be caused, not controlled” Peter Drucker 1909-2005 “… is not what the supplier puts in. It is what the customer gets out and is willing to pay for. Customers pay only for what is of use to them and gives them value. Nothing else constitutes quality” Joseph Duran 1904-2008 “Fitness for use. Fitness is defined by the customer” Robert Pirsig 1928-2017 “… is like modern art. We may not be able to define great modern art; but we frequently recognise it when we see it.”
- 6. Commercial in Confidence How ISO and PMBOK define quality. “The degree to which a set of inherent characteristics fulfills requirements.”
- 7. Commercial in Confidence PMBOK, 5th Edition PMBOK: Project Quality Management Quality management has 2 goals: 1. Ensuring a quality end-product. 2. Ensuring that all of the processes involved during the project lifecycle are carried out efficiently. By managing quality, project managers can ensure a successful project and increased customer satisfaction.
- 8. Commercial in Confidence Dr Martin Barnes’ Iron Triangle
- 9. Commercial in Confidence Are you working in QA? • Quality Assurance is the process for managing quality. • Processes are put in place to ensure the correct steps are done in the correct way. • If the correct processes are in place there is greater confidence that the actual results will turn out as expected • Quality Control is checking or detecting the output for quality, also known as “testing”
- 10. Commercial in Confidence How does IT fare… overshooting the runway
- 11. Commercial in Confidence Testing the quality in. Let us un-curdle the milk!
- 12. Commercial in Confidence Quality: What happens if you do, or don’t, do it
- 13. Commercial in Confidence Quality conformance: The Customer I want a car that: 1. Has 4 wheels 2. Has working brake lights 3. Has air conditioning Also, I do not want it to rob the engine of power
- 15. Commercial in Confidence Delivering to specification = quality? 4 Wheels Air Conditioning Working brake lights Air Con Power Source Doesn’t rob engine of power! You didn’t specifically tell me “no dents”
- 16. Commercial in Confidence Business results delivered from the technology vortex
- 17. Commercial in Confidence Performance (Does it meet my core need) Features (Anything additionally appealing) Reliability (Will not fail over a time period) Conformance (Precision to meet standards) Durability (Lifespan) Serviceability (Speed to repair) Aesthetics (Personal preferences) Perceived Quality (Indirect measures) Dimensions of quality. Garvin, Harvard Business School
- 18. Commercial in Confidence Induction: Absence of evidence vs evidence of absence 1. How do you know that you covered what you needed to? 2. Zero defects detected = bug free? 3. Do you think you can now predict quality issues? Getting philosophical about quality SEXTUS EMPERICUS, c. 200AD: A universal rule could not be established from an incomplete set of particular instances DAVID HUME, c. 1700’s 1. You cannot generalise based on limited observations 2. Sequences of events from the past cannot predict the future
- 19. Commercial in Confidence Checking or testing? Checking • Unambiguous binary result against specification • Compares an output to an expected output • Can be performed by machines as the result is self-decidable • Finds presence of defects Testing • Gain an understanding about the fitness for purpose of the product • Combines checking with an observation to either an implicit or explicit expected result • Performed through exploration, discovery, and learning • Humans evaluate results • Raises questions of use cases, usability and richer defects not mechanically detected
- 20. Commercial in Confidence Why do you test? Why do you want to automate?
- 21. Commercial in Confidence How to improve quality.
- 22. Commercial in Confidence We can no longer only optimise for efficiency in the 21st century http://www.afr.com/leadership/atlassian-cofounder-scott-farquhar-challenges-business-focus-on-efficiency-20180409-h0yju7 “…eliminating variables and optimising output rates worked for factory workers in the 19th century but made little sense now.” We must adapt to, and plan for change.
- 23. Test Automation
- 24. Commercial in Confidence Get local contractors to manually test Do the same thing cheaper offshore using labour arbitrage and tick-boxes If we can describe what to do in enough detail that someone halfway around the world can do it, why not just automate it? WHY doesn't testing fix quality? Evolution of the testing industry
- 25. Commercial in Confidence So why automate? • Because technology is fun and we love tinkering with the latest tools • Increases motivation and optimism • Operational efficiency for checking • Consistency • Stable metrics (e.g. tests per hour) • Auditability • Speed to deliver • Cost • Liberate test analysts to be analysts not “checkers” or get rid of testers altogether • Because we believe that consistent and robotic checking is the solution to quality
- 26. Commercial in Confidence The technology vortex
- 28. Commercial in Confidence • A test suite using record and playback broke the next release (recommended approach only for short term contractors) • Limited validations meant test automation suite completed • The Test Automator went on holidays and nobody else could use it • Automated tests were later found to exercise only 5 of 200 web services, team believed coverage was high • Maintenance of a test suite exceeded the manual execution time each release • Lots of defects found but it was the script that was faulty not the software Horror stories.
- 29. Commercial in Confidence • Able to complete a 15 person days testing overnight, freeing up testing team to be analysts, not executors • Expanding test coverage beyond what humans could possibly achieve • Repetitious data creation • Identification of variations using screen diffs that human testers had not detected • API level tests quickly implemented that had high ROI being fast, efficient, and stable • Automation implemented in build process, fast feedback identified variances soon after check-in Development is additive. Automation can be a labour saviour.
- 30. Commercial in Confidence • Start with the idea, end with the customer • Employ quality methods and measures across the lifecycle • Plan for quality, and follow through with the plan • Use Humans who understand the context • Always involve and listen to your customers, as they are the ultimate measure of quality • Do perform Quality Control (checking) to confirm delivery • As close to source of creation as possible • Automate repetition & breadth • Align verification to business goals and risks • Remember that checking (alone) is not quality - really test: explore, discover, and learn How to really assure quality.
- 31. Is test automation really the solution to your quality problem? Yes No Maybe Dude, are you serious?
- 32. Commercial in Confidence Lets ask our some of our friends…. RANDOM TECH GUY: It can do more consistent and thorough checking, early and regularly. CROSBY: If this creates a culture of quality in the fabric of the organization then it can play a part. ADVOCATE: Employ checks early. Don’t spoil the product’s potential by waiting until its packaged before thinking about quality. CUSTOMER: Doing your same checks only faster won’t make me happy. Maybe you could get me involved so you deliver what I wanted? JURAN: Dude, fitness is measured by the customer. Involve the customer, already! DRUCKER: What’s the customer going to get out of it? DEMING: Sounds like better management of design, engineering and testing. Do it. But don’t forget the customer. PIRSIG: Quality is like modern art, you know it when you see it. Test automation is a cool modern art but only one appreciated by techno boffins not customers.
- 33. YOU DECIDE
Editor's Notes
- Hands up who has used test automation before? Can anyone offer a definition of quality?
- Phil Crosby – businessman and author of QM practices. Involved in Zero Defects program Peter Drucker – management consultant and author and business guru Joseph Duran – engineer and management consultant and author. Evangelist for quality management Robert Pirsig – philosopher and author – Zen and the art of motorcycle maintenance: an inquiry into values W Edwards Deming – American engineer, author, management consultant. Brought quality to Japan under behest of General MacArthur and then brought it back to USA to become more competitive starting with Ford in 1981
- $2.8 billion is similar to the GDP of East Timor (World Bank 2017) $1.8 billion is the cost of the new Sunshine Coast University Hospital, which the IT overspend would have paid for a good portion of it
- Hands up how many people have been involved in the reactive process is the last 2 years?