Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Sustainable Automation Frameworks by Kelsey Shannahan
Next
Download to read offline and view in fullscreen.

0

Share

Download to read offline

Introduction to test for non testers

Download to read offline

A brief introduction to test for the non-tester. Can be used for both business and development, although it is primarily focused on developers and persons interested in becoming testers.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Introduction to test for non testers

  1. 1. Introduction to test: What is test, and who is the tester? •Mattias Lönnqvist •Kuoni Nordic IT •STOCKHOLM, 2013-11-27
  2. 2. Agenda • Presentation • Introduction to testing • • • • • What is testing? Why do we test? Test Phases Requirements When do we stop? • Who is the tester? • • • • • Beizer Cognitive psychology Biological differences Assumptions Cognitive dissonance • • • • • Is all lost? • <- Briefly! Why does it matter? Test tools Summary Q&A | 1
  3. 3. Presentation • Mattias Lönnqvist • QA Test Manager at Kuoni Nordic IT (Previously: OE IT Online) • Been at Kuoni since June 1st 2012 • Been working in IT since 1997 and with test since 2004. • ISTQB certified at foundation level and advanced level (test manager) • Worked as a consultant in test for 5 years before going to Kuoni • Will give you a quick background on testing • Please do interrupt if you have questions • A nice guy  | 2
  4. 4. What is testing? Introduction • What testing is NOT • What testing IS • It’s not the quality process • ”Testing is measuring quality” • Only quality measurement, (Ariane 5 Flight 501) Hetzel 1988 • William Ariane 5's first test flight i.e. not theJune 1996 failed, with the rocket selfon 4 Quality process destructing 37 seconds after launch because of a malfunction in the control software. • It’s not a guarantee • ”Testing is the execution of • Products will never be software in order to show that it bugfre A data conversion from 64-bit floating doesn’t work” point value to 16-bit signed integer value to • Glenford Myers 1979 be stored in a variable representing horizontal bias caused a processor trap (operand error) • It’s not a cost because the floating point value • Example: The Ariane 5 rocket was too large • We save more on testing a 16-bit signed •integer. $800 million Cost: to be represented by than the cost • It’s not a waste of time • It helps improving quality | 3
  5. 5. What is testing? From a business perspective Cos t Reduced Cost = reduced Quality Reduced Time = reduced Busines Quality (or increased cost) s Increased Quality = increased Oriente cost/time d Quality Tim e Testing | 4
  6. 6. What is testing: The V-model / the W-model | 5
  7. 7. Why do we test? Introduction • We are measuring quality • Testing is gaining knowledge • To keep our deadlines • We cant keep our deadline if we dont know if it works • Not loosing money • Ariane 5 rocket: $ 800.000.000 • Heathrow terminal 5: £25.000.000 (failed luggage handling) • Forsmark 2006: 2 (of 4) backup generators didn’t start on power shortage. Cost unknown. • etc • To make our customers happy • We need them to have confidence in us • Errare humanum est • Code is written by humans, and humans make mistakes • Deadlines causes stress and stress induces errors • Changing development and maintenance environments • Lacking or incorrect requirements | 6
  8. 8. Why do we test? The numbers • Standish group 1995 – The Chaos Report • 31,1% of all (software) projects are never completed • 52,7% of all projects cost 189% of estimated cost • In 1995, in the US, 81 billions were spent on projects that were cancelled or never finished • 59 billions were spent on projects that were finished that year • Completed projects had in average 42% of initial requirements met • 9% of all projects in larger companies were considered succesfull • The biggest reason for this was incomplete requirements • The numbers are better now, but not good • Standish report costs money, which is why I use the 95 report • Info-tech research group 2006 • 50% of additional work in software projects is caused by problems with requirements • 70% of failed projects fails because of problems with requirements • Testing is connected to requirements! | 7
  9. 9. Why do we test – Requirements • • • • • • • Verification of requirements Business specifies requirements Development transforms requirement into code Testing verifies code against requirements Requirements can never cover 100% Requirements can never be unambiguous Requirements can be misunderstood • • • • • • • • • What is this? [Example on requirements] 1. Carries objects or people 2. Low maintenance 3. Low cost 4. Versatile The developers interpret as: System testers interpret as: Acceptance testers interpret as: No! This is what business meant: Myth: I can’t determine if the code is correct Truth: Yes, you can! | 8
  10. 10. Why do we test: the costs • What are the costs? • Test environment • Test resources • Time spent on testing • Etc • What are the benefits? • Compare to cost of cancelling a release • Compare to cost of fixing a bug in live environment • Compare to cost of decreased sales • Compare to cost of reduction in Brand value Source: Barry Boehm 2007 | 9
  11. 11. Why do we test: The importance • How bad is a bug? • Disaster? • Titanic, Ariane, Patriot Missile, Landing system, etc • Scale of disaster is proportional to the belief it cannot fail • Sometimes called ”Famous last words” • Bad? • Word 97 crashed all the time in first release • Software wipes database • Trivial? • Something is ugly, spelling error, etc. • Catch 22 • If you look for defects and find none you are pleased • Finding errors destroys confidence • Testing is looking for errors you don’t wish to find… | 10
  12. 12. Why do we test – Summary Statement • • • • • • • • • • • True or False? Because Software contains bugs/errors? To estimate / measure quality in Software? To spend time between code complete and go live? To prove that there are no bugs? Because testing is a part of software development? Because Errors can cost a lot? To reduce risk of upset customers? To reduce risk of loosing business? To stay the best in the travel business? Because we like to point out others mistakes? To be able to determine if the software can go live? | 11
  13. 13. The test phases – simplified description • Developers: • Develop code • Compile • Check in • Unit test • (sometimes) integration test • Own testers • Smoke test • System test • Regression test • Re-test • Customer • Acceptence test • This is the customer sign-off, stating if they are satisfied • No acceptance test = we accept the quality, whatever it is… | 12
  14. 14. The test phases – One (of many) way to split up the work • Unit tests • Developers test their new code • Does it compile? Does it run? • System test • Developers make sure system works after new code is added • Smoke test • Developers make sure that testers can do their work before handing over • System Integration Test (SIT) • Testers make sure the system is working • Testers make sure the system is working with all other systems • User Acceptance Test (UAT) • Customer (might be internal) makes sure the delivered software matches their requirements • This phase should only find bugs relating to requirements (mismatch) • All other bugs should be found in the previous phases! | 13
  15. 15. Test-phases in a waterfall world (but can be described Agile) Start We have a project that needs testing Requirements Requirements Insufficient resources, scope, etc Plannin g Create test plan, allocate resources Requirements Execute test cases Specificati Requirements Non-functional requirements Create test cases from requirements Unclear requirement Retest bugs Unclear or missing requirements on Functional requirements A good test case is: Execute test cases Testin 1. Effective (finds bugs) 2. g Examplatory (covers several areas) 3. Maintainable (easy to change / update / keep current) 4. Low cost (easy to use, doesnt take a lot of time) Requirements Report bugs Result 5. Possible to automate (sometimes) s Repor ts End Create test report Requirements Requirements Project delivered | 14
  16. 16. Requirements – why are they so hard? • We are dealing with humans • Humans interpret • Humans are different • We have different • Roles • Interpretations | 15
  17. 17. When do we stop testing? Statement about end of testing True or False? • We can never test enough • We stop testing when the customer is satisfied • We stop testing when we have proved that the system has no flaws / errors / bugs • We stop testing when we ”feel” quality is good enough • Whether we are done depends on … • • • • Who decides if exit-criteria is fullfilled What are our risks What is our test case coverage What is our S-curve | 16
  18. 18. When do we stop: The S-curve This example is for a typical project Applies to all projects Applies to all models Answers: • Can we release now? • How close to ”ready for delivery” are we? • How are we progressing? Risk Decision Point deploy now? Law of Diminshing Returns Low Detection Rate Cumulative Defects • • • • Test on or implement now Undetermined time saving High Detection Rate Time Requirements Analysis Design Code Test Implementation date | 17
  19. 19. Who is the tester? • We are dealing with humans • We are controlled by our brains • Cognitive psychology • How / what we percieve • How / what we see • How / what we hear • Test Psychology • Understanding how we, as humans affect testing • We are not machines | 18
  20. 20. What is a tester? • Beizers phases of testing (Boris Beizer) • Phase 0 : No difference between testing and debugging • i.e. Testing is something done by developers only • Phase 1: The purpose of testing is to show that the Software works • i.e. We can prove that the software is flawless • Phase 2: The purpose of testing is to show that the Software doesn’t work • i.e. Our purpose is to find mistakes • Phase 3: Purpose of testing is to reduce risks in software • i.e. We reduce risk of errors in delivered software • Phase 4: Testing is not an act, it is a mental discipline that results in lowrisk software without much testing effort • i.e. testing is a part of everything • This is what exploratory testing focuses on | 19
  21. 21. Cognitive Psychology • My focus is on human memory functions • How we learn • How we remember • How we recall information • Human memory consists of: • Short term memory – here and now • That is why you should take notes when testing • Or you will forget what the bug was • (But you may still remember that you found a bug) • Medium term memory – last month • Repeating copies from short term memory • Long term memory – childhood (etc) | 20
  22. 22. Human memory – Short term memory • • • • You can only remember 7 +/- 2 items (5 to 9) Needs to be actively copied to long term memory Trivial errors often caused by short term memory overload Major problem with short term memory • I forgot what the problem is •  • If your brilliant idea isnt copied to long term memory you loose it (but may come up with it again) • That is why you should take notes when testing • Or you will forget what the bug was • (But you may still remember that you found a bug) | 21
  23. 23. Human memory – Medium term memory (working memory) • Working memory • is more a process than a part of the brain • Analysis causes tension • The mind must solve the problem to relieve the tension • Result is ”feel good factor” • Major problem with working memory • ”Feel good factor” undermines correct understanding • Desire to solve leads humans to jump to conclusions • Solution may be wrong • E.g. When finding a work-around to do something that isnt working the way it should, we may be too pleased with our solution to remember the bug | 22
  24. 24. Human memory – Long term memory • Affects Your world • The way you view things Your childhood • Your personality Right and Wrong • Your temperament Your parents and family • The way the ”penny life drops” • Education • Familiar concepts • Formative years • Easily accepted • You never forget anything in the long term memory • New ideas & changes • Resisted • But you might not be able to recall it • • • • | 23
  25. 25. Biological differences • Biological differences in GENERAL • Not all men, and not all women • Women are better at multitasking • Women have a smaller brain, but a more developed brain stem (the brains information highway) • Less women has color blindness • Important if you use color to carry information • Men are better AND worse at math • Women more in the middle on Bell curve • Some biological differences enhanced by society • E.g. high pitched voices in Japanese women | 24
  26. 26. Assumptions • The brain makes assumptions from stored knowledge • The brain will find a solution • But it doesnt have to be correct • The penny drops … • Do you know what you see? • Or do you see what you know? | 25
  27. 27. Assumptions – an example • Read the following colors and say them loud • Dont think; time is money • We have a deadline • Ready? | 26
  28. 28. | 27
  29. 29. Assumptions – Contradicting associations • Color and word doesn’t match • Asking your brain to associate the wrong color with the name of the color causes a conflict • The conflict creates overload that increases due to quick slide change (0,5 seconds) • Cognitive dissonance (more later) • Now take a look at the next slide… | 28
  30. 30. Raise your hand if you spotted the error… | 29
  31. 31. • Let us look again … • See it? | 30
  32. 32. Assumptions - Blinded by the grammar rules • English grammar is a set of rules • You know the rules and apply them by ”motor skill” • Your brain is conditioned to apply the rules, even if what is present breaks the rules • As a result: • You may not ”see” what is actually present | 31
  33. 33. Cognitive dissonance • Do you believe what you see? • Or • Do you see what you believe? | 32
  34. 34. Cognitive dissonance - example 1(color blindness) | 33
  35. 35. Cognitive dissonance - example 2 (it shouldn’t move) | 34
  36. 36. Cognitive dissonance - example 3 (might not work on projector) | 35
  37. 37. Cognitive dissonance - summary • About 15% of all males are color blind • Around 1% of all women • Females can interpret patterns more consistently then men • If the information is in a color or if it can be interpreted in more than one way … • How do you know that you will interpret it correctly | 36
  38. 38. Interpretations • I am a tester • In what field? • What do you do? • Functional or nonfunctional? • Manual or automated? • Etc • What is obvious to you might not be obvious to other • The better you know an area, the more informed questions can you ask | 37
  39. 39. Who is the tester? Summary • To understand the tester we need to understand • • • • • Cognitive Psychology (memory) Biological differences Assumptions Cognitive dissonance Interpretations • Why does it all matter? | 38
  40. 40. Why does it all matter - Overload • Maximum efficiency at 7 +/- 2 tasks • Overload affects recall (memory) • You have not only forgotten • But you cant remember that you forgot • Test items are overlooked • Risk is introduced • Impairs ability to interpret (correctly) • You may not be self aware of overload | 39
  41. 41. Why does it all matter – Cognitive dissonance • You’ll remove 30-40% of your own errors • Which is why coders shouldn’t test their own code • Don’t proof read your own work • Dont design tests for software you have written yourself • Let others challenge your mind set, your work and your documents • Welcome this! | 40
  42. 42. Is all lost ? • What can I do about this? • Be aware of your limitations • Be aware of overload • Create your own fortress of solitude: • Set the alarm for 60 minutes and: • Turn off the phone • Put on head phones • Put up a ”Do not disturb” sign • Dont run the same test twice • Switch with someone else | 41
  43. 43. Test tools (in brief) • Screen dumps • An image says more than 1000 words • Test case tools • To store test cases for easy access • TFS / Spira / Excel / etc • With or without Gherkin / Specflow • Bug reporting • TFS / Spira / Google docs / etc • Methods • Exploratory • Sessions | 42
  44. 44. Summary • Remember that testing is PART of the quality process • Be aware of your limitations • Both your own and those of others • Dont expect testers to find bugs if they cant focus • Do you know what you see or do you see what you know? • Utilize domain knowledge (business, end users, etc) • No one knows better than them what the software is supposed to do! • Not me, not the project manager, not the developers • This is the strength and purpose of UAT! • Avoid overload! • Find a way to create undisturbed working space • Humans interpret • If you are uncertain about a requirement, ask … | 43
  45. 45. Q&A • Questions? • Please do ask… • There are no stupid questions • Only stupid lecturers …  | 44

A brief introduction to test for the non-tester. Can be used for both business and development, although it is primarily focused on developers and persons interested in becoming testers.

Views

Total views

774

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

9

Shares

0

Comments

0

Likes

0

×