Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Digital QA Summit - Karen N. Johnson - Testing Explained

2,610 views

Published on

opening keynote at the Digital QA Summit in Chicago October 2014

Published in: Technology
  • Be the first to comment

Digital QA Summit - Karen N. Johnson - Testing Explained

  1. 1. TESTING EXPLAINED AN OVERVIEW OF SOFTWARE TESTING Karen N. JohnsonDigital QA Summit
  2. 2. Do you find yourself needing to “check” something in production immediately after go live? Do you need to investigate how a page or a campaign looks in multiple browsers or on a mobile device? Without realizing it, you may be performing the role of a software tester in addition to your primary role. What is a software tester?  In this session Karen offers 1) a brief overview of the software testing profession, 2) a summary of different types of testing, 3) a list of resources to learning more about testing. If testing is part of what you or your team need to juggle to “make it happen” come attend this session.   About this talk … Testing Explained © Karen N. Johnson 2014
  3. 3. •  Software Test Consultant •  Published Author (Beautiful Testing) •  Co-founder of WREST: Workshop on Regulated Software Testing •  Website: www.karennicolejohnson.com or www.karennjohnson.com •  Twitter: @karennjohnson Karen N. Johnson Testing Explained © Karen N. Johnson 2014
  4. 4. Do you get asked…
  5. 5. Hey, can you “just” check this out? Testing Explained © Karen N. Johnson 2014
  6. 6. Can you make sure all the parts to the campaign “look good?” Testing Explained © Karen N. Johnson 2014
  7. 7. But do you feel unsure how to test? Testing Explained © Karen N. Johnson 2014
  8. 8. to software testing Testing Explained © Karen N. Johnson 2014
  9. 9. Testing is more than checking things out. Testing Explained © Karen N. Johnson 2014
  10. 10. Software testing is a profession with it’s own “world.” Testing Explained © Karen N. Johnson 2014
  11. 11. When you already have a fulltime job and a profession? But how do you immerse yourself in another world? Testing Explained © Karen N. Johnson 2014
  12. 12. Here’s look at some of the essential topics in the software testing community.
  13. 13. Waterfall & Agile Development Testing Explained © Karen N. Johnson 2014 ¨  There are many different ways to develop and test software. ¨  In past years, the Waterfall method was the most common software development methodology. ¨  In current times, an Agile development process is now more common.
  14. 14. Waterfall Development Testing Explained © Karen N. Johnson 2014
  15. 15. Agile Development Testing Explained © Karen N. Johnson 2014 The Agile Manifesto ¨  We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: ¤  Individuals and interactions over Processes and tools ¤  Working software over Comprehensive documentation ¤  Customer collaboration over Contract negotiation ¤  Responding to change over Following a plan
  16. 16. Acronyms: we love them! ¨  ATDD: Acceptance Test-Driven Development ¨  BDD: Behavior-driven development ¨  TDD: Test-driven development Testing Explained © Karen N. Johnson 2014
  17. 17. Checking vs. Testing Testing Explained © Karen N. Johnson 2014 ¨  Within the profession of testing, there is a long-time discussion known as “checking vs. testing.” ¨  Often less experienced people in testing and/or people who are not professional testers “test” in such a manner that they are “checking” software works. ¨  In these situations, people are not exploring, they are not looking for issues, they are checking to confirm software works versus looking to expose or discover issues.
  18. 18. Checking vs. Testing Testing Explained © Karen N. Johnson 2014 ¨  If you never read an article on testing, if you never read a blog post on checking vs. testing, learn this one concept: Know when you are checking to confirm software is working vs. testing with the intention of discovering possible issues. ¨  Do not confuse the two approaches (checking vs. testing) as meaning the same thing. ¨  There is nothing wrong with “checking” software or “confirming” that software works but this is not the same as intentionally looking to discover defects.
  19. 19. Tested = Checked + Explored Testing Explained © Karen N. Johnson 2014 In Elisabeth Hendrickson’s book, Explore It!, she states that neither checking nor exploring is enough. “You’re not done testing until you’ve checked that the software meets expectations and you’ve explored whether there are additional risks. A comprehensive test strategy incorporates both approaches.”
  20. 20. Exploratory vs. Scripted Testing Testing Explained © Karen N. Johnson 2014 ¨  Within the profession of testing, there is a long-time discussion on the use of scripted vs. exploratory testing. ¨  Let’s look at each format.
  21. 21. Scripted testing: a definition Testing Explained © Karen N. Johnson 2014 Scripted testing follows a path that is written by the tester themselves or someone else. The script includes test steps that are documented. It is expected that the tester will follow the steps as detailed. The tester’s job in a scripted environment is to follow each step and report findings (if any).
  22. 22. Exploratory testing: a definition Testing Explained © Karen N. Johnson 2014 James Bach’s definition: “Exploratory testing is simultaneous learning, test design, and test execution.” This style of testing requires that your brain be fully engaged at all times. Cem Kaner’s definition: “Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”
  23. 23. Exploratory & Scripted Testing Testing Explained © Karen N. Johnson 2014 ¨  As it turns out, you don’t have to choose one form over another. You could use a blended approach.
  24. 24. Schools of Software Testing Testing Explained © Karen N. Johnson 2014 ¨  Around 2003, several people in the testing field constructed a way to explain different core approaches to software testing. The construct was referred to as the schools of testing. ¨  The schools of testing have been greatly debated over the years. Some people have been so opposed to the construct of the schools concept, they have refused to even discuss the construct or acknowledge the schools as a way of discussing differing views.
  25. 25. Schools of Software Testing Testing Explained © Karen N. Johnson 2014 ¨  In 2003, the schools were defined by Bret Pettichord as: ¤  Analytical School ¤  Factory School ¤  Quality Assurance ¤  Context-Driven School
  26. 26. Schools of Software Testing Testing Explained © Karen N. Johnson 2014 ¨  You don’t have to be involved in the debate. ¨  You don’t have to choose a school. ¨  But it can be helpful to know this “argument” exists in the testing profession. ¨  It can be helpful to know what school a tester identifies with.
  27. 27. The Seven Basic Principles of the Context-Driven School 1.  The value of any practice depends on its context. 2.  There are good practices in context, but there are no best practices. 3.  People, working together, are the most important part of any project’s context. 4.  Projects unfold over time in ways that are often not predictable. 5.  The product is a solution. If the problem isn’t solved, the product doesn’t work. 6.  Good software testing is a challenging intellectual process. 7.  Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
  28. 28. The Original Quadrants of Testing Testing Explained © Karen N. Johnson 2014 From Brian Marick’s blog
  29. 29. The Original Quadrants of Testing Testing Explained © Karen N. Johnson 2014 Support programming vs. critique product ¤  “When it comes to supporting programming, tests are mainly about preparing and reassuring.” ¤  “On the other side of the division, tests are about uncovering prior mistakes and omissions.”
  30. 30. Agile Testing Quadrants Testing Explained © Karen N. Johnson 2014 From the book, Agile Testing by Lisa Crispin and Janet Gregory
  31. 31. The Quadrants are being reconsidered Testing Explained © Karen N. Johnson 2014 ¨  Gojko Adzic http://gojko.net/2013/10/21/lets-break-the-agile-testing-quadrants/ ¨  Markus Gärtner http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/ ¨  Elisabeth Hendrickson http://www.slideshare.net/ehendrickson/the-thinking-tester-evolved?ref=http:// www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/
  32. 32. Certification & Standards Testing Explained © Karen N. Johnson 2014 ¨  There are several certification programs available. ¨  There are several boards such as IEEE and ISO, that devise standards around testing. ¨  And there are testers who are in agreement with certifications and/or standards. ¨  And there are testers who are strongly opposed.
  33. 33. Types of Testing Testing Explained © Karen N. Johnson 2014 ¨  There are numerous types of testing. Each type has a focal point and serves a different purpose. For example, user acceptance testing vs. performance testing. ¨  In some cases, a type of testing will be more likely to occur at different times during the development process. For example, unit testing vs. system testing.
  34. 34. Black Box Testing Testing Explained © Karen N. Johnson 2014 ¨  Black box testing is one of the most common terms used in testing. ¨  The basic premise is the person testing does not know how the code was written and does not have insight inside the “box” or the system. ¨  Black box testing is testing without a technical understanding of how a program was written. This is why the “box” (or the item being tested) is “black” (because we don’t know what is inside.)
  35. 35. White Box or Glass Box Testing Testing Explained © Karen N. Johnson 2014 ¨  The opposite of black box testing. White box testing means the tester has an inner understanding of how the program was written, this form of testing allows for more targeted technical testing. ¨  Much of the time, black box testing is the type of testing being used.
  36. 36. Testing Explained © Karen N. Johnson 2014 An example … Suppose I am testing the credit card entry part of an ecommerce website. Black box testing: •  Try using an invalid credit card White box testing: •  Review the code •  Learn after three invalid credit card entries, the account is locked •  Enter three invalid attempts check error handling and messaging With black box testing, I might try an invalid credit card once, see an error message and not continue to test because I don’t know about the three attempt condition.
  37. 37. Unit Testing Testing Explained © Karen N. Johnson 2014 Martin Fowler’s description: “The common properties of unit tests — small scope, done by the programmer herself, and fast — mean that they can be run very frequently when programming.”
  38. 38. Integration Testing Testing Explained © Karen N. Johnson 2014 Paul Jorgensen’s definition: Integration is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing. Lee Copeland’s definition: In integration we assemble units together into subsystems and finally into systems. It is possible for units to function perfectly in isolation but to fail when integrated.
  39. 39. System Testing Testing Explained © Karen N. Johnson 2014 Lee Copeland’s definition: A system consists of all of the software (and possibly hardware, user manuals, training materials, etc.) that make up the product delivered to the customer. System testing focuses on defects that arise at this highest level of integration. Typically system testing includes many types of testing: functionality, usability, security, internationalization and localization, reliability and availability, capacity, performance, backup and recovery, portability, and many more.
  40. 40. UAT or User Acceptance Testing Testing Explained © Karen N. Johnson 2014 Lee Copeland’s definition: Acceptance testing is defined as that testing, which when completed successfully, will result in the customer accepting the software and giving us their money.
  41. 41. Testing Explained © Karen N. Johnson 2014 An example … Suppose I am testing the purchasing process of an ecommerce website. Unit testing: Build a unit test for credit card expiration date. Integration testing: When the code for credit card processing is integrated with the code that handlings shipping information, I can build an integration test. At the system level, I might build testing from the start of product selection through the entire place an order, credit card and shipping process; a test that brings all the pieces together.
  42. 42. Regression Testing Testing Explained © Karen N. Johnson 2014 ¨  A common form of testing; used to retest to ensure changes in one part of a program have not introduced issues in another part of a program.
  43. 43. The “ilities” of testing ¨  Software testers commonly refer to a series of terms as the “ilities” of testing. ¨  The “ilities” are often a challenge to test but are often the same items that customers are often most concerned about. Testing Explained © Karen N. Johnson 2014
  44. 44. The “ilities” of testing ¨  Capability ¨  Reliability ¨  Usability ¨  Security ¨  Scalability ¨  Performance ¨  Installability ¨  Compatibility ¨  Supportability ¨  Testability ¨  Maintainability ¨  Portability ¨  Localizability Testing Explained © Karen N. Johnson 2014 Source: James Bach, Rapid Software Testing
  45. 45. More types of testing ¨  Stress ¨  Multi-user ¨  Cookies ¨  Compatibility ¨  Developer ¨  Unit ¨  Integration ¨  System ¨  User Acceptance ¨  Installation ¨  API/web services ¨  Black box ¨  White or glass box ¨  Gray box ¨  Functional ¨  Automated ¨  Regression ¨  Security ¨  Data ¨  Exploratory ¨  Scripted ¨  Performance Testing Explained © Karen N. Johnson 2014
  46. 46. Some “words” around testing Testing Explained © Karen N. Johnson 2014 ¨  Risk ¨  Heuristics ¨  Oracles
  47. 47. But if you only learn one question to ask, let the question be … What if? Testing Explained © Karen N. Johnson 2014
  48. 48. •  What if? Is a challenge question. •  What if? Is the type of question that begs for investigation and discovery. •  What if? Does not require technical training or knowledge. Testing Explained © Karen N. Johnson 2014
  49. 49. During storyboarding and design is a great time to use: What if? Testing Explained © Karen N. Johnson 2014
  50. 50. Testing Explained © Karen N. Johnson 2014
  51. 51. Testers focus on where and how a story can breakdown. Testing Explained © Karen N. Johnson 2014
  52. 52. Testing Explained © Karen N. Johnson 2014
  53. 53. “State transitions provide a Bermuda-Triangle-like moment in time where “things” can go wrong for a user or system. And once a user or system is in a bad state, they can be awkwardly trapped or lost.” Testing Explained © Karen N. Johnson 2014
  54. 54. State transition “When a system must remember something about what has happened before or when valid and invalid orders of operations exist, state- transition diagrams are excellent tools to record this information.” Source: Lee Copeland. A Practitioner's Guide to Software Test Design Testing Explained © Karen N. Johnson 2014
  55. 55. Testing Explained © Karen N. Johnson 2014
  56. 56. Some final advice …
  57. 57. 1. You will never find everything. Testing Explained © Karen N. Johnson 2014 The number of combinations and possibilities is greater than what you can test.
  58. 58. There are more ways to test than there are hours in the day. 2. You will never have enough time. Testing Explained © Karen N. Johnson 2014
  59. 59. If you have more ideas for testing than time, choose what you will test wisely and get help from your team. 3. You cannot do it all alone. Testing Explained © Karen N. Johnson 2014
  60. 60. Build testing into the job regardless of who in your agency gets it done. Do not allow testing to be an optional activity. 4. Don’t allow testing to be an extra bill to the client. Testing Explained © Karen N. Johnson 2014
  61. 61. Note-taking, mind mapping, voice recording: whatever it takes, don’t lose a great idea. 5. Capture your test ideas. Testing Explained © Karen N. Johnson 2014
  62. 62. 6. To find issues, you have to test beyond the simple path. Testing Explained © Karen N. Johnson 2014
  63. 63. 7. Ask questions. Especially ask: “what if?” Testing Explained © Karen N. Johnson 2014
  64. 64. Email: karen@karennjohnson.com Twitter: @karennjohnson Karen N. Johnson Testing Explained © Karen N. Johnson 2014
  65. 65. Checking vs. Testing Testing Explained © Karen N. Johnson 2014 ¨  James Bach, Blog: Testing and Checking Refined, http://www.satisfice.com/blog/archives/856 ¨  Michael Bolton, Blog: Testing vs. Checking ¨  http://www.developsense.com/blog/2009/08/testing-vs-checking/ ¨  Elisabeth Hendrickson, Explore It! ¨  Ministry of Testing, http://www.ministryoftesting.com/2012/07/mindmaptesting-and-checking/
  66. 66. Exploratory vs. Scripted Testing Testing Explained © Karen N. Johnson 2014 ¨  James Bach, paper: Exploratory Testing Explained (10pp), http://www.satisfice.com/articles/et-article.pdf#page=1&zoom=auto,-55,792 ¨  James Bach, video, Understanding exploratory testing (6 minute video), https://www.youtube.com/watch?v=I-ItEKt_N_s ¨  James Bach, Session-Based Test Management (1 page), http://www.satisfice.com/sbtm/ ¨  James Bach, What is Exploratory Testing? And How it Differs from Scripted Testing ¨  http://www.satisfice.com/articles/what_is_et.shtml ¨  Cem Kaner, slide set (180pp) A Tutorial in Exploratory Testing, http://www.kaner.com/pdfs/QAIExploring.pdf ¨  Michael Bolton, Resources on Exploratory Testing, Metrics, and Other Stuff, http://www.developsense.com/resources.html ¨  Gregory Mooney, Exploratory vs. Scripted Testing: One or the Other, or Both?, http://blog.smartbear.com/testing/exploratory-vs-scripted-testing-one-or-the-other-or-both/
  67. 67. Schools of Testing Testing Explained © Karen N. Johnson 2014 ¨  Bret Pettichord, Schools of Software Testing https://www.prismnet.com/~wazmo/papers/four_schools.pdf ¨  Bret Pettichord, http://www.testingeducation.org/conference/wtst_pettichord_FSofST2.pdf ¨  Cem Kaner, Blog: Schools of software testing, http://kaner.com/?p=15 ¨  Context-driven testing, http://context-driven-testing.com/ ¨  James Bach, http://www.satisfice.com/blog/archives/134
  68. 68. Quadrants of Testing Testing Explained © Karen N. Johnson 2014 ¨  Brian Marick blog post: http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2 ¨  Lisa Crispin, Janet Gregory, Agile Testing (book) ¨  Gojko Adzic http://gojko.net/2013/10/21/lets-break-the-agile-testing-quadrants/ ¨  Markus Gärtner http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/ ¨  Elisabeth Hendrickson http://www.slideshare.net/ehendrickson/the-thinking-tester-evolved?ref=http:// www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/
  69. 69. Rapid Testing ¨  Rapid testing is a method, a series of techniques and a class, all in one taught by James Bach. ¨  James is arguably the most known tester across the globe. ¨  James posts the slides to his Rapid Software Testing class on his website. www.satisfice.com Testing Explained © Karen N. Johnson 2014
  70. 70. BBST: the Black Box Software Testing ¨  Cem Kaner offers a series of classes taught through slides and videos. ¨  Cem posts the materials to his classes: http://www.testingeducation.org/BBST/ Testing Explained © Karen N. Johnson 2014

×