Exploratory Testing: Finding the Music of Software Investigation


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Exploratory Testing: Finding the Music of Software Investigation

  1. 1. Exploratory Testing: Finding the Music of Software Investigation My friend Steve is an exceptional first time I saw him test software, I was classical guitarist. Watching him reminded of my friend Steve. This time perform is inspiring – he has a rare the tension and resolution wasn’t mastery over the instrument and has related to music composition or the spent years developing his craft. Steve execution of techniques on a musical can also explain the techniques he is instrument. Instead, the tension and using while he is playing, to teach and resolution revolved around ideas. James demonstrate how a student can learn would simultaneously design and and improve their own skills. Steve can execute tests based on his curiosity make a guitar sing, and says that music about the application. He would get is about tension and resolution. If music feedback on a test, learn from it and is all tension, you get uncomfortable as a design a new test. The tension was listener. If it only resolves, it is boring, generated by the questioning nature of tedious repetition. Steve extends this his tests, and the resolution concept to the actual physical actions emerged from the results that a guitarist employs to create certain of those tests. There was sounds. For example, if you play with a something almost lot of tension, you will limit your ability musical in this interplay to do certain tasks. To make music, you between the mind of the need to find a balance between tension tester and the and resolution, and to find this balance, application being you need a mix of knowledge, skill and tested. This shouldn’t creativity. be surprising; as a software tester, Like Steve, my friend James Bach is also James has a well- exceptionally skilled. James isn’t a developed mix of guitarist, he is a software tester. James is knowledge, skill also inspiring to watch while he and creativity. practices his craft. He is a master of skilled exploratory testing: As a software simultaneous test design, execution and testing learning1. James can also explain the consultant and testing techniques he uses while he is musician, I testing, to instruct testing students. The meet a lot of Jonathan Kohl, Kohl Concepts Inc. © 2007 1
  2. 2. Exploratory Testing: Finding the Music of Software Investigation skilled testers who do amazing work. Both software testing and music can be Through experience and a lot of trial interpreted and performed in a variety and error, they have developed skills of ways. they can’t easily explain. Unfortunately, Western classical music is often highly with software testing, there aren’t as scripted in the form of sheet music. many obvious avenues for skill Compositions are written in a language development as there are for musicians. that performers can interpret with their Many software testers don’t realize that voices or instruments. Despite a detailed there are learnable exploratory testing well-disseminated shared “language” skills they can develop to help them for printed music, it is difficult to become even more valuable to software perform music exactly the way the development teams. composer intended, particularly with When I work in a new organization, it musical pieces that have been around doesn’t take long for me to get for centuries, because we don’t have the introduced to the testers who are highly composer around anymore to consult. valued. Often, when I ask testers about The opposite of playing from sheet the times they did their best work, they music is improvisation, creating apologize for breaking the rules: “I unrehearsed, unscripted music. A didn’t follow any pre-scripted test cases, continuum exists between these two and I didn’t really follow the regular styles, because a composition is open to testing process.” As they describe how at least some interpretation by the they came through for the team on a performer, and some performers crucial bug discover, they outline embellish more than others. Software activities that I identify with skilled that plays music is very precise in exploratory testing. Little do they know repeating what is input from sheet that this is often precisely why they are music, but is rarely as pleasant to listen so effective as testers. They have to as a real performer. Music can be developed analysis and investigative boring and tedious when played by a skills that give them the confidence to computer program, and full of life when use the most powerful testing tool we played by a musician. At the other end have at our disposal: the human mind. of the spectrum, successful Exploratory testing is something that improvisation requires skill, and top testers naturally engage in, but because performers study to develop a large it is the opposite of scripted testing, it is breadth and depth of musical theory misunderstood and sometimes and technical proficiency on their discouraged. In an industry that instruments in order to successfully and encourages pre-scripted testing creatively improvise. processes, many testers aren’t aware In testing, test scripts that are written that there are other ways of testing other down are also open to interpretation by than writing and following test scripts. 2
  3. 3. Jonathan Kohl, Kohl Concepts Inc. © 2007 the test executor. Automating these tests is the only way to guarantee that they will be repeated exactly the same way, but like automating music, the lack of interpretation in execution can limit the results. A computer can only find the problems we predict and program it to find. Repeating scripted tests over and over can get boring, tedious, and may musician only feel like idea resolution, without plays a wrong the vital tension created by curiosity. At note, we really notice it.) In software the other end of the spectrum, there is testing, exploring and improvisation, improvisational testing: exploratory even when done wrong, can often lead testing. Pure exploratory testing means to wonderful sources of new that my next test is completely shaped information. Inappropriate by my current ideas, without any interpretations can be a hazard in preconceptions. Pure scripted testing musical performances, but on software and pure exploratory testing are on projects, accidents, or “playing the opposite ends of a continuum. wrong notes”, can lead to important discoveries. Furthermore, software This analogy of music and software projects are faced with risk, and testing isn’t perfect however. Music is exploratory testing allows for us to performed for entertainment purposes instantaneously adjust to new risks. or as practice for musicians who are developing their skills. The end goal is What does skilled exploratory testing entertainment for listeners, skill look like? Here is scripted testing and development, and the enjoyment of the exploratory testing in action. In one test musician. Software testing on the other effort, I came across a manual test script hand isn’t generally done for and its automated counterpart which entertainment, instead it is used to had been written several releases ago. discover information. As Cem Kaner They were for an application I was says, software testing is an investigative unfamiliar with, using technology I was activity to provide quality-related barely acquainted with. I had never run information about software2. To gather these tests before, so I ran the automated different kinds of information, we want test first to try to learn more about what to be open to different interpretations, was being tested. It passed, but the test and to be able to look at a problem in execution and results logging didn’t many different ways. In music, provide much information other than improvisation can have negative effects “test passed.” To me, this is the when used at an inappropriate time or equivalent of the emails I get that say: in an inappropriate manner. (When a “Congratulations! You may already be a 3
  4. 4. Exploratory Testing: Finding the Music of Software Investigation winner!” Statements like that on their what caused failures. I didn’t want to own, without some sort of corroboration write these tests down because I wanted mean very little. to adjust them on the fly so I could quickly learn more. Writing them down I didn’t learn much from my initial would interrupt the flow of discovery, effort: running the automated test didn’t and I wasn’t sure what tests I wanted to reveal more information about the repeat later. application or the technology. Since learning is an important part of testing I ran another test, and without the work, I delved more deeply. I moved on scripted tests to limit my observation, to the manual test script, and followed noticed something that raised my each step. When I got to the end, I suspicions: the application became checked for the expected results, and sluggish. Knowing that letting time pass sure enough, the actual result I observed in a system can cause some problems to matched what was predicted in the intensify, I decided to try a different script. Time to pass the test and move kind of test. I would follow the last part on, right? I still didn’t understand of the manual test script, wait a few exactly what was going on with the test minutes, and then thoroughly inspect and I couldn’t take responsibility for the system. I ran this new test, and the those test results completely on blind system felt even more sluggish than faith. That violates my purpose as a before. The application messaging tester; if I believed everything worked showed me the system was working as advertised, why test at all? properly, but the sluggish behaviour Furthermore, experience has taught me was a symptom of a larger problem not that tests can be wrong, particularly as exposed by the original tests I had they get out of date. Re-running the performed. scripted tests provided no new Investigating behind the user interface, I information, so it was time leave the found that the application was silently scripted tests behind. failing; while it was recording database One potential landmine in providing transactions as completing successfully, quality-related software information is it was actually deleting the data. We had tunnel vision. Scripted tests have a side actually been losing data ever since I ran effect of creating blinders - narrowing the first tests. Even though the tests your observation space. To widen my appeared to pass, the application was observation possibilities, I began to failing in a serious manner. If I had transition from scripted testing to relied only on the scripted manual and exploratory testing. I began creating automated tests, this would have gone new tests by adding variability to the undetected, resulting in a catastrophic existing manual test, and I was able to failure in production. Furthermore, if I get a better idea of what worked and had taken the time to write down the 4
  5. 5. Jonathan Kohl, Kohl Concepts Inc. © 2007 tests first, and then execute them, I come up with ways to manage their would most likely have missed this testing thought processes. Skilled window of opportunity that allowed me exploratory testers use mental tricks to to find the source of the problem. help keep their thinking sharp and Merely running the scripted tests only consistent. Two tricks testers use to kick felt like repeating an idea resolution, start their brains are heuristics and didn’t lead to any interesting (problem-solving approaches) and discoveries. On the other hand, the mnemonics (memory aids)3. interplay between the tension and Musicians use similar techniques, and resolution of exploratory testing ideas may recognize “the circle of fifths” as a quickly led to a very important heuristic to follow if they get lost in an discovery. Due to results like this, I improvised performance. (This isn’t a don’t tend to use many procedural, pre- guarantee though, a heuristic may or scripted manual test cases in my own may not work for you. When a heuristic personal work. is inappropriate, you simply try So how did I find a problem that was another.) Musicians tend to have large waiting like a time-bomb for a customer toolboxes of heuristics, and also use to stumble upon? I treated the test mnemonics as well. One example is scripts for what they were: imperfect “Every Good Boy Does Fine” which is sources of information that could used to remember the notes “EGBDF” severely limit my abilities to observe on the lines of a staff. Skilled testers use useful information about the similar tools to remember testing ideas application. Before, during and after test and techniques. execution, I designed and re-designed I’m sometimes called in as an outside tests based on my observations. I also tester to test an application that is had a bad feeling when I ran the test. nearing completion. If the software I’ve learned to investigate those product is new, a technique I might use unsettled feelings rather than suppress is the “First Time User” heuristic. With them because feelings of tension don’t very little application information, and fit into a script or process; often, this using only the information available to rapid investigation leads to important the first time user, I begin testing. It’s discoveries. I didn’t let the scripts important for me to know as little as dictate to me what to test, or what possible about the application at this success meant. I had confidence that time, because once I know too much, I skilled exploratory testing would can’t really test the way a first-time user confirm or deny the answers supplied would. by the scripted tests. To start testing in these situations, I Testers who have learned to use their often use a mnemonic I developed creativity and intelligence when testing called “MUTII”. (A composer named 5
  6. 6. Exploratory Testing: Finding the Music of Software Investigation Nicolo Mutii helps me remember it.) Can I easily implement the tasks given This mnemonic helps me maintain the information and design of the consistency in the way I think about product? testing. Expanding the mnemonic: Before I start testing the application, I Market—The targeted constituency of gather information about the market users this software is intended for. For and the users from the business. This example, “the finance department”, or helps frame the kinds of tests I will “medium sized accounting firms”. develop when I use the software for the first time. If I’m not familiar with the Users—The actual users who will use the market and users, I will also ask for software. Who are the users? What do typical tasks engaged in by the users. they do? What are their motivations for using our software? When I start testing, I open my notebook to take notes of my Tasks—What are the tasks that the users observations and thoughts, and any will use this software for? What are bugs I find. (See Figure 1.) I begin some typical tasks in their work? designing a test in my mind, execute it Information—What does the product tell with the software, and observe the me about the tasks it automates, and results. I keep repeating this process, how I can perform them? changing tests, and referring back to my Implementation—Is the software easy to MUTII mnemonic. Not only does each use as a first time user? Is it reliable? letter of the mnemonic help me frame 6 Figure 1: Excerpt from exploratory testing session notes
  7. 7. Jonathan Kohl, Kohl Concepts Inc. © 2007 my testing, but the acronym helps me tests. Exploratory testing—like quickly design and execute many tests improvising—helps me adapt my under each section as I go. I may also thinking and my actions based on what use other heuristics and mnemonics as I the software is telling me. This is a test, if I find areas to explore differently powerful concept. You can seize upon or more deeply. opportunities as soon as you observe them. Furthermore, you can adapt As I work through the application, I quickly to project risks, and discover may find mismatches between the and explore new ones. By developing application and the market it is intended skills to manage your thinking about for, and the information supplied to testing, you no longer have to wait for users. If I have trouble figuring out the spontaneous discoveries to appear out software’s purpose and how to use it, so of thin air, and not be able to explain will end users. This is important why you found a particular problem, or usability information that I write down repeat it. in my notes. If the software isn’t usable, it isn’t going to sell. I invariably find Developing exploratory testing skill bugs in this first testing session. I puts you in charge of your testing explore them, take notes so I can report approach. Skilled software testing, like them later, and design new tests around skilled musicianship, is often referred those bugs. After the session is over, I to as “magic”, simply because it is will have bugs to report and usability misunderstood. Music follows a set of questions to ask. I will now have a patterns, heuristics and techniques. If model developed in my mind for testing you know a handful, you can make this software. My brain will work on music quite readily. Getting there by this model constantly, even when I’m trial and error takes a lot longer, and not testing, and other testing activities you’ll have a hard time explaining how will help build this and other models of you got there once you arrive. Testing the software. using pure observation and trial and error can be effective, but can be Using heuristics and mnemonics helps effective much more quickly if there is a me be consistent when testing, but I system to frame it. don’t let them rule my testing actions. If I observe something suspicious, I Skilled exploratory testing can be a explore it. If something feels wrong, I powerful way of thinking about testing. investigate, and confirm or deny that However, it is often misunderstood, feeling with defensible facts. It’s feared and discouraged. When we common to switch from heuristics and dictate that all tests must be scripted, we mnemonics to pure free-form discourage the wonderful tension and improvisation and back again, or to resolution in testing, driven by a curious improvise around highly structured thinker. We limit the possibilities of our 7
  8. 8. Exploratory Testing: Finding the Music of Software Investigation software testing leading to new, them. In the right hands, software tools important discoveries. We also hamper and processes, much like musical our ability to identify and adapt to instruments, enable us to realize the emerging project risks. In environments results we seek. There are many ways to that are dominated by technology, it perform music, and there are many shouldn’t be surprising that we ways to test software. Skilled constantly look to tools and processes exploratory testing is another effective for solutions. But tools and processes on thinking tool to add to the testing their own are stupid things. They still repertoire. require human intelligence behind Jonathan Kohl is a software testing consultant with Kohl Concepts Inc., based in Calgary, Alberta, Canada. Jonathan writes about and speaks on software testing. Read more of his work at www.kohl.ca. Contact Jonathan at jonathan@kohl.ca. 1 Bach, James. (2003) Exploratory Testing Explained http://www.satisfice.com/articles/et-article.pdf 2 Kaner, Cem. (2004). The Ongoing Revolution in Software Testing. Presented at Software Test & Performance Conference, December, 2004, Baltimore, MD http://www.kaner.com/pdfs/TheOngoingRevolution.pdf 3 Bach, James. (1999, November) Heuristic Risk-Based Testing. Software Testing and Quality Engineering Magazine http://www.satisfice.com/articles/hrbt.pdf 8