Your SlideShare is downloading. ×
  • Like
Wit magazine april_2008
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Wit magazine april_2008


WhatIsTesting magazine. Published in April 2008

WhatIsTesting magazine. Published in April 2008

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. What Is Test ng India’s 1st Software Testing Magazine April’08, Issue Reduce number of testing scenarios with this insight on - Cover Story Pg 21 Risk based TestingInside this issue Are you TEAM ready? Putting the “Test” back in “Test plan” Pg 57 Some things I have learned in Pg 44 Software Testing Pg Morning Ideas Pg 04 Have I got a deal for you? Finally Usability Testing Pg 28 Pg 40 14 With little Preparation & Teamwork you can pave the way for new Team Memers. and more...
  • 2. Welcome to   is the first conference being organized by PureConferences in India.  We intend making  an experience for all participants by creating a platform for professionals and experts from the testing community of the Globe  will revolve around ’Agility in Testing’The theme will explore and expand on the need for testing teams to adapt to the changing demands of release and quality. will cover diverse topics. Keynotes, Tutorials, and Paper presentations by renowned speakers known nationally and internationally will be one of the highlights of the conference. Let’s mark the conference program in our calendar today itself.  will offer opportunities to the partners to associate themselves with this endeavor and showcase their products in the conference. We are sure you want to participate and/or present a paper or tutorial. Do visit us at
  • 3. Issue1: Vol 1: April 2008 ContentsCover Story Is bad testing pg eating your 21 Pr fits? 04 FROM THE DESK: Morning Ideas by Sunil Gupta Good design is key for a good and reliable product 08 THE PROCESS ROOM: Software Testing: ~ A field par excellence by Sudhir & Anjan Software testing as a field has brought in a high level of confidence in customers and users and has thus far provided far more challenges to the new age engineers and academicians. 12 TEAM MANAGEMENT: Are You Ready? by Michael Bolton With a little preparation and teamwork, you can pave the way for a new team member. 21 COVER STORY: Is Bad Testing Eating Your Profits? by Rex Black Testing should focus on mitigating specific risks to the quality of the system. Sequence of test execution should be driven by associated risk... 28 THAT TIME OF YEAR: Have I Got A Deal For You by Matthew Heusser The methods and extend of our testing are all choices; you could even say they are trade-offs. 33 LEARN HOW TO TEST: What a Tester Should Know, even After Midnight by Hans Schaefer Testing The Normal Way is Not Enough.... Where you find defects, dig deeper! 40 LEARN HOW TO TEST: Finally Usability Testing? by Erik van Veenendaal Will it finally happen!Other Stories 44 FROM DEV DESK: Some Things I’ve Learned in Software Testing by Jonathan Kohl 50 CONTROVERSY CORNER: QA is More Challenging than Development by Mrityunjay 53 Is scripted testing bad? by Vipul Kocher 57 Putting the “Test” back in Test Plan by Paul Carvalho 1
  • 4. Tea testers look ataspects of dried & brewed tea leaves and the liquor Likewise, PureTesting goes into various aspects & nuances of soĞware testing We build innovati ve, end-to-end solutions, and manage critical testing processes, and reduce total cost of producing quality soĞware • Banking & Finacial Services • Datacom & Telecom • Pharmaceuticals • Embedded Systems • eLearning • EAI & ISVs Test Consulting • Testing Services • Testing Products India • USA • UK • NZ +91 (120) 4621010; Global Software Test Consulting & Services company
  • 5. Editor’s Note Release of any software is a moment of mixed emotions, pleasure andpain, joy and anxiety -- joyousness of seeing the hard work resultingin a release and the anxiety of the unfound bugs that could have beenfound and fixed. After a lot of effort, this magazine is also being released with similaremotions. We take great pleasure in introducing Version 1.0 of WITmagazine. We hope to have many more versions of this magazine, bothmajor and minor, in years to come and hopefully with minimum numberof patches. We also hope that the future versions of the magazine willhave more features, will be more robust, will have an expanded userbase and establish itself as a dominant player. Instead of the usual Editorial about the contents of the magazinewhich you anyway can find out from the table of content, we take thisopportunity to share with you the reasons for this magazine and whoare the people who have contributed to it. Many of the technical magazines “grow-up” with time. The articlesbecome more “philosophical” and more abstract. Readers who areregular subscribers too grow with the magazine but most of the newcomers start finding the magazine “too abstract” for their liking. WITmagazine hopes to keep itself relevant to both old readers who grow upwith the magazine and new readers who are either new to the testingprofession or new to the magazine. Our sincere thanks to the initial group of people who helpedbrainstorm the concept of the magazine. They are – Bernard Homès,Danny R. Faught, Johanna Rothman, Kamesh Pemmaraju, MatthewHeusser, Michael Kelly, Rex Black, Scott Barber, Stefan Steurs. Withoutthem the magazine would not have become a living reality. All goodthings about the magazine are because of this group, the authors andother well wishers. All bugs are because of us. For putting together this version of the WIT magezine, I acknowledgethe efforts of Ashwin Razdan for collaborating with the authors andcollecting all the articles; Parag Sapre for designing the elaborategraphics; Anushree Tewari for providing editing support, and SatishThakur for giving it the final shape in which you see the WIT magezine.Hope you find this version readable and useful and we hope to have abetter V 1.1. ~ Vipul 3
  • 6. From Daily Life Morning Ideas the Morning Desk IdeasThough a good design is key for a good and reliableproduct, a robust test and evaluation of the productonly ensures that the product meets all-explicit andimplicit requirements and is ready for deployment andusage.O ne morning when I was engrossed bags had torn almost at the same place. in my newspaper, my daughter Since it was not possible to repair the bag created panic as her school in ten minutes, thus she carried her stuff in bag had given way at one of the another bag but this episode left me thinking. shoulder belts. I realized that the bag was made to meetI was annoyed at her letting the functional requirements of the studentme know only when I was such as adequate space to carry stuff,completely torn while she would compartments for lunch box, pencil box andhave seen this happening over water bottle. The bag was also providing athe last few days but my bigger good aesthetics but lacked the robustnessreason of annoyance was due in terms of handling the stress that comesto the fact this was the 3rd or on the bag with daily usage....4th time the different school4
  • 7. by Sunil Gupta, Head of Testing Practice : Flextronics Software Systems by For product organizations, the composition and organization of the test team plays a very crucial role in the success of the product. Enough time Composition and and energy needs to be spent by senior test organization of the test team folks to plays a very crucial role in • Visualize the product (software and the success of the product. hardware) • Look at the various field configurationsTwo things emerged when I extended my thought • Identify the tools required for testingprocess further. • Look at the FOUR views:One, product making whether it is a school o User viewbag or a large and complex telecom product o Maintenance Viewrequires good understanding and research interms of features, stress areas, usability, etc o Functional/Operational Viewwithout which the product would give numerous o Performance Viewproblems to the user as the school bag gave to In order to achieve a good understanding ofmy daughter. the product before the exhaustive test cycleSecond, testing plays a very crucial role to commences, a few people from the test teamensure that the product meets all requirements need to be dedicated to carry out the Testincluding the ability of the product to handle Engineering tasks (as defined above). This setnormal and abnormal stress conditions. of people should have the mandate to carry out the verification and validation tasks.If we take the school bag example, it is veryimportant for the product manufacturer to In the school bag example, the User View would ensure that the various be to look at all the ways in which the school stitches in the bag take up bag would be used by a user. There would be the full bag load else there several user scenarios such as: are going to be numerous • Student lifting the bag from one belt dissatisfied customers like me. • Student lifting the bag using the side belt and loading on its back Though a good design is key for a good and reliable product, • Student fully loading the bag with books but a robust test and evaluation weight may not be too muchof the product only ensures that the product • Studentmeets all-explicit and implicit requirements and partiallyis ready for deployment and usage. loading theIt is seen across the IT and software industry books butthat the organizations give a lot of focus to hiring the weightand developing software development teams may havebut do not focus enough on developing testing exceeded theteams. The test team is put in place either verylate on the programs or the test teams are notorganized well enough to provide the vital valueexpected from the Test team. 5
  • 8. Morning Ideas Suddenly I realized that it was 9:00 hrs by the watch and I had a 9:30 am meeting with the Product testing and Quality Group to review the Post Release Defect Density of the products that we shipped in the deployment is a very last quarter and had only 30 minutes to reach involved and rigorous task office. Thanks to god, for my home is only a 7 minutes drive from office. recommended weight Bye for now……Maintenance View would be to look at aspectssuch as ease of cleaning and washing the bag;ease of repairing the bag and how easy it is tofind the spares such as zippers, buckles, etc.Functional or Operational view is anotherimportant aspect of testing. Tester has toconsider all the functional aspects of theproduct. In this example, aspects such asfollowing would be tested: • Ease of keeping books in the bag • Access to all the pockets of the bag • Ease of closing and opening the bag • Ease of lifting the bag on shoulder and on the back • Enough space to keep books, lunch box, and water bottle, etc.Last but not the least, the Performance view ofthe product in terms of load handling, abilityto bear stress due to regular use throughload balancing is considered. In the example,tester has to take following aspects intoconsideration: • Testing the bag for various load and stress situations • Testing the bag for full load capacity • Testing the key joints/stitches under the stress situations • Identify the weak areas and strong areas of the productProduct testing and deployment is a veryinvolved and rigorous task and involves the bestof people and the best of the practices.6
  • 9. Neilsoft is a specialist engineering services & solutions company focused on helping our clients enhance their product engineering efficiency. Our services in the software product engineering domain span the entire product lifecycle including development, testing, and localization engineering.T E S T I N G E X P E R T I S E Glo ba ise l ert de Compatibility p Testing liv ex e n ry ai In Te mo in on m st st st ti Do al in Te gra g de la g tio te l In n rage Sto S Partnerships with ind ec uri C AE Performance Functionality ty Testing Testing M CA PL M C AD ustr y le s M Tes sse in ed ul ti st at ad ti- ng ce g Te tom by ers te ro u A p st bu Certification ro Testing & ven Pro | Pune Bangalore Canton Chicago Cambridge Dalian ) +91-20-2605 3003 Focus verticals: Construction Industrial Machinery Software Transportation Energy
  • 10. The Software Testing: ~ A field par excellence Process Software Corner Testing ~ a Field Par ExcellenceApart from giving fillip to the economy, Software test-ing as a field has brought in a high level of confidencein customers and users and has thus far provided farmore challenges to the new age engineers and acad-emicians.Introduction and ITeS segment known as Software Testing. Renewed growth of the Software Industry hasAwareness on Software Quality has, over the last helped testing achieve the position of beingcouple of decades, become one of the defining called the watchdog of quality. Apart from givingfeatures not only in Indian software sector but fillip to the economy, Software testing as a fieldalso from a global perspective. Thanks to the has brought in a high level of confidence indotcom crash which was an eye opener for customers and users and has thus far providedeveryone in the field to look back and focus far more challenges to the new age engineerson enhancing the quality of the products that and academicians. Looking at some of theare already in the market for many years. The astounding figures:quality initiatives taken through the last coupleof decades, have now become all pervasive and 1. Bangalore has seen a 24% growth in itsthus given rise to a new avenue within the IT recruitment drive in software testing. It is8
  • 11. by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL, Bangalore Facts from Industry 1. $3.0 billion of $4.6 billion in outsourced Composition and organization testing is sent offshore, throwing up of the test team plays a very opportunities for companies in India that thrive on low-cost knowledge workers. ~ crucial role in the success of Source Aztech Software - Bangalore the product. 2. Testing could make up to 25-50 percent of software budgets. Independent testing is growing at 50-65 percent while the part forecasted that India needs about 18000 of work done offshore is growing at 35-40 test engineers in the year 2005-2006. percent. ~ Source - Partha Iyengar, vice-2. With India having more than 89 companies president at industry researcher Gartner at SEI CMM Level 5 assessment; 275 Indian 3. Testing service team at Wipro has jumped software and BPO companies acquired four-fold to 2,400 in two years. In the quality certification; and more and more nine months to December, revenue grew IT-ITES companies have dedicated quality 90 percent to $64 million, three times departments responsible for developing and the industry average. ~Source - C.P. deploying the quality policies and reviewing Gangadharaiah, VP Testing services, Wipro them. 4. The global market for software testing is3. NASSCOM survey has indicated that the around $13 billion. In India alone, the road ahead for the Indian software and demand for software testing professionals services companies within the quality is expected to touch 20,000 to 30,000 arena was largely dictated and tuned to by December 2006. . ~Source – Internet the developments taking place in the global Search marketplace.4. The future would see organizations move Recent Trends from quality assurance to business With this background, we can ask ourselves assurance and focus on information security - what conclusion we want to drive from it? complying with international legislations, Why suddenly is everyone so statistical about developing new global delivery capabilities gathering all these figures? To take a close look and enhancing quality processes in new below we see from 50’s onwards till 80’s either business areas. the products were not tested or if they were then it was out of compulsion.This is testimony to the fact that softwaretesting is poised to grow at a phenomenal rate Quality as a term itself took the real shapein future. With all these happenings, no one can under the leadership of Borris Beizer who gavedeny the conclusion that software testing has the awareness to the industry and the worldgrown as an independent field within IT/ITeS about its very existence. The role of quality hasservice segments. And it has given a clarion become a necessity than a need in the softwarecall to everyone within the professional arena, industry.specifically academicians, to closely monitor, If we start looking into history, we will findanalyze and push forward the new horizons of thousands of such cases where a small mistakethe technology movement that software testing in the software has brought havoc to life andas a field is going to provide. Looking at some money. The most recent one is the Columbiamore interesting statistics on testing industry: Space Shuttle crash. All such incidents have 9
  • 12. Software Testing: ~ A field par excellenceespoused within the IT-geeks eagerness towards human life such as Insurance, Healthcare,“checking before delivering” approach. On the Banking, Financial services, Pharmaceuticals,contrary, there was time when products were logistics etc. What this means is that, softwarereleased without or with minimal testing done. developed in each of these Industry verticalsOf course, many a time, these are dictated by need testing – some rigorously and some whichhow the market wants them. Time-to-market can give adequate confidence in the product. Atwas probably more important than quality the end, software testing too is spanning acrossof the product. But, time has changed post all these Industry verticals. When it comes todotcom crash era and software quality has the domains, the field looks even more lucrativemaximum thrust in the market today, be it by the amount of challenge each one of themconsumer products, electronics, manufacturing provide to the individuals involved in testing.or software. And quite rightly, software testing Therefore, as software testing started to supportis enjoying its share of this quality conscious the development activities in these domainmarket. areas, it became increasingly necessary for the test engineers to gain knowledge and specialize Testing as a field has now been more within each of these domain areas. This is strategic and approach-centric another area where there is a huge potential thus adding new dimensions as for test engineers to grow and make an impact. well as many challenges to it. This led to different teams being formed, some Previously testing used to be an heterogeneous and some homogeneous groups afterthought activity in the post to cater to testing software in these domains. release phase when the clients The requirements of the client keep oncall and say “we found something wrong in growing and we as organizations need to meetyour delivered software”. But with time, it is no the demands of the end users to stay in thislonger the case. Any small error today is a big business. This translates to the whole gamutissue from a client perspective. With more and of technology issues starting from software,more third party software testing companies hardware to third-party products. Software cangrooming overtime, the availability of a talented be legacy application running on a mainframeresource pool, making timely deliveries and the or the latest technologies such as .NET, J2EEcommitment to meet the quality expectations etc... Technology as we know has advanced fromof the customer has added many feathers to the days of mainframes to today’s applicationsIndia’s software testing market. Needless to running on a PC based Linux systems. However,make a note that Software testing as a field at the same time, applications developed on thewithin IT/ITeS has now emerged as a strong legacy technologies have not gone away or havearea of business solutions and a source of not been replaced with new technology; they aremajor revenue generator and emerging more as still being used in Banks, Insurance companiesa serious business goal as well an intrinsic partwithin the IT/ITeS organization culture.Future Challenges There is software in almost every aspect of human life....Domain and Technology Perspective software developed... needThe growing demand from within the customergroups as well the growing demand of the testing -- some rigourouslyorganizations to meet and extend their client and some which can givesupport has made software testing as a field adequate confident in thenothing but a paramount knowledge pool.There is software in almost every aspect of product.10
  • 13. by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL, Bangaloreand many others. This even complicates the Conclusionscenarios wherein the vendors have to support Moving a little off track to give a comparativeboth old applications as well as new applications. analysis we see in the past Public AdministrationThis tells us how complex it can be for testing has struggled a lot to give itself an individualto handle such complex set-ups. Besides, the identity and for a long time shared a part of itsapplications run on varied platforms from glory with either Political science or Sociology.different vendors including: Windows, Linux, But today it has gained a public reputation andMacintosh, IRIX, AIX, unix & Mainframe. We has become a part of everyone’s life.are currently looking from a macro level. Sincewith the growing numbers of different flavors of So does it hold true for software testing which hasan individual OS (e.g. Linux as such has Linux remained cocooned within Software EngineeringWorkstation2.0/3/0, Advanced Server 2.0/3.0, or Software Development Life Cycle. But nowEnterprise server 2.0/3.0 etc.) the coverage from the field has not only gained reputation but alsotesting point of view becomes immeasurable. has stood on its own. Software testing as a fieldDoes that not give a test engineer enough has come of age bearing the pain of “We can dochallenge to work on so many platforms? Of without you” to share the glory of “With you wecourse yes, it’s not that developers only code are right through”. A field that is still growing,for making an application run over different shining and awaiting to flourish still needs a lotplatforms but the test engineers too have their of planning & strategizing from all sections inown way of reaching end to end. Apart from this the Industry and academicians. As we need towe can also predict a good deal of exploration remember the following lines:in areas like Client-Server, Web (2-tier, 3-tier) “The Road to Excellence is a never endingapplications, Mainframe application and an Road” ~H. James Harrington, The Improvementendless list of such applications. ProcessWhat we discussed as far as technology isconcerned is only the tip of the iceberg. Thereseems to be an endless list and mushroomingapplication areas to be explored in the time tocome. 11
  • 14. Team Are You Ready?Mana-gmentAreYouReady?With a little preparation andteamwork, you can pave the wayfor a new team member.Use a checklist to make sure you have everythingready before the new team member arrives.● Compile all necessary documents into one binder that can be given to any new arrivals.● Be sure to provide some context about the company and the team to help the newbie get acclimated.12
  • 15. by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.I f your company is typical, it is hiring agreement. Make sure this, too, is prepared more and more temporary workers while in advance, and that all agree on each of its scrambling for every bit of efficiency and points before the contractor’s first day. If your value. Hiring a contractor to help with human resources department has policies andtemporary problems can be cost-effective, but procedures manual appropriate for your hire,wasting his or her time undermines the purpose make a copy available to the contractor earlyof the exercise and costs hundreds of dollars in the game. Human resources will typicallyper day. Moreover, people simply don’t work as require various kinds of information fromwell when they feel uninformed, frustrated, or your new hire. Be sure to ask in advance whatstymied. On the other hand, a contractor who information they’ll need so that you can passcan hit the ground running will make better on any questions to the contractor prior torelationships, will be more effective, and will her arrival. Human resources or security willuse your time and hers productively. typically supply ID badges, pass cards, access codes, and keys to the office, restrooms, or This article describes ways to prepare other secure facilities. These items should befor a new contractor; however, the tips here available and tested.could just as easily be applied to a temporaryemployee, a new permanent employee, or in If the contractor is arriving from out of thesome cases, a transfer into your department. I country, make sure that your human resourcesdon’t assume any particular job description or department and the contractor have fulfilled alltitle for the person, nor do I assume anything requirements related to visas, work permits,about the person’s technical skill in any given and the like. The accounting people will needcategory. Neither should you make such to deal with administrative and taxation issues.assumptions. A crackerjack developer may be Make their jobs easier by coordinating andcompletely oblivious to network configuration relaying the required information in advance,issues — especially in a new environment both from your staff and from the contractor.— and a brilliant documentation writer might Issues related to eligibility, tax numbers, taxbe flummoxed by a cryptic voicemail system. forms, and withholding information should beYour contractor’s time is best used to solve the sorted out concurrently with the contract.problem for which you’re hiring her, not herown infrastructural or contractual issues.Contracts and Administrivia Restrooms with locks butBefore beginning work, your contractor will no keys are scourges upontypically sign a contract outlining the scope of the land. When I started atthe work to be done, the deliverables expected, one small outfit, it was aand a schedule detailing when each element week before security couldof the work should be completed. The morespecific the contract, the less opportunity there produce a restroom keyis for miscommunication and disagreement, for me. It wasn’t only a bother for meand the better your interests are protected. to ask my new colleagues for a key—IYour contractor may have requirements that areoutside the scope of your company’s standard was interrupting their work, too.contract, or she may have to provide certainkinds of documents relating to work eligibility. At this company, the restrooms wereMake sure these kinds of issues are ironed out locked, but the door to the server roombefore the contractor is slated to begin work. was always open.Most contracts come with a non-disclosure 13
  • 16. Are You Ready? Elizabeth, a European There are few things that make a contractor contractor of my happier than being paid on time and with a minimum of hassle. Ask your accounting acquaintance, was hired department what they’ll need from the contractor to work in an American to pay invoices promptly. company. She submitted Ensure that expense and reimbursement policies an invoice, covering salary are clear from the get-go. If your company uses and expenses, which was a standard expense reconciliation form, make sure that a few copies or an electronic template rejected on the basis of a couple are available to your contractor. Again, your of questions from the accounting most friendly and helpful contact person in department about her tax withholding accounting should be available to the contractor status. She dropped in to investigate to help sort out confusion or difficulties. the problem by speaking with Mark, IT Issues the accountant who had sent her the Notify the IT support department that a new hire mail rejecting the invoice. “I don’t have is on her way, and request the specific things time,” said Mark. “Not even for just a you’ll need and the date you’ll need them. Most new arrivals will need at least one computer set couple of questions?” asked Elizabeth. up, linked “No,” Mark replied. “It would be a to the network, loaded with tools, attached to different story if you were an employee printers, and tested. If you’re hiring a developer rather than a contractor. But,” he said or tester, additional terminals or workstations imperiously, “I have more important may be necessary. Don’t forget to tell the IT department about this, and above all, don’t strategic things to do.” This incident assume that they’ll know what the new arrival occurred at a company where more needs. Your new arrival will need access to than half the staff was made up of various areas of the network, so make sure contractors. she has the appropriate set of access rights. If your company’s security is very restrictive—for Eventually the problem was sorted example, limited out, but it meant inconvenience and Internet access—notify your contractor exasperation for Elizabeth and her beforehand so that other arrangements can be made for Net-based research tools and manager, and embarrassment for resources. An analog line might be required Mark’s manager. On the other hand, for some purposes. If specific tools—examples Elizabeth’s final report included a include compilers, configuration management subtle recommendation—which Mark’s programs, and testing utilities—are required, have these loaded and tested on the contractor’s manager followed—to give Mark less machine in advance. Be sure to follow up, strategic work to do. Whether Mark is verifying that the work has been done and currently doing strategic work or not, that resources, configuration settings, network policies, and default passwords are documented. he’s doing it somewhere else. Make sure that email accounts have been set14
  • 17. We challenge you. At GrapeCity, we challenge you with assignments on the latest .NET and Java technologies, enabling you to emerge a winner in this game called life. Lines of Business Business SolutionsGrapeCity is a software development multinational company Custom Applicationswith over 650 employees across Asia and the United States. For Mobile Applicationsmore than 25 years, we have brought optimized software solutions CRMand services to enterprises around the world. ERPWe rely on four key principles to help our clients achieve their goals: Technical Servicesthoroughly understanding our clients’ business objectives, providinghighly personalized experiences, maintaining a strong emphasis on Mobile Portingquality, and adhering to the highest ethical standards. Quality Assurance Technical Support Send your resume to Component Development now! Technical ToolsGrapeCity India Pvt. Ltd.A-15, Sector 62, Noida 201307 Phone: +91 120 247 0123 Fax: +91 120 247
  • 18. Are You Ready?up and checked by sending amessage to the new arrival’s Note the people who are most likely to beaddress, and then check with helpful in answering questions or fixingthe IT department to be surethe message has been received problems, and provide their contact the new mailbox. In addition,obtain a document from the IT voicemail system. Have the telephone installed,department on how to change network login and a number ready for the contractor when sheand email passwords. arrives. Make sure that this number is listed in the company directory — and remember thatEveryone needs a working telephone and directories can exist on the network, in the voicemail system, and with the receptionist. Despite everyone’s best intentions, a company’s information infrastructure can be complex The new arrival can often and confusing to a new arrival. Have the IT help people to notice department provide the name of a specific old bumps in the road. I person—a guide or sponsor who is helpful and knowledgeable about the company—to help the was brought into a test contractor navigate through network access, organization to help email configuration, and printer problems. In identify problems in testing addition, make sure that your contractor will be able to find her default printer, email address, efficiency. There were two types of network login name and password, and IP network drops in the test lab—one address. Try to counsel colleagues against for the company’s internal network sending email announcements saying things and one for the test network. These like “This week’s build can be found in the usual places.” Instead, make the details explicit. weren’t labeled, since everyone who used the lab knew which was which. I Providing Context suggested that this shouldn’t be taken The contractor has been hired to accomplish for granted as one of the locals helped some task or solve some problem. If people within your company are having difficulty me find the right connection, and I sorting out an issue, imagine the difficulty that used a bit of masking tape to label the a stranger will face! drops nearest to my workstation. The Even people who have worked inside your next day, a fairly senior tester reported company and who know the general environment a defect that was later found to be need background information when they spurious, at a fairly significant waste of arrive in a new position. This is yet another motivating force for your company to develop time and effort. The misunderstanding clear specifications, design documents, plans, happened because he had plugged the schedules, budgets, and so on. Start by writing a test machine into the wrong network brief description of what the company and your division do, and prepare an organizational chart connection. The day after that, labels or departmental roster. At the very least, identify magically appeared on all of the yourself and your direct superiors and reports, network drops. along with the other people with whom you expect the contractor will be working. As you’re16
  • 19. by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.preparing this information,note the people who are most The processes and documents thatlikely to be helpful in answeringquestions or fixing problems, you prepare for the arrival of oneand provide their contactinformation, so as to minimize contractor will be highly recyclable.the amount of time that the and you’ll quickly have a list that is easilycontractor must wait—unproductively—for maintained and printed.answers. Document the key business processesand workflows associated with the new arrival’s One of the benefits of traveling, from the pointtask at hand. If you provide answers early, there of view of many contractors, is using free timewill be less time spent wondering and askingquestions.Don’t get hung up on tools or formatting: A At one company where Idocument composed using a text editor or was a full-timer, we hostedeven a pen and paper is infinitely superior to adocument that doesn’t exist at all. When offered a number of developersa choice between a detailed Notepad file and an during an outsourcedempty Gantt chart, I’ll pick the former every time. project. We created a listSimply make sure that the information exists, of local restaurants andand that it can be found and read easily. Make ita mandate from the beginning, and check with tourist attractions by sending outthe new arrival often. If the new arrival feels email to everyone in the department,that she is being forced to do detective work, soliciting one suggestion from eachencourage her to keep a log of what she neededto know and what she found. Make sure that person. I compiled the suggestionsone of her deliverables is a list of the things that into a single document, removed thewill be useful to her successor, and be sure to duplicates, and posted the results tobudget time for her to prepare it. an internal discussion board. A few ofCity Guide the suggestions inspired still more.If your new arrival is coming in from out of town, Some might suggest that discussionshe’ll feel more welcome with a guide and some of restaurants on company time andidea of where important services can be found. company bulletin boards is wasteful.If your company is providing accommodations,make sure that the contractor’s lodgings are Our experience was that the one-quiet, secure, comfortable, and reliable, and suggestion technique cost next to nothat they’re ready for the contractor by the time and was a way of getting to knowend of the first day of work, at the latest. If people—and people who knew eacharranging accommodations is the contractor’sresponsibility, provide a list of facilities in a other better almost always workedvariety of price ranges. While it’s nice to be close smarter and work, don’t strand the contractor near anindustrial park without food or services nearby. We also found that a handy list made itPrepare a list of restaurants in the area that easy to book company gatherings, tocater to a variety of tastes and budgets. provide directions, and so on.Ask your department for recommendations, 17
  • 20. Are You Ready?to explore their surroundings. Survey your people she’ll need to know. An introductionstaff for a number of interesting— and possibly from you is likely to carry more significanceundiscovered— places to visit in the area. If than one from a subordinate, so be preparedthose places are somewhat off the beaten path, to do this yourself. Even if you are routinelyask the person making the recommendation to busy, getting your employees and contractorsprovide a link to one of the several useful Web- to work together is a tremendously worthwhilebased mapping services. If everyone pitches in a investment. A contractor’s effectiveness oftensuggestion or two, compiling them into a helpful depends upon being able to meet and connectguide shouldn’t be burdensome. When you’re with other people. Most contractors meet theirfinished, you should have a complete package new colleagues in business meetings, but moreof general information that can be passed to informal circumstances can lead to friendlierany new employee or visitor on paper, via email, relationships. On the introductory tour, takeor on an internal Web site. Ask the new arrival your time. Cover the important places and people,frequently for feedback on the checklist, and but be relaxed and try to avoid a hard-and-fastrefine it accordingly. agenda. Spend a few moments introducing the new contractor to each staff member. ExplainOn the Job quite generally what the new arrival is thereMake sure that you schedule some time on to do, and describe your staff member’s role,the contractor’s first day for introductions and how each may be able to help the other.and a tour of the essentials. You should be Don’t just concentrate on peers and colleagues.prepared to walk around for at least an hour Spend a few moments chatting with other usefulwith the new arrival, introducing her to the contacts with whom your new arrival will need to interact—people in network services, reception, the mail room, and accounts payable. As you’re The kitchen or lunchroom walking around, show the new person all of the is not only the place useful and necessary places around the office. for refueling but also a An early stop should be the restroom. If keys or pass cards are required, make sure that the good place to meet and contractor is given them upon her arrival. The IT converse with others. department should have set up a default printer Linger for a while and and informed you which one it is. Include this encourage chatting, even on the tour, along with specialty printers and other equipment that might be useful. Also if it’s not directly related to the task on the tour, include visits to the office supply at hand. Spend a little petty cash cupboard, the photocopier, the first aid kit, and on group lunches, outside coffee the lunchroom. For the duration of the contract, the contractor is a member of your company breaks, or visits to the pub with your and your department, and thus should be contractor and the rest of your staff. accorded at least the same kind of assistance If these informal approaches seem and respect given to your permanent employees. of questionable value, consider that If you are the contractor’s supervisor, you must be prepared to go to bat for her, and the rest of productivity and effectiveness are your organization should be prepared to take often based on personal relationships the same approach. and rapport—do what you can to foster No matter what the task, there’s nothing worse them. than having to research and answer the same questions over and over again. Keep a list of18
  • 21. by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.frequently asked questions, their answers, and complete, can be reused and refined, but rarelywhere someone can go for more information. In will you have to start it again from scratch.many cases the FAQ information can be posted You’ll need to review things periodically, buton an internal Web site, captured in a database the benefits will be immediate: happier, better-within workgroup software, or collected in a prepared, and—most importantly— morepiece of email. If you don’t have such things set productive workers.up, consider making the collection of FAQs atask for the contractor, making such a documentpart of the contractor’s package of deliverables,since your contractor will have plenty of insighton what’s important for a new arrival to know.Reuse and RecycleAll this might sound like a lot of work— and thefirst time you do it, it will be.Luckily, the processes and documents that youprepare for the arrival of one contractor will behighly recyclable; they can be used again andagain, regardless of whether the new arrival isa temporary or full-time employee. Since you’llbe consulting with other departments, you canexchange ideas and plans, and improve the lotof new arrivals companywide. The work, onceChecklist for a New Team * Policies and procedures manualMember * Expense and reimbursement policyPrepare for Arrival * Expense forms* Signed contract * Telephone and voicemail instructions* HR info from contractor * Telephone directory, with key contacts* Keys, badges, and codes ready and highlightedtested * Internet policy* Visa, work permit, and eligibility * Password and login instructionsconfirmed * Software resource manuals* Resolution of tax issues * Names of key contact people in IT, HR,* Telephone and voicemail set up Accounting, etc.* Phone number listed in directory * Organizational chart* Set up computer: * Division responsibilities networked * Overview of company all applications installed and tested * Key business processes and workflows email account related to job login * Guide to city, if applicable passwords * List of frequently asked questions (with answers) printer connection Documents to Provide 19
  • 22. CORPORATE OVERVIEWHexaware is a leading global provider of IT and BPO services. Over 40 of our clients are Fortune 100 companies. Weenable clients achieve competitive advantage by co-developing innovative IT/Process capabilities delivered throughflexible business models. We focus and have achieved leadership positions in domains such as HR, Banking andFinancial Services, Insurance, Leasing and Transportation. On the technology front we specialize in BusinessAnalytics, Enterprise Applications, Application Modernization/Management and Independent Testing. Founded in1990, we are a US$ 153 Million company with over 4000 employees in 16 locations worldwide. We are currently rankedas No. 11 Software Company in India. TECHNICAL EXPERTISE Our Key Offerings Technology – Legacy Systems, Client Server, J2EE BR / FS/SR Specification Industry Specific Industry Specific Business Requirement Analysis Gap Analysis Functional Support and .NET. Solutions Solutions Operating Systems – Airlines Software Testing Existing Methodology Review Existing Tools Evaluation Multiple Virtual System Process Consulting Process Recommendations Manufacturing & High (MVS), technology System Integration Black Box - Functional Testing UNIX variants and Window Banking & Securities White Box - Interface Testing Testing (SIT) Grey Box - Regression Testing based. Insurance Performance Load Testing Stress/Volume Testing AUTOMATION TOOLS Technology Practices Technology Practices Testing Analysis and Interpretation EXPERTISE Application Management Test Management Win Runner, Load Runner Enterprise Application Test Automation Functional Automation Load Test Automation and Test Director - Mercury Integration Interactive. People Soft UAT UAT Preparation & Execution Roll Out / Production Support Functional, Performance Oracle User Training Siebel Support and Test Management Tools E-Business solutions Fail Over / Availability Multi-tier Availability - Rational Software. Low Resource Testing Endurance Tests Silk Test and Silk Performer from Segue Software.HITS(HEXAWARE INDEPENDENT TESTING SERVICES) OVERVIEW Team of 250+ with varied skill sets Domain focused, Automation driven E2E testing solutions Independent testing labs based in Chennai and Mumbai Showcases projects spanning a diverse spectrum of Domains – BFS, Insurance, ERP and Airlines Technologies - Mainframe, J2EE, Siebel, People soft and SAP Expertise in test automation tools like Win runner, Quick Test Pro, Load Runner, Test Director, Rational suite. More than 35% have hands on experience in Mercury tools. Proven Test Methodologies from end-user perspective to meet the customers expectations under real-world business scenario. Offshore – Onsite Delivery model providing significant cost savings More than 1000 Man years experience Hexaware Technologies Hexaware Towers, Bldg. No. 152, Millennium Business Park, 51/3, G.N Chetty Road, TTC Industrial Area, TN S 3 "A" Bl k M h
  • 23. Cover by Rex Black, President, RBCS, Inc., Bulverde, TX Story Is bad testing eating your Pr fits?Using Quality Risks toGuide Testing EffortTesting should focus on mitigating specific risks to thequality of the system. Sequence of test execution shouldbe driven by associated risk...S ince the 1980s, when Bill Hetzel Categories of Quality Risks and Boris Beizer published their Computer software, hardware, and systems can influential books on software testing, fail in the most amazing and various ways. Some we have known that testing should be bugs are not functional problems, but fall intorisk-based. Testing should focus on mitigating other quality risk categories. We’ll take a look atspecific risks to the quality of the system. The some categories of quality risks to stimulate oursequence of test execution and the total effort thinking about what to test.expended on any given test should be driven bythe level of risk associated with that test. In the Functionality1990s, people like Rick Craig, Paul Gerrard, andI independently created some ways to achieve The most obvious quality risk category is thatsystematic risk-based testing, often by adapting of functionality. There’s always a chance thatideas from other forms of engineering, such as the system does not provide some function itinsurance and medicine. In this article and the should. The function can include a capability,next one, we’ll talk about ways to perform risk feature, processing, input or output.based testing. For example, if you’re testing a calculator21 21
  • 24. Using Quality Risks to Guide Testing Effort 4 Unacceptable performance load above 500 Transaction Processing Time 3 Unacceptable performance at 2 any load Unacceptable performance degeneration above time 1 Acceptable performance Transaction Arrival Rate 1000 Figure 1: Performance bugsprogram, the lack of an add capability would fall performance. The shaded area at the bottom ofinto this category. So would a situation where the figure represents the required performance,the add capability was implemented, but the “+” which is one second or less transactionkey didn’t cause the function to be activated. processing time at a load of up to 1,000The add capability might be implemented and transactions per minute.accessible, but might work only on integers, not Risks to system quality in the area of performancereal numbers. The add operation, when carried include the possibility that the system respondsout, might give the wrong result, as in 2+2=5. to input, produces output, orOr there could be some strange side effect processes data too slowly underwhere the operation is handled but the all levels of load. The straightresult is not at all as expected, as in a dashed line shows this possibility.divide function where 2 divided by 2 The system might perform fine upreturned one, but in Roman numeral to some level of load, but have anformat. unacceptable non-linearity in the performance curve, as shown by thePerformance and Reliability steep curved dotted line. Finally, the systemIn Figure 1, you see a graphical representation might perform within specifications during anof three types of quality risks in the area of initial test run, but subsequent tests—when the22 22
  • 25. by Rex Black, President, RBCS, Inc., Bulverde, TX include seeing functionality, performance, or reliability problems due to the rate of Reliability problems exist computational, data, and communication flows. when the system fails to For example, the performance of database function normally each and management systems often start to degrade when the database management system load every time. exceeds 80% of the rated transactions per minute or the allowed number of simultaneoussystem is not rebooted between tests—might connections.reveal unacceptable performance degradation,as shown in the family of three gently curved Installation and Deinstallationlines. Various things can go wrong when installingThe risk of performance degradation—and programs. These include situations where theindeed, behavioral degradation of any sort— program won’t install or the installation processover time brings us to the reliability category causes damage to the system.of risks to system quality. Reliability problemsexist when the system fails to function normally The latter kind of problem once was ubiquitouseach and every time. Such a reliability bug can on Windows systems, with theexhibit itself as an intermittent functional appropriate nickname “DLL hell.”problem. Sometimes, such bugs Here, one program overwroteappear after a long period of runtime libraries used by another programor after running under heavy load. during its installation process.I had such a problem with Windows Deinstallation can also create2000 while writing this article. Extended problems. Sometimes, deinstallationuptime between reboots usually resulted in does not completely remove files andcharacters in some applications taking on the undo changes. Sometimes, deinstallationwrong font or the wrong font size. causes damage to the system.You might have seen another type of reliabilitybug, too, where the system functions normallywhen it functions, but crashes or hangsunpredictably. Such bugs can occur after some Various things can go wronglong period of runtime or after running under during installing & deinstallingheavy load. Alternatively, such bugs can betruly random. I recently saw such a bug when programs.a security patch caused reboots on a systemevery third or fourth time the system booted.Stress, Capacity, and Volume OperationsRisks to system quality in the area of capacity We also have those risks associated withinclude seeing functionality, performance, or recurring operating activities.reliability problems due to resource depletion. These include activities carried out at the endFor example, the performance of operating of some operational period, such as end-of-systems often starts to degrade once the system quarter or end-of-month closing out of accounts.consumes more than 80% of the available hard Problems can arise especially with failures todrive or memory space. archive inactive logical records in a safe andRisks to system quality in the area of volume recoverable manner, as well as archiving of data that should remain active. In addition, the 23
  • 26. Using Quality Risks to Guide Testing Effort exhibit their effects a long way from their origin. Side-effects of changes can impair or even break The effects of changes, even cohabiting, interacting, or underlying software. small, localized, isolated The adding of yet-another-feature can lead to changes, are not always systemic performance and capacity issues. small, localized, or isolated. Usability and User Interfacetiming of archiving and the calculation of when One special category of non-functional qualitya period has ended and a new period begun can risks relates to the usefulness and usability offail. the system by the intended user or operator. Specific usability and user interface bugsFor systems with databases, there can be regular can vary. In some cases, systems presentrequirements to compact or repair the databases. cumbersome interfaces that do not followIn addition, for such systems, operators normal or expected workflows, leading tooften must back up data and configurations user frustration, confusion, and mistakes. Inregularly, along with verifying the restore other cases, functionality, while present in theprocess. There’s a strong risk of side effects system, is practically inaccessible to the userlike performance problems related to backup orrestore operations, the inaccessibility of someor all system functionality during backup or Data quality poses a majorrestore operations, and the failure of backup or category of quality risks forrestore operations under full system capacity many systems.conditions. Sometimes, operations activitieshappen automatically, in the background. Forexample, rollback logs for a database often or operator. Systems can be inappropriatelyneed to be purged automatically to avoid major difficult for the users to learn, leadingperformance and reliability bugs. to abandonment, mistakes, and inefficiency. Finally, users andRegression operators can find instructional,By regression, I mean when a help, and error messages thatpreviously-working feature, function, are misleading, confusing, orcapability, or behavior fails following a misspelled.change. The problem we have in systemtesting is that software/hardware systems, Data Qualitybeing digital, tend to be discrete rather than Many systems exist primarily to do interestingcontinuous in their failure modes. things with data. Inputs are processed,The effects of changes, even small, localized, transformed, and stored. Databases areisolated changes, are not always small, localized, queried, records linked through foreign keys,or isolated. How many times have you heard a and integrity constraints enforced. Outputs arestatement like, “How could that have happened? displayed, printed, and transmitted to otherI only changed one line of code.” system.A small change, even one line of code, can affect So, data quality poses a major category ofthe behavior of the rest of the system. A small quality risks for many systems. The systemchange can cause incompatible data to enter might corrupt or lose data. The system mightshared data sets, including dynamic dataflows store bad or nonsense data in a databaseand static databases, resulting in bugs that without integrity constraints. Databases shared across multiple systems can allow data which is24
  • 27. by Rex Black, President, RBCS, Inc., Bulverde, TXvalid for one system to be accessed by another Of course, if the system does not support thesystem which does not know how to handle character sets used by the local language, wethat data, resulting in failures that are removed have a localization problem. Those languagesin time and feature space from the source of the that use the Roman alphabet, such as English,problem. German, and Spanish, have single-byte character sets. Some languages that use otherFurther complicating these situations, it is alphabets, such as Russian and Hebrew, havesometimes difficult to restore the system to single-byte character sets, too.working state after a badly-handled errorcondition. In some cases, the system, in the Localization also has operational implications.course of succumbing to the error, damages For example, time zones and time formats varyconfiguration files or static data stores in a way based on locale. Does the time change in thethat’s hard to fix later. The data quality bug in the summer? (In the United States, this is calledexpense reporting package I mentioned earlier “daylight savings times,” but it goes by differentis an example of this kind of misbehavior. names around the world.) If so, when does adjusted time begin and end? AreDate and Time Handling dates written “month-day-year” (asIn addition to having to handle in the United States) or in the morecommon errors, many systems must logical “day-month-year” format (asalso handle dates. This has created in much of the rest of the world)?significant information-technologyproblems, including the infamous Configuration and“Y2K bugs” that consumed huge Compatibilityproportions of company and governmentIT budgets in the late 1990s and contributed to Quality risks related tothe depression in the high-technology sector inthe early 2000s. In addition to having to handle Localization may be inonce-in-a-hundred-lifetimes type of events like the categories of GUI ora new millennium, systems must frequently operations.handle events like leap years.Many systems must deal with expiration dates. Another family of risks to system quality livesLicenses expire. Credit cards expire. Insurance in the areas of configuration and compatibility.policies expire. After some period of time, the Different users configure both the hardwareright of a bank to disqualify a borrower based on and software of systems differently. A familya derogatory like a bankruptcy can also expire. of systems may support various hardwareFailure to handle such expiry events properly configurations. Often, the number of potentialcan expose a company to serious financial and combinations of configurations is huge andlegal risks. testing inappropriate combinations can result in risk of failure in the field.LocalizationLocalization refers to the ability of a system to Standards and Regulatorysupport local customs, languages, norms, and Compliancelaws. There are two broad categories of quality More risks to the quality of the system existrisks related to localization. One category in the areas of standards and regulations. Arelates to the user interface, and the other to system can work properly but be excluded fromoperations. the market by failing to meet standards, either 25
  • 28. Using Quality Risks to Guide Testing Efforteffectively (no one will buy it) or legally (thegovernment won’t let anyone buy it or sell it). When you haveSecurity communicating components,With the growth of the Internet, awareness of timing and coordinationthe risks associated with security has increased.Security risks are legion. They include viruses between them is a concern.and worms, criminals breaking into servers,vandals causing denial of service attacks, Can You Think of Other Qualitybugging and intercepting e-mail and Internet Risks?communications, and more. Yes, and so could I. But isn’t this list already too long? What if you tried to cover all theTiming and Coordination quality risks discussed so far? Could you testWhen you have communicating components, or them all? Would they fit into the budget andshared components, timing and coordination schedule context of the project? Would youbetween those components is a concern. For end up wasting time and effort on unimportantexample, with an e-commerce system, how long problems? Surely any attempt to test this wholeshould the system wait between events like set of quality risks would be neither effectivemouse-clicks and submitted screens? At what nor efficient!point can it safely conclude that the customer So, what shall we do? In the sequelhas abandoned the transaction? to this article, we will discuss someAs another example, consider the ideas of how to trim the infiniteautomated teller machine. What number of things you could testhappens when the central computer into a finite list of risks you shouldtimes out or the network goes down? mitigate.What if that happens in the midst of atransaction? If we’re testing an inventory About the Author:system, what happens when two salespeopletry to sell the same item at the same time?DocumentationFor many people the quality of the documentationsignificantly affects their experience of qualitywhen using the system. Documentation qualityproblems include being technically incorrect, ofcourse, especially the examples. The user canfind the documentation insulting or offensive,teeming with grammatical or spelling errors, orafflicted with distracting cosmetic or formatting With a quarter-century of software and systemsproblems. engineering experience, Rex Black is president and principal consultant of RBCS, Inc., a leaderDocumentation refers not only to hard copy, but in software, hardware, and systems testing.also to electronic documentation. Help screens,installation instructions, error messages, andwizards are a form of documentation. This article is based on an excerpt from Rex Black’s book, Pragmatic Software Testing, published by Wiley26
  • 29. That Have I got a deal for you? Time of YearHave I gota DEALfor you …?A s this is our first issue, we are in a unique giving something up – time, money, energy, or place—: This issue will set the ground effort, in exchange for something we value more. rules for every issue that follows. Future If you don’t think it’s like that in the world ofissues will build on this one, so we have to lay software testing … think again.the right foundation. I would like to start that Having an independent test group splitsfoundation with a simple assertion: management attention and costs money, but it also decreases risk and provides an impartial The methods and extend of our testing are assessment of the quality of the software. all choices; you could even say they are Using bug-tracking software costs money and creates the risk of “information overload”, but trade-offs. also ensures that every incident is logged and tracked. With a complete listLife is all about trades. We trade of incidents, a change controla “work week” of our lives for a board (CCB) can determinepay check, then we trade that which defects should be fixedpay check in for a home, for food, in the next release, by priority,and utilities or clothing. We trade but that invariably slows downsome of our freedom for a family. bug fixes and decreases the personal touch.At each stage of the game, we are28
  • 30. by Matthew Heusser, Socialtext, California, US Documented test cases are to enter into the equation, and that worries me great, right? We couldn’t a bit. live without documented test Okay, more than a bit. It worries me a lot. It’s cases, could we? Well, maybe. downright irresponsible to suggest that there is Time spent documenting testcases is time that could be spent testing. Thereis also some evidence to suggest that following Not all trade-offs are equal;the same test scripts again and again leadsto thinking about problems in the same way, the trick is to carefully weighwhich could cause a miss. Many times, finding the pros and cons beforea bug (or a repeatable error condition) is like making a decision....playing a game of twenty questions; how likelyare those scripted test cases to get the rightanswer if they lack feedback? one single way to do it – a single one-dimensionalRequirements-driven testing requires written scale for goodness. The real suggestion that therequirements, and “locks in” the feature set at “Best Practice” crowd is that you listen to them,the beginning of the cycle, when we have the doing things the way they suggest, instead ofleast information about what the users actually thinking for yourself. The trade-off is your brainneed and how the software will support them. for their brain. Keep in mind, the guru never setReviews and Inspections of documented test foot in your office and doesn’t even know whatcases slow down the test process, again taking product you make.time away from test execution. Exploratory When pressed on this issue,testing requires talent and practice; exploratory most gurus admit that, oftesting in the hands of someone without course, “best” doesn’t reallyexperience could just be a waste of time. mean “best.” To them, “BestHow can your team get “better?” Mentoring, Practice” is just a short way ofCoaching, and Skill Development are all very saying “A practice that is goodhard to measure and hard to evaluate. Test enough for enough people enough of the timeProcess Improvement is easier to measure, but that it’s probably worth considering.” If thatalso of more questionable value. were really the case, then there are a number of alternatives; James Bach of Satisfice recentlyNot all trade-offs are equal; the trick is to listed several on his blog, which you can find atcarefully weigh the pros and cons before making decision – to consciously realize what you aretrading away, what you are trading for, and if • “Here’s a practice I find interesting”the switch is worth it. The question is, what • “Here’s what I would recommend for thisdo you value more? Which outcome is “better” situation”for your company, with your customers, sellingyour product, at this point in time? • “Here’s my favorite practice for dealing with {x}”I’m harping on this point for a reason. Notbecause it’s so blatantlyzing obvious (it is), • “Person {x} attributes Practice {Y} for hisbut because so few people are talking about success. Maybe you’d like to learn aboutit. Instead of talking about trade-offs, they it”are talking about “Best Practices”, which, Notice that all of those examples includedapparently, always solve all of your problems conditionals, and that best practices are free ofwithout introducing any new problems. The all conditionals. When someone consistentlyfact that this defies common sense doesn’t seem 29
  • 31. Have I got a deal for you? 7) Only through judgment and skill, exercised At each stage of the game, cooperatively throughout the entire project,we are giving something up... are we able to do the right things at the right times to effectively test our products in exchange for something - we value more. The greek philosopher Socrates is credited withuses the phrase “best practices” and never saying that the unexamined life is not worthseems to use any conditionals, my radar goes living. We spend thousands of hours a yearup … then I go to protect my wallet. at work. Examining our work life and strivingSome gurus admit that they don’t know your to constantly improve would seem a naturalsituation. They explain when a practice might extension of that, and it is what this magazinefit and when it might not; they talk about the is all about.pros and cons of the practice. Some experts and Of course, the choice is yours. Each issue willconsultants admit that all new practices involve cost a little bit of your money, and a little bita trade-off, and they are up-front about it. more of your time. Would you like to trade?At WhatIsTesting, we want to follow that example.Every issue of WhatIsTesting will contain someideas, strategies, techniques, practices, andperhaps some philosophy. The ideas might fitinto your workplace, they might not, or perhaps About the Author:you can use them later, but not now.As I mentioned at the beginning of this essay,this issue will set the ground rules for everythingthat follows, so I want to start it out right. Iconsider myself a member of the context-drivenschool of testing, that has the following basicprinciples:1) The value of any practice depends on its context.2) There are good practices in context, but there are no best practices.3) People, working together, are the most important part of any project’s context. Matt Heusser is a Member of the Technical Staff at Socialtext, where he is a lead in the QA4) Projects unfold over time in ways that are group. often not predictable. You can read Matt by email at Matt.Heusser@5) The product is a solution. If the problem isn’t <> solved, the product doesn’t work. or read his blog online at xndev.blogspot.com6) Good software testing is a challenging <> intellectual process.30
  • 32. Learn by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board How To TestWhat aTesterShouldKnow, evenAfter MidnightTesting The Normal Way is Not literature are widely practiced.Enough. As test execution is generally one of the lastSystematic testing of software or systems can activities in a project, it is very often run underbe learned, just like any engineering discipline. severe budget and time pressure. This meansThere are tester knowledge certification that the testers have a hidden or open motivationschemes like the ISTQB certification, which is to compromise on the thoroughness of theiran internationally recognized, independent and job. For example, as detecting defects alwaysnon-profit organization, and there delays both testing and the deliveryare standards (ISTQB Glossary, or acceptance of the product, thereBS 7925, IEEE 829, 1008, 1012). is a hidden pressure not to findAt least the books and most of the any defects, or at least not seriousstandards have been around for a ones. Testing is just “done”. Thislong time, and many techniques are job is then not very motivating,widely accepted. This means testing investment in learning is scarce,can actually be studied and then people try to find a different job,executed in some systematic way. and testing does not improve overThis does not mean that the typical time.testing methods described in the 33
  • 33. What a Tester Should Know, even After MidnightCan we Do Testing in a Better problems and defects, a mindset that questionsWay? everything is likely to result in better testing.Glenford Myers, in 1979, defined testing Part of the tester’s task is to report incidents.differently: The aim of Testing is to find defects. This is not easy because it requires describingHe used this definition because it motivates the problem in such a way that the other sidepeople. Having a negative focus, trying to break accepts it as important enough to do somethingthe product, leads to finding more and more about it. It means not only collecting moreserious defects than just checking whether the information about the problem but also toproduct “works”. think about how to “sell” the bug report to the responsible person.Most people do as they are told. If they are toldto find as many defects as possible, they will try There is definitely more to it. A tester should always be on the outlook for more challenges in field of testing. This paper is only the beginning Test in a better way: of what you may look for. - Clearly define the purpose of testing. The Purpose of Testing - Continuously try to learn There are a lot of possible goals for testing. One main, but possibly boring, purpose is to more. measure the quality of a product. Testing is - A mindset that questions then considered just a usual measuring device. Just usual measuring is not much fun, but everything is likely to result in a necessary job, which must be done well. better testing. However there are questions a tester should - Report the bug with a view to ask, in order to measure optimally. The main question is which quality characteristics are “sell” it to responsible person. most important to know, and how precisely the measurement must be do so. If they are told to get the job done (and Another definition of testing is trying to createexplicitly or inherently passing the message confidence in the product. Confidence, however,that defects delay the progress), people will try is best built by trying to destroy it (and notnot to find defects, or they will overlook many. succeeding in doing so). It is like scientific work: Someone proposes a new theory, and all theThus the first problem is to clearly define the educated specialists in the area try all they canpurpose of testing, and make the purpose to show that it is wrong.perfectly clear to the testers.Second problem relates to change. Softwareproducts are growing more and more complex. Defects clump together: theyThe focus of the requirements is changing, are social!for example emphasizing more security, Defects often have a commoninteroperability and usability. This leads to cause!changes in the requirements on the testing job.Thus, a tester should continuously try to learn Try to find common causes,more. and then follow them!The next problem is the mindset of testers. Where you find defects, digAs one of the purposes of testing is finding deeper!34
  • 34. by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications BoardAfter trying this for some time, unsuccessfully, input and misuse.the new theory is slowly adopted. This view There are three kinds of tests: The good, the badsupports Myers’ definition of software testing: and the ugly tests: Good means everything isFind defects! The approach is a pessimist fine, all inputs are right. Bad means inputs areapproach. The pessimist believes «this probably wrong. The ugly tests is where all wrong thingsdoes not work» and tries to show it. Every defect happen simultaneously: Someone restarts thefound is then a success. server while you do your reservation and at thePeople function by motivation. The definition same time sets the machine date wrong, …of testing as actively searching for bugs is Another way to look at risk is to asses the possiblemotivating, and people find more bugs this way. damage of the failure. This may be difficult toIt works in two ways: One is by designing more analyze but one can make a beginning by askingdestructive or just more test cases. The other “what is the worst thing that can happen if thisone is by having a closer look at the results, function, feature or request fails?”analyzing details a not-so-motivated testerwould overlook. In the latter case this may Better tests forces developers to do better work,mean to analyze results that are not directly inform management about risks, and lead tovisible at the screen, but are deep inside some lower cost of software creation. Testers bring bad News, but that is their job. Nobody loves speed checks on the motorway! But speed checks A tester should try to find make our roads safer, and we all benefit. defects! Continuous Learning Defects may appear at places Continuous learning is required in nearly any where you do not see them job. But for testers it is absolutely essential. easily, i.e. not on screen output! Testing is risk mitigation.files, databases, buffers or at other places in What determines risk?the network. What happens if some input isDefects tend to cluster together. We should take wrong?a deeper look in areas where we find defects.The trouble is that this could be a self fulfillingprophecy. We are likely to find more defectsbecause of our focus in that are. In most cases, testing is done somehowOne more definition of testing is «measuring systematically, using some black box approach.risk». This means that testing is a kind of risk In addition, test design may follow heuristics.mitigation, part of risk management. Testers Any black box approach may leave importantshould have an idea about product risk, as well areas un-covered. Any heuristic is incomplete,as risk management. as it is dependent on personal experience (or onA very basic method for this is looking at the learnt experience from others. And white boxusage profile, and at possible damage. A tester testing does not uncover errors of omission. Itshould ask which kind of user would do what all comes down to: If there is some aspect theand how often. How will different users use the tester doesn’t know, it is not tested. Hence thesystem? How will this vary over time? And it tester should know as much as possible. Butis very important to take into account wrong how? 35
  • 35. What a Tester Should Know, even After MidnightA tester needs programming experience. There your colleagues, and go to testing conferences!are lots of programming bugs, even after unittesting by programmers. The tester should A Critical Mindsethave an idea of problems posed by a particular Don’t believe anything!programming language. Most testers who havetested software written in C or C++ know that As a tester, don’t assume anything. Itthey should check for memory leaks or memory may be wrong! Designer, specifiers, andoverwriting. Those who used early versions of programmers assume a lot. It may beMFC (Microsoft Foundation Classes) know that difficult to ask because you may lookthe program would crash for any date greater stupid asking. Someone who could answerthan a particular date “We need to check may be far away or not easily available.what that date was and with which version You wouldn’t even realize that there may beof MFC” another interpretation. Sometimes people you ask also don’t know, or you get someThe tester needs design experience. Much design sarcastic about contracts and communication amongstvarious modules. If the tester has no idea about Using the pessimist view, you may as well assumearchitectures and their inherent problems, she that any assumption is probably wrong.will have trouble planning integration tests. There are ways to overcome the trouble that youThe tester needs domain experience. System may seem stupid. Learn how to deal with people,testing is about implementing the right solution, learn how to interview, learn how to be self-doing what the domain requires. Can you test confident. Ask someone else. Read, review, sleepa railway interlocking system? (Eh, what does over it, and try to find different interpretations.interlocking mean, by the way...?). At least some You may need a lawyer’s mindset.domain experience is a must to test the system If you don’t get an answer, have it on your issuesand communicate with different stakeholders.The trouble is that this may not be enough. Don’t assume! Ask!Testing for today’s stakeholders may definitely If nobody else asks the rightbe not enough. There are totally new ways a question, you might do so!system may be used or interfaced with othersystems and a tester should try to anticipate at Think about new possibilities,least part of this! A tester should always try to unknown problems, and thefind new ways to look at the object under test, stuff you approaches, and new viewpoints. Think «out of the box»!And finally: testers should try to use the newestapproaches and technology. You have to learn list. But don’t just assume anything! Don’t takethem. Read testing books, look for and learn things for granted! And especially don’t believetools, study journals, participate in discussion that the system is probably rightgroups, special interest groups, discuss with Defects Learn more, about everything! Nobody loves the messenger with bad News! Programming, architecture, As a tester, most of what you report is bad news. new domains, users, tools, The bad News is the bugs, or issues to call them anything! in a neutral way. Textbooks generally handle this area well. There is issue reporting, registration,36
  • 36. by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Boardhandling, removal, retesting, regression testing. The second problem is worse: Sometimes weWe know all this. But there is something extra experience failures, and we cannot recreateto it, and that is not there in the books: them. These issues are called intermittent bugs. They are especially difficult if they 1 - An issue is only interesting for a tester if introduce system crashes. Upon restarting the it is accepted as a defect and repaired. system, any corrupted data in the memory may 2 - There are defects, which are the result of be deleted, destroying the evidence. In many running many test cases in a row in some cases, intermittent bugs are the result of long- very special order. term corruption of some resource or memory.The first problem is one of salesmanship and An example is memory leaks. Some function indiscipline. As a tester, one has a sales job. the program does not return unused memoryNobody is interested in spending any money on when finishing. But because there is a lot ofrepairing defects. They will only be repaired if available memory, this can go on for a long time,they are important enough. Thus, as a tester you until the memory is depleted. Other resourceshave to report an issue in such a way that the may also be depleted. As an example, the Marsdeveloper understands that it must be repaired. Explorer ceased working after 18 days due toThe damage must look big, the probability of it too many files accumulated. In many real timeoccurring must look big, and the issue must be embedded systems, the tasks are restarted ateasy to repeat. certain intervals, in order to cancel out possible corruption of resources.Thus the tester should not just write an issueon the issues list. The tester should think: Are The trouble is that ANY shared resource canthere any other related scenarios, which would be corrupted. It comes down to checking themake this problem worse? Are there any more not very easily visible outputs such as files,stakeholders interested in this? Is this really memory pointers, buffers, databases, messagesthe whole problem or is there more to it? It may to remote systems, registry and many otheralso mean to invent and run some more test resources. It could even be the result of a racecases. Cem Kaner has presented some excellent condition, which depends on the exact timingmaterial on this cause which is available at of some parallel tasks. And intermittent bugshis website at normally require a whole sequence of test cases,BugAdvocacy.pdf not just one input and output.Finally, there are the human issues, about However, if intermittent bugs occur, it is a gooddiplomacy, politeness etc. A tester should idea to be able to rerun the same sequence ofmake sure not to hurt anyone personally when test cases, maybe even with the same timing,reporting an issue. and do more checking than before. James Bach has a good guide to investigating such problems available at james/archives/000197.html. For every issue (or bug), One final problem: You may be wrong yourself. research more about it! Humans err. Testers are humans. This means Make sure you report it as a you overlook problems; you misunderstand risk, and as the whole risk! outputs, and some of the problems you think you have found are actually not problems. Defect reporting is a sales job! Be self-critical: Sometimes it is your fault. Mistrust your memory. It is restricted. This Be diplomatic when reporting means it is better to take notes, to log issues! what you did, what the system did, what 37
  • 37. What a Tester Should Know, even After Midnight test will be used. If nothing can be said, the Analyze even intermittent test should be directed at any use, testing problems! robustness. This means especially test cases Log everything you do in for wrong inputs are interesting. testing! One technique is the basis of most black box testing: Equivalence partitioning. It helps to save Log everything you see, and look at more remote details! Make it possible to rerun your A tester must be able to state tests, with more logging and what coverage a test has analysis tools switched on! achieved! is installed etc. You cantrust notes more than your memory. test effort, and it can be applied to derive otherThe Late Night Tester’s Toolbox test techniques. As a tester, you should know it,How should a tester work? What should a tester but also be aware that it has caveats: Black boxalways keep in mind when working? testing may leave out important aspects. YouOne main trouble is to test “everything”. This is should also be aware that combination testingfar too much. It can never be achieved. But the is interesting. Lee Copeland (Copeland 2004)tester should have some ideas about what is has published a good introduction.tested and what not, or what is tested more or Another big problem is test environment thatless. That means test coverage. has not been validated. The test environmentThere are three very basic concepts of coverage, should be prepared and tested early. Waiting forand they can be applied to any notation for the test environment to work can kill any testing Follow the usage profile if possible! You may be wrong – don’t trust yourself 100%! If not possible, test for robustness against wrong Take notes of what you do! input.example, a control flow diagram, a data flow effort. Also remember that the defect may notdiagram, a state transition diagram, a call be in the object under test, but in the test datagraph, system architecture or a use case. or the output analysis. Be self-critical! • Basic coverage is executing every box. And finally, there is test automation. Running tests using automated tools helps regression • The next level is testing every connection. testing. But test automation is more than that: • This should be the minimum when testing. If there is more time, the next level is combining things, for example pairs of connections. Equivalence partitioning is a good basic technique!Next, a test should follow the usage profile. Thisis difficult, especially in module and subsystem Remember combination testing!testing. But as a tester, one should at least tryto get some idea about how the object under38
  • 38. by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications BoardTools may read specifications and automatically • ISEB: Information Systems Examinationsgenerate test cases. Tools may automatically Board of British Computer Society.create or restore test environments. Tools may used to manage the testing effort and the Qualifications/ISEB/ has run a certificationtest material. Remember that automaton can scheme for software testers since applied to any part of the software testing • ISTQB: International Softwarelife cycle and not just test execution. Testing Qualifications Board. Develops and runs an international software tester certification scheme. Test the test environment – well before test execution! • ISTQB Glossary: media/glossary-current.pdf Check you test data! • Kaner 99: C. Kaner, J. Falk, H. Q. Nguyen, “Testing Computer Software (3rd ed.), John Wiley, 1999.Selected References • Kaner bugadvoc: A presentation about how a tester should report issues. http://www. • Bach 2005: James Bach. A blog note about possible causes of intermittent bugs: http:// • Myers 79: Glenford Myers: The Art of Software Testing, John Wiley, 1979. • Beizer 95: Boris Beizer, Black Box Testing, John Wiley, 1995 • Schaefer 2004; Hans Schaefer, “What Software People should have known about • Better Software Magazine, www. Software Testing 10 years ago - What they definitely should know today. Why they still Very practical! don’t know it and don’t use it”, EuroSTAR • BS7925: British Standard: www. 2004• Copeland 2004: Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House, 2004. • Crispin: Lisa Crispin, Tip House, Testing Automate testing tasks! Extreme Programming, Addison-Wesley, 2002, also Be aware that there is more crispin/XPTesterBOR.htm to automation than using test robots! • Gilb 2003: “Testers Rights: What Test should demand from others, and why?”., Keynote at EuroSTAR 2003 • GTB: German Testing Board: www.german- The German Testing Board developed an earlier version of the nowadays-existing ISTQB certification. • IEEE Standards: See 39
  • 39. Take Control Of Your Quality Virtusa uality Assurance Services Enterprises know well that quality can make or break customer loyalty and brand reputation. However, the growing complexity, constantly changing market needs, and time to market pressures make software quality an elusive challenge. Virtusa QA lifecycle services allow you to take control of your QA processes, projects, people, and priorities, while strengthening your QA capabilities, minimizing risk and reducing cost. QA Assessment QA Offerings - QA Outsourcing Evaluation - Tools Evaluation - Quick Wins Workshops QA Ass ess men t QA Process optimization - Process Improvement Framework QA Pro ces s - Health Check Assessment & Scorecard Opt imiz atio n Testing & Test Automation Tes tin g & - Functional Tes t Au tom ati on Automated Testing Manual Testing - Performance - Certification QA - Internationalization Ou tso ur cin g - Usability QA Outsourcing - SQA Outsourcing and Management - Test Team Establishment/Augmentation - ‘Quick Win’ Test Project Consultancy Mission-Critical Software Product QA Heritage Leverage Virtusa’s eight years of experience in Offering Various domain & State Of The Art SQA Labs Tools Expertise testing and certifying hundreds of commercial Shipping, 5% products and mission-critical solutions to rapidly Retail, 5% Financial, 30% Publishing, 15% improve QA effcienceis. Reduce cost and time to market by leveraging Virtusa’s global delivery Health model, world-class QA professionals, and state- care, 15% Telecom, 15% Insurance, 15% of-the-art labs, tools and technologies. About Virtusa Virtusa is a global provider of custom IT solutions and outsourcing services to Global 2000 enterprises and software vendors in the financial services, insurance, media and communications, high technology and other industries. Our deep domain strength, custom solutions and services, and a unique platforming methodology enable our clients to reduce costs, improve business performance and accelerate revenue generation. We deliver these results by integrating a consultative approach with our global delivery model. 2000 West Park Drive, Westborough, MA 01581 Phone: 508 389 7300 Fax: 508 366 9901 © 2006 Virtusa Corporation. All rights reserved. Contact:
  • 40. Learn by Erik van Veenendaal, Improve Quality Services BV : The Netherlands How To TestFinallyUsabilityTesting?will it finally happen?A new testing magazine that has chosen software code that is being developed world- web testing as its central theme for wide is directly or indirectly related to the user- the first issue! I assume that this also interface, we still only dedicate a minimum means attention is provided to one of percentage of our test effort to usabilitythe most critical success factors for websites: testing. Yet, there are many so-called bestusability and thus usability testing. In fact practices available. Last week I was involvedusability has been identified in various surveys in a usability test that very clearly showed thatas one of the main reasons for project success thorough attention for usability throughout thein any IT project! For a number of years I’ve development process (requirements, standards,been lecturing on this topic at test conferences. expert reviews, usability testing) can deliverAt EuroSTAR, I have received the best tutorial great results. The users were enthusiasticaward for Discount Usability Testing and regarding the new system that supported theirour company, Improve Quality Services, tasks in an efficient, effective, and user-friendlyregularly runs courses on Usability Testing. manner. This is also possible in real-life IT-Yet, I don’t have the impression that all this projects!pioneering work has had much impact on the As far as I’m concerned every (senior) tester thattesting community. During the last EuroSTAR operates at system level should, in addition to theconference, an usability expert from South conventional functional test design techniquesAfrica stated that “usability (testing) will only (boundary value analysis, equivalencestart to be performed and receive the right level partitioning, decision tables etc.), also have theof attention, when testing for level of knowledge and skillsfunctionality is under control”. that allows them to apply aAn interesting thought, which number of relatively easy toas far as I’m concerned, is very use usability test techniques.much true. Perhaps we were The first things that comejust not ready for it, but are to my mind in this contextwe ready now…? are the Heuristic EvaluationIt remains strange that when technique developed byno less than 40% of the Jacob Nielsen (www.useit. 41
  • 41. Finally Usability Testing? Will it finally happen?com) and the SUMI questionnaire method (www. During a recent conference, a number Both techniques are relatively enthusiastic former participants of myeasy to learn, do not require more than one or usability courses told me they were now usingtwo days of testing effort and therefore have a the techniques mentioned above in their testgreat cost/benefit ratio. Recently, the British projects with good results. Usability testing,Computer Society (BCS) Special Interest Group will it finally happen?in Software Testing developed a testing standard For queries or comments you can contact Erikin the area of usability testing that is certainly van Veenendaal ( to get acquainted with ( Sufficient possibilities A paper on discount usability testing canexist, therefore, to make a good, low-cost but be downloaded from www.improveqs.nlthorough start with usability testing; preferablyin co-operation with developers. Prevention isalways better than cure.42
  • 42. Canarys Automation Pvt. Ltd, # 135, 7th Main, 4th Block Jayanagar, Bangalore – 560011 Ph: +91-80-26539915, +91-80-26642922; info@ecanarys.comCanarys is glad to inform that we are Partners to Radview Software, providing Sales & SupportServices to customers in India. Please find a brief on the Black Box Testing Solutions offered byRadview:Only RadView provides software-testing tools that feature JavaScript test-agenda creation forperformance testing, load testing, and functional testing of website applications. Throughout theweb application development lifecycle, the seamless integration between RadViews performancetesting / load testing products (WebLOAD and WebRM) and functional testing product (WebFT)provides you with the necessary tools to make certain your web application is scalable androbust.Award-winning WebLOAD can help companies deploy high performing e-business applicationsby modeling and anticipating the real-life demands on their applications. By accuratelysimulating Internet user behavior and predicting capacity requirements, WebLOAD reportsbottlenecks, constraints, and weak links within an application – before they cost you downtime,lost sales, or more importantly, lost customers.WebFT is the industrys first Web-centric testing solution that efficiently verifies the functionalaccuracy of Web applications and ensures applications perform as expected.Speed implementation time — use of open standards scripting language reduces the amount ofstaff training requiredImprove quality — resulting from higher quality Web applicationsReduce test script development costs — leverage single transferable scripts for fast, easy reuseof test scenarios between development teamsSpeed Web application deployment — streamline the testing process through the use of singletransferable scripts.For more information please contact: Canarys Automation Pvt. Ltd, # 135, 7th Main, 4th Block Jayanagar, Bangalore – 560011 Ph: +91-80-26539915, +91-80-26642922
  • 43. Some Things I’ve Learned in Software Testing Some Things I’ve Learned in Software TestingS everal years ago, when I started out as that any error messages that appeared were a software tester, my goals for learning due to mistakes I made using the software. software testing were clear. I needed Then, a senior tester encouraged me to follow to learn how a software company my instincts: he told me that if somethingoperated, learn about the software I was bothered me, I should log it as a potentialassigned to test, and learn how to test software. problem instead of letting it go by. As I tested,After a while, I enjoyed finding and reporting I learned that when I had a nagging feelingdifficult-to-reproduce bugs; I felt pleased that that something was wrong, there usually waswe were finding and fixing issues that could a problem in the software. Sometimes it wasnegatively affect customers. a usability issue, or an awkward spot in the application that needed more investigation, butHow did I go from a nervous student to becoming whatever the issue, it was almost always wortha confident tester? At first, I read everything taking the time to investigate further. EvenI could about testing, the product we were now, it is tempting not to listen to that voiceworking on, and the systems we used. When inside that tells me something is wrong withI couldn’t find anything to read, the software, but every time I don’t payI asked coworkers to teach me attention, I miss at least one bug that Ithings. I also learned a lot through regret later. Occasionally, my instinct ismentoring from more senior wrong, but more often than not, thatdevelopers and testers. sense of feeling uncomfortable withIn the end, I learned the most by the software helps me uncoverdoing. (I still do.) When I first started important information.out in testing, my view of what a bug looked I also learn a lot from mylike was quite narrow. Like many users, I felt44
  • 44. by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canadamistakes, especially when I try something, fail I also read statements like: “Quality Assurance isand try something else. Do you learn from your much more than software testing.” In this line ofmistakes and from trying new techniques? Who thinking, it is common to suggest that QA shouldcan you talk to in the organization that can help be responsible for the development process, thatyou learn more about testing? quality is only as good as the process it follows. However, in my testing experience working forThere is More Than One Way to small companies whose survival depended onTest Software satisfying paying customers, the process took aIt didn’t take long for my experience to show back seat to the product. We strove for productme that much of popular software testing excellence over process excellence. If a processinformation is really folklore. It was interesting helped us deliver a product that met businessto read, but I soon discovered that when I tried to goals, we followed it. If a process got in theapply popular testing wisdom (frequently called way of those business goals, we scrapped it.“Quality Assurance”) on real projects, Everything we did needed to be aligned withmy efforts failed. One practice involved the goals of the business if we were to survive.writing all the test cases first (pre- This key lesson shaped my thinking asscripting test cases) based on the a software tester.project documentation, and then My desire to learn more about softwaretesting using these procedural test testing grew because I learned the mostscripts later. Trying this practice and provided the best business resultscaused me a lot of problems. I from actually using and testing software.found that testers would run out of time I wanted to find more techniquesto actually test because they were so compatible with what I was learning aboutbusy writing about what they were software testing. So I reread the first book I’dgoing to test. As things changed over the life ever read on software testing: Testing Computerof the project, we got bogged down maintaining Software by Cem Kaner, Jack Falk and Hungdocumentation. And, when we started testing, Nguyen. Again it resonated with me, and I beganthese pre-scripted test cases really narrowed to read everything I could find that was writtenour focus, often making us miss entire classes by Cem Kaner. Through him, I discoveredof tests. James Bach and later Brian Marick. That wasIn the software start-up I was working for, it when I found my niche. These were people whowas important for each member of the team thought about testing the way I did. My ownto provide value to the goals of the business. experiences and ideas were echoed in theirA development manager then complained to work. Their writing was true to the point andme that we spent so much time developing backed by experience, but most importantly,and maintaining test cases that we didn’t do when I tried what they recommended, it worked.nearly enough actual testing. Once I took that These three testing thinkers, along with Bretto heart, the business was pleased with the Pettichord, went on to found the Context-Drivenresults I provided by using exploratory testing Testing focus on what was important to the business The Context-Driven Testing School has identifiedand to the customer. Previously, I had focused different schools of thought in software testing.on writing lots of documentation (test plans Bret Pettichord has outlined four of them in hisand test cases) before testing. At first it felt a bit Four Schools of Software Testing2 presentation:wrong, like we weren’t following “the rules,” butthe development team and our customers were • Analytical Schoolhappier with our results. That carried more • Factory Schoolweight with me. 45
  • 45. Some Things I’ve Learned in Software Testing • Quality School testing thought3. Bret Pettichord identifies this as the “Agile” school of thought. There may be • Context-Driven School other schools as well. What is important as aThe Analytical School draws heavily on tester is to understand there are different waysmathematics, uses a lot of tools and techniques of thinking about and communicating testingfor code coverage and uses metrics to try to ideas, and recognize which ones you identifymeasure conformance to a specification. This with. That will help you get a focus on whatschool proposes that software reliability can to learn, and will help you understand wherebe calculated. Some testers in this school others are coming from.use complex mathematical models for quality Some people find the identification of schoolsprediction. of thought in testing objectionable, and thisThe Factory School generally expresses that that work by the contextualists is somewhattesting needs to be predictable, repeatable controversial. When I first discovered it Iand software is validated from written was reminded of different schools ofrequirements. This school of thought in other disciplines that Ithought is often interpreted so studied in university. For example,that testing becomes a clerical in philosophy classes, whentask. (Note: many Agile and Lean I learned of another school oftesting ideas are variations of thought in logic, I wanted to learn asthis school of thought. This much as I could to see if it alteredshouldn’t be surprising. Many of my thinking. If I found ideas inthe ideas in this movement come a particular school of thought thatstraight from manufacturing where resonated with me, I added them to myautomation, repeatable processes and own repertoire. The schools of thought ineliminating waste are important factors.) testing idea shouldn’t be something that bothers us as testers; instead, it can be usedThe Quality School encourages conformance as a model for learning. If I get stuck studyingto a particular process. Often, the tester’s the same old techniques and processes, I canjob is to police the process, and make sure that use this model to identify other schools ofeveryone is following it. The belief is that if we thought that I am less familiar with, and beginfollow the process properly, our product quality researching them.goals will succeed as well. As a tester, what works for you, in a givenThe Context-Driven School believes testing is a situation? Have some techniques worked wellskilled activity, and the usefulness of a practice on some projects, but not so well on others? Isis dependant on the context it is used in. Not all there a particular school of thought that youcompanies developing software are the same, haven’t studied before that might help you onand it is irresponsible to try to standardize what you are testing right now?processes for all software development. Differentcompanies have very different needs and those Software Testing isneeds are far too diverse to standardize to one Multidisciplinarytrue way of testing. Testers in this school usewhatever testing tools or techniques are the I once worked on an article on tester rolesmost effective to help them reach their testing with my colleague Michael Kelly, who is alsogoals. a software tester. It became clear to us how many different disciplines software testersSince Four Schools of Software Testing was draw from. Different testers described thepublished, Cem Kaner has identified Test- following disciplines as influencing their work:Driven Development as another school of46
  • 46. by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canadageneral systems thinking, history, software that influence me, and provide enormousdevelopment, construction, law, psychology, possibilities for learning. My background infamily therapy, adult education theory, writing, music and sports, and my education alsowriting and performing music, experimental influences the way I approach learning andscience, among others. Older, more established testing. What are your influences in life, anddisciplines have often already dealt the same how much do you draw on these other influenceskinds of problems we are trying to deal with when testing software? What other disciplinesin software development. Applying successful remind you of your work in software testing?principles from other disciplines can be arewarding source of learning. Testers Can Learn From OtherI often explain software testing ideas by Rolesusing an intersection of ideas from different I’ve worked in roles other than software tester,fields. Three that I think of frequently are and each of these taught me somethingphilosophy, business and technology. Testing, important about software philosophy, involves asking questions. Not Working as a programmer, I learned how toonly do I question the assumptions made in the look at applications differently. Understandingproduct I’m testing, in our development efforts, the application from a source code perspectiveand in the process we adhere to, but I constantly helped me develop a catalogue of potential areaschallenge my own assumptions about testing. in a particular programming language that areStudying philosophy and practicing critical more susceptible to bugs. It also provides athinking skills especially help me when I’m better understanding of where to find usefulgenerating test ideas. information for reporting a bug. For example,Business considerations help me focus the work web applications usually have a server logI’m doing. Am I adding value on a team? Is that may have debugging information usefulwhat I’m doing helping or hindering the for reporting problems. As a testergoals of the organization? Is my testing wearing a developer hat, I was thefocusing on what is important to the recipient of bug reports, rathergoals of the company, or am I blindly than a generator. It helped mefollowing a process? Understanding understand how a poor bug reporthow businesses work, how they can frustrate themselves, and how they stay In a technical writing role, I learned aboutfunctioning is important to testing. design and how design (UI or architectural)Technology and testing are related because we can affect the quality of a product. I noticed ause tools, and interact with designers, developers correlation between complicated designs andand users in order to test. We rely on computer the amount of bugs found. I learned that if Itechnology not only in software development, had trouble writing user documentation for abut in most businesses situations. As testers, particular feature, it would likely be a sourcewe must have a certain level of skill to be able of bugs. Writing down requirements and designto use technology. Since technology moves so ideas from the point of view of a user exposedquickly, it is difficult to keep up and it seems there are infinite technological areas to Working in sales taught me about the bigexplore as testers. We might specialize in testing picture, about how a software tester can servea particular technology, or we may depend on the business goals of an organization. A companycertain tools for our testing. does not necessarily survive because they arePhilosophy, business and technology are areas such-and-such CMM level, or follow the 12 practices of Extreme Programming, or because 47
  • 47. Some Things I’ve Learned in Software Testingthey use .NET or Java, or other technical or Practicing Software Testing isprocedural issues we find so important in Importantdevelopment. When I worked in sales, the harsh We can also learn by practicing softwarereality was that a company survives by fulfilling testing skills. For example, I regularly practicean important equation: my testing skills, and ask my testing friends Revenues – Expenses = Profit to help me develop my skills. One of these friends is James Bach, who taught me how toThis equation must be greater than zero in order be more consistent with my testing approach.for the company to survive. He found that when he gave me a number ofI learned customers cared that the product practice testing problems, my test approachhelped them solve a particular problem. They was inconsistent. James helped me developdidn’t care what technology was used to solve my testing ideas into a consistent system whenit. They cared that the product was easy-to-use, solving different kinds of testing problems. Nowthat it helped them in their work, and that it when I am faced with a testing problem, I havewas affordable. It was interesting to see that a strategy and a game-plan instead of an ad-they would put up with application failures, hoc approach. Practicing testing with someonebut that they were not as tolerant of usability else helped identify an area of testing weaknessissues. When you deal with customers, you get for me to address and improve.a different view of the product. Testers serve I’ve also worked with programmers andcustomers, and it is in our best interests to asked for feedback on how to provide betterbe sure that they are satisfied and impressed information when logging bugs. Early in mywith our product. I learned that it is much career when I was first testing web applications,more important to be testing what is important I wasn’t investigating beyond the error I saw onto the customer rather than trying to fulfill the screen. One of my programmer colleaguessome generally-accepted testing practice. It is taught me how to find more information in logabsolutely paramount to do software testing files. I practiced developing my web testingfrom a user’s perspective. Testers need to investigation skills by:understand the company, what its goals are,and to figure out how testing work maps to • Finding relevant server logs on different webthose goals. application architecturesUnderstanding how an application works from a • Reading error messages in the server logstechnical perspective, and being able to describe • Changing the level of logging to get morethat in a document helps me test more effectively. information on the serverIt helps me ask questions that people I workwith find useful. Working directly with potential • Saving relevant information and talkingand existing customers has helped me focus on about possible causes with administratorstheir needs in my testing. Understanding the and developersgoals of the business helps me prioritize my Because I practiced, when I next tested the webmission in testing on a particular task. application, gathering relevant information fromWhat other jobs have you had other than in server logs to use for a bug report was secondsoftware development and testing? What have nature. It took much less time to do after I hadyou learned on those jobs that you could apply practiced those testing? Are there specific skills you developed This is just one very simple example, but itin other jobs that give you an advantage as a helped my testing effectiveness. There are manytester? skills we can develop, and many techniques we can practice. What specific skills you rely48
  • 48. by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canadaon as a tester? Can you name them? Can you Learning software testing is a rewarding,identify technical skills, investigation and career-long journey. We all learn differently,problem-solving skills and softer skills related and I encourage you to try different ideas andto communication, and working with others? see what works for you. Don’t be afraid to fail when you try something new; we often learnConclusion more from our failures than successes. QualityAt the end of each section of this article, I have guru W. Edwards Deming said: “Drive out fear.”posed questions for you, the reader. If you would Don’t be afraid to try something new, if youlike to learn more about software testing, I urge have an open mind, you are guaranteed to learnyou to answer these questions as best you can. something from it.Take out a pencil and paper, and write downyour answers to the questions. Read them over,and see if there are any new areas for study for About the Authoryou to pursue in testing. You should have some Jonathan Kohl is the founder and principalnew ideas about software testing you’d like to software testing consultant with Kohl Conceptslearn more about. Do you want to learn more Inc., based in Calgary, Alberta, Canada. A notedabout testing techniques? How about different testing thinker, Jonathan is recognized as aschools of thought or definitions on testing? Are leader in the exploratory testing community. Heyou interested in developing your technical or is a popular author and speaker who believesprogramming skills? Would you like to learn to that testing is a challenging intellectual craft.use a test automation tool? How about learning Jonathan’s blog on software development andmore about the end users of your software? testing issues is one of the most well-read testingWith software testing, there are many avenues blogs in the industry. Jonathan is a regularfor learning, and the more you learn, the more contributor to Better Software magazine, as anyou realize how much more there is to learn. author and technical editor. Check out otherI constantly learn more about software testing writings at I hope I have spurred your thinking on howyou can learn more about software testing too.Even though I am no longer a beginner I still dothe following: Notes 1 Kaner, Bach, Pettichord, Marick. The • Read about software testing and other Seven Basic Principles of the Context- disciplines Driven School http://www.context-driven- • Attend and contribute to workshops and conferences on software testing 2 Pettichord, Bret.(2003-2007.) Four Schools • Identify skills and techniques I am of Software Testing using when testing, and explain them to ~wazmo/papers/four_schools.pdf colleagues 3 Kohl, Jonathan. (2005) TDD - A Fifth • Participate in mailing lists, forums and other School of Software Testing http://www. communities of practice related to software testing • Work with others skilled practitioners who constantly challenge me to do better • Practice, practice, practice 49
  • 49. Sierra Atlantic Testing Competency CenterOur Testing CompetencyWith over 12 years of experience, Sierra Atlantic is an established offshoring services companyusing CMM Level 5, ISO 9001:2002, and BS7799 certified process. Using our testing competencycenter SATIN (Sierra Atlantic Testing Intelligence), we create speedy, low cost and high-qualitysoftware. This in turn is delivered through our well-defined engagement models and processmethodologies and incorporates the industry’s best practices.State-of-the-Art FacilitiesOur state-of-the-art Interoperability Lab provides compatibility / interoperability testing servicesto software product companies. The scope of this service includes: Hardware / OS platforms Enterprise Applications Middleware platforms Application Servers Web Servers Databases BrowsersWith over 250 person years of experience in software testing services, our resources have in-depth understanding of various testing methodologies and technologies. The following graphillustrates this: 7% 8% Only manual testing 7% WinRunner 10% Test Director 28% QTP LoadRunner Silk Test Rational Robot 23% 17%The following instances help demonstrate our competency.A leading Enterprise Application product vendorSierra Atlantic’s Interoperability Lab enabled the testing of a leading Enterprise Applicationvendor’s product on platforms such as Sun Solaris, HP-UX and IBM-AIX. This testing allowed thecustomer to remotely log in and execute tests using its staff without having to build its own testenvironments, an expensive and time-consuming process.A leading portal product vendorThis client of ours required its product to be tested and certified across a variety of web serversand platforms. The product also required to be tested for compliance under Rule 508. SierraAtlantic’s Interoperability Lab created multiple functional and compliance test cases and anautomation strategy based on Mercury test automation tools. As a result, the client was able tocut in-house costs, and the testing rendered its product totally compatible and fully interoperablewith other leading enterprise applications and infrastructure products.About Sierra AtlanticFounded in 1993, Sierra Atlantic is a leader in offshoring enterprise applications, helping ourcustomers optimize their investments in Oracle, PeopleSoft, SAP and Siebel. We offer completelifecycle application management solutions -- development, implementation, integration, upgradeand support -- using our NShore™ methodology. We integrate these point solutions intoApplication Networks® that enable seamless business processes within and outside theenterprise.
  • 50. by Mrityunjay :QAis MoreChallengingthanDevelopmentI come from a background in product product functionality and issues first hand, and development and have been in QA for about delivering a complex product to the market. I am a year or so, that too in a senior management very convinced that in a real world, it is easier capacity and not necessarily an engineer. to don a developer hat than a QA hat. In thisIt saddens me when I interview some good QA article, I attempt to show in a very simplisticengineers and they seem almost apologetic way what I mean by this statement.about the fact that they are into testing, and Of course I say real world, because in an idealmore so when they are into manual testing and world, a Dev will be much more like a QA andnot automation. This probably stems from the QA will be much more like a Dev, and they havefact that writing code is considered as the best joint ownership of quality of the product. The(and smartest) aspect of software development company I work in doesn’t claim to be the idealand anyone not doing that is considered less company, but I find it much closer to it thansmart. many other companies I have seen.I started thinking about whether this is really Let me start with an example. We are giventhe case, that writing code is the smartest part a functionality to implement: List all primeof the software development lifecycle and what numbers between two given numbers X and Y.I came out was the reverse; that writing code Now here is what a developer has to deal with:is probably the easier part of the development,it is testing and Quality Assurance (QA) which ● Find/implement an algorithm to check if ais tougher. This is based on my discussions given number is a prime number.with the QA engineers and leads, looking at the ● Implement a logic which cycles through 51
  • 51. QA is More Challenging than Development Test Entry Design Test Entry F I Functional Requirement G Test Entry U Design R Test Entry E Product Requirement - Test Entry 1 Design Test Entry Functional Requirement Test Entry Design Test Entry every number between X and Y, and apply risky since some of the options are being left the algorithm in #1 above. out and hence some issues may still remain. ● Unit test this with some combinations of X A QA engineer who is a white box tester may and Y and make sure output makes sense. look at the code and decide there are 232 options only because the developer used a longA QA engineer who relies on black box testing variable. This is nowhere better than infinity.and hasn’t looked at the implementation will Some choices have to be exercised, probablyknow that there are infinite inputs to this based on decision points in the code. Againimplementation and so he/she will try to pick a risky affair, as QA is likely to miss many,some intelligent inputs which are more likely to including non-function, parts, like dependencyproduce issues (think equivalent partitioning, on CPU and memory, impact of running on sayboundary values, etc). However, this is decidedly52
  • 52. by Mrityunjay :Japanese OS, etc. in the given time and money through various heuristics and techniques, none of which areNote some of the differences in the roles in a scientifically proven to discover all the defects.developer and QA here: This is decidedly tougher job than development, ● A developer decides on an algorithm and since the choices and the implications of choices unambiguously sticks to it, implements are much better known and understood when a and delivers to QA. For a QA, life is full of developer chooses a specific algorithm and its ambiguities and risk-taking. A week-at- implementation. Given the fact that a functional heart QA is not a good QA. requirement is broadly defined, it is easier for a developer to meet those requirements and verify ● A QA has to second-guess Dev to figure they are met, than for a QA to verify developers’ out what are the likely error scenarios if output, since they are very precisely defined he/she wants to avoid running an infinite- (algorithms and code) and hence so many more input problem. This is analogous to 20- ways of proving failure exist. questions game, where you think of a word and someone has to guess it by asking only To summarize my arguments above, there are 20 questions. This is tough. two basic points I have made: ● Think of the implication if a developer ● It is easier to drop a needle in a haysack (or or a QA goofs up. If a Dev goofs up, it is bug in a product) than to find it (discover supposed to be caught by QA, causing bugs, defect before release). The latter requires design changes, etc, a costly affair indeed. you to be much smarter, customer-focused, However, if a QA does so, customers hit with good problem-solving abilities. those defects, and this is a much higher ● QA problem is an undefined problem and cost to the company, in terms of money, hence solving it takes very different skills credibility and market share. Which and lots of smartness in addition to hard role is easier? work.In a complex product, locating a bug is However, don’t get me wrong: theselike searching for a needle in the haysack; arguments take the example of a typicalyou can apply heuristics, but it still is developer role and a typical QA role anda hard and challenging work, scary too, not of individuals who play them. As I havewhen you know that a missed needle can mentioned before, a good developer indeedcause immense customer frustration, dons the QA hat many times, verifies theand management backlash. algorithms he/she has produced and onlyLet’s look at it from another perspective. then allows it to be given to QA. So do notThe waterfall model of product development is use these arguments against any individual;a tree structure. A product requirement spawns I have been a developer, and I would not lovemany functional requirements, each functional to hear these arguments because I thoughtrequirement spawns many design and algorithm I donned QA hats many times to producepieces, which in turn spawn many test cases good quality code. Same goes for many of my(see fig 1). developer friends out there. Thus, even in such a simplified scenario Now that we know doing QA takes a lot of(where next phase only doubles the entities), a smartness and skill, what does it take to beproduct requirement generates 8 test entities a great QA? Well, the trick is to think like afor verification. No wonder then that the QA developer sometimes, and that will be the topichas to play the game of probabilities: trying to for a forthcoming issue.find maximum number of high impact defects 53
  • 53. Is scripted testing bad? 25 20 15 10 5 Is 0 scripted testing bad?W hen I started my testing career the bugs I caught were the result of conditions over a decade ago, my manager that occur in the real world but are impossible gave me two directives: to think about while designing test cases. And there were many bugs that I did not find rather the bugs found me. With these findings my ● Raise thirty bugs every day respect for writing test cases took a beating. ● 80% of your bugs should be coming from Exploratory testing had begun getting talked test cases that you would have written prior about when I had formed my own philosophy to the test execution and technique of mixing test cases and ad- hoc testing and it struck a chord immediately.While I was able to accomplish the task of Exploratory testing described what I was doingraising thirty bugs every day, I have not been all my life. Now I had a name for to achieve the second target even aftertwelve years of trying really hard. To figure out However, there was another trend that I noticedwhat was going wrong I analyzed the bugs, - scripted testing, which I had been doing all mycreated checklists, got my test cases reviewed, life, started getting painted as a bad practice.learnt about various test design techniques and To set the context right, let me describe whatused them. is generally meant by scripted and exploratory testing.Sure, I was able to improve the percentage ofbugs being found but I never could achieve a “Exploratory testing is simultaneousfigure anywhere close to 80%. This made me learning, test design, and test execution”question the assumption that most of the bugs be caught by pre-written test cases. Many of54
  • 54. by Vipul Kocher, CEO, PureTesting, NOIDA, India“Scripted testing means that learning andtest design happens prior to test execution.” Can there be scripted-Before you read on the rest of the article, please exploratory testingpause for a moment and answer honestly if you continuum: meaning thereby,consider scripted testing a bad practice. scripted and exploratoryIf you are careful about the contexts you are testing can be mixed in anylikely to have answered “It depends… scriptedtesting might be useful in some situations” percentage ?and that is the truth. Unfortunately, contextsin which scripted testing is useful do not often testing at one extreme and exploratory testingget elaborated resulting in a message “scripted at the other extreme. Then, there is this idea oftesting is bad.” scripted-exploratory testing continuum which means that scripted and exploratory testingToday scripted testing is taking a beating from can be mixed in any percentageexploratory testing and to take a stance againstit is seen as a sign of enlightenment. The stance Unfortunately there exists no definition formay vary from “scripted testing is bad” to this type of testing. But isn’t this nameless,“amount of scripted testing to be done depends definition-less continuum what most testers,on context” but in all cases the underlying at least the good ones, have been using?message that gets conveyed is “scripted testing The process being ‘create scripted tests, useis bad.” It is unfortunate that there are very few exploratory methods to create more tests, runadvocates of scripted testing who are actively them, learn from them, create more tests andwriting in favor of scripted testing or explaining run purely exploratory tests...’the contexts in which scripted testing is good. To me it would be more natural if somebody provided a definition for this mix of “scripted”Is scripted testing really bad? and “exploratory” testing. Let me call thisBefore we can answer the question “is type of testing as Natural testing. With thisscripted testing bad” we first need to agree definition, I can call myself a “natural tester”on the definitions of scripted testing and also and not necessarily a “scripted” or “exploratory”understand the contexts in which scripted tester because in real life I am neither. This istesting is good or bad. the position most of the testers I interact with, take. They are neither pure exploratory nor pureScripted and Exploratory scripted testers. We start our testing wheneverTesting: definition is important our organizational goals permit us to which can be at the requirements stage or late in the cycleWithout going into definitions of scripted and when development is about to be completed. Weexploratory testing the only questions I would write test cases which help us understand thelike to ask are: system and in the process of writing test cases ● Who has created the definitions of scripted we uncover bugs, raise uncomfortable questions testing? and help improve the product. When we execute tests, we use our pre-written tests, come up ● Who has defined exploratory testing? with more test and scenarios to explore theIf the definition of scripted testing comes from product; we document some of these discoveredthe exploratory camp or vice-versa, the definition tests and discard the rest. We mix testing usingwould naturally portray one as more negative test cases (scripted testing) with guided ad-hocthan the other. The definitions put scripted (exploratory) testing. This is “natural testing” to 55
  • 55. Is scripted testing bad? pre-designed test cases. In exploratory testing test There are numerous advantages of “scripted” testing over “exploratory” testing, some of them design, execution and being: learning happen at the same • The act of writing test cases promotes logical time whereas scripted testing and more complete thinking. uses pre-designed test cases. • Written test cases can be reviewed. This is not a small advantage. The review is likely tous and we fall in neither the exploratory nor the result in better test cases and consequentlyscripted camp. better testing.Painting scripted testing as a • When test cases are written along withbad practice the requirements being written, many requirement bugs can be found and removedSome of the arguments against scripted testing even before any code is written.include: • Test cases can be shared with developers ● Scripted testing does not promote thinking resulting in removal of bugs even before about the problem in different ways. Scripted they can be “created”. test cases mechanize the act of testing, taking intelligence away from testing. • Written test cases can be used as an aid for training new testers. Statements like above make people believe that scripted testing promotes stilted • Even if requirements change, the impact of thinking. If you define scripted testing requirement change as related to testing can as “following the pre-written test cases be addressed in a better way with written VERBATIM” then the statement would make test cases. sense. If this is how scripted testing was In my experience of testing, many complex to be defined, I would not like to be found products including some which are used by siding with scripted testing. Of course, there millions of users, review of written test cases is no need to define scripted testing in this has resulted in better understanding of testing manner because that is not how most testers and consequently better execution of testing by test. Unless there is a client demand, most all testers involved in the review and not only testers do not follow the same test script the person who reviewed the test cases. again and again. They use test script only as a guide and not the gospel that has to be Of course, there are situations where one followed verbatim. Testers vary their data, does not have the time to write test cases. But their actions, and the sequence of actions. enough people write about these contexts. My purpose to write this article was to defend what ● Scripted testing wastes time. Time is wasted most people today would not defend. I do not in writing test cases whereas it could be favor exploratory testing over scripted testing or spent testing. Also requirements change vice-versa. I favor using whichever style gives often, resulting in time wastage when test the most benefit in a given situation, rather cases have to be reworked. than clinging to one end or the other of theThe difference between scripted and exploratory continuum.testing is said to be that in exploratory testingtest design, execution and learning happen atthe same time whereas scripted testing uses56
  • 56. Putting the “Test” back in Test PlanPutting the“Test”back inTest PlanU ntil fairly recently, many people who What to do when Confusion sets called themselves Software Testers in came by their jobs by accident rather I suddenly found myself in the uncomfortable than by design. This placed many position of not really knowing something thatof us in the difficult position of having to learn I thought I already understood. What do I dohow to do our job while we were trying to do that now? Where do I start?job. One of my biggest challenges was trying tofigure out what goes into a Test Plan. I began by asking people I worked with. I networked a bit and asked other testers forIn the first company that I worked for, the answer advice, but most of the time no one would saywas straightforward enough. A Test Plan was much because they thought it was confidentialsimply a commonly-grouped set of test cases to share information like that. I even went tothat was executed by a Tester. For example, a day-long workshop on “Managing Systemsthere was a Print Test Plan, a GUI Test Plan, a Testing Projects”. While I found the workshopMath Test Plan, and so on. No problem. to be very insightful, I didn’t believe everythingDoubt in that definition began to creep into the instructor had said because a lot of it wentmy mind when I started to network with other against what I had learned so far on my own.Testers in town and found out that Test Plans For example, a Test Plan was not supposed tomeant different things to different people. When have any test cases in it. This just increasedI got my first Team Lead role in the next company my doubt and anxiety.I worked for, I discovered that the Test Plan Back at my desk, faced with an empty MicrosoftI was being asked to produce was something Word document in front of me, I searched thecompletely different. It was supposed to be a Internet for Test Plan templates and examples.planning document of some kind. I needed more opinions and references than58
  • 57. by Paul Carvalho I became quite proficient at creating these Test Plan documents and customising them Acknowledging that prior for the needs for each project. I even became familiar with some industry standards for Test experience and knowledge Documentation1. based upon someone else’s Then I started to notice an interesting pattern incomplete knowledge is the emerge with the more projects I worked on. For biggest hurdle one thing, no one from any other department ever read any of these documents, even when they asked me to produce one. For anotherjust my word against my workshop notes. It thing, no matter how I modified this document,was at that time that I stumbled upon a few I never really found it useful in the day-to-mailing lists focussed on Software Testing and day management of the test projects I workedQuality Assurance. People on those lists were on. Finally, creating and maintaining thevery friendly and helpful and in no time I had test documentation always seemed to take aseveral examples to help me develop my first significant amount of time and effort away fromtest plans. the actual testing activity itself.The biggest hurdle I had to overcome was Naturally I tried to find ways to improve theacknowledging that my prior experience and efficiency of creating and maintaining theseknowledge were based upon someone else’s documents, but after a while I just started toincomplete knowledge. Until that moment, I question the usefulness of the document entirely.thought that I knew what a test plan was. Then One day I realised that I had fallen into the traphow was it that the definition changed when I of forgetting to question why I even had to createchanged companies? I had to unlearn the ways one. This time when I asked the question theof the Dark Side. response disturbed me: “Because it is expected that you create this documentation.”Test Plan 1.0 Wait a minute! Hold the phone! Who expects meMany of the examples and templates that I had to create this documentation? Why do I have towere very similar to each other with some minor create this documentation? Where’s the valuevariations between them. In general, I found in that? I’m the one doing the job here and I’mthat a Test Plan contained elements like the seeing that I’m wasting my time creating andfollowing: maintaining documents and procedures that ● Overview and Objectives don’t actually help me do my job any better. Where’s the sense in that? ● Testing Scope and Strategy (Features To Be Tested, Not Tested, etc.) A Brief Glimpse into ● Approach, Entry and Exit Criteria, etc. Project Management By sheer coincidence, it was at ● Responsibilities and Staff that moment that I attended a ● Schedule and Deliverables (Test Reports, Project Management workshop. I etc.) had requested it because I was struggling to manage several ● Resource Requirements (Test different test groups and projects Environment, Tools, etc.) simultaneously. An interesting ● Risks and Contingencies thing happened after attending that workshop: I realised that I 59
  • 58. Putting the “Test” back in Test Plan Development is no different Nature abhors a vacuum, When you stop creating Test Plans, you run into all kinds of resistance, especially from and Software Development is Management. There are some good reasons no different to create extra documentation and there are many bad reasons. To be clear, “Because it’snever wanted to create a Test Plan again. expected” is not a good reason.During the course I learned about the Project One valid reason for producing someManagement Institute2, we covered the Project documentation is to help you learn from whatManagement Body of Knowledge (PMBOK), you did on a test project. Being faced withand worked through several examples in all of the missed bugs now being reported byorder to understand how to apply the correct Customers through the Support departmentknowledge, tools and techniques in order to after a product ships is a harsh wake-up call. Ifsuccessfully manage a project to completion. I you don’t have a record of the who, what, where,highly recommend such a course for any Test when, how, or why you did or didn’t test certainManager. One of the most significant aspects things, it is more difficult for you to learn howof the course for me was learning about the to improve and reduce the number of escapedelements of a Project Work Plan: bugs in the next project. ● The Project Team It is important to note that a Project Plan doesn’t ● Project Charter completely replace a Test Plan though. While the project plan may outline all the players, ● Scope Management Plan the general work breakdown structure and so ● Risk Management Plan on, there is at least one thing not captured to any useful detail by a Project Manager. One ● Communications/Reportin of the single most important things that a Test Plan Team needs to own on any project is the Test ● Work Breakdown Structure Strategy. ● Master Budget It took me a few years to become reasonably good at developing and implementing an effective Test ● Task Responsibility Strategy. I think it is a key skill for any Tester or Test Manager. My learning path involved “going ● Critical Path Schedule back to square one” so that I could really knowWow. That was it! I finally understood what it what I needed to be effective. Nowadays therewas that a Test Plan must have originally tried are some good references and workshops thatto accomplish. At some time, before Project can help jumpstart you to reasonable success.Management became a formally recognisedrole in a Software Development organisation, Test Plan 2.0 – A Fresh Startit must have been commonplace for each of In the end, it comes down to this: you will neverthe teams to be required to produce their own be able to test everything; you will never findlittle mini project work plans. Since that need all the bugs; and you will never have as muchdoesn’t exist anymore, it was time to get with time, people, or resources as you really want.the program and stop wasting my time creating Starting with that in mind, your Test Strategytest documentation that clearly no one needed needs to help you work with what you have,anymore. in the time you have, to do the best job youNature abhors a vacuum, and Software possibly can.60
  • 59. by Paul Carvalho our strategy as the risks and priorities change throughout the development phase. This is Why do certain test practices Agile Development. Things change and we need or approaches seem to be to adapt our testing strategy along with the new information as it arises, whenever it arises. successful sometimes but not The next time you are asked to provide a “Test always? Plan”, remember to focus more on the “Test” than the “Plan”. If you find you are putting too muchFor me, the best way to accomplish this kind effort on the latter, you should take a seriousof “plan” was to use a Risk-Based Testing look at the value that the documentation effortapproach3. A few authors in particular had is providing your organisation. Time is money,the most influence on me during my on-the- and the last time I checked, I’m getting paid tojob education: James Bach4, Cem Kaner5, and test.Ross Collard. It was at that time that I alsocame to learn about the Context-Driven School Referencesof Software Testing.6 1. An example of an Industry Standard forContext-Driven Testing was an interesting Test Documentation is IEEE Standard 829paradigm shift that helped me to understand the - see piece of the test management puzzle: whydo certain test practices or approaches seem 2. To learn more about the Project Managementto be successful sometimes but not always? Institute, you can find them on the web atBecause that’s the way it is. Success depends a great number of factors that change with 3. For an excellent article to get you startedalmost every project. To be a successful Test on Risk-Based Testing, read James Bach’sManager requires that you are a keen observer article “Heuristic Risk-Based Testing”,of these influencing factors and can adapt your available on his web site at’s dynamics and approaches to suit the of your current situation. Darwin had itright when he said that things in nature adapt 4. James Bach: to find out more about hisor die. We should too. work and for excellent resources on Test Methodology, explore his web site atArmed with this new knowledge, I am now comfortable developing Test Strategiestailored to suit the needs of each project that 5. Cem Kaner: author, professor, andI work on. I take no offence when I am asked advocate of great software testing practices,to produce a “Test Plan”. Instead of providing he maintains several useful web sites.a document that someone else thinks I need, I His personal site is http://www.kaner.create something that I find useful – a usable, com/ but most of his work and Testingworking Test Strategy. Instead of worrying educational materials can be found for freeabout templates, cover pages, and document at systems, I worry about Quality Criteria, 6. To find out more about Context-DrivenTechnical Risks, and Test Techniques. School of Software Testing, check outA typical Test Strategy document of mine is one or two pages in length, and I don’t or pick up the book “Lessons Learnedworry about formatting or templates anymore. in Software Testing, A Context-DrivenIt sits on a network drive that is accessible to Approach” by Kaner, Bach and Pettichord.the whole development team, and we update 61
  • 60. Accelrys has over twenty years of innovation and technology leadership in delivering solutions that transform discovery and development research. The company provides market-leading modeling, simulation and informatics software, data pipelining and platform technology, and solutions consulting and support services. A unique combination of expertise in both life and materials science lets Accelrys serve a diverse range of clients, including some of San Diego, CA USA the world’s leading pharmaceutical, biotechnology, chemical, nanotechnology, government and academic research organizations. Cambridge, UK Accelrys is a global company. Our office in Bangalore, India undertakes Bangalore, India engineering and quality control tasks.Burlington, MA USA Tokyo, Japan For more information, visit our website: Paris, France
  • 61. EXPRESS YOURSELF“WhatIsTesting” magazine is written for software and application development mangers, projectmanagers, team leaders and Test & QA managers. The magazine is not written for the entrylevel tester running test scripts all day, but rather has a larger more strategic view of the entireapplication life cycle.The reader is a decision maker within an IT department. Articles in the magazine should provideuseful information to help him/her understand trends and emerging technologies, come to gripwith new and timeless challenges, adopt best practices and ultimately make better decisions toimprove software quality.PresentationArticles should be in the range of 1500 to 4000 words, and be written for practitioners in the field;with a style that is down to earth, practical, interesting and original. Avoid jargon, speculativetheory and abstract concepts; instead, focus on practical information that can help our readerstoday or in the near term.The purpose of an article is to transfer useful knowledge to the magazine’s readers, rather than topromote the author’s accomplishments or the products, services and technologies offered by theauthor’s employer.Don’t start the article with long fancy anecdote, a “first-the-earth-cooled” historical timeline oran academic-style abstract that says, “This article will do such and such”. Just get right into thesubject matter. Also, don’t include footnotes or references. Refer to other sources, such as book,magazine articles or Web sites, within the main body of the article. Be sure to define terms andacronyms that may not be familiar to the reader; incase of doubt, define.Include, annotated source code, only when it helps explain the concepts in the article. Also, addcopious figure, tables, listings, screen captures and other graphics to make the article easier tounderstand. Please avoid bulleted points within our prose; it is hard to read such lists. If you feelthat a list can best be expressed using bullets, consider pulling it into a separate tale or textbox.Do not assume that the reader has specialized experience or knowledge about particularmethodologies, platforms or tools. If the article is written about a specific methodology, platform ortool, present that dependency in the initial proposal. In the manuscript, include a short introductionor overview of prerequisite background concepts, if appropriate.Article ReviewsAll proposals and manuscript submissions should be sent to the magazine by the article’s author.If there are multiple authors, one author will be designated as the lead author. The editors maysend article, proposals and finalized manuscripts for peer review. In some cases, the editors mightask perform a peer technical review of the manuscript.Once the necessary review has been completed’ the article will be sent back to the article’s authorfor final approval.
  • 62. Welcome to TEST 2008 agility is synonymous with the testing team’s resourcefulness and ability to respond quicklyTest2008 is the first conference being organized to changing contexts. Organizations needby PureConferences in India. Our conference to catch the ‘window of opportunity’ whenwill provide a platform for international releasing a product to remain profitable. At theand national test professionals to interact same time, we, as customers only want to useand participate. Speakers from around 10 ‘quality’ products.countries, such as USA, UK, France, Sweden,Canada, Italy, Netherlands, including India will We are hoping to have intense discussionsdeliverkeynotes, tutorials, and papers during around the theme to make it a learningthe conference. We intend to involve academia experience for all participants. The theme isand institutions of learning with our conference. broad enough and will cover diverse topics.We also intend to make Test2008 a ‘green’ Keynotes, Tutorials, and Paper presentationsconference as far as possible. Additionally, we by renowned speakers known internationallywill institute two test scholarships for educating and in our country will be the highlight of thepromising students unable to finance their conference.studies. Please mark theTest2008 conference program inAnimated discussions among testers frequently your calendar. Our conference will offer partnersthrew up the requirement of agility in testing theopportunities to associate themselves withand the challenges associated with it. Creating this endeavor and showcase their products anda balance between conflicting and changing participate in the conference.demands of quality by customers and timely If you want to present a paper or tutorial, pleasereleases by organizations is a real challenge. fill in theSpeaker Submission Form. To attendAgility in Testing is the theme of Test2008--Test the conference, please Register.Excellence through Speed and Technology.Agility represents nimbleness, resourcefulness, We welcome all delegates to our conference.and adaptability. In the world of testing,