Wit magazine april_2008


Published on

WhatIsTesting magazine. Published in April 2008

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Wit magazine april_2008

  1. 1. What Is Test ng India’s 1st Software Testing Magazine April’08, Issue Reduce number of testing scenarios with this insight on - Cover Story Pg 21 Risk based TestingInside this issue Are you TEAM ready? Putting the “Test” back in “Test plan” Pg 57 Some things I have learned in Pg 44 Software Testing Pg Morning Ideas Pg 04 Have I got a deal for you? Finally Usability Testing Pg 28 Pg 40 14 With little Preparation & Teamwork you can pave the way for new Team Memers. and more...
  2. 2. Welcome to   is the first conference being organized by PureConferences in India.  We intend making  an experience for all participants by creating a platform for professionals and experts from the testing community of the Globe  will revolve around ’Agility in Testing’The theme will explore and expand on the need for testing teams to adapt to the changing demands of release and quality. will cover diverse topics. Keynotes, Tutorials, and Paper presentations by renowned speakers known nationally and internationally will be one of the highlights of the conference. Let’s mark the conference program in our calendar today itself.  will offer opportunities to the partners to associate themselves with this endeavor and showcase their products in the conference. We are sure you want to participate and/or present a paper or tutorial. Do visit us at http://www.test2008.in
  3. 3. Issue1: Vol 1: April 2008 ContentsCover Story Is bad testing pg eating your 21 Pr fits? 04 FROM THE DESK: Morning Ideas by Sunil Gupta Good design is key for a good and reliable product 08 THE PROCESS ROOM: Software Testing: ~ A field par excellence by Sudhir & Anjan Software testing as a field has brought in a high level of confidence in customers and users and has thus far provided far more challenges to the new age engineers and academicians. 12 TEAM MANAGEMENT: Are You Ready? by Michael Bolton With a little preparation and teamwork, you can pave the way for a new team member. 21 COVER STORY: Is Bad Testing Eating Your Profits? by Rex Black Testing should focus on mitigating specific risks to the quality of the system. Sequence of test execution should be driven by associated risk... 28 THAT TIME OF YEAR: Have I Got A Deal For You by Matthew Heusser The methods and extend of our testing are all choices; you could even say they are trade-offs. 33 LEARN HOW TO TEST: What a Tester Should Know, even After Midnight by Hans Schaefer Testing The Normal Way is Not Enough.... Where you find defects, dig deeper! 40 LEARN HOW TO TEST: Finally Usability Testing? by Erik van Veenendaal Will it finally happen!Other Stories 44 FROM DEV DESK: Some Things I’ve Learned in Software Testing by Jonathan Kohl 50 CONTROVERSY CORNER: QA is More Challenging than Development by Mrityunjay 53 Is scripted testing bad? by Vipul Kocher 57 Putting the “Test” back in Test Plan by Paul Carvalho WhatIsTesting.com 1
  4. 4. Tea testers look ataspects of dried & brewed tea leaves and the liquor Likewise, PureTesting goes into various aspects & nuances of soĞware testing We build innovati ve, end-to-end solutions, and manage critical testing processes, and reduce total cost of producing quality soĞware • Banking & Finacial Services • Datacom & Telecom • Pharmaceuticals • Embedded Systems • eLearning • EAI & ISVs Test Consulting • Testing Services • Testing Products India • USA • UK • NZ www.puretesting.com +91 (120) 4621010; info@puretesting.com Global Software Test Consulting & Services company
  5. 5. Editor’s Note Release of any software is a moment of mixed emotions, pleasure andpain, joy and anxiety -- joyousness of seeing the hard work resultingin a release and the anxiety of the unfound bugs that could have beenfound and fixed. After a lot of effort, this magazine is also being released with similaremotions. We take great pleasure in introducing Version 1.0 of WITmagazine. We hope to have many more versions of this magazine, bothmajor and minor, in years to come and hopefully with minimum numberof patches. We also hope that the future versions of the magazine willhave more features, will be more robust, will have an expanded userbase and establish itself as a dominant player. Instead of the usual Editorial about the contents of the magazinewhich you anyway can find out from the table of content, we take thisopportunity to share with you the reasons for this magazine and whoare the people who have contributed to it. Many of the technical magazines “grow-up” with time. The articlesbecome more “philosophical” and more abstract. Readers who areregular subscribers too grow with the magazine but most of the newcomers start finding the magazine “too abstract” for their liking. WITmagazine hopes to keep itself relevant to both old readers who grow upwith the magazine and new readers who are either new to the testingprofession or new to the magazine. Our sincere thanks to the initial group of people who helpedbrainstorm the concept of the magazine. They are – Bernard Homès,Danny R. Faught, Johanna Rothman, Kamesh Pemmaraju, MatthewHeusser, Michael Kelly, Rex Black, Scott Barber, Stefan Steurs. Withoutthem the magazine would not have become a living reality. All goodthings about the magazine are because of this group, the authors andother well wishers. All bugs are because of us. For putting together this version of the WIT magezine, I acknowledgethe efforts of Ashwin Razdan for collaborating with the authors andcollecting all the articles; Parag Sapre for designing the elaborategraphics; Anushree Tewari for providing editing support, and SatishThakur for giving it the final shape in which you see the WIT magezine.Hope you find this version readable and useful and we hope to have abetter V 1.1. ~ Vipul WhatIsTesting.com 3
  6. 6. From Daily Life Morning Ideas the Morning Desk IdeasThough a good design is key for a good and reliableproduct, a robust test and evaluation of the productonly ensures that the product meets all-explicit andimplicit requirements and is ready for deployment andusage.O ne morning when I was engrossed bags had torn almost at the same place. in my newspaper, my daughter Since it was not possible to repair the bag created panic as her school in ten minutes, thus she carried her stuff in bag had given way at one of the another bag but this episode left me thinking. shoulder belts. I realized that the bag was made to meetI was annoyed at her letting the functional requirements of the studentme know only when I was such as adequate space to carry stuff,completely torn while she would compartments for lunch box, pencil box andhave seen this happening over water bottle. The bag was also providing athe last few days but my bigger good aesthetics but lacked the robustnessreason of annoyance was due in terms of handling the stress that comesto the fact this was the 3rd or on the bag with daily usage....4th time the different school4 WhatIsTesting.com
  7. 7. by Sunil Gupta, Head of Testing Practice : Flextronics Software Systems by For product organizations, the composition and organization of the test team plays a very crucial role in the success of the product. Enough time Composition and and energy needs to be spent by senior test organization of the test team folks to plays a very crucial role in • Visualize the product (software and the success of the product. hardware) • Look at the various field configurationsTwo things emerged when I extended my thought • Identify the tools required for testingprocess further. • Look at the FOUR views:One, product making whether it is a school o User viewbag or a large and complex telecom product o Maintenance Viewrequires good understanding and research interms of features, stress areas, usability, etc o Functional/Operational Viewwithout which the product would give numerous o Performance Viewproblems to the user as the school bag gave to In order to achieve a good understanding ofmy daughter. the product before the exhaustive test cycleSecond, testing plays a very crucial role to commences, a few people from the test teamensure that the product meets all requirements need to be dedicated to carry out the Testincluding the ability of the product to handle Engineering tasks (as defined above). This setnormal and abnormal stress conditions. of people should have the mandate to carry out the verification and validation tasks.If we take the school bag example, it is veryimportant for the product manufacturer to In the school bag example, the User View would ensure that the various be to look at all the ways in which the school stitches in the bag take up bag would be used by a user. There would be the full bag load else there several user scenarios such as: are going to be numerous • Student lifting the bag from one belt dissatisfied customers like me. • Student lifting the bag using the side belt and loading on its back Though a good design is key for a good and reliable product, • Student fully loading the bag with books but a robust test and evaluation weight may not be too muchof the product only ensures that the product • Studentmeets all-explicit and implicit requirements and partiallyis ready for deployment and usage. loading theIt is seen across the IT and software industry books butthat the organizations give a lot of focus to hiring the weightand developing software development teams may havebut do not focus enough on developing testing exceeded theteams. The test team is put in place either verylate on the programs or the test teams are notorganized well enough to provide the vital valueexpected from the Test team. WhatIsTesting.com 5
  8. 8. Morning Ideas Suddenly I realized that it was 9:00 hrs by the watch and I had a 9:30 am meeting with the Product testing and Quality Group to review the Post Release Defect Density of the products that we shipped in the deployment is a very last quarter and had only 30 minutes to reach involved and rigorous task office. Thanks to god, for my home is only a 7 minutes drive from office. recommended weight Bye for now……Maintenance View would be to look at aspectssuch as ease of cleaning and washing the bag;ease of repairing the bag and how easy it is tofind the spares such as zippers, buckles, etc.Functional or Operational view is anotherimportant aspect of testing. Tester has toconsider all the functional aspects of theproduct. In this example, aspects such asfollowing would be tested: • Ease of keeping books in the bag • Access to all the pockets of the bag • Ease of closing and opening the bag • Ease of lifting the bag on shoulder and on the back • Enough space to keep books, lunch box, and water bottle, etc.Last but not the least, the Performance view ofthe product in terms of load handling, abilityto bear stress due to regular use throughload balancing is considered. In the example,tester has to take following aspects intoconsideration: • Testing the bag for various load and stress situations • Testing the bag for full load capacity • Testing the key joints/stitches under the stress situations • Identify the weak areas and strong areas of the productProduct testing and deployment is a veryinvolved and rigorous task and involves the bestof people and the best of the practices.6 WhatIsTesting.com
  9. 9. Neilsoft is a specialist engineering services & solutions company focused on helping our clients enhance their product engineering efficiency. Our services in the software product engineering domain span the entire product lifecycle including development, testing, and localization engineering.T E S T I N G E X P E R T I S E Glo ba ise l ert de Compatibility p Testing liv ex e n ry ai In Te mo in on m st st st ti Do al in Te gra g de la g tio te l In n rage Sto S Partnerships with ind ec uri C AE Performance Functionality ty Testing Testing M CA PL M C AD ustr y le s M Tes sse in ed ul ti st at ad ti- ng ce g Te tom by ers te ro u A p st bu Certification ro Testing & ven Pro testing@neilsoft.com | www.neilsoft.com/testing.htm Pune Bangalore Canton Chicago Cambridge Dalian ) +91-20-2605 3003 Focus verticals: Construction Industrial Machinery Software Transportation Energy
  10. 10. The Software Testing: ~ A field par excellence Process Software Corner Testing ~ a Field Par ExcellenceApart from giving fillip to the economy, Software test-ing as a field has brought in a high level of confidencein customers and users and has thus far provided farmore challenges to the new age engineers and acad-emicians.Introduction and ITeS segment known as Software Testing. Renewed growth of the Software Industry hasAwareness on Software Quality has, over the last helped testing achieve the position of beingcouple of decades, become one of the defining called the watchdog of quality. Apart from givingfeatures not only in Indian software sector but fillip to the economy, Software testing as a fieldalso from a global perspective. Thanks to the has brought in a high level of confidence indotcom crash which was an eye opener for customers and users and has thus far providedeveryone in the field to look back and focus far more challenges to the new age engineerson enhancing the quality of the products that and academicians. Looking at some of theare already in the market for many years. The astounding figures:quality initiatives taken through the last coupleof decades, have now become all pervasive and 1. Bangalore has seen a 24% growth in itsthus given rise to a new avenue within the IT recruitment drive in software testing. It is8 WhatIsTesting.com
  11. 11. by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL, Bangalore Facts from Industry 1. $3.0 billion of $4.6 billion in outsourced Composition and organization testing is sent offshore, throwing up of the test team plays a very opportunities for companies in India that thrive on low-cost knowledge workers. ~ crucial role in the success of Source Aztech Software - Bangalore the product. 2. Testing could make up to 25-50 percent of software budgets. Independent testing is growing at 50-65 percent while the part forecasted that India needs about 18000 of work done offshore is growing at 35-40 test engineers in the year 2005-2006. percent. ~ Source - Partha Iyengar, vice-2. With India having more than 89 companies president at industry researcher Gartner at SEI CMM Level 5 assessment; 275 Indian 3. Testing service team at Wipro has jumped software and BPO companies acquired four-fold to 2,400 in two years. In the quality certification; and more and more nine months to December, revenue grew IT-ITES companies have dedicated quality 90 percent to $64 million, three times departments responsible for developing and the industry average. ~Source - C.P. deploying the quality policies and reviewing Gangadharaiah, VP Testing services, Wipro them. 4. The global market for software testing is3. NASSCOM survey has indicated that the around $13 billion. In India alone, the road ahead for the Indian software and demand for software testing professionals services companies within the quality is expected to touch 20,000 to 30,000 arena was largely dictated and tuned to by December 2006. . ~Source – Internet the developments taking place in the global Search marketplace.4. The future would see organizations move Recent Trends from quality assurance to business With this background, we can ask ourselves assurance and focus on information security - what conclusion we want to drive from it? complying with international legislations, Why suddenly is everyone so statistical about developing new global delivery capabilities gathering all these figures? To take a close look and enhancing quality processes in new below we see from 50’s onwards till 80’s either business areas. the products were not tested or if they were then it was out of compulsion.This is testimony to the fact that softwaretesting is poised to grow at a phenomenal rate Quality as a term itself took the real shapein future. With all these happenings, no one can under the leadership of Borris Beizer who gavedeny the conclusion that software testing has the awareness to the industry and the worldgrown as an independent field within IT/ITeS about its very existence. The role of quality hasservice segments. And it has given a clarion become a necessity than a need in the softwarecall to everyone within the professional arena, industry.specifically academicians, to closely monitor, If we start looking into history, we will findanalyze and push forward the new horizons of thousands of such cases where a small mistakethe technology movement that software testing in the software has brought havoc to life andas a field is going to provide. Looking at some money. The most recent one is the Columbiamore interesting statistics on testing industry: Space Shuttle crash. All such incidents have WhatIsTesting.com 9
  12. 12. Software Testing: ~ A field par excellenceespoused within the IT-geeks eagerness towards human life such as Insurance, Healthcare,“checking before delivering” approach. On the Banking, Financial services, Pharmaceuticals,contrary, there was time when products were logistics etc. What this means is that, softwarereleased without or with minimal testing done. developed in each of these Industry verticalsOf course, many a time, these are dictated by need testing – some rigorously and some whichhow the market wants them. Time-to-market can give adequate confidence in the product. Atwas probably more important than quality the end, software testing too is spanning acrossof the product. But, time has changed post all these Industry verticals. When it comes todotcom crash era and software quality has the domains, the field looks even more lucrativemaximum thrust in the market today, be it by the amount of challenge each one of themconsumer products, electronics, manufacturing provide to the individuals involved in testing.or software. And quite rightly, software testing Therefore, as software testing started to supportis enjoying its share of this quality conscious the development activities in these domainmarket. areas, it became increasingly necessary for the test engineers to gain knowledge and specialize Testing as a field has now been more within each of these domain areas. This is strategic and approach-centric another area where there is a huge potential thus adding new dimensions as for test engineers to grow and make an impact. well as many challenges to it. This led to different teams being formed, some Previously testing used to be an heterogeneous and some homogeneous groups afterthought activity in the post to cater to testing software in these domains. release phase when the clients The requirements of the client keep oncall and say “we found something wrong in growing and we as organizations need to meetyour delivered software”. But with time, it is no the demands of the end users to stay in thislonger the case. Any small error today is a big business. This translates to the whole gamutissue from a client perspective. With more and of technology issues starting from software,more third party software testing companies hardware to third-party products. Software cangrooming overtime, the availability of a talented be legacy application running on a mainframeresource pool, making timely deliveries and the or the latest technologies such as .NET, J2EEcommitment to meet the quality expectations etc... Technology as we know has advanced fromof the customer has added many feathers to the days of mainframes to today’s applicationsIndia’s software testing market. Needless to running on a PC based Linux systems. However,make a note that Software testing as a field at the same time, applications developed on thewithin IT/ITeS has now emerged as a strong legacy technologies have not gone away or havearea of business solutions and a source of not been replaced with new technology; they aremajor revenue generator and emerging more as still being used in Banks, Insurance companiesa serious business goal as well an intrinsic partwithin the IT/ITeS organization culture.Future Challenges There is software in almost every aspect of human life....Domain and Technology Perspective software developed... needThe growing demand from within the customergroups as well the growing demand of the testing -- some rigourouslyorganizations to meet and extend their client and some which can givesupport has made software testing as a field adequate confident in thenothing but a paramount knowledge pool.There is software in almost every aspect of product.10 WhatIsTesting.com
  13. 13. by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL, Bangaloreand many others. This even complicates the Conclusionscenarios wherein the vendors have to support Moving a little off track to give a comparativeboth old applications as well as new applications. analysis we see in the past Public AdministrationThis tells us how complex it can be for testing has struggled a lot to give itself an individualto handle such complex set-ups. Besides, the identity and for a long time shared a part of itsapplications run on varied platforms from glory with either Political science or Sociology.different vendors including: Windows, Linux, But today it has gained a public reputation andMacintosh, IRIX, AIX, unix & Mainframe. We has become a part of everyone’s life.are currently looking from a macro level. Sincewith the growing numbers of different flavors of So does it hold true for software testing which hasan individual OS (e.g. Linux as such has Linux remained cocooned within Software EngineeringWorkstation2.0/3/0, Advanced Server 2.0/3.0, or Software Development Life Cycle. But nowEnterprise server 2.0/3.0 etc.) the coverage from the field has not only gained reputation but alsotesting point of view becomes immeasurable. has stood on its own. Software testing as a fieldDoes that not give a test engineer enough has come of age bearing the pain of “We can dochallenge to work on so many platforms? Of without you” to share the glory of “With you wecourse yes, it’s not that developers only code are right through”. A field that is still growing,for making an application run over different shining and awaiting to flourish still needs a lotplatforms but the test engineers too have their of planning & strategizing from all sections inown way of reaching end to end. Apart from this the Industry and academicians. As we need towe can also predict a good deal of exploration remember the following lines:in areas like Client-Server, Web (2-tier, 3-tier) “The Road to Excellence is a never endingapplications, Mainframe application and an Road” ~H. James Harrington, The Improvementendless list of such applications. ProcessWhat we discussed as far as technology isconcerned is only the tip of the iceberg. Thereseems to be an endless list and mushroomingapplication areas to be explored in the time tocome. WhatIsTesting.com 11
  14. 14. Team Are You Ready?Mana-gmentAreYouReady?With a little preparation andteamwork, you can pave the wayfor a new team member.Use a checklist to make sure you have everythingready before the new team member arrives.● Compile all necessary documents into one binder that can be given to any new arrivals.● Be sure to provide some context about the company and the team to help the newbie get acclimated.12 WhatIsTesting.com
  15. 15. by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.I f your company is typical, it is hiring agreement. Make sure this, too, is prepared more and more temporary workers while in advance, and that all agree on each of its scrambling for every bit of efficiency and points before the contractor’s first day. If your value. Hiring a contractor to help with human resources department has policies andtemporary problems can be cost-effective, but procedures manual appropriate for your hire,wasting his or her time undermines the purpose make a copy available to the contractor earlyof the exercise and costs hundreds of dollars in the game. Human resources will typicallyper day. Moreover, people simply don’t work as require various kinds of information fromwell when they feel uninformed, frustrated, or your new hire. Be sure to ask in advance whatstymied. On the other hand, a contractor who information they’ll need so that you can passcan hit the ground running will make better on any questions to the contractor prior torelationships, will be more effective, and will her arrival. Human resources or security willuse your time and hers productively. typically supply ID badges, pass cards, access codes, and keys to the office, restrooms, or This article describes ways to prepare other secure facilities. These items should befor a new contractor; however, the tips here available and tested.could just as easily be applied to a temporaryemployee, a new permanent employee, or in If the contractor is arriving from out of thesome cases, a transfer into your department. I country, make sure that your human resourcesdon’t assume any particular job description or department and the contractor have fulfilled alltitle for the person, nor do I assume anything requirements related to visas, work permits,about the person’s technical skill in any given and the like. The accounting people will needcategory. Neither should you make such to deal with administrative and taxation issues.assumptions. A crackerjack developer may be Make their jobs easier by coordinating andcompletely oblivious to network configuration relaying the required information in advance,issues — especially in a new environment both from your staff and from the contractor.— and a brilliant documentation writer might Issues related to eligibility, tax numbers, taxbe flummoxed by a cryptic voicemail system. forms, and withholding information should beYour contractor’s time is best used to solve the sorted out concurrently with the contract.problem for which you’re hiring her, not herown infrastructural or contractual issues.Contracts and Administrivia Restrooms with locks butBefore beginning work, your contractor will no keys are scourges upontypically sign a contract outlining the scope of the land. When I started atthe work to be done, the deliverables expected, one small outfit, it was aand a schedule detailing when each element week before security couldof the work should be completed. The morespecific the contract, the less opportunity there produce a restroom keyis for miscommunication and disagreement, for me. It wasn’t only a bother for meand the better your interests are protected. to ask my new colleagues for a key—IYour contractor may have requirements that areoutside the scope of your company’s standard was interrupting their work, too.contract, or she may have to provide certainkinds of documents relating to work eligibility. At this company, the restrooms wereMake sure these kinds of issues are ironed out locked, but the door to the server roombefore the contractor is slated to begin work. was always open.Most contracts come with a non-disclosure WhatIsTesting.com 13
  16. 16. Are You Ready? Elizabeth, a European There are few things that make a contractor contractor of my happier than being paid on time and with a minimum of hassle. Ask your accounting acquaintance, was hired department what they’ll need from the contractor to work in an American to pay invoices promptly. company. She submitted Ensure that expense and reimbursement policies an invoice, covering salary are clear from the get-go. If your company uses and expenses, which was a standard expense reconciliation form, make sure that a few copies or an electronic template rejected on the basis of a couple are available to your contractor. Again, your of questions from the accounting most friendly and helpful contact person in department about her tax withholding accounting should be available to the contractor status. She dropped in to investigate to help sort out confusion or difficulties. the problem by speaking with Mark, IT Issues the accountant who had sent her the Notify the IT support department that a new hire mail rejecting the invoice. “I don’t have is on her way, and request the specific things time,” said Mark. “Not even for just a you’ll need and the date you’ll need them. Most new arrivals will need at least one computer set couple of questions?” asked Elizabeth. up, linked “No,” Mark replied. “It would be a to the network, loaded with tools, attached to different story if you were an employee printers, and tested. If you’re hiring a developer rather than a contractor. But,” he said or tester, additional terminals or workstations imperiously, “I have more important may be necessary. Don’t forget to tell the IT department about this, and above all, don’t strategic things to do.” This incident assume that they’ll know what the new arrival occurred at a company where more needs. Your new arrival will need access to than half the staff was made up of various areas of the network, so make sure contractors. she has the appropriate set of access rights. If your company’s security is very restrictive—for Eventually the problem was sorted example, limited out, but it meant inconvenience and Internet access—notify your contractor exasperation for Elizabeth and her beforehand so that other arrangements can be made for Net-based research tools and manager, and embarrassment for resources. An analog line might be required Mark’s manager. On the other hand, for some purposes. If specific tools—examples Elizabeth’s final report included a include compilers, configuration management subtle recommendation—which Mark’s programs, and testing utilities—are required, have these loaded and tested on the contractor’s manager followed—to give Mark less machine in advance. Be sure to follow up, strategic work to do. Whether Mark is verifying that the work has been done and currently doing strategic work or not, that resources, configuration settings, network policies, and default passwords are documented. he’s doing it somewhere else. Make sure that email accounts have been set14 WhatIsTesting.com
  17. 17. We challenge you. At GrapeCity, we challenge you with assignments on the latest .NET and Java technologies, enabling you to emerge a winner in this game called life. Lines of Business Business SolutionsGrapeCity is a software development multinational company Custom Applicationswith over 650 employees across Asia and the United States. For Mobile Applicationsmore than 25 years, we have brought optimized software solutions CRMand services to enterprises around the world. ERPWe rely on four key principles to help our clients achieve their goals: Technical Servicesthoroughly understanding our clients’ business objectives, providinghighly personalized experiences, maintaining a strong emphasis on Mobile Portingquality, and adhering to the highest ethical standards. Quality Assurance Technical Support Send your resume to Component Development jobs.india@grapecity.com now! Technical ToolsGrapeCity India Pvt. Ltd.A-15, Sector 62, Noida 201307 Phone: +91 120 247 0123 Fax: +91 120 247 0124www.grapecity.com
  18. 18. Are You Ready?up and checked by sending amessage to the new arrival’s Note the people who are most likely to beaddress, and then check with helpful in answering questions or fixingthe IT department to be surethe message has been received problems, and provide their contact information.in the new mailbox. In addition,obtain a document from the IT voicemail system. Have the telephone installed,department on how to change network login and a number ready for the contractor when sheand email passwords. arrives. Make sure that this number is listed in the company directory — and remember thatEveryone needs a working telephone and directories can exist on the network, in the voicemail system, and with the receptionist. Despite everyone’s best intentions, a company’s information infrastructure can be complex The new arrival can often and confusing to a new arrival. Have the IT help people to notice department provide the name of a specific old bumps in the road. I person—a guide or sponsor who is helpful and knowledgeable about the company—to help the was brought into a test contractor navigate through network access, organization to help email configuration, and printer problems. In identify problems in testing addition, make sure that your contractor will be able to find her default printer, email address, efficiency. There were two types of network login name and password, and IP network drops in the test lab—one address. Try to counsel colleagues against for the company’s internal network sending email announcements saying things and one for the test network. These like “This week’s build can be found in the usual places.” Instead, make the details explicit. weren’t labeled, since everyone who used the lab knew which was which. I Providing Context suggested that this shouldn’t be taken The contractor has been hired to accomplish for granted as one of the locals helped some task or solve some problem. If people within your company are having difficulty me find the right connection, and I sorting out an issue, imagine the difficulty that used a bit of masking tape to label the a stranger will face! drops nearest to my workstation. The Even people who have worked inside your next day, a fairly senior tester reported company and who know the general environment a defect that was later found to be need background information when they spurious, at a fairly significant waste of arrive in a new position. This is yet another motivating force for your company to develop time and effort. The misunderstanding clear specifications, design documents, plans, happened because he had plugged the schedules, budgets, and so on. Start by writing a test machine into the wrong network brief description of what the company and your division do, and prepare an organizational chart connection. The day after that, labels or departmental roster. At the very least, identify magically appeared on all of the yourself and your direct superiors and reports, network drops. along with the other people with whom you expect the contractor will be working. As you’re16 WhatIsTesting.com
  19. 19. by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.preparing this information,note the people who are most The processes and documents thatlikely to be helpful in answeringquestions or fixing problems, you prepare for the arrival of oneand provide their contactinformation, so as to minimize contractor will be highly recyclable.the amount of time that the and you’ll quickly have a list that is easilycontractor must wait—unproductively—for maintained and printed.answers. Document the key business processesand workflows associated with the new arrival’s One of the benefits of traveling, from the pointtask at hand. If you provide answers early, there of view of many contractors, is using free timewill be less time spent wondering and askingquestions.Don’t get hung up on tools or formatting: A At one company where Idocument composed using a text editor or was a full-timer, we hostedeven a pen and paper is infinitely superior to adocument that doesn’t exist at all. When offered a number of developersa choice between a detailed Notepad file and an during an outsourcedempty Gantt chart, I’ll pick the former every time. project. We created a listSimply make sure that the information exists, of local restaurants andand that it can be found and read easily. Make ita mandate from the beginning, and check with tourist attractions by sending outthe new arrival often. If the new arrival feels email to everyone in the department,that she is being forced to do detective work, soliciting one suggestion from eachencourage her to keep a log of what she neededto know and what she found. Make sure that person. I compiled the suggestionsone of her deliverables is a list of the things that into a single document, removed thewill be useful to her successor, and be sure to duplicates, and posted the results tobudget time for her to prepare it. an internal discussion board. A few ofCity Guide the suggestions inspired still more.If your new arrival is coming in from out of town, Some might suggest that discussionshe’ll feel more welcome with a guide and some of restaurants on company time andidea of where important services can be found. company bulletin boards is wasteful.If your company is providing accommodations,make sure that the contractor’s lodgings are Our experience was that the one-quiet, secure, comfortable, and reliable, and suggestion technique cost next to nothat they’re ready for the contractor by the time and was a way of getting to knowend of the first day of work, at the latest. If people—and people who knew eacharranging accommodations is the contractor’sresponsibility, provide a list of facilities in a other better almost always workedvariety of price ranges. While it’s nice to be close smarter and harder.to work, don’t strand the contractor near anindustrial park without food or services nearby. We also found that a handy list made itPrepare a list of restaurants in the area that easy to book company gatherings, tocater to a variety of tastes and budgets. provide directions, and so on.Ask your department for recommendations, WhatIsTesting.com 17
  20. 20. Are You Ready?to explore their surroundings. Survey your people she’ll need to know. An introductionstaff for a number of interesting— and possibly from you is likely to carry more significanceundiscovered— places to visit in the area. If than one from a subordinate, so be preparedthose places are somewhat off the beaten path, to do this yourself. Even if you are routinelyask the person making the recommendation to busy, getting your employees and contractorsprovide a link to one of the several useful Web- to work together is a tremendously worthwhilebased mapping services. If everyone pitches in a investment. A contractor’s effectiveness oftensuggestion or two, compiling them into a helpful depends upon being able to meet and connectguide shouldn’t be burdensome. When you’re with other people. Most contractors meet theirfinished, you should have a complete package new colleagues in business meetings, but moreof general information that can be passed to informal circumstances can lead to friendlierany new employee or visitor on paper, via email, relationships. On the introductory tour, takeor on an internal Web site. Ask the new arrival your time. Cover the important places and people,frequently for feedback on the checklist, and but be relaxed and try to avoid a hard-and-fastrefine it accordingly. agenda. Spend a few moments introducing the new contractor to each staff member. ExplainOn the Job quite generally what the new arrival is thereMake sure that you schedule some time on to do, and describe your staff member’s role,the contractor’s first day for introductions and how each may be able to help the other.and a tour of the essentials. You should be Don’t just concentrate on peers and colleagues.prepared to walk around for at least an hour Spend a few moments chatting with other usefulwith the new arrival, introducing her to the contacts with whom your new arrival will need to interact—people in network services, reception, the mail room, and accounts payable. As you’re The kitchen or lunchroom walking around, show the new person all of the is not only the place useful and necessary places around the office. for refueling but also a An early stop should be the restroom. If keys or pass cards are required, make sure that the good place to meet and contractor is given them upon her arrival. The IT converse with others. department should have set up a default printer Linger for a while and and informed you which one it is. Include this encourage chatting, even on the tour, along with specialty printers and other equipment that might be useful. Also if it’s not directly related to the task on the tour, include visits to the office supply at hand. Spend a little petty cash cupboard, the photocopier, the first aid kit, and on group lunches, outside coffee the lunchroom. For the duration of the contract, the contractor is a member of your company breaks, or visits to the pub with your and your department, and thus should be contractor and the rest of your staff. accorded at least the same kind of assistance If these informal approaches seem and respect given to your permanent employees. of questionable value, consider that If you are the contractor’s supervisor, you must be prepared to go to bat for her, and the rest of productivity and effectiveness are your organization should be prepared to take often based on personal relationships the same approach. and rapport—do what you can to foster No matter what the task, there’s nothing worse them. than having to research and answer the same questions over and over again. Keep a list of18 WhatIsTesting.com
  21. 21. by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.frequently asked questions, their answers, and complete, can be reused and refined, but rarelywhere someone can go for more information. In will you have to start it again from scratch.many cases the FAQ information can be posted You’ll need to review things periodically, buton an internal Web site, captured in a database the benefits will be immediate: happier, better-within workgroup software, or collected in a prepared, and—most importantly— morepiece of email. If you don’t have such things set productive workers.up, consider making the collection of FAQs atask for the contractor, making such a documentpart of the contractor’s package of deliverables,since your contractor will have plenty of insighton what’s important for a new arrival to know.Reuse and RecycleAll this might sound like a lot of work— and thefirst time you do it, it will be.Luckily, the processes and documents that youprepare for the arrival of one contractor will behighly recyclable; they can be used again andagain, regardless of whether the new arrival isa temporary or full-time employee. Since you’llbe consulting with other departments, you canexchange ideas and plans, and improve the lotof new arrivals companywide. The work, onceChecklist for a New Team * Policies and procedures manualMember * Expense and reimbursement policyPrepare for Arrival * Expense forms* Signed contract * Telephone and voicemail instructions* HR info from contractor * Telephone directory, with key contacts* Keys, badges, and codes ready and highlightedtested * Internet policy* Visa, work permit, and eligibility * Password and login instructionsconfirmed * Software resource manuals* Resolution of tax issues * Names of key contact people in IT, HR,* Telephone and voicemail set up Accounting, etc.* Phone number listed in directory * Organizational chart* Set up computer: * Division responsibilities networked * Overview of company all applications installed and tested * Key business processes and workflows email account related to job login * Guide to city, if applicable passwords * List of frequently asked questions (with answers) printer connection Documents to Provide WhatIsTesting.com 19
  22. 22. CORPORATE OVERVIEWHexaware is a leading global provider of IT and BPO services. Over 40 of our clients are Fortune 100 companies. Weenable clients achieve competitive advantage by co-developing innovative IT/Process capabilities delivered throughflexible business models. We focus and have achieved leadership positions in domains such as HR, Banking andFinancial Services, Insurance, Leasing and Transportation. On the technology front we specialize in BusinessAnalytics, Enterprise Applications, Application Modernization/Management and Independent Testing. Founded in1990, we are a US$ 153 Million company with over 4000 employees in 16 locations worldwide. We are currently rankedas No. 11 Software Company in India. TECHNICAL EXPERTISE Our Key Offerings Technology – Legacy Systems, Client Server, J2EE BR / FS/SR Specification Industry Specific Industry Specific Business Requirement Analysis Gap Analysis Functional Support and .NET. Solutions Solutions Operating Systems – Airlines Software Testing Existing Methodology Review Existing Tools Evaluation Multiple Virtual System Process Consulting Process Recommendations Manufacturing & High (MVS), technology System Integration Black Box - Functional Testing UNIX variants and Window Banking & Securities White Box - Interface Testing Testing (SIT) Grey Box - Regression Testing based. Insurance Performance Load Testing Stress/Volume Testing AUTOMATION TOOLS Technology Practices Technology Practices Testing Analysis and Interpretation EXPERTISE Application Management Test Management Win Runner, Load Runner Enterprise Application Test Automation Functional Automation Load Test Automation and Test Director - Mercury Integration Interactive. People Soft UAT UAT Preparation & Execution Roll Out / Production Support Functional, Performance Oracle User Training Siebel Support and Test Management Tools E-Business solutions Fail Over / Availability Multi-tier Availability - Rational Software. Low Resource Testing Endurance Tests Silk Test and Silk Performer from Segue Software.HITS(HEXAWARE INDEPENDENT TESTING SERVICES) OVERVIEW Team of 250+ with varied skill sets Domain focused, Automation driven E2E testing solutions Independent testing labs based in Chennai and Mumbai Showcases projects spanning a diverse spectrum of Domains – BFS, Insurance, ERP and Airlines Technologies - Mainframe, J2EE, Siebel, People soft and SAP Expertise in test automation tools like Win runner, Quick Test Pro, Load Runner, Test Director, Rational suite. More than 35% have hands on experience in Mercury tools. Proven Test Methodologies from end-user perspective to meet the customers expectations under real-world business scenario. Offshore – Onsite Delivery model providing significant cost savings More than 1000 Man years experience Hexaware Technologies Hexaware Towers, Bldg. No. 152, Millennium Business Park, 51/3, G.N Chetty Road, TTC Industrial Area, TN S 3 "A" Bl k M h
  23. 23. Cover by Rex Black, President, RBCS, Inc., Bulverde, TX Story Is bad testing eating your Pr fits?Using Quality Risks toGuide Testing EffortTesting should focus on mitigating specific risks to thequality of the system. Sequence of test execution shouldbe driven by associated risk...S ince the 1980s, when Bill Hetzel Categories of Quality Risks and Boris Beizer published their Computer software, hardware, and systems can influential books on software testing, fail in the most amazing and various ways. Some we have known that testing should be bugs are not functional problems, but fall intorisk-based. Testing should focus on mitigating other quality risk categories. We’ll take a look atspecific risks to the quality of the system. The some categories of quality risks to stimulate oursequence of test execution and the total effort thinking about what to test.expended on any given test should be driven bythe level of risk associated with that test. In the Functionality1990s, people like Rick Craig, Paul Gerrard, andI independently created some ways to achieve The most obvious quality risk category is thatsystematic risk-based testing, often by adapting of functionality. There’s always a chance thatideas from other forms of engineering, such as the system does not provide some function itinsurance and medicine. In this article and the should. The function can include a capability,next one, we’ll talk about ways to perform risk feature, processing, input or output.based testing. For example, if you’re testing a calculator21 WhatIsTesting.com WhatIsTesting.com 21
  24. 24. Using Quality Risks to Guide Testing Effort 4 Unacceptable performance load above 500 Transaction Processing Time 3 Unacceptable performance at 2 any load Unacceptable performance degeneration above time 1 Acceptable performance Transaction Arrival Rate 1000 Figure 1: Performance bugsprogram, the lack of an add capability would fall performance. The shaded area at the bottom ofinto this category. So would a situation where the figure represents the required performance,the add capability was implemented, but the “+” which is one second or less transactionkey didn’t cause the function to be activated. processing time at a load of up to 1,000The add capability might be implemented and transactions per minute.accessible, but might work only on integers, not Risks to system quality in the area of performancereal numbers. The add operation, when carried include the possibility that the system respondsout, might give the wrong result, as in 2+2=5. to input, produces output, orOr there could be some strange side effect processes data too slowly underwhere the operation is handled but the all levels of load. The straightresult is not at all as expected, as in a dashed line shows this possibility.divide function where 2 divided by 2 The system might perform fine upreturned one, but in Roman numeral to some level of load, but have anformat. unacceptable non-linearity in the performance curve, as shown by thePerformance and Reliability steep curved dotted line. Finally, the systemIn Figure 1, you see a graphical representation might perform within specifications during anof three types of quality risks in the area of initial test run, but subsequent tests—when the22 WhatIsTesting.com WhatIsTesting.com 22
  25. 25. by Rex Black, President, RBCS, Inc., Bulverde, TX include seeing functionality, performance, or reliability problems due to the rate of Reliability problems exist computational, data, and communication flows. when the system fails to For example, the performance of database function normally each and management systems often start to degrade when the database management system load every time. exceeds 80% of the rated transactions per minute or the allowed number of simultaneoussystem is not rebooted between tests—might connections.reveal unacceptable performance degradation,as shown in the family of three gently curved Installation and Deinstallationlines. Various things can go wrong when installingThe risk of performance degradation—and programs. These include situations where theindeed, behavioral degradation of any sort— program won’t install or the installation processover time brings us to the reliability category causes damage to the system.of risks to system quality. Reliability problemsexist when the system fails to function normally The latter kind of problem once was ubiquitouseach and every time. Such a reliability bug can on Windows systems, with theexhibit itself as an intermittent functional appropriate nickname “DLL hell.”problem. Sometimes, such bugs Here, one program overwroteappear after a long period of runtime libraries used by another programor after running under heavy load. during its installation process.I had such a problem with Windows Deinstallation can also create2000 while writing this article. Extended problems. Sometimes, deinstallationuptime between reboots usually resulted in does not completely remove files andcharacters in some applications taking on the undo changes. Sometimes, deinstallationwrong font or the wrong font size. causes damage to the system.You might have seen another type of reliabilitybug, too, where the system functions normallywhen it functions, but crashes or hangsunpredictably. Such bugs can occur after some Various things can go wronglong period of runtime or after running under during installing & deinstallingheavy load. Alternatively, such bugs can betruly random. I recently saw such a bug when programs.a security patch caused reboots on a systemevery third or fourth time the system booted.Stress, Capacity, and Volume OperationsRisks to system quality in the area of capacity We also have those risks associated withinclude seeing functionality, performance, or recurring operating activities.reliability problems due to resource depletion. These include activities carried out at the endFor example, the performance of operating of some operational period, such as end-of-systems often starts to degrade once the system quarter or end-of-month closing out of accounts.consumes more than 80% of the available hard Problems can arise especially with failures todrive or memory space. archive inactive logical records in a safe andRisks to system quality in the area of volume recoverable manner, as well as archiving of data that should remain active. In addition, the WhatIsTesting.com 23
  26. 26. Using Quality Risks to Guide Testing Effort exhibit their effects a long way from their origin. Side-effects of changes can impair or even break The effects of changes, even cohabiting, interacting, or underlying software. small, localized, isolated The adding of yet-another-feature can lead to changes, are not always systemic performance and capacity issues. small, localized, or isolated. Usability and User Interfacetiming of archiving and the calculation of when One special category of non-functional qualitya period has ended and a new period begun can risks relates to the usefulness and usability offail. the system by the intended user or operator. Specific usability and user interface bugsFor systems with databases, there can be regular can vary. In some cases, systems presentrequirements to compact or repair the databases. cumbersome interfaces that do not followIn addition, for such systems, operators normal or expected workflows, leading tooften must back up data and configurations user frustration, confusion, and mistakes. Inregularly, along with verifying the restore other cases, functionality, while present in theprocess. There’s a strong risk of side effects system, is practically inaccessible to the userlike performance problems related to backup orrestore operations, the inaccessibility of someor all system functionality during backup or Data quality poses a majorrestore operations, and the failure of backup or category of quality risks forrestore operations under full system capacity many systems.conditions. Sometimes, operations activitieshappen automatically, in the background. Forexample, rollback logs for a database often or operator. Systems can be inappropriatelyneed to be purged automatically to avoid major difficult for the users to learn, leadingperformance and reliability bugs. to abandonment, mistakes, and inefficiency. Finally, users andRegression operators can find instructional,By regression, I mean when a help, and error messages thatpreviously-working feature, function, are misleading, confusing, orcapability, or behavior fails following a misspelled.change. The problem we have in systemtesting is that software/hardware systems, Data Qualitybeing digital, tend to be discrete rather than Many systems exist primarily to do interestingcontinuous in their failure modes. things with data. Inputs are processed,The effects of changes, even small, localized, transformed, and stored. Databases areisolated changes, are not always small, localized, queried, records linked through foreign keys,or isolated. How many times have you heard a and integrity constraints enforced. Outputs arestatement like, “How could that have happened? displayed, printed, and transmitted to otherI only changed one line of code.” system.A small change, even one line of code, can affect So, data quality poses a major category ofthe behavior of the rest of the system. A small quality risks for many systems. The systemchange can cause incompatible data to enter might corrupt or lose data. The system mightshared data sets, including dynamic dataflows store bad or nonsense data in a databaseand static databases, resulting in bugs that without integrity constraints. Databases shared across multiple systems can allow data which is24 WhatIsTesting.com
  27. 27. by Rex Black, President, RBCS, Inc., Bulverde, TXvalid for one system to be accessed by another Of course, if the system does not support thesystem which does not know how to handle character sets used by the local language, wethat data, resulting in failures that are removed have a localization problem. Those languagesin time and feature space from the source of the that use the Roman alphabet, such as English,problem. German, and Spanish, have single-byte character sets. Some languages that use otherFurther complicating these situations, it is alphabets, such as Russian and Hebrew, havesometimes difficult to restore the system to single-byte character sets, too.working state after a badly-handled errorcondition. In some cases, the system, in the Localization also has operational implications.course of succumbing to the error, damages For example, time zones and time formats varyconfiguration files or static data stores in a way based on locale. Does the time change in thethat’s hard to fix later. The data quality bug in the summer? (In the United States, this is calledexpense reporting package I mentioned earlier “daylight savings times,” but it goes by differentis an example of this kind of misbehavior. names around the world.) If so, when does adjusted time begin and end? AreDate and Time Handling dates written “month-day-year” (asIn addition to having to handle in the United States) or in the morecommon errors, many systems must logical “day-month-year” format (asalso handle dates. This has created in much of the rest of the world)?significant information-technologyproblems, including the infamous Configuration and“Y2K bugs” that consumed huge Compatibilityproportions of company and governmentIT budgets in the late 1990s and contributed to Quality risks related tothe depression in the high-technology sector inthe early 2000s. In addition to having to handle Localization may be inonce-in-a-hundred-lifetimes type of events like the categories of GUI ora new millennium, systems must frequently operations.handle events like leap years.Many systems must deal with expiration dates. Another family of risks to system quality livesLicenses expire. Credit cards expire. Insurance in the areas of configuration and compatibility.policies expire. After some period of time, the Different users configure both the hardwareright of a bank to disqualify a borrower based on and software of systems differently. A familya derogatory like a bankruptcy can also expire. of systems may support various hardwareFailure to handle such expiry events properly configurations. Often, the number of potentialcan expose a company to serious financial and combinations of configurations is huge andlegal risks. testing inappropriate combinations can result in risk of failure in the field.LocalizationLocalization refers to the ability of a system to Standards and Regulatorysupport local customs, languages, norms, and Compliancelaws. There are two broad categories of quality More risks to the quality of the system existrisks related to localization. One category in the areas of standards and regulations. Arelates to the user interface, and the other to system can work properly but be excluded fromoperations. the market by failing to meet standards, either WhatIsTesting.com 25
  28. 28. Using Quality Risks to Guide Testing Efforteffectively (no one will buy it) or legally (thegovernment won’t let anyone buy it or sell it). When you haveSecurity communicating components,With the growth of the Internet, awareness of timing and coordinationthe risks associated with security has increased.Security risks are legion. They include viruses between them is a concern.and worms, criminals breaking into servers,vandals causing denial of service attacks, Can You Think of Other Qualitybugging and intercepting e-mail and Internet Risks?communications, and more. Yes, and so could I. But isn’t this list already too long? What if you tried to cover all theTiming and Coordination quality risks discussed so far? Could you testWhen you have communicating components, or them all? Would they fit into the budget andshared components, timing and coordination schedule context of the project? Would youbetween those components is a concern. For end up wasting time and effort on unimportantexample, with an e-commerce system, how long problems? Surely any attempt to test this wholeshould the system wait between events like set of quality risks would be neither effectivemouse-clicks and submitted screens? At what nor efficient!point can it safely conclude that the customer So, what shall we do? In the sequelhas abandoned the transaction? to this article, we will discuss someAs another example, consider the ideas of how to trim the infiniteautomated teller machine. What number of things you could testhappens when the central computer into a finite list of risks you shouldtimes out or the network goes down? mitigate.What if that happens in the midst of atransaction? If we’re testing an inventory About the Author:system, what happens when two salespeopletry to sell the same item at the same time?DocumentationFor many people the quality of the documentationsignificantly affects their experience of qualitywhen using the system. Documentation qualityproblems include being technically incorrect, ofcourse, especially the examples. The user canfind the documentation insulting or offensive,teeming with grammatical or spelling errors, orafflicted with distracting cosmetic or formatting With a quarter-century of software and systemsproblems. engineering experience, Rex Black is president and principal consultant of RBCS, Inc., a leaderDocumentation refers not only to hard copy, but in software, hardware, and systems testing.also to electronic documentation. Help screens,installation instructions, error messages, andwizards are a form of documentation. This article is based on an excerpt from Rex Black’s book, Pragmatic Software Testing, published by Wiley26 WhatIsTesting.com