ST&QA 05-06 2010


Published on

How to reach ST&QA
Letters to the editor may be sent to:

About Software Test Professionals (STP)
STP provides more than 50,000 software professionals with information, education, training
and professional networking opportunities.

724 S Tejon Suite C • Colorado Springs CO • 80903

About ST&QA
ST&QA is published bi-monthly by STP (Jan, Mar, May, July, Sept, Nov). It is a printed magazine geared towards the software test and
quality assurance professional. Subscription to ST&QA is an STP Pro Membership benefit.
If you would like to subscribe to ST&QA, visit and join the Pro Membership.

Editorial Submissions
Visit to learn more about submitting editorial to ST&QA.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

ST&QA 05-06 2010

  1. 1. VO LU M E 7 ISSUE 5 J U N E 2 MAGAZINE 10 14 20Omega Tester: Performance Testing Implementing Six Sigma: What HappenedTesting with a Team within the Agile Improvement in Software to Software Test &of One Methodologies Development Performance Magazine? BRAND NEW Page A new look, 8 for STP new names, and an expanded vision!
  2. 2. Join & the Power of MEMBERSHIP is in Your HandsMembership Levels:BASIC – (free & limited access) The current digital edition PRO – Full access to the community, resources, Crews,of ST&QA magazine, Test Q&A Report, STP Insider and conference discounts and knowledge sharing opportunitiesselected web content. that matter most to you.EXECUTIVE – The highest level of membership designed What’s a Crew? Crews are online communities of membersfor senior-level executives. Contact us for details. interested in sharing ideas and knowledge with others in the field. Crew benefits include: • Opportunities to interact with peers for concept sharing, learning, debating and networking • Download content created by the industry to help improve the practices at your organization • Discussions of current trends with industry experts • Access to other professionals who can help with idea generation or test case/development ideas Leverage the Power. Join Today! • Networking with professionals in your geographic region / 877.257.9531 for online or face-to-face meet-ups • Offer your leadership on subject matter that relates to your area of expertise
  3. 3. Volume 7 / Issue 5 @ S o f t wa r eTe s t Pr o S o f t wa r eTe s t Pr o f e s s i o n a l s “ for STP ” BRAND Page 8 New A new look, new names, and an E X P A N D E D V I S I O N. 10 The Omega Tester Contributors Whether it’s cost cutting or a headlong rush to “agile” development, a lot of test teams are understaffed. Let me put it this way, if your test team is not understaffed then it soon will be… 14 Performance Testing within the Agile Methodologies Performance testing is an important component of the overall software development process, owing to the fact that the final indication of success is user satisfaction and how well the system responds under load. 20 Six Sigma Part IV: Improvement in JamesBACH GorankaBJEDOV Independent Test Advocate, Risk Analysis & Assessment, ImadMOULINE CTO, Gomez, Web Division Software Development Author, Buccaneer-Scholar Product Testing, Google, Inc. of Compuware We can improve software development in two primary ways: software itself (and, possibly, the testing) or the development process. 28 Raising the Bar on Quality Web Performance As a QA/testing professional, you are responsible for protecting your organization’s brand, customer experiences and online revenue by finding and resolving problems before your Web end-users are impacted. 32 Exploratory Performance Testing Have you ever found yourself in a situation where you are organizing ThomasO’MARA KimPRIES JonQUIGLEY functional testing of a product, when suddenly, somebody comes up and Founder of Director of Product Integrity Manager of Verification and asks about performance? TEOInnovations, Inc. & Reliability, Stoneridge, Inc. Test Group, Volvo Truck4 /
  4. 4. N E W M E M B ER S H I P M O D E L N E W E V E N T FO C U S N E W T R A I N I N G O FFER I N G MAGAZINE NEW WEBSITE W e know that the role of testing is critical to providing a quality product, but the focus on quality is bigger than simply testing software. We want to include more content that helps drive the conversation toward improving JANUARY 2010 communications between teams, increase awareness of the impact quality has on costs of development, and be a source of information organizations need to support software testing as not just a function, but a strategy in the ultimate goal of delivering quality software to the end user. Our vision for the “new” magazine will be to improve the reputation of MARCH 2010 software development by highlighting the importance of testing and quality within the software development process.What has happened to We plan to continue offering valuable content for the software tester by providing articles that help software testers better understand how, why, and what to test. In addition, we also want to provide content that supports the importance of the tester’s role in the quality process, and how “quality” is a critical component of software development. We will be listening to our community and closely monitoring the feedback we receive regarding each edition. “Pro” Members of the Software Test Professionals association will have anThe former magazine Software Test & Performance area specifically designed to discuss eachis now Software Test & Quality Assurance issue and make suggestions for improving the magazine going forward. You can findMagazine. This is only one of the changes taking place that discussion by joining the “ST&QAhere at the association of Software Test Professionals (STP). Crew”. Basic Members can forward theirThe overall strategy of the new team at STP is to create a full-service comments directly to me by submitting aassociation which will not only include publishing the magazine, comment to editor@softwaretestpro.combut will incorporate a new membership model, new event focus, andeventually a new training offering to round out the services needed.This model is supported by a new website that incorporates the bestaspects of social media platforms. Print subscriptions are free to STPSo why did we feel it was necessary to change the name of the Pro Members. Digital downloads ofmagazine to support this new strategy? There were a number of the current editions of the magazinereasons but the most important reason was the concept that a name are included in the Basic Communityshould be a reflection of the publication’s vision. By adding “Quality Membership. We also want to promoteAssurance” (QA) we wanted to convey the message that we will the community by publishing articlesincorporate more content on the overall process of quality as opposed from community members. You canto the specific practice of performance testing. Performance testing will submit articles on the new websitecontinue to be a focus (two major articles in this edition) but by adding under the Community tab.QA we can expand the conversation to include the role of testing in the (click “contribute”)overall process of the software development life cycle. Volume 7 / Issue 5 / June 2010 5
  5. 5. SOFTWARETEST | PROFESSIONALS Letter from the Editor I f you have ever had the opportunity to make a living at something you love, you know how fortunate you feel. Too many people go to a job every day instead of building a career. For the last eight years I had the opportunity to build a career around what I love; helping people thrive. The most rewarding aspect of being involved with membership is the people that join an association believe what they do is more than a job. Members of associations are the cream of the crop in any industry. They understand the value of belonging to an organized group of peers willing to help each other. This edition of Software Test & Quality Assurance is our introduction to you, our community. We have been busy building the infrastructure to deliver the services needed to become the most respected association in our industry. The STP team brings to you over 100 years of association experience, and we have built our careers delivering the highest quality experience to our members. We invite you to join us on this journey to excellence. Many of you have already embraced us and we thank you. Please feel free to reach out to any “ W E A R E C R E AT I N G one of us with ideas, thoughts, or comments. You will find the team scattered throughout this edition or on our new website at T H E AS SO CI ATIO N W HICH We have included a number of articles outlining the new association, W I L L B R I N G PA S S I O N AT E name changes, and the difference between social media and associations. Articles include: P EO P L E TO G E T H E R TO I N F LU E N C E T H E n Magazine Name Change: The focus of the new bi-monthly magazine, Software Test and Quality Assurance, is on testing but it also focuses on I N D U ST RY ’ S FU T U R E .” how testing interacts with other functions within the software development life cycle and it expands the conversation into software development. RICH HAND n New Association Name/Logo: Our new name - Software Test Professionals (STP) reflects the vision of the testing industry; provides timely, valuable industry Editor-in Chief: Rich Hand information as well as encourages member communication and collaboration. n New Membership Structure: By joining an association with a thumb on Editor: Yvonne Hamilton the pulse of the software testing industry, your membership will offer you the development tools, leadership opportunities and resources to advance Creative Director: Scott Hanson your career. Association membership truly is a valuable differentiator in today’s competitive job market. Media Sales: Cheri Bruno n New Association Website: You will find valuable member resources offering industry specific information as well as opportunities to participate How to reach ST&QA including joining or leading a crew on topical discussions, attending Letters to the editor may be sent to: training classes or industry events and more. While reading this magazine, you will be introduced to the website About Software Test Professionals (STP) developers who created the new association website. You’ll learn about the STP provides more than 50,000 software development and testing process they implored as well as a new interactive professionals with information, education, training feature of the website – Crews – the best of social media for the software test and professional networking opportunities. professional. You will learn more about the membership structure as well 724 S Tejon Suite C • Colorado Springs CO • 80903 719.476.0301 as industry focused articles such as: “The Omega Tester” by James Bach, “Performance Testing within the Agile Methodologies” by Thomas O’Mara About ST&QA “Six Sigma Part IV: Improvement in Software Development” by Jon Quigley ST&QA is published bi-monthly by STP (Jan, Mar, May, July, Sept, Nov). It is a printed and Kim Pries, “Raising the Bar on Quality Web Performance” by Imad magazine geared towards the software test and Mouline, and “Exploratory Performance Testing” by Goranka Bjedov. Don’t quality assurance professional. Subscription miss our new features, “Ask The Tester” hosted by Matt Heuesser with guest to ST&QA is an STP Pro Membership benefit. expert Jon Bach, and the “Cartoon Tester” created by Andy Glover. If you would like to subscribe to ST&QA, visit and join the Pro Membership. I am truly excited to introduce all the new changes at STP to you. I am Back Issues of ST&P confident we are creating THE association which will bring passionate Pro Members may access electronic back people together to influence the industry’s future. issues at Your feedback is always welcome. Email me personally and let me know Editorial Submissions what you think of the new changes at STP: Visit to learn more about submitting editorial to ST&QA.6 / RichHAND
  6. 6. WHERE TEST LEADERSHIP, MANAGEMENT & STRATEGY CONVERGE CONFERENCE & EXPO 2010 / 877.257.9531 Conference Discounts – Save up to $500 OCTOBER EARLY, EARLY BIRD: REGISTER BY AUGUST 20 19-21, 2010 to receive $400 off any full conference package FINAL EARLY BIRD: REGISTER BY SEPTEMBER 15 to receive $200 off any full conference package Las Vegas STP PRO MEMBERSHIP: Join now to get $100 savings off any full conference package Microsoft “Father of Agile Development” MITRE ROSS SMITH KELLI VRLA KENT BECK MARK SCHARENBROICH ROBERT MARTINProductivity Games Improve Top Ten Leadership Skills: Surviving G Forces NICE BIKE: 2010’s Top 25 Most Dangerous Software Quality & Thriving in the 21st Century in the Organization Fueling Performance with Passion Application Security Weaknesses Special Introduction by Microsoft’s first salaried employee, MARC McDONALD Senior Software Development Engineer, Microsoft Corporation
  7. 7. Meet the Team 1 2 3 BRAND Peggy LIBBEY 1 CEO, NEW for STPOver 30 years of experience in finance and executivemanagement. Peggy is the leader and respected arbiterof all conflicts between team members. She is the pivotpoint of our organization and enjoys being behind thescenes instead of on the “stage”, although she is greatwhen she needs to be front and center. Katie LUTZE 2 Director of Financeklutze@softwaretestpro.comAs talented as a director at Circ De Soleil, Katie canmake an Excel spreadsheet perform in some of themost interesting calculations. She is a brilliant financialanalyst which has made her a valued business analystwhen we are developing strategy. 3 Director ofHENDERSON Fiona Eventsfhenderson@softwaretestpro.comAs a certified meeting planner, Fiona has earned thereputation for creating, managing, and implementing themost coveted events in the IT industry. She prides herselfin delivering the best conference experience for everyattendee. She is most often seen in the back of a generalsession room calling the shots on headset. Don’t let hersmile fool you; she is usually managing chaos behindthe curtain.8 /
  8. 8. NEWFOR | STPT here is a new team at the helm who has are always evolving, and our profession is a combined 100 years of experience in continually evolving. Whether it is new testing tools association management. We are looking and technology, new methods and practices, or forward to becoming the place where improved practices in programming, we need to stay software test professionals will come to in tune with the changes in our industry; and thereshare, network, learn, and participate in an effort are changes as a professional body we may want toto expand the role and the reputation of software promote in the industry as well. By expanding ourquality and testing. The new model is dependent on name, we describe the expanded scope of what wethe professionals in the industry buying into our hope to accomplish. We believe the potential we haveconcept and trusting us to represent them when people as a community to impact our industry is our newsletters and magazines, or attend our The potential lies in the community’s participation andevents and training. Our success doesn’t rely on the the contributions you make to help drive the directionpeople at STP being subject matter experts; it relies on of the industry. We want to put you in the “driver’ssubject matter experts utilizing STP to build a wealth seat” and amplify the voice of software testing andof knowledge and expertise through participation in quality assurance professionals.the association, and the association promoting the As a new organization, we will need to gain your trust,professionals within the community. respect, and confidence. We understand that and willSo what’s in a name? The new name, Software Test continue to reach out and invite you to participate onProfessionals, is about people. We are focused on the an advisory board, be a presenter at a conference orpeople responsible for software testing and managers webinar, and give you the opportunity to showcaseresponsible for quality assurance. Although the your expertise to the industry. We are here to improvesoftware test professional is the focus, there are many your professional career which, in turn, improves thefunctions performed in the Software Development Life association’s reputation. It is a symbiotic relationshipCycle that depend on each other for success. built on mutual trust, and vendor neutrality.THE ORGANIZATION FORMERLY KNOWN AS SOFTWARE TEST& PERFORMANCE COLLABORATIVE, AND THE MAGAZINE KNOWNAS SOFTWARE TEST & PERFORMANCE MAGAZINE HAVE A NEWLOOK, NEW NAMES, AND AN EXPANDED VISION. THE ACRONYMSTP HAS REMAINED THE SAME, BUT THE WORDS ANDVISION ARE NEW.There are many topics and functions that are specific We do not endorse products but we engage all vendorsto each discipline, but there are also many reasons to be thought leaders and to sponsor activities withinwhy software testers, quality assurance managers, the community. We provide a resource about all of thesystem architects, business analysts, project pertinent information and available technology to helpmanagers, and programmers have shared interests in the individual professional reach the best decision fora topic. We plan to address both the specific discipline their organization. No one product or methodology isand the bigger picture, and that is why we changed right for every organization, and we are here to servethe magazine’s name. Second, we wanted to highlight the entire industry, not just a particular segment. Wethe “new and improved” concept, but you will be the have found neutrality to be the best way to build trustultimate judge on whether or not we have improved. and credibility.We look forward to the challenge! You are invited to join the membership. We believeThe magazine formerly known as Software Test & the information, services, and networking opportunitiesPerformance is now Software Test & Quality Assurance. we provide will appeal to everyone in our industry.We received community feedback and based on the The new association is very inexpensive and the moneyfeedback we decided that this new name was best invested in our membership ends up supporting thesuited to represent the changes we are hoping to make. services provided to the membership. We hope youWe appreciate all of the feedback we received from consider Software Test Professionals a good investment.the community, and will continue to reach out as thisnew organization develops. The most rewarding aspectof managing a true association is that it is alwaysevolving. It is always evolving because we as people Volume 7 / Issue 5 / June 2010 9
  9. 9. OMEGA TESTER: TESTING WITH A TEAM OF ONE IN THE MOVIE “OMEGA MAN”, CHARLTON HESTON PLAYED A SCIENTIST- WARRIOR LEFT ALIVE, AND ALONE, AFTER A PLAGUE KILLS EVERYBODY ON THE PLANET. WELL, HE’S NOT TOTALLY ALONE. HE ALSO HAS TO FIGHT A LOT OF MUTANT ZOMBIES. THAT BRINGS ME TO TESTING. Whether it’s cost cutting or a headlong rush to “agile” development, a lot of test teams are understaffed these days. Let me put it this way, if your test team is not understaffed, then it soon will be. Many of the testers I meet are the only tester on their projects. Like Charlton Heston in the movie, the “omega tester” must be canny By JamesBACH and resourceful. Hey, and even on larger teams, your teammates may be working in different areas of the product, or your team may be spread out all over the world. You may feel alone, even on a big project.1 HERE ARE THE GENERAL ISSUES WITH BEING AN OMEGA TESTER: Anything you do means something else won’t get done. (Don’t tell me you’re going to “multitask.” Multitasking is for managers and other people who don’t do anything mentally demanding.) The moment you sit down and really focus on testing something, every other part of the product, every communication, every other preparation or learning must be put on hold. While you are testing, you are a single-threaded process, and the queue grows quickly while your head is down in your work.2 You feel too busy to plan and prepare. There may be manuals to read, developers and users to interview, test tools and virtual test platforms to configure. You need to learn enough about the product to imagine the risks and get a picture of your test strategy. On top of that, there are lots of meetings you could and probably should go to. But as soon as the product is in your hands, the pressure will be on you to test now, now, now! a database and sends it to you for testing. These days that’s not a lot of work – for HIM; for you – welcome to the House of Pain. You immediately have the problem of creating a potentially large and sophisticated test data set. Sure, you can skip that step and just add a few records to test with. That would be a start; but tell me – in what future sprint will your job be to do the deep data testing? They will always add something new, and you will always be expected to jump on it. Developers “Hey wait,” you say, “I’m on a Scrum project, we create functionality, but they also automatically take it one sprint at a time.” Yes, it would be nice create performance problems, scalability problems, if your test preparation activities mapped to the reliability problems, usability problems, etc. They development process. But there’s no particular create these problems for free (I’ve never seen “create reason why they should. Testing and development bugs” on a development schedule, yet there always are not symmetrical activities. A small amount of seem to be enough time to create them), but it isn’t development can create a huge test problem. For free for you to find them. instance, imagine that a developer slaps together10 /
  10. 10. TEAMOF | ONE3 You feel too busy to mind your infrastructure. As you test, you learn. You create files and take crude notes. Your desktop becomes littered with these things. This represents a kind of technical debt that grows harder to support as the project moves along. You need time to stop testing and organize your notes, consolidate and organize your test data, create a coverage outline or update your risk lists. A good infrastructure for your test project gives you efficiency, but you may feel that efficiency is a luxury you can’t afford. Indeed, how many firefighters, in the midst of a disaster, decide to change the oil in the fire truck?4 You feel too busy for curiosity. Testing is a search process. Not only that, it’s a heuristic search. That means there is no sure way to find what we are looking for, and no optimal way of searching for it. There are certain obvious kinds of testing to do, but that’s not enough for excellent testing. Excellent testing requires that you test more than the obvious things in the obvious ways. You must indulge your curiosity. You must play. You must open yourself to the unexpected. For many testers, this can be hard to justify (a lot of my teaching and coaching focuses on solving that problem). It looks like you are goofing off, instead of doing important work. So, when the pressure is on, curiosity is easy to put on the shelf. You then become a little more like a fancy robot; a little less like a powerful human tester.5 No one understands what you do. Programmers and testers both do work that is intangible. The difference is that programmers have a tangible finale: the software. The end product of testing is less tangible, and efforts to make it more tangible mean spending less time actually testing. Many people think testing is easy (“just try it and see if it works!”) and they wonder why it takes you so long to find those “obvious” bugs. At least if you are in a team of testers, you can commiserate and share the burden of explaining the impossibility of complete testing for the 37th time. As an omega tester, you just feel surrounded by monsters. What to do about this? My omega tester suggestions consist of technical ideas and social ideas. Technical Ideas: Forget test cases and scripts. Let risk be your compass. “Gone Bug Finding”. Pick a Think test activities and the A common test strategy practice risk or an area of the product testing story. I don’t think of is to organize testing around a and test it. Record your testing myself as creating test cases standard, such as a specification charter – a sentence or two when I work, I think of myself as or requirements document. about the objective of the doing testing. Sometimes that Those things are important session. This is important results in specific documentation but as an organizing principle because as a lone tester you or some other sort of test artifact, I suggest risk instead. By that need to be more definite about but not necessarily. Test cases I mean focusing your testing the progress you make. It’s too are not testing. Configuring the on the more important product easy to let email or meetings product, operating it, observing it, areas and looking for the more drain all your time away and and evaluating it – that’s testing. important kinds of bugs. At first find you’ve gone a whole day Because you’re overloaded, I you may not have a strong idea without really testing anything. suggest thinking about your about what the risks are, but Use unexpected lulls for strategy in terms of what you keep asking yourself “what are infrastructure, preparation, or need to do, or what you need to the things we are most worried re-assessment. These lulls will create in order to do it. A detailed about NOW?” and as the project occur more or less regularly on documented test procedure is goes on you will learn this better. any project. I’m talking about usually neither of those things. Test in short, sharp, those times when you’re waiting Focus on the activity of testing, uninterrupted sessions. for a new build because they can’t or think about the test report Session based testing means get the thing installed properly on you are trying to produce (it may that you set aside a block of time your test server, or any other time be an oral report), and you’ll get – I prefer 90 minute blocks – put when your testing is blocked. You more testing done. a sign up at your desk that says need to take those opportunities Volume 7 / Issue 5 / June 2010 11
  11. 11. to look over your strategy, try to think ahead, or pull quality.” You should challenge that thinking right from your notes and files together into a better order. the start. Several articles could be written about how to do that. It will have to suffice for me to say that there Use concise test documentation. I’m talking about is no theory of testing and no tool of testing capable of lists, outlines, or tables. I like to use a wiki for this these demonstrating that a product is bug free. There is no days (I wrote this using SocialText). Or Google Docs set of such tools, theories, techniques that can do that. works, too. Something online, shareable, that has a way There is no way at all to do that. All you can promise to roll back history. When I say concise I mean write is to look for problems in thoughtful, reasonable ways. what you need and no more than that. Think tables and And forget about ensuring quality. You support the lists and outlines. Avoid writing detailed instructions. team in producing a good product. You are part of the Don’t write things that are easily handled by training. process of making a good product, but you are not Unless you are testing Microsoft Word, writing is not the one who controls quality. The chief executive of testing, and for darn sure formatting is not testing. the project is the only one who can claim to have that Use recording tools to shorten bug investigation power (and he doesn’t really control quality either, but time. They say that scripted testing is better than don’t tell him that). exploratory testing when it comes to bug investigation. Co-locate. It may not be in your power to huddle This is completely untrue. Having a list of steps that around the same desks as the programmers, but baby, you produced two months ago is no more helpful than get as close as you can. It makes a profound difference a list you made as you went along, except you are two in your productivity if you are within ear and eye range months wiser today, and the notes you record today of the people creating the product. My feeling is that are fresh in your mind. Of course, even better is if you it doubles my productivity – at least – to work ten feet don’t write down your steps at all – because they were away versus ten miles away, and quadruples it versus captured automatically by your recording tool. There being ten time zones away. are various commercial and open source tools you can use for this. I like BBTestAssistant, TestExplorer, Practice, practice, practice explaining testing. or Spector. But there’s also Wink, which is free. Also It’s not enough to know how to test. If you are in remember there are log files, especially if you are a team of one, congratulations, you now have to be testing something online. Or use a digital camera. ready to explain testing, defend testing, and teach testing. Over and over. To the same people, too, Do lots of exploratory scenario testing. because they will keep forgetting what you tell them. This means that you test according to realistic and One way to practice is to blog about testing, or write complex sequences of user behavior organized around about it in a forum like the Context-Driven forum a coherent scenario. The major downside of it is that ( to do it well requires that you get real users to help with it or that you have a deep understanding of real Advocate for testability. Testing and development are users, how they think, and what they will do with not symmetrical. A small decision made by a designer, your product. Exploratory scenario testing would be making little work for him, can be a whole lotta work the same as beta testing if beta testing were highly for you. You’ve got to get in there, gently, and help organized, focused on challenging situations rather them understand the testing impact of what they than just everyday situations, and done by people add or change. Don’t expect anyone to know about passionate to find bugs. testability unless you tell them what that is. Basically, it boils down to two big things: A) can you see it? B) can Now for the social ideas. Unlike Charlton Heston, you control it? That’s why I push for rich log files and who was being chased by zombies most of the time in a scriptable interface (I’m happy to test through a GUI, that movie, you need social skills to be a really good but I also want to get underneath the GUI to control omega tester. It’s your social skills that will save you. the product better and see it better). Social Ideas: Advocate for strong pre-testing. I don’t worry about this so much when I’m working with a good test team. Declare high service aspirations. For instance, It can be hard to get developers to work methodically. when I work with a development team, I give them a It is especially hard to push for that without sounding one page declaration of all the things I promise to do like a nag. But if you are a one-person team, you don’t for them. It documents twelve commitments I make have much choice. You need to sit down and level to them as a tester. I don’t want them to see me as a with them: unless they do unit testing and paired nagging burden. I want to ease their burden, not add programming, or something similarly effective at clearing to it. I will serve my team in any way I can to achieve out the knucklehead bugs before it comes to you, then that. By making my commitment clear, I find that I you are going to be overwhelmed with bugs to find, foster a similar attitude in return. That makes my job and that will slow you down. You’ll be like one of those easier, because their help is vital for me to do my job contestants in a “money booth” trying to grab flying cash well. The success of the following, begins with this one. while it whirlwinds all over the place. If you like sports Set low technical expectations. “Yay, we hired a metaphors, tell them you want be a good goalie for the tester! Now we’ll ship without bugs!” People will think team, but even you can’t defend against multiple shots testing is easy. People will think your job is to “ensure on goal all at the same time.12 /
  12. 12. Avoid bureaucratic bug reporting protocols. Bug Who’s the Real Zombie?investigation and reporting take up a lot of your time. If you do a good job being an omega tester, you’ll findThe more time it takes, the less time you have to find that the other people on the team, over time (it may takenew bugs. But if you test alongside the programmer several projects), will come to respect and rely upon you.whose code it is, you’ll find that reproducing the bug Otherwise, especially if you fall into a formulaic, uninspired,and writing a report become mostly unnecessary. or complainy approach to your work, the project team mayHe sees what you’re doing and immediately reacts come to see you as the mutant zombie in their midst. Oneto it. I’ve found this to be a wonderful way to test sign of this is that they run away when you shuffle towardsomething in its early stages. I’ve also experienced them with your arms stiffly out as if groping for brain mattera project where my testers were required by our or “unambiguous requirements.” Don’t be like that, okay?client to use a cumbersome bug tracking system thatreminded us of filing tax forms. We estimated that welost 30 minutes of test time for each bug we reported,over and above the time it took us to actuallyinvestigate and describe the steps to reproduce.On top of that, we were not allowed to talk to the About The Authorprogrammers. We didn’t work near them, never metthem, or heard from them. It was an outrageous James Bach: If Devil’s Advocate were a televised sport, James would play itwaste of time. If you are a team of one, tell your professionally. A conscientious objector to high school, he skipped school andmanagement that you need to minimize your report taught himself programming. Apple Computer tried him out as test manager,grinding and maximize your bug finding. and he never looked back. He worked in Silicon Valley for nearly ten years, now independent and one of the most outspoken advocates of skilled sapient testing. James Co-authored “Lessons Learned in Software Testing”, and wroteMy Favorite Advice for Omega Testers... “Secrets of a Buccaneer-Scholar”.If you are an omega tester, feeling surroundedby creatures who have tester-like qualities (arms,legs, human DNA, etc.) and yet are not like you,do not be dismayed. Befriend them. Get them tohelp you. Here’s the secret: people will help youtest if you make it easy and fun for them. Let’scall these people beta testers.Beta testers can help you test, but don’t givethem any paperwork to do. Blech. I was once abeta tester on an early PDA (one of the very earliestever, which was called the Magic Link or somethinglike that). I noticed only after having the device fora couple of weeks that Sony had expected me tofill out a minute-by-minute log of exactly what Idid with their product. Ha ha ha. No, I justdoodled in some fake data and returned it.Many things are expected of a true professionaltester. You should be able to report and justifyyour test coverage and your oracles. You need tounderstand risks and use test tools. But pushnone of that onto your beta testers. Instead,create a carnival atmosphere of old fashionedbug huntin’. If I can get my tester helpers allin the same room for a testing event, I makesure they are fed. I pair them up and give eachpair a bell to ring when they think they found aproblem. Beta testers are motivated by the respectthey receive and the feeling of being listened to.I’ve also had very good experiences with bugbounties – paying people who are not on the testteam for each critical bug that they find. You cando this with actual cash money, or do it with playmoney that can be used to purchase items at anauction you hold at the end of the project (thelatter method allows you to completely controlthe budget). Volume 7 / Issue 5 / June 2010 13
  13. 13. By ThomasO’MARA Performance testing is an importantPERFORMANCE component of the overall software development process, owing to the fact that the final indication ofTESTING success is user satisfaction and how well the system responds under load. Since agile development and testing have been around for a while and being used by development and testing teams successfully, aWITHIN THE focus on performance testing early in the developmentAGILE cycle needs a closer examination of how much of an impact it will have on the final outcome.METHODOLOGIES What and Why of Agile Methodologies The Software Development Life Cycle (SDLC) is the venerable overarching structure that every development initiative has usedSoftware engineering and the programs since the beginning to manage the overall development process,generated from such initiatives have indeed but it too is evolving rapidly into a series of hybrid approacheschanged the way business is conducted in put together to satisfy a particular need. Even more now than ever the SDLC is forcibly being modified due to external marketevery enterprise large or small spanning the pressures and changes internally as the needs and requirementsglobe. Furthermore, the way software is being of the business units change and become more complex.constructed and managed today is rapidly Before agile development and testing such as: Test Drivenchanging as the environments, development Development (TDD), Scrum, or Extreme Programming (XP), the most widely used methodology is the waterfall method. Thistools, and methodologies evolve to fit the method is based on the building construction metaphor andunique needs of every organization. employs a sequential approach to software development. The waterfall approach delayed testing until after the application was constructed and only then was the code set transferred to the testing team. As you can see, performance testing was most likely last on the list to be performed, if ever. With the agile approach, both the development and testing teams are engaged in the early stages of construction and all processes are done in parallel to meet the short term cycle or sprint. This will ensure an accelerated return on investment (ROI), value delivered to the business unit early on, and a higher quality product that closely matches the original intent. Still, with all of the new technology and methodologies available, system performance is a key indication of success. Therefore, a closer look at fundamental aspects of performance testing is needed. Characteristics of Agile Performance Testing Performance testing with its ability to quantify the application’s response and throughput is an integral part of any software development testing initiative. Unfortunately it can be an afterthought, or employed only if the application does not respond to the user’s expectation. To use performance testing effectively in the rapid cycle development process, the developers and testers need to adopt an agile way of thinking and quickly put these ideas to work early on. Some of the more key points of performance testing are:14 /
  14. 14. AGILE | METHODOLOGIESPerformance Objectives of development. This brings us to the THE MAJORITY OFThe service level objective or agreement essence of agile testing: testing at the(i.e. response time, throughput, the component level. ENTERPRISE LEVELnumber of concurrent users, andadditional factors that contribute to SOFTWARE DEVELOPMENTthe user’s experience) is between the Component Level Performance Testing PROJECTS EMPLOY THEIT managers and the stake holdersand needs to be defined in the early Within the agile environment, the 3-TIER ARCHITECTURE TOstages of requirements gathering. service and persistence layers of 3-tierThis will establish a clear set of architectures are normally constructed SOME EXTENT, SO TESTINGguidelines on system architecture, first to meet the wire up of the presentationapplication design, and construction. THE VARIOUS LAYERS AT layer. During this phase the data modelTest Data Creation is defined, the data access framework DIFFERENT STAGES OF is chosen, and the domain objects areWhen the initial requirements for the DEVELOPMENT IN EITHER ready to transcend the tiers. That’s whenfirst development cycle are completed, the testing and development teams comethe data model defined, and the service together and define the test scripts needed THE AGILE OR NON-AGILEand persistence layers starting to take to generate the unit tests for test drivenshape, the development team and the ENVIRONMENT IS ESSENTIAL. development and performance unit will need to preparea dummy dataset large enough to Each component of the service andemulate a real world workload. This persistence tier such as SQL statementswill become the baseline starting point and stored procedures, service layer classfor performance testing cycles as the libraries, and web services needs to beapplication matures into a production taken into consideration when defining aready code set. performance testing plan. This will provide the necessary guidance on how to isolateTrend Analysis each component for drill down analysisIn agile development and testing, it is should a resource contention arise, thusimportant to run performance tests ensuring the development team deliversand analyze the data at the end of every a high quality application that will meetcycle or sprint, primarily to see how the its end-to-end performance goals.application design and construction isresponding when the system is undera predefined load. This information is Conclusionparamount in the agile process to provide The majority of enterprise levelthe necessary feedback to meet the software development projects employrequired goals as each of the iterations the 3-tier architecture to some extent,transpire during the development process. so testing the various layers at different stages of development in either the agileDevelopment to Production Life Cycle or non-agile environment is essential.When the application is mature enough However, up to this point, most automatedto yield performance results, testing in performance testing is done via thea real world environment is crucial for a presentation layer or user interface tosuccessful implementation. So it’s good reach the service and persistence layerspractice to make sure all of the required with user data. This can cause delays ifsystems are allocated in the initial phases the user interface is constructed late andof development, so performance testing not wired up to the service layer.on the production level hardware is useful About The Authorin determining what is needed as the Load testing at the component levelapplication grows to its final configuration. without the user interface present is Thomas O’Mara is the founder critical to success of the agile testing of TEOInnovations, Inc. andWhen the performance objectives project. Not only will such testing keep has been building, programming,have been met, the test data is being development on track to meet the sprint and testing computer basedgenerated, and the physical system is goals, but will uncover any potential systems for almost 30 years.being prepared, planning on how to bottlenecks early on to save time, You can reach him ateffectively test the application is critical. money, and resources in the process.Since most of the testing is being done atthe Unit level, only service componentswill be available in the early stages Volume 7 / Issue 5 / June 2010 15
  15. 15. DEVELOPINGA | COMMUNITY Featured Crews Notifications Member Profile16 /
  16. 16. Meet the TeamDEVELOPERS 1Developing a Communitynot just Programming 2a WebsiteWith all of the new offerings atSoftware Test Professionals (STP),we had to highlight our new website, 3and what better way than to talk to thedevelopers responsible for the developmentand coding of the new site. I sat down withSystem Architects Patrick McDavid andCharles Gibson to discuss the new website. 1 Director of IT ROVANSEK JanetteI found it very interesting. Janette has been in IT for 15 years and was responsibleBy RichHAND for the systems at HDI for 8 years. Currently she heads up the team here at STP and is the CRM and one person Help Desk. She is often found smiling next to Abbie at conference registration and training events.I came away from our discussions with a sense of their Patrick McDAVID true understanding and commitment to our business, 2 Systems Architect and thought our audience would appreciate their vision and commitment to this community. They are developers A member of the team responsible for the new STP but they are the kind of developers that are truly empathetic website, Patrick spent 5 years creating and developingto the role of the tester. Their past experience in the association systems for HDI. Known here as P-diddy, he is most often found on the UCCS campus engaged in philosophicalbusiness has really shaped the way they developed the new arguments with his soon to be graduated peers.Software Test Professionals website. Although when I askedthem about the testing process, the reaction was eye-opening. Charles GIBSON 3 Systems Architect cgibson@softwaretestpro.comWhy a New Website? A member of the systems team, a creature of agile,We already had a website, so it is a fundamental question to raise – and all around cool guy, Charles has spent 8 years in software development. A former HDI member, he canwhy create a new website? What was the need? most often be found instructing Krav Maga. No needThe previous management team here at STP outsourced the to worry about security at STP.responsibility of the website to a third party, and they coded in Rubyon Rails. Charles and Patrick felt that in the interest of time, andthe ability to have greater flexibility and agility, it would be better Volume 7 / Issue 5 / June 2010 17
  17. 17. to consider coding in a language best in our environment.” Patrick and occasionally arguing access they both had previously worked agreed and stated “The process was to the functionality these guys with. Patrick stated, “We had the very informal working with a beta were on track with the business opportunity to beta Microsoft’s new software; changing features on the expectations. In our discussion, .Net 4.0, which was a very attractive fly; finding new stuff we could use Patrick and Charles firmly agreed opportunity for us.” So the decision all made it more of a creation than as stated by Patrick, “the website was made to use .Net 4.0. According strictly coding from requirements”. is really just the beginning of a to Patrick and Charles, it was Their methodology seemed to work continual development process that exciting and a bit of a risk to use but you – the community – will be will evolve with the membership a beta version of Microsoft’s newest the ultimate judge. and community.” They not only offering. They both agreed that it understand that development will How much direction did you get was really cool to start from scratch continue, they are encouraging from the business owners? They with no legacy to worry about. feedback by having a “Crew” both agreed the initial meetings with for people to interact with them The development team had a the business owners were helpful to suggest future features and blank canvas to work with when but unnecessary. They said that functionality. Charles added, “I they embarked on the project. with tongue in cheek but I was not can’t wait to see how members “Having built communities in the that sure. Having worked with them use the site so we can enhance past, the new language gave us at our last association they were well the areas they really want, and the flexibility we needed to create known for creating website features an interactive website for the on their own. Charles added, “We develop features we have not even considered yet based on their membership, which in the end knew in our heads what worked and feedback”. I truly saw excitement in was the driving factor that helped what didn’t at the last association so their faces when they talked about us decide to start from scratch,” we wanted to make sure we included software testers actually providing both Patrick and Charles agreed. the capabilities members utilized feedback that they can take and often and minimize the time spent Patrick brought up the point, “Beta turn into reality. Between the on things members rarely used.” is very interesting to use because it lack of legacy, and the agility and They both know the business and gives you the new bells and whistles flexibility of their infrastructure, agreed that their experience was but it comes with risk.” Looking at they believe the changes will come the key resource used to develop the new website it definitely looks fast and furious if the community the site. I think they welcomed the like it was worth it! wants them. As the membership feedback from the business team What “process” or development guy here, this is music to my ears! to keep them from “wandering” into methodology did you use to the realm of cool that had the risk What is the feature you are most create the new website? This of eating up precious time. excited about for our membership question generated a chuckle, and community? Hands down, For development teams in large laugh, sigh, and a long pause both Patrick and Charles believe the shops with stringent requirements before Charles spoke up and said ease of sharing and contributing and process this may seem “Informal Agile.” Charles had content is the foundation of what we unstructured but as a business spent a number of years in more needed to provide for a professional stakeholder, one of the reasons developed Agile shops so he was community. These guys have a I think it worked was Patrick’s cautious to clarify that it was a natural instinct for associations and Charles’ strong knowledge little bit of paired coding, iterative and know that the association of the business of associations development, loose design and can only prosper if the members and community engagement. plan activities, but mostly loose participate. Patrick referred to methods. Charles stated, “We did Having been part of the status the ability to share content as the a lot of things Agile suggests, we meetings, I was confident after “thought leadership promotion tool.” just did it in a way that worked reviewing, listening, This content sharing will build the Badges Featured Articles Industry Tweets ST&QA Magazine Online18 /
  18. 18. DEVELOPINGA | COMMUNITYmembership’s foundation of knowledge along with in their mind the things they needed to revisit in an effortindividual member reputations. Charles and Patrick to have a better quality product. They were consideringhave built in a reputation model that awards points things they may not have, if their work wasn’t going to beto “activities” that provide value to the community. tested by a large group of testers from our community. TheirWhen a member reaches a certain number of points conversation was respectful but they were clearly anxiousthey achieve trophies but the real value is the and concerned about the bugs that could be found. Theymember’s profile is updated to let others know the would not rule out the possibility of testers finding problemsstatus of their membership reputation. The more a that could delay launch. They were in no way willing to saymember participates and contributes, and if what with any degree of confidence that the launch would takethey contribute garners good feedback, individuals place on the projected date. It was partly out of modesty butreceive additional privileges and status within the also a respect for the possibilities that having testers combingmembership. Charles and Patrick are both looking the website, this community may find flaws that would bringforward to the feedback regarding this feature. to the forefront issues that could delay the project.The other feature Charles and Patrick get excited They know they are developing a site for the most criticalabout is the Crew feature. Crews are the foundation audience in the IT industry. They understand the necessityof the membership. Crews are groups of members for testing and they know the site will be under constantthat have similar interests and can collaborate by scrutiny from the membership. It is a relationship they hopesharing content, video, podcasts, and comments on to develop, and are looking forward to the scrutiny becausetopics of interest. Pro Members can create a crew in the end, the quality of this website will have to be theand can be part of multiple crews. best. If we have thousands of testers on the site - utilizing it to share; contribute; learn; build relationships; and theThere will be some crews we hope will be extremely membership is willing to provide us with the feedback - ourvaluable such as the “Ask the Tester” - a crew where developers will have created the website all of us at Softwarequestions will be answered quickly by an industry Test Professionals could only dream about.leader as well as others within the crew. We alsohave a crew for people interested in getting publishedin the magazine or simply talking about articles inthe magazine. The idea of the crew was summed upby Charles “members can share, develop, and utilizecrews any way they find beneficial to improving their POSITION YOURSELFmembership experience”. Charles is really pumpedabout crews and Patrick is pumped about contentsharing. But I can say with confidence they can’twait for “opening day”.And then I changed this mood of excitement and AS A CENTER OF INFLUENCEspeculation, and exuberance by asking this:Are you ready for the testers we are aboutto unleash on the website? I can only say the Join other senior-level professionals to network,comments between Patrick and Charles started exchange ideas and share information in face- to-face meetings. Meetings include industryflying, “Oh man they are going to hammer us. Theyare going to challenge our word length fields, our…; they continued on as I sat back, listened, andlearned what the role of testing is really about for facilitators, forward-thinking topics and thethe developer. What I heard was both Patrick andCharles listing the areas of code and functionality ability to network with executivesthat might be vulnerable. They were fine tuning facing the same industry issues and challenges. Network at a higher-level. Be Influential. W W W. S O F T WA R E T E S T P R O . C O / October 19-21, 2010 / Las Vegas CONTACT US TODAY! 1.877.257.9531 / Volume 7 / Issue 5 / June 2010 19
  19. 19. By JonQUIGLEY and KimPRIES IMPLEMENTING “ W E CA N I M PROV E TH E SIX SIGMA: SOFTWARE ITSELF (AND, POSSIBLY, THE TESTING) OR WE CAN IMPROVE THE DEVELOPMENT PROCESS.” Improvement in Software Development W e can improve software development in two primary ways: This important tool can be quite we can improve the software itself (and, possibly, the testing) difficult to implement well. While it is or we can improve the development process. We have already easy to list and document the lesson covered the preceding three Six Sigma Phases (Define, Measure, learned, it is not so easy to put this and Analyze) in previous articles. Now we take what we have information into a format that makes learned and use that information to improve our software development. for easy searching and retrieval. It is the retrieval that is important. Improving the Software Itself We have seen lessons learned documented in word processors, Training software developers is one obvious method for improving spreadsheets, data bases, and free software development. All people may be created equal, but not all form storage (for example, Microsoft programmers are equal. However, simply saying ‘training’ isn’t really OneNote). Even with a data base, good enough. We need to consider what our measurement and analysis we need to have a good system of phases have told us about our software issues. One typical suite of training search categories and terms or classes might include the following for a hypothetical embedded software nobody will be able to retrieve the development group (‘Our-Department’): information promptly. If the work to review the lessons becomes a burden, the task will starve away n Introduction to Our-Department n In-circuit emulation and we will have lost an opportunity coding practices and standard to profit from past blunders. If we n Pitfalls in C and how for the C language are using textual information, we to avoid them n Our Department specific can use search engines like Google software handling procedures n Memory mapping the processor Desktop to help us find information; an older approach using ASCII files n Use of static testers n Lessons learned review would involve the use of the grep tool n Use of dynamic testers n Design architectures common on UNIX-types of operating systems. Once retrieval becomes efficient, we have the beginnings of We call it ‘training’ but, to some extent, we are really talking about defect prevention program. We don’t indoctrination into our organization’s practices for software development. think it is inappropriate to make Some of these practices will fall under the heading of lessons learned— the use of the lessons learned data where we have captured failures from the past as well as solutions to those base a part of the software engineer’s failures. It does not do much good to critique failures and understand them performance evaluation. if there is no learning from this review. Yet, in our experience, this is often what really happens. Frequent reviews of our lessons learned file or data In a sense, while we call it training, base by the entire team is not wasted time. We have seen what we call the what we are really talking about is a ‘stupidity loop’—a phenomenon often tied into the turnover rate of developers, continually learning organization 1. especially with senior developers. We can detect the duration of the stupidity This is more than learning from the loop by the regularity with which we see the same mistakes begin to recur. failures. This includes boundary Hence, the reviews of the lessons learned file allow our entire team to learn spanning for best practices, which from the past without necessarily having all the original players present. is not necessarily a euphemism for 1 SENGE, P.M. 1990. The Fifth Discipline. London: Century Business20 /