• Save
ST&QA 05-06 2010
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


ST&QA 05-06 2010

Uploaded on

How to reach ST&QA...

How to reach ST&QA
Letters to the editor may be sent to: editor@softwaretestpro.com

About Software Test Professionals (STP)
STP provides more than 50,000 software professionals with information, education, training
and professional networking opportunities.

724 S Tejon Suite C • Colorado Springs CO • 80903

About ST&QA
ST&QA is published bi-monthly by STP (Jan, Mar, May, July, Sept, Nov). It is a printed magazine geared towards the software test and
quality assurance professional. Subscription to ST&QA is an STP Pro Membership benefit.
If you would like to subscribe to ST&QA, visit softwaretestpro.com and join the Pro Membership.

Editorial Submissions
Visit softwaretestpro.com to learn more about submitting editorial to ST&QA.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 2

http://www.slashdocs.com 2

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. VO LU M E 7 ISSUE 5 J U N E 2 010www.softwaretestpro.com MAGAZINE 10 14 20Omega Tester: Performance Testing Implementing Six Sigma: What HappenedTesting with a Team within the Agile Improvement in Software to Software Test &of One Methodologies Development Performance Magazine? BRAND NEW Page A new look, 8 for STP new names, and an expanded vision!
  • 2. Join & the Power of MEMBERSHIP is in Your HandsMembership Levels:BASIC – (free & limited access) The current digital edition PRO – Full access to the community, resources, Crews,of ST&QA magazine, Test Q&A Report, STP Insider and conference discounts and knowledge sharing opportunitiesselected web content. that matter most to you.EXECUTIVE – The highest level of membership designed What’s a Crew? Crews are online communities of membersfor senior-level executives. Contact us for details. interested in sharing ideas and knowledge with others in the field. Crew benefits include: • Opportunities to interact with peers for concept sharing, learning, debating and networking • Download content created by the industry to help improve the practices at your organization • Discussions of current trends with industry experts • Access to other professionals who can help with idea generation or test case/development ideas Leverage the Power. Join Today! • Networking with professionals in your geographic region www.softwaretestpro.com / 877.257.9531 for online or face-to-face meet-ups • Offer your leadership on subject matter that relates to your area of expertise
  • 3. Volume 7 / Issue 5 @ S o f t wa r eTe s t Pr o S o f t wa r eTe s t Pr o f e s s i o n a l s “ for STP ” BRAND Page 8 New A new look, new names, and an E X P A N D E D V I S I O N. 10 The Omega Tester Contributors Whether it’s cost cutting or a headlong rush to “agile” development, a lot of test teams are understaffed. Let me put it this way, if your test team is not understaffed then it soon will be… 14 Performance Testing within the Agile Methodologies Performance testing is an important component of the overall software development process, owing to the fact that the final indication of success is user satisfaction and how well the system responds under load. 20 Six Sigma Part IV: Improvement in JamesBACH GorankaBJEDOV Independent Test Advocate, Risk Analysis & Assessment, ImadMOULINE CTO, Gomez, Web Division Software Development Author, Buccaneer-Scholar Product Testing, Google, Inc. of Compuware We can improve software development in two primary ways: software itself (and, possibly, the testing) or the development process. 28 Raising the Bar on Quality Web Performance As a QA/testing professional, you are responsible for protecting your organization’s brand, customer experiences and online revenue by finding and resolving problems before your Web end-users are impacted. 32 Exploratory Performance Testing Have you ever found yourself in a situation where you are organizing ThomasO’MARA KimPRIES JonQUIGLEY functional testing of a product, when suddenly, somebody comes up and Founder of Director of Product Integrity Manager of Verification and asks about performance? TEOInnovations, Inc. & Reliability, Stoneridge, Inc. Test Group, Volvo Truck4 / www.softwaretestpro.com
  • 4. N E W M E M B ER S H I P M O D E L N E W E V E N T FO C U S N E W T R A I N I N G O FFER I N G MAGAZINE NEW WEBSITE W e know that the role of testing is critical to providing a quality product, but the focus on quality is bigger than simply testing software. We want to include more content that helps drive the conversation toward improving JANUARY 2010 communications between teams, increase awareness of the impact quality has on costs of development, and be a source of information organizations need to support software testing as not just a function, but a strategy in the ultimate goal of delivering quality software to the end user. Our vision for the “new” magazine will be to improve the reputation of MARCH 2010 software development by highlighting the importance of testing and quality within the software development process.What has happened to We plan to continue offering valuable content for the software tester by providing articles that help software testers better understand how, why, and what to test. In addition, we also want to provide content that supports the importance of the tester’s role in the quality process, and how “quality” is a critical component of software development. We will be listening to our community and closely monitoring the feedback we receive regarding each edition. “Pro” Members of the Software Test Professionals association will have anThe former magazine Software Test & Performance area specifically designed to discuss eachis now Software Test & Quality Assurance issue and make suggestions for improving the magazine going forward. You can findMagazine. This is only one of the changes taking place that discussion by joining the “ST&QAhere at the association of Software Test Professionals (STP). Crew”. Basic Members can forward theirThe overall strategy of the new team at STP is to create a full-service comments directly to me by submitting aassociation which will not only include publishing the magazine, comment to editor@softwaretestpro.combut will incorporate a new membership model, new event focus, andeventually a new training offering to round out the services needed.This model is supported by a new website that incorporates the bestaspects of social media platforms. Print subscriptions are free to STPSo why did we feel it was necessary to change the name of the Pro Members. Digital downloads ofmagazine to support this new strategy? There were a number of the current editions of the magazinereasons but the most important reason was the concept that a name are included in the Basic Communityshould be a reflection of the publication’s vision. By adding “Quality Membership. We also want to promoteAssurance” (QA) we wanted to convey the message that we will the community by publishing articlesincorporate more content on the overall process of quality as opposed from community members. You canto the specific practice of performance testing. Performance testing will submit articles on the new websitecontinue to be a focus (two major articles in this edition) but by adding under the Community tab.QA we can expand the conversation to include the role of testing in the (click “contribute”)overall process of the software development life cycle. Volume 7 / Issue 5 / June 2010 5
  • 5. SOFTWARETEST | PROFESSIONALS Letter from the Editor I f you have ever had the opportunity to make a living at something you love, you know how fortunate you feel. Too many people go to a job every day instead of building a career. For the last eight years I had the opportunity to build a career around what I love; helping people thrive. The most rewarding aspect of being involved with membership is the people that join an association believe what they do is more than a job. Members of associations are the cream of the crop in any industry. They understand the value of belonging to an organized group of peers willing to help each other. This edition of Software Test & Quality Assurance is our introduction to you, our community. We have been busy building the infrastructure to deliver the services needed to become the most respected association in our industry. The STP team brings to you over 100 years of association experience, and we have built our careers delivering the highest quality experience to our members. We invite you to join us on this journey to excellence. Many of you have already embraced us and we thank you. Please feel free to reach out to any “ W E A R E C R E AT I N G one of us with ideas, thoughts, or comments. You will find the team scattered throughout this edition or on our new website at www.softwaretestpro.com. T H E AS SO CI ATIO N W HICH We have included a number of articles outlining the new association, W I L L B R I N G PA S S I O N AT E name changes, and the difference between social media and associations. Articles include: P EO P L E TO G E T H E R TO I N F LU E N C E T H E n Magazine Name Change: The focus of the new bi-monthly magazine, Software Test and Quality Assurance, is on testing but it also focuses on I N D U ST RY ’ S FU T U R E .” how testing interacts with other functions within the software development life cycle and it expands the conversation into software development. RICH HAND n New Association Name/Logo: Our new name - Software Test Professionals (STP) reflects the vision of the testing industry; provides timely, valuable industry Editor-in Chief: Rich Hand information as well as encourages member communication and collaboration. rhand@softwaretestpro.com n New Membership Structure: By joining an association with a thumb on Editor: Yvonne Hamilton Yhamilton@softwaretestpro.com the pulse of the software testing industry, your membership will offer you the development tools, leadership opportunities and resources to advance Creative Director: Scott Hanson your career. Association membership truly is a valuable differentiator in shanson@softwaretestpro.com today’s competitive job market. Media Sales: Cheri Bruno cbruno@softwaretestpro.com n New Association Website: You will find valuable member resources offering industry specific information as well as opportunities to participate How to reach ST&QA including joining or leading a crew on topical discussions, attending Letters to the editor may be sent to: training classes or industry events and more. editor@softwaretestpro.com While reading this magazine, you will be introduced to the website About Software Test Professionals (STP) developers who created the new association website. You’ll learn about the STP provides more than 50,000 software development and testing process they implored as well as a new interactive professionals with information, education, training feature of the website – Crews – the best of social media for the software test and professional networking opportunities. professional. You will learn more about the membership structure as well 724 S Tejon Suite C • Colorado Springs CO • 80903 719.476.0301 as industry focused articles such as: “The Omega Tester” by James Bach, “Performance Testing within the Agile Methodologies” by Thomas O’Mara About ST&QA “Six Sigma Part IV: Improvement in Software Development” by Jon Quigley ST&QA is published bi-monthly by STP (Jan, Mar, May, July, Sept, Nov). It is a printed and Kim Pries, “Raising the Bar on Quality Web Performance” by Imad magazine geared towards the software test and Mouline, and “Exploratory Performance Testing” by Goranka Bjedov. Don’t quality assurance professional. Subscription miss our new features, “Ask The Tester” hosted by Matt Heuesser with guest to ST&QA is an STP Pro Membership benefit. expert Jon Bach, and the “Cartoon Tester” created by Andy Glover. If you would like to subscribe to ST&QA, visit softwaretestpro.com and join the Pro Membership. I am truly excited to introduce all the new changes at STP to you. I am Back Issues of ST&P confident we are creating THE association which will bring passionate Pro Members may access electronic back people together to influence the industry’s future. issues at www.stqamag.com. Your feedback is always welcome. Email me personally and let me know Editorial Submissions what you think of the new changes at STP: rhand@softwaretestpro.com. Visit softwaretestpro.com to learn more about submitting editorial to ST&QA.6 / www.softwaretestpro.com RichHAND
  • 6. WHERE TEST LEADERSHIP, MANAGEMENT & STRATEGY CONVERGE CONFERENCE & EXPO 2010 www.STPCon.com / 877.257.9531 Conference Discounts – Save up to $500 OCTOBER EARLY, EARLY BIRD: REGISTER BY AUGUST 20 19-21, 2010 to receive $400 off any full conference package FINAL EARLY BIRD: REGISTER BY SEPTEMBER 15 to receive $200 off any full conference package Las Vegas STP PRO MEMBERSHIP: Join now to get $100 savings off any full conference package Microsoft “Father of Agile Development” MITRE ROSS SMITH KELLI VRLA KENT BECK MARK SCHARENBROICH ROBERT MARTINProductivity Games Improve Top Ten Leadership Skills: Surviving G Forces NICE BIKE: 2010’s Top 25 Most Dangerous Software Quality & Thriving in the 21st Century in the Organization Fueling Performance with Passion Application Security Weaknesses Special Introduction by Microsoft’s first salaried employee, MARC McDONALD Senior Software Development Engineer, Microsoft Corporation
  • 7. Meet the Team 1 2 3 BRAND Peggy LIBBEY 1 CEO, Presidentplibbey@softwaretestpro.com NEW for STPOver 30 years of experience in finance and executivemanagement. Peggy is the leader and respected arbiterof all conflicts between team members. She is the pivotpoint of our organization and enjoys being behind thescenes instead of on the “stage”, although she is greatwhen she needs to be front and center. Katie LUTZE 2 Director of Financeklutze@softwaretestpro.comAs talented as a director at Circ De Soleil, Katie canmake an Excel spreadsheet perform in some of themost interesting calculations. She is a brilliant financialanalyst which has made her a valued business analystwhen we are developing strategy. 3 Director ofHENDERSON Fiona Eventsfhenderson@softwaretestpro.comAs a certified meeting planner, Fiona has earned thereputation for creating, managing, and implementing themost coveted events in the IT industry. She prides herselfin delivering the best conference experience for everyattendee. She is most often seen in the back of a generalsession room calling the shots on headset. Don’t let hersmile fool you; she is usually managing chaos behindthe curtain.8 / www.softwaretestpro.com
  • 8. NEWFOR | STPT here is a new team at the helm who has are always evolving, and our profession is a combined 100 years of experience in continually evolving. Whether it is new testing tools association management. We are looking and technology, new methods and practices, or forward to becoming the place where improved practices in programming, we need to stay software test professionals will come to in tune with the changes in our industry; and thereshare, network, learn, and participate in an effort are changes as a professional body we may want toto expand the role and the reputation of software promote in the industry as well. By expanding ourquality and testing. The new model is dependent on name, we describe the expanded scope of what wethe professionals in the industry buying into our hope to accomplish. We believe the potential we haveconcept and trusting us to represent them when people as a community to impact our industry is significant.read our newsletters and magazines, or attend our The potential lies in the community’s participation andevents and training. Our success doesn’t rely on the the contributions you make to help drive the directionpeople at STP being subject matter experts; it relies on of the industry. We want to put you in the “driver’ssubject matter experts utilizing STP to build a wealth seat” and amplify the voice of software testing andof knowledge and expertise through participation in quality assurance professionals.the association, and the association promoting the As a new organization, we will need to gain your trust,professionals within the community. respect, and confidence. We understand that and willSo what’s in a name? The new name, Software Test continue to reach out and invite you to participate onProfessionals, is about people. We are focused on the an advisory board, be a presenter at a conference orpeople responsible for software testing and managers webinar, and give you the opportunity to showcaseresponsible for quality assurance. Although the your expertise to the industry. We are here to improvesoftware test professional is the focus, there are many your professional career which, in turn, improves thefunctions performed in the Software Development Life association’s reputation. It is a symbiotic relationshipCycle that depend on each other for success. built on mutual trust, and vendor neutrality.THE ORGANIZATION FORMERLY KNOWN AS SOFTWARE TEST& PERFORMANCE COLLABORATIVE, AND THE MAGAZINE KNOWNAS SOFTWARE TEST & PERFORMANCE MAGAZINE HAVE A NEWLOOK, NEW NAMES, AND AN EXPANDED VISION. THE ACRONYMSTP HAS REMAINED THE SAME, BUT THE WORDS ANDVISION ARE NEW.There are many topics and functions that are specific We do not endorse products but we engage all vendorsto each discipline, but there are also many reasons to be thought leaders and to sponsor activities withinwhy software testers, quality assurance managers, the community. We provide a resource about all of thesystem architects, business analysts, project pertinent information and available technology to helpmanagers, and programmers have shared interests in the individual professional reach the best decision fora topic. We plan to address both the specific discipline their organization. No one product or methodology isand the bigger picture, and that is why we changed right for every organization, and we are here to servethe magazine’s name. Second, we wanted to highlight the entire industry, not just a particular segment. Wethe “new and improved” concept, but you will be the have found neutrality to be the best way to build trustultimate judge on whether or not we have improved. and credibility.We look forward to the challenge! You are invited to join the membership. We believeThe magazine formerly known as Software Test & the information, services, and networking opportunitiesPerformance is now Software Test & Quality Assurance. we provide will appeal to everyone in our industry.We received community feedback and based on the The new association is very inexpensive and the moneyfeedback we decided that this new name was best invested in our membership ends up supporting thesuited to represent the changes we are hoping to make. services provided to the membership. We hope youWe appreciate all of the feedback we received from consider Software Test Professionals a good investment.the community, and will continue to reach out as thisnew organization develops. The most rewarding aspectof managing a true association is that it is alwaysevolving. It is always evolving because we as people Volume 7 / Issue 5 / June 2010 9
  • 9. OMEGA TESTER: TESTING WITH A TEAM OF ONE IN THE MOVIE “OMEGA MAN”, CHARLTON HESTON PLAYED A SCIENTIST- WARRIOR LEFT ALIVE, AND ALONE, AFTER A PLAGUE KILLS EVERYBODY ON THE PLANET. WELL, HE’S NOT TOTALLY ALONE. HE ALSO HAS TO FIGHT A LOT OF MUTANT ZOMBIES. THAT BRINGS ME TO TESTING. Whether it’s cost cutting or a headlong rush to “agile” development, a lot of test teams are understaffed these days. Let me put it this way, if your test team is not understaffed, then it soon will be. Many of the testers I meet are the only tester on their projects. Like Charlton Heston in the movie, the “omega tester” must be canny By JamesBACH and resourceful. Hey, and even on larger teams, your teammates may be working in different areas of the product, or your team may be spread out all over the world. You may feel alone, even on a big project.1 HERE ARE THE GENERAL ISSUES WITH BEING AN OMEGA TESTER: Anything you do means something else won’t get done. (Don’t tell me you’re going to “multitask.” Multitasking is for managers and other people who don’t do anything mentally demanding.) The moment you sit down and really focus on testing something, every other part of the product, every communication, every other preparation or learning must be put on hold. While you are testing, you are a single-threaded process, and the queue grows quickly while your head is down in your work.2 You feel too busy to plan and prepare. There may be manuals to read, developers and users to interview, test tools and virtual test platforms to configure. You need to learn enough about the product to imagine the risks and get a picture of your test strategy. On top of that, there are lots of meetings you could and probably should go to. But as soon as the product is in your hands, the pressure will be on you to test now, now, now! a database and sends it to you for testing. These days that’s not a lot of work – for HIM; for you – welcome to the House of Pain. You immediately have the problem of creating a potentially large and sophisticated test data set. Sure, you can skip that step and just add a few records to test with. That would be a start; but tell me – in what future sprint will your job be to do the deep data testing? They will always add something new, and you will always be expected to jump on it. Developers “Hey wait,” you say, “I’m on a Scrum project, we create functionality, but they also automatically take it one sprint at a time.” Yes, it would be nice create performance problems, scalability problems, if your test preparation activities mapped to the reliability problems, usability problems, etc. They development process. But there’s no particular create these problems for free (I’ve never seen “create reason why they should. Testing and development bugs” on a development schedule, yet there always are not symmetrical activities. A small amount of seem to be enough time to create them), but it isn’t development can create a huge test problem. For free for you to find them. instance, imagine that a developer slaps together10 / www.softwaretestpro.com
  • 10. TEAMOF | ONE3 You feel too busy to mind your infrastructure. As you test, you learn. You create files and take crude notes. Your desktop becomes littered with these things. This represents a kind of technical debt that grows harder to support as the project moves along. You need time to stop testing and organize your notes, consolidate and organize your test data, create a coverage outline or update your risk lists. A good infrastructure for your test project gives you efficiency, but you may feel that efficiency is a luxury you can’t afford. Indeed, how many firefighters, in the midst of a disaster, decide to change the oil in the fire truck?4 You feel too busy for curiosity. Testing is a search process. Not only that, it’s a heuristic search. That means there is no sure way to find what we are looking for, and no optimal way of searching for it. There are certain obvious kinds of testing to do, but that’s not enough for excellent testing. Excellent testing requires that you test more than the obvious things in the obvious ways. You must indulge your curiosity. You must play. You must open yourself to the unexpected. For many testers, this can be hard to justify (a lot of my teaching and coaching focuses on solving that problem). It looks like you are goofing off, instead of doing important work. So, when the pressure is on, curiosity is easy to put on the shelf. You then become a little more like a fancy robot; a little less like a powerful human tester.5 No one understands what you do. Programmers and testers both do work that is intangible. The difference is that programmers have a tangible finale: the software. The end product of testing is less tangible, and efforts to make it more tangible mean spending less time actually testing. Many people think testing is easy (“just try it and see if it works!”) and they wonder why it takes you so long to find those “obvious” bugs. At least if you are in a team of testers, you can commiserate and share the burden of explaining the impossibility of complete testing for the 37th time. As an omega tester, you just feel surrounded by monsters. What to do about this? My omega tester suggestions consist of technical ideas and social ideas. Technical Ideas: Forget test cases and scripts. Let risk be your compass. “Gone Bug Finding”. Pick a Think test activities and the A common test strategy practice risk or an area of the product testing story. I don’t think of is to organize testing around a and test it. Record your testing myself as creating test cases standard, such as a specification charter – a sentence or two when I work, I think of myself as or requirements document. about the objective of the doing testing. Sometimes that Those things are important session. This is important results in specific documentation but as an organizing principle because as a lone tester you or some other sort of test artifact, I suggest risk instead. By that need to be more definite about but not necessarily. Test cases I mean focusing your testing the progress you make. It’s too are not testing. Configuring the on the more important product easy to let email or meetings product, operating it, observing it, areas and looking for the more drain all your time away and and evaluating it – that’s testing. important kinds of bugs. At first find you’ve gone a whole day Because you’re overloaded, I you may not have a strong idea without really testing anything. suggest thinking about your about what the risks are, but Use unexpected lulls for strategy in terms of what you keep asking yourself “what are infrastructure, preparation, or need to do, or what you need to the things we are most worried re-assessment. These lulls will create in order to do it. A detailed about NOW?” and as the project occur more or less regularly on documented test procedure is goes on you will learn this better. any project. I’m talking about usually neither of those things. Test in short, sharp, those times when you’re waiting Focus on the activity of testing, uninterrupted sessions. for a new build because they can’t or think about the test report Session based testing means get the thing installed properly on you are trying to produce (it may that you set aside a block of time your test server, or any other time be an oral report), and you’ll get – I prefer 90 minute blocks – put when your testing is blocked. You more testing done. a sign up at your desk that says need to take those opportunities Volume 7 / Issue 5 / June 2010 11
  • 11. to look over your strategy, try to think ahead, or pull quality.” You should challenge that thinking right from your notes and files together into a better order. the start. Several articles could be written about how to do that. It will have to suffice for me to say that there Use concise test documentation. I’m talking about is no theory of testing and no tool of testing capable of lists, outlines, or tables. I like to use a wiki for this these demonstrating that a product is bug free. There is no days (I wrote this using SocialText). Or Google Docs set of such tools, theories, techniques that can do that. works, too. Something online, shareable, that has a way There is no way at all to do that. All you can promise to roll back history. When I say concise I mean write is to look for problems in thoughtful, reasonable ways. what you need and no more than that. Think tables and And forget about ensuring quality. You support the lists and outlines. Avoid writing detailed instructions. team in producing a good product. You are part of the Don’t write things that are easily handled by training. process of making a good product, but you are not Unless you are testing Microsoft Word, writing is not the one who controls quality. The chief executive of testing, and for darn sure formatting is not testing. the project is the only one who can claim to have that Use recording tools to shorten bug investigation power (and he doesn’t really control quality either, but time. They say that scripted testing is better than don’t tell him that). exploratory testing when it comes to bug investigation. Co-locate. It may not be in your power to huddle This is completely untrue. Having a list of steps that around the same desks as the programmers, but baby, you produced two months ago is no more helpful than get as close as you can. It makes a profound difference a list you made as you went along, except you are two in your productivity if you are within ear and eye range months wiser today, and the notes you record today of the people creating the product. My feeling is that are fresh in your mind. Of course, even better is if you it doubles my productivity – at least – to work ten feet don’t write down your steps at all – because they were away versus ten miles away, and quadruples it versus captured automatically by your recording tool. There being ten time zones away. are various commercial and open source tools you can use for this. I like BBTestAssistant, TestExplorer, Practice, practice, practice explaining testing. or Spector. But there’s also Wink, which is free. Also It’s not enough to know how to test. If you are in remember there are log files, especially if you are a team of one, congratulations, you now have to be testing something online. Or use a digital camera. ready to explain testing, defend testing, and teach testing. Over and over. To the same people, too, Do lots of exploratory scenario testing. because they will keep forgetting what you tell them. This means that you test according to realistic and One way to practice is to blog about testing, or write complex sequences of user behavior organized around about it in a forum like the Context-Driven forum a coherent scenario. The major downside of it is that (software-testing@yahoogroups.com). to do it well requires that you get real users to help with it or that you have a deep understanding of real Advocate for testability. Testing and development are users, how they think, and what they will do with not symmetrical. A small decision made by a designer, your product. Exploratory scenario testing would be making little work for him, can be a whole lotta work the same as beta testing if beta testing were highly for you. You’ve got to get in there, gently, and help organized, focused on challenging situations rather them understand the testing impact of what they than just everyday situations, and done by people add or change. Don’t expect anyone to know about passionate to find bugs. testability unless you tell them what that is. Basically, it boils down to two big things: A) can you see it? B) can Now for the social ideas. Unlike Charlton Heston, you control it? That’s why I push for rich log files and who was being chased by zombies most of the time in a scriptable interface (I’m happy to test through a GUI, that movie, you need social skills to be a really good but I also want to get underneath the GUI to control omega tester. It’s your social skills that will save you. the product better and see it better). Social Ideas: Advocate for strong pre-testing. I don’t worry about this so much when I’m working with a good test team. Declare high service aspirations. For instance, It can be hard to get developers to work methodically. when I work with a development team, I give them a It is especially hard to push for that without sounding one page declaration of all the things I promise to do like a nag. But if you are a one-person team, you don’t for them. It documents twelve commitments I make have much choice. You need to sit down and level to them as a tester. I don’t want them to see me as a with them: unless they do unit testing and paired nagging burden. I want to ease their burden, not add programming, or something similarly effective at clearing to it. I will serve my team in any way I can to achieve out the knucklehead bugs before it comes to you, then that. By making my commitment clear, I find that I you are going to be overwhelmed with bugs to find, foster a similar attitude in return. That makes my job and that will slow you down. You’ll be like one of those easier, because their help is vital for me to do my job contestants in a “money booth” trying to grab flying cash well. The success of the following, begins with this one. while it whirlwinds all over the place. If you like sports Set low technical expectations. “Yay, we hired a metaphors, tell them you want be a good goalie for the tester! Now we’ll ship without bugs!” People will think team, but even you can’t defend against multiple shots testing is easy. People will think your job is to “ensure on goal all at the same time.12 / www.softwaretestpro.com
  • 12. Avoid bureaucratic bug reporting protocols. Bug Who’s the Real Zombie?investigation and reporting take up a lot of your time. If you do a good job being an omega tester, you’ll findThe more time it takes, the less time you have to find that the other people on the team, over time (it may takenew bugs. But if you test alongside the programmer several projects), will come to respect and rely upon you.whose code it is, you’ll find that reproducing the bug Otherwise, especially if you fall into a formulaic, uninspired,and writing a report become mostly unnecessary. or complainy approach to your work, the project team mayHe sees what you’re doing and immediately reacts come to see you as the mutant zombie in their midst. Oneto it. I’ve found this to be a wonderful way to test sign of this is that they run away when you shuffle towardsomething in its early stages. I’ve also experienced them with your arms stiffly out as if groping for brain mattera project where my testers were required by our or “unambiguous requirements.” Don’t be like that, okay?client to use a cumbersome bug tracking system thatreminded us of filing tax forms. We estimated that welost 30 minutes of test time for each bug we reported,over and above the time it took us to actuallyinvestigate and describe the steps to reproduce.On top of that, we were not allowed to talk to the About The Authorprogrammers. We didn’t work near them, never metthem, or heard from them. It was an outrageous James Bach: If Devil’s Advocate were a televised sport, James would play itwaste of time. If you are a team of one, tell your professionally. A conscientious objector to high school, he skipped school andmanagement that you need to minimize your report taught himself programming. Apple Computer tried him out as test manager,grinding and maximize your bug finding. and he never looked back. He worked in Silicon Valley for nearly ten years, now independent and one of the most outspoken advocates of skilled sapient testing. James Co-authored “Lessons Learned in Software Testing”, and wroteMy Favorite Advice for Omega Testers... “Secrets of a Buccaneer-Scholar”.If you are an omega tester, feeling surroundedby creatures who have tester-like qualities (arms,legs, human DNA, etc.) and yet are not like you,do not be dismayed. Befriend them. Get them tohelp you. Here’s the secret: people will help youtest if you make it easy and fun for them. Let’scall these people beta testers.Beta testers can help you test, but don’t givethem any paperwork to do. Blech. I was once abeta tester on an early PDA (one of the very earliestever, which was called the Magic Link or somethinglike that). I noticed only after having the device fora couple of weeks that Sony had expected me tofill out a minute-by-minute log of exactly what Idid with their product. Ha ha ha. No, I justdoodled in some fake data and returned it.Many things are expected of a true professionaltester. You should be able to report and justifyyour test coverage and your oracles. You need tounderstand risks and use test tools. But pushnone of that onto your beta testers. Instead,create a carnival atmosphere of old fashionedbug huntin’. If I can get my tester helpers allin the same room for a testing event, I makesure they are fed. I pair them up and give eachpair a bell to ring when they think they found aproblem. Beta testers are motivated by the respectthey receive and the feeling of being listened to.I’ve also had very good experiences with bugbounties – paying people who are not on the testteam for each critical bug that they find. You cando this with actual cash money, or do it with playmoney that can be used to purchase items at anauction you hold at the end of the project (thelatter method allows you to completely controlthe budget). Volume 7 / Issue 5 / June 2010 13
  • 13. By ThomasO’MARA Performance testing is an importantPERFORMANCE component of the overall software development process, owing to the fact that the final indication ofTESTING success is user satisfaction and how well the system responds under load. Since agile development and testing have been around for a while and being used by development and testing teams successfully, aWITHIN THE focus on performance testing early in the developmentAGILE cycle needs a closer examination of how much of an impact it will have on the final outcome.METHODOLOGIES What and Why of Agile Methodologies The Software Development Life Cycle (SDLC) is the venerable overarching structure that every development initiative has usedSoftware engineering and the programs since the beginning to manage the overall development process,generated from such initiatives have indeed but it too is evolving rapidly into a series of hybrid approacheschanged the way business is conducted in put together to satisfy a particular need. Even more now than ever the SDLC is forcibly being modified due to external marketevery enterprise large or small spanning the pressures and changes internally as the needs and requirementsglobe. Furthermore, the way software is being of the business units change and become more complex.constructed and managed today is rapidly Before agile development and testing such as: Test Drivenchanging as the environments, development Development (TDD), Scrum, or Extreme Programming (XP), the most widely used methodology is the waterfall method. Thistools, and methodologies evolve to fit the method is based on the building construction metaphor andunique needs of every organization. employs a sequential approach to software development. The waterfall approach delayed testing until after the application was constructed and only then was the code set transferred to the testing team. As you can see, performance testing was most likely last on the list to be performed, if ever. With the agile approach, both the development and testing teams are engaged in the early stages of construction and all processes are done in parallel to meet the short term cycle or sprint. This will ensure an accelerated return on investment (ROI), value delivered to the business unit early on, and a higher quality product that closely matches the original intent. Still, with all of the new technology and methodologies available, system performance is a key indication of success. Therefore, a closer look at fundamental aspects of performance testing is needed. Characteristics of Agile Performance Testing Performance testing with its ability to quantify the application’s response and throughput is an integral part of any software development testing initiative. Unfortunately it can be an afterthought, or employed only if the application does not respond to the user’s expectation. To use performance testing effectively in the rapid cycle development process, the developers and testers need to adopt an agile way of thinking and quickly put these ideas to work early on. Some of the more key points of performance testing are:14 / www.softwaretestpro.com
  • 14. AGILE | METHODOLOGIESPerformance Objectives of development. This brings us to the THE MAJORITY OFThe service level objective or agreement essence of agile testing: testing at the(i.e. response time, throughput, the component level. ENTERPRISE LEVELnumber of concurrent users, andadditional factors that contribute to SOFTWARE DEVELOPMENTthe user’s experience) is between the Component Level Performance Testing PROJECTS EMPLOY THEIT managers and the stake holdersand needs to be defined in the early Within the agile environment, the 3-TIER ARCHITECTURE TOstages of requirements gathering. service and persistence layers of 3-tierThis will establish a clear set of architectures are normally constructed SOME EXTENT, SO TESTINGguidelines on system architecture, first to meet the wire up of the presentationapplication design, and construction. THE VARIOUS LAYERS AT layer. During this phase the data modelTest Data Creation is defined, the data access framework DIFFERENT STAGES OF is chosen, and the domain objects areWhen the initial requirements for the DEVELOPMENT IN EITHER ready to transcend the tiers. That’s whenfirst development cycle are completed, the testing and development teams comethe data model defined, and the service together and define the test scripts needed THE AGILE OR NON-AGILEand persistence layers starting to take to generate the unit tests for test drivenshape, the development team and the ENVIRONMENT IS ESSENTIAL. development and performance testing.business unit will need to preparea dummy dataset large enough to Each component of the service andemulate a real world workload. This persistence tier such as SQL statementswill become the baseline starting point and stored procedures, service layer classfor performance testing cycles as the libraries, and web services needs to beapplication matures into a production taken into consideration when defining aready code set. performance testing plan. This will provide the necessary guidance on how to isolateTrend Analysis each component for drill down analysisIn agile development and testing, it is should a resource contention arise, thusimportant to run performance tests ensuring the development team deliversand analyze the data at the end of every a high quality application that will meetcycle or sprint, primarily to see how the its end-to-end performance goals.application design and construction isresponding when the system is undera predefined load. This information is Conclusionparamount in the agile process to provide The majority of enterprise levelthe necessary feedback to meet the software development projects employrequired goals as each of the iterations the 3-tier architecture to some extent,transpire during the development process. so testing the various layers at different stages of development in either the agileDevelopment to Production Life Cycle or non-agile environment is essential.When the application is mature enough However, up to this point, most automatedto yield performance results, testing in performance testing is done via thea real world environment is crucial for a presentation layer or user interface tosuccessful implementation. So it’s good reach the service and persistence layerspractice to make sure all of the required with user data. This can cause delays ifsystems are allocated in the initial phases the user interface is constructed late andof development, so performance testing not wired up to the service layer.on the production level hardware is useful About The Authorin determining what is needed as the Load testing at the component levelapplication grows to its final configuration. without the user interface present is Thomas O’Mara is the founder critical to success of the agile testing of TEOInnovations, Inc. andWhen the performance objectives project. Not only will such testing keep has been building, programming,have been met, the test data is being development on track to meet the sprint and testing computer basedgenerated, and the physical system is goals, but will uncover any potential systems for almost 30 years.being prepared, planning on how to bottlenecks early on to save time, You can reach him ateffectively test the application is critical. articles@teoinnovations.com. money, and resources in the process.Since most of the testing is being done atthe Unit level, only service componentswill be available in the early stages Volume 7 / Issue 5 / June 2010 15
  • 15. DEVELOPINGA | COMMUNITY Featured Crews Notifications Member Profile16 / www.softwaretestpro.com
  • 16. Meet the TeamDEVELOPERS 1Developing a Communitynot just Programming 2a WebsiteWith all of the new offerings atSoftware Test Professionals (STP),we had to highlight our new website, 3and what better way than to talk to thedevelopers responsible for the developmentand coding of the new site. I sat down withSystem Architects Patrick McDavid andCharles Gibson to discuss the new website. 1 Director of IT ROVANSEK JanetteI found it very interesting. jrovansek@softwaretestpro.com Janette has been in IT for 15 years and was responsibleBy RichHAND for the systems at HDI for 8 years. Currently she heads up the team here at STP and is the CRM and one person Help Desk. She is often found smiling next to Abbie at conference registration and training events.I came away from our discussions with a sense of their Patrick McDAVID true understanding and commitment to our business, 2 Systems Architect and thought our audience would appreciate their vision pmcdavid@softwaretestpro.com and commitment to this community. They are developers A member of the team responsible for the new STP but they are the kind of developers that are truly empathetic website, Patrick spent 5 years creating and developingto the role of the tester. Their past experience in the association systems for HDI. Known here as P-diddy, he is most often found on the UCCS campus engaged in philosophicalbusiness has really shaped the way they developed the new arguments with his soon to be graduated peers.Software Test Professionals website. Although when I askedthem about the testing process, the reaction was eye-opening. Charles GIBSON 3 Systems Architect cgibson@softwaretestpro.comWhy a New Website? A member of the systems team, a creature of agile,We already had a website, so it is a fundamental question to raise – and all around cool guy, Charles has spent 8 years in software development. A former HDI member, he canwhy create a new website? What was the need? most often be found instructing Krav Maga. No needThe previous management team here at STP outsourced the to worry about security at STP.responsibility of the website to a third party, and they coded in Rubyon Rails. Charles and Patrick felt that in the interest of time, andthe ability to have greater flexibility and agility, it would be better Volume 7 / Issue 5 / June 2010 17
  • 17. to consider coding in a language best in our environment.” Patrick and occasionally arguing access they both had previously worked agreed and stated “The process was to the functionality these guys with. Patrick stated, “We had the very informal working with a beta were on track with the business opportunity to beta Microsoft’s new software; changing features on the expectations. In our discussion, .Net 4.0, which was a very attractive fly; finding new stuff we could use Patrick and Charles firmly agreed opportunity for us.” So the decision all made it more of a creation than as stated by Patrick, “the website was made to use .Net 4.0. According strictly coding from requirements”. is really just the beginning of a to Patrick and Charles, it was Their methodology seemed to work continual development process that exciting and a bit of a risk to use but you – the community – will be will evolve with the membership a beta version of Microsoft’s newest the ultimate judge. and community.” They not only offering. They both agreed that it understand that development will How much direction did you get was really cool to start from scratch continue, they are encouraging from the business owners? They with no legacy to worry about. feedback by having a “Crew” both agreed the initial meetings with for people to interact with them The development team had a the business owners were helpful to suggest future features and blank canvas to work with when but unnecessary. They said that functionality. Charles added, “I they embarked on the project. with tongue in cheek but I was not can’t wait to see how members “Having built communities in the that sure. Having worked with them use the site so we can enhance past, the new language gave us at our last association they were well the areas they really want, and the flexibility we needed to create known for creating website features an interactive website for the on their own. Charles added, “We develop features we have not even considered yet based on their membership, which in the end knew in our heads what worked and feedback”. I truly saw excitement in was the driving factor that helped what didn’t at the last association so their faces when they talked about us decide to start from scratch,” we wanted to make sure we included software testers actually providing both Patrick and Charles agreed. the capabilities members utilized feedback that they can take and often and minimize the time spent Patrick brought up the point, “Beta turn into reality. Between the on things members rarely used.” is very interesting to use because it lack of legacy, and the agility and They both know the business and gives you the new bells and whistles flexibility of their infrastructure, agreed that their experience was but it comes with risk.” Looking at they believe the changes will come the key resource used to develop the new website it definitely looks fast and furious if the community the site. I think they welcomed the like it was worth it! wants them. As the membership feedback from the business team What “process” or development guy here, this is music to my ears! to keep them from “wandering” into methodology did you use to the realm of cool that had the risk What is the feature you are most create the new website? This of eating up precious time. excited about for our membership question generated a chuckle, and community? Hands down, For development teams in large laugh, sigh, and a long pause both Patrick and Charles believe the shops with stringent requirements before Charles spoke up and said ease of sharing and contributing and process this may seem “Informal Agile.” Charles had content is the foundation of what we unstructured but as a business spent a number of years in more needed to provide for a professional stakeholder, one of the reasons developed Agile shops so he was community. These guys have a I think it worked was Patrick’s cautious to clarify that it was a natural instinct for associations and Charles’ strong knowledge little bit of paired coding, iterative and know that the association of the business of associations development, loose design and can only prosper if the members and community engagement. plan activities, but mostly loose participate. Patrick referred to methods. Charles stated, “We did Having been part of the status the ability to share content as the a lot of things Agile suggests, we meetings, I was confident after “thought leadership promotion tool.” just did it in a way that worked reviewing, listening, This content sharing will build the Badges Featured Articles Industry Tweets ST&QA Magazine Online18 / www.softwaretestpro.com
  • 18. DEVELOPINGA | COMMUNITYmembership’s foundation of knowledge along with in their mind the things they needed to revisit in an effortindividual member reputations. Charles and Patrick to have a better quality product. They were consideringhave built in a reputation model that awards points things they may not have, if their work wasn’t going to beto “activities” that provide value to the community. tested by a large group of testers from our community. TheirWhen a member reaches a certain number of points conversation was respectful but they were clearly anxiousthey achieve trophies but the real value is the and concerned about the bugs that could be found. Theymember’s profile is updated to let others know the would not rule out the possibility of testers finding problemsstatus of their membership reputation. The more a that could delay launch. They were in no way willing to saymember participates and contributes, and if what with any degree of confidence that the launch would takethey contribute garners good feedback, individuals place on the projected date. It was partly out of modesty butreceive additional privileges and status within the also a respect for the possibilities that having testers combingmembership. Charles and Patrick are both looking the website, this community may find flaws that would bringforward to the feedback regarding this feature. to the forefront issues that could delay the project.The other feature Charles and Patrick get excited They know they are developing a site for the most criticalabout is the Crew feature. Crews are the foundation audience in the IT industry. They understand the necessityof the membership. Crews are groups of members for testing and they know the site will be under constantthat have similar interests and can collaborate by scrutiny from the membership. It is a relationship they hopesharing content, video, podcasts, and comments on to develop, and are looking forward to the scrutiny becausetopics of interest. Pro Members can create a crew in the end, the quality of this website will have to be theand can be part of multiple crews. best. If we have thousands of testers on the site - utilizing it to share; contribute; learn; build relationships; and theThere will be some crews we hope will be extremely membership is willing to provide us with the feedback - ourvaluable such as the “Ask the Tester” - a crew where developers will have created the website all of us at Softwarequestions will be answered quickly by an industry Test Professionals could only dream about.leader as well as others within the crew. We alsohave a crew for people interested in getting publishedin the magazine or simply talking about articles inthe magazine. The idea of the crew was summed upby Charles “members can share, develop, and utilizecrews any way they find beneficial to improving their POSITION YOURSELFmembership experience”. Charles is really pumpedabout crews and Patrick is pumped about contentsharing. But I can say with confidence they can’twait for “opening day”.And then I changed this mood of excitement and AS A CENTER OF INFLUENCEspeculation, and exuberance by asking this:Are you ready for the testers we are aboutto unleash on the website? I can only say the Join other senior-level professionals to network,comments between Patrick and Charles started exchange ideas and share information in face- to-face meetings. Meetings include industryflying, “Oh man they are going to hammer us. Theyare going to challenge our word length fields, our…; they continued on as I sat back, listened, andlearned what the role of testing is really about for facilitators, forward-thinking topics and thethe developer. What I heard was both Patrick andCharles listing the areas of code and functionality ability to network with executivesthat might be vulnerable. They were fine tuning facing the same industry issues and challenges. Network at a higher-level. Be Influential. W W W. S O F T WA R E T E S T P R O . C O Mwww.STPCon.com / October 19-21, 2010 / Las Vegas CONTACT US TODAY! 1.877.257.9531 / rhand@softwaretestpro.com Volume 7 / Issue 5 / June 2010 19
  • 19. By JonQUIGLEY and KimPRIES IMPLEMENTING “ W E CA N I M PROV E TH E SIX SIGMA: SOFTWARE ITSELF (AND, POSSIBLY, THE TESTING) OR WE CAN IMPROVE THE DEVELOPMENT PROCESS.” Improvement in Software Development W e can improve software development in two primary ways: This important tool can be quite we can improve the software itself (and, possibly, the testing) difficult to implement well. While it is or we can improve the development process. We have already easy to list and document the lesson covered the preceding three Six Sigma Phases (Define, Measure, learned, it is not so easy to put this and Analyze) in previous articles. Now we take what we have information into a format that makes learned and use that information to improve our software development. for easy searching and retrieval. It is the retrieval that is important. Improving the Software Itself We have seen lessons learned documented in word processors, Training software developers is one obvious method for improving spreadsheets, data bases, and free software development. All people may be created equal, but not all form storage (for example, Microsoft programmers are equal. However, simply saying ‘training’ isn’t really OneNote). Even with a data base, good enough. We need to consider what our measurement and analysis we need to have a good system of phases have told us about our software issues. One typical suite of training search categories and terms or classes might include the following for a hypothetical embedded software nobody will be able to retrieve the development group (‘Our-Department’): information promptly. If the work to review the lessons becomes a burden, the task will starve away n Introduction to Our-Department n In-circuit emulation and we will have lost an opportunity coding practices and standard to profit from past blunders. If we n Pitfalls in C and how for the C language are using textual information, we to avoid them n Our Department specific can use search engines like Google software handling procedures n Memory mapping the processor Desktop to help us find information; an older approach using ASCII files n Use of static testers n Lessons learned review would involve the use of the grep tool n Use of dynamic testers n Design architectures common on UNIX-types of operating systems. Once retrieval becomes efficient, we have the beginnings of We call it ‘training’ but, to some extent, we are really talking about defect prevention program. We don’t indoctrination into our organization’s practices for software development. think it is inappropriate to make Some of these practices will fall under the heading of lessons learned— the use of the lessons learned data where we have captured failures from the past as well as solutions to those base a part of the software engineer’s failures. It does not do much good to critique failures and understand them performance evaluation. if there is no learning from this review. Yet, in our experience, this is often what really happens. Frequent reviews of our lessons learned file or data In a sense, while we call it training, base by the entire team is not wasted time. We have seen what we call the what we are really talking about is a ‘stupidity loop’—a phenomenon often tied into the turnover rate of developers, continually learning organization 1. especially with senior developers. We can detect the duration of the stupidity This is more than learning from the loop by the regularity with which we see the same mistakes begin to recur. failures. This includes boundary Hence, the reviews of the lessons learned file allow our entire team to learn spanning for best practices, which from the past without necessarily having all the original players present. is not necessarily a euphemism for 1 SENGE, P.M. 1990. The Fifth Discipline. London: Century Business20 / www.softwaretestpro.com
  • 20. IMPLEMENTINGSIX | SIGMAprocuring the best practices of other controlled, we increase the likelihood n Combinatorial testing: whereinorganizations. We may also be able of producing a truly robust module. we stimulate the inputs of theto affect the software development We can practice encapsulation system and observe the outputsprocess by benchmarking our own with any language, although the of the system using a recipe thatorganizational practices against truly object-oriented languages exercises all inputs in a particularthose of a well-reputed software help us to enforce these practices. pattern (for example, pairwisehouse with who we are not in We also want to be sure that our testing tests all pairs)competition. The benchmarking code will check for simple flaws likeexercise should provide ideas for overflows and underflows—we can’t n Stochastic testing (ortesting in our own organization expect the run-time produced by exploratory): we allow the test(note: we test because many times, the compiler/linker to do this for engineer to use their ‘gut’ andthe ideas of other organizations are us. With embedded code, we must knowledge of the product to explorenot simply drop-in practices). make sure we are not burdening the the behavioral envelope of the microcontroller excessively while software; sometimes, we can useFor developers not in the embedded maintaining the safety of our code tools to help with this approachsoftware arena, the same kind of (a sad truth sometimes). The more (for example, a genetic algorithm)approach applies. We want to capture risky the result of an over- or under-what we think the developers need to flow, the more important the practice n Severe testing: we challenge theknow and we want to indoctrinate the of deliberate protection will be. We software by doing horrible thingsteam regarding known development have seen substantial protection to the data bus, the input values,issues that have occurred before. written into code that controlled a the hardware—anything goes withThis indoctrination includes instilling fighter aircraft, since the results of a this kind of testing. This testingthe organization’s risk philosophy failure were likely to be catastrophic. can exceed the design limits toand extends to why some things are The approach worked well. ferret out marginal performancedone the way they are done; in short, issues. In fact, it is not out of linewe describe our practice and how to for the testing to go all the way toperform them as well as the rationale the destruct limit—by so doing, we Improving the Software Testing establish the true design limit, thefor doing it that way. The approachapplies regardless of whether we failure limit, and the destruct limit,are looking at web development, We recommend thus characterizing the product. A MULTIMODALenterprise applications, or database management. Occasionally, an approach likeIn addition to indoctrination/ approach to testing, severe testing will produce specious failures; that is, failure we wouldtraining, we can also reuse known particularly if our measurement never expect to see in the field.good code as long as we spell out However, any failure indicates somethe rules for doing so. In fact, reuse and analysis phases suggest kind of weakness in the code. Weof code (and pieces of hardware for that we are letting defects can then make a conscious decisionthat matter) can become another through to our customers. whether or not we should take actiontopic for the training part of software on this failure information. Thisimprovement. For code reuse to is much better than not looking atwork well, the library modules we what the risk may be; furthermore,are going to reuse must be well- in the unlikely event we do see thesedocumented and, well, modular. In failures in the field, we already knowsome cases, these components can Severe something about them and have a Testingexist as executable code in the form remedy prepared in anticipation ofof dynamic link libraries; in other these dire consequences.cases, we may have to incorporate Stochastic Testingsource code into our new source code Test suites can be just as(definitely less desirable since this reusable as reusable software.approach can lead to degradation In fact, for regression testing Combinatorialif the developers touch the library Testing (testing to verify we haven’t addedcode). As we already know, some defects to already good code);languages lend themselves to we want to reuse the previousmodularity better than others. collection of tests to perform our Compliance Testing regression testing. Our Six SigmaEncapsulation (or decoupling) measurement and analysis phasesis often a language-based approach will tell us if the regression testingto implementing good practices, performs up to expectations. Thisalthough we can accomplish In general, in the embedded world, is a good candidate for automation,decoupling by focusing our attention we use the following approaches especially for organizations that haveon older languages that do not most often: an iterative product developmentexplicitly support it. If we can methodology. In electronics productencapsulate a software object such n Compliance testing: testing to manufacturing facilities, it is notthat access to its methods is well- the requirements specification uncommon to see automated Volume 7 / Issue 5 / June 2010 21
  • 21. test equipment that is used to exercise the product; in effect, performing Product development process extremely high-speed regression tests. We recommend the same kind of knowledge of the staff is particularly approach to regression testing—using technology to speed us through the helpful. We are not talking about tedium of ensuring we did not decrease the quality of the software. the knowledge of the players for their particular part of the We can use the results of exploratory testing to add to our existing set of development puzzle, but knowledge formal tests; in effect, practicing continuous improvement with our test of how the pieces should go suite. Consequently, we can expect the test suite for a specific product together in general. Colleges and to grow as we move through the development project—we are learning universities teach engineers, both about the product as we proceed. The expansion of the test suite should hardware and software, about the be welcomed rather than rejected because it allows us to tighten down technical details of the development on the product with every iteration of the test suite and each release. An process. Something as rudimentary additional benefit of the regular growth of the test suite is that the testing as configuration management becomes more stringent as we move towards release of our software. needs are usually not covered and certainly the role of product testing Test automation can improve our situation by taking our test suite and is not. The staff’s knowledge of the applying it to the product at machine speeds as we have already mentioned entire puzzle can come from the use in our section regression testing. An additional benefit arises because of well-defined software architecture we often do not have to have any kind of human intervention and we document. The architecture can, therefore, run our test suite day and night, thereby improving our document, if well-crafted, will throughput on test case achievement. In our case, we are also set up to often see little modification make a video recording of the testing for subsequent review in the case of during the development of the a logged failure. Sometimes automation may not be practical for a specific software; yet, it is the foundation product, but we recommend automation whenever possible. for understanding the puzzle and all the pieces as a whole. This The automation, partially or fully, of a test suite allows our test engineers document can be a key reference and the software engineers to move towards a test, analyze, and fix (TAAF) when modifications become regime. TAAF is a means for using our competitive edge in testing to rapidly necessary as new features are cycle through the software testing in the interest of flushing out as many required or fixes need to be made. defects as we can. We continue to measure defect arrival rates and, when feasible, we can compare our results to a Rayleigh model, which may Design tools can be helpful, provide us with a statistical basis for authorizing a release of a product particularly when it is possible to produce usable code directly Test from the design tool. We know for TAAF IS A MEANS FOR USING example, that the International Telecommunications Union OUR COMPETITIVE EDGE IN TESTING standards Z.100, Z.105, and Z.120 have been implemented TO RAPIDLY CYCLE THROUGH THE in software by Telelogic. This Analyze design software produces SOFTWARE TESTING IN THE INTEREST pseudocode and a graphical representation simultaneously OF FLUSHING OUT AS MANY DEFECTS during the software design phase. The code can be tested within the Fix AS WE CAN. design tool and, when it is ready for outside-the-tool testing, the design can be compiled directly. Improving the Software Development Process Other tools such as Matlab/ Simulink from Mathworks Prioritizing tasks can be difficult, especially when our team is under the have similar capabilities. hammer to deliver promptly (everything and now). Because many myths surround a highly-cognitive field like software engineering, it can be Design tools that are decoupled difficult to determine what the highest priorities are. For example, some enough from the code such that organizations find a complete and comprehensive requirements document they cannot either produce code far outweighs the benefits of design documentation. Other organizations or a compiled image are generally may find or believe the reverse. less useful, although we have seen research indicating that the effort Additionally, we can use some simple financial tools to help us prioritize of working in the latest version of the tasks within our development process. We have used a tool that the Universal Modeling Language combines net present value with some qualitative assessments and (UML) produced software with provides a quick pseudo-quantitative calculation of priorities. When mixing fewer issues than similar software factor types, we need to be careful that the value from the NPV calculation developed without the discipline does not swamp the qualitative data. of presenting the design in an abstract format. We need to use We can usually detect a prioritizing issue when we hear our engineers say these tools with care so that they something like “I couldn’t do the frimfram function because I didn’t have time.” don’t become a collection of pretty22 / www.softwaretestpro.com
  • 22. IMPLEMENTINGSIX | SIGMApictures to dazzle the customer this approach will provide a (or functions or features) andand upper management. Any basis for starting the review as break them into ‘bite-size’ chunks.reasonably competent modeling well as providing a baseline for We can then put these items ontool actually represents a grammar; problem-checking. an electronic white board or a realhence, the structure of the model one and ask the developers to pullshould be verifiable electronically. Proof of correctness is a rarely the feature from the setup area toIf this feature is not available, the used technique, yet a powerful the work area to be followed by theresults from the design tool will be approach to software development. complete area when this discretepalliative but not corrective. When we prove that software is portion of the project is now correct, we assume the specified complete. The Kanban approachDesign reviews are often touted requirements are themselves correct is excellent for job release to theas a means for improving software. by some definition of correctness. developers but may make forOur experience suggests that The requirements may be expressed headaches with the planners.reviews may be useful if the in a formal language such as Z or the The solution to the planning issueparticipants approach the presented B-Method or some other systematic lies with reasonably accuratedesign as if they were hungry approach. The requirements are estimation of the work time forpiranha and the design was a piece really the Achilles heel of this each Kanban card/ticket.of fresh meat. A design review that process. Extensive documentationdoesn’t have metaphoric blood does not necessarily help. There is Regardless of which choices weflowing under the door probably still much interpretation by multiple make, the software developmentisn’t accomplishing much. The parties, common understandings are process must be measured:primary issue with this approach more difficult to reach, and we may first, to be sure we didn’t makecan be tied to the following items: think a common understanding of it worse and, second, to ensure the documentation exists when such we have a detectable improvementn Lack of use of measurement is not the case. Computer scientists in performance. and analysis information have theorized about formaln Failure to use the lessons methods for years and both the U.S. Conclusion and Next Step learned file Department of Defense and NASA have taken a look at the usefulness There are many ways to improven Inadequate critiques from the of the approach, particularly with the quality of the development participants (the critique should safety-critical software. The difficulty deliverables. You must start with not start when the actual review of the approach has always been an understanding of your problem starts – come prepared) the level of mathematical and areas and where you can don Lack of interest or time put logical sophistication required. the most good with the least in up front for the review effort. Making changes to the Once we have code, we can derive design environment also meansFor design reviews to have any the coherence of that code against understanding the variables,chance of having an impact on the requirements. We do not need particularly those that arethe end design result, the due a doctorate in symbolic logic to be directly changing because ofdiligence in the critique has to able to do this, but we do need to the improvement actions. Thesebe greater than the due diligence have a fundamental idea of how to variables we will have derived fromduring the design work. The approach the correctness problem. the measurement and analyzeconsequences for taking this Even so, the correctness approach phases of our Six Sigma problem-review lightly should have some has the benefit of being less solving approach. This meansimpact upon the reviewers. Having qualitative and more stringent keeping your eyes open forpeople showing up with marked than the design review approach. the unintended consequences.up copies of the design – should The next time we will discuss:be a requirement. Those that do Increasing tempo helps us Control – the approaches we use tonot perform this up front work, especially when we are trying make our improvements sustainable.have not delivered and are to implement a TAAF (Test, Analyze,evaluated accordingly. And Fix) regimen to our testing and software development. From theUnfortunately, these items project management perspective,suggest that design reviews are we might implement scrum ashardly deterministic and may a simple means of acceleratinghave significant qualitative issues. development and improving projectIt is when looking back over your oversight. We have used scrum About The Authorsshoulder that you notice that for line management and we knowa particular failure should have / from experience that the scrum Jon Quigley is manager of thecould have been caught had there approach will improve the verification and test group atbeen a critical design review. cadence of accomplishment. Volvo Truck.One potential method for improvingthe design review would be to start Kim Pries is director of product We can take scrum to the nextwith a standardized checklist based, integrity and reliability for Stoneridge, step and implement a Kanbanin part, on our lessons learned which designs and manufactures system. In the Kanban approach electronic automotive components.documentation. At a minimum, we will take all of our user stories Volume 7 / Issue 5 / June 2010 23
  • 23. ASKTHE | TESTER MATTHEW HEUSSER “ I F YO U T R AV E L I N Read Matt’s Blog Posts @ M O ST O F T H E P O PU L A R www.softwaretestpro.com/blog S O F T WA R E T EST I N G C I RC L ES , YO U ’ V E PRO B A B LY H E A R D O F JonBACH J O N B AC H .” M anager for Corporate Intellect and lead consultant at Quardev (a software testing and technical writing service company in Seattle), Jon is the co-creator of Session-Based Test Management and a longtime champion of exploratory testing. You may not know that Jon is also a trained journalist and author of the autobiographical “Above the Clouds” where he tells a story about his relationship with his father author Richard Bach. Jon is also a blogger on the sapience and humanity in software testing. In this inaugural column of “Ask the Tester”, we’ve asked Jon to address questions about testing and test management. Want more? Jon Bach will now answer your questions. Go online and visit the “Ask the Tester” Crew and join the conversation – Pro Members can ask question and interact What’s the future of testing? Are we going to have to learn to be testers with Jon directly! and programmers to deal with automation and to be more useful in Email your questions, name and location dealing with code issues, or will there be a shift in focus to some other with ‘Ask the Tester’ as subject line to: aspect of process and use of tools? matt.heusser@gmail.com Michael Furmaniuk Cambridge, MA BACH: I think so, yes. Just like India, New Zealand, Russia, and even more important to establish how certifications have a chilling Europe). Software Test Engineers your personal reputation as effect on our industry in that will likely become extinct as a tester. More testers will employers don’t even look at more companies will look to understand this as the economy your resume unless you have a SDETs and programmers to fill fluctuates. “Better, cheaper, cert, automation and notions of testing roles. To compound this, faster” will still be the mantra. Agile will have a similar effect. candidates will quickly learn How you meet that perception is Don’t have Ruby, Selenium, or that to be employed, they will by being (or appearing to be) more Fitnesse experience? Don’t know have to become programmers, indispensable on their project how to code? No job for you. The and then it becomes a self- than other candidates. I’m not future will be much the same as fulfilling prophecy — no more talking mere intelligence, but by the current. The perception will STEs, only SDETs. If companies being more valuable to them in grow: “Why hire a Test Engineer need testers, not programmers, your tactics and approach and when we can advertise for a they will think that all they need in ways they appreciate. The more Programmer-in-Test who will to do is crowdsource — find a kinds of agile apps that show use tools and automation to do company that has 10,000 regular promise (Selenium, Fitnesse, the work of 4 testers?” Decision- folks (not skilled testers) sign up Cucumber, etc), the more word makers are going to continue to to find bugs in a short duration will spread. The good news is find the cheapest way to test, so with their iPhone or PC or iPad. that social networking is here to using labor in other countries Agile values and principles will stay. It is not a trend; it is more will still be prevalent. But even continue to take hold as decision- and more an online “university.” though the value of testing there makers will want to ship more Twitter is a tool that enables this. is not as rich as it is onshore, kinds of software more quickly For colleagues who need to those countries will improve at and will think Agile will be the build their reputation or create as more take it seriously as a way to do it whether or not it a professional community that craft (evidenced by the Weekend really is the right fit for them. can help them, tools like this Tester groups sprouting up in With all of this, it will become will be a must.24 / www.softwaretestpro.com
  • 24. Meet the TeamIn many fields there are BACH: All testers should be aware of how theya few principles that are solve problems. That’s why I like doing puzzles,fundamental; they are riddles, brain teasers, and recall games likeinextricably linked to each those found on Sporcle. I’m trying to learn whataspect of the discipline. it feels like to think, so I can recognize when I’mWhat do you think are the not thinking, or when I’ve slipped into autopilot.few fundamental principles Likewise, I want testers to know what it feels likeor skills for testing – that to be confused or stuck or unsure, instead ofall testers should know just dismissing what feels to them like a simpleand have ingrained within headache. Going to a testing conference is onethem? And how do you thing, but hanging around in the bar or the open 1think a new tester should areas afterward is another — that’s when reallearn them? learning happens with colleagues. Jumping to conjectures instead of conclusions is another Kaleb Pederson Spokane, WA important skill. I have a slide deck from my “Craft of Bug Investigation” talk that spells out some of these skills, as well as those used in exploration. 2 BACH: By definition, good testers are goodHow do you think good testers because they are valued. But how didtesters can be valued? they get there? Think of the best testers youIn your opinion, what does have worked with or testers you have seen at athe learning curve of a conference that you respect. What are they doing?tester look like? What can For me, a good tester’s value is their ability towe learn from that? What 3 think of meaningful vulnerabilities and constructshould be improved to ways to reveal them. They are passionate andmake testing more valuable curious. They are aware that their relationshipfor the organizations with with programmers and stakeholders is just asrespect for sharing instead important as test techniques and tools they use.of selling? They know they’re of service and don’t mind Jeroen Rosink The Netherlands adapting to new missions. They know that people 1 Director of CARACOSTAS Abbie see them as the bringer of bad news, so it helps to talk about what’s working, too. When my wife Training calls, it’s never good news. I flinch when I see her acaracostas@softwaretestpro.com name on the caller ID, so I take a deep breath The most outgoing member of the team, Abbie was and get ready to ask how I can be of service to responsible for training at HDI for 3 years and held many her. But that doesn’t stop me from wanting her roles over her 8 years at HDI. She can be seen at every event smiling and greeting members and customers with to call me more just to say “I love you.” a bright smile and warm greeting. She is the epitome of great customer service. Our upcoming training courses will be guided by her experience in working with members to create the types of offerings our community needs. BACH: If quality = value, then no, in my Rick BAUCOMIs the quality of software experience, quality is the same as it ever was. 2 Director of Interactive Communicationsdelivery increasing or My brother wrote two posts with respect to rbaucom@softwaretestpro.comdeclining, and what “quality is dead.” His point was that we don’t Social media is a key communications platform andmakes you think so? expect software to work anymore. I disagree, Rick is the guy who gets the communications to you. Lanette Creamer to some extent. I expect software to work, but He is passionate about utilizing social networks and Seattle, WA the latest e-mail practices to provide the community I understand when it doesn’t. What I don’t the information they need, the way they want it! understand is why people blame testing. It could be that testers found that bug you found, but the Triage Team thought it was a low risk 3 Director ofBRUNO Cheri Sales and decided to punt on it. I can forgive bugs I cbruno@softwaretestpro.com don’t care about, but what I can’t forgive is if Cheri has been creating the most effective campaigns the app makes me think in a way to which I’m for vendors that highlight thought leadership and value uncomfortably unaccustomed. I’ve used test for the communities they serve. A dedicated professional tools that “worked” (no crashes, hangs or data bringing customers and vendors together in ways that benefit both the vendor and the customer. Often seen loss) but gave me a headache because they running through the streets of Boston, she has been forced me to think in a convoluted way. creating value for 20 years. Volume 7 / Issue 5 / June 2010 25
  • 25. Why does testing cost so much? 3 I have to configure and operate it topology; the cost of outsourcing it to a crowdsouring company or Markus Gaertner 4 I have to come up with test ideas Hamburg, Germany a test lab; or to do beta testing or 5 I have to see if my stakeholders create a usability program; and the BACH: Not sure how to answer find value in those tests or know costs to fix the things we find or at that. I’ve been asked to be part of to what extent they trust me least to investigate what it might a book on the cost of testing, and 6 I have to take time to express cost; and the cost of developer I told the authors that I think the or create those ideas as mistakes... It’s amazing that important issue to discuss is not executable tests software is even affordable after all cost, but VALUE. I would pay more of that. Of course, testing doesn’t (cost) for something that has value 7 I have to operate the application have to cost a thing. If you’re first to to me. Likewise, software might as I test and observe the outcomes market or have a great idea, people be free (no cost), but it may totally 8 I have to understand how I’m may want it, bugs and all. I saw this waste my time or be so buggy that it going to know if the tests passed while I worked on Microsoft Flight has no value to me. But the spirit of or failed Simulator 2004. The community your question may have to do with could not wait to get their hands I have to report those outcomes in time and dollars — testing takes on the new airplanes, scenery and such a way that it is likely to lead time because of all of the systemic flight models. It had many bugs, to an action of some kind interactions inherent in software but it mostly met what people (hardware, devices, operating 9 I have to react to that action and wanted it to do. We had more system, memory, etc.) as well as the run follow-up tests assuming my testing to do, of course but beta number of combinations and risks. mission hasn’t changed testing is a great way to build a Here’s a quick rundown of what All of this seems simple to people community and get a sense of what happens for me that might make it like you and me. It’s just what we needs to be fixed at a relatively “expensive” with respect to time: need do. But to actually DO it, low cost, in my experience. takes time. Then there are the tools 1 I’m handed something (somehow) we need to support our work; the to test (in some way) cost of the machines (though these 2 I have to learn about it (model days there are VMs and the cloud); it, compare it to what I know, the time to re-configure those ask questions, get answers, machine or the network; the time understand those answers) to setup or teardown or rebuild a BACH: No one says you have to be shackled in your job. Work the What would you recommend for career job, sure, but be aware of your options. It’s normal these days to advancement and developmental focus work for several companies for short durations. The question “Are for a Lone Gun tester (i.e. one who is the you happy where you are?” is tougher than people make it out to be sole testing resource for his company)? because happiness is relative and multi-dimensional. For example, How can one get ahead when they feel I’ve been in situations where I didn’t like my job, but it was better than as though they are just running faster the alternatives I knew about. Once I knew about more alternatives, to stand still? the game changed. Develop your own (public) portfolio. Blog, Tweet, Michael Larsen San Francisco, CA comments on posts. Get your name out there and stand for something. Talk about your failures and half-baked ideas. Give content away. Be helpful. It’s like the Agile notion: fail fast and adapt. How much does it cost to fix a bug as BACH: I don’t know how to answer that because I subscribe to the a function of the interval between bug Context-Driven School of Thinking which says “it depends.” Why? In creation and bug fixing? (Feel free to this case, sometimes it’s cheaper to procrastinate. Ever jump feet first relate your answer to a denial of the into something only to have the mission change two days later making existence of Technical Debt.) your work obsolete? I once found a bug that was left open for 183 days. The programmer told the triage team it was a particularly nasty bug Steve Poling Grand Rapids, MI dependent on a third-party control. He knew it would take a week to fix, and other things were more valuable to his time on the project. The triage team agreed. Then one day, it was fixed. Just like that. In one day. In fact, in one hour. Turns out that third-party had developed a new SDK which had a library that fixed the problem. All he had to do was implement it. Voila. Bug open for 183 days, fixed in no time, instead of having him engineer a solution for a whole week. Y2K was the same principle. Sometimes it’s cheaper to wait to see if the bug is reported or cared about until you fix it. Other times, though, that’s bad policy. It depends.26
  • 26. ASKTHE | TESTER BACH: I have no project experience with perf and Agile, but I do have How do we plan for performance experience with something known as Agile-fall. Sounds from your question like testing in an Agile Environment? it’s the same. That is, a blending of waterfall and Agile principles where work is Should test sprints be in parallel done in sprints (an Agile notion) but the sprints involve dependent phase gates with the development sprints? (Waterfall) - a design sprint, a dev sprint, a testing sprint. Some Agilists will Ravi V. Bangalore, India decry this, but on the project I was on, it worked well that way. We didn’t care what the name of it was, we just cared that it helped us know when we weredelivering what was intended. Now, as far as performance goes in an iterative release, the only thing I can tell you isthat it’s heuristic -it’s fallible. Even though the whole system isn’t built yet, those perf results may be useful to provokeconversations about quality. Assuming performance testing is a measure of how speedy the task flow in the wholesystem (each component of the software working together), then you won’t have the best idea about that until thosepieces come together. My go to guy for this is Scott Barber. He’s thoughtful and knowledgeable in this space andseems to have a great public portfolio for you to take advantage of. BACH: Seriously. It could be that the skill you need is not “automation”, but Say I’m a test automation engineer. toolsmithing or questioning or risk analysis. Are you trying to use a computer What are the critical skills? to run 10,000 tests in 10 minutes, or come up with a script to grep and parse Software Development (coding) error messages in a 1-meg log file? Both are known as “automation”, but because testing – or something else? the missions are completely different, different skills are required. Maybe the Gergely Brautigam Hungary critical skill is being able to install, configure, and use a tool from the outside to do your automation. Maybe the critical skill is being able to think about howyou’re going to automate your existing set of simple smoke tests, so risk analysis skills are more important. At Quardev,each customer seems to mention “automation”, but means it differently. Do they want a build engineer to focus on simplebatch processing? Do they want to take their manual tests and put them all into some kind of capture-replay tool? Do theywant someone to be skilled enough to write unit tests after their previous programmer quit? Do they want someone to dokeyword-driven acceptance testing with Fitnesse? Each of those needs a different set of skills, but what they all have incommon is thoughtful questioning to deliver the approach that’s right for the objective. As a manager, how do you inspire 9 Back them up in a conflict they 17 Pretend that you are shipping testers to love their job? Especially had with management. tomorrow and see what they the ones who think testing is a would do about it at a time 10 Demonstrate testing to them but dirty job.” when management may think show your thinking (mistakes, Parimala Shankaraiah “we found everything”. Bangalore, India assumptions, etc.) as you test. 18 Try out a Weekend Tester 11 Have them email someone in ourBACH: I’ve tried many ways: business like Michael Bolton or session, and if they’re shy, just have them lurk.1 Help them midwife their ideas. Lanette Creamer, who have good ideas and love responding to 19 Take an existing bug and don’t2 Catch them doing something cool. honest questions from colleagues. tell them what it is, but have them3 Be an example (as in your own try to reproduce it by pairing up 12 Pretend that the developer forbade blog, Parimala, about looking for with someone. them to test something and see the book). what they would do about that. 20 Remind them of other metaphors4 Pretend you’re the new guy and — like how testers are heroes 13 Have a friendly competition of ask them for tips and advice. like the Secret Service, Men in who can find the best bug or5 Tell them your failures and invite Black, bodyguards, or crime create a flash mob for them to them to suggest what you could scene detectives. (I appreciate share their ideas or borrow from have done differently. the question, because all of those others (the parkcalc one last ideas were within me and just6 Find a way to “Dogfood” the app month is a good example). took your inspiring question to you’re testing – to not just pretend 14 Go to a testing conference but be bring it out in 10 minutes of to be a user, but find a way they sure to hangout with other testers thinking and rapid typing.) would actually use it themselves. AFTER the conference day is done.7 Ask them what movies or actors 15 Allow them a safe space to fail inspire them. but also show their smarts. Next issue we’ll feature Michael Bolton as8 Solve one problem for them OR 16 Invite them to give YOU a our guest expert. No, not the singer, the allow them to solve one problem brainteaser or puzzle that YOU consulting software tester; read about him at www.developsense.com. for you. have to solve in front of them. Volume 7 / Issue 5 / June 2010 27
  • 27. RAISINGTHE | BAR28 / www.softwaretestpro.com
  • 28. RAISING THE BARON QUALITY WEB PERFORMANCE:Best Practices to Find and ResolveProblems Inside and Outside the Firewall By ImadMOULINE CTO, Gomez, the Web Performance Division of Compuware QA and Testing: A Critical Line of Defense As a QA/testing professional, you are responsible for protecting your organization’s brand, customer experiences and online revenue by finding and resolving problems before your web end-users are impacted. However, to date, it’s estimated that 57 percent of application performance issues are actually found by customers(1), and that 65 percent of applications are deployed without proper load testing to ensure they can scale to support desired traffic levels(2). The key to ensuring a high performance website or application, even under load, is to realistically load test from the “outside-in” — from where your end- users and customers are — so you can find and resolve more problems both inside and outside the firewall. This is the only way to realistically “stress test” across the entirety of today’s highly complex and dynamic web applications. Ultimately, this gives QA and testing professionals a direct hand in determining business outcomes through the delivery of consistently superior web experiences, regardless of traffic volumes. The Web Performance Imperative There’s no doubt that poorly performing websites and applications can have a direct impact on the bottom line. For example, Amazon has correlated every 100 millisecond delay in processing web page views to a one percent revenue decline for the corresponding time period(3). In addition, Microsoft’s Bing found that a two second slowdown reduces revenue per user by 4.3 percent. Considering the traffic these sites generate, that’s a significant amount of business. This impact is growing more pronounced every day, as end-user expectations for superior Web experiences soar to new heights. Quite simply, today’s web end-users have little to no tolerance for slow page downloads and unreliable web applications, even during periods of peak traffic when they know a site is likely inundated with visitors. As evidence, a recent survey conducted found that 67 percent of consumers expect websites to work regardless of how many visitors a site may have at any given time. In addition, 78 percent indicated they will readily switch to a competitor’s site if they encounter slowdowns, errors and transaction problems. 1 Aberdeen Group- “Customers are won or lost in one second” 2 OAUG ResearchLine study, focused on the value of monitoring, testing and quality assurance of Web applications, conducted by Unisphere Research in partnership with Oracle Corp. 3 Greg Linden presentation, “Make Data Useful,” delivered at Velocity Conference 2009. Volume 7 / Issue 5 / June 2010 29
  • 29. RAISINGTHE | BAR The Challenge of Increasing Complexity Here’s a sampling of the performance issues that a With stakes this high, QA and testing professionals behind-the-firewall approach may miss: have a huge role to play in terms of making sure web n Third-party services: poor performance pages and applications deliver consistently excellent under load, blocking content delivery, performance throughout peak traffic periods. But a service level agreement (SLA) violations primary challenge is that today’s feature-rich websites and applications include components, content and n Major ISPs: network peering problems, outages services delivered from not just the data center but also n CDNs: oversubscribed POP, poor numerous third-party providers. Consider an online sales routing optimization transaction, which may include shopping cart, product catalog and check-out functionalities from external n Local ISPs: network peering problems, third-parties – in fact, it’s estimated that the average inconsistent connectivity web application today connects to 8.17 distinct hosts n Browsers: browser/device incompatibility, to complete a transaction. In addition to third-party page size too big, poorly performing JavaScript services, it’s likely your applications traverse a complex web application delivery chain path (including ISPs, Regarding third-party services specifically, your tests CDNs, desktops, mobile devices and browsers) before need to “drill down” to reveal how each application ultimately being delivered to end-users around the world. component performs under load. This can be the The fact is, poor performance anywhere in the chain can key to enforcing third-party SLAs and optimizing an degrade the end-user experience and reflect poorly on application’s performance as a whole. But you also you. In other words, when any “link” in the chain fails to need to test performance from the “outside in,” i.e. scale and performance dips as a result, end-users won’t from the end-user’s perspective – the last step of the care who or what is the faulty component – they will Web application delivery chain and the only place simply hold the site owner responsible, inflicting harm you can get a realistic view into the performance of upon the brand, customer satisfaction and revenues. applications, when subjected to an extremely wide range of possible Web “noise.” This increased complexity essentially requires QA and testing professionals to proactively manage performance and stress-test across the entire range of components Ensuring Performance for Key Geographies that ultimately color end-users’ experiences, including Whether you know it or not, the performance of components from beyond the firewall. QA and your web application varies for end-users around testing teams that today use only lab-based load and the world depending on their location and unique performance testing need to re-assess their testing connection scenarios, i.e. local ISP, high broadband, methodology and ask three fundamental questions: dial-in, etc. So any optimization efforts must begin with a measure of how your web application is performing n Are we testing all of our internal and for these geographically dispersed end-users. Only external components? then can you identify which end-user segments may be experiencing a performance degradation and then n Are we incorporating a geographical load trace back through the web application delivery chain testing element into our testing plans, to verify to identify, isolate and resolve the issues. performance for our key regions and markets? Some QA and testing professionals are already n Are we focused on validating the end-user taking a step beyond inside-the-firewall testing by experience, or simply stress testing our generating and applying production-equivalent loads internal systems? using external Internet backbones or cloud locations to assess application performance impact. While a first step towards understanding the end-user What Internal Testing Approaches Miss perspective, this approach fails to test across all the Traditional inside-the-firewall testing tools that QA performance-impacting variables that ultimately and testing professionals might currently use are built comprise the full application and assemble in the on the philosophy of generating load and measuring end-user’s browser. So this approach still falls short performance behind the firewall. While valuable, this in terms of revealing how load truly impacts the kind of internal testing only tells part of the story experience of key end-user segments. since it only helps identify problems rooted in the data center. More often than you may think, when testing teams leverage load generated from the cloud only, the average However, your end-users don’t live in data centers — response time metric can remain under the target goal, they are located around the world, at the end of a long hiding major discrepancies and response time variations and complex web application delivery chain. When you across regions. For example, a cloud-generated load test generate load and measure performance behind the might report that the application’s response time meets firewall only, you have no visibility into the true end- a target goal of four seconds. But this information may user experience, or how elements beyond the firewall hide major discrepancies and response time variations will scale under load. And all components and services across regions. What if peak load causes a slowdown need to perform optimally in order to deliver quality for a localized ISP, meaning that end-users in a key Web experiences across all end-users and regions. revenue-generating region are actually experiencing a30 / www.softwaretestpro.com
  • 30. Meet the Teamresponse time of 10 seconds or higher? The only way to identify this type ofproblem is to combine load generated from the cloud with load generated fromreal end-user desktops in these key regions around the world.As a QA/testing professional, you need to partner with your businesscounterparts to understand your organization’s key target markets – andthen ensure that your load testing program is set up to measure howyour application will perform under load for those markets – before anymajor application launch, new code, or major marketing campaigns areimplemented. This proactive strategy can help you identify and remedypotential issues and optimize performance for key end-user segments. For 1example, you may be able to configure your web applications for optimalperformance in a region’s most predominant browser (Firefox for Europeanusers, Internet Explorer for North American users); or, you may determinethat your cloud service provider has become a bottleneck, and use thisknowledge to prompt additional infrastructure investments.Tapping the Necessary Testing ResourcesThe testing approaches described above present additional challenges worth 2noting. Specifically, inside-the-firewall approaches often require dedicatedhardware to execute tests, which may not be available or is too costly. Inaddition, those using the cloud for load testing should be aware that thecloud is, by its very nature, a shared resource which means testing oftenhas to be pre-scheduled and often delayed.These challenges – combined with the other shortcomings already discussed– should compel QA and testing teams to consider test and measurementsolutions that are available in a SaaS model. These solutions leverage 3worldwide testing networks to provide organizations with an on-demandview into end-users’ experiences, the first step of an “outside-in” approachand the only way to truly load test the entire web application delivery chain.These solutions also require no additional hardware or software purchases,and make testing services more readily available whenever they are needed.This is especially important when you are working on high-volume testinginitiatives that require thousands of concurrent virtual users to accessyour application, or when time-to-market is of the essence for a revenue- Rich HANDgenerating web page or application. 1 Director of Membership & Publications rhand@softwaretestpro.comConclusion Rich has 30 years of business experience and is most often found evangelizing the benefits of membership.Best practices indicate that load testing across the web application delivery He spent 7 years at HDI and spends most of his daychain, from the perspective of your geographically dispersed end-users, writing articles, creating webinars, and preparing contentis critical to ensuring optimal performance. An “outside in” approach to for the membership.load testing can help QA and testing teams guarantee their web pages andapplications will deliver when they’re needed the most whether your testing Yvonne HAMILTON 2 Marketing Communications Managerefforts are focused on releasing new features or applications; adding third-party services; launching new marketing campaigns; or getting ready for yhamilton@softwaretestpro.compeak traffic. In this way, QA and testing professionals can assume a pivotal Yvonne is the newest member of the team butrole in driving desired business outcomes for their organizations. comes with a large repository of marketing and brand management experience in professional services. She can mostly be found combing through content and editing Rich’s writing to make sure he actually sounds like he knows what he’s talking about. Scott HANSON 3 Creative Director shanson@softwaretestpro.com About The Author The most creative member of the STP team, Scott spent 8 years creating for HDI. Every visual Imad Mouline is the CTO of Gomez, the Web performance division of Compuware. He is you encounter at STP in our newsletters, magazine, a veteran of software architecture, research and development and is a recognized expert and conferences was created by this man. in web application development, testing and performance management. His breadth of Most often he can be found riding his Harley 2i&B knowledge spans web 2.0, cloud computing, web browsers, web application architecture around the Rocky Mountains of Colorado. and infrastructure, and SaaS. Prior to Gomez, Mouline was CTO at S1 Corp. Volume 7 / Issue 5 / June 2010 31
  • 31. “ DO NOT TRY TO USE LOGIC A ND RE ASONING IN YOUR DISCUSSION. EXPLORATORY PERFORMANCE RE A LIZE TH AT NOTHING SAID IS COMING FROM E XPERIENCE – A ND YOU TESTING ARE NOT REQUIRED TO DO A NY THING OTHER TH A N ACK NOWLEDGE THIS.” By GorankaBJEDOV H ave you ever found yourself in a situation where you are organizing functional testing of a product, when suddenly, somebody comes up and asks about performance? And before you even had a chance to explain that you haven’t even started thinking about non-functional requirements, they have launched into a long discussion about how important this product is, how detailed the tests need to be, and are talking about hundred thousand users simulation with correct distribution of use cases... Before you know it, you are in a meeting, requirements are being thrown out like snow-balls, and every person who is even remotely associated with the product has strong opinions on what you should do, how it should be done, which tools About The Author should be used, how the results should be reported and so on. And, if you have ever done any performance work, you realize that the things these people are Known for her technical talking about will take months, if not longer, while they have made it very clear achievements and her they expect you to provide them with the results in a few weeks. refreshing wit, Goranka Bjedov has worked at Google Welcome to my world. I have been doing performance testing for Google since where she has planned and January 2005, and have dealt with more Google services than any other person implemented product testing at the company. (I do not really have proof of this, and it is certainly possible that and has been responsible for there is a person out there who has secretly tested more than 50 products, but risk analysis and assessment I doubt it.) So, here is my story – what have I learned, what works, what does for the past five years. Her not work, and why I am now comfortable doing things that would have probably current testing interests caused me to break into a cold sweat ten years ago. include both server and client- side performance, robustness, and scalability testing for Web- based applications. Before Google, Goranka held senior What Does Not Work test engineering positions at This was probably the most difficult lesson for me to learn, and I still have an Network Appliance and AT&T extremely hard time accepting it. I wish it were not true. I wish I were imagining Labs. Previously, she was things. But, rule number one, if you find yourself in the meeting I described above an Associate Professor in is: Do not try to use logic and reasoning in your discussion. Actually, do not discuss the School of Engineering at anything at all. Realize that nothing said is coming from experience, instead, people Purdue University. A speaker are talking about their hopes, their ideas, their emotions – and you are not required at numerous testing and to do anything other than acknowledge this. Wait for the conversation to die down, performance conferences, and once it is done, acknowledge everything that was said as if it made sense, tell Goranka also has authored the team that you will immediately start working on what they have suggested many papers, presentations, (using their scenarios, their tools, their ideas, etc.) however, you suspect the work and two textbooks. will take a very long time. So, in parallel, you will quickly throw together a couple of performance tests to get a feel for the system, to help better design detailed scripts and guide the overall performance testing process. By this point in time, you probably know who is the development lead and who are the developers working on most critical operations. You probably understand what are the critical functions, and even though you may not know everything as accurately as you would like, you can do an amazing amount of useful performance work, in a short time, as long as you do not get involved in long discussions about what and how it should be done.32 / www.softwaretestpro.com
  • 32. EXPLORATORYPERFORMANCE | TESTINGLesson Learned: System PerformanceDo not waste time trying to explain to people why 120they are wrong. They are not thinking and analyzingthe situation, the data, time-lines and what can andcannot be done. They are reacting emotionally to 100the wrongly perceived risks and importance of theirservice. Same as you will never meet a parent who willdescribe their child as below average, not very smart or 80completely spoiled, you will never find a developmentteam that will feel the same way about their baby/project. Coming in, and pointing out that the baby is 60ugly, isn’t very important and could be left in a cornerunattended, will certainly not win you any friends.(Trust me, I have done this many times and it never 40led to a satisfactory resolution.) 20What WorksWhat works, invariably, is agreeing with thedevelopment team in principle. Yes, their product 0is very important (to them). Yes, their product is 0 5 10 15 20 25 30very complex and difficult (is there a softwareproduct that isn’t?). Yes, the time-lines are very tight Load [threads]and will require all of your time and concentration, latency [ms] throughout [t/s]so, you will need the team’s full cooperation and helpto make progress. Get the list of names of people who FIGUREshould be informed on a daily basis of what is going System Performance Graphon. Get the contact people in case you need a question 1answered quickly. Never discuss finding problems,because, clearly, their product will have none.Instead, make the request from the perspectiveof your inadequacy – in case you get stuck notbeing able to understand what just happened, Resource Usagewho is the person that will be available to help?By this point in time, you will have a list of operations/ 100transactions the team considers important. You mayhave to add one or two that you feel are missing.There is no hard and fast rule here, but if there is 90a transaction that somebody considers important,and you can include it in your tests, do so. Try to 80limit your test to 10-15 transactions at most, becausemore than that could make for more complex tests.Remember, this does not mean you will not be able 70to change this later, it just sets your starting point.You are now ready to start exploratory performancetesting, and gain an understanding for how your 60system behaves. By its nature, exploratory testingdoesn’t follow a prescribed algorithm – you tend tomodify your future actions based on the results you 50have just obtained. However, there is an overall goalthat you are trying to accomplish, and your actionswill be chosen with that goal in mind. 40 0 5 10 15 20 25 30End Goal LoadWhen I finish performance tests, I give my team the CPU Memorytwo graphs that show overall behavior of the system asa function of increased load. Additionally, I frequently FIGUREprovide a list of performance data for every individual System Resource Consumption Graphtransaction in a tabular format – I find that people 2quickly catch on to the concerns I have if the datais logically graphed, and also presented in a table. Note: the graphs shown above are not from anySo, what I am hoping to accomplish is something particular product or service, instead, they are createdlike the graphs shown in Figures 1 and 2: by me to show a reasonable system behavior which you may encounter. Volume 7 / Issue 5 / June 2010 33
  • 33. I find these graphs incredibly useful, and, in my Instead of focusing on getting the perfect performance experience, every development team I have ever worked testing load, I will look over the application and try and with has felt the same. The path to obtaining the graphs, sort though the different categories of things a person provided everything works well, is relatively simple and could do on the site: straight forward. The overall benefits of preforming this work, even with imperfect load, is multi-fold: 1. a user could search for a single word term 1. it is easy to visually inspect the behavior of the 2. a user could search for a specified phrase system and see if it makes sense 3. a user could search for multiple words 2. it is easy to see why/where the system is limited, 4. a user could perform any of the built-in which allows the development team to make better calculations or conversions choices in future changes 5. a user could look at the different page of 3. it is easy to pick a good load for a benchmark results, once those are presented 4. it is easy to describe the behavior of the system 6. a user could leave the site at any point in time to non-technical people 5. almost any load is likely to find big functional problems: Assuming I have absolutely no information, I would deadlocks, bottle-necks, priority inversion, etc. make some guesses: maybe 50% of the users will type 6. you have a better understanding of relative costs (in a single word search. Another 30% will search using terms of system resources) of different transactions multiple words, while only 10% will type an exact phrase. The remaining 10% of the users will use the built-in functions. Why do I pick these numbers? No particular Starting Exploratory Performance Testing reasons, I need to pick something, this makes sense to I would like to illustrate my approach on a simple me and I am perfectly happy to proceed. Somebody else service which I use for demonstration purposes would pick a different set of percentages, and that would frequently: Google Search. have been perfectly fine. Think of this as making a cup of coffee on a new coffee-maker and with unknown coffee beans. You will make a guess for how much sugar/milk/ cream/etc. you want. Your guess is not better or worse than the guess of a different person, using the machine right before or after you. You will end up with a cup of coffee, and if the cup is terrible, you will repeat the process and change some of the parameters. I will run these preliminary choices by a person more familiar with the service – just to see if I am completely off. If necessary, I will adjust my guesses and proceed. Creating the Performance Test Script For a project like this, I would use the tool called JMeter (http://jakarta.apache.org/jmeter/). However, even in this category, it would be difficult to go FIGURE Default Google Search page for user wrong – there are many excellent performance testing 3 goranka@google.com tools provided by both vendors and the open source community. Use whichever tool you are most familiar Let us assume that I do not know anything about the with. In addition to JMeter, I will do most of my search patterns, have never worked on it, and have no scripting using FireFox browser and I always make logs from production to guide my work (not an unusual sure that LiveHTTPHeaders extension is installed and situation to be in). Where to start, what to do? This is enabled. Note – these are my personal choices and a very important crossroad which can confound the they work for me – since I do most of my work from a performance tester and stop any progress. Clearly, Linux workstation, I find them essential. However, you what you want to do is pick the right transactions, add can find an equivalent set of tools for every other major the right time delays to those transactions, have the operating system, and I would definitely suggest doing right distribution of users simulating exactly what will some reading on the topic and exploring if you have be happening in production... or do you? never done any work in the area before. There is nothing wrong with having all these things as In many cases, you can use JMeter’s recording ability close to the production as you can get them. But, if you to create your first performance testing script. While I have no information, you run the risk of searching for never use the recorder, when teaching JMeter, I really something that does not exist and wasting the time you like the ability. Basically, you can insert JMeter as a do not have. You do not need all these things. What you proxy between your browser and the outside world, and do need, instead, is some load – any load will do! – it will automatically record every request the browser and you need to report your results correctly. makes and allow you to obtain your first script.34 / www.softwaretestpro.com
  • 34. EXPLORATORYPERFORMANCE | TESTINGMy personal preference is to use LiveHTTPHeaders and consumption data I may be interested in: memory usage,look at the traffic between the browser and the Web, CPU usage, disk I/O, and anything else I may need.and script the requests directly. Depending on the operating system and the performanceI tend to do most of my performance script development testing tool you have chosen, you may have to do moreiteratively: I will script one HTTP request at a time, play or less work in this area. Most vendor tools come fullyit back, correct any errors and continue until all of my equipped with the monitoring capabilities, while mostrequests have been scripted. Both the recorder and the open source tools require you to solve this problemLiveHTTPHEaders will catch all of the AJAX requests as on your own. I have a very strong preference for usingwell, and will allow me to adjust my script accordingly. whichever monitoring solution will be in operation once the service is deployed and in production – so in my case, I tend to use Google specific performance monitoring solutions, which will not be shown in this paper. How long should the performance test run for? This is another one of those experience variables that different people will have different solutions for. I still have a preference for 2 hour runs, although in a lot of cases I find I can get very good information in runs as short as 10 – 15 minutes. You could say the answer depends on the level of risk you are willing to tolerate, however it should always be longer than the time it takes for the system to reach its steady state. This means that you will have to warm up your caches, pre-populate your database, and do anything else necessary to bring the system to the state you expect it to be in during production. FIGURE The results from the verification test run with a single A very simple JMeter script thread repeating five times are shown in Figures 5 and 6. 4A couple of things to note about the script:1. At this time, it has only one HTTP sampler. You can control the mix of requests by controlling the queries in the file, you could add multiple different samplers and control the throughput of each or you could have different thread groups with different number of threads. What you choose to do depends solely on your preference. In this case, to make the script very simple and fit on one page, I am controlling my query mix through the query file.2. All of the defaults are specified in the HTTP Request Defaults, and can actually be read from the command line when starting the script. This allows for easy execution in different environments, and provides reasonable defaults. FIGURE Verifying the script is working correctly3. There are three different listeners – View Results Tree, Aggregate Report and Summary Report. During 5 with View Results Tree the regular performance run, I would disable View Results Tree listener, as I tend to think of it as a perfect debugging tool. It allows me to see every bit of information exchanged between the server and the client, and it also has an option of presenting the data in HTML format. (Note: if your application is JavaScript heavy, you may have to use a lot of imagination to recognize it, however, I find this ability quite useful for quick inspection during the script development time even in that case.)Running the Script and Collecting the DataMy first run is always with the smallest possible load FIGURE Performance test results for the verification– one thread. Additionally, I will set up the servicemonitoring, and collect all of the basic resource 6 run using Aggregate Report listener Volume 7 / Issue 5 / June 2010 35
  • 35. Note: In many situations, I receive requests to transform the system performance graph into a latency vs. throughput 1. Figure 5 shows that there were a total of five graph. Clearly, this can be done easily, but I find the requests made, all of them successful (marked by resulting graph misleading and prefer to look at the a little green triangle with a check-mark inside) graph as shown in Figure 1. While the reason for this 2. The tab of the View Results Tree is set to Response may be a simple nitpick, I believe that the x-axis of any Data tab, and instead of displaying the results in line graph should show independent variables, while the y-axis shows dependent variables. Since the only simple text (default) I have selected the Render HTML thing I actually control is the number of threads making button on the bottom of the screen, allowing me to requests and not the throughput itself, I prefer to leave easily see if the results match my expectations. this graph as shown, since it more accurately displays 3. The resulting image does not look like the results what actually goes on. This is my preference – you are Google would return (JMeter will not render JavaScript certainly free to do whatever you like. correctly), however, I can easily see that the query is The really interesting performance work starts here. «performance testing» it was read from the file correctly, As I mentioned, this is a set of graphs obtained on a and the results returned match what I expected. well-behaved system, so we are not even looking for potential problems. What the project team is usually 4. The Aggregate Report listener, at the same time, interested in are the following questions: correctly post-processed all of my performance run data, and calculated both the throughput, 1. What is the maximum throughput of our system? and major latency statistics. 2. What is the maximum sustainable throughput of our system? Post Processing your Performance Run Data 3. What load should be chosen as a maximum Most of the reasonable performance testing tools will value for safe operation? automatically post-process your transaction data 4. If the system needs to be modified in order to and calculate average transactions per second the increase its performance characteristics, where system sustained during the run. Additionally, you should we start? will find several different latency values – minimum and maximum, average, median, 90%, etc... It seems 5. How can I tell this is a well-behaved system and very natural to take the average or median as your I am not missing a huge underlying problem? representative numbers and move on. Unfortunately, this is a very bad decision which will prevent you from 6. What would be a good load for a performance understanding the real user experience. 90% latency is benchmark? the typical value the system should be judged by, and in All of these questions can be answered by analyzing our some cases, a system may require even higher standards (95% or 99% users received a response in the amount of graphs shown in Figures 1 and 2. time less than X seconds is not an unusual requirement). What is the maximum throughput of our system? Obtaining these values is not particularly difficult even if This question is relatively easy to answer – you can just the tool doesn’t provide them for you – all that is needed read it off the throughput graph on Figure 1. Maximum is sorting the data set and reading off the results. throughput is the top of the hump shown in that graph System resource consumption data is simply averaged (about 35 transactions per second in the graph shown). out. These four values (90% latency, throughput, Considering that I am peacefully discussing «the hump» average CPU usage and average memory usage) are and that I have already stated this is a well-behaved the beginning of the final performance graph. For the system, you can infer that I am not surprised to see smallest interesting load (one thread/virtual user) it there, and that I do not consider it a side-effect of a going as fast as possible (not including any thinking bad test execution or some other anomaly. I expect to time between requests) the system is represented by these four values. These values are the beginning of the see the hump (it may be more or less pronounced) in graphs shown in Figures 1 and 2. any system that is designed to queue requests which cannot be processed as soon as they arrive. Which tends to be most of the systems I work with. Completing the Preliminary Scalability Test “The hump” is the overhead you pay for creating, The scalability test consists of many individual maintaining and processing the queue. Prior to this performance test runs with different loads. After each point (to the left of the hump on the graph) the system run is post-processed, the machine consumption data was fast enough and it processed requests as quickly (averages for CPU, memory and anything else you may as they came in. The top of the hump is the optimal find important) and the system performance data (90% operation of the system – the requests are coming in at latency and throughput) are graphed either individually maximum processing rate, thus not triggering the need or combined into two separate graphs. My personal to queue but using all the processing power to respond. preference is to create a machine consumption graph with all machine variables on a single graph and a The only useful value of analyzing the hump, however separate graph showing throughput and latency as is understanding how expensive queue keeping is. functions of load. See Figures 1 and 2 again. I expect to pay about 5% in maximum throughput, but if I see the cost being more than 10%, it is something36 / www.softwaretestpro.com
  • 36. EXPLORATORYPERFORMANCE | TESTINGI will discuss with the development team. Keep in mind, How can I tell this is a well-behaved system andperformance testing is interesting only when it comes I am not missing a huge underlying problem?to the analysis part. It is interesting precisely because You cannot. If you are concerned, once you have all ofthere are no hard and fast answers and every data these runs completed and all the graphs ready, by allpoint, graph or result obtained needs to be understood means, consult a performance testing expert. However, inin the context of the system analyzed. my experience, most common performance problems areThe only other thing the hump tells you is where easy to see and even diagnose during these exploratoryyour queuing system kicks in. If you are familiar with performance runs, and they will show themselves the firstqueuing theory, however, this will also tell you that the time you start a run with multiple parallel threads. Mostlatency of your requests to the right of the hump will often, you will notice your error rates going up – a clearbe dominated by the length of the queue, and will tell indication that the system is not behaving as it should.you little about any other part of the system. Thus,choosing the benchmark load that is to the right of What would be a good load for a performancethe hump makes sense only if what you are interested benchmark? Usually, to get to the point where youin is the performance of that particular subsystem. have Figures 1 and 2, and a well behaved system, youWhat is the maximum sustainable throughput of will spend some time debugging the problems, workingour system? That question is also not too difficult – with new patched up versions and iterating. The processthe maximum sustainable throughput is the tail is interesting, and typically provides a lot of valuableend of the throughput graph on Figure 1 (about 32 feedback to the team. However, what you can do oncetransactions per second in this case). This is an the exploratory performance testing is complete,important number – because it tells you how fast your where you work really shines and provides the mostsystem will be able to cope with requests coming in, value is if it is turned into a well-selected benchmark.even when overloaded. For capacity planning purposes, Basically, you are creating a «unit» performance test,this should be the absolute maximum number you ever that can run on a frequent (nightly?) basis and canreport, because your system will not be able to support detect performance degradation quickly. With this typeany higher throughput. of information, you never put your development team in a situation where they have to sort through monthsWhat load should be chosen as a maximum value of code changes looking for one or two lines that havefor safe operation? This is a tricky question, and seriously degraded system performance.it very much depends on things that have not beenaddressed yet. The true answer is – it depends. What So, what should a good benchmark do?is the importance of the system? What are the risks 1. It will target the parts of the system we are interested inassociated with the system not being available or 2. It will try and eliminate any environment related noiseresponding very slowly? The answers to thesequestions will greatly influence the answer to 3. It will try and simulate all parts of the system thewhat the maximum value for safe operation is. way we expect them to run in productionHowever, for Unix-based systems, from experience, My rule of thumb is very simple – a good benchmarkI will pick a point where no system resource (memory, (unless you are interested in evaluating your queuingCPU, disk I/O) is operating at higher than 80% of sub-system, of course) should be to the left of thethe maximum load, on average. The reason for this is maximum throughput hump in Figure 1. So, technically,fairly simple – if overloaded for an extended period of anything below 20 threads (in this example) would betime, Unix OS itself can become flaky and not behave acceptable, anything above it not. I usually select a loadthe way you expect. Once that happens, you will run that corresponds to whatever has made any averageinto strange behaviors that are difficult to reproduce, system characteristic close to 80%. Looking at Figure 2,analyze or debug. So, from experience, I will pick the that would indicate something around 15 threads. I was80% maximum usage as my rule-of-thumb cut-off recently asked why I would not select something muchpoint. For the system described by Figure 1 and 2, lower than that – wouldn’t 5 threads do as good of a job?this would be somewhere around 15 thread load. Truthfully, there is no reason why you could not runIf the system needs to be modified in order to a benchmark with 5 threads only. Chances are, youincrease its performance characteristics, where would be able to detect performance degradations inshould we start? A secondary benefit of completing that setup as well (there is no theoretical reason whyexploratory performance tests for a system is finding the idea should be rejected outright.) The reason why I prefer to use a relatively high load is twofold:out additional information about the system that canbe incredibly useful for future benchmarking efforts. 1. It is more accurate representation of what theFor example, the system represented by Figures 1 system is likely to experience once deployedand 2 is clearly CPU limited. This means that any 2. At higher loads, you are more likely to trigger additionaloptimizations in the code should be done to trade off problems which may remain hidden otherwise. Whilememory benefits for CPU benefits. It also means that this is not strictly a performance issue, it certainlyany increases in CPU utilization will impact the system makes sense to test as much as possible.throughput much faster than changes in memoryutilization, and therefore, benchmark tests should bemore sensitive about increases in CPU usage. Happy Performance Testing! Volume 7 / Issue 5 / June 2010 37
  • 37. SOCIAL | NETWORKING PEGGY LIBBEY You can reach Peggy at: plibbey@softwaretestpro.com vs. Associations Social Networking Associations and Social Networking — are they the same, are they somewhat similar, or are they completely different? To answer that question, consider the definitions for specific topic. A closer look at the networking taking each according to Google web definitions: place will probably reveal that much of the activity is generated by vendors, recruiters, and individuals interested in self promotion. There is definitely activity An association is a formal organization of that could be described as valuable. The availability of people or groups of people. It can also be targeted information that is “free” on the web is at an an act of consorting or joining with others; all time high. One simply has to have the time to find the state of being connected together or the it if in fact, it does exist in a form that is available. process of bringing ideas or events together in memory or imagination. Now consider how an association functions. Social networking is the interaction between An association can also provide information and people who share a common interest. More the opportunity to network with others. However, specifically, it is using internet network remember an association is a formal organization groups (such as Facebook and Twitter) which means there is more structure and the to network and communicate using shared foundation on which it is formed is more specific interests, related skills or geographical and targeted. An association may provide the location between consumers and businesses. opportunity to network virtually, but it will also provide opportunities to network in person through local meetings and annual conferences. It will Certainly the definitions would indicate that they are also focus specifically on improving the careers of at least similar. The advent of social media sites has individuals, the efficiencies of organizations, and made virtual networking much easier for individuals provide validation to ideas and practices within an with similar interests; but are these social networks industry. All of this serves to improve the reputation a replacement for the professional association? of the profession served by the association. Social networking tools are an inexpensive way Professional associations focus on enabling the for individuals to connect with others of similar members of the community to create, validate, interests. A web search will reveal thousands of groups and publish the results of the activities conducted with thousands of members seemingly “networking.” within the association framework. Associations are Keep in mind the difference between social networking a place where professionals gather to share, learn, for purely social reasons and social networking for build relationships, contribute knowledge, and professional reasons where the goal may be to find validate the industry’s direction with the goal of the answer to a problem or information about a improving the careers of the individuals.38 / www.softwaretestpro.com
  • 38. THECARTOON | TESTERAssociations are about creating anenvironment of trust between the membersof the community.A member of an association demonstratesa commitment to not only their own career,but also that of the industry in whole.An association provides the platform tobuild trust and reputation through not onlyonline mediums but also networking events,educational offerings, conferences, and more.The knowledge base of the industry as offeredby the association is continually changing.The changes are validated as the knowledgeis debated and vetted by the communityto help create the accepted practices,methodologies, training courses, certificationstandards, and terminology used withinan industry.Associations reinvest in the membershipand industry they serve. They go beyondproviding a technology platform and a placeto gather; they take the information andknowledge that is shared and turn it intodocumentation that becomes the body ofknowledge for an industry. Associationsbecome an archive of work, a “genealogy” ofideas that trace the development of a functioninto a leading edge profession that can anddoes change the course of entire industries.Each association is unique in that it respondsto the needs of its members. What may beappropriate for one industry or professionmay not necessarily adapt well to a differentassociation. The association is ultimatelydefined by its members.As we embark on the launch of anassociation focused on software testprofessionals, we are making a commitmentto providing the opportunities outlined above.We look forward to a mutually beneficialrelationship as we encourage commitmentfrom our professionals—commitment atwhatever level is right for you. With yourhelp, we intend to build an association thatwill not only offer you the professional About The Cartoon Testerdevelopment and support you need, but Andy Glover is a software test team leader, in thiswill also offer many opportunities for you role he tests windows and web applications for theto participate in a community of your pharmaceutical industry as well as manage a teampeers in a profession that continues to of 5 software testers. He follows a number of testinggrow in importance. blogs and when ever possible he adds posts and comments in test forums. This year Andy has started a blog, the cartoon tester, where he posts cartoons relating to testers and testing in the hope that some will provide a lesson or insight into the craft of software testing. http://cartoontester.blogspot.com Volume 7 / Issue 5 / June 2010 39