U test whitepaper_10_tips_web_app_testing

  • 354 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
354
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Ten Tips of Web App TestingHow to test and launch a world-class web applicationWHITEPAPER: March, 2010
  • 2. Table of ContentsIntroduction: Testing on a Moving Target…………………………………………………………………………………………….……………………….…….… 2 New Challenges, Old Solutions …………………………….………………………………………………………………..…………………….………………..……….. 2 The Crowdsourcing Advantage………………………………….………………………………………………………………………………………..……………………. 2 Testing Types………………………………………………………………………………………………………………………………………………………………….……………………….. 3 Functional Testing………………………………………………..……………………………………………..…………………………………………………………..……. 3 Usability Testing……………………………………..…………………………………………………………….…………………………………………………………..……. 3 Load Testing ……………………………………………………………..………………………………………………………..………………………………………………..……. 4 Web Testing Checklist ………………………………………………………………………………………………………………………………………………………………..……. 4Web Testing Tips: Launching a Higher Quality Web Application……………..……………………………………………..….……….. 5 1. Don’t Be a Slave To Your Metrics ………………………………………………………………………………………………………………………………..……. 5 2. Know Thy Third Party Apps ……………………………………………………………………………………………………………………………………………..……. 5 3. Understand The "Testing Managers Dilemma" ……………………………………………………………………………………………..……. 5 4. Invest In Testers, Then Tools…………………………………………………………...……………………………………………………………..…………………… 6 5. Protect Users, Save Yourself ……………………………………………………….…………………………………..………………………………………………….. 6 6. Multimedia, Multi-Problems ……………………………………………………………………………………………………….…….………………………….……. 7 7. Forfeit The Blame Game ……………..………………………………………………………………………………………….………………………………...…………. 7 8. Be Everywhere …………………………………….……………………………………………..……………………………………………………………………………………..…… 7 9. Plugins: Proceed With Caution …………………………………………………………………..………………………………………………………………..…… 8 10. Respect The Dead (IE6), Make Way For The New (HTML 5)……………………………………………….……………………. 8About uTest……………………………………………………………………………………………………………………………………………………………………………………………………. 9 "Why is there never time to do it right, but always time to do it over?" - Colonel Mike Mullane Retired NASA AstronautWHITEPAPER: 10 Tips of Web Testing 1
  • 3. Testing on a Moving Target (the Web)New Challenges, Old Solutions for Testing Your Web ApplicationEstablishing procedure amidst a constant state of change may seem like an absurdity (and a losingbattle), but this is what today’s software companies face when it comes to testing their webapplications. Aside from an expanding matrix of new browsers, plugins, third party apps, programminglanguages and more, there are now thousands, if not “Complexity kills. It sucks the life out ofmillions, of new users from all around the globe. Ofcourse, none of this was in the manual! developers, it makes products difficult to plan, build and test, it introduces securityThus, the purpose of this whitepaper is twofold. While challenges, and it causes end-user andour ‘10 Tips’ section will help you stay on top on of the administrator frustration.”latest trends in web app testing, our introduction willoutline the basics of functional, usability and load - Ray Ozzietesting for web-based applications. Along the way, Chief Software Architect, Microsoftwe’ll cover topics such as:  Third Party Apps: Your product is only part of the equation - what about the satellite apps that are orbiting your web application?  Cloud Computing: With opportunity comes risk. There’s a ton of upside to cloud computing, but what happens to your web app when it fails to functional properly?  Multimedia: How will your web app perform for users with different versions of Flash, Acrobat, Java, etc.? Have you thought about HTML5 and the challenges of managing H.264 vs. Theora?  Plugins: While the use of plugins may be slowing, the use of extensions are not. Learn why plugins and their extensions should be a testing focal point.  Localization: Obviously, your application needs to work where your users reside. Discover why most software companies underestimate the importance of localization in their web testing.  Deadlines: Your web application will never be perfect, so how will you know when it’s safe to launch? It’s called the “Testing Managers Dilemma” and we’ll show you how to deal with it.  Metrics: As one testing expert put it, “When your car is about to go off a cliff, it’s a weird time to be thinking about gas mileage and drag coefficients.” Find out how metrics can mislead.The Crowdsourcing AdvantageWhile the following material is suited for companies that rely on traditional staffing models for testing(i.e. internal QA teams or offshoring firms), this whitepaper will be especially useful for those wholeverage some aspect of crowdsourced (or community-based) testing.To learn more about the growing trend of crowdsourcing - and how companies of all sizes can use it tocomplement their in-house efforts – check out the 8 Essentials of Crowdsourcing whitepaper.WHITEPAPER: 10 Tips of Web Testing 2
  • 4. Web Applications: Testing TypesFunctional TestingNo surprises here. The most frequent and critical task that QA takes on before launch is to performcomprehensive functional testing. This can include - but is not limited to - testing of:  Web Forms: This includes sign-up, sign-in, contact and purchase forms; database queries, checking for cross-site scripting/AJAX vulnerabilities and other issues related to incorrect inputs.  Links: Ensuring that all outgoing and internal links are functional and accurate.  GUI: Despite back-end test automation, front end “For a successful technology, reality problems with web applications are almost always discovered by human testers, including issues must take precedence over public related to browser compatibility or CSS flaws. relations, for nature cannot be fooled.”  Language: Does your web app properly support Unicode? Can it accept and handle foreign - Richard Feynman characters? This is a growing concern for Renowned Scientist developers and should be considered as an essential part of your testing efforts.  Security: Security testing – including checks for open redirects, cookie tampering, SQL injections and other vulnerabilities that could harm your users (and reputation) – should also be performed prior to launch.Bottom line: Does your web application work when, where and how your users need it to work?Usability TestingBeta testing may be suitable for some large companies (mainly those with name recognition) who wantto solicit user feedback, but it is NOT an effective substitute for usability testing, which should includeprofessional analysis of:  Graphics: Have you identified and addressed the major GUI bugs in your web app? Are the color schemes easy to understand (in every country of your user base)? Many of these flaws will not be reported by beta users, but they should be known about and fixed before launch.  Feature Set: This can include the intuitiveness of shopping lists, online checkouts and other e- commerce actions; the accuracy of search results, data entry, sign-up forms and other features.  Accessibility: Is your application accessible for all users, including the visually impaired? Does your app involve parental or administrator restrictions? Be aware of your legal obligations.  Benchmarking: How does your web application stand up to that of your closest competitors? What about your own previous versions? What areas does it compare favorably or unfavorably?  Overall Usability: Beta testers - if they report bugs at all - are likely to respond with “this-site- sucks” or “this-site-is-great” type of analysis. When launching a web application, professional, non-biased analysis is essential.Bottom line: Is your web application intuitive and highly usable for your target audience?WHITEPAPER: 10 Tips of Web Testing 3
  • 5. Load TestingHow will your web application hold up when it needs to most? Synthetic load tools are helpful insimulating these types of scenarios, but with the assistance of real testers, you can obtain a completepicture of how your application performs under stress.Load testing should, at the very least, involve one of the following methods:  Live Load Testing: A team of live testers (preferably from where your users reside – whether it be North America or a global audience) will test your application simultaneously, enabling you to see how your app performs under real-world, non- “The real value of tests is not that simulated load. This process, as its name would indicate, requires NO automated tools. Ideal for web they detect bugs in the code, but apps that contain Flash, streaming video or other types that they detect inadequacies in of multimedia that are difficult to evaluate with the methods, concentration and simulated load. skill of those who design and produce the code.”  Simulated Load Testing: Using the simulated load testing tool of your choice (more on this below) you can - Tony Hoare obtain a realistic snapshot of your web applications Software Testing Expert performance under peak synthetic usage.  Hybrid Load Testing: A combination of the aforementioned methods, this process involves having live testers perform functional testing on your web application while under automated load. This lets you see bugs or performance issues that only show themselves under peak loads.Bottom line: Will your web application perform as expected under maximum pressure? Will it crash?Will the performance degrade?Web Testing Checklist: A SummaryRegardless of whether you are performing functional, usability or load testing, it is imperative that eachof these practices achieve maximum testing coverage that matches your user base across:  Location: If you have a global user base, why would you only test your web application in your office? It’s therefore critical to extend your testing coverage beyond your borders if needed.  Language: If your application is available in multiple languages, you must verify that nothing is lost in translation (like the intuitiveness of your content, error messages and core features).  Operating Systems: Your app needs to work seamlessly across all the different flavors and versions of Windows, Mac and Linux.  Browsers: The days of Internet Explorer hegemony are over. Today’s web applications must be tested thoroughly against multiple versions of IE, Firefox, Chrome, Safari and others.And now, the Ten Tips of Web Testing….WHITEPAPER: 10 Tips of Web Testing 4
  • 6. The 10 Tips of Web Testing1. Don’t Be a Slave to Your Metrics When testing a web application, it’s easy to become overwhelmed by the data that that’s available to you. It can drag you in all sorts of directions and distract you from more important matters – if you allow it to. Don’t. Said testing expert Michael Bolton of DevelopSense: “Some people enslave numbers. They make numbers work “When your car is about to go off a too hard, and too often. I’ve seen organizations collect piles cliff, it’s a weird time to be thinking of data about defect escape ratios and defect detection about gas mileage and drag percentages. They hire market research firms and calculate coefficients; better to take the right the ratio of happy customers to unhappy customers. But control action—look out the the aggregated data doesn’t tell you anything specific on window and steer or use the brake how to make things better for the unhappy customers.” until you’re back on course.” In other words, while comprehensive testing will present - Michael Bolton you with a ton of raw data, the most important information CEO, DevelopSense will still come from your testers, managers and customers.2. Know Thy Third-Party Apps When launching a web application, remember that it won’t operate in a vacuum. There are many third-party applications that could interfere with its performance, apps like:  Live chat  Checkout processes  Search plug-ins  RSS Feeds  Embedded videos or audio players  Ad servers or embedded ads  Web analytics packages  Blogs, forums and message boards  Social networking modules or toolbars (like Tweet streams or Digg) Do any of these apply to your web application? To your users, these are part of your app.3. Understand the Testing Manager’s Dilemma As any honest test manager will tell you, it is NOT his or her responsibility to make the final decision about when the product is completed – that is clearly the role of the product owner. Testing managers have enough to worry about. Between time limits and budget constraints, there’s no shortage of obstacles to must deal with – so don’t make them the final arbiter of product readiness.WHITEPAPER: 10 Tips of Web Testing 5
  • 7. Besides, to a good testing manager, their job is never complete. In theory, software testing could go on forever! Despite knowing better, software executives regularly pressure their testing managers into providing an answer, and end up being told what exactly they want to hear. Also, it is okay to fall off the agile wagon every once and awhile. With the short sprint cycles and constant deadlines, it’s amazing that more companies don’t wind up there more often. But just because you can’t maintain a truly agile schedule 24/7, doesn’t mean that certain aspects of your development (like testing consistently) should ever be abandoned.4. Invest In Testers, Then Tools While bug-tracking systems and automated test tools are an essential part of any test team, there’s a price to be paid for relying on them too heavily – literally and figuratively. Consider automated tools, for instance. Automation will never find bugs for use cases that haven’t been conceived and documented. It simply helps to ensure that old bugs don’t resurface and that the main test cases are clear. “I urge you not to use expensive tools, even if they work. Never let your Conversely, live testers will find bugs that are new or manager buy them. Because unique to your app - which is probably most of them. In expensive tools become something short, ignoring real-world testing at the expense of tools you MUST use, even if they don’t is a recipe for disaster. work. A free tool may be freely Speaking of expense, if you dole out big money for abandoned. This gives you flexibility.” elaborate test tools (and expect an even bigger ROI), - James Bach you will feel compelled to keep using it, regardless of its effectiveness. In this regard, software companies would Testing Author and Expert be wise to follow the advice James Bach gave to individual testers (see sidebar).5. Protect Your Users, Save Yourself To have a safe, secure web application is obviously a no-brainer – nobody launches with anything less in mind. What’s not so obvious is the way to achieve this type safety for your web app. To point you in the right direction, here are a few questions to ask yourself before your next launch:  Is your web app behaving the way your privacy policy claims it does?  Can a tester easily get an app to cough up the private data of another user? For example, if they see “user_id=232” in the URL, what happens if they change it to “user_id=231”? Do they get to see someone else’s personal data?  Are you sharing personally identifiable information about your users with third-parties like Salesforce.com or Google Analytics? What about the company that’s hosting your app?  What happens if your web app is cached when it shouldn’t be? Does it share the wrong data with people? What happens if you actually want caching? Is it sending out the right things to be cached? Are end users actually seeing a benefit?WHITEPAPER: 10 Tips of Web Testing 6
  • 8.  Is your site vulnerable to common security exploits like XSS, injection flaws, broken authentication, flawed session management, unvalidated redirects and forwards? Since these areas are sometimes overlooked by in-house teams, it can be helpful to leverage testers who are skilled in testing, but new to your application.6. Multimedia, Multi-Problems How much does your web application rely on Flash, HTML 5, Java and other rich-media tools? Even if you answered “a little bit”, multimedia testing - including load testing - should become a critical component of your testing from this day forward. Of course, many in-house teams will find they have neither the time nor the headcount to achieve testing across all versions of multimedia players. If this is the case, you should strongly consider leveraging a community of professional testers to quickly and easily find the versions you need.7. Forfeit the Blame Game Reproducing defects, filing bug reports and “verifying” fixes are tasks commonly believed to be bottlenecks in the testing process. They are not. While they can comprise close to 20% of a test team’s “Testers need to really keep reminding duties - and while testing teams are often blamed for the developers that it’s not personal. delays in these processes - it is often actually a On the other hand, it’s important to bottleneck on the development side (where the state that not every functionality issue engineers are competing with their own challenges, is a bug. It’s important to have a tester obstacles and deadlines). that can tell the difference between an If test teams can work with development (rather than enhancement and a true bug. I know, it blame them) to improve the quality of the software sounds so ridiculously trivial but finding prior to code completion, then it will improve the quality testers who can also understand speed of the whole system. Buggy software can seem the nuances of business is key.” like the test team’s fault, but it is important realize that other factors are contributing the pace of each - Jack Margo development cycle - so don’t take it out on your SVP of Web Operations, DevShed testing team. Making developers better, helping them understand failures and the factors that cause them will mean fewer bugs to find in the future. Testers are quality gurus and that means teaching those responsible for anti-quality what they are doing wrong and where they could improve.8. Be Everywhere Does your web application work in New Zealand? What about Singapore? If your web app is localized – that is, if your users are dispersed throughout the globe – then you’ll need to testWHITEPAPER: 10 Tips of Web Testing 7
  • 9. language translation, currency conversion and other location-specific features. Aside from functionality, it is equally important to run usability testing with users from around the world. Neither in-house QA teams nor outsourcing firms can efficiently complete such a task. It would be (and indeed, has been) time-consuming, costly and impractical. And thus, it has been ignored. However, crowdsourcing enables you to easily leverage a global community of professional testers, letting you handpick the testers you need to fill out your testing coverage matrix.9. Plugins: Proceed With Caution Plugins remain a significant problem for web application developers. In fact, most of the security issues on the web today are a direct result of Flash and Acrobat bugs. In many cases, people end up with ancient versions of Acrobat reader that can run random exploits simply because they forgot to make the needed updates. Does this sound familiar? Also, while the sheer number of browser plugins has declined in recent years, extensions have gained in popularity. While less common (and usually only found on more advanced users’ browsers), extensions can dramatically impact how an app will work if they are buggy. As such, they too will require testing prior to launch.10. Respect The Dead – Make Way For The New It’s no secret that IE6 is on its way out, but large organizations can’t seem to kill it off just yet. As frustrating as it may be, support for older browsers can be critical, especially for B2B applications. Conversely, the next wave of HTML5 is going to introduce a flurry of competing multimedia standards. “HTML5 vs. Flash” is going to be a big debate in the years ahead, and companies will have to make a very important choice: Either they deal with Flash’s security issues and lack of support on mobile, or they opt for HTML5’s confusing vendor specific standards (H.264 vs. Theora). Also, CSS3 is slowly making its presence felt. In fact, many web apps already use it for additional layout effects. If you go down this road, how will non-CSS3 browsers deal with this? Cloud computing is another major trend to keep in mind when testing your web app. How will your application behave when different parts of the cloud are misbehaving? What happens to your lead “The Internet? Is that thing still around?” generation efforts, for example, if your CRM system malfunctions? Or if Omniture goes down for an - Homer Simpson hour? What effect will this have on your users? Typical Web User Consider these cloud issues when testing. While some trends will die before their time, others will persist far beyond what’s anticipated. For web application companies, it is critical to be mindful of these trends and remain on the offensive. Then again, you don’t have much of a choice.WHITEPAPER: 10 Tips of Web Testing 8
  • 10. About uTestHeadquartered near Boston, uTest is the worlds largest marketplace for software testing services. Thecompany provides real-world QA services through its community of 40,000+ professional testers fromover 170 countries around the world. To date, more than 500 companies - from web start-ups toenterprise software firms - have signed up to get their apps tested by the uTest community.uTest enables companies to launch higher quality products; get their desktop, web and mobileapplications to market faster; and control the cost of testing. Customers specify their requirements fortester experience, location, language, OS and browser, and uTest selects the right testers for eachproject. And because uTest is on-demand, companies only pay for completed test cycles.A brief online demo is available at www.utest.com/demo. uTest can be contacted at:uTest, Inc.153 Cordaville RoadSouthborough, MA 01772p: 1.800.445.3914e: info@utest.comw: www.utest.comWHITEPAPER: 10 Tips of Web Testing 9