Tonight’s discussion pertains the fastest growing segment of software development – Internet or Web applications.
Overview: Industry Facts and Misconceptions Type of risk mitigation performed key similarities to legacy and c/s development Planning Iterative Industry statistics Discussion of key differences Aspects unique to Web testing Address Web Testing Processes employed by Testmasters, Inc. Establishing QA Internet Test Plan Creating test Template Environment/Test Lab Discussion of tools to aid in maintaining quality systems How we can learn from history.
The Internet is a global network of networks connecting millions of users worldwide via many computer networks using a simple standard common addressing system and communications protocol. The connection between the different networks are called &quot;gateways&quot;. These gateways serve to transfer electronic data worldwide. The World Wide Web a distributed hypertext-based information system developed at CERN , is a globally interconnected network of hypermedia information based on the Internet A uniform addressing scheme for locating resources on the Web (e.g., URLs ). Protocols , for access to named resources over the Web (e.g., HTTP ). Hypertext, for easy navigation among resources (e.g. HyperText Markup Language or HTML , a standard format for describing the structure of documents for transmission of hypermedia documents). HTML documents are ASCII files with embedded codes for logical markup, format (text styles, document titles, paragraphs, tables) and hyperlinks. a set of servers that respond to requests from browsers (or clients ) for those documents. WWW also interfaces with other standard protocols (FTP, Telnet, NNTP, WAIS, gopher, ...) and their data formats. Underlying the user interface represented by web browsers, is the network and the protocols that travel the wires to the servers or &quot;engines&quot; that process requests, and return the various media. The protocol of the web is known as HTTP, for HyperText Transfer Protocol. The opportunities are new and exciting. The challenges facing Quality Assurance managers is rather daunting considering the rush to “get products out the door”. The one common theme we would like you to leave here tonight Change to meet new challenges but not forget the basics because they still apply. Life cycle Requirements Standards/ Test early and test often/ catch errors as early as possible in the life cycle Respect the iterative nature of software testing Planning – critical to providing continuous quality software Change control – essential to maintaining quality – this is not an option. Discuss different types of business activities and impact on risks. Before I go on, I would like to do what most consultants do, is first ask you a few Questions. If you would answer by raising your hand if the statement fits you. I have significant experience in testing Web applications I have just recently gotten involved with Web application development or testing. The applications I work with are directly available to the public. The applications I work with are primarily for defined users – B to B or internal users.
Typical retail customer utilization Rapid growth in using the Web business to business and internal to organizations.
Technology, both hardware and software, are changing so rapidly that it is not only difficult to keep up with the new technology but satisfy the differences posed in the marketplace. Processing speeds, connectivity speeds, memory – is the solution for the old 486 customer the same as the new pentium machines? Do you want to down play your technical capability to the lowest denominator? At what level do you target? Important questions in keeping you on track. Management Perception of time : It’s called RAD and managers loved it. No longer these extensive delays due to mainframe Worked with clients to create accelerated testing processes for the mainframe. Client outsourced much of their support for RAD from a third party since they were a legacy mainframe based group. Guess what took longer – same basic changes in both but who was completed first – the legacy. Why, life cycle, planning and controlling activities. Business managers want it yesterday. Months are days and days are hours. Management perception of quality: get it in faster. Quality should be perfect but don’t let that get in the way of completing the project in accord with my aggressive schedule. Testing? You mean these expensive resources make that many mistakes? Minimal patience and equally difficult to get the attention warranted – until major disaster occurs.
One of the most critical aspects of establishing consistent and timely change to Web applications is the use of effective plans. The RAD nature of Web development often impedes the establishment of meaningful plans. We have come to believe that planning is a key factor that should not be compromised. Our experience has shown, that the absence of planning has dysfunctional consequences. Establishing a common set of templates and checklists to plan creation will expedite the process. All plans will not be the same. The plans will vary based on:
Walkthrough of the test plan template; hyperlink to QTM – go to tools and templates – click on test plan for internet.
Configuration Configuration refers to the way the test environment is set up and the assortment of components that make up the test environment. Configuration can refer to either hardware, software, or the combination of both. The ability to establish a configuration that models the targeted production environment is the single most important factor in the effectiveness of the resulting test environment. Scalability Scalability is defined as the ability of a test component’s parameters of operation to expand without necessitating major changes or fundamental redesign in the test system. For example, the test environment configuration should support testing of a single application or be expandable to support testing of many applications with a minimum of configuration changes. Recoverability In order for the test environment to be effective, it is imperative that it incorporate the ability to recover from any deviations from the baseline state back to a specific point-in-time. Procedures must be defined to address such issues as system, disk, and/or database crashes, database corruptions, and/or baseline version restores, as well as reestablishment of the system to a particular point in time for proper staging of tests. Controllability One of the most important characteristics of an effective test environment is the ability to remain stable and consistent throughout the use of the environment. Stability within the environment provides personnel with the capacity to execute tests in a more efficient and precise manner. In order to retain stability throughout the environment, it is essential that all items and aspects of the environment be strictly controlled. Controllability is having the capacity to provide stability and consistency of the entire test environment including network, hardware, and software configurations, as well as mechanisms provided to maintain facility, network, and software integrity. Control is necessary to provide the test and environment with minimal external interference to the test focus as possible. Manageability Manageability of the defined test environment is necessary to ensure the highest productivity within testing. The ability to control the environment directly relates to how you manage data changes, release deliveries, environmental changes, facilities, and personnel. These are all equally important to the success of testing. Management of the test environment must include consistent and repeatable practices. Consistent management practices within the test environment refers to the ability to manage and adhere to defined policies and procedures for all applications staged within the environment throughout all phases of the development lifecycle and the use of the environment. PC Cards 3COM Etherlink III Adaptec SCSI Monitors/Video Cards Samsung SyncMaster STB Velocity Printers HP OfficeJet HP DeskJet Modems/Network Cards IBM 56K modem 3COM ISDN LAN modem Software Another primary component of the test environment is the software of the application under test. It is important to define and document the specific attributes of the application, such as the programming language used, version under test, etc. Operating Systems The test environment may need to support a wide representation of significant operating systems, which will be used primarily for compatibility testing. The following list provides an example of different types of operating systems. Sun Solaris Microsoft Windows 95 Microsoft Windows 98 Linux Network Support In order to avoid interfering with the corporate network, it may be necessary to set up a dedicated test environment network. Determine the type of network to be supported. Some different types of networks are: Token ring LAN WAN Star Point-to-Point Databases The database needs of the application must be determined. Some examples of the databases are: Sybase Informix IDMS Oracle Change Control/Migration Process Any change to the test environment changes the structure and configuration of the testing to be performed. It is imperative that the test environment be controlled in a precise and traceable manner. The manner in which this can be accomplished is to establish clear and concise change control and migration procedures for changes to and integration of these changes into the environment. Change management is one of the most fundamental systems-management functions that must be enforced to ensure robustness--serviceability, performance, availability, reliability, and supportability of the test environment. In order to ensure that the test environment remains able to support valid testing, it is important to know precisely what is contained in the test environment at all times. Documented instructions for performing tasks such as environment builds and instructions for recording environment changes should be followed. Change management best practices encompass the control of all changes made in the test environment, including operating systems, software upgrades and modifications, applications, application changes to online transactions and batch jobs, application transactions, batch jobs additions and deletions, changes to hardware configurations, and network modifications such as performance-related enhancements. Click here ( Change Control Process Flow ) to view an example of a standard change control process. Other equipment needed in the test environment Equipment such as cabling, data storage, UPS units, and computer media should be included on the inventory list. Support Software There are many different types of support software. Test tools are the most commonly used support software. Test tools encompass a broad range of automated and manual aids. Tools span automated capture/playback tools to performance measurement. Listed below are some of the different categories of test tools. Capture/Playback tools Capture/Playback tools are designed to capture user operations including keystrokes, mouse activity, and display output. Once this information has been captured, the tools can then automatically play back the captured results. Development Testing Tools Development testing tools can span a number of different test tool categories. These tools can assist with the development of tests and are most useful as a means of implementing test cases. Some development tools include: test data generation tools, coverage analysis tools and requirements-based test design tools. Metrics (Measurement) Tools Test Measurement or Test Coverage tools provide an automated means of determining what parts of the program have been tested or not tested, providing indication where additional tests need to be performed. Compare Utilities Compare Utilities provide the user with the ability to compare data from different sources. These sources can be data files, text files or databases. Environment Management Utilities These utilities allow the user to monitor network activity within an environment. Performance Assessment Tools Stress test tools or load and performance test tools facilitate testing the systems response time. Recovery Tools Recovery tools provide a means to recreate data from an initial captured state. This category includes tools such as backup/restore tools and disk-cloning tools. Test Management Tools Test management primarily relates to the implementation testing phase: system testing, system integration testing and acceptance testing. Test management tools provide some or all of the following features: A repository to warehouse test definitions, scripts and expected results A means of viewing and reporting tests useful in test walkthroughs A method of orchestration and sequencing of tests to be performed A process for tracking and reporting test results Management reporting of test development and execution status Some test management tools are linked directly to on-line drivers, while others are stand-alone products that solely focus on test data management. Recovery tools to aid in restoring files and environments. Setting up clean machines for the lab, etc. Archival processes and programs to aid in rapid storage and retrieval of information. Trouble shooting processes and tools – sniffers,etc.
Test execution should essentially start at the beginning stages of your development. Through use of checklists and templates, much of the requirements can be confirmed early and avoids redesign later. Is this a new concept? No of course not. I stated earlier that the common theme here is that those disciplines that helped us succeed in the past will help us moving forward. In discussing the practices used in test execution, we must be aware of the impact the type of application has on the work to be performed. All testing execution processes benefit significantly from regular use of standard checklists and forms. The likelihood is your test will be repeated. Build equity in what you do. Creating effective manual processes can be transformed with the use of automated tools. Business applications – databases – complex functionality. GUI Testing You have seen the checklists for GUI Testing and Usability Testing. Adhering to these structured disciplines, enables subsequent integration of automation that can reduce the monotony of retesting your applications. Because if you make a change, you need to test. If you make a change, you need to test. Usability Testing will ensure that your customers have a pleasant experience while at your site. Visual Presentation Color Appropriate use of color to attract attention; reduce eyestrain; appeal to the customer. Color use is consistent when designating specific functionality, e.g., hyperlinks. Layout Fonts Content The content is user-friendly and appropriate to the expected audience. Content is laid out in a way that is consistent with the delivery medium. Spelling and grammar are correct. User Interface Operation Keyboard, mouse, and browser navigation buttons perform as expected. Routing is logical and intuitive. Labels Menu items, buttons, tool bars, and the like contain meaningful/descriptive labels; placement is consistent from screen to screen. Forms Field sizes are consistent with the data to be entered. Required fields are identified. ‘ Clickable’ Objects Clickable text and objects intuitively represent the function called. Graphics Placement and use of graphics is consistent with the text. Display Stability Window Sizing Screen Resolution All text, menus, buttons, forms, message boxes, etc. display properly at common screen resolutions. Scrolling A scroll bar is present when text/graphics exceed the length and or width of the display window. Scrolling does not cause distortion. Navigation Browser All browser functions perform as intended using both the keyboard and mouse. Functions can be performed in any sequence without error. Steps are retraceable. Application performs equally on multiple browsers and/or browser versions. Site All site navigation features work correctly and in any sequence. There are no conflicts between browser and site navigation. Site navigation is logical and intuitive. Navigation buttons, bars, tabs, or menus are consistently placed throughout the application; e.g., left-oriented, bottom-oriented, etc. Site Map A map of the site should be provided for complex sites to aid the users in finding their way around the site. When present, all links from the map to the site are verified to work properly. Links Internal All links are easily identifiable and call the proper document or function. External Users can easily return to your site after reviewing an external link. Menus and Toolbars Lists Menu lists are complete, drop down when selected, and are descriptive. Menu items are listed in order of usage or importance. All menu items work as expected, calling the appropriate document or function. When an item is selected the list becomes inactive. Buttons Labels are descriptive of the function of the button; all buttons perform as expected and are disabled after selection. Graphics Graphics intuitively represent the function they represent, perform as expected, and are disabled after selection. Form Functions Data Entry All check boxes, radio buttons, and text fields perform as expected. Required data fields are enforced. Data validations are performed accurately, returning an understandable message to the user. Data within the specified range of values, data type, etc. are accepted and processed accurately. Data Retrieval Data entered into and accepted by the application are accurately retrieved via the provided means, such as e-mail, inquiry, report, etc. Data are as entered; field lengths are adequate to display all entered and accepted characters. Multi-Media Synchronization Video All video output plays accurately from beginning to end. Video can be stopped and restarted without loss. Video is paused when other functions are selected and restarts when/if appropriate. Audio All audio output plays accurately from beginning to end. Audio can be stopped and restarted without loss. Audio is paused when other functions are selected and restarts when/if appropriate. Audio and video are synchronized, when appropriate, and remain in synch when stopped and restarted. Slides All slide shows play accurately from beginning to end. Shows can be stopped and restarted without loss. Slide shows are paused when other functions are selected and restart when/if appropriate. Slides are synchronized with audio and video, as appropriate. Animations Animations work as intended System Functions Data Entry All data required to accomplish system functions is entered into the system via manual or automated processes. Data is properly edited to avoid erroneous processing. Edits perform accurately on all permutations of valid data; edits accurately process data outside, on, and inside boundary values. Field sizes are adequate to accommodate data. Processing All calculations/algorithms are performed accurately. Data is properly routed and displayed within the application. Processes meet expected results. Data Storage All inbound and calculated data is properly stored either for temporary or permanent access as required by the application. Field sizes and definitions are accurate and accommodate all valid occurrences. Data Retrieval Data is accurately retrieved and displayed. Retrieval is efficient. User is properly informed of excessive wait times (via hour-glass or other notification). Security Application is appropriately and adequately secured based on its type, exposure to the company, and exposure to customers. Security includes access to the system, authorized use of specific functions within the system, as well as secure transmission of sensitive data. Controls Application provides adequate controls to ensure accurate processing, balancing, and verification of results. All records are accurately processed; records are neither dropped nor duplicated during processing. Back Office Application outputs appropriately and accurately support back office functions (e.g., order processing, credit card processing, inventory management, billing and receipts). Interfaces to 3 rd Party Vendors Inputs Data received from other applications is accurately processed. Rejected data is handled correctly and with sufficient controls to support reprocessing and avoid loss of data. Outputs Data sent to other applications is processed and formatted properly. Data produces accurate results when processed by the other applications. The systems afford appropriate controls to avoid loss of data.
There are a slew of tools out there in the market place. Some are extremely effective. The processes you put into place make the difference as to the success of the tools.
The Web offers business an exciting new delivery system that enables companies to reach out to there customers and for their customers to reach out to them. This can be a very good thing – but it can be a very bad thing if you frustrate the customer by: not allowing access (closing the door on a 24 hour store), being too slow, confusing the customer, etc. etc. etc. When you are on the Web, you are in a glass box. If your zippers open, someone is going to notice. So, we have some new challenges to mingle with these new found opportunities. With the introduction of new technology there seems to be
Testing Web Based Applications Differences in Traditional Legacy Testing vs. Web Systems Testing
AGENDA <ul><li>Overview </li></ul><ul><li>Discussion of Key Differences </li></ul><ul><li>Web Testing Processes </li></ul><ul><li>Tools of the Trade </li></ul><ul><li>Learning from History </li></ul><ul><li>Q & A Session </li></ul>
Overview <ul><li>Web Testing is Different but…. </li></ul><ul><ul><li>Life Cycle Methodology </li></ul></ul><ul><ul><li>Planning * Planning * Planning </li></ul></ul><ul><ul><li>Change Control </li></ul></ul><ul><li>Types of Business Activity </li></ul><ul><ul><li>Corporate Home Page </li></ul></ul><ul><ul><li>eCommerce Activity </li></ul></ul><ul><ul><ul><li>On-line Ordering </li></ul></ul></ul><ul><ul><ul><li>On-line Payment </li></ul></ul></ul><ul><ul><li>Business to Business </li></ul></ul><ul><ul><li>Structured Inquiry vs. Query Reports </li></ul></ul><ul><ul><li>Front-end to Legacy Systems </li></ul></ul><ul><li>Audience Mix </li></ul><ul><li>Terms </li></ul>
WWW Survey Results Demographics <ul><li>Survey Respondents </li></ul><ul><li>38.5 % Female (41 % 2000) </li></ul><ul><li>(Compared to 5% in 1994) </li></ul><ul><li>82 .5 % Vote </li></ul><ul><li>Most Important Issue Facing the Internet? </li></ul><ul><ul><ul><li>Privacy (30%) </li></ul></ul></ul><ul><ul><ul><li>Censorship (24%) </li></ul></ul></ul><ul><ul><ul><li>Navigation (16 %) </li></ul></ul></ul>"Copyright 1994-1998 Georgia Tech Research Corporation. All rights Reserved. Source: GVU's WWW User Survey www.gvu.gatech.edu/user_surveys"
WWW Survey Results Technical Demographics <ul><li>55% < 33.6K </li></ul><ul><li>39 % Speed upgrades within last year </li></ul><ul><li>71 % have not switched browsers in last 12 months </li></ul><ul><li>46% have created a Web Page </li></ul>"Copyright 1994-1998 Georgia Tech Research Corporation. All rights Reserved. Source: GVU's WWW User Survey www.gvu.gatech.edu/user_surveys"
Tremendous Web Growth <ul><li>64 Million U.S. Adults regular Internet users (NUA) </li></ul><ul><li>56 % of US companies will sell their products on-line in 2000 – up from 24% in 1998. (NUA) </li></ul><ul><li>800 Million Web pages (NEC Research). </li></ul><ul><li>In 1999, 80 million Americans had Internet Connectivity compared to 3 million in 1993. (U.S. Department of Commerce) </li></ul>
Public Web Sites <ul><li>U.S. Owns largest share of sites </li></ul><ul><li>Percentage of U.S. Internet Connected Households soars to 60% in 2000. </li></ul><ul><li>(On-line Computer Library Center) </li></ul>
What they use the Web for.. <ul><li>Information Gathering </li></ul><ul><li>Community Services </li></ul><ul><li>Some Shopping (but not for everything) </li></ul><ul><li>Game Playing </li></ul><ul><li>Novelty Effect Wearing Off </li></ul>
Toward Quality Systems <ul><li>…the market for technology and services is not a market for novelty any more, but a market for quality , which increasingly has to serve an experienced user base. This will open a host of new possibilities for companies willing to question and improve existing tools and services. </li></ul><ul><li>2000 Pfeiffer Consulting </li></ul>
Discussion of Key Differences <ul><li>Direct Access into Company </li></ul><ul><ul><li>No Customer Service Middle-man </li></ul></ul><ul><ul><li>Security Risks </li></ul></ul><ul><ul><li>Critical Access and Timing </li></ul></ul><ul><ul><li>Can’t Shut It Down </li></ul></ul><ul><li>Plethora of Environments </li></ul><ul><ul><li>Computers/Operating Systems/Browsers </li></ul></ul><ul><ul><li>Settings/Custom Desktop </li></ul></ul><ul><ul><li>Software Languages </li></ul></ul><ul><ul><li>Embedded Components </li></ul></ul>
Differences (continued) <ul><li>Rapidly Changing Technology </li></ul><ul><li>Management Perception of Time </li></ul><ul><li>Management Perception of Quality </li></ul><ul><li>The Need for Speed </li></ul><ul><ul><li>Time to Market </li></ul></ul><ul><ul><li>Operational Performance </li></ul></ul><ul><li>Training and Education of the User </li></ul>
Test Planning Key to Achieving Quality Results <ul><li>Vary Approach Based on Critical Factors </li></ul><ul><ul><ul><li>Type of Application </li></ul></ul></ul><ul><ul><ul><li>Risks </li></ul></ul></ul><ul><ul><ul><li>Timing </li></ul></ul></ul><ul><ul><ul><li>Size of Effort </li></ul></ul></ul><ul><li>Test Plan Checklist Checklist </li></ul><ul><li>Review of Test Plan Template Internet Test Plan Template </li></ul>Web Testing Processes
Test Plan Template <ul><li>Introduction </li></ul><ul><li>Objectives and Tasks </li></ul><ul><ul><li>2.1 Objectives </li></ul></ul><ul><ul><li>2.2 Tasks </li></ul></ul><ul><li>Risks </li></ul><ul><ul><li>3.1 Identify and prioritize risk areas </li></ul></ul><ul><ul><li>3.2 Identify test dependencies on external milestones </li></ul></ul><ul><ul><li>3.3 Contingency plan for risks </li></ul></ul><ul><li>Test Strategy </li></ul><ul><ul><li>4.1 Functional Testing </li></ul></ul><ul><ul><li>4.2 GUI Testing </li></ul></ul><ul><ul><li>4.3 Usability Testing </li></ul></ul><ul><li>Hardware & Software Requirements </li></ul><ul><li>Environment Requirements </li></ul><ul><ul><li>6.1 Client Operating Systems </li></ul></ul><ul><ul><li>6.2 Configuration of hardware/ software environment </li></ul></ul><ul><ul><li>6.3 Clean and Dirty Machines </li></ul></ul><ul><li>Test and Project Schedules </li></ul><ul><li>Resources </li></ul><ul><li>Test Procedures </li></ul><ul><li>Control Procedures </li></ul><ul><li>Retention Policy </li></ul><ul><li>Approval Process </li></ul><ul><li>Template and Samples </li></ul>Web Testing Processes
Creating Test Scenarios <ul><li>Checklists for Common Testing Activities </li></ul><ul><li>GUI Checklist </li></ul><ul><li>Use of Matrices / Application Test Templates Generic Matrix Results Tracking </li></ul><ul><li>Create Positive and Negative Tests </li></ul><ul><li>Required Fields, Boundary Values, and Data Types </li></ul>Web Testing Processes