deeper IBM Global Business Services – Application ServicesPresentation Transcript
April 19, 2010 Version 1.0 Application Testing Services University of Stellenbosch – System Testing Disclaimer This document does not constitute a formal offer, but serves to outline the discussion elements for contractual consideration based on IBM’s standard terms and conditions.
The current situation
Ad Hoc Statements John Roodt : Senior Architect “… you guys are brutal. Brutally honest…” Jolene Saayman : Lead Java Developer “… if it weren't for you I would have left this company a long time ago…” Adam Granger : Lead Java Developer “… you are our safety net…” Michael Kiergard : Customer Project Manager “… you must be involved. I trust you guys…. I don’t trust the developers…” Karen Greeves : Project Delivery Executive “… I can rest assured that after it has been through your hands, then the application is pretty solid…” Meike Voight : SAS Developer “… but what are they going to test if I have already tested it?...” Cherryl Lundt : Project Manager “… I now want to involve you on all my projects...” “ As a software tester, the fate of the world rests on your shoulders. This statement is not an exaggeration if you accept the dual premises that computer software runs the modern world and that all software has bugs – then you will quickly reach the inescapable conclusion that unless the most disruptive bugs are removed, the world as we know it will grind to a halt.” Scott Loveland, Geoffrey Miller, Richard Prewitt, Jr., and Michael Shannon have collectively spent more than 70 years in the software testing trenches. They currently hunt bugs in the IBM mainframe development laboratory in Poughkeepsie, New York. Collectively they have spent more than 70 years in the software testing trenches.
Being a tester
Test Strategy : Example case study of mature client
What people think we are : Testers are negative thinkers, they complain, they like to break things, they take a special thrill in delivering bad news.
BUT WE PROTEST !!
Who we really are : Tester’s don’t complain, we offer evidence. Tester’s don’t like to break things, we dispel the illusion that things work. Testers don’t enjoy giving bad news, we enjoy freeing our clients from the thrall of false belief.
To define the role of a testing group is not easy. The following metaphor might best explain this role:
"A project is like a road trip. Some projects are simple and routine, like driving to the store in broad daylight. But most projects are more like driving a truck off-road, in the mountains at night. Those projects need headlights. The tester is the headlight . The tester illuminates the road ahead so the developers and managers, however they struggle with the map, can at least see where they are, what they're about to run over, and how close they are to the cliff."
The detailed mission of testing groups might differ from company to company, but behind those details is a common factor. Testing is done to find information. Critical decisions about the project or the product are made on the basis of the information found by the test group.
IBM Testing Services - Overview Provide integrated testing services that leverage test best practices and highly skilled test professionals from across IBM
Deliver flexible testing services
Exploit technology to deliver innovative solutions Leverage leading practices and Global Reach
Lead industry through proven processes, methods, and tools
Maintain high customer satisfaction through delivery excellence and measurable business value
Provide testing infrastructure On-Demand
Test Team IBM Provision of experienced test resources to IBM &/or Customer Projects Testing Project and/or Programme Take responsibility for all aspects of testing for a project or programme. Build estimates, provide the team, provide management support Consultancy Services Managed Testing Services “ Testing on Demand” for a subset of testing scope or full Outsourced Testing Test Process Automation Test Environment Management Test Support (Systems, Integration, OAT, UAT) Technical Testing Performance, Load and Stress Education & Skills Transfer Component Solutions Web Testing Test Leadership Legacy Transformation Test Automation Test Process Assessment Capacity Planning Complex Architecture Performance Scalability & Interoperability Software Product Testing Full Lifecycle Test Services tailored
Test Strategy – Example Case study Business Case Feasibility Analysis Specification Design Architectural Design Module Design Construction Module Test Acceptance Test System Test Implementation Integration Test Business Realization Business Process Owners Business IT Vendor or Int IT Business Governance
Test Strategy – Example Case Study Business Case Feasibility Analysis Specification Design Architectural Design Module Design Construction Module Test Acceptance Test System Test Implementation Integration Test Business Realization Business Process Owners Business IT Vendor or Int IT
Change / Decision register
Non Functional Requirement Specification
Software System Requirement Specification
Technical Architecture Definition
AsIs and ToBe model
Logical Data Model
System Context Diagram
Project Management Plan
System Design Document
Physical Data Model
Detailed Test Plan
Implementation Requirement Specification
Operational Acceptance Test plan
Unit Test report
Integration Test report
Data Migration Test report
Factory Acceptance report
Application support documentation
Global HelpDesk documentation
Operational Management support
User Acceptance Test report
Stress and Performance report
Application Impact Assessment
Operations Acceptance test report
Maintenance and Support Contracts
Handover Report to Business Owner
Close Down report
Test Strategy – Example Case Study Test plans AUT, AIT, FAT Test cases AUT, AIT, FAT Execute AUT Execute AIT Execute FAT Test plans SIT, UAT, SPT Test cases SIT, UAT, SPT Execute SIT Execute SPT Execute UAT Promote PROD Warranty Period Design Analysis Construct and Vendor test Cust Test Impl Prod Execute OAT Execute AIA
Testing Methodology - Simplified “V”-Model of Testing Capture Requirements Design System Implement System Component Test Integration/System Test Acceptance Tests
Test team involved on day one of project
Requirements, high-level design are primary drivers for behavioral test phases
Low-level design, implementation are primary drivers for structural testing
Features dropped during development (left side of V) rather than slipping test phase entry dates or violating test entry criteria
Application Concept Develop Tests Develop Tests Develop Tests System Tests Integration Tests Unit Tests Verification Validation
Testing Methodology - Validation ARE WE BUILDING THE RIGHT SYSTEM? Ideal Reviewers : Review Granularity Business Owners Business Representative Business Analyst Conceptual Application Business Owner Architects Test Engineers Prototypes Simulations High Level Designs Developers Deployers Architects Detailed Design Execution Activities For review phases Requirements/Specification reviews Architectural Design reviews Detailed Design reviews Test Planning
Testing Methodology - Verification ARE WE BUILDING THE SYSTEM RIGHT? Ideal Reviewers : Review Granularity Developers DB / Net / System Administrators Deployers Structural (“White Box”) Test Engineers Test Technicians (Some) Developers Behavioral (“Black Box”) Tech Support / Help Desk Sales / Marketing Business Analyst / User Customer Live (“Alpha/Beta”) Execution Activities For review phases Unit Component Integration System Acceptance Pilot
Test Process – Main process
Test Process – Define Mission
Test Process - Verify Test Approach
Test Process – Validate Build Stability
Test Process – Test and Evaluate
Test Process – Achieve Acceptable Mission
Test Process – Improve Test Assests
Simplified Test Process
What every developer needs to know : 1. The product is more than Software
Kit OS Infrastructure
What every developer needs to know : 2. Quality is more than the lack of bugs (FURPS)
Does it solve the problem when it works Is it practical to use Does it keep on working Is it economical to build and maintain
What every developer needs to know : 3. Quality Assurance is more than testing Quality Assurance is everything you do to minimize the risk of failure and to promote excellence
Process definition and improvements
Inspections and active testing
Experience based improvements
What every developer needs to know : 4. Testing is hard to do
Anticipate your users:
… to examine a product that is..
...using a process that is often…
Think of the permutations!!
Functions vs Input data vs state of data
Products vs platforms
Expectations of customers
Versions of the product
What every developer needs to know : 5. You can make testing easier
Document your design
User internal error checking
Test units before Integration
Tell the tester what is new, concerns you or is shaky
Test the build yourself…….First
Evolve product in functional layers.. Ie Iterative development with testability in mind
Build in testability
Characteristics of a good tester
Know at least some programming . There’s a popular myth that testing can be staffed with people who have little or no programming knowledge. Since they’re testing software, without know some programming, they can’t have any real insights into the kinds of bugs that come into software and the likeliest place to find them.
Know the application . The ideal tester has deep insights into how the users will exploit the program’s features and the kinds of cockpit errors that users are likely to make.
Intelligence . The single most important quality for testers (just as for programmers) is raw intelligence, good testers, just as programmers, are smart people.
Hyper-sensitivity to little things . Good testers notice little things that others miss or ignore. Testers see symptoms, not bugs.
Tolerance for chaos . People react to chaos and uncertainty in different ways. If the tester waits for all issues to be fully resolved before starting test design or testing, he won't get started until after the software has been shipped. Testers have to be flexible and be able to drop things when blocked and move on to another thing that is not blocked. Testers always have many irons in the fire.
People skills . You can be an effective programmer even if you are hostile and anti-social; that won’t work for a tester. Testers can take a lot of abuse from outraged programmers. A sense of humor and a thick skin will help the tester survive. Testers may have to be diplomatic when confronting a programmer with a fundamental goof. Diplomacy, tact, a ready smile- all work to the independent tester’s advantage.
Tenacity (Merriam-Webster’s dictionary explains it as: persistent in maintaining or adhering to something valued or habitual). An ability to reach compromises and consensus can be at the expense of tenacity. The best testers are both socially adept and tenacious where it matters. The best testers are so skilful at it that the programmer never realizes they’ve been had.
Organized . There’s just too much to keep track of to trust to memory. Good testers use files, Databases and all the other accoutrements of an organized mind.
Sceptical . That doesn’t mean hostile. I mean skepticism in the sense that nothing is taken for granted and that all is fit to be questioned. Only tangible evidence in documents, specifications, code and test results matter. While they may patiently listen to the reassuring words from the programmers ("Trust me. I know where the bugs are.") - and do it with a smile - they ignore such in-substantive assurances.
Self-sufficient and tough . If they need love, they don’t expect to get it on the job. They can’t be looking for interaction between them and programmers as a source of ego-gratification and/ or nurturing. Their ego is gratified by finding bugs, with few misgivings about the pain (in the programmers) that such finding might engender.
Cunning . Systematic test techniques such as syntax testing and automatic test generators have reduced the need for much cunning, but the need is still with us and undoubtedly always will be because it will never be possible to systematize all aspects of testing. There will always be room for that off-beat thinking that will lead to a test case that exposes a really bad bug.
Technology hungry . They hate dull, repetitive work; they’ll do it for a while if they have to, but not for long. The silliest thing for a human to do, in their mind, is to pound on a keyboard when they’re surrounded by computers.
Honest . Testers are fundamentally honest and incorruptible. They’ll compromise if they have to, but they’ll righteously agonize over it. This fundamental honesty extends to a brutally realistic understanding of their own limitations as a human being.