Your SlideShare is downloading. ×
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
software-testing-framework 3
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

software-testing-framework 3

786

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
786
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Software Testing Framework Document version: 2.0Harinath V Pudipeddihari.nath@sqae.comhttp://www.sqae.com
  • 2. Table of ContentsTable of Contents ...........................................................................................2Revision History .............................................................................................4Testing Framework .........................................................................................51.0 INTRODUCTION ........................................................................................................................... 51.2 TRADITIONAL TESTING CYCLE ...........................................................................52.0 VERIFICATION AND VALIDATION TESTING STRATEGIES................................... 62.1 VERIFICATION STRATEGIES ..............................................................................6 2.1.1 REVIEW’ S ......................................................................................7 2.1.2 INSPECTIONS ..................................................................................8 2.1.3 WALKTHROUGHS ..............................................................................82.2 VALIDATION STRATEGIES ................................................................................83.0 TESTING TYPES ............................................................................................................................ 93.1 WHITE BOX TESTING .....................................................................................9W HITE BOX TESTING TYPES................................................................................. 10 3.1.1 BASIS PATH TESTING ...................................................................... 10 3.1.2 FLOW GRAPH N OTATION ................................................................... 10 3.1.3 CYCLOMATIC COMPLEXITY ................................................................. 10 3.1.4 GRAPH MATRICES .......................................................................... 10 3.1.5 CONTROL STRUCTURE TESTING ........................................................... 10 3.1.5.1 Condition Testing ........................................................... 10 3.1.5.2 Data Flow Testing .......................................................... 103.1.6 LOOP TESTING ........................................................................................ 11 3.1.6.1 Simple Loops .......................................................................... 11 3.1.6.2 Nested Loops .......................................................................... 11 3.1.6.3 Concatenated Loops ................................................................. 11 3.1.6.4 Unstructured Loops .................................................................. 113.2 BLACK BOX TESTING ................................................................................... 11BLACK BOX TESTING TYPES ................................................................................. 11 3.2.1 GRAPH BASED TESTING METHODS ....................................................... 11 3.2.2 EQUIVALENCE PARTITIONING .............................................................. 11 3.2.3 BOUNDARY VALUE ANALYSIS .............................................................. 12 3.2.4 COMPARISON TESTING ..................................................................... 12 3.2.5 ORTHOGONAL ARRAY TESTING ............................................................ 12 3.3 SCENARIO BASED TESTING (SBT).......................................................... 12 3.4 EXPLORATORY TESTING ....................................................................... 134.0 STRUCTURAL SYSTEM TESTING TECHNIQUES ........................................................ 135.0 FUNCTIONAL SYSTEM TESTING TECHNIQUES......................................................... 134.0 TESTING PHASES ...................................................................................................................... 144.2 UNIT TESTING ........................................................................................... 154.3 INTEGRATION TESTING ................................................................................. 15 4.3.1 TOP- DOWN I NTEGRATION.................................................................. 15Software Testing Framework V2.0 2 of 25
  • 3. 4.3.2 BOTTOM- UP I NTEGRATION ................................................................. 154.4 SMOKE TESTING......................................................................................... 164.5 SYSTEM TESTING ........................................................................................ 16 4.5.1. RECOVERY TESTING ....................................................................... 16 4.5.2. SECURITY TESTING ........................................................................ 16 4.5.3. STRESS TESTING .......................................................................... 16 4.5.4. PERFORMANCE TESTING .................................................................. 16 4.5.5. REGRESSION TESTING .................................................................... 174.6 ALPHA TESTING ......................................................................................... 174.7 USER ACCEPTANCE TESTING ........................................................................... 174.8 BETA TESTING ........................................................................................... 175.0 METRICS ......................................................................................................................................... 176.0 TEST MODELS .............................................................................................................................. 196.1 THE ‘V’ MODEL .......................................................................................... 196.2 THE ‘W’ MODEL ......................................................................................... 206.3 THE BUTTERFLY MODEL ................................................................................ 217.0 DEFECT TRACKING PROCESS.............................................................................................. 238.0 TEST PROCESS FOR A PROJECT ........................................................................................ 249.0 DELIVERABLES ........................................................................................................................... 25Software Testing Framework V2.0 3 of 25
  • 4. Revision HistoryVersion No. Date Author Notes1.0 August 6, 2003 Harinath Initial Document Creation and Posting on web site.2.0 December 15, Harinath Renamed the document to Software 2003 Testing Framework V2.0 Modified the structure of the document. Added Testing Models section Added SBT, ET testing types.Next Version of this framework would include Test Estimation Procedures and MoreMetrics.Software Testing Framework V2.0 4 of 25
  • 5. Testing FrameworkThrough experience t hey det erm ined, t hat t here should be 30 defect s per 1000 linesof code. I f t est ing does not uncover 30 defect s, a logical solut ion is t hat t he t estprocess was not effective.1.0 IntroductionTest ing plays an im port ant role in t oday’s Syst em Developm ent Life Cycle. DuringTesting, we follow a systematic procedure to uncover defects at various stages of thelife cycle.This fram ework is aim ed at providing t he reader various Test Types, Test Phases,Test Models and Test Met rics and guide as t o how t o perform effect ive Test ing in t heproject.All t he definit ions and st andards m ent ioned in t his fram ework are exist ing one’s. Ihave not alt ered any definit ions, but where ever possible I t ried t o explain t hem insim ple words. Also, t he fram ework, approach and suggest ions are m y experiences.My int ent ion of t his fram ework is t o help Test Engineers t o underst and t he concept sof t est ing, various t echniques and apply t hem effect ively in t heir daily work. Thisframework is not for publication or for monetary distribution.I f you have any queries, suggest ions for im provem ent s or any point s found m issing,kindly write back to me.1.2 Traditional Testing CycleLet us look at t he t radit ional Soft ware Developm ent life cycle. The figure belowdepicts the same. Requirements Requirements Design Design Te st Code Code Test Maintenance Maintenance Fig A Fig BI n t he above diagram ( Fig A) , t he Test ing phase com es aft er t he Coding is com plet eand before the product is launched and goes into maintenance.Software Testing Framework V2.0 5 of 25
  • 6. But , t he recom m ended t est process involves t est ing in every phase of t he life cycle( Fig B) . During t he requirem ent phase, t he em phasis is upon validat ion t o det erm inet hat t he defined requirem ent s m eet t he needs of t he proj ect . During t he design andprogram phases, t he em phasis is on verificat ion t o ensure t hat t he design andprogram s accom plish t he defined requirem ent s. During t he t est and inst allat ionphases, t he em phasis is on inspect ion t o det erm ine t hat t he im plem ent ed syst emmeets the system specification.The chart below describes the Life Cycle verification activities.Life Cycle Phase Verification ActivitiesRequirements • Determine verification approach. • Determine adequacy of requirements. • Generate functional test data. • Determine consistency of design with requirements.Design • Determine adequacy of design. • Generate structural and functional test data. • Determine consistency with designProgram (Build) • Determine adequacy of implementation • Generat e st ruct ural and funct ional t est dat a for programs.Test • Test application system.Installation • Place tested system into production.Maintenance • Modify and retest.Throughout the entire lifecycle, neither development nor verification is a straight- lineact ivit y. Modificat ions or correct ions t o a st ruct ure at one phase will requiremodifications or re- verification of structures produced during previous phases.2.0 Verification and Validation Testing Strategies2.1 Verification StrategiesThe Verificat ion St rat egies, persons / t eam s involved in t he t est ing, and t hedeliverable of that phase of testing is briefed below:Verification Performed By Explanation DeliverableStrategyRequirements Users, Developers, Requirement Reviewed andReviews Test Engineers. Review’s help in approved base lining desired statement of requirements to requirements. build a system.Design Reviews Designers, Test Design Reviews help System Design Engineers in validating if the Document, design meets the Hardware Design requirements and Document. build an effective system.Code Walkthroughs Developers, Code Walkthroughs Software ready for Subject Specialists, help in analyzing the initial testing by Test Engineers. coding techniques the developer. and if the code is meeting the coding standardsSoftware Testing Framework V2.0 6 of 25
  • 7. Code Inspections Developers, Formal analysis of Software ready for Subject Specialists, the program source testing by the Test Engineers. code to find defects testing team. as defined by meeting system design specification.2.1.1 Review’sThe focus of Review is on a work product ( e.g. Requirem ent s docum ent , Code et c.) .Aft er t he work product is developed, t he Proj ect Leader calls for a Review. The workproduct is dist ribut ed t o t he personnel who involves in t he review. The m ainaudience for t he review should be t he Proj ect Manager, Proj ect Leader and t heProducer of the work product.Major reviews include the following:1. In Process Reviews2. Decision Point or Phase End Reviews3. Post Implementation ReviewsLet us discuss in brief about t he above m ent ioned reviews. As per st at ist ics Reviewsuncover over 65% of t he defect s and t est ing uncovers around 30% . So, it ’s veryimportant to maintain reviews as part of the V&V strategies.In- Process ReviewIn- Process Review looks at t he product during a specific t im e period of a life cycle,such as act ivit y. They are usually lim it ed t o a segm ent of a proj ect , wit h t he goal ofident ifying defect s as work progresses, rat her t han at t he close of a phase or evenlater, when they are more costly to correct.Decision- Point or Phase- End ReviewThis review looks at t he product for t he m ain purpose of det erm ining whet her t ocont inue wit h planned act ivit ies. They are held at t he end of each phase, in asem iform al or form al way. Defect s found are t racked t hrough resolut ion, usually byway of t he exist ing defect t racking syst em . The com m on phase- end reviews areSoftware Requirements Review, Critical Design Review and Test Readiness Review. • The Soft w a r e Re qu ir e m e n t s Re vie w is aim ed at validat ing and approving t he docum ent ed soft ware requirem ent s for t he purpose of est ablishing a baseline and ident ifying analysis packages. The Developm ent Plan, Soft ware Test Plan, Configurat ion Managem ent Plan are some of the documents reviews during this phase. • The Cr it ica l D e sign Re vie w baselines t he det ailed design specificat ion. Test cases are reviewed and approved. • The Te st Re a din e ss Re vie w is perform ed when t he appropriat e applicat ion com ponent s are near com plet ing. This review will det erm ine the readiness of the application for system and acceptance testing.Post Implementation ReviewThese reviews are held aft er im plem ent at ion is com plet e t o audit t he process basedon act ual result s. Post - I m plem ent at ion reviews are also known as Postmortems andare held t o assess t he success of t he overall process aft er release and ident ify anyopport unit ies for process im provem ent . They can be held up t o t hree t o six m ont hsafter implementation, and are conducted in a format.Software Testing Framework V2.0 7 of 25
  • 8. There are three general classes of reviews:1. Informal or Peer Review2. Semiformal or Walk- Through3. Format or InspectionsPe e r Re vie w is generally a one- to- one m eet ing bet ween t he aut hor of a workproduct and a peer, init iat ed as a request for im port regarding a part icular art ifact orproblem . There is no agenda, and result s are not form ally report ed. These reviewsoccur on an as needed basis throughout each phase of a project.2.1.2 InspectionsA knowledgeable individual called a m oderat or, who is not a m em ber of t he t eam ort he aut hor of t he product under review, facilit at es inspect ions. A recorder whorecords the defects found and actions assigned assists the moderator. The meeting isplanned in advance and m at erial is dist ribut ed t o all t he part icipant s and t hepart icipant s are expect ed t o at t end t he m eet ing well prepared. The issues raisedduring t he m eet ing are docum ent ed and circulat ed am ong t he m em bers present andthe management.2.1.3 WalkthroughsThe aut hor of t he m at erial being reviewed facilit at es walk- Through. The part icipant sare led t hrough t he m at erial in one of t wo form at s; t he present at ion is m ade wit houtint errupt ions and com m ent s are m ade at t he end, or com m ent s are m adet hroughout . I n eit her case, t he issues raised are capt ured and published in a reportdist ribut ed t o t he part icipant s. Possible solut ions for uncovered defect s are notdiscussed during the review.2.2 Validation StrategiesThe Validat ion St rat egies, persons / t eam s involved in t he t est ing, and t hedeliverable of that phase of testing is briefed below:Validation Performed By Explanation DeliverableStrategyUnit Testing. Developers / Test Testing of single Software unit Engineers. program, modules, ready for testing or unit of code. with other system component.Integration Testing. Test Engineers. Testing of integrated Portions of the programs, modules, system ready for or units of code. testing with other portions of the system.System Testing. Test Engineers. Testing of entire Tested computer computer system. system, based on This kind of testing what was specified usually includes to be developed. functional and structural testing.Production Developers, Test Testing of the whole Stable application.Environment Engineers. computer systemTesting. before rolling out to the UAT.Software Testing Framework V2.0 8 of 25
  • 9. User Acceptance Users. Testing of computer Tested andTesting. system to make sure accepted system it will work in the based on the user system regardless of needs. what the system requirements indicate.Installation Test Engineers. Testing of the SuccessfullyTesting. Computer System installed during the application. Installation at the user place.Beta Testing Users. Testing of the Successfully application after the installed and installation at the running client place. application.3.0 Testing TypesThere are two types of testing: 1. Functional or Black Box Testing, 2. Structural or White Box Testing.Before t he Proj ect Managem ent decides on t he t est ing act ivit ies t o be perform ed, itshould have decided t he t est t ype t hat it is going t o follow. I f it is t he Black Box,t hen t he t est cases should be writ t en addressing t he funct ionalit y of t he applicat ion.I f it is t he Whit e Box, t hen t he Test Cases should be writ t en for t he int ernal andfunctional behavior of the system.Funct ional t est ing ensures t hat t he requirem ent s are properly sat isfied by t heapplicat ion syst em . The funct ions are t hose t asks t hat t he syst em is designed t oaccomplish.Structural testing ensures sufficient testing of the implementation of a function.3.1 White Box TestingWhit e Box Test ing; also know as glass box t est ing is a t est ing m et hod where t hetester involves in testing the individual software programs using tools, standards etc.Using white box testing methods, we can derive test cases that:1) Guarant ee t hat all independent pat hs wit hin a m odule have been exercised atlease once,2) Exercise all logical decisions on their true and false sides,3) Execute all loops at their boundaries and within their operational bounds, and4) Exercise internal data structures to ensure their validity.Advantages of White box testing:1) Logic errors and incorrect assum pt ions are inversely proport ional t o t heprobability that a program path will be executed.2) Oft en, a logical pat h is not likely t o be execut ed when, in fact , it m ay be execut edon a regular basis.3) Typographical errors are random.Software Testing Framework V2.0 9 of 25
  • 10. White Box Testing TypesThere are various t ypes of Whit e Box Test ing. Here in t his fram ework I will addressthe most common and important types.3.1.1 Basis Path TestingBasis pat h t est ing is a whit e box t est ing t echnique first proposed by Tom McCabe.The Basis pat h m et hod enables t o derive a logical com plexit y m easure of aprocedural design and use t his m easure as a guide for defining a basis set ofexecut ion pat hs. Test Cases derived t o exercise t he basis set are guarant eed t oexecute every statement in the program at least one time during testing.3.1.2 Flow Graph NotationThe flow graph depict s logical cont rol flow using a diagram m at ic not at ion. Eachstructured construct has a corresponding flow graph symbol.3.1.3 Cyclomatic ComplexityCyclom at ic com plexit y is a soft ware m et ric t hat provides a quant it at ive m easure ofthe logical complexity of a program. When used in the context of a basis path testingm et hod, t he value com put ed for Cyclom at ic com plexit y defines t he num ber forindependent pat hs in t he basis set of a program and provides us wit h an upperbound for t he num ber of t est s t hat m ust be conduct ed t o ensure t hat all st at em ent shave been executed at lease once.An independent pat h is any pat h t hrough t he program t hat int roduces at least onenew set of processing statements or a new condition.Computing Cyclomatic ComplexityCyclom at ic com plexit y has a foundat ion in graph t heory and provides us wit hextremely useful software metric. Complexity is computed in one of the three ways:1. The num ber of regions of t he flow graph corresponds t o t he Cyclom at iccomplexity.2. Cyclomatic complexity, V(G), for a flow graph, G is defined as V (G) = E- N+2Where E, is the number of flow graph edges, N is the number of flow graph nodes.3. Cyclomatic complexity, V (G) for a flow graph, G is also defined as: V (G) = P+1Where P is the number of predicate nodes contained in the flow graph G.3.1.4 Graph MatricesThe procedure for deriving t he flow graph and even det erm ining a set of basis pat hsis am enable t o m echanizat ion. To develop a soft ware t ool t hat assist s in basis pat htesting, a data structure, called a graph matrix can be quite useful.A Graph Mat rix is a square matrix whose size is equal to the number of nodes on theflow graph. Each row and colum n corresponds t o an ident ified node, and m at rixentries correspond to connections between nodes.3.1.5 Control Structure TestingDescribed below are some of the variations of Control Structure Testing. 3.1.5.1 Condition Testing Condit ion t est ing is a t est case design m et hod t hat exercises t he logical conditions contained in a program module. 3.1.5.2 Data Flow Testing The dat a flow t est ing m et hod select s t est pat hs of a program according t o t he locations of definitions and uses of variables in the program.Software Testing Framework V2.0 10 of 25
  • 11. 3.1.6 Loop TestingLoop Test ing is a whit e box t est ing t echnique t hat focuses exclusively on t he validit yof loop const ruct s. Four classes of loops can be defined: Sim ple loops, Concat enat edloops, nested loops, and unstructured loops. 3.1.6.1 Simple Loops The following set s of t est s can be applied t o sim ple loops, where ‘n’ is t he maximum number of allowable passes through the loop. 1. Skip the loop entirely. 2. Only one pass through the loop. 3. Two passes through the loop. 4. ‘m’ passes through the loop where m<n. 5. n- 1, n, n+1 passes through the loop. 3.1.6.2 Nested Loops I f we ext end t he t est approach for sim ple loops t o nest ed loops, t he num ber of possible tests would grow geometrically as the level of nesting increases. 1. Start at the innermost loop. Set all other loops to minimum values. 2. Conduct sim ple loop t est s for t he innerm ost loop while holding t he out er loops at t heir m inim um it erat ion param et er values. Add ot her t est s for out - of- range or exclude values. 3. Work out ward, conduct ing t est s for t he next loop, but keeping all ot her out er loops at minimum values and other nested loops to “typical” values. 4. Continue until all loops have been tested. 3.1.6.3 Concatenated Loops Concat enat ed loops can be t est ed using t he approach defined for sim ple loops, if each of t he loops is independent of t he ot her. However, if t wo loops are concat enat ed and t he loop count er for loop 1 is used as t he init ial value for loop 2, then the loops are not independent. 3.1.6.4 Unstructured Loops Whenever possible, t his class of loops should be redesigned t o reflect t he use of the structured programming constructs.3.2 Black Box TestingBlack box t est ing, also known as behavioral t est ing focuses on t he funct ionalrequirem ent s of t he soft ware. All t he funct ional requirem ent s of t he program will beused to derive sets of input conditions for testing.Black Box Testing TypesThe following are the most famous/frequently used Black Box Testing Types. 3.2.1 Graph Based Testing Methods Soft ware t est ing begins by creat ing a graph of im port ant obj ect s and t heir relationships and t hen devising a series of t est s t hat will cover t he graph so that each objects and their relationships and then devising a series of tests that will cover t he graph so t hat each obj ect and relat ionship is exercised and error are uncovered. 3.2.2 Equivalence Partitioning Equivalence part it ioning is a black box t est ing m et hod t hat divides t he input domain of a program into classes of data from which test cases can be derived. EP can be defined according to the following guidelines:Software Testing Framework V2.0 11 of 25
  • 12. 1. I f an input condit ion specifies a range, one valid and one t wo invalid classes are defined. 2. I f an input condit ion requires a specific value, one valid and t wo invalid equivalence classes are defined. 3. I f an input condit ion specifies a m em ber of a set , one valid and one invalid equivalence class are defined. 4. If an input condition is Boolean, one valid and one invalid class are defined. 3.2.3 Boundary Value Analysis BVA is a t est case design t echnique t hat com plem ent s equivalence part it ioning. Rat her t han select ing any elem ent of an equivalence class, BVA leads t o t he select ion of t est cases at t he “ edges” of t he class. Rat her t han focusing solely on input conditions, BVA derives test cases from the output domain as well. Guidelines for BVA are sim ilar in m any respect s t o t hose provided for equivalence partitioning. 3.2.4 Comparison Testing Sit uat ions where independent versions of soft ware be developed for crit ical applicat ions, even when only a single version will be used in t he delivered com put er based syst em . These independent versions from t he basis of a black box testing technique called Comparison testing or back- to- back testing. 3.2.5 Orthogonal Array Testing The orthogonal array testing method is particularly useful in finding errors associated with region faults – an error category associated with faulty logic within a software component.3.3 Scenario Based Testing (SBT)Dr.Cem Kaner in “ A Pat t ern for Scenario Test ing” has explained scenario BasedTesting in great detail that can be found at www.testing.com.What is Scenario Based Test ing and How/ Where is it useful is an int erest ingquestion. I shall explain in brief the above two mentioned points.Scenario Based Test ing is cat egorized under Black Box Test s and are m ost helpfulwhen t he t est ing is concent rat ed on t he Business logic and funct ional behavior of t heapplicat ion. Adopt ing SBT is effect ive when t est ing com plex applicat ions. Now, everyapplicat ion is com plex, t hen it ’s t he t eam s call as t o im plem ent SBT or not . I wouldpersonally suggest using SBT when t he funct ionalit y t o t est includes various feat uresand funct ions. A best exam ple would be while t est ing banking applicat ion. Asbanking applicat ions require ut m ost care while t est ing, handling various funct ions ina single scenario would result in effective results.A sam ple t ransact ion ( scenario) can be, a cust om er logging int o t he applicat ion,checking his balance, t ransferring am ount t o anot her account , paying his bills,checking his balance again and logging out.In brief, use Scenario Based Tests when: 1. Testing complex applications. 2. Testing Business functionality.When designing scenarios, keep in mind: 1. The scenario should be close to the real life scenario. 2. Scenarios should be realistic. 3. Scenarios should be traceable to any/combination of functionality. 4. Scenarios should be supported by sufficient data.Software Testing Framework V2.0 12 of 25
  • 13. 3.4 Exploratory TestingExplorat ory Test s are cat egorized under Black Box Test s and are aim ed at t est ing inconditions when sufficient time is not available for testing or proper documentation isnot available.Exploratory testing is ‘Testing while Exploring’. When you have no idea of how theapplication works, exploring the application with the intent of finding errors can betermed as Exploratory Testing.Performing Exploratory TestingThis is one big question for many people. The following can be used to performExploratory Testing: • Learn the Application. • Learn the Business for which the application is addressed. • Learn the technology to the maximum extent on which the application has been designed. • Learn how to test. • Plan and Design tests as per the learning.4.0 Structural System Testing TechniquesThe following are the structural system testing techniques.Technique Description ExampleStress Determine system performance Sufficient disk space with expected volumes. allocated.Execution System achieves desired level of Transaction turnaround proficiency. time adequate.Recovery System can be returned to an Evaluate adequacy of operational status after a failure. backup data.Operations System can be executed in a Determine systems can normal operational status. run using document.Compliance System is developed in accordance Standards follow. with standards and procedures.Security System is protected in accordance Access denied. with importance to organization.5.0 Functional System Testing TechniquesThe following are the functional system testing techniques.Technique Description ExampleRequirements System performs as specified. Prove system requirements.Regression Verifies that anything unchanged Unchanged system still performs correctly. segments function.Error Handling Errors can be prevented or Error introduced into the detected and then corrected. test.Manual Support The people- computer interaction Manual procedures works. developed.Intersystems. Data is correctly passed from Intersystem parameters system to system. changed.Control Controls reduce system risk to an File reconciliation acceptable level. procedures work.Software Testing Framework V2.0 13 of 25
  • 14. Parallel Old systems and new system are Old and new system can run and the results compared to reconcile. detect unplanned differences.4.0 Testing Phases Requirement Study Requirement Checklist Software Requirement Specification Software Requirement Functional Specification Specification Checklist Functional Specification Document Functional Specification Architecture Design Document Architecture Design Detailed Design Document Coding Functional Specification Unit Test Case Documents Document Unit Test Case Document Design Document System Test Case Document Functional Specification Document Integration Test Case Document Unit/Integration/System Regression Test Case Test Case Documents Document Functional Specification Document Performance Test Cases and Scenarios Performance Criteria Software Requirement Specification Regression Test Case User Acceptance Test Case Document Documents/Scenarios Performance Test Cases and ScenariosSoftware Testing Framework V2.0 14 of 25
  • 15. 4.2 Unit TestingGoal of Unit t est ing is t o uncover defect s using form al t echniques like BoundaryValue Analysis ( BVA) , Equivalence Part it ioning, and Error Guessing. Defect s anddeviat ions in Dat e form at s, Special requirem ent s in input condit ions ( for exam pleText box where only num eric or alphabet s should be ent ered) , select ion based onCom bo Box’s, List Box’s, Opt ion but t ons, Check Box’s would be ident ified during t heUnit Testing phase.4.3 Integration TestingI nt egrat ion t est ing is a syst em at ic t echnique for const ruct ing t he program st ruct urewhile at t he sam e t im e conduct ing t est s t o uncover errors associat ed wit hint erfacing. The obj ect ive is t o t ake unit t est ed com ponent s and build a programstructure that has been dictated by design.Usually, the following methods of Integration testing are followed: 1. Top- down Integration approach. 2. Bottom- up Integration approach.4.3.1 Top- down IntegrationTop- down int egrat ion t est ing is an increm ent al approach t o const ruct ion of programst ruct ure. Modules are int egrat ed by m oving downward t hrough t he cont rolhierarchy, beginning wit h t he m ain cont rol m odule. Modules subordinat e t o t he m aincont rol m odule are incorporat ed int o t he st ruct ure in eit her a dept h- first or breadt h-first manner. 1. The Integration process is performed in a series of five steps: 2. The main control module is used as a test driver and stubs are substituted for all components directly subordinate to the main control module. 3. Depending on t he int egrat ion approach select ed subordinat e st ubs are replaced one at a time with actual components. 4. Tests are conducted as each component is integrated. 5. On com plet ion of each set of t est s, anot her st ub is replaced wit h t he real component. 6. Regression t est ing m ay be conduct ed t o ensure t hat new errors have not been introduced.4.3.2 Bottom- up IntegrationButton- up int egrat ion t est ing begins const ruct ion and t est ing wit h at om ic m odules(i.e. components at the lowest levels in the program structure). Because componentsare int egrat ed from t he but t on up, processing required for com ponent s subordinat eto a given level is always available and the need for stubs is eliminated. 1. A Bot t om - up int egrat ion st rat egy m ay be im plem ent ed wit h t he following steps: 2. Low level com ponent s are com bined int o clust ers t hat perform a specific software sub function. 3. A driver is written to coordinate test case input and output. 4. The cluster is tested. 5. Drivers are rem oved and clust ers are com bined m oving upward in t he program structure.Software Testing Framework V2.0 15 of 25
  • 16. 4.4 Smoke Testing“Smoke testing might be a characterized as a rolling integration strategy”.Sm oke t est ing is an int egrat ion t est ing approach t hat is com m only used when“shrink- wrapped” soft ware product s are being developed. I t is designed as a pacingm echanism for t im e- critical projects, allowing the software team to assess its projecton a frequent basis.The sm oke t est should exercise t he ent ire syst em from end t o end. Sm oke t est ingprovides benefits such as:1) Integration risk is minimized.2) The quality of the end- product is improved.3) Error diagnosis and correction are simplified.4) Progress is easier to asses.4.5 System TestingSyst em t est ing is a series of different t est s whose prim ary purpose is t o fullyexercise t he com put er based syst em . Alt hough each t est has a different purpose, allwork t o verify t hat syst em elem ent s have been properly int egrat ed and performallocated functions.The following tests can be categorized under System testing: 1. Recovery Testing. 2. Security Testing. 3. Stress Testing. 4. Performance Testing.4.5.1. Recovery TestingRecovery t est ing is a syst em t est t hat focuses t he soft ware t o fall in a variet y ofways and verifies t hat recovery is properly perform ed. I f recovery is aut om at ic,reinit ializat ion, checkpoint ing m echanism s, dat a recovery and rest art are evaluat edfor correct ness. I f recovery requires hum an int ervent ion, t he m ean- time- to- repair(MTTR) is evaluated to determine whether it is within acceptable limits.4.5.2. Security TestingSecurit y t est ing at t em pt s t o verify t hat prot ect ion m echanism s built int o a syst emwill, in fact , prot ect it from im proper penet rat ion. During Securit y t est ing, passwordcracking, unaut horized ent ry int o t he soft ware, net work securit y are all t aken int oconsideration.4.5.3. Stress TestingSt ress t est ing execut es a syst em in a m anner t hat dem ands resources in abnorm alquant it y, frequency, or volum e. The following t ypes of t est s m ay be conduct edduring stress testing; • Special t est s m ay be designed t hat generat e t en int errupt s per second, when one or two is the average rate. • I nput dat a rat es m ay be increases by an order of m agnit ude t o det ermine how input functions will respond. • Test Cases that require maximum memory or other resources. • Test Cases that may cause excessive hunting for disk- resident data. • Test Cases that my cause thrashing in a virtual operating system.4.5.4. Performance TestingPerform ance t est s are coupled wit h st ress t est ing and usually require bot h hardwareand software instrumentation.Software Testing Framework V2.0 16 of 25
  • 17. 4.5.5. Regression TestingRegression testing is the re- execution of some subset of tests that have already beenconducted to ensure that changes have not propagated unintended side affects.Regression may be conducted manually, by re- executing a subset of al test cases orusing automated capture/playback tools.The Regression test suit contains three different classes of test cases: • A representative sample of tests that will exercise all software functions. • Additional tests that focus on software functions that are likely to be affected by the change. • Tests that focus on the software components that have been changed.4.6 Alpha TestingThe Alpha testing is conducted at the developer sites and in a controlled environmentby the end- user of the software.4.7 User Acceptance TestingUser Accept ance t est ing occurs j ust before t he soft ware is released t o t he cust om er.The end- users along with the developers perform the User Acceptance Testing with acertain set of test cases and typical scenarios.4.8 Beta TestingThe Bet a t est ing is conduct ed at one or m ore cust om er sit es by t he end- user of t hesoft ware. The bet a t est is a live applicat ion of t he soft ware in an environm ent t hatcannot be controlled by the developer.5.0 MetricsMet rics are t he m ost im port ant responsibilit y of t he Test Team . Met rics allow fordeeper underst anding of t he perform ance of t he applicat ion and it s behavior. Thefine t uning of t he applicat ion can be enhanced only wit h m et rics. I n a t ypical QAprocess, there are many metrics which provide information.The following can be regarded as the fundamental metric:IEEE Std 982.2 - 1988 defines a Functional or Test Coverage Metric. It can be usedto measure test coverage prior to software delivery. It provide a measure of thepercentage of the software tested at any point during testing.It is calculated as follows:Function Test Coverage = FE/FTWhereFE is the number of test requirements that are covered by test cases that wereexecuted against the softwareFT is the total number of test requirementsSoftware Release MetricsThe software is ready for release when:1. It has been tested with a test suite that provides 100% functional coverage, 80%branch coverage, and 100% procedure coverage.2. There are no level 1 or 2 severity defects.3. The defect finding rate is less than 40 new defects per 1000 hours of testing4. The software reaches 1000 hours of operation5. Stress testing, configuration testing, installation testing, Naïve user testing,usability testing, and sanity testing have been completedSoftware Testing Framework V2.0 17 of 25
  • 18. IEEE Software Maturity MetricIEEE Std 982.2 - 1988 defines a Software Maturity Index that can be used todetermine the readiness for release of a software system. This index is especiallyuseful for assessing release readiness when changes, additions, or deletions aremade to existing software systems. It also provides an historical index of the impactof changes. It is calculated as follows:SMI = Mt - ( Fa + Fc + Fd)/MtWhereSMI is the Software Maturity Index valueMt is the number of software functions/modules in the current releaseFc is the number of functions/modules that contain changes from the previousreleaseFa is the number of functions/modules that contain additions to the previous releaseFd is the number of functions/modules that are deleted from the previous releaseReliability MetricsPerry offers the following equation for calculating reliability.Reliability = 1 - Number of errors (actual or predicted)/Total number oflines of executable codeThis reliability value is calculated for the number of errors during a specified timeinterval.Three other metrics can be calculated during extended testing or after the system isin production. They are:MTTFF (Mean Time to First Failure)MTTFF = The number of time intervals the system is operable until its first failureMTBF (Mean Time Between Failures)MTBF = Sum of the time intervals the system is operableNumber of failures for the time periodMTTR (Mean Time To Repair)MTTR = sum of the time intervals required to repair the systemThe number of repairs during the time periodSoftware Testing Framework V2.0 18 of 25
  • 19. 6.0 Test ModelsThere are various models of Software Testing. Here in this framework I wouldexplain the three most commonly used models: 1. The ‘V’ Model. 2. The ‘W’ Model. 3. The Butterfly Model6.1 The ‘V’ ModelThe following diagram depicts the ‘V’ Model Requirements Acceptance Tests System Tests Specification Integration Tests Architecture Unit Tests Detailed Design CodingThe diagram is self- explanatory. For an easy understanding, look at the followingtable:SDLC Phase Test Phase1. Requirements 1. Build Test Strategy. 2. Plan for Testing. 3. Acceptance Test Scenarios Identification.2. Specification 1. System Test Case Generation.3. Architecture 1. Integration Test Case Generation.4. Detailed Design 1. Unit Test Case GenerationSoftware Testing Framework V2.0 19 of 25
  • 20. 6.2 The ‘W’ ModelThe following diagram depicts the ‘W’ model: Regression Requirements Requirements Round 3 Review Performance Testing Regression Specification Round 2 Specification System Review Testing Architecture Regression Architecture Integration Round 1 Review Testing Detailed Design Unit Design Testing Review Code Code WalkthroughThe ‘W’ model depicts that the Testing starts from day one of the initiation of theproject and continues till the end. The following table will illustrate the phases ofactivities that happen in the ‘W’ model:SDLC Phase The first ‘V’ The second ‘V’1. Requirements 1. Requirements Review 1. Build Test Strategy. 2. Plan for Testing. 3. Acceptance (Beta) Test Scenario Identification.2. Specification 2. Specification Review 1. System Test Case Generation.3. Architecture 3. Architecture Review 1. Integration Test Case Generation.4. Detailed Design 4. Detailed Design Review 1. Unit Test Case Generation.5. Code 5. Code Walkthrough 1. Execute Unit Tests 1. Execute Integration Tests. 1. Regression Round 1. 1. Execute System Tests. 1. Regression Round 2. 1. Performance Tests 1. Regression Round 3 1. Performance/Beta TestsSoftware Testing Framework V2.0 20 of 25
  • 21. In the second ‘V’, I have mentioned Acceptance/Beta Test Scenario Identification. This is because, the customer might want to design the Acceptance Tests. In this case as the development team executes the Beta Tests at the client place, the same team can identify the Scenarios. Regression Rounds are performed at regular intervals to check whether the defects, which have been raised and fixed, are re- tested. 6.3 The Butterfly Model The t est ing act ivit ies for t est ing soft ware product s are preferable t o follow t he Butterfly Model. The following picture depicts the test methodology.Test Design Test AnalysisTest Execution Fig: Butterfly Model I n t he Butterfly m odel of Test Developm ent , t he left wing of t he but t erfly depict s the Te st An a lysis. The right wing depict s t he Te st D e sign , and finally t he body of t he but t erfly depict s t he Te st Ex e cu t ion . How t his exact ly happens is described below. Test Analysis Analysis is t he key fact or which drives in any planning. During t he analysis, t he analyst understands the following: • Verify t hat each requirem ent is t agged in a m anner t hat allows correlat ion of t he tests for that requirement to the requirement itself. (Establish Test Traceability) • Verify traceability of the software requirements to system requirements. • Inspect for contradictory requirements. • Inspect for ambiguous requirements. • Inspect for missing requirements. • Check t o m ake sure t hat each requirem ent , as well as t he specificat ion as a whole, is understandable. • I dent ify one or m ore m easurem ent , dem onst rat ion, or analysis m et hod t hat m ay be used to verify the requirement’s implementation (during formal testing). • Creat e a t est “ sket ch” t hat includes t he t ent at ive approach and indicat es t he test’s objectives. During Test Analysis t he required docum ent s will be carefully st udied by t he Test Personnel, and the final Analysis Report is documented. The following documents would be usually referred: 1. Software Requirements Specification. 2. Functional Specification. 3. Architecture Document. Software Testing Framework V2.0 21 of 25
  • 22. 4. Use Case Documents.The An a lysis Re por t would consist of t he underst anding of t he applicat ion, t hefunct ional flow of t he applicat ion, num ber of m odules involved and t he effect ive TestTime.Test DesignThe right wing of t he but t erfly represent s t he act of designing and im plem ent ing t het est cases needed t o verify t he design art ifact as replicat ed in t he im plem ent at ion.Like test analysis, it is a relatively large piece of work. Unlike test analysis, however,t he focus of t est design is not t o assim ilat e inform at ion creat ed by ot hers, but rat hert o im plem ent procedures, t echniques, and dat a set s t hat achieve t he t est ’sobjective(s).The out put s of t he t est analysis phase are t he foundat ion for t est design. Eachrequirem ent or design const ruct has had at least one t echnique ( a m easurem ent ,dem onst rat ion, or analysis) ident ified during t est analysis t hat will validat e or verifythat requirement. The tester must now implement the intended technique.Soft ware t est design, as a discipline, is an exercise in t he prevent ion, det ect ion, andelim inat ion of bugs in soft ware. Prevent ing bugs is t he prim ary goal of soft waret est ing. Diligent and com pet ent t est design prevent s bugs from ever reaching t heim plem ent at ion st age. Test design, wit h it s at t endant t est analysis foundat ion, ist herefore t he prem iere weapon in t he arsenal of developers and t est ers for lim it ingthe cost associated with finding and fixing bugs.During Test Design, basing on t he Analysis Report t he t est personnel would developthe following: 1. Test Plan. 2. Test Approach. 3. Test Case documents. 4. Performance Test Parameters. 5. Performance Test Plan.Test ExecutionAny test case should adhere to the following principals: 1. Accurate – tests what the description says it will test. 2. Economical – has only the steps needed for its purpose. 3. Repeatable – tests should be consistent, no matter who/when it is executed. 4. Appropriate – should be apt for the situation. 5. Traceable – the functionality of the test case should be easily found.During t he Test Execut ion phase, keeping t he Proj ect and t he Test schedule, t he t estcases designed would be execut ed. The following docum ent s will be handled duringthe test execution phase:1. Test Execution Reports.2. Daily/Weekly/monthly Defect Reports.3. Person wise defect reports.After the Test Execution phase, the following documents would be signed off.1. Project Closure Document.2. Reliability Analysis Report.3. Stability Analysis Report.4. Performance Analysis Report.5. Project Metrics.Software Testing Framework V2.0 22 of 25
  • 23. 7.0 Defect Tracking ProcessThe Defect Tracking process should answer the following questions: 1. When is the defect found? 2. Who raised the defect? 3. Is the defect reported properly? 4. Is the defect assigned to the appropriate developer? 5. When was the defect fixed? 6. Is the defect re- tested? 7. Is the defect closed?The defect tracking process has to be handled carefully and managed efficiently.The following figure illustrates the defect tracking process: The Tester/Developer finds the Bug. Reports the Defect in the Defect Tracking Tool. Status “Open” The concerned Developer is informed The Developer fixes the Defect The Developer If the Defect re- changes the Status to occurs, the status “Resolved” changes to “Re- Open” The Tester Re- Tests and changes Status to “Closed”Defect ClassificationThis sect ion defines a defect Severit y Scale fram ework for det erm ining defectcrit icalit y and t he associat ed defect Priorit y Levels t o be assigned t o errors foundsoftware.Software Testing Framework V2.0 23 of 25
  • 24. The defects can be classified as follows:Classification DescriptionCritical There is s funct ionalit y block. The applicat ion is not able t o proceed any further.Major The applicat ion is not working as desired. There are variat ions in the functionality.Minor There is no failure report ed due t o t he defect , but cert ainly needs to be rectified.Cosmetic Defects in the User Interface or Navigation.Suggestion Feature which can be added for betterment.Priority Level of the DefectThe priorit y level describes t he t im e for resolut ion of t he defect . The priorit y levelwould be classified as follows:Classification DescriptionImmediate Resolve the defect with immediate effect.At the Earliest Resolve the defect at the earliest, on priority at the second level.Normal Resolve the defect.Later Could be resolved at the later stages.8.0 Test Process for a ProjectI n t his sect ion, I would explain how t o go about planning your t est ing act ivit ieseffect ively and efficient ly. The process is explained in a t abular form at giving t hephase of testing, activity and person responsible.For t his, I assum e t hat t he proj ect has been ident ified and t he t est ing t eam consist sof five personnel: Test Manager, Test Lead, Senior Test Engineer and 2 TestEngineer’s.SDLC Phase Testing Phase/Activity Personnel1. Requirements 1. Study the requirements for Test Manager / Test Lead Testability. 2. Design the Test Strategy. 3. Prepare the Test Plan. 4. Identify scenarios for Acceptance/Beta Tests2. Specification 1. Identify System Test Cases / Test Lead, Senior Test Scenarios. Engineer, and Test Engineers. 2. Identify Performance Tests.3. Architecture 1. Identify Integration Test Cases / Test Lead, Senior Test Scenarios. Engineer, and Test Engineers. 2. Identify Performance Tests.4. Detailed 1. Generate Unit Test Cases Test Engineers.DesignSoftware Testing Framework V2.0 24 of 25
  • 25. 9.0 DeliverablesThe Deliverables from the Test team would include the following: 1. Test Strategy. 2. Test Plan. 3. Test Case Documents. 4. Defect Reports. 5. Status Reports (Daily/weekly/Monthly). 6. Test Scripts (if any). 7. Metric Reports. 8. Product Sign off Document.Software Testing Framework V2.0 25 of 25

×