Software Architecture-based Regression Testing
Upcoming SlideShare
Loading in...5
×
 

Software Architecture-based Regression Testing

on

  • 522 views

 

Statistics

Views

Total Views
522
Views on SlideShare
521
Embed Views
1

Actions

Likes
0
Downloads
7
Comments
0

1 Embed 1

http://www.slideshare.net 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Software Architecture-based Regression Testing Software Architecture-based Regression Testing Presentation Transcript

  • Software Architecture-based Regression Testing Henry Muccini Software Engineering and Architecture Group Università degli Studi dell'Aquila I-67100 L'Aquila, Italy muccini@di.univaq.it Marcio Dias Debra J. Richardson Department of Computer Science and Donald Bren School of Information e-Science Research Institute, and Computer Sciences, University of Durham, UK University of California Irvine, USA
  • My main research areas: Analysis » SEA Group - Software Engineering and Architecture Group » Software Architecture Analysis: - Model-Checking SAs (the CHARMY framework) - SA-based Testing - SA-based Regression Testing (the SARTE project) - Model-checking driven Testing (the ModTest approach) » Product Line: - Modeling Product Line Architecture SEA Group - Testing and Model Checking of Product Line 2 © 2005 by H. Muccini, WADS 2005
  • Our Experience on SA-based analysis SA conformance to requirements through MC C identify validate the SA model h Requirements M e conformance o c with respect to selected d drive k SA model functional properties e i functional Charmy Project l n Software M.C. properties Architecture [NOK] g refine SA OK [test case selection] [www.di.univaq.it/charmy] Software Extract SA-level Architecture Test cases T e provide confidence on the s SATC implementation t fulfillment to its i System Map SATC n Implementation into real architectural spec g Test Cases SA-based Testing [NOK] Test Test Fault removal Exec Procedures SEA Group 3 © 2005 by H. Muccini, WADS conformance to SA through Testing Implementation 2005
  • ModTest: Model-Checking driven Testing SA conformance to requirements through MC a1 C Requirements identify Specification h Requirements a2 M e Architectural SA Revision o c Specification d drive k SA model a3 e i functional Charmy l Software M.C. properties Analysis n a4 Architecture [NOK] Critical g refine SA OK [test case selection] NOK sub-systems identification OK a5 a7 TeStor System T a6 Implementation e Test spec Test Case s System Implementation t Implementation i a8 n [NOK] Test Test Case Test g Procedures Execution Implementation Fault removal Exec Revision NOK OK SEA GroupImplementation conformance to SA through Testing System Release 4 © 2005 by H. Muccini, WADS 2005
  • Our Recent experience in SA-based Analysis » Industrial Experience - PSTDA Italy [ICSE00,ICSE01] - Telcordia - Marconi [FME 03] - Siemens [ITM 04] » Academic Experience - [FASE 04][IEEE TSE04] - [CBSE 05][COMPSAC05] SEA Group 5 © 2005 by H. Muccini, WADS 2005
  • Considerations » What happens if the (architectural) model changes? - Usually, we need to remake analysis from scratch Software Architecture SA2 Software Architecture SA1 (version 2) (version 1) Component B Component B Component C' Component A Component C Component A Component Y SEA Group 6 © 2005 by H. Muccini, WADS 2005
  • How changes affect ModTest SA conformance to requirements through MC C identify h Requirements M e o c SA spec d drive k SA model e i functional l M.C. properties n Software Architecture [NOK] Test Case g refine SA OK [test case selection] selection T e Test Test spec s System implementation t Implementation i n [NOK] Test Test Test g Fault removal Exec Procedures execution SEA Group Implementation conformance to SA through Testing 7 © 2005 by H. Muccini, WADS 2005
  • SA-based Regression Testing [WADS05] [COMPSAC05] Software Architecture SA2 Software Architecture SA1 (version 2) (version 1) Component B Component B SA evolution Component C' Component A Component C Component A Component Y Test Reuse Code Code1 evolution Code2 Code (version 1) (version 2) SEA Group Test Reuse Test Reuse 8 © 2005 by H. Muccini, WADS 2005
  • Traditional Regression Testing » Test modified software to provide a certain confidence that no new errors are introduced into previously tested code. » Two key phases: i) testing the program P with respect to a specified test suite T, and ii) when a new version P' is released, regression testing of the modified version P' versus a test suite T' » Selective RT: - Goal: selecting T' as a “relevant” subset of T > t1 in T is included in T ' if there is the potential that it could produce different results on P' than it did on P SEA Group 9 © 2005 by H. Muccini, WADS 2005
  • First phase: SA-based Code Testing » The code conformance to the SA has been already tested Software Architecture “S” Code “P” XClient Component Class ------- Connector ClientA ClientB Component ------ ------ ClientC ------- Component Identify relevant Refine SA-level tests Run TC over P and SA-level Test Cases into Code-level tests Evaluate Results SEA Group Step 4 Steps 0 - 2 Step 3 10 © 2005 by H. Muccini, WADS 2005
  • SArTe – Goal 1 (code evolution) Software Architecture SA1 » Test Conformance of a Modified (version 1) Implementation P' to the initial SA Component B - Context: Component C Component A > P correctly implements the SA S > P' modifies P: some objects are modified, and/or some new objects are introduced. - Goal: Test the conformance of P' with respect to S, > while reusing previous test Code1 information for selective regression Code2 testing, thereby reducing the test cases (version 1) (version 2) that must be retested. Test Reuse - To handle Architectural Drift SEA Group 11 © 2005 by H. Muccini, WADS 2005
  • SArTe – Goal 2 (SA evolution) » Test Conformance of an Evolved Software Architecture SA2 Software Architecture Software Architecture SA1 (version 2) (version 1) Component B - Context: Component B Component C' > P correctly implements the SA S Component A Component C Component A Component Y > S’ modifies S, adding or removing components Test > A modified implementation P' may have Reuse been also developed. - Goal: Test the conformance of P’ with respect to S', Code2 Code (version 2) > while reusing previous test information Test for selective RT, thereby reducing the Reuse test cases that must be retested. SEA Group 12 © 2005 by H. Muccini, WADS 2005
  • Goal 1: P changes SA-based Code Testing Step 0 SA-spec for S Step 1 SA-based Testing Criterion Regression Testing Step 2 - Goal 1 - Extract SA-level Step A Test Cases (ATCs) Build the GP Build the GP’ graph for P graph for P’ Step 3 Map ATCs into Code-level Test Cases Step B as a way to select T for P Compare GP with GP’ Step 4 Step C Run T over P Create a Test History and Evaluate Results Step D Select T' for P' SEA Group 13 © 2005 by H. Muccini, WADS 2005
  • Considerations » Differences with respect to traditional code-based selective RT techniques: - code-level test cases are always selected starting from a well formalized architectural specification. - the oracle in SA-based RT is the software architecture specification itself. » Advantages: - as in traditional RT, we reduce the size of the test suite for P', eliminating all those tests which do not need to be reapplied to P', and - when conformance faults are detected, we can gather information on how to adjust the initial architecture. SEA Group 14 © 2005 by H. Muccini, WADS 2005
  • Goal 2: SA changes SA-based Regression Testing SA-based Code Testing - Goal 2 - Step a Step 0 SA-spec for S’’ SA-spec for S Step b Step 1 Testing Criterion Testing Criterion Step 2 Extract SA-level Step c Test Cases (ATCs) Compare S and S’’ Step 3 Map ATCs into Code-level Step d Test Cases as a way to select T for P Select ATCs’’ from S that need to be retested in S’’ Step 4 Step e Run T over P Map ATCs’’ into Code-level Test and Evaluate Results Cases as a way to select T’’ for P’’ Step f SEA Group Run T’’ over P’’ and Evaluate Results 15 © 2005 by H. Muccini, WADS 2005
  • Goal 2 Idea » Compare the two architectural specifications to identify changed/unchanged portions of the SA. - Both structural and behavioral changes are taken into account > We compare the topology changes (if the SA structure changed) > We compare the behavioral changes (if the SA behavior changed) - and, in a fashion similar to traditional code-based RT, > ATC needs to be re-run in S'‘, if it traverses a path modified when moving from S to S'' SEA Group 16 © 2005 by H. Muccini, WADS 2005
  • Experiment 1: the Elevator SA Elevator System Elevator System Version 1 Version 2 ElevatorADT1 ElevatorADT1 ElevatorADT2 ElevatorPanel1 ElevatorPanel2 ElevatorPanel1 Scheduler BuildingPanel BuildingPanel SEA Group 17 © 2005 by H. Muccini, WADS 2005
  • Experiment 2: the Cargo Router System Cargo Router, Version 2 Cargo Router, Version 1 Note: In Version2, the connection (*) Port Vehicle Warehouse Clock is replaced with the connections (**) (P) (V) (W) (C) Bus1 Planner (P) NextShipment (NS) (**) (*) Bus2 Bus2A (**) Bus2B Port Vehicle Warehouse Port Vehicle Warehouse Artist (PA) Artist (VA) Artist (WA) Artist2 (PA2) Artist2 (VA2) Artist2 (WA2) Bus3 Bus3A CargoRouter CargoRouter2 (CR) Bus3C (CR2) Planner Artist (PlA) Translator (T) Bus4 Graphics Binding (GB) SEA Group a) b) 18 © 2005 by H. Muccini, WADS 2005
  • Future Work » To reconstruct the actual architecture when the first goal determines that the code no longer conforms to the initial SA » Regression Testing of Component-based Software Architectures » Regression-based Analysis of ModTest » Apply/refine this approach into real systems (SiemensC.N.X., Terma GmbH) SEA Group 19 © 2005 by H. Muccini, WADS 2005
  • Contact Information » Henry Muccini - Dipartimento di Informatica, Universita' dell'Aquila, L'Aquila, Italy - muccini@di.univaq.it » Marcio Dias - Department of Computer Science and e-Science Research Institute, University of Durham, UK - marcio.dias@dur.ac.uk » Debra J. Richardson - Donald Bren School of Information and Computer Sciences, University of California Irvine, USA - djr@ics.uci.edu SEA Group 20 © 2005 by H. Muccini, WADS 2005
  • SA diff e1 e1 e1 e1 e1 e1 in(x) in(x) in(x) S0 S1 S0 S0 S1 S1 in(a) in(a) in(a) e10 e1 e10 e10 e1 e1 e4 e4 e4 out(z) out(z) out(z) e4 e4 e4 S2 in(y) S2 S2 in(y) in(y) S4 S4 S4 S5 S5 S5 e8 in(z) e5 e8 e8 in(z) in(z) e5 out(w) out(w) e7 e6 S3 e7 e7 S3 S3 S5' S5' in(b) in(b) in(b) e6 e6 S7 e0 S7 S7 e0 e0 S6 out(c) S6 S6 out(c) out(c) e9 e9 e9 e6 e6 e6 ATC1 in T ATC2 in T ATC1 in T’ SEA Group 21 © 2005 by H. Muccini, WADS 2005