MTS: Controllable Test Objects
Upcoming SlideShare
Loading in...5
×
 

MTS: Controllable Test Objects

on

  • 286 views

Invited Talk, ISSTA 2nd International Workshop on End-to-end Test Script Engineering ...

Invited Talk, ISSTA 2nd International Workshop on End-to-end Test Script Engineering
July 16, 2012, Minneapolis.
Limitations of x-unit testing framework, MTS testing framework that combined test objects with procedural aspects of TTCN.

Statistics

Views

Total Views
286
Views on SlideShare
286
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

MTS: Controllable Test Objects MTS: Controllable Test Objects Presentation Transcript

  • MTS: Controllable Test Objects Robert V. Binder2nd International Workshop on End-to-end Test Script Engineering July 16, 2012 Minneapolis Copyright © 2012, Robert V. Binder
  • Overview• Testing in Flatland• Overview of MTS• Lessons Learned• Q&AJuly 16, 2012 MTS - Controllable Test Objects 2
  • July 16, 2012 MTS - Controllable Test Objects 3
  • The xUnit Phenomena jUnit cppUnitJuly 16, 2012 MTS - Controllable Test Objects 4
  • Incremental Testing Framework «uses» TestRunner +Contains TestLog -Uses TestObject 1 1 1 1 0..1 TestSuite -Contains TestCase -Contains 0..1 0..* 0..* MyTestSuite MyTestObjectJuly 16, 2012 MTS - Controllable Test Objects 5
  • Testing in Flatland TestCases TestSuites • Run A A pqrstv p s p q B C • Run B q t B r s t u r s C r u • Run C C t u t uJuly 16, 2012 MTS - Controllable Test Objects 6
  • Straight TrackJuly 16, 2012 MTS - Controllable Test Objects 7
  • xUnit’s One Track Strategy• Define and nest groups • Obstacle to any rich• Run any sequence of strategy groups – Select, repeat• No support for group or – Skip, Halt member – Time – Repeat – Broadcast – Select – SynchronizeJuly 16, 2012 MTS - Controllable Test Objects 8
  • Composition Isn’t Enough• atom == test case • How to do ( p | q )• ( ) group• * quantification • How to do (t ( p | q ))*• | select (or) xUnit is like reg exp without * ? + | A formal language without alternation and Kleene starJuly 16, 2012 MTS - Controllable Test Objects 9
  • Composition Isn’t Enough Why Repeat? Why Select?• Iterate over collection • Respond to environment• Iterate over generation configuration/conditions strategy • Skip• Performance test over – Depends on earlier interval – Don’t halt long complex test run• Capacity test to limits – Count failures for reliability estimateJuly 16, 2012 MTS - Controllable Test Objects 10
  • Straight TrackJuly 16, 2012 MTS - Controllable Test Objects 11
  • Switching TracksJuly 16, 2012 MTS - Controllable Test Objects 12
  • MTS ARCHITECTURE AND FEATURESJuly 16, 2012 MTS - Controllable Test Objects 13
  • MTS: Any App, Any Platform Console Host Agent Host MTS Console MTS Test TEST RUN Agent REPORTS Agent Host MTS Remote Agent MTS Test Agent Client Under Test Client Host Under Test MTS Remote Agent MTS Remote Client Agent Under Test Host Under Test may be Server Client Host Under Test  Cell Phone Under Test  PDA  Desktop Server Host Under Test MTS Remote Agent  Server  Embedded Processor Server  Network Equipment Under Test  Access Point  Base Station Server Host Under TestJuly 16, 2012 MTS - Controllable Test Objects 14
  • MTS Approach• Test harness for distributed, end to end testing – xUnit: composable test objects – TTCN: adapters, concurrency, … – Tcl: defacto standard for test script automationJuly 16, 2012 MTS - Controllable Test Objects 15
  • About TTCN-3• Procedural DSL similar • Well-suited to model-based to CORBA IDL testing – Generate TTCN from MSCs, FSMs, etc.• Point of Observation – Generate impl source from TTCN and Control (adapters) – Compile & run impl• Modular, but not • Same test suite can be used necessarily composable on many implementationsJuly 16, 2012 MTS - Controllable Test Objects 16
  • Tcl• Open source iut.set {1} interpreter iut.set {$x}• LISP-ish evaluation iut.set {[genX{$y}]} iut.set {[eval $codeForX]}July 16, 2012 MTS - Controllable Test Objects 17
  • Tcl• Open source interpreter• LISP-ish evaluation• Object-oriented [incr Tcl]July 16, 2012 MTS - Controllable Test Objects 18
  • MTS::TestObject MTS::TestObject MTS Framework +testObjectHeadline +getTestObjectClassName() +getTestObjectHeadline() +getTestObjectFileName() Adapter +getTestObjectDir() +getRemoteHostStatus()+connect() +getTestRunRepeatCount() YourTestObject+open() +getTestRunIteration() #require() MyTestObject+close()+test() #setup()+record() #test() +require() PPC 2002 #cleanup() +setup() WM5 #ensure() +test() #check() +cleanup() +getImage() #run() +ensure() #submit() WebService #userLogEntry() #userLogImage() +getServiceWSDL() #evaluateCompareFrame() +parseWSDL() #replayTestActionSequence() +queryForDictionary() +queryForXML() July 16, 2012 MTS - Controllable Test Objects 19
  • TestObject methodsSkip rest of test object when its preconditions arent met. requireAutomatically called when a TestObject is run.Create starting conditions for a test; allocate necessary resources. setupRun tests. The users implementation is expected to execute test testactions leading to one or more test verdicts.Restore pretest conditions; release resources the user test object cleanuphas claimed.Skip or halt rest of a test run when postconditions arent met. ensureAutomatically called after cleanup completes.July 16, 2012 MTS - Controllable Test Objects 20
  • Using require ::itcl::class aTestCase{ inherit TestObject ::itcl::body aTestCase::require {} { if {[getRemoteHostStatus] eq "Connected"} { set remoteHostconnection true } else { userLogEntry "Remote Host not ready, skipping" set remoteHostconnection false } return $remoteHostconnection } ::itcl::body aTestCase::test {} { … }July 16, 2012 MTS - Controllable Test Objects 21
  • Completion Actionsreturn Current Test object Caller Test object Test objects run stack CounterskeywordOK No effect, Destruct No effect. No effect. No (ensure) change.SKIP Destruct Continue with the No effect. No next test object. change.HALT Destruct Immediately exit the Terminate the test run. No method that called Percolate the HALT to the change. this test object. run stack.EXCP Capture current Tcl Immediately exit the Terminate the test run. Increment errorInfo. Destruct the method that called Percolate the EXCP to all error current test object. this test object. test objects in the run counter. July 16, 2012 stack. MTS - Controllable Test Objects 22
  • Using ensure and Completion Actions ::itcl::class checkDir {inherit TestObject ::itcl::body checkDir::require {} { … } ::itcl::body checkDir::setup {} { … } ::itcl::body checkDir::test {} { … } ::itcl::body checkDir::cleanup {} { … } ::itcl::body checkDir::ensure { set currentDir [pwd] if { ! [regexp {mydir} $currentDir]} { return SKIP } else { return OK } }July 16, 2012 MTS - Controllable Test Objects 23
  • MTS run commandrun <target> repeat • Hardcoded target<times> <arg list> run myTestObject• “Sources in” target file • Target name in a variable TestObject subclass run $anyTestObject• Runs this TestObject• Returns control to the • Function computes target run [eval whichTestObject] callerJuly 16, 2012 MTS - Controllable Test Objects 24
  • Selection and Repetition in MTS How to do ( p | q ) How to do (t ( p | q ))*::itcl::body pq::test{} { ::itcl::body mySuite::test{reps} { if {[rhStatus]} { run strategy1 -repeat $reps run remoteTest } } else { run localTest ::itcl::body strategy1::test{} { } run t} run pq } July 16, 2012 MTS - Controllable Test Objects 25
  • TestObject helpersStart other test objects or create a test suite run <target>Evaluate and log pass/fail conditions check <expression>Time any action during a test run Timer <method>Object interface for test run as a whole testRun <method>Monitor remote host resource utilization rpm <method>Check the remote host getRHostStatus myHostPerform data-driven tests submit <args>Annotate test run reports userLogEntry <args> July 16, 2012 MTS - Controllable Test Objects 26
  • IMPLEMENTING MODAL CLASS TEST PATTERNJuly 16, 2012 MTS - Controllable Test Objects 27
  • Modal Class Test PatternTwoPlayerGame T wo Play erG am e ( ) αTwoPlayerGame p1 _S tart ( ) / p2 _S tart ( ) / ThreePlayerGame( ) /TwoPlayerGame( )+TwoPlayerGame() s im u lat eV olle y( ) G am e S tarte d s im u lat eV olle y( ) α+p1_Start( )+p1_WinsVolley( ) p1_Start( ) / p3_Start( )/-p1_AddPoint( ) p1 _W ins V olle y ( ) p2 _W ins V olle y ( ) simulateVolley( ) simulateVolley( ) [th is .p 1_ Sc ore ( ) < 20 ] / [th is .p 2_ Sc ore ( ) < 20 ] / Game Started+p1_IsWinner( ) th is .p 1A ddP oin t( ) th is .p 2A ddP oin t( )+p1_IsServer( ) s im u lat eV olle y( ) p1 _W ins V olle y ( ) / s im u lat eV olle y( )+p1_Points( ) s im u lat eV olle y( ) p2_Start( ) /+p2_Start( ) P la ye r 1 P la ye r 2 simulateVolley( )+p2_WinsVolley( ) S erv ed S erv ed p1_WinsVolley( ) /-p2_AddPoint( ) p2 _W ins V olle y ( ) / simulateVolley( ) s im u lat eV olle y( )+p2_IsWinner( ) p1 _W ins V olle y ( ) p2 _W ins V olle y ( )+p2_IsServer( ) [th is .p 1_ Sc ore ( ) = = 20] / [th is .p 2_ Sc ore ( ) = = 20] /+p2_Points( ) th is .p 1A ddP oin t( ) th is .p 1A ddP oin t( ) p2_WinsVolley( )+~( ) p1_WinsVolley( ) [this.p2_Score( ) < 20] / p3_WinsVolley( ) P la ye r 1 P la ye r 2 [this.p1_Score( ) < 20] / this.p2AddPoint( ) [this.p3_Score( ) < 20] / p1 _Is W in ner( ) / Won Won p2 _Is W in ner( ) / this.p1AddPoint( ) simulateVolley( ) this.p3AddPoint( ) retu rn T R UE ; retu rn T R UE ; simulateVolley( ) simulateVolley( ) ~( ) ~( ) p1_WinsVolley( ) / p2_WinsVolley( ) / ω simulateVolley( ) simulateVolley( ) Player 1 Player 2 Player 3 Served Served Served p2_WinsVolley( ) / p3_WinsVolley( ) / simulateVolley( ) simulateVolley( ) ThreePlayerGame p1_WinsVolley( ) p3_WinsVolley( ) [this.p1_Score( ) == 20] / [this.p3_Score( ) == 20] / T h ree P la y erG a m e ( ) / T wo P la y erG am e ( ) α this.p1AddPoint( ) this.p3AddPoint( ) p 3_ S tart ( ) / p3_WinsVolley( ) /ThreePlayerGame G a m e S ta rt ed s im ulat eV o lley ( ) simulateVolley( ) p 3_ W ins V o lle y( ) / p2_WinsVolley( )+ThreePlayerGame() s im ulat eV o lley ( ) [this.p2_Score( ) == 20] /+p3_Start( ) p 3_ W ins V o lle y( ) this.p1AddPoint( ) [t his .p 3_ S co re ( ) < 2 0] /+p3_WinsVolley( ) th is . p3 A dd P oint ( ) T w oP lay erG am e ( )-p3_AddPoint( ) s im ulat eV o lley ( ) p 1_ W ins V o lle y( ) /+p3_IsWinner( ) s im ulat eV o lley ( )+p3_IsServer( ) P la y er 3+p3_Points( ) S erv e d p1_IsWinner( ) / p2_IsWinner( ) / p3_IsWinner( ) /+~( ) p 2_ W ins V o lle y( ) / return TRUE; Player 1 return TRUE; Player 2 Player 3 return TRUE; s im ulat eV o lley ( ) Won Won Won p 3_ W ins V o lle y( ) [t his .p 3_ S co re ( ) = = 2 0] / th is . p3 A dd P oint ( ) ~( ) ~( ) ~( ) ω P la y er 3 W on p 3_ Is W in ne r( ) / ret urn T R UE ; ~( ) ω July 16, 2012 MTS - Controllable Test Objects 28
  • Modal Class Test Design 1 ThreePlayerGame( ) 2 p1_Start( ) 8 P layer 2 Served 3 p2_Start( ) 4 p3_Start( ) 11 P layer 3 Served 5 p1_WinsVolley( ) 6 p1_WinsVolley( )[this.p1_Score( ) < 20] P layer 1 Served 17 om ega *7 7 p1_WinsVolley( ) [this.p1_Score( ) == 20] P layer 1 W on 14 8 p2_WinsVolley( ) P layer 1 W on 9 p2_WinsVolley( ) [this.p2_Score( ) < 20] *6 P layer 1 Served 10 p2_WinsVolley( ) [this.p2_Score( ) == 20] 2 Sneak path tests *9 P layer 2 Served not shown 11 P layer 3 Served 1 3 alpha Gam eS tarted P layer 2 Served 17 om ega * 10 P layer 2 W on 15 P layer 2 W on Note iterations 5 P layer 1 Served 4 * 12 P layer 3 Served 17 om ega 11 p3_WinsVolley( ) * 13 P layer 3 W on 16 12 p3_WinsVolley( ) [this.p3_Score( ) < 20] P layer 3 Served P layer 3 W on 13 p3_WinsVolley( ) [this.p3_Score( ) == 20] 8 P layer 2 Served 14 p1_IsWinner( ) 15 p2_IsWinner( ) 16 p3_IsWinner( ) 5 P layer 1 Served 17 ~( )July 16, 2012 MTS - Controllable Test Objects 29
  • Implementing Modal Class Design xUnit MTS• TestCase for each transition (20) • TestObject for each transition (20)• Hand hack TestSuite for each • Top level TestObject uses run to root to leaf path (17) and one to run them all call transition test objects – addTest – Iteration trival – addTestSuite • Parameterized strategies• How to iterate transitions? – N+• Embed strategy in single large test – Markov object? – Very smelly – Etc.• Develop code generator that writes • Parameterized state spec test suites? July 16, 2012 MTS - Controllable Test Objects 30
  • 1 ThreePlayerGame( )2 p1_Start( ) 8 P layer 2 Served3 p2_Start( )4 p3_Start( ) 11 P layer 3 Served5 p1_WinsVolley( )6 p1_WinsVolley( )[this.p1_Score( ) < 20] P layer 1 Served 17 om ega *77 p1_WinsVolley( ) [this.p1_Score( ) == 20] P layer 1 W on 14 Dude, where’s my object?8 p2_WinsVolley( ) P layer 1 W on9 p2_WinsVolley( ) [this.p2_Score( ) < 20] *6 P layer 1 Served10 p2_WinsVolley( ) [this.p2_Score( ) == 20] 2 *9 P layer 2 Served 11 P layer 3 Served 1 3 alpha Gam eS tarted P layer 2 Served 17 om ega * 10 P layer 2 W on 15 P layer 2 W on 5 P layer 1 Served 4 * 12 P layer 3 Served xUnit MTS 17 om ega11 p3_WinsVolley( ) * 13 P layer 3 W on 1612 p3_WinsVolley( ) [this.p3_Score( ) < 20] P layer 3 Served P layer 3 W on13 p3_WinsVolley( ) [this.p3_Score( ) == 20] 8 P layer 2 Served14 p1_IsWinner( )15 p2_IsWinner( )16 p3_IsWinner( ) 5 P layer 1 Served17 ~( ) 1 8 15 1 6 11 1 8 15 2 9 16 2 7 12 2 9 16 3 10 17 8 3 10 17 3 13 11 21 4 11 18 4 18 4 9 14 5 12 19 5 12 19 5 10 15 6 13 20 6 13 20 16 7 14 7 14 TestCase objects TestSuite objects TestObject objects July 16, 2012 MTS - Controllable Test Objects 31
  • Controllable Test Objects Design Goal TTCN xUnit MTS Select, Iterate, any level Yes No Yes Generate with MBT strategies Yes Yes Yes Logical/Physical separation Yes No Yes Platform agnostic/robust Yes ? Yes Channel agnostic/robust Yes ? Yes Distributed control Yes No Yes Minimize Test Agent resources NA No Yes Minimize brittleness Yes No Yes Intuitive interaction No Yes Yes Composable No Yes YesJuly 16, 2012 MTS - Controllable Test Objects 32
  • MTS hard problem queue• Protocol to drive remote target host IUT from test host – Bi-directional, lightweight (BEEP ?) – Platform and stack agnostic – Target environment control (remote command line) – Distributed adapter RPC• Forced reflection to generate adapters• Logical/physical mapping in generative models• Scalable asynchronous remote agents – Control strategy – Provisioning – ProtocolJuly 16, 2012 MTS - Controllable Test Objects 33
  • Question Time Bob Binder rvbinder@gmail.com www.robertvbinder.comJuly 16, 2012 MTS - Controllable Test Objects 34