The document proposes a method to scale out automated test cases across multiple machines without modifying test case code. It involves adding a synchronization attribute class that is called by test case command classes to decouple synchronization logic from test code. The attribute class queries a central server to ensure critical resources are acquired sequentially across test runs. When tested on up to four machines, response times increased only for four machines likely due to reaching server capacity, showing the approach can scale tests effectively.

1. Scale out the test cases
X-BU Partner Integration Engineering
Tao Zhou
Tao.Zhou@emc.com
Abstract:
Thispaperwill introduce asimple waytoscale outtest caseson manymachines.The wayto scale out
testcases will notrequire code changestoexistingtestcodes,whichusing reflectiontechnologyto
decouple the synchronizationcode andtestcasescode.
1. Introduction
For automationtesting,there will havemore thanthousandsof testcases.Inregressionphase, there is
alwayspainto handle tremendouscasesinshorttime:if all testcasesrun sequentiallyinseparated
environments,thatwill takesmore time tocomplete testingthanexpected,butif casesare parallel
running,thatwill require extraeffortto synchronize the critical resource requests betweenthe test
cases. Is there an easywayto scale out these testcasestoparallel executionwithoutresource
confliction?The paperwilldescribehowtoscale outyour testcasesin an easyway.
2. The synchronization architecture
How to synchronize the resource request inautomationtestcases? Forthe testcasesare runningon
differentmachines,soitisintuitional tohave aserver(Figure 1) to synchronize the requests.
Figure 1

2. The serverside implementationcouldsimple andstraightforward,butthe difficultpartisthe client
code,for the followingreasons:
 The existedtestcasesare designedwithoutconsiderthe parallel executioncases.
 The testingenvironmentmaychange frequently.
 Addingthe synchronize code intestcasesneedsextra effectforQA.
 There will heterogeneoustestcases,forexample,testcase A will require resource 1, 2, 3 in
differentphases,buttestcase B will only requireresource 2inspecified.SowhenA isusing
resource 1 and 3 the case B coulduse resource 2.
Is there a panaceato serve all testcases? Yes,the paperwill introduce the methodwhichwill easily
transferthe single-processtestcasestomulti-processestestcases.
3. Client side software design
As previous sectionmentioned,there are many issues fordesignthe synchronize code forclientside,in
the section,there isa simple scenariotodemonstrate how toresolve theseissues.
The picture (Figure 2) showsthe simplifiedcode whichcontainsatestcase of creatinga Lun. When
several instance of the case runsimultaneously, the storage will getmore thanone requestforcreating
Lun at a moment,then the instances will be failed.
Figure 2
Andthe followingpicture (Figure 3) showsthe synchronizedcode addedtothe testcase,now when
several instances are runningsimultaneously,butonstorage side,the requestforcreating Lunisone by
one,the orderof requestingissynchronizedbyserver, sothese instanceswillbe success.

3. Figure 3
As showingthe picture above,thereisno extraworkfor testcase code, onlywork is justadd an
attribute tothe New-EmcLunclass. The attribute isadecoupledwithtestcases,andeasilybe
implemented.
So the attribute classperfectlyresolvesall the issuesmentionedinprevioussection, butwhythe
attribute classhasthe magicpower?
The picture below (Figure 4) isa typicallytestcasesstructure,the testcasescall the commandclasses,
and commandclasses finallycall all kindsof CLI viaadapter class to execute the commandintestcases.
Figure 4
What’sthe attribute classdois markthe critical resource forcommandclasses.Whenthe command
classcallsthe adapterclassto execute CLIcommand,thenthe adapterclassqueries the attributesof

4. commandclass then getthe critical resource ID inrunningtime environmentfromattributevalues,then
queryto servertomake sure the critical resource isnotusingby othertestcases.
Figure 5
The magic powerof the designisthat the critical resource ID isgetfrom runningtime environment,only
thingthe testcases designerneedtoknow isjustsettingthe attribute of commandclass. For the same
commandclass,whenthe environmentisdifferent,there will generate differentcritical environmentID.
Andthe test casescode is decoupled withsynchronization code,there will getsynchronizedtestcases
withoutchange anytestcases code.
4. Server side software design
Serverside software designisstraightforwardandsimple,only one concernisresource deadlock.For
example if clientA requestresource 1and2, meanwhile clientBrequest2 and 3, clientCrequest3 and
1, if resource isassignedseparately,the deadlockscenariowillhappen.The simplewaytoresolve the
issue isto setatomiclockfor each resource request,onlyall requestedresourceare available,the
resource will be acquired.

5. Figure 6
5. Application
To get more strictlytestingresultof the scale-outabilityof the design.There are choosingthe most
resource consumingcasesandall casesrequestssame resources. There definestwoparametersto
measure the scale-outability:
The over headrate: 𝑅 𝑜ℎ =
𝑇𝑛
𝑇1
; 𝑇𝑛 is whentestcasesrunningonN machines,the sumof time forN
machinescostfor testcases, 𝑇1 is the time withonlyone machine,the time of runningall testcases. The
rate isnear the value 1 meansthe systemisa scalable system.
The response time rate: 𝑅 𝑟𝑒𝑠 =
𝑅𝑒𝑠 𝑛
𝑅𝑒𝑠1
; 𝑅𝑒𝑠 𝑛 isthe average response time of server (the resource [Like
storage,Host] whichtestcasesare using) whentestcasesare runningonN machine, 𝑅𝑒𝑠1 isthe
average response time whentestcasesare runningon1 machine.
Form the picture below,the 𝑅 𝑜ℎ value isnear1 whenN =3 and 2, whichmeansthe overheadisverylow
if testcases are runningon 3 machines; the 𝑅 𝑟𝑒𝑠 turnhighwhenN=4, whichmeanswhen4clientsare
requestingthe service onserver,the serverperformance isdegradedsharply,thatmaycausedby
reachingthe maximumof servercapability.Sothe 𝑅 𝑜ℎ increase onN=4 iscausesby 𝑅 𝑟𝑒𝑠 increasing. So
the scalabilityof the testisdependsonthe servercapability.

6. Figure 7
6. Summary
Thispaperintroducedaneasyimplementationof synchronization.Withthe implementation,the test
casescouldeasilyscale out on manymachines.
0
0.5
1
1.5
2
2.5
3
1 2 3 4
Roh
Rres
Numof Machine