Context: Testing is essential to improve the correctness of software systems. Metamorphic testing (MT) is an approach especially suited when the system under test lacks oracles, or they are expensive to compute. However, building an MT environment for a particular domain (e.g., cloud simulation, model transformation, machine learning) requires substantial effort. Objective: Our goal is to facilitate the construction of MT environments for specific domains. Method: We propose a model-driven engineering approach to automate the construction of MT environments. Starting from a meta-model capturing the domain concepts, and a description of the domain execution environment, our approach produces an MT environment featuring comprehensive support for the MT process. This includes the definition of domain-specific metamorphic relations, their evaluation, detailed reporting of the testing results, and the automated search-based generation of follow-up test cases. Results: Our method is supported by an extensible platform for Eclipse, called Gotten. We demonstrate its effectiveness by creating an MT environment for simulation-based testing of data centres and comparing with existing tools; its suitability to conduct MT processes by replicating previous experiments; and its generality by building another MT environment for video streaming APIs. Conclusion: Gotten is the first platform targeted at reducing the development effort of domain-specific MT environments. The environments created with Gotten facilitate the specification of metamorphic relations, their evaluation, and the generation of new test cases.

1. www.uam.es
Automated engineering
of domain-specific
metamorphic testing environments
Pablo Gómez-Abajo1, Pablo C. Cañizares1, Alberto Núñez2,
Esther Guerra1 and Juan de Lara1
1Universidad Autónoma de Madrid, Madrid, Spain
2Universidad Complutense de Madrid, Madrid, Spain

2. www.uam.es
Publication
• "Automated engineering of domain-specific metamorphic testing
environments". Pablo Gómez-Abajo, Pablo C. Cañizares, Alberto Núñez,
Esther Guerra, Juan de Lara. In Information and Software Technology
(Elsevier). Volume 157, May 2023, 107164. JCR 3.9 - Q2 - Position:
29/108 - Area: Computer Science / Software Engineering. 2022.
1/21

3. www.uam.es
Motivation
• Testing is essential to improve the correctness of software
systems
• Metamorphic testing (MT) is an approach suited when
• The system under test lacks oracles
• They are expensive to compute
• However
• Building an MT environment for a particular domain requires
substantial effort
• e.g., cloud simulation, model transformation, machine learning, etc.
2/21

4. www.uam.es
Metamorphic Testing
• MT uses metamorphic relations (MRs) to determine if the
execution of the test cases is correct
• MT studies the relations between test inputs and the resulting
outputs
• Calculating the sum of two numbers sum(a, b), we may define
the MR
• For a = 1, b = 6, we may generate the follow-ups
• c = 2, d = 5, x = 3
MRi If the result of two sums are equal, then sum(a, b) = sum(c, d) =>
MRo
The result of the same sums but increasing the first
operand by an arbitrary number x should be equal
sum(sum(a, x), b) = sum(sum(c, x), d)
• c = 6, d = 1, x = 0 • ...
3/21

5. www.uam.es
Proposal
• We propose an MDE approach to automate the construction of MT
environments
• Our approach starts from …
• A meta-model capturing the domain concepts
• A description of the domain execution environment
• … and produces an MT environment that supports the full MT process
• Definition and evaluation of domain-specific MRs
• Detailed reporting of the testing results
• Automated search-based generation of follow-up test cases
4/21

6. www.uam.es
Proposal
• We propose an MDE approach to automate the construction of MT
environments
• Our approach starts from …
• A meta-model capturing the domain concepts
• A description of the domain execution environment
• … and produces an MT environment that supports the full MT process
• Definition and evaluation of domain-specific MRs
• Detailed reporting of the testing results
• Automated search-based generation of follow-up test cases
Gotten
4/21

7. www.uam.es
Overview
Application
expert
Create
domain MMs
Define SuT(s)
execution
start
EMF
ext.
point
Domain
expert
Define MRs
Fine-tune
follow-up
generation
mrDSL
fowDSL
Create input
test cases
Generate
follow-ups
Metamorphic
testing
Define MRs
satisfactory
results?
end
Tester
[yes]
[no]
GOTTEN GOTTEN
5/21

8. www.uam.es
Running example: Cloud simulators
6/21

9. www.uam.es
1. metamodel datacentre "/sample.gotten/model/datac.ecore" with m1, m2
2. models "/sample.gotten/model/dcmodels"
3. metamodel workload "/sample.gotten/model/workload.ecore" with w1, w2
4. models "/sample.gotten/model/workloads"
5.
6. datacentre input Features {
7. context DataCentre def: CPU: Int = racks->collect
8. (numBoards*board.nodesPerBoard*board.nodeType.CPUCores*board.nodeType.CPUSpeed)->sum()
9. context DataCentre def: NMachines: Int = racks->collect
10. (numBoards*board.nodesPerBoard*board.nodeType.CPUCores)->sum()
11. ...
12. }
13. workload input Features {
14. context WorkloadSet def: Workload: Set = workloads->collect(Traces)
15. }
16. output Features {
17. Time : Long
18. Energy : Long
19. }
20. Processor {
21. Name: String
22. Version: String
23. }
24. MetamorphicRelations {
25. MR1 = [(( CPU(m1) > CPU(m2) ) and ((w1) == (w2)) ) implies ((Energy(m1) <= Energy(m2)))]
26. MR2 = [(( NMachines(m1) > NMachines(m2)) and ((w1) == (w2)) )
27. implies ((NMachines(m1)/NMachines(m2)) >= (Energy(m1)/Energy(m2)))]
28. ...
29. }
The mrDSL: Specification of MRs
7/21

10. www.uam.es
1. followups for datacentre using MR1
2. with source path = "/sample.gotten/model/dcmodels"
3. and output folder = "/sample.gotten/model/dcmodels"
4.
5. CPU -> decrease [1] NodeType.CPUCores keeping {NodeType.CPUCores > 0};
6. decrease [10] NodeType.CPUSpeed keeping {NodeType.CPUSpeed > 0}
7.
8. maximize ( CPU(m1) - CPU(m2) )
9.
10. maxSolutions 50
11. iterations 1
12. algorithms [Random, NSGAII, NSGAIII, eMOEA]
The fowDSL: Configuration of follow-ups
generation
8/21

11. www.uam.es
The Gotten architecture
GOTTEN
CORE
MOMOT
Processor
Dissect
CloudSim
Input test
models
Follow-up
test models
MOMOT
programs
HENSHIN
rules
mrDSL
spec.
fowDSL
configs.
MT
interactive
report
GOTTEN
1
2
4
3
5
6
8
CloudSim
processor
Dissect
processor
7 7
9/21

12. www.uam.es
2
3
1
4
5
6
7
The Gotten tool
10/21

13. www.uam.es
Evaluation
RQ1: Effectiveness of Gotten
Comparison with FwCloudMeT
RQ2: Suitability of generated MT tools
Perform a full MT process for the cloud simulation
domain and obtain similar results to an ad-hoc solution
RQ3: Case studies in other domains
Gotten for cloud simulators and video streaming APIs
11/21

14. www.uam.es
RQ1: Effectiveness of Gotten to
specify MT environments (1/3)
Subsystem
Dissect CloudSimStorage
FwCloudMeT Gotten FwCloudMeT Gotten
Engine 4341 574 4415 539
MRs 3174 10 3207 10
Follow-ups 2480 20 2513 20
Executor
(processor)
2254 564 2328 509
Total 12249 1168 12463 1078
LoC
12/21

15. www.uam.es
RQ1: Effectiveness of Gotten to
specify MT environments (1/3)
Subsystem
Dissect CloudSimStorage
FwCloudMeT Gotten FwCloudMeT Gotten
Engine 4341 574 4415 539
MRs 3174 10 3207 10
Follow-ups 2480 20 2513 20
Executor
(processor)
2254 564 2328 509
Total 12249 1168 12463 1078
• A new processor in Gotten required around 1100 LoC
• A new processor in FwCloudMeT required one order
of magnitude more code (around 12000 LoC)
13/21

16. www.uam.es
RQ2: Suitability of Gotten environments for MT (2/3)
MR1
MR1i
The cloud 𝑚1 has a better CPU system than 𝑚2. The workloads 𝜔1 and 𝜔2 are equal
𝑀𝑅1𝑖 = [ CPU(m1) > CPU(m2) and w1 == w2 ]
MR1o
The energy required to execute 𝜔1 over 𝑚1 should be less than or equal to the energy required to execute 𝜔2 over
𝑚2
𝑀𝑅1𝑜 = [ Energy(m1) <= Energy(m2) ]
MR2
MR2i
The cloud 𝑚1 contains more physical machines than the cloud 𝑚2. The workloads 𝜔1 and 𝜔2 are equal
𝑀𝑅2𝑖 = [ NMachines(m1) > NMachines(m2) and w1 == w2 ]
MR2o
The ratio between the number of machines of 𝑚1 and 𝑚2 should be greater than or equal to the ratio between the
energy consumption required to execute 𝜔1 over 𝑚1 and the one required to execute 𝜔2 over 𝑚2
𝑀𝑅2𝑜 = [ NMachines(m1) / NMachines(m2) >= Energy(m1) / Energy(m2) ]
MR3
MR3i
The cloud 𝑚1 has a better storage system than 𝑚2. The workloads 𝜔1 and 𝜔2 are equal
𝑀𝑅3𝑖 = [ Storage(m1) > Storage(m2) and w1 == w2 ]
MR3o
The time required to execute 𝜔1 over 𝑚1 should be less than or equal to the time required to execute 𝜔2 over 𝑚2
𝑀𝑅3𝑜 = [ Time(m1) <= Time(m2) ]
14/21

17. www.uam.es
RQ2: Suitability of Gotten environments for MT (2/3)
MR4
MR4i
The cloud 𝑚1 has a better network system than 𝑚2. The workloads 𝜔1 and 𝜔2 are equal
𝑀𝑅4𝑖 = [ Network(m1) > Network(m2) and w1 == w2 ]
MR4o
The time required to execute 𝜔1 over 𝑚1 should be less than or equal to the time required to execute 𝜔2 over 𝑚2
𝑀𝑅4𝑜 = [ Time(m1) <= Time(m2) ]
MR5
MR5i
The cloud 𝑚1 has a better memory system than 𝑚2. The workloads 𝜔1 and 𝜔2 are equal
𝑀𝑅5𝑖 = [ Memory(m1) > Memory(m2) and w1 == w2 ]
MR5o
The time required to execute 𝜔1 over 𝑚1 should be less than or equal to the time required to execute 𝜔2 over 𝑚2
𝑀𝑅5𝑜 = [ Time(m1) <= Time(m2) ]
MR6
MR6i
The clouds 𝑚1 and 𝑚2 are equal. The workload 𝜔1 contains 𝜔2
𝑀𝑅6𝑖 = [ m1 == m2 and Workload(w1)−>includes(Workload(w2)) ]
MR6o
The time required to execute 𝜔2 over 𝑚2 should be less than or equal to the time required to execute 𝜔1 over 𝑚1
𝑀𝑅6𝑜 = [ Time(m2) <= Time(m1) ]
15/21

18. www.uam.es
Environment Simulator MR1 MR2 MR3 MR4 MR5 MR6
Original results
CloudSim 100 40 100 100 0 100
Dissect 100 51 100 99 0 100
Gotten
CloudSim 100 45 100 100 0 100
Dissect 100 80 100 100 0 100
• Both tools agree that the two simulators satisfy MR1, MR3,
MR4, and MR6, but not MR5
• Regarding MR2, it differs because
• The nature of the test generation algorithms (search vs. heuristics)
• The relation between the ratios of the physical machines and energy
consumption is not met in the simulated environments as we expected
RQ2: Suitability of Gotten environments
for MT (2/3)
16/21

19. www.uam.es
Environment Simulator MR1 MR2 MR3 MR4 MR5 MR6
Original results
CloudSim 100 40 100 100 0 100
Dissect 100 51 100 99 0 100
Gotten
CloudSim 100 45 100 100 0 100
Dissect 100 80 100 100 0 100
• We could perform a full MT process for the cloud simulation
domain using a Gotten environment
• Our results are in line with those obtained with FwCloudMeT, a
specific tool for MT of cloud simulators
RQ2: Suitability of Gotten environments
for MT (2/3)
17/21

20. www.uam.es
• Gotten for Video Streaming APIs
RQ3: MT environments for different
domains (3/3)
18/21

21. www.uam.es
RQ3: MT environments for different
domains (3/3)
• We could create MT environments for different
domains using Gotten
• We could define an infrastructure enabling the reuse of
• The meta-model
• The test cases
• The catalogue of MRs for different SuTs
19/21

22. www.uam.es
• Gotten automates the construction of MT environments
for any domain
• Our approach offers
• Two DSLs to describe MRs
• Strategies for generating follow-ups
• An extension point to connect the MT environment with the
SuT
Conclusions
20/21

23. www.uam.es
• We will improve the follow-ups generation process to
support more powerful synthesis of strings
• e.g., using Wordnet
• We are currently extending the mrDSL language with
more primitives
• We also plan to use Gotten for MT in typical MDE
tasks
• Model transformation
• Code generation
Future work
21/21

24. www.uam.es
https://g0tten.github.io/gotten/

25. www.uam.es
Example: Video Streaming APIs

26. www.uam.es
1. metamodel videostream "/video/model/VideoStream.ecore" with m1, m2
2. models "/video/model/videotc"
3.
4. videostream input Features {
5. context VideoAPITest def: IsFullSearch: Boolean =
6. (request.oclIsTypeOf(SearchVideo)
7. and request.oclAsType(SearchVideo).maxResults = -1)
8. context SearchVideo def: SearchOrder: Int = orderType
9. context SearchVideo def: UntilYear: Int = until.year
10. context SearchVideo def: FromYear: Int = from.year
11. context SearchVideo def: Radius: Double = position.radius
12. }
13. output Features {
14. NVideos : Long
15. Results: Set
16. }
17. Processor {
18. Name: String
19. Version: String
20. }
21. MetamorphicRelations {
22. MR1 = [(IsFullSearch(m1) and (SearchOrder(m1) <> SearchOrder(m2)))
23. implies (NVideos(m1) == NVideos(m2))]
24. MR2 = [(IsFullSearch(m1) and (UntilYear(m1) < FromYear(m2)))
25. implies (Results(m1)->excludes(Results(m2)))]
26. MR3 = [(IsFullSearch(m1) and (Radius(m1) > Radius(m2)))
27. implies (Results(m1)->includes(Results(m2)))]
28. }
The mrDSL: Specification of MRs

27. www.uam.es
1. followups for videostream using MR1
2. with source path = "/video/model/videotc"
3. and output folder = "/video/model/videotc"
4.
5. SearchOrder -> require SearchVideo.orderType <> NONE
6.
7. maximize (SearchOrder(m2) - SearchOrder(m1))
8.
9. maxSolutions 3
10. iterations 1
11. algorithms [Random, NSGAII, NSGAIII, eMOEA]
The fowDSL: Configuration of follow-ups
generation

28. www.uam.es
The Gotten architecture
GOTTEN
CORE
MOMOT
Processor
Vimeo
Youtube
Input test
models
Follow-up
test models
MOMOT
programs
HENSHIN
rules
mrDSL
spec.
fowDSL
configs.
MT
interactive
report
GOTTEN
1
2
4
3
5
6
8
Youtube
API
Vimeo
API
7 7