The document proposes a novel approach called PaDMTP for path directed source test case generation and prioritization using metamorphic testing in Python. It aims to address limitations in traditional testing like incomplete coverage and lack of automation. The approach generates test cases using Python constraint solving and prioritizes them using path tracing. It was implemented on sample programs and evaluated against random and adaptive random testing using mutation analysis, showing improved fault detection effectiveness with PaDMTP.

2. A NOVEL APPROACH FOR
PATH DIRECTED SOURCE
TEST CASE GENERATION
AND PRIORITIZATION USING
PYTHON

3. OUTLINE
Outline
Conclusion
Introduction
Research
Methodology
Proposed
Solution
Implementation
Validation
01
02
03 04
05
06

4. INTRODUCTION

5. LIMITATIONS OF TRADITIONAL TESTING
PRACTICES
• It is difficult to develop test cases that account for all feasible
circumstances and inputs, which might result in problems being
undetected.
• Traditional testing techniques can take a long time, especially in big
and complicated projects, which could cause product releases to be
delayed.
• Because testers are fallible, manual testing might result in errors, the
failure to notice flaws, or inconsistent test findings.
• Manual testing is continually running the same test cases, which can
be tedious and increase the likelihood of errors.
• Unaware biases among testers might cause them to concentrate on
certain features or locations while potentially ignoring other

6. TEST ORACLE PROBLEM
• In software testing, a test oracle is a mechanism used to determine the
correctness of the test results.
• The test oracle problem arises when there is no clear and definitive way to
verify the correctness of the test output automatically.
• It is challenging to determine the expected output for certain test cases,
especially in complex systems or when the expected result is not well-
defined.
• Inadequate or inaccurate test oracles may lead to false positives (incorrectly
flagging valid results as defects) or false negatives (failing to detect actual
defects).

7. To overcome these issues,
Metamorphic Testing was
Introduced in 1998 (Tsong
Yueh 1998).
Tsong Yueh, C. S. C., Cheung; Yiu, Siu-Ming; (1998). Metamorphic Testing: A New Approach for Generating
Next Test Cases, The Hong Kong University of Science and Technology.

8. WHAT IS METAMORPHIC TESTING?
• Metamorphic testing is a property-based testing approach used to verify the
correctness of a program without requiring an explicit oracle.
• It is particularly useful when a test oracle is difficult to define or when the
expected output is not readily available.
• Instead of verifying individual test cases, metamorphic testing focuses on
the relationship between inputs and their corresponding outputs.
John-Mathews, J.-M. (2022). "How to test ML models? #4 Metamorphic testing." Retrieved July, 2023, from
https://www.giskard.ai/knowledge/how-to-test-ml-models-4-metamorphic-testing.

9. WHAT IS METAMORPHIC TESTING?
(CONTD.)
• Metamorphic testing is based on the concept of metamorphic relations,
which describe the expected relationships between inputs and outputs. They
are also called follow-up test cases.
• A metamorphic relation is a high-level specification of how outputs should
change when inputs are transformed or mutated.
• Test cases are created by applying input transformations or mutations to the
original input and then comparing the output with the transformed input.
• If the metamorphic relation holds, the output should satisfy the specified
relationship, even though the actual output may be different from the
expected output. Otherwise, it indicates the presence of defects in the
program or an issue with the test case.

10. IMPORTANCE OF METAMORPHIC TESTING
• It does not require a predefined expected output for each test case, making
it valuable when it is challenging or impractical to define oracles.
• It complements traditional testing techniques and can significantly increase
test coverage, helping to detect defects that might be missed by other
methods.
• It can be applied during various stages of development to identify issues
early, facilitating prompt rectification and minimizing downstream impact.

11. IMPORTANCE OF METAMORPHIC TESTING
(CONTD.)
• It is applicable to different types of software and domains, offering a
versatile testing approach for various projects.
• It can reveal subtle and hidden defects that may not be apparent in
individual test cases but become apparent through the application of
metamorphic relations.

12. CHALLENGES OF METAMORPHIC TESTING
• Generating Effective and Diverse Test Cases
• Designing appropriate input transformations to create meaningful test cases.
• Ensuring sufficient coverage of input space to reveal potential defects.
• Avoiding redundancy in test cases while maintaining diversity.
• Prioritizing Test Cases for Efficient Fault Detection
• Identifying critical metamorphic relations for prioritization.
• Balancing the trade-off between high-priority relations and comprehensive
coverage.
• Developing strategies to optimize the order of test case execution.

13. RESEARCH METHODOLOGY

14. OBJECTIVES FOR THIS RESEARCH STUDY
• Conduct an analysis of existing approaches and techniques in metamorphic
testing, identifying their limitations specifically in path-directed source test
case generation and prioritization.
• Propose a novel methodology, that integrates path-directed techniques with
metamorphic testing principles, aiming to enhance the generation and
prioritization of source test cases.
• Implement the methodology using the Python programming language,
leveraging its flexibility and widespread adoption in the software testing
community.

15. OBJECTIVES (CONTD.)
• Evaluate the effectiveness and efficiency of our approach through
experiments conducted on various software systems, comparing the results
with existing approaches.
• Provide comprehensive guidelines and recommendations for practitioners
and researchers on the application of the methodology in real-world
software testing scenarios.
• Contribute to the advancement of metamorphic testing techniques by
addressing the challenges associated with path-directed source test case
generation and prioritization.

16. RESEARCH QUESTIONS
RQ1. How does the integration of machine learning techniques, specifically
in Python, enhance the fault detection effectiveness of using the
generated source test cases?
RQ2. To what extent does this approach outperform existing source test
case generation techniques in terms of fault detection capabilities and
overall testing effectiveness?
RQ3. What is the computational overhead associated with the
implementation of this approach for generating source test cases, and
how does it impact the efficiency and scalability of the testing process?

17. SEARCH STRATEGY
• Source for Literature Search:
• Google Scholar
• DBLP
• Keywords Used During Search:
• Metamorphic Testing
• Software Testing

18. LITERATURE REVIEW
Title Year Tools & Techniques Area of Research
Evaluation of Metamorphic Testing
for Edge Detection in MRI Brain
Diagnostics
2022
MT through test case
generation technique
Healthcare,
Apply metamorphic
testing techniques on X-
rays, CT, and MRI
An innovative approach for testing
bioinformatics programs using
metamorphic testing
2018 MT through Ling Pipe Bioinformatics
Automated metamorphic testing on
the analyses of feature models
2011 Mutation testing Feature models
Testing Web Enabled Simulation at
Scale Using
2021 Regression testing
Facebook’s web enabled
simulation
Testing scientific software: A
systematic literature review
2014
MT through regression
testing
Testing scientific software
Automatic System Testing of
Programs without Test Oracles
2009
Amsterdam, Mutation
testing, SVM
To check metamorphic
relations on java codes
Metamorphic Testing on Nuclide
Inventory Tool
2020 Use of MT to guide MR
Whole project is divided
into three layers i.e.,
physics algo and code

19. PROPOSED SOLUTION

20. PROPOSED APPROACH
• PaDMTP = (Path Directed Source Test Case Generation and
Prioritization in Metamorphic Testing using Python)
• Our approach is divided into two parts:
• Source Test Case Generation
• Source Test Case Prioritization

21. SOURCE TEST CASE GENERATION
This part has the following steps:
1. Python Constraint Solving
• During this step, the Python Constraint module was used to generate a
diverse range of outputs that adhered to the specified constraints of the
problem being addressed.
2. Path Constraints
• The constraints established in the previous stage are compiled into a
thorough list (also known as Source Test Cases) in this phase, which
serves as the basis for further processing and improvement of our
methodology.

22. SOURCE TEST CASE PRIORITIZATION
This part has the following steps:
3. Python Path Tracing
• In this phase, we made use of the Python Trace module's capabilities for
line-level tracing. We were able to obtain a thorough route for a
particular test case by carefully capturing the sequence of executed
statements using line-level tracing.
4. Test Case Prioritization
• In this phase, we made use of the Python Trace module's capabilities for
line-level tracing. We have been able to obtain a thorough route for a
particular test case by carefully capturing the sequence of executed
statements using line-level tracing.

23. PROPOSED
SOLUTION
WORKFLOW

24. IMPLEMENTATION

25. TOOLS & TECHNIQUES
• Tool
• Jupyter Notebook
• Python Modules
• Python Constraint = For Constraint Solving
• Python Trace = For Path Tracing
• MutPy = For Mutation Testing

26. TOOL INTERFACE - MAIN

27. TOOL INTERFACE - PATH TRACING

28. TOOL INTERFACE - FOLLOW UP TEST CASES
(MRS)

29. TOOL RESULTS

30. VALIDATION

31. OBJECT PROGRAMS
• The proposed approach was implemented on the following
programs:
S.
No.
Program Name Description
1. MyCode
There are two arguments specified for this
function. Its main objective is to identify
the output by using these input values as
inputs in a series of calculations and
conditional operations.
2. HCF
The Python program that we used may be
used to get the Highest Common Factor
(HCF) of two input integers.
3. LCM
The Least Common Multiple (LCM) between
two specified numbers is the main purpose
of this Python function.
4. DIFF
In the Python programme under tests, the
positive difference between two provided
numbers is determined.

32. TESTING TECHNIQUES UTILIZED
Each of the object programs was subjected to the following
testing techniques:
• PaDMTP (Proposed Approach)
• Random Testing (RT)
• Adaptive Random Testing (ART)

33. TESTING TECHNIQUES UTILIZED (CONTD.)
• PaDMTP
• 4 Types of Metamorphic Relations (MRs) were created for each object
program. They are as follows
Metamorphic Relation Description Positive Condition
Additive
Add a positive
constant
Increase or Remain
Inclusive Add a new element Increase or Remain
Permutative Permutate a value Remain
Multiplicative Multiply by a constant Increase or Remain

34. TESTING TECHNIQUES UTILIZED (CONTD.)
• To validate / compare the results of all testing techniques, we
used Mutation Testing
• Python Module, MutPy, was used for this purpose

35. RESULTS – MUTATION TESTING - MYCODE
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Fault
Detection
Rate
k% Test Cases
MyCode
PaDMTP RT ART

36. RESULTS – MUTATION TESTING - HCF
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Fault
Detection
Rate
k Test Cases
HCF
PaDMTP RT ART

37. RESULTS – MUTATION TESTING - LCM
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Fault
Detection
Rate
k% Test Cases
LCM
PaDMTP RT ART

38. RESULTS – MUTATION TESTING - DIFF
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Fault
Detection
Rate
k Test Cases
Positive Difference
PaDMTP RT ART

39. RESULTS – PAIRWISE T-TEST – (PADMTP VS
RT)
Object Programs T-Stat P-Value Effect Size
MyCode 9.000 8.538 x 10 -06 42.426
HCF 0.042 9.677 x 10 -01 0.205
LCM 0.892 3.958 x 10 -01 4.639
DIFF 2.106 6.452 x 10 -02 5.346

40. RESULTS – PAIRWISE T-TEST – (PADMTP VS
ART)
Object Programs T-Stat P-Value Effect Size
MyCode 9.000 8.538 x 10 -06 11.767
HCF 0.795 4.472 x 10 -01 3.797
LCM 1.125 2.899 x 10 -01 5.282
DIFF 1.685 1.263 x 10 -01 7.639

41. CONCLUSION

42. CONCLUSION
• Metamorphic Testing (MT):
• It effectively addresses the oracle problem in software testing.
• It complements traditional methods by generating unique test cases.
• Research interest increased in source test case generation with metamorphic relations
(MRs).
• Our Novel Approach for Source Test Case Generation:
• Path-directed approach using Python.
• Leveraging Python path tracer and constraint solver.
• Obtaining program path constraints for extensive path coverage.
• Improving fault detection effectiveness.

43. CONCLUSION (CONTD.)
• Benefits of Our Approach:
• Comprehensive Path Coverage: Achieving extensive coverage of execution
paths.
• Enhanced Fault Detection: Significantly improving the effectiveness of fault
detection.
• Efficient Testing: Prioritization technique for test case diversity and testing
efficiency.
• Experimental Evaluations:
• Four representative programs used for evaluation.