SlideShare a Scribd company logo
1 of 31
Download to read offline
OCAT: Object Capture based
        Automated Testing


                       Hojun Jaygarl, Carl K. Chang
                                       Iowa State University
                                          Sunghun Kim
        The Hong Kong University of Science and Technology
                                                 Tao Xie
                             North Carolina State University


                                        ISSTA 2010
Problem




          Generating object inputs is hard
     in Object-Oriented (OO) unit testing




2
Automated Test Generation (ATG)



    Automatically generate test inputs for a unit.


    Reduce manual efforts in OO unit testing.




3
Two Main Types of ATG Techniques
   Direct object construction
       Directly assign values to object fields of the object instance
       E.g., Korat [ISSTA'02]


   Method sequence generation
       Generate method sequences that can produce an object
        instance under construction
       E.g., Randoop [ICSE’07], Pex [TAP'08]
Results of the State of the Arts in ATG
       Pex
           Pex automatically generates test inputs.
           Dynamic Symbolic Execution
           21% of branch coverage [Thummalapenta et al. FSE '09] for
            QuickGraph, a C# graph library.
       Randoop
           Randoop automatically generates method sequences
           Random but feedback directed
           58% of branch coverage [Thummalapenta et al. FSE '09]
           45% according to our evaluation for Apache Common Collections
            3.2
    5
Case of not-covered branches
Cause of not-                   # of branches             Explanation
covered branches
Insufficient object             135 (46.3%)               unable to generate desirable object instances
                                                          required to cover certain branches.
String comparison               61 (20.9%)                Difficult to randomly find a desirable string to satisfy
                                                          such constraints, since the input space of string type
                                                          values is huge.

Container object access 39 (13.4%)                        Not easy to create a certain size of a container with
                                                          necessary elements.

Array comparison                25 (8.6%)                 Not easy to create a certain size of an array with
                                                          necessary elements.

Exception branches              18 (6.1%)                 These branches have a particularity of exception
                                                          handling code that handles run-time errors

Environmental setting           9 (3.1%)
                                                                    Almost 90%
                                                          Hard to get environment variables and file-system
                                                          structure.

Non-deterministic               4 (1.3%)                  Hard to handle multi-threading and user interactions
branch


 6              Run Randoop for three projects Apache Commons, XML Security, and JSAP
                 We randomly selected 10 source code files from each subject and investigated the causes of uncovered branches.
Not-Covered Branch Example




                             Checks the
                             validity of
                               “doc”




7
Limitations of Current Approaches
       Korat
           Require manual efforts for writing class invariants and value
            domains.
       Pex
           Pex uses built-in simple heuristics for generating fixed
            sequences, which are often ineffective.
       Randoop
           Random approach cannot generate relevant sequences that
            produce desirable object instances.


    8
Idea
       Practical approach (high coverage)


       Reflect real usage


       Easier process

                             Object capture based
                              Automated Testing

    9
OCAT Overview




10
Capturing Process

    Instrument the target program


    Capture objects
        From program execution
        From system test execution




    11
Instrumentation
    Byte code instrumentation




    12
Serialization
    Get a type and a concrete state of captured objects.


    Prune redundant objects.


    A concrete state representation for the state-equality
     checking [Xie et al. ASE’04]




    13
Serialized Object Example
    A list that has bank accounts




    14
Object Generation
    Feed captured objects to an existing automated method
     sequence generation technique.


                                 Method
          Object                             Sequences
                                Sequences                Execution
         Capturing   Captured   Generation
                      objects
                                             Generated
                                              objects




    15
Usage of Captured Objects
    Captured objects are de-serialized and used as test inputs.
    Evolved Objects

                Captured         foo(A)                    New
              instance of A     returns A             instance of A


        Indirect usage

              Captured            bar(A)                  New
            instance of A        returns B            instance of B




    16
Method Sequence with Captured Objects




17
Evaluation - Setup




18
Evaluation - Captured Objects
    Q1 How much can OCAT
     improve code coverage
     through captured object
     instances?


    19.0% improved from
     45.2%




    19
Evaluation- Captured Objects
    28.5% and 17.3% improved from 29.6% and 54.1%




    20
Object Mutation
Approach – Object Mutation (MTT)
AbstractReferenceMap




 22
Evaluation- Mutated Objects
    Q2 How much can mutated object instances further improve
     code coverage?




    23
Evaluation- Total
    25.5% improved on average, with maximum 32.7%
                             32.7%

22.9%


                                              20.9%




    24
Why is It a Feasible Idea?
   Reflect real usage


   Potential for being desirable inputs in
    achieving new branch coverage


   Capturing objects is easy


25
Discussion
    Object Capturing Process
        Problem: OCAT’s coverage depends on captured objects
        Capturing objects is an easy process (but we still need to
         capture “good-enough” objects).


    Captured Objects and Software Evolution
        Problem: software evolves and objects are changing.
        Captured object instances may be obsolete and not be valid
         anymore.

    26
Discussion
        Branches to Cover
         Problem: Still not-covered branches are more than 20%.
         Cross-system object capturing
             objects can be captured from system A and used for system B.
         Static analysis
             currently we use a simple static analysis.
         Iterative process
             two phases, object generation and object mutation, can be
              iteratively applied



    27
Threats to Validity
    Software under test might not be representative
        Our three subjects may yield better or worse OCAT
         coverage than that of other software projects.


    Our object capturing relies on the existing tests
        OCAT test coverage reported in this paper depends on
         the quality and quantity of existing tests.




    28
Conclusions
    Problem
         Hard to generate desirable object instances in OO unit
          testing.


        OCAT approach
         Capture objects from program execution.
         Generate more objects with method sequence generation.
         Mutate directly the captured objects to try to cover those
          not-yet-covered branches.


    29
Conclusions
    Results
        OCAT helps Randoop to achieve high branch coverage
         averagely 68.5%
        25.5% improved (with maximum 32.7%) from only 43.0%
         achieved by Randoop alone.
    Future work
        Enhance static analysis part
        Apply captured objects to other state-of-the-art techniques
         (e.g., parameterized unit testing, concolic testing, and
         model checking approaches).
     
    30
         Release the tool.
Thank you!




31

More Related Content

What's hot

STAR: Stack Trace based Automatic Crash Reproduction
STAR: Stack Trace based Automatic Crash ReproductionSTAR: Stack Trace based Automatic Crash Reproduction
STAR: Stack Trace based Automatic Crash ReproductionSung Kim
 
Software Defect Prediction on Unlabeled Datasets
Software Defect Prediction on Unlabeled DatasetsSoftware Defect Prediction on Unlabeled Datasets
Software Defect Prediction on Unlabeled DatasetsSung Kim
 
Mining Assumptions for Software Components using Machine Learning
Mining Assumptions for Software Components using Machine LearningMining Assumptions for Software Components using Machine Learning
Mining Assumptions for Software Components using Machine LearningLionel Briand
 
Testing Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal PerspectiveTesting Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal PerspectiveLionel Briand
 
Dissertation Defense
Dissertation DefenseDissertation Defense
Dissertation DefenseSung Kim
 
Applications of Machine Learning and Metaheuristic Search to Security Testing
Applications of Machine Learning and Metaheuristic Search to Security TestingApplications of Machine Learning and Metaheuristic Search to Security Testing
Applications of Machine Learning and Metaheuristic Search to Security TestingLionel Briand
 
Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)Sung Kim
 
System Testing of Timing Requirements based on Use Cases and Timed Automata
System Testing of Timing Requirements based on Use Cases and Timed AutomataSystem Testing of Timing Requirements based on Use Cases and Timed Automata
System Testing of Timing Requirements based on Use Cases and Timed AutomataLionel Briand
 
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...Lionel Briand
 
IRJET- Object Detection and Recognition using Single Shot Multi-Box Detector
IRJET- Object Detection and Recognition using Single Shot Multi-Box DetectorIRJET- Object Detection and Recognition using Single Shot Multi-Box Detector
IRJET- Object Detection and Recognition using Single Shot Multi-Box DetectorIRJET Journal
 
Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Lionel Briand
 
Joshua bloch effect java chapter 3
Joshua bloch effect java   chapter 3Joshua bloch effect java   chapter 3
Joshua bloch effect java chapter 3Kamal Mukkamala
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsLionel Briand
 
Automated and Scalable Solutions for Software Testing: The Essential Role of ...
Automated and Scalable Solutions for Software Testing: The Essential Role of ...Automated and Scalable Solutions for Software Testing: The Essential Role of ...
Automated and Scalable Solutions for Software Testing: The Essential Role of ...Lionel Briand
 
Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...
Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...
Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...Feng Zhang
 
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCLOCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCLLionel Briand
 
Keynote SBST 2014 - Search-Based Testing
Keynote SBST 2014 - Search-Based TestingKeynote SBST 2014 - Search-Based Testing
Keynote SBST 2014 - Search-Based TestingLionel Briand
 
Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: ...
Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: ...Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: ...
Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: ...Lionel Briand
 
A practical guide for using Statistical Tests to assess Randomized Algorithms...
A practical guide for using Statistical Tests to assess Randomized Algorithms...A practical guide for using Statistical Tests to assess Randomized Algorithms...
A practical guide for using Statistical Tests to assess Randomized Algorithms...Lionel Briand
 
Transfer defect learning
Transfer defect learningTransfer defect learning
Transfer defect learningSung Kim
 

What's hot (20)

STAR: Stack Trace based Automatic Crash Reproduction
STAR: Stack Trace based Automatic Crash ReproductionSTAR: Stack Trace based Automatic Crash Reproduction
STAR: Stack Trace based Automatic Crash Reproduction
 
Software Defect Prediction on Unlabeled Datasets
Software Defect Prediction on Unlabeled DatasetsSoftware Defect Prediction on Unlabeled Datasets
Software Defect Prediction on Unlabeled Datasets
 
Mining Assumptions for Software Components using Machine Learning
Mining Assumptions for Software Components using Machine LearningMining Assumptions for Software Components using Machine Learning
Mining Assumptions for Software Components using Machine Learning
 
Testing Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal PerspectiveTesting Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal Perspective
 
Dissertation Defense
Dissertation DefenseDissertation Defense
Dissertation Defense
 
Applications of Machine Learning and Metaheuristic Search to Security Testing
Applications of Machine Learning and Metaheuristic Search to Security TestingApplications of Machine Learning and Metaheuristic Search to Security Testing
Applications of Machine Learning and Metaheuristic Search to Security Testing
 
Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)
 
System Testing of Timing Requirements based on Use Cases and Timed Automata
System Testing of Timing Requirements based on Use Cases and Timed AutomataSystem Testing of Timing Requirements based on Use Cases and Timed Automata
System Testing of Timing Requirements based on Use Cases and Timed Automata
 
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...
 
IRJET- Object Detection and Recognition using Single Shot Multi-Box Detector
IRJET- Object Detection and Recognition using Single Shot Multi-Box DetectorIRJET- Object Detection and Recognition using Single Shot Multi-Box Detector
IRJET- Object Detection and Recognition using Single Shot Multi-Box Detector
 
Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...
 
Joshua bloch effect java chapter 3
Joshua bloch effect java   chapter 3Joshua bloch effect java   chapter 3
Joshua bloch effect java chapter 3
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance Systems
 
Automated and Scalable Solutions for Software Testing: The Essential Role of ...
Automated and Scalable Solutions for Software Testing: The Essential Role of ...Automated and Scalable Solutions for Software Testing: The Essential Role of ...
Automated and Scalable Solutions for Software Testing: The Essential Role of ...
 
Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...
Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...
Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...
 
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCLOCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
 
Keynote SBST 2014 - Search-Based Testing
Keynote SBST 2014 - Search-Based TestingKeynote SBST 2014 - Search-Based Testing
Keynote SBST 2014 - Search-Based Testing
 
Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: ...
Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: ...Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: ...
Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: ...
 
A practical guide for using Statistical Tests to assess Randomized Algorithms...
A practical guide for using Statistical Tests to assess Randomized Algorithms...A practical guide for using Statistical Tests to assess Randomized Algorithms...
A practical guide for using Statistical Tests to assess Randomized Algorithms...
 
Transfer defect learning
Transfer defect learningTransfer defect learning
Transfer defect learning
 

Viewers also liked

Crowd debugging (FSE 2015)
Crowd debugging (FSE 2015)Crowd debugging (FSE 2015)
Crowd debugging (FSE 2015)Sung Kim
 
A Survey on Automatic Software Evolution Techniques
A Survey on Automatic Software Evolution TechniquesA Survey on Automatic Software Evolution Techniques
A Survey on Automatic Software Evolution TechniquesSung Kim
 
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...Sung Kim
 
Random testing & prototyping
Random testing & prototypingRandom testing & prototyping
Random testing & prototypingVipul Rastogi
 
Tensor board
Tensor boardTensor board
Tensor boardSung Kim
 
Time series classification
Time series classificationTime series classification
Time series classificationSung Kim
 

Viewers also liked (6)

Crowd debugging (FSE 2015)
Crowd debugging (FSE 2015)Crowd debugging (FSE 2015)
Crowd debugging (FSE 2015)
 
A Survey on Automatic Software Evolution Techniques
A Survey on Automatic Software Evolution TechniquesA Survey on Automatic Software Evolution Techniques
A Survey on Automatic Software Evolution Techniques
 
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
 
Random testing & prototyping
Random testing & prototypingRandom testing & prototyping
Random testing & prototyping
 
Tensor board
Tensor boardTensor board
Tensor board
 
Time series classification
Time series classificationTime series classification
Time series classification
 

Similar to OCAT: Object Capture based Automated Testing (ISSTA 2010)

Anomaly Detection with Azure and .NET
Anomaly Detection with Azure and .NETAnomaly Detection with Azure and .NET
Anomaly Detection with Azure and .NETMarco Parenzan
 
Anomaly Detection with Azure and .net
Anomaly Detection with Azure and .netAnomaly Detection with Azure and .net
Anomaly Detection with Azure and .netMarco Parenzan
 
Introduction to Snabbkaffe
Introduction to SnabbkaffeIntroduction to Snabbkaffe
Introduction to SnabbkaffeEMQ
 
Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Ap...
Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Ap...Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Ap...
Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Ap...knowdiff
 
Testing survey by_directions
Testing survey by_directionsTesting survey by_directions
Testing survey by_directionsTao He
 
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceEnabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceLionel Briand
 
The Status of ML Algorithms for Structure-property Relationships Using Matb...
The Status of ML Algorithms for Structure-property Relationships Using Matb...The Status of ML Algorithms for Structure-property Relationships Using Matb...
The Status of ML Algorithms for Structure-property Relationships Using Matb...Anubhav Jain
 
Android Malware 2020 (CCCS-CIC-AndMal-2020)
Android Malware 2020 (CCCS-CIC-AndMal-2020)Android Malware 2020 (CCCS-CIC-AndMal-2020)
Android Malware 2020 (CCCS-CIC-AndMal-2020)Indraneel Dabhade
 
findbugs Bernhard Merkle
findbugs Bernhard Merklefindbugs Bernhard Merkle
findbugs Bernhard Merklebmerkle
 
Accelerated test case - Anish bhanu
Accelerated test case - Anish bhanuAccelerated test case - Anish bhanu
Accelerated test case - Anish bhanuRoopa Nadkarni
 
Product defect detection based on convolutional autoencoder and one-class cla...
Product defect detection based on convolutional autoencoder and one-class cla...Product defect detection based on convolutional autoencoder and one-class cla...
Product defect detection based on convolutional autoencoder and one-class cla...IAESIJAI
 
IEEE 2015 Java Projects
IEEE 2015 Java ProjectsIEEE 2015 Java Projects
IEEE 2015 Java ProjectsVijay Karan
 
IEEE 2015 Java Projects
IEEE 2015 Java ProjectsIEEE 2015 Java Projects
IEEE 2015 Java ProjectsVijay Karan
 
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
Performance Tuning -  Memory leaks, Thread deadlocks, JDK toolsPerformance Tuning -  Memory leaks, Thread deadlocks, JDK tools
Performance Tuning - Memory leaks, Thread deadlocks, JDK toolsHaribabu Nandyal Padmanaban
 
QTP/UFT Overview and Installation
QTP/UFT Overview and InstallationQTP/UFT Overview and Installation
QTP/UFT Overview and InstallationMurageppa-QA
 
URBAN OBJECT DETECTION IN UAV RESNETpptx
URBAN OBJECT DETECTION IN UAV RESNETpptxURBAN OBJECT DETECTION IN UAV RESNETpptx
URBAN OBJECT DETECTION IN UAV RESNETpptxbalajimankena
 
A framework and approaches to develop an in-house CAT with freeware and open ...
A framework and approaches to develop an in-house CAT with freeware and open ...A framework and approaches to develop an in-house CAT with freeware and open ...
A framework and approaches to develop an in-house CAT with freeware and open ...Tetsuo Kimura
 

Similar to OCAT: Object Capture based Automated Testing (ISSTA 2010) (20)

Icsm08a.ppt
Icsm08a.pptIcsm08a.ppt
Icsm08a.ppt
 
Anomaly Detection with Azure and .NET
Anomaly Detection with Azure and .NETAnomaly Detection with Azure and .NET
Anomaly Detection with Azure and .NET
 
Anomaly Detection with Azure and .net
Anomaly Detection with Azure and .netAnomaly Detection with Azure and .net
Anomaly Detection with Azure and .net
 
Introduction to Snabbkaffe
Introduction to SnabbkaffeIntroduction to Snabbkaffe
Introduction to Snabbkaffe
 
Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Ap...
Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Ap...Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Ap...
Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Ap...
 
Testing survey by_directions
Testing survey by_directionsTesting survey by_directions
Testing survey by_directions
 
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceEnabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial Intelligence
 
The Status of ML Algorithms for Structure-property Relationships Using Matb...
The Status of ML Algorithms for Structure-property Relationships Using Matb...The Status of ML Algorithms for Structure-property Relationships Using Matb...
The Status of ML Algorithms for Structure-property Relationships Using Matb...
 
Cots testing
Cots testingCots testing
Cots testing
 
Android Malware 2020 (CCCS-CIC-AndMal-2020)
Android Malware 2020 (CCCS-CIC-AndMal-2020)Android Malware 2020 (CCCS-CIC-AndMal-2020)
Android Malware 2020 (CCCS-CIC-AndMal-2020)
 
findbugs Bernhard Merkle
findbugs Bernhard Merklefindbugs Bernhard Merkle
findbugs Bernhard Merkle
 
Accelerated test case - Anish bhanu
Accelerated test case - Anish bhanuAccelerated test case - Anish bhanu
Accelerated test case - Anish bhanu
 
Product defect detection based on convolutional autoencoder and one-class cla...
Product defect detection based on convolutional autoencoder and one-class cla...Product defect detection based on convolutional autoencoder and one-class cla...
Product defect detection based on convolutional autoencoder and one-class cla...
 
IEEE 2015 Java Projects
IEEE 2015 Java ProjectsIEEE 2015 Java Projects
IEEE 2015 Java Projects
 
IEEE 2015 Java Projects
IEEE 2015 Java ProjectsIEEE 2015 Java Projects
IEEE 2015 Java Projects
 
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
Performance Tuning -  Memory leaks, Thread deadlocks, JDK toolsPerformance Tuning -  Memory leaks, Thread deadlocks, JDK tools
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
 
QTP/UFT Overview and Installation
QTP/UFT Overview and InstallationQTP/UFT Overview and Installation
QTP/UFT Overview and Installation
 
URBAN OBJECT DETECTION IN UAV RESNETpptx
URBAN OBJECT DETECTION IN UAV RESNETpptxURBAN OBJECT DETECTION IN UAV RESNETpptx
URBAN OBJECT DETECTION IN UAV RESNETpptx
 
A framework and approaches to develop an in-house CAT with freeware and open ...
A framework and approaches to develop an in-house CAT with freeware and open ...A framework and approaches to develop an in-house CAT with freeware and open ...
A framework and approaches to develop an in-house CAT with freeware and open ...
 
kanimozhi2019.pdf
kanimozhi2019.pdfkanimozhi2019.pdf
kanimozhi2019.pdf
 

More from Sung Kim

DeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
DeepAM: Migrate APIs with Multi-modal Sequence to Sequence LearningDeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
DeepAM: Migrate APIs with Multi-modal Sequence to Sequence LearningSung Kim
 
Deep API Learning (FSE 2016)
Deep API Learning (FSE 2016)Deep API Learning (FSE 2016)
Deep API Learning (FSE 2016)Sung Kim
 
Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)
Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)
Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)Sung Kim
 
Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)
Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)
Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)Sung Kim
 
How We Get There: A Context-Guided Search Strategy in Concolic Testing (FSE 2...
How We Get There: A Context-Guided Search Strategy in Concolic Testing (FSE 2...How We Get There: A Context-Guided Search Strategy in Concolic Testing (FSE 2...
How We Get There: A Context-Guided Search Strategy in Concolic Testing (FSE 2...Sung Kim
 
CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)
CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)
CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)Sung Kim
 
Source code comprehension on evolving software
Source code comprehension on evolving softwareSource code comprehension on evolving software
Source code comprehension on evolving softwareSung Kim
 
A Survey on Dynamic Symbolic Execution for Automatic Test Generation
A Survey on  Dynamic Symbolic Execution  for Automatic Test GenerationA Survey on  Dynamic Symbolic Execution  for Automatic Test Generation
A Survey on Dynamic Symbolic Execution for Automatic Test GenerationSung Kim
 
Survey on Software Defect Prediction
Survey on Software Defect PredictionSurvey on Software Defect Prediction
Survey on Software Defect PredictionSung Kim
 
MSR2014 opening
MSR2014 openingMSR2014 opening
MSR2014 openingSung Kim
 
Personalized Defect Prediction
Personalized Defect PredictionPersonalized Defect Prediction
Personalized Defect PredictionSung Kim
 
Automatic patch generation learned from human written patches
Automatic patch generation learned from human written patchesAutomatic patch generation learned from human written patches
Automatic patch generation learned from human written patchesSung Kim
 
The Anatomy of Developer Social Networks
The Anatomy of Developer Social NetworksThe Anatomy of Developer Social Networks
The Anatomy of Developer Social NetworksSung Kim
 
A Survey on Automatic Test Generation and Crash Reproduction
A Survey on Automatic Test Generation and Crash ReproductionA Survey on Automatic Test Generation and Crash Reproduction
A Survey on Automatic Test Generation and Crash ReproductionSung Kim
 
How Do Software Engineers Understand Code Changes? FSE 2012
How Do Software Engineers Understand Code Changes? FSE 2012How Do Software Engineers Understand Code Changes? FSE 2012
How Do Software Engineers Understand Code Changes? FSE 2012Sung Kim
 
Defect, defect, defect: PROMISE 2012 Keynote
Defect, defect, defect: PROMISE 2012 Keynote Defect, defect, defect: PROMISE 2012 Keynote
Defect, defect, defect: PROMISE 2012 Keynote Sung Kim
 
Predicting Recurring Crash Stacks (ASE 2012)
Predicting Recurring Crash Stacks (ASE 2012)Predicting Recurring Crash Stacks (ASE 2012)
Predicting Recurring Crash Stacks (ASE 2012)Sung Kim
 
Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzz...
Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzz...Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzz...
Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzz...Sung Kim
 
Software Development Meets the Wisdom of Crowds
Software Development Meets the Wisdom of CrowdsSoftware Development Meets the Wisdom of Crowds
Software Development Meets the Wisdom of CrowdsSung Kim
 
BugTriage with Bug Tossing Graphs (ESEC/FSE 2009)
BugTriage with Bug Tossing Graphs (ESEC/FSE 2009)BugTriage with Bug Tossing Graphs (ESEC/FSE 2009)
BugTriage with Bug Tossing Graphs (ESEC/FSE 2009)Sung Kim
 

More from Sung Kim (20)

DeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
DeepAM: Migrate APIs with Multi-modal Sequence to Sequence LearningDeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
DeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
 
Deep API Learning (FSE 2016)
Deep API Learning (FSE 2016)Deep API Learning (FSE 2016)
Deep API Learning (FSE 2016)
 
Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)
Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)
Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)
 
Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)
Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)
Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)
 
How We Get There: A Context-Guided Search Strategy in Concolic Testing (FSE 2...
How We Get There: A Context-Guided Search Strategy in Concolic Testing (FSE 2...How We Get There: A Context-Guided Search Strategy in Concolic Testing (FSE 2...
How We Get There: A Context-Guided Search Strategy in Concolic Testing (FSE 2...
 
CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)
CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)
CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)
 
Source code comprehension on evolving software
Source code comprehension on evolving softwareSource code comprehension on evolving software
Source code comprehension on evolving software
 
A Survey on Dynamic Symbolic Execution for Automatic Test Generation
A Survey on  Dynamic Symbolic Execution  for Automatic Test GenerationA Survey on  Dynamic Symbolic Execution  for Automatic Test Generation
A Survey on Dynamic Symbolic Execution for Automatic Test Generation
 
Survey on Software Defect Prediction
Survey on Software Defect PredictionSurvey on Software Defect Prediction
Survey on Software Defect Prediction
 
MSR2014 opening
MSR2014 openingMSR2014 opening
MSR2014 opening
 
Personalized Defect Prediction
Personalized Defect PredictionPersonalized Defect Prediction
Personalized Defect Prediction
 
Automatic patch generation learned from human written patches
Automatic patch generation learned from human written patchesAutomatic patch generation learned from human written patches
Automatic patch generation learned from human written patches
 
The Anatomy of Developer Social Networks
The Anatomy of Developer Social NetworksThe Anatomy of Developer Social Networks
The Anatomy of Developer Social Networks
 
A Survey on Automatic Test Generation and Crash Reproduction
A Survey on Automatic Test Generation and Crash ReproductionA Survey on Automatic Test Generation and Crash Reproduction
A Survey on Automatic Test Generation and Crash Reproduction
 
How Do Software Engineers Understand Code Changes? FSE 2012
How Do Software Engineers Understand Code Changes? FSE 2012How Do Software Engineers Understand Code Changes? FSE 2012
How Do Software Engineers Understand Code Changes? FSE 2012
 
Defect, defect, defect: PROMISE 2012 Keynote
Defect, defect, defect: PROMISE 2012 Keynote Defect, defect, defect: PROMISE 2012 Keynote
Defect, defect, defect: PROMISE 2012 Keynote
 
Predicting Recurring Crash Stacks (ASE 2012)
Predicting Recurring Crash Stacks (ASE 2012)Predicting Recurring Crash Stacks (ASE 2012)
Predicting Recurring Crash Stacks (ASE 2012)
 
Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzz...
Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzz...Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzz...
Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzz...
 
Software Development Meets the Wisdom of Crowds
Software Development Meets the Wisdom of CrowdsSoftware Development Meets the Wisdom of Crowds
Software Development Meets the Wisdom of Crowds
 
BugTriage with Bug Tossing Graphs (ESEC/FSE 2009)
BugTriage with Bug Tossing Graphs (ESEC/FSE 2009)BugTriage with Bug Tossing Graphs (ESEC/FSE 2009)
BugTriage with Bug Tossing Graphs (ESEC/FSE 2009)
 

Recently uploaded

The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 

Recently uploaded (20)

The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 

OCAT: Object Capture based Automated Testing (ISSTA 2010)

  • 1. OCAT: Object Capture based Automated Testing Hojun Jaygarl, Carl K. Chang Iowa State University Sunghun Kim The Hong Kong University of Science and Technology Tao Xie North Carolina State University ISSTA 2010
  • 2. Problem Generating object inputs is hard in Object-Oriented (OO) unit testing 2
  • 3. Automated Test Generation (ATG) Automatically generate test inputs for a unit. Reduce manual efforts in OO unit testing. 3
  • 4. Two Main Types of ATG Techniques  Direct object construction  Directly assign values to object fields of the object instance  E.g., Korat [ISSTA'02]  Method sequence generation  Generate method sequences that can produce an object instance under construction  E.g., Randoop [ICSE’07], Pex [TAP'08]
  • 5. Results of the State of the Arts in ATG  Pex  Pex automatically generates test inputs.  Dynamic Symbolic Execution  21% of branch coverage [Thummalapenta et al. FSE '09] for QuickGraph, a C# graph library.  Randoop  Randoop automatically generates method sequences  Random but feedback directed  58% of branch coverage [Thummalapenta et al. FSE '09]  45% according to our evaluation for Apache Common Collections 3.2 5
  • 6. Case of not-covered branches Cause of not- # of branches Explanation covered branches Insufficient object 135 (46.3%) unable to generate desirable object instances required to cover certain branches. String comparison 61 (20.9%) Difficult to randomly find a desirable string to satisfy such constraints, since the input space of string type values is huge. Container object access 39 (13.4%) Not easy to create a certain size of a container with necessary elements. Array comparison 25 (8.6%) Not easy to create a certain size of an array with necessary elements. Exception branches 18 (6.1%) These branches have a particularity of exception handling code that handles run-time errors Environmental setting 9 (3.1%) Almost 90% Hard to get environment variables and file-system structure. Non-deterministic 4 (1.3%) Hard to handle multi-threading and user interactions branch 6  Run Randoop for three projects Apache Commons, XML Security, and JSAP We randomly selected 10 source code files from each subject and investigated the causes of uncovered branches.
  • 7. Not-Covered Branch Example Checks the validity of “doc” 7
  • 8. Limitations of Current Approaches  Korat  Require manual efforts for writing class invariants and value domains.  Pex  Pex uses built-in simple heuristics for generating fixed sequences, which are often ineffective.  Randoop  Random approach cannot generate relevant sequences that produce desirable object instances. 8
  • 9. Idea  Practical approach (high coverage)  Reflect real usage  Easier process Object capture based Automated Testing 9
  • 11. Capturing Process  Instrument the target program  Capture objects  From program execution  From system test execution 11
  • 12. Instrumentation  Byte code instrumentation 12
  • 13. Serialization  Get a type and a concrete state of captured objects.  Prune redundant objects.  A concrete state representation for the state-equality checking [Xie et al. ASE’04] 13
  • 14. Serialized Object Example  A list that has bank accounts 14
  • 15. Object Generation  Feed captured objects to an existing automated method sequence generation technique. Method Object Sequences Sequences Execution Capturing Captured Generation objects Generated objects 15
  • 16. Usage of Captured Objects  Captured objects are de-serialized and used as test inputs.  Evolved Objects Captured foo(A) New instance of A returns A instance of A  Indirect usage Captured bar(A) New instance of A returns B instance of B 16
  • 17. Method Sequence with Captured Objects 17
  • 19. Evaluation - Captured Objects  Q1 How much can OCAT improve code coverage through captured object instances?  19.0% improved from 45.2% 19
  • 20. Evaluation- Captured Objects  28.5% and 17.3% improved from 29.6% and 54.1% 20
  • 22. Approach – Object Mutation (MTT) AbstractReferenceMap 22
  • 23. Evaluation- Mutated Objects  Q2 How much can mutated object instances further improve code coverage? 23
  • 24. Evaluation- Total  25.5% improved on average, with maximum 32.7% 32.7% 22.9% 20.9% 24
  • 25. Why is It a Feasible Idea?  Reflect real usage  Potential for being desirable inputs in achieving new branch coverage  Capturing objects is easy 25
  • 26. Discussion  Object Capturing Process  Problem: OCAT’s coverage depends on captured objects  Capturing objects is an easy process (but we still need to capture “good-enough” objects).  Captured Objects and Software Evolution  Problem: software evolves and objects are changing.  Captured object instances may be obsolete and not be valid anymore. 26
  • 27. Discussion  Branches to Cover  Problem: Still not-covered branches are more than 20%.  Cross-system object capturing  objects can be captured from system A and used for system B.  Static analysis  currently we use a simple static analysis.  Iterative process  two phases, object generation and object mutation, can be iteratively applied 27
  • 28. Threats to Validity  Software under test might not be representative  Our three subjects may yield better or worse OCAT coverage than that of other software projects.  Our object capturing relies on the existing tests  OCAT test coverage reported in this paper depends on the quality and quantity of existing tests. 28
  • 29. Conclusions  Problem  Hard to generate desirable object instances in OO unit testing.  OCAT approach  Capture objects from program execution.  Generate more objects with method sequence generation.  Mutate directly the captured objects to try to cover those not-yet-covered branches. 29
  • 30. Conclusions  Results  OCAT helps Randoop to achieve high branch coverage averagely 68.5%  25.5% improved (with maximum 32.7%) from only 43.0% achieved by Randoop alone.  Future work  Enhance static analysis part  Apply captured objects to other state-of-the-art techniques (e.g., parameterized unit testing, concolic testing, and model checking approaches).  30 Release the tool.

Editor's Notes

  1. Nice to meet you. I’m Hojun Jaygarl from Iowa State University. I’m gonna present the paper called OCAT, Object capture-based Automated Testing. Professor Sunghun Kim from HKUST, and Tao Xie from NCSU helped me to publish this paper and Professor Carl Chang is my advisor.
  2. As all people know in this room, generating objects is hard. Object-oriented program are commonly used in these days even in the firmware level. However, many automated test input generating algorithms have low structural coverage (e.g., less than 50% branch coverage) Why? because, search space for object instances is huge. For example, Assume there is target branch that needs an ArrayList which has 500 elements of BankAccount and check 499 th elements of it. It’s very hard to generate it.
  3. As a preliminary experiment, we’ve investigated what’s the main causes of not-covered branches from Randoop. We select 30 source files from 3 open-source projects and categorize the causes.
  4. This is the one example of not-covered branch with insufficient object inputs. As you seen in this source, this is a constructor of Algorithm class and it needs an object input of Document class. a constructor of super class actually checks the validity of Document class, thus without having the Document class, we cannot generate Algorithm instance also, therefore, we cannot test any method in Algorithm class.
  5. Current approaches to generate object input has some limitations. Korat needs manual efforts for writing class invariants and value domains. Pex uses .. Randoop cannot generate..
  6. This slide shows an overview idea of OCAT. First of all, we capture objects from program execution such as system tests, regression tests, normal or planned program execution. After capturing objects, we seed these objects into a method sequence generation technique. By generating method sequences with captured object, more objects can be generated. Since these inputs reflect real usage, capturing these inputs and exploiting them in automated testing provide great potential for being desirable in achieving new branch coverage. After using method sequence generation, we checks not-covered branches. We are doing this because method sequence generation approach might quickly cover most branches, but cannot touch some complex branches. So, we do static analysis of not-covered branches. If we can get a value that needs for the not-covered branches, we directly mutated captured objects based on the SMT solver’s answer. All these captured and generated and mutated objects can be used and test inputs and achieve high code coverage.
  7. Let’s talk about capturing process. First, we need to instrument the target program. And then, by running the target program we can capture objects.
  8. This is an example of captured program. Actually the instrumentation is done in the byte code level of Java. We instrument the code that call our capturing library into entry points of method calls. So now by executing the program that uses this Algorithm class, we can capture Algorithm class as a receiver, and Document instance and string algorithmURI.
  9. If the capture method is called, the method gets a type and a concrete state of an object to be stored. and we keep types and states of all stated objects to maintain an object repository with redundant objects eliminated. We use a concrete state representation for the state-equality checking.
  10. This is an example to show how actually object instances are saved. Assume we captured an array list of bank accounts, that will be saved as an xml file. BankAccount has three fields and we can see the values of fields in the XML.
  11. OCAT generates the more objects by feeding captured objects to an existing automated method sequence generation technique. The captured objects are evolved to achieve high coverage by constructing and executing method sequences.
  12. When captured objects are used for method sequence generation, it uses in two ways. It can be directly used as you see in the first diagram, and it can be used indirectly as you see in the second diagram. If the directly used one generate more objects, it can be used to test other methods.
  13. This is the example of a method sequence that generated with captured objects. Assume there is a target method that needs a input of Map container. Method sequence generator produces method invocations of loading captured objects. TreeMap, String, BeanMap and integer. By generating method sequences, we produce more object instance by evolving capture objects.
  14. Instead of having toy projects for evaluation, we use open-source projects. Apache Commons Collections (ACC) : is an extended collection library that provides new utilities, such as buffer, queue, and map. Apache XML Security (AXS) : is a library implementing the XML Digital Signature Specification and XML Encryption Specification. JSAP : is a command-line argument parser.
  15. Now I’m going to present the first result of our approach. So, how much can OCAT improve code coverage through captured object instance with method sequence generation? The first line is our approach and blue line is Randoop and the dotted line shows a coverage of our execution for capturing objects. We measured based on the number of tests has been generated by both our approach and randoop. As a result, 19% has been improved from 45.2% which Randoop achived.
  16. You can see the result of XML security is significant because this open source project needs lots of legitimate XML inputs. Even if we have strong string constrain solver, it’s difficult to generate inputs like XML data. These results suggest that captured object instances plays an important role to improve code coverage by assisting an existing testing technique. We observe that the captured object instances are influenced by the original set of tests provided with the subject system. However, the OCAT’s achieved coverage is much higher than the coverage achieve by simply executing these original tests because captured object instances can lead a random generation technique to cover more branches.
  17. Captured and generated objects may not be able to cover all branches. CAPTIG mutates captured objects to try to satisfy constraints on conditionals related to not-yet-covered branches. SMT: Satifiability Modulo Theory
  18. This is the example of mutation. In the AbstractReferenceMap class, there is one branch that has not been covered, since ATG cannot generate the instance of AbstractRefereceMap. We already know the true path of this branch has not been covered, so we extract precondition of it by static analysis. We put the extracted precondition into SMT solver, and get a value. Because we have captured objects, we simply change the object instance with the value we got and check whether the new mutated object instance satisfies the branch condition.
  19. OCAT (mutated objects) can still improve approximately 4% of code coverage. This might not be significant since we are using very simple static analysis technique. We still can see the improvement which is not trivial.
  20. Overall, 22.9% of branch coverage improved from for ACC 32.7% “” for AXS and 20.9% for JSAP. On average 25.5% has been increased
  21. The captured object instances reflect real usage of object instances. Capturing these object instances and exploiting them provide great potential for being desirable in achieving new branch coverage Capturing objects is easier than writing a specification or an initial test case.
  22. graphic
  23. graphic
  24. Testing object-oriented (OO) software is critical OO languages have been commonly used. Many automated test input generating algorithms low structural coverage (e.g., less than 50% branch coverage) Challenges search space for object instances is huge. near half of difficult-to-cover branches require desirable objects. The main difference between unit tests of procedural languages (such as C) and object-oriented languages (such as C++ and Java) is that in procedural languages, a tester only needs to prepare input arguments for tested functions plus global variables, but in object-oriented languages, a tester needs to prepare both re- ceiver and argument object instances.