Recommended
More Related Content
Similar to Test Recommendation using Machine Learning - GHC 2020
Similar to Test Recommendation using Machine Learning - GHC 2020 (20)
Recently uploaded
Recently uploaded (20)
Test Recommendation using Machine Learning - GHC 2020
- 2. Test Recommendation using Machine Learning BY SADIYA HAMEED (SALESFORCE)
- 3. Problem Monolithic codebase Million+ tests Hard for developers to know tests impacted by their changes CI system’s increasing time to feedback and resource consumption Growing number of tests and growing rate of incoming changes
- 4. Idea Prioritized list of relevant tests for every changelist
- 6. Mix of Signals FTW! Research Paper
- 8. Features Code Coverage Text Similarity Features Based on Historical Data Failure History Recommender File to Test Recommender User to Test Recommender
- 9. CodeCoverage Java (JaCoCo/Clover), Javascript (Instanbul) More lines covered -> higher score Using line numbers for better precision Figure: Code Coverage of a Test for a File
- 10. Features Code Coverage Text Similarity Features Based on Historical Data Failure History Recommender File to Test Recommender User to Test Recommender
- 11. TextSimilarity Path text similarity Content text similarity Use Abstract Syntax Tree (AST) NLP Pre-processing N-gram Stop Word Removal Case Conversion Stemming Search using Lucene based Elasticsearch Figure 1: Abstract Syntax Tree (Source: Wikipedia License: CC) Figure 2: NLP Pre-processing of a test name
- 12. Features Code Coverage Text Similarity Features Based on Historical Data Failure History Recommender File to Test Recommender User to Test Recommender
- 13. Failure History Recommender Change agnostic Feature Areas of code base currently under active development Frequency and Recency of Test Failure Negatively impacted by test non-determinism Figure: Test Recency and Decay Score
- 14. File to Test Recommender File to test score based on CI history Collaborative Filtering on files Higher weights for directly related tests, frequently failing tests, and those related to more file in the changelist Negatively impacted by test non-determinism Figure 1: File to test relationship Figure 2: File to file relationship
- 15. User to Test Recommender Change agnostic feature User to test score based on CI history Collaborative Filtering on users Users tend to not break the same test Users moved around through internal developer mobility Figure: User to Test Scores
- 16. TheModel Simple Linear Regression since all features provide recommendations Weights determined by training on historical data from CI system Regression model instead of classification model
- 17. Model Validation No clear test to change relevance data Validate against failures assigned to a change by our CI system
- 18. Recall What we missed Recall = True Positives True Positives + False Negatives Where, True Positives: tests we recommended for a change and they failed on that change False Negatives: tests we did not recommend for a change but they failed on that change True Positives + False Negatives: Total failing tests for a change
- 19. Precision What was recommended but didn’t fail Precision = True Positives True Positives + False Positives Where, True Positives: tests we recommended for a change and they failed on that change False Positives: tests we recommended for a change that did not fail on that change True Positives + False Positives: Total recommended tests for a change
- 20. For every change with failures On average, we catch 60% - 80% failures By running top 2000 recommended tests Out of the approx. million supported tests Results
- 21. Current Uses Local testing during development Automated remote testing at code review stage Testing in CI System
- 22. Conclusion
- 23. References Test Prioritizer Blog Post Test Prioritizer Research Paper