Chakkrit (Kla) Tantithamthavorn, profile picture
Uploaded byChakkrit (Kla) Tantithamthavorn
3,663 views

AI-Driven Software Quality Assurance in the Age of DevOps

The document discusses the challenges and methodologies surrounding software quality assurance, particularly in the context of defect prediction models. It emphasizes the impact of automated parameter optimization on the accuracy and interpretation of these models, highlighting that incorrect application can lead to misleading insights and poor decisions. The research indicates that while some statistical techniques perform well, guidance specific to software engineering is essential for effective implementation.

Embed presentation

Downloaded 23 times
Dr. Chakkrit (Kla) Tantithamthavorn Lecturer, Faculty of Information Technology,
 Monash University, Australia. @klainfohttp://chakkrit.com chakkrit.tantithamthavorn@monash.edu AI-Driven Software Quality Assurance 
 in the Age of DevOps
Australian Taxation Office’s service outages from software bugs lead to $4.2 billion dollars lost 2
3 Software Quality Assurance practices like 
 code review and testing are still time-consuming - Large and complex: 1 billion lines of code - Intense quality assurance activities: - 17K code reviews - 100 million test cases
3 Software Quality Assurance practices like 
 code review and testing are still time-consuming - Large and complex: 1 billion lines of code - Intense quality assurance activities: - 17K code reviews - 100 million test cases It is infeasible to exhaustively review and test 
 source code within the limited time and resources
4 Defect models play a significant role in software quality management Release Defect
 Model Module A Module C Module B Module D Module A Module C Module B Module D
4 Predict
 future software 
 defects Defect models play a significant role in software quality management Release Defect
 Model Module A Module C Module B Module D Module A Module C Module B Module D
4 Predict
 future software 
 defects Explain 
 what makes software fail Defect models play a significant role in software quality management Release Defect
 Model Module A Module C Module B Module D Module A Module C Module B Module D
4 Predict
 future software 
 defects Explain 
 what makes software fail Defect models play a significant role in software quality management Release Defect
 Model Module A Module C Module B Module D Module A Module C Module B Module D Develop 
 empirical theories related to software quality
5 Defect models play a significant role in software quality management Lewis et al., ICSE’13 Mockus et al., BLTJ’00 Ostrand et al., TSE’05 Kim et al., FSE’15 Naggappan et al., ICSE’06 Zimmermann et al., FSE’09 Caglayan et al., ICSE’15 Tan et al., ICSE’15 Shimagaki et al., ICSE’16 Defect models become widespread in many large software organisations Explain 
 what makes software fail Develop 
 empirical theories related to software quality Predict
 future software 
 defects
6 Today research toolkits are easily accessible
7 Statistical
 Model Training
 Corpus Classifier 
 Parameters (7) Model
 Construction Performance
 Measures Data 
 Sampling (2) Data Cleaning and Filtration (3) Metrics Extraction and Normalization (4) Descriptive Analytics (+/-) Relationship to the Outcome Y X x Software
 Repository Software
 Dataset Clean
 Dataset Studied Dataset Outcome Studied Metrics Control Metrics +~ (1) Data Collection Predictive 
 Analytics Prescriptive Analytics (8) Model Validation (9) Model Analysis and Interpretation Importance 
 Score Testing
 Corpus PredictionsPerformance
 Estimates Patterns In reality, defect modelling is detailed and complex
8 A lack of practical guidelines of defect modelling have a negative impact on software quality management Misleading 
 insights Managers take wrong technical decisions Developers waste time and resources Wrong 
 predictions
8 A lack of practical guidelines of defect modelling have a negative impact on software quality management Misleading 
 insights Managers take wrong technical decisions Developers waste time and resources Wrong 
 predictions Empirical investigation is needed 
 to derive practical guidelines for defect modelling
9 Metric Selection Model Construction Model Evaluation Control Metrics
 [ICSE-SEIP’18] Correlation
 [TSE’16] Model Statistics
 [ICSE-SEIP’18] Model Interpretation Class Imbalance
 [Under Review] The Risks of Unsound
 Defect
 Models Data Quality Issue Reports
 [ICSE’15] Feature Selection
 [ICSME’18] Interpretation
 [Under Review] Model Validation
 [TSE’17] Measures
 [ICSE-SEIP’18] Releases
 [Under Review] Metrics
 [On-going] Universal Models
 [Under Review] Ranking 
 [On-going] Time-Wise
 [On-going] Parameters
 [ICSE’16,TSE’18]
9 Metric Selection Model Construction Model Evaluation Control Metrics
 [ICSE-SEIP’18] Correlation
 [TSE’16] Model Statistics
 [ICSE-SEIP’18] Model Interpretation Class Imbalance
 [Under Review] The Risks of Unsound
 Defect
 Models Data Quality Issue Reports
 [ICSE’15] Feature Selection
 [ICSME’18] Interpretation
 [Under Review] Model Validation
 [TSE’17] Measures
 [ICSE-SEIP’18] Releases
 [Under Review] Metrics
 [On-going] Universal Models
 [Under Review] Ranking 
 [On-going] Time-Wise
 [On-going] Parameters
 [ICSE’16,TSE’18]
10 Defect prediction models have configurable parameters that control their characteristics Most of the widely-used classification techniques require at least one parameter setting Based on the literature analysis of 
 300+ defect studies
10 Defect prediction models have configurable parameters that control their characteristics Most of the widely-used classification techniques require at least one parameter setting Based on the literature analysis of 
 300+ defect studies #trees for 
 random forest #clusters for 
 k-nearest neighbors
10 Defect prediction models have configurable parameters that control their characteristics Most of the widely-used classification techniques require at least one parameter setting Based on the literature analysis of 
 300+ defect studies #trees for 
 random forest #clusters for 
 k-nearest neighbors 80% of top-50 highly-cited defect studies rely on a default setting [IST’16]
10 Defect prediction models have configurable parameters that control their characteristics Most of the widely-used classification techniques require at least one parameter setting Based on the literature analysis of 
 300+ defect studies #trees for 
 random forest #clusters for 
 k-nearest neighbors 80% of top-50 highly-cited defect studies rely on a default setting [IST’16] Even within the R toolkit, 2 random forest packages have different default settings
What is the impact of optimization techniques on defect models? 11 Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32.
What is the impact of optimization techniques on defect models? 11 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32.
What is the impact of optimization techniques on defect models? 11 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32.
What is the impact of optimization techniques on defect models? 11 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? (RQ3) What are the best classification techniques for defect models when automated parameter optimization is applied? Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32.
12 A comprehensive framework to extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset
12 A comprehensive framework to extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting
12 A comprehensive framework to extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting Optimized
 Model Default-setting
 Model
12 A comprehensive framework to extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting Optimized
 Model Default-setting
 Model Calculate
 accuracy The accuracy of
 optimized model The accuracy of
 default model
12 A comprehensive framework to extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting Optimized
 Model Default-setting
 Model Calculate
 accuracy The accuracy of
 optimized model The accuracy of
 default model Rank metrics by importance score The ranking of
 metrics in 
 optimized model The ranking of
 metrics in 
 default model
12 A comprehensive framework to extensively evaluate the impact of optimization techniques on defect models 26 classification techniques 
 (e.g., C5.0, Random Forest) 12 performance measures 
 (e.g., AUC, F-measure) 4 optimization algorithms 
 (e.g., genetic algorithm, and differential evolution) Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting Optimized
 Model Default-setting
 Model Calculate
 accuracy The accuracy of
 optimized model The accuracy of
 default model Rank metrics by importance score The ranking of
 metrics in 
 optimized model The ranking of
 metrics in 
 default model
13 Case Study Setup Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Code Metrics
 (e.g., lines of code
 code complexity) Process Metrics
 (e.g., #commits
 #added_lines) Human Factor
 (e.g., #dev,
 ownership) 600 - 10,000 modules
 11-48% defective ratios 18 software releases 15-38 software metrics
Large Medium Small ● ● ● ● ● ● ● ● 0.0 0.1 0.2 0.3 0.4 C 5.0 AdaBoost AVN N etC ART PC AN N etN N etFDA M LPW eightD ecayM LP LM TG PLS LogitBoostKN N xG BTreeG BM N B R BF SVM R adial G AM Boos AUCPerformanceImprovement 14 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? Approach: Compute the accuracy difference between optimized models and default-setting models AUCDifference(Optimized-Default)
Large Medium Small ● ● ● ● ● ● ● ● 0.0 0.1 0.2 0.3 0.4 C 5.0 AdaBoost AVN N etC ART PC AN N etN N etFDA M LPW eightD ecayM LP LM TG PLS LogitBoostKN N xG BTreeG BM N B R BF SVM R adial G AM Boos AUCPerformanceImprovement 14 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? 9 of the 26 studied classification techniques have a large accuracy improvement Approach: Compute the accuracy difference between optimized models and default-setting models AUCDifference(Optimized-Default)
Large Medium Small ● ● ● ● ● ● ● ● 0.0 0.1 0.2 0.3 0.4 C 5.0 AdaBoost AVN N etC ART PC AN N etN N etFDA M LPW eightD ecayM LP LM TG PLS LogitBoostKN N xG BTreeG BM N B R BF SVM R adial G AM Boos AUCPerformanceImprovement 14 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? 9 of the 26 studied classification techniques have a large accuracy improvement Approach: Compute the accuracy difference between optimized models and default-setting models AUCDifference(Optimized-Default)
Large Medium Small ● ● ● ● ● ● ● ● 0.0 0.1 0.2 0.3 0.4 C 5.0 AdaBoost AVN N etC ART PC AN N etN N etFDA M LPW eightD ecayM LP LM TG PLS LogitBoostKN N xG BTreeG BM N B R BF SVM R adial G AM Boos AUCPerformanceImprovement 14 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? 9 of the 26 studied classification techniques have a large accuracy improvement Approach: Compute the accuracy difference between optimized models and default-setting models Optimization substantially improves the accuracy of defect prediction models AUCDifference(Optimized-Default)
15 (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Optimized
 Model Default
 Model Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank in default models
15 (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Optimized
 Model Default
 Model Calculate an importance score for each metric Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank in default models
15 (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Optimized
 Model Default
 Model Calculate an importance score for each metric Ranking of metrics
 in optimized models Ranking of metrics
 in default models Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank in default models
15 (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Optimized
 Model Default
 Model Calculate an importance score for each metric Ranking of metrics
 in optimized models Ranking of metrics
 in default models Calculate the rank difference for each metric Diff %metrics 0 60% -1 20% Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank in default models
16 (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank for default models 0% 20% 40% 60% 80% 100% 0 -1 -2 -3 -4 Percentageofmetrics Rank difference
16 (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank for default models 36% of the top-rank metrics in optimized models 
 do not appear at the top-rank for default models 0% 20% 40% 60% 80% 100% 0 -1 -2 -3 -4 Percentageofmetrics Rank difference
16 (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank for default models 36% of the top-rank metrics in optimized models 
 do not appear at the top-rank for default models 0% 20% 40% 60% 80% 100% 0 -1 -2 -3 -4 Percentageofmetrics One-third of the most important metrics are missing 
 if we do not apply parameter optimization Rank difference
17 (RQ3) What is the best classification techniques for defect models when automated parameter optimization is applied? Approach: Compute the average rank of classification techniques across the studied datasets A JMLR’14 paper finds that random forest is top-performing classifiers for machine learning datasets
17 (RQ3) What is the best classification techniques for defect models when automated parameter optimization is applied? Approach: Compute the average rank of classification techniques across the studied datasets A JMLR’14 paper finds that random forest is top-performing classifiers for machine learning datasets Optimized xGBTree Optimized C5.0 Optimized RF Surprisingly, the best technique like Random Forest in ML domains is not always be the best in SE domains
17 (RQ3) What is the best classification techniques for defect models when automated parameter optimization is applied? Approach: Compute the average rank of classification techniques across the studied datasets A JMLR’14 paper finds that random forest is top-performing classifiers for machine learning datasets Optimized xGBTree Optimized C5.0 Optimized RF Surprisingly, the best technique like Random Forest in ML domains is not always be the best in SE domains Domain-specific modelling guidelines are critically needed
What is the impact of optimization techniques on defect models? 18 Optimization substantially improves the accuracy of defect prediction models One-third of the most important metrics are missing if we do not apply parameter optimization The best technique (RF) in ML domain might not always be the best in SE domain when parameter optimization is applied Automated parameter optimization should be applied for defect prediction models
19 “This is a much needed contribution both for researchers and for practitioners: 
 
 Researchers will find a checklist for the quality assurance of their defect modelling methods. 
 
 Practitioners, that is software quality experts in companies, will avoid a false interpretation of their data.” 
 
 - An Anonymous Reviewer -
20 Open Challenges: 
 Faster dev. speed, but Slower QA activities Software companies are shifting 
 from long to rapid release cycles
20 Open Challenges: 
 Faster dev. speed, but Slower QA activities Software companies are shifting 
 from long to rapid release cycles Large volume of code changes
20 Open Challenges: 
 Faster dev. speed, but Slower QA activities Software companies are shifting 
 from long to rapid release cycles Large volume of code changes Slow CI builds and tests
20 Open Challenges: 
 Faster dev. speed, but Slower QA activities How to develop an AI agent to accurately and intelligently remove software defects prior to CI build runs? Software companies are shifting 
 from long to rapid release cycles Large volume of code changes Slow CI builds and tests
21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps
21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects
21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects Explain the Nature
 of Software Defects
21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects Explain the Nature
 of Software Defects Generate Actionable 
 Guidelines
21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects Explain the Nature
 of Software Defects Generate Actionable 
 Guidelines Bug Hunter
21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects Explain the Nature
 of Software Defects Generate Actionable 
 Guidelines Bug Hunter This research project is expected to reduce software defects and operating costs, while accelerating development productivity for Australian software industry
22 I'm actively recruiting Ph.D. students Jirayus Jiarpakdee Nov 2017 - current
 Publications: ICSME’18, ICSE’19,
 TSE’19, more to come … Benefits - Tuition Fees Scholarships - Full stipends ($27,353 per annum in 2018, indexed annually) - International Travel Funding - Work with Experts in AI/ML/DataMining/SE - Access to Monash HPC clusters - Possible Domestic and International Internships Current Ph.D. student:
23
23
23
23
23 Dr. Chakkrit (Kla) Tantithamthavorn @klainfohttp://chakkrit.com chakkrit.tantithamthavorn@monash.edu

More Related Content

PDF
Towards a Better Understanding of the Impact of Experimental Components on De...
byChakkrit (Kla) Tantithamthavorn
 
PDF
An Empirical Comparison of Model Validation Techniques for Defect Prediction ...
byChakkrit (Kla) Tantithamthavorn
 
PDF
Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...
byChakkrit (Kla) Tantithamthavorn
 
PDF
Software Analytics In Action: A Hands-on Tutorial on Mining, Analyzing, Model...
byChakkrit (Kla) Tantithamthavorn
 
PDF
Automated parameter optimization should be included in future 
defect predict...
byChakkrit (Kla) Tantithamthavorn
 
PDF
The Impact of Class Rebalancing Techniques on the Performance and Interpretat...
byChakkrit (Kla) Tantithamthavorn
 
PDF
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
byChakkrit (Kla) Tantithamthavorn
 
PDF
Mining Software Defects: Should We Consider Affected Releases?
byChakkrit (Kla) Tantithamthavorn
 
Towards a Better Understanding of the Impact of Experimental Components on De...
byChakkrit (Kla) Tantithamthavorn
 
An Empirical Comparison of Model Validation Techniques for Defect Prediction ...
byChakkrit (Kla) Tantithamthavorn
 
Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...
byChakkrit (Kla) Tantithamthavorn
 
Software Analytics In Action: A Hands-on Tutorial on Mining, Analyzing, Model...
byChakkrit (Kla) Tantithamthavorn
 
Automated parameter optimization should be included in future 
defect predict...
byChakkrit (Kla) Tantithamthavorn
 
The Impact of Class Rebalancing Techniques on the Performance and Interpretat...
byChakkrit (Kla) Tantithamthavorn
 
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
byChakkrit (Kla) Tantithamthavorn
 
Mining Software Defects: Should We Consider Affected Releases?
byChakkrit (Kla) Tantithamthavorn
 

What's hot

PDF
Leveraging HPC Resources to Improve the Experimental Design of Software Analy...
byChakkrit (Kla) Tantithamthavorn
 
PDF
The adoption of machine learning techniques for software defect prediction: A...
byRAKESH RANA
 
PPTX
Feature Selection Techniques for Software Fault Prediction (Summary)
bySungdoGu
 
DOC
Testing survey by_directions
byTao He
 
PPT
Experiments on Design Pattern Discovery
byTim Menzies
 
PDF
Software testing effort estimation with cobb douglas function a practical app...
byeSAT Publishing House
 
PPT
Promise 2011: "An Iterative Semi-supervised Approach to Software Fault Predic...
byCS, NcState
 
PDF
A Review on Parameter Estimation Techniques of Software Reliability Growth Mo...
byEditor IJCATR
 
PDF
Test Case Optimization and Redundancy Reduction Using GA and Neural Networks
byIJECEIAES
 
PDF
Dc35579583
byIJERA Editor
 
PDF
Software testing strategy
byijseajournal
 
PPTX
A software fault localization technique based on program mutations
byTao He
 
PDF
Software testing defect prediction model a practical approach
byeSAT Journals
 
PPT
Complexity Measures for Secure Service-Orieted Software Architectures
byTim Menzies
 
PPT
Using Developer Information as a Prediction Factor
byTim Menzies
 
PDF
A metrics suite for variable categorizationt to support program invariants[
byIJCSEA Journal
 
PPTX
StratCel: A Strategy-Centric Approach to the Design of End-User Debugging Tools
byhciresearch
 
PPT
Assessing the Reliability of a Human Estimator
byTim Menzies
 
DOC
Abstract.doc
bybutest
 
PDF
Effectiveness of test case
byijseajournal
 
Leveraging HPC Resources to Improve the Experimental Design of Software Analy...
byChakkrit (Kla) Tantithamthavorn
 
The adoption of machine learning techniques for software defect prediction: A...
byRAKESH RANA
 
Feature Selection Techniques for Software Fault Prediction (Summary)
bySungdoGu
 
Testing survey by_directions
byTao He
 
Experiments on Design Pattern Discovery
byTim Menzies
 
Software testing effort estimation with cobb douglas function a practical app...
byeSAT Publishing House
 
Promise 2011: "An Iterative Semi-supervised Approach to Software Fault Predic...
byCS, NcState
 
A Review on Parameter Estimation Techniques of Software Reliability Growth Mo...
byEditor IJCATR
 
Test Case Optimization and Redundancy Reduction Using GA and Neural Networks
byIJECEIAES
 
Dc35579583
byIJERA Editor
 
Software testing strategy
byijseajournal
 
A software fault localization technique based on program mutations
byTao He
 
Software testing defect prediction model a practical approach
byeSAT Journals
 
Complexity Measures for Secure Service-Orieted Software Architectures
byTim Menzies
 
Using Developer Information as a Prediction Factor
byTim Menzies
 
A metrics suite for variable categorizationt to support program invariants[
byIJCSEA Journal
 
StratCel: A Strategy-Centric Approach to the Design of End-User Debugging Tools
byhciresearch
 
Assessing the Reliability of a Human Estimator
byTim Menzies
 
Abstract.doc
bybutest
 
Effectiveness of test case
byijseajournal
 

Similar to AI-Driven Software Quality Assurance in the Age of DevOps

DOCX
A Novel Approach to Improve Software Defect Prediction Accuracy Using Machine...
byShakas Technologies
 
PDF
Practical Guidelines to Improve Defect Prediction Model – A Review
byinventionjournals
 
PDF
Software Defect Prediction Using Radial Basis and Probabilistic Neural Networks
byEditor IJCATR
 
PDF
Survey on Software Defect Prediction (PhD Qualifying Examination Presentation)
bylifove
 
PPTX
Survey on Software Defect Prediction
bySung Kim
 
PDF
Survey on Software Defect Prediction
bylifove
 
PDF
Defect Prediction: Accomplishments and Future Challenges
byYasutaka Kamei
 
PPTX
Software engineering practices and software quality empirical research results
byNikolai Avteniev
 
PDF
A Review on Software Fault Detection and Prevention Mechanism in Software Dev...
byiosrjce
 
PDF
F017652530
byIOSR Journals
 
PDF
Thesis Final Report
bySadia Sharmin
 
PDF
Software Defects and SW Reliability Assessment
byKristine Hejna
 
PDF
Reliable software in a continuous integration/continuous deployment (CI/CD) e...
byAnn Marie Neufelder
 
PPTX
Efficient Defect Management in Software Management
byGenQE
 
PDF
Predicting Software Defects Using Bayesian Network Approach
bytheijes
 
PDF
Software Defect Trend Forecasting In Open Source Projects using A Univariate ...
byCSCJournals
 
PDF
Intelligent Software Defect Prediction 1st Edition Xiaoyuan Jing
bybabinvaderx4
 
PPT
A Regression Analysis Approach for Building a Prediction Model for System Tes...
byMIMOS Berhad/Open University Malaysia/Universiti Teknologi Malaysia
 
PDF
Software testing.pdf
bySwagatGogoi3
 
PDF
the-top-ten-things-that-have-been-proven-to-effect-software-reliability-1.pdf
bymattcs901
 
A Novel Approach to Improve Software Defect Prediction Accuracy Using Machine...
byShakas Technologies
 
Practical Guidelines to Improve Defect Prediction Model – A Review
byinventionjournals
 
Software Defect Prediction Using Radial Basis and Probabilistic Neural Networks
byEditor IJCATR
 
Survey on Software Defect Prediction (PhD Qualifying Examination Presentation)
bylifove
 
Survey on Software Defect Prediction
bySung Kim
 
Survey on Software Defect Prediction
bylifove
 
Defect Prediction: Accomplishments and Future Challenges
byYasutaka Kamei
 
Software engineering practices and software quality empirical research results
byNikolai Avteniev
 
A Review on Software Fault Detection and Prevention Mechanism in Software Dev...
byiosrjce
 
F017652530
byIOSR Journals
 
Thesis Final Report
bySadia Sharmin
 
Software Defects and SW Reliability Assessment
byKristine Hejna
 
Reliable software in a continuous integration/continuous deployment (CI/CD) e...
byAnn Marie Neufelder
 
Efficient Defect Management in Software Management
byGenQE
 
Predicting Software Defects Using Bayesian Network Approach
bytheijes
 
Software Defect Trend Forecasting In Open Source Projects using A Univariate ...
byCSCJournals
 
Intelligent Software Defect Prediction 1st Edition Xiaoyuan Jing
bybabinvaderx4
 
A Regression Analysis Approach for Building a Prediction Model for System Tes...
byMIMOS Berhad/Open University Malaysia/Universiti Teknologi Malaysia
 
Software testing.pdf
bySwagatGogoi3
 
the-top-ten-things-that-have-been-proven-to-effect-software-reliability-1.pdf
bymattcs901
 

Recently uploaded

PDF
TPPmark2025 Kenta Inoue's answer 12/04/2025
byKenta Inoue
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PDF
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
PPTX
RISE with SAP for Automotive - S4 HANA Value.pptx
byAmira Jemi
 
PDF
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
PDF
Greetings All Students Update 3 by Mia Corp
by©LDMMIA, ©Reiki Yoga
 
PDF
eResource Scheduler Enterprise Resource Management and Scheduling Software.pdf
byeResource Scheduler
 
PDF
AI and Computer Architecture: 200 Years Together
byDmitry Zinoviev
 
PPTX
Basics of Identity Access Management In mordern Infrastructure
byPrinceXavier18
 
PDF
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
PDF
Guided Substation Engineering with CoMPAS: Simplifying IEC 61850
byDanBrown980551
 
PPTX
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
PDF
Safeguarding AI-Based Financial Infrastructure
byYasir Naveed Riaz
 
PDF
Session 1 - Solving Semi-Structured Documents with Document Understanding
byDianaGray10
 
DOCX
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
PPTX
Combining Induction and Transduction for Abstract Reasoning.pptx
byallen578468
 
PDF
Zero Trust & Defense-in-Depth: The Future of Critical Infrastructure Security
byYasir Naveed Riaz
 
PPTX
The Landscape of Computer Software Development Companies
byYash Singh
 
PDF
Knowing and Doing: Knowledge graphs, AI, and work
bymarainglezakis1
 
PDF
The Ultimate Guide to Problem Management Dashboards for IT Teams.pdf
byomnex systems
 
TPPmark2025 Kenta Inoue's answer 12/04/2025
byKenta Inoue
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
RISE with SAP for Automotive - S4 HANA Value.pptx
byAmira Jemi
 
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
Greetings All Students Update 3 by Mia Corp
by©LDMMIA, ©Reiki Yoga
 
eResource Scheduler Enterprise Resource Management and Scheduling Software.pdf
byeResource Scheduler
 
AI and Computer Architecture: 200 Years Together
byDmitry Zinoviev
 
Basics of Identity Access Management In mordern Infrastructure
byPrinceXavier18
 
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
Guided Substation Engineering with CoMPAS: Simplifying IEC 61850
byDanBrown980551
 
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
Safeguarding AI-Based Financial Infrastructure
byYasir Naveed Riaz
 
Session 1 - Solving Semi-Structured Documents with Document Understanding
byDianaGray10
 
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
Combining Induction and Transduction for Abstract Reasoning.pptx
byallen578468
 
Zero Trust & Defense-in-Depth: The Future of Critical Infrastructure Security
byYasir Naveed Riaz
 
The Landscape of Computer Software Development Companies
byYash Singh
 
Knowing and Doing: Knowledge graphs, AI, and work
bymarainglezakis1
 
The Ultimate Guide to Problem Management Dashboards for IT Teams.pdf
byomnex systems
 

AI-Driven Software Quality Assurance in the Age of DevOps

  • 1.
    Dr. Chakkrit (Kla)Tantithamthavorn Lecturer, Faculty of Information Technology,
 Monash University, Australia. @klainfohttp://chakkrit.com chakkrit.tantithamthavorn@monash.edu AI-Driven Software Quality Assurance 
 in the Age of DevOps
  • 2.
    Australian Taxation Office’sservice outages from software bugs lead to $4.2 billion dollars lost 2
  • 3.
    3 Software Quality Assurancepractices like 
 code review and testing are still time-consuming - Large and complex: 1 billion lines of code - Intense quality assurance activities: - 17K code reviews - 100 million test cases
  • 4.
    3 Software Quality Assurancepractices like 
 code review and testing are still time-consuming - Large and complex: 1 billion lines of code - Intense quality assurance activities: - 17K code reviews - 100 million test cases It is infeasible to exhaustively review and test 
 source code within the limited time and resources
  • 5.
    4 Defect models playa significant role in software quality management Release Defect
 Model Module A Module C Module B Module D Module A Module C Module B Module D
  • 6.
    4 Predict
 future software 
 defects Defectmodels play a significant role in software quality management Release Defect
 Model Module A Module C Module B Module D Module A Module C Module B Module D
  • 7.
    4 Predict
 future software 
 defects Explain
 what makes software fail Defect models play a significant role in software quality management Release Defect
 Model Module A Module C Module B Module D Module A Module C Module B Module D
  • 8.
    4 Predict
 future software 
 defects Explain
 what makes software fail Defect models play a significant role in software quality management Release Defect
 Model Module A Module C Module B Module D Module A Module C Module B Module D Develop 
 empirical theories related to software quality
  • 9.
    5 Defect models playa significant role in software quality management Lewis et al., ICSE’13 Mockus et al., BLTJ’00 Ostrand et al., TSE’05 Kim et al., FSE’15 Naggappan et al., ICSE’06 Zimmermann et al., FSE’09 Caglayan et al., ICSE’15 Tan et al., ICSE’15 Shimagaki et al., ICSE’16 Defect models become widespread in many large software organisations Explain 
 what makes software fail Develop 
 empirical theories related to software quality Predict
 future software 
 defects
  • 10.
    6 Today research toolkitsare easily accessible
  • 11.
    7 Statistical
 Model Training
 Corpus Classifier 
 Parameters (7) Model
 Construction Performance
 Measures Data
 Sampling (2) Data Cleaning and Filtration (3) Metrics Extraction and Normalization (4) Descriptive Analytics (+/-) Relationship to the Outcome Y X x Software
 Repository Software
 Dataset Clean
 Dataset Studied Dataset Outcome Studied Metrics Control Metrics +~ (1) Data Collection Predictive 
 Analytics Prescriptive Analytics (8) Model Validation (9) Model Analysis and Interpretation Importance 
 Score Testing
 Corpus PredictionsPerformance
 Estimates Patterns In reality, defect modelling is detailed and complex
  • 12.
    8 A lack ofpractical guidelines of defect modelling have a negative impact on software quality management Misleading 
 insights Managers take wrong technical decisions Developers waste time and resources Wrong 
 predictions
  • 13.
    8 A lack ofpractical guidelines of defect modelling have a negative impact on software quality management Misleading 
 insights Managers take wrong technical decisions Developers waste time and resources Wrong 
 predictions Empirical investigation is needed 
 to derive practical guidelines for defect modelling
  • 14.
    9 Metric Selection Model Construction ModelEvaluation Control Metrics
 [ICSE-SEIP’18] Correlation
 [TSE’16] Model Statistics
 [ICSE-SEIP’18] Model Interpretation Class Imbalance
 [Under Review] The Risks of Unsound
 Defect
 Models Data Quality Issue Reports
 [ICSE’15] Feature Selection
 [ICSME’18] Interpretation
 [Under Review] Model Validation
 [TSE’17] Measures
 [ICSE-SEIP’18] Releases
 [Under Review] Metrics
 [On-going] Universal Models
 [Under Review] Ranking 
 [On-going] Time-Wise
 [On-going] Parameters
 [ICSE’16,TSE’18]
  • 15.
    9 Metric Selection Model Construction ModelEvaluation Control Metrics
 [ICSE-SEIP’18] Correlation
 [TSE’16] Model Statistics
 [ICSE-SEIP’18] Model Interpretation Class Imbalance
 [Under Review] The Risks of Unsound
 Defect
 Models Data Quality Issue Reports
 [ICSE’15] Feature Selection
 [ICSME’18] Interpretation
 [Under Review] Model Validation
 [TSE’17] Measures
 [ICSE-SEIP’18] Releases
 [Under Review] Metrics
 [On-going] Universal Models
 [Under Review] Ranking 
 [On-going] Time-Wise
 [On-going] Parameters
 [ICSE’16,TSE’18]
  • 16.
    10 Defect prediction modelshave configurable parameters that control their characteristics Most of the widely-used classification techniques require at least one parameter setting Based on the literature analysis of 
 300+ defect studies
  • 17.
    10 Defect prediction modelshave configurable parameters that control their characteristics Most of the widely-used classification techniques require at least one parameter setting Based on the literature analysis of 
 300+ defect studies #trees for 
 random forest #clusters for 
 k-nearest neighbors
  • 18.
    10 Defect prediction modelshave configurable parameters that control their characteristics Most of the widely-used classification techniques require at least one parameter setting Based on the literature analysis of 
 300+ defect studies #trees for 
 random forest #clusters for 
 k-nearest neighbors 80% of top-50 highly-cited defect studies rely on a default setting [IST’16]
  • 19.
    10 Defect prediction modelshave configurable parameters that control their characteristics Most of the widely-used classification techniques require at least one parameter setting Based on the literature analysis of 
 300+ defect studies #trees for 
 random forest #clusters for 
 k-nearest neighbors 80% of top-50 highly-cited defect studies rely on a default setting [IST’16] Even within the R toolkit, 2 random forest packages have different default settings
  • 20.
    What is theimpact of optimization techniques on defect models? 11 Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32.
  • 21.
    What is theimpact of optimization techniques on defect models? 11 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32.
  • 22.
    What is theimpact of optimization techniques on defect models? 11 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32.
  • 23.
    What is theimpact of optimization techniques on defect models? 11 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? (RQ2) How much does the interpretation of defect models change when automated parameter optimization is applied? (RQ3) What are the best classification techniques for defect models when automated parameter optimization is applied? Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32.
  • 24.
    12 A comprehensive frameworkto extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset
  • 25.
    12 A comprehensive frameworkto extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting
  • 26.
    12 A comprehensive frameworkto extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting Optimized
 Model Default-setting
 Model
  • 27.
    12 A comprehensive frameworkto extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting Optimized
 Model Default-setting
 Model Calculate
 accuracy The accuracy of
 optimized model The accuracy of
 default model
  • 28.
    12 A comprehensive frameworkto extensively evaluate the impact of optimization techniques on defect models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting Optimized
 Model Default-setting
 Model Calculate
 accuracy The accuracy of
 optimized model The accuracy of
 default model Rank metrics by importance score The ranking of
 metrics in 
 optimized model The ranking of
 metrics in 
 default model
  • 29.
    12 A comprehensive frameworkto extensively evaluate the impact of optimization techniques on defect models 26 classification techniques 
 (e.g., C5.0, Random Forest) 12 performance measures 
 (e.g., AUC, F-measure) 4 optimization algorithms 
 (e.g., genetic algorithm, and differential evolution) Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Defect 
 Dataset Default 
 Setting Construct 
 Defect
 Models Optimized 
 Setting Optimized
 Model Default-setting
 Model Calculate
 accuracy The accuracy of
 optimized model The accuracy of
 default model Rank metrics by importance score The ranking of
 metrics in 
 optimized model The ranking of
 metrics in 
 default model
  • 30.
    13 Case Study Setup ChakkritTantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering (TSE) (2018), pp. 1-32. Code Metrics
 (e.g., lines of code
 code complexity) Process Metrics
 (e.g., #commits
 #added_lines) Human Factor
 (e.g., #dev,
 ownership) 600 - 10,000 modules
 11-48% defective ratios 18 software releases 15-38 software metrics
  • 31.
    Large Medium Small ● ● ● ● ● ●● ● 0.0 0.1 0.2 0.3 0.4 C 5.0 AdaBoost AVN N etC ART PC AN N etN N etFDA M LPW eightD ecayM LP LM TG PLS LogitBoostKN N xG BTreeG BM N B R BF SVM R adial G AM Boos AUCPerformanceImprovement 14 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? Approach: Compute the accuracy difference between optimized models and default-setting models AUCDifference(Optimized-Default)
  • 32.
    Large Medium Small ● ● ● ● ● ●● ● 0.0 0.1 0.2 0.3 0.4 C 5.0 AdaBoost AVN N etC ART PC AN N etN N etFDA M LPW eightD ecayM LP LM TG PLS LogitBoostKN N xG BTreeG BM N B R BF SVM R adial G AM Boos AUCPerformanceImprovement 14 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? 9 of the 26 studied classification techniques have a large accuracy improvement Approach: Compute the accuracy difference between optimized models and default-setting models AUCDifference(Optimized-Default)
  • 33.
    Large Medium Small ● ● ● ● ● ●● ● 0.0 0.1 0.2 0.3 0.4 C 5.0 AdaBoost AVN N etC ART PC AN N etN N etFDA M LPW eightD ecayM LP LM TG PLS LogitBoostKN N xG BTreeG BM N B R BF SVM R adial G AM Boos AUCPerformanceImprovement 14 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? 9 of the 26 studied classification techniques have a large accuracy improvement Approach: Compute the accuracy difference between optimized models and default-setting models AUCDifference(Optimized-Default)
  • 34.
    Large Medium Small ● ● ● ● ● ●● ● 0.0 0.1 0.2 0.3 0.4 C 5.0 AdaBoost AVN N etC ART PC AN N etN N etFDA M LPW eightD ecayM LP LM TG PLS LogitBoostKN N xG BTreeG BM N B R BF SVM R adial G AM Boos AUCPerformanceImprovement 14 (RQ1) What is the impact of automated parameter optimization on the accuracy of defect models? 9 of the 26 studied classification techniques have a large accuracy improvement Approach: Compute the accuracy difference between optimized models and default-setting models Optimization substantially improves the accuracy of defect prediction models AUCDifference(Optimized-Default)
  • 35.
    15 (RQ2) How muchdoes the interpretation of defect models change when automated parameter optimization is applied? Optimized
 Model Default
 Model Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank in default models
  • 36.
    15 (RQ2) How muchdoes the interpretation of defect models change when automated parameter optimization is applied? Optimized
 Model Default
 Model Calculate an importance score for each metric Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank in default models
  • 37.
    15 (RQ2) How muchdoes the interpretation of defect models change when automated parameter optimization is applied? Optimized
 Model Default
 Model Calculate an importance score for each metric Ranking of metrics
 in optimized models Ranking of metrics
 in default models Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank in default models
  • 38.
    15 (RQ2) How muchdoes the interpretation of defect models change when automated parameter optimization is applied? Optimized
 Model Default
 Model Calculate an importance score for each metric Ranking of metrics
 in optimized models Ranking of metrics
 in default models Calculate the rank difference for each metric Diff %metrics 0 60% -1 20% Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank in default models
  • 39.
    16 (RQ2) How muchdoes the interpretation of defect models change when automated parameter optimization is applied? Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank for default models 0% 20% 40% 60% 80% 100% 0 -1 -2 -3 -4 Percentageofmetrics Rank difference
  • 40.
    16 (RQ2) How muchdoes the interpretation of defect models change when automated parameter optimization is applied? Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank for default models 36% of the top-rank metrics in optimized models 
 do not appear at the top-rank for default models 0% 20% 40% 60% 80% 100% 0 -1 -2 -3 -4 Percentageofmetrics Rank difference
  • 41.
    16 (RQ2) How muchdoes the interpretation of defect models change when automated parameter optimization is applied? Approach: Compute the percentage of the top-rank metrics in optimized models that appears at the top-rank for default models 36% of the top-rank metrics in optimized models 
 do not appear at the top-rank for default models 0% 20% 40% 60% 80% 100% 0 -1 -2 -3 -4 Percentageofmetrics One-third of the most important metrics are missing 
 if we do not apply parameter optimization Rank difference
  • 42.
    17 (RQ3) What isthe best classification techniques for defect models when automated parameter optimization is applied? Approach: Compute the average rank of classification techniques across the studied datasets A JMLR’14 paper finds that random forest is top-performing classifiers for machine learning datasets
  • 43.
    17 (RQ3) What isthe best classification techniques for defect models when automated parameter optimization is applied? Approach: Compute the average rank of classification techniques across the studied datasets A JMLR’14 paper finds that random forest is top-performing classifiers for machine learning datasets Optimized xGBTree Optimized C5.0 Optimized RF Surprisingly, the best technique like Random Forest in ML domains is not always be the best in SE domains
  • 44.
    17 (RQ3) What isthe best classification techniques for defect models when automated parameter optimization is applied? Approach: Compute the average rank of classification techniques across the studied datasets A JMLR’14 paper finds that random forest is top-performing classifiers for machine learning datasets Optimized xGBTree Optimized C5.0 Optimized RF Surprisingly, the best technique like Random Forest in ML domains is not always be the best in SE domains Domain-specific modelling guidelines are critically needed
  • 45.
    What is theimpact of optimization techniques on defect models? 18 Optimization substantially improves the accuracy of defect prediction models One-third of the most important metrics are missing if we do not apply parameter optimization The best technique (RF) in ML domain might not always be the best in SE domain when parameter optimization is applied Automated parameter optimization should be applied for defect prediction models
  • 46.
    19 “This is amuch needed contribution both for researchers and for practitioners: 
 
 Researchers will find a checklist for the quality assurance of their defect modelling methods. 
 
 Practitioners, that is software quality experts in companies, will avoid a false interpretation of their data.” 
 
 - An Anonymous Reviewer -
  • 47.
    20 Open Challenges: 
 Fasterdev. speed, but Slower QA activities Software companies are shifting 
 from long to rapid release cycles
  • 48.
    20 Open Challenges: 
 Fasterdev. speed, but Slower QA activities Software companies are shifting 
 from long to rapid release cycles Large volume of code changes
  • 49.
    20 Open Challenges: 
 Fasterdev. speed, but Slower QA activities Software companies are shifting 
 from long to rapid release cycles Large volume of code changes Slow CI builds and tests
  • 50.
    20 Open Challenges: 
 Fasterdev. speed, but Slower QA activities How to develop an AI agent to accurately and intelligently remove software defects prior to CI build runs? Software companies are shifting 
 from long to rapid release cycles Large volume of code changes Slow CI builds and tests
  • 51.
    21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps
  • 52.
    21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects
  • 53.
    21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects Explain the Nature
 of Software Defects
  • 54.
    21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects Explain the Nature
 of Software Defects Generate Actionable 
 Guidelines
  • 55.
    21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects Explain the Nature
 of Software Defects Generate Actionable 
 Guidelines Bug Hunter
  • 56.
    21 Future Research Agenda: AI-Driven Software Quality Assurance in the Age of DevOps Accurate Predictions
 of Future Defects Explain the Nature
 of Software Defects Generate Actionable 
 Guidelines Bug Hunter This research project is expected to reduce software defects and operating costs, while accelerating development productivity for Australian software industry
  • 57.
    22 I'm actively recruitingPh.D. students Jirayus Jiarpakdee Nov 2017 - current
 Publications: ICSME’18, ICSE’19,
 TSE’19, more to come … Benefits - Tuition Fees Scholarships - Full stipends ($27,353 per annum in 2018, indexed annually) - International Travel Funding - Work with Experts in AI/ML/DataMining/SE - Access to Monash HPC clusters - Possible Domestic and International Internships Current Ph.D. student:
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
    23 Dr. Chakkrit (Kla)Tantithamthavorn @klainfohttp://chakkrit.com chakkrit.tantithamthavorn@monash.edu