Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Learning for Testing Prioritization- an industrial case study Slide 1 Learning for Testing Prioritization- an industrial case study Slide 2 Learning for Testing Prioritization- an industrial case study Slide 3 Learning for Testing Prioritization- an industrial case study Slide 4 Learning for Testing Prioritization- an industrial case study Slide 5 Learning for Testing Prioritization- an industrial case study Slide 6 Learning for Testing Prioritization- an industrial case study Slide 7 Learning for Testing Prioritization- an industrial case study Slide 8 Learning for Testing Prioritization- an industrial case study Slide 9 Learning for Testing Prioritization- an industrial case study Slide 10 Learning for Testing Prioritization- an industrial case study Slide 11 Learning for Testing Prioritization- an industrial case study Slide 12 Learning for Testing Prioritization- an industrial case study Slide 13
Upcoming SlideShare
Summit2013 eventos onto quad
Next

1 Like

Share

Learning for Testing Prioritization- an industrial case study

Talk given by Ben Busjaeger, Software Engineering Architect at Salesforce, at FSE 2016: ACM SIGSOFT on November 2016

Modern cloud-software providers, such as Salesforce.com, increasingly adopt large-scale continuous integration environments. In such environments, assuring high developer productivity is strongly dependent on conducting testing efficiently and effectively. Specifically, to shorten feedback cycles, test prioritization is popularly used as an optimization mechanism for ranking tests to run by their likelihood of revealing failures. To apply test prioritization in industrial environments, we present a novel approach (tailored for practical applicability) that integrates multiple existing techniques via a systematic framework of machine learning to rank. Our initial empirical evaluation on a large realworld dataset from Salesforce shows that our approach significantly outperforms existing individual techniques.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Learning for Testing Prioritization- an industrial case study

  1. 1. Learning for Test Prioritization An Industrial Case Study Benjamin Busjaeger Salesforce Tao Xie University of Illinois, Urbana-Champaign ACM SIGSOFT FSE 2016
  2. 2. Large Scale Continuous Integration 2K+ Modules 600K+ Tests 2K+ Engineers 500+ Changes/Day Main repository ... 350K+ Source files
  3. 3. Motivation Before Commit After Commit Objective Reject faulty changes Detect test failures as quickly as possible Current Run hand-picked tests Parallelize (8000 concurrent VMs) & batch (1-1000 changes per run) Problem Too complex for human Feedback time constrained by capacity & batching complicates fault assignment Desired Select top-k tests likely to fail Prioritize all tests by likelihood of failure
  4. 4. Insight: Need Multiple Existing Techniques ● Heterogeneous languages: Java, PL/SQL, JavaScript, Apex, etc. ●Non-code artifacts: metadata and configuration ●New/recently-failed tests more likely to fail Test code coverage of change Textual similarity between test and change Test age and recent failures
  5. 5. Insight: Need Scalable Techniques ●Complex integration tests: concurrent execution ●Large data volume: 500+ changes, 50M+ test runs, 20TB+ results per day → Our approach: Periodically collect coarse-grained measurements Test code coverage of change Textual similarity between test and change Test age and recent failures
  6. 6. Formal model for learning from past test results New Approach: Learning for Test Prioritization Test code coverage of change Textual similarity between test and change Test age and recent failures Change Test Ranking → Implementation currently in pilot use at Salesforce
  7. 7. Empirical Study: Setup ●Test results of 45K tests for ~3 month period ●In this period, 711 changes with ≥1 failure • 440 for training • 271 for testing ●Collected once for each test: • Test code coverage • Test text content ●Collected continuously: • Test age and recent failures
  8. 8. ● New approach achieves highest average recall at all cutoff points • 50% failures detected with top 0.2% • 75% failures detected with top 3% Results: Test selection (before commit) New Approach
  9. 9. ● New approach achieves highest APFD with least variance • Median: 85% • Average: 99% Results: Test prioritization (after commit) New Approach
  10. 10. Results: Side Benefits Invalid Assignments Flaky Tests
  11. 11. Summary ● Main insights gained in conducting test prioritization in industry ● Novel learning-based approach to test prioritization ● Implementation currently in pilot use at Salesforce ● Empirical evaluation using a large Salesforce dataset ● Ongoing work: add features, word2vec, deep learning, cost-aware
  12. 12. Thank you Q&A
  13. 13. Summary ● Main insights gained in conducting test prioritization in industry ● Novel learning-based approach to test prioritization ● Implementation currently in pilot use at Salesforce ● Empirical evaluation using a large Salesforce dataset ● Ongoing work: add features, word2vec/deep learning, cost-aware
  • JoannaWeekly

    Nov. 30, 2021

Talk given by Ben Busjaeger, Software Engineering Architect at Salesforce, at FSE 2016: ACM SIGSOFT on November 2016 Modern cloud-software providers, such as Salesforce.com, increasingly adopt large-scale continuous integration environments. In such environments, assuring high developer productivity is strongly dependent on conducting testing efficiently and effectively. Specifically, to shorten feedback cycles, test prioritization is popularly used as an optimization mechanism for ranking tests to run by their likelihood of revealing failures. To apply test prioritization in industrial environments, we present a novel approach (tailored for practical applicability) that integrates multiple existing techniques via a systematic framework of machine learning to rank. Our initial empirical evaluation on a large realworld dataset from Salesforce shows that our approach significantly outperforms existing individual techniques.

Views

Total views

1,403

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

0

Shares

0

Comments

0

Likes

1

×