Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Shattr: Combinatorial Group Testing for Software Integration

Combinatorial Group Testing for Software Integration. Leslie Wu.

Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP)

Today’s datacenter-scale distributed systems deploy software-as-a-service:

A pageview on Google may trigger upwards of 50 services (Jeff Dean)

View on Amazon.com may access more than 100 services (AMZN CTO)

Our work: apply “group testing” to software integration in-the-large.

Goal: accelerate integration-defect root cause analysis

  • Be the first to comment

  • Be the first to like this

Shattr: Combinatorial Group Testing for Software Integration

  1. 1. Shattr: Combinatorial Group Testing for Software Integration Leslie Wu CS343, 2007 June
  2. 2. Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO)
  3. 3. Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO) • Our work: apply “group testing” to software integration in-the-large • Goal: accelerate integration-defect root cause analysis
  4. 4. Background • Group Testing: Syphilis in World War II draftees • method invented Robert Dorfman, an economist • Electronics testing (1960s-) • Pooling designs in Biology (1990s-) • Software Integration (modern day)
  5. 5. Related work • Combinatorial Group Testing (Du and Hwang 2000) • Pooling Designs in Biology (Du and Hwang 2006) • Web services / Delta debugging (Zeller) • Group Testing on Complexes vs. Isolated Components • (More details in final report)
  6. 6. Problem statement • Testing a group g of services means: • Upgrade all services in g • Perform an automated integration test • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)
  7. 7. Problem statement • Testing a group g of services means: C • Upgrade all services in g • Perform an automated integration test A B • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)
  8. 8. Integration cost metrics 1) Number of integration tests (traditional group testing) 2) Depth of recursion tree (inspired by pooling designs) • More tests means more test machines required • Deeper recursion tree means longer time to isolate integration defect
  9. 9. Methods for Software Integration 1. Bottom-up integration 2. Top-down 3. Shattr-style
  10. 10. Traditional integration tree • Trees and branches Mozilla.org
  11. 11. Bottom-up integration After integration Before integration n=8 Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
  12. 12. Bottom-up integration OK A C B Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
  13. 13. Bottom-up integration Failure! ...what’s the root cause? OK OK OK A C B Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
  14. 14. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  15. 15. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  16. 16. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  17. 17. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  18. 18. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  19. 19. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  20. 20. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  21. 21. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
  22. 22. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
  23. 23. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
  24. 24. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
  25. 25. Simulation • (Ruby)
  26. 26. Simulation Results A sample: (n=128) Time # tests (avg.) Tree 7 32 Shattr 4 28 (More details in report)
  27. 27. Contributions • Novel application of “Group Testing for 1-Complexes” to Software Integration • Proposed graph-theoretic model for Software Integration as a social system • Introduced software service integration cost metrics • Describe several integration algorithms • Implemented simulation in Ruby, more analysis in report • Demonstrated potential way to find integration defects more quickly • Literature review
  28. 28. Future work • Remove restrictions: generalize to k-complexes • More specific graph model, interaction not generally complete graph • More data-driven -- integration defect count and probability • Core theory: practical group testing on complexes relatively unstudied • Prototyping in the wild: is group testing feasible in practice?

×