Shattr: Combinatorial Group Testing for Software Integration

  • 1,123 views
Uploaded on

Combinatorial Group Testing for Software Integration. Leslie Wu. …

Combinatorial Group Testing for Software Integration. Leslie Wu.

Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP)

Today’s datacenter-scale distributed systems deploy software-as-a-service:

A pageview on Google may trigger upwards of 50 services (Jeff Dean)

View on Amazon.com may access more than 100 services (AMZN CTO)

Our work: apply “group testing” to software integration in-the-large.

Goal: accelerate integration-defect root cause analysis

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,123
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
















Transcript

  • 1. Shattr: Combinatorial Group Testing for Software Integration Leslie Wu CS343, 2007 June
  • 2. Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO)
  • 3. Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO) • Our work: apply “group testing” to software integration in-the-large • Goal: accelerate integration-defect root cause analysis
  • 4. Background • Group Testing: Syphilis in World War II draftees • method invented Robert Dorfman, an economist • Electronics testing (1960s-) • Pooling designs in Biology (1990s-) • Software Integration (modern day)
  • 5. Related work • Combinatorial Group Testing (Du and Hwang 2000) • Pooling Designs in Biology (Du and Hwang 2006) • Web services / Delta debugging (Zeller) • Group Testing on Complexes vs. Isolated Components • (More details in final report)
  • 6. Problem statement • Testing a group g of services means: • Upgrade all services in g • Perform an automated integration test • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)
  • 7. Problem statement • Testing a group g of services means: C • Upgrade all services in g • Perform an automated integration test A B • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)
  • 8. Integration cost metrics 1) Number of integration tests (traditional group testing) 2) Depth of recursion tree (inspired by pooling designs) • More tests means more test machines required • Deeper recursion tree means longer time to isolate integration defect
  • 9. Methods for Software Integration 1. Bottom-up integration 2. Top-down 3. Shattr-style
  • 10. Traditional integration tree • Trees and branches Mozilla.org
  • 11. Bottom-up integration After integration Before integration n=8 Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
  • 12. Bottom-up integration OK A C B Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
  • 13. Bottom-up integration Failure! ...what’s the root cause? OK OK OK A C B Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
  • 14. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  • 15. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  • 16. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  • 17. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  • 18. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  • 19. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  • 20. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
  • 21. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
  • 22. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
  • 23. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
  • 24. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
  • 25. Simulation • (Ruby)
  • 26. Simulation Results A sample: (n=128) Time # tests (avg.) Tree 7 32 Shattr 4 28 (More details in report)
  • 27. Contributions • Novel application of “Group Testing for 1-Complexes” to Software Integration • Proposed graph-theoretic model for Software Integration as a social system • Introduced software service integration cost metrics • Describe several integration algorithms • Implemented simulation in Ruby, more analysis in report • Demonstrated potential way to find integration defects more quickly • Literature review
  • 28. Future work • Remove restrictions: generalize to k-complexes • More specific graph model, interaction not generally complete graph • More data-driven -- integration defect count and probability • Core theory: practical group testing on complexes relatively unstudied • Prototyping in the wild: is group testing feasible in practice?