Shattr: Combinatorial Group Testing for Software Integration

1,497 views
1,440 views

Published on

Combinatorial Group Testing for Software Integration. Leslie Wu.

Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP)

Today’s datacenter-scale distributed systems deploy software-as-a-service:

A pageview on Google may trigger upwards of 50 services (Jeff Dean)

View on Amazon.com may access more than 100 services (AMZN CTO)

Our work: apply “group testing” to software integration in-the-large.

Goal: accelerate integration-defect root cause analysis

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,497
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
















  • Shattr: Combinatorial Group Testing for Software Integration

    1. 1. Shattr: Combinatorial Group Testing for Software Integration Leslie Wu CS343, 2007 June
    2. 2. Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO)
    3. 3. Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO) • Our work: apply “group testing” to software integration in-the-large • Goal: accelerate integration-defect root cause analysis
    4. 4. Background • Group Testing: Syphilis in World War II draftees • method invented Robert Dorfman, an economist • Electronics testing (1960s-) • Pooling designs in Biology (1990s-) • Software Integration (modern day)
    5. 5. Related work • Combinatorial Group Testing (Du and Hwang 2000) • Pooling Designs in Biology (Du and Hwang 2006) • Web services / Delta debugging (Zeller) • Group Testing on Complexes vs. Isolated Components • (More details in final report)
    6. 6. Problem statement • Testing a group g of services means: • Upgrade all services in g • Perform an automated integration test • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)
    7. 7. Problem statement • Testing a group g of services means: C • Upgrade all services in g • Perform an automated integration test A B • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)
    8. 8. Integration cost metrics 1) Number of integration tests (traditional group testing) 2) Depth of recursion tree (inspired by pooling designs) • More tests means more test machines required • Deeper recursion tree means longer time to isolate integration defect
    9. 9. Methods for Software Integration 1. Bottom-up integration 2. Top-down 3. Shattr-style
    10. 10. Traditional integration tree • Trees and branches Mozilla.org
    11. 11. Bottom-up integration After integration Before integration n=8 Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
    12. 12. Bottom-up integration OK A C B Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
    13. 13. Bottom-up integration Failure! ...what’s the root cause? OK OK OK A C B Image from http://www.gigaflop.demon.co.uk/comp/chapt3.htm
    14. 14. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
    15. 15. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
    16. 16. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
    17. 17. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
    18. 18. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
    19. 19. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
    20. 20. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
    21. 21. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
    22. 22. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
    23. 23. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
    24. 24. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests suffice to determine defective edge
    25. 25. Simulation • (Ruby)
    26. 26. Simulation Results A sample: (n=128) Time # tests (avg.) Tree 7 32 Shattr 4 28 (More details in report)
    27. 27. Contributions • Novel application of “Group Testing for 1-Complexes” to Software Integration • Proposed graph-theoretic model for Software Integration as a social system • Introduced software service integration cost metrics • Describe several integration algorithms • Implemented simulation in Ruby, more analysis in report • Demonstrated potential way to find integration defects more quickly • Literature review
    28. 28. Future work • Remove restrictions: generalize to k-complexes • More specific graph model, interaction not generally complete graph • More data-driven -- integration defect count and probability • Core theory: practical group testing on complexes relatively unstudied • Prototyping in the wild: is group testing feasible in practice?

    ×