### SlideShare for iOS

by Linkedin Corporation

FREE - On the App Store

Combinatorial Group Testing for Software Integration. Leslie Wu. ...

Combinatorial Group Testing for Software Integration. Leslie Wu.

Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP)

Today’s datacenter-scale distributed systems deploy software-as-a-service:

A pageview on Google may trigger upwards of 50 services (Jeff Dean)

View on Amazon.com may access more than 100 services (AMZN CTO)

Our work: apply “group testing” to software integration in-the-large.

Goal: accelerate integration-defect root cause analysis

- Total Views
- 1,659
- Views on SlideShare
- 1,656
- Embed Views

- Likes
- 0
- Downloads
- 1
- Comments
- 0

http://www.slideshare.net | 3 |

- 1. Shattr: Combinatorial Group Testing for Software Integration Leslie Wu CS343, 2007 June
- 2. Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO)
- 3. Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO) • Our work: apply “group testing” to software integration in-the-large • Goal: accelerate integration-defect root cause analysis
- 4. Background • Group Testing: Syphilis in World War II draftees • method invented Robert Dorfman, an economist • Electronics testing (1960s-) • Pooling designs in Biology (1990s-) • Software Integration (modern day)
- 5. Related work • Combinatorial Group Testing (Du and Hwang 2000) • Pooling Designs in Biology (Du and Hwang 2006) • Web services / Delta debugging (Zeller) • Group Testing on Complexes vs. Isolated Components • (More details in ﬁnal report)
- 6. Problem statement • Testing a group g of services means: • Upgrade all services in g • Perform an automated integration test • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)
- 7. Problem statement • Testing a group g of services means: C • Upgrade all services in g • Perform an automated integration test A B • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)
- 8. Integration cost metrics 1) Number of integration tests (traditional group testing) 2) Depth of recursion tree (inspired by pooling designs) • More tests means more test machines required • Deeper recursion tree means longer time to isolate integration defect
- 9. Methods for Software Integration 1. Bottom-up integration 2. Top-down 3. Shattr-style
- 10. Traditional integration tree • Trees and branches Mozilla.org
- 11. Bottom-up integration After integration Before integration n=8 Image from http://www.gigaﬂop.demon.co.uk/comp/chapt3.htm
- 12. Bottom-up integration OK A C B Image from http://www.gigaﬂop.demon.co.uk/comp/chapt3.htm
- 13. Bottom-up integration Failure! ...what’s the root cause? OK OK OK A C B Image from http://www.gigaﬂop.demon.co.uk/comp/chapt3.htm
- 14. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
- 15. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
- 16. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
- 17. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
- 18. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
- 19. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
- 20. Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!
- 21. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests sufﬁce to determine defective edge
- 22. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests sufﬁce to determine defective edge
- 23. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests sufﬁce to determine defective edge
- 24. Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests sufﬁce to determine defective edge
- 25. Simulation • (Ruby)
- 26. Simulation Results A sample: (n=128) Time # tests (avg.) Tree 7 32 Shattr 4 28 (More details in report)
- 27. Contributions • Novel application of “Group Testing for 1-Complexes” to Software Integration • Proposed graph-theoretic model for Software Integration as a social system • Introduced software service integration cost metrics • Describe several integration algorithms • Implemented simulation in Ruby, more analysis in report • Demonstrated potential way to ﬁnd integration defects more quickly • Literature review
- 28. Future work • Remove restrictions: generalize to k-complexes • More speciﬁc graph model, interaction not generally complete graph • More data-driven -- integration defect count and probability • Core theory: practical group testing on complexes relatively unstudied • Prototyping in the wild: is group testing feasible in practice?

Full NameComment goes here.