Shattr: Combinatorial Group Testing for Software Integration

Shattr: Combinatorial Group Testing for Software Integration Leslie Wu CS343, 2007 June

Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO)

Motivation • Software testing under-researched but important in practice (Amitabh Srivastava, MSFT VP) • Today’s datacenter-scale distributed systems deploy software-as-a-service: • A pageview on Google may trigger upwards of 50 services (Jeff Dean) • View on Amazon.com may access more than 100 services (AMZN CTO) • Our work: apply “group testing” to software integration in-the-large • Goal: accelerate integration-defect root cause analysis

Background • Group Testing: Syphilis in World War II draftees • method invented Robert Dorfman, an economist • Electronics testing (1960s-) • Pooling designs in Biology (1990s-) • Software Integration (modern day)

Related work • Combinatorial Group Testing (Du and Hwang 2000) • Pooling Designs in Biology (Du and Hwang 2006) • Web services / Delta debugging (Zeller) • Group Testing on Complexes vs. Isolated Components • (More details in ﬁnal report)

Problem statement • Testing a group g of services means: • Upgrade all services in g • Perform an automated integration test • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)

Problem statement • Testing a group g of services means: C • Upgrade all services in g • Perform an automated integration test A B • Graph model of integration: each vertex corresponds to a service • Problem: Find the defective (“positive”) edge • Assume only one bad edge (k-complexes for k=1)

Integration cost metrics 1) Number of integration tests (traditional group testing) 2) Depth of recursion tree (inspired by pooling designs) • More tests means more test machines required • Deeper recursion tree means longer time to isolate integration defect

Methods for Software Integration 1. Bottom-up integration 2. Top-down 3. Shattr-style

Traditional integration tree • Trees and branches Mozilla.org

Bottom-up integration After integration Before integration n=8 Image from http://www.gigaﬂop.demon.co.uk/comp/chapt3.htm

Bottom-up integration OK A C B Image from http://www.gigaﬂop.demon.co.uk/comp/chapt3.htm

Bottom-up integration Failure! ...what’s the root cause? OK OK OK A C B Image from http://www.gigaﬂop.demon.co.uk/comp/chapt3.htm

Natural divide-and-conquer • Example: • Given 64 services/components, divide into four subsets of size 16 • Test all (4 choose 2) pairs of subsets (6 edges), recurse on positive edge • Doesn’t scale if you need to increase branching factor!

Group Testing for 1-Complexes • Shattr-style: • Instead, perform group tests on larger, overlapping subsets • ...recurse on intersection of positive subsets • Example: 3 tests sufﬁce to determine defective edge

Simulation • (Ruby)

Simulation Results A sample: (n=128) Time # tests (avg.) Tree 7 32 Shattr 4 28 (More details in report)

Contributions • Novel application of “Group Testing for 1-Complexes” to Software Integration • Proposed graph-theoretic model for Software Integration as a social system • Introduced software service integration cost metrics • Describe several integration algorithms • Implemented simulation in Ruby, more analysis in report • Demonstrated potential way to ﬁnd integration defects more quickly • Literature review

Future work • Remove restrictions: generalize to k-complexes • More speciﬁc graph model, interaction not generally complete graph • More data-driven -- integration defect count and probability • Core theory: practical group testing on complexes relatively unstudied • Prototyping in the wild: is group testing feasible in practice?

Shattr: Combinatorial Group Testing for Software Integration

by Leslie W on May 07, 2010

Accessibility

Tags

Upload Details

Usage Rights

1 Embed 3

Statistics

Shattr: Combinatorial Group Testing for Software Integration — Presentation Transcript