Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dealing with combinatorial explosions and boring tests

15,220 views

Published on

This presentation explains parameterized tests, theory tests, and generative testing. It also explains single mode faults and double mode faults and shows how to reduce the number of test cases when there's an combinatorial explosion. Lot's of JUnit examples.

Published in: Software
  • Be the first to comment

Dealing with combinatorial explosions and boring tests

  1. 1. JDays 2015 AlexanderTarnowski Dealing with combinatorial explosionsand boring tests
  2. 2.  Developer (2000→) Java, Perl, C, C++, Groovy, C#, PHP, Visual Basic, Assembler  Trainer – TDD, Unit testing, Clean Code, WebDriver, Specification by Example  Developer mentor  Writer  Scrum Master  Coach in training WhoamI? https://www.crisp.se/konsulter/alexander-tarnowski alexander_tar alexander.tarnowski@crisp.se
  3. 3. ”We want our customers to be able to compute their car insurance premiums online. Online quotes are in our favor, since we outprice our competitors!” Who is Tim? Image: stockimages/freedigitalphotos.net Alexander Tarnowski
  4. 4. Age Premium for males Premium for females 18-23 1.75 1.575 24-59 1.0 0.9 60+ 1.35 1.215 Business rules Alexander Tarnowski
  5. 5. @Test public void maleDriversAged18() { assertEquals(1.75, new PremiumRuleEngine() .getPremiumFactor(18, Gender.MALE), 0.0); } The first test Age Premium for males Premium for females 18-23 1.75 1.575 24-59 1.0 0.9 60+ 1.35 1.215 Alexander Tarnowski
  6. 6. @Test public void maleDriversAged23() { assertEquals(1.75, new PremiumRuleEngine() .getPremiumFactor(23, Gender.MALE), 0.0); } The second test Age Premium for males Premium for females 18-23 1.75 1.575 24-59 1.0 0.9 60+ 1.35 1.215 Alexander Tarnowski
  7. 7. And this could go on… Image: imagerymajestic/freedigitalphotos.net Alexander Tarnowski
  8. 8.  Silly names  Repetitive test structure  Boredom Smells and insights Image: Mister GC/freedigitalphotos.net Alexander Tarnowski
  9. 9.  How many equivalence classes?  How many boundary values?  Would we test drive all of them? How many tests are needed? 0 0.5 1 1.5 2 18-23 24-59 60+ Male Female Alexander Tarnowski
  10. 10. @RunWith(Parameterized.class) public class PremiumAgeIntervalsTest { private double expectedPremiumFactor; private int age; private Gender gender; public PremiumAgeIntervalsTest(double expectedPremiumFactor, int age, Gender gender) { this.expectedPremiumFactor = expectedPremiumFactor; this.age = age; this.gender = gender; } @Parameters public static Collection<Object[]> data() { return Arrays.asList(new Object[][]{ {1.75, 18, Gender.MALE}, {1.75, 23, Gender.MALE}, {1.0, 24, Gender.MALE}, {1.0, 59, Gender.MALE}, {1.35, 60, Gender.MALE}, {1.575, 18, Gender.FEMALE}, {1.575, 23, Gender.FEMALE}, {0.9, 24, Gender.FEMALE}, {0.9, 59, Gender.FEMALE}, {1.215, 60, Gender.FEMALE}} ); } @Test public void verifyPremiumFactor() { assertEquals(expectedPremiumFactor, new PremiumRuleEngine() .getPremiumFactor(age, gender), 0.0); } } Theparameterized version Alexander Tarnowski
  11. 11. @RunWith(Parameterized.class) public class PremiumAgeIntervalsTest { @Parameter(value = 0) public double expectedPremiumFactor; @Parameter(value = 1) public int age; @Parameter(value = 2) public Gender gender; @Parameters(name = "Case {index}: Expected {0} for {1} year old {2}s") public static Collection<Object[]> data() { return Arrays.asList(new Object[][]{ {1.75, 18, Gender.MALE}, {1.75, 23, Gender.MALE}, {1.0, 24, Gender.MALE}, {1.0, 59, Gender.MALE}, {1.35, 60, Gender.MALE}, {1.575, 18, Gender.FEMALE}, {1.575, 23, Gender.FEMALE}, {0.9, 24, Gender.FEMALE}, {0.9, 59, Gender.FEMALE}, {1.215, 60, Gender.FEMALE}} ); } @Test public void verifyPremiumFactor() { assertEquals(expectedPremiumFactor, new PremiumRuleEngine() .getPremiumFactor(age, gender), 0.0); } } Alternative syntax Alexander Tarnowski
  12. 12.  Focus on data  Allow comparing a set of predefined inputs with some predefined output  Make checking simple  Turn tests with silly names into data-driven tests Parameterized tests Alexander Tarnowski
  13. 13.  Interface  Happy path  Learning  Error handling TDD vsTesting TDD Testing(checking)  Equivalence partitions  Boundary values  Decision tables  State machines Alexander Tarnowski
  14. 14. @Test public void aHamburgerIsnHealthyFood() { assertThat(Menu.HAMBURGER.getCalories(), greaterThan(200)); } Speaking offood… Alexander Tarnowski
  15. 15. Hamburgers contain the least amount of calories among common fast foods Theory Alexander Tarnowski
  16. 16. public class CalorieComparisonTest { public static List<FastFood> foods() { return Arrays.asList(Menu.FISH_BURGER, Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH, Menu.HOTDOG); } @Test public void hamburgersContainTheLeastAmountOfCaloriesAmongFastFoods() { for (FastFood food : foods()) assertThat(Menu.HAMBURGER.getCalories(), is(lessThan(food.getCalories()))); } } } Proving the theory Alexander Tarnowski
  17. 17. @RunWith(Theories.class) public class CalorieComparisonTest { @DataPoints public static List<FastFood> foods() { return Arrays.asList(Menu.FISH_BURGER, Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH, Menu.HOTDOG); } @Theory public void hamburgersContainTheLeastAmountOfCaloriesAmongFastFoods(FastFood food) { assertThat(Menu.HAMBURGER.getCalories(), is(lessThan(food.getCalories()))); } } Atheorytest Alexander Tarnowski
  18. 18.  No fast food meal contains less than 500 calories! Amore interesting theory Image: marin/freedigitalphotos.net Alexander Tarnowski
  19. 19. @RunWith(Theories.class) public class FastFoodMealTheoryTest { @DataPoints public static List<Main> mainCourses() { return Arrays.asList(Menu.HAMBURGER, Menu.FISH_BURGER, Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH, Menu.HOTDOG); } @DataPoints public static List<SideOrder> sideOrders() { return Arrays.asList(Menu.SMALL_FRENCH_FRIES, Menu.LARGE_FRENCH_FRIES, Menu.APPLE_PIE, Menu.SMALL_CHOCOLATE_MILKSHAKE); } @DataPoints public static List<Beverage> bevereges() { return Arrays.asList(Menu.MEDIUM_COKE, Menu.LARGE_DIET_COKE, Menu.MEDIUM_LATTE, Menu.LARGE_LATTE); } @Theory public void noFastFoodMealContainsLessThan500calories(Main main, SideOrder sideOrder, Beverage beverage) { assumeThat(beverage.isDiet(), is(false)); assertThat(main.getCalories() + sideOrder.getCalories() + beverage.getCalories(), is(greaterThan(500))); } } Alexander Tarnowski
  20. 20.  Feed the test with all main courses and all side orders and all beverages  Cartesian product: mains X side orders X beverages  assumeThat prunes some combinations Behind the scenes (Hamburger, Small french fries, Medium coke) (Hamburger, Small french fries, Large diet coke) (Hamburger, Small french fries, Medium latte) (Hamburger, Small french fries, Large latte) (Hamburger, Large french fries, Medium coke) (Hamburger, Large french fries, Large diet coke) (Hamburger, Large french fries, Medium latte) (Hamburger, Large french fries, Large latte) … Alexander Tarnowski
  21. 21.  Are built around ”for all” type of reasoning  Can’t pair specific data points with specific results  Let you work with the Cartesian product of multiple variables Theories Alexander Tarnowski
  22. 22. Let’s do the Caesarcipher… A B C D E F G H I J K L M O P Q R S T U V X Y Z S T U V X Y Z A B C D E F G H I J K L M O P Q R CAESAR=USXKSJ Alexander Tarnowski
  23. 23. Does it work? … by borrowing an onlineimplementation Alexander Tarnowski
  24. 24.  For a bunch of different arbitrary strings…  ... and a bunch of different offsets...  ... try the following: CaesarCipher.decode(CaesarCipher.encode(string, offset), offset) Dream scenario Alexander Tarnowski
  25. 25. @RunWith(Theories.class) public class CaesarCipherTest { @Theory public void caesarCipherRoundTrip(@RandomString(maxLength = 128) String plainText, @TestedOn(ints = {0, 1, 2, 10, 26, 27, 1000}) int offset) { assertEquals(plainText, CaesarCipher.decode(CaesarCipher.encode(plainText, offset), offset)); } } We can do that! Alexander Tarnowski
  26. 26. RandomString.java: @Retention(RetentionPolicy.RUNTIME) @ParametersSuppliedBy(RandomStringSupplier.class) public @interface RandomString { int maxLength(); } RandomStringSupplier.java: public class RandomStringSupplier extends ParameterSupplier { @Override public List<PotentialAssignment> getValueSources(ParameterSignature signature) throws Throwable { RandomString annotation = signature.getAnnotation(RandomString.class); int length = (int) (Math.random() * annotation.maxLength()); final String s = RandomStringUtils.randomAlphanumeric(length); return Arrays.asList(PotentialAssignment.forValue("random string", s)); } } @RandomString Alexander Tarnowski
  27. 27. RandomStringsSupplier.java: public class RandomsStringSupplier extends ParameterSupplier { @Override public List<PotentialAssignment> getValueSources(ParameterSignature signature) throws Throwable { List<PotentialAssignment> values = new ArrayList<>(); RandomStrings annotation = signature.getAnnotation(RandomStrings.class); Generator<String> stringGenerator = strings(integers(1, 128, Distribution.INVERTED_NORMAL), characters()); for (int i = 0; i < annotation.count(); i++) { values.add(PotentialAssignment.forValue("random string", stringGenerator.next())); } return values; } } QuickCheck style Alexander Tarnowski
  28. 28. Examples of generators  booleans()  dates(Long low, Long high, TimeUnit precision)  fixedValues(T... values)  strings(Generator<Integer> length, Generator<Character> characters)  arrays(Generator<? extends T> content, Class<T> type)  excludeValues(Generator<T> generator, T... excluded)  sortedLists(Generator<T> content, int low, int high) net.java.quickcheck public interface Generator<T> { /** * Generates the next instance. * * @return a newly created instance */ public T next(); } Alexander Tarnowski
  29. 29.  Anything goes!  May involve huge domains  Often involves inverse functions Generative testing Alexander Tarnowski
  30. 30.  Now we can execute thousands of tests!  But what if we want the opposite? Congratulations! Alexander TarnowskiImage: zole4/freedigitalphotos.net
  31. 31. Back to car insurance premiums Gender Male Female Age interval 18-24 25-59 60+ Yearly mileage 0 1-1000 1001-3000 3001-6000 6001+ Safety features None Airbag ABS HIP Multiple Brand Nissan Volvo Ferrari Toyota Ford Volkswagen Driving record Model Driver Average Joe Unlucky Uma Bad Judgement Jed Dangerous Dan 2 x 3 x 5 x 5 x 6 x 5 = 4500 Alexander Tarnowski
  32. 32.  One parameter causes the error  Only 6 tests are needed! Single mode faults Brand Drivingrecord Yearlymileage Safetyfeatures Ageinterval Gender Nissan ModelDriver 0 None 18-24 Male Volvo AverageJoe 1-1000 Airbag 25-59 Female Ferrari UnluckyUma 1001-3000 ABS 60+ - Toyota BadJudgementJed 3001-6000 HIP - - Ford DangerousDan 6001+ Multiple - - Volkswagen - - - - - Alexander Tarnowski
  33. 33.  A combination of two parameters causes the error  Run through a tool that computes all pairs (or look up in a table of orthogonal arrays)  Only ~40 tests are needed! Double mode faults Alexander Tarnowski
  34. 34. Finding all pairs by hand Row Variable 1 Variable 2 Variable 3 1 A X Q 2 A X R 3 A Y Q 4 A Y R 5 B X Q 6 B X R 7 B Y Q 8 B Y R Alexander Tarnowski
  35. 35.  Theoretical foundation: orthogonal arrays  Reduce the number of tests from thousands to just a few  Great to put into parameterized tests Finding Single and Doublemode faults Alexander Tarnowski
  36. 36.  Unit tests are examples  Parameterized tests make writing many similar tests easy  Theory tests introduce general statements about program elements  Generative tests – Anything goes!  Single mode faults & double mode faults – Reduce the number of tests and feed parameterized tests Summary Alexander Tarnowski
  37. 37.  https://leanpub.com/developer_testing  http://web.archive.org/web/20110808084654/http://shareandenjoy.saff.net/tdd- specifications.pdf  https://github.com/junit-team/junit/wiki/Theories  https://github.com/pholser/junit-quickcheck  http://www.satisfice.com/tools.shtml Some resources Alexander Tarnowski

×