Mutation testing 설명 및 예제

1. MUTATION TESTING
Quis custodiet ipsos custodies?
By COLE / ACT LAB

2. Quis custodiet ipsos custodies?
뀌ㅅ 꾸ㅅ또디엗 입소ㅅ 꾸ㅅ또데ㅅ
Who watches the watchmen?
누가 감시자를 감시할 것인가?

3. What is Mutation ?

4. What is Mutation ?
Original
Mutation
A mutation is a change that occurs in our DNA sequence, either
due to mistakes when the DNA is copied or as the result of
environmental factors such as UV light and cigarette smoke.

5. What is Mutation in SW ?
Mutation은
프로그램 내에서 의도적으로
결함을 유발시키기 위한 작은 변경 사항

6. What is Mutation in SW ?
If (a && b) { … }  If (a || b) { … }

7. What is Mutation Testing?
Mutation testing은
기존 테스트의 품질을 평가하는 Test
기존 Test 에서 다루는 코드를 약간 변경을 가하고 기존 Test suit가 변경
사항을 감지하고 거부하는지 여부를 확인하는 Test
변경사항을 감지 및 거부하지 않으면 테스트가 코드의 복잡성을 전부 수용
하지 못하고 하나 이상의 요소가 테스트되지 않은 상태임을 나타냄

8. CODE TESTS
MUTATION
TESTING
Verify Quality of Verify Quality of

9. 변경사항 감지 및 거부 O  Mutant Killed
변경사항 감지 및 거부 X  Mutant Survived

10. Value Mutations
변수를 변경 해서 프로그램 내 에러 탐지
가장 일반적인 방법으로는 상수를 변경하거나, 한 개 변수를 매우 크거나 작은 수로 변경
Decision Mutations
결정/조건 요소를 변경 설계 상의 에러 탐지
arithmetic operators(+, -, *, /, %) 변경
relational operators and logical operators (AND, OR , NOT) 변경
Statement Mutations
Line 삭제나 복제를 통해 에러 탐지
Mutation Testing Types

11. into
MUTATORS: CONDITION BOUNDARY
into
into
into
>
<
>=
<=
>=
<=
>
<

12. into
MUTATORS: NEGATE CONDITONALS
into
into
into
==
!=
>=
<=
>=
<=
<
>
into
into
<
>
>=
<=

13. into
MUTATORS: ETC
into
become
( doSomething(); is removed )
If( a == b )
+
return a;
If( true )
-
return 0;
Remove void invocations

14. Mutation Testing Benefits
개발자가 생각하지 못한 새로운 오류를 찾아줌
기존의 테스트 방법으로는 찾을 수 없는 숨겨진 결함을 찾아줌
디버깅 및 유지 관리가 이전보다 훨씬 쉬움

15. Why is not widely used?
Begins in 1971, Richard J. Lipton
Testing Maturity
Integration into the SW development process
Technical Problem – Brute Force

16. Code Coverage
‘Test coverage is a useful tool for finding
untested parts of a codebase‘
Martin Fowler

17. public class Calculator {
int valueDisplayed;
public Calculator() { this.valueDisplayed = 0; }
public Calculator(int initialValue) {
this.valueDisplayed = initialValue;
}
public void add(int x) { this.valueDisplayed += x; }
public void power(int x) {
this.valueDisplayed = (int) ;
Math.pow(this.valueDisplayed, x);
}
public int getResult() { return this.valueDisplayed; }
public void set(int x) { this.valueDisplayed = x; }
public boolean setConditional(int x, boolean yesOrNo) {
if(yesOrNo) {
set(x);
return true;
} else {
return false;
}
}
}
@Test
public void testAddition() {
Calculator calculator = new Calculator();
calculator.add(2);
assertEquals(calculator.getResult(), 2);
}
@Test
public void testPower() {
Calculator calculator = new Calculator(2);
calculator.power(3);
assertEquals(calculator.getResult(), 8);
}
@Test
public void testConditionalSetTrue() {
Calculator calculator = new Calculator();
assertEquals(calculator.setConditional(2, true), true);
}
@Test
public void testConditionalSetFalse() {
Calculator calculator = new Calculator();
assertEquals(calculator.setConditional(3, false), false);
}

21. public class Calculator {
int valueDisplayed;
public Calculator() { this.valueDisplayed = 0; }
public Calculator(int initialValue) {
this.valueDisplayed = initialValue;
}
public void add(int x) { this.valueDisplayed += x; }
public void power(int x) {
this.valueDisplayed = (int) ;
Math.pow(this.valueDisplayed, x);
}
public int getResult() { return this.valueDisplayed; }
public void set(int x) { this.valueDisplayed = x; }
public boolean setConditional(int x, boolean yesOrNo) {
if(yesOrNo) {
set(x);
return true;
} else {
return false;
}
}
}
@Test
public void testAddition() {
Calculator calculator = new Calculator();
calculator.add(2);
assertEquals(calculator.getResult(), 2);
}
@Test
public void testPower() {
Calculator calculator = new Calculator(2);
calculator.power(3);
assertEquals(calculator.getResult(), 8);
}
@Test
public void testConditionalSetTrue() {
Calculator calculator = new Calculator();
assertEquals(calculator.setConditional(2, true), true);
}
@Test
public void testConditionalSetFalse() {
Calculator calculator = new Calculator();
assertEquals(calculator.setConditional(3, false), false);
}

22. @Test
public void testConditionalSetTrue() {
Calculator calculator = new Calculator();
assertEquals(calculator.setConditional(2, true), true);
assertEquals(calculator.getResult(), 2);
}

23. MUTATION TESTING IS SLOW ?
단일 테스트케이스는 수행에 큰 문제가 없지만 테스트케이스가 많은 경
우 수행 시간이 증가될 수 있음
• 1000 classes , 10 unit test / class, 2ms / unit test
• 1000 x 10 x 2ms = 20sec
• 8 mutants / class : 1000 classes x 8 = 8000 mutants
• Brute force : 8000 x 20sec = 1day 20hour 26min 40sec
• Smart testing : 8000 x 10 x 2ms = 2min 40sec

24. •muJava A mutation tool for Java that includes class-level operators
•Mutate.py A Python script to mutate C-programs
•Mutator A source-based multi-language commercial mutation analyzer for concurrent Java, Ruby, JavaScript, PHP
•Bacterio Mutation testing tool for multi-class Java systems
•Javalanche Bytecode-based mutation testing tool for Java
•Major Compiler-integrated mutation testing framework for Java
•Jumble Bytecode-based mutation testing tool for Java
•PIT Bytecode-based mutation testing tool for Java
•Stryker Mutation testing tool for JavaScript
•Mutant AST based mutation testing tool for Ruby
•Jester Source-based mutation testing tool for Java
•Judy Mutation testing tool for Java
•Heckle Mutation testing tool for Ruby
•NinjaTurtles IL-based mutation testing tool for .NET and Mono
•Nester Mutation testing tool for C#
•Humbug Mutation testing tool for PHP
•MuCheck Mutation analysis library for Haskell
Mutation Testing Framework

25. Reference
• https://www.tutorialspoint.com/software_testing_dictionary/mu
tation_testing.htm
• http://www.inf.ed.ac.uk/teaching/courses/st/2011-
12/Resource-folder/09_mutation.pdf
• http://www.codeaffine.com/2015/10/05/what-the-heck-is-
mutation-testing/
• http://www.ontestautomation.com/an-introduction-to-
mutation-testing-and-pit/