Tested by Monkeys
The end of banana software!
Dr. Jeremias Rößler Founder ReTest
2
3
Hello!
4
Bug
5
Change
Bug
6
Change
Bug
Test
7
Change
Bug
Test
8
Change
Bug
Test
9
Change
Bug
Test
10
Change
Bug
Test
11
Change
Bug
Test
12
Change
Bug
Test
13
Change
Bug
Test
14
Change
Bug
Test
15
Change
Bug
Test
16
Change
Bug
Test
17
Change
Bug
Test
18
Change
Bug
Test
19
Change
Bug
Test
20
SURVEY
21
Who has ever tested manually?
SURVEY
22
Who has ever regression-tested manually?
Who has ever tested manually?
SURVEY
23
Who has ever regression-tested manually?
Who has ever tested manually?
Who liked it?
SURVEY
24
SURVEY
Why didn’t you automate regression testing?
25
Mehr-Aufwand
Zeit
Einsparung
Time
Additional
Effort
Saved Effort
26
AUTOMATION
Maintenance effort
Risk
Test Coverage
27
Problem:
- high testing effort
- high test automation effort
28
Infinite-Monkey-Theorem:
A monkey hitting keys at random on a typewriter
for a long enough time
will type the complete works of William Shakespeare.
29
We replace the typewriter with a computer…
Infinite Monkey
30
public static void main(String... args) throws Exception {
Robot robot = new Robot();
while (true) {
robot.mouseMove(random.nextInt(maxX), random.nextInt(maxY));
robot.mousePress(InputEvent.BUTTON1_DOWN_MASK);
robot.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);
robot.delay(200);
for (char inputChar : randomString().toCharArray()) {
robot.keyPress((int) inputChar);
robot.keyRelease((int) inputChar);
robot.delay(10);
}
robot.keyPress(KeyEvent.VK_ENTER);
robot.keyRelease(KeyEvent.VK_ENTER);
}
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
Infinite Monkey
31
DEMO
32
33
34
Intelligent Monkey
35
public static void main(String... args) throws Exception {
WebDriver driver = new FirefoxDriver();
driver.get("http://www.retest.de");
while (true) {
List<WebElement> links = driver.findElements(By.tagName("a"));
links.get(random.nextInt(links.size())).click();
Thread.sleep(500);
List<WebElement> fields =
driver.findElements(By.xpath("//input[@type='text']"));
WebElement field = fields.get(random.nextInt(fields.size()));
field.sendKeys(randomString());
Thread.sleep(500);
}
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
36
Intelligent Monkey
DEMO
37
Intelligent Monkey
38
39
- gremlins.js
- NetFlix Chaos Monkey
- UI/Application Exerciser Monkey
Open Source Products
40
- quick and easy to implement
- finds bugs
Traditional monkey testing
but:
- relevant bugs?
- not intelligent
- not trainable
- needs sandboxing
41
Intelligent Monkey
?
42
43
44
Problem space
Problem
space
45
46
47
48
- bases on realistic use case
- covers corner cases
- what about coverage?
Trainable monkey testing
49
50
Several searches in parallel
51Source: https://www.youtube.com/watch?v=yJpLUP93pRo
52Source: http://cnx.org/content/m47304/latest/
© Robert Bear and David Rintoul
53
Gen
Exon
Intron
Exon
Tower of Hanoi
54
© André Karwath aka Aka
Source: http://en.wikipedia.org/wiki/File:Tower_of_Hanoi_4.gif
55
Tower of Hanoi
Possibilities
63
>2
56
9.223.372.036.854.775.808
Tower of Hanoi
Possibilities
57
Tower of Hanoi
Possibilities
292.471.208 Years
58
Genetic Algorithm
efficiency
baseline: 150 mio. years
with GA: 66 seconds
59
60
- bases on realistic use case
- optimises for coverage
- very effective
intelligent monkey testing
61
How do you bring AIinto testing?
62
AI in Testautomation is a hype.
Wolfgang Platz,
Founder & CPO of Tricentis
“
63
AI in Testing?
– so far I do not see anyone doing it
in a useful way.
Antony Edwards,
CTO of TestPlant
“
64
Which bugs can the monkey find?
65
66
Oracle Problem
67
When is a bug a bug?
When it’s not a feature!
68
What is a bug?
Without specification, there are no bugs
— only surprises.
Brian Kernighan
“
69
Why do we test?
After
implementation?
1 + 1 = 3
2 + 2 = 4
70
71
72
73
Why do we test?
After
implementation?
1 + 1 = 3
2 + 2 = 4
After
change?
1 + 1 = 3
1 + 1 = 5
74
regression-test
!=
test
75
regression-test
==
version control
76
77
78
79
80
- oracle problems hinders AI
- regression testing needs no oracle
- VCS for the GUI
functional monkey testing
81
Intelligenter Affe
Summary
Monkey-Testing:
cheaper
faster
better
multiplies manual tests
Difference Testing
little maintenance
no wasteful testing
yet more complete
82
83
Coming 2017Mobile Web
84
https://retest.rocks
@ReTest_en
@roesslerj
Dr. Jeremias Rößler Founder ReTest
Questions?
https://www.facebook.com/retest.de
https://retest.rocksroessler@retest.de
86
1. if (name[0] == '.') continue;
Is it a bug?
.
..
text.c
other.files
87
Is it a bug?
88
Is it a bug?

Jeremias Rößler