When the British were ruling in colonial India, they were concerned about the number of venomous cobra snakes in Delhi. The government therefore offered a bounty for every dead cobra. Initially this was a successful strategy as large numbers of snakes were killed for the reward. Eventually, however, enterprising people began to breed cobras for the income. When the government became aware of this, the reward program was scrapped, causing the cobra breeders to set the now-worthless snakes free. As a result, the wild cobra population further increased. The apparent solution for the problem made the situation even worse. Our tests which we assume to be a safeguard for our code might just make our situation even worse!

1. WHO WILL TEST YOUR TESTS ?
Yahya Poonawala @meetykp
Priti Biyani @pritibiyani

2. 2https://flic.kr/p/9uZhDS 2

3. TESTSCode
works
Reduce
Bugs
Reduce
cost of
change
Allow
Refactoring
Faster
development
Reduce
fear
3

4. NOW
WHEN YOU HEAR ….
4

5. Oh, it failed?
Don’t worry. Just re-trigger it!
5

6. Oh, it failed?
Strange; it passed on my machine!
6

7. Oh, it failed?
Run it on agent 007!
It’s an agent issue!
7

8. Oh, it failed?
I know. It fails at 12AM!
8

9. 9

10. Now, again,
WHY DO WE WRITE TESTS?
1
0

11. Tests prove that the code actually works
Tests prove that the code
sometimes works
1
1

12. Tests reduce bugs
Tests itself might have bugs
1
2

13. Tests reduce the cost of change
Change? Are you kidding?
1
3

14. Tests helps in faster development.
Testing is slowing me down 
1
4

15. NON-DETERMINISTIC
TESTS
A.K.A “Flaky Tests”
1
5

16. 1
6http://www.trovatten.com/long-game/

17. 1
7

18. 1
8
http://tom-doherty.com/wp-content/uploads/2014/02/craigslist-waste-time.jpg

19. 1
9

20. ILLUSION OF SUCCESS!
2
0

21. ILLUSION OF FAILURE!
2
1

22. 2
2

23. 2
3

24. DO NOT DELETE YOUR
FLAKY TESTS!!
2
4

25. CAUSES OF FLAKINESS
2
5

26. Thread 1::
if( name is not Yahya ) {
…
fputs( name, file )
…
}
Thread 2::
Initialize name to Yahya

27. CONCURRENCY
2
7

28. Initialize lock;
Thread 1::
Acquire lock
if( name is not Yahya) {
. . .
fputs( name, file )
. . .
}
Release lock
Thread 2::
Acquire lock
Initialize name to Yahya
Release lock

29. TIME
2
9
http://imgs.xkcd.com/comics/bug.png

30. EXTERNAL SYSTEM CALLS
3
0

31. UNORDERED COLLECTIONS
3
1

32. 3
2
https://www.vardot.com/sites/default/files/2.jpg

33. # pseudo-code
makeAsyncCall;
sleep(2000);
readResponse;

34. ASYNCHRONOUS
WAITING
3
4

35. make asynchronous call;
While (response received?)
if timeout then
throw Test Timeout Exception
else
sleep(10)
. . .
read response

36. RESOURCE LEAKS
3
6

37. # //pseudo-code
FILE_NAME = “content.txt”
// Test 1:
FileReader.read(FILE_NAME).to be empty
// Test2: Write to the file
data = “XP Conf 2015”
FileWriter.write(FILE_NAME, data)
FileReader.read(FILE_NAME) == data
// Test 3: Updates the file
append_data = “Schedule for talk”
complete_data = “XP Conf 2015. Schedule for talk ”
FileWrite.append(FILE_NAME, append_data)
FileReader.read(FILE_NAME) == complete_data

38. TEST ORDER DEPENDENCY
3
8

39. INPUT / OUTPUT
3
9

40. 4
0

41. WHAT’S YOUR STRATEGY?
4
1

42. STOP CALLING YOUR BUILD
FLAKY!!
4
2

43. Identify
Flaky tests
Quarantine
PlanFix Tests
Monitor
4
3

44. IT’S NOT ALWAYS YOUR
TESTS
4
4

45. DO NOT GET
OVERWHELMED BY FAILING
TESTS
4
5

46. REMEMBER!
4
6

47. LOVE YOUR TEST CODE !
4
7

48. There is no such thing as
FLAKY TEST!
If its flaky, its not a test !
4
8

49. AND THE FUTURE…
4
9

50. REFERENCES
• An Empirical Analysis of Flaky tests
• Eradicating Non-Determinism in Tests – Martin
Fowler
• No more flaky tests on the GO team - Pavan
Sudarshan
• Top 12 reasons to write unit tests
• http://tech.just-eat.com/2015/03/30/how-to-fix-
flaky-tests/
5
0

51. THANK YOU
yahya.poonawala@gmail.com
priti.biyani6@gmail.com