This presentation discusses our build process and the awesome things that come from integrating a Continuous Testing methodology into your workflow.

1. iOS Continuous Testing
For fun and profit

2. Ideas, not tools

3. Ideas, not tools
Okay, some tools

4. Why test?

5. Why test?
1 in X “fixed” bugs are not really fixed

6. Why test?
1 in 4 “fixed” bugs are not really fixed

7. Why test?
1 in X “implemented” features break the build

8. Why test?
1 in 5 “implemented” features break the build

9. Why test?
Over ___% of all reopened issues are reopened more
than once

10. Why test?
Over half of all reopened issues are reopened more
than once

11. Why test?
__% of all reopened issues are reopened 3 or more
times

12. Why test?
25% of all reopened issues are reopened 3 or more
times

13. Unit Testing (SenTest)

14. iOS App Development Workflow Guide:
Unit Testing Applications

15. UI Automation is
AWFUL!

16. UI Automation is
AWFUL!
• I’ve personally filed over 15 bugs.

17. UI Automation is
AWFUL!
• I’ve personally filed over 15 bugs.
• All are still open.

18. UI Automation is
AWFUL!
• I’ve personally filed over 15 bugs.
• All are still open.
• 4.2 broke more than it fixed

19. UI Automation is
AWFUL!
• I’ve personally filed over 15 bugs.
• All are still open.
• 4.2 broke more than it fixed
• Continuing the trend of fail

20. Time spent since August
60
45
30
15
0
Instruments Bug Tracker Blogging HN Reddit RSS Email

21. Integration Testing (KIF)

28. Step 1

29. Step 1
• NSMutableArray *steps = [NSMutableArray array];

30. Step 1
• NSMutableArray *steps = [NSMutableArray array];
[steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create
Album"]];

31. Step 1
• NSMutableArray *steps = [NSMutableArray array];
[steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create
Album"]];
[steps addObject:[KIFTestStep
stepToWaitForViewWithAccessibilityLabel:@"createAlbumView"]];

32. Step 1
• NSMutableArray *steps = [NSMutableArray array];
[steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create
Album"]];
[steps addObject:[KIFTestStep
stepToWaitForViewWithAccessibilityLabel:@"createAlbumView"]];
•

33. Step 2

34. Step 2
[steps addObject:[KIFTestStep stepToEnterText:name
intoViewWithAccessibilityLabel:@"albumName"]];

35. Step 3

36. Step 3
if (primary) [steps addObject:[KIFTestStep
stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];

37. Step 3
if (primary) [steps addObject:[KIFTestStep
stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];
[steps addObject:[KIFTestStep
stepToTapViewWithAccessibilityLabel:@"createAlbumButton"]];

38. Step 3
if (primary) [steps addObject:[KIFTestStep
stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];
[steps addObject:[KIFTestStep
stepToTapViewWithAccessibilityLabel:@"createAlbumButton"]];
[steps addObject:[KIFTestStep
stepToWaitForAbsenceOfViewWithAccessibilityLabel:@"createAlbumVi
ew"]];

39. Step 4

40. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

41. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

42. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];

43. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;

44. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){

45. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){

46. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){
found = YES;

47. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){
found = YES;
}

48. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){
found = YES;
}
}

49. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){
found = YES;
}
}
KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);

50. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){
found = YES;
}
}
KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
return KIFTestStepResultSuccess;

51. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){
found = YES;
}
}
KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
return KIFTestStepResultSuccess;
}]];

52. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){
found = YES;
}
}
KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
return KIFTestStepResultSuccess;
}]];

53. Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Global *g = [Global shared];
BOOL found = NO;
for(Album *a in g.albums){
if ([a.name isEqualToString:name]){
found = YES;
}
}
KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
return KIFTestStepResultSuccess;
}]];
•

54. Step 5

55. Step 5
#if RUN_KIF_TESTS

56. Step 5
#if RUN_KIF_TESTS
[[PhotoWalletTestController sharedInstance]
startTestingWithCompletionBlock:^{

57. Step 5
#if RUN_KIF_TESTS
[[PhotoWalletTestController sharedInstance]
startTestingWithCompletionBlock:^{
exit([[PhotoWalletTestController sharedInstance]
failureCount]);

58. Step 5
#if RUN_KIF_TESTS
[[PhotoWalletTestController sharedInstance]
startTestingWithCompletionBlock:^{
exit([[PhotoWalletTestController sharedInstance]
failureCount]);
}];

59. Step 5
#if RUN_KIF_TESTS
[[PhotoWalletTestController sharedInstance]
startTestingWithCompletionBlock:^{
exit([[PhotoWalletTestController sharedInstance]
failureCount]);
}];
#endif

63. Keep It Functional
https://github.com/square/KIF

64. Why continuously test?

65. Why continuously test?

66. Why continuously test?
Don’t break the build!

67. Why continuously test?
Don’t hold up deployment!

68. Why continuously test?
Refactor with confidence!

69. Myth #1: I don’t have time
Variations
• I’m a one-man shop
• My iOS game is not mission-critical
• Testing takes time away from writing new
code

70. Bugs Tests

71. Bugs Tests
70
56
42
28
14
0
1.2 1.3 1.4 1.5 1.6 1.7

72. Continuous Testing
Bugs Tests
70
56
42
28
14
0
1.2 1.3 1.4 1.5 1.6 1.7

73. Continuous Testing
Bugs Tests
70
56
42
28
14
0
1.2 1.3 1.4 1.5 1.6 1.7

74. Ship Dates Over Time
•

75. Ship Dates Over Time
Continuous Testing
•

76. Ship Dates Over Time
Continuous Testing
•
Shipping 40 Days Sooner

77. Experimenting with
TDD

78. Experimenting with
TDD

79. Experimenting with
TDD

80. Experimenting with
TDD

81. Experimenting with
TDD

82. Experimenting with
TDD

83. Nobody likes to test
• We need something fun
• We need something easy
• We need something automatic
• We need something obviously beneficial

84. Hudson / Jenkins
Enterprise tools are not your friend

87. Enterprise tools are not your friend

88. Introducing buildbot
Testing made fun

89. GLaDOS is alive!
She sends mail

90. GLaDOS is alive!
She uses our bugtracker

91. GLaDOS is alive!
She taunts developers

92. GLaDOS is alive!
She merges things on GitHub

93. GLaDOS is alive!
She files bugs

94. buildbot in practice
Testing made easy

95. zero-friction testing

96. zero-friction testing
• Test by default

97. zero-friction testing
• Test by default
• Command-line support w/ work.py

98. zero-friction testing
• Test by default
• Command-line support w/ work.py
• One-click testing from the bug tracker

99. zero-friction testing
• Test by default
• Command-line support w/ work.py
• One-click testing from the bug tracker
• Human tester selected automagically to
code review every patch

100. zero-friction testing

101. zero-friction testing
• Errors

102. zero-friction testing
• Errors
• Warnings

103. zero-friction testing
• Errors
• Warnings
• Analyzer results

104. zero-friction testing
• Errors
• Warnings
• Analyzer results
• Command-U tests

105. zero-friction testing
• Errors
• Warnings
• Analyzer results
• Command-U tests
• Merge failures

106. zero-friction testing
• Errors
• Warnings
• Analyzer results
• Command-U tests
• Merge failures
• KIF UI integration tests
All before a reviewer looks at the patch

107. zero-friction testing
Unified error summary
Detailed log files

108. test once, test forever
• Reviewers only get sane patches
• Once you write a failing test, never
look at the patch again

109. Integration made easy
When the tests pass, GLaDOS merges the feature in
and closes the ticket.

110. AL
RN LY
TE ON
IN E
USDeployments made easy
Only one person knows
how to do this!

111. AL
RN LY
TE ON
IN E
USDeployments made easy

112. AL
RN LY
TE ON
IN E
USDeployments made easy

113. AL
RN LY
TE ON
IN E
USDeployments made easy

114. AL
RN LY
TE ON
IN E
USDeployments made easy

115. AL
RN LY
TE ON
IN E
USDeployments made easy

116. AL
RN LY
TE ON
IN E
USDeployments made easy

117. Setup

118. buildbot: the bad news
we’re the only user

119. buildbot: the bad news
• Large buy-in
• work.py - workflow
• git - version control
• FogBugz - bug tracking
• GitHub - source hosting

120. buildbot: the bad news
Lots of our defaults aren’t configurable

121. buildbot: the bad news
Hard to retrofit into your existing workflow

122. buildbot: the bad news
Lots of undocumented behavior

123. EN E
OP RC
S OUbuildbot
http://github.com/drewcrawford/buildbot

124. OK, now what?

125. OK, now what?
• Test.

126. OK, now what?
• Test.
• Test early, test often.

127. OK, now what?
• Test.
• Test early, test often.
• Test reasonably.

128. OK, now what?
• Test.
• Test early, test often.
• Test reasonably.
• Test automatically.

129. OK, now what?
• Test.
• Test early, test often.
• Test reasonably.
• Test automatically.
• Think about your workflow

130. • Tiny iOS Developer
• Mix of contracts &
products
• Many other dev tools
like buildbot

131. Drew Crawford
http://drewcrawfordapps.com
drew@drewcrawfordapps.com
http://sealedabstract.com