Successfully reported this slideshow.
iOS Continuous Testing       For fun and profit
Ideas, not tools
Ideas, not tools    Okay, some tools
Why test?
Why test?1 in X “fixed” bugs are not really fixed
Why test?1 in 4 “fixed” bugs are not really fixed
Why test?1 in X “implemented” features break the build
Why test?1 in 5 “implemented” features break the build
Why test?Over ___% of all reopened issues are reopened more                     than once
Why test?Over half of all reopened issues are reopened more                      than once
Why test?__% of all reopened issues are reopened 3 or more                       times
Why test?25% of all reopened issues are reopened 3 or more                       times
Unit Testing (SenTest)
iOS App Development Workflow Guide:         Unit Testing Applications
UI Automation is    AWFUL!
UI Automation is         AWFUL!• I’ve personally filed over 15 bugs.
UI Automation is         AWFUL!• I’ve personally filed over 15 bugs. • All are still open.
UI Automation is         AWFUL!• I’ve personally filed over 15 bugs. • All are still open.• 4.2 broke more than it fixed
UI Automation is         AWFUL!• I’ve personally filed over 15 bugs. • All are still open.• 4.2 broke more than it fixed • C...
Time spent since August60453015 0     Instruments Bug Tracker Blogging   HN   Reddit   RSS   Email
Integration Testing (KIF)
Step 1
Step 1•   NSMutableArray *steps = [NSMutableArray array];
Step 1  •   NSMutableArray *steps = [NSMutableArray array];  [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityL...
Step 1  •   NSMutableArray *steps = [NSMutableArray array];   [steps addObject:[KIFTestStep stepToTapViewWithAccessibility...
Step 1  •   NSMutableArray *steps = [NSMutableArray array];   [steps addObject:[KIFTestStep stepToTapViewWithAccessibility...
Step 2
Step 2[steps addObject:[KIFTestStep stepToEnterText:nameintoViewWithAccessibilityLabel:@"albumName"]];
Step 3
Step 3if (primary) [steps addObject:[KIFTestStepstepToTapViewWithAccessibilityLabel:@"albumPrimary"]];
Step 3  if (primary) [steps addObject:[KIFTestStep  stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];   [steps addObj...
Step 3  if (primary) [steps addObject:[KIFTestStep  stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];   [steps addObj...
Step 4
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTest...
Step 4    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"    executionBlock:^KIFTestStepResult...
Step 4    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"    executionBlock:^KIFTestStepResult...
Step 4    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"    executionBlock:^KIFTestStepResult...
Step 4    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"    executionBlock:^KIFTestStepResult...
Step 4    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"    executionBlock:^KIFTestStepResult...
Step 5
Step 5#if RUN_KIF_TESTS
Step 5    #if RUN_KIF_TESTS   [[PhotoWalletTestController sharedInstance]startTestingWithCompletionBlock:^{
Step 5    #if RUN_KIF_TESTS   [[PhotoWalletTestController sharedInstance]startTestingWithCompletionBlock:^{      exit([[Ph...
Step 5    #if RUN_KIF_TESTS   [[PhotoWalletTestController sharedInstance]startTestingWithCompletionBlock:^{       exit([[P...
Step 5    #if RUN_KIF_TESTS   [[PhotoWalletTestController sharedInstance]startTestingWithCompletionBlock:^{       exit([[P...
Keep It Functionalhttps://github.com/square/KIF
Why continuously test?
Why continuously test?
Why continuously test?      Don’t break the build!
Why continuously test?     Don’t hold up deployment!
Why continuously test?     Refactor with confidence!
Myth #1: I don’t have time                Variations• I’m a one-man shop• My iOS game is not mission-critical• Testing tak...
Bugs   Tests
Bugs               Tests7056422814 0     1.2   1.3     1.4   1.5           1.6   1.7
Continuous Testing            Bugs                  Tests7056422814 0     1.2   1.3       1.4    1.5           1.6   1.7
Continuous Testing            Bugs                  Tests7056422814 0     1.2   1.3       1.4    1.5           1.6   1.7
Ship Dates Over Time•
Ship Dates Over Time       Continuous Testing•
Ship Dates Over Time          Continuous Testing•     Shipping 40 Days Sooner
Experimenting with      TDD
Experimenting with      TDD
Experimenting with      TDD
Experimenting with      TDD
Experimenting with      TDD
Experimenting with      TDD
Nobody likes to test• We need something fun• We need something easy• We need something automatic• We need something obviou...
Hudson / JenkinsEnterprise tools are not your friend
Enterprise tools are not your friend
Introducing buildbot      Testing made fun
GLaDOS is alive!     She sends mail
GLaDOS is alive!   She uses our bugtracker
GLaDOS is alive!   She taunts developers
GLaDOS is alive!  She merges things on GitHub
GLaDOS is alive!     She files bugs
buildbot in practice      Testing made easy
zero-friction testing
zero-friction testing• Test by default
zero-friction testing• Test by default• Command-line support w/ work.py
zero-friction testing• Test by default• Command-line support w/ work.py• One-click testing from the bug tracker
zero-friction testing• Test by default• Command-line support w/ work.py• One-click testing from the bug tracker• Human tes...
zero-friction testing
zero-friction testing• Errors
zero-friction testing• Errors• Warnings
zero-friction testing• Errors• Warnings• Analyzer results
zero-friction testing• Errors• Warnings• Analyzer results• Command-U tests
zero-friction testing• Errors• Warnings• Analyzer results• Command-U tests• Merge failures
zero-friction testing  • Errors  • Warnings  • Analyzer results  • Command-U tests  • Merge failures  • KIF UI integration...
zero-friction testingUnified error summary  Detailed log files
test once, test forever • Reviewers only get sane patches • Once you write a failing test, never   look at the patch again
Integration made easyWhen the tests pass, GLaDOS merges the feature in              and closes the ticket.
AL     RN LY  TE ONIN E  USDeployments made easy                 Only one person knows                     how to do this!
AL     RN LY  TE ONIN E  USDeployments made easy
AL     RN LY  TE ONIN E  USDeployments made easy
AL     RN LY  TE ONIN E  USDeployments made easy
AL     RN LY  TE ONIN E  USDeployments made easy
AL     RN LY  TE ONIN E  USDeployments made easy
AL     RN LY  TE ONIN E  USDeployments made easy
Setup
buildbot: the bad news     we’re the only user
buildbot: the bad news • Large buy-in  • work.py - workflow  • git - version control  • FogBugz - bug tracking  • GitHub - ...
buildbot: the bad news  Lots of our defaults aren’t configurable
buildbot: the bad news Hard to retrofit into your existing workflow
buildbot: the bad news    Lots of undocumented behavior
EN E        OP RC        S OUbuildbothttp://github.com/drewcrawford/buildbot
OK, now what?
OK, now what?• Test.
OK, now what?• Test.• Test early, test often.
OK, now what?• Test.• Test early, test often.• Test reasonably.
OK, now what?• Test.• Test early, test often.• Test reasonably.• Test automatically.
OK, now what?• Test.• Test early, test often.• Test reasonably.• Test automatically.• Think about your workflow
•   Tiny iOS Developer•   Mix of contracts &    products•   Many other dev tools    like buildbot
Drew Crawfordhttp://drewcrawfordapps.comdrew@drewcrawfordapps.comhttp://sealedabstract.com
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
Upcoming SlideShare
Loading in …5
×

iOS Continuous Testing

3,017 views

Published on

This presentation discusses our build process and the awesome things that come from integrating a Continuous Testing methodology into your workflow.

Published in: Technology, Business
  • Be the first to comment

iOS Continuous Testing

  1. 1. iOS Continuous Testing For fun and profit
  2. 2. Ideas, not tools
  3. 3. Ideas, not tools Okay, some tools
  4. 4. Why test?
  5. 5. Why test?1 in X “fixed” bugs are not really fixed
  6. 6. Why test?1 in 4 “fixed” bugs are not really fixed
  7. 7. Why test?1 in X “implemented” features break the build
  8. 8. Why test?1 in 5 “implemented” features break the build
  9. 9. Why test?Over ___% of all reopened issues are reopened more than once
  10. 10. Why test?Over half of all reopened issues are reopened more than once
  11. 11. Why test?__% of all reopened issues are reopened 3 or more times
  12. 12. Why test?25% of all reopened issues are reopened 3 or more times
  13. 13. Unit Testing (SenTest)
  14. 14. iOS App Development Workflow Guide: Unit Testing Applications
  15. 15. UI Automation is AWFUL!
  16. 16. UI Automation is AWFUL!• I’ve personally filed over 15 bugs.
  17. 17. UI Automation is AWFUL!• I’ve personally filed over 15 bugs. • All are still open.
  18. 18. UI Automation is AWFUL!• I’ve personally filed over 15 bugs. • All are still open.• 4.2 broke more than it fixed
  19. 19. UI Automation is AWFUL!• I’ve personally filed over 15 bugs. • All are still open.• 4.2 broke more than it fixed • Continuing the trend of fail
  20. 20. Time spent since August60453015 0 Instruments Bug Tracker Blogging HN Reddit RSS Email
  21. 21. Integration Testing (KIF)
  22. 22. Step 1
  23. 23. Step 1• NSMutableArray *steps = [NSMutableArray array];
  24. 24. Step 1 • NSMutableArray *steps = [NSMutableArray array]; [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"CreateAlbum"]];
  25. 25. Step 1 • NSMutableArray *steps = [NSMutableArray array]; [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"CreateAlbum"]]; [steps addObject:[KIFTestStepstepToWaitForViewWithAccessibilityLabel:@"createAlbumView"]];
  26. 26. Step 1 • NSMutableArray *steps = [NSMutableArray array]; [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"CreateAlbum"]]; [steps addObject:[KIFTestStepstepToWaitForViewWithAccessibilityLabel:@"createAlbumView"]]; •
  27. 27. Step 2
  28. 28. Step 2[steps addObject:[KIFTestStep stepToEnterText:nameintoViewWithAccessibilityLabel:@"albumName"]];
  29. 29. Step 3
  30. 30. Step 3if (primary) [steps addObject:[KIFTestStepstepToTapViewWithAccessibilityLabel:@"albumPrimary"]];
  31. 31. Step 3 if (primary) [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"albumPrimary"]]; [steps addObject:[KIFTestStepstepToTapViewWithAccessibilityLabel:@"createAlbumButton"]];
  32. 32. Step 3 if (primary) [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"albumPrimary"]]; [steps addObject:[KIFTestStepstepToTapViewWithAccessibilityLabel:@"createAlbumButton"]]; [steps addObject:[KIFTestStepstepToWaitForAbsenceOfViewWithAccessibilityLabel:@"createAlbumView"]];
  33. 33. Step 4
  34. 34. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
  35. 35. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
  36. 36. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared];
  37. 37. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO;
  38. 38. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){
  39. 39. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){
  40. 40. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES;
  41. 41. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; }
  42. 42. Step 4[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } }
  43. 43. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requestedalbum %@",g.albums]);
  44. 44. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requestedalbum %@",g.albums]); return KIFTestStepResultSuccess;
  45. 45. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requestedalbum %@",g.albums]); return KIFTestStepResultSuccess; }]];
  46. 46. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requestedalbum %@",g.albums]); return KIFTestStepResultSuccess; }]];
  47. 47. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requestedalbum %@",g.albums]); return KIFTestStepResultSuccess; }]]; •
  48. 48. Step 5
  49. 49. Step 5#if RUN_KIF_TESTS
  50. 50. Step 5 #if RUN_KIF_TESTS [[PhotoWalletTestController sharedInstance]startTestingWithCompletionBlock:^{
  51. 51. Step 5 #if RUN_KIF_TESTS [[PhotoWalletTestController sharedInstance]startTestingWithCompletionBlock:^{ exit([[PhotoWalletTestController sharedInstance]failureCount]);
  52. 52. Step 5 #if RUN_KIF_TESTS [[PhotoWalletTestController sharedInstance]startTestingWithCompletionBlock:^{ exit([[PhotoWalletTestController sharedInstance]failureCount]); }];
  53. 53. Step 5 #if RUN_KIF_TESTS [[PhotoWalletTestController sharedInstance]startTestingWithCompletionBlock:^{ exit([[PhotoWalletTestController sharedInstance]failureCount]); }];#endif
  54. 54. Keep It Functionalhttps://github.com/square/KIF
  55. 55. Why continuously test?
  56. 56. Why continuously test?
  57. 57. Why continuously test? Don’t break the build!
  58. 58. Why continuously test? Don’t hold up deployment!
  59. 59. Why continuously test? Refactor with confidence!
  60. 60. Myth #1: I don’t have time Variations• I’m a one-man shop• My iOS game is not mission-critical• Testing takes time away from writing new code
  61. 61. Bugs Tests
  62. 62. Bugs Tests7056422814 0 1.2 1.3 1.4 1.5 1.6 1.7
  63. 63. Continuous Testing Bugs Tests7056422814 0 1.2 1.3 1.4 1.5 1.6 1.7
  64. 64. Continuous Testing Bugs Tests7056422814 0 1.2 1.3 1.4 1.5 1.6 1.7
  65. 65. Ship Dates Over Time•
  66. 66. Ship Dates Over Time Continuous Testing•
  67. 67. Ship Dates Over Time Continuous Testing• Shipping 40 Days Sooner
  68. 68. Experimenting with TDD
  69. 69. Experimenting with TDD
  70. 70. Experimenting with TDD
  71. 71. Experimenting with TDD
  72. 72. Experimenting with TDD
  73. 73. Experimenting with TDD
  74. 74. Nobody likes to test• We need something fun• We need something easy• We need something automatic• We need something obviously beneficial
  75. 75. Hudson / JenkinsEnterprise tools are not your friend
  76. 76. Enterprise tools are not your friend
  77. 77. Introducing buildbot Testing made fun
  78. 78. GLaDOS is alive! She sends mail
  79. 79. GLaDOS is alive! She uses our bugtracker
  80. 80. GLaDOS is alive! She taunts developers
  81. 81. GLaDOS is alive! She merges things on GitHub
  82. 82. GLaDOS is alive! She files bugs
  83. 83. buildbot in practice Testing made easy
  84. 84. zero-friction testing
  85. 85. zero-friction testing• Test by default
  86. 86. zero-friction testing• Test by default• Command-line support w/ work.py
  87. 87. zero-friction testing• Test by default• Command-line support w/ work.py• One-click testing from the bug tracker
  88. 88. zero-friction testing• Test by default• Command-line support w/ work.py• One-click testing from the bug tracker• Human tester selected automagically to code review every patch
  89. 89. zero-friction testing
  90. 90. zero-friction testing• Errors
  91. 91. zero-friction testing• Errors• Warnings
  92. 92. zero-friction testing• Errors• Warnings• Analyzer results
  93. 93. zero-friction testing• Errors• Warnings• Analyzer results• Command-U tests
  94. 94. zero-friction testing• Errors• Warnings• Analyzer results• Command-U tests• Merge failures
  95. 95. zero-friction testing • Errors • Warnings • Analyzer results • Command-U tests • Merge failures • KIF UI integration testsAll before a reviewer looks at the patch
  96. 96. zero-friction testingUnified error summary Detailed log files
  97. 97. test once, test forever • Reviewers only get sane patches • Once you write a failing test, never look at the patch again
  98. 98. Integration made easyWhen the tests pass, GLaDOS merges the feature in and closes the ticket.
  99. 99. AL RN LY TE ONIN E USDeployments made easy Only one person knows how to do this!
  100. 100. AL RN LY TE ONIN E USDeployments made easy
  101. 101. AL RN LY TE ONIN E USDeployments made easy
  102. 102. AL RN LY TE ONIN E USDeployments made easy
  103. 103. AL RN LY TE ONIN E USDeployments made easy
  104. 104. AL RN LY TE ONIN E USDeployments made easy
  105. 105. AL RN LY TE ONIN E USDeployments made easy
  106. 106. Setup
  107. 107. buildbot: the bad news we’re the only user
  108. 108. buildbot: the bad news • Large buy-in • work.py - workflow • git - version control • FogBugz - bug tracking • GitHub - source hosting
  109. 109. buildbot: the bad news Lots of our defaults aren’t configurable
  110. 110. buildbot: the bad news Hard to retrofit into your existing workflow
  111. 111. buildbot: the bad news Lots of undocumented behavior
  112. 112. EN E OP RC S OUbuildbothttp://github.com/drewcrawford/buildbot
  113. 113. OK, now what?
  114. 114. OK, now what?• Test.
  115. 115. OK, now what?• Test.• Test early, test often.
  116. 116. OK, now what?• Test.• Test early, test often.• Test reasonably.
  117. 117. OK, now what?• Test.• Test early, test often.• Test reasonably.• Test automatically.
  118. 118. OK, now what?• Test.• Test early, test often.• Test reasonably.• Test automatically.• Think about your workflow
  119. 119. • Tiny iOS Developer• Mix of contracts & products• Many other dev tools like buildbot
  120. 120. Drew Crawfordhttp://drewcrawfordapps.comdrew@drewcrawfordapps.comhttp://sealedabstract.com

×