SlideShare a Scribd company logo
iOS Continuous Testing
       For fun and profit
Ideas, not tools
Ideas, not tools
    Okay, some tools
Why test?
Why test?
1 in X “fixed” bugs are not really fixed
Why test?
1 in 4 “fixed” bugs are not really fixed
Why test?
1 in X “implemented” features break the build
Why test?
1 in 5 “implemented” features break the build
Why test?
Over ___% of all reopened issues are reopened more
                     than once
Why test?
Over half of all reopened issues are reopened more
                      than once
Why test?
__% of all reopened issues are reopened 3 or more
                       times
Why test?
25% of all reopened issues are reopened 3 or more
                       times
Unit Testing (SenTest)
iOS App Development Workflow Guide:
         Unit Testing Applications
UI Automation is
    AWFUL!
UI Automation is
         AWFUL!
• I’ve personally filed over 15 bugs.
UI Automation is
         AWFUL!
• I’ve personally filed over 15 bugs.
 • All are still open.
UI Automation is
         AWFUL!
• I’ve personally filed over 15 bugs.
 • All are still open.
• 4.2 broke more than it fixed
UI Automation is
         AWFUL!
• I’ve personally filed over 15 bugs.
 • All are still open.
• 4.2 broke more than it fixed
 • Continuing the trend of fail
Time spent since August

60

45

30

15

 0
     Instruments Bug Tracker Blogging   HN   Reddit   RSS   Email
Integration Testing (KIF)
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
Step 1
Step 1

•   NSMutableArray *steps = [NSMutableArray array];
Step 1

  •   NSMutableArray *steps = [NSMutableArray array];

  [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create
Album"]];
Step 1

  •   NSMutableArray *steps = [NSMutableArray array];

   [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create
Album"]];
   [steps addObject:[KIFTestStep
stepToWaitForViewWithAccessibilityLabel:@"createAlbumView"]];
Step 1

  •   NSMutableArray *steps = [NSMutableArray array];

   [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create
Album"]];
   [steps addObject:[KIFTestStep
stepToWaitForViewWithAccessibilityLabel:@"createAlbumView"]];

  •
Step 2
Step 2


[steps addObject:[KIFTestStep stepToEnterText:name
intoViewWithAccessibilityLabel:@"albumName"]];
Step 3
Step 3

if (primary) [steps addObject:[KIFTestStep
stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];
Step 3

  if (primary) [steps addObject:[KIFTestStep
  stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];

   [steps addObject:[KIFTestStep
stepToTapViewWithAccessibilityLabel:@"createAlbumButton"]];
Step 3

  if (primary) [steps addObject:[KIFTestStep
  stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];

   [steps addObject:[KIFTestStep
stepToTapViewWithAccessibilityLabel:@"createAlbumButton"]];

   [steps addObject:[KIFTestStep
stepToWaitForAbsenceOfViewWithAccessibilityLabel:@"createAlbumVi
ew"]];
Step 4
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

 Global *g = [Global shared];
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

 Global *g = [Global shared];
 BOOL found = NO;
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

 Global *g = [Global shared];
 BOOL found = NO;
 for(Album *a in g.albums){
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

 Global *g = [Global shared];
 BOOL found = NO;
 for(Album *a in g.albums){
    if ([a.name isEqualToString:name]){
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

 Global *g = [Global shared];
 BOOL found = NO;
 for(Album *a in g.albums){
    if ([a.name isEqualToString:name]){
        found = YES;
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

 Global *g = [Global shared];
 BOOL found = NO;
 for(Album *a in g.albums){
    if ([a.name isEqualToString:name]){
        found = YES;
    }
Step 4
[steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

 Global *g = [Global shared];
 BOOL found = NO;
 for(Album *a in g.albums){
    if ([a.name isEqualToString:name]){
        found = YES;
    }
 }
Step 4
    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
    executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

    Global *g = [Global shared];
    BOOL found = NO;
    for(Album *a in g.albums){
       if ([a.name isEqualToString:name]){
           found = YES;
       }
    }
    KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
Step 4
    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
    executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

    Global *g = [Global shared];
    BOOL found = NO;
    for(Album *a in g.albums){
       if ([a.name isEqualToString:name]){
           found = YES;
       }
    }
    KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
    return KIFTestStepResultSuccess;
Step 4
    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
    executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

       Global *g = [Global shared];
       BOOL found = NO;
       for(Album *a in g.albums){
          if ([a.name isEqualToString:name]){
              found = YES;
          }
       }
      KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
       return KIFTestStepResultSuccess;
   }]];
Step 4
    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
    executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

       Global *g = [Global shared];
       BOOL found = NO;
       for(Album *a in g.albums){
          if ([a.name isEqualToString:name]){
              found = YES;
          }
       }
      KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
       return KIFTestStepResultSuccess;
   }]];
Step 4
    [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation"
    executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {

       Global *g = [Global shared];
       BOOL found = NO;
       for(Album *a in g.albums){
          if ([a.name isEqualToString:name]){
              found = YES;
          }
       }
      KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested
album %@",g.albums]);
       return KIFTestStepResultSuccess;
   }]];


  •
Step 5
Step 5

#if RUN_KIF_TESTS
Step 5

    #if RUN_KIF_TESTS
   [[PhotoWalletTestController sharedInstance]
startTestingWithCompletionBlock:^{
Step 5

    #if RUN_KIF_TESTS
   [[PhotoWalletTestController sharedInstance]
startTestingWithCompletionBlock:^{
      exit([[PhotoWalletTestController sharedInstance]
failureCount]);
Step 5

    #if RUN_KIF_TESTS
   [[PhotoWalletTestController sharedInstance]
startTestingWithCompletionBlock:^{
       exit([[PhotoWalletTestController sharedInstance]
failureCount]);
   }];
Step 5

    #if RUN_KIF_TESTS
   [[PhotoWalletTestController sharedInstance]
startTestingWithCompletionBlock:^{
       exit([[PhotoWalletTestController sharedInstance]
failureCount]);
   }];
#endif
iOS Continuous Testing
iOS Continuous Testing
iOS Continuous Testing
Keep It Functional



https://github.com/square/KIF
Why continuously test?
Why continuously test?
Why continuously test?
      Don’t break the build!
Why continuously test?
     Don’t hold up deployment!
Why continuously test?
     Refactor with confidence!
Myth #1: I don’t have time
                Variations



• I’m a one-man shop
• My iOS game is not mission-critical
• Testing takes time away from writing new
  code
Bugs   Tests
Bugs               Tests
70

56

42

28

14

 0
     1.2   1.3     1.4   1.5           1.6   1.7
Continuous Testing
            Bugs                  Tests
70

56

42

28

14

 0
     1.2   1.3       1.4    1.5           1.6   1.7
Continuous Testing
            Bugs                  Tests
70

56

42

28

14

 0
     1.2   1.3       1.4    1.5           1.6   1.7
Ship Dates Over Time


•
Ship Dates Over Time
       Continuous Testing



•
Ship Dates Over Time
          Continuous Testing



•

     Shipping 40 Days Sooner
Experimenting with
      TDD
Experimenting with
      TDD
Experimenting with
      TDD
Experimenting with
      TDD
Experimenting with
      TDD
Experimenting with
      TDD
Nobody likes to test

• We need something fun
• We need something easy
• We need something automatic
• We need something obviously beneficial
Hudson / Jenkins
Enterprise tools are not your friend
iOS Continuous Testing
iOS Continuous Testing
Enterprise tools are not your friend
Introducing buildbot
      Testing made fun
GLaDOS is alive!
     She sends mail
GLaDOS is alive!
   She uses our bugtracker
GLaDOS is alive!
   She taunts developers
GLaDOS is alive!
  She merges things on GitHub
GLaDOS is alive!
     She files bugs
buildbot in practice
      Testing made easy
zero-friction testing
zero-friction testing

• Test by default
zero-friction testing

• Test by default
• Command-line support w/ work.py
zero-friction testing

• Test by default
• Command-line support w/ work.py
• One-click testing from the bug tracker
zero-friction testing

• Test by default
• Command-line support w/ work.py
• One-click testing from the bug tracker
• Human tester selected automagically to
  code review every patch
zero-friction testing
zero-friction testing
• Errors
zero-friction testing
• Errors
• Warnings
zero-friction testing
• Errors
• Warnings
• Analyzer results
zero-friction testing
• Errors
• Warnings
• Analyzer results
• Command-U tests
zero-friction testing
• Errors
• Warnings
• Analyzer results
• Command-U tests
• Merge failures
zero-friction testing
  • Errors
  • Warnings
  • Analyzer results
  • Command-U tests
  • Merge failures
  • KIF UI integration tests
All before a reviewer looks at the patch
zero-friction testing
Unified error summary




  Detailed log files
test once, test forever

 • Reviewers only get sane patches
 • Once you write a failing test, never
   look at the patch again
Integration made easy
When the tests pass, GLaDOS merges the feature in
              and closes the ticket.
AL
     RN LY
  TE ON
IN E
  USDeployments made easy




                 Only one person knows
                     how to do this!
AL
     RN LY
  TE ON
IN E
  USDeployments made easy
AL
     RN LY
  TE ON
IN E
  USDeployments made easy
AL
     RN LY
  TE ON
IN E
  USDeployments made easy
AL
     RN LY
  TE ON
IN E
  USDeployments made easy
AL
     RN LY
  TE ON
IN E
  USDeployments made easy
AL
     RN LY
  TE ON
IN E
  USDeployments made easy
Setup
buildbot: the bad news
     we’re the only user
buildbot: the bad news

 • Large buy-in
  • work.py - workflow
  • git - version control
  • FogBugz - bug tracking
  • GitHub - source hosting
buildbot: the bad news
  Lots of our defaults aren’t configurable
buildbot: the bad news
 Hard to retrofit into your existing workflow
buildbot: the bad news
    Lots of undocumented behavior
EN E
        OP RC
        S OUbuildbot
http://github.com/drewcrawford/buildbot
OK, now what?
OK, now what?

• Test.
OK, now what?

• Test.
• Test early, test often.
OK, now what?

• Test.
• Test early, test often.
• Test reasonably.
OK, now what?

• Test.
• Test early, test often.
• Test reasonably.
• Test automatically.
OK, now what?

• Test.
• Test early, test often.
• Test reasonably.
• Test automatically.
• Think about your workflow
•   Tiny iOS Developer

•   Mix of contracts &
    products

•   Many other dev tools
    like buildbot
Drew Crawford
http://drewcrawfordapps.com
drew@drewcrawfordapps.com


http://sealedabstract.com

More Related Content

Similar to iOS Continuous Testing

Sustainable TDD
Sustainable TDDSustainable TDD
Sustainable TDD
Steven Mak
 
Gojko Adzic Cucumber
Gojko Adzic CucumberGojko Adzic Cucumber
Gojko Adzic Cucumber
Skills Matter
 
GoCracow #5 Bartlomiej klimczak - GoBDD
GoCracow #5 Bartlomiej klimczak - GoBDDGoCracow #5 Bartlomiej klimczak - GoBDD
GoCracow #5 Bartlomiej klimczak - GoBDD
Bartłomiej Kiełbasa
 
Test legacy apps with Behat
Test legacy apps with BehatTest legacy apps with Behat
Test legacy apps with Behat
agpavlakis
 
Functions for nothing, and your tests for free
Functions for nothing, and your tests for freeFunctions for nothing, and your tests for free
Functions for nothing, and your tests for free
George Pollard
 
Dev Day 2024: Jonathan Frere - Playwright: Das Beste aus dem Dramatiker herau...
Dev Day 2024: Jonathan Frere - Playwright: Das Beste aus dem Dramatiker herau...Dev Day 2024: Jonathan Frere - Playwright: Das Beste aus dem Dramatiker herau...
Dev Day 2024: Jonathan Frere - Playwright: Das Beste aus dem Dramatiker herau...
emmaberlin1
 
Refactoring In Tdd The Missing Part
Refactoring In Tdd The Missing PartRefactoring In Tdd The Missing Part
Refactoring In Tdd The Missing Part
Gabriele Lana
 
Objective C 基本介紹
Objective C 基本介紹Objective C 基本介紹
Objective C 基本介紹
Giga Cheng
 
Property Based Testing in PHP
Property Based Testing in PHPProperty Based Testing in PHP
Property Based Testing in PHP
vinaikopp
 
Test Driven Development with JavaFX
Test Driven Development with JavaFXTest Driven Development with JavaFX
Test Driven Development with JavaFX
Hendrik Ebbers
 
Unit Testing Lots of Perl
Unit Testing Lots of PerlUnit Testing Lots of Perl
Unit Testing Lots of Perl
Workhorse Computing
 

Similar to iOS Continuous Testing (11)

Sustainable TDD
Sustainable TDDSustainable TDD
Sustainable TDD
 
Gojko Adzic Cucumber
Gojko Adzic CucumberGojko Adzic Cucumber
Gojko Adzic Cucumber
 
GoCracow #5 Bartlomiej klimczak - GoBDD
GoCracow #5 Bartlomiej klimczak - GoBDDGoCracow #5 Bartlomiej klimczak - GoBDD
GoCracow #5 Bartlomiej klimczak - GoBDD
 
Test legacy apps with Behat
Test legacy apps with BehatTest legacy apps with Behat
Test legacy apps with Behat
 
Functions for nothing, and your tests for free
Functions for nothing, and your tests for freeFunctions for nothing, and your tests for free
Functions for nothing, and your tests for free
 
Dev Day 2024: Jonathan Frere - Playwright: Das Beste aus dem Dramatiker herau...
Dev Day 2024: Jonathan Frere - Playwright: Das Beste aus dem Dramatiker herau...Dev Day 2024: Jonathan Frere - Playwright: Das Beste aus dem Dramatiker herau...
Dev Day 2024: Jonathan Frere - Playwright: Das Beste aus dem Dramatiker herau...
 
Refactoring In Tdd The Missing Part
Refactoring In Tdd The Missing PartRefactoring In Tdd The Missing Part
Refactoring In Tdd The Missing Part
 
Objective C 基本介紹
Objective C 基本介紹Objective C 基本介紹
Objective C 基本介紹
 
Property Based Testing in PHP
Property Based Testing in PHPProperty Based Testing in PHP
Property Based Testing in PHP
 
Test Driven Development with JavaFX
Test Driven Development with JavaFXTest Driven Development with JavaFX
Test Driven Development with JavaFX
 
Unit Testing Lots of Perl
Unit Testing Lots of PerlUnit Testing Lots of Perl
Unit Testing Lots of Perl
 

Recently uploaded

The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
Zilliz
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
ZachWylie3
 
kk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdfkk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdf
KIRAN KV
 
Gen AI: Privacy Risks of Large Language Models (LLMs)
Gen AI: Privacy Risks of Large Language Models (LLMs)Gen AI: Privacy Risks of Large Language Models (LLMs)
Gen AI: Privacy Risks of Large Language Models (LLMs)
Debmalya Biswas
 
It's your unstructured data: How to get your GenAI app to production (and spe...
It's your unstructured data: How to get your GenAI app to production (and spe...It's your unstructured data: How to get your GenAI app to production (and spe...
It's your unstructured data: How to get your GenAI app to production (and spe...
Zilliz
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
FIDO Alliance
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
DianaGray10
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
SelfMade bd
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
Step-By-Step Process to Develop a Mobile App From Scratch
Step-By-Step Process to Develop a Mobile App From ScratchStep-By-Step Process to Develop a Mobile App From Scratch
Step-By-Step Process to Develop a Mobile App From Scratch
softsuave
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 

Recently uploaded (20)

The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
 
kk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdfkk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdf
 
Gen AI: Privacy Risks of Large Language Models (LLMs)
Gen AI: Privacy Risks of Large Language Models (LLMs)Gen AI: Privacy Risks of Large Language Models (LLMs)
Gen AI: Privacy Risks of Large Language Models (LLMs)
 
It's your unstructured data: How to get your GenAI app to production (and spe...
It's your unstructured data: How to get your GenAI app to production (and spe...It's your unstructured data: How to get your GenAI app to production (and spe...
It's your unstructured data: How to get your GenAI app to production (and spe...
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
Step-By-Step Process to Develop a Mobile App From Scratch
Step-By-Step Process to Develop a Mobile App From ScratchStep-By-Step Process to Develop a Mobile App From Scratch
Step-By-Step Process to Develop a Mobile App From Scratch
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 

iOS Continuous Testing

  • 1. iOS Continuous Testing For fun and profit
  • 3. Ideas, not tools Okay, some tools
  • 5. Why test? 1 in X “fixed” bugs are not really fixed
  • 6. Why test? 1 in 4 “fixed” bugs are not really fixed
  • 7. Why test? 1 in X “implemented” features break the build
  • 8. Why test? 1 in 5 “implemented” features break the build
  • 9. Why test? Over ___% of all reopened issues are reopened more than once
  • 10. Why test? Over half of all reopened issues are reopened more than once
  • 11. Why test? __% of all reopened issues are reopened 3 or more times
  • 12. Why test? 25% of all reopened issues are reopened 3 or more times
  • 14. iOS App Development Workflow Guide: Unit Testing Applications
  • 16. UI Automation is AWFUL! • I’ve personally filed over 15 bugs.
  • 17. UI Automation is AWFUL! • I’ve personally filed over 15 bugs. • All are still open.
  • 18. UI Automation is AWFUL! • I’ve personally filed over 15 bugs. • All are still open. • 4.2 broke more than it fixed
  • 19. UI Automation is AWFUL! • I’ve personally filed over 15 bugs. • All are still open. • 4.2 broke more than it fixed • Continuing the trend of fail
  • 20. Time spent since August 60 45 30 15 0 Instruments Bug Tracker Blogging HN Reddit RSS Email
  • 29. Step 1 • NSMutableArray *steps = [NSMutableArray array];
  • 30. Step 1 • NSMutableArray *steps = [NSMutableArray array]; [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create Album"]];
  • 31. Step 1 • NSMutableArray *steps = [NSMutableArray array]; [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create Album"]]; [steps addObject:[KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"createAlbumView"]];
  • 32. Step 1 • NSMutableArray *steps = [NSMutableArray array]; [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Create Album"]]; [steps addObject:[KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"createAlbumView"]]; •
  • 34. Step 2 [steps addObject:[KIFTestStep stepToEnterText:name intoViewWithAccessibilityLabel:@"albumName"]];
  • 36. Step 3 if (primary) [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"albumPrimary"]];
  • 37. Step 3 if (primary) [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"albumPrimary"]]; [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"createAlbumButton"]];
  • 38. Step 3 if (primary) [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"albumPrimary"]]; [steps addObject:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"createAlbumButton"]]; [steps addObject:[KIFTestStep stepToWaitForAbsenceOfViewWithAccessibilityLabel:@"createAlbumVi ew"]];
  • 40. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
  • 41. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) {
  • 42. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared];
  • 43. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO;
  • 44. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){
  • 45. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){
  • 46. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES;
  • 47. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; }
  • 48. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } }
  • 49. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested album %@",g.albums]);
  • 50. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested album %@",g.albums]); return KIFTestStepResultSuccess;
  • 51. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested album %@",g.albums]); return KIFTestStepResultSuccess; }]];
  • 52. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested album %@",g.albums]); return KIFTestStepResultSuccess; }]];
  • 53. Step 4 [steps addObject:[KIFTestStep stepWithDescription:@"Testing album creation" executionBlock:^KIFTestStepResult(KIFTestStep *step, NSError *__autoreleasing *error) { Global *g = [Global shared]; BOOL found = NO; for(Album *a in g.albums){ if ([a.name isEqualToString:name]){ found = YES; } } KIFTestCondition(found, error, [NSString stringWithFormat:@"Failed to create requested album %@",g.albums]); return KIFTestStepResultSuccess; }]]; •
  • 56. Step 5 #if RUN_KIF_TESTS [[PhotoWalletTestController sharedInstance] startTestingWithCompletionBlock:^{
  • 57. Step 5 #if RUN_KIF_TESTS [[PhotoWalletTestController sharedInstance] startTestingWithCompletionBlock:^{ exit([[PhotoWalletTestController sharedInstance] failureCount]);
  • 58. Step 5 #if RUN_KIF_TESTS [[PhotoWalletTestController sharedInstance] startTestingWithCompletionBlock:^{ exit([[PhotoWalletTestController sharedInstance] failureCount]); }];
  • 59. Step 5 #if RUN_KIF_TESTS [[PhotoWalletTestController sharedInstance] startTestingWithCompletionBlock:^{ exit([[PhotoWalletTestController sharedInstance] failureCount]); }]; #endif
  • 66. Why continuously test? Don’t break the build!
  • 67. Why continuously test? Don’t hold up deployment!
  • 68. Why continuously test? Refactor with confidence!
  • 69. Myth #1: I don’t have time Variations • I’m a one-man shop • My iOS game is not mission-critical • Testing takes time away from writing new code
  • 70. Bugs Tests
  • 71. Bugs Tests 70 56 42 28 14 0 1.2 1.3 1.4 1.5 1.6 1.7
  • 72. Continuous Testing Bugs Tests 70 56 42 28 14 0 1.2 1.3 1.4 1.5 1.6 1.7
  • 73. Continuous Testing Bugs Tests 70 56 42 28 14 0 1.2 1.3 1.4 1.5 1.6 1.7
  • 74. Ship Dates Over Time •
  • 75. Ship Dates Over Time Continuous Testing •
  • 76. Ship Dates Over Time Continuous Testing • Shipping 40 Days Sooner
  • 83. Nobody likes to test • We need something fun • We need something easy • We need something automatic • We need something obviously beneficial
  • 84. Hudson / Jenkins Enterprise tools are not your friend
  • 87. Enterprise tools are not your friend
  • 88. Introducing buildbot Testing made fun
  • 89. GLaDOS is alive! She sends mail
  • 90. GLaDOS is alive! She uses our bugtracker
  • 91. GLaDOS is alive! She taunts developers
  • 92. GLaDOS is alive! She merges things on GitHub
  • 93. GLaDOS is alive! She files bugs
  • 94. buildbot in practice Testing made easy
  • 97. zero-friction testing • Test by default • Command-line support w/ work.py
  • 98. zero-friction testing • Test by default • Command-line support w/ work.py • One-click testing from the bug tracker
  • 99. zero-friction testing • Test by default • Command-line support w/ work.py • One-click testing from the bug tracker • Human tester selected automagically to code review every patch
  • 103. zero-friction testing • Errors • Warnings • Analyzer results
  • 104. zero-friction testing • Errors • Warnings • Analyzer results • Command-U tests
  • 105. zero-friction testing • Errors • Warnings • Analyzer results • Command-U tests • Merge failures
  • 106. zero-friction testing • Errors • Warnings • Analyzer results • Command-U tests • Merge failures • KIF UI integration tests All before a reviewer looks at the patch
  • 107. zero-friction testing Unified error summary Detailed log files
  • 108. test once, test forever • Reviewers only get sane patches • Once you write a failing test, never look at the patch again
  • 109. Integration made easy When the tests pass, GLaDOS merges the feature in and closes the ticket.
  • 110. AL RN LY TE ON IN E USDeployments made easy Only one person knows how to do this!
  • 111. AL RN LY TE ON IN E USDeployments made easy
  • 112. AL RN LY TE ON IN E USDeployments made easy
  • 113. AL RN LY TE ON IN E USDeployments made easy
  • 114. AL RN LY TE ON IN E USDeployments made easy
  • 115. AL RN LY TE ON IN E USDeployments made easy
  • 116. AL RN LY TE ON IN E USDeployments made easy
  • 117. Setup
  • 118. buildbot: the bad news we’re the only user
  • 119. buildbot: the bad news • Large buy-in • work.py - workflow • git - version control • FogBugz - bug tracking • GitHub - source hosting
  • 120. buildbot: the bad news Lots of our defaults aren’t configurable
  • 121. buildbot: the bad news Hard to retrofit into your existing workflow
  • 122. buildbot: the bad news Lots of undocumented behavior
  • 123. EN E OP RC S OUbuildbot http://github.com/drewcrawford/buildbot
  • 126. OK, now what? • Test. • Test early, test often.
  • 127. OK, now what? • Test. • Test early, test often. • Test reasonably.
  • 128. OK, now what? • Test. • Test early, test often. • Test reasonably. • Test automatically.
  • 129. OK, now what? • Test. • Test early, test often. • Test reasonably. • Test automatically. • Think about your workflow
  • 130. Tiny iOS Developer • Mix of contracts & products • Many other dev tools like buildbot

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n
  35. \n
  36. \n
  37. \n
  38. \n
  39. \n
  40. \n
  41. \n
  42. \n
  43. \n
  44. \n
  45. \n
  46. \n
  47. \n
  48. \n
  49. \n
  50. \n
  51. \n
  52. \n
  53. \n
  54. \n
  55. \n
  56. \n
  57. \n
  58. \n
  59. \n
  60. \n
  61. \n
  62. \n
  63. \n
  64. \n
  65. \n
  66. \n
  67. \n
  68. \n
  69. \n
  70. \n
  71. \n
  72. \n
  73. \n
  74. \n
  75. \n
  76. \n
  77. \n
  78. \n
  79. \n
  80. \n
  81. \n
  82. \n
  83. \n
  84. \n
  85. \n
  86. \n
  87. \n
  88. \n
  89. \n
  90. \n
  91. \n
  92. \n
  93. \n
  94. \n
  95. \n
  96. \n
  97. \n
  98. \n
  99. \n
  100. \n
  101. \n
  102. \n
  103. \n
  104. \n
  105. \n
  106. \n
  107. \n
  108. \n
  109. \n
  110. \n
  111. \n
  112. \n
  113. \n
  114. \n
  115. \n
  116. \n
  117. \n
  118. \n
  119. \n
  120. \n
  121. \n
  122. \n
  123. \n
  124. \n
  125. \n
  126. \n
  127. \n
  128. \n
  129. \n
  130. \n
  131. \n
  132. \n
  133. \n
  134. \n
  135. \n
  136. \n
  137. \n
  138. \n
  139. \n
  140. \n