Testing iOS appswith Frank  Stewart Gleadow   @stewgleadow
why have  tests?
why have  tests?
Testing Approaches
“It seems tobe working”
“Nothing isobviously broken”
Testing iPhone Apps      iPhone App
iPhone App  Server
iPhone App             Solved?  Server
iPhone App   ?             Solved?  Server
Does MVC help?
UIViewUIViewController                   Model    Server
UIViewUIViewController                   Model    Server
UIViewUIViewController    Solved?        Model    Server
UIView          ?UIViewController    Solved?        Model    Server
UIViewController   UIView
UIViewController         UIView          50% of native code?
UIViewController   UIView
AutomatedFunctional Testing
Potential Testing Tools                 hmmm...
Potential Testing Tools• Frank and UISpec   hmmm...
Potential Testing Tools• Frank and UISpec   hmmm...• Sikuli
Potential Testing Tools• Frank and UISpec   hmmm...• Sikuli• FoneMonkey
Potential Testing Tools• Frank and UISpec   hmmm...• Sikuli• FoneMonkey• iCuke
Potential Testing Tools• Frank and UISpec   hmmm...• Sikuli• FoneMonkey• iCuke• UIAutomation
UISpec
UISpec- (void)itShouldHaveDefaultUsers;{    [[app.tableView.label text:@"User"] should].exist;}
UISpec
Cucumber
CucumberScenario: [name of the scenario]   Given [some statement]   When [some action]   Then [some expected result]   And...
to be perfectly Frank...
Tests        iPhone          App
Tests                          UISpec   Cucumber / Ruby                     ?                         iPhone              ...
Tests                Frank                     Server                          UISpec   Cucumber / Ruby                   ...
Tests                Frank                     Driver                              Frank                              Serv...
Tests                Frank                     Driver                                          Frank                      ...
Frankusesaccessibilitylabels
frankly.my_dear       do |i|  dont_give_a_damn!end
frankly.my_dear       do |i|  dont_give_a_damn!end    UISpec   UIQuery     Frank    Frankly
frankly.my_dear        do |i|  dont_give_a_damn!end    UISpec   UIQuery            [app.tableView.label text:@"User"];    ...
frankly.my_dear        do |i|  dont_give_a_damn!end    UISpec   UIQuery            [app.tableView.label text:@"User"];    ...
Frank StepsScenario: default users should be present at startup   When I start the app   Then I should see “Users”
Frank StepsScenario: default users should be present at startup   When I start the app   Then I should see “Users”  Then /...
Demo
The Road Ahead
The Road Ahead Behind
Stewart Gleadowsgleadow@thoughtworks.com      @stewgleadow
References•   github.com/moredip/frank & http://groups.google.com/group/frank-discuss•   code.google.com/p/uispec & http:/...
Images•   http://www.myfreewallpapers.net/movies/pages/    frankenstein-02.shtml•   http://www.workbloom.net/wp-content/up...
Frank iOS Testing
Frank iOS Testing
Frank iOS Testing
Frank iOS Testing
Frank iOS Testing
Upcoming SlideShare
Loading in...5
×

Frank iOS Testing

7,241

Published on

A recent talk I gave on using Frank for automated functional testing with iOS development.

Published in: Technology, News & Politics
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,241
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
72
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • \n
  • - talk covers native apps only\n- audience different from original\n- linking up a handful of open source tools to create a monster\n- the tools that it links together\n- the reason frank needs to exist\n
  • - in a team environment, need to know if you’ve broken another part of the app\n- cost of bugs in production is too high (bad reviews kill you, one chance)\n- executable documentation (so should be business readable => cucumber - DHH)\n- onboarding team members quickly (agile app)\n\n\n
  • - in a team environment, need to know if you’ve broken another part of the app\n- cost of bugs in production is too high (bad reviews kill you, one chance)\n- executable documentation (so should be business readable => cucumber - DHH)\n- onboarding team members quickly (agile app)\n\n\n
  • - in a team environment, need to know if you’ve broken another part of the app\n- cost of bugs in production is too high (bad reviews kill you, one chance)\n- executable documentation (so should be business readable => cucumber - DHH)\n- onboarding team members quickly (agile app)\n\n\n
  • - broad spectrum of testing approaches\n- skewed towards the ‘absolutely none’ end\n\n
  • - try it out, press a few buttons (perhaps even mail it to the user)\n- it seems to be working => nothing is obviously broken\n* not as strong a statement (try sending that to a client)\n- manual testing takes too long\n\n
  • - try it out, press a few buttons (perhaps even mail it to the user)\n- it seems to be working => nothing is obviously broken\n* not as strong a statement (try sending that to a client)\n- manual testing takes too long\n\n
  • - try it out, press a few buttons (perhaps even mail it to the user)\n- it seems to be working => nothing is obviously broken\n* not as strong a statement (try sending that to a client)\n- manual testing takes too long\n\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - break down into smaller components\n-> not just all in your app delegate and view controllers\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - views: hard to test that a view looks correct, or an animation looks right, must be manual\n- controllers: keep it just plumping* (don’t let them creep)\n* if it goes wrong it should be obvious (but things break)\n- messiest & most prone to change?\n- the bits the user actually sees\n
  • - views: hard to test that a view looks correct, or an animation looks right, must be manual\n- controllers: keep it just plumping* (don’t let them creep)\n* if it goes wrong it should be obvious (but things break)\n- messiest & most prone to change?\n- the bits the user actually sees\n
  • - need something that interacts with the app like a user\n- selenium/webdriver equivalents\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • - BDD tool modelled on RSpec\n- useful tool, not a huge community about it\n- uses a query language called UIQuery\n- difficult to run as part of a CI build (Perryn)\n- output hard to read\n- not business readable\n
  • - BDD tool modelled on RSpec\n- useful tool, not a huge community about it\n- uses a query language called UIQuery\n- difficult to run as part of a CI build (Perryn)\n- output hard to read\n- not business readable\n
  • - BDD tool modelled on RSpec\n- useful tool, not a huge community about it\n- uses a query language called UIQuery\n- difficult to run as part of a CI build (Perryn)\n- output hard to read\n- not business readable\n
  • - easy to integrate with CI\n
  • - describe the behaviour of the app in plain English\n- business readable\n-> think at a higher level\n
  • - stringing other open source libraries together to make Frank\n- put it together to make the monster\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - enabled accessibility in OS X Universal Access and iPhone Simulator\n(not sure about on the actual device, is it always on?)\n- leads to less fragile tests\n* tests can use the same domain language as the app\n(not bound to the view hierarchy or XML representation)\n- at worst, the app becomes more accessible (if done right)\n* forces the tests to use the domain language of the user\n
  • - Frankly is like UIQuery without the . and [ ]\n
  • - Frankly is like UIQuery without the . and [ ]\n
  • - Frankly is like UIQuery without the . and [ ]\n
  • - Frankly is like UIQuery without the . and [ ]\n
  • - symbiote: talk to a running app interactively\n- view DOM & accessibility label\n- flash element\n- write the most general selector possible (less brittle tests)\n\n
  • - symbiote: talk to a running app interactively\n- view DOM & accessibility label\n- flash element\n- write the most general selector possible (less brittle tests)\n\n
  • - symbiote: talk to a running app interactively\n- view DOM & accessibility label\n- flash element\n- write the most general selector possible (less brittle tests)\n\n
  • - lots of predefined steps for many common cases\n- views exist, accessibility falling back to labels\n- run in simulator, kill, wipe the cache\n
  • \n
  • - not the bill gates book -> information highway\n- easier install (frank-cucmber ruby gem)\n- better integration with CI\n* launch_app_headless\n* iphone_sim\n* xcodebuild instead of Applescript\n- recording test runs as movies for test artifacts\n* (already implemented, just haven’t looked into it)\n
  • - not the bill gates book -> information highway\n- easier install (frank-cucmber ruby gem)\n- better integration with CI\n* launch_app_headless\n* iphone_sim\n* xcodebuild instead of Applescript\n- recording test runs as movies for test artifacts\n* (already implemented, just haven’t looked into it)\n
  • - not the bill gates book -> information highway\n- easier install (frank-cucmber ruby gem)\n- better integration with CI\n* launch_app_headless\n* iphone_sim\n* xcodebuild instead of Applescript\n- recording test runs as movies for test artifacts\n* (already implemented, just haven’t looked into it)\n
  • - screenshots and image based regression testing\n- even better CI capability\n- merge with iCuke\n- integrate with UIAutomation\n* less UISpec, add javascript steps?\n- better on-device testing\n* Apple knows devices/OS, not testing\n* give us the hooks\n
  • \n
  • \n
  • \n
  • Frank iOS Testing

    1. 1. Testing iOS appswith Frank Stewart Gleadow @stewgleadow
    2. 2. why have tests?
    3. 3. why have tests?
    4. 4. Testing Approaches
    5. 5. “It seems tobe working”
    6. 6. “Nothing isobviously broken”
    7. 7. Testing iPhone Apps iPhone App
    8. 8. iPhone App Server
    9. 9. iPhone App Solved? Server
    10. 10. iPhone App ? Solved? Server
    11. 11. Does MVC help?
    12. 12. UIViewUIViewController Model Server
    13. 13. UIViewUIViewController Model Server
    14. 14. UIViewUIViewController Solved? Model Server
    15. 15. UIView ?UIViewController Solved? Model Server
    16. 16. UIViewController UIView
    17. 17. UIViewController UIView 50% of native code?
    18. 18. UIViewController UIView
    19. 19. AutomatedFunctional Testing
    20. 20. Potential Testing Tools hmmm...
    21. 21. Potential Testing Tools• Frank and UISpec hmmm...
    22. 22. Potential Testing Tools• Frank and UISpec hmmm...• Sikuli
    23. 23. Potential Testing Tools• Frank and UISpec hmmm...• Sikuli• FoneMonkey
    24. 24. Potential Testing Tools• Frank and UISpec hmmm...• Sikuli• FoneMonkey• iCuke
    25. 25. Potential Testing Tools• Frank and UISpec hmmm...• Sikuli• FoneMonkey• iCuke• UIAutomation
    26. 26. UISpec
    27. 27. UISpec- (void)itShouldHaveDefaultUsers;{ [[app.tableView.label text:@"User"] should].exist;}
    28. 28. UISpec
    29. 29. Cucumber
    30. 30. CucumberScenario: [name of the scenario] Given [some statement] When [some action] Then [some expected result] And [another expectation]
    31. 31. to be perfectly Frank...
    32. 32. Tests iPhone App
    33. 33. Tests UISpec Cucumber / Ruby ? iPhone App
    34. 34. Tests Frank Server UISpec Cucumber / Ruby iPhone App
    35. 35. Tests Frank Driver Frank Server UISpec Cucumber / Ruby iPhone App
    36. 36. Tests Frank Driver Frank Server “frankly” UISpec Cucumber / Ruby iPhone App
    37. 37. Frankusesaccessibilitylabels
    38. 38. frankly.my_dear do |i| dont_give_a_damn!end
    39. 39. frankly.my_dear do |i| dont_give_a_damn!end UISpec UIQuery Frank Frankly
    40. 40. frankly.my_dear do |i| dont_give_a_damn!end UISpec UIQuery [app.tableView.label text:@"User"]; Frank Frankly
    41. 41. frankly.my_dear do |i| dont_give_a_damn!end UISpec UIQuery [app.tableView.label text:@"User"]; Frank Frankly app tableView label text:’User’
    42. 42. Frank StepsScenario: default users should be present at startup When I start the app Then I should see “Users”
    43. 43. Frank StepsScenario: default users should be present at startup When I start the app Then I should see “Users” Then /^I should see "([^"]*)"$/ do |expected_mark| check_element_exists("view marked:#{expected_mark}") end
    44. 44. Demo
    45. 45. The Road Ahead
    46. 46. The Road Ahead Behind
    47. 47. Stewart Gleadowsgleadow@thoughtworks.com @stewgleadow
    48. 48. References• github.com/moredip/frank & http://groups.google.com/group/frank-discuss• code.google.com/p/uispec & http://groups.google.com/group/uispec• softnoise.wordpress.com/2010/11/14/ios-setting-up-a-test-environment/• cukes.info• cuke4ninja.com• The RSpec Book
    49. 49. Images• http://www.myfreewallpapers.net/movies/pages/ frankenstein-02.shtml• http://www.workbloom.net/wp-content/uploads/2009/09/ road-ahead.jpg• http://upload.wikimedia.org/wikipedia/commons/archive/ 0/07/20090605224904!IMac_aluminium.png• http://themachoresponse.blogspot.com/2010/03/frankly-my- dear-i-dont-give-damn.html• http://www.jesseshunting.com/images/confused_sign_post.jpg
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×