Successfully reported this slideshow.

Frank iOS Testing

6

Share

Loading in …3
×
1 of 54
1 of 54

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Frank iOS Testing

  1. 1. Testing iOS apps with Frank Stewart Gleadow @stewgleadow
  2. 2. why have tests?
  3. 3. why have tests?
  4. 4. Testing Approaches
  5. 5. “It seems to be working”
  6. 6. “Nothing is obviously broken”
  7. 7. Testing iPhone Apps iPhone App
  8. 8. iPhone App Server
  9. 9. iPhone App Solved? Server
  10. 10. iPhone App ? Solved? Server
  11. 11. Does MVC help?
  12. 12. UIView UIViewController Model Server
  13. 13. UIView UIViewController Model Server
  14. 14. UIView UIViewController Solved? Model Server
  15. 15. UIView ? UIViewController Solved? Model Server
  16. 16. UIViewController UIView
  17. 17. UIViewController UIView 50% of native code?
  18. 18. UIViewController UIView
  19. 19. Automated Functional Testing
  20. 20. Potential Testing Tools hmmm...
  21. 21. Potential Testing Tools • Frank and UISpec hmmm...
  22. 22. Potential Testing Tools • Frank and UISpec hmmm... • Sikuli
  23. 23. Potential Testing Tools • Frank and UISpec hmmm... • Sikuli • FoneMonkey
  24. 24. Potential Testing Tools • Frank and UISpec hmmm... • Sikuli • FoneMonkey • iCuke
  25. 25. Potential Testing Tools • Frank and UISpec hmmm... • Sikuli • FoneMonkey • iCuke • UIAutomation
  26. 26. UISpec
  27. 27. UISpec - (void)itShouldHaveDefaultUsers; { [[app.tableView.label text:@"User"] should].exist; }
  28. 28. UISpec
  29. 29. Cucumber
  30. 30. Cucumber Scenario: [name of the scenario] Given [some statement] When [some action] Then [some expected result] And [another expectation]
  31. 31. to be perfectly Frank...
  32. 32. Tests iPhone App
  33. 33. Tests UISpec Cucumber / Ruby ? iPhone App
  34. 34. Tests Frank Server UISpec Cucumber / Ruby iPhone App
  35. 35. Tests Frank Driver Frank Server UISpec Cucumber / Ruby iPhone App
  36. 36. Tests Frank Driver Frank Server “frankly” UISpec Cucumber / Ruby iPhone App
  37. 37. Frank uses accessibility labels
  38. 38. frankly.my_dear do |i| dont_give_a_damn! end
  39. 39. frankly.my_dear do |i| dont_give_a_damn! end UISpec UIQuery Frank Frankly
  40. 40. frankly.my_dear do |i| dont_give_a_damn! end UISpec UIQuery [app.tableView.label text:@"User"]; Frank Frankly
  41. 41. frankly.my_dear do |i| dont_give_a_damn! end UISpec UIQuery [app.tableView.label text:@"User"]; Frank Frankly app tableView label text:’User’
  42. 42. Frank Steps Scenario: default users should be present at startup When I start the app Then I should see “Users”
  43. 43. Frank Steps Scenario: default users should be present at startup When I start the app Then I should see “Users” Then /^I should see "([^"]*)"$/ do |expected_mark| check_element_exists("view marked:'#{expected_mark}'") end
  44. 44. Demo
  45. 45. The Road Ahead
  46. 46. The Road Ahead Behind
  47. 47. Stewart Gleadow sgleadow@thoughtworks.com @stewgleadow
  48. 48. References • github.com/moredip/frank & http://groups.google.com/group/frank-discuss • code.google.com/p/uispec & http://groups.google.com/group/uispec • softnoise.wordpress.com/2010/11/14/ios-setting-up-a-test-environment/ • cukes.info • cuke4ninja.com • The RSpec Book
  49. 49. Images • http://www.myfreewallpapers.net/movies/pages/ frankenstein-02.shtml • http://www.workbloom.net/wp-content/uploads/2009/09/ road-ahead.jpg • http://upload.wikimedia.org/wikipedia/commons/archive/ 0/07/20090605224904!IMac_aluminium.png • http://themachoresponse.blogspot.com/2010/03/frankly-my- dear-i-dont-give-damn.html • http://www.jesseshunting.com/images/confused_sign_post.jpg

Editor's Notes

  • \n
  • - talk covers native apps only\n- audience different from original\n- linking up a handful of open source tools to create a monster\n- the tools that it links together\n- the reason frank needs to exist\n
  • - in a team environment, need to know if you’ve broken another part of the app\n- cost of bugs in production is too high (bad reviews kill you, one chance)\n- executable documentation (so should be business readable => cucumber - DHH)\n- onboarding team members quickly (agile app)\n\n\n
  • - in a team environment, need to know if you’ve broken another part of the app\n- cost of bugs in production is too high (bad reviews kill you, one chance)\n- executable documentation (so should be business readable => cucumber - DHH)\n- onboarding team members quickly (agile app)\n\n\n
  • - in a team environment, need to know if you’ve broken another part of the app\n- cost of bugs in production is too high (bad reviews kill you, one chance)\n- executable documentation (so should be business readable => cucumber - DHH)\n- onboarding team members quickly (agile app)\n\n\n
  • - broad spectrum of testing approaches\n- skewed towards the ‘absolutely none’ end\n\n
  • - try it out, press a few buttons (perhaps even mail it to the user)\n- it seems to be working => nothing is obviously broken\n* not as strong a statement (try sending that to a client)\n- manual testing takes too long\n\n
  • - try it out, press a few buttons (perhaps even mail it to the user)\n- it seems to be working => nothing is obviously broken\n* not as strong a statement (try sending that to a client)\n- manual testing takes too long\n\n
  • - try it out, press a few buttons (perhaps even mail it to the user)\n- it seems to be working => nothing is obviously broken\n* not as strong a statement (try sending that to a client)\n- manual testing takes too long\n\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - one big amorphous blob -> maybe there are parts to test\n- test client/server separately\n* testing the bits where computers talk to computers is easier\n- rspec/cucumber (dhh rant?)\n* native side should be as small as possible (or not native at all?)\n* native side is Safari?\n
  • - break down into smaller components\n-> not just all in your app delegate and view controllers\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - done the server, look at the iPhone app specifically\n- can unit test the M in MVC\n* (coverage and a little bit more patchy than that)\n- is it possible for the model to be cross platform?\n* Rhodes? c# w/ Mono? C? Javascript?\n
  • - views: hard to test that a view looks correct, or an animation looks right, must be manual\n- controllers: keep it just plumping* (don’t let them creep)\n* if it goes wrong it should be obvious (but things break)\n- messiest & most prone to change?\n- the bits the user actually sees\n
  • - views: hard to test that a view looks correct, or an animation looks right, must be manual\n- controllers: keep it just plumping* (don’t let them creep)\n* if it goes wrong it should be obvious (but things break)\n- messiest & most prone to change?\n- the bits the user actually sees\n
  • - need something that interacts with the app like a user\n- selenium/webdriver equivalents\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • TOOLS FOR - actually remote controlling your app (like Selenium or WebDriver)\nSikuli - uses image analysis (fragile?)\n- simulator only unless you jailbreak your phone and run a VNC server on it\nFoneMonkey - comes from a record/playback model (does have scripting now)\niCuke - very similar to Frank but uses XML/xpath\n- pushes more work to the Ruby side, very little Objective C logic\n- potentially lead to brittle tests as XML structure more likely to change?\n- more “touch at X Y” kind of tests\nUIAutomation - Instruments + javascript + jasmine-iphone\n- could not consistently run and automate in the build, no applescript\n
  • - BDD tool modelled on RSpec\n- useful tool, not a huge community about it\n- uses a query language called UIQuery\n- difficult to run as part of a CI build (Perryn)\n- output hard to read\n- not business readable\n
  • - BDD tool modelled on RSpec\n- useful tool, not a huge community about it\n- uses a query language called UIQuery\n- difficult to run as part of a CI build (Perryn)\n- output hard to read\n- not business readable\n
  • - BDD tool modelled on RSpec\n- useful tool, not a huge community about it\n- uses a query language called UIQuery\n- difficult to run as part of a CI build (Perryn)\n- output hard to read\n- not business readable\n
  • - easy to integrate with CI\n
  • - describe the behaviour of the app in plain English\n- business readable\n-> think at a higher level\n
  • - stringing other open source libraries together to make Frank\n- put it together to make the monster\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - embeds an HTTP server inside the app to send commands to\n(using a modified main.m file or conditional compilation using precompile flags)\n- interact with via cucumber with Ruby steps\n- more predefined steps than iCuke\n* should run on the device (no dynamic libraries)\n* published as bonjour device\n\n
  • - enabled accessibility in OS X Universal Access and iPhone Simulator\n(not sure about on the actual device, is it always on?)\n- leads to less fragile tests\n* tests can use the same domain language as the app\n(not bound to the view hierarchy or XML representation)\n- at worst, the app becomes more accessible (if done right)\n* forces the tests to use the domain language of the user\n
  • - Frankly is like UIQuery without the . and [ ]\n
  • - Frankly is like UIQuery without the . and [ ]\n
  • - Frankly is like UIQuery without the . and [ ]\n
  • - Frankly is like UIQuery without the . and [ ]\n
  • - symbiote: talk to a running app interactively\n- view DOM & accessibility label\n- flash element\n- write the most general selector possible (less brittle tests)\n\n
  • - symbiote: talk to a running app interactively\n- view DOM & accessibility label\n- flash element\n- write the most general selector possible (less brittle tests)\n\n
  • - symbiote: talk to a running app interactively\n- view DOM & accessibility label\n- flash element\n- write the most general selector possible (less brittle tests)\n\n
  • - lots of predefined steps for many common cases\n- views exist, accessibility falling back to labels\n- run in simulator, kill, wipe the cache\n
  • \n
  • - not the bill gates book -> information highway\n- easier install (frank-cucmber ruby gem)\n- better integration with CI\n* launch_app_headless\n* iphone_sim\n* xcodebuild instead of Applescript\n- recording test runs as movies for test artifacts\n* (already implemented, just haven’t looked into it)\n
  • - not the bill gates book -> information highway\n- easier install (frank-cucmber ruby gem)\n- better integration with CI\n* launch_app_headless\n* iphone_sim\n* xcodebuild instead of Applescript\n- recording test runs as movies for test artifacts\n* (already implemented, just haven’t looked into it)\n
  • - not the bill gates book -> information highway\n- easier install (frank-cucmber ruby gem)\n- better integration with CI\n* launch_app_headless\n* iphone_sim\n* xcodebuild instead of Applescript\n- recording test runs as movies for test artifacts\n* (already implemented, just haven’t looked into it)\n
  • - screenshots and image based regression testing\n- even better CI capability\n- merge with iCuke\n- integrate with UIAutomation\n* less UISpec, add javascript steps?\n- better on-device testing\n* Apple knows devices/OS, not testing\n* give us the hooks\n
  • \n
  • \n
  • \n
  • ×