The document emphasizes the importance of implementing a robust testing process for mobile app development following continuous delivery principles, highlighting challenges such as App Store distribution and device variability. It details best practices and tools like Xamarin.UITest and Calabash for automating tests, as well as team structures that promote shared responsibilities in testing. Key insights include improvements in efficiency, reduced bug incidents, and a focus on stable test suites to enhance overall app quality and development speed.

1. Get Your Testing
Process in Place
Niels Frydenholm
Senior iOS developer
nfrydenholm@ebay.com

2. Get Your Testing
Process in Place
Karl Krukow
Technical Lead,, Xamarin Test Cloud
@karlkrukow

4. “Continuous Delivery is a software
development discipline where you build
software in such a way that the software
can be released to production at any time.”
Martin Fowler
ThoughtWorks

5. Continuous Delivery

6. Mobile Continuous Delivery Challenges
• App Store Distribution Model
•Maturity - No Industry Standards (yet)
• Complicated Tool Setup
• Realistic tests are challenging
• Tested on your users’ devices
• Vary Device Conditions - Device settings, Network…

7. Towards Continuous Delivery for Mobile
Aspects
Tooling
People
Process

8. Towards Continuous Delivery for Mobile
Aspects Examples
Tooling
People
Process
• Xamarin.UITest, Calabash
• Jenkins, TeamCity, TFS, …
• Xamarin Test Cloud

9. Towards Continuous Delivery for Mobile
Aspects Examples
Tooling
People
Process
• Xamarin.UITest, Calabash
• Jenkins, TeamCity, TFS, …
• Xamarin Test Cloud
• Cross-functional Teams
• Behaviour Driven Development
• Shared responsibility

10. Xamarin.UITest & Calabash

11. Xamarin.UITest & Calabash
Xamarin.UITest
• Write Tests in C#
• Run with NUnit
• Xamrin/Visual Studio or CLI
Calabash
• Write Tests in Ruby
• Supports BDD, run with Cucumber
• RubyMine or CLI

12. Xamarin.UITest - Architecture
Your App
Test Cloud
Agent

13. Calabash - Architecture
Your App
Test Cloud
Agent
!
Scenario: Login
Given I am on the Login screen
When I login as "Nat"
Then I should go to the Assignments

14. Calabash - Architecture
Your App
Test Cloud
Agent
Cucumber
!
Scenario: Login
Given I am on the Login screen
When I login as "Nat"
Then I should go to the Assignments

15. Calabash - Architecture
Your App
Test Cloud
Agent
Cucumber
!
Scenario: Login
Given I am on the Login screen
When I login as "Nat"
Then I should go to the Assignments

16. Get Your Testing
Process in Place
Niels Frydenholm
Senior iOS developer
nfrydenholm@ebay.com

17. Agenda
■ From story to running tests
■ Improvements in development and release cycle
■ Demo
■ Best practices (based on our learnings)
■ Final words
■ Questions?

18. eBay Classifieds, Denmark
Two popular consumer apps and sites
■ DBA - DK’s #1 Classifieds site
■ BilBasen - DK’s #1 Online Vehicle Marketplace
A few numbers
■ Approx. 1/4 of the danish population is using the
DBA app
■ Mobile traffic share is 57% for DBA
■ Approx 1/8 of the danish population is using the
BilBasen app
■ Mobile traffic share is 54% for BilBasen

19. How we are organised
The team
■ 4 iOS developers / (backend)
■ 1 backend developer
■ 1 QA expert
■ 1 Scrum master

20. From Story
to Running Tests

21. Implementing a story
Before sprint
■ Product owner (PO) introduces story
■ Team and PO agrees on accept criteria’s
New feature idea
Lots of questions

22. Implementing a story
Before sprint
■ Product owner (PO) introduces story
■ Team and PO agrees on accept criteria’s
New feature idea
Lots of questions
Implement feature
Write tests
Review
Test
In Sprint
■ Developer(s) implements story and tests
■ Unit, Integration and UI tests
■ Done with code and tests are green
■ Review code (and tests)
■ Manuel test from other team member
■ Inspect code coverage to ensure test quality

23. Continuous Integration
■ Jenkins on a couple of Mac Mini’s
- Unit -> Integration -> UI tests
- UI tests only in Simulator
■ A subset after commit
■ A subset in Xamarin Cloud
■ All UI tests nightly
!
■ Keep status visible and Green!

24. Focus on Automated Tests
Has Improved ‘Everything’

25. Releases 2013
DBA versions
JAN
2.0 2.0.1 2.1 2.2 2.3 2.3.1 2.4 3.0
FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
BilBasen versions
4.2 5.0 5.0.1 5.0.2
Refactored tests
to Page objects
4.1.2
4.1.3

26. 3.3.1 3.5
3.2 3.3 3.4.1
3.0.1 3.1 3.2.1 3.4 3.4.2
FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
5.0.4 5.0.5 5.0.6 5.1 5.2 5.2.2
5.2.1
3.2.3
Releases 2014
DBA versions
3.1.1 3.2.2
JAN
5.0.3
BilBasen versions

27. Development improvements
In early 2013
• Limited refactoring - risk too high
• All changes = huge test effort
•Many bugs were not found until later…
• Lots of “legacy code”
• See first bullet

28. Development improvements
In early 2013 Now
• Limited refactoring - risk too high
• All changes = huge test effort
•Many bugs were not found until later…
• Lots of “legacy code”
• See first bullet
• Planned refactoring of larger areas
• Boy scout clean up
• Adding tests to missing areas
• (Most) bugs are found immediately after
commit
• QA can spend time on better things
■ Identify critical business areas
■ Find the crazy bugs

29. Project example
BilBasen - new search
■ Search criteria pages and search logic
was totally re-written
■ From hardcoded options in the app
■ To meta driven from the server
■ Server side meta data was build as
part of a project
■ Saved searches now in sync with web

30. BilBasen - new search
Test results
■ Most scenarios were already covered
with UI tests
■ Manuel test effort
■ About one day of testing
■ Focused on network timings
■ Compare results with website
■ No serious bugs were found

31. Release improvements
In early 2013
• Entire team spent 2-5 days on test runs
and bug fixes
• Serious bugs could be found in last
minute
• Big releases to save the overhead
• Releases were often delayed

32. Release improvements
In early 2013 Now
• Entire team spent 2-5 days on test runs
and bug fixes
• Serious bugs could be found in last
minute
• Big releases to save the overhead
• Releases were often delayed
• Upload when all tests are green (all the
time!)
• PO flexibility to change scope of
release
• 1 hour test run for some releases
■ Bugs are very rarely found

33. Demo

34. Best Practices
(based on our learnings)

35. Best Practices
• Structure your test code well
• Page objects
• Replace “sleeps” with “wait_for”

36. Page objects and wait_for
Before page objects (Ruby example)
Scenario: Check that change classification clears
matrixdata with warning
Given I am logged in as "Buyer"
And I am on the SYI hub
And I select classification "Hovedtelefoner"
And I set "Type" to “In-ear”
And I set price to "250"
!

37. Page objects and wait_for
Before page objects (Ruby example)
Scenario: Check that change classification clears
matrixdata with warning
Given I am logged in as "Buyer"
And I am on the SYI hub
And I select classification "Hovedtelefoner"
And I set "Type" to “In-ear”
And I set price to "250"
!
And /^I set price to "(.*?)"$/ do |price|
macro 'I swipe up'
sleep(1)
touch("view marked:'Price'")
sleep(0.5)
set_text("view marked:'Price'", price)
sleep(0.5)
touch("view marked:'OK'")
sleep(1)
end

38. Page object and wait_for
With page objects (Ruby example)
And /^I set price to "(.*?)"$/ do |price|
@page.write_price(price)
end

39. Page object and wait_for
With page objects (Ruby example)
And /^I set price to "(.*?)"$/ do |price|
@page.write_price(price)
end
SellYourItemPage class (page object)
def write_price(price)
scroll_and_wait_for_row_with_mark("priceCell")
touch("view marked:'Price'")
keyboard_enter_text price
close_keyboard
end
def select_bundle(bundle_name)
..logic here..
end
!
def write_headline(headline)
..logic here..
end

40. Best Practices
• Structure your test code well
• Page objects
• replace “sleeps” with “wait_for”
• Ensure stable test suite execution
• Give failed tests an extra run

41. Stability in your CI setup
• UIAutomation + Simulator can often fail when running
a big test suite
• Re-run failed tests an extra time
• Cucumber rerun formatter
• NUnit-retry
• Red builds should be trusted, investigated and fixed
• Only use this if test suite starts to be unstable

42. Stability in your CI setup
• UIAutomation + Simulator can often fail when running
a big test suite
• Re-run failed tests an extra time
• Cucumber rerun formatter
• NUnit-retry
• Red builds should be trusted, investigated and fixed
• Only use this if test suite starts to be unstable

43. Best Practices
• Structure your test code well
• Page objects
• replace “sleeps” with “wait_for”
• Ensure stable test suite execution
• Give failed tests an extra
• Run tests on real devices
• Xamarin Test Cloud
• Screen sizes, OS versions

45. Best Practices
• Structure your test code well
• Page objects
• replace “sleeps” with “wait_for”
• Ensure stable test suite execution
• Give failed tests an extra run
• Run tests on real devices
• Xamarin Test Cloud
• Screen sizes, OS versions
• Strive for fast test execution
• Use different test suites
• Avoid too much UI interaction for
setup/teardown operations

46. Fast test execution
• Use the “Backdoor” to the app
• Xamarin.UITest.IApp.Invoke
• Invoke(“method”, param)
• Invoke logic - without using the UI
• Great for Setup / Teardown behaviour
• Backdoor logic should be tested with UI
interactions in other tests

47. Fast test execution
• Use the “Backdoor” to the app
• Xamarin.UITest.IApp.Invoke
• Invoke(“method”, param)
• Invoke logic - without using the UI
• Great for Setup / Teardown behaviour
• Backdoor logic should be tested with UI
interactions in other tests
•We use it for things like
• Create a user / Login
• Create listings (things for sale)
• Cleaning up
• Delete favorites
• Delete users listings

48. Fast test execution - backdoor optimised
Scenario: I can manage my listing and see the changes
Given I am logged in as "UniqueSeller" using quick login
And I have created a listing for "Hovedtelefoner" and is on
the SYI VIP
Then I see the VIP for "Her kommer en rimelig lang tekst"
!
When I go back to my listings page
And the created listing is in the list
…more steps omitted…

49. Fast test execution - backdoor optimised
Scenario: I can manage my listing and see the changes
Given I am logged in as "UniqueSeller" using quick login
And I have created a listing for "Hovedtelefoner" and is on
the SYI VIP
Then I see the VIP for "Her kommer en rimelig lang tekst"
!
When I go back to my listings page
And the created listing is in the list
…more steps omitted…
1 scenario (1 passed)
19 steps (19 passed)
0m34.054s
1 scenario (1 passed)
19 steps (19 passed)
1m16.610s

50. Best Practices
• Structure your test code well
• Page objects
• replace “sleeps” with “wait_for”
• Ensure stable test suite execution
• Give failed tests an extra run
• Run tests on real devices
• Xamarin Test Cloud
• Screen sizes, OS versions
• Strive for fast test execution
• Use different test suites
• Avoid too much UI interaction for
setup/teardown operations
• Run tests for all active branches in CI
• Automate job creation

51. Feature builds
■ All jobs required for a branch
■ Red builds are accepted for a period
■ Only merge to master with green builds

52. Final words

53. Start today…
• QA is a Team effort / responsibility
• Bugs/Crashes will still live in the app
• Do not expect it to be effortless to get
started
• It will quickly pay off

54. Start today…
• QA is a Team effort / responsibility
• Bugs/Crashes will still live in the app
• Do not expect it to be effortless to get
started
• It will quickly pay off
See it as an investment that gives you
• Higher app quality (and code quality)
• Time to focus on new important features
• Faster time to market - with less bugs
• Happy customers - they like frequent
updates of high quality
• Happy and calm QA
• Happy and brave developers

55. Questions?
Niels Frydenholm
@nfrydenholm