This document summarizes a presentation on cross-platform mobile test automation using Xamarin and SpecFlow. It discusses using Xamarin to build native mobile apps with shared C# code, SpecFlow for writing acceptance criteria in Gherkin, and testing the user interface of mobile apps using Appium. It also describes testing at the controller layer for faster test execution. Automating tests is recommended to improve quality and provide faster feedback, but manual testing still has value for exploring apps and hard-to-automate checks.

1. COPYRIGHT, TECHTALK - WWW.TECHTALK.AT
ANDREAS WILLICH (@SABOTAGEANDI)
CHRISTIAN HASSA (@CHRISHASSA)
Mobile Quality Crew Meetup, Vienna, April 23rd 2015
Cross Mobile Testautomation with
Xamarin & SpecFlow

2. 2
Topics
• Xamarin
• Test automation in agile projects
• Specification-By-Example
• SpecFlow
• UI automation on mobile devices
• Mobile app automation with SpecFlow

3. 3
Xamarin

4. 4
Native development for each platform

5. 5
Abstracting the platform

6. 6
Xamarin approach

7. 7
Compilation and run time
Android APK
Android/Linux kernel
IL code for platform
specific profile
VS solution with
projects
iOS APP
Android APK
Shared Code
(C#,VB.NET)
Mono CLR for iOS
(Monotouch)
Mono CLR for
Android
(Monodroid)
Multi-Platform
profile assembly
Dalvik
Android.*
Java.*
IL+JIT
AOT ARM Binary
iOS App (C#)
Android App
(C#)
Xamarin.iOS
profile assembly
Xamarin.Android
profile assembly
iOS/Cocoa Touch
iOS APP
ARM
binary

8. 8
Native mobile apps with C#/Xamarin
1. Reuse existing .NET/C# Know-How
2. Powerful platform and tools
(Linq, Xml, Events/Delegates, Parallel Programming, Visual Studio/TFS, …)
3. Shared code for multi-platform apps

9. 9
Required Know-How
C#

10. 10
Xamarin.iOS
C#

11. 11
Xamarin.Android
twitterListView.setOnItemClickListener (new OnItemClickListener () {
public void onItemClick (AdapterView<?> parent, View view, int position, long id) {
Intent tweetDetails = new Intent (TwitterScreen.this, TweetDetailsScreen.class);
tweetDetails.putExtra ("TweetID", TwitterFeed[position].ID);
startActivity (tweetDetails);
}
};
twitterListView.ItemClick += (s, e) => {
var tweetDetails = new Intent (this, typeof (TweetDetailsScreen));
tweetDetails.PutExtra ("TweetID", TwitterFeed[e.Position].ID);
StartActivity (tweetDetails);
};
C#

12. 12
Development environment
OS Mac OSX, Windows Windows
UI Designer Android, iOS
Build Android, iOS (via Xamarin Build Host on Mac OSX)
Debug Android (Device and ARM/x86 emulator)
iOS (Device and emulator on Mac)

13. 13
.NET libaries availabile on Xamarin
Assembly feature set
mscorlib.dll SL
OpenTK.dll OpenGL/OpenAL
System.dll SL+
System.Core.dll SL
System.Data.dll NET35-
System.Data.Service.Client.dll vollständiger OData Client
System.Json.dll SL
System.Runtime.Serialization.dll SL
System.ServiceModel.dll SL (WCF)
System.ServiceModel.Web.dll SL+
System.Transactions.dll NET35
System.Web.Services NET35-
System.Xml.dll NET35
System.Xml.Linq.dll NET35
Mono.CompilerService
s.SymbolWriter.dll Für Compiler
Mono.Data.Sqlite.dll
ADO.NET für
SQLite
Mono.Data.Tds.dll
TDS Protokoll
Support
Mono.Security.dll Cryptographic API
monotouch.dll iOS Bindings
Mono.Android.dll Android Bindings

14. 14
Xamarin.Forms
One UI definition for 3 platforms (iOS, Android, WP)
in XAML (data binding)
translates to native UI controls on each platform
can be mixed with custom UI

15. 15
Test Automation
in
agile projects

16. 16
Test automation becomes expensive when …
• automating manual tests
• automation makes tests
unreadable
• automating only after
implementation
structure
readability
right moment

17. 17
Structure
Manual tests
Asserts Multiple combined
features
Structure ACT-ASSERT-
ACT-ASSERT-
ACT-ASSERT-
…
Dependent features
Long test path with
high chance to break
Cause and impact of
error hard to trace
Automated Check
Single aspect of a
single feature
ARRANGE –
ACT –
ASSERT
Independent features
Short test path with
lower chance to break
Cause and impact of
error easy to relate

18. 18
What to automate
User
journeys
Acceptance-
criteria
Units
exploratory
testing
Test automation pyramid
Source: Mike Cohn many
few hard
easy
Automatability

19. 19
// Go to web page 'http://localhost:40001/' using new browser instance
BrowserWindow localhostBrowser = BrowserWindow.Launch(
new System.Uri(this.RecordedMethod1Params.Url));
// Click 'Register found item' link
Mouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9));
// Click 'Save' button
Mouse.Click(uISpeichernButton, new Point(44, 14));
int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9));
// Click 'Register found item' link
Mouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7));
// Click 'Save' button
Mouse.Click(uISpeichernButton, new Point(34, 11));
int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9));
Assert.IsTrue(fundNr1 + 1 == fundNr2);
// Click 'Close' button
Mouse.Click(uICloseButton, new Point(26, 11));
Readability

20. 20
A readable test case
Scenario: New found items should receive a
consecutive number for the current year
Given the previous found item of the
current year had the number 145
When I register a new found item
Then the last found item of the
current year should have the number 146

21. 21
When to test (right moment)
Acceptance criteria checks
(ATDD, BDD)
Unit Tests
(TDD)
business view
technical view
Exploratory tests
User Journey tests
Performance, Scalability,
Usability,Security, …
productdefinition
productcritique
New dimension: defining the product
Synergy: Specification of requirements and tests
Agile Testing Quadrants: Brian Marick

22. 22
Automated checks don’t replace manual testing
User
journeys
Acceptance-
criteria
Units
exploratory
testing
manual check
in definition of done
for stories
Main success paths
Undiscovered
acceptance criteria
No/(fewer) manual
regression checks
Fewer paths
are enough
More time
for exploration

23. 23
Specification-By-Example

24. 24
Agile “requirements”
Specification-
By-Example
Image from: 50 Quick ideas to improve your user stories: https://leanpub.com/50quickideas
Why?
Outcome
How?
Specification

25. 25
Agile specifications
define upfront define just-in-time
Code
Reminder
for a
conversation
Goals
Impacts
Epics
User Stories
Acceptance
criteria
Why?
Outcome
How?
Specification

26. 26
public void CalculateDiscount(Order order)
{
if (order.Customer.IsNew)
order.FinalAmount =
Math.Round(order.Total * 9/10);
}
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
We’d like to encourage new customers to buy in our shop.
Therefore we want offer 10% discount for their first order.
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com

27. 27
Acceptance
criteria
Agile specifications
define upfront define just-in-time
Code
Goals
Impacts
Epics
User Stories
Bugs
Why?
Outcome
How?
Specification

28. 28
Specification with examples
Examples …
make abstract descriptions better understandable
are usually not formally documented or exchanged
Examples Tests
Requirements
describe validate
fulfillment of
consist of

29. 29
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
public void CalculateDiscount(Order order)
{
if (order.Customer.IsNew)
order.FinalAmount =
Math.Round(order.Total * 9/10);
}
We’d like to encourage new customers to buy in our shop.
Therefore we want offer 10% discount for their first order.
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com

30. 30
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
New customers should receive 10% discount:
Given the customer hasn‘t ordered yet
When the customer adds a book for EUR 37.50 to the shopping cart
Then the shopping cart shows a total price of EUR 33.75.

31. 31
Actually, this is not quite right:
Books on sale should be excluded!
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com

32. 32
“Happy
Path”
Technical
feasibility
Exceptions,
border cases
3 Amigos
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com

33. 33
Collecting Acceptance Criteria
“I would try to put a book into the
shopping cart …”
“I would try to remove a book
from the shopping cart…”
“I’d check whether the shopping cart
is empty, when I enter the shop …”
Books can be added to
shopping cart.
Books can be removed from
shopping cart.
Shopping cart should be empty
when entering the shop.
... ? …
As a potential customer
I want to collect books in a shopping cart
So that I can order several books at once.
“Imagine this story is
already implemented:
How would you verify it?”
“I would try to add 1000 books to
the shopping cart …”

34. 34
Exploration through examples
Books in catalogue:
Title Author
Specification-By-Example Gojko Adzic
Impact Mapping Gojko Adzic
Explore It! Elisabeth Hendrickson
Competitive Engineering Tom Gilb
… I want to find books in the catalogue by author and title
Search for … Books found …
Spec Specification-By-Example
Hend Explore It!
et Explore It!, Competitive Engineering
Context
Action
Assertion

35. 35
Key examples: Breaking the model
Books in catalogue:
Title Author
Specification-By-Example Gojko Adzic
Impact Mapping Gojko Adzic
Explore It! Elisabeth Hendrickson
Competitive Engineering Tom Gilb
… I want to find books in the catalogue by author and title
Search for … Books found …
Spec Specification-By-Example
Hend Explore It!
et Explore It!, Competitive Engineering
What happens, if I search for
“Explore Specification”?
Can I search for single
characters, e.g. “e”?
Is the number of search results
limited, or paged?
Is the search also performed in the
sub-title of a book?

36. 36
Purpose of the examples
• Shared understanding:
acceptance criteria
• Documentation:
specification details
• Regression-tests:
violated specifications

37. 37
SpecFlow

38. 38
SpecFlow – BDD for .NET
#62 in Visual Studio Gallery based on popularity (Apr 23 2015)
#303 most downloaded package on NuGet
in the last 6 weeks (Apr 23 2015)
~1‘400 unique daily visitors on specflow.org
> 30’000 active users
> 40 contributors

39. 39
Feature: Description of feature or user story
Szenariogrundriss: Beschreibung des Akzeptanzkriteriums
Szenario: Beschreibung des Akzeptanzkriteriums
Gherkin Feature Files
Background: context for all scenarios in the feature file
Scenario: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Given/When/Then: automated scenario steps
Scenario Outline: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Given/When/Then: automated scenario steps with <place holder>
Examples: table with examples for <place holders>Examples: table with examples for <place holders>
Given: automated scenario steps
@tagname
@tagname
@tagname
@tagname

40. 40
Abstract acceptance criteria
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Books can be removed from the shopping basket
Shopping basket is initially empty
The same book can be added multiple times to the shopping
basket

41. 41
Scenarios: Examples in Gherkin
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Given my shopping basket is empty
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 1 copy of “Harry Potter”

42. 42
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Scenarios: Examples in Gherkin
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
The same book can be added multiple times to the shopping basket

43. 43
The same book can be added multiple times to the shopping basket
Structure of Scenarios
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Title: Describes intention/abstract acceptance criterion
Arrange: Context, describes state of the system
Act: Execution of the feature
Assert: Assertion of observable behaviour
And I should see the warning: “Book already existed in basket”
Triple-A
constraint
“Checks”
Chaining
up steps

44. 44
Automated continuous validation
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
System
„Step Definitions“ are binding individual steps to an automatable interface of the application.
Automatable
interface
UI
Automation
Automation does not necessarily have to bind to the UI.
Automatability of system is supported/evolving with development.

45. 45
UI automation
on mobile devices

46. 46
Available tools/libraries
• Android
• Android 2.3+: Instrumentation
• Android 4.2+: UiAutomator
• selendroid
• Appium
• Robotium
• Calaba.sh
• Xamarin.UITest
• iOS
• UIAutomation
• Appium
• ios-driver
• Calaba.sh
• Xamarin.UITest
• Frank
• KIF (Keep it functional)

47. 47
Selenium: cross platform, existing know how
• Android
• selendroid
• Appium
• iOS
• Appium
• ios-driver

48. 48
Appium (www.appium.io)
• Provides WebDriver API (JSON Wire Protocol)
• Uses native UI Automation library
• Can test native & hybrid Apps, mobile web apps
• Clients for C#, Java, Ruby, Python, PHP, JavaScript
• Works with devices and emulators

49. 49
Examples

50. 50
Automation with SpecFlow and Appium

51. 51
SpecFlow & Appium
• Write acceptance criteria independent of the platform
• Write bindings against a Selenium- like- API
• Run it on a devices or emulator

52. 52
Demo

53. 53
SpecFlow automation
on the
controller layer

54. 54
Different levels of automation
Controller
Business Layer
Data Layer
Model
View
Device
automation
Trigger behaviour
through controller
Assert behaviour
on model, db, ..
Setup pre-conditions
through service
interfaces
Out-of-process
In-process

55. 55
How?
• Reuse the same feature files
• Write new bindings against the
Controller/ViewModel/…

56. 56
Demo

57. 57
Faster feedback from tests
UI Automation Test Controller Test

58. 58
Why is it faster?
• No overhead for deploying and starting the app
• No need to wait for UI animations to finish
• No delays for searching UI controls

59. 59
Advantages Controller Testing
• Faster
• Better testable App architecture
(Xamarin.Forms & MVVM)

60. 60
Controller tests cannot fully replace UI tests
• UI tests are still needed
• Diversity of mobile devices
• Changes of manufacturer
• Different Hardware
 Screen sizes
 Densities
 …
• Automated UI tests are still faster than manual testing

61. 61
Conclusions

62. 62
Excuses for avoiding mobile test automation
• “This is just a small app”
• “Only complex end-end tests would make sense”
• “Maintaining automated tests is too costly”
• Diversity of devices
• Small apps also run into regressions
• Bad feedback on app store cannot be undone

63. 63
Automated testing in mobile App development
• Business readable automated checks
• specification of acceptance criteria
• test first development
• Testability is an ongoing architectural concern
• every new feature needs to be testable
• not every acceptance criterion needs to be tested through UI
• Automated tests don’t replace manual testing entirely
• exploratory testing
• assertions that are hard to automate
• …

64. 64
Questions and further information
Andreas Willich (awi@techtalk.at)
Christian Hassa (ch@techtalk.at)