1. 1
30 MINUTEN
Welcome!
How to build your Mobile App
Competence Center?
Webinar Series..
#5 App Testing and Distribution
Remco de Vos, Project Manager
Peter Broekroelofs, CTO
2. Agenda
2
• A Mobile Maturity Model for Your App Center - Summary
• App Test Process and various elements and roles in testing
• Automatic Testing
• Our Approach to testing in various phases (definition, development, acceptance)
• App Distribution
• App Lifecycle Management
3. Companies struggle in transition to strategic mobility
3
Opportunistic Strategic Mobile-First
The majority of companies A small minority of companies Very few, if any, companies
A reactive IT department Mobility Center of Excellence: C-level
attention, self-empowered lines of
business, a responsive IT department
Low
Organization-wide strategic focus
Medium High
Most of the time customers only. Some-
times siloed employee classes, typically
field and sales forces and applications
addressing basic informational interactions
Limited extendibility of architecture
Addresses large subsets of both
dedicatedly and occasional mobile workers
and more sophisticated offerings to
customers. Transactional driven
Affects all mobile workers and internal
activities and sophisticated customer
engagement
Common architecture for mobility Common architecture for mobility and
integrated into most IT business processes
Siloed point solutions Sophisticated administration and
management tools; voice, data and
integrated communications services
Integrated platform capabilities and
ubiquitous connectivity
Few formal policies with decisions heavily
user-influenced
Policy-driven approach for management,
security and compliance
Policy-driven and 'factory' approach to
mobile innovation, re-casting business
workflows
Proportion of companies
Mobile strategy center of
gravity
Level of business model
innovation
Users
Architecture
Technologies
Policies
4. The Opportunistic Market in Transition
Experimentation is giving way to a more thoughtful approach to mobility. Organizations are taking
a step back and rethinking how best to maximize the value of mobility.
3
1
"The early days of mobile adoption were characterized by experimentation and unfettered
departmental demand for mobile apps. These apps, funded by the business, tend to be natively
developed, and are built quickly and cheaply and often without coordination with the rest of the
organization or a view to long-term sustainment and value maximization.
2
Such experimentation and piloting are necessary for organizations to test and learn about mobility, with
the lessons learned that enterprise mobility can show significant business value, and that not embracing
enterprise mobility will put the organization at a competitive disadvantage.
Source:
Gartner,
2013
6. How to grow your
mobile maturity?
6
1. What is your Mobile Strategy Maturity goal?
2. How are you performing currently?
3. What are the gaps?
4. Define gaps that matter most
5. Prioritize and close the gaps
7. 7
1. Strategy and Organization
2. Initiation and Design
3. Security and Management
4. Development and Maintenance
5. Test and Distribution
6. Backend and Integration
Mobile Maturity Model - Assessment Axes
8. 8
Mobile Maturity Model - Level determination
OPPORTUNISTIC
Developing
Aware
STRATEGIC
Optimising
Practicing
MOBILE FIRST
Leading
Opportunistic Strategic Mobile First
Strategy & Organisation
Practice observed 1 ✔ x
Practice observed 2 ✔ x
Initiation & Design
Practice observed 1 ✔ x
Practice observed 2 ✔
Development & Maintenance
Practice observed 1 x
16. App Test and Distribution Axis
16
App Testing
Opportunistic Strategic Mobile First
• Testing is performed on emulators and a
few mainstream devices.
• A test process focusses on requirements
and features in happy flow.
• Testing is performed on a representative
subset of devices.
• Client has a decicated testing discipline.
Testers are involved from project start
and part of the scrum team.
• More attention for exceptions.
• Testing is partly automated with (cross
platform) unit - and functional test tools
on a representative subset of devices.
• Client has implemented testing at
various levels, including e.g. design/
prototype usability testing, integration
testing and performance testing.
Practices Observed
Test definition
Test tooling
Test set
Test process
18. Unit testing Example
function TestSuiteByteArray:UnitTest_createWithOddHexNumber()
local data = ByteArray.newFromHex("0xA20");
Assertions.assertEquals("0x0a20", data:toHex());
end
function TestSuiteByteArray:UnitTest_createWithCapitalAndNonCapitalLettersHex()
local data = ByteArray.newFromHex("0xA20f34");
Assertions.assertEquals("0xa20f34", data:toHex());
end
function TestSuiteByteArray:UnitTest_createByteArraynewFromBase64String()
local data = ByteArray.newFromBase64("og80");
Assertions.assertEquals("og80", data:toBase64());
end
function TestSuiteByteArray:UnitTest_bytesExtremeValuesWorkOk()
local data = ByteArray.newFromHex("0x0080FF"); -- 0, 128 , 255
Assertions.assertEquals(0, data:get(1));
Assertions.assertEquals(128, data:get(2));
Assertions.assertEquals(255, data:get(3));
End
19. Functional testing Example
Scenario: Test the button listener
Given I am on the mainpage
And I go to the "Manual/UI/Button/Button listener" menu
Then I should see "M2AT.08.06.030"
Then I drag from "clickButton" to "pressButton"
And I should not see "Press OK to continue"
Then I drag from "pressButton" to "clickButton"
And I should see "Press OK to continue"
And I press the "OK" button
20. Manual versus automatic testing
Automatic testing Pro's
• Fast verification on every code
commit possible for large
percentage of functionality
• Promotes test early and often
policy, especially for unit
testing
• Investing in testscripts gets
stronger on multiple OS-s and
form factors from the same
source
Advanced
app
solu8ons
|
20
Automatic testing Cons:
• Additional costs, continuously
(every large change)
• Flexibility loss in GUI (costs)
• Requires (regression) test setup
for multiple devices / OS’s
• Covers likely only most
important flows
• Hardly ‘waterproof’; manual
verification remains.
• Break even point for apps with
short lifetime hard to reach
How dynamic to
setup which tests?
21. How do we perform our testing?
Advanced app solutions | 21
• Definition Phase
• Development Phase
• Acceptance Phase
22. When the app goes into beta and the first bug reports arrive…
Advanced app solutions | 22
23. Phases in a project – Definition Phase 1/2
Advanced app solutions | 23
Contract / Project plan
• Definition of target devices /
operating systems
• Definition of ‘Reference Set’ #
• Severity Matrix
• Definition of Severity
# A ‘reference set’ is a selected set of devices to meet the specifications and behavior of the most commonly used devices of
a specific target group.
24. Phases in a project – Definition Phase 2/2
Advanced app solutions | 24
Testplan
• Which tests? (integration, performance, method of acceptance, etc)
• Process of reporting & tools
• Test arrangements
• Distribution during test phase
Scrum Team
• Defines ‘Definition of Ready for test’
• Defines ‘Definition of Done’
• Creates Use Stories with Acceptance criteria, Business Logic & Exceptions for validation in testing
26. Phases in a project – Development phase 1/3
Advanced app solutions | 26
• List of devices (‘reference set’) – testing on devices & compliancy.
• Compliancy list & ticking your devices
• Builds are created and published on internal fast installation location
• Usage of the tool ‘M2Borrow’
• A library of all our devices with OS’s
• Barcode based
• Easy track & trace from borrowers
28. Phases in a project – Development phase 2/3
Advanced app solutions | 28
• Working in JIRA with tickets
• Open > In Progress > Resolved > Closed
• Important: signalize defects as soon as possible.
Example
‘Defini.on
of
Ready’
29. Phases in a project – Development phase 3/3
Advanced app solutions | 29
• ‘Resolved > Closed’
• Note: ‘Resolved > Closed’ is in some cases
assigned to the Product Owner of the customer.
Example
‘Defini.on
of
Done’
30. Phases in a project – Acceptance phase
Advanced app solutions | 30
• Optional specific tests (performance, load/stress, etc)
• Acceptance builds are distributed to the customer (published on fast installation location and/or
TestFlight and in near future on new S2M Appstore.)
• Customer verifies functionality on variety of users (‘Family & Friends’ testing)
• Submission preparation
31. Advanced app solutions | 31
Imagine..
Mismatching team logo for Steaua Bucharest
Blooper intermezzo (3)
32. App Test and Distribution Axis
32
App Distribution
Opportunistic Strategic Mobile First
Ad hoc distribution to the app stores.
Internal distribution via mail or website.
Partial automation of the distribution, including
monitoring. A process with measureable gate
before 'going live' and in sync with multi
channel approach and compelling events.
Use of enterprise stores
App distribution policy fully integrated in the
Customer Lifecyle
Focus on customer engagement.
B2E distribution via integrated enterprise store
for all OS-s. Centrally managed with policy
based device and application management
Practices Observed
B2C distribution: App stores
App Lifecycle Management
B2E distribution: Enterprise store
MAM integration
33. Advanced app solutions | 33
App Distribution
• Final verification testing for submission
• App Stores
• Apple iTunes, Google Play, Windows Market place, Blackberry World
• MDM / MAM (Good, MobileIron, XenMobile)
• Mobile Device/Application Management systems & enterprise app stores
• Lean, fast and easy app distribution mechanism
• Via Service2Media enterprise app store with In House Certificates
34. Add-on: App Lifecycle Management
Advanced app solutions | 34
• Suggestion: use a version control mechanism apart from the stores, like our M2Version service.
• Verifies on every start-up with a back-end if a business rule applies for a specific audience
(app version, OS (version), language/country, etc)
• Allows pop-up messages to specific group of users on start-up for i.e.
• Voluntary updates with a link to a (different) app or URL
• Mandatory updates with a link to a (different) app or URL
• Maintenance or outage messages with optional URL
• Retirements of an app
• Etc.
35. Are you interested in a Mobile Strategy
Maturity Assessment?
anette@service2media.com
35
