Implement Combinatorial Test Patterns for Better Mobile and IoT Testing

T10
Track Session
4/21/2016 2:00 PM

"Implement Combinatorial Test
Patterns for Better Mobile and IoT
Testing"

Presented by:
Jon Hagar
Grand Software Testing

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 · 904-278-0524 · info@techwell.com · www.techwell.com

Jon Hagar
Jon Hagar is a systems software engineer and testing consultant, supporting
software product integrity and verification and validation (V&V), with a
specialization in mobile and embedded software system testing. For more than
thirty years, Jon has worked in software testing and engineering projects. He
authored Software Test Attacks to Break Mobile and Embedded Devices;
consults, presents, teaches, and writes regularly in many forums on software
testing and V&V; and is lead editor/author on committees including OMG UTP
model-based test standard, IEEE 1012 V&V plans, and ISO/IEEE/IEC 29119
software test standard. Contact Jon at jon.d.hagar@gmail.com.

Implement Combinatorial Test Patterns for
Better Mobile and IoT Testing
Jon D. Hagar, Consultant,
embedded@ecentral.com
1
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
• Scary stories
• It only takes a few minutes of using an App before users like or hate it
• Worse than that. . .
– Many users will post a poor social media review of the app or device
– You may be on the nightly news (bad press is not good)
– A question I get a lot, “how do we deal with fragmentation?”
• So You want to be
– Part of the billions of devices
» You want to be GREAT
2
The Mobile-IoT Opportunity

What is on the Horizon for Testing
Hype Cycle for Emerging Technologies, 2014
Reference: Gartner report 2014
What We Will Cover
• Introduction and definitions
• Problem example
• A combinatorial test attack pattern
• Some Combinatorial (CT) Tools
– Demo
• Wrap up
4

Basic Definitions
• Test – the act of conducting experiments on something to determine
the quality (ies) and provide information
– Many methods, techniques, approaches, levels, context
– Considerations: input, environment, output, instrumentation
• Quality (ies) – Value to someone (that they will pay for)
– Functional
– Non-functional
– It “works”
– Does no harm
• Are there (critical) bugs?
5
The Mobile-IoT Embedded Space
6
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
Many Options
Huge
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)

• Embedded – Software contained in “specialized” hardware…
• Mobile and handheld devices—small, held in the hand, connected to
communication networks, including
– Cell and smart phones – apps
– Tablets
– Medical devices
• IoT – Internet of Things are traditional devices with software and comms added
• Mobile, Handheld, IoT typically have:
– Many of the problems of classic embedded systems
– The power of PCs/IT
– More user interfaces than classic embedded systems
– Fast and frequent updates
• Devices are “evolving” with more power, resources, apps, etc.
• Mobile and IoT are (currently) the “hot” area of computers/software
You know what they are. . . Right?
Embedded, IoT, Mobile and Handheld?
Test Brakes
What’s this?
• Embedded – Software contained in “specialized” hardware…
– Minimal networking-communication
PLUS
• Mobile and handheld smart devices—small,
held in the hand, highly connected (web, cloud, servers,….)
• IoT – Internet of Things are “traditional” and new
devices with software and communication added
What is an IoT device?
Test Brakes

IoT – 2.5 Main Segments (and a name a few categories)
9
Must be Interoperable
Across segments
Industrial Value
2x Consumer
Industrial 4.0 Middle Consumer
Home
- Security
- Monitoring &
control
- Infotainment
Human
- Health
- Fitness
- Info
Vehicles
- driverless
- monitoring
- Infotainment
Office
- Security
- Energy
- Worker info
Medical
- Health monitor/control
- Records
Retail
- Ordering
- Checkout
- Advertize
Cities/States/Nation
- Health
- Safety
- Info
- Control and Monitor
Transportation
- Vehicles
- Navigation
- Logistics
Worksite/Factories
- Ops
- Control
- Info
Defining Software Capabilities
• James Whittaker defines 4 fundamental capabilities that all
software possesses
1. Software accepts inputs from its environment
2. Software produces output and transmits it to its environment
3. Software stores data internally in one or more data structures
4. Software performs computations using input or stored data
• To this, we expand and refine based on an mobile-IoT context:
– Within time
– Using specialized hardware (as sub of items 1 and 2 above) and control
– Security and privacy
– Different development lifecycle constraints
10

Attack-based Testing Patterns
What is an attack?
• A pattern (of testing) based on a common mode of failure
seen over and over
– Some see this as a negative, when it is really a positive
– Attacks seek the “bugs” that may be in the software
– May include or use classic test techniques and test concepts
• Lee Copeland’s book on test design
• Many other good books
• A Pattern (more than a process) which must be modified
for the context at hand to do the testing
• Testers learn mental attack patterns when
working over the years in a specific domain
Example Attacks
(from “Software Test Attacks to Break Mobile and Embedded Devices”)
• Attack 1: Static Code Analysis
• Attack 2: Finding White–Box Data Computation Bugs
• Attack 3: White–Box Structural Logic Flow Coverage
• Attack 4: Finding Hardware–System Unhandled Uses in
Software
• Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs
• Attack 6: Long Duration Control Attack Runs
• Attack 7: Breaking Software Logic and/or Control Laws
• Attack 8: Forcing the Unusual Bug Cases
• Attack 9 Breaking Software with Hardware and System
Operations
• 9.1 Sub–Attack: Breaking Battery Power
• Attack 10: Finding Bugs in Hardware–Software
Communications
• Attack 11: Breaking Software Error Recovery
• Attack 12: Interface and Integration Testing
• 12.1 Sub–Attack: Configuration Integration Evaluation
• Attack 13: Finding Problems in Software–System Fault
Tolerance
• Attack 14: Breaking Digital Software Communications
• Attack 15: Finding Bugs in the Data
• Attack 16: Bugs in System–Software Computation
• Attack 17: Using Simulation and Stimulation to Drive
Software Attacks
• Attack 18: Bugs in Timing Interrupts and Priority Inversion
• Attack 19: Finding Time Related Bugs
• Attack 20: Time Related Scenarios, Stories and Tours
• Attack 21: Performance Testing Introduction
• Attack 22: Finding Supporting (User) Documentation
Problems
• Sub–Attack 22.1: Confirming Install–ability
• Attack 23: Finding Missing or Wrong Alarms
• Attack 24: Finding Bugs in Help Files
• Attack 25: Finding Bugs in Apps
• Attack 26: Testing Mobile and Embedded Games
• Attack 27: Attacking App–Cloud Dependencies
• Attack 28 Penetration Attack Test
• Attack 28.1 Penetration Sub–Attacks: Authentication —
Password Attack
• Attack 28.2 Sub–Attack Fuzz Test
• Attack 29: Information Theft—Stealing Device Data
• Attack 29.1 Sub Attack –Identity Social Engineering
• Attack 30: Spoofing Attacks
• Attack 30.1 Location and/or User Profile Spoof Sub–Attack
• Attack 30.2 GPS Spoof Sub–Attack
• Attack 31: Attacking Viruses on the Run in Factories or
PLCs
• Attack 32: Using Combinatorial Tests
• Attack 33: Attacking Functional Bugs

In Mobile-IoT Many Example Combinations:
Standards, Interfaces, Protocols, Platforms, Software, and Data Patterns
13
Network-Comm
Many of these Combinations Will Need Testing
Exercise: How should we test these?
(How do you do it now?)
14
How many tests are needed?
Coverage of combinations?
How do we find errors?

Combinatorial Testing (CT) Math Offers Solutions
• CT has long history of Usage
• CT uses many tools
• CT is still underused
• CT has some cool possibilities
• CT should be one of the attack techniques used
• Find out how CT can help your testing
15Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Math-based Testing
Testing is a sampling problem:
How can Math aide testing?
• Test systematically the numbers of devices, configurations,
networks, etc.
• Use sampling in environments and quality control
• Use sampling of data from the input domain space
• Use Big Data Analytics to feed testing
16Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

Pattern Attack 32: Combinatorial Tests
17
• When to apply this attack?
– There are numerous related variables and variable values which
interact
– Validation Analysis Upfront
– Testing throughout the life cycle and in Maintenance Mode
• What faults make this attack successful?
– Untested configuration combinations
– Data “bugs”
• Who conducts this attack?
– Tester, analyst
• Where is this attack conducted?
– Tool running in the lab or field
• How to determine if the attack exposes failures?
– A test fails to meet success criteria
– Hard crash - NIST Data
Attack 32: Combinatorial Test Patterns
18
• How to conduct this attack – basic pattern
– Identify combinatorial situation
– Identify combinatorial tool
– Identify variables
– Identify values
– Identify constraints on values
– Enter variables and values into tool with constraints
– Exercise resulting combinations in usage scenario
tests or automated tests
– Look for failures
– Repeat and refine as needed

• Android
Example Usage:
Numbers of data choices, devices and configurations
19
• Hardware
• IoT
devices
• Data
• Routers
• IoT Home
Protocols
How many Tests?
10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests!
Using the ACTS Combinatorial Tool:
Example
20
Parameters:
Andriod AppPlatform
[Device 1, Device 2, Device 3, Device 4, Device 6,
Device 7, Device 8, Device 9, Device 10]
IoTProtocolHome [true, false]
IoT Devices
[Refrig, Stove, mircrowave, TV, front door, Garage
door, Home gaurd, Stereo, Temp Control, Lights,
Drapes, Water Heater, window openers]
Routers [0, 1, 2, 3, 4, 5]
Comm providers
[Cell1, Broadband, cable, Cell 2, Space based,
Vendor godzilla]
Data [1, 0, -1, 99999, -99999, 100, -200]
Test Case# Andriod AppPlatform IoTsHome IoTDevices Routers
Comm
providers Data
0 Device 1 false Refrig 1 Broadband 0
1 Device 2 true Refrig 2 cable -1
2 Device 3 false Refrig 3 Cell 2 99999
3 Device 4 true Refrig 4
Space
based -99999
4 Device 6 false Refrig 5
Vendor
godzilla 100
5 Device 7 true Refrig 0 Cell1 -200
119 Tests

Other Statistical Tools to Consider
General Technique
Concept
Tool Examples (Note
1)
Examples of where
technique can be used
Specific sub- technique
examples
Combinatorial Testing
ACT [4], Hexawise[5]
rdExpert [6]
PICT[7]
Medical, Automotive,
Aerospace, Information
Tech, avionics, controls,
User interfaces
Pairwise, orthogonal arrays,
3-way, and up to 6 way
pairing are now available
Design of Experiments
(DOE)
DOE ProXL[8]
DOE++ [9]
JMP [10]
Hardware, systems, and
software testing where
there are "unknowns"
needing to be evaluated
Taguchi [12]
DOE
Random Testing
Random number
generator feature
used from most
systems or languages
Chip makers,
manufacturing quality
control in hardware
selection
Testing with randomly
generated numbers includes:
fuzzing and use in model-
based simulations
Statistical Sampling SAS [10]
Most sciences, engineering
experiments, hardware
testing, and manufacturing
Numerous statistical
methods are included with
most statistical tools
Software Black box
Domain Testing
Mostly used in manual
test design, though
some tools are now
coming available [11]
All environments and types
of software tests. These
are “classic” test
techniques, but still
underused
Equivalence Class, Boundary
Value Analysis, decision
tables (Note 2)
Many Variables and Choices
22
And the ACTS tool in real-time (be on the high wire)

Tool Demo
Link To ACTS Tools
23
Expanding Combinatorial Testing
• For Dev-Ops
– Sampling user data
– Model-based testing
– Advanced data selection
• Support domain testing
• Do test without an Oracle
– NIST Study
– Combine with automation
– Run 4-to-6 way combo’s
– Look for Major Crashes
24

Summary
Common Mobile-IoT Problem
• Data selection
• Dealing with numbers of
configurations
– Hardware, Software, Protocol,
etc.
• Testing within time and
budget
Overlooked Solutions
• Data analysis with sampling
– Classic testing
• Combinatorial Testing with
tools
– Test Automation (not a
requirement)
• Reduce combinations to fit
within budget and schedule
25
•
•
•
•
•
•
•
•
•
•
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”

Book List
• “Software Test Attacks to Break Mobile and Embedded Devices”
Jon D. Hagar, 2013
• “How to Break Software” James Whittaker, 2003
– And his other “How To Break…” books
• “A Practitioner’s Guide to Software Test Design” Copeland, 2004
• “Introduction to Combinatorial Testing” D. Richard Kuhn Raghu N. Kacker Yu Lei ,
2013
•
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”

Implement Combinatorial Test Patterns for Better Mobile and IoT Testing

More Related Content

What's hot

Viewers also liked

Similar to Implement Combinatorial Test Patterns for Better Mobile and IoT Testing

More from Josiah Renaudin

Recently uploaded

Implement Combinatorial Test Patterns for Better Mobile and IoT Testing