The document discusses the unique challenges of IoT software testing, highlighting issues such as complex hardware-software integration, data analytics, security, and efficient device configuration testing. Jon Hagar, an expert in software testing, emphasizes the need for improved testing methodologies and tools to address the risks associated with IoT devices. Key challenges include safety, privacy, and the sheer number of test configurations required in this rapidly evolving technological landscape.

1. T20
Session - IoT Testing
4/27/17 3:15 PM
IoT Software Testing Challenges: The
IoT World Is Really Different
Presented by:
Jon Hagar
Grand Software Testing
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - https://www.techwell.com/

2. Jon Hagar
Grand Software Testing
Jon Hagar is a systems software engineer and testing consultant, supporting
software product integrity and verification and validation (V&V), with a
specialization in mobile and embedded software system testing. For more than
thirty years, Jon has worked in software testing and engineering projects. He
authored Software Test Attacks to Break Mobile and Embedded Devices;
consults, presents, teaches, and writes regularly in many forums on software
testing and V&V; and is lead editor/author on committees including OMG UTP
model-based test standard, IEEE 1012 V&V plans, and ISO/IEEE/IEC 29119
software test standard. Contact Jon at jon.d.hagar@gmail.com.

3. IoT Software Testing Challenges:
Is the IoT World Really Different?
Jon D. Hagar, Consultant, Grand Software Testing
embedded@ecentral.com
1
• Industrial Revolution 4 powered by IoT
• IoT - all the problems of Hardware, IT and Mobile
• Worse than that. . .
– IoT Devices Kill
– 4-10 trillion USD in next 10 years?????????????
Copyright 2016 Jon D. Hagar - “Software Test Attacks to Break Mobile and Embedded Devices”
2
The Embedded and IoT Opportunity

4. IoT – 2.5 Main Segments
(to name a few)
3
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Must be Interoperable
Across segments
Industrial Value
2x Consumer
Industrial 4.0 Middle Consumer
Home
- Security
- Monitoring &
control
- Infotainment
Human
- Health
- Fitness
- Info
Vehicles
- driverless
- monitoring
- Infotainment
Office
- Security
- Energy
- Worker info
Medical
- Health monitor/control
- Records
Retail
- Ordering
- Checkout
- Advertize
Cities/States/Nation
- Health
- Safety
- Info
- Control and Monitor
Transportation
- Vehicles
- Navigation
- Logistics
Worksite/Factories
- Ops
- Control
- Info
State of Dev and Testing Today
• Requirements verification checking
- Software Coverage
• Risk–based analysis and testing
• Scientific context driven exploratory testing
NOT ENOUGH
Copyright 2016 Jon D. Hagar - “Software Test Attacks to Break Mobile and Embedded Devices”

5. Today’s Top 5 IoT Test Challenges
1. Complex and unique IoT hardware combined with software
2. Data Analytics
3. Numbers of devices and configurations to test
3. Privacy and Security
4. Communication, Connectivity and Integration
- Third-Party software/hardware
5. Safety
Others (not covered today):
• Lifecycle – unified hardware-software dev-test-ops (a dream)
– Tools to support development, ops and tests
– Cost and schedule
– Concurrent software and hardware development
• Integrated Dev-Ops for IoT
• Ubiquitous User Interface (UI)
• Resource limitations
• CM
• Performance and real-time
• And the list goes on (other presentations)
5Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
5
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
(Cloud)
Many Options
Huge
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)
Cyber-Physical
Systems (today)
Challenge: Complex and Unique Hardware
Combined with Software = IoT
• Is it Top Down
• Is it Bottom Up
• Who is first – hardware, software, dev, ops, test, system
YES
6Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

6. Solution Space
• Agile Software
• Dev-Ops-Test
• Hardware
• Software
• SYSTEMS THINKING
– Modeling and Data
7Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Systems Thinking Implies Modeling
• Hardware and electronics modeled for years
• Systems modeling growing
• Software modeling started
• Formal test modeling use tools
– Infancy
8Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

7. 9
An Example Test Flow with Modeling for IoT
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Challenge: Data Analytics
• Everyone talks about data analytics
– Everyone but most testers
• Testers need Patterns
10Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

8. The Evolution of Computers =>
The Evolution of Data Usage
11Copyright 2015 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
(Cloud)
Many Options
Huge
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)
Data Used
by
The Few
Data Pulled
(from whole web)
App that
Customizes a Data Slice
Limited
Data
(if any)
We do not
know yet
? ? ?
Testers Should Be Looking Towards the Data
• Real-time Usage Data
• Error and performance logging
• Error-Use taxonomies
Test attacks
Tools – AI, Stats, Neural Networks, Self-organizing data…..
12Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Self
Organizing
Tester
TESTERS ARE IGNORING IOT BIG DATA

9. Scared Yet?
Just wait, there is more to come
13Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Challenge: Numbers of devices & configurations
(and how to test efficiently)
IoT is Mobile Fragmentation Problem x 10 (or 100, 1,000?)
Combinatorial Testing
Risk-based testing
Hitting “sweet” spot of market timing
* Warning what is “sweet spot” for IoT?
14Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

10. • Andriod
Math to Rescue:
Numbers of devices and configurations
15
• Routers
• IoT
devices
• Data
• Comm
Channels
• IoT Home
Protocol
How many Tests
to address: data, configurations, devices
comms, resources, integration, resources
10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests
Embedded
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Pattern Example
Using the ACTS Combinatorial Tool
16
Parameters:
Andriod AppPlatform
[Device 1, Device 2, Device 3, Device 4, Device 6,
Device 7, Device 8, Device 9, Device 10]
IoTProtocolHome [true, false]
IoT Devices
[Refrig, Stove, mircrowave, TV, front door, Garage
door, Home gaurd, Stereo, Temp Control, Lights,
Drapes, Water Heater, window openers]
Routers [0, 1, 2, 3, 4, 5]
Comm providers
[Cell1, Broadband, cable, Cell 2, Space based,
Vendor godzilla]
Data [1, 0, -1, 99999, -99999, 100, -200]
Test Case# Andriod AppPlatform IoTsHome IoTDevices Routers
Comm
providers Data
0 Device 1 false Refrig 1 Broadband 0
1 Device 2 true Refrig 2 cable -1
2 Device 3 false Refrig 3 Cell 2 99999
3 Device 4 true Refrig 4
Space
based -99999
4 Device 6 false Refrig 5
Vendor
godzilla 100
5 Device 7 true Refrig 0 Cell1 -200
119 Test
Sample
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

11. Scary:
Most Testers Can Learn
Combinatorial Testing
But few use it
17Paper 14ASTC-0015/2014-01-2148
Challenge: Privacy and Security
(should be on the top of the list and I always talk it?)
• I’ve talked about it for decades
• People on Ski Lifts ask me about this
• We (the industry) are NOT getting better
18Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

12. Security Bugs
• Hacking Pacemakers
• Common programming errors (developer level)
= Zero Day Exploit
• Not Testing in the System environment
• Holes in stress and unusual case tests
• IoT denial of service hack 2016
19Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Privacy – Safety Restricted Data
• Different from security
»More of an issue in some countries legally
• Examples I might not want exposed
20Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

13. Exploratory Testing - Definition
• Quoting James Bach: “The plainest definition of exploratory
testing is test design and test execution at the same time. This
is the opposite of scripted testing (predefined test procedures,
whether manual or automated). Exploratory tests, unlike
scripted tests, are not defined in advance and carried out
precisely according to plan.”
http://www.satisfice.com/articles/what_is_et.shtml
Credit to Jean Ann Harison2013Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Pattern attack–based
For Security/Privacy Use
Attack Pattern-based Testing
What is an attack?
• A pattern (for testing) based on a common mode of failure
or information need seen over and over
– Maybe seen as a negative, when it is really a positive
– Goes after the “bugs” that may be in the software
– May include or use classic test techniques and test concepts
• See Lee Copeland’s book on test design and many other good test
books
• A Pattern (more than a process) which must be modified
for the context at hand to do the testing
• Testers learn mental attack patterns when
working over the years in a specific domain
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”

14. • Attack 28 Penetration Attack Test
• Attack 28.1 Penetration Sub–Attacks: Authentication — Password
• Attack 28.2 Sub–Attack Fuzz Test
• Attack 29: Information Theft—Stealing Device Data
• Attack 29.1 Sub Attack –Identity Social Engineering
• Attack 30: Spoofing Attacks
• Attack 30.1 Location and/or User Profile Spoof Sub–Attack
• Attack 30.2 GPS Spoof Sub–Attack
Security Attacks
(from “Software Test Attacks to Break Mobile and Embedded Devices”)
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Internal Security Testing Hacks Are Only a Start
• External Crowd Source Security Hacking - GOOD
• More internal attacks – BETTER
• Bad guys find “zero” day using attacks
• Data analytics and scientific exploration - BEST
24Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

15. Challenge:
25Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Where &
How
Do we test?
Comm Connectivity Concerns
• Time lag - Speed
• Data correctness
• Different configurations to integrate
• Data completeness
• Product impacted by security hacks
26Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

16. Connectivity Factors to Test
• Test environment Brownouts and dips?
• Interface Protocol options
Which ones?
• Device options
• Chaos Network Engineering
http://principlesofchaos.org/
27Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
• Test on the Production Field System
• Be careful
• Solves system-of-system IoT field testing???
Exploration with Chaos Engineering
Paper 14ASTC-0015/2014-01-2148 28

17. Dev-Testing Options for Connectivity
Test Early
• Model-based analysis
• Math-based analysis
Test Often
• Test labs
• Safety & Security – Sand Box
Test Consistently
• Risk-based testing
• Requirements verification checking
• Automation
29Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Integration Is Hard
• Lesson learned from big systems
– Many failures come from integration bugs
• Testing your IoT device in standalone
May not work in the system
Who owns the integration?
30Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

18. Sub-Challenge: Third-Party software/hardware
• Weakest link
– Who/what will it be?
– How do you communicate, integrate, etc.
31Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
IoT Device Made Up of Pieces/Parts
• Off-the-shelf
– Hardware
– Software
– Edge Device <-> Comm Network (Fog) <-> Cloud
• Many Providers of Off-the-Shelf
• Pieces/parts may change during dev and ops
• Tempting to Trust……. BUT
Verification Evaluation is a MUST (use the data)
32Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

19. Steps in Testing Third Party
• Risk Evaluation
• Selection of vendors
– Assess (test) the decision up front
• Regression test all changes (which will happen)
• Test during “set” based design????
• Verify functions/qualities and explore as an ongoing effort
33Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Challenge: Safety
• IoT will and has killed people
– Medical
– Industrial
– Smart cars
– Smarter cities…………
34Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

20. Safety Pitfalls
• Letting users be the Testers
• Common programming errors (developer level)
• Forgetting the hardware
• Missing environments
• Stress cases and unusual cases missed
• Not doing full system-level testing with hw and sw in loop
35Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Solution Space for Safety
• Spending more time and money
– V&V
– IV&V
• Standards (controversy) and regulations
– Formal testing
– More to come
36Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

21. Mobile-IoT Challenge Summary
• To defeat an enemy, you must know the bug
• The IoT test data is limited,
– What exists has implications
• Improve our Patterns
– IoT Dev-Test Book in 2016 - LeanPub.com - Hagar
• Software will be in very nearly everything
– Testing may be a limiting factor
– Failures will happen but minimize
37Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
References (my favorite books)
• “Software Test Attacks to Break Mobile and Embedded Devices”
– Jon Hagar
– IoT Tests Book in 2016 - LeanPub.com - Hagar
•
–
•
•
•
•
•
–
–
–
–
–
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”

22. More Resources
•
•
•
•
–
•
•
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”