The Internet of Things (IoT) is omnipresent. More and more hardware devices get connected and will collect and share huge amounts of data in the near future. This progress will lead to a digital and hyper-connected world. Though, in such growing networks of interconnected things, quality assurance (QA) will become a continuous challenge. Especially aspects like conformance, interoperability and security but also performance and robustness will require an increased attention from QA perspective.

1. Axel Rennoch, Alexander Kaiser, Sascha Hackel
Software Quality Days 2019
16. January 2019, Vienna, Austria
FUNCTIONAL AND NON-FUNCTIONAL TESTING WITH
IOT-TESTWARE

2. 2
• IoT Testing
− Challenges and scope
− IoT test language: TTCN-3
• Project IoT-T
− Eclipse IoT-Testware
− Standardization & Certification
• Summary and outlook
AGENDA

3. 3
• Mirai botnet, October 2016:
− botnet using insecure configured IoT-devices (~100.000)
− attack causes blackout and disruption
(e.g. Amazon, Netflix, Twitter, Github)
• Wannacry, May 2017
− ransomeware affecting the whole world (e.g. hospitals in the U.K.)
• KRACK: Key Reinstallation Attack, October 2017
− Replay attack on Wi-Fi Protected Access protocol
• Spectre and Meltdown, January 2018
− Spectre: vulnerability that allows observable side effects from mispredicted
speculative executions
− Meltdown: hardware vulnerability that allows to read all memory
MOTIVATION FOR QUALITY

4. 4
TRENDS IN IOT

5. 5
IOT ARCHITECTURE
The Three Software Stacks Required for IoT Architectures, Eclipse IoT Working Group, September 2016
telemetry
commands
telemetry
commands

6. 6
 Connectivity
options
 Throughput
 Latency
 Power efficiency
 Packet size
 Wide portfolio of competences required
 Devices (sensors, HW, embedded SW)
 Platforms (Cloud, platform domain knowledge)
 Applications (SW, dashboard, business logic)
TECHNICAL SCOPE
 IoT platforms
 360+ worldwide
 IoT protocols
 Rich selection
 IP-based
 non-IP based
Cellular
netw.
Cellular: 4G, NB-IoT, Cat-M1,EC-GSM…[5G];
non-cellular: Wifi, LoRa, Sigfox ,Zigbee, BLE, … connectivity
MQTT MQTT-SN CoAPAMQP3OPC
UA
XMPPHTTP1 Web
Socket21|2|3
applicationIoT application logic
TCP UDP
SMS
DTLSDTLS
IPv4/IPv6(6loWPAN)
transport
TLS/SSL
IoT services layer services…

7. 7
• IoT devices,
 Mikrocontroller (MCU),
 Gateways (Bosch XDK, IoT starterkits)
• IoT platforms
 RIOT, relayr, Thread, mbed…
 service layer (oneM2M, FiWare)
• IoT protocols
 Constrained Application Protocol (CoAP)
 MQ Telemetry Transport (MQTT)
IoT challenges: complexity, asynchronism, resource constraints, long operation phase
STARTING: TEST OBJECTS
LPWAN
LoRa
oneM2M

8. 8
After the acceptance and system tests there will be
a long operation phase => new test phase „operation“
LONG OPERATION LIFETIME

9. 9
INTEGRATION OF SEVERAL TESTING APPROACHES
IoT
Testing
Software
Testing
System
Testing
Security
Testing
Test
Automation
Protocol
Testing

10. 10
• Less resources needed (time and money)
• Avoid human mistakes due to manually testing
• During test development and execution
• Speed-up of regression tests and product time-to-market
TEST AUTOMATION

11. 11
MULTIPLE TEST CONFIGURATION (SAMPLES)

12. 12
• Toolset (selection of available means)
Protocol tester/monitor (Eclipse Titan, Wireshark)
Test devices (RFID kit, Bluetooth test device)
GUI tester (Selenium, SikuliX, Chrome headless)
Web services tester (soapUI)
…
• Public Testsuites (in development)
 Application of a standardized notation
 Abstract and platform-independent
TESTWARE

13. 13
IOT TEST LANGUAGE

14. 14
• TTCN-3 is the Testing and Test Control Notation
• Internationally standardized testing language for formally defining test
scenarios.
• Designed purely for testing
CHALLENGE TEST AUTOMATION
testcase Hello_Bob () {
p.send(“How do you do?“);
alt {
[]p.receive(“Fine!“);
{setverdict( pass )};
[else]
{setverdict( inconc )} //Bob asleep!
}
}

15. 15
• One test technology for different tests
 Distributed, platform-independent testing
 Integrated graphical test development, documentation and analysis
 Adaptable, open test environment
• Areas of Testing
 Conformance and functional testing
 Interoperability and integration testing
 Real-time, performance, load and stress testing
 Security testing
 Regression testing
• Used for system and product qualification and certification
DESIGN PRINCIPLES OF TTCN-3

16. Eclipse IoT-Testware
THE IOT-T PROJECT

17. 17
© Fraunhofer FOKUS
IOT QUALITY
1.
Conformance
2.
Robustness /
Security
3.
Performance
Protocol conformance
Test Suites

18. 18
Take available software and tools …
… and adding public testuites as a result of insights from IoT testing:
IOT-TESTWARE
…
https://projects.eclipse.org/projects/technology.iottestware

19. 19
• Supplement to running and active Eclipse projects
 Paho, OM2M, Titan
• New project at Eclipse Foundation:
https://projects.eclipse.org/projects/technology.iottestware
 TTCN-3 test suites for CoAP, MQTT, OPC-UA, LoRa?
• Assured licenses for users
• Currently in cooperation with
relayr GmbH, Ericsson, LAAS/CNRS, itemis AG, Spirent Communications,
Easy Global Market, Iskratel/Sintesio, …
THE ECLIPSE PROJECT

20. 20
SAMPLE TESTSUITE STRUCTURE: MQTT
 Broker as SUT
 All mandatory message data fields
 Regular and illegal data
(Fixed/variable header, payload)
 Protocol features
 General
 Connect/disconnect (session)
 Subscribe/unsubscribe
 Immediate publish
 Last will and Testament (LWT)
 Heartbeats keepAlive values
 Topic
 Error handling
 Client as SUT
 …

21. 21
TEST DEVELOPMENT SAMPLE: MQTT
TESTZIEL-KATALOG
 Test configurations
 Test Suite Structure
 Test purpose (catalogue)
 Test implementation (TTCN-3)

22. 22
MQTT BROKER EVALUATION (CONFORMANCE, APRIL 2018)
Broker PASS FAIL INCONCLUSIVE
# % # % # %
HiveMQ ? 39 86,67% 4 8,89% 2 4,44%
Mosquitto 1.4.15 38 84,44% 5 11,11% 2 4,44%
VerneMQ 1.3.1 37 82,22% 5 11,11% 3 6,67%
EMQ 2.2 35 77,78% 8 17,78% 2 4,44%
lannister ? 31 68,89% 12 26,67% 2 4,44%
ActiveMQ 5.15.3 29 64,44% 14 31,11% 2 4,44%
aedes v0.33.0 26 57,78% 17 37,78% 2 4,44%
RSMB ? 26 57,78% 17 37,78% 2 4,44%
RabbitMQ 3.7.4 19 42,22% 26 57,78% 0 0,00%
Mosca 2.8.1 19 42,22% 24 53,33% 2 4,44%
HBMQTT 0.9 17 37,78% 28 62,22% 0 0,00%
Moquette 0.10 16 35,56% 29 64,44% 0 0,00%

23. 23
© Fraunhofer FOKUS
BROKER CONFORMANCE OVER TIME
40 38 39 37 36 35
19 19
3 5 3 5 7 8
24 24
2 2 3 3 2 2 2 2
0
5
10
15
20
25
30
35
40
45
50
Mosquitto
1.4.14
Mosquitto
1.4.15
VerneMQ
1.1.0
VerneMQ
1.3.1
EMQ 2.0 EMQ 2.2 Mosca 2.5.1Mosca 2.8.1
Pass Fail Inconclusive
Testcases
-2 -2 -1 0

24. 24
• Vulnerability scanner:
 in particular for web applications, zero-day/fuzzing, consideration of data
bases, traffic/network analyser, program code scanner
• Penetration tester, e.g. “SQL injection”
• Intrusion detection tools
• Load test/Scalability
• Further utilities: Model-based testing (UML testing profile) and risk modelling
TESTWARE: SECURITY

25. 25
FUZZING APPROACH
CoAP
ATS
CoAP
ETS
Fuzzed
Data
SUT
ATS: Abstract Test Suite
ETS: Executable TS

26. 26
Results for CoAP:
- Initially, 4421 fuzzed test data for CoAP were generated
- After sending the data to a (local) CoAP server, it crashed after date “1107”
https://www.fokus.fraunhofer.de/de/sqc/security_testing
https://github.com/fraunhoferfokus/Fuzzino/blob/master/doc/Fuzzino_XML_Description.pdf
FUZZINO RESULTS AND RESOURCES

27. 27
PERFORMANCE TESTING: HIGH-LEVEL VIEW
Eclipse Titan
RIoT
SUTLoad
htop

28. 28
© Fraunhofer FOKUS
PERFORMANCE TESTING: WORKFLOW
(1)
(1) Create/choose State Machine
(2) Configure scenario
(3) Start Performance Test Suite
a. Observe RIoT verdicts
b. Observe IUT process with htop
(CPU / Mem. Usage)
(2)
Titan /
RIoT
SUT
htop
(3)
Load

29. 29
1. Control Logic
1. State Machine
2. Configuration Files
2. Applib
1. CoAP / MQTT / LwM2M / HTTP
2. Model-base Testing
3. CLL
1. Loadtest Framework
2. Functiontest Framework
3. Etc.
© Fraunhofer FOKUS
RIOT UNDER THE HOOD
TTCN-3
&
Titan
User
space

30. 30
COAP STATE MACHINE: EXAMPLE
S0 S1
Start Traffic Case
CoAP rsp: 2.05: verdict = pass
CoAP rsp: not 2.05: verdict = fail
Timeout : verdict = fail
© Fraunhofer FOKUS
S0: idle state
S1: initiated, test case execution of => CoAP GET /some/resource/123

31. 31
COAP STATE MACHINE: EXAMPLE CONT.
S0 S1
init
© Fraunhofer FOKUS
S2: CoAP GET /some/resource/123 S4: CoAP DELETE /some/resource/123
S3: CoAP PUT /some/resource/123
S2
set verdict
!2.05
S3
!2.01
S4
2.02
2.05
2.01
rnd*
rnd*
Scenario:
„randomly GET / PUT or DELETE
a single resource from multiple/
parallel CoAP Clients“4.04

32. 32
Source: Eclipse Titan user guide
© Fraunhofer FOKUS
EXCURSUS: TTCN-3 PARALLEL EXECUTION

33. 33
TS
COAP PERFORMANCE: EXAMPLE
© Fraunhofer FOKUS
SUT
PTC
PTC
…
MTC
CoAP: Req / Resp
GET | PUT | DELETE

34. 34
(ALMOST) COMPLETE IOT-TESTWARE
…
SUT

35. Standardization & Certification
THE IOT-T PROJECT

36. 36
• New Working Group (TST) will develop
IoT test catalogues and specifications (not covered elsewhere)
• The types of testing include
conformance, interoperability, security and performance testing
• The initial technical focus will be:
− IoT network layer
(communication protocols, node connectivity, edge computing etc.),
− Basic security of IoT devices
ETSI TC MTS

37. 37
MTS TST WORK PROGRAMME
IEC 62443-4-2
CoAP
MQTT
LoRaWAN
Vul. database
https://portal.etsi.org/tb.aspx?tbid=860&SubTB=860

38. 38
BASE SECURITY CERTIFICATION SCOPE
The Three Software Stacks Required for IoT Architectures, Eclipse IoT Working Group, September 2016
IoT-Testlab Scope
(basic security level
certification)
telemetry
commands
telemetry
commands

39. Putting everything together
SUMMARY AND OUTLOOK

40. 40
IOT-TESTWARE
BIG PICTURE
…
ETS
Reporting
Logging
TP: Test Purpose
TSS: Test Suite Structure
ATS: Abstract Test Suite
ETS: Executable TS
SUT: System Under Test
TSS TP ATS

41. 41
Advanced testing technology:
 Open source IoT-Testware (code):
 External (open source) SW
Standardized IoT test purposes:
SUMMARY

42. 42
• Adding more protocols to IoT-Testware
AMQP, LWM2M, 6LoWPAN, LPWAN
• Increased security level for certification
• Cooperation/liaisons (in preparation) with
ETSI TC Cyber/SmartM2M, oneM2M, OPC Foundation ...
OUTLOOK

43. 43
CONTACTS
Thank you for your attention!
https://www.fokus.fraunhofer.de/en/sqc
https://relayr.io/