#Interactive Session by Pradipta Biswas and Sucheta Saurabh Chitale, "Navigating the IoT Performance Testing Landscape" at #ATAGTR2023. #ATAGTR2023 was the 8th Edition of Global Testing Retreat. To know more about #ATAGTR2023, please visit: https://gtr.agiletestingalliance.org/

1. December 9, 2023
Navigating the IoT Performance
Testing Landscape
Sucheta Chitale
Pradipta Biswas
Global Testing Retreat #ATAGTR2023
2, 3 & 8, 9 December 2023

2. Speaker Introduction
Performance Test Manager,
Sogeti, Capgemini
Performance Test Lead,
Sogeti, Capgemini
Sucheta Chitale Pradipta Biswas

3. Agenda
01 What is IoT ?
02 Quick Stats!
03 Performance Testing in IoT
04 Difference between Web-based PT and IoT PT
05 IoT Performance Testing – Focus Areas & tools
06 NFRs for different Use Cases
07
Challenges and best practices
08
Diagrammatic representation
09
Example

4. What is IoT ??

5. IoT Applications & Types
Consumer IoT Industrial IoT
Wearables
Fitness bands
Smart gears
Smart home
Manufacturing
Pharmaceuticals
Supply Chain
Healthcare
Oil & Natural Gas Mining
Logistics
Agriculture

6. Quick Stats
29B
Devices by 2027 acc. Statista
483 $B
Spending by 2027 acc. Statista

7. Performance Testing in IoT
Performance testing in IoT (Internet
of Things) is a crucial aspect of
ensuring the reliability, scalability,
and efficiency of IoT systems.
IoT performance testing is the
process of evaluating the
performance and responsiveness of
IoT devices, networks, and
applications under various
conditions. It aims to ensure that
IoT systems can handle the
expected load, maintain reliability,
and provide optimal user
experiences.
Scalability
Key Objectives
Reliability
Latency and Responsiveness
Throughput
Security
Resource Utilization

8. Difference between Web based PT and IoT PT
Key Differences Web-based PT IoT PT
Communication Protocol HTTP / HTTPS, FTP, SMTP, JDBC, etc. MQTT, CoAP, AMQP, UDP, XMPP, HTTP, etc
Simulation End user behaviour simulation Simulation of devices/ sensors
Types of Workloads
considered
Usually, the user-initiated workload is
considered
Device-initiated and IoT platform-initiated
workloads are considered
Requests / Responses
Users create the requests and receive
the responses
IoT devices create requests and receive responses
as well as requests, and they also provide responses
Performance Test Coverage Application Layer Physical, Network and Transport layers
Big Data and BI
Not all web applications require
performance analysis of Big data
processing and BI analytics
Performance analysis of huge volumes of data from
IoT devices are analyzed using Big data, BI and
analytics solutions for intelligent decision making in
all IoT applications

9. End to End
IoT Performance Testing
API
Performance
Device Simulation
Gateway Scalability
Simulation of Different
Network Speeds
Volume performance
assessment, batch jobs ,
Interface, reports performance
End User Customer Experience,
Mobile App, Front End
Performance Experience
IoT Cloud Platform Performance,
Scalability, Resilience
IoT Performance Testing – Focus Areas
Neoload
OpenText LoadRunner
JMeter
Locust
IoTify
Bevywise IoT Simulator
AWS IoT Simulator
Azure IoT Device Telemetry Simulator
Tools

10. Connected Factories
Medical Devices Monitoring
Remote Fleet Management
Smart Grid/Meters
 Use Case: Industry 4.0, Predictive maintenance,
Preventive Maintenance, Equipment Uptime, Connected
Floor
 NFRs: IoT Platform Stability, Scalability & Reliability.
Concurrent Assets, Data transmission through Various
networks
 Use Case: Track Vehicle Location, Monitor Vehicle Health,
Monitor Parameters such as speed, fuel, brakes etc.
 NFRs: IoT Platform Stability, Scalability & Reliability.
Concurrent Vehicles, Vehicle To Cloud Transmission,
App/Dashboard Performance Data transmission through
Various networks
 Use Case: Remote metrics management, safety, billing &
metering
 NFRs: Billing Systems Performance, Responsiveness through
different network conditions, Multi meter connectivity,
Integrations
 Use Case: Remote Device management,
Upgrades, Multi device connectivity
 NFRs: Concurrent devices connectivity,
Responsiveness through different network
conditions, device upgrades
NFRs for different Use Cases

11. Performance Testing for MQTT using JMeter
Simulation On-premise
LG
Cloud LG PUBLISH A
MESSAGE
SUBSCRIBE TO
A TOPIC
APPLICATION UNDER TEST
APP SERVER
DB SERVER
MQTT BROKER

12. Example of an IoT Performance Testing
Scenario: Smart Home Security System
System Components:
IoT Cameras: Installed around the house to monitor security.
IoT Sensors: Detect motion and door/window openings.
Central Hub: Collects data from cameras and sensors and
controls the system.
Mobile App: Allows users to monitor and control the
security system remotely.
Performance load testing: simulate large number of users
accessing the mobile app simultaneously, checking live camera
feeds and receiving real-time alerts from sensors.
Performance Stress Testing: Gradually increase the number of
IoT devices connected to the hub beyond its recommended
capacity. Continuously monitor system behaviour.
Network Performance Testing: Simulate varying network speeds
and connectivity interruptions to assess how the IoT devices and
the mobile app handle such scenarios.
Performance Scalability Testing: Gradually increase the number
of IoT devices and sensors in the system to evaluate its ability to
scale both vertically (adding more resources to a single device)
and horizontally (adding more devices).
IoT performance testing is crucial for ensuring the reliability and
functionality of IoT systems, especially in scenarios where real-time data and
responses are critical, such as smart home security systems. By conducting
thorough performance testing, we can identify and address potential issues
before they impact users or result in system failures.

13. Challenges & Best Practices in IoT PT
Scalability
Diversity of
Devices and
Protocols
Data volume,
validation
and Integrity
Edge and Fog
Computing
Real-Time
Analytics &
Insights
Firmware
Updates
Security
Challenges
Handle large
volume of data
and
connections
simultaneously.
Require
support for
various
communication
standards.
Simulate large
volumes of
data at a high
velocity.
Evaluate the
performance of
these
distributed
processing
components.
Assessing
speed and
accuracy of
analytics.
Firmware
updates in IoT
devices.
Vulnerability to
security threats
& performance
degradation.
Create realistic
workloads that
simulate
expected real-
world
scenarios.
Simulate
various network
conditions,
ensure
compatibility,
performance
across the
spectrum.
Validate data
integrity.
Test the
performance of
edge
computing
components,
implement OTA
updates.
Generate
actionable
insights based
on the
collected IoT
data.
Test with
firmware
updates,
Rollback,
Fallback
mechanism.
Integrate
security testing
into
performance.
Best
Practices

14. THANK
YOU