This document discusses testing for Internet of Things (IoT) devices. It describes the challenges of testing IoT hardware, firmware, software and cloud integration. It proposes building a modular, scalable and reliable end-to-end automated testing environment to test IoT devices and scenarios. Example test cases are provided to demonstrate turning lights on and off from a mobile app and wireless switch. Vision system and test automation approaches are also summarized.

1. Testing in
the world of
Internet of
Things
Dawid Pacia
Tomasz Janiszewski

2. “
The best way to predict the future
is to invent it.
Alan Kay, Xerox PARC, 1971

3. What is IoT?
The Internet of Things (IoT) is the
network of physical objects - devices,
vehicles, buildings and other items
embedded with electronics, software,
sensors, and network connectivity - that
enables these objects to collect and
exchange data.
https://en.wikipedia.org/wiki/Internet_of_Things

4. What is going to be the next
"big thing"? IoT?

5. Silvair products
● Based on TI CC2540/41 &
Nordic nRF51/52
● Bluetooth Low Energy v4.0
● Radio range: up to 500 meters

6. Silvair products

7. Silvair products
● Android 5.0+ app
● iOS 9.0+ app

8. Testing Scope &
Challenges
Hardware
○ Drivers
○ Switches
○ Sensors
○ Luminaries
○ Motors
○ Light sources / bulbs
○ Multi-protocol
gateways
Firmware
○ Bluetooth Low
Energy v4.0
protocol support
○ Silvair Smart Mesh
protocol support
○ Provisioning
○ Discovery
○ Groups and Scenes
○ OTAU
○ High Security
Software
○ Network setup
○ On / Off, dimming, color change
○ Associations
○ Factory Reset
○ OTAU
○ Cloud

9. “In the beginning there was
only Chaos”
● Event-driven development (for CES, BT World, etc.)
● Chaotic environment with constantly changing
priorities (on a daily basis)
● Lack of time to test SF/HW/FW on all test levels
● Ignoring product/project risk analysis
● No TDD/FDD/XP, lack of Unit tests
● PoC….. PoC everywhere
● No documentation
● Not enough testers!

11. Main focus
● Building and maintaining test environment
● Test automation
● Helping other teams with their testing activities

12. Let’s imagine a simple test
scenario...
Story: The user turns on a light bulb from an app
and turns it off from a wireless switch
Step 1: Launch mobile app and create a home
network with 1x wireless switch and 2x light
bulbs
Step 2: Turn on the bulbs from an app
Step 3: Turn off the bulbs from a switch

13. Flashing
App
installation
Devices
pre-configuration
START Init
Test scenario flow:
Changing
bulb state
(Step 2)
Switch
control
(Step 3)
Test suite
execution
Closure
activities
Creating “Home”
network
(Step 1)
END

14. Simple… right?
Well... not quite...

15. ○ How to flash a BLE module with the firmware
which is inside a bulb or a switch?
○ How to control hardware buttons on a switch on
demand?
○ How to verify whenever a bulb has been turned on
or not?
○ How to verify if desired color, brightness, color
temperature or saturation level has been reached?
Challenges:

16. ○ How to combine the results from multiple
sources (mobile app, firmware, hardware, cloud)
and be sure that the test cases is passed and
functionality under test is working as expected?
○ How to to know which component has failed
and when?
○ How to detect potential problems like
unpleasant LED flickering while dimming?
○ How to reset everything to default state at the
end of the test scenario?
Challenges:

17. Hardware
Firmware
Embedded
platforms
Cloud
Vision
system
Continuous
integration
Electric
circuits
Mobile
applications
AUTOMATIZATIONAutomatic
reporting
REST
Multiple
platform
s

18. Want big
impact?
Use big image.
THEY SAID THIS JOB WOULD BE EASY
YOU GOTTA BE KITTEN ME

19. Let’s make it
step by
step !
● Following common methodology
● Facilitating external processes
● Automation tools freedom
● Considered architecture
● Brainstorming sessions
● Team autonomy

20. “...Possibilities kills creativity...”
Requirements of IoT E2E Test
Environment
● Dirt-cheap!
● Simplicity - built with a KISS
("Keep it simple, stupid!")
methodology, extremely
low entry point for system
newcomers
● Modularity - the system
should handle flawlessly a
multiple of different DUT’s
(Devices Under Test)

21. Requirements of IoT E2E Test
Environment
● Scalability - a multiple
instances of DUT modules
and Test Arbiters should be
able to operate
simultaneously without any
hassle
● Reliability and stability - the
testing environment should
be able to operate with a
MTBF of 7 days
“...Possibilities kills creativity...”

22. Let’s prototype!

23. Every engineering
starts with LEGO

24. Cloud
REST API
SSH
UART
Module Flashing
M
A
S
T
E
R
SSH
REST
API
B
L
E
DEBUGGER UART
FW DB
AUTH
TOKEN
/ REST
B
L
E
DEBUGGER

26. Switch Control

27. class switchTest(unittest.TestCase):
self.authorization = ["rpi-devDav.local","qateam","123"]
…
self.channelOff2 = self.config["channelOff2"]
def testOnOff(self):
switch = switchLib(self.config["switchAddress"], self.config["login"],
self.config["password"], self.config["platform"], self.channelOn1,
self.channelOn2, self.channelOff1)
switch.switchClick(self.channelOn1)
…
switch.switchHold(self.channelOn1,3)
switch.switchHold(self.channelOff1,3)
switch.switchWakeUp()
switch.factortReset()
def tearDown(self):
pass
Switch Control

28. Vision System

29. The very first thing - prepare your
workstation!
● Unchangeable
conditions
○ Lighting (!)
○ Filters
● Stability (!)
● Usability
○ Effort in
reconfiguration
○ Reusage
● Costs

30. The very first thing - prepare your workstation!

31. Trigger
signal
Set proper
parameters /
disable automatic
Send
output
How it really looks like:
Take a snapshot,
image
preprocessing
(filtering)
Kill the
processIs camera
buffer
released?
Image
processing (blob
detection, check
elements, etc.)
Thresholding
Create
binary ROI
matrix
Trigger
signal
Gather value
matrix
Send
output
How we would like it:
Vision System

32. v4l2-ctl -c exposure_auto=1 | v4l2-ctl -c exposure_auto_priority=0 | v4l2-ctl
-c white_balance_temperature_auto=0 | v4l2-ctl -c absolute_exposure=171
Set proper parameters
(always manually!)
*Collection of libraries which adds a thin abstraction layer on top of video4linux2 devices. The purpose is to make it easy for
application writers to support a wide variety of devices without having to write seperate code for different devices in the same class.
brightness (int) : min=0 max=255 step=1 default=128 value=128
contrast (int) : min=0 max=255 step=1 default=32 value=32
saturation (int) : min=0 max=255 step=1 default=32 value=32
white_balance_temperature_auto(bool) : default=1 value=0
gain (int) : min=0 max=255 step=1 default=0 value=0
power_line_frequency (menu) : min=0 max=2 default=2 value=2
white_balance_temperature (int) : min=0 max=10000 default=4000 value=4000
sharpness (int) : min=0 max=255 step=1 default=24 value=24
backlight_compensation (int) : min=0 max=1 step=1 default=1 value=1
exposure_auto (menu) : min=0 max=3 default=3 value=1
exposure_absolute (int) : min=1 max=10000 default=166 value=171
exposure_auto_priority (bool) : default=0 value=0

33. ...and this is why
[109, 5, 2,
6, 2, 3,
3, 3, 104]
Good Exposure
(value 171)
[173, 58, 33,
54, 47, 51,
38, 48, 169]
Overexposure
(value 800)

34. Image preprocessing
(filtering)
filtering (blur),
thresholding
noise reduction,
morphological
(erosion)
(146, 193, 255)
(255, 64, 17)

35. Response analysis

36. Purpose of the extended
blob detection
# Filter by Area.
params.filterByArea = True
params.minArea = 1500
# Filter by Circularity
# (4 * PI * Area) / (Perimeter^2)
params.filterByCircularity = True
params.minCircularity = 0.6
# Filter by Inertia
# It measures how elongated a shape is
params.filterByInertia = True
params.minInertiaRatio = 0.8
# Filter by Convexity
# Area of the Blob / Area of its
# convex hull
params.filterByConvexity = True
params.minConvexity = 0.87

37. Purpose of the extended
blob detection - example
BAD VERY BAD OK

38. QA CI
Environment
Test Arbiter
JIRA TEST
RESULTS
iOS iOS
APK APK Cloud
TEST
RESULTS
E2E Test
Environment
Appium Appium

39. Test Arbiter
class Silvair(unittest.TestCase):
driver = None
app = None
@classmethod
def setUpClass(cls):
…
…
cls.restAPIClientBulb1 =
RestAPIClient(cls.config["restAPI_IP_bulb1"],cls.usernameRestAPI,cls.passwordRestAPI,
cls.projectName)
cls.restAPIClientBulb2 =
RestAPIClient(cls.config["restAPI_IP_bulb2"],cls.usernameRestAPI,cls.passwordRestAPI,
cls.projectName)
cls.restAPIClientSwitch =
RestAPIClient(cls.config["restAPI_IP_switch"],cls.usernameRestAPI,cls.passwordRestAP
I,cls.projectName)
cls.restAPIClientBulb1.fullFlashingFlow(cls.config["bulbOrderID"])
cls.restAPIClientBulb2.fullFlashingFlow(cls.config["bulbOrderID"])
cls.restAPIClientSwitch.fullFlashingFlow(cls.config["switchOrderID"])

40. Test Arbiter
...
print("[LOG] " + datetime.now().strftime("%Y-%m-%d %H:%M:%S") + " Appium
server started.")
downloadFromS3(cls.config["s3ApkPath"],
os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + "/apk",
cls.config["APP_NAME"])
cls.driver = create_driver('Android', configFile, bool(int(cls.config["reinstall_app"])))
cls.driver.implicitly_wait(5)
cls.bulbMac1 = cls.config["bulbMac1"]
cls.bulbMac2 = cls.config["bulbMac2"]
cls.switchMac = cls.config["switchMac"]
cls.retryNumber = int(cls.config["retryNumber"])
cls.bulbSlot1 = int(cls.config["bulbSlot1"])
cls.bulbSlot2 = int(cls.config["bulbSlot2"])
cls.bulbProvisioningWorkaround = cls.config["bulbProvisioningWorkaround"]
cls.factoryResetWorkaround = cls.config["factoryResetWorkaround"]
cls.HWbulbWorkaround = cls.config["HWbulbWorkaround"]

41. Test Arbiter
...
cls.androidLib = android(cls.driver)
cls.bulbEnv = BulbEnv(cls.config["deviceNumber"],
cls.config["resolutionX"], cls.config["resolutionY"], cls.config["rowsNum"],
cls.config["columnsNum"])
cls.switchEnv = switchLib(cls.config["switchAddress"], cls.config["login"],
cls.config["password"], cls.config["platform"], cls.config["channelOn1"],
cls.config["channelOn2"], cls.config["channelOff1"])
cls.app = Pages(cls.driver, cls.switchEnv)
cls.e2eEnv = e2eLib(cls.app, cls.bulbEnv, cls.switchEnv,
cls.config["minThreshold_on"], cls.config["maxThreshold_on"])

42. Number of commits 212
Number of code lines added 69293
Number of code lines deleted 46028
Number of Github
contributors
3
Number of Jenkins jobs 18
Number of builds 777
QA CI Environment
QA CI
Environment
Test Arbiter
JIRA
TEST
RESULTS
TEST
RESULTS
DUT DUT DUT

44. def test12_associations(self):
self.app.clickDeviceOnAddedList(self.switchMac)
self.switchEnv.switchWakeUp()
self.e2eEnv.switch_turnOffBulbs(self.config["channelOff2"], [self.bulbSlot1,
self.bulbSlot2], 0, self.retryNumber)
self.app.doAssociation(self.switchMac, 0, "none")
self.switchEnv.switchWakeUp()
self.e2eEnv.switch_turnOffBulbs(self.config["channelOff2"], [self.bulbSlot1,
self.bulbSlot2], 0, self.retryNumber)
self.app.swipeUp()
self.app.doAssociation(self.switchMac, 1, self.bulbMac1, "Lightness")
self.driver.back()
Tests creation? Couldn’t be
simplier !

45. def test18_FRswitchHW(self):
self.androidLib.resetBT()
self.switchEnv.switchWakeUp()
self.app.addDevice(self.switchMac)
self.app.findAddedDeviceOnList(self.switchMac)
self.e2eEnv.switch_FR(self.switchMac, 5)
def test08_provisioningOfNodOn(self):
self.switchEnv.switchFactoryReset()
self.switchEnv.switchWakeUp()
self.app.addDevice(self.switchMac)
self.app.findAddedDeviceOnList(self.switchMac)
Tests creation? Couldn’t be
simplier !
def test14_sharing(self):
self.app.share("Gmail")
self.app.share("Save to Drive")

46. USEYOURBRAIN!
IoT BAT

47. ● Long time SSH connection (fast switch state
change)
● PWM high frequency (UART testing)
● Electrostatic Discharge
● Busy camera buffer
● Appium server / flashing / setting devices test
steps timeouts
● Different screen sizes / language versions /
platforms
but where is a problem...
Just a few problems met...

48. ● Long time SSH connection
● PWM high frequency (UART
testing)
● Electrostatic Discharge
● Busy camera buffer
● Appium server / flashing /
setting devices test steps
timeouts
● Different screen sizes /
language versions /
platforms
...there is a solution !
● Cert
Authorization
● I/O stream log
deletion
● Humidifier
● Monitoring camera
process and usage
● Moved as
set-up activity
● Gather necessary
informations and
determine while class
initiation

49. … unfortunately, not for all
of them :(

50. Performance Mobile Phone
Number of ADV packets received from a Bulb
within 15 minutes The percentage of
packets received
[%]Clean RF environment
Noisy RF environment
(1200 packets per
second)
VERY GOOD Galaxy Tab 7.0 (T230) 3921 2312 59
GOOD
LG G2 3395 1594 47
Nexus 6 5294 1948 37
Samsung Galaxy S6 5818 2130 37
AVERAGE
Samsung Galaxy Note 4 5070 1607 32
Nexus 9 5377 1541 29
Sony Xperia Z2 5862 1604 27
Samsung Galaxy S5 5784 1494 26
Nexus 5 5964 1518 25
BAD
Samsung Galaxy S5 mini 6397 1197 19
Samsung Galaxy Note 10.1 5268 751 14
Samsung Galaxy S3 6018 641 11
VERY BAD
Xiaomi Mi3 3237 278 9
LG G2 mini 4081 328 8
Samsung Galaxy A3 4050 319 8
Samsung Galaxy S4 mini 4037 329 8
HTC One M8 3868 279 7
Sony Xperia T3 4040 283 7

51. Takeaway...
● Does Scrum really work? Yes, it does!
● Think big, but act small
● Iterate and refine
● Unleash team creativity
● Focus on interoperability
● Modularity is the key
● Incorporate RBT - focus on risky functionalities,
e.g. OTAU
● Physics is ruthless
● IoT is also hardware
● Security!
Want to know more about IoT protocols? Just grab
our free ebook at: https://silvair.com/media

52. Thanks!
Any questions?
You can find us at:
Dawid Pacia:
◎ e-mail: paciadawid@gmail.com
◎ / paciadawid
Tomasz Janiszewski:
◎ e-mail: janiszewski.tomasz@gmail.com
◎ / tjaniszewski
https://silvair.com/careers