Presentation from meetup "IoT Under Test" organized by Silvair. Łukasz shared with you: ✅ how testing IoT product differs from a simpler tech stack (and how it's similar) ✅ what's a good model of thinking about the quality ✅ how we @ Silvair approach challenges of Quality Assurance

1. Keeping IoT stack in quality check

2. Keeping IoT stack in quality check
Łukasz Dudek
Software Engineer & Delta Team Leader
lukasz.dudek@silvair.com

3. Agenda
1. Massive stack challenge
2. Round Earth model
3. Testing components
4. Soft quality checkpoints
5. Beyond our product
6. Summary

4. Massive stack
●
Embedded firmware
●
Licensing platform
●
MaTE (flashing device)
●
Commissioning platform
●
Mobile app
●
Monitoring backend
●
Gateway

5. Massive stack – also in tests!
●
Embedded firmware
●
Licensing platform
●
MaTE (flashing device)
●
Commissioning platform
●
Mobile app
●
Monitoring backend
●
Gateway

6. Massive stack = challenge
●
Many:
– Components
– Technologies
– Connection points

7. Round Earth model
●
Test strategy
●
Concentric spheres
– Development concerns “inside”
– UX concerns “outside”
– Risk grows outwards
●
Test pyramid alternative
– Not only “more UTs than E2Es”
Image source: https://phys.org/news/2015-12-earth-layers.html
Idea source: https://www.satisfice.com/blog/archives/4947

8. Round Earth model - layers
●
Surface = what users see
●
Sub-systems / services
●
Units / Components
●
Environment / framework
●
Also:
– User input
– Data flows
– Configuration
Image source: https://phys.org/news/2015-12-earth-layers.html
Idea source: https://www.satisfice.com/blog/archives/4947

9. Round Earth model - outtakes
●
Bottom-up quality
●
Automate low-level checks
●
Encourage penetrability
●
Push tests down
●
Less top-level tests
Image source: https://phys.org/news/2015-12-earth-layers.html
Idea source: https://www.satisfice.com/blog/archives/4947

10. Round Earth model - outtakes for IoT
●
Huge UI layer
– GUIs
– Light / diodes
– Buttons / switches
●
More gluing code needed
●
Many different services
Image source: https://phys.org/news/2015-12-earth-layers.html
Idea source: https://www.satisfice.com/blog/archives/4947

11. Testing components
●
Common ideas
●
Different tools
●
Best practices for each
●
Static Code Analysis
– Easy
– Useful
Image source: https://www.mindtheproduct.com/2016/02/what-the-hell-are-ci-cd-and-devops-a-cheatsheet-for-the-rest-of-us/

12. Testing components - levels
●
Unit
●
Functional
●
Smoke
●
Regression
●
End-to-end (integration)
●
Exploratory
– Only manual level

13. Testing components – challenges
●
Tests with hardware
– Long
– Flaky
– Many hardware vendors
●
Mobile app’s tests
– Real devices
Image source: https://www.mindtheproduct.com/2016/02/what-the-hell-are-ci-cd-and-devops-a-cheatsheet-for-the-rest-of-us/

14. End-to-end testing challenges
●
Include every component
– Hardware for each
●
Tools to access all APIs
●
Fully automated

15. Soft quality checkpoints
●
Test Community
– Weekly meetings
– Day-to-day QA issues
●
Impact meetings
– Thrice per release
– Non-technical people with POs
– Business impact of features

16. Beyond our product
●
Our product = our country
●
International law = Bluetooth Mesh standard
– PTS test suite
●
Interoperability is a must
●
Manual tests of 3rd party
– Apps
– Sea of devices

17. Summary
●
IoT technology stacks are big
●
Common QA concepts
●
Common best practices
– CI/CD as paradigm
●
Best tools for each job
●
Harder integration
– Inside & outside

18. Questions?

19. Thank you!