Every day, embedded devices are becoming more powerful and capable of running more elaborate applications. Among these applications are Web-based ones, enabling to leverage features from the Web APIs to the embedded context, either through a generic browser running a traditional Web application or through a customized Web engine tightly integrated within the system. But such capabilities usually bring new challenges, like testing user interactions with the application using the embedded device's specific I/O methods, such as gestures, or inspecting Web application internals with JavaScript. In this context, using a browser automation framework such as WebDriver, which is a W3C standard supported by WebKit Web engine, allows testing Web-based applications on such devices as if the user were actually using it, alongside running custom JS code. In this session, we will cover why we need browser automation for testing on certain types of embedded devices, with a focus on WebDriver as the proposed tool to achieve that goal. We will also discuss WebDriver's main features and limitations, as well as other possible approaches and frameworks that could be considered for this kind of task. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://eoss24.sched.com/event/1aeSx/automated-testing-for-web-based-systems-on-embedded-devices-lauro-moura-igalia

1. Automated testing of
Embedded Web Applications
Lauro Moura
lmoura@igalia.com
1 / 28

2. About me
At Igalia since 2020
WebKit team
QA, WebDriver
Other work done in the past:
EFL (JS and C# bindings)
Maemo (Python bindings, Nix WebKit port)
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
2 / 28

3. About Igalia
Specialized Open Source consultancy, founded in
2001
Fully remote, HQ in A Coruña (Spain). Flat
structure.
Top contributors to all the main Web Rendering
Engines
WebKit, Chromium, Gecko and Servo
Active contributor to other areas and OSS
projects
V8, SpiderMonkey, JSC, LLVM, Node.js, GStreamer,
Mesa, Linux Kernel...
Members of several working groups:
W3C, WHATWG, WPT, TC39, OpenJS, Test262,
Khronos... Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
3 / 28

4. Agenda
Embedded Web Apps
Some Challenges in Web Apps
Challenges of Embedded Web Apps
Browser automation overview
Browser automation tools
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
4 / 28

5. Web Applications
"Write once, run anywhere!"
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
5 / 28

6. Web Applications
Fast development cycles
Fast deployment
Large ecosystem
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
6 / 28

7. Web Apps in real life
"Write once, run _mostly_ anywhere!"
"Works better with $BROWSER"
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
7 / 28

8. Why does this happen?
Engine differences: Blink vs WebKit vs Gecko
Browser diferences: Chrome vs Brave vs Edge, Safari vs GNOME Web
Versioning: Canary, Stable, ESR
Browser engines != Desktop browsers != Mobile browsers
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
8 / 28

9. It's also about hardware
Desktop? Tablet? Phone?
4k? 1080p? Scaled or native resolution?
What kind of interaction? Mouse? Gestures?
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
9 / 28

10. Embedded Web Applications
Not just a generic site running on a generic mobile browser
Load specific resources...
(e.g. using a local server)
... from specific browsers
We can expand the browser with new APIs
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
10 / 28

11. Embedded Web
Applications in real life
But we might have different devices...
...with different hardware capabilities...
(e.g. different GPUs, screens)
Or different browser versions
Thankfully, still less variables
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
11 / 28

12. Other differences
Generic websites are usually sandboxed
Embedded web apps might need to interact with the system
Testing the web app in a live system is a must
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
12 / 28

13. But how to test a web app?
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
13 / 28

14. Automation != Testing
Automation is a tool
Crawling the web
Filing taxes
Running test steps
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
14 / 28

15. Originally, mimick user actions
Navigate to $URL
Find input for value "name"
Enter text "John Doe"
Find button with label "Confirm"
Click this button
Search for message "User created"
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
15 / 28

16. Brief history of browser automation
Early 2000's efforts
Selenium + WebDriver = Selenium WebDriver
W3C WebDriver standard in 2018
Supported by major browsers
WebDriver "Classic"
HTTP-based protocol (remote connections)
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
16 / 28

17. WebDriver Classic, under the hood
Commands
Responds
User Browser
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
17 / 28

18. Mimicking the user is not enough
Logic moving into the frontend
Asynchronous tasks
Polling is tricky
Developer tools/inspectors evolution
Enabling two-way communication with the browser
Automation tools based on these protocols
Browser-specific solutions?
in active development o/
W3C WebDriver BiDi
WebKit support incoming
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
18 / 28

19. WebDriver BiDi, under the hood
Commands
Events
User Browser
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
19 / 28

20. WebDriver Bidi
Based on WebSockets
Extends WebDriver Classic
Still perform user actions
Monitor the browser state without polling
Monitor messages
log
Observe changes
DOM
Intercept and mock requests
network
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
20 / 28

21. What about Browser
Automation Tools?
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
21 / 28

22. Selenium
One of the most well known tools around
Deeply influenced the WebDriver standard
JavaScript, Python, Java, Ruby, C#...
Selenium Grid
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
22 / 28

23. WebDriverIO
Also based on the W3C WebDriver
Written in JavaScript
Built-in support for Cucumber, Mocha and Jasmine test frameworks
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
23 / 28

24. Playwright
Puppeteer background
Developed by Microsoft
Autowaits feature
Depends on patched Firefox or WebKit, though...
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
24 / 28

25. Recap
Embedded Web Apps run in controlled scenarios
But testing how it integrates in the system is crucial
WebDriver BiDi features allows better testing of such apps
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
25 / 28

26. Questions?
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
26 / 28

27. Thank you!
Automated testing for Web-based embedded systems
Lauro Moura, 2023-04-18
27 / 28

28. 28 / 28