UI testing is an essential part of software development. Despite technological progress, the automation of UI tests is still considered too complex to function completely without manual intervention. In addition to classical selector-based approaches, more and more image-based methods are being pursued. This talk will cover the "state of the art" of UI testing with an overview of tools and techniques. In particular, current problems and future developments will be discussed. Furthermore, it will be shown which kind of UI representations are used by today's test tools and how the addressing of elements in the user interface is done. In addition, the role of artificial intelligence in the different approaches is shown and a prediction of testing tools of the future is presented on the basis of current research.

1. The Future of
UI Testing

2. © askui
User interface 2

3. © askui
Element localisation 3
Login Page
Login
Visual selector
<button>
Type: ‚submit‘
Text: ‚Login‘
Human Computer
Code selector

4. © askui
4
DOM and Shadow DOM
Shadow DOM
Subtree of DOM elements
XPath selector
CSS selector
DOM
XPath selector
CSS selector

5. © askui
5
iFrame
Embed another full document within HTML document
XPath selector
CSS selector
Only if moved into iFrame before selecting

6. © askui
6
XPath and CSS selectors
/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/main[1]/a
rticle[1]/div[1]/div[1]/div[1]/div[1]/section[1]/div[1]/di
v[1]/div[1]/div[1]/div[1]/section[1]/div[1]/div[1]/div[1
]/div[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]/inpu
t[1]
XPath
//input[@id=’userId’]
CSS selector
#userId

7. © askui
Demo 7
Let’s get started!

8. © askui
8
Toolspace
Code
Formal (DOM)
Visual
Graphical Programming
Smart Selector

9. © askui
Why are there challenges? Example case 9

10. © askui
10
Canvas
Renders graphics into 2D plane
XPath selector
CSS selector
Canvas
Use coordinates/AI. Use case for visual regression

11. © askui
Smart selectors 11
Appl€
Apl
Appie
Apple
Optical Character Recognition (OCR)

12. © askui
Smart selectors 12
Appl€
Apl
Appie
Apple
Optical Character Recognition (OCR) Picture in Picture

13. © askui
Smart selectors 13
Appl€
Apl
Appie
Apple
Optical Character Recognition (OCR) Picture in Picture
Relations

14. © askui
Smart selectors 14
Appl€
Apl
Appie
Apple
Optical Character Recognition (OCR) Picture in Picture
Relations Self-healing Tests
Sign In
<button_Login>
Type: ‚submit‘
ID: anmelden
Testscript:
“Click on
button
‘sign in’”
Testscript:
“Click on
button
‘sign in’”

15. © askui
Demo 15
Let’s get started!

16. © askui
16
Toolspace
TestComplete
Code
Formal (DOM)
Visual
Graphical Programming
Smart Selector

17. © askui
17
Visual (Regression) Testing

18. © askui
Outlook with AI 18

19. © askui
Demo 19
Let’s get started!

20. © askui
20
Toolspace
TestComplete
Code
Formal (DOM)
Visual
Graphical Programming
Smart Selector

21. © askui
Outlook with AI 21
src: https://visualqa.org/ src: https://twitter.com/, @levelsio
Visual Question Answering GPT-3

22. 22

23. askui GmbH
c/o CyberLab
Haid-und-Neu-Str. 18
76131 Karlsruhe
johannes.dienst@askui.com
www.askui.com
23