KraQA #42 - Paweł Dyrek - Machine learning powered regression - narzędzie wspierające testy aplikacji web'owych oparte na sztucznej inteligencji i porównywaniu obrazów

1. Machine learning
powered regression
December 2019

2. Agenda:
Problem
Solutions
Our suggested solution
Some numbers
Current work
Machine learning
The road ahead

3. Regression testing takes a lot of effort
Problem

4. Regression testing takes a lot of effort
Problem
Manual testing
Create automated testing suite
Solutions

5. Automated snapshot testing with ML support
Our idea

6. Our idea
Open source tool
From QAs for QAs by QAs

7. Some numbers
Implemented for 3 live products
Estimated 1 month of work to achieve 80% regression coverage even for
complex applications.
1 QA supported by this tool is an equivalent of approximately 4 QA engineers.

8. E-commerce example: Language International
~50000 different landing pages
60 languages on site
Compatible with multiple viewports and devices
Result:
50000 pages * 60 languages * 10 devices = 30 000 000 pages to test before each release
Some numbers

9. Some numbers: 30 000 000
Impossible to test manually.
Several months to cover with tests by a team of QA engineers with classical approach.
Maintenance/update of old tests is easily achieved by replacing screenshots almost immediately.
In classical approach QA needs to change the source code of the tests.
10000 tests, 3000 tests failed, 100 files to update - inefficient to update, would take a week for QA team.
We will take ½ day to validate and replace screenshots as a screenshot compares to 50 classic tests on average.

10. Current work - Web

11. Current work - Intelligent crawler

12. Current work - test execution
Graphic UI
API library - Java or PHP

13. USP vs the biggest competition
Intelligent crawler to identify the dynamic part of the page and reach the different application states.
Generate tests automatically for the most common cases.

14. USP vs the market
By QAs and for QAs - optimized to boost the quality.
Reach desired coverage in weeks instead of months.
Build an efficient QA team - focus on what is important and automate the rest.

15. Machine learning - layout and item type recognition

16. Machine learning - layout and item type recognition

17. Machine learning - layout and item type recognition
Google Vision AI

18. Machine learning - layout and item type recognition
Tensorflow & Keras

19. Machine learning - object detection

20. Machine learning - mobile status interface ignore

21. Machine learning - mobile status interface ignore

22. The road ahead - automated Selenium scripts generation

23. The road ahead
We still don’t have a modern UI for test cases and test suites management.
Generate more sophisticated test data sets.
Add user management in a more sophisticated way.

24. The road ahead
Open source project development:
https://github.com/codete/QAcheck

25. The road ahead
Code preparation

26. The road ahead
Future goals definition

27. The road ahead
Hackathon contest
January/February 2020

28. Where to find us
https://github.com/codete/QAcheck

29. Q&A

30. Thank you!