Using ML to make your UI tests more robust
•Download as PPTX, PDF•
1 like•573 views
Common practice is to write lots of unit tests and API tests and only a few User Interface tests. Why? Because UI tests are brittle. Change one thing and all the other tests unravel. But what if we could use ML to help us out. Many of our apps have the same functionality such as login, checkout, share, pay etc. In this session we'll look at how we can use Object Detection and labeling techniques to make our UI tests more robust with a fraction of the code.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Using ML to make your UI tests more robust
Similar to Using ML to make your UI tests more robust (20)
More from Godfrey Nolan
More from Godfrey Nolan (20)
Recently uploaded
Recently uploaded (13)
Using ML to make your UI tests more robust
- 2. UI API Unit Tests 8/28/2019 AI Testing 2
- 3. 8/28/2019 AI Testing 3
- 4. 8/28/2019 AI Testing 4
- 5. 8/28/2019 AI Testing 5
- 6. 8/28/2019 AI Testing 6
- 7. 8/28/2019 AI Testing 7
- 8. 8/28/2019 AI Testing 8
- 9. 8/28/2019 AI Testing 9
- 10. 8/28/2019 AI Testing 10
- 11. 8/28/2019 AI Testing 11 Get Data Train & Test Adjust Deploy Test & Validate
- 12. 8/28/2019 AI Testing 12 Get Data Train & Test Adjust Deploy Test & Validate
- 13. 8/28/2019 AI Testing 13 Images Train.csv Test.csv Train.record Test.record Training the model Evaluating the model Generate CVS file Generate TFRecords Generate TFRecords
- 14. 8/28/2019 AI Testing 14 Laggards Leaders Today, most fall in this range TestingEfficiency&Efficacy Time Manual Testing Testing Automation Continuous Testing Intelligent Continuous Testing https://www.ca.com/en/blog-highlight/artificial-intelligence-and-continuous-testing-its-the-next-big-thing-really.html
- 15. 8/28/2019 AI Testing 15 AccelQ Acellere Acellere AppDiff APImetrics Appium Classifier Plugin Applitools Appvance Assister Automorph AutonomIQ Codebeat Codebots Codota Decibel Insight Deckard AI Deep Code DiffBlue Eggplant Endtest EvoSuite Fedr8 Firedrop Functionize Ghost Inspector Kite Logz Mabl Memorio Near Opsani Parasoft SOAtest Prodo Qordoba Qualicen RainforestQA ReTest Saucelabs Sealights Selenium with AI Sema Sourcegraph Source{d} Stepsize Tara Intelligence Tesabot Test AI Test Complete Test Craft Testim Test Sigma The Grid Tosca Uizard Yotako
- 16. 8/28/2019 AI Testing 16 NLP Auto generate tests Unit tests only Enhanced version of Selenium Low code Limited number apps Test management tool XPATH helper tools
- 17. 8/28/2019 AI Testing 17 Self Healing Flexible Generic solution Web and Mobile Test more than one Export code for CI Repeatable Quick No False Positives Not Expensive Scalable Still funded
- 18. 8/28/2019 AI Testing 18 Flexible Generic solution Web and Mobile Test more than one Export code for CI Repeatable Quick No False Positives Not Expensive Scalable Still funded Self Healing
- 19. 8/28/2019 AI Testing 19
- 20. 8/28/2019 AI Testing 20
- 21. 8/28/2019 AI Testing 21
- 22. 8/28/2019 AI Testing 22
- 23. 8/28/2019 AI Testing 23
- 24. 8/28/2019 AI Testing 24
- 25. 8/28/2019 AI Testing 25
- 26. 8/28/2019 AI Testing 26
- 27. 8/28/2019 AI Testing 27
- 28. 8/28/2019 AI Testing 28
- 29. 8/28/2019 AI Testing 29 Get Data Train & Test Adjust Deploy Test & Validate
- 30. 8/28/2019 AI Testing 30
- 31. 8/28/2019 AI Testing 31 Flexible Generic solution Web and Mobile Test more than one Export code for CI Repeatable Quick No False Positives Not Expensive Scalable Still funded Self Healing
- 32. 8/28/2019 AI Testing 32 Selenium with AI Selenium with AI Selenium with AI Selenium with AI Fix Patch Generation Bug Detection Triggers Trigger Patch Generator Sapienz Auto Triage Validated Revision
- 33. 8/28/2019 AI Testing 33
- 34. 8/28/2019 AI Testing 34 How to apply AI to testing by Jeremias Rößler https://www.gartner.com/doc/reprints?id=1-5U7BFII&ct=181120&st=sb https://www.youtube.com/watch?v=EcTmKXdYvtM https://github.com/testdotai/appium-classifier-plugin https://www.kaggle.com/testdotai/common-mobile-web-app-icons AI for element selection - Jason Arbon http://testerstories.com/ https://david.rothlis.net/testdotai-appium-plugin/ https://code.fb.com/developer-tools/finding-and-fixing-software-bugs-automatically-with-sapfix-and-sapienz/ https://riis.com/blog/building_better_ui_tests_with_ai/
Editor's Notes
- Hey I’m Godfrey Nolan, own a mobile company in Detroit, MI.
- Mike Cohn’s agile testing pyramid – which is 10 years old this year unit tests test the methods, the API or service layer tests the REST APIs using Postman or RestAssure and the GUI is tested using Espresso or Appium Why do we do it? For insurance purposes, to stop the finger pointing between the backend and the mobile team IMHO we do it to extend the life of the software, it’s IT’s dirty little secret that apps only last a couple years before someone decides to rewrite them Anything that has been around for a while has an inertia, where it takes way longer to add a feature to a 2 yr old mobile app than it did when it first came out You can see that it’s a triangle for a reason, we should have lots of unit tests, many API tests and then a much smaller number of UI tests – which you think would be the most
- Ok, we can take a look at a simple login screen for our app, takes a username and password and a Sign in Button. We also have the create account, and need help – for forgot passwords etc.
- If you’re new to testing you’ll probably just write GUI tests for each any every scenario When you first start testing, it’s almost like all you do are GUI tests. ‘cos that’s what the user sees Quickly there can be hundreds of these tests in our tests suite. Maybe all of your tests are really well written Espresso, Selenium or Appium Or maybe all of your tests used the recording tool that you edited to get working.
- So when we make a subtle change such as going from username to email then all of our tests are going to fail There are too many dependencies to manage manually
- So if we change the field from username to email address like in the previous slide then all of our other tests cases are going to fail people only run diff tests….explain (TODO) People write less UI tests
- What we need to do is invert the pyramid or at least turn it into a square not a triangle? What do we want the tool to do? => need a testing tool that can recognize logins, carts, registrations So the tests aren't as brittle. => need a tool that can teach itself Need to work on Android, not just the web performs some type of self maintenance, fixes code automatically, self healing => Generic testing tool - one app to rule them all (not everyone’s going to want that)
- Before we go much further got to say there really isn’t enough time to talk about the morals of AI or just how scary some of the predictions are Yes this is going to mean manual testers are going to need to change how they do their job But that’s been the same for many a year. Context is key – we can’t teach the AI context. So a generic tool might be a long way off but a specific type of tool could work. This has implications for developers in the future
- We have can decompose this mobile app or any other web page into standard components =>shopping cart =>search bar =>data card with two texts boxes and a corresponding image So what we want is an app that will use machine learning/object detection to break apart an app into easily recognizable elements such as login, registration, forgot password, Nueral Networks can learn many different types of login buttons o NN know what aren’t login buttons o NN know if a login button moves, changes size, position, color, etc. o “Sees and Thinks like a User”
- We have can decompose this fitness app =>fab button =>hamburger menu =>alert =>home button =>user etc
- To train a network we need to get a largish sample size of images, say 5000 We label each of the images to show what we’re looking for - in this case it’s person and kite We train with 70% of the data and then test our model on the remaining 30% with the hope that a very small percentage are not recognized AI testing is basically doing exactly the same thing.
- So where do we get the data? First place to look is Kaggle, test.ai have a largish dataset of mobile icons. Many of the icons came from https://thenounproject.com/ If we need to find a cart image then we can can use the custom cart image driver.findElement('-custom', 'ai:cart'); It will find the cart element
- I took this from the Tensorflow github repo and it shows how we put it all together. Test.ai uses Tensorflow and the Kaggle data to train the model which we then use as an Appium plugin
- Here’s where we are at the moment. There are the beginnings of something that meets our definition of Nirvana But so far I don’t see anything works out of the box on all apps.
- There’s been a significant growth in the number of tools that claim to be AI testing tools. So many that it’s hard to know where to begin Don’t believe the Hype - be skeptical - most of them just help you write tests quicker – it’s debatable that some have any sort of AI.
- Tools fall into a very similar type of categories NLP - natural language processing Auto Generate tests Unit tests only Enhanced version of Selenium Low code Limited number apps Test management tool XPATH helper tools
- Self Healing - can fix any simple coding error, recommit and then test again Flexible - if I change from username to email, I don’t have to rewrite everything Generic solution - should be able to work on any app in the app store, Google Play Web and Mobile - iOS and Android Test more than one Page/Activity/ViewExport code for CI Repeatable Quick No False Positives Not prohibitively expensive - transparent pricing Scalable - multi browser, multi device Still funded - support?
- Test.ai is using object detection, uses TensorFlow Jason Arbon is the creator of the tool Appium Plugin - Appium with a brain You can try it yourself as everything is up on github They did get funding from Google 11m Does have a higher learning curve than the other examples
- # install appium $ npm install -g appium # install test-ai dependencies $ brew install pkg-config cairo pango libpng jpeg giflib # install test-ai $ npm install test-ai-classifier --save # install python unit testing framework and python Appium client. $ pip3 install pytest Appium-python-client
- TO DO: Show the two apps being tested TO DO: Show all the prerequisites Pre-req’s – install node, python3, Appium, In this example we’re starting up the FandangoNow app and telling it to find the search bar It’s been trained for approx 100 commands or images for finding - cart, search, login, send, up down buttons etc. and you can train more yourself
- Import the Appium drivers
- Point the test at the genymotion device You get this from running adb devices
- Running this against FandangoNow, for the moment we’re hardcoding the location of the apk
- Here’s where we load the test.ai module
- This is the address of the Appium web interface, if you used Appium before this is where you normally create your tests
- This is where we do the actual test, we’re looking for a search button Looking at the app I’ve highlighted the search, in top rh corner of the app on the nav bar, marked with a red rectangle Once we find it we send a click event
- So here’s the result of our test. I’m also highlighting how our demo test gets called.
- You can find a list of the objects or commands your can find using test.ai There’s over 100 at the moment which should suit most needs Our objective is to create dynamic data in 1000’s of APKs, i.e. automated registration and then login to apps to see what’s been saved insecurely.
- Like I said the dataset is open source, so if you have something you want to test, for example there was a diary icon in the fitness app That isn’t in the existing list then you can add that to the dataset and retain the machine learning so it’ll work for you requirements.
- Ok so we want this to be a generic test, we want to pass in more apks and search for a single movie, say Thor Here’s two more app with search buttons with different search buttons, one high one low. DEMO: do the second demo.
- Maybe you don’t need a flexible solution like we need maybe you need something very specific, Facebook created Sapienz Sapienz is Facebooks app for testing Facebook, Instagram and Whatsapp only. I left it to the end as it’s not publicly available but it’s interesting to help see where AI testing is headed
- Sapienz and Sapfix have a true self-healing functionality. Testing is all about passing tests, so if we have failure then mutate the code using templates and if it passes the tests then commit the code, otherwise revert. SAY THAT TWICE so if we have failure then mutate the code using templates and if it passes the tests then commit the code, otherwise revert.
- Are we there yet? Not that intelligent yet to call it AI and It's not machine learning either. I would say it's just Intelligent Test Automation Coding. Mostly all about efficiency not AI Mostly all about fixing tests quickly not AI Have we reached Nirvana. If you’re only testing a small number of apps then I think you can use something like test.ai Facebook – SELF HEALING WTF!! WHY DO WE WANT TO DO IT? Dynamic testing security – I’ll let you know how we get on
- Any questions??