Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AI and Machine Learning for Testers


Published on

Artificial intelligence (AI) is the most important technology for software testers to understand today. All software will soon have AI-powered components, and they are unlike anything you’ve ever tested before. As risky as AI can be, it is a powerful weapon for testers to solve some of their most painful testing challenges today. The web was great, mobile is interesting, but AI will truly change the way you build and test all software. Jason Arbon gives a brief introduction to AI and machine learning (ML) so you can nod your head knowingly when the topics come up. Explore how products that leverage machine learning are tested at Google, Microsoft, and new startups. Learn the basics of labeling data, training sets, testing sets, measuring quality, and the risks of retraining neural networks. Even learn how to apply AI and ML to your own testing work today. Join Jason to get a glimpse into the new world where we will work hand-in-hand with our new AI bot friends. Don’t miss the AI train—it will change everything.

Published in: Software

AI and Machine Learning for Testers

  1. 1.                 W6   Special  Topics   5/10/17  11:30             AI  and  Machine  Learning  for  Testers     Presented  by:         Jason  Arbon     Appdiff,  Inc.     Brought  to  you  by:                 350  Corporate  Way,  Suite  400,  Orange  Park,  FL  32073     888-­‐-­‐-­‐268-­‐-­‐-­‐8770  ·∙·∙  904-­‐-­‐-­‐278-­‐-­‐-­‐0524  -­‐  -­‐            
  2. 2.   Jason  Arbon     Jason  Arbon  is  the  CEO  of  Appdiff,  which  is  redefining  how  enterprises  develop,  test,   and  ship  mobile  apps  with  zero  code  and  zero  setup  required.  He  was  formerly  the   director  of  engineering  and  product  at,  where  he  led   product  strategy  to  deliver  crowdsourced  testing  via  more  than  250,000  community   members  and  created  the  app  store  data  analytics  service.  Jason  previously  held   engineering  leadership  roles  at  Google  and  Microsoft  and  coauthored  How  Google   Tests  Software  and  App  Quality:  Secrets  for  Agile  App  Teams.    
  3. 3. AI and Machine Learning for Testers Jason Arbon, CEO @Appdiff
  4. 4. Ai for Test Automation 3 Relevant Context Testing Neural Net Ranker Personalized Web Search and Chrome Test Automation AI for Mobile Test Automation
  5. 5. Ai for Test Automation Agenda AI For Testing Testing AI Future
  6. 6. Appdiff Presentation5 The Real Data Scientists Joanne Tseng Data Scientist Appdiff mission: Transform app development with automation & insights Francis Iannacci Lead Data Scientist
  7. 7. AI for App Testing Features Complexity increases exponentially as new features and states interact with existing features Tests Test coverage grows linearly because tests can only be added one at a time Time Complexity/Coverage COVERAGE GAP 6 Testing Needs AI
  8. 8. Our Story7 Definition
  9. 9. When will AI Start Testing?
  10. 10. When: Today
  11. 11. Our Story10 AI for Testing
  12. 12. What We Do11 ML : Subjective or complex labels via Humans
  13. 13. Our Story12 AI for Testing: Input Reduce input space 150 legit actions per page 10 clicks deep 2x10^65 paths Train like a human/tester
  14. 14. Our Story13 AI for Testing
  15. 15. Our Story14 AI for Testing
  16. 16. Our Story15 AI Driven Test Flows
  17. 17. Our Story16 AI Driven Test Flows
  18. 18. Our Story17 AI Driven Test Flows
  19. 19. Our Story18 Coverage
  20. 20. Coverage
  21. 21. Our Story20 AI bots can test almost any app
  22. 22. Our Story21 Benchmarked Peformance
  23. 23. Our Story22 Automation Coverage: Bots 2/3rds Existing Tests Long Sequence of Dependant Actions and Verifications Basic Tasks (Login, Search, Create Account, Add items to Card, etc.) Specific Sequences of Events with Specific Input (search for ‘beanie babies’, etc.
  24. 24. Our Story23 Automation Coverage: Bots soon 100%
  25. 25. Our Story24 Automation Coverage: Bots... 10X Canonical Tests Learning Tests Cross-app
  26. 26. portfolio
  27. 27. Our Story26 Appdiff Represents the Evolution of Software Quality 1 AD HOC TESTING Reactively test 2 MANUAL TESTING Proactively test 3 TEST AUTOMATION Automate repetition 4 AI-DRIVEN APPROACH Accelerate coverage App QA App Automation QA App
  28. 28. How to test AI?
  29. 29. Our Story28 Testing AI: Action / Flow Graphs
  30. 30. Our Story29 Testing AI: App Generation and Traversal
  31. 31. Our Story30 Testing AI: Graph Generation and Traversal
  32. 32. What We Do31 Testing AI: Drift Networks are randomly initialized Same overall correctness score can mean diff performance on subsets of data
  33. 33. What is Next?
  34. 34. What We Do33 DARPA HARD
  35. 35. What We Do34 Humans vs Bots Humans Train Bots in Course of Work Humans Remain to Personalize and Re-train bots.
  36. 36. What We Do35 Testing Flow with AI
  37. 37. What We Do36 AI Test Description Language: AIT Focus on Intent, not “how” Human Readable App-Independant
  38. 38. Our Story37 AI for Testing
  39. 39. What We Do Features Complexity increases exponentially as new features and states interact with existing features Tests Test coverage grows linearly because tests can only be added one at a time Time Complexity/Coverage COVERAGE GAP 38 Testing Needs AI
  40. 40. Jason Arbon, CEO
  41. 41. Our Story40 Help Labeling
  42. 42. Our Story41 Teaching AI How To Test
  43. 43. What We Do42 ML: Page Label Training
  44. 44. As long as there is software, there will be software testing Humans Code
  45. 45. Our Story44 AI for Testing
  46. 46. Appendix: How It Works45 Like a Redline for Your App Agile development leaves teams struggling to achieve adequate test coverage. Automatically identify changes to UX across versions of your app. Version X Version Y Version X Version Y 1.2s 1.8s Slower AddedRemoved Performance UX
  47. 47. Testing with AI?