Artificial intelligence (AI) is the most important technology for software testers to understand today. All software will soon have AI-powered components, and they are unlike anything you’ve ever tested before. As risky as AI can be, it is a powerful weapon for testers to solve some of their most painful testing challenges today. The web was great, mobile is interesting, but AI will truly change the way you build and test all software. Jason Arbon gives a brief introduction to AI and machine learning (ML) so you can nod your head knowingly when the topics come up. Explore how products that leverage machine learning are tested at Google, Microsoft, and new startups. Learn the basics of labeling data, training sets, testing sets, measuring quality, and the risks of retraining neural networks. Even learn how to apply AI and ML to your own testing work today. Join Jason to get a glimpse into the new world where we will work hand-in-hand with our new AI bot friends. Don’t miss the AI train—it will change everything.

1.
W6
Special
Topics
5/10/17
11:30
AI
and
Machine
Learning
for
Testers
Presented
by:
Jason
Arbon
Appdiff,
Inc.
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Jason
Arbon
Jason
Arbon
is
the
CEO
of
Appdiff,
which
is
redefining
how
enterprises
develop,
test,
and
ship
mobile
apps
with
zero
code
and
zero
setup
required.
He
was
formerly
the
director
of
engineering
and
product
at
Applause.com/uTest.com,
where
he
led
product
strategy
to
deliver
crowdsourced
testing
via
more
than
250,000
community
members
and
created
the
app
store
data
analytics
service.
Jason
previously
held
engineering
leadership
roles
at
Google
and
Microsoft
and
coauthored
How
Google
Tests
Software
and
App
Quality:
Secrets
for
Agile
App
Teams.

3. AI and Machine
Learning for Testers
Jason Arbon, CEO @Appdiff

5. Ai for Test
Automation
3
Relevant Context
Testing Neural Net Ranker
Personalized Web Search and
Chrome Test Automation
AI for Mobile Test Automation

6. Ai for Test Automation
Agenda
AI For Testing
Testing AI
Future

7. Appdiff Presentation5
The Real Data Scientists
Joanne Tseng
Data Scientist @appdiff.com
Appdiff mission:
Transform app
development with
automation & insights
Francis Iannacci
Lead Data Scientist @appdiff.com

8. AI for App Testing
Features
Complexity increases
exponentially as new features and
states interact with existing
features
Tests
Test coverage grows linearly
because tests can only be added
one at a time
Time
Complexity/Coverage
COVERAGE
GAP
6
Testing Needs AI

9. Our Story7
Definition

10. When will AI Start
Testing?

11. When: Today

12. Our Story10
AI for Testing

13. What We Do11
ML : Subjective or complex labels via Humans

14. Our Story12
AI for Testing: Input
Reduce input space
150 legit actions per
page
10 clicks deep
2x10^65 paths
Train like a
human/tester

15. Our Story13
AI for Testing

16. Our Story14
AI for Testing

17. Our Story15
AI Driven Test Flows

18. Our Story16
AI Driven Test Flows

19. Our Story17
AI Driven Test Flows

20. Our Story18
Coverage

21. Coverage

22. Our Story20
AI bots can test almost any app

23. Our Story21
Benchmarked Peformance

24. Our Story22
Automation Coverage: Bots 2/3rds Existing Tests
Long Sequence of Dependant
Actions and Verifications
Basic Tasks (Login, Search,
Create Account, Add items to
Card, etc.)
Specific Sequences of Events
with Specific Input (search for
‘beanie babies’, etc.

25. Our Story23
Automation Coverage: Bots soon 100%

26. Our Story24
Automation Coverage: Bots... 10X
Canonical Tests
Learning Tests Cross-app

27. portfolio

28. Our Story26
Appdiff Represents the Evolution of Software Quality
1
AD HOC
TESTING
Reactively test
2
MANUAL
TESTING
Proactively test
3
TEST
AUTOMATION
Automate repetition
4
AI-DRIVEN
APPROACH
Accelerate coverage
App
QA
App
Automation
QA
App

29. How to test AI?

30. Our Story28
Testing AI: Action / Flow Graphs

31. Our Story29
Testing AI: App Generation and Traversal

32. Our Story30
Testing AI: Graph Generation and Traversal

33. What We Do31
Testing AI: Drift
Networks are randomly initialized
Same overall correctness score can
mean diff performance on subsets of
data

34. What is Next?

35. What We Do33
DARPA HARD

36. What We Do34
Humans vs Bots
Humans Train Bots in Course of Work
Humans Remain to Personalize and Re-train bots.

37. What We Do35
Testing Flow with AI

38. What We Do36
AI Test Description Language: AIT
Focus on Intent, not “how”
Human Readable
App-Independant

39. Our Story37
AI for Testing

40. What We Do
Features
Complexity increases
exponentially as new features and
states interact with existing
features
Tests
Test coverage grows linearly
because tests can only be added
one at a time
Time
Complexity/Coverage
COVERAGE
GAP
38
Testing Needs AI

41. hi@appdiff.com
Jason Arbon, CEO
jason@appdiff.com

42. Our Story40
Help Labeling

43. Our Story41
Teaching AI How To Test

44. What We Do42
ML: Page Label Training

45. As long as there is software,
there will be software testing
Humans Code

46. Our Story44
AI for Testing

47. Appendix: How It Works45
Like a Redline for Your App
Agile development leaves teams struggling
to achieve adequate test coverage.
Automatically identify changes to UX
across versions of your app.
Version X Version Y Version X Version Y
1.2s 1.8s
Slower
AddedRemoved
Performance UX

48. Testing with AI?