Advancing the State of The Art in AI and Testing - DevOps Next

#devopsnext-testing-tools
LIVE Q&A
EVENT HELP
#devopsnext-event-help
PRE-RECORDED

Advancing the State of the
Art in AI and Testing
TA R I Q K I N G | C H I E F S C I E N T I S T

4 | DevOps Next 2020
Confidentiality Statement
The information contained in this document is strictly confidential, privileged, and
only for the information of the intended recipient. The information contained in this
document may not be otherwise used, disclosed, copied, altered, or distributed
without the prior written consent of Perforce Software, Inc.

INTRODUCTION
o AI FOR FUNCTIONAL TESTING: RECAP
BEYOND FUNCTIONAL TEST AUTOMATION
o AI FOR TESTING UI DESIGN
o AI FOR TESTING VIDEOS
o AI FOR TESTING GAMES
WRAP UP
AGENDA

FUNCTIONAL UI TESTING: WEB AND MOBILE
AI FOR FUNCTIONAL TESTING

REUSABLE TESTS + PERFORMANCE MEASUREMENTS = POWERFUL INSIGHTS
PERFORMANCE BENCHMARKS

AI FOR TESTING:
BEYOND FUNCTIONAL TEST AUTOMATION
Functionality
Usability
Videos
GamesAccessibility
Security

AI FOR UI DESIGN TESTING

DESIGNING HIGH QUALITY MOBILE USER EXPERIENCES
developer.android.com/designdeveloper.apple.com/design/human-interface-guidelines
MOBILE UI GUIDELINES

When possible, present choices.
Make data entry as efficient as possible. Consider
using a picker or table instead of a text field, for
example, because it’s easier to choose from a list
of predefined options than to type a response.
Give text-titled buttons enough room.
If your toolbar includes multiple buttons, the text of
those buttons may appear to run together, making
the buttons indistinguishable. Add separation by
inserting fixed space between the buttons.
HIGS: HUMAN INTERFACE GUIDELINES FOR iOS
MOBILE UI GUIDELINES

TRAIN BOTS TO RECOGNIZE GOOD VS. BAD DESIGN
HIG TESTING WITH AI

A HIG VIOLATION IN A MAJOR APPLICATION IN THE APPSTORE
HIG TESTING WITH AI

ACCESSIBILITY (A11Y)

WEB CONTENT ACCESSIBILITY GUIDELINES (WCAG)
A11Y GUIDELINES

/AGENT-X/AGENT
http://github.com/tariqking
EXTENDING A.G.E.N.T WITH OPEN-SOURCE ACCESSIBILITY TOOLS
A11Y TESTING WITH AI

AGENT FRAMEWORK
INSERT
FRAMEWORK
HERE
Lighthouse
EXTENDING A.G.E.N.T WITH OPEN-SOURCE ACCESSIBILITY TOOLS

PNSQC 2019 TECHNICAL PAPER AND PRESENTATION

How would you determine if a
mobile app login was trustworthy
just by looking at it?

Sample of Least Trusted Apps Sample of Most Trusted Apps

Familiar Brands
Language Matches Region
Few Screen Elements
70 TESTERS (7 MINS) VS. 1 BOT (3 SECS) TESTERS LEARNED FROM AI
TESTING UI TRUST WITH AI
HUMAN VS. AI TRUSTWORTHINESS EVALUATION

TRUSTWORTHY?
QUALITATIVE QUANTITATIVE
“Well it seems
trustworthy”
“It in the 90th
percentile of
the trusted
apps”
HUMAN VS. AI TRUSTWORTHINESS EVALUATION
TESTING UI TRUST WITH AI

Rating Label
5 Excellent
4 Good
3 Fair
2 Poor
1 Bad
4
5
MANUAL APPROACH: MEAN OPINION SCORE (MOS)
VIDEO QUALITY

CURRENT AUTOMATION: FRAME BY FRAME FAILURE MODE DETECTION
VIDEO QUALITY

Scene Analysis: Human Focus AreasVideo Category InformationCommon Failure Modes
TRAINING BOTS TO TEST VIDEO QUALITY LIKE A HUMAN
TESTING VIDEOS WITH AI

Actual vs. Predicted MOS values for Asian Fusion Video Category

AUTOMATION CHALLENGES
o Graphical, Animated, 2D/3D
o Infinite State Space
o Frequent Updates, DLC
o Customized Interaction Models
o Declarative vs. Imperative Tests
o Finicky Audience
AI PLAYS GAMES, WHY NOT TEST THEM TOO?
TESTING GAMES WITH AI

Real-Time Object Recognition in Video Games
REINFORCEMENT LEARNING
o Goal-Based Test Specification
Test Script:
1. Open Treasure Chest
2. Equip Pump Shotgun
3. Score a Kill
o Bots learn by trial and error to
accomplish tasks.
o Reward-based model allows AI to
improve or find optimal solution.
o When game changes, bots can
autonomously re-learn how to
accomplish tasks in new environment.
LEVERAGING REINFORCEMENT LEARNING FOR GAMEPLAY TESTING

Object-Recognition: Character Object-Recognition: Back Bling
AUTOMATING IN-GAME STORES: OBJECT DETECTION

Trained
Image
Visual Diff on
Glitched Asset
VISUAL DIFFING HEATMAP: BUG DETECTION

TAKING VISUAL DIFFING TO THE NEXT LEVEL

Advancing the State of
the Art in AI and Testing
Tariq King, Chief Scientist
Jason Arbon, CEO
AI FOR UI DESIGN TESTING
How accessible is it?
Tariq King enjoys breaking things and is currently the Chief
Scientist at test.ai. He started his career in academia as a tenure-
track professor and later transitioned into the software industry.
Tariq previously held positions as a test architect, manager,
director, and head of quality and performance engineering. He
has published over 40 articles in peer-reviewed software testing
books, journals, conferences, and workshops. He is a member of
the ACM and IEEE Computer Society, and serves as a board
member of several international software engineering and testing
conferences.
Jason Arbon is a test nerd and currently the CEO of test.ai,
where his mission is to automate the testing of the world’s apps
with AI. He has been the director of product and engineering at
uTest/Applause.com. Jason previously held engineering
leadership roles at Google (Chrome/Search) and Microsoft
(WindowsCE, SQL Server, BizTalk, Bing). He also co-authored
the books: How Google Tests Software and App Quality: Secrets
for Agile App Teams. In his spare time, Jason likes to read up on
Artificial General Intelligence and consciousness and is working
on a new personalized search engine.
AI for Testing Web Content Accessibility Guidelines
Does it look trustworthy?
Is it usable?
AI for Testing Mobile
Human Interface Guidelines

AI FOR GAME TESTINGAI FOR VIDEO STREAM TESTING
AI bots perform scene analysis to detect human focus areas.
AI bots mimic human judgement on video stream quality.
AI bots use reinforcement learning to execute gameplay tests.
AI bots test in-game store purchase of an axe for a character.

ACKNOWLEDGMENTS
DIONNY SANTIAGO
YASHAS MAVINAKERE
JASON STREDWICK
JUSTIN PHILLIPS
JONATHAN BELTRAN
PATRICK ALT
DAVID ADAMO
GREGG TABOT
YINLIN FU
XANDER SONG
CHRIS NAVRIDES

#devopsnext-testing-tools
LIVE SLACK Q&A

Advancing the State of The Art in AI and Testing - DevOps Next

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Advancing the State of The Art in AI and Testing - DevOps Next

Similar to Advancing the State of The Art in AI and Testing - DevOps Next (20)

More from Perfecto by Perforce

More from Perfecto by Perforce (16)

Recently uploaded

Recently uploaded (20)

Advancing the State of The Art in AI and Testing - DevOps Next