Rapid advancement in digital technologies have accelerated the need to deliver more value to customers, faster. This ever-increasing demand for both speed and quality has put software testing and delivery under tremendous stress. How can you advance your development and testing capabilities to meet digital market demand, today and in the future? Is it only about technology and automation? What about testers—will new practices such as AI save us? Join this web seminar for a discussion with Forrester Research’s Diego Lo Giudice and Perfecto’s Tzvika Shahaf about the future state of testing. Learn what’s next for test and dev teams and the people who drive them. We’ll talk about: -Where the development and testing market is headed and how you compare -Test automation challenges and how to overcome them -What’s next in test automation tools and technologies -Testing and development predictions for 2019

1. THE FUTURE OF TESTING:
TRENDS, TACTICS, AND
PREDICTIONS

2. TZVIKA SHAHAF
• Director of Product Management, Perfecto
• Co-Author of Continuous Testing for DevOps Professionals
• Working with Fortune 500 enterprises and leading tech companies worldwide
helping them optimize their testing strategy and achieve Continuous Testing
GUEST SPEAKER: DIEGO LO GIUDICE
• Vice President & Principal Analyst, Forrester Research
• Focus: Application Development & Delivery
• Leading expert on SDLC processes and practices
• Areas of interest include: agile development, agile and lean transformations,
agile development sourcing strategies and services, agile testing practices
and tools, DevOps, and software testing and quality
SPEAKERS

3. AGENDA
• The state of continuous testing, trends, and challenges
• Continuous testing & AI – How AI can help close the gaps
• What’s Next: Predictions for the future of continuous testing / test
automation

4. © 2017 FORRESTER. REPRODUCTION PROHIBITED.
Testing trends and predictions
Diego Lo Giudice, Vice President, Principal Analyst
@dlogiudice

5. 5© 2017 FORRESTER. REPRODUCTION PROHIBITED.
The Combined Force of Digital Transformation
& Customer Experience Creates Pressure on
Application Delivery Teams To Go Faster
Digital Transformation Customer Experience
Feature
Release
Cycle

6. 6© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Agile+DevOps and Continuos Delivery are increasingly what
teams are adopting to be successfull............ and testing is
right in the middle of it !
Test ReleaseBuild
MonitorIdeate/Plan
Developers UsersFeedback loop
Delivery Pipeline
Insights
Data/analytics
Continuous SW Delivery

7. 7© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Where are we with testing today ?

8. 8© 2017 FORRESTER. REPRODUCTION PROHIBITED.
8%
9%
20%
22%
25%
26%
26%
28%
30%
31%
31%
31%
33%
44%
51%
None of the above
Other (please specify)
Lack of automation tools and/or infrastructure (e.g., Agile PM tools, automated builds, test…
Scaling Agile at the enterprise by replicating success throughout the organization to more teams
Connecting existing product portfolio management (PPM) with Agile projects
Lack of Agile testing skills (e.g., unit testing, test automation, testing in sprints, exploratory…
Lack of dedicated cross functional teams
Lack of Agile skills in downstream or DevOps (XP, development, testing, continuous integration,…
Lack of Agile executive leadership
Incompatibility of our existing funding process to support Agile projects
Lack of Agile skills in project management practices, backlog grooming, planning, requirements,…
Lack of Product Owners commitment from the business
Incompatibility of our existing budgeting process to support Agile projects
Lack of skilled Product Owners from the business
People behavioral change (true buy-in into lean and agile values and principles)
Which of the following are barriers to your organization’s Agile adoption? (Check all that apply)
Base: 232 professionals with knowledge of their firm’s Agile practices
Source: Forrester’s Q3 2017 Global Agile Software Online Survey
Testing appears in many of the barriers and challenges in scaling
Agile and DevOps
Skills
Skills
Automation

9. 9© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Clients of leading continuous testing automation tool vendors: levels
of automation are over all the place and generally low…
CapGemini
World Quality
Report 2018
shows below
20% for all.

10. 10© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Test
automation is
not growing or
improving….

11. 11© 2016 Forrester Research, Inc. Reproduction Prohibited
Performance matters,
what broke !....why ?
36% of my Forrester
inquiries are on non
functional testing -
2/3rds are on
performance testing.

12. 12
Device Fragmentation &
Various OS Platforms

13. 13© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Testing
If you thought
Test Automation
was hard!
Think how hard
it is in
continuous
delivery at
speed!

14. 14© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Can AI help improve software testing ?
Source: Forrester - How AI will change software development and applications

15. 15© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Where can AI
play in the
extended SDLC
and specifically
in Testing ?
Source Forrester : How AI Will Change Software
Development And Applications ?

16. 16© 2017 FORRESTER. REPRODUCTION PROHIBITED.
37% use AI and
ML to test better
and faster to
increase
quality…What
are they doing ?
Source Forrester : To Be Published soon “The Path
To Autonomous Testing: Augment Human Testers
First.”

17. 17© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Source Forrester: Augmented Intelligence Unlocks The Intelligence In AI

18. 18© 2017 FORRESTER. REPRODUCTION PROHIBITED.
AI can augment
testers
intelligence ?
Developers
Technical testers
Business testers
AI

19. FORRESTER.COM
Thank you
© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Diego Lo Giudice
dlogiudice@forrester.com

20. CONTINUOUS TESTING HAS NEVER BEEN MORE COMPLEX
Augmented RealityVirtual Reality
Progressive Web
Artificial Intelligence
Mobile IoT
CONTINUOUS
TESTING FOR
DIGITAL

21. Reporting
Creation Execution
Lab
Tools setup
Create
(Authoring & maintenance)
Debugging
Execution
Scheduling
Validations
Orchestration
Test environment
Desktop web
Artifacts collection
Mobile devicesMobile Emulators
Triaging
Dashboards
Analytics
MARKET TREND – AI IN TESTING

22. AI-Based Self-Healing:
• Network Connectivity
• Device Lock
• Effective Reboot
LEVERAGING AI TO OPTIMIZE AUTOMATION – LAB

23. Easy test automation creation & maintenance
• Robust & intelligent object addressing
• Self-healing executions
Powerful web-based IDE
• No coding skills required
• Script debugging & low maintenance
Test management functionality
• Manage script/suites and prioritize (if not using CI)
LEVERAGING AI FOR AUTOMATION CREATION

24. Codeless Selenium
LEVERAGING AI FOR AUTOMATION CREATION

25. Reasons for “not stable”
80% of issues have a pattern52% success rate
10% of devices, causing
80% of lab issues
Lab
25%
Orchest
ration
25%
Scripts
& FW
50%
FAILURE REASON
Objects Coding Time Other
Scripts & FW issues
Device in use No Device
Orchestration issues
Networking Stability Lock Other
Lab issues

26. Reasons for “not stable”
80% of issues have a pattern52% success rate
Lab
25%
Orchest
ration
25%
Scripts
& FW
50%
FAILURE REASON
Objects Coding Time Other
Scripts & FW issues
Device in use No Device
Orchestration issues
Networking Stability Lock Other
Lab issues
What’s
wrong
With my
Scripts
What’s wrong
With my Lab
What’s wrong
With my Executions
10% of devices, causing 80%
of lab issues
0%
10%
20%
30%
40%
Issues

27. Recognize objects
Transcribe speech – Accessibility
Make quality related decisions based on data
Identify Trends and/or patterns
Security use cases – Identify signatures e.g.
27© 2015, Perfecto Mobile Ltd. All Rights Reserved.
KEY ML USE CASES IN TEST AUTOMATION

28. CarCat
Train
Glass
Dog
Sea
Sun
Table
Tree
TRAINING AI – THE PHOTO LINGO

29. PopupApp not installed
Elementnotfound
App didn’t launch
Capability not supported
App Crashed
Keyboard
Device in Use
Trust
TRAINING AI – THE TESTING LINGO

30. Stabilize Execution By Eliminating Noise
• Pop ups
• Home screen
• Keyboard open
Enhanced Reporting and Analysis
TRAINING THE TESTING AI ENGINE
IMAGE RECOGNITION

31. TRAINING THE TESTING AI ENGINE – IMAGE
RECOGNITION
AI-BASED AUTO-DETECTION OF POPUPS

32. TRAINING THE TESTING AI ENGINE – TEXT
ANALYSISAI-BASED PROCESS TO IDENTIFY & EXCLUDE FALSE NEGATIVE TEST
REUSLTS

33. TRAINING THE TESTING AI ENGINE – IMAGE
RECOGNITION
AI-BASED AUTO-DETECTION OF POPUPS

34. # Category
1 Test Authoring
2 Test Maintenance
3 Test Execution
4 Test Analysis
5 Tools Maturity
6 Available Integrations
7 Req. Test/Dev Skillset
8 Testing Env.
9 Testing Types
10 Supported App Types
Test Engineers/Developers Manual Testers (Developers?)
Traditional Test Automation
Define Manual Flows, BDD Style, etc.
Test Code & Reusable Functions (Java, JS,
etc.)
Define Objects, POM, use Object Spy
Structured Page Based Test Steps
Define Visual Validations and Assertions
Longer Time To Develop, Complex
Changes required pro-actively, SCM
Supported
Configure Env. (TestNG Data Provider),
Execute Locally/CI/Cloud
Flexible, OSS, Commercial
High, Including Guidelines, Doc’s
Plenty, OSS, Defect Management, etc.
Medium-High
IDE
API, Load, Functional
All Types
ML Based Test Automation
Record Test Flows (No Coding – in most cases)
Groups Reusable Functions
Objects Generated On-The-Fly (transparent to user)
Flow Based Scenarios
Part of Test Authoring Recording/Authoring
Shorter Time, Advanced Capabilities
Self-Healing/Correction Automatically handled/Object
Scoring/Local SCM Controlled
Configure Env., Execution Management Built-In,
Locally/CI/Cloud
Relies on ML/AI Tool Vendor in Most Cases
Emerging Technology, Web More Advanced Than
Mobile
Most capabilities built-in, some exists
Low-Medium
ML UI
Mostly Functional (E2E) & API
Mostly Web
MINDSET AND WORKFLOW CHANGES IN TEST
AUTOMATION

35. • Microservices & Cloud
• Big Data and Intelligent Testing
• Service Virtualization
• Codeless Test Authoring (Native)
• ML/AI Across DevOps Activities
• Testing AI
TRENDS AND FUTURE OF DEVOPS
(PREDICTIONS)

36. Thank You