More Related Content Similar to Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice (20) More from Sauce Labs (20) Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice1. © 2019 FORRESTER. REPRODUCTION PROHIBITED.
Accelerate Your Digital Agenda with
Continuous Testing
Diego Lo Giudice, VP & Principal Analyst serving AD&D
April 2019
2. 2© 2019 FORRESTER. REPRODUCTION PROHIBITED.
We need to test more,
better and faster..
..continuously
4. 4© 2016 Forrester Research, Inc. Reproduction Prohibited
http://assets.bwbx.io/images/i0dwq8cFhDmE/v1/-1x-1.jpg
Too much speed?
Poor quality?
Or both?
7. 7© 2019 Forrester Research, Inc. Reproduction Prohibited
53%
Base: 232 tech professionals at firms that have implemented or are implementing Agile.
Source: Forrester’s Q3 2017 Global Agile Software Application Development Online Survey
Expect Agile to
improve technical
quality
8. 8© 2019 Forrester Research, Inc. Reproduction Prohibited
63%
Base: 232 tech professionals at firms that have implemented or are implementing Agile.
Source: Forrester’s Q3 2017 Global Agile Software Application Development Online Survey
Expect Agile to
improve functional
quality
9. 9© 2019 Forrester Research, Inc. Reproduction Prohibited
76%
Base: 232 tech professionals at firms that have implemented or are implementing Agile.
Source: Forrester’s Q3 2017 Global Agile Software Application Development Online Survey
Expect Agile to
improve customer
experience
10. 10© 2019 FORRESTER. REPRODUCTION PROHIBITED.
We need to test more,
better and faster.
Speed Can’t Come
Without Quality !
Quality@speed
12. 12© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Digital
Transformation
Is The Goal For
Every Industry
Base: 354 business and technology leaders
Source: Forrester’s Q1 2018 Digital Process Automation Survey
13. 13© 2019 FORRESTER. REPRODUCTION PROHIBITED.
But,most digital transformation initiatives are struggling
Source: “The Sorry State Of Digital Transformation In 2018” Forrester report.
Base: 1,559 business and technology decision makers
14. 14© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Many Enterprise
Developers
Aspire To Do
DevOps
Yes, our group
actively
embraces
DevOps
21%
Yes, our group actively
embraces DevOps in
conjunction with Agile
32%
Some of our
processes or tools
resemble those used
in DevOps, but we
have not officially
adopted DevOps
25%
No, but adoption
of DevOps is an
internal priority
5%
No, and we
have no plans to
adopt DevOps
11%
Don't
know
6%
Would you characterize your group's current
processes and structure as resembling DevOps?
Base 1358 Enterprise Developers and Development Managers
Source: Forrester Business Technographics Developer Survey, 2018
But only these really
get it!
15. 15© 2019 FORRESTER. REPRODUCTION PROHIBITED.
But Software Delivery Speed Seems Sluggish
Base: 3,228 Global Developers,
Source: Forrester Analytics Business Technographics Global Developer Survey, 2018.
9%
18%
26%
20%
More than once per month
Once per month
Once per quarter
Twice per year
How frequently do you release applications ?
16. 16© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Development Velocity Is Stuck – App Releases
Base (2018): 3,228 Developers
Base: 546-719 Developers who work for a software company, for internal IT, or in technology services
Source: Forrester Business Technographics Developer Survey, 2014-2018
0%
5%
10%
15%
20%
25%
30%
We average one
release a year (or
less frequently)
We average two
releases a year
We average one
release per quarter
We average one
release a month
We average one
release a week
We average a release
a day
We release many
times per day
"How often does your team or teams release applications?"
2018 2017 2016 2015 2014
One release a
month
17. 17© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Testing Is Part Of The Problem
Base: 232 tech professionals at firms that have implemented or are implementing Agile.
Source: Forrester’s Q3 2017 Global Agile Software Application Development Online Survey
20%
22%
25%
26%
26%
28%
30%
31%
Lack of automation tools and/or infrastructure
Scaling Agile at the enterprise level
Connecting Agile with existing product portfolio management
(PPM)
Lack of dedicated cross functional teams
Lack of Agile testing skills
Lack of Agile skills in downstream or DevOps
Lack of Agile executive leadership
Lack of Agile skills in project management practices
Which of the following are barriers to your organization’s Agile
adoption? (Check all that apply) (not all answers shown)
a
18. 18© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Developers time on testing is closer to 1 hour a day
19. 19© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Test automation plateaus too, and manual testing
resists to go away
20. 20© 2019 FORRESTER. REPRODUCTION PROHIBITED.
As Continuous Delivery Matures Performance Testing
Is The Next Blocker
Performance done at the end is complex
and costly
Centralized performance testing isn’t scalable
Completely disrupts continuous delivery
21. 21© 2019 Forrester Research, Inc. Reproduction Prohibited
How can we address
these challenges?
22. 22© 2019 FORRESTER. REPRODUCTION PROHIBITED.
From Digital Business Excellence To SW Delivery
Execution Excellence….Your new mantra…!
Business Must
Master Digital
To Stay In
Business
Software
Powers The
Digital
Experience
Software Execution
Excellence Equals
Business Execution
Excellence
25. 25© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Sprint definition of "done" includes
your sprint testing goals (Done
done, test data strategy,
automation, cntrl. Gates…)
Define Test Scenarios using a
DSL (e.g: Gherkin)
Include Performance Testing in
Acceptance Criteria
Feature toggles, AB Multivariate…
Prioritize bugs on backlog
PO, Testers and Developers
26. 26© 2019 FORRESTER. REPRODUCTION PROHIBITED.
End to End Testing
Integration Testing
Functional
Testing
Non-Functional
Testing
Unit Testing
Make Sure Its Clear Who Needs To Test What….
Only Developers
Buz mostly manual, Mostly Technical Testers,
Dev Testers (API testing, automation).
Technical Testers (Specialized),
Dev Testers (slowly catching up..shift left)
Mostly Technical Testers, Dev Testers (SVT
tools).
Buz, Mostly Technical Testers
27. 27© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Expert Agile Shops Are Better At It...
• 69% Have a tester embedded in
delivery team
• 36% Practice test-driven
development (TDD)
• 20% Practice Behavior Driven
Development (BDD)
• 58% Report improved functional
quality
• 50% Report improved technical
quality
• 57% Have a tester embedded in
delivery team
• 15% Practice test-driven
development (TDD)
• 12% Practice Behaviour Driven-
Development (BDD)
• 37% Report improved functional
quality
• 33% Report improved technical
quality
Agile Experts Agile Neophytes
Base: 130 tech professionals who have achieved “Expert” status by achieving significant scale and quality of Agile adoption. 102 tech professionals who have achieved
“Neophyte” status by achieving limited scale and quality of Agile adoption.
Source: Forrester’s Q3 2017 Global Agile Software Application Development Online Survey
28. 28© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Design automation of functional
tests (and test data).
Build automation of functional
tests (UI, API).
Build automation for non
functional (SVT, SL PT, Security)
Scriptless, scripting, coding
Exploratory testing (manual)
Technical Testers, Dev-testers,
Business testers
29. 29© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Source: See The Forrester report “Shift Performance Testing Left To Streamline App Delivery”.
30. 30© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Shift Left Performance Testing (SLPT)
Developers
profile code,
smoke test,
leverage
baselines at
unit/
component
level
Product
Owners and
Developers
become
stewards of
Performance
(With Testers)
Overtime E2E
Performance
Testing
Wanes As
SLPT
Becomes The
Norm.
A new set of
lightweight
agile SLPT
tools provide
developers
what they
need to get
the job done
31. 31© 2019 FORRESTER. REPRODUCTION PROHIBITED.
United Airlines experienced
an 80 % Performance
Issues drop in production
with SLPT
32. 32© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Version Control test assets
with source code
Integrate in CI/CD to
automate all test execution
(unit, functional, non-
functional)
33. 33© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Automated integration
testing (Service
Virtualization)
Automated governance for
release automation or
feature deployment
34. 34© 2019 FORRESTER. REPRODUCTION PROHIBITED.
idea Customer
value
control points
versioned
source repository
Code
Environ
Configs
Test assets
Continuous
Integration & Testing
artifact
repository
“built”
artifacts
Backlog
Release
Decision
Release
Automation
vendors
open
source
Developers
EA,
Testers
Ops,
Security
EA,
Developers,
Ops,
QA,
Security
DevOps Automates Testing Governance
CONTROL POINTS IN THE CONTINUOUS DELIVERY PIPELINE
35. 35© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Production monitoring
feeds your testing
Machine Learning predicts
what to test next
Integrated AB testing in
SDLC helps improve CX
36. 36© 2018 Forrester Research, Inc. Reproduction Prohibited
A/B Testing improves customer experience
37. 37© 2019 FORRESTER. REPRODUCTION PROHIBITED.
How the use of AI
optimized and
reduced overall
testing time from
years to days for
DELL EMC
26
World Quality Report | 30th October 2018 © 2018 Capgemini. All rights reserved.
Context at DellEMC Servers
Server Configuration Elements
Chassis
Processor
Memory DIMM
Memory Configuration
Hard Disk Drive (HDD)
Non-Volatile Memory (NVM)
Embedded Systems Management
Power Management BIOS
Power Supply
Bezel
Network Daughter Card
RAID Controller
Network Interface Card (NIC)
Host Bus Adapter (HBA)
Additional PCIe Cards
Cooling
465 Trillion
Test Configurations!!
AI Technologies used:
• Machine learning
• Rule-based
system
38. 38© 2019 FORRESTER. REPRODUCTION PROHIBITED.
We need to test more,
We need to test faster,
We need to test better,
We need to test smarter!
40. The Push From Within: A Journey
of Transformation
Claude Jones, Sr. Director of Engineering | Walmart Labs