The world has changed. We feel it in the work environment. We see it in the numbers. We smell it in the questions. Much that once was, is lost, for few remember it, even less still use it. In the name of agility and lean processes, we have turned a blind eye to quality, favouring quick execution over customer delight. And some things that should not have been forgotten were lost. Join Jay Rao and Vishal Anand to explore the new world of “agile delivery”. Understand the myths vs the real picture, the anti-patterns to watch out for and how you can look forward to understanding new measurements for delivery, and various functions within – from architecture to deployment – and where each of us has opportunities that align with common sense and outcomes

1. 46
440
170
times
more frequent code deployments
times
faster lead time from commit to deploy
times
faster mean time to recover

2. Brave New World
A wider perspective of our opportunities

3. Fresh look at measurements in the new Agile Delivery world
What are we talking about today?
The Delivery and Service conundrum
What is beyond Automation?
Testers new Tree of Life

4. Core Tenets
Outcomes over Output
Global optima over local optima

5. Dev
Ops
Typical Measures
- Velocity
- Utilization
- Lines of Code
- Defect density
- Cost of Quality Typical Measures
- Uptime
- Performance
- Recovery time

6. Measures are proxies
They represent an idea

7. Dev
Ops
I want to deliver
change faster! Less changes keep the
system more stable!
Throughput!
Stability!

8. What does the data say?
Deployment Frequency
Lead time
MTTR
(higher means faster response)
Change Failure Rate
(higher means less change failure)
High Maturity Medium Maturity Low Maturity
How is this
possible?

10. High rate of change can and does co-exist with high quality of service

11. “Cease dependence on inspection to achieve
quality. Eliminate the need for inspection by
building quality into the product in first place”
- Edward Deming

12. Does this mean no QC?

13. On the contrary

14. Dev
Ops
Traditional Testing –
preserve throughput and
quality using automation
and more
New addition to Testing
– measure and enable
service in sync with Dev
Throughput!
Stability!

15. Automation – Past & Now
15
The current future of Automation – What Next?
Record and Playback Agile Automation
Framework Driven
(Datasheets, Library Functions)
Quick
Regression
Early
Automation
Sustained
Quality
Business
Readiness
Continuous
Automation
Continuous Delivery
CI
Continuous
Integration
Continuous
Delivery
Continuous
Deployment
Continuous Delivery is a Trending
Agile Topic
Integrated Cloud AutomationNon-Conventional Automation
Shift-left through virtualization & Early
Automation
Multi-channel, multi-platform
testing
Distributed Computing
Infrastructure
Test Lab
High Performance
More emphasis on GUI Automation Coverage driven automation process Search for Singular Tooling Solution
Automation across Lifecycle Defect detection to Defect Prevention Right viable tools & Integrated tooling ecosystem
BUT THIS IS TEST AUTOMATION

16. GUI Tests
API
Automation
Unit Tests
10%
20%
70%
10%
20%
70%
Manual Testing
Manual
Exploratory
Testing
GUI
Test
s
Manual
Exploratory
Testing
Integration Test
Biz Logic Acceptance Test
Unit Tests
Workflow Test
E2E Flow Test
1%
4%
6%
9%
BUT THIS IS STILL TEST AUTOMATION
Source: Mike Cohn, AgileFAQs

17. Build Quality thru Automation in process
Image Source:: http://geekinc.ca/business-process-management-and-henry-ford/ Image Source:: http://loomstate.blogspot.co.uk/2017/01/toyoda-and-japanese-
industrial.html
1908 1924

18. Lifecycle Automation
Cognitive QA
1 2
Automate everything. Build fixtures for every step.
Build tests for every step.
Automate the decision process. Use AI on QA
data to derive test suite, better design and
even better requirements – all the way to
service expectations
Highly evolved Orgs are 44x more likely to reuse testing patterns
– State of DevOps 2018

19. Computers perform repetitive tasks – people solve problems

20. BDD
Tech. Design
Coding
Test Design
Unit Tests
Logic/UI/API/Web Services/DB
Integration Tests
Biz. Workflow TestsUAT UI Tests
NFT Regression
Test Data
Feedback from users
Exploratory Tests
Uptime
Performance
Recovery (MTTR)
Change Failure Rate

21. Markets that do not exist cannot be analysed
- Christensen (1997,143)
Early feedback and response is critical to the success of the product

22. Feedback
Basic
Performance
Delight
Everything
Else
?
Should these
even exist?
What kind of feedback can we do? And what is the impact?

23. A model for
Prioritizing
Solutions
using
Feedback
Right
Thing
Wrong
Thing
Wrong
Way
Right
Way
Quick but
unsustainable
wins
Enduring
Success
Slow failure Fast failure

24. Takeaways