This document discusses the "shift left" testing strategy to overcome issues caused by late testing. It outlines how ambiguous requirements, compressed development timelines, and mounting technical debt can lead to a downward spiral of lower quality releases over time. The key aspects of the proposed shift left strategy are: 1. Creating test specifications before development begins using Gherkin syntax to make requirements objectively measurable. 2. Executing all test cases daily, even during development, to prove software delivery and contain issues early. 3. Measuring test coverage against requirements daily to accurately track percentage of delivery and foresee risks. 4. Leveraging cloud-based hyperscaled testing to maximize parallel test throughput and allow all tests

1. THE “SHIFT LEFT” STRATEGY
OVERCOMING THE DOWNWARD SPIRAL OF DOOM
Darian Rashid
darian@clearskytestautomation.com

2. 2

3. 3
Late?
Too many defects?
Not enough content delivered?
How did we get here?

4. 4
DeployDesign Development TestRequirement
Create Test Cases
LETS REWIND
First time we
validate
requirements
Development has most likely overrun its time
Test capacity is constrained
Not enough people and time to run System, Integration and Regression effectively

5. 5

6. 6
Fix easy issues first
“Cannot replicate”
Test “Coverage”
Technical debt mountain
How many issues can be
differed?
How many issues from
previous releases continue to
be ignored?

7. AMBIGUITY IN REQUIREMENTS?
7
Leads to missed expectations
and rework
Discovered well into
development cycle
Each reworked item could:
Impact delivery schedule
Incur fines
WELL, YES AND NO…
Best case: Spend time arguing that
the delivery met the requirements
Worst Case: Implement CRs during
testing
• Ambiguous requirements
• Test cases created after development has begun

8. “We just finished up
our sprints. Now we’re
gonna test.”
BUT..

9. 9
THE CODEBASE

10. 10
THE CODEBASE
Brittle code
Deeply coupled,
interconnected web
Usually new features on top of
legacy code
Refactored last in the Regan era

11. 11
THE TEAM

12. 12

13. 13

14. 14
EMERGENCY PATCHES
Haven’t accounted for these in the release
Are already overcommitted for the next release
Cannot drop deliverables
RESULT: A compressed time frame for the next release

15. 15
THE “DOWNWARD SPIRAL OF DOOM”
Ambiguous
requirements &
over-
commitment &
technical debt
Compressed
development
timeframe
Compressed
Q/A
timeframe
Rushed delivery
& slashed test
matrix
Rework &
brittle code
Low quality
release
Production
issues &
patches
EACH
RELEASE
WORSE
THAN THE
LAST

16. 16

17. Prove Delivery
Run Manual
Tests
Create Test
Specifications
Create
Development
Design
Documents
Develop Code/
Execute Tests,
Daily
Create
Customer
Requirements
THE PARADIGM SHIFT
PROVE SOFTWARE DELIVERY BY:
1. Creating test specifications BEFORE development begins
2. Execute ALL test cases DAILY, even during development
3. Measure test coverage against requirements DAILY
Customer
Requirements
Objectively measurable
test specifications
CONTAIN ISSUES

18. CONVENTIONAL GHERKIN
Create Adult Customer Profile with Preferred Phone as Home
Phone
1.Launch the eCustomer Application
2.Click on ‘or register’ link
GIVEN I am at the Home page
WHEN I click on the “Register Now” Link
THEN I will be taken to the Register screen
AND I will see the following:
3.Enter the following Mandatory Details:
(a) enter the valid 10 digit Home phone number
(b)Preferred Phone: Home
(c)Preferred Contact Method: Email
(d)Enter the other details given in the default demographics
section of test case initialization table
WHEN I enter the following:
THEN I will be taken to the Contact Information Screen
AND I should see the following
Title First Name Middle Name Last Name … Answer
(blank) (blank) (blank) (blank) … (blank)
First Name Middle Name Last Name … Answer
Miss Bethany Jones … 9021
First Name Middle Name Last Name … Answer
Miss Bethany Jones … 9021
COMPARE LEFT AND RIGHT
What should I see when I click on this
link? How do I know which page I’m
taken to? How do I know that data
loaded that page correctly. If this page
isn’t right, the rest of the test cannot
proceed.
Each cell is a test case. The number of
output values to check against will
depend on the test case.
What other details? Which details
are valid details? Which are invalid?

19. 19
TEST SPECIFICATIONS
Represent the behaviors of the user and the system
Created in a simple, human-readable manner
Objectively measurable
Used as the primary customer and internal acceptance criteria
Become the requirements for the development organization
Proves delivery

20. Create Customer
Requirements
Create Test
Specifications in
Gherkin
Create
Development
Design Documents
Develop Code
Run Regression
Tests Multiple
Times Daily
Prove Delivery
Test Specs. Pass. Feature
complete, add to
regression suite.
Unit
Testing
Manual
Testing
PROVE DELIVERY
• Creating
OBJECTIVELY
MEASURABLE test
specifications
• Execute EVERY test
cases EVERY day
• Measure test
coverage against
requirements DAILY
END RESULTS
• Maintain quality
• Accurately show %
delivery
• Foresee Risk

21. 21
Execute EVERY Test EVERY Day?

22. 22
BUT…
Number of Tests 1 VM
1000 8.3 Hours
3000 25 Hours
5000 41.6 Hours
10,000 83.3 Hours
Average Test Execution Time: 2 minutes

23. 23
THE ANSWER
HYPERSCALE ON THE CLOUD

24. 24
MAXIMIZE TEST THROUGHPUT THROUGH MASSIVE-
SCALE PARALLEL TESTING USING THE CLOUD
CLEAR
SKY
CORE
RUNNERS

25. 25
SYSTEM CAN’T TAKE THE HEAT? KEEP SCALING!
CLEAR
SKY
CORE
RUNNERS
LOAD
BALANCER

26. THE DEMO

27. 27
THE CONCLUSION
Create unambiguous test specifications, used as requirements
Run EVERY test EVERY day to create a safety net
Use hyperscaled automation on the cloud to scale
Test case
Instances under test
KEEP QUALITY CONSTANT DAILY