Chuck Schneider, VP Application Platform Development & Testing Strategy at Cerner, shares his 6 top tips to ensure that your software testing strategy works the first time. Chuck presented these slides at the Gartner Application Strategies & Solutions Summit 2019 in Las Vegas. This presentation highlights the challenges encountered throughout Cerner's testing journey, and how you can avoid them! Cerner is a world leading Healthcare Information Technology company working on securing better healthcare for the world.

1. 6 Top Tips to a Testing Strategy That Works
Chuck Schneider
Vice President of Platform Engineering
and
Testing Strategy - Cerner Corp.

2. Cerner Corporation

3. Cerner’s Scale
Over 8 billion orders placed annually
500 applications (C#, C++, Java RCP, Browser, Android, iOS)
475,000 concurrent users worldwide
1415 Eggplant users trained across 9 countries
Q1 2018 357 automated scripts; 2064 reusable handlers
Q3 2019 6,482 automated scripts; 16,499 reusable handlers
Over 250,000 automated tests ran headless in past 12 months
Over 2,000,000 launches of Eggplant in past 12 months
Average Concurrent Executions: 31154 mobile devices in our Device Lab
15,000 Jenkins sessions executed per day

4. Test Design and Shift Left
Test design
and shift left
1. Eliminate the tests you don’t need!
2. Shift Left
• Unit tests, API tests, Functional tests
• Isolate the architecture
3. Test Design
• “Living Tests”
• Organized repository
• Reusable Handlers
The Value - Control Costs, Maintainability

5. Test Data Management
Test design
and shift left

6. Test Data Management
Test design
and shift left
1. Determine approach
• Database resets (snapshots)
• Database record generation
• Use testing automation scripts
2. Identify types of data
• Users
• Activity
• Reference
• Configuration
3. Optimize each test’s data needs
• Will need to generate this data prior to each test
The Value - Repeatability

7. Test Automation
Test design
and shift left

8. Test Automation
Test design
and shift left
1. Understand the applications you are testing
2. Start with the tests you run the most
3. Minimize the use of hardcoded values
4. Version your tests (tests are code)
The Value - Efficiency and Quality

9. Controlled Environments
Test design
and shift left

10. Controlled Environments
Test design
and shift left
1. Create & replicate from a master environment
2. Control the testing environment
• Control access
• Control the data builds and configuration
3. Determine needs for on demand & shared
environments
The Value - Repeatability

11. Continuous Integration
Test design
and shift left

12. Continuous Integration
Test design
and shift left
1. Create headless processes
• Credentials
• Data Creation triggers
• Identifying and/or spinning up SUTs
2. Associate tests with code
3. Design code pipeline management
The Value - Early feedback loop of errors

13. CI PIPELINE
Engineer starts
1
2
Engineer
coding/testing on
branch
Engineer checks-in
code
3 4
Automated Tests
5
Package/Release
76
Automated Tests
Deploy

14. Dashboards
Test design
and shift left

15. Dashboards
Test design
and shift left
1. Define each pillar's success metrics
• Static code analysis
• Which data is auto generated
• Test status per run/release
• Environment refresh/creations
• Adoption
2. Automate the measurements
3. Share dashboards broadly
The Value - Visibility of successes and failures

16. World Class Testing Ecosystem
To create and
run efficient
and
maintainable
tests
With controlled
test data
That does not
require human
intervention
In on demand
virtual
environments
In a
continuous
manner
Which
captures,
collects and
communicates
coverage &
results
Test design
and shift left
TDM
RQM,
Eggplant,
Selenium EOD CD Dashboard

17. Where to Start?

18. State the Value for the Company
IMPROVE QUALITY REDUCE COSTS RELEASE MORE
OFTEN
HIGHER CUSTOMER
SATISFACTION

19. Get Clear on your Goals
• Increase frequency of releases by X%
• Reduce production defects by X%
• Reduce testing release cycle by X%
• Percent of automated tests run per release
• Pipeline Health = how long a test stays in a failed state

20. Assess Current State
• Evaluate where your biggest inefficiencies are today
• Where are you in the Maturity Model today?
• Phase 1 - Manual tests
• Phase 2 - Automating Manual Tests
• Phase 3 - Automating the Process
• Phase 4 - CI Pipeline Adoption
• Phase 5 - Well Established CI Pipeline

21. Define Future State
• Determine your desired Maturity Model Phase for each of your
products
• What tooling and infrastructure will you need to purchase
• Two risks when purchasing any tool
• Will the tool meet your expectations
• Will the tool be adopted
• Culture changes
• Role responsibilities may change
• What skills are needed

22. Create Incremental Value
• Determine where to start
• What is causing the most friction?
• Spinning up an environment
• Automate tests
• Improve test designs
• Create milestones
• Associate value and costs to each milestone
• Define how each milestone will be measured for success
• Regression tests run nightly in a headless manner
• Increase Unit testing coverage to X%
• Reduce hours spent creating test data by X%
• X% of tests are automated and run for each release

23. All Aboard
• Everyone needs to be onboard
• Sell this to your
• Testers
• Learn automation skills
• Reduce repetitive mundane tasks
• More time to find bugs
• Engineers
• Faster feedback on quality of code
• Design an automated process
• Executives
• Efficiencies in the development process
• Faster to market
• Higher Quality

24. Questions or
for a copy of these slides contact me
Chuck Schneider at
www.linkedin.com/in/chuck-schneider