Penny McVay shares how her team approached improving the quality of a large policy-writing application for a global insurance carrier. The application has many pieces and parts, thousands of lines of code are changed monthly, and the business depends on a stable application. To mitigate risk, the team investigated the question of how to predict where testing needed to focus. The regression test suite the organization had built was difficult to maintain, identified few defects, and took hours of effort to run. QA needed not only to optimize the regression suite but also to determine how to mitigate the most risk. To accomplish this, the team collected historical data to predict significant impact areas including what is being written in production, what transactions are being exercised in production, and what defects are occurring in today’s production environment. From this information, the QA team rewrote the regression suite to eliminate waste and provide more value to the business customers.

1. BW6
Session
6/8/16 1:30 PM
Predictive Test Planning to Improve
System Quality
Presented by:
Penny McVay
Liberty Mutual
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.techwell.com/

2. Penny McVay
Liberty Mutual
Penny McVay manages a large, global-wide quality assurance team for the
commercial insurance technology division of Liberty Mutual. Penny began her
career in the insurance industry and has experience in business functional areas
and IT. From a COBOL programmer to a developer to earning her PMP
certification, Penny's experiences have shaped her approach to managing QA as a
true partner in collaborating on large IT projects for the past fourteen years.
Penny resides in Hamilton, Ohio, and when she's not knee-deep into quality
assurance practices, Penny enjoys spending time with her family and camping.

3. 6/2/16
1
Increase Quality through Predictive Modeling
Penny McVay
CI Quality Assurance
Liberty Mutual Insurance 5/8/2016 2
Who Am I
38 years of Insurance Industry experience
• IT Experience - 17 years
– Developer: Cobol II, CICS, Easytrieve
– Project Management: Large Tier 1 projects and implemented SDLC
– Quality Assurance: Start up of Ohio Casualty Group Testing
• Experience prior to Liberty Mutual
– 8 years test leadership for a new Product and Policy Admin System
in Commercial Lines small to medium business units

4. 6/2/16
2
Liberty Mutual Insurance 5/8/2016 3
Main Message
Quality Assurance has many tools and
methodologies that can assure quality while
reducing cost.
Liberty Mutual Insurance 5/8/2016 4
How?
1. Combinatorial Analysis
2. Shift left influencing quality at the beginning of the cycle
3. Automation
4. Risk-Based Testing

5. 6/2/16
3
Liberty Mutual Insurance 5/8/2016 5
The Challenge
Determine the scope of testing to maintain the
quality and dependability of a large-policy
writing application with many pieces, parts and
thousands of lines of code that are updated
monthly.
Liberty Mutual Insurance 5/8/2016 6
Background: Predictive Modeling
Collected data to find significant defects where
the impact would be most harmful to the
business.
Questions Asked
• What products are written in production?
• What transactions are exercised in production?
• Where are defects occurring in the production environment?

6. 6/2/16
4
Liberty Mutual Insurance 5/8/2016 7
Predictive Modeling Results
An automated regression suite that was:
• large
• difficult to maintain
• identified few defects
• took hours of work to execute
Liberty Mutual Insurance 5/8/2016 8
Next Steps
Decided not only optimize the regression, but
rewrite the regression to provide more value.

7. 6/2/16
5
Liberty Mutual Insurance 5/8/2016 9
Predictive Modeling
1. Wrote a new regression suite using the predictive modeling
approach
– Test cases designed based on business data
– Test cases included defect-prone areas based on data
– Eliminated waste by including testing based on statistics
2. Rewrote in a new tool to improve performance and
maintainability
Liberty Mutual Insurance 5/8/2016 10
Other Important Tasks
1. Planned feedback loop created to verify as data changed
so regression could also be updated
2. Buy-in from key partners (developers, systems analysts,
and business partners)
3. Designed a two-tiered regression suite

8. 6/2/16
6
Liberty Mutual Insurance 5/8/2016 11
Business Data Analysis
What policies are written in the field?
• How many new business policies were written?
• How many renewals were being issued annually?
– How many were manual, automated, or due to conversion from
another application?
• How many subsequent transactions were applied to new
business and renewal policies?
– Cancellations, Audits, Reinstatements and Endorsements
Liberty Mutual Insurance 5/8/2016 12
Defect Analysis
1. Rank the LOBs based on how they make up the book of
business (largest to smallest)
2. Rank the LOBs based on the largest amount of defects
identified in the previous 12 months
3. Determine other factors impacting creation of defects
– Monoline vs multi-line
– Multi-state vs single state
– Defects found regardless of state (Common requirements by State)

9. 6/2/16
7
Liberty Mutual Insurance 5/8/2016 13
Design of Automation
Wrote based on the book of business and
where history indicated there would be the
most defects.
• LOB Example:
• 4.2% of policies issued
– 70% Single LOB only
– 39% Single State only
– 47% All States
Reduced regression test scripts from 428 scripts to 40
Liberty Mutual Insurance 5/8/2016 14
Two-Tiered Regression
Static: based on the historical data.
• Remains mostly unchanged
• Meets the criteria of the “design of automation”
– based on production data analysis and defect data analysis
• Updates are needed if there are significant changes to the
book of business
– Production data pulled on a regular schedule to determine if
there has been a change in the book of business

10. 6/2/16
8
Liberty Mutual Insurance 5/8/2016 15
Two-Tiered Regression
Dynamic: based on current project releases
introducing risk to the application.
• Based on projects moving into production, risk test scripts
are developed and executed for 90 days
• Reviewed after 90 days to determine if the regression needs
to be revised to include this functionality
• Not meant to be long-term
Liberty Mutual Insurance 5/8/2016 16
Results & Lessons Learned
1. Efficiencies gained by analyzing business risk
2. Re-use of regression automation is relative
– Anything high-risk is in the regression suite
3. Re-use of automation is easier
– Data now outside of the automation, allowing for easy
data changes with datasheets feeding into the
automation
– Project teams can easily identify automation pertinent to
their work using the data analysis
4. Test Catalog of automation created in order for the teams to
understand what is in the regression suite

11. 6/2/16
9