The document discusses initiatives at Northwell Health's Testing Center of Excellence (TCoE) to make testing more efficient and data-driven. It describes Northwell Health's large size and the TCoE's goals to improve quality, reduce time to market, and lower costs. Key initiatives discussed include implementing risk-based testing to prioritize testing based on risk, reducing wait times in end-to-end testing through automation and coordination, and increasing automation of testing and management of test data to speed up testing cycles and ensure consistent artifacts. The TCoE aims to transform testing through metrics, processes, tool optimization, and expanding its services across Northwell Health.

1. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

2. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Enabling a data-driven testing CoE
by Harnessing the true power of Risk-Based Tes
Ian Jablonski, Director, Northwell Health Information Services
Vinay Chandra, Director, Deloitte Consulting LLP

3. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
Introduction of speakers
Background and Experience
• Over 25 years of Information Technology experience in
Financial Services
• Over 15 years of experience in building and leading
Centralized Software Quality Assurance Teams
• Managed several Transformations and several
geographically diverse QA Centers of Excellence.
Background and Experience
• Recognized as a thought leader with demonstrated skills in
conceptualizing and implementing more than 20 large scale
Testing CoEs
• Focused on utilizing newer techniques that help evolve Testing
organizations through their transformational journey thereby
increasing focus on early defect detection and defect
prevention
Ian Jablonski Vinay Chandra
Ian Jablonski
Director, Testing Center of Excellence
Northwell Health Information Systems
ijablonski@northwell.edu
Vinay Chandra
Managing Director,
Deloitte Consulting LLP
vinchandra@Deloitte.com

4. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
Agenda
1. Northwell Health Testing CoE background
2. Foundations of a Data Driven Testing CoE
3. TCoE Transformational Themes
4. Key Initiatives
1. Risk Based Testing
2. Reduced Wait Times
3. Automation Testing
5. Leveraging Tools Across the TCoE
6. In Summary

5. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
Northwell Health Testing CoE Background
Northwell Health Introduction TCoE Background
• The Northwell Health TCoE was
established to achieve higher
quality releases through
professional test services within
project timelines
• The TCoE centralizes core testing
capabilities, delivering cost-
effective and standardized testing
services
• The TCoE has more than 140
practitioners serving 20% of the
enterprise, and as an independent
advisor, aims to expand testing
services throughout all of Northwell
Health with a defined level of
quality, increased automation, and
maximizing cost benefits
TCoE Mission Statement
As a trusted partner, the TCoE
provides software quality
assurance services across the health
system that will drive improved
quality, reduced time to market,
and lower costs through repeatable,
leading edge processes that are
metrics driven and focused on
continuous improvement.
• Nation’s 14th largest health system
in the US with $12b in revenue
• Nation’s 7th largest physician group
practice with nearly 2,750 full-time
physicians
• Largest private employer in New
York State
• 61,000 employees
• 15,000+ nurses
• 11,000 + affiliated physicians
• 6,600 beds
• 450 outpatient physician practices
• 21 hospitals serving 8 million
people
• Largest ambulance transport
system on the East Coast US

6. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
Foundations of a Data Driven TCoE
TCoE
Mission
Faster Time to Market Improved Quality
Metrics
• Test Design and Test Execution Coverage
• Defect Leakage / RSI
• Assets by System
Reduced Cost
Metrics
• Savings through Automation
• Defect Density
• Overall Cost, Cost by Role, Cost per Work Unit
Metrics
• Automation Coverage
• Resource Utilization / Skills Inventory
• Test Design and Test Execution Cycle Time
Enablers
• Solid Framework and Centralized Repository
• Periodic realignment of Processes
• Metrics driven decision making and SLAs
• Shift Left Techniques
Enablers
• Automation expanded beyond regression testing
• Optimized TCoE toolsets
• Demand Management and Cross Training
• Follow the Sun operating model
Enablers
• Automation Leverage
• Staff Leadership Internally
• Span of Control
• Offshore and Nearshore labor model
• Expanded Services and Cross Training
A Data Driven TCoE will ensure a Quicker, Better and Cheaper Testing organization

7. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Environment Management
Specialized Testing Test OptimizationDemand Management
Process Management TCoE ExpansionTest Management
• Prepare metrics definition
• Define SLAs
• Define efficiency and improvement
• Transform from test to quality
• Risk Based Testing
• Regression Testing
• UAT Support
• Involvement in all phases of PLC
• Shift left strategy
• Wait time reduction
• Shared Services
• Develop and Implement a Skills
Cross Training Plan
• Forecasting, capacity and demand
management process improvements
• Expand TCoE footprint to additional
service lines
• Knowledge Transfer
• Develop and Implement Environment Management
Strategy
• Develop and Implement Release Management Strategy
• Develop and Implement Test Data Management Strategy
• Tool strategy and maintenance
• Automation testing
• Performance testing
• Service Virtualization
TCoE Transformation Themes
The following Transformation Themes are integral in moving from the initial setup to an optimized TCoE

8. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Key Initiative - Risk Based Testing (RBT)
• Focused workflow/pathway oriented
approach requires additional number of test
cases
• Dependent on testers’ native application
knowledge
• Technology-focused testing approach that
may not always meet business intent
• Level of effort may not reflect level of risk
• Identification of critical defect at an early
stage of testing
• Considerable Decrease in QA cost
• Faster Time-to-Market
• Increase involvement between Business,
Technology and Testing
• Collaborative working model between
SME’s and testers
• Business can make informed decision
on the Risk vs. Testing Level for release
acceptability
Conventional Testing Risk Based Testing
VS.
RBT is a methodology to test the right things at the right time with the right amount of rigor. RBT
also seeks to improve quality by increasing visibility across business and technology stakeholders.

9. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Our RBT Approach
Statistical Risk Based Testing
approach uses mathematical models
and Statistical Sampling to arrive at a
targeted optimal scope for testing
Examples:
• Testing using Orthogonal Arrays/
Combinatorial testing
• Other Statistical Sampling
Methods (Random, Systematic,
Stratified and clustered Sampling)
Business Oriented Risk Based Testing
leverages business experience to focus
testing efforts based on Financial,
Operational, and Technical exposure
Leveraging both approaches can
help realize the benefits of RBT

10. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Key Initiative- E2E Wait Time Reduction
Scheduler
TCoE
2hrs
Registration
TCoE
1 hr
Oncology
Non-TCoE
1 hr
Coding
TCoE
1 hr
Billing
TCoE
2 hrs
Charge Entry
TCoE
1 hr
Radiology
TCoE
2 hrs
Cardiology
TCoE
1 hr
EKG Order
TCoE
2 hrs
Lab
TCoE
2 hrs
Order Entry
TCoE
4 hrs
Patient
Accounting
TCoE
3 hrs
4 hrs
4 hrs
6 hrs
4 hrs
4 hrs
2 hrs
2 hrs
3 hrs
6 hrs
2 hrs
3 hrs
4 hrs
4 hrs
B D A
B A F T
B D M
B A F T
B D M F T
B A F T
B D M
B D M
B D M
B D M
B D A F T
B D A F T
Day 1 Day 2 Day 3 Day 4 Day 5
Current State
Total Execution Effort:
70 hrs
Application Execution:
22 hrs
Coordination/Wait Time:
48 hrs
LEGEND
D – Interface has dependencies for
completion
M/A – Batch is Automated or run Manually
B – Interface is Batch
F – Batch run frequency has an influence
T – Batch run timing has an influence
Professional Charges
Technical Charges
Real-time Interfaces
B D M A F T

11. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
Approaches used to reduce the wait times and speed up the End to End testing process
E2E Wait Time Reduction Approach
1
2
3
4
5
Analyzed Interfaces - Identified opportunities in interfaces to
reduce wait times in End to End Testing
Automation - Found opportunities to automate application
interactions and interfaces such that wait times are reduced
Environment Management - Analyzed opportunities related to
factors that cause environment unavailability/wait times
QA Lead Coordination efforts - Explored opportunities to reduce
QA Lead coordination efforts during the life cycle of test execution
Shared Services - Explored opportunities to use shared services
for testing, i.e. using the same resource for multiple applications

12. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Issue to Impact
Key Initiative - Automation and Test Data Management
Long Testing Cycles
All phases of testing were manual,
resulting in long testing cycles.
Handoff between applications for
SIT was manual which induced
unwanted wait times when testers
were not collocated
Test Data Generation
Requests from downstream
systems were queued up due to
unavailability of testing resources to
generate required test data.
Inconsistent Artifacts
Test execution results were not
documented in a consistent and
centralized manner by each
participating application testing
group. This caused challenges in
meeting audit requirements.
Report Validation
Verification and validation of reports
which contains huge volume and
large combination of data.
Verification of reports can be, both
time consuming and erroneous.

13. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Our Approach to Automation and TDM
Automatic application screen capture
and upload to HP QC for documentation
of results and to support test audits
Leveraged Deloitte’s Data Conversion &
Validation (DCAV) tool that enabled
automated data comparison for
converted data and enabled analysis of
discrepancies through an audit report
Implementing a robust test automation
framework helped reduce E2E test
cycle time by 35%. Automated over
2,000 system test cases across top
80% of applications by testing volume
Registered over 30,000 patients and
placed over 25,000 orders for
downstream functional testing saving
over 15,000 hours of manual effort

14. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Leveraging tools across the TCoE
• Industry Print is Deloitte's time-
tested business process modeling
method, tool, and collection of
cross-functional business process
designs
• The repository of Industry Prints
provides a starting point for
business process design and
transformation efforts
• HP Quality Center (HPQC)
empowers organizations to
manage the core application
lifecycle, granting application
teams the crucial visibility and
collaboration needed for
predictable, repeatable, and
adaptable delivery of modern
applications
Industry Print 5 HP QC
• Sprinter speeds up the authoring
and execution of manual
software testing, and eliminates
repetitive tasks—such as
injecting data, running repetitive
tests on different platforms, and
keeping track of steps
• Deloitte’s Test Workbench™ is
an industry contextualized
integrated testing platform. It is
designed to be a one stop shop
for testing activities that can also
help harness Deloitte's industry
expertise for effective and
efficient test delivery
HP Sprinter Test Workbench™

15. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
In Summary
Achievements and Next Steps
• RBT approach optimized the total number of test cases by 70%
• An adjustable, customizable, and reusable test case archive established to assess and predict business risk
• 85% optimization achieved in data selection
• Execution effort reduction of 75%
• Achieved defect density of 0.253
• Reduced the cost to risk ratio by identifying and prioritizing high risk areas
• Testing effort reduced by 40%
Achievements till date
• Transform from a Testing organization to a Quality organization
• Expand TCoE services to the enterprise
• Testing perceived as an trusted partner and a strong decision support system
• Focus on continuous improvements to further drive down cost of quality
Next Steps

16. Thank you