Benefits Evaluation of Health Information Technology Projects
•Download as PPTX, PDF•
2 likes•454 views
The document discusses benefits evaluation, which involves measuring and analyzing the impacts of an intervention or change. It outlines the benefits evaluation process, including planning evaluations, defining indicators, collecting pre-implementation data, and analyzing post-implementation data. The goal is to demonstrate achievement of project objectives, improve future projects, and inform decisions about current and future initiatives.
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Benefits Evaluation of Health Information Technology Projects
Similar to Benefits Evaluation of Health Information Technology Projects (20)
Recently uploaded
Recently uploaded (20)
Benefits Evaluation of Health Information Technology Projects
- 1. Shawn Quait, RN, BScN HITenlightenment.com
- 2. • Understand what Benefits Evaluation is and why it is important • Understand how project benefits fit into organizational strategy and strategic priorities • Understand the Benefits Evaluation process HITenlightenment.com
- 4. Activity in which the impacts of an intervention or change are measured and analyzed Pre-Implementation Compare Post-Implementation Benefits HITenlightenment.com
- 5. • Inform decisions of current/future projects • Engage stakeholders • Advance investment in IM initiatives • Enhance end user adoption • Focus project efforts • Demonstrate achievement of project objectives • Improve outcomes of future projects • Inform solution design HITenlightenment.com
- 7. Benefits Net Government / Benefits Ministry Initiatives System Quality Quality of Care Use Service Access Quality User Information Satisfaction Quality Productivity Organizational Strategy and Strategic Priorities HITenlightenment.com
- 9. Monitoring and Controlling PMBOK Initiation (I) Planning (II) Executing (III) Closing (IV) Go Live IT Project Initiation Planning Design Build and Test Delivery Support Lifecycle Post-Implementation Plan Benefits Measurement (I) Data Collection and Analysis (III) Benefits Evaluate Technology Define Indicators (II) Evaluation Benefits (IV) Operationalize Pre-Implementation Data Collection and Analysis (III) Indicators (V) HITenlightenment.com
- 10. Monitoring and Controlling PMBOK Initiation (I) Planning (II) Executing (III) Closing (IV) Go Live IT Project Initiation Planning Design Build and Test Delivery Support Lifecycle Post-Implementation Plan Benefits Measurement (I) Data Collection and Analysis (III) Benefits Evaluate Technology Define Indicators (II) Evaluation Benefits (IV) Operationalize Pre-Implementation Data Collection and Analysis (III) Indicators (V) HITenlightenment.com
- 11. • Benefit Evaluation Planning • Scope, budget, schedule, deliverables and resources • Governance structure • Stakeholders and audience for report • Evaluation indicators and studies • Plan is drafted in initiation, refined in planning, and finalized in the design phase HITenlightenment.com
- 12. • Each project will have a benefits evaluation lead • Responsibilities • Stakeholder identification and relations • Planning and coordination of the benefits evaluation • Formation of benefits evaluation working group • Status reporting HITenlightenment.com
- 13. Monitoring and Controlling PMBOK Initiation (I) Planning (II) Executing (III) Closing (IV) Go Live IT Project Initiation Planning Design Build and Test Delivery Support Lifecycle Post-Implementation Plan Benefits Measurement (I) Data Collection and Analysis (III) Benefits Evaluate Technology Define Indicators (II) Evaluation Benefits (IV) Operationalize Pre-Implementation Data Collection and Analysis (III) Indicators (V) HITenlightenment.com
- 14. Defining Indicators consists of 4 steps 1. Create Benefits Description List 2. Prioritize Benefits 3. Create Indicator Definitions 4. Secure Approval for Indicator Definitions List HITenlightenment.com
- 15. Benefit Reduction in medication incidents. Benefit Domain: Drug Information System Program Ranking: Classification Major Benefits Dimension: Net Benefits 5 Category: Quality Sub-Category: Patient Safety Definition Number of Medication Incidents (numerator) per X number of medications ordered (denominator). Benchmark or 100% Reduction in medication incidents. Target Benefit Realisation Immediately post implementation. Time Frame Method / Source Measurement Numerator: Number of Medication Incidents Denominator: Total number of medications Phase ordered Pre Measurement/Analysis Responsibility: Measurement/Analysis Responsibility: Implementation Benefits Evaluation Lead Benefits Evaluation Lead Data Capture: Medication incident reports. Data Capture: Through the pharmacy information system Assumptions / Constraints: Only medication Indicator Incidents that have resulted from errors in the Assumptions / Constraints: The total number medication ordering and transcription processes of medications, rather than medication orders Definition (including unclear handwriting) will be counted will be considered to eliminate discrepancies in eliminating those due to adverse drug reactions. comparisons between units due to variations in Example However, those due to drug interactions should the number of medications within each order. be included. The present medication incident report should carry adequate information to determine the cause. Post Data Capture: Medication incident reports. Data Capture: Through the pharmacy Implementation information system. Assumptions / Constraints: Same as in Pre- Implementation. Assumptions / Constraints: Same as in Pre- Implementation.
- 16. • Patient Satisfaction with system (If applicable) • User Satisfaction with system implementation • System Utilization HITenlightenment.com
- 17. • Try to make data collection and objective and quantitative wherever possible • Example: Collecting data using a question on a survey “% of respondents who felt that unclear medication orders were significantly reduced”, rather than actually measuring it “Number of unclear orders per 100 medications ordered”
- 18. Monitoring and Controlling PMBOK Initiation (I) Planning (II) Executing (III) Closing (IV) Go Live IT Project Initiation Planning Design Build and Test Delivery Support Lifecycle Post-Implementation Plan Benefits Measurement (I) Data Collection and Analysis (III) Benefits Evaluate Technology Define Indicators (II) Evaluation Benefits (IV) Operationalize Pre-Implementation Data Collection and Analysis (III) Indicators (V) HITenlightenment.com
- 19. The Data Collection and Analysis phase consists of 4 steps: 1. Develop Data Collection Process 2. Trial Data Collection 3. Data Collection Proper 4. Data Analysis HITenlightenment.com
- 20. Monitoring and Controlling PMBOK Initiation (I) Planning (II) Executing (III) Closing (IV) Go Live IT Project Initiation Planning Design Build and Test Delivery Support Lifecycle Post-Implementation Plan Benefits Measurement (I) Data Collection and Analysis (III) Benefits Evaluate Technology Define Indicators (II) Evaluation Benefits (IV) Operationalize Pre-Implementation Data Collection and Analysis (III) Indicators (V) HITenlightenment.com
- 21. Benefits Evaluation Report • Benefits Achieved • Benefits Not Achieved • Negative Impacts • Lessons Learned • Evaluation Sustainability Plan • Recommendations formed HITenlightenment.com
- 22. Monitoring and Controlling PMBOK Initiation (I) Planning (II) Executing (III) Closing (IV) Go Live IT Project Initiation Planning Design Build and Test Delivery Support Lifecycle Post-Implementation Plan Benefits Measurement (I) Data Collection and Analysis (III) Benefits Evaluate Technology Define Indicators (II) Evaluation Benefits (IV) Operationalize Pre-Implementation Data Collection and Analysis (III) Indicators (V)
- 23. • Evaluation continues after solution is operational • Benefits indicators selected for ongoing measurement become performance measures • Information Management Monthly report will display the performance measures as appropriate HITenlightenment.com
- 24. Monitoring and Controlling PMBOK Initiation (I) Planning (II) Executing (III) Closing (IV) Go Live IT Project Initiation Planning Design Build and Test Delivery Support Lifecycle Plan Benefits Measurement (I) Post-Implementation Data Collection and Develop Preliminary Refine and Finalize Benefits Evaluation Plan Analysis (III) Benefits Evaluation -Augment Benefits Description list with 1. Trial data collection Plan workflow benefits 2. Conduct data -Define scope of -Refine and finalize Benefits Description List collection proper benefits evaluation -Refine and finalize indicator definitions and 3. Analyze data -Define stakeholders evaluation studies -Administer User -Define audience for -Refine and update Benefits Evaluation Plan Satisfaction Survey final report -Review final Benefits Evaluation Plan with -Administer Patient -Define indicators governance structure and sign off Satisfaction Survey (If and evaluation applicable) studies - Consider ethics of measurement plan -Create first draft of Evaluate Technology Benefits Evaluation Benefits (IV) Benefits Plan -Complete Benefits Evaluation -Review initial plan Evaluation Report with governance - Present benefits structure evaluation results to governance structure as part of project Define Indicators (II) closure 1.Create Benefits Description List 2.Prioritize benefits 3.Create Indicator definitions Operationalize 4.Secure approval for Indicator definitions from governance structure Indicators (V) -Identify operational performance measures Pre-Implementation Data Collection and Analysis (III) - Develop reporting 1. Develop data collection process strategy for 2. Trial data dollection operational 3. Conduct data collection proper performance 4. Analyze data measures
- 25. Connect with me on LinkedIn/shawnquait Shawn Quait, RN, BScN HITenlightenment.com
Editor's Notes
- Evaluating the Benefits of Health Information Technology Projects
- ObjectivesThe objectives of this session are to: - Understand what Benefits Evaluation is and why it is important- Understand how project benefits fit into organizational strategy and strategic priorities- Understand the Benefits Evaluation process
- What is Benefits Evaluation and why is it important?
- What is Benefits Evaluation?Benefits evaluation is an activity in which the impacts of an intervention or change are measured and analyzed. The aim of a benefits evaluation effort is to make a comparison between the pre and post change environments.
- Why is Benefits Evaluation Important?Evaluating the benefits of IM project implementations will:- Inform decisions of current/future projects - ensures the organization has an objective view of the real benefits of an IM initiative- Engage stakeholders - use evidence to enhance patient, staff, and executive support for future IM initiatives- Advance investment in IM initiatives - through demonstrating the value of investments, we advance further investment- Enhance end user adoption - understanding the benefits of IM initiatives will encourage and ultimately improve end user adoption- Focus project efforts - focus the objectives and priorities of the project towards the benefits- Demonstrate achievement of project objectives - allow you to demonstrate that you have achieved what you have set out to do- Improve outcomes of future projects - use lessons learned from benefits evaluation to inform decision making for future projects- Inform solution design - findings will influence decisions on technology system functionality and workflow is adjusted based on project goals
- How do project benefits fit into organizational strategy and strategic priorities?
- Benefits Evaluation FrameworkCanada Health Infoway definesbenefits as having six conceptual dimensions. The quality of the information, system, and service can affect the extent of system use, intention to use, and user satisfaction. In turn, system usage and satisfaction can lead to positive and negative impacts at the individual and organizational levels, which are collectively viewed as Net Benefits (Quality of Care, Access, and Productivity). It is important that IM project benefits align with organization strategy and strategic priorities. To ensure alignment organizational strategy and strategic priorities will influence the benefits selected as priority to analyze. Sometimes projects are driven by Government or other Ministry Initiatives, and this is reflected in the model as an influence of the benefits examined.
- What is the Benefits Evaluation process?
- Benefits Evaluation Process OverviewThis diagram illustrates how the benefits evaluation phases fit into the PMO methodology and IT project lifecycle. The phases are not discrete, and do overlap. We will present a brief overview of the activities of each phase of the benefits evaluation process.
- Benefits Evaluation Process Overview: Plan Benefits Measurement
- Plan Benefits EvaluationScope - The scope outlines the proposed benefits under study and has a description of each benefit area. The scope of the benefit evaluation and corresponding descriptions are added to the project charter. Only benefits that will actually be measured should be included in the project charter. Benefits should be SMART (Specific, Measurable, Attainable, Realistic, and Time Driven).Budget - A separate budget is not developed for the benefits evaluation, however the costs should be accounted for as part of the main project. Schedule/Deliverables/Resources - The schedule, deliverables, and resources for the benefits evaluation work package should be defined and included in the overarching project plan. Governance structure - A governance structure should be established for benefits evaluation (e.g. steering committee). The governance structure is responsible for the review and approval of the initial and final benefits evaluation plan, and any changes or updates to this plan. Stakeholders and audience for report - Stakeholders can be internal or external to the organization, and should include any group or individual who may affect or be affected by the achievement of the project’s objectives (e.g. committees, management, patients, staff, etc.). From the stakeholders you identify those that will be the audience for the report. Evaluation indicators and studies - There should be a proposed indicator and measurement strategy defined for each benefit.PMO Methodology - The expectation is that a preliminary benefits evaluation plan is created in the initiation phase, refined in the planning stage, and finalized in the design phase. Final approval for the benefits evaluation plan should be obtained from the governance structure. All indicators should be defined and approved by the end of the design phase.
- Benefits Evaluation Lead Each project will have a person designated by the project manager as the benefits evaluation lead. If it is a smaller project, the project manager themselves may need to fulfill this position. Clinically focused projects will have a clinical benefits evaluation lead, and technically focussed projects will have a business benefits evaluation lead. Depending on the scope of the project it may be necessary to have both a clinical/business lead. Responsibilities include: - Stakeholder identification and relations- Planning and coordination of the benefits evaluation work package - Formation of benefits evaluation working group - Status reporting
- Benefits Evaluation Process Overview: Define Indicators
- Define IndicatorsStep 1: Create Benefits Description ListThe first step in defining indicators is to create a list of expected benefits from the project implementation. This list of expected benefits becomes the driver of indicators to use. To help identify important benefits to evaluate you can use strategies such as: brainstorming, literature review, workflow analysis, analyzing negative impacts, examining potential impacts on corporate and program KPIs, and consulting with experts for their opinion. To ensure the benefits measurement is unbiased, it is important to identify and present the negative impacts of project (e.g. CPOE will increase Physician ordering time). Presenting and evaluating these impacts is imperative for an objective benefits measurement. Step 2: Prioritize BenefitsThe second step is to prioritize the benefits description list by ranking benefits by their feasibility and importance. Indicators are re-assessed for inclusion based on importance and feasibility of measurement. Step 3: Create Indicator DefinitionsThe third step is to create a clear and succinct indicator definition for each benefit from the Benefits Description List. Part of the definition process is to describe where or how the required data can be obtained for the indicators both pre and post implementation.Step 4: Secure Approval for Indicator Definitions ListKeep the final number of indicators to a manageable level. The actual value of the primary project would depend on the complexity, priority level and the affected business area. Extensive and costly systems that have a direct impact on patient safety and quality of care will naturally have more indicators. A fair figure for such a project may be 10 to 20. Obtain final approval for the final list of indicator definitions from the governance structure. All indicators should be defined and approved by the end of the design phase of the project lifecycle.
- Indicator Definition Example
- Mandatory IndicatorsPatient Satisfaction with system - Should be measured if patient and/or care provider workflows are directly impacted (Survey background and administration instructions included in toolkit). User Satisfaction with system implementation - Should be measured for every system implementation (Survey background and administration instructions included in toolkit). System Utilization - Always should be measured. You must include a few indicators based on system utilization in post implementation studies. The extent of benefits realized through the solution is directly dependent upon the level of utilization (e.g. CPOE system - A 50% reduction of medication incidents at 50% system utilization vs. that at 100% system utilization).
- Qualitative data (e.g. user/patient testimonials). Try to make data collection objective and quantitative, rather than subjective and qualitative wherever possible (e.g. Collecting data using a question on a survey “% of respondents who felt that unclear medication orders were significantly reduced”, rather than actually measuring it “Number of unclear orders per 100 medications ordered”).
- Benefits Evaluation Process Overview: Pre-Implementation and Post-Implementation Data Collection and Analysis
- Data Collection and AnalysisThe Data Collection and Analysis phase consists of 4 steps: 1. Develop Data Collection Process (Consider sources of data, modes of data collection, methods of recording data, who will perform the data collection, timing of data collection, and sampling) 2. Trial Data Collection (Very important step as it will help to validate your data collection process and forms, assess the resource requirements for the data collection, and evaluate the volume of data you will get, and most importantly prevent wasting resources and costly mistakes)3. Data Collection Proper (Once the data collection forms and plan are completed you can initiate data collection proper)Pre-Implementation - Baseline measurements should be made before any changes have occurred. Post-implementation - Although pre-implementation phase of benefits measurement coincides with the regular project phases the post implementation does not. Benefits monitoring will have a more flexible end date and can be expected to extend well beyond the project’s end date. There may be a delay between the system implementation and the realization of the benefits (e.g. if the benefits depends on the capture of additional data which will be populated over a period of time; or on the adoption of a new working practice which may take time to become embedded). Staffing for post implementation measurement will need to be planned in advanced to ensure effective completion. As a general rule of thumb three months is acceptable for post implementation data collection to initiate, however for complex projects an evaluation at six months may be necessary. 4. Data Analysis - Once the data collection is completed the analysis process can begin. The depth of analysis depends on the target audience.
- Benefits Evaluation Process Overview: Evaluate Technology Benefits
- Evaluate Technology Benefits Final comparison of pre and post benefits data is completed to evaluate the benefits of the IM project. The report is organized to illustrate how the information management project benefits align with the Benefits Evaluation Framework Final report is created and presented. Benefits Evaluation Report - Benefits Achieved - Analysis/comparison between pre and post implementation environments.- Benefits Not Achieved - Outline the reasons for not achieving/abandoning and whether any remedial action can be undertaken.- Negative Impacts - Analysis/comparison between pre and post implementation environments. - Lessons Learned - Analysis of lessons learned from the benefits measurement.- Evaluation Sustainability - Plan for ongoing benefits evaluation should be clearly articulated with reference to indicators to be used once - system is operational. - Recommendations – Any recommendations formed as a result of the benefits evaluation.
- Benefits Evaluation Process Overview: Operationalize Indicators
- Operationalize Indicators Evaluation continues after solution is operational. The project and benefit evaluation plan should have well thought out arrangements for the handover of benefits monitoring responsibilities after closing of the project. The benefit indicators that are selected for on-going measurement become performance measures. Performance Measurement is defined as a periodic, repetitive activity of measuring a process or its output to determine if there has been a change. Performance indicators will be reported on in the Information Management Monthly report as appropriate (e.g. If an indicator is reported on quarterly it will display in the report quarterly).
- Benefits Evaluation Summary