Outcomes Evaluation: The Future for CME


Published on

Published in: Health & Medicine, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Outcomes Evaluation: The Future for CME

  1. 1. Educational Outcomes Evaluation: The Future for CME James C. Leist, EdD Staff Consultant Alliance for Continuing Medical Education
  2. 2. Disclosure  Part time CME consultant  This format and some of the content were developed by Donald Moore, Vanderbilt University, Robert Addleton, Medical Association of Georgia and Derek Dietze, Improve CME
  3. 3. Plan for Session Challenge-based Approach – Scenario – Independent Reflection – Small and Large Group Work – Mini Lecture – Further Reflection and Integration
  4. 4. Challenge Scenario  In her recent meeting with the Hospital CEO, Maureen O. Measure, BSN, MEd, the CME Director for Hopewell General Hospital, was asked about the value of CME. Maureen noted that they have done many programs for all physicians in the hospital system.  The CEO asked whether CME improved physician practice and patient care.  Maureen said that she didn’t have any data.
  5. 5. Challenge Scenario  The CEO asked if QI data were needed and agreed to appoint her to the QI Committee for the Hospital. She agreed anxiously.  In her third meeting with the QI Committee, Maureen listened to a report about the varied levels of physician prescriptions of ACE inhibitors for patients with CHF. How could the physician behavior be changed to assure that at least 80% of patients received the appropriate dosage of ACE I?
  6. 6. Challenge Scenario Questions  What can a CME provider contribute to this scenario?  If a CME intervention is provided, what outcomes should be evaluated and how?
  7. 7. Challenge Individual Reflection For the next 5 minutes, reflect on the challenges listed below: – What are Outcomes? – How do you evaluate the outcomes of a CME activity for this scenario?
  8. 8. Challenge Small Group  In your small groups in the next 10 minutes, share the results of your individual reflection on the definition of outcomes and how you would address the outcomes of a CME activity for this scenario.  Designate one member of the small group to share your results with the large group, time permitting.
  9. 9. Outcomes Evaluation Mini-Lecture  Learning Objectives for session – Define Educational Outcomes – Describe how CME providers in the Hospital/Health Systems Venue can contribute to more sophisticated educational outcomes – Apply measurement of educational outcomes to one CME activity when you return to your venue
  10. 10. Outcomes Defined  An outcome is defined as – the result, or effect, of an event – the consequences of an action  In CME, an educational outcome is the result, or effect, of CME activities  Evaluation is the process for measuring outcomes  Key to Outcomes Evaluation is to plan for the results that are desired – NEEDS ASSESSMENT DATA ARE CRITICAL
  11. 11. Yogi Berraism “If you don’t know where you’re goin’, you ain’t gonna get there” “Yogi Berra”
  12. 12. Process for Evaluating Outcomes  Conduct a needs assessment - practice gap  Determine if education can address the gap  Identify the results of the educational intervention  Develop/Implement the intervention/s - target audience and Faculty  Measure the results/outcomes  Share the results with appropriate stakeholders  Repeat and Improve the process
  13. 13. Practice Gap - Individual and/or Organizational  Current Practice(Problem/Need) - Current level  Desired Result - Practice/Performance level desired  Gap - The Difference between the desired result and the current practice  Will an educational intervention/interventions address the identified gap.
  14. 14. A Continuum, or Levels, of Educational Outcomes/Results  Attendance  Satisfaction  Learning - KSA (Knows)  Competence (Knows/Shows How)  Performance/Behavior in Practice - Individual and Organizational (Does)  Patient Outcomes  Community Health Status
  15. 15. Evaluation - A Continuum of Educational Outcomes  Current Practice - Generally – Attendance – Satisfaction – Learning (Knowledge, Skills and Attitudes)  Most Desirable, More Difficult and Required – Competence – Performance/Behavior in Practice - Individual and Organizational – Patient Outcomes
  16. 16. Outcome Evaluation by Venues Venues Results Office- based Practice Hospital/ Health System Specialty Societies Med. Ed. Comm. Co. Medical School Attendance/ Satisfaction X X X X X Learning - KSA X X X X X Competence X X X X X Practice X X X? X? X? Patient Outcomes X X X? X? Community X X X X?
  17. 17. How do we Evaluate Outcomes?  Accurate data-based needs assessment of desired/current practice required to document need for change and provide baseline for measurement of improvement in practice.  Educational Formats  Data Collection Strategies – Indirect – Self-report by physicians – Direct – Actual performance data that describes performance, or result of the performance
  18. 18. Educational Formats  Conferences – Advantages – Disadvantages  Regularly Scheduled Series - RSS – Advantages – Disadvantages
  19. 19. Data Collection Strategies - Indirect  Outcome – Physicians will prescribe ACE inhibitors in the appropriate dosage for patients with CHF in every case when it is indicated  Measure – Prescriptions for ACE inhibitors  Methods – Survey; focus group; interview cardiologists
  20. 20. Data Collection Strategies - Direct  Outcome – Physicians will prescribe ACE inhibitors in the appropriate dosage for patients with CHF in every case when it is indicated  Measure – Prescriptions for ACE inhibitors  Methods – Chart audit; chart recall; prescription pads; observation; QI/PI data
  21. 21. Learning (Knows)  Methods for Measuring Knowledge, Skills and Attitudes – Post program evaluations  Listing of what you learned (most important thing that you learned) – Pre- and Post - Tests – Others?
  22. 22. Competence (Knows/Shows How) Minimum evaluation standard  Methods for Measuring Competence – Post Test with Case Study/Scenario-Asks how – Commitment to Change with Explanation - What do you plan to change and how? – Demonstration of how to apply a skill learned in the CME activity – Action Plan for use of knowledge learned. – Others?
  23. 23. Performance/Behavior in Practice  Methods for Measuring Performance in Practice – Indirect (1-3 Months)  Follow up Survey - Self Report  Observed practice  Others? – Direct (3-12 Months)  Monitor Clinical Data - organizational  Chart Audit - individual and organizational  Others?
  24. 24. Patient Outcome  Methods for measuring patient outcomes – Monitor health status data for target audience – Monitor patient practices related to need addressed – Others?
  25. 25. Value of CME Formats(Conference/RSS) in Each Outcome Level  Learning  Competence  Performance  Patient Outcome
  26. 26. Challenge Scenario Questions  What can a CME provider contribute to this scenario?  If a CME intervention is provided, what outcomes should be evaluated and how?
  27. 27. How Do We Evaluate CME in Changing Physician Practice/Performance?  Conduct effective Needs Assessment to establish a baseline of performance, or health status, and identify the gap between what is and what should be. (EXAMPLE)  Numerous RCTs have shown that in patients with CHF, ACE Inhibitors – Reduce mortality by 20-25% – Increase functional status and quality of life – Decrease hospitalization and death by 35-40%  ACE Inhibitors are cost effective  Currently, ACE-I are prescribed for only 50% of pts.
  28. 28. How Do We Evaluate CME in Changing Physician Practice/Performance?  Re-measure the specific practice after the educational intervention to assure desired result. (EXAMPLE)  Measure performance after CME/activity/s – Method  Direct Measurement of performance  Chart Audit – Results: 75% of the prescriptions written by physicians for their CHF patients were ACE I, when indicated
  29. 29. Role for CME Provider - 1  Ask the right questions when planning CME activity. – Are there data to reflect a need and provide a baseline for performance of individual/organization? – What result do you want from the CME intervention?  Work directly with your defined population, such as a physician practice, or your institution, to identify the gap in practice/health status.  Partner with units that have data that reflect need for change in practice/health status and that continually re- measure performance. (P&T Comm., QI/PI Comm.)
  30. 30. Role for CME Provider - 2  Learn about performance measurement and measure and share results  Learn to do chart audits, or teach physician office staff to conduct audits, or gather other data, for assessment of performance/health status as a standard procedure.
  31. 31. Role for CME Provider - 3  Make physicians Aware of gap, Agree on need to change, Adopt the new practice and Adhere to new practice.(Committee members or Influential physicians)  Use educational practices of – Predisposing – Create awareness of need to change – Enabling – Provide tools (education) to make change – Reinforcing – Follow up to support needed change
  32. 32. CME Profession Competencies  Adult/Organizational Learning Principles  Educational Interventions  Performance Measurement  Systems Thinking  Partnering  Leadership  Administration/ Management  Self Assessment and Lifelong Learning
  33. 33. CME Profession Competencies  Performance Measurement – Use appropriate data to assess educational and administrative performance of the CME Program  Partnering – Identify and collaborate with key partners and stakeholders in accomplishing their CME mission  Systems Thinking – Continually assess individual and organizational performance and make improvements through relevant learning experiences.
  34. 34. Lessons Learned  Accurate Needs Assessment Data a Necessary Foundation For Outcomes Evaluation  Re-measure Interventions  Use 3 CME Profession Competencies in Outcomes Evaluation – Performance Measurement – Partnering – Systems Thinking  CME must provide outcomes evaluation to contribute to effective health care in the future  Role of CME Provider is expanding from a meeting planner to learning facilitator, or learning consultant  DATA, DATA, DATA
  35. 35. Other Lessons or QUESTIONS?
  36. 36. What should YOU do when you return to work in your setting/venue?  Identify one area where performance or health status is necessary for improvement  Ask the right questions in the planning process to assure need and desired results  Re-measure performance or health status after educational intervention/s to document results  Develop at least one new competency  Promote the effectiveness and value of CME
  37. 37. References/Resources  ACME Evaluating Educational Outcomes-An Electronic Workbook for CME Providers www.acme-assn.org/workbook.shtml (Available Spring 2009)  CMExchange  AMA Physician Consortium on Performance Development, www.ama-assn/ama/pub/category/2946.html