Dubrowski assessment and evaluation

2,176 views

Published on

Panel presentation for the Pain Symposium 2010

Published in: Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,176
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
124
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Dubrowski assessment and evaluation

  1. 1. Adam Dubrowski PhD<br />Models for Assessing Learning and Evaluating Learning Outcomes<br />
  2. 2. www.wordle.net<br />
  3. 3. Acknowledgement<br />Dr. Kathryn Parker, Learning Institute<br />
  4. 4. Introduction<br />
  5. 5. Introduction<br />Is Eli OK?<br />
  6. 6. Introduction<br />Environment<br />Intake<br />Interaction<br />Eli<br />Reaction<br />Function<br />
  7. 7. Content<br />Assessment<br />Evaluation: Models and Frameworks <br />Working Example: Evaluation of a Hand Hygiene Program<br />Assessment Instruments<br />Standards and Rigor: Assessment Instruments<br />Working Example: OSATS<br />Standards and Rigor: Evaluations<br />Summary and Take-home Message <br />
  8. 8. Introduction<br />Formative<br />Assessments<br />Summative<br />
  9. 9. Introduction<br />Evaluations<br />Assessments<br />
  10. 10. Introduction<br />Process<br />Evaluations<br />Outcome<br />
  11. 11. Introduction<br />Process<br />Evaluations<br />Outcome<br />
  12. 12. Introduction<br />Alkin, M.C. & Christie, C.A. (2004). The evaluation theory tree. In Alkin, M.C. (ed.), Evaluation Roots: Tracing Theorists’ Views and Influences. London: Sage Publications.<br />
  13. 13. Introduction<br />
  14. 14. Reflections<br />Where would you sit on the Evaluation Tree?<br />What about others in your “system”?<br />
  15. 15. Introduction<br />Alkin, M.C. & Christie, C.A. (2004). The evaluation theory tree. In Alkin, M.C. (ed.), Evaluation Roots: Tracing Theorists’ Views and Influences. London: Sage Publications.<br />
  16. 16. Introduction<br />
  17. 17. Models and Frameworks<br />CIPP Evaluation Model (Stufflebeam, 1983)<br />Kirkpatrick’s Model of Evaluating Training Programs (Kirkpatrick, 1998)<br />Moore’s Evaluation Framework (Moore et al., 2009)<br />Miller’s Framework for Clinical Assessments (Miller, 1990) <br />
  18. 18. CIPP<br />CIPP: Context, Input, Process, Product<br />Stufflebeam, 1983; Steinert, 2002<br />
  19. 19. CIPP<br />Context<br />What is the relation of the course to other courses?<br />Is the time adequate?<br />Should courses be integrated or separate?<br />
  20. 20. CIPP<br />Inputs<br />What are the entering ability, learning skills and motivation of students?<br />What is the students’ existing knowledge?<br />Are the objectives suitable?<br />Does the content match student abilities?<br />What is the theory/practice balance?<br />What resources/equipment are available?<br />How strong are the teaching skills of teachers?<br />How many students/teachers are there?<br />
  21. 21. CIPP<br />Process<br />What is the workload of students?<br />Are there any problems related to teaching/learning?<br />Is there effective 2-way communication?<br />Is knowledge only transferred to students, or do they use and apply it?<br />Is the teaching and learning process continuously evaluated?<br />Is teaching and learning affected by practical/institutional problems?<br />
  22. 22. CIPP<br />Product<br />Is there one final exam at the end or several during the course?<br />Is there any informal assessment?<br />What is the quality of assessment?<br />What are the students’ knowledge levels after the course?<br />How was the overall experience for the teachers and for the students?<br />
  23. 23. CIPP<br />Methods used to evaluate the curriculum<br />Discussions<br />Informal conversation or observation<br />Individual student interviews<br />Evaluation forms<br />Observation in class by colleagues<br />Performance test<br />Questionnaire<br />Self-assessment<br />Written test<br />
  24. 24. CIPP<br />CIPP focuses on the process and informs the program/curriculum for future improvements.<br /> Limited emphasis on the outcome.<br />
  25. 25. Kirkpatrick’s Model of Evaluating Training Programs <br />Results<br />Behavior<br />Learning<br />Reaction<br />
  26. 26. Kirkpatrick’s Model of Evaluating Training Programs <br />Results<br />Behavior<br />Learning<br />Reaction<br />Reaction: The degree to which the expectations of the participants about the setting and delivery of the learning activity were met <br />Questionnaires completed by attendees after the activity <br />
  27. 27. Kirkpatrick’s Model of Evaluating Training Programs <br />Results<br />Behavior<br />Learning<br />Reaction<br />Learning: The degree to which participants recall and demonstrate in an educational setting what the learning activity intended them to be able to do <br />Tests and observation in educational setting<br />
  28. 28. Kirkpatrick’s Model of Evaluating Training Programs <br />Results<br />Behavior<br />Learning<br />Reaction<br />Behavior: The degree to which participants do what the educational activity intended them to be able to do in their practices <br />Observation of performance in patient care setting; patient charts; administrative databases <br />
  29. 29. Kirkpatrick’s Model of Evaluating Training Programs <br />Results: The degree to which the health status of patients as well as the community of patients changes due to changes in the practice<br />Health status measures recorded in patient charts or administrative databases, epidemiological data and reports<br />Results<br />Behavior<br />Learning<br />Reaction<br />
  30. 30. Kirkpatrick’s Model of Evaluating Training Programs <br />The importance of Kirkpatrick’s Model is in its ability to identify a range of dimensions that needs to be evaluated in order to inform us about the educational quality of a specific program.<br />It focuses on the outcomes.<br />However, it provides limited information about the individual learner.<br />
  31. 31. Moore’s Framework<br />Results<br />Behavior<br />Learning<br />Reaction<br />Does<br />Shows How <br />Knows How <br />Knows<br />
  32. 32. Miller’s Framework for Clinical Assessments<br />Does<br />(Performance)<br />Shows How <br />(Competence)<br />Knows How <br />(Procedural Knowledge)<br />Knows<br />(Declarative Knowledge)<br />
  33. 33. Miller’s Framework for Clinical Assessments<br />Does<br />(Performance)<br />Shows How <br />(Competence)<br />Knows How <br />(Procedural Knowledge)<br />Knows<br />(Declarative Knowledge)<br />Knows: Declarative knowledge. The degree to which participants state what the learning activity intended them to know<br />Pre- and posttests of knowledge. <br />
  34. 34. Miller’s Framework for Clinical Assessments<br />Does<br />(Performance)<br />Shows How <br />(Competence)<br />Knows How <br />(Procedural Knowledge)<br />Knows<br />(Declarative Knowledge)<br />Knows how: Procedural knowledge. The degree to which participants state how to do what the learning activity intended them to know how to do <br />Pre- and posttests of knowledge <br />
  35. 35. Miller’s Framework for Clinical Assessments<br />Does<br />(Performance)<br />Shows How <br />(Competence)<br />Knows How <br />(Procedural Knowledge)<br />Knows<br />(Declarative Knowledge)<br />Shows how: The degree to which participants show in an educational setting how to do what the learning activity intended them to be able to do <br />Observation in educational setting <br />
  36. 36. Miller’s Framework for Clinical Assessments<br />Does: Performance. The degree to which participants do what the learning activity intended them to be able to do in their practices.<br />Observation of performance in patient care setting; patient charts; administrative databases <br />Does<br />(Performance)<br />Shows How <br />(Competence)<br />Knows How <br />(Procedural Knowledge)<br />Knows<br />(Declarative Knowledge)<br />
  37. 37. Miller’s Framework for Clinical Assessments<br />The importance of Miller’s Framework is in its ability to identify learning objectives and link them with appropriate testing contexts (where) and instruments (how). <br />
  38. 38. Assumption<br />Moore’s and Kirkpatrick’s models assume a relationship between the different levels.<br />If learners are satisfied, they will learn more, will be able to demonstrate the new skills, transfer them to the clinical setting, and consequently the health of patients and communities will improve!<br />
  39. 39. Models assumptions are not met!<br />
  40. 40. Building Skills, Changing Practice: Simulator Training for Hand Hygiene Protocols<br />Canadian Institutes of Health Research<br />Partnerships for Health System Improvement<br />A. McGeer, MA. Beduz, A. Dubrowski<br />
  41. 41. Purpose<br />Current models of knowledge delivery about proper hand hygiene rely on didactic session<br />However, transfer to practice is low<br />
  42. 42. Purpose<br />Does<br />Shows How <br />Knows How <br />Knows<br />Clinical practice<br />Simulation<br />Didactic courses<br />
  43. 43. Purpose<br />The primary purpose was to investigate the effectiveness of simulation-based training of hand hygiene on transfer of knowledge to clinical practice <br />
  44. 44. Project<br />Recruitment<br />Clinical monitoring (audits)<br />Δ in Behavior<br />Simulation<br />Δ in Learning<br />Reaction<br />Clinical monitoring (audits)<br />
  45. 45. Results: Reactions<br />Simulation Design Scale<br />Educational Practices Questionnaire<br />159respondents - average response:<br />‘Agree’ or ‘Strongly Agree’ (4 or 5 on a 5-pt scale) to each of 40 items<br />Rated 39 of these items as ‘Important’ (4 on a 5-pt scale)<br />
  46. 46. Results: Learning<br />Moment 1 Moment 2 Moment 3 Moment 4<br />
  47. 47. Results: Behaviour<br />
  48. 48. Conclusions<br />No transfer of knowledge<br />No clear relationship between the assessments of reactions, learning and behaviour.<br />
  49. 49. Summary<br />
  50. 50. Summary<br />Outcome based models of program evaluation may be providing limited use (i.e. they are mostly on the research branch).<br />Need new, more complex models that incorporate both processes and outcomes (i.e. span across more branches).<br />
  51. 51. Summary<br />Assessment instruments, which feed these models are critical for successful evaluations.<br />We need to invest efforts in standardization and rigorous development of these instruments.<br />Evaluations<br />Assessments<br />
  52. 52. Assessment Instruments<br />
  53. 53. Assessment Instruments<br />
  54. 54. Standards and Rigor<br />
  55. 55. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments that apply to measuring three properties:<br />Distinguish between two or more groups, <br />assess change over time,<br />predict future status.<br />
  56. 56. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />The concept to be measured needs to be defined properly and should match its intended use.<br />
  57. 57. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />Reliability is the degree to which the instrument is free of random error, which means free from errors in measurement caused by chance factors that influence measurement.<br />
  58. 58. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />Validity is the degree to which the instrument measures what it purports to measure.<br />
  59. 59. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />Responsiveness is an instrument’s ability to detect change over time.<br />
  60. 60. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />Interpretability is the degree to which one can assign easily understood meaning to an instrument's score.<br />
  61. 61. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />Burden refers to the time, effort and other demands placed on those to whom the instrument is administered (respondent burden) or on those who administer the instrument (administrative burden)<br />
  62. 62. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />Alternative means of administration include self report, interviewer-administered, computer assisted, etc. Often it is important to know whether these modes of administration are comparable<br />
  63. 63. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />Cultural and Language adaptations or translations.<br />
  64. 64. Scientific Advisory Committee of the Medical Outcomes Trust<br />The SAC defined a set of eight key attributes of instruments:<br />The Conceptual and Measurement Model<br />Reliability<br />Validity<br />Responsiveness or sensitivity to change<br />Interpretability<br />Burden<br />Alternative Forms of Administration<br />Cultural And Language Adaptations<br />
  65. 65. Scientific Advisory Committee of the Medical Outcomes Trust<br />Do our assessment instruments fit these criteria?<br />
  66. 66. Working Example: OSATS<br />Objective Assessments of Technical Skills<br />Bell-ringer exam (12 minutes per station)<br />Minimum of 6-8 stations<br />Direct or video assessments of technical performance<br />
  67. 67. Working Example: OSATS<br />The Conceptual and Measurement Model<br />Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative "bench station" examination. Am J Surg. 1997 Mar;173(3):226-30.<br />Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997 Feb;84(2):273-8. <br />
  68. 68. Working Example: OSATS<br />Reliability and Validity<br />Faulkner H, Regehr G, Martin J, Reznick R. Validation of an objective structured assessment of technical skill for surgical residents. Acad Med. 1996 Dec;71(12):1363-5.<br />Kishore TA, Pedro RN, Monga M, Sweet RM. Assessment of validity of an OSATS for cystoscopic and ureteroscopic cognitive and psychomotor skills. J Endourol. 2008 Dec;22(12):2707-11. <br />Goff B, Mandel L, Lentz G, Vanblaricom A, Oelschlager AM, Lee D, Galakatos A, Davies M, Nielsen P. Assessment of resident surgical skills: is testing feasible? Am J Obstet Gynecol. 2005 Apr;192(4):1331-8; discussion 1338-40. VAL<br /> <br />Martin JA, Reznick RK, Rothman A, Tamblyn RM, Regehr G. Who should rate candidates in an objective structured clinical examination? Acad Med. 1996 Feb;71(2):170-5. <br />
  69. 69. Working Example: OSATS<br />Alternative Forms of Administration<br />Maker VK, Bonne S. Novel hybrid objective structured assessment of technical skills/objective structured clinical examinations in omprehensiveperioperative breast care: a three-year analysis of outcomes. J Sualrg Educ. 2009 Nov-Dec;66(6):344-51. <br />Lin SY, Laeeq K, Ishii M, Kim J, Lane AP, Reh D, Bhatti NI. Development and<br />pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy. 2009 May-Jun;23(3):354-9. <br />Goff BA, VanBlaricom A, Mandel L, Chinn M, Nielsen P. Comparison of objective, structured assessment of technical skills with a virtual reality hysteroscopy trainer and standard latex hysteroscopy model. J Reprod Med. 2007 May;52(5):407-12. <br />Datta V, Bann S, Mandalia M, Darzi A. The surgical efficiency score: a feasible, reliable, and valid method of skills assessment. Am J Surg. 2006, Sep;192(3):372-8. <br />
  70. 70. Working Example: OSATS<br />Adaptations<br />Setna Z, Jha V, Boursicot KA, Roberts TE. Evaluating the utility of workplace-based assessment tools for specialty training. Best Pract Res ClinObstetGynaecol. 2010 Jun 30. <br />Dorman K, Satterthwaite L, Howard A, Woodrow S, Derbew M, Reznick R, Dubrowski A. Addressing the severe shortage of health care providers in Ethiopia: bench model teaching of technical skills. Med Educ. 2009 Jul;43(7):621-7. <br />
  71. 71. Working Example: OSATS<br />The SAC defined a set of eight key attributes of instruments:<br /><ul><li>The Conceptual and Measurement Model
  72. 72. Reliability
  73. 73. Validity</li></ul> Responsiveness or sensitivity to change<br /> Interpretability<br /> Burden<br /><ul><li>Alternative Forms of Administration
  74. 74. Cultural And Language Adaptations</li></li></ul><li>Working Example: OSATS<br />Applications to program evaluation<br />Chipman JG, Schmitz CC. Using objective structured assessment of technical skills to evaluate a basic skills simulation curriculum for first-year surgical residents. J Am Coll Surg. 2009 Sep;209(3):364-370. <br />Siddighi S, Kleeman SD, Baggish MS, Rooney CM, Pauls RN, Karram MM. Effects of an educational workshop on performance of fourth-degree perineal laceration repair. Obstet Gynecol. 2007 Feb;109(2 Pt 1):289-94. <br />VanBlaricom AL, Goff BA, Chinn M, Icasiano MM, Nielsen P, Mandel L. A new curriculum for hysteroscopy training as demonstrated by an objective structured assessment of technical skills (OSATS). Am J Obstet Gynecol. 2005 Nov;193(5):1856-65. <br />Anastakis DJ, Wanzel KR, Brown MH, McIlroy JH, Hamstra SJ, Ali J, Hutchison CR, Murnaghan J, Reznick RK, Regehr G. Evaluating the effectiveness of a 2-year curriculum in a surgical skills center. Am J Surg. 2003 Apr;185(4):378-85. <br />
  75. 75. Standards and Rigor<br />Evaluations<br />Assessments<br />
  76. 76. The Joint Committee on Standards for Educational Evaluation (1990) <br />Utility Standards <br />The utility standards are intended to ensure that an evaluation will serve the information needs of intended users. <br />
  77. 77. The Joint Committee on Standards for Educational Evaluation (1990) <br />Utility Standards<br />Feasibility Standards<br />The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.<br />
  78. 78. The Joint Committee on Standards for Educational Evaluation (1990) <br />Utility Standards<br />Feasibility Standards<br />Propriety Standards<br />The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.<br />
  79. 79. The Joint Committee on Standards for Educational Evaluation (1990) <br />Utility Standards<br />Feasibility Standards<br />Propriety Standards<br />Accuracy Standards<br />The accuracy standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.<br />
  80. 80. Conclusions<br />
  81. 81. Conclusions<br />Process<br />Evaluations<br />Outcome<br />
  82. 82. Conclusions<br />Evaluations<br />Assessments<br />
  83. 83. Conclusions<br />Three philosophies/paradigms:<br /><ul><li>Use
  84. 84. Methods
  85. 85. Value</li></li></ul><li>Conclusions<br />Where would you sit on the Evaluation Tree?<br />What about others in your “system”?<br />
  86. 86. Take-home<br />Need more complex evaluation models that look at both processes and outcomes<br />Choices will depend on the intended use (i.e. the branch on the Evaluation Tree)<br />Both the evaluation models and the assessments instruments need standardization and rigor.<br />
  87. 87. Beast<br />Environment<br />Intake<br />Interaction<br />Fluffy<br />Reaction<br />Function<br />

×