Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Competency assessment boot camp basics with tp


Published on

This is a presentation used to teach the basics of competency development and assessment. It demonstrates the use of the audience response system, Turning Point.

Published in: Education, Business, Technology
  • Be the first to comment

  • Be the first to like this

Competency assessment boot camp basics with tp

  1. 1. What is your biggest challenge in training & competency assessment?
  2. 2. Competency Assessment Boot Camp Basics for Labs Caer Vitek Education Coordinator for Graduate Medical Education Department of Laboratory Medicine and Pathology May 5, 2009
  3. 3. Disclosures <ul><li>Caer Vitek has nothing to disclose. </li></ul>
  4. 4. I have previously downloaded or reviewed today’s session. <ul><li>True </li></ul><ul><li>False </li></ul>
  5. 5. How long have you worked in a laboratory? <ul><li>Under two years </li></ul><ul><li>2-4 years </li></ul><ul><li>5-10 years </li></ul><ul><li>More than 10 years </li></ul>
  6. 6. My involvement in training laboratory staff <ul><li>Totally responsible for developing and providing training </li></ul><ul><li>Key developer of training </li></ul><ul><li>Work as part of a training team </li></ul><ul><li>Do some training as part of my job </li></ul><ul><li>I am not involved in training </li></ul>
  7. 7. Pre-test
  8. 8. Analyzing the results of a proficiency test is an example of a(n) ___________ objective. <ul><li>Cognitive </li></ul><ul><li>Formative </li></ul><ul><li>Psychomotor </li></ul><ul><li>Affective </li></ul><ul><li>Confirmative </li></ul>
  9. 9. Multiple choice tests are <ul><li>the most effective form of assessment. </li></ul><ul><li>only to be used to assess objective factual knowledge. </li></ul>
  10. 10. Learning Objectives (At the completion of today’s session, the audience members will be able to:) <ul><li>Define key characteristics of effective learning objectives </li></ul><ul><li>Revise learning objectives to enhance assessment </li></ul><ul><li>Select assessment tools based on learning objectives and level of skill/knowledge </li></ul>
  11. 11. Why are objectives important? (WIIFM) <ul><li>Clarify expectations for </li></ul><ul><ul><li>learners </li></ul></ul><ul><ul><li>trainers </li></ul></ul><ul><li>Provide a framework to evaluate the learning </li></ul>
  12. 12. How do you create training for labs? <ul><li>Identify need </li></ul><ul><li>Develop plan </li></ul><ul><li>Gather materials </li></ul><ul><li>Assessment </li></ul><ul><li>What are new task(s) or process(es)? </li></ul><ul><li>How do my learner(s) learn best? </li></ul><ul><li>What will I need to develop in order to master this new task? </li></ul><ul><li>How will I know if they’ve learned it? </li></ul>
  13. 13. Fundamental training for labs? <ul><li>Identify need </li></ul><ul><li>Develop plan </li></ul><ul><li>Gather materials </li></ul><ul><li>Assessment </li></ul><ul><li>Evaluate </li></ul><ul><li>Example: </li></ul><ul><li>New process </li></ul><ul><li>Map out process, identify critical steps </li></ul><ul><li>Bones, treats, leash </li></ul><ul><li>Competency </li></ul><ul><li>Successes & revisions </li></ul>
  14. 14. Basic #1: Be clear <ul><li>Be clear: </li></ul><ul><ul><li>Do I know my learner ? </li></ul></ul><ul><ul><li>Are the goals clear? </li></ul></ul><ul><ul><li>Are the objectives clear? </li></ul></ul><ul><ul><li>Is the plan clear? </li></ul></ul>
  15. 15. Goal, objective or strategy? <ul><li>GOAL= </li></ul><ul><li>broad, generalized statement about what is to be learned </li></ul>OBJECTIVE= Specifies what the learner is to accomplish
  16. 16. What is the strategy? <ul><li>GOAL= </li></ul><ul><li>broad, generalized statement about what is to be learned </li></ul>OBJECTIVE= Specifies what the learner is to accomplish STRATEGY= Specifies how the learner will learn
  17. 17. View DVDs of presentations on . . . . <ul><li>Goal </li></ul><ul><li>Objective </li></ul><ul><li>Strategy </li></ul><ul><li>Assessment </li></ul>
  18. 18. Basic #2: Know your goal <ul><li>Then consider strategy & assessment </li></ul>
  19. 19. Authentic assessment What do you use?
  20. 20. How will I assess my learners? <ul><li>Common mistake: </li></ul><ul><li>Choose the assessment tool, then force-fit the objectives </li></ul><ul><li>Better: Work backwards </li></ul><ul><li>OR </li></ul><ul><li>concurrently </li></ul>
  21. 21. Basic #3: Choose from a toolbox <ul><li>Assessment Toolbox </li></ul><ul><li>Observe skills & behaviors </li></ul><ul><li>Rating scale </li></ul><ul><li>Checklist </li></ul><ul><li>Direct testing </li></ul><ul><li>Case studies </li></ul><ul><li>Unknowns </li></ul><ul><li>Others: Rubric, Portfolio </li></ul>
  22. 22. Know when they know (assessment of competency) <ul><li>Match the learning/behavior with the assessment </li></ul><ul><li>Target behavior performance reflects how well the trainee can carry out a task(s) </li></ul><ul><li>Role of standards </li></ul><ul><li>Importance of self-assessment </li></ul>
  23. 23. Bloom’s Taxonomy                                                                                       NEW Version
  24. 24. Writing learning objectives <ul><li>A </li></ul><ul><li>B </li></ul><ul><li>C </li></ul><ul><li>D </li></ul>
  25. 25. Writing learning objectives <ul><li>A udience= learner </li></ul><ul><li>B </li></ul><ul><li>C </li></ul><ul><li>D </li></ul>
  26. 26. A udience <ul><li>Who is your learner? </li></ul>
  27. 27. Writing learning objectives <ul><li>A udience= learner </li></ul><ul><li>B ehavior= able to know or to do </li></ul><ul><li>C </li></ul><ul><li>D </li></ul>
  28. 28. Writing learning objectives <ul><li>A udience= learner </li></ul><ul><li>B ehavior= able to know or to do </li></ul><ul><li>C ondition= circumstances under which objectives must be completed </li></ul><ul><li>D egree= standard of acceptable performance </li></ul>
  29. 29. Writing learning objectives <ul><li>A udience= learner </li></ul><ul><li>B ehavior= able to know or to do </li></ul><ul><li>C ondition= circumstances under which objectives must be completed </li></ul><ul><li>D egree= standard of acceptable performance </li></ul>
  30. 30. Types of objectives Cognitive (Knowledge) Psychomotor (Skill) Affective (Emotion)
  31. 31. Bloom’s Taxonomy                                                                                       NEW Version
  32. 32. Bloom’s Taxonomy for Cognitive Objectives <ul><li>Remember: Can the learner recall or remember the information? </li></ul><ul><li>define, duplicate, list, memorize, recall, repeat, reproduce state </li></ul><ul><li>Understand: Can the learner explain ideas or concepts? </li></ul><ul><li>classify, describe, discuss, explain, identify, locate, recognize * , report, select, translate, paraphrase </li></ul><ul><li>Apply: Can the learner use the information in a new way? </li></ul><ul><li>choose, demonstrate, dramatize, employ, illustrate, interpret, operate, schedule, sketch, solve, use, write </li></ul>
  33. 33. Bloom’s Taxonomy for Cognitive Objectives (cont.) <ul><li>Analyse : Can the learner distinguish between the different parts? </li></ul><ul><li>appraise, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test </li></ul><ul><li>Evaluate: Can the learner justify a stand or decision? </li></ul><ul><li>appraise, argue, defend, judge, select, support, value, evaluate </li></ul><ul><li>Can the learner create new product or point of view? </li></ul><ul><li>assemble, construct, create, design, develop, formulate, write </li></ul>Create:
  34. 34. B ehavior
  35. 35. And now, a word about verbs: <ul><li>Understand </li></ul><ul><li>Appreciate </li></ul><ul><li>Learn </li></ul><ul><li>Improve </li></ul><ul><li>Increase </li></ul><ul><li>Motivate </li></ul><ul><li>Know </li></ul><ul><li>Really know </li></ul><ul><li>Approach </li></ul><ul><li>Grasp the significance of </li></ul><ul><li>Be familiar with </li></ul><ul><li>Expand horizons </li></ul><ul><li>Become </li></ul><ul><li>Grow </li></ul><ul><li>Think critically </li></ul><ul><li>Recognize * </li></ul>
  36. 36. And now, a word about verbs: <ul><li>Understand </li></ul><ul><li>Appreciate </li></ul><ul><li>Learn </li></ul><ul><li>Improve </li></ul><ul><li>Increase </li></ul><ul><li>Know </li></ul><ul><li>Really know </li></ul><ul><li>Use judiciously for goals! </li></ul><ul><li>Approach </li></ul><ul><li>Grasp the significance of </li></ul><ul><li>Expand horizons </li></ul><ul><li>Become </li></ul><ul><li>Grow </li></ul><ul><li>Think critically </li></ul><ul><li>Recognize * </li></ul>Do NOT use for objectives
  37. 37. C onditions <ul><li>Learning </li></ul><ul><ul><li>Environment </li></ul></ul><ul><ul><li>Materials </li></ul></ul><ul><ul><li>Standard Operating Procedures (SOP) </li></ul></ul>
  38. 38. D egree=Standards <ul><li>How </li></ul><ul><li>much? </li></ul><ul><li>many? </li></ul><ul><li>often? </li></ul><ul><li>well? </li></ul><ul><li>will I know when there is success? </li></ul><ul><li>Examples: </li></ul><ul><li>Until the syringe is empty. </li></ul><ul><li>All 96 wells </li></ul><ul><li>At the start of every run. </li></ul><ul><li>No more than 1 error. </li></ul><ul><li>When the tubes are in position and the red light is on. </li></ul>
  39. 39. Templates <ul><li>After the training period the learner will be able to perform ________(task)_________. The task must be performed under the following conditions: ____(condition)______, _____(condition)_____, and ____(condition)______. The following standards must be met: ____(standard)_____, _____(standard)____, and ____(standard)_____. </li></ul><ul><li>Given a ____(condition)_____, ____(condition)____, and ____(condition)_____, perform _______________(task)_________________. The task must be performed as ____________(standards)_____________. </li></ul>
  40. 40. Knowledge-based objective <ul><li>Compare and contrast the different methods used for glucose, electrolyte, and coagulation (ACT, PT, PTT) testing in the critically ill patient, including differences between whole blood and plasma-based testing </li></ul>
  41. 41. Rewriting Learning Objectives to be assessable <ul><li>Before: The learner will appreciate the principles of quality assurance. </li></ul><ul><li>After: </li></ul>
  42. 42. At the completion of this training, the learner will be able to: <ul><li>Demonstrate/describe a systematic approach to reading PT evaluation reports. </li></ul><ul><li>Identify two resources available to assist with report interpretation. </li></ul><ul><li>Demonstrate/describe a systematic method for investigating PT failures. </li></ul><ul><li>Identify three types of unacceptable PT results that warrant investigation. </li></ul><ul><li>Differentiate appropriate from inappropriate methods of investigating unacceptable PT results. </li></ul><ul><li>Accurately complete a Proficiency Test Failure Reporting form. </li></ul><ul><li>What recommendations would you offer to enhance these? </li></ul>
  43. 43. The learner will list the safety features on the FUTURO 5000. <ul><li>Remember </li></ul><ul><li>Understand </li></ul><ul><li>Apply </li></ul><ul><li>Analyze </li></ul><ul><li>Evaluate </li></ul><ul><li>Create </li></ul>
  44. 44. The learner will compare the safety features on the FUTURO 5000 to the FUTURO 2000. <ul><li>Remember </li></ul><ul><li>Understand </li></ul><ul><li>Apply </li></ul><ul><li>Analyse </li></ul><ul><li>Evaluate </li></ul><ul><li>Create </li></ul>
  45. 45. Basic #4: Know when they know <ul><li>How well must they master the information or technique? </li></ul><ul><li>often? </li></ul><ul><li>well? </li></ul><ul><li>many? </li></ul><ul><li>much? </li></ul><ul><li>will we know it is OK? </li></ul><ul><li>(combination) </li></ul>
  46. 46. Know when they know (assessment of competency) <ul><li>Knowledge & Comprehension: </li></ul><ul><ul><li>Quizzes and exams </li></ul></ul><ul><ul><ul><li>multiple choice, matching, fill-in-the-blank, </li></ul></ul></ul><ul><ul><ul><li>direct observation </li></ul></ul></ul><ul><li>Application, Analysis & Evaluation: </li></ul><ul><ul><li>Real-life, authentic problems </li></ul></ul><ul><ul><ul><li>proficiency testing, unknowns </li></ul></ul></ul><ul><li>Creation: </li></ul><ul><ul><li>Portfolio, presentations </li></ul></ul>
  47. 47. Keys to success <ul><li>Build into training: </li></ul><ul><ul><li>Steps for the learner to self-assess </li></ul></ul>
  48. 48. Assessing lower-level competencies Performance-content matrix Attitude Interpersonal Procedure Principles/Rules Concept Fact APPLICATION RECALL CONTENT
  49. 49. Assessing higher-level competencies <ul><li>What does an authentic problem look like in your lab? </li></ul><ul><li>What does a learner need to have/do to solve it? </li></ul>
  50. 51. Given a pipette____________ 100 ul accurately. <ul><li>Measure and aliquot </li></ul><ul><li>Dispense </li></ul><ul><li>Dispell </li></ul><ul><li>Mix a suspension </li></ul>
  51. 52. The learner will list and describe the safety features on the FUTURO 5000. <ul><li>Remember </li></ul><ul><li>Understand </li></ul><ul><li>Apply </li></ul><ul><li>Analyse </li></ul><ul><li>Evaluate </li></ul><ul><li>Create </li></ul>TRICK QUESTION-UNFAIR! Beware of compounding-don’t do it
  52. 53. What are the effects of compounding? <ul><li>Compounding: using 2 or more behavioral and/or knowledge verbs in one objective </li></ul><ul><ul><li>“list” and “compare” </li></ul></ul><ul><ul><li>“compare” and “contrast” </li></ul></ul><ul><li>Makes assessment difficult, challenging or impossible! </li></ul>
  53. 54. Basics <ul><li>Be clear </li></ul><ul><li>Know your goal(s) </li></ul><ul><li>Use a good toolbox </li></ul><ul><li>Know when you’ve achieved the goal (and when you haven’t) </li></ul>
  54. 55. RESOURCES <ul><li>Revised Bloom’s Taxonomy: </li></ul>
  55. 56. Contact information: <ul><li>Caer Vitek: [email_address] </li></ul>
  56. 57. Performance-based objective <ul><li>Analyze, contrast and evaluate various service options that meet clinical needs for rapid turn-around time and analytical precision, </li></ul><ul><li>Review advantages and disadvantages of each option, present your conclusion. </li></ul>
  57. 58. Characteristics of learning objectives <ul><li>S pecific </li></ul><ul><li>M easurable </li></ul><ul><li>A cceptable (to you and your learner) </li></ul><ul><li>R ealistic </li></ul><ul><li>T ime-bound </li></ul>
  58. 59. Rules for Writing Single Response Questions <ul><li>1) Focus on an important topic, usually a common or </li></ul><ul><li>critical clinical problem; avoid esoterica and “zebras” </li></ul><ul><li>2) Assess application of knowledge, not recall </li></ul><ul><li>3) Pose clinical decision-making tasks that are within the </li></ul><ul><li>education/experience of examinees </li></ul><ul><li>4) Pose a clear question in the lead-in </li></ul><ul><ul><li>can you answer it without looking at the options? </li></ul></ul><ul><li>5) Use homogeneous distractors </li></ul><ul><li>6) Avoid technical flaws </li></ul>
  59. 60. Grammatical Cues <ul><li>The option(s) does not flow from the stem </li></ul><ul><ul><li>The minor differences among organisms of the </li></ul></ul><ul><ul><li>same kind are known as </li></ul></ul><ul><ul><li>A. Heredity </li></ul></ul><ul><ul><li>B. Variations </li></ul></ul><ul><ul><li>C. Adaptation </li></ul></ul><ul><ul><li>D. Natural selection </li></ul></ul>
  60. 61. Logical Cues <ul><li>A subset of options are collectively exhaustive. </li></ul><ul><li>Crime is </li></ul><ul><li>A. Equally distributed among the social classes </li></ul><ul><li>B. Overrepresented among the poor </li></ul><ul><li>C. Overrepresented among the middle class and rich </li></ul><ul><li>D. Primarily an indication of psychosexual maladjustment </li></ul><ul><li>E. Reaching a plateau of tolerability for the nation </li></ul>
  61. 62. Absolute Terms <ul><li>Terms such as ‘always’ or ‘never’ are used in options. </li></ul><ul><li>In patients with advanced dementia, Alzheimer’s type, the memory defect </li></ul><ul><li>A. Can be treated adequately with lecithin </li></ul><ul><li>B. Could be a sequela of early parkinsonism </li></ul><ul><li>C. Is never seen in patients with neurofibrillary tangles </li></ul><ul><li>C. Is never severe </li></ul><ul><li>D. Possibly involves the cholinergic system </li></ul>
  62. 63. Long Correct Answer <ul><li>The correct answer is longer, more specific, or more </li></ul><ul><li>complete than the other options. </li></ul><ul><li>Secondary gain is </li></ul><ul><li>A. Synonymous with malingering </li></ul><ul><li>B. A frequent problem in obsessive-compulsive disorder </li></ul><ul><li>C. A complication of a variety of illnesses and tends to prolong many of them </li></ul><ul><li>D. Never seen in organic brain damage </li></ul>
  63. 64. Word Repeats <ul><li>A word or phrase is included in the stem and correct </li></ul><ul><li>answer. </li></ul><ul><li>A 58-year-old man with a history of heavy alcohol </li></ul><ul><li>use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called </li></ul><ul><ul><li>A. Depersonalization </li></ul></ul><ul><ul><li>B. Derailment </li></ul></ul><ul><ul><li>C. Derealization* </li></ul></ul><ul><ul><li>D. Focal memory defect </li></ul></ul>
  64. 65. Options are long, complicated or doubled <ul><li>Systematic geography differs from regional geography in that </li></ul><ul><li>A. Systematic geography deals, in the main, with physical geography, while regional geography concerns itself essentially with the field of human geography </li></ul><ul><li>B. Systematic geography studies a region systematically while regional geography is concerned only with descriptive account of a region </li></ul><ul><li>C. Systematic geography studies a single phenomenon in its distribution over the earth in order to supply generalizations for regional geography, which studies the arrangement of phenomena in one given area* </li></ul>
  65. 66. Numeric data are not stated consistently <ul><li>Following a second episode of infection, what is the the </li></ul><ul><li>likelihood that a woman is infertile? </li></ul><ul><li>A. Less than 20% </li></ul><ul><li>B. 20% to 30% </li></ul><ul><li>C. Greater than 50% </li></ul><ul><li>D. 90% </li></ul><ul><li>E. 75% </li></ul>
  66. 67. Options in an nonlogical order <ul><li>The population of Denmark is </li></ul><ul><li>A. 2 million </li></ul><ul><li>B. 15 million </li></ul><ul><li>C. 4 million </li></ul><ul><li>D. 7 million </li></ul>
  67. 68. “ None of the Above” is used as an option <ul><li>Which city is closest to New York City? </li></ul><ul><li>A. Boston </li></ul><ul><li>B. Chicago </li></ul><ul><li>C. Dallas </li></ul><ul><li>D. Los Angeles </li></ul><ul><li>E. None of the above </li></ul>