Download the presentation

252
-1

Published on

Published in: Health & Medicine, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
252
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • AR Specialist Salary Range: 32k-35k
    Accountant Salary Range: 42k-45k
  • Standards are the “target”. To hit the target, it must be clear in concept and language; preferably from an authoritative source.
    The criteria for judging the performance should be as self-evident as possible so the LEARNER can apply them without much if any assistance from the instructor
    The time allotted for application, practice and feedback is the MOST important time of a SAM session, in particular. It should be ample, unhurried
    The SAM structure should build in and stimulate reflection on one’s practice, analysis of one’s errors – not only the what, but the how and the why
    Personal reflection is most important in a self-assessment, but at least on occasion, input from respected colleagues may broaden and deepen the personal reflection. (This is especially useful if the learner is having problems accurately comparing his/her own performance to models/standards – has a “blind spot”.)
    Let’s take a look at FEEDBACK as the make-or-break element of self assessment in the next slide.
  • In contrast, CME may be
    Method: at least some of the time, a more passive reception of instruction
    Mindset: intellectual curiosity
    Goal: information gain
    Catalyst: presentation of information
    Response: add to store of knowledge
  • Active learning opportunities – this is a big *creative* challenge!
    Sequenced offerings, building on the previous
    Methods enabling transfer to the practice setting
    Assignments requiring thoughtful engagement
    Mentoring in reflective attitudes such as Dewey’s open-mindedness, engagement, responsibility
    These are hard to do:
    Our didactic approaches aren’t bad, in fact we’ve gotten very good at presenting lots of info in well-organized ways in short amounts of time.
    But… let’s self-evaluate our own habitual approaches as we get into SAMs… is there a way to greater effectiveness?
    The literature offers a prime opportunity in terms of “authentic assessments” (next slide)
  • Examples of authentic assessment:
    Performances: Any technique that requires learners to generate their own responses rather than to select among responses provided for them. Authentic assessments are a subset of the above that “closely simulate or actually replicate challenges faced by professionals.” Oral exams are a type of performance assessments that are authentic to the extent they resemble problems radiologists will actually encounter in practice!
    Example: Physics’ Self-Directed Educational Projects – cite the DR Physics MCST QA example here, and refer to ABR website for more.
    Pose real dilemmas – not just to unambiguous and unproblematic questions, but to sometimes unanswerable (or unsure) questions. Some helps in doing “dilemma” type assessment: A. make the assessment interactive – ask learners to state their justification in response to targeted questions, confront them with counterexamples, and request comment and critique on other points of view. B. Make the assessment reiterative. Use recurring tasks to self-assess progress over time, e.g., in a series of SAMs on the same topic. C. Use assessment tasks that will evoke common misunderstandings to see if the learner can recognize and overcome a misconception.
    So… if we use authentic assessments, how do we evaluate them? (next slide)
  • Clear & very specific descriptive language – not focused on “correct/incorrect”, not praising or blaming, not interpreting or analyzing the error, not advising on how to improve! (Looking for an example here, maybe from nursing.)
    Again, standards should be authoritative (shared by learner & profession), appropriate to the task at hand, and rigorous (these are not novices but practicing professionals; SAMs are aimed toward making experts.)
    Immediacy is absolutely key to effective feedback, and therefore indispensable in SAMs. Why? Because the learner is *presently* engaged, and his/her conceptions are therefore fluid or in an “unfrozen” state – can change before “refreezing”.
    Letting learners see and analyze their own errors – more engaging, more transferable, more enduring.
    But what about those MCQs that are supposed to form the posttest of SAMs? (next slide)
  • Download the presentation

    1. 1. Item-Writing Workshop Gary J. Becker, M.D.1 Jennifer L. Bosma, PhD Executive Director AED for Administration American Board of Radiology 1 Member, Board of Directors, American Board of Medical Specialties American Roentgen Ray Society San Diego, CA May 2, 2010
    2. 2. A Few Very Basic Principles about Objective Tests • A primary purpose: communicate what isA primary purpose: communicate what is importantimportant • Powerful motivatorsPowerful motivators • Help fill instructional gaps by encouragingHelp fill instructional gaps by encouraging students to read broadly independentlystudents to read broadly independently • Because of powerful influence on students, weBecause of powerful influence on students, we should develop tests that further ourshould develop tests that further our educational goalseducational goals
    3. 3. About Multiple Choice Questions (MCQs) • ObjectiveObjective • Efficient; sample across domainEfficient; sample across domain • If well written, can test all cognitive levelsIf well written, can test all cognitive levels • Obviates several of shortcomings of T/F,Obviates several of shortcomings of T/F, and is better discriminatorand is better discriminator • The workhorseThe workhorse
    4. 4. • CMECME • SAMsSAMs • Qualifying (“written”) clinical examsQualifying (“written”) clinical exams • Qualifying (“written”) physics examsQualifying (“written”) physics exams • Many of the MOC exam itemsMany of the MOC exam items Multiple Choice Questions Standard MCQ is item format of choice for:
    5. 5. When oral Coumadin causes coagulopathy, what is theWhen oral Coumadin causes coagulopathy, what is the method that most rapidly corrects it?method that most rapidly corrects it? A.A. Transfusion of whole bloodTransfusion of whole blood B.B. Transfusion of fresh frozen plasmaTransfusion of fresh frozen plasma C.C. Intramuscular injection of vitamin KIntramuscular injection of vitamin K D.D. Intravenous injection of thrombinIntravenous injection of thrombin E.E. Intravenous injection of epsilon aminocaproic acidIntravenous injection of epsilon aminocaproic acid Typical MCQ measures knowledge, comprehension
    6. 6. • 6 levels of intellectual behavior important in learning6 levels of intellectual behavior important in learning** • FromFrom simple recallsimple recall toto most abstractmost abstract • >95% test questions: recall level, lowest>95% test questions: recall level, lowest • Highest levels remain difficult to achieve with MCQsHighest levels remain difficult to achieve with MCQs Benjamin Bloom’s Taxonomy * Educational psychology theory, 1956
    7. 7. Benjamin Bloom’s Taxonomy: 6 Levels of Intellectual Behavior Important in Learning 1. Knowledge 2. Comprehension 3. Application 4. Analysis 5. Synthesis 6. Evaluation “Written” “Written” Oral Oral Oral Oral Define, Label, Order, List, Name Classify, Describe, Identify, Recognize Interpret, Solve, Choose, Demonstrate Analyze, Calculate, Distinguish, Compare Assemble, Compose, Design, Manage, Create Argue, Assess, Defend, Estimate, Rate, Support
    8. 8. Application/Analysis
    9. 9. Synthesis/Evaluation/Management
    10. 10. R-Type Extended Matching Items
    11. 11. Anatomy of an R-Type Extended Matching Item • Theme • Option list • Lead-in statement • Stems (3-4)
    12. 12. THEME: Arteriopathies OPTION LIST: A. Acute embolus B. Adventitial cycstic disease C. Arteriovenous malformation D. Atherosclerosis E. Behcet Disease F. Blue Toe Syndrome G. Entrapment H. Ergotism I. Fetal Alcohol Syndrome J. Fibromuscular dysplasia, intimal type K. Fibromuscular dysplasia, medial type L. Giant cell arteritis M. Hypothenar hammer syndrome N. Kawasaki disease O. Leriche syndrome P. Leukocytoclastic vasculitis Q. Marfan syndrome R. Mycotic aneurysm S. Normal T. Polyarteritis nodosa U. Pseudoaneurysm V. Raynaud syndrome W. Subclavian steal syndrome X. Swyer-James-McLeod syndrome Y. Takayasu arteritis Z.Thromboangiitis obliterans LEAD-IN STATEMENT: Each of the following vignettes provides a brief clinical case description. For each numbered vignette, provide the correct lettered diagnosis. Each diagnosis may be used once, more than once, or not at all. 1. 67-year-old woman with a 2-year history of progressive arm fatigue when carrying groceries KEY: L 2. 32-year-old man with hepatitis B surface antigen seropositivity presents with acute right flank pain KEY: T 3. 43-year-old woman transported by emergency medical technicians following a motor vehicle accident KEY: U 4. 71-year-old man with severe dyspnea admitted to ICU; BP is 220/120 despite 4 anti-hypertensive medications; diffuse rales; Cockcroft-Galt estimated GFR is 39mL/min KEY: D
    13. 13. THEME: Arteriopathies OPTION LIST: A.Acute embolus B.Adventitial cycstic disease C.Arteriovenous malformation D.Atherosclerosis E.Behcet Disease F.Blue Toe Syndrome G.Entrapment H.Ergotism I.Fetal Alcohol Syndrome J.Fibromuscular dysplasia, intimal type K.Fibromuscular dysplasia, medial type L.Giant cell arteritis M.Hypothenar hammer syndrome N.Kawasaki disease O.Leriche syndrome P.Leukocytoclastic vasculitis Q.Marfan syndrome with aortic dissection R.Mycotic aneurysm S.Normal T.Polyarteritis nodosa U.Pseudoaneurysm V.Raynaud syndrome W.Subclavian steal syndrome X.Swyer-James-McLeod syndrome Y.Takayasu arteritis Z.Thromboangiitis obliterans LEAD-IN STATEMENT: Each of the following vignettes provides a brief clinical case description. For each numbered vignette, provide the correct lettered diagnosis. Each diagnosis may be used once, more than once, or not at all. 1. 25-year-old male runner with 4-month history of right calf claudication at ½ mile . Past history and review of systems are negative. KEY: G 2. 56-year-old man with recent aortic valve replacement has malaise, night sweats, and back pain; Temp: 38.3o C. WBCs: 17,400 with 83% bands KEY: R 3. 37-year-old woman non-smoker on daily medication for headache has 3-week history of bilateral lower limb claudication. Femoral pulses are faint; popliteal, posterior tibial, and dorsalis pedis pulses are absent. KEY: H 4. 42-year old tall, thin male with acute severe back pain KEY: X
    14. 14. Example: Extended Matching Solitary Liver Mass For each patient whose clinical and imaging information is shown, select the most appropriate diagnosis from the list below. Each option may be used once, more than once, or not at all. a. Focal nodular hyperplasia h. Biliary cystadenoma b. Liver cell adenoma i. Lymphoma c. Cavernous hemangioma j. Solitary metastasis d. Inflammatory pseudotumor k. Intrahepatic cholangiocarcinoma e. Pyogenic abscess l. Hepatocellular carcinoma f. Fungal abscess m. Fibrolamellar carcinoma g. Nodular focal fat deposition
    15. 15. 1: 35 year old woman who underwent sonographic evaluation for mild abdominal discomfort. She was referred for MR imaging to characterize a solitary liver mass seen on that ultrasound. Images are obtained 30 seconds (A), 70 seconds (B) and 1 hour (C) after administration of gadopendetate intravenously. A B C a. Focal nodular hyperplasia b. Liver cell adenoma c. Cavernous hemangioma d. Inflammatory pseudotumor e. Pyogenic abscess f. Fungal abscess g. Nodular focal fat deposition h. Biliary cystadenoma i. Lymphoma j. Solitary metastasis k. Intrahepatic cholangiocarcinoma l. Hepatocellular carcinoma m. Fibrolamellar carcinoma
    16. 16. A B C D 2. 60 year old man with abdominal pain. Imaging performed elsewhere showed a liver mass, and he is referred for CT imaging to characterize it. Images are obtained before (A), 20 seconds after (B), 50 seconds after (C), and 5 minutes after (D) intravenous administration of iodinated contrast material. a. Focal nodular hyperplasia b. Liver cell adenoma c. Cavernous hemangioma d. Inflammatory pseudotumor e. Pyogenic abscess f. Fungal abscess g. Nodular focal fat deposition h. Biliary cystadenoma i. Lymphoma j. Solitary metastasis k. Intrahepatic cholangiocarcinoma l. Hepatocellular carcinoma m. Fibrolamellar carcinoma
    17. 17. Strengths of an R-Type Extended Matching Item • Tests beyond knowledge, comprehension • More analysis, management, top of pyramid • Better discriminator than T/F or A-type • Comparable results with one-third fewer items when compared with A-type • Theoretically, can test more of domain in same time
    18. 18. Script Concordance Testing
    19. 19. Script theory of medical expertise: 1990 (Schmidt et al) “Experienced practitioners possess elaborate networks of knowledge fitted to the regular tasks they perform called scripts.” Scripts allow the clinician to determine the most likely diagnosis, strategies of investigation, and treatment options. Script Concordance Test, an assessment tool: 2000 (Charlin et al)
    20. 20. Script Concordance Testing
    21. 21. Script Concordance
    22. 22. Script Concordance Testing Probes the ability of the individual to interpret and utilize data in specific clinical settings Serves as an indicator of clinical reasoning Script Concordance Testing discriminates between highly experienced and lesser experienced individuals. Will require pre-test familiarity
    23. 23.  The main purpose of the ABMS MemberThe main purpose of the ABMS Member Boards, including the ABR, isBoards, including the ABR, is to improveto improve healthcare safety and quality in the U.S.healthcare safety and quality in the U.S. by establishing standards for:by establishing standards for: • Initial certificationInitial certification • Maintenance of CertificationMaintenance of Certification
    24. 24. General Competencies • Medical KnowledgeMedical Knowledge • Patient CarePatient Care • Interpersonal and Communication SkillsInterpersonal and Communication Skills • ProfessionalismProfessionalism • Practice-based Learning and ImprovementPractice-based Learning and Improvement • Systems-based PracticeSystems-based Practice  ABMS, September 1999ABMS, September 1999
    25. 25. • Professional standing (I)Professional standing (I) • Lifelong learning and self-assessment (II)Lifelong learning and self-assessment (II) -CME-CME -SAMs-SAMs • Cognitive expertise (III)Cognitive expertise (III) -Examinations-Examinations • Practice performance (IV)Practice performance (IV) -Many include self-assessment, blend in Part 2-Many include self-assessment, blend in Part 2  ABMS, March 2000ABMS, March 2000 Maintenance of Certification Components & Item Writing Opportunities
    26. 26. ABR Volunteer Opportunities http://www.theabr.org
    27. 27. • Qualifying (“written”) examsQualifying (“written”) exams • Oral certifying examsOral certifying exams • MOC examsMOC exams • SAMsSAMs • Computer-based Core ExamComputer-based Core Exam • Computer-based Certifying ExamComputer-based Certifying Exam ABR Volunteer Opportunities in Diagnostic Radiology For those skilled in item-writing:
    28. 28. Activity Time Commitment Travel Method Interaction with Colleagues Qualifying (“written”) exams: Write test questions 10-25 hrs / year; 3-year term No Web-based submission Feedback by 1-2 people Oral certifying exams: Serve as oral examiner 5 days 1st week of June; state preference for re- invitation Yes 1st day orientation; 2-4 full days of examining; 13-16 candidates per day 15-20 colleagues, same sub- specialty, + 175-200 DRs total; social opportunities MOC exams: Serve as subspecialty committee member for MOC exam committee 1 in-person meeting / year; 1-2 electronic meetings / year; 3-year term Yes Committee meeting; follow-up electronically Committee of colleagues in same specialty meets in- person & electronically
    29. 29. Activity Time Commitment Travel Method Interaction with Colleagues SAMs: Serve as reviewer of SAM applications 1-2 hrs / review; 1-6 reviews / yr; 3-year term No Email; FTP Occasional with other reviewers Future core exam: Serve as committee member to develop exam 1 in-person mtg & 2-4 electronic mtgs per year; 3-yr term Yes 1x /yr Tucson Committee meeting; follow-up electronically Committee of colleagues in same specialty meets in- person & electronically over 3 years Future certifying exam: Serve as subspecialty committee member for MOC exam 1 in-person mtg & 2-4 electronic mtgs per year; 3-yr term Yes 1x /yr Tucson Committee meeting; follow-up electronically Committee of colleagues in same specialty meets in-
    30. 30. What Are the Essential Ingredients of Self-Assessment?  Clear standardsClear standards  ““Self-evident” criteriaSelf-evident” criteria  Time for application, practice, feedbackTime for application, practice, feedback  Reflection on practiceReflection on practice  personalpersonal  peer stimulatedpeer stimulated
    31. 31. What’s “self” about SAMs?  Robust self-assessment employs a unique…Robust self-assessment employs a unique…  Method: Reflective examination of selfMethod: Reflective examination of self  Mindset: Attitude change fromMindset: Attitude change from concealment to opennessconcealment to openness  Goal: Self-improvementGoal: Self-improvement  Catalyst: Application, practice, andCatalyst: Application, practice, and feedbackfeedback  Response: Changes in own practiceResponse: Changes in own practice
    32. 32. Findings from studies of effective adult instruction  What is more effective thanWhat is more effective than didactic presentations?didactic presentations?  ActiveActive  LongitudinalLongitudinal  TransferableTransferable  Engaging assignmentsEngaging assignments  MentoringMentoring
    33. 33. Authentic Assessments • Tasks that closely simulate or actuallyTasks that closely simulate or actually replicate challenges faced in practicereplicate challenges faced in practice • Use of genuine examples of the learner’sUse of genuine examples of the learner’s workwork • Use of models with which learner comparesUse of models with which learner compares • Pose real dilemmas; draw on commonPose real dilemmas; draw on common misconceptionsmisconceptions  Interactive respondingInteractive responding  Reiterative respondingReiterative responding
    34. 34. Feedback  Instructor:Instructor: Describes what the learner did/did not doDescribes what the learner did/did not do References standards of quality / competenceReferences standards of quality / competence Delivers immediatelyDelivers immediately  Learner:Learner: Self-perceives both correct answers and errorsSelf-perceives both correct answers and errors Analyzes causesAnalyzes causes
    35. 35. SAM Data
    36. 36. SAMs by Category April 2010 Category Number of SAMs Qualified Number of SAMs Currently Available ARRS SAMs Qualified ARRS SAMs currently available Musculoskeletal 47 24 15 12 Cardiac 21 6 5 4 Thoracic 36 12 8 6 Gastrointestinal 55 17 11 6 Genitourinary 34 14 10 8 Neuroradiology 89 33 9 7 Vascular and Interventional 64 9 6 4 Nuclear Medicine 62 14 2 2 Ultrasound 22 5 3 2 Pediatric 30 3 3 1 Breast 38 11 2 1 Other 25 2 0 0 Non-Interpretive Skills 71 19 7 5 Total 594 169 81 58
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×