Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Download the presentation

375 views

Published on

Published in: Health & Medicine, Technology
  • Be the first to comment

Download the presentation

  1. 1. Item-Writing Workshop Gary J. Becker, M.D.1 Jennifer L. Bosma, PhD Executive Director AED for Administration American Board of Radiology 1 Member, Board of Directors, American Board of Medical Specialties American Roentgen Ray Society San Diego, CA May 2, 2010
  2. 2. A Few Very Basic Principles about Objective Tests • A primary purpose: communicate what isA primary purpose: communicate what is importantimportant • Powerful motivatorsPowerful motivators • Help fill instructional gaps by encouragingHelp fill instructional gaps by encouraging students to read broadly independentlystudents to read broadly independently • Because of powerful influence on students, weBecause of powerful influence on students, we should develop tests that further ourshould develop tests that further our educational goalseducational goals
  3. 3. About Multiple Choice Questions (MCQs) • ObjectiveObjective • Efficient; sample across domainEfficient; sample across domain • If well written, can test all cognitive levelsIf well written, can test all cognitive levels • Obviates several of shortcomings of T/F,Obviates several of shortcomings of T/F, and is better discriminatorand is better discriminator • The workhorseThe workhorse
  4. 4. • CMECME • SAMsSAMs • Qualifying (“written”) clinical examsQualifying (“written”) clinical exams • Qualifying (“written”) physics examsQualifying (“written”) physics exams • Many of the MOC exam itemsMany of the MOC exam items Multiple Choice Questions Standard MCQ is item format of choice for:
  5. 5. When oral Coumadin causes coagulopathy, what is theWhen oral Coumadin causes coagulopathy, what is the method that most rapidly corrects it?method that most rapidly corrects it? A.A. Transfusion of whole bloodTransfusion of whole blood B.B. Transfusion of fresh frozen plasmaTransfusion of fresh frozen plasma C.C. Intramuscular injection of vitamin KIntramuscular injection of vitamin K D.D. Intravenous injection of thrombinIntravenous injection of thrombin E.E. Intravenous injection of epsilon aminocaproic acidIntravenous injection of epsilon aminocaproic acid Typical MCQ measures knowledge, comprehension
  6. 6. • 6 levels of intellectual behavior important in learning6 levels of intellectual behavior important in learning** • FromFrom simple recallsimple recall toto most abstractmost abstract • >95% test questions: recall level, lowest>95% test questions: recall level, lowest • Highest levels remain difficult to achieve with MCQsHighest levels remain difficult to achieve with MCQs Benjamin Bloom’s Taxonomy * Educational psychology theory, 1956
  7. 7. Benjamin Bloom’s Taxonomy: 6 Levels of Intellectual Behavior Important in Learning 1. Knowledge 2. Comprehension 3. Application 4. Analysis 5. Synthesis 6. Evaluation “Written” “Written” Oral Oral Oral Oral Define, Label, Order, List, Name Classify, Describe, Identify, Recognize Interpret, Solve, Choose, Demonstrate Analyze, Calculate, Distinguish, Compare Assemble, Compose, Design, Manage, Create Argue, Assess, Defend, Estimate, Rate, Support
  8. 8. Application/Analysis
  9. 9. Synthesis/Evaluation/Management
  10. 10. R-Type Extended Matching Items
  11. 11. Anatomy of an R-Type Extended Matching Item • Theme • Option list • Lead-in statement • Stems (3-4)
  12. 12. THEME: Arteriopathies OPTION LIST: A. Acute embolus B. Adventitial cycstic disease C. Arteriovenous malformation D. Atherosclerosis E. Behcet Disease F. Blue Toe Syndrome G. Entrapment H. Ergotism I. Fetal Alcohol Syndrome J. Fibromuscular dysplasia, intimal type K. Fibromuscular dysplasia, medial type L. Giant cell arteritis M. Hypothenar hammer syndrome N. Kawasaki disease O. Leriche syndrome P. Leukocytoclastic vasculitis Q. Marfan syndrome R. Mycotic aneurysm S. Normal T. Polyarteritis nodosa U. Pseudoaneurysm V. Raynaud syndrome W. Subclavian steal syndrome X. Swyer-James-McLeod syndrome Y. Takayasu arteritis Z.Thromboangiitis obliterans LEAD-IN STATEMENT: Each of the following vignettes provides a brief clinical case description. For each numbered vignette, provide the correct lettered diagnosis. Each diagnosis may be used once, more than once, or not at all. 1. 67-year-old woman with a 2-year history of progressive arm fatigue when carrying groceries KEY: L 2. 32-year-old man with hepatitis B surface antigen seropositivity presents with acute right flank pain KEY: T 3. 43-year-old woman transported by emergency medical technicians following a motor vehicle accident KEY: U 4. 71-year-old man with severe dyspnea admitted to ICU; BP is 220/120 despite 4 anti-hypertensive medications; diffuse rales; Cockcroft-Galt estimated GFR is 39mL/min KEY: D
  13. 13. THEME: Arteriopathies OPTION LIST: A.Acute embolus B.Adventitial cycstic disease C.Arteriovenous malformation D.Atherosclerosis E.Behcet Disease F.Blue Toe Syndrome G.Entrapment H.Ergotism I.Fetal Alcohol Syndrome J.Fibromuscular dysplasia, intimal type K.Fibromuscular dysplasia, medial type L.Giant cell arteritis M.Hypothenar hammer syndrome N.Kawasaki disease O.Leriche syndrome P.Leukocytoclastic vasculitis Q.Marfan syndrome with aortic dissection R.Mycotic aneurysm S.Normal T.Polyarteritis nodosa U.Pseudoaneurysm V.Raynaud syndrome W.Subclavian steal syndrome X.Swyer-James-McLeod syndrome Y.Takayasu arteritis Z.Thromboangiitis obliterans LEAD-IN STATEMENT: Each of the following vignettes provides a brief clinical case description. For each numbered vignette, provide the correct lettered diagnosis. Each diagnosis may be used once, more than once, or not at all. 1. 25-year-old male runner with 4-month history of right calf claudication at ½ mile . Past history and review of systems are negative. KEY: G 2. 56-year-old man with recent aortic valve replacement has malaise, night sweats, and back pain; Temp: 38.3o C. WBCs: 17,400 with 83% bands KEY: R 3. 37-year-old woman non-smoker on daily medication for headache has 3-week history of bilateral lower limb claudication. Femoral pulses are faint; popliteal, posterior tibial, and dorsalis pedis pulses are absent. KEY: H 4. 42-year old tall, thin male with acute severe back pain KEY: X
  14. 14. Example: Extended Matching Solitary Liver Mass For each patient whose clinical and imaging information is shown, select the most appropriate diagnosis from the list below. Each option may be used once, more than once, or not at all. a. Focal nodular hyperplasia h. Biliary cystadenoma b. Liver cell adenoma i. Lymphoma c. Cavernous hemangioma j. Solitary metastasis d. Inflammatory pseudotumor k. Intrahepatic cholangiocarcinoma e. Pyogenic abscess l. Hepatocellular carcinoma f. Fungal abscess m. Fibrolamellar carcinoma g. Nodular focal fat deposition
  15. 15. 1: 35 year old woman who underwent sonographic evaluation for mild abdominal discomfort. She was referred for MR imaging to characterize a solitary liver mass seen on that ultrasound. Images are obtained 30 seconds (A), 70 seconds (B) and 1 hour (C) after administration of gadopendetate intravenously. A B C a. Focal nodular hyperplasia b. Liver cell adenoma c. Cavernous hemangioma d. Inflammatory pseudotumor e. Pyogenic abscess f. Fungal abscess g. Nodular focal fat deposition h. Biliary cystadenoma i. Lymphoma j. Solitary metastasis k. Intrahepatic cholangiocarcinoma l. Hepatocellular carcinoma m. Fibrolamellar carcinoma
  16. 16. A B C D 2. 60 year old man with abdominal pain. Imaging performed elsewhere showed a liver mass, and he is referred for CT imaging to characterize it. Images are obtained before (A), 20 seconds after (B), 50 seconds after (C), and 5 minutes after (D) intravenous administration of iodinated contrast material. a. Focal nodular hyperplasia b. Liver cell adenoma c. Cavernous hemangioma d. Inflammatory pseudotumor e. Pyogenic abscess f. Fungal abscess g. Nodular focal fat deposition h. Biliary cystadenoma i. Lymphoma j. Solitary metastasis k. Intrahepatic cholangiocarcinoma l. Hepatocellular carcinoma m. Fibrolamellar carcinoma
  17. 17. Strengths of an R-Type Extended Matching Item • Tests beyond knowledge, comprehension • More analysis, management, top of pyramid • Better discriminator than T/F or A-type • Comparable results with one-third fewer items when compared with A-type • Theoretically, can test more of domain in same time
  18. 18. Script Concordance Testing
  19. 19. Script theory of medical expertise: 1990 (Schmidt et al) “Experienced practitioners possess elaborate networks of knowledge fitted to the regular tasks they perform called scripts.” Scripts allow the clinician to determine the most likely diagnosis, strategies of investigation, and treatment options. Script Concordance Test, an assessment tool: 2000 (Charlin et al)
  20. 20. Script Concordance Testing
  21. 21. Script Concordance
  22. 22. Script Concordance Testing Probes the ability of the individual to interpret and utilize data in specific clinical settings Serves as an indicator of clinical reasoning Script Concordance Testing discriminates between highly experienced and lesser experienced individuals. Will require pre-test familiarity
  23. 23.  The main purpose of the ABMS MemberThe main purpose of the ABMS Member Boards, including the ABR, isBoards, including the ABR, is to improveto improve healthcare safety and quality in the U.S.healthcare safety and quality in the U.S. by establishing standards for:by establishing standards for: • Initial certificationInitial certification • Maintenance of CertificationMaintenance of Certification
  24. 24. General Competencies • Medical KnowledgeMedical Knowledge • Patient CarePatient Care • Interpersonal and Communication SkillsInterpersonal and Communication Skills • ProfessionalismProfessionalism • Practice-based Learning and ImprovementPractice-based Learning and Improvement • Systems-based PracticeSystems-based Practice  ABMS, September 1999ABMS, September 1999
  25. 25. • Professional standing (I)Professional standing (I) • Lifelong learning and self-assessment (II)Lifelong learning and self-assessment (II) -CME-CME -SAMs-SAMs • Cognitive expertise (III)Cognitive expertise (III) -Examinations-Examinations • Practice performance (IV)Practice performance (IV) -Many include self-assessment, blend in Part 2-Many include self-assessment, blend in Part 2  ABMS, March 2000ABMS, March 2000 Maintenance of Certification Components & Item Writing Opportunities
  26. 26. ABR Volunteer Opportunities http://www.theabr.org
  27. 27. • Qualifying (“written”) examsQualifying (“written”) exams • Oral certifying examsOral certifying exams • MOC examsMOC exams • SAMsSAMs • Computer-based Core ExamComputer-based Core Exam • Computer-based Certifying ExamComputer-based Certifying Exam ABR Volunteer Opportunities in Diagnostic Radiology For those skilled in item-writing:
  28. 28. Activity Time Commitment Travel Method Interaction with Colleagues Qualifying (“written”) exams: Write test questions 10-25 hrs / year; 3-year term No Web-based submission Feedback by 1-2 people Oral certifying exams: Serve as oral examiner 5 days 1st week of June; state preference for re- invitation Yes 1st day orientation; 2-4 full days of examining; 13-16 candidates per day 15-20 colleagues, same sub- specialty, + 175-200 DRs total; social opportunities MOC exams: Serve as subspecialty committee member for MOC exam committee 1 in-person meeting / year; 1-2 electronic meetings / year; 3-year term Yes Committee meeting; follow-up electronically Committee of colleagues in same specialty meets in- person & electronically
  29. 29. Activity Time Commitment Travel Method Interaction with Colleagues SAMs: Serve as reviewer of SAM applications 1-2 hrs / review; 1-6 reviews / yr; 3-year term No Email; FTP Occasional with other reviewers Future core exam: Serve as committee member to develop exam 1 in-person mtg & 2-4 electronic mtgs per year; 3-yr term Yes 1x /yr Tucson Committee meeting; follow-up electronically Committee of colleagues in same specialty meets in- person & electronically over 3 years Future certifying exam: Serve as subspecialty committee member for MOC exam 1 in-person mtg & 2-4 electronic mtgs per year; 3-yr term Yes 1x /yr Tucson Committee meeting; follow-up electronically Committee of colleagues in same specialty meets in-
  30. 30. What Are the Essential Ingredients of Self-Assessment?  Clear standardsClear standards  ““Self-evident” criteriaSelf-evident” criteria  Time for application, practice, feedbackTime for application, practice, feedback  Reflection on practiceReflection on practice  personalpersonal  peer stimulatedpeer stimulated
  31. 31. What’s “self” about SAMs?  Robust self-assessment employs a unique…Robust self-assessment employs a unique…  Method: Reflective examination of selfMethod: Reflective examination of self  Mindset: Attitude change fromMindset: Attitude change from concealment to opennessconcealment to openness  Goal: Self-improvementGoal: Self-improvement  Catalyst: Application, practice, andCatalyst: Application, practice, and feedbackfeedback  Response: Changes in own practiceResponse: Changes in own practice
  32. 32. Findings from studies of effective adult instruction  What is more effective thanWhat is more effective than didactic presentations?didactic presentations?  ActiveActive  LongitudinalLongitudinal  TransferableTransferable  Engaging assignmentsEngaging assignments  MentoringMentoring
  33. 33. Authentic Assessments • Tasks that closely simulate or actuallyTasks that closely simulate or actually replicate challenges faced in practicereplicate challenges faced in practice • Use of genuine examples of the learner’sUse of genuine examples of the learner’s workwork • Use of models with which learner comparesUse of models with which learner compares • Pose real dilemmas; draw on commonPose real dilemmas; draw on common misconceptionsmisconceptions  Interactive respondingInteractive responding  Reiterative respondingReiterative responding
  34. 34. Feedback  Instructor:Instructor: Describes what the learner did/did not doDescribes what the learner did/did not do References standards of quality / competenceReferences standards of quality / competence Delivers immediatelyDelivers immediately  Learner:Learner: Self-perceives both correct answers and errorsSelf-perceives both correct answers and errors Analyzes causesAnalyzes causes
  35. 35. SAM Data
  36. 36. SAMs by Category April 2010 Category Number of SAMs Qualified Number of SAMs Currently Available ARRS SAMs Qualified ARRS SAMs currently available Musculoskeletal 47 24 15 12 Cardiac 21 6 5 4 Thoracic 36 12 8 6 Gastrointestinal 55 17 11 6 Genitourinary 34 14 10 8 Neuroradiology 89 33 9 7 Vascular and Interventional 64 9 6 4 Nuclear Medicine 62 14 2 2 Ultrasound 22 5 3 2 Pediatric 30 3 3 1 Breast 38 11 2 1 Other 25 2 0 0 Non-Interpretive Skills 71 19 7 5 Total 594 169 81 58

×