Harnessing the Power of Categories: Using Categories to Improve Exam Questions
•Download as PPTX, PDF•
0 likes•301 views
Presented by Donna K Czarnecki, Lead CBT, Data, and Core Coordinator, University of Nebraska Medical Center, College of Medicine Item analysis and review can often be a daunting process for educators. This presentation will focus on how the category feature in ExamSoft can be used to foster faculty collaboration in reviewing banked items (exam questions) to improve their performance. This webinar will also include discussion around how to identify problem items and introduce a system to track needed and completed question changes.
Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Harnessing the Power of Categories: Using Categories to Improve Exam Questions
Similar to Harnessing the Power of Categories: Using Categories to Improve Exam Questions (20)
More from ExamSoft
More from ExamSoft (20)
Recently uploaded
Recently uploaded (20)
Harnessing the Power of Categories: Using Categories to Improve Exam Questions
- 1. HARNESSING THE POWER OF CATEGORIES: USING CATEGORIES TO IMPROVE EXAM QUESTIONS Donna K. Czarnecki, BS Lead Computer-Based Testing, Data and Core Coordinator University of Nebraska Medical Center, College of Medicine
- 2. For Assessment Building To Students To Faculty & Program Work with Focused Set of Items Facilitates Review/Improvement Process
- 3. Reasons to Categorize – Manage your Items • Allows you and your faculty to work with smaller sets of specific questions. • No need to duplicate questions in another question folder. (andyourfaculty)
- 4. Questions by Category • Subset of questions from Questions by Folder area • Questions are tagged with categories to group them together. • Think of these groups of questions as being “sorted” out of the Questions by Folder database. Questions by Folder Topic Faculty Name Course Academic Year
- 5. Requires Review • Does not meet formatting requirements • Has poor statistical properties/was removed from scoring Two main reasons an item requires review:
- 6. Requires Review Does not meet formatting requirements: • NBME Item Writing Guide • Institutional Preferences
- 7. Requires Review Does not meet formatting requirements: • NBME = National Board of Medical Examiners • NBME Item Writing Guide • http://www.nbme.org/PDF/ItemWriting_ 2003/2003IWGwhole.pdf
- 8. Requires Review Does not meet formatting requirements: • NBME Item Writing Services http://www.nbme.org/IWW/ • Online Interactive Item Writing Tutorial • Item Writing Seminars for Individuals • Item Writing Workshops for Groups
- 9. Requires Review Does not meet formatting requirements: • Institutional preferences • Bloom’s Taxonomy • http://www.bloomstaxonomy.org/
- 10. Requires Review Does not meet formatting requirements: • A few examples: • Not phrased as a question • Includes an “all of the above” or “none of the above” answer option • True/False • Includes improper level of questioning (diagnosis for M1s, dosing for M2s)
- 11. Requires Review - Does not meet formatting requirements:
- 12. Requires Review Has poor statistical properties/ was removed from scoring • Low difficulty index or point biserial correlation • Removed from scoring due to poor stats
- 13. Requires Review Has poor statistical properties/ was removed from scoring • Low difficulty index or point biserial correlation • Difficulty below 0.50 • Discriminator/point biserial below 0.20 • Negative discriminator/point biserial
- 14. Requires Review Has poor statistical properties/ was removed from scoring • Removed from scoring due to poor stats • Tagged with the appropriate academic year the question was removed from an exam.
- 15. Requires Review - Has poor statistical properties/ was removed from scoring
- 16. Requires Review - Review process for faculty 1. Go through all exam questions and tag if review is required
- 17. Requires Review - Review process for faculty 2. Set up review category for faculty to access
- 18. Requires Review - Review process for faculty 3. Sort/search for review categories in appropriate question/course folder
- 19. Requires Review - Review process for faculty 3. Sort/search for review categories in appropriate question/course folder
- 20. Requires Review - Review process for faculty 3. Sort/search for review categories in appropriate question/course folder
- 21. Requires Review - Review process for faculty 3. Sort/search for review categories in appropriate question/course folder
- 22. Requires Review - Review process for faculty 4. Tag questions for inclusion into review category
- 23. Requires Review - Review process for faculty 5. Provide faculty with location of questions to review (including a deadline)
- 24. Requires Review – Archiving Bad Questions • For those questions that are not revised or negotiated to remain the same, we add the word “ARCHIVE” at the front of the questions title. • This indicates to us that the question should not be used on an assessment. • By doing this, the question will still remain categorized in the Questions by Folder area, so the faculty can still see it, but it will not be used on the exam in its current state. • Eventually the “ARCHIVE” questions will be removed from the folders, and then eventually deleted from the system.
- 25. Requires Review Categories 2016 M1 Item Review Function Course Structure Course Cellular Processes Course Neurosciences Course Questions by Folder
- 26. Questions by Category • Questions assigned to a category are not a separate set of questions. • A question can only be in one folder but many different categories. • Do NOT delete questions from category bins, or you will delete them from the database permanently! • Instead, remove the category assignment from a question to remove it from that category bin.
- 27. Questions??? My contact info: Donna K. Czarnecki, B.S. Lead Computer-Based Testing, Data and Core Coordinator Office of Medical Education, College of Medicine University of Nebraska Medical Center 402.559.4043 I dczarnec@unmc.edu
Editor's Notes
- There are many benefits to using categories. During the last webinar I discussed using the Categories area to physically build assessments. Today I will focus on their usefulness for reviewing and improving exam questions.
- Questions by category are basically a subset or group of questions from the Folder area. Each institution’s categories are based on their specific needs. Here at UNMC we have the illustrated categories plus many more. Think of it as a way to filter or sort out certain questions you want to see or work with.
- At UNMC we are guided by two things when determining if a question meets our formatting requirements. One is the NBME Writing Guide and the other is our own particular institution’s preferences.
- First let me explain that NBME stands for the National Board of Medical Examiners. As you may remember from my introduction, I work with the first and second year medical students at our university. The NBME offers a variety of educational materials and instructional opportunities to help enhance the quality of test items used by educators and assessors in the healthcare professions around the world. The NBME's Item Writing Manual, Constructing Written Test Questions for the Basic and Clinical Sciences, is a resource for faculty members at schools of medicine, schools for healthcare professionals, those invested in the assessment of competence of healthcare providers, and others interested in learning how to write better quality test items. A link to the downloadable manual is listed. As just mentioned, this information is not just limited to use in the healthcare professions so I encourage you to take a look.
- In addition to the writing guide manual, NBME offers several other item writing services with their tutorials, seminars and workshops.
- Another major guiding force here at UNMC is the use of Bloom’s Taxonomy. This topic alone needs its own webinar, which guess what? Has already been done. You can find these webinars on the ExamSoft website and plenty of information on the web. Because of this, I will not get very deep into this topic. Suffice it to say that we try to use higher level taxonomic questions with our students to prepare them for their board exams and licensure.
- I have listed a few examples here of types of questions that do not meet our formatting requirements. The first one – Not phrased as a question – is exactly that. These are actually question stems that end with a semicolon and continue on into the question foils to complete a sentence. All of the Above and None of the Above questions do not contain a simple answer or response and often cue the student to the correct answer. True/False questions require personal interpretation and may be confusing to students The last example is something specific for our students and institution.
- Here is a list of categories we use for inappropriate formatting and a few other categories, some of which are more specific to our institution. There are two other important formatting issues I would like to mention: Remove Negative Phrasing – these are questions that contain the word “NOT” in the question stem or include All of the following EXCEPT: Two Answer Choice Allowed – these are questions that were determined after an exam to have had two possible correct answer.
- Here is another major category, and sub categories, regarding the actual statistical performance of the exam questions. It can be broken up into two areas: the actual performance stats and whether or not it was removed from scoring.
- The first thing we look at with our stats is the difficulty. This is the percentage of students that answered the question correctly. If the difficulty is below 0.50, then we look at the discriminator or point biserial. If it is below 0.20 the question is tagged with poor statistical properties, and usually removed from the exam. Negative discriminators also indicate a possible problem and are noted for review.
- Questions that are removed from scoring after an exam are tagged with this information for the appropriate academic year.
- Here is a screenshot of how we have our categories set up for poor statistics The last category is designed to be used by a course director if they need the question looked over by a content expert.
- Unfortunately someone has to go through and tag all of the questions that need to be reviewed. Don’t worry, this process is not as bad as it sounds. This task was performed by myself and my colleague once we had imported all of our questions. I would highly recommend that you do this as questions are added to your database. We imported over 4,000 questions in our first year, so this was quite a project to go through all of the questions after that.
- In my last webinar I talked about setting up a “construction zone” under the Categories area to build assessments. Using a similar method, we created an area for the faculty to review their questions. We set up the main category “2016 M1 Item Review” and then set up sub-categories based on the different courses. This allows the appropriate course director the ability to only see and review questions associated with their course.
- Use the “Advanced Search” area located in the upper right area of your screen.
- Search for questions with the category “Requires Review.” Scroll down and select done.
- Select the course folder where all of the questions are located.
- Select “All Questions” in the status field and then select “Search.”
- You will now have a list of all of the questions from that course folder that “Requires Review.” Select all and add the review category tags to all of the question using the bulk add feature. Make sure to select “Update Categories” when you are all done.
- All of this leads us back to the listing of questions, by course, in the 2016 Review category. This is the location for your faculty to access their questions. Once they click on the review category for their course, they will see all of the question they need to review. Based on this screenshot, if the Neuroscience Course Director click on his course review category, he would see the 129 questions located there. Most importantly, make sure to give the Course Director and/or Faculty a deadline to complete their review.
- For those questions that are not revised or negotiated to remain the same, we add the word “ARCHIVE” at the front of the questions title. This indicates to us that the question should not be used on an assessment. By doing this, the question will still remain categorized in the Questions by Folder area, so the faculty can still see it, but it will not be used on the exam in its current state. Eventually the “ARCHIVE” questions will be removed from the folders, and then eventually deleted from the system.
- Here is a diagram to help with the process for those that are more visual. The entire circle shows all of the questions in the ExamSoft database, located in the Questions by Folders area. Inside of the entire circle is a small subset of questions that need to be reviewed. Inside of the review circle are the course subsets. Remember, these are not separate sets of questions. They are just sorted out by categories to make them easier to review for the appropriate person.
- And finally – a very important reminder.