Learning by doing aalhe presentation handout

1,699 views

Published on

This is the handout version of a workshop I gave to the Association for the Assessment

Published in: Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,699
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Add animations, add emphasis to the action verbs, vague/fuzzy words, etc.
  • Add animation box saying, “In this case, we can combine many specific objectives into a single, efficient one.
  • Add more specific instructions once vignettes are written
  • Test blueprint process used to improve the assessment instrument.Course Leaning ObjectivesI started with the list of new learning objectives (shows picture) for the whole courseIdentify test “topics” from learning objectivesUsing this list I generated what test topics I wanted to cover with the assessment, which included removing course-specific items that students wouldn’t know when coming into the classAllocate percent of test to each topicTake the topics and allocate a % of the test to each one based on importance/number of learning objectives inherent to itExpand each topic to as many “concepts” as possibleSimilar to a brainstorming session, use course content to expand each topic as far as possible to include as many individual concepts as possibleRedundancy is OKDon’t worry about redundant concepts at this point, it is better to be inclusiveReallocate %Look at your list and reallocate the % of the test based on number of conceptsCollapse list of concepts to remove redundancyNow is when you identify the redundant items and shift/delete them to make a more parsimonious blueprint “overlapping” conceptsThe key is to look for “overlapping” concepts, or concepts that go together due to how you plan to ask the questions (give example of research design/statistics)Final allocation of test %Use the reduced list of concepts to create a final allocation of test %Create/modify itemsUse this list as a base for modifying and/or adding items to your assessment.
  • Test blueprint process used to improve the assessment instrument.Course Leaning ObjectivesI started with the list of new learning objectives (shows picture) for the whole courseIdentify test “topics” from learning objectivesUsing this list I generated what test topics I wanted to cover with the assessment, which included removing course-specific items that students wouldn’t know when coming into the classAllocate percent of test to each topicTake the topics and allocate a % of the test to each one based on importance/number of learning objectives inherent to itExpand each topic to as many “concepts” as possibleSimilar to a brainstorming session, use course content to expand each topic as far as possible to include as many individual concepts as possibleRedundancy is OKDon’t worry about redundant concepts at this point, it is better to be inclusiveReallocate %Look at your list and reallocate the % of the test based on number of conceptsCollapse list of concepts to remove redundancyNow is when you identify the redundant items and shift/delete them to make a more parsimonious blueprint “overlapping” conceptsThe key is to look for “overlapping” concepts, or concepts that go together due to how you plan to ask the questions (give example of research design/statistics)Final allocation of test %Use the reduced list of concepts to create a final allocation of test %Create/modify itemsUse this list as a base for modifying and/or adding items to your assessment.
  • Test blueprint process used to improve the assessment instrument.Course Leaning ObjectivesI started with the list of new learning objectives (shows picture) for the whole courseIdentify test “topics” from learning objectivesUsing this list I generated what test topics I wanted to cover with the assessment, which included removing course-specific items that students wouldn’t know when coming into the classAllocate percent of test to each topicTake the topics and allocate a % of the test to each one based on importance/number of learning objectives inherent to itExpand each topic to as many “concepts” as possibleSimilar to a brainstorming session, use course content to expand each topic as far as possible to include as many individual concepts as possibleRedundancy is OKDon’t worry about redundant concepts at this point, it is better to be inclusiveReallocate %Look at your list and reallocate the % of the test based on number of conceptsCollapse list of concepts to remove redundancyNow is when you identify the redundant items and shift/delete them to make a more parsimonious blueprint “overlapping” conceptsThe key is to look for “overlapping” concepts, or concepts that go together due to how you plan to ask the questions (give example of research design/statistics)Final allocation of test %Use the reduced list of concepts to create a final allocation of test %Create/modify itemsUse this list as a base for modifying and/or adding items to your assessment.
  • Test blueprint process used to improve the assessment instrument.Course Leaning ObjectivesI started with the list of new learning objectives (shows picture) for the whole courseIdentify test “topics” from learning objectivesUsing this list I generated what test topics I wanted to cover with the assessment, which included removing course-specific items that students wouldn’t know when coming into the classAllocate percent of test to each topicTake the topics and allocate a % of the test to each one based on importance/number of learning objectives inherent to itExpand each topic to as many “concepts” as possibleSimilar to a brainstorming session, use course content to expand each topic as far as possible to include as many individual concepts as possibleRedundancy is OKDon’t worry about redundant concepts at this point, it is better to be inclusiveReallocate %Look at your list and reallocate the % of the test based on number of conceptsCollapse list of concepts to remove redundancyNow is when you identify the redundant items and shift/delete them to make a more parsimonious blueprint “overlapping” conceptsThe key is to look for “overlapping” concepts, or concepts that go together due to how you plan to ask the questions (give example of research design/statistics)Final allocation of test %Use the reduced list of concepts to create a final allocation of test %Create/modify itemsUse this list as a base for modifying and/or adding items to your assessment.
  • Be sure to describe that “NOTA” and “AOTA” effectively turn the question into a simple T/F item.
  • Learning by doing aalhe presentation handout

    1. 1. “Learning by Doing”By Patrick B. BarlowPhD Candidate in Evaluation, Statistics, & MeasurementThe University ofTennesseeA Hands onApproach to Multiple ChoiceAssessment
    2. 2. A Bit of Background• Currently a PhD Candidate in Evaluation, Statistics, & Measurement atTheUniversity ofTennessee• Been working in higher education assessment since 2008• Began working on aTeagle Foundation grant teaching an “Assessment 101” coursewith Dr. Phil Kramer at Saint John’s University (MN).• Recently (2011) shifted my focus to assessment in Graduate MedicalEducation, specifically in evidence-based medicine, research methods, andstatistics.• Developing a new biostatistics and clinical epidemiology assessmentinstrument for my dissertation using an Item ResponseTheory (IRT)approach.
    3. 3. Objectives ofToday’sWorkshopAfter today’s workshop, participants will be able to:• Translate vague course learning goals to measurable learning objectives• Develop an effective test blueprint for their multiple-choice assessment• Create utilization-focused assessments that focus on giving students real-worldapplication of concepts, and giving instructors useful feedback on student learning.• Write effective item stems and response options that reflect best practices in itemwriting.• Critique their existing assessment instruments for areas of improvement.I Hope!
    4. 4. Today’sWorkshop at a Glance• Part I• LearningObjectives• Test Blueprints• Activity I• Writing learning objectives from vague coursegoals• Part II• Defining Multiple Choice Questions (MCQs)• Best practices in MCQ writing• Challenges to writing effective MCQs• Activity II• “Assessment 911!”
    5. 5. FundamentalsWhat areWeTalking AboutToday?• What are MCQ Assessments?• Why• When• But…• Most professors are not trained as assessment experts• Assessment as a stick vs. carrot• Questions of Use
    6. 6. Back to BasicsWriting High Quality Learning Objectives
    7. 7. Integrated Perspective• Integrated Course Design (Fink, 2003)• Learning objectives inform assessmentstrategies, which inform teaching &learning activities.• All components are integrated &support one another.LearningObjectivesAssessmentTeaching &Learning ActivitiesSituational Factors
    8. 8. Goals vs. Objectives• For this workshop, we define learning “goals” and learning “objectives” as(Suskie, 2009):• Goal: how the students will be different as a result of the course.• EX: “Students will gain a working knowledge of the fundamental concepts of epidemiologicresearch methods and statistics as well as the ability to comfortably read research articles intheir specialty.”• Objective: a specific, measurable outcome that describes the details of each goal.• EX: “Students will be able to identify different scales of measurement for variables”
    9. 9. Writing a High Quality Objective• Be concise• Neither too vague nor too specific—find abalance• Focus on the results• What should the students have after the coursenot while in the course?• Use action words• Avoid vague or fuzzy terms such as “learn,”“demonstrate,” or “know”.• Write in a teamAdapted from Suskie, 2009
    10. 10. Moving fromVague Goal to Measurable ObjectiveStatistics 101 Course Example• Vague learning goal… Students willunderstand basicstatistics conceptsDescriptive statisticsHypothesisTestingScales ofmeasurementStudents will be able to formulate a nulland alternative hypothesis for a givenresearch scenario• Concepts related to thegoal• Measurable objective forone of those concepts
    11. 11. Importance of Being Parsimonious• Sometimes the list of specific concepts would become too cumbersome iftranslated into a long list of learning objectives…Students will be able to apply a cross-sectionalstudy design to their own research areaStudents will be able to apply a cohort studydesign to their own research areaStudents will be able to apply a case-controlstudy design to their own research areaStudents will be able to apply an RCT studydesign to their own research areaStudents will be able toapply the four keyepidemiologic studydesigns to their ownresearch area.
    12. 12. YourTurn…• In groups of three to four…• Choose an area of research and/or teaching tofocus on for the activity.• Feel free to choose among your own interests, orpick something as a group.• Use what we have discussed to think of thefollowing:• Two learning goals• At least three specific, measurable learningobjectives that could be used to describe thosegoals.
    13. 13. Test BlueprintsUsing LearningObjectives to PlanAssessment Structure
    14. 14. What is aTest Blueprint?• In general, a test blueprint refers to the allocation of items or sections of atest according to certain concepts or ideas (Suskie, 2009)
    15. 15. Why Use aTest Blueprint?• Makes tests easier to construct (think of it likea test “recipe”)• Assures that you address relevant information• Assists with generating study materials forstudents• Makes “teaching to the test” more like“teaching with the test”• If the test only asks questions on what you seeas important, then you would hope yourteaching reflects these topics.
    16. 16. One Approach for Creating aTest Blueprint• Ingredients• 1 Course syllabus• 1 Set of LearningObjectives• 1 Set of lecture materials orother classroom activities• CalculatorIdentify Key Learning ObjectivesCreate test “topics”Allocate fractions of the test toeach topicRevise the list of topics & repeatThe Basic Cycle
    17. 17. “after thismoduleparticipantsshould beable to…”1. Start with course learningobjectivesBiostatistics & Clinical Epidemiology AssessmentExampleApply eachdesign to theirown area ofresearchIdentify majorepidemiologicresearch designs
    18. 18. 1. Start with course learningobjectives2. Identify test “topics” fromlearning objectivesBiostatistics & Clinical Epidemiology AssessmentExampleApply eachdesign to theirown area ofresearchIdentify majorepidemiologicresearch designsResearchDesign (20%)
    19. 19. 1. Start with course learningobjectives2. Identify test “topics” fromlearning objectives3. Expand each topic to as many“concepts” as possibleBiostatistics & Clinical Epidemiology AssessmentExampleIdentify majorepidemiologicresearch designsIdentify an RCTIdentify a Case-Control StudyIdentify aCohort StudyIdentify aCross-Sectional StudyApply an RCTApply a Case-Control StudyApply a CohortStudyApply a Cross-Sectional StudyApply eachdesign to theirown area ofresearch
    20. 20. 1. Start with course learningobjectives2. Identify test “topics” fromlearning objectives3. Expand each topic to as many“concepts” as possible4. Collapse list of concepts toremove redundancy5. Create/modify itemsBiostatistics & Clinical Epidemiology AssessmentExampleIdentify majorepidemiologicresearch designs“after thismoduleparticipantsshould beable to…”RCTCase-ControlStudyCohort StudyCross-SectionalStudyIdentify majorepidemiologic researchdesigns, and apply eachdesign to their ownresearch area
    21. 21. For Discussion• What is your approach for designing an objective assessment?• What has worked out well for you in the past?• What has not worked out so well for you?• How do we, as assessment experts, describe the utility of making testblueprints to our colleagues or clients?
    22. 22. Section II: WritingEffective MCQTest ItemsDefining MultipleChoice Questions (MCQs)Best practices in MCQ writingChallenges to writing effective MCQs
    23. 23. Section IIWriting Effective MCQTest Items• Defining multiple choice items• Two main types of MCQs• Characteristics of multiple choice items that emphasize higher-order thinking• Writing effective item prompts/stems• Writing answer choices and distractors• Challenges to writing multiple choice items• Item independence• Avoiding “testwise” students
    24. 24. Defining MCQs• We will be comparing two types of MCQs inthis workshop• True / False MCQs• Examinees select all true options such as“OptionA, option B, both A & B, or Neither Anor B”• Simple true / false questions• One-Best-Answer MCQs• Examinees select a single best response usuallyfrom 4 or more optionsAdapted from Case & Swanson, 1998
    25. 25. ExamplesTrue / False MCQ• Which of the following is the correct formulafor hydrogen peroxide?a) H2Ob) CO2c) H4O2d) 2(HO)One-Best-Answer MCQA lab assistant finds an unmarked beaker containinga clear, viscous liquid.The assistant further tests theunknown liquid to find that it is non-flammable, andslightly acidic (pH = 6.2), and has a boiling point of150 oC.What chemical is most likely in the unmarkedbeaker?a) H2Ob) CO2c) H4O2d) 2(HO)
    26. 26. What’s the Difference?True / False MCQs can only be judged as absolutely true or absolutelyfalse.There are no shades of grey.• Which of the following is thecorrect formula for hydrogenperoxide?a) H2Ob) CO2c) H4O2d) 2(HO)DCBAFalse True
    27. 27. What’s the Difference?DCB ALeastCorrectMostCorrectA lab assistant finds an unmarked beakercontaining a clear, viscous liquid.The assistantfurther tests the unknown liquid to find that itis non-flammable, and slightly acidic (pH = 6.2),and has a boiling point of 150 oC.What chemical is most likely in the unmarkedbeaker?a) H2Ob) CO2c) H4O2d) 2(HO)On the other hand, one-best-answer MCQs can be qualitativelyranked from “least correct” to “most correct”
    28. 28. For Discussion…• Which MCQ format would do a better job atmeasuring higher-order thinking skills (e.g.application of knowledge)?Why?• Would the preferred type of MCQ differ byfield of study?Why or why not?
    29. 29. Recommendation• Experts recommend that test writers do not use true / false MCQs• Choosing the single “correct” answer is problematic when the concept is not actuallyan absolutely “true” or “false” dichotomy.Example:A student collects information on the socioeconomic status of a group of 1000incoming first-year college students. She wishes to create a report on the typical familyincome for her sample.What would be the most accurate way to report this data?a) Report the mean of the incomeb) Report the range of the incomesc) Report the median of the incomesd) Report the standard deviation of the incomes
    30. 30. Writing Effective Item Stems• A traditional format for a one-best-answer MCQ is to have a stem followedby the lead in question.For Example…• A research team conducts a randomized, prospective study to compare a 1-day, 4-drug regimen with a 7-day, 3-drugregimen in their efficacy at eradicating a particular infection.They are hypothesizing that the two groups would not differin the proportion of patients whose infection was eradicated by a clinically meaningful amount (+/- 15% eradicationpercentage)What statistical analysis would be the most appropriate to way to test this hypothesis?a) Chi SquareTest of independenceb) Independent samples t-Testc) Non-inferiority test for proportionsd) Equivalence test for proportions
    31. 31. Key Points of Effective Stems• Be clear, concise, and avoid vague terms• Define all necessary terms & avoid jargon• Ask a complete question, that is, the student should not have to read the responseoptions to understand what they’re being asked.• Avoid using negatively-worded stems such as “Which of these are not…” or“All of these except”• Avoid “Which of the following…”• Try not to introduce construct irrelevant difficulty• Items should focus on important concepts only• Avoid verbal or grammatical associations between the stem and answers
    32. 32. So,What DoesThat Mean?• In a research study, the age of the participants was 26years +/- 5 years (mean +/- standard deviation).Whichof the following statements is the most correct?• It is 95% certain that the true mean lies within theinterval of 16-36 years.• Most of the patients were aged 26 years; the remainderwere aged between 21 and 31 years.• Approximately 95% of the patients were aged between16 and 36 years.• No patients were younger than age 16 or older than age36.Vague Stem ExampleAdapted from Windish, Hout, & Green 2007; & Ayers, Schulzinger, de laTeja, & GrayWhite, 2009)
    33. 33. So,What DoesThat Mean?• In a research study, the age of the participants was 26years +/- 5 years (mean +/- standard deviation).Whichof the following statements is the most correct?• It is 95% certain that the true mean lies within theinterval of 16-36 years.• Most of the patients were aged 26 years; the remainderwere aged between 21 and 31 years.• Approximately 95% of the patients were aged between16 and 36 years.• No patients were younger than age 16 or older than age36.Vague Stem ExampleNegatively-WordedStem ExampleAdapted from Windish, Hout, & Green 2007; & Ayers, Schulzinger, de laTeja, & GrayWhite, 2009)• All of the statements about women’s rights after theRevolution are true except…• Married women could not sign contracts or ownproperty• Women could not vote in Massachusetts• Women were not offered higher education• Women could not vote in any state
    34. 34. Construct Irrelevant Difficulty?• Several Sources• Unnecessary background information• Too little background information (i.e. vaguestem)• Concepts that do not belong to the construct ofinterest (use a blueprint!)• Grammatical, spelling, or other languagedifficulties• “Speededness”• Example…
    35. 35. The Bottom Line“Remove all the barriers that will keepa knowledgeable student fromanswering the item correctly...[and]Remove all clues that will help a less-than-knowledgeable student answerthe item correctly”(Suskie 2009, pg. 170)
    36. 36. For Discussion• Time to Share, nobody’s perfect!• We have all written questions that violate oneor more of these guidelines whether we knowit or not.Thinking back on assessments thatyou have created, what are some of the morecommon “errors” you find yourself making?• My own example(s)…
    37. 37. Writing EffectiveResponse Options
    38. 38. Dos & Don’ts of Effective Response OptionsDos• Order responses logically• Use vertical rather than horizontal responses• Make all options similar length• Write distractors that are intrinsically possibleor true statements• Use homogeneous distractorsDon’ts• Use “all of the above” or “none of the above”*• Use verbal clues or absolutes such as “always,”“never,” “usually,” etc• Use two-part or double-barreled options*• Use response options that can be answered bylooking at other questions• Include grammatical “cues” in the responseoptionsCase & Swanson (1998); Suskie (2009); & Brunnquell, A., Degirmenci,U., Kreil, S., Kornhuber, J., & Weih, M. (2011).
    39. 39. Dos & Don’tsA Couple of ExamplesDo…• About 25 of the 92 natural elements are knownto be essential to life.Which four elementsmake up approximately 96% of living matter?a) Carbon, sulfur, phosphorus, hydrogenb) Carbon, hydrogen, nitrogen, oxygenc) Carbon, oxygen, hydrogen, calciumd) Carbon, sodium, chlorine, nitrogenDon’t…• Which of the following statements best distinguisheshypotheses from theories in science?a) Hypotheses usually are relatively narrow in scope;theories have broad explanatory power.b) Theories are hypotheses that have been provedc) Theories are proved true in all cases; hypotheses areusually falsified by tests.d) Hypotheses are guesses; theories are correct answersmake distractors similar length “longest answer = correct answer”Adapted from multiple exams in a Biology 1001 Course, Spring 2013; NorthHennepinCommunityCollege; Brooklyn Park, MN 55445
    40. 40. Dos & Don’tsA Couple of ExamplesWrite plausible options Use “all (none) of the above”• Which thinker had a major impact on early-twentieth-century writers, leading them toreimagine human identity in radically new ways?a) Sigmund Freudb) Sir James Frazerc) Friedrich Nietzsched) All of the above• Which of the following phrases best characterizes thelate-nineteenth century aesthetic movement whichwidened the breach between artists and the readingpublic, sowing the seeds of modernism?a) Art for intellect’s sakeb) Art for God’s sakec) Art for the massesd) Art for art’s sakeAdapted from the Norton Anthology of English Literature Multiple ChoiceQuizzes (http://www.wwnorton.com/college/english/nael/quiz.htm)Do… Don’t…
    41. 41. Challenges toWriting MCQs“Interlocking” ItemsAvoidingTestwise Students
    42. 42. “Interlocking” or “Interdependent” Items• Items can become interlocked or dependent on one another when:• Response options or stems of one item are related to (or give the answers to) another question• The same passage, calculation, graph, etc. is used for multiple items• Using common examples, passages, etc. is an attractive option when writing tests forefficiencies sake.• However, it disadvantages students who don’t understand the example (even when they may knowthe concept)• Interlocking items are especially a concern when using Item ResponseTheory (IRT)approaches to test construction, which assumes all items are locally independent of oneanother.(DeAyala, 2009; DeMars, 2010; & Suskie, 2009)
    43. 43. An Example of Interlocking Items1.What is the most likely value forthe slope of this graph?2. What is the most likelyY-intercept for the graph above?3. A mathematics faculty memberhas 10 years of experience at theuniversity. What should be herapproximate income?• An institutional researcher at a large university is conducting a salaryequity analysis on the university employees. One of these analyses isto look at the relationship between employee income and years ofexperience. Use their results below to answer questions 1-5Image from http://allpsych.com/researchmethods/images/scatterplot.gif
    44. 44. “Cheating”Avoiding GivingTestwise Students an Easy Answer• There are several specific errors in responseoption writing that lead students to pick upon the correct answer without actuallyknowing the concept in question.This“testwiseness” introduces a host of issueswith test reliability and validity byinadvertently lowering the difficulty of thetest.
    45. 45. “Cheating”Avoiding GivingTestwise Students an Easy Answer• Grammatical cues• Testwise students will easily eliminate options that do not grammatically connect totheir stem• Logical cues—not all answers are plausible or can be ranked in “correctness”• Often seen when test writers need to quickly think of the “D” or “E” option just to have4-5 response options.• Absolute terms such as “always” and “never”• Testwise students are far more likely to eliminate options that deal in absolutes, butthis flaw can be avoided through including the words in the stem rather than theoptions.Key issues related to testwiseness (Case & Swanson, 1998)
    46. 46. “Cheating”Avoiding GivingTestwise Students an Easy Answer• Long correct answer• Testwise students can easily see when the correct answer is written more completely, moredetailed, or otherwise longer than the others.• Repeated words• If a response option uses the same word as the stem, then it is more likely to be the correctanswer; however, repeating the word in all options is a great strategy.• Convergence• Correct answer is the option that has the most in common with other options such as themiddle number in a math MCQ (i.e. it won’t be an extreme number)Key issues related to testwiseness (Case & Swanson, 1998)
    47. 47. For DiscussionHow do you avoid testwiseness in your own assessment practices?
    48. 48. FinalThoughtsWhat is theTakeAway Message Here?
    49. 49. Feeling Overwhelmed?• Never fear, no test is perfect…• As an example, a recent review found at least three formal item writing flaws in all 40of the continuing medical education items published in the New EnglandJournal ofMedicine (Stagnaro-Green & Downing, 2006).
    50. 50. A Couple Parting Recommendations• Always begin with a clear set of learning objectives and test blueprint• Write items in teams, and seek a peer reviewer whenever possible• Work to observe some of the basic dos and don’ts we covered today, but remember toalways balance rigor with practicality!• Developing assessments, just like conducting assessments, is an iterative process.Asyou conduct assessment within a cycle of feedback and revision, so must you makechanges to the instrument itself to generate a better product.
    51. 51. Final ActivityMCQ 911!• You’ve been asked by a colleague to review their latest classroomassessment instrument. He has not had any formal training in writing testquestions, and is very worried he will not get accurate measurements of hisstudents’ knowledge.• Use what you have learned about writing effective MCQ items to give yourcolleague’s assessment some “emergency care.”
    52. 52. Final ActivityMCQ 911!• Instructions:• Review the MCQs in your groups• What are the writing flaws your colleague has made in their items?Which ones are the most severe?• What suggestions or changes would you have for you colleague in order to fix some of these flaws?• Things to look for:• Vague, confusing, or negatively-worded stems• Response option length• “All (none) of the above” options• Cues for testwise students such as grammatical cues, logical cues, “long-answer=right answer”, andconvergence• Construct irrelevant difficulty (look at the “TestTopics” for a guide)• Interlocking items
    53. 53. References• Brunnquell, A., Degirmenci, U., Kreil, S., Kornhuber, J., & Weih, M. (2011). Web-based application to eliminate five contraindicated multiple-choice question practices. Evaluation & the health professions, 34(2), 226–38. doi:10.1177/0163278710370459• Case, S., & Swanson, D. (1998). Constructing written test questions for the basic and clinical sciences. Retrieved from http://ibmi3.mf.uni-lj.si/mf/fakulteta/prenova/stomatologija/mcq.pdf• Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309–319.doi:10.1037/1040-3590.7.3.309• DeMars, C. (2010). Item response theory. Oxford University Press, USA.• De Ayala, R.J. (2009). The theory and practice of item response theory. Guilford Press, USA.• Downing, S., & Baranowski, R. (1995). Item type and cognitive ability measured: the validity evidence for multiple true-false items in medicalspecialty certification. Applied measurement …. Retrieved from http://www.tandfonline.com/doi/full/10.1207/s15324818ame0802_5• Downing, S. M. (2005).The effects of violating standard item writing principles on tests and students: the consequences of using flawed testitems on achievement examinations in medical education. Advances in health sciences education : theory and practice, 10(2), 133–43.doi:10.1007/s10459-004-4019-5• Fink, L.D. (2003) Creating Significant Learning Experiences: An Integrated Approach to Designing Courses. John Wiley & Sons, Inc, USA.• Jozefowicz, R. F., Koeppen, B. M.,Case, S.,Galbraith, R., Swanson, D., & Glew, R. H. (2002).The quality of in-house medical schoolexaminations. Academic medicine : journal of the Association of American Medical Colleges, 77(2), 156–61. Retrieved fromhttp://www.ncbi.nlm.nih.gov/pubmed/11841981• Suskie, L. (2009). Assessing Student Learning: A Common Sense Guide, 2nd Edition. John Wiley & Sons, Inc, USA.• Windish, D. M., Huot, S. J., & Green, M. L. (2007). Medicine Residents’ Understanding of the Biostatistics and Results in the MedicalLiterature. JAMA:The Journal of the American Medical Association, 298(9), 1010–1022. Retrieved from http://jama.ama-assn.org/content/298/9/1010.abstract

    ×