Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Question clarity and why its important Liz Norman ANZCVS 2019


Published on

Presentation for examiner training at the Australian and NZ College of Veterinary Scientists Examiners Workshop 9-10 February 2019

Published in: Education
  • Have you ever used the help of ⇒ ⇐? They can help you with any type of writing - from personal statement to research paper. Due to this service you'll save your time and get an essay without plagiarism.
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Question clarity and why its important Liz Norman ANZCVS 2019

  1. 1. Massey University | | 0800 MASSEY Clarity and why it’s important Liz Norman Massey University
  2. 2. Why clarity is important • We want the candidate to do the task we envisaged, not something else – Validity – measuring what we intend to measure • We want to minimise irrelevant difficulty – Validity – measuring what we intend to measure
  3. 3. Thinking about thinking
  4. 4. Schemas and sterotypes • All of us develop schemas or stereotypes • Help us categorise and process complex information quickly • Particular features of questions trigger certain schemas and hence expectations. • Triggered without us being aware of it and before the reading of the text reaches consciousness • Affect our interpretation of the question.
  5. 5. Thinking about thinking • When candidates read a question, they must form a mental model of the task required and begin to plan their response. • They begin to form a model before they have even finished reading the question. • The model they form is influenced by their expectations and their pre-existing mental schemas and sterotypes
  6. 6. Elephant Riding Jan Kemp Climbing up the back of an elephant you spring into the toehold of its tail held in place by the mahout grab the ropes strapped round its belly & haul yourself up. She rises from buckled knees under you moves like a ship you’re high under the hanging ashoka leaves as you flow forward her fly-bitten ears grey sails flap. she flings the odd young-leaved branch into her mouth with her triumphant trunk.
  7. 7. Effects of anxiety and time stress on thinking • Decreased working memory • Decreased processing capacity • Use of type 1 processing increases – Automatic thinking – Use of schemas and sterotypes • Lowered ability to concentrate on relevant information and suppress irrelevant information Can lead to misunderstanding of the intent of the question and the task asked of them
  8. 8. Implications for question writing • There may be problems with questions that you can’t see – your schemas are different • The text at the beginning of a question will have more influence on the schema elicited than the text you place at the end. • Images have more influence on the expectations developed than text
  9. 9. Implications for question writing • Anxiety and time stress increase schema use – We might be just measuring propensity to anxiety or writing speed • Each candidate will have different schemas – We might be just measuring cultural backgrounds or experience with examinations • Problems with questions may not affect all candidates equally and therefore may be hard to detect.
  10. 10. Contextualising Qs • Context is good because it brings relevance and authenticity • Allows assessment of concrete or specific examples not abstract concepts or generalisations • Allows assessment of applied learning (doing not just knowing) All these carry with them a potential for bias.
  11. 11. Contextualising Qs Potential problems: • More words used • Use of colloquial expressions or culturally-specific terms • Differential familiarity • Requirement for sifting • May activate schemas that interfere with thinking
  12. 12. Ahmed&Pollitt(2007)Improvingthequalityofcontextualizedquestions:Anexperimentalinvestigation offocus.AssessmentinEducation:Principles,Policy&Practice,14(2),201-232.
  13. 13. Ahmed&Pollitt(2007)Improvingthequalityofcontextualizedquestions:Anexperimentalinvestigation offocus.AssessmentinEducation:Principles,Policy&Practice,14(2),201-232.
  14. 14. Focussed contextualisation • Contextualisation works best when well focussed. • The most salient aspects of the context are also the main issues addressed in the question. • Makes Qs more likely to elicit helpful schemas than unhelpful ones, in candidates with relevant knowledge. • Does not make Qs easier for candidates that lack relevant knowledge. Ahmed, A., & Pollitt, A. (2007). Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy and Practice, 14(2), 201- 232.
  15. 15. Recommendations for maximising clarity • Use clear language that is as simple as possible while still being precise • Contextualise purposefully for what it brings to the task • Avoid contradicting expectations or switching schemas part way through a question
  16. 16. Wording of questions – examples of problems
  17. 17. Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.
  18. 18. Outline your approach to confirming the initial clinical diagnosis and a management and prevention plan for this problem. This discussion should include an outline on further observations taken about ….
  19. 19. A veterinarian asks your for assistance in designing a protocol for the delivery of a vaccine for cats in their practice. What factors would you take into consideration in designing this protocol?
  20. 20. Are there any clinical features which can help you determine this patient’s prognosis?
  21. 21. State what you believe is your most likely diagnosis.
  22. 22. Discuss commonly found tumours and tumour- like disorders associated with the oral cavity and dental tissues of the horse.
  23. 23. How would you localise the site of the lesion? Answer provided in the marking scheme: Spinal lesion between T3 and L3
  24. 24. Write notes on a) considerations in the selection of stockpersons and animal attendants
  25. 25. Describe and give reasons for your further clinical examination (if any), and recommendations to the owner/trainer for diagnostics and treatment. Explain what you think is really important and why. Explain if you think there is any controversy in treatment plans.
  26. 26. Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)
  27. 27. Which of the following statements is TRUE? a. A ventral hernia is classified as a false hernia b. A diaphragmatic hernia is classified as a true hernia c. Umbilical hernias are classified as false hernias d. False hernias have a hernia sac e. Umbilical hernias are relatively uncommon
  28. 28. Briefly outline your interpretation of the laboratory results. Do not just state the abnormalities.
  29. 29. Discuss your approach to the management of large colon volvulus.
  30. 30. A client rings you up to arrange his quarterly visit as he is frustrated by ongoing problems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also had a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also think she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear. On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present. You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)
  31. 31. Example from a job interview – manager position Relationships with other managers and service leaders as well as staff members from other units is vital to successful performance in this role. Tell us how you have developed an effective working relationship with an influential person within one of your roles.
  32. 32. Steps for checking clarity • Ensure the wording gives an instruction. • Ensure that the question scope is clear. Specify it if necessary. • Check the question asks for what you are rewarding in your marking scheme—all of it. • Check that understanding the task does not rely on information that might not be common knowledge in different parts of the world. • Consider how the phrasing of the question may distract or misdirect candidates and rephrase if necessary. • Use the simplest language possible while still being precise. This includes word choice and sentence structure. Remove all unnecessary wording. • Check that the discourse of the question mirrors the discourse of the response you require. • Check spelling, grammar, and punctuation. Explain all abbreviations unless knowledge of them is considered part of the curriculum.
  33. 33. “Effective item writers are trained, not born … “ Downing and Haladyna 2006, Handbook of test development ,p. 11