Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Clarity
and why it’s important
Liz Norman
Massey University
http://www.slidehare....
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
“Effective item writers are trained,
not born....”
Downing and Haladyna (2006) Ha...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Validity
Measuring what we
want to measure
and
Not measuring what we
don’t want t...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
What we don’t want to measure
• Ability to take tests
• Ability to write legibly ...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Why clarity is important
• We want the candidate to do the task we
envisaged, not...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Communication
The examination questions are the question setter’s
expression of t...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
General issues with question wording
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Expectations and stereotypes
Examples:
•“treatment”
•All differential diagnoses v...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Crisp et al (2008) Tales of the expected: the influence of students‘ expectations...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Discuss the diagnosis and management of a cat
with both chronic renal failure and...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Contextualising questions
• Context is good because it brings relevance and
authe...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Contextualising questions
Other disadvantages:
•More words used
•More intended / ...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
You are presented with a poorly controlled diabetic
dog. The dog is an 8 year old...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
List the possible causes of consistently high blood
glucose values in a dog being...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Question 1
A crate of 12 cans of cola costs $4.20. How much
do 7 crates of cola c...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Guidelines for using context
• If you are going to use context, use a natural,
re...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
a) After several weeks in a tropical
climate the body can adapt by
increasing max...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Don’t write questions, write tasks
What is your
diagnosis?
State the most likely ...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Instructional verb examples
Compare: to find similarities between things, or to l...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Specify boundaries of the answer
Species
eg. “in both dogs and cats....”
Quantiti...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
What are the
clinical signs of
hypothyroidism in
dogs and how do
they arise?
List...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Guidance
Question 1
Discuss the use of insulin for the treatment of diabetes mell...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Guidance
You have been contracted by a farmer producing Pacific oysters
(Crassost...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Purpose of marking schemes
•To help you during question writing
 What content is...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Wording of questions – examples of problems
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Name two (2) diagnostic tests you would run next
to investigate the cause of this...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Outline your approach to confirming the initial
clinical diagnosis and a manageme...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
A veterinarian asks you for assistance in designing
a protocol for the delivery o...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Are there any clinical features which can help you
determine this patient’s progn...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
State what you believe is your most likely
diagnosis.
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Discuss commonly found tumours and tumour-like
disorders associated with the oral...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
How would you localise the site of the lesion?
Answer provided in marking scheme:...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Write notes on
a) Considerations in the selection of
stockpersons and animal atte...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Briefly outline your interpretation of results. Do not
just state the abnormaliti...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Sentence-level clarity
• Simple sentence structures
• Grammatically correct
• Rem...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Question timing
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
How long will it take the candidate?
• Unstructured tasks – open ended
• Need to ...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Reading rates reported by 104
medical students at the Mercer University
School of...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Poppy is a 9 year old 12 kg female neutered fox terrier cross who has been on
tre...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
A client rings you up to arrange his quarterly visit as he is frustrated by ongoi...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Handwriting speed for three writing tasks
Writing task
Handwriting speed (wpm)
Me...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Effect of time stress
Time stress
•Increases the use of schemas (stereotyping)
•D...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Oral (viva voce) Examinations
Advantages
•A student’s [practitioner’s] ability to...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Oral (viva voce) Examinations
Disadvantages
•Scarey! [may be influenced by person...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Oral Examinations – Design and Delivery
• Identify question set according to Subj...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Oral Examinations – Design and Delivery
•Prepare a marking scheme (as for written...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Conditions under which
Oral Examinations are valid, reliable and fair
• Ensure it...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Key points
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Written papers - checking wording
• Check the question asks for what you are
rewa...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Written papers - checking timing
• Allow reading time of 70 words per minute (onl...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Oral Examinations
• Included within blueprinting process
• Check quality of image...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Next session’s task:
• Finalise your question wording
• Print a copy of your Qs a...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
End of the day
• Discussion on progress and challenges
• Feedback loops between e...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Question / topic:
Criterion
Unsatisfactory Novice Proficient Weighting
0 1 2 3 4 ...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Oral Examinations
•Mirrors oral communications typical of
professional practice
•...
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Oral Examinations
BUT
•Resource intensive
•Judgements based on limited evidence
•...
Upcoming SlideShare
Loading in …5
×

3 clarity

270 views

Published on

ANZCVS Examiners Workshop Feb 2016

Published in: Education
  • Be the first to comment

  • Be the first to like this

3 clarity

  1. 1. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Clarity and why it’s important Liz Norman Massey University http://www.slidehare.net/liznorman Sharanne Raidal, BVSc, PhD, FANZCVS, GradDipEd Australian and New Zealand College of Veterinary Scientists Examiner Workshop, 6 – 7th February 2016
  2. 2. SCHOOL OF ANIMAL AND VETERINARY SCIENCES “Effective item writers are trained, not born....” Downing and Haladyna (2006) Handbook of Test Development, page 11
  3. 3. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Validity Measuring what we want to measure and Not measuring what we don’t want to measure Content domain Exam 2 Oral Credentials domain Exam 1 Prac Practice domain
  4. 4. SCHOOL OF ANIMAL AND VETERINARY SCIENCES What we don’t want to measure • Ability to take tests • Ability to write legibly and fast • Ability to rote learn whole pages of textbooks or review articles – prewriting • Ability to write down a huge series of unconnected facts in no particular order • Ability to research examiner’s fields of interest and rote learn impressive aspects of that • Ability to interpret what examiners are thinking
  5. 5. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Why clarity is important • We want the candidate to do the task we envisaged, not something else • We want to minimise irrelevant difficulty • Validity – measuring what we intend to measure
  6. 6. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Communication The examination questions are the question setter’s expression of the question setter’s task. The candidate’s answer represents the candidate’s expression of the candidate’s interpretation of the questions. The marker evaluates the marker’s interpretation of the candidate’s expression of the candidate’s answer. The marker uses the marker’s interpretation of the setter’s expression of the setter’s task to evaluate the candidate’s answer. Modified from Pollitt & Ahmed (1999) New Model of the Question Answering Process. IAEA. Bled. Slovenia
  7. 7. SCHOOL OF ANIMAL AND VETERINARY SCIENCES General issues with question wording
  8. 8. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Expectations and stereotypes Examples: •“treatment” •All differential diagnoses vs those only applicable in a particular case •Expectation that Qs will ask about what something is rather than what it is not •Expectation of hard questions
  9. 9. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Crisp et al (2008) Tales of the expected: the influence of students‘ expectations on question validity and implications for writing exam questions. Ed Res 50(1): 95-115.
  10. 10. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Discuss the diagnosis and management of a cat with both chronic renal failure and hyperthyroidism. (24 marks)
  11. 11. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Contextualising questions • Context is good because it brings relevance and authenticity • Allows assessment of concrete or specific examples not abstract concepts or generalisations • Allows assessment of applied learning (doing not just knowing) • All these carry with them a potential for bias
  12. 12. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Contextualising questions Other disadvantages: •More words used •More intended / unintended demand •Familiarity – schemas / stereotypes •Focus may direct candidates to the wrong aspects •Images can be particularly distracting
  13. 13. SCHOOL OF ANIMAL AND VETERINARY SCIENCES You are presented with a poorly controlled diabetic dog. The dog is an 8 year old female spayed Labrador weighing 36 kg with a body condition score of 7/9. The dog has been on porcine lente insulin (Caninsulin) twice daily for 3 months. Various doses have been tried during this time and the dog is currently receiving 70 units twice daily. The dog is still polyuric and polydipsic, with 3+ to 4+ glucosuria. Discuss the possible causes of poor control which you should consider when evaluating this dog.
  14. 14. SCHOOL OF ANIMAL AND VETERINARY SCIENCES List the possible causes of consistently high blood glucose values in a dog being treated with subcutaneous insulin injections for the treatment of diabetes mellitus. Outline your approach to investigating these possible causes.
  15. 15. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Question 1 A crate of 12 cans of cola costs $4.20. How much do 7 crates of cola cost? Question 2 A ski pass costs $42.0. How much would it cost for 7 days.
  16. 16. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
  17. 17. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Guidelines for using context • If you are going to use context, use a natural, realistic one • Contextualise purposefully for what it brings to the task • Only use images when the question could not be asked without them • Don’t decorate!
  18. 18. SCHOOL OF ANIMAL AND VETERINARY SCIENCES a) After several weeks in a tropical climate the body can adapt by increasing maximum swat rate to 3.5 L/h. at what rate, in watts, is energy being removed when sweating that rapidly. Assume all of the sweat evaporates. (3 marks) b) If a 50 kg girl stopped seating because of severe dehydration, how long would it take for her body to rise to a dangerous level of 40o C if originally she was sweating at a rate of 3.5 L/h to maintain a body temperature of 36o C? (2 marks)
  19. 19. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Don’t write questions, write tasks What is your diagnosis? State the most likely diagnosis State the most likely diagnosis and explain your reasoning. Discuss the differential diagnoses you would consider in this case.
  20. 20. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Instructional verb examples Compare: to find similarities between things, or to look for characteristics and features that resemble each other. Contrast: to find differences or to distinguish between things. Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject. Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something. Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes. Illustrate: to use a picture, diagram or example to clarify a point.
  21. 21. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Specify boundaries of the answer Species eg. “in both dogs and cats....” Quantities and amounts eg. “Provide 5 reasons why.....” With reference to eg. “With reference to published research from...” Time eg. “in the first 24 hours” Part of the question eg. “for one of your differentials....”
  22. 22. SCHOOL OF ANIMAL AND VETERINARY SCIENCES What are the clinical signs of hypothyroidism in dogs and how do they arise? List the three most common owner- observed clinical signs of hypothyroidism in dogs and explain how thyroid hormone deficiency leads to each of these signs.
  23. 23. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Guidance Question 1 Discuss the use of insulin for the treatment of diabetes mellitus in cats. (25 marks) Question 2 a) Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats. (10 marks) b) Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks) c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus. (10 marks)
  24. 24. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Guidance You have been contracted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks) Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider, and detail how you might further investigate potential causes, and what advice you would provide.
  25. 25. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Purpose of marking schemes •To help you during question writing  What content is important  Whether the Q asks what you intended it to ask  Whether it is do-able in the time available •To help you during question marking  Helps you decide whether an answer is good enough to be awarded a mark  Facilitates reliable and fair marking
  26. 26. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Wording of questions – examples of problems
  27. 27. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.
  28. 28. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Outline your approach to confirming the initial clinical diagnosis and a management and prevention plan for this problem. This discussion should include an outline on further observations taken about .....
  29. 29. SCHOOL OF ANIMAL AND VETERINARY SCIENCES A veterinarian asks you for assistance in designing a protocol for the delivery of a vaccine for cats in their practice. What factors would you take into consideration in designing this protocol?
  30. 30. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Are there any clinical features which can help you determine this patient’s prognosis?
  31. 31. SCHOOL OF ANIMAL AND VETERINARY SCIENCES State what you believe is your most likely diagnosis.
  32. 32. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Discuss commonly found tumours and tumour-like disorders associated with the oral cavity and dental tissues of the horse.
  33. 33. SCHOOL OF ANIMAL AND VETERINARY SCIENCES How would you localise the site of the lesion? Answer provided in marking scheme: Spinal lesion between T3 and L3
  34. 34. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Write notes on a) Considerations in the selection of stockpersons and animal attendants
  35. 35. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Briefly outline your interpretation of results. Do not just state the abnormalities.
  36. 36. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Sentence-level clarity • Simple sentence structures • Grammatically correct • Remove superfluous words
  37. 37. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Question timing
  38. 38. SCHOOL OF ANIMAL AND VETERINARY SCIENCES How long will it take the candidate? • Unstructured tasks – open ended • Need to control the demands of the question carefully • Question reading time • Writing speed
  39. 39. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Reading rates reported by 104 medical students at the Mercer University School of Medicine, 2010 Number (%) of students Number of pages read per hour Reading rate (WPM) 13 (13%) ≤ 6 ≤ 50 55 (53%) 7 – 12 51 - 100 18 (17%) 13 - 18 101 - 150 16 (15%) 19 - 24 151 - 200 2 (2%) ≥ 25 ≥ 201 Klatt and Klatt (2011) Acad Med 86: 1079 - 1083
  40. 40. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Poppy is a 9 year old 12 kg female neutered fox terrier cross who has been on treatment for diabetes mellitus with Caninsulin (porcine lente insulin) for 4 months. She is currently receiving 5 units subcutaneously at 7:30am and 5 units at 7:30pm. At the time of each injection she is fed a mixture of ‘Optimum’ dog roll and Pedigree Pal Meaty Bites. Her last visit was 2 weeks ago when her dose of insulin was decreased by 2 units. Since the last visit Poppy has been happy and active and is eating all her food at mealtimes. Her body weight has not changed since the last visit and her body condition score is 5/9. Her owner reports her to be drinking about 600 mL of water daily. Her owner is very happy with Poppy’s progress and comments that she is her normal self again. There have been no signs of hypoglycaemia. You detect no abnormalities on physical examination and admit Poppy to perform a serial blood glucose curve. Her owner has administered the morning insulin dose and fed her as usual. You have been provided with the results obtained. Based on your assessment of the clinical information and the glucose curve shown, what dose of insulin would you recommend Poppy be given now?
  41. 41. SCHOOL OF ANIMAL AND VETERINARY SCIENCES A client rings you up to arrange his quarterly visit as he is frustrated by ongoing problems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also ha d a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also thinks she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear. On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present. You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)
  42. 42. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Handwriting speed for three writing tasks Writing task Handwriting speed (wpm) Mean SD Range Handwriting speed test (3 min writing test) 37.83 4.71 27.0 – 51.7 Exam 1 (2 h exam with prior knowledge of questions) 17.75 2.78 11.8 – 25.7 Exam 2 (2 h exam without prior knowledge of questions) 15.37 3.12 9.4 – 23.2 Summers & Catarro (2003) Australian Occupational Therapy Journal 50(3): 148 - 157
  43. 43. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Effect of time stress Time stress •Increases the use of schemas (stereotyping) •Decreases working memory processing capacity •Decreases the ability to maintain relevant information and suppress irrelevant information
  44. 44. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Oral (viva voce) Examinations Advantages •A student’s [practitioner’s] ability to verbally explain / communicate on a subject is a good measure of their understanding •Provide a measure of candidate’s verbal communication skills •Use of different (visual) cues •Alternative assessment technique - interactive •May provide a more authentic assessment of ‘real world’ skills – thinking on feet, progressive disclosure, no ‘cues’ in question •Rapid marking turnaround
  45. 45. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Oral (viva voce) Examinations Disadvantages •Scarey! [may be influenced by personality] •Shorter time period – may limit the amount of material that can be covered relative to a written examination •Consistency may be hard to achieve •More subjective? •Harder to ‘re-grade’ / defend [College procedures] •Delivered asynchronously (confidentiality / security issues)
  46. 46. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Oral Examinations – Design and Delivery • Identify question set according to Subject Guidelines (for each day examined) • Situate oral exam content within your blueprint • Determine an appropriate way to deliver questions and cues (images, data, other) • Decide appropriate questions – more colloquial delivery than written papers, but aim for consistency • Identify / predict need for additional cues  Avoid prompting / coaching candidates during exam
  47. 47. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Oral Examinations – Design and Delivery •Prepare a marking scheme (as for written / prac) and considering qualitative and quantitative (content) aspects of potential answers •Ensure images (or other props) are of adequate quality •Be able to respond to problems during exam delivery - candidate nerves - problems with delivery / marking •Consider what you will say and do during the exam (be prepared)
  48. 48. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Conditions under which Oral Examinations are valid, reliable and fair • Ensure items focus on capabilities required for professional practice (decision making, reasoning) • Appropriate content (defined by peers, profession) • Exam items are within the scope of professional practice, at an appropriate level • Adequate sampling of questions • Examiner training (issues and methods) • Inter-examiner variation monitored, addressed • Consider potential for bias (pre- and post-exam) Memon et al (2010) Oral Assessment and Postgraduate Medical Examinations: Establishing Conditions for Validity, Reliability and Fairness. Adv in Health Sci Educ 15(2): 277-289
  49. 49. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Key points
  50. 50. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Written papers - checking wording • Check the question asks for what you are rewarding in your marking scheme • Ensure the wording gives an instruction • Ensure the scope is clear – may need to be specified • Consider how the phrasing of the Q may distract or misdirect candidates • Simplify wording if necessary • Check grammar and punctuation
  51. 51. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Written papers - checking timing • Allow reading time of 70 words per minute (only necessary to check unusually wordy examinations) • Allow writing speed of no more than 20 words per minute (pref 15 words per minute)
  52. 52. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Oral Examinations • Included within blueprinting process • Check quality of images / prompts • Ensure clarity of your questions and marking scheme (as for writtens) • Consider delivery (and responses) to candidates • Evaluate marking scheme • CONSISTENCY
  53. 53. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
  54. 54. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Next session’s task: • Finalise your question wording • Print a copy of your Qs and marking schemes • Exchange your exam with someone from another discipline • Give constructive feedback
  55. 55. SCHOOL OF ANIMAL AND VETERINARY SCIENCES End of the day • Discussion on progress and challenges • Feedback loops between examiners and chapters • How best for the BoE to support examiners • How best to administratively support examiners
  56. 56. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Question / topic: Criterion Unsatisfactory Novice Proficient Weighting 0 1 2 3 4 5 Domain knowledge Little or no awareness of relevant information; frequent factual errors or omission of key information Able to recall sufficient factual information relevant to topic; some errors or key information omitted Well informed, excellent recall of relevant information; few or no errors, all/most key info covered X2 Application of info Little or no understanding of topic; information presented randomly and with limited application to case scenario Satisfactory understanding of topic, limited or simplistic application to case scenario Good to excellent understanding of topic; information intelligently applied to case scenario X2 Use of veterinary terminology Unable to use, or limited use of, correct terminology Able to use correct terminology, some errors in pronunciation Fluent use of appropriate terminology X1 Holistic impression Examiner is uncomfortable with student’s preparedness for professional practice; major deficiencies identified on > 1 occasion Examiner is satisfied that student can operate safely in a supportive environment Examiner is impressed with student’s abilities; demonstrated knowledge, critical thinking and communication ability suggest advanced readiness for professional practice Comments
  57. 57. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Oral Examinations •Mirrors oral communications typical of professional practice •Can test limits of knowledge and understanding (weak and strong candidates) •Good test of: • clinical reasoning (the cognitive processes which constitute professional thinking) • Interpersonal competence • Intra- personal qualities (confidence, self- awareness) Memon et al (2010) Oral Assessment and Postgraduate Medical Examinations: Establishing Conditions for Validity, Reliability and Fairness. Adv in Health Sci Educ 15(2): 277-289
  58. 58. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Oral Examinations BUT •Resource intensive •Judgements based on limited evidence •May be difficult to defend on appeal •Unfamiliar (potentially difficult) for both examiners and candidate •Can be difficult to distinguish between what the candidate says and how they say it (discourse) •Performance limiting anxiety Memon et al (2010) Oral Assessment and Postgraduate Medical Examinations: Establishing Conditions for Validity, Reliability and Fairness. Adv in Health Sci Educ 15(2): 277-289

×