Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Oral exams Liz Norman ANZCVS 2019

1,077 views

Published on

Presentation for examiner training at the Australian and NZ College of Veterinary Scientists Examiners Workshop 9-10 February 2019

Published in: Education
  • You can ask here for a help. They helped me a lot an i`m highly satisfied with quality of work done. I can promise you 100% un-plagiarized text and good experts there. Use with pleasure! ⇒ www.HelpWriting.net ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • If you’re looking for a great essay service then you should check out ⇒ www.WritePaper.info ⇐. A friend of mine asked them to write a whole dissertation for him and he said it turned out great! Afterwards I also ordered an essay from them and I was very happy with the work I got too.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Oral exams Liz Norman ANZCVS 2019

  1. 1. Massey University | massey.ac.nz | 0800 MASSEY Oral exams Liz Norman Massey University
  2. 2. insight reliability Oral exams: strengths and limitations
  3. 3. insight reliability Oral exams: strengths and limitations • depth of knowledge • analytical and reasoning capabilities • cognitive processes • decision making abilities • oral communication
  4. 4. insight reliability Oral exams: strengths and limitations • depth of knowledge • analytical and reasoning capabilities • cognitive processes • decision making abilities • oral communication • few cases/topics sampled • lack of examiner agreement • assessing candidates on different cases/topics • assessing different numbers of cases/topics
  5. 5. insight reliability Oral exams: strengths and limitations • Ability to tailor questions and probing increases insight • Long cases allow you to explore depth • Standardisation of questions and probing increases reliability • More short cases increases reliability
  6. 6. insight reliability Oral exams: strengths and limitations • Ability to tailor questions and probing increases insight • Pre-prepared probes and marking schemes increases reliability
  7. 7. insight reliability Oral exams: strengths and limitations 1. Finding the right balance 2. Reducing examiner-examiner unreliability
  8. 8. Reducing examiner-examiner unreliability • Examiner training – Use of exemplars of different standard – Practice in assessment • Advance preparation of questions and prompts – Well-structured questions aimed at assessing appropriate skills – Clear and straightforward to answer • Advance preparation of marking schemes – Common picture of performance levels: Familiarisation, discussion, and agreement between examiners on the marking schemes – Well structured marking schemes which are easy to apply and well-understood by examiners
  9. 9. Reducing examiner-examiner unreliability • Allowing only preplanned questions and prompts – Planning for contingencies • Putting candidates at ease as much as possible • Reducing examiner load as much as possible • Monitor examiner agreement • Consensus decision making
  10. 10. Finding the right balance: blueprinting • Don’t aim to assess factual knowledge in oral exams – written exams are better for this • Aim to assess: – Analytical, problem solving, reasoning skills – Cognitive processes – Decision-making abilities – Judgement – Oral communication – Professionalism, values, ethics (if these are part of the subject guidelines)
  11. 11. Finding the right balance: questions • Make your questions short, straightforward and clear • Centre on clinical scenarios • Get straight to the point. – Remove unnecessary information – Consider even saying what the point is: “This question is going to be about (disease x) and I want to explore your approach to management.” • Keep Qs short - aim for new Q every 5 mins
  12. 12. Finding the right balance: questions • Make Qs open-ended – Require interpretation, problem solving, reasoning, judgement, or decisions – do not have one right answer • Do not ask for personal experience or opinion – Eg: “How is this usually managed in your practice?”, “What is your opinion on?” – It’s “unmarkable” content – Can throw candidates. Differential effect on those not “testwise”
  13. 13. Broad types of oral Qs Recall: Questions that ask candidates only to recall facts, principles, or other information. – List the most common causes of (x). – Define (x). – What is the most common disease associated with (x)? – Explain two mechanisms by which (x) results in (y). – Describe the differences between Type 1 and Type 2 (x). McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.
  14. 14. Broad types of oral Qs Recall: Questions that ask candidates only to recall facts, principles, or other information. Interpretive: Questions that require interpretation of information, and using principles to deal with new situations. – Describe the changes on this radiograph. – Interpret the results of this biochemistry panel. – Describe the abnormalities you see (in this image). McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.
  15. 15. Broad types of oral Qs Recall: Questions that ask candidates only to recall facts, principles, or other information. Interpretive: Questions that require interpretation of information, and using principles to deal with new situations. Problem solving: Questions that require finding a solution. – What is the list of differentials for these findings? – Outline you management of this case. – What would be the best next step in this case? – Can you account for the clinical findings in this case? McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.
  16. 16. Finding the right balance: prompts, probes • Plan them in advance but they can never be entirely standardised • Only plan a few – need to move on to next case/topic
  17. 17. Types of prompts and probes Exploring the depth of knowledge and higher order abilities – Can you explain your reasoning? – What are the implications of each of these options? – What is the justification for your decision? – What evidence supports your judgement in this case? – Explain how you arrived at that conclusion – Because? – Why is that? – What would you do if your approach does not work?
  18. 18. Types of prompts and probes Seeking more detail from the candidate – Any other differentials? – Can you tell me which specific drug you would use? – Which particular suture pattern would be most appropriate? – Which particular signs are leading you to conclude that? – Can you elaborate on that?
  19. 19. Types of prompts and probes Clarifying the question being asked or redirecting the candidate – What I am getting at here is… – Can I get you to focus on (x) in this case? – Can you tell me all the options rather than what you would do? – Just coming back to something you mentioned earlier about (x), can you tell me… Asking about things that the candidate has missed out – Can you tell me any other implications? What about implications for the whole farm? – One thing you didn’t mention was (x). Can you tell me about that?
  20. 20. Discourse Three different types of discourse in oral exams (Roberts, 2000) • Personal experience – talk that deals with personal experience and feelings and often involves narratives and stories • Professional – talk of professionals to clients and colleagues to colleagues • Institutional – principles: abstract, analytic, depersonalised. Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375
  21. 21. Discourse • Important to frame your question in the same discourse that you expect an answer • Usually you expect an answer in institutional discourse. – Candidates may not know this, especially those inexperienced in this type of exam, from ethnic minorities or overseas trained. – Causes candidate uncertainty and can result in differential bias in exam scores • But in oral exams its easy to slip into one or other of the other discourses Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375
  22. 22. The following question combines elements of both professional and personal experience. Question: If someone took out a personal complaint against you, how would you react? Answer: That's where my personal stress management plan comes in . . . . Although the theme of the question was professional and personal, the candidate responded in an institutional way. Instead of describing how she would feel (personal experience discourse) or what she did in her practice in relation to a specific case (professional discourse), she produced an abstract and analytical statement (institutional discourse) that satisfied the examiners as a response they could mark. Head nodding and mutual smiles provided confirmation. Quote from Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375
  23. 23. Ancillary materials • Need to have clear function • Use them purposefully, not as decoration • Ensure they clearly show what they are intended to show, when displayed in the way candidates will see them – Check with someone who did not see the case
  24. 24. Marking schemes • Rate on only a few criteria eg – Factual recall – Analysis and interpretation of data – Problem solving – Verbal presentation • Plus overall rating • Separately grade each main question/topic • Provide descriptors for 3-4 levels of performance eg: clear fail, marginal fail, bare pass, good pass
  25. 25. Conducting oral exams Create the right climate • Introductions and non-threatening questions (eg weather) • Convey a sense of genuine interest • Adapt body language to candidate’s needs – Some need eye contact, some find it off-putting – Candidate interpretations of “mhm” and “uhuh”: • Encouragement to go on • Stop talking so I can ask the next Q • Well done you have passed • Avoid humour
  26. 26. Not to much structure, not too little • Plan all the questions and possible prompts including how these should be worded • But on the day don’t read as if from a script • Adapt your language and body language to the candidate • Change the order or leave out probes if necessary • Use clarification and redirection only if necessary • Give all candidates the same information • Metacommunicate
  27. 27. Dealing with issues When the candidate can’t answer or says very little • Ensure candidate has had enough time to respond (but don’t give them endless time) • Do not give hints • Ask the candidate if they would like the question clarified. If they do, be careful not to – give hints or additional information – change the discourse or requirements of the answer • Often best just to move on to next question
  28. 28. Dealing with issues When candidates go off track or take too long • Use clarifying or redirecting questions – What I am getting at here is… – Can I get you to focus on (x) in this case? – Can you tell me all the options rather than what you would do? – Just coming back to something you mentioned earlier about (x), can you tell me…
  29. 29. Dealing with issues When candidates are exceedingly nervous • Trying to help often makes things worse • Poor performance may be the cause: move on to the next question
  30. 30. Scoring candidates • Record grade for each question as soon as it is finished • If you are thinking of giving quite a high grade think “What would the candidate have to have done to earn an even higher grade?” – helps examiners use the whole scale • At end of exam record your holistic judgement as well as sum/average of scores • Do not confer with your co-examiners until you have recorded your own judgement • Discuss your judgements and reach a consensus decision

×