Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

1 blueprinting

446 views

Published on

ANZCVS Examiners Workshop Feb 2016

Published in: Education
  • Be the first to comment

  • Be the first to like this

1 blueprinting

  1. 1. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting and drafting questions Liz Norman Massey University Sharanne Raidal, BVSc, PhD, FANZCVS, GradDipEd http://consultatutorblog.files.wordpress.com/2010/06/examstress.gif Australian and New Zealand College of Veterinary Scientists Examiner Workshop, 6 – 7th February 2016
  2. 2. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Overview • Blueprinting and drafting questions • Grading criteria and marking schemes • Clarity, timing; oral examinations http://www.slidehare.net/liznorman http://toonclips.com/600/4997.jpg
  3. 3. SCHOOL OF ANIMAL AND VETERINARY SCIENCES What is a blueprint? blueprint, n. something which acts as a plan model or template OED Online http://www.oed.com https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg
  4. 4. SCHOOL OF ANIMAL AND VETERINARY SCIENCES What is a blueprint? • Specifies what’ going to be in the exam  Content / topics (breadth)  Level (depth) https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg
  5. 5. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Purpose of blueprinting • To document sampling • To show that your examination is representative of all that could be examined and is sufficient Content domain Exam 2 Oral Essay Credentials domain Exam 1 Prac Practice domain
  6. 6. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Why blueprint? • To show that we have a representative sample • To permit extrapolation to content (practice) domain • To plan how each component assessed Content domain Exam 2 Oral Essay Credentials domain Exam 1 Prac Practice domain
  7. 7. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Content of questions • Scope of topics • Other relevant considerations - eg. species body systems • Scope of skills and abilities
  8. 8. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Subject guidelines • Represent the entire content domain (ie. they define all that is examinable) • Specify both scope (breadth) and level (depth) of the knowledge, skills, attitudes and judgements required • For Fellowship, some aspects are assessed in the credentials document (and process and training)
  9. 9. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Subject guidelines The candidate will have a detailed knowledge of: The aetiology, pathogenesis and pathophysiology of cardiac, renal, respiratory, alimentary, musculoskeletal, endocrine, ophthalmological and neurological organ dysfunction in the cat and the dog.
  10. 10. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Level of questions; skills & abilities required Level – depth  Levels of knowing and understanding Thinking processes required by the discipline •Surface vs deep •Fact recall vs application •Blooms taxonomy •SOLO taxonomy
  11. 11. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Level of questions; skills & abilities required Fact recall: Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring several facts). Applied (higher order): Questions that require the use of facts or concepts, the solution of a diagnostic of physiologic problem, the perception of a relationship or other process(es) beyond recalling discrete fact. From: Peitzman et al. (1990) Academic Medicine , 65(9): S59-60
  12. 12. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Level of questions; skills & abilities required Bloom’s taxonomy http://julietovar.edublogs.org/files/2011/05/blooms-taxonomy-1k4snjn.JPG
  13. 13. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Level of questions; skills & abilities required Bloom’s taxonomy http://www.psia-nw.org/wp-content/uploads/Blooms_Taxonomy.jpg Recall Application
  14. 14. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Level of questions; skills & abilities required Bloom’s instructional verbs Create: compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare. Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate. Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise. Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch. Understand: translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell. Remember: define, repeat, record, list, state, recall, name. Recall Apply
  15. 15. SCHOOL OF ANIMAL AND VETERINARY SCIENCES http://www.pescholar.com/wp-content/uploads/2013/03/Solo-SS.png
  16. 16. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Level of questions; skills & abilities required SOLO taxonomy http://litre.ncsu.edu/solowrittencomposite.gif
  17. 17. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Level of questions skills & abilities required SOLO taxonomy http://www.mbms.org.uk/faculties/science/assets/SOLO/solo- taxonomy-explained-full.jpg
  18. 18. SCHOOL OF ANIMAL AND VETERINARY SCIENCES SOLO taxonomy Prestructural Question may be rephrased at the answer; almost completely misses the point of the question Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking
  19. 19. SCHOOL OF ANIMAL AND VETERINARY SCIENCES SOLO taxonomy Prestructural Question may be rephrased at the answer; almost completely misses the point of the question Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking Quantitative change Qualitative change
  20. 20. SCHOOL OF ANIMAL AND VETERINARY SCIENCES SOLO taxonomy Prestructural Question may be rephrased at the answer; almost completely misses the point of the question Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking Deep Surface
  21. 21. SCHOOL OF ANIMAL AND VETERINARY SCIENCES SOLO taxonomy Prestructural Question may be rephrased at the answer; almost completely misses the point of the question Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking Application Recall
  22. 22. SCHOOL OF ANIMAL AND VETERINARY SCIENCES SOLO taxonomy Prestructural Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise Multistructural Combine, classify, describe, enumerate, list, do algorithm, apply method, account for, execute, formulate, solve, conduct, prove, complete, characterise Relational Analyse, compare, contrast, integrate, relate, explain causes, apply theory, argue, implement, plan, summarise, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt Extended abstract Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, critique, reason
  23. 23. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Level of questions; skills & abilities required Skills and abilities •What is contained in subject guidelines? •Skills beyond cognitive? •Recognition / interpretation? •Communication? - diagnostic images - cytology / pathology - reports - statistics
  24. 24. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting - process • Mapping to scope and level
  25. 25. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting - process • Breadth by learning outcome Written paper 1 Written paper 2 Practical examination Oral examination LO1 x x x LO2 x x LO3 x x x LO4 x
  26. 26. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting - process • Breadth by topic Patho- physiology Investigation & diagnosis Treatment & management Gastrointestinal P1Q1 P1Q1, P2Q4 Cardiovascular P1Q4 P2Q2 P2Q2 Nervous P1Q3, P2Q1 Endocrine P1Q3 P2Q3 Musculoskeletal P2Q5
  27. 27. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting - process • Breadth by topic and depth Patho- physiology Investigation & diagnosis Treatment & management recall higher recall higher recall higher Gastrointestinal P1Q1 P1Q1 P2Q4 Cardiovascular P1Q4 P2Q2 P2Q2 Nervous P1Q3, P2Q1 Endocrine P1Q3 P2Q3 Musculoskeletal P2Q5
  28. 28. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting - process • Breadth by species Species Number of Qs Percent Small animal 15 52% Farm animal 4 14% Horse 5 17% Exotic 3 10% Lab 1 3% All 1 3%
  29. 29. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting - process • Content coverage x depth Topic 1 30% Topic 2 30% Topic 3 40% Total Knowledge of terms 2 5 5 12 Comprehension of principles 4 3 4 11 Application of principles 3 3 3 9 Analysis of situations 3 2 5 10 Evaluation of situations 3 2 3 8 Total questions 15 15 20 50 http://scoring.msu.edu/written.html
  30. 30. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting Enables you to plan and check that your exam, or set of exams, covers the content (and skills) it is meant to cover, at an appropriate level. Without this you cannot extrapolate performance on this exam to performance in the whole domain. Validity
  31. 31. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Should questions be difficult? Norm-referenced vs standards-based grading
  32. 32. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Should questions be difficult? Appropriate demand vs irrelevant difficulty Practice domain Content domain
  33. 33. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand What makes questions difficult? •Familiarity / novelty •Complexity – components, links •Scale – components, links •Resources utilised (provided, generated) •Abstraction – concrete experience, concepts, principles •Task strategy – simple, stepwise, integrated •Guidance Adapted from Hughes et al (1998) The development of a tool for gauging demands of GCSE and A level exam questions. Brit Ed Res Assoc Conference.
  34. 34. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Familiarity / novelty Question 1 Explain the physiological actions of insulin. Question 2 Explain the physiological actions of ghrelin.
  35. 35. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Familiarity / novelty Question 1 List four (4) clinical signs of diabetes mellitus in dogs. Question 2 List four (4) clinical signs of feline hypersomatotropism.
  36. 36. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Complexity, scale – components, links Question 1 Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs. Question 2 Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.
  37. 37. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Complexity, scale – components, links Question 1 Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs and cats. Question 2 Compare and contrast the clinical signs of diabetes mellitus with those of hypoadrenocorticism in dogs and cats.
  38. 38. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Resources utilised – provided, generated Describe the skull. (30 marks)
  39. 39. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Abstraction – concrete experience, concepts, principles Discuss the concept of quality of life in terms of biological function, “feelings” and natural existence. Describe how both classical conditioning and operant conditioning are involved in cows confidently entering the milking shed and letting down.
  40. 40. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Task and response strategies – simple, stepwise, integrated Question 1 Discuss the use of insulin for the treatment of diabetes mellitus in cats. (25 marks)
  41. 41. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Task and response strategies – simple, stepwise, integrated Question 2 a)Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats. (10 marks) b)Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks) c)Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus. (10 marks)
  42. 42. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Task and response strategies – simple, stepwise, integrated Guidance Describe the skull. (30 marks)
  43. 43. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Task and response strategies – simple, stepwise, integrated Guidance You are presented with a 7 year old male Warmblood gelding with exercise intolerance. You auscultate an arrhythmia with a variable pulse and order an ECG. Define the type and give a specific name for the arrhythmia demonstrated in the rhythm strip below. (2 marks)
  44. 44. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Controlling difficulty and demand Task and response strategies – simple, stepwise, integrated Guidance You have been contracted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks) Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider, and detail how you might further investigate potential causes, and what advice you would provide.
  45. 45. SCHOOL OF ANIMAL AND VETERINARY SCIENCES What’s different about oral questions? • Thinking on your feet • Can’t revisit or edit • Tests verbal communication skills (as well as knowledge) • Interactive – can prompt and redirect, can ask candidate to explain their answer
  46. 46. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Key points
  47. 47. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting • Ensure the examination (as a whole) representatively samples from the domain specified for the examination • Membership – this is all the learning outcomes • Fellowship – this is all the learning outcomes, except those that can only be sampled in the credentials document
  48. 48. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Blueprinting • Ensure the examination (as a whole) representatively samples from the domain specified for the examination • Consider - topics - level - species, body systems, modalities - skills - other things (eg. Dx vs mgt, principles vs application)
  49. 49. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Structuring the exam • Ensure you follow subject guidelines explicitly • Shorter vs longer questions • Scenario vs direct questions • Difficulty, guidance
  50. 50. SCHOOL OF ANIMAL AND VETERINARY SCIENCES Next session’s task: • Check your blueprint - content - level • Classify your draft Qs according to SOLO taxonomy • Redraft Qs if necessary
  51. 51. SCHOOL OF ANIMAL AND VETERINARY SCIENCES SOLO taxonomy Prestructural Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise Multistructural Combine, classify, describe, enumerate, list, do algorithm, apply method, account for, execute, formulate, solve, conduct, prove, complete, characterise Relational Analyse, compare, contrast, integrate, relate, explain causes, apply theory, argue, implement, plan, summarise, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt Extended abstract Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, critique, reason

×