Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Blueprinting and drafting questions Liz Norman ANZCVS 2019

1,105 views

Published on

Presentation for examiner training at the Australian and NZ College of Veterinary Scientists Examiners Workshop 9-10 February 2019

Published in: Education
  • Have u ever tried external professional writing services like ⇒ www.HelpWriting.net ⇐ ? I did and I am more than satisfied.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Grab 5 Free Shed Plans Now! Download 5 Full-Blown Shed Plans with Step-By-Step Instructions & Easy To Follow Blueprints! ☞☞☞ https://url.cn/I86oXShh
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I purchased your e-book, How to Reverse Diabetes. This is really an amazing life-saving book, it's a blessing beyond words. I've downloaded your program because my dad has been suffering from diabetes for a very long time but now, thanks to your help, his blood sugar level is well within normal range and all his symptoms are gone. I cannot explain in words how much this book has meant to me and my family. I just want to let you know that your program has totally changed my life; you truly are a blessing for humanity. People need to hear your message because it can save many many people’s lives. The world needs more people like you. From the bottom of my heart, thank you thank you thank you! ★★★ https://bit.ly/2swQ6OO
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I have recently read your book and must congratulate you on the most informative and revolutionary contents. Your book has drastically changed my life and the way I view health. I am so grateful for your discovery of this information and for sharing it with the world. ★★★ http://t.cn/A6vI6Tyi
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Blueprinting and drafting questions Liz Norman ANZCVS 2019

  1. 1. Massey University | massey.ac.nz | 0800 MASSEY Blueprinting and drafting questions Liz Norman Massey University
  2. 2. What is a blueprint? "blueprint, n.". Something which acts as a plan, model, or template OED Online http://www.oed.com Railway and harbour report, Vancouver, B.C. by City of Vancouver Archives, Attribution License
  3. 3. Why we need to blueprint Its important that we sample representatively from the content domain (all that it is possible to examine) This is so we can generalise performance to the whole content domain
  4. 4. Constructing blueprints • Table or grid with at least 2 dimensions • Dimensions could include: – Learning outcomes – Diseases or presenting problem – Body system – Species – Knowledge domain: anatomy, pathophysiology, diagnosis, management, prognosis, epidemiology, investigations, ethics & law – Skills: interpretation, reasoning, decision making, evaluation, problem solving – Cognitive level: factual recall, analysis and interpretation of data, problem solving – Degree of abstraction: principles (theoretical), applied (concrete) – Location of question in the examination components
  5. 5. Constructing blueprints • Begin by considering the curriculum • What knowledge, skills, and attitudes are examinable? • Subject guidelines
  6. 6. Body system by knowledge domain Written paper 1 Written paper 2 Oral examination Practical examination LO1 X X X LO2 X X LO3 X X X LO4 X
  7. 7. Body system by knowledge domain by exam component Pathophysiology Investigation and diagnosis Treatment and management Gastrointestinal P1Q1 P1Q1, P2Q4 OQ3 Cardiovascular P1Q4 P2Q2, OQ1 P2Q2 Nervous P1Q3, P2Q1 Endocrine P1Q3 OQ2 P2Q3 Respiratory P2Q5
  8. 8. Proportion of exam by species species number of Qs percent small animal 15 52% farm 5 17% horse 4 14% exotic 3 10% lab 1 3% all 1 3%
  9. 9. Body system by knowledge domain by cognitive level Pathophysiology Investigation and diagnosis Treatment and management recall analyse, interpret problem solve recall analyse, interpret problem solve recall analyse, interpret problem solve Gastrointes tinal P1Q1 P1Q1 P2Q4 OQ3 Cardiovasc ular P1Q4 P2Q2 OQ1 P2Q2 Nervous P1Q2 P2Q1 Endocrine P1Q3 OQ2 P2Q3 Respiratory P2Q5
  10. 10. Body system by knowledge domain by subject of question Written paper 2 Pathophysiology Investigation and diagnosis Treatment and management Gastrointestinal Investigation of exocrine pancreatic insufficiency Cardiovascular Investigation of atrial fibrillation Management of cardiomyopathy Nervous Diagnosis of L2-3 spinal neoplasia Endocrine Management of diabetic ketoacidosis Respiratory Diagnosis of feline asthma Management of feline asthma
  11. 11. Weighting questions Impact Weight Frequency Weight Less important 1 Rarely seen 1 Essential 2 Relatively common 2 High impact 3 Very common 3 McLaughlin, K., Lemaire, J., & Coderre, S. (2005). Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach, 27(6), 544-547
  12. 12. Drafting questions
  13. 13. Creating appropriate tasks • Questions are actually tasks • We need the tasks to be things that Members or Fellows need to be able to do. • This ensures we can validly extrapolate from performance on the exam, to performance in the Membership or Fellowship domain.
  14. 14. Creating appropriate tasks • Emphasis should be on assessing the candidate’s – understanding – skills in analysing and interpreting information – skills in problem solving – judgement • Not recall of information
  15. 15. Recall-higher order classification Fact recall Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts) Applied (higher order) Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.
  16. 16. Webb’s depth of knowledge Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7-25. doi:10.1080/08957340709336728. recall Recall information or perform a simple step, identify, measure, describe, explain simple ideas skill/concept Requires more than one step, comparing, interpreting, estimating, making observations, explaining, organising and displaying data strategic thinking Requires planning and using evidence, explaining reasoning, make conjectures, draw conclusions, solving problems extended thinking Complex reasoning, planning developing and thinking over an extended period of time, making multiple connections, synthesis of ideas into new concepts
  17. 17. Knowledge-understanding classification Knowledge knowing about; a body of coherent facts; can be thought of as right or wrong. Understanding knowing how and why; the meaning of facts; the theory that links facts and provides meaning; how sense is made of facts to enable them to be applied to analysis, synthesis, evaluation; to be able to explain why particular facts or skills are applicable to a particular situation; to know which fact to apply when; to be able to create new knowledge or modify or adapt an idea to a new situation. Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and Curriculum Development.
  18. 18. SOLO taxonomy Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). New York: Academic Press.
  19. 19. Pre-structural Uni-structural Multi-structural Relational Extended abstract
  20. 20. Pre-structural Uni-structural Multi-structural Relational Extended abstract Quantitative change Qualitative change
  21. 21. Pre-structural Uni-structural Multi-structural Relational Extended abstract knowledge understanding
  22. 22. Pre-structural Uni-structural Multi-structural Relational Extended abstract surface deep
  23. 23. Pre-structural Uni-structural Multi-structural Relational Extended abstract recall application
  24. 24. Pre-structural Uni-structural Multi-structural Relational Extended abstract recall skill/concept strategic thinking extended thinking
  25. 25. Lack of knowledge or common knowledge only. Answer addresses a single aspect or a few aspects. Oversimplified, reductionist. Pre-structural Uni-structural Multi-structural Relational Extended abstract
  26. 26. Multi structural Signs to look for in answers (not all need to be present) • Multiple aspects considered and may be comprehensive • No particular order to aspects presented, or serial structure “he said, she said” • Inclusion of irrelevant/less important material • Lacks integration such as causal explanation or compare and contrasting • Replication of material from sources – rote learned or reproduced without significant transformation
  27. 27. Relational Signs to look for in answers (not all need to be present) • Aspects explained relative to one another • Logically organised answer • Analysis and or synthesis • Compares similarities and differences • Integrates multiple levels (eg: molecular, biochemical, systemic) • Expresses reasons, explains implications, or reaches a conclusion • Expresses relative importance, value, significance of aspects. • Selective answer that addresses the point of the question and may be shorter than a multistructural answer • Uses the language of the discipline - terminology and phrasing • Relates answer to examples or experience • Relates answer to organising principles of the discipline • Evaluates inconsistencies
  28. 28. Lack of knowledge or common knowledge only. Answer addresses a single aspect or a few aspects. Oversimplified, reductionist. Answer addresses multiple aspects and may well be comprehensive. Connections simple or lacking. Answer addresses multiple aspects and also how they integrate and inter-relate. Makes connections between aspects. Knows their relative importance/value/significance. Goes beyond a relational answer within and beyond a domain, including to areas not experienced or only imagined. Has more originality, creativity, meta- connections, and utilisation of overarching principles. Pre-structural Uni-structural Multi-structural Relational Extended abstract
  29. 29. Question writing
  30. 30. Specifying tasks – give an instruction Compare: to find similarities between things, or to look for characteristics and features that resemble each other. Contrast: to find differences or to distinguish between things. Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject. Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something. Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes. Illustrate: to use a picture, diagram or example to clarify a point.
  31. 31. Specifying scope • Do you want a general answer about this condition that applies to all possibilities or one that specifically applies to this case? • Do you want an answer to include – all possible options? – those that are currently available in this region of the world? – those that are available under cost constraints? • Writing marking scheme helps you get the scope right
  32. 32. Specify boundaries of the answer • Species e.g. “in both dogs and cats…” • Quantities and amounts e.g. “Provide 5 reasons why…” • With reference to e.g. “ With reference to the published research from ..” • Time eg: “in the first 24 hours” • Part of the question Eg: “for one of your differentials….”
  33. 33. Construct irrelevance When the skills, abilities or knowledge required to answer examination questions is not part of the domain we are trying to assess. What are we actually assessing?
  34. 34. Important sources of construct irrelevance • Unclear tasks – because the candidate may do something other than what we intended them to do. • Unanticipated and irrelevant difficulty and demand • Insufficient time to complete the task (fully)
  35. 35. Controlling task difficulty and demand • Tasks need to be at an appropriate level of difficulty • They don’t need to be “tricky” in order to be difficult
  36. 36. Some factors that affect demand and difficulty • Type of operation • Degree of novelty • Number of components or ideas involved • Whether resources are provided or need to be generated by the candidate • The question wording and any images, diagrams, or tables provided • Degree of abstraction • Response strategy – simple, stepwise, integrated • Guidance provided
  37. 37. Some factors that affect demand and difficulty • Type of operation – Simple steps vs synthesis, interpretation, evaluation
  38. 38. Some factors that affect demand and difficulty • Type of operation • Degree of novelty • Number of components or ideas involved • Whether resources are provided or need to be generated by the candidate • The question wording and any images, diagrams, or tables provided • Degree of abstraction • Response strategy – simple, stepwise, integrated • Guidance provided
  39. 39. Question 1: Explain the physiological actions of insulin. Question 2: Explain the physiological actions of ghrelin.
  40. 40. Some factors that affect demand and difficulty • Type of operation • Degree of novelty • Number of components or ideas involved • Whether resources are provided or need to be generated by the candidate • The question wording and any images, diagrams, or tables provided • Degree of abstraction • Response strategy – simple, stepwise, integrated • Guidance provided
  41. 41. Question 1: Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs. Question 2: Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.
  42. 42. Some factors that affect demand and difficulty • Type of operation • Degree of novelty • Number of components or ideas involved • Whether resources are provided or need to be generated by the candidate • The question wording and any images, diagrams, or tables provided • Degree of abstraction • Response strategy – simple, stepwise, integrated • Guidance provided
  43. 43. For EACH ECG, make a recommendation for management of the horse and explain your reasoning. Discuss the recommended management for a horse with atrial fibrillation and explain your reasoning.
  44. 44. Some factors that affect demand and difficulty • Type of operation • Degree of novelty • Number of components or ideas involved • Whether resources are provided or need to be generated by the candidate • The question wording and any images, diagrams, or tables provided • Degree of abstraction • Response strategy – simple, stepwise, integrated • Guidance provided
  45. 45. Some factors that affect demand and difficulty • Type of operation • Degree of novelty • Number of components or ideas involved • Whether resources are provided or need to be generated by the candidate • The question wording and any images, diagrams, or tables provided • Degree of abstraction • Response strategy – simple, stepwise, integrated • Guidance provided
  46. 46. Discuss the concept of quality of life in terms of biological function, “feelings” and natural existence. Describe how both classical conditioning and operant conditioning are involved in cows confidently entering the milking shed and letting down.
  47. 47. Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)
  48. 48. Some factors that affect demand and difficulty • Type of operation • Degree of novelty • Number of components or ideas involved • Whether resources are provided or need to be generated by the candidate • The question wording and any images, diagrams, or tables provided • Degree of abstraction • Response strategy – simple, stepwise, integrated • Guidance provided
  49. 49. Describe the skull. (30 marks)
  50. 50. Question 1 Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks) Question 2 a) Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats (10 marks) b) Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks) c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).
  51. 51. Some factors that affect demand and difficulty • Type of operation • Degree of novelty • Number of components or ideas involved • Whether resources are provided or need to be generated by the candidate • The question wording and any images, diagrams, or tables provided • Degree of abstraction • Response strategy – simple, stepwise, integrated • Guidance provided
  52. 52. You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks) Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide.
  53. 53. Pacing examinations • Reading time: 40 words per minute • Writing time: 20 words per minute • Marking schemes help you check the pacing
  54. 54. Key points
  55. 55. Blueprinting • Ensure the examination (as a whole) representatively samples from the domain specified for the examination – Topics – Level – Species – Other things eg: • diagnosis vs management • principles vs application
  56. 56. Question writing • Choose a task that will allow candidates to demonstrate higher order skills • Phrase your “question” as an instruction • Define the scope and boundaries of the answer • Remove all irrelevant difficulty • Keep level of demand appropriate for the level of examination (Fellowship vs Membership) and the time available
  57. 57. What you have to watch out for • Design assessment tasks that request an integrated and coherently structured response. Not: “write short notes on”, “use a table to compare” • Relational responses can be reproduced, so design tasks that require an original application of related knowledge. • Decide on the nature of connections expected at the level of expertise being assessed. Knowledge connections can range from very basic to expert.
  58. 58. References and further reading Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). New York: Academic Press. Biggs, J. (1992). A qualitative approach to grading students. HERDSA News, 14(3), 3-6.

×