Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Loughborough DHD seminar


Published on

  • Be the first to comment

  • Be the first to like this

Loughborough DHD seminar

  1. 1. Diagnosing students’ data handling skills Simon Bates Ross Galloway Helen Maynard-Casely Hilary Singer Kate Slaughter
  2. 2. <ul><li>Diagnostic tests: what, why and how? </li></ul><ul><li>Existing diagnostic tests </li></ul><ul><li>Why investigate data handling skills? </li></ul><ul><li>Development of the diagnostic </li></ul><ul><li>Initial results </li></ul><ul><li>Development of supporting resources </li></ul><ul><li>Summary </li></ul>Outline
  3. 3. Diagnostic tests: what? <ul><li>Standardised, statistically validated test. </li></ul><ul><li>Often in multiple choice question (MCQ) format. </li></ul><ul><li>Targeted to address key learning outcomes. </li></ul><ul><li>Often used to identify misconceptions. </li></ul>
  4. 4. Diagnostic tests: why? <ul><li>Ausubel’s Dictum: “ Ascertain what the student knows and teach accordingly ” </li></ul>“ Ascertain what the student misunderstands and teach accordingly ”
  5. 5. Diagnostic tests: how? <ul><li>Often administered using ‘pre / post’ scheme. </li></ul>Pre-test Respond Instruct Post-test
  6. 6. Existing diagnostic tests <ul><li>Scientific reasoning (Lawson) </li></ul><ul><li>Student attitudes (CLASS) </li></ul><ul><li>Graphs (TUG-K) </li></ul><ul><li>Forces and statics (FCI) </li></ul><ul><li>Electricity and magnetism (BEMA, homegrown) </li></ul><ul><li>Quantum mechanics (QMCS, QPCS, homegrown) </li></ul><ul><li>Light and spectroscopy Concept Inventory (Bader) </li></ul><ul><li>Data handling (UBC/homegrown) </li></ul><ul><li>… </li></ul>
  7. 7. Force Concept Inventory … a mature, established diagnostic test.
  8. 8. Labs: the implicit curriculum <ul><li>In practical work, we expect students to acquire data analysis skills in parallel to practical abilities. </li></ul><ul><li>Frequently, these generic skills are not explicitly taught and not effectively assessed. </li></ul>
  9. 9. HEA Student Employability Profiles
  10. 10. Typical question (from UBC pre-prototype)
  11. 11. A
  12. 12. B
  13. 13. C
  14. 14. D
  15. 15. A B C D
  16. 16. No statistically significant difference between 1 st , 2 nd and 4 th year classes. Response profiles from Edinburgh Physics classes
  17. 17. Where we went from here… <ul><li>Build on pre-prototype UBC instrument. </li></ul><ul><li>Expand number of questions, increase scope. </li></ul><ul><li>Ensure generic nature so wide applicability: physics, chemistry, engineering, other STEM, etc. </li></ul><ul><li>Ensure test is robust and reliable. </li></ul>
  18. 18. Diagnostic test development process Beichner R. J., (1994) Am. J. Physics 62 , 750.
  19. 19. Data Handling Diagnostic – prototype test instrument 23 item MCQ test to address generic data analysis skills. Wide applicability to all STEM disciplines. Topics covered include: <ul><li>precision and accuracy </li></ul><ul><li>approximations </li></ul><ul><li>lines of best fit </li></ul><ul><li>errors and uncertainties </li></ul><ul><li>derivation of quantitative relationships from graphical data </li></ul>
  20. 20. Trial phase deployment <ul><li>Prototype test taken by > 1200 students </li></ul><ul><li>Test deployed in 10 institutions across UK and Ireland (7 Physics & 4 Chemistry depts.) </li></ul><ul><li>Test deployed to all undergraduate years (1 st – 5 th ) </li></ul>
  21. 21. Validity and reliability evaluation <ul><ul><li>Validity </li></ul></ul><ul><ul><li>test covers appropriate range of skills </li></ul></ul><ul><ul><li>test does actually evaluate what you intended </li></ul></ul><ul><ul><li>Reliability </li></ul></ul><ul><ul><li>degree of consistency between scores when test repeated </li></ul></ul><ul><ul><li>test has ability to discriminate student ability </li></ul></ul><ul><ul><li>Employ statistical tests and analysis, for individual items and for whole test. </li></ul></ul><ul><ul><li>e.g. BEMA: Phys. Rev. Spec. Topics PER 2 010105 (2006) </li></ul></ul><ul><ul><li>DHD found to perform satisfactorily </li></ul></ul>
  22. 22. Student performance from the trial deployment Histogram of student scores
  23. 23. Easy Medium Hard Question correct response rates (all students)
  24. 24. Question response profiles
  25. 25. Question 18 and response profile
  26. 26. Question correct response rates broken down by class
  27. 27. Question correct response rates broken down by class
  28. 28. Edinburgh Physics year-by-year mean test scores
  29. 29. Edinburgh Physics year-by-year question correct response rates  2 consistency test p -values: 1 st 2 nd 3 rd 4 th 5 th PhD 0.038 0.971 0.985 0.934 0.701 0.238
  30. 30. <ul><li>Augment test with stand-alone interactive learning resources </li></ul><ul><li>– diagnose and address misconceptions. </li></ul>Where do we go from here?
  31. 31. Learning resources <ul><li>Stand-alone and web-based, to support rather than supplant other instruction. </li></ul><ul><li>Accessible direct from questions and by category. </li></ul><ul><li>Feature informal explanations, step-by-step guides, animations, practice exercises and self-test questions. </li></ul>
  32. 33. Example: mean, standard deviation and standard error
  33. 34. <ul><li>Random samples drawn from Gaussian distribution. </li></ul><ul><li>mean = 10 = 5 </li></ul><ul><li>Track evolution of mean , standard deviation and standard error as number of measurements N increases. </li></ul>
  34. 35. <ul><li>Diagnostic tests have been shown to effectively identify learning challenges. </li></ul><ul><li>Data handling skills are important and valued but frequently fall into the ‘implicit curriculum’. </li></ul><ul><li>We have developed a valid, reliable and effective Data Handling Diagnostic. </li></ul><ul><li>We have found that development of these skills and overall student proficiency is not what it might be. </li></ul><ul><li>The diagnostic instrument and its supporting learning resources are ready for use. </li></ul>Summary
  35. 36. <ul><li>Acknowledgements: </li></ul><ul><li>We gratefully acknowledge project grant support from the HEA Physical Sciences Centre and the assistance of our partner institutions in the pilot phase. We also gratefully acknowledge Doug Bonn and James Day of the University of British Columbia for initial ideas and for allowing re-use of three of their items. </li></ul>[email_address] [email_address]