Diagnosing Student Understanding of Data Analysis Techniques

1,271 views
1,210 views

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,271
On SlideShare
0
From Embeds
0
Number of Embeds
208
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Diagnosing Student Understanding of Data Analysis Techniques

  1. 1. Physics Education Research The University of EdinburghDiagnosing Student Understanding of Data Analysis Techniques Simon Bates Ross Galloway Helen Maynard-Casely Hilary Singer Kate SlaughterPhysics Education Research Group, University of Edinburgh, UK AAPT Summer Meeting 2011, Omaha NE, Aug 2011
  2. 2. Physics Education Research The University of EdinburghOutline •  Why investigate data handling skills? •  Development of the diagnostic •  Student performance on the diagnostic •  Development of supporting resources •  Summary
  3. 3. Physics Education Research The University of EdinburghLabs: the implicit curriculum •  In practical work, we expect students to acquire data analysis skills in parallel to practical abilities. •  Frequently, these generic skills are not explicitly taught and not effectively assessed.
  4. 4. Physics Education Research The University of Edinburgh Higher Education Academy Student Employability Profiles
  5. 5. Physics Education Research The University of EdinburghTypical question (from UBC pre-prototype)
  6. 6. Physics Education Research The University of Edinburgh A!
  7. 7. Physics Education Research The University of Edinburgh B!
  8. 8. Physics Education Research The University of Edinburgh C!
  9. 9. Physics Education Research The University of Edinburgh D!
  10. 10. Physics Education Research The University of Edinburgh A! B! C! D!
  11. 11. Physics Education Research The University of EdinburghResponse profilesfrom EdinburghPhysics classes No statistically significant difference between 1st, 2nd and 4th year classes.
  12. 12. Physics Education Research The University of EdinburghWhere we went from here… •  Build on pre-prototype UBC instrument. •  Expand number of questions, increase scope. •  Ensure generic nature so wide applicability: physics, chemistry, engineering, other STEM, etc. •  Ensure test is robust and reliable.
  13. 13. Physics Education Research The University of EdinburghDiagnostic test development process Beichner R. J., (1994) Am. J. Physics 62, 750.
  14. 14. Physics Education Research The University of EdinburghData Handling Diagnostic – prototype test instrument 23 item MCQ test to address generic data analysis skills. Wide applicability to all STEM disciplines. Topics covered include: o  precision and accuracy o  approximations o  lines of best fit o  errors and uncertainties o  derivation of quantitative relationships from graphical data
  15. 15. Physics Education Research The University of EdinburghTrial phase deployment•  Prototype test taken by > 1200 students•  Test deployed in 10 institutions across UK and Ireland (7 Physics & 4 Chemistry depts.)•  Test deployed to all undergraduate years (1st – 5th)
  16. 16. Physics Education Research The University of EdinburghValidity and reliability evaluation Validity •  test covers appropriate range of skills •  test does actually evaluate what you intended Reliability •  degree of consistency between scores when test repeated •  test has ability to discriminate student ability Employ statistical tests and analysis, for individual items and for whole test. e.g. BEMA: Phys. Rev. Spec. Topics PER 2 010105 (2006)     DHD found to perform satisfactorily
  17. 17. Physics Education Research The University of EdinburghStudent performance from the trial deployment
  18. 18. Physics Education Research The University of Edinburgh
  19. 19. Physics Education Research The University of Edinburgh
  20. 20. Physics Education Research The University of Edinburgh
  21. 21. Physics Education Research The University of Edinburgh
  22. 22. Physics Education Research The University of Edinburgh
  23. 23. Physics Education Research The University of Edinburgh‘Phase space’ of class response profiles
  24. 24. Physics Education Research The University of Edinburgh
  25. 25. Physics Education Research The University of Edinburgh PG‘Phase space’ of class response profiles
  26. 26. Physics Education Research The University of Edinburgh Faculty PG‘Phase space’ of class response profiles
  27. 27. Physics Education Research The University of Edinburgh χ2 consistency test p-values: 1st 2nd 3rd 4th 5th PG0.038 0.971 0.985 0.934 0.701 0.238
  28. 28. Physics Education Research The University of EdinburghData handling skill development•  1st and 2nd year → explicit instruction → improvement•  Later years → implicit development → no improvement•  Postgrad TA’s not more proficient than undergrads they are instructing•  Faculty significantly more proficient Broadly consistent with findings at University of British Columbia Day & Bonn, PRST-PER 7, 010114, (2011)
  29. 29. Physics Education Research The University of EdinburghWhere do we go from here? •  Augment test with stand-alone interactive learning resources – diagnose and address misconceptions.
  30. 30. Physics Education Research The University of EdinburghLearning resources •  Stand-alone and web-based, to support rather than supplant other instruction. •  Accessible direct from questions and by category. •  Feature informal explanations, step-by-step guides, animations, practice exercises and self-test questions.
  31. 31. Physics Education Research The University of Edinburgh
  32. 32. Physics Education Research The University of EdinburghExample: mean, standard deviation and standard error
  33. 33. Physics Education Research The University of Edinburgh•  Random samples drawn from Gaussian distribution.•  mean = 10 std.dev. = 5•  Track evolution of mean, standard deviation and standard error as number of measurements N increases.
  34. 34. Physics Education Research The University of EdinburghSummary •  Data handling skills are important and valued but frequently fall into the ‘implicit curriculum’. •  We have developed a valid and reliable Data Handling Diagnostic. •  We have found that development of these skills and overall student proficiency is not what it might be. •  Part of the problem may be that lab Teaching Assistants also have serious gaps in their knowledge. •  Explicit instruction in data handling skills seems to be required.
  35. 35. Physics Education Research The University of EdinburghAcknowledgements:We gratefully acknowledge project grant support from the HEA Physical Sciences Centre and the assistance of our partner institutions in the pilot phase. We also gratefully acknowledge Doug Bonn and James Day of the University of British Columbia for initial ideas and for allowing re-use of three of their items.Poster at PERCEdPER group website bit.ly/EdPERTalk slides on Slideshare EdPER_talksRoss.Galloway@ed.ac.uk S.P.Bates@ed.ac.uk

×