This document summarizes the development of a diagnostic test to evaluate students' data analysis skills. The authors developed a 23-question multiple choice test covering key skills like precision, accuracy, error analysis and graph interpretation. They administered the test to over 1200 students across 10 institutions, finding that skills development is not consistent and later-year students did not necessarily perform better. They propose augmenting the test with online learning resources to help address misconceptions and make data analysis skills development more explicit in curricula.

1. Physics Education Research
The University of Edinburgh
Diagnosing Student Understanding of
Data Analysis Techniques
Simon Bates Ross Galloway
Helen Maynard-Casely
Hilary Singer Kate Slaughter
Physics Education Research Group, University of Edinburgh, UK
AAPT Summer Meeting 2011, Omaha NE, Aug 2011

2. Physics Education Research
The University of Edinburgh
Outline
• Why investigate data handling skills?
• Development of the diagnostic
• Student performance on the diagnostic
• Development of supporting resources
• Summary

3. Physics Education Research
The University of Edinburgh
Labs: the implicit curriculum
• In practical work, we expect students
to acquire data analysis skills in
parallel to practical abilities.
• Frequently, these generic skills are not
explicitly taught and not effectively
assessed.

4. Physics Education Research
The University of Edinburgh
Higher Education Academy
Student Employability Profiles

5. Physics Education Research
The University of Edinburgh
Typical question (from UBC pre-prototype)

6. Physics Education Research
The University of Edinburgh
A!

7. Physics Education Research
The University of Edinburgh
B!

8. Physics Education Research
The University of Edinburgh
C!

9. Physics Education Research
The University of Edinburgh
D!

10. Physics Education Research
The University of Edinburgh
A!
B!
C!
D!

11. Physics Education Research
The University of Edinburgh
Response profiles
from Edinburgh
Physics classes
No statistically
significant difference
between 1st, 2nd and
4th year classes.

12. Physics Education Research
The University of Edinburgh
Where we went from here…
• Build on pre-prototype UBC instrument.
• Expand number of questions, increase scope.
• Ensure generic nature so wide applicability:
physics, chemistry, engineering, other STEM, etc.
• Ensure test is robust and reliable.

13. Physics Education Research
The University of Edinburgh
Diagnostic test development process
Beichner R. J., (1994)
Am. J. Physics 62, 750.

14. Physics Education Research
The University of Edinburgh
Data Handling Diagnostic – prototype test instrument
23 item MCQ test to address generic
data analysis skills. Wide applicability
to all STEM disciplines.
Topics covered include:
o precision and accuracy
o approximations
o lines of best fit
o errors and uncertainties
o derivation of quantitative
relationships from graphical data

15. Physics Education Research
The University of Edinburgh
Trial phase deployment
• Prototype test taken
by > 1200 students
• Test deployed in 10
institutions across UK
and Ireland (7
Physics & 4
Chemistry depts.)
• Test deployed to all
undergraduate years
(1st – 5th)

16. Physics Education Research
The University of Edinburgh
Validity and reliability evaluation
Validity
• test covers appropriate range of skills
• test does actually evaluate what you intended
Reliability
• degree of consistency between scores when
test repeated
• test has ability to discriminate student ability
Employ statistical tests and analysis, for
individual items and for whole test.
e.g. BEMA: Phys. Rev. Spec. Topics PER 2 010105 (2006)
DHD found to perform satisfactorily

17. Physics Education Research
The University of Edinburgh
Student performance from the trial deployment

18. Physics Education Research
The University of Edinburgh

19. Physics Education Research
The University of Edinburgh

20. Physics Education Research
The University of Edinburgh

21. Physics Education Research
The University of Edinburgh

22. Physics Education Research
The University of Edinburgh

23. Physics Education Research
The University of Edinburgh
‘Phase space’ of class response profiles

24. Physics Education Research
The University of Edinburgh

25. Physics Education Research
The University of Edinburgh
PG
‘Phase space’ of class response profiles

26. Physics Education Research
The University of Edinburgh
Faculty
PG
‘Phase space’ of class response profiles

27. Physics Education Research
The University of Edinburgh
χ2 consistency test p-values:
1st 2nd 3rd 4th 5th PG
0.038 0.971 0.985 0.934 0.701 0.238

28. Physics Education Research
The University of Edinburgh
Data handling skill development
• 1st and 2nd year → explicit instruction → improvement
• Later years → implicit development → no improvement
• Postgrad TA’s not more proficient than undergrads they
are instructing
• Faculty significantly more proficient
Broadly consistent with findings at University of British Columbia
Day & Bonn, PRST-PER 7, 010114, (2011)

29. Physics Education Research
The University of Edinburgh
Where do we go from here?
• Augment test with stand-alone interactive
learning resources
– diagnose and address
misconceptions.

30. Physics Education Research
The University of Edinburgh
Learning resources
• Stand-alone and web-based, to support rather than
supplant other instruction.
• Accessible direct from questions and by category.
• Feature informal explanations, step-by-step guides,
animations, practice exercises and self-test questions.

31. Physics Education Research
The University of Edinburgh

32. Physics Education Research
The University of Edinburgh
Example: mean, standard deviation and standard error

33. Physics Education Research
The University of Edinburgh
• Random samples
drawn from
Gaussian
distribution.
• mean = 10
std.dev. = 5
• Track evolution of
mean, standard
deviation and
standard error as
number of
measurements N
increases.

34. Physics Education Research
The University of Edinburgh
Summary
• Data handling skills are important and valued but
frequently fall into the ‘implicit curriculum’.
• We have developed a valid and reliable Data Handling
Diagnostic.
• We have found that development of these skills and
overall student proficiency is not what it might be.
• Part of the problem may be that lab Teaching Assistants
also have serious gaps in their knowledge.
• Explicit instruction in data handling skills seems to be
required.

35. Physics Education Research
The University of Edinburgh
Acknowledgements:
We gratefully acknowledge project grant support from the HEA Physical
Sciences Centre and the assistance of our partner institutions in the pilot
phase. We also gratefully acknowledge Doug Bonn and James Day of the
University of British Columbia for initial ideas and for allowing re-use of
three of their items.
Poster at PERC
EdPER group website bit.ly/EdPER
Talk slides on Slideshare EdPER_talks
Ross.Galloway@ed.ac.uk
S.P.Bates@ed.ac.uk