Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
STATISTICAL MODELLING FOR IMPROVEMENT OF SECONDARY AND TERTIARY PHYSICS ASSESSMENTS
PHYSICS: EXTERNAL RESULTS LEVEL 3  BY GENDER (2010)
LEVEL 3 EXTERNAL STANDARDS  (2010 RESULTS)
PROFILES OF EXPECTED PERFORMANCE (PEPs) <ul><li>Expected percentage band for each grade, based on </li></ul><ul><li>statis...
2010 PEP BANDS  ACTUAL:  N = 32.3%;  A = 46.2%;  M = 15.1%;  E = 6.4%
COHORT STRENGTH & STANDARD DIFFICULTY <ul><li>S k   =  Σ j   ρ kj n kj   [   R kj ( j ) –  R j ( j )  ]   /   Σ j   n kj <...
RESULTS: PHYSICS 90522
PHYSICS 90522  Principal Components Analysis
PHYSICS 90254: Waves (Level 2)
ECONOMICS 90631 (Level 3) Market failure and government interventions
CHEMISTRY 90696 (Level 3)  Oxidation-reduction processes
STATISTICS 90643 (Level 3) Probability
PHYSICS 90522: ITEM LOADINGS THE FIRST & SECOND DIMENSIONS
INTERPRETATION?   <ul><li>FIRST DIMENSION   </li></ul><ul><li>Understanding of atoms, photons and nuclei </li></ul><ul><li...
MEASURES OF INTERNAL CONSISTENCY <ul><li>Inter-item Correlation, Item Total Correlation & </li></ul><ul><li>Cronbach Alpha...
INTER-ITEM & ITEM TOTAL CORRELATIONS  (CRONBACH ALPHA = 0.76)
EXAMINATIONS IN PHYSICS . . .  <ul><li>Always assess one dominant dimension (cognitive </li></ul><ul><li>construct) and at...
ITEM RESPONSE THEORY   <ul><li>T he probability of obtaining in a single item a particular grade or better ( Not Achieved ...
ITEM-CHARACTERISTIC CURVE (DICHOTOMOUS) DIFFICULTY :  the ability at which P = 0.5  DISCRIMINATION:  slope at that point
ITEM-CHARACTERISTIC CURVE (POLYTOMOUS)
IRT PARAMETERS   <ul><li>Item discriminations & grade difficulties estimated </li></ul><ul><li>using various models </li><...
90522: EXAMPLE ITEM PARAMETERS   (Graded Response Model)
PHYSICS 90522: HISTOGRAM OF CANDIDATE ABILITIES (PERFORMANCES)
PHYSICS 90522: ITEM CHARACTERISTIC CURVES
 
 
THE SUFFICIENCY METHOD OF ASSIGNING OVERALL GRADES <ul><li>For example: </li></ul><ul><ul><li>5A  =>  Achieved  </li></ul>...
ALIGNMENT OF ACTUAL FINAL GRADES WITH ABILITY-BASED GRADES
THE SCORE-GRADE METHOD   7  8 5  6 3  4 1  2 0 E M A N
  SCORE-BASED MARKING  <ul><li>Improved discrimination between grades  </li></ul><ul><li>Fairer outcomes  </li></ul><ul><l...
NEW ZEALAND SCHOLARSHIP
AWARDING SCHOLARSHIPS IN PHYSICS
RELATIONSHIP BETWEEN PERFORMANCE IN NCEA LEVEL 3 PHYSICS AND SCHOLARSHIP PHYSICS
DIFFERENTIAL ITEM FUNCTIONING (DIF) Two or more groups of candidates, matched for ability, perform differently on a partic...
DIF: ILLUSTRATION ( Item 1 from the 2010 Geography 90704 examination)
<ul><li>[email_address] </li></ul><ul><li>04 – 463 - 4251 </li></ul>
Upcoming SlideShare
Loading in …5
×

16.00 o6.2 d lillis

387 views

Published on

Physikos 2: David Lillis

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

16.00 o6.2 d lillis

  1. 1. STATISTICAL MODELLING FOR IMPROVEMENT OF SECONDARY AND TERTIARY PHYSICS ASSESSMENTS
  2. 2. PHYSICS: EXTERNAL RESULTS LEVEL 3 BY GENDER (2010)
  3. 3. LEVEL 3 EXTERNAL STANDARDS (2010 RESULTS)
  4. 4. PROFILES OF EXPECTED PERFORMANCE (PEPs) <ul><li>Expected percentage band for each grade, based on </li></ul><ul><li>statistical information & changes in examination </li></ul><ul><li>format etc </li></ul><ul><li>Provide guidance to examiners & markers </li></ul><ul><li>Ensure results that are consistent with the standards </li></ul><ul><li>and that standards are maintained </li></ul>
  5. 5. 2010 PEP BANDS ACTUAL: N = 32.3%; A = 46.2%; M = 15.1%; E = 6.4%
  6. 6. COHORT STRENGTH & STANDARD DIFFICULTY <ul><li>S k = Σ j ρ kj n kj [ R kj ( j ) – R j ( j ) ] / Σ j n kj </li></ul><ul><li>D k = Σ j ρ kj n kj [ R kj ( j ) – R kj ( k ) ] / Σ j n kj </li></ul>
  7. 7. RESULTS: PHYSICS 90522
  8. 8. PHYSICS 90522 Principal Components Analysis
  9. 9. PHYSICS 90254: Waves (Level 2)
  10. 10. ECONOMICS 90631 (Level 3) Market failure and government interventions
  11. 11. CHEMISTRY 90696 (Level 3) Oxidation-reduction processes
  12. 12. STATISTICS 90643 (Level 3) Probability
  13. 13. PHYSICS 90522: ITEM LOADINGS THE FIRST & SECOND DIMENSIONS
  14. 14. INTERPRETATION? <ul><li>FIRST DIMENSION </li></ul><ul><li>Understanding of atoms, photons and nuclei </li></ul><ul><li>SECOND DIMENSION </li></ul><ul><li>Item type (qualitative or quantitative responses) </li></ul>
  15. 15. MEASURES OF INTERNAL CONSISTENCY <ul><li>Inter-item Correlation, Item Total Correlation & </li></ul><ul><li>Cronbach Alpha </li></ul><ul><li>Range between – 1 and + 1 </li></ul><ul><li>Optimal between about 0.4 and 0.85 </li></ul>
  16. 16. INTER-ITEM & ITEM TOTAL CORRELATIONS (CRONBACH ALPHA = 0.76)
  17. 17. EXAMINATIONS IN PHYSICS . . . <ul><li>Always assess one dominant dimension (cognitive </li></ul><ul><li>construct) and at least one other dimension </li></ul><ul><li>For practical purposes, form a psychometric scale, </li></ul><ul><li>but not a perfect scale </li></ul>
  18. 18. ITEM RESPONSE THEORY <ul><li>T he probability of obtaining in a single item a particular grade or better ( Not Achieved , Achieved , Merit or Excellence ), for a candidate of ability θ (measured over the entire assessment) , is given by the expression: </li></ul><ul><li>P j = { 1 + exp [ ka ( b j – θ ) ] } -1 </li></ul><ul><li>j indexes the assessment grades Achieved (A) or better, Merit (M) or better, and Excellence (E) </li></ul><ul><li>θ is the calculated ability (which you can also think of as a measure of performance) </li></ul><ul><li>a is the item discrimination </li></ul><ul><li>b j is the estimated difficulty of gaining either an A or better, M or better, or an E grade for the item. </li></ul><ul><li>k = -1.7 (scales the logistic curve to approximate a cumulative ogive) </li></ul>
  19. 19. ITEM-CHARACTERISTIC CURVE (DICHOTOMOUS) DIFFICULTY : the ability at which P = 0.5 DISCRIMINATION: slope at that point
  20. 20. ITEM-CHARACTERISTIC CURVE (POLYTOMOUS)
  21. 21. IRT PARAMETERS <ul><li>Item discriminations & grade difficulties estimated </li></ul><ul><li>using various models </li></ul><ul><li>Ability estimated from performance on all items, </li></ul><ul><li>taking account of item difficulties & discriminations </li></ul><ul><li>The most objective measure of performance? </li></ul>
  22. 22. 90522: EXAMPLE ITEM PARAMETERS (Graded Response Model)
  23. 23. PHYSICS 90522: HISTOGRAM OF CANDIDATE ABILITIES (PERFORMANCES)
  24. 24. PHYSICS 90522: ITEM CHARACTERISTIC CURVES
  25. 27. THE SUFFICIENCY METHOD OF ASSIGNING OVERALL GRADES <ul><li>For example: </li></ul><ul><ul><li>5A => Achieved </li></ul></ul><ul><ul><li>3As + 3Ms => Merit </li></ul></ul><ul><ul><li>2Ms + 2Es => Excellence </li></ul></ul>
  26. 28. ALIGNMENT OF ACTUAL FINAL GRADES WITH ABILITY-BASED GRADES
  27. 29. THE SCORE-GRADE METHOD 7 8 5 6 3 4 1 2 0 E M A N
  28. 30. SCORE-BASED MARKING <ul><li>Improved discrimination between grades </li></ul><ul><li>Fairer outcomes </li></ul><ul><li>Reduction in year-to-year variability </li></ul>
  29. 31. NEW ZEALAND SCHOLARSHIP
  30. 32. AWARDING SCHOLARSHIPS IN PHYSICS
  31. 33. RELATIONSHIP BETWEEN PERFORMANCE IN NCEA LEVEL 3 PHYSICS AND SCHOLARSHIP PHYSICS
  32. 34. DIFFERENTIAL ITEM FUNCTIONING (DIF) Two or more groups of candidates, matched for ability, perform differently on a particular item.
  33. 35. DIF: ILLUSTRATION ( Item 1 from the 2010 Geography 90704 examination)
  34. 36. <ul><li>[email_address] </li></ul><ul><li>04 – 463 - 4251 </li></ul>

×