0
Achieving consistency of teachers’    judgements and measuring progress        against national standards:    Curriculum k...
National Standards design• Validity of teacher judgements is  emphasised:  – Wide range of evidence  – Over-reliance on fo...
Achieving reliability• Important to teacher acceptance of the  standards:  – Common understanding of standards  – Confiden...
The Progress and Consistency         Tool (PaCT)• Designed to support teachers to make  reliable judgements without sacrif...
Design of PaCTEach sequence of standards (reading, writing, mathematics):  – Conceptualised as a developmental    continuu...
The form of the rubrics                          An item        An illustration
Guidelines and principles  for developing progressions1. Aspects should represent salient stages of   cognitive developmen...
Guidelines and principlesfor developing progressions (ctd.)5. Aspects should be associated with cognitive   development an...
Aspects for the mathematics rubric (Gill Thomas)Strand      AspectsNumber and    • Additive thinkingalgebra       • Multip...
Aspects for the reading rubric (Sue Douglas)1. Making sense of text2. Applying vocabulary knowledge (with Writing)3. Using...
Aspects for the writing rubric (Sue Douglas)1.Encoding2.Sentence Structure (different types of sentences &sentence beginni...
The components of an illustration                               Annotation                      The interpretation of the ...
More expert About the  sameLess expert  Unsure
Implementing the framework• Defining the rubrics; describing the  aspects and items• Developing the illustrations• Collect...
Developing the frameworkAnalysis of sample data- Collect judgements on each aspect for  a sample of students- Sample shoul...
Utility of item response analysis• Creates a scale with equal-interval  properties.  – Ideal for measuring progress• Locat...
Developing the frameworkStandard setting  – When the overall continuum for each sequence of    standards has been construc...
Rubric design: Standard setting            ✔   ✔✪      ✔                ✔        ✔                             ✔
Test information• Teachers can also enter results for a  selection of tests; e.g.,    – PAT    – GLOSS    – asTTle• It is ...
Bands showing probable location of                       a student                 95% confidence                 68% conf...
Measuring and reporting                                      progress                          5     4Year 5              ...
Problematic issues• Making the tool mandatory• Housing and usage of data
Upcoming SlideShare
Loading in...5
×

Nznsa conference michael johnston 29 aug 2012

275

Published on

Dr Michael Johnston's presentation to NZ Normal Schools confere

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
275
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Once the teachers professional judgments have been synthesised by the tool into an OTJ – the OTJ can be considered against other standards assessment results (eg e-asTTle, PAT).
  • Transcript of "Nznsa conference michael johnston 29 aug 2012"

    1. 1. Achieving consistency of teachers’ judgements and measuring progress against national standards: Curriculum knowledge deployed within a psychometric frameworkMichael Johnston29 August 2012
    2. 2. National Standards design• Validity of teacher judgements is emphasised: – Wide range of evidence – Over-reliance on formal testing discouraged – No national testing• Reliability will be a challenge: – Overall judgement must be formed from diverse evidence on multiple criteria
    3. 3. Achieving reliability• Important to teacher acceptance of the standards: – Common understanding of standards – Confidence that they can make judgements with reasonable national consistency• Exemplars and moderation help but are not sufficient – A given student does not usually resemble the exemplar – Social moderation is not designed to produce consistency
    4. 4. The Progress and Consistency Tool (PaCT)• Designed to support teachers to make reliable judgements without sacrificing validity. – A diverse evidence base and professional judgement remains at the forefront. The framework gives structure to the professional judgement.• Will provide a mechanism to measure progress.
    5. 5. Design of PaCTEach sequence of standards (reading, writing, mathematics): – Conceptualised as a developmental continuum, underpinned by a quantitative scale – Decomposed into a set of aspects, each comprising a series of (sequential) items stages.• Each item comprises annotated illustrations.• Collectively, the aspects comprise a
    6. 6. The form of the rubrics An item An illustration
    7. 7. Guidelines and principles for developing progressions1. Aspects should represent salient stages of cognitive development and/or curriculum progression.2. Aspects should cover the range of relevant cognitive functions important to the standard.3. The number of aspects for each standard should be manageable for teachers.4. Observational evidence for each item (stage) of each aspect must be readily available to, and recognisable by, teachers.
    8. 8. Guidelines and principlesfor developing progressions (ctd.)5. Aspects should be associated with cognitive development and learning rather than with year levels.6. The aspects defined for a given standard need not all entail the same numbers of stages.7. The extent of each stage on each progression in relation to the overall continuum will be determined empirically, not defined by the standards writers.
    9. 9. Aspects for the mathematics rubric (Gill Thomas)Strand AspectsNumber and • Additive thinkingalgebra • Multiplicative thinking • Patterns and relationships • Using symbols and expressionsMeasurement • Measurement senseand • Geometrical thinkinggeometryStatistics • Statistical investigations • Statistical literacy and probability
    10. 10. Aspects for the reading rubric (Sue Douglas)1. Making sense of text2. Applying vocabulary knowledge (with Writing)3. Using reading to organise for learning4. Locating and using information and ideas in informational texts (continuous and non- continuous/in print)5. Locating and using information and ideas in informational texts (continuous and non- continuous/on-line)6. Interpreting and responding to ideas, information and experiences in literary texts (in print and on-line)7. Identifying and reflecting on the way writers use ideas and language to influence their readers.
    11. 11. Aspects for the writing rubric (Sue Douglas)1.Encoding2.Sentence Structure (different types of sentences &sentence beginnings; grammar; punctuation)3.Applying vocabulary knowledge (with Reading)4.Using writing to think and organise for learning5.Using writing (in print) to communicate knowledge andunderstanding6.Using writing (online) to communicate knowledge andunderstanding7.Creating texts for literary purposes8.Creating texts to influence others
    12. 12. The components of an illustration Annotation The interpretation of the response & why it ‘fits’ Student response What the child does or says (work sample, interview transcript) Prepared for TAG, 14 August 2012
    13. 13. More expert About the sameLess expert Unsure
    14. 14. Implementing the framework• Defining the rubrics; describing the aspects and items• Developing the illustrations• Collection and analysis of sample data – Determine quality of rubrics, items and illustrations – Calibrate the measurement scale• Standard setting
    15. 15. Developing the frameworkAnalysis of sample data- Collect judgements on each aspect for a sample of students- Sample should be representative of Year levels, geographic regions, school deciles and demographics- The scale for each sequence of standards will be constructed from these data using item-response analysis.
    16. 16. Utility of item response analysis• Creates a scale with equal-interval properties. – Ideal for measuring progress• Locates each stage of each progression on this overall scale.
    17. 17. Developing the frameworkStandard setting – When the overall continuum for each sequence of standards has been constructed, a standard- setting exercise will be required to define overall Year-level boundaries. – Standard setting will require holistic judgements by expert judges made using the same evidence base as that supporting the scale construction exercise. – Important to be clear that boundaries are stochastic, not deterministic
    18. 18. Rubric design: Standard setting ✔ ✔✪ ✔ ✔ ✔ ✔
    19. 19. Test information• Teachers can also enter results for a selection of tests; e.g., – PAT – GLOSS – asTTle• It is envisaged the role of test information will as a self-moderation process.
    20. 20. Bands showing probable location of a student 95% confidence 68% confidence Year 4Continuum Year 3 Year 2 Year 1 Observational Test
    21. 21. Measuring and reporting progress 5 4Year 5 3Year 4 Performance (logits) 2Stand ard 1 Year 3 0Year -1 2 -2 2011 (Year 2) 2012 (Year 3) 2013 (Year 4) 2014 (Year 4) 2015 (Year 5) Year and schooling year
    22. 22. Problematic issues• Making the tool mandatory• Housing and usage of data
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×