Once the teachers professional judgments have been synthesised by the tool into an OTJ – the OTJ can be considered against other standards assessment results (eg e-asTTle, PAT).
Transcript of "Nznsa conference michael johnston 29 aug 2012"
Achieving consistency of teachers’ judgements and measuring progress against national standards: Curriculum knowledge deployed within a psychometric frameworkMichael Johnston29 August 2012
National Standards design• Validity of teacher judgements is emphasised: – Wide range of evidence – Over-reliance on formal testing discouraged – No national testing• Reliability will be a challenge: – Overall judgement must be formed from diverse evidence on multiple criteria
Achieving reliability• Important to teacher acceptance of the standards: – Common understanding of standards – Confidence that they can make judgements with reasonable national consistency• Exemplars and moderation help but are not sufficient – A given student does not usually resemble the exemplar – Social moderation is not designed to produce consistency
The Progress and Consistency Tool (PaCT)• Designed to support teachers to make reliable judgements without sacrificing validity. – A diverse evidence base and professional judgement remains at the forefront. The framework gives structure to the professional judgement.• Will provide a mechanism to measure progress.
Design of PaCTEach sequence of standards (reading, writing, mathematics): – Conceptualised as a developmental continuum, underpinned by a quantitative scale – Decomposed into a set of aspects, each comprising a series of (sequential) items stages.• Each item comprises annotated illustrations.• Collectively, the aspects comprise a
The form of the rubrics An item An illustration
Guidelines and principles for developing progressions1. Aspects should represent salient stages of cognitive development and/or curriculum progression.2. Aspects should cover the range of relevant cognitive functions important to the standard.3. The number of aspects for each standard should be manageable for teachers.4. Observational evidence for each item (stage) of each aspect must be readily available to, and recognisable by, teachers.
Guidelines and principlesfor developing progressions (ctd.)5. Aspects should be associated with cognitive development and learning rather than with year levels.6. The aspects defined for a given standard need not all entail the same numbers of stages.7. The extent of each stage on each progression in relation to the overall continuum will be determined empirically, not defined by the standards writers.
Aspects for the mathematics rubric (Gill Thomas)Strand AspectsNumber and • Additive thinkingalgebra • Multiplicative thinking • Patterns and relationships • Using symbols and expressionsMeasurement • Measurement senseand • Geometrical thinkinggeometryStatistics • Statistical investigations • Statistical literacy and probability
Aspects for the reading rubric (Sue Douglas)1. Making sense of text2. Applying vocabulary knowledge (with Writing)3. Using reading to organise for learning4. Locating and using information and ideas in informational texts (continuous and non- continuous/in print)5. Locating and using information and ideas in informational texts (continuous and non- continuous/on-line)6. Interpreting and responding to ideas, information and experiences in literary texts (in print and on-line)7. Identifying and reflecting on the way writers use ideas and language to influence their readers.
Aspects for the writing rubric (Sue Douglas)1.Encoding2.Sentence Structure (different types of sentences &sentence beginnings; grammar; punctuation)3.Applying vocabulary knowledge (with Reading)4.Using writing to think and organise for learning5.Using writing (in print) to communicate knowledge andunderstanding6.Using writing (online) to communicate knowledge andunderstanding7.Creating texts for literary purposes8.Creating texts to influence others
The components of an illustration Annotation The interpretation of the response & why it ‘fits’ Student response What the child does or says (work sample, interview transcript) Prepared for TAG, 14 August 2012
Implementing the framework• Defining the rubrics; describing the aspects and items• Developing the illustrations• Collection and analysis of sample data – Determine quality of rubrics, items and illustrations – Calibrate the measurement scale• Standard setting
Developing the frameworkAnalysis of sample data- Collect judgements on each aspect for a sample of students- Sample should be representative of Year levels, geographic regions, school deciles and demographics- The scale for each sequence of standards will be constructed from these data using item-response analysis.
Utility of item response analysis• Creates a scale with equal-interval properties. – Ideal for measuring progress• Locates each stage of each progression on this overall scale.
Developing the frameworkStandard setting – When the overall continuum for each sequence of standards has been constructed, a standard- setting exercise will be required to define overall Year-level boundaries. – Standard setting will require holistic judgements by expert judges made using the same evidence base as that supporting the scale construction exercise. – Important to be clear that boundaries are stochastic, not deterministic