Moderation is desirable but not mandatory and is described in the NQC documents as a quality control process. Within a moderation process, adjustments to student results should be made prior to their finalisation of the results if the judgements of the assessor have been determined to be too harsh or lenient. Similarly, moderation can lead to adjustments to student results if the assessment tools have been determined to be too easy and/or difficult. Adjustments are made therefore to the students’ results prior to finalisation. This process helps to bring standards across RTOs into alignment and therefore ensure fairness and comparability of standards across the sector. Although moderation is desirable within the VET Sector, particularly in high risk assessments, it is not necessary under the AQTF as in many instances the benefits may not outweigh the costs.
Here is also some examples of the outcomes of a moderation process. Like validation, it could lead to recommendations for improvements to the tool, but unlike validation, it may require altering students’ results prior to finalisation to bring standards into alignment. There may also be a requirement for some form of accountability.
This table provides a good summary as to the difference between validation and moderation. It can be found in the NQC Implementation guide.
Although yet to be pursued at the national level within the VET Sector, statistical moderation could be used to ensure that RTO based assessments are comparable throughout the nation, particularly if grades or marks are to be reported. However, to implement this moderation process, some form of a common assessment task(s) would need to be introduced at a national level in the VET sector (e.g., external exam or standardised assessment tools) to moderate the organisation-based assessments. If a common assessment task was used to statistically moderate organisation-based assessments, the statistical moderation process would maintain the rank order of the candidates’ scores (as determined by the assessor/organisation) but it would bring the distributions of scores across groups of candidates (from other organisations or assessors) within the same units within a qualification into alignment. That is, statistical moderation adjusts the organisation-based assessments in accordance with candidates’ performances on common external tasks. It should be acknowledged that any adjustment to a candidate’s scores is determined by the external scores for the whole organisation’s cohort, not by the candidate’s own external score. It is also important to note that statistical moderation does not change the rank order of candidates, as determined by the organisation’s scores. A candidate given the top score for an assessment task by his/her organisation would have the top score after statistical moderation, no matter how they performed on the external task. The process recognises that organisations are in the best position to make comparative judgements about the performance of their candidates and these comparative judgements are not changed as a result of the statistical moderation. Statistical moderation entails adjusting the level and spread of each organisation’s assessments of its candidates in a particular qualification, to match the level and spread of the same candidates’ scores on a common external task. If a common assessment task was to be completed by all candidates across the nation or within an industry area, it could become the common standard against which organisation’s assessments could be compared. At a national level, the organisation-based assessments could be statistically moderated using:A common exam across all qualifications based on measuring generic/employability skills.Qualification specific national exams (similar to those used for licensing purposes).National common assessment tools within each qualification that would need to be judged centrally. The major benefit of statistical moderation is that it provides the strongest form of quality control over organisation-based assessments. It can also be less expensive to implement and maintain (if paper-based) than external moderation processes. It would however require the introduction of some form of common assessment task(s) at the national level. If the common assessment task was paper-based (as has been typically implemented in other educational sectors due to reduced costs associated with the implementation and scoring procedures), then any adjustments to candidate results would be limited to estimates of candidates’ cognitive skills (i.e., knowledge and understanding); and therefore may have limited face and content validity within the VET sector.
External Approaches (Validation and Moderation)There are various external approaches to assessment validation and moderation. One approach would be for an external person (or a panel of people) to visit the organisation to judge the way in which candidates’ evidence were collected and judged against the Unit(s) of Competency. Differences between the local and external assessment judgements could then be either:Discussed and reconciled accordingly (i.e., if conducted for moderation purposes); and/or Discussed to identify ways in which improvements to future assessment practices could be undertaken (i.e., if conducted for validation purposes). An alternative external approach would be for samples of assessment tools and/or judged candidate evidence to be sent to a central location for specialist assessors to review directly against the Unit(s) of Competency. The specialist external assessors could be representatives of the relevant national Industry Skills Council (ISC) and/or the relevant state/territory registering bodies. Again, differences between the organisation and the external-based assessments could then be discussed (e.g., for validation) and/or reconciled (e.g., for moderation) at a distance. There are a number of benefits from using external moderators/validators. These include the potential to:Offer authoritative interpretations of the standards specified within Units of Competency;Improve consistency of the standards across locations by identifying local bias and/or misconceptions (if any);Offer advice to organisations and assessors on assessment approaches and procedures; andObserve actual assessment processes in real time as opposed to simply reviewing assessment products (if site visits are included). In relation to moderation, although external approaches have greater quality control over the assessment processes and outcomes than consensus meetings, they have less quality control than statistical approaches.
Typically consensus meetings involve assessors reviewing their own and their colleagues’ assessment tools and outcomes as part of a group. It can occur within and/or across organisations. It is typically based on agreement within a group on the appropriateness of the assessment tools and assessor judgements for a particular unit(s) of competency. A major strength of consensus meetings is that assessors are directly involved in all aspects of assessment and gain professionally by learning not only how and what to assess, but what standards to expect from their candidates. It also enables assessors to develop strong networks and promotes collegiality. Another benefit from consensus meetings is that it provides opportunity for sharing materials/resources among assessors. If used for moderation purposes, consensus meetings however provide less quality control than external and statistical approaches as again, they can be influenced by local values and expectations.
Questions are often asked about how can validation processes be systematic. The Assessor Guide provides some guidance here as to what a plan for validation may include. Implementation of the plan goes towards the notion of ‘systematic’.
The Assessor Guide provides some advice as to what validation model could be suitable to different RTOs or contexts.
An project related to the initial validation and moderation papers, was one undertaken by Shelley, Chloe and Andrea regarding how validation was implemented in diverse settings. The findings indicated that in general the RTOs were struggling with implementing a systematic process and the identified barriers were….However, the RTOs were deploying a range of strategies. Refer to Handout. Discuss this with the group.
Finally, the NQC Code of Practice has included this table – and it is worthwhile talking with your group what strategies your RTO use. Get comments back. Pull all comments together and close.
1. TVET Australia Assessment, validation and moderationA power point presentation developed by the NQC tosupport information sessions on assessment, validation andmoderation
3. DisclaimerThis work has been produced on behalf of the NationalQuality Council with funding provided through the AustralianGovernment Department of Education, Employment andWorkplace Relations and state and territory governments.The views expressed in this work are not necessarily thoseof the Australian Government or state and territorygovernments
4. AcknowledgementThis presentation was designed to support the interactiveinformation sessions that formed part of the NQC’scommunication and dissemination strategy: NQC products:validation and communication. Reports and materials whichfocus on validation and moderation may be downloadedfrom the NQC website athttp:www.nqwc.tvetaustralia.com.au/nqc_publicationsThis work was produced for the National Quality Council by Andrea Bateman, Quorum QA Australia Pty Ltd Chloe Dyson, Quorum QA Australia Pty Ltd
5. Validation Validation is a quality review process. It involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes.NQC Implementation Guide: Validation and Moderation 2009
6. Outcomes of validationRecommendations for future improvements Context and conditions for the assessment Task/s to be administered to the candidates Administration instructions Criteria used for judging the quality of performance (e.g. the decision making rules, evidence requirements etc) Guidelines for making reasonable adjustments to the way in which the evidence of performance was gathered to ensure that the expected standard of performance specified within the Unit(s) of Competency has not been altered Recording and reporting requirements.
7. Moderation Moderation is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.NQC Implementation Guide: Validation and Moderation 2009
8. Outcomes of moderation Recommendations for future improvement and adjustments to assessor judgements (if required) and Recommendations for improvement to the assessment tools Adjusting the results of a specific cohort of candidates prior to the finalisation of results and Requesting copies of final candidate assessment results in accordance with recommended actions.
9. Validation vs ModerationFeatures Validation ModerationAssessment Quality Review Quality ControlQualityManagementTypePrimary Purpose Continuous improvement Bring judgements and standards into alignment.Timing On-going Prior to the finalisation of candidate resultsFocus Assessment tools; and Assessment tools, and; Candidate Evidence Candidate Evidence, (including including assessor assessor judgements) judgements (mandatory) (desirable only)Type of Assessor Partnerships Consensus MeetingsApproaches Consensus Meetings External (moderators or External (validators or panels)
10. Types of Approaches - Statistical Limited to moderation Yet to be pursued at the national level in VET Requires some form of common assessment task at the national level Adjusts level and spread of RTO based assessments to match the level and spread of the same candidates scores on a common assessment task Maintains RTO-based rank ordering but brings the distribution of scores across groups of candidates into alignmentStrength Strongest form of quality controlWeakness Lacks face validity, may have limited content validity
11. Types of Approaches - External Types Site Visit Versus Central AgencyStrengths Offer authoritative interpretations of standards Improve consistency of standards across locations by identifying local bias and/or misconceptions (if any) EducativeWeakness Expensive Less control than statistical
12. Types of Approaches – Assessor Partnerships Validation only Informal, self-managed, collegial Small group of assessors May involve: Sharing, discussing and/or reviewing one another’s tools and/or judgements Benefit Low costs, personally empowering, non-threatening May be easily organised Weakness Potential to reinforce misconceptions and mistakes Ref: Implementation Guide, Assessment Fact
13. Types of Approaches - Consensus Typically involves reviewing their own & colleagues assessment tools and judgements as a group Can occur within and/or across organisations Strength Professional development, networking, promotes collegiality and sharing Weakness Less quality control than external and statistical approaches as they can also be influenced by local values and expectations Requires a culture of sharing
14. Systematic Validation (consensus)Indicators Yes/No ActionIs there a plan for assessment validation (including validation ofRPL assessment) in place?Does your plan:•Determine the sample of units of competency to be validated overa set period of time•Provide dates for proposed validation activities•Include details about who will participate in assessmentvalidation, including the Chair of consensus panels, if relevant•Include a strategy to ensure that all relevant staff are involved•Identify what processes and materials will be used forimplementing and recording the outcomes of assessmentvalidationDoes your RTO have terms of reference in place to guide the workof consensus panels?Does your RTO have validation materials (policy, procedure, forms)in place that cause participants to engage effectively in validation?Does your RTO have a process for monitoring the action taken as aresult of validation?Does your RTO have a process and plan in place for reviewing the Ref: Assessor Guideeffectiveness of assessment validation?
15. System considerations What is the most appropriate approach to validation?Condition Suggested approachWhenever my RTO conducts internal Consider including external representationvalidation few opportunities for improvement on your validation panelariseOur assessors are contractors and cannot Consider establishing assessor validationcome to validation consensus meetings partnerships at your local level, but ensurebecause my RTO can’t afford to pay for their that improvements identified are recordedtime and some are located interstate and fed back to other assessors and formalisedOur RTO conducts high risk units related to Consider consensus moderation, ideallylicensing, where the licensing authority has with external representation on your panel.mandated the use of assessment tools itprovidesOur RTO is new and assessors do not have Consider inviting an external person witha lot of experience expertise in assessment tool design to validation consensus meetings
16. Assessment Quality Management Quality Assurance Quality Control Quality Review (Input approach) (Outcome approach) (Retrospective approach)Examples include: Examples include: Examples Include: Industry competency standards Moderation in which adjustments to Monitoring and auditing of registered as the benchmarks for assessor judgements are made to training organisations assessment overcome differences in the difficulty of the assessment tool and/or Review and validation of assessment National assessment principles severity of the judgement. tools, processes and outcomes to identify future improvements. Minimum qualifications for assessors (i.e., TAE40110) Follow-up surveys with key stakeholders (e.g., student destination Development of a Professional surveys, employer feedback on how Code of Practice well the assessment outcomes predicted workplace performance). Standardisation of reporting formats Assessment Guidelines and Policy Documents Benchmark examples of varying levels of performances Assessment tool banks Common assessment tasks Exemplar assessment tools Panelling, Piloting and/or Trialling of assessment tools. Professional development programs/workshops for assessors
17. Quality management in diverse settings Identified barriers: Structural (i.e., the organizational and resource aspects) – financial, variations of definitions across key documents Process (i.e., the practices and activities that take place) – rolling enrolments, partnering arrangements, workloads Personal factors (i.e., the attitudinal, assessment literacy and expectations of the key players). Strategies deployed by RTOs Refer to Handout – Quality management processes in diverse settings.