DisclaimerThis work has been produced on behalf of the NationalQuality Council with funding provided through the AustralianGovernment Department of Education, Employment andWorkplace Relations and state and territory governments.The views expressed in this work are not necessarily thoseof the Australian Government or state and territorygovernments
AcknowledgementThis presentation was designed to support the interactiveinformation sessions that formed part of the NQC’scommunication and dissemination strategy: NQC products:validation and communication. Reports and materials whichfocus on validation and moderation may be downloadedfrom the NQC website athttp:www.nqwc.tvetaustralia.com.au/nqc_publicationsThis work was produced for the National Quality Council by Andrea Bateman, Quorum QA Australia Pty Ltd Chloe Dyson, Quorum QA Australia Pty Ltd
Validation Validation is a quality review process. It involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes.NQC Implementation Guide: Validation and Moderation 2009
Outcomes of validationRecommendations for future improvements Context and conditions for the assessment Task/s to be administered to the candidates Administration instructions Criteria used for judging the quality of performance (e.g. the decision making rules, evidence requirements etc) Guidelines for making reasonable adjustments to the way in which the evidence of performance was gathered to ensure that the expected standard of performance specified within the Unit(s) of Competency has not been altered Recording and reporting requirements.
Moderation Moderation is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.NQC Implementation Guide: Validation and Moderation 2009
Outcomes of moderation Recommendations for future improvement and adjustments to assessor judgements (if required) and Recommendations for improvement to the assessment tools Adjusting the results of a specific cohort of candidates prior to the finalisation of results and Requesting copies of final candidate assessment results in accordance with recommended actions.
Validation vs ModerationFeatures Validation ModerationAssessment Quality Review Quality ControlQualityManagementTypePrimary Purpose Continuous improvement Bring judgements and standards into alignment.Timing On-going Prior to the finalisation of candidate resultsFocus Assessment tools; and Assessment tools, and; Candidate Evidence Candidate Evidence, (including including assessor assessor judgements) judgements (mandatory) (desirable only)Type of Assessor Partnerships Consensus MeetingsApproaches Consensus Meetings External (moderators or External (validators or panels)
Types of Approaches - Statistical Limited to moderation Yet to be pursued at the national level in VET Requires some form of common assessment task at the national level Adjusts level and spread of RTO based assessments to match the level and spread of the same candidates scores on a common assessment task Maintains RTO-based rank ordering but brings the distribution of scores across groups of candidates into alignmentStrength Strongest form of quality controlWeakness Lacks face validity, may have limited content validity
Types of Approaches - External Types Site Visit Versus Central AgencyStrengths Offer authoritative interpretations of standards Improve consistency of standards across locations by identifying local bias and/or misconceptions (if any) EducativeWeakness Expensive Less control than statistical
Types of Approaches – Assessor Partnerships Validation only Informal, self-managed, collegial Small group of assessors May involve: Sharing, discussing and/or reviewing one another’s tools and/or judgements Benefit Low costs, personally empowering, non-threatening May be easily organised Weakness Potential to reinforce misconceptions and mistakes Ref: Implementation Guide, Assessment Fact
Types of Approaches - Consensus Typically involves reviewing their own & colleagues assessment tools and judgements as a group Can occur within and/or across organisations Strength Professional development, networking, promotes collegiality and sharing Weakness Less quality control than external and statistical approaches as they can also be influenced by local values and expectations Requires a culture of sharing
Systematic Validation (consensus)Indicators Yes/No ActionIs there a plan for assessment validation (including validation ofRPL assessment) in place?Does your plan:•Determine the sample of units of competency to be validated overa set period of time•Provide dates for proposed validation activities•Include details about who will participate in assessmentvalidation, including the Chair of consensus panels, if relevant•Include a strategy to ensure that all relevant staff are involved•Identify what processes and materials will be used forimplementing and recording the outcomes of assessmentvalidationDoes your RTO have terms of reference in place to guide the workof consensus panels?Does your RTO have validation materials (policy, procedure, forms)in place that cause participants to engage effectively in validation?Does your RTO have a process for monitoring the action taken as aresult of validation?Does your RTO have a process and plan in place for reviewing the Ref: Assessor Guideeffectiveness of assessment validation?
System considerations What is the most appropriate approach to validation?Condition Suggested approachWhenever my RTO conducts internal Consider including external representationvalidation few opportunities for improvement on your validation panelariseOur assessors are contractors and cannot Consider establishing assessor validationcome to validation consensus meetings partnerships at your local level, but ensurebecause my RTO can’t afford to pay for their that improvements identified are recordedtime and some are located interstate and fed back to other assessors and formalisedOur RTO conducts high risk units related to Consider consensus moderation, ideallylicensing, where the licensing authority has with external representation on your panel.mandated the use of assessment tools itprovidesOur RTO is new and assessors do not have Consider inviting an external person witha lot of experience expertise in assessment tool design to validation consensus meetings
Assessment Quality Management Quality Assurance Quality Control Quality Review (Input approach) (Outcome approach) (Retrospective approach)Examples include: Examples include: Examples Include: Industry competency standards Moderation in which adjustments to Monitoring and auditing of registered as the benchmarks for assessor judgements are made to training organisations assessment overcome differences in the difficulty of the assessment tool and/or Review and validation of assessment National assessment principles severity of the judgement. tools, processes and outcomes to identify future improvements. Minimum qualifications for assessors (i.e., TAE40110) Follow-up surveys with key stakeholders (e.g., student destination Development of a Professional surveys, employer feedback on how Code of Practice well the assessment outcomes predicted workplace performance). Standardisation of reporting formats Assessment Guidelines and Policy Documents Benchmark examples of varying levels of performances Assessment tool banks Common assessment tasks Exemplar assessment tools Panelling, Piloting and/or Trialling of assessment tools. Professional development programs/workshops for assessors
Quality management in diverse settings Identified barriers: Structural (i.e., the organizational and resource aspects) – financial, variations of definitions across key documents Process (i.e., the practices and activities that take place) – rolling enrolments, partnering arrangements, workloads Personal factors (i.e., the attitudinal, assessment literacy and expectations of the key players). Strategies deployed by RTOs Refer to Handout – Quality management processes in diverse settings.