Setting and assessing learning standards
•Download as PPTX, PDF•
0 likes•437 views
Presentation at meeting of Australian Learning and Teaching Fellows on 23 April 2012.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (17)
Similar to Setting and assessing learning standards
Similar to Setting and assessing learning standards (20)
Recently uploaded
Recently uploaded (20)
Setting and assessing learning standards
- 1. Setting and assessing standards Status, options and challenges Mark Freeman mark.freeman@sydney.edu.au mark.freeman@abdc.edu.au
- 2. Outline 1. Regulatory context 2. Setting learning standards 3. Assessing learning standards 4. Evaluating moderation options 5. Q & A Support for this project has been provided by the Australian Business Deans Council, the Institute of Chartered Accountants in Australia, CPA Australia and the Australian Government Office for Learning and Teaching. The views expressed in this presentation do not necessarily reflect the views of any of these stakeholders.
- 3. I have I can‟t I said that I had taught hear him taught him, not Snoopy whistle that he had to whistle learned
- 4. Definitions Standards “a definite level of excellence or attainment, or a definite degree of any quality viewed as a prescribed object of endeavour or as the recognised measure of what is adequate for some purpose, so established by authority, custom, or consensus” (Sadler, 2009) Learning outcomes “the expression of the set of knowledge, skills and the application of the knowledge and skills a person has acquired and is able to demonstrate as a result of learning” (AQF, 2011) Learning standards “the explicit levels of attainment required of and achieved by students and graduates, individually and collectively, in defined areas of knowledge and skills” (DEEWR, 2011)
- 5. It’s coming!
- 7. Provider Threshold Qualifications TEQSA Teaching and Learning Research Information “At this point [the learning and teaching standards] are not threshold standards [but] (ie. internal „Fitness-for-purpose‟ „standards‟ external) what the government may choose to do in the Higher Education Standards Panel consult Ministers future remains to be seen.“ Commissioners Teaching standards separate from learning standards The Australian 24 Aug 2011
- 8. TEQSA legislation ...take account of external standards.. e.g. published discipline standards... ...standards intended ...and ..actually achieved ....are benchmarked ...awards ...meet the corresponding specifications ...described in the AQF TEQSA regulatory risk framework
- 9. T & L Standards Discussion Paper - principle 3 & 5 “TEQSA is not the only custodian of standards, nor are higher education institutions. This responsibility is distributed and shared more widely, including with disciplinary communities and professional associations” “Institutional standards for teaching and learning will differ but all institutions must meet or surpass national standards”
- 10. Setting learning standards “Discipline communities will „own‟ and take responsibility for implementing teaching and learning standards (working with professional bodies and other stakeholders where appropriate) within the academic traditions of collegiality, peer review, pre-eminence of disciplines and, importantly, academic autonomy” DEEWR (2009, p. 32)
- 11. 9 discipline groups in 4 waves – 11 sets defined Jul’09 Arts, Social Sciences & Humanities Business, Management & Economics Engineering & ICT Feb’10 Creative & Performing Arts Health, Medicine & Vet Science Law Jul’10 Architecture, Design & Building Science Feb’11 Education
- 12. Business, Management and Economics Accounting Degrees Banking, Finance & Related Fields • Bachelor Business Information Studies • Management Business Master (Entry) • Master (Advanced) Economics and Econometrics Hospitality Management Cycles Human Resource Industrial Relations setting 1. Agenda Management 2. Awareness raising 3. Consultation International Business Organisation Management 4. Dissemination Marketing Sport and Recreation Engagement • 2,100 participants nationally Tourism Management Missing• (eg. Australian universities 38 Logistics) • 21 private/other providers • 20 others (eg professional and peak bodies)
- 13. Provider A Provider B Provider C Provider D Judgement Judgement Judgement Judgement + Knowledge Knowledge Knowledge Knowledge + Application Application Application Application + Communication Communication Communication + Communication + & Teamwork & Teamwork & Teamwork & Teamwork Self Self Self Self + Management Management Management Management Small business Quantitative Professional Public sector Regional Chalk + talk Online learning Problem-based Team-based learning learning learning
- 17. Q1 & 2 Indiv Small group Indiv
- 19. Assessing learning standards 1. Perceptions – employers, graduates, professional bodies eg. AGS/CEQ; professional body accreditation 2. Common test – ACER, CLA, AHELO “many graduates already subjected to skills testing for employment” 3. External moderation – UK, Go8, Krause, ABDC-Prof Bodies “Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities” (Tenet 6: Price et al, 2008)
- 20. Moderation initiatives QVS Krause-Scott et al Achievement Mat Scope Multiple Multiple Accounting Level Bachelor Bachelor Bach + Mast HEI grouping Go8 11 across Start 10 across Reviewers 1 academic 1 academic 2 aca/professionals Data selection Stratified Stratified Randomised Sample size 5% HD/D/C/P/F 1 HD/D/C/P/F per partner 5 Products All unit‟s tasks All unit‟s tasks Specific threshold Intent Verification (QA) Qlty assurance & Qlty QA & QE Enhancement (QE) Authority Top-down Top-down Bottom-up
- 21. Achievement Matters Project Aims 1. Evidence of accounting academic standards • External, double-blind, peer-reviewed • Benchmark against national consensus (Bachelor & Master) • All HEP types 2. A model process for obtaining and using evidence • Assessing inputs & outputs • Quality enhancement & assurance 3. Professional learning and capacity building Rationale: Improve, self-regulate, avoid perverse options Pilot: Adelaide, Curtin, Deakin, Griffith, Monash, RMIT, Southern Cross, Sydney, USQ, UWA, UWS
- 22. Pilot cycle: Threshold standard written communication Graduates of a Bachelor/Master (Entry) degree would be expected to justify and communicate accounting advice and ideas in straightforward/diverse collaborative contexts involving both accountants and non-accountants. Master (entry): Diverse = Several competing or new qualitative perspectives and/or quantitative perspectives characterised by considerable data items, over multiple variables and known relationships between them. Bachelor: Straightforward = few qualitative perspectives and/or quantitative perspectives characterised by considerable data items over multiple variables and known relationships between them
- 23. Reaching consensus on assessment task validity Assess Enter Compare Pre-F2F Consensus Agree F2F Implement Post-F2F 2
- 24. Calibrating and grading to the standard Assess Enter Compare Pre-F2F Consensus Agree F2F Apply • to assignment if student Post-F2F • to marking if faculty 2
- 25. I‟m confident rating assessment requirements and students‟ work
- 26. Calibration – task validity Individual results pre-workshop • Min & max (n=26) NA A • Mean ±1 SD Group results at workshop • Small groups (n=5) NA A • Consensus
- 27. Calibration – UG student 1 Individual results pre-workshop • Min & max (n=26) NM M • Mean ±1 SD Group results at workshop • Small groups (n=5) NM M • Consensus
- 28. Calibration – PG student 1 Individual results pre-workshop • Min & max (n=26) NM M • Mean ± 1 SD Group results at workshop • Small groups (n=5) NM M • Consensus
- 29. Confirmation – PG student 5 Individual results at workshop • Min & max (n=20) NM M • Mean ±1 SD Group results at workshop • Small groups (n=5) NM M • Consensus
- 30. Participant feedback Having to enter my feedback into SPARK caused me to reflect on the reasons for my judgement I expect this project will help establish national agreement on academic standards between accounting degree providers and with employers
- 31. Impact on academics Strongly Strongly Disagree Agree Post-workshop 3 Pre-workshop 3 Pre-workshop 1
- 33. 3
- 34. Q3 & 4 Indiv Small group Indiv
- 35. Challenges TEQSA 1. How will standards be set and monitored in a way that is sensible, fair, accepted and still economic? HEI 1. How should we engage in disciplines setting standards? 2. How should agreed disciplinary learning standards be implemented into our curriculum? 3. How can we best participate in collaborative initiatives assessing achievement against national benchmarks? 4. How should evidence from our participation in national moderation projects be reported and used? 5. What systems changes and professional development are needed here to prepare for the standards agenda?
- 36. Thank you
Editor's Notes
- Linus has the blanket and is talking to Charlie Brown
- TEQSA - will be a statutory authority of the Federal Government established in the second half of 2010 and >AUQA functionsProvider standards – National protocols and ESOS (eg. appro staffing profile to deliver educational niche; academics undertaking scholarship, and research if supervising HDR students) ……http://www.mceecdya.edu.au/mceecdya/national_protocols_for_higher_education_mainpage,15212.html Qualification standards: Australian Qualifications FrameworkLearning and Teaching Standards: Academic Standards, Learning outcomesResearch standards: Excellence in Research in AustraliaInformation standards – for the market and regulatorsQA model not yet finalized, but it is likely it more expansive and have a stronger remit over AUQA. From 2012 TEQSA will: build upon self-regulation processes and will accredit providers unis in 2012 and 2013 VETevaluate the performance of institutions and programs against nationally-agreed standards (measured by minimum learning outcomes); evaluate at-risk areas using a proportionate model; use a range of sanctions to promote high quality encourage best practiceprovide greater national consistency
- The course of study meets the Qualification Standards.1.2 There are robust internal processes for design and approval of the course of study, which:• provide realistic projections of the demand and resources required for the course of study;• take account of external standards and requirements, e.g. published discipline standards, professional accreditation, input from relevant external stakeholders, and comparable standards at other higher education providers; and,• provide for appropriate development of key graduate attributes in students including English language proficiency.2.1 Resourcing for the course of study is adequate to meet the higher education provider’s projected enrolments for the course of study and for students to achieve the expected learning outcomes3.1 Admission criteria for the course of study:• are appropriate for the Qualification Standards level of the course of study and required learning outcomes;3.2 The higher education provider ensures that students who are enrolled are sufficiently competent in the English language to participate effectively in the course of study and achieve its expected learning outcomes, and sets English language entry requirements accordingly.4.2 The higher education provider ensures that staff who teach students in the course of study:• are appropriately qualified in the relevant discipline for their level of teaching (qualified to at least one Qualification Standards level higher than the course of study being taught or with equivalent professional experience);• have a sound understanding of current scholarship and/or professional practice in the discipline that they teach;• have an understanding of pedagogical and/or adult learning principles relevant to the student cohort being taught;• engage students in intellectual inquiry appropriate to the level of the course of study and unit being taught; and,• are advised of student and other feedback on the quality of their teaching and have opportunities to improve their teaching.5.4 The higher education provider maintains, monitors and acts on comparative data on the performance of students in the course of study, including information on the performance of student cohorts by entry pathway, mode of study and place of study, such data to include: student attrition; student progress; course completions; and grade distributions.5.5 The academic standards intended to be achieved by students and the standards actually achieved by students in the course of study are benchmarked against similar accredited courses of study offered by other higher education providers.5.6 The higher education provider is able to demonstrate appropriate progression and completion rates and that students who complete the course of study have attained key graduate attributes including an appropriate level of English language proficiency.
- Additional LO and higher standards beyond threshold LO quite possible and very much encoruaged as part of Total number of meetings held: 489Total number of attendees at meetings: 7090 Number of persons on your email contact list: 3579
- Diversity protectedIndividual HEI choose additional LO (eg. 9 if 7 threshold LO and wish to emphasise something like behavioural or quantiative outcomes); higher learning outcomes (eg. a pass at Uni XXX might be far in excess of the threshold)Autonomy protectedIndividual HEI choose how deliver (eg. PBL, lect/tutes; WIL; TBL) & assess (capstones; e-portfolio)Reputations protected Sample graded work is reviewed by registered peer experts who review HEP standards against threshold LOSupports accreditation & benchmarkingAcademics and professionals working together (extending prof and international accreditation)HOME – Opportunity for 1 or 2 questions before moving on?
- Beverley What capabilitiesshould be(lots)How design to implement and map (some)How do we evidence them ( a few)How improve from benchmarking (almost none)Participating providers nominate their peer reviewers and implement the necessary processes to identify and collect the relevant inputs and outputs. Five pieces of student sample work for each learning outcome under review from each HEP will be randomly selected, de-identified and sent to the Independent Assessment Data Coordinator (IADC). Related assessment requirements and supporting documentation will also be submitted. The required number of pieces is considered sufficient to gauge standards without making the process too onerous, but will be re-evaluated after the pilot. Prior to peer review, each participant will undergo training in the software (SPARKPLUS) used to collect peer reviews and grade exemplars to calibrate standards. The first national face-to-face workshop is in Darwin immediately following the AFAANZ conference.The IADC will oversee anonymity of sample work (and assessment requirements) and allocate two peers for (double blind) review of the work. Each item sampled will be graded online as ‘meeting’ or ‘not meeting’ the threshold learning outcomes. Each HEP will submit their own grading of meeting/not meeting benchmarked against the specific national thresholds under review. The latter could well be different to the grade the piece received originally when all assessment criteria were considered. There will be no change to the mark awarded to the student by the HEP arising from any peer review feedback.The IADC will oversee tracking, aggregation and return of confidential anonymous reviews to the relevant HEP, together with the aggregate results. Only individual qualitative comments are returned initially to promote internal quality enhancement discussions at each HEP.Participants discuss possible intended quality enhancements to their respective programs at a face-to-face forum and clarify explanations for grade variances from national standards allocated by peer reviewers (distributed at the forum).
- Beverley What capabilitiesshould be(lots)How design to implement and map (some)How do we evidence them ( a few)How improve from benchmarking (almost none)Participating providers nominate their peer reviewers and implement the necessary processes to identify and collect the relevant inputs and outputs. Five pieces of student sample work for each learning outcome under review from each HEP will be randomly selected, de-identified and sent to the Independent Assessment Data Coordinator (IADC). Related assessment requirements and supporting documentation will also be submitted. The required number of pieces is considered sufficient to gauge standards without making the process too onerous, but will be re-evaluated after the pilot. Prior to peer review, each participant will undergo training in the software (SPARKPLUS) used to collect peer reviews and grade exemplars to calibrate standards. The first national face-to-face workshop is in Darwin immediately following the AFAANZ conference.The IADC will oversee anonymity of sample work (and assessment requirements) and allocate two peers for (double blind) review of the work. Each item sampled will be graded online as ‘meeting’ or ‘not meeting’ the threshold learning outcomes. Each HEP will submit their own grading of meeting/not meeting benchmarked against the specific national thresholds under review. The latter could well be different to the grade the piece received originally when all assessment criteria were considered. There will be no change to the mark awarded to the student by the HEP arising from any peer review feedback.The IADC will oversee tracking, aggregation and return of confidential anonymous reviews to the relevant HEP, together with the aggregate results. Only individual qualitative comments are returned initially to promote internal quality enhancement discussions at each HEP.Participants discuss possible intended quality enhancements to their respective programs at a face-to-face forum and clarify explanations for grade variances from national standards allocated by peer reviewers (distributed at the forum).
- Indiv: 64707263938758799287757479477669635753708280777790 Av=73.36Stdev=12.17881Teams:3643413945Consensus: 40
- Indiv: 64707263938758799287757479477669635753708280777790 Av=73.36Stdev=12.17881Teams:3643413945Consensus: 40
- Indiv: 1628313131333436394343444445454647505253566063759094 Av=47.26923 SD=18.01679Teams:3643413945Consensus: 40
- Indiv: 7354655968575286687468625761636653444161.6315810.65213 Av=61.6 Stdev=10.7Teams:6660736549Consensus: 60
- INTERACTION 2: HWO MIGHT YOU APPLY THIS PROCESS IN YOUR CONTEXT?
- TEQSA will have best practice database – but what else?Questions for GU?Eg. how will GU disciplines provide valid and reliable evidence of LS and TS? How will GU undertake curriculum renewal processes to ensure PLO are achieved?