Setting and assessing learning standards


Published on

Presentation at meeting of Australian Learning and Teaching Fellows on 23 April 2012.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Setting and assessing learning standards

  1. 1. Setting and assessing standards Status, options and challenges Mark
  2. 2. Outline 1. Regulatory context 2. Setting learning standards 3. Assessing learning standards 4. Evaluating moderation options 5. Q & ASupport for this project has been provided by the Australian Business Deans Council, the Institute of CharteredAccountants in Australia, CPA Australia and the Australian Government Office for Learning and Teaching. Theviews expressed in this presentation do not necessarily reflect the views of any of these stakeholders.
  3. 3. I have I can‟t I said that I hadtaught hear him taught him, notSnoopy whistle that he hadto whistle learned
  4. 4. DefinitionsStandards“a definite level of excellence or attainment, or a definite degree of anyquality viewed as a prescribed object of endeavour or as the recognisedmeasure of what is adequate for some purpose, so established byauthority, custom, or consensus” (Sadler, 2009)Learning outcomes“the expression of the set of knowledge, skills and the application of theknowledge and skills a person has acquired and is able to demonstrate asa result of learning” (AQF, 2011)Learning standards“the explicit levels of attainment required of and achieved by students andgraduates, individually and collectively, in defined areas of knowledge andskills” (DEEWR, 2011)
  5. 5. It’s coming!
  6. 6. Provider Threshold QualificationsTEQSA Teaching and Learning Research Information “At this point [the learning and teaching standards] are not threshold standards [but] (ie. internal „Fitness-for-purpose‟ „standards‟ external) what the government may choose to do in the Higher Education Standards Panel consult Ministers future remains to be seen.“ Commissioners Teaching standards separate from learning standards The Australian 24 Aug 2011
  7. 7. TEQSA legislation...take account of external standards.. e.g. published discipline standards......standards intended ...and ..actually achieved ....are benchmarked...awards the corresponding specifications ...described in the AQFTEQSA regulatory risk framework
  8. 8. T & L Standards Discussion Paper - principle 3 & 5“TEQSA is not the only custodian of standards, nor arehigher education institutions. This responsibility isdistributed and shared more widely, including withdisciplinary communities and professional associations”“Institutional standards for teaching and learning will differbut all institutions must meet or surpass national standards”
  9. 9. Setting learning standards “Discipline communities will „own‟ and take responsibility for implementing teaching and learning standards (working with professional bodies and other stakeholders where appropriate) within the academic traditions of collegiality, peer review, pre-eminence of disciplines and, importantly, academic autonomy” DEEWR (2009, p. 32)
  10. 10. 9 discipline groups in 4 waves – 11 sets defined Jul’09 Arts, Social Sciences & Humanities Business, Management & Economics Engineering & ICT Feb’10 Creative & Performing Arts Health, Medicine & Vet Science Law Jul’10 Architecture, Design & Building Science Feb’11 Education
  11. 11. Business, Management and EconomicsAccounting Degrees Banking, Finance & Related Fields • BachelorBusiness Information Studies • Management Business Master (Entry) • Master (Advanced)Economics and Econometrics Hospitality Management CyclesHuman Resource Industrial Relations setting 1. AgendaManagement 2. Awareness raising 3. ConsultationInternational Business Organisation Management 4. DisseminationMarketing Sport and Recreation Engagement • 2,100 participants nationallyTourism Management Missing• (eg. Australian universities 38 Logistics) • 21 private/other providers • 20 others (eg professional and peak bodies)
  12. 12. Provider A Provider B Provider C Provider DJudgement Judgement Judgement Judgement +Knowledge Knowledge Knowledge Knowledge +Application Application Application Application +Communication Communication Communication + Communication +& Teamwork & Teamwork & Teamwork & TeamworkSelf Self Self Self +Management Management Management Management Small businessQuantitative ProfessionalPublic sector RegionalChalk + talk Online learning Problem-based Team-basedlearning learning learning
  13. 13. Q1 & 2Indiv Small group Indiv
  14. 14. Assessing learning standards
  15. 15. Assessing learning standards1. Perceptions – employers, graduates, professional bodieseg. AGS/CEQ; professional body accreditation2. Common test – ACER, CLA, AHELO“many graduates already subjected to skills testing for employment”3. External moderation – UK, Go8, Krause, ABDC-Prof Bodies“Assessment is largely dependent upon professional judgement andconfidence in such judgement requires the establishment of appropriateforums for the development and sharing of standards within and betweendisciplinary and professional communities” (Tenet 6: Price et al, 2008)
  16. 16. Moderation initiatives QVS Krause-Scott et al Achievement MatScope Multiple Multiple AccountingLevel Bachelor Bachelor Bach + MastHEI grouping Go8 11 across Start 10 acrossReviewers 1 academic 1 academic 2 aca/professionalsData selection Stratified Stratified RandomisedSample size 5% HD/D/C/P/F 1 HD/D/C/P/F per partner 5Products All unit‟s tasks All unit‟s tasks Specific thresholdIntent Verification (QA) Qlty assurance & Qlty QA & QE Enhancement (QE)Authority Top-down Top-down Bottom-up
  17. 17. Achievement Matters ProjectAims1. Evidence of accounting academic standards • External, double-blind, peer-reviewed • Benchmark against national consensus (Bachelor & Master) • All HEP types2. A model process for obtaining and using evidence • Assessing inputs & outputs • Quality enhancement & assurance3. Professional learning and capacity buildingRationale: Improve, self-regulate, avoid perverse optionsPilot: Adelaide, Curtin, Deakin, Griffith, Monash, RMIT, Southern Cross, Sydney, USQ, UWA, UWS
  18. 18. Pilot cycle: Threshold standard written communication Graduates of a Bachelor/Master (Entry) degree would be expected to justify and communicate accounting advice and ideas in straightforward/diverse collaborative contexts involving both accountants and non-accountants. Master (entry): Diverse = Several competing or new qualitative perspectives and/or quantitative perspectives characterised by considerable data items, over multiple variables and known relationships between them. Bachelor: Straightforward = few qualitative perspectives and/or quantitative perspectives characterised by considerable data items over multiple variables and known relationships between them
  19. 19. Reaching consensus on assessment task validity Assess Enter ComparePre-F2F Consensus AgreeF2F ImplementPost-F2F 2
  20. 20. Calibrating and grading to the standard Assess Enter ComparePre-F2F Consensus AgreeF2F Apply • to assignment if studentPost-F2F • to marking if faculty 2
  21. 21. I‟m confident rating assessment requirements and students‟ work
  22. 22. Calibration – task validityIndividual results pre-workshop• Min & max (n=26) NA A• Mean ±1 SDGroup results at workshop• Small groups (n=5) NA A• Consensus
  23. 23. Calibration – UG student 1Individual results pre-workshop• Min & max (n=26) NM M• Mean ±1 SDGroup results at workshop• Small groups (n=5) NM M• Consensus
  24. 24. Calibration – PG student 1Individual results pre-workshop• Min & max (n=26) NM M• Mean ± 1 SDGroup results at workshop• Small groups (n=5) NM M• Consensus
  25. 25. Confirmation – PG student 5Individual results at workshop• Min & max (n=20) NM M• Mean ±1 SDGroup results at workshop• Small groups (n=5) NM M• Consensus
  26. 26. Participant feedbackHaving to enter my feedback into SPARKcaused me to reflect on the reasons for myjudgementI expect this project will help establishnational agreement on academic standardsbetween accounting degree providers andwith employers
  27. 27. Impact on academics Strongly Strongly Disagree Agree Post-workshop 3 Pre-workshop 3 Pre-workshop 1
  28. 28. 3
  29. 29. Q3 & 4Indiv Small group Indiv
  30. 30. ChallengesTEQSA1. How will standards be set and monitored in a way that is sensible, fair, accepted and still economic?HEI1. How should we engage in disciplines setting standards?2. How should agreed disciplinary learning standards be implemented into our curriculum?3. How can we best participate in collaborative initiatives assessing achievement against national benchmarks?4. How should evidence from our participation in national moderation projects be reported and used?5. What systems changes and professional development are needed here to prepare for the standards agenda?
  31. 31.  Thank you