Download to read offline
Uploaded byMark Freeman
Using 'double blind peer review' to assure academic standards
This document summarizes a presentation about how the accounting field uses double blind peer review (DBPR) to ensure learning outcomes are achieved. It discusses how DBPR involves anonymous external reviewers calibrating their reviews of de-identified student work. The results are used by institutions to benchmark standards against other programs and identify areas for improvement. The process is coordinated across multiple Australian universities and institutions to establish a national benchmarking system for assessing accounting learning standards.
More Related Content
Similar to Using 'double blind peer review' to assure academic standards
Using 'double blind peer review' to assure academic standards
- 1. Proving to improve:how Accounting uses double blind peer review to assure TLOs CUBEnet Forum Canberra 13 Dec 2013
- 2.
- 3. 1.2 There arerobust internal processes for design and approval of the course of study which take account of external standards and requirements, e.g. published discipline standards …, and comparable standards at other higher education providers. Scott Ray 4/4/09 http://www.flickr.com/photos/15075617@N05/3418528723
- 4. Knowledge Self management Application Accounting graduate Master (entry) Bachelor graduatesare able to justify and communicate accounting advice and ideas in straightforward diverse collaborative contexts involving both accountants and non-accountants Communication & Teamwork Judgement
- 5. Knowledge Self management Application Accounting graduate Masters emerging /advanced Communication & Teamwork Judgement minimal Bachelor graduatesare able to exercise judgement under supervision to solve routine accounting problems in straightforward contexts using social, ethical, economic, regulatory and global perspectives complex
- 6. 5.5 The academic standardsintended to be achieved by students and the outcomes actually achieved by students in the course of study are benchmarked against similar accredited courses of study.” Martin Morris 20/2/12 http://www.flickr.com/photos/martin55/6929721749/ “external AoL”
- 7.
- 8. Calibration – Taskvalidity Written communication 91% Individual confidence pre-workshop Individual results pre-workshop • Min & max (n=26) NA A • Mean 1 SD Group results at workshop • Small groups (n=5) • Consensus NA A
- 9. Calibration – UGstudent 1 ....S2, S3, Written communication S4, S5 78% Individual confidence pre-workshop Individual results pre-workshop • Min & max (n=26) • Mean 1 SD NM M NM M Group results at workshop • Small groups (n=5) • Consensus
- 10. Provider selects reviewers Project selects standards& degree for review Provider uses results (Project ensures consensus) 3 anonymous reviewers submit (Project allocates 2 externals) External calibration & review Provider selects tasks Project generates random student ID to submit Reviewers calibrate Provider submits de-identified data
- 11. Feedback to submittinginstitution Summary of results • Using the results • Next steps
- 12. Feedback to submittinginstitution Appendix 1 – detailed comments
- 13. Your uni predictsthat the BSc (BioMed) is likely to be selected by TEQSA as a sample degree for scrutiny in 2015. How would you demonstrate external AOL? 1. Rely on sample data reviewed as part of QVS 2. Select benchmarking partner dept to assess sample assignments & exams in final UoS for 5% students in each grade band 3. Pay an external academic to assess 4. Lobby ACDS to establish a national ‘double blind peer review’ benchmarking process that includes calibration 5. Lobby CUBEnet to do DBPR benchmarking & calibration 6. Do nothing
- 14. Achievement Matters: ExternalPeer Review of Accounting Learning Standards Australian Business Deans Council Institute of Chartered Accountants in Australia CPA Australia Office for Learning and Teaching Project team : Adelaide (B.Howieson), Deakin (K.Watty), RMIT (B.O’Connell, P.de Lange), Sydney (M.Freeman), UWA (P. Hancock), UWS (A.Abraham) Participants from 17 providers Websites achievementmatters.com.au disciplinestandards.pbworks.com Disclaimer The views expressed in this presentation do not necessarily reflect the views of any of these stakeholders
- 15.
Editor's Notes
- #2 In order to play in many fields you need to meet a minimum standard.For example: Womens high jump introduced in 1928 Olympics. – Ethel Catherwood (Canada) jumped 1.59m (5 ft 2 inches). But 2012 Olympics qualification standard was 1.96m (6ft 5inches – set by Intern Assoc of Athletics Federation). Like deregulation of banking sector required APRA’s role to be strengthened, moving to demand-driven HE system has meant a new & stronger regulator. TEQSA (on video) has invited us to take responsibility for our future by setting and assessing academics standards Outline for today – we will cover some Background (including process)State of playLearningsNext steps
- #3 15:28-22:3042:13-
- #4 External – AQF and Discipline standards
- #5 2010Interprets AQF ie. eg. develop TLO for accounting bachelors (lev 7) and coursework masters (lev 9)degreesBuilds on previous program focus eg. AACSB assurance of learning where program LO matter (not unit of study LO)Highly consultative - Meet with multiple levels of distributed leaders. Multiple cycles. In total 2,100 participants nationally, representing 38 Australian universities, 21 private/other providers, 20 others (eg professional and peak bodies)Five LS - Fewer thresholds than any other discipline. Acctngdepts encouraged to diversely build (eg. Internationalisation)Resulted in jointly-funded initiative to assess learning outcomes benchmarked against standardsResulted in other business disciplines setting standards using same approach (eg. Marketing, econs)
- #6 2010Interprets AQF ie. eg. develop TLO for accounting bachelors (lev 7) and coursework masters (lev 9)degreesBuilds on previous program focus eg. AACSB assurance of learning where program LO matter (not unit of study LO)Highly consultative - Meet with multiple levels of distributed leaders. Multiple cycles. In total 2,100 participants nationally, representing 38 Australian universities, 21 private/other providers, 20 others (eg professional and peak bodies)Five LS - Fewer thresholds than any other discipline. Acctngdepts encouraged to diversely build (eg. Internationalisation)Resulted in jointly-funded initiative to assess learning outcomes benchmarked against standardsResulted in other business disciplines setting standards using same approach (eg. Marketing, econs)
- #7 HE is quite unlike other top export industries (iron ore, coal and gold), since the quality or standard of HE can not be measured exactly. Hence the importance for TEQSA to monitor and protect “Brand Australia”.HE is not like high jump where the standard can be measured exactly. It is more like diving where experienced judges make independent decisions. In fact this mirrors what is valued in vetting a manuscript for publication – it is double-blind peer reviewed.
- #9 “I am confident rating the capacity of assessment requirements to allow students to demonstrate the national learning standard for oral communication in accounting”Indiv: 64707263938758799287757479477669635753708280777790 Av=73.36Stdev=12.17881Teams:3643413945Consensus: 40
- #10 “I am confident rating students' written communication ability benchmarked against the national standard”Indiv: 64707263938758799287757479477669635753708280777790 Av=73.36Stdev=12.17881Teams:3643413945Consensus: 40
- #15 Just flash upStop presentation here