Using external assurance of learning as prove to stakeholders and to improve practice

1. External Assurance of Learning Models ‐
Accountability and Quality Improvement
ABDC 25 July 2014
www.slideshare.net/markalistairfreeman

2. Achievement Matters: External Peer Review of Accounting Learning Standards
Australian Business Deans Council
CPA Australia
Institute of Chartered Accountants in Australia
Office for Learning and Teaching
Project team : Adelaide (B.Howieson), Deakin (K.Watty), RMIT (B.O’Connell,
P.de Lange), Sydney (M.Freeman), UWA (P. Hancock), UWS (A.Abraham)
Participants from 17 providers
Websites
achievementmatters.com.au
disciplinestandards.pbworks.com
Disclaimer
The views expressed in this presentation do not necessarily reflect the views of
any of these stakeholders

3. Standard
“a definite level of excellence or attainment.....
so established by authority, custom, or
consensus” (Sadler, 2013)
eg. Learning standard
a.k.a threshold learning outcome (TLO)
Learning outcome
Learner knows, understands and can do…..
independently, on demand and in a range of
contexts
Definitions
Source Ray 4/4/09 http://www.flickr.com/photos/15075617@N05/3418528723

4. Providers
Graduate outcomes
Students
Self regulation
Market forces
National regulation
Prof regulation

5. article.wn.com
Need better grad outcomes for
Accounting degrees. What changes?
With peer
What evidence prioritise to know
if improving?

6. Market
22% employers would recruit more grads if higher number “appropriate available”
23% domestic bachelor grad self-report seeking FT employment in AGS.
NESB 38%. International bachelor? Conversion masters?
Private providers compete. eg. internships; small engaging classes; some big brands
(GCA, 2014)

7. NOT interested in “record of completion”
but “demonstrable competence” say CPA Aust (HES 2009)
Accredited by AACSB +/or EQUIS (CPAA guidelines, 2011)
Lobby by CPAA for Accounting on Skills Occupation List (2014)
Professional

8. AoL standards (AACSB, 2003)
1/3Aust graduates applying for
residency sub standard English
communication (Birrell, 2007)
39% pressure to pass full fee
students work “not good enough”
(NTEU, 2012)
Self regulation
Source: Symington(18Oct2009) at commons.wikimedia.org

9. 9
1.1 The course of study meets the
Qualification Standards.
1.2 There are robust internal
processes for design and approval of the
course of study which take account of
external standards and requirements,
e.g. published discipline standards,
professional accreditation, input from
relevant external stakeholders, and
comparable standards at other higher
education providers.
Legislation 2012
Source: http://curiosoando.com/que-es-el-freno-de-servicio

10. Martin Morris 20/2/12 http://www.flickr.com/photos/martin55/6929721749/
5.5 The academic standards
intended to be achieved by
students and the outcomes
actually achieved by students
in the course of study are
benchmarked against similar
accredited courses of study.
External AoL

11. 1
1
Source Ray 4/4/09 http://www.flickr.com/photos/15075617@N05/3418528723
1.5.1 The expected learning outcomes for each course
of study are specified, consistent with the level and
field of education of the qualification awarded and
informed by national and/or international
comparators.
1.5.2 The specified learning outcomes for each
course of study encompass discipline-related and
generic outcomes, including…:
HESP propose
Reference points
AQF (2nd ed)
Discipline learning outcomes statements
Professional requirements

12. Martin Morris 20/2/12
http://www.flickr.com/photos/martin55/6929721749/
5.3.3 Review and improvement
activities include regular external
referencing against comparable courses of
study, including by referencing:
a) the progress of student cohorts
through courses of study, attrition rates
and completion times and rates, and
b) the grading of students’ achievement
of learning outcomes for selected units
of study within courses of study
HESP propose

13. article.wn.com
The BCom is 1 of 5 degrees
up for TEQSA scrutiny in 2015.
Challenge
What evidence will you prioritise?

14. 5 students work given to 6 experienced EE in 4 disciplines. (2013)
Findings
• Only 1 jointly highest (of 5) by all 6 EE in discipline
• 9 of 20 ranked both best and worst (of 5)
Call for calibration again
UK model
“The idea that a single external examiner could
make a comparative judgement on the national,
and indeed international, standard of a
programme has always been flawed” (2012)

15. “Assessment is largely dependent upon professional
judgement and confidence in such judgement
requires the establishment of appropriate forums for
the development and sharing of standards within and
between disciplinary and professional communities”
Tenet 6: Price et al (2008)
UK trial forums hospitality management (Rust 2009)
Literature
….just like research

16. By + 4 Accounting
2,100 participants nationally
38 Australian universities
21 private/other providers
20 professional/peak bodies and
employers
ABDC leadership
2010

17. Accounting
graduate
Knowledge
Application
Judgement
Communication
& Teamwork
Self
management
Bachelor graduates are able to
exercise judgement
under supervision to solve
routine accounting problems
in straightforward contexts
using social, ethical, economic,
regulatory and global
perspectives
Masters
minimal
emerging/advanced
complex
^

18. TLO 1
TLO 2
TLO 3
TLO 4
TLO 5
TLO 1
TLO 2
TLO 3
TLO 4
TLO 5
TLO 1
TLO 2
TLO 3
TLO 4
TLO 5
Blended
learning
Online distance
learning
Problem-based
learning
Team-based
learning
Provider CProvider B Provider DProvider AApply
TLOs
+
Quantitative
Public sector
Professional Small
business
Regional
+
+
International
+
International
International
International
International
TLO 4
TLO 3
TLO 2
TLO 1
TLO 5

19. Assess TLOs
Achievement Matters project aims
1. Obtain external peer-reviewed evidence of accounting learning
outcomes in all types of higher education providers, benchmarked
against the learning standards.
2. A model process for assessing learning outcomes against standards (that
is sustainable, reliable and efficient), satisfying external quality
assurance needs and motivating continuous improvement.
3. Professional development of academics about assessment.
4. Enhanced understandings in the external environment of our model for
assessing achievement of learning standards

20. UWA
Curtin
(16)
UniSA
Adelaide
(20)
(2)
(3)
(4)
JCU
USQ
QUT
Griffith
(16)
UWS UTS
Macquarie
SCU(65)
RMIT (46)
Monash
Deakin
Kaplan
Holmesglen

21. Assess Enter Compare
Pre-F2F
F2F
Consensus Agree
Post-F2F
Apply
Confirm
2011-14 Calibrate

22. 22
Remotelyassess

23. 23
Remotely
compare de-
identified rating
& feedback

24. Calibration – Task validity
Individual confidence pre‐workshop
Individual results pre‐workshop
• Min & max (n=33)
• Mean ±1 SD
Group results at workshop
• Small groups (n=6)
• Consensus
91%
NA A
NA A
Oral communication
(12) (21)

25. Oral communication
Individual confidence pre‐workshop
Individual results pre‐workshop
• Min & max (n=33)
• Mean ±1 SD
Group results at workshop
• Small groups (n=6)
• Consensus
NM M
NM M
85%
....S2, S3, S4, S5
Calibration – UG Student 1
(10) (23)

26. Date Location Learning standards calibrated Level
Jul 2011 Darwin • Written Communication Bachelor
Sep 2011 Melbourne • Written Communication Master
Feb 2012 Adelaide • Written Communication Bachelor
Sep 2012 Sydney • Knowledge
• Written Communication
Master
Feb 2013 Adelaide • Knowledge
• Oral Communication
Master
Jul 2013 Perth • Application
• Judgement
• Oral Communication
Bachelor
Feb 2014 Adelaide • Application
• Judgement
• Teamwork
Master
Jul 2014 Sydney • Teamwork
• Self management
• Written communication
Bachelor
Calibration workshops

27. 2
7
Grounded & trusted suggestions to improve assessments
• Reduced use of group assessable task & same mark
• Increased weighting in grade
• Expanded use of past examples as student calibration activity
• Greater use of self-assessment resources eg. writersdiet.com
• More authentic eg. brief CFO/board & email SH friend
• Added triangulation eg. impromptu Q&A
Standard deviation reduced
• Less likely to have an unreliable review
• Avoid perverse consequences
Calibration – Impact

28. Martin Morris 20/2/12 http://www.flickr.com/photos/martin55/6929721749/
Live review

29. 2
Live review report

30. 3
Appendix

31. Achievement Matters model
1. Useful to prove (to external bodies) & to improve achievement
2. Collaborate with industry & professional bodies
3. Fits culture
4. Reduce potential bias
– Review only if participated in calibration against explicit standards
– Experiments show standard deviation halved
– Double-blind peer review with third external if any disagreement
– Student work randomly selected so no risk of ranking sample
5. Efficient
– Assessment tasks aligned to discipline standards (not units of study)
– Live reviews immediately after calibration follow annual conference
– Electronic reviews: easy to allocate and collect, aggregate & return
– Professional development & shared resources
– Potential for deans council to coordinate for each discipline

32. This project will help establish
national agreement on learning
standards between accounting
degree providers
This project will help establish
national agreement on learning
standards between accounting
employers and accounting degree
providers
Future - Accounting

33. Quality Verification Achievement Matters Inter‐university Review
Primary intent QA QA & QE QA & QE
Scope Selected final UoS Selected degree standards Selected final UoS
Disciplines Multiple (7) Accounting Multiple (12)
Level Bachelor Bachelor + Master Bachelor
HEI grouping Group of Eight 17 HEI across sector 11 unis across sector
Reviewers 1 academic 2 anonymous academics +/or
professionals (3rd potentially)
1 academic per partner
Calibrated Not explicitly Yes by workshops Not explicitly
Products viewed Inputs & outputs in
final unit of study
Tasks & outputs in degree
evidencing standard(s)
Tasks (inputs) & outputs in
final unit of study
Data selection Stratified random Fully random Stratified random
Sample size
outputs
5% all assessments
(available) per grade
5 assessments per standard 1 assessment per grade
(HD/D,C,P,F) = 4
Reviews Manual submission
files & aggregation
Online submission & auto
aggregation
Manual submission files &
aggregation
Authority Institutionally top-
down
Disciplinary ground-up Institutionally top-down

34. Decide with peers
The Bachelor of Business has been selected as one of five degrees to evidence
the Threshold Standards in the upcoming TEQSA review at your institution in
2015. The best evidence of external assessment is:
A. Use AGS % employed and % satisfied with graduate attribute development
B. Access validated standardised test for final year students to do
C. Pay experienced external academic to verify grades on final year UoS sample
D. Select partner department to mutually verify grades on final year UoS sample
E. Argue existing external accreditation system (eg. AACSB, EPAS) suffices
F. Lobby existing university system (eg. Go8, IRU) to focus in 2014
G. Lobby ABDC to coordinate external review system around TLOs with calibration

35. > 1?
Future

36. Thank you