Apresentação realizada pela professora Maryam Omari, no ENANGRAD 2018, realizado pela ANGRAD na FECAP, São Paulo, de 24 a 26 de agosto.

1. Assurance of Learning
The ECU Story
PROFESSOR MARYAM OMARI
EXECUTIVE DEAN
SCHOOL OF BUSINESS AND LAW
EDITH COWAN UNIVERSITY

2. EDITH COWAN UNIVERSITY

3. Edith Cowan University
• Large multi-campus university
• 100+ year history, 27 years as a university
• ~28,000 students
• International students from
100+ countries
• 2018
• QS, Top 150 under 50
• THE, Top 100 Asia-Pacific Universities
• THE World University Rankings, in the 500-
600

4. QILT 2018 - Released May 16, 2018
• The top public university in Australia for student
experience – 2nd year in a row
• Also rated top public university in Australia for:
• Teaching Quality
• Skills Development
• Student support
• Business and Management Discipline – top public
university in Australia
• Overall Education Experience
• Teaching Quality
• Skills Development
• Student Support

6. ECU Renewal and Change
2015
• April 1, appointment of the new Vice-Chancellor
• Review of the academic units at ECU
2016
• Academic restructure: 8 new Schools
• Appointment of 8 new Executive Deans
• New leadership team in the School of
Business and Law

7. Our Strategic Intent | 2017–2021
ECU’S PURPOSE
To transform lives and enrich society through education and research.
ECU’S VISION
To be recognised for our world-ready graduates and leading edge
research.
SBL’S MISSION
The School of Business and Law develops industry-ready graduates
through high quality teaching and learning experiences and engaged,
applied research.

8. Our Four Pillars
• Enhancing Learning and Teaching
• Advancing Research and
Knowledge Translation
• Growing Internationalisation
• Ensuring Organisational
Sustainability

9. The School of Business and Law
• January 2016 - New name, Executive Dean and leadership team
• Has grown by over 30% in academic staff numbers in just 2 years
• Huge quantum leap in research and quality publications
• Teaching outcomes going from strength to strength
• February 2016 - AACSB accreditation journey began
• May 2017 – Eligibility Application (EA) submitted
• May 2018 – First mentor visit
• September 2018 – iSER will be submitted

10. What is Assurance of Learning (AoL) about?

11. Meet the AOL team at ECU!

12. The essence of AoL | Herding Cats
• Scoping
• Review
• Standardisation
• Measurement
• Improvement
• Keep the cycle going to eventually close the loop!
• Most importantly: Have good people driving it!
Image: Photo still from EDS ad campaign ‘Cat Herders’, launched at the Superbowl in 2000. Agency: Fallon Worldwide

13. Who are the stakeholders in the AoL process?
• Students
• Industry/potential employers
• Our Course Consultative Committees and Advisory Board
• Professional associations
• Alumni
• Staff
• The wider university
• The higher education sector
Image: Rodion Kutsaev accessed at Unsplash, Feb 21, 2018

14. Who is ultimately responsible?

15. The QA processes we already had in place
• Annual unit reviews
• Annual course reviews
• Professional accreditations
• Ad-hoc program reviews given new markets and industry
developments
• 3 yearly/Triennial reviews of our international offerings
• 5 yearly comprehensive course reviews with benchmarking against
other institutions and courses

16. What did we do to make it better?
• Regular seminars and use of technologies to
facilitate the process: 450+ hrs of TEL training
• Teaching and learning grants to support and
encourage:
• 5 grants a year since 2016 totaling $15K AUD/annum,
• 3-6 conference sponsorships between $15K - $18K
AUD/annum,
• $60K AUD for curriculum renewal in 2017 across 4
discipline areas
• Built and strengthened a team to drive the
initiative
• Identified champions for the cause

17. What did we already do to assure learning?
• 2012 onwards - The Australian Qualifications Framework (AQF)
describes Course Learning Outcomes (CLOs) for all degree types
• 2016 onwards – with the commencement of the AACSB journey, the
CLOs were refocused around 3 School based core themes:
1. Applying knowledge
2. Industry-readiness
3. Developing an ethical, professional identity

18. Our Undergraduate CLOs
THEME 1
APPLIED KNOWLEDGE
THEME 2
INDUSTRY READINESS
THEME 3
AN ETHICAL PROFESSIONAL
IDENTITY
UGCLO1
Applying Knowledge
UGCLO2
Solving Problems
& Thinking Critically
UGCLO3
Communicating
Effectively
UGCLO4
Working in Teams
UGCLO5
Creating &
Innovating
UGCLO6
Reflection &
Feedback
UGCLO7
Behaving Professionally
& Ethically

19. Our Postgraduate CLOs
THEME 1
APPLIED KNOWLEDGE
THEME 2
INDUSTRY READINESS
THEME 3
AN ETHICAL PROFESSIONAL
IDENTITY
PGCLO1
Mastering Knowledge
PGCLO2
Solving Problems
& Thinking Critically
PGCLO3
Communicating &
Working in Teams
PGCLO4
Initiating & Completing
a Project
PGCLO5
Managing & Leading
Ethically

20. AoL Implementation
• CLO confirmation and curriculum mapping
• Developing assessments
• Developing marking criteria and rubrics
• Measuring student achievement
• Reporting on students’ achievement
• Identifying improvements
• Implementing improvements
• Evaluating the impact of improvements

21. The Roll-out!2016 2017 2018 2019
Tranche Course Jul–Dec Jan–June Jul–Dec Jan–June Jul–Dec Jan–June Jul–Dec
Pilot MPA M0 Refine CLOs M3 Criteria and Rubrics
M1 Curriculum Map M4 Assess Outcomes
M2 Assessment Strategy M5 Report and Analysis
M6 Identify Improvements
MHRM M0 Refine CLOs M3 Criteria and Rubrics
M1 Curriculum Map M4 Assess Outcomes
M2 Assessment Strategy M5 Report and Analysis
M6 Identify Improvements
M Marketing and Innovation M0 Refine CLOs M3 Criteria and Rubrics
M1 Curriculum Map M4 Assess Outcomes
M2 Assessment Strategy M5 Report and Analysis
M6 Identify Improvements
Tranche 1 MBA M1 Curriculum Map M4 Assess Outcomes
M2 Assessment Strategy M5 Report and Analysis
M3 Criteria and Rubrics M6 Identify Improvements
MBAI M1 Curriculum Map M4 Assess Outcomes
M2 Assessment Strategy M5 Report and Analysis
M3 Criteria and Rubrics M6 Identify Improvements
B Int. Resort Hotel Management M1 Curriculum Map
B Hospitality and Tourism Management M1 Curriculum Map
M3 Criteria and Rubrics
M4 Assess Outcomes
M5 Report and Analysis
M6 Identify Improvements
M7 Implement Improvements
M3 Criteria and Rubrics
M4 Assess Outcomes
M5 Report and Analysis
M6 Identify Improvements
M7 Implement Improvements
M2 Assessment Strategy
M7 Implement Improvements
M7 Implement Improvements
M7 Implement Improvements
M7 Implement Improvements
M7 Implement Improvements
M2 Assessment Strategy

22. Overview of Assessment Results for the Pilot
PGCLO1
Mastering Knowledge
PGCLO2
Solving Problems
and Thinking
Critically
PGCLO3
Communicating and
Working in Teams
PGCLO4
Initiating and
Completing a Project
PGCLO5
Managing and
Leading Ethically
Master of
Professional
Accounting
Group data Not enough data Missed Written
Communication
Group data CLOs overlapped in
rubric
Master of Human
Resource
Management
Rubric too late Rubric late or not
used
Unit didn’t run Rubric late or not
used
Unit didn’t run
Master of Marketing
& Innovation
Management
Unit didn’t run HURRAH! Mostly group data Unit didn’t run Group data

23. Assessing - Bachelor of Commerce (Accounting)
Course Learning Outcomes Subordinate
Learning Outcome n (students) Does Not Meet
Expectations (%)
Meets
Expectations (%)
Exceeds
expectations (%)
UGCLO1: Applying knowledge UGCLO1.1 80 6 56 38
UGCLO1.2 59 10 10 80
UGCLO2: Solving problems & thinking critically Aggregated 59 14 5 81
UGCLO2.1 59 19 29 53
UGCLO2.1 59 12 5 83
UGCLO3: Communicating effectively Aggregated 54 2 63 35
UGLCO3.1 54 4 61 35
UGCLO3.2 54 0 33 67
UGCLO4: Working in teams N/A 54 0 31 69
UGCLO5: Creating & innovating N/A 59 12 5 83
UGCLO6: Reflection & feedback N/A No data collected yet. First iteration in 2019
UGCLO7: Behaving professionally and ethically N/A No data collected yet. First iteration in 2019

24. Assessing – Master of Business Administration Int.
Course Learning Outcomes Subordinate Learning
Outcome
n (students) Does Not Meet
Expectations (%)
Meets
Expectations (%)
Exceed
Expectations (%)
PGCLO1: Mastering knowledge Aggregated 47 2 55 43
PGCLO1.1 & 1.2 47 2 53 45
PGCLO1.3 47 2 51 47
PGCLO1.4 47 2 55 43
PGCLO2: Solving problems & thinking critically Aggregated 71 51 21 28
PGCLO3: Communicating and working in teams Aggregated 72 4 54 42
PGLCO3.1-W 72 13 43 44
PGLCO3.1-O 72 4 49 47
PGCLO3.2 72 4 53 43
PGCLO4: Initiating and completing a project Aggregated 47 4 57 38
PGCLO4.1 47 0 57 43
PGCLO4.2 47 2 60 38
PGCLO5: Managing and leading ethically Aggregated 67 13 40 46

25. Designing Assessments
Identification of assessment instruments and measures
• Assessment instruments should be direct (assignment, case study,
exam), with the use of indirect instruments to support (student
evaluations, graduate survey, employer feedback).
• Assessments must measure individual performance against the
outcome/objective.
• Establish benchmarks and descriptors to evaluate student
performance (marking rubrics).

26. Reviewing Data
Collection, analysis and dissemination of assessment information
• Student performance against outcomes and objectives must be
collected and analysed by a committee.
• The collection process should be systematic across the review
cycles.
• The School will collect (direct measure) data every semester through
our Learning Management System (LMS).

27. Use of data for improvement
Using assessment information for continuous improvement
• Must document the process for continuous improvement using the
performance data gathered and analysis undertaken: ‘close the
loop’.
• Examples of student performance on assessment measures should
be used to demonstrate the continuous improvement of curricula.

28. Never send a boy to do
a man’s job.
• AoL was more complex to implement
than anticipated.
• AoL rests on assessment artefacts which
are not always integral to the
development of units (afterthoughts).
• Curriculum mapping revealed some
gaps, misalignments and differing
interpretations.
• In order to meet timelines, quick fixes
were applied with reviews to follow.
Image: Accessed at Lakeview consulting: http://www.lakeviewconsulting.net/mickis-blog/

29. The way forward
• Move away from compliance and move
towards continuous improvement
• Reframe the work of course assessment so
that it enhances the learning experience.
• Celebrate the AoL process, not the
achievement of compliance.
• Be more directive, and provide more
support and encouragement for AoL
processes.
• Foster constructive conversations and
actions about assessment throughout the
development process to ‘close the loop’.
• Recognise and reward the development
processes around AoL through performance
management to close the gap between
espoused and enacted quality assurance.
• Inject new blood!
Image source: Hello I’m Nik from Unsplash, accessed on 7th August 2018

30. Learnings
• AoL, we should look at it as a continuous improvement process, the
work is never truly done!
• Expect the unexpected all the time!
• You need a firm but patient driver for the initiative
• Circumstances beyond ones control – timetabling
• Non-compliance – inadvertent and intentional
• The ‘yes, yes’ people can be the ‘no, no’ people!
• It’s a JOURNEY!
Image: Filip Bunkens, accessed at Unsplash, 6th February 2018

31. 19/09/2018 Assurance of learning & accreditation: Re-aligning practices and attitudes 31
Thank you for your time and attention!