Webinar conducted for ODLAA (Open & Distance Learning Association of Australia) on “Impacts of Authentic Assessment on the Development of Graduate Attributes” by Shironica Karunanayaka and Som Naidu (Based on the Journal Article: Distance Education, 42(2), 231-252. https://doi.org/10.1080/01587919.2021.1920206) 21 October 2021 - 9.30AM (AEST)

1. Karunanayaka_Naidu_ODLAA_Webinar

2. Authentic Assessments
▪Authentic assessments (AA) are
based on real-world issues
where learners are required to
perform either real-world tasks,
or tasks closely resembling real-
world tasks, to demonstrate
meaningful application of
essential knowledge and skills
(Wiggins, 1998).
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 2
▪ AA provide the training for
problem solving in real world
settings.
▪ AA is critical in the development
of graduate attributes among
learners.
▪ Designing AA and ascertaining its
impacts on the achievement of
desirable graduate attributes is
challenging.

3. Graduate Attributes
• Graduate Attributes (GA) are skills, knowledge, attitudes and values
which contribute to the profession.
• GA are the qualities, attitudes, behaviors, values and ethics built into
the learning process.
• There is growing acceptance of the need to explicitly embed GA in
assessment as it is seen as the strongest evidence of their achievement.
• It is crucial that educational institutions explicitly target the
development of these GAs with the design of their teaching, learning,
and assessment strategies.
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 3

4. Design of Authentic Assessments
• Design of AA is connected with the design of authentic learning
experiences.
• AA is informed by the theory of Situated Cognition where knowledge is
constructed within and linked to the activity, context, and culture in
which it is learned (Brown, Collins & Duguid, 1989).
• The design of such learning experiences and activities have the potential
to facilitate the development of desired graduate attributes.
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 4

5. Research Questions
• Key Research Question:
How does authentic assessment impact the development of GAs?
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 5
Sub-Research Questions:
1. How and in which ways have the online authentic assessment tasks
designed in the online CPD program supported development of GAs
among the practitioners?
2. What are the impacts of authentic assessments designed in the online
CPD program on developing the desired GAs among the practitioners?
3. What challenges were faced by the practitioners in achieving the
desired GAs by engaging in the online authentic assessment tasks?

6. Context
• Online CPD program on the adoption of OER and OEP, at OUSL.
• Use of scenario-based approach in the design of this program.
• The integration of authentic learning scenarios linked with
authentic learning activities and assessment tasks.
• Our paper deals with ascertaining the potential impacts on
practitioners of the authentic learning and assessment strategies,
in terms of achieving the learning outcomes and developing
desired graduate attributes.
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 6

7. Methodology
• An explorative study on the potential impacts of authentic
assessment strategies that were designed for the four online courses
on OER and OEP, using a mixed-methods approach.
• Both quantitative and qualitative data were collected, analyzed, and
discussed within this investigation.
• Fifteen (15) practitioners who had successfully completed all four
online courses and received the ‘certificate of successful completion’
of the online CPD program, were the participants of this study.
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 7

8. Design
of AA
Tasks
21.10.2021

9. Example: Creation
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 9

10. Example: Peer Discussion
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 10

11. Example: Self-Reflection
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 11

12. GAs and GGAs Identified
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 12
Generic GA (GGA) Graduate Attributes (GA)
Scholarship (S) 1. Demonstrate a substantive level of knowledge and
understanding in OER and OEP.
Application (A) 2. Use efficiently and effectively, practical skills and
enquiry within the area of OER and OEP.
Critical thinking (CT) 3. Deal with complex issues in a systematic manner and
make sound judgments.
Creativity (C) 4. Exercise initiative and personal responsibility.
Self-Regulation (SR) 5. Advance knowledge and develop additional skills.
Problem-solving (PS) 6. Demonstrate self-direction and confidence in solving
problems.
Self-directed Professional
Development (SPD)
7. Carry out independent studies for professional
development.

13. Data Collection and Analysis
▪ Development and validation of an Instrument to capture the
development of GA:
▪ Graduate Attributes & Authentic Assessments (GAAA) Rubric
▪ An in-depth content analysis of all assignments submitted by the 15
participants to seek evidence of the development of GAs and GGAs.
▪ 60 Artefacts (Assessment 1)
▪ 60 Discussion forum entries (Assessment 2)
▪ 60 Self-Reflections (Assessment 3)
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 13

14. Results
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 14

15. Overall Scores: Creation
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 15
Course
Assignment 1 – Creation S A CT C SR PS SPD
M1 Create a graphical representation on OER and related
concepts. 2.47 1.63 1.27 1.90 0.00 0.60 0.00
M2 Create a set of strategies for efficient searching and
evaluation of OER
1.93 1.83 1.57 1.23 0.23 0.77 0.07
M3 Create an OER by adapting existing OER by revising and/or
remixing
1.83 1.70 1.00 1.77 0.67 0.43 0.00
M4 Create a plan to promote integration of OER and adoption
of OEP at institutional level 1.50 2.43 1.27 2.17 0.97 1.60 2.40

16. Overall Scores: Creation - Findings
• Overall highest S; A; C
• Variations among four courses, due
to the differences in creations.
• M1 – high S; C; A; CT
• M2 – high S; A; CT
• M3 – high S; C; A
• M4 – high SPD; A; C; PS
• Using a variety of creation type of
assessments is desirable.
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 16
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00
S
A
CT
C
SR
PS
SPD
Total Score Value Indicator
GGA
M1 M2 M3 M4

17. Overall Scores: Peer Discussion
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 17
Course
S A CT C SR PS SPD
M1
1.63 1.37 1.57 1.63 1.20 0.60 0.23
M2
1.60 1.97 2.03 1.70 1.47 1.00 0.23
M3
1.63 2.10 1.53 1.93 1.17 1.00 0.33
M4
1.17 1.60 1.17 1.70 0.87 0.67 0.47

18. Overall Scores: Peer Discussion - Findings
• Overall highest – A; C; CT; S
• Associations show a similar pattern
among the four courses, since the
assessment task is similar in all.
• High – A; C; CT; S
• Low – SPD; PS; SR
• Peer discussions is a powerful tool
to develop GAs yet need to allocate
adequate time to engage.
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 18
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00
S
A
CT
C
SR
PS
SPD
Total Score Value Indicator
GGA
M1 M2 M3 M4

19. Overall Scores: Self-Reflection
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 19
Course
S A CT C SR PS SPD
M1
1.33 1.73 1.50 1.40 2.20 1.53 1.80
M2
1.37 1.93 1.37 1.40 1.93 1.60 1.70
M3
1.43 2.17 1.53 1.73 2.07 1.63 1.70
M4
1.33 1.93 1.57 1.73 2.10 1.37 1.97

20. Overall Scores: Self-Reflection - Findings
• Overall highest – SR; A; SPD
• Associations show a similar
pattern among the four courses,
since the assessment task is
similar in all.
• High – All GGAs
• Self Reflections are useful to
develop GAs, yet allocation of
adequate time is needed.
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 20
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00
S
A
CT
C
SR
PS
SPD
Total Score Indicator
GGA
M1 M2 M3 M4

21. Challenges
Category (GGA) Codes
(Challenges)
Supportive Quotes (Examples)
Scholarship (S) Representing
knowledge
…It was interesting and somewhat challenging to find a way of
representing all strands of the assignment challenge on one diagram and
on one page!
Cognitive
load
…I had to skim through so much and file it for later reading. The more I
went into the detail of the plan, the more complicated it became.
Criteria …I concerned only about the 5 criteria they have mentioned in the
assessment rubric when I prepare the map...
Application (A) Technical
issues
…Even if you use OER Commons or search using Google Advanced/ open
license, the searches often come up with copyrighted resources!
Time
constraints
…I always felt that every activity that I completed had even better scope
to complete it if I had spent more time…
Critical Thinking
(CT)
Criteria …I acknowledged that I had not really engaged wholly with the spirit of a
peer-facilitated discussion. It was more of a tick-box exercise…
Cognitive
load
…if there are a great many postings on a forum, I switch off as I cannot
follow all of what is being discussed. As a result, I did not read the
contributions in other group forums…
21.10.2021 21

22. Challenges (contd.)
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 22
Category
(GGA)
Codes
(Challenges)
Supportive Quotes (Examples)
Creativity
(C)
Workload …I realized that I needed to do far more research and talk with others to
create something of utility…
Time
constraints
…The OER produced is very much an early prototype - I have a lot more work
to do on it…I realized that the task is more challenging and time-consuming
than first envisaged…
Self-
Regulation
(SR)
Cognitive load …Also searching the internet throws up so much material that It can be
overwhelming, and useful material can be buried deep…Curation and
record-keeping is another challenge…
Inadequate
peer support
…I was the first in our group to upload the assignment but received least
feedback initially from peers which really made me sad...
Lack of
feedback
…some were not giving critical feedback…just saying well done or good job,
etc…
Non-interest …My way of thinking and learning can be a double-edged sword. If I am not
interested in something, I switch off completely and may miss useful
learning opportunities…

23. Challenges (contd.)
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 23
Category
(GGA)
Codes
(Challenges)
Supportive Quotes (Examples)
Problem
Solving (PS)
Time
constraints
…The frustration and time involved in producing what was a very
incomplete product got me thinking that it might be easier to create my
own OER from scratch and forget about reuse, remixing etc…
Technical
issues
…I had to remove some elements to reduce the size and to be able to
attach it to my post…
Self-directed
Professional
Development
(SPD)
Criteria …I included information under the specified headings as that was what
specified in the assignment criteria. I’d done so differently if I wasn't
trying such criteria and, of course if I had more time :)
Skill
limitations
…I wish to improve my search skills in finding suitable OER materials of
quality and relevance. I also wish to explore tools and application for
curating resources…

24. Concluding Thoughts
• We have been able to demonstrate that authentic assessment
activities supported the development of targeted GGAs in this context.
• Our findings highlight the critical role that that authentic assessment
activities play in the development of desirable graduate attributes.
21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 24

25. 21.10.2021 Karunanayaka_Naidu_ODLAA_Webinar 25
Acknowledgement
• Commonwealth Educational Media
Centre for Asia (CEMCA)
• Open University of Sri Lanka (OUSL)
• The CPDMOOC Development Team
• CPDMOOC Participants
• Palika Jayasinghe and Randima
Jayasekara who contributed to the
data analysis process.