In recent years there has been growing interest in the use of e-learning tools that are able to adapt to suit the ability levels, needs, or preferences of individual learners. In this project we aim to test the utility of an adaptive e-learning study tool within the context of a large undergraduate Psychology course (approximately 700 students). The study tool and a number of associated summative tests are hosted on the course’s Blackboard Learning Management System. Pilot data indicates that students that use the tool perform significantly better on the summative tests compared to non-users (t[683] = 4.35, p <0.001). We examine the relationship in the context of 1) learning analytics data that can be obtained via Blackboard, and 2) a number of known psychological predictors of academic success. Delivered at Innovate and Educate: Teaching and Learning Conference by Blackboard. 24 -27 August 2015 in Adelaide, Australia.

1. Cognitive, personality and behavioural
predictors of academic success in a large
undergraduate program.
Dr Matthew Dry
University of Adelaide, School of Psychology

2. Acknowledgements … and a disclaimer ….
• My co-authors are Clemence Due, Anna Chur-Hansen, and
Nicholas Burns (all from Adelaide Uni School of Psychology)
• This research has been partly funded by an Office of Learning
and Teaching Seed grant
• We have received technical support from McGraw-Hill, but
none of the authors are affiliated with the company, and they
have not provided any financial support for this research!
2

3. I am a psychologist ….. and an academic …
3
• I am interested in human
behaviour in general
• As a teacher I’m particularly
interested in why some
students do well, (and
others not so well) and the
factors that play a role in
this.

4. Online learning tools
• Come in all sorts of colours and flavours – quizzes, videos,
interactive experiments, puzzles, etc
• It is generally claimed (or assumed?) that they aid student
learning
• But assessing their actual utility is hard
4

5. Talk Outline
1. The Context
2. The Tool
– How LearnSmart works
– Initial Findings
3. The Research
– The psychological factors influencing academic success
– The current study
– Results
4. Blackboard data
5. General Discussion, etc
5

6. The Context – Large Undergraduate Psychology Course
• Psychology 1A (Semester 1) N = 700
• Psychology 1B (Semester 2) N = 600
• Each semester covers six core concept areas, e.g. in 1B:
– Developmental psychology
– Statistics
– Personality
– Motivation & Emotion
– Learning
– Intelligence
6

7. The Context – Large Undergraduate Psychology Course
• Most of the assessment, and all of the course content is hosted
on or accessed via BlackBoard
• Assessment tasks:
– Online quizzes assessing the six topic areas (MAEs – Module
Assessment Exercises) = 20%
– A research report = 15%
– Research Participation = 10%
– End-of-semester exam = 55%
7

8. The Context – Large Undergraduate Psychology Course
• The textbook we use is Passer, M. W. & Smith, R. E. (2013).
Psychology – The Science of Mind and Behaviour (Australian
Edition). McGraw-Hill: North Ryde, NSW
– Most introductory psychology texts cover exactly the same material
– Even the chapters tend to be in the same order
– Like most textbooks it comes with a range of online supplementary
materials
– The reason I adopted this textbook was because of the LearnSmart tool
8

9. The Tool - LearnSmart®
• Online quiz covering the textbook chapter content
9

10. The Tool - LearnSmart®
10

11. The Tool - LearnSmart®
11

12. The Tool - LearnSmart®
• LearnSmart is adapts to the student’s ability level based on
– Correct versus incorrect answers
– The metacognitive data (I know it/I think so/Unsure/ No idea)
• If students get questions right and are well calibrated the questions get
harder
• If students get questions wrong or are poorly calibrated the questions get
easier
• It seemed to me that students would be more motivated to perform this task
because the degree of challenge would be matched to the student … so I set
it as an extension task for students to complete if they wanted or to ignore if
they chose…
12

13. The Tool - LearnSmart®
• Each of the six course topics is assessed via an online quiz
(MAE)
• Students that had completed the LearnSmart task associated
with each of these topics prior to the MAE did better than the
other students:
– This was a statistically significant difference
– It was consistent across topic areas and cohorts
– It amounted to a difference of around 5-15%
13

14. Why would this happen?
• Is it the tool? And what aspects of the tool in particular?
– Adaptation?
– Metacognitive aspect (calibration)?
– Immediacy of feedback?
• Is it something to do with the type of student that chose to use
the tool?
– We know that there are certain psychological variables that affect
academic success …
14

15. The Psychology of Academic Success
• There is a large body of literature investigating the affect of
individual differences in psychological variables on academic
success:
– IQ affects success. Smart students tend to do better than not-so-smart
students.
– Personality affects success:
• Two of the Big 5 personality trait (conscientiousness & openness)
• Epistemic curiosity, need for cognition
• Learning Style
15

16. The chicken or the egg?
• Students that used the tool did better academically, AND
• Smarter/more conscientious students might be more likely to
use the tool, BUT
• Smarter/more conscientious students tend do better
academically, SO
• It is clear that we need to do some more research before we
can decide if the tool is actually having an effect!
16

17. The Current Study
• Does the LearnSmart tool have an impact upon academic
success above and beyond what we would expect from these
individual differences?
– Psychology 1A students (Semester 1)
– Students participated for course credit. N = 278 (194 female)
– Intellectual abilities tasks (Ravens APM, CAB – inductive reasoning)
– Personality measures (Conscientiousness, Openness to Experience,
Epistemic Curiosity, Need for Cognition)
– Behavioral measure (LearnSmart usage)
– Outcome variable (Exam performance for Semester 1)
17

18. The Current Study
• Does the LearnSmart tool have an impact upon academic
success above and beyond what we would expect from these
individual differences?
– Psychology 1A students (Semester 1)
– Students participated for course credit. N = 278 (194 female)
– Intellectual abilities tasks (Ravens APM, CAB – inductive reasoning)
– Personality measures (Conscientiousness, Openness to Experience,
Epistemic Curiosity, Need for Cognition)
– Behavioral measure (LearnSmart usage)
– Outcome variable (Exam performance for Semester 1)
18

19. Measuring intellectual ability – Ravens APM
19
This one is
relatively easy ….
But this one is
quite hard!

20. Measuring Personality Trait – Need For Cognition
20

21. Comparing LearnSmart Users (n = 159) and Non-Users (n = 119)
Measure Cohen’s d p-value
Intellectual Abilities CAB-I 0.07 .54
Ravens APM 0.01 .94
Personality Traits Conscientiousness 0.45 <.001
Epistemic Curiosity 0.42 <.001
Need for Cognition 0.41 <.001
Openness to Experience 0.32 .009
21
Users did not differ from Non-users on the intellectual abilities measures:
Being clever does not make you more or less likely to use the tool

22. Comparing LearnSmart Users (n = 159) and Non-Users (n = 119)
Measure Cohen’s d p-value
Intellectual Abilities CAB-I 0.07 .54
Ravens APM 0.01 .94
Personality Traits Conscientiousness 0.45 <.001
Epistemic Curiosity 0.42 <.001
Need for Cognition 0.41 <.001
Openness to Experience 0.32 .009
22
Users were more significantly more conscientious and open to experience,
and had a higher degree of epistemic curiosity and need for cognition.

23. Correlation matrix for variables
1 2 3 4 5 6 7
1. CAB-I -
2. Ravens APM .50
3. Conscientiousness .06 .03
4. Epistemic Curiosity .03 .03 .38
5. Need for Cognition .11 .15 .44 .66
6. Openness to Experience .13 .12 -.09 .32 .36
7. LearnSmart Usage .07 .03 .20 .14 .16 .12
8. Exam .22 .22 .14 .05 .13 .17 .31
23

24. Correlation matrix for variables
1 2 3 4 5 6 7
1. CAB-I -
2. Ravens APM .50
3. Conscientiousness .06 .03
4. Epistemic Curiosity .03 .03 .38
5. Need for Cognition .11 .15 .44 .66
6. Openness to Experience .13 .12 -.09 .32 .36
7. LearnSmart Usage .07 .03 .20 .14 .16 .12
8. Exam .22 .22 .14 .05 .13 .17 .31
24
Intellectual ability predicts exam performance

25. Correlation matrix for variables
1 2 3 4 5 6 7
1. CAB-I -
2. Ravens APM .50
3. Conscientiousness .06 .03
4. Epistemic Curiosity .03 .03 .38
5. Need for Cognition .11 .15 .44 .66
6. Openness to Experience .13 .12 -.09 .32 .36
7. LearnSmart Usage .07 .03 .20 .14 .16 .12
8. Exam .22 .22 .14 .05 .13 .17 .31
25
Intellectual ability predicts exam performance – but
not LearnSmart usage

26. Correlation matrix for variables
1 2 3 4 5 6 7
1. CAB-I -
2. Ravens APM .50
3. Conscientiousness .06 .03
4. Epistemic Curiosity .03 .03 .38
5. Need for Cognition .11 .15 .44 .66
6. Openness to Experience .13 .12 -.09 .32 .36
7. LearnSmart Usage .07 .03 .20 .14 .16 .12
8. Exam .22 .22 .14 .05 .13 .17 .31
26
The personality traits all predict LearnSmart usage

27. Correlation matrix for variables
1 2 3 4 5 6 7
1. CAB-I -
2. Ravens APM .50
3. Conscientiousness .06 .03
4. Epistemic Curiosity .03 .03 .38
5. Need for Cognition .11 .15 .44 .66
6. Openness to Experience .13 .12 -.09 .32 .36
7. LearnSmart Usage .07 .03 .20 .14 .16 .12
8. Exam .22 .22 .14 .05 .13 .17 .31
27
The personality traits predict exam performance
(except epistemic curiosity)

28. Correlation matrix for variables
1 2 3 4 5 6 7
1. CAB-I -
2. Ravens APM .50
3. Conscientiousness .06 .03
4. Epistemic Curiosity .03 .03 .38
5. Need for Cognition .11 .15 .44 .66
6. Openness to Experience .13 .12 -.09 .32 .36
7. LearnSmart Usage .07 .03 .20 .14 .16 .12
8. Exam .22 .22 .14 .05 .13 .17 .31
28
Importantly – LearnSmart usage predicts exam
performance

29. Predicting Exam Performance - Regression
• We compared two regression models predicting exam results:
• Model 1.
– Exam = CAB-I + APM + C + EC + NFC + O
– R2 = .11, F(7, 271) = 5.36, p < .001
• Model 2.
– Exam = CAB-I + APM + C + EC + NFC + O + LS
– R2 = .17, F(7, 270) = 8.14, p < .001
• R2 change = .06, F(1, 270) = 22.2, p < .001
29

30. Predicting Exam Performance - Regression
• Relative importance
regression indicates that
LearnSmart usage accounts
for around 46% of the
explained variance,
intellectual abilities for 32%,
and openness to experience
around 11%
30
% R2
CAB-I 14.2
Ravens APM 17.5
Conscientiousness 7.4
Epistemic Curiosity 1.6
Need for Cognition 2.9
Openness to Experience 10.7
LearnSmart 45.6

31. Summary – Semester 1 study
• The psychological measures give insight into users vs non-
users
• They did not differ in regards to intellectual ability.
– Clever students are no more or less likely to make use of the tool than
the other students
• But they did differ in regards to personality.
– Users scored higher on Conscientiousness, Epistemic Curiosity, Need for
Cognition and Openness to Experience
31

32. Summary – Semester 1 study
• The psychological measures and LearnSmart usage predicted
exam performance
• But LearnSmart usage was the strongest predictor
– LearnSmart usage predicted exam performance even when controlling
for individual differences in personality and intellectual ability
32

33. Summary – Semester 1 study
• Does the LearnSmart tool have an impact upon academic
success above and beyond what we would expect from these
individual differences?
• YES!
• This tool appears to actually have a positive impact upon
academic performance!
33

34. Looking Ahead – Semester 2 study
• In Semester 2 we have mandated a minimum usage of the tool for
course credit (5%)
• All students are required to complete the tests of cognitive abilities
and personality measures as part of the major assignment
– They can choose not to give consent for their data to be used for research
purposes (good research is ethical research!)
• We are also collecting a range of other variables that may be
informative
– Attitudes to learning, predictions of achievement, metacognitive data, etc
34

35. What about Blackboard?
• Psych 1A/B students interface with Blackboard for the majority
of the content and assessment
• This is potentially a rich data-source
– One of the reasons I’m here is that I want to know more about the type
of data that can be extracted from Blackboard
• To date I have run some simple analyses using data from
Blackboard …
35

36. What can Blackboard tell us about academic success?
• There is a significant relationship between submission latency
and performance (r = .31, p< .001) … but plenty of students
are submitting at the last moment and doing well.36
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
0
10
20
30
40
50
60
70
80
90
100
Submission time prior to due-date (Days)
MAE%

37. What can Blackboard tell us about academic success?
• The same pattern holds for the subset of students that
we have the psychological data for (r = .28, p< .001).
37
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
0
10
20
30
40
50
60
70
80
90
100
Submission time prior to due-date (Days)
MAE%

38. What can Blackboard tell us about academic success?
• There is a weak but significant relationship with
conscientiousness (r = .14, p< .05), but no significant
relationship with any of the other psychological variables38
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
0
25
50
75
100
125
150
175
200
225
250
Submission time prior to due-date (Days)
Conscientiousness

39. What can Blackboard tell us about academic success?
• Students are submitting their assignments at all hours
… and this doesn’t seem to affect how well they do.
39
0 2 4 6 8 10 12 14 16 18 20 22 24
0
10
20
30
40
50
60
70
80
90
100
Time of day submitted
MAE%

40. What can Blackboard tell us about academic success?
• When we have the full data-set of psychological measures and
LearnSmart usage for the entire cohort we will be able to
match this up with other behavioural data from BlackBoard
– Course Access
– Missed submissions
– Usage of other supplementary materials
40

41. Final thoughts … future questions …
• Data from Semester 1 indicates that LearnSmart usage has an
impact on academic success above and beyond what we would
expect given individual differences in intellectual ability and
personality traits
• This suggests the tool is actually working
– What is it about the tool that causes this?
– What other sorts of tools might lead to similar improvements?
41

42. Take-Home Message!
• The psychological variables predict academic success and
behavioural patterns
– There are interesting and meaningful relationships between these
variables
– You should consider including psychological measures in your
investigations
– Without controlling for these psychological variables you cannot make
any strong conclusions about learning analytics/behavioural data and
academic outcomes
42

43. Questions?
43

44. Contact: matthew.dry@adelaide.edu.au
44