This presentation reports on the efficacy of a mobile learning intervention that combined ‘push notifications’ and game principles within a timed quiz app. An institutional interdisciplinary case study was conducted which compared rates of student retention and academic performance with their usage of a purpose-designed learning app. Leading up to lectures the app pushed daily quizzes to students’ personal mobile devices and then rewarded them with feedback, points, badges and a position on a leaderboard. During this session, the findings of this study will be discussed and conclusions made in regards to what findings mean for the future research into higher education learning enabled via mobile app technologies.

1. Dan Laurence @D4n_
Higher Education & Game Principles:
Context, Theory & Application
Senior Learning Designer & Educational Game Designer - La Trobe University

2. Todays Session
3 x 10 minute blocks, followed each by question time (and game time).
Aims: 1. To explore the context of Higher Education & the challenge of student retention.
2. To consider game principles as part of an engagement solution.
3. To demonstrate a specific implementation of game design and it’s outcomes.
The game Kahoot will be used later in this session. If you have a spare moment you may
download the app (on your device) in preparation at: www.kahoot.it
2

3. Higher Education – The Context
A pitched battleground to retain disengaging students.
If the average student fee is around $17,000.
Every 1% decrease in student retention costs…
$4-16 million the same year in lost fees.
Double if you factor in cost of acquisition.
…and that is just in the first year.
These dollar values represent peoples dreams & aspirations.
3
- Adams et al, 2010
Hobson’s retention project

4. How the Government Sees Higher Education
4

5. Who are these students and why do they leave?
Many risk factors and correlations:
• part-time
• mature-age
• studying online
• first year
• 1st in family to attend tertiary study
• low socio-economic status
• Indigenous
• regional
• student with a disability
“Student attrition is a wicked problem.
It could be 'solved' by excluding from
university study everyone except the
previously educationally successful,
rich, healthy, unencumbered young
people without children, who live in
cities and study full-time, on campus.
But that solution would create some
other issues that would be somewhat
unpalatable to a developed economy
and egalitarian society.”
Marcia Devlin
DVC Learning and Quality at Federation University
Some mitigation possible, but factors largely beyond control.

6. What influences students to discontinue study?
- Adams et al, 2010 – Hobson’s retention project
After enrolment what concerns can we address?

7. What are students specific complaints?
- The Student Surveys (CSHE, 2004 & AUSSIE, 2008)
• only 50% of students felt that staff were available to
discuss student work
• many students perceive that staff are not accessible
• fewer than 30% of students felt that teaching staff took
an interest in their progress
• 30% of students reported that they did not receive
helpful academic feedback
Students expect: regular constructive feedback.

8. 8
Student Personas
What are students
actually feeling and
thinking?

9. 9
Student Journeys
A tool to provide
insights into the
student experience
“Pain Points”

10. Can’t we just reach out to these students?
• Academics don’t have the time to
respond to every student.
• Feedback needs to be of a high quality.
• Feedback needs to be timely.
• Feedback needs to be more regular.
• Feedback isn’t feedback unless
students can respond to it.
10
The only solutions that are practical for large classes are automated.
By Mosborne01 (Own work) CC BY-SA 3.0, via Wikimedia Commons

11. Regular Feedback Also Aids Effective Recall of Facts
The Ebbinghaus
Forgetting Curve
illustrates the simple
point that repetition
increases the recall
of facts.
11

12. The Formative Assessment Challenge
Tests and quizzes are usually associated with marks.
If assessments aren’t associated with marks, students wont do them, will they?
How can we make unmarked, affordable, automated feedback systems that students will
actually want to engage with?
First we should ask what do students actually like doing?
12

13. What do students like?
• social media, cats
• games, movies & music
• SMS & other short messaging
• personal email, but rarely school emails.
Modern students access these increasingly via
mobile phone,
not desktop PCs, not a website & not a LMS.

14. The requirements of an engagement solution
14
• Engaging, or put simply
• Sense of progress & regular feedback
• Social & interactive
• Mobile first
Question Time: 2-5 minutes

15. Block Two: Game Principles as Part of the Solution
We will limit ourselves here to what has become known as “simple gamification”.
Characterised by Points, Badges & Leaderboards
(Werbach’s PBL model, 2012).
15
What is a Game?
Game principles can be observed everywhere in the world.
The animal kingdom, the Legal profession, the business world, religion and yes… education
There are many different definitions and many different types of game.

16. Engagement & Flow
The state of flow is characterised thus:
• There are clear goals every step of the way.
• There is immediate feedback to one’s actions.
• There is a balance between challenges and skills.
• Action and awareness are merged.
• Distractions are excluded from consciousness.
• There is no worry of failure.
• Self-consciousness disappears.
• The sense of time becomes distorted.
• The activity becomes an end in itself.
(Mihaly Csikszentmihaliyi, 1990)
This is the state games aspire towards

17. Types of Rewards
• Points/marks/currency
• Adversarial
• Social/collaborative
• Explorative
17
(Richard Bartle, 1996)
Bartle’s Player Types

18. Markzewski’s Gamification User Types Hexad
18
…Intrinsic and Extrinsic MotivationDifferent strokes for different folks

19. Winning is great, unless it isn’t
Many studies and theories align with the broad contention that:
extrinsic rewards can sometimes decrease intrinsic motivation & performance!
19
Only certain types of extrinsic
rewards (such as praise) can
increase intrinsic motivation.
For complex or creative tasks using
extrinsic rewards can lead to
poorer performance.
If/then rewards tend to narrow
our focus and limit cognitive
capacity.
(Ledford, Gerhart, & Fang, 2013)
Gamification/extrinsic rewards increase performance on simple tasks*
*(Arely, Gneezy, Lowenstein & Mazar, 2008)

20. Where are Game Principles Best Used?
Where a knowledge foundation is required - before it can be applied, etc..
• Sciences, Chemistry, Physics, etc.
• Accounting, Finance, Economics, etc.
• Computer Science, Programming, etc.
Creates a point to check & feedback to a student their level of understanding.20
theoretical knowledge
threshold knowledgecore concepts
abstract knowledge
Simple tasks

21. Let’s test this…
Using your mobile devices go to:
www.Kahoot.it
When prompted enter the game pin:
486299
21

22. The Problem With Leaderboards…
Being at the bottom of a leaderboard (a loser) is demotivating and disengaging.
These are the students we should be reaching out to most!
Solutions:
Anonymous: deidentify student names.
The Plus/Minus solution: only show students the ten peers closest to them in score.
Points Ramp: As the semester progresses increase the point value of questions.
22

23. Games are about rules
Agreed set of rules and common understandings.
Break the rules and your contract is void.
The architecture of dissatisfaction is:
an unmet expectation, or unfulfilled ideal.
Set clear expectations!
This includes clear instructions and ‘on-boarding’
Any perceived unfairness results in:
• Decreased extrinsic motivation/engagement
• Low student satisfaction/retention
23
By Jérôme Dessommes - ÉCRIVAINS CONSULT®, via Wikimedia Commons

24. An Engagement Solution
24
• Clear Expectations
• Regular constructive feedback
(engagement loops)
• Needs to be controllable/curatable
(many games aren’t)
• Points & badges & leaderboards (PBL model)
• Rich media, videos
• Social - leaderboards & chat forums
• Mobile first - uses ‘push-notifications’
• A ‘modern’ interface i.e. Progress trees/bars
Engagement Loops

25. What are the LMS/VLE options?
LMS/VLE’s :
Blackboard 9.x Blackboard Ultra Moodle/Moodlerooms Canvas?
Collaborate Prime Curatr MOOC platforms
25
• Question types: Multiple choice questions, fill in the blank, short answer, equations,
• Gamification: Points, badges, leaderboard, levels, progression trees
• Analytics: Dashboard, notifications
• Student information/enrolment system: Integrated, importable/exportable
• Open standards compliance: SCORM, Tin Can API
In our assessment, LMS’s struggle to provide quality mobile interfaces
& often lack key functions.

26. What are some other web based options?
Web Apps (browser based)
Kahoot (hybrid) Quizlet ProProfs GoConqr
Riddle Learningpod Socrative That Quiz
TestMoz Gnowledge QuizStar QuestionPro
Hot Potatoes Quizinator Edpuzzle Verso
26
Publisher Apps (browser based)
Learning Catalytics (Pearsons)
Cengage - Mindtap
• Question Types: Multiple Choice Questions, Fill in the Blank, Short Answer, Equations, Flashcards
• Media Types: Audio, Images and/or Video
• Gamification: Points, Badges, Leaderboard, Levels
• Social features: Forums, leaderboard
• Analytics: Dashboard, notifications
• Content Authoring: is it your platform or someone else’s?
• Content Bank: OER’s? Personalisation/Branding/Advertising
Mobile feature integration still lacking! - Push Notifications, Geolocation,
but mostly about usability

27. What are some mobile first options?
Mobile Apps (iOS & Android):
27
Q-Stream Duo Lingo Mathspace Nearpod
Quizling QuizClash iSpring Quizmaker Questbase
A range of great things about these apps.
The issues we had were:
• Inability to curate content
• Lack of game elements
• Unsustainable cost
• Practical implementation issues

28. What went on the wish list?
28
Needs to be both USEFUL and EASY to use:
• Easy to use web portal for question creation
• Well documented
• Import/export external question sets
• Real time learning analytics dashboard
• Import/export use data (currently csv)

29. Block Two – game principles as part of an
engagement solution.
Question Time: 2-5 minutes
29

30. So after all that here’s what we did…
30
We consulted broadly with game design/programming lecturers, students and graduates.
Using all our research, background data and a teaching grant we created a simple first version.
Featuring:
• Timed multiple choice questions
• Pre-set badges
• Leaderboard

31. And for the Teacher
We created a back-end web portal to create the multiple choice questions:
31
Complete with basic engagement analytics

32. Research Design
The first iteration performed a simple comparison with previous semesters (year on year).
We wanted control groups, but couldn’t disadvantage those students in the control group.
We sought and received ethics approval on the following hypotheses.
1. Is there a relationship between introducing a mobile app into the classroom and
improved student retention rates?
2. Is there a relationship between the use of the app and improved academic performance?
3. Is student performance within the app related to academic performance?
4. Does the existence of other blended content impact on the effectiveness of the app?
5. Do students respond positively to the introduction of apps?
32

33. Did gamification make a difference?
33
“The introduction of the app in second
semester of 2015 was associated with a
positive improvement in student
retention (calculated at 12.23%)
compared to semester one 2015, a
positive improvement of 9.22%...”
“…students who used the app on
average achieved marks that were
7.03% higher than students who did
not use the app.”

34. Did gamification make a difference?
34
"I love the
competitiveness,
ease of access and
simple to use."
Strong correlation between
app use & academic
performance (r .40).
Positive qualitative feedback:
"The app questions were
great I really appreciated
the daily reminders."
"So much easier to access
than the online quiz.”
Negative feedback (during mid-semester break):
“Where have the questions gone, is it broken?”

35. Follow up research…
35
One of the big questions is the influence of mobile optimisation on our study.
The next semester we were given approval to use a control group.
The control group was given an LMS based quiz as a substitute, however made possible by
an unexpected upgrade to our Blackboard installation students also received the BB
Student mobile version of the quiz with push-notifications.
So our control group essentially had no timer, no badges and no leaderboard.
This should allow us to exclude the influence of mobile optimisation
and discern the unique contribution that gamification contributes.

36. Thankyou. Final Questions?
Follow @D4n_ for updates.
For enquiries or interest in research collaboration please contact:
d.laurence@latrobe.edu.au
36

37. References
Richard Bartle (1996). Hearts, Clubs, Diamonds, Spades: Players Who suit MUDs.
E.L. Deci, R. Koestner, R.M. Ryan. Extrinsic Rewards and Intrinsic Motivation in Education: Reconsidered Once Again,
Review of Educational Research, Spring 2001, Vol. 71, pp 1-27
D. Ariely, U. Gneezy, G. Lowenstein, & N. Mazar. Large Stakes and Big Mistakes. Federal Reserve Bank of Boston
Working Paper No. 05-11, July 2005; NY Times, 20 Nov 2008.
Ledford, G. E., Gerhart, B., & Fang, M. (2013). Negative effects of extrinsic rewards on intrinsic motivation: More
smoke than fire.
Csíkszentmihályi, M. (1990). Flow: The psychology of optimal experience. New York: Harper and Row.
Werbach, K. & Hunter, D. (2012), For the Win: How Game Thinking Can Revolutionize Your
Business, Wharton School, Philadelphia
Ebbinghaus, H (1885/1913). Memory: A Contribution to Experimental Psychology.
37