SlideShare a Scribd company logo
1 of 20
Evaluating the Usability of a
Study Support Mobile App in
Higher Education
Mohammad Khalil,
Jacqueline Wong,
Martine Baars,
Farshida Zafar, &
Barbara Wasson
mLearn Conference, Delft
18.09.2019
Background & Motivation
◉ Self-study is an important factor in student success (Wijnen et
al., 2017)
◉ Self-study entails self-regulated learning (SRL) and use of
effective learning strategies
◉ Self-testing can be use for self-monitoring as part of SRL (Baars,
2014)
2
Jacqueline Wong, 18.09.19, Delft
Background & Motivation
◉ Smartphones account for 75% of mobile devices used in
educational context (Wu et al., 2012)
◉ Growing number of apps used for learning
◉ Few studies examined the usability of self-study apps!
◉ Research on usability for self-study apps can help to:
○ Improve apps and enhance user experience
○ Carry out finer research
○ Support teacher instructions based on the difficulties students are
facing
○ Implement support on app use for self-regulated learning, and in
turn, improve learning performance
3
“
4
Usability is the extent
to which a system, product or service can be used by
specified users to achieve specified goals
with effectiveness, efficiency, and satisfaction
in a specified context of use”
-ISO FDIS 9241-210
Jacqueline Wong, 18.09.19, Delft
What is the EUR App?
Jacqueline Wong, 18.09.19, Delft
6
Eur App
● App developed in 2016
● Made by the students for
the students!
● Led by a lecturer in the
law faculty - Farshida
Zafar
● Formative assessment
tool with a focus on
gamification
1. Teacher Perspective
7
● Teachers have control over the content.
● Teachers can drop in 250 multiple choice questions divided over categories
● Teachers can reference to a literature or video for feedback
● Quick changes can be done ‘on the fly’
● Simple descriptive analytics (successful students and percentage of
answers)
Jacqueline Wong, 18.09.19, Delft
2. Student-face design
8
● Four game modalities (practice, compete, test, and battle)
● Availability of instant feedback in practice mode
● The app focuses on knowledge testing and entertaining
● Gamification elements are available (i.e., leaderboard, time
restrictions...etc.)
Jacqueline Wong, 18.09.19, Delft
9
Eur App - Screenshots
Jacqueline Wong, 18.09.19, Delft
10
Jacqueline Wong, 18.09.19, Delft
The Research
Study
Jacqueline Wong, 18.09.19, Delft
12
Research Questions
◉ How often did the students use the app
and which mode was mostly used for
studying in the course?
◉ What were the students’ perceived
usability and enjoyment of the app?
Jacqueline Wong, 18.09.19, Delft
Sample Profile
◉ Survey sent to 800 students
◉ Year 1 Bachelor students from Erasmus University
◉ Course “Introduction to constitutional and administrative law”
◉ Response rate was 10.12%
◉ Female : Male (65% : 36%)
13
Criteria used for evaluation
14
◉ Learnability: How easy is the application to learn and
start working on.
◉ Efficiency: How efficient is the system in terms of
productivity
◉ Memorability: How memorable is the application when
a user leaves it for a while
◉ Errors: The application should have a low error profile
without catastrophic bugs
◉ Satisfaction: How pleasant is the application to use
and are the users satisfied of its functionalities?
◉ Enjoyment: Closely related to motivation by which we
examine whether students were enjoyed using the
application.
◉“It was easy for me to learn how to use
the app and navigate through it”
◉I was able to use the app without
frustration.
◉I remembered how to use the app when I
leave it for a while
◉The app gave error messages when I
made a mistake
◉I was satisfied with the app as a study
tool
◉I enjoyed using the practice mode of the
game app
Nielsen (1994) and
Mun & Hwang (2003)
How often did the students use the app and
which mode was mostly used for studying in the
course?
15
RQ1
What were the students’ perceived
usability and enjoyment of the app?
16
Criteria Cronbach’s a Mean
Learnability (4 items) .702 5.59
Efficiency (3 items) .716 5.43
Memorability (2 items) 4.31
Errors (2 items) 4.80
Satisfaction (4 items) .818 5.75
Enjoyment (5 items) .707 5.26
RQ2
Results
Mode usage
The practice and test
modes were the most
used ones. The battle
and compete modes
were not as highly used.
App usage
Around 50% of the
students used the app
once a week to practice.
60% of the students used
it to assess their
knowledge
Highest usability
criteria
Satisfaction (e.g., app is
user-friendly) was the
highest criterion of the
usability testing
Lowest usability criterions
Memorability (e.g.,
remembering how the app
works, app recalling same
page…) was the lowest criterion
of the usability testing
Other facts
Efficiency and enjoyment
of the app were
perceived on average by
the students
17
Study key outcomes
◉ Students used the app as more of a diagnostic tool rather than
a study tool
◉ Perhaps, students prefer modalities that provide feedback
rather than models with gamification
◉ According to Nielsen (1994), the application can be considered
“Stable”
◉ The gamification models of the app should be improved
18
Any questions ?
Thanks!
19
Mohammad Khalil
mohammad.khalil@uib.no
Jacqueline Wong
wong@essb.eur.nl
Martine Baars Farshida Zafar Barbara Wasson
Any questions ?
You can find me at:
◉ wong@essb.eur.nl
◉ mohammad.khalil@uib.no
Thanks!
20
Mohammad
Khalil,
Jacqueline
Wong,
Martine Baars,
Farshida Zafar, &
Barbara Wasson
You can find me at:
◉ wong@essb.eur.nl
◉ mohammad.khalil@uib.no

More Related Content

What's hot

My feedback poster1
My feedback poster1My feedback poster1
My feedback poster1
Tom Duff
 
Download File (ppt)
Download File (ppt)Download File (ppt)
Download File (ppt)
Videoguy
 
Altc 2010 paper_0188_cb_dh
Altc 2010 paper_0188_cb_dhAltc 2010 paper_0188_cb_dh
Altc 2010 paper_0188_cb_dh
Claire B
 
Comparative and Non-Comparative Evaluation Studies
Comparative and Non-Comparative Evaluation StudiesComparative and Non-Comparative Evaluation Studies
Comparative and Non-Comparative Evaluation Studies
u068717
 

What's hot (19)

Student Response Systems Presentation Final
Student Response Systems Presentation FinalStudent Response Systems Presentation Final
Student Response Systems Presentation Final
 
Comparative and non-comparative evaluation in Educational technology
Comparative and non-comparative evaluation in Educational technologyComparative and non-comparative evaluation in Educational technology
Comparative and non-comparative evaluation in Educational technology
 
My feedback poster1
My feedback poster1My feedback poster1
My feedback poster1
 
STUDY OF STUDENTS AND INFORMATION TECHNOLOGY, 2014
STUDY OF STUDENTS AND INFORMATION TECHNOLOGY, 2014STUDY OF STUDENTS AND INFORMATION TECHNOLOGY, 2014
STUDY OF STUDENTS AND INFORMATION TECHNOLOGY, 2014
 
Participatory Management of Safe School Project by using The PAOR Workshop Pr...
Participatory Management of Safe School Project by using The PAOR Workshop Pr...Participatory Management of Safe School Project by using The PAOR Workshop Pr...
Participatory Management of Safe School Project by using The PAOR Workshop Pr...
 
Teacher Safe: Digital application as an occupational Safety tool
Teacher Safe: Digital application as an occupational Safety toolTeacher Safe: Digital application as an occupational Safety tool
Teacher Safe: Digital application as an occupational Safety tool
 
Openness to integrating technology.eli 2011 template
Openness to integrating technology.eli 2011 templateOpenness to integrating technology.eli 2011 template
Openness to integrating technology.eli 2011 template
 
Download File (ppt)
Download File (ppt)Download File (ppt)
Download File (ppt)
 
Integrating Smartphones and Tablet Devices into EMS Education
Integrating Smartphones and Tablet Devices into EMS Education Integrating Smartphones and Tablet Devices into EMS Education
Integrating Smartphones and Tablet Devices into EMS Education
 
Comparative and non-comparative evaluation in educational technology
Comparative and non-comparative evaluation in educational technologyComparative and non-comparative evaluation in educational technology
Comparative and non-comparative evaluation in educational technology
 
Online student self assessment, skylar csun
Online student self assessment, skylar csunOnline student self assessment, skylar csun
Online student self assessment, skylar csun
 
Robots in nursing education
Robots in nursing educationRobots in nursing education
Robots in nursing education
 
Altc 2010 paper_0188_cb_dh
Altc 2010 paper_0188_cb_dhAltc 2010 paper_0188_cb_dh
Altc 2010 paper_0188_cb_dh
 
IT Tralee Colloquium presentation
IT Tralee Colloquium presentation IT Tralee Colloquium presentation
IT Tralee Colloquium presentation
 
Comparative and Non-Comparative Evaluation Studies
Comparative and Non-Comparative Evaluation StudiesComparative and Non-Comparative Evaluation Studies
Comparative and Non-Comparative Evaluation Studies
 
UMassD Mobile Usage Survey Results
UMassD Mobile Usage Survey ResultsUMassD Mobile Usage Survey Results
UMassD Mobile Usage Survey Results
 
Formative assessment & feedback with student response tools - PETE&C
Formative assessment & feedback with student response tools - PETE&CFormative assessment & feedback with student response tools - PETE&C
Formative assessment & feedback with student response tools - PETE&C
 
Cnie
CnieCnie
Cnie
 
Measuring Mobile Learning - EDLD 5315
Measuring Mobile Learning - EDLD 5315Measuring Mobile Learning - EDLD 5315
Measuring Mobile Learning - EDLD 5315
 

Similar to Evaluating the Usability of a Study Support Mobile App in Higher Ed

CapstonePaperFinal
CapstonePaperFinalCapstonePaperFinal
CapstonePaperFinal
Leia Dolphy
 
The readiness of academic staff at South Valley University to develop and imp...
The readiness of academic staff at South Valley University to develop and imp...The readiness of academic staff at South Valley University to develop and imp...
The readiness of academic staff at South Valley University to develop and imp...
Alaa Sadik
 
Suh an explanatory study of high school teachers’ integration of mobile learning
Suh an explanatory study of high school teachers’ integration of mobile learningSuh an explanatory study of high school teachers’ integration of mobile learning
Suh an explanatory study of high school teachers’ integration of mobile learning
Sylvia Suh
 
Models of evaluation in educational technology
Models of evaluation in educational technologyModels of evaluation in educational technology
Models of evaluation in educational technology
السيف البتار
 
Models of evaluation in educational technology
Models of evaluation in educational technologyModels of evaluation in educational technology
Models of evaluation in educational technology
u083486
 
Designing and Evaluating a Contextual Mobile Application to Support Situated ...
Designing and Evaluating a Contextual Mobile Application to Support Situated ...Designing and Evaluating a Contextual Mobile Application to Support Situated ...
Designing and Evaluating a Contextual Mobile Application to Support Situated ...
HCI Lab
 

Similar to Evaluating the Usability of a Study Support Mobile App in Higher Ed (20)

Enhancers and challenges to impactful mobile learning
Enhancers and challenges to impactful mobile learningEnhancers and challenges to impactful mobile learning
Enhancers and challenges to impactful mobile learning
 
CapstonePaperFinal
CapstonePaperFinalCapstonePaperFinal
CapstonePaperFinal
 
Digital tool presentation: AppsGyeser
Digital tool presentation: AppsGyeserDigital tool presentation: AppsGyeser
Digital tool presentation: AppsGyeser
 
Introduction to evaluation in educational technology
Introduction to evaluation in educational technology Introduction to evaluation in educational technology
Introduction to evaluation in educational technology
 
Users' evaluation of apps for Language Learning
Users' evaluation of apps for Language LearningUsers' evaluation of apps for Language Learning
Users' evaluation of apps for Language Learning
 
PowerPoint-based quizzes in wave motion: Performance and experiences of students
PowerPoint-based quizzes in wave motion: Performance and experiences of studentsPowerPoint-based quizzes in wave motion: Performance and experiences of students
PowerPoint-based quizzes in wave motion: Performance and experiences of students
 
The readiness of academic staff at South Valley University to develop and imp...
The readiness of academic staff at South Valley University to develop and imp...The readiness of academic staff at South Valley University to develop and imp...
The readiness of academic staff at South Valley University to develop and imp...
 
Approaching the Bleeding Edge: Possibilities & Practices for Learning Technol...
Approaching the Bleeding Edge: Possibilities & Practices for Learning Technol...Approaching the Bleeding Edge: Possibilities & Practices for Learning Technol...
Approaching the Bleeding Edge: Possibilities & Practices for Learning Technol...
 
Feedback as Dialogue
Feedback as DialogueFeedback as Dialogue
Feedback as Dialogue
 
Suh an explanatory study of high school teachers’ integration of mobile learning
Suh an explanatory study of high school teachers’ integration of mobile learningSuh an explanatory study of high school teachers’ integration of mobile learning
Suh an explanatory study of high school teachers’ integration of mobile learning
 
Models of evaluation in educational technology
Models of evaluation in educational technologyModels of evaluation in educational technology
Models of evaluation in educational technology
 
Evaluating The A Self Based Learning Format The Microsoft Word Documents
Evaluating The A Self Based Learning Format The Microsoft Word DocumentsEvaluating The A Self Based Learning Format The Microsoft Word Documents
Evaluating The A Self Based Learning Format The Microsoft Word Documents
 
Mecar2010
Mecar2010Mecar2010
Mecar2010
 
Tece ws01 20130626 final public
Tece ws01 20130626 final publicTece ws01 20130626 final public
Tece ws01 20130626 final public
 
Intentional Use of iPads by Teacher Educators
Intentional Use of iPads by Teacher EducatorsIntentional Use of iPads by Teacher Educators
Intentional Use of iPads by Teacher Educators
 
Models of evaluation in educational technology
Models of evaluation in educational technologyModels of evaluation in educational technology
Models of evaluation in educational technology
 
Mobile learning
Mobile learningMobile learning
Mobile learning
 
Making connections: Global Student Technology Survey
Making connections: Global Student Technology SurveyMaking connections: Global Student Technology Survey
Making connections: Global Student Technology Survey
 
Module 4 Assignment
Module 4 AssignmentModule 4 Assignment
Module 4 Assignment
 
Designing and Evaluating a Contextual Mobile Application to Support Situated ...
Designing and Evaluating a Contextual Mobile Application to Support Situated ...Designing and Evaluating a Contextual Mobile Application to Support Situated ...
Designing and Evaluating a Contextual Mobile Application to Support Situated ...
 

More from Mohammad Khalil

Gamification in MOOCs: a Review of the State of the Art
Gamification in MOOCs: a Review of the State of the ArtGamification in MOOCs: a Review of the State of the Art
Gamification in MOOCs: a Review of the State of the Art
Mohammad Khalil
 

More from Mohammad Khalil (20)

Learning analytics in a time of pandemics
Learning analytics in a time of pandemicsLearning analytics in a time of pandemics
Learning analytics in a time of pandemics
 
Compbiomed presentation
Compbiomed presentationCompbiomed presentation
Compbiomed presentation
 
The use of Learning Analytics to explore sequences of self-regulated learning...
The use of Learning Analytics to explore sequences of self-regulated learning...The use of Learning Analytics to explore sequences of self-regulated learning...
The use of Learning Analytics to explore sequences of self-regulated learning...
 
The unbearable lightness of consent: Mapping MOOC providers’ response to consent
The unbearable lightness of consent: Mapping MOOC providers’ response to consentThe unbearable lightness of consent: Mapping MOOC providers’ response to consent
The unbearable lightness of consent: Mapping MOOC providers’ response to consent
 
Gamification in MOOCs: a Review of the State of the Art
Gamification in MOOCs: a Review of the State of the ArtGamification in MOOCs: a Review of the State of the Art
Gamification in MOOCs: a Review of the State of the Art
 
Learning Analytics & MOOCs- Data Issues Perspective
Learning Analytics & MOOCs- Data Issues PerspectiveLearning Analytics & MOOCs- Data Issues Perspective
Learning Analytics & MOOCs- Data Issues Perspective
 
EMOOCs2018 announcement
EMOOCs2018 announcementEMOOCs2018 announcement
EMOOCs2018 announcement
 
Why Undergraduates Enroll in MOOCs?
Why Undergraduates Enroll in MOOCs?Why Undergraduates Enroll in MOOCs?
Why Undergraduates Enroll in MOOCs?
 
Learning Analytics in Massive Open Online Courses - PhD Defense
Learning Analytics in Massive Open Online Courses - PhD DefenseLearning Analytics in Massive Open Online Courses - PhD Defense
Learning Analytics in Massive Open Online Courses - PhD Defense
 
Does Gamification in MOOC discussion forums Work?
Does Gamification in MOOC discussion forums Work?Does Gamification in MOOC discussion forums Work?
Does Gamification in MOOC discussion forums Work?
 
When Learning Analytics Meets MOOCs - a Review on iMooX Case Studies
When Learning Analytics Meets MOOCs - a Review on iMooX Case StudiesWhen Learning Analytics Meets MOOCs - a Review on iMooX Case Studies
When Learning Analytics Meets MOOCs - a Review on iMooX Case Studies
 
Engaging Learning Analytics in MOOCs: the good, the bad, and the ugly
Engaging Learning Analytics in MOOCs: the good, the bad, and the uglyEngaging Learning Analytics in MOOCs: the good, the bad, and the ugly
Engaging Learning Analytics in MOOCs: the good, the bad, and the ugly
 
Technology Enhanced Learning - MOOCs and Learning Analytics
Technology Enhanced Learning - MOOCs and Learning AnalyticsTechnology Enhanced Learning - MOOCs and Learning Analytics
Technology Enhanced Learning - MOOCs and Learning Analytics
 
EMOOCs2016 MOOC Conference 2016
EMOOCs2016 MOOC Conference 2016EMOOCs2016 MOOC Conference 2016
EMOOCs2016 MOOC Conference 2016
 
On Using Learning Analytics to Track the Activity of Interactive MOOC Videos
On Using Learning Analytics to Track the Activity of Interactive MOOC VideosOn Using Learning Analytics to Track the Activity of Interactive MOOC Videos
On Using Learning Analytics to Track the Activity of Interactive MOOC Videos
 
What is Learning Analytics about? A Survey of Different Methods Used in 2013-...
What is Learning Analytics about? A Survey of Different Methods Used in 2013-...What is Learning Analytics about? A Survey of Different Methods Used in 2013-...
What is Learning Analytics about? A Survey of Different Methods Used in 2013-...
 
iOS Human Interface Guidelines (HCI)
iOS Human Interface Guidelines (HCI)iOS Human Interface Guidelines (HCI)
iOS Human Interface Guidelines (HCI)
 
a STEM MOOC for School Children - What does Learning Analytics Tell us?
a STEM MOOC for School Children - What does Learning Analytics Tell us?a STEM MOOC for School Children - What does Learning Analytics Tell us?
a STEM MOOC for School Children - What does Learning Analytics Tell us?
 
De-identification in Learning Analytics
De-identification in Learning AnalyticsDe-identification in Learning Analytics
De-identification in Learning Analytics
 
Learning Analytics: Principles and Constraints
Learning Analytics: Principles and ConstraintsLearning Analytics: Principles and Constraints
Learning Analytics: Principles and Constraints
 

Recently uploaded

ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
ashishpaul799
 
Neurulation and the formation of the neural tube
Neurulation and the formation of the neural tubeNeurulation and the formation of the neural tube
Neurulation and the formation of the neural tube
SaadHumayun7
 

Recently uploaded (20)

Research Methods in Psychology | Cambridge AS Level | Cambridge Assessment In...
Research Methods in Psychology | Cambridge AS Level | Cambridge Assessment In...Research Methods in Psychology | Cambridge AS Level | Cambridge Assessment In...
Research Methods in Psychology | Cambridge AS Level | Cambridge Assessment In...
 
Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17
 
[GDSC YCCE] Build with AI Online Presentation
[GDSC YCCE] Build with AI Online Presentation[GDSC YCCE] Build with AI Online Presentation
[GDSC YCCE] Build with AI Online Presentation
 
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
 
“O BEIJO” EM ARTE .
“O BEIJO” EM ARTE                       .“O BEIJO” EM ARTE                       .
“O BEIJO” EM ARTE .
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdf
 
The Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. HenryThe Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. Henry
 
philosophy and it's principles based on the life
philosophy and it's principles based on the lifephilosophy and it's principles based on the life
philosophy and it's principles based on the life
 
Removal Strategy _ FEFO _ Working with Perishable Products in Odoo 17
Removal Strategy _ FEFO _ Working with Perishable Products in Odoo 17Removal Strategy _ FEFO _ Working with Perishable Products in Odoo 17
Removal Strategy _ FEFO _ Working with Perishable Products in Odoo 17
 
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...
 
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxslides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).
 
Mbaye_Astou.Education Civica_Human Rights.pptx
Mbaye_Astou.Education Civica_Human Rights.pptxMbaye_Astou.Education Civica_Human Rights.pptx
Mbaye_Astou.Education Civica_Human Rights.pptx
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
 
Post Exam Fun(da) Intra UEM General Quiz - Finals.pdf
Post Exam Fun(da) Intra UEM General Quiz - Finals.pdfPost Exam Fun(da) Intra UEM General Quiz - Finals.pdf
Post Exam Fun(da) Intra UEM General Quiz - Finals.pdf
 
Neurulation and the formation of the neural tube
Neurulation and the formation of the neural tubeNeurulation and the formation of the neural tube
Neurulation and the formation of the neural tube
 
Gyanartha SciBizTech Quiz slideshare.pptx
Gyanartha SciBizTech Quiz slideshare.pptxGyanartha SciBizTech Quiz slideshare.pptx
Gyanartha SciBizTech Quiz slideshare.pptx
 
2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx
 
Basic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & EngineeringBasic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
 
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdfINU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
 

Evaluating the Usability of a Study Support Mobile App in Higher Ed

  • 1. Evaluating the Usability of a Study Support Mobile App in Higher Education Mohammad Khalil, Jacqueline Wong, Martine Baars, Farshida Zafar, & Barbara Wasson mLearn Conference, Delft 18.09.2019
  • 2. Background & Motivation ◉ Self-study is an important factor in student success (Wijnen et al., 2017) ◉ Self-study entails self-regulated learning (SRL) and use of effective learning strategies ◉ Self-testing can be use for self-monitoring as part of SRL (Baars, 2014) 2 Jacqueline Wong, 18.09.19, Delft
  • 3. Background & Motivation ◉ Smartphones account for 75% of mobile devices used in educational context (Wu et al., 2012) ◉ Growing number of apps used for learning ◉ Few studies examined the usability of self-study apps! ◉ Research on usability for self-study apps can help to: ○ Improve apps and enhance user experience ○ Carry out finer research ○ Support teacher instructions based on the difficulties students are facing ○ Implement support on app use for self-regulated learning, and in turn, improve learning performance 3
  • 4. “ 4 Usability is the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” -ISO FDIS 9241-210 Jacqueline Wong, 18.09.19, Delft
  • 5. What is the EUR App? Jacqueline Wong, 18.09.19, Delft
  • 6. 6 Eur App ● App developed in 2016 ● Made by the students for the students! ● Led by a lecturer in the law faculty - Farshida Zafar ● Formative assessment tool with a focus on gamification
  • 7. 1. Teacher Perspective 7 ● Teachers have control over the content. ● Teachers can drop in 250 multiple choice questions divided over categories ● Teachers can reference to a literature or video for feedback ● Quick changes can be done ‘on the fly’ ● Simple descriptive analytics (successful students and percentage of answers) Jacqueline Wong, 18.09.19, Delft
  • 8. 2. Student-face design 8 ● Four game modalities (practice, compete, test, and battle) ● Availability of instant feedback in practice mode ● The app focuses on knowledge testing and entertaining ● Gamification elements are available (i.e., leaderboard, time restrictions...etc.) Jacqueline Wong, 18.09.19, Delft
  • 9. 9 Eur App - Screenshots Jacqueline Wong, 18.09.19, Delft
  • 12. 12 Research Questions ◉ How often did the students use the app and which mode was mostly used for studying in the course? ◉ What were the students’ perceived usability and enjoyment of the app? Jacqueline Wong, 18.09.19, Delft
  • 13. Sample Profile ◉ Survey sent to 800 students ◉ Year 1 Bachelor students from Erasmus University ◉ Course “Introduction to constitutional and administrative law” ◉ Response rate was 10.12% ◉ Female : Male (65% : 36%) 13
  • 14. Criteria used for evaluation 14 ◉ Learnability: How easy is the application to learn and start working on. ◉ Efficiency: How efficient is the system in terms of productivity ◉ Memorability: How memorable is the application when a user leaves it for a while ◉ Errors: The application should have a low error profile without catastrophic bugs ◉ Satisfaction: How pleasant is the application to use and are the users satisfied of its functionalities? ◉ Enjoyment: Closely related to motivation by which we examine whether students were enjoyed using the application. ◉“It was easy for me to learn how to use the app and navigate through it” ◉I was able to use the app without frustration. ◉I remembered how to use the app when I leave it for a while ◉The app gave error messages when I made a mistake ◉I was satisfied with the app as a study tool ◉I enjoyed using the practice mode of the game app Nielsen (1994) and Mun & Hwang (2003)
  • 15. How often did the students use the app and which mode was mostly used for studying in the course? 15 RQ1
  • 16. What were the students’ perceived usability and enjoyment of the app? 16 Criteria Cronbach’s a Mean Learnability (4 items) .702 5.59 Efficiency (3 items) .716 5.43 Memorability (2 items) 4.31 Errors (2 items) 4.80 Satisfaction (4 items) .818 5.75 Enjoyment (5 items) .707 5.26 RQ2
  • 17. Results Mode usage The practice and test modes were the most used ones. The battle and compete modes were not as highly used. App usage Around 50% of the students used the app once a week to practice. 60% of the students used it to assess their knowledge Highest usability criteria Satisfaction (e.g., app is user-friendly) was the highest criterion of the usability testing Lowest usability criterions Memorability (e.g., remembering how the app works, app recalling same page…) was the lowest criterion of the usability testing Other facts Efficiency and enjoyment of the app were perceived on average by the students 17
  • 18. Study key outcomes ◉ Students used the app as more of a diagnostic tool rather than a study tool ◉ Perhaps, students prefer modalities that provide feedback rather than models with gamification ◉ According to Nielsen (1994), the application can be considered “Stable” ◉ The gamification models of the app should be improved 18
  • 19. Any questions ? Thanks! 19 Mohammad Khalil mohammad.khalil@uib.no Jacqueline Wong wong@essb.eur.nl Martine Baars Farshida Zafar Barbara Wasson
  • 20. Any questions ? You can find me at: ◉ wong@essb.eur.nl ◉ mohammad.khalil@uib.no Thanks! 20 Mohammad Khalil, Jacqueline Wong, Martine Baars, Farshida Zafar, & Barbara Wasson You can find me at: ◉ wong@essb.eur.nl ◉ mohammad.khalil@uib.no