Best Practices for Evaluating Mobile Learning

Best Practices for Evaluating
Mobile Learning:
Real Stories from Around the Globe
Dr. Julie A. Evans
CEO, Project Tomorrow
June 25, 2018
@SpeakUpEd
@JulieEvans_PT

Dr. Julie A. Evans
Chief Executive Officer, Project Tomorrow
Chief Researcher, Speak Up Research Project
Mobile Learning Project Evaluator since 2007
Also, skilled in the evaluation of blended learning
and digital content usage in K-12 and higher
education
Thank you for being here!

Special Guest:
Hiram Lopez-Landin
Senior Analyst, Wireless Reach
Qualcomm

Today’s discussion topics
▪ About our work
o Project Tomorrow and the Speak Up Research Project
o Qualcomm Wireless Reach
▪ Why are evaluations of mobile learning important today?
▪ State of mobile learning today & evaluations for mobile learning
▪ Let’s talk about 3 different mobile learning projects
▪ My top 8 tips and strategies for evaluating mobile learning
▪ Extended learning – links to some additional resources
▪ Your thoughts, insights, questions

Tweet with us!
@JulieEvans_PT @SpeakUpEd
@hlopezlandin @QCWirelessReach

About Project Tomorrow (www.tomorrow.org)
▪ Nonprofit education organization supporting K-12 education since 1996
▪ Mission is to ensure all of today’s students are well prepared for the future
▪ Programs and research focus on role of digital tools within the education
ecosystem – believe in power of STEAM to support student preparation for
college and career success
o Speak Up Research Project on Digital Learning: collecting & reporting
on the authentic feedback of K-12 stakeholders to inform federal, state
& local programs and policies

About the Speak Up Project (www.tomorrow.org/speakup)
▪ Annual research project since 2003
▪ Uses online surveys + focus groups
▪ Facilitated 100% through schools and districts
▪ We design online surveys to collect feedback from
your K-12 Students, Teachers, Parents, Administrators,
and Community Members
▪ All K-12 schools – public, private, parochial, charter,
virtual - are eligible to participate
▪ Project Tomorrow manages all data collection and
reporting for you - 100% free service
▪ Schools get summary report with all locally collected
data + state and national data for benchmarks
We share national data with federal, state and local policymakers to inform programs and funding

About the Speak Up Project (www.tomorrow.org/speakup)
Survey question topics include:
✓ Use of technology to support learning
✓ School climate for innovation
✓ College and career ready skill development/interest
✓ Leadership challenges
✓ Teachers’ needs for professional learning
✓ How do different stakeholders value digital learning
✓ Emerging trends w/digital tools, content and resources
✓ New classroom models: mobile, blended, flipped
✓ School to home communications
✓ Designing the ultimate school
❖ Activities
❖ Attitudes
❖ Aspirations
Since 2003, 5.4 million K-12 stakeholders have
submitted a Speak Up survey

11
45
PROGRAMS
17
COUNTRIES
150
STAKEHOLDERS
270K
PEOPLE
Wireless Reach Education Portfolio Impact to Date

12
17
9
7
6
2
2
1
1
United States
East Asia
Southeast Asia
South Asia
Middle…
Latin America
Sub-Saharan…
Europe
Programs by Region
Percent of female
beneficiaries in all
education programs
49%
Organizations served
in education portfolio570
Wireless Reach Education 10 Years of Accomplishments
Devices used in
education portfolio11,000
All figures include aggregated data from 2006 to 2017

13
Education Strategy
Wireless Reach education
programs are designed to address
the barriers to adoption of wireless
technology in the classroom
including the need for:
• Digital content & assessment
• Infrastructure & IT support
• Privacy & security

Why is it important to
evaluate our mobile
learning projects?

Why is it important to evaluate our mobile learning projects?
Are our investments yielding a return on investment?

What do you see?
Are we all seeing
“evaluation” or even
“mobile learning”
the same way?

State of Mobile Learning: selected research
findings from Speak Up 2017
State of Mobile Learning: selected research findings from Speak Up 2017
Learning on the go – anytime, anywhere

Survey Audience # of Surveys Submitted
K-12 Students 340,927
Teachers & Librarians 34,833
Parents 23,159
Administrators 3,249
Community Members 4,611
About schools and districts: 3,641 districts, 10,619 schools
29% urban, 34% rural, 37% suburban, 68% title 1 eligible
National participation in Speak Up 2017: 406,779

@Project Tomorrow 2017
Students have more personal access to mobile devices than ever before
35%
50%
79%
91%
53%
57% 55%
39%
32% 35%
48%
54%
10%
15% 14% 13%15% 12%
K-2 Gr 3-5 Gr 6-8 Gr 9-12
Smartphone Tablet Laptop Digital Reader Smartwatch

Teachers and mobile devices within learning
Classroom access is changing rapidly
In class
access
2015 2017
BYOD 17% 23%
1:1 in school
assignment
20% 31%
1:1 in school
+ home
assignment
12% 14%
Cart access 31% 30%
No access 31% 18%
Not surprising: teachers want
more PD on how to
effectively integrate mobile
devices within instruction

Administrators say mobile learning = positive results
▪ 49% of school principals say they see positive academic results from their 1:1
mobile learning programs
Benefits of mobile learning Elementary
school
principals
Middle school
principals
High school
principals
Students develop topics more fully 82% 83% 77%
Students are more engaged in learning 72% 62% 55%
Preparation for college/work 71% 71% 71%
Increased access to online content 69% 73% 73%
Learning is more personalized 65% 60% 57%
Students are collaborating more 57% 58% 58%
Improved teacher-student communications 48% 58% 63%

But what metrics are school and
district leaders using to measure
outcomes from mobile learning?

Turn to a neighbor and share:
How are you evaluating the success of your mobile learning programs?
How satisfied are you with your evaluation process?

Administrators say these are the top ways they are measuring the impact
of their digital initiatives:
Metrics % of district
administrators
nationwide
Which ones are you
using?
Student engagement levels 77%
Classroom observations 69%
Student feedback 62%
Student achievement results 60%
Teacher buy-in 54%
Teacher feedback 53%
Parent feedback 37%

Two of the 8 essentials:
✓ Purposeful planning for mobile
device usage
✓ Measuring project results with
meaningful metrics

✓ Purposeful planning
for mobile device
usage
▪ What are your goals?
▪ What outcomes do
you want to see?
▪ Why is this important
for your school or
district?
✓ Measuring project
results with
meaningful metrics
▪ Aligning metrics with
goals
▪ Leveraging
appropriate tools
▪ Being open to what
you don’t know yet
Evaluating mobile learning begins with project planning

Introduction to 3 mobile learning projects, the metrics they used/are
using to evaluate program success & lessons learned that can inform
your efforts:
❑ “Making Learning Mobile” – Chicago
❑ “STEAMing Ahead with Mobile Learning – San Diego
❑ “Classroom of the Future” – Turkey
Let’s get real!

“Making Learning Mobile” – Chicago
o Three year focused study on the impact of
extended Internet access and mobile learning
on student achievement
o Study group: 5th grade students and their
teachers at Falconer Elementary School in
Chicago, IL
o Students each assigned a Samsung Android
tablet with 4G LTE enabled education
broadband to use at school and at home
Partners: Kajeet, Qualcomm Wireless Reach, Project Tomorrow

Objectives of the project was to learn more about:
✓ How did the students and teachers use the tablets and extended Internet
connectivity for learning?
✓ What benefits did they derive from those experiences?
✓ What was the impact of the experiences on student learning and overall
academic achievement?
Evaluation metrics included:
• Feedback data from students and teachers – analysis of changes over 3 years
• Classroom observations to evaluate depth of usage and impact of home access
• Analysis of student achievement results

Key findings from the study:
1. Tablet became an indispensable learning tool
2. Tablets and extended connectivity enabled greater
communications between students and teachers
3. Benefits to students included development of
college/career ready skills and new learning
behaviors
4. Teachers’ skills, motivation and readiness for tablet
integration impacted effectiveness of usage
5. Purposeful integration within math = higher
student achievement and growth

Insights from the evaluation of this project:
▪ Build your evaluation methodology and tools to be flexible and open to “surprises”
▪ Understand that the teacher adoption/assimilation/adaption process may be a
three year adventure – be reasonable with project goals and the timeline for
intended outcomes, and adjust your logical framework accordingly
▪ Reality alert: traditional data collection and analysis tools may not work for
mobile learning
▪ With mobile learning, it is essential to collect and analyze both quantitative and
qualitative data – feedback matters
▪ Keep eyes wide open for lessons learned – these are gold!

“STEAMing Ahead with Mobile Learning” – San Diego
Partners: San Diego Public Library, e3 Civic High School,
Qualcomm Wireless Reach, Project Tomorrow
9th graders were provided with 4GL tablet
that included an AR app to learn about the
STEAM elements within the Central Library
Dome design, engineering and construction

✓ How a contextually relevant learning experience for students could introduce
students to the role of STEAM within construction and design
✓ The role of tablets and 4GL in increasing student engagement in learning
✓ How to effectively increase student interest in STEAM career fields
✓ The ways in which this application could be used within other curricular fields
and with the general public
• Feedback data from students and teachers
• Observation of the learning activity with “person on the street” interviews
• Analysis of changes in students’ interest levels in STEAM career fields

Key findings from the study:
1. Students said use of AR content increased their engagement
in learning
2. Benefits included having opportunity to work collaboratively
with classmates on a real world activity; enjoyment level
surprised the students!
3. Different learning modalities within the experience appealed
to different students: videos, images, text, assessment
4. 40% of the were more interested in a STEAM career field as a
result of this learning experience
5. Teachers’ excited about how to use AR in other curricular
areas; library staff exploring ways to bring same content to
general public

Insights from the evaluation of this project:
▪ A small project evaluation period demands tighter adherence to project goals.
Project goals therefore must be clearly defined with broad buy-in from all
stakeholders and partners.
▪ Critical to use the right tools to collect data. Example: to show change in attitudes,
leverage pre and post data collection to “see” process of discovery and new ideas
▪ Reality alert: students are not familiar with all new technologies and thus, the
awareness/tool learning process must be taken into account within the analysis of
impact
▪ With a new technology such as AR, bring teachers into the evaluation
development process upfront so that the analysis of the outcomes is based upon a
realistic understanding of the environment

“Classroom of the Future” – Turkey
Partners: Turkish Education Association, Qualcomm Wireless Reach
▪ 7th grade students and their teachers from 7
schools have Qualcomm-enabled tablets
with wireless connectivity to support
learning in school and at home
▪ Initial focus was on English instruction
classes but project has expanded to include
other subject areas now
▪ Teachers were provided with specialized
professional development to support this
project

✓ How the inclusion of mobile devices in the learning environment enhances student
learning and engagement particularly in reading and writing
✓ How the mobile learning experience improves students’ perceptions of their
technology skills especially as an important global workplace skill
✓ If the teacher PD and the use of the devices in the classroom results in increases in
teachers’ capacity, comfort and confidence in implementing mobile learning
• Feedback data from students and teachers – pre and post surveys
• Aligning pre and post survey data elements directly with project goals
• Linking teacher data with student data for analysis of impact

Key findings from the study (so far!):
1. Students and teachers agree: the use of the tablets with wireless
connectivity both at school and at home are important to support learning.
2. 3/4 of the teachers reported: mobile learning environment resulted in
improved student engagement, enhanced quality of their students’ work and
students taking greater responsibility for their own learning.
3. The availability of the 1:1 tablets for student use at school and at home
increased the students’ overall use of technology for learning - 86% of the
students developed stronger technology skills during the school year.
4. With the new tools and training, 88% of the teachers reported greater
communications with their students and 76% noted an increase in their
overall effectiveness as a teacher.

Interim insights from the evaluation of this project:
▪ Even as project expands to include new teachers and schools, it is important to
keep the objectives and evaluation plan stable so that you can evaluate for growth
and maturity alongside the success of the replication.
▪ Analyzing teacher and student data at the class/school level side by side provided
valuable insights into the relationship between usage, personal valuation and
impact.
▪ Monitoring is not the same as evaluation. But monitoring is important for a valid
evaluation.
▪ A good evaluation demands good data. To get good data, you have to ask the right
questions.

Selection of tips and
strategies you should think
about when evaluating your
own mobile learning
projects
8

Selection of tips and strategies to use when evaluating your own mobile learning projects
1. Evaluation is not magic. Write out a plan and do that at the
same time you are developing your operational plan for
implementation so it aligns to your objectives for your project .

2. The quality of the data will determine the validity of your
evaluation – and thus, what you will learn. You should always
worry about the validity of your data!

3. Use the right tool for the right objectives in your plan. Some
tools work better than others for certain objectives.

3. Use the right tool for the right objectives in your plan. Some
tools work better than others for certain objectives.
4. Evaluation is ongoing – full year and year to year

5. Just because you use a mobile device and you are in education don’t
assume that you know how to evaluate a mobile learning project.
Evaluation is tricky business to get right!

5. Just because you use a mobile device and you are in education does not
mean that you automatically know how to evaluate a mobile learning
project. This is tricky business to get right!
6. Keep your eyes wide open for surprises and unintended outcomes –
mobile learning is not “jello” yet. This field is still developing and there
is no recipe or formula for success yet.

5. Just because you use a mobile device and you are in education does not
mean that you automatically know how to evaluate a mobile learning
project. This is tricky business to get right!
6. Keep your eyes wide open for surprises and unintended outcomes –
mobile learning is not “jello” yet. This field is still developing and there
is no recipe or formula for success yet.
7. Be realistic about your outcomes especially regarding the timeframes in
which you expect to see those outcomes. To get to the big impact
outcomes takes a lot of work in the classroom!

8. Most importantly, an evaluation of a
mobile learning project is about so much
more than mobile devices. It is really
about the transformative learning
experiences at your school.
Take advantage of that!

More resources available for extended learning
https://www.tomorrow.org/publications/MakingLearningMobile.html
▪ 3 years of evaluation reports from the Making Learning Mobile project
https://www.qualcomm.com/media/documents/files/united-states-
making-learning-mobile-mlm-project.pdf
▪ Case study on the Qualcomm Wireless Reach site
Additional information about the projects featured today:
Making Learning Mobile – Chicago

More resources available for extended learning
Additional information about the projects featured today:
STEAMing Ahead with Mobile Learning – San Diego
https://www.qualcomm.com/media/documents/files/united-
states-steaming-ahead-with-mobile-learning.pdf
Classroom of the Future – Turkey
(not published yet – hard copies provided today for you!)

More resources available at www.tomorrow.org
National Speak Up reports and infographics
Targeted and thematic reports
Digital learning trends
Community engagement
Mobile learning – infographic!
Games in the classroom
Blended learning outcomes
Presentations, podcasts and webinars
Services: consulting, workshops, evaluation and efficacy studies
✓ We have expertise in the evaluation of mobile implementations, new
classroom models and digital content usage
New Speak Up 2017 reports released thru Sept 2018

Plan now for your district to participate in Speak Up 2018
Online surveys for:
o K-12 students – individual + group
o Parents – English and Spanish
o Teachers
o Librarians/Media Specialists
o School Site & District Administrators
o Technology Leaders
o Community Members &
Communications Officers
Surveys open Oct 2018 – Jan 2019
Learn more www.tomorrow.org/speakup
100% free service to all schools and districts – including reports with comparison data

Your questions,
comments,
thoughts

Best Practices for Evaluating Mobile Learning

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best Practices for Evaluating Mobile Learning

Similar to Best Practices for Evaluating Mobile Learning (20)

More from Julie Evans

More from Julie Evans (20)

Recently uploaded

Recently uploaded (20)

Best Practices for Evaluating Mobile Learning