Metrics and Messaging: What Works when Evaluating Blended Learning Implementations

1. Julie A. Evans, Ed.D.
Chief Executive Officer, Project Tomorrow
@JulieEvans_PT
Metrics and Messaging: What Works when
Evaluating Blended Learning Implementations
Tuesday, January 29, 2019 @ 2 pm

2. “Innovating is a process, not an event.”
Blended: Using Disruptive Innovation
to Improve Schools
by Michael B. Horn and Heather Staker

3. Today’s discussion
▪ Introduction to Project Tomorrow and the Speak Up Research Project
▪ Key research highlights about blended learning implementations
▪ Metrics for measuring impact – step by step process we recommend
▪ Messaging value and impact to your community
▪ Other resources for you
▪ Your thoughts, comments, ideas! Let’s talk!

4. Sign up to get a copy of today’s presentation!
1. Sign up on the table sheets
2. Sign up on the Google form
http://bit.ly/2HubYkw

5. Getting to know you!

6. Tweet
with us!
Share
your
thoughts!
@JulieEvans_PT @SpeakUpEd
@FETC #FETC #blendedlearning

7. About Project Tomorrow (www.tomorrow.org)
▪ Nonprofit education organization supporting K-12 education since 1996
▪ Mission is to ensure all of today’s students are well prepared for the future
▪ Programs and research focus on role of digital tools within the education
ecosystem – believe in power of STEAM to support student preparation for
college and career success
o Speak Up Research Project on Digital Learning: collecting & reporting
on the authentic feedback of K-12 stakeholders to inform federal, state
& local programs and policies
o Efficacy and evaluation studies on impact of digital learning

8. About the Speak Up Project (www.tomorrow.org/speakup)
▪ Annual research project since 2003
▪ Uses online surveys + focus groups
▪ Facilitated 100% through schools and districts
▪ We design online surveys to collect feedback from
your K-12 Students, Teachers, Parents, Administrators
and Community Members
▪ All K-12 schools – public, private, parochial, charter,
virtual - are eligible to participate
▪ Project Tomorrow manages all data collection and
reporting for you - 100% free service
▪ Schools get summary report with all locally collected
data + state and national data for benchmarks
We share national data with federal, state and local policymakers to inform programs and funding

9. About the Speak Up Project (www.tomorrow.org/speakup)
Standard survey question topics include:
✓ Use of technology to support learning
✓ School climate for innovation
✓ College and career ready skill development/interest
✓ Leadership challenges and capacities
✓ Teachers’ needs for professional learning
✓ How do different stakeholders value digital learning
✓ Emerging trends w/digital tools, content and resources
✓ New classroom models: mobile, blended, flipped
✓ School to home communications
✓ Student safety online & digital citizenship
✓ Designing the ultimate school
Since 2003, over 5.4 million K-12 stakeholders have submitted a Speak Up survey

10. Key trends from the Speak Up Research
• Greater emphasis on linking technology to students’ global skill preparation
• Interest, acceptance and implementation of new learning models
• Students’ self-directed digital learning influencing expectations for school
• Meet the new digital parent – new demands, higher aspirations
• Thinking beyond engagement to understand outcomes from digital learning
• Increasing criticality for Internet connectivity – at school and at home
• Digital learning is a metaphor today for education transformation
• The strong connection between leadership and effectiveness

11. About our Efficacy & Evaluation Studies
Contracted research projects to understand the efficacy of digital learning
▪ Extension of our Speak Up research – 40+ studies since 2007
▪ Specialization in evaluating mobile learning, new classroom models and the
use of digital content within instruction
▪ Study types: feedback studies, efficacy studies, secondary data analysis,
evaluations of outcomes, control & treatment groups
▪ Clients include K-12 districts, foundations, companies, nonprofit orgs
▪ Outcomes inform education planning, new initiatives, community
engagement strategies, development of new income, understanding
product impact, new solution development

12. Speak Up Research Findings:
State of blended learning in K-12 schools
Background and context: research results

13. Survey Audience # of Surveys Submitted
K-12 Students 340,927
Teachers & Librarians 34,833
Parents 23,159
Administrators 3,249
Community Members 4,611
Speak Up 2017 Data Set: 406,779 respondents
About participating schools and districts: 3,641 districts, 10,619 schools
29% urban, 34% rural, 37% suburban, 68% title 1 eligible

14. Per the Clayton Christensen Institute
(https://www.christenseninstitute.org/), blended learning is
a structured education program which includes both
traditional learning as well as online learning. In a blended
learning model, students spend some time in a traditional
classroom with a teacher providing instruction, and some
time following an online curriculum (at home or at school)
which allows them to have some control over the time,
place, path, or pace of their learning.
Definition of blended learning used within our research projects:

15. 22%
26%
26%
23%
26%
24%
26%
44%
59%
59%
54%
52%
49%
50%
0% 10% 20% 30% 40% 50% 60% 70%
ELEMENTARY SCHOOLS
MIDDLE SCHOOLS
HIGH SCHOOLS
URBAN COMMUNITIES
SUBURBAN COMMUNITIES
RURAL COMMUNITIES
TITLE 1 SCHOOLS
Blended learning implemented Demonstrated positive academic results from blended learning
Where is blended learning being implemented within K-12? Everywhere!

16. What do you think are the
benefits of implementing
blended learning environments
in schools?

17. What do principals say are the benefits of implementing blended learning?
Principals who have implemented blended learning cite these outcomes:
1. Extends learning opportunities for students (66% of the principals agree)
2. Increases personalization of instruction (61%)
3. Increases student engagement in learning (53%)
4. Students are more responsible for their own learning (48%)
5. Enhances the relevancy of the curriculum (48%)
6. Builds stronger school to home communications (46%)
7. Student academic outcomes are improved (45%)
8. Teachers’ skills using technology are improved and technology is used more
effectively (40%)

18. What do principals say are the challenges of implementing blended learning?
Middle school principals identify these as key challenges:
▪ Need to educate parents about at-home responsibilities with blended learning (64%)
▪ How to develop a student culture of responsibility for educational destiny (54%)
▪ Ensuring that every child has access to safe, convenient, sustained and appropriate
online access beyond the school campus (49%)
▪ Imperative that schools identify best practices to implement in the classroom (45%)
▪ Also necessary to re-think traditional instructional practices that support blended
learning (43%)
▪ Determining how to measure the impact of blended learning on educational goals
(42%)

19. How are school and district
administrators currently
evaluating the impact of their
digital initiatives?

20. How are
you
measuring
the impact
of digital
learning
initiatives?0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Teacher buy-
in/feedback
Student
achievement
results
Student
feedback
Class
observations
Student
engagement
All District Admin District Admin - small sized districts
District Admin - medium sized districts District Admin - large sized districts

21. Efficacy and Evaluation Studies:
Evaluating impact of blended learning
Background and context: research results

22. Efficacy and Evaluation Studies:
Evaluating impact of blended learning
Two different blended learning impact studies designed and conducted
by Project Tomorrow:
A. Poudre Global Academy (CO) - 2015
o Evaluating impact of blended learning on student achievement
B. 9 K-8 parochial schools in Los Angeles – 3 year study – 2019
o Evaluating impact of blended learning on student achievement,
enrollment and school financial health

23. Efficacy and Evaluation Studies:
Evaluating impact of blended learning
B. Study: 9 K-8 parochial schools in Los Angeles
o Why blended learning?
▪ Desire to improve academic outcomes and close achievement
gap – disadvantaged students, students with limited EL
▪ Motivation to change teacher skills using technology
▪ Motivation to update teacher instructional practices
▪ Upgrade profile of school to meet parent expectations
▪ Competitive pressures from charter schools
▪ Goal to better prepare students for demanding high school
experiences, college readiness and career opportunities

24. Efficacy and Evaluation Studies:
Evaluating impact of blended learning
B. Study: 9 K-8 parochial schools in Los Angeles
o Evaluating impact of blended learning on student achievement
▪ Secondary data analysis
▪ Reading Gr 2-8
▪ Math Gr 2-8
▪ Standardized assessments used in all 9 schools
▪ 3 school years of data
▪ Schoolwide and grade level views (aggregated student data)
▪ 2015 – 2018
Analysis theme: to see evidence of effectiveness = consistency and
constancy of growth and achievement – schoolwide + at pivotal grades

25. In common variables we
know about
Divergent variables we
know about
Variables we don’t know
about
• Grade levels
• School culture
• Blended learning
rotation model
• Commitment to
closing student
achievement gaps
• School leadership
commitment to
blended learning
• Provider model
• Curriculum products
• Implementation years
• School environment
• Prior emphasis on data
or gap analysis
• Teacher turnover
• Student demographics
• Technology platforms
• Teacher effectiveness
• Principal effectiveness
• Teacher skills
• Teacher attitudes or
values
• Fidelity to the model
• Parental support or
satisfaction
• Attendance issues
B. Study: 9 parochial schools in Los Angeles – study limitations
Result: need to use both traditional metrics and new metrics for the data analysis
Result: examine pivotal grades of 3rd and 8th

26. B. Study: 9 parochial schools in Los Angeles – key findings – achievement
o Maturity with a blended learning model matters. Schools with 3+ years of
experience with blended learning had stronger performance in reading and math
based upon the review of traditional achievement metrics that emphasis student
growth.
o Blended learning provides evidence of student proficiency as well as growth.
Thinking beyond growth to proficiency measures, tangible outcomes are evident
from the blended learning implementations in the following areas:
▪ Sustained increases in students’ scaled scores in reading in grades 3 and 8
▪ Accomplishment of a minimum of one grade level growth in reading and math
in grades 3 and 8
▪ Eight of nine schools had their 8th graders graduate in spring 2017 with reading
levels that are at or with 10% of being classified as college ready

27. B. Study: 9 parochial schools in Los Angeles – key findings – achievement
o Maturity with a blended learning model matters.
0
10
20
30
40
50
60
70
80
90
Median PR Median SPG % above 65 PR % proficient above
CAASPP
School 6 School 4 School 8 School 2 School 1
School 3 School 9 School 7 School 5
Schools 5, 6,
7 and 9
each had a
minimum of
3 years
experience
with
blended
learning

28. B. Study: 9 parochial schools in Los Angeles – key findings – achievement
o Blended learning provides evidence of student proficiency as well as growth.
Grade 3 Reading – Comparative Median Scaled Scores and Annual Growth

29. B. Study: 9 parochial schools in Los Angeles – key findings – achievement
o Blended learning provides evidence of student proficiency as well as growth.
Grade 8 Reading – Comparative Lexile Levels
Lexile range for grade 8:
1050 – beginning of 8th grade
1100 – beginning of 9th grade
+ 1200 – college ready reading

30. Today’s discussion
▪ Introduction to Project Tomorrow and the Speak Up Research Project
▪ Key research highlights about blended learning implementations
▪ Metrics for measuring impact – step by step process we recommend
▪ Messaging value and impact to your community
▪ Other resources for you
▪ Your thoughts, comments, ideas! Let’s talk!

31. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact
of your blended learning programs
1. Identify your
goal(s) for
blended
learning
2. Determine
the value of
those goal(s)
3. Establish the
necessary
proof cases
4. Verify
realism of
measuring
outcomes
5. Agree on
logistics of
measurement
and reporting
6. Determine
most
appropriate
messaging for
outcomes

32. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact
of your blended learning programs
1. Identify your
goal(s) for
blended
learning
▪ Back to the beginning – what are you trying to achieve?
▪ Does blended learning as an initiative fit that goal?
▪ Is there some modicum of anticipated success?
▪ Be careful about “goal bingeing”
▪ “One goal is a goal, two goals are a goal and a half, three goals = no plan”
Common driving goal for all 9 LA schools: closing
achievement gap through increased academic
achievement for all students

33. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact
of your blended learning programs
2. Determine
the value of
those goal(s)
▪ What does success achieving this goal mean to your district, your school &
to you?
▪ How does achieving this goal support your organizational vision or mission?
▪ Does the VOI (value of investment) align with the effort?
▪ Do we have stakeholder buy-in on the value of this goal?
▪ What happens if we don’t do this …. What is the impact of not achieving
this goal?
Common value proposition: proving academic achievement
with blended learning will result in
increased enrollments and financial stability

34. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact
of your blended learning programs
3. Establish the
necessary
proof cases
▪ What is the physical representation of success with this goal?
▪ What constitutes adequate proof of success with this goal?
▪ What types of proof are necessary – ex: quantitative vs. qualitative?
▪ Is there stakeholder agreement as to what constitute adequate proof?
▪ Does the proof case carry the same weight with different audiences?
▪ Do your audiences need an education as to why this proof is valuable?
Common proof case: what matters to parents are test score
results, acceptance into prestigious high schools, engaged
happy children, teachers that care about their child

35. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact
of your blended learning programs
4. Verify
realism of
measuring
outcomes
▪ Is it realistic and possible to measure outcomes to support that proof case?
▪ Is it realistic and possible to collect supporting data for the proof case?
▪ Will the measured outcomes be valid?
▪ Will the measured outcomes be acceptable or adequate to pass muster
with the key stakeholders?
Realistic and possible: secondary data analysis of existing
achievement data
Not realistic or possible (at this time): measuring levels of
engagement or teacher concern for children

36. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact
of your blended learning programs
5. Agree on
logistics of
measurement
and reporting
▪ Agreement is essential here: are we all on the same page?
▪ Big questions to ask: what, where, when, who
▪ Who should do the data collection and analysis to establish validity?
▪ Who should create the reporting to ensure validity?
▪ Are the analytical and reporting processes appropriate for the task?
Agreement: Buy-in from principals as to the data analysis
process and sharing data with the third party evaluator;
value of the third party from a reporting perspective

37. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact
of your blended learning programs
6. Determine
most
appropriate
messaging for
outcomes
▪ Back to the beginning: what are you trying to achieve?
▪ Does the evaluation findings support those goals? Or are there surprises?
▪ What do your stakeholders need to understand the value of the process – a
lot of data, analyzed information, next steps …. ?
▪ How to articulate your messaging so that it achieves your goals?
▪ Solicit feedback on your messaging and monitor reactions
Messaging outcomes: Selecting high impact statements
that resonate with targeted audiences (ex: 8th graders
reading at college ready levels)

38. Project Tomorrow’s recommendations for a step-by-step process for evaluating the
impact of . . . . . any of your digital learning initiatives
1. Identify your
goal(s) for
blended
learning
2. Determine
the value of
those goal(s)
3. Establish the
necessary
proof cases
4. Verify
realism of
measuring
outcomes
5. Agree on
logistics of
measurement
and reporting
6. Determine
most
appropriate
messaging for
outcomes

39. Today’s discussion
▪ Introduction to Project Tomorrow and the Speak Up Research Project
▪ Key research highlights about blended learning implementations
▪ Metrics for measuring impact – step by step process we recommend
▪ Messaging value and impact to your community
▪ Other resources for you
▪ Your thoughts, comments, ideas! Let’s talk!

40. What conditions or elements
have to exist to make a blended
learning implementation
successful?

41. ❖ Supportive, visionary, involved leadership
❖ Teacher confidence and comfort with tools and process
❖ Continuity of staff as model matures
❖ Appropriate tools
❖ Culture that values data and improvement
❖ Classroom practices and habits that work
❖ Student growth mindset: empowerment around learning
❖ Parents that understand and value personalized learning
❖ Maturity with model: move from growth to proficiency
❖ Be open to revisions, re-invention, re-imagining of your initiative as it
matures
From Project Tomorrow research: what makes blended learning successful?
These 10 elements are essential:

42. More resources available at www.tomorrow.org
National Speak Up reports and infographics
Targeted and thematic reports
Educational Equity
Digital learning trends
School communications
Mobile learning
Blended learning outcomes
Presentations, podcasts and webinars
Services: consulting, workshops, evaluation and efficacy studies
✓ We have expertise in the evaluation of mobile implementations, new
classroom models and digital content usage

43. Invitation for your school or district to participate in Speak Up 2018
Online surveys for:
o K-12 students – individual + group
o Parents – English and Spanish
o Teachers
o Librarians/Media Specialists
o School Site & District Administrators
o Technology Leaders
o Community Members
Surveys open now through June 28, 2019
Learn more www.tomorrow.org/speakup
100% free service to all schools and districts – including reports with comparison data

44. Sign up to get a copy of today’s presentation!
1. Sign up on the table sheets
2. Sign up on the Google form
http://bit.ly/2HubYkw

45. Your thoughts,
comments,
ideas
Let’s talk!

46. Thank you for joining me today!
Please review this session on the mobile app.
Julie A. Evans, Ed.D.
Chief Executive Officer, Project Tomorrow
@JulieEvans_PT
jevans@tomorrow.org
Metrics and Messaging: What Works when Evaluating
Blended Learning Implementations
1/29/19 @ 2 pm – C142