The Real Metrics that Educators use to Identify and Evaluate New Products for Classroom Usage

1. Insights from the Speak Up 2017 Research Results
So Many Products, So Much Data:
The Real Metrics that Educators Use to Identify and Evaluate
New Products for Classroom Usage
Dr. Julie A. Evans
CEO, Project Tomorrow
April 18, 2018

3. How do we determine the quality of
digital learning products and tools?
How do we meaningfully measure
impact and ROI?
So Many Products, So Much Data

4. Let’s see what
you think
Let’s see what you think!

5. What do you think school principals nationwide said?
What attribute is most important when evaluating the quality of
digital products for classroom usage?
A. Product is available with multiple reading levels
B. Product is research based
C. Product demonstrates student achievement results
D. Product includes teacher training
E. Product is created by a practicing teacher
@JulieEvans_PT @SpeakUpEd @asugsvsummit

6. What do you think district administrators nationwide said?
What metric is most effective for measuring the impact or ROI of
your digital initiatives?
A. Feedback from teachers
B. Student achievement results
C. Students’ level of engagement in learning
D. Observations in the classroom
E. Feedback from students
@JulieEvans_PT @SpeakUpEd @asugsvsummit

7. Let’s see what
you think
Stay tuned ….

8. What is the state of
digital learning in our
schools and districts
today?

9. About the Speak Up Project (www.tomorrow.org/speakup)
▪ Annual research project since 2003
▪ Uses online surveys + focus groups
▪ Facilitated 100% through schools and districts
▪ Free service for districts: schools get summary report
with all locally collected data + state and national data
for benchmarks
▪ National data is shared with federal, state and local
policymakers to inform programs and funding
▪ Largest collection of authentic feedback from K-12
stakeholders about state of digital learning today
Since 2003, over 5.4 million K-12 stakeholders
have submitted a Speak Up survey

10. Big picture trends from our research …
• Greater emphasis on students’ global skill preparation using digital tools
• Value of personalized learning on the rise
• Interest and acceptance of new learning models
• Increasing criticality for Internet connectivity – at school and at home
• Learning is a 24/7 enterprise for students
• New expectations from parents – digital aspirations esp. for communications
• Strong desire to understand real outcomes from digital learning
• Digital learning is a metaphor today for education transformation

11. @Project Tomorrow 2017

12. What do you see?
Are we seeing the same
thing?

13. “Without data, you are
just another person with
an opinion …”
Introducing the Speak Up Research Project to inform
new discussions and better decision-making around
using technology to transform education

14. What is the state of digital
learning in our schools and
districts today?
Data Source:
Speak Up 2017 results – 406,000 K-12 stakeholders

15. Insights from the Speak Up 2017 Research Results
Wake up issue for majority of our nation’s Superintendents
Closing the achievement gap
Best way to do this?
Make our teachers more effective
Integrate college and career ready
skills into curriculum
Leverage digital tools, content and
resources to personalize learning
for all
@JulieEvans_PT @SpeakUpEd @asugsvsummit

16. Speak Up Maturity – Growth Model: District Digital Initiatives
Insights from the Speak Up 2017 Research Results
Mature Initiatives
▪ Cloud computing
applications
▪ Use of video within
instruction
▪ Online tests
▪ Students have periodic
access to devices
▪ Using data to improve
instruction
▪ Using social media to
communicate with parents
Developing Initiatives
▪ Online PD for teachers
▪ Online classes for students
▪ Online textbooks
▪ Students assigned devices
to use at school only
Emerging Initiatives
▪ Blended learning models
▪ Flipped learning models
▪ Competency based models
▪ Use of OER content
▪ Students assigned devices
to use at school and take
home
@JulieEvans_PT @SpeakUpEd @asugsvsummit

17. Speak Up Maturity – Growth Model: District Digital Initiatives
Insights from the Speak Up 2017 Research Results
Mature Initiatives
▪ Cloud computing
applications
▪ Use of video within
instruction
▪ Online tests
▪ Students have periodic
access to devices
▪ Using data to improve
instruction
▪ Using social media to
communicate with parents
Developing Initiatives
▪ Online PD for teachers
▪ Online classes for students
▪ Online textbooks
▪ Students assigned devices
to use at school only
Emerging Initiatives
▪ Blended learning models
▪ Flipped learning models
▪ Competency based models
▪ Use of OER content
▪ Students assigned devices
to use at school and take
home
From low to high anticipated growth

18. Insights from the Speak Up 2017 Research Results
In what subjects is technology being used most effectively?
@JulieEvans_PT @SpeakUpEd @asugsvsummit
Mathematics
English/Reading
Science

19. Insights from the Speak Up 2017 Research Results
What is the value proposition for administrators?
Digital learning …..
✓ Increases student engagement in learning
✓ Improves quality, relevancy and consistency of instructional materials
✓ Improves teachers’ skills using technology
✓ Provides way for learning to be personalized for every student
✓ Identifies the school/district as “innovative” compared to peers
@JulieEvans_PT @SpeakUpEd @asugsvsummit

20. Insights from the Speak Up 2017 Research Results
68%
32%
0%
62%
38%
0%
44%
53%
1%
48% 48%
2%
Extremely important Somewhat important Not important
District Administrators School Principals Teachers Parents
How important is the
use of technology
within learning for a
student’s future
success?
@JulieEvans_PT @SpeakUpEd @asugsvsummit

21. Insights from the Speak Up 2017 Research Results
What is holding back greater adoption of digital learning?
Administrators report three key barriers to greater adoption:
✓ Motivating and training teachers to change their instructional practices
✓ Providing infrastructure to support high impact usage
✓ Determining quality and appropriateness of digital products for
classroom usage
@JulieEvans_PT @SpeakUpEd @asugsvsummit

22. How do we determine the quality of
digital learning products and tools?
So Many Products, So Much Data

23. Insights from the Speak Up 2017 Research Results
@JulieEvans_PT @SpeakUpEd @asugsvsummit
How do we determine the quality of digital learning products and tools?
School site principals identify:
✓ Product features
✓ Source trustworthiness

24. Insights from the Speak Up 2017 Research Results
Considered “very important” product features – for determining quality
Product features Elementary
school principals
Middle school
principals
High school
principals
✓ Aligned to curriculum 89% 84% 82%
✓ Multiple reading levels 88% 84% 82%
✓ Content is current 86% 82% 83%
✓ Content can be modified by teacher 79% 78% 77%
✓ Rich data set derived from the
content
75% 68% 65%
✓ Individual student accounts 69% 60% 61%
✓ Includes teacher PD 65% 61% 60%
✓ Available on multiple devices 60% 55% 63%
✓ Online assessments included 59% 57% 57%

25. Insights from the Speak Up 2017 Research Results
Considered “very important” sources for trustworthiness
Source attributes Elementary school
principals
Middle school
principals
High school
principals
✓ Content is research based 84% 79% 79%
✓ Demonstrated student achievement
results from product usage
78% 72% 70%
✓ Created by practicing teachers 53% 48% 56%
✓ Teachers have evaluated the product 53% 49% 54%
✓ Content created by an expert org 42% 43% 44%
✓ Referred by a trusted colleague 35% 33% 36%

26. Let’s see what
you think
Our audience polls ….

27. What do you think school principals nationwide said?
What attribute is most important when evaluating the quality of
digital products for classroom usage?
A. Product is available with multiple reading levels
B. Product is research based
C. Product demonstrates student achievement results
D. Product includes teacher training
E. Product is created by a practicing teacher
@JulieEvans_PT @SpeakUpEd @asugsvsummit

28. What do you think school principals nationwide said?
What attribute is most important when evaluating the quality of
digital products for classroom usage?
A. Product is available with multiple reading levels (84%)
B. Product is research based (82%)
C. Product demonstrates student achievement results (75%)
D. Product includes teacher training (63%)
E. Product is created by a practicing teacher (53%)
@JulieEvans_PT @SpeakUpEd @asugsvsummit

29. How do we meaningfully measure
impact and ROI?
So Many Products, So Much Data

30. Insights from the Speak Up 2017 Research Results
What administrators say are essential conditions for a
successful digital learning initiative
Funding $
Teacher buy-inParent buy-in
PoliciesTechnology Capacity
@JulieEvans_PT @SpeakUpEd @asugsvsummit

31. Insights from the Speak Up 2017 Research Results
@JulieEvans_PT @SpeakUpEd @asugsvsummit
What types of decisions are informed by data?
54% of school principals say regularly use data
26% say rarely
87% of school principals say regularly use data
8% say rarely

32. What do you think district administrators nationwide said?
What metric is most effective for measuring the impact or ROI of
your digital initiatives?
A. Feedback from teachers
B. Student achievement results
C. Students’ level of engagement in learning
D. Observations in the classroom
E. Feedback from students
@JulieEvans_PT @SpeakUpEd @asugsvsummit

33. What do you think district administrators nationwide said?
What metric is most effective for measuring the impact or ROI of
your digital initiatives?
A. Feedback from teachers (53%)
B. Student achievement results (60%)
C. Students’ level of engagement in learning (77%)
D. Observations in the classroom (69%)
E. Feedback from students (62%)
@JulieEvans_PT @SpeakUpEd @asugsvsummit

34. Insights from the Speak Up 2017 Research Results
@JulieEvans_PT @SpeakUpEd @asugsvsummit
What metrics are most effective for evaluating or measuring the
impact of technology initiatives?
Quantitative
1. Student achievement
results
Qualitative
1. Student engagement
2. Classroom observations
3. Student feedback
4. Teacher buy-in/feedback
5. Student project quality

35. Insights from the Speak Up 2017 Research Results
@JulieEvans_PT @SpeakUpEd @asugsvsummit
Why are the metrics mostly qualitative?
▪ Difficult to identify “impact” or “ROI”
▪ Difficult to isolate variables for evaluation quantitatively
▪ Lack of expertise in how to do this in a K-12 environment
▪ Digital learning space is still evolving and maturing
▪ Lack of knowledge or skill on how to use data or measure for impact
▪ Competing priorities

36. We are still determining
the right “lens” for
identifying and evaluating
digital products for
classroom usage

37. Learn more @ www.tomorrow.org
National Speak Up reports and infographics
Targeted and thematic reports
Digital learning trends
Community engagement
Mobile learning
Games in the classroom
Blended learning outcomes
Presentations, podcasts and webinars
Services: consulting, workshops, evaluation and efficacy studies
✓ We have expertise in the evaluation of mobile implementations, new
classroom models and digital content usage
New Speak Up 2017 reports to be released in Spring 2018

38. Insights from the Speak Up 2017 Research Results
So Many Products, So Much Data:
The Real Metrics that Educators Use to Identify and Evaluate
New Products for Classroom Usage
Dr. Julie A. Evans
CEO, Project Tomorrow
JEvans@tomorrow.org
@JulieEvans_PT
Thank you for joining me today!