1.
1
DEMYSTIFYING EVIDENCE IN EDTECH
Molly B. Zielezinski PhD
November 2019

2.
2
ABOUT MBZ LABS
Ed Tech Industry
Education Research
Teaching Practice

3.
3
National Context
The Every Student Succeeds
Act (ESSA) requires states to
document and enforce
accountability plans
including 5 school level
indicators.
SHIFTING CONTEXT
Why care about evidence?
Local Context
In accordance with ESSA,
evidence based programs
can be purchased with
government funds. Also,
EdTech can offer solutions for
capturing evidence of school
level indicators.

4.
4
ESSENTIAL QUESTIONS
What does it claim to do?
What are the core features of the
product? What outcomes is the
company promising?
Does it work?
Does engagement with the product
lead to the expected outcomes?
For whom?
With which populations can you
expect the relevant outcomes?
How do you know?
What can the company offer in
terms of documentation of the
expected outcomes in prior
contexts?
What to ask when vetting EdTech

5.
5
TYPES OF EVIDENCE
FOUNDATIONAL FORMATIVE SUMMATIVE
• Literature Review
• Logic Model
• Feasibility Study
• User Surveys
• Focus Groups
• Interview Series
• Usability Study/Pilot
• Case Study
• Implementation Study
• Correlational Study
• Pre-post Study
• Quasi-experimental Study
• Randomized Control Trial

6.
6
FOUNDATIONAL FORMATIVE SUMMATIVE
Literature Review
Logic Model
Feasibility Study
Focus Groups
User Interview Series
Usability Study/Pilot
Case Study
Implementation Study
Correlational Study
Pre-post Study
Quasi-experimental Study
Randomized Control Trial
TYPES OF EVIDENCE
User Surveys

7.
7
LEVELS OF EVIDENCE
Source: Pearson

8.
88
COLLECTING
EVIDENCE
1 Company Website
2 Ask Company
3 Check References
4 Digital Promise or CASEL Badge
5 ISTE EdTech Advisor
6 EvidenceforESSA.org
Where to look

9.
99
WEIGHING THE EVIDENCE
1 How many sources of evidence are there?
What types?
2 How recently was the evidence collected?
Has the product changed since then?
3 What is the highest level of rigor in the evidence base?
Does the context match yours? Does it qualify for ESSA funding?

10.
10
GOVERNMENT FUNDING
TITLE II
Focus: Teacher & principal training & recruiting
Who: States and districts
What: Face to face PD about digital tools, TOSAs,
online PD, systems for data collection &
management
TITLE I
Focus: Support for low income students
Who: LEAs & Districts focused on lowest
achieving students
What: Technology capacity, infrastructure,
and content; Support evidence gathering and
analysis
TITLE IV
Focus: Student support & academic enrichment
(SSAE)
Who: States and districts
What: Provide access to help educators
discover, use, & share digital content, tools that
support external collaboration; build capacity &
infrastructure
TITLE III
Focus: Support for ELLs
Who: States and districts
What: Supplemental PD focused on improving
instruction for ELLS that uses digital or online methods;
digital materials, curricula and programs
SOURCES
OF EDTECH
FUNDING

11.
11
LINKS TO LEARN MORE
• Title I and School Improvement Grants
• Titles II & III
• Title IV
• Government Summary of Funding Opportunities
• What to look for beyond RCTs
Email molly@mbzlabs for a copy of this presentation or follow @mbzlabs on twitter
ADDITIONAL FUNDING SOURCES
• School Improvement Grants
• ERATE
• Rural and Low-Income Schools Program

12.
12
MAKING IT WORK FOR YOU
Leverage Government
Funding
Try Before You Buy
Build Efficacy into the
Purchase
Monitor Effectiveness
Early and Often

13.
1313
Questions?
Molly@mbzlabs.com
@MBZ Labs

14.
14
SUPPLEMENTARY SLIDES

15.
15
FOUNDATIONAL FORMATIVE SUMMATIVE
Literature Review
Logic Model
Feasibility Study
Focus Groups
User Interview Series
Usability Study/Pilot
Case Study
Implementation Study
Correlational Study
Pre-post Study
Quasi-experimental Study
Randomized Control Trial
TYPES OF EVIDENCE
User Surveys