The document discusses using data to assess educational technology initiatives and interventions. It provides a six-step process for conducting research: 1) ask a question, 2) define indicators, 3) identify a sample, 4) identify a method, 5) gather data, and 6) analyze results. Key recommendations include identifying clear learning goals connected to the technology, measuring progress towards those goals using random samples, and using assessment data to improve instruction.

1. Did we change?
Justin Reich
Doctoral Student, Harvard Graduate School of
Education
Co-Director, EdTechTeacher.org
@bjfr
Blog: edtechresearcher.org

5. Using Data to Address Instruction
Identify a
Problem of
Practice
Gather data
about the
problem
Examine
Instruction
Develop an
Action Plan
Act and
Assess
Further Reading: Data Wise and Data Wise in Action

6. The lower school at the American School of
Brussels is working on increasing
connections between home and school and
having parents more aware of and involved in
children’s classroom experiences.
They are requiring all elementary teachers to
create blogs to help build these connections.
How could you assess, short term and long
term, whether this initiative is working?
Scenario 1

8. How would you test French
Fry Quality at McDonalds?

9. The Fakename School District has a Wordpress
install for classroom blogging.
Teachers have been asked to consider Will
Richardson’s definition of
blogging, emphasizing student
autonomy, creativity, and attentiveness to a
real audience
Bud Hunt wanted to write his MA thesis about
Scenario 2

10. What did Bud do?
• He took a sample of all of his school’s blog
posts
• He read them
• He categorized them
• He drew some conclusions about the
kinds of learning taking place based on
that analysis

13. American School of Doha has a 6-12 one-to-
one program. They are implementing a 1-1
program in the fifth grade.
The tech directors want to assess the
efficacy of their intervention.
What questions do you have? What could
we do?
Scenario 3

14. What does
awesome look
like?

15. The Newmarket High School has just
purchased 300 iPads to allow them to pilot a 1
to 1 program in the ninth grade.
Teachers have been asked to create “21st
century learning environments,” and they can
use the iPads however they wish.
How could you assess whether this pilot
should expand to the entire high school?
Scenario 4

16. Using Data to Address Instruction
Identify a
Problem of
Practice
Gather data
about the
problem
Examine
Instruction
Develop an
Action Plan
Act and
Assess
Further Reading: Data Wise and Data Wise in Action

17. Six Steps for Research
1. Ask a Question
2. Define Indicators
3. Identify a Sample
4. Identify a Method
5. Gather Data
6. Analyze the Results

18. Ask a Question
Good questions ask about how technology
changes people’s experience of learning.
Two helpful prompts:
What is the learning goal before technology?
Is technology supposed to change the
learning goal?

19. Define Indicators
• Patrick Larkin said he wants students to
be more engaged. What are indicators of
engagement?
• Chris Lehmann said he wants students to
be able to ask good questions. What are
indicators that students ask good
questions?

20. Identify a Sample
• Whole Population
• Convenience Sample- whoever you can
get
• Purposive Sample- a few, well-chosen
people
• Random sample- often the best method

21. Identify a Method
• Online Survey
• Interview (phone or in person)
• Classroom observation (online or in person)
• Teaching Materials Review (online or in print)
• Looking at Student Work (online or in print)
• Web Analytics
• Test Score Analysis

22. Gather Data
• Structured Data Collection Allows for
Structured Data Analysis
– Ask all of your random sample of parents the
same four questions, and you can compare
their answers
• Unstructured Data Collection Allows for
Surprising Insight

23. Analyze the Results
• Carefully crunch the numbers OR
• Just look at stuff and see what you think
• How will the data make your school
better?

24. Rules
• Write your name on the back of your cards
• Gain points by presenting research plans
• Start Research with a Question Card
• All cards only go face up, cards only connect on lines
• Extend Research with Sample, sampling, method, and
other cards
• All current cardholders must agree to add a new card
• Cards only add constraints, anything else is allowed
• Call Justin when you present, total points times
multiplier is score

25. Presentations
• What is the question?
• What are the indicators?
• What methods will allow you to assess
those indicators?
• How will the data gathered help you
answer to the question?
• What could you do with your results to
improve the school?

26. Final Thoughts
• Identify learning goals connected to
technology investments and interventions
• Measure progress towards those goals
• Draw random samples rather than
assessing everyone
– (and actually make them random)
• Leverage partnerships with universities
and other organizations
• Use assessment data to reflect upon and
improve instruction

27. Visit us at EdTechTeacher.org!
justin@edtechteacher.org
tom@edtechteacher.org