Brown Bag presentation by Barry Fishman and Bill Penuel at Northwestern University School of Education and Social Policy on Design-Based Implementation Research (DBIR), presented on Thursday, May 23rd, 2013

1. Design-Based
Implementation Research:
Working in Partnership(s) to Transform the
Relationship of Research and Practice
Bill Penuel (@bpenuel)
University of Colorado Boulder
Barry Fishman (@barryfishman)
University of Michigan
May 23, 2013

2. Acknowledgments
• National Science Foundation 1054086
• Annie Allen, University of Colorado
Britte Haugan Cheng, SRI International
Nora Sabelli, SRI International
Andy Krumm, SRI International

3. Where We Begin
• Many promising educational interventions have been
developed, validated, tested... Then what?
• The majority fade away as funding ends or attention
turns elsewhere
• A few are sustained in (or near)
the contexts where they were
developed
• A very few are brought to scale and
are used across settings/contexts
• Why is the rate of “success” so low?

4. A Validity Problem
• Interventions are usually developed in
“hothouse” environments
• Researchers seek to reduce sources of
variation in evaluations to increase internal
validity
• Funded research focuses more on
developing and validating interventions from
basic research (Type I translation) than on
understanding or closing gaps between
research and practice (Type II translation)

5. Type I Translation
Type II
Translation
What is being
translated?
Translating principles
from basic learning
research into
interventions
Translating
interventions
developed for one or a
few settings into
interventions that are
scalable to many
settings
What kind of
research is
involved?
Design-based research
Efficacy and
effectiveness trials
Implementation
research
(for example, DBIR)

6. Type I Translation
Type II
Translation
What kinds of
questions does
translational
research answer?
What/how do people learn
from this design?
What do problems in
learning or implementation
suggest about redesign?
What kinds of capacities
are required to implement
this design?
What supports are needed
for people implementing
the design to adapt it in
ways congruent with the
design’s core principles?
Who is involved?
Learning scientists,
classroom teachers,
subject matter experts,
often also software
developers
Learning scientists,
organizational researchers,
teacher leaders, school and
district administrators,
often also publishers and
enterprise software
engineers

7. Design-Based Research
• The objective of DBR is to develop theory, not
just empirically tune “what works”
• “Design experiments create the conditions for
developing theories yet must place these
theories in harm’s way.”
• The theory is expected to do “real work in
practical educational contexts”
(Cobb, Confrey, diSessa, Lehrer, Schauble, 2003)

8. Design-Based
Implementation Research
• Blends learning sciences and policy
implementation research traditions and
methods
• Learning sciences: iterative, collaborative,
guided by and informing theories of
learning/teaching
• Policy implementation: focus on conditions for
implementation effectiveness, guided by and
informing theories of institutional change and
organizational learning. Focus on the design of
systems and infrastructure

9. DBIR Principles
1. A focus on jointly-defined problems of teaching and
learning practice
2. A commitment to iterative, collaborative design
3. A concern with developing knowledge and theory
through disciplined inquiry
4. A goal of developing capacity for sustaining change
in systems
Source: Penuel, W. R., Fishman, B., Cheng, B. H., & Sabelli, N. (2011).
Organizing research and development at the intersection of
learning, implementation, and design. Educational Researcher,
40(7), 331–337.

10. Example: The Center for Learning
Technologies in Urban Schools

11. Commitment to Iteration &
Collaboration
• All work unfolded over multiple phases of pilot testing
and revision, starting with a small core of teachers and
then expanding
• Design teams formed “work circles” that included
university and school personnel
• Project leadership worked to forge new relationships
within and across institutions
– E.g., Bringing curriculum and technology leadership
together in Detroit Public Schools
– E.g., Convening technology leadership across districts

12. Concern for Developing Knowledge
and Theory Through Inquiry
• The goal throughout was not only to create better
materials
• Goal was to develop new knowledge by developing
the theory of educative curriculum materials, with a
range of subgoals, e.g.:
– Study of teacher learning through professional
development that linked PD to student outcomes
– Research on cognition supported by technology
– Evolving theory of system capacity for supporting
reform, and how to design for systemic change

13. A Concern with Developing
Capacity to Sustain Change

14. Evidence Standards for DBIR

15. Design and
Development
Efficacy
Trials
Effectiveness
Studies
Involvement of R&D Team
Involvement of Evaluators & Practitioners

16. Design and
Development
Efficacy
Trials
Effectiveness
Studies
Involvement of R&D Team
Involvement of Evaluators & Practitioners
Sustainability?
+0.50
+0.00

17. Replication… Really?
• Organizational replication is often thought of
as a process that yields reliable results at the
expense of local and professional control
• Peurach and Glazer (2011) argue for a
“knowledge-based logic” in which program
developers collaborate with schools to
produce, use, improve, and retain practical
knowledge

18. Key Differences from (old) IES Model
• Teams plan for scaling and sustainability from
the start
• Designers maintain their involvement
throughout, because design is ongoing and
iterative
• Powerful innovations are not expected to
survive in the wild without changes at the
system level

19. Generating Forms of Evidence in an
Era of “Fast Science”
• Get it mostly right, fast.
– How do you know you “got it right”?
– What’s a “minimum viable” innovation to test?
• Fail early, fail often.
– How can you put your theory in harm’s way from the
start?
– What documentation can benefit your own and other
teams?
– When do you do the “big test”?

20. Matching Phase of Development
to Phase of Research
Phase of Development Driving Questions Sources of Evidence
Problem Negotiation What problem of
practice should be the
focus of our joint work?
Available data from
multiple sectors
Research evidence
Perspectives and values
of stakeholders
(including nonschool
actors)
Co-design What should be the
focus of our work?
To what extent do teams
leverage the diverse
expertise of
stakeholders?
Design Rationales
Ethnographic accounts
of design processes

21. Matching Phase of Development
to Phase of Research
Phase of Development Driving Questions Sources of Evidence
Early implementation How do implementers
adapt the innovation to
their local contexts?
How do implementers use
the innovation to
reconstruct their practice?
What are the appropriate
measures of impact?
Observations of
implementation
Interviews
Assessment design
Efficacy What is the potential
impact of the innovation on
teaching and learning?
What mediates impacts on
learning?
Randomized Controlled
Trials
Interrupted Time Series
Designs
Explanatory Case
Studies

22. Matching Phase of Development
to Phase of Research
Phase of Development Driving Questions Sources of Evidence
“Translation” (Type II) What supports are needed
to implement the program
effectively?
What are the conditions for
sustainability?
Experimental
comparisons of different
means of support
Explanatory comparative
case analysis

23. Challenges in DBIR
• Establishing shared relevance
• Balancing rewards across partners
• Collaborating across disciplines
• Synchronizing time tables and cycles
• Earning trust

24. Funders Interested in DBIR
• National Science Foundation
• US Department of Education
– RELs
– Research-practice partnerships
– Continuous Improvement Research in Education
• William T. Grant Foundation
– Use of research evidence RFP
– Funding of work in research-practice partnerships
(white paper, Coburn-Penuel study, Carnegie
study)

25. Coming Soon
Also: http://tinyurl.com/dbir-aera