This presentation shares how and why outcome mapping processes and principles enriched an internal self-evaluation process that was grounded in participatory action research methods toward deliberative democratic curriculum evaluation.

1. Outcome
Mapping for
Planning
Evaluations in
American K–12
Urban
Education:
Potent
Possibilities
American Evaluation Association
2015 Conference
Author: Tabia Lee, Ed.D.
© 2015, e-mail for permissions
drtlee11@gmail.com +1 916-588-7776

2. Schools as
Complex Adaptive
Social-Ecological
Systems The
Evaluation
Process
Teachers as
Curriculum
Evaluators
Outcome Mapping &
Internal Self-
Evaluation Planning
Potent
Possibilities
Presentation Roadmap
I.
II.
IV.
III.
V.

4. Locating Evaluation Studies & Action Research
Usefulness
in facilitating
decision making &
transformations
Discovery
or validation of
generalizations
Hypothesis-
driven &
product-focused
Decision-
driven &
process-focused
Goals&Aims
Drivers & Focus
EVALUATION STUDIES
& ACTION RESEARCH
GENERAL
EDUCATIONAL
RESEARCH

5. Schools and Wicked Problems
Adapted from Rittel & Webber, 1973

6. Continuous,
Recursive, &
Divergent
Unpredictable
Outcomes
Sociopolitical
Collaborative
Teaching
&
Learning
EmergentCreates
Reality
Source: Guba &
Lincoln, 1989
The
Evaluation
Process

7. development
accountability
inside outside
internal
evaluation
external
evaluation
bottom-uptop-down
self-
evaluation
others
The Evaluation Cube
Source: Schratz, 1999

8. Senge’s 5
Disciplines
& School
Change
Personal
Mastery
Systems
Thinking
Mental
Models
Team
Learning
Shared
Vision
Source: Cropper, 2003

9. Linear Perspective Nonlinear Perspective
Metaphor Machine Living System
Part & Whole
Relationship
 Hierarchy
 Sum of the Parts
 Fractal Geometry
 Emergence
Understanding
of Schools
Formal
Organizations
Learning
Organizations/
Communities
Cause & Effect
One-Way Causality
(Open-Loop)
Seeking Stability
 Webs of Reciprocal
Causality (Closed-Loop)
 Dynamic Equilibrium
Locus of Change External External & Internal
Schools as Social Systems

10. The Ecology of Educational Environments
Macro-
system
Exo-
system
Meso-
system
Micro-
system
Source:
Bronfenbrenner, 1976

11. Knowledge Cultures
as a Nested System Individual
Knowledge
Local
Community
Knowledge
Specialized
Knowledge
Organizational
Knowledge
Holistic
Knowledge
KEY
Collective
Thinking,
Learning,
& Inquiry
is
All the decision-making
knowledges
generating synergy

12. Simple
Complex Complicated
Chaotic


A B
A B
A
B
C
D
E
A B

13. HABITUAL
PRACTICES
PERCEIVED
REALITY
CORE
BELIEFS
ASSUMPTIONS
(examined and unexamined)
HABITUAL
PRACTICES
REALITY
CORE
BELIEFS
SYSTEMS
THINKING
Versus
Incompetent
Systems
Competent
Systems

14. Schools as Complex Adaptive Systems
Generate
System-wide
Patterns
Agents
Interact Source: Patterson,
Holladay, & Eoyang, 2012

15. Schools as Complex Adaptive
Social-Ecological Systems (CASS)
Schools
as
CASS
Diverse &
Dynamic Agents
Non-Linear &
Unpredictable
Change & Co-
Evolution Rules
Flow of
Information
Feedback Loops
Nested
Systems
Structure
EmergenceInteraction
Dependent
but
Autonomous
Source:
Keshavarz,
Nutbeam,
Rowling, &
Khavarpour,
2010

16. Teachers as Curriculum Evaluators
Improvement
of Teaching
Improvement
of Learning
Political &
Social
Accountability
Professional &
Organizational
Development
Significant and Lasting Change is Teacher-Led

17. Inquiry
Design
Framework
Collective
Learning Spiral
(P4D4)
Participatory
Action
Research Outcome
Mapping
Deliberative
Democratic
Curriculum
Evaluation

18. Participatory
Action
Research
Participatory
Practical
&
Collaborative
Critical
Transforms
Theory &
Practice
Reflexive
Emancipatory
Social
Process
Documents the
Change Process

19. Outcome
Mapping
Principles
Actor-Centered
Development
&
Behavior
Change
Non-Linearity
&
Contribution
Participation
&
Accountability
Continuous
Learning &
Flexibility
Source: Jones &
Hearn, 2009

20. Collective
Learning
Spiral
(P4D4)

21. Curriculum Evaluation Approaches
Source: MacDonald, 1976
Bureaucratic Autocratic Democratic

22. Democratic Evaluation Orientations
Discursive
Sampwith the people
le text
1
3
2
Dialogue
Deliberation
Inclusion
1
2
3
1
1
2
2 3
3
Adapted from: Hanberger, 2004
Less More

23. 0.
Situation
Analysis
1.
Intentional
Design
2.
Evaluation
Planning
Develop
START Reports
Reports
Reports
3.
Outcome &
Performance
Monitoring
GO TO
START
Collective
Inquiry
Flow

24. Cycle 0: Situation Analysis
Literature
&
Document
Review
Work
Teams
Surveys
&
Classroom
Observations
Boundary Partner,
Influence, & SWOT
Analyses
Interviews &
conversations
Observe
the Landscape
Analyze
the Patterns

25. Schools &
Systems
Thinking Student-
Centered
Learning
Constructivist
& Inquiry-
Based
Varied
Measures of
Achievement
Commitment
to Life-Long
Learning
A Whole-
Community
Approach
Culture of
Teaming
Indicators
of
Readiness
Source: Benson,
LaVigne, Marlin, &
Yates, 2010

26. Influence Analysis
High
Low
Low High
LevelofPower
Level of Interest
Mixed Blessing Supportive
Marginal Non-Supportive
Monitor Defend Against
InvolveCollaborate With
KEY
Teacher
Admin
Support
Staff
Solid
Flexible
Not
Movable
T
A
SS
A
1
A
3
A
2
A
4
SS
1T
6
T
5
T
3
T
2
T
4
T
1
T
7
T
8
T
9
T
10
SS
2
T
11

27. Cycle 1: Intentional Design
Outcome
Challenges Progress
Markers
Organizational
Practices
Strategy
Maps
Vision & Mission
Collective
Visioning
Collective
Action
Monitoring
Priorities

28. Vision &
Mission
Teachers critically examine the civic curriculum and collaborate
to support the development of students’ civic competence.
Increased Civic Competency
Whole-
School
Effort
Support
Networks
&
Processes
Civic
Education
Resources
Informed
Teachers
Transdisciplinary
teacher teams promote
local and global civic competence using
authentic learning activities.

29. Mapping the Strategy
Individual/Collective
STRATEGIES
Environmental
STRATEGIES
Establish Newsletter and Good Citizen of the
Month Billboard (I-2)
Conduct Student and Teacher Surveys and
Interviews (I-1)
Organize School-wide Civic Activities (I-2)
Report
Civic
Findings
& News
to
Faculty
(E-2)
Implement
Observ-
ation
Teams
(E-3)
Establish
Critical
Friends
Groups &
Processes
(E-3)
Develop
Observ-
ation
Criteria
(E-1)
Facilitate Mock Election Workshops and Training (I-2)
Distribute
Civic
Lesson
Tips,
Plans, &
Materials
(E-2)

30. Cycle 2: Evaluation Planning
Determine
Evaluation
Questions
Establish
Timelines
Theory of Change
Chart the
Pathways
Plan
the Route
Identify
Information
Sources

31. Theory of Change Excerpt
The Project intends to see Social Studies
Teachers (SST) who recognize the importance
of increasing students’ civic competency
Teachers commit to action for increased student civic competency
Surveys Interviews
Report results
in workshops
80% SST participate
SST reflect
& engage in
generative
dialogue
SST
collab-
orate
on civic
curri-
culum
action
plans
Assumption
Goal
Outcome
Challenge
Strategy
Expect to See
Like to See
Love to See
KEY
“Hard” and “soft” data
inform about civic
competence levels,
curricular strengths,
and challenges
Increased
ownership &
responsibility

32. Cycle 3: Monitoring & Evaluation
Set
Monitoring
Priorities
Performance
Journal
Strategy
Journal
Analyze Cycle
0, 1, & 2 Data
Focus the
Monitoring
Document
the Changes
Outcome
Journals

33. Documenting Teacher Change
20% regularly facilitated
learning opportunities related
to civic competency
61% reported students
never researched local
community problems in class
44% reported they never
discussed civic responsibility
with students
72% regularly facilitated
learning opportunities related
to civic competency
30% reported students
never researched local
community problems in class
32% reported they never
discussed civic responsibility
with students
Prior to Project Two Years Into Project

34. Documenting Student Change
11% of students reported
that civic competency was
regularly addressed in Social
Studies class
62% reported they never
talked about the importance
of voting in Social Studies class
20% were able to correctly
identify the current mayor
Prior to Project Two Years Into Project
55% of students reported
that civic competency was
regularly addressed in Social
Studies class
16% reported they never
talked about the importance
of voting in Social Studies class
66% were able to correctly
identify the current mayor

35. Lessons Learned
More extensively targeting
all subject-matter teachers
More deeply engaging
Administrators
Building networks with
other area schools
Intensified whole-school
focus on civic competence
A network of educators
committed to civic education
that is less susceptible to
being dismantled

36. Development Empowerment
Potent Possibilities

37. References
Aslin, H. J. & Brown, V. A. (2004). Towards whole of community engagement: A practical toolkit (MDBC Publication No. 14/04). Canberra City,
Australian Capital Territory: Murray-Darling Basin Commission.
Benson, T., LaVigne, A., Marlin, S., & Yates, J. (2010). Indicators of site readiness for innovation: Systems thinking in schools. Retrieved from
http://watersfoundation.org/wp-content/uploads/2013/08/Indicators-of-site-readiness.pdf
Brodnick, R. J., Jr. (2000). Conceptualizing social systems: A critical argument for the nonlinear perspective (Doctoral dissertation). Available from
ProQuest Dissertations and Theses database. (UMI No. 9965960)
Bronfenbrenner, U. (1976). The experimental ecology of education. Paper presented at American Educational Research Association Annual
Meeting, San Francisco, CA.
Brown, V. A. (2001). Planners and the planet. Australian Planner, 38(2), 67–73. doi:10.1080/07293682.2001.9657941
Brown, V. A. (2004). The more we are together... Collaborative decision making, social planning, and sustainability. Australian Planner, 41(3), 42–
45. doi:10.1080/07293682.2004.9982371
Brown, V. A., & Lambert, J. (2013). Collective learning for transformational change: A guide to collaborative action. New York, NY: Routledge.
Clarke, A. E. (2005). Situational analysis: Grounded theory after the postmodern turn. Thousand Oaks, CA: Sage Publications.
Cropper, B. (2003). Fact-file 2: Unpacking the 5 disciplines... Becoming a learning organisation. Retrieved from
http://www.thechangeforum.com/factfiles/FactFile_2-The_5_Disciplines.pdf
Crosby, D. A. (2014). In Imago Naturae: The ultimacy of nature as a closing of the gaps. In B. Donaldson (Ed.), Beyond the bifurcation of nature: A
common world for animals and the environment (pp. 180–191). Newcastle upon Tyne, UK: Cambridge Scholars Publishing.
de Guerre, D. W. (2001). Doing action research in one's own organization: An ongoing conversation over time. Systemic Practice and Action
Research, 15(4), 331–349. doi:10.1023/A:1016348421584
Dooley, K. J. (1996). A nominal definition of complex adaptive systems. The Chaos Network, 8(1), 2–3.
Dooley, K. J. (1997). A complex adaptive systems model of organizational change. Nonlinear Dynamics, Psychology, and Life Sciences, (1)1, 69–97.
doi:10.1023/A:1022375910940
Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping: Building learning and reflection into development programs. Ottawa, Ontario,
Canada: International Development Research Centre.
Eguren, I. R. (2011). Theory of change: A thinking and action approach to navigate in the complexity of social change processes. The Hague, The
Netherlands: Humanist Institute for Development Cooperation (Hivos).
Eoyang, G. H. (2001). Conditions for self-organizing in human systems (Doctoral dissertation). Available from ProQuest Dissertations and Theses
database. (UMI No. 3040770)

38. References
Forrester, J. W. (1995). Counterintuitive behaviors of social systems. [based on October 7, 1970 testimony for the Subcommittee on Urban
Growth of the Committee on Banking and Currency, U.S. House of Representatives]. Retrieved from
http://static.clexchange.org/ftp/documents/roadmaps/RM1/D-4468-2.pdf (Original work published 1971)
Giancola, J. M., & Hutchison, J. K. (2005). Transforming the culture of school leadership: Humanizing our practice. Thousand Oaks, CA: Corwin
Press.
Greenwood, D. J., Whyte, W. F., Harkavy, I. (1993). Participatory action research as a process and a goal. Human Relations, 46(2), 175–192.
doi:10.1177/001872679304600203
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Thousand Oaks, CA: Sage Publications.
Hanberger, A. Democratic governance and evaluation. Paper presented at the Sixth European Evaluation Society Conference, Berlin, Germany.
Retrieved from http://www.edusci.umu.se/digitalAssets/66/66094_hanbergergovernance04.pdf
Henderson, J. G., & Gornick, R. (2007). Transformative curriculum leadership (3rd ed.). Upper Saddle River, NJ: Pearson Education.
Jones, H., & Hearn, S. (2009, October). Outcome mapping: A realistic alternative for planning monitoring, and evaluation. Overseas
Development Institute Background Note. Retrieved from http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-
files/5058.pdf
Kemmis, S., & McTaggart, R. (2005). Participatory action research: Communicative action and the public sphere. In N. K. Denzin & Y. S. Lincoln
(Eds.), The SAGE handbook of qualitative research (3rd ed., 559–604). Thousand Oaks, CA: Sage Publications.
Keshavarz, N., Nutbeam, D., Rowling, L., & Khavarpour, F. (2010). Schools as social complex adaptive systems: A new way to understand the
challenges of introducing the health promoting schools concept. Social Science and Medicine, 70(10), 1467–1474.
doi:10.1016/j.socscimed.2010.01.034
Krathwohl, D. R. (1998). Methods of educational and social science research: An integrated approach (2nd ed.). Long Grove, IL: Waveland Press.
Kurtz, C. F., & Snowden, D. J. (2003). The new dynamics of strategy: Sense-making in a complex and complicated world. IBM Systems Journal,
42(3), 462–483. doi:10.1147/sj.423.0462
MacBeath, J. (1999). Schools must speak for themselves: The case for school self-evaluation. New York, NY: Routledge.
MacDonald, B. (1976). Evaluation and the control of education. In D. Tawney (Ed.), Curriculum evaluation today: Trends and implications (pp.
125–136). London, UK: Macmillan.
Mendelow, A. L. (1981). Environmental scanning—The impact of the stakeholder concept. Proceedings of the International Conference on
Information Systems, USA, 2, 407–418. Retrieved from http://aisel.aisnet.org/icis1981/20
Merry, U. (1995). Coping with uncertainty: Insights from the new sciences of chaos, self-organization, and complexity. Westport, CT: Praeger.

39. Mertens, D.M. (2005). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed
methods (2nd ed.). Thousand Oaks, CA: Sage Publications.
Morgan, G. (2006). Images of organization (Updated ed.). Thousand Oaks, CA: Sage Publications.
Organizational Research Services. (2004). Theory of change: A practical tool for action, results and learning. Retrieved from
http://www.aecf.org/m/resourcedoc/aecf-theoryofchange-2004.pdf
Patterson, L., Holladay, R., & Eoyang, G. (2012). Radical rules for schools: Adaptive action for complex change. Circle Pines, MN: Human Systems
Dynamics Institute.
Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage Publications.
Richmond , B. (2004). An introduction to systems thinking. Lebanon, NH: isee systems.
Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. doi:10.1007/BF01405730
Roduner, D., & Hartmann, O. (2009, October). Module on step-0: How to prepare for OM Intentional Design. Outcome Mapping Ideas
(Paper No. 3). Retrieved from http://www.outcomemapping.ca/download/simonhearn_en_OMidea3.pdf
Savage, G. T., Nix, T. W., Whitehead, C. J., & Blair, J. D. (1991). Strategies for assessing and managing organizational stakeholders. Academy of
Management Executive, 5(2), 61–75. doi:10.5465/ame.1991.4274682
Schön, D. A. (1995). Knowing-in-action: The new scholarship requires a new epistemology. Change, 27(6), 26–34.
Senge, P. M. (2006). The fifth discipline: The art and practice of the learning organization (Revised ed.). New York, NY: Doubleday.
Sergiovanni, T. J. (1994). Building community in schools. San Francisco, CA: Jossey-Bass.
Shapiro, I. (2005). Theories of change. Beyond Intractibility (G. Burgess & H. Burgess, Eds.). Conflict Information Consortium, University of
Colorado, Boulder. Retrieved from http://www.beyondintractability.org/essay/theories-of-change
Themessl-Huber, M. T., & Grutsch, M. A. (2003). The shifting locus of control in participatory evaluations. Evaluation, 9(1), 92–111. doi:
10.1177/1356389003009001006
Wadsworth, Y. (1998). What is Participatory Action Research? Action Research International, Paper 2. Retrieved from:
http://www.scu.edu.au/schools/gcm/ar/ari/p-ywadsworth98.html
Watkins, A., & Wilber, K. (2015). Wicked and wise: How to solve the world’s toughest problems. Croydon, UK: Urbane Publications.
Zmuda, A., Kuklis, R., & Kline, E. (2004). Transforming schools: Creating a culture of continuous improvement. Alexandria, VA: Association for
Supervision and Curriculum Development.
References

40. Visual SourcesSlide 1: Cartoon evaluation image from Scheers & Wilson-Grau, 2008, GAN-Net Impact Community of Practice Meeting Presentation
Slide 3: Project Information graphic template from www.powerframeworks.com
Slide 4: Schools and Wicked Problems adapted from Rittel & Webber (1973); graphic template from www.SlideHunter.com
Slide 5: Locating Evaluation and Action Research adapted from Krathwohl (1998 )using positional mapping as described by Clarke (2005)
Slide 6: The Evaluation Process was adapted from Guba & Lincoln (1989) graphic template remixed from www.fppt.com
Slide 7: The Evaluation Cube was adapted from Schratz (1999) as cited in Themessl-Huber & Grutsch (2003) and MacBeath (1999)
Slide 9: Schools as Social Systems adapted from Brodnick, 2000, p. 120
Slide 11: Knowledge Cultures as a Nested System visual representation remixed from: solvingforpattern.org; Material from Valerie A. Brown
Slide 12: Cynefin Framework adapted from Kurtz & Snowden, 2004
Slide 13: Competent System Vs. Incompetent System from Zmuda, Kuklis, & Kline, 2004, p. 43
Slide 14: Complex Adaptive Systems quasicrystal graphic from http://wallpapers24k.blogspot.com/2012_05_01_archive.html networked agents
image from http://necsi.edu/research/networks/man53/man53.png
Slide 15: Schools as CASS graphic template remixed from www.fppt.com
Slide 16: Teachers as Curriculum Evaluators is adapted from from MacBeath (1999); graphic template from www.showeet.com
Slide 17: Inquiry Design Framework graphic template from www.SlideHunter.com
Slide 18: Participatory Action Research graphic template from www.showeet.com
Slide 20: Collective Learning Spiral graphic template remixed from www.slideteam.net; material from Aslin & Brown, 2004; Brown (2001, 2004);
Brown & Lambert (2013).
Slide 22: Democratic Evaluation Orientations graphic template remixed from www.showeet.com. Information adapted from Hanberger (2004)
Slide 23: Collective Inquiry Flow circular flow of process graphic template from www.slideteam.net
Slide 24: Cycle infinity ribbon graphic templates from www.powerframeworks.com
Slide 25: Schools and Systems Thinking graphic template from www.fppt.com
Slide 26: The Influence analysis representation fuses the insights of Mendelow (1981) and Savage, Nix, Whitehead, & Blair (1991), as a visual
representation of the positioning of the Project’s boundary partners.
Slide 28: Vision and Mission graphic template from www.fppt.com
Slide 29: Mapping the Strategy graphic template from www.SlideHunter.com
Slide 33: Documenting Teacher Change graphic template from www.slidehunter.com
Slide 35: Lessons Learned graphic template from www.powerframeworks.com
Slide 36: Teachers as curriculum evaluators handshake remixed from https://commons.wikimedia.org/wiki/File:Handshake2.svg ; Vertical Spiral
image from http://i2.wp.com/www.makeyourbestself.com/wp-content/uploads/2015/04/upwardspiral.jpg?resize=699%2C649

41. Outcome Mapping for Planning Evaluations in
American K–12 Urban Education: Potent
Possibilities by Tabia Lee is licensed under
a Creative Commons Attribution-NonCommercial
4.0 International License.
Please contact for permissions:
Tabia Lee, Ed.D.
916-588-7776, drtlee11@gmail.com