1. Roundtable: Cognitive Mapping as a tool
for program and evaluation design
Joshua Penman, Carlos Echeverria-Estrada &
Agnieszka Rykaczewska
2. We acknowledge the invaluable support this team
received from CGU faculty, in particular from the
project P.I., Dr. Tarek Azzam, and the staff responsible
for administering the SEPA grant at City of Hope.
3. Evaluation of the San Gabriel Valley’s SEPAC (K-8
Research Education Program)
May 2012 – Start of San Gabriel
Valley Science Education Partnership
Award Collaborative (SGV SEPAC) –
2012/13
Summer 2012 – CGU was hired
2012/2015 – evaluation questions
and design, yearly data collection
and reporting separately by method.
2015/16 – Logic model and program
mapping (four components),
questions about use and learning to
CoH staff.
2016/17 – Use of logic model and
program mapping for new gran
design & application.
Current data collection
April 2017 - Grant ends.
4. Cognitive Mapping
• Spatial location of elements (i.e., concepts) in a network,
relevant for the interpretation of relationships between
concepts (Coronges et al., 2007; Huff, 1990).
• In a conceptual map, nodes represent the concepts in
the knowledge domain, and the strings represent the
links between these concepts (Curşeu, Schalk, & Schruijer, 2010).
7. So, why Cognitive Mapping?
• Traditional logic model did not capture
underlying mechanisms involved in the
program effectiveness
• Maps can be used as tools, as a means of
applying embodied theory to a particular
application (Conole, Dyke, Oliver & Seale, 2004)
• Enables reflection on implementation that
elevates the importance of feedback
Source: http://www.clipartkid.com/man-
with-question-mark-cliparts/
8. Cognitive Mapping: Using Century, Rudnick & Freeman’s
Fidelity of Implementation (FOI) 1
Basic FOI Framework:
Categories of Critical Components
Procedural Educative Pedagogical
Student
engagement
Structural Critical Components Instructional Critical Components
1 Century, J., Rudnick, M., & Freeman, C. (2010). A Framework for Measuring Fidelity of Implementation: A Foundation for Shared
Language and Accumulation of Knowledge. American Journal Of Evaluation, 31(2), 199-218.
14. Did it help?
To Program Staff:
• The process helped clarify outcomes and how they were
achieved.
• The logic model process map helped to explain “why” does the
evaluation feedback actually mattered.
• Reassured that program in practice was aligned with the original
goals of the Grant
• Helpful in identifying critical features of implementation for future
programming
15. To Evaluation Team:
• Assess what measures are appropriate for each element in the
logic model (process & outcomes chain)
• Identified alignment
• Identified gaps
• It helped improve observations, knowing the background of the
program items.
• This last summer observations: considerably less general and
more attention to interactions between students and
teachers, and how they were affecting each other.
16. To Other Audiences:
• This tool addresses mostly
the organizational side of
the program. It can be used
in different subject domains
and with different theoretical
frameworks (e.g., Six
Sigma).
Source: http://blog.mitchcommgroup.com/ann-marie-
satterfield/how-you-reach-a-diverse-audience
17. Table discussion
• Raise your hand if you have any
experience with using approaches like
this? What’s your experience?
19. Table discussion
• Raise your hand if you have…
• What potential benefits do you see?
• Pro’s & con’s?
20. Table discussion
• Raise your hand if you have…
• What potential benefits do you see?
• Pro’s & Con’s?
• What was the role of the stakeholder theory?
21. Table discussion
• Raise your hand if you have…
• What potential benefits do you see?
• Pro’s & Con’s?
• What was the role of the stakeholder theory?
• How would this work with another theoretical
framework other than Century?
22. References:
Century, J., Rudnick, M., & Freeman, C. (2010). A Framework for Measuring Fidelity of
Implementation: A Foundation for Shared Language and Accumulation of Knowledge.
American Journal Of Evaluation, 31(2), 199-218.
Conole, G., Dyke, M., Oliver, M., & Seale, J. (2004). Mapping Pedagogy and Tools for
Effective Learning Design. Computers And Education, 43(1-2), 17-33.
Curşeu, P.L., Schalk, R., & Schruijer, S. (2010). The Use of Cognitive Mapping in Eliciting
and Evaluating Group Cognitions. Journal Of Applied Social Psychology, 40(5), 1258-
1291.
***SHORTEN THE STORY***
The San Gabriel Valley Science Education Partnership Award Collaborative (SGV SEPAC) is a partnership between City of Hope (COH), a Comprehensive Cancer Center and biomedical research institute, and the Duarte Unified School District (DUSD), an 80% minority school district in the San Gabriel Valley of Los Angeles County.
Focus on 3rd aim: establishing a K-8 research education program.
These include visits to 2nd grade classrooms by COH scientists and field trip visits to COH by 2nd, 5th, and 8th grade students. All students in these grade levels are included in the year’s activities. Second grade students learn about genetic traits and inheritance and then visit COH to tour the grounds and engage in fun exploration activities. Fifth grade students visit COH labs and learn about fundamental concepts related to the human body. Eighth grade students do hands-on science centered on the topic of DNA. These activities are designed to whet their appetite for research and pique their interest in applying for the summer research program during their high school years.
Hodgkinson and Clarkson (2005) argued that the process of cognitive mapping consists of four steps: (a) knowledge elicitation; (b) construction; (c) analysis; and (d) aggregation or comparison.
In the knowledge-elicitation stage, the authors distinguished between ideographic and nomothetic approaches. In the ideographic approach, the researcher collects the concepts that are used by the individual or the group to describe a particular task or the way in which a particular task is (to be) performed. In nomothetic elicitation, the researcher provides predefined concepts based on theoretical models or own hypotheses.
Hodgkinson and Clarkson (2005) argued that the process of cognitive mapping consists of four steps: (a) knowledge elicitation; (b) construction; (c) analysis; and (d) aggregation or comparison.
In the knowledge-elicitation stage, the authors distinguished between ideographic and nomothetic approaches. In the ideographic approach, the researcher collects the concepts that are used by the individual or the group to describe a particular task or the way in which a particular task is (to be) performed. In nomothetic elicitation, the researcher provides predefined concepts based on theoretical models or own hypotheses.
The traditional LM was not powerful enough to invite to the reflection of why delving into the implementation is so important for program improvement.
Limited to descriptive elements of general trends, but skips the details of underlying processes to make the program successful.
Also, poor visualization of such processes did not invite to reflection on implementation.
Useful to evaluation team and CoH’s staff.
***NESZKA***
Looking for a framework that helped the evaluation unravel the processes involved in a successful implementation, the team unpackaged the Framework for Measuring Fidelity of Implementation by Century, Rudnick and Freeman.
-They follow the same colors
***JOSH***
***JOSH***
Logic model of the mapping process
6 Categories
32 Items
43 Arrows on this sheet.
It just looks like spaghetti: arrows crossing over each other are confusing. It looks like what most of our clients think about their program: “That everything links to everything else. Everything we do is important.”
PS: In this, I think I gave an extra link for each thing that the others linked to, so we could perform a rudimentary network analysis to see what the most important items were. But that wasn’t very helpful.
***JOSH***
6 categories
25 items
27 Arrows on this sheet.
6 attributes (the bullet points within items)
3 levels (items stacked on top of each other)
Two types of level:
Component (it can be divided up into these things)
Additive: it builds on top of another item.
In addition, we have two special use cases:
Linked: multiple items on the same level linked with + sign to indicate that they are expected to be synergistic.
Attributes (describe the thing or a component)
Why use Pipes instead of Arrows? - J: We’re not sure; I think we wanted to make them bigger and draw more attention to them so that the client is less focused on the program components, and more focused on how they connect to each other. The really important thing, the “why” for having any given component is because of what it connects to.
Also, we think it looks a little bit like an old pipe game that used to
***NESZKA***
To Program Staff:
- They could say that “teachers weren’t effective” year after year; but how did teachers do that; . . . such a critical piece for effectiveness and impact. - They accepted it; they realized that in the next program they’re going to have to track implementation more carefully.
The traditional logic model addresses how things are supposed to happen, but doesn’t address “why” nearly as much.
In the case of the San Gabriel Valle SEPAC, process mapping captured the underlying mechanism of the interventions. This visual representation showed the program was still pursuing the original purposes, even after a significant staff turnovers during the course of the grant.
To Program Staff:
- They could say that “teachers weren’t effective” year after year; but how did teachers do that; . . . such a critical piece for effectiveness and impact. - They accepted it; they realized that in the next program they’re going to have to track implementation more carefully.
The traditional logic model addresses how things are supposed to happen, but doesn’t address “why” nearly as much.
In the case of the San Gabriel Valle SEPAC, process mapping captured the underlying mechanism of the interventions. This visual representation showed the program was still pursuing the original purposes, even after a significant staff turnovers during the course of the grant.
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
