This document discusses using cognitive mapping as an evaluation tool for an education program. Cognitive mapping creates a visual representation of the relationships between key concepts in a program. The evaluation team used cognitive mapping to create a logic model that mapped the underlying mechanisms of a science education program in more detail than a traditional logic model. Stakeholder feedback found the cognitive map clarified outcomes and how they were achieved. It also helped identify critical implementation features and alignment between the program goals and evaluation. The cognitive mapping approach incorporates theoretical frameworks like fidelity of implementation and can be adapted to other subject domains or theories.
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...SarahG_SS
Presentation at the OE Global Conference held in Banff, Canada in April 2015. This presentation introduces the development of the evaluation strategy, using Utilization Focused Evaluation (UFE), for the ROER4D Project (http://roer4d.org/).
You can access the abstract here: http://conference.oeconsortium.org/2015/presentation/developing-an-evaluation-strategy-to-gain-insights-into-a-multi-national-project-roer4d/
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...SarahG_SS
Presentation at the OE Global Conference held in Banff, Canada in April 2015. This presentation introduces the development of the evaluation strategy, using Utilization Focused Evaluation (UFE), for the ROER4D Project (http://roer4d.org/).
You can access the abstract here: http://conference.oeconsortium.org/2015/presentation/developing-an-evaluation-strategy-to-gain-insights-into-a-multi-national-project-roer4d/
Identifying and changing key curriculum design practicesJisc
Examining the process of how institutions identify and then seek to change the curriculum design processes and practices. (This session complements the main conference session on curriculum design).
Jisc conference 2011
Seda small grants final report 2013 powell varga atkins uo l v1-1sarra saffron powell
Report on academic staff's perspectives on and practices in digital literacies at the University of Liverpool. We examined the relationship between attitudes towards digital literacies ad critical reflection in discipline specific contexts. Includes a workshop for educational practitioners to try and supporting resources.
EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...EUmoocs
This hands-on workshop will work with learning design tools and with massive open online courses (MOOCs) on the FutureLearn platform to explore how learning design can be used to influence the choice and design of learning analytics. This workshop will be of interest to people who are involved in the design or presentation of online courses, and to those who want to find out more about learning design, learning analytics or MOOCs. Participants will find it helpful to have registered for FutureLearn and explored the platform for a short time in advance of the workshop.
This presentation was given during the EMMA Summer School, that took place in Ischia (Italy) on 4-11 July 2015.
More info on the website: http://project.europeanmoocs.eu/project/get-involved/summer-school/
Follow our MOOCs: http://platform.europeanmoocs.eu/MOOCs
Design and deliver your MOOC with EMMA: http://project.europeanmoocs.eu/project/get-involved/become-an-emma-mooc-provider/
Tools May 2015 Logic models for prog.docxturveycharlyn
Tools
May 2015
Logic models for program
design, implementation, and
evaluation: Workshop toolkit
Karen Shakman
Sheila M. Rodriguez
Education Development Center, Inc.
Overview
This Logic Model Workshop Toolkit is designed to
help practitioners learn the overall purpose of a logic
model, the different elements of a logic model, and
the appropriate steps for developing and using a logic
model for program evaluation. This toolkit includes a
facilitator workbook, a participant workbook, and a
slide deck.
At Education Development Center, Inc.
U . S . D e p a r t m e n t o f E d u c a t i o n
U.S. Department of Education
Arne Duncan, Secretary
Institute of Education Sciences
Sue Betka, Acting Director
National Center for Education Evaluation and Regional Assistance
Ruth Curran Neild, Commissioner
Joy Lesnick, Associate Commissioner
Amy Johnson, Action Editor
Chris Boccanfuso, Project Officer
REL 2015–057
The National Center for Education Evaluation and Regional Assistance (NCEE) conducts
unbiased large-scale evaluations of education programs and practices supported by federal
funds; provides research-based technical assistance to educators and policymakers; and
supports the synthesis and the widespread dissemination of the results of research and
evaluation throughout the United States.
May 2015
This report was prepared for the Institute of Education Sciences (IES) under Contract
ED-IES-12-C-0009 by Regional Educational Laboratory Northeast & Islands administered
by Education Development Center, Inc. The content of the publication does not neces
sarily reflect the views or policies of IES or the U.S. Department of Education nor does
mention of trade names, commercial products, or organizations imply endorsement by the
U.S. Government.
This REL report is in the public domain. While permission to reprint this publication is
not necessary, it should be cited as:
Shakman, K., & Rodriguez, S. M. (2015). Logic models for program design, implementation,
and evaluation: Workshop toolkit (REL 2015–057). Washington, DC: U.S. Department of
Education, Institute of Education Sciences, National Center for Education Evaluation and
Regional Assistance, Regional Educational Laboratory Northeast & Islands. Retrieved
from http://ies.ed.gov/ncee/edlabs.
This report is available on the Regional Educational Laboratory website at http://ies.ed.gov/
ncee/edlabs.
http://ies.ed.gov/ncee/edlabs
http://ies.ed.gov/ncee/edlabs
http://ies.ed.gov/ncee/edlabs
Contents
Facilitator workbook
Introduction to workshop 3
Session I. Learning about logic models (2 hours) 5
Session II. From logic models to program and policy evaluation (1.5 hours) 30
Appendix A. Simple logic model A-1
Appendix B. Template for simple logic model B-1
Appendix C. Sample logic model for college readiness program C-1
Appendix D. Example of a logic model for an educator evaluation system theory of acti ...
Skills & ideas for #ProblemGamblingKTE Anne Bergen
Skills & ideas for #ProblemGamblingKTE. (2104). Part of the "Moving Research Forward" Workshop Series for the Ontario Problem Gambling Research Centre.
An institutional perspective on analytics that focusses on a particular tool developed using an agile methodology to visualise learner behaviours in MOOCs via Sankey diagrams.
Assessing the Implementation of Authentic, Client-Facing Student Projects in ...craigmmacdonald
User Experience (UX) is often cited as one of the fastest growing occupations, creating opportunities across nearly every sector for individuals skilled in the application of user-centered design principles and methods. Many Information and Library Schools have responded to this demand by introducing more UX coursework into their curriculum, but the proliferation of agile software development and lean product design has incentivized organizations to look for experienced individuals for UX roles, even those that are considered entry-level. As a result, aspiring information professionals face a paradoxical situation in which they are required to have UX experience before they can gain UX experience. This article provides an assessment of one institution's efforts to overcome this experience gap by offering opportunities for students to participate in three types of authentic client-facing UX projects. Through surveys of students and clients served over four academic years, we provide a set of lessons learned and recommended best practices for incorporating project-based learning opportunities into UX courses.
Models of curriculum evaluation and application in educationalKoledafe Olawale
Curriculum can be defined as the planned and guided learning experiences and intended learning outcomes, formulated through the systematic reconstruction of knowledge and experiences, under the auspices of the school, for the learners’ continuous and willful growth in personal social competence (Tanner & Tanner, 1975)
More Related Content
Similar to Evaluation2016_Session2667_Mapping_Programs_Roundtable_10272016_draft3_AEAtemplate_Tarek (1) (1)
Identifying and changing key curriculum design practicesJisc
Examining the process of how institutions identify and then seek to change the curriculum design processes and practices. (This session complements the main conference session on curriculum design).
Jisc conference 2011
Seda small grants final report 2013 powell varga atkins uo l v1-1sarra saffron powell
Report on academic staff's perspectives on and practices in digital literacies at the University of Liverpool. We examined the relationship between attitudes towards digital literacies ad critical reflection in discipline specific contexts. Includes a workshop for educational practitioners to try and supporting resources.
EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...EUmoocs
This hands-on workshop will work with learning design tools and with massive open online courses (MOOCs) on the FutureLearn platform to explore how learning design can be used to influence the choice and design of learning analytics. This workshop will be of interest to people who are involved in the design or presentation of online courses, and to those who want to find out more about learning design, learning analytics or MOOCs. Participants will find it helpful to have registered for FutureLearn and explored the platform for a short time in advance of the workshop.
This presentation was given during the EMMA Summer School, that took place in Ischia (Italy) on 4-11 July 2015.
More info on the website: http://project.europeanmoocs.eu/project/get-involved/summer-school/
Follow our MOOCs: http://platform.europeanmoocs.eu/MOOCs
Design and deliver your MOOC with EMMA: http://project.europeanmoocs.eu/project/get-involved/become-an-emma-mooc-provider/
Tools May 2015 Logic models for prog.docxturveycharlyn
Tools
May 2015
Logic models for program
design, implementation, and
evaluation: Workshop toolkit
Karen Shakman
Sheila M. Rodriguez
Education Development Center, Inc.
Overview
This Logic Model Workshop Toolkit is designed to
help practitioners learn the overall purpose of a logic
model, the different elements of a logic model, and
the appropriate steps for developing and using a logic
model for program evaluation. This toolkit includes a
facilitator workbook, a participant workbook, and a
slide deck.
At Education Development Center, Inc.
U . S . D e p a r t m e n t o f E d u c a t i o n
U.S. Department of Education
Arne Duncan, Secretary
Institute of Education Sciences
Sue Betka, Acting Director
National Center for Education Evaluation and Regional Assistance
Ruth Curran Neild, Commissioner
Joy Lesnick, Associate Commissioner
Amy Johnson, Action Editor
Chris Boccanfuso, Project Officer
REL 2015–057
The National Center for Education Evaluation and Regional Assistance (NCEE) conducts
unbiased large-scale evaluations of education programs and practices supported by federal
funds; provides research-based technical assistance to educators and policymakers; and
supports the synthesis and the widespread dissemination of the results of research and
evaluation throughout the United States.
May 2015
This report was prepared for the Institute of Education Sciences (IES) under Contract
ED-IES-12-C-0009 by Regional Educational Laboratory Northeast & Islands administered
by Education Development Center, Inc. The content of the publication does not neces
sarily reflect the views or policies of IES or the U.S. Department of Education nor does
mention of trade names, commercial products, or organizations imply endorsement by the
U.S. Government.
This REL report is in the public domain. While permission to reprint this publication is
not necessary, it should be cited as:
Shakman, K., & Rodriguez, S. M. (2015). Logic models for program design, implementation,
and evaluation: Workshop toolkit (REL 2015–057). Washington, DC: U.S. Department of
Education, Institute of Education Sciences, National Center for Education Evaluation and
Regional Assistance, Regional Educational Laboratory Northeast & Islands. Retrieved
from http://ies.ed.gov/ncee/edlabs.
This report is available on the Regional Educational Laboratory website at http://ies.ed.gov/
ncee/edlabs.
http://ies.ed.gov/ncee/edlabs
http://ies.ed.gov/ncee/edlabs
http://ies.ed.gov/ncee/edlabs
Contents
Facilitator workbook
Introduction to workshop 3
Session I. Learning about logic models (2 hours) 5
Session II. From logic models to program and policy evaluation (1.5 hours) 30
Appendix A. Simple logic model A-1
Appendix B. Template for simple logic model B-1
Appendix C. Sample logic model for college readiness program C-1
Appendix D. Example of a logic model for an educator evaluation system theory of acti ...
Skills & ideas for #ProblemGamblingKTE Anne Bergen
Skills & ideas for #ProblemGamblingKTE. (2104). Part of the "Moving Research Forward" Workshop Series for the Ontario Problem Gambling Research Centre.
An institutional perspective on analytics that focusses on a particular tool developed using an agile methodology to visualise learner behaviours in MOOCs via Sankey diagrams.
Assessing the Implementation of Authentic, Client-Facing Student Projects in ...craigmmacdonald
User Experience (UX) is often cited as one of the fastest growing occupations, creating opportunities across nearly every sector for individuals skilled in the application of user-centered design principles and methods. Many Information and Library Schools have responded to this demand by introducing more UX coursework into their curriculum, but the proliferation of agile software development and lean product design has incentivized organizations to look for experienced individuals for UX roles, even those that are considered entry-level. As a result, aspiring information professionals face a paradoxical situation in which they are required to have UX experience before they can gain UX experience. This article provides an assessment of one institution's efforts to overcome this experience gap by offering opportunities for students to participate in three types of authentic client-facing UX projects. Through surveys of students and clients served over four academic years, we provide a set of lessons learned and recommended best practices for incorporating project-based learning opportunities into UX courses.
Models of curriculum evaluation and application in educationalKoledafe Olawale
Curriculum can be defined as the planned and guided learning experiences and intended learning outcomes, formulated through the systematic reconstruction of knowledge and experiences, under the auspices of the school, for the learners’ continuous and willful growth in personal social competence (Tanner & Tanner, 1975)
Similar to Evaluation2016_Session2667_Mapping_Programs_Roundtable_10272016_draft3_AEAtemplate_Tarek (1) (1) (20)
1. Roundtable: Cognitive Mapping as a tool
for program and evaluation design
Joshua Penman, Carlos Echeverria-Estrada &
Agnieszka Rykaczewska
2. We acknowledge the invaluable support this team
received from CGU faculty, in particular from the
project P.I., Dr. Tarek Azzam, and the staff responsible
for administering the SEPA grant at City of Hope.
3. Evaluation of the San Gabriel Valley’s SEPAC (K-8
Research Education Program)
May 2012 – Start of San Gabriel
Valley Science Education Partnership
Award Collaborative (SGV SEPAC) –
2012/13
Summer 2012 – CGU was hired
2012/2015 – evaluation questions
and design, yearly data collection
and reporting separately by method.
2015/16 – Logic model and program
mapping (four components),
questions about use and learning to
CoH staff.
2016/17 – Use of logic model and
program mapping for new gran
design & application.
Current data collection
April 2017 - Grant ends.
4. Cognitive Mapping
• Spatial location of elements (i.e., concepts) in a network,
relevant for the interpretation of relationships between
concepts (Coronges et al., 2007; Huff, 1990).
• In a conceptual map, nodes represent the concepts in
the knowledge domain, and the strings represent the
links between these concepts (Curşeu, Schalk, & Schruijer, 2010).
7. So, why Cognitive Mapping?
• Traditional logic model did not capture
underlying mechanisms involved in the
program effectiveness
• Maps can be used as tools, as a means of
applying embodied theory to a particular
application (Conole, Dyke, Oliver & Seale, 2004)
• Enables reflection on implementation that
elevates the importance of feedback
Source: http://www.clipartkid.com/man-
with-question-mark-cliparts/
8. Cognitive Mapping: Using Century, Rudnick & Freeman’s
Fidelity of Implementation (FOI) 1
Basic FOI Framework:
Categories of Critical Components
Procedural Educative Pedagogical
Student
engagement
Structural Critical Components Instructional Critical Components
1 Century, J., Rudnick, M., & Freeman, C. (2010). A Framework for Measuring Fidelity of Implementation: A Foundation for Shared
Language and Accumulation of Knowledge. American Journal Of Evaluation, 31(2), 199-218.
14. Did it help?
To Program Staff:
• The process helped clarify outcomes and how they were
achieved.
• The logic model process map helped to explain “why” does the
evaluation feedback actually mattered.
• Reassured that program in practice was aligned with the original
goals of the Grant
• Helpful in identifying critical features of implementation for future
programming
15. To Evaluation Team:
• Assess what measures are appropriate for each element in the
logic model (process & outcomes chain)
• Identified alignment
• Identified gaps
• It helped improve observations, knowing the background of the
program items.
• This last summer observations: considerably less general and
more attention to interactions between students and
teachers, and how they were affecting each other.
16. To Other Audiences:
• This tool addresses mostly
the organizational side of
the program. It can be used
in different subject domains
and with different theoretical
frameworks (e.g., Six
Sigma).
Source: http://blog.mitchcommgroup.com/ann-marie-
satterfield/how-you-reach-a-diverse-audience
17. Table discussion
• Raise your hand if you have any
experience with using approaches like
this? What’s your experience?
19. Table discussion
• Raise your hand if you have…
• What potential benefits do you see?
• Pro’s & con’s?
20. Table discussion
• Raise your hand if you have…
• What potential benefits do you see?
• Pro’s & Con’s?
• What was the role of the stakeholder theory?
21. Table discussion
• Raise your hand if you have…
• What potential benefits do you see?
• Pro’s & Con’s?
• What was the role of the stakeholder theory?
• How would this work with another theoretical
framework other than Century?
22. References:
Century, J., Rudnick, M., & Freeman, C. (2010). A Framework for Measuring Fidelity of
Implementation: A Foundation for Shared Language and Accumulation of Knowledge.
American Journal Of Evaluation, 31(2), 199-218.
Conole, G., Dyke, M., Oliver, M., & Seale, J. (2004). Mapping Pedagogy and Tools for
Effective Learning Design. Computers And Education, 43(1-2), 17-33.
Curşeu, P.L., Schalk, R., & Schruijer, S. (2010). The Use of Cognitive Mapping in Eliciting
and Evaluating Group Cognitions. Journal Of Applied Social Psychology, 40(5), 1258-
1291.
***SHORTEN THE STORY***
The San Gabriel Valley Science Education Partnership Award Collaborative (SGV SEPAC) is a partnership between City of Hope (COH), a Comprehensive Cancer Center and biomedical research institute, and the Duarte Unified School District (DUSD), an 80% minority school district in the San Gabriel Valley of Los Angeles County.
Focus on 3rd aim: establishing a K-8 research education program.
These include visits to 2nd grade classrooms by COH scientists and field trip visits to COH by 2nd, 5th, and 8th grade students. All students in these grade levels are included in the year’s activities. Second grade students learn about genetic traits and inheritance and then visit COH to tour the grounds and engage in fun exploration activities. Fifth grade students visit COH labs and learn about fundamental concepts related to the human body. Eighth grade students do hands-on science centered on the topic of DNA. These activities are designed to whet their appetite for research and pique their interest in applying for the summer research program during their high school years.
Hodgkinson and Clarkson (2005) argued that the process of cognitive mapping consists of four steps: (a) knowledge elicitation; (b) construction; (c) analysis; and (d) aggregation or comparison.
In the knowledge-elicitation stage, the authors distinguished between ideographic and nomothetic approaches. In the ideographic approach, the researcher collects the concepts that are used by the individual or the group to describe a particular task or the way in which a particular task is (to be) performed. In nomothetic elicitation, the researcher provides predefined concepts based on theoretical models or own hypotheses.
Hodgkinson and Clarkson (2005) argued that the process of cognitive mapping consists of four steps: (a) knowledge elicitation; (b) construction; (c) analysis; and (d) aggregation or comparison.
In the knowledge-elicitation stage, the authors distinguished between ideographic and nomothetic approaches. In the ideographic approach, the researcher collects the concepts that are used by the individual or the group to describe a particular task or the way in which a particular task is (to be) performed. In nomothetic elicitation, the researcher provides predefined concepts based on theoretical models or own hypotheses.
The traditional LM was not powerful enough to invite to the reflection of why delving into the implementation is so important for program improvement.
Limited to descriptive elements of general trends, but skips the details of underlying processes to make the program successful.
Also, poor visualization of such processes did not invite to reflection on implementation.
Useful to evaluation team and CoH’s staff.
***NESZKA***
Looking for a framework that helped the evaluation unravel the processes involved in a successful implementation, the team unpackaged the Framework for Measuring Fidelity of Implementation by Century, Rudnick and Freeman.
-They follow the same colors
***JOSH***
***JOSH***
Logic model of the mapping process
6 Categories
32 Items
43 Arrows on this sheet.
It just looks like spaghetti: arrows crossing over each other are confusing. It looks like what most of our clients think about their program: “That everything links to everything else. Everything we do is important.”
PS: In this, I think I gave an extra link for each thing that the others linked to, so we could perform a rudimentary network analysis to see what the most important items were. But that wasn’t very helpful.
***JOSH***
6 categories
25 items
27 Arrows on this sheet.
6 attributes (the bullet points within items)
3 levels (items stacked on top of each other)
Two types of level:
Component (it can be divided up into these things)
Additive: it builds on top of another item.
In addition, we have two special use cases:
Linked: multiple items on the same level linked with + sign to indicate that they are expected to be synergistic.
Attributes (describe the thing or a component)
Why use Pipes instead of Arrows? - J: We’re not sure; I think we wanted to make them bigger and draw more attention to them so that the client is less focused on the program components, and more focused on how they connect to each other. The really important thing, the “why” for having any given component is because of what it connects to.
Also, we think it looks a little bit like an old pipe game that used to
***NESZKA***
To Program Staff:
- They could say that “teachers weren’t effective” year after year; but how did teachers do that; . . . such a critical piece for effectiveness and impact. - They accepted it; they realized that in the next program they’re going to have to track implementation more carefully.
The traditional logic model addresses how things are supposed to happen, but doesn’t address “why” nearly as much.
In the case of the San Gabriel Valle SEPAC, process mapping captured the underlying mechanism of the interventions. This visual representation showed the program was still pursuing the original purposes, even after a significant staff turnovers during the course of the grant.
To Program Staff:
- They could say that “teachers weren’t effective” year after year; but how did teachers do that; . . . such a critical piece for effectiveness and impact. - They accepted it; they realized that in the next program they’re going to have to track implementation more carefully.
The traditional logic model addresses how things are supposed to happen, but doesn’t address “why” nearly as much.
In the case of the San Gabriel Valle SEPAC, process mapping captured the underlying mechanism of the interventions. This visual representation showed the program was still pursuing the original purposes, even after a significant staff turnovers during the course of the grant.
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?
a) Raise your hand if you have any experience with using approaches like this?
What’s your experience?
b) What potential benefits do you see?
What potential trade-off’s do you see? (Pro’s & Con’s)
What is the role of the stakeholder theory?
How would this work with another framework other than Century?