SlideShare a Scribd company logo
1 of 51
Program Theory & The Theory-
Driven Approach to Evaluation
        Jeffrey Sheldon, M. A., Ed. M.
School of Behavioral & Organizational Sciences
        Claremont Graduate University
           The Claremont Colleges
               25 October 2006




                                                 1
What does this model tell us?




                                2
Challenges in Program Evaluation
• Inadequate program conceptualization.

• Poor program implementation.

• Insensitive program evaluation.

• Poor stakeholder – evaluator relations.

• Scarcity of cumulative knowledge and wisdom.

                                                 3
Black Box Evaluation

• Evaluation of program outcomes without the benefit
  of an articulated program theory that provides
  insight into what is presumed to be causing those
  outcomes, and why.
                        Rossi, Lipsey & Freeman (2004)




                                                         4
Contingency View
• No single best way to conduct program evaluation.

• The choice of approaches and methods for program
  evaluation should be situational.

• The individual natures of programs and the
  uniqueness of evaluation purposes and contextual
  circumstances require use of a range of evaluation
  approaches and methods.
                                           Chen (2005)
                                                      5
Theories Used in Evaluation
• Evaluation theory.
   – Guides evaluation practice, e.g., empowerment
     evaluation, theory-driven evaluation, goals-free
     evaluation.

• Social science theory.
   – Theory from the extant literature, e.g., Social
     Learning Theory, Theory of Reasoned Action.

• Program theory
   – Stakeholder theory – varies by program.
                                                        6
Program Theory

• The set of assumptions about the manner in which
  the program relates to the social benefits it is
  expected to produce and the strategy and tactics the
  program has adopted to achieve its goals and
  objectives. Within program theory we can
  distinguish impact theory – the nature of the change
  in social conditions brought about by program action
  and process theory – the program’s organizational
  plan and service utilization plan.
                       Rossi, Lipsey & Freeman (2004)
                                                         7
Program Theory
• Assumptions made by stakeholders about what action
  is required to solve a social problem and why the
  problem will respond to this action.

• Perception of nature of problem is from experience,
  conventional wisdom, discussions with peers, etc…

• Solution to the problem is practicable.



                                                        8
Assumptions
• Prescriptive – the action that is required to
  solve a social problem.

  – Explained by the Process Model


• Descriptive – why the problem will respond to
  the action.

  – Explained by the Impact Model

                                                  9
Process Model

• Components and activities program designers and key
  stakeholders see as necessary for program success.

• Example: Wired With Wisdom Parent Recruitment
   Letters from Principals
   Champion Parents: phone, email, personal contact
   Children of target parents



                                                   10
Process Model                      Principal
                                                                               Positive
                                                                                Letter
                                                                                          Negative
                                                                                           Letter
                                                                                                        Incentive
                                                                                                          Letter
                                                                                                                    Follow-up
                                                                                                                      Letter



                                     Project
                                     Manager                                                                                    Effects intrinsic
                                                                                                                                   motivation
                                                                                                                                 leading to use

                                                                          Phone
Recruitment
 Program          WWK                PTO/PTA                             E - Mail
                                                  Champs
                                      Officers
                                                                         Personal             Parents                               Or
                                                                        Follow-up
                                                                                                                                  No effect on
                                                                                                                                   intrinsic
                                     Computer                                                                                     motivation
                                     Instructor                                                                                 leading to non-
                                                                                                                                      use


                                                           Internet training




                                     Teachers
                                                                                             Students



 Direct influence/communication:


 Direct action:


 Two-way influence/communication:

                                                                                                                                         11
Components of Process Model

• Intervention and service delivery protocols.

• Implementing organization: assess, enhance,
  and ensure its capacity.

• Program implementers: recruit, train and
  maintain both competency and commitment.

                                                 12
Components of Process Model

• Associate organizations/community partners:
  establishing collaborations.

• Ecological context: seek its support at micro
  and macro levels.

• Target population: identify, recruit, screen,
  serve.
                                                  13
Impact Model

• Assumptions about causal processes through which
  intervention is supposed to work.

• Example:

Parent recruitment   Increases or        Successful recruitment,

Methods              Reduces Intrinsic   use of program
                     Motivators



                                                                   14
Impact Model
                                Reduce
                               Perceived
                                Barriers




                                Increase
                               Perceived
                                Benefits

      Wired With Wisdom
  Parent Recruitment Program
                                                  Successful    Well managed
            (Letters
                                                recruitment =   family internet
          Phone calls
            E - mails                               use of                        Safer children
                                                                environment &
                                                 Wired With
       Personal contact)                                          safety plan
                                                   Wisdom
                                 Increase
                                Perceived
                               Susceptibility




                                  Increase
                                 Perceived
                                  Severity




                                 Increase
                                  Action
                                   Cues



                                                                                             15
Components of the Impact Model
• Intervention/treatment

• Determinants
   – Mediators
   – Moderators


• Goals/outcomes:
   – Distal
   – Intermediate
   – Proximal
                                      16
Logic Models & Program Theory




                                17
Program Theory Considerations

•   Parsimony (the core of the program).
•   Precision of relationships.
•   Bi-directional approach.
•   Program-effect decay functions.
•   Dose-response functions.
•   Mediators.
•   Moderators.

                                           18
Program-effect Decay Functions




                                 19
Dose-Response Functions




                          20
Direct Effects Model




                       21
One Mediator Model




                     22
Indirect Effects Model




                         23
Multiple Mediator Effects Model




                                  24
Moderator of Mediator Effect Model




                                     25
Moderator of Mediator-Outcome
        Relationship




                                26
Small Group Exercise
Based on the following model, describe the program,
its actions and the changes that are expected to
result. Use the questions below as a guide.

    What is the intervention/treatment?
    What are the mediators?
    What are the moderators?
    What are the proximal outcomes?
    What is the distal outcome?


                                                      27
Computers In Our Future




                          28
Theory Calls Evaluation Practitioner’s
             Attention To:
• Which stage or stages of the program cycle will be
  the focus of the evaluation?

• What do stakeholders want from the evaluation?

• What evaluation options potentially fit the given
  program’s context?

• What trade-offs among these options will be most
  profitable?
                                                       29
Using Program Theory to Design
               Evaluations
• Compels evaluators to be thoughtful before acting.

• Enhances understanding of program.

• Program assumptions used as scaffolding for the
  study.

• Informs method choices – qualitative, quantitative or
  mixed, that is, the contingency view!
                                                       30
Using Program Theory to Design
               Evaluations
• Highlights elements of program activity that deserve
  attention in the evaluation.

• Helps tailor evaluations to answer the most important
  questions (remember, parsimony).

• Heightens evaluation responsiveness and sensitivity.

• Increases validity – construct & internal.
                                                         31
Using Program Theory to Design
               Evaluations
• Fosters cumulative wisdom.

• Helps evaluators meet American Evaluation
  Association professional evaluation standards –
  Utility, Feasibility, Practicality, Accuracy.

• Can choose to collect data on linkage mechanisms
  assumed to be operative in one theory or in several
  theories.

• Can direct the evaluation toward investigating one
  link in the theory chain.                             32
Is Theory-Driven Evaluation
        Methodologically Rigorous?

“The tie-in, or relationship, of the theory-driven
  approach with our best methodological work is
  impressive. Think about the way we establish the
  validity of constructs in experimental research. In
  essence, construct validation requires a theory, an
  understanding of the hypothetical network of causal
  associations and non-causal relationships among the
  variables that we might try to understand”
                                             Crano (2003)
                                                        33
Theory-driven Evaluation
           The CDC Framework
• Engage stakeholders – evaluability assessment.

• Describe the program through the action and change
  models.

• Formulate & prioritize evaluation questions.

• Focus the evaluation design.



                                                       34
Theory-driven Evaluation
            The CDC Framework

• Gather credible evidence through rigorous scientific
  methods.

• Justify conclusions.

• Ensure utilization and lessons learned.




                                                         35
CDC Evaluation Framework




                           36
Effective Theory-Driven Evaluations

1. Future action directedness.

   –   Useful to stakeholders.

   –   Assessing merit is a means rather than an end.

   –   Provides useful information for stakeholders to
       improve current or future programs.

                                                         37
Effective Theory-Driven Evaluations

1. Scientific and stakeholder credibility.

   –   Follows scientific methods and principles to
       optimize validity and reliability.

   –   Responding to stakeholders’ values, views,
       concerns, and needs.



                                                      38
Effective Theory-Driven Evaluations

1. Holistic Approach

  –   Intrinsic value.

  –   Context.




                                        39
Explicating Program Theory: Basics
Facilitated by Evaluator:

• Stakeholders reflectively examine what they are
  doing.

• Stakeholders identify elements that are essential for
  achieving program goals.

• Stakeholders articulate causal relationships.


                                                          40
Explicating Program Theory: Process
• Face-to-face meetings with stakeholders (working group or
  intensive interview)

• Facilitating conceptualization of program:
   – “Tell me how your program works.”
   – “What do you want your program to do?”
   – “What circumstance does it mitigate or need it meets?”
   – “Who does it impact?”

• Theorizing methods – backward reasoning (start with intended
  outcomes) to inputs, forward reasoning, or both.
                                                              41
Explicating Program Theory: Results
• Stake-holder buy-in & support of evaluation.

• Systematic understanding of stakeholder views,
  needs, and values.

• Utilization of knowledge produced by the evaluation

   –   Conceptual (understanding/education)
   –   Instrumental (decision-support)
   –   Process (making use of the logic of the evaluation)
   –   Symbolic (justify a priori decisions)
   –   Influence                                         42
Evaluation Questions Hierarchy




                                 43
Types of Theory-Driven Evaluations

• Action Model  Theory-driven process evaluation.

• Change Model  Theory-driven outcome evaluation.

• Action Model + Change Model  Integrated
  Theory-driven process/outcome evaluation.




                                                     44
Theory-Driven Process Evaluation
• Systematically assess how the following major
  components of an action model are being
  implemented in the field:

   –   Intervention and service delivery protocols
   –   Target populations
   –   Implementing organization
   –   Implementers
   –   Associate organizations/partners
   –   Ecological support                            45
Theory-Driven Outcome/Impact Evaluation
 • Serves accountability and program improvement needs by
   investigating underlying causal mechanisms.

 • Comments on construct ability

 • Increases internal validity.

 • Generates two kinds of information:
    – Assesses if program achieving its predetermined goals
    – Investigates why and how program succeeds or does not succeed


                                                                      46
Integrated Process – Outcome/Impact
                Evaluation
Implementation of
parent recruitment
action model



Parent recruitment           Increases or         Successful recruitment,

Methods                  Reduces Intrinsic        use of program
                         Motivators



(Action theory of success)                   (Conceptual theory of success)

                                                                            47
References
• Bickman, L. (Ed.) (1987). Using program theory in evaluation.
  New Directions for Program Evaluation, No., 47. San
  Francisco, CA: Jossey-Bass.

• Birkmayer, J. D., & Weiss, C. H. (2000). Theory-based
  evaluation in practice. Evaluation Review, 21(4), 407 – 431.

• Chen, H. T. (2005). Practical Program Evaluation: Assessing,
  and Improving Planning, Implementation and Effectiveness.
  Thousand Oaks, CA: Sage.

• Chen, H. T. (1990). Theory-Driven Evaluation. Newbury Park,
  CA: Sage


                                                                 48
References
• Chen, H. T., & Rossi, P. H. (1983). Evaluating with sense: The
  theory-driven approach. Evaluation Review, 7, 283 – 302.

• Chen, H. T., & Rossi, P. H. (1987). The theory-driven
  approach to evaluation. Evaluation and Program Planning,
  10, 95 – 103.

• Crano, W. D. (2003). Theory-driven evaluation and construct
  validity. In S. I. Donaldson and M. Scriven (Eds.), Evaluating
  Social Programs and Problems: Visions for the New
  Millennium. Mahwah, NJ: Erlbaum.

• Donaldson, S. I. (Forthcoming). Program theory-driven
  evaluation science: Strategies and applications. Mahwah, NJ:
  Erlbaum.
                                                                   49
References
• Donaldson, S. I. (2002). Theory-Driven Evaluation in the
  New Millennium. In S. I. Donaldson and M. Scriven (Eds.).
  Evaluating Social Programs and Problems: Visions for the
  New Millennium. Mahwah, NJ: Erlbaum.

• Donaldson, S. I., & Lipsey, M. (2006). Roles for theory in
  contemporary evaluation practice: Developing practical
  knowledge. In. I. Shaw, J. C. Greene, and M. H. Mark (Eds.).
  The SAGE Handbook of Evaluation, Thousand Oaks, CA:
  Sage.

• Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004).
  Program Evaluation: Alternative Approaches and Practical
  Guidelines (3rd Ed.). Boston, MA: Pearson.

• Lipsey, M. (1988). Practice and malpractice in evaluation
  research. Evaluation Practice, 8(4), 5 – 24.                   50
References
• Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004).
  Evaluation: A Systematic Approach (7th Ed.). Thousand Oaks,
  CA: Sage.

• Shadish, W. R., Cook, T., & Leviton, L. (1991). Foundations
  of Program Evaluation: Theories of Practice. Newbury Park,
  CA: Sage.

• Weiss, C. H. (1997). How can theory-based evaluation make
  greater headway? Evaluation Review, 21(4), 501 – 524.

• Weiss, C. H. (1998). Evaluation: Methods for Studying
  Programs and Policies (2nd Ed.). Upper Saddle River, NJ:
  Prentice Hall.

                                                              51

More Related Content

Similar to Program Theory Lecture University of San Diego October 2006

Edu Portal Solution
Edu Portal SolutionEdu Portal Solution
Edu Portal SolutionMandar Joshi
 
Developing Individual Development Plans (IDP)
Developing Individual Development Plans (IDP)Developing Individual Development Plans (IDP)
Developing Individual Development Plans (IDP)Seta Wicaksana
 
Total Performance Scoecard
Total Performance ScoecardTotal Performance Scoecard
Total Performance ScoecardNimra Gull
 
The Impact of Nonprofit Academic Programs on the Practice of Nonprofit Leader...
The Impact of Nonprofit Academic Programs on the Practice of Nonprofit Leader...The Impact of Nonprofit Academic Programs on the Practice of Nonprofit Leader...
The Impact of Nonprofit Academic Programs on the Practice of Nonprofit Leader...heathercarpenter
 
Feedback and communication report
Feedback and communication reportFeedback and communication report
Feedback and communication reportjesseba
 
K marvel week 6 leadership academy
K marvel week 6 leadership academyK marvel week 6 leadership academy
K marvel week 6 leadership academyKam Marvel
 
Using research to_create_effective_on-line
Using research to_create_effective_on-lineUsing research to_create_effective_on-line
Using research to_create_effective_on-linetrtkaren
 
D6 e6 outcome based measurement from theory to implementation
D6 e6 outcome based measurement from theory to implementationD6 e6 outcome based measurement from theory to implementation
D6 e6 outcome based measurement from theory to implementationocasiconference
 
2010 bush foundation inputs, process indicators, and intermediate outcomes
2010 bush foundation inputs, process indicators, and intermediate outcomes2010 bush foundation inputs, process indicators, and intermediate outcomes
2010 bush foundation inputs, process indicators, and intermediate outcomesChristopher Thorn
 
Evaluating and communicating your project
Evaluating and communicating your project Evaluating and communicating your project
Evaluating and communicating your project mycommunitylocality
 
A Guide to Logic Models – Grant Writing
A Guide to Logic Models – Grant WritingA Guide to Logic Models – Grant Writing
A Guide to Logic Models – Grant Writing4Good.org
 
Janet May's Assessing Online Learning Process Maturity: the e-Learning Maturi...
Janet May's Assessing Online Learning Process Maturity: the e-Learning Maturi...Janet May's Assessing Online Learning Process Maturity: the e-Learning Maturi...
Janet May's Assessing Online Learning Process Maturity: the e-Learning Maturi...Alexandra M. Pickett
 
Management of Remote Workers_Best Practice principles and processes
Management of Remote Workers_Best Practice principles and processesManagement of Remote Workers_Best Practice principles and processes
Management of Remote Workers_Best Practice principles and processesCharles Cotter, PhD
 

Similar to Program Theory Lecture University of San Diego October 2006 (20)

Edu Portal Solution
Edu Portal SolutionEdu Portal Solution
Edu Portal Solution
 
Developing Individual Development Plans (IDP)
Developing Individual Development Plans (IDP)Developing Individual Development Plans (IDP)
Developing Individual Development Plans (IDP)
 
Putting Program Evaluation to Work for You
Putting Program Evaluation to Work for YouPutting Program Evaluation to Work for You
Putting Program Evaluation to Work for You
 
Employer Internship Toolkit
Employer Internship ToolkitEmployer Internship Toolkit
Employer Internship Toolkit
 
Total Performance Scoecard
Total Performance ScoecardTotal Performance Scoecard
Total Performance Scoecard
 
The Impact of Nonprofit Academic Programs on the Practice of Nonprofit Leader...
The Impact of Nonprofit Academic Programs on the Practice of Nonprofit Leader...The Impact of Nonprofit Academic Programs on the Practice of Nonprofit Leader...
The Impact of Nonprofit Academic Programs on the Practice of Nonprofit Leader...
 
Feedback and communication report
Feedback and communication reportFeedback and communication report
Feedback and communication report
 
K marvel week 6 leadership academy
K marvel week 6 leadership academyK marvel week 6 leadership academy
K marvel week 6 leadership academy
 
Using research to_create_effective_on-line
Using research to_create_effective_on-lineUsing research to_create_effective_on-line
Using research to_create_effective_on-line
 
Keep the fires burning
Keep the fires burningKeep the fires burning
Keep the fires burning
 
Computerized Grading System 2016
Computerized Grading System 2016Computerized Grading System 2016
Computerized Grading System 2016
 
Goal setting
Goal settingGoal setting
Goal setting
 
D6 e6 outcome based measurement from theory to implementation
D6 e6 outcome based measurement from theory to implementationD6 e6 outcome based measurement from theory to implementation
D6 e6 outcome based measurement from theory to implementation
 
2010 bush foundation inputs, process indicators, and intermediate outcomes
2010 bush foundation inputs, process indicators, and intermediate outcomes2010 bush foundation inputs, process indicators, and intermediate outcomes
2010 bush foundation inputs, process indicators, and intermediate outcomes
 
Evaluating and communicating your project
Evaluating and communicating your project Evaluating and communicating your project
Evaluating and communicating your project
 
Leadership Development and Your Business
Leadership Developmentand Your BusinessLeadership Developmentand Your Business
Leadership Development and Your Business
 
A Guide to Logic Models – Grant Writing
A Guide to Logic Models – Grant WritingA Guide to Logic Models – Grant Writing
A Guide to Logic Models – Grant Writing
 
Janet May's Assessing Online Learning Process Maturity: the e-Learning Maturi...
Janet May's Assessing Online Learning Process Maturity: the e-Learning Maturi...Janet May's Assessing Online Learning Process Maturity: the e-Learning Maturi...
Janet May's Assessing Online Learning Process Maturity: the e-Learning Maturi...
 
Stake's Model of curriculum Evaluation
Stake's Model of curriculum Evaluation Stake's Model of curriculum Evaluation
Stake's Model of curriculum Evaluation
 
Management of Remote Workers_Best Practice principles and processes
Management of Remote Workers_Best Practice principles and processesManagement of Remote Workers_Best Practice principles and processes
Management of Remote Workers_Best Practice principles and processes
 

More from Jeffrey Andrew Sheldon, M.A., Ed.M.

Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019Jeffrey Andrew Sheldon, M.A., Ed.M.
 
Lecture 9 strategic planning for iranian ng os - dr. jeff sheldon - march 2019
Lecture 9   strategic planning for iranian ng os - dr. jeff sheldon - march 2019Lecture 9   strategic planning for iranian ng os - dr. jeff sheldon - march 2019
Lecture 9 strategic planning for iranian ng os - dr. jeff sheldon - march 2019Jeffrey Andrew Sheldon, M.A., Ed.M.
 
Developing Leadership Capacity to Influence the Self-Evaluating Organization
Developing Leadership Capacity to Influence the Self-Evaluating Organization Developing Leadership Capacity to Influence the Self-Evaluating Organization
Developing Leadership Capacity to Influence the Self-Evaluating Organization Jeffrey Andrew Sheldon, M.A., Ed.M.
 
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016Jeffrey Andrew Sheldon, M.A., Ed.M.
 
Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides
Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final SlidesDissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides
Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final SlidesJeffrey Andrew Sheldon, M.A., Ed.M.
 
Geita District Mwanza Region Northern Tanzania April 2007 J Sheldon
Geita District Mwanza Region Northern Tanzania April 2007 J SheldonGeita District Mwanza Region Northern Tanzania April 2007 J Sheldon
Geita District Mwanza Region Northern Tanzania April 2007 J SheldonJeffrey Andrew Sheldon, M.A., Ed.M.
 
Democratic Republic Of Congo Education System Transformation December 2006 J ...
Democratic Republic Of Congo Education System Transformation December 2006 J ...Democratic Republic Of Congo Education System Transformation December 2006 J ...
Democratic Republic Of Congo Education System Transformation December 2006 J ...Jeffrey Andrew Sheldon, M.A., Ed.M.
 

More from Jeffrey Andrew Sheldon, M.A., Ed.M. (14)

Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
 
Lecture 9 strategic planning for iranian ng os - dr. jeff sheldon - march 2019
Lecture 9   strategic planning for iranian ng os - dr. jeff sheldon - march 2019Lecture 9   strategic planning for iranian ng os - dr. jeff sheldon - march 2019
Lecture 9 strategic planning for iranian ng os - dr. jeff sheldon - march 2019
 
Developing Leadership Capacity to Influence the Self-Evaluating Organization
Developing Leadership Capacity to Influence the Self-Evaluating Organization Developing Leadership Capacity to Influence the Self-Evaluating Organization
Developing Leadership Capacity to Influence the Self-Evaluating Organization
 
Final Products Report
Final Products ReportFinal Products Report
Final Products Report
 
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
 
Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides
Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final SlidesDissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides
Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides
 
Operacion Pedro Pan December 2006 J Sheldon
Operacion Pedro Pan December 2006 J SheldonOperacion Pedro Pan December 2006 J Sheldon
Operacion Pedro Pan December 2006 J Sheldon
 
Mc Cord Hosptial Final Products Report August 2005 J Sheldon
Mc Cord Hosptial Final Products Report August 2005 J SheldonMc Cord Hosptial Final Products Report August 2005 J Sheldon
Mc Cord Hosptial Final Products Report August 2005 J Sheldon
 
Geita District Mwanza Region Northern Tanzania April 2007 J Sheldon
Geita District Mwanza Region Northern Tanzania April 2007 J SheldonGeita District Mwanza Region Northern Tanzania April 2007 J Sheldon
Geita District Mwanza Region Northern Tanzania April 2007 J Sheldon
 
Democratic Republic Of Congo Education System Transformation December 2006 J ...
Democratic Republic Of Congo Education System Transformation December 2006 J ...Democratic Republic Of Congo Education System Transformation December 2006 J ...
Democratic Republic Of Congo Education System Transformation December 2006 J ...
 
Ces Conference Presentation June 2006 J Sheldon
Ces Conference Presentation June 2006 J SheldonCes Conference Presentation June 2006 J Sheldon
Ces Conference Presentation June 2006 J Sheldon
 
Aea Conference Presentation November 2007 J Sheldon
Aea Conference Presentation November 2007 J SheldonAea Conference Presentation November 2007 J Sheldon
Aea Conference Presentation November 2007 J Sheldon
 
Aea Conference Presentation November 2007 J Sheldon
Aea Conference Presentation November 2007 J SheldonAea Conference Presentation November 2007 J Sheldon
Aea Conference Presentation November 2007 J Sheldon
 
Aea Ces Conference Poster October 2005 J Sheldon
Aea Ces Conference Poster October 2005 J SheldonAea Ces Conference Poster October 2005 J Sheldon
Aea Ces Conference Poster October 2005 J Sheldon
 

Recently uploaded

4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptxmary850239
 
Keynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designKeynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designMIPLM
 
Concurrency Control in Database Management system
Concurrency Control in Database Management systemConcurrency Control in Database Management system
Concurrency Control in Database Management systemChristalin Nelson
 
The Contemporary World: The Globalization of World Politics
The Contemporary World: The Globalization of World PoliticsThe Contemporary World: The Globalization of World Politics
The Contemporary World: The Globalization of World PoliticsRommel Regala
 
ICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfVanessa Camilleri
 
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17Celine George
 
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxINTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxHumphrey A Beña
 
How to Add Barcode on PDF Report in Odoo 17
How to Add Barcode on PDF Report in Odoo 17How to Add Barcode on PDF Report in Odoo 17
How to Add Barcode on PDF Report in Odoo 17Celine George
 
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...Nguyen Thanh Tu Collection
 
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...Postal Advocate Inc.
 
Integumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptIntegumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptshraddhaparab530
 
4.18.24 Movement Legacies, Reflection, and Review.pptx
4.18.24 Movement Legacies, Reflection, and Review.pptx4.18.24 Movement Legacies, Reflection, and Review.pptx
4.18.24 Movement Legacies, Reflection, and Review.pptxmary850239
 
4.16.24 Poverty and Precarity--Desmond.pptx
4.16.24 Poverty and Precarity--Desmond.pptx4.16.24 Poverty and Precarity--Desmond.pptx
4.16.24 Poverty and Precarity--Desmond.pptxmary850239
 
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxMULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxAnupkumar Sharma
 
Q4-PPT-Music9_Lesson-1-Romantic-Opera.pptx
Q4-PPT-Music9_Lesson-1-Romantic-Opera.pptxQ4-PPT-Music9_Lesson-1-Romantic-Opera.pptx
Q4-PPT-Music9_Lesson-1-Romantic-Opera.pptxlancelewisportillo
 
Presentation Activity 2. Unit 3 transv.pptx
Presentation Activity 2. Unit 3 transv.pptxPresentation Activity 2. Unit 3 transv.pptx
Presentation Activity 2. Unit 3 transv.pptxRosabel UA
 
Measures of Position DECILES for ungrouped data
Measures of Position DECILES for ungrouped dataMeasures of Position DECILES for ungrouped data
Measures of Position DECILES for ungrouped dataBabyAnnMotar
 
Transaction Management in Database Management System
Transaction Management in Database Management SystemTransaction Management in Database Management System
Transaction Management in Database Management SystemChristalin Nelson
 

Recently uploaded (20)

4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx
 
Keynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designKeynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-design
 
Concurrency Control in Database Management system
Concurrency Control in Database Management systemConcurrency Control in Database Management system
Concurrency Control in Database Management system
 
The Contemporary World: The Globalization of World Politics
The Contemporary World: The Globalization of World PoliticsThe Contemporary World: The Globalization of World Politics
The Contemporary World: The Globalization of World Politics
 
ICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdf
 
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
 
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxINTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
 
How to Add Barcode on PDF Report in Odoo 17
How to Add Barcode on PDF Report in Odoo 17How to Add Barcode on PDF Report in Odoo 17
How to Add Barcode on PDF Report in Odoo 17
 
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
 
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
 
Paradigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTAParadigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTA
 
Integumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptIntegumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.ppt
 
4.18.24 Movement Legacies, Reflection, and Review.pptx
4.18.24 Movement Legacies, Reflection, and Review.pptx4.18.24 Movement Legacies, Reflection, and Review.pptx
4.18.24 Movement Legacies, Reflection, and Review.pptx
 
4.16.24 Poverty and Precarity--Desmond.pptx
4.16.24 Poverty and Precarity--Desmond.pptx4.16.24 Poverty and Precarity--Desmond.pptx
4.16.24 Poverty and Precarity--Desmond.pptx
 
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxMULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
 
Q4-PPT-Music9_Lesson-1-Romantic-Opera.pptx
Q4-PPT-Music9_Lesson-1-Romantic-Opera.pptxQ4-PPT-Music9_Lesson-1-Romantic-Opera.pptx
Q4-PPT-Music9_Lesson-1-Romantic-Opera.pptx
 
Presentation Activity 2. Unit 3 transv.pptx
Presentation Activity 2. Unit 3 transv.pptxPresentation Activity 2. Unit 3 transv.pptx
Presentation Activity 2. Unit 3 transv.pptx
 
Measures of Position DECILES for ungrouped data
Measures of Position DECILES for ungrouped dataMeasures of Position DECILES for ungrouped data
Measures of Position DECILES for ungrouped data
 
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptxYOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
 
Transaction Management in Database Management System
Transaction Management in Database Management SystemTransaction Management in Database Management System
Transaction Management in Database Management System
 

Program Theory Lecture University of San Diego October 2006

  • 1. Program Theory & The Theory- Driven Approach to Evaluation Jeffrey Sheldon, M. A., Ed. M. School of Behavioral & Organizational Sciences Claremont Graduate University The Claremont Colleges 25 October 2006 1
  • 2. What does this model tell us? 2
  • 3. Challenges in Program Evaluation • Inadequate program conceptualization. • Poor program implementation. • Insensitive program evaluation. • Poor stakeholder – evaluator relations. • Scarcity of cumulative knowledge and wisdom. 3
  • 4. Black Box Evaluation • Evaluation of program outcomes without the benefit of an articulated program theory that provides insight into what is presumed to be causing those outcomes, and why. Rossi, Lipsey & Freeman (2004) 4
  • 5. Contingency View • No single best way to conduct program evaluation. • The choice of approaches and methods for program evaluation should be situational. • The individual natures of programs and the uniqueness of evaluation purposes and contextual circumstances require use of a range of evaluation approaches and methods. Chen (2005) 5
  • 6. Theories Used in Evaluation • Evaluation theory. – Guides evaluation practice, e.g., empowerment evaluation, theory-driven evaluation, goals-free evaluation. • Social science theory. – Theory from the extant literature, e.g., Social Learning Theory, Theory of Reasoned Action. • Program theory – Stakeholder theory – varies by program. 6
  • 7. Program Theory • The set of assumptions about the manner in which the program relates to the social benefits it is expected to produce and the strategy and tactics the program has adopted to achieve its goals and objectives. Within program theory we can distinguish impact theory – the nature of the change in social conditions brought about by program action and process theory – the program’s organizational plan and service utilization plan. Rossi, Lipsey & Freeman (2004) 7
  • 8. Program Theory • Assumptions made by stakeholders about what action is required to solve a social problem and why the problem will respond to this action. • Perception of nature of problem is from experience, conventional wisdom, discussions with peers, etc… • Solution to the problem is practicable. 8
  • 9. Assumptions • Prescriptive – the action that is required to solve a social problem. – Explained by the Process Model • Descriptive – why the problem will respond to the action. – Explained by the Impact Model 9
  • 10. Process Model • Components and activities program designers and key stakeholders see as necessary for program success. • Example: Wired With Wisdom Parent Recruitment Letters from Principals Champion Parents: phone, email, personal contact Children of target parents 10
  • 11. Process Model Principal Positive Letter Negative Letter Incentive Letter Follow-up Letter Project Manager Effects intrinsic motivation leading to use Phone Recruitment Program WWK PTO/PTA E - Mail Champs Officers Personal Parents Or Follow-up No effect on intrinsic Computer motivation Instructor leading to non- use Internet training Teachers Students Direct influence/communication: Direct action: Two-way influence/communication: 11
  • 12. Components of Process Model • Intervention and service delivery protocols. • Implementing organization: assess, enhance, and ensure its capacity. • Program implementers: recruit, train and maintain both competency and commitment. 12
  • 13. Components of Process Model • Associate organizations/community partners: establishing collaborations. • Ecological context: seek its support at micro and macro levels. • Target population: identify, recruit, screen, serve. 13
  • 14. Impact Model • Assumptions about causal processes through which intervention is supposed to work. • Example: Parent recruitment Increases or Successful recruitment, Methods Reduces Intrinsic use of program Motivators 14
  • 15. Impact Model Reduce Perceived Barriers Increase Perceived Benefits Wired With Wisdom Parent Recruitment Program Successful Well managed (Letters recruitment = family internet Phone calls E - mails use of Safer children environment & Wired With Personal contact) safety plan Wisdom Increase Perceived Susceptibility Increase Perceived Severity Increase Action Cues 15
  • 16. Components of the Impact Model • Intervention/treatment • Determinants – Mediators – Moderators • Goals/outcomes: – Distal – Intermediate – Proximal 16
  • 17. Logic Models & Program Theory 17
  • 18. Program Theory Considerations • Parsimony (the core of the program). • Precision of relationships. • Bi-directional approach. • Program-effect decay functions. • Dose-response functions. • Mediators. • Moderators. 18
  • 25. Moderator of Mediator Effect Model 25
  • 26. Moderator of Mediator-Outcome Relationship 26
  • 27. Small Group Exercise Based on the following model, describe the program, its actions and the changes that are expected to result. Use the questions below as a guide. What is the intervention/treatment? What are the mediators? What are the moderators? What are the proximal outcomes? What is the distal outcome? 27
  • 28. Computers In Our Future 28
  • 29. Theory Calls Evaluation Practitioner’s Attention To: • Which stage or stages of the program cycle will be the focus of the evaluation? • What do stakeholders want from the evaluation? • What evaluation options potentially fit the given program’s context? • What trade-offs among these options will be most profitable? 29
  • 30. Using Program Theory to Design Evaluations • Compels evaluators to be thoughtful before acting. • Enhances understanding of program. • Program assumptions used as scaffolding for the study. • Informs method choices – qualitative, quantitative or mixed, that is, the contingency view! 30
  • 31. Using Program Theory to Design Evaluations • Highlights elements of program activity that deserve attention in the evaluation. • Helps tailor evaluations to answer the most important questions (remember, parsimony). • Heightens evaluation responsiveness and sensitivity. • Increases validity – construct & internal. 31
  • 32. Using Program Theory to Design Evaluations • Fosters cumulative wisdom. • Helps evaluators meet American Evaluation Association professional evaluation standards – Utility, Feasibility, Practicality, Accuracy. • Can choose to collect data on linkage mechanisms assumed to be operative in one theory or in several theories. • Can direct the evaluation toward investigating one link in the theory chain. 32
  • 33. Is Theory-Driven Evaluation Methodologically Rigorous? “The tie-in, or relationship, of the theory-driven approach with our best methodological work is impressive. Think about the way we establish the validity of constructs in experimental research. In essence, construct validation requires a theory, an understanding of the hypothetical network of causal associations and non-causal relationships among the variables that we might try to understand” Crano (2003) 33
  • 34. Theory-driven Evaluation The CDC Framework • Engage stakeholders – evaluability assessment. • Describe the program through the action and change models. • Formulate & prioritize evaluation questions. • Focus the evaluation design. 34
  • 35. Theory-driven Evaluation The CDC Framework • Gather credible evidence through rigorous scientific methods. • Justify conclusions. • Ensure utilization and lessons learned. 35
  • 37. Effective Theory-Driven Evaluations 1. Future action directedness. – Useful to stakeholders. – Assessing merit is a means rather than an end. – Provides useful information for stakeholders to improve current or future programs. 37
  • 38. Effective Theory-Driven Evaluations 1. Scientific and stakeholder credibility. – Follows scientific methods and principles to optimize validity and reliability. – Responding to stakeholders’ values, views, concerns, and needs. 38
  • 39. Effective Theory-Driven Evaluations 1. Holistic Approach – Intrinsic value. – Context. 39
  • 40. Explicating Program Theory: Basics Facilitated by Evaluator: • Stakeholders reflectively examine what they are doing. • Stakeholders identify elements that are essential for achieving program goals. • Stakeholders articulate causal relationships. 40
  • 41. Explicating Program Theory: Process • Face-to-face meetings with stakeholders (working group or intensive interview) • Facilitating conceptualization of program: – “Tell me how your program works.” – “What do you want your program to do?” – “What circumstance does it mitigate or need it meets?” – “Who does it impact?” • Theorizing methods – backward reasoning (start with intended outcomes) to inputs, forward reasoning, or both. 41
  • 42. Explicating Program Theory: Results • Stake-holder buy-in & support of evaluation. • Systematic understanding of stakeholder views, needs, and values. • Utilization of knowledge produced by the evaluation – Conceptual (understanding/education) – Instrumental (decision-support) – Process (making use of the logic of the evaluation) – Symbolic (justify a priori decisions) – Influence 42
  • 44. Types of Theory-Driven Evaluations • Action Model  Theory-driven process evaluation. • Change Model  Theory-driven outcome evaluation. • Action Model + Change Model  Integrated Theory-driven process/outcome evaluation. 44
  • 45. Theory-Driven Process Evaluation • Systematically assess how the following major components of an action model are being implemented in the field: – Intervention and service delivery protocols – Target populations – Implementing organization – Implementers – Associate organizations/partners – Ecological support 45
  • 46. Theory-Driven Outcome/Impact Evaluation • Serves accountability and program improvement needs by investigating underlying causal mechanisms. • Comments on construct ability • Increases internal validity. • Generates two kinds of information: – Assesses if program achieving its predetermined goals – Investigates why and how program succeeds or does not succeed 46
  • 47. Integrated Process – Outcome/Impact Evaluation Implementation of parent recruitment action model Parent recruitment Increases or Successful recruitment, Methods Reduces Intrinsic use of program Motivators (Action theory of success) (Conceptual theory of success) 47
  • 48. References • Bickman, L. (Ed.) (1987). Using program theory in evaluation. New Directions for Program Evaluation, No., 47. San Francisco, CA: Jossey-Bass. • Birkmayer, J. D., & Weiss, C. H. (2000). Theory-based evaluation in practice. Evaluation Review, 21(4), 407 – 431. • Chen, H. T. (2005). Practical Program Evaluation: Assessing, and Improving Planning, Implementation and Effectiveness. Thousand Oaks, CA: Sage. • Chen, H. T. (1990). Theory-Driven Evaluation. Newbury Park, CA: Sage 48
  • 49. References • Chen, H. T., & Rossi, P. H. (1983). Evaluating with sense: The theory-driven approach. Evaluation Review, 7, 283 – 302. • Chen, H. T., & Rossi, P. H. (1987). The theory-driven approach to evaluation. Evaluation and Program Planning, 10, 95 – 103. • Crano, W. D. (2003). Theory-driven evaluation and construct validity. In S. I. Donaldson and M. Scriven (Eds.), Evaluating Social Programs and Problems: Visions for the New Millennium. Mahwah, NJ: Erlbaum. • Donaldson, S. I. (Forthcoming). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum. 49
  • 50. References • Donaldson, S. I. (2002). Theory-Driven Evaluation in the New Millennium. In S. I. Donaldson and M. Scriven (Eds.). Evaluating Social Programs and Problems: Visions for the New Millennium. Mahwah, NJ: Erlbaum. • Donaldson, S. I., & Lipsey, M. (2006). Roles for theory in contemporary evaluation practice: Developing practical knowledge. In. I. Shaw, J. C. Greene, and M. H. Mark (Eds.). The SAGE Handbook of Evaluation, Thousand Oaks, CA: Sage. • Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines (3rd Ed.). Boston, MA: Pearson. • Lipsey, M. (1988). Practice and malpractice in evaluation research. Evaluation Practice, 8(4), 5 – 24. 50
  • 51. References • Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A Systematic Approach (7th Ed.). Thousand Oaks, CA: Sage. • Shadish, W. R., Cook, T., & Leviton, L. (1991). Foundations of Program Evaluation: Theories of Practice. Newbury Park, CA: Sage. • Weiss, C. H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501 – 524. • Weiss, C. H. (1998). Evaluation: Methods for Studying Programs and Policies (2nd Ed.). Upper Saddle River, NJ: Prentice Hall. 51