SlideShare a Scribd company logo
1 of 24
Download to read offline
Evaluation Design

Curriculum Evaluation (EDU 5352)
      Mina Badiei (GS31016)
What Is Evaluation Design?
• The plan for an evaluation project is called a
  "design“.
• It is a particularly vital step to provide an
  appropriate assessment.
• A good design offers an opportunity to
  maximize the quality of the evaluation, helps
  minimize and justify the time and cost
  necessary to perform the work.
Design Process
• 1. Identifying evaluation questions and issues
• 2. Identifying research designs and
  comparisons
• 3. Sampling methods
• 4. Data collection instruments
• 5. Collecting and coding qualitative data
Evaluation Design Approaches


                             Mixed-
Quantitative   Qualitative
                             Method
Quantitative approach
• Quantitative data can be counted, measured, and
  reported in numerical form and answer questions
  such as who, what, where, and how much.
• The quantitative approach is useful for describing
  concrete phenomena and for statistically
  analyzing results.
• Data collection instruments can be used with
  large numbers of study participants.
• Data     collection    instruments      can     be
  standardized, allowing for easy comparison
  within and across studies.
• Experimental designs tend to be rigorous in that they control
                      for external factors and enable you to argue, with some
                      degree of confidence, that your findings are due to the
  Experimental        effects of the program rather than other, unrelated, factors.
                    • They are rarely applicable in educational settings where
                      there is a chance that students may be denied an opportunity
                      to participate in a program because of the evaluation design.


                    • Quasi-experimental designs are those in which participants
                      are matched beforehand, or after the fact, using statistical
     Quasi-           methods
                    • These studies offer a reasonable solution for schools or
  Experimental        districts that cannot randomly assign students to different
                      programs, but still desire some degree of control so that they
                      can make statistical statements about their findings.



                    • it is intended to demonstrate trends or changes over time.
Time-series Study   • the purpose of the design is not to examine the impact of an
                      intervention ,but simply to explore and describe changes in
                      the construct of interest.
                    • Time series offer more data points, but because there is little
                      control over extraneous factors.
• it is intended to show a snapshot in time.
                  • it might be used to answer questions like:
Cross-sectional   • -what do parents think about our school?
                  • -what do parents see as the strengths and
                    weaknesses the school environment?




                  • case studies are those which seek to follow
                    program implementation or impact on an
                    individual, group, or organization, such as a
                    school or classroom.
Case-studies      • However, case studies are an excellent way to
                    collect evidence of program effectiveness, to
                    increase understanding of how an intervention is
                    working in particular settings, and to inform a
                    larger study to be conducted later.
Types of Experimental Design
Post-test only design              Pre-post design
•   the least complicated of the   • it is employed when a
    experimental design.
                                     pretreatment measure can
• it has three steps:
1) decide what comparisons are       supply useful information.
desired and meaningful.            • in Field-trial stage this
2) the students in two or more       design in common to use.
comparison groups must be
similar.
3) collect the information after
the posttest to determine
whether the differences
occurred.
Example:
• Equilibrium Effects of Education Policies: A Quantitative
  Evaluation
By : Giovanni Gallipoli, Costas Meghir, Giovanni L. Violante
The paper compares partial and general equilibrium effects of alternative education
policies on the distribution of education and earnings. The numerical counterpart of
the model, parameterized through a variety of data sources, education enrollment
responses which are broadly in line with reduce-form estimates. Through numerical
simulations, they compare the effects of alternative policy interventions on optimal
education decisions, inequality, and output. It’s a king of Quasi-Experimental design.
Qualitative Approach
• Qualitative data are reported in narrative form.
• Qualitative approach can provide important insights into how well a
  program is working and what can be done to increase its impact.
• Qualitative data can also provide information about how
  participants – including the people responsible for operating the
  program as well as the target audience – feel about the program.
• It promotes understanding of diverse stakeholder perspectives
  (e.g., what the program means to different people).
• Stakeholders, funders, policymakers, and the public may find
  quotes and anecdotes easier to understand and more appealing
  than statistical data.
Observation


Interview


Focus Groups


Document Studies


Key-informants
observation
• Observational techniques are methods by which an individual or
  individuals gather firsthand data on programs, processes, or
  behaviors being studied.
• They provide evaluators with an opportunity to collect data on a
  wide range of behaviors, to capture a great variety of
  interactions, and to openly explore the evaluation topic.
• By directly observing operations and activities, the evaluator can
  develop a holistic perspective, i.e., an understanding of the context
  within which the project operates.
• Observational approaches also allow the evaluator to learn about
  things the participants or staff may be unaware of or that they are
  unwilling or unable to discuss in an interview or focus group.
When to use observations
•  Observations can be useful during both the formative and summative
  phases of evaluation. For example, during the formative
  phase, observations can be useful in determining whether or not the
  project is being delivered and operated as planned.
• In the hypothetical project, observations could be used to describe the
  faculty development sessions, examining the extent to which participants
  understand the concepts, ask the right questions, and are engaged in
  appropriate interactions.
• Observations during the summative phase of evaluation can be used to
  determine whether or not the project is successful. The technique would
  be especially useful in directly examining teaching methods employed by
  the faculty in their own classes after program participation.
Interviews

• Interviews provide very different data from observations: they
  allow the evaluation team to capture the perspectives of project
  participants, staff, and others associated with the project.
• In the hypothetical example, interviews with project staff can
  provide information on the early stages of the implementation and
  problems encountered.
• An interview, rather than a paper and pencil survey, is selected
  when interpersonal contact is important and when opportunities
  for follow up of interesting comments are desired.
• Two types of interviews are used in evaluation research: structured
  interviews, in which a carefully worded questionnaire is
  administered; and in depth interviews, in which the interviewer
  does not follow a rigid form.
Contd..
structured interviews:              In-depth interviews:
                                    • the interviewers seek to encourage
• the emphasis is on obtaining          free and open responses, and
  answers to carefully phrased          there may be a trade off between
  questions.                            comprehensive coverage of topics
• Interviewers are trained to           and in depth and limited set of
  deviate only minimally from the       questions.
  question wording to ensure        • In depth interviews also encourage
  uniformity of interview               capturing of respondents’
  administration.                       perceptions in their own words.
                                        This allows the evaluator to
                                        present the meaningfulness of the
                                        experience from the respondent’s
                                        perspective.
                                    • In depth interviews are conducted
                                        with individuals or with a small
                                        group of individuals.
When to use interviews
• interviews can be used at any stage of the evaluation process. They
  are especially useful in answering questions such as those
  suggested by Patton (1990):
• What does the program look and feel like to the participants? To
  other stakeholders?
• What are the experiences of program participants?
• What do stakeholders know about the project?
• What thoughts do stakeholders knowledgeable about the program
  have concerning program operations, processes, and outcomes?
• What are participants’ and stakeholders’ expectations?
• What features of the project are most salient to the participants?
• What changes do participants perceive in themselves as a result of
  their involvement in the project?
Focus Groups

• Focus groups combine elements of both interviewing and
  participant observation.
• The focus group session is, indeed, an interview not a
  discussion group, problem-solving session, or decision-
  making group. (Patton, 1990)
• The hallmark of focus groups is the explicit use of the
  group interaction to generate data and insights that would
  be unlikely to emerge without the interaction found in a
  group.
• Focus groups are a gathering of 8 to 12 people who share
  some characteristics relevant to the evaluation. Originally
  used as a market research tool to investigate the appeal of
  various products.
Contd.
• the focus group technique has been adopted by
  other fields, such as education, as a tool for data
  gathering on a given topic.
• It conducted by experts take place in a focus
  group facility that includes recording apparatus
  (audio and/or visual) and an attached room with
  a one-way mirror for observation. There is an
  official recorder who may or may not be in the
  room.
• Participants are paid for attendance and
  provided with refreshments.
When to use focus groups
• When conducting evaluations, focus groups are useful
  in answering the same type of questions as in-depth
  interviews, except in a social context.
• Specific applications of the focus group method in
  evaluations include :
    - identifying and defining problems in project
implementation;
    -identifying project strengths, weaknesses, and
recommendations;
   -assisting with interpretation of quantitative findings;
   -obtaining perceptions of project outcomes and
impacts; and generating new ideas.
Other Qualitative Methods

•   Document Studies: defined a document as "any written or recorded material"
    not prepared for the purposes of the evaluation or at the request of the
    inquirer. Documents can be divided into two major categories: public
    records, and personal documents (Guba and Lincoln, 1981).
•   Key Informant :A key informant is a person (or group of persons) who has
    unique skills or professional background related to the issue/intervention
    being evaluated, is knowledgeable about the project participants, or has
    access to other information of interest to the evaluator.
•   Key informants can help the evaluation team better understand the issue
    being evaluated, as well as the project participants, their
    backgrounds, behaviors, and attitudes, and any language or ethnic
    considerations. They can offer expertise beyond the evaluation team. They are
    also very useful for assisting with the evaluation of curricula and other
    educational materials. Key informants can be surveyed or interviewed
    individually or through focus groups.
Example:
• A Qualitative Evaluation Process for Educational Programs
  Serving Handicapped Students in Rural Areas
By : LUCILLE ANNESEZEPH
The paper describes a qualitative methodology designed to evaluate special education programs
in rural areas serving students with severe special needs. A rationale is provided for the use of
the elements of aesthetic criticism as the basis of methodology, and specific descriptions of the
steps for its implementation and validation are provided. Some practical limitations and particular
areas of usefulness are also discussed.
Mix method
• In recent years evaluators of educational and social programs have
  expanded their methodological repertoire with designs that include
  the use of both qualitative and quantitative methods. Such practice,
  however, needs to be grounded in a theory that can meaningfully
  guide the design and implementation of mixed-method evaluations.
• In many cases a mixture of designs can work together as a design
  for evaluating a large, complex program.
• The ideal evaluation combines quantitative and qualitative
  methods. A mixed-method approach offers a range of perspectives
  on a program's processes and outcomes.
• For example, the impact of a reading intervention on student
  performance may be compared for all students in a school over a
  period of time using repeated measures from exams administered
  for this purpose, but could also include more focused case studies
  of particular classes to learn about crucial implementation issues.
benefits
• It increases the validity of your findings by allowing you to
  examine the same phenomenon in different ways.
• It can result in better data collection instruments. For
  example, focus groups can be invaluable in the
  development or selection of a questionnaire used to gather
  quantitative data.
• It promotes greater understanding of the findings.
  Quantitative data can show that change occurred and how
  much change took place, while qualitative data can help
  you and others understand what happened and why.
• It offers something for everyone. Some stakeholders may
  respond more favorably to a presentation featuring charts
  and graphs. Others may prefer anecdotes and stories.
Example:
• A Mixed Methods Evaluation of a 12-Week Insurance-Sponsored
  Weight Management Program Incorporating Cognitive–Behavioral
  Counseling
By:Christiaan Abildso , Sam Zizzi, Diana Gilleland, James Thomas, and Daniel Bonner
A sequential mixed methods approach was used to assess the physical
and psychosocial impact of a 12-week cognitive–behavioral weight
management program and explore factors associated with weight loss.
Quantitative data revealed program completion rate and mean
percentage weight loss that compare favorably with other
interventions, and differential psychosocial impacts on those losing
more weight. Telephone interviews revealed four potential
mechanisms for these differential impacts: (a) fostering
accountability, (b) balancing perceived effort and success, (c)
redefining ‘‘success,’’ and (d) developing cognitive flexibility.

More Related Content

What's hot

Evaluation Principles: Theory-Based, Utilization-Focused, Participatory
Evaluation Principles: Theory-Based, Utilization-Focused, ParticipatoryEvaluation Principles: Theory-Based, Utilization-Focused, Participatory
Evaluation Principles: Theory-Based, Utilization-Focused, ParticipatoryMHTP Webmastere
 
The Theory of Change Approach
The Theory of Change ApproachThe Theory of Change Approach
The Theory of Change ApproachEric Graig
 
Qualitative research
Qualitative researchQualitative research
Qualitative researchAbhinav Kp
 
Program Evaluation: Forms and Approaches by Helen A. Casimiro
Program Evaluation: Forms and Approaches by Helen A. CasimiroProgram Evaluation: Forms and Approaches by Helen A. Casimiro
Program Evaluation: Forms and Approaches by Helen A. CasimiroHelen Casimiro
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluationSudipta Barman
 
Impact evaluation
Impact evaluationImpact evaluation
Impact evaluationCarlo Magno
 
M& e slide share
M& e   slide shareM& e   slide share
M& e slide shareSelf
 
Experimental research
Experimental researchExperimental research
Experimental researchizzajalil
 
Qualitative research methodology 1
Qualitative research methodology 1Qualitative research methodology 1
Qualitative research methodology 1KaleemSarwar2
 
what is a needs assessment , How to write a needs assessment
 what is a needs assessment , How to write a needs assessment what is a needs assessment , How to write a needs assessment
what is a needs assessment , How to write a needs assessmentNeveenJamal
 
Monitoring And Evaluation Presentation
Monitoring And Evaluation PresentationMonitoring And Evaluation Presentation
Monitoring And Evaluation PresentationEquiGov Institute
 
Monitoring and evaluation Learning and Development
Monitoring and evaluation Learning and DevelopmentMonitoring and evaluation Learning and Development
Monitoring and evaluation Learning and DevelopmentSESH SUKHDEO
 
Correlational research
Correlational researchCorrelational research
Correlational researchAzura Zaki
 
Participatory Action Research
Participatory Action ResearchParticipatory Action Research
Participatory Action ResearchReynante Tagum
 

What's hot (20)

Evaluation Principles: Theory-Based, Utilization-Focused, Participatory
Evaluation Principles: Theory-Based, Utilization-Focused, ParticipatoryEvaluation Principles: Theory-Based, Utilization-Focused, Participatory
Evaluation Principles: Theory-Based, Utilization-Focused, Participatory
 
Triangulation
TriangulationTriangulation
Triangulation
 
The Theory of Change Approach
The Theory of Change ApproachThe Theory of Change Approach
The Theory of Change Approach
 
ACTION RESEARCH
ACTION RESEARCHACTION RESEARCH
ACTION RESEARCH
 
Qualitative research
Qualitative researchQualitative research
Qualitative research
 
Triangulation research
Triangulation researchTriangulation research
Triangulation research
 
Program Evaluation: Forms and Approaches by Helen A. Casimiro
Program Evaluation: Forms and Approaches by Helen A. CasimiroProgram Evaluation: Forms and Approaches by Helen A. Casimiro
Program Evaluation: Forms and Approaches by Helen A. Casimiro
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluation
 
Impact evaluation
Impact evaluationImpact evaluation
Impact evaluation
 
Some Challenges of M&E
Some Challenges of M&ESome Challenges of M&E
Some Challenges of M&E
 
M& e slide share
M& e   slide shareM& e   slide share
M& e slide share
 
Key informant interview
Key informant interviewKey informant interview
Key informant interview
 
Experimental research
Experimental researchExperimental research
Experimental research
 
Qualitative research methodology 1
Qualitative research methodology 1Qualitative research methodology 1
Qualitative research methodology 1
 
what is a needs assessment , How to write a needs assessment
 what is a needs assessment , How to write a needs assessment what is a needs assessment , How to write a needs assessment
what is a needs assessment , How to write a needs assessment
 
Monitoring And Evaluation Presentation
Monitoring And Evaluation PresentationMonitoring And Evaluation Presentation
Monitoring And Evaluation Presentation
 
Triangulation
TriangulationTriangulation
Triangulation
 
Monitoring and evaluation Learning and Development
Monitoring and evaluation Learning and DevelopmentMonitoring and evaluation Learning and Development
Monitoring and evaluation Learning and Development
 
Correlational research
Correlational researchCorrelational research
Correlational research
 
Participatory Action Research
Participatory Action ResearchParticipatory Action Research
Participatory Action Research
 

Similar to Evaluation design

Research- 2 Selecting a Study Design.pptx
Research- 2 Selecting a Study Design.pptxResearch- 2 Selecting a Study Design.pptx
Research- 2 Selecting a Study Design.pptxAttiqUrRehmanMemon
 
Data collection instruments (2)
Data collection instruments (2)Data collection instruments (2)
Data collection instruments (2)pesantez666
 
Data Collection Instruments.
Data Collection Instruments.Data Collection Instruments.
Data Collection Instruments.Auver2012
 
UNIT 1 Business Research Method by Dr. Rashmi Maini-1.pptx
UNIT 1 Business Research Method by Dr. Rashmi Maini-1.pptxUNIT 1 Business Research Method by Dr. Rashmi Maini-1.pptx
UNIT 1 Business Research Method by Dr. Rashmi Maini-1.pptxHarshkumarSingh74
 
Specific techniques of curriculum evaluation
Specific techniques of curriculum evaluationSpecific techniques of curriculum evaluation
Specific techniques of curriculum evaluationCynthia Calupas
 
Tools of evaluation ppt
Tools of evaluation pptTools of evaluation ppt
Tools of evaluation pptkitttu singh
 
Chapter 8.pptx
Chapter 8.pptxChapter 8.pptx
Chapter 8.pptxMUSEAWALE1
 
Rsi 26 5-solooooooo
Rsi 26 5-soloooooooRsi 26 5-solooooooo
Rsi 26 5-soloooooooDilshad Shah
 
Classroom Based Assessment Tools and Techniques 27-09-2022.ppt
Classroom Based Assessment Tools and Techniques 27-09-2022.pptClassroom Based Assessment Tools and Techniques 27-09-2022.ppt
Classroom Based Assessment Tools and Techniques 27-09-2022.pptNasirMahmood976516
 
Barbara dale jones presentation-frf event_17032015_final
Barbara dale jones presentation-frf event_17032015_finalBarbara dale jones presentation-frf event_17032015_final
Barbara dale jones presentation-frf event_17032015_finalbarbaradj02
 
Keys to success with assessment and evaluation
Keys to success with assessment and evaluationKeys to success with assessment and evaluation
Keys to success with assessment and evaluationFrank Cervone
 
Managing Quality In Qualitative Research
Managing Quality In Qualitative ResearchManaging Quality In Qualitative Research
Managing Quality In Qualitative ResearchMike Crabb
 
Monitoring and evaluation2
Monitoring and evaluation2 Monitoring and evaluation2
Monitoring and evaluation2 PCPD Palestine
 
Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Shayne Botor
 
Qualitative Research Methods
Qualitative Research MethodsQualitative Research Methods
Qualitative Research MethodsAnil Sharma
 
Evaluation the many faces
Evaluation   the many facesEvaluation   the many faces
Evaluation the many facesleesha roberts
 
Mass media research unit 2
Mass media research unit 2 Mass media research unit 2
Mass media research unit 2 Shubhangi Mishra
 
research design (B.COM).pptx
research design (B.COM).pptxresearch design (B.COM).pptx
research design (B.COM).pptxSINUSARJA
 

Similar to Evaluation design (20)

Research- 2 Selecting a Study Design.pptx
Research- 2 Selecting a Study Design.pptxResearch- 2 Selecting a Study Design.pptx
Research- 2 Selecting a Study Design.pptx
 
Data collection instruments (2)
Data collection instruments (2)Data collection instruments (2)
Data collection instruments (2)
 
Data Collection Instruments.
Data Collection Instruments.Data Collection Instruments.
Data Collection Instruments.
 
UNIT 1 Business Research Method by Dr. Rashmi Maini-1.pptx
UNIT 1 Business Research Method by Dr. Rashmi Maini-1.pptxUNIT 1 Business Research Method by Dr. Rashmi Maini-1.pptx
UNIT 1 Business Research Method by Dr. Rashmi Maini-1.pptx
 
Specific techniques of curriculum evaluation
Specific techniques of curriculum evaluationSpecific techniques of curriculum evaluation
Specific techniques of curriculum evaluation
 
Pilot study-research
Pilot study-researchPilot study-research
Pilot study-research
 
Tools of evaluation ppt
Tools of evaluation pptTools of evaluation ppt
Tools of evaluation ppt
 
Chapter 8.pptx
Chapter 8.pptxChapter 8.pptx
Chapter 8.pptx
 
Rsi 26 5-solooooooo
Rsi 26 5-soloooooooRsi 26 5-solooooooo
Rsi 26 5-solooooooo
 
Classroom Based Assessment Tools and Techniques 27-09-2022.ppt
Classroom Based Assessment Tools and Techniques 27-09-2022.pptClassroom Based Assessment Tools and Techniques 27-09-2022.ppt
Classroom Based Assessment Tools and Techniques 27-09-2022.ppt
 
Barbara dale jones presentation-frf event_17032015_final
Barbara dale jones presentation-frf event_17032015_finalBarbara dale jones presentation-frf event_17032015_final
Barbara dale jones presentation-frf event_17032015_final
 
Keys to success with assessment and evaluation
Keys to success with assessment and evaluationKeys to success with assessment and evaluation
Keys to success with assessment and evaluation
 
Managing Quality In Qualitative Research
Managing Quality In Qualitative ResearchManaging Quality In Qualitative Research
Managing Quality In Qualitative Research
 
Evaluation methodology of practices of science communication
Evaluation methodology of practices of science communicationEvaluation methodology of practices of science communication
Evaluation methodology of practices of science communication
 
Monitoring and evaluation2
Monitoring and evaluation2 Monitoring and evaluation2
Monitoring and evaluation2
 
Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Botor_project_research_methodology_2016
Botor_project_research_methodology_2016
 
Qualitative Research Methods
Qualitative Research MethodsQualitative Research Methods
Qualitative Research Methods
 
Evaluation the many faces
Evaluation   the many facesEvaluation   the many faces
Evaluation the many faces
 
Mass media research unit 2
Mass media research unit 2 Mass media research unit 2
Mass media research unit 2
 
research design (B.COM).pptx
research design (B.COM).pptxresearch design (B.COM).pptx
research design (B.COM).pptx
 

More from Mina Badiei

Conceptual Framework
Conceptual FrameworkConceptual Framework
Conceptual FrameworkMina Badiei
 
Theory in Education
Theory in EducationTheory in Education
Theory in EducationMina Badiei
 
Theory in education
Theory in educationTheory in education
Theory in educationMina Badiei
 
STAGE OF CURRICULUM DEVELOPMENT AND EVALUATION IN UPDATING THE ENTIRE CURRICULUM
STAGE OF CURRICULUM DEVELOPMENT AND EVALUATION IN UPDATING THE ENTIRE CURRICULUMSTAGE OF CURRICULUM DEVELOPMENT AND EVALUATION IN UPDATING THE ENTIRE CURRICULUM
STAGE OF CURRICULUM DEVELOPMENT AND EVALUATION IN UPDATING THE ENTIRE CURRICULUMMina Badiei
 
curriculum inquiry
curriculum inquirycurriculum inquiry
curriculum inquiryMina Badiei
 
Bases of curriculum planning part 1
Bases of curriculum planning part 1Bases of curriculum planning part 1
Bases of curriculum planning part 1Mina Badiei
 
Innovation in instruction
Innovation in instruction  Innovation in instruction
Innovation in instruction Mina Badiei
 

More from Mina Badiei (8)

Conceptual Framework
Conceptual FrameworkConceptual Framework
Conceptual Framework
 
Lecture 5
Lecture 5Lecture 5
Lecture 5
 
Theory in Education
Theory in EducationTheory in Education
Theory in Education
 
Theory in education
Theory in educationTheory in education
Theory in education
 
STAGE OF CURRICULUM DEVELOPMENT AND EVALUATION IN UPDATING THE ENTIRE CURRICULUM
STAGE OF CURRICULUM DEVELOPMENT AND EVALUATION IN UPDATING THE ENTIRE CURRICULUMSTAGE OF CURRICULUM DEVELOPMENT AND EVALUATION IN UPDATING THE ENTIRE CURRICULUM
STAGE OF CURRICULUM DEVELOPMENT AND EVALUATION IN UPDATING THE ENTIRE CURRICULUM
 
curriculum inquiry
curriculum inquirycurriculum inquiry
curriculum inquiry
 
Bases of curriculum planning part 1
Bases of curriculum planning part 1Bases of curriculum planning part 1
Bases of curriculum planning part 1
 
Innovation in instruction
Innovation in instruction  Innovation in instruction
Innovation in instruction
 

Recently uploaded

LEAD6001 - Introduction to Advanced Stud
LEAD6001 - Introduction to Advanced StudLEAD6001 - Introduction to Advanced Stud
LEAD6001 - Introduction to Advanced StudDr. Bruce A. Johnson
 
UNIT I Design Thinking and Explore.pptx
UNIT I  Design Thinking and Explore.pptxUNIT I  Design Thinking and Explore.pptx
UNIT I Design Thinking and Explore.pptxGOWSIKRAJA PALANISAMY
 
POST ENCEPHALITIS case study Jitendra bhargav
POST ENCEPHALITIS case study  Jitendra bhargavPOST ENCEPHALITIS case study  Jitendra bhargav
POST ENCEPHALITIS case study Jitendra bhargavJitendra Bhargav
 
EDD8524 The Future of Educational Leader
EDD8524 The Future of Educational LeaderEDD8524 The Future of Educational Leader
EDD8524 The Future of Educational LeaderDr. Bruce A. Johnson
 
ASTRINGENTS.pdf Pharmacognosy chapter 5 diploma in Pharmacy
ASTRINGENTS.pdf Pharmacognosy chapter 5 diploma in PharmacyASTRINGENTS.pdf Pharmacognosy chapter 5 diploma in Pharmacy
ASTRINGENTS.pdf Pharmacognosy chapter 5 diploma in PharmacySumit Tiwari
 
3.14.24 Gender Discrimination and Gender Inequity.pptx
3.14.24 Gender Discrimination and Gender Inequity.pptx3.14.24 Gender Discrimination and Gender Inequity.pptx
3.14.24 Gender Discrimination and Gender Inequity.pptxmary850239
 
BBA 205 BE UNIT 2 economic systems prof dr kanchan.pptx
BBA 205 BE UNIT 2 economic systems prof dr kanchan.pptxBBA 205 BE UNIT 2 economic systems prof dr kanchan.pptx
BBA 205 BE UNIT 2 economic systems prof dr kanchan.pptxProf. Kanchan Kumari
 
Metabolism , Metabolic Fate& disorders of cholesterol.pptx
Metabolism , Metabolic Fate& disorders of cholesterol.pptxMetabolism , Metabolic Fate& disorders of cholesterol.pptx
Metabolism , Metabolic Fate& disorders of cholesterol.pptxDr. Santhosh Kumar. N
 
Pharmacology chapter No 7 full notes.pdf
Pharmacology chapter No 7 full notes.pdfPharmacology chapter No 7 full notes.pdf
Pharmacology chapter No 7 full notes.pdfSumit Tiwari
 
Quantitative research methodology and survey design
Quantitative research methodology and survey designQuantitative research methodology and survey design
Quantitative research methodology and survey designBalelaBoru
 
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...gdgsurrey
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection
 
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...Nguyen Thanh Tu Collection
 
DLL Catch Up Friday March 22.docx CATCH UP FRIDAYS
DLL Catch Up Friday March 22.docx CATCH UP FRIDAYSDLL Catch Up Friday March 22.docx CATCH UP FRIDAYS
DLL Catch Up Friday March 22.docx CATCH UP FRIDAYSTeacherNicaPrintable
 
Arti Languages Pre Seed Send Ahead Pitchdeck 2024.pdf
Arti Languages Pre Seed Send Ahead Pitchdeck 2024.pdfArti Languages Pre Seed Send Ahead Pitchdeck 2024.pdf
Arti Languages Pre Seed Send Ahead Pitchdeck 2024.pdfwill854175
 
Plant Tissue culture., Plasticity, Totipotency, pptx
Plant Tissue culture., Plasticity, Totipotency, pptxPlant Tissue culture., Plasticity, Totipotency, pptx
Plant Tissue culture., Plasticity, Totipotency, pptxHimansu10
 
Material Remains as Source of Ancient Indian History & Culture.ppt
Material Remains as Source of Ancient Indian History & Culture.pptMaterial Remains as Source of Ancient Indian History & Culture.ppt
Material Remains as Source of Ancient Indian History & Culture.pptBanaras Hindu University
 
LEAD5623 The Economics of Community Coll
LEAD5623 The Economics of Community CollLEAD5623 The Economics of Community Coll
LEAD5623 The Economics of Community CollDr. Bruce A. Johnson
 
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptxAUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptxiammrhaywood
 
BBA 205 BUSINESS ENVIRONMENT UNIT I.pptx
BBA 205 BUSINESS ENVIRONMENT UNIT I.pptxBBA 205 BUSINESS ENVIRONMENT UNIT I.pptx
BBA 205 BUSINESS ENVIRONMENT UNIT I.pptxProf. Kanchan Kumari
 

Recently uploaded (20)

LEAD6001 - Introduction to Advanced Stud
LEAD6001 - Introduction to Advanced StudLEAD6001 - Introduction to Advanced Stud
LEAD6001 - Introduction to Advanced Stud
 
UNIT I Design Thinking and Explore.pptx
UNIT I  Design Thinking and Explore.pptxUNIT I  Design Thinking and Explore.pptx
UNIT I Design Thinking and Explore.pptx
 
POST ENCEPHALITIS case study Jitendra bhargav
POST ENCEPHALITIS case study  Jitendra bhargavPOST ENCEPHALITIS case study  Jitendra bhargav
POST ENCEPHALITIS case study Jitendra bhargav
 
EDD8524 The Future of Educational Leader
EDD8524 The Future of Educational LeaderEDD8524 The Future of Educational Leader
EDD8524 The Future of Educational Leader
 
ASTRINGENTS.pdf Pharmacognosy chapter 5 diploma in Pharmacy
ASTRINGENTS.pdf Pharmacognosy chapter 5 diploma in PharmacyASTRINGENTS.pdf Pharmacognosy chapter 5 diploma in Pharmacy
ASTRINGENTS.pdf Pharmacognosy chapter 5 diploma in Pharmacy
 
3.14.24 Gender Discrimination and Gender Inequity.pptx
3.14.24 Gender Discrimination and Gender Inequity.pptx3.14.24 Gender Discrimination and Gender Inequity.pptx
3.14.24 Gender Discrimination and Gender Inequity.pptx
 
BBA 205 BE UNIT 2 economic systems prof dr kanchan.pptx
BBA 205 BE UNIT 2 economic systems prof dr kanchan.pptxBBA 205 BE UNIT 2 economic systems prof dr kanchan.pptx
BBA 205 BE UNIT 2 economic systems prof dr kanchan.pptx
 
Metabolism , Metabolic Fate& disorders of cholesterol.pptx
Metabolism , Metabolic Fate& disorders of cholesterol.pptxMetabolism , Metabolic Fate& disorders of cholesterol.pptx
Metabolism , Metabolic Fate& disorders of cholesterol.pptx
 
Pharmacology chapter No 7 full notes.pdf
Pharmacology chapter No 7 full notes.pdfPharmacology chapter No 7 full notes.pdf
Pharmacology chapter No 7 full notes.pdf
 
Quantitative research methodology and survey design
Quantitative research methodology and survey designQuantitative research methodology and survey design
Quantitative research methodology and survey design
 
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
 
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
 
DLL Catch Up Friday March 22.docx CATCH UP FRIDAYS
DLL Catch Up Friday March 22.docx CATCH UP FRIDAYSDLL Catch Up Friday March 22.docx CATCH UP FRIDAYS
DLL Catch Up Friday March 22.docx CATCH UP FRIDAYS
 
Arti Languages Pre Seed Send Ahead Pitchdeck 2024.pdf
Arti Languages Pre Seed Send Ahead Pitchdeck 2024.pdfArti Languages Pre Seed Send Ahead Pitchdeck 2024.pdf
Arti Languages Pre Seed Send Ahead Pitchdeck 2024.pdf
 
Plant Tissue culture., Plasticity, Totipotency, pptx
Plant Tissue culture., Plasticity, Totipotency, pptxPlant Tissue culture., Plasticity, Totipotency, pptx
Plant Tissue culture., Plasticity, Totipotency, pptx
 
Material Remains as Source of Ancient Indian History & Culture.ppt
Material Remains as Source of Ancient Indian History & Culture.pptMaterial Remains as Source of Ancient Indian History & Culture.ppt
Material Remains as Source of Ancient Indian History & Culture.ppt
 
LEAD5623 The Economics of Community Coll
LEAD5623 The Economics of Community CollLEAD5623 The Economics of Community Coll
LEAD5623 The Economics of Community Coll
 
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptxAUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
 
BBA 205 BUSINESS ENVIRONMENT UNIT I.pptx
BBA 205 BUSINESS ENVIRONMENT UNIT I.pptxBBA 205 BUSINESS ENVIRONMENT UNIT I.pptx
BBA 205 BUSINESS ENVIRONMENT UNIT I.pptx
 

Evaluation design

  • 1. Evaluation Design Curriculum Evaluation (EDU 5352) Mina Badiei (GS31016)
  • 2. What Is Evaluation Design? • The plan for an evaluation project is called a "design“. • It is a particularly vital step to provide an appropriate assessment. • A good design offers an opportunity to maximize the quality of the evaluation, helps minimize and justify the time and cost necessary to perform the work.
  • 3. Design Process • 1. Identifying evaluation questions and issues • 2. Identifying research designs and comparisons • 3. Sampling methods • 4. Data collection instruments • 5. Collecting and coding qualitative data
  • 4. Evaluation Design Approaches Mixed- Quantitative Qualitative Method
  • 5. Quantitative approach • Quantitative data can be counted, measured, and reported in numerical form and answer questions such as who, what, where, and how much. • The quantitative approach is useful for describing concrete phenomena and for statistically analyzing results. • Data collection instruments can be used with large numbers of study participants. • Data collection instruments can be standardized, allowing for easy comparison within and across studies.
  • 6. • Experimental designs tend to be rigorous in that they control for external factors and enable you to argue, with some degree of confidence, that your findings are due to the Experimental effects of the program rather than other, unrelated, factors. • They are rarely applicable in educational settings where there is a chance that students may be denied an opportunity to participate in a program because of the evaluation design. • Quasi-experimental designs are those in which participants are matched beforehand, or after the fact, using statistical Quasi- methods • These studies offer a reasonable solution for schools or Experimental districts that cannot randomly assign students to different programs, but still desire some degree of control so that they can make statistical statements about their findings. • it is intended to demonstrate trends or changes over time. Time-series Study • the purpose of the design is not to examine the impact of an intervention ,but simply to explore and describe changes in the construct of interest. • Time series offer more data points, but because there is little control over extraneous factors.
  • 7. • it is intended to show a snapshot in time. • it might be used to answer questions like: Cross-sectional • -what do parents think about our school? • -what do parents see as the strengths and weaknesses the school environment? • case studies are those which seek to follow program implementation or impact on an individual, group, or organization, such as a school or classroom. Case-studies • However, case studies are an excellent way to collect evidence of program effectiveness, to increase understanding of how an intervention is working in particular settings, and to inform a larger study to be conducted later.
  • 8. Types of Experimental Design Post-test only design Pre-post design • the least complicated of the • it is employed when a experimental design. pretreatment measure can • it has three steps: 1) decide what comparisons are supply useful information. desired and meaningful. • in Field-trial stage this 2) the students in two or more design in common to use. comparison groups must be similar. 3) collect the information after the posttest to determine whether the differences occurred.
  • 9. Example: • Equilibrium Effects of Education Policies: A Quantitative Evaluation By : Giovanni Gallipoli, Costas Meghir, Giovanni L. Violante The paper compares partial and general equilibrium effects of alternative education policies on the distribution of education and earnings. The numerical counterpart of the model, parameterized through a variety of data sources, education enrollment responses which are broadly in line with reduce-form estimates. Through numerical simulations, they compare the effects of alternative policy interventions on optimal education decisions, inequality, and output. It’s a king of Quasi-Experimental design.
  • 10. Qualitative Approach • Qualitative data are reported in narrative form. • Qualitative approach can provide important insights into how well a program is working and what can be done to increase its impact. • Qualitative data can also provide information about how participants – including the people responsible for operating the program as well as the target audience – feel about the program. • It promotes understanding of diverse stakeholder perspectives (e.g., what the program means to different people). • Stakeholders, funders, policymakers, and the public may find quotes and anecdotes easier to understand and more appealing than statistical data.
  • 12. observation • Observational techniques are methods by which an individual or individuals gather firsthand data on programs, processes, or behaviors being studied. • They provide evaluators with an opportunity to collect data on a wide range of behaviors, to capture a great variety of interactions, and to openly explore the evaluation topic. • By directly observing operations and activities, the evaluator can develop a holistic perspective, i.e., an understanding of the context within which the project operates. • Observational approaches also allow the evaluator to learn about things the participants or staff may be unaware of or that they are unwilling or unable to discuss in an interview or focus group.
  • 13. When to use observations • Observations can be useful during both the formative and summative phases of evaluation. For example, during the formative phase, observations can be useful in determining whether or not the project is being delivered and operated as planned. • In the hypothetical project, observations could be used to describe the faculty development sessions, examining the extent to which participants understand the concepts, ask the right questions, and are engaged in appropriate interactions. • Observations during the summative phase of evaluation can be used to determine whether or not the project is successful. The technique would be especially useful in directly examining teaching methods employed by the faculty in their own classes after program participation.
  • 14. Interviews • Interviews provide very different data from observations: they allow the evaluation team to capture the perspectives of project participants, staff, and others associated with the project. • In the hypothetical example, interviews with project staff can provide information on the early stages of the implementation and problems encountered. • An interview, rather than a paper and pencil survey, is selected when interpersonal contact is important and when opportunities for follow up of interesting comments are desired. • Two types of interviews are used in evaluation research: structured interviews, in which a carefully worded questionnaire is administered; and in depth interviews, in which the interviewer does not follow a rigid form.
  • 15. Contd.. structured interviews: In-depth interviews: • the interviewers seek to encourage • the emphasis is on obtaining free and open responses, and answers to carefully phrased there may be a trade off between questions. comprehensive coverage of topics • Interviewers are trained to and in depth and limited set of deviate only minimally from the questions. question wording to ensure • In depth interviews also encourage uniformity of interview capturing of respondents’ administration. perceptions in their own words. This allows the evaluator to present the meaningfulness of the experience from the respondent’s perspective. • In depth interviews are conducted with individuals or with a small group of individuals.
  • 16. When to use interviews • interviews can be used at any stage of the evaluation process. They are especially useful in answering questions such as those suggested by Patton (1990): • What does the program look and feel like to the participants? To other stakeholders? • What are the experiences of program participants? • What do stakeholders know about the project? • What thoughts do stakeholders knowledgeable about the program have concerning program operations, processes, and outcomes? • What are participants’ and stakeholders’ expectations? • What features of the project are most salient to the participants? • What changes do participants perceive in themselves as a result of their involvement in the project?
  • 17. Focus Groups • Focus groups combine elements of both interviewing and participant observation. • The focus group session is, indeed, an interview not a discussion group, problem-solving session, or decision- making group. (Patton, 1990) • The hallmark of focus groups is the explicit use of the group interaction to generate data and insights that would be unlikely to emerge without the interaction found in a group. • Focus groups are a gathering of 8 to 12 people who share some characteristics relevant to the evaluation. Originally used as a market research tool to investigate the appeal of various products.
  • 18. Contd. • the focus group technique has been adopted by other fields, such as education, as a tool for data gathering on a given topic. • It conducted by experts take place in a focus group facility that includes recording apparatus (audio and/or visual) and an attached room with a one-way mirror for observation. There is an official recorder who may or may not be in the room. • Participants are paid for attendance and provided with refreshments.
  • 19. When to use focus groups • When conducting evaluations, focus groups are useful in answering the same type of questions as in-depth interviews, except in a social context. • Specific applications of the focus group method in evaluations include : - identifying and defining problems in project implementation; -identifying project strengths, weaknesses, and recommendations; -assisting with interpretation of quantitative findings; -obtaining perceptions of project outcomes and impacts; and generating new ideas.
  • 20. Other Qualitative Methods • Document Studies: defined a document as "any written or recorded material" not prepared for the purposes of the evaluation or at the request of the inquirer. Documents can be divided into two major categories: public records, and personal documents (Guba and Lincoln, 1981). • Key Informant :A key informant is a person (or group of persons) who has unique skills or professional background related to the issue/intervention being evaluated, is knowledgeable about the project participants, or has access to other information of interest to the evaluator. • Key informants can help the evaluation team better understand the issue being evaluated, as well as the project participants, their backgrounds, behaviors, and attitudes, and any language or ethnic considerations. They can offer expertise beyond the evaluation team. They are also very useful for assisting with the evaluation of curricula and other educational materials. Key informants can be surveyed or interviewed individually or through focus groups.
  • 21. Example: • A Qualitative Evaluation Process for Educational Programs Serving Handicapped Students in Rural Areas By : LUCILLE ANNESEZEPH The paper describes a qualitative methodology designed to evaluate special education programs in rural areas serving students with severe special needs. A rationale is provided for the use of the elements of aesthetic criticism as the basis of methodology, and specific descriptions of the steps for its implementation and validation are provided. Some practical limitations and particular areas of usefulness are also discussed.
  • 22. Mix method • In recent years evaluators of educational and social programs have expanded their methodological repertoire with designs that include the use of both qualitative and quantitative methods. Such practice, however, needs to be grounded in a theory that can meaningfully guide the design and implementation of mixed-method evaluations. • In many cases a mixture of designs can work together as a design for evaluating a large, complex program. • The ideal evaluation combines quantitative and qualitative methods. A mixed-method approach offers a range of perspectives on a program's processes and outcomes. • For example, the impact of a reading intervention on student performance may be compared for all students in a school over a period of time using repeated measures from exams administered for this purpose, but could also include more focused case studies of particular classes to learn about crucial implementation issues.
  • 23. benefits • It increases the validity of your findings by allowing you to examine the same phenomenon in different ways. • It can result in better data collection instruments. For example, focus groups can be invaluable in the development or selection of a questionnaire used to gather quantitative data. • It promotes greater understanding of the findings. Quantitative data can show that change occurred and how much change took place, while qualitative data can help you and others understand what happened and why. • It offers something for everyone. Some stakeholders may respond more favorably to a presentation featuring charts and graphs. Others may prefer anecdotes and stories.
  • 24. Example: • A Mixed Methods Evaluation of a 12-Week Insurance-Sponsored Weight Management Program Incorporating Cognitive–Behavioral Counseling By:Christiaan Abildso , Sam Zizzi, Diana Gilleland, James Thomas, and Daniel Bonner A sequential mixed methods approach was used to assess the physical and psychosocial impact of a 12-week cognitive–behavioral weight management program and explore factors associated with weight loss. Quantitative data revealed program completion rate and mean percentage weight loss that compare favorably with other interventions, and differential psychosocial impacts on those losing more weight. Telephone interviews revealed four potential mechanisms for these differential impacts: (a) fostering accountability, (b) balancing perceived effort and success, (c) redefining ‘‘success,’’ and (d) developing cognitive flexibility.

Editor's Notes

  1. The use of interviews as a data collection method begins with the assumption that the participants’ perspectives are meaningful, knowable, and able to be made explicit, and that their perspectives affect the success of the project.
  2. The technique inherently allows observation of group dynamics, discussion, and firsthand insights into the respondents’ behaviors, attitudes, language, etc.