SlideShare a Scribd company logo
1 of 30
REFLECTIVE PRACTICE, COLLABORATION, AND 
STAKEHOLDER COMMUNICATION: 
WHERE DOES THE FIELD OF EVALUATION STAND? 
Tiffany L. Smith, University of Wisconsin, Stout 
Gary J. Skolits, University of Tennessee, Knoxville 
Patrick B. Barlow, University of Iowa 
EVALUATION 2014
IN THIS PRESENTATION 
• What is Reflective Practice? 
• Statement of the Problem 
• Study Purpose and Research Questions 
• Design and Methodology 
• Participants 
• Findings 
• Conclusions and Recommendations
WHAT IS REFLECTIVE PRACTICE? 
• One of six essential competencies for 
program evaluators identified by 
Stevahn, King, Ghere, & Minnema 
(2005) 
• Highlighted in the Program Evaluation 
Standards (2011) 
• Discussed theoretically and practically 
by a number of evaluation authors
Reflective practice (RP) is critical 
and deliberate inquiry into 
professional practice in order to 
gain a deeper understanding of 
oneself, others, and the meaning 
that is shared among individuals. 
This can happen during practice 
and after the fact, and can either 
be done alone or with others 
(Forrester, 2010; Peters, 1991; 
Schön, 1983).
STATEMENT OF THE PROBLEM 
• Currently, the evaluation community does not know 
evaluators’ opinions, perceptions, or the extent to 
which evaluators use RP in their evaluation work. 
• As the field of evaluation is changing, it is important 
to explore the element of reflection in evaluation 
practice.
STUDY PURPOSE AND RESEARCH QUESTIONS 
The purpose of this study was to determine how 
professional evaluators view RP, the extent and manner in 
which they engage in RP behaviors, and how evaluators 
conceptualize whether RP efforts affect, if at all, the 
evaluation process. 
The following research questions guided this study: 
1. How do professional evaluators conceptualize the notion 
of reflective practice as it relates to their evaluation 
work? 
2. In what ways do professional evaluators engage in 
behaviors associated with reflective practice? 
3. In what ways do evaluators perceive reflective practice 
as having a collaborative element in evaluation?
DESIGN AND METHODOLOGY 
• Qualitative descriptive design through individual 
semi-structured interviews. 
• “Basic” qualitative research, grounded in 
constructionism, where “individuals construct 
reality in interaction with their social worlds” 
(Merriam, 2009, p. 22). 
• Snowball sampling of 19 evaluators with 10 or 
more years of program evaluation experience.
PARTICIPANTS 
• 19 program evaluators with a range of experience 
from 10-49 years (9 had 10-30, 10 had 31+) 
• 11 female, 8 male 
• 9 practitioners, 10 academics 
• Range of different settings for evaluation work, 
some claiming to be “generalists” 
• 15 had doctoral degrees 
• 17 “serendipitous” evaluators
RESULTS 
What did I find?
OVERARCHING THEMES 
Collaboration/Multiple Perspectives 
Learning and Feedback 
Self Awareness and Introspection 
Critical thinking and questioning 
Intuition versus Purpose
RESEARCH QUESTION ONE 
HOW DO PROFESSIONAL EVALUATORS CONCEPTUALIZE THE NOTION OF REFLECTIVE PRACTICE AS IT RELATES TO THEIR 
EVALUATION WORK? 
Theme Sub-Themes # Discussed 
(1) Learning 
Feedback/Improvement 
Metaevaluation/Metaanalysis 
Stakeholder/Organizational Learning 
Capacity Building 
Facilitation 
19 
(2) Multiple Perspectives 
Collaboration 
Triangulation 
Sources of Information 
19 
(3) Intuition versus Purpose 
Intuition 
Purpose 
Both Intuition and Purpose 
2 
7 
10 
(4) Thinking 
Thinking Back 
Critical Thinking 
Evaluative Thinking 
Understanding Context 
17 
(5) Questioning 
General Questioning 
Process Questioning 
Improvement Questioning 
16 
(6) Self-Awareness 
Previous Experiences 
Worldview 
General Self-Awareness 
10
LEARNING 
“If you don’t reflect you can go from evaluation to 
evaluation to evaluation and within the evaluation and 
just not be learning and improving on what you’re 
doing.”
MULTIPLE PERSPECTIVES “What you’re doing is you’re saying 
what are the different sources of 
information, and it’s not necessarily 
triangulation… in the truest sense of 
the word where you’re talking about 
can I confirm this piece of data with 
that piece of data with another 
piece of data… but rather this notion 
that there are going to be multiple 
perspectives on any given set of 
evaluation questions, on any given 
set of… questions, indicators, 
instruments, findings, there are 
going to be multiple perspectives of 
those and you have to take into 
account the fact that there are 
multiple perspectives.”
INTUITION V. PURPOSE 
“You could almost say there is a 
Zen of practice, and part of that 
is being present not just in the 
moment, but present all the way 
through the experience so that 
you can look back… it’s almost 
like there’s two of you so… it’s a 
cognitive process, it’s an intuitive 
process, and it’s an iterative 
process, where you review, you 
think, you study what you’re 
doing, you study what you have 
done. You look at the results that 
came out of that and then you 
make a judgment accordingly.”
INTUITION V. PURPOSE 
“I think that if it’s more purposeful 
then… there’s a greater likelihood that... 
reflection will improve your practice. So 
to me reflective practice, you do it for a 
purpose. You reflect because you know 
you don’t know everything and you want 
to figure that out, and… try to do better 
next time, or try to find the things that 
worked really well and do them again.”
THINKING 
“And my idea of reflection is thinking… I 
mean you think all the time in evaluation, 
it’s… it’s your weapon, thinking. I mean 
you can’t be without it for a minute.”
RESEARCH QUESTION TWO 
IN WHAT WAYS DO PROFESSIONAL EVALUATORS ENGAGE IN BEHAVIORS ASSOCIATED WITH REFLECTIVE PRACTICE? 
Theme Sub-Themes # Discussed 
(1) Sharing 
Sharing with the Evaluation Community 
Meetings 
Dialogue/ Discussion 
Steering Committees / Advisory Boards 
Debriefing/Briefing 
19 
(2) Individual Reflective Practice 
Introspection 
Journaling 
Voice Memos 
Checklists 
Reviewing Resources 
18 
(3) Prevalence 
Negotiation 
Pre-Evaluation 
Design 
Implementation 
Analysis 
Reporting 
Post-Evaluation 
18 
(4) Evaluation 
Self-Evaluation 
Metaevaluation/Metaanalysis 
Feedback 
11 
(5) Using Guidelines 
Program Evaluation Standards 
Guiding Principles 
Essential Competencies 
Ethical Codes 
3
INDIVIDUAL REFLECTIVE PRACTICE 
“Just sitting in your chair at 
the end of the day and going 
what worked well, what 
didn’t work well, and how am 
I going to resolve this 
challenge? I got feedback 
about something, what do I 
think about that feedback? 
So… that’s the kind of 
ongoing reflective practice.”
PREVALENCE 
“I can’t imagine doing an 
evaluation that is not extremely 
questioning and reflective from 
beginning to end.”
EVALUATION 
“I define it as basically evaluating what I 
do. So it’s the sort of… meta-way of doing 
your work, metaevaluation; which can 
happen both formally and informally. 
Reflective practice, it’s about not making 
assumptions that you know what you’re… 
that you know what you’re doing.”
RESEARCH QUESTION THREE 
IN WHAT WAYS DO EVALUATORS PERCEIVE REFLECTIVE PRACTICE AS HAVING A COLLABORATIVE ELEMENT IN EVALUATION? 
COLLABORATION WAS DEFINITIONAL 
“There cannot be any reflective practice process without the collaborative 
element. I mean if it’s just you reflecting on your own, well that’s great… you 
know. It’s like you’re swimming in your own sweat… your boundaries are the 
boundaries of your reflective practice.”
TRAINING IN REFLECTIVE PRACTICE 
“To me, reflection… critical reflection… critical 
analysis… reflective practice is an inherent part of 
science. Not an add-on. It’s inherent to it. From 
the start of educating children on how scientists 
work, how to do science, reflection ought to be 
part of the process they use… back to the days 
when you’re doing your first science fair 
experiment, all the way through. If you are 
teaching evaluation then reflective practice ought 
to be inherent, infused so that there’s no need for 
kind of a special course or special attention to 
reflective practice. It ought to be like breathing. 
But you know that’s possibly not what happens.”
TRAINING IN REFLECTIVE PRACTICE 
“There certainly should be experiences that are conscious and 
deliberately constructed that are designed to provide you with a 
forum where you can develop your ability to think critically or 
be a reflective practitioner.”
TRAINING IN REFLECTIVE PRACTICE 
“You can never underestimate what people 
don’t understand. So my first reaction was 
gonna be no, anybody knows how to 
reflect… but maybe they don’t you know, 
and who wants to chance it. You know, I 
think picking a couple of themes on what 
to reflect on… work with stakeholders, and 
your data… and then people sharing stories 
about reflecting, and role modeling 
reflecting on something would be helpful 
to students and practitioners.”
CONCLUSIONS AND 
RECOMMENDATIONS 
What does it mean and where do we go from here?
DISCUSSION 
• Although participants are not using 
the words “reflective practice,” they 
are engaging in the behaviors 
associated with the notion. 
• Participants’ conceptualizations of 
RP were in line with both the 
essential competencies for program 
evaluators (Stevahn, King, Ghere, & 
Minnema, 2005) and the Program 
Evaluation Standards (Yarbrough, 
Shulha, Hopson, & Caruthers, 
2011).
DISCUSSION 
• Patton (2012) expressed suspicion with 
regard to how much evaluators actually 
use RP systematically in their evaluation 
work… 
• Overall, participants’ conceptualization of 
RP appears to be above and beyond the 
call of duty presented by the essential 
competencies for program evaluators 
described by Stevahn and her colleagues 
(2005).
IMPLICATIONS FOR PRACTICE 
• How professional evaluators label RP in evaluation 
• A more systematic approach to RP in evaluation 
• Ensuring professional evaluators build time into 
evaluation for reflection (alone and with others) 
• Training professional evaluators in RP and its 
associated behaviors
RECOMMENDATIONS FOR FUTURE RESEARCH 
• Case study research on evaluators 
• Follow-up with a quantitative examination 
• Perspectives of RP in evaluation from practitioners 
in different countries 
• Replication with less experienced evaluators
Thank you for your time! 
QUESTIONS OR COMMENTS? 
CONTACT INFORMATION: 
Tiffany Smith, Ph. D 
University of Wisconsin, Stout 
smithtif@uwstout.edu 
Gary Skolits, Ed. D. 
University of Tennessee, 
Knoxville 
gskolits@utk.edu 
Patrick Barlow, Ph. D. 
University of Iowa 
patrick-barlow@uiowa.edu

More Related Content

What's hot

CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 5-Reflective cycle-AIOU-8611
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 5-Reflective cycle-AIOU-8611CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 5-Reflective cycle-AIOU-8611
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 5-Reflective cycle-AIOU-8611Ek ra
 
Reflective Practice Session
Reflective Practice SessionReflective Practice Session
Reflective Practice Sessionjaclynne
 
What is reflection
What is reflectionWhat is reflection
What is reflectionAamir Ali
 
Reflective Piece Final assignment portfolio
Reflective Piece Final assignment  portfolioReflective Piece Final assignment  portfolio
Reflective Piece Final assignment portfoliormg6449
 
Reflective Practice: Formulating Your Teaching Experience (ppt)
Reflective Practice: Formulating Your Teaching Experience (ppt)Reflective Practice: Formulating Your Teaching Experience (ppt)
Reflective Practice: Formulating Your Teaching Experience (ppt)JosetteLB
 
Collaborative Task - Gibbs Model of Reflection
Collaborative Task - Gibbs Model of ReflectionCollaborative Task - Gibbs Model of Reflection
Collaborative Task - Gibbs Model of ReflectionNovie Isaac
 
Reflective Practice Group Presentation by Jessica, Rich, and Candace
Reflective Practice Group Presentation by Jessica, Rich, and CandaceReflective Practice Group Presentation by Jessica, Rich, and Candace
Reflective Practice Group Presentation by Jessica, Rich, and CandaceCandace Ramey Rivera
 
Reflective practice
Reflective practiceReflective practice
Reflective practicewcctlc
 
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 7-Reflective and Critical Wri...
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 7-Reflective and Critical Wri...CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 7-Reflective and Critical Wri...
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 7-Reflective and Critical Wri...Ek ra
 
Reflective practice (1)
Reflective practice (1)Reflective practice (1)
Reflective practice (1)hannahtyreman
 
Reflecting on ‘reflective practice’
Reflecting on ‘reflective practice’Reflecting on ‘reflective practice’
Reflecting on ‘reflective practice’Lisa Thyer
 
Reflective Writing
Reflective WritingReflective Writing
Reflective Writingerikbohemia
 
NCV 2 Human & Social Development Hands-On Support Slide Show - Module 1
NCV 2 Human & Social Development Hands-On Support Slide Show - Module 1NCV 2 Human & Social Development Hands-On Support Slide Show - Module 1
NCV 2 Human & Social Development Hands-On Support Slide Show - Module 1Future Managers
 
Reflective Practice Presentation
Reflective Practice PresentationReflective Practice Presentation
Reflective Practice PresentationJennifer Fenton
 

What's hot (20)

CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 5-Reflective cycle-AIOU-8611
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 5-Reflective cycle-AIOU-8611CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 5-Reflective cycle-AIOU-8611
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 5-Reflective cycle-AIOU-8611
 
Reflective practice
Reflective practiceReflective practice
Reflective practice
 
Reflective Practice Session
Reflective Practice SessionReflective Practice Session
Reflective Practice Session
 
What is reflection
What is reflectionWhat is reflection
What is reflection
 
Reflective Piece Final assignment portfolio
Reflective Piece Final assignment  portfolioReflective Piece Final assignment  portfolio
Reflective Piece Final assignment portfolio
 
Reflective Practice
Reflective PracticeReflective Practice
Reflective Practice
 
Reflective Practice
Reflective PracticeReflective Practice
Reflective Practice
 
Reflective thinking
Reflective thinkingReflective thinking
Reflective thinking
 
Reflective Practice: Formulating Your Teaching Experience (ppt)
Reflective Practice: Formulating Your Teaching Experience (ppt)Reflective Practice: Formulating Your Teaching Experience (ppt)
Reflective Practice: Formulating Your Teaching Experience (ppt)
 
Collaborative Task - Gibbs Model of Reflection
Collaborative Task - Gibbs Model of ReflectionCollaborative Task - Gibbs Model of Reflection
Collaborative Task - Gibbs Model of Reflection
 
Reflective Practice Group Presentation by Jessica, Rich, and Candace
Reflective Practice Group Presentation by Jessica, Rich, and CandaceReflective Practice Group Presentation by Jessica, Rich, and Candace
Reflective Practice Group Presentation by Jessica, Rich, and Candace
 
Reflective practice 1
Reflective practice 1Reflective practice 1
Reflective practice 1
 
Reflective practice
Reflective practiceReflective practice
Reflective practice
 
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 7-Reflective and Critical Wri...
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 7-Reflective and Critical Wri...CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 7-Reflective and Critical Wri...
CRITICAL THINKING AND REFLECTIVE PRACTICES-Unit 7-Reflective and Critical Wri...
 
Reflective practice (1)
Reflective practice (1)Reflective practice (1)
Reflective practice (1)
 
Reflecting on ‘reflective practice’
Reflecting on ‘reflective practice’Reflecting on ‘reflective practice’
Reflecting on ‘reflective practice’
 
Reflective Writing
Reflective WritingReflective Writing
Reflective Writing
 
NCV 2 Human & Social Development Hands-On Support Slide Show - Module 1
NCV 2 Human & Social Development Hands-On Support Slide Show - Module 1NCV 2 Human & Social Development Hands-On Support Slide Show - Module 1
NCV 2 Human & Social Development Hands-On Support Slide Show - Module 1
 
Reflective Practice Presentation
Reflective Practice PresentationReflective Practice Presentation
Reflective Practice Presentation
 
Reflective learning
Reflective learning Reflective learning
Reflective learning
 

Similar to Reflective Practice, Collaboration, and Stakeholder Communication

C2 l reflective pedagogy spring 2011
C2 l reflective pedagogy   spring 2011C2 l reflective pedagogy   spring 2011
C2 l reflective pedagogy spring 2011making_connections
 
Handout: To Be a Reflective Evaluation Practitioner
Handout: To Be a Reflective Evaluation PractitionerHandout: To Be a Reflective Evaluation Practitioner
Handout: To Be a Reflective Evaluation PractitionerTiffany Smith
 
Self-reflection- Dr Ryan Thomas Williams
Self-reflection- Dr Ryan Thomas WilliamsSelf-reflection- Dr Ryan Thomas Williams
Self-reflection- Dr Ryan Thomas WilliamsRyan Williams
 
Reflective learning & critical thinking
Reflective learning & critical thinkingReflective learning & critical thinking
Reflective learning & critical thinkingPIR BUX JOKHIO
 
Tutorial 1 kaedah penyelidikan
Tutorial 1 kaedah penyelidikanTutorial 1 kaedah penyelidikan
Tutorial 1 kaedah penyelidikanDiyana Mahmud
 
PGCAP week2 reflecting and developing (CORE Sep11)
PGCAP week2 reflecting and developing (CORE Sep11)PGCAP week2 reflecting and developing (CORE Sep11)
PGCAP week2 reflecting and developing (CORE Sep11)Academic Development
 
Kay Sambell Keynote Edinburgh 2014
Kay Sambell Keynote Edinburgh 2014Kay Sambell Keynote Edinburgh 2014
Kay Sambell Keynote Edinburgh 2014Kay Sambell
 
Reflexivity lecture and continuous exercise
Reflexivity lecture and continuous exerciseReflexivity lecture and continuous exercise
Reflexivity lecture and continuous exerciseMarkL07
 
Civic engagement evaluation
Civic engagement evaluationCivic engagement evaluation
Civic engagement evaluationJohn Hamerlinck
 
Using mental models for product success
Using mental models for product successUsing mental models for product success
Using mental models for product successProduct Anonymous
 
Action research handbook
Action research handbookAction research handbook
Action research handbookkokpa91
 
Toronto Public Relation is a Discipline of Depth
Toronto Public Relation is a Discipline of DepthToronto Public Relation is a Discipline of Depth
Toronto Public Relation is a Discipline of Depthrobinmethew
 

Similar to Reflective Practice, Collaboration, and Stakeholder Communication (20)

C2 l reflective pedagogy spring 2011
C2 l reflective pedagogy   spring 2011C2 l reflective pedagogy   spring 2011
C2 l reflective pedagogy spring 2011
 
Handout: To Be a Reflective Evaluation Practitioner
Handout: To Be a Reflective Evaluation PractitionerHandout: To Be a Reflective Evaluation Practitioner
Handout: To Be a Reflective Evaluation Practitioner
 
2.28.11 webinar
2.28.11 webinar2.28.11 webinar
2.28.11 webinar
 
Self-reflection- Dr Ryan Thomas Williams
Self-reflection- Dr Ryan Thomas WilliamsSelf-reflection- Dr Ryan Thomas Williams
Self-reflection- Dr Ryan Thomas Williams
 
Reflective writing
Reflective writingReflective writing
Reflective writing
 
Reflective writing
Reflective writingReflective writing
Reflective writing
 
Reflective learning & critical thinking
Reflective learning & critical thinkingReflective learning & critical thinking
Reflective learning & critical thinking
 
Tutorial 1 kaedah penyelidikan
Tutorial 1 kaedah penyelidikanTutorial 1 kaedah penyelidikan
Tutorial 1 kaedah penyelidikan
 
PGCAP week2 reflecting and developing (CORE Sep11)
PGCAP week2 reflecting and developing (CORE Sep11)PGCAP week2 reflecting and developing (CORE Sep11)
PGCAP week2 reflecting and developing (CORE Sep11)
 
Critical writing + first part of session
Critical writing + first part of sessionCritical writing + first part of session
Critical writing + first part of session
 
The importance of the reflexive practice in career guidance and counselling
The importance of the reflexive practice in career guidance and counsellingThe importance of the reflexive practice in career guidance and counselling
The importance of the reflexive practice in career guidance and counselling
 
Critical Thinking
Critical ThinkingCritical Thinking
Critical Thinking
 
Kay Sambell Keynote Edinburgh 2014
Kay Sambell Keynote Edinburgh 2014Kay Sambell Keynote Edinburgh 2014
Kay Sambell Keynote Edinburgh 2014
 
Action research
Action researchAction research
Action research
 
Reflexivity lecture and continuous exercise
Reflexivity lecture and continuous exerciseReflexivity lecture and continuous exercise
Reflexivity lecture and continuous exercise
 
Abc reg 2018 unborn world
Abc reg 2018 unborn worldAbc reg 2018 unborn world
Abc reg 2018 unborn world
 
Civic engagement evaluation
Civic engagement evaluationCivic engagement evaluation
Civic engagement evaluation
 
Using mental models for product success
Using mental models for product successUsing mental models for product success
Using mental models for product success
 
Action research handbook
Action research handbookAction research handbook
Action research handbook
 
Toronto Public Relation is a Discipline of Depth
Toronto Public Relation is a Discipline of DepthToronto Public Relation is a Discipline of Depth
Toronto Public Relation is a Discipline of Depth
 

Recently uploaded

Indexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfIndexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfChristalin Nelson
 
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...DhatriParmar
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection
 
Tree View Decoration Attribute in the Odoo 17
Tree View Decoration Attribute in the Odoo 17Tree View Decoration Attribute in the Odoo 17
Tree View Decoration Attribute in the Odoo 17Celine George
 
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxBIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxSayali Powar
 
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...Nguyen Thanh Tu Collection
 
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Osopher
 
Employablity presentation and Future Career Plan.pptx
Employablity presentation and Future Career Plan.pptxEmployablity presentation and Future Career Plan.pptx
Employablity presentation and Future Career Plan.pptxryandux83rd
 
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptxDecoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptxDhatriParmar
 
Congestive Cardiac Failure..presentation
Congestive Cardiac Failure..presentationCongestive Cardiac Failure..presentation
Congestive Cardiac Failure..presentationdeepaannamalai16
 
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...HetalPathak10
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...Nguyen Thanh Tu Collection
 
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvRicaMaeCastro1
 
MS4 level being good citizen -imperative- (1) (1).pdf
MS4 level   being good citizen -imperative- (1) (1).pdfMS4 level   being good citizen -imperative- (1) (1).pdf
MS4 level being good citizen -imperative- (1) (1).pdfMr Bounab Samir
 

Recently uploaded (20)

Introduction to Research ,Need for research, Need for design of Experiments, ...
Introduction to Research ,Need for research, Need for design of Experiments, ...Introduction to Research ,Need for research, Need for design of Experiments, ...
Introduction to Research ,Need for research, Need for design of Experiments, ...
 
Indexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfIndexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdf
 
prashanth updated resume 2024 for Teaching Profession
prashanth updated resume 2024 for Teaching Professionprashanth updated resume 2024 for Teaching Profession
prashanth updated resume 2024 for Teaching Profession
 
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
 
Tree View Decoration Attribute in the Odoo 17
Tree View Decoration Attribute in the Odoo 17Tree View Decoration Attribute in the Odoo 17
Tree View Decoration Attribute in the Odoo 17
 
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxBIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
 
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
 
Spearman's correlation,Formula,Advantages,
Spearman's correlation,Formula,Advantages,Spearman's correlation,Formula,Advantages,
Spearman's correlation,Formula,Advantages,
 
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
 
Paradigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTAParadigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTA
 
Employablity presentation and Future Career Plan.pptx
Employablity presentation and Future Career Plan.pptxEmployablity presentation and Future Career Plan.pptx
Employablity presentation and Future Career Plan.pptx
 
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
 
Faculty Profile prashantha K EEE dept Sri Sairam college of Engineering
Faculty Profile prashantha K EEE dept Sri Sairam college of EngineeringFaculty Profile prashantha K EEE dept Sri Sairam college of Engineering
Faculty Profile prashantha K EEE dept Sri Sairam college of Engineering
 
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptxDecoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
 
Congestive Cardiac Failure..presentation
Congestive Cardiac Failure..presentationCongestive Cardiac Failure..presentation
Congestive Cardiac Failure..presentation
 
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
 
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
 
MS4 level being good citizen -imperative- (1) (1).pdf
MS4 level   being good citizen -imperative- (1) (1).pdfMS4 level   being good citizen -imperative- (1) (1).pdf
MS4 level being good citizen -imperative- (1) (1).pdf
 

Reflective Practice, Collaboration, and Stakeholder Communication

  • 1. REFLECTIVE PRACTICE, COLLABORATION, AND STAKEHOLDER COMMUNICATION: WHERE DOES THE FIELD OF EVALUATION STAND? Tiffany L. Smith, University of Wisconsin, Stout Gary J. Skolits, University of Tennessee, Knoxville Patrick B. Barlow, University of Iowa EVALUATION 2014
  • 2. IN THIS PRESENTATION • What is Reflective Practice? • Statement of the Problem • Study Purpose and Research Questions • Design and Methodology • Participants • Findings • Conclusions and Recommendations
  • 3. WHAT IS REFLECTIVE PRACTICE? • One of six essential competencies for program evaluators identified by Stevahn, King, Ghere, & Minnema (2005) • Highlighted in the Program Evaluation Standards (2011) • Discussed theoretically and practically by a number of evaluation authors
  • 4. Reflective practice (RP) is critical and deliberate inquiry into professional practice in order to gain a deeper understanding of oneself, others, and the meaning that is shared among individuals. This can happen during practice and after the fact, and can either be done alone or with others (Forrester, 2010; Peters, 1991; Schön, 1983).
  • 5. STATEMENT OF THE PROBLEM • Currently, the evaluation community does not know evaluators’ opinions, perceptions, or the extent to which evaluators use RP in their evaluation work. • As the field of evaluation is changing, it is important to explore the element of reflection in evaluation practice.
  • 6. STUDY PURPOSE AND RESEARCH QUESTIONS The purpose of this study was to determine how professional evaluators view RP, the extent and manner in which they engage in RP behaviors, and how evaluators conceptualize whether RP efforts affect, if at all, the evaluation process. The following research questions guided this study: 1. How do professional evaluators conceptualize the notion of reflective practice as it relates to their evaluation work? 2. In what ways do professional evaluators engage in behaviors associated with reflective practice? 3. In what ways do evaluators perceive reflective practice as having a collaborative element in evaluation?
  • 7. DESIGN AND METHODOLOGY • Qualitative descriptive design through individual semi-structured interviews. • “Basic” qualitative research, grounded in constructionism, where “individuals construct reality in interaction with their social worlds” (Merriam, 2009, p. 22). • Snowball sampling of 19 evaluators with 10 or more years of program evaluation experience.
  • 8. PARTICIPANTS • 19 program evaluators with a range of experience from 10-49 years (9 had 10-30, 10 had 31+) • 11 female, 8 male • 9 practitioners, 10 academics • Range of different settings for evaluation work, some claiming to be “generalists” • 15 had doctoral degrees • 17 “serendipitous” evaluators
  • 10. OVERARCHING THEMES Collaboration/Multiple Perspectives Learning and Feedback Self Awareness and Introspection Critical thinking and questioning Intuition versus Purpose
  • 11. RESEARCH QUESTION ONE HOW DO PROFESSIONAL EVALUATORS CONCEPTUALIZE THE NOTION OF REFLECTIVE PRACTICE AS IT RELATES TO THEIR EVALUATION WORK? Theme Sub-Themes # Discussed (1) Learning Feedback/Improvement Metaevaluation/Metaanalysis Stakeholder/Organizational Learning Capacity Building Facilitation 19 (2) Multiple Perspectives Collaboration Triangulation Sources of Information 19 (3) Intuition versus Purpose Intuition Purpose Both Intuition and Purpose 2 7 10 (4) Thinking Thinking Back Critical Thinking Evaluative Thinking Understanding Context 17 (5) Questioning General Questioning Process Questioning Improvement Questioning 16 (6) Self-Awareness Previous Experiences Worldview General Self-Awareness 10
  • 12. LEARNING “If you don’t reflect you can go from evaluation to evaluation to evaluation and within the evaluation and just not be learning and improving on what you’re doing.”
  • 13. MULTIPLE PERSPECTIVES “What you’re doing is you’re saying what are the different sources of information, and it’s not necessarily triangulation… in the truest sense of the word where you’re talking about can I confirm this piece of data with that piece of data with another piece of data… but rather this notion that there are going to be multiple perspectives on any given set of evaluation questions, on any given set of… questions, indicators, instruments, findings, there are going to be multiple perspectives of those and you have to take into account the fact that there are multiple perspectives.”
  • 14. INTUITION V. PURPOSE “You could almost say there is a Zen of practice, and part of that is being present not just in the moment, but present all the way through the experience so that you can look back… it’s almost like there’s two of you so… it’s a cognitive process, it’s an intuitive process, and it’s an iterative process, where you review, you think, you study what you’re doing, you study what you have done. You look at the results that came out of that and then you make a judgment accordingly.”
  • 15. INTUITION V. PURPOSE “I think that if it’s more purposeful then… there’s a greater likelihood that... reflection will improve your practice. So to me reflective practice, you do it for a purpose. You reflect because you know you don’t know everything and you want to figure that out, and… try to do better next time, or try to find the things that worked really well and do them again.”
  • 16. THINKING “And my idea of reflection is thinking… I mean you think all the time in evaluation, it’s… it’s your weapon, thinking. I mean you can’t be without it for a minute.”
  • 17. RESEARCH QUESTION TWO IN WHAT WAYS DO PROFESSIONAL EVALUATORS ENGAGE IN BEHAVIORS ASSOCIATED WITH REFLECTIVE PRACTICE? Theme Sub-Themes # Discussed (1) Sharing Sharing with the Evaluation Community Meetings Dialogue/ Discussion Steering Committees / Advisory Boards Debriefing/Briefing 19 (2) Individual Reflective Practice Introspection Journaling Voice Memos Checklists Reviewing Resources 18 (3) Prevalence Negotiation Pre-Evaluation Design Implementation Analysis Reporting Post-Evaluation 18 (4) Evaluation Self-Evaluation Metaevaluation/Metaanalysis Feedback 11 (5) Using Guidelines Program Evaluation Standards Guiding Principles Essential Competencies Ethical Codes 3
  • 18. INDIVIDUAL REFLECTIVE PRACTICE “Just sitting in your chair at the end of the day and going what worked well, what didn’t work well, and how am I going to resolve this challenge? I got feedback about something, what do I think about that feedback? So… that’s the kind of ongoing reflective practice.”
  • 19. PREVALENCE “I can’t imagine doing an evaluation that is not extremely questioning and reflective from beginning to end.”
  • 20. EVALUATION “I define it as basically evaluating what I do. So it’s the sort of… meta-way of doing your work, metaevaluation; which can happen both formally and informally. Reflective practice, it’s about not making assumptions that you know what you’re… that you know what you’re doing.”
  • 21. RESEARCH QUESTION THREE IN WHAT WAYS DO EVALUATORS PERCEIVE REFLECTIVE PRACTICE AS HAVING A COLLABORATIVE ELEMENT IN EVALUATION? COLLABORATION WAS DEFINITIONAL “There cannot be any reflective practice process without the collaborative element. I mean if it’s just you reflecting on your own, well that’s great… you know. It’s like you’re swimming in your own sweat… your boundaries are the boundaries of your reflective practice.”
  • 22. TRAINING IN REFLECTIVE PRACTICE “To me, reflection… critical reflection… critical analysis… reflective practice is an inherent part of science. Not an add-on. It’s inherent to it. From the start of educating children on how scientists work, how to do science, reflection ought to be part of the process they use… back to the days when you’re doing your first science fair experiment, all the way through. If you are teaching evaluation then reflective practice ought to be inherent, infused so that there’s no need for kind of a special course or special attention to reflective practice. It ought to be like breathing. But you know that’s possibly not what happens.”
  • 23. TRAINING IN REFLECTIVE PRACTICE “There certainly should be experiences that are conscious and deliberately constructed that are designed to provide you with a forum where you can develop your ability to think critically or be a reflective practitioner.”
  • 24. TRAINING IN REFLECTIVE PRACTICE “You can never underestimate what people don’t understand. So my first reaction was gonna be no, anybody knows how to reflect… but maybe they don’t you know, and who wants to chance it. You know, I think picking a couple of themes on what to reflect on… work with stakeholders, and your data… and then people sharing stories about reflecting, and role modeling reflecting on something would be helpful to students and practitioners.”
  • 25. CONCLUSIONS AND RECOMMENDATIONS What does it mean and where do we go from here?
  • 26. DISCUSSION • Although participants are not using the words “reflective practice,” they are engaging in the behaviors associated with the notion. • Participants’ conceptualizations of RP were in line with both the essential competencies for program evaluators (Stevahn, King, Ghere, & Minnema, 2005) and the Program Evaluation Standards (Yarbrough, Shulha, Hopson, & Caruthers, 2011).
  • 27. DISCUSSION • Patton (2012) expressed suspicion with regard to how much evaluators actually use RP systematically in their evaluation work… • Overall, participants’ conceptualization of RP appears to be above and beyond the call of duty presented by the essential competencies for program evaluators described by Stevahn and her colleagues (2005).
  • 28. IMPLICATIONS FOR PRACTICE • How professional evaluators label RP in evaluation • A more systematic approach to RP in evaluation • Ensuring professional evaluators build time into evaluation for reflection (alone and with others) • Training professional evaluators in RP and its associated behaviors
  • 29. RECOMMENDATIONS FOR FUTURE RESEARCH • Case study research on evaluators • Follow-up with a quantitative examination • Perspectives of RP in evaluation from practitioners in different countries • Replication with less experienced evaluators
  • 30. Thank you for your time! QUESTIONS OR COMMENTS? CONTACT INFORMATION: Tiffany Smith, Ph. D University of Wisconsin, Stout smithtif@uwstout.edu Gary Skolits, Ed. D. University of Tennessee, Knoxville gskolits@utk.edu Patrick Barlow, Ph. D. University of Iowa patrick-barlow@uiowa.edu

Editor's Notes

  1. Competencies are a set of skills that evaluators should have Standards are a set of guidelines referred to by evaluators and stakeholders to see how evaluators should behave Theoretically – organizational learning in evaluation, evaluation use, decision-making in evaluation, etc.
  2. Note range of experience – most of these were very experienced evaluators and a number of them are well known in the evaluation field. Note – academics were practicing evaluators as well, they just practice in an academic setting. Note – generalist meaning not specific to one field, but take evaluations as they come. Note – serendipitous meaning “fell into evaluation” – this is still a new field, and most didn’t expect to end up in an evaluation profession.
  3. RQ 1 – 6 major themes RQ 2 – 5 major themes RQ 3 – 3 major themes Training in RP FIRST – who were my participants???
  4. Learning through feedback from others and focusing on improvement Including formal or informal metaevaluation/metaanalysis Helping stakeholders and the organization to learn, including facilitation One participant made a general observation regarding learning and reflection in evaluation. They stated…
  5. The idea of multiple perspectives was brought up in terms of including multiple perspectives through collaboration with other individuals; taking into account different sources of data and triangulating; and considering different sources of information available in the evaluation or program context. One participant discussed the idea of triangulation with stakeholders. She said…
  6. Participants were asked whether RP in evaluation was intuitive or purposeful. Overwhelmingly participants claimed it was both. Those who claimed it was purposeful made the distinction that there is indeed an intuitive element, but that RP is a purposeful practice. Respondents who thought of it as intuitive really focused on the idea that it is just something that they do in their everyday practice as humans. Intuition was talked about in terms of being in the moment and operating with your past experiences and background to guide you. Participants chose words and phrases like “natural,” “emotional,” “personalities,” and “human nature.” This participant discussed the intuitive element of RP, but also discussed the purposeful element. She said…
  7. In the context of RP as purposeful, it was considered an intentional process with tools and/or structure to it. Participants used words like “systematic,” “explicit,” “rigorous,” or “intentional.” One participant said…
  8. Some participants talked about thinking generally, mentioning that reflection is a process of thinking through the evaluation, while other participants talked about thinking back over past work with an eye towards improvement. Other responses centered on critical or evaluative thinking, which was reported to be more systematic. Finally, some participants focused on understanding the context of the program they are operating in.
  9. Journaling Voice Memos Checklists Reviewing Resources Introspection
  10. Throughout! Some specific areas of evaluation discussed –Negotiation, Pre-Evaluation, Design, Implementation, Analysis, Reporting, Post-Evaluation
  11. Self-Evaluation, formal Metaevaluation/Metaanalysis, general Feedback
  12. With regard to those who discussed collaboration in terms of RP, 13 participants started discussing the collaborative element of RP before the interviewer asked, 4 of which brought up collaboration early on in their definition of RP. ONE PARTICIPANT didn’t bring up the notion of collaboration in the context of RP
  13. When participants were asked if there should be any sort of training in RP, 3 said no need.
  14. 5 Unsure – Thought it may be useful but didn’t know how to do it, or thought it was built into the learning process.
  15. 11 YES, it would be good for evaluators to receive training in RP