This document summarizes a presentation on using reflective practice to enhance evaluators' professional activities. It introduces the four presenters and their backgrounds in evaluation and reflective practice. It then discusses why reflection is important for evaluators, defining it as one of the essential competencies. The Data Model of reflection, which includes the steps of Describe, Analyze, Theorize, and Act, is introduced as a framework for structured reflection. A case example is provided of how an evaluator used the Data Model to reflect on a challenging evaluation project experiencing issues due to changes in requirements.
Kurt lewin’s action research model - OD process - Organizational Change and...manumelwin
The process of change involves three steps
Unfreezing: Faced with a dilemma or disconfirmation, the individual or group becomes aware of a need to change.
Changing: The situation is diagnosed and new models of behavior are explored and tested.
Refreezing: Application of new behavior is evaluated, and if reinforcing, adopted.
Reflective Practice, Collaboration, and Stakeholder Communication: Where Does...Tiffany Smith
This handout accompanies the 2014 AEA research presentation entitled "Reflective Practice, Collaboration, and Stakeholder Communication: Where Does the Field of Evaluation Stand?"
Kurt lewin’s action research model - OD process - Organizational Change and...manumelwin
The process of change involves three steps
Unfreezing: Faced with a dilemma or disconfirmation, the individual or group becomes aware of a need to change.
Changing: The situation is diagnosed and new models of behavior are explored and tested.
Refreezing: Application of new behavior is evaluated, and if reinforcing, adopted.
Reflective Practice, Collaboration, and Stakeholder Communication: Where Does...Tiffany Smith
This handout accompanies the 2014 AEA research presentation entitled "Reflective Practice, Collaboration, and Stakeholder Communication: Where Does the Field of Evaluation Stand?"
Implementing an Organisational Intervention for Work-related Stress: an Actio...Mike Pye
John Hamilton is a dual qualified chartered safety and health practitioner and HR practitioner with extensive experience working in large complex organisations in the public and private sector. John is research active, using evidence based practice to design and implement high participation wellbeing interventions, corporate health and safety governance systems, and employee engagement programmes. John recently completed a PhD in organisational health and wellbeing at Lancaster University and explains his study in this presentation.
Presentation by Terri Manning, Associate Vice President for Institutional Research/Director of the Center for Applied Research, Central Piedmont Community College; LACCD AtD Liaison at the 2nd Annual LACCD AtD Retreat
Reflective Practice, Collaboration, and Stakeholder CommunicationTiffany Smith
This presentation accompanies the 2014 AEA research presentation entitled "Reflective Practice, Collaboration, and Stakeholder Communication: Where Does the Field of Evaluation Stand?"
invoNET 2012 Presentation.
Public involvement in research: assessing impact through a realist evaluation.
Presenters: David Evans, Vito Laterza & Rosie Davies
Introductions: Simon Denegri, Chair of INVOLVE
Implementing an Organisational Intervention for Work-related Stress: an Actio...Mike Pye
John Hamilton is a dual qualified chartered safety and health practitioner and HR practitioner with extensive experience working in large complex organisations in the public and private sector. John is research active, using evidence based practice to design and implement high participation wellbeing interventions, corporate health and safety governance systems, and employee engagement programmes. John recently completed a PhD in organisational health and wellbeing at Lancaster University and explains his study in this presentation.
Presentation by Terri Manning, Associate Vice President for Institutional Research/Director of the Center for Applied Research, Central Piedmont Community College; LACCD AtD Liaison at the 2nd Annual LACCD AtD Retreat
Reflective Practice, Collaboration, and Stakeholder CommunicationTiffany Smith
This presentation accompanies the 2014 AEA research presentation entitled "Reflective Practice, Collaboration, and Stakeholder Communication: Where Does the Field of Evaluation Stand?"
invoNET 2012 Presentation.
Public involvement in research: assessing impact through a realist evaluation.
Presenters: David Evans, Vito Laterza & Rosie Davies
Introductions: Simon Denegri, Chair of INVOLVE
Looking for a great presentation and workshop on strategic thinking and leadership and their role in creating an inspiring and awesome strategic vision and plan? Look no further. Please feel free to call me if you would like me to put on a 2 hour overview of this workshop for your organization or you would like to discuss any points further. My number is 612-310-3803.
Training needs analysis, skills auditing, training evaluation, calculating training ROI and strategic learning and development best practice principles and processes
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
Francesca Gottschalk - How can education support child empowerment.pptxEduSkills OECD
Francesca Gottschalk from the OECD’s Centre for Educational Research and Innovation presents at the Ask an Expert Webinar: How can education support child empowerment?
How to Make a Field invisible in Odoo 17Celine George
It is possible to hide or invisible some fields in odoo. Commonly using “invisible” attribute in the field definition to invisible the fields. This slide will show how to make a field invisible in odoo 17.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
Normal Labour/ Stages of Labour/ Mechanism of LabourWasim Ak
Normal labor is also termed spontaneous labor, defined as the natural physiological process through which the fetus, placenta, and membranes are expelled from the uterus through the birth canal at term (37 to 42 weeks
1. DEMYSTIFYING REFLECTIVE PRACTICE:
USING THE DATA MODEL TO ENHANCE
EVALUATORS’ PROFESSIONAL ACTIVITIES
“Insights and innovation await us only if we are capable of stepping
outside the frenzied worlds of data and distraction that wash over us…
time for reflection is an open invitation to discover what awaits us…”
(Forrester, 2011, pp. 216-217)
Tiffany L. Smith
John M. Peters
Gary J. Skolits
Patrick B. Barlow
Expert Lecture
American Evaluation Association 2013
Friday, October 18th, 1:45-2:30pm
2. INTRODUCTION: WHO ARE WE?
TIFFANY SMITH
JOHN PETERS
GARY SKOLITS
PATRICK BARLOW
A doctoral candidate in
the Evaluation, Statistics
and Measurement
program at the
University of Tennessee
Knoxville. She has taken
two courses on
Reflective Practice and
has interned with the
Institute for Reflective
Practice at the University
of Tennessee with John
Peters. Her primary
research interests
include stakeholder
communication and
involvement as well as
reflective practice in
evaluation.
The Director of the
Tennessee Teaching and
Learning Center’s
Institute for Reflective
Practice. The primary
mission of the Institute is
to promote reflective
practice by individuals
and organizations served
by the University of
Tennessee, Knoxville. Its
service is based on the
belief that addressing
and managing change in
individual lives and
organizations demands
flexible thinking
processes by persons
capable of proactive
interpersonal
communication.
The Director of the
Institute for Assessment
and Evaluation at the
University of Tennessee’s
in the College of
Education, Health, and
Human Sciences. He is a
tenured associate
professor in the
Evaluation, Statistics, an
d Measurement Ph.D.
program. His research
interests include
evaluation methods, the
evaluation of
educational
interventions, P-16 and
college access program
evaluation as well as
higher education
accountability.
A doctoral candidate in
the
Evaluation, Statistics, an
d Measurement program
at the University of
Tennessee. He works as a
Statistical and Research
Design Consultant for
the University of
Tennessee Graduate
School of Medicine. His
primary research areas
are in higher education
assessment and teaching
statistics and research
methods in the social
sciences, epidemiology, a
nd clinical medical
fields.
3. WHY REFLECT?
“Reflective practice is more than spending ten
minutes at the end of an evaluation
congratulating oneself on getting the damn
thing done” (Patton, 2012, p. 401).
4. REFLECTIVE PRACTICE IN EVALUATION
• One of six Essential Competencies for program
evaluators, defined as:
– “being acutely aware of personal evaluation
preferences, strengths, and limitations; selfmonitoring the results of actions intended to
facilitate effective evaluation studies; and planning
how to enhance future endeavors”
(Stevhan, King, Ghere, & Minnema, 2005, p. 46)
5. HOW DOES REFLECTION IMPROVE
EVALUATION PRACTICE?
• Can be used as a means for selfawareness, professional growth and
development, improved ethical practice, dialogue
and stakeholder communication, and for learning
in organizations
• The reflection process can happen while engaging
in daily practice as well as after the fact, either
alone or with others.
• Reflection involves being a student of the actions
that you make, and the study of those actions is
6. Patton (2012) reports that “in speeches and
workshops at professional evaluation
association meetings, I like to ask for a show
of hands of those who systematically reflect
on evaluations they have conducted for
learning and further professional
development. Few hands go up; typically, in
fact, no one raises a hand” (p. 400).
7. A SHOW OF HANDS?
Who systematically reflects on evaluations they
have conducted for learning and further
professional development?
Why Not?
Why is reflective practice not a part of regular
evaluation practice?
8. WHY IS RP NOT A PART OF REGULAR
EVALUATION PRACTICE?
• Perhaps there is a lack of awareness of the
purpose of reflection in evaluation?
• Unclear what reflective practice is in the first
place?
9. WHAT IS RP, ANYWAY?
According to Peters (1991), “reflective practice
involves more than simply thinking about what
one is doing and what one should do next. It
involves identifying one’s assumptions and
feelings associated with practice, theorizing
about how these assumptions and feelings are
functionally or dysfunctionally associated with
practice, and acting on the basis of the resulting
theory of practice” (p. 89).
10. THE DATA MODEL FOR REFLECTION
•
•
•
•
DESCRIBE
ANALYZE
THEORIZE
ACT
11. HOW HAS DATA BEEN USED AS A REFLECTIVE
PRACTICE EFFORT?
• Mainly to guide individual and group
reflection on a variety of work-related
issues, train teams of workers, and for
supervisory training.
• Also as major component of action research
projects in community education
programs, business applications, and in higher
education settings
12. DESCRIBE
• A detailed account of the situation, task, or
incident that has happened in one’s practice.
• Concerned with identifying the specifics of the
situation at hand.
– Context of the situation, the setting in which it
occurred, who was involved, etc.
• Use as much detail as possible, so as to reflect on
the whole of the problem and its setting.
• Everything that you know about the situation and
context are key to this step of the reflective
process.
13. DESCRIBE
• This might prove to be harder than one might
expect
– A description only involves accounts of what actually
occurred
• Description is devoid of any why inclinations or
explanations in order that a clear picture of the
problem is painted (Peters, 2009).
• This portion of the DATA model could prove to be
beneficial if one includes other colleagues, be it
evaluators or stakeholders.
14. ANALYZE
• The why that it was necessary to avoid during
the Describe phase gets its turn.
• Identification of factors contributing to the
situation.
• Examination of one’s assumptions, biases, and
preconceived notions
15. ANALYZE
• Beliefs, rules, motives, and facts should be things that
the practitioner is aware of during this stage of
reflection.
• Through the first two steps of DATA, problem is at least
tentatively outlined
– The practitioner should have a very clear and holistic
understanding of the problem, its context, and why it has
come up in their practice.
• This will in turn produce a practical question of how to
solve the problem.
– How, specifically, can I change my practice to produce a
better outcome?
16. THEORIZE
• The focus is on answering the practical question
introduced in the analysis.
• Refers to deriving a practical theory from the
description and analysis in order to improve one’s
practice.
– A practical theory refers to understanding the structure
and implications of a theory’s use in professional practice.
• This does not need to be a scholarly theory, derived
from the literature, although previous literature may
influence this section
• This theory comes from the practice in which one
works.
17. THEORIZE
• At the end of Theorize, the practitioner should be able
to ask What am I going to do, and Why is this solution
better than other potential solutions?
• How many options do you have?
• Is there relevant literature or research to indicate that
this option is better over others?
• In practice, what has worked before, and how well?
• The Theorize portion of DATA is intended to provide
practitioners with a theoretical solution to their
concern.
18. ACT
• Taking action.
• What are you going to do about it, specifically?
• Moving forward based on a practical theory which is
derived from reflection on the situation, including the
what and the why.
• When one takes action, it is in the context and for the
betterment of their practice.
• This action tests the practical theory that was
developed via this reflective process.
• This can lead to further reflection on one’s practice and
further professional development.
20. Michael is a seasoned evaluator who has
worked many different evaluation projects over
the past 20 years. The following is a case of
Michael utilizing the DATA model of reflective
practice in order to understand and solve a
dilemma in his evaluation work.
21. DESCRIBE
At the beginning of the evaluation project:
• Contract awarded for 3 years of funding
– Serving a true community need
• Stakeholders and evaluators come together to define the
evaluation design, data collection, schedules, procedures
• Client not familiar with evaluation or working with
evaluators and very reluctant and cautious
– Displaying signs of confusing evaluator with auditor
• Client staffs project and project data collection and
reporting based on evaluation design
22. DESCRIBE
• About a year in, change in requirements from funding agency
–
–
–
–
Reporting is vastly increased
Increase in the amount of data to be collected by tenfold
Complications for collecting the data are greatly increased
Project itself is being redefined by new data collection requirements
• Changes mandated at the policy level resulting from inner turmoil
in funding agency
• This issue requires a total renegotiation of the client’s data
collection process as well as the evaluation design, breaking all of
the prior agreements.
• Data collection is complex, training implemented that they never
counted on
23. Due to this issue, the evaluation is struggling.
Evaluative progress is not being made.
24. ANALYZE
• Why is this situation occurring?
– Evaluator’s assumption that no major policy change
would occur was totally inaccurate; after 20 years of
practice, never has such a change been encountered
– Examination of how evaluator’s behavior could have
helped to create the situation
– Examine angers and frustrations
– Evaluator and client, and now multiple stakeholders
come to new understandings about what is expected
and required
• Sense of a weak relationship, a weak tie, and a
mistrust in the stakeholders
25. Practical Question: How can we work to
reestablish trust in the evaluation and program
environment?
26. THEORIZE
• Utilization of Program Evaluation Standards (as well as ethical
guidelines, Essential Competencies, Utilization Focused evaluation)
to decide on the most effective means of establishing trust.
– Some specific Standards: F1, F2, F4, U1, U2, U3, U4, U8, P2, P4, P7
• Discusses options with trusted colleagues
• After thinking about other options, including termination of the
project, Michael decides that the best strategy would be:
– Face to face discussion with primary stakeholders
• Review the status of the relationship prior to the change (contracting, working
relationship)
• Review the relationship after the change
• Gauge the perspectives of stakeholders
• Is there a basis for continuing? Want?
• Both the evaluation staff and the client must be willing to put the cards on the
table and understand that this was an imposed change.
27. ACT
• Michael acts on the basis of a practical theory.
• Renegotiation of agreement
• Re-establishment of trust through critical
conversation with stakeholders
28. USING DATA IN EVALUATION
• How did reflection on the situation help
Michael to come up with a plan of action?
– How could it have improved his professional
practice? Way of thinking?
• When is DATA practical for evaluators?
• Should RP be an alone thing? Or is this best
done collaboratively?
29. USING DATA IN ACTION RESEARCH: THE
DATA-DATA APPROACH
• DATA-DATA has been used to plan and conduct more
than 125 projects in higher education, community
education, and business.
–
–
–
–
Design
Analyze
Theorize
Act
• Using the second DATA when you want to have more
confidence in the result of your thinking and action
that results from the first DATA.
• The systematic design work and careful analysis in the
second DATA accomplishes this.
30. WHAT DOES REFLECTION PROVIDE FOR
EVALUATION PRACTITIONERS?
• A way to step back from daily practice and to
create an encompassing understanding of the
issues and situations that we face
• Reflecting helps us to learn more about context
• Reflection provides us an opportunity to
investigate our own assumptions and how they
influence our actions.
• How can RP be integrated into evaluation work?
31. Can anyone think of a way that the DATA
model can be used in their evaluation work?
32. QUESTIONS?
CONTACT INFORMATION:
Tiffany Smith: tsmith92@utk.edu
John Peters: jpeters@utk.edu
Gary Skolits: gskolits@utk.edu
Patrick Barlow: pbarlow1@utk.edu
Institute for Reflective Practice: irp@utk.edu
Editor's Notes
Reflective practice can sound like a mystical and puzzling concept for practitioners and scholars alike. However, stepping back from our practice to ask the questions, What is it that just happened?, What is happening now?, Why?, and Would I benefit from changing my current way of doing things?, can be very beneficial to evaluation practice. However, reflection does not appear to be utilized often in order to improve evaluation practice (Patton, 2012). The aim of this work is to provide evaluators with a critical and systematic approach to reflective practice that is practical enough to be used in regular evaluation practice.
This competency is a part of a set of competencies that have as their goal to improve training, enhance reflection, advance evaluation research, and for the continual professional development of the field. Similarly, it is worth noting that the Joint Committee on Standards for Educational Evaluation claim that the Program Evaluation Standards, available as a guide for evaluators in order to perform useful and effective evaluations, provide “guidance and [encourage] reflective practice” (Yarbrough, Shulha, Hopson, & Caruthers, 2011, p. xii).
The remainder of this presentation aims to provide an encompassing definition of reflective practice, a practical framework for reflection through the DATA model, and an example of the use of DATA for critical examination into evaluation practice.
The DATA Model involves identifying one’s assumptions, beliefs, values, and motivations, considering how they are intuitively associated with practice, and acting on the basis of a practical theory (Peters, 1991; Peters, 2009). Utilizing DATA is not merely an introspective process; it is action-based and can be utilized for the purpose of enhancing professional practice. It is also important to note that the model is recursive, primarily in the sense that reflection is not linear, but instead we tend to reflect on each of the steps of the model at different times during the process.
After this slide, turn over to Gary
F1 Project Management Evaluations should use effective project management strategies. F2 Practical Procedures Evaluation procedures should be practical and responsive to the way the program operates. F4 Resource Use Evaluations should use resources effectively and efficiently. U1 Evaluator Credibility Evaluations should be conducted by qualified people who establish and maintain credibility in the evaluation context. U2 Attention to Stakeholders Evaluations should devote attention to the full range of individuals and groups invested in the program and affected by its evaluation. U3 Negotiated Purposes Evaluation purposes should be identified and continually negotiated based on the needs of stakeholders. U4 Explicit Values Evaluations should clarify and specify the individual and cultural values underpinning purposes, processes, and judgments. U8 Concern for Consequences and Influence Evaluations should promote responsible and adaptive use while guarding against unintended negative consequences and misuse. P2 Formal Agreements Evaluation agreements should be negotiated to make obligations explicit and take into account the needs, expectations, and cultural contexts of clients and other stakeholders. P4 Clarity and Fairness Evaluations should be understandable and fair in addressing stakeholder needs and purposes. P7 Fiscal Responsibility Evaluations should account for all expended resources and comply with sound fiscal procedures and processes.