Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016

STRATEGIC LEARNING, RESEARCHAND EVALUATIONAT
THEVIRGINIAG. PIPER CHARITABLETRUST
Jeffrey A. Sheldon, Ph.D.
4 January 2016
Virginia G. Piper Charitable Trust

Overview
• Global Vision for Strategic Learning, Research, and Evaluation
• Creating and Supporting a Context for Learning and Knowledge Management
• Approaching and Assessing the Impact of Investments
• Planning and Refining Grant-making
• Work Approach 1: Fostering Empowerment
• Work Approach 2: Fostering Self-Determination
• Projected Collaborative Accomplishments
• Facing Challenges
Virginia G. Piper Charitable Trust - 2

GlobalVision for Strategic Learning,Research,and Evaluation
• A supportive context for learning and evaluation (i.e., culture, leadership, systems and structures,
communications, and teams) is created and sustained.
• Learning across the Trust is in real-time, is communicatively grounded, and based on Adult Learning Theory.
• Everyone within the Trust is involved in learning such that the Trust becomes a community of learners.
• Generating knowledge for learning:
• Through evaluation and research.
• Gathered from individuals within and without the Trust.
• Gathered through the Trust’s “collective wisdom.”
• Gathered from grantees.
• Gathered from external sources (e.g., peer reviewed and gray literature, and from other similarly engaged
philanthropic and human service organizations).
• Evaluation capacity is built within the Trust so that everyone understands its inherent importance, everyone
becomes better consumers of evaluation, everyone can contribute within their specific role(s), everyone can
teach others about evaluation, and everyone can contribute to organizational learning.

GlobalVision for Strategic Learning,Research,and Evaluation
• Grantees evaluation capacity is built so they become integral to the overall research, evaluation, and learning
enterprise by being better consumers and producers of evaluative knowledge.
• A community of learners is created within each initiative so grantees can learn from each other and the Trust
can learn from them.
• A rigorous research agenda is developed within each initiative, there is publishing in peer reviewed journals
and other media as appropriate, and presentations are made at conferences (e.g., American Evaluation
Association).
• Strategic learning, research, and evaluation are integral in a complex ecological system of interrelated parts.
• Strategic learning, research, and evaluation informs the planning and refinement of the Trust’s grant-making
endeavors.

• Organizational learning is the best way to address critical issues facing organizations today because they must
be ready to receive and act upon new information for change to occur (Bowen, Rose, & Ware, 2006).
• Contextual factors influence how and to what extent learning will occur in an organization (Cousins &
Leithwood, 1986; Cousins & Shulha, 2002).
• Culture, leadership, communication, teamwork, and systems/structures are considered to be the elements of
organizational context necessary for organizational learning (Shulha & Cousins, 1997).
• Transformational leadership, supportive structure, and culture are the key factors promoting the learning
process (Lam & Pang, 2003).
• Evaluation contributes to individual, group, and organizational learning (Preskill & Torres, 1999; Russ-Eft,
Atwood, & Egherman, 2002).
• An organization’s culture and context influence the extent to which evaluation occurs in support of learning
and decision-making (Preskill & Torres, 2000).
Creatingand Supporting a Context for Learningand Knowledge Management

• Is the Trust’s culture supportive?
• Examples of collaboration and problem-solving behaviors:
– There is respect for different perspectives and opinions.
– Staff is continuously looking for ways to improve processes and initiatives.
– Opportunities are provided to think about and reflect on work.
– There is more concern about how work contributes to the success of the Trust rather than on individual
success.
– Problems or issues are generally viewed as opportunities to learn.
• Examples of risk-taking behaviors:
– Mistakes are viewed as opportunities for learning.
– There is continuous questioning abut how the Trust is doing, what it can do better, and what is
working.
– There is a willingness to take risks in the course of doing the Trust’s work.
– There is a commitment to being innovative and forward looking.
– There is confidence that mistakes or failures will not have a negative consequences.
• Examples of participatory decision-making behaviors:
– Individuals trust themselves and each other.
– Individual capacity to learn is the Trust’s greatest resource.
– Data/information is used to inform decision-making.
– Asking questions and raising issues about the Trust’s work is encouraged.
– Offering dissenting opinions and alternative viewpoints is encouraged.

• Is the Trust’s leadership transformational by supporting learning and knowledge management?
• Officers believe that the Trust’s success depends upon learning from daily practices.
• Officers support the sharing of knowledge and skills among staff members.
• Officers provide the necessary time and support for systemic, long-term change.
• Officers use data/information to inform decision-making.
• Officers take on the role of coaching, mentoring and facilitating learning.
• Officers help staff understand the value of experimentation and the learning that can result from such
endeavors.
• Officers admit when they don’t know the answer to a question.
• Officers make realistic commitments for staff (e.g. time, resources, and workload).
• Officers understand that staff has different learning styles and learning needs.
• Officers are more concerned with serving the Trust than with seeking personal power or gain.
• Officers are open to constructive feedback from others.
• Officers model the importance of learning through their own efforts to learn.

• Does the Trust have systems and structures in place that will support learning and knowledge management?
• Examples of an open and accessible work environment:
– There is little bureaucratic red tape when trying to do something new or different.
– The Trust is designed to allow for easy and frequent communication amongst staff.
– There are few boundaries that keep staff from working together.
– All staff are available to participate in meetings.
• Examples of rewards and recognitions system and practices:
– There is recognition or rewards for learning new knowledge and skills.
– Staff are recognized or rewarded for helping solve Trust problems.
– The current reward/appraisal system recognizes, in some way, team learning and performance.
– Staff are recognized or rewarded for helping others learn.
– Staff are recognized or rewarded for experimenting with new ideas.
• Examples of the relationship between the Trust’s work to its goals:
– Staff understands how its work relates to the goals and mission of the Trust.
– Staff performance goals are clearly aligned with the Trust’s strategic goals.
– Staff meets work deadlines.

• Do the Trust’s communications support learning and knowledge management?
• Examples of availability:
– Information is gathered from staff, grantees, board members, community partners or other stakeholders
to gauge how well the Trust is doing.
– Currently available information tells the Trust what it needs to know about the effectiveness of
initiatives.
– There are adequate records of past change efforts and what happened as a result.
• Examples of dissemination:
– There are existing systems to manage and disseminate information for those who need and can use it.
– Staff is cross-trained to perform various job functions.
– Officers and staff have access to the information needed to make work-related decisions.
– Technologies are used to communicate with one another.
– When new information that would be helpful to others is learned or discovered it gets disseminated to
those individuals.

• Does the Trust perform work in teams?
• The Trust operates via, or is transitioning towards, a team-based structure.
• Work at the Trust is sometimes conducted as part of a working group that is or could be identified as a
“team.”
• Teams are an effective way to meet Trust goals.
• Training is provided on how to work as a team member.
• Is The Trust engaged in evaluation?
• The integration of evaluation activities into the Trust’s work has enhanced the quality of decision-making.
• It has been worthwhile to integrate evaluation activities into the Trust’s daily work practices.
• There would be support if the Trust tried to do more evaluation work.
• Evaluation helps the Trust provide better initiatives.
• Doing more evaluation would make it easier to convince Trust staff of needed changes.
• This would be a good time to begin, renew or intensify efforts to conduct evaluations.
• There are evaluation processes in place enabling the Trust to review how well changes it makes are
working.
Creating and Supporting a Context for Learningand Knowledge Management

Approaching andAssessing the Impact of Investments
• Each of the Trust’s funded programs is unique therefore evaluation design needs a theory-driven approach
based on two models:
• Action Model  Theory-driven process evaluation - It is prescriptive in that it explains the action that is
required to solve a social problem by highlighting the components and activities program designers and
key stakeholders see as necessary for program success.
• Change Model  Theory-driven outcome evaluation - It is descriptive in that it explains why the
problem will respond to the action by highlighting the assumptions about causal processes through which
an intervention is supposed to work.
• Action Model + Change Model  Integrated Theory-driven process/outcome evaluation.

• Action Model (Process)
• Intervention and service delivery protocols.
• Implementing organization: assess, enhance, and ensure its capacity.
• Program implementers: recruit, train and maintain both competency and commitment.
• Associate organizations/community partners: establishing collaborations.
• Ecological context: seek its support at micro and macro levels.
• Target population: identify, recruit, screen, serve.
• Change Model (Outcomes)
• Intervention/treatment
• Determinants
– Mediators
– Moderators
• Goals/outcomes:
– Distal
– Intermediate
– Proximal

WWK
Principal
Project
Manager
PTO/PTA
Officers
Computer
Instructor
Parents
Effects intrinsic
motivation
leading to use
Teachers
Students
Champs
Positive
Letter
Negative
Letter
Incentive
Letter
Phone
E - Mail
Personal
Follow-up
Follow-up
Letter
Internet training
No effect on
intrinsic
motivation
leading to non-
use
Direct influence/communication:
Direct action:
Two-way influence/communication:
Example of an
Action Model
Or
Recruitment
Program

Example of a
Change Model
Safer children
Reduce
Reluctance
Reduce
Ignorance
Reduce
Denial
Reduce
Embarrassment
Reduce
Time
Wired With Wisdom
Parent Recruitment Program
(Letters
Phone calls
E - mails
Personal contact)
Increase
Experience
Increase
Concern
Successful
recruitment =
use of
Wired With
Wisdom
Well managed
family internet
environment &
safety plan
Reduce Risk
Increase
Expectations
Increase Mental
Readiness

• Effective Evaluations Using the Theory - Driven Approach:
• Future action directedness
– Useful to stakeholders.
– Assessing merit, worth, or value is a means rather than an end.
– Provides useful information for stakeholders to improve current or future programs.
• Scientific and stakeholder credibility
– Follow scientific methods and principles to optimize validity and reliability.
– Responds to stakeholders’values, views, concerns, and needs.
• Holistic Approach
– Intrinsic value.
– Considers program context.
• Compels evaluator to be thoughtful before acting because she thoroughly understands the program.
• Helps tailor evaluations to answer the most important questions (i.e., parsimony) and informs methodological
choices.
• Heightens evaluation sensitivity and validity.
• Helps evaluator meet American Evaluation Association professional evaluation standards – Utility,
Feasibility, Propriety, Accuracy.
• Fosters cumulative wisdom.

• Face-to-face meetings with stakeholders (working group or intensive interview).
• Develop a systematic understanding of stakeholder views, needs, and values.
• Facilitate conceptualization of a program:
• Stakeholders reflectively examine what they are doing.
• Stakeholders identify elements that are essential for achieving program goals.
• Stakeholders articulate causal relationships.
• Theorizing methods – backward reasoning (start with intended outcomes) to inputs, forward reasoning, or
both.
• Increases stakeholders’buy-in and support for evaluation.
• Work with stakeholders to focus the evaluation design (e.g., questions, data, methods, analysis, etc.).
• Gather credible evidence using rigorous methods based on contingency (e.g., qual., quant., or mixed).
• Work with stakeholders to examine data and justify conclusions.
• Work with stakeholders to ensure utilization and lessons learned are widely disseminated.
• Utilization of the evaluation can include:
• Conceptual (knowledge produced used for understanding/education).
• Instrumental (knowledge produced used for decision-support).
• Process (making use of the logic of the evaluation).
• Symbolic (knowledge used to justify a priori decisions).

Planningand Refining Grant-making
• Identify the issues/problems that need to be solved based in part on previous grant-making experience and
criteria internal to the Trust.
• Articulate assumptions about how the problems can be addressed with available resources:
• Existing knowledge of the issue.
• The activities of others in the field.
• Previous grant-making experience.
• How the Trust’s support can address the problem and achieve desired results.
• Develop theoretical action and change models or logic models to express assumptions; these are updated
throughout the planning/refinement process.
• Scan the field to assess needs and identify gaps by taking stock of the social, political, and economic
context—where it is currently and where it may be heading.
• Goal: Identify areas or strategies where additional resources can have significant impact.
• Methods: Secondary research, discussions, gathering and convening, surveys, and network mapping.
• Topics: Populations of concern, community needs, context, key players, strategies, other funders,
successful strategies/what works, and gaps.

Planningand Refining Grant-making
• Take stock of internal capacity through a frank and thorough assessment.
• Develop a clear understanding of the Trust’s capabilities so that it can align what is needed with what it
can realistically contribute.
• Internal capacity includes: Finances, human assets, intellectual capital, social capital, comparative
advantage/influence.
• Validate assumptions through a "quality review" with other stakeholders to ensure that the logic is sound,
assumptions are accurate, and relevant influential factors are accounted for.
• Implement, evaluate, and plan again as an ongoing, iterative process.

WorkApproach 1: FosteringEmpowerment
• A long-term process individuals, organizations, and communities undertake for themselves rather than
something done to or for them (Lather, 1991, 1992; Claridge, 1996)
• Empowered outcomes: individuals or aggregate bodies of individuals engaging in behaviors that permit
effective pursuit of planned social change, for example, working with others, managing resources, learning
decision-making skills, influencing policy, accessing government, sharing leadership, obtaining needed
resources, being networked to others, etc. (Schulz et al., 1995; Zimmerman, 2000).
• Three interrelated components (Zimmerman, Israel, Schulz, & Checkoway, 1992).
• Interactional: Developing knowledge, skills, and strategies to become leaders/advocates for
change.
• Intrapersonal: Appraisal/critical awareness of, and the motivation and agency to change one’s
life circumstances (i.e., self-efficacy).
• Behavioral: Individual-level actions requiring leadership that benefits individuals and
communities by influencing social policy.

WorkApproach 2: FosteringSelf-Determination
• A general, organismic theory of human motivation explaining the relationship between three sets of
motivational processes—intrinsic, extrinsic, and amotivational —and self-determined behaviors (Ryan &
Connell, 1989).
• To be self-determined, experience well–being, and have optimal health (e.g., self-esteem, self-actualization,
vitality, positive affect, and work engagement) people have to satisfy three basic psychological needs
endowed by nature (Koestner, Ryan, Bernieri, & Holt, 1984):
• Competence: efficacious performance of behaviors based on positive feedback and communication.
• Autonomy: choice in how to perform a task is provided, the task is recognized as being important, and
feelings about the task are acknowledged.
• Relatedness: meaningful connections to others and a sense of belonging; the opposite is alienation and
inauthenticity.
• The collective research on SDT suggests that when basic psychological needs are supported, intrinsic
motivation and integrated extrinsic motivation are most likely to be evident.

Projected CollaborativeAccomplishmentsWithin 3 – 6 Months
• Contextual assessment conducted with internal Trust team to determine readiness for organizational learning and
evaluative inquiry using quantitative (e.g., the ROLE) and qualitative (e.g., observation and content analysis)
methods.
• Qualitative research methods will be used to determine knowledge needs and ascertain knowledge, skills,
experiences, and attitudes about evaluation by group (see below) and across the Trust:
• Interviews/focus groups of recent grantees within each initiative (random sample).
• Interviews with all current grantees (full census)/formulate action and change models and evaluation plans.
• Interviews with board of trustee members (full census, n = 8).
• Interviews with outside board committee members (full census, n = 4).
• Interviews with officers (full census, n = 3).
• Interviews with staff (full census, n = 13).
• Information gathered through quantitative and qualitative research will be collaboratively analyzed to make
meaning of the data, synthesized, and reported back to each group along with a cross-group report.
• Will have canvassed other charitable trusts/foundations with similar initiatives to inform the Trust’s best practices in
research, evaluation, and strategic learning.
• A framework for evaluation, research, and strategic learning that maps onto the Trust’s strategic plan as informed
by data and input from Trust leadership, staff et al. is at or near completion.

OvercomingChallenges using the GettingTo Outcomes Model
• Focus: The problem related to pediatric vision we’re trying to solve or unmet need to be fulfilled based on
secondary or external data.
• Target: Identify goals, target population, and desired outcomes of a pediatric vision screening and follow-up
care initiative.
• Adopt: Find existing programs and best practices regarding pediatric vision screening and follow-up care
worth adopting (must be evidence-based or a promising practice based on preliminary research).
• Adapt: Modify the program or best practice to fit children’s vision screening and follow-up care needs.
• Resources: Assess capacity (e.g., staff, financing, etc.) to implement the initiative.
• Plan: Make a plan for getting started by developing action and change theoretical models, and formulating
and prioritizing evaluation questions.
• Monitor: Track planning and implementation process.
• Evaluate: Evaluate initiative’s success in achieving desired results by using rigorous scientific methods to
answer evaluation questions.
• Improve: Make a plan for continuous quality improvement from the results of evaluative inquiry.
• Sustain: Consider how to keep the initiative going if it is successful.

Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016

Similar to Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016 (20)

Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016