Integrating Gender into Climate Change Adaptation Programs: A Research and Ca...
The Decision Support Practices And Research Needs Of Nonprofits
1. The Decision-support Practices and Training Needs of Nonprofits A Proposal for Fine-tuning Nonprofit Management Education Julie Grace Grey - June 2011
2. The Situation Past is Prologue Current Nonprofit Management Education Nonprofit Management in Practice Movement to Professionalize Evaluation Peer-reviewed Journal National Association Research and Practice Taxonomy of Essential Competencies University Offerings
3. Program Logic Model & Planning Cycle Planning Cycle: Surveillance –> Evaluation –>Analytic Studies –> Policy Development
4. Survey of Oregon nonprofits 358 Respondents 252 Selected for Analysis
5. Survey Tool Demographics Personnel Capacity Evaluation Vision & Planning Data Analysis and Decision-support Dissemination and Usage Top 5 Data Needs Wish List
6. Specifies Program Theory 57% of nonprofits have evaluation requirements 51% use a logic model (step-wise use of data) 33% use population-based outcome evaluations 14% conduct population-based evaluations at multiple geographic levels (community impact)
7. Level of Analysis “Analyzes Data” 57% analyzed client data for patterns. 25% conducted trend analysis. 22% evaluated client data by geographic area An even deeper level of analysis would use time-series data to develop predictive models in support of planning and decision making.
8. Study Groups Nonprofits without evaluation requirements – 43% Nonprofits with evaluation requirements – 57% Nonprofits that consistently conduct population-based outcome evaluations – 27% Nonprofits whose reports/analyses are consistently used by external groups – 13%
9. Nonprofits that Consistently Conduct Population-based Outcome Evaluations Have staff that can: Respond to special requests for data Perform basic descriptive statistics Analyze data by geographic levels Synthesize and interpret research data Analyze time-series data Estimate potential population coverage Etc… But they do not attend to evaluation use…
10. Influential Nonprofits Evaluations linked to broader management plan Reports used by management/board Reports useful to external audiences Voluminous data management activities Dynamic communication between stakeholders Staff role in evaluation activities
11. Top 5 Items on Data Needs Wish List Grant Writing and Fundraising Collaboration w/ Local Universities & Each Other Program Evaluation & Outcome Assessment Technical Assistance – computer soft/hardware Community-wide Database
12. Recommendations Management Programs with Nonprofit Certification Program Evaluation Theory & Methods Quantitative Data Analysis for Managers Management Programs without NPM Certification Concentration in Program Evaluation including Program Evaluation Theory & Methods Quantitative Data Analysis for Managers Qualitative Methods, Survey Design, & Report Writing Mediation & Conflict Resolution Organizational Leadership Grant Writing Project Management Vertical Practicum Series x 3
13. Future Directions Nonprofit Evaluation Assistance Services…. Evaluation Capacity Building Workshops & Classes Coaching (Catalyst Approach) Direct Services Logic Model Development Program Evaluation Survey Design/Analysis Technical Assistance (Software Applications)
Editor's Notes
My name is Julie. I am interested in new theories and directions in education and workforce training. My project surveys the decision-support practices and training needs of nonprofit organizations and outlines additional management training in the area of program evaluation.
What I wanted to do with this project is linkpast experiences with management education to develop additional but needed training for nonprofit managers.Current nonprofit education was developed long ago and includes a broad and rich curriculum that is both relevant and essential – including organizational behavior, marketing, budget & Finance, Law and ethics… But curriculums are not static and must adapt to changed demands in workplace.Now more than ever – to substantiate funding – government agencies and nonprofits must demonstrate organizational value to stakeholders demanding evidence of performance in achieving community-wide and often politically charged goals.The situation is so severe that there is a movement towards professionalizing the field of program evaluation. Indeed a constellation of events coalesce to make it so:There is a peer-reviewed journal and a respected national association. There is a lot of discussion among researchers and practitioners surrounding development of a set of essential competencies for program evaluators.The Essential Competencies is the latest effort to develop standard competencies to facilitate improved evaluation training, enhance reflective practice, to enumerate specific skills, knowledge, and dispositions.And, finally universities are beginning to take note. In the spring of 2011, the University of Oregon began it’s nonprofit management degree program which includes a course in program evaluation – but that is rare so far.
Briefly describe the basic logic model…..A nonprofit that can clearly articulate, with appropriate evidence, a chain of logic or reasoning demonstrating how their programs and services contribute to changes in client behavior or to community goals have evaluation designs that have the greatest potential for programmatic, policy, and legislative influence. They can link project-level results to community-level outcomes.
Survey of Oregon nonprofits was designed to describe research practices and needs AND to measure the extent to which evaluation practices comply with systematic inquiry and situational analysis categories of the Taxonomy of Essential Competencies for Program Evaluators.
Demographics – useful for testing the reliability/validity of the data as well as for descriptive statistics ==county of operation, respondent’s role, activity code, evaluation requirements.Personnel Capacity – Number of volunteers and paid staff, total hours per week, staff able to conduct specific decision-support activities: trend analysis, pop estimation, synthesis & interpretation, basic descriptive statistics, etc…Evaluation Vision & Planning – looks at the overall vision and plan for evaluation – include data from multiple sources? Synthesize and interpret research findings? Identify data needs of providers of similar services or advocates or the legislature? Is data collected in one step of the program used to inform other steps in the evaluation? – a backdoor to logic model design.Data Analysis and Decision-support – use relevant geographic data and at what point in the evaluation process do they use such data? Engage in specific decision-support practices like 1) conduct a survey, 2) estimate the potential client-base, 3) analyze data for patterns, 4) analyze data for trends,5) evaluate GIS data, 6) conduct population-based evaluations 7) PBE at multiple geographic levels.Dissemination and Usage - looks at ways data are translated into decisions about programs or policies - who contributes to the analyses? Do they use their own data, do outside advocates use their data in support of their agenda’s?
Arguably, one of the most important subcategories of the “Essential Competencies” is “Specifies program theory” – aka a logic model.57% with evaluation requirements – 51.4% use data collected in one step of the planning cycle to inform activities in other steps – an internal step-wise use of data that indicates use of a logic model. But only 33.3% can identify using a logic model (pop-based) that has a chance for program/policy/leg influence – and only 14% do so at multiple geographic levels to pinpoint precisely their impact in the community.What this shows is that funder-imposed evaluation requirements do not guarantee program evaluation competence.
Research shows that the level of analysis makes a difference in studying and understanding effectiveness. And, an example of how my research is unique and contributory to evaluation research is the level of analysis. One subcategory of the Essential Competencies - “Analyzes data” – is nondescript so I’m demonstrating levels of analysis.Of nonprofits with evaluation requirements 57% reported the nonprofit analyzed client data for patterns BUT 25% reported analyzing time-series data for trends – in the first instance finding outliers and anomalies but not – as in the second case – identifying changes to the client-base or impact to the community over time.An even deeper level of analysis – using time-series data to develop predictive models to support decision-making likely has far fewer practitioners.
Essentially, in my study, the extent to which a nonprofit complies with the essential competencies breaks down to four groups with unique organizational behaviors and characteristics.
Nonprofits that consistently conduct population-based outcome evaluations are technically skilled evaluators in a better position to defend their missions.BUT – they do not attend to issues of evaluation use – an even higher level of performance is displayed by influential nonprofits.
Nonprofits that consistently conduct population-based outcome evaluations show the highest compliance with the essential competencies than other study groups and distinct organizational behaviors and characteristics are evident.Evaluations linked to broader management strategy: Strategic plan or quality assurance effortsReports used internally by management & board membersProduce information useful to external audiencesEngage in voluminous data management and analysis activities (data collection, trend analysis, pop estimation, surveys, synthesis and interpretation, lit reviews….)Dynamic exchange of information (legislature, advocates, providers of similar studies)Role of staff in evaluation activities – empowered to respond to survey for example – and has input into the meaning of the data.
