It's about the plan: how to ensure that your program evaluation gives you the results you need
•
1 like•483 views
This presentation was given at the California Academic & Research Libraries (CARL) conference in April 2010 in Sacramento, CA
Recommended
More Related Content
What's hot
What's hot (20)
Similar to It's about the plan: how to ensure that your program evaluation gives you the results you need
Similar to It's about the plan: how to ensure that your program evaluation gives you the results you need (20)
Recently uploaded
Recently uploaded (20)
It's about the plan: how to ensure that your program evaluation gives you the results you need
- 1. It’s About the Plan: How to ensure that your program evaluation gives you the results you need Dominique Turnbow Susan Shepherd CARL Conference, April 10, 2010
- 2. Evaluation Questions Is the implementation of the program consistent with the goals specified in the Project Plan? Does the current structure of the Program work well for stakeholders?
- 3. Stakeholders Colleges’ Staff First- and second-year students Provosts Library Public Service Heads College Outreach Liaisons
- 5. Benefits of Planning Clarity
- 6. Benefits of Planning Improved Outcome
- 7. Benefits of Planning Final report writes itself
- 8. Considerations Lead time - planning our evaluation took about 6 months; you’ll want to apply the process to a somewhat substantial research question.
- 9. Considerations Process is rigorous – you may encounter impatience from your team members. Perseverance and strong leadership help.
- 10. Considerations Beware of “analysis paralysis”- don’t over analyze. Keep consulting the plan.
- 12. Now, its your turn…
Editor's Notes
- Dominique, slides 1-2 (10 minutes)Welcome, Introductions, feel free to ask questions throughoutI was first introduced to evaluation planning while taking a course at San Diego State University about program evaluation for my instructional design degree. In my class, I worked on a project for real clients that were trying to evaluate the effectiveness of their outreach to teach parents about the importance of nutrition and exercise so that they could help their children make good lifestyle choices. It was during this project that I saw the value of applying this process to qualitative evaluation. We all know that outreach is a difficult thing to measure since it often lacks hard numbers; instead evaluations rely on affective means. The process can help you focus your evaluation to try to get the data needed to evaluate a project to make the decisions you need. Now, I want to provide a brief description of our study and findings. Remember, this presentation will focus on the plan, but we thought it would be helpful to share a little overview at the beginning. We conducted an evaluation of the College Outreach Program, a pilot program targeting UCSD first- and second-year undergraduatesto promote library services and resources. The program was structured so that each of UCSD's 6 colleges had a "College Outreach Liaison" who was tasked with communicating with the students through their Student Affairs office and college Provost.
- After the 3rd year of the pilot program, we designed and conducted this evaluation to answer tw0 questions: Is the implementation of the program consistent with the goals specified in the Project Plan? (internal stakeholders – to the library)Does the current structure of the Program work well for stakeholders? (external stakeholders)
- Dominique (5 minutes)Five stakeholder groups were identified and invited to participate in an online survey via SurveyMonkey. These groups represented campus leaders, staff and students that were part of or impacted by the COP. Each of the online surveys asked the participants about their opinions of the Program (as it relates to the specific group) and recommendations for improvement and/or change. Responses from all groups were generally positive and favored the College Outreach Program.Colleges’ Staff Valued having one person to contact in the libraries.Students• Over 66% (n=115) of the students surveyed remember getting information about the UCSD Libraries during their orientation in a variety of ways.ProvostsThe Program is appreciated by the CollegesPublic Services Management Group (PSMG)• Generally favored the program, but felt that it might take up more staff time than they had to offer.College Outreach Liaisons (COL)• COLs believe there is value in an outreach program to new freshman and transfer students. Many of their concerns have to do with the structure, sustainability, and support of the Program.Currently, the UCSD Libraries are undergoing a strategic planning process that will help administrators decide if providing outreach to this group of students is a priority.
- Susan (slides 6-10= 10 minutes)– it forces you to be very clear about what you are trying to accomplish and gives you sort of a roadmap to keep referring back to. The method we’re going to lead you through has several components which will help you to examinethe purpose of your program. Then you’ll take a close look at the stakeholders in your program, which helps to prevent overlooking people who may be affected, or who may be relevant to the evaluation.Then, developing the evaluation questions and then making sure the survey instruments ask what is needed to answer those questions forces discipline on the process.
- – following the steps of the process leads to a better evaluation outcome because it keeps you focused.
- – [j/k] –Referring back to the evaluation plan gives you chunks of the report to just drop into place. Keeping the evaluation questions in focus while you’re analyzing the data and writing the report helps to avoid getting distracted by extraneous points.
- It also took us about 6 months to conduct the evaluation, or about a year from start to finish. However, I’m convinced it could take a lot less time. We were a committee We defined 5 stakeholder groups. It would have taken less time with fewer groups=> surveys=> data One of our survey groups was first- and second- year undergrads. This led to lots of data to analyze A lot of our survey data was qualitative, thus taking a little longer to synthesize.
- Dominique (1-2 minutes)
- Dominique (10 minutes)Put up a PDF of our plan on the screen. The plan has seven parts. You can see these on your Activity worksheet. Description – provides the context a reader will need to understand what and why you are doing this evaluation. This section should be short, but include all pertinent details. (This was one of the more difficult sections for us to write). Purpose – includes the broad evaluation questions you would like to answer. Stakeholders – identify important stakeholders and why they are important to the study. Limitations – no study is perfect. Include your factors that may impact your evaluation (e.g. time, resources, access to key stakeholders, etc.) Evaluation Questions/Issues – a table that breaks your broad evaluation questions into sub-questions that will focus your evaluation; also identify the stakeholders that will provide answers to your questions. Questions/Procedures – a table the source and type of information required to answer each broad question; also includes a brief description of your data collection strategy (e.g. interviews, surveys, etc.) Management Plan – provides a general timeline for the evaluation; include information about the task, person responsible. This is a guideline and can be fluid, but it is good to have something to refer to…working meetings once/month
- Activity (about 20 minutes) and reporting (about 10 minutes)