Recommended
Recommended
More Related Content
What's hot
Similar to Count Your Impact: Getting Started with Outcome-Based Evaluation
Similar to Count Your Impact: Getting Started with Outcome-Based Evaluation (20)
More from library_research_service
More from library_research_service (11)
Recently uploaded
Recently uploaded (20)
Count Your Impact: Getting Started with Outcome-Based Evaluation
- 1. COUNT YOUR IMPACT: GETTING STARTED WITH OUTCOME-BASED EVALUATION Linda Hofschire | @lindahofschire www.lrs.org | @lrs_co | ripl.lrs.org
- 3. OUTCOME-BASED EVALUATION PROJECT Outcome-Based Evaluation Project Project name: Senior Computer Class Series Description: My library offers a class series for seniors that covers basic computer /technology skills
- 4. DATA TRIFECTA Inputs Outputs Outcomes
- 5. INPUT MEASURES What effort was expended on the program/resource/service? Inputs
- 8. staff hours
- 9. OUTPUT MEASURES How many… services were provided? programs were offered? people attended? Outputs
- 10. number of registered participants
- 11. number of events offered
- 12. total attendance at events
- 13. number of minutes spent reading
- 14. circulation
- 15. OUTCOME MEASURES What difference does the program/resource/service make? (the why) Outcomes
- 16. Children’s enjoyment of reading increased
- 17. children maintained or improved reading skills
- 18. children were more likely to read by choice
- 19. library developed long-term partnership with local school district
- 20. DATA TRIFECTA Inputs Outputs Outcomes
- 21. ACTIVITY: WHAT IS YOUR DATA TRIFECTA? Inputs Outputs Outcomes
- 22. Inputs Outputs Outcomes Funds expended: • Copies of handouts • Marketing campaigns at local senior centers Staff hours: • Curriculum development • Instruction Number of events offered Number of registrants Number of attendees Digital literacy Confidence using technology Ongoing partnerships with local senior centers
- 23. ACTIVITY: WHAT IS YOUR DATA TRIFECTA? Outcome-Based Evaluation Project Project name: Description: Inputs Outputs Outcomes
- 24. Library A Library CLibrary B
- 25. Library A • 100% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills Library CLibrary B
- 26. Library A • 100% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills Library CLibrary B • 25 children participated • 100 children participated • 50 children participated
- 27. Library A • 100% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills Library CLibrary B • 25 children participated • 100 children participated • 50 children participated • $5,000 • $10,000 • $2,500
- 28. Library A • 100% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills Library CLibrary B • 25 children participated • 100 children participated • 50 children participated • $5,000 • $10,000 • $2,500 • $200/child • $100/child • $50/child
- 30. IntermediateImmediate Long-term OUTCOMES Learn how to complete online job application
- 31. IntermediateImmediate Long-term OUTCOMES Learn how to complete online job application Get an interview
- 32. IntermediateImmediate Long-term OUTCOMES Learn how to complete online job application Get an interview Get a job
- 33. IntermediateImmediate Long-term OUTCOMES Learn how to complete online job application Get an interview Get a job
- 34. LINKING IMMEDIATE OUTCOMES TO LONG-TERM IMPACT Reading for pleasure associated with: Increased empathy Improved relationships Reduced depression symptoms Reduced risk of dementia The Reading Agency, 2015
- 35. LINKING OUTPUTS TO LONG-TERM IMPACT Reading more associated with… Improved reading skills Better development of vocabulary, writing, and grammar Chin & Krashen (2007)
- 36. Any successes measuring long-term impact?
- 39. INDICATORS Outcome Indicator Increased enjoyment of reading Parent response to survey question
- 40. INDICATORS Outcome Indicator Increased enjoyment of reading Parent response to survey question Increased digital literacy
- 41. INDICATORS Outcome Indicator Increased enjoyment of reading Parent response to survey question Increased digital literacy Demonstrate ability to text, send an email with an attachment, etc.
- 42. After participating in the Summer Reading Program, did your child’s reading habits change? Please indicate if the following things increased, stayed the same, or decreased for your child: Increased Stayed the Same Decreased Enjoyment of reading Reading skills Reading by his/her choice SURVEY
- 43. While I was on a computer in the computer center today, I . . . SURVEY Looked for employment (job search, resume, application, etc.). Communicated with someone (email, chat, Facebook, etc.). Used the computer for entertainment (ex: videos, music, games). Used government resources (licenses, taxes, unemployment, disability, welfare, immigration, etc.). Did schoolwork. Found health information.
- 44. Please let us know in your own words the difference the 2014 Summer Reading Program has made for your child(ren) and family: SURVEY—OPEN ENDED QUESTIONS
- 45. As a first time parent it's really helpful to have tangible ideas about ways to get my child engaged in early literacy skills. There were some recommended activities I hadn't thought of trying!
- 46. I loved being able to spend time with my child and bond over my love for books. My son's reading endurance has really grown this summer, thanks to the reading program. He was very motivated to read for 40 minutes and sometimes he read for even longer stretches.
- 47. Please tell us how the computer center helps you or your community: SURVEY—OPEN ENDED QUESTIONS
- 48. Probably the most valuable resource, dollar for dollar, available to community. I have found jobs…researched… located tax information, and have done school work over the years. Thank you!
- 49. The access to the internet afforded by the public library is most probably the only reason I am not completely bereft of any and all computer skills. Were it not for this access, as well as the assistance rendered via classes offered, I would most likely be unemployed if not unemployable.
- 50. Other ideas for open-ended questions?
- 52. PLA-PROJECT OUTCOME Early Childhood Literacy Education/Lifelong Learning Digital Inclusion Job Skills Economic Development Civic/Community Engagement Summer Reading
- 53. IMPACT SURVEY
- 54. CONTINUUM
- 55. CONTINUUM
- 56. CONTINUUM
- 57. POSTCARD
- 58. PRIZE DRAWING
- 60. ACTIVITY: OUTCOME INDICATORS AND METHODS FOR MEASUREMENT Outcome Indicators: Outcome=Digital literacy Indicator #1=Class attendee is able to send an email with an attachment Indicator #2=Class attendee is able to like the library Facebook page
- 61. ACTIVITY: OUTCOME INDICATORS AND METHODS FOR MEASUREMENT How will you measure outcomes? Skill demonstration (indicators listed above) Pre-post continuum to assess confidence using technology
- 62. ACTIVITY: OUTCOME INDICATORS AND METHODS FOR MEASUREMENT Outcome Indicators: How will you measure outcomes?
- 63. I’ve collected some outcome data, now how do I use it?
- 64. Strategic Planning Management Communication DATA USE
- 65. Area of strategic focus: Nurture children’s learning by providing programs and services that instill a love of reading and advance reading skills. STRATEGIC PLANNING
- 66. After participating in the Summer Reading Program, did your child’s reading habits change? Please indicate if the following things increased, stayed the same, or decreased for your child: Increased Stayed the Same Decreased Enjoyment of reading Reading skills Reading by his/her choice SURVEY
- 67. Implications of OBE results for: Budgeting Staffing Programs (focus, number offered, size, timing, etc.) MANAGEMENT
- 68. Library A • 100% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills • 85% of participating children increased or maintained reading skills Library CLibrary B • 25 children participated • 100 children participated • 50 children participated • $5,000 • $10,000 • $2,500 • $200/child • $100/child • $50/child
- 69. Internal stakeholders External stakeholders COMMUNICATION
- 71. ACTIVITY: OUTCOME INDICATORS AND METHODS FOR MEASUREMENT Identify uses for data in… 1) Strategic planning: 2) Management: 3) Communication:
- 72. ACTIVITY: OUTCOME INDICATORS AND METHODS FOR MEASUREMENT Identify uses for data in… 1) Strategic planning: One of my library’s strategic priorities is to provide lifelong learning. An OBE of our senior tech classes will help us to determine whether we’re achieving this priority or if we need to make adjustments.
- 73. ACTIVITY: OUTCOME INDICATORS AND METHODS FOR MEASUREMENT Identify uses for data in… 2) Managing: Are we achieving our desired outcomes? If not, do we need to adjust staff hours, budget, etc.?
- 74. ACTIVITY: OUTCOME INDICATORS AND METHODS FOR MEASUREMENT Identify uses for data in… 3) Communicating: Our mayor recently announced an initiative to improve the lives of seniors in our community. This is a multi- faceted initiative that will address health, wellbeing, financial and digital literacy, etc. We can demonstrate the library’s unique role in supporting this initiative by sharing our OBE data with her office.
- 75. THANK YOU! Linda Hofschire | @lindahofschire Hofschire_L@cde.state.co.us www.lrs.org | @lrs_co | ripl.lrs.org
Editor's Notes
- NS To evaluate and report on our projects and services we need to collect three types of data: Input, Output, and Outcome measures. I call this the Data Trifecta. Today we are going to talk a lot about outcome measures because that how we measure community engagement. But, in order to understand library outcomes you really need inputs and outputs as well. So, let’s take a look at each on of these measures in a library context.
- NS First…Input measures Defined as: Quantifying the effort expended on a program Stuff you put into the project (hence the name). Money and staff tend to be the biggies in the input measures category. For my examples I used summer reading programs in public libraries. It is a both a staple of library programming, as well as an evolving service in public libraries. It is a classic example of community engagement. Funds expended Number of staff Staff hours Staff salaries
- NS Second… Output Measures Defined as: Quantifying the level of services and materials provided What did you spend the money on? What happened as a result of inputs? For my examples I used summer reading programs in public libraries. It is a both a staple of library programming, as well as an evolving service in public libraries. It is a classic example of community engagement. Funds expended Number of staff Staff hours Staff salaries
- NS Third measure in the data trifecta is Outcome Measures Quantifying the effect a service has on the program’s stated objectives (or goals) Ultimately, what effect do want the program to have? On of my colleagues always says this is where the people come in. What changed for them? Did they learn a skill? Did they acquire knowledge? Did they change their behavior? Do they do their job better now? Is their library a better place? What changed for these people? What difference did the project make? NS So, here are some examples of possible outcomes for a summer reading program. Notice, the first three are outcomes for library patrons—namely children. However, the last outcome is library related, not patron related.
- NS So, as you’ve probably already gathered these measures are linear. You put resources in—those are inputs Those inputs yield services or things—those are outputs Those outputs impact people & their organizations—those are outcomes
- LH
- LH
- LH
- NS Example is conceptual concentrate on the relationships between the measures Example will Illustrate the importance of the data trifecta Help understand the difference between input, outputs, & outcomes How they fit together 3 public libraries Evaluate summer reading program Goal: children attending summer reading program will increase or maintain reading skills Summer slide Similar communities - 300 school age children Look at 3 possible measures – taking advantage of the data trifecta
- NS Outcome measure Library A 100% children participating in SRP increased or maintained reading skills Libraries B & C 85% increased or maintained reading skills Add output measure…
- NS Output measure – number of participants in SRP Library A – 25 children Library B – 100 children Library C – 50 children Calculate # of children increased or maintained reading skills 85 children in Library B ~43 children in Library C Add input measure…
- NS Input measure: SRP budget Library A - $5,000 Library B – $10,000 Library C - $2,500 Adds even more detail to the evaluation See differences between the SRP One last measure calculated measure input & output data
- NS Ratio Cost per child Library A - $200/child Library B - $100/child Library C - $50/child Added data – added to understanding Don’t you look at Library C differently now vs beginning Simplistic example More data in real life Data Trifecta Adds understanding Truer picture Detailed Truthful picture
- LH
- LH
- LH
- NS Ratio Cost per child Library A - $200/child Library B - $100/child Library C - $50/child Added data – added to understanding Don’t you look at Library C differently now vs beginning Simplistic example More data in real life Data Trifecta Adds understanding Truer picture Detailed Truthful picture
- LH
- LH
- LH
- LH