Evaluating Digital Literacy programs


Published on

This webinar is specifically developed for anyone currently managing a computer center, librarians involved in computer-related programs, those who support digital literacy programs, or BTOP grant recipients. You will leave with an understanding of how to track the impact of your public access computers and computer training programs.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • TechSoup Global currently operates its product donation programs with partners in 32 countries. With the number of supporting corporations currently at 37 (offering 392 different donated or discounted products) TechSoup Global has provided approximately 4.9 million donated technology products since program inception to NGOs around the world. The retail price of these donations is roughly equivalent to US$1.4 billion.
  • Evaluating Digital Literacy programs

    1. 1. How to Evaluate Your Digital Literacy Program June 30, 2011 Audio is only available by calling this number: Conference Call: 866-740-1260; Access Code: 6339392 Sponsored by
    2. 2. Using ReadyTalk <ul><li>Chat & raise hand </li></ul><ul><li>All lines are muted </li></ul><ul><li>If you lose your internet connection, reconnect using the link emailed to you. </li></ul><ul><li>If you lose your phone connection, re-dial the phone number and re-join. </li></ul><ul><li>ReadyTalk support: 800-843-9166 </li></ul>
    3. 3. You are being recorded… <ul><li>This seminar will be available on the TechSoup website along with past webinar presentations: </li></ul><ul><ul><li>www.techsoup.org/go/webinars </li></ul></ul><ul><li>You will receive a link to this presentation, material and links. </li></ul><ul><li>After the webinar, you can ask follow-up questions in our Accessible Technology and Public Computing Community Forum: </li></ul><ul><ul><li>http://bit.ly/tsg-pcc </li></ul></ul><ul><li>Twitter hashtags: #techsoup </li></ul>
    4. 4. How to Evaluate Your Digital Literacy Program Presenters: Linda Hofschire & Erick Ruiz Sponsored by This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License .
    5. 5. Community Technology Network
    6. 6. Today’s Speakers Linda Hofschire Colorado State Library Facilitator: Kami Griffiths , Community Technology Network Assisting with chat questions: Sarah Washburn , TechSoup Global Stephanie Gerding , TechSoup Global Susan Burkholder , Colorado State Library Erick Ruiz Instructional Design Consultant
    7. 7. Agenda <ul><li>Overview of evaluation fundamentals </li></ul><ul><li>Colorado State Library’s BTOP program </li></ul><ul><li>Systems used to track impact </li></ul><ul><li>Long term benefit of tracking data </li></ul><ul><li>Tools used to track </li></ul><ul><li>Q&A </li></ul>
    8. 8. Poll <ul><li>Are you a BTOP grantee? </li></ul><ul><ul><li>Yes </li></ul></ul><ul><ul><li>No </li></ul></ul>
    9. 9. To measure or NOT to measure <ul><li>Before you begin </li></ul><ul><li>Program objectives must be developed </li></ul><ul><li>Stakeholders must agree </li></ul><ul><li>Realities </li></ul><ul><li>ROI is not for every organization or program </li></ul><ul><li>Are you prepared to handle a negative ROI? </li></ul><ul><li>ROI measurement can add to your program costs </li></ul>Phillips, P. (2002). The Bottomline on ROI . Atlanta, GA: CEP Press
    10. 10. What is ROI? <ul><li>ROI is a measure of accountability </li></ul><ul><li>ROI is a mathematical explanation, comparing program costs to program benefits. </li></ul><ul><ul><li>Benefit Cost Ration (BCR) </li></ul></ul><ul><ul><ul><li>BCR = Program Benefits/Program Costs </li></ul></ul></ul><ul><ul><li>Return on Investment (ROI) </li></ul></ul><ul><ul><ul><li>ROI (%) = (Net Program Benefits/Program Costs) X 10 </li></ul></ul></ul>
    11. 11. What is ROI? <ul><li>Tangible program benefits: expressed quantitatively </li></ul><ul><ul><li>BCR and/or ROI </li></ul></ul><ul><li>Intangible program benefits: expressed qualitatively </li></ul><ul><ul><li>Benefits that can not be quantified </li></ul></ul><ul><ul><li>Benefits that can not be isolated to your program </li></ul></ul>
    12. 12. ROI Levels <ul><li>First... </li></ul><ul><ul><li>Program objectives must be developed </li></ul></ul><ul><li>Then... </li></ul><ul><ul><li>Level 1: Reaction & Satisfaction </li></ul></ul><ul><ul><li>Level 2: Learning </li></ul></ul><ul><ul><li>Level 3: Application </li></ul></ul><ul><ul><li>Level 4: Business Impact </li></ul></ul><ul><ul><li>Level 5: ROI </li></ul></ul>
    13. 13. Level 1: Reaction & Satisfaction <ul><li>Key Questions </li></ul><ul><li>Was the program relevant? </li></ul><ul><li>Was the program important? </li></ul><ul><li>Did the program provide new information? </li></ul><ul><li>Do participants intend to use what was learned? </li></ul><ul><li>Would Participants recommend the program to others? </li></ul><ul><li>Method </li></ul><ul><li>Online Surveys </li></ul><ul><li>Face-to-face interviews </li></ul><ul><li>Focus Groups </li></ul><ul><li>Random Sampling </li></ul>Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
    14. 14. Level 2: Learning <ul><li>Key Questions </li></ul><ul><li>Did participants acquire the knowledge and skills presented in the program? </li></ul><ul><li>Do participants know how to apply what they learned? </li></ul><ul><li>Are participants confident to apply what they learned? </li></ul><ul><li>Method </li></ul><ul><li>Testing </li></ul><ul><ul><li>Norm Referenced Testing </li></ul></ul><ul><ul><li>Criterion Referenced Testing </li></ul></ul>Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
    15. 15. Level 3: Application <ul><li>Key Questions </li></ul><ul><li>How effective are participants at applying what they learned? </li></ul><ul><li>How frequently are participants applying what they learned? </li></ul><ul><li>What support are participants receiving? </li></ul><ul><li>Method </li></ul><ul><li>Participant and “other” </li></ul><ul><li>Online Surveys </li></ul><ul><li>Face-to-face interviews </li></ul><ul><li>Focus Groups </li></ul><ul><li>Random Sampling </li></ul>Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
    16. 16. Level 4: Business Impact <ul><li>Key Questions </li></ul><ul><li>To what extent did application of learning improve the measures the program was intended to improve? </li></ul><ul><li>How do you know it was your program that produced these measures? </li></ul><ul><li>Method </li></ul><ul><li>Survey partners </li></ul><ul><li>Survey participants </li></ul><ul><ul><li>Online Surveys </li></ul></ul><ul><ul><li>Interviews </li></ul></ul><ul><ul><li>Focus Groups </li></ul></ul>Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
    17. 17. Level 5: Return On Investment <ul><li>Key Questions </li></ul><ul><li>Do the monetary benefits of the improvement in business impact measures out-weigh the cost of the program? </li></ul><ul><li>Method </li></ul><ul><li>Financial Measures </li></ul><ul><li>Tangible Benefits Data </li></ul><ul><li>Intangible Benefits Data </li></ul>ROI (%) = (Net Program Benefits/Program Costs) x 100 Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
    18. 18. Colorado Computer Centers: Bridging Colorado’s Great Digital Divide
    19. 19. 81 Public Computer Centers (PCCs)
    20. 20. PCCs in Action
    21. 21. By the Numbers… <ul><li>450 desktops </li></ul><ul><li>700 laptops </li></ul><ul><li>70 ADA compliant workstations </li></ul><ul><li>10,000 trainings </li></ul>
    22. 22. <ul><li>Areas of Focus </li></ul><ul><li>Basic computer skills </li></ul><ul><li>Workforce/employment </li></ul><ul><li>Health </li></ul><ul><li>Resources for adult education </li></ul><ul><li>ESL </li></ul>Partnerships 9 partners, including the Gates Foundation, College in Colorado, the Workforce Development Council, and the Colorado Department of Labor
    23. 23. Three Components of Our BTOP Evaluation <ul><li>Outputs </li></ul><ul><li>Class evaluations </li></ul><ul><li>Immediate outcomes </li></ul>
    24. 24. Planning the Evaluation
    25. 25. Data Map Data element Report Collection Point Quarterly Report Annual Report Grant Application Compliance Officer Website Personnel $ X X Broadband speed X X X X Type of class X X X Equipment $ X X Scheduled one-to-one tutoring sessions X Hours open per 48-hr weekend X X X X
    26. 26. Laying the Groundwork <ul><li>Pretest </li></ul><ul><li>Establish baseline </li></ul>
    27. 27. Outputs
    28. 28. Class Evaluation
    29. 29. Outcomes <ul><li>For classes: </li></ul><ul><ul><li>Basic technology skills </li></ul></ul><ul><ul><li>Employment </li></ul></ul><ul><ul><li>Business </li></ul></ul><ul><ul><li>Health </li></ul></ul><ul><li>For open lab users: </li></ul><ul><ul><li>General </li></ul></ul>
    30. 30. Class Outcomes—Employment <ul><li>Please check all that apply: </li></ul><ul><li>After taking this class, I am better able to… </li></ul><ul><li>Create or work on a resume </li></ul><ul><li>Search for a job online </li></ul><ul><li>Learn about a potential employer </li></ul><ul><li>Learn skills for current or new job </li></ul>
    31. 31. General Outcomes <ul><li>Please check all that apply: </li></ul><ul><li>During my recent visit to the computer center, I… </li></ul><ul><li>Did schoolwork </li></ul><ul><li>Looked for employment </li></ul><ul><li>Communicated with someone (ex., email, online chat, instant messaging, Facebook) </li></ul>
    32. 32. Assessing Outcomes Over Time <ul><li>Select a sub-sample </li></ul><ul><li>Pre-/post-test </li></ul><ul><li>Case study </li></ul><ul><li>Tell the story behind the numbers </li></ul>
    33. 33. Benefits Beyond the Grant Period
    34. 34. Benefits Beyond the Grant Period
    35. 35. Benefits Beyond the Grant Period <ul><li>Use results for budgeting, program planning, advocacy, etc. </li></ul><ul><li>Ongoing evaluation </li></ul>
    36. 36. Tools/Resources <ul><li>Impact survey http://impactsurvey.org/ </li></ul><ul><li>Colorado State Library’s BTOP site http://coloradovirtuallibrary.org/btop/ </li></ul><ul><li>Computer reservations system http://code.google.com/p/powerline/ </li></ul>
    37. 37. Questions? Submit your questions using the chat box.
    38. 38. Continue the Discussion… <ul><li>Additional questions can be answered by posting in in our Community Forum: </li></ul><ul><ul><li>http://bit.ly/tsg-pcc </li></ul></ul>
    39. 39. Thank you to our Webinar Sponsor! ReadyTalk offers dedicated product demos for TechSoup organizations 4 times per week. For more information: pages.readytalk.com/techsoup.html
    40. 40. Thank you! Please complete the post event survey! Kami Griffiths, kami@ctnbayarea.org