• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Evaluating Digital Literacy programs
 

Evaluating Digital Literacy programs

on

  • 637 views

This webinar is specifically developed for anyone currently managing a computer center, librarians involved in computer-related programs, those who support digital literacy programs, or BTOP grant ...

This webinar is specifically developed for anyone currently managing a computer center, librarians involved in computer-related programs, those who support digital literacy programs, or BTOP grant recipients. You will leave with an understanding of how to track the impact of your public access computers and computer training programs.

Statistics

Views

Total Views
637
Views on SlideShare
637
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • TechSoup Global currently operates its product donation programs with partners in 32 countries. With the number of supporting corporations currently at 37 (offering 392 different donated or discounted products) TechSoup Global has provided approximately 4.9 million donated technology products since program inception to NGOs around the world. The retail price of these donations is roughly equivalent to US$1.4 billion.

Evaluating Digital Literacy programs Evaluating Digital Literacy programs Presentation Transcript

  • How to Evaluate Your Digital Literacy Program June 30, 2011 Audio is only available by calling this number: Conference Call: 866-740-1260; Access Code: 6339392 Sponsored by
  • Using ReadyTalk
    • Chat & raise hand
    • All lines are muted
    • If you lose your internet connection, reconnect using the link emailed to you.
    • If you lose your phone connection, re-dial the phone number and re-join.
    • ReadyTalk support: 800-843-9166
  • You are being recorded…
    • This seminar will be available on the TechSoup website along with past webinar presentations:
      • www.techsoup.org/go/webinars
    • You will receive a link to this presentation, material and links.
    • After the webinar, you can ask follow-up questions in our Accessible Technology and Public Computing Community Forum:
      • http://bit.ly/tsg-pcc
    • Twitter hashtags: #techsoup
  • How to Evaluate Your Digital Literacy Program Presenters: Linda Hofschire & Erick Ruiz Sponsored by This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License .
  • Community Technology Network
  • Today’s Speakers Linda Hofschire Colorado State Library Facilitator: Kami Griffiths , Community Technology Network Assisting with chat questions: Sarah Washburn , TechSoup Global Stephanie Gerding , TechSoup Global Susan Burkholder , Colorado State Library Erick Ruiz Instructional Design Consultant
  • Agenda
    • Overview of evaluation fundamentals
    • Colorado State Library’s BTOP program
    • Systems used to track impact
    • Long term benefit of tracking data
    • Tools used to track
    • Q&A
  • Poll
    • Are you a BTOP grantee?
      • Yes
      • No
  • To measure or NOT to measure
    • Before you begin
    • Program objectives must be developed
    • Stakeholders must agree
    • Realities
    • ROI is not for every organization or program
    • Are you prepared to handle a negative ROI?
    • ROI measurement can add to your program costs
    Phillips, P. (2002). The Bottomline on ROI . Atlanta, GA: CEP Press
  • What is ROI?
    • ROI is a measure of accountability
    • ROI is a mathematical explanation, comparing program costs to program benefits.
      • Benefit Cost Ration (BCR)
        • BCR = Program Benefits/Program Costs
      • Return on Investment (ROI)
        • ROI (%) = (Net Program Benefits/Program Costs) X 10
  • What is ROI?
    • Tangible program benefits: expressed quantitatively
      • BCR and/or ROI
    • Intangible program benefits: expressed qualitatively
      • Benefits that can not be quantified
      • Benefits that can not be isolated to your program
  • ROI Levels
    • First...
      • Program objectives must be developed
    • Then...
      • Level 1: Reaction & Satisfaction
      • Level 2: Learning
      • Level 3: Application
      • Level 4: Business Impact
      • Level 5: ROI
  • Level 1: Reaction & Satisfaction
    • Key Questions
    • Was the program relevant?
    • Was the program important?
    • Did the program provide new information?
    • Do participants intend to use what was learned?
    • Would Participants recommend the program to others?
    • Method
    • Online Surveys
    • Face-to-face interviews
    • Focus Groups
    • Random Sampling
    Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
  • Level 2: Learning
    • Key Questions
    • Did participants acquire the knowledge and skills presented in the program?
    • Do participants know how to apply what they learned?
    • Are participants confident to apply what they learned?
    • Method
    • Testing
      • Norm Referenced Testing
      • Criterion Referenced Testing
    Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
  • Level 3: Application
    • Key Questions
    • How effective are participants at applying what they learned?
    • How frequently are participants applying what they learned?
    • What support are participants receiving?
    • Method
    • Participant and “other”
    • Online Surveys
    • Face-to-face interviews
    • Focus Groups
    • Random Sampling
    Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
  • Level 4: Business Impact
    • Key Questions
    • To what extent did application of learning improve the measures the program was intended to improve?
    • How do you know it was your program that produced these measures?
    • Method
    • Survey partners
    • Survey participants
      • Online Surveys
      • Interviews
      • Focus Groups
    Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
  • Level 5: Return On Investment
    • Key Questions
    • Do the monetary benefits of the improvement in business impact measures out-weigh the cost of the program?
    • Method
    • Financial Measures
    • Tangible Benefits Data
    • Intangible Benefits Data
    ROI (%) = (Net Program Benefits/Program Costs) x 100 Phillips, P. & Phillips, J. (2005). Return On Investment Basics . Alexandria, VA: ASTD Press
  • Colorado Computer Centers: Bridging Colorado’s Great Digital Divide
  • 81 Public Computer Centers (PCCs)
  • PCCs in Action
  • By the Numbers…
    • 450 desktops
    • 700 laptops
    • 70 ADA compliant workstations
    • 10,000 trainings
    • Areas of Focus
    • Basic computer skills
    • Workforce/employment
    • Health
    • Resources for adult education
    • ESL
    Partnerships 9 partners, including the Gates Foundation, College in Colorado, the Workforce Development Council, and the Colorado Department of Labor
  • Three Components of Our BTOP Evaluation
    • Outputs
    • Class evaluations
    • Immediate outcomes
  • Planning the Evaluation
  • Data Map Data element Report Collection Point Quarterly Report Annual Report Grant Application Compliance Officer Website Personnel $ X X Broadband speed X X X X Type of class X X X Equipment $ X X Scheduled one-to-one tutoring sessions X Hours open per 48-hr weekend X X X X
  • Laying the Groundwork
    • Pretest
    • Establish baseline
  • Outputs
  • Class Evaluation
  • Outcomes
    • For classes:
      • Basic technology skills
      • Employment
      • Business
      • Health
    • For open lab users:
      • General
  • Class Outcomes—Employment
    • Please check all that apply:
    • After taking this class, I am better able to…
    • Create or work on a resume
    • Search for a job online
    • Learn about a potential employer
    • Learn skills for current or new job
  • General Outcomes
    • Please check all that apply:
    • During my recent visit to the computer center, I…
    • Did schoolwork
    • Looked for employment
    • Communicated with someone (ex., email, online chat, instant messaging, Facebook)
  • Assessing Outcomes Over Time
    • Select a sub-sample
    • Pre-/post-test
    • Case study
    • Tell the story behind the numbers
  • Benefits Beyond the Grant Period
  • Benefits Beyond the Grant Period
  • Benefits Beyond the Grant Period
    • Use results for budgeting, program planning, advocacy, etc.
    • Ongoing evaluation
  • Tools/Resources
    • Impact survey http://impactsurvey.org/
    • Colorado State Library’s BTOP site http://coloradovirtuallibrary.org/btop/
    • Computer reservations system http://code.google.com/p/powerline/
  • Questions? Submit your questions using the chat box.
  • Continue the Discussion…
    • Additional questions can be answered by posting in in our Community Forum:
      • http://bit.ly/tsg-pcc
  • Thank you to our Webinar Sponsor! ReadyTalk offers dedicated product demos for TechSoup organizations 4 times per week. For more information: pages.readytalk.com/techsoup.html
  • Thank you! Please complete the post event survey! Kami Griffiths, kami@ctnbayarea.org