Lessons learned from eXtension cop evaluation efforts.netc.2010


Published on

NETC 2010 Conference Presentation

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Lessons learned from eXtension cop evaluation efforts.netc.2010

  1. 1. Lessons Learned from eXtension CoP Evaluation Efforts Michael Lambur eXtension Evaluation and Research Leader
  2. 2. Overall Lessons Learned <ul><li>Posting content to a website does not ensure outcomes/impact (i.e., If we build it they will come) </li></ul><ul><li>CoP’s who have documented outcomes/impact to date have engaged with their clientele—internal and external </li></ul><ul><li>It’s all about good program development using a new medium </li></ul>
  3. 3. CoP Evaluation Efforts <ul><li>Entrepreneurs and their Communities </li></ul><ul><ul><li>Surveyed CoP members on five key resource areas: case studies, information briefs, state and local policies, research references, and webinars </li></ul></ul><ul><ul><li>Most planned to use the website for information for public presentations, articles, and other reports </li></ul></ul><ul><ul><li>86% of the respondents rated webinars as useful or very useful </li></ul></ul>
  4. 4. CoP Evaluation Efforts <ul><li>Financial Security for All </li></ul><ul><ul><li>Posted a survey link on all their FAQ’s </li></ul></ul><ul><ul><li>27 people responded </li></ul></ul><ul><ul><li>70.4% indicated that the FAQ provided information that was useful or very useful </li></ul></ul><ul><ul><li>37% indicated that the FAQ helped them to learn quite a bit more or a lot more new information </li></ul></ul><ul><ul><li>66.7% indicated that that they may put or definitely plan to put the information to use </li></ul></ul>
  5. 5. CoP Evaluation Efforts <ul><li>HorseQuest </li></ul><ul><ul><li>Learning Modules </li></ul></ul><ul><ul><ul><li>Reported a 17% increase in knowledge scores pre to post </li></ul></ul></ul><ul><ul><ul><li>79% indicated a willingness to recommend Learning Modules to others </li></ul></ul></ul>
  6. 6. CoP Evaluation Efforts <ul><li>HorseQuest </li></ul><ul><ul><li>Webcasts </li></ul></ul><ul><ul><ul><li>17 webcasts offered attended by 600 people </li></ul></ul></ul><ul><ul><ul><li>58% rated them as very useful </li></ul></ul></ul><ul><ul><ul><li>Reported a 15% increase in knowledge scores pre to post </li></ul></ul></ul><ul><ul><ul><li>76% planned to make at least one change in their horse management practices </li></ul></ul></ul><ul><ul><ul><li>Archived webcasts are being used in local programming </li></ul></ul></ul>
  7. 7. CoP Evaluation Efforts <ul><li>HorseQuest </li></ul><ul><ul><li>YouTube </li></ul></ul><ul><ul><ul><li>111 video clips have been posted to the HorseQuest YouTube channel </li></ul></ul></ul><ul><ul><ul><li>Videos have been viewed 125,277 times </li></ul></ul></ul><ul><ul><ul><li>The channel has 310 individual subscribers </li></ul></ul></ul><ul><ul><ul><li>68% of the videos have been rated, with 75% receiving the top rating of 5 stars </li></ul></ul></ul><ul><ul><ul><li>There have been 239 self-referrals from the channel to the eXtension HorseQuest website, with an average of 6.81 page views per visit </li></ul></ul></ul>
  8. 8. CoP Evaluation Efforts <ul><li>Imported Fire Ants </li></ul><ul><ul><li>Conducted an eXtension adoption survey at the Alabama Association of County Agriculture Agents and Specialists </li></ul></ul><ul><ul><li>74 individuals responded to the survey </li></ul></ul><ul><ul><li>69% indicated that they were familiar with eXtension; 58% indicated that they had used eXtension resources </li></ul></ul><ul><ul><li>21 indicated that the Imported Fire Ants resource area was very useful </li></ul></ul>
  9. 9. CoP Evaluation Efforts <ul><li>Just in Time Parenting </li></ul><ul><ul><li>Parenting Newsletter </li></ul></ul><ul><ul><ul><li>Parents subscribe and receive a monthly newsletter during the first year after their child’s birth </li></ul></ul></ul><ul><ul><ul><li>After they receive their 12 th newsletter, parents will be invited to complete an online survey to document how much they have learned, ways they have changed their behavior, and satisfaction with JITP </li></ul></ul></ul><ul><ul><ul><li>Based on pilot study results, 81% (n=738) who opened messages clicked forward to the appropriate newsletter </li></ul></ul></ul>
  10. 10. CoP Evaluation Efforts <ul><li>Livestock and Poultry Environmental Learning Center </li></ul><ul><ul><li>Newsletter </li></ul></ul><ul><ul><ul><li>Online survey completed by 147 newsletter subscribers working with livestock and poultry producers </li></ul></ul></ul><ul><ul><ul><li>64% rated webcasts, 52% rated subject matter pages, 46% rated the website, and 46% rated the newsletter as very useful </li></ul></ul></ul>
  11. 11. CoP Evaluation Efforts <ul><li>Livestock and Poultry Environmental Learning Center </li></ul><ul><ul><ul><li>62% indicated that the website was better than others in the quality of information provided </li></ul></ul></ul><ul><ul><ul><li>Knowledge gains were documented </li></ul></ul></ul><ul><ul><ul><li>50% of livestock and poultry producers reported that they had implemented one or more new practices to protect air or water quality </li></ul></ul></ul><ul><ul><ul><li>70% of county Extension agents used the website </li></ul></ul></ul><ul><ul><ul><li>84% of Extension specialists used the website to research an answer for a producer or agent </li></ul></ul></ul>
  12. 12. So Where Do We Go From Here? <ul><li>Using social media applications to engage clientele (e.g., Twitter, Facebook, YouTube, Blogs, etc.) </li></ul>
  13. 13. Strategy Blocks and Social Media Tools (Kanter, 2010)
  14. 14. Evaluating Social Media Applications <ul><li>Engagement </li></ul><ul><li>Forrester Research (Villa, 2009): </li></ul><ul><ul><li>Involvement: site visitors, time spent, page views, reach, frequency </li></ul></ul><ul><ul><li>Interaction: contributions to blogs, content creation and uploads </li></ul></ul><ul><ul><li>Intimacy: consumer attitudes, perceptions, and feelings </li></ul></ul><ul><ul><li>Influence: likelihood that consumers will recommend or advocate </li></ul></ul>
  15. 15. Evaluating Social Media Applications <ul><li>Papadakis (Kaushik, 2008): </li></ul><ul><ul><li>Kind: the degree to which customers are positively or negatively engaged </li></ul></ul><ul><ul><li>Degree: the degree to which customers are involved—low to high </li></ul></ul>
  16. 16. Evaluating Social Media Applications <ul><li>Involvement and interaction/degree can be measured with analytics (quantitative “head” data) </li></ul><ul><li>Intimacy and influence/kind need to come directly from the customer (qualitative “heart” data) </li></ul>
  17. 17. Evaluating Social Media Applications <ul><li>Involvement and interaction/degree: </li></ul><ul><ul><li>GA: unique visits, frequency of visits, recency of visit, depth of visit, length of visit, time spent on site </li></ul></ul><ul><ul><li>Twitter: Following, followers, tweets, Tweetrush, Tweetlists, Tweetvolume </li></ul></ul><ul><ul><li>Facebook: Fans, profile data, wall postings, events </li></ul></ul><ul><ul><li>YouTube: Views, ratings, links, audiences </li></ul></ul>
  18. 18. Evaluating Social Media Applications <ul><li>Intimacy and influence/kind: </li></ul><ul><ul><li>Comment analysis </li></ul></ul><ul><ul><li>Content analysis of the “conversation” </li></ul></ul><ul><ul><li>Evaluative questions asked in the conversation </li></ul></ul><ul><ul><li>Links to online surveys </li></ul></ul>
  19. 19. References <ul><li>Kanter, Beth. (2010). 9 ways nonprofits can excel using social media. http://www.bethkanter.org/socialmedia-nonprofits-excel/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+bethblog+%28Beth%27s+Blog%29 </li></ul><ul><li>Kaushik, A. (2008). Measuring online engagement: What role does web analytics play? http://www.kaushik.net/avinash/2008/01 </li></ul><ul><li>Villa, J. (2009). What does engagement really mean? http://www.mediapost.com/publications/index.cfm?fa=Articles.showArticle&art_aid=111974&passFuseAction=PublicationsSearch.showSearchReslts&art_searched=jose%20villa&page_number=11 </li></ul>