National eXtension Conference Presentation by Barbara O'Neill, Molly Herndon, & Martie Gillen

1. Measuring Impact:
Methods and Metrics for
Face-to-Face Classes and
Social Media Outreach
Barbara O’Neill, Ph.D., CFP®, CRPC®
Distinguished Professor, Rutgers University
oneill@aesop.rutgers.edu

2. Are You Evaluating Your
Educational Programs?
If so, how?

3. The Million Dollar “So
What?” Question…
At the end of the day…did your educational
program make a difference?
How do you know?

4. • What gets measured gets funded
• With evaluation results, you can…
– assess impact of programs on learners
– see if you accomplished what you planned
– know if a program was “worth it”
– celebrate success and learn from failure
– make informed decisions to improve, hold, or fold programs
– promote your program and win public support
We are in an “Accountability Era”

5. Consider a Third Party
Program Evaluator
• Some funders require this (e.g., FINRA Investor
Education Foundation)
• No potential for conflict of interest by one entity
performing both the education and evaluation role
• Takes evaluation “monkey” off busy educators’ back
• Build into grant funding requests

6. • See http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
• Begins with the end in mind
• Explains what a program is and what it will accomplish
• Shows relationships between inputs, outputs, and outcomes
Introducing the Logic Model

7. Building a Strong
Financial Education Program
INPUTS OUTPUTS OUTCOMES
Program
investments
Activities Participation Short Medium
What
We
Invest
What
We Do
Who We
Reach
What Results
SO WHAT??
What is the VALUE?
Long-
term

8. Staff
Money
Time
Volunteers
Partners
Equipment/Technology
Policies
Research
INPUTS

9. OUTPUTS
What We Do Who We Reach
ACTIVITIES
Assess needs and assets
Design curriculum
Educate students
Conduct workshops
Facilitate learning groups
Sponsor conferences
Work with the media
Partner – collaborate
PARTICIPATION
Participants
Clients
Customers
Users
Groups
Reactions - Satisfaction

10. OUTCOMES
What Results for Individuals, Organizations, Communities..…
SHORT
Learning
Awareness
Knowledge
Attitudes
Skills
Opinion
Aspirations
Motivation
MEDIUM
Action
Behavior
Practice
Decisions
Policies
Social action
LONG-TERM
Conditions
Human
Economic
Civic
Environment

11. OUTCOMES
Commonly Measured Items
That Are Not Outcomes
• Participant satisfaction
• Number of people taught
• Units of education completed
• Number of events held
• Time and money spent
• Level of effort

12. Question
What are some outcomes for
your educational programs?

13. Impact Evaluation Data
Collection Methods
• Surveys (Paper or Online)
– Post-evaluation only (short programs)
– Pre- and post-evaluation
– Follow-up (e.g., 3 months later)
• Focus groups
• Interviews
• Observations
• Tests of knowledge/ability
• RARE: Control groups and longitudinal studies

14. Typical Survey Questions
• General reactions to the program
• Changes in knowledge
• Changes in motivation, confidence, and abilities
• Intended changes in behavior
• Actual changes in behavior
• Future programming needs and preferences
• Demographics of participants
• Qualitative/open-ended responses

15. Post-Then-Pre
(Retrospective) Evaluation
Method
• Compares knowledge and attitudes before and
after a financial education intervention
• Administered once at the end of the intervention
• Helps avoid biases due to people thinking they
know more than they know

16. Case Example:
Humpty Dumpty
“Humpty Dumpty sat on a wall,
Humpty Dumpty had a great fall,
All the king’s horses and all the king’s men
Couldn’t put Humpty together again”

17. So Who/What Was
Humpty Dumpty?
Probably
NOT what
you think!

18. Post-Then-Pre (Retrospective)
Evaluation Method
• Why? Because people don’t know what they don’t know!
• Who knew what Humpty Dumpty really was?
• Five point rating scale
– 1 = Strongly Disagree
– 5 = Strongly Agree
• Post: After listening to Barb, I know the history of the
Humpty Dumpty nursery rhyme
• Pre: Before listening to Barb, I knew the history of the
Humpty Dumpty nursery rhyme

19. Post-Then-Pre Evaluation
• Also known as a “Retrospective” evaluation
• Helps identify changes in knowledge, attitudes, and
behavior

20. Another Post-Then-Pre
Evaluation Example
Big Advantage: Avoids the problem of learners under-
estimating what they don’t know before a program

21. Critical Incident Technique
• Qualitative evaluation research method where
program participants tell personal stories
• Often used with “Train the Trainer” programs for
professionals (e.g., teachers, librarians)
• Incidents are categorized and deemed successful
(positive results) or unsuccessful (negative results)
http://www.joe.org/joe/2013june/tt2.php (Journal of Extension article)

22. More About the Critical
Incident Technique
http://www.adb.org/documents/information/knowledge-
solutions/the-critical-incident-technique.pdf
•Ask subjects to describe- through interviews-incidents that
they handled well or poorly (need not be spectacular events)
– Example: How did library staff handle patrons’ personal finance
questions?
•Provides rich personal perspectives
•Pre-intervention and Post-intervention comparison of
number and type of incidents

23. Critical Incident Technique
Example: NYPL Grant
Program Title: Money Matters

24. CIT Questions in NYPL
Financial Education
Evaluation
• Think about your experiences helping patrons with personal finance
questions. Remember a time when you had a successful experience
helping someone with these types of questions. Please write down what
happened.
• What made this a successful, positive experience?
• Think about your experiences helping patrons with personal finance
questions. Remember a time when you had an unsuccessful experience
helping someone with these types of questions. Please write down what
happened.
• What made this an unsuccessful or challenging experience?

25. CIT Summary in Final NYPL
Project Evaluation Report
“The change in the percentages of the types of incidents is indicative of
change in attitudes and abilities. Also, a new category appeared in the
successful incidents, Increased Knowledge/Confidence/Satisfaction,
which was not present in the Pre-training Survey. Furthermore, for the
unsuccessful critical incidents, the category Lack of Training/
Knowledge which was the most frequently seen theme for the Pre-
training Survey was not found at all in the Post-training Survey.
These findings resonate with quantitative findings and provide confirming
evidence that the Money Matters training has been successful in
improving participants’ knowledge of personal finance and their
ability to be successful in handling patron inquiries. Additionally, staff
members provided eloquent testament in their qualitative responses that
their skills have been enhanced and their attitudes and behaviors became
more positive for ably handling personal finance queries” (Radford, 2013).

26. Social Media
Evaluation
Methods

27. Triangulation (Multiple
Methods) Evaluation Approach
• Unique Twitter hashtag: #eXasw
• Follow-up follower/friend survey
• Follow-up project participant survey
• bit.ly analytics to determine number of clicks on unique
embedded links
• Pre- and post-ASW Twitter influence metrics (Klout score)

28. Klout Score Progression
• Klout is a measure of a person’s “influence” on twitter
• Based on an algorithm with factors such as number of
followers and number of retweets
• Go to www.klout.com and log in with Twitter or Facebook
– 2011: 11.22 to 19.68
– 2012: 20.3 to 29.3
– 2013: 32.6 to 39.6
– 2014: 29.76 to 38.84
– 2015: 23.8 to 30.7

29. Twitter Chat
Evaluation Methods
• Online survey (e.g., Qualtrics) link embedded into final tweets
(with prizes as an incentive to complete)
Dr. Barbara O'Neill @moneytalk1Apr 29
Please let us know if U found #SSHWchat helpful & take this brief survey:
https://rutgers.qualtrics.com/SE/?SID=SV_9X0V77kUKhOqlWB … Will pull winners at 1:30 pm #sshwchat
• TweetReach: http://tweetreach.com/ or Hashtracking:
https://www.hashtracking.com/ (type in hashtag to pull a report)
• Follow-up contact from participants (Twitter direct messages, e-
mail, etc.)
• Traffic to Web site during and after a chat
• Other?

30. Sample TweetReach
Report

31. Archive a Twitter Chat
With Storify and Track Use
https://storify.com/ Storify is an organized collection of tweets
with links, photos, etc. to “tell a story” from Twitter content
Video: https://www.youtube.com/watch?v=l9iHniFjiVc
Storify Samples:
https://storify.com/RutgersNJAES/small-steps-to-health-and-wealth
https://storify.com/RutgersSEBS/cook-douglass-community-day-2014-at-rutgers
https://storify.com/wisebread/how-are-you-saving-for-retirement
https://storify.com/JerryBuchko/mcpd2014-twitter-chat-archive-1

32. One Last Step:
Share Your Program
Evaluation Results
WHAT?
SO
WHAT?
NOW
WHAT?

33. Who Needs to Know
About the Results of
Your Educational
Programs? Why?

34. Impact Statements:
Intentions
As a result of participating in this financial education
program, X% of participants reported that they…
• plan to do/use/adopt…
• are more knowledgeable about…
• are more confident in their ability to…
• are more likely than before to do/use/adopt…
• will do/use/adopt…
…a particular attitude, piece of information, or behavior.

35. Social Media
Impact Reports
• Number of participants
• TweetReach report outreach numbers
• Participant survey data
• Data from other feedback methods
• Sample Evaluation Report:
http://www.slideshare.net/BarbaraONeill/ssh
w-twitter-chat-impact-statement-0414

36. Public Value
Approach
Private benefits accrue to program participants
Public benefits are benefits to those who did not
participate in a program
© 2007 Regents of the University of Minnesota

37. Public Value Statements
• Focus on an outcome that matters to
stakeholder(s)
• Use stakeholder’s language
• Avoid jargon and empty words
• Should be short and believable

38. Participants in a nutrition program…
have been shown to increase their
consumption of fruits and
vegetables…
which has been shown to reduce the
incidence of diabetes…
which directly improves
quality of life for
participants.
which reduces public
health costs for all
citizens.
© 2007 Regents of the University of Minnesota

39. Participants
Changes
Outcome
Private benefits Public value
© 2007 Regents of the University of Minnesota

40. Public Value
Statement Template
When you support _______________________program,
participants will ______________________________,
(changes)
which leads to _______________________________,
(outcomes)
which will benefit other community members by
_____________________________________________.
(public value)
© 2007 Regents of the University of Minnesota

41. Question
What are some public values
for your educational
programs?

42. • Launched in 2012
• Active online teaching community comprised of
– Cooperative Extension faculty
– Military family service professionals from the DoD, branch services
– Non-governmental organizations
• Federal government grant-funded program is a collaborative effort of
– eXtension
– Cooperative Extension
– DoD
– NIFA, USDA

43. Military Families Learning Network
Personal Finance (MFLNPF)
•Personal Finance Concentration Area
– Builds the financial management capability and
educational background of financial counselors
and educators

44. Military Families Learning Network
Personal Finance
• Monthly web conferences offer credentialed
participants the opportunity to earn continuing
education unit (CEU) credit at no cost.
• Since 2012, the MFLNPF Concentration Area has
provided certificates for almost 8,000 CEUs to
personal finance web conference participants
through the delivery of 61 online web
conferences on a wide variety of personal finance
topics.

45. • Link for webinars presented by all MFLN concentration areas:
http://www.extension.org/pages/62581/military-families-learning-network-
webinars
–Seven topic areas including Personal Finance
• Webinars are free and require no advance registration
• All webinars are marketed and archived on an eXtension Learn page
–Example: https://learn.extension.org/events/2149
MFLN Webinars

46. 2012-2015 MFLNPF Webinars
• All MFLNPF webinars are archived at
https://learn.extension.org/events/tag/mfln%20personal%20fin
ance
• First webinar: 2/2/12- Housing and Foreclosure for Military
Families
• Last 2015 webinar: 12/1/15- 2015 Personal Finance Year in
Review
• Total of 56 webinars in four years (14 webinars per year)

47. More About MFLNPF Webinars
• Received 2014 AFCPE Outstanding Educational
Program Award
• Often include two presenters on a topic
–One for a general overview and one
for military-specific applications
• Available online 24/7/365
• CEU credit available upon 80% score on a 10-
question quiz
Image derived from photo by
Dr. Irene Leech, Virginia Tech

48. Webinar Evaluation
48

49. Webinar Evaluation
49

50. Webinar Evaluation
50

51. Measuring Virtual Engagement
PF SMS iconsPF SMS icons
51
 Facebook Insights
 Sprout Social
 Google Analytics

52. PF SMS iconsPF SMS icons
52
Facebook Insights

53. PF SMS iconsPF SMS icons
53
Sprout Social

54. PF SMS iconsPF SMS icons
54
Google Analytics

55. PF SMS iconsPF SMS icons
55
 Social Media measurement tools can help
online communities of practice measure their
impact within their communities.
 This data is valuable to stakeholders who wish
to see proof of interaction and engagement
among online community members.
 Many social media measurement tools are free
and easy to use.

56. Questions and
Comments?
Barbara O'Neill, Ph.D., CFP®, CRPC
Extension Specialist in Financial Resource Management
and Distinguished Professor, Rutgers University
Phone: 848-932-9126
E-mail: oneill@aesop.rutgers.edu
Internet: http://njaes.rutgers.edu/money/
Twitter: http://twitter.com/moneytalk1