1. Personal Finance
Outcome Assessments:
Methods and Metrics for
Face-to-Face Classes and
Social Media Outreach
Barbara O’Neill, Ph.D., CFP®, CRPC®
Distinguished Professor, Rutgers University
oneill@aesop.rutgers.edu
3. The Million Dollar “So
What?” Question…
At the end of the day…did your financial
education program make a difference?
How do you know?
4. The Current State of Financial
Education Program Evaluation
• Current evaluation efforts are still far from
satisfactory
• General lack of evaluation capacity
• Evaluation is often treated as an after-thought
• Outcomes (e.g., measures of changed behavior) are
often confused with program outputs (e.g., number
of participants)
5. • What gets measured gets funded
• With evaluation results, you can…
– assess impact of programs on learners
– see if you accomplished what you planned
– know if a program was “worth it”
– celebrate success and learn from failure
– make informed decisions to improve, hold, or fold programs
– promote your program and win public support
We are in an “Accountability Era”
6. Consider a Third Party
Program Evaluator
• Some funders require this (e.g., FINRA Investor
Education Foundation and IMLS)
• No potential for conflict of interest by one entity
performing both education and evaluation role
• Takes evaluation “monkey” off busy educators’ back
• Build into grant funding requests
8. Building a Strong
Financial Education Program
INPUTS OUTPUTS OUTCOMES
Program
investments
Activities Participation Short Medium
What
We
Invest
What
We Do
Who We
Reach
What Results
SO WHAT??
What is the VALUE?
Long-
term
10. OUTPUTS
What We Do Who We Reach
ACTIVITIES
Assess needs and assets
Design curriculum
Educate students
Conduct workshops
Facilitate learning groups
Sponsor conferences
Work with the media
Partner – collaborate
PARTICIPATION
Participants
Clients
Customers
Users
Groups
Reactions - Satisfaction
11. OUTCOMES
What Results for Individuals, Organizations, Communities..…
SHORT
Learning
Awareness
Knowledge
Attitudes
Skills
Opinion
Aspirations
Motivation
MEDIUM
Action
Behavior
Practice
Decisions
Policies
Social action
LONG-TERM
Conditions
Human
Economic
Civic
Environment
12. OUTCOMES
Commonly Measured Items
That Are Not Outcomes
• Participant satisfaction
• Number of people taught
• Units of education completed
• Number of events held
• Time and money spent
• Level of effort
14. Impact Evaluation Data
Collection Methods
• Surveys (Paper or Online)
– Post-evaluation only (short programs)
– Pre- and post-evaluation
– Follow-up (e.g., 3 months later)
• Focus groups
• Interviews
• Observations
• Tests of knowledge/ability
• RARE: Control groups and longitudinal studies
15. Typical Survey Questions
• General reactions to the program
• Changes in knowledge
• Changes in motivation, confidence, and abilities
• Intended changes in behavior
• Actual changes in behavior
• Future programming needs and preferences
• Demographics of participants
• Qualitative/open-ended responses
16. Post-Then-Pre
(Retrospective) Evaluation
Method
• Compares knowledge and attitudes before and
after a financial education intervention
• Administered once at the end of the intervention
• Helps avoid biases due to people thinking they
know more than they know
17. Case Example:
Humpty Dumpty
“Humpty Dumpty sat on a wall,
Humpty Dumpty had a great fall,
All the king’s horses and all the king’s men
Couldn’t put Humpty together again”
19. Post-Then-Pre (Retrospective)
Evaluation Method
• Why? Because people don’t know what they don’t know!
• Who knew what Humpty Dumpty really was?
• Five point rating scale
– 1 = Strongly Disagree
– 5 = Strongly Agree
• Post: After listening to Barb, I know the history of the
Humpty Dumpty nursery rhyme
• Pre: Before listening to Barb, I knew the history of the
Humpty Dumpty nursery rhyme
20. Post-Then-Pre Evaluation
• Also known as a “Retrospective” evaluation
• Helps identify changes in knowledge, attitudes, and
behavior
22. Critical Incident Technique
• Qualitative evaluation research method where
program participants tell personal stories
• Often used with “Train the Trainer” programs for
professionals (e.g., teachers, librarians)
• Incidents are categorized and deemed successful
(positive results) or unsuccessful (negative results)
http://www.joe.org/joe/2013june/tt2.php (Journal of Extension article)
23. More About the Critical
Incident Technique
http://www.adb.org/documents/information/knowledge-
solutions/the-critical-incident-technique.pdf
• Ask subjects to describe- through interviews-incidents that
they handled well or poorly (need not be spectacular events)
– Example: How did library staff handle patrons’ personal finance
questions?
• Provides rich personal perspectives
• Pre-intervention and Post-intervention comparison of
number and type of incidents
25. CIT Questions in NYPL
Financial Education
Evaluation
• Think about your experiences helping patrons with personal finance
questions. Remember a time when you had a successful experience
helping someone with these types of questions. Please write down what
happened.
• What made this a successful, positive experience?
• Think about your experiences helping patrons with personal finance
questions. Remember a time when you had an unsuccessful experience
helping someone with these types of questions. Please write down what
happened.
• What made this an unsuccessful or challenging experience?
26. CIT Summary in Final NYPL
Project Evaluation Report
“The change in the percentages of the types of incidents is indicative of
change in attitudes and abilities. Also, a new category appeared in the
successful incidents, Increased Knowledge/Confidence/Satisfaction,
which was not present in the Pre-training Survey. Furthermore, for the
unsuccessful critical incidents, the category Lack of Training/
Knowledge which was the most frequently seen theme for the Pre-
training Survey was not found at all in the Post-training Survey.
These findings resonate with quantitative findings and provide confirming
evidence that the Money Matters training has been successful in
improving participants’ knowledge of personal finance and their
ability to be successful in handling patron inquiries. Additionally, staff
members provided eloquent testament in their qualitative responses that
their skills have been enhanced and their attitudes and behaviors became
more positive for ably handling personal finance queries” (Radford, 2013).
28. America Saves Week (ASW)
Social Media Campaign
• Conducted by eXtension (online program of the
Cooperative Extension System), 2011-2015
• Prepared “cut and paste” tweets and FB messages on
saving, debt/expense reduction, America Saves, ASW
• Project cooperators from eXtension, America Saves
campaigns, military, others
29. Triangulation (Multiple
Methods) Evaluation
Approach
• Unique Twitter hashtag: #eXasw
• Follow-up follower/friend survey
• Follow-up project participant survey
• bit.ly analytics to determine number of clicks on unique
embedded links
• Pre- and post-ASW Twitter influence metrics (Klout score)
31. Follower/Friend Survey
Results: 2012 Example
• Response: 50% “very helpful” and 50% “helpful”
• 54% visited ASW links; 38% planned to
• 8% joined AS; 38% planned to; 16% already joined AS;
38% had not joined AS
• Mostly positive comments about SM messages
• One negative comment about seeing repeated messages
and the need for more action steps
32. Professional Collaborator
Survey Results: 2012 Example
• 22/22 Klout scores increased
– Average from 20.3 to 29.3 (vs. 11.22 to 19.68 in 2011)
• Positive comments (e.g., wanted more “cut and
paste” messages, improved SM skills, ease of use,
continue in 2013)
• State Governor and Senator started following one
participant
• Reported likes, retweets, and direct messages
33. Klout Score Progression
• Klout is a measure of a person’s “influence” on twitter
• Based on an algorithm with factors such as number of
followers and number of retweets
• Go to www.klout.com and log in with Twitter or Facebook
– 2011: 11.22 to 19.68
– 2012: 20.3 to 29.3
– 2013: 32.6 to 39.6
– 2014: 29.76 to 38.84
34. Project Take-Aways
• Social media messages are great for increasing
awareness but do not appear to prompt much action
• Project participants like having “cut and paste” social
media messages
• Twitter influence (Klout score) of project participants
became noticeably higher over time
• Social media is an effective outreach tool
35. Twitter Hashtags
• A key component of Twitter use is the hashtag (#)
symbol followed by a series of unique characters
• The hashtag serves like a “magnet” to link together the
tweets of Twitter users who are interested in the same
topic (e.g., credit) or who are linked together in some
way (e.g., professional conference attendees)
• By using a hashtag, users can effectively hold a
“conversation” with one another about a topic of
interest
36. Twitter Chats
• Online users gather at a specified time (e.g., every
Friday at 3 pm ET) to “discuss” (read: tweet about)
certain topics, questions, or issues.
• Synchronous learning environment where participants
interact in real time using their Twitter user names
(a.k.a.,”handles”).
• As with tweets at conferences, the “glue” that holds
Twitter chats together is the designated hashtag
• Twitter chat hashtag creates a running stream of
tweets so that everyone can follow the “conversation.”
37. More About Twitter Chats
• Chat hashtags are often derived from a chat sponsor’s
name (example: #wbchat for WiseBread); industry
(example: #creditchat); or program title (#SSHWchat
for the Small Steps to Health and Wealth™ program)
• A Twitter chat hashtag creates a running stream of
tweets so that everyone can follow the “conversation”
• The formatting convention used to follow Twitter chat
threads is Q1 for Question 1 and A1 for participant
responses to that question
38. Twitter Chat
Evaluation Methods
• Online survey (e.g., Qualtrics) link embedded into final tweets
(with prizes as an incentive to complete)
Dr. Barbara O'Neill @moneytalk1Apr 29
Please let us know if U found #SSHWchat helpful & take this brief survey:
https://rutgers.qualtrics.com/SE/?SID=SV_9X0V77kUKhOqlWB … Will pull winners at 1:30 pm #sshwchat
• TweetReach statistics: http://tweetreach.com/ (type in chat
hashtag to pull a report)
• Follow-up contact from participants (Twitter direct messages,
e-mail, etc.)
• Traffic to Web site during and after a chat
• Other?
40. #eXASchat
Evaluation Results
• Four consecutive Monday evenings in February 2014 to
promote America saves and saving money
• Hashtag: #eXASchat
• 69 participants
• Estimated reach: 42,838 (estimated overall reach of tweets in report)
• Exposure impressions: 249,382 (based on participants’
followers; tells how many tweets were sent by users to their followers)
• Number of tweets sent by users with > 1,000 followers
(96th percentile of all Twitter users): 38
41. #SSHWchat
Evaluation Results
• Held at noon ET on 4/29/14 for Financial Literacy Month
• Topic: Strategies to improve health and finances
• 34 participants
• Estimated reach: 25,294 (estimated overall reach of tweets in report)
• Exposure impressions: 50,553
• Six-question Qualtrics survey: 29% response rate
– 100% of respondents reported learning something new
– 100% of respondents reported planned behavior changes
• Comment: “Please have one monthly, if possible”
42. Archive a Twitter Chat
With Storify and Track Use
https://storify.com/ Storify is an organized collection of tweets
with links, photos, etc. to “tell a story” from Twitter content
Video: https://www.youtube.com/watch?v=l9iHniFjiVc
Storify Samples:
https://storify.com/RutgersNJAES/small-steps-to-health-and-wealth
https://storify.com/RutgersSEBS/cook-douglass-community-day-2014-at-rutgers
https://storify.com/wisebread/how-are-you-saving-for-retirement
https://storify.com/JerryBuchko/mcpd2014-twitter-chat-archive-1
44. Who Needs to Know
About the Results of
Your Financial Education
Programs? Why?
45. Tips for Telling Your Story
• Use simple descriptive statistics (e.g., counts,
percentages, and averages)
• Don’t use jargon
• Don’t overstate your results (e.g., causality)
• Blend quantitative and qualitative data
• Clearly describe who the results represent (i.e.,
demographic characteristics of participants)
• Be honest about your program’s strengths and
weaknesses, while highlighting the positive
46. Impact Statements:
Intentions
As a result of participating in this financial education
program, X% of participants reported that they…
• plan to do/use/adopt…
• are more knowledgeable about…
• are more confident in their ability to…
• are more likely than before to do/use/adopt…
• will do/use/adopt…
…a particular attitude, piece of information, or behavior.
47. Impact Statements:
Actions
As a result of participating in this financial education
program, X% of participants reported that they…
• are now doing…
• did…
• used…
• increased their knowledge of…
• adopted…
• changed…
… a particular attitude, piece of information, or behavior.
48. Social Media
Impact Reports
• Number of participants
• TweetReach report outreach numbers
• Participant survey data
• Data from other feedback methods
• Sample Evaluation Report:
http://www.slideshare.net/BarbaraONeill/ssh
w-twitter-chat-impact-statement-0414
49. Public Value
Approach
Private benefits accrue to program participants
Public benefits are benefits to those who did not
participate in a program
It is possible to exclude non-payers from
participating in programs—and enjoying private
benefits—but it is not possible to exclude non-payers
from receiving public benefits of programs
50. How Do Financial
Education Programs
Create Public Value?
Does it narrow an information gap?
Does it address a crucial concern about fairness?
Does one person’s participation benefit people who do
not participate in the program?
Does one person’s participation reduce costs for
others?
Does the program improve upon market outcomes?
51. Program
participants…
have been shown to change their
behavior in specific ways…
that have been shown to lead to
specific outcomes…
that directly benefit
the participants.
that generate public
value.
52. Participants in a nutrition program…
have been shown to increase their
consumption of fruits and
vegetables…
which has been shown to reduce the
incidence of diabetes…
which directly improves
quality of life for
participants.
which reduces public
health costs for all
citizens.
54. Public Value Statements
• Focus on an outcome that matters to
stakeholder(s)
• Use stakeholder’s language
• Avoid jargon and empty words
• Should be short and believable
55. Public Value
Statement Template
When you support _______________________program,
participants will ______________________________,
(changes)
which leads to _______________________________,
(outcomes)
which will benefit other community members by
_____________________________________________.
(public value)
56. Activity #2
Write a public value
statement for your financial
education program?
60. Questions and
Comments?
Barbara O'Neill, Ph.D., CFP®, CRPC
Extension Specialist in Financial Resource Management
and Distinguished Professor, Rutgers University
Phone: 848-932-9126
E-mail: oneill@aesop.rutgers.edu
Internet: http://njaes.rutgers.edu/money/
Twitter: http://twitter.com/moneytalk1