Presentation from Dr. Saugato Datta, managing director of Ideas 42, U.S.

1. Saugato Datta
Managing Director, ideas42
Behaviorally Optimizing Higher Education:
Improving Student Success through
Behavioral Science
ESF Flanders Conference on Social Innovation
Brussels, Belgium. October 25-26, 2015
Follow us on Twitter
@ideas42

2. agenda for session
2
esf flanders conference
on social innovation
Day 1: Introduction to Behavioral Science & ideas42
• Why behavioral insights in Post-Secondary Education?
• Case Study 1: Increasing On-Time Financial Aid Applications at Arizona
State
Key Psychology: Social Norms
• Case Study 2: Increasing Uptake of Tutoring Services at West Kentucky
Community and Technical College
Key Psychologies: Time Inconsistency and Procrastination

3. agenda for session
3
esf flanders conference
on social innovation
Day 2
• Recap of Cases
• How did we get here? Overview of ideas42 RFP Process in Post-
Secondary Education
• Discussion: What do we need for innovation?

4. first, a small quiz

5. © 2015 ideas42 5
question 1
A cup and a saucer cost $1.10 in total. The
cup costs a dollar more than the saucer.
How much does the saucer cost?
cents

6. © 2015 ideas42 6
question 2
​If it takes 5 machines 5 minutes to make 5
widgets, how long will it take 100 machines
to make 100 widgets?
minutes

7. © 2015 ideas42 7
question 3
​In a pond, there is a patch of lily pads.
Every day, the patch doubles in size. If it
takes 24 days for the patch to cover the
entire pond, how long would it take for the
patch to cover half the pond?
days

8. © 2015 ideas42 8
quiz answers
• A cup and a saucer cost $1.10 in total. The cup
costs a dollar more than the saucer.
How much does the saucer cost?
• If it takes 5 machines 5 minutes to make 5
widgets, how long will it take 100 machines to
make 100 widgets?
• In a pond, there is a patch of lily pads. Every day,
the patch doubles in size. If it takes 24 days for
the patch to cover the entire pond, how long
would it take for the patch to cover half the pond?
$0.05 $0.10
5 min 100 min
23 days 12 days
incorrect
(but illuminating)correct

9. 9

10. © 2015 ideas42 10
do these gentlemen want to get fit?

11. © 2015 ideas42 11
what do we do?
Use the theories of behavioral science to
design solutions to some of the world’s most
persistent social problems.

12. © 2015 ideas42 12
With many partners in many different domains…

13. © 2015 ideas42 13
In many countries across the world…

14. © 2015 ideas42 14
ideas42 projects around the world

15. © 2015 ideas42 15
Including internal government ‘nudge units’
Operational/Past:
• Chicago Nudge Unit
• 4 staff seconded to White House Social and
Behavioral Sciences Team
• Western Cape, South Africa
In development:
• Advisory role in World Bank’s new Global
Insights Unit (GINI)
• NYC and Boston nudge units

16. © 2015 ideas42 16
why work in post-secondary education?

17. © 2015 ideas42 17
Just 59% of full-time, four-year college students
graduate within six years
Source: https://nces.ed.gov/fastfacts/display.asp?id=40

18. © 2015 ideas42 18
Just 29% of students in full-time community college
students graduate within 1.5x the standard time period
Source: http://nces.ed.gov/programs/coe/indicator_cva.asp

19. © 2015 ideas42 19
College is hard.
(in ways we overlook)

20. to understand why, a small (fun) test

27. BLUE

28. why do behavioral work in post-secondary education?
Student retention and performance depend on
behavioral phenomena: self-control, attention,
cognitive bandwidth, salience, etc….

29. self control is hard

30. 30©2013 ideas42

31. © 2015 ideas42 31
what we dowhat we intend ≠
Typically, when a
program seeks
to change this
We design it
to change this
We actually
need to
change this
limited self-control, inattention, status quo
bias, etc. mean that…

32. © 2015 ideas42 32
which may explain why only 29% of full-time community college
students graduate within 1.5x the standard time period
Source: http://nces.ed.gov/programs/coe/indicator_cva.asp

33. © 2015 ideas42 33
work in post-secondary education
Using the theories of behavioral science to
improve student outcomes in colleges and
universities across the US.

34. © 2015 ideas42 34
EXAMPLE – FAFSA FORMS
Traditional perspective:
- They don’t want to go to college
- They can’t afford it
- They don’t understand its value, etc.
Behavioral perspective:
- FAFSA forms are too hard
- They make families feel too “unsophisticated” to
go to college
Problem: Students are not applying for
financial aid
©2013 ideas42 34

35. © 2015 ideas42 35© 2015 ideas42 35
Increasing FAFSA Applications with
Behavioral Design
​Spring Semester 2015

36. © 2015 ideas42 36
1. In 2011-2013 http://money.cnn.com/2014/03/07/pf/college/fafsa-financial-aid/
Problem: only 18% of students file the FAFSA by ASU’s
priority deadline, and many eligible students never file
• Priority filers are guaranteed maximum
aid package
• Nationally, priority filers are offered 2x as
much aid as those who apply later
• Nationally, at least 2 million students did
not receive grants they qualify for
because they did not file the FAFSA1

37. © 2015 ideas42 37
Diagnosed six primary barriers to file from survey,
literature review, and conversations with stakeholders
Students do not understand
what information they need
to file the FAFSA
Students do not adequately
plan to collect the
information they’ll need
(once they know)
Parent information is
difficult to obtain and may
require significant effort by
both student and parent
1
2
3
Priority deadline not
salient at the right time
Inaccurate mental model of
who receives financial aid
Misperceived social norms
of how many students
submit the FAFSA
4
5
6

38. © 2015 ideas42 38
beforeprioritydeadline
Intervention 1: students Intervention 2: parents
• Population: 63,000 continuing students
• Treatment group: 8 behavioral emails sent to students
- 3 sub-treatments compared BE email, short BE
email, and peer BE email
• Control group: 1 standard ASU email sent to students*
• Population: 22,000 continuing students with parent
emails on file
• Treatment group: 2 behavioral emails sent to parents
• Control group: no communications*
RCT evaluated behavioral communications to students
and parents before priority deadline
*Note this is standard protocol for ASU

39. © 2015 ideas42 39
Intervention 1: design targeted identified barriers to
action for students

40. © 2015 ideas42 40
Intervention 1: design targeted identified barriers to
action for students
Students do not understand what
information they need to file the FAFSA
Students do not adequately plan to
collect the information they’ll need (once
they know)
Parent information is difficult to
obtain and may require significant effort
on part of both student and parent
1
2
3
3
2
2
1

41. © 2015 ideas42 41
Intervention 1: design targeted identified barriers to
action for students
4
Priority deadline not salient at the right time
4
Inaccurate mental model of who receives
financial aid
5
Misperceived social norms of how many
students submit the FAFSA
6
4
5
6

42. © 2015 ideas42 42
Intervention 2: design targeted identified barriers to
action for parents
4
1
3
Students do not understand what
information they need to file the FAFSA
Students do not adequately plan to
collect the information they’ll need (once
they know)
Parent information is difficult to
obtain and may require significant effort
on part of both student and parent
1
2
3

43. © 2015 ideas42 43
Intervention 2: design targeted identified barriers to
action for parents
4
4
6
5
Priority deadline not salient at the right time
Inaccurate mental model of who receives
financial aid
Misperceived social norms of how many
students submit the FAFSA
4
5
6

44. © 2015 ideas42 44
**indicates significance at the 95% level
Interventions 1 and 2 increased priority FAFSA filers by
as much as 72%
29%
40%
44%
50%
0%
10%
20%
30%
40%
50%
60%
Control Student emails only
(averaged)
Parent emails only Both student and parent
emails
Number of Priority FAFSA Filers by Treatment Condition
(Interventions 1 & 2)
**
**
**+38%
+52%
+72%

45. © 2015 ideas42 45
**Indicates significance at the 95% level
29%
40%
44%
50%
0%
10%
20%
30%
40%
50%
60%
Control Student emails only
(averaged)
Parent emails only Both student and parent
emails
Number of FAFSA Filers by Treatment Condition
(at End of Study Period )
** **
**
Interventions increased priority FAFSA filers by up to 72%
+72%
*No difference in Intervention 1 between behavioral email conditions; all
increased relative to control

46. © 2015 ideas42 46
Insights for future projects
• Email intervention was effective, nearly doubling rate of filing by priority deadline
and increasing applications overall
• Parents are a valuable communication channel; merit careful consideration of
when to involve them
• Despite aggressive campaign, 66% of students said they received “the right
amount” of emails; only 7% reported getting “too many”
• Project is scalable and cost-effective (vs $90 H&R block experiment)

47. © 2015 ideas42 47 47©2013 ideas42

48. © 2015 ideas42 48
IMPLICIT NORMS
©2013 ideas42 48

49. © 2015 ideas42 49
NEGATIVE SOCIAL NORMS
©2013 ideas42 49

50. © 2015 ideas42 50
PERCEPTIONS OF SOCIAL NORMS
©2013 ideas42 50

51. © 2015 ideas42 51
FEATURES OF SOCIAL NORMS
Peer groups
reinforce norms
Visibility of
behavior is key
Framing
matters
©2013 ideas42 51

52. © 2015 ideas42 52
Framing
matters
FRAME THE BEHAVIOR
©2013 ideas42 52

53. © 2015 ideas42 53
FRAME THE BEHAVIOR
Framing
matters
©2013 ideas42 53

54. © 2015 ideas42 54
Visibility of
behavior is key
FOCUS ON THE RIGHT BEHAVIOR
©2013 ideas42 54

55. © 2015 ideas42 55
Visibility of
behavior is key
FOCUS ON THE RIGHT BEHAVIOR
©2013 ideas42 55

56. © 2015 ideas42 56
DEFINING THE PEER GROUP
Peer groups
reinforce norms
©2013 ideas42 56

57. © 2015 ideas42 57
DEFINING THE PEER GROUP
“A lot of people I know
don’t go to college”
“66% of high school seniors
go to college”
©2013 ideas42 57

58. © 2015 ideas42 58
GROUP
ACTIVITY #1
• Think of a time when you started something new and at
first felt like you didn’t fit in (immediately felt that you fit in).
• Take your favorite of the above examples and dig a little
deeper into that situation. What were the details (location,
mood, surroundings, actions of others) that prevented you
from feeling like you fit in (helped you feel you belonged)?
• Discuss with your group.
• Can you think of a situation faced by the population your
organization serves where a similar issue might exist?

59. © 2015 ideas42 59© 2015 ideas42 59
Increasing Uptake of On-campus
Tutoring at West Kentucky
Community and Technical College
​Dana Guichon, Andrew White, Katy Davis, and Piyush Tantia
​2014-2015

60. © 2015 ideas42 60
Problem: students do not use academic support services
(tutoring) provided by the college
• 35% of students fail at least 1 class per semester
• 20% of students withdraw from at least 1 class per
semester
High rates or students failing or withdrawing
from classes
Low utilization of academic support services at
the Tutoring Center
• 3-4% of enrolled students use tutoring per semester

61. © 2015 ideas42 61
Availability of tutoring was not salient to students.
Feedback on coursework provided too late, thus students do not seek academic help at
the optimal time.
Students hold misperceptions around tutoring and/or the Tutoring Center.
Steps associated with scheduling a tutoring appointment create hassles.
Faculty play a large role in guiding students toward tutoring, yet few students receive
actionable encouragement from professors.
Diagnosis focused on five factors limiting tutoring
uptake

62. © 2015 ideas42 62
Randomized Trial 1 – Fall 2014
4,455
degree-seeking
students
treatment
4 ideas42
emails
industry
standard
1 Starfish
template email
control
No emails
randomly sorted

63. © 2015 ideas42 63
Behaviorally designed treatment emails targeted identified
diagnoses
Reduce hassles associated with scheduling
a tutoring appointment by providing clear
and actionable information
1
2
Correct misperceptions around tutoring
3 Set a deadline so that students seek tutoring
at the optimal time

64. © 2015 ideas42 64
Intervention led more students to start going to tutoring
and doubled the number of tutoring sessions attended
*Significant to the 90% level; **Significant to the 95% level
Results look at students that started to go to tutoring on or after 10/14, which was the date the intervention was launched
Proportion of students that started
attending tutoring
Number of tutoring sessions
attended
1,14%
1,68%
1,82%
0,00%
0,40%
0,80%
1,20%
1,60%
2,00%
33
56
80
0
20
40
60
80
100
**

65. © 2015 ideas42 65
Randomized Trial 2 – Spring 2015
261
faculty teaching
that semester
treatment
8 ideas42 emails
control
no emails
randomly sorted

66. © 2015 ideas42 66
Behaviorally designed treatment emails targeted identified
diagnoses
Encourage faculty to send
referrals at the optimal time in
the semester.
1
2
Peer testimonial around the
benefit of tutoring for students
3
Reduce hassles associated
with making a referral for
tutoring

67. © 2015 ideas42 67
Intervention tripled the number of faculty sending
referrals and doubled the number of referrals sent
*Significant to the 90% level; **Significant to the 95% level
Proportion of faculty that sent
referrals
Number of referrals sent
**
6,45%
18,98%
0,00%
4,00%
8,00%
12,00%
16,00%
20,00%
24,00%
Control Treatment
49
106
0
20
40
60
80
100
120
Control Treatment

68. © 2015 ideas42 68
Source: Faculty responses to emails sent
Faculty appreciated the email campaign
Thank you! I look forward to
providing my students with
these resources. As a former
WKCTC student I benefitted
tremendously from The
Tutoring Center.
Dr. Baker, I
appreciate this
idea; a reminder
can sometimes do
the trick and I will
be more cognizant
of tutoring for my
students.

69. © 2015 ideas42 69
Randomized Trial 3 – Spring 2015
3,121
degree-seeking
students on
financial aid
treatment
9 ideas42 emails
control
no emails
randomly sorted

70. © 2015 ideas42 70
Behaviorally designed treatment emails targeted identified
diagnoses
Reduce hassles associated with
scheduling a tutoring appointment.
1
2
Peer testimonial from a student
sharing a tutoring success story and
correcting misperceptions around
what to expect at a tutoring session.

71. © 2015 ideas42 71
4,99%
6,71%
0,00%
2,00%
4,00%
6,00%
8,00%
10,00%
12,00%
Control Treatment
Treatment students were 34% more likely to attend
tutoring and attended 53% more tutoring sessions
*Significant to the 90% level; **Significant to the 95% level
Sample included all students taking at least one class for which tutoring was offered and excluded three students who had each attended over 20 tutoring sessions during the semester
Proportion of students that
attended tutoring
Number of tutoring sessions
attended
*
171
263
0
50
100
150
200
250
300
Control Treatment
*

72. © 2015 ideas42 72
Source: Student responses to the emails sent
Students appreciated the email campaign
Just words of
encouragement throughout
the semester would be
awesome. I work full time
and have a family.
Sometimes I get frustrated
and just need words of
encouragement.
…your
encouragement
was extremely
valuable in
motivating me to
take action

73. © 2015 ideas42 73
Suppose I offer you a choice…
$100
today
$105
next month
Or

74. © 2015 ideas42 74
Which would you choose?
1. $100 today
2. $105 next month
1 2
50% 50%

75. © 2015 ideas42 75
Now suppose I offer you the following choice…
$100
In 6 months
$105
In 7 months
Or

76. © 2015 ideas42 76
Which would you choose?
1. $100 in 6 months
2. $105 in 7 months
1 2
50% 50%

77. © 2015 ideas42 77 77
CONTEXTUAL TRIGGERS
Present-biased preferences often play out when something has immediate costs
(often small, but feel big and immediate) and benefits in the future (may be large,
but feel small or distant)
Benefits
Costs

78. © 2015 ideas42 78
PROCRASTINATION
Putting off doing something at the point where you
previously predicted you would do it.
Definition

79. © 2015 ideas42 79 79
PROCRASTINATION

80. © 2015 ideas42 80
PROCRASTINATION: INSIGHT #1
80
It’s about Deferral
• You postpone something you’ve previously
decided to do, and still in principle want to
do
Change Mind

81. © 2015 ideas42 81
Today
....
..
Reality
PROCRASTINATION
Reality
.....
Reality
....

82. © 2015 ideas42 82
Open-ended
processes with
no fixed
deadlines
WHEN DO WE PROCRASTINATE?

83. © 2015 ideas42 83
Open-ended
processes with
no fixed
deadlines
Processes with
too many
component
steps
WHEN DO WE PROCRASTINATE?

84. © 2015 ideas42 84
Open-ended
processes with
no fixed
deadlines
Processes with
too many
component
steps
Actions whose
benefits lie far
ahead but costs
are borne now
WHEN DO WE PROCRASTINATE?

85. © 2015 ideas42 85
HASSLE FACTORS
Small roadblocks that need to be performed before you
complete an action
Definition

86. © 2015 ideas42 86
HASSLE FACTORS
86
HASSLE FACTORS: INSIGHT #1
Small hassles can have a big impact because they play into our other problem
psychologies. For example, if we never procrastinated, some hassles would be less
powerful.

87. © 2015 ideas42 87
HASSLE FACTORS: INSIGHT #2
Effect is disproportionate to the size of the hassle. In other words, a small hassle
can have an enormous impact.
87
Devoto, F. Duflo, E., Dupas, P., Pariente, W., Pons, V. (2011) “Happiness on Tap: Piped Water Adoption in Urban
Morocco.” American Economic Journal: Public Policy, September 2011
Experiment: Households were
given help with the
administrative steps needed to
get a piped water connection.
0
0,2
0,4
0,6
0,8
1
1,2
%
signing
up for
piped
water
control treatment
Results of Piped Water Adoption in Morocco
10%
70%

88. © 2015 ideas42 88
HASSLE FACTOR TRIGGERS
Processes with
many
(sub)steps to
complete
Processes with
too many kinds
of steps to be
taken

89. © 2015 ideas42 89
GROUP
ACTIVITY #1
Part 1
• Think of times where you had a big deadline that you
failed to meet (were able to meet)
• Take your favorite of the above examples and dig a little
deeper into that situation. What were the details (timing,
location, mood, presence of others) that prevented you
from meeting the goal (helped you reach the goal)?
• Discuss with your group.
Part 2
• Discuss with your group a case (ideally among the
population your organization serves) where people fail to
meet their goals.
• What has your institution done to help?
• What could you do to help in the future?

90. © 2015 ideas42 90
+

91. © 2015 ideas42 91
rfp: a request for problems, not proposals
Source: http://nces.ed.gov/programs/coe/indicator_cva.asp
647 people across 600
organizations
72 attendees for RFP webinar
70 organizations interested in
applying

92. © 2015 ideas42 92
RFP APPLICATIONS RECEIVED
56Applications Received
6
5
7
17
2
2
3
7
1
6
Community Based
4-Year Institution
National Networks
Non-Profits
Research Groups
High Schools
Ed Tech
Community Colleges
Government
Misc.
15 Financial Aid
10 Summer Melt
9 Utilization of
Resources
9 Persistence
3 College Savings
10 Other
16 17
23
Preparation
Transition
Persistence&
Completion

93. © 2015 ideas42 93
APPLICATION RATING PROCESS
7 dimensions:
Overall Feel
Client
Pool
Organizational
Capacity
Leadership
Buy-In
Scalability Intervention
Promise
Appropriate
Project
Scope

94. © 2015 ideas42 94
NARROWING THE LIST
Eliminated 12 based on pool size and
implementation promise
Arrived at final list through rankings and group
discussion
Final pool included both strong potential
partners and thought leaders in the space
1
2
3

95. © 2015 ideas42 95
+
+

96. © 2015 ideas42 96
Pay attention to the individual’s inattention
You might think that your customer is making an active choice not to use your product. Are you sure s/he
considered it?
Inattention

97. © 2015 ideas42 97
general lesson: define, diagnose, design … then test
DEFINE DIAGNOSE DESIGN TEST
REDEFINE
PROBLEM
FIND ANOTHER
BOTTLENECK
DISENTANGLE
PRESUMPTIONS
INTERVENTION
CONCEPT
CONTEXT
RECONNAISSANCE
BEHAVIORAL
MAP
HYPOTHESIZED
BOTTLENECKS
POLISH
INTERVENTION
DETERMINE
FEASABILITY
INITIAL
EXPERIMENT
DESIGN
ACTIONABLE
BOTTLENECKS
SCALABLE
INTERVENTION
DEFINED
PROBLEM

98. © 2015 ideas42 98
Today’s focus is tomorrow’s neglect
The pressing needs of today are far more important than what might arise three months from now.
Present Bias
TODAY

99. © 2015 ideas42 99
Small hassles can have large effects
Small hassles can trigger the “not now” response, and lead to procrastination and inaction.
Hassle Factors

100. © 2015 ideas42 100
How can we stimulate
innovation?
• What sorts of institutions should
be targeted?
• How can we insert the
necessary technical expertise
• How do we ensure rigor in
measurement?
• How can we make innovation
ongoing and iterative?

101. Saugato Datta
Managing Director, ideas42
Behaviorally Optimizing College:
Improving Student Success through
Behavioral Science
October 25-26, 2015
Follow us on Twitter
@ideas42

102. © 2015 ideas42 102© 2015 ideas42 102
Increasing Applications to
Work-Study Jobs
at Arizona State University
​Andrew White, Nicki Cohen, Alissa Fishbane, and Piyush Tantia
​Spring Semester 2015

103. © 2015 ideas42 103
Problem: few eligible students apply for work-study
jobs in ASU’s new SEED jobs program
20%
80%
Low Hiring Rate
for SEED jobs
Position filled Position open
11%
89%
Low Application Rate
for SEED jobs
Applied Did not apply
Fall 2014 data

104. © 2015 ideas42 104
Incorrect mental models, non-salient deadlines, and
hassle factors prevent students from applying
Students did not have the correct mental model about
work- study jobs
Application deadline not salient
Hassle factors in the application process prevented action

105. © 2015 ideas42 105
Randomized controlled trial used one treatment arm
2,335
eligible freshmen
treatment
12 ideas42 emails
control
12 standard emails
randomly sorted

106. © 2015 ideas42 106
Shape the right mental model of SEED jobs
— emphasize financial and academic
benefits in clear language
Make deadline salient and force a moment
of choice
Reduce hassle factors with plan-making
activity
1
12 emails targeted identified diagnoses
2
3

107. © 2015 ideas42 107
More unique applicants and applications
Trend towards more hires
**significant at the 95% level
1. Not a significant result. A large portion of applications were never reviewed due to organizational constraints
109
140
-
50
100
150
control treatment
Number of Unique Applicants
+30
%
304
475
-
100
200
300
400
500
control treatment
Number of Applications
+60
%
** **
Number of Hires1
55
hires
in treatment
50
hires
in control
<

108. © 2015 ideas42 108
Sinclair Project
Summary & Results
September 16, 2015
Using behavioral science to do good

109. © 2015 ideas42 109
DEFINE (THE PROBLEM): Ideas42 worked with Sinclair on the
problem of late registration
The Problem
Too many students register too late in the process. In 2014, a third of
continuing, non-audit students registered for classes during the last month of the
registration window and over a thousand students registered in the last week.
Why It Matters
• Sinclair has evidence that late registration correlates with decreased academic
success and retention.
• Students who don’t register for classes early get locked out of classes and
sections that they need, which impedes on-time completion and retention.
• When students register late it is difficult to project course demand and offer the
courses students will need and want.

110. © 2015 ideas42 110
DIAGNOSE: WE IDENTIFIED THE KEY BEHAVIORAL
BOTTLENECKS IMPEDING EARLY REGISTRATION
#1: Students are anchored to the perceived deadline as the day to act. This
could be the registration deadline or the start of the semester.
#2: Registration isn’t salient for most students, particularly those that are not
on campus everyday.
#3: There is little perceived value in registering early. Students don’t realize
the consequences for delaying registration.
#4: Students appear to have various mental models (e.g., “don’t register until
you’re sure”) which lead them to do something other than register early.
#5: The registration process contains a number of complex choices and
hassle factors which lead to procrastination.

111. © 2015 ideas42 111
DESIGN: WE CREATED TWO INTERVENTIONS TO ADDRESS
THE BOTTLENECKS
Advising InterventionMessaging Intervention
» Students were assigned opt-out “appointments” with
academic advisors via email and text message. In the
appointments, advisors encouraged them to update
their My Academic Plan (MAP) and register for Fall
courses.
» Daytime appointments took place during regular
advising walk-in hours. Students with an appointment
were able to skip the normal line.
» Evening appointments were conducted over the
phone. Advisors called students directly at the
prescribed time.
» ideas42 sent students in the treatment group a series
of emails and text messages
» Students were sent 1-2 messages a week over the
course of 12 weeks, 19 messages in total.
» Messages were sent in email-text pairs.

112. © 2015 ideas42 112
Diagnosis  Design: THE INTERVENTIONs WERE TARGETED
AT THE DIAGNOSED BEHAVIORAL BOTTLENECKS
Advising
Intervention
Messaging
Intervention
#1:
Anchoring to
Deadline
#2:
Salience of
Registration
#3:
Little
Perceived
Value
#4:
Different
Mental Models
#5:
Hassle
Factors

113. © 2015 ideas42 113
TEST: WE evaluated THE INTERVENTIONS USING A
randomized control trial
Control Treatment
Control
6,355 students
did not receive
either
intervention
6,390 students
received the
Messaging
intervention
alone
Treatment
2,600 students
received the
Advising
intervention
alone
2,556 students
received both
interventions
Messaging Intervention
Advising
Intervention
»ideas42 ran a 2x2 intervention design to test the effects of the
interventions separately and combined.

114. © 2015 ideas42 114
TABLE OF CONTENTS
I. Project Recap: Define, Diagnose, Design, Test
II. Key Results
III. Subgroup Analysis
IV. Discussion of Results
114

115. © 2015 ideas42 115
KEY OUTCOME VARIABLE: EARLY REGISTRATION DEFINED
AS ON OR BEFORE JULY 17
APRIL
S M T W Th F S
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30
MAY
S M T W Th F S
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31
JUNE
S M T W Th F S
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30
JULY
S M T W Th F S
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31
AUGUST
S M T W Th F S
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
115

116. © 2015 ideas42 116
33,1%
35,4%
0%
5%
10%
15%
20%
25%
30%
35%
40%
No Messaging
(Control)
Messaging
%Studentswhoregisteredearly **
MESSAGING INTERVENTION increased early reg
By 2.3 percentage points, or 6.9 percent.
** = significant at 95% level
All students, collapsing across advising treatment

117. © 2015 ideas42 117
STUDENTS WHO OPENED AT LEAST ONE MESSAGING intervention
EMAIL REGISTERED AT EVEN HIGHER RATES
26,8%
21,0%
42,7%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
No Messaging (Control) No Opened Messages Opened at Least One
Message
%Studentsgoingto
advisingthissummer
• Positive effect of opening email on registration, but there may be selection
effects

118. © 2015 ideas42 118
MESSAGING EMAILS HAD GENERALLY HIGH OPEN RATES THAT
DECLINED OVER TIME
• Two-thirds of all students who received a messaging email opened at
least one of them.
43,0%
22,6%
12,9%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Weeks 1-4 Weeks 5-8 Weeks 9-12
Emailopenrate(%)

119. © 2015 ideas42 119
29,5%
31,4%
0%
5%
10%
15%
20%
25%
30%
35%
40%
No Advising
(Control)
Advising
Studentswhoregisteredearly
**
ADVISING INTERVENTION increased early reg
by 1.9 percentage points, or 6.4 percent
** = significant at 95% level
Excluding students who registered before advising treatment began, and groups Sinclair specified

120. © 2015 ideas42 120
The advising intervention led MORE students to ENGAGE WITH
advising
38%
47%
0%
10%
20%
30%
40%
50%
No Advising
(Control)
Advising
%Studentsengagingwith
Advisingfrom5/4to7/31
• The intervention made students more likely to engage with Advising between
May 4 and July 31 (not necessarily at the scheduled time).
• The range of what constitutes “engagement” requires further discussion.

121. © 2015 ideas42 121
Messaging & Advising EACH increased early registration BUT NO
ADDED BENEFIT OF BOTH
***
• The Messaging and Advising interventions both increased early registration,
but there was no statistically significant additional benefit of a student
receiving both.
26,8%
32,1%
30,4%
32,4%
0%
5%
10%
15%
20%
25%
30%
35%
40%
None Messaging Only Advising Only Messaging & Advising
%Studentswhoregisteredearly
** ** **

122. © 2015 ideas42 122
The ADVISING intervention INCREASED Fall semester RETENTION
(7th day)
***
47,1% 46,3%
48,9% 48,0%
0%
10%
20%
30%
40%
50%
60%
None Messaging Only Advising Only Messaging & Advising
%Studentswhoregistered
**
• Based on data updated on August 30, 2015
• ** = significant at 95% level

123. © 2015 ideas42 123
The ADVISING intervention INCREASED fall semester RETENTION
(14th day)
***
47,2% 46,3%
48,8% 48,1%
0%
10%
20%
30%
40%
50%
60%
None Messaging Only Advising Only Messaging & Advising
%Studentswhoregistered
*
• Based on data updated on September 7, 2015
• * = significant at 90% level

124. © 2015 ideas42 124
TABLE OF CONTENTS
I. Project Recap: Define, Diagnose, Design, Test
II. Key Results
III. Subgroup Analysis
IV. Discussion of Results
124

125. © 2015 ideas42 125
THE INTERVENTIONS HAD STRONGER EFFECTS ON SOME
VULNERABLE GROUPS BUT ON NOT OTHERS
Group Messaging Advising Combined
Overall +20% +13% +21%
Minority Students +18% +18% +36%
Developmental +32% +35% +34%
First-Generation +24% +2% +21%
No EFC +17% +9% +15%
Full-Time +21% +16% +31%
Part-Time +18% +10% +8%
• No consistent pattern emerged in the subgroup analysis.
125

126. © 2015 ideas42 126
TABLE OF CONTENTS
I. Project Recap: Define, Diagnose, Design, Test
II. Key Results
III. Subgroup Analysis
IV. Discussion of Results
126

127. © 2015 ideas42 127
SUMMARY OF KEY RESULTS
 Both the Messaging and Advising interventions succeeded in increasing early
registration.
 The Advising intervention increased overall registration (retention) in addition to early
registration.
 Subgroup analysis showed both interventions helped students taking developmental
courses more than other students, but the pattern wasn’t consistent.
 Students opened emails in the first 4 weeks of the Messaging intervention at a much
higher rate (43.0%) than in the last 4 weeks (12.9%).
 The mechanism of the Advising intervention’s success appears to be getting more
students to engage with Advising.

128. © 2015 ideas42 128
TAKEAWAYS & Recommendations FOR SINCLAIR
Advising
 Consider formalizing an outbound calling program in Advising as an effective way to
increase early registration and retention.
 Engaging with Advising is good for students, so finding innovative ways to increase such
engagement can drive positive outcomes.
Messaging
 Behaviorally designed reminder messages that keep registration top of mind, facilitate
next steps, and promote interim deadlines are effective in prompting action on key
priorities.
 Students opened emails in the first 4 weeks at much higher rates (43.0%) than in the last
4 weeks (12.9%), so limit overall messaging volume by focusing on the first part of the
registration period.
 Continue experimenting with text messages to nudge critical actions.