Presentation regarding the outcomes, implementation and operation of Starfish as a potential retention intervention.

1. Closing the Loop: Improving Student
Retention through Analytics
A report on SUNY Oswego's Starfish Early Alert
pilot
Michelle Bandla Coordinator, First Year Programs
Greg Ketcham Director, Academic Programs,
Division of Extended Learning
Rameen Mohammadi Associate Provost for Undergraduate and Special Programs
Elizabeth Schmitt Professor of Economics
SUNY CIT May 2014

2. Agenda
• Why Starfish?
• Introduction to Starfish Pilot
• Early Alert and Connect
• Project Goals
• Specific Expectations/Sample Communications
• Pilot Assessment
• Usage data
• Faculty Focus Groups and Changes in Spring 2014
• How to assess?
• Comparing pilot and non-pilot groups
• Pilot Implementation Issues
• Angel integration
• Faculty training and participation
• Other issues
Slide 2

3. Why Starfish?
• Problem: Retention and persistence rates below
institution goals
• Institution vitality
• Student success in meeting educational and life goals
• Accountability
• Nationally SUNY comprehensives are in the 75-
80th percentile of 4-year publics in persistence
measures
• But Oswego’s retention and graduation rates are
lower than most SUNY comprehensives
Slide 3

4. Why Starfish?
• Solution: IPAS
• Integrated Planning and Advisement Services
• “…leverage technology to enhance and streamline
course advising ... tracking of student progress and
targeting of support services, in order to improve
student academic decision-making as well as
institutional strategies to help students stay on track
to degree attainment.” [Gates Foundation LRM Market Acceleration Project,
February 2013]
Slide 4

5. Why does an IPAS help?
• Metacognition
• Dunning & Kruger (1999), Dunning et al (2004)
• Early targeted communications offer more effective
feedback to struggling students
• Earlier intervention gives us more options
• Communication to build connection
• Student satisfaction with their classroom experience
(Pascarella 2011)
• Connection to resources
• Connection to instructors

6. Why Starfish?
• Oswego had been evaluating IPAS vendors for
over two years
• Constantly changing product
• Major financial commitment
• SUNY adoption?
• The (unsuccessful ) Gates grant opportunity
pushed us to focus on a vendor and move
forward with a pilot
Slide 6

7. Why Starfish?
• Vendor choice: Starfish
• Integrates with Angel, Tutortrac, Noel Levitz CSI, Banner
• Early Alert and Connect products
• Starfish Pilot, 2013-14
• 967 students
• Freshmen and Transfers (Biology, Psychology, Public Justice,
Business)
• Criteria: Department interest, past success rates, good
indicator of campus experience
Slide 7

8. Introduction to Starfish Pilot
Early Warning & Student Tracking System
• Student concerns can be manually
flagged when observed or through
periodic surveys.
• Automatic flags are raised based on
online grade book data.
• Recommendations can be
communicated to the student.
• Appropriate personnel are informed of
flags raised and cleared.
• Close the loop with faculty when
progress is made.
Slide 8
Educational Support Networking System
• Each student has personal My Success
Network of instructors and campus
staff.
• Online scheduling for in-person
meetings – both individual and group
appointments.
• Appointment confirmation by email.
• Documentation of contact with students
through private and shared notes.
• Service catalog of available campus
support resources.

9. Introduction to Starfish Pilot
• Goals for the Pilot:
• Participation in Progress Surveys: Progress surveys provides
instructors the opportunity to raise concerns or kudos for pilot students in
their course sections.
• Notification to students of concerns: Email notifications
sent to students that include comments provided.
• Notification to advisors and coordinated outreach to
flagged students: Email notifications sent to advisors. Advisors to
outreach to pilot students and close the loop with instructors.

10. Specific Expectations
Slide 10
Instructors
• Raise
flags/kudos
when
appropriate.
• Submit
progress
survey when
emailed.
• Follow up with
student on
flags raised
• Clear flag when
appropriate
Advisors/Staff
• Receive email
notifications
from Starfish.
• Record follow
up actions in
Starfish:
• Follow up
with students
with multiple
flags
• Add
comments
• Clear flags
• Close the
Loop
Students
• Receive email
notifications
when flagged.
• Contact
appropriate
support office.
• Meet with
Instructors and
Advisors
Everyone
• Provide us with
your feedback!

11. Example Survey Announcement
To: {Instructor Name}
From: Starfish@oswego.edu
Subject: [Starfish] Progress Survey for {Deployment Date}
Dear {Instructor Name},
This spring is the second semester of our year-long Starfish Early Alert Pilot. You are receiving this email because you have one
or more of the pilot students in your course(s). You have a critical role in this process as you best know how the student is doing
in your course. Starfish provides us with the communication tools to identify the struggling students and intervene earlier in order
to help them succeed.
Approximately 1000 first year and transfer students are in the PILOT cohort from the following majors: Biology/Zoology, Public
Justice, Psychology and those in the School of Business.With over 3,600 performance flags and kudos generated in the fall,
faculty participation exceeded expectations. We are hoping that you will continue to help us through this pilot and utilize Starfish
to report on the academic progress of our students.
There will be two Progress Surveys deployed this semester through Starfish prior to mid-term in order to receive early
performance feedback about our PILOT students. Starfish system offers many other tools to aid in assisting our students that
some faculty have already found useful. For example, you have the ability to identify “To Do” items for students, use the Starfish
appointment features that synchronize with Google Calendar, or take advantage of the “Zoom In” option that uses your Angel
gradebook to raise flags or generate kudos for students. To learn more about the functionality of Starfish, please follow this link:
Oswego.edu/starfish
Here is a short YouTube video explaining how to submit a progress survey:
http://www.youtube.com/watch?v=NMMPaZxDMP8&feature=youtu.be
Thank you for your participation! We appreciate your dedication to student success.
On behalf of the Starfish Implementation Team,
Rameen Mohammadi
Associate Provost

12. Example Student Email Notification
From: starfish@oswego.edu
ReplyTo: starfish@oswego.edu
Subject: [Starfish College Early Alert] Progress Concern: CourseName
Dear StudentName,
You have been flagged for FlagName in CourseName.
The professor in this class has indicated that you are not making adequate progress and she/he is concerned about your ability to
successfully complete the class.
[if comments] Below are your instructor’s comments:
These are my comments!
Please contact your instructor to discuss these issues and to make a plan to improve your performance in the course. Refer to the
course syllabus for contact information for your professor.
We also recommend that you contact your advisor to discuss this issue and any other issues you are experiencing this semester.
The College has free tutoring services to help you improve your academic performance. In addition, advisement and personal counseling
services are available to support academic and career planning, as well as to assist you with personal concerns.
To seek tutoring, please contact the Office of Learning Services:
Address: 173 Campus Center
Email: ols@oswego.edu
website: http://www.oswego.edu/academics/support/OLS.html
To seek Major or Career Exploration contact The Compass
Address: 145 Campus Center
Email: compass@oswego.edu
website: http://www.oswego.edu/student/success/compass.html
To seek personal counseling, please contact the Counseling Center:
Address: 113 Walker Health Center
Phone: 315-312-441
website: http://www.oswego.edu/student/services/counseling_services.html
Sincerely,
Student Success Team

13. Usage Data 2013-2014
Fall 2013 Spring 2014*
Students 967 977
Courses and labs 711 704
Instructors 360 383
Advisors 100 131
Courses /w Completed Surveys 429 (60%) 431 (61%)
Academic Flags 1440 765
Kudos 1951 768
System Flags (3 flags) 199 55
System Flags (6 flags) 76 6
Slide 13
*Significant changes in the flag structure occurred between these semesters as a
result of faculty feedback

14. Usage Data 2013-2014
• Fall focus group feedback
• Fewer flag categories
• Better defined default options
• Clearing flags—what does it mean?
• Clarifying the advisor role
Slide 14

15. Tracking Items Changes Fall 2013 to Spring 2014
Fall 2013 Flags Spring 2014 Flags
Attendance Concern
Excessive Absences
In Danger of Failing
In Danger of Failing
Low Quiz/Test Scores
Incomplete or Unsatisfactory Graded Work
Low Participation
Failure to complete Assignments
Behavioral Concern
General/Behavioral Concern
General Concern

16. Tracking Items Changes Fall 2013 to Spring 2014
Fall 2013 Kudos Spring 2014 Kudos
Keep Up the Good Work
Outstanding Academic Performance Outstanding Academic Performance
Showing Improvement Showing Improvement

17. Assessment Plan
• Performance Differences between Fall Cohorts (2012
and 2013)
• Note: selectivity matrix differs across these cohorts
• Course specific performance Fall 2013: Pilot vs. non-Pilot
• Compare to Fall 2012
• Fall to Spring 2013-14
• Retention
• Good standing
• Fall to Fall Retention (not yet available)
• GPAs for the 2013-14 academic year (not yet available)
• Student, faculty and staff user surveys (in progress)
Slide 17

18. Some Early Data
• Overall (freshman and transfers), there were no
significant differences (Fall-Spring) in
• Semester withdrawals
• Term GPA
• Instances of disqualification, academic warning or
attainment of honors (Dean's list, President's list).
• This is robust to subgroup comparisons
• Psychology Transfers, Psychology Freshmen, Biology
Transfers, etc.
Slide 18

19. Overall Comparison, Pilot Cohorts
Fall Semesters
Slide 19
Year Student Type # # % # % # %
Fall 2012 All 756 38 5.0% 117 15.5% 149 19.7%
Fall 2013 All 894 33 3.7% 141 15.8% 189 21.1%
Fall 2012 New Freshman 440 21 4.8% 58 13.2% 93 21.1%
Fall 2013 New Freshman 580 20 3.4% 84 14.5% 118 20.3%
Fall 2012 New Transfer 316 17 5.4% 59 18.7% 56 17.7%
Fall 2013 New Transfer 314 13 4.1% 57 18.2% 71 22.6%
Dismissed Warning Pres/Dean's List
Differences shown here are not statistically significant.

20. Other Grade Data
• Fall semester grade comparisons
• Fall 2013
• Pilot vs. nonpilot cohorts in selected courses
• Fall 2012 vs. 2013
• Compare the pilot-eligible cohorts (first year students
in certain majors)
• Preliminary analysis finds little difference
Slide 20

21. Fall 2013 Average GPA
Slide 21
CHE
111 N GPA
Pilot 181 2.45*
Non-pilot 124 2.76
*significant at 5% level
BIO
203 N GPA
Pilot 22 2.11*
Non-pilot 84 2.43
PSY
100 N GPA
Pilot 114 2.62*
Non-pilot 157 2.35
No difference:
ANT 111, BIO 110, CHE 209, ECO 101, HIS 100, HIS 202,
HIS 203, MAT 104, MAT 158, MAT 208, MAT 210, MUS
101, SPA 102, THT 110
Selected Courses, All Sections

22. Pilot-eligible comparison, Fall 2012 vs. 2013
Slide 22
MAT
208 N GPA
2012 47 1.88*
2013 42 2.83
*significant at 5% level
ECO
101 N GPA
2012 216 2.42*
2013 287 2.65
No difference:
BIO 110, BIO 203, CHE 111, CHE 209,HIS 100, HIS 202,
HIS 203, MAT 104, MAT 158, MAT 210, PSY 100
Note: selectivity in the Fall 2012 cohort is greater than the
Fall 2013 cohort
Selected Courses, All Sections

23. Eco 101 800, Fall 2013
• Principles of Microeconomics
• 410 students
• Mastery learning quiz system
• i<clicker sessions every class (attendance proxy)
Slide 23
*significant at 5% level
N
Course
Average
Missed
i<clicker
sessions
Missed
module
quizzes
Pilot 249 71.46 3.35* 1.45*
Non-pilot 161 70.30 4.45 2.22

24. What to make of the preliminary data?
• Is it reasonable to expect big differences in the
first semester of the pilot?
• Faculty usage
• Limited integration
• Intervention options and follow-through
• Is the action at the center or at the tails of the
distribution?
Slide 24

25. Instructor Comments
CSC 102 Business Information System and Computing Tools
249 students, multiple sections
Mike Smith wrote:
My experience with Starfish was a positive one, I have larger classes ( for fall '13 I
had one of 88 and 3 of 56 ). On my initial flagging of items I was swarmed with
students... I think prior to this even though I would stand in front of the class and
warn them "Task 'X' is due tomorrow,” it was easier for them to put off
(procrastinate) the task until it was too late. Most of the students seemed to
respond to Starfish as an official warning, and spoke to me about the task with an
intended solution.
Slide 25

26. Instructor Comments
Eco 101 Principles of Microeconomics
420 students, large section
John Kane Wrote:
While I have for many years been providing students with current grade
report and class averages in ANGEL, the alerts and kudos sent through
Starfish seemed to have resulted in more focused attention from
students. Students that were falling behind in class were more likely
to stop by to discuss their performance after receiving alert
notifications. Students performing well were more likely to visit my
office after receiving "kudos" through Starfish. In a class of 420
students, students often feel a bit lost in the crowd. These
notifications helped reduce this barrier.
Slide 26

27. Pilot Implementation Issues
• Technical issues
• Angel integration
• Integration of Starfish and Google calendars
• Integration of Tutortrac data, NL CSI
• Flags within and across courses
• Usage issues
• Faculty buy-in
• Effective interventions--What does it mean to close the loop?
• Full integration with other software
• Decision-time comes before Fall 2014 firm enrollment
Slide 27

28. Advice for Adopters
• Allow time for implementation, faculty training
• Put an assessment plan in place
• What data? When will you gather it? Analyze it?
• Get feedback early and often from end users
• Don’t be afraid to make changes mid-pilot based on
feedback
• Keep communications open with your vendor
• Trouble shooting
• Requests for new features
• Successes from other user campuses
Slide 28

29. Next Direction
• Starfish decision
• Discontinue?
• Continue pilot?
• Full implementation?
• What is a reasonable time frame to expect
results?
Slide 29