Diabesties was an iPhone app built to help college students with Type 1 Diabetes better manage their condition through tracking and social support. This project uses machine learning algorithms to predict user churn after a user's first week of engagement. View the github repo here to see the full analysis and code: www.github.com/nadaataiyab/diabesties A youtube vide of the live presentation is available here: https://www.youtube.com/watch?v=6jJtakvCEqA&t=1s

1. Predicting User Churn for a Digital Health App
Galvanize Data Science Immersive
Capstone Project
Presentation to Ayogo, Oct 2017
Nadaa Taiyab

2. BACKGROUND
• Investment Banking
• International Development
• Health Tech Startups (Finance & Operations)
• Health Coaching

3. DIABESTIES APP
• IPhone app built by Ayogo in partnership
with the College Diabetes Network

4. P
• Track glucose, insulin,
carbs, and mood
• Share data with your
“diabestie”
• Export data

5. PROJECT:
PREDICT USER CHURN
Use machine learning
algorithms to predict which
users would stop using the
app

6. WHY CHURN MODELING?
• Increase persistence by intervening with users at
risk of churn (eg. phone call, email, reminder,
etc.)
• Reduce cost and improve effectiveness of
interventions
• Instructive for future app design and predictive
modeling

7. TECHNOLOGIES
seaborn

8. DATA
• MySQL Database
• 3 years of data (2012-
2015)

9. 3080
Users
51,125
Logs
387,188
Clicks
User Data
(Demographics
)
Behavioral
Data

10. User Data
What can we learn about the users?

11. 30% were college age (18-25), but the average age was
37.

12. 51% had Type 1 Diabetes and 42% had Type II
Diabetes.

13. 58% of participants were
female

14. 50% would not specify ethnicity, so hard to draw firm
conclusions,
but likely that majority were caucasian. and living in US.

15. DEMOGRAPHICS
• Many of actual participants differed from
intended target of college students with Type 1
Diabetes
• Huge age range, including a lot of people in late
thirties
• Ethnicity and location inconclusive, but likely
majority caucasian in the US

16. Behavioral
Data
What can we learn about how users interacted
with the app?

17. Used mainly as a glucose tracker.
Insulin and mood tracked 30% of the
time.
97% of entries included
glucose log

18. Many users tried the app a few times and quit.
A few users logged hundreds of times over 2-3 years.
85% entered data 0-10
times
15% entered more than 10
times

19. 23% of users were active more than 30 days, though only 15%
logged more than 10 times, indicating stop-start usage pattern.

20. DIABESTIE EFFECT
• Only 3% of users had a diabestie
• Of users with a diabestie, 44% used the app
for more than a week
• Diabestie didn’t seem to affect churn

21. BEHAVIORAL DATA
• Mainly used a glucose tracker
• Small core of 10-15% of users that are persistent
• Diabestie no effect

22. Churn Model
Can we use machine learning
algorithms to predict churn?

23. FEATURE ENGINEERING
• Behavioral Data
• # page views
• # total entries
• # moods,
• # notes
• # A1C updates
• User Data
• age, gender,
ethnicity,
education
• diabetes type
• diabestie: yes / no

24. CHURN DEFINITION
Users who completed less than ten
additional entries after the first week of
use

25. CLASSIFIER MODELS
• Logistic Regression
• Random Forest
• Gradient Boosted Trees
• AdaBoost

26. Gradient Boosted Trees had highest Area Under the Curve
(AUC), indicating that it had the most predictive power
True Positive Rate (Sensitivity) = True Positives / (True Positives + False Negatives)
False Positive Rate = 1 - (True Negatives / (True Negatives + False Positives))

27. MODEL SCORES
Logistic
Regression
Random
Forest
Gradient
Boost AdaBoost
Accuracy 92% 92% 94% 93%
Precision 93% 95% 95% 95%
Recall 98% 96% 98% 97%
AUC = Area Under the
Curve
Accuracy = (True Positives + True Negatives) / Total Sample
Size
Recall or Sensitivity = True Positive / (True Positive + False
Negative)
Precision = True Positives / (True Positive + False
Positive)

28. CONFUSION MATRIX
n= 770 Predicted
Churn
Predicted
Not Churn
Actual
Churn
True
Positive
692
False
Negative
16
708
Actual
Not Churn
False
Positive
34
True
Negative
28
62
726 44
Results from Gradient Boosted
Trees

29. Behavioral features had greater predictive power than
demographics
except for age.
Feature importance scores calculated
using
Random Forest algorithm

30. TAKE AWAYS
• How an app is used can evolve in unexpected ways
• Behavioral data has more predictive power than demographics
• Improve Data Quality
• Structured data for location (maybe for a1c and glucose too)
• Safeguards against data entry errors
• Timestamp data important
• Better app usage data
• Possibly improve model by only including users that made at least one entry

31. Nadaa Taiyab
nadaa.taiyab@gmail.com
github.com/nadaataiyab/diabesties