Dustin McIntosh analyzed user interactions with a product to understand different user types and how to improve the user experience. The analysis found that users can be classified as novices or experts based on characteristics of their SQL queries. Novices tended to make more SQL errors than experts. The analysis also found that some users who completed the tutorial did not continue using the product due to frustration, likely because the tutorial did not provide a good onboarding experience. Personalizing the product response based on whether a user is a novice or expert could help improve the user experience and reduce churn.

1. Analyzing Analysts
Dustin McIntosh
Mode Analytics
Understanding the Queriers Behind the Queries

2. Goal: Understand user base and how
they interact with the product
Novices
Power Users
● Who are the users?
● Which SQL errors do they make?
● How can we make users'
experience better?
● Why do users churn?
● Can we personalize product
response for different users?

3. Goal: Understand user base and how
they interact with the product
● Who are the users?
● Which SQL errors do they make?
● How can we make users'
experience better?
● Why do users churn?
● Can we personalize product
response for different users?
Novices
Power Users

4. Time
Churn – Are users getting frustrated?
Successful
query
SQL
error

5. Churn – Are users getting frustrated?

6. Churn – Are users getting frustrated?
No!

7. Churning Users = Tutorial Users
Recommendation: prompt tutorial users to connect their
own data at tutorial conclusion

8. Can we personalize product
response for different users?
Expert?
Novice?New User

9. Classifying Experts vs. Novices
Bag of
SQL
keywords
Formatting based:
– White space
– Parentheses
Keyword diversity
Is it an Error?
Error type
Random
Forest

10. Bag of
SQL
keywords
Formatting based:
– White space
– Parentheses
Keyword diversity
Is it an Error?
Error type
Random
Forest
AUC = 0.66
Classifying Experts vs. Novices

11. The Differences Between Experts
and Novices

12. Summary
● Who are the users?
● Which SQL errors do they make?
● How can we make users'
experience better?
● Do users churn out of frustration?
● Can we personalize product
response for different users?
Expert?
Novice?

13. Dustin McIntosh
EFavDB.com

14. Extra Slides

15. What errors do they make?
Recommendation: prominently display or auto-correct table/column
names

16. Top-heavy nature of SQL

17. Non-tutorial users also not getting
frustrated
All churned users Churned users that have
connected their own data

18. Correlations of the Features

19. Compare Using top 5 features
All features
Top 5 features

20. Histograms of important features
novices
experts