This document introduces challenges in classifying customer feedback at Cookpad, including having a large number of categories, multi-label feedback, and imbalanced data. It discusses an initial heuristic approach that was infeasible, and a second trial using simple machine learning with multiple binary SVM classifiers to handle the multi-label problem. This approach rebalanced the data and achieved high precision and recall in evaluations, reducing the time spent on feedback tagging by customer support teams by half. Future work to improve the system is also discussed.

1. 3 Challenges in
Customer Feedback
Classification
ML Kitchen #8
April 12, 2018
Van Phu Quang Huy

2. About Me
• Van Phu Quang Huy
• Github: @vanhuyz
• Doing machine learning and developing new services in
Cookpad R&D

3. What I played recently
Transform gorilla faces to human faces using CycleGAN
Repository: https://github.com/vanhuyz/CycleGAN-TensorFlow

4. Today’s talk
Introduce a very first approach to customer
support automation

5. Background
• About cookpad recipe sharing service (cookpad.com)
• 90M worldwide monthly average users
• 68 countries / 22 languages

6. Background
• About customer feedback
• extremely important in service development
• each feedback must be delivered to right staff/
engineers

7. Customer Feedback Box on
cookpad.com

8. Feedback Examples
•
It was easy, I tried with my child. It was fun.
•
There are abundant recipes. Useful.
•
After updating, the search ranking disappears, it is very
inconvenient.

9. User Support Team have to classify all customer
feedback into about 100 tags everyday

10. Internal tool (before)
User Support Team have to manually choose tags from
long tag list

11. Problems
• User Support Team have to classify all customer
feedback into about 100 tags everyday
• Time consuming (60 min/day)
• Boring

12. Problems
• User Support Team have to classify all customer
feedback into about 100 tags everyday
• Time consuming (60 min/day)
• Boring

13. It’s already 2018…

14. Solution: feedback
auto-classification?

15. Feedback Classification:
3 Challenges
• Open set and large number of categories
• Multi-label problem
• Imbalanced data

16. 1. Open set and large number
of categories
• Large number of categories: currently 100+
• Open set problem:
• unseen categories in training (as new tags will be
added in future)

17. 2. Multi-label problem
1 feedback can have multi tags, for example:
“
”
“By sending cooksnap, the recipe will come out soon when I cook the
next time, it is extremely convenient and useful. There are many
recipes from various people. I am happy to use it. Let’s keep the good
work.”
should be tagged as Cooksnap, Positive

18. 3. Imbalanced data
• Almost half of feedback
are related to Tokubai ※
• New services (e.g.
Amazon Echo Alexa,
storeTV, etc) have very
few feedback
• Many services have
already been closed then
those data become
obsolete
※ These services currently belong to Tokubai inc.

19. Open Question
How do you design your system to
solve this problem?

20. Rules of Machine Learning
https://developers.google.com/machine-learning/rules-of-ml/
• Don’t be afraid to launch a product without machine
learning.
• Choose machine learning over a complex heuristic.
• Keep the first model simple and get the infrastructure
right.

21. Rules of Machine Learning
https://developers.google.com/machine-learning/rules-of-ml/
• Don’t be afraid to launch a product without machine
learning.
• Choose machine learning over a complex heuristic.
• Keep the first model simple and get the infrastructure
right.

22. First trial: Heuristic
• Manually build a dictionary for each tag
• e.g. feedback includes a word from [‘search’, ‘find’,
‘keyword’] should be tagged as Search
• This achieved high precision but very low recall
• Also, building dictionaries for 100 tags is infeasible

23. Rules of Machine Learning
https://developers.google.com/machine-learning/rules-of-ml/
• Don’t be afraid to launch a product without machine
learning.
• Choose machine learning over a complex heuristic.
• Keep the first model simple and get the infrastructure
right.

24. Second trial:
Simple Machine Learning

25. Look back 3 challenges
• Multi-label problem?
• Multiple binary classifiers can handle that!
• Open set and large number of categories?
• If a new tag is added, just train a new binary classifier
• Imbalanced data?
• Rebalance data by using all positive samples, but only
select randomly negative samples from the rest

26. Rules of Machine Learning
https://developers.google.com/machine-learning/rules-of-ml/
• Don’t be afraid to launch a product without machine
learning.
• Choose machine learning over a complex heuristic.
• Keep the first model simple and get the infrastructure
right.

27. Pick a first algorithm
Note that we need to train ~100 binary classifiers

28. I choose you!
Support Vector Machine

29. Evaluation of 81 classifiers
on validation set

30. Infrastructure

31. Internal tool (after)
User Support Team only have to choose tags from
few tag suggestions

32. Results
• User Support Team have to classify all customer
feedback into about 100 tags everyday
• Time consuming (60 min/day) 30 min/day
• Boring Fun!

33. Results
• User Support Team have to classify all customer
feedback into about 100 tags everyday
• Time consuming (60 min/day) 30 min/day
• Boring Fun!

34. Future Work
• Visualize/Evaluate results using obtained data during
operation
• Collect negative samples (suggested tags which are not
chosen by operators) to improve classifiers
• Other ways to deal with imbalanced data https://
github.com/scikit-learn-contrib/imbalanced-learn
• Deep learning?
• Aim for 100% auto-tagging?

35. Summary
• Introduced a problem in Customer Service
• Introduced a very first solution for customer feedback
classification
• Successfully reduced labor time of customer feedback
tagging by half