This document discusses methods for measuring the quality of online services. It describes how major companies like Google, Facebook, and Netflix collect data through user behavior, panel surveys, and direct user feedback at different stages of their services. Panel surveys can provide insights but have limitations, while user behavior data is abundant but noisy. The document also provides examples of how to design panel surveys and side-by-side evaluations to assess search engine result pages. It concludes that the best approach is to combine various data collection methods depending on the service characteristics and lifecycle.

1. Measuring the Quality
of Online Service
Jin Young Kim
Senior Applied Scientist
Microsoft Web Search and AI

2. About Jin Young Kim
• Data Scientist at Microsoft
• Quantified Self Enthusiast
(10 years of happiness tracking)
• Author of ‘Hello, Data Science’
(#1 Bestseller in Korea)

3. Data is the ingredients for all these issues
• Data for training and evaluating ML models
• Data for discovering the defect and issues
• Data for monitoring the health of existing service
• Data for measuring the value of new service

4. Issues in Online Service Development
• Planning
• How to set business objective & plan?
• Implementation
• How to train and improve ML models?
• Evaluation
• How much are users satisfied with the service?
Plan
ExecuteEvaluate

5. How can we collect data for these purposes?

6. Case Study: Data Collection for Restaurants
• Customer Behavior
• Facial expression
• Quantity of leftovers
• Pace of dining
Only limited type of data is
available, possibly with lots
of noise

7. Case Study: Data Collection for Restaurants
• Panel Survey
• Satisfaction for Food
• Satisfaction for Service
• Satisfaction for Environment
Survey can provide
insights into customer
satisfaction, but with
some caveat

8. Data Collection for Online Service
• User Behavior
• Various ‘signals’ from behavioral data
• Limited type of data is available, with lots of noise
• Needs substantial user base required
• Panel Survey
• Hire a group of panels, or use crowdsourcing
• Collect feedback for all aspects of service quality
• Cost of hiring and maintaining panel

9. Data Collection for Online Service (2)
• Direct User Feedback
• Request real-time feedback from customers
• Typically low response rate, with potential nuisance
• Widely used for personalized services (i.e., recommendation)
Panel Survey User Behavior
User Feedback

11. How does major online service companies
collect data for measurement?

12. Search Engine: Google / Bing
• Early stage: panel-based survey
• Late stage: user behavior-based experiments
• Source: Google

13. How to evaluate the quality of this SERP?

14. Social Network: Facebook
• Before: use only user behavior
• Nowadays: user behavior + panel survey + user feedback
• Source: Slate / Quora
We could expose contents users are
actually satisfied instead of click-baits by
using panel survey and user feedback in
addition to signals from user behavior
- Julie Zhuo, Product Design VP at
Facebook

15. User feedback for Facebook News Feed

16. Recommendation System: Netflix
• Combine user feedback and behavior for measurement
• Source: Netflix

17. Movie Recommendations from Netflix
Algorithm
A
Algorithm
B
Can you tell if algorithm A vs. B is better?
Even the users
themselves
can’t!!!

18. Movie Recommendations from Netflix (2)
Results below are more relevant, but users engage more with the above

19. So, how should I collect data for my service?
• What signals can we extract out of user behavior?
• Are there incentives for users to provide feedback?
Service
Characteristics
• Do you already have substantial volume of active users?
• Can a panel evaluate user experience as a substitute?
Feasibility of
Collection
• Do you have marketing budget for building a user base,
or for a panel survey?
Cost of
Collection

20. How to evaluate the quality of this SERP?

21. Evaluation based on
user behavior
• Which result did users click?
• Is click the only measure of satisfaction?
• How long did a user stay on a result?
• Is longer dwell-time already better?
• Do users perform search repeatedly?
• Does loyalty mean satisfaction?
User behavior is an important clue, but a noisy one.

22. How can you design a panel survey for SERP
evaluation?
How would you evaluate
the search results for
query ‘crowdsourcing’?
Bad
Good
Excellent
Perfect
Q: Who do you think so?

23. Alternative: Evaluating a Webpage
How would you evaluate
the search results for
query ‘crowdsourcing’?
Bad
Good
Excellent
Perfect
Q: Who do you think so?

24. Alternative: Side-by-Side SERP Evaluation
Q: How would you
compare two results?
Left much better
Left slightly better
About the same
Right slightly better
Right much better
Q: Why do you think so?

25. Conclusions

26. Summary…
• As a first step in data science, plan on collecting high-quality data
• Combine various data collection methods depending on the
characteristics and lifecycle of service
• It takes a lot of consideration to get the panel survey done right

27. For more information…
• What you need to know about data even if you’re not a Data Scientist
• SIGIR’2015 Tutorial on Offline Search Evaluation
• Offline Evaluation for Information Retrieval
Foundation and Trend in IR Journal (To Appear)