Talk from the 12th ACM Conference on Recommender Systems, Vancouver, Industry Track Why standard monitoring practices are not enough to monitor algorithmic services and how to derive metrics that can detect a degradation of quality in real time Contact: rmminusrslash at twitter or lina.weichbrodt@gmail.com

1. MEASURING
OPERATIONAL
QUALITY OF
RECOMMENDATIONS
Lina Weichbrodt
12th ACM Conference on Recommender Systems
04-10-2018

2. 2
Speaker: Lina Weichbrodt @rmminusrslash
● Research Engineer in the recommendation algorithms team at
Zalando
● Develop scalable machine learning products
● Zalando: Europe’s leading online fashion platform, 4.5 B
revenue in 2017, 17 European countries

3. 3
Article Recommendation
How can we monitor
recommendation quality in real time?
PROBLEM

4. 4
OUTLINE
● Problem Analysis
○ Status Quo Algorithmic Service Monitoring
○ Risk Areas
● Solution
○ Definition of Response Quality
○ How to Select a Quality Metric
● Case Study

5. 5
PROBLEM ANALYSIS

6. 6
STATUS QUO ALGORITHMIC SERVICE MONITORING
ZMON
● Status Quo: speed and errors during processing
(latency and availability)
● Good, but not enough!

7. 7
RISK AREAS: OVERVIEW

8. 8
● The input data changes. Typical reasons might be a client
application that releases a bug (e.g., lowercasing a case
sensitive identifier) or changes a feature in a way that affects
the data distribution such as allowing all users to use the
product cart instead of previously allowing it only for logged
in users. If the change is not detected training data and
serving data can diverge.
● The model is updated and the new version is inferior to the
previous one.
● The latest deployment of the stack that processes the
request and serves the model contains a bug.
RISK AREAS

9. 9
● Changes concerning external services lead to performance
loss. An example in an e-commerce setting is switching to a
different microservice to obtain article metadata used for
filtering the recommendations.
● Changes in configuration (applied filters, algorithms, A/B
test configuration) are faulty or lead to unforeseen quality
degradation.
RISK AREAS

10. 10
We need quality monitoring of
delivered recommendations
CONCLUSION

11. 11
SOLUTION

12. 12
● Typical definition of successful response: fast enough,
no http errors.
● Suggested definition: fast enough, successful
according to the business case.
Example: For personalized recommendations on Zalando
Home Page we expect…
DEFINITION OF RESPONSE QUALITY
more than 95%
http 200 OK responses
under 200ms.

13. 13
● Typical definition of successful response: fast enough,
no http errors.
● Suggested definition: fast enough, successful
according to the business case.
Example: For personalized recommendations on Zalando
Home Page we expect…
DEFINITION OF RESPONSE QUALITY
more than 65% of responses have
at last four articles in under 200ms.
more than 95%
http 200 OK responses
under 200ms.

14. 14
● Quality metric(s) are business case dependent
-> choose your own!
● Measuring the objective or user perceived quality is
(most likely) not possible. That is not a problem, simple
heuristics are already effective.
HOW TO SELECT A QUALITY METRIC

15. 15
● Suggested criteria
● comparable across models
● simple and easy to understand
● can be collected in real time
● allows for actionable alerting on problems
HOW TO SELECT A QUALITY METRIC

16. 16
CASE STUDY

17. 17
● Recommendation Team:
○ Dozens of business cases for website, apps and
emails
○ Heavy load with up to 4000 req/sec
○ Low latency requirements
CASE STUDY: ZALANDO

18. 18
● Background: Recommendations are created from a
sequence of configurations (model and filtering rules)
● Simple quality metric
○ Good response: top 5 positions are from best
configuration
○ Poor response: less than 5 articles or the top 5
articles contain a fallback (e.g. popular item)
CASE STUDY: SELECT QUALITY METRIC
Best Config
Config 2
….
Worst Config

19. 19
CASE STUDY: IMPLEMENTATION

20. 20
CASE STUDY: ALERTING
Use case XY

21. 21
● DO: add specific alerts for important use cases
● DO: if you have so many use cases that choosing
individual thresholds is not reasonable for all of them
add coarse alerts e.g. success percentage <50 %
CASE STUDY: ALERTING

22. 22
CASE STUDY: ANALYSIS
Use case XY

23. 23
● Goal: Detect bugs in the new version of the stack
● DO: Deploy model changes and code changes
separately
● Deployment process:
○ Two stack versions are live at the same time
○ Traffic is being gradually switched over
○ Measure each minute the difference of good and
bad percentages between the two stacks -> A/A
test
● Distribution of difference: average is expected to be 0 if
no bug was introduced
CASE STUDY: DEPLOYMENT MONITORING

24. 24
CASE STUDY: DEPLOYMENT MONITORING
Use cases

25. 25
● Insights
○ Online monitoring for data driven services must
include quality metrics
○ Definition of quality metrics and their acceptable
level is business case dependent
○ A very simplistic metric is already very useful
CONTRIBUTIONS

26. 26
● Quality Metrics
○ allow for real-time quality monitoring
○ will detect problems that are hard to spot
○ easy to integrate with existing tools
○ can be used to set user expectations about the
service performance
CONTRIBUTIONS

27. 27
Work with us in Berlin!
The Reco Algorithms Team is hiring a
Principal Research Engineer.