Recommendation is one of the most traditional and wide spread use case of Machine Learning. In this talk we want to showcase, how an advanced recommendation engine can be served at scale in Glance. Glance is an AI-powered, content driven, personalised Screen Zero (lockscreen) platform for mobile, which is used by over 26M DAU users in India. The talk will take you through each component of a recommendation engine and in the end will showcase the learnings which we got from our experiments to make it functional at scale.
Recommendation Problem Formulation:
A recommendation problem can be framed in 2 ways:
1. Missing value problem / Prediction problem
2. Ranking top k items
The factors to keep in mind for any recommendation are:
1. User Signal
i. Eg: rating, like, dislike
ii. Time spent
iv. Etc etc
2. Utility Matrix - [User, Item] matrix
3. Types of ratings
c. Unary- these are positive preference entries. For example item buying matrix, where we only have
the buying (1 ) but nothing for dislike or not buying.
Types of Recommendation Systems
There are broadly 4 types of recommendation systems prevalent:
1. Collaborative Filtering
a. User based
b. Item based
2. Content Based Filtering
3. Knowledge Based
4. Hybrid Systems
Collaborative Filtering: User
1. Do not need any domain expertise.
2. Explore is done through similar user tastes.
1. Sparsity in signals
3. Cold start
Collaborative Filtering: Item
1. Domain knowledge not required.
2. More accurate.
3. High interpretability.
4. More stable than user based CF.
2. Explore is minimum.
3. Cold item problem.
Content Based Filtering
1. Learns from user’s past interactions.
2. Requires Domain Knowledge.
1. Learn user’s preference from interaction with combination of item attributes.
2. Preferred methodologies:
a. Similarity based methods. Ex. Cosine Similarity estimation.
a. ML models - Regression, SVM etc.
Content Based Recommendation
1. No cold start problem for items.
2. Interpretation is available
1. No learning from neighbors.
2. Cold start problem for users
3. Can only be as good as deﬁnition of item attribute.
a. NETFLIX- “They paid people to watch films and tag them with all kinds of metadata. This process is so
sophisticated and precise that taggers receive a 36-page training document that teaches them how to rate
movies on their sexually suggestive content, goriness, romance levels, and even narrative elements like
Personalized screen zero
Glance is an AI-powered, content driven, personalised Screen Zero (lockscreen) platform for mobile.
Glance enables discovery of content for users every time they unlock their mobile.
Multiple formats and vernacular
Scale of the problem:
- 81 Billion Glances a month
- 65% of all smart phones in India have Glance
- Average time spent is 22 min.
- 26 Million Daily Active Users.
Need for recommendation:
- Currently there are 17 categories for every Glance article
- Glance content is in 4 different vernacular: English, Telugu, Hindi and Tamil.
- 100,000 Glances are consumed daily by users.
- Lock Screen is really meaningful to a user.
Neural Network based Recommendation Approach
- Should work with multiple data sources: Image, Video & Text
- Should be scalable to serve millions of user.
- Latency should be low to serve in real time.
- Cost Effective to be scaled properly.
MODELLINGUSER INTEREST ITEM PROFILE
RECENCY VIRALITY VERNACULAR
FACTORS KEPT IN MIND
Breaking down a GlanceGLANCE
- News / Features
- Video / Article
- In app
- Article Image
Lesson 1: Architecture Decisions
1. Respect your heterogeneous and unique system
a. Our Learning:
i. We explored 4 different ways to serve the user a personalised feed based on the model
ii. Factors in our case were: Latency, Refresh rate and Change in Proﬁle
2. Move fast and break things is not always fun
a. Our Learning:
i. Consider the scale of your application even in your MVP as this can be deciding factor in
Data Ingestion Architecture
Overall data flow
● Because of heterogeneous sources and scale of data, pre-processing is done
in real time.
● Pre-processing is done using Azure Stream Analytics and Azure Logic App
Lesson 2: Serving Architecture
Batch Predictions Real-time Prediction
Make offline predictions for the users and
Make predictions has and when required.
Advantages: Fast, Model can be complex. Advantages: Reactive, Cost
Disadvantage: Wasteful, 3X more
Disadvantage: Model complexity needs to
Our Learning: Take best of both worlds. Keep complexity oﬄine.
● The scoring service is done with ‘just-in time’
ranking for all glances for incoming users
● The advantage of this architecture is that it’s
tightly coupled with the existing glance
● Our architecture made it scalable and
compute effective by reducing it 3X compared
to batch predictions.
Lesson 3: Containerize and Code Hygiene
1. Code Hygiene is extremely important
a. Our Learning
i. Triple check coding standards and documentation of code for easy passover and
2. Train and Serve Environments should match
a. Our Learning:
i. Kubernetes allowed high replicability, scalability and easy container management.
Lesson 4: Log and Monitor
A/B Testing framework Logging of all results in
the online world
Predefined and agreed
DS and business
metrics to track
- Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language
understanding." arXiv preprint arXiv:1810.04805 (2018).
- Redmon, Joseph, and Ali Farhadi. "Yolov3: An incremental improvement." arXiv preprint
- He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE
conference on computer vision and pattern recognition. 2016.