Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Predictive quality metrics @ tinyclues - Artem Kozhevnikov - Tinyclues

859 views

Published on

Artem Kozhevnikov, lead Data Scientist, will present some quality metrics commonly followed @ tinyclues in order to evaluate the model predictive power. Those metrics are going beyond well known technical metrics like AUC or RMSE and seem to be important in the context of CRM campaigns targeting.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Predictive quality metrics @ tinyclues - Artem Kozhevnikov - Tinyclues

  1. 1. Predictive Quality Metrics Artem Kozhevnikov Lead Data Scientist Tinyclues
  2. 2. SAAS SOLUTION USING FIRST PARTY DATA COMPATIBLE WITH ANY MARKETING STACK DESIGNED FOR MARKETERS TINYCLUES IN A FEW WORDS DEEP AI FOR CUSTOMER ACTIVATION DRIVES REVENUE AND ENGAGEMENT ACROSS ALL CHANNELS UP IN 2 WEEKS 60 employees // 30 R&D // Hiring 20
  3. 3. YOU DON’T KNOW WHAT YOUR CUSTOMERS ARE INTERESTED IN TODAY INTENT-DRIVEN MARKETING YOUR STRATEGIC OPPORTUNITY TO DRIVE REVENUE FROM ALL YOUR CUSTOMERS IS UNDERSERVED BY YOUR MASS TARGETED CAMPAIGNS 1% § ONSITE RECOMMENDATIONS § REMARKETING MESSAGES § DISPLAY RETARGETING Works well for recent visitors, but is rapidly repetitive and inefficient 99% ?
  4. 4. Success Stories RETAIL +30% CAMPAIGN REVENUE RETAIL / FASHION +151% REVENUE PER EMAIL HOSPITALITY +178% CAMPAIGN REVENUE HOSPITALITY +30% CAMPAIGN REVENUE E-COMMERCE +305% CAMPAIGN REVENUE RETAIL / FASHION +60% REVENUE PER EMAIL E-COMMERCE +80% CAMPAIGN REVENUE 10M+ 10M+ 5M+ 10M+ TRAVEL +115% CAMPAIGN REVENUE 10M+
  5. 5. FROM TOPIC TO TARGET TO MESSAGE TO REVENUE
  6. 6. Comments : • This is topic centric formulation (unlikely for onsite recommendation system) • Need to score all users, in particular, those without recent activity • In reality, our goals are more complex, we follow various campaign related ROI metrics : • CTR, CR, Opt out, Attributed Revenue, ... Predictive Problem Given a Topic, for each user u ∈ Users we want to build a predictive score such that users with higher score will have higher probability of conversion (buying) after receiving a communication about Topic through a given channel (like Email, Notifications, Facebook Custom Audience, ...).
  7. 7. Sample size vs Data Relevance tradeoff : • A. contains much more information than sparse C., • but A. is not directly related to CRM targeting problem as C. does 3 models 1 10 100 1 000 10 000 100 000 1 000 000 All sales Topic sales Topic sales attributed to email Topic sales attributed to single Topic email campaign Monthly sales count C A B
  8. 8. • A. has only Implicit Feedback (only positive) information • To set a (binary) classification problem for A. we need to define a contrast, or negative response. • There is no canonical definition for negative response : • u ∈ User at random ? • user u ∈ User that bought some other products ? • … • Scores for A. problem are not calibrated • For B. and C. we can use Explicit Feedback, so their scores are calibrated • Collecting robust feedback takes several days (delayed response) • You need to implement explore/exploit strategy to have more efficient learning for C. 3 models
  9. 9. Data Base Impression Click Channel Data Relational DataBase (Simplified Schema) User Campaign Optout Product Purchase Campaign Attributes Product AttributesPageview WebSearch Add To Cart Attribution Rules
  10. 10. Unsupervised learning • Many sparse and long-tail categorical fields in relational and time-dependent data • Heavy relying of socio-demographic fields (zipcode, firstname, age, …) • Various clustering methods for heavy tail distributions, no feature hasher • Bank of latent representations • Bayesian frameworks allowing finer meta-parameters control • Long Time series aggregation (several years of logs through different event tables) Feature Engineering Highlights
  11. 11. Latent Feature’s Bank
  12. 12. Unsupervised Feature propagation Multi-layer Unsupervised Module Multidimensional sparse tensor (DataBase) Asynchronous, daily updates Raw sparse features Scoring A. Scoring B. Scoring C. Latent Features Bank Cold Start Features Warm Start & Channel Specific Features Scoring Micro Services
  13. 13. • Train/Test AUCs at different points of pipeline • Robustness • Aggregations (average, min, max) of AUCs over most frequently used Topics • Pre/Post campaign evaluation (“in time” generalization robustness) • Accuracy/Recall at extract point • NLL, RMSE Predictive Metrics : AUC
  14. 14. • Calibration ratio = sum(observed) / sum(predicted) • Works only for calibrated scores • Monitor calibration ratio over different Topics • Independent of features engineering • Predictive Debug : simple method to see how well a feature is taken into account by model • In Pre/Post campaign shoot Predictive Metrics : Calibration Age >= 40 nb_message nb_clickers CTR sum(proba_is_clicker) mean(proba_is_clicker) Calibration ratio True 1304544 29350 2.25% 20645.54 1.58% 1.42 False 701544 21747 3.1% 30904.78 4.4% 0.7 All 2006088 51097 2.55% 51550.32 2.57% 0.99
  15. 15. • 20/80 rule : 20% of buyers make 80% of sales • You may get a very high AUCs baselines by taking very simple intensity features (top buyers, …) • Idea : compare the model of specific topic T against one of generic topic G (containing all products) • Specificity – lift with respect to generic model : Predictive Metrics : Specificity
  16. 16. Same AUCs, Different Specificity
  17. 17. • Multi-topics context : • we want to communicate on several Topics at the same moment • with some pressure constraints (<= 1 message weekly per user) Predictive Metrics : Overlaps Extract(Topic1) Extract(Topic2) Extract(Topic3)
  18. 18. SAME AUCs, DIFFERENT OVERLAPS
  19. 19. 1. Solution for an approximating problem may provide you with to very good (unsupervised) features 2. Focus on how AI is going to change your system behavior and try to find adequate offline metrics Takeaways
  20. 20. • Long term CRM planning optimization • Automatic Predictive Setup • Large scale industrialization of the predictive modules and tools Next challenges
  21. 21. Questions ? WE ARE HIRING! And many more….

×