SmartNews TechNight Vol.5 : AD Data Engineering in practice: SmartNews Ads裏のデータシステム

15,819 views

Published on

SmartNews Ads裏のデータシステムでは、様々なソースからデータを集め、広告クリックの予測を初め、ユーザプロファイリングや配信ターゲティングなど色んなアルゴリズムが動いています。このセッションでは、この一年間SmartNew Adsにおけるデータシステムの実践、特に広告クリックシステムの進化を話します!

Published in: Technology

SmartNews TechNight Vol.5 : AD Data Engineering in practice: SmartNews Ads裏のデータシステム

  1. 1. Data Engineering In Practice: SmartNews Ads裏のDMP System Lan
  2. 2. Who am I • Lan • Veteran hacker but new in AD world • someone who can make a computer do what he wants—whether the computer wants to or not. (http://paulgraham.com/gba.html) • ex-{Rakuten, GREE} • Distribution System, Info Retrieval, ML
  3. 3. Today’s Talk • DMP in SmartNews Ads • #1. Prediction • #2. Targeting • Future Work & Summary
  4. 4. DMP = Data Management Platform
  5. 5. DMP in SmartNews Ads • Private DMP ( 90%+1st-party data ) • Data Collect, Clean, Aggregation • ID Mapping • User Profiling • User Clustering • CTR / CVR Prediction • Lookalike • Custom Audience
  6. 6. DMP Clusters AD delivery cluster AD Log in S3 Kinesis AD tracker Video AD delivery cluster DMP streaming Audience Data in DynamoDBRDB Hadoop ML Analytics Models & Targeting SmartNews Log ML Small company but not small data •Article Meta > 200K/day •Article x {read, share, read_related …} •Channel x {subscribe, preview, view, …} •Push, Live, Weather, Setting, … •Survey result •Audience Data > 14M (~5M MAU) •AD Meta •AD History •AD Conversions •AD Optout • Managed/Compressed Data > 130TB • Lookalike seeds • ~1TB Data for training CTR prediction model •> 1M unique features •User Demographics •Device •Locations •…
  7. 7. #1 Prediction
  8. 8. Pick up an AD to feed here
  9. 9. Similar to Recommendation but DIFFERENT • optimization goal • accuracy of the probability
  10. 10. More than Ranking • When we do AD auction • eCPM (effective Cost per Mille) = CTR (Click Through Rate) x CPC (Cost per Click) • Suppose we have • CTRad1=0.05 > CTRad2=0.04 > CTRad3=0.03 • CPCad1 = 10JPY, CPCad2 = 13JPY, CPCad3 = 20JPY(winner) • but if: pCTRad1 = 0.2 (winner) > pCTR’ad2 = 0.1 > pCTR’ad3 = 0.03 • then we lost 0.1JPY potential income
  11. 11. The CTR(CVR) prediction Problem μ(a, u, c) = p(click | a,u,c)
  12. 12. CTR Prediction v1 • Train and scoring daily • One GBDT (Gradient Boosting Decision Tree) model per AD campaign • using ~1month’s data • Hundreds of small batches inside Hadoop Yarn • Quick and Simple • dev in 1 month • pick up best features for every campaign • minutes ~ 1 hour for model training • explainable Tree models • no need for AD feature • Same approach for CVR prediction (CPC / CVR = CPA (Cost Per Acquisition) ) delivery result User Features generate samples Yarn Users predictions sample model scoring sample model scoring sample model scoring …
  13. 13. Metrics • NE (Normalized Cross- Entropy) • the average log loss when using predicted CTR / the average log loss per impression • https://facebook.com//download/321355358042503/adkdd_2014_camera_ready_junfeng.pdf • AUC (Area under the ROC curve, AUROC) • measure ranking quality • others: Precision/Recall, ECS(Effective catalog size), CTR / CVR / Sales, etc
  14. 14. Review of CTR Prediction v1 • Marked improvement, moderate AUC & NE • And • hard to do overall tuning • hard to prediction online (feature set differs) • latency for new campaigns • relatively poor performance to new campaigns (cold start) • lost the connections between campaigns even for the same advertiser • …
  15. 15. CTR Prediction v2 • A simple model for all • AD feature added • Dynamic features extraction • All calculation distributed • GBDT + LogisticRegression • Train once per day, scoring twice
  16. 16. About the Features • >1M unique features, sparse • GBDT provides great feature engineering • (sometimes) feature engineering is kind of intuition and trial-and-error • demographic, device, location, reading interests… • AD history is helpful • Feature Hashing, Binarization & Discretization, …
  17. 17. Performance improvement
  18. 18. #2 Targeting Watabe TamTam Komiya Takei Ikeishi Nagase Lan Niku Game Beer Snack Costume Gourmet Princess
  19. 19. It’s difficult comparing to
  20. 20. Profiling User by Statistics and ML • Gender Prediction (precision: 0.90+), Age Prediction, … • News Channel / Source Preference • AD Slot Preference • …
  21. 21. Standard Targeting • Female in Kansai who subscribes Travel Channel
  22. 22. Lookalike Targeting
  23. 23. Lookalike Targeting • Our solution • Solve it as an classification problem • Seed user as Positive Sample • While all targeting candidates as Negative Sample (w/ random sampling ) • based on Spark MLlib Logistic Regression • 30%~50% CVR↑ comparing to normal targeting
  24. 24. Article Keyword Targeting Keyword Realtime Calculating Reach UU Only user who exceeds a certain read-time threshold will be included
  25. 25. Custom Audience SmartNews AD tracker Send any custom event (S2S req, web beacon, etc) Event Audience BloomFilter Obj Updating per Several Minutes Your Service / App / Site SmartNews AD Delivery Cluster AD targeting / Delete Targeting Lookalike Lookalike Targeting
  26. 26. Future Work
  27. 27. Targeting Audience by Interests
  28. 28. Collect Negative Signal to Optimize UX
  29. 29. Summary of My 1st SmartNews Year • Challenge place. We’re startup so we can move quick and break things • Learn from the industry leaders. Keep trial-and-error. • Number don’t lie. Don’t trust your intuition over number. • But if you really doubt the number, look closely. there may be BUG hidden.

×