Hila Becker, Marta Arias, "Real-time ranking with concept drift using expert advice", in Proceedings of the Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '07), 86-94

1. Real-time Ranking with Concept Drift Using Expert Advice Hila Becker and Marta Arias Center for Computational Learning Systems Columbia University

3. Problem Setting + + + - + - - - time + + + - + - - - + + + - + - - - + + + - + - - - ? . . . t-1 t 1 2 3 ? ? ? ? ? ? ? M + Feature Vector x = x 1 ,x 2 ,…,x n Label y

5. Concept Drift + + + + + - + + - - - - - - time

6. Ensemble Methods time

8. Weighted Majority Algorithm e 1 . . . e 2 e 3 e N N Experts 1 0 0 1 ? w 1 *1 + w 2 *0 + w 3 *0 + . . . + w N *1 >0.5 <0.5 1 0 1

10. Online Ranking Algorithm e 1 . . . e 2 e 3 e B w 1 w 2 w 3 w B ? F1 F4 F3 F2 F5 F4 F2 F1 F3 F5 F1 F3 F5 F4 F2 F1 F3 F4 F2 F5 F1 F3 F4 F2 F5 F3 F1 F4 F2 F5 e B+1 e B+2 w B+1 w B+2

11. Performance – Summer 05

12. Performance – Winter 06

18. Performance Metric ranking outages pAUC=17/24=0.7 0 8 0 7 0 6 1 5 0 4 1 3 1 2 0 1 8 7 6 5 4 3 2 1 1 2 3

19. Budget Variation