Real-time ranking with concept drift using expert advice
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Real-time ranking with concept drift using expert advice

  • 1,003 views
Uploaded on

Hila Becker, Marta Arias, "Real-time ranking with concept drift using expert advice", in Proceedings of the Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining......

Hila Becker, Marta Arias, "Real-time ranking with concept drift using expert advice", in Proceedings of the Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '07), 86-94

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,003
On Slideshare
995
From Embeds
8
Number of Embeds
3

Actions

Shares
Downloads
5
Comments
0
Likes
0

Embeds 8

http://www.linkedin.com 6
http://www.slideshare.net 1
http://www.slideee.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Given an infinite amount of continuous measurement, how do we model them in order to capture possibly time-evolving trends and patterns in the stream, compute the optimal model and make time critical decisions.
  • Compute weighted average, divide into bins [i/epsilon,i+1/epsilon], compute the mean and std. div for the bin and check if can make confident prediction. (Fx-mean-std*t > cost/transaction
  • here, the rules of the game change

Transcript

  • 1. Real-time Ranking with Concept Drift Using Expert Advice Hila Becker and Marta Arias Center for Computational Learning Systems Columbia University
  • 2. Dynamic Ranking
    • Continuous arrival of data over time
    • Set of items to rank
      • Dynamic features
      • Adapt to Changes
    • Given a list of electrical grid components, produce a ranking according to failure susceptibility
  • 3. Problem Setting + + + - + - - - time + + + - + - - - + + + - + - - - + + + - + - - - ? . . . t-1 t 1 2 3 ? ? ? ? ? ? ? M + Feature Vector x = x 1 ,x 2 ,…,x n Label y
  • 4. Challenges
    • Changes in underlying distribution
      • Hidden
      • Concept drift
      • Adapt learning model to improve predictions
    • Finite storage space
      • Sample from the data
      • Discard old or irrelevant information
  • 5. Concept Drift + + + + + - + + - - - - - - time
  • 6. Ensemble Methods time
  • 7. Weighted Expert Ensembles
    • Associate a weight with each expert
    • Measure of belief in expert performance
    • Weights used in final prediction
      • Use only the best expert
      • Weighted average of predictions
    • Update the weights after every prediction
  • 8. Weighted Majority Algorithm e 1 . . . e 2 e 3 e N N Experts 1 0 0 1 ? w 1 *1 + w 2 *0 + w 3 *0 + . . . + w N *1 >0.5 <0.5 1 0 1
  • 9. Modified Weighted Majority
    • Different Constrains for data streams
      • Incorporate new data
      • Static vs. Dynamic set of experts
    • Ranking Algorithm
      • Loss function – 1-normalized average rank of positive examples
      • Combine Predictions – weighted average rank
  • 10. Online Ranking Algorithm e 1 . . . e 2 e 3 e B w 1 w 2 w 3 w B ? F1 F4 F3 F2 F5 F4 F2 F1 F3 F5 F1 F3 F5 F4 F2 F1 F3 F4 F2 F5 F1 F3 F4 F2 F5 F3 F1 F4 F2 F5 e B+1 e B+2 w B+1 w B+2
  • 11. Performance – Summer 05
  • 12. Performance – Winter 06
  • 13. Contributions
    • Additive weighted ensemble based on the Weighted Majority algorithm
    • Algorithm adapted to ranking
    • Experiments on a Real-world datastream
      • Outperform traditional approaches
      • Explore performance/complexity tradeoffs
  • 14. Future Work
    • Ensemble diversity control
    • Exploit re-occurring contexts
      • Use knowledge of cyclic patterns
      • Revive old experts
    • Change detection
    • Statistical estimation of predicting ensemble size
  • 15. Ensemble Methods
    • Static ensemble with online learners [Hulten ’01]
    • Use batch-learners as experts
      • Can use many learning algorithms
      • Loses interpretability
    • Additive ensembles
      • Train an expert at constant intervals [Street and Kim ’01]
      • Train an expert when performance declines [Kolter ’05]
  • 16. Ensemble Pruning
    • Additive ensembles can grow infinitely large
    • Criteria for removing experts
      • Age - retire oldest model [Chu and Zaniolo ‘04]
      • Performance
        • Worst in the ensemble
        • Below a minimal threshold [Stanley ’01]
      • Instance-based Pruning [Wang et al. ’03]
  • 17. Dealing with a moving set of experts
    • Introduce new parameters
      • B: “budget” (max number of models) set to 100
      • p: new models weight percentile in [0,100]
      •  : age penalty in (0,1]
    • If too many models (more than B), drop models with poor q-score, where
      • q i = w i • pow(  , age i )
      • I.e.,  is rate of exponential decay
  • 18. Performance Metric ranking outages pAUC=17/24=0.7 0 8 0 7 0 6 1 5 0 4 1 3 1 2 0 1 8 7 6 5 4 3 2 1 1 2 3
  • 19. Budget Variation
  • 20. Data Streams
    • Continuous arrival of data over time
    • Real-world applications
      • Consumer shopping patterns
      • Weather prediction
      • Electricity load forecasting
    • Increased attention
      • Companies collect data
      • Traditional approaches do not apply