Métodos Adaptativos de Minería de Datos y Aprendizaje
              para Flujos de Datos.


                              ...
Minería de Datos y Aprendizaje para Flujos
   de Datos con Cambio de Concepto

                             Extraer inform...
Minería de Datos y Aprendizaje para Flujos
   de Datos con Cambio de Concepto

                             Extraer inform...
Minería de Datos Masivos


Explosión de Datos en los últimos años

            : 100 millones búsquedas por día

         ...
Minería de Datos Masivos

              Datos Masivos
2007
   Universo Digital: 281 exabytes (mil millones de gigabytes)
 ...
Minería de Datos Masivos

              Datos Masivos
2007
   Universo Digital: 281 exabytes (mil millones de gigabytes)
 ...
Minería de Datos Masivos

              Datos Masivos
2007
   Universo Digital: 281 exabytes (mil millones de gigabytes)
 ...
Minería de Datos Masivos

Koichi Kawana
Simplicidad significa conseguir el máximo efecto con los
mínimos medios.


        ...
Minería de Datos Masivos

Koichi Kawana
Simplicidad significa conseguir el máximo efecto con los
mínimos medios.


        ...
Introducción: Data Streams
Data Streams
    Secuencia potencialmente infinita
    Gran cantidad de datos: espacio sublineal...
Introducción: Data Streams
Data Streams
    Secuencia potencialmente infinita
    Gran cantidad de datos: espacio sublineal...
Introducción: Data Streams
Data Streams
    Secuencia potencialmente infinita
    Gran cantidad de datos: espacio sublineal...
Introducción: Data Streams
Data Streams
    Secuencia potencialmente infinita
    Gran cantidad de datos: espacio sublineal...
Introducción: Data Streams
Data Streams
    Secuencia potencialmente infinita
    Gran cantidad de datos: espacio sublineal...
Introducción: Data Streams
Problema
                12, 35, 21, 42, 5, 43, 57, 2, 45, 67
Dados n números no ordenados, enc...
Introducción: Data Streams
Problema
                12, 35, 21, 42, 5, 43, 57, 2, 45, 67
Dados n números no ordenados, enc...
Introducción: Data Streams
Problema
                12, 35, 21, 42, 5, 43, 57, 2, 45, 67
Dados n números no ordenados, enc...
Outline



1   Introduction


2   ADWIN : Concept Drift Mining


3   Hoeffding Adaptive Tree


4   Conclusions




       ...
Data Streams


Data Streams
At any time t in the data stream, we would like the per-item
processing time and storage to be...
Data Streams Approximation Algorithms
Frequency moments
Frequency moments of a stream A = {a1 , . . . , aN }:
            ...
Data Streams Approximation Algorithms


           1011000111 1010101

Sliding Window
We can maintain simple statistics ov...
Data Streams Approximation Algorithms


           10110001111 0101011

Sliding Window
We can maintain simple statistics o...
Data Streams Approximation Algorithms


           101100011110 1010111

Sliding Window
We can maintain simple statistics ...
Data Streams Approximation Algorithms


           1011000111101 0101110

Sliding Window
We can maintain simple statistics...
Data Streams Approximation Algorithms


           10110001111010 1011101

Sliding Window
We can maintain simple statistic...
Data Streams Approximation Algorithms


           101100011110101 0111010

Sliding Window
We can maintain simple statisti...
Outline



1   Introduction


2   ADWIN : Concept Drift Mining


3   Hoeffding Adaptive Tree


4   Conclusions




       ...
Data Mining Algorithms with Concept Drift

No Concept Drift                 Concept Drift

         DM Algorithm
         ...
Data Mining Algorithms with Concept Drift

No Concept Drift                 Concept Drift

         DM Algorithm          ...
Time Change Detectors and Predictors: A
          General Framework

                                    Estimation
      ...
Time Change Detectors and Predictors: A
          General Framework

                                          Estimation
...
Time Change Detectors and Predictors: A
          General Framework

                                                Estim...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Window Management Models

                      W = 101010110111111

Equal  fixed size              Total window against
su...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 1



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initialize Window W
2 for e...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 1 W1 = 01010110111111



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initial...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 10 W1 = 1010110111111



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initial...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 101 W1 = 010110111111



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initial...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 1010 W1 = 10110111111



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initial...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 10101 W1 = 0110111111



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initial...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 101010 W1 = 110111111



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initial...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 1010101 W1 = 10111111



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initial...
Algorithm ADWIN
Example
W = 101010110111111
W0 = 10101011 W1 = 0111111



ADWIN: A DAPTIVE W INDOWING A LGORITHM
1 Initial...
Algorithm ADWIN
Example
W = 101010110111111 |µW0 − µW1 | ≥ εc : CHANGE DET.!
                      ˆ    ˆ
W0 = 101010110 W...
Algorithm ADWIN
Example
W = 101010110111111 Drop elements from the tail of W
W0 = 101010110 W1 = 111111



ADWIN: A DAPTIV...
Algorithm ADWIN
Example
W = 01010110111111 Drop elements from the tail of W
W0 = 101010110 W1 = 111111



ADWIN: A DAPTIVE...
Algorithm ADWIN [BG07]



ADWIN has rigorous guarantees (theorems)
    On ratio of false positives
    On ratio of false n...
Algorithm ADWIN [BG07]

Theorem
At every time step we have:
 1   (Few false positives guarantee) If µt remains constant wi...
Outline



1   Introduction


2   ADWIN : Concept Drift Mining


3   Hoeffding Adaptive Tree


4   Conclusions




       ...
Classification
                    Example
                    Contains   Domain      Has        Time
Data set that       “...
Classification
Assume we have to classify the following new instance:
 Contains Domain         Has         Time
  “Money”  ...
Decision Trees




Basic induction strategy:
    A ← the “best” decision attribute for next node
    Assign A as decision ...
Hoeffding Tree / CVFDT

Hoeffding Tree : VFDT

    Pedro Domingos and Geoff Hulten.
    Mining high-speed data streams. 20...
VFDT / CVFDT

Concept-adapting Very Fast Decision Trees: CVFDT

    G. Hulten, L. Spencer, and P. Domingos.
    Mining tim...
Decision Trees: CVFDT




No theoretical guarantees on the error rate of CVFDT
CVFDT parameters :
 1   W : is the example ...
Decision Trees: Hoeffding Adaptive Tree


Hoeffding Adaptive Tree:
     replace frequency statistics counters by estimator...
What is MOA?

{M}assive {O}nline {A}nalysis is a framework for online learning
from data streams.




    It is closely re...
Ensemble Methods
      http://www.cs.waikato.ac.nz/∼abifet/MOA/




New ensemble methods:
   ADWIN bagging: When a change ...
Outline



1   Introduction


2   ADWIN : Concept Drift Mining


3   Hoeffding Adaptive Tree


4   Conclusions




       ...
Conclusions

Adaptive and parameter-free methods based in
     replace frequency statistics counters by ADWIN
         don...
Upcoming SlideShare
Loading in …5
×

Métodos Adaptativos de Minería de Datos y Aprendizaje para Flujos de Datos.

1,618 views
1,507 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,618
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
29
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Métodos Adaptativos de Minería de Datos y Aprendizaje para Flujos de Datos.

  1. 1. Métodos Adaptativos de Minería de Datos y Aprendizaje para Flujos de Datos. Albert Bifet LARCA: Laboratori d’Algorismica Relacional, Complexitat i Aprenentatge Departament de Llenguatges i Sistemes Informàtics Universitat Politècnica de Catalunya Junio 2009, Santander
  2. 2. Minería de Datos y Aprendizaje para Flujos de Datos con Cambio de Concepto Extraer información de secuencia potencialmente infinita de data datos que varian con el tiempo usando pocos recursos usando ADWIN La Desintegración de la ADaptive Sliding WINdow: Persistencia de la Memoria Ventana deslizante adaptativa 1952-54 sin parámetros Salvador Dalí 2 / 29
  3. 3. Minería de Datos y Aprendizaje para Flujos de Datos con Cambio de Concepto Extraer información de secuencia potencialmente infinita de data datos que varian con el tiempo usando pocos recursos usando ADWIN La Desintegración de la ADaptive Sliding WINdow: Persistencia de la Memoria Ventana deslizante adaptativa 1952-54 sin parámetros Salvador Dalí 2 / 29
  4. 4. Minería de Datos Masivos Explosión de Datos en los últimos años : 100 millones búsquedas por día : 20 millones transacciones por día 1,000 millones de transacciones de tarjetas de credito por mes 3,000 millones de llamadas telefónicas diarias en EUA 30,000 millones de e-mails diarios, 1,000 millones de SMS Tráfico de redes IP: 1,000 millones de paquetes por hora por router 3 / 29
  5. 5. Minería de Datos Masivos Datos Masivos 2007 Universo Digital: 281 exabytes (mil millones de gigabytes) La cantidad de información creada excedió el almacenaje disponible por primera vez Green Computing Estudio y práctica de como usar recursos informáticos eficientemente. Algorithmic Efficiency Una de las principales maneras de hacer Green Computing 4 / 29
  6. 6. Minería de Datos Masivos Datos Masivos 2007 Universo Digital: 281 exabytes (mil millones de gigabytes) La cantidad de información creada excedió el almacenaje disponible por primera vez Green Computing Estudio y práctica de como usar recursos informáticos eficientemente. Algorithmic Efficiency Una de las principales maneras de hacer Green Computing 4 / 29
  7. 7. Minería de Datos Masivos Datos Masivos 2007 Universo Digital: 281 exabytes (mil millones de gigabytes) La cantidad de información creada excedió el almacenaje disponible por primera vez Green Computing Estudio y práctica de como usar recursos informáticos eficientemente. Algorithmic Efficiency Una de las principales maneras de hacer Green Computing 4 / 29
  8. 8. Minería de Datos Masivos Koichi Kawana Simplicidad significa conseguir el máximo efecto con los mínimos medios. Donald Knuth “... we should make use of the idea of limited resources in our own education. We can all benefit by doing occasional "toy" programs, when artificial restrictions are set up, so that we are forced to push our abilities to the limit. “ 5 / 29
  9. 9. Minería de Datos Masivos Koichi Kawana Simplicidad significa conseguir el máximo efecto con los mínimos medios. Donald Knuth “... we should make use of the idea of limited resources in our own education. We can all benefit by doing occasional "toy" programs, when artificial restrictions are set up, so that we are forced to push our abilities to the limit. “ 5 / 29
  10. 10. Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1, . . . , n}. Sea π−1 la permutación π con un elemento que falta. π−1 [i] llega en orden creciente Tarea: Determinar el número que falta 6 / 29
  11. 11. Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1, . . . , n}. Sea π−1 la permutación π con un elemento que falta. π−1 [i] llega en orden creciente Tarea: Determinar el número que falta 6 / 29
  12. 12. Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1, . . . , n}. Usar un vector n-bit para Sea π−1 la permutación π con un memorizar todos elemento que falta. los numeros π−1 [i] llega en orden creciente (espacio O(n) ) Tarea: Determinar el número que falta 6 / 29
  13. 13. Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1, . . . , n}. Data Streams: Sea π−1 la permutación π con un espacio elemento que falta. O(log(n)). π−1 [i] llega en orden creciente Tarea: Determinar el número que falta 6 / 29
  14. 14. Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1, . . . , n}. Almacenar Sea π−1 la permutación π con un n(n + 1) − ∑ π−1 [j]. elemento que falta. 2 j≤i π−1 [i] llega en orden creciente Tarea: Determinar el número que falta 6 / 29
  15. 15. Introducción: Data Streams Problema 12, 35, 21, 42, 5, 43, 57, 2, 45, 67 Dados n números no ordenados, encontrar un número que esté en la mitad superior de la lista ordenada. 2, 5, 12, 21, 35 42, 43, 45, 57, 67 Algoritmo Elegir k números aleatoriamente. Devolver el número mayor. Análisis La probabilidad de que la solución sea incorrecta es la probabilidad de que todos los k números estén en la mitad inferior : (1/2)k Para tener probabilidad δ usaremos k = log 1/δ muestras 7 / 29
  16. 16. Introducción: Data Streams Problema 12, 35, 21, 42, 5, 43, 57, 2, 45, 67 Dados n números no ordenados, encontrar un número que esté en la mitad superior de la lista ordenada. 2, 5, 12, 21, 35 42, 43, 45, 57, 67 Algoritmo Elegir k números aleatoriamente. Devolver el número mayor. Análisis La probabilidad de que la solución sea incorrecta es la probabilidad de que todos los k números estén en la mitad inferior : (1/2)k Para tener probabilidad δ usaremos k = log 1/δ muestras 7 / 29
  17. 17. Introducción: Data Streams Problema 12, 35, 21, 42, 5, 43, 57, 2, 45, 67 Dados n números no ordenados, encontrar un número que esté en la mitad superior de la lista ordenada. 2, 5, 12, 21, 35 42, 43, 45, 57, 67 Algoritmo Elegir k números aleatoriamente. Devolver el número mayor. Análisis La probabilidad de que la solución sea incorrecta es la probabilidad de que todos los k números estén en la mitad inferior : (1/2)k Para tener probabilidad δ usaremos k = log 1/δ muestras 7 / 29
  18. 18. Outline 1 Introduction 2 ADWIN : Concept Drift Mining 3 Hoeffding Adaptive Tree 4 Conclusions 8 / 29
  19. 19. Data Streams Data Streams At any time t in the data stream, we would like the per-item processing time and storage to be simultaneously O(log k (N, t)). Approximation algorithms Small error rate with high probability ˜ An algorithm (ε, δ )−approximates F if it outputs F for which ˜ − F | > εF ] < δ . Pr[|F 9 / 29
  20. 20. Data Streams Approximation Algorithms Frequency moments Frequency moments of a stream A = {a1 , . . . , aN }: v Fk = ∑ fik i=1 where fi is the frequency of i in the sequence, and k ≥ 0 F0 : number of distinct elements on the sequence F1 : length of the sequence F2 : self-join size, the repeat rate, or as Gini’s index of homogeneity Sketches can approximate F0 , F1 , F2 in O(log v + log N) space. Noga Alon, Yossi Matias, and Mario Szegedy. The space complexity of approximation the frequency moments. 1996 10 / 29
  21. 21. Data Streams Approximation Algorithms 1011000111 1010101 Sliding Window We can maintain simple statistics over sliding windows, using O( 1 log2 N) space, where ε N is the length of the sliding window ε is the accuracy parameter M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. 2002 11 / 29
  22. 22. Data Streams Approximation Algorithms 10110001111 0101011 Sliding Window We can maintain simple statistics over sliding windows, using O( 1 log2 N) space, where ε N is the length of the sliding window ε is the accuracy parameter M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. 2002 11 / 29
  23. 23. Data Streams Approximation Algorithms 101100011110 1010111 Sliding Window We can maintain simple statistics over sliding windows, using O( 1 log2 N) space, where ε N is the length of the sliding window ε is the accuracy parameter M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. 2002 11 / 29
  24. 24. Data Streams Approximation Algorithms 1011000111101 0101110 Sliding Window We can maintain simple statistics over sliding windows, using O( 1 log2 N) space, where ε N is the length of the sliding window ε is the accuracy parameter M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. 2002 11 / 29
  25. 25. Data Streams Approximation Algorithms 10110001111010 1011101 Sliding Window We can maintain simple statistics over sliding windows, using O( 1 log2 N) space, where ε N is the length of the sliding window ε is the accuracy parameter M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. 2002 11 / 29
  26. 26. Data Streams Approximation Algorithms 101100011110101 0111010 Sliding Window We can maintain simple statistics over sliding windows, using O( 1 log2 N) space, where ε N is the length of the sliding window ε is the accuracy parameter M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. 2002 11 / 29
  27. 27. Outline 1 Introduction 2 ADWIN : Concept Drift Mining 3 Hoeffding Adaptive Tree 4 Conclusions 12 / 29
  28. 28. Data Mining Algorithms with Concept Drift No Concept Drift Concept Drift DM Algorithm DM Algorithm input output input output - Counter5 - - - Counter4 Static Model Counter3 6 Counter2 Counter1 - Change Detect. 13 / 29
  29. 29. Data Mining Algorithms with Concept Drift No Concept Drift Concept Drift DM Algorithm DM Algorithm input output input output - Counter5 - - Estimator5 - Counter4 Estimator4 Counter3 Estimator3 Counter2 Estimator2 Counter1 Estimator1 13 / 29
  30. 30. Time Change Detectors and Predictors: A General Framework Estimation - xt - Estimator 14 / 29
  31. 31. Time Change Detectors and Predictors: A General Framework Estimation - xt - Estimator Alarm - - Change Detect. 14 / 29
  32. 32. Time Change Detectors and Predictors: A General Framework Estimation - xt - Estimator Alarm - - Change Detect. 6 6 ? - Memory 14 / 29
  33. 33. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 1 01010110111111 [Dasu+ 06] 15 / 29
  34. 34. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 10 1010110111111 [Dasu+ 06] 15 / 29
  35. 35. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 101 010110111111 [Dasu+ 06] 15 / 29
  36. 36. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 1010 10110111111 [Dasu+ 06] 15 / 29
  37. 37. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 10101 0110111111 [Dasu+ 06] 15 / 29
  38. 38. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 101010 110111111 [Dasu+ 06] 15 / 29
  39. 39. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 1010101 10111111 [Dasu+ 06] 15 / 29
  40. 40. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 10101011 0111111 [Dasu+ 06] 15 / 29
  41. 41. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 101010110 111111 [Dasu+ 06] 15 / 29
  42. 42. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 1010101101 11111 [Dasu+ 06] 15 / 29
  43. 43. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 10101011011 1111 [Dasu+ 06] 15 / 29
  44. 44. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 101010110111 111 [Dasu+ 06] 15 / 29
  45. 45. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 1010101101111 11 [Dasu+ 06] 15 / 29
  46. 46. Window Management Models W = 101010110111111 Equal fixed size Total window against subwindows subwindow 1010 1011011 1111 10101011011 1111 [Kifer+ 04] [Gama+ 04] Equal size adjacent subwindows ADWIN: All Adjacent subwindows 1010101 1011 1111 10101011011111 1 [Dasu+ 06] 11 15 / 29
  47. 47. Algorithm ADWIN Example W = 101010110111111 W0 = 1 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  48. 48. Algorithm ADWIN Example W = 101010110111111 W0 = 1 W1 = 01010110111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  49. 49. Algorithm ADWIN Example W = 101010110111111 W0 = 10 W1 = 1010110111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  50. 50. Algorithm ADWIN Example W = 101010110111111 W0 = 101 W1 = 010110111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  51. 51. Algorithm ADWIN Example W = 101010110111111 W0 = 1010 W1 = 10110111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  52. 52. Algorithm ADWIN Example W = 101010110111111 W0 = 10101 W1 = 0110111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  53. 53. Algorithm ADWIN Example W = 101010110111111 W0 = 101010 W1 = 110111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  54. 54. Algorithm ADWIN Example W = 101010110111111 W0 = 1010101 W1 = 10111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  55. 55. Algorithm ADWIN Example W = 101010110111111 W0 = 10101011 W1 = 0111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  56. 56. Algorithm ADWIN Example W = 101010110111111 |µW0 − µW1 | ≥ εc : CHANGE DET.! ˆ ˆ W0 = 101010110 W1 = 111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  57. 57. Algorithm ADWIN Example W = 101010110111111 Drop elements from the tail of W W0 = 101010110 W1 = 111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  58. 58. Algorithm ADWIN Example W = 01010110111111 Drop elements from the tail of W W0 = 101010110 W1 = 111111 ADWIN: A DAPTIVE W INDOWING A LGORITHM 1 Initialize Window W 2 for each t 0 3 do W ← W ∪ {xt } (i.e., add xt to the head of W ) 4 repeat Drop elements from the tail of W 5 until |µW0 − µW1 | ≥ εc holds ˆ ˆ 6 for every split of W into W = W0 · W1 7 ˆ Output µW 16 / 29
  59. 59. Algorithm ADWIN [BG07] ADWIN has rigorous guarantees (theorems) On ratio of false positives On ratio of false negatives On the relation of the size of the current window and change rates Other methods in the literature: [Gama+ 04], [Widmer+ 96], [Last 02] don’t provide rigorous guarantees. 17 / 29
  60. 60. Algorithm ADWIN [BG07] Theorem At every time step we have: 1 (Few false positives guarantee) If µt remains constant within W , the probability that ADWIN shrinks the window at this step is at most δ . 2 (Few false negatives guarantee) If for any partition W in two parts W0 W1 (where W1 contains the most recent items) we have |µW0 − µW1 | ε, and if 3 max{µW0 , µW1 } 4n ε ≥ 4· ln min{n0 , n1 } δ then with probability 1 − δ ADWIN shrinks W to W1 , or shorter. 18 / 29
  61. 61. Outline 1 Introduction 2 ADWIN : Concept Drift Mining 3 Hoeffding Adaptive Tree 4 Conclusions 19 / 29
  62. 62. Classification Example Contains Domain Has Time Data set that “Money” type attach. received spam describes e-mail yes com yes night yes features for yes edu no night yes deciding if it is no com yes night yes spam. no edu no day no no com no day no yes cat no day yes Assume we have to classify the following new instance: Contains Domain Has Time “Money” type attach. received spam yes edu yes day ? 20 / 29
  63. 63. Classification Assume we have to classify the following new instance: Contains Domain Has Time “Money” type attach. received spam yes edu yes day ? 20 / 29
  64. 64. Decision Trees Basic induction strategy: A ← the “best” decision attribute for next node Assign A as decision attribute for node For each value of A, create new descendant of node Sort training examples to leaf nodes If training examples perfectly classified, Then STOP, Else iterate over new leaf nodes 21 / 29
  65. 65. Hoeffding Tree / CVFDT Hoeffding Tree : VFDT Pedro Domingos and Geoff Hulten. Mining high-speed data streams. 2000 With high probability, constructs an identical model that a traditional (greedy) method would learn With theoretical guarantees on the error rate 22 / 29
  66. 66. VFDT / CVFDT Concept-adapting Very Fast Decision Trees: CVFDT G. Hulten, L. Spencer, and P. Domingos. Mining time-changing data streams. 2001 It keeps its model consistent with a sliding window of examples Construct “alternative branches” as preparation for changes If the alternative branch becomes more accurate, switch of tree branches occurs 23 / 29
  67. 67. Decision Trees: CVFDT No theoretical guarantees on the error rate of CVFDT CVFDT parameters : 1 W : is the example window size. 2 T0 : number of examples used to check at each node if the splitting attribute is still the best. 3 T1 : number of examples used to build the alternate tree. 4 T2 : number of examples used to test the accuracy of the alternate tree. 24 / 29
  68. 68. Decision Trees: Hoeffding Adaptive Tree Hoeffding Adaptive Tree: replace frequency statistics counters by estimators don’t need a window to store examples, due to the fact that we maintain the statistics data needed with estimators change the way of checking the substitution of alternate subtrees, using a change detector with theoretical guarantees Summary: 1 Theoretical guarantees 2 No Parameters 25 / 29
  69. 69. What is MOA? {M}assive {O}nline {A}nalysis is a framework for online learning from data streams. It is closely related to WEKA It includes a collection of offline and online as well as tools for evaluation: boosting and bagging Hoeffding Trees with and without Naïve Bayes classifiers at the leaves. 26 / 29
  70. 70. Ensemble Methods http://www.cs.waikato.ac.nz/∼abifet/MOA/ New ensemble methods: ADWIN bagging: When a change is detected, the worst classifier is removed and a new classifier is added. Adaptive-Size Hoeffding Tree bagging 27 / 29
  71. 71. Outline 1 Introduction 2 ADWIN : Concept Drift Mining 3 Hoeffding Adaptive Tree 4 Conclusions 28 / 29
  72. 72. Conclusions Adaptive and parameter-free methods based in replace frequency statistics counters by ADWIN don’t need a window to store examples, due to the fact that we maintain the statistics data needed with ADWINs using ADWIN as change detector with theoretical guarantees, Summary: 1 Theoretical guarantees 2 No parameters needed 3 Higher accuracy 4 Less space needed 29 / 29

×