fast memory efficient local outlier factor

1. DEPARTMENT OF COMPUTER ENGINEERING
K.K. Wagh Institute of Engineering Education and Research , Nashik
Presented By
Diksha R. Gupta (Roll No : 07)
Exam Seat No:- 6960
Guided By
Prof. N. M. Shahane
Seminar
On
Fast Memory Efficient Local Outlier
Detection in Data Streams

2. Contents
• Introduction
• Literature Survey
• Related Work
• Local Outlier Factor: LOF
• MILOF: Memory efficient ILOF algorithm
And the MILOF_F variant
• Comparison of Time Complexity in Different
Implementations
• Conclusion
• References
1/14

3. Introduction
• Outlier detection has received considerable attention
in the field of data mining because of the need to
detect unusual events in a variety of applications,
such as fraud detection, human gait analysis and
intrusion detection.
• A variety of outlier detection algorithms has been
proposed for use on static data sets which have a
finite number of samples.
2/14

4. Cont…….
• Data streams are also generated at a high data
rate, hence making the task of outlier detection
even more challenging.
• This is a significant problem that arises in many
real applications.
• For example, an outlier detection system in
wireless sensor networks must work with the
limited memory in each sensor node in order to
detect rare events in near real time.
3/14

5. Cont….
Fig. Scatter plots of a two-dimensional
synthetic data set evolving over time,
containing normal data points (circles) and
outliers (stars)
4/14

6. Literature Survey
Sr. No Paper Name Related Work
1 Local Outlier Detection for Data
Streams in Sensor Networks
LOF and iLOF
algorithms
2 Anomaly Detection: A Survey Hodge and Austin
[2004] provide an
extensive survey of
anomaly detection
techniques developed in
machine learning and
statistical domains.
3 Fast Memory Efficient Local Outlier
Detection in Data Streams
LOF AND ILOF,
MILOF algorithm
5/14

7. Related Work
• Three categories of approaches have been
proposed to detect outliers in data streams:-
1. Distribution based
2. Clustering based
3. Distance based approaches
6/14

8. There are two type of algorithms used :-
1. LOF (Local Outlier Factor)
2. MILOF (memory efficient incremental
local outlier detection)
7/14

9. Local Outlier Factor: LOF
• Reachability distance from p to o:
-where k is a user-specified parameter
• Local reachability density of p:
• LOF (Local outlier factor) of an object p
8/14

10. Fig. Three phases of MiLOF on a 2-D data stream with dimensions
(d1; d2).
MILOF: Memory efficient ILOF
algorithm
And the MILOF_F variant
9/14

11. Algorithm Existin
g data
points
Efficient
Implementatio
n
Simple
Implementati
on
Insertion Total Insertion Total
MiLOF/MiLOF_F b + c Log(b+c) nlog(b+c) b + c n(b + c)
iLOF n Log(n) nlog(n) n n*n
Comparison of Time Complexity in Different
Implementations
10/14

12. Conclusion
• Memory efficiency is an important requirement in the field of
outlier detection for data streams.
• There are two approaches to tackle the problem of identifying
local outliers.
• The first algorithm (MiLOF) has substantial practical
significance in improving widely cited methods (LOF and
incremental LOF).
• iLOF requires saving all previous data points to compute local
outlier factors.
• MiLOF method avoids this difficulty by summarizing subsets
of the previous data and accumulating an evolutionary history
of the streaming information.
• The MiLOF saves computation time as well as memory.
11/14

13. REFERENCES
1. S. Papadimitriou, H. Kitagawa, P. B. Gibbons, and C.
Faloutsos, “LOCI: Fast outlier detection using the local
correlation integral,” in Proc. Int. Conf. Data Eng., 2003, pp.
315–326.
2. D. Pokrajac, A. Lazarevic, and L. J. Latecki, “Incremental
local outlier detection for data streams,” in Proc. IEEE Symp.
Comput. Intell. Data Mining, 2007, pp. 504–515.
3. S. Rajasegarar, C. Leckie, and M. Palaniswami, “Anomaly
detection in wireless sensor networks,” IEEE Wireless
Commun., vol. 15, no. 4, pp. 34–40, Aug. 2008.
4. M. Gupta, J. Gao, C. C. Aggarwal, and J. Han, “Outlier
detection for temporal data: A survey,” IEEE Trans. Knowl.
Data Eng., vol. 25, no. 1, pp. 1–20, Jan. 2013.
12/14

14. Thank You