H K Yoon, profile picture
Uploaded byH K Yoon
190 views

Outlier Analysis.pdf

The document covers various statistical concepts including measures of central tendency, variability, association, and probabilistic models. It discusses techniques for outlier detection, time series analysis, and multivariate analysis, along with a survey of algorithms ranging from traditional statistical methods to modern machine learning approaches. Additionally, it addresses challenges in unsupervised learning and methods for data representation, including clustering and dimensionality reduction.

Technology◦
Related topics:
Deep Learning•

Embed presentation

Download to read offline
Contents
• (1) 중심경향성: Ungrouped Data • Mode, Mean, Median • Percentile, Quantile/Quartile • (2) 변동성: Ungrouped Data • Range & IQR (Interquartile Range) • MAD (Mean Absolute Deviation), Variance, Standard Deviation • 분산 및 표준편차 • Unbiased estimator • Z-score • (3) Measures of Shape • Moment • Skewness와 Kurtosis • (4) 연관성 (Association) 측도 • Correlation
• 기본개념 • Experiment, (Elementary) , Event와 Sample Space , … • 조건부 확률과 Bayes’ rule • 조건부 확률 법칙 P(X | Y) = (P(X ∩ Y))/(P(Y)) = (P(X)•((Y|X))/(P(Y)) • 독립성 여부의 검정: P(X | Y) = P(X) and P(Y| X) = P(Y) • Bayes’ Rule • P(Xi | Y) = 𝑃 𝑋𝑖 •𝑃(𝑌|𝑋𝑖) 𝑃 𝑋1 •𝑃 𝑌 𝑋1 + 𝑃 𝑋2 •𝑃 𝑌 𝑋2 +⋯+𝑃 𝑋𝑛 •𝑃(𝑌|𝑋𝑛) • Odds • 확률변수와 확률분포 • 확률변수 • = a variable that contains the outcomes of a chance experiment • 확률분포 • 이산분포: 이항분포, Poisson분포 • 연속분포: 균일분포, 정규분포, t-분포, 지수분포, 𝜒2 분포
• 추정 • 정규분포 • t-분포 • 가설검정 • p-Value를 이용한 가설검정 • p-value = 관측된 유의수준 (level of significance) • defines the smallest value of 𝛼 for which the H0 can be rejected. • “α 가 p보다 커야만 H0를 reject 가능”
• Variate(s) • Univariate • Bi-variate • 공분산향렬과 SSCP • Multivariate • Multivariate • 다변량 확률분포 • 다변량정규분포 • 다변량 분석기법 • PCA • Factor Analysis • MDS • ANOVA/ANCOVA/MANOVA/MANCOVA/… • 다변량 다중회귀분석 (MVMLR)
• 주요 이슈 • Subjectivity, Interestingness and noise • Subjective judgement, as to what constitutes a “sufficient” deviation • In real applications, the data may be embedded in a significant amount of noise • Noise 문제 • Representation의 문제 • Normality vs. Anomaly • 특성공학 • 이상치 분석과 데이터 모델
• 알고리즘 분류 (1) • Outlier scores: quantify level of “outlierness” • Binary labels: “outlier? or not?” • the threshold is based on the statistical distribution of the scores. • 알고리즘 분류 (2)
• qualitative techniques, • 전문가 의견 • time series analysis • 관심사항: patterns and pattern changes • 과거 데이터가 중요한 역할 • causal models. • 인과관계 • 과거 데이터가 중요한 역할 • Regression • 독립변수와 종속변수
단변량 Time Series 모델 • AR Model • 자기상관 (Autocorrelation) • 정상성 (Stationarity)과 ADF Test • Differencing a Time Series • Autocorrelation에서의 Lags • Partial Autocorrelation • AR 모델 정의 • Yule-Walker Equation 이용한 AR 추정 • MA Model • MA 모델 정의 • MA Model의 Fitting • Stationarity • AR vs. MA 모델 선택 • Model Retraining을 통한 다단계 예측 • 최적의 MA Order를 찾기 위한 Grid Search
• ARMA 모델 • 모델 정의 • ARMA(1,1) Model의 Fitting • Automated Hyperparameter Tuning • Grid Search • 성능향상을 위한 Tuning • ARIMA 모델 • 모델 정의 • SARIMA 모델 • 모델 정의 • regular AR part φp • seasonal AR part • regular MA part θq • seasonal MA part • regular integration part; order d • seasonal integration part; order D • Coefficient of seasonality s
• Multivariate Time Series Models • SARIMAX 모델 • SARIMA 모델에 외생변수 (X)를 추가 • VAR 모델 • Since VAR model proposes one model for multiple target variables, it regroups those variables as a vector • VAR 계수의 추정 • VARMAX 모델 • VAR model에 MV 항을 추가하고 외생변수 허용 • V for vector indicating that it’s a multivariate model • AR for autoregression • MA for moving average • X for the use of exogenous variables (in addition to the endogenous variables)
• Linear Regression • kNN, 의사결정 트리/Random Forest • XGBoost와 LightGBM
• RNN/LSTM • Predicting a Sequence Rather Than a Value • SimpleRNN • GRU, LSTM • Prophet 모델 (Facebook) • an automated procedure for building forecasting models developed by Facebook. • Input possibilities are • Seasonality of any regular order • Holidays • Additional regressors • hyperparameters • Fourier order of the seasonality: A higher order means more flexibility. • changepoint_prior_scale plays on the trend: The higher the value, the more flexible the trend. • holidays_prior_scale: The lower it is, the less important the holidays are for the model. • prior scale for the seasonality • DeepAR 모델
• Euclidean 거리 • Manhattan 거리 • Minkowski 거리 • Cosine 거리
• Distance Distribution-based Techniques • to model entire data set to be normally distributed about its mean in the form of a multivariate Gaussian distribution. • Let ҧ 𝜇 be d-dimensional (row) data set, and Σ be its d x d covariance matrix. • Then, the probability distribution 𝑓( ത 𝑋) for a d-dimensional (row vector) data point X is: • |Σ| = determinant of covariance matrix. • 지수부: (half) squared Mahalanobis distance of the data point X to the centroid μ of the data. = outlier score
• Extreme-Value Analysis • 극값을 판별해 내는 것 • = Probabilistic Tail Inequalities • Markov Inequality • Chebychev Inequality • … • determine statistical tails of the underlying distribution. • Univariate • Box Plots • 다변량 데이터에서의 극값분석 • Depth-Based Methods – Convex hull 분석 • Deviation-Based Methods • Angle-based • Extreme-value analysis is usually required as a final step on these modeled deviations
• 개요 • 확률모형에서는 “likelihood fit of a data point to a generative model is the outlier score”. • 예 • GMM • EM • 장단점 • 장점 • 다양한 경우에 적용 가능 (any data type or mixed data type), as long as an appropriate generative model is available for each mixture component. • 단점 • 분포를 특정하기 어려운 경우. • As the number of model parameters increases, over-fitting becomes more common.
• 일반형 • a convex non-linear programming – OLS • Model the data along lower-dimensional subspaces using linear correlations • Hyperplane과 데이터와의 거리 → outlier scores. • PCA • 행렬분해 • Spectral Models • Some variations of matrix decomposition (ex: PCA) used in certain types of data such as graphs and networks, are called spectral models. • They are used commonly for clustering graph data, and are often used in order to identify anomalous changes in temporal sequences of graphs.
• 개념 • Clustering method • Density-based methods • Nearest-neighbor methods
• 개념 • outliers increase the minimum code length (i.e., minimum length of the summary) required to describe a data set as they represent deviations from natural attempts to summarize data. • 예(1) • 예(2) multidimensional data sets • 확률모델: a data set in terms of generative model parameters, such as a mixture of Gaussian distributions or a mixture of exponential power distributions. • 군집화 / 밀도기반 요약 : describes a data set in terms of cluster descriptions, histograms, or other summarized representations, along with maximum error tolerances. • PCA / spectral 모델: describes the data in terms of lower dimensional subspaces of projection of multi-dimensional data or a latent representation of a network. • FP mining : describes the data in terms of an underlying code book of frequent patterns.
• High-dimension • Subspace outlier detection • Assumption: “outliers are often hidden in the unusual local behavior of low-dimensional subspaces, and this deviant behavior is masked by full-dimensional analysis”. • High-dimensional space에서 데이터는 sparse 및 almost equidistant. • → outlier scores become less distinguishable. • Outliers are best emphasized in a lower-dimensional local subspace of relevant attributes.
• Max Voting • 주로 classification 에 적용. • 다수 모델로 각각의 데이터를 예측 – 이를 ‘vote’로 처리. • 예: 영화에 대한 평점 • 기법 • Averaging과 Weighted Averaging • Stacking • Blending • Bagging 및 Boosting
• Outlier 분석 ensemble 의 2 종류 : • sequential ensembles • a given algorithm or set of algorithms are applied sequentially, so that future applications of the algorithms are influenced by previous applications, in terms of either modifications of the base data for analysis or in terms of the specific choices of the algorithms. • 최종 산출물: either a weighted combination of, or the final result of the last application. (예) 분류모델에서 boosting methods may be considered examples of sequential ensembles. • independent ensembles • different algorithms, or different instantiations of the same algorithm are applied to either the complete data or portions of the data. The choices made about the data and algorithms applied are independent of the results obtained from these different algorithmic executions. • 최종산출물: executions are combined together in order to obtain more robust outliers.
• 범주형 데이터, 텍스트 및 Mixed Attributes • categorical attributes that take on discrete unordered values. • Mixed attribute data contain both numerical and categorical attributes. • Regression-based models can be used in a limited way over discrete attribute values, • 대책 • convert the discrete data to binary data by creating one attribute for each categorical value. Such methods can be more easily extended to text • 모델 적용 • LSA (latent semantic analysis) • Clustering • proximity-based methods • probabilistic models • frequent pattern mining • 데이터 내에서의 Dependency 문제 • 시계열 데이터 • Discrete Sequence 데이터 • 그래프, 네트워크 형, …
• 모델링: f의 추정? • Prediction • Inference • Resampling과 Cross Validation • 지도학습 vs. 비지도학습
• Feature (특성) • A feature is a numeric representation of raw data. • Simple Numbers • Scalars, vectors, spaces • Counts • Binarization. Quantization or binning • Feature Scaling (Normalization) • Min-max scaling • Standardization (variance scaling) • Feature Selection Bucketing Crossing Hashing Embedding
• Log 변환 • 텍스트 데이터 • Flat Vectors • Bag-of-words, Bag-of-N-Grams • Filtering • Stopwords, Frequency-based filtering, Stemming • Semantic기법 • Parsing, tokenization, Phrase Detection, TF-IDF • 범주형 변수 - Encoding • One-hot encoding, Dummy coding • 차원축소와 행렬분해 • PCA, SVD • 모델 적용 • LSA (latent semantic analysis), Clustering, 확률모형 • Data Value에서의 Dependency 문제
• kNN • KNN graph (k-nearest neighbor graph)? • a graph in which 2 vertices p and q are connected by an edge, if the distance between p and q is among the k-th smallest distances from p to other objects from P. • has a vertex for each point, and a directed edge from p to q whenever q is a nearest neighbor of p, a point whose distance from p is minimum among all the given points other than p itself. • (변형 1) 1-NNG • Directions of the edges are ignored and NNG is defined instead as an undirected graph. However, the nearest neighbor relation is not a symmetric one. • (변형 2) FNG (farthest neighbor graph)
• Outlier Detection using In-degree Number (ODIN) • 각 data point의 in-degree를 계산 • in-degree = the number of nearest neighbour sets to this point belongs. • In-degree값이 크면 ; more confidence of this point belonging to some dense region in the space. • In-degree값이 작으면 ; it’s not part of many nearest neighbour sets • 즉, is kind of isolated in the space. • the reverse of KNN.
• SVM • 개념 • Linear SVM vs. Non-Linear SVM • One-Class Classification • 1. Outlier Detection • 2. AD in Acoustic Signals • 3. Novelty Detection and many others. • One-class SVM (1) • to ensure the widest street • maximize 2/|w| == to minimizing 1/2*(|w|^2). • + Lagrange multiplier → • w is a vector of random weights. • alpha = Lagrange multiplier, • y = either +1 or -1 i.e., class of the sample, • x = samples from data.
비지도학습 일반론 • 차원축소 • Linear Projection • PCA, SVD • Random projection • Manifold Learning • Isomap • T-SNE • Dictionary learning • ICA, Latent Dirichlet Allocation • 군집화 • K-Means • Hierarchical Clustering • DBSCAN • 혼합모형/EM • 딥러닝 기반 비지도학습 • Feature Extraction • Autoencoders • Unsupervised Pretraining • 생성모델과 네트워크 모델 • RBM • Deep Belief Networks • GAN • Sequential Data 적용 • Hidden Markov model • 강화학습과 비지도학습 • Semi-supervised Learning
• 비지도학습 • 목적: interesting pattern과 숨겨진 데이터 속성을 찾는 것 • = 자율학습(unsupervised learning) (vs. 지도학습(supervised)) • Can we visualize data? • Can we find meaningful subgroups of observations or variables? • Challenges • EDA - goal is not as clearly defined • 객관적 성능측정이 쉽지 않다 - don’t know the “right answer” • High-dimensional data • 대표적 적용 예 • As a stand-alone tool to get insight into data distribution • As a preprocessing step for other algorithms
• 군집모형 • 군집 (cluster) = a subset of data which are similar.
• K-Means • K-Means 기반 이상탐지 • we can define outliers by ourselves. • define what is a ‘far’ distance • define how many data points should be outliers. • outlier/anomaly • a data point far from the centroid of its cluster
• LOF (Local Outlier Factors) • 개요 • identify an outlier considering the density of the neighborhood. • 특히 데이터의 밀도 (density of the data)가 일정치 않을 때 효과가 큼 • = ratio of the average LRD of K neighbors of A to the LRD of A. • LRD of each point is used to compare with the average LRD of its K neighbors. • If the point is not an outlier (inlier), the ratio of average LRD of neighbors is approximately equal to the LRD of a point. In that case, LOF is nearly equal to 1. • If the point is an outlier, LRD of a point < average LRD of neighbors. Then LOF value will be high. • If LOF> 1, is considered as an outlier, but not always true. • 관련 개념 • Reachability distance (RD) • Local reachability density (LRD) • Local Outlier Factor (LOF)
• LOF ≈ 1 similar density as neighbors • LOF < 1 higher density than neighbors (normal point) • LOF > 1 lower density than neighbors (anomaly)
• K-distance와 K-Neighbors • K-distance • = distance between the point, and it’s Kth nearest neighbor. • K-neighbors, Nₖ(A), includes a set of points that lie in or on the circle of radius K-distance. • Reachability Distance (RD) • = maximum of K-distance of Xj and the distance between Xi and Xj. • Local RD (LRD) K-distance of A with K=2
• Mixture Model • model the data in terms of a mixture of several components, where each component has a simple parametric form (예: Gaussian). • assuming class mixture component is known and estimating class membership given parameters. • Mixtures of {Sequences, Curves, …} • 생성모형 • select a component ck for individual i • generate data according to p(Di | ck) • p(Di | ck) can be very general • GMM (Gaussian Mixture Model) • Multivariate Gaussian models
• EM (Expectation-Maximization) • Latent variable model • Algorithm • Expectation • Maximization
• Neuron과 Artificial Nodes • 개별 신경망의 특징을 결정하는 요소: • 활성함수 • step, sigmoid, tanh, relu • Network topology (or architecture) • 모델이 가진 뉴론의 수 + 연결된 layer의 수 • Training 알고리즘 • Gradient descent, Newton method, Conjugate gradient, … • 학습 – BP through Gradient Descent • Computation Graph
• 기본형 • 감독형 딥러닝 • CNN, RNN • 무감독형 딥러닝 • Autoencoder, RBM • 강화학습 • Q-Learning, Policy-Gradient • 응용 • CNN • RNN
• 딥러닝 모델 https://link.springer.com/article/10.1007/s00530-020-00694-1
• Python기반 딥러닝 프레임워크 • TensorFlow와 Keras • TensorFlow • Keras 이용 • R interface • PyTorch • 기타 주요 라이브러리
• 개념 • Anomaly Detection (AD) OR novelty detection • Normality Representation • ☞ 기술통계 • Measures of Frequency • Measures of Central Tendency • Measures of Dispersion • Anomaly representation • Outlier detection 알고리즘에서의 2가지 출력 양식 • Outlier scores: quantify level of “outlier-ness”  outlier tendency. • Binary labels: “Whether a data point is an outlier or not” • 주요 이슈 • Subjectivity, Interestingness and noise • the data may be embedded in a significant amount of noise
• RNN • 개념 • RNN은 지금 들어온 입력데이터와 과거에 입력 받았던 데이터를 동시에 고려 • 장단점 • (장점) see how previous layer is stimulated → NN interprets sequences much better. • (단점) more parameters to be calculated A recurrent neuron (왼쪽) unrolled through time (오른쪽)
• Long short-term memory models • 목적: 기존의 RNN의 문제점 해결: • Vanishing gradients와 Exploding gradients • inability to remember or forget certain aspects of input sequences • 특징: previous time step으로부터 previous output뿐 아니라 state 정보도 함께 전달받음. • 동작원리 • Output control: How much an output neuron is stimulated by the previous output and current state • Memory control: How much of previous state will be forgotten • Input control: How much of the previous output and new state (memory) will be considered to determine the new current state • These are trainable and optimized
• 정의 • shallow, 2-layer NNs constituting DBN (deep-belief networks) • Restriction = no intra-layer communication.
• Reconstruction • activations of hidden layer no.1 become input in a backward pass. • Forward pass – RBM to predict node activations: p(a | x; w). • Backward pass - RBM attempts to estimate p(x | a; w). • Reconstruction을 통해 입력데이터의 PDF를 추측 (= generative learning) • 추정된 PDF와 실제 PDF의 거리계산 - Kullback Leibler Divergence. • Kullback-Leibler (KL) divergence measures divergence of two probability distributions, p and q. 이를 통해 p(x, a)
• Autoencoder • PCA와 유사하지만 보다 flexible. • Target output is its input (x) in a different form (x’). • dimensionality of input = dimensionality of the output • essentially what we want is x’ = x. x’ = Decode (Encode(x)) • Autoencoder를 이용한 이상탐지 • If a point in feature space lies far away from the majority of the points (meaning it holds different properties), the autoencoder learns the distribution - an anomaly. • 즉, model more or less correctly re-generates the images leading to low loss values. • We use these reconstruction loss values as the anomaly scores • The higher the scores, the higher the chances of input being an anomaly.
• LSTM-based Encoder-Decoder for Anomaly Detection • 정상데이터 (MV TSA)를 Unsupervised 방법으로 학습하고 이상치를 탐지하는 모델 • 특징 • LSTM-Encoder와 LSTM-Decoder로 구성 • Encoder는 다변량 데이터를 압축하여 feature로 변환. • Decoder는 Encoder에서 받은 feature를 이용 Encoder에서 받은 다변량 데이터를 재구성 • Reconstruction Error 계산 • MSE Loss를 이용하여 학습 But 추론 과정에서 Error 계산 방법은 Absolute Error를 활용.
• Self-Organizing Maps • 자기조직화 지도 • 특징 • Competitive learning by BP • 일종의 DR 기법 • 동작원리 • Components • Initialization • Competition • Cooperation • Adaptation https://arxiv.org/pdf/1312.5753.pdf source; wikipedia
• 사이버보안 이상징후 판단 • Malware 분석 • Network traffic 분석 • 센서 네트워크
• 모터 결함 진단 (Fault Diagnosis) • = 고장진단 (현 상태가 고장인지 여부) + 고장 예측 • 예: 유도 전동기 (Induction motor) • DBN (Deep Belief Network) • A generative graphical model • Stacking RBMs • 진동 (주파수) 측정 데이터를 이용하여 학습 • → Contrastive divergence using • Gibbs sampling
• Chemistry/Materials Science • Medical Outlier
• Anomaly detection problem Complexities • Unknownness • Anomalies are associated with many unknowns, e.g., instances with unknown abrupt behaviors, data structures, and distributions. They remain unknown until actually occur. • Heterogeneous anomaly classes. • Anomalies are irregular, and thus, one class of anomalies may demonstrate completely different abnormal characteristics from another class of anomalies. • Rarity and class imbalance • → unavailability of large-scale labeled data in most applications = class imbalance • Diverse types of anomaly. • 3 different types of anomaly have been explored. • Point anomalies • Conditional anomalies • Group anomalies • 3 different types of anomaly have been explored. • Point anomalies • Conditional anomalies = contextual anomalies • Group anomalies = collective anomalies
• Deep Anomaly Detection가 해결 시도하는 문제 • CH1: Low anomaly detection recall rate. • CH2: Anomaly detection in high-dimensional and/or not-independent data. • CH3: Data-efficient learning of normality/abnormality. • CH4: Noise-resilient anomaly detection. • CH5: Detection of complex anomalies. • CH6: Anomaly explanation.
• Deep Anomaly Detection 접근법의 3가지 Frameworks
Outlier Analysis.pdf

More Related Content

PDF
Relational knowledge distillation
byNAVER Engineering
 
PPTX
라이브 서비스를 위한 게임 서버 구성
byHyunjik Bae
 
PDF
개발을잘하고싶어요-네이버랩스 송기선님
byNAVER D2
 
PDF
두 번째 startlink.live: 오현석 (algoshipda) - 초심자를 위한 알고리즘 공부 전략
byStartlink
 
PDF
Document Summarization
byPratik Kumar
 
PPTX
딥러닝 - 역사와 이론적 기초
byHyungsoo Ryoo
 
PPTX
12 questions pour bien lancer son programme OKR
byAgile En Seine
 
PDF
Multi Task Learning for Recommendation Systems
byVaibhav Singh
 
Relational knowledge distillation
byNAVER Engineering
 
라이브 서비스를 위한 게임 서버 구성
byHyunjik Bae
 
개발을잘하고싶어요-네이버랩스 송기선님
byNAVER D2
 
두 번째 startlink.live: 오현석 (algoshipda) - 초심자를 위한 알고리즘 공부 전략
byStartlink
 
Document Summarization
byPratik Kumar
 
딥러닝 - 역사와 이론적 기초
byHyungsoo Ryoo
 
12 questions pour bien lancer son programme OKR
byAgile En Seine
 
Multi Task Learning for Recommendation Systems
byVaibhav Singh
 

What's hot

PDF
[261] 실시간 추천엔진 머신한대에 구겨넣기
byNAVER D2
 
PDF
RoFormer: Enhanced Transformer with Rotary Position Embedding
bytaeseon ryu
 
PPTX
프라우드넷의 연결 유지 기능과 홀펀칭-윤현민
byHyunjik Bae
 
PPTX
Reinforcement Learning
byDongHyun Kwak
 
PPTX
Intro to Deep Reinforcement Learning
byKhaled Saleh
 
PDF
게임서버프로그래밍 #2 - IOCP Adv
bySeungmo Koo
 
PDF
A brief introduction to recent segmentation methods
byShunta Saito
 
PPTX
Shap
byGiovanni Bruner
 
PDF
인공지능개론 (머신러닝 중심)
bySK(주) C&C - 강병호
 
PPTX
게임 분산 서버 구조
byHyunjik Bae
 
PDF
ddpg seminar
by민재 정
 
PPTX
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
bySri Ambati
 
PDF
How To Become Better Engineer
byDaeMyung Kang
 
PDF
알파고 해부하기 1부
byDonghun Lee
 
PDF
Introduction to natural language processing
byMinh Pham
 
PPTX
Ndc14 분산 서버 구축의 ABC
byHo Gyu Lee
 
PDF
Python과 Git으로 만드는 모바일 게임 패치 시스템
byYoungtaek Oh
 
PPT
Линейная регрессия
byDEVTYPE
 
PDF
Detailed Introduction To LLAMA by Meta 2024
byRituPatel551417
 
PPTX
딥 러닝 자연어 처리 학습을 위한 PPT! (Deep Learning for Natural Language Processing)
byWON JOON YOO
 
[261] 실시간 추천엔진 머신한대에 구겨넣기
byNAVER D2
 
RoFormer: Enhanced Transformer with Rotary Position Embedding
bytaeseon ryu
 
프라우드넷의 연결 유지 기능과 홀펀칭-윤현민
byHyunjik Bae
 
Reinforcement Learning
byDongHyun Kwak
 
Intro to Deep Reinforcement Learning
byKhaled Saleh
 
게임서버프로그래밍 #2 - IOCP Adv
bySeungmo Koo
 
A brief introduction to recent segmentation methods
byShunta Saito
 
Shap
byGiovanni Bruner
 
인공지능개론 (머신러닝 중심)
bySK(주) C&C - 강병호
 
게임 분산 서버 구조
byHyunjik Bae
 
ddpg seminar
by민재 정
 
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
bySri Ambati
 
How To Become Better Engineer
byDaeMyung Kang
 
알파고 해부하기 1부
byDonghun Lee
 
Introduction to natural language processing
byMinh Pham
 
Ndc14 분산 서버 구축의 ABC
byHo Gyu Lee
 
Python과 Git으로 만드는 모바일 게임 패치 시스템
byYoungtaek Oh
 
Линейная регрессия
byDEVTYPE
 
Detailed Introduction To LLAMA by Meta 2024
byRituPatel551417
 
딥 러닝 자연어 처리 학습을 위한 PPT! (Deep Learning for Natural Language Processing)
byWON JOON YOO
 

Similar to Outlier Analysis.pdf

PDF
Decision tree and ensemble
byDanbi Cho
 
PDF
AI 바이오 (2_3일차).pdf
byH K Yoon
 
PPT
Machine learning........................
byssuserc13e6a
 
PPTX
Seminar9
bykim taegong
 
PDF
ML 60'~80' new paradigm 1
by민석 김
 
PPTX
Machine learning
bysujinkim136
 
PDF
Decision Tree Intro [의사결정나무]
byHyunwoo Kim
 
PDF
Decision tree
bySEMINARGROOT
 
PPTX
Deep Learning for AI (2)
byDongheon Lee
 
PPTX
R v01 rprogamming_basic01 (R 프로그래밍 기본)
byBuskersBu
 
PDF
Support vector machines
byJinho Lee
 
PDF
1시간만에 머신러닝 개념 따라 잡기
bySungmin Kim
 
PDF
Dm ml study_roadmap
byKang Pilsung
 
PPTX
Elementary statistical inference1
bySEMINARGROOT
 
PDF
Incheon National University - EATED SRA
byssuser58d6dc2
 
PPTX
Murpy's Machine Learning:14. Kernel
byJungkyu Lee
 
PPTX
Kernel Method
by수철 박
 
PPTX
PPT - Deep and Confident Prediction For Time Series at Uber
byJisang Yoon
 
PDF
제 14회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [BICS팀] : Boaz Industry Classification Standard
byBOAZ Bigdata
 
PPTX
Anomaly detection
by철 김
 
Decision tree and ensemble
byDanbi Cho
 
AI 바이오 (2_3일차).pdf
byH K Yoon
 
Machine learning........................
byssuserc13e6a
 
Seminar9
bykim taegong
 
ML 60'~80' new paradigm 1
by민석 김
 
Machine learning
bysujinkim136
 
Decision Tree Intro [의사결정나무]
byHyunwoo Kim
 
Decision tree
bySEMINARGROOT
 
Deep Learning for AI (2)
byDongheon Lee
 
R v01 rprogamming_basic01 (R 프로그래밍 기본)
byBuskersBu
 
Support vector machines
byJinho Lee
 
1시간만에 머신러닝 개념 따라 잡기
bySungmin Kim
 
Dm ml study_roadmap
byKang Pilsung
 
Elementary statistical inference1
bySEMINARGROOT
 
Incheon National University - EATED SRA
byssuser58d6dc2
 
Murpy's Machine Learning:14. Kernel
byJungkyu Lee
 
Kernel Method
by수철 박
 
PPT - Deep and Confident Prediction For Time Series at Uber
byJisang Yoon
 
제 14회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [BICS팀] : Boaz Industry Classification Standard
byBOAZ Bigdata
 
Anomaly detection
by철 김
 

More from H K Yoon

PDF
AI 바이오 (4일차).pdf
byH K Yoon
 
PDF
Nlp and transformer (v3s)
byH K Yoon
 
PDF
Open stack and k8s(v4)
byH K Yoon
 
PDF
Open source Embedded systems
byH K Yoon
 
PDF
빅데이터, big data
byH K Yoon
 
PPT
Sensor web
byH K Yoon
 
PPT
Tm기반검색v2
byH K Yoon
 
AI 바이오 (4일차).pdf
byH K Yoon
 
Nlp and transformer (v3s)
byH K Yoon
 
Open stack and k8s(v4)
byH K Yoon
 
Open source Embedded systems
byH K Yoon
 
빅데이터, big data
byH K Yoon
 
Sensor web
byH K Yoon
 
Tm기반검색v2
byH K Yoon
 

Recently uploaded

PDF
Benefits of Using the IAC 500 Sensor for Ambient Conditions
byCS Instruments
 
PPTX
Compare and contrast types of attacks.pptx
bysyednaqihassan14
 
PDF
A Brief Introduction About Christopher Elwell Woburn
byChristopher Elwell Woburn
 
PDF
The map to conquer linear algebra for IT engineer
byakipii ogaoga
 
PDF
splunk-peak-threat-hunting-framework.pdf
bylobster6719
 
PDF
MAD (1).pdf Mobile application develipment
byprathiT1
 
PDF
Top 10 API Automation Testing Tools: Features, Pros & Cons
byhenrycavill30521
 
PPT
Information and Communication Technologies and Power
byJeffrey Hart
 
PDF
IAC 500 Sensor - Humidity Measurement Device
byCS Instruments
 
PDF
What is Voice User Interface (VUI) Definition & Examples.pdf
byDesign Studio UI UX
 
PPTX
Why 2026 Could Be a Turning Point for Decentralized Exchanges.pptx
bycalliemorgan193
 
PDF
“Lessons from Yesterday's Tomorrowland” by Scott M. Graffius
byScott M. Graffius
 
PPTX
Pizza Chain Market Data Scraping for Better Insights Report.pptx
byWeb Data Crawler
 
PPTX
AI Bot Traffic Surge: Retail Fraud Threat for Age-Restricted Websites
byFTx Identity
 
PPTX
Salesforce Spring 26 Release Key .pptx
byMehediHasan939830
 
PDF
Ai In Courts Ai in courts AI in court AI in court
byZainKarim7
 
PPT
HDTV and DTV Standards: The United States Opts for a Digital HDTV Standard
byJeffrey Hart
 
PDF
Beyond BBB: Practical Alternatives to Posterior Approximation in Bayesian Neu...
byTomasz Kusmierczyk
 
PPTX
Scrape YouTube Video Data for Influencer Analytics.pptx
byWeb Data Crawler
 
PDF
BO049FHDMO 0.49 Inch Micro OLED Datasheet
bySyluxdisplay
 
Benefits of Using the IAC 500 Sensor for Ambient Conditions
byCS Instruments
 
Compare and contrast types of attacks.pptx
bysyednaqihassan14
 
A Brief Introduction About Christopher Elwell Woburn
byChristopher Elwell Woburn
 
The map to conquer linear algebra for IT engineer
byakipii ogaoga
 
splunk-peak-threat-hunting-framework.pdf
bylobster6719
 
MAD (1).pdf Mobile application develipment
byprathiT1
 
Top 10 API Automation Testing Tools: Features, Pros & Cons
byhenrycavill30521
 
Information and Communication Technologies and Power
byJeffrey Hart
 
IAC 500 Sensor - Humidity Measurement Device
byCS Instruments
 
What is Voice User Interface (VUI) Definition & Examples.pdf
byDesign Studio UI UX
 
Why 2026 Could Be a Turning Point for Decentralized Exchanges.pptx
bycalliemorgan193
 
“Lessons from Yesterday's Tomorrowland” by Scott M. Graffius
byScott M. Graffius
 
Pizza Chain Market Data Scraping for Better Insights Report.pptx
byWeb Data Crawler
 
AI Bot Traffic Surge: Retail Fraud Threat for Age-Restricted Websites
byFTx Identity
 
Salesforce Spring 26 Release Key .pptx
byMehediHasan939830
 
Ai In Courts Ai in courts AI in court AI in court
byZainKarim7
 
HDTV and DTV Standards: The United States Opts for a Digital HDTV Standard
byJeffrey Hart
 
Beyond BBB: Practical Alternatives to Posterior Approximation in Bayesian Neu...
byTomasz Kusmierczyk
 
Scrape YouTube Video Data for Influencer Analytics.pptx
byWeb Data Crawler
 
BO049FHDMO 0.49 Inch Micro OLED Datasheet
bySyluxdisplay
 

Outlier Analysis.pdf

  • 2.
  • 5.
    • (1) 중심경향성:Ungrouped Data • Mode, Mean, Median • Percentile, Quantile/Quartile • (2) 변동성: Ungrouped Data • Range & IQR (Interquartile Range) • MAD (Mean Absolute Deviation), Variance, Standard Deviation • 분산 및 표준편차 • Unbiased estimator • Z-score • (3) Measures of Shape • Moment • Skewness와 Kurtosis • (4) 연관성 (Association) 측도 • Correlation
  • 6.
    • 기본개념 • Experiment,(Elementary) , Event와 Sample Space , … • 조건부 확률과 Bayes’ rule • 조건부 확률 법칙 P(X | Y) = (P(X ∩ Y))/(P(Y)) = (P(X)•((Y|X))/(P(Y)) • 독립성 여부의 검정: P(X | Y) = P(X) and P(Y| X) = P(Y) • Bayes’ Rule • P(Xi | Y) = 𝑃 𝑋𝑖 •𝑃(𝑌|𝑋𝑖) 𝑃 𝑋1 •𝑃 𝑌 𝑋1 + 𝑃 𝑋2 •𝑃 𝑌 𝑋2 +⋯+𝑃 𝑋𝑛 •𝑃(𝑌|𝑋𝑛) • Odds • 확률변수와 확률분포 • 확률변수 • = a variable that contains the outcomes of a chance experiment • 확률분포 • 이산분포: 이항분포, Poisson분포 • 연속분포: 균일분포, 정규분포, t-분포, 지수분포, 𝜒2 분포
  • 7.
    • 추정 • 정규분포 •t-분포 • 가설검정 • p-Value를 이용한 가설검정 • p-value = 관측된 유의수준 (level of significance) • defines the smallest value of 𝛼 for which the H0 can be rejected. • “α 가 p보다 커야만 H0를 reject 가능”
  • 9.
    • Variate(s) • Univariate •Bi-variate • 공분산향렬과 SSCP • Multivariate • Multivariate • 다변량 확률분포 • 다변량정규분포 • 다변량 분석기법 • PCA • Factor Analysis • MDS • ANOVA/ANCOVA/MANOVA/MANCOVA/… • 다변량 다중회귀분석 (MVMLR)
  • 12.
    • 주요 이슈 •Subjectivity, Interestingness and noise • Subjective judgement, as to what constitutes a “sufficient” deviation • In real applications, the data may be embedded in a significant amount of noise • Noise 문제 • Representation의 문제 • Normality vs. Anomaly • 특성공학 • 이상치 분석과 데이터 모델
  • 13.
    • 알고리즘 분류(1) • Outlier scores: quantify level of “outlierness” • Binary labels: “outlier? or not?” • the threshold is based on the statistical distribution of the scores. • 알고리즘 분류 (2)
  • 15.
    • qualitative techniques, •전문가 의견 • time series analysis • 관심사항: patterns and pattern changes • 과거 데이터가 중요한 역할 • causal models. • 인과관계 • 과거 데이터가 중요한 역할 • Regression • 독립변수와 종속변수
  • 16.
    단변량 Time Series모델 • AR Model • 자기상관 (Autocorrelation) • 정상성 (Stationarity)과 ADF Test • Differencing a Time Series • Autocorrelation에서의 Lags • Partial Autocorrelation • AR 모델 정의 • Yule-Walker Equation 이용한 AR 추정 • MA Model • MA 모델 정의 • MA Model의 Fitting • Stationarity • AR vs. MA 모델 선택 • Model Retraining을 통한 다단계 예측 • 최적의 MA Order를 찾기 위한 Grid Search
  • 17.
    • ARMA 모델 •모델 정의 • ARMA(1,1) Model의 Fitting • Automated Hyperparameter Tuning • Grid Search • 성능향상을 위한 Tuning • ARIMA 모델 • 모델 정의 • SARIMA 모델 • 모델 정의 • regular AR part φp • seasonal AR part • regular MA part θq • seasonal MA part • regular integration part; order d • seasonal integration part; order D • Coefficient of seasonality s
  • 18.
    • Multivariate TimeSeries Models • SARIMAX 모델 • SARIMA 모델에 외생변수 (X)를 추가 • VAR 모델 • Since VAR model proposes one model for multiple target variables, it regroups those variables as a vector • VAR 계수의 추정 • VARMAX 모델 • VAR model에 MV 항을 추가하고 외생변수 허용 • V for vector indicating that it’s a multivariate model • AR for autoregression • MA for moving average • X for the use of exogenous variables (in addition to the endogenous variables)
  • 19.
    • Linear Regression •kNN, 의사결정 트리/Random Forest • XGBoost와 LightGBM
  • 20.
    • RNN/LSTM • Predictinga Sequence Rather Than a Value • SimpleRNN • GRU, LSTM • Prophet 모델 (Facebook) • an automated procedure for building forecasting models developed by Facebook. • Input possibilities are • Seasonality of any regular order • Holidays • Additional regressors • hyperparameters • Fourier order of the seasonality: A higher order means more flexibility. • changepoint_prior_scale plays on the trend: The higher the value, the more flexible the trend. • holidays_prior_scale: The lower it is, the less important the holidays are for the model. • prior scale for the seasonality • DeepAR 모델
  • 22.
    • Euclidean 거리 •Manhattan 거리 • Minkowski 거리 • Cosine 거리
  • 23.
    • Distance Distribution-basedTechniques • to model entire data set to be normally distributed about its mean in the form of a multivariate Gaussian distribution. • Let ҧ 𝜇 be d-dimensional (row) data set, and Σ be its d x d covariance matrix. • Then, the probability distribution 𝑓( ത 𝑋) for a d-dimensional (row vector) data point X is: • |Σ| = determinant of covariance matrix. • 지수부: (half) squared Mahalanobis distance of the data point X to the centroid μ of the data. = outlier score
  • 25.
    • Extreme-Value Analysis •극값을 판별해 내는 것 • = Probabilistic Tail Inequalities • Markov Inequality • Chebychev Inequality • … • determine statistical tails of the underlying distribution. • Univariate • Box Plots • 다변량 데이터에서의 극값분석 • Depth-Based Methods – Convex hull 분석 • Deviation-Based Methods • Angle-based • Extreme-value analysis is usually required as a final step on these modeled deviations
  • 26.
    • 개요 • 확률모형에서는“likelihood fit of a data point to a generative model is the outlier score”. • 예 • GMM • EM • 장단점 • 장점 • 다양한 경우에 적용 가능 (any data type or mixed data type), as long as an appropriate generative model is available for each mixture component. • 단점 • 분포를 특정하기 어려운 경우. • As the number of model parameters increases, over-fitting becomes more common.
  • 27.
    • 일반형 • aconvex non-linear programming – OLS • Model the data along lower-dimensional subspaces using linear correlations • Hyperplane과 데이터와의 거리 → outlier scores. • PCA • 행렬분해 • Spectral Models • Some variations of matrix decomposition (ex: PCA) used in certain types of data such as graphs and networks, are called spectral models. • They are used commonly for clustering graph data, and are often used in order to identify anomalous changes in temporal sequences of graphs.
  • 28.
    • 개념 • Clusteringmethod • Density-based methods • Nearest-neighbor methods
  • 29.
    • 개념 • outliersincrease the minimum code length (i.e., minimum length of the summary) required to describe a data set as they represent deviations from natural attempts to summarize data. • 예(1) • 예(2) multidimensional data sets • 확률모델: a data set in terms of generative model parameters, such as a mixture of Gaussian distributions or a mixture of exponential power distributions. • 군집화 / 밀도기반 요약 : describes a data set in terms of cluster descriptions, histograms, or other summarized representations, along with maximum error tolerances. • PCA / spectral 모델: describes the data in terms of lower dimensional subspaces of projection of multi-dimensional data or a latent representation of a network. • FP mining : describes the data in terms of an underlying code book of frequent patterns.
  • 30.
    • High-dimension • Subspaceoutlier detection • Assumption: “outliers are often hidden in the unusual local behavior of low-dimensional subspaces, and this deviant behavior is masked by full-dimensional analysis”. • High-dimensional space에서 데이터는 sparse 및 almost equidistant. • → outlier scores become less distinguishable. • Outliers are best emphasized in a lower-dimensional local subspace of relevant attributes.
  • 32.
    • Max Voting •주로 classification 에 적용. • 다수 모델로 각각의 데이터를 예측 – 이를 ‘vote’로 처리. • 예: 영화에 대한 평점 • 기법 • Averaging과 Weighted Averaging • Stacking • Blending • Bagging 및 Boosting
  • 33.
    • Outlier 분석ensemble 의 2 종류 : • sequential ensembles • a given algorithm or set of algorithms are applied sequentially, so that future applications of the algorithms are influenced by previous applications, in terms of either modifications of the base data for analysis or in terms of the specific choices of the algorithms. • 최종 산출물: either a weighted combination of, or the final result of the last application. (예) 분류모델에서 boosting methods may be considered examples of sequential ensembles. • independent ensembles • different algorithms, or different instantiations of the same algorithm are applied to either the complete data or portions of the data. The choices made about the data and algorithms applied are independent of the results obtained from these different algorithmic executions. • 최종산출물: executions are combined together in order to obtain more robust outliers.
  • 35.
    • 범주형 데이터,텍스트 및 Mixed Attributes • categorical attributes that take on discrete unordered values. • Mixed attribute data contain both numerical and categorical attributes. • Regression-based models can be used in a limited way over discrete attribute values, • 대책 • convert the discrete data to binary data by creating one attribute for each categorical value. Such methods can be more easily extended to text • 모델 적용 • LSA (latent semantic analysis) • Clustering • proximity-based methods • probabilistic models • frequent pattern mining • 데이터 내에서의 Dependency 문제 • 시계열 데이터 • Discrete Sequence 데이터 • 그래프, 네트워크 형, …
  • 38.
    • 모델링: f의추정? • Prediction • Inference • Resampling과 Cross Validation • 지도학습 vs. 비지도학습
  • 40.
    • Feature (특성) •A feature is a numeric representation of raw data. • Simple Numbers • Scalars, vectors, spaces • Counts • Binarization. Quantization or binning • Feature Scaling (Normalization) • Min-max scaling • Standardization (variance scaling) • Feature Selection Bucketing Crossing Hashing Embedding
  • 41.
    • Log 변환 •텍스트 데이터 • Flat Vectors • Bag-of-words, Bag-of-N-Grams • Filtering • Stopwords, Frequency-based filtering, Stemming • Semantic기법 • Parsing, tokenization, Phrase Detection, TF-IDF • 범주형 변수 - Encoding • One-hot encoding, Dummy coding • 차원축소와 행렬분해 • PCA, SVD • 모델 적용 • LSA (latent semantic analysis), Clustering, 확률모형 • Data Value에서의 Dependency 문제
  • 43.
    • kNN • KNNgraph (k-nearest neighbor graph)? • a graph in which 2 vertices p and q are connected by an edge, if the distance between p and q is among the k-th smallest distances from p to other objects from P. • has a vertex for each point, and a directed edge from p to q whenever q is a nearest neighbor of p, a point whose distance from p is minimum among all the given points other than p itself. • (변형 1) 1-NNG • Directions of the edges are ignored and NNG is defined instead as an undirected graph. However, the nearest neighbor relation is not a symmetric one. • (변형 2) FNG (farthest neighbor graph)
  • 44.
    • Outlier Detectionusing In-degree Number (ODIN) • 각 data point의 in-degree를 계산 • in-degree = the number of nearest neighbour sets to this point belongs. • In-degree값이 크면 ; more confidence of this point belonging to some dense region in the space. • In-degree값이 작으면 ; it’s not part of many nearest neighbour sets • 즉, is kind of isolated in the space. • the reverse of KNN.
  • 45.
    • SVM • 개념 •Linear SVM vs. Non-Linear SVM • One-Class Classification • 1. Outlier Detection • 2. AD in Acoustic Signals • 3. Novelty Detection and many others. • One-class SVM (1) • to ensure the widest street • maximize 2/|w| == to minimizing 1/2*(|w|^2). • + Lagrange multiplier → • w is a vector of random weights. • alpha = Lagrange multiplier, • y = either +1 or -1 i.e., class of the sample, • x = samples from data.
  • 47.
    비지도학습 일반론 • 차원축소 •Linear Projection • PCA, SVD • Random projection • Manifold Learning • Isomap • T-SNE • Dictionary learning • ICA, Latent Dirichlet Allocation • 군집화 • K-Means • Hierarchical Clustering • DBSCAN • 혼합모형/EM • 딥러닝 기반 비지도학습 • Feature Extraction • Autoencoders • Unsupervised Pretraining • 생성모델과 네트워크 모델 • RBM • Deep Belief Networks • GAN • Sequential Data 적용 • Hidden Markov model • 강화학습과 비지도학습 • Semi-supervised Learning
  • 48.
    • 비지도학습 • 목적:interesting pattern과 숨겨진 데이터 속성을 찾는 것 • = 자율학습(unsupervised learning) (vs. 지도학습(supervised)) • Can we visualize data? • Can we find meaningful subgroups of observations or variables? • Challenges • EDA - goal is not as clearly defined • 객관적 성능측정이 쉽지 않다 - don’t know the “right answer” • High-dimensional data • 대표적 적용 예 • As a stand-alone tool to get insight into data distribution • As a preprocessing step for other algorithms
  • 50.
    • 군집모형 • 군집(cluster) = a subset of data which are similar.
  • 51.
    • K-Means • K-Means기반 이상탐지 • we can define outliers by ourselves. • define what is a ‘far’ distance • define how many data points should be outliers. • outlier/anomaly • a data point far from the centroid of its cluster
  • 52.
    • LOF (LocalOutlier Factors) • 개요 • identify an outlier considering the density of the neighborhood. • 특히 데이터의 밀도 (density of the data)가 일정치 않을 때 효과가 큼 • = ratio of the average LRD of K neighbors of A to the LRD of A. • LRD of each point is used to compare with the average LRD of its K neighbors. • If the point is not an outlier (inlier), the ratio of average LRD of neighbors is approximately equal to the LRD of a point. In that case, LOF is nearly equal to 1. • If the point is an outlier, LRD of a point < average LRD of neighbors. Then LOF value will be high. • If LOF> 1, is considered as an outlier, but not always true. • 관련 개념 • Reachability distance (RD) • Local reachability density (LRD) • Local Outlier Factor (LOF)
  • 53.
    • LOF ≈1 similar density as neighbors • LOF < 1 higher density than neighbors (normal point) • LOF > 1 lower density than neighbors (anomaly)
  • 54.
    • K-distance와 K-Neighbors •K-distance • = distance between the point, and it’s Kth nearest neighbor. • K-neighbors, Nₖ(A), includes a set of points that lie in or on the circle of radius K-distance. • Reachability Distance (RD) • = maximum of K-distance of Xj and the distance between Xi and Xj. • Local RD (LRD) K-distance of A with K=2
  • 56.
    • Mixture Model •model the data in terms of a mixture of several components, where each component has a simple parametric form (예: Gaussian). • assuming class mixture component is known and estimating class membership given parameters. • Mixtures of {Sequences, Curves, …} • 생성모형 • select a component ck for individual i • generate data according to p(Di | ck) • p(Di | ck) can be very general • GMM (Gaussian Mixture Model) • Multivariate Gaussian models
  • 57.
    • EM (Expectation-Maximization) •Latent variable model • Algorithm • Expectation • Maximization
  • 60.
    • Neuron과 ArtificialNodes • 개별 신경망의 특징을 결정하는 요소: • 활성함수 • step, sigmoid, tanh, relu • Network topology (or architecture) • 모델이 가진 뉴론의 수 + 연결된 layer의 수 • Training 알고리즘 • Gradient descent, Newton method, Conjugate gradient, … • 학습 – BP through Gradient Descent • Computation Graph
  • 61.
    • 기본형 • 감독형딥러닝 • CNN, RNN • 무감독형 딥러닝 • Autoencoder, RBM • 강화학습 • Q-Learning, Policy-Gradient • 응용 • CNN • RNN
  • 62.
  • 64.
    • Python기반 딥러닝프레임워크 • TensorFlow와 Keras • TensorFlow • Keras 이용 • R interface • PyTorch • 기타 주요 라이브러리
  • 65.
    • 개념 • AnomalyDetection (AD) OR novelty detection • Normality Representation • ☞ 기술통계 • Measures of Frequency • Measures of Central Tendency • Measures of Dispersion • Anomaly representation • Outlier detection 알고리즘에서의 2가지 출력 양식 • Outlier scores: quantify level of “outlier-ness”  outlier tendency. • Binary labels: “Whether a data point is an outlier or not” • 주요 이슈 • Subjectivity, Interestingness and noise • the data may be embedded in a significant amount of noise
  • 67.
    • RNN • 개념 •RNN은 지금 들어온 입력데이터와 과거에 입력 받았던 데이터를 동시에 고려 • 장단점 • (장점) see how previous layer is stimulated → NN interprets sequences much better. • (단점) more parameters to be calculated A recurrent neuron (왼쪽) unrolled through time (오른쪽)
  • 68.
    • Long short-termmemory models • 목적: 기존의 RNN의 문제점 해결: • Vanishing gradients와 Exploding gradients • inability to remember or forget certain aspects of input sequences • 특징: previous time step으로부터 previous output뿐 아니라 state 정보도 함께 전달받음. • 동작원리 • Output control: How much an output neuron is stimulated by the previous output and current state • Memory control: How much of previous state will be forgotten • Input control: How much of the previous output and new state (memory) will be considered to determine the new current state • These are trainable and optimized
  • 70.
    • 정의 • shallow,2-layer NNs constituting DBN (deep-belief networks) • Restriction = no intra-layer communication.
  • 71.
    • Reconstruction • activationsof hidden layer no.1 become input in a backward pass. • Forward pass – RBM to predict node activations: p(a | x; w). • Backward pass - RBM attempts to estimate p(x | a; w). • Reconstruction을 통해 입력데이터의 PDF를 추측 (= generative learning) • 추정된 PDF와 실제 PDF의 거리계산 - Kullback Leibler Divergence. • Kullback-Leibler (KL) divergence measures divergence of two probability distributions, p and q. 이를 통해 p(x, a)
  • 72.
    • Autoencoder • PCA와유사하지만 보다 flexible. • Target output is its input (x) in a different form (x’). • dimensionality of input = dimensionality of the output • essentially what we want is x’ = x. x’ = Decode (Encode(x)) • Autoencoder를 이용한 이상탐지 • If a point in feature space lies far away from the majority of the points (meaning it holds different properties), the autoencoder learns the distribution - an anomaly. • 즉, model more or less correctly re-generates the images leading to low loss values. • We use these reconstruction loss values as the anomaly scores • The higher the scores, the higher the chances of input being an anomaly.
  • 73.
    • LSTM-based Encoder-Decoderfor Anomaly Detection • 정상데이터 (MV TSA)를 Unsupervised 방법으로 학습하고 이상치를 탐지하는 모델 • 특징 • LSTM-Encoder와 LSTM-Decoder로 구성 • Encoder는 다변량 데이터를 압축하여 feature로 변환. • Decoder는 Encoder에서 받은 feature를 이용 Encoder에서 받은 다변량 데이터를 재구성 • Reconstruction Error 계산 • MSE Loss를 이용하여 학습 But 추론 과정에서 Error 계산 방법은 Absolute Error를 활용.
  • 74.
    • Self-Organizing Maps •자기조직화 지도 • 특징 • Competitive learning by BP • 일종의 DR 기법 • 동작원리 • Components • Initialization • Competition • Cooperation • Adaptation https://arxiv.org/pdf/1312.5753.pdf source; wikipedia
  • 78.
    • 사이버보안 이상징후판단 • Malware 분석 • Network traffic 분석 • 센서 네트워크
  • 79.
    • 모터 결함진단 (Fault Diagnosis) • = 고장진단 (현 상태가 고장인지 여부) + 고장 예측 • 예: 유도 전동기 (Induction motor) • DBN (Deep Belief Network) • A generative graphical model • Stacking RBMs • 진동 (주파수) 측정 데이터를 이용하여 학습 • → Contrastive divergence using • Gibbs sampling
  • 80.
  • 82.
    • Anomaly detectionproblem Complexities • Unknownness • Anomalies are associated with many unknowns, e.g., instances with unknown abrupt behaviors, data structures, and distributions. They remain unknown until actually occur. • Heterogeneous anomaly classes. • Anomalies are irregular, and thus, one class of anomalies may demonstrate completely different abnormal characteristics from another class of anomalies. • Rarity and class imbalance • → unavailability of large-scale labeled data in most applications = class imbalance • Diverse types of anomaly. • 3 different types of anomaly have been explored. • Point anomalies • Conditional anomalies • Group anomalies • 3 different types of anomaly have been explored. • Point anomalies • Conditional anomalies = contextual anomalies • Group anomalies = collective anomalies
  • 83.
    • Deep AnomalyDetection가 해결 시도하는 문제 • CH1: Low anomaly detection recall rate. • CH2: Anomaly detection in high-dimensional and/or not-independent data. • CH3: Data-efficient learning of normality/abnormality. • CH4: Noise-resilient anomaly detection. • CH5: Detection of complex anomalies. • CH6: Anomaly explanation.
  • 85.
    • Deep AnomalyDetection 접근법의 3가지 Frameworks