This document discusses unsupervised machine learning approaches for detecting outliers and data defects, including clustering, PCA, and autoencoders. It proposes using an autoencoder model to learn hidden patterns in unlabeled data attributes and then detect invalid records that do not conform to the learned patterns. The approach involves preparing AutoML pipelines for clustering, PCA, decision trees, and autoencoders, then using them in an ensemble model to generate rules for detecting outliers and data quality issues, which would be presented to users for validation.

1. AI for Data Quality:
Epic 3a. Unsupervised methods

2. Data Defects
●
Outlier data means this data is totally different from the others
●
The key idea of this approach is to automatically discover domain-specific
patterns and generate rules for outliers
●
There are many outlier detection approaches including probabilistic and
statistical models, linear correlation analysis, proximity-based detection,
and supervised outlier detection
●
An unsupervised machine learning approach called autoencoder will be used
to learn hidden patterns in the attributes of the unlabelled data records
Country City Postcode Count
USA Michigan 100
UK Birmingham DY4 200
USA Birmingham 35251 100
USA Birmingham B1 3

3. Clustering
●
Clustering is an unsupervised technique that has
been widely used to investigate properties of the
data through grouping similar data into several
categories.
●
The similarity of the records are measured using
distance functions, such as Euclidean and
Manhattan distances.
●
Distance-based clustering algorithms cannot
derive the complex non-linear relationships that
exist among attributes of the data.

4. PCA
●
Principal Component Analysis (PCA) is a
representation learning approach that investigates
the relationships among the data attributes.
●
With this approach the features of correlated
attributes are converted into a set of linearly
uncorrelated attributes called principal
components.
●
The PCA representation learning can only
investigate the linear relationships among the
attributes.

5. Autoencoder
●
Autoencoder is a type of neural network that efficiently models complex
associations among attributes of the data through the composition of
several layers of non-linearity
●
A trained autoencoder model with its parameters learned to best describe
the patterns in the input data records:
– 1. Reconstructs the data records using the patterns discovered during the training
– 2. Detects invalid records - records that do not conform to the patterns discovered by
the autoencoder

6. Proposed approach
●
Prepare AutoML pipelines
for Cluster, PCA, Decision
Tree and Autoencoder
models
●
Use them in an ensemble
model (MajorityVote or
SVM)
●
Generate AutoDQ rules and
present them for User
validation in natural
language ordered by some
metric/confidence level
“City Birmingham with postcode not
typical for UK West Midland should
be flagged as an error with proposed
correction for country column =
‘USA’.”