The document outlines a syllabus focused on unsupervised learning techniques, emphasizing clustering and dimensionality reduction. It explains the definitions and applications of clustering, types of clustering algorithms (including k-means, density-based, hierarchical, and distribution-based), and their various applications across fields such as marketing, biology, and medical diagnostics. Additionally, it provides a detailed explanation of the k-means clustering algorithm, its iterative process, and an example for better understanding.

1. Unit IV
Syllabus: Unsupervised Learning Techniques: Clustering, K-Means, Limits of K-Means,
Using Clustering for Image Segmentation, Using Clustering for Preprocessing, Using
Clustering for Semi-Supervised Learning, DBSCAN, Gaussian Mixtures. Dimensionality
Reduction: The Curse of Dimensionality, Main Approaches for Dimensionality Reduction,
PCA, Using Scikit-Learn, Randomized PCA, Kernel PCA.
Unsupervised Learning Techniques: Clustering,
1. What is Clustering? What are the main applications of clustering algorithms?
Illustrate. [7M] July – 2023 Set -4[Remember]
In real world, not every data we work upon has a target variable. This kind of data cannot be
analyzed using supervised learning algorithms. We need the help of unsupervised algorithms.
One of the most popular type of analysis under unsupervised learning is Cluster analysis. When
the goal is to group similar data points in a dataset, then we use cluster analysis. In practical
situations, we can use cluster analysis for customer segmentation for targeted advertisements, or
in medical imaging to find unknown or new infected areas and many more use cases.
What is Clustering ?
The task of grouping data points based on their similarity with each other is called Clustering or
Cluster Analysis. This method is defined under the branch of Unsupervised Learning, which
aims at gaining insights from unlabelled data points, that is, unlike supervised learning we don’t
have a target variable.
Clustering aims at forming groups of homogeneous data points from a heterogeneous dataset. It
evaluates the similarity based on a metric like Euclidean distance, Cosine similarity, Manhattan
distance, etc. and then group the points with highest similarity score together.
For Example, In the graph given below, we can clearly see that there are 3 circular clusters
forming on the basis of distance.

2. Now it is not necessary that the clusters formed must be circular in shape. The shape of clusters
can be arbitrary. There are many algortihms that work well with detecting arbitrary shaped
clusters.
For example, In the below given graph we can see that the clusters formed are not circular in
shape.
Types of Clustering
Broadly speaking, there are 2 types of clustering that can be performed to group similar data
points:
 Hard Clustering: In this type of clustering, each data point belongs to a cluster
completely or not. For example, Let’s say there are 4 data point and we have to cluster
them into 2 clusters. So each data point will either belong to cluster 1 or cluster 2.
Data Points Clusters
A C1
B C2
C C2

3. Data Points Clusters
D C1
 Soft Clustering: In this type of clustering, instead of assigning each data point into a
separate cluster, a probability or likelihood of that point being that cluster is evaluated.
For example, Let’s say there are 4 data point and we have to cluster them into 2 clusters.
So we will be evaluating a probability of a data point belonging to both clusters. This
probability is calculated for all data points.
Data Points Probability of C1 Probability of C2
A 0.91 0.09
B 0.3 0.7
C 0.17 0.83
D 1 0
Uses of Clustering
Now before we begin with types of clustering algorithms, we will go through the use cases of
Clustering algorithms. Clustering algorithms are majorly used for:
 Market Segmentation – Businesses use clustering to group their customers and use
targeted advertisements to attract more audience.
 Market Basket Analysis – Shop owners analyze their sales and figure out which items are
majorly bought together by the customers. For example, In USA, according to a study
diapers and beers were usually bought together by fathers.
 Social Network Analysis – Social media sites use your data to understand your browsing
behaviour and provide you with targeted friend recommendations or content
recommendations.
 Medical Imaging – Doctors use Clustering to find out diseased areas in diagnostic images
like X-rays.
 Anomaly Detection – To find outliers in a stream of real-time dataset or forecasting
fraudulent transactions we can use clustering to identify them.
 Simplify working with large datasets – Each cluster is given a cluster ID after clustering
is complete. Now, you may reduce a feature set’s whole feature set into its cluster ID.

4. Clustering is effective when it can represent a complicated case with a straightforward
cluster ID. Using the same principle, clustering data can make complex datasets simpler.
There are many more use cases for clustering but there are some of the major and common use
cases of clustering. Moving forward we will be discussing Clustering Algorithms that will help
you perform the above tasks.
Types of Clustering Algorithms
At the surface level, clustering helps in the analysis of unstructured data. Graphing, the shortest
distance, and the density of the data points are a few of the elements that influence cluster
formation. Clustering is the process of determining how related the objects are based on a metric
called the similarity measure. Similarity metrics are easier to locate in smaller sets of features. It
gets harder to create similarity measures as the number of features increases. Depending on the
type of clustering algorithm being utilized in data mining, several techniques are employed to
group the data from the datasets. In this part, the clustering techniques are described. Various
types of clustering algorithms are:
1. Centroid-based Clustering (Partitioning methods)
2. Density-based Clustering (Model-based methods)
3. Connectivity-based Clustering (Hierarchical clustering)
4. Distribution-based Clustering
We will be going through each of these types in brief.
1. Centroid-based Clustering (Partitioning methods)
Partitioning methods are the most easiest clustering algorithms. They group data points on the
basis of their closeness. Generally, the similarity measure chosen for these algorithms are
Euclidian distance, Manhattan Distance or Minkowski Distance. The datasets are separated into a
predetermined number of clusters, and each cluster is referenced by a vector of values. When
compared to the vector value, the input data variable shows no difference and joins the cluster.
The primary drawback for these algorithms is the requirement that we establish the number of
clusters, “k,” either intuitively or scientifically (using the Elbow Method) before any clustering
machine learning system starts allocating the data points. Despite this, it is still the most popular
type of clustering. K-means and K-medoids clustering are some examples of this type clustering.
2. Density-based Clustering (Model-based methods)
Density-based clustering, a model-based method, finds groups based on the density of data
points. Contrary to centroid-based clustering, which requires that the number of clusters be
predefined and is sensitive to initialization, density-based clustering determines the number of
clusters automatically and is less susceptible to beginning positions. They are great at handling
clusters of different sizes and forms, making them ideally suited for datasets with irregularly

5. shaped or overlapping clusters. These methods manage both dense and sparse data regions by
focusing on local density and can distinguish clusters with a variety of morphologies.
In contrast, centroid-based grouping, like k-means, has trouble finding arbitrary shaped clusters.
Due to its preset number of cluster requirements and extreme sensitivity to the initial positioning
of centroids, the outcomes can vary. Furthermore, the tendency of centroid-based approaches to
produce spherical or convex clusters restricts their capacity to handle complicated or irregularly
shaped clusters. In conclusion, density-based clustering overcomes the drawbacks of centroid-
based techniques by autonomously choosing cluster sizes, being resilient to initialization, and
successfully capturing clusters of various sizes and forms. The most popular density-based
clustering algorithm is DBSCAN.
3. Connectivity-based Clustering (Hierarchical clustering)
A method for assembling related data points into hierarchical clusters is called hierarchical
clustering. Each data point is initially taken into account as a separate cluster, which is
subsequently combined with the clusters that are the most similar to form one large cluster that
contains all of the data points.
Think about how you may arrange a collection of items based on how similar they are. Each
object begins as its own cluster at the base of the tree when using hierarchical clustering, which
creates a dendrogram, a tree-like structure. The closest pairings of clusters are then combined
into larger clusters after the algorithm examines how similar the objects are to one another.
When every object is in one cluster at the top of the tree, the merging process has finished.
Exploring various granularity levels is one of the fun things about hierarchical clustering. To
obtain a given number of clusters, you can select to cut the dendrogram at a particular height.
The more similar two objects are within a cluster, the closer they are. It’s comparable to
classifying items according to their family trees, where the nearest relatives are clustered
together and the wider branches signify more general connections. There are 2 approaches for
Hierarchical clustering:
 Divisive Clustering: It follows a top-down approach, here we consider all data points to
be part one big cluster and then this cluster is divide into smaller groups.
 Agglomerative Clustering: It follows a bottom-up approach, here we consider all data
points to be part of individual clusters and then these clusters are clubbed together to
make one big cluster with all data points.
4. Distribution-based Clustering
Using distribution-based clustering, data points are generated and organized according to their
propensity to fall into the same probability distribution (such as a Gaussian, binomial, or other)
within the data. The data elements are grouped using a probability-based distribution that is
based on statistical distributions. Included are data objects that have a higher likelihood of being
in the cluster. A data point is less likely to be included in a cluster the further it is from the
cluster’s central point, which exists in every cluster.

6. A notable drawback of density and boundary-based approaches is the need to specify the clusters
a priori for some algorithms, and primarily the definition of the cluster form for the bulk of
algorithms. There must be at least one tuning or hyper-parameter selected, and while doing so
should be simple, getting it wrong could have unanticipated repercussions. Distribution-based
clustering has a definite advantage over proximity and centroid-based clustering approaches in
terms of flexibility, accuracy, and cluster structure. The key issue is that, in order to
avoid overfitting, many clustering methods only work with simulated or manufactured data, or
when the bulk of the data points certainly belong to a preset distribution. The most popular
distribution-based clustering algorithm is Gaussian Mixture Model.
Applications of Clustering in different fields:
1. Marketing: It can be used to characterize & discover customer segments for marketing
purposes.
2. Biology: It can be used for classification among different species of plants and animals.
3. Libraries: It is used in clustering different books on the basis of topics and information.
4. Insurance: It is used to acknowledge the customers, their policies and identifying the
frauds.
5. City Planning: It is used to make groups of houses and to study their values based on
their geographical locations and other factors present.
6. Earthquake studies: By learning the earthquake-affected areas we can determine the
dangerous zones.
7. Image Processing: Clustering can be used to group similar images together, classify
images based on content, and identify patterns in image data.
8. Genetics: Clustering is used to group genes that have similar expression patterns and
identify gene networks that work together in biological processes.
9. Finance: Clustering is used to identify market segments based on customer behavior,
identify patterns in stock market data, and analyze risk in investment portfolios.
10. Customer Service: Clustering is used to group customer inquiries and complaints into
categories, identify common issues, and develop targeted solutions.
11. Manufacturing: Clustering is used to group similar products together, optimize
production processes, and identify defects in manufacturing processes.
12. Medical diagnosis: Clustering is used to group patients with similar symptoms or
diseases, which helps in making accurate diagnoses and identifying effective treatments.
13. Fraud detection: Clustering is used to identify suspicious patterns or anomalies in
financial transactions, which can help in detecting fraud or other financial crimes.

7. 14. Traffic analysis: Clustering is used to group similar patterns of traffic data, such as peak
hours, routes, and speeds, which can help in improving transportation planning and
infrastructure.
15. Social network analysis: Clustering is used to identify communities or groups within
social networks, which can help in understanding social behavior, influence, and trends.
16. Cybersecurity: Clustering is used to group similar patterns of network traffic or system
behavior, which can help in detecting and preventing cyberattacks.
17. Climate analysis: Clustering is used to group similar patterns of climate data, such as
temperature, precipitation, and wind, which can help in understanding climate change and
its impact on the environment.
18. Sports analysis: Clustering is used to group similar patterns of player or team
performance data, which can help in analyzing player or team strengths and weaknesses
and making strategic decisions.
19. Crime analysis: Clustering is used to group similar patterns of crime data, such as
location, time, and type, which can help in identifying crime hotspots, predicting future
crime trends, and improving crime prevention strategies.
K-Means
2. Describe K means clustering algorithm. [7M] July – 2023 Set -1 Understand]
K-Means Clustering
K-means clustering is a popular unsupervised machine learning algorithm used to group data
points into K clusters based on their similarity. The K in K-means represents the number of
clusters that the algorithm will create.

8. The algorithm starts by selecting K random centroids (points in the feature space) as the initial
cluster centers. It then assigns each data point to the nearest centroid based on the Euclidean
distance between the data point and the centroid. Next, it computes the mean of each cluster,
which becomes the new centroid for that cluster. The algorithm repeats these steps until the
centroids no longer change or a maximum number of iterations is reached.
K-means clustering aims to minimize the sum of squared distances between each data point and
its assigned centroid. This means that the algorithm is trying to find the cluster centroids that
minimize the variance within each cluster and maximize the distance between clusters.
How K-Means Clustering Work?
K-means clustering works by iteratively assigning data points to the nearest cluster centroid,
computing the mean of each cluster, and updating the centroids until convergence is reached.
The following steps summarize the K-means clustering process:

9. 1. Initialization: Select the number of clusters (K) and randomly initialize K cluster
centroids in the feature space.
2. Assignment: Assign each data point to the nearest cluster centroid based on the Euclidean
distance between the data point and each centroid.
3. Update: Compute the mean of each cluster, which becomes the new centroid for that
cluster.
4. Repeat: Repeat steps 2 and 3 until the centroids no longer change or a maximum number
of iterations is reached.
5. Convergence: Once convergence is reached, the K clusters represent groups of data
points that are similar to each other and dissimilar to data points in other clusters.
To explain this process further, let’s consider an example where we want to cluster a dataset of
points in a two-dimensional feature space.
1. Initialization: We randomly select K centroids in the feature space. Let’s say we want to
create three clusters, so we randomly select three points as the initial centroids.
2. Assignment: We calculate the Euclidean distance between each data point and each
centroid. We assign each data point to the nearest centroid and form clusters.
3. Update: We calculate the mean of each cluster, which becomes the new centroid for that
cluster.
4. Repeat: We repeat steps 2 and 3 until the centroids no longer change or a maximum
number of iterations is reached.
5. Convergence: Once convergence is reached, the K clusters represent groups of data
points that are similar to each other and dissimilar to data points in other clusters. We can
use these clusters to analyze and understand the underlying patterns in the data.
In summary, K-means clustering is an iterative process that partitions data points into K clusters
based on their similarity. It is an unsupervised learning algorithm that is widely used in data
mining, pattern recognition, and image segmentation.
Example of K-means clustering
Cluster the following eight points (with (x, y) representing locations) into three clusters:
A1(2, 10), A2(2, 5), A3(8, 4), A4(5, 8), A5(7, 5), A6(6, 4), A7(1, 2), A8(4, 9).
Initial cluster centers are: A1(2, 10), A4(5, 8) and A7(1, 2).
The distance function between two points a = (x1, y1) and b = (x2, y2) is defined as → Ρ(a, b) =
|x2 — x1| + |y2 — y1|
Use K-Means Algorithm to find the three cluster centers after the second iteration.

10. We follow the above-discussed K-Means Clustering Algorithm:
→ Iteration-01:
 We calculate the distance of each point from each of the centers of the three clusters.
 The distance is calculated by using the given distance function.
The following illustration shows the calculation of the distance between point A1(2, 10) and
each of the centers of the three clusters:
Calculating Distance Between A1(2, 10) and C1(2, 10)
Ρ(A1, C1)
= |x2 — x1| + |y2 — y1|
= |2–2| + |10–10|
= 0
Calculating Distance Between A1(2, 10) and C2(5, 8)
Ρ(A1, C2)
= |x2 — x1| + |y2 — y1|
= |5–2| + |8–10|
= 3 + 2
= 5
Calculating Distance Between A1(2, 10) and C3(1, 2)
Ρ(A1, C3)
= |x2 — x1| + |y2 — y1|
= |1–2| + |2–10|
= 1 + 8
= 9
In a similar manner, we calculate the distance of other points from each of the centers of the
three clusters.
Next,
 We draw a table showing all the results.
 Using the table, we decide which point belongs to which cluster.
 The given point belongs to that cluster whose center is nearest to it.

11. From here, New clusters are-
Cluster-01:
The first cluster contains points-
 A1(2, 10)
Cluster-02:
The second cluster contains points-
 A3(8, 4)
 A4(5, 8)
 A5(7, 5)
 A6(6, 4)
 A8(4, 9)
Cluster-03:
The third cluster contains points-
 A2(2, 5)

12.  A7(1, 2)
Now,
 We re-compute the new cluster clusters.
 The new cluster center is computed by taking mean of all the points contained in that
cluster.
For Cluster-01:
 We have only one point A1(2, 10) in Cluster-01.
 So, the cluster center remains the same.
For Cluster-02:
Center of Cluster-02
= ((8 + 5 + 7 + 6 + 4)/5, (4 + 8 + 5 + 4 + 9)/5)
= (6, 6)
For Cluster-03:
Center of Cluster-03
= ((2 + 1)/2, (5 + 2)/2)
= (1.5, 3.5)
This is the completion of Iteration-01.
→ Iteration-02:
 We calculate the distance of each point from each of the centers of the three clusters.
 The distance is calculated by using the given distance function.
The following illustration shows the calculation of the distance between point A1(2, 10) and
each of the centers of the three clusters:
Calculating Distance Between A1(2, 10) and C1(2, 10)
Ρ(A1, C1)
= |x2 — x1| + |y2 — y1|
= |2–2| + |10–10|
= 0
Calculating Distance Between A1(2, 10) and C2(6, 6)
Ρ(A1, C2)

13. = |x2 — x1| + |y2 — y1|
= |6–2| + |6–10|
= 4 + 4
= 8
Calculating Distance Between A1(2, 10) and C3(1.5, 3.5)
Ρ(A1, C3)
= |x2 — x1| + |y2 — y1|
= |1.5–2| + |3.5–10|
= 0.5 + 6.5
= 7
In a similar manner, we calculate the distance of other points from each of the centers of the
three clusters.
Next,
 We draw a table showing all the results.
 Using the table, we decide which point belongs to which cluster.
 The given point belongs to that cluster whose center is nearest to it.

14. From here, New clusters are:
Cluster-01:
The first cluster contains points-
 A1(2, 10)
 A8(4, 9)
Cluster-02:
The second cluster contains points-
 A3(8, 4)
 A4(5, 8)
 A5(7, 5)
 A6(6, 4)
Cluster-03:
The third cluster contains points-
 A2(2, 5)
 A7(1, 2)
Now,
 We re-compute the new cluster clusters.
 The new cluster center is computed by taking mean of all the points contained in that
cluster.
For Cluster-01:
Center of Cluster-01
= ((2 + 4)/2, (10 + 9)/2)
= (3, 9.5)
For Cluster-02:
Center of Cluster-02
= ((8 + 5 + 7 + 6)/4, (4 + 8 + 5 + 4)/4)
= (6.5, 5.25)
For Cluster-03:
Center of Cluster-03

15. = ((2 + 1)/2, (5 + 2)/2)
= (1.5, 3.5)
This is the completion of Iteration-02.
After the second iteration, the center of the three clusters is:
 C1(3, 9.5)
 C2(6.5, 5.25)
 C3(1.5, 3.5)
3. Using K means clustering algorithm form two clusters for given data. [7M] July – 2023
Set -1[Apply] – Refer Notes
Steps of K-Means Algorithm:
1. Choose the K number of clusters.
2. Select K random points from graph or K data points from dataset as centroid of the
clusters.
3. Assign each data points to the closest centroid to form K clusters.
4. Keep iterating following until we find optimal centroid so that assignments to the data
points doesn’t change:
a. Compute the distance between data points and centroids.
b. Assign each data point to the cluster that is closer than other cluster (centroid).
c. Compute the centroids for the clusters by taking the average of all data points of that
cluster.
Height in cm and Weight in Kgs

16. Step 1: Let’s choose K = 2 as number of clusters.
Step 2: In this step we can either take 2 random data points or 2 data points from our dataset as
centroid of the clusters.
Let’s take first two data points as the two centroids C1(185, 72) and
C2(170, 56).
Step 3 & 4: Let’s take Point P3(168,60). In which cluster it will go?
We have to calculate the distance between both the clusters and assign new data point in that
cluster which has minimum distance between them.
Distance between C1 and P3, D13
= sqrt( (C1.X - P3.X)² + (C1.Y - P3.Y)² )

17. = sqrt( (185 - 168)² + (72–60)² )
= sqrt( 289+ 144 )
= sqrt( 433 )
= 20.80
Distance between C2 and P3, D23
= sqrt( (C2.X — P3.X)² + (C2.Y — P3.Y)² )
= sqrt( (170–168)² + (56–60)² )
= sqrt( 4+ 16 )
= sqrt( 20 )
= 4.48
So the point P3 will go into C2 cluster. Now we have to calculate the average of the centroid C2
(as it has minimum distance from Point P3) and newly added point and assign it back to centroid.
This will be our new centroid.
C2 = ( ( 170 + 168)/2 , (60 + 56)/2 )
= ( 169 , 58 )
As of now we have two clusters with following data points:
Cluster 1: C1
Cluster 2: C2, P3
Like this, we have to group all the remaining data points into these two clusters so that the final
cluster would be :
Cluster 1: C1, P4, P5, P6, P7, P8, P9, P10, P11, P12
Cluster 2: C2, P3

18. Now we have to check if there are any reassignment of data points into different clusters. It can be
checked using above steps. In our case this is the final clusters.
Using Clustering for Image Segmentation
4. How can we use clustering for image segmentation? Explain. [7M] July – 2023 Set -
3[Remember]
Introduction
In simpler terms, image segmentation is a process of dividing an image into multiple parts or
segments, where each segment represents a unique object or region within the image. This
process can be used in a variety of applications, including medical imaging, autonomous driving,
surveillance, and image editing.
Image segmentation is a challenging task due to the variability of objects and scenes in real-
world images. Some common methods used for image segmentation include thresholding,
clustering, edge detection, and deep learning-based approaches such as convolutional neural
networks (CNNs).
What is Image Segmentation?
Image segmentation is the process of dividing an image into multiple segments or regions, each
of which represents a meaningful object or part of the image. The primary goal of image
segmentation is to extract useful information from images, simplify image representation, and
enable higher-level computer vision tasks.
 Extracting Useful Information: Image segmentation allows us to identify and extract
useful information from an image that may be difficult or impossible to obtain without

19. segmenting the image. For example, in medical imaging, image segmentation can be used
to identify and isolate specific organs or tissues for further analysis or diagnosis.
 Simplifying Image Representation: Images can be complex and contain a lot of
information. By segmenting an image into meaningful regions or objects, we can simplify
the image representation and make it easier to analyze. This can be particularly useful for
computer vision tasks such as object recognition or tracking, where the focus is on
specific regions of an image rather than the entire image.
 Enabling Higher-level Computer Vision Tasks: Image segmentation is a critical step in
many higher-level computer vision tasks, such as object detection, semantic
segmentation, and instance segmentation. By segmenting an image into regions or
objects, we can better understand the spatial relationships between different parts of the
image, which can be used to classify and identify objects, track their movements, or
generate 3D models.
Need for Image Segmentation
Image segmentation is needed in computer vision for object recognition, image editing, and
medical imaging. It is essential to train machine learning algorithms to recognize and classify
different objects. Image segmentation is also used to isolate specific regions of an image for
editing and identifying different tissues or organs in medical images.
 Object Recognition: Image segmentation is essential for object recognition in computer
vision. By segmenting an image into meaningful objects or regions, we can train machine
learning algorithms to recognize and classify different objects. This is useful in many
applications, such as autonomous vehicles, surveillance systems, and robotics.

20.  Image Editing: Image segmentation is also used in image editing software to isolate
specific regions of an image for editing. For example, in photo editing software, image
segmentation can be used to select and enhance a particular object or remove unwanted
objects from an image.
 Medical Imaging: Image segmentation is critical in medical imaging applications, such
as identifying tumours or lesions in MRI or CT scans. By segmenting an image into
different regions or objects, doctors can identify and diagnose medical conditions more
accurately and efficiently.
 Augmented Reality: Image segmentation is also necessary for augmented reality
applications, where virtual objects are overlaid onto the real-world environment. By
segmenting an image into different objects or regions, AR systems can identify and track
real-world objects to enhance the user's experience.
 Video Analysis: Image segmentation is used in video analysis to track and monitor
objects over time. By segmenting an image into different objects or regions, we can track
the movements of these objects, which is useful in applications such as traffic monitoring
or surveillance.
Clustering
Clustering involves dividing an image into groups based on their similarity. The most common
clustering algorithms used for image segmentation are K-means and Mean-shift. Clustering is
useful for image segmentation and images with multiple regions, but it may not work well with
images that contain noise or outliers.
1. K-Means Clustering K-means clustering and Mean Shift Clustering are two common
unsupervised machine learning algorithms used for clustering and grouping data points
based on their similarity.

21. K-means clustering is a simple and widely used clustering algorithm for image
segmentation that aims to partition data points into k clusters based on their similarity in
feature space. It works by first randomly initializing k cluster centroids and then
iteratively assigning each data point to the closest centroid based on a distance metric,
such as Euclidean distance. After all data points have been assigned to their closest
centroids, the centroids are updated to the mean of all data points assigned to that cluster.
This process is repeated until the centroids converge or a maximum number of iterations
is reached.
2. Mean Shift Clustering Mean Shift Clustering is a more advanced clustering algorithm in
image segmentation that does not require a pre-specified number of clusters like k-means.
Instead, it works by iteratively shifting each data point towards a higher density region in
feature space until it converges to a mode or local maxima of the probability density
function. This is achieved by applying a kernel function to each data point to estimate its
density and then shifting it towards the weighted mean of all nearby data points within a
certain radius or bandwidth.
Compared to k-means, Mean Shift Clustering is more robust to noise and can identify
non-linearly separated clusters. However, it is computationally more expensive and may
not be suitable for large datasets. Both algorithms are widely used in applications such as
image segmentation, pattern recognition, and data analysis.

22. Clustering Clustering is a image segmentation technique that involves grouping pixels in an
image based on their similarity in color, texture, or other features. One common clustering
algorithm used in image segmentation is k-means clustering, which we discussed earlier.
We can apply k-means clustering to an image in OpenCV as follows:
Step 1: Import the necessary libraries
Here, we import the necessary libraries and read in the input image 'image.jpg' using
the cv2.imread() function of OpenCV.
import cv2
import numpy as np
# Read image
img = cv2.imread('image.jpg')
Step 2: Reshape image to a 2D array of pixels
We reshape the 3D input image into a 2D array of pixels using the reshape() function of NumPy.
# Reshape image to a 2D array of pixels
pixels = img.reshape((-1,3))
Here, we specify the shape of the new array as (-1,3), where -1 is inferred automatically based on
the size of the original array and 3 represents the number of channels in the image (BGR).
Step 3: Convert pixel values to float32 for image segmentation

23. We convert the pixel values to float32 using the np.float32() function of NumPy. This is required
for applying the k-means clustering algorithm.
# Convert pixel values to float32
pixels = np.float32(pixels)
Step 4: Apply k-means clustering for image segmentation We apply k-means clustering to the
pixel values using the cv2.kmeans() function of OpenCV.
# Apply k-means clustering
k = 5
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 10, 1.0)
flags = cv2.KMEANS_RANDOM_CENTERS
compactness, labels, centers = cv2.kmeans(pixels, k, None, criteria, 10,
flags)
 Here, we specify the number of clusters k as 5, and the termination criteria for the
algorithm as 10 iterations or a change in compactness value of less than 1.0.
 We also specify the flag for selecting random initial centres
as cv2.KMEANS_RANDOM_CENTERS.
 The function returns the compactness value, the labels assigned to each pixel, and the
centre values of each cluster.
Step 5: Reshape labels and centres to the original image shape for image segmentation
We reshape the labels assigned to each pixel and the centre values of each cluster to the original
image shape using the reshape() function and the img.shape attribute of the input image. We also
convert the centre values to the uint8 data type using the np.uint8() function of NumPy. We
assign the cluster centre values to the corresponding pixel labels to obtain a segmented image.
# Reshape labels and centres to the original image shape
labels = labels.reshape(img.shape[:2])
centers = np.uint8(centers)
res = centers[labels]
Step 6: Display the clustered images
Finally, we display the output of the image segmentation using the cv2.imshow() function of
OpenCV and wait for a key press before closing the window using
the cv2.waitKey() and cv2.destroyAllWindows() functions.
# Display clustered image
cv2.imshow('clustered', res)
cv2.waitKey(0)
cv2.destroyAllWindows()

24. Input Image
Output Image

25. Using Clustering for Preprocessing
5. How can we use clustering for Preprocessing? Explain. [7M] July – 2023 Set -
2[Remember]
Clustering is an unsupervised learning technique that groups similar data points together. It
analyzes the data to identify inherent structures and creates clusters where data points within a
cluster share common characteristics.
How is it used in Preprocessing?
There are several ways clustering can be used to preprocess data for machine learning models:
1. Identify Outliers: Clustering algorithms can help identify data points that fall far away
from the main clusters. These outliers can significantly impact the performance of
machine learning models. By identifying them through clustering, you can choose to
remove them or handle them appropriately before feeding the data to your model.
2. Dimensionality Reduction: Sometimes datasets can have many features (dimensions).
Clustering can be used to group similar features together, effectively reducing the
dimensionality of the data. This can improve the efficiency and accuracy of machine
learning models, especially those that struggle with high dimensionality.
3. Feature Engineering: Clustering can reveal hidden patterns in your data that can be used
to create new features. For instance, if you cluster customer data, you might discover
distinct customer segments. These segments can be transformed into new features for
your model.
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
# Create sample data
data, _ = make_blobs(n_samples=200, centers=4, random_state=0)
# Create the plot
fig, ax = plt.subplots()
# Scatter plot the data points
ax.scatter(data[:, 0], data[:, 1])
# Define cluster centers
cluster_centers = [[4,5], [2,2], [8,5], [6,1]]
# Plot the cluster centers
for center in cluster_centers:
ax.plot(center[0], center[1], marker='o', markersize=15, color='red')
# Set labels
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_title('Sample Data with Cluster Centers')

26. # Show the plot
plt.show()
Understanding make_blobs
The make_blobs function in scikit-learn is designed to generate synthetic datasets with well-
defined clusters. This makes it a great tool to illustrate how clustering works. Here's how it's
used:
 n_samples: The total number of data points to generate.
 centers: The number of clusters to create.
 cluster_std: Standard deviation of the clusters. Controls the spread of the data points
within each cluster.
 random_state: Sets a seed for reproducibility purposes.
Preprocessing with Clustering
Here's how clustering could be used as part of preprocessing on a dataset like the one generated
by make_blobs:
1. Outlier Detection:
o Apply a clustering algorithm (like K-Means or DBSCAN).

27. o Points that don't fit well into any cluster are potential outliers.
o Investigate these outliers and decide if they should be removed or kept based on
domain knowledge and how they might affect your model.
2. Feature Engineering:
o Cluster assignments can become new categorical features.
o For example, you could add a column "cluster_label" to your dataset, indicating
which cluster each data point belongs to. This can help your model uncover
patterns related to cluster membership.
Note: Since make_blobs creates data specifically designed to have well-separated clusters, the
need for preprocessing is often less pronounced.
Using Clustering for Semi-Supervised Learning
6. How can we use clustering for semi-supervised learning? Explain[7M] July – 2023
Set -4[Remember]
Clustering can be utilized in semi-supervised learning to enhance the performance of models
when labeled data is scarce. Here's how clustering can be integrated into semi-supervised
learning:
1. Initial Cluster Formation: Initially, unlabeled data is clustered using an unsupervised
clustering algorithm such as k-means, hierarchical clustering, or DBSCAN. These
clusters can represent potential classes or groupings within the data.
2. Label Propagation: Once the clusters are formed, labels from a small subset of data points
(seed points) can be propagated to the entire cluster. This assumes that data points within
the same cluster share similar characteristics and thus can be assigned the same label.
Various techniques can be used for label propagation, such as assigning the most frequent
label in the cluster or propagating labels based on a probabilistic framework.
3. Model Training: With the expanded labeled dataset obtained from label propagation, a
semi-supervised learning model can be trained. This model can be a variety of algorithms
such as support vector machines, decision trees, neural networks, etc., depending on the
nature of the problem.
4. Iterative Refinement: The process can be iterated by re-assigning labels to the unlabeled
data based on the updated model's predictions. This iterative refinement can improve the
quality of label propagation and subsequently enhance the performance of the semi-
supervised learning model.
5. Evaluation: Finally, the trained semi-supervised learning model can be evaluated using a
separate validation set or through cross-validation to assess its performance.
By incorporating clustering into semi-supervised learning, we can effectively leverage the
structure of the unlabeled data to augment the labeled dataset, thereby improving the robustness

28. and generalization capabilities of the model, especially in scenarios where labeled data is limited
or expensive to obtain.
Unsupervised learning is a type of machine learning where the data is unlabeled, meaning it has
no predefined categories. The algorithm has to discover the structure of the data on its own.
In the unsupervised clustering section of the image, there are data points represented by hearts.
These data points are unlabeled and the task is to group them into classes. The algorithm does
this by finding similarities between the data points. For instance, it might group data points that
are close together in space together.
Semi-supervised clustering, on the other hand, uses a combination of labeled and unlabeled data.
In the semi-supervised clustering section of the image, there are data points represented by
circles and squares. The circles are the labeled data points, and the squares are the unlabeled data
points. The algorithm uses the labeled data points to learn the structure of the data and then uses
this knowledge to cluster the unlabeled data points.

29. So, in essence, semi-supervised clustering leverages a small amount of labeled data to improve
the clustering of a larger amount of unlabeled data. This can be useful in situations where
labeling data is expensive or time-consuming.
DBSCAN
7. What is Density based clustering? Describe DBSCAN clustering algorithm. [7M]
July – 2023 Set -2[Remember]
Clustering analysis is an unsupervised learning method that separates the data points into several
specific bunches or groups, such that the data points in the same groups have similar properties
and data points in different groups have different properties in some sense.
It comprises of many different methods based on different distance measures. E.g. K-Means
(distance between points), Affinity propagation (graph distance), Mean-shift (distance between
points), DBSCAN (distance between nearest points), Gaussian mixtures (Mahalanobis distance
to centers), Spectral clustering (graph distance), etc.
Centrally, all clustering methods use the same approach i.e. first we calculate similarities and
then we use it to cluster the data points into groups or batches. Here we will focus on
the Density-based spatial clustering of applications with noise (DBSCAN) clustering method.
Why do we need a Density-Based clustering algorithm like DBSCAN when we already have
K-means clustering?
K-Means clustering may cluster loosely related observations together. Every observation
becomes a part of some cluster eventually, even if the observations are scattered far away in the
vector space. Since clusters depend on the mean value of cluster elements, each data point plays
a role in forming the clusters. A slight change in data points might affect the clustering outcome.
This problem is greatly reduced in DBSCAN due to the way clusters are formed. This is usually
not a big problem unless we come across some odd shape data.
Another challenge with k-means is that you need to specify the number of clusters (“k”) in order
to use it. Much of the time, we won’t know what a reasonable k value is a priori.
What’s nice about DBSCAN is that you don’t have to specify the number of clusters to use it.
All you need is a function to calculate the distance between values and some guidance for what
amount of distance is considered “close”. DBSCAN also produces more reasonable results
than k-means across a variety of different distributions. Below figure illustrates the fact:

30. Density-Based Clustering Algorithms
Density-Based Clustering refers to unsupervised learning methods that identify distinctive
groups/clusters in the data, based on the idea that a cluster in data space is a contiguous region of
high point density, separated from other such clusters by contiguous regions of low point density.
Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is a base algorithm for
density-based clustering. It can discover clusters of different shapes and sizes from a large
amount of data, which is containing noise and outliers.
The DBSCAN algorithm uses two parameters:
 minPts: The minimum number of points (a threshold) clustered together for a region to be
considered dense.
 eps (ε): A distance measure that will be used to locate the points in the neighborhood of any
point.
These parameters can be understood if we explore two concepts called Density Reachability and
Density Connectivity.
Reachability in terms of density establishes a point to be reachable from another if it lies within
a particular distance (eps) from it.
Connectivity, on the other hand, involves a transitivity based chaining-approach to determine
whether points are located in a particular cluster. For example, p and q points could be connected
if p->r->s->t->q, where a->b means b is in the neighborhood of a.
There are three types of points after the DBSCAN clustering is complete:
 Core — This is a point that has at least m points within distance n from itself.
 Border — This is a point that has at least one Core point at a distance n.
 Noise — This is a point that is neither a Core nor a Border. And it has less than m points within
distance n from itself.

31. Algorithmic steps for DBSCAN clustering
 The algorithm proceeds by arbitrarily picking up a point in the dataset (until all points have
been visited).
 If there are at least ‘minPoint’ points within a radius of ‘ε’ to the point then we consider all
these points to be part of the same cluster.
 The clusters are then expanded by recursively repeating the neighborhood calculation for each
neighboring point
DBSCAN in action
Parameter Estimation
Every data mining task has the problem of parameters. Every parameter influences the algorithm
in specific ways. For DBSCAN, the parameters ε and minPts are needed.
 minPts: As a rule of thumb, a minimum minPts can be derived from the number of
dimensions D in the data set, as minPts ≥ D + 1. The low value minPts = 1 does not make
sense, as then every point on its own will already be a cluster. With minPts ≤ 2, the result will
be the same as of hierarchical clustering with the single link metric, with the dendrogram cut at
height ε. Therefore, minPts must be chosen at least 3. However, larger values are usually better

32. for data sets with noise and will yield more significant clusters. As a rule of thumb, minPts =
2·dim can be used, but it may be necessary to choose larger values for very large data, for
noisy data or for data that contains many duplicates.
 ε: The value for ε can then be chosen by using a k-distance graph, plotting the distance to
the k = minPts-1 nearest neighbor ordered from the largest to the smallest value. Good values
of ε are where this plot shows an “elbow”: if ε is chosen much too small, a large part of the
data will not be clustered; whereas for a too high value of ε, clusters will merge and the
majority of objects will be in the same cluster. In general, small values of ε are preferable, and
as a rule of thumb, only a small fraction of points should be within this distance of each other.
 Distance function: The choice of distance function is tightly linked to the choice of ε, and has
a major impact on the outcomes. In general, it will be necessary to first identify a reasonable
measure of similarity for the data set, before the parameter ε can be chosen. There is no
estimation for this parameter, but the distance functions need to be chosen appropriately for the
data set.
DBSCAN Python Implementation Using Scikit-learn
Let us first apply DBSCAN to cluster spherical data.
We first generate 750 spherical training data points with corresponding labels. After that
standardize the features of your training data and at last, apply DBSCAN from the sklearn
library.
import numpy as np
from sklearn.cluster import DBSCAN
from sklearn import metrics
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler
# Generate sample data
centers = [[1, 1], [-1, -1], [1, -1]]
X, labels_true = make_blobs(n_samples=750, centers=centers, cluster_std=0.4,
random_state=0)
X = StandardScaler().fit_transform(X)
# Compute DBSCAN
db = DBSCAN(eps=0.3, min_samples=10).fit(X)

33. core_samples_mask = np.zeros_like(db.labels_, dtype=bool)
core_samples_mask[db.core_sample_indices_] = True
labels = db.labels_
# Number of clusters in labels, ignoring noise if present.
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
n_noise_ = list(labels).count(-1)
print('Estimated number of clusters: %d' % n_clusters_)
print('Estimated number of noise points: %d' % n_noise_)
print("Homogeneity: %0.3f" % metrics.homogeneity_score(labels_true, labels))
print("Completeness: %0.3f" % metrics.completeness_score(labels_true, labels))
print("V-measure: %0.3f" % metrics.v_measure_score(labels_true, labels))
print("Adjusted Rand Index: %0.3f"
% metrics.adjusted_rand_score(labels_true, labels))
print("Adjusted Mutual Information: %0.3f"
% metrics.adjusted_mutual_info_score(labels_true, labels))
print("Silhouette Coefficient: %0.3f"
% metrics.silhouette_score(X, labels))
# Plot result
import matplotlib.pyplot as plt
%matplotlib inline
# Black removed and is used for noise instead.
unique_labels = set(labels)
colors = [plt.cm.Spectral(each)
for each in np.linspace(0, 1, len(unique_labels))]
for k, col in zip(unique_labels, colors):
if k == -1:
# Black used for noise.
col = [0, 0, 0, 1]
class_member_mask = (labels == k)
xy = X[class_member_mask & core_samples_mask]

34. plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
markeredgecolor='k', markersize=14)
xy = X[class_member_mask & ~core_samples_mask]
plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
markeredgecolor='k', markersize=6)
plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()

35. DBSCAN to cluster spherical data
The black data points represent outliers in the above result. Next, apply DBSCAN to cluster non-spherical data.
import numpy as np
import matplotlib.pyplot as plt
from sklearn import metrics
from sklearn.datasets import make_circles
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import DBSCAN
X, y = make_circles(n_samples=750, factor=0.3, noise=0.1)
X = StandardScaler().fit_transform(X)
y_pred = DBSCAN(eps=0.3, min_samples=10).fit_predict(X)
plt.scatter(X[:,0], X[:,1], c=y_pred)
print('Number of clusters: {}'.format(len(set(y_pred[np.where(y_pred != -1)]))))

36. print('Homogeneity: {}'.format(metrics.homogeneity_score(y, y_pred)))
print('Completeness: {}'.format(metrics.completeness_score(y, y_pred)))
print("V-measure: %0.3f" % metrics.v_measure_score(labels_true, labels))
print("Adjusted Rand Index: %0.3f"
% metrics.adjusted_rand_score(labels_true, labels))
print("Adjusted Mutual Information: %0.3f"
% metrics.adjusted_mutual_info_score(labels_true, labels))
print("Silhouette Coefficient: %0.3f"
% metrics.silhouette_score(X, labels))

37. DBSCAN to cluster non-spherical data
Which is absolutely perfect. If we compare with K-means it would give a completely incorrect
output like:
K-means clustering result

38. Gaussian Mixtures
8. What is a Gaussian mixture? What tasks can you use it for? Explain. [7M] July –
2023 Set -3[Remember]
Have you ever wondered how machine learning algorithms can effortlessly categorize complex
data into distinct groups?
Gaussian Mixture Models (GMMs) play a pivotal role in achieving this task.
Recognized as a robust statistical tool in machine learning and data science, GMMs excel in
estimating density and clustering data.
In this article, I will dive into the world of Gaussian Mixture Models, explaining their
importance, functionality, and application in various fields.
Gaussian Mixture Models Overview
Imagine blending multiple Gaussian distributions to form a single model. This is precisely what
a Gaussian Mixture Model does.
At its heart, GMM operates on the principle that a complex, multi-modal distribution can be
approximated by a combination of simpler Gaussian distributions, each representing a different
cluster within the data.
The essence of GMM lies in its ability to determine cluster characteristics such as mean,
variance, and weight.
The mean of each Gaussian component gives us a central point, around which the data points are
most densely clustered.
The variance, on the other hand, provides insight into the spread or dispersion of the data points
around this mean. A smaller variance indicates that the data points are closely clustered around
the mean, while a larger variance suggests a more spread-out cluster.
The weights in a GMM are particularly significant. They represent the proportion of the dataset
that belongs to each Gaussian component.
In a sense, these weights embody the strength or dominance of each cluster within the overall
mixture. Higher weights imply that a greater portion of the data aligns with that particular
Gaussian distribution, signifying its greater prominence in the model.
This triad of parameters — mean, variance, and weight — enables GMMs to model the data with
remarkable flexibility. By adjusting these parameters, a GMM can shape itself to fit a wide
variety of data distributions, whether they are tightly clustered, widely dispersed, or overlapping
with one another.
One of the most powerful aspects of GMMs is their capacity to compute the probability of each
data point belonging to a particular cluster.

39. This is achieved through a process known as ‘soft clustering’, as opposed to ‘hard clustering’
methods like K-Means.
In soft clustering, instead of forcefully assigning a data point to a single cluster, GMM assigns
probabilities that indicate the likelihood of that data point belonging to each of the Gaussian
components.
Algorithms
Model Representation
At its core, a GMM is a combination of several Gaussian components.
These components are defined by their mean vectors, covariance matrices, and weights,
providing a comprehensive representation of data distributions.
The probability density function of a GMM is a sum of its components, each weighted
accordingly.
Notation:
 K: Number of Gaussian components
 N: Number of data points
 D: Dimensionality of the data
GMM Parameters:
 Means (μ): Center locations of Gaussian components.
 Covariance Matrices (Σ): Define the shape and spread of each component.
 Weights (π): Probability of selecting each component.
Model Training
Training a GMM involves setting the parameters using available data. The Expectation-
Maximization (EM) technique is often employed, alternating between the Expectation (E) and
Maximization (M) steps until convergence.
Expectation-Maximization
During the E step, the model calculates the probability of each data point belonging to each
Gaussian component. The M step then adjusts the model’s parameters based on these
probabilities.
Clustering and Density Estimation
Post-training, GMMs cluster data points based on the highest posterior probability. They are also
used for density estimation, assessing the probability density at any point in the feature space.
Implementation of Gaussian Mixture Models

40. This code generates some sample data from two different normal distributions and uses a
Gaussian Mixture Model from Scikit-learn to fit this data.
It then predicts which cluster each data point belongs to and visualizes the data points with their
respective clusters.
The centers of the Gaussian components are marked with red ‘X’ symbols.
The resulting plot provides a visual representation of how the GMM has clustered the data.
After fitting the Gaussian Mixture Model to the data, a new data point at coordinates [2,2] is
defined.
The predict_proba method of the GMM object is then used to calculate the probability of this
new data point belonging to each of the two clusters.
The resulting probabilities are printed, and the data points, Gaussian centers, and the new data
point are plotted for visualization.

41. Use Cases of Gaussian Mixture Models
GMMs find application in a diverse range of fields:
 Anomaly Detection: Identifying unusual data patterns.
 Image Segmentation: Grouping pixels in images based on color or texture.
 Speech Recognition: Assisting in the recognition of phonemes in audio data.
 Handwriting Recognition: Simulating different handwriting styles.
 Customer Segmentation: Grouping customers with similar behaviors or preferences.
 Data Clustering: Finding natural groups in data.
 Computer Vision: Object detection and background removal.
 Bioinformatics: Analyzing gene expression data.
 Recommendation Systems: Personalizing user experiences.
 Medical Imaging: Tissue classification and abnormality detection.
 Finance: Asset price modeling and risk management.

42. Advantages and Disadvantages of Gaussian Mixture Models
Advantages
 Flexibility in Data Representation: GMMs adeptly represent complex data structures.
 Probabilistic Approach: They provide probabilities for cluster assignments, aiding in
uncertainty estimation.
 Soft Clustering: GMMs offer probabilistic cluster assignments, allowing for more
nuanced data analysis.
 Effective in Overlapping Clusters: They accurately model data with overlapping
clusters.
 Density Estimation Capabilities: Useful in understanding the underlying distribution of
data.
 Handling Missing Data: GMMs can estimate parameters even with incomplete data sets.
 Outlier Detection: Identifying data points that do not conform to the general pattern.
 Scalability and Simplicity: Effective in handling large datasets and relatively easy to
implement.
 Interpretable Parameters: Provides meaningful insights into cluster characteristics.
Disadvantages
 Challenges in Determining Component Number: Misjudgment in component number
can lead to overfitting or underfitting.
 Initialization Sensitivity: The outcome is influenced by the initial parameter settings.
 Assumption of Gaussian Distribution: Not always applicable if data do not adhere to
Gaussian distributions.
 Curse of Dimensionality: High-dimensional data can complicate the model.
 Convergence Issues: Problems arise when dealing with singular covariance matrices.
 Resource Intensive for Large Datasets: Computing and memory requirements can be
substantial.
Gaussian mixture models are versatile and can be used for various tasks, including:
1. Clustering: GMMs can be used for clustering similar data points together. Each
component in the mixture model can represent a cluster, and data points are assigned to
the component with the highest probability.

43. 2. Density Estimation: GMMs can be used to estimate the probability density function of a
dataset. This is useful in applications such as anomaly detection, where regions of low
probability density may indicate anomalies.
3. Semi-supervised Learning: GMMs can be used in semi-supervised learning scenarios
where only a portion of the data is labeled. By modeling the underlying distribution of the
data, GMMs can help make predictions on unlabeled data points.
4. Generative Modeling: GMMs can be used as generative models to generate new
samples similar to the training data. By sampling from the learned mixture model, new
data points can be generated.
5. Dimensionality Reduction: GMMs can be used for dimensionality reduction by fitting
the model to the data and then representing each data point by the posterior probabilities
of belonging to each component. This can be particularly useful for visualizing high-
dimensional data in lower dimensions.
Dimensionality Reduction: The Curse of Dimensionality
9. What is Curse of Dimensionality? How to find the solution for it? Explain. [7M] July –
2023 Set -1[Remember]
Curse of Dimensionality describes the explosive nature of increasing data dimensions and its
resulting exponential increase in computational efforts required for its processing and/or
analysis. This term was first introduced by Richard E. Bellman, to explain the increase in volume
of Euclidean space associated with adding extra dimensions, in area of dynamic programming.
Today, this phenomenon is observed in fields like machine learning, data analysis, data mining
to name a few. An increase in the dimensions can in theory, add more information to the data
thereby improving the quality of data but practically increases the noise and redundancy during
its analysis.
Behavior of a Machine Learning Algorithms — Need for data points and Accuracy of
Model
In machine learning, a feature of an object can be an attribute or a characteristic that defines it.
Each feature represents a dimension and group of dimensions creates a data point. This
represents a feature vector that defines the data point to be used by a machine learning
algorithm(s). When we say increase in dimensionality it implies an increase in the number of
features used to describe the data. For example, in the field of breast cancer research, age,
number of cancerous nodes can be used as features to define the prognosis of the breast cancer
patient. These features constitute the dimensions of a feature vector. But other factors like past
surgeries, patient history, type of tumor and other such features help a doctor to better determine
the prognosis. In this case by adding features, we are theoretically increasing the dimensions of
our data.
As the dimensionality increases, the number of data points required for good performance of any
machine learning algorithm increases exponentially. The reason is that, we would need more
number of data points for any given combination of features, for any machine learning model to

44. be valid. For example, let’s say that for a model to perform well, we need at least 10 data points
for each combination of feature values. If we assume that we have one binary feature, then for its
21 unique values (0 and 1) we would need 2¹x 10 = 20 data points. For 2 binary features, we
would have 2² unique values and need 2² x 10 = 40 data points. Thus, for k-number of binary
features we would need 2ᵏ x 10 data points.
Hughes (1968) in his study concluded that with a fixed number of training samples, the
predictive power of any classifier first increases as the number of dimensions increase, but after a
certain value of number of dimensions, the performance deteriorates. Thus, the phenomenon of
curse of dimensionality is also known as Hughes phenomenon.

45. Graphical Representation of Hughes Principle
Effect of Curse of Dimensionality on Distance Functions:
For any point A, lets assume distₘᵢₙ(A) is the minimum distance between A and its nearest
neighbor and distₘₐₓ(A) is the maximum distance between A and the farthest neighbor.
That is, for a d — dimensional space, given n-random points, the distₘᵢₙ(A) ≈ distₘₐₓ(A) meaning,
any given pair of points are equidistant to each other.

46. Therefore, any machine learning algorithms which are based on the distance measure including
KNN(k-Nearest Neighbor) tend to fail when the number of dimensions in the data is very high.
Thus, dimensionality can be considered as a “curse” in such algorithms.
Solutions to Curse of Dimensionality:
One of the ways to reduce the impact of high dimensions is to use a different measure of distance
in a space vector. One could explore the use of cosine similarity to replace Euclidean distance.
Cosine similarity can have a lesser impact on data with higher dimensions. However, use of such
method could also be specific to the required solution of the problem.
Other methods:
Other methods could involve the use of reduction in dimensions. Some of the techniques that can
be used are:
1. Forward-feature selection: This method involves picking the most useful subset of features
from all given features.
2. PCA/t-SNE: Though these methods help in reduction of number of features, but it does not
necessarily preserve the class labels and thus can make the interpretation of results a tough task.
10. How can you evaluate the performance of a dimensionality reduction algorithm on
your dataset? Explain. [7M] July – 2023 Set -4[Remember]
Dimensionality reduction is the process of reducing the number of features (or dimensions) in a
dataset while retaining as much information as possible. This can be done for a variety of
reasons, such as to reduce the complexity of a model, to improve the performance of a learning
algorithm, or to make it easier to visualize the data. There are several techniques for
dimensionality reduction, including principal component analysis (PCA), singular value
decomposition (SVD), and linear discriminant analysis (LDA). Each technique uses a different
method to project the data onto a lower-dimensional space while preserving important
information.
What is Dimensionality Reduction?
Dimensionality reduction is a technique used to reduce the number of features in a dataset while
retaining as much of the important information as possible. In other words, it is a process of
transforming high-dimensional data into a lower-dimensional space that still preserves the
essence of the original data.
In machine learning, high-dimensional data refers to data with a large number of features or
variables. The curse of dimensionality is a common problem in machine learning, where the
performance of the model deteriorates as the number of features increases. This is because the
complexity of the model increases with the number of features, and it becomes more difficult to
find a good solution. In addition, high-dimensional data can also lead to overfitting, where the
model fits the training data too closely and does not generalize well to new data.

47. Dimensionality reduction can help to mitigate these problems by reducing the complexity of the
model and improving its generalization performance. There are two main approaches to
dimensionality reduction: feature selection and feature extraction.
Feature Selection:
Feature selection involves selecting a subset of the original features that are most relevant to the
problem at hand. The goal is to reduce the dimensionality of the dataset while retaining the most
important features. There are several methods for feature selection, including filter methods,
wrapper methods, and embedded methods. Filter methods rank the features based on their
relevance to the target variable, wrapper methods use the model performance as the criteria for
selecting features, and embedded methods combine feature selection with the model training
process.
Feature Extraction:
Feature extraction involves creating new features by combining or transforming the original
features. The goal is to create a set of features that captures the essence of the original data in a
lower-dimensional space. There are several methods for feature extraction, including principal
component analysis (PCA), linear discriminant analysis (LDA), and t-distributed stochastic
neighbor embedding (t-SNE). PCA is a popular technique that projects the original features onto
a lower-dimensional space while preserving as much of the variance as possible.
Why is Dimensionality Reduction important in Machine Learning and Predictive
Modeling?
An intuitive example of dimensionality reduction can be discussed through a simple e-mail
classification problem, where we need to classify whether the e-mail is spam or not. This can
involve a large number of features, such as whether or not the e-mail has a generic title, the
content of the e-mail, whether the e-mail uses a template, etc. However, some of these features
may overlap. In another condition, a classification problem that relies on both humidity and
rainfall can be collapsed into just one underlying feature, since both of the aforementioned are
correlated to a high degree. Hence, we can reduce the number of features in such problems. A 3-
D classification problem can be hard to visualize, whereas a 2-D one can be mapped to a simple
2-dimensional space, and a 1-D problem to a simple line. The below figure illustrates this
concept, where a 3-D feature space is split into two 2-D feature spaces, and later, if found to be
correlated, the number of features can be reduced even further.

48. Components of Dimensionality Reduction
There are two components of dimensionality reduction:
 Feature selection: In this, we try to find a subset of the original set of variables, or
features, to get a smaller subset which can be used to model the problem. It usually
involves three ways:
1. Filter
2. Wrapper
3. Embedded
 Feature extraction: This reduces the data in a high dimensional space to a lower
dimension space, i.e. a space with lesser no. of dimensions.
Methods of Dimensionality Reduction
The various methods used for dimensionality reduction include:
 Principal Component Analysis (PCA)
 Linear Discriminant Analysis (LDA)
 Generalized Discriminant Analysis (GDA)
Dimensionality reduction may be both linear and non-linear, depending upon the method used.
The prime linear method, called Principal Component Analysis, or PCA, is discussed below.
Principal Component Analysis
This method was introduced by Karl Pearson. It works on the condition that while the data in a
higher dimensional space is mapped to data in a lower dimension space, the variance of the data
in the lower dimensional space should be maximum.

49. It involves the following steps:
 Construct the covariance matrix of the data.
 Compute the eigenvectors of this matrix.
 Eigenvectors corresponding to the largest eigenvalues are used to reconstruct a large
fraction of variance of the original data.
Hence, we are left with a lesser number of eigenvectors, and there might have been some data
loss in the process. But, the most important variances should be retained by the remaining
eigenvectors.
Advantages of Dimensionality Reduction
 It helps in data compression, and hence reduced storage space.
 It reduces computation time.
 It also helps remove redundant features, if any.
 Improved Visualization: High dimensional data is difficult to visualize, and
dimensionality reduction techniques can help in visualizing the data in 2D or 3D, which
can help in better understanding and analysis.
 Overfitting Prevention: High dimensional data may lead to overfitting in machine
learning models, which can lead to poor generalization performance. Dimensionality
reduction can help in reducing the complexity of the data, and hence prevent overfitting.
 Feature Extraction: Dimensionality reduction can help in extracting important features
from high dimensional data, which can be useful in feature selection for machine learning
models.
 Data Preprocessing: Dimensionality reduction can be used as a preprocessing step before
applying machine learning algorithms to reduce the dimensionality of the data and hence
improve the performance of the model.
 Improved Performance: Dimensionality reduction can help in improving the performance
of machine learning models by reducing the complexity of the data, and hence reducing
the noise and irrelevant information in the data.
Disadvantages of Dimensionality Reduction
 It may lead to some amount of data loss.
 PCA tends to find linear correlations between variables, which is sometimes undesirable.

50.  PCA fails in cases where mean and covariance are not enough to define datasets.
 We may not know how many principal components to keep- in practice, some thumb
rules are applied.
 Interpretability: The reduced dimensions may not be easily interpretable, and it may be
difficult to understand the relationship between the original features and the reduced
dimensions.
 Overfitting: In some cases, dimensionality reduction may lead to overfitting, especially
when the number of components is chosen based on the training data.
 Sensitivity to outliers: Some dimensionality reduction techniques are sensitive to outliers,
which can result in a biased representation of the data.
 Computational complexity: Some dimensionality reduction techniques, such as manifold
learning, can be computationally intensive, especially when dealing with large datasets.
Important points:
 Dimensionality reduction is the process of reducing the number of features in a dataset
while retaining as much information as possible.
This can be done to reduce the complexity of a model, improve the performance of a
learning algorithm, or make it easier to visualize the data.
 Techniques for dimensionality reduction include: principal component analysis (PCA),
singular value decomposition (SVD), and linear discriminant analysis (LDA).
 Each technique projects the data onto a lower-dimensional space while preserving
important information.
 Dimensionality reduction is performed during pre-processing stage before building a
model to improve the performance
 It is important to note that dimensionality reduction can also discard useful information,
so care must be taken when applying these techniques.
Evaluating the performance of a dimensionality reduction algorithm on
your dataset
1. Data reconstruction error:
One way to measure how well a dimensionality reduction algorithm preserves the information in
the original data is to calculate the data reconstruction error. This is the difference between the
original data and the reconstructed data, which is obtained by applying the inverse
transformation of the dimensionality reduction algorithm. The lower the data reconstruction
error, the better the algorithm is at retaining the essential features of the data. You can use
various metrics to quantify the data reconstruction error, such as mean squared error, root mean
squared error, or mean absolute error.
2. Data compression ratio
Another way to evaluate a dimensionality reduction algorithm's performance is to look at the
data compression ratio. This is the ratio of the size of the original data to the size of the reduced
data. The higher the data compression ratio, the more efficient the algorithm is at reducing the
dimensionality of the data. However, a high data compression ratio does not necessarily mean a
high quality of data reconstruction. You need to balance the trade-off between data compression
and data reconstruction when choosing a dimensionality reduction algorithm.

51. 3. Data visualization quality
A third way to assess a dimensionality reduction algorithm's performance is to examine the data
visualization quality. This is especially relevant when you reduce the dimensionality of the data
to two or three dimensions, which can be easily plotted and visualized. The data visualization
quality reflects how well the reduced data captures the patterns, clusters, and outliers in the
original data. You can use various methods to evaluate the data visualization quality, such as
scatter plots, heat maps, or silhouette plots.
4. Data classification accuracy
A fourth way to gauge a dimensionality reduction algorithm's performance is to check the data
classification accuracy. This is applicable when you have a supervised learning problem, where
you want to predict the class labels of the data based on some features. The data classification
accuracy is the proportion of correctly predicted labels out of the total number of labels. You can
use different classifiers, such as logistic regression, k-nearest neighbors, or support vector
machines, to test the data classification accuracy. The higher the data classification accuracy, the
better the algorithm is at preserving the discriminative power of the data.
5. Algorithm complexity and scalability
A fifth way to evaluate a dimensionality reduction algorithm's performance is to consider the
algorithm complexity and scalability. This refers to how fast and how well the algorithm can
handle large and high-dimensional datasets. The algorithm complexity and scalability depend on
factors such as the computational cost, the memory usage, and the convergence rate of the
algorithm. You can use various metrics to measure the algorithm complexity and scalability,
such as time complexity, space complexity, or iteration complexity.
6. Algorithm suitability and robustness
A sixth way to judge a dimensionality reduction algorithm's performance is to assess the
algorithm suitability and robustness. This means how well the algorithm fits the characteristics
and the objectives of the data and the problem. The algorithm suitability and robustness depend
on aspects such as the data type, the data distribution, the data noise, and the data interpretation.
You can use various methods to test the algorithm suitability and robustness, such as cross-
validation, sensitivity analysis, or parameter tuning.
Main Approaches for Dimensionality Reduction
11. What are the main motivations for reducing a dataset’s dimensionality? What are the
main drawbacks? [7M] July – 2023 Set -2[Remember]
Reducing the dimensionality of a dataset refers to the process of reducing the number of features
or variables in the dataset while preserving as much relevant information as possible. This
process is often employed in machine learning and data analysis for various reasons, and it
comes with its own set of motivations and drawbacks:
Motivations for reducing dataset dimensionality:
1. Curse of Dimensionality: High-dimensional datasets often suffer from the curse of
dimensionality, where the data becomes sparse and computational complexity increases

52. exponentially with the number of dimensions. Reducing dimensionality can alleviate this
problem by making the dataset more manageable.
2. Computational Efficiency: Fewer dimensions generally mean less computational
resources are required for tasks such as training machine learning models, making
predictions, or conducting analyses.
3. Visualization: It's easier to visualize and interpret data in lower dimensions, especially
when dealing with 2D or 3D plots. Dimensionality reduction techniques like PCA
(Principal Component Analysis) or t-SNE (t-distributed Stochastic Neighbor Embedding)
are commonly used for visualization purposes.
4. Noise Reduction: By focusing on the most relevant features, dimensionality reduction
techniques can help in filtering out noisy or irrelevant information, improving model
performance.
5. Overfitting Mitigation: High-dimensional datasets are more prone to overfitting, where a
model learns noise in the data instead of the underlying patterns. Reducing
dimensionality can help in reducing overfitting by focusing on the most informative
features.
Drawbacks of reducing dataset dimensionality:
1. Information Loss: The primary drawback of dimensionality reduction is the potential loss
of information. Removing dimensions means discarding some variation in the data,
which can lead to a loss of important patterns or relationships.
2. Complexity and Interpretability: Dimensionality reduction techniques often introduce
complexity into the analysis, making it harder to interpret the transformed data.
Understanding the relationships between the original features and the reduced dimensions
can be challenging.
3. Algorithm Dependence: The effectiveness of dimensionality reduction techniques can
vary depending on the nature of the dataset and the specific algorithms used. What works
well for one dataset may not work as effectively for another.
4. Computational Cost of Transformation: While reducing dimensionality can lead to
computational efficiency in subsequent tasks, the process of dimensionality reduction
itself can be computationally expensive, particularly for large datasets.
5. Parameter Tuning: Many dimensionality reduction techniques involve hyperparameters
that need to be tuned, which can be time-consuming and require careful experimentation
to achieve optimal results.
Overall, while dimensionality reduction can offer significant benefits in terms of computational
efficiency, visualization, and noise reduction, it's essential to consider the trade-offs and potential
drawbacks, particularly regarding information loss and increased complexity.
12. Explain the process of reducing the dimension by using Manifold Learning. [7M]
July – 2023 Set -3[Understand]
The curse of dimensionality
A large number of machine learning datasets involve thousands and sometimes millions of
features. These features can make training very slow. In addition, there is plenty of space in high
dimensions making the high-dimensional datasets very sparse, as most of the training instances
are quite likely to be far from each other. This increases the risk of overfitting since the

53. predictions will be based on much larger extrapolations as compared to those on low-
dimensional data. This is called the curse of dimensionality.
There are two main approaches for dimensionality reduction: Projection and Manifold Learning.
Here, we will focus on the latter.
What is Manifold Learning?
What is a manifold?
A two-dimensional manifold is any 2-D shape that can be made to fit in a higher-dimensional
space by twisting or bending it, loosely speaking.
What is the Manifold Hypothesis?
“The Manifold Hypothesis states that real-world high-dimensional data lie on low-dimensional
manifolds embedded within the high-dimensional space.”
In simpler terms, it means that higher-dimensional data most of the time lies on a much closer
lower-dimensional manifold. The process of modeling the manifold on which training instances
lie is called Manifold Learning.
Locally Linear Embedding (LLE)
Locally Linear Embedding (LLE) is a Manifold Learning technique that is used for non-linear
dimensionality reduction. It is an unsupervised learning algorithm that produces low-dimensional
embeddings of high-dimensional inputs, relating each training instance to its closest neighbor.
How does LLE work?
For each training instance x(i)
, the algorithm first finds its k nearest neighbors and then tries to
express x(i)
as a linear function of them. In general, if there are m training instances in total, then
it tries to find the set of weights w which minimizes the squared distance between x(i)
and its
linear representation.
So, the cost function is given by
where wi,j =0, if j is not included in the k closest neighbors of i.
Also, it normalizes the weights for each training instance x(i)
,

54. Finally, each high-dimensional training instance x(i)
is mapped to a low-dimensional
(say, d dimensions) vector y(i)
while preserving the neighborhood relationships. This is done by
choosing d-dimensional coordinates which minimize the cost function,
Here the weights wi,j are kept fixed while we try to find the optimum coordinates y(i)
Implementation using scikit-learn
The following code is used to implement LLE using scikitlearn’s
LocallyLinearEmbedding class:
from sklearn.manifold import LocallyLinearEmbedding
lle= LocallyLinearEmbedding(n_neighbors=5, n_components=2)
#X_transformed=lle.fit_transform(X)
Now, we will compare LLE with PCA and t-SNE on the swiss roll dataset.
#Importing the required libraries
from collections import OrderedDict
from functools import partial
from time import time
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.ticker import NullFormatter
from sklearn import manifold, datasets
from sklearn.decomposition import PCA
#This import is needed to silence pyflakes
Axes3D
#Then we load the swiss roll dataset
n_points = 1000
X, color = datasets.make_swiss_roll(n_points, random_state=0)
n_neighbors = 10
n_components = 2
# Creating the plot
fig = plt.figure(figsize=(15, 8))
fig.suptitle("Manifold Learning with %i points, %i neighbors"
% (1000, n_neighbors), fontsize=14)
# Adding 3d scatter plot
ax = fig.add_subplot(231, projection='3d')
ax.scatter(X[:, 0], X[:, 1], X[:, 2], c=color, cmap=plt.cm.Spectral)
ax.view_init(4, -72)
# Making a dictionary 'methods' containing LLE, t-SNE and PCA
LLE = partial(manifold.LocallyLinearEmbedding,
n_neighbors, n_components, eigen_solver='auto')
methods = OrderedDict()

55. methods['LLE'] = LLE(method='standard')
methods['t-SNE'] = manifold.TSNE(n_components=n_components, init='pca',
random_state=0)
methods['PCA']=PCA(n_components=2)
# Plotting the results
for i, (label, method) in enumerate(methods.items()):
t0 = time()
Y = method.fit_transform(X)
t1 = time()
print("%s: %.2g sec" % (label, t1 - t0))
ax = fig.add_subplot(2, 3, 2 + i+(i>1))
ax.scatter(Y[:, 0], Y[:, 1], c=color, cmap=plt.cm.Spectral)
ax.set_title("%s (%.2g sec)" % (label, t1 - t0))
ax.xaxis.set_major_formatter(NullFormatter())
ax.yaxis.set_major_formatter(NullFormatter())
ax.axis('tight')
plt.show()
Output: We get the following plot:
PCA
13. Explain the concept of PCA for Compression. [7M] July – 2023 Set -4[Understand]
In today’s digital age, images are a critical element in any online platform or website. Images can
convey information, evoke emotions, and enhance user experience. However, images are also a
major contributor to website load times and storage requirements. Large images can slow down

56. page load times, which can lead to frustrated users and lower search engine rankings. That’s why
image compression is a crucial aspect of website optimization.
There are various image compression techniques available, ranging from lossy to lossless
compression. Lossy compression techniques like JPEG, PNG, and GIF use algorithms to discard
unnecessary data from an image file to reduce its size. However, these techniques have
limitations, including loss of image quality and difficulty in optimizing the compression
parameters.
Principal Component Analysis (PCA) is a promising technique for image compression that
addresses some of the limitations of other techniques. In this article, we’ll take a closer look at
how PCA works for image compression, its advantages and limitations, and how it compares to
other compression techniques.
What is Principal Component Analysis (PCA)?
PCA is a mathematical technique for transforming data from its original form into a new set of
variables known as principal components. These primary components are orthogonal to one
another, which means they are unrelated and independent.
The PCA process entails identifying the principal components that explain the greatest amount of
variation in the data while minimizing information loss. The first principal component accounts
for the most variance, followed by the second, and so on.
PCA has various applications in different fields, including finance, engineering, and computer
science. In image processing, PCA is used to reduce the dimensions of an image without losing
essential features.
The idea behind PCA for image compression is to represent an image as a linear combination of
its principal components, which reduces the image size and storage requirements. PCA can also
be used in conjunction with other compression techniques, such as Huffman encoding, to further
reduce the size of the compressed image.
How does PCA work for image compression?
The PCA process for image compression involves the following steps:
1. Convert the image to grayscale: To simplify the image’s complexity, convert it to
grayscale. This step is optional and depends on the application.
2. Split the image into smaller blocks: Divide the image into smaller blocks of equal size,
typically 8x8 or 16x16 pixels. The block size can vary depending on the application and
image content.
3. Flatten each block into a column vector: Reshape each block into a column vector of
pixel values.
4. Compute the covariance matrix: Calculate the covariance matrix of the column vectors.
The covariance matrix represents the relationship between the pixels in each block.
5. Compute the eigenvectors and eigenvalues: Find the eigenvectors and eigenvalues of the
covariance matrix. The eigenvectors represent the principal components of the image,
and the eigenvalues represent their corresponding variance.
6. Choose the number of principal components: Select the number of principal components
to retain based on the desired level of compression and image quality.
7. Encode the image: Encode the image by representing each block as a linear combination
of the retained principal components. This step requires multiplying the original block
vector by the selected eigenvectors.

57. 8. Compress the encoded image: Apply compression techniques such as Huffman encoding
to further reduce the size of the encoded image.
The resulting compressed image will have a reduced size compared to the original image while
retaining most of its essential features. The degree of compression and image quality depends on
the number of principal components selected.
Advantages of PCA for image compression
PCA has several advantages over other image compression techniques, including:
1. Efficient use of storage space: PCA compression can significantly reduce the storage
space required to store an image without significant loss of image quality.
2. Faster loading times: Smaller image sizes lead to faster page load times, which can
enhance user experience.
3. Improved website speed: Faster loading times can also improve website speed, which is a
crucial factor in search engine rankings.
4. Enhanced user experience: High-quality images are essential for providing an excellent
user experience, and PCA can preserve the essential features of an image while reducing
its size.
5. Increased SEO rankings: Faster loading times and improved website speed can lead to
higher search engine rankings, which is crucial for online visibility and traffic.
Limitations of PCA for image compression
PCA has some limitations that need to be considered, including:
1. Loss of image quality: Like other compression techniques, PCA compression can result
in a loss of image quality, depending on the number of principal components selected.
2. Sensitivity to image content: PCA works well for images with a smooth and continuous
structure, but it may not be as effective for images with sharp edges or high-frequency
content.
3. High computational requirements: The computation involved in PCA compression can be
intensive, especially for large images or high-resolution images.
4. Compression parameters optimization: Choosing the optimal number of principal
components and compression parameters can be challenging, and the process may require
trial and error.
5. Limited application to specific types of images: PCA is suitable for compressing certain
types of images, such as photographs, but may not be as effective for compressing other
types of images, such as logos or text.
How does PCA compare to other compression techniques?
PCA has several advantages over other compression techniques, such as JPEG and PNG, in
terms of compression efficiency, image quality, and speed. However, each technique has its
strengths and weaknesses, and the choice of technique depends on the specific application and
requirements.
JPEG compression is a lossy compression technique that uses the Discrete Cosine Transform
(DCT) to compress images. JPEG compression is suitable for images with smooth color
transitions and can achieve high compression ratios while maintaining acceptable image quality.

58. However, JPEG compression can result in image artifacts and blockiness, especially at low
bitrates.
PNG compression is a lossless compression technique that uses a combination of different
compression algorithms, including deflate, to compress images. PNG compression is suitable for
images with sharp edges and text and can achieve higher compression ratios than other lossless
compression techniques. However, PNG compression may not be as effective for images with
smooth color transitions.
14. Can PCA be used to reduce the dimensionality of a highly non-linear dataset?
Explain. [7M] July – 2023 Set -3[Understand]
 PCA is unsupervised machine learning technique(i.e. we don't have output/target in
dataset).
 It comes under Feature Extraction technique.
 PCA goal if to reduce Curse of Dimensionality problem or to reduce dimensions of data
or to select only important features/column from plethora of unwanted columns, can say.
keeping crux of data unhampered.
What is Dimension Reduction?
 It is a process of converting data set having vast dimensions into a data set with lesser
dimensions.
 It ensures that converted data set conveys similar information concisely.
Example:
 below graph shows two dimensions x1 and x2.
 x1 represents measurement in cm.
 x2 represents measurement in inches.

59. In ML:
 using both these dimensions convey similar information.
 Also, they introduce a lot of noise in system. So, it is better to use just one dimension.
Using dimension reduction techniques :
 convert dimensions of data from 2 dimensions (x1 and x2) to 1 dimension (z1).
 it makes data relatively easier to explain.
Steps involved in PCA Algorithm are as follows :
Step-01: Get data.
Step-02: Compute mean vector (µ).
Step-03: Subtract mean from given data.
Step-04: Calculate covariance matrix.
Step-05: Calculate eigen-vectors and eigen-values of covariance matrix.
Step-06: Choosing components and forming a feature vector.
Step-07: Deriving new data set.
Why do we need dimensionality reduction ?
 outcome is less misleading, means model accuracy improves.
 if we have fewer dimensions(less data) that will require less computing.
 less data means that our algorithms will train faster.
 if data will be less, storage space required will be less. (reduced time space complexity)
 dimensionality Reduction removes redundant features, noise and helps in visualising data
using 2D/3D plots.
When not to use dimensionality reduction ?
 if you don't want to loose some information, information loss will possibly degrading
performance of subsequent training algorithms.
 if you don't have computationally intensive machines.
 transformed features are often hard to interpret.

60.  it makes independent variables less interpretable.
What is Curse of Dimensionality ?
 curse of dimensionality problem arises while working with data in higher dimensions.
 as number of features increase, number of samples increases, hence model becomes more
complex.
 more number of features, more chances of overfitting.
 a machine learning model that is trained on a large number of features, gets increasingly
dependent on data it was trained on and in turn overfitted, resulting in poor performance
on real data, beating purpose.
 fewer features our training data has, lesser assumptions our model makes and simpler it
will be.
PCA can be used to reduce dimensionality of a highly non-linear dataset ?
 Yes, PCA can be used to significantly reduce dimensionality of highly non-linear dataset
because it can at least get rid of useless dimensions.
 If there are no useless dimensions, reducing dimensionality with PCA will lose too much
information.
What are some Limitations of PCA ?
 never work well for non-linearly correlated data.always considers low variance
components in data as noise and recommend to get rid off these low variance
components. In real life, sometimes these components play a major role in tasks like
supervised learning.
 if variables are correlated, PCA can achieve dimensionality reduction.
 if variables are not correlated, PCA just orders them according to their variances.
 PCA always finds orthogonal principal components. Sometimes, data demands non-
orthogonal principal components to represent it. Standard PCA fails to so.
Why orthogonal rotation is necessary in PCA ?
 orthogonal rotation account maximum variance of training data set.
 if components do not rotates , effect of PCA will diminish and one have to select more
number of components to explain variance in training data set.
Why to standardise data before applying PCA ?
 based on covariance matrix of original variables pca finds new directions, as covariance
matrix are sensitive to standardisation of variables. generally, people do standardisation
to assign equal weights to all variables.
 while using features of different scales, people gets misleading directions. But, it is not
necessary to standardise variables, if all variables are on same scale.
Need not to remove highly correlated variables before applying PCA ?
 NO, PCA shoots all highly correlated variables on same Principal Component-
(Eigenvector), not different ones, thats why not needed to remove highly correlated
variables
If all eigenvectors are same then PCA won’t be able to select principal components because in
that case, all principal components will be equal.
Way to evaluate performance of PCA on given dataset
 in general one can say dimensionality reduction algorithm is performing well if algorithm
like PCA eliminates a lot of dimensions from given dataset without losing too much
information.

61.  when using dimensionality reduction as a preprocessing step before ML algorithm, than
one can simply measure performance of that second algorithm, if dimensionality
reduction did not lose too much information, then algorithm should perform just as well
as when using original dataset.
* Principal Component Analysis is a well-known dimension reduction techniqu
* It transforms variables into a new set of variables called as principal components
* These principal components are linear combination of original variables and are orthogonal
* First principal component accounts for most of possible variation of original data
* Second principal component does its best to capture variance in data
* There can be only two principal components for a two-dimensional data set.e
Higher dimensional data is not good for any ML model, to protect our model we use different
dimensionality reduction techniques.
Dimensionality Reduction:
It can be done by two ways:
1. Feature Selection: go with either 1. Forward Feature Selection or 2. Backword Feature
Elimination
* Always select feature which have more variance and remove which have less
* If variance of features are same i.e. features shows linear or kind of linear relationship
Than Feature Selection will not work
* Feature Extraction will come into rescue, now apply PCA kind of techniques
PCA will make new set of features from existing features and will select new subset from new
set of features, which according to PCA is most important.
Feature Extraction: go with either 1. PCA 2. LDA or 3. tSne
Using Scikit- Learn
15. What is Scikit Learn and What it is used for? What is the difference between Scikit
Learn and sklearn? [7M] [Remember]
scikit-learn is definitely one of the most commonly used packages when it comes to Machine
Learning and Python. However, a lot of newcomers get confused about the naming of the package
itself due to the fact that it looks to appear with two distinct names; scikit-learn and sklearn.
In this, we will discuss whether there’s any difference between the two packages in the first place.
Additionally, we’ll discuss whether it matters which one you install and import in your source
code.

62. What is scikit-learn
The project was originally started back in 2007 as part of the Google Summer of Code while the
first public release was made in early 2010.
scikit-learn is an open source Machine Learning Python package that offers functionality
supporting supervised and unsupervised learning. Additionally, it provides tools for model
development, selection and evaluation as well as many other utilities including data pre-
processing functionality.
More specifically, scikit-learn’s main functionality includes classification, regression, clustering,
dimensionality reduction, model selection and pre-processing. sThe library is very simple to use
and most importantly efficient as it is built on NumPy, SciPy and matplotlib.
Is there any difference between scikit-learn and sklearn?
The short answer is no. scikit-learn and sklearn both refer to the same package however, there are
a couple of things you need to be aware of.
Firstly, you can install the package by using either of scikit-learn or sklearn identifiers however, it
is recommended to install scikit-learn through pip using the skikit-learn identifier.
If you install the package using the sklearn identifier and then run pip list you will notice the
annoying sklearn 0.0 entry:
$ pip install sklearn
$ pip list
Package Version
------------- -------
joblib 1.0.1
numpy 1.21.2
pip 19.2.3
scikit-learn 0.24.2
scipy 1.7.1
setuptools 41.2.0
sklearn 0.0
threadpoolctl 2.2.0
Additionally, if you now attempt to uninstall sklearn, the package won’t be uninstalled:

63. $ pip uninstall sklearn
$ pip list
Package Version
------------- -------
joblib 1.0.1
numpy 1.21.2
pip 19.2.3
scikit-learn 0.24.2
scipy 1.7.1
setuptools 41.2.0
threadpoolctl 2.2.0
Essentially, sklearn is a dummy project on PyPi that will in turn install scikit-learn. Therefore, if
you uninstall sklearn you are just uninstalling the dummy package, and not the actual package
itself.
Now despite how you installed scikit-learn, you must import it in your code using
the sklearn identifier:
import sklearn
If you attempt to import the package using the scikit-learn identifier, you will end up with
a SyntaxError:
>>> import sklearn
>>> import scikit-learn
File "<stdin>", line 1
import scikit-learn
^
SyntaxError: invalid syntax
Even if you try to import it with __import__() in order to deal with the hyphen in the package’s
name, you will still get a ModuleNotFoundError:
>>> __import__('scikit-learn')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'scikit-learn'
In general, you are advised to install the library using the scikit-learn identifier (i.e. pip install
scikit-learn) but in your source code, you must import it using the sklearn identifier (i.e. import
sklearn).

64. Randomized PCA, Kernel PCA
16. In what cases would you use Incremental PCA, Randomized PCA & Kernel PCA?
Explain. [7M] July – 2023 Set -2 [Understand]
Python – Variations of Principal Component Analysis
Principal Component Analysis (PCA) is an unsupervised dimensionality reduction and
visualization technique. It is often referred to as a linear technique because the mapping of new
features is given by the multiplication of features by the matrix of PCA eigenvectors. It works by
simply identifying the hyperplane that lies close to the data and then projecting the data onto it in
order to maximize the variance. Due to the simplistic approach PCA follows, it is widely used in
data mining, bioinformatics, psychology, etc. Most of us are unaware of the fact that there are
various versions of this algorithm out there which are better than the conventional approach.
Let’s look at them one by one.
Principal Component Analysis (PCA) is a technique for dimensionality reduction and feature
extraction that is commonly used in machine learning and data analysis. It is implemented in
many programming languages, including Python.
There are several variations of PCA that have been developed to address specific challenges or
improve performance. Some of the variations of PCA in Python include:
Kernel PCA: This variation of PCA uses a kernel trick to transform the data into a higher-
dimensional space where it is more easily linearly separable. This can be useful for handling
non-linearly separable data.
Incremental PCA: This variation of PCA allows for the analysis of large datasets that cannot be
fit into memory all at once. It is useful for handling big data problems.
Sparse PCA: This variation of PCA adds a sparsity constraint to the PCA problem, which
encourages the algorithm to find a lower-dimensional representation of the data with fewer non-
zero components.
Robust PCA: This variation of PCA is designed to handle datasets with outliers or noise. It
separates the data into a low-rank component and a sparse component, where the sparse
component represents the outliers or noise.
Non-negative Matrix Factorization (NMF): This variation of PCA is used to factorize a non-
negative matrix into two non-negative matrices, where the resulting matrices have non-negative
elements.
PCA with L1-Regularization: This variation of PCA adds L1 regularization term to the PCA
optimization problem. This can be useful for handling high-dimensional datasets with many
correlated features.
Randomized PCA:
This is an extension to PCA which uses approximated Singular Value Decomposition(SVD) of

65. data. Conventional PCA works in O(n*p2
) + O(p3
) where n is the number of data points and p is
the number of features whereas randomized version works in O(n*d*2) + O(d3
) where d is the
number of principal components. Thus, it is blazing fast when d is much smaller than n.
sklearn provides a method randomized_svd in sklearn.utils.extmath which can be used to do
randomized PCA. This method returns three matrices: U which is an m x m matrix, Sigma is
an m x n diagonal matrix, and V^T is the transpose of an n x n matrix where T is a superscript.
Another way to use sklearn.decomposition.PCA and change the svd_solver hyperparameter from
‘auto’ to ‘randomized’ or ‘full’. However, Scikit-learn automatically uses randomized PCA if
either p or n exceeds 500 or the number of principal components is less than 80% of p and n.
Randomized PCA is a variation of Principal Component Analysis (PCA) that is designed to
approximate the first k principal components of a large dataset efficiently. Instead of computing
the eigenvectors of the covariance matrix of the data, as is done in traditional PCA, randomized
PCA uses a random projection matrix to map the data to a lower-dimensional subspace. The first
k principal components of the data can then be approximated by computing the eigenvectors of
the covariance matrix of the projected data.
Randomized PCA has several advantages over traditional PCA:
1. Scalability: Randomized PCA can handle large datasets that are not possible to fit into
memory using traditional PCA.
2. Speed: Randomized PCA is much faster than traditional PCA for large datasets, making
it more suitable for real-time applications.
3. Sparsity: Randomized PCA is able to handle sparse datasets, which traditional PCA is
not able to handle well.
4. Low-rank approximation: Randomized PCA can be used to obtain a low-rank
approximation of a large dataset, which can then be used for further analysis or
visualization.
Code:
# Python3 program to show the working of
# randomized PCA
# importing libraries
import numpy as np
from sklearn.decomposition import PCA
from sklearn.utils.extmath import randomized_svd
# dummy data
X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
# creates instance of PCA with randomized svd_solver
pca = PCA(n_components = 2, svd_solver ='randomized')

66. # This function takes a matrix and returns the
# U, Sigma and V ^ T elements
U, S, VT = randomized_svd(X, n_components = 2)
# matrix returned by randomized_svd
print(f"Matrix U of size m * m: {U}n")
print(f"Matrix S of size m * n: {S}n")
print(f"Matrix V ^ T of size n * n: {VT}n")
# fitting the pca model
pca.fit(X)
# printing the explained variance ratio
print("Explained Variance using PCA with randomized svd_solver:",
pca.explained_variance_ratio_)
Incremental PCA:
The major problem with PCA and most of the dimensionality reduction algorithms is that they
require whole data to fit in the memory at a single time and as the data is very huge at times thus
it becomes very difficult to fit in memory.
Fortunately, there is an algorithm called Incremental PCA which is useful for large training
datasets as it splits the data into min-batches and feeds it to Incremental PCA one batch at a time.
This is called as on-the-fly learning. As not much data is present in the memory at a time thus
memory usage is controlled.

67. Scikit-Learn provides us with a class called as sklearn.decomposition.IncrementalPCA using
which we can implement this.
Code:
# Python3 program to show the working of
# incremental PCA
# importing libraries
import numpy as np
from sklearn.decomposition import IncrementalPCA
# dummy data
X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
# specify the number of batches
no_of_batches = 3
# create an instance of IncrementalPCA
incremental_pca = IncrementalPCA(n_components = 2)
# fit the data in batches
for batch in np.array_split(X, no_of_batches):
incremental_pca.fit(batch)
# fit and transform the data
final = incremental_pca.transform(X)
# prints an 2d-array (as n_components = 2)
print(final)
Kernel PCA:
Kernel PCA is yet another extension of PCA using a kernel. The kernel is a mathematical
technique using which we can map instances to very high dimensional space called the feature
space, enabling non-linear classification and regression with Support Vector Machines(SVM).

68. This is usually employed in novelty detections and image de-noising.
Scikit-Learn provides a class KernelPCA in sklearn.decomposition which can be used to perform
Kernel PCA.
Code:
# Python3 program to show the working of
# Kernel PCA
# importing libraries
import numpy as np
from sklearn.decomposition import KernelPCA
# dummy data
X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
# creating an instance of KernelPCA using rbf kernel
kernel_pca = KernelPCA(n_components = 2, kernel ="rbf", gamma = 0.03)
# fit and transform the data
final = kernel_pca.fit_transform(X)
# prints an 2d-array (as n_components = 2)
print(final)
KernelPCA is unsupervised thus there is no obvious measure to select the best kernel. As we
usually use dimensionality reduction as a step in supervised learning algorithms so we can use a
pipeline with GridSearchCV for selecting optimal hyperparameters and then using those
hyperparameters (kernel and gamma) to get the best classification accuracy.
Principal Component Analysis (PCA) is a technique widely used for dimensionality reduction
and feature extraction. However, there are variations of PCA tailored for specific scenarios:
1. Incremental PCA:
 Use Cases: Incremental PCA is useful when dealing with large datasets that
cannot fit into memory all at once. It processes data in small batches or chunks,

69. making it feasible to handle datasets that are too large to be processed in a single
step.
 Explanation: Instead of computing the covariance matrix of the entire dataset,
Incremental PCA computes it incrementally, updating the eigenvectors and
eigenvalues as new data batches are processed. This allows for memory-efficient
processing of large datasets by breaking down the computation into manageable
steps.
2. Randomized PCA:
 Use Cases: Randomized PCA is suitable for large datasets where the
computational cost of the standard PCA becomes prohibitive. It can significantly
speed up the computation of PCA while maintaining a good approximation of the
principal components.
 Explanation: Randomized PCA approximates the principal components by using
random projections. By employing randomization techniques, it reduces the
computational complexity of the PCA algorithm, making it more efficient for
large datasets. It provides an approximate solution to PCA with much faster
computation times compared to the traditional method, albeit sacrificing some
precision.
3. Kernel PCA:
 Use Cases: Kernel PCA is beneficial when dealing with nonlinear relationships
within the data. It is particularly useful for dimensionality reduction in cases
where linear methods like traditional PCA may not capture the underlying
structure effectively.
 Explanation: Kernel PCA applies the kernel trick, which implicitly maps the data
into a higher-dimensional space where nonlinear relationships can be captured
more effectively. By using a kernel function, such as the Gaussian kernel (RBF
kernel) or polynomial kernel, it computes the principal components in this
transformed space. Kernel PCA is powerful for tasks like manifold learning,
where the data lies on a nonlinear manifold, or when dealing with datasets with
complex nonlinear relationships.
In summary, Incremental PCA is suitable for handling large datasets in a memory-efficient
manner, Randomized PCA is effective for speeding up PCA computation on large datasets while
maintaining good approximation, and Kernel PCA is advantageous when dealing with nonlinear
relationships in the data that cannot be captured by linear methods like traditional PCA. Each
variant of PCA is tailored to address specific challenges and requirements in different scenarios.
Kernel PCA
17. Explain about Kernel PCA in detail. [7M] July – 2023 Set -1[Understand]
PRINCIPAL COMPONENT ANALYSIS: is a tool which is used to reduce the dimension of
the data. It allows us to reduce the dimension of the data without much loss of information. PCA
reduces the dimension by finding a few orthogonal linear combinations (principal components)
of the original variables with the largest variance. The first principal component captures most of
the variance in the data. The second principal component is orthogonal to the first principal
component and captures the remaining variance, which is left of first principal component and so

70. on. There are as many principal components as the number of original variables. These principal
components are uncorrelated and are ordered in such a way that the first several principal
components explain most of the variance of the original data.
KERNEL PCA: PCA is a linear method. That is it can only be applied to datasets which are
linearly separable. It does an excellent job for datasets, which are linearly separable. But, if we
use it to non-linear datasets, we might get a result which may not be the optimal dimensionality
reduction. Kernel PCA uses a kernel function to project dataset into a higher dimensional feature
space, where it is linearly separable. It is similar to the idea of Support Vector Machines. There
are various kernel methods like linear, polynomial, and gaussian.
Kernel Principal Component Analysis (KPCA) is a technique used in machine learning for
nonlinear dimensionality reduction. It is an extension of the classical Principal Component
Analysis (PCA) algorithm, which is a linear method that identifies the most significant features
or components of a dataset. KPCA applies a nonlinear mapping function to the data before
applying PCA, allowing it to capture more complex and nonlinear relationships between the data
points.
In KPCA, a kernel function is used to map the input data to a high-dimensional feature space,
where the nonlinear relationships between the data points can be more easily captured by linear
methods such as PCA. The principal components of the transformed data are then computed,
which can be used for tasks such as data visualization, clustering, or classification.
One of the advantages of KPCA over traditional PCA is that it can handle nonlinear relationships
between the input features, which can be useful for tasks such as image or speech recognition.
KPCA can also handle high-dimensional datasets with many features by reducing the
dimensionality of the data while preserving the most important information.
However, KPCA has some limitations, such as the need to choose an appropriate kernel function
and its corresponding parameters, which can be difficult and time-consuming. KPCA can also be
computationally expensive for large datasets, as it requires the computation of the kernel matrix
for all pairs of data points.
Code: Create a dataset that is nonlinear and then apply PCA to the dataset.
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=500, noise=0.02, random_state=417)
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.show()

71. Code: Let’s apply PCA to this dataset.
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)
plt.title("PCA")
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y)
plt.xlabel("Component 1")
plt.ylabel("Component 2")
plt.show()
As you can see PCA failed to distinguish the two classes.
Code: Applying kernel PCA on this dataset with RBF kernel with a gamma value of 15.

72. from sklearn.decomposition import KernelPCA
kpca = KernelPCA(kernel='rbf', gamma=15)
X_kpca = kpca.fit_transform(X)
plt.title("Kernel PCA")
plt.scatter(X_kpca[:, 0], X_kpca[:, 1], c=y)
plt.show()
In the kernel space the two classes are linearly separable. Kernel PCA uses a kernel function to
project the dataset into a higher-dimensional space, where it is linearly separable. Finally, we
applied the kernel PCA to a non-linear dataset using scikit-learn.
Kernel Principal Component Analysis (PCA) is a technique for dimensionality reduction in
machine learning that uses the concept of kernel functions to transform the data into a high-
dimensional feature space. In traditional PCA, the data is transformed into a lower-dimensional
space by finding the principal components of the covariance matrix of the data. In kernel PCA,
the data is transformed into a high-dimensional feature space using a non-linear mapping
function, called a kernel function, and then the principal components are found in this high-
dimensional space.
Advantages of Kernel PCA:
1. Non-linearity: Kernel PCA can capture non-linear patterns in the data that are not
possible with traditional linear PCA.
2. Robustness: Kernel PCA can be more robust to outliers and noise in the data, as it
considers the global structure of the data, rather than just local distances between data
points.
3. Versatility: Different types of kernel functions can be used in kernel PCA to suit different
types of data and different objectives.
4. Kernel PCA can handle nonlinear relationships between the input features, allowing for
more accurate dimensionality reduction and feature extraction compared to traditional
linear PCA.
5. It can preserve the most important information in high-dimensional datasets while
reducing the dimensionality of the data, making it easier to visualize and analyze.

73. 6. Kernel PCA can be used for a variety of tasks, including data visualization, clustering,
and classification.
7. It is a well-established and widely used technique in machine learning, with many
available libraries and resources for implementation.
Disadvantages of Kernel PCA:
1. Complexity: Kernel PCA can be computationally expensive, especially for large datasets,
as it requires the calculation of eigenvectors and eigenvalues.
2. Model selection: Choosing the right kernel function and the right number of components
can be challenging and may require expert knowledge or trial and error
3. Choosing an appropriate kernel function and its parameters can be challenging and may
require expert knowledge or extensive experimentation.
4. Kernel PCA can be computationally expensive, especially for large datasets, as it requires
the computation of the kernel matrix for all pairs of data points.
5. It may not always be easy to interpret the results of kernel PCA, as the transformed data
may not have a clear interpretation in the original feature space.
6. Kernel PCA is not suitable for datasets with many missing values or outliers, as it
assumes a complete and consistent dataset.