The document covers the fundamentals of exploratory data analysis (EDA) and its significance in data science, highlighting the phases of data analysis, types of data, and measurement scales. It emphasizes the importance of EDA in deriving insights from data and discusses various software tools and visualization techniques used in the process. Additionally, the document details methods for data transformation and descriptive statistics, along with different ways to visualize data effectively.

1. Data Exploratory, Feature
Engineering and Visualization
Dr.M.Shanthi,
ADS
ODD SEM-2024-2025

2. Unit-I
EDA Fundamentals-Understanding Data Science
– Significance of EDA – Making Sense of Data-
Comparing EDA with Classical and Bayesian
Analysis- Software Tools available for EDA.

3. Understanding Data Science
• Data Science: Scientific Study of Data.
• Data science involves cross-disciplinary knowledge from computer science,
statistics and mathematics.
• Data Analysis Phases:
1. Data Requirements
2. Data Collection
3. Data Processing
4. Data cleaning
5. EDA  Transformation, Descriptive Statistics.
6. Modeling and Algorithm
7. Data Product
8. Communication – Data Visualization

4. The Significance of EDA
- Different fields (science, economics, engineering, and
marketing) accumulate and store the data in electronic
formats.
- Appropriate and well established decision should be made
using the data collected.
- Impossible to take decisions from datasets without the help
computer programs.
- Data mining  data insights & make further decisions.
- Exploratory Data Analysis is a first Exercise in data mining.
- Visualize the data  to understand , create hypotheses for
further analysis.

5. The Significance of EDA
• EDA reveals ground truth of the content 
without making any underlying assumptions.
• Scientists uses (EDA)  what type of
modeling and hypothesis can be created.
• EDA
- summarizing data,[pandas]
- statistical data,[scipy]
- visualization of the data. [matplotlib]

6. Steps in EDA
• Problem Definition
- Define the business problem
- Data analysis plan execution
- Main deliverables
- Obtaining the current status of the data
- Performing cost/benefit Analysis
• Data Preparation
- Sources of the data
-Define data schemas and tables
-Main characteristics of the data
-Clean the dataset
- Delete the non-relevant datasets
- Transform the data
- Divide the data into required chunks for analysis
• Data Analysis
- Summarizing the data
- finding the hidden correlation and relationships among the data
• Development and representation of the results.
- Graphs
- Summary Tables
- Plots

7. Making sense of Data
Type of data analysis?
1. Numerical data[Quantitative data]
 Discrete data (fixed and distinct values)
Ex: Country code variable
Rank for students
 Continuous data
Infinite number of numerical values within a
specific range

8. Making Sense of Data
2. Categorical data[Qualitative data]
Categorical data represents the characteristics of an object.
Example:
Gender
Marital status
Movie Genres
Blood Type
Types of drugs
Types:
Binary categorical variable can take exactly two values anyone will be
selected. Dichotomous variable.
Polytomous variable can take more than two possible values. (Marital status)

9. Measurement Scales
• Most of the categorical dataset follows either
nominal or ordinal measurement scales.
- Nominal
- Ordinal
- Interval
- ratio

10. Measurement Scales
Nominal
• labeling variables without any quantitative value.
• The scales are generally referred to as labels.
• Scales are mutually exclusive / do not carry numerical values.
Example:
• What is the gender?
Male, Female, Third gender/Non-binary
I prefer not to answer, Other
• The languages that are spoken in a particular country
Tamil, Telugu, Malayalam etc.
• Biological Species
• Parts Speech in grammar
Important Note: Someone uses numbers for labels in the nominal measurement sense, they
have no concrete numerical value or meaning.
 No form of arithmetic calculation can be made on nominal measures.

11. Measurement Scales
In case of a Nominal dataset, you can certainly know the following:
Frequency rate at which a label occurs over a period of time
within the dataset.
Proportion Dividing the frequency by the total number of events
Percentage  compute the percentage of each proportion
Visualize  Pie chart or Bar Chart
Nominal scale: Pie chart or Bar Chart
Important note:
Type of data  Computation Type of model  Type of
visualization

12. Measurement Scales
Ordinal
- Difference between Ordinal and Nominal scale is the
order.
- Order of the values is significant factor.
- Represented by Likert scale.
Diagram need to be attached:
- Ordinal scales as an order of ranking.
- Median  measure of central tendency.
- Average is not permitted.

13. Measurement Scales
Interval
• The order and exact differences between the
values are significant.
• Used in statistics
• Measure of central tendencies i.e.
mean,median,mode and standard deviations.
• Example : Temperature.

14. Measurement Scales
Ratio:
The order, exact values and absolute zero
Possible to apply descriptive and inferential statistics.
 Central tendencies, Measure of dispersion(scattering the data/distribution)
 coefficient variation(ratio of measure of dispersion around the mean).
Examples:
- Dose amount
- Reaction rate
- Flow rate
- Concentration
- Pulse
- Weight
- Length

15. Measurement Scales

16. Comparing EDA with Classical and Bayesian
Analysis

17. Software tools available for EDA
• Python
• R programming Language
• Weka
• KNIME

18. Visual Aids for EDA
• Line Chart
• Bar Chart
• Scatter Plot
• Area Plot and stacked plot
• Pie Chart
• Table chart
• Polar Chart
• Histogram
• Lollipop Chart
• Choosing the best Chart
• Other Libraries to explore

19. Line Chart
• Line chart is used to illustrate the relationship
between two or more continuous variable.
• Matplotlib library
• Example:
- Date vs Stock_price

20. Lollipop chart
• A Lollipop chart can be used to display ranking
in the data.
• It is similar to an ordered bar chart.
• The line and the circle on the top gives nice
illustration of different types of cars and their
associated miles.

21. Bar Chart
• Bar charts are frequently used.
• To distinguish objects between distinct
collections in order to track variations over
time.
• Bars can be drawn horizontally or vertically to
represent the categorical variables.
• Example: Pharmacy in Norway keeps track of
the amount of Zoloft sold every month.

22. Table Chart
• A table chart combines a bar chart and a table.
• Example: Consider the standard LED bulbs that
come in different wattages.
• Based on two categorical variables: The year
and wattage. The number of units sold in a
particular year.

23. Histogram
• Histogram plots are used to depict the
distribution of any continuous variable.
• These types of plots are very popular in statistical
analysis.
• To find out the distribution we can go with
histogram plot.
• Example: Frequency vs years of experience with
python programming.

24. Scatter Plot
• Scatter plots are also called scatter graphs,
scatter charts.
• Cartesian co-ordinates x,y.

25. Cartesian Co-Ordinates

26. Polar Co-ordinates

27. Polar Chart

28. Data Transformation
• Merging database-style dataframes
• Transformation techniques
• Benefits of data transformation

29. Data transformation
• Concat
• Concat with an axis
• Merge
inner join
outer join
left join
right join
index
• Reshaping and pivoting
stacking
unstacking

30. Transformation Techniques
• Data Duplication
• Replacing values
• Handling missing data

31. Transformation Techniques
• Dropping the Missing Values
– Row-wise
– Column-wise
– Based on threshold

32. Transformation Techniques
• Filling the Missing Values
- Fill by zero value
- Fill by Forward/Backward Filling
- Fill by interpolating method

33. Descriptive Statistics
• Simple summaries of the entire dataset.
Central Tendencies
Mean
Median
Mode

34. Descriptive Statistics
Mean/Average might not be the best representation of the dataset ?
Measure of Dispersion
1. Standard Deviation
2. Variance
3. Skewness ( Measure of Symmetry and Asymmetry Variable)
Positive Skewness
Symmetrical
Negative Skewness
4. Kurtosis (Heaviness of the tail of the distribution)
( 0 ) Mesokurtic
(+3) Leptokurtic
(-1) Platykurtic
5. Percentile ( Measure the percentage of values in any dataset that lie below a
certain value)
25%
50%
75%
100%
6. Quartiles
- Visualization of Quartiles

35. Skewness
• Asymmetry of the variable in the dataset
about its mean.
• Positive
• Negative
• Symmetrical

36. Skewness
function: df.skew()

37. Kurtosis
Function= df.kurt()
• Kurtosis is a statistical measure that illustrates
how heavily the tails of distribution differ
from those of a normal distribution.
• Identify whether a given distribution contains
extreme values.
• Measure of outlier presence in a given
distribution.
• High kurtosis  high Outliers.

38. Kurtosis

39. Kurtosis
• There are three types of Kurtosis:
Mesokurtic  0
Leptokurtic  (K>3) High Flat  High
Outliers
Platykurtic (K<0) Low Outliers

40. Percentile
Function = np.percentile(attribute,50)
• Measure the percentage of values in any
dataset that lie below a certain value.

41. Quartiles
• Quartiles are values that split the given
dataset into quarters.

42. Grouping Datasets
• Groupby Mechanisms
- Grouping by features, hierarchically
- Aggregating a dataset by groups
- Applying custom aggregation functions to
groups
- Transforming a dataset groupwise

43. Grouping the Datasets
• Selecting a subset of columns
• Max and Min
• Mean