ml

1. Data Quality and Preprocessing:
Essential Concepts for
Engineering Students
Welcome, third-year engineering students! Today, we'll delve into the critical realm of
data quality and preprocessing. In an increasingly data-driven world, understanding how
to handle and refine raw data is paramount for any successful engineering project.

2. Chapter 1
Why Data Quality
Matters in Engineering
In the world of engineering, data is the bedrock of innovation and
decision-making. Whether you're designing a new system, optimising a
process, or predicting system failures, the reliability of your outputs hinges
entirely on the quality of your input data.
Poor data quality leads to inaccurate models, wrong decisions,
and costly errors.
Data preprocessing improves reliability by cleaning and
preparing data for analysis.
Engineers spend up to 80% of their time fixing data issues
before modelling.

3. The Five Data Quality Challenges
Problem 1: Missing Values
What? Data points absent from a dataset (e.g., sensor failure, incomplete
surveys or unexpected power outages). These gaps can severely impact the
integrity of your analysis.
Example: Imagine a complex industrial system, like a chemical reactor. A
critical temperature sensor might miss readings for several hours due to a
faulty connection or system glitch. This creates a gap in the time-series
data crucial for process control.
Impact: Many algorithms cannot directly handle gaps in data, leading to
biased or incomplete analysis. It can obscure trends and compromise the
accuracy of predictive models, potentially causing system inefficiencies or
safety hazards.
Solutions
• Remove records with missing data (only if the proportion is very
small).
• Impute missing values using statistical methods (mean, median,
mode) or more advanced predictive models (e.g., regression, k-
NN imputation).
• Utilise domain knowledge to estimate missing entries, especially
in engineering contexts where physical laws or operational
limits can guide reasonable approximations.

4. Problem 2: Redundancy (Duplicate Data)
What? Redundancy refers to repeated or duplicated records within
datasets. This isn't just about exact copies, but also highly similar
entries that represent the same real-world entity.
Example: In a large manufacturing plant's inventory system, the same
spare part might be logged multiple times under slightly different
descriptions or part numbers due to manual entry errors or integration
issues from different departments (e.g., "Gearbox Assembly-Type A"
and "Assy, Gearbox, Type A").
Impact: Duplicate data inflates dataset size, distorts statistical
measures (e.g., counting unique parts), and wastes storage and
processing resources. It can also lead to inconsistent reporting and
faulty supply chain management decisions.
Solutions
Identify duplicates using unique identifiers (if
available) or advanced record linkage techniques
that match records based on similarity across
multiple attributes.
Carefully remove or merge duplicate records,
prioritising the most complete or accurate entry while
preserving critical information from all original
entries.

5. Problem 3: Inconsistency
Inconsistency arises when data points conflict or contradict each other across different records or data sources, making it challenging to establish a single source of truth.
What? Conflicting or contradictory data values for the same entity, often arising from varied data entry methods, lack of standardization, or
integration of disparate systems.
Example: Consider a national infrastructure project database where road segment lengths are recorded in metres in one system and kilometres in
another without proper conversion. Or, a bridge's "last inspection date" is recorded as "2023-01-15" in the maintenance log but "15/01/23" in the
digital asset management system, with another entry stating "January 15, 2022" for the same inspection.
Impact: Inconsistent data confuses algorithms, especially those relying on precise numerical comparisons or chronological order. It reduces the
trustworthiness of analytical results, leading to flawed project planning, inaccurate asset valuation, or misallocation of resources.
1 2 3

6. Problem 4: Noise
What? Noise refers to random errors, irrelevant information, or distortion present in data. It's usually a result of measurement
inaccuracies, data transmission errors, or extraneous factors not relevant to the core phenomenon being measured.
Example: In real-time monitoring of a vibration sensor on a bridge, sudden erroneous spikes in readings might occur due to transient
electrical interference from nearby construction equipment or electromagnetic disturbances, even though the bridge's structural
integrity remains unchanged.
Impact: Noisy data can mask true underlying patterns, trends, and relationships, making it difficult for algorithms to learn effectively. It
reduces model accuracy, leads to false alarms in monitoring systems, and can result in incorrect diagnoses or premature maintenance
actions in engineering applications.

7. Problem 5: Outliers
Outliers are data points that significantly deviate from other observations. While sometimes noise, they can also represent genuine, extreme events that require careful consideration.
What? Data points significantly different from others in a dataset. They can be valid but unusual observations, or they can be
errors (which might also be considered noise).
Example: In a quality control process for manufacturing circuit boards, a machine might consistently operate between 20-80°C.
However, a single reading suddenly shows 500°C. This extreme value is an outlier that could indicate a sensor malfunction, a
temporary system fault, or even a critical overheating event.
Impact: Outliers can severely distort statistical measures such as mean and standard deviation, leading to misinterpretations of
data distributions. During model training, they can pull the model away from the true underlying patterns, resulting in poor
predictive performance or biased results.
Detection
Detect outliers using statistical tests (e.g., Z-scores, IQR method) or visualization
Decision
Decide whether to remove, transform (e.g., log transformation to reduce skew), or
Robust Algorithms
Use robust algorithms that are less sensitive to outliers, such as median-based

8. Summary Table: Problems, Examples & Solutions
This table provides a concise overview of the data quality challenges we've discussed, along with their key characteristics and common solutions.
Missing Values Sensor data gaps in environmental monitoring. Imputation, removal (if minor), domain-specific
estimation.
Redundancy Duplicate equipment IDs in an asset register. Deduplication, sophisticated record linkage.
Inconsistency Conflicting material properties from different tests. Standardisation, data fusion, rigorous validation rules.
Noise Erroneous spikes in vibration sensor readings. Smoothing techniques, filtering, regression for noise
reduction.
Outliers Extreme stress test result outside expected bounds. Statistical detection, transformation, use of robust
algorithms.

9. Chapter 2
Why Preprocessing is Critical for
Engineering Success
Data preprocessing isn't just a technical step; it's a foundational practice that underpins all successful data-driven
engineering endeavors. It’s the difference between a robust, reliable system and one prone to failure.

10. Takeaway: Mastering Data Quality is Key to Engineering
Excellence
As future engineers, your ability to handle data effectively will define your success in an increasingly complex and interconnected world.
Always inspect and clean your data before analysis.
Never assume raw data is perfect. Always start with a thorough data
quality assessment.
Use appropriate techniques tailored to each data
quality problem.
There's no one-size-fits-all solution. Choose methods wisely based on
the type and context of the data issue.
Remember: “Garbage in, garbage out” — quality data
leads to quality results.
The performance of your models and the reliability of your decisions are
directly tied to the integrity of your data.
Start building your data preprocessing skills today
for future-ready engineering.
Embrace these techniques as fundamental tools in your engineering
toolkit.
Thank you!