Name Topping PizzaSize Number of Slices
Alice Pineapple Large 8
Bob Pepperoni Medium -2
Charlie Cheese Extra Large 12
Daniela Pepperoni Small 16
Eduardo "Just veggies!" Small 4
Frank Pepperoni ??? 8
Georgia 123 Medium 8
Spot the Dirty Data
Your first mission is to spot the 'dirty data' in a fun, real-life scenario!
Instructions
Imagine you're organizing a surprise pizza party for 20 friends. You send out an online poll asking
everyone to choose their favorite pizza topping. Here's the data you received:
2.
Spot the DirtyData......
• Spot any "dirty data" or errors in the responses.
• Identify how you would clean the data to make it usable.
Point out the issues
Questions
Just like in this pizza party example, when you're collecting data for
analysis, things can go wrong—people give inconsistent, missing, or
outright bizarre responses. Your job as data analysts is to clean it up!
Lesson!
3.
Process the collecteddata to ensure it is accurate, consistent, and usable. This involves
correcting errors, handling missing values, and formatting the data appropriately.
DATA CLEANING
&PREPARATION
3
Key Steps
1.Identify and Understand the Data - Before cleaning, it’s essential to review the dataset and
understand: The structure and source of the data, what is missing, inaccurate, or irrelevant.
2.Remove Duplicates - Ensure that each data entry is unique unless there is a valid reason for
duplicates
3.Handle Missing Data - Remove: If a data point is unnecessary or too incomplete, Fill: Use
average or other statistical methods to estimate missing values, or backfill with previous entries
if appropriate.
4.Standardize Data Formats: Ensure consistency in formats such as dates, text cases, number
formatting, etc.
5.Fix Errors and Outliers: Correct obvious mistakes or unrealistic values (e.g., ages above 120
years). Use logic or domain knowledge to decide what is realistic.
4.
When cleaning data,it is important to adhere to certain rules and best practices to maintain the
integrity and reliability of the data:
Best Practices in Data cleaning and preparation
• Maintain Original Data Integrity: Always keep a backup of the original dataset before starting
any cleaning process. This ensures that you can reference or recover the original data if
necessary.
• Use Clear Naming Conventions: Ensure that column headers and variables are properly
labeled and understandable to others. This makes it easier to interpret and analyze the data.
• Document Your Changes: Keep a log of all data transformations and cleaning operations. This
helps track what was done, especially when working with complex datasets.
• Be Consistent with Rules: Apply consistent rules when handling missing values, duplicates,
and outliers. Avoid arbitrary decisions that could skew the results.
Ethical issues in data cleaning
• Transparency: Data cleaning processes must be transparent. Misleading cleaning practices
(e.g., hiding data that doesn’t fit the narrative) can result in unethical outcomes.
• Bias: Care must be taken to avoid introducing bias during data cleaning. Overzealous cleaning
may result in the exclusion of outliers or data points that could be important for the analysis.
• Data Privacy: If you're working with sensitive data (e.g., personal identifiers, health data),
ethical guidelines and legal regulations. Anonymize data where appropriate.
• Data Ownership: Understand the ownership of the data you are working with. If you are
handling proprietary or third-party data, ensure you have permission to clean, modify, or use
the data in your analysis.
• Accuracy vs. Modification: The objective is to clean data for accuracy, not to alter it for specific
outcomes. Any changes must be justified by errors or inconsistencies rather than desired
results.
5.
Common Excel functionsfor Data cleaning
Removes extra spaces from text except for single spaces between words. =TRIM(A2)
• Example:
⚬ Original Data: " John Doe "
⚬ Cleaned Data: "John Doe" (Leading and trailing spaces removed)
TRIM
Removes non-printable characters from text. =CLEAN(B2).
Often, when importing data from other sources (like web scraping or databases), non-printable
characters may appear. CLEAN helps to remove these characters.
Example:
• Original Data: "John Doe@#?"
• Cleaned Data: "John Doe"
CLEAN
Finds specific text or numbers and replaces them with new content. Use this when you need to
standardize data entries, such as correcting typos, changing abbreviations to full words, or
updating outdated terms.
Example:
• Original Data: "Mgr"
• Find "Mgr" Replace with "Manager"
→
• Cleaned Data: "Manager"
FIND and REPLACE
6.
Common Excel functionsfor Data cleaning
Replaces specific text within a string with another text. =SUBSTITUTE(C2, "Old", "New")
Similar to FIND and REPLACE, but it's more dynamic and can be applied to specific cells within a
formula.
• Example:
⚬ Original Data: "Old Value"
⚬ Formula: =SUBSTITUTE(C2, "Old", "New")
⚬ Cleaned Data: "New Value"
SUBSTITUTE
Standardizes text case to uppercase, lowercase, or proper case (first letter capitalized).
• =UPPER(D2) (Converts text to uppercase)
• =LOWER(D2) (Converts text to lowercase)
• =PROPER(D2) (Capitalizes the first letter of each word)
Examples:
• Original Data: "john doe"
• UPPER: "JOHN DOE"
• LOWER: "john doe"
• PROPER: "John Doe"
UPPER, LOWER, and PROPER
7.
Common Excel functionsfor Data cleaning
Splits data into separate columns based on a delimiter (e.g., comma, space).
Select the column Go to Data Text to Columns Choose delimiter (e.g., comma, space).
→ → →
When you have data that needs to be separated into different columns, like splitting full names
into first and last names.
Example:
• Original Data: "John, Doe"
• Cleaned Data: "John" in one column, "Doe" in another
TEXT TO COLUMNS or SPLIT (in Power
Query)
Automatically detects and removes duplicate rows.
Go to Data Remove Duplicates Select columns to check for duplicates.
→ →
Use this to clean datasets that contain repeated entries, such as duplicate customer records or
survey responses.
REMOVE DUPLICATES
8.
Common Excel functionsfor Data cleaning
Prevents errors (e.g., #DIV/0!) from showing up in cells by replacing them with alternative text.
=IFERROR(E2/F2, "Error"). Provides cleaner results when dealing with mathematical errors, such as
division by zero.
IFERROR
Checks whether a cell is empty. =ISBLANK(G2). Identifies missing data that needs to be filled in or
addressed.
ISBLANK
Remove blanks shortcut using F5 and CTRL -
REMOVE BLANKS
Ensures that only valid data is entered into specific cells by restricting entries (e.g., allowing only
whole numbers or dates).
Steps: Go to Data tab Data Validation Set the rules (e.g., numbers only).
→ →
Prevents errors during data entry by enforcing consistent formats.
DATA VALIDATION
9.
Common Excel functionsfor Data cleaning
Combines text from multiple cells into one cell.
=CONCATENATE(H2, " ", I2) or =H2 & " " & I2
Useful for combining data from separate columns, such as first and last names.
CONCATENATE or &
CONCAT
Similar to CONCATENATE but can combine a range of cells more easily. =CONCAT(A2:A10)
Useful for combining strings from multiple cells into one without needing to reference each cell
individually.
IF, AND, OR
IF: Performs a logical test and returns one value for TRUE and another for FALSE.
• Formula: =IF(A2 > 50, "Pass", "Fail")
AND: Returns TRUE if all conditions are met, FALSE otherwise.
• Formula: =AND(A2 > 50, B2 < 100)
OR: Returns TRUE if any condition is met, FALSE otherwise.
• Formula: =OR(A2 > 50, B2 > 50)
Combine conditions for cleaning data or setting validation rules, such as flagging outliers or
errors.
10.
Data Cleaning withPower Query
Power Query is a data transformation and connectivity tool found in Excel and Power BI. It helps
you to:
• Import data from various sources.
• Clean and transform data without writing complex formulas.
• Refresh data with a click whenever the source data changes
Key Features
• Automate Repetitive Tasks: Power Query saves your steps, so you don’t have to repeat them.
• Data from Multiple Sources: Pull data from databases, websites, CSV files, Excel workbooks,
and more.
• Intuitive Interface: You can visually transform your data without needing programming
knowledge.
Basic Power Query Workflow
• Load Data: Import data from different sources.
• Transform Data: Clean, filter, and reshape your data.
• Apply & Load: Apply the transformations and load the clean data back into Excel.
11.
Power Query DataCleaning Tools
Power Query is a data transformation and connectivity tool found in Excel and Power BI. It helps
you to:
• Import data from various sources.
• Clean and transform data without writing complex formulas.
• Refresh data with a click whenever the source data changes
Here are some common Power Query features for data cleaning:
Replace Values: Automatically replace incorrect or unwanted values (e.g., replace “N/A” with a
blank).
• Go to the Home tab and click Replace Values.
Trim Text: Remove extra spaces from the beginning and end of text strings.
• In the Transform tab, click Trim.
Remove Errors: If your dataset contains errors (like broken links), you can quickly remove them.
• Select the column, right-click, and choose Remove Errors.
Change Data Types: Ensure that columns are correctly typed (e.g., dates as dates, numbers as
numbers).
• Select the column, then in the Transform tab, choose the correct data type.
Unpivot Columns: Convert columns into rows to transform data from a wide format into a tall
format.
• Useful for time-series or survey data where variables are spread across multiple columns.
12.
Assignment: Data CleaningExercise
To practice data cleaning techniques using Excel/Power Query and apply best practices and ethical considerations to
prepare a dataset for analysis.
Objective:
Instructions
Download a sample dataset containing some typical data quality issues (e.g., missing values, duplicates, inconsistent
formatting). You can download one from an open-source repository like Kaggle or a government data portal.
Task
• Remove Duplicates: Identify and remove any duplicate rows in the dataset.
• Fix Inconsistent Formats: Standardize date and text formats (e.g., convert all names to title case using the
PROPER function).
• Handle Missing Data: Identify any missing data and apply appropriate techniques (e.g., removing, backfilling, or
using averages).
• Clean Text Data: Use the TRIM and CLEAN functions to remove unnecessary spaces and special characters.
• Validate Entries: Apply Data Validation rules to ensure only valid data can be entered in specific columns (e.g., for
numerical entries, date fields).
• Error Handling: Use the IFERROR function to manage cells with errors, replacing error messages with "N/A" or
another placeholder.
• Create Documentation: Write a short document (100-150 words) summarizing the data cleaning steps you took
and any assumptions or decisions made during the process (e.g., how you handled missing data).
Reflect on and briefly describe the ethical considerations involved in cleaning the dataset. Were there
any personal identifiers or sensitive information? What steps did you take to ensure data privacy and
avoid bias?
Ethical Consideration
