The document discusses the challenges of data profiling, highlighting its manual, time-consuming, and error-prone nature. It outlines steps for effective data profiling using Python and the limitations of the pandas library. Additionally, it mentions alternative packages like pydqc and missingno for summarizing and visualizing data.

1. Getting To Know Any Dataset
In 4 Lines Of Python

2. A Dataset And a Person Walk Into a Bar

3. A Dataset And a Person Walk Into a Bar

4. What's the Problem?
Time Consumming
Its Manual and Not Fun
Error Prone
Many Skills Required

5. Data Profiling
Definition

6. Data Profiling Steps
1.Managing the input.
2.Performing the computation.
3.Managing the output.

7. Data Profiling Steps
1.Managing the input.
2.Performing the computation.
3.Managing the output.

8. Data Profiling Steps
1.Managing the input.
2.Performing the computation.
3.Managing the output.
Note: Pandas allow to read into memory different dataset
types
like Excel, Feather, CSV,Parquet, Databases and More

9. CLICK FOR
MAGIC

10. No Silver Bullet
• Pandas profiling SHINES on non-nested datasets, can be fixed by Flatten
nested datasets.
• Pandas requires a lot of RAM, can be fixed by Sa`mpling.
• Input should be successfully Loaded.
• Pandas profiling doesn't support all profile tasks. :(
• Pandas profiling treat ordinal columns as categorical. :(

11. Honorable Mention
Packagespydqc:
• Data summary report for table.
• Summarize difference between two data tables (useful for
comparing training set with test set, comparing the same
data
table from two different snapshot dates, etc.)
missingno:
• Missing data visualization module for Python.

12. Data Profiling
In Theory
Additional Resources