The Importance & Impact of Data Wrangling/Data Prep

REIN IN YOUR DATA
The Importance & Impact
of Data Wrangling/Data Prep
© Copyright 2019 Data-Core Systems, Inc. | datacoresystems.com

Business Statement
Data-Core is in the Business of providing
Customized IT & IT-enabled Services to SMB
Customers in Specific Vertical Segment
For more information: www.datacoresystems.com

About Data-Core
❖ Incorporated in 1988 as a Delaware corporation
❖ Headquartered in Philadelphia, PA with offices
worldwide
❖ Provider of IT and IT-enabled services
❖ Started AI-Labs in India for training professionals
❖ Data-Core is part of the DC Group, a diversified
multinational conglomerate founded in 1930

Data-Core: Quick Facts
❖ 1500 Employees worldwide
❖ Locations
US:
• Philadelphia, PA
• Bristol, PA
• Las Vegas, NV
❖ Certifications & Compliance
• HIPPA: Healthcare Privacy
• ISO 9001: Quality Standard
• ISO 27001: Security Standard
• PCI: Data Security
• CMMi: Process Improvement
Offshore:
• Kolkata, India
• Mumbai, India

IT Projects Healthcare BPS Consulting AI Labs
• BI/Analytics
• AI/Machine
Learning
• Data Wrangling
• Cloud Services
• API Integration
• SAP
• TIBCO
• .Net/ Java
• Salesforce
• Microsoft
• Data Science
Training
• PG Diploma
• PG Certificate
• Revenue Cycle
• Lockbox Processing
• Medical Claims
• Nimbus Platform
• Patient Engagement
• Health Analytics
• Media Intelligence
• Ad Monitoring
• RUZIVO Platform
• e-Publishing
• Mobile/e-book
• PubFlow Platform
• Legacy Applications
Practice Areas

Transition to the Cloud with
minimal investment
Scalable infrastructure with
close monitoring of capacity
Azure Government Cloud
implementation for security
API Integration:
Connecting systems
Data Wrangling:
Prepping data
Real-time Analytics:
Visualizing data
Machine Learning/ AI:
Predictive Analytics
Implementing off-the-
shelf applications
Developing Customized
applications
Integrating with
existing ERP solutions
Automated testing for
functional & regression
Data
IT Project Practice - Services
Cloud
Application

You can have data without information but you cannot have
information without data.
~ Daniel Keys Moran
Data is like garbage. You would better know what you are going to
do with it before you collect it.
~ Mark Twain
Dirty data is a business problem not an IT problem.
~ Gartner
From the Experts

❖ Data Scientists/Analysts spend too much time preparing the data
❖ Utilizing data scientists to carry out data preparation process
❖ Mapping of raw unstructured data
❖ Data Analytics teams are highly depending on data engineering team to prepare data for analysis
❖ Preparing data without completely understanding business goals or context of the use case
❖ Manual Data Preparation process can hinder collaboration and efficiency of the process
automation
Bottlenecks with Data Preparation

Data Preparation
60%Data Analysis
12%
Data Modeling
10%
Development
10%
Data Integration
5%
Other
3%
Time Consumption(%)
Bottlenecks with Data Preparation

Data Science Pipeline

Data Sources

Types of Raw Data

Unstructured Semi-StructuredStructured
Text files
CSV files
Excel files
PDF
NoSQL
Excel files
XML
JSON
Email
Relational Data
NoSQL
Media Files
NoSQL
Types of Raw Data

Problems with Raw Data

Data with Null Values
Data with Missing Values
Order No. Order Date Region Rep Item Units Unit Cost Total
1 9/1/14 Central Smith Desk 2 125.00 250.00
9 4/10/15 Central Andrews Pencil Null 1.99 131.34
10 12/12/14 Central Smith Pencil 67 1.29 86.43
11 4/18/14 Central Andrews Pencil 20 1.99 Null
14 6/25/14 Central Morgan Pencil 90 4.99 449.10
18 11/8/14 East Parent Pen 15 5.00 Null
1 9/1/14 Central Smith Desk 2 125.00 250.00
9 4/10/15 Central Andrews Pencil 66 1.99 131.34
18 11/8/14 East Parent Pen 15 19.99 299.85
5 1/15/15 Central Gill Binder 8.99
7 5/14/15 Gill Pencil 1.29 68.37
19 9/18/14 East Jones Pen Set 16 15.99 255.84
21 22/10/14 Jones Pen 64 575.36
26 8/24/15 West Sorvino Desk 3 275.00 825.00
28 10/14/15 West Thompson Binder 19.99 1,139.43
5 1/15/15 Central Gill Binder 46 8.99 413.54
7 5/14/15 Central Gill Pencil 53 1.29 68.37
21 10/22/14 East Jones Pen 64 8.99 575.36
28 10/14/15 West Thompson Binder 57 19.99 1,139.43

Backdated Data
3 2/9/1450 Central Jardine Pencil 36 4.99 179.64
4 8/7/15 Central Kivell Pen Set 42 23.95 1,005.90
27 3/15/20014 West Sorvino Pencil 56 2.99 167.44
29 09/27/159 West Sorvino Pen 76 1.99 151.24
3 2/9/14 Central Jardine Pencil 36 4.99 179.64
4 8/7/15 Central Kivell Pen Set 42 23.95 1,005.90
27 3/15/14 West Sorvino Pencil 56 2.99 167.44
29 9/27/15 West Sorvino Pen 76 1.99 151.24
Data with Mismatched Values
13 2/1/15 Center Smith Binder 87 15.00 1,305.00
15 2015 Dec 5 Central Jardine Binder 94 19.99 1,879.06
21 22/10/14 East Jones Pen 64 8.99 575.36
24 Jan 5 2015 East Jones Pencil 95 1.99 189.05
26 8/24/15 W Sorvino Desk 3 275.00 825.00
29 Sept 27 14 West Sorvino Pen 76 1.99 151.24
13 2/1/15 Central Smith Binder 87 15.00 1,305.00
15 12/4/15 Central Jardine Binder 94 19.99 1,879.06
21 10/22/14 East Jones Pen 64 8.99 575.36
24 1/6/14 East Jones Pencil 95 1.99 189.05
29 9/27/15 West Sorvino Pen 76 1.99 151.24

Disparate Data
Unnormalized Data
Order
No.
Order Date Region Rep Item Units Unit Cost Total
6 3/24/15 Central Jardine Pen Set 50 4.99 FALSE
8 7/21/15 Central Morgan Pen Set -55 12.49 686.95
14 6/25/14 Central Morgan Pencil -90 4.99 449.10
15 2015 Dec 5 Central Jardine Binder 94 19.99 1,879.06
16 11/25/14 Central Kivell Pen Set 96 4.99 FALSE
22 12/29/14 E Parent Pen Set 40 15.99 FALSE
6 3/24/15 Central Jardine Pen Set 50 4.99 249.50
8 7/21/15 Central Morgan Pen Set 55 12.49 686.95
15 12/4/15 Central Jardine Binder 94 19.99 1,879.06
16 11/25/14 Central Kivell Pen Set 96 4.99 479.04
22 12/29/14 East Parent Pen Set 74 15.99 1,183.26
11 4/18/14 Central Andrews Pencil 20 1.99 Null
23 7/29/14 East Parent Binder 81 19.99 1,619.19
25 4/27/15 East Howard Pen 96 4.99 479.04
23 7/29/14 East Parent Binder 81 19.99 1,619.19
25 4/27/15 East Howard Pen 96 4.99 479.04

Data Wrangling/Preparation Steps

Data Wrangling/Preparation with
Excel

Data Sources
On-premises
Cloud
Hybrid
Data Types
Structured
Unstructured
Semi-structured
Import Data
Live Streaming/
Real-time Data
Historical Data
Pre-requisites
Basic
Knowledge
about the tool
(Excel)
Default – 10MB
Maximum – 2GB
File Size

Features
Easy Application Integrations
Easy to use and readily available tool
Records live data from different data sources
Analytics Features
Easy to import data from multiple data sources
Limitation
Data Integrity Issues
Version Control Issues
Difficult to append data from multiple data sources
Less Scalability
Limited file/spreadsheet size

SSIS

❖ SQL Server Integration Services (SSIS)
✓ A high performance integration and transformation tool used to perform data extraction, data
transformation and loading data (ETL) to various destinations.
✓ Component of Microsoft SQL Server database software.
❖ Extract-Transform-Load (ETL) Process
✓ Data Extraction is the process where data is collected from various sources like text files, XML
files, Excel files etc.
✓ The Data Transformation process where the collected data is transformed as per the
requirements before it is loaded into that destination storage.
✓ The Load process is when the data collected from various sources is transformed and is then
loaded to the destination storage.

❖ SSIS Features:
✓ Data Cleaning
✓ Intelligent Data Handling
✓ Data Integration
✓ Flexible concurrent data processing to multiple varied destinations
✓ Visual Studio, SQL Server and Custom code integration
❖ SSIS Limitations
✓ The limitation on the data volume/file size is directly dependent of the system memory (RAM)
where the SSIS package is running as SSIS loads the data into the memory.
✓ SSIS cannot process real-time/live streaming data.

Power BI

Python

Demo

Azure ML Studio

Azure Data Factory

Tableau Prep

Trifacta

❖ Data Wrangling/Preparation tools can benefit Data Scientists to focus on the data analysis and other counter
part activities rather than utilizing 60% of their time on data preparation processes
❖ Data preparation tools and platforms overall enables the data engineering team to prepare data for analysis
more efficiently and therefore help the Data Analytics teams with valuable data
❖ Data Preparation tools and platforms have advanced features mapping of raw unstructured data, providing
live previews of the data and AI features that provide suitable suggestions based on the dataset and give an
overview of the overall data flow
❖ Data wrangling provides automation process features that help to prepare the data without any redundant
data preparation process
Summary

Discussion

Thank You!

The Importance & Impact of Data Wrangling/Data Prep

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

The Importance & Impact of Data Wrangling/Data Prep