Implementing a data_science_project (Python Version)_part1

Arthur Samuel (1959)
Machine Learning is
the field of study
that gives computers
the ability to learn
without being

Project Description & Checklist
Data Loading, Merging and Visualisation
Feature Cleaning, Selection & Transformation
Machine Learning Algorithm Adoption
Model Performance Evaluation
Outline
Model Validation, Fine-Tuning & Ensembling

The Description
To use machine learning
techniques to perform
exploratory and predictive
analyses on crime data.

Project Description, Resources & Checklist
The Datasets
Additional data
(to be sourced later)
Dataset D
?
!
Data on the location
(i.e. geographical
coordinates) of the
police stations across
the country.
Dataset C
Data on the names of
police station and the
population that fall
under their
jurisdiction.
Dataset B
Data on crime
reported across the
country and the
respective police
stations
(2015/ 2016).
Dataset A

Checklist
Checklist 1
Is it a supervised, unsupervised or reinforcement machine
learning project?

Unsupervised
Learning
Computer
learns by
searching

Unsupervised
Learning
Aims at
finding
patterns

Outcome feature is known
Task driven
Fits data
Its goal is to predict values in
continuous (regression) or categorical
(classification) format
Example, in retail business, predict
the credit worthiness of a a potential
customer.
Re-Inforcement
Learning
Unsupervised
Learning
Supervised
Learning
Outcome feature is unknown.
Data driven
Clusters data
Its goal is to find patterns
(clustering) in the data.
Example: Segment clients by socio-
demographic characteristics.
Outcome feature is unknown.
Circumstance driven.
Decides on data
Its goal is to learn how to decide
under a given circumstance.
Example: In forex trading, adjust the
take-loss or take-profit based on the
performance of the traded currency.

Id Province Police Station Population Burglary
AB123 Gauteng Dunnottar 10479 141
AB123 North West Mmabatho 134138 773
Id Province Police Station Population Frequent Crime
AB123 Gauteng Dunnottar 10479 Burglary
AB123 North West Mmabatho 134138 Arson
Label
Supervised Learning
Labelled Data
Label

Id Province Police Station Population Burglary Crime Type
AB123 Gauteng Dunnottar 10479 141 Burglary
AB123 North West Mmabatho 134138 773 Arson
Unsupervised Learning
Unlabelled Data

Checklist
Checklist 1
Checklist 2
Is it a supervised or unsupervised machine learning project?
Is it a classification or regression task?

Id Province Police Station Population Burglary
AB123 Gauteng Dunnottar 10479 141
AB123 North West Mmabatho 134138 773
Regression
Id Province Police Station Population Frequent Crime
AB123 Gauteng Dunnottar 10479 Burglary
AB123 North West Mmabatho 134138 Arson
Classification
Supervised Learning
Labelled Data
The values are
continuous
The values are
categorical

Checklist
Checklist 1
Checklist 2
Is it a supervised, unsupervised or reinforcement machine
learning project?
Is it a classification or regression task?
Checklist 3 Identify the target feature or features to be clustered
Checklist 4 Can I get extra data or feature to boost my project?

Checklist 5
Checklist 6
What are the available solutions to the problem?
How do I intend to measure the performance of my model?
Checklist 7 How will my solution be deployed and utilised?
Checklist

Video
AudioText
ImageAlpha
Numeric $1,000
Male Female
No
Yes
2014-08-21
10-5
2.0
1
This is a quote by Napoleon Hill.
do small things in a great way.
If you cannot do great things
Data Loading, Merging & Visualisation
Data Form

Data Location
Computer | Server | Web | Cloud.
Where is the dataset located?
Data Form
Numeric | Text | Image | Audio | Video.
The dataset is what form? Alpha-
Data Size
byte, megabyte, gigabyte or terabyte.
How big is the dataset? Is the size in kilo
Analysis Platform
Can I analyse it on my computer or I need to engage the
Data Flow
as a stream or in batches?
Is it a real time data? Does it come
Data Loading Checklist
service of cloud based computing provider e.g. Microsoft Azure,
Amazon web service (AWS), google cloud etc.

Data Loading Steps
Step 1
 a webpage will open Type Jupyter Notebook Anaconda PromptStart Menu
Start the Jupyter notebook or your
LET’S DEMONSTRATE THIS
It is assumed that you have already installed Anaconda

Anaconda
In your Windows Start Menu,
type in Anaconda or browse
to find anaconda prompt
Click on Anaconda prompt and a
command prompt will appear

Type Jupyter Notebook and press Enter.
A webpage will come up.
Jupyter notebook

To change the title click on the
default type and type your title.
Select this each time you want to write code
This is where you will enter your code. Each
time you press Alt+Enter to run your codes
another one will appear.
This box can be in different mode.
Code | Markdown |Raw NBConvert |Heading
Select this each time you want to write comments. It
support HTML codes.
This has option for HTML, LaTex, rest codes to be run.
Select this each time you want to make heading.

Data Loading Steps
Step 2
Step 1
import os
os.getcwd()
os.chdir('C:/Anaconda3')
Import the python module for checking & changing your directory
a webpage will open Type Jupyter Notebook Anaconda PromptStart Menu
LET’S SEE THE CODE ON JUPYTER NOTEBOOK

Data Loading Steps
Step 3
Step 2
Step 1
import pandas as pd
Import the python module for loading data i.e. pandas
import os
os.getcwd()
a webpage will open Type Jupyter Notebook Anaconda PromptStart Menu

Data Loading Steps
Step 4
Step 3
Step 2
Step 1
Dataset=pd.read_csv(‘C:/MyDataset.csv’)
Load the data
import pandas as pd
import os
os.getcwd()
 a webpage will open Type Jupyter Notebook Anaconda PromptStart Menu
The other kind of data
Format that you can load

That is the folder where you put your dataset.
Note the direction of the slash (/)
If you want it like (), type r’s
r’CAnacondaMyData.csv’

Data Loading Steps
Step 4
Step 3
Step 2
Step 1
Dataset=pd.read_csv(‘C:/MyDataset.csv’)
Load the data
import pandas as pd
import os
os.getcwd()
 Webpage will open Type Jupyter Notebook Anaconda PromptStart Menu

Project Data Loading
Viewing the top 5 Records
DatasetA
The dataset is in csv (comma delimited) format
Dataset A - Crime Reported and Police Station

DatasetA

Reshaping the dataset
DatasetA
Province Police_Station Crime_Category Period_2015_2016
Eastern Cape Aberdeen All theft not mentioned elsewhere 51
Eastern Cape Aberdeen Theft out of or from motor vehicle 7
Eastern Cape Aberdeen Theft of motor vehicle and motorcycle 2
Eastern Cape Aberdeen Stock-theft 20
Long Format
Province Police_Station All theft not
mentioned elsewhere
Theft out of or from
motor vehicle
Theft of motor vehicle
and motorcycle
Stock-theft
Eastern Cape Aberdeen 51 7 2 20
Wide Format

DatasetA
Reshaping (Pivoting) the dataset from "long" to "wide" format
We need to flatten the data frame.

DatasetA
Flattening the pivoted dataset

DatasetA
Check the datasets for duplicates
This is a major checklist before merging this dataset with the other datasets.

Dataset B - Police Station and the Population that they Cover
DatasetB
The dataset is in xlsx (MS excel) format

DatasetB
Viewing the attributes of the features

Dataset C - Police Station and their Geo-Coordinates
DatasetC
The dataset is in tsv (tab delimited) format

DatasetC
Viewing the attributes of the features

Total Records = 1142
Feature
Police_Station
LongitudeY
LatitudeX
Dataset C
Feature
Police_Station
population_estimate
Dataset B
Feature
Province
Police_Station
Crime_Category
Period_2015_2016
Dataset A

Datasets Merging
Province
Police_Station
Crime_Category
Period_2015_2016
Police_Station
population_estimate
Police_Station
LongitudeY
LatitudeX
1143
1140 1142

Datasets Merging
Merging Dataset A & B
Note: Dataset A contains more records than Dataset B. Hence, Dataset A is the universal dataset.

Datasets Merging
Merging Dataset A_B with Dataset C
Merging …

Please subscribe to my youtube channel for the
other versions
And like the video on linkedin and youtube

Implementing a data_science_project (Python Version)_part1

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Implementing a data_science_project (Python Version)_part1

Similar to Implementing a data_science_project (Python Version)_part1 (20)

More from Dr Sulaimon Afolabi

More from Dr Sulaimon Afolabi (12)

Recently uploaded

Recently uploaded (20)

Implementing a data_science_project (Python Version)_part1