Course Overview
1
Data Manipulation&
Analysis with
Pandas
2
Data Manipulation
with Numpy
3
Visualizing data with
Matplotlib
4
Advance data
visualization with
Seaborn
5
Machine learning
with Scikit-Learn
6
Project
Topics Covered inthis Section
1. Introduction to Pandas (Overview & installation)
2. Import files using Pandas
3. Inspecting data with Pandas (Head, tail, info, describe etc.)
4. Slicing & dicing data with Pandas
5. Creating new features with Pandas
6. Aggregating data with Pandas
7. Joining datasets with Pandas
What is Pandas?
Pandas is the primary package for performing data
analysis tasks in Python.
Pandas derives its name from panel data analysis and
is the fundamental package that provides relational
data structures (think Excel, SQL type) and a host of
capabilities to play with those data structures.
It is the most widely used package in Python for data
analysis tasks, and is very good to work with cross
sectional, time series, and panel data analysis.
7.
Why Pandas?
• Ithas a tabular data structure that can hold both homogenous and
heterogenous data. Programming is much easier & faster.
• Very good indexing capabilities that makes data alignment and merging easy.
• Good time series functionality. No need to use different data structures for time
series and cross sectional data. Allows for both ordered and unordered time-
series data.
• A host of statistical functions developed around NumPy and pandas that makes
a researcher’s task easy and fast.
• Easily handles data manipulation and cleaning.
• Easy to expand and shorten data sets. Comprehensive merging, joins, and
group by functionality to join multiple data sets.
8.
Installing Pandas
Pandas isa data wrangling library, pull up a computer & let’s start by installing it:
Installation depending on your environment (ie you installed conda, or have pip):
● pip install pandas
● conda install pandas
Python doesn’t load all of the libraries available to it by default. We have to add an
import statement to our code in order to use library functions. To import a library,
we use the syntax import libraryName. If we want to give the library a nickname to
shorten the command, we can add as nickNameHere.
An example of importing the pandas library using common nickname pd is below.
● import pandas as pd #This will import data into your workspace
Using Pandas wecan import several formats of data like
● CSV
● Excel
● Text
● JSON
● SQL files
● Web pages/ HTML
All these formats are read using Pandas as a Data Frame.
In this section, we will focus on reading CSV file and see how we can manage
Data Frame with various function using Pandas.
# Example
df = pd.read_csv("filename.csv")
Importing Files
This is initialand important step after loading file, to check whether the file has
loaded correctly and properly.
Following are functions to check:
● Checking the head & tail of the data frame.
● Getting data information on variables types.
● Understanding the characteristics of data.
● Statistical summary on the variables.
● Dot operator or square bracket to check a specific column.
● Locating specific range or array of columns & rows.
Inspecting Data
In data manipulation,slicing and dicing is a key activity to subset the data as per
the needs of the analysis. For ex:
1. Reducing the data for only a particular Product OR Geography
2. Taking customers who are working only in corporate
3. Analyzing Sales Representatives that are in corporate sales and sale only one
product
Slicing and Dicing Data
During a DataScience project cycle for various purposes it might be needed to
create a new feature to better understand data and imporve modeling.
Most importantly it is used for Modeling with desire for good fit. Feature
engineering, also known as feature creation, is the process of constructing new
features from existing data to train a machine learning model.
Here are some of the feature creation option with Pandas:
1. Creating new features using conditional assignment of values
2. Creating dummy variables
Creating new Features
Many wonderful resultscan be achieved on aggregating key values can lead to
statistical significant features and right feature for modeling.
Here are some of aggregate functions
● mean(): Compute mean of groups
● sum(): Compute sum of group values
● size(): Compute group sizes
● count(): Compute count of group
● first(): Compute first of group values
● last(): Compute last of group values
● max(): Compute max of group values
Aggregating Data
Pandas has full-featured,high performance in-memory join operations that are very similar to relational
databases like SQL.
With use of Merge function or join function, we can set or input what kind of join needs to be performed
with indicating of common key.
Type of Joins:
Inner Join Left Join Right Join Full Outer Join
Joining Data
Editor's Notes
#8 Each time we call a function that’s in a library, we use the syntax LibraryName.FunctionName. Adding the library name with a . before the function name tells Python where to find the function. In the example above, we have imported Pandas as pd. This means we don’t have to type out pandas each time we call a Pandas function.
#12 df.shape() #This tells you how large a DataFrame is and is in the format (rows, columns)
df.head() #Returns the first 5 rows of the given DataFrame.
df.tail() #Returns the last 5 rows of the given DataFrame.
df.dtypes or df.variable.dtype #See the data-type of one or more columns
df.describe() #Gives the summary statistics of the data set….count,min,max,sum etc.
df.variable.astype() #Convert a column to another data type say for example: converting integers to floats or vice versa
#14 # Select rows 0, 1, 2 (row 3 is not selected)
Surveys_df[0:3]
# Select the first 5 rows (rows 0, 1, 2, 3, 4)
surveys_df[:5]
# Select the last element in the list
# (the slice starts at the last element, and ends at the end of the list)
surveys_df[-1:]
#16 # Reset the index values to the second dataframe appends properly
survey_sub_last10 = survey_sub_last10.reset_index(drop=True)
# drop=True option avoids adding new index column with old index values
#20 merged_inner = pd.merge(left=survey_sub, right=species_sub, left_on='species_id', right_on='species_id')
# In this case `species_id` is the only column name in both dataframes, so if we skipped `left_on`
# And `right_on` arguments we would still get the same result
# What's the size of the output data?
merged_inner.shape
merged_inner
merged_left = pd.merge(left=survey_sub, right=species_sub, how='left', left_on='species_id', right_on='species_id')
merged_left
