What is Data Science?
Data Science is a combination of multiple disciplines that
uses statistics, data analysis, and machine learning to
analyze data and to extract knowledge and insights from
it.
By using Data Science, companies are able to make:
•Better decisions (should we choose A or B)
•Predictive analysis (what will happen next?)
•Pattern discoveries (find pattern, or maybe hidden
information in the data)
Where is Data Science Needed?
• Data Science is used in many industries in the world
today, e.g. banking, consultancy, healthcare, and
manufacturing.
• Examples of where Data Science is needed:
Data Science can be applied in nearly
every part of a business where data is
available. Examples are:
• Consumer goods
• Stock markets
• Industry
• Politics
• Logistic companies
• E-commerce
How Does a Data Scientist Work?
• A Data Scientist requires expertise in several
backgrounds:
• Machine Learning
• Statistics
• Programming (Python or R)
• Mathematics
• Databases
Here is how a Data Scientist works:
1.Ask the right questions - To understand the business problem.
2.Explore and collect data - From database, web logs, customer
feedback, etc.
3.Extract the data - Transform the data to a standardized format.
4.Clean the data - Remove erroneous values from the data.
5.Find and replace missing values - Check for missing values and
replace them with a suitable value (e.g. an average value).
6.Normalize data - Scale the values in a practical range (e.g. 140 cm is
smaller than 1,8 m. However, the number 140 is larger than 1,8. - so
scaling is important).
7.Analyze data, find patterns and make future predictions.
8.Represent the result - Present the result with useful insights in a
way the "company" can understand.
What is Data?
• Data is a collection of information.
• One purpose of Data Science is to structure data,
making it interpretable and easy to work with.
Data can be categorized into two groups:
• Structured data
• Unstructured data
Unstructured Data
• Unstructured data is not organized. We must organize
the data for analysis purposes.
Structured Data
• Structured data is organized and easier to work with.
How to Structure Data?
• We can use an array or a database table to structure or
present data.
• Example of an array:
• [80, 85, 90, 95, 100, 105, 110, 115, 120, 125]
Try this:
Array=[80, 85, 90, 95, 100, 105, 110, 115, 120, 125]
print(Array)
Database Table
• A database table is a table with structured data.
• The following table shows a database table with health
data extracted from a sports watch:
Database Table Structure
A database table consists of column(s)
and row(s):
Variables
• A variable is defined as something that can be
measured or counted.
• Examples can be characters, numbers or time.
• In the example under, we can observe that each column
represents a variable.
Data Science & Python
• Python is a programming language widely used by Data
Scientists.
• Python has in-built mathematical libraries and
functions, making it easier to calculate mathematical
problems and to perform data analysis.
Python Libraries
• Python has libraries with large collections of mathematical
functions and analytical tools.
• In this course, we will use the following libraries:
• Pandas - This library is used for structured data operations, like
import CSV files, create dataframes, and data preparation
• Numpy - This is a mathematical library. Has a powerful N-
dimensional array object, linear algebra, Fourier transform, etc.
• Matplotlib - This library is used for visualization of data.
• SciPy - This library has linear algebra modules
Data Science - Python DataFrame
• Create a DataFrame with Pandas
• A data frame is a structured representation of data.
import pandas as pd
d = {'col1': [1, 2, 3, 4, 7], 'col2': [4, 5, 6, 9, 5], 'col3':
[7, 8, 12, 1, 11]}
df = pd.DataFrame(data=d)
print(df)
We write pd. in front of DataFrame() to let Python know that we want to activate the
DataFrame() function from the Pandas library.
Be aware of the capital D and F in DataFrame!
Example 1
Count the number of columns:
• count_column = df.shape[1]
print(count_column)
Example 2
Count the number of rows:
count_row = df.shape[0]
print(count_row)
Data Science Functions
• This chapter shows three commonly used functions
when working with Data Science: max(), min(), and
mean().
The max() function
• The Python max() function is used to find the highest value in an
array.
• Ex
Average_pulse_max = max(80, 85, 90, 95, 100, 105, 110, 115, 120, 125)
print (Average_pulse_max)
The mean() function
The NumPy mean() function is used to find the average value of an
array.
Example:
import numpy as np
Calorie_burnage = [240, 250, 260, 270, 280, 290, 300, 310, 320, 330]
Average_calorie_burnage = np.mean(Calorie_burnage)
print(Average_calorie_burnage)
Data Science - Data Preparation
• Extract and Read Data With Pandas
• import pandas as pd
sample_data = pd.read_csv(‘music.csv’)
sample_data
Data Cleaning
• import pandas as pd
• sample_data = pd.read_csv("music.csv")
• X= sample_data.drop(columns=['genre'])
• print(X)
Data Categories
• Data can be split into three main categories:
1.Numerical - Contains numerical values. Can be divided
into two categories:
1.Discrete: Numbers are counted as "whole". Example: You cannot
have trained 2.5 sessions, it is either 2 or 3
2.Continuous: Numbers can be of infinite precision. For example, you
can sleep for 7 hours, 30 minutes and 20 seconds, or 7.533 hours
2.Categorical - Contains values that cannot be measured up
against each other. Example: A color or a type of training
3.Ordinal - Contains categorical data that can be measured
up against each other. Example: School grades where A is
better than B and so on
Data Types
We can use the info() function to list the data types
within our data set:
Ex: print(sample_data.info())
Analyze the Data
When we have cleaned the data set, we can start
analyzing the data.
We can use the describe() function in Python to
summarize data:

Lecture3.pptx

  • 1.
    What is DataScience? Data Science is a combination of multiple disciplines that uses statistics, data analysis, and machine learning to analyze data and to extract knowledge and insights from it.
  • 2.
    By using DataScience, companies are able to make: •Better decisions (should we choose A or B) •Predictive analysis (what will happen next?) •Pattern discoveries (find pattern, or maybe hidden information in the data)
  • 3.
    Where is DataScience Needed? • Data Science is used in many industries in the world today, e.g. banking, consultancy, healthcare, and manufacturing. • Examples of where Data Science is needed:
  • 4.
    Data Science canbe applied in nearly every part of a business where data is available. Examples are: • Consumer goods • Stock markets • Industry • Politics • Logistic companies • E-commerce
  • 5.
    How Does aData Scientist Work? • A Data Scientist requires expertise in several backgrounds: • Machine Learning • Statistics • Programming (Python or R) • Mathematics • Databases
  • 6.
    Here is howa Data Scientist works: 1.Ask the right questions - To understand the business problem. 2.Explore and collect data - From database, web logs, customer feedback, etc. 3.Extract the data - Transform the data to a standardized format. 4.Clean the data - Remove erroneous values from the data. 5.Find and replace missing values - Check for missing values and replace them with a suitable value (e.g. an average value). 6.Normalize data - Scale the values in a practical range (e.g. 140 cm is smaller than 1,8 m. However, the number 140 is larger than 1,8. - so scaling is important). 7.Analyze data, find patterns and make future predictions. 8.Represent the result - Present the result with useful insights in a way the "company" can understand.
  • 7.
    What is Data? •Data is a collection of information. • One purpose of Data Science is to structure data, making it interpretable and easy to work with.
  • 8.
    Data can becategorized into two groups: • Structured data • Unstructured data
  • 9.
    Unstructured Data • Unstructureddata is not organized. We must organize the data for analysis purposes.
  • 10.
    Structured Data • Structureddata is organized and easier to work with.
  • 11.
    How to StructureData? • We can use an array or a database table to structure or present data. • Example of an array: • [80, 85, 90, 95, 100, 105, 110, 115, 120, 125]
  • 12.
    Try this: Array=[80, 85,90, 95, 100, 105, 110, 115, 120, 125] print(Array)
  • 13.
    Database Table • Adatabase table is a table with structured data. • The following table shows a database table with health data extracted from a sports watch:
  • 14.
    Database Table Structure Adatabase table consists of column(s) and row(s):
  • 15.
    Variables • A variableis defined as something that can be measured or counted. • Examples can be characters, numbers or time. • In the example under, we can observe that each column represents a variable.
  • 17.
    Data Science &Python • Python is a programming language widely used by Data Scientists. • Python has in-built mathematical libraries and functions, making it easier to calculate mathematical problems and to perform data analysis.
  • 18.
    Python Libraries • Pythonhas libraries with large collections of mathematical functions and analytical tools. • In this course, we will use the following libraries: • Pandas - This library is used for structured data operations, like import CSV files, create dataframes, and data preparation • Numpy - This is a mathematical library. Has a powerful N- dimensional array object, linear algebra, Fourier transform, etc. • Matplotlib - This library is used for visualization of data. • SciPy - This library has linear algebra modules
  • 19.
    Data Science -Python DataFrame • Create a DataFrame with Pandas • A data frame is a structured representation of data. import pandas as pd d = {'col1': [1, 2, 3, 4, 7], 'col2': [4, 5, 6, 9, 5], 'col3': [7, 8, 12, 1, 11]} df = pd.DataFrame(data=d) print(df) We write pd. in front of DataFrame() to let Python know that we want to activate the DataFrame() function from the Pandas library. Be aware of the capital D and F in DataFrame!
  • 20.
    Example 1 Count thenumber of columns: • count_column = df.shape[1] print(count_column) Example 2 Count the number of rows: count_row = df.shape[0] print(count_row)
  • 21.
    Data Science Functions •This chapter shows three commonly used functions when working with Data Science: max(), min(), and mean().
  • 23.
    The max() function •The Python max() function is used to find the highest value in an array. • Ex Average_pulse_max = max(80, 85, 90, 95, 100, 105, 110, 115, 120, 125) print (Average_pulse_max)
  • 24.
    The mean() function TheNumPy mean() function is used to find the average value of an array. Example: import numpy as np Calorie_burnage = [240, 250, 260, 270, 280, 290, 300, 310, 320, 330] Average_calorie_burnage = np.mean(Calorie_burnage) print(Average_calorie_burnage)
  • 25.
    Data Science -Data Preparation • Extract and Read Data With Pandas • import pandas as pd sample_data = pd.read_csv(‘music.csv’) sample_data
  • 26.
    Data Cleaning • importpandas as pd • sample_data = pd.read_csv("music.csv") • X= sample_data.drop(columns=['genre']) • print(X)
  • 27.
    Data Categories • Datacan be split into three main categories: 1.Numerical - Contains numerical values. Can be divided into two categories: 1.Discrete: Numbers are counted as "whole". Example: You cannot have trained 2.5 sessions, it is either 2 or 3 2.Continuous: Numbers can be of infinite precision. For example, you can sleep for 7 hours, 30 minutes and 20 seconds, or 7.533 hours 2.Categorical - Contains values that cannot be measured up against each other. Example: A color or a type of training 3.Ordinal - Contains categorical data that can be measured up against each other. Example: School grades where A is better than B and so on
  • 28.
    Data Types We canuse the info() function to list the data types within our data set: Ex: print(sample_data.info())
  • 29.
    Analyze the Data Whenwe have cleaned the data set, we can start analyzing the data. We can use the describe() function in Python to summarize data:

Editor's Notes

  • #2 Data Science is about data gathering, analysis and decision-making. Data Science is about finding patterns in data, through analysis, and make future predictions.
  • #4 For route planning: To discover the best routes to ship To foresee delays for flight/ship/train etc. (through predictive analysis) To create promotional offers To find the best suited time to deliver goods To forecast the next years revenue for a company To analyze health benefit of training To predict who will win elections
  • #6 A Data Scientist must find patterns within the data. Before he/she can find the patterns, he/she must organize the data in a standard format.
  • #13 It is common to work with very large data sets in Data Science.
  • #14 This dataset contains information of a typical training session such as duration, average pulse, calorie burnage etc.
  • #15 A row is a horizontal representation of data. A column is a vertical representation of data.
  • #17 There are 6 columns, meaning that there are 6 variables (Duration, Average_Pulse, Max_Pulse, Calorie_Burnage, Hours_Work, Hours_Sleep). There are 11 rows, meaning that each variable has 10 observations. ***But if there are 11 rows, how come there are only 10 observations? It is because the first row is the label, meaning that it is the name of the variable.
  • #20 Example Explained Import the Pandas library as pd Define data with column and rows in a variable named d Create a data frame using the function pd.DataFrame() The data frame contains 3 columns and 5 rows Print the data frame output with the print() function
  • #21 Why Can We Not Just Count the Rows and Columns Ourselves? If we work with larger data sets with many columns and rows, it will be confusing to count it by yourself. You risk to count it wrongly. If we use the built-in functions in Python correctly, we assure that the count is correct.
  • #23 The data set above consists of 6 variables, each with 10 observations: Duration - How long lasted the training session in minutes? Average_Pulse - What was the average pulse of the training session? This is measured by beats per minute Max_Pulse - What was the max pulse of the training session? Calorie_Burnage - How much calories were burnt on the training session? Hours_Work - How many hours did we work at our job before the training session? Hours_Sleep - How much did we sleep the night before the training session? We use underscore (_) to separate strings because Python cannot read space as separator.
  • #25 We write np. in front of mean to let Python know that we want to activate the mean function from the Numpy library.
  • #26 1.Before analyzing data, a Data Scientist must extract the data, and make it clean and valuable. 2. Example Explained Import the Pandas library Name the data frame as sample_data.
  • #28 By knowing the type of your data, you will be able to know what technique to use when analyzing them.