Data science uses statistics, machine learning, and data analysis to extract knowledge and insights from data. It allows companies to make better decisions, predictive analysis, and discover patterns. Data science is used across many industries and can be applied anywhere data is available, such as consumer goods, stock markets, logistics, and e-commerce. A data scientist requires expertise in machine learning, statistics, programming, mathematics, and databases to explore and analyze data, find patterns, and make predictions to help businesses.
Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...
What is Data Science
1. What is Data Science?
Data Science is a combination of multiple disciplines that
uses statistics, data analysis, and machine learning to
analyze data and to extract knowledge and insights from
it.
2. By using Data Science, companies are able to make:
•Better decisions (should we choose A or B)
•Predictive analysis (what will happen next?)
•Pattern discoveries (find pattern, or maybe hidden
information in the data)
3. Where is Data Science Needed?
• Data Science is used in many industries in the world
today, e.g. banking, consultancy, healthcare, and
manufacturing.
• Examples of where Data Science is needed:
4. Data Science can be applied in nearly
every part of a business where data is
available. Examples are:
• Consumer goods
• Stock markets
• Industry
• Politics
• Logistic companies
• E-commerce
5. How Does a Data Scientist Work?
• A Data Scientist requires expertise in several
backgrounds:
• Machine Learning
• Statistics
• Programming (Python or R)
• Mathematics
• Databases
6. Here is how a Data Scientist works:
1.Ask the right questions - To understand the business problem.
2.Explore and collect data - From database, web logs, customer
feedback, etc.
3.Extract the data - Transform the data to a standardized format.
4.Clean the data - Remove erroneous values from the data.
5.Find and replace missing values - Check for missing values and
replace them with a suitable value (e.g. an average value).
6.Normalize data - Scale the values in a practical range (e.g. 140 cm is
smaller than 1,8 m. However, the number 140 is larger than 1,8. - so
scaling is important).
7.Analyze data, find patterns and make future predictions.
8.Represent the result - Present the result with useful insights in a
way the "company" can understand.
7. What is Data?
• Data is a collection of information.
• One purpose of Data Science is to structure data,
making it interpretable and easy to work with.
8. Data can be categorized into two groups:
• Structured data
• Unstructured data
11. How to Structure Data?
• We can use an array or a database table to structure or
present data.
• Example of an array:
• [80, 85, 90, 95, 100, 105, 110, 115, 120, 125]
13. Database Table
• A database table is a table with structured data.
• The following table shows a database table with health
data extracted from a sports watch:
15. Variables
• A variable is defined as something that can be
measured or counted.
• Examples can be characters, numbers or time.
• In the example under, we can observe that each column
represents a variable.
16.
17. Data Science & Python
• Python is a programming language widely used by Data
Scientists.
• Python has in-built mathematical libraries and
functions, making it easier to calculate mathematical
problems and to perform data analysis.
18. Python Libraries
• Python has libraries with large collections of mathematical
functions and analytical tools.
• In this course, we will use the following libraries:
• Pandas - This library is used for structured data operations, like
import CSV files, create dataframes, and data preparation
• Numpy - This is a mathematical library. Has a powerful N-
dimensional array object, linear algebra, Fourier transform, etc.
• Matplotlib - This library is used for visualization of data.
• SciPy - This library has linear algebra modules
19. Data Science - Python DataFrame
• Create a DataFrame with Pandas
• A data frame is a structured representation of data.
import pandas as pd
d = {'col1': [1, 2, 3, 4, 7], 'col2': [4, 5, 6, 9, 5], 'col3':
[7, 8, 12, 1, 11]}
df = pd.DataFrame(data=d)
print(df)
We write pd. in front of DataFrame() to let Python know that we want to activate the
DataFrame() function from the Pandas library.
Be aware of the capital D and F in DataFrame!
20. Example 1
Count the number of columns:
• count_column = df.shape[1]
print(count_column)
Example 2
Count the number of rows:
count_row = df.shape[0]
print(count_row)
21. Data Science Functions
• This chapter shows three commonly used functions
when working with Data Science: max(), min(), and
mean().
22.
23. The max() function
• The Python max() function is used to find the highest value in an
array.
• Ex
Average_pulse_max = max(80, 85, 90, 95, 100, 105, 110, 115, 120, 125)
print (Average_pulse_max)
24. The mean() function
The NumPy mean() function is used to find the average value of an
array.
Example:
import numpy as np
Calorie_burnage = [240, 250, 260, 270, 280, 290, 300, 310, 320, 330]
Average_calorie_burnage = np.mean(Calorie_burnage)
print(Average_calorie_burnage)
25. Data Science - Data Preparation
• Extract and Read Data With Pandas
• import pandas as pd
sample_data = pd.read_csv(‘music.csv’)
sample_data
26. Data Cleaning
• import pandas as pd
• sample_data = pd.read_csv("music.csv")
• X= sample_data.drop(columns=['genre'])
• print(X)
27. Data Categories
• Data can be split into three main categories:
1.Numerical - Contains numerical values. Can be divided
into two categories:
1.Discrete: Numbers are counted as "whole". Example: You cannot
have trained 2.5 sessions, it is either 2 or 3
2.Continuous: Numbers can be of infinite precision. For example, you
can sleep for 7 hours, 30 minutes and 20 seconds, or 7.533 hours
2.Categorical - Contains values that cannot be measured up
against each other. Example: A color or a type of training
3.Ordinal - Contains categorical data that can be measured
up against each other. Example: School grades where A is
better than B and so on
28. Data Types
We can use the info() function to list the data types
within our data set:
Ex: print(sample_data.info())
29. Analyze the Data
When we have cleaned the data set, we can start
analyzing the data.
We can use the describe() function in Python to
summarize data:
Editor's Notes
Data Science is about data gathering, analysis and decision-making.
Data Science is about finding patterns in data, through analysis, and make future predictions.
For route planning: To discover the best routes to ship
To foresee delays for flight/ship/train etc. (through predictive analysis)
To create promotional offers
To find the best suited time to deliver goods
To forecast the next years revenue for a company
To analyze health benefit of training
To predict who will win elections
A Data Scientist must find patterns within the data. Before he/she can find the patterns, he/she must organize the data in a standard format.
It is common to work with very large data sets in Data Science.
This dataset contains information of a typical training session such as duration, average pulse, calorie burnage etc.
A row is a horizontal representation of data.
A column is a vertical representation of data.
There are 6 columns, meaning that there are 6 variables (Duration, Average_Pulse, Max_Pulse, Calorie_Burnage, Hours_Work, Hours_Sleep).
There are 11 rows, meaning that each variable has 10 observations.
***But if there are 11 rows, how come there are only 10 observations?
It is because the first row is the label, meaning that it is the name of the variable.
Example Explained
Import the Pandas library as pd
Define data with column and rows in a variable named d
Create a data frame using the function pd.DataFrame()
The data frame contains 3 columns and 5 rows
Print the data frame output with the print() function
Why Can We Not Just Count the Rows and Columns Ourselves?
If we work with larger data sets with many columns and rows, it will be confusing to count it by yourself. You risk to count it wrongly. If we use the built-in functions in Python correctly, we assure that the count is correct.
The data set above consists of 6 variables, each with 10 observations:
Duration - How long lasted the training session in minutes?
Average_Pulse - What was the average pulse of the training session? This is measured by beats per minute
Max_Pulse - What was the max pulse of the training session?
Calorie_Burnage - How much calories were burnt on the training session?
Hours_Work - How many hours did we work at our job before the training session?
Hours_Sleep - How much did we sleep the night before the training session?
We use underscore (_) to separate strings because Python cannot read space as separator.
We write np. in front of mean to let Python know that we want to activate the mean function from the Numpy library.
1.Before analyzing data, a Data Scientist must extract the data, and make it clean and valuable.
2. Example Explained
Import the Pandas library
Name the data frame as sample_data.
By knowing the type of your data, you will be able to know what technique to use when analyzing them.