Big Data & Analytics (Conceptual and Practical Introduction)

Big Data & Analytics
Workshop
Yaman Hajja, Ph.D.
yamanhajja@gmail.com
March 24, 2017

1
Introduction
What is data?
Data
is a set of values of qualitative or quantitative variables.
Yaman Hajja | Big Data & Analytics

1
Introduction
What is data?
Data
is any sequence of one or more symbols given meaning by
speciﬁc act(s) of interpretation. [In Computing].

1
Introduction
What is data?
Data
is any sequence of one or more symbols given meaning by
speciﬁc act(s) of interpretation. [In Computing].
Data Information
Data requires interpretation to become information.

2
Data is the new oil of the digital economy
Data in the 21st century is like oil in the 18th century.
Data is the new oil of
the digital economy.

2
Data is the new oil of the digital economy
Data in the 21st century is like oil in the 18th century.
Data is the new oil of
the digital economy.
Data infrastructure should become a proﬁt center.

3
Types of data
Types of data. Translation of document hosted by João Netoat.

4
Open Data
Open Data
is the idea that some data should be freely available to everyone
to use and republish as they wish, without restrictions from
copyright, patents or other mechanisms of control.

4
Open Data
Open Data
is the idea that some data should be freely available to everyone
to use and republish as they wish, without restrictions from
copyright, patents or other mechanisms of control.
Example:
Linked Datasets as of August 2014. Tungsten Tide.

5
Datasets for data science projects
Example:
analyticsvidhya
Example:
kaggle
Example:
drivendata
opendatasoft
opendatainception

6
What is data analysis?
Data analysis: also known as data analytics, is a process of
inspecting, cleansing, transforming, and modeling data with the
goal of discovering useful information, suggesting conclusions,
and supporting decision-making.
Data analysis has multiple facets and approaches,
encompassing diverse techniques under a variety of names, in
different business, science, and social science domains.

7
What is data analysis?
Statistical data
Statistical data?
Statistical analysis:
is a component of data analytics. In the context of business
intelligence (BI), statistical analysis involves collecting and
scrutinizing every data sample in a set of items from which
samples can be drawn.
A sample,
in statistics, is a representative selection drawn from a total
population.

8
Data analysis process

9
Data Information Intelligence

10
Understanding Big Data

11
Understanding Big Data
Big Data
is a term for data sets that are so
large or complex that traditional data
processing application softwares are
inadequate to deal with them.
Challenges include capture, storage,
analysis, data curationa
, search,
sharing, transfer, visualization, querying,
updating and information privacy.
aorganization and integration of data collected from
various sources

12
Big Data Characteristics
3 Vs
1. Volume: big data doesn’t sample; it just observes and tracks
what happens
2. Velocity: big data is often available in real-time
3. Variety: big data draws from text, images, audio, video; plus it
completes missing pieces through data fusion

13
Who can deal with Big Data?

14
Who can deal with Big Data?

15
Multidisciplinary!!!

16
Big Data tools

17
Big Data tools

18
some Big Data facts
Big Data and Business Analytics Revenues Forecast to Reach
$150.8 Billion This Year, Led by Banking and Manufacturing
Investments, According to from International Data Corporation
(IDC), an increase of 12.4% over 2016.
Twenty-ﬁve years ago, data was growing at a rate of 100GB a
day. Now, data grows at a rate of almost 50,000GB a second.
The world today is awash in data. In 2015, mankind produced as
much information as was created in all previous years of human
civilization. Every time we send a message, make a call, or
complete a transaction, we leave digital traces.

19
Data scientists vs data analysts

20
Data Visualization
Data visualization is a general term that describes any effort to
help people understand the signiﬁcance of data by placing it in a
visual context. Patterns, trends and correlations that might go
undetected in text-based data can be exposed and recognized
easier with data visualization software.

21
Example: Data Visualized
Charter value
NPLs
Exchange rate
M1
15
15.5
16
16.5
17
17.5
18
Chartervalue%
2
4
6
8
10
12
14
16
18
20
22 2002m1
2002m7
2003m1
2003m7
2004m1
2004m7
2005m1
2005m7
2006m1
2006m7
2007m1
2007m7
2008m1
2008m7
2009m1
2009m7
2010m1
2010m7
2011m1
2011m7
2012m1
2012m7
2013m1
2013m7
2014m1
2014m7
2015m1
2015m7
Time (2002 M1 - 2015 M8)
NPLs % Money supply M1 % pa
Exchange rate Charter value %
NPls of Malaysia banking system over M1, exchange rate, and charter value (2002 M1 - 2015 M8)

22
Example#2: Data Visualized
Capital
GDP
NPLs
-12
-10
-8
-6
-4
-2
0
2
4
6
8
10
12
14
16 1998m1
1998m7
1999m1
1999m7
2000m1
2000m7
2001m1
2001m7
2002m1
2002m7
2003m1
2003m7
2004m1
2004m7
2005m1
2005m7
2006m1
2006m7
2007m1
2007m7
2008m1
2008m7
2009m1
2009m7
2010m1
2010m7
2011m1
2011m7
2012m1
2012m7
2013m1
2013m7
2014m1
2014m7
2015m1
NPLs % GDP growth % Capital ratio %
NPLs of Malaysia banking system over business cycle (GDP) (1998 M1 - 2015 M3) with capital ratio

23
Example#3: Data Visualized
NPLs
Lending rate
Unemploment
Inflation
-3
-2
-1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1998q1
1998q3
1999q1
1999q3
2000q1
2000q3
2001q1
2001q3
2002q1
2002q3
2003q1
2003q3
2004q1
2004q3
2005q1
2005q3
2006q1
2006q3
2007q1
2007q3
2008q1
2008q3
2009q1
2009q3
2010q1
2010q3
2011q1
2011q3
2012q1
2012q3
2013q1
2013q3
2014q1
2014q3
2015q1
Time (1998 Q1 - 2015 Q1)
NPLs % Lending interest rate %
Inflation (CP) % Unemloyment %
NPLs of Malaysia banking system over the business cycle (1998 Q1 - 2015 Q1)

24
Visualization Types

25
Social Network Analysis
Social network analysis (SNA) is the process of investigating
social structures through the use of network and graph
theories.
It characterizes networked structures in terms of nodes
(individual actors, people, or things within the network) and the
ties, edges, or links (relationships or interactions) that connect
them. Examples of social structures commonly visualized
through social network analysis include social media networks.

26
Example of Social Network Analysis
Data visualization of Facebook relationships

27
Network Theory Tools

28
What exactly is the meaning of an API?
Application Programming Interface (API)
Application Programming
Interface (API)
API is a particular set of rules (’code’)
and speciﬁcations that software
programs can follow to communicate
with each other.
It serves as an interface between
different software programs and
facilitates their interaction, similar to the
way the user interface facilitates
interaction between humans and
computers.

29
What exactly is the meaning of an API?
Application Programming Interface (API)
API is a set of subroutine definitions, protocols, and tools for building
application software.
It is a set of clearly defined methods of communication between
various software components. A good API makes it easier to develop
a computer program by providing all the building blocks, which are
then put together by the programmer.
An API may be for a web-based system, operating system, database
system, computer hardware or software library. An API specification
can take many forms, but often includes specifications for routines,
data structures, object classes, variables or remote calls.
Microsoft Windows API, the C++ Standard Template Library and Java
APIs are examples of different forms of APIs.

30
API

31
Example of web API
Shiny Weather Data
A web API is an application programming interface (API) for
either a web server or a web browser.
Shiny Weather Data is a web service making different sources of
European gridded climate data available in hourly time series
formats used by common building performance modeling tools.
This web service has been around for a while and has a steadily
growing user group of professional building modelers as well as
students and researchers.
satellite-based time series of solar irradiation for the actual
weather conditions as well as for clear-sky conditions
Portfolio Visualizer

32
Predictive Analytics
Predictive analytics is the branch of
the advanced analytics which is used to
make predictions about unknown future
events.
Predictive analytics uses many
techniques from data mining, statistics,
modeling, machine learning, and
artiﬁcial intelligence to analyze current
data to make predictions about future.

33
Predictive Analytics

34
Probability and Statistics
Probability is the measure of the likelihood that an event will
occur. Probability is quantiﬁed as a number between 0 and 1
(where 0 indicates impossibility and 1 indicates certainty). The
higher the probability of an event, the more certain that the event
will occur.
A simple example is the tossing of a coin. Since the coin is
unbiased, the two outcomes ("head" and "tail") are both equally
probable; the probability of "head" equals the probability of
"tail". Since no other outcomes are possible, the probability is
1/2 (or 50%), of either "head" or "tail".

35
Probability Theory
Probability Theory is the branch of mathematics concerned
with probability, the analysis of random phenomena.
The central objects of probability theory are random variables,
stochastic processes, and events: mathematical abstractions of
non-deterministic events or measured quantities that may either
be single occurrences or evolve over time in an apparently
random fashion.
Example

36
Statistics
Statistics as "a branch of mathematics dealing with the
collection, analysis, interpretation, and presentation of masses of
numerical data". Merriam-Webster dictionary.
In applying statistics to, e.g., a scientiﬁc, industrial, or social
problem, it is conventional to begin with a statistical population or
a statistical model process to be studied.
Populations can be diverse topics such as "all people living in a
country" or "every atom composing a crystal".
Statistics deals with all aspects of data including the planning of
data collection in terms of the design of surveys and
experiments.

37
Normal Distribution
Normal (or Gaussian) distribution is a very common continuous
probability distribution. Normal distributions are important in
statistics and are often used in the natural and social sciences to
represent real-valued random variables whose distributions are
not known.
LINK (Normal Distribution).

38
Normal Distribution
Probability density function
Figure: The red curve is the standard normal distribution

39
Other Distributions

40
p-value
The P value, or calculated probability, is the probability of ﬁnding the
observed, or more extreme, results when the null hypothesis (H0) of a
study question is true – the deﬁnition of ’extreme’ depends on how
the hypothesis is being tested.
- LINK.
- Seeing Theory website.

41
what is Regression Analysis?
Regression analysis is a form of predictive modelling technique
which investigates the relationship between a dependent (target)
and independent variable (s) (predictor).
This technique is used for forecasting, time series modelling and
finding the causal effect relationship between the variables. For
example, relationship between rash driving and number of road
accidents by a driver is best studied through regression.
Regression analysis is an important tool for modelling and
analyzing data.
There are multiple benefits of using regression analysis.
They are as follows:
*** It indicates the significant relationships between dependent
variable and independent variable.
*** It indicates the strength of impact of multiple independent
variables on a dependent variable.

42
Linear Regression
It is one of the most widely known
modeling technique. Linear
regression is usually among the
ﬁrst few topics which people pick
while learning predictive
modeling.
Linear Regression establishes a
relationship between dependent
variable (Y) and one or more
independent variables (X) using
a best ﬁt straight line (also
known as regression line).

43
Linear Regression. Cont.
It is represented by an equation
Y = α + βX + e, where a is
intercept, β is slope of the line
and e is error term. This equation
can be used to predict the value
of target variable based on given
predictor variable(s).

44
Data Modeling then Forecasting (Simulation
of the model) Example.
0
1
2
3
-1
-.5
0
.5
-.5
0
.5
1
-1
-.5
0
.5
1
-1
-.5
0
.5
-.05
0
.05
.1
0
.1
.2
.3
-.1
-.05
0
.05
-.15
-.1
-.05
0
.05
-.02
0
.02
.04
.06
-2
-1
0
1
-3
-2
-1
0
1
-1
0
1
2
-5
0
5
10
-2
-1
0
1
-.1
-.05
0
.05
-.1
-.05
0
.05
-.05
0
.05
.1
-.1
0
.1
.2
.3
-.1
-.05
0
.05
-.2
-.1
0
.1
0
.1
.2
.3
.4
-.2
-.1
0
.1
-.2
0
.2
.4
0
.1
.2
.3
0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30
M1 : M1
LENR : M1
CA^2 : M1
CA : M1
NPLs : M1
M1 : LENR
LENR : LENR
CA^2 : LENR
CA : LENR
NPLs : LENR
M1 : CA^2
LENR : CA^2
CA^2 : CA^2
CA : CA^2
NPLs : CA^2
M1 : CA
LENR : CA
CA^2 : CA
CA : CA
NPLs : CA
M1 : NPLs
LENR : NPLs
CA^2 : NPLs
CA : NPLs
NPLs : NPLs
95% CI Orthogonalized IRF
Step (1-month)
impulse : response. Generated by Monte-Carlo with 1000 reps. Based on VAR model.
Impulse-Response Functions

45
Back to R Programming
How to fetch stock data?
Example: How to fetch stock data?
Financial time series forecasting – an easy approach
Yahoo Finance

46
R - Linear Regression
Example
Linear Regression in R.

47
R - Linear Regression
Example
Advanced R

48
Artificial intelligence (AI)
Definition
AI is intelligence exhibited by machines. In computer science,
the field of AI research defines itself as the study of "intelligent
agents": any device that perceives its environment and takes
actions that maximize its chance of success at some goal.
The term "artificial intelligence" is applied when a machine
mimics "cognitive" functions that humans associate with other
human minds, such as "learning" and "problem solving" (known
as Machine Learning).
In August 2001, robots beat humans in a simulated financial
trading competition.

49
Artificial intelligence (AI)
List of programming languages for artificial intelligence
Definition
Python is widely used for Artificial Intelligence. They have a lot of
different AIs with corresponding packages: General AI, Machine
Learning, Natural Language Processing and Neural Networks.
Companies like Narrative Science use Python to create an
artificial intelligence for Narrative Language Processing.
MATLAB.
C++
.

50
Machine learning
Definition
Machine learning is the subfield of computer science that gives
computers the ability to learn without being explicitly
programmed. Evolved from the study of pattern recognition
and computational learning theory in artificial intelligence,
machine learning explores the study and construction of
algorithms that can learn from and make predictions on
data—such algorithms overcome following strictly static program
instructions by making data driven predictions or decisions,
through building a model from sample inputs.
Machine learning is employed in a range of computing tasks
where designing and programming explicit algorithms with good
performance is difficult or infeasible; example applications
include spam filtering, optical character recognition (OCR),
search engines and computer vision.

51
Machine learning
Deﬁnition +
Machine learning is a branch in computer science that studies
the design of algorithms that can learn. Typical machine learning
tasks are concept learning, function learning or “predictive
modeling”, clustering and ﬁnding predictive patterns.
These tasks are learned through available data that were
observed through experiences or instructions, for example.
Machine learning hopes that including the experience into its
tasks will eventually improve the learning. The ultimate goal is to
improve the learning in such a way that it becomes automatic, so
that humans like ourselves don’t need to interfere any more.

52
Machine learning
Figure: The machine learning process starts with raw data and ends up with
a model derived from that data.

53
Common Machine Learning Algorithms
Naïve Bayes Classiﬁer Algorithm
K Means Clustering Algorithm
Support Vector Machine Algorithm
Apriori Algorithm
Linear Regression
Logistic Regression
Artiﬁcial Neural Networks
Random Forests
Decision Trees
Nearest Neighbours (k-nearest neighbours "KNN" )

54
The Role of [R] in machine learning
Much of the work done by a data scientist involves statistics. For
example, machine learning algorithms commonly apply some
kind of statistical technique to prepared data.
But doing this kind of work can sometimes require programming.
What programming language is best for statistical computing?
The answer is clear: It’s the open-source language called R.
Created in New Zealand more than 20 years ago, R has
become the lingua franca for writing code in this area. In
fact, it’s hard to ﬁnd a data scientist who doesn’t know R.
Example: Machine Learning in R using (k-nearest neighbours)
algorithm.

55
Machine learning

56
Data mining
Definition
Data mining is the computational process of discovering
patterns in large data sets involving methods at the intersection
of artificial intelligence, machine learning, statistics, and
database systems.
It is an interdisciplinary subfield of computer science

57
Data mining
Deﬁnition 2
Data in digital form are available everywhere. It can be used to
predict the future. Usually the statistical approach is used. Data
mining is an extension of traditional data analysis and statistical
approaches in that it incorporates analytical techniques drawn
from a range of disciplines.
Data mining covers the entire process of data analysis,
including data cleaning and preparation and visualization of the
results, and how to produce predictions in real-time so that
speciﬁc goals are met.
Source

58
Data mining process and concept
Figure: Data mining is actually a part of the knowledge discovery process (KDD: knowledge
discovery from data). Data mining can be considered as a step in an iterative knowledge
discovery process which is shown in the above ﬁgure (Fayyad & Patetsky-Shapiro & Smith, 1996)

59
Data mining in "Risk Management"
Data mining creates models through data analysis and
prediction to help solve problems involving both project feasibility
and risk management.
Data mining has been used to analyze a database containing
information on a person’s history, achievements, and expertise.
The goal was to develop a proﬁle of the maturity of a certain
project involving the resource capacity, especially human capital.

60
Data mining tools

61
Data mining Cont.
Why Data Mining?
It helps to discover reasons for success and failure.
It helps to understand your customers, products etc.
It improves your organization by mining large sized databases.
SQL Data Mining Algorithms
Set of clusters illustrating how to relate the cases in dataset.
Decision Tree forecasts about the outcome and its after-effects.
Set of Rules explain how to group the products in a transaction.

62
World wide data
Move On To Big Data!!!

Big Data & Analytics (Conceptual and Practical Introduction)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Big Data & Analytics (Conceptual and Practical Introduction)

Similar to Big Data & Analytics (Conceptual and Practical Introduction) (20)

Recently uploaded

Recently uploaded (20)

Big Data & Analytics (Conceptual and Practical Introduction)