Master thesis

Platzhalter für Bild, Bild auf Titelfolie hinter das Logo einsetzen
Avinash Bapu Sreenivas
4359023
Numerical Investigation for Information
Tracking of Noisy and Non-smooth data in
Large-scale Statistics

Overview
 Introduction
 Regularization and its importance
 Total Variation (TV) regularization
• Principle
• Formula
• Importance
 Determination of regularization parameter
• Eye-balling (Trial and error) method
• L-curve method
• Normalized Cumulative Periodogram
• Generalized Cross Validation
 Summary
23. April 2021 | Referent | Information tracking of noisy data | Seite 2

Introduction
1. Understanding the term “large-scale data”
2. Complexities surrounding large-scale data
• Measurement
error
• Sample error
• Human error
Large scale data
Large volume
High sample size
High growth rate
Characteristics
Complexities
Statistical analysis of
large scale data
Tedious
Complex
Computationally
challenging
Reasons
These types of errors results in additional meaningless information termed as
“noise”.

3. Major consequence of noise in field of engineering,
Apply
Derivatives are
crucial in engineering
Finite difference
method
Noisy
data
Amplification
of noise
Therefore , regularization methods are employed
to overcome the amplification

Regularization and its importance
1. Aim of regularization
•to overcome the problem of overfitting of data arising from highly unstable RSS.
•achieved by the introduction of an additional term, known as penalty term.
2. General representation,
3. Detailed understanding of the aforementioned terms,
•Data fidelity term,
Derivative of “f” Penalty term Data fidelity
Residual Sum Square (RSS)

Contd.
• Penalty ,
Based on the penalty term , some of the well known regularization are mentioned below ,
Regularization parameter
Regularization term
Ridge (L2)
LASSO (L1)
[Least Absolute Shrinkage and
Selection Operator]
TV regularization

Comparison between L1 & L2 regularization
1. Geometric representation
L2
L1

Total Variation (TV) regularization
1. General form of TV regularization,
2. Basic principle of TV regularization
• General form of Euler-Lagrange equation,
RSS Penalty term
Derivative
of
solution ‘u’
a) Gradient descent (using
Euler-Lagrange equation)
b) Lagged diffusivity fixed point
method

Contd.
Applying Euler-Lagrange equation coupled with gradient descent method to TV
regularization,
• Final equation
Final equation
Solved Lagged diffusivity

Contd.
23. April 2021 | Referent | Kurztitel der Präsentation | Seite 10
4. Standard deviation increases Stronger parameter values for
satisfactory results
3. Importance of
• Jump discontinuities inclusion
• Determination of discontinuous derivative

Eye-balling technique
1. Trial and error method
2. Large scale data a) Inefficient
b) Infeasible
3.

1. Numerical and approximated function values of TF1
0 1 2 3 4 5 6 7
Data points
-1
-0.5
0
0.5
1
1.5
Function
values
Comparision between numerical function vs approximated function
known function
alpha=0.0005
alpha=0.21
alpha=5
Over-fit
Under-fit
Good-fit
Contd.

Test function 1
Data points
Given function
Standard deviation
of
AWGN
Noisy function
TEST FUNCTION 1
0 1 2 3 4 5 6
Data points
-1.5
-1
-0.5
0
0.5
1
1.5
Function
values
Graphical representation of tabular values
given data
noisy data

L-curve method
1. Estimation of L-curve parameter:
Determine
the solution
norm (a)
Calc to residual
norm (b)
Determine
log10 (a)
&
log10 (b)
For
each
RP
value
Plot a graph
(log10b , log10a)

Contd.
2. Estimation of curvature of L-curve graph:
Determine
curvature using
3 consecutive
points of a curve
distance
formula
Plot
Select

Results regarding TF1
Function values
Curvature plot

Results for Test function 2
Data points
Given function
Standard
deviation of
AWGN
Noisy function
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
-0.5
0
0.5
1
1.5
2
2.5
3
Function
values
given function
Noist function

Results for Test function 3
Data points
Given function
Standard
deviation of
AWGN
Noisy function
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Data points
-20
0
20
40
60
80
100
120
140
160
Function
points
given function
Noisy function

Contd.
Advantages and disadvantages of L-curve:
Advantages:
1. Highly robust method
2. Clear indication of over-fit and under-fit condition
3. “corner” balance between the two conditions
Disadvantages:
1. Special cases
2. Increase in problem size
2 “corners”
• Global corner
• New corner
Over-regularization
or
Large parameter value
Manually check for better
yielding solution

Normalized Cumulative Periodogram (NCP)
1. Aim and objective of NCP
 NCP mainly deals with residual vector ,
 Extraction of relevant information from data such that only noise remains in
 Optimal regularization parameter leads to the transition of from various signal
dominance to white-noise characteristics.
Discrete Fourier
transform of r
Periodogram
Length of residual
vector
NCP of residual
vector
For
each
RP
value
Discrete Fourier transform of r Periodogram
Length of residual vector
NCP of residual vector

Results for TF1
0 50 100 150 200 250 300
length of residual vector
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
NCP
values
NCP graph for various regularization parameter (RP)
RP(12)=0.56
RP(2)=0.06
RP(40)=1.96
RP(25)=1.21
RP(8)=0.36
RP(19)=0.91
RP(33)=1.61
RP(18)=0.86
RP(20)=0.96
RP(1)=0.01
Gaussian white noise characteristics

Contd.

Results for TF2 and TF3
0 50 100 150 200 250
length of residual vector
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
NCP
values
NCP graph for various regularization parameter
Each line indicates NCP values for 48 different RP values
Test function 2
Test function 3

Contd.
Advantages and disadvantages of NCP:
Advantages:
1. Computationally inexpensive
2. Deals with only residual vector More stable for high regularization parameter
Disadvantages:
1. Deals mainly with white noise
2. Dealing with different types of noise
NOTE : For a noise vector
Covariance matrix : Colored noise model
• Constant spectral density
• Random errors independent to each other
where,
Power spectral density
Frequency
White noise
Covariance matrix must known
White noise model

Generalized Cross Validation (GCV)
1. Objective of GCV:
 Helps determine the optimal regularization parameter even when,
a) exact data are unknown
b) No knowledge about noise variance
Hence proves to be an effective statistical tool.
2. Mathematics behind GCV:
 System definition :
 GCV is given by : where,
GCV Dataset Identity
matrix
Differentiation
operator
Noisy
data
Trace of
a matrix

Results of GCV
Test function 1 Test function 2
Optimal regularization parameter Minimum of GCV function
NOTE:

Advantages and disadvantages of GCV:
Advantages:
Unknown noise level
Disadvantages:
In small sample
• noise with smaller standard deviation is hard to capture
• Characterized by under-regularization (small parameter value)
• GCV can be employed
• Provides satisfactory parameter value

Application of TV regularization in data-driven process
Problem description
0 1 2 3 4 5 6 7 8 9 10
tspan
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
polynomial
v
l
a
ues
polynomial graph of the form
-0.3x+0.2x2
+0.1x
3
+0.5x
4
values

Contd.
Convergence and optimal parameter value

Contd.
Analytical values
Above graph proves that the optimal parameter
provides satisfactory results

Summary
1. Regularization is one of the best possibilities for tracking information from noisy data.
2. Total variation regularization is investigated thoroughly and the following conclusions are
drawn,
» Optimal regularization parameter plays a crucial role during the analysis of noisy data.
» Determination of optimal parameter is of paramount importance but complex.
» Different methods are tested to determine the optimal parameter value.
 L-curve
 Normalized Cumulative Periodogram (NCP)
 Generalized Cross Validation (GCV)
3. Data-driven system was simply employed to verify the results.
Differs in philosophy but
provides satisfactory values

Discussion
Thank you
all for
your patience

Appendix

Results for TF2
Derivative values
Function values

Results for TF3
Derivative values
Function values

2. Based on the penalty term , the following 2 properties are discussed below
 Robustness
•L1:
•L2:
 Computational effort
•L1:
•L2:
3. Sparsity
 Shrinkage of coefficients to zeros
more robust
less robust
L1-norm More computational effort
L2-norm Less computational effort

Description of test functions
Data points
Given function
Standard deviation
of
AWGN
Noisy function
TEST FUNCTION 1
0 1 2 3 4 5 6
Data points
-1.5
-1
-0.5
0
0.5
1
1.5
Function
values
given data
noisy data

Data points
Given function
Standard deviation
of
AWGN
Noisy function
TEST FUNCTION 2
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Data points
-0.5
0
0.5
1
1.5
2
2.5
3
Function
values
given function
Noist function

Data points
Given function
Standard deviation
of
AWGN
Noisy function
TEST FUNCTION 3
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Data points
0
50
100
150
Function
values
given function
noisy function

Generalized Cumulative Periodogram (GCV)
1. Objective of GCV:
 Helps determine the optimal regularization parameter even when,
a) exact data are unknown
b) No knowledge about noise variance
Hence proves to be an effective statistical tool.
2. Fundamental formula surrounding GCV:
GCV Dataset Identity
matrix
Noisy
data
Trace of
a matrix

Comparison of known and determined function using GCV for TF1
Satisfactory results

Concept behind Data-driven (sparse regression) method
1. Aim of data-driven
» Obtaining governing equation (PDE’s) in “spatiotemporal” existence.
2. Principle of data-driven
» Efficient determination of coefficients at a fixed spatial location.
3. General formulation of data-driven (sparse regression) method,
» Non-linear form of a PDE,
Unknowns
Partial derivative
w.r.t time or space
Parameters
(coefficients)

» Discretization of nonlinear PDE ,
Right hand side of
nonlinear PDE
Matrix consisting of
derivative parameters
coefficients of the PDE
Sparse vector
Parameter to be determined

Master thesis

Recommended

Recommended

More Related Content

Similar to Master thesis

Similar to Master thesis (20)

Recently uploaded

Recently uploaded (20)

Master thesis