This document discusses classical optimization theory and methods for finding extrema of functions. It defines local and global minima/maxima, and presents necessary and sufficient conditions for identifying extrema. Specifically, it states that a necessary condition is for the gradient to be zero at a point, while sufficient conditions involve the Hessian matrix being positive/negative definite. It then introduces the Newton-Raphson method for numerically solving the necessary condition of setting the gradient to zero. An example demonstrates applying Newton-Raphson to find an extremum of a sample function.

1. Classical Optimization
Theory
B SV P SuryaTeja
K Rohit
B SuryaTej
Mrudul Nekkanti

2. Find Out the type of extreme points in
the following figures.

3. Now what about this?
• This one has no
minima or maxima.
• The minima or
maxima are defined in
a specific region
which, in other words,
we call as constrained
problems.

4. What is minimum or maximum?
▪ Minimum, in mathematical terms, for a function can be defined as
▪ f (X0 + h) > f (X0) V X0 ∈ domain
▪ Similarly, maximum can be defined, for a function, as the following
▪ f (X0 + h) < f (X0) V X0∈ domain
▪ Here “h” is a small value and tends to zero.
▪ These are local maxima and minima, because we are not basing them
for the whole domain.

5. ▪ But if we take the smallest value of all the local minimas, then the
value is called a Global Minima.
▪ If we take the largest of all the local maximas, then it is called as
Global Maxima.

6. Necessary Conditions
▪ We are going to develop necessary and sufficient conditions for an n-
variable function f(X) to have extrema.
▪ It is assumed that the first and second order partial derivatives of f(X)
are continuous for all X.
▪ Theorem 1: A necessary condition for X0 to be an extreme point of f(X)
is that ∇ f(X0) = 0

7. Sufficient conditions
▪ Theorem: A sufficient condition for a stationary point X0 to be an
extreme point is that the Hessian matrix H evaluated at X0 satisfy the
following conditions
i. H is positive definite, if X0 is a minimum point
ii. H is negative definite, if X0 is a maximum point

8. Hessian Matrix

9. The Newton Raphson Method
▪ The necessary condition, sometimes ∇ f(X) = 0 , can be difficult to
solve numerically.
▪ So we use an iterative method called Newton Raphson method,
which helps solving simultaneous nonlinear equations.
▪ The method is mentioned in the next slides.

10. ▪ Consider the simultaneous equation f i(X) = 0, i = 1,2,3 … m
▪ ByTaylor’s expression at a given point Xk , we can write the whole
expression in the following form
f i(X) ~ f i(Xk) + ∇ fi(Xk)(X - Xk)
Changing the equation will give us the following expression
f i(Xk) + ∇ fi(Xk)(X - Xk) = 0
This can be written as Ak +Bk(X - Xk) = 0
OR X = Xk - B-1
k Ak (Bkis non singular)

11. ▪ The whole idea of this method is to start from an initial point and
then move on by using the above equation to find a point until it
converges.
▪ This process is done until 2 successive points are almost equal.
▪ For a single variable function this can be shown as
xk + 1 = xk -
𝒇(𝒙k)
𝒇′(𝒙k)

12. Example
Demonstrate Newton Raphson Method on the following
g(x) = (3x - 2)2(2x - 3)2
First find out f(x) = g’(x) = 72x3 - 234x2 + 241x – 78
Then follow the newton Raphson equation for a single variable that is
shown below.
xk + 1 = xk -
𝒇(𝒙k)
𝒇′(𝒙k)

13. Solving
k xk f(xk)/f'(xk) xk+1
0 10 2.967892314 7.032107686
1 7.032107686 1.97642875 5.055678936
2 5.055678936 1.314367243 3.741311693
3 3.741311693 0.871358025 2.869953668
4 2.869953668 0.573547408 2.29640626
5 2.29640626 0.371251989 1.925154272
6 1.925154272 0.230702166 1.694452106
7 1.694452106 0.128999578 1.565452528
8 1.565452528 0.054156405 1.511296123
9 1.511296123 0.010864068 1.500432055
10 1.500432055 0.000431385 1.50000067
11 1.50000067 6.70394E-07 1.5

14. ▪ It converges at 1.5
▪ Taking some other initial value we can converge at the other points.
Initial values 1 and 0.5 should give the other 2 extreme points.

15. questions

16. Thank You!