Java Programming Notes for Beginners by Arun Umraossuserd6b1fd
Â
Shot notes for quick revision. Not explained extensively but suitable for last night preparation. Fit for CBSE Class XII board students for their last minute preparation.
Think Like Scilab and Become a Numerical Programming Expert- Notes for Beginn...ssuserd6b1fd
Â
Notes for Scilab Programming. This notes includes the mathematics used behind scilab numerical programming. Illustrated with suitable graphics and examples. Each function is explained well with complete example. Helpful to beginners. GUI programming is also explained.
Basic ForTran Programming - for Beginners - An Introduction by Arun Umraossuserd6b1fd
Â
ForTran Programming Notes for Beginners. With complete example. Covers full syllabus. Well structured and commented for easy - self understanding. Good for Senior Secondary Students.
Assembly Codes in C Programmes - A Short Notes by Arun Umraossuserd6b1fd
Â
Notes for integration of Assembly language into C programming language. Suitable for novice programmer in C for embedded systems or emulators or simulators.
Java Programming Notes for Beginners by Arun Umraossuserd6b1fd
Â
Shot notes for quick revision. Not explained extensively but suitable for last night preparation. Fit for CBSE Class XII board students for their last minute preparation.
Think Like Scilab and Become a Numerical Programming Expert- Notes for Beginn...ssuserd6b1fd
Â
Notes for Scilab Programming. This notes includes the mathematics used behind scilab numerical programming. Illustrated with suitable graphics and examples. Each function is explained well with complete example. Helpful to beginners. GUI programming is also explained.
Basic ForTran Programming - for Beginners - An Introduction by Arun Umraossuserd6b1fd
Â
ForTran Programming Notes for Beginners. With complete example. Covers full syllabus. Well structured and commented for easy - self understanding. Good for Senior Secondary Students.
Assembly Codes in C Programmes - A Short Notes by Arun Umraossuserd6b1fd
Â
Notes for integration of Assembly language into C programming language. Suitable for novice programmer in C for embedded systems or emulators or simulators.
Notes and Description for Xcos Scilab Block Simulation with Suitable Examples...ssuserd6b1fd
Â
This notes on xcos are similar to Simulink of Matlab. Suitable for easy block drawing and simulations for systems or mathematical problems. This notes also explains about the working of blocks, mathematics behind them - used for block computation - their structures and input-output data.
Vector spaces over a general field are treated and used to create interesting and practical algebras and geometries. Quotient spaces and dual spaces are emphasized. Algebras and geometries based on vector spaces are introduced. In particular, the exterior algebra of a vector space is constructed and given geometric interpretation. Scalar product spaces are introduced for scalar products of general type, and the Hodge star is treated two different ways for scalar product spaces. The topics are presented in a manner that facilitates self-study through inline exercises interspersed throughout. A basic algebra background is required for successful study of the material; readers should know what an Abelian group is and be able to do basic matrix algebra.
Notes and Description for Xcos Scilab Block Simulation with Suitable Examples...ssuserd6b1fd
Â
This notes on xcos are similar to Simulink of Matlab. Suitable for easy block drawing and simulations for systems or mathematical problems. This notes also explains about the working of blocks, mathematics behind them - used for block computation - their structures and input-output data.
Vector spaces over a general field are treated and used to create interesting and practical algebras and geometries. Quotient spaces and dual spaces are emphasized. Algebras and geometries based on vector spaces are introduced. In particular, the exterior algebra of a vector space is constructed and given geometric interpretation. Scalar product spaces are introduced for scalar products of general type, and the Hodge star is treated two different ways for scalar product spaces. The topics are presented in a manner that facilitates self-study through inline exercises interspersed throughout. A basic algebra background is required for successful study of the material; readers should know what an Abelian group is and be able to do basic matrix algebra.
Partial differential equations, graduate level problems and solutions by igor...Julio Banks
Â
The physical world is driving by the laws of mathematics, more specifically PDE (Partial Differential Equations). FEA (Finite Element Analysis) and CFD (Computation Fluid Dynamics) are the numerical methods utilized to model physical events described by PDEs.
Stochastic Processes and Simulations â A Machine Learning Perspectivee2wi67sy4816pahn
Â
Written for machine learning practitioners, software engineers and other analytic professionals interested in expanding their toolset and mastering the art. Discover state-of-the-art techniques explained in simple English, applicable to many modern problems, especially related to spatial processes and pattern recognition. This textbook includes numerous visualization techniques (for instance, data animations using video libraries in R), a true test of independence, simple illustration of dual confidence regions (more intuitive than the classic version), minimum contrast estimation (a simple generic estimation technique encompassing maximum likelihood), model fitting techniques, and much more. The scope of the material extends far beyond stochastic processes.
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Â
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...Amil Baba Dawood bangali
Â
Contact with Dawood Bhai Just call on +92322-6382012 and we'll help you. We'll solve all your problems within 12 to 24 hours and with 101% guarantee and with astrology systematic. If you want to take any personal or professional advice then also you can call us on +92322-6382012 , ONLINE LOVE PROBLEM & Other all types of Daily Life Problem's.Then CALL or WHATSAPP us on +92322-6382012 and Get all these problems solutions here by Amil Baba DAWOOD BANGALI
#vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore#blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #blackmagicforlove #blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #Amilbabainuk #amilbabainspain #amilbabaindubai #Amilbabainnorway #amilbabainkrachi #amilbabainlahore #amilbabaingujranwalan #amilbabainislamabad
Explore the innovative world of trenchless pipe repair with our comprehensive guide, "The Benefits and Techniques of Trenchless Pipe Repair." This document delves into the modern methods of repairing underground pipes without the need for extensive excavation, highlighting the numerous advantages and the latest techniques used in the industry.
Learn about the cost savings, reduced environmental impact, and minimal disruption associated with trenchless technology. Discover detailed explanations of popular techniques such as pipe bursting, cured-in-place pipe (CIPP) lining, and directional drilling. Understand how these methods can be applied to various types of infrastructure, from residential plumbing to large-scale municipal systems.
Ideal for homeowners, contractors, engineers, and anyone interested in modern plumbing solutions, this guide provides valuable insights into why trenchless pipe repair is becoming the preferred choice for pipe rehabilitation. Stay informed about the latest advancements and best practices in the field.
Cosmetic shop management system project report.pdfKamal Acharya
Â
Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry.
Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks.
Data file handling has been effectively used in the program.
The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.
2. The Hong Kong University of Science and Technology
Department of Mathematics
Clear Water Bay, Kowloon
Hong Kong
Copyright câ 2012 by Jeffrey Robert Chasnov
This work is licensed under the Creative Commons Attribution 3.0 Hong Kong License.
To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/hk/
or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco,
California, 94105, USA.
3. Preface
What follows are my lecture notes for Math 3311: Introduction to Numerical
Methods, taught at the Hong Kong University of Science and Technology. Math
3311, with two lecture hours per week, is primarily for non-mathematics majors
and is required by several engineering departments.
All web surfers are welcome to download these notes at
http://www.math.ust.hk/~machas/numerical-methods.pdf
and to use the notes freely for teaching and learning. I welcome any comments,
suggestions or corrections sent by email to jeffrey.chasnov@ust.hk.
iii
9. 1.7. SPECIAL NUMBERS 3
1.7 Special numbers
Smallest exponent: e = 0000 0000, represents denormal numbers (1.f â 0.f)
Largest exponent: e = 1111 1111, represents 屉, if f = 0
e = 1111 1111, represents NaN, if f Ìž= 0
Number Range: e = 1111 1111 = 28
- 1 = 255 reserved
e = 0000 0000 = 0 reserved
so, p = e - 127 is
1 - 127 †p †254-127
-126 †p †127
Smallest positive normal number
= 1.0000 0000 · · · · ·· 0000à 2â126
â 1.2 Ã 10â38
bin: 0000 0000 1000 0000 0000 0000 0000 0000
hex: 00800000
MATLAB: realmin(âsingleâ)
Largest positive number
= 1.1111 1111 · · · · ·· 1111à 2127
= (1 + (1 â 2â23
)) Ã 2127
â 2128
â 3.4 Ã 1038
bin: 0111 1111 0111 1111 1111 1111 1111 1111
hex: 7f7fffff
MATLAB: realmax(âsingleâ)
Zero
bin: 0000 0000 0000 0000 0000 0000 0000 0000
hex: 00000000
Subnormal numbers
Allow 1.f â 0.f (in software)
Smallest positive number = 0.0000 0000 · · · · · 0001 à 2â126
= 2â23
à 2â126
â 1.4 Ã 10â45
1.8 Examples of computer numbers
What is 1.0, 2.0 & 1/2 in hex ?
1.0 = (â1)0
à 2(127â127)
à 1.0
Therefore, ð = 0, ð = 0111 1111, ð = 0000 0000 0000 0000 0000 000
bin: 0011 1111 1000 0000 0000 0000 0000 0000
hex: 3f80 0000
2.0 = (â1)0
à 2(128â127)
à 1.0
Therefore, ð = 0, ð = 1000 0000, ð = 0000 0000 0000 0000 0000 000
bin: 0100 00000 1000 0000 0000 0000 0000 0000
hex: 4000 0000
10. 4 CHAPTER 1. IEEE ARITHMETIC
1/2 = (â1)0
à 2(126â127)
à 1.0
Therefore, ð = 0, ð = 0111 1110, ð = 0000 0000 0000 0000 0000 000
bin: 0011 1111 0000 0000 0000 0000 0000 0000
hex: 3f00 0000
1.9 Inexact numbers
Example:
1
3
= (â1)0
Ã
1
4
à (1 +
1
3
),
so that ð = ð â 127 = â2 and ð = 125 = 128 â 3, or in binary, ð = 0111 1101.
How is ð = 1/3 represented in binary? To compute binary number, multiply
successively by 2 as follows:
0.333 . . . 0.
0.666 . . . 0.0
1.333 . . . 0.01
0.666 . . . 0.010
1.333 . . . 0.0101
etc.
so that 1/3 exactly in binary is 0.010101 . . . . With only 23 bits to represent ð,
the number is inexact and we have
ð = 01010101010101010101011,
where we have rounded to the nearest binary number (here, rounded up). The
machine number 1/3 is then represented as
00111110101010101010101010101011
or in hex
3ððððððð.
1.9.1 Find smallest positive integer that is not exact in
single precision
Let ð be the smallest positive integer that is not exact. Now, I claim that
ð â 2 = 223
à 1.11 . . . 1,
and
ð â 1 = 224
à 1.00 . . . 0.
The integer ð would then require a one-bit in the 2â24
position, which is not
available. Therefore, the smallest positive integer that is not exact is 224
+ 1 =
16 777 217. In MATLAB, single(224
) has the same value as single(224
+1). Since
single(224
+1) is exactly halfway between the two consecutive machine numbers
224
and 224
+2, MATLAB rounds to the number with a final zero-bit in f, which
is 224
.
11. 1.10. MACHINE EPSILON 5
1.10 Machine epsilon
Machine epsilon (ðmach) is the distance between 1 and the next largest number.
If 0 †ð¿ < ðmach/2, then 1 + ð¿ = 1 in computer math. Also since
ð¥ + ðŠ = ð¥(1 + ðŠ/ð¥),
if 0 †ðŠ/ð¥ < ðmach/2, then ð¥ + ðŠ = ð¥ in computer math.
Find ðmach
The number 1 in the IEEE format is written as
1 = 20
à 1.000 . . . 0,
with 23 0âs following the binary point. The number just larger than 1 has a 1
in the 23rd position after the decimal point. Therefore,
ðmach = 2â23
â 1.192 Ã 10â7
.
What is the distance between 1 and the number just smaller than 1? Here,
the number just smaller than one can be written as
2â1
à 1.111 . . . 1 = 2â1
(1 + (1 â 2â23
)) = 1 â 2â24
Therefore, this distance is 2â24
= ðmach/2.
The spacing between numbers is uniform between powers of 2, with logarith-
mic spacing of the powers of 2. That is, the spacing of numbers between 1 and
2 is 2â23
, between 2 and 4 is 2â22
, between 4 and 8 is 2â21
, etc. This spacing
changes for denormal numbers, where the spacing is uniform all the way down
to zero.
Find the machine number just greater than 5
A rough estimate would be 5(1+ ðmach) = 5+5ðmach, but this is not exact. The
exact answer can be found by writing
5 = 22
(1 +
1
4
),
so that the next largest number is
22
(1 +
1
4
+ 2â23
) = 5 + 2â21
= 5 + 4ðmach.
1.11 IEEE double precision format
Most computations take place in double precision, where round-off error is re-
duced, and all of the above calculations in single precision can be repeated for
double precision. The format is
ð
â â
0
ð
â â
1 2 3 4 5 6 7 8 9 1011
ð
â â
12
· · · · · · · ·
63
12. 6 CHAPTER 1. IEEE ARITHMETIC
# = (â1) ð
à 2 ðâ1023
à 1.f
where s = sign
e = biased exponent
p=e-1023 = exponent
1.f = significand (use binary point)
1.12 Roundoff error example
Consider solving the quadratic equation
ð¥2
+ 2ðð¥ â 1 = 0,
where ð is a parameter. The quadratic formula yields the two solutions
ð¥Â± = âð ±
âïž
ð2 + 1.
Consider the solution with ð > 0 and ð¥ > 0 (the ð¥+ solution) given by
ð¥ = âð +
âïž
ð2 + 1. (1.1)
As ð â â,
ð¥ = âð +
âïž
ð2 + 1
= âð + ð
âïž
1 + 1/ð2
= ð(
âïž
1 + 1/ð2 â 1)
â ð
(ïž
1 +
1
2ð2
â 1
)ïž
=
1
2ð
.
Now in double precision, realmin â 2.2 Ã 10â308
and we would like ð¥ to be
accurate to this value before it goes to 0 via denormal numbers. Therefore,
ð¥ should be computed accurately to ð â 1/(2 à realmin) â 2 à 10307
. What
happens if we compute (1.1) directly? Then ð¥ = 0 when ð2
+ 1 = ð2
, or
1 + 1/ð2
= 1. That is 1/ð2
= ðmach/2, or ð =
â
2/
â
ðmach â 108
.
For a subroutine written to compute the solution of a quadratic for a general
user, this is not good enough. The way for a software designer to solve this
problem is to compute the solution for ð¥ as
ð¥ =
1
ð +
â
ð2 + 1
.
In this form, if ð2
+ 1 = ð2
, then ð¥ = 1/2ð which is the correct asymptotic form.
13. Chapter 2
Root Finding
Solve ð(ð¥) = 0 for ð¥, when an explicit analytical solution is impossible.
2.1 Bisection Method
The bisection method is the easiest to numerically implement and almost always
works. The main disadvantage is that convergence is slow. If the bisection
method results in a computer program that runs too slow, then other faster
methods may be chosen; otherwise it is a good choice of method.
We want to construct a sequence ð¥0, ð¥1, ð¥2, . . . that converges to the root
ð¥ = ð that solves ð(ð¥) = 0. We choose ð¥0 and ð¥1 such that ð¥0 < ð < ð¥1. We
say that ð¥0 and ð¥1 bracket the root. With ð(ð) = 0, we want ð(ð¥0) and ð(ð¥1)
to be of opposite sign, so that ð(ð¥0)ð(ð¥1) < 0. We then assign ð¥2 to be the
midpoint of ð¥0 and ð¥1, that is ð¥2 = (ð¥0 + ð¥1)/2, or
ð¥2 = ð¥0 +
ð¥1 â ð¥0
2
.
The sign of ð(ð¥2) can then be determined. The value of ð¥3 is then chosen as
either the midpoint of ð¥0 and ð¥2 or as the midpoint of ð¥2 and ð¥1, depending on
whether ð¥0 and ð¥2 bracket the root, or ð¥2 and ð¥1 bracket the root. The root,
therefore, stays bracketed at all times. The algorithm proceeds in this fashion
and is typically stopped when the increment to the left side of the bracket
(above, given by (ð¥1 â ð¥0)/2) is smaller than some required precision.
2.2 Newtonâs Method
This is the fastest method, but requires analytical computation of the derivative
of ð(ð¥). Also, the method may not always converge to the desired root.
We can derive Newtonâs Method graphically, or by a Taylor series. We again
want to construct a sequence ð¥0, ð¥1, ð¥2, . . . that converges to the root ð¥ = ð.
Consider the ð¥ ð+1 member of this sequence, and Taylor series expand ð(ð¥ ð+1)
about the point ð¥ ð. We have
ð(ð¥ ð+1) = ð(ð¥ ð) + (ð¥ ð+1 â ð¥ ð)ðâ²
(ð¥ ð) + . . . .
7
14. 8 CHAPTER 2. ROOT FINDING
To determine ð¥ ð+1, we drop the higher-order terms in the Taylor series, and
assume ð(ð¥ ð+1) = 0. Solving for ð¥ ð+1, we have
ð¥ ð+1 = ð¥ ð â
ð(ð¥ ð)
ðâ²(ð¥ ð)
.
Starting Newtonâs Method requires a guess for ð¥0, hopefully close to the root
ð¥ = ð.
2.3 Secant Method
The Secant Method is second best to Newtonâs Method, and is used when a
faster convergence than Bisection is desired, but it is too difficult or impossible
to take an analytical derivative of the function ð(ð¥). We write in place of ðâ²
(ð¥ ð),
ðâ²
(ð¥ ð) â
ð(ð¥ ð) â ð(ð¥ ðâ1)
ð¥ ð â ð¥ ðâ1
.
Starting the Secant Method requires a guess for both ð¥0 and ð¥1.
2.3.1 Estimate
â
2 = 1.41421356 using Newtonâs Method
The
â
2 is the zero of the function ð(ð¥) = ð¥2
â 2. To implement Newtonâs
Method, we use ðâ²
(ð¥) = 2ð¥. Therefore, Newtonâs Method is the iteration
ð¥ ð+1 = ð¥ ð â
ð¥2
ð â 2
2ð¥ ð
.
We take as our initial guess ð¥0 = 1. Then
ð¥1 = 1 â
â1
2
=
3
2
= 1.5,
ð¥2 =
3
2
â
9
4 â 2
3
=
17
12
= 1.416667,
ð¥3 =
17
12
â
172
122 â 2
17
6
=
577
408
= 1.41426.
2.3.2 Example of fractals using Newtonâs Method
Consider the complex roots of the equation ð(ð§) = 0, where
ð(ð§) = ð§3
â 1.
These roots are the three cubic roots of unity. With
ðð2ðð
= 1, ð = 0, 1, 2, . . .
the three unique cubic roots of unity are given by
1, ðð2ð/3
, ðð4ð/3
.
With
ððð
= cos ð + ð sin ð,
15. 2.4. ORDER OF CONVERGENCE 9
and cos (2ð/3) = â1/2, sin (2ð/3) =
â
3/2, the three cubic roots of unity are
ð1 = 1, ð2 = â
1
2
+
â
3
2
ð, ð3 = â
1
2
â
â
3
2
ð.
The interesting idea here is to determine which initial values of ð§0 in the complex
plane converge to which of the three cubic roots of unity.
Newtonâs method is
ð§ ð+1 = ð§ ð â
ð§3
ð â 1
3ð§2
ð
.
If the iteration converges to ð1, we color ð§0 red; ð2, blue; ð3, green. The result
will be shown in lecture.
2.4 Order of convergence
Let ð be the root and ð¥ ð be the ðth approximation to the root. Define the error
as
ð ð = ð â ð¥ ð.
If for large ð we have the approximate relationship
|ð ð+1| = ð|ð ð| ð
,
with ð a positive constant, then we say the root-finding numerical method is of
order ð. Larger values of ð correspond to faster convergence to the root. The
order of convergence of bisection is one: the error is reduced by approximately
a factor of 2 with each iteration so that
|ð ð+1| =
1
2
|ð ð|.
We now find the order of convergence for Newtonâs Method and for the Secant
Method.
2.4.1 Newtonâs Method
We start with Newtonâs Method
ð¥ ð+1 = ð¥ ð â
ð(ð¥ ð)
ðâ²(ð¥ ð)
.
Subtracting both sides from ð, we have
ð â ð¥ ð+1 = ð â ð¥ ð +
ð(ð¥ ð)
ðâ²(ð¥ ð)
,
or
ð ð+1 = ð ð +
ð(ð¥ ð)
ðâ²(ð¥ ð)
. (2.1)
16. 10 CHAPTER 2. ROOT FINDING
We use Taylor series to expand the functions ð(ð¥ ð) and ðâ²
(ð¥ ð) about the root
ð, using ð(ð) = 0. We have
ð(ð¥ ð) = ð(ð) + (ð¥ ð â ð)ðâ²
(ð) +
1
2
(ð¥ ð â ð)2
ðâ²â²
(ð) + . . . ,
= âð ð ðâ²
(ð) +
1
2
ð2
ð ðâ²â²
(ð) + . . . ;
ðâ²
(ð¥ ð) = ðâ²
(ð) + (ð¥ ð â ð)ðâ²â²
(ð) +
1
2
(ð¥ ð â ð)2
ðâ²â²â²
(ð) + . . . ,
= ðâ²
(ð) â ð ð ðâ²â²
(ð) +
1
2
ð2
ð ðâ²â²â²
(ð) + . . . .
(2.2)
To make further progress, we will make use of the following standard Taylor
series:
1
1 â ð
= 1 + ð + ð2
+ . . . , (2.3)
which converges for |ð| < 1. Substituting (2.2) into (2.1), and using (2.3) yields
ð ð+1 = ð ð +
ð(ð¥ ð)
ðâ²(ð¥ ð)
= ð ð +
âð ð ðâ²
(ð) + 1
2 ð2
ð ðâ²â²
(ð) + . . .
ðâ²(ð) â ð ð ðâ²â²(ð) + 1
2 ð2
ð ðâ²â²â²(ð) + . . .
= ð ð +
âð ð + 1
2 ð2
ð
ðâ²â²
(ð)
ðâ²(ð) + . . .
1 â ð ð
ðâ²â²(ð)
ðâ²(ð) + . . .
= ð ð +
(ïž
âð ð +
1
2
ð2
ð
ðâ²â²
(ð)
ðâ²(ð)
+ . . .
)ïž (ïž
1 + ð ð
ðâ²â²
(ð)
ðâ²(ð)
+ . . .
)ïž
= ð ð +
(ïž
âð ð + ð2
ð
(ïž
1
2
ðâ²â²
(ð)
ðâ²(ð)
â
ðâ²â²
(ð)
ðâ²(ð)
)ïž
+ . . .
)ïž
= â
1
2
ðâ²â²
(ð)
ðâ²(ð)
ð2
ð + . . .
Therefore, we have shown that
|ð ð+1| = ð|ð ð|2
as ð â â, with
ð =
1
2
â
â
â
â
ðâ²â²
(ð)
ðâ²(ð)
â
â
â
â ,
provided ðâ²
(ð) Ìž= 0. Newtonâs method is thus of order 2 at simple roots.
2.4.2 Secant Method
Determining the order of the Secant Method proceeds in a similar fashion. We
start with
ð¥ ð+1 = ð¥ ð â
(ð¥ ð â ð¥ ðâ1)ð(ð¥ ð)
ð(ð¥ ð) â ð(ð¥ ðâ1)
.
We subtract both sides from ð and make use of
ð¥ ð â ð¥ ðâ1 = (ð â ð¥ ðâ1) â (ð â ð¥ ð)
= ð ðâ1 â ð ð,
17. 2.4. ORDER OF CONVERGENCE 11
and the Taylor series
ð(ð¥ ð) = âð ð ðâ²
(ð) +
1
2
ð2
ð ðâ²â²
(ð) + . . . ,
ð(ð¥ ðâ1) = âð ðâ1 ðâ²
(ð) +
1
2
ð2
ðâ1 ðâ²â²
(ð) + . . . ,
so that
ð(ð¥ ð) â ð(ð¥ ðâ1) = (ð ðâ1 â ð ð)ðâ²
(ð) +
1
2
(ð2
ð â ð2
ðâ1)ðâ²â²
(ð) + . . .
= (ð ðâ1 â ð ð)
(ïž
ðâ²
(ð) â
1
2
(ð ðâ1 + ð ð)ðâ²â²
(ð) + . . .
)ïž
.
We therefore have
ð ð+1 = ð ð +
âð ð ðâ²
(ð) + 1
2 ð2
ð ðâ²â²
(ð) + . . .
ðâ²(ð) â 1
2 (ð ðâ1 + ð ð)ðâ²â²(ð) + . . .
= ð ð â ð ð
1 â 1
2 ð ð
ðâ²â²
(ð)
ðâ²(ð) + . . .
1 â 1
2 (ð ðâ1 + ð ð) ðâ²â²(ð)
ðâ²(ð) + . . .
= ð ð â ð ð
(ïž
1 â
1
2
ð ð
ðâ²â²
(ð)
ðâ²(ð)
+ . . .
)ïž (ïž
1 +
1
2
(ð ðâ1 + ð ð)
ðâ²â²
(ð)
ðâ²(ð)
+ . . .
)ïž
= â
1
2
ðâ²â²
(ð)
ðâ²(ð)
ð ðâ1 ð ð + . . . ,
or to leading order
|ð ð+1| =
1
2
â
â
â
â
ðâ²â²
(ð)
ðâ²(ð)
â
â
â
â |ð ðâ1||ð ð|. (2.4)
The order of convergence is not yet obvious from this equation, and to determine
the scaling law we look for a solution of the form
|ð ð+1| = ð|ð ð| ð
.
From this ansatz, we also have
|ð ð| = ð|ð ðâ1| ð
,
and therefore
|ð ð+1| = ð ð+1
|ð ðâ1| ð2
.
Substitution into (2.4) results in
ð ð+1
|ð ðâ1| ð2
=
ð
2
â
â
â
â
ðâ²â²
(ð)
ðâ²(ð)
â
â
â
â |ð ðâ1| ð+1
.
Equating the coefficient and the power of ð ðâ1 results in
ð ð
=
1
2
â
â
â
â
ðâ²â²
(ð)
ðâ²(ð)
â
â
â
â ,
and
ð2
= ð + 1.
18. 12 CHAPTER 2. ROOT FINDING
The order of convergence of the Secant Method, given by ð, therefore is deter-
mined to be the positive root of the quadratic equation ð2
â ð â 1 = 0, or
ð =
1 +
â
5
2
â 1.618,
which coincidentally is a famous irrational number that is called The Golden
Ratio, and goes by the symbol Ί. We see that the Secant Method has an order
of convergence lying between the Bisection Method and Newtonâs Method.
19. Chapter 3
Systems of equations
Consider the system of ð linear equations and ð unknowns, given by
ð11 ð¥1 + ð12 ð¥2 + · · · + ð1ð ð¥ ð = ð1,
ð21 ð¥1 + ð22 ð¥2 + · · · + ð2ð ð¥ ð = ð2,
...
...
ð ð1 ð¥1 + ð ð2 ð¥2 + · · · + ð ðð ð¥ ð = ð ð.
We can write this system as the matrix equation
Ax = b,
with
A =
â
â
â
â
â
ð11 ð12 · · · ð1ð
ð21 ð22 · · · ð2ð
...
...
...
...
ð ð1 ð ð2 · · · ð ðð
â
â
â
â
â
, x =
â
â
â
â
â
ð¥1
ð¥2
...
ð¥ ð
â
â
â
â
â
, b =
â
â
â
â
â
ð1
ð2
...
ð ð
â
â
â
â
â
.
3.1 Gaussian Elimination
The standard numerical algorithm to solve a system of linear equations is called
Gaussian Elimination. We can illustrate this algorithm by example.
Consider the system of equations
â3ð¥1 + 2ð¥2 â ð¥3 = â1,
6ð¥1 â 6ð¥2 + 7ð¥3 = â7,
3ð¥1 â 4ð¥2 + 4ð¥3 = â6.
To perform Gaussian elimination, we form an Augmented Matrix by combining
the matrix A with the column vector b:
â
â
â3 2 â1 â1
6 â6 7 â7
3 â4 4 â6
â
â .
Row reduction is then performed on this matrix. Allowed operations are (1)
multiply any row by a constant, (2) add multiple of one row to another row, (3)
13
20. 14 CHAPTER 3. SYSTEMS OF EQUATIONS
interchange the order of any rows. The goal is to convert the original matrix
into an upper-triangular matrix.
We start with the first row of the matrix and work our way down as follows.
First we multiply the first row by 2 and add it to the second row, and add the
first row to the third row:
â
â
â3 2 â1 â1
0 â2 5 â9
0 â2 3 â7
â
â .
We then go to the second row. We multiply this row by â1 and add it to the
third row: â
â
â3 2 â1 â1
0 â2 5 â9
0 0 â2 2
â
â .
The resulting equations can be determined from the matrix and are given by
â3ð¥1 + 2ð¥2 â ð¥3 = â1
â2ð¥2 + 5ð¥3 = â9
â2ð¥3 = 2.
These equations can be solved by backward substitution, starting from the last
equation and working backwards. We have
â2ð¥3 = 2 â ð¥3 = â1
â2ð¥2 = â9 â 5ð¥3 = â4 â ð¥2 = 2,
â3ð¥1 = â1 â 2ð¥2 + ð¥3 = â6 â ð¥1 = 2.
Therefore, â
â
ð¥1
ð¥2
ð¥3
â
â =
â
â
2
2
â1
â
â .
3.2 ð¿ð decomposition
The process of Gaussian Elimination also results in the factoring of the matrix
A to
A = LU,
where L is a lower triangular matrix and U is an upper triangular matrix.
Using the same matrix A as in the last section, we show how this factorization
is realized. We have
â
â
â3 2 â1
6 â6 7
3 â4 4
â
â â
â
â
â3 2 â1
0 â2 5
0 â2 3
â
â = M1A,
where
M1A =
â
â
1 0 0
2 1 0
1 0 1
â
â
â
â
â3 2 â1
6 â6 7
3 â4 4
â
â =
â
â
â3 2 â1
0 â2 5
0 â2 3
â
â .
21. 3.2. ð¿ð DECOMPOSITION 15
Note that the matrix M1 performs row elimination on the first column. Two
times the first row is added to the second row and one times the first row is
added to the third row. The entries of the column of M1 come from 2 = â(6/â3)
and 1 = â(3/â3) as required for row elimination. The number â3 is called the
pivot.
The next step is
â
â
â3 2 â1
0 â2 5
0 â2 3
â
â â
â
â
â3 2 â1
0 â2 5
0 0 â2
â
â = M2(M1A),
where
M2(M1A) =
â
â
1 0 0
0 1 0
0 â1 1
â
â
â
â
â3 2 â1
0 â2 5
0 â2 3
â
â =
â
â
â3 2 â1
0 â2 5
0 0 â2
â
â .
Here, M2 multiplies the second row by â1 = â(â2/ â 2) and adds it to the
third row. The pivot is â2.
We now have
M2M1A = U
or
A = Mâ1
1 Mâ1
2 U.
The inverse matrices are easy to find. The matrix M1 multiples the first row by
2 and adds it to the second row, and multiplies the first row by 1 and adds it
to the third row. To invert these operations, we need to multiply the first row
by â2 and add it to the second row, and multiply the first row by â1 and add
it to the third row. To check, with
M1Mâ1
1 = I,
we have â
â
1 0 0
2 1 0
1 0 1
â
â
â
â
1 0 0
â2 1 0
â1 0 1
â
â =
â
â
1 0 0
0 1 0
0 0 1
â
â .
Similarly,
Mâ1
2 =
â
â
1 0 0
0 1 0
0 1 1
â
â
Therefore,
L = Mâ1
1 Mâ1
2
is given by
L =
â
â
1 0 0
â2 1 0
â1 0 1
â
â
â
â
1 0 0
0 1 0
0 1 1
â
â =
â
â
1 0 0
â2 1 0
â1 1 1
â
â ,
which is lower triangular. The off-diagonal elements of Mâ1
1 and Mâ1
2 are simply
combined to form L. Our LU decomposition is therefore
â
â
â3 2 â1
6 â6 7
3 â4 4
â
â =
â
â
1 0 0
â2 1 0
â1 1 1
â
â
â
â
â3 2 â1
0 â2 5
0 0 â2
â
â .
22. 16 CHAPTER 3. SYSTEMS OF EQUATIONS
Another nice feature of the LU decomposition is that it can be done by over-
writing A, therefore saving memory if the matrix A is very large.
The LU decomposition is useful when one needs to solve Ax = b for x when
A is fixed and there are many different bâs. First one determines L and U using
Gaussian elimination. Then one writes
(LU)x = L(Ux) = b.
We let
y = Ux,
and first solve
Ly = b
for y by forward substitution. We then solve
Ux = y
for x by backward substitution. When we count operations, we will see that
solving (LU)x = b is significantly faster once L and U are in hand than solving
Ax = b directly by Gaussian elimination.
We now illustrate the solution of LUx = b using our previous example,
where
L =
â
â
1 0 0
â2 1 0
â1 1 1
â
â , U =
â
â
â3 2 â1
0 â2 5
0 0 â2
â
â , b =
â
â
â1
â7
â6
â
â .
With y = Ux, we first solve Ly = b, that is
â
â
1 0 0
â2 1 0
â1 1 1
â
â
â
â
ðŠ1
ðŠ2
ðŠ3
â
â =
â
â
â1
â7
â6
â
â .
Using forward substitution
ðŠ1 = â1,
ðŠ2 = â7 + 2ðŠ1 = â9,
ðŠ3 = â6 + ðŠ1 â ðŠ2 = 2.
We now solve Ux = y, that is
â
â
â3 2 â1
0 â2 5
0 0 â2
â
â
â
â
ð¥1
ð¥2
ð¥3
â
â =
â
â
â1
â9
2
â
â .
Using backward substitution,
â2ð¥3 = 2 â ð¥3 = â1,
â2ð¥2 = â9 â 5ð¥3 = â4 â ð¥2 = 2,
â3ð¥1 = â1 â 2ð¥2 + ð¥3 = â6 â ð¥1 = 2,
and we have once again determined
â
â
ð¥1
ð¥2
ð¥3
â
â =
â
â
2
2
â1
â
â .
23. 3.3. PARTIAL PIVOTING 17
3.3 Partial pivoting
When performing Gaussian elimination, the diagonal element that one uses
during the elimination procedure is called the pivot. To obtain the correct
multiple, one uses the pivot as the divisor to the elements below the pivot.
Gaussian elimination in this form will fail if the pivot is zero. In this situation,
a row interchange must be performed.
Even if the pivot is not identically zero, a small value can result in big round-
off errors. For very large matrices, one can easily lose all accuracy in the solution.
To avoid these round-off errors arising from small pivots, row interchanges are
made, and this technique is called partial pivoting (partial pivoting is in contrast
to complete pivoting, where both rows and columns are interchanged). We will
illustrate by example the LU decomposition using partial pivoting.
Consider
A =
â
â
â2 2 â1
6 â6 7
3 â8 4
â
â .
We interchange rows to place the largest element (in absolute value) in the pivot,
or ð11, position. That is,
A â
â
â
6 â6 7
â2 2 â1
3 â8 4
â
â = P12A,
where
P12 =
â
â
0 1 0
1 0 0
0 0 1
â
â
is a permutation matrix that when multiplied on the left interchanges the first
and second rows of a matrix. Note that Pâ1
12 = P12. The elimination step is
then
P12A â
â
â
6 â6 7
0 0 4/3
0 â5 1/2
â
â = M1P12A,
where
M1 =
â
â
1 0 0
1/3 1 0
â1/2 0 1
â
â .
The final step requires one more row interchange:
M1P12A â
â
â
6 â6 7
0 â5 1/2
0 0 4/3
â
â = P23M1P12A = U.
Since the permutation matrices given by P are their own inverses, we can write
our result as
(P23M1P23)P23P12A = U.
24. 18 CHAPTER 3. SYSTEMS OF EQUATIONS
Multiplication of M on the left by P interchanges rows while multiplication on
the right by P interchanges columns. That is,
P23
â
â
1 0 0
1/3 1 0
â1/2 0 1
â
â P23 =
â
â
1 0 0
â1/2 0 1
1/3 1 0
â
â P23 =
â
â
1 0 0
â1/2 1 0
1/3 0 1
â
â .
The net result on M1 is an interchange of the nondiagonal elements 1/3 and
â1/2.
We can then multiply by the inverse of (P23M1P23) to obtain
P23P12A = (P23M1P23)â1
U,
which we write as
PA = LU.
Instead of L, MATLAB will write this as
A = (Pâ1
L)U.
For convenience, we will just denote (Pâ1
L) by L, but by L here we mean a
permutated lower triangular matrix.
For example, in MATLAB, to solve Ax = b for x using Gaussian elimination,
one types
x = A â b;
where â solves for x using the most efficient algorithm available, depending on
the form of A. If A is a general ð Ã ð matrix, then first the LU decomposition
of A is found using partial pivoting, and then ð¥ is determined from permuted
forward and backward substitution. If A is upper or lower triangular, then
forward or backward substitution (or their permuted version) is used directly.
If there are many different right-hand-sides, one would first directly find the
LU decomposition of A using a function call, and then solve using â. That is,
one would iterate for different bâs the following expressions:
[LU] = lu(A);
y = L â b;
x = U â ðŠ;
where the second and third lines can be shortened to
x = U â (L â b);
where the parenthesis are required. In lecture, I will demonstrate these solutions
in MATLAB using the matrix A = [â2, 2, â1; 6, â6, 7; 3, â8, 4]; which is the
example in the notes.
3.4 Operation counts
To estimate how much computational time is required for an algorithm, one can
count the number of operations required (multiplications, divisions, additions
and subtractions). Usually, what is of interest is how the algorithm scales with
25. 3.4. OPERATION COUNTS 19
the size of the problem. For example, suppose one wants to multiply two full
ð Ã ð matrices. The calculation of each element requires ð multiplications and
ð â 1 additions, or say 2ð â 1 operations. There are ð2
elements to compute
so that the total operation count is ð2
(2ð â 1). If ð is large, we might want to
know what will happen to the computational time if ð is doubled. What matters
most is the fastest-growing, leading-order term in the operation count. In this
matrix multiplication example, the operation count is ð2
(2ð â 1) = 2ð3
â ð2
,
and the leading-order term is 2ð3
. The factor of 2 is unimportant for the scaling,
and we say that the algorithm scales like O(ð3
), which is read as âbig Oh of n
cubed.â When using the big-Oh notation, we will drop both lower-order terms
and constant multipliers.
The big-Oh notation tells us how the computational time of an algorithm
scales. For example, suppose that the multiplication of two large ðà ð matrices
took a computational time of ð ð seconds. With the known operation count
going like O(ð3
), we can write
ð ð = ðð3
for some unknown constant ð. To determine how long multiplication of two
2ð Ã 2ð matrices will take, we write
ð2ð = ð(2ð)3
= 8ðð3
= 8ð ð,
so that doubling the size of the matrix is expected to increase the computational
time by a factor of 23
= 8.
Running MATLAB on my computer, the multiplication of two 2048 Ã 2048
matrices took about 0.75 sec. The multiplication of two 4096 Ã 4096 matrices
took about 6 sec, which is 8 times longer. Timing of code in MATLAB can be
found using the built-in stopwatch functions tic and toc.
What is the operation count and therefore the scaling of Gaussian elimina-
tion? Consider an elimination step with the pivot in the ðth row and ðth column.
There are both ð â ð rows below the pivot and ð â ð columns to the right of the
pivot. To perform elimination of one row, each matrix element to the right of
the pivot must be multiplied by a factor and added to the row underneath. This
must be done for all the rows. There are therefore (ð â ð)(ð â ð) multiplication-
additions to be done for this pivot. Since we are interested in only the scaling
of the algorithm, I will just count a multiplication-addition as one operation.
To find the total operation count, we need to perform elimination using ðâ1
pivots, so that
op. counts =
ðâ1âïž
ð=1
(ð â ð)2
= (ð â 1)2
+ (ð â 2)2
+ . . . (1)2
=
ðâ1âïž
ð=1
ð2
.
26. 20 CHAPTER 3. SYSTEMS OF EQUATIONS
Three summation formulas will come in handy. They are
ðâïž
ð=1
1 = ð,
ðâïž
ð=1
ð =
1
2
ð(ð + 1),
ðâïž
ð=1
ð2
=
1
6
ð(2ð + 1)(ð + 1),
which can be proved by mathematical induction, or derived by some tricks.
The operation count for Gaussian elimination is therefore
op. counts =
ðâ1âïž
ð=1
ð2
=
1
6
(ð â 1)(2ð â 1)(ð).
The leading-order term is therefore ð3
/3, and we say that Gaussian elimination
scales like O(ð3
). Since LU decomposition with partial pivoting is essentially
Gaussian elimination, that algorithm also scales like O(ð3
).
However, once the LU decomposition of a matrix A is known, the solution
of Ax = b can proceed by a forward and backward substitution. How does
a backward substitution, say, scale? For backward substitution, the matrix
equation to be solved is of the form
â
â
â
â
â
â
â
ð1,1 ð1,2 · · · ð1,ðâ1 ð1,ð
0 ð2,2 · · · ð2,ðâ1 ð2,ð
...
...
...
...
...
0 0 · · · ð ðâ1,ðâ1 ð ðâ1,ð
0 0 · · · 0 ð ð,ð
â
â
â
â
â
â
â
â
â
â
â
â
â
â
ð¥1
ð¥2
...
ð¥ ðâ1
ð¥ ð
â
â
â
â
â
â
â
=
â
â
â
â
â
â
â
ð1
ð2
...
ð ðâ1
ð ð
â
â
â
â
â
â
â
.
The solution for ð¥ð is found after solving for ð¥ ð with ð > ð. The explicit solution
for ð¥ð is given by
ð¥ð =
1
ðð,ð
â
â ðð â
ðâïž
ð=ð+1
ðð,ð ð¥ ð
â
â .
The solution for ð¥ð requires ð â ð + 1 multiplication-additions, and since this
must be done for ð such ðâ²
ð , we have
op. counts =
ðâïž
ð=1
ð â ð + 1
= ð + (ð â 1) + · · · + 1
=
ðâïž
ð=1
ð
=
1
2
ð(ð + 1).
27. 3.5. SYSTEM OF NONLINEAR EQUATIONS 21
The leading-order term is ð2
/2 and the scaling of backward substitution is
O(ð2
). After the LU decomposition of a matrix A is found, only a single forward
and backward substitution is required to solve Ax = b, and the scaling of the
algorithm to solve this matrix equation is therefore still O(ð2
). For large ð,
one should expect that solving Ax = b by a forward and backward substitution
should be substantially faster than a direct solution using Gaussian elimination.
3.5 System of nonlinear equations
A system of nonlinear equations can be solved using a version of Newtonâs
Method. We illustrate this method for a system of two equations and two
unknowns. Suppose that we want to solve
ð(ð¥, ðŠ) = 0, ð(ð¥, ðŠ) = 0,
for the unknowns ð¥ and ðŠ. We want to construct two simultaneous sequences
ð¥0, ð¥1, ð¥2, . . . and ðŠ0, ðŠ1, ðŠ2, . . . that converge to the roots. To construct these
sequences, we Taylor series expand ð(ð¥ ð+1, ðŠ ð+1) and ð(ð¥ ð+1, ðŠ ð+1) about the
point (ð¥ ð, ðŠ ð). Using the partial derivatives ð ð¥ = ðð/ðð¥, ð ðŠ = ðð/ððŠ, etc., the
two-dimensional Taylor series expansions, displaying only the linear terms, are
given by
ð(ð¥ ð+1, ðŠ ð+1) = ð(ð¥ ð, ðŠ ð) + (ð¥ ð+1 â ð¥ ð)ð ð¥(ð¥ ð, ðŠ ð)
+ (ðŠ ð+1 â ðŠ ð)ð ðŠ(ð¥ ð, ðŠ ð) + . . .
ð(ð¥ ð+1, ðŠ ð+1) = ð(ð¥ ð, ðŠ ð) + (ð¥ ð+1 â ð¥ ð)ð ð¥(ð¥ ð, ðŠ ð)
+ (ðŠ ð+1 â ðŠ ð)ð ðŠ(ð¥ ð, ðŠ ð) + . . . .
To obtain Newtonâs method, we take ð(ð¥ ð+1, ðŠ ð+1) = 0, ð(ð¥ ð+1, ðŠ ð+1) = 0 and
drop higher-order terms above linear. Although one can then find a system of
linear equations for ð¥ ð+1 and ðŠ ð+1, it is more convenient to define the variables
Îð¥ ð = ð¥ ð+1 â ð¥ ð, ÎðŠ ð = ðŠ ð+1 â ðŠ ð.
The iteration equations will then be given by
ð¥ ð+1 = ð¥ ð + Îð¥ ð, ðŠ ð+1 = ðŠ ð + ÎðŠ ð;
and the linear equations to be solved for Îð¥ ð and ÎðŠ ð are given by
(ïž
ð ð¥ ð ðŠ
ð ð¥ ð ðŠ
)ïž (ïž
Îð¥ ð
ÎðŠ ð
)ïž
=
(ïž
âð
âð
)ïž
,
where ð, ð, ð ð¥, ð ðŠ, ð ð¥, and ð ðŠ are all evaluated at the point (ð¥ ð, ðŠ ð). The two-
dimensional case is easily generalized to ð dimensions. The matrix of partial
derivatives is called the Jacobian Matrix.
We illustrate Newtonâs Method by finding the steady state solution of the
Lorenz equations, given by
ð(ðŠ â ð¥) = 0,
ðð¥ â ðŠ â ð¥ð§ = 0,
ð¥ðŠ â ðð§ = 0,
28. 22 CHAPTER 3. SYSTEMS OF EQUATIONS
where ð¥, ðŠ, and ð§ are the unknown variables and ð, ð, and ð are the known
parameters. Here, we have a three-dimensional homogeneous system ð = 0,
ð = 0, and â = 0, with
ð(ð¥, ðŠ, ð§) = ð(ðŠ â ð¥),
ð(ð¥, ðŠ, ð§) = ðð¥ â ðŠ â ð¥ð§,
â(ð¥, ðŠ, ð§) = ð¥ðŠ â ðð§.
The partial derivatives can be computed to be
ð ð¥ = âð, ð ðŠ = ð, ð ð§ = 0,
ð ð¥ = ð â ð§, ð ðŠ = â1, ð ð§ = âð¥,
â ð¥ = ðŠ, â ðŠ = ð¥, â ð§ = âð.
The iteration equation is therefore
â
â
âð ð 0
ð â ð§ ð â1 âð¥ ð
ðŠ ð ð¥ ð âð
â
â
â
â
Îð¥ ð
ÎðŠ ð
Îð§ ð
â
â = â
â
â
ð(ðŠ ð â ð¥ ð)
ðð¥ ð â ðŠ ð â ð¥ ð ð§ ð
ð¥ ð ðŠ ð â ðð§ ð
â
â ,
with
ð¥ ð+1 = ð¥ ð + Îð¥ ð,
ðŠ ð+1 = ðŠ ð + ÎðŠ ð,
ð§ ð+1 = ð§ ð + Îð§ ð.
The MATLAB program that solves this system is contained in newton system.m.
29. Chapter 4
Least-squares
approximation
The method of least-squares is commonly used to fit a parameterized curve to
experimental data. In general, the fitting curve is not expected to pass through
the data points, making this problem substantially different from the one of
interpolation.
We consider here only the simplest case of the same experimental error for
all the data points. Let the data to be fitted be given by (ð¥ð, ðŠð), with ð = 1 to
ð.
4.1 Fitting a straight line
Suppose the fitting curve is a line. We write for the fitting curve
ðŠ(ð¥) = ðŒð¥ + ðœ.
The distance ðð from the data point (ð¥ð, ðŠð) and the fitting curve is given by
ðð = ðŠð â ðŠ(ð¥ð)
= ðŠð â (ðŒð¥ð + ðœ).
A least-squares fit minimizes the sum of the squares of the ððâs. This minimum
can be shown to result in the most probable values of ðŒ and ðœ.
We define
ð =
ðâïž
ð=1
ð2
ð
=
ðâïž
ð=1
(ïž
ðŠð â (ðŒð¥ð + ðœ)
)ïž2
.
To minimize ð with respect to ðŒ and ðœ, we solve
ðð
ððŒ
= 0,
ðð
ððœ
= 0.
23
30. 24 CHAPTER 4. LEAST-SQUARES APPROXIMATION
Taking the partial derivatives, we have
ðð
ððŒ
=
ðâïž
ð=1
2(âð¥ð)
(ïž
ðŠð â (ðŒð¥ð + ðœ)
)ïž
= 0,
ðð
ððœ
=
ðâïž
ð=1
2(â1)
(ïž
ðŠð â (ðŒð¥ð + ðœ)
)ïž
= 0.
These equations form a system of two linear equations in the two unknowns ðŒ
and ðœ, which is evident when rewritten in the form
ðŒ
ðâïž
ð=1
ð¥2
ð + ðœ
ðâïž
ð=1
ð¥ð =
ðâïž
ð=1
ð¥ð ðŠð,
ðŒ
ðâïž
ð=1
ð¥ð + ðœð =
ðâïž
ð=1
ðŠð.
These equations can be solved either analytically, or numerically in MATLAB,
where the matrix form is
(ïžâïž ð
ð=1 ð¥2
ð
âïž ð
ð=1 ð¥ðâïž ð
ð=1 ð¥ð ð
)ïž (ïž
ðŒ
ðœ
)ïž
=
(ïžâïž ð
ð=1 ð¥ð ðŠðâïž ð
ð=1 ðŠð
)ïž
.
A proper statistical treatment of this problem should also consider an estimate
of the errors in ðŒ and ðœ as well as an estimate of the goodness-of-fit of the data
to the model. We leave these further considerations to a statistics class.
4.2 Fitting to a linear combination of functions
Consider the general fitting function
ðŠ(ð¥) =
ðâïž
ð=1
ð ð ð ð(ð¥),
where we assume ð functions ð ð(ð¥). For example, if we want to fit a cubic
polynomial to the data, then we would have ð = 4 and take ð1 = 1, ð2 = ð¥,
ð3 = ð¥2
and ð4 = ð¥3
. Typically, the number of functions ð ð is less than the
number of data points; that is, ð < ð, so that a direct attempt to solve for
the ð ðâs such that the fitting function exactly passes through the ð data points
would result in ð equations and ð unknowns. This would be an over-determined
linear system that in general has no solution.
We now define the vectors
y =
â
â
â
â
â
ðŠ1
ðŠ2
...
ðŠ ð
â
â
â
â
â
, c =
â
â
â
â
â
ð1
ð2
...
ð ð
â
â
â
â
â
,
and the ð-by-ð matrix
A =
â
â
â
â
â
ð1(ð¥1) ð2(ð¥1) · · · ð ð(ð¥1)
ð1(ð¥2) ð2(ð¥2) · · · ð ð(ð¥2)
...
...
...
...
ð1(ð¥ ð) ð2(ð¥ ð) · · · ð ð(ð¥ ð)
â
â
â
â
â
. (4.1)
31. 4.2. FITTING TO A LINEAR COMBINATION OF FUNCTIONS 25
With ð < ð, the equation Ac = y is over-determined. We let
r = y â Ac
be the residual vector, and let
ð =
ðâïž
ð=1
ð2
ð .
The method of least squares minimizes ð with respect to the components of c.
Now, using ð to signify the transpose of a matrix, we have
ð = r ð
r
= (y â Ac) ð
(y â Ac)
= y ð
y â c ð
A ð
y â y ð
Ac + c ð
A ð
Ac.
Since ð is a scalar, each term in the above expression must be a scalar, and since
the transpose of a scalar is equal to the scalar, we have
c ð
A ð
y =
(ïž
c ð
A ð
y
)ïž ð
= y ð
Ac.
Therefore,
ð = y ð
y â 2y ð
Ac + c ð
A ð
Ac.
To find the minimum of ð, we will need to solve ðð/ðð ð = 0 for ð = 1, . . . , ð.
To take the derivative of ð, we switch to a tensor notation, using the Einstein
summation convention, where repeated indices are summed over their allowable
range. We can write
ð = ðŠð ðŠð â 2ðŠðAðð ð ð + ððA ð
ððA ðð ðð.
Taking the partial derivative, we have
ðð
ðð ð
= â2ðŠðAðð
ðð ð
ðð ð
+
ððð
ðð ð
A ð
ððA ðð ðð + ððA ð
ððA ðð
ððð
ðð ð
.
Now,
ððð
ðð ð
=
{ïž
1, if ð = ð;
0, otherwise.
Therefore,
ðð
ðð ð
= â2ðŠðAðð + A ð
ððA ðð ðð + ððA ð
ððA ðð.
Now,
ððA ð
ððA ðð = ððA ððA ðð
= A ððA ðð ðð
= A ð
ððA ðð ðð
= A ð
ððA ðð ðð.
32. 26 CHAPTER 4. LEAST-SQUARES APPROXIMATION
Therefore,
ðð
ðð ð
= â2ðŠðAðð + 2A ð
ððA ðð ðð.
With the partials set equal to zero, we have
A ð
ððA ðð ðð = ðŠðAðð,
or
A ð
ððA ðð ðð = A ð
ðð ðŠð,
In vector notation, we have
A ð
Ac = A ð
y. (4.2)
Equation (4.2) is the so-called normal equation, and can be solved for c by
Gaussian elimination using the MATLAB backslash operator. After construct-
ing the matrix A given by (4.1), and the vector y from the data, one can code
in MATLAB
ð = (ðŽâ²
ðŽ)â(ðŽâ²
ðŠ);
But in fact the MATLAB back slash operator will automatically solve the normal
equations when the matrix A is not square, so that the MATLAB code
ð = ðŽâðŠ;
yields the same result.
33. Chapter 5
Interpolation
Consider the following problem: Given the values of a known function ðŠ = ð(ð¥)
at a sequence of ordered points ð¥0, ð¥1, . . . , ð¥ ð, find ð(ð¥) for arbitrary ð¥. When
ð¥0 †ð¥ †ð¥ ð, the problem is called interpolation. When ð¥ < ð¥0 or ð¥ > ð¥ ð the
problem is called extrapolation.
With ðŠð = ð(ð¥ð), the problem of interpolation is basically one of drawing a
smooth curve through the known points (ð¥0, ðŠ0), (ð¥1, ðŠ1), . . . , (ð¥ ð, ðŠ ð). This is
not the same problem as drawing a smooth curve that approximates a set of data
points that have experimental error. This latter problem is called least-squares
approximation.
Here, we will consider three interpolation algorithms: (1) polynomial inter-
polation; (2) piecewise linear interpolation, and; (3) cubic spline interpolation.
5.1 Polynomial interpolation
The ð + 1 points (ð¥0, ðŠ0), (ð¥1, ðŠ1), . . . , (ð¥ ð, ðŠ ð) can be interpolated by a unique
polynomial of degree ð. When ð = 1, the polynomial is a linear function;
when ð = 2, the polynomial is a quadratic function. There are three standard
algorithms that can be used to construct this unique interpolating polynomial,
and we will present all three here, not so much because they are all useful, but
because it is interesting to learn how these three algorithms are constructed.
When discussing each algorithm, we define ð ð(ð¥) to be the unique ðth degree
polynomial that passes through the given ð + 1 data points.
5.1.1 Vandermonde polynomial
This Vandermonde polynomial is the most straightforward construction of the
interpolating polynomial ð ð(ð¥). We simply write
ð ð(ð¥) = ð0 ð¥ ð
+ ð1 ð¥ ðâ1
+ · · · + ð ð.
27
34. 28 CHAPTER 5. INTERPOLATION
Then we can immediately form ð + 1 linear equations for the ð + 1 unknown
coefficients ð0, ð1, . . . , ð ð using the ð + 1 known points:
ðŠ0 = ð0 ð¥ ð
0 + ð1 ð¥ ðâ1
0 + · · · + ð ðâ1 ð¥0 + ð ð
ðŠ2 = ð0 ð¥ ð
1 + ð1 ð¥ ðâ1
1 + · · · + ð ðâ1 ð¥1 + ð ð
...
...
...
ðŠ ð = ð0 ð¥ ð
ð + ð1 ð¥ ðâ1
ð + · · · + ð ðâ1 ð¥ ð + ð ð.
The system of equations in matrix form is
â
â
â
â
â
ð¥ ð
0 ð¥ ðâ1
0 · · · ð¥0 1
ð¥ ð
1 ð¥ ðâ1
1 · · · ð¥1 1
...
...
...
...
ð¥ ð
ð ð¥ ðâ1
ð · · · ð¥ ð 1
â
â
â
â
â
â
â
â
â
â
ð0
ð1
...
ð ð
â
â
â
â
â
=
â
â
â
â
â
ðŠ0
ðŠ1
...
ðŠ ð
â
â
â
â
â
.
The matrix is called the Vandermonde matrix, and can be constructed using
the MATLAB function vander.m. The system of linear equations can be solved
in MATLAB using the â operator, and the MATLAB function polyval.m can
used to interpolate using the ð coefficients. I will illustrate this in class and
place the code on the website.
5.1.2 Lagrange polynomial
The Lagrange polynomial is the most clever construction of the interpolating
polynomial ð ð(ð¥), and leads directly to an analytical formula. The Lagrange
polynomial is the sum of ð + 1 terms and each term is itself a polynomial of
degree ð. The full polynomial is therefore of degree ð. Counting from 0, the ðth
term of the Lagrange polynomial is constructed by requiring it to be zero at ð¥ ð
with ð Ìž= ð, and equal to ðŠð when ð = ð. The polynomial can be written as
ð ð(ð¥) =
(ð¥ â ð¥1)(ð¥ â ð¥2) · · · (ð¥ â ð¥ ð)ðŠ0
(ð¥0 â ð¥1)(ð¥0 â ð¥2) · · · (ð¥0 â ð¥ ð)
+
(ð¥ â ð¥0)(ð¥ â ð¥2) · · · (ð¥ â ð¥ ð)ðŠ1
(ð¥1 â ð¥0)(ð¥1 â ð¥2) · · · (ð¥1 â ð¥ ð)
+ · · · +
(ð¥ â ð¥0)(ð¥ â ð¥1) · · · (ð¥ â ð¥ ðâ1)ðŠ ð
(ð¥ ð â ð¥0)(ð¥ ð â ð¥1) · · · (ð¥ ð â ð¥ ðâ1)
.
It can be clearly seen that the first term is equal to zero when ð¥ = ð¥1, ð¥2, . . . , ð¥ ð
and equal to ðŠ0 when ð¥ = ð¥0; the second term is equal to zero when ð¥ =
ð¥0, ð¥2, . . . ð¥ ð and equal to ðŠ1 when ð¥ = ð¥1; and the last term is equal to zero
when ð¥ = ð¥0, ð¥1, . . . ð¥ ðâ1 and equal to ðŠ ð when ð¥ = ð¥ ð. The uniqueness of
the interpolating polynomial implies that the Lagrange polynomial must be the
interpolating polynomial.
5.1.3 Newton polynomial
The Newton polynomial is somewhat more clever than the Vandermonde poly-
nomial because it results in a system of linear equations that is lower triangular,
and therefore can be solved by forward substitution. The interpolating polyno-
mial is written in the form
ð ð(ð¥) = ð0 + ð1(ð¥ â ð¥0) + ð2(ð¥ â ð¥0)(ð¥ â ð¥1) + · · · + ð ð(ð¥ â ð¥0) · · · (ð¥ â ð¥ ðâ1),
35. 5.2. PIECEWISE LINEAR INTERPOLATION 29
which is clearly a polynomial of degree ð. The ð + 1 unknown coefficients given
by the ðâs can be found by substituting the points (ð¥ð, ðŠð) for ð = 0, . . . , ð:
ðŠ0 = ð0,
ðŠ1 = ð0 + ð1(ð¥1 â ð¥0),
ðŠ2 = ð0 + ð1(ð¥2 â ð¥0) + ð2(ð¥2 â ð¥0)(ð¥2 â ð¥1),
...
...
...
ðŠ ð = ð0 + ð1(ð¥ ð â ð¥0) + ð2(ð¥ ð â ð¥0)(ð¥ ð â ð¥1) + · · · + ð ð(ð¥ ð â ð¥0) · · · (ð¥ ð â ð¥ ðâ1).
This system of linear equations is lower triangular as can be seen from the
matrix form
â
â
â
â
â
1 0 0 · · · 0
1 (ð¥1 â ð¥0) 0 · · · 0
...
...
...
...
...
1 (ð¥ ð â ð¥0) (ð¥ ð â ð¥0)(ð¥ ð â ð¥1) · · · (ð¥ ð â ð¥0) · · · (ð¥ ð â ð¥ ðâ1)
â
â
â
â
â
â
â
â
â
â
ð0
ð1
...
ð ð
â
â
â
â
â
=
â
â
â
â
â
ðŠ0
ðŠ1
...
ðŠ ð
â
â
â
â
â
,
and so theoretically can be solved faster than the Vandermonde polynomial. In
practice, however, there is little difference because polynomial interpolation is
only useful when the number of points to be interpolated is small.
5.2 Piecewise linear interpolation
Instead of constructing a single global polynomial that goes through all the
points, one can construct local polynomials that are then connected together.
In the the section following this one, we will discuss how this may be done using
cubic polynomials. Here, we discuss the simpler case of linear polynomials. This
is the default interpolation typically used when plotting data.
Suppose the interpolating function is ðŠ = ð(ð¥), and as previously, there are
ð + 1 points to interpolate. We construct the function ð(ð¥) out of ð local linear
polynomials. We write
ð(ð¥) = ðð(ð¥), for ð¥ð †ð¥ †ð¥ð+1,
where
ðð(ð¥) = ðð(ð¥ â ð¥ð) + ðð,
and ð = 0, 1, . . . , ð â 1.
We now require ðŠ = ðð(ð¥) to pass through the endpoints (ð¥ð, ðŠð) and (ð¥ð+1, ðŠð+1).
We have
ðŠð = ðð,
ðŠð+1 = ðð(ð¥ð+1 â ð¥ð) + ðð.
36. 30 CHAPTER 5. INTERPOLATION
Therefore, the coefficients of ðð(ð¥) are determined to be
ðð =
ðŠð+1 â ðŠð
ð¥ð+1 â ð¥ð
, ðð = ðŠð.
Although piecewise linear interpolation is widely used, particularly in plotting
routines, it suffers from a discontinuity in the derivative at each point. This
results in a function which may not look smooth if the points are too widely
spaced. We next consider a more challenging algorithm that uses cubic polyno-
mials.
5.3 Cubic spline interpolation
The ð + 1 points to be interpolated are again
(ð¥0, ðŠ0), (ð¥1, ðŠ1), . . . (ð¥ ð, ðŠ ð).
Here, we use ð piecewise cubic polynomials for interpolation,
ðð(ð¥) = ðð(ð¥ â ð¥ð)3
+ ðð(ð¥ â ð¥ð)2
+ ðð(ð¥ â ð¥ð) + ðð, ð = 0, 1, . . . , ð â 1,
with the global interpolation function written as
ð(ð¥) = ðð(ð¥), for ð¥ð †ð¥ †ð¥ð+1.
To achieve a smooth interpolation we impose that ð(ð¥) and its first and
second derivatives are continuous. The requirement that ð(ð¥) is continuous
(and goes through all ð + 1 points) results in the two constraints
ðð(ð¥ð) = ðŠð, ð = 0 to ð â 1, (5.1)
ðð(ð¥ð+1) = ðŠð+1, ð = 0 to ð â 1. (5.2)
The requirement that ðâ²
(ð¥) is continuous results in
ðâ²
ð(ð¥ð+1) = ðâ²
ð+1(ð¥ð+1), ð = 0 to ð â 2. (5.3)
And the requirement that ðâ²â²
(ð¥) is continuous results in
ðâ²â²
ð (ð¥ð+1) = ðâ²â²
ð+1(ð¥ð+1), ð = 0 to ð â 2. (5.4)
There are ð cubic polynomials ðð(ð¥) and each cubic polynomial has four free
coefficients; there are therefore a total of 4ð unknown coefficients. The number
of constraining equations from (5.1)-(5.4) is 2ð+2(ðâ1) = 4ðâ2. With 4ðâ2
constraints and 4ð unknowns, two more conditions are required for a unique
solution. These are usually chosen to be extra conditions on the first ð0(ð¥) and
last ð ðâ1(ð¥) polynomials. We will discuss these extra conditions later.
We now proceed to determine equations for the unknown coefficients of the
cubic polynomials. The polynomials and their first two derivatives are given by
ðð(ð¥) = ðð(ð¥ â ð¥ð)3
+ ðð(ð¥ â ð¥ð)2
+ ðð(ð¥ â ð¥ð) + ðð, (5.5)
ðâ²
ð(ð¥) = 3ðð(ð¥ â ð¥ð)2
+ 2ðð(ð¥ â ð¥ð) + ðð, (5.6)
ðâ²â²
ð (ð¥) = 6ðð(ð¥ â ð¥ð) + 2ðð. (5.7)
37. 5.3. CUBIC SPLINE INTERPOLATION 31
We will consider the four conditions (5.1)-(5.4) in turn. From (5.1) and (5.5),
we have
ðð = ðŠð, ð = 0 to ð â 1, (5.8)
which directly solves for all of the ð-coefficients.
To satisfy (5.2), we first define
âð = ð¥ð+1 â ð¥ð,
and
ðð = ðŠð+1 â ðŠð.
Now, from (5.2) and (5.5), using (5.8), we obtain the ð equations
ððâ3
ð + ððâ2
ð + ððâð = ðð, ð = 0 to ð â 1. (5.9)
From (5.3) and (5.6) we obtain the ð â 1 equations
3ððâ2
ð + 2ððâð + ðð = ðð+1, ð = 0 to ð â 2. (5.10)
From (5.4) and (5.7) we obtain the ð â 1 equations
3ððâð + ðð = ðð+1 ð = 0 to ð â 2. (5.11)
It is will be useful to include a definition of the coefficient ð ð, which is now
missing. (The index of the cubic polynomial coefficients only go up to ð â 1.)
We simply extend (5.11) up to ð = ð â 1 and so write
3ð ðâ1â ðâ1 + ð ðâ1 = ð ð, (5.12)
which can be viewed as a definition of ð ð.
We now proceed to eliminate the sets of ð- and ð-coefficients (with the ð-
coefficients already eliminated in (5.8)) to find a system of linear equations for
the ð-coefficients. From (5.11) and (5.12), we can solve for the ð ð-coefficients
to find
ðð =
1
3âð
(ðð+1 â ðð) , ð = 0 to ð â 1. (5.13)
From (5.9), we can solve for the ð ð-coefficients as follows:
ðð =
1
âð
(ïž
ðð â ððâ3
ð â ððâ2
ð
)ïž
=
1
âð
(ïž
ðð â
1
3âð
(ðð+1 â ðð) â3
ð â ððâ2
ð
)ïž
=
ðð
âð
â
1
3
âð (ðð+1 + 2ðð) , ð = 0 to ð â 1. (5.14)
We can now find an equation for the ð-coefficients by substituting (5.8),
(5.13) and (5.14) into (5.10):
3
(ïž
1
3âð
(ðð+1 â ðð)
)ïž
â2
ð + 2ððâð +
(ïž
ðð
âð
â
1
3
âð(ðð+1 + 2ðð)
)ïž
=
(ïž
ðð+1
âð+1
â
1
3
âð+1(ðð+2 + 2ðð+1)
)ïž
,
38. 32 CHAPTER 5. INTERPOLATION
which simplifies to
1
3
âð ðð +
2
3
(âð + âð+1)ðð+1 +
1
3
âð+1 ðð+2 =
ðð+1
âð+1
â
ðð
âð
, (5.15)
an equation that is valid for ð = 0 to ð â 2. Therefore, (5.15) represent ð â 1
equations for the ð+1 unknown ð-coefficients. Accordingly, we write the matrix
equation for the ð-coefficients, leaving the first and last row absent, as
â
â
â
â
â
â
â
. . . . . . . . . . . . missing . . . . . .
1
3 â0
2
3 (â0 + â1) 1
3 â1 . . . 0 0 0
...
...
...
...
...
...
...
0 0 0 . . . 1
3 â ðâ2
2
3 (â ðâ2 + â ðâ1) 1
3 â ðâ1
. . . . . . . . . . . . missing . . . . . .
â
â
â
â
â
â
â
â
â
â
â
â
â
â
ð0
ð1
...
ð ðâ1
ð ð
â
â
â
â
â
â
â
=
â
â
â
â
â
â
â
â
missing
ð1
â1
â ð0
â0
...
ð ðâ1
â ðâ1
â ð ðâ2
â ðâ2
missing
â
â
â
â
â
â
â
â
.
Once the missing first and last equations are specified, the matrix equation
for the ð-coefficients can be solved by Gaussian elimination. And once the ð-
coefficients are determined, the ð- and ð-coefficients can also be determined from
(5.13) and (5.14), with the ð-coefficients already known from (5.8). The piece-
wise cubic polynomials, then, are known and ð(ð¥) can be used for interpolation
to any value ð¥ satisfying ð¥0 †ð¥ †ð¥ ð.
The missing first and last equations can be specified in several ways, and
here we show the two ways that are allowed by the MATLAB function spline.m.
The first way should be used when the derivative ðâ²
(ð¥) is known at the endpoints
ð¥0 and ð¥ ð; that is, suppose we know the values of ðŒ and ðœ such that
ðâ²
0(ð¥0) = ðŒ, ðâ²
ðâ1(ð¥ ð) = ðœ.
From the known value of ðŒ, and using (5.6) and (5.14), we have
ðŒ = ð0
=
ð0
â0
â
1
3
â0(ð1 + 2ð0).
Therefore, the missing first equation is determined to be
2
3
â0 ð0 +
1
3
â0 ð1 =
ð0
â0
â ðŒ. (5.16)
From the known value of ðœ, and using (5.6), (5.13), and (5.14), we have
ðœ = 3ð ðâ1â2
ðâ1 + 2ð ðâ1â ðâ1 + ð ðâ1
= 3
(ïž
1
3â ðâ1
(ð ð â ð ðâ1)
)ïž
â2
ðâ1 + 2ð ðâ1â ðâ1 +
(ïž
ð ðâ1
â ðâ1
â
1
3
â ðâ1(ð ð + 2ð ðâ1)
)ïž
,
39. 5.4. MULTIDIMENSIONAL INTERPOLATION 33
which simplifies to
1
3
â ðâ1 ð ðâ1 +
2
3
â ðâ1 ð ð = ðœ â
ð ðâ1
â ðâ1
, (5.17)
to be used as the missing last equation.
The second way of specifying the missing first and last equations is called
the not-a-knot condition, which assumes that
ð0(ð¥) = ð1(ð¥), ð ðâ2(ð¥) = ð ðâ1(ð¥).
Considering the first of these equations, from (5.5) we have
ð0(ð¥ â ð¥0)3
+ ð0(ð¥ â ð¥0)2
+ ð0(ð¥ â ð¥0) + ð0
= ð1(ð¥ â ð¥1)3
+ ð1(ð¥ â ð¥1)2
+ ð1(ð¥ â ð¥1) + ð1.
Now two cubic polynomials can be proven to be identical if at some value of ð¥,
the polynomials and their first three derivatives are identical. Our conditions of
continuity at ð¥ = ð¥1 already require that at this value of ð¥ these two polynomials
and their first two derivatives are identical. The polynomials themselves will be
identical, then, if their third derivatives are also identical at ð¥ = ð¥1, or if
ð0 = ð1.
From (5.13), we have
1
3â0
(ð1 â ð0) =
1
3â1
(ð2 â ð1),
or after simplification
â1 ð0 â (â0 + â1)ð1 + â0 ð2 = 0, (5.18)
which provides us our missing first equation. A similar argument at ð¥ = ð¥ ð â 1
also provides us with our last equation,
â ðâ1 ð ðâ2 â (â ðâ2 + â ðâ1)ð ðâ1 + â ðâ2 ð ð = 0. (5.19)
The MATLAB subroutines spline.m and ppval.m can be used for cubic spline
interpolation (see also interp1.m). I will illustrate these routines in class and
post sample code on the course web site.
5.4 Multidimensional interpolation
Suppose we are interpolating the value of a function of two variables,
ð§ = ð(ð¥, ðŠ).
The known values are given by
ð§ðð = ð(ð¥ð, ðŠ ð),
with ð = 0, 1, . . . , ð and ð = 0, 1, . . . , ð. Note that the (ð¥, ðŠ) points at which
ð(ð¥, ðŠ) are known lie on a grid in the ð¥ â ðŠ plane.
40. 34 CHAPTER 5. INTERPOLATION
Let ð§ = ð(ð¥, ðŠ) be the interpolating function, satisfying ð§ðð = ð(ð¥ð, ðŠ ð). A
two-dimensional interpolation to find the value of ð at the point (ð¥, ðŠ) may be
done by first performing ð + 1 one-dimensional interpolations in ðŠ to find the
value of ð at the ð + 1 points (ð¥0, ðŠ), (ð¥1, ðŠ), . . . , (ð¥ ð, ðŠ), followed by a single
one-dimensional interpolation in ð¥ to find the value of ð at (ð¥, ðŠ).
In other words, two-dimensional interpolation on a grid of dimension (ð +
1) Ã (ð + 1) is done by first performing ð + 1 one-dimensional interpolations
to the value ðŠ followed by a single one-dimensional interpolation to the value
ð¥. Two-dimensional interpolation can be generalized to higher dimensions. The
MATLAB functions to perform two- and three-dimensional interpolation are
interp2.m and interp3.m.
41. Chapter 6
Integration
We want to construct numerical algorithms that can perform definite integrals
of the form
ðŒ =
â«ïž ð
ð
ð(ð¥)ðð¥. (6.1)
Calculating these definite integrals numerically is called numerical integration,
numerical quadrature, or more simply quadrature.
6.1 Elementary formulas
We first consider integration from 0 to â, with â small, to serve as the building
blocks for integration over larger domains. We here define ðŒâ as the following
integral:
ðŒâ =
â«ïž â
0
ð(ð¥)ðð¥. (6.2)
To perform this integral, we consider a Taylor series expansion of ð(ð¥) about
the value ð¥ = â/2:
ð(ð¥) = ð(â/2) + (ð¥ â â/2)ðâ²
(â/2) +
(ð¥ â â/2)2
2
ðâ²â²
(â/2)
+
(ð¥ â â/2)3
6
ðâ²â²â²
(â/2) +
(ð¥ â â/2)4
24
ðâ²â²â²â²
(â/2) + . . .
6.1.1 Midpoint rule
The midpoint rule makes use of only the first term in the Taylor series expansion.
Here, we will determine the error in this approximation. Integrating,
ðŒâ = âð(â/2) +
â«ïž â
0
(ïž
(ð¥ â â/2)ðâ²
(â/2) +
(ð¥ â â/2)2
2
ðâ²â²
(â/2)
+
(ð¥ â â/2)3
6
ðâ²â²â²
(â/2) +
(ð¥ â â/2)4
24
ðâ²â²â²â²
(â/2) + . . .
)ïž
ðð¥.
35
42. 36 CHAPTER 6. INTEGRATION
Changing variables by letting ðŠ = ð¥ â â/2 and ððŠ = ðð¥, and simplifying the
integral depending on whether the integrand is even or odd, we have
ðŒâ = âð(â/2)
+
â«ïž â/2
ââ/2
(ïž
ðŠðâ²
(â/2) +
ðŠ2
2
ðâ²â²
(â/2) +
ðŠ3
6
ðâ²â²â²
(â/2) +
ðŠ4
24
ðâ²â²â²â²
(â/2) + . . .
)ïž
ððŠ
= âð(â/2) +
â«ïž â/2
0
(ïž
ðŠ2
ðâ²â²
(â/2) +
ðŠ4
12
ðâ²â²â²â²
(â/2) + . . .
)ïž
ððŠ.
The integrals that we need here are
â«ïž â
2
0
ðŠ2
ððŠ =
â3
24
,
â«ïž â
2
0
ðŠ4
ððŠ =
â5
160
.
Therefore,
ðŒâ = âð(â/2) +
â3
24
ðâ²â²
(â/2) +
â5
1920
ðâ²â²â²â²
(â/2) + . . . . (6.3)
6.1.2 Trapezoidal rule
From the Taylor series expansion of ð(ð¥) about ð¥ = â/2, we have
ð(0) = ð(â/2) â
â
2
ðâ²
(â/2) +
â2
8
ðâ²â²
(â/2) â
â3
48
ðâ²â²â²
(â/2) +
â4
384
ðâ²â²â²â²
(â/2) + . . . ,
and
ð(â) = ð(â/2) +
â
2
ðâ²
(â/2) +
â2
8
ðâ²â²
(â/2) +
â3
48
ðâ²â²â²
(â/2) +
â4
384
ðâ²â²â²â²
(â/2) + . . . .
Adding and multiplying by â/2 we obtain
â
2
(ïž
ð(0) + ð(â)
)ïž
= âð(â/2) +
â3
8
ðâ²â²
(â/2) +
â5
384
ðâ²â²â²â²
(â/2) + . . . .
We now substitute for the first term on the right-hand-side using the midpoint
rule formula:
â
2
(ïž
ð(0) + ð(â)
)ïž
=
(ïž
ðŒâ â
â3
24
ðâ²â²
(â/2) â
â5
1920
ðâ²â²â²â²
(â/2)
)ïž
+
â3
8
ðâ²â²
(â/2) +
â5
384
ðâ²â²â²â²
(â/2) + . . . ,
and solving for ðŒâ, we find
ðŒâ =
â
2
(ïž
ð(0) + ð(â)
)ïž
â
â3
12
ðâ²â²
(â/2) â
â5
480
ðâ²â²â²â²
(â/2) + . . . . (6.4)
43. 6.2. COMPOSITE RULES 37
6.1.3 Simpsonâs rule
To obtain Simpsonâs rule, we combine the midpoint and trapezoidal rule to
eliminate the error term proportional to â3
. Multiplying (6.3) by two and
adding to (6.4), we obtain
3ðŒâ = â
(ïž
2ð(â/2) +
1
2
(ð(0) + ð(â))
)ïž
+ â5
(ïž
2
1920
â
1
480
)ïž
ðâ²â²â²â²
(â/2) + . . . ,
or
ðŒâ =
â
6
(ïž
ð(0) + 4ð(â/2) + ð(â)
)ïž
â
â5
2880
ðâ²â²â²â²
(â/2) + . . . .
Usually, Simpsonâs rule is written by considering the three consecutive points
0, â and 2â. Substituting â â 2â, we obtain the standard result
ðŒ2â =
â
3
(ïž
ð(0) + 4ð(â) + ð(2â)
)ïž
â
â5
90
ðâ²â²â²â²
(â) + . . . . (6.5)
6.2 Composite rules
We now use our elementary formulas obtained for (6.2) to perform the integral
given by (6.1).
6.2.1 Trapezoidal rule
We suppose that the function ð(ð¥) is known at the ð + 1 points labeled as
ð¥0, ð¥1, . . . , ð¥ ð, with the endpoints given by ð¥0 = ð and ð¥ ð = ð. Define
ðð = ð(ð¥ð), âð = ð¥ð+1 â ð¥ð.
Then the integral of (6.1) may be decomposed as
â«ïž ð
ð
ð(ð¥)ðð¥ =
ðâ1âïž
ð=0
â«ïž ð¥ ð+1
ð¥ ð
ð(ð¥)ðð¥
=
ðâ1âïž
ð=0
â«ïž â ð
0
ð(ð¥ð + ð )ðð ,
where the last equality arises from the change-of-variables ð = ð¥â ð¥ð. Applying
the trapezoidal rule to the integral, we have
â«ïž ð
ð
ð(ð¥)ðð¥ =
ðâ1âïž
ð=0
âð
2
(ðð + ðð+1) . (6.6)
If the points are not evenly spaced, say because the data are experimental values,
then the âð may differ for each value of ð and (6.6) is to be used directly.
However, if the points are evenly spaced, say because ð(ð¥) can be computed,
we have âð = â, independent of ð. We can then define
ð¥ð = ð + ðâ, ð = 0, 1, . . . , ð;
44. 38 CHAPTER 6. INTEGRATION
and since the end point ð satisfies ð = ð + ðâ, we have
â =
ð â ð
ð
.
The composite trapezoidal rule for evenly space points then becomes
â«ïž ð
ð
ð(ð¥)ðð¥ =
â
2
ðâ1âïž
ð=0
(ðð + ðð+1)
=
â
2
(ð0 + 2ð1 + · · · + 2ð ðâ1 + ð ð) . (6.7)
The first and last terms have a multiple of one; all other terms have a multiple
of two; and the entire sum is multiplied by â/2.
6.2.2 Simpsonâs rule
We here consider the composite Simpsonâs rule for evenly space points. We
apply Simpsonâs rule over intervals of 2â, starting from ð and ending at ð:
â«ïž ð
ð
ð(ð¥)ðð¥ =
â
3
(ð0 + 4ð1 + ð2) +
â
3
(ð2 + 4ð3 + ð4) + . . .
+
â
3
(ð ðâ2 + 4ð ðâ1 + ð ð) .
Note that ð must be even for this scheme to work. Combining terms, we have
â«ïž ð
ð
ð(ð¥)ðð¥ =
â
3
(ð0 + 4ð1 + 2ð2 + 4ð3 + 2ð4 + · · · + 4ð ðâ1 + ð ð) .
The first and last terms have a multiple of one; the even indexed terms have a
multiple of 2; the odd indexed terms have a multiple of 4; and the entire sum is
multiplied by â/3.
6.3 Local versus global error
Consider the elementary formula (6.4) for the trapezoidal rule, written in the
form
â«ïž â
0
ð(ð¥)ðð¥ =
â
2
(ïž
ð(0) + ð(â)
)ïž
â
â3
12
ðâ²â²
(ð),
where ð is some value satisfying 0 †ð †â, and we have used Taylorâs theorem
with the mean-value form of the remainder. We can also represent the remainder
as
â
â3
12
ðâ²â²
(ð) = O(â3
),
where O(â3
) signifies that when â is small, halving of the grid spacing â de-
creases the error in the elementary trapezoidal rule by a factor of eight. We call
the terms represented by O(â3
) the Local Error.
46. 40 CHAPTER 6. INTEGRATION
Integration using Simpsonâs rule (6.5) with grid size â yields
ðŒ =
â
3
(ïž
ð(ð) + 4ð(ð) + ð(ð)
)ïž
â
â5
90
ðâ²â²â²â²
(ð),
where ð is some value satisfying ð †ð †ð.
Integration using Simpsonâs rule twice with grid size â/2 yields
ðŒ =
â
6
(ïž
ð(ð) + 4ð(ð) + 2ð(ð) + 4ð(ð) + ð(ð)
)ïž
â
(â/2)5
90
ðâ²â²â²â²
(ðð) â
(â/2)5
90
ðâ²â²â²â²
(ð ð),
with ðð and ð ð some values satisfying ð †ðð †ð and ð †ð ð †ð.
We now define
ð1 =
â
3
(ïž
ð(ð) + 4ð(ð) + ð(ð)
)ïž
,
ð2 =
â
6
(ïž
ð(ð) + 4ð(ð) + 2ð(ð) + 4ð(ð) + ð(ð)
)ïž
,
ðž1 = â
â5
90
ðâ²â²â²â²
(ð),
ðž2 = â
â5
25 · 90
(ïž
ðâ²â²â²â²
(ðð) + ðâ²â²â²â²
(ð ð)
)ïž
.
Now we ask whether ð2 is accurate enough, or must we further refine the cal-
culation and go to Level 2? To answer this question, we make the simplifying
approximation that all of the fourth-order derivatives of ð(ð¥) in the error terms
are equal; that is,
ðâ²â²â²â²
(ð) = ðâ²â²â²â²
(ðð) = ðâ²â²â²â²
(ð ð) = ð¶.
Then
ðž1 = â
â5
90
ð¶,
ðž2 = â
â5
24 · 90
ð¶ =
1
16
ðž1.
Then since
ð1 + ðž1 = ð2 + ðž2,
and
ðž1 = 16ðž2,
we have for our estimate for the error term ðž2,
ðž2 =
1
15
(ð2 â ð1).
Therefore, given some specific value of the tolerance tol, if
â
â
â
â
1
15
(ð2 â ð1)
â
â
â
â < tol,
then we can accept ð2 as ðŒ. If the tolerance is not achieved for ðŒ, then we
proceed to Level 2.
The computation at Level 2 further divides the integration interval from ð
to ð into the two integration intervals ð to ð and ð to ð, and proceeds with the
47. 6.4. ADAPTIVE INTEGRATION 41
above procedure independently on both halves. Integration can be stopped on
either half provided the tolerance is less than tol/2 (since the sum of both errors
must be less than tol). Otherwise, either half can proceed to Level 3, and so on.
As a side note, the two values of ðŒ given above (for integration with step
size â and â/2) can be combined to give a more accurate value for I given by
ðŒ =
16ð2 â ð1
15
+ O(â7
),
where the error terms of O(â5
) approximately cancel. This free lunch, so to
speak, is called Richardsonâs extrapolation.
49. Chapter 7
Ordinary differential
equations
We now discuss the numerical solution of ordinary differential equations. These
include the initial value problem, the boundary value problem, and the eigen-
value problem. Before proceeding to the development of numerical methods, we
review the analytical solution of some classic equations.
7.1 Examples of analytical solutions
7.1.1 Initial value problem
One classic initial value problem is the ð ð¶ circuit. With ð the resistor and ð¶
the capacitor, the differential equation for the charge ð on the capacitor is given
by
ð
ðð
ðð¡
+
ð
ð¶
= 0. (7.1)
If we consider the physical problem of a charged capacitor connected in a closed
circuit to a resistor, then the initial condition is ð(0) = ð0, where ð0 is the initial
charge on the capacitor.
The differential equation (7.1) is separable, and separating and integrating
from time ð¡ = 0 to ð¡ yields
â«ïž ð
ð0
ðð
ð
= â
1
ð ð¶
â«ïž ð¡
0
ðð¡,
which can be integrated and solved for ð = ð(ð¡):
ð(ð¡) = ð0 ðâð¡/ð ð¶
.
The classic second-order initial value problem is the ð ð¿ð¶ circuit, with dif-
ferential equation
ð¿
ð2
ð
ðð¡2
+ ð
ðð
ðð¡
+
ð
ð¶
= 0.
Here, a charged capacitor is connected to a closed circuit, and the initial condi-
tions satisfy
ð(0) = ð0,
ðð
ðð¡
(0) = 0.
43
50. 44 CHAPTER 7. ORDINARY DIFFERENTIAL EQUATIONS
The solution is obtained for the second-order equation by the ansatz
ð(ð¡) = ð ðð¡
,
which results in the following so-called characteristic equation for ð:
ð¿ð2
+ ð ð +
1
ð¶
= 0.
If the two solutions for ð are distinct and real, then the two found exponential
solutions can be multiplied by constants and added to form a general solution.
The constants can then be determined by requiring the general solution to satisfy
the two initial conditions. If the roots of the characteristic equation are complex
or degenerate, a general solution to the differential equation can also be found.
7.1.2 Boundary value problems
The dimensionless equation for the temperature ðŠ = ðŠ(ð¥) along a linear heat-
conducting rod of length unity, and with an applied external heat source ð(ð¥),
is given by the differential equation
â
ð2
ðŠ
ðð¥2
= ð(ð¥), (7.2)
with 0 †ð¥ †1. Boundary conditions are usually prescribed at the end points of
the rod, and here we assume that the temperature at both ends are maintained
at zero so that
ðŠ(0) = 0, ðŠ(1) = 0.
The assignment of boundary conditions at two separate points is called a two-
point boundary value problem, in contrast to the initial value problem where the
boundary conditions are prescribed at only a single point. Two-point boundary
value problems typically require a more sophisticated algorithm for a numerical
solution than initial value problems.
Here, the solution of (7.2) can proceed by integration once ð(ð¥) is specified.
We assume that
ð(ð¥) = ð¥(1 â ð¥),
so that the maximum of the heat source occurs in the center of the rod, and
goes to zero at the ends.
The differential equation can then be written as
ð2
ðŠ
ðð¥2
= âð¥(1 â ð¥).
The first integration results in
ððŠ
ðð¥
=
â«ïž
(ð¥2
â ð¥)ðð¥
=
ð¥3
3
â
ð¥2
2
+ ð1,
51. 7.1. EXAMPLES OF ANALYTICAL SOLUTIONS 45
where ð1 is the first integration constant. Integrating again,
ðŠ(ð¥) =
â«ïž (ïž
ð¥3
3
â
ð¥2
2
+ ð1
)ïž
ðð¥
=
ð¥4
12
â
ð¥3
6
+ ð1 ð¥ + ð2,
where ð2 is the second integration constant. The two integration constants are
determined by the boundary conditions. At ð¥ = 0, we have
0 = ð2,
and at ð¥ = 1, we have
0 =
1
12
â
1
6
+ ð1,
so that ð1 = 1/12. Our solution is therefore
ðŠ(ð¥) =
ð¥4
12
â
ð¥3
6
+
ð¥
12
=
1
12
ð¥(1 â ð¥)(1 + ð¥ â ð¥2
).
The temperature of the rod is maximum at ð¥ = 1/2 and goes smoothly to zero
at the ends.
7.1.3 Eigenvalue problem
The classic eigenvalue problem obtained by solving the wave equation by sepa-
ration of variables is given by
ð2
ðŠ
ðð¥2
+ ð2
ðŠ = 0,
with the two-point boundary conditions ðŠ(0) = 0 and ðŠ(1) = 0. Notice that
ðŠ(ð¥) = 0 satisfies both the differential equation and the boundary conditions.
Other nonzero solutions for ðŠ = ðŠ(ð¥) are possible only for certain discrete values
of ð. These values of ð are called the eigenvalues of the differential equation.
We proceed by first finding the general solution to the differential equation.
It is easy to see that this solution is
ðŠ(ð¥) = ðŽ cos ðð¥ + ðµ sin ðð¥.
Imposing the first boundary condition at ð¥ = 0, we obtain
ðŽ = 0.
The second boundary condition at ð¥ = 1 results in
ðµ sin ð = 0.
Since we are searching for a solution where ðŠ = ðŠ(ð¥) is not identically zero, we
must have
ð = ð, 2ð, 3ð, . . . .
52. 46 CHAPTER 7. ORDINARY DIFFERENTIAL EQUATIONS
The corresponding negative values of ð are also solutions, but their inclusion
only changes the corresponding values of the unknown ðµ constant. A linear
superposition of all the solutions results in the general solution
ðŠ(ð¥) =
ââïž
ð=1
ðµ ð sin ððð¥.
For each eigenvalue ðð, we say there is a corresponding eigenfunction sin ððð¥.
When the differential equation can not be solved analytically, a numerical
method should be able to solve for both the eigenvalues and eigenfunctions.
7.2 Numerical methods: initial value problem
We begin with the simple Euler method, then discuss the more sophisticated
Runge-Kutta methods, and conclude with the Runge-Kutta-Fehlberg method,
as implemented in the MATLAB function ode45.m. Our differential equations
are for ð¥ = ð¥(ð¡), where the time ð¡ is the independent variable, and we will make
use of the notation Ëð¥ = ðð¥/ðð¡. This notation is still widely used by physicists
and descends directly from the notation originally used by Newton.
7.2.1 Euler method
The Euler method is the most straightforward method to integrate a differential
equation. Consider the first-order differential equation
Ëð¥ = ð(ð¡, ð¥), (7.3)
with the initial condition ð¥(0) = ð¥0. Define ð¡ ð = ðÎð¡ and ð¥ ð = ð¥(ð¡ ð). A Taylor
series expansion of ð¥ ð+1 results in
ð¥ ð+1 = ð¥(ð¡ ð + Îð¡)
= ð¥(ð¡ ð) + Îð¡ Ëð¥(ð¡ ð) + O(Îð¡2
)
= ð¥(ð¡ ð) + Îð¡ð(ð¡ ð, ð¥ ð) + O(Îð¡2
).
The Euler Method is therefore written as
ð¥ ð+1 = ð¥(ð¡ ð) + Îð¡ð(ð¡ ð, ð¥ ð).
We say that the Euler method steps forward in time using a time-step Îð¡,
starting from the initial value ð¥0 = ð¥(0). The local error of the Euler Method
is O(Îð¡2
). The global error, however, incurred when integrating to a time ð, is
a factor of 1/Îð¡ larger and is given by O(Îð¡). It is therefore customary to call
the Euler Method a first-order method.
7.2.2 Modified Euler method
This method is of a type that is called a predictor-corrector method. It is also
the first of what are Runge-Kutta methods. As before, we want to solve (7.3).
The idea is to average the value of Ëð¥ at the beginning and end of the time step.
That is, we would like to modify the Euler method and write
ð¥ ð+1 = ð¥ ð +
1
2
Îð¡
(ïž
ð(ð¡ ð, ð¥ ð) + ð(ð¡ ð + Îð¡, ð¥ ð+1)
)ïž
.
53. 7.2. NUMERICAL METHODS: INITIAL VALUE PROBLEM 47
The obvious problem with this formula is that the unknown value ð¥ ð+1 appears
on the right-hand-side. We can, however, estimate this value, in what is called
the predictor step. For the predictor step, we use the Euler method to find
ð¥ ð
ð+1 = ð¥ ð + Îð¡ð(ð¡ ð, ð¥ ð).
The corrector step then becomes
ð¥ ð+1 = ð¥ ð +
1
2
Îð¡
(ïž
ð(ð¡ ð, ð¥ ð) + ð(ð¡ ð + Îð¡, ð¥ ð
ð+1)
)ïž
.
The Modified Euler Method can be rewritten in the following form that we will
later identify as a Runge-Kutta method:
ð1 = Îð¡ð(ð¡ ð, ð¥ ð),
ð2 = Îð¡ð(ð¡ ð + Îð¡, ð¥ ð + ð1),
ð¥ ð+1 = ð¥ ð +
1
2
(ð1 + ð2).
(7.4)
7.2.3 Second-order Runge-Kutta methods
We now derive all second-order Runge-Kutta methods. Higher-order methods
can be similarly derived, but require substantially more algebra.
We consider the differential equation given by (7.3). A general second-order
Runge-Kutta method may be written in the form
ð1 = Îð¡ð(ð¡ ð, ð¥ ð),
ð2 = Îð¡ð(ð¡ ð + ðŒÎð¡, ð¥ ð + ðœð1),
ð¥ ð+1 = ð¥ ð + ðð1 + ðð2,
(7.5)
with ðŒ, ðœ, ð and ð constants that define the particular second-order Runge-
Kutta method. These constants are to be constrained by setting the local error
of the second-order Runge-Kutta method to be O(Îð¡3
). Intuitively, we might
guess that two of the constraints will be ð + ð = 1 and ðŒ = ðœ.
We compute the Taylor series of ð¥ ð+1 directly, and from the Runge-Kutta
method, and require them to be the same to order Îð¡2
. First, we compute the
Taylor series of ð¥ ð+1. We have
ð¥ ð+1 = ð¥(ð¡ ð + Îð¡)
= ð¥(ð¡ ð) + Îð¡ Ëð¥(ð¡ ð) +
1
2
(Îð¡)2
šð¥(ð¡ ð) + O(Îð¡3
).
Now,
Ëð¥(ð¡ ð) = ð(ð¡ ð, ð¥ ð).
The second derivative is more complicated and requires partial derivatives. We
have
šð¥(ð¡ ð) =
ð
ðð¡
ð(ð¡, ð¥(ð¡))
]ïž
ð¡=ð¡ ð
= ð ð¡(ð¡ ð, ð¥ ð) + Ëð¥(ð¡ ð)ð ð¥(ð¡ ð, ð¥ ð)
= ð ð¡(ð¡ ð, ð¥ ð) + ð(ð¡ ð, ð¥ ð)ð ð¥(ð¡ ð, ð¥ ð).
54. 48 CHAPTER 7. ORDINARY DIFFERENTIAL EQUATIONS
Therefore,
ð¥ ð+1 = ð¥ ð + Îð¡ð(ð¡ ð, ð¥ ð) +
1
2
(Îð¡)
2 (ïž
ð ð¡(ð¡ ð, ð¥ ð) + ð(ð¡ ð, ð¥ ð)ð ð¥(ð¡ ð, ð¥ ð)
)ïž
. (7.6)
Second, we compute ð¥ ð+1 from the Runge-Kutta method given by (7.5).
Substituting in ð1 and ð2, we have
ð¥ ð+1 = ð¥ ð + ðÎð¡ð(ð¡ ð, ð¥ ð) + ðÎð¡ð
(ïž
ð¡ ð + ðŒÎð¡, ð¥ ð + ðœÎð¡ð(ð¡ ð, ð¥ ð)
)ïž
.
We Taylor series expand using
ð
(ïž
ð¡ ð + ðŒÎð¡, ð¥ ð + ðœÎð¡ð(ð¡ ð, ð¥ ð)
)ïž
= ð(ð¡ ð, ð¥ ð) + ðŒÎð¡ð ð¡(ð¡ ð, ð¥ ð) + ðœÎð¡ð(ð¡ ð, ð¥ ð)ð ð¥(ð¡ ð, ð¥ ð) + O(Îð¡2
).
The Runge-Kutta formula is therefore
ð¥ ð+1 = ð¥ ð + (ð + ð)Îð¡ð(ð¡ ð, ð¥ ð)
+ (Îð¡)2
(ïž
ðŒðð ð¡(ð¡ ð, ð¥ ð) + ðœðð(ð¡ ð, ð¥ ð)ð ð¥(ð¡ ð, ð¥ ð)
)ïž
+ O(Îð¡3
). (7.7)
Comparing (7.6) and (7.7), we find
ð + ð = 1,
ðŒð = 1/2,
ðœð = 1/2.
There are three equations for four parameters, and there exists a family of
second-order Runge-Kutta methods.
The Modified Euler Method given by (7.4) corresponds to ðŒ = ðœ = 1 and
ð = ð = 1/2. Another second-order Runge-Kutta method, called the Midpoint
Method, corresponds to ðŒ = ðœ = 1/2, ð = 0 and ð = 1. This method is written
as
ð1 = Îð¡ð(ð¡ ð, ð¥ ð),
ð2 = Îð¡ð
(ïž
ð¡ ð +
1
2
Îð¡, ð¥ ð +
1
2
ð1
)ïž
,
ð¥ ð+1 = ð¥ ð + ð2.
7.2.4 Higher-order Runge-Kutta methods
The general second-order Runge-Kutta method was given by (7.5). The general
form of the third-order method is given by
ð1 = Îð¡ð(ð¡ ð, ð¥ ð),
ð2 = Îð¡ð(ð¡ ð + ðŒÎð¡, ð¥ ð + ðœð1),
ð3 = Îð¡ð(ð¡ ð + ðŸÎð¡, ð¥ ð + ð¿ð1 + ðð2),
ð¥ ð+1 = ð¥ ð + ðð1 + ðð2 + ðð3.
The following constraints on the constants can be guessed: ðŒ = ðœ, ðŸ = ð¿ + ð,
and ð + ð + ð = 1. Remaining constraints need to be derived.
55. 7.2. NUMERICAL METHODS: INITIAL VALUE PROBLEM 49
The fourth-order method has a ð1, ð2, ð3 and ð4. The fifth-order method
requires up to ð6. The table below gives the order of the method and the number
of stages required.
order 2 3 4 5 6 7 8
minimum # stages 2 3 4 6 7 9 11
Because of the jump in the number of stages required between the fourth-
order and fifth-order method, the fourth-order Runge-Kutta method has some
appeal. The general fourth-order method starts with 13 constants, and one then
finds 11 constraints. A particularly simple fourth-order method that has been
widely used is given by
ð1 = Îð¡ð(ð¡ ð, ð¥ ð),
ð2 = Îð¡ð
(ïž
ð¡ ð +
1
2
Îð¡, ð¥ ð +
1
2
ð1
)ïž
,
ð3 = Îð¡ð
(ïž
ð¡ ð +
1
2
Îð¡, ð¥ ð +
1
2
ð2
)ïž
,
ð4 = Îð¡ð (ð¡ ð + Îð¡, ð¥ ð + ð3) ,
ð¥ ð+1 = ð¥ ð +
1
6
(ð1 + 2ð2 + 2ð3 + ð4) .
7.2.5 Adaptive Runge-Kutta Methods
As in adaptive integration, it is useful to devise an ode integrator that au-
tomatically finds the appropriate Îð¡. The Dormand-Prince Method, which is
implemented in MATLABâs ode45.m, finds the appropriate step size by compar-
ing the results of a fifth-order and fourth-order method. It requires six function
evaluations per time step, and constructs both a fifth-order and a fourth-order
method from these function evaluations.
Suppose the fifth-order method finds ð¥ ð+1 with local error O(Îð¡6
), and the
fourth-order method finds ð¥â²
ð+1 with local error O(Îð¡5
). Let ð be the desired
error tolerance of the method, and let ð be the actual error. We can estimate ð
from the difference between the fifth- and fourth-order methods; that is,
ð = |ð¥ ð+1 â ð¥â²
ð+1|.
Now ð is of O(Îð¡5
), where Îð¡ is the step size taken. Let Îð be the estimated
step size required to get the desired error ð. Then we have
ð/ð = (Îð¡)5
/(Îð)5
,
or solving for Îð,
Îð = Îð¡
(ïž ð
ð
)ïž1/5
.
On the one hand, if ð < ð, then we accept ð¥ ð+1 and do the next time step
using the larger value of Îð. On the other hand, if ð > ð, then we reject
the integration step and redo the time step using the smaller value of Îð. In
practice, one usually increases the time step slightly less and decreases the time
step slightly more to prevent the waste of too many failed time steps.
56. 50 CHAPTER 7. ORDINARY DIFFERENTIAL EQUATIONS
7.2.6 System of differential equations
Our numerical methods can be easily adapted to solve higher-order differential
equations, or equivalently, a system of differential equations. First, we show how
a second-order differential equation can be reduced to two first-order equations.
Consider
šð¥ = ð(ð¡, ð¥, Ëð¥).
This second-order equation can be rewritten as two first-order equations by
defining ð¢ = Ëð¥. We then have the system
Ëð¥ = ð¢,
Ëð¢ = ð(ð¡, ð¥, ð¢).
This trick also works for higher-order equation. For another example, the third-
order equation
...
ð¥ = ð(ð¡, ð¥, Ëð¥, šð¥),
can be written as
Ëð¥ = ð¢,
Ëð¢ = ð£,
Ëð£ = ð(ð¡, ð¥, ð¢, ð£).
Now, we show how to generalize Runge-Kutta methods to a system of dif-
ferential equations. As an example, consider the following system of two odes,
Ëð¥ = ð(ð¡, ð¥, ðŠ),
ËðŠ = ð(ð¡, ð¥, ðŠ),
with the initial conditions ð¥(0) = ð¥0 and ðŠ(0) = ðŠ0. The generalization of the
58. 52 CHAPTER 7. ORDINARY DIFFERENTIAL EQUATIONS
The standard definitions of the derivatives give the first-order approximations
ðŠâ²
(ð¥) =
ðŠ(ð¥ + â) â ðŠ(ð¥)
â
+ O(â),
ðŠâ²
(ð¥) =
ðŠ(ð¥) â ðŠ(ð¥ â â)
â
+ O(â).
The more widely-used second-order approximation is called the central differ-
ence approximation and is given by
ðŠâ²
(ð¥) =
ðŠ(ð¥ + â) â ðŠ(ð¥ â â)
2â
+ O(â2
).
The finite difference approximation to the second derivative can be found from
considering
ðŠ(ð¥ + â) + ðŠ(ð¥ â â) = 2ðŠ(ð¥) + â2
ðŠâ²â²
(ð¥) +
1
12
â4
ðŠâ²â²â²â²
(ð¥) + . . . ,
from which we find
ðŠâ²â²
(ð¥) =
ðŠ(ð¥ + â) â 2ðŠ(ð¥) + ðŠ(ð¥ â â)
â2
+ O(â2
).
Sometimes a second-order method is required for ð¥ on the boundaries of the
domain. For a boundary point on the left, a second-order forward difference
method requires the additional Taylor series
ðŠ(ð¥ + 2â) = ðŠ(ð¥) + 2âðŠâ²
(ð¥) + 2â2
ðŠâ²â²
(ð¥) +
4
3
â3
ðŠâ²â²â²
(ð¥) + . . . .
We combine the Taylor series for ðŠ(ð¥ + â) and ðŠ(ð¥ + 2â) to eliminate the term
proportional to â2
:
ðŠ(ð¥ + 2â) â 4ðŠ(ð¥ + â) = â3ðŠ(ð¥) â 2âðŠâ²
(ð¥) + O(h3
).
Therefore,
ðŠâ²
(ð¥) =
â3ðŠ(ð¥) + 4ðŠ(ð¥ + â) â ðŠ(ð¥ + 2â)
2â
+ O(â2
).
For a boundary point on the right, we send â â ââ to find
ðŠâ²
(ð¥) =
3ðŠ(ð¥) â 4ðŠ(ð¥ â â) + ðŠ(ð¥ â 2â)
2â
+ O(â2
).
We now write a finite difference scheme to solve (7.8). We discretize ð¥ by
defining ð¥ð = ðâ, ð = 0, 1, . . . , ð + 1. Since ð¥ ð+1 = 1, we have â = 1/(ð + 1).
The functions ðŠ(ð¥) and ð(ð¥) are discretized as ðŠð = ðŠ(ð¥ð) and ðð = ð(ð¥ð). The
second-order differential equation (7.8) then becomes for the interior points of
the domain
âðŠðâ1 + 2ðŠð â ðŠð+1 = â2
ðð, ð = 1, 2, . . . ð,
with the boundary conditions ðŠ0 = ðŽ and ðŠ ð+1 = ðµ. We therefore have a linear
system of equations to solve. The first and ðth equation contain the boundary
conditions and are given by
2ðŠ1 â ðŠ2 = â2
ð1 + ðŽ,
âðŠ ðâ1 + 2ðŠ ð = â2
ð ð + ðµ.
59. 7.3. NUMERICAL METHODS: BOUNDARY VALUE PROBLEM 53
The second and third equations, etc., are
âðŠ1 + 2ðŠ2 â ðŠ3 = â2
ð2,
âðŠ2 + 2ðŠ3 â ðŠ4 = â2
ð3,
. . .
In matrix form, we have
â
â
â
â
â
â
â
â
â
2 â1 0 0 . . . 0 0 0
â1 2 â1 0 . . . 0 0 0
0 â1 2 â1 . . . 0 0 0
...
...
...
...
...
...
...
...
0 0 0 0 . . . â1 2 â1
0 0 0 0 . . . 0 â1 2
â
â
â
â
â
â
â
â
â
â
â
â
â
â
â
â
â
â
ðŠ1
ðŠ2
ðŠ3
...
ðŠ ðâ1
ðŠ ð
â
â
â
â
â
â
â
â
â
=
â
â
â
â
â
â
â
â
â
â2
ð1 + ðŽ
â2
ð2
â2
ð3
...
â2
ð ðâ1
â2
ð ð + ðµ
â
â
â
â
â
â
â
â
â
.
The matrix is tridiagonal, and a numerical solution by Guassian elimination
can be done quickly. The matrix itself is easily constructed using the MATLAB
function diag.m and ones.m. As excerpted from the MATLAB help page, the
function call ones(m,n) returns an m-by-n matrix of ones, and the function call
diag(v,k), when v is a vector with n components, is a square matrix of order
n+abs(k) with the elements of v on the k-th diagonal: ð = 0 is the main diag-
onal, ð > 0 is above the main diagonal and ð < 0 is below the main diagonal.
The ð Ã ð matrix above can be constructed by the MATLAB code
M=diag(-ones(n-1,1),-1)+diag(2*ones(n,1),0)+diag(-ones(n-1,1),1); .
The right-hand-side, provided f is a given n-by-1 vector, can be constructed
by the MATLAB code
b=h^2*f; b(1)=b(1)+A; b(n)=b(n)+B;
and the solution for u is given by the MATLAB code
y=Mâb;
7.3.2 Shooting method
The finite difference method can solve linear odes. For a general ode of the form
ð2
ðŠ
ðð¥2
= ð(ð¥, ðŠ, ððŠ/ðð¥),
with ðŠ(0) = ðŽ and ðŠ(1) = ðµ, we use a shooting method. First, we formulate
the ode as an initial value problem. We have
ððŠ
ðð¥
= ð§,
ðð§
ðð¥
= ð(ð¥, ðŠ, ð§).
60. 54 CHAPTER 7. ORDINARY DIFFERENTIAL EQUATIONS
The initial condition ðŠ(0) = ðŽ is known, but the second initial condition ð§(0) = ð
is unknown. Our goal is to determine ð such that ðŠ(1) = ðµ.
In fact, this is a root-finding problem for an appropriately defined function.
We define the function ð¹ = ð¹(ð) such that
ð¹(ð) = ðŠ(1) â ðµ.
In other words, ð¹(ð) is the difference between the value of ðŠ(1) obtained from
integrating the differential equations using the initial condition ð§(0) = ð, and
ðµ. Our root-finding routine will want to solve ð¹(ð) = 0. (The method is called
shooting because the slope of the solution curve for ðŠ = ðŠ(ð¥) at ð¥ = 0 is given
by ð, and the solution hits the value ðŠ(1) at ð¥ = 1. This looks like pointing a
gun and trying to shoot the target, which is ðµ.)
To determine the value of ð that solves ð¹(ð) = 0, we iterate using the Secant
method, given by
ð ð+1 = ð ð â ð¹(ð ð)
ð ð â ð ðâ1
ð¹(ð ð) â ð¹(ð ðâ1)
.
We need to start with two initial guesses for ð, solving the ode for the two
corresponding values of ðŠ(1). Then the Secant Method will give us the next value
of ð to try, and we iterate until |ðŠ(1) â ðµ| < tol, where tol is some specified
tolerance for the error.
7.4 Numerical methods: eigenvalue problem
For illustrative purposes, we develop our numerical methods for what is perhaps
the simplest eigenvalue ode. With ðŠ = ðŠ(ð¥) and 0 †ð¥ †1, this simple ode is
given by
ðŠâ²â²
+ ð2
ðŠ = 0. (7.9)
To solve (7.9) numerically, we will develop both a finite difference method and
a shooting method. Furthermore, we will show how to solve (7.9) with homo-
geneous boundary conditions on either the function ðŠ or its derivative ðŠâ²
.
7.4.1 Finite difference method
We first consider solving (7.9) with the homogeneous boundary conditions ðŠ(0) =
ðŠ(1) = 0. In this case, we have already shown that the eigenvalues of (7.9) are
given by ð = ð, 2ð, 3ð, . . . .
With ð interior points, we have ð¥ð = ðâ for ð = 0, . . . , ð + 1, and â =
1/(ð + 1). Using the centered-finite-difference approximation for the second
derivative, (7.9) becomes
ðŠðâ1 â 2ðŠð + ðŠð+1 = ââ2
ð2
ðŠð. (7.10)
Applying the boundary conditions ðŠ0 = ðŠ ð+1 = 0, the first equation with ð = 1,
and the last equation with ð = ð, are given by
â2ðŠ1 + ðŠ2 = ââ2
ð2
ðŠ1,
ðŠ ðâ1 â 2ðŠ ð = ââ2
ð2
ðŠ ð.