Numerical Solutions for Linear System of Equations

1. Contents
1 NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS 2
1.1 Linear System of Equations in General Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Linear System of Equations in Matrix Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Homogeneous and Non-Homogeneous Linear System of Equations . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Consistency of Linear System of Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5 Solution of Linear System of Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5.1 Gauss Elimination Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5.2 Gauss-Jordan Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5.3 Inverse of A if Exists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5.4 Iterative Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5.5 Choleski Decomposition (Square-Root) Method for Positive Definite System of Matrices . . . . . . 10
1.5.6 LU Decomposition Using Gauss Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.7 Contacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1

2. Chapter 1
NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
1.1 Linear System of Equations in General Form
a11x1 + a12x2 + a13x3 + ... + a1nxn = b1
a21x1 + a22x2 + a23x3 + ... + a2nxn = b2
a31x1 + a32x2 + a33x3 + ... + a3nxn = b3
.
.
.
an1x1 + an2x2 + an3x3 + ... + annxn = bn
1.2 Linear System of Equations in Matrix Form
AX = B
A =













a11 a12 a13 ... a1n
a21 a22 a23 ... a2n
a31 a32 a33 ... a3n
. . . .
. . . .
. . . .
an1 an2 an3 ... ann













, X =













x1
x2
x3
.
.
.
xn













, B =













b1
b2
b3
.
.
.
bn













1.3 Homogeneous and Non-Homogeneous Linear System of Equations
• If B = 0 ⇒ Homogeneous linear system of equations.
• If B = 0 ⇒ Non-homogeneous linear system of equations.
1.4 Consistency of Linear System of Equations
• If AX = B has no solution ⇒ Inconsistent linear system of equations.
• If AX = B has at least one solution ⇒ Consistent linear system of equations.
– The linear system of equations has one solution if the number of unknowns is equal to the number of indepen-
dent equations (A has inverse or |A| = 0).
2

3. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
– The linear system of equations has more than one solution if the number of unknowns is greater than the
number of independent equations(A has not inverse or |A| = 0).
NOTE: Any homogeneous linear system of equations is a consistent linear system of equations.
1.5 Solution of Linear System of Equations
1.5.1 Gauss Elimination Method
1. Backward substitution (Upper triangular system)
× × × ×
× × × ×
A
× × × ×
× × × ×
×
×
B
×
×
i=n
i=1 a1i − b1
i=n
i=1 a2i − b2
Check
i=n
i=1 a3i − b3
i=n
i=1 a4i − b4
Raws operations
↓ ↓ ↓ ...
× × × ×
0 × × ×
0 0 × ×
0 0 0 ×
×
×
×
×
Then apply backward substitution to get the unknowns.
Example
Consider the following linear system of equations
x + 6y + 5z = 28
2x + 4y + 9z = 37
3x + 7y + 8z = 41
A B Check Raws operations
1 6 5
2 4 9
3 7 8
28
37
41
−16
−22
−23
−2 × R1 + R2 → R2
−3 × R1 + R3 → R3
1 6 5
0 −8 −1
0 −11 −7
28
−19
−43
−16
10
25
−
11
8
× R2 + R3 → R3
1 6 5
0 −8 −1
0 0 45/8
28
−19
−135/8
−16
10
45/4
z =
135
45
= 3
y =
−19 + z
−8
= 2
x = 8 − 5z − 6y = 2
Mohamed Mohamed El-Sayed Atyya Page 3 of 13

4. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
2. Forward substitution (Lower triangular system)
× × × ×
× × × ×
A
× × × ×
× × × ×
×
×
B
×
×
i=n
i=1 a1i − b1
i=n
i=1 a2i − b2
Check
i=n
i=1 a3i − b3
i=n
i=1 a4i − b4
Raws operations
↓ ↓ ↓ ...
× 0 0 0
× × 0 0
× × × 0
× × × ×
×
×
×
×
Then apply forward substitution to get the unknowns.
Example
Consider the following linear system of equations
x + 6y + 5z = 28
2x + 4y + 9z = 37
3x + 7y + 8z = 41
A B Check Raws operations
1 6 5
2 4 9
3 7 8
28
37
41
−16
−22
−23
−
−9
8
× R3 + R2 → R2
−
−5
8
× R3 + R1 → R1
−7/8 13/8 0
−11/8 −31/8 0
3 −7 8
19/8
−73/8
41
−13/8
31/8
−23
1
8
× R1 → R1
1
8
× R2 → R2
−7 13 0
−11 −31 0
3 −7 8
19
−73
41
−13
31
−23
13
31
× R2 + R1 → R1
−360/31 0 0
−11 −31 0
3 7 8
−360/31
−73
41
0
31
−23
x = 1
y =
−73 + 11x
−31
= 2
z =
41 − 3x − 7y
8
= 3
Mohamed Mohamed El-Sayed Atyya Page 4 of 13

5. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
1.5.2 Gauss-Jordan Method
× × × ×
× × × ×
A
× × × ×
× × × ×
×
×
B
×
×
i=n
i=1 a1i − b1
i=n
i=1 a2i − b2
Check
i=n
i=1 a3i − b3
i=n
i=1 a4i − b4
Raws operations
↓ ↓ ↓ ...
I X
Example
Consider the following linear system of equations
x + 6y + 5z = 28
2x + 4y + 9z = 37
3x + 7y + 8z = 41
A B Check Raws operations
1 6 5
2 4 9
3 7 8
28
37
41
−16
−22
−23
−2 × R1 + R2 → R2
−3 × R1 + R3 → R3
1 6 5
0 −8 −1
0 −11 −7
28
−19
−43
−16
10
25
−
1
8
× R2 → R2
1 6 5
0 1 1/8
0 0 45/8
28
−19/8
−135/8
−16
−5/4
45/4
−6 × R2 + R1 → R1
11 × R2 + R3 → R3
1 0 17/4
0 1 1/8
0 0 45/8
55/4
−19/8
−135/8
−17/2
−5/4
45/4
−
8
45
× R3 → R3
1 0 17/4
0 1 1/8
0 0 1
55/4
−19/8
3
−17/2
−5/4
−2
−
17
4
× R3 + R1 → R1
−
1
8
× R3 + R2 → R2
1 0 0
0 1 0
0 0 1
1
2
3
0
−1
−2
x = 1
y = 2
z = 3
Mohamed Mohamed El-Sayed Atyya Page 5 of 13

6. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
1.5.3 Inverse of A if Exists
A I Check Raws operations
↓ ↓ ↓ ...
I A−1
X = A−1
B
Example
Consider the following linear system of equations
x + 6y + 5z = 28
2x + 4y + 9z = 37
3x + 7y + 8z = 41
A I Check Raw operations
1 6 5
2 4 9
3 7 8
1 0 0
0 1 0
0 0 1
11
14
17
−2 × R1 + R2 → R2
−3 × R1 + R3 → R3
1 6 5
0 −8 −1
0 −11 −7
1 0 0
−2 1 0
−3 0 1
11
−8
−16
−
1
8
× R2 → R2
1 6 5
0 1 0.125
0 −11 −7
1 0 0
0.25 −0.125 0
−3 0 1
11
1
−16
−6 × R2 + R1 → R1
11 × R2 + R3 → R3
1 0 4.25
0 1 0.125
0 0 −5.625
−0.5 0.75 0
0.25 −0.125 0
−0.25 −1.375 1
5
1
−5
−
1
5.625
× R3 → R3
1 0 4.25
0 1 0.125
0 0 1
−0.5 0.75 0
0.25 −0.125 0
2/45 11/45 −8/45
5
1
8/9
−0.125 × R3 + R2 → R2
−4.25 × R3 + R1 → R1
1 0 0
0 1 0
0 0 1
−31/45 −13/45 34/45
11/45 −7/45 1/45
2/45 11/45 −8/45
11/9
8/9
8/9
∴ A−1
=




−31/45 −13/45 34/45
11/45 −7/45 1/45
2/45 11/45 −8/45








x
y
z



 = A−1
B =




1
2
3




Mohamed Mohamed El-Sayed Atyya Page 6 of 13

7. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
1.5.4 Iterative Techniques
Consider the linear system of equations,
a11x1 + a12x2 + a13x3 + ... + a1nxn = b1
a21x1 + a22x2 + a23x3 + ... + a2nxn = b2
a31x1 + a32x2 + a33x3 + ... + a3nxn = b3
.
.
.
an1x1 + an2x2 + an3x3 + ... + annxn = bn
Rewrite it as,
x1 =
1
a11
[b1 − a12x2 − a13x3 − ... − a1nxn]
x2 =
1
a22
[b2 − a21x1 − a23x3 − ... − a2nxn]
x3 =
1
a33
[b3 − a31x1 − a32x2 − ... − a3nxn]
.
.
.
xn =
1
ann
bn − an1x1 − an2x2 − ... − an(n−1)xn−1
In general,
xn =
1
ann
bn −
n
i=1
anixi ; ∀ i = n
To get a solution with this method we should satisfy this conditions:
1. ann = 0 ∀ n ; existence condition.
2. A matrix should be diagonally dominant, |aii| > n
j=1 |aij| ∀ j; j = i ; convergence condition.
Methods of solution:
1. Jacobi method
We start the solution by an initial vector,
x(0)
= x
(0)
1 x
(0)
2 x
(0)
3 ... x
(0)
n
T
Then get the fist iteration values,
x
(1)
1 =
1
a11
b1 − a12x
(0)
2 − a13x
(0)
3 − ... − a1nx(0)
n
x
(1)
2 =
1
a22
b2 − a21x
(0)
1 − a23x
(0)
3 − ... − a2nx(0)
n
x
(1)
3 =
1
a33
b3 − a31x
(0)
1 − a32x
(0)
2 − ... − a3nx(0)
n
.
.
.
x(1)
n =
1
ann
bn − an1x
(0)
1 − an2x
(0)
2 − ... − an(n−1)x
(0)
n−1
Mohamed Mohamed El-Sayed Atyya Page 7 of 13

8. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
In general,
x(k)
n =
1
ann
bn −
n
i=1
anix
(k−1)
i ; ∀ i = n
Continue till get a solution with a certain percentage of error.
Example
Consider the following linear system of equations
x + y + 9z = 11
7x + 3y + 2z = 12
−2x + 8y + 4z = 10
and initial values
x(0)
= 0 0 0
T
with accuracy k = 5 ⇒ x(i+1)
− x(i)
< 0.5 × 10−5
A|B =



1 1 9
7 3 2
−2 8 4
11
12
10


 not diagonally dominant
⇒ A|B =



7 3 2
−2 8 4
1 1 9
12
10
11



∴ x(i+1)
=
1
7
12 − 3y(i)
− 2z(i)
y(i+1)
=
1
8
10 − 2x(i)
− 4z(i)
z(i+1)
=
1
9
11 − x(i)
− y(i)
Iteration No. x y z
0 0 0 0
1 1.714286 1.250000 1.222222
2 0.829365 1.067461 0.892857
3 1.001700 1.010912 1.011463
4 0.992048 0.994694 0.998599
5 1.002674 0.998713 1.001473
6 1.000131 0.999932 0.999850
7 1.000072 1.000108 0.999993
8 0.999975 1.0000215 0.999980
9 0.999997 0.999999 1.000002
10 0.999999 0.999998 1.000000
| error| 0.000002 0.000001 0.000002
2. Gauss-Seidel method
We start the solution by an initial vector,
x(0)
= x
(0)
1 x
(0)
2 x
(0)
3 ... x
(0)
n
T
Mohamed Mohamed El-Sayed Atyya Page 8 of 13

9. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
Then get the fist iteration values,
x
(1)
1 =
1
a11
b1 − a12x
(0)
2 − a13x
(0)
3 − ... − a1nx(0)
n
x
(1)
2 =
1
a22
b2 − a21x
(1)
1 − a23x
(0)
3 − ... − a2nx(0)
n
x
(1)
3 =
1
a33
b3 − a31x
(1)
1 − a32x
(1)
2 − ... − a3nx(0)
n
.
.
.
x(1)
n =
1
ann
bn − an1x
(1)
1 − an2x
(1)
2 − ... − an(n−1)x
(0)
n−1
In general,
x(k)
n =
1
ann
bn −
i<n
i=1
anix
(k)
i +
i=n
i>n
anix
(k−1)
i
Continue till get a solution with a certain percentage of error.
Example
Consider the following linear system of equations
x + y + 9z = 11
7x + 3y + 2z = 12
−2x + 8y + 4z = 10
and initial values
x(0)
= 0 0 0
T
with accuracy k = 5 ⇒ x(i+1)
− x(i)
< 0.5 × 10−5
A|B =



1 1 9
7 3 2
−2 8 4
11
12
10


 not diagonally dominant
⇒ A|B =



7 3 2
−2 8 4
1 1 9
12
10
11



∴ x(i+1)
=
1
7
12 − 3y(i)
− 2z(i)
y(i+1)
=
1
8
10 − 2x(i+1)
− 4z(i)
z(i+1)
=
1
9
11 − x(i+1)
− y(i+1)
Mohamed Mohamed El-Sayed Atyya Page 9 of 13

10. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
Iteration No. x y z
0 0 0 0
1 1.714286 1.678572 0.845238
2 0.753401 1.015731 1.025652
3 0.985929 0.983656 1.003379
4 1.006039 0.999820 0.999349
5 1.000263 1.000391 0.999927
6 0.999853 0.999999 1.000016
7 0.999996 0.999991 1.000001
8 1.000004 1.000001 0.999999
9 1.000000 1.000001 1.000000
| error| 0.000004 0.000000 0.000001
1.5.5 Choleski Decomposition (Square-Root) Method for Positive Definite System of
Matrices
Theorem: A symmetrical n × n matrix is positive definite if and only if det(Ak) > 0 ;
k = 1, 2, 3, ..., n. where Ak is the k × k matrix formed by the kth
first row and
column of matrix A.
Theorem: For a symmetric positive definite matrix A, |aij|2
≤ |aiiajj| and the maximum
element of A lies on the main diagonal.
Theorem: Let A be a symmetric positive definite matrix, there is a unique upper-triangular
matrix R with the diagonal elements, such that RT
R = A or LLT
= A. Where
L = lower-triangular matrix, L = RT
.
A = LLT










a11 a12 a13 ... a1n
a21 a22 a23 ... a2n
. . . .
. . . .
. . . .
an1 an2 an3 ... ann










=










l11
l21 l22
. . .
. . .
. . .
ln1 ln2 . . . lnn




















l11 l21 . . . ln1
l22 . . . ln2
. .
. .
. .
lnn










l11 =
√
a11
lij =
1
lji
aij −
j−1
k=1
likljk ; j = 1, 2, 3, ..., i − 1
lii = aii −
i−1
k=1
l2
ik ; i = 1, 2, 3, ..., n
li−1, j = 0 ; j = i, i + 1, ..., n
Also,
A = RT
R
AX = B ⇒ RT
RX = B
RT
[RX − D] = AX − B ⇒ RT
D = B , RX = D
Mohamed Mohamed El-Sayed Atyya Page 10 of 13

11. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
Example
Consider the following linear system of equations
7x + y + 5z = 13
x + 8y + 2z = 11
5x + 2y + 9z = 16
7 1 5
1 8 2
5 2 9
⇒
55/7 9/7
9/7 38/7
⇒ 287/55
Pivot 1 =
√
7 Pivot 2 = 55/7 Pivot 3 = 287/55
R =




√
7 1/
√
7 5/
√
7
0 55/7 9/
√
385
0 0 287/55




RT
=




√
7 0 0
1/
√
7 55/7 0
5/
√
7 9/
√
385 287/55




RT
D = B ⇒ D = 13/
√
7 64/
√
385 287/55
T
RX = D ⇒ X = 1 1 1
T
For clarification
a11 a12 a13
a21 a22 a23
a31 a32 a33
⇒
b11 b12
b21 b22
⇒ c11
Pivot 1 =
√
a11 Pivot 2 =
√
b11 Pivot 3 =
√
c11
b11 = a22 −
a12
Pivot 1
×
a21
Pivot 1
b12 = a23 −
a13
Pivot 1
×
a21
Pivot 1
b21 = a32 −
a12
Pivot 1
×
a31
Pivot 1
b22 = a33 −
a13
Pivot 1
×
a31
Pivot 1
c11 = b22 −
b12
Pivot 2
×
b21
Pivot 2
R =



r11 r12 r13
0 r22 r23
0 0 r33



r11 =
a11
Pivot 1
r12 =
a12
Pivot 1
Mohamed Mohamed El-Sayed Atyya Page 11 of 13

12. 1.5. SOLUTION OF LINEAR SYSTEM OF EQUATIONS
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
r13 =
a13
Pivot 1
r22 =
b11
Pivot 2
r23 =
b12
Pivot 2
r33 = Pivot 3
1.5.6 LU Decomposition Using Gauss Elimination
Consider a linear system of equations
AX = B
Which A can be expressed as
A = LU
Then we get
LUX = B
L [UX − D] = AX − B ⇒ LD = B , UX = D
Example
Consider the following linear system of equations
x + 6y + 5z = 28
2x + 4y + 9z = 37
3x + 7y + 8z = 41
A B Check Raw operations
1 6 5
2 4 9
3 7 8
28
37
41
−16
−22
−23
−2 × R1 + R2 → R2
−3 × R1 + R3 → R3
1 6 5
0 −8 −1
0 −11 −7
28
−19
−43
−16
10
25
−
11
8
× R2 + R3 → R3
1 6 5
0 −8 −1
0 0 −45/8
28
−19
−135/8
−16
10
45/4
U =




1 6 5
0 −8 −1
0 0 −45/8




L =




1 0 0
2 1 0
3 11/8 1




LD = B ⇒ D = 28 −19 −135/8
T
UX = D ⇒ X = 1 2 3
T
Note: As we see the raw operation in general is aij × Rj + Ri → Ri, this used to generate L matrix as lij = −aij
Mohamed Mohamed El-Sayed Atyya Page 12 of 13

13. 1.6. REFERENCES
CHAPTER 1. NUMERICAL SOLUTIONS FOR
LINEAR SYSTEM OF EQUATIONS
1.6 References
1. A First Course in Numerical Analysis, 2 nd edition, Anthony Ralston and Philip Rabinowitz.
2. Elementary Numerical Analysis, An Algorithmic Approach, 3 rd edition, S. D. Conte and Carl de Boor.
3. Numerical Methods for Engineers, 6 th edition, Steven C. Chapra and Raymond P. Canale.
4. Numerical Methods and Algorithms, Milan Kubicek, Miroslava Dubcova, Drahoslava Janovska, VSCHT Praha 2005.
1.7 Contacts
mohamed.atyya94@eng-st.cu.edu.eg
Mohamed Mohamed El-Sayed Atyya Page 13 of 13