Chapter two

Computer Science and Engineering Program
Fundamental of the Analysis of Alg.
Efficiency
Gedamu A.
alemugedamu@yahoo.com
Image processing and Computer vision SIG
Some of the sides are exported from different sources to clarify the topic

Analysis of algorithms
 Issues:
 Correctness
 space efficiency
 time efficiency
 optimality
 Approaches:
 theoretical analysis
 empirical analysis

Space Analysis
 When considering space complexity, algorithms are divided
into those that need extra space to do their work and those that
work in place.
 Space analysis would examine all of the data being stored to
see if there were more efficient ways to store it.
 Example : As a developer, how do you store the real numbers ?
 Suppose we are storing a real number that has only one place of precision
after the decimal point and ranges between -10 and +10.
 How many bytes you need ?

Space Analysis
 Example : As a developer, how do you store the real numbers ?
 Suppose we are storing a real number that has only one place of precision
after the decimal point and ranges between -10 and +10.
 How many bytes you need ?
 Most computers will use between 4 and 8 bytes of memory.
 If we first multiply the value by 10. We can then store this as an integer
between -100 and +100. This needs only 1 byte, a savings of 3 to 7 bytes.
 A program that stores 1000 of these values can save 3000 to 7000 bytes.
 It makes a big difference when programming mobile or PDAs or when you
have large input .

Theoretical analysis of time efficiency
Time efficiency is analyzed by determining the number of repetitions of the
basic operation as a function of input size
 Basic operation: the operation that contributes the most towards the
running time of the algorithm
T(n) ≈ copC(n)
running time execution time
for basic operation
or cost
Number of times
basic operation
is executed
input size
Note: Different basic operations may cost differently!

Why Input Classes are Important?
 Input determines what the path of execution through an
algorithm will be.
 If we are interested in finding the largest value in a list of N
numbers, we can use the following algorithm:

Why Input Classes are Important?
 If the list is in decreasing order,
 There will only be one assignment done before the loop starts.
 If the list is in increasing order,
 There will be N assignments (one before the loop starts and N -1 inside the
loop).
 Our analysis must consider more than one possible set of input,
because if we only look at one set of input, it may be the set that is
solved the fastest (or slowest).

Input size and basic operation examples
Problem Input size measure Basic operation
Searching for key in a list
of n items
Number of list’s items, i.e. n Key comparison
Multiplication of two
matrices
Matrix dimensions or total
number of elements
Multiplication of two
numbers
Checking primality of a
given integer n
n’size = number of digits (in
binary representation)
Division
Typical graph problem #vertices and/or edges
Visiting a vertex or
traversing an edge

Importance of the analysis
 It gives an idea about how fast the algorithm
 If the number of basic operations
C(n) = ½ n (n-1) = ½ n2 – ½ n ≈ ½ n2
How much longer if the algorithm doubles its input
size?
 Increasing input size increases the complexity
 We tend to omit the constants because they have no effect with large inputs
 Everything is based on estimation
T(n) ≈ copC(n)
4
)(
2
1
)2(
2
1
)(
)2(
2
2
)(
)2(

n
n
nCC
nCC
OP
OP
nT
nT

Empirical analysis of time efficiency
 Select a specific (typical) sample of inputs
 Use physical unit of time (e.g., milliseconds)
or
Count actual number of basic operation’s executions
 Analyze the empirical/experimental data

case to consider in analysis
Best-case, average-case, worst-case
For some algorithms, efficiency depends on form of input:
 Worst case: Cworst(n) – maximum over inputs of size n
 Best case: Cbest(n) – minimum over inputs of size n
 Average case: Cavg(n) – “average” over inputs of size n
 The toughest to do

Best-case, average-case, worst-case
 Average case: Cavg(n) – “average” over inputs of size n
 Determine the number of different groups into which all possible input
sets can be divided.
 Determine the probability that the input will come from each of these
groups.
 Determine how long the algorithm will run for each of these groups.
n is the size of the input,
m is the number of groups,
pi is the probability that the input will be from
group i,
ti is the time that the algorithm takes for input
from group i.

Example: Sequential search
 Worst case
 Best case
 Average case
n key comparisons
1 comparison
(n+1)/2, assuming K is in A

Computing the Average Case for the Sequential
search
 Neither the Worst nor the Best case gives the yield to the actual performance of an algorithm
with random input.
 The Average Case does
 Assume that:
 The probability of successful search is equal to p(0≤ p ≤1)
 The probability of the first match occurring in the ith position is the same for every i .
 The probability of a match occurs at ith position is p/n for every i
 In the case of unsuccessful search , the number of comparison is n with probability (1-
p).
)1(
2
)1(
)1.(
2
)1(
.
)1.(]...321[
)1.(]....3.2.1[)(
pn
np
pn
nn
n
p
pnn
n
p
pn
n
p
n
n
p
n
p
n
p
nCavg








Computing the Average Case for the Sequential
search
 If p =1 (I found the key k)
 The average number of comparisons is (n+1)/2
 If p=0
 The average number of key comparisons is n
 The average Case is more difficult than the Best and Worst cases
)1(
2
)1(
)1.(
2
)1(
.
)1.(]...321[
)1.(]....3.2.1[)(
pn
np
pn
nn
n
p
pnn
n
p
pn
n
p
n
n
p
n
p
n
p
nCavg








16
Mathematical Background

 Logarithms

Logarithms
 Which Base ?
Loga n = Loga b Logb n
Loga n = c Logb n
In terms of complexity , we tend to
ignore the constant

 ..

Types of formulas for basic operation’s count
 Exact formula
e.g., C(n) = n(n-1)/2
 Formula indicating order of growth with specific multiplicative
constant
e.g., C(n) ≈ 0.5 n2
 Formula indicating order of growth with unknown
multiplicative constant
e.g., C(n) ≈ cn2

Order of growth
 Of greater concern is the rate of increase in operations for an
algorithm to solve a problem as the size of the problem
increases.
 This is referred to as the rate of growth of the algorithm.

Order of growth
 The function based on x2 increases slowly at first, but as the problem size
gets larger, it begins to grow at a rapid rate.
 The functions that are based on x both grow at a steady rate for the entire
length of the graph.
 The function based on log x seems to not grow at all, but this is because it
is actually growing at a very slow rate.

Values of some important functions as n  

Order of growth
Second point to consider :
 Because the faster growing functions increase at such a
significant rate, they quickly dominate the slower-growing
functions.
 This means that if we determine that an algorithm’s
complexity is a combination of two of these classes, we will
frequently ignore all but the fastest growing of these terms.
 Example : if the complexity is
we tend to ignore 30x term

Classification of Growth
Asymptotic order of growth
A way of comparing functions that ignores constant factors and small
input sizes.
 O(g(n)): class of functions f(n) that grow no faster than g(n)
 All functions with smaller or the same order of growth as g(n)
 Ω(g(n)): class of functions f(n) that grow at least as fast as g(n)
 All functions that are larger or have the same order of growth as g(n)
 Θ(g(n)): class of functions f(n) that grow at same rate as g(n)
 Set of functions that have the same order of growth as g(n)
)(),()1(5.0),(5100),( 23222
nOnnOnnnOnnOn 
),(5100),()1(5.0),( 2223
nnnnnnn 
)( 22
nbnan 

Big-oh
•O(g(n)): class of functions t(n) that grow no faster than g(n)
• if there exist some positive constant c and some nonnegative n0
such that
0)()( nnallforncgnt 
)O(n5100nthatProve:Ex 2

5101
101n101n
5)nall(forn100n5100n
0
2



nandc
You may come up with different c and n0

Big-omega
Ω(g(n)): class of functions t(n) that grow at least as fast as g(n)
?)(
)()(
23
0
nnthatprove
nnallforncgnt


01
0
0
23


nandc
nallfornn

Big-theta
Θ(g(n)): class of functions t(n) that grow at same rate as g(n)
02 )(1)()( nnallforngcntngc 
021 ,, nandcc
gettoneedYou

(g(n)), functions that grow at least as fast as g(n)
(g(n)), functions that grow at the same rate as g(n)
O(g(n)), functions that grow no faster than g(n)
g(n)
>=
<=
=
Summary

Theorem
 If t1(n)  O(g1(n)) and t2(n)  O(g2(n)), then
t1(n) + t2(n)  O(max{g1(n), g2(n)}).
 The analogous assertions are true for the -notation and -
notation.
 Implication: The algorithm’s overall efficiency will be
determined by the part with a larger order of growth, i.e., its
least efficient part.
 For example, 5n2 + 3nlogn  O(n2)
Proof. There exist constants c1, c2, n1, n2 such that
t1(n)  c1*g1(n), for all n  n1
t2(n)  c2*g2(n), for all n  n2
Define c3 = c1 + c2 and n3 = max{n1,n2}. Then
t1(n) + t2(n)  c3*max{g1(n), g2(n)}, for all n  n3

Some properties of asymptotic order of growth
 f(n)  O(f(n))
 f(n)  O(g(n)) iff g(n) (f(n))
 If f (n)  O(g (n)) and g(n)  O(h(n)) , then f(n)  O(h(n))
If f1(n)  O(g1(n)) and f2(n)  O(g2(n)) , then
f1(n) + f2(n)  O(max{g1(n), g2(n)})
Also, 1in (f(i)) =  (1in f(i))

Orders of growth of some important functions
 All logarithmic functions loga n belong to the same class
(log n) no matter what the logarithm’s base a > 1 is
because
 All polynomials of the same degree k belong to the same class:
aknk + ak-1nk-1 + … + a0  (nk)
 Exponential functions an have different orders of growth for
different a’s
ann bba log/loglog 

Basic asymptotic efficiency classes
1 constant
log n logarithmic
n linear
n log n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial

Time efficiency of nonrecursive algorithms
General Plan for Analysis
 Decide on parameter n indicating input size
 Identify algorithm’s basic operation
 Determine worst, average, and best cases for input of size n
 Set up a sum for the number of times the basic operation is executed
 Simplify the sum using standard formulas and rules.

Example 1: The Largest Element
T(n) = 1in-1 (1) = n-1 = (n) comparisons

Example 2: Element uniqueness problem
T(n) = 0in-2 (i+1jn-1 (1))
= 0in-2 (n-i+1) =
= ( ) comparisons2
n
)(
2
1
2
)1(
1.....)2()1(1
11
22
2
0
nn
nn
nnin
lu
n
i
u
li











Example 3: Matrix multiplication
T(n) = = ( ) multiplications
Ignored addition for simplicity
3
n  






1
0
1
0
1
0
1
n
i
n
j
n
k

Example 4: Counting binary digits
It cannot be investigated the way the previous examples are.
The halving game: Find integer i such that n/ ≤ 1.
Answer: i ≤ log n. So, T(n) = (log n) divisions.
Another solution: Using recurrence relations.
i
2

Plan for Analysis of Recursive Algorithms
 Decide on a parameter indicating an input’s size.
 Identify the algorithm’s basic operation.
 Check whether the number of times the basic op. is executed may
vary on different inputs of the same size. (If it may, the worst,
average, and best cases must be investigated separately.)
 Set up a recurrence relation with an appropriate initial condition
expressing the number of times the basic op. is executed.
 Solve the recurrence (or, at the very least, establish its solution’s
order of growth) by backward substitutions or another method.

Example 1 : Factorial: Recursive Algorithm
43
The stopping condition is n=0
A repetitive algorithm uses recursion whenever the algorithm
appears within the definition itself.

Example 1: Recursive evaluation of n!
Definition: n ! = 1  2  … (n-1)  n for n ≥ 1 and 0! = 1
Recursive definition of n!: F(n) = F(n-1)  n for n ≥ 1 and
F(0) = 1
Size:
Basic operation:
Recurrence relation:
n
multiplication
M(n) = M(n-1) (for F(n-1) ) + 1 (for the n * F(n-1))
 M(0) = 0

Solving the recurrence for M(n)
M(n) = M(n-1) + 1, M(0) = 0 (no multiplication when n=0)
M(n) = M(n-1) + 1
 = (M(n-2) + 1) + 1 = M(n-2) + 2
 = (M(n-3) + 1) + 2 = M(n-3) + 3
 …
 = M(n-i) + i
 = M(0) + n
 = n
The method is called backward substitution.

Example 2: The Tower of Hanoi Puzzle
46
Towers of Hanoi (aka Tower of Hanoi) is a mathematical puzzle
invented by a French Mathematician Edouard Lucas in 1983.
•Initially the game has few discs arranged in the increasing
order of size in one of the tower.
• The number of discs can vary, but there are only three towers.
•The goal is to transfer the discs from one tower another tower.
However you can move only one disk at a time and you can
never place a bigger disc over a smaller disk. It is also
understood that you can only take the top most disc from any
given tower.

Example 2: The Tower of Hanoi Puzzle
Recurrence for number of moves: M(n) = 2M(n-1) + 1

1
2
3
4
5
6
7
0 Move Sequence for d = 3

The Tower of Hanoi Puzzle
 Here's how to find the number of moves needed to transfer larger
numbers of disks from post A to post C, when M = the number of
moves needed to transfer n-1 disks from post A to post C:
 for 1 disk it takes 1 move to transfer 1 disk from post A to post C;
 for 2 disks, it will take 3 moves: 2M + 1 = 2(1) + 1 = 3
 for 6 disks... ?

Explicit Pattern
 Number of Disks Number of Moves
1 1
2 3
3 7
4 15
5 31
6 63

Powers of two help reveal the pattern:
 Number of Disks (n) Number of Moves
1 21 - 1 = 2 - 1 = 1
2 22 - 1 = 4 - 1 = 3
3 23 - 1 = 8 - 1 = 7
4 24 - 1 = 16 - 1 = 15
5 25 - 1 = 32 - 1 = 31
6 26 - 1 = 64 - 1 = 63

Fascinating fact
So the formula for finding the number of steps it takes to transfer
n disks from post A to post C is:
2
n - 1

Solving recurrence for number of moves
M(n) = 2M(n-1) + 1, M(1) = 1
M(n) = 2M(n-1) + 1
= 2(2M(n-2) + 1) + 1 = 2^2*M(n-2) + 2^1 + 2^0
= 2^2*(2M(n-3) + 1) + 2^1 + 2^0
= 2^3*M(n-3) + 2^2 + 2^1 + 2^0
= …
= 2^(n-1)*M(1) + 2^(n-2) + … + 2^1 + 2^0
= 2^(n-1) + 2^(n-2) + … + 2^1 + 2^0
= 2^n - 1

Fibonacci Sequence
 The Fibonacci numbers:
0, 1, 1, 2, 3, 5, 8, 13, 21, …
 The Fibonacci recurrence:
F(n) = F(n-1) + F(n-2)
F(0) = 0
F(1) = 1
 F(n) = F(n-1) + F(n-2) or F(n) - F(n-1) - F(n-2) = 0
General 2nd order linear homogeneous recurrence with
constant coefficients:
aX(n) + bX(n-1) + cX(n-2) = 0
What Is the
Important of
Fibonacci
Sequence?

Chapter two

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Chapter two

Similar to Chapter two (20)

Recently uploaded

Recently uploaded (20)

Chapter two