Student of Midnapore College (Autonomous).

1. Design & Analysis of
Algorithms
Design the Algorithm
Analysis the Algorithm

2. Introduction
• Algorithm: Idea behind the Program
• Pseudo-Code: Natural Language + Flavour of
Programming Language (Even that can be used to
describe an algorithm)
• Program: Set of Instruction
• Process: Program in Execution

3. Algorithms
Finiteness: An algorithm must always terminate after a
finite number of steps.
Definiteness: Each step of an algorithm must be precisely
defined; the actions to be carried out must be rigorously
and unambiguously specified for each case.
Input: An algorithm has zero or more inputs, i.e, quantities
which are given to it initially before the algorithm begins.
Output: An algorithm has one or more outputs i.e,
quantities which have a specified relation to the inputs.
Effectiveness: An algorithm is also generally expected to
be effective. This means that all of the operations to be
performed in the algorithm must be sufficiently basic that
they can in principle be done exactly and in a finite length
of time.

4. The problem of sorting

5. Insertion Sort

6. Example of Insertion Sort

7. Example of Insertion Sort

8. Example of Insertion Sort

9. Example of Insertion Sort

10. Example of Insertion Sort

11. Example of Insertion Sort

12. Example of Insertion Sort

13. Example of Insertion Sort

14. Example of Insertion Sort

15. Example of Insertion Sort

16. Example of Insertion Sort

17. Running Time
• The running time depends on the input: an already
sorted sequence is easier to sort.
• Parameterize the running time by the size of the
input, since short sequences are easier to sort than
long ones.
• Generally, we seek upper bounds on the running
time, because everybody likes a guarantee.

18. Kinds of analyses
Worst-case: (usually)
• T(n) = maximum time of algorithm on any input of
size n.
Average-case: (sometimes)
• T(n) = expected time of algorithm over all inputs of
size n.
• Need assumption of statistical distribution of inputs.
Best-case:
• Cheat with a slow algorithm that works fast on some
input.

19. Machine-Independent time
The RAM Model
Machine independent algorithm design depends on a
hypothetical computer called Random Acces Machine (RAM).
Assumptions:
• Each simple operation such as +, -, if ...etc takes exactly
one time step.
• Loops and subroutines are not considered simple
operations.
• Each memory acces takes exactly one time step.

20. Machine-independent time
What is insertion sort’s worst-case time?
• It depends on the speed of our computer,
• relative speed (on the same machine),
• absolute speed (on different machines).
BIG IDEA:
• Ignore machine-dependent constants.
• Look at growth of
“Asymptotic Analysis”


n
n
T as
)
(

21. Machine-independent time: An example
A pseudocode for insertion sort ( INSERTION SORT ).
INSERTION-SORT(A)
1 for j  2 to length [A]
2 do key  A[ j]
3  Insert A[j] into the sortted sequence A[1,..., j-1].
4 i  j – 1
5 while i > 0 and A[i] > key
6 do A[i+1]  A[i]
7 i  i – 1
8 A[i +1]  key

22. Analysis of INSERTION-SORT(contd.)
1
]
1
[
8
)
1
(
1
7
)
1
(
]
[
]
1
[
6
]
[
0
5
1
1
4
1
0
]
1
1
[
sequence
sorted
the
into
]
[
Insert
3
1
]
[
2
]
[
2
1
times
cost
SORT(A)
-
INSERTION
8
2
7
2
6
2
5
4
2
1




























n
c
key
i
A
t
c
i
i
t
c
i
A
i
A
t
c
key
i
A
and
i
n
c
j
i
n
j
A
j
A
n
c
j
A
key
n
c
A
length
j
n
j j
n
j j
n
j j
do
while
do
to
for

23. Analysis of INSERTION-SORT(contd.)
)
1
(
)
1
(
)
1
(
)
(
2
6
2
5
4
2
1 






 



n
j
j
n
j
j t
c
t
c
n
c
n
c
c
n
T
).
1
(
)
1
( 8
2
7 


 

n
c
t
c
n
j
j
The total running time is

24. Analysis of INSERTION-SORT(contd.)
The best case: The array is already sorted.
(tj =1 for j=2,3, ...,n)
)
1
(
)
1
(
)
1
(
)
1
(
)
( 8
5
4
2
1 







 n
c
n
c
n
c
n
c
n
c
n
T
).
(
)
( 8
5
4
2
8
5
4
2
1 c
c
c
c
n
c
c
c
c
c 









25. Analysis of INSERTION-SORT(contd.)
•The worst case: The array is reverse sorted
(tj =j for j=2,3, ...,n).
)
1
2
/
)
1
(
(
)
1
(
)
( 5
2
1 




 n
n
c
n
c
n
c
n
T
)
1
(
)
2
/
)
1
(
(
)
2
/
)
1
(
( 8
7
6 




 n
c
n
n
c
n
n
c
n
c
c
c
c
c
c
c
n
c
c
c )
2
/
2
/
2
/
(
)
2
/
2
/
2
/
( 8
7
6
5
4
2
1
2
7
6
5 









2
)
1
(
1




n
n
j
n
j
c
bn
an
n
T 

 2
)
(

26. Growth of Functions
Although we can sometimes determine the exact
running time of an algorithm, the extra precision is not
usually worth the effort of computing it.
For large inputs, the multiplicative constants and lower
order terms of an exact running time are dominated by
the effects of the input size itself.

27. Asymptotic Notation
The notation we use to describe the asymptotic running
time of an algorithm are defined in terms of functions
whose domains are the set of natural numbers.
Asymptotic notations are the
mathematical notations used to describe the running time
of an algorithm when the input tends towards a particular
value or a limiting value.
 
...
,
2
,
1
,
0

N

28. O-notation
• For a given function , we denote by the set
of functions
• We use O-notation to give an asymptotic upper bound of
a function, to within a constant factor.
• means that there existes some constant c
s.t. is always for large enough n.
)
(n
g ))
(
( n
g
O










0
0
all
for
)
(
)
(
0
s.t.
and
constants
positive
exist
there
:
)
(
))
(
(
n
n
n
cg
n
f
n
c
n
f
n
g
O
))
(
(
)
( n
g
O
n
f 
)
(n
cg

)
(n
f

29. Ω-Omega notation
• For a given function , we denote by the
set of functions
• We use Ω-notation to give an asymptotic lower bound on
a function, to within a constant factor.
• means that there exists some constant c s.t.
is always for large enough n.
)
(n
g ))
(
( n
g












0
0
all
for
)
(
)
(
0
s.t.
and
constants
positive
exist
there
:
)
(
))
(
(
n
n
n
f
n
cg
n
c
n
f
n
g
))
(
(
)
( n
g
n
f 

)
(n
f )
(n
cg


30. -Theta notation
• For a given function , we denote by the set
of functions
• A function belongs to the set if there exist
positive constants and such that it can be “sand-
wiched” between and or sufficienly large n.
• means that there exists some constant c1
and c2 s.t. for large enough n.
)
(n
g ))
(
( n
g













0
2
1
0
2
1
all
for
)
(
)
(
)
(
c
0
s.t.
and
,
,
constants
positive
exist
there
:
)
(
))
(
(
n
n
n
g
c
n
f
n
g
n
c
c
n
f
n
g
)
(n
f ))
(
( n
g

1
c 2
c
)
(
1 n
g
c )
(
2 n
g
c
Θ
))
(
(
)
( n
g
n
f 

)
(
)
(
)
( 2
1 n
g
c
n
f
n
g
c 


31. Asymptotic notation
Graphic examples of and .

 ,
, O

32. 2
2
2
2
1 3
2
1
n
c
n
n
n
c 


2
1
3
2
1
c
n
c 


Example 1.
Show that
We must find c1 and c2 such that
Dividing bothsides by n2 yields
For
)
(
3
2
1
)
( 2
2
n
n
n
n
f 



)
(
3
2
1
,
7 2
2
0 n
n
n
n 




33. Theorem
• For any two functions and , we have
if and only if
)
(n
g
))
(
(
)
( n
g
n
f 

)
(n
f
)).
(
(
)
(
and
))
(
(
)
( n
g
n
f
n
g
O
n
f 



34. Because :
)
2
(
5
2
2
3 n
n
n 



Example 2.
)
2
(
5
2
2
3
)
( n
n
n
n
f 




)
2
(
5
2
2
3 n
O
n
n 



35. Example 3.
6
100
3
3
,
3
for
since
)
(
6
100
3 2
2
2
2






 n
n
n
c
n
O
n
n

36. Example 3.
3
when
6
100
3
,
1
for
since
)
(
6
100
3
6
100
3
3
,
3
for
since
)
(
6
100
3
2
3
3
2
2
2
2
2















n
n
n
n
c
n
O
n
n
n
n
n
c
n
O
n
n

37. Example 3.
c
n
n
cn
c
n
O
n
n
n
n
n
n
c
n
O
n
n
n
n
n
c
n
O
n
n




















when
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
1
for
since
)
(
6
100
3
6
100
3
3
,
3
for
since
)
(
6
100
3
2
2
2
3
3
2
2
2
2
2

38. Example 3.
100
when
6
100
3
2
,
2
for
since
)
(
6
100
3
when
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
1
for
since
)
(
6
100
3
6
100
3
3
,
3
for
since
)
(
6
100
3
2
2
2
2
2
2
2
3
3
2
2
2
2
2





























n
n
n
n
c
n
n
n
c
n
n
cn
c
n
O
n
n
n
n
n
n
c
n
O
n
n
n
n
n
c
n
O
n
n

39. Example 3.
3
when
6
100
3
,
3
for
since
)
(
6
100
3
100
when
6
100
3
2
,
2
for
since
)
(
6
100
3
when
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
1
for
since
)
(
6
100
3
6
100
3
3
,
3
for
since
)
(
6
100
3
3
2
3
2
2
2
2
2
2
2
2
3
3
2
2
2
2
2






































n
n
n
n
c
n
n
n
n
n
n
n
c
n
n
n
c
n
n
cn
c
n
O
n
n
n
n
n
n
c
n
O
n
n
n
n
n
c
n
O
n
n

40. Example 3.
100
when
6
100
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
3
for
since
)
(
6
100
3
100
when
6
100
3
2
,
2
for
since
)
(
6
100
3
when
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
1
for
since
)
(
6
100
3
6
100
3
3
,
3
for
since
)
(
6
100
3
2
2
3
2
3
2
2
2
2
2
2
2
2
3
3
2
2
2
2
2














































n
n
n
cn
c
n
n
n
n
n
n
n
c
n
n
n
n
n
n
n
c
n
n
n
c
n
n
cn
c
n
O
n
n
n
n
n
n
c
n
O
n
n
n
n
n
c
n
O
n
n

41. Example 3.
apply.
and
both
since
)
(
6
100
3
100
when
6
100
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
3
for
since
)
(
6
100
3
100
when
6
100
3
2
,
2
for
since
)
(
6
100
3
when
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
1
for
since
)
(
6
100
3
6
100
3
3
,
3
for
since
)
(
6
100
3
2
2
2
2
3
2
3
2
2
2
2
2
2
2
2
3
3
2
2
2
2
2



















































O
n
n
n
n
n
n
cn
c
n
n
n
n
n
n
n
c
n
n
n
n
n
n
n
c
n
n
n
c
n
n
cn
c
n
O
n
n
n
n
n
n
c
n
O
n
n
n
n
n
c
n
O
n
n

42. Example 3.
applies.
only
since
)
(
6
100
3
apply.
and
both
since
)
(
6
100
3
100
when
6
100
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
3
for
since
)
(
6
100
3
100
when
6
100
3
2
,
2
for
since
)
(
6
100
3
when
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
1
for
since
)
(
6
100
3
6
100
3
3
,
3
for
since
)
(
6
100
3
3
2
2
2
2
2
3
2
3
2
2
2
2
2
2
2
2
3
3
2
2
2
2
2
O
n
n
n
O
n
n
n
n
n
n
cn
c
n
n
n
n
n
n
n
c
n
n
n
n
n
n
n
c
n
n
n
c
n
n
cn
c
n
O
n
n
n
n
n
n
c
n
O
n
n
n
n
n
c
n
O
n
n
























































43. Example 3.
applies.
only
since
)
(
6
100
3
applies.
only
since
)
(
6
100
3
apply.
and
both
since
)
(
6
100
3
100
when
6
100
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
3
for
since
)
(
6
100
3
100
when
6
100
3
2
,
2
for
since
)
(
6
100
3
when
3
,
any
for
since
)
(
6
100
3
3
when
6
100
3
,
1
for
since
)
(
6
100
3
6
100
3
3
,
3
for
since
)
(
6
100
3
2
3
2
2
2
2
2
3
2
3
2
2
2
2
2
2
2
2
3
3
2
2
2
2
2




























































n
n
n
O
n
n
n
O
n
n
n
n
n
n
cn
c
n
n
n
n
n
n
n
c
n
n
n
n
n
n
n
c
n
n
n
c
n
n
cn
c
n
O
n
n
n
n
n
n
c
n
O
n
n
n
n
n
c
n
O
n
n

44. Standard notations and common functions
• Floors and ceilings
    1
1 




 x
x
x
x
x

45. Standard notations and common functions
• Logarithms:
)
lg(lg
lg
lg
)
(log
log
log
ln
log
lg 2
n
n
n
n
n
n
n
n
k
k
e





46. Standard notations and common functions
• Logarithms:
For all real a>0, b>0, c>0, and n
b
a
a
a
n
a
b
a
ab
b
a
c
c
b
b
n
b
c
c
c
a
b
log
log
log
log
log
log
log
)
(
log
log






47. Standard notations and common functions
• Logarithms:
b
a
c
a
a
a
a
b
a
c
b
b
b
b
log
1
log
log
)
/
1
(
log
log
log





48. Standard notations and common functions
• Factorials
For the Stirling approximation:























n
e
n
n
n
n
1
1
2
! 
0

n
)
lg
(
)
!
lg(
)
2
(
!
)
(
!
n
n
n
n
n
o
n
n
n






49. Types of Functions
O(1) ------ Constant
Ex.: f(n)= 2 or 5 or 5000 (any constant)  O(1)
O(logn) ------ logarithmic
Ex.: alogn+b  O(logn)
O(n)  Linear
Ex.: an+b =O(n)
O(n^2)  Quadratic
Ex.: an^2+bn+c
O(n^3)  Cubic
Ex.: an^3+bn^2+cn+d
O(a^n) like 2^n, 3^n, n^n  Exponential

50. Compare Classes of Functions
1<logn<√n<n<nlogn< n^2<n^3<…<2^n<3^n<…<n^n

51. Machine Independent
• Time Complexity of an Algorithm:
Number of Primitive computations.
Relationship between the running time
of an algorithm and the size of its input
• Does not depend on Software and
Hardware

52. A Model of the Computer: RAM
• An informal model of the random-access machine (RAM)
• Basic types in the RAM model
• ◮ integer and floating-point numbers
• ◮ limited size of each “word” of data (e.g., 64 bits)
• Basic steps in the RAM model
• ◮ operations involving basic types
• ◮ load/store: assignment, use of a variable
• ◮ arithmetic operations: addition, multiplication, division,
etc.
• ◮ branch operations: conditional branch, jump
• ◮ subroutine call
A basic step in the RAM model takes a constant time