This document provides an introduction to algorithms and their analysis. It discusses algorithm design techniques like insertion sort and asymptotic analysis using big O notation to describe an algorithm's running time. Insertion sort is analyzed as an example, with its best, average, and worst case times shown to be O(n), O(n^2), and O(n^2) respectively, where n is the input size. Common functions and standard asymptotic notations are also introduced.

1. AlgorithmsAlgorithms

2. IntroductionIntroduction
• The methods of algorithm design form one of the core
practical technologies of computer science.
• The main aim of this lecture is to familiarize the student
with the framework we shall use through the course
about the design and analysis of algorithms.
• We start with a discussion of the algorithms needed to
solve computational problems. The problem of sorting is
used as a running example.
• We introduce a pseudocode to show how we shall
specify the algorithms.

3. AlgorithmsAlgorithms
• The word algorithm comes from the name of a Persian
mathematician Abu Ja’far Mohammed ibn-i Musa al
Khowarizmi.
• In computer science, this word refers to a special
method useable by a computer for solution of a problem.
The statement of the problem specifies in general terms
the desired input/output relationship.
• For example, sorting a given sequence of numbers into
nondecreasing order provides fertile ground for
introducing many standard design techniques and
analysis tools.

4. The problem of sortingThe problem of sorting

5. Insertion SortInsertion Sort

6. Example of Insertion SortExample of Insertion Sort

7. Example of Insertion SortExample of Insertion Sort

8. Example of Insertion SortExample of Insertion Sort

9. Example of Insertion SortExample of Insertion Sort

10. Example of Insertion SortExample of Insertion Sort

11. Example of Insertion SortExample of Insertion Sort

12. Example of Insertion SortExample of Insertion Sort

13. Example of Insertion SortExample of Insertion Sort

14. Example of Insertion SortExample of Insertion Sort

15. Example of Insertion SortExample of Insertion Sort

16. Example of Insertion SortExample of Insertion Sort

17. Analysis of algorithmsAnalysis of algorithms
The theoretical study of computer-program
performance and resource usage.
What’s more important than performance?
• modularity
• correctness
• maintainability
• functionality
• robustness
• user-friendliness
• programmer time
• simplicity
• extensibility
• reliability

18. Analysis of algorithmsAnalysis of algorithms
Why study algorithms and performance?
• Algorithms help us to understand scalability.
• Performance often draws the line between what is feasible
and what is impossible.
• Algorithmic mathematics provides a language for talking
about program behavior.
• The lessons of program performance generalize to other
computing resources.
• Speed is fun!

19. Running TimeRunning Time
• The running time depends on the input: an already
sorted sequence is easier to sort.
• Parameterize the running time by the size of the
input, since short sequences are easier to sort than
long ones.
• Generally, we seek upper bounds on the running
time, because everybody likes a guarantee.

20. Kinds of analysesKinds of analyses
Worst-case: (usually)
• T(n) = maximum time of algorithm on any input of
size n.
Average-case: (sometimes)
• T(n) = expected time of algorithm over all inputs of
size n.
• Need assumption of statistical distribution of inputs.
Best-case:
• Cheat with a slow algorithm that works fast on some
input.

21. Machine-Machine-IIndependent timendependent time
The RAM Model
Machine independent algorithm design depends on a
hypothetical computer called Random Acces Machine (RAM).
Assumptions:
• Each simple operation such as +, -, if ...etc takes exactly
one time step.
• Loops and subroutines are not considered simple
operations.
• Each memory acces takes exactly one time step.

22. Machine-independent timeMachine-independent time
What is insertion sort’s worst-case time?
• It depends on the speed of our computer,
• relative speed (on the same machine),
• absolute speed (on different machines).
BIG IDEA:
• Ignore machine-dependent constants.
• Look at growth of
“Asymptotic Analysis”
∞→nnT as)(

23. Machine-independent time: An exampleMachine-independent time: An example
A pseudocode for insertion sort ( INSERTION SORT ).
INSERTION-SORT(A)
1 for j ← 2 to length [A]
2 do key ← A[ j]
3 ∇ Insert A[j] into the sortted sequence A[1,..., j-1].
4 i ← j – 1
5 while i > 0 and A[i] > key
6 do A[i+1] ← A[i]
7 i ← i – 1
8 A[i +1] ← key

24. Analysis of INSERTION-SORT(contd.)Analysis of INSERTION-SORT(contd.)
1]1[8
)1(17
)1(][]1[6
][05
114
10]11[sequence
sortedtheinto][Insert3
1][2
][21
timescostSORT(A)-INSERTION
8
27
26
25
4
2
1
−←+
−−←
−←+
>>
−−←
−−⋅⋅
∇
−←
←
∑
∑
∑
=
=
=
nckeyiA
tcii
tciAiA
tckeyiAandi
ncji
njA
jA
ncjAkey
ncAlengthj
n
j j
n
j j
n
j j
do
while
do
tofor

25. Analysis of INSERTION-SORT(contd.)Analysis of INSERTION-SORT(contd.)
)1()1()1()(
2
6
2
5421 −++−+−+= ∑∑
==
n
j
j
n
j
j tctcncnccnT
).1()1( 8
2
7 −+−+ ∑
=
nctc
n
j
j
The total running time is

26. Analysis of INSERTION-SORT(contd.)Analysis of INSERTION-SORT(contd.)
The best case: The array is already sorted.
(tj =1 for j=2,3, ...,n)
)1()1()1()1()( 85421 −+−+−+−+= ncncncncncnT
).()( 854285421 ccccnccccc +++−++++=

27. Analysis of INSERTION-SORT(contd.)Analysis of INSERTION-SORT(contd.)
•The worst case: The array is reverse sorted
(tj =j for j=2,3, ...,n).
)12/)1(()1()( 521 −++−+= nncncncnT
)1()2/)1(()2/)1(( 876 −+−+−+ ncnncnnc
ncccccccnccc )2/2/2/()2/2/2/( 8765421
2
765 +−−++++++=
2
)1(
1
+
=∑
=
nn
j
n
j
cbnannT ++= 2
)(

28. Growth of FunctionsGrowth of Functions
Although we can sometimes determine the exact
running time of an algorithm, the extra precision is not
usually worth the effort of computing it.
For large inputs, the multiplicative constants and lower
order terms of an exact running time are dominated by
the effects of the input size itself.

29. Asymptotic NotationAsymptotic Notation
The notation we use to describe the asymptotic running
time of an algorithm are defined in terms of functions
whose domains are the set of natural numbers
{ }...,2,1,0=N

30. O-notationO-notation
• For a given function , we denote by the
set of functions
• We use O-notation to give an asymptotic upper bound of
a function, to within a constant factor.
• means that there existes some constant c
s.t. is always for large enough n.
)(ng ))(( ngO






≥≤≤
=
0
0
allfor)()(0
s.t.andconstantspositiveexistthere:)(
))((
nnncgnf
ncnf
ngO
))(()( ngOnf =
)(ncg≤)(nf

31. ΩΩ--OmegaOmega notationnotation
• For a given function , we denote by the
set of functions
• We use Ω-notation to give an asymptotic lower bound on
a function, to within a constant factor.
• means that there exists some constant c s.t.
is always for large enough n.
)(ng ))(( ngΩ






≥≤≤
=Ω
0
0
allfor)()(0
s.t.andconstantspositiveexistthere:)(
))((
nnnfncg
ncnf
ng
))(()( ngnf Ω=
)(nf )(ncg≥

32. --ThetaTheta notationnotation
• For a given function , we denote by the
set of functions
• A function belongs to the set if there
exist positive constants and such that it can be
“sand- wiched” between and or sufficienly
large n.
• means that there exists some constant c1
and c2 s.t. for large enough n.
)(ng ))(( ngΘ






≥≤≤≤
=Θ
021
021
allfor)()()(c0
s.t.and,,constantspositiveexistthere:)(
))((
nnngcnfng
nccnf
ng
)(nf ))(( ngΘ
1c 2c
)(1 ngc )(2 ngc
Θ
))(()( ngnf Θ=
)()()( 21 ngcnfngc ≤≤

33. Asymptotic notationAsymptotic notation
Graphic examples of and .ΩΘ ,, O

34. 2
2
22
1 3
2
1
ncnnnc ≤−≤
21
3
2
1
c
n
c ≤−≤
Example 1.Example 1.
Show that
We must find c1 and c2 such that
Dividing bothsides by n2 yields
For
)(3
2
1
)( 22
nnnnf Θ=−=
)(3
2
1
,7 22
0 nnnn Θ=−≥

35. TheoremTheorem
• For any two functions and , we have
if and only if
)(ng
))(()( ngnf Θ=
)(nf
)).(()(and))(()( ngnfngOnf Ω==

36. Because :
)2(5223 nnn Ω=+−
Example 2.Example 2.
)2(5223)( nnnnf Θ=+−=
)2(5223 nOnn =+−

37. Example 3.Example 3.
610033,3forsince)(61003 2222
+−>==+− nnncnOnn

38. Example 3.Example 3.
3when61003,1forsince)(61003
610033,3forsince)(61003
2332
2222
>+−>==+−
+−>==+−
nnnncnOnn
nnncnOnn

39. Example 3.Example 3.
cnncncnOnn
nnnncnOnn
nnncnOnn
><≠+−
>+−>==+−
+−>==+−
when3,anyforsince)(61003
3when61003,1forsince)(61003
610033,3forsince)(61003
22
2332
2222

40. Example 3.Example 3.
100when610032,2forsince)(61003
when3,anyforsince)(61003
3when61003,1forsince)(61003
610033,3forsince)(61003
2222
22
2332
2222
>+−<=Ω=+−
><≠+−
>+−>==+−
+−>==+−
nnnncnnn
cnncncnOnn
nnnncnOnn
nnncnOnn

41. Example 3.Example 3.
3when61003,3forsince)(61003
100when610032,2forsince)(61003
when3,anyforsince)(61003
3when61003,1forsince)(61003
610033,3forsince)(61003
3232
2222
22
2332
2222
><+−=Ω≠+−
>+−<=Ω=+−
><≠+−
>+−>==+−
+−>==+−
nnnncnnn
nnnncnnn
cnncncnOnn
nnnncnOnn
nnncnOnn

42. Example 3.Example 3.
100when61003,anyforsince)(61003
3when61003,3forsince)(61003
100when610032,2forsince)(61003
when3,anyforsince)(61003
3when61003,1forsince)(61003
610033,3forsince)(61003
22
3232
2222
22
2332
2222
>+−<Ω=+−
><+−=Ω≠+−
>+−<=Ω=+−
><≠+−
>+−>==+−
+−>==+−
nnncncnnn
nnnncnnn
nnnncnnn
cnncncnOnn
nnnncnOnn
nnncnOnn

43. Example 3.Example 3.
apply.andbothsince)(61003
100when61003,anyforsince)(61003
3when61003,3forsince)(61003
100when610032,2forsince)(61003
when3,anyforsince)(61003
3when61003,1forsince)(61003
610033,3forsince)(61003
22
22
3232
2222
22
2332
2222
ΩΘ=+−
>+−<Ω=+−
><+−=Ω≠+−
>+−<=Ω=+−
><≠+−
>+−>==+−
+−>==+−
Onnn
nnncncnnn
nnnncnnn
nnnncnnn
cnncncnOnn
nnnncnOnn
nnncnOnn

44. Example 3.Example 3.
applies.onlysince)(61003
apply.andbothsince)(61003
100when61003,anyforsince)(61003
3when61003,3forsince)(61003
100when610032,2forsince)(61003
when3,anyforsince)(61003
3when61003,1forsince)(61003
610033,3forsince)(61003
32
22
22
3232
2222
22
2332
2222
Onnn
Onnn
nnncncnnn
nnnncnnn
nnnncnnn
cnncncnOnn
nnnncnOnn
nnncnOnn
Θ≠+−
ΩΘ=+−
>+−<Ω=+−
><+−=Ω≠+−
>+−<=Ω=+−
><≠+−
>+−>==+−
+−>==+−

45. Example 3.Example 3.
applies.onlysince)(61003
applies.onlysince)(61003
apply.andbothsince)(61003
100when61003,anyforsince)(61003
3when61003,3forsince)(61003
100when610032,2forsince)(61003
when3,anyforsince)(61003
3when61003,1forsince)(61003
610033,3forsince)(61003
2
32
22
22
3232
2222
22
2332
2222
ΩΘ≠+−
Θ≠+−
ΩΘ=+−
>+−<Ω=+−
><+−=Ω≠+−
>+−<=Ω=+−
><≠+−
>+−>==+−
+−>==+−
nnn
Onnn
Onnn
nnncncnnn
nnnncnnn
nnnncnnn
cnncncnOnn
nnnncnOnn
nnncnOnn

46. Standard notations and common functionsStandard notations and common functions
• Floors and ceilings
    11 +<≤≤<− xxxxx

47. Standard notations and common functionsStandard notations and common functions
• Logarithms:
)lg(lglglg
)(loglog
logln
loglg 2
nn
nn
nn
nn
kk
e
=
=
=
=

48. Standard notations and common functionsStandard notations and common functions
• Logarithms:
For all real a>0, b>0, c>0, and n
b
a
a
ana
baab
ba
c
c
b
b
n
b
ccc
ab
log
log
log
loglog
loglog)(log
log
=
=
+=
=

49. Standard notations and common functionsStandard notations and common functions
• Logarithms:
b
a
ca
aa
a
b
ac
bb
bb
log
1
log
log)/1(log
loglog
=
=
−=

50. Standard notations and common functionsStandard notations and common functions
• Factorials
For the Stirling approximation:












Θ+





π=
ne
n
nn
n
1
12!
0≥n
)lg()!lg(
)2(!
)(!
nnn
n
non
n
n
Θ=
ω=
=