1. Data Structures and Algorithms
Lecture 1: Asymptotic Notations

2. Basic Terminologies
• Algorithm
– Outline
– Essence of a computational procedure
– Step by step instructions
• Program
– Implementation of an Algorithm in some
programming language
• Data Structure
– Organization of Data needed to solve the problem
effectively
– Data Structure you already know: Array

3. Algorithmic Problem
Specification of
Input
?
Specification of
Output as a
function of Input
• Infinite number of input instances satisfying the
specification
• E.g.: Specification of Input:
• A sorted non-decreasing sequence of natural numbers
of non-zero, finite length:
• 1, 20,908,909,1000,10000,1000000
• 3

4. Algorithmic Solution
Input instance
adhering to
Specification
Algorithm
Output related
to the Input as
required
• Algorithm describes the actions on the input
instances
• Infinitely many correct algorithms may exist
for the same algorithmic problem

5. Characteristics of a Good Algorithm
• Efficient
– Small Running Time
– Less Memory Usage
• Efficiency as a function of input size
– The number of bits in a data number
– The number of data elements

6. Measuring the running time
• Experimental Study
– Write a program that implements the algorithm
– Run the program with data sets of varying size and
composition
– Use system defined functions like clock() to get an
measurement of the actual running time

7. Measuring the running time
• Limitations of Experimental Study
– It is necessary to implement and test the
algorithm in order to determine its running time.
– Experiments can be done on a limited set of
inputs, and may not be indicative of the running
time on other inputs not included in the
experiment
– In order to compare 2 algorithms same hardware
and software environments must be used.

8. Beyond Experimental Study
• We will develop a general methodology for
analyzing the running time of algorithms. This
approach
– Uses a high level description of the algorithm
instead of testing one of its implementations
– Takes into account all possible inputs
– Allows one to evaluate the efficiency of any
algorithm in a way that is independent of the
hardware and software environment.

9. Pseudo - Code
• A mixture of natural language and high level
programming concepts that describes the main
ideas behind a generic implementation of a data
structure or algorithm.
• E.g. Algorithm arrayMax(A,n)
– Input: An array A storing n integers
– Output: The maximum element in A.
currMax
A[0]
for i
1 to n-1 do
if currMax < A[i] then currMax
return currMax
A[i]

10. Pseudo - Code
• It is more structured that usual prose but less
formal than a programming language
• Expressions:
– Use standard mathematical symbols to describe
numeric and Boolean expressions
– Use ‘
’ for assignment instead of “=”
– Use ‘=’ for equality relationship instead of “==”
• Function Declarations:
– Algorithm name (param1 , param2)

11. Pseudo - Code
• Programming Constructs:
–
–
–
–
–
Decision structures: if…then….[else….]
While loops: while…..do…..
Repeat loops: Repeat………until…….
For loop: for……do………..
Array indexing: A[i], A[I,j]
• Functions:
– Calls: return_type function_name(param1 ,param2)
– Returns: return value

12. Analysis of Algorithms
• Primitive Operation:
– Low level operation independent of programming
language.
– Can be identified in a pseudo code.
• For e.g.
– Data Movement (assign)
– Control (branch, subroutine call, return)
– Arithmetic and Logical operations (e.g. addition
comparison)
– By inspecting the pseudo code, we can count the
number of primitive operations executed by an
algorithm.

13. Example: Sorting
OUPUT
Permutation of the sequence of
numbers in non-decreasing order
INPUT
Sequence of
numbers
a1,a2,a3,a4,….an
2, 5 , 4, 10, 7
Sort
Correctness (Requirements for the
output)
For any given input the algorithm
halts with the output:
• b1 < b2 < b3 < b4 < ….. < bn
• b1, b2, b3, b4,…. bn is a permutation
of a1, a2, a3, a4, …. an
b1,b2,b3,b4,….bn
2, 4 , 5, 7, 10
Running Time depends on
• No. of elements (n)
• How sorted (partial) the numbers are?
• Algorithm

14. Insertion Sort
• Used while playing cards
3
4
6
8
9
A
1
i j
Strategy
• Start “empty
handed”
• Insert a card in the
right position of the
already sorted hand
• Continue until all
cards are inserted or
sorted
7
2
5
1
n
INPUT: An array of Integers A[1..n]
OUTPUT: A permutation of A such that A[1] <=
A[2] <= A[3]<= ..<=A[n]
For j
2 to n do
key A[j]
Insert A*j+ into sorted sequence A*1…j-1]
i
j-1
while i >0 and A[i] >key
do A[i+1] A[i]
i- A[i+1]
key

15. Analysis of Insertion Sort
• For j
2 to n do
•
key A[j]
•
Insert A[j] into sorted
sequence A*1…j-1]
•
i
j-1
•
while i >0 and A[i] >key
•
do A[i+1] A[i]
•
i- •
A[i+1] key
Total Time = n(c1 + c2 + c3 + c7) +
Cost
c1
c2
Times
n
n-1
c3
c4
c5
c6
c7
n-1
n-1
(c4 + c5 + c6)–( c2 +c3 +c5+c6+c7)
tj counts the number of times the values have to be shifted in one iteration

16. Best / Average/ Worst Case:
Difference made by tj
• Total Time = n(c1 + c2 + c3 + c7) +
+c3 +c5+c6+c7)
• Best Case
(c4 + c5 + c6)–( c2
– Elements already sorted
– tj = 1
– Running time = f(n) i.e. Linear Time
• Worst Case
– Elements are sorted in inverse order
– tj = j
– Running time = f(n2) i.e. Quadratic Time
• Average Case
– tj = j/2
– Running Time = f(n2) i.e. Quadratic Time

17. Best / Average/ Worst Case
• For a Specific input size say n
Running Time (sec)
5
Worst Case
4
Average Case
3
Best Case
2
1
A B C D E F G H I J K L M N
Input instance

18. Best / Average/ Worst Case
• Varying input size
Worst case
Running Time (sec)
50
Average Case
40
Best Case
30
20
10
10
20
30
40
Input Size
50
60

19. Best / Average/ Worst Case
• Worst Case:
– Mostly used
– It is an Upper bound of how bad a system can be.
– In cases like surgery or air traffic control knowing
the worst case complexity is of crucial importance.
• Average case is often as bad as worst case.
• Finding an average case can be very difficult.

20. Asymptotic Analysis
• Goal: Simplify analysis of running time by getting
rid of details which may be affected by specific
implementation and hardware
– Like rounding 1000001 to 1000000
– 3n2 to n2
• Capturing the essence: How the running time of
an algorithm increases with the size of the input
in the limit
– Asymptotically more efficient algorithms are best for
all but small inputs.

21. Asymptotic Notation
• The “big Oh” (O)
Notation
– Asymptotic Upper Bound
– f(n) = O(g(n)), if there
exists constants c and n0
such that f(n) ≤ cg(n) for n
≥ n0
– f(n) and g(n) are functions
over non negative nondecreasing integers
• Used for worst case
analysis
c. g(n)
f(n)
n0
n
f(n) = O(g(n))

22. Examples
• E.g. 1:
–
–
–
–
–
f(n) = 2n + 6
g(n) = n
c =4
n0 = 3
Thus, f(n) = O(n)
• E.g. 2:
– f(n) = n2
– g(n) = n
– f(n) is not O(n) because there exists no
constants c and n0 such that f(n) ≤
cg(n) for n ≥ n0
• E.g. 3:
–
–
–
–
–
f(n) = n2 + 5
g(n) = n2
c=2
n0 = 2
Thus, f(n) = O(n2)

23. Asymptotic Notation
• Simple Rules:
– Drop lower order terms and constant factors
• 50 nlogn is O(nlogn)
• 7n – 3 is O(n)
• 8n2logn + 5n2 + n is O(n2logn)
– Note that 50nlogn is also O(n5) but we will
consider it to be O(nlogn) it is expected that the
approximation should be of as lower value as
possible.

24. Comparing Asymptotic Analysis of
Running Time
• Hierarchy of f(n)
– Log n < n < n2 < n3 < 2n
• Caution!!!!!
– An algorithm with running time of 1,000,000 n is
still O(n) but might be less efficient than one
running in time 2n2, which is O(n2) when n is not
very large.

25. Examples of Asymptotic Analysis
• Algorithm: prefixAvg1(X)
• Input: An n element array X of numbers
• Output: An n element array A of numbers such
that A[i] is the average of elements X[0]…X[i].
for i 0 to n-1 do
a 0
for j 0 to i do
a a+X[j]
A[i] a/(i+1)
return A
n iterations
1 Step
i iterations
i = 0 , 1, …., i-1
• Analysis: Running Time O(n2) roughly.

26. Examples of Asymptotic Analysis
• Algorithm: prefixAvg2(X)
• Input: An n element array X of numbers
• Output: An n element array A of numbers such
that A[i] is the average of elements X[0]…X[i].
s 0
for i 0 to n do
s s + X[i]
A[i] s/(i+1)
return A
• Analysis: Running time is O(n)

27. Asymptotic Notation (Terminologies)
• Special cases of Algorithms
– Logarithmic : O(log n)
– Linear: O(n)
– Quadratic: O(n2)
– Polynomial: O(nk), k ≥ 1
– Exponential: O(an), a>1
• “Relatives” of Big Oh
– Big Omega (Ω(f(n)) : Asymptotic Lower Bound
– Big Theta (Θ(f(n)): Asymptotic Tight Bound

28. Asymptotic Notation
• The big Omega (Ω) Notation
– Asymptotic Lower bound
– f(n) = Ω(g(n), if there exists
constants c and n0 such that
c.g(n) ≤ f(n) for n > n0
• Used to describe best case
running times or lower
bounds of asymptotic
problems
– E.g: Lower bound of
searching in an unsorted
array is Ω(n).

29. Asymptotic Notation
• The Big Theta (Θ) notation
– Asymptotically tight bound
– f(n) = Θ(g(n)), if there exists
constants c1, c2 and n0 such
that c1.g(n) ≤ f(n) ≤ c2.g(n) for
n > n0
– f(n) = Θ(g(n)), iff f(n) = O(g(n))
and f(n) = Ω(g(n),
– O(f(n)) is often misused
instead of Θ(f(n))

30. Asymptotic Notation
• There are 2 more notations
– Little oh notation (o), f(n) = o(g(n))
• For every c there should exist a no. n0 such that f(n) ≤ o(g(n) for n >
n0.
– Little Omega notation (ω)
• Analogy with real numbers
–
–
–
–
–
f(n) = O(g(n)) ,
f(n) = Ω(g(n)),
f(n) = Θ(g(n)),
f(n) = o(g(n)),
f(n) = ω(g(n)),
f≤g
f≥g
f=g
f<g
f>g
• Abuse of Notation:
– f(n) = O(g(n)) actually means f(n) є O(g(n))

31. Comparison of Running times

32. Assignment
• Write a C program to implement i) Insertion
sort and ii) Bubble Sort.
• Perform an Experimental Study and
graphically show the time required for both
the program to run.
• Consider varied size of inputs and best case
and worst case for each of the input sets.