CCS 3102 Lecture 3_ Complexity theory.pdf

Design and Analysis of
Algorithms
Complexity theory
Department of Computer Science

Complexity theory
❖The complexity of an algorithm is simply the
amount of work the algorithm performs to complete
its task.
❖Complexity theory is the study of the cost of
solving interesting problems. It measures the
amount of resources needed.
 Time
 Space
❖Two aspects
 Upper bounds: give a fast algorithm
 Lower bounds: no algorithm is faster.
Monday, March 6, 2023
Design and Analysis of Computer Algorithm 2

What is Algorithm Analysis?
❖How to estimate the time required for an
algorithm
❖Techniques that drastically reduce the
running time of an algorithm
❖A mathemactical framework that more
rigorously describes the running time of an
algorithm
Monday, March 6, 2023 3
Design and Analysis of Computer Algorithm

Algorithm Analysis(1/5)
❖Measures the efficiency of an algorithm or its
implementation as a program as the input size
becomes very large
❖We evaluate a new algorithm by comparing its
performance with that of previous approaches
 Comparisons are asymptotic analyses of classes of
algorithms
❖We usually analyze the time required for an
algorithm and the space required for a
datastructure

Algorithm Analysis (2/5)
❖Many criteria affect the running time of an
algorithm, including
 speed of CPU, bus and peripheral hardware
 design think time, programming time and
debugging time
 language used and coding efficiency of the
programmer
 quality of input (good, bad or average)

❖Programs derived from two algorithms for solving
the same problem should both be
 Machine independent
 Language independent
 Environment independent (load on the
system,...)
 Amenable to mathematical study
 Realistic

❖We estimate the algorithm's performance based on
the number of key and basic operations it requires
to process an input of a given size
❖For a given input size n we express the time T to
run the algorithm as a function T(n)
❖Concept of growth rate allows us to compare
running time of two algorithms without writing two
programs and running them on the same computer

❖Formally, let T(A,L,M) be total run time for
algorithm A if it were implemented with language L
on machine M. Then the complexity class of
algorithm A is
O(T(A,L1,M1) U O(T(A,L2,M2)) U O(T(A,L3,M3)) U ...
❖Call the complexity class V; then the complexity of
A is said to be f if V = O(f)
❖The class of algorithms to which A belongs is said
to be of at most linear/quadratic/ etc.

Performance Analysis
❖Predicting the resources which are required by
an algorithm to perform its task.
❖An algorithm is said to be efficient and fast, if it
takes less time to execute and consumes less
memory space. The performance of an
algorithm is measured on the basis of following
properties :
 Space Complexity
 Time Complexity

Input Size
❖Time and space complexity
 This is generally a function of the input size
▪ E.g., sorting, multiplication
 How we characterize input size depends:
▪ Sorting: number of input items
▪ Multiplication: total number of bits
▪ Graph algorithms: number of nodes & edges
▪ Etc

11
Running Time
❖ Most algorithms transform input
objects into output objects.
❖ The running time of an
algorithm typically grows with
the input size.
❖ Average case time is often
difficult to determine.
❖ We focus on the worst case
running time.
 Easier to analyze
 Crucial to applications such as
games, finance and robotics
0
20
40
60
80
100
120
Running
Time 1000 2000 3000 4000
Input Size
best case
average case
worst case
Design and Analysis of Computer Algorithm Monday, March 6, 2023

Running Time
❖Number of primitive steps that are executed
 Except for time of executing a function call, most
statements roughly require the same amount of
time
▪ y = m * x + b
▪ c = 5 / 9 * (t - 32 )
▪ z = f(x) + g(y)
❖We can be more exact if need be

Space Complexity
❖ Its the amount of memory space required by the algorithm,
during the course of its execution.
❖ It must be taken seriously for multi-user systems and in
situations where limited memory is available.
❖ An algorithm generally requires space for following
components :
❖ Instruction Space: Its the space required to store the executable version
of the program. This space is fixed, but varies depending upon the
number of lines of code in the program.
❖ Data Space: Its the space required to store all the constants and
variables value.
❖ Environment Space : Its the space required to store the environment
information needed to resume the suspended function.

Constant Space Complexity
int square (int a)
{
return a*a;
}
❖If any algorithm requires a fixed amount of
space for all input values then that space
complexity is said to be Constant Space
Complexity

Linear Space Complexity
int sum (int A[ ], int n)
{
int sum = 0, i;
for (i = 0; i < n; i++)
sum = sum + A[i];
return sum;
}
❖If the amount of space required by an algorithm is
increased with the increase of input value, then that
space complexity is said to be Linear Space
Complexity

Time Complexity
❖The time complexity of an algorithm is the total
amount of time required by an algorithm to
complete its execution.
❖If any program requires fixed amount of time for
all input values then its time complexity is said
to be Constant Time Complexity
int sum (int a, int b)
{
return a+b;
}

Linear Time Complexity
❖If the amount of time required by an algorithm is
increased with the increase of input value then
that time complexity is said to be Linear Time
Complexity

Asymptotic Performance
❖How does the algorithm behave as the
problem size gets very large?
 Running time
 Memory/storage requirements
 Bandwidth/power requirements/logic gates/etc.

Order of Growth
❖This refers to the rate of growth of the running time
of an algorithm
❖Only the leading term of a formula that matters
since the lower-order terms are relatively
insignificant for large n
❖E.g. given an2+bn+c for some constants a, b and
c, the leading term is an2
❖We also ignore the leading term’s constant
coefficient since it has less significant than the rate
of growth. Hence we write Ɵn2(“theta of n-
squared”)

Rate of Growth
❖Consider the example of buying elephants and
goldfish:
Cost: cost_of_elephants + cost_of_goldfish
Cost ~ cost_of_elephants (approximation)
❖The low order terms in a function are relatively
insignificant for large n
n4 + 100n2 + 10n + 50 ~ n4
i.e., we say that n4 + 100n2 + 10n + 50 and n4
have the same rate of growth

Function of Growth rate

Asymptotic notation
❖Asymptotic efficiency of algorithms is when we
look at input sizes large enough to make only
the order of growth of the running time relevant
❖Studying how the running time of an algorithm
increases with the size of input in the limit as
the size of input increases without bound
❖Asymptotically more efficient algorithm is best
for all but very small inputs

Big O Notation
❖Big O notation: asymptotic “less than”:
❖f(n)  O(g(n)) if and only if  c > 0, and n0 , so that
0 ≤ f(n) ≤ cg(n) for all n ≥ n0
❖f(n) = O(g(n)) means f(n)  O(g(n)) (i.e., at most)
 We say that g(n) is an asymptotic upper bound
for f(n).
❖It also means:
❖E.g. f(n) = O(n2) if there exists c, n0 > 0 such that
f(n) ≤ cn2, for all n ≥ n0.
lim ≤ c
n→
f(n)
g(n)

Big-Oh Rules
❖If f(n) is a polynomial of degree d, then f(n) is O(nd),
i.e.,
1. Drop lower-order terms
2. Drop constant factors
❖Use the smallest possible class of functions
 Say “2n is O(n)” instead of “2n is O(n2)”
❖Use the simplest expression of the class
 Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

When to use – big Oh
❖ Big “oh” - asymptotic upper bound on the growth of
an algorithm
❖ When do we use Big Oh?
1. Theory of NP-completeness
2. To provide information on the maximum number of
operations that an algorithm performs
 Insertion sort is O(n2) in the worst case
▪ This means that in the worst case it performs at
most cn2 operations

Omega  Notation
❖  notation: asymptotic “greater than”:
❖f(n)   (g(n)) if and only if  c > 0, and n0 ,
so that 0 ≤ cg(n) ≤ f(n) for all n ≥ n0
❖f(n) =  (g(n)) means f(n)   (g(n)) (i.e, at
least)
 We say g(n) is an asymptotic lower bound of
f(n).
❖It also means: lim ≥ c
n→
f(n)
g(n)

When to use – Omega
Omega - asymptotic lower bound on the growth of an
algorithm or a problem*
When do we use Omega?
1. To provide information on the minimum number of
operations that an algorithm performs
 Insertion sort is (n) in the best case
▪ This means that in the best case its instruction
count is at least cn,
 It is (n2) in the worst case
▪ This means that in the worst case its instruction
count is at least cn2

When to use – Omega cont.
2.To provide information on a class of algorithms
that solve a problem
 Sort algorithms based on comparison of keys are
(nlgn) in the worst case
▪ This means that all sort algorithms based only on
comparison of keys have to do at least cnlgn
operations
 Any algorithm based only on comparison of keys
to find the maximum of n elements is (n) in
every case
▪ This means that all algorithms based only on
comparison of keys to find maximum have to do at
least cn operations

Theta  Notation
❖Θ notation: asymptotic “equality”:
❖Combine lower and upper bound
❖Means tight: of the same order
❖𝑓 𝑛 ∈ Θ 𝑔 𝑛 if and only if ∃ 𝑐1, 𝑐2 > 0, and 𝑛0 ,
such that 𝑐1𝑔 𝑛 ≤ 𝑓 𝑛 ≤ 𝑐2𝑔 𝑛 for any 𝑛 ≥ 𝑛0
❖𝑓 𝑛 = 𝛩 𝑔 𝑛 means 𝑓 𝑛 ∈ 𝛩 𝑔 𝑛
 We say g(n) is an asymptotically tight bound for
f(n).
❖It also means:
 𝑐1 ≤ lim
𝑛→∞
𝑓(𝑛)
𝑔(𝑛)
≤ 𝑐2

When to use - Theta
❖Theta - asymptotic tight bound on the growth rate
 Insertion sort is (n2) in the worst and average
cases
▪ The means that in the worst case and average
cases insertion sort performs cn2 operations
 Binary search is (lg n) in the worst and average
cases
▪ The means that in the worst case and average
cases binary search performs clgn operations
❖Note: We want to classify an algorithm using Theta.
❖Little “oh” - used to denote an upper bound that is not
asymptotically tight. n is in o(n3). n is not in o(n)

Illustration
𝑓 𝑛 = 𝑂(𝑔 𝑛 ) 𝑓 𝑛 = Ω(𝑔 𝑛 ) 𝑓 𝑛 = Θ(𝑔 𝑛 )

Does 5n+2 O(n)?
Proof: From the definition of Big Oh, there must exist c>0
and integer N>0 such that 0  5n+2cn for all nN.
Dividing both sides of the inequality by n>0 we get:
0  5+2/nc.
2/n  2, 2/n>0 becomes smaller when n increases
There are many choices here for c and N.
If we choose N=1 then c  5+2/1= 7.
If we choose c=6, then 0  5+2/n6. So N  2.
In either case (we only need one!) we have a c>o and N>0
such that 0  5n+2cn for all n  N. So the definition is
satisfied and 5n+2 O(n)

Is 5n-20  (n)?
Proof: From the definition of Omega, there must exist c>0 and
integer N>0 such that 0  cn  5n-20 for all nN
Dividing the inequality by n>0 we get: 0  c  5-20/n for all nN.
20/n  20, and 20/n becomes smaller as n grows.
There are many choices here for c and N.
Since c > 0, 5 – 20/n >0 and N >4
For example, if we choose c=4, then 5 – 20/n  4 and N 
20
In this case we have a c>o and N>0 such that 0  cn  5n-20
for all n  N. So the definition is satisfied and 5n-20   (n)

Is 1/2n2-3n O(n2)?

.
and
So
.
all
for
1/4.
c
Choose
finite
for
0
Since
6.
Since
.
that
such
and
exist
must
There
12
4
1
12
3
2
1
4
1
.
2
/
1
,
3
and
3
2
1
0
0
3
2
1
0
get
we
0
2
by
Dividing
all
for
3
2
2
1
2
0
0
0
=
=

−

=



−


−




−




N
/
c
n
n
c
n
/n
N
,
c
n
c
n
N
n
n
n
cn
N
c
N
Is 1/2n2-3n (n2)?

Kinds of Analysis
❖Worst case (usually)
 Provides an upper bound on running time
 An absolute guarantee
❖Average case (sometimes)
 Provides the expected running time
 Very useful, but treat with care: what is “average”?
▪ Random (equally likely) inputs
▪ Real-life inputs
❖Best-case: (rarely)
 Cheat with a slow algorithm that works fast on
some input.

Empirical Analysis of Algorithms
❖In practice, we will often need to resort to empirical
rather than theoretical analysis to compare algorithms.
 We may want to know something about performance of
the algorithm “on average” for real instances.
 Our model of computation may not capture important
effects of the hardware architecture that arise in
practice.
 There may be implementation details that affect
constant factors and are not captured by asymptotic
analysis.
❖For this purpose, we need a methodology for comparing
algorithms based on real-world performance.

Issues to Consider
❖Empirical analysis introduces many more factors
that need to be controlled in some way.
 Test platform (hardware, language, compiler)
 Measures of performance (what to compare)
 Benchmark test set (what instances to test on)
 Algorithmic parameters
 Implementation details
❖It is much less obvious how to perform a rigorous
analysis in the presence of so many factors.
❖Practical considerations prevent complete testing.

Measures of Performance
❖For the time being, we focus on sequential
algorithms.
❖What is an appropriate measure of performance?
❖What is the goal?
 Compare two algorithms.
 Improve the implementation of a single algorithm.
❖Possible measures
 Empirical running time (CPU time, wallclock)
 Representative operation counts

Measuring Time
❖There are three relevant measures of time taken by a
process.
 User time measures the amount of time (number of
cycles taken by a process in “user mode.”
 System time the time taken by the kernel executing on
behalf of the process.
 Wallclock time is the total “real” time taken to execute
the process.
❖Generally speaking, user time is the most relevant,
though it ignores some important operations (I/O, etc.).
❖Wallclock time should be used cautiously/sparingly, but
may be necessary for assessment of parallel codes,

Representative Operation Counts
❖In some cases, we may want to count operations, rather
than time
 Identify bottlenecks
 Counterpart to theoretical analysis
❖What operations should we count?
 Profilers can count function calls and executions of
individual lines of code to identify bottlenecks.
 We may know a priori what operations we want to
measure
(example: comparisons and swaps in sorting).

Test Sets
❖It is crucial to choose your test set well.
❖The instances must be chosen carefully in order to allow
proper conclusions to be drawn.
❖We must pay close attention to
 their size,
 inherent difficulty,
 and other important structural properties.
❖This is especially important if we are trying to distinguish
among multiple algorithms.
❖Example: Sorting

Comparing Algorithms
❖Given a performance measure and a test set, the
question still arises how to decide which algorithm is
“better.”
❖We can do the comparison using some sort of summary
statistic.
 Arithmetic mean
 Geometric mean
 Variance
❖ Performance profiles allow comparison of algorithms
across an entire test set without loss of information.
❖ They provide a visual summary of how algorithms
compare on a performance measure across a test set

Accounting for Stochasticity
❖In empirical analysis, we must take account of the fact
that running times are inherently stochastic.
❖If we are measuring wallclock time, this may vary
substantially for seemingly identical executions.
❖In the case of parallel processing, stochasticity may also
arise due to asynchronism (order of operations).
❖In such case, multiple identical runs may be used to
estimate the affect of this randomness.
❖If necessary, statistical analysis may be used to analyze
the results, but this is beyond the scope of this course.

Empirical versus Theoretical Analysis
❖For sequential algorithms, asymptotic analysis is often
good enough for choosing between algorithms.
❖It is less ideal with respect to tuning of implementation
details.
❖For parallel algorithms, asymptotic analysis is far more
problematic.
❖The details not captured by the model of computation
can matter much more.
❖There is an additional dimension on which we must
compare algorithms: scalability

CCS 3102 Lecture 3_ Complexity theory.pdf

Recommended

Recommended

More Related Content

Similar to CCS 3102 Lecture 3_ Complexity theory.pdf

Similar to CCS 3102 Lecture 3_ Complexity theory.pdf (20)

Recently uploaded

Recently uploaded (20)

CCS 3102 Lecture 3_ Complexity theory.pdf