1. Dept of CS & IT
University of Sargodha
Advance Analysis of
Algorithms
Fall 2011
Fahad Maqbool
MS Computer Science
BS Software Engineering
1
2. Adv Analysis of Algo
Steps involved in writing a computer program:
• Problem formulation & specification
• Design of solution
• Implementation
• Testing
• Documentation
• Evaluation of the solution
2
3. Adv Analysis of Algo
Problem formulation & specification
• Half the battle is knowing what problem to solve.
• Most problems have no simple precise specification.
• Some problems are impossible to formulate.
• Example: Gourmet recipe, World peace, Etc.
3
4. Adv Analysis of Algo
How to formulate & specify a problem?
• Identify the problem parameters.
• Expressing the problem by a formal model.
• Looking for a solution for the model.
• In the absence of a solution, discover about the model.
4
5. Adv Analysis of Algo
How to model and solve a problem?
• Using any branch of Mathematics or Science.
• Identifying problems of numerical nature.
• With a suitable model, attempt to find a solution in terms of that
model
• Simultaneous linear equations for electrical circuits.
• Differential equations for predicting chemical reaction rates.
5
6. Adv Analysis of Algo
Algorithm
• Finite sequence of instructions.
• Each instruction having a clear meaning.
• Each instruction requiring finite amount of effort.
• Each instruction requiring finite time to complete.
6
7. Adv Analysis of Algo
Algorithm
• Finite sequence of instructions.
An input should not take the program in an infinite loop
• Each instruction having a clear meaning.
Very subjective. What is clear to me, may not be clear to you.
• Each instruction requiring finite amount of effort.
Very subjective. Finite on a super computer or a 486?
• Each instruction requiring finite time to complete.
Very subjective. 1 min, 1 hr, 1 year or a lifetime?
7
8. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
GIVEN: A complex intersection.
OBJECTIVE: Traffic light with minimum phases.
SOLUTION:
• Identify permitted turns, going straight is a “turn”.
• Make group of permitted turns.
• Make the smallest possible number of groups.
• Assign each phase of the traffic light to a group.
8
9. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
D
E
C
Rawalpindi
B
A
An intersection
Roads C and E are one way, others are two way.
9
10. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
There are 13 permitted turns.
Some turns such as AB (from A to B) and EC can be carried
out simultaneously.
Other like AD and EB cross each other and can not be carried
out simultaneously.
The traffic light should permit AB and EC simultaneously, but
should not allow AD and EB. 10
11. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
D
AB & AC E
C
AD & EB
B
A
An intersection
11
12. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
SOLUTION:
• We model the problem using a structure called graph
G(V,E).
• A graph consists of a set of points called vertices V, and lines
connecting the points, called edges E.
•Drawing a graph such that the vertices represent turns.
• Edges between those turns that can NOT be performed
simultaneously.
12
13. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
D
E AB AC AD
C
BA BC BD
B DA DB DC
A
An intersection
EA EB EC ED
Partial graph of incompatible turns
13
14. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
Partial table of incompatible turns
14
15. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
SOLUTION:
The graph can aid in solving our problem.
A coloring of a graph is an assignment of a color to each
vertex of the graph, so that no two vertices connected by an
edge have the same color.
Our problem is of coloring the graph of incompatible turns
using as few colors as possible.
15
16. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
More on SOLUTION (Graph coloring):
Graph coloring problem has been studied for decades.
The theory of algorithms tells us a lot about it.
Unfortunately this belongs to a class of problems called as
NP-Complete problems.
For such problems, all known solutions are basically “try all
possibilities”
In case of coloring, try all assignments of colors. 16
17. Adv Analysis of Algo
Using a model to solve a complicated traffic light
problem
Approaches to attempting NP-Complete problems:
1. If the problem is small, might attempt to find an optimal
solution exhaustively.
2. Look for additional information about the problem.
3. Change the problem a little, and look for a good, but not
necessarily optimal solution.
An algorithm that quickly produces good but not
necessarily optimal solutions is called a heuristic.
17
18. Using model to solve complicated traffic light problem
A reasonable heuristic for graph coloring is the greedy
“algorithm”.
Try to color as many vertices as possible with the first color,
and then as many uncolored vertices with the second color,
and so on.
The approach would be:
1. Select some uncolored vertex, and color with new color.
2. Scan the list of uncolored vertices.
3. For each uncolored vertex, determine whether it has an edge to any
vertex already colored with the new color.
4. If there is no such edge, color the present vertices with the new
color.
18
19. Using model to solve complicated traffic light problem
This is called “greedy”, because it colors a vertex,
whenever it can, without considering potential
drawbacks.
3
1 5 2
Three colors used
3
4
1 5 2
Two colors used
(optimum)
4
19
20. Analysis of Algo
Using a model to solve a complicated traffic light
problem
Work SMART
not
HARD
20
21. Questions that will be answered
• What is a “good” or "efficient" program?
• How to measure the efficiency of a program?
• How to analyse a simple program?
• How to compare different programs?
• What is the big-O notation?
• What is the impact of input on program performance?
• What are the standard program analysis techniques?
• Do we need fast machines or fast algorithms?
21/31
22. Which is better?
The running time of a program.
Program easy to understand?
Program easy to code and debug?
Program making efficient use of resources?
Program running as fast as possible?
22/31
23. Measuring Efficiency
Ways of measuring efficiency:
• Run the program and see how long it takes
• Run the program and see how much memory it uses
•Lots of variables to control:
• What is the input data?
• What is the hardware platform?
• What is the programming language/compiler?
• Just because one program is faster than another right now, means it
will always be faster?
23/31
24. Measuring Efficiency
Want to achieve platform-independence
• Use an abstract machine that uses steps of time and units
of memory, instead of seconds or bytes
• - each elementary operation takes 1 step
• - each elementary instance occupies 1 unit of memory
24/31
25. A Simple Example
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A
int Sum(int A[], int N) {
int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}
How should we analyse this?
25/31
26. A Simple Example
Analysis of Sum
1.) Describe the size of the input in terms of one ore more
parameters:
- Input to Sum is an array of N ints, so size is N.
2.) Then, count how many steps are used for an input of that size:
- A step is an elementary operation such as
+, <, =, A[i]
26/31
27. A Simple Example
Analysis of Sum (2)
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A
int Sum(int A[], int N {
int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5
6 7 1,2,8: Once
return s; 3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3
27/31
28. Analysis: A Simple Example
How 5N+3 Grows
Estimated running time for different values of N:
N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps
As N grows, the number of steps grow in linear proportion to
N for this Sum function.
28/31
29. Analysis: A Simple Example
What Dominates?
What about the 5 in 5N+3? What about the +3?
• As N gets large, the +3 becomes insignificant
•5 is inaccurate, as different operations require varying
amounts of time
What is fundamental is that the time is linear in N.
Asymptotic Complexity: As N gets large, concentrate on the
highest order term:
• Drop lower order terms such as +3
• Drop the constant coefficient of the highest order term i.e. N
29/31
30. Analysis: A Simple Example
Asymptotic Complexity
• The 5N+3 time bound is said to "grow asymptotically" like N
• This gives us an approximation of the complexity of the
algorithm
• Ignores lots of (machine dependent) details, concentrate
on the bigger picture
30/31
31. Comparing Functions
Definition: If f(N) and g(N) are two complexity functions, we say
f(N) = O(g(N))
(read "f(N) as order g(N)", or "f(N) is big-O of g(N)")
if there are constants c and N0 such that for N N0, T(N) cN
f(N) c g(N)
for all sufficiently large N.
31/31
33. Comparing Functions
100n2 Vs 5n3, which one is better?
Differenec of functions
20000
10000
0
-10000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
-20000
-30000
-40000
-50000
-60000
-70000
-80000
-90000
33/31
34. Comparing Functions
Why is this useful?
As inputs get larger, any algorithm of a smaller order will
be more efficient than an algorithm of a larger order
0.05 N2 = O(N2)
Time (steps)
3N = O(N)
Input (size)
N = 60
34/31
35. Comparing Functions
What is the relationship between the following?
What is the polynomial representation for the following?
f1(n)
f2(n)
f3(n)
F(n)
f4(n)
Input size (n)
35/31
36. Big-O Notation
• Think of f(N) = O(g(N)) as
" f(N) grows at most like g(N)" or
" f grows no faster than g"
(ignoring constant factors, and for large N)
Important:
• Big-O is not a function!
• Never read = as "equals"
• Examples:
5N + 3 = O(N)
37N5 + 7N2 - 2N + 1 = O(N5)
36/31
38. Size does matter
Common Orders of Growth
Let N be the input size, and b and k be constants
O(k) = O(1) Constant Time
Increasing Complexity
O(logbN) = O(log N) Logarithmic Time
O(N) Linear Time
O(N log N)
O(N2) Quadratic Time
O(N3) Cubic Time
...
O(kN) Exponential Time
38/31
39. Size does matter
What happens if we double the input size N?
N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~1076
If one execution takes 10-6 seconds,
2256 will take about 1061 centuries
39/31
40. Size does matter
Big Numbers
Suppose a program has run time O(n!) and the run time
for n = 10 is 1 second
For n = 12, the run time is 2 minutes
For n = 14, the run time is 6 hours
For n = 16, the run time is 2 months
For n = 18, the run time is 50 years
For n = 20, the run time is 200 centuries
40/31
41. Standard Analysis Techniques
Constant time statements
Simplest case: O(1) time statements
• Assignment statements of simple data types
int x = y;
• Arithmetic operations:
x = 5 * y + 4 - z;
• Array referencing:
A[j] = 5;
• Array assignment:
j, A[j] = 5;
• Most conditional tests:
if (x < 12) ...
41/31
42. Standard Analysis Techniques
Analyzing Loops
Any loop has two parts:
1. How many iterations are performed?
2. How many steps per iteration?
int sum = 0,j;
for (j=0; j < N; j++)
sum = sum +j;
- Loop executes N times (0..N-1)
- 4 = O(1) steps per iteration
- Total time is N * O(1) = O(N*1) = O(N)
42/31
43. Standard Analysis Techniques
Analyzing Loops (2)
What about this for-loop?
int sum =0, j;
for (j=0; j < 100; j++)
sum = sum +j;
- Loop executes 100 times
- 4 = O(1) steps per iteration
- Total time is 100 * O(1) = O(100 * 1) = O(100) = O(1)
43/31
44. Standard Analysis Techniques
Analyzing Loops (3)
What about while-loops?
Determine how many times the loop will be executed:
bool done = false;
int result = 1, n;
scanf("%d", &n);
while (!done){
result = result *n;
n--;
if (n <= 1) done = true;
}
Loop terminates when done == true, which happens
after N iterations. Total time: O(N)
44/31
45. Standard Analysis Techniques
Nested Loops
Treat just like a single loop and evaluate each level of
nesting as needed:
int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;
Start with outer loop:
- How many iterations? N
- How much time per iteration? Need to evaluate inner loop
Inner loop uses O(N) time
Total time is N * O(N) = O(N*N) = O(N2)
PRODUCT RULE
45/31
46. Standard Analysis Techniques
Nested Loops (2)
What if the number of iterations of one loop depends on the
counter of the other?
int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;
Analyze inner and outer loop together:
- Number of iterations of the inner loop is:
0 + 1 + 2 + ... + (N-1) = O(N2) How?
46/31
47. Standard Analysis Techniques
Digression
When doing Big-O analysis, we sometimes have to compute
a series like:
1 + 2 + 3 + ... + (N-1) + N
What is the complexity of this?
Remember Gauss:
47/31
48. Standard Analysis Techniques
Sequence of Statements
For a sequence of statements, compute their complexity functions
individually and add them up
for (j=0; j < N; j++)
for (k =0; k < j; k++)
sum = sum + j*k; O(N2)
for (l=0; l < N; l++)
sum = sum -l; O(N)
printf("sum is now %f", sum); O(1)
Total cost is O(N2) + O(N) +O(1) = O(N2)
SUM RULE
48/31
49. Standard Analysis Techniques
Conditional Statements
What about conditional statements such as
if (condition)
statement1;
else
statement2;
where statement1 runs in O(N) time and statement2 runs in O(N2) time?
We use "worst case" complexity: among all inputs of size N, what is the maximum
running time?
The analysis for the example above is O(N2)
49/31
50. Fast machine Vs Fast Algorithm
Get a 10 times fast computer, that can do a job in 103
seconds for which the older machine took 104
seconds .
Comparing the performance of algorithms with time
complexities T(n)s of n, n2 and 2n (technically not an
algorithm) for different problems on both the
machines.
Question: Is it worth buying a 10 times fast machine?
50/31
