2. Course Objectives:
To develop problem solving abilities using mathematical theories.
To apply algorithmic strategies while solving problems.
To analyze performance of different algorithmic strategies in terms of time and space.
To develop time and space efficient algorithms.
To study algorithmic examples in distributed and concurrent environments
To Understand Multithreaded and Distributed Algorithms
3. Course Outcomes:
• On completion of the course, student will be able to
• CO1: Formulate the problem
• CO2: Analyze the asymptotic performance of algorithms
• CO3: Decide and apply algorithmic strategies to solve given
problem
• CO4: Find optimal solution by applying various methods
• CO5: Analyze and Apply Scheduling and Sorting Algorithms.
• CO6: Solve problems for multi-core or distributed or concurrent
environments
4. What will we study?
• Look at some classical algorithms on different kinds of
problems
• How to design an algorithm
• How to show that an algorithm works correctly
• How to analyze the performance of the algorithm
7. Algorithms
• Algorithm: Any well-defined computational procedure
that takes some value, or set of values, as input and
produces some value, or set of values, as output.
• Or; Algorithm: A method of solving a problem, using a
sequence of well-defined steps
• Example: Sorting problem
• Input: A sequence of n numbers <a1,a2,..,an>
• Output: A permutation of the input sequence such that
a1’<=a2’<=…..<=an’
8. Instances of a problem
• An algorithm is said to be correct if for every input instance,
it halts with the correct output
• An instance of a problem consists of all inputs needed to
compute a solution to the problem
• A correct algorithm solves the given computational
problem.
• An incorrect algorithm might not halt at all on some input
instance, or it might halt with other than the desired answer
9. What kind of
problem are
solved by
algorithms?
The Human
Genome Project
Identify all the
100,000 genes in
human DNA
Determine the
sequences of the 3
billion chemical
base pairs of DNA
The Internet
applications
Quickly access and
retrieve large
amount of
information such
as Google
10. Algorithms as a technology
• Efficiency:
• Different algorithms solve the same problem often
differ noticeably in their efficiency
• These differences can be much more significant
than difference due to hardware and software
11. Example
• The insertion sort takes time roughly equal to
c1n2(c1 is constant) to sort n items. But merge sort
takes time roughly equal to c2nlogn (c2 is constant)
• For example, assume a faster computer A
(1010instructions/sec) running insertion sort against
a slower computer B (107instructions/sec) running
merge sort.
• Suppose that c1=2，c2=50 and n= 107.
• the execution time of computer A is 2(107)2/
1010instructions/sec = 20,000seconds
• the execution time of computer B is 50 · 107lg
107/ 107instructions/sec = 1,163seconds
13. Sumarian
Arithmetic
Tablets (Tablettes
sumeriennes de
Shuruppek, Jestin, 1937,
Plates XXI and CXLII)
The first Algorithms:
Jean-Luc Chabert, A History of Algorithms: From the Pebble to the Microchip.
Springer Science & Business Media. pp. 7–8 (2012). ISBN 9783642181924.
History of Arithmetic Algorithms: for addition,
subtraction, multiplication, division, sums of fractions, approx.
square root:
• Sumerian [c. 3500 - 2000 BCE]
• Babylonian [c. 2500 BCE]
• Egyptian [c. 1550 BCE]
• Used a combination of base 10 and base 60 for specifying
numbers.
• Algorithms were generally explained by example
calculations, with specified numerical values, rather than
abstractly.
Babylonian
Arithmetic
Tablet
(Mathematics
Keilschrifttexte,
Neugebauer, 1935,
Plate 43)
15. • Diophantus of Alexandria [ca. 3rd century AD] author of Arithmetica a Greek
mathematician
• Gave procedures for solving algebraic equations.
• 1st Greek mathematician who recognized fractions as numbers - allowed
positive rational numbers for the coefficients of polynomials and their
solutions.
• 1st use of symbols as "place holders" for numbers.
• Diophantine equation: a polynomial equation, usually involving two or more
unknowns, where the only solutions (of interest) are the integer ones.
16. • Example Linear Diophantine Equation:
• Metrodorus [5th-century AD]: gives a Puzzle (“The Diophantus epitaph”) for calculating his age:
• “Here lies Diophantus,' the wonder behold. Through art algebraic, the stone tells how old:
• 'God gave him his boyhood one-sixth of his life,
• One twelfth more as youth while whiskers grew rife;
• And then yet one-seventh ere marriage begun;
• In five years there came a bouncing new son.
• Alas, the dear child of master and sage
• After attaining half the measure of his father's life chill fate took him.
• After consoling his fate by the science of numbers for four years, he ended his life.”
• The Puzzle implies that Diophantus' age x in years can be expressed as
• x = x / 6 + x / 12 + x / 7 + 5 + x / 2 + 4
• which can be solved as x = 84.
17. • Muhammad ibn Mūsā al-Khwārizmī [c. 780 – c. 850 AD]
• a Persian mathematician and astronomer who worked at the House of Wisdom.
• The word “algorithm” is derived from his name.
• the word al-jabr literally means "completion" or "rejoining”, which are processes in his algebra.
• Originally, “algorithm” referred to sets of rules and techniques used by algebra to solve
algebraic equations.
• Later “algorithm” was generalized to refer to any set of rules or techniques.
• al-Khwārizmī’s Al-jabr (Algebra) [c. 820s AD]: book “The Compendious Book on Calculation by
Completion and Balancing”.
• Described algorithms (mostly due to Diophantus of Alexandria) for solving algebraic equations
(but did not use symbols in his algorithms).
• al-Khwārizmī’s Algorithm to solve any linear or quadratic equation:
• Began by reducing the equation to one of six standard forms (where b and c are positive
integers):
• (1) ax2 = bx, (2) ax2 = c, (3) bx = c, (4) ax2 + bx = c, (5) ax2 + c = bx, (6) bx + c = ax2
• Then expressed the solutions by use of square-root function and the arithmetic operations
• al-Khwārizmī’s “On the Calculation with Hindu Numerals” [c. 820 AD]:
described the Hindu–Arabic numeral system, which spread throughout
the Middle East and Europe. A page from al-Khwārizmī's
Algebra
Muhammad
ibn Musa al-
Khwarizmi
stamp
18. Motivation for his
Algebra:
• “...what is easiest and most useful
in arithmetic, such as men
constantly require in cases of
inheritance, legacies, partition,
lawsuits, and trade, and in all their
dealings with one another, or
where the measuring of lands, the
digging of canals, geometrical
computations, and other objects of
various sorts and kinds are
concerned.”
19. History of Algorithm
• Archimedes [ca. 300 BCE]:
• Greek mathematician
• 1st algorithm for approximately computing π up to any given precision.
• Estimated π by using inscribed and circumscribed polygons.
• Newton’s Algorithm [1669]: The most important numerical algorithm ever invented !
• Iterative algorithm for find an approximate root of a continuous function f(x).
• Joseph Fourier [1822]: Fourier Transform. Determines a decomposition of a time-dependant signal at n points into a sum of n
sinusoidal functions of distinct frequencies.
• James Cooley & John Tukey’s [1965]: Fast Fourier Transform (FFT) Algorithm.
• Randomized Algorithm: an algorithm that uses random numbers in its execution.
• Monte Carlo Randomized Algorithms:
• Randomized algorithms that make use of the average of multiple repeated randomized executions to approximate a given value.
• Monte Carlo Estimation of π
20. History of Algorithm
• Earliest Method: Georges-Louis Leclerc & Comte de Buffon
• Buffon’s Needles [18th Century]: Randomized approximation algorithm using short segments (needles) rather than points.
• Metropolis Algorithm [1946]: Monte Carlo algorithm for sampling from multi-dimensional distributions.
• Linear Programming Problem [1947]: a set of m linear inequalities over n variables, and a linear objective vector.
• A very important combinatorial optimization problem, with many applications.
• George Dantzig's Simplex Algorithm for Linear Programming [1947]:
• On each iteration: makes a pivot, that changes linear basis to improve in optimization direction.
• von Neumann[ca. 1950s]:
• Suggested 1st interior point algorithm for Linear Programming
• Karmarkar, N. [1984]: A provably efficient algorithm for Linear Programming
21. History of Algorithm
• Efficient Algorithms:
• An algorithm is termed efficient if it runs in time polynomial in the input size.
• The SAT Problem:
• SAT is the problem where you are input a propositional logical formula F, and you wish to find an assignment of its Boolean
variables that makes F true.
• Example: formula F = (X OR NOT(Y)) AND (NOT(X) OR Y) has satisfying assignment X = 1 and Y = 1.
• There is a large class of Combinatorial Problems that likely have no efficient algorithms:
• Steve Cook and Richard Karp showed that a large class of combinatorial problems (known as NP complete problems) can be
represented as SAT problems
• So if any NP complete problem has an efficient algorithm, then they all do.
• But no one has found such an efficient algorithm, and most mathematicians believe there is no such efficient algorithm.
22. Algorithms
• An algorithm is an exact specification of how to solve a computational
problem
• An algorithm must specify every step completely, so a computer can
implement it without any further “understanding”
• An algorithm must work for all possible inputs of the problem.
• Algorithms must be:
• Correct: For each input produce an appropriate output
• Efficient: run as quickly as possible, and use as little memory as possible – more about
this later
• There can be many different algorithms for each computational problem.
23. Correctness of Euclid’s Algorithm
• Theorem: When Euclid’s GCD algorithm terminates, it returns the
mathematical GCD of x and y.
• Notation: Let g be the GCD of the original values of x and y.
• Loop Invariant Lemma: For all k 0, The values of x, y after k rounds of the
loop satisfy GCD(x,y)=g.
• Proof of lemma: next slide.
• Proof of Theorem: The method returns when y=0. By the loop invariant
lemma, at this point GCD(x,y)=g. But GCD(x,0)=x for every integer x (since
x|0 and x|x). Thus g=x, which is the value returned by the code.
• Still Missing: The algorithm always terminates.
26. Proving Correctness
• Counterexample (indirect proof )
• Induction (direct proof )
• Loop Invariant
• proof by cases/enumeration
• proof by chain of is
• proof by contradiction
• proof by contrapositive
27. Proving Correctness
-Proof by Counterexample
• Used to prove statements false, or algorithms either in-correct or non-optimal
• Identify a case for which something is NOT true
• If the proof seems hard or tricky, sometimes a counterexample works
• Sometimes a counterexample is just easy to see, and can shortcut a proof
• If a counterexample is hard to nd, a proof might be easier
28. Proving Correctness
-Proof by Counterexample
• Greedy Algorithms
Denition 11.2 (Greedy Algorithm) An algorithm that selects the best choice at each step,
instead of considering all sequences of steps that may lead to an optimal solution.
• It's usually straight-forward to nd a greedy algorithm that is feasible, but hard
to nd a greedyalgorithm that is optimal
• Either prove the solution optimal, or nd a counterexample such that the
algorithm yields a non-optimal solution
• An algorithm can be greedy even if it doesn't produce an optimal solution
29. Proving Correctness
-Proof by Counterexample
• Interval Scheduling: Simple Greedy Algorithm
Input: Array A of requests q : fstart; finishg such that (q1 = fs1; f1g; q2 = fs2; f2g; : : : qn = fsn; fng)
Output: S is the set of talks scheduled
Schedule(A):
1 Sort talks by start time; reorder so that s1 s2 : : : sn
2 S = ;
3 for j = 1 to n:
4 if qj is compatible with S:
5 // The current request doesn't conict with any others we've chosen
6 S = S U qj // Add it to the set of scheduled
7 return S
30. Proving Correctness
-Proof by Counterexample
Summary: Counterexamples
• Sometimes it's easy to provide a counterexample
• It's usually enough to provide a counterexample to prove something wrong or False
• In algorithms, particularly useful for proving heuristics or greedy algorithms wrong or non-
optimal
31. Proving Correctness
-Proof by Induction
Insert Algorithm
Insert(A, e)
1 Add(A, e) // Add e at the end of A
2 for i = A:length 1 to 1:
3 while A[i + 1] < A[i]:
4 A[i + 1] = A[i] // Move the larger one to the end
We want to prove that for any element e and any list A:
• 1. The resulting list after Insert(A; e) is sorted
• 2. The resulting list after Insert(A; e) contains all of the elements of A, plus element e.
32. Proving Correctness
-Proof by Induction
Proving:
• Let's say P(n) is dened for any list A and element e as:
• If sorted(A) and length(A) = n, then sorted(insert(A; l)) and elements(insert(A; e)) =
elements(A)U{e} sorted(A) indicates the list is sorted.
• We want to prove that P(n) holds for all n 0.
33. Proving Correctness
-Proof by Loop Invariant
• Built o proof by induction.
• Useful for algorithms that loop.
Formally: nd loop invariant, then prove:
1. Dene a Loop Invariant
2. Initialization
3. Maintenance
4. Termination
Informally:
1. Find p, a loop invariant
2. Show the base case for p
3. Use induction to show the rest.
34. Proving Correctness
• Proof by Loop Invariant Examples
• Proof by Loop Invariant Is...
• Invariant: something that is always true
• After nding a candidate loop invariant, we prove:
• 1. Initialization: How does the invariant get initialized?
• 2. Loop Maintenance: How does the invariant change at each pass through the loop?
• 3. Termination: Does the loop stop? When?
35. Proving Correctness
• Loop Invariant Proof Examples
• We have a few examples:
• Linear Search
• Insertion Sort
• Bubble Sort
• Merge Sort
• LinearSearch(A; v)
• 1 for j = 1 toA:length:
• 2 if A[j] == v:
• 3 return j
• 4 return NIL
36. Proving Correctness
• Loop Invariant . At the start of each iteration of the for loop on line 1, the subarray A[1 : j 1]
does not
• contain the value v
• Initialization Prior to the rst iteration, the array A[1 : j 1] is empty (j == 1). That (empty)
subarray
• does not contain the value v.
37. Proving Correctness
Maintenance Line 2 checks whether A[j] is the desired value (v). If it is, the algorithm will
return j, thereby
terminating and producing the correct behavior (the index of value v is returned, if v is
present). If
A[j] 6= v, then the loop invariant holds at the end of the loop (the subarray A[1 : j] does not
contain
the value v).
Termination The for loop on line 1 terminates when j > A:length (that is, n). Because each
iteration of
a for loop increments j by 1, then j = n + 1. The loop invariant states that the value is not
present
in the subarray of A[1 : j 1]. Substituting n + 1 for j, we have A[1 : n]. Therefore, the value is
not
present in the original array A and the algorithm returns NIL.
38. Proving Correctness
• Insertion Sort
InsertionSort(A)
1 for i = 1 to A:length
2 j = i
3 while j > 0 and A[j 1] > A[j]
4 Swap(A[j], A[j 1])
5 j = j 1
• Invariant A[0 : i 1] are sorted
• Initialization At the top of the rst loop, this is A[0 : 0], which is vacuously true.
• Maintenance An inner loop where we start from i and work our way down, swapping values until
we nd
• the location for a[i] in the sorted section of the data
• Termination And the end of the for loop, i = len(A). That means that the array A[0 : A:length 1]
is
• now sorted, which is the entire array.
39. • Bubble Sort: Outer Loop
BubbleSort(A)
1 for i = 1 to A.length - 1
2 for j = A.length to i + 1
3 if A[j] < A[j - 1]
4 Swap(A[j], A[j - 1]
• Invariant At the start of each iteration of the for loop on line 1, the subarray A[1 : i 1] is sorted
• Initialization Prior to the rst iteration, the array A[1 : i 1] is empty (i = 1). That (empty) subarray is
sorted by denition.
• Maintenance Given the guarantees of the inner loop, at the end of each iteration of the for loop at line 1,
the value at A[i] is the smallest value in the range A[i : A:range]. Since the values in A[i : i 1] were
sorted and were less than the value in A[i], the values in the range A[1 : i] are sorted.
• Termination The for loop at line 1 ends when i equals A:length 1. Based on the maintenance proof, this
means that all values in A[1 : A:length 1] are sorted and less than the value at A[length]. So, by
denition, the values in A[1 : A:length] are sorted.
