The document discusses limitations of algorithms and methods for establishing lower bounds on algorithmic complexity. It covers four main topics: [1] efficiency classes and lower bounds, [2] decision trees for deriving lower bounds, [3] adversary arguments and problem reduction techniques for lower bounds, and [4] classifications of problem complexity including P, NP, NP-complete, and exponential time problems.
This document discusses limitations of algorithms and lower bounds on algorithm efficiency. It begins by defining lower bounds as estimates of the minimum work needed to solve a problem. It then provides examples of problems and their known lower bounds, including sorting arrays and multiplying matrices. The document discusses methods for establishing lower bounds, such as trivial counting arguments, information theory using decision trees, adversary arguments, and problem reduction. It provides examples for each method.
This document discusses lower bounds and limitations of algorithms. It begins by defining lower bounds and providing examples of problems where tight lower bounds have been established, such as sorting requiring Ω(nlogn) comparisons. It then discusses methods for establishing lower bounds, including trivial bounds, decision trees, adversary arguments, and problem reduction. The document explores different classes of problems based on complexity, such as P, NP, and NP-complete problems. It concludes by examining approaches for tackling difficult combinatorial problems that are NP-hard, such as using exact algorithms, approximation algorithms, and local search heuristics.
The document discusses sorting algorithms. It defines sorting as arranging a list of records in a certain order based on their keys. Some key points made:
- Sorting is important as it enables efficient searching and other tasks. Common sorting algorithms include selection sort, insertion sort, mergesort, quicksort, and heapsort.
- The complexity of sorting in general is Θ(n log n) but some special cases allow linear time sorting. Internal sorting happens in memory while external sorting handles data too large for memory.
- Applications of sorting include searching, finding closest pairs of numbers, checking for duplicates, and calculating frequency distributions. Sorting also enables efficient algorithms for computing medians, convex hulls, and
This document discusses dynamic programming and greedy algorithms. It begins by defining dynamic programming as a technique for solving problems with overlapping subproblems. Examples provided include computing the Fibonacci numbers and binomial coefficients. Greedy algorithms are introduced as constructing solutions piece by piece through locally optimal choices. Applications discussed are the change-making problem, minimum spanning trees using Prim's and Kruskal's algorithms, and single-source shortest paths. Floyd's algorithm for all pairs shortest paths and optimal binary search trees are also summarized.
The document provides an introduction to the analysis of algorithms. It discusses key concepts like the definition of an algorithm, properties of algorithms, common computational problems, and basic issues related to algorithms. It also covers algorithm design strategies, fundamental data structures, and the fundamentals of analyzing algorithm efficiency. Examples of algorithms for computing the greatest common divisor and checking for prime numbers are provided to illustrate algorithm design and analysis.
This document provides an overview of brute force and divide-and-conquer algorithms. It discusses various brute force algorithms like computing an, string matching, closest pair problem, convex hull problems, and exhaustive search algorithms like the traveling salesman problem and knapsack problem. It also analyzes the time efficiency of these brute force algorithms. The document then discusses the divide-and-conquer approach and provides examples like merge sort, quicksort, and matrix multiplication. It provides pseudocode and analysis for mergesort. In summary, the document covers brute force and divide-and-conquer techniques for solving algorithmic problems.
This document provides an overview of the CS303 Computer Algorithms course taught by Dr. Yanxia Jia. It discusses the importance of algorithms, provides examples of classic algorithm problems like sorting and searching, and summarizes common algorithm design techniques and data structures used, including arrays, linked lists, stacks, queues, heaps, graphs and trees.
Algorithm Design and Complexity - Course 5Traian Rebedea
This document provides an overview of greedy algorithms and their use in solving optimization problems. It discusses key aspects of greedy algorithms including making locally optimal choices at each step, optimal substructures, and the greedy choice property. Two problems addressed in detail are the activity selection problem and building Huffman trees. The activity selection problem can be solved optimally using a greedy approach by always selecting the activity with the earliest finish time. Huffman trees provide data compression by assigning codes to characters based on frequency, with more common characters having shorter codes, and can be constructed greedily by repeatedly combining the two subtrees with lowest weight.
This document discusses limitations of algorithms and lower bounds on algorithm efficiency. It begins by defining lower bounds as estimates of the minimum work needed to solve a problem. It then provides examples of problems and their known lower bounds, including sorting arrays and multiplying matrices. The document discusses methods for establishing lower bounds, such as trivial counting arguments, information theory using decision trees, adversary arguments, and problem reduction. It provides examples for each method.
This document discusses lower bounds and limitations of algorithms. It begins by defining lower bounds and providing examples of problems where tight lower bounds have been established, such as sorting requiring Ω(nlogn) comparisons. It then discusses methods for establishing lower bounds, including trivial bounds, decision trees, adversary arguments, and problem reduction. The document explores different classes of problems based on complexity, such as P, NP, and NP-complete problems. It concludes by examining approaches for tackling difficult combinatorial problems that are NP-hard, such as using exact algorithms, approximation algorithms, and local search heuristics.
The document discusses sorting algorithms. It defines sorting as arranging a list of records in a certain order based on their keys. Some key points made:
- Sorting is important as it enables efficient searching and other tasks. Common sorting algorithms include selection sort, insertion sort, mergesort, quicksort, and heapsort.
- The complexity of sorting in general is Θ(n log n) but some special cases allow linear time sorting. Internal sorting happens in memory while external sorting handles data too large for memory.
- Applications of sorting include searching, finding closest pairs of numbers, checking for duplicates, and calculating frequency distributions. Sorting also enables efficient algorithms for computing medians, convex hulls, and
This document discusses dynamic programming and greedy algorithms. It begins by defining dynamic programming as a technique for solving problems with overlapping subproblems. Examples provided include computing the Fibonacci numbers and binomial coefficients. Greedy algorithms are introduced as constructing solutions piece by piece through locally optimal choices. Applications discussed are the change-making problem, minimum spanning trees using Prim's and Kruskal's algorithms, and single-source shortest paths. Floyd's algorithm for all pairs shortest paths and optimal binary search trees are also summarized.
The document provides an introduction to the analysis of algorithms. It discusses key concepts like the definition of an algorithm, properties of algorithms, common computational problems, and basic issues related to algorithms. It also covers algorithm design strategies, fundamental data structures, and the fundamentals of analyzing algorithm efficiency. Examples of algorithms for computing the greatest common divisor and checking for prime numbers are provided to illustrate algorithm design and analysis.
This document provides an overview of brute force and divide-and-conquer algorithms. It discusses various brute force algorithms like computing an, string matching, closest pair problem, convex hull problems, and exhaustive search algorithms like the traveling salesman problem and knapsack problem. It also analyzes the time efficiency of these brute force algorithms. The document then discusses the divide-and-conquer approach and provides examples like merge sort, quicksort, and matrix multiplication. It provides pseudocode and analysis for mergesort. In summary, the document covers brute force and divide-and-conquer techniques for solving algorithmic problems.
This document provides an overview of the CS303 Computer Algorithms course taught by Dr. Yanxia Jia. It discusses the importance of algorithms, provides examples of classic algorithm problems like sorting and searching, and summarizes common algorithm design techniques and data structures used, including arrays, linked lists, stacks, queues, heaps, graphs and trees.
Algorithm Design and Complexity - Course 5Traian Rebedea
This document provides an overview of greedy algorithms and their use in solving optimization problems. It discusses key aspects of greedy algorithms including making locally optimal choices at each step, optimal substructures, and the greedy choice property. Two problems addressed in detail are the activity selection problem and building Huffman trees. The activity selection problem can be solved optimally using a greedy approach by always selecting the activity with the earliest finish time. Huffman trees provide data compression by assigning codes to characters based on frequency, with more common characters having shorter codes, and can be constructed greedily by repeatedly combining the two subtrees with lowest weight.
The document discusses the framework for analyzing the efficiency of algorithms by measuring how the running time and space requirements grow as the input size increases, focusing on determining the order of growth of the number of basic operations using asymptotic notation such as O(), Ω(), and Θ() to classify algorithms based on their worst-case, best-case, and average-case time complexities.
Algorithm Design and Complexity - Course 4 - Heaps and Dynamic ProgammingTraian Rebedea
Course 4 for the Algorithm Design and Complexity course at the Faculty of Engineering in Foreign Languages - Politehnica University of Bucharest, Romania
Linear Discriminant Analysis and Its Generalization일상 온
The brief introduction to the linear discriminant analysis and some extended methods. Much of the materials are taken from The Elements of Statistical Learning by Hastie et al. (2008).
Divide and Conquer Algorithms - D&C forms a distinct algorithm design technique in computer science, wherein a problem is solved by repeatedly invoking the algorithm on smaller occurrences of the same problem. Binary search, merge sort, Euclid's algorithm can all be formulated as examples of divide and conquer algorithms. Strassen's algorithm and Nearest Neighbor algorithm are two other examples.
Algorithm Design and Complexity - Course 1&2Traian Rebedea
Courses 1 & 2 for the Algorithm Design and Complexity course at the Faculty of Engineering in Foreign Languages - Politehnica University of Bucharest, Romania
Algorithm Design and Complexity - Course 3Traian Rebedea
The document provides an overview of recursive algorithms and complexity analysis. It discusses recursive algorithms, divide and conquer design technique, and several examples of recursive algorithms including Towers of Hanoi, Merge Sort, and Quick Sort. For recursive algorithms, it explains how to analyze their running time using recurrence relations. It then covers four methods for solving recurrence relations: iteration, recursion trees, substitution method, and master theorem. The substitution method and master theorem are described as the most rigorous mathematical approaches.
Iterative improvement is an algorithm design technique for solving optimization problems. It starts with a feasible solution and repeatedly makes small changes to the current solution to find a solution with a better objective function value until no further improvements can be found. The simplex method, Ford-Fulkerson algorithm, and local search heuristics are examples that use this technique. The maximum flow problem can be solved using the iterative Ford-Fulkerson algorithm, which finds augmenting paths in a flow network to incrementally increase the flow from the source to the sink until no more augmenting paths exist.
The document proposes a new method called Spectral Regression Discriminant Analysis (SRDA) to address the computational challenges of Linear Discriminant Analysis (LDA) on large, high-dimensional datasets. SRDA combines spectral graph analysis and regression to reduce the time complexity of LDA from quadratic to linear. It works by using the eigenvectors of the within-class scatter matrix to define a regression problem, the solution of which provides the projection vectors that maximize class separability. Experiments on four datasets show SRDA has comparable classification accuracy to LDA but can scale to much larger problems.
This document provides an overview of key concepts related to random processes, including:
1) Random processes can be described statistically using properties like the mean, variance, autocorrelation, and spectral density.
2) Stationary and ergodic processes have statistical properties that do not vary over time or between samples.
3) Autocorrelation describes the dependency of a process on past values, while spectral density describes the frequency content.
4) Cross-correlation and cross-spectral density characterize the relationship between two random processes.
5) The response of a linear system to a random input can be related to the input's spectral density using the system's transfer function.
In this presentation we describe the formulation of the HMM model as consisting of states that are hidden that generate the observables. We introduce the 3 basic problems: Finding the probability of a sequence of observation given the model, the decoding problem of finding the hidden states given the observations and the model and the training problem of determining the model parameters that generate the given observations. We discuss the Forward, Backward, Viterbi and Forward-Backward algorithms.
The document discusses order statistics and medians. It defines order statistics as the ith smallest element in a data set and notes that the median is the middle element when the data set size is odd or the average of the two middle elements when the size is even. It then describes algorithms for finding the minimum, maximum, second smallest element, and any order statistic in expected linear time using a randomized selection algorithm. Finally, it provides an overview of generic programming in C++ using templates for functions and classes.
This document discusses analyzing the time efficiency of recursive algorithms. It provides a general 5-step plan: 1) choose a parameter for input size, 2) identify the basic operation, 3) check if operation count varies, 4) set up a recurrence relation, 5) solve the relation to determine growth order. It then gives two examples - computing factorial recursively and solving the Tower of Hanoi puzzle recursively - to demonstrate applying the plan. The document also briefly discusses algorithm visualization using static or dynamic images to convey information about an algorithm's operations and performance.
Constraint satisfaction problems (CSPs) define states as assignments of variables to values from their domains, with constraints specifying allowable combinations. Backtracking search assigns one variable at a time using depth-first search. Improved heuristics like most-constrained variable selection and least-constraining value choice help. Forward checking and constraint propagation techniques like arc consistency detect inconsistencies earlier than backtracking alone. Local search methods like min-conflicts hill-climbing can also solve CSPs by allowing constraint violations and minimizing them.
This document provides an overview of the Data Structures I course. It outlines the course objectives of becoming familiar with problem solving, algorithms, data structures, and tracing algorithms. The course will cover fundamentals of data structures and algorithms, static and dynamic data structures, searching and sorting algorithms, recursion, abstract data types, stacks, queues and trees. Exams, labs, participation and quizzes will be used for grading. Pseudo-code is introduced as a way to express algorithms independent of a programming language. Examples of algorithms for determining even/odd numbers and computing weekly wages are provided.
Unit 1: Fundamentals of the Analysis of Algorithmic Efficiency, Units for Measuring Running Time, PROPERTIES OF AN ALGORITHM, Growth of Functions, Algorithm - Analysis, Asymptotic Notations, Recurrence Relation and problems
This is the second lecture in the CS 6212 class. Covers asymptotic notation and data structures. Also outlines the coming lectures wherein we will study the various algorithm design techniques.
Discusses the concept of Language Models in Natural Language Processing. The n-gram models, markov chains are discussed. Smoothing techniques such as add-1 smoothing, interpolation and discounting methods are addressed.
1) The lower bound theory is used to determine if an algorithm is the most efficient possible by comparing its time complexity f(n) to a lower bound complexity g(n).
2) For comparison-based sorting algorithms, the decision tree model shows that any sorting algorithm requires at least O(nlogn) comparisons in the worst case.
3) This lower bound of O(nlogn) time complexity for comparison-based sorting algorithms means that no algorithm can sort faster than nlogn time using only element comparisons.
This chapter discusses limitations on algorithmic power and methods for establishing lower bounds on algorithms. It introduces lower bounds as estimates of the minimum amount of work needed to solve a problem. Several methods are presented for establishing lower bounds, including trivial lower bounds based on input/output sizes, decision trees to model comparisons, adversary arguments, and reducing one problem to another with a known lower bound. Examples are given for sorting, searching, and matrix multiplication.
This document discusses lower bounds and limitations of algorithms. It begins by defining lower bounds and providing examples of problems where tight lower bounds have been established, such as sorting requiring Ω(nlogn) comparisons. It then discusses methods for establishing lower bounds, including trivial bounds, decision trees, adversary arguments, and problem reduction. The document covers several examples to illustrate these techniques. It also discusses the complexity classes P, NP, and NP-complete problems. Finally, it discusses approaches for tackling difficult combinatorial problems that are NP-hard, including exact and approximation algorithms.
The document discusses the framework for analyzing the efficiency of algorithms by measuring how the running time and space requirements grow as the input size increases, focusing on determining the order of growth of the number of basic operations using asymptotic notation such as O(), Ω(), and Θ() to classify algorithms based on their worst-case, best-case, and average-case time complexities.
Algorithm Design and Complexity - Course 4 - Heaps and Dynamic ProgammingTraian Rebedea
Course 4 for the Algorithm Design and Complexity course at the Faculty of Engineering in Foreign Languages - Politehnica University of Bucharest, Romania
Linear Discriminant Analysis and Its Generalization일상 온
The brief introduction to the linear discriminant analysis and some extended methods. Much of the materials are taken from The Elements of Statistical Learning by Hastie et al. (2008).
Divide and Conquer Algorithms - D&C forms a distinct algorithm design technique in computer science, wherein a problem is solved by repeatedly invoking the algorithm on smaller occurrences of the same problem. Binary search, merge sort, Euclid's algorithm can all be formulated as examples of divide and conquer algorithms. Strassen's algorithm and Nearest Neighbor algorithm are two other examples.
Algorithm Design and Complexity - Course 1&2Traian Rebedea
Courses 1 & 2 for the Algorithm Design and Complexity course at the Faculty of Engineering in Foreign Languages - Politehnica University of Bucharest, Romania
Algorithm Design and Complexity - Course 3Traian Rebedea
The document provides an overview of recursive algorithms and complexity analysis. It discusses recursive algorithms, divide and conquer design technique, and several examples of recursive algorithms including Towers of Hanoi, Merge Sort, and Quick Sort. For recursive algorithms, it explains how to analyze their running time using recurrence relations. It then covers four methods for solving recurrence relations: iteration, recursion trees, substitution method, and master theorem. The substitution method and master theorem are described as the most rigorous mathematical approaches.
Iterative improvement is an algorithm design technique for solving optimization problems. It starts with a feasible solution and repeatedly makes small changes to the current solution to find a solution with a better objective function value until no further improvements can be found. The simplex method, Ford-Fulkerson algorithm, and local search heuristics are examples that use this technique. The maximum flow problem can be solved using the iterative Ford-Fulkerson algorithm, which finds augmenting paths in a flow network to incrementally increase the flow from the source to the sink until no more augmenting paths exist.
The document proposes a new method called Spectral Regression Discriminant Analysis (SRDA) to address the computational challenges of Linear Discriminant Analysis (LDA) on large, high-dimensional datasets. SRDA combines spectral graph analysis and regression to reduce the time complexity of LDA from quadratic to linear. It works by using the eigenvectors of the within-class scatter matrix to define a regression problem, the solution of which provides the projection vectors that maximize class separability. Experiments on four datasets show SRDA has comparable classification accuracy to LDA but can scale to much larger problems.
This document provides an overview of key concepts related to random processes, including:
1) Random processes can be described statistically using properties like the mean, variance, autocorrelation, and spectral density.
2) Stationary and ergodic processes have statistical properties that do not vary over time or between samples.
3) Autocorrelation describes the dependency of a process on past values, while spectral density describes the frequency content.
4) Cross-correlation and cross-spectral density characterize the relationship between two random processes.
5) The response of a linear system to a random input can be related to the input's spectral density using the system's transfer function.
In this presentation we describe the formulation of the HMM model as consisting of states that are hidden that generate the observables. We introduce the 3 basic problems: Finding the probability of a sequence of observation given the model, the decoding problem of finding the hidden states given the observations and the model and the training problem of determining the model parameters that generate the given observations. We discuss the Forward, Backward, Viterbi and Forward-Backward algorithms.
The document discusses order statistics and medians. It defines order statistics as the ith smallest element in a data set and notes that the median is the middle element when the data set size is odd or the average of the two middle elements when the size is even. It then describes algorithms for finding the minimum, maximum, second smallest element, and any order statistic in expected linear time using a randomized selection algorithm. Finally, it provides an overview of generic programming in C++ using templates for functions and classes.
This document discusses analyzing the time efficiency of recursive algorithms. It provides a general 5-step plan: 1) choose a parameter for input size, 2) identify the basic operation, 3) check if operation count varies, 4) set up a recurrence relation, 5) solve the relation to determine growth order. It then gives two examples - computing factorial recursively and solving the Tower of Hanoi puzzle recursively - to demonstrate applying the plan. The document also briefly discusses algorithm visualization using static or dynamic images to convey information about an algorithm's operations and performance.
Constraint satisfaction problems (CSPs) define states as assignments of variables to values from their domains, with constraints specifying allowable combinations. Backtracking search assigns one variable at a time using depth-first search. Improved heuristics like most-constrained variable selection and least-constraining value choice help. Forward checking and constraint propagation techniques like arc consistency detect inconsistencies earlier than backtracking alone. Local search methods like min-conflicts hill-climbing can also solve CSPs by allowing constraint violations and minimizing them.
This document provides an overview of the Data Structures I course. It outlines the course objectives of becoming familiar with problem solving, algorithms, data structures, and tracing algorithms. The course will cover fundamentals of data structures and algorithms, static and dynamic data structures, searching and sorting algorithms, recursion, abstract data types, stacks, queues and trees. Exams, labs, participation and quizzes will be used for grading. Pseudo-code is introduced as a way to express algorithms independent of a programming language. Examples of algorithms for determining even/odd numbers and computing weekly wages are provided.
Unit 1: Fundamentals of the Analysis of Algorithmic Efficiency, Units for Measuring Running Time, PROPERTIES OF AN ALGORITHM, Growth of Functions, Algorithm - Analysis, Asymptotic Notations, Recurrence Relation and problems
This is the second lecture in the CS 6212 class. Covers asymptotic notation and data structures. Also outlines the coming lectures wherein we will study the various algorithm design techniques.
Discusses the concept of Language Models in Natural Language Processing. The n-gram models, markov chains are discussed. Smoothing techniques such as add-1 smoothing, interpolation and discounting methods are addressed.
1) The lower bound theory is used to determine if an algorithm is the most efficient possible by comparing its time complexity f(n) to a lower bound complexity g(n).
2) For comparison-based sorting algorithms, the decision tree model shows that any sorting algorithm requires at least O(nlogn) comparisons in the worst case.
3) This lower bound of O(nlogn) time complexity for comparison-based sorting algorithms means that no algorithm can sort faster than nlogn time using only element comparisons.
This chapter discusses limitations on algorithmic power and methods for establishing lower bounds on algorithms. It introduces lower bounds as estimates of the minimum amount of work needed to solve a problem. Several methods are presented for establishing lower bounds, including trivial lower bounds based on input/output sizes, decision trees to model comparisons, adversary arguments, and reducing one problem to another with a known lower bound. Examples are given for sorting, searching, and matrix multiplication.
This document discusses lower bounds and limitations of algorithms. It begins by defining lower bounds and providing examples of problems where tight lower bounds have been established, such as sorting requiring Ω(nlogn) comparisons. It then discusses methods for establishing lower bounds, including trivial bounds, decision trees, adversary arguments, and problem reduction. The document covers several examples to illustrate these techniques. It also discusses the complexity classes P, NP, and NP-complete problems. Finally, it discusses approaches for tackling difficult combinatorial problems that are NP-hard, including exact and approximation algorithms.
Undecidable Problems - COPING WITH THE LIMITATIONS OF ALGORITHM POWERmuthukrishnavinayaga
This document discusses algorithms and their analysis. It begins by defining key properties of algorithms like their lower, upper, and tight bounds. It then discusses different techniques for determining algorithm lower bounds such as trivial, information theoretical, adversary, and reduction arguments. Decision trees are presented as a model for representing algorithms that use comparisons. Lower bounds proofs are given for sorting and searching algorithms. The document also covers polynomial time versus non-polynomial time problems, as well as NP-complete problems. Specific algorithms are analyzed like knapsack, traveling salesman, and approximation algorithms.
Undecidable Problems and Approximation AlgorithmsMuthu Vinayagam
The document discusses algorithm limitations and approximation algorithms. It begins by explaining that some problems have no algorithms or cannot be solved in polynomial time. It then discusses different algorithm bounds and how to derive lower bounds through techniques like decision trees. The document also covers NP-complete problems, approximation algorithms for problems like traveling salesman, and techniques like branch and bound. It provides examples of approximation algorithms that provide near-optimal solutions when an optimal solution is impossible or inefficient to find.
This document provides an overview of key algorithm analysis concepts including:
- Common algorithmic techniques like divide-and-conquer, dynamic programming, and greedy algorithms.
- Data structures like heaps, graphs, and trees.
- Analyzing the time efficiency of recursive and non-recursive algorithms using orders of growth, recurrence relations, and the master's theorem.
- Examples of specific algorithms that use techniques like divide-and-conquer, decrease-and-conquer, dynamic programming, and greedy strategies.
- Complexity classes like P, NP, and NP-complete problems.
This document provides an introduction to algorithms and data structures. It begins by defining an algorithm and discussing their importance and characteristics. Some examples of algorithms are presented, including Euclid's algorithm for finding the greatest common divisor and the sieve of Eratosthenes for generating primes. The document then discusses analyzing algorithms to evaluate efficiency and optimality. Important problem types like sorting, searching, and graph problems are introduced. Finally, fundamental data structures like arrays, linked lists, stacks, queues, and graphs are overviewed.
There are three main limitations of algorithms:
1) Some problems cannot be solved by any algorithm.
2) Some problems can be solved algorithmically but not in polynomial time.
3) Some problems can be solved in polynomial time but existing algorithms are not efficient.
Lower bounds help establish the minimum efficiency of algorithms. There are four main methods to determine lower bounds: trivial counting arguments, information theory, adversary arguments, and problem reduction. Decision trees can also help analyze comparison-based algorithms and establish lower bounds.
The document discusses brute force and exhaustive search approaches to solving problems. It provides examples of how brute force can be applied to sorting, searching, and string matching problems. Specifically, it describes selection sort and bubble sort as brute force sorting algorithms. For searching, it explains sequential search and brute force string matching. It also discusses using brute force to solve the closest pair, convex hull, traveling salesman, knapsack, and assignment problems, noting that brute force leads to inefficient exponential time algorithms for TSP and knapsack.
The document discusses various sorting algorithms and their time complexities, including:
- Insertion sort runs in O(n^2) time in the worst case.
- Merge sort and heap sort run in O(nlogn) time in the worst case.
- Any comparison-based sorting algorithm requires Ω(nlogn) time.
- Counting sort and radix sort can run in O(n) time by avoiding comparisons, but have additional requirements on the key range.
1. Counting sort and radix sort can sort in linear time O(n) by exploiting properties of the input rather than just comparisons. Counting sort assumes integers as input and radix sort assumes digitized numbers.
2. Bucket sort also runs in linear time if inputs are uniformly distributed between 0 and 1. It divides the range into buckets and distributes inputs into the corresponding buckets which are then sorted.
3. The comparison-based lower bound of Ω(nlogn) does not apply to these algorithms because they do not rely solely on comparisons. Counting sort counts occurrences rather than comparing, and radix/bucket sort distribute into buckets based on digits/positions rather than comparisons.
This document discusses algorithms and their analysis. It defines an algorithm as a finite sequence of unambiguous instructions that terminate in a finite amount of time. It discusses areas of study like algorithm design techniques, analysis of time and space complexity, testing and validation. Common algorithm complexities like constant, logarithmic, linear, quadratic and exponential are explained. Performance analysis techniques like asymptotic analysis and amortized analysis using aggregate analysis, accounting method and potential method are also summarized.
The document discusses algorithms and their analysis. It defines an algorithm as a step-by-step procedure for solving a problem and analyzes the time complexity of various algorithms. Key points made include:
1) Algorithms are analyzed based on how many times their basic operation is performed as a function of input size n.
2) Common time complexities include O(n) for sequential search, O(n^3) for matrix multiplication, and O(log n) for binary search.
3) Recursive algorithms like fibonacci are inefficient and iterative versions improve performance by storing previously computed values.
This document outlines a course on data structures and algorithms. It includes the following topics: asymptotic and algorithm analysis, complexity analysis, abstract lists and implementations, arrays, linked lists, stacks, queues, trees, graphs, sorting algorithms, minimum spanning trees, hashing, and more. The course objectives are to enable students to understand various ways to organize data, understand algorithms to manipulate data, use analyses to compare data structures and algorithms, and select relevant structures and algorithms for problems. The document also lists reference books and provides outlines on defining algorithms, analyzing time/space complexity, and asymptotic notations.
The document discusses the divide and conquer algorithmic approach. It defines divide and conquer as dividing a problem into smaller subproblems, solving the subproblems independently, and then combining the solutions to solve the original problem. The key steps are listed as divide, conquer, and combine. Examples provided of problems solved using divide and conquer include binary search, min-max problems, merge sort, and Karatsuba's multiplication algorithm. Advantages include solving difficult problems efficiently while disadvantages include potential slowdowns from recursion and redundant solving of subproblems.
This document provides an overview and introduction to the CPEN 204 Data Structures course. It outlines the course grading breakdown and covers topics like data structures, algorithms, abstract data types, analysis using Big O notation, and examples of linear and non-linear data structures. Common data structures like arrays, linked lists, trees, queues and stacks that should already be known are also listed.
The document discusses algorithms and data structures. It defines an algorithm as a well-defined procedure that takes input and produces output. Algorithms are used for calculation, data processing, and automated reasoning. The document discusses different ways of describing algorithms including natural language, flowcharts, and pseudo code. It also discusses analyzing the time complexity of algorithms using asymptotic notation such as Big-O, Omega, and Theta notation. Recursive algorithms and solving recurrences are also covered.
The document discusses algorithms and their role in computing. It defines an algorithm as a well-defined computational procedure that takes input and produces output. Algorithms must be correct, producing the right output for each input, and efficient. It presents examples of computational problems and their algorithms. The document also discusses pseudo-code for writing algorithms and the divide-and-conquer technique for algorithm design, which breaks problems into smaller subproblems.
The document summarizes several linear sorting algorithms, including bucket sort, counting sort, general bucket sort, and radix sort. Counting sort runs in O(n+k) time and O(k) space, where k is the range of integer keys, and is stable. Radix sort uses a stable sorting algorithm like counting sort to sort based on each digit of d-digit numbers, resulting in O(d(n+k)) time for sorting n numbers with d digits in the range [1,k].
Euclid's algorithm is an efficient method for computing the greatest common divisor (GCD) of two numbers. It works by repeatedly finding the remainder of dividing the larger number by the smaller number, and then setting the larger number equal to the smaller number and the smaller number equal to the remainder, until the smaller number is zero. The last non-zero remainder is the GCD. The time complexity of Euclid's algorithm is O(log n) where n is the smaller of the two input numbers. Algorithm analysis techniques such as worst-case, best-case, average-case analysis and asymptotic notations can be used to formally analyze the efficiency of algorithms.
Euclid's algorithm is an efficient method for computing the greatest common divisor (GCD) of two numbers. It works by repeatedly finding the remainder of dividing the larger number by the smaller number, and then setting the larger number equal to the smaller number and the smaller number equal to the remainder, until the smaller number is zero. The last non-zero remainder is the GCD. The time complexity of Euclid's algorithm is O(log n) where n is the smaller of the two input numbers. Algorithm analysis techniques such as worst-case, best-case, average-case analysis and asymptotic notations can be used to formally analyze the efficiency of algorithms.
1. Limitations of Algorithm Power
Objectives
We now move into the third and final major theme for this course.
1. Tools for analyzing algorithms.
2. Design strategies for designing algorithms.
3. Identifying and coping with the limitations of algorithms.
Efficiency of an algorithm
• By establishing the asymptotic efficiency class
• The efficiency class for selection sort (quadratic) is lower. Does this mean that selection
sort is a “better” algorithm?
– Like comparing “apples” to “oranges”
• By analyzing how efficient a particular algorithm is compared to other algorithms for the
same problem
– It is desirable to know the best possible efficiency any algorithm solving this
problem may have – establishing a lower bound
Lower Bounds
Lower bound: an estimate on a minimum amount of work needed to solve a given problem
Examples:
• number of comparisons needed to find the largest element in a set of n numbers
• number of comparisons needed to sort an array of size n
• number of comparisons necessary for searching in a sorted array
• number of multiplications needed to multiply two n-by-n matrices
Lower bound can be
– an exact count
– an efficiency class (Ω)
• Tight lower bound: there exists an algorithm with the same efficiency as the lower bound
Problem Lower bound Tightness
sorting Ω(nlog n) yes
searching in a sorted array Ω(log n) yes
element uniqueness Ω(nlog n) yes
n-digit integer multiplication Ω(n) unknown
multiplication of n-by-n matrices Ω(n2) unknown
Methods for Establishing Lower Bounds
• trivial lower bounds
• information-theoretic arguments (decision trees)
2. • adversary arguments
• problem reduction
Trivial Lower Bounds
Trivial lower bounds: based on counting the number of items that must be processed in input
and generated as output
Examples
• finding max element
• polynomial evaluation
• sorting
• element uniqueness
• computing the product of two n-by-n matrices
Conclusions
• may and may not be useful
• be careful in deciding how many elements must be processed
Decision Trees
Decision tree — a convenient model of algorithms involving Comparisons in which: (i)
internal nodes represent comparisons (ii) leaves represent outcomes
Decision tree for 3-element insertion sort
Deriving a Lower Bound from Decision Trees
• How does such a tree help us find lower bounds?
– There must be at least one leaf for each correct output.
– The tree must be tall enough to have that many leaves.
• In a binary tree with l leaves and height h,
3. h ≥ log2 l
Decision Tree and Sorting Algorithms
Decision Tree and Sorting Algorithms
• Number of leaves (outcomes) ≥ n!
• Height of binary tree with n! leaves ≥ log2n!
• Minimum number of comparisons in the worst case ≥ log2n! for any comparison-based
sorting algorithm
• log2n! ≈ n log2n
• This lower bound is tight (e.g. mergesort)
Decision Tree and Searching a Sorted Array
Decision Tree and Searching a Sorted Array
4. • Number of leaves (outcomes) = n + n+1 = 2n+1
• Height of ternary tree with 2n+1 leaves ≥ log3 (2n+1)
• This lower bound is NOT tight (the number of worst-case comparisons for binary search
is log2 (n+1), and log3 (2n+1) ≤ log2 (n+1))
• Can we find a better lower bound or find an algorithm with better efficiency than binary
search?
Decision Tree and Searching a Sorted Array
Decision Tree and Searching a Sorted Array
• Consider using a binary tree where internal nodes also serve as successful searches and
leaves only represent unsuccessful searches
• Number of leaves (outcomes) = n + 1
• Height of binary tree with n+1 leaves ≥ log2 (n+1)
• This lower bound is tight
Decision-tree example
5. Decision-tree model
A decision tree can model the execution of any comparison sort:
• One tree for each input size n.
• View the algorithm as splitting whenever it compares two elements.
• The tree contains the comparisons along all possible instruction traces.
• The running time of the algorithm = the length of the path taken.
• Worst-case running time = height of tree.
Any comparison sort can be turned into a Decision tree
6.
7. Decision Tree Model
• In the insertion sort example, the decision tree reveals all possible key comparison
sequences for 3 distinct numbers.
• There are exactly 3!=6 possible output sequences.
• Different comparison sorts should generate different decision trees.
• It should be clear that, in theory, we should be able to draw a decision tree for ANY
comparison sort algorithm.
• Given a particular input sequence, the path from root to the leaf path traces a particular
key comparison sequence performed by that comparison sort.
- The length of that path represented the number of key comparisons performed by
the sorting algorithm.
• When we come to a leaf, the sorting algorithm has determined the sorted order.
• Notice that a correct sorting algorithm should be able to sort EVERY possible output
sorted order.
• Since, there are n! possible sorted order, there are n! leaves in the decision tree.
• Given a decision tree, the height of the tree represent the longest length of a root to leaf
path.
• It follows the height of the decision tree represents the largest number of key
comparisons, which is the worst-case running time of the sorting algorithm.
“Any comparison based sorting algorithm takes (n logn) to sort a list of n distinct
elements in the worst-case.”
– any comparison sort ← model by a decision tree
– worst-case running time ← the height of decision tree
“Any comparison based sorting algorithm takes (n logn) to sort a list of n distinct elements
in the worst-case.”
• We want to find a lower bound ( ) of the height of a binary tree that has n! Leaves.
◊ What is the minimum height of a binary tree that has n! leaves?
8. • The binary tree must be a complete tree (recall the definition of complete tree).
• Hence the minimum (lower bound) height is θ(log2(n!)).
• log2(n!)
= log2(n) + log2(n-1) + …+ log2(n/2)+….
≥ n/2 log2(n/2) = n/2 log2(n) – n/2
So, log2(n!) = (n logn).
• It follows the height of a binary tree which has n! leaves is at least (n logn) ◊ worst-
case running time is at least (n logn)
• Putting everything together, we have
“Any comparison based sorting algorithm takes (n logn) to sort a list of n distinct
elements in the worst-case.”
Adversary Arguments
Adversary argument: a method of proving a lower bound by playing role of adversary that makes
algorithm work the hardest by adjusting input
Example: “Guessing” a number between 1 and n with yes/no questions
Adversary: Puts the number in a larger of the two subsets generated by last question
Lower Bounds by Problem Reduction
Idea: If problem P is at least as hard as problem Q, then a lower bound for Q is also a lower
bound for P.
Hence, find problem Q with a known lower bound that can be reduced to problem P in question.
Example: Euclidean MST problem
• Given a set of n points in the plane, construct a tree with minimum total length that
connects the given points. (considered as problem P)
• To get a lower bound for this problem, reduce the element uniqueness problem to it.
(considered as problem Q)
• If an algorithm faster than n log n exists for Euclidean MST, then one exists for element
uniqueness also. Aha! A contradiction! Therefore, any algorithm for Euclidean MST
must take Ω(n log n) time.
Classifying Problem Complexity
Is the problem tractable, i.e., is there a polynomial-time (O(p(n)) algorithm that solves it?
Possible answers:
• yes (give examples)
• no
– because it’s been proved that no algorithm exists at all
(e.g., Turing’s halting problem)
– because it’s been proved that any algorithm for the problem takes exponential
time
• unknown
9. Problem Types: Optimization and Decision
• Optimization problem: find a solution that maximizes or minimizes some objective
function
• Decision problem: answer yes/no to a question
Many problems have decision and optimization versions.
E.g.: traveling salesman problem
• optimization: find Hamiltonian cycle of minimum length
• decision: find Hamiltonian cycle of length ≤ m
Decision problems are more convenient for formal investigation of their complexity.
Class P
P: the class of decision problems that are solvable in O(p(n)) time, where p(n) is a polynomial of
problem’s input size n
Examples:
• searching
• element uniqueness
• graph connectivity
• graph acyclicity
• primality testing (finally proved in 2002)
Class NP
NP (nondeterministic polynomial): class of decision problems whose proposed solutions can be
verified in polynomial time = solvable by a nondeterministic polynomial algorithm
A nondeterministic polynomial algorithm is an abstract two-stage procedure that:
• generates a random string purported to solve the problem
• checks whether this solution is correct in polynomial time
By definition, it solves the problem if it’s capable of generating and verifying a solution on one
of its tries
Why this definition?
• led to development of the rich theory called “computational complexity”
Example: CNF satisfiability
Problem: Is a boolean expression in its conjunctive normal form (CNF) satisfiable, i.e., are there
values of its variables that makes it true?
This problem is in NP. Nondeterministic algorithm:
• Guess truth assignment
• Substitute the values into the CNF formula to see if it evaluates to true
Example: (A | ¬B | ¬C) & (A | B) & (¬B | ¬D | E) & (¬D | ¬E)
Truth assignments:
ABCDE
0 0 0 0 0
. . .
1 1 1 1 1
Checking phase: O(n)
10. What problems are in NP?
• Hamiltonian circuit existence
• Partition problem: Is it possible to partition a set of n integers into two disjoint subsets
with the same sum?
• Decision versions of TSP, knapsack problem, graph coloring, and many other
combinatorial optimization problems. (Few exceptions include: MST, shortest paths)
• All the problems in P can also be solved in this manner (but no guessing is necessary), so
we have:
P ⊆ NP
• Big question: P = NP ?
P = NP ?
• One of the most important unsolved problems is computer science is whether or not
P=NP.
– If P=NP, then a ridiculous number of problems currently believed to be very
difficult will turn out have efficient algorithms.
– If P≠NP, then those problems definitely do not have polynomial time solutions.
• Most computer scientists suspect that P ≠ NP. These suspicions are based partly on the
idea of NP-completeness.
NP-Complete Problems
A decision problem D is NP-complete if it’s as hard as any problem in NP, i.e.,
• D is in NP
• every problem in NP is polynomial-time reducible to D
NP problems
NP -complete
problem
Other NP-complete problems obtained through polynomial-
time reductions from a known NP-complete problem
11. NP problems
known
NP -complete
problem
candidate
for NP -
completeness
Examples: TSP, knapsack, partition, graph-coloring and hundreds of other problems of
combinatorial nature
General Definitions: P, NP, NP-hard, NP-easy, and NP-complete
• Problems
- Decision problems (yes/no)
- Optimization problems (solution with best score)
• P
- Decision problems (decision problems) that can be solved in polynomial time
- can be solved “efficiently”
• NP
- Decision problems whose “YES” answer can be verified in polynomial
time, if we already have the proof (or witness)
• co-NP
- Decision problems whose “NO” answer can be verified in polynomial time,
if we already have the proof (or witness)
• e.g. The satisfiability problem (SAT)
- Given a boolean formula
(x1 ∨ x2 ∨ x3 ∨ x4 ) ∧ (x5 ∨ x6 ∨ x7 ) ∧ x8 ∧ x9
is it possible to assign the input x1...x9, so that the formula evaluates to TRUE?
- If the answer is YES with a proof (i.e. an assignment of input value), then we can check the
proof in polynomial time (SAT is in NP)
- We may not be able to check the NO answer in polynomial time (Nobody really knows.)
• NP-hard
- A problem is NP-hard iff an polynomial-time algorithm for it implies a polynomial-
time algorithm for every problem in NP
- NP-hard problems are at least as hard as NP problems
• NP-complete
- A problem is NP-complete if it is NP-hard, and is an element of NP (NP-easy)
• Relationship between decision problems and optimization problems
12. - every optimization problem has a corresponding decision problem
- optimization: minimize x, subject to constraints
- yes/no: is there a solution, such that x is less than c?
- an optimization problem is NP-hard (NP-complete)
if its corresponding decision problem is NP-hard (NP-complete)
• How to know another problem, A, is NP-complete?
- To prove that A is NP-complete, reduce a known NP-complete problem to A
• Requirement for Reduction
- Polynomial time
- YES to A also implies YES to SAT, while
NO to A also implies No to SAT
Examples of NP-complete problems
Vertex cover
• Vertex cover
- given a graph G=(V,E), find the smallest number of vertexes that cover each edge
- Decision problem: is the graph has a vertex cover of size K?
• Independent set
- independent set: a set of vertices in the graph with no edges between each pair of nodes.
- given a graph G=(V,E), find the largest independent set
- reduction from vertex cover:
• Set cover
- given a universal set U, and several subsets S1,...Sn
- find the least number of subsets that contains each elements in the universal set
Polynomial (P) Problems
• Are solvable in polynomial time
• Are solvable in O(nk), where k is some constant.
• Most of the algorithms we have covered are in P
Nondeterministic Polynomial (NP) Problems
• This class of problems has solutions that are verifiable in polynomial time.
– Thus any problem in P is also NP, since we would be able to solve it in
polynomial time, we can also verify it in polynomial time
NP-Complete Problems
• Is an NP-Problem
• Is at least as difficult as an NP problem (is reducible to it)
• More formally, a decision problem C is NP-Complete if:
13. – C is in NP
– Any known NP-hard (or complete) problem ≤p C
– Thus a proof must show these two being satisfied
Exponential Time Algorithms
Examples
• Longest path problem: (similar to Shortest path problem, which requires polynomial
time) suspected to require exponential time, since there is no known polynomial
algorithm.
• Hamiltonian Cycle problem: Traverses all vertices exactly once and form a cycle.
Reduction
• P1 : is an unknown problem (easy/hard ?)
• P2 : is known to be difficult
If we can easily solve P2 using P1 as a subroutine then P1 is difficult
Must create the inputs for P1 in polynomial time.
* P1 is definitely difficult because you know you cannot solve P2 in polynomial time unless you
use a component that is also difficult (it cannot be the mapping since the mapping is known to be
polynomial)
Decision Problems
Represent problem as a decision with a boolean output
– Easier to solve when comparing to other problems
– Hence all problems are converted to decision problems.
P = {all decision problems that can be solved in polynomial time}
NP = {all decision problems where a solution is proposed, can be verified in polynomial time}
NP-complete: the subset of NP which are the “hardest problems”
14. Alternative Representation
• Every element p in P1 can map to an element q in P2 such that p is true (decision
problem) if and only if q is also true.
• Must find a mapping for such true elements in P1 and P2, as well as for false elements.
• Ensure that mapping can be done in polynomial time.
• *Note: P1 is unknown, P2 is difficult
Cook’s Theorem
• Stephen Cook (Turing award winner) found the first NP-Complete problem, 3SAT.
Basically a problem from Logic.
Generally described using Boolean formula.
A Boolean formula involves AND, OR, NOT operators and some variables.
Ex: (x or y) and (x or z), where x, y, z are boolean variables.
Problem Definition – Given a boolean formula of m clauses, each containing ‘n’
boolean variables, can you assign some values to these variables so that the
formula can be true?
Boolean formula: (x v y v ẑ) Λ (x v y v ẑ)
Try all sets of solutions. Thus we have exponential set of possible solutions. So it
is a NPC problem.
• Having one definite NP-Complete problem means others can also be proven NP-
Complete, using reduction.