The document discusses analyzing algorithms through mathematical analysis and order-of-growth hypotheses. It covers estimating running time through experiments that measure time as input size increases. Doubling the input size provides the exponent in a power law model. Order-of-growth hypotheses assume algorithms follow patterns like constant, logarithmic, linear, quadratic, or cubic time as a function of input size. Precise mathematical models are difficult but approximations suffice by focusing on dominant terms.
EuroPython 2017 - PyData - Deep Learning your Broadband Network @ HOMEHONGJOO LEE
45 min talk about collecting home network performance measures, analyzing and forecasting time series data, and building anomaly detection system.
In this talk, we will go through the whole process of data mining and knowledge discovery. Firstly we write a script to run speed test periodically and log the metric. Then we parse the log data and convert them into a time series and visualize the data for a certain period.
Next we conduct some data analysis; finding trends, forecasting, and detecting anomalous data. There will be several statistic or deep learning techniques used for the analysis; ARIMA (Autoregressive Integrated Moving Average), LSTM (Long Short Term Memory).
Adaptive Noise Cancellation using Multirate TechniquesIJERD Editor
International Journal of Engineering Research and Development is an international premier peer reviewed open access engineering and technology journal promoting the discovery, innovation, advancement and dissemination of basic and transitional knowledge in engineering, technology and related disciplines.
The Shortest Path Tour Problem is an extension to the normal Shortest Path Problem and appeared in the scientific literature in Bertsekas's dynamic programming and optimal control book in 2005, for the first time. This paper gives a description of the problem, two algorithms to solve it. Results to the numeric experimentation are given in terms of graphs. Finally, conclusion and discussions are made.
EuroPython 2017 - PyData - Deep Learning your Broadband Network @ HOMEHONGJOO LEE
45 min talk about collecting home network performance measures, analyzing and forecasting time series data, and building anomaly detection system.
In this talk, we will go through the whole process of data mining and knowledge discovery. Firstly we write a script to run speed test periodically and log the metric. Then we parse the log data and convert them into a time series and visualize the data for a certain period.
Next we conduct some data analysis; finding trends, forecasting, and detecting anomalous data. There will be several statistic or deep learning techniques used for the analysis; ARIMA (Autoregressive Integrated Moving Average), LSTM (Long Short Term Memory).
Adaptive Noise Cancellation using Multirate TechniquesIJERD Editor
International Journal of Engineering Research and Development is an international premier peer reviewed open access engineering and technology journal promoting the discovery, innovation, advancement and dissemination of basic and transitional knowledge in engineering, technology and related disciplines.
The Shortest Path Tour Problem is an extension to the normal Shortest Path Problem and appeared in the scientific literature in Bertsekas's dynamic programming and optimal control book in 2005, for the first time. This paper gives a description of the problem, two algorithms to solve it. Results to the numeric experimentation are given in terms of graphs. Finally, conclusion and discussions are made.
How to Generate Personalized Tasks and Sample Solutions for Anonymous Peer Re...Mathias Magdowski
Keynote Presentation at the Doctoral Student Meeting 2021 of the IEEE German EMC Chapter
Abstract: In order to dissuade our students from bulimic learning and to motivate them to deal with electrical engineering already during the semester, we have developed a concept of personalized tasks with anonymous peer review. All students receive their own assignment by e-mail, can solve it and submit their solution as an explanatory video via a learning management system for correction. The video submission was chosen because not only the result but also the process of solving the problem can be documented much better and can be corrected or evaluated. In order to reduce the correction effort for the teachers, the students assess each other using a sample solution that is also personalized. The process runs automatically and is therefore easily scalable. Compared to simple multiple-choice or numerical value-and-unit tasks, the calculation method and approach as well as sketches, circuit diagrams and charts can also be evaluated well. This contribution describes how the tasks and sample solutions can be automatically generated in LaTeX with the help of the packages PGFPlots and Circuitikz.
REDUCING TIMED AUTOMATA: A NEW APPROACHijistjournal
Today model checking is the most useful verification method for real time systems, so there is a serious need for improving its efficiency with respect to both time and resources. In this paper we present a new approach for reducing timed automata. In fact regions of a region automaton are aggregated according to a coarse equivalence class partitioning based on traces. We will show that the proposed algorithm terminates and preserves original timed automaton. Proposed algorithms are implemented by model transformation with Atom3 tool.
The asynchronous parallel algorithms are developed to solve massive optimization problems in a distributed data system, which can be run in parallel on multiple nodes with little or no synchronization. Recently they have been successfully implemented to solve a range of difficult problems in practice. However, the existing theories are mostly based on fairly restrictive assumptions on the delays, and cannot explain the convergence and speedup properties of such algorithms. In this talk we will give an overview on distributed optimization, and discuss some new theoretical results on the convergence of asynchronous parallel stochastic gradient algorithm with unbounded delays. Simulated and real data will be used to demonstrate the practical implication of these theoretical results.
Design limitations and its effect in the performance of ZC1-DPLLIDES Editor
The paper studies the dynamics of a conventional
positive going zero crossing type digital phase locked loop
(ZC1-DPLL) taking non-ideal responses of the loop constituent
blocks into account. The finite width of the sampling pulses
and the finite propagation delay of the loop subsystems are
properly modeled mathematically and the system dynamics is
found to change because of their influence considered
separately. However, when these two are taken simultaneously,
the system dynamics can be made nearly equivalent to that of
the ideal system. Through an extensive numerical simulation
a set of optimum parameters to overcome design limitations
have been obtained.
Digital Signal Processing[ECEG-3171]-Ch1_L03Rediet Moges
This Digital Signal Processing Lecture material is the property of the author (Rediet M.) . It is not for publication,nor is it to be sold or reproduced.
#Africa#Ethiopia
Digital Signal Processing[ECEG-3171]-Ch1_L04Rediet Moges
This Digital Signal Processing Lecture material is the property of the author (Rediet M.) . It is not for publication,nor is it to be sold or reproduced.
#Africa#Ethiopia
new optimization algorithm for topology optimizationSeonho Park
authors devise new convex approximation called DQA which utilizes information of two consecutive points at iterates. Also, to guarantee global convergence, filter method is illustrated.
How to Generate Personalized Tasks and Sample Solutions for Anonymous Peer Re...Mathias Magdowski
Keynote Presentation at the Doctoral Student Meeting 2021 of the IEEE German EMC Chapter
Abstract: In order to dissuade our students from bulimic learning and to motivate them to deal with electrical engineering already during the semester, we have developed a concept of personalized tasks with anonymous peer review. All students receive their own assignment by e-mail, can solve it and submit their solution as an explanatory video via a learning management system for correction. The video submission was chosen because not only the result but also the process of solving the problem can be documented much better and can be corrected or evaluated. In order to reduce the correction effort for the teachers, the students assess each other using a sample solution that is also personalized. The process runs automatically and is therefore easily scalable. Compared to simple multiple-choice or numerical value-and-unit tasks, the calculation method and approach as well as sketches, circuit diagrams and charts can also be evaluated well. This contribution describes how the tasks and sample solutions can be automatically generated in LaTeX with the help of the packages PGFPlots and Circuitikz.
REDUCING TIMED AUTOMATA: A NEW APPROACHijistjournal
Today model checking is the most useful verification method for real time systems, so there is a serious need for improving its efficiency with respect to both time and resources. In this paper we present a new approach for reducing timed automata. In fact regions of a region automaton are aggregated according to a coarse equivalence class partitioning based on traces. We will show that the proposed algorithm terminates and preserves original timed automaton. Proposed algorithms are implemented by model transformation with Atom3 tool.
The asynchronous parallel algorithms are developed to solve massive optimization problems in a distributed data system, which can be run in parallel on multiple nodes with little or no synchronization. Recently they have been successfully implemented to solve a range of difficult problems in practice. However, the existing theories are mostly based on fairly restrictive assumptions on the delays, and cannot explain the convergence and speedup properties of such algorithms. In this talk we will give an overview on distributed optimization, and discuss some new theoretical results on the convergence of asynchronous parallel stochastic gradient algorithm with unbounded delays. Simulated and real data will be used to demonstrate the practical implication of these theoretical results.
Design limitations and its effect in the performance of ZC1-DPLLIDES Editor
The paper studies the dynamics of a conventional
positive going zero crossing type digital phase locked loop
(ZC1-DPLL) taking non-ideal responses of the loop constituent
blocks into account. The finite width of the sampling pulses
and the finite propagation delay of the loop subsystems are
properly modeled mathematically and the system dynamics is
found to change because of their influence considered
separately. However, when these two are taken simultaneously,
the system dynamics can be made nearly equivalent to that of
the ideal system. Through an extensive numerical simulation
a set of optimum parameters to overcome design limitations
have been obtained.
Digital Signal Processing[ECEG-3171]-Ch1_L03Rediet Moges
This Digital Signal Processing Lecture material is the property of the author (Rediet M.) . It is not for publication,nor is it to be sold or reproduced.
#Africa#Ethiopia
Digital Signal Processing[ECEG-3171]-Ch1_L04Rediet Moges
This Digital Signal Processing Lecture material is the property of the author (Rediet M.) . It is not for publication,nor is it to be sold or reproduced.
#Africa#Ethiopia
new optimization algorithm for topology optimizationSeonho Park
authors devise new convex approximation called DQA which utilizes information of two consecutive points at iterates. Also, to guarantee global convergence, filter method is illustrated.
Algorithm and its Properties
Computational Complexity
TIME COMPLEXITY
SPACE COMPLEXITY
Complexity Analysis and Asymptotic notations.
Big-oh-notation (O)
Omega-notation (Ω)
Theta-notation (Θ)
The Best, Average, and Worst Case Analyses.
COMPLEXITY Analyses EXAMPLES.
Comparing GROWTH RATES
Safalta Digital marketing institute in Noida, provide complete applications that encompass a huge range of virtual advertising and marketing additives, which includes search engine optimization, virtual communication advertising, pay-per-click on marketing, content material advertising, internet analytics, and greater. These university courses are designed for students who possess a comprehensive understanding of virtual marketing strategies and attributes.Safalta Digital Marketing Institute in Noida is a first choice for young individuals or students who are looking to start their careers in the field of digital advertising. The institute gives specialized courses designed and certification.
for beginners, providing thorough training in areas such as SEO, digital communication marketing, and PPC training in Noida. After finishing the program, students receive the certifications recognised by top different universitie, setting a strong foundation for a successful career in digital marketing.
Biological screening of herbal drugs: Introduction and Need for
Phyto-Pharmacological Screening, New Strategies for evaluating
Natural Products, In vitro evaluation techniques for Antioxidants, Antimicrobial and Anticancer drugs. In vivo evaluation techniques
for Anti-inflammatory, Antiulcer, Anticancer, Wound healing, Antidiabetic, Hepatoprotective, Cardio protective, Diuretics and
Antifertility, Toxicity studies as per OECD guidelines
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Dr. Vinod Kumar Kanvaria
Exploiting Artificial Intelligence for Empowering Researchers and Faculty,
International FDP on Fundamentals of Research in Social Sciences
at Integral University, Lucknow, 06.06.2024
By Dr. Vinod Kumar Kanvaria
Delivering Micro-Credentials in Technical and Vocational Education and TrainingAG2 Design
Explore how micro-credentials are transforming Technical and Vocational Education and Training (TVET) with this comprehensive slide deck. Discover what micro-credentials are, their importance in TVET, the advantages they offer, and the insights from industry experts. Additionally, learn about the top software applications available for creating and managing micro-credentials. This presentation also includes valuable resources and a discussion on the future of these specialised certifications.
For more detailed information on delivering micro-credentials in TVET, visit this https://tvettrainer.com/delivering-micro-credentials-in-tvet/
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
How to Build a Module in Odoo 17 Using the Scaffold MethodCeline George
Odoo provides an option for creating a module by using a single line command. By using this command the user can make a whole structure of a module. It is very easy for a beginner to make a module. There is no need to make each file manually. This slide will show how to create a module using the scaffold method.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
it describes the bony anatomy including the femoral head , acetabulum, labrum . also discusses the capsule , ligaments . muscle that act on the hip joint and the range of motion are outlined. factors affecting hip joint stability and weight transmission through the joint are summarized.
How to Add Chatter in the odoo 17 ERP ModuleCeline George
In Odoo, the chatter is like a chat tool that helps you work together on records. You can leave notes and track things, making it easier to talk with your team and partners. Inside chatter, all communication history, activity, and changes will be displayed.
2. Running time
“ As soon as an Analytic Engine exists, it will necessarily guide the
future
course of the science. Whenever any result is sought by its aid,
the question
how many
times do you
have to turn
the crank?
Charles Babbage (1864) Analytic Engine
2
3. Reasons to analyze algorithms
Predict performance.
this course (COS 226)
Compare algorithms.
Provide guarantees.
Understand theoretical basis. theory of algorithms (COS 423)
client gets poor performance because programmer
did not understand performance characteristics
Primary practical reason: avoid performance bugs.
3
4. Some algorithmic successes
Discrete Fourier transform.
• Break down waveform of N samples into periodic components.
• Applications: DVD, JPEG, MRI, astrophysics, ….
• Brute force: N2 steps.
Freidrich Gauss
• FFT algorithm: N log N steps, enables new technology. 1805
time
quadratic
64T
32T
16T
linearithmic
8T
linear
size 1K 2K 4K 8K
Linear, linearithmic, and quadratic
4
5. Some algorithmic successes
N-body Simulation.
• Simulate gravitational interactions among N bodies.
• Brute force: N2 steps.
• Barnes-Hut: N log N steps, enables new research.
Andrew Appel
PU '81
time
quadratic
64T
32T
16T
linearithmic
8T
linear
size 1K 2K 4K 8K
Linear, linearithmic, and quadratic
5
6. ‣ estimating running time
‣ mathematical analysis
‣ order-of-growth hypotheses
‣ input models
‣ measuring space
6
7. Scientific analysis of algorithms
A framework for predicting performance and comparing algorithms.
Scientific method.
• Observe some feature of the universe.
• Hypothesize a model that is consistent with observation.
• Predict events using the hypothesis.
• Verify the predictions by making further observations.
• Validate by repeating until the hypothesis and observations agree.
Principles.
• Experiments must be reproducible.
• Hypotheses must be falsifiable.
Universe = computer itself.
7
8. Experimental algorithmics
Every time you run a program you are doing an experiment!
Why is
my program so slow ?
First step. Debug your program!
Second step. Choose input model for experiments.
Third step. Run and time the program for problems of increasing size.
8
9. Example: 3-sum
3-sum. Given N integers, find all triples that sum to exactly zero.
Application. Deeply related to problems in computational geometry.
% more 8ints.txt
30 -30 -20 -10 40 0 10 5
% java ThreeSum < 8ints.txt
4
30 -30 0
30 -20 -10
-30 -10 40
-10 0 10
9
10. 3-sum: brute-force algorithm
public class ThreeSum
{
public static int count(long[] a)
{
int N = a.length;
int cnt = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
for (int k = j+1; k < N; k++) check each triple
if (a[i] + a[j] + a[k] == 0)
cnt++;
return cnt;
}
public static void main(String[] args)
{
int[] a = StdArrayIO.readLong1D();
StdOut.println(count(a));
}
}
10
12. Measuring the running time
Q. How to time a program?
A. Automatic.
Stopwatch stopwatch = new Stopwatch();
ThreeSum.count(a);
double time = stopwatch.elapsedTime();
StdOut.println("Running time: " + time + " seconds");
client code
public class Stopwatch
{
private final long start = System.currentTimeMillis();
public double elapsedTime()
{
long now = System.currentTimeMillis();
return (now - start) / 1000.0;
}
}
implementation
12
13. 3-sum: initial observations
Data analysis. Observe and plot running time as a function of input size N.
N time (seconds) †
1024 0.26
2048 2.16
4096 17.18
8192 137.76
† Running Linux on Sun-Fire-X4100
13
14. Empirical analysis
Log-log plot. Plot running time vs. input size N on log-log scale.
slope = 3
power law
Regression. Fit straight line through data points: c N a. slope
Hypothesis. Running time grows cubically with input size: c N 3.
14
15. Prediction and verification
Hypothesis. 2.5 × 10 –10 × N 3 seconds for input of size N.
Prediction. 17.18 seconds for N = 4,096.
Observations. N time (seconds)
4096 17.18
4096 17.15 agrees
4096 17.17
Prediction. 1100 seconds for N = 16,384.
Observation. N time (seconds)
agrees
16384 1118.86
15
16. Doubling hypothesis
Q. What is effect on the running time of doubling the size of the input?
N time (seconds) † ratio
512 0.03 -
1024 0.26 7.88
2048 2.16 8.43
4096 17.18 7.96
8192 137.76 7.96
numbers increases running time increases
by a factor of 2 by a factor of 8
lg of ratio is
exponent in power law
(lg 7.96 ≈ 3)
Bottom line. Quick way to formulate a power law hypothesis.
16
17. Experimental algorithmics
Many obvious factors affect running time:
• Machine.
• Compiler.
• Algorithm.
• Input data.
More factors (not so obvious):
• Caching.
• Garbage collection.
• Just-in-time compilation.
• CPU use by other applications.
Bad news. It is often difficult to get precise measurements.
Good news. Easier than other sciences.
e.g., can run huge number of experiments
17
18. ‣ estimating running time
‣ mathematical analysis
‣ order-of-growth hypotheses
‣ input models
‣ measuring space
18
19. Mathematical models for running time
Total running time: sum of cost × frequency for all operations.
• Need to analyze program to determine set of operations.
• Cost depends on machine, compiler.
• Frequency depends on algorithm, input data.
Donald Knuth
1974 Turing Award
In principle, accurate mathematical models are available.
19
20. Cost of basic operations
operation example nanoseconds †
integer add a + b 2.1
integer multiply a * b 2.4
integer divide a / b 5.4
floating point add a + b 4.6
floating point multiply a * b 4.2
floating point divide a / b 13.5
sine Math.sin(theta) 91.3
arctangent Math.atan2(y, x) 129.0
... ... ...
† Running OS X on Macbook Pro 2.2GHz with 2GB RAM
20
21. Cost of basic operations
operation example nanoseconds †
variable declaration int a c1
assignment statement a = b c2
integer compare a < b c3
array element access a[i] c4
array length a.length c5
1D array allocation new int[N] c6 N
2D array allocation new int[N][N] c7 N 2
string length s.length() c8
substring extraction s.substring(N/2, N) c9
string concatenation s + t c10 N
Novice mistake. Abusive string concatenation.
21
22. Example: 1-sum
Q. How many instructions as a function of N?
int count = 0;
for (int i = 0; i < N; i++)
if (a[i] == 0) count++;
operation frequency
variable declaration 2
assignment statement 2
less than comparison N+1
equal to comparison N
between N (no zeros)
and 2N (all zeros)
array access N
increment ≤ 2N
22
23. Example: 2-sum
Q. How many instructions as a function of N?
int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0) count++;
1
0 + 1 + 2 + . . . + (N − 1) = N (N − 1)
operation frequency 2
N
=
2
variable declaration N+2
assignment statement N+2
less than comparison 1/2 (N + 1) (N + 2)
equal to comparison 1/2 N (N − 1)
tedious to count exactly
array access N (N − 1)
increment ≤ N2
23
24. Tilde notation
• Estimate running time (or memory) as a function of input size N.
• Ignore lower order terms.
- when N is large, terms are negligible
- when N is small, we don't care
Ex 1. 6 N 3 + 20 N + 16
~ 6N3
Ex 2. 6 N 3 + 100 N 4/3 + 56
~ 6N3
Ex 3. 6 N 3 + 17 N 2 lg N + 7 N
~ 6N3
discard lower-order terms
(e.g., N = 1000 6 trillion vs. 169 million)
f (N)
Technical definition. f(N) ~ g(N) means lim = 1
N→ ∞ g(N)
€
24
25. Example: 2-sum
Q. How long will it take as a function of N?
int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0) count++; "inner loop"
operation frequency cost total cost
variable declaration ~ N c1 ~ c1 N
assignment statement ~ N c2 ~ c2 N
less than comparison ~ 1/2 N 2
c3 ~ c3 N 2
equal to comparison ~ 1/2 N 2
array access ~ N2 c4 ~ c4 N 2
increment ≤ N2 c5 ≤ c5 N 2
total ~ cN2
25
26. Example: 3-sum
478 Algorithms and Data
Q. How many instructions as a function of N?
public class ThreeSum
{ the leading term of math
public static int count(int[] a)
{
pressions by using a m
int N = a.length; device known as the til
int cnt = 0; 1
We write f (N) to re
for (int i = 0; i < N; i++)
quantity that, when divid
for (int j = i+1; j < N; j++) N
inner
for (int k = j+1; k < N; k++) ~N / 2
2
approaches 1 as N grow
loop
if (a[i] + a[j] + a[k] == 0) ~N / 6
3 write N g (N)N (Nf − 1)(N −to in
= (N) 2)
3 3!
cnt++; g (N) f (N)1approaches 1
∼ N 3
6
return cnt; depends on input data With this notation, we
}
complicated parts of an
public static void main(String[] args) that represent small val
{
int[] a = StdArrayIO.readInt1D(); ample, the if statement
int cnt = count(a); is executed N 3/6 tim
StdOut.println(cnt);
}
N (N 1)(N 2)/6 N 3/
}
Remark. Focus on instructions in inner loop; ignore everything else!N/3, which certainly, whe
Anatomy of a program’s statement execution frequencies N 3/6, approaches 1 as N
notation is useful when t
ter the leading term are r 26
27. Mathematical models for running time
In principle, accurate mathematical models are available.
In practice,
• Formulas can be complicated.
• Advanced mathematics might be required.
• Exact models best left for experts.
costs (depend on machine, compiler)
TN = c1 A + c2 B + c3 C + c4 D + c5 E
A = variable declarations
B = assignment statements frequencies
C = compare (depend on algorithm, input)
D = array access
E = increment
Bottom line. We use approximate models in this course: TN ~ c N3.
27
28. ‣ estimating running time
‣ mathematical analysis
‣ order-of-growth hypotheses
‣ input models
‣ measuring space
28
29. Common order-of-growth hypotheses
To determine order-of-growth:
• Assume a power law TN ~ c N a.
• Estimate exponent a with doubling hypothesis.
• Validate with mathematical analysis.
N time (seconds) †
Ex. ThreeSumDeluxe.java
1,000 0.43
Food for thought. How is it implemented?
2,000 0.53
4,000 1.01
8,000 2.87
16,000 11.00
32,000 44.64
64,000 177.48
observations
Caveat. Can't identify logarithmic factors with doubling hypothesis.
29
30. Common order-of-growth hypotheses
Linearithmic. We use the term linearithmic to describe programs whose running
time for a problem of size N has order of growth N log N. Again, the base of the
Good news. the smallrelevant. For example, CouponCollector (PROGRAM 1.4.2) is lin-
logarithm is not set of functions
earithmic.N, prototypical example 2, mergesort (see NROGRAM 4.2.6). Several im-
1, log The N, N log N, N is N3, and 2P
portant problems have natural solutions that are quadratic but clever algorithms
suffices to describe order-of-growth of typical algorithms. impor-
that are linearithmic. Such algorithms (including mergesort) are critically
tant in practice because they enable us to address problem sizes far larger than
could be addressed with quadratic solutions. In the next section, we consider a
general design technique for developing
time growth rate
linearithmic algorithms. name T(2N) / T(N)
1024T
Quadratic. A 1 constant 1
exponential
typical program whose
c
tic
cubi
ea c
i
hm
dra
512T
running time has order of growth N 2
r
rit
qua
log N logarithmic ~1
ea
lin
lin has two nested for loops, used for some
calculation involving all pairs linear ele-
N of N 2
64T ments. The force update double loop in
NBody (PROGRAM 3.4.2) is a linearithmicof
N log N prototype ~2
the programs in this classification, as is
the elementaryN2 quadratic
sorting algorithm Inser- 4
8T tion (PROGRAM 4.2.4).
4T
N 3
cubic 8
2T Cubic. Our example for this section,
logarithmic
N
2 exponential T(N)
ThreeSum,
is cubic (its running time has
T
constant order of growth N 3) because it has three
size 1K 2K 4K 8K 1024K
nested for loops, to process all triples of
N elements. The running time of matrix factor for
Orders of growth (log-log plot)
multiplication, as implemented in SEC- doubling hypothesis
TION 1.4 has order of growth M 3 to mul-
tiply two M-by-M matrices, so the basic matrix multiplication algorithm is often
considered to be cubic. However, the size of the input (the number of entries in the 30
31. Common order-of-growth hypotheses
growth
name typical code framework description example
rate
1 constant a = b + c; statement add two numbers
while (N > 1)
log N logarithmic { N = N / 2; ... } divide in half binary search
for (int i = 0; i < N; i++)
N linear { ... } loop find the maximum
divide
N log N linearithmic [see lecture 5] mergesort
and conquer
for (int i = 0; i < N; i++)
N2 quadratic for (int j = 0; j < N; j++) double loop check all pairs
{ ... }
for (int i = 0; i < N; i++)
for (int j = 0; j < N; j++)
N3 cubic for (int k = 0; k < N; k++) triple loop check all triples
{ ... }
exhaustive check all
2N exponential [see lecture 24]
search possibilities
31
32. Practical implications of order-of-growth
Q. How long to process millions of inputs?
Ex. Population of NYC was "millions" in 1970s; still is.
seconds equivalent
1 1 second
Q. How many inputs can be processed in minutes?
10 10 seconds
Ex. Customers lost patience waiting "minutes" in 1970s;
102 1.7 minutes
they still do.
103 17 minutes
104 2.8 hours
105 1.1 days
For back-of-envelope calculations, assume:
106 1.6 weeks
107 3.8 months
processor instructions
decade
speed per second 108 3.1 years
1970s 1 MHz 106 109 3.1 decades
1980s 10 MHz 107 1010 3.1 centuries
1990s 100 MHz 108 … forever
2000s 1 GHz 109 1017 age of universe
32
33. Practical implications of order-of-growth
problem size solvable in minutes time to process millions of inputs
growth
rate
1970s 1980s 1990s 2000s 1970s 1980s 1990s 2000s
1 any any any any instant instant instant instant
log N any any any any instant instant instant instant
tens of hundreds of
N millions billions minutes seconds second instant
millions millions
hundreds of hundreds of tens of
N log N millions millions hour minutes seconds
thousands millions seconds
tens of
N2 hundreds thousand thousands decades years months weeks
thousands
N3 hundred hundreds thousand thousands never never never millennia
33
34. Practical implications of order-of-growth
effect on a program that
runs for a few seconds
growth
name description
rate
time for 100x size for 100x
more data faster computer
1 constant independent of input size - -
log N logarithmic nearly independent of input size - -
N linear optimal for N inputs a few minutes 100x
N log N linearithmic nearly optimal for N inputs a few minutes 100x
N2 quadratic not practical for large problems several hours 10x
N3 cubic not practical for medium problems several weeks 4-5x
2N exponential useful only for tiny problems forever 1x
34
35. ‣ estimating running time
‣ mathematical analysis
‣ order-of-growth hypotheses
‣ input models
‣ measuring space
35
36. Types of analyses
Best case. Running time determined by easiest inputs.
Ex. N-1 compares to insertion sort N elements in ascending order.
Worst case. Running time guarantee for all inputs.
Ex. No more than ½N2 compares to insertion sort any N elements.
Average case. Expected running time for "random" input.
Ex. ~ ¼ N2 compares on average to insertion sort N random elements.
36
37. Commonly-used notations
notation provides example shorthand for used to
10 N 2
provide
Tilde leading term ~ 10 N 2 10 N 2 + 22 N log N
approximate model
10 N 2 + 2 N +37
N2
asymptotic classify
Big Theta Θ(N 2) 9000 N 2
growth rate algorithms
5 N 2 + 22 N log N + 3N
N2
develop
Big Oh Θ(N 2) and smaller O(N 2) 100 N
upper bounds
22 N log N + 3 N
9000 N 2
develop
Big Omega Θ(N 2) and larger Ω(N 2) N5
lower bounds
N 3 + 22 N log N + 3 N
38. Commonly-used notations
Ex 1. Our brute-force 3-sum algorithm takes Θ(N 3) time.
Ex 2. Conjecture: worst-case running time for any 3-sum algorithm is Ω(N 2).
Ex 3. Insertion sort uses O(N 2) compares to sort any array of N elements;
it uses ~ N compares in best case (already sorted) and ~ ½N 2 compares in the
worst case (reverse sorted).
Ex 4. The worst-case height of a tree created with union find with path
compression is Θ(N).
Ex 5. The height of a tree created with weighted quick union is O(log N).
base of logarithm absorbed by big-Oh
1
loga N = logb N
logb a
38
39. Predictions and guarantees
Theory of algorithms. Worst-case running time of an algorithm is O(f(N)).
Advantages
• describes guaranteed performance. time/memory
• O-notation absorbs input model.
f(N)
Challenges values represented
•
by O(f(N))
Cannot use to predict performance.
• Cannot use to compare algorithms.
input size
39
40. Predictions and guarantees
Experimental algorithmics. Given input model,average-case
running time is ~ c f(N).
Advantages. time/memory
• Can use to predict performance.
• Can use to compare algorithms.
c f(N)
values represented
Challenges. by ~ c f(N)
• Need to develop accurate input model.
• May not provide guarantees.
input size
40
41. ‣ estimating running time
‣ mathematical analysis
‣ order-of-growth hypotheses
‣ input models
‣ measuring space
41
42. Typical memory requirements for primitive types in Java
Bit. 0 or 1.
Byte. 8 bits.
Megabyte (MB). 210 bytes ~ 1 million bytes.
Gigabyte (GB). 220 bytes ~ 1 billion bytes.
type bytes
boolean 1
byte 1
char 2
int 4
float 4
long 8
double 8
42
43. Typical memory requirements for arrays in Java
Array overhead. 16 bytes on a typical machine.
type bytes type bytes
char[] 2N + 16 char[][] 2N2 + 20N + 16
int[] 4N + 16 int[][] 4N2 + 20N + 16
double[] 8N + 16 double[][] 8N2 + 20N + 16
one-dimensional arrays two-dimensional arrays
Q. What's the biggest double[] array you can store on your computer?
typical computer in 2008 has about 1GB memory
43
44. uble instance vari-
Typical memory requirements for objects in Java
rograms create mil-
nformation needed
Object overhead. 8 bytes on a typical machine.
ansparency). A refer-
Reference. 4 bytes on a typical machine.
data type contains a
ytes for theEach Complex object consumes 24 bytes of memory.
Ex 1.
reference
eded for the object’s
am 3.2.1) public class Complex
32 bytes 8 bytes overhead for object
Charge {
object
private double re; 8 bytes
overhead
uble rx; private double im; 8 bytes
uble ry; rx
... double
uble q; ry
} values 24 bytes
q
gram 3.2.6) 24 bytes
Complex
object
overhead
ouble re;
ouble im; re double
im values
rary) 12 bytes 44
45. object (Program 3.2.6) 24 bytes
c class Typical
Complex memory requirements for objects in Java
object
overhead
ivate double re;
ivate double im; re double
Object overhead. im
8 bytes on a typical machine.
values
Reference. 4 bytes on a typical machine.
ct (Java library) 12 bytes
ic class Color
Ex 2. A String of length N consumes 2N + 40 bytes.
object
overhead
rivate byte r;
rivate byte g; r g b a
rivate byte b;
rivate byte a;
public class String
byte 8 bytes overhead for object
values
{
private int offset; 4 bytes
object (Program 3.3.4) 16 bytes + string + vector
private int count; 4 bytes
c class Document
private int hash;
object 4 bytes
overhead
ivate String id;private char[] value; 4 bytes for reference
ivate Vector profile; id
... references (plus 2N + 16 bytes for array)
profile
}
2N + 40 bytes
ect (Java library) 24 bytes + char array
c class String
object
overhead
ivate char[] val;
ivate int offset; value reference
ivate int count; offset
ivate int hash; int
count values
hash
45
46. Example 1
Q. How much memory does this program use as a function of N ?
public class RandomWalk {
public static void main(String[] args) {
int N = Integer.parseInt(args[0]);
int[][] count = new int[N][N];
int x = N/2;
int y = N/2;
for (int i = 0; i < N; i++) {
// no new variable declared in loop
...
count[x][y]++;
}
}
}
46
47. Example 2
Q. How much memory does this code fragment use as a function of N ?
...
int N = Integer.parseInt(args[0]);
for (int i = 0; i < N; i++) {
int[] a = new int[N];
...
}
Remark. Java automatically reclaims memory when it is no longer in use.
47
48. Out of memory
Q. What if I run out of memory?
% java RandomWalk 10000
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
% java -Xmx 500m RandomWalk 10000
...
% java RandomWalk 30000
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
% java -Xmx 4500m RandomWalk 30000
Invalid maximum heap size: -Xmx4500m
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.
48
49. Turning the crank: summary
In principle, accurate mathematical models are available.
In practice, approximate mathematical models are easily achieved.
Timing may be flawed?
• Limits on experiments insignificant compared to
other sciences.
• Mathematics might be difficult?
• Only a few functions seem to turn up.
• Doubling hypothesis cancels complicated constants.
Actual data might not match input model?
• Need to understand input to effectively process it.
• Approach 1: design for the worst case.
• Approach 2: randomize, depend on probabilistic guarantee.
49
Editor's Notes
DFT = fast way to multiply two univariate polynomials
FFT reference: Runge-K&#xF6;nig (1924), Cooley-Tukey (1965) rediscovered FFT and popularized it. Discovered by Gauss at age of 28, two years before Fourier's paper on Fourier transforms!
medical imaging = CAT scan, MR
astrophysicists want N = 64 gigapoints to search for pulsars; seismologists want N = 10 gigapoints to predict earthquakes
Appel designed his fast multipole method for senior thesis in Physics at Princeton! (actually can be made linear)
physicists want N = # atoms in the universe
assumes no integer overflow
Run ThreeSum.java in class from the command-line and wait for it to finish, perhaps up to N = 4096.
Ask students to make predictions.
log-log plot identifies power law relationships
Very accurate predictions.
4x input size -> 64x increase in running time for cubic algorithm
put more emphasis on ratios with larger N since lots of noise contained in data for smaller N
if connected to Internet, some hacker is using your computer to send spam, steal your money, etc.
Good news: compared to other scientists, running computational experiments is easy. Don't need to kill cows, blow up islands.
most famous modeling by Knuth - detailed results to accurate predict running time
Knuth's idea seems obvious in hindsight, but in the 1960s, programmers said it would never be possible to analyze the running time of a computer program since they are too complicated. Knuth's TAOCP is now a classic treatise on the subject.
consider putting in instruction counts for N = 10000
N^3 / 6 arises because N choose 3 is the number of triples i, j, k
this partially explains reason why small set of functions
arises from straightforward composition of loops, conditionals, functions
didn't put column for cost - in 1970s, machine cost millions of $$$
Ex. Population of NYC was &#x201C;millions&#x201D; in 1970s; still is
Ex. Customers lost patience waiting &#x201C;minutes&#x201D; in 1970s; still do
Note: not an issue for 3-sum since best, average, and worst are all almost the same.
dependence on input implied ????
table contains typical memory requirements for primitive types
2d array is represented as an array of arrays: e.g., for double[][], there is one array with N references to arrays (4N + 16) and N double[] arrays (N (8N + 16)) for a total of 8N^2 + 20N + 16 bytes.
A. ~ 4N^2
What to do in last case? Need better algorithm that uses less memory.
Mathematics might be difficult.
It is known that properties of singularities of functions in the complex plane play a role in analysis of many algorithms
Leading term might not be good enough?
Typical programs have bottleneck in one function.
Debugging tools are available to identify bottlenecks.
Timing may be flawed.
Different results on different computers.
Different results on same computer at different times.