The document discusses heuristic algorithms and their applications. It describes various heuristic algorithms including greedy algorithms, hill climbing, simulated annealing, tabu search, and ant colony algorithms. It provides examples of how these algorithms can be applied to job scheduling problems and the traveling salesman problem. The advantages of heuristic algorithms are that they can find approximate solutions quickly without needing to derive formulas, but the disadvantage is solutions are not guaranteed to be optimal.

1. 1
Heuristic Approach for OptimizationHeuristic Approach for Optimization
ANG SovannANG Sovann

2. 2
Content(1)
1. Introduction
1.1. History
1.2. Heuristic meaning
1.3. Heuristic value
1. Heuristic Algorithms
2.1. Greedy best first algorithm
2.2. Hill climbing algorithm
2.3. Simulated annealing
2.4. Tabu search

3. 3
Content(2)
2.5. Ant colonies
3. Applications
3.1. Job Scheduling problem
3.2. Travelling salesman problem
3.3. Other problems
4. Advantages and disadvantages
5. Conclusion

4. 4
1. Introduction
1.1. History
1.2. Heuristic meaning
1.3. Heuristic Value

5. 5
1.1. History
• In 1957, George Polya wrote an influential book called
‘‘How to solve it’’
• In 1963, Newell, Shaw and Simon stated: A process that may
solve a given problem, but offer no guarantee of doing so.
• In 1983 Kirkpatrick, Gelatt and Vecchi: Simulated Annealing
(SA)
• In 1990 Dueck and Scheuer: Threshold Accepting (TA)
• In 1992 Coloni, Dorigo and Maniezzo: Ant Colonies (AC)
• In 1997 Glover and Laguna : Tabu Search (TS)

6. 6
•The heuristic means ‘‘to find ’’or “to discover ’’.
• Heuristics method is a procedure that is used to
find the optimize and designed for solving
problems more quickly.
•It does not guarantee and not accurate algorithms
but it may be considered as approximately.
1.2. Heuristic meaning

7. 7
1.3. Heuristic Value
2 8 3
1 6 4
7 5
2 8 3
1 4
7 6 5
2 3
1 8 4
7 6 5
1 3
8 4
7 6 5
2
3
1 8 4
7 6 5
2
1 3
8 4
7 6 5
2
-5
h = -3
h = -3
h = 0
h = -4
-5
Initial state
v
Goal state
h = -2
h = -1

8. 8
2. Heuristic Algorithms
2.1. Greedy Best Search Algorithm
2.2. Hill climbing
2.3. Simulated Annealing Algorithm
2.4. Tabu Search Algorithm
2.5. Ant colony Algorithm

9. 9
What is Greedy Algorithm?
•A Greedy algorithm is an algorithm that follows
the problem solving heuristics of making the
locally optimal choice at each stagewith the hope
of finding a global optimum.
•In simple case, greedy often give you the best
solution.
2.1. Greedy Best-first Algorithm (1)

10. 10
•The Greedy Strategy:
– At each step, make the “best” next choice.
– Never backtrack or change past choices.
– Never look ahead to see if this choice has
negative consequences.
2.1. Greedy Best-first Algorithm (2)

11. 11
2.1. Greedy Best-first Algorithm (3)
A
B
C
D
E
Z
4
2
1
5
10
11
2
3
6
A
B C
B DE
D
ZE
2
2+1=3
4
2+8=132+10=12
3+5=8
8+2=10 8+6=14
Z10+3=13
Shortest path network.
• Source A, destination Z.
• Length i = length of edge i.
Shortest path problem: find shortest
directed path from A to Z.
=2+1+5+2+3=13
Cost of path = sum of edge costs in path
9

12. 12
•In large and complex case, greedy doesn’t always
give you the best solution, because it’s just search
and take the best choice that you can reach from
the current state.
•It takes longer time than any other algorithms for
big case of problem.
2.1. Greedy Best-first Algorithm (4)

13. 13
2.2. Hill climbing Algorithm (1)

14. 14
2 8 3
1 6 4
7 5
2 8 3
1 6 4
7 5
2 8 3
1 4
7 6 5
2 8 3
1 6 4
7 5
2 8 3
1 6 4
7 5
2 8 3
1 4
7 6 5
2 8 3
1 4
7 6 5
2 3
1 8 4
7 6 5
2 8 3
1 4
7 6 5
2 3
1 8 4
7 6 5
2 3
1 8 4
7 6 5
2 3
1 8 4
7 6 5
1 2 3
8 4
7 6 5
h = -4
h = -5h = -5
h = -3
h = -4h = -4
h = -3
h = -4
h = -3
h = -3
h = -2
h = -1
1 2 3
8 4
7 6 5
h = 0
1 2 3
8 4
7 6 5
h = 0
Goal state
current state
h = -4
h = -3
h = -2
h = -1
h = 0

15. 15
2.2. Hill climbing Algorithm (3)
Goal
maximum
• Problems:
– Local Maximum: peaks that aren’t the highest
point in the space
– Plateaus: the space has a broad flat region that
gives the search algorithm no direction (random
walk)
Local maximum

16. 16
2.3. Simulated Annealing (1)
• Slowly cool down a heated solid, so that particles arrange
in the ground energy state
0
200
400
600
800
1000
1200
0 20 40 60 80 100 120 140 160
AxisTitle
Axis Title

17. 17
2.3. Simulated Annealing (1)
Initialize:
– initial solution x ,
– highest temperature Th,
– and coolest temperature Tl
T= Th
When the temperature is higher than Tl
While not in equilibrium
Search for the new solution X’
Accept or reject X’ according to Metropolis Criterion
End
Decrease the temperature T
End
Algorithm

18. 18
2.3. Simulated Annealing (2)
Metropolis Criterion
•Let :
– X be the current solution and X’ be the new solution
– C(x) be the energy state (cost) of x
– C(x’) be the energy state of x’
• Probability Paccept = exp [(C(x)-C(x’))/ T]
• Let N = Random(0,1)
• Unconditional accepted if
– C(x’) < C(x), the new solution is better
• Probably accepted if
– C(x’) >= C(x), the new solution is worse .
– Accepted only when N < Paccept

19. 19
2.3. Simulated Annealing (3)
• Slowly cool down a heated solid, so that particles arrange
in the ground energy state C(x’) >= C(x), the new solution is
worse.
Accepted only when
N < Paccept

20. 20
2.4. Tabu search Algorithm (1)
• Tabu search works like hill climbing, but it
maintains a tabu list to avoid getting stuck in local
optima.

21. 21
2.4. Tabu search Algorithm (2)
Initial solution
(i in S)
Create a candidate
list of solutions
Evaluate solutions
Choose the best
admissible solution
Stopping conditions
satisfied ?
Update Tabu &
Aspiration
Conditions
Final solution
No Yes
Flowchart

22. 22
•Technique for solving problems which can be
expressed as finding good paths through
graphs.
•Originally applied to Traveling Salesman
Problem (TSP).
2.5. Ant colony algorithm

23. 23
2.5. Ant colony algorithm(2)
Naturally Observed Ant Behavior

24. 24
2.5. Ant colony algorithm(3)
Naturally Observed Ant Behavior
An obstacle has blocked path!

25. 25
2.5. Ant colony algorithm(4)
Naturally Observed Ant Behavior

26. 26
2.5. Ant colony algorithm(5)
Naturally Observed Ant Behavior
Shorter path reinforced.

27. 27
How the ant can find the shortest path?
•Use ‘Pheromone’ to communicate
•Lay Pheromone trail on the ground which as a signal
to attract other ants
•The more ants follow the trail , the stronger
Pheromone trail, more likely ants follow it
•Pheromone is built up on the shortest path
2.5. Ant colony algorithm(6)

28. 28
d = 0.5
d = 0.5
d = 1
d = 1
E
D
CH
B
A
(a)
E
D
CH
B
A
(b)
30 ants
30 ants
15 ants
15 ants
15 ants
15 ants
t = 0 E
D
CH
B
A
(c)
30 ants
30 ants
20 ants
20 ants
10 ants
10 ants
t = 1
2.5. Ant colony algorithm(cont.)

29. 29
• The majority of ants follow the short path
2.5. Ant colony algorithm(cont.)

30. 30
Ex. Single machine scheduling with due-dates
These jobs have to be done; their length
represents the time they will take.
A
B
C
D
E
3.1. Job Scheduling problem

31. 31
3.1. Job Scheduling problem
Ex. Single machine scheduling with due-dates
A
B
C
D
E
Each has a ‘due date’, when it needs to be finished
3pm
3:30pm
5pm
4pm
4:30pm
• Only one ‘machine’ is
available to process these
jobs, so can do just one at a
time.
e.g. machine might be
human tailor, photocopier…

32. 32
3.1. Job Scheduling problem
• A is 70min late Average lateness =46min
• B is 30min early (0 lateness)
• C is 60min late
• D is 50min late Max lateness=90min
• E is 50min late
A due 3pm B – 3:30 C - 5pm D – 4pm E -4:30
2 pm 3 pm 5 pm4 pm 6 pm

33. 33
3.1. Job Scheduling problem
• A is 70min late Average lateness =46min
• B is 30min early (0 lateness)
• C is 60min late
• D is 50min late Max lateness=70min
• E is 50min late
A due 3pmB – 3:30 C - 5pmD – 4pm E -4:30
3 pm 5 pm4 pm 6 pm2 pm

34. 34
3. 2 Traveling Salesman Problem
A Salesman wishes to travel around a given set of cities, and
return to the beginning, covering the smallest total distance

35. 35
• useful when solving issues that do
not require a formula
• Can be used in dynamic applications
(adapts to changes such as new
distances etc.)
• It can provide quick solution
approximately for new/revised standards to be
recognized by NIST
4. Advantages and Disadvantages(1)
Advantages:

36. 36
4. Advantages and Disadvantages(2)
Advantages:
• Save time in defining formula.
• Help to understand the problem more
easily.
• It can be resolved problem efficiently and
faster.

37. 37
4. Advantages and Disadvantages(3)
Disadvantages:
• Solutions can not guarantee that the
best solution
• The algorithm used for heuristic solution
has many conditions
• Technical more complex when have large
information, or large problem.

38. 38
5. Conclusion (1)
• Minimize/Maximize the costs of production
with Non-deterministic Polynomial time
problem (NP-hard)
•
Heuristic algorithm is very useful to:

39. 39
5. Conclusion (2)
And also used in:
• Artificial Intelligence
• Computer science
• Mathematical optimization