PWang_PhD_Thesis_Final

Department of Mechanical Engineering
Date: January 14, 2012
I HEREBY RECOMMEND THAT THE THESIS PREPARED UNDER MY
SUPERVISION BY
Peter Chang-Chun Wang
ENTITLED
A Poly-Hybrid Particle Swarm Optimization Method with
Intelligent Parameter Adjustment
BE ACCEPTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE
DEGREE OF
DOCTOR OF PHILOSOPHY IN MECHANICAL ENGINEERING
Terry E. Shoup, Ph.D Weijia Shang, Ph.D.
Thesis Advisor Thesis Reader
Timothy Hight, Ph.D.
Thesis Reader
Christopher Kitts, Ph.D.
Thesis Reader
Peter J. Woytowitz, Ph.D.
Thesis Reader

2
A Poly-Hybrid Particle Swarm Optimization Method
With Intelligent Parameter Adjustment
Prepared By:
Peter C. Wang
Dissertation
Submitted in Partial Fulfillment of the Requirements
For the Degree of Doctor of Philosophy
In Mechanical Engineering
In the School of Engineering at
Santa Clara University, January 14, 2012
Santa Clara, California

3
DEDICATION
I would like to thank God for His guidance of our family throughout our lives and
blessing us. He gave us peace, health, and happiness and the addition of my daughter,
Jane, to the family was the greatest gift we got from Him.
To my lovely wife, Betty, for being with me, her love and patience, and taking
care of the family while I was pursuing my Ph.D. studies and working full-time. It is a
blessing to have her as my wife.
To my parents, Jack and Mary, and brothers, Harry and Walter, for being my
family, their support and encouragement throughout my life.

4
ACKNOWLEDGEMENT
To Professor Shoup, his guidance, patience and understanding throughout my
Ph.D. studies.
To my Ph.D. committee members Professors Shang, Hight, Kitts, Chiappari and
Woytowitz for their guidance and suggestions.

5
TABLE OF CONTENTS
1. Introduction……………...…..................................................13
1.1 325 BCE to 1939 Period………………..………....…...…………...15
1.2 1939 to 1990 Period..…………………….…….……………..……..16
1.3 1990 to 2011 Period………………………..….……….…………….16
2. Parameter Sensitivity Study of Particle Swarm Method…………...19
2.1 Parameter Sensitivity of the Particle Swarm Method…………….20
2.1.1 Velocity…………………………………………………..…..22
2.1.2 Swarm Population Size………………..………………..……..25
2.1.3 Iteration………………………………………………..……..28
2.1.4 Dimension of the Design Space…………………………….…31
2.2 Other Consideration in Using the PSO Methods………………....35
3. Parameter Sensitivity Study of Pattern Search Methods………..36
3.1 Nelder-Mead Simplex Method……………………………………..37
3.1.1 Computational Results & Design of Experiments……...…39
3.1.2 P aram et er S en s i t i v i t y S t u d y o f NM S i m p l ex S earch
Method………………………………………………………..39
3.1.3 Results………………………………………………………..40
3.2 Hooke & Jeeves Method……………………………………………45
3.2.1 Computational Results, Design of Experiments, Parameter
Sensitivity Study of Hooke & Jeeves Method…………….45
3.2.2 Results………………………………………………………..46
3.3 The Random Search Method…………………………….…………50
4. The Poly-Hybrid PSO Method……………………………………….51
4.1 Current Search Algorithms for Optimization……………………..52
4.2 The Proposed Poly-Hybrid PSO Method………………………..…53
4.3 Classic Benchmark Problems……………………………………...64
4.4 Experiment and Results...…………………………………………….64
4.4.1 Experiment……………………………………………………64
4.4.2 Results……………………………………………………….65
4.5 A Guideline for Choosing the Right Algorithm…………………..72
4.6 Conclusions………………………………………………………….72
5. Conclusion & Future Research……………………………………73
6. List of References………………………………………………………74
7. Appendix…………………………………………………………………81

6
LIST OF TABLES
1.1 General Steps of the PSO Algorithm……………………………………..17
2.1 List of Popular Benchmark Problems………………………………………20
2.2 PSO Sensitivity Study Range of Values Used……………………………21
2.3 PSO Parameter Sensitivity Study Summary………………………………..34
3.1 Parameter Sensitivity Ranges……………………………………………..40
3.2 List of Test Functions……………………………………………………..40
3.3 Comparisons of Suggested Coefficients for NM Simplex Method…….41
3.4 Wang and Shoup NM Simplex Method Sensitivity Studies………………42
3.5 Hooke & Jeeves Method Sensitivity Studies…………………………….49
4.1 A Listing of Frequently Acknowledged Best Choice Methods for
Optimization……………………………………………………………………..53

7
LIST OF FIGURES
1.1 Design Problem Solving Process…………………………………………13
1.2 History Timeline of Optimization Methods Developed……………...15
2.1 Sensitivity of the PSO Method to the Velocity Parameter for the
Spherical Benchmark Test Function…………………...……………… 22
Griewank Benchmark Test Function……………………………………23
Schaffer F6 Benchmark Test Function……………………………… 23
Rosenbrock Benchmark Test Function…………………………………24
Rastrigin Benchmark Test Function……………………………………24
2.6 Sensitivity of the PSO Method to the Population Parameter for the
Spherical Benchmark Test Function……………………………………25
Schaffer F6 Benchmark Test Function…………………………………28
2.11 Sensitivity of the PSO Method to the Iteration Parameter for the
Schaffer F6 Benchmark Test Function…………………………………..31
2.16 Sensitivity of the PSO Method to the Dimension Parameter for the
Griewank Benchmark Test Function……………………………………..33
Rastrigin Benchmark Test Function……………………………………..33
Schaffer F6 Benchmark Test Function…………………………………..34

8
2.21 Outliers in the PSO Swarm……………………………………………….35
3.1 Sensitivity of the Simplex Optimization Method to the Reflection
Coefficient (α) for the Seven Benchmark Test Function………..……41
3.2 Sensitivity of the Simplex Optimization Method to the Contraction
Coefficient (β)Parameter for the Seven Different Benchmark Test
Function…………………………………………………………………….43
3.3 Sensitivity of the Simplex Optimization Method to the Size (ζ)
Param eter fo r the Seven Different Benchm ark Test Function
………………………………………………………………………………44
3.4 Sensitivity of the Simplex Optimization Method to the Expansion
( γ ) P a r a m e t e r f o r t h e S e v e n D i f f e r e n t B e n c h m a r k T e s t
Function……………………………………………………………………45
3.5 Sensitivity of the Hooke & Jeeves Optimization Method to the
A c c e l e r a t i o n F a c t o r ( δ ) f o r t h e S p h e r i c a l B e n c h m a r k T e s t
Function…………………………………………………………………….47
A c c e l e r a t i o n F a c t o r ( δ ) f o r t h e G r i e w a n k B e n c h m a r k T e s t
Function…………………………………………………………………….47
A c c e l e r a t i o n F a c t o r ( δ ) f o r t h e R a s t r i g i n B e n c h m a r k T e s t
Function…………………………………………………………………..48
Accel erati on Facto r (δ) fo r t he Ro senb rock Ben ch mark T es t
Function…………………………………………………………………..48
Acceleration Factor (δ) for the Schaffer F6 Benchmark Test
Function…………………………………………………………………….49
4.1 Poly-Hybrid Flow Chart Showing the Information Flow from the
Central Controller to Each Method and Vice Versa…………………54
4.2 Poly-Hybrid Graphical Representation (Initial)……………………...55
4.3 Poly-Hybrid Graphical Representation (Final)…………..……………..56
4.4 Overview of the Poly-Hybrid PSO Flowchart…………………………..57
4.5 Detail of the Central Control Flowchart…………………………………58
4.6 Detail of the PSO Flow-chart…………………………………………..59
4.7 Detail of the Random Flow-chart………………………………………...60
4.8 Detail of the Simplex Flow-chart……………………….……………...60
4.9 Detail of the HJ Flow-chart……………………………….…………….60
4.10 Algorithm and True Merit Value Difference Comparison (Spherical)65
4.11 A Comparison of Merit Evaluations Required to Achieve an Acceptable
Solution for the PSO method When Solving the Nine Test Function
(Spherical)………………………………………………………………..66
4.12 A Comparison of Run Time Required to Achieve an Acceptable
Solution for the PSO Method When Solving the Nine Test Functions
( S p h e r i c a l ) … … … … … … … … … … … … … … … … . . … … … . . 6 6

9
4.13 A l g o r i t h m a n d T r u e M e r i t V a l u e D i f f e r e n c e C o m p a r i s o n
(Griewank)………………………………………………………....………67
4.14 A Comparison of Merit Evaluations Required to Achieve an Acceptable
Solution for the PSO Method When Solving the Nine Test Function
(Griewank)…………….…………..……………………………………....67
4.15 A Comparison of Run Time Required to Achieve an Acceptable
Solution for the PSO Method When Solving the Nine Test Function
(Griewank)………………………………………………..………………..68
4.16 Individual Performance of the Hooke & Jeeves Using the Griewank
Function Between Merit Evaluations and Merit Values Found………69
4.17 Individual Performance of the Simplex Method Using the Griewank
Function Between Merit Evaluations and Merit Values Found………69
4.18 Individual Performance of the Random Method Using the Griewank
Function Between Merit Evaluations and Merit Values
Found……………………………………………………………………....70
4.19 Individual Performance of the PSO Method Using the Griewank
F u n c t i o n B e t w e e n M e r i t E v a l u a t i o n s a n d M e r i t V a l u e s
Found……………………………………………………………………...70
4.20 Performance of the Poly-Hybrid PSO Method Using the Griewank
F u n c t i o n B e t w e e n M e r i t E v a l u a t i o n s a n d M e r i t V a l u e s
Found……………………………………………………………………..71

10
NOTATION
xi(t+1) is the new position of a particle
xi(t) is the current position of a particle
vi(t+1) is the new velocity of a particle
Reflection (α) is a parameter used by the Simplex method for reflection
Expansion (γ) is a parameter used by the Simplex method for expansion
Contraction (β) is a parameter of the Simplex method to contract the simplex
Simplex Size (ζ) is a parameter of the Simplex method to scale the size of the simplex
Acceleration Factor (δ) is a parameter of the Hooke & Jeeves method used to move in the
design space
w is the inertia weight (from experimental tests, 0.9 -> 0.4)
vi(t) is the current velocity of a particle
n1, n2 are ~ 2.0 (from experimental tests)
r1, r2 are random numbers generated between 0 and 1
pi is the best position for a neighboring particle
xi(t) is the current position of a particle

11
pbest is the best position for a global particle
∆t is the change in time
Fopt is the optimum value
Pexp is the new vertex generated for an expansion
Pcont is the new vertex generated for a contraction
Pi is the new vertex generated for a size
Prefl is the new vertex generated for a reflection
Pcent is the center vertex of a simplex
Phigh is the highest value vertex of a simplex
Plow is the lowest value vertex of a simplex

12
ABSTRACT
This thesis presents results of a new hybrid optimization method which combines
the best features of four traditional optimization methods together with an intelligent
adjustment algorithm to speed convergence on unconstrained and constrained
optimization problems. It is believed that this is the first time that such a broad array of
methods has been employed to facilitate synergistic enhancement of convergence.
Particle swarm optimization is based on swarm intelligence inspired by the social
behavior and movement dynamics of bird flocking, fish schooling, and swarming theory.
This method has been applied for structural damage identification, neural network
training, and reactive power optimization. It is also believed that this is the first time an
intelligent parameter adjustment algorithm has been applied to maximize the
effectiveness of individual component algorithms within a hybrid method. A
comprehensive sensitivity analysis of the traditional optimization methods within the
hybrid group is used to demonstrate how the relationship among the design variables in a
given problem can be used to adjust algorithm parameters. The new method is
benchmarked using 11 classical test functions and the results show that the new method
outperforms eight of the most recently published search methodologies.

13
Chapter 1: Introduction
Optimization is the process of moving toward improvement given a set of
requirements and constraints. Although optimization occurs in a wide range of fields
where decisions are needed, it is especially helpful to the various disciplines of
engineering when it is applied to the design problem solving process shown in figure 1.1
(after [49]).
In this situation, a set of requirements and constraints is used to create a merit
function equation to be maximized or minimized. The constraints on the problem are
also used to create mathematical relationships that must be satisfied in the solution
process. Perhaps the most important part of the solution process is that a scheme or
algorithm is used that allows the extremum to be approached in an efficient way. This
scheme or algorithm is known as an optimization method. With all optimization methods
the goal is to minimize the effort required to reach a solution or to maximize the desired
benefit that the solution provides. In the fields of engineering, optimization is widely
used in the automotive, energy, and aerospace/defense industries to save cost, material,
and time.
Figure 1.1 Design problem solving process (after [49])
Design/Problem Solving
Recognize
the Need
Create a
Design
Prepare a
Model
Test &
Evaluate the
Model
Communicate
the Design
Improve the
Design
Iteration
Optimizatio
Recognize
the Need
Create a
Design
Prepare a
Model
Test &
Evaluate the
Model
Communicate
the Design
Improve the
Design
Iteration
Optimization

14
Optimization methods are traditionally classified as either indirect or direct.
Indirect methods are those schemes that use calculus theories to identify places in the
design space where an extremum occurs. Direct methods are those schemes that are
based on experimentation with various allowable values of the design parameters within
the allowable design space in an effort to discover information about where an extremum
may lie. Indirect methods are typically difficult to apply because of their mathematical
rigor but provide highly meaningful results. Direct methods are typically easy to apply
because of their experimental simplicity, but provide limited information about the
location of the optimum unless a large number of experiments are used. For this reason,
the advent of the digital computer was a pivotal moment in the field of optimization
where a switch from indirect to direct methods occurred. Indeed, as the computational
capability of the modern digital computer has increased, this transition from indirect to
direct methods has accelerated. Thus, it seems logical to categorize the history of
optimization methods into three eras based on periods of time. These three eras are the
time between 325 BCE and 1939, the time between 1939 and 1990, and the time between
1990-and the present. These three periods will now be used to highlight the historical
development of optimization.
Figure 1.2 shows a timeline list of the optimization methods developed during the
325 BCE-1939, 1939-1990, and 1990-2011 periods. The details of each method are well
explained in detail by authors such as Rao [39] and Shoup and Mistree [49]. In this
chapter, only a brief summary is presented. Before the computer was invented in 1939,
many of the optimization theories were based on the indirect method of the calculus of
variations. These theories later formed the basis for some of the direct methods which
came into prominence in the 1939-1990 periods as computational power increased. In
the 1990-2011 periods, direct methods gained increased popularity in research because of
the increases in complexity of engineering problems and because of the increases in
computational capability that has become available. It was during this third era that
research has been initiated to relate evolutionary development to optimization methods.
This unique approach attempts to discover if there is some intelligence that exists in

15
nature that can be applied to the mathematics of optimization. This approach has resulted
in the development of the genetic algorithms and swarm intelligence methods.
1.1 325 BCE-1939 Period
Calculus was invented during this period (circa 1684). The well-known gradient
method was presented by Cauchy in 1847. This method is a first order optimization
algorithm that takes steps proportional to the magnitude of the local gradient of the
function at the current point to find a trajectory leading to a local extremum. It is a strong
local search method. This method led to developments of the conjugate gradient methods
and has been included in hybrid methods developed in later times. As optimization
problems have become more complicated and have required more computational power,
the invention of the digital computer gave rise to the birth of the 1939-1990 periods.
Figure 1.2 History timeline of optimization methods developed
Interpolation
Methods
Elimination
Methods
Direct Methods
Integer Nonlinear Programming
Linear Programming
Geometric Programming
Dynamic Programming
Indirect Methods
201119901939325 BCE
Stochastic Programming
Integer Linear Programming
Genetic Programming
Swarm Intelligence (Particle
Swarm, Ant, Bee, etc )
Pre-Calculus
Calculus
of
Variation
Gradient Method
Interpolation
Methods
Elimination
Methods
Direct Methods
Integer Nonlinear Programming
Linear Programming
Geometric Programming
Dynamic Programming
Indirect Methods
201119901939325 BCE
Stochastic Programming
Integer Linear Programming
Genetic Programming
Swarm Intelligence (Particle
Swarm, Ant, Bee, etc )
Pre-Calculus
Calculus
of
Variation
Gradient Method

16
1.2 1939-1990 Period
When the digital computer was introduced in 1939, this new device provided a tool
that allowed direct methods to become more computationally feasible. As a result, the
field of optimization saw renewed research interest and a new emphasis began on the
development of optimization methods. A large number of new optimization techniques
thus emerged and several well defined new areas in optimization theory soon followed.
During this period, the pattern search methods and random search methods were
developed.
Pattern search methods are known for their simplicity and are popular in the
optimization field because their performance is better than methods that rely on gradient
calculations. They are superior to gradient methods when used on merit surfaces that
have sharply defined ridges that can occur as a result of the imposed constraints. The
most popular pattern search methods are the Hooke & Jeeves method, the Rosenbrock
method, and the Nelder-Mead Simplex method.
1.3 1990-2011 Period
In the most recent era, modern optimization methods have emerged for solving
increasingly complex and mathematically challenging engineering problems.
Since the research in this dissertation is strongly dependent on one of the swarm
intelligence methods, the Particle Swarm Optimization (PSO) method, a more detailed
explanation of how it works will now be presented. Generally, the following are the
steps for the PSO.

17
Particle Swarm Optimization
Step Description
1 Initialize population in hyperspace.
2 Calculate the merit values of individual particles. This value is the new particle’s position.
3 If the particle’s new position is better than its previous best position, update to the new position.
4 Determine the best particle (according to particle’s previous best positions).
5 Update a particle’s velocities with vi(t+1)= vi(t) + n1 * r1 * (pi - xi(t)) + n2 * r2 * (pbest - xi(t)).
6 Move particles to their new positions with xi(t+1)= xi(t)+ vi(t+1).
7 Go to step 2 until stopping criteria are satisfied (max iteration reached or minimum error satisfied).
Where,
• vi(t+1) is new velocity of the particle
• w is inertia weight (from experimental tests, 0.9 -> 0.4)
• vi(t) is current velocity of the particle
• n1, n2 are ~ 2.0 (from experimental tests)
• r1, r2 are random numbers generated between 0 and 1
• pi is the best position for neighboring particle
• xi(t) is the current position of the particle
• pbest is the best position for the global particle
• xi(t+1) is the new position of a particle
• ∆t=1
The advantages to the PSO method are that it is derivative free, it serves as a good
global search algorithm since it is capable of moving through the design space without
being trapped at a local optimum and it is simple to implement. The disadvantages are its
weak local search ability that sometimes causes premature convergence, its characteristic
of having too many input parameters, its propensity to have no defined termination
requirements, and its inherent difficulty in determining the best set of PSO parameters
(the population, the maximum velocity, the maximum position, the error function, etc…)
to produce an optimum solution.
Table 1.1 General steps of the PSO algorithm [after 3]

18
Since its development in 1995, the PSO method has seen improvements aimed at
improving its convergence by providing a damping factor in the velocity equation. These
methods are the CPSO, FIPS, and GCPSO and are described in detail by Rao [39].
The PSO and its modified versions are good global search algorithms and are
flexible in moving through the design space; however, help is needed for narrowing the
search, since sometimes it converges prematurely. Classical methods like pattern search
methods can help with this premature convergence. This can reduce the number of merit
evaluations needed and can thus reduce convergence time.
This thesis is organized with chapter 1 presenting a brief history of optimization
methods developed through 2011. Chapter 2 presents a parameter sensitivity study of the
particle swarm optimization method and how each parameter affects the merit function
values. Chapter 3 presents a parameter sensitivity study of the Nelder-Mead Simplex
method and how each parameter in this method affects the merit function values.
Chapter 4 presents a new poly-hybrid PSO method and shows how it is an improvement
in the field of optimization. Chapter 5 summarizes the study and presents future research.
A list of references and appendix follow to conclude the thesis.

19
Chapter 2: Parameter Sensitivity Study of the Particle Swarm Method
Every modern optimization method currently in use requires the user to select
certain parameters prior to starting the optimization process. Examples of these
parameters might be such things as initial boundaries on the design space, initial starting
points for a search, an initial exploration step size, acceleration factors, shrinkage factors,
the number of merit evaluations in an initial exploration step, etc. A careful selection of
these key starting parameters can often lead to rapid convergence to a very good solution.
A poor selection of these starting parameters can often lead to slow convergence or no
convergence at all to a usable solution. For this reason, in the field of optimization, a
recent approach to making the best use of various algorithms is to investigate the
sensitivity of parameter choices utilizing benchmark test problems to determine which
parameters are most sensitive to the overall success of the optimization process. Some of
the most important examples of the use of the sensitivity analysis are by Fan & Sahara
[46], M. Clerc [55], and Eberhart & Shi [4]. This analysis process is known as a
sensitivity analysis and proceeds by holding all input parameters but one fixed, and
running the optimization algorithm for selected choices of the remaining single parameter.
Since the research associated with this dissertation is aimed at developing a new
hybrid optimization method that will involve the use of the PSO method and several other
traditional methods, it is important that we begin our investigation with a parameter
sensitivity analysis for the PSO method and these other traditional methods. Chapters 2
of the thesis will present sensitivity studies of the particle swarm optimization method
and Chapter 3 will present similar information for the pattern search methods. The
presentation in Chapter 2 will show how the PSO velocity, the PSO population size, the
PSO iteration parameter, and the number of dimensions of the problem influence the
quality of a solution found. Dejong’s Test Functions [29] and Eason and Fenton’s [2] test
functions will be used to conduct the sensitivity study.

20
The results of this sensitivity will be summarized and a gap analysis will be
presented to suggest where improvements are needed for the current particle swarm
optimization method.
All sensitivity studies in this research were implemented using a Dell Inspiron
laptop with Pentium dual core CPU and 64 bit operating system. Coding was done in
Visual Basic for Excel.
2.1 Parameter Sensitivity Study of the Particle Swarm Method
The key parameters used to start the PSO method are the velocity, the population,
the iteration parameter, and the number of dimensions of the problem. We will study
these parameters and their impact on finding the true optimum solution using the five
DeJong Test Functions [29] and six Eason and Fenton [2] problems shown in Table 2.1.
The study is done such that one variable at a time is varied, while all others are fixed.
Table 2.1 List of popular benchmark problems [after 2 and 29]Table 2.1 List of Popular Benchmark Problems
Functions
Dynamic
Range (Xmax)
Error
Criterion
Optimum
Value
Minimum
Optimum X
Equation
Spherical 10.00 1.00E-02 0.00E+00 0.0
Rosenbrock 100.00 1.00E+02 0.00E+00 1.0
Rastrigin 5.12 1.00E+02 0.00E+00 0.0
Griewank 600.00 5.00E-02 0.00E+00 0.0
Shaffer's F6 10.00 1.00E-05 0.00E+00 0.0
EF2 42.00 1.00E-01 -3.30E+00 (20,11,15)
EF4 10.00 1.00E-01 0.00E+00 (1,1,1,1)
EF5 2.00 1.00E-01 0.00E+00 (1,1)
EF6 5.00 1.00E-01 1.62E+00 (1.2867,0.53047)
EF7 125.00 1.00E-01 -5.68E+00 (22.3, 0.5, 125)
EF8 3.00 1.00E-01 1.74E+00 (1.7435,2.0297)
∑=
=
D
i
ixxf
1
2
)(
[ ]∑
−
−+−+⋅⋅=
1
222
)1()1(100)(
D
i
iii xxxxf
∑=
⋅⋅⋅−+⋅=
D
i
ii xxDxf
1
2
)))2cos(10((10)( π
∑ ∏= =
+−=
D
i
D
i
ii
i
xx
xf
1 1
2
1))(cos()
4000
()(
[ ]
[ ]222
222
)(001.01
5.0)(sin
5.0)(
yx
yx
xf
++
−+
+=
1000
)( 321 xxx
xf
⋅⋅−
=
[ ] )1()1(8.19)1()1(1.10)1()(90)1()(100)( 42
3
4
2
2
2
3
22
34
2
3
23
12 −⋅−⋅+−+−⋅+−+−⋅+−+−⋅= xxxxxxxxxxxf
3
1
23
12 )1()(100)( xxxxf −+−⋅=
10
)592.01044.0(
)(
3
21
3
1
3
2
3
1
−−−
⋅+⋅+⋅⋅
=
xxxxx
xf
7
2
32
4
1
10
0201.0
)(
xxx
xf
⋅⋅⋅−
=
10
)(
)100()1(
12
)(
4
23
2
2
2
1
2
1
2
22
1 





⋅
+⋅
+
+
++
=
xx
xx
x
x
x
xf
[35]
[35]
[35]
[35]
[35]
[10]
[10]
[10]
[10]
[10]
[10]
Functions
Dynamic
Range (Xmax)
Error
Criterion
Optimum
Value
Minimum
Optimum X
Equation
Spherical 10.00 1.00E-02 0.00E+00 0.0
Rosenbrock 100.00 1.00E+02 0.00E+00 1.0
Rastrigin 5.12 1.00E+02 0.00E+00 0.0
Griewank 600.00 5.00E-02 0.00E+00 0.0
Shaffer's F6 10.00 1.00E-05 0.00E+00 0.0
EF2 42.00 1.00E-01 -3.30E+00 (20,11,15)
EF4 10.00 1.00E-01 0.00E+00 (1,1,1,1)
EF5 2.00 1.00E-01 0.00E+00 (1,1)
EF6 5.00 1.00E-01 1.62E+00 (1.2867,0.53047)
EF7 125.00 1.00E-01 -5.68E+00 (22.3, 0.5, 125)
EF8 3.00 1.00E-01 1.74E+00 (1.7435,2.0297)
∑=
=
D
i
ixxf
1
2
)(
[ ]∑
−
−+−+⋅⋅=
1
222
)1()1(100)(
D
i
iii xxxxf
∑=
⋅⋅⋅−+⋅=
D
i
ii xxDxf
1
2
)))2cos(10((10)( π
∑ ∏= =
+−=
D
i
D
i
ii
i
xx
xf
1 1
2
1))(cos()
4000
()(
[ ]
[ ]222
222
)(001.01
5.0)(sin
5.0)(
yx
yx
xf
++
−+
+=
1000
)( 321 xxx
xf
⋅⋅−
=
[ ] )1()1(8.19)1()1(1.10)1()(90)1()(100)( 42
3
4
2
2
2
3
22
34
2
3
23
12 −⋅−⋅+−+−⋅+−+−⋅+−+−⋅= xxxxxxxxxxxf
3
1
23
12 )1()(100)( xxxxf −+−⋅=
10
)592.01044.0(
)(
3
21
3
1
3
2
3
1
−−−
⋅+⋅+⋅⋅
=
xxxxx
xf
7
2
32
4
1
10
0201.0
)(
xxx
xf
⋅⋅⋅−
=
10
)(
)100()1(
12
)(
4
23
2
2
2
1
2
1
2
22
1 





⋅
+⋅
+
+
++
=
xx
xx
x
x
x
xf
[35]
[35]
[35]
[35]
[35]
[10]
[10]
[10]
[10]
[10]
[10]
[29]
[29]
[29]
[29]
[29]
[2]
[2]
[2]
[2]
[2]
[2]
1.00
E+00

21
Table 2.2 shows the starting parameters for the PSO method whose values will be
varied during this parameter sensitivity study, and those values to be fixed during the
sensitivity study while another is varied. The major parameters used for the sensitivity
study are the population (i.e. the number of particles to put into the design space), the
dimension of the optimization problem, the maximum velocity each particle can have to
move in the design space, the maximum number of iterations before the algorithm
terminates or if it reaches the minimum error first. The range of values was chosen
arbitrarily although fixed parameter values used during sensitivity runs for the population,
the maximum iterations, the dimension of problem, the global/local, and inertia weights
were set as the same default values as was used in the original PSO research work [44].
It should be noted that the optimum values found in the sensitivity study can change for
each run, because of the random number generator built into the PSO method, thus, the
values shown will change. It should also be noted that, of the 8 parameters shown in
Table 2.2, only four of these have been found to have a significant impact on the solution
convergence. Thus these four will be the focus of this sensitivity analysis.
PSO Sensitivity Study
Parameter Values Used for Study Values Fixed During Study
Maximum Velocity (Vmax) 0.1 to 500 80
Swarm Population Size 1 to 128 40
Maximum Iteration 100 to 1000 1000
Dimension of Design Space 2 to 8 Depends on Function
Parameters Not Studied
Minimum Error 1E-5 to 100 Depends on Function
Global/Local 0.0
Inertia Weight 0.9 to 0.1 Linearly
Maximum Position 5.12 to 600 Depends on Function
Table 2.2 PSO sensitivity study range of values used

22
2.1.1 Velocity
Figures 2.1 through 2.5 show the results of sensitivity studies conducted on the
DeJong Test Functions. Generally, with the exception of the results for the Schaffer F6
and Griewank Functions, the study shows that the variation in velocity resulted in a
variation of optimum solutions. Even for these two test functions it is obvious that a zero
value choice for starting velocity is not a helpful choice. For the other test functions, the
results of the parameter study show a scattered sensitivity variation on the order of
magnitude of around 10. This performance illustrates both the strength and the weakness
of the PSO method, which is good in exploring the design space, but may be poor in local
convergence. This problem is illustrated in figures 2.1 through 2.5.
Spherical Function [29]
Fopt = 0
0.00E+00
1.00E-03
2.00E-03
3.00E-03
4.00E-03
5.00E-03
6.00E-03
7.00E-03
8.00E-03
9.00E-03
1.00E-02
0 100 200 300 400 500 600
Velocity
Fopt
Figure 2.1 Sensitivity of the PSO method to the velocity
parameter for the Spherical [29] benchmark test function

23
Griewank Function [29]
Fopt = 0
0.00E+00
2.00E+00
4.00E+00
6.00E+00
8.00E+00
1.00E+01
1.20E+01
0 100 200 300 400 500 600
Velocity
Fopt
Shaffer F6 Function [29]
Fopt = 0
0.00E+00
1.00E-04
2.00E-04
3.00E-04
4.00E-04
5.00E-04
0 100 200 300 400 500 600
Velocity
Fopt
parameter for the Griewank [29] benchmark test function
parameter for the Schaffer F6 [29] benchmark test function

24
Rosenbrock Function [29]
Fopt = 1.0
0.00E+00
2.00E+01
4.00E+01
6.00E+01
8.00E+01
1.00E+02
1.20E+02
0 100 200 300 400 500 600
Velocity
Fopt
Rastrigrin Function [29]
Fopt = 0
0.00E+00
5.00E+00
1.00E+01
1.50E+01
2.00E+01
2.50E+01
3.00E+01
3.50E+01
4.00E+01
4.50E+01
5.00E+01
0 100 200 300 400 500 600
Velocity
Fopt
In general, other than the Rosenbrock and Rastrigin functions, which have very
high error criterion, what we conclude about the velocity sensitivity analysis is that
parameter for the Rastrigin [29] benchmark test function
parameter for the Rosenbrock [29] benchmark test function

25
velocity values less than 50 are likely to give bad results while values between 50 and
500 seem to be generally equally useful.
2.1.2 Swarm Population Size
Figures 2.6 through 2.10 show the sensitivity study for the swarm population size
versus the optimum solution found. Generally, as the population size increases, it results
in better quality solutions; although, it is shown that at some larger population values, it
actually causes the optimum solution found to be worse compared to those of the small
population.
0.0000
0.0500
0.1000
0.1500
0.2000
0.2500
0.3000
0.3500
0.4000
0.4500
0.5000
0 20 40 60 80 100 120 140
Fopt
Population
Fopt=0
Figure 2.6 Sensitivity of the PSO method to the population

26
Fopt=1
0.0000
20.0000
40.0000
60.0000
80.0000
100.0000
120.0000
140.0000
160.0000
0 20 40 60 80 100 120 140
Population
Fopt
Fopt=0
0.0000
0.2000
0.4000
0.6000
0.8000
1.0000
1.2000
0 20 40 60 80 100 120 140
Population
Fopt

27
Rastrigin Function [29]
Fopt=0
0.0000
20.0000
40.0000
60.0000
80.0000
100.0000
120.0000
0 20 40 60 80 100 120 140
Population
Fopt

28
Schaffer F6 Function [29]
Fopt=0
0.0000
0.0020
0.0040
0.0060
0.0080
0.0100
0.0120
0 20 40 60 80 100 120 140
Population
Fopt
2.1.3 Iteration
Figures 2.11 through 2.15 show the sensitivity study for iteration versus the
optimum solution found. Generally, as the number of iterations increase beyond 100, it
resulted in only slightly better optimum solution convergence. Thus we conclude that the
solution sensitivity to iteration size is relatively low.

29
Fopt=0
0.0000
0.0500
0.1000
0.1500
0.2000
0.2500
0.3000
0.3500
0.4000
0.4500
0.5000
0 200 400 600 800 1000 1200
Iterations
Fopt
Fopt=1
0.0000
50.0000
100.0000
150.0000
200.0000
250.0000
300.0000
350.0000
400.0000
450.0000
500.0000
0 200 400 600 800 1000 1200
Iterations
Fopt
Figure 2.11 Sensitivity of the PSO method to the iteration

30
Fopt=0
0.0000
0.2000
0.4000
0.6000
0.8000
1.0000
1.2000
1.4000
0 200 400 600 800 1000 1200
Iterations
Fopt
Fopt=0
0.0000
10.0000
20.0000
30.0000
40.0000
50.0000
60.0000
70.0000
0 200 400 600 800 1000 1200
Iterations
Fopt

31
Fopt=0
0.0000
0.0050
0.0100
0.0150
0.0200
0.0250
0.0300
0.0350
0.0400
0 200 400 600 800 1000 1200
Iterations
Fopt
2.1.4 Dimension of the Design Space
Figures 2.16 and 2.20 show the sensitivity results for the number of dimensions in
the design space versus the optimum solution found. As expected, as the number of
dimensions gets higher, the result is worse optimum solution convergence. This is a
general characteristic of all optimization algorithms.

32
Fopt=0
0.0000
0.0500
0.1000
0.1500
0.2000
0.2500
0.3000
0.3500
0.4000
0.4500
0.5000
2 3 4 5 6 7 8
Dimension
Fopt
Fopt=1
0.0000
10.0000
20.0000
30.0000
40.0000
50.0000
60.0000
70.0000
80.0000
90.0000
2 3 4 5 6 7 8
Dimension
Fopt
Figure 2.16 Sensitivity of the PSO method to the dimension

33
Fopt=0
0.0000
0.0500
0.1000
0.1500
0.2000
0.2500
0.3000
0.3500
0.4000
2 3 4 5 6 7 8
Dimension
Fopt
Fopt=0
0.0000
20.0000
40.0000
60.0000
80.0000
100.0000
120.0000
2 3 4 5 6 7 8
Dimension
Fopt

34
Fopt=0
0.0000000
0.0000020
0.0000040
0.0000060
0.0000080
0.0000100
0.0000120
2 3 4 5 6 7 8
Dimension
Fopt
Parameter Sensitivity
Velocity Moderate to Low
Population High
Iteration Moderate to Low
Dimension Moderate to Low
Particle Swarm Optimization
Table 2.3 shows the summary of the PSO parameter sensitivity study. Maximum
velocity, maximum number of iteration and number of dimensions do not show high
sensitivity for all the benchmark problems, but, generally, optimum solutions get better as
these are increased. Population sensitivity shows high consistency among all benchmark
problems such that as it increases, the optimum solution converges faster. The question
remains about how large the population should be so as not to waste run time and the
number of merit evaluations. Therefore, this parameter will further be studied and
developed in chapter 4.
Table 2.3 PSO parameter sensitivity study summary

35
2.2 Other Considerations in Using the PSO methods
Figure 2.21 (after [6]) shows a common problem for the particle swarm method.
Upon reaching the convergence criteria (maximum number of iterations reached or
minimum error reached), there are particles shown in figure 2.21 that are outliers. This
means that they represent wasted time and wasted effort for the search process because
they do not add value or contribute meaning to the optimum found. Although there are
many positive attributes to the PSO method, this particular characteristic behavior is a
negative factor that must be endured if one is to use the PSO method. Since many other
optimization methods such as pattern search methods do not have this particular
disadvantage, it seems logical to try to combine the PSO method with some of these other
methods in an attempt to gain the best advantages of the combination of methods and to
limit the disadvantages of the combination. Such methods are known as hybrid methods.
Before we propose a new hybrid method, it is useful, for the sake of completeness, to
look at the parameter sensitivity of those other methods that we will ultimately use for
our innovative hybrid approach. Thus in Chapter 3 we will do a parameter sensitivity
analysis for two pattern search methods and the random search method. Then in Chapter
4 we will propose a new hybrid method based on the addition of these three methods to
the PSO method.
OutliersOutliers
Figure 2.21 Outliers in the
PSO swarm [after 6]

36
Chapter 3:
This chapter will present a parameter sensitivity study of the pattern search
methods used in the poly-hybrid PSO method. Specifically, the parameters used in the
Simplex and Hooke & Jeeves methods. These methods are both strong in local
convergence and are derivative-free methods. Section 3.3 will present the Random
search method used in the poly-hybrid PSO. This method is used to optimize the design
space; thus, saving extra merit evaluations and run time.
Parameter Sensitivity Study of the Pattern Search Methods
Pattern search methods do not require the gradient of the problem to be optimized
and thus are easy to use especially on functions that are not continuous or easily
differentiable. Pattern search methods are also known as direct-search, derivative-free, or
black box methods.
There are three pattern search methods that are traditionally used in optimization
studies. These are: the Nelder-Mead Simplex method [61], the Hooke & Jeeves method
[60], and the Rosenbrock Pattern Search Method [62]. Of these three, the Hooke &
Jeeves method is similar to the Rosenbrock method. Since we are hoping to assemble a
new hybrid method that has the greatest diversity of components, we have arbitrarily
elected to use the Hooke & Jeeves method rather than the Rosenbrock method. Thus for
purposes of this research work, we will perform the parameter sensitivity analysis for the
Hooke and Jeeves Method and the Nelder-Mead Simplex method. Each of these two
methods includes a set of parameters for the user to input before execution of the
algorithm to search for the optimum solution. Hooke & Jeeves has parameters to define
the design space, a parameter to define the initial exploration step size, a parameter to
define the acceleration factor, and a parameter to define the final exploration step size.
Nelder-Mead Simplex method has parameters to define the design space, to define the
minimum step size at convergence, to define the side length of the simplex, to define the

37
reflection coefficient, to define the contraction coefficient, and to define the expansion
coefficient [49].
A parameter sensitivity study will be conducted on each of these two pattern
search methods with the goal of better understanding how each parameter can affect each
algorithm in converging to an optimum solution. The Nelder-Mead Simplex method
sensitivity study was previously conducted by Fan and Zahara [46]. The study in this
thesis is an extension of their analysis by providing a larger range for the parameters to
understand better the behavior or pattern. In order to maintain consistency, the same
benchmark problems as Fan and Zahara [46] will be used in this study. The results of
this study will be used to create an automatic parameter adjuster algorithm for
manipulating the various input parameters during the hybrid method operation for the
purpose of enhancing and speeding convergence. This is explained in chapter 4. Let us
look first at the Nelder-Mead simplex method.
3.1 Nelder-Mead Simplex Method
Nelder-Mead (NM) Simplex Method is a popular and simple direct search
technique that has been widely used in unconstrained optimization problems [15 & 46].
It is derivative free; however, the solution outcome is very sensitive to the choice of
initial points and the method does not guarantee convergence to a global optimum. Fan-
Zahara [46] conducted a parameter sensitivity study using a select set of test cases to
suggest best parameter values based on the highest percentage rate of successful
minimization. Begambre-Laier [39] used a strategy to control the Particle Swarm
Optimization parameters based on the Nelder-Mead Simplex Method in identifying
structural damage. Although the suggested parameter values for the Nelder-Mead
Simplex Method resulted in good convergence, a better understanding of the parameter’s
behavior in producing the optimum solutions can be of great utility for current and future
research in using the NM Simplex method for hybrid or in creating an intelligent
adjustment algorithm. A sensitivity analysis was conducted previously by Fan and

38
Zahara [46] and it is the goal of this section to extend Fan-Zahara and Begambre-Laier’s
parameter sensitivity study by searching for a common pattern and provide better
understanding of the parameter’s behavior in producing optimum solutions.
Fan and Zahara did an excellent job in outlining the original NM Simplex
procedure and their work is documented in their classic paper [46]. The following list is
the equations from [after 46] that will be used in this investigation to support the
discussion of the results.
Reflection (α): Prefl = (1+α)Pcent – αPhigh (1)
Expansion (γ): Pexp = γPrefl – (1-γ)Pcent (2)
Contraction (β): Pcont = βPhigh – (1-β)Pcent (3)
Simplex Size (ζ): Pi = ζ Pi + (1- ζ)Plow (4)
Where,
Pexp is the new vertex generated for an expansion
Pcont is the new vertex generated for a contraction
Pi is the new vertex generated for size
Prefl is the new vertex generated for a reflection
Pcent is the center vertex of a simplex
Phigh is the highest value vertex of a simplex
Plow is the lowest value vertex of a simplex
α is the reflection coefficient
γ is the expansion coefficient
β is the contraction coefficient
ζ is the size of the simplex
The above mathematical equations (1-4) show how the four parameters affect the simplex
movement in the design space.

39
3.1.1 Computational Results & Design of Experiments
In this section, the sensitivity study of the NM Simplex Search Method
parameters will be explored and the empirical results reported.
This study uses the same assumptions specified in Fan-Zahara’s paper [46] for
stopping criterion, for the initial starting points, and for the coefficient intervals. The
numerical computations for this study have been done using a Dell Inspiron laptop with
Pentium dual core CPU and 64-bit operating system. The code was prepared using
Visual Basics for Excel.
3.1.2 Parameter Sensitivity Study of NM Simplex Search Method
In this section, the parameter sensitivity of the NM Simplex method will be
studied using the same benchmark functions used in Fan-Zahara’s paper [46] to keep the
comparison of results consistent. This study is conducted by using the original
coefficients for the NM simplex method listed in Fan-Zahara’s paper [46]. These
coefficients are listed below for convenience.
Reflection (α) is 1.0
Expansion (γ) is 2.0
Contraction (β) is 0.5
Simplex Size (ζ) is 0.5
For this sensitivity study, the ranges of parameters are listed in Table 3.1. Each of
the four NM simplex parameters is varied according to the ranges given in Table 3.1
while keeping the other three parameters fixed at the original coefficients values. For
example, in the sensitivity study for reflection coefficient (α), it is varied from 0.50 to
2.00 with an increment of 0.25 while keeping the expansion (γ) at 2.0, contraction (β) at
0.5, and simplex size (ζ) at 0.5.

40
RangeSimplex
Parameters Minimum Maximum
Increment
Reflection (α) 0.50 2.00 0.25
Contraction (β) 0.25 0.75 0.25
Expansion (γ) 1.50 3.00 0.25
Size of Simplex (ζ) 0.25 0.75 0.25
# Test Function Dimesions Global Optimum Value Function
1 B2 2 0
2 Beale 2 0
3 Booth 2 0
4 Wood 4 0
5 Extended Rosenbrock 4 0
6 Rastrigin 4 0
7 Sphere 4 0
7.0)4cos(4.0)3cos(3.02)( 21
2
2
2
1 +⋅⋅⋅−⋅⋅⋅−⋅+= xxxxxf ππ
∑=
−−=
3
1
2
21 ))1(()(
i
i
i xxyxf
2
21
2
21 )52()72()( −++−+= xxxxxf
2
42
2
1
2
42
2
1
2
3
22
34
2
1
2
1
22
12 ))()10(())2()10(()1())()90(()1())10()( xxxxxxxxxxxf −+−++−+−+−+−=
−
∑=
−− −+−=
2
1
2
12
22
122 ))1()(100()(
i
iii xxxxf
∑=
+−=
10
1
2
)10)2cos(10()(
i
ii xxxf π
∑=
=
30
1
2
)(
i
ixxf
3.1.3 Results
Figure 3.1 shows the sensitivity results for the reflection coefficient (α). It shows
that the optimum solution quickly converges to the true value when the reflection
coefficient (α) is greater than 0.75 for all test cases. For coefficient values greater than
0.75, the optimum solution does not result in larger oscillations compared to values
between 0.5 and 0.75. This agrees with Fan and Zahara’s paper [46], which suggested
that the best value is 1.5, while the original value suggested for the NM Simplex method
Table 3.1 Parameter sensitivity ranges
Table 3.2 List of test functions [after 46, 51]
[46]
[46]
[46]
[46]
[46]
[46]
[46]

41
is 1.00. This further suggests that the reflection coefficient (α) can impact the optimum
solution greatly by increasing the value to 0.75 or greater. This makes sense because
from equation (1), this means that the value for the generation of the new vertex, Prefl, has
to be larger to help the NM simplex method to expand further into the design space in
search of the true solution. The average and standard deviations for the reflection
coefficient (α) are found as 1.29 and 0.466 as shown in table 3.3.
Reflection vs. Optimum Value
1.00E-09
1.00E-07
1.00E-05
1.00E-03
1.00E-01
1.00E+01
1.00E+03
1.00E+05
1.00E+07
1.00E+09
1.00E+11
1.00E+13
0.50 0.70 0.90 1.10 1.30 1.50 1.70
Alpha
OptimumValue(LogScale)
Wood Rastrigin Sphere Rosenbrock Beale B2 Booth
NM Simplex
Coefficient
Original NM Simplex Fan and Zahara Wang and Shoup
Reflection (α) 1.00 1.50
Average = 1.29,
σ = 0.47
Expansion (γ) 2.00 2.75
Average = 2.29,
σ = 0.44
Contraction (β) 0.50 0.75
Average = 0.47,
σ = 0.17
Simplex Size (ζ) 0.50 0.50
Average = 0.57,
σ = 0.19
Table 3.3 Comparisons of suggested coefficients for NM Simplex Method [51]
Test Function [46, 51]
Fopt = 0
Fopt
Figure 3.1 Sensitivity of the simplex optimization method to the reflection
coefficient (α) for the seven benchmark test functions [46, 51]

42
Figure 3.2 shows the sensitivity results for the contraction coefficient (β). It shows that
the optimum solution quickly converges to the true value when the contraction coefficient
(β) is greater than 0.5 for all test cases. This agrees with Fan and Zahara’s paper [50],
which suggests that the best value is 0.75, while the value for the original NM simplex is
0.5. This suggests that the contraction coefficient (β) can impact the optimum solution
greatly by increasing the value to 0.5 and greater. This makes sense because from
equation (3), when the value of the generation of the new vertex, Pcont, for contraction
coefficient (β) is larger, more flexibility can be provided to expand in the design space by
increasing the contraction coefficient (β). The average and standard deviation found for
the contraction coefficient (β) are 0.47 and 0.17 as shown in table 3.3. Table 3.4 shows a
summary of the impact parameters used to study the sensitivity of the Simplex method.
Wang and Shoup NM Simplex Method Sensitivity Studies
Reflection (α) 0.5 to 2.0 A=0.5, Beta=0.5, Gamma=2
Size ( ζ ) 0.25 to 0.75 Alpha=1, Beta=0.5, Gamma=2
Expansion (γ) 1.5 to 3 Alpha=1, Beta=0.5, A=0.5
Contraction (β) 0.25 to 0.75 Alpha=1, A=0.5, Gamma=2
Starting Point for the Search 5.12 to 600 Depends on Function
Test for Minimum Step Size at Convergence 1E-5 to 100 Depends on Function
Table 3.4 Wang and Shoup NM Simplex Method sensitivity studies [51]

43
Contraction vs. Optimum Value
1.00E-09
1.00E-07
1.00E-05
1.00E-03
1.00E-01
1.00E+01
1.00E+03
1.00E+05
1.00E+07
0.25 0.35 0.45 0.55 0.65 0.75
Beta
Figure 3.3 shows the sensitivity results for the size parameter (ζ) of the simplex
coefficient. It shows that the optimum solution quickly converges to true value when the
size parameter coefficient (ζ) is greater than 0.5 for all test cases. This agrees with Fan
and Zahara’s paper [46], which suggested that the best value is 0.5, which also is the
suggested value from the original work of Nelder and Mead. This suggests that the size
parameter (ζ) of the Simplex method; although, not changing the optimum solution too
greatly compared to the reflection (α) and contraction (β) coefficients can contribute to
solution progress when the values are 0.5 or greater. This also makes sense because from
equation (4), with the size parameter (ζ) of the simplex, Pi, increasing to a larger size
coefficient (ζ), allows more flexibility to move in the design space. The average and
standard deviation for this parameter are 0.57 and 0.19 as shown in Table 3.3.
Fopt = 0
Fopt
Figure 3.2 Sensitivity of the Simplex Optimization Method to the contraction coefficient
(β) parameter for the seven different benchmark test function [46, 51]

44
Simplex Size vs. Optimum Value
1.00E-09
1.00E-08
1.00E-07
1.00E-06
1.00E-05
1.00E-04
1.00E-03
1.00E-02
1.00E-01
1.00E+00
1.00E+01
0.25 0.35 0.45 0.55 0.65 0.75
Simplex Size
Figure 3.4 shows the sensitivity results for the expansion coefficient (γ). This
figure shows that the optimum solution does not oscillate as much as the reflection (α)
and contraction (β) coefficients when varying the values. It also shows that the optimum
solution converges better when the expansion coefficient is greater than 2.0. This again
agrees with Fan and Zahara’s paper [46], which suggested the best value, is 2.75, while
the original value recommended for the NM Simplex is 2.0. This conclusion also makes
sense because from equation (2), the larger the expansion coefficient (γ), the larger the
extension of the search space for each exploration, Pexp. The average and standard
deviation found are 2.3 and 0.44 as shown in table 3.3.
Fopt = 0
Fopt
Figure 3.3 Sensitivity of the Simplex Optimization Method to the size (ζ)
parameter for the seven different benchmark test function [46, 51]

45
Expansion vs. Optimum Value
1.00E-09
1.00E-08
1.00E-07
1.00E-06
1.00E-05
1.00E-04
1.00E-03
1.00E-02
1.00E-01
1.00E+00
1.00E+01
1.75 1.95 2.15 2.35 2.55 2.75 2.95
Gamma
3.2 Hooke & Jeeves Method
Hooke & Jeeves [39] is an easily programmed climbing technique that does not
require calculation of derivatives. The algorithm has ridge following properties and is
based on the premise that any set of design moves that have been successful in early
experiments is worth trying again. This method assumes unimodality and is used to find
the minimum of a multivariable unconstrained function of the form
Merit=F(X1, X2, X3,.….Xn,).
The method uses a pattern move procedure and extrapolates along a line between
a new base point and the previous base point. The distance moved beyond the best base
point is somewhat larger than the distance between the base points by a factor of δ. This
factor is called an acceleration factor and is recommend to be equal to or greater than 1.0.
3.2.1 Computational Results, Design of Experiments, Parameter Sensitivity Study
of Hooke & Jeeves Method
Fopt
Figure 3.4 Sensitivity of the Simplex Optimization Method to the expansion
(γ) parameter for the seven different benchmark test function [46, 51]
Fopt = 0

46
In this section, the sensitivity study of the Hooke and Jeeves (HJ) Search Method
parameters will be explored and the empirical results reported. This study uses a Dell
Inspiron laptop with Pentium dual core CPU and 64-bit operating system. Coding is
done in Visual Basics for Excel.
The Hooke & Jeeves method has one important parameter that affects the
optimum solution. This is the acceleration factor (δ). Depending on the acceleration
factor (δ), this can either overshoot or undershoot the optimum solution, so there is great
value in studying this parameter. The other parameters, starting points, initial exploration
step size, and the final exploration step size, are usually determined by the user or by
experience gained from executing this algorithm, so, these variable will not be studied.
3.2.2 Results
Figures 3.5 to 3.9 show plots of the optimum solution versus acceleration factor (δ)
for the DeJong Test Cases. Figure 3.5 shows how the optimum solution for the Spherical
Function is affected by varying the acceleration factor (δ). It is seen that the best value
for the acceleration factor (δ) for this test case is around 1.1, 1.3, 1.8, or 1.9 and the worst
is around 1.7. Figure 3.6 shows how the optimum solution for the Griewank Function is
affected by varying the acceleration factor (δ). It is seen that the best value for the
acceleration factor (δ) for this test case is around 2 and the worst is around 1.7. Figure
3.7 shows how the optimum solution for the Rastrigin Function is affected by varying the
acceleration factor (δ). It is seen that the best value for the acceleration factor (δ) for this
test case is around 1.7 and the worst is around 1.3. Figure 3.8 shows how the optimum
solution for the Rosenbrock Function is affected by varying the acceleration factor (δ). It
is seen that the best value for the acceleration factor (δ) for this test case is around 1.1,
1.2, 1.5, 1.7 and the worst is around 1.9. Figure 3.9 shows how the optimum solution for
the Schaffer F6 Function is affected by varying the acceleration factor (δ). It is seen that

47
the best value for the acceleration factor (δ) for this test case is around 1.8 and the worst
is around 1.5.
Spherical-HJ
0.00E+00
2.00E-20
4.00E-20
6.00E-20
8.00E-20
1.00E-19
1.0 1.2 1.4 1.6 1.8 2.0
Acceleration Factor
OptimumSolution
Griewangk-HJ
0.0000E+00
5.0000E+01
1.0000E+02
1.5000E+02
2.0000E+02
2.5000E+02
3.0000E+02
3.5000E+02
4.0000E+02
1.0 1.2 1.4 1.6 1.8 2.0
Acceleration Factor
OptimumSolutionFopt
Fopt
Fopt = 0
Fopt = 0
Figure 3.5 Sensitivity of the Hooke & Jeeves optimization method to the
acceleration factor (δ) for the Spherical benchmark test function [29]
acceleration factor (δ) for the Griewank benchmark test function [29]

48
Rastrigin-HJ
0.0000E+00
1.0000E-01
2.0000E-01
3.0000E-01
4.0000E-01
5.0000E-01
6.0000E-01
7.0000E-01
1.0 1.2 1.4 1.6 1.8 2.0
Acceleration Factor
OptimumSolution
Rosenbrock-HJ
0.0000E+00
5.0000E+02
1.0000E+03
1.5000E+03
2.0000E+03
1.0 1.2 1.4 1.6 1.8 2.0
Acceleration Factor
OptimumSolution
Fopt
Fopt
Fopt = 0
Fopt = 1
acceleration factor (δ) for the Rastrigin benchmark test function [29]
acceleration factor (δ) for the Rosenbrock benchmark test function [29]

49
Fopt=0
1.5260007040E-01
1.5260007045E-01
1.5260007050E-01
1.5260007055E-01
1.5260007060E-01
1.5260007065E-01
1.5260007070E-01
1.5260007075E-01
1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0
Acceleration Factor
Fopt
It was found that no consistent relationship exists among the test cases, but it
shows how the optimum values vary randomly based on the acceleration factor (δ)
chosen for this method. Because of the lack of consistent relationship, it makes a strong
case for the benefit of automatically adjusting itself per iteration.
Hooke & Jeeves Method Sensitivity Studies
Acceleration Factor (δ) 1.1 to 2.0 Initial Exploration Step Size=1.1
Starting Point for the Search 5.12 to 600 Depends on Function
Initial Exploration Step Size 1.1
Final Exploration Step Size 1E-5 to 100 Depends on Function
Table 3.5 Hooke & Jeeves method sensitivity studies
acceleration factor (δ) for the Schaffer F6 benchmark test function [29]

50
3.3 The Random Search Method
The random search method [49] is the fourth method that will be employed to
develop the poly-hybrid method in Chapter 4 that follows. The parameters used for the
random search method are the upper and lower bounds, which define the design space,
and the minimum error used to terminate the algorithm specified by the user. In most
engineering problems, the designer likes to specify the lower and upper boundaries and
the minimum error from customer’s requirements and let the Random Search Method
find a first order optimum value for preliminary design. Thus, a parameter sensitivity
study for the random search method is not as useful and will not be done at this point.
Nevertheless, the Random Search Method can serve a valuable role in a poly-hybrid
method in that it helps the PSO part of the composite algorithm to have the best initial
design space before the poly-hybrid method begins its composite search process.

51
Chapter 4:
The Poly-Hybrid PSO Method
Since the initial development of the particle swarm optimization method (PSO) in
1995, there have been a number of improvements in its deployment and numerical
implementation. These have included improvements in initial particle selection [6],
improvements in choosing velocities for particle movement [8], and improvements in
overall schemes for the evaluation of convergence [7]. Recently, a few researchers have
extended the utility of the PSO method by combining it with one or two other direct
search algorithms that are known to have broad utility [9]. The success of these efforts
shows much promise; however, the hybrid approach has two minor disadvantages.
First, since each direct search method has its own inherent advantages and unique
capabilities, the addition of multiple methods adds robustness to the computational
versatility of the hybrid algorithm, but at the same time adds complexity to the
optimization process and will thus expand the size and complexity of the implementation
code.
Second, since each of the traditional direct methods has its own unique parameter
set to adjust factors that determine such things as initial step size, convergence
adjustment factors, acceleration factors, deceleration factors as a solution is approached,
and other factors unique to individual algorithms, it seems reasonable to assume that
there is a need for an intelligent scheme for tracking and adjusting these factors. This
also will add complexity to the implementation code.
Thus it is the purpose of this research to show that combining more than two
direct search methods with the PSO method (i.e. a poly-hybrid method) can produce
better results than would be possible by adding only one direct search method (i.e. a
mono-hybrid method). This hypothesis will be validated in what now follows. It is also

52
the purpose of this research to show how intelligence based scheme can be employed to
track the search parameters needed for various methods in a way that allows optimal
choices for required parameter values as the search process progresses. This also will be
presented in what now follows in this dissertation. It is believed that these two
contributions to the state of knowledge in optimization methodology constitute an
important enhancement to the field of optimization. This chapter will be devoted to
explaining how the new poly-hybrid PSO method is structured to accomplish the two
improvements that are being sought. Benchmark tests are provided to that show how well
this new algorithm performs.
Before we explain the assembly of the poly-hybrid method, it is useful to look
briefly at recent trends in optimization algorithms.
4.1 Current Search Algorithms for Optimization
The algorithms and hybrid methods shown in Table 4.1 are most frequently
acknowledged in the literature to be best choices for overall solution efficiency for
constrained and unconstrained optimization.

53
Method Name/ Description
Researchers, date and reference
number
PSO Particle Swarm Optimization Method Eberhart & Kennedy, 1995, [19]
CPSO Constriction Particle Swarm Optimization Method Eberhart & Shi, 2000, [20]
GCPSO
Guaranteed Convergence Particle Swarm Optimization
Method
Van den Bergh, 2001, [53]
FIPS Fully Informed Particle Swarm Method Kennedy & Mendes, 2003, [21]
HJ Hooke and Jeeves Pattern Search Method Hooke & Jeeves, 1961, [38]
Simplex Simplex Method Pattern Search Method Nelder & Mead, 1965, [14]
PSO-Simplex Bi-Hybrid PSO and Simplex Fan & Zahara, 2007, [46]
PSO-Hooke &
Jeeves
Bi-Hybrid PSO and Hooke & Jeeves Tseng & Lin, 2009, [28]
What this table clearly shows is that the latest trend in optimization algorithms is
that combined methods have considerable promise in advancing the state of the art in this
field.
4.2 The Proposed Poly-Hybrid PSO Method
After experimenting with various combinations of these recent methods described
in the previous section, this research determined that a very promising poly-hybrid
method can be assembled by using the following four methods: PSO, H&J, Random
Search, and Nelder-Mead Simplex. The reasoning in assembling these four methods is as
follows:
1. The PSO method’s strength in global exploration and simplicity in code,
Table 4.1 A listing of frequently acknowledged best choice methods for optimization

54
2. The use of the two popular and best known pattern search methods requiring no
derivatives, which are simple and efficient, and capable of moving along any
surface with rapid local convergence,
3. The random method’s ability to shrink the design space per iteration, resulting in
saving run time and merit evaluations.
These combinations resulted in having a composite method that has both global
and local convergence strengths and have good capabilities in reducing the design space
to save run time and merit evaluations.
Figure 4.1 shows the flow diagram of the poly-hybrid method.
In general, the central control serves as the “intelligence” of the poly-hybrid
method. It controls, manages, and processes the data (e.g., minimum error and optimum
value found) passed from the four methods. Based on the inputs it receives, it makes a
CENTRAL CONTROL
PSO RANDOM
HJ SIMPLEX
MAP KEY
INFO FLOW FROM CENTRAL TO METHOD
INFO FLOW FROM METHOD TO CENTRAL
CENTRAL CONTROL
PSO RANDOM
HJ SIMPLEX
MAP KEY
MAP KEY
Figure 4.1 Poly-hybrid flow-chart showing the information flow from the
central controller to each method and vice versa [50]

55
decision on the best course of action to enhance convergence (e.g., going back to a
particular method for better convergence, for updating parameters, etc.), and for passing
back these decisions to the respective methods so that convergence can be improved.
This “intelligence” process is continued until all four methods are exhausted (i.e., none of
the four methods can result in better convergence as seen when the minimum error no
longer changes or when the maximum number of iterations has been reached.
In figures 4.2 and 4.3, graphical examples of the poly-hybrid method are
presented to show how this works.
X
f(X)
Random
Nelder Mead Simplex
Hooke & Jeeves
PSO
Initial Final NM
Simplex Value
Initial Final
HJ Value
Initial Final
Random Value
Initial Final PSO
Value
Design Space
Initial Start Value
X
f(X)
Random
Nelder Mead Simplex
Hooke & Jeeves
PSO
Random
Nelder Mead Simplex
Hooke & Jeeves
PSO
Initial Final NM
Simplex Value
Initial Final
HJ Value
Initial Final
Random Value
Initial Final PSO
Value
Design Space
Initial Start Value
Figure 4.2 Poly-hybrid graphical representation (initial) [50]

56
X
f(X)
Random
Nelder Mead Simplex
Hooke & Jeeves
PSO
Final NM Simplex Value
Final HJ Value
Final
Random
Value
Final PSO Value
New Design Space
New Start for HJ,
NM, Random
Particle Deleted b/c Outside Band
Final Design Space
X
f(X)
Random
Nelder Mead Simplex
Hooke & Jeeves
PSO
Random
Nelder Mead Simplex
Hooke & Jeeves
PSO
Final NM Simplex Value
Final HJ Value
Final
Random
Value
Final PSO Value
New Design Space
New Start for HJ,
NM, Random
Particle Deleted b/c Outside Band
Final Design Space
Figure 4.2 shows the poly-hybrid method during the first iteration. Initially, after
the user enters the required inputs (e.g. problem dimension, global/local coefficient,
maximum velocity, inertia weight, maximum number of iterations, minimum error,
maximum position), each of the four methods, starting from the same initial values given
by the user run independently and produces the best minimum error and optimum
solutions found by the method per iteration and passed to central control for decision
making and processing. Figure 4.4 shows an overview flowchart of how the poly-hybrid
method works.
The central control, shown in figure 4.5, compares the minimum error and
optimum solutions found from each method per iteration cycle and based on which is
least, it makes the decision to see which of the methods to use for the second, third,
fourth, etc. iteration cycles to gain further improvement in the solution. If the choice is to
proceed with the Hooke and Jeeves or Simplex methods, the controller will first adjust
the HJ & Simplex parameters per iteration cycle either by increasing or decreasing the
parameters by 10% and using the best location found from the previous iteration as the
new starting point. The controller will proceed with the method chosen and will give
Figure 4.3 Poly-hybrid graphical representation (final) [50]

57
back another optimum solution and minimum error found during that iteration cycle.
This new location and design space will be passed back to the central control for
evaluation and decision for the next method for further improvement. If the choice is to
use the PSO method, the controller will use the best location found from the previous
iteration as the new starting point, and the particles will be redistributed about that point
by the random generator in the velocity equation. The PSO method will then be rerun
and a new optimum solution and minimum error will be found and returned to the central
controller. This process continues until either the maximum number of iterations is
expended or until the prescribed degree of convergence is reached.
Figures 4.4-4.9 show the flow charts and block diagrams of the poly-hybrid PSO
method, central controller, each of the four methods used in the poly-hybrid PSO method
in detail.
Figure 4.4 Overview of the poly-hybrid PSO flowchart [50]
User’s Input to Poly-Hybrid Method
Initial
minimum error
set for Simplex,
HJ, PSO,
Random
Initial HJ and
Simplex
parameters value
are randomly set
Using sampling
formula to
calculate
population size
Call PSO Call Random
Call Hooke &
Jeeves
Call Simplex
Return
Minimum Error
Produced
Return
Maximum
Position
Return
Minimum Error
Produced
Return
Minimum Error
Produced
Central Control
Maximum number of
iterations reached or
minimum error reached?
STOP
YES
NO
Initial
minimum error
set for Simplex,
HJ, PSO,
Random
Initial HJ and
Simplex
parameters value
are randomly set
Using sampling
formula to
calculate
population size
Initial
minimum error
set for Simplex,
HJ, PSO,
Random
Initial HJ and
Simplex
parameters value
are randomly set
Using sampling
formula to
calculate
population size
Call PSO Call Random
Call Hooke &
Jeeves
Call Simplex
Return
Minimum Error
Produced
Return
Maximum
Position
Return
Minimum Error
Produced
Return
Minimum Error
Produced
Central Control
Maximum number of
iterations reached or
minimum error reached?
STOP
YES
NO

58
Figure 4.5 Detail of the central control flowchart [50]

59
Figure 4.2, shows the initial convergence results of the NM Simplex, HJ, and
Random methods that produced values at local optimums, but not global, and are stalled
at these solutions. These are typical, because local methods that act independently tend
to get stalled on local optimum. The optimum solution found for the PSO method is near
the global optimum, but the method has difficulty landing on a local optimum. It seems
reasonable that a combination of these methods might help this situation as explained in
the following paragraph.
Figure 4.2 shows the poly-hybrid method after the first iteration.
The central control module determines by using the decision making process outlined in
figure 4.5, the best optimum value and minimum error found in figure 4.2 to be that from
the PSO method, so, the original initial values for Random, HJ, NM Simplex are now set
to the new values the PSO method had in figure 4.3. Then the method again proceeds
according to figures 4.6 through 4.9 using these new inputs with new design space
determined from what was learned in the first iteration, to reduce the original space even
further. The result is saved run time and reduced merit evaluations needed.
Figure 4.6 Detail of the PSO flow-chart [50]

60
The design space continues to be reduced whenever the poly-hybrid method,
through the central controller, finds that it is appropriate. Because of the PSO method’s
Figure 4.7 Details of the Random flow-chart [50]
Figure 4.8 Details of the Simplex flow-chart
[50]
Figure 4.9 Details of the HJ flow-chart [50]

61
strength in global exploration, it has the unique ability to move the random, NM Simplex
and HJ methods away from places in the design space where the search is stalled. When
these conditions exist the algorithm will encourage a local exploration in a new, reduced
part of the design space away from the stall at the local optimum. The result is better
convergence. Thus the best characteristics of the PSO method can be used to augment
the shortcomings of the other three methods and visa versa. So, with these methods
together, the result is finding the true optimum solution using the logic found in figure
4.3.
The poly-hybrid method blends together the robustness of the PSO method, the
simplicity of the H&J method, the multimodality capability of the random search method,
and the acceleration capabilities of the simplex method. Since this poly-hybrid method
combines more than two traditional algorithms, it satisfies the first of the criteria
proposed in the introduction to this chapter. The central block, as explained in the above
paragraph, is where the decision is made about which of the multi-methods will have
control at any given time. The outer blocks of the figure show where decisions are made
concerning parameter selection and where decisions are made about whether to continue
with this algorithm or to return to the central control to index to another method.
Figures 4.4-4.9 show flowcharts of the poly-hybrid PSO method. The decision of
a particular algorithm will be accomplished in the central block, figure 4.5, of this
diagram and will be based on the following logic:
1. Based on the user’s input (e.g. problem dimension, global/local coefficient,
maximum velocity, inertia weight, maximum number of iterations, minimum
error, maximum position), it sets final minimum error cut-off values by
multiplying user defined minimum error by 10%.
2. Initial minimum error cut-off for Simplex, HJ, PSO, and Random methods use the
user defined minimum error input.
3. Initial HJ and Simplex parameters values are randomly set.

62
4. Initial population size is calculated by using Six Sigma [40] sampling formula for
values at 99% confidence.
5. The PSO method is called and the result is a minimum error produced after
running the method.
6. The Random method is called and the result is a new maximum position. The
minimum error is passed to the central control.
7. The HJ method is called and the result is the minimum error produced after
running the method.
8. The Simplex method is called and the result is the minimum error produced after
running the method.
9. The various minimum errors are compared after running the PSO, Random, HJ
and Simplex methods.
10. The HJ or Simplex parameters are modified by either increasing or decreasing
these by 10% and that specific method is called to re-evaluate.
11. The algorithm stops when the maximum number of iterations has been reached or
the final minimum error is achieved.
The use of a central controller to manage search methods has never been tried
before in research. Current research studies [57-59] use a bi-hybrid approach involving
only two methods, which bounce between each other to produce results. In the poly-
hybrid method the central controller, after a preliminary run of the methods with fixed
iterations and runtime, is able to pick, based on the minimum errors found in each
method, which method to use to further improve convergence. This approach helps to
save run time by saving merit evaluations, because the controller knows based on the
previous results, which method is best to use for further improvements. Thus, this method
is often successful in producing good results where bi-hybrid methods can fail. The
central controller automatically updates the parameters used in the HJ and Simplex
methods per iteration cycle by either increasing or decreasing the values by 10%, and
uses the sampling formula for values to update population size per run. It eliminates dead

63
weight particles; it further improves convergence, and ultimately reduces the number of
evaluations and thus the overall run time.
The decisions about parameter selection will be made as follows:
1. Based on the minimum errors found from the preliminary run of the random, PSO,
HJ, and Simplex methods, the controller compares and evaluates performance.
2. If the HJ and/or Simplex methods can be improved further, the controller will
increases or decrease parameters, depending on the current values, by 10%.
3. This new set parameter values will be sent to the HJ and/or Simplex methods for
re-evaluations.
4. Once step 3 is done, the decision process goes back to step 1.
The decision about termination will be made as follows:
1. A minimum error defined initially by the user has been reached.
2. This minimum error is preliminary used for all methods as a stopping criterion.
3. The final minimum error is the user-defined value multiplied by 0.10.
4. The poly-hybrid method will stop when it reaches this final minimum error or
when the number of maximum iterations specified by the user is met.
With this decision process, and the application of the intelligence module to each
sub-algorithm, the second criterion proposed in the introduction to this work is now
satisfied.
Let us now look at how well the poly-hybrid method performs when applied to
standard benchmark test problems. The software program to be used will be Visual
Basics for Excel.

64
4.3 Classic Benchmark Problems
The classical benchmark tests [2&29] previously described in Table 2.1 are now
applied to the poly-hybrid method to demonstrate its performance capabilities when
compared with other methods. Only the Spherical and Griewank test function results will
be posted in this thesis, since other test functions show similar behaviors. Reference [50]
shows the results for all the test functions.
4.4 Experiment and Results
As stated earlier in this dissertation, the problems were run on a Dell Inspiron
laptop with Pentium dual core CPU and 64-bit operating system. Coding is done in
Visual Basics for Excel.
4.4.1 Experiment
The following shows the results of the nine algorithms tested with eleven classical
benchmark problems. The objective function, constraints, minimum error, and parameter
range used for each algorithm is reported from previous published papers [2] and [29]
and listed again in table 2.1. In each figure, the case number, test case name, optimum
value found for the problem, difference between the true published value versus the
algorithm’s answer, number of merit evaluations it took to get the solution, and run time,
are shown in each of the tables shown. The random number generator used to generate
random positions and velocities uses the same seed to minimize any bias in the solution.
This type of random number generator is called sequential random sample and for
example, if a seed of 3 is assigned, every tenth number starting with 3 will be selected, so,
3, 13, 23, 33, etc. It has the advantage of being quick and easy to explain for the results
of this work.

65
There were 9 algorithms tested using 11 classical benchmark problems. These 9
algorithms are the PSO, GCPSO, CPSO, FIPS, Simplex, HJ, PSO-Simplex, PSO-HJ, and
Intelligent Poly-Hybrid PSO.
4.4.2 Results
Figures 4.10 to 4.15 show the delta comparison, merit evaluations, and run time,
respectively, for all 9 algorithms tested in this paper using the Spherical and Griewank
Test Functions. Delta is the difference between the true merit values of each benchmark
test functions and the average merit values produced using each algorithm.
Figure 4.10 Algorithm and true merit value
difference comparison (Spherical) [50]

66
Figure 4.11 A comparison of merit evaluations required to achieve an
acceptable solution for the PSO method when solving the nine test
function (Spherical) [50]
Figure 4.12 A comparison of run time required to achieve an acceptable solution for
the PSO method when solving the nine test functions (Spherical) [50].

67
Figure 4.14 A comparison of merit evaluations required to achieve an acceptable
solution for the PSO method when solving the nine test function (Griewank) [50]
Figure 4.13 Algorithm and true merit value
difference comparison (Griewank) [50]

68
The poly-hybrid method performed the best in both test functions in terms of delta
comparison. It is one to two orders of magnitude better than the runner-up PSO-Simplex
method and the PSO-HJ hybrid method, which performed third best. The results shown
for delta comparison reinforce the accepted notion that hybrids typically perform better
than methods that are not hybrids, in this case, the PSO, CPSO, FIPS, HJ, GCPSO, and
Simplex. The reason is that these six other methods lack strengths in local convergence
that hybrid methods provide. In addition, it is shown that the use of additional methods
in the hybrid, or poly-hybrid approach further improves convergence as shown for the
PSO-HJ-NM-Random method. It should be noted, however, that this convergence
improvement results in a sacrifice in run time and merit evaluations as can be seen in
figures 4.11, 4.12, 4.14 and 4.15.
Figures 4.16 to 4.20 show individual performances of the Hooke & Jeeves,
Simplex, Random, and Particle Swarm Optimization and the Poly-Hybrid Method, for the
Griewank Function.
Figure 4.15 A comparison of run time required to achieve an acceptable solution for the
PSO method when solving the nine test function (Griewank) [50]

69
HJ Performance on Griwank Function
60
65
70
75
80
85
90
95
100
105
110
115
120
125
130
135
140
145
150
155
160
0 50 100 150 200 250
# of Tim es Merit Function Evaluation
OptimumValueFound
Figure 4.16 Individual performance of the Hooke & Jeeves Method using the
Griewank Function between merit evaluations and merit values found [50]
NM Performance on Griwank Function
2
2.5
3
3.5
4
4.5
5
5.5
6
6.5
7
0 50 100 150 200
# Tim es Merit Function Evaluation
OptimumValueFound
Figure 4.17 Individual performance of the Simplex Method using the Griewank Function
between merit evaluations and merit values found [50]
Fopt = 0
Fopt = 0
FoptFopt

70
Random Performance on Griwank Function
0
20
40
60
80
100
120
140
160
180
200
0 50 100 150 200
# Times Merit Function Evaluation
OptimumValueFound
PSO Performance on Griwank Function
-20
0
20
40
60
80
100
120
0 50 100 150 200 250
# of Tim es Merit Function Evaluation
OptimumValueFound
Figure 4.18 Individual performance of the Random Method using the
Griewank Function between merit evaluations and merit values found [50]
Figure 4.19 Individual performance of the PSO Method using the Griewank
function between merit evaluations and merit values found [50]
Fopt = 0
Fopt = 0
FoptFopt

71
Figures 4.16 to 4.18 shows the Hooke & Jeeves, Simplex, and Random methods;
although, trying to converge to the true optimum value have the well known problem
with local search methods of getting stalled at a local optimum. It also demonstrates its
strength in local convergence. Figure 4.19 shows that the Particle Swarm Optimization
Method is not converging throughout the run, but rather is oscillating about a mean value
around 30, which is well away from the true optimum value, but demonstrates its strength
in being able to move in the design space. Figure 4.20 shows that the poly-hybrid
method, combines the strength of each method together to cover the weakness of others.
This method is able to use the PSO to move in the design space, and the Hooke & Jeeves,
Random and Simplex methods for local convergence strengths near to the global
optimum. This results in convergence to the true optimum value of 0 for the Griewank
Function.
Poly-Hybrid Performance on Griewank Function
-1
0
1
2
3
4
5
6
7
8
9
0 20 40 60 80 100 120 140 160 180 200
# of Times Merit Function Evaluated
OptimumValueFound
Figure 4.20 Performance of the poly-hybrid PSO Method using the Griewank
function between merit evaluations and merit values found [50]
Fopt = 0
Fopt

72
4.5 A guideline for choosing the right algorithm
It should be noted that for optimization problems having a low number of
dimensions, say less than 4, some of the other algorithms can outperform the poly-hybrid
PSO method. This is because if the number of dimensions is less, or if problem is
mathematically well behaved, using the poly-hybrid PSO method is needlessly complex;
although it does still give good results. An obvious, but important good rule of thumb is
depending on the problem, number of dimensions, number of merit evaluations and run
time desired; one should pick an algorithm that is best suited for the situation. For
example, a simple, uni-modal problem can be solved fairly efficiently by most classical
methods and the use of a more complex algorithm may not be justified. It is observed
from the parameter sensitivity studies that the adjustment of variables can cause huge
varying of the merit values, so, it is important to take caution of this change.
4.6 Conclusions
In this chapter, the poly-hybrid PSO method is shown to have strength in better
converging to the true optimum solution when compared to other traditional methods
because it combines the best capabilities of a global search method, a random search
method, and two pattern search methods. With the automatic parameter adjuster, the
poly-hybrid method has a strong advantage in finding the best parameters to use to get
the best solution. This adjustment process can, at times come at the price of requiring
additional run time and merit evaluations. It should be noted that the poly-hybrid method
removes “outlier” particles in the PSO portion of the process and this can result in
efficiency in solution progress.
Visual Basic code for the poly-hybrid PSO method is included in appendix 6 of
this dissertation. This code uses REM statements to explain what the various sections do.
For purposes of better understanding, this appendix has been divided into modules that
handle the various parts of the poly-hybrid PSO method.

73
Chapter 5:
Conclusion & Future Research
This research has successfully shown that the PSO optimization method, when
used in combination, with two or more traditional methods to form a hybrid approach,
has a strong potential to improve the convergence process well beyond that expected
from a single algorithm. This advancement in the state of the art suggests that the future
for poly-hybrid methods has great potential for future progress in design optimization.
Although it is beyond the scope of this investigation, the success of this research
suggests that there are other research paths that now seem fruitful for further
investigation. For example, further work is needed to investigate the effect of finding an
optimum solution that terminates at an error that is much less than the user defined error,
thus the solution can be further improved. The reason is the Hooke & Jeeves and
Simplex Methods sometimes get stalled at the local solution and because the error in
either of these methods is less than that for the PSO method. In this case the algorithm
never bounces back to the PSO to move in the design space, thus, causing premature
convergence. Further investigations on improving this aspect of convergence seem to
have good potential for future work. In addition, the benefits of combining more than
two existing methods in a hybrid method have been shown in this research work to have
strong potential for future success in accelerating convergence. This suggests that trying
other pattern search methods in a poly-hybrid approach might also yield good results. In
addition, research on new ways to make the poly-hybrid method smarter about using and
picking the methods automatically during operation should be investigated.

74
List of References
1 A. Ismael F. Vaz and Luis N. Vicente, “A Particle Swarm Pattern Search Method
for Bound Constrained Global Optimization,” Journal of Global Optimization,
Volume 39, Number 2, 197-219, DOI:10.1007/s10898-007-9133-5.
2 E.D. Eason and R.G. Fenton, “A Comparison of Numerical Optimization
Methods for Engineering Design,” Journal of Engineering for Industry:
Transactions of the ASME, February 1974.
3 R.C. Eberhart and J. Kennedy, “A new optimizer using particle swarm theory,”
Proceedings of the Sixth International Symposium on Micromachine and Human
Science, Nagoya, Japan, 1995, 39-43.
4 R.C. Eberhart and Y. Shi, “Comparing inertia weights and constriction factors in
particle swarm optimization,” Proceedings of the IEEE Congress on Evolutionary
Computation (CEC 2000), San Diego, CA, 2000 84-88.
5 R.C. Eberhart and Y. Shi, “Comparison between genetic algorithms and particle
swarm optimization,” Evolutionary Programming VII: Proceedings of the Seventh
Annual Conference on Evolutionary Programming, San Diego, CA. 1998.
6 R.C. Eberhart and Y. Shi, “Particle swarm optimization: developments,
applications and resources,” Proceedings of the IEEE Congress on Evolutionary
Computation (CEC 2001), Seoul, Korea. 2001.
7 R.C. Eberhart, P.K. Simpson and R.W. Dobbins, “Computational Intelligence PC
tools, 1st ed,” Boston, MA: Academic press professional, 1996.
8 F. Van den Bergh, “An Analysis of Particle Swarm Optimizers,” Ph.D.
Dissertation, University of Pretoria, Pretoria, 2001.
9 P.C. Fourie and A.A. Groenwold, “Particle swarms in size and shape
optimization,” Proceedings of the International Workshop on Multidisciplinary
Design Optimization, Pretoria, South Africa, 2000, 97-106.
10 Hong Zhou, Waiman Cheung, and Lawrence C. Leung, “Minimizing Weighted
Tardiness of Job-Shop Scheduling Using a Hybrid Genetic Algorithm,” European
Journal of Operational Research, Volume 194, Issue 3, 1 May 2009, 637-649.

75
11 Hui Liu, Zixing Cai, and Yong Wang, “Hybridizing Particle Swarm Optimization
with Differential Evolution for Constrained Numerical and Engineering
Optimization,” Applied Soft Computing, Volume 10, Issue 2, March 2010, 629-
640.
12 A. Ismael F. Vaz and Luis N. Vicente, “A particle swarm pattern search method
for bound constrained global optimization,” J. Glob Optim, 39, 2007, 197-219.
13 D. Ioan, G. Ciuprina and I. Munteanu, “Intelligent-particle swarm optimization.
Proceedings of 6th International Workshop on Optimization and Inverse
Problems in Electromagnetism (OIPE 2000),” Torino, Italy, 2000.
14 J.A. Nelder and R. Mead, “A Simplex Method for Function Minimization,”
Computer Journal 7,196), 308-313.
15 J.A. Nelder and R. Mead, “A simplex method for function minimization,”
Computer Journal 7, 1965, 308-313.
16 Jiang Chuanwen and Etorre Bompard, “A Hybrid Method of Chaotic Particle
Swarm Optimization and Linear Interior for Reactive Power Optimization,”
Mathematics and Computers in Simulation, Volume 68, Issue 1, 3 February 2005,
57-65.
17 Jing-Ru-Zhang, Jun Zhang, Tat-Ming Lok and Michael R. Lyu, “A Hybrid
Particle Swarm Optimization Back Propagation Algorithm for Feedforward
Neural Network Training,” Applied Mathematics and Computation, Volume 185,
Issue 2, 15 February 2007, 1026-1037.
18 J. Kennedy and R.C. Eberhart, “A discrete binary version of the particle swarm
algorithm,” Proceedings of the World Multiconference on Systemics, Cybernetics
and Informatics 1997, Piscataway, NJ., 1997, 4104-4109.
19 J. Kennedy and R.C. Eberhart, “Particle swarm optimization,” Proceedings of
IEEE International Conference on Neural Networks, Piscataway, NJ, 1995, 1942-
1948.
20 J. Kennedy and R.C. Eberhart, “The particle swarm: social adaptation in
information processing systems.” in Corne, D., Dorigo, M., and Glover, F. (eds.)
New Ideas in Optimization London, UK: McGraw-Hill, 1999.

76
21 J. Kennedy and R. Mendes, “Neighborhood topologies in fully-informed and best-
of-neighborhood particle swarms,” Proceedings of the 2003 IEEE International
Workshop on Soft Computing in Industrial Applications 2003 (SMCia/03), 2003,
45-50.
22 J. Kennedy and W. Spears, “Matching algorithms to problems: an experimental
test of the particle swarm and some genetic algorithms on the multimodal problem
generator,” Proceedings of IEEE Congress on Evolutionary Computation (CEC
1998), Anchorage, Alaska, USA, 1998.
23 P.J. Angeline, “Evolutionary optimization versus particle swarm optimization:
philosophy and performance differences,” Evolutionary Programming VII:
Proceedings of the Seventh Annual Conference on Evolutionary Programming,
1998.
24 J. Kennedy, “The behavior of particles,” Evolutionary Programming VII
Proceedings of the Seventh Annual Conference on Evolutionary Programming,
San Diego, CA., 1998, 581-589.
25 J. Kennedy, “The particle swarm: social adaptation of knowledge,” Proceedings
of IEEE International Conference on Evolutionary Computation, Indianapolis,
IN., 1997, 303-308.
26 J. Kennedy, “Thinking is social: experiments with the adaptive culture model,”
Journal of Conflict Resolution, vol. 42, 1998, 56-76.
27 L. De Giovanni and F. Pezzella, “An Improved Genetic Algorithm for the
Distributed and Flexible Job-Shop Scheduling Problem,” European Journal of
Operational Research, Volume 200, Issue 2, 16 January 2010, 395-408.
28 Lin-Yu Tseng and Ya-Tai Lin, “A Hybrid Genetic Local Search Algorithm for the
Permutation Flowshop Scheduling Problem,” European Journal of Operational
Research, Volume 198, Issue 1, 1 October 2009, 84-92.
29 Michael Meissner, Michael Schmuker and Gisbert Schneider, “Optimized Particle
Swarm (OPSO) and its application to artificial neural network training,” BMC
Bioinformatics, 7:125, 2006.

PWang_PhD_Thesis_Final

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Similar to PWang_PhD_Thesis_Final

Similar to PWang_PhD_Thesis_Final (20)

PWang_PhD_Thesis_Final