Upcoming SlideShare
×

# Nature-Inspired Metaheuristic Algorithms for Optimization and Computational Intelligence

0 views
2,961 views

Published on

This is the supplementary materials for the International Conference: FedCSIS2011 tutorial given by Xin-She Yang.

Published in: Education
2 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
0
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
193
0
Likes
2
Embeds 0
No embeds

No notes for slide

### Nature-Inspired Metaheuristic Algorithms for Optimization and Computational Intelligence

1. 1. Xin-She YangMetaheuristic Algorithms forOptimization and ComputationalIntelligence@FedCSIS2011A more detailed PDF (about 80 pages with Matlab/Octave codes) is provided separately.∗ c Xin-She Yang 2011∗https://sites.google.com/site/metaheuristicbook/fedcsis2011
2. 2. Metaheuristic Algorithms for Optimization andComputational IntelligenceXin-She Yang1Abstract From the algorithm point of view, the essence of solving any optimization and compu-tational problem is to ﬁnd the right ‘smart algorithm’ to complete the task in the most efﬁcientand cost-effective way. Such smart algorithms for computational intelligence also are crucial to anycomputation. However, truly intelligent algorithms are yet to be developed. The right choice of analgorithm can be crucially important in ﬁnding the right solutions for a given problem. More specif-ically, for optimization (and thus computational intelligence) purpose, there exist a diverse rangeof algorithms. Loosely speaking, we have gradient-based algorithms, derivative-free algorithms andmetaheuristics. Modern metaheuristic algorithms are often nature-inspired, and they are suitablefor global optimization. In this tutorial, we will brieﬂy introduce optimization algorithms such ashill-climbing, trust-region method, simulated annealing, differential evolution, particle swarm opti-mization, harmony search, ﬁreﬂy algorithm and cuckoo search.1 IntroductionAlgorithms for optimization are more diverse than the types of optimization, though the right choiceof algorithms is an important issue. There are a wide range of optimization algorithms, and a detaileddescription of each can take up a whole book of more than several hundred pages or a series of longlectures. Therefore, in this tutorial, we will introduce a few important algorithms selected from awide range of optimization algorithms [4, 28, 32], with a focus on the metaheuristic algorithmsdeveloped after the 1990s. This selection does not mean that the algorithms not described here arenot popular. In fact, they may be equally widely used. For more details, please refer to more advancedreferences [9, 28, 31, 37].2 Derivative-Based AlgorithmsDerivative-based or gradient-based algorithms use the information of derivatives. They are very efﬁ-cient as local search algorithms, but may have the disadvantage of being trapped in a local optimumif the problem of interest is not convex. It is required that the objective function is sufﬁciently smoothso that its ﬁrst (and often second) derivatives exist. Discontinuity in objective functions may rendersuch methods unsuitable. One of the classical examples is the Newton’s method, while a modernexample is the method of conjugate gradient. Gradient-base methods are widely used in many appli-cations and discrete modelling [3, 21].2.1 Newton’s Method and Hill-ClimbingOne of the most widely used algorithms is Newton’s method, which is a root-ﬁnding algorithm aswell as a gradient-based optimization algorithm [11]. For a given function f (x), its Tayler expansions1 Mathematics and Scientiﬁc Computing, National Physical Laboratory, UK (xin-she.yang@npl.co.uk) 1
3. 3. 2 1 f (x) = f (xn ) + (∇ f (xn ))T ∆ x + ∆ xT ∇2 f (xn )∆ x + ..., (1) 2in terms of ∆ x = x − xn about a ﬁxed point xn leads to the following iterative formula x = xn − H −1 ∇ f (xn ), (2)where H −1 (x(n) ) is the inverse of the symmetric Hessian matrix H = ∇2 f (xn ), which is deﬁned as  2  ∂ f ∂2 2 ∂ x1 ... ∂ x ∂fxn  1    H(x) ≡ ∇2 f (x) ≡  . . . . . . . (3)  2  ∂ f ∂ f2 ∂ xn ∂ x . . . ∂ x2 1 n Starting from an initial guess vector x(0) , the iterative Newton’s formula for the nth iterationbecomes x(n+1) = x(n) − H −1 (x(n) )∇ f (x(n) ). (4) In order to speed up the convergence, we can use a smaller step size α ∈ (0, 1] and we have themodiﬁed Newton’s method x(n+1) = x(n) − α H −1 (x(n) )∇ f (x(n) ). (5) It is often time-consuming to calculate the Hessian matrix using second derivatives. In this case,a simple and yet efﬁcient alternative is to use an identity matrix I to approximate H so that H −1 = I,which leads to the quasi-Newton method x(n+1) = x(n) − α I∇ f (x(n) ). (6)In essence, this is the steepest descent method. For a maximization problem, the steepest descent becomes a hill-climbing. That is, the aim isto climb up to the highest peak or to ﬁnd the highest possible value of an objective f (x) from thecurrent point x(n) . From the Taylor expansion of f (x) about x(n) , we have f (x(n+1) ) = f (x(n) + ∆ s) ≈ f (x(n) + (∇ f (x(n) ))T ∆ s, (7)where ∆ s = x(n+1) − x(n) is the increment vector. Since we are trying to ﬁnd a better (higher) approx-imation to the objective function, it requires that f (x(n) + ∆ s) − f (x(n) ) = (∇ f )T ∆ s > 0. (8)From vector analysis, we know that the inner product uT v of two vectors u and v is the largest whenthey are parallel. Therefore, we have ∆ s = α ∇ f (x(n) ), (9)where α > 0 is the step size. In the case of minimization, the direction ∆ s is along the steepestdescent in the negative gradient direction. It is worth pointing out that the choice of the step size α is very important. A very small stepsize means slow movement towards the local optimum, while a large step may overshoot and subse-quently makes it move far away from the local optimum. Therefore, the step size α = α (n) shouldbe different at each iteration and should be chosen so as to maximize or minimize the objectivefunction, depending on the context of the problem.