Piece of research on the effect of dimensionality on the interaction among variables

1. Large Scale Problems in Practice:
The effect of dimensionality on the
interaction among variables
Fabio Caraffini∗, Ferrante Neri∗ and Giovanni Iacca∗∗
∗De Montfort University, Leicetser, UK
∗∗RWTH Aachen University, Germany
EvoKNOW, Amsterdam, April 2017

2. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Large Scale Optimisation Problem (LSOP)
LSOPs are optimisation problems plagued by a high number of
design variables (hundreds or even thousands!).
A strict definition cannot be given for LSOP
but such problems are becoming more and more frequent
thus requires further investigation!
Fabio Caraffini EvoKNOW 2017

3. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Hard (large scale) life:
curse of dimensionality
The size of a domain grows
exponentially with the number of
dimensions!
The search space cannot be
covered:
large pop size = high structural
bias
large pop size = no convergence.
Fabio Caraffini EvoKNOW 2017

4. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Hard (large scale) life:
curse of dimensionality
Dimensionality also affects intrinsic features
of the search space itself.
As a square has four sides (1D lines), a cube has six sides (2D
squares) and a tesseract has eight sides (3D cubes):
thus a tesseract of side x has a high “surface volume” of 8x3
.
Similary, hyper-spherical domains behave similarly as:
their surface is Sn(r) = 2π
n
2
Γ( n
2 ) rn−1
,
their volume Vn(r) = π
n
2
Γ( n
2 +1) rn
,
with a ratio Vn(r)
Sn(r) ∝ 1
n .
In high dimensions uniformly sampled points are likely to be lo-
cated on the surface as the volume is a small fraction of the search
space!
Fabio Caraffini EvoKNOW 2017

5. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
How to best address LSOPs then?
We may think that, in such a vast space, exploration plays the major role
however, the most successfully algorithms in the literature implement
exploitation mechanisms:
by employing local searchers,
by means of µ-populations,
by decomposing the search space,
by addressing a variable at a time. . .
Fabio Caraffini EvoKNOW 2017

6. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
What is the common denominator of these approaches?
How can it be formally explained?
The search logic attempts to quickly achieve improvements with
mechanisms similar to those used for separable problems.
We tried and give a formal explanation of this phenomenon by looking at
the correlation amongst design variables (despite correlation=separability):
On the basis of the usual algorithm and experimental setting, what
happens to the correlation among the variables when the
dimensionality grows?
Fabio Caraffini EvoKNOW 2017

7. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Estimating the Correlation between Pairs of Variables
Step 1: “covariance matrix adaptation”.
We “evolved” a covariance matrix C by borrowing the same working
principle of the CMA-ES algorithm with rank-µ-update and weighted
recombination.
As the covariance matrix adapts itself to a basin of a ac traction, it
can be seen as an approximation of the Hessian matrix, thus giving us
information on the local behaviour of the fitness landscape.
Fabio Caraffini EvoKNOW 2017

8. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Estimating the Correlation between Pairs of Variables
Step 2: “correlation index”.
From C we worked out the Pearson (ρ) and the Spearman (τ) correlations:
|ρi,j | =
Ci,j
√
Ci,i Cj,j
=





1 |ρ1,2| |ρ1,3| ... |ρ1,n|
X 1 |ρ2,3| ... |ρ2,n|
X X 1 ... |ρ3,n|
... ... ... ... ...
X X X X 1





|τi,j | =
m
k=1(rk,i − ¯Ri
) m
k=1(rk,j − ¯Rj
)
m
k=1(rk,i − ¯Ri
)
2 m
k=1(rk,j − ¯Rj
)
2
=





1 |τ1,2| |τ1,3| ... |τ1,n|
X 1 |τ2,3| ... |τ2,n|
X X 1 ... |τ3,n|
... ... ... ... ...
X X X X 1





to obtain the global Pearson (ς) and the Spearman (ϕ) correlation indices:
ς = 2
n2−n
n−1
i=1
n
j=i+1 |ρi,j | ϕ = 2
n2−n
n−1
i=1
n
j=i+1 |τi,j |
Fabio Caraffini EvoKNOW 2017

9. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Test problems
We checked (50 runs) ς and ϕ over
19 test problems from SISC2010 in 10, 30, 50, 100, 500 and
1000 dimensions;
28 test problems from CEC2013 in 10, 30, 50 dimensions;
24 test problems from BBOB2010 in 10, 30, 50 and 100
dimensions;
15 test problems from CEC2013_LSGO in 1000 dimensions.
Fabio Caraffini EvoKNOW 2017

10. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Results
Graphical representation
Figure: Correlation indices for
f13 of SISC2010
Figure: Correlation indices for f8
of CEC2013.
Fabio Caraffini EvoKNOW 2017

11. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Results
Observations
For all the 86 problems it appears clear that the correlation
amongst variables decay when the dimensionality.
As a general trend, optimization problems with at least 100
dimensions show a weak correlation, while problems in 500 and
1000 dimensions show a nearly null correlation.
Regardless of the specific problem, all the LSOPs appear
always characterized by uncorrelated variables.
Separable problems ⇒ low correlation regardless of the n value
(N.B. low correlation separability!)
Fabio Caraffini EvoKNOW 2017

12. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Conclusions and way forward
We noted that in practice common experimental conditions
impose a growing shallowness of the search with the increase
of dimensionality.
Under these conditions LSOPs have a weak correlation
between variables, thus, a practically efficient approach is to
avoid exploratory diagonal moves and exploit the directions
along the axes.
Further studies will propose a lighter replacement for the
CMA-ES part, as well as algorithmic components exploiting
the knowledge from the (weak) correlation between pairs of
variables.
Fabio Caraffini EvoKNOW 2017

13. Large scale optimisation problems LSOPs in practice: experimental set-up Results Conclusions
Fabio Caraffini EvoKNOW 2017