ES-Rank is an evolutionary strategy approach for learning to rank documents. It uses a (1+1)-evolutionary strategy with Gaussian*exp(Cauchy) random mutation to evolve a ranking model. Experimental results on three datasets show that ES-Rank performs competitively in terms of accuracy metrics like MAP and NDCG compared to 14 other ranking techniques, while having reasonable runtimes. It considers the whole training dataset with each iteration to effectively learn rankings for large collections.

1. ES-Rank: Evolutionary Strategy
Learning to Rank Approach
Osman AliOsman Ali SadekSadek Ibrahim, Dario LandaIbrahim, Dario Landa--SilvaSilva
ASAP Research GroupASAP Research Group
The UniversityThe University oof Nottingham, UKf Nottingham, UK
April,April, 20172017

2. Outline of the PresentationOutline of the Presentation
Background
Information Retrieval System Architecture
Learning to Rank Architecture
Problem Statement
The Proposed ES-Rank
2
High-Level Overview
Detailed steps
Experimental Studies
Experimental Study Settings.
Experimental Results.
Conclusion

3. Information Retrieval (IR) System ArchitectureInformation Retrieval (IR) System Architecture
3Figure 1: Typical IR System Architecture

4. Typical Structure of a LTR DatasetTypical Structure of a LTR Dataset
4
In general, there are three categories of LTR methods:
point-wise, pair-wise and list-wise. Hybrid approaches
have also been proposed.

5. Learning to RankLearning to Rank ArchitectureArchitecture
5

6. Some of the limitations of existing Learning to Rank techniques:
Not considering the whole training dataset in each
learning iteration. Performance vs Computation Effort.
Problem StatementProblem Statement
6
Computational runtime and memory size requirements
for some of the big datasets. For example, rt-rank and
Ranklib packages.
The large variations in the characteristics of big datasets
demands their full consideration for more effective
learning.

7. Start IR System Using
Various Query-
Document Features
Document
Collection
OR
Users search the IR
system for Information
Proposed Approach (highProposed Approach (high--level description)level description)
7
Gathering Relevance
Feedback by IR System from
Users to Create the IR
Query-Document Pairs
Use Query-Document Pairs Training Set for
Evolving a Ranking Model using ES-Rank
OR
WWW

8. The Proposed Approach (ESThe Proposed Approach (ES--Rank)Rank)
Start
Initialize Chromosomes for
Parent and Offspring
Previous mutation
good and stopping
Mutate chromosome on
R genes using previous
NO
Yes
Stopping
criterion
reached?
End
YES
8
Mutate chromosome on
R random genes with
Gaussian*exp(Cauchy)
random number
good and stopping
criterion not
reached?
R genes using previous
mutation step
Mutated chromosome
better than current
chromosome and
stop criterion not
reached?
Mutated chromosome
becomes the current
chromosome
NO
Yes
NO

9. Initialization
Stopping Criterion
Random Mutation with
The Proposed Approach (ESThe Proposed Approach (ES--Rank)Rank)
9
Objective Function
Random Mutation with
adaptive procedure
according to pervious
iteration procedure.

10. Experimental Study SettingsExperimental Study Settings
• Each dataset includes the training, validation and test sets.
• ES-Rank and the other compared methods are applied to the training set in
order to evolve a linear ranking function.
• The performance of the linear ranking function is assessed using the test set
to obtain the predictive performance of the learning algorithm.to obtain the predictive performance of the learning algorithm.
• The comparison used three well-known LETOR datasets and two evaluation
fitness metrics.
10
LETOR Datasets used in experiments
Dataset Queries
Query-URL
Pairs
Features Relevance Labels No. Folds
MQ2007 1692 69623 46 {0,1,2} 5
MQ2008 784 15211 46 {0,1,2} 5
MSLR-
WEB10K
10000 1200192 136 {0,1,2,3,4} 5

11. Experimental Study SettingsExperimental Study Settings
• The experimental study conducted was a comparison between ES-Rank and
14 other Evolutionary and Machine Learning techniques.
• The default parameter settings suggested for the 14 Evolutionary and
Machine Learning techniques were used as described in well-known packagesMachine Learning techniques were used as described in well-known packages
(Ranklib, rt-Rank, LAGEP, SVM-Rank, Sofia-ml).
• ES-Rank uses (1+1)-Evolutionary Strategy with Gaussian*exp(Cauchy) Random
Number as the mutation step. The mutation has a random probability
(Random Walk) that adapts according to the performance in the previous
iteration. The number of evolving iterations was set to 1300 in these
experiments. 11

12. Experimental ResultsExperimental Results
[Mean Average Precision (MAP) Results][Mean Average Precision (MAP) Results]
ES-Rank performs the best in 7 of the 15 data folds
.
9

13. Experimental ResultsExperimental Results
[Normalized Discounted Cumulative Gain (NDCG) Results][Normalized Discounted Cumulative Gain (NDCG) Results]
ES-Rank performs the best in 4 of the 15 data folds.
.
13

14. Experimental Results (Time Evaluation)Experimental Results (Time Evaluation)
Algorithm MSLR-WEB10K MQ2008 MQ2007 AvgTime (s)
RankBoost 3720 15 74 1269.667
RankSVM 32409 19 23 10817
ListNET 18005 45 95 6048.333
AdaRank 3600 11 20 1210.333
MART 1200 8 11 406.3333
Coordinate Ascent 25200 37 240 8492.333
14
Coordinate Ascent 25200 37 240 8492.333
LambdaMART 3720 9 11 1246.667
RankNET 10800 33 96 3643
Random Forest 3660 27 55 1247.333
Linear Regression 157 2 3 54
RankGP 26020 375 390 8928.333
Combined Ranking and
Regression
10803 42 51 3632
LambdaRank 18015 46 142 6067.667
IGBRT 36750 274 253 12425.67
ES-Rank 1800 35 51 628.6667

15. Experimental Results (Statistical Summary)Experimental Results (Statistical Summary)
Algorithms mean sd se(mean) cv
AdaRank 0.4605 0.0758 0.0138 0.1646
Combined.Regression.Ranking 0.4318 0.0406 0.0074 0.0941
Coordinate.Ascent 0.4776 0.0638 0.0116 0.1330
ES-Rank 0.4779 0.0596 0.0109 0.1247
IGBRT 0.4565 0.0563 0.0145 0.1234
LambdaMART 0.4774 0.0634 0.0116 0.1328
15
LambdaMART 0.4774 0.0634 0.0116 0.1328
LambdaRank 0.3247 0.0939 0.0171 0.2892
Linear.Regression 0.4425 0.0527 0.0096 0.1192
ListNET 0.4099 0.1034 0.0189 0.2522
MART 0.4750 0.0624 0.0114 0.1313
Random.Forest 0.4771 0.0661 0.0121 0.1384
RankBoost 0.4621 0.0770 0.0141 0.1666
RankGP 0.4197 0.0372 0.0068 0.0887
RankNET 0.4143 0.1060 0.0194 0.2559
RankSVM 0.4043 0.0707 0.0129 0.1748

16. ConclusionConclusion
There are issues in improving IR using learning to rank approaches,
particularly on large document collections.
The proposes ES-Rank performs well against fourteen other techniques in
terms of NDCG@10 and MAP.
16
terms of NDCG@10 and MAP.
The proposed ES-Rank considers the whole instance (Query-Document Pairs)
in each learning iteration which helps to improve the performance of the
evolved ranking model.
The computational run-time of ES-Rank is reasonable given its performance in
terms of accuracy (third fastest after Linear Regression and MART).