This document discusses automated approaches for selecting robust systematic trading strategies. It presents several techniques: system parameter permutation to evaluate all possible outcomes and address overfitting; domain regression to select strategies where train and test performance are correlated; step-down correlation minimization to choose uncorrelated strategies; and bootstrap sampling with random test sets instead of fixed periods. The goal is to estimate out-of-sample performance using all available backtest data through techniques like system parameter permutation, domain regression, and estimating profit and loss shortfall.

1. Automated Selection and
Robustness
For Systematic Trading Strategies
Dr Tom Starke

2. • Traditional approach
Parameter optimisation on train set
Validation on test set
This is what we can do now:
System Parameter Permutation
Supervised compound metric optimisation
Domain Regression
Step-Down Correlation Minimisation
Strategy Selection Automation
Walk-forward optimisation

3. Traditional Approach

4. Train Data
Test Data

5. Bad result - bad luck
Is there another way?

6. Too many parameters is bad news
for systematic trading strategies
But what is too many?

7. • Data mining bias very real
Overfitting is hard to detect
Solution:
Use all available backtest data

8. System Parameter
Permutation
Evaluation all possible outcomes*
ning Data Mining from Bias to Benefit Through System Parameter Permu

9. Mean should
be positive
PnL Distribution of all backtest results
Pitfall: parameter space has unreasonable bounds

10. Domain Regression:
The train performance informs the test
performance.

11. Train performance informs test performance
Correlation of results from backtest of train and test sets

12. BAD!

13. • Picking a single parameter set may not
produce very good results
Selecting several of the best strategies increases
the probability of arriving closer to the
regression line

14. Step-Down Low-Correlation Approach

15. Parameter sets should be as uncorrelated as possible

16. One More:
Use random subsamples instead of fixed test
period

17. Train
Test

18. Randomised test sets

19. Estimating PnL Shortfall

20. BRAC*- Build,Rebuild and Compare
Out-of-sample performance scales with the
mean of the PnL distribution of N
subsamples.
*Building Reliable Trading Systems - Keith
Fitschen

21. Choosing subsamples for
BRAC test

22. Mean PnL shortfall scales with OOS performance
Performance of full data set
Adjusted mean PnL of
subsamples

23. Bootstrapping
Expected
OOS PnL

24. Compound Metrics
Manually supervised ML

25. Optimisation: Monte-Carlo vs Grid Search
GA’s may not give the full story.

26. Walk-forward optimisation

27. System Parameter Permutation Profitability
Domain Regression OOS Expectation
Step-Down Selection Optimising Outcome
BRAC PnL Shortfall
Walk-Forward Optimisation Algorithm
Human-Supervised Metric
Construction
Output Metric
Bootstrapping PnL Shortfall

28. Questions?