DEMPSTER et al.: COMPUTATIONAL LEARNING TECHNIQUES FOR INTRADAY FX TRADING 745can be partly explained by the unsatisfactory performance of Evolutionary learning encompasses sets of algorithms that areexchange rate models based on economic fundamentals. They inspired by Darwinian evolution. GAs are population-based op-cite Frankel and Rose  who state that no model based on timization algorithms first proposed by Holland . They havesuch standard fundamentals like money supplies, real incomes, since become an active research area within the artificial intelli-interest rates, and current-account balances will ever succeed in gence community and have been successfully applied to a broadexplaining or predicting a high percentage of the variation in the range of hard problems. Their success is in part due to their sev-exchange rate, at least at short or medium-term frequencies. eral control parameters that allow them to be highly tuned to the A number of researchers have examined net returns due to specific problem at hand. GP is an extension proposed by Kozavarious trading rules in the foreign exchange markets , . , whose original goal was to evolve computer programs.The general conclusion is that trading rules are sometimes able Pictet et al.  employ a GA to optimize a class of exponen-to earn significant returns net of transaction costs and that this tially weighted moving average rules, but run into serious over-cannot be easily explained as compensation for bearing risk. fitting and poor out-of-sample performance. They report 3.6%Neely and Weller  note however that academic investigation to 9.6% annual excess returns net of transaction costs, but asof technical trading has not been consistent with the practice the models of Olsen and Associates are not publicly availableof technical analysis. As noted above, technical trading is most their results are difficult to evaluate. Neely and Weller  re-popular in the foreign exchange markets where the majority of port that for their GA approach, although strong evidence ofintraday foreign exchange traders consider themselves technical predictability in the data is measured out-of-sample when trans-traders. They trade throughout the day using high-frequency action costs are set to zero, no evidence of profitable trading op-data but aim to end the day with a net open position of zero. portunities arise when transaction costs are applied and tradingThis is in contrast to much of the academic literature which has is restricted to times of high market activity.tended to take much longer horizons into account and only con-sider daily closing prices. IV. REINFORCEMENT LEARNING Goodhart and O’Hara  provide a thorough survey of past Reinforcement learning has so far found only a few financialwork investigating the statistical properties of high-frequency applications. The reinforcement learning technique is stronglytrading data, which has tended to look only at narrow classes of influenced by the theory of MDPs, which evolved from attemptsrules. Goodhart and Curcio  examine the usefulness of resis- to understand the problem of making sequences of decisionstance levels published by Reuters and also examine the perfor- under uncertainty when each decision can depend on the pre-mance of various filter rules identified by practitioners. Demp- vious decisions and their outcomes. The last decade has wit-ster and Jones ,  examine profitability of the systematic nessed the merging of ideas from the reinforcement learning andapplication of the popular channel and head-and-shoulders pat- control theory communities . This has expanded the scopeterns to intraday FX trading at various frequencies, including of dynamic programming and allowed the approximate solutionwith an overlay of statistically derived filtering rules. In subse- of problems that were previously considered intractable.quent work ,  upon which this paper expands, they apply Although reinforcement learning was developed indepen-a variety of technical trading rules to trade such data (see also dently of MDPs, the integration of these ideas with the theoryTan ) and also study a genetic program which trades com- of MDPs brought a new dimension to RL. Watkins binations of these rules on the same data . None of these was instrumental in this advance by devising the method ofstudies report any evidence of significant profit opportunities, -learning for estimating action-value functions. The naturebut by focussing on relatively narrow classes of rules their re- of reinforcement learning makes it possible to approximatesults do not necessarily exclude the possibility that a search over optimal policies in ways that put more effort into learning toa broader class would reveal profitable strategies. Gencay et al. make good decisions for frequently encountered situations in fact assert that simple trading models are able to earn at the expense of less effort for less frequently encounteredsignificant returns after transaction costs in various foreign ex- situations . This is a key property which distinguisheschange markets using high frequency data. reinforcement learning from other approaches for approximate solution of MDP’s. As fundamental research in reinforcement learning advances, III. GENETIC ALGORITHMS applications to finance have started to emerge. Moody et al.  examine a recurrent reinforcement learning algorithm that seeks In recent years, the application of artificial intelligence (AI) to optimize an online estimate of the Sharpe ratio. They alsotechniques to technical trading and finance has experienced sig- compare the recurrent RL approach to that of -learning.nificant growth. Neural networks have received the most atten-tion in the past and have shown varying degrees of success. V. APPLYING OPTIMIZATION METHODS TO TECHNICALHowever recently there has been a shift in favor of user-trans- TRADINGparent, nonblack box evolutionary methods like GAs and in par-ticular genetic programming (GP). An increasing amount of at- In this paper, following , , , we consider tradingtention in the last several years has been spent on these genetic rules defined in terms of eight popular technical indicators usedapproaches which have found financial applications in option by intraday FX traders. They include both buy and sell signalspricing ,  and as an optimization tool in technical trading based on simple trend-detecting techniques such as moving av-applications , , . erages as well as more complex rules. The indicators we use are
746 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 4, JULY 2001the price channel breakout, adaptive moving average, relative information (their current position) than the heuristic and MDPstrength index, stochastics, moving average convergence/diver- methods and we might thus expect them to perform better. Thegence, moving average crossover, momentum oscillator, and GA method also has an extra constraint restricting the com-commodity channel index. A complete algorithmic description plexity of the rules it can generate which is intended to stopof these indicators can be found in , . overfitting of the in-sample data. To define the indicators, we first aggregate the raw tick datainto (here) quarter-hourly intervals, and for each compute the VI. APPLYING RL TO THE TECHNICAL TRADING PROBLEMbar data—the open, close, high, and low FX rates. Most of theindicators use only the closing price of each bar, so we will The ultimate goal of reinforcement learning based tradingintroduce the notation to denote the closing GBP:USD FX systems is to optimize some relevant measure of trading systemrate (i.e., the dollar value of one pound) of bar (here we use performance such as profit, economic utility or risk-adjusted re-boldface to indicate random entities). turn. A standard RL framework has two central components; We define the market state at time as the binary string an agent and an environment. The agent is the learner and de-of length 16 giving the buy and sell pounds indications of the cision maker that interacts with the environment. The environ-eight indicators, and define the state space as the ment consists of a set of states and available actions for the agentset of all possible market states. Here a 1 represents a trading in each state.recommendation for an individual indicator whose entry is oth- The agent is bound to the environment through perceptionerwise 0. In effect, we have constructed from the available tick and action. At a given time step the agent receives input ,data a discrete-time data series: at time (the end of the bar which is representative of some state , where is the setinterval) we see , compute and must choose whether or not of all possible states in the environment. As mentioned in theto switch currencies based on the values of the indicators in- previous section, is defined here as being a combination ofcorporated in and which currency is currently held. We con- the technical indicator buy and sell pounds decisions prependedsider this time series to be a realization of a binary string valued to the current state of the agent (0 for holding dollars and 1 forstochastic process and make the required trading decisions by pounds). The agent then selects an action wheresolving an appropriate stochastic optimization problem. telling it to hold pounds ( ) or dollars ( ) over Formally, a trading strategy is a function the next timestep. This selection is determined by the agent’s , , for some current position ( , dollars, policy ( , i.e., defined in our case as the trading strategy)or , pounds), telling us whether we should hold pounds ( ) which is a mapping from states to probabilities of selecting eachor dollars ( ) over the next timestep. It should be noted that of the possible actions.although our trading strategies are formally Markovian (feed- For learning to occur while iteratively improving the tradingback rules), some of our technical indicators require a number strategy (policy) over multiple passes of the in-sample data, theof periods of previous values of to decide the corresponding agent needs a merit function that it seeks to improve. In RL, this0-1 entries in . The objective of the trading strategies used in is a function of expected return which is the amount of returnthis paper is to maximize the expected dollar return (after trans- the agent expects to get in the future as a result of moving for-action costs) up to some horizon : ward from the current state. At each learning episode for every time-step the value of the last transition is communicated to the agent by an immediate reward in the form of a scalar rein- (1) forcement signal . The expected return from a state is therefore defined aswhere denotes expectation, is the proportional transactioncost, and is chosen with the understanding that trading strate-gies start in dollars, observe and then have the opportunity toswitch to pounds. Since we do not have an explicit probabilistic (2)model for how FX rates evolve, we cannot perform the expecta-tion calculation in (1), but instead adopt the familiar approach ofdividing our data series into an in-sample region, over which we where is the discount factor and is the final time step.optimize the performance of a candidate trading strategy, and an Note that the parameter determines the “far-sightedness” ofout-of-sample region where the strategy is ultimately tested. the agent. If then and the agent myopically The different approaches utilized solve slightly different tries to maximize reward only at the next time-step. Conversely,versions of the in-sample optimization problem. The simple as the agent must consider rewards over an increasingheuristic and Markov Chain methods find a rule which takes as number of future time steps to the horizon. The goal of the agentinput a market state and outputs one of three possible actions: is to learn over a large number of episodes a policy mapping ofeither “hold pounds,” “hold dollars” (switching currencies if which maximizes for all as the limitnecessary) or “stay in the same currency.” of the approximations obtained from the same states at the pre- The GA and RL approaches find a rule which takes as input vious episode.the market state and the currency currently held, and chooses In our implementation, the agent is directly attempting tobetween two actions: either to stay in the same currency or maximize (1). The reward signal is therefore equivalent to ac-switch. Thus the RL and GA method are given slightly more tual returns achieved from each state at the previous episode.
DEMPSTER et al.: COMPUTATIONAL LEARNING TECHNIQUES FOR INTRADAY FX TRADING 747This implies that whenever the agent remains in the base cur- analysis of the algorithm to enable convergence proofs. As arency, regardless of what happens to the FX rate, the agent is bootstrapping approach, -learning estimates the -value func-neither rewarded nor penalized. tion of the problem based on estimates at the previous learning Often RL problems have a simple goal in the form of a single episode. The -learning update is the backward recursionstate which when attained communicates a fixed reward andhas the effect of delaying rewards from the current time pe-riod of each learning episode. Maes and Brookes  show (5)that immediate rewards are most effective—when they are fea-sible. RL problems can in fact be formulated with separate state where the current state-action pair fromspaces and reinforcement rewards in order to leave less of a the previous learning episode. At each iteration (episode) of thetemporal gap between performance and rewards. In particular learning algorithm, the action-value pairs associated with all theit has been shown that successive immediate rewards lead to ef- states are updated and over a large number of iterations their `fective learning. Mataric  demonstrates the effectiveness of values converge to optimality for (4). We note that there aremultiple goals and progress estimators, for example, a reward some parameters in (5): in particular, the learning rate refersfunction which provides instantaneously positive and negative to the extent with which we update the current -factor basedrewards based upon “immediate measurable progress relative to on future rewards, refers to how “far-sighted” the agent isspecific goals.” and a final parameter of the algorithm is the policy followed It is for this reason that we chose to define the immediate in choosing the potential action at each time step. -learningreward function (2) rather than to communicate the cumulative has been proven to converge to the optimal policy regardless ofreward only at the end of each trading episode. the policy actually used in the training period . We find that In reinforcement learning the link between the agent and the following a random policy while training yields the best results.environment in which learning occurs is the value function . In order for the algorithm to converge, the learning rateIts value for a given state is a measure of how “good” it is for must be set to decrease over the course of learning episodes.an agent to be in that state as given by the total expected fu- Thus has been initially set to 0.15 and converges downwardsture reward from that state under policy . Note that since the to 0.000 15 at a rate of , where is theagent’s policy determines the choice of actions subsequent episode (iteration) number which runs from 0 to 10 000. Theto a state, the value function evaluated at a state must depend parameter has been set to 0.9999 so that each state has fullon that policy. Moreover, for any two policies and we say sight of future rewards in order to allow faster convergence tothat is preferred to , written , if and only if , the optimal. . Under suitable technical conditions there will With this RL approach we might expect to be able to outper-always be at least one policy that is at least as good as all other form all the other approaches on the in-sample data set. How-policies. Such a policy is called an optimal policy and is the ever on the out-of-sample data set, in particular at higher slip-target of any learning agent within the RL paradigm. To all op- page values, we suspect that some form of generalization of thetimal policies is associated the optimal value function , which input space would lead to more successful performance.can be defined in terms of a dynamic programming recursion as (3) VII. APPLYING THE GENETIC ALGORITHM The approach chosen extends the genetic programming work Another way to characterize the value of a state is to con- initiated in  and . It is based on the premise that prac-sider it in terms of the values of all the actions that can be taken titioners typically base their decisions on a variety of technicalfrom that state assuming that an optimal policy is followed signals, which process is formalized by a trading rule. Such asubsequently. This value is referred to as the -value and is rule takes as input a number of technical indicators and gener-given by ates a recommended position (long £, neutral, or long $). The agent applies the rule at each timestep and executes a trade if the rule recommends a different position to the current one. (4) Potential rules are constructed as binary trees in which theThe optimal value function expresses the obvious fact that the terminal nodes are one of our 16 indicators yielding a Booleanvalue of a state under an optimal policy must equal the expected signal at each timestep and the nonterminal nodes are thereturn for the best action from that state, i.e., Boolean operators AND, OR, and XOR. The rule is evaluated recursively. The value of a terminal node is the state of the associated indicator at the current time; and the value of a nonterminal node is the associated Boolean function applied toThe functions and provide the basis for learning algo- its two children. The overall value of the rule is the value ofrithms for MDPs. the root node. An overall rule value of one (true) is interpreted -learning  was one of the most important breakthroughs as a recommended long £ position and zero (false) is takenin the reinforcement learning literature . In this method, the as a recommended neutral position. Rules are limited to alearned action-value function directly approximates the op- maximum depth of four (i.e., a maximum of 16 terminals) totimal action-value function and dramatically simplifies the limit complexity. An example rule is shown in Fig. 1. This
748 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 4, JULY 2001definition of a rule generalizes that used in ,  whichallows trees in the comb form of Fig. 1, but to depth 10. The fitness score of such a rule is defined as the total returncumulated over the appropriate data period [cf. (1)], i.e. Fig. 1. An example rule. (6)The genetic algorithm is used to search the space of all suchrules and is tuned to favor rules that trade successfully (i.e.,achieve high fitness scores) in the in-sample training period. Aninitial population of 250 rules is randomly generated and eachrule is evaluated according to (6). A new population of rules isgenerated from this in which high scoring rules are preferred tolow scoring rules. This bias means that the fitness scores of thenew population should be greater than those of the old popula-tion. New rules are generated by two processes: crossover andmutation. To use crossover two parent rules are selected from the Fig. 2. Genetic algorithm crossover.current population. The selection process is biased toward fitter(better performing) rules: all rules in the current population with a Markov chain on a small set of market states and replacesare ranked in order of fitness score, and are chosen with the problem of maximizing the profit made over the in-samplea probability linearly proportional to their rank. A random period with that of maximizing a total expected discounted re-subtree is chosen from each parent rule and these two subtrees turn assuming Markov dynamics. This approach is described inare swapped between the parent rules to create two new rules, detail in Section VIII-A.each of which inherits characteristics from both parents. This The second method is a simple heuristic: with each state weprocess is shown in Fig. 2. Potential subtrees to swap are associate a number (which will be interpreted as the expectedchecked to ensure that the resulting new rules would not exceed rise in the exchange FX rate over the next trading periodsthe specified maximum tree depth. Two new rules meeting this for some ) and consider strategies which buy pounds if thiscriterion are inserted into the new population. For mutation, a number exceeds one threshold, and sell pounds when it fallssingle rule is selected from the current population (again biased below a second threshold. We then optimize over and the twotoward the better performers) and a random node is replaced thresholds to maximize the in-sample profit. More details on thiswith a random node of the same type (e.g., an AND might method are given in Section VIII-B.become an OR). The mutated rule is then inserted into the new These two methods were used to benchmark the success ofpopulation. the true computational learning approaches in maximizing the New rules are generated using 75% crossover and 25% mu- in-sample profit. Neither are likely to attain the true optimum,tation until a total of 250 new rules are generated, when the new but they should perform reasonably well. The heuristic waspopulation is evaluated for fitness scores. The average score of more successful at solving the in-sample problem with nonzerothe new population should be greater than that of the old due transaction costs than any of the other approaches, but it doesto the favoritism shown to the better performing rules in the old not perform particularly well out-of-sample due to over-fitting.population. This process is repeated 100 times (generations) andthe best rule found during the entire run is selected as the output A. A Markov-Chain Linear Programming Approximationof the genetic algorithm. The rules found by this process exhibit a number of desirable Recall that denotes the market-state at time (the values ofproperties. First, with careful tuning of the GA they should per- the indicator recommendations on which the trading decision atform well in-sample. Second, a rule can be understood by hu- time is based). In this section we will let denote the set ofmans: it is clear what the rule does, even if it is not clear why it all market states which occur in the in-sample dataset.does it. Thirdly, this structure limits (but does not prevent) how We will construct a controlled Markov chain on the set ofmuch a rule can learn in detail (i.e., overfit) the training data set. pairs where denotes a market state present in theIt is to be expected that this enforced generalization will lead to in-sample dataset and indicatesbetter out-of-sample performance with a possible reduction in whether or not pounds are currently held. The controls at eachin-sample performance. timestep are 0 or 1, indicating the currency we wish to hold over the next timestep as before. Denoting by the number of times state appears VIII. MARKOV CHAIN AND SIMPLE HEURISTIC in the in-sample dataset (excluding the final data-point), and In addition to the RL and GA methods we will consider two by the number of times state is immediately fol-alternative approaches. The first replaces the in-sample dataset lowed by state , we define a controlled Markov chain ,
DEMPSTER et al.: COMPUTATIONAL LEARNING TECHNIQUES FOR INTRADAY FX TRADING 749 , by fixing and choosing the prob-ability of a transition from to using controlto beFor , we define an approximation to the ex-pected dollar return over the next timestep given we are in state and hold currency asWe are now in a position to replace the problem of maximizingthe in-sample return with the problem Fig. 3. Cumulative in-sample monthly returns at no slippage. (7)where is a feedback map from the state-space of the Markovchain to the set of controls and we treat the termas zero (since we must start in dollars). The constant is a dis-count factor, chosen arbitrarily to be equal to . This is anapproximation to the objective (1). Since the set is quite small, this problem can be solvedexactly using the technique of linear programming. To see this,observe that the solution to (7) is characterized by the optimalvalue function defined for all asthe optimal value (1) when . The functionsatisfies the dynamic programming recursion Fig. 4. Cumulative out-of-sample monthly returns at no slippage. (8)whose solution is unique under various conditions (it sufficesthat the Markov chain has a finite statespace and that , which is the case here). But a solution to the system abovecan be obtained by solving the linear programsubject to for all and Fig. 5. Cumulative in-sample monthly returns at 1 bp slippage. B. A Simple Heuristic (9) One objection to the method of the previous section is that when transaction costs are large the solution obtained to the problem in (7) above may perform badly over the actualwhich must yield the required optimal value function . The in-sample period; worse even than the trivial strategy ‘alwaysoptimal action in state can be extracted from the hold dollars’ with a return of 0. As an alternative, we consideroptimal value function by choosing any maximizing a heuristic defined in terms of three parameters :the right-hand side of (8). The solution to the LP is found foreach in-sample period using the CPLEX commercial LP solver. when (10)
750 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 4, JULY 2001Fig. 6. Cumulative out-of-sample monthly returns at 1 bp slippage. Fig. 8. Cumulative out-of-sample monthly returns at 4 bp slippage.Fig. 7. Cumulative in-sample monthly returns at 4 bp slippage. Fig. 9. Cumulative in-sample monthly returns at 8 bp slippage. when (11)The expected value in (10) and (11) is just the expected returnavailable if we held pounds for the next days given that the cur-rent market state is . Since we do not have a stochastic processmodel for FX rate movements, this expectation must also be es-timated from the in-sample data (assuming the ergodic theoremholds) aswhere is the set . For a several classes of stochastic process models for FX dy-namics, the optimal strategy for both very low and very hightransaction costs has the form of (10) and (11), making it a plau- Fig. 10. Cumulative out-of-sample monthly returns at 8 bp slippage.sible heuristic in general. The optimization of the three parameters of the heuristic is a from January 1994 until January 1998 using a moving windownonconvex multiextremal problem and for each in-sample pe- of one year for training (fitting) followed by one month out-of-riod is solved by a simple genetic algorithm. sample testing. The cumulative (without reinvestment) returns over the IX. NUMERICAL EXPERIMENTS period are shown in odd-numbered Figs. 3–9 for selected The results reported below were obtained by applying the ap- in-sample (fitting) cases and evenly-numbered Figs. 4–10 forproaches described above to the GBP:USD exchange rate data corresponding out-of-sample (testing) cases. The annualised
DEMPSTER et al.: COMPUTATIONAL LEARNING TECHNIQUES FOR INTRADAY FX TRADING 751 TABLE I OUT-OF-SAMPLE AVERAGE ANNUAL RETURNSaverage monthly returns are shown in Table I for the variousapproaches in the out-of-sample case. In-sample fitting performance has been shown for com-pleteness to demonstrate the learning ability of the variousapproaches. It is clear that on the in-sample data set, thesimple heuristic approach consistently outperforms all the Fig. 11. Indicators used by the genetic algorithm by slippage.other methods except in the no slippage case, when all methodswere able to fit the data to essentially the same degree. Theout-of-sample test results demonstrate, however, that theheuristic approach was in fact significantly overfitting the data. For the out-of-sample back-tests, we note that the genetic al-gorithm and reinforcement learning approaches tended to out-perform the others at lower slippage values. In order to gainfurther insight into the overall best performing GA, a plot ofhow often it inferred rules using each indicator for each slip-page value is shown in Fig. 11. Fig. 12 shows the frequencieswith which the GA employed specific indicators over the en-tire four year data period, aggregated for all slippage values andinto quarters, with considerable variability in the patterns evi-dent. The GAs reduction in trading frequency with decreasingtransaction costs is demonstrated dramatically in Fig. 13. Sim-ilar results apply to the other methods with the exception of theheuristic, whose high trading frequency at realistic transaction Fig. 12. Relative frequencies over time of indicators used by the geneticcosts leads to its poor performance in out-of-sample back-tests. algorithm.Data on the dealing frequency of all the different approaches isgiven in Table II in order to shed light on the risk profiles ofthe different methods. These results are discussed further in thefinal Section X. In order to evaluate the relative risk-adjusted performance ofthe trading models further, we now consider several risk mea-sures found in the financial literature. A risk-adjusted measurecommonly used to evaluate portfolio models is the Sharpe ratio,defined as Sharpe Ratio (12)The Sharpe ratio evaluations of our trading models, as shownin Table III demonstrate that on the dataset used we are able togain significant risk-adjusted returns up to a slippage value of 1bp. However the Sharpe ratio is numerically unstable for small Fig. 13. Genetic algorithm position duration by slippage.variances of returns and cannot consider the clustering of profitand loss trades , . Furthermore the Sharpe ratio penal- TABLE IIizes strategies for upside volatility and its definition in terms of OUT-OF-SAMPLE AVERAGE MONTHLY DEALING FREQUENCYsummary statistics means that a strategy can appear to be suc-cessful but in fact suffers from significant drawdown . Max-imum drawdown over a certain period of lengthis defined as (13)
752 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 4, JULY 2001 TABLE III hundred signal states that occur within a given year. Each of OUT-OF-SAMPLE SHARPE RATIOS the four trading strategies is then evaluated out-of-sample at 15-min intervals on the next month of indicator signals and this back-testing process is then rolled forward a month and repeated for a total of 48 months. It is evident that in-sample, all approaches were able to infer successful trading strategies and also notable that the ge- netic algorithm consistently underperforms the other methods in-sample. This is undoubtably due to the constraint imposed on TABLE IV OUT-OF-SAMPLE DRAWDOWNS the complexity of the rules which was specifically imposed to avoid overfitting. By contrast, the non-GA approaches—in par- ticular the trading threshold heuristic—may end up exploiting noise in the in-sample data set. At zero-slippage (no costs of trading), however, we find that all approaches are able to infer similar strategies and perform similarly out-of-sample. There is evidence that the non-GA approaches do in fact overfit as the GA outperforms the other methods with nonzero transaction TABLE V costs in the out-of-sample cases up to 8 bp slippage. OUT-OF-SAMPLE STIRLING RATIOS The fact that the techniques investigated here return positive results both in-sample and in out-of-sample back-tests implies that there is useful information in technical indicators that can be exploited. This is consistent with the tenets of technical anal- ysis and contradictory to the efficient market hypothesis. Fur- thermore, the GAs relatively good out-of-sample performance demonstrates that using a combination of technical indicators leads to better performance than using the individual indica-where and are the total returns of the periods from tors themselves. In fact, Dempster and Jones ,  demon-to and as defined by (6) respectively. In our case was strate that with a few exceptions these indicators are largelydefined as on a monthly basis and the mean over the out-of- unprofitable on the same data when considered in isolation.sample back-test period is reported in Table IV. We therefore Figs. 11 and 12 demonstrate that some indicators also conveyalso quote the Stirling ratio, defined as more information than others depending on the slippage value and the market state. We note that the relative strength index Stirling Ratio (14) (buy/sell) indicators are not used at zero transaction costs but as the slippage is increased, the GA tends to favor them. Indi- cators such as price channel breakout, stochastics, and movingwhich is the average monthly return divided by the maximum average crossover (buy) are very important at zero slippage, butdrawdown within that month. This value averaged over the 48 the GA appears to disregard the information provided by them atmonthly back-test periods is reported in Table V. higher slippage values. At zero slippage the GA is able to infer The current RL implementation requires about eight minutes successful strategies without using the current position. How-CPU time on a 650 MHz Athlon per single training opti- ever, at higher transaction costs, knowing the current positionmization (episode). The GA is implemented in the interpreted becomes very important. This lends credence to the argumentlanguage Scheme, but evaluation is parallelised over multiple that this extra position information tends to favor the RL and GAsimilar CPUs. It also takes about eight minutes CPU time approaches, since the Markov chain approach and the heuristicper optimization on a single machine. The Markov chain and did not have this information available to them.heuristic approaches execute in four seconds and approximately The RL approach, the Markov chain and the heuristic all ex-four minutes, respectively. ploit the fact that of the market possible states only a few hundred actually occur in the in-sample period. This number X. DISCUSSION AND CONCLUSION is small enough that each state may be considered individually In this paper we have developed three trading strategies based when deciding a strategy. However, there are two problems withon computational learning techniques and one simple heuristic such a rule when it is back-tested out-of-sample.based on trading thresholds over a fixed horizon. The strategies Firstly, we may encounter a state in the out-of-sample databased on the genetic (programming) algorithm (GA) and rein- which was not present in the in-sample data. In that case someforcement ( -) learning train at 15-min intervals on the buy-sell arbitrary action must be made and both the RL and the Markovsignals from eight popular technical trading indicators—some chain method choose to hold their current position. This may beof which require a number of previous observations—and cur- a disadvantage if many new states are encountered and it also ig-rent positions over a one year period of GBP:USD FX data, nores the fact that some new states may be very similar to stateswhile the Markov chain strategy uses the entire set of training which were present in the in-sample period. The genetic algo-data to estimate the relative transition frequencies of the few rithm on the other hand generates a trading rule in the in-sample
DEMPSTER et al.: COMPUTATIONAL LEARNING TECHNIQUES FOR INTRADAY FX TRADING 753training period whose structure tends to take the same actions in  M. P. Taylor and H. Allen, “The use of technical analysis in the foreignsimilar states. exchange market,” J. Int. Money Finance, vol. 11, pp. 304–314, June 1992. Secondly, there is a severe danger that these approaches  J. A. Frankel and A. K. Rose, “A survey of empirical research on nominalmay learn too well (overfit) the specific in-sample data; in exchange rates,” Nat. Bureau Economic Res., Working Paper No. 4865,other words the in-sample problem they attempt to solve is too Sept. 1994.  R. Levich and L. Thomas, “The significance of technical trading rulespecific. Indeed the simple heuristic method demonstrates this profits in the foreign exchange market: A bootstrap approach,” J. Int.quite clearly: it achieves excellent in-sample performance but Money Finance, vol. 12, pp. 451–474, Oct. 1993.is mediocre out-of-sample and terrible at realistic transaction  C. R. Osler and K. Chang, “Head and shoulders: Not just a flaky pattern,” Federal Reserve Bank of New York, Staff Papers no. 4, Aug. 1995.costs. The limit on the complexity of the GA is an artificial  C. Neely and P. Weller, “Intraday technical trading in the foreign ex-constraint which reduces the opportunity for the GA to overfit change market,” Federal Reserve Bank of St. Louis, Working Paperwhile not prohibiting simple trading rules. This limit effectively 99-016A, 1999.  C. Goodhart and M. O’Hara, “High frequency data in financial markets:forces the GA to work with generalized (classes of specific) Issues and applications,” J. Empirical Finance, vol. 4, pp. 73–114, 1997.states. Thus we hope to improve the current reinforcement  C. Goodhart and R. Curcio, When support/resistance levels are broken,learning approach by forcing state generalization and also by can profits be made? Evidence from the foreign exchange market, London School of Economics, July 1992.improving its convergence properties.  M. A. H. Dempster and C. M. Jones, “Can channel pattern trading be Another current avenue of research is to find constraints for successfully automated?,” European J. Finance, 2001, to be published.  , Can technical pattern trading be successfully automated? 2. Headthe in-sample optimization problem which force state general- and shoulders, Centre Financial Res., Judge Inst. Management, Univ.ization (such as the rule-complexity constraint in the GA ap- Cambridge, July 1999.proach), but for which a heuristic similar to that of Section  C. M. Jones, “Automated Technical Foreign Exchange Trading with High Frequency Data,” Ph.D. dissertation, Centre Financial Res., JudgeVIII-B can be applied. Inst. Management, Univ. Cambridge, June 1999. A further goal is the exploration of generalization methods  M. A. H. Dempster and C. M. Jones, The profitability of intra-day FXin the context of a broader RL approach. Neuro-dynamic trading using technical indicators, , Centre Financial Res., Judge Inst. Management, Univ. Cambridge, Dec. 2000.programming (NDP)  attempts to combine neural networks  G. W. Tan, “Topics in foreign exchange trading systems,” M.S. thesis,with the central ideas of dynamic programming in order to Centre Financial Res., Judge Inst. Management, Univ. Cambridge, Juneaddress this goal. NDP’s employ parametric representations 1999.  M. A. H. Dempster and C. M. Jones, “A real-time adaptive tradingof the value function (such as artificial neural networks) to system using genetic programming,” Quant. Finance, 2001, to beovercome the curse of dimensionality. The free parameters are published.tuned using regression or stochastic approximation methods  R. Gencay, G. Ballocchi, M. Dacorogna, R. Olsen, and O. Pictet, Real-time trading models and the statistical properties of foreignused in combination with classical dynamic programming exchange rates, in Internal Document RAG.1988-12-01, Olsen &methods. Further, Wilson’s X-classifier system  attempts to Associates, Zurich, Switzerland, 1998.merge ideas from reinforcement learning with those from the  S. Chen and W. Lee, “Option pricing with gas: A second report,” in Proc. IEEE Int. Conf. Neural Networks, vol. 1, 1997, pp. 21–25.Classifier System community in order to incorporate general-  N. K. Chidambaran, C. H. J. Lee, and J. R. Trigueros, “An adaptiveization into a -Learning-like framework. We also intend to evolutionary approach to option pricing via genetic programming,”investigate this approach in the present context in the future. in Genetic Programming 1998: Proc. 3rd Annu. Conf., J. R. Koza, W. Banzhaf, K. Chellapilla, K. Deb, M. Dorigo, D. B. Fogel, M. H. Garzon, D. E. Goldberg, H. Iba, and R. Riolo, Eds. Madison, WI, July 22–25, 1998, pp. 38–41.  J. H. Holland, Adaptation in Natural and Artificial Systems. Ann ACKNOWLEDGMENT Arbor, MI: Univ. Michigan Press, 1975.  J. R. Koza, Genetic Programming: On the Programming of Computers The authors would like to thank Dr. C. Jones for his insightful by Means of Natural Selection. Cambridge, MA: MIT Press, 1992.  O. V. Pictet, M. M. Dacorogna, B. Chopard, M. Oudsaidene, R. Schirru,comments, advice, and proofreading of the paper, and J. Scott and M. Tomassini, “Using genetic algorithm for robust optimizationfor lending his expertise in Scheme. in financial applications,” Neural Network World, vol. 5, no. 4, pp. 573–587, 1995.  D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Program- ming. Waltham, MA: Athena Scientific, 1996.  C. Watkins, “Learning from Delayed Reward,” Ph.D. dissertation, Dept. REFERENCES Eng., Univ. Cambridge, 1989.  R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduc-  B. LeBaron, “Technical trading rule profitability and foreign exchange tion. Cambridge, MA: MIT Press, 1998. intervention,” J. Int. Economics, vol. 49, pp. 124–143, 1999.  J. Moody, L. Wu, Y. Liao, and M. Saffell, “Performance functions and re-  A. W. Lo and A. C. MacKinlay, “Stock market prices do not follow inforcement learning for trading systems and portfolios,” J. Forecasting, random walks: Evidence froma simple specification test,” Rev. Financial vol. 17, pp. 441–470, 1998. Studies, vol. 1, pp. 41–66, 1988.  P. Maes and R. Brooks, “Learning to coordinate behaviors,” in Proc. 8th  , A Nonrandom Walk Down Wall Street. Princeton, NJ: Princeton Nat. Conf. Artificial Intell., July 29–Aug. 3, 1990, pp. 796–802. Univ. Press, 1999.  M. Mataric, “Learning in multi-robot systems,” in IJCAI-95 Workshop  C. J. Neely, P. A. Weller, and R. Dittmar, “Is technical analysis in the on Adaptation and Learning in Multiagent Systems, 1995, pp. 32–37. foreign exchange market profitable? A genetic programming approach,”  S. Wilson, “Classifier fitness based on accuracy,” J. Evolutionary J. Financial Quantitative Anal., pp. 405–426, Dec. 1997. Comput., vol. 3, no. 2, pp. 149–175, 1995.
754 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 4, JULY 2001M. A. H. Dempster received the B.A. degree from Toronto University, Toronto, Yazann Romahi received the B.Eng. degree in electronics and computer sci-ON, Canada, the M.A. degree from Oxford University, Oxford, U.K., and the ence and the M.Sc. degree in artificial intelligence from Edinburgh University,M.S. and Ph.D. degrees from Carnegie Mellon University, Pittsgurgh, PA, all in Edinburgh, U.K. He is pursuing the Ph.D. degree at the Centre for Financial Re-mathematics. search at the Judge Institute of Management Studies in the University of Cam- He is currently Professor of Management Studies (Finance and Management bridge.Science), Director of the Centre for Financial Research and Director of Re- His Ph.D. work is involved in the investigation of FX Market Dynamics andsearch at the Judge Institute of Management Studies in the University of Cam- optimal trading in the FX markets in the presence of transaction costs usingbridge. His present research interests include mathematical and computational artificial intelligence-based techniques.finance and economics, optimization and nonlinear analysis, stochastic systems,algorithm analysis, decision support, and applications software and telecom-munications systems modeling. He is author of more than 100 published re-search articles and reports and is author, editor or translator of eight booksincluding Mathematics of Derivative Securities (Cambridge, U.K.: CambridgeUniv. Press, 1997). Dr. Dempster is joint Editor-in-Chief of Quantitative Finance and AssociateEditor of Stochastics and Stochastic Reports, Journal of Economic Dynamicsand Control, Computational Finance, and Computational Economics. G. W. P. Thompson received the M.A. and Ph.D. degrese in mathematics fromTom W. Payne received the M.A. degree in engineering and computer science the Queen’s College, Cambridge University, Cambridge, U.K.from Clare College, Cambridge University, Cambridge, U.K., and the M.Sc. He is a Research Associate at the Centre for Financial Research at the Judgedegree in artificial intelligence from Edinburgh University, Edinburgh, U.K. He Institute of Management Studies in the University of Cambridge. His Ph.D.is currently pursuing the Ph.D. degree at the Centre for Financial Research at work investigated Gaussian models for interest-rates, numerical techniques forthe Judge Institute of Management Studies in the University of Cambridge. pricing complex options in Gaussian markets, and the optimal trading of se- His work focuses on the application on genetic algorithms to building suc- curities in the presence of transaction costs. His current research interests alsocessful automated trading systems. include optimization, particularly stochastic programming.