Approaching Real-Time Business Intelligence: Trading at the Speed of Light


Published on

Talk given at the Big Data conference in Edmonton

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Approaching Real-Time Business Intelligence: Trading at the Speed of Light

  1. 1. Approaching Real-Time Business Intelligence Trading at the Speed of Light Sean McClure, Ph.D. Business Analytics, Excellerate Inc.
  2. 2. Overview ThemesIntroducing Excellerate MetricsReal-Time BI and HFT Information at High Frequency Data Mining Strategies at High Frequency Big Data Developing and Deploying Models Meta Data Executing and Monitoring Real-Time Systems
  3. 3. About Us! Business Intelligence Service Providers• Dedicated to bringing top quality business intelligence expertise to successful growing organizations (SGOs);• Aggressively researching industry best practices and best-in-breed software tools to deliver high-end analytics and data mining expertise;• Business Intelligence model supported by Subject Matter Experts (SMEs) in key business areas.
  4. 4. Real -Time Business Intelligence Defining “Real-Time” Three types of latency1:• Data latency: time taken to collect and store the data;• Analysis latency: time taken to analyze the data and turn it into actionable information; and• Action latency: the time taken to react to the information and take action. Approaching “zero” latency• Real-time business intelligence technologies are designed to reduce all three latencies as close to zero as possible;• Traditional BI only seeks to reduce data latency.
  5. 5. Real -Time Business IntelligenceReal Time BI in various industriesDebit and Credit Fraud DetectionMarketing 1Inventory ControlSupply-chain OptimizationCustomer relationship management (CRM)Dynamic pricing and yield managementData validationOperational intelligence and risk management 2Call center optimizationTransportation industryFinance (biggest candidates) 3
  6. 6. High Frequency Trading (HFT) Best Case Study for “Real-Time” Intelligence High• Trading platform transacts Execution Latency large number of orders at Traditional long- very fast speeds; term Investing• Complex algorithms analyze multiple markets and Algorithmic/ High- execute orders based on electronic trading frequency market conditions; Trading• Faster execution speeds Low more profitable than slower Short Position Holding Period Long execution speeds. Figure 1 – Types of Trading6 In the U.S., high-frequency trading accounts for ~73% of all equity trading volume5
  7. 7. High Frequency InformationProperties of Tick Data – Quote, Trade, Price and Volume Information Date/time quote originated Provided by other Highest price available for market participants sale of the security through limit orders• Timestamp• Security ID Lowest price entered for• Bid Price buying the security• Ask Price• Available bid volume Total demand• Available ask volume Total supply• Last trade price• Last trade size Price at which the last trade in the• Option-specific data security cleared Actual size of the last executed trade
  8. 8. High Frequency InformationRecent microstructure research and advances in econometricmodeling tell us there are unique characteristics to tick data; irregularly spaced Tick Data quotes arriving randomly very short time intervals time(compare to low-frequency)Irregularities wealth of information not available in low-frequencydata; Inter-trade durations signal changes in market volatility, liquidity, and other variables.Volume of data allows for statistically precise inferences. Number of observations in single day of tick data = 30 years of daily observations
  9. 9. High Frequency Information Modeling the Arrivals of Tick Data creates a host of opportunities not available at low-frequency• Time distance between quote arrivals carries information time quote processes Duration models trade processes price processes Estimate the factors affecting the volume processes duration between ticks High Trade Duration Higher likelihood of unobserved bad news Low Trade Low Price Duration Low Volume Duration Higher likelihood Duration Increased of unobserved Increased Liquidity Volatility good news Absence of TradeLack of news, low levelsof liquidity, trading halts, trader motivations
  10. 10. High Frequency InformationData sampling methods overcome irregularities in high-frequency datafor ease of processing Traditional Approach Linear Time-Weighted Interpolation7 Minute 1 Minute 2 Minute 3 Minute 1 Minute 2 Minute 3 Figure A Figure B t tlastquote ˆ qt qt ,last ˆ qt qt ,last (qt ,next qt ,last ) t next tlastMost modern computational techniques have been developedto work with regularly spaced data (easy to process)High frequency data-sampling methods developed toovercome irregularities in tick data by sampling at www.excellerate4success.compredetermined periods of time
  11. 11. High Frequency InformationSecurity Price Adjustments to Information The price of the security in the inefficient market begins adjusting before/after the news becomes public ( “information leakage” and “overshooting”) Many solid trading strategies exploit both the information leakage and overshooting to generate consistent profits Efficient and Inefficient Markets Inefficient market response Good News Bad News Efficient Efficient market response market response Information Arrival Time Information Arrival Time Inefficient market response Figure 2 Incorporation of information in efficient and inefficient markets6
  12. 12. High-Frequency StrategiesTrading on High-Frequency Information Traders leverage state-of-the-art IT technology to implement trading strategies that have high-frequency opportunities; High-frequency trading strategies typically fall into four main categories8. HFT-based Strategies Electronic Statistical Liquidity Others Liquidity Arbitrage Detection Provision Spread Market Neutral Sniffing/Pinging/ Latency Capturing Arbitrage Sniping Arbitrage Cross Asset, Cross Quote Matching Short Term Rebates Market & ETF Momentum
  13. 13. High-Frequency Strategies Liquidity Provision Strategies - Spread Capturing Liquidity providers profit from the spread between bid and ask prices by continuously buying and selling securities; Executed predominantly using limit orders Ask Asking Price Market Buy Orders Market Sell Orders Bid-Ask Spread Market Price Limit Buy Orders Limit Sell Orders Bid Offer PriceHigh-speed transmission of orders andlow-latency execution required forsuccessful implementation of liquidityprovision strategies. Market Transactions
  14. 14. High-Frequency StrategiesPredictions based on Real-Time access to the Limit Order Book (LOB) Direction of market price • Shape of limit order book is movement predictive of impending changes in market price10 • Exploited by market-maker traders; buy sell • Depends on probability distribution for arriving market orders; Direction of market price movement • Shape can be estimated when book not observable. buy sell
  15. 15. High-Frequency StrategiesStatistical Arbitrage “Stat-Arb” rests squarely on data mining. It finds statistical relationships in large amounts of data and builds a model of those relationships; Leverages states of the art technology to profit from small and short-lived discrepancies between securities; Arbitrageurs generate profits by selling the asset on the market where it is valued higher and simultaneously buying it on another market where it is valued lower.
  16. 16. High-Frequency Strategies Detecting Statistical Anomalies in Price Levels Identify securities that trade in frequency unit Once gap in prices reverse, close out position/stop loss Measure difference between prices of Sij ,t Si ,t S j ,t ,t 1, T identified securities Monitor and act upon differences in security pricesSt Si , S j, E S 2 SSt Si , S j, E S 2 S Select most stable T 2 relationships mini, j t 1 Sij ,t T Estimate T 1 2 distributional 1 St St E St E St St T 1t properties of the T 1 t 1 difference
  17. 17. High-Frequency StrategiesFundamental Arbitrage Strategies by Asset Class6 Asset Class Fundamental Arbitrage Strategy Foreign Exchange Triangular Arbitrage Foreign Exchange Uncovered Interest Parity (UIP) Arbitrage Equities Different Equity Classes of the Same Issuer Equities Market Neutral Arbitrage Equities Liquidity Arbitrage Equities Large-to-Small Information Spillovers Futures and the Underlying Asset Basis Trading Indexes and ETFs Index Composition Arbitrage Options Volatility Curve
  18. 18. Model Development/Deployment Model DevelopmentModels used in HFT Ideas • Linear Econometric Models • Academic research and • Autoregressive (AR) Estimation proprietary extensions • Moving Average (MA) Estimation • Autoregressive Moving Average (ARMA) • Cointegration Tools Volatility Modeling • To model observed volatility • Modeling predominantly clustering = ARMA or GARCH in Matlab /R, • c++ for back-tests and transition into production NonLinear Econometric Models Allows for modeling of complex nontrivial relationships in data Back Testing • Taylor series expansion • Threshold autoregressive model • Modeled relationships tested on lengthy • Markov switching model spans of tick data • Nonparametric estimation • Forecasting validity • Neural Networks • Various market situations
  19. 19. Model Development/Deployment Back-Testing Econometric Models Model Accuracy Analysis6Point Forecasts Accuracy Curve• predict price will reach certain level /point Random Forecast• regression of realized values 100 from historical data against out of Model C sample forecasts Model A Hit Rate (%)Directional Forecasts• makes decisions to enter into positions based on expectations of system going up or down (without target) Model BAccuracy Curves 0.0 Miss Rate (%) 100 %• compares the accuracy of probabilistic forecasts• HFT models done with TSA curves
  20. 20. Executing Real-Time Systems Execution Optimization Algorithms • Algorithms spanning order-execution processes • Designed to optimize trading execution once the buy- and-sell decisions have been made elsewhere best way to route the order to the exchange best point in time to execute a submitted order (non-market order) best sequence of sizes in which the order should be optimally processedCommon Types1) Market Aggressiveness Selection algorithms designed to choose between market1) and limit orders for optimal execution;2) Price-Scaling algorithms designed to select the best execution price according to1) the pre-specified trading benchmarks; and3) Size-optimization algorithms that determine the optimal ways to break down large1) parcels to minimize adverse costs (cost of market impact) trading lots into smaller
  21. 21. Executing Real-Time Systems Execution Optimization Algorithms Market Aggressiveness Price-Scaling Size Optimization Selection • Tries to obtain the best • Tries to trade with• Balances passive and price for the strategy position undetected aggressive trading using optimization Strike Algorithm • Large order packets are min Cost ( ) Risk( ) • Minimizes the cost of broken up for least execution relative to a benchmark amount of market impactCost ( ) Eo P( ) Pb (“Stealth Trading”) • Designed to capture gains inRisk ( ) ( ( )) periods of favorable pricesP( ) P f ( X , ) g( X ) ( ) 2 min Et Pt 1 ( t ) Pb ,tPb Benchmark execution price tP(a) Realized execution price Pt 1 ( t ) Realized price (a) Deviation of trading outcome t Trading aggressivenessP Market price at order entry Pb ,t Benchmark pricef ( X , a) Market Impact due to trade Plus Algorithmg( X ) Price impact due to info leak Wealth Algorithm
  22. 22. Executing Real-Time Systems HFT Business Cycle 1 1 – 4: run-time Receive/archive real- time tick data on 5 – 6: post-trade securities of interest6 2Ensure trading costs incurred Apply back-tested during execution are within econometric models to the acceptable ranges tick data obtained in 1 Each functions built with independent alert systems that notify monitoring personnel of problems,5 unusual patterns etc. Evaluate trading performance relative topredetermined benchmarks 3 Send orders and keep track of open positions/P&L values 4 Monitor run-time trading behavior, compare with predefined parameters, manage the run-time trading risk
  23. 23. Summary ThemesIntroducing Excellerate MetricsReal-Time BI and HFT Data Mining Information at High Frequency Big Data Strategies at High Frequency Developing and Deploying Models Meta Data Executing and Monitoring Real-Time Systems
  24. 24. Thank You Sean McClure, Ph.D. Business Analytics, Excellerate Inc.
  25. 25. References1) Richard Hackathorn, "Active Data Warehousing: From Nice to Necessary," Teradata Magazine (June 2006), AR-48352) High-Frrequency Trading, A Practical Guide to Algorithmic Strategies and Trading Systems, Aldridge, Wiley Trading, 20107) Dacorogna, M.M., R. Gencay, U.A. Muller, R. Olsen and O.V. Pictet, 2001. An introduction to High-Frequency Finance. Academic Press: San Diego, CA.8) High-Frequency Trading; Gomber, Arndt, Lutat, Uhle, Deutsche Borse Group9) Cao, C., O. Hansch and X. Wang, 2004. “The Informational Content of an Open Limit Order Book” Penn State University.