1. 1
INFORMS Philadelphia
November 2015
Bin Weng ( Email: bzw0018@auburn.edu)
Ph.D. Candidate of Industrial and System Engineering
Mohamed A. Ahmed (Email: mza0068@auburn.edu)
M.S. Candidate of Industrial and System Engineering
Fadel M. Megahed (Email: fmegahed@auburn.edu)
Assistant Professor of Industrial and System Engineering
Stock Market Prediction Using
Disparate Data Sources

2. 2Stock Market Prediction
 Why?
• The stock market is one
of the most important
way for companies to
raise money.
• About 48% Americans
invested in the stock
market as 2015 (CNBC).
• The successful prediction
of a stock’s future price
could yield significant
PROFIT.

3. 3Stock Market Prediction
 How?
Guess? Fundamental Analysis
Technical Analysis (Charting) Technological Methods

4. 4Stock Market Prediction
 Ray Dalio’s $165B Bridgewater Associates will start
a new artificial-intelligence unit to use predictive
analysis for trades. (Bloomberg, 2015)

5. 5Related Works
Paper Index Selected Papers
[1]
Predicting Financial Markets: Comparing Survey,
News, Twitter and Search Engine Data
[2] A fusion model of HMM, ANN and GA for stock market forecasting
[3] Twitter mood predicts the stock market
[4] Stock Market Prediction System with Modular Neural Networks
[5]
Empirical evaluation of an automated intraday stock recommendation
system incorporating both market data and textual news
[6] A Hybrid Machine Learning System for Stock Market Forecasting
[7]
Market Index and Stock Price Direction Prediction using Machine
Learning Techniques: An empirical study on the KOSPI and HSI
[8] Stock Market Prediction Using Disparate Data Sources (Proposed)

6. 6Related Works
Paper
Data Model
Target
Type of Stock
Market
Data
Technical
Indicator
Social
Media
News
Secondary
Variable
Time
Series
Logistic
Regression
Decision
Trees
Neural
Networks
Support
Vector
Machines
IT Index
Mix of
companies
[1]    
Price
Volume

[2] 

Price 
[3]    Movement 
[4]  

Buy and sell
signal

[5]       
Price
Volume

[6]   Movement 
[7]    Movement  
[8]          Movement  

7. 7Research Motivation
 Which sources of data have the most correlation
with the stock market time series?
 Which logical target has the best prediction
capability with regards to the stock movement?
 Which technological model is best at predicting
the stock movement?
 Can we construct a better model using disparate
data sources?

8. 8Data Sources

9. 9Process Overview

10. 10Data Sources
 Social Media and Internet Data
• “Financial news articles play a large role in influencing the
movement of a stock as humans react to the information.” (M.
Nardo etc. 2015)
• “Data on changes in how often financially
related Wikipedia pages were viewed have contained early signs
of stock market moves.” (H. Moat etc. 2013)
• Blog communication exhibits remarkable
predictive power. (M. Choudhury etc. 2008)

11. 11Data Sources
 Secondary Variables
• The data from Social Media and Internet always have high variability
(e.g. Moving Average, Momentum, Relative Strength Index).
• If the upward or downward movement in predicting variables had an
effect on the target movement?
• What range of the primary variables have predicting power over the
targets?
0
500
1000
1500
2000
2500
3000
3500
1/2/2014 2/2/2014 3/2/2014 4/2/2014 5/2/2014 6/2/2014
Google News & Blogs

12. 12Target Matrix
Target
Type
Method
1 Open (i+1) – Close (i)
2 Open (i+1) – Open (i)
3 Close (i+1) – Close (i)
4 Close (i+1) – Open (i)
5 Volume of trades moves as previous day

13. 13Data Fusion

14. 14Feature Selection
• Simplification of model
• Shorter training times
• Improve accuracy
• Enhanced generalization by reducing overfitting

15. 15Feature Selection
 Chord Diagram

16. 16Feature Selection
 Method :
Recursive feature
elimination (RFE)
 Coding : Python
with multiple
feature selection
package
Pseudo Code of RFE
* Code is available on https://github.com/binweng/SFS

17. 17Feature Selection
Target Variables
Target 1
Close Open High Low P/E Ratio
Wiki_3_day_disparity Wiki_5_day_disparity Wiki_10_day_disparity Wiki_Momentum_1 Wiki_ROC
Google_MA_5 Google_EMA_3 Google_3_Day_disparity Google_5_day_disparity RSI
Stochastic Ocillater Wiki_RSI Google_MA_4 William %R Google_MA_3
Target 2
Close Open High Low P/E Ratio
Wiki_5_day_disparity Wiki_Move Wiki_MA3_Move Wiki_EMA5_Move
Wiki_5day_disparity_M
ove
Google_EMA5_Move
Google_3day_disparity_
Move
Google_ROC_Move Google_RSI_Move Wiki_3_day_disparity
Stochastic Ocillater RSI_Move Wiki_RSI_Move Google_MA_6 Google_Move
Target 3
Close Open High P/E Ratio Stochastic_Move
Wiki_Monentum_1 Wiki_Move Wiki_MA3_Move Wiki_EMA5_Move Wiki_ROC_Move
Google_EMA5_Move
Google_3day_disparity_
Move
Google_ROC_Move Google_RSI_Move Wiki_10_day_disparity
RSI_Move Wiki_RSI_Move Wiki_3_day_disparity Google_Move Google_MA5_Move
Target 4
Close Open High Low P/E Ratio
RSI_Move Wiki_10_day_Disparity Wiki_Move Wiki_MA3_Move Wiki_EMA5_Move
Google_Move
Google_3day_disparity_
Move
Google_ROC_Move Google_RSI_Move William %R
Stochastic Ocillater Stochastic_Move
Wiki_3day_disparity_M
ove
Wiki_ROC_Move Wiki_RSI_Move
Target 5
Close Open High Low William %R
Wiki_Monentum_1 Wiki_RSI Google_MA_2 Google_MA_3 Google_MA_4
Google_MA_9 Google_3_day_disparity Google_5_day_disparity
Google_10_day_disparit
y
Wiki_10_day_disparity
Wiki_3_day_disparity Wiki_5_day_disparity Google_MA_6 Google_MA_7 Google_MA_8

18. 18Model Comparison

19. 19Model Comparison
Source: http://scikit-learn.org/stable/tutorial/machine_learning_map/

20. 20Model Comparison

21. 21Experimental Result
Paper 1 – B. Nair etc., 2010 Paper 2 – A. Chen, 2003

22. 22Experimental Result
• Comparison of Model Accuracy by information input

23. 23Experimental Result
• Evaluate the model using AUC

24. 24Experimental Result
Target Coincidence Matrix for SVM
Target1
Training 0 1 Testing 0 1
0 55 113 0 60 95
1 27 229 1 34 183
Target2
Training 0 1 Testing 0 1
0 160 28 0 156 39
1 37 180 1 32 164
Target3
Training 0 1 Testing 0 1
0 147 46 0 164 32
1 30 172 1 31 174
Target4
Training 0 1 Testing 0 1
0 150 31 0 165 34
1 34 172 1 31 179
Target5
Training 0 1 Testing 0 1
0 177 29 0 183 37
1 130 61 1 125 54

25. 25Target Matrix
Target
Type
Method
1 Open (i+1) – Close (i)
2 Open (i+1) – Open (i)
3 Close (i+1) – Close (i)
4 Close (i+1) – Open (i)
5 Volume of trades as previous day

26. 26Evaluation
 10 – fold cross validation

27. 27Evaluation
 Cross validation result

28. 28Evaluation
Accuracy: 82% - 89%

29. 29Moving Prediction

30. 30Conclusion
• Disparate sources of data help predict the stock market.
• Multiple targets’ prediction results can be used in
conjunction to successfully track stock market
movements.
• Decision tree model and support vector machine model
perform the best interchangeably with different
combinations of input data.
• With all the types of input data, SVMs performed best.

31. 31Future Work
• Identifying and adding into a more inclusive form of
this model, new sources of data that have a predictive
effect on the movement of the stock market, like twitter
sentiment and market news textual analysis.
• Include linguistic modeling, clustering, and controlling
methods like fuzzy theory in obtaining the predictions
of price range.
Fuzzy Membership FunctionFuzzy System

32. 32
INFORMS Philadelphia
November 2015
Bin Weng ( Email: bzw0018@auburn.edu)
Ph.D. Candidate of Industrial and System Engineering
Mohamed A. Ahmed (Email: mza0068@auburn.edu)
M.S. Candidate of Industrial and System Engineering
Fadel M. Megahed (Email: fmegahed@auburn.edu)
Assistant Professor of Industrial and System Engineering
Stock Market Prediction Using
Disparate Data Sources