Given the resurgence of neural network-based techniques in recent years, it is important for data science practitioner to understand how to apply these techniques and the tradeoffs between neural network-based and traditional statistical methods. This lecture discusses two specific techniques: Vector Autoregressive (VAR) Models and Recurrent Neural Network (RNN). The former is one of the most important class of multivariate time series statistical models applied in finance while the latter is a neural network architecture that is suitable for time series forecasting. I’ll demonstrate how they are implemented in practice and compares their advantages and disadvantages. Real-world applications, demonstrated using python and Spark, are used to illustrate these techniques. While not the focus in this lecture, exploratory time series data analysis using time-series plot, plots of autocorrelation (i.e. correlogram), plots of partial autocorrelation, plots of cross-correlations, histogram, and kernel density plot, will also be included in the demo. The attendees will learn – the formulation of a time series forecasting problem statement in context of VAR and RNN – the application of Recurrent Neural Network-based techniques in time series forecasting – the application of Vector Autoregressive Models in multivariate time series forecasting – the pros and cons of using VAR and RNN-based techniques in the context of financial time series forecasting – When to use VAR and when to use RNN-based techniques

1. Jeffrey Yau
Chief Data Scientist, AllianceBernstein, L.P.
Lecturer, UC Berkeley Masters of Information Data
Science
Time Series Forecasting Using
Recurrent Neural Network and
Vector Autoregressive Model

2. Motivation - Why MTSA?
2
Daily Interest Rates
Days
InterestRates

3. Agenda
Section I: Time series forecasting problem formulation
Section II: Multivariate (vs. univariate) time series
forecasting
Section III: Two (of the many) approaches to this problem:
– Vector Autoregressive (VAR) Models
– Long Short Term Memory (LSTM) Network
• Formulation
• Implementation
Section IV: Comparison of the two approaches
Section V: A few words on Spark Implementation
3

4. Section I
• Time series forecasting problem formulation
• Multivariate (vs. univariate) time series forecasting
• Two (of the many) approaches to this problem:
– Vector Autoregressive (VAR) Models
– Long Short Term Memory (LSTM) Network
• Formulation
• Implementation
• Comparison of the two approaches
• A few words on Spark Implementation
4

5. Forecasting: Problem Formulation
• Forecasting: predicting the future values of the series
using current information set
• Current information set consists of current and past
values of the series and other exogenous series
5

6. Time Series Forecasting Requires Models
7
A statistical model or a
machine learning
algorithm
Information Set:
Forecast
horizon: h

7. What models?
9
A model, f(), could be as simple as “a rule” - naive model:
The forecast for tomorrow is the observed value today
Forecast
horizon: h=1
Information Set:

8. “Rolling” Average Model
10
It could be a slightly more complicated model
The forecast for time t+1 is an average of the observed
values from a predefined, past k time periods
Forecast horizon: h=1 Information Set:

9. More Sophisticated Model
11
… but there are other, more sophisticated model that
utilize the information set much more efficiently and aim
at optimizing some metrics

10. Section II
• Time series forecasting problem formulation
• Multivariate (vs. univariate) time series forecasting
• Two (of the many) approaches to this problem:
– Vector Autoregressive (VAR) Models
– Long Short Term Memory (LSTM) Network
• Formulation
• Implementation
• Comparison of the two approaches
• A few words on Spark Implementation
12

11. Univariate Time Series Models
Statistical relationship is unidirectional
13
values from its own series
exogenous seriesDynamics of series y

12. Autoregressive Integrated Moving Average
(ARIMA) Model
14
values from the series shocks / “error” terms exogenous series
Limitation: cannot model cross equation dynamics (or
cross series dependency)

13. Multivariate Time Series Modeling
15
Kequations
lag-1 of the K series
lag-p of the K series
exogenous series
Dynamics of each of the series Interdependence among the series

14. Section III.A
• Time series forecasting problem formulation
• Multivariate (vs. univariate) time series forecasting
• Two (of the many) approaches to this problem:
– Vector Autoregressive (VAR) Models
– Long Short Term Memory (LSTM) Network
• Formulation
• Implementation
• Comparison of the two approaches
• A few words on Spark Implementation
16

15. Joint Modeling of Multiple Time Series
17

16. ● a system of linear equations of the K series being
modeled
● only applies to stationary series
● non-stationary series can be transformed into
stationary ones using simple differencing (note: if the
series are not co-integrated, then we can still apply
VAR ("VAR in differences"))
Vector Autoregressive (VAR) Models
18

17. Vector Autoregressive (VAR) Model of
Order 1
19
K series 1-lag each
AsystemofKequations

18. General Steps to Build VAR Model
20
1. Load the series
2. Train/validation/test split the series
3. Conduct exploratory time series data analysis on the training set
4. Determine if the series are stationary
5. Transform the series
6. Build a model on the transformed series
7. Model diagnostic
8. Model selection (based on some pre-defined criterion)
9. Conduct forecast using the final, chosen model
10. Inverse-transform the forecast
11. Conduct forecast evaluation
Iterative

19. ETSDA - Index of Consumer Sentiment
21
autocorrelation function
(ACF) graph
Partial autocorrelation
function (PACF) graph

20. ETSDA - Beer Consumption Series
22
Autocorrelation function

21. Transforming the Series
23
Take the simple-difference of the natural logarithmic transformation of
the series
note: difference-transformation
generates missing values

22. ETSDA on the Transformed Series
24

23. Is the method we propose capable of answering the following questions?
● What are the dynamic properties of these series? Own lagged
coefficients
● How are these series interact, if at all? Cross-series lagged coefficients
VAR Model Proposed
25

24. VAR Model Estimation and Output
26

25. VAR Model Output - Estimated Coefficients
27put your #assignedhashtag here by setting the footer in view-header/footer

26. VAR Model Output - Var-Covar Matrix
28

27. VAR Model Diagnostic
29
UMCSENT Beer

28. VAR Model Selection
30
Model selection, in the case of VAR(p), is the choice of the order and
the specification of each equation
Information criterion can be used for model selection:

29. VAR Model - Inverse Transform
31
Don’t forget to inverse-transform the forecasted series!

30. VAR Model - Forecast Using the Model
32
The Forecast Equation:

31. VAR Model Forecast
33
where T is the last observation period and l is the lag

32. What do the result mean in this context?
34
Don’t forget to put the result in the existing context!

33. Section III.B
• Time series forecasting problem formulation
• Multivariate (vs. univariate) time series forecasting
• Two (of the many) approaches to this problem:
– Vector Autoregressive (VAR) Models
– Long Short Term Memory (LSTM) Network
• Formulation
• Implementation
• Comparison of the two approaches
• A few words on Spark Implementation
35

34. Standard feed-forward network
36
• network architecture does
not have "memory" built in
• inputs are processed
independently
• no “device” to keep the past
information

35. Vanilla Recurrent Neural Network (RNN)
37
A structure that can retain past information, track the state of the
world, and update the state of the world as the network moves forward

36. Limitation of Vanilla RNN Architecture
38
Exploding (and vanishing) gradient problems
(Sepp Hochreiter, 1991 Diploma Thesis)

37. Long Short Term Memory (LSTM) Network
39

38. LSTM: Hochreiter and Schmidhuber (1997)
40
The architecture of memory cells and gate units from the original
Hochreiter and Schmidhuber (1997) paper

39. LSTM
41
Another representation of the architecture of memory cells and gate
units: Greff, Srivastava, Koutnık, Steunebrink, Schmidhuber (2016)

40. (Vanilla) LSTM: 1-minute Overview
42
Chris Colah’s blog gives an excellent introduction to vanilla RNN and LSTM:
http://colah.github.io/posts/2015-08-Understanding-LSTMs/#fnref1
•
Use memory cells and gated units to information flow
LSTM Memory
Cell
ht-1 h
t
Training uses Backward Propagation Through Time (BPTT)
memory cell (state)
hidden state
current input
output gate
forget gate
input gate

41. Implementation in Keras
Some steps to highlight:
• Formulate the series for a RNN supervised learning regression problem
(i.e. (Define target and input tensors)
• Scale all the series
• Define the (initial) architecture of the LSTM Model
○ Define a network of layers that maps your inputs to your targets and
the complexity of each layer (i.e. number of neurons)
○ Configure the learning process by picking a loss function, an
optimizer, and metrics to monitor
• Produce the forecasts and then reverse-scale the forecasted series
• Calculate loss metrics (e.g. RMSE, MAE)
43
Note that stationarity, as defined previously, is not a requirement

42. Consumer Sentiment and Housing Starts
44

43. LSTM Architecture Design and Training -
A Simple Illustration
45

44. LSTM: Forecast Results (1)
46
Monthly series:
● Training set: 1981 Jan - 2016 Dec
● Test set: 2017 Jan to 2018 Apr

45. LSTM: Forecast vs Observed Series
47
Housing Stars Consumer Sentiment

46. Section IV
• Time series forecasting problem formulation
• Multivariate (vs. univariate) time series forecasting
• Two (of the many) approaches to this problem:
– Vector Autoregressive (VAR) Models
– Long Short Term Memory (LSTM) Network
• Formulation
• Implementation
• Comparison of the two approaches
• A few words on Spark Implementation
48put your #assignedhashtag here by setting the footer in view-header/footer

47. VAR vs. LSTM: Data Type
macroeconomic time
series, financial time
series, business time
series, and other numeric
series
49
images, texts, all the
numeric time series (that
can be modeled by VAR)
VAR LSTM

48. layers of non-linear
transformations of input
series
VAR vs. LSTM: Parametric form
a linear system of
equations - highly
parameterized
50
VAR LSTM

49. • not require
stationarity
VAR vs. LSTM: Stationarity Requirement
• applied to stationary
time series only
• its variant (e.g. Vector
Error Correction
Model) can be applied
to co-integrated series
51
VAR LSTM

50. • data preprocessing is a
lot more involved
• model training and
hyper-parameter
tuning are time
consuming
VAR vs. LSTM: Model Implementation
• data preprocessing is
straight-forward
• model training time is
fast
52
VAR LSTM

51. Section IV
• Time series forecasting problem formulation
• Multivariate (vs. univariate) time series forecasting
• Two (of the many) approaches to this problem:
– Vector Autoregressive (VAR) Models
– Long Short Term Memory (LSTM) Network
• Formulation
• Implementation
• Comparison of the two approaches
• A few words on Spark Implementation
53put your #assignedhashtag here by setting the footer in view-header/footer

52. Implementation using Spark
54
Straightforward to use Spark for all the steps (outlined above) up to model
training, such as ...
Ordering in the series matterWait … Not So Fast!

53. Flint: A Time Series Library for Spark
55
Need time-series aware libraries whose operations preserve the natural
order of the series, such as Flint, open sourced by TwoSigma
(https://github.com/twosigma/flint)
- can handle a variety of datafile types, manipulate and aggregate time
series, and provide time-series-specific functions, such as (time-series)
join, (time-series) filtering, time-based windowing, cycles, and
intervalizing)
Example: Defining a moving average of the beer consumption series

54. A Few Words on TSA Workflow in Spark
56
Load, pre-process, analyze,
transform series using
time-series aware libraries
Conduct Hyperparameter
Tuning in a distributed
model and use Spark ML
Pipeline

55. Thank You!
57