The document discusses a framework for improving graph-based spatiotemporal forecasting by addressing the interplay between global and local effects in time series forecasting models. It outlines methodologies for incorporating learnable node embeddings and regularization techniques that enhance model performance, particularly in transfer learning contexts. Empirical results from various experiments demonstrate the effectiveness of the proposed approaches in enhancing forecasting accuracy and efficiency.

1. Quang-Huy Tran
Network Science Lab
Dept. of Artificial Intelligence
The Catholic University of Korea
E-mail: huytran1126@gmail.com
2024-04-22
Taming Local Effects in Graph-based
Spatiotemporal Forecasting
Andrea Cini et al.
NeurIPS’37: 2023 Conference on Neural Information Processing Systems

2. 2
OUTLINE
• MOTIVATION
• INTRODUCTION
• PROBLEM FORMULATION
• METHODOLOGY
• EXPERIMENT & RESULT
• CONCLUSION

3. 3
MOTIVATION
• Graph-based methods are effective in forecasting collections of time series.
• Deep learning methods combine sequence-processing operators with message
passing.
o A single (global) inductive model is trained to predict any time series associated with a node.
o It is common practice to use node-specific (local) parameters to help node identification.
 Improved modeling of local effects and accuracy.
 Inductive capabilities compromised.
Overview graph

4. 4
INTRODUCTION
• Understanding the interplay between globality and locality in graph-based models.
• A methodological framework for designing node-specific components effectively.
• Methods to ease transferability of the global components to new nodes/graphs.

5. 5
PROBLEM FORMULATION
• We consider a set of 𝑁 correlated time series, where each 𝑖-th time series associated
with:
o an observation vector Xt = 𝑥𝑡
𝑖
∈ ℝ𝑑𝑥 at each time step t.
o a vector of exogenous variable Ut = 𝑢𝑡
𝑖
∈ ℝ𝑑𝑢 at each time step t;
Collections of time series

6. 6
PROBLEM FORMULATION
Relational Information
• Assume the existence of functional dependencies between time
series:
o i.e. forecasts for one time series can be improved by
accounting for the past values of other time series.
• Model pairwise relationships existing at time step t with
adjacency matrix 𝐴 ∈ ℝ𝑁×𝑁.
o A can be asymmetric (directed graph).
• Spatial is called the dimension spanning the time series
collection.

7. 7
PROBLEM FORMULATION
Time Series Forecasting
• Multi – step time series forecasting:
o Given a window of observations 𝑋𝑡−𝑊:𝑡, the goal is to predict the next 𝐻 observations 𝑋𝑡:𝑡+𝐻
𝑋𝑡:𝑡+𝐻 = 𝑓(𝑋𝑡−𝑤:𝑡)
• The deep learning approach to forecasting consists of
training:
o A global neural network (NN).
o local node specification NN.

8. 8
METHODOLOGY
Global and Local Forecasting
• Global forecasting model: if its parameters are fitted to a group of time series.
o All learnable parameters are shared.
o More data available for training.
o Can be used in inductive learning.
• Local forecasting model: specific to a single time series.
o Capture better series-specific dynamics.
o Often require shorter input windows.
o Or reduced model capacity.

9. 9
METHODOLOGY
Relational inductive biases
• Both approaches share a drawback: dependencies across time are often discarded.
Message Passing Neural Network for Spatiotemporal GNNs: Spatiotemporal
Message Passing (STMP)
• Embed relational information as an
architectural bias into the processing.
• Graph Neural Network (GNN) provide
appropriate neural operators.

10. 10
METHODOLOGY
Architecture
• a sequence of three operators:
• Encoding layer: an MLP.
• A stack of L STMP layers.
• Readout layer: an MLP.

11. 11
METHODOLOGY
Spatiotemporal message passing - Globality
• The cornerstone operator in STGNNs is the STMP layer
where 𝜌𝑙: update function of layer 𝑙 and 𝛾𝑙: message function of layer 𝑙.

12. 12
METHODOLOGY
Globality and Locality in STGNNs
• Limitation of Global model:
• Hybrid global-local STGNNs with
specialized local components:
o Struggle to model local effects.
o Require large model capacity or impractically
long windows.
o Node-level effects are captured more
efficiently than by fully global models.
o Forecasting accuracy on the task is usually
higher empirically.

13. 13
METHODOLOGY
Limits of global-local STGNNs
• Local components in a global STGNN’s disadvantages:
o Model’s inductive capabilities are compromised (hard to handle unseen time series).
o The number of learnable parameters can be much larger compared to fully global model.

14. 14
METHODOLOGY
Learnable node embeddings
• Mitigate drawbacks by using node embeddings, a table of learnable parameter
associated with each node:
o Fed into global STGNN and learned end-to-end.
• Amortize cost of specializing the model to each time series:
o A single vector for each node is added to the model’s parameters.
o Same vector can be used in multiple components of the architecture.
• Transfer the learned model to a different set of 𝑁′
time series more easily:
o Only 𝑁′𝑑𝑣 parameters need to be tuned, while the shared components are fixed.
o The embedding space can be regularized to better fit embeddings of new nodes.

15. 15
METHODOLOGY
Structuring the Embedding space
• Two strategies for regularizing the embedding space:
o Variational: a smoother embedding space to enable interpolation.
 Model each node embedding as sample from a multivariate Gaussian under sampling t
where (𝜇𝑖, 𝜎𝑖): learnable (local) parameters.
where 𝑃 = 𝑁 0, Ι : prior,
𝐷𝐾𝐿 : Kullback-Leibler divergence.
𝛽: controls the regularization strength.

16. 16
METHODOLOGY
Structuring the Embedding space
• Two strategies for regularizing the embedding space:
o Clustering: make clusters in the latent space to improve interpretability.
 Add a matrix 𝐶 ∈ ℝ𝐾×𝑑𝑣 of K ≪ 𝑁 learnable centroids and a cluster assignment matrix 𝑆 ∈
ℝ𝑁×𝐾
containing node-cluster pair scores.
where 𝜏: hyperparameter.

17. 17
EXPERIMENT AND RESULT
EXPERIMENT
• Measurement:
o Mean Absolute Errors (MAE)
• Dataset:
o GPVAR(-L): a graph with 20 communities resulting in a network.
o Traffic flow forecasting: METR-LA and PEMS-BAY. For transfer learning, PEMS03, PEMS04, PEMS07,
and PEMS08 dataset are used.
o Electric load forecasting : CER-E
dataset, a collection of energy
consumption.
o Air quality monitoring: AQI dataset
collects hourly measurements of
pollutant PM2.5 in China.

18. 18
• Baseline:
o RNN: global univariate RNN sharing the same parameters across the time series.
o FC-RNN: multivariate RNN taking as input the time series as if they were a multivariate one.
o LocalRNNs: local univariate RNNs with different sets of parameters for each time series.
o DCRNN[1]: recurrent T&S model with the Diffusion Convolutional operator.
o AGCRN[2]: T&S global-local Adaptive Graph Convolutional Recurrent Network.
o GraphWaveNet[3]: deep T&S spatiotemporal convolutional network.
EXPERIMENT AND RESULT
EXPERIMENT
[1] Li, Y., Yu, R., Shahabi, C., & Liu, Y. (2017). Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926.
[2] Bai, L., Yao, L., Li, C., Wang, X., & Wang, C. (2020). Adaptive graph convolutional recurrent network for traffic forecasting. Advances in neural information processing systems, 33, 17804-17815.
[3] Wu, Z., Pan, S., Long, G., Jiang, J., & Zhang, C. (2019). Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint arXiv:1906.00121.

19. 19
EXPERIMENT AND RESULT
RESULT

20. 20
EXPERIMENT AND RESULT
RESULT – Visualization Analysis

21. 21
EXPERIMENT AND RESULT
RESULT – Additional Experiments Results
• Analysis on Local components.
o The performance of reference architecture with and without local components

22. 22
EXPERIMENT AND RESULT
RESULT – Additional Experiments Results
• Analysis on Transfer Learning

23. 23
EXPERIMENT AND RESULT
RESULT – Additional Experiments Results
• Analysis on Transfer Learning

24. 24
CONCLUSION
• Investigate the impact of locality and globality in graph-based spatiotemporal
forecasting architectures.
• Propose a framework to explain empirical results associated with the use of trainable
node embeddings
o discuss different architectures and regularization techniques to account for local effects.
• The proposed methodologies are thoroughly empirically validated:
o effective in a transfer learning context.
• Future works can build on the results presented here and study alternative, and even
more transferable, methods to account for local effects.