Probabilistic time series modelling using Bayesian neural networks

Abstract

Interpretable models play a key role in structural health monitoring (SHM) systems dealing with time series data. Interpretable models provide users with meaningful insights extracted from the raw data such as the level describing the irreversible structural behaviour, the trend describing its rate of change, and the seasonality isolating the reversible effects caused by environmental factors. Bayesian dynamic linear models (BDLM) [1] are a special type of state-space models (SSM); this probabilistic method provides interpretable results by decomposing raw data into reversible and irreversible effects. In BDLM, the harmonic and non-harmonic reversible effects can be modelled by the periodic components. However, estimating the parameters for these components relies on optimization techniques. The key limitation preventing the scaling of BDLM to a large number of time series is that it requires feature engineering to define the dependencies between the model's components. The long-short term memory (LSTM) [2] is a widely used neural network architecture to model time series. LSTM is capable of modelling complex patterns such that it could replace the BDLM's periodic components, and automatically model the non-linear dependencies within and between time series. The limitation of the existing LSTM methods is that they either rely on deterministic parameter estimation which fails to take into account epistemic uncertainties and only provides point-estimates for predictions, or they rely on gradient-based optimization techniques that are incompatible with SSM. As a result, up to now LSTM could not be coupled analytically with probabilistic time series models.

The objective of this paper is twofold; first, we develop the formulation for Bayesian LSTM neural networks (TAGI-LSTM) taking into consideration both epistemic and aleatory uncertainties which enables providing the parameter and predictive uncertainties. The weight and bias parameters of TAGI-LSTM are estimated analytically using the Bayes' theorem instead of the optimization-based backpropagation algorithm. Second, we couple TAGI-LSTM with BDLM in a probabilistic manner resulting in a novel Bayesian hybrid model inheriting the best features from both models. Our method provides interpretable results along with the prediction uncertainties while not requiring feature engineering nor optimization. The results from case studies show that the hybrid model provides a competitive performance compared with BDLM while being scalable to process a thousand of time series.

References

[1] James-A. Goulet. Bayesian dynamic linear models for structural health monitoring. Structural Control and Health Monitoring, 24(12):e2035, 2017

[2] Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural computation, 9:1735–80, 1997