EGU2020-10385_presentation LSTM algorithm

Using AutoRegressive Integrated Moving Average
and Gaussian Processes with LSTM neural
networks to predict discrete geomagnetic signals
Authors: Laurentiu Asimopolos 1, Alexandru Stanciu 2,
Natalia-Silvia Asimopolos 1, Bogdan Balea 1,
Andreea Dinu 2, Adrian-Aristide Asimopolos 3
1 Geological Institute of Romania, Surlari Geomagnetic Observatory, BUCHAREST,
Romania (asimopolos@gmail.com)
2 National Institute for Research and Development in Informatics BUCHAREST, Romania
(alex@ici.ro)
3 Polytechnic University of Bucharest, Faculty of Transportation, BUCHAREST, Romania
(adrian.asimopolos@gmail.com)
Session EMRP2.5 – Electromagnetic induction in geophysics

In this paper, we present the results obtained for the geomagnetic data acquired
at the Surlari Observatory, located about 30 Km North of Bucharest - Romania.
The observatory database contains records from the last seven solar cycles, with
different sampling rates.
We used AR, MA, ARMA and ARIMA (AutoRegressive Integrated Moving
Average) type models for time series forecasting and phenomenological
extrapolation. ARIMA model is a generalization of an autoregressive moving
average (ARMA) model, fitted to time series data to predict future points in the
series.
We made spectral analysis using Fourier Transform, that gives us a relevant
picture of the frequency spectrum of the signal component, but without locating it
in time, while the wavelet analysis provides us with information regarding the time
of occurrence of these frequencies.
Wavelet allows local analysis of magnetic field components through variable
frequency windows. Windows with longer time intervals allow us to extract low-
frequency information, medium-sized intervals of different sizes lead to medium-
frequency information extraction, and very narrow windows highlight the high-
frequencies or details of the analysed signals.

We extend the study of geomagnetic data analysis and predictive modelling by
implementing a Long Short-Term Memory (LSTM) recurrent neural network that is capable
of modelling long-term dependencies and is suitable for time series forecasting. This method
includes a Gaussian process (GP) model in order to obtain probabilistic forecasts based on the
LSTM outputs.
The evaluation of the proposed hybrid model is conducted using the Receiver Operating
Characteristic (ROC) Curve that provides a probabilistic forecast of geomagnetic storm
events.
In addition, reliability diagrams are provided in order to support the analysis of the
probabilistic forecasting models.
The implementation of the solution for predicting certain geomagnetic parameters is
implemented in the MATLAB language, using the Toolbox Deep Learning Toolbox, which
provides a framework for the design and implementation of deep learning models.
Also, in addition to using the MATLAB environment, the solution can be accessed,
modified, or improved in the Jupyter Notebook computing environment.

REMARKS ABOUT SURLARI GEOMAGNETIC OBSERVATORY
Geomagnetic field study in Romanian stations has started with irregular measurements, late XIX-th
century. In 1943, the foundation of SNGO marks the beginning of a new era in the systematic study of
geomagnetic field by a continuous registration of its variations and by carrying out standard absolute
measurements in a fundamental station. Observatory location was thoroughly established, so that meets the
geomorphological and technological criteria. SNGO is located in Căldăruşani - Surlari astro-geodetic polygon,
in an area without magnetic field anomalies or significant local heterogeneity of electrical conductivity in the
basement and a sufficient distance from major industrial sources of disturbance.
The Observatory covers an area of 3.56 hectares and comprises seven buildings and underground
laboratory for geomagnetic sensors. Inside are installed specialized equipment’s to multiparametric monitoring
of fields of earth. These buildings were made during 1943-1969. The underground laboratory and the main
buildings were renovated and modernized in 2006-2008. The design of the special geomagnetic recording
laboratories was made after some well verified. Inside laboratories were built 18 specially designed pillars
embedded deep in the ground, which are mounted high-resolution sensors.

General view in SNGO
Absolute measurement laboratory in SNGO
The first magnetogram in October 16, 1943
in SNGO

Short description of network and infrastructure features at the SNGO:
• NAT local network access implemented natively by the Cisco-880 router
• VPN Remote access server in the Observer's local network for users located
in other locations implemented through Cisco IOS 12.4T
• DHCP automatic configuration of network settings implemented with ISC
dhcpd 3.1.1
• DNS address resolution by name and vice versa implemented with ISC BIND
9
• Integrated LDAP authentication of users in the local network implemented
with open ldap 2 2.4.12
• NTP clock synchronization service for all computers connected to the local
network implemented with ntpd 4.2.4
• Database server for storing data from all Surlari Geomagnetic Observatory
acquisition systems implemented with MySql 5.0.67
• Web server hosts the observatory's web site deployed with Apache 2 2.2.10
2) Automatically, periodically (e.g., averaged over 5 seconds or 60 seconds) to the
Web server.
• FTP server file transfer services for Intuitive users using Intranet vsftpd 2.0.7
• File server server for general use file storage with Samba 3.2.4

For implementation, these services have been grouped into 4 categories, taking into
account their specificity, the need for hardware resources to run, and critical dependency
between them. For implementation, the solution was chosen as each group to run on a
separate server. Cisco-880 router - runs NAT and VPN server services (already implemented
and active).
netserv * - server for common network services: DHCP, DNS, LDAP
dbserv * - database server
filer * - file server
webserv * - Web server and FTP
To reduce costs and reduce the number of computers installed, all are virtual machines
running under VMware Server 2 on a single physical server for a high reliability (RAID 1 for
dual hard drives, dual NIC mounted in fail-over architecture, UPS with monitoring software).
Modernization of the network infrastructure has been carried out so that it can provide all
the services necessary for the exploitation.
Description of each unit within the data server
Schematic overview of the database
acquisition and management program.

The software package consists of the following components:
- SQLServer.VI on the database server is designed to transmit measured values in
two ways:
1) upon request, to applications installed on computers of different users; users
can log in with a username and password to access the data stored on the database
server;
2) Automatically, periodically (e.g., averaged over 5 seconds or 60 seconds) to the
Web server.
Client VI, with multiple installations on different users' computers, has the role of
allowing them to create on their own computers text files containing measured
values retrieved from the database server (figure 3 and 4);
This allows the user to select the start and end moments of the time period for which
the SQL Server VI, application will be requested, copy the measured values;
The user can specify, where appropriate, the path and file name in which the
measured measurement values transmitted by the SQL Server VI application are
saved;
- Web Server VI is installed on the Web server and receives the measured values,
periodically transmitted by SQL Server VI and will save them in text files; The files
are daily, the file name is DD-MM-YY, and the files within a month will be stored in
the same folder, the monthly folder name being MM-YY.

- Java Client VI, is installed on the Web server and is called by the actions of the
various web page visitors and allows viewing of measured values in graphical form.
These software applications respect the SNGO internal network and public network
connections, thus centralizing data acquisition and online transmission on Web
Server.
Standard processing of geomagnetic data according to INTERMAGNET
requirements includes calculating the average at one minute of Geomagnetic
components.
Values and obtaining provisional data files, establishing the base level of records as
well as adopting the baseline level and making definitive data.
This infrastructure and organizing of database from SNGO allow us to monitoring
and make all processing procedures of data according to geomagnetic observatories
reequipments.

THE GEOMAGNETIC EQUIPMENT USED IN SURLARI OBSERVATORY
An important operation in the functioning of magnetometers is the calibration for establishing the
sensitivity of each component records (nT/mm). This is done by means of DC powered coils placed in the
directions perpendicular to the axis sensors. The DC power have a known intensity. The two systems have
operated continuously until 2003, producing analogue records on photographic paper.
It is also very important to obtain base level of records with absolute measurements made with theodolite
Matting Weissenberg, ground inductor and oscillations box in a first phase and then theodolite Matting
Weissenberg and quartz horizontal magnetometer for H component QHM. In addition to these devices in 1968
was brought Varian proton precession magnetometer measuring the scalar value of the total field.
Triaxial magnetometer MAG-03MC allows simultaneous recording of the components Hx (North
direction), Hy (on the East) and Hz (vertical direction down) of the geomagnetic field. The sensors of this
magnetometer are magnetic inductive type and are made of coils with a large number of turns and a magnetic
core with high permeability (Permalloy).
Characteristic response (floor type) of this magnetic sensor, shows a very good functionality for a band of
frequencies between 1Hz and 2000Hz

MAG 03 DAM logger has two connectors for analog input signal from two flux gate magnetometers and a RS-
232 connector with 25 pins for output signal. Logger is controlled by software developed in FORTRAN that allows
selecting the number of channels to be recorded (from 1-6), the choice of sampling rate (between one second and 10
seconds) and the measuring range. The sampling rate refers to the frequency of purchase and not the storage, it is 10
times lower.
MAG-01h DI Fluxgate Magnetometer produced by Bartington Instruments Ltd., England, with THEO 010b
nonmagnetic theodolite, measuring declination and inclination of the geomagnetic field in absolute terms. These
measurements are used to establish the base level of permanent records.
This type of magnetometer can be used mounted on a Wild T1 theodolite nonmagnetic. It can measure both
declination and inclination of the geomagnetic field with an accuracy of ± one second sexagesimal degree.
Geometrics G-856 proton precession magnetometer are used for absolute measurements of total magnetic field.
FGE vector magnetometer was built by Danish Meteorological Institute using three commercial fluxgate
sensors, mounted in a block of marble 12x12x12cm3 through the quartz tube is offset coils that ensure maximum
stability or drift to 3 nT / year. Variation with temperature of recorded values due sensors is below 0.2 nT/°C and of the
electronics, as 0.1 nT /°C.
GSM90 Overhauser proton magnetometer is a scalar magnetometer designed for magnetic observatories
and other applications (Volcanology), where stability and accuracy are absolutely necessary.
Torsion photoelectric magnetometer PSM

ARIMA models are the most general class of models for forecasting a time series which
can be made to be “stationary” by differencing, perhaps in conjunction with nonlinear
transformations. A random variable that is a time series is stationary if its statistical properties
are all constant over time. A stationary series has no trend. This condition means that its
autocorrelations (correlations with its own prior deviations from the mean) remain constant
over time, or equivalently, that its power spectrum remains constant over time. A random
variable of this form can be viewed as a combination of signal and noise, and the signal could
be a pattern of fast or slow mean reversion, or sinusoidal oscillation, or rapid alternation in
sign, and it could also have a periodical component. An ARIMA model can be viewed as a
“filter” that tries to separate the signal from the noise, and the signal is then extrapolated into
the future to obtain forecasts.
ARIMA(p,d,q) forecasting equation for a stationary time series is a linear (i.e., regression-
type) equation in which the predictors consist of lags of the dependent variable and/or lags of
the forecast errors.
The acronym ARIMA stands for Auto-Regressive Integrated Moving Average. Lags
of the stationarized series in the forecasting equation are called "autoregressive" terms, lags of
the forecast errors are called "moving average" terms, and a time series which needs to be
differenced to be made stationary is said to be an "integrated" version of a stationary series.
Random-walk and random-trend models, autoregressive models, and exponential smoothing
models are all special cases of ARIMA models.
A non-periodical ARIMA model is classified as an "ARIMA(p,d,q)" model, where:
• p is the number of autoregressive terms,
• d is the number of non-periodical differences needed for stationarity, and
• q is the number of lagged forecast errors in the prediction equation.

The forecasting equation is constructed as follows. First, let y denote the dth difference of
Y, which means:
If d=0: yt = Yt
If d=1: yt = Yt - Yt-1
If d=2: yt = (Yt - Yt-1) - (Yt-1 - Yt-2) = Yt - 2Yt-1+ Yt-2
The second difference of Y (the d=2 case) is not the difference from 2 periods ago. Rather, it
is the first-difference-of-the-first difference, which is the discrete analogue of a second
derivative, i.e., the local acceleration of the series rather than its local trend.
In terms of y, the general forecasting equation is:
ŷt = μ + ϕ1 yt-1 +…+ ϕp yt-p - θ1et-1 -…- θqet-q
Here the moving average parameters (θ’s) are defined so that their signs are negative in the
equation, following the convention introduced by Box and Jenkins.

Matlab code:
STSFMs_ARIMA.AR = [];
STSFMs_ARIMA.MA = [];
Lags = num2cell(nan(1,STSFMs_ARIMA.Seasonality));
STSFMs_ARIMA.SAR = Lags;
STSFMs_ARIMA.SMA = Lags;
else
STSFMs_ARIMA = arima('SMALags',12,'D',1,'SARLags',12,...
'Seasonality',12);
D1 = LagOp({1 -1},'Lags',[0,1]);
D12 = LagOp({1 -1},'Lags',[0,12]);
D = D1*D12;
dFinal_test = filter(D,Final_test);
STSFMs_ARIMA_1 = arima('SMALags', 12, 'D',1,'SARLags',12,'MALags',1,'ARLags',1,...
'Seasonality',12);
STSFMs_ARIMA_Minitab = arima('SMALags', 12,'SMA',0.9852,
'D',1,'SARLags',12,'SAR',0.1264,'MALags',1,'MA',0.0500,'ARLags',1,'AR',0.9688,...
'Seasonality',12,'Constant',-0.0002256,'Variance',2);
max_order = 3;
[aic_matrix,bic_matrix] = ARMA_model_order_tuning(Final_test,max_order);
end

Long Short Term Memory (LSTM) networks are a type of recurrent neural network
that is capable of long-term dependencies.
Conceptually, a recurring LSTM unit tries to "remember" all the past knowledge about
which the network is seen so far and to "forget" the irrelevant data. This is done by
introducing different layers with activation functions called "gates" for different purposes.
Each recurrent LSTM unit also maintains a vector called an "internal cell state" that
conceptually describes the information that was chosen to be retained by the previous
recurrent LSTM unit. An LSTM network comprises four different gates for different
purposes, as described below:
• Forget gate: determines the extent to which previous data can be forgotten.
• Input gate: determines the information to be written in the internal cell state.
• Input modulation gate: it is considered as a part of the input gate and is used to
modulate the information that the input gate will write on the internal cell state internally
adding nonlinearity to the information and normalizing the information. This is done to
reduce learning time, ensuring faster convergence. Although the actions of this gate are less
important than the others and are often treated as a concept that offers finesse, it is good
practice to include this in the structure of the LSTM unit.
• Output gate: determines which output (next hidden layer) is generated from the
internal state of the LSTM unit.

The architecture of an
LSTM cell
The implementation of the solution for predicting certain geomagnetic parameters is
implemented in the MATLAB language, using the Deep Learning Toolbox. It provides a
framework for the design and implementation of Deep Learning networks, with the
development of convolutional neural networks (ConvNets, CNN) and recurrent LSTM
networks for the classification or regression of image, time series and text data.
Also, in addition to using the MATLAB environment, the solution can be accessed,
modified or improved in the Jupyter Notebook computing environment.

Stages of implementing the solution
1. The data can be read as follows:
format short
omni_data = readtable('fisier.txt');
omni_data.Properties.VariableNames = {'Var1' 'Var2'
'Var3' 'Var4' 'Var5' 'Var6' 'Var7' 'Var8'};
data = table2cell(omni_data(:,8));
data = [data{:}];
2. Training data partitioning - 70% and testing - 30%:
numTimeStepsTrain = floor(0.7*numel(data));
dataTrain = data(1:numTimeStepsTrain+1);
dataTest = data(numTimeStepsTrain+1:end);
3. Data normalization
mu = mean(dataTrain);
sig = std(dataTrain);
dataTrainStandardized = (dataTrain - mu) / sig;

4. Preparation of training data
XTrain = dataTrainStandardized(1:end-1);
YTrain = dataTrainStandardized(2:end);
5. Defining the Long Short Term Memory network
numFeatures = 1;
numResponses = 1;
numHiddenUnits = 150;
layers = [ ...
sequenceInputLayer(numFeatures)
lstmLayer(numHiddenUnits)
fullyConnectedLayer(numResponses)
regressionLayer];
options = trainingOptions('adam', ...
'MaxEpochs',150, ...
'GradientThreshold',1, ...
'InitialLearnRate',0.005, ...
'LearnRateSchedule','piecewise', ...
'LearnRateDropPeriod',125, ...
'LearnRateDropFactor',0.2, ...
'Verbose',0, ...
'Plots','training-progress');

6. Training of the LSTM network
net = trainNetwork(XTrain,YTrain,layers,options);
7. Normalization of test data
dataTestStandardized = (dataTest - mu) / sig;
XTest = dataTestStandardized(1:end-1);
8. Effective prediction of the variable selected for analysis, based on the observed
data
net = resetState(net);
net = predictAndUpdateState(net,XTrain);
YPred = [];
numTimeStepsTest = numel(XTest);
for i = 1:numTimeStepsTest
[net,YPred(:,i)] =
predictAndUpdateState(net,XTest(:,i),'ExecutionEnvironment','gpu');
end

9. Calculation of the square error
%unstandardize data
YPred = sig*YPred + mu;
YTest = dataTest(2:end);
rmse = sqrt(mean((YPred-YTest).^2))
10. Display of observed data (YTest) and forecasted data (YPred)
figure
plot(dataTrain(1:end-1))
hold on
idx =
numTimeStepsTrain:(numTimeStepsTrain+numTimeStepsTest);
plot(idx,[data(numTimeStepsTrain) YPred],'.-')
hold off
ylabel("Values")
title("Forecast")
legend(["Observed" "Forecast"])

For the experimental part, we used an LSTM model to predict the Dst values 1h, 2h,
and 3h ahead, and a Gaussian Process classifier to predict the level of the geomagnetic
storms.
The training dataset has four months of hourly Dst values, starting from March 1st,
2001 until June 28th, 2001. This period includes the geomagnetic storm event that
occurred between March 31st – 1st April, as shown in next figure:
Training data,
1st March 2001 – 28th June 2001

The Gaussian Process multiclass classifier is used for geomagnetic storm prediction
depending on the level of activity. For instance, for Dst < -250 nT there is super
geomagnetic storm, for -50 nT < Dst < -250 nT there is an intense storm, and for Dst > -
50 nT a moderate storm. For a 3h ahead prediction, we obtained a 0.98 ROC AUC, as
presented in next figure:
ROC curve of the
Gaussian Process classifier

We have also analyzed the geomagnetic storm that occurred between December
14th and December 16th, 2006, as presented in next figure:
Geomagnetic storm
between 14th
December 2006 – 16th
December 2006

Geomagnetic activity
predicted by the LSTM
model (3h ahead)
The 3h ahead estimation of the Dst is presented in next figure:

ROC curve for the Gaussian
Process classifier (3h ahead)
Reliability diagram of the GP
classifier (3h ahead)
The evaluation of the Gaussian Process classifier is presented in the next two figures:
The LSTM model had an RMSE of 12.12 and a correlation coefficient of 0.83.

CONCLUSIONS
Forecasting time series data is an important subject in many domains, inclusive in
geomagnetism. Traditionally, there are several techniques to effectively forecast the next
lag of time series data such as univariate Auto Regressive (AR), univariate Moving Average
(MA), Simple Exponential Smoothing (SES), and more notably Auto Regressive Integrated
Moving Average (ARIMA) with its many variations. In particular, ARIMA model has
demonstrated its outperformance in precision and accuracy of predicting the next lags of
time series.
Long Short Term Memory (LSTM) is a recurrent neural network (RNN) that enables
support for time series and sequence data in a network. LSTM performs additive
interactions, which can help improve gradient flow over long sequences during training.
LSTM are best suited for learning long-term dependencies.

ACKNOWLEDGEMENT
We address our thanks by the Romanian Ministry of Education and Research for
financing of the projects:
•The realization of 3D geological / geophysical models for the characterization
of some areas of economic and scientific interest in Romania, with Contract no.
49N / 2019.
•Institutional capacities and services for research, monitoring and forecasting
of risks in extra-atmospheric space, acronym SAFESPACE, project
Nr.16PCCDI/2018, within PNCDIII.

EGU2020-10385_presentation LSTM algorithm

More Related Content

Similar to EGU2020-10385_presentation LSTM algorithm

Recently uploaded

EGU2020-10385_presentation LSTM algorithm

Editor's Notes