This document summarizes a presentation about using recurrent neural networks like LSTMs and AWS Lambda for real-time energy price forecasting and optimization. The objectives are to provide a cost-effective real-time market price forecasting service and scalable distributed solution. LSTMs are well-suited for this task as they can learn nonlinear relationships in time series data and remember long-term dependencies. The architecture proposed uses AWS Lambda to run prediction code in response to new time series data and store results in S3.

1. OPEN DATA SCIENCE
CONFERENCE
Real Time Training and Forecasting using LSTM/RNNs &
AWS Lambda
Anoop Vasant Kumar
Data Scientist
Hitachi Consulting, UK

2. Understanding the problem
Nuclear Power Phase Out
Switch to low carbon,
environment friendly
and affordable energy.
Forecast of Energy Price
Trading Optimization
and Energy Price
Forecast
Generation Optimization
Power Generation
Optimization

3. Project objective #1:
Cost effective method to provide real
time market price forecasting service.

4. Project objective #2:
A scalable and distributed solution,
that forecasts prices for each of the
100+ energy products based on live
market input feeds.

5. High Price 48 Euro
Generation Cost 30 Euro
Low Price 10 Euro
T-5 T-4 T-3 T-3 T-2 T-1 T
Time Series of a typical energy product

6. Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product

7. Model
A model that could learn complex
nonlinear relationships
Multiple features with each input
being a time series on it own
Ability to remember the past trend
and sequence
Ability to consider long term
dependencies in data
Ability to improve forecasts and
update model parameters real
time based on new data inputs

8. Correlation of Products

9. Recurrent Neural Networks
are networks with loops in
them, allowing information
to persist.
It makes use of sequential
information unlike
traditional neural
networks.
RNN - A network that remembers

10. Long Short Term Memory - Recurrent Neural Networks
Remembering long term dependencies is their
default nature.
Think of LSTMs as exactly same as RNNs except
the method is which the hidden state is calculated,
is slightly different!.

11. Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product

12. LSTM - Unrolled

13. X
y

14. Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product

15. anoop.vasant.kumar@gmail.com
Performance
Product 2
Date upto which
Neural Network
is Trained : 2016-
07-24

16. Compute as-a service
Provisioning and managing of the servers to run code.
Event-driven compute service - runs code in response
to events, such as changes to data in an Amazon
S3 bucket.
Runs code on a high-availability compute
infrastructure and performs all of the
administration.
Compute Infrastructure - AWS Lambda

17. anoop.vasant.kumar@gmail.com
High Level Architecture /Neural Networks on the Cloud
High Speed Data
Connection
Scheduled Daily Lambda Trigger
PullDataFeedsand
DataPreprocess
Pandas
DataFrame
Prd 1
Prd 2
Prd 3
Object Upload
S3 Object Upload Event Notification
Prediction Lambda Triggered Per Object Upload
Prd 120
Prd 5
Prd 4
Prd 3
Prd 1
Prd 2
Prd 120
Prd 1
Prediction Date
2016-08-03
Key Value
Prd1 36
Prd2 29
…
Prd 120 41
Amazon API
Gateway
Time Series Product 2 in Market abc
Time Series - Product a
Time Series Product b
Preprocessed Data
Predicted Prices

18. X
y

19. anoop.vasant.kumar@gmail.com
Email: anoop.vasant.kumar@gmail.com
Twitter: @vasant_anoop