https://www.bigdataspain.org/2016/program/thu-turning-idea-into-data-driven-production-system-energy-load-forecasting-case-study.html https://www.youtube.com/watch?v=Df59GmPcVag&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&t=26s&index=58

2. BIG DATA SPAIN 2016 1
BIG DATA SPAIN 2016
© 2015 The MathWorks, Inc.
Turning an idea into a Data-Driven
Production System
An Energy Load Forecasting Case Study
Lucas García
Senior Application Engineer
MathWorks

3. BIG DATA SPAIN 2016 2
What is Energy Forecasting?
From Wikipedia:
Energy forecasting is a broad term that refers to
"forecasting in the energy industry".
It includes - but is not limited to - forecasting demand
(load) and price of electricity, fossil fuels (natural
gas, oil, coal) and renewable energy sources (RES;
hydro, wind, solar).

4. BIG DATA SPAIN 2016 3
What is Data Analytics?
• What happened?Descriptive
• Why did it happen?Diagnostics
• What will happen?Predictive
• What should be done?Prescriptive
Turn large volumes of complex data into actionable information
Data Decisions

5. BIG DATA SPAIN 2016 4
Data Analytics – Using Data to Make Better Decisions
Develop Predictive
Models
Access and Explore
Data
Preprocess Data
Integrate Analytics with
Systems

6. BIG DATA SPAIN 2016 5
Goal:
 Implement a tool for easy and accurate computation of day-ahead system load forecast
Requirements:
 Acquire and clean data from multiple
sources
 Accurate predictive model
 Easily deploy to production environment
Case Study: Day-Ahead Energy Load Forecasting

7. BIG DATA SPAIN 2016 6
The Data
mis.nyiso.com/public/
NYISO Energy Load Data
cdo.ncdc.noaa.gov/qclcd_ascii/
National Climatic Data Center Weather Data

8. BIG DATA SPAIN 2016 7
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction

9. BIG DATA SPAIN 2016 8
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
1

10. BIG DATA SPAIN 2016 9
Data Analytics Workflow
Files
Databases
Sensors
Access and Explore
Data
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
 Repositories – SQL, NoSQL, etc.
 File I/O – Text, Spreadsheet, etc.
 Web Sources – RESTful, JSON, etc.
Business and Transactional Data
Engineering, Scientific and Field Data
 Real-Time Sources – Sensors, GPS, etc.
 File I/O – Image, Audio, etc.
 Communication Protocols – OPC (OLE for
Process Control), CAN (Controller Area
Network), etc.

11. BIG DATA SPAIN 2016 10
Data Analytics Workflow
Files
Databases
Sensors
Access and Explore
Data
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
 Data aggregation
– Different sources (files, web, etc.)
– Different types (images, text, audio, etc.)
 Data clean up
– Poorly formatted files
– Irregularly sampled data
– Redundant data, outliers, missing data etc.
 Data specific processing
– Signals: Smoothing, resampling, denoising,
Wavelet transforms, etc.
– Images: Image registration, morphological
filtering, deblurring, etc.
 Dealing with out of memory data (big data)
Challenges

12. BIG DATA SPAIN 2016 11
Data Analytics Workflow
Files
Databases
Sensors
Access and Explore
Data
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
 Point and click tools to access
variety of data sources
 High-performance environment
for big data
Files
Signals
Databases
Images
 Built-in algorithms for data
preprocessing including sensor,
image, audio, video and other
real-time data
MATLAB Analytics work
with business and
engineering data
1

13. BIG DATA SPAIN 2016 12
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
1 2

14. BIG DATA SPAIN 2016 13
Data Analytics Workflow
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Challenges
 Lack of data science expertise
 Feature Extraction – How to transform
data to best represent the system?
– Requires subject matter expertise
– No right way of designing features
 Feature Selection – What attributes or
subset of data to use?
– Entails a lot of iteration – Trial and error
– Difficult to evaluate features
 Model Development
– Many different models
– Model Validation and Tuning
 Time required to conduct the analysis
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction

15. BIG DATA SPAIN 2016 14
Data Analytics Workflow
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
MATLAB enables
domain experts to
do Data Science
2
Apps Language
 Easy to use apps
 Wide breadth of tools to facilitate
domain specific analysis
 Examples/videos to get started
 Automatic MATLAB code
generation
 High speed processing of large
data sets

16. BIG DATA SPAIN 2016 15
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
1 2 3

17. BIG DATA SPAIN 2016 16
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
 End user: Operators, Analysts,
Administrative Staff, customers etc.
 Different target platforms:
– Cluster or Cloud environment
– Standalone desktop applications
– Server based Web and enterprise systems
– Embedded hardware
 Different Interfaces: C++, Java, Python,
.NET etc.
 Need to translate analytics to production
environment
Challenges

18. BIG DATA SPAIN 2016 17
Integrate analytics with systems
MATLAB
Runtime
C, C++ HDL PLC
Embedded Hardware
C/C++ ++
Excel
Add-in Java
Hadoop/
Spark
.NET
MATLAB
Production
Server
Standalone
Application
Enterprise Systems
Python
MATLAB Analytics
run anywhere
3

19. BIG DATA SPAIN 2016 18
MATLAB
Desktop
Deployed Analytics
MATLAB Production Server
MATLAB
Production
Server
Web
Application
Server
MATLAB
Production Server
RequestBroker
CTF
Apache Tomcat
Web Server/
Webservice
Weather
Data
Energy
Data
Predictive
Models
Train in
MATLAB

20. BIG DATA SPAIN 2016 19
Key Takeaways
 Utilize all of your data
 Apply advanced analytics techniques
 Operationalize analytics to enterprise
systems and embedded devices
MATLAB Analytics work
with business and
engineering data
1
MATLAB enables
domain experts to do
Data Science
2
3MATLAB Analytics
run anywhere

21. BIG DATA SPAIN 2016 20
Thank you!
Stay tuned: Twitter: @MATLAB | LinkedIn: https://www.linkedin.com/company/the-mathworks_2
% Send me your feedback:
% lucas.garcia@mathworks.com
% Twitter: @mathinking