LeS Data ScienceS

1. Les Data Sciences
L’exemple du problème du
‘Unit Commitment’
Alain Chabrier
Decision Optimization STSM, IBM
alain.chabrier@ibm.com
@AlainChabrier

2. IBM Cloud / Data Science Elite/ September, 2018 / © 2018 IBM Corporation

4. https://demanda.ree.es/visiona/seleccionar-sistema

5. Le problème du ‘Unit Commitment’
5
• Thermiques
• Renouvelables
• Contraintes
opérationnelles
Unités
• Configuration
du réseau
• Opérations
Réseau
• Demande
planifiée et non
planifiée
Charge
• Renouvelables
• Demande non
planifiée
Variabilité
• Court terme
• Moyen terme
• Long terme
Planification

6. Les défis
6
Variabilité des sources renouvelables qui ont une part
de plus en plus grande, et de la charge
Une production renouvelable insuffisante peut
provoquer une défaillance globale du réseau.
Eviter les coupures par surcharge involontaire en
provisionnant une réserve (‘spinning reserve’),
La réserve amène une augmentation des coûts de
production!
→ Besoin d’adapter dynamiquement la planification et
la réserve en fonction de prévisions stochastiques de la
génération renouvelable et de la demande.

7. LOT OF DATA
7

8. KNOWN DATA
Grid infrastructure
Committed
industrial
consumption
Units operating
limits
8

9. Renewable production
Demand
UNKNOWN DATA
9

10. International electricity price
Gas/oil
competition
on heating
OTHER’S DECISIONS
10

11. Which units to
operate and
how much?
MY DECISIONS
11

12. Known data
ANALYZE
DISPLAY, EXPLORE,…
Descriptive Analytics
Unknown data
PREDICT
CLASSIFY, …
Predictive Analytics
Someone else’s decisions
PLAY
COMPETE,…
Game Theory
Your decisions
OPTIMIZE
DECIDE, PLAN, SCHEDULE, …
Prescriptive Analytics
12

13. Intention vs Technology
13
Predictive Analytics
Any type of analytics
that predicts some
unknowns, events or
future outcomes
Prescriptive Analytics
Any type of analytics that
advises actions or
decisions whose execution
is likely to bring a given
aimed result.

14. Techniques to prescribe good decisions
I apply rules
Mum told me to put one pair
of socks and a T Shirt per day.
She told me then to put one
book per week, and then toy if
it fits.
I learnt from my mother.
I don’t know about the limits, or
constraints or rules, but I saw my
mother pack luggage many times
and I learnt how to do it alone.
I optimize my luggage.
I know what are Airlines size and
weight limits. I know what are the
mandatory items per day or
week. I know my preferences for
optional items.
BUSINESS RULES
MANAGEMENT
MACHINE LEARNING DECISION OPTIMIZATION
My family is going on holiday to the beach, and airline has luggage limits…
How to decide about luggage?
known policy sub optimal no need to
formulate
Need data,
bias
optimal need to formulate
14

15. Formulation du modèle
15
OPL Python

16. Machine learning Decision optimization
Historical Data Weather prediction
Habits Commitments
Recommended
maintenance UnitsSchedule
Capacity Grid
Operations Maintenance
Consumers
Forecast Plan
Two Types of Science for Two types of Data
16

17. Train ML
model
Solve DO
model
Machine Learning vs Decision Optimization
Machine Learning 101
• Basic (supervised): you know the answer
and you train the machine how to find it
• More advanced – unsupervised,
reinforcement, & deep learning
Decision Optimization 101
• You don’t know the answer, and you
provide the machine the rules on what
is a good and a bad solution
• More advanced – robust/stochastic/…
Sample data
Prediction,
pattern,
classification
Observations
Predicted
data
(optional)
Business
goals
Business
constraints
Unknowns
(decision variables)
Decisions,
plans,
schedules
Solve
trained
model
Known
data
17

18. • AI & data science as a team effort - Lets AI experts, data
scientists, analysts, stakeholders collaborate to collect,
share, explore, analyze data to derive insights, train
models, and share/deploy resulting assets
• Projects provide a secure environment in which teams or
individuals work with and analyze data using
• Connect cloud and on prem data sources
• Refine – clean and shape data for analysis/ ML
• Dashboards – visual analytics and sharing of insights
• ML/DL – train with Spark ML, ScikitLearn, SPSS, …
and deploy to Watson Machine Learning service for prod
• Flows – create flows running on Spark or SPSS
• Notebooks – Jupyter + Spark int., comments, versions, share
as link, Git integration, Remote Spark integration, …
now with flexible Environment options
• Decision Optimization – solve complex problems
• RStudio integrated with Spark
• Integration with other IBM AI and Data capabilities
• Integrates with Watson Knowledge Catalog, Watson
Machine Learning, and Watson Open Scale
• Integrates with Watson AI Services & IBM Analytics Engine
• Integrates with IBM/third-party cloud & on-prem data services
Try it at https://www.ibm.com/cloud/watson-studio
Now available in Dallas, Frankfurt, Tokyo, London
Watson Studio
18Think 2019 / 6974 / February 14, 2019 / © 2019 IBM Corporation

19. Create & train ML Models
• Create Model from Project Assets tab
• Automatic – prepare data and create model
• Manual – user prepares data and selects model
• Modeller Flow – create and train model in a flow
• Notebook – create and train model through code
• Deploy model to WML
• the model becomes available through REST API
• get URL & code snippets from Implementation tab
• try model with different values in test tab
• Invoke model from any client
• Notebooks in Watson Studio or elsewhere
• Apps on IBM Cloud
• Other apps or other clients
• Alternatively: Export model and run where needed
• Own container images
• Mobile apps
19Think 2019 / 6974 / February 14, 2019 / © 2019 IBM Corporation

20. DO Model Builder
(in WS Local, soon on Cloud)
• Model Builder
Simple 4-step guided builder
• OPL (Optimization Programming Language)
Import and run OPL models.
• Interactive Dashboard
Understand and share the optimization results
and insights through a visual, interactive
interface.
• Scenario Comparison
Explore trade-offs between different action
plans in a single view.
• Debug, tune, validate with Scenarios and
Dashboard
• Scenario Management from Python
Create, duplicate, update, solve scenarios
directly from Python.
20Think 2019 / 6974 / February 14, 2019 / © 2019 IBM Corporation

21. 2121
USE CASE
Efficiently leverage
renewable power to
satisfy demand
Predict renewable
generation
Optimize renewables &
tooling in different
production platforms
IBM’s Data Science Elite help
Red Electrica de España plan
renewable energy production
more efficiently
CASE STUDY “… the work done has been a lot
and of great quality and we know
that in so little time much more
has been done than initially
imagined.
Our team is really satisfied with
everything achieved and for
having had the opportunity to
work with a so competent IBM
team.”
Mustafa Pezic
Red Eléctrica de España
EXPECTED BENEFIT
Unify predictive and
optimization tooling via DO
for WS, enabling future
machine learning use
cases.
Better predict wind
generation
probabilistically.
UNIQUE CHALLENGE
Wind generation difficult
to predict on the Spanish
Islands
Planning models with
uncertainty are complex
Predictive & optimization
tooling in different
platforms
21

22. Optimiser l’incertain (1/3)
22
Max: 400MW
High Fixed cost.
130MW70MW
320MW280MW
Plan: 200MW
Plan: 100MW
100MW
300MW
Deterministic Optimization

23. Optimiser l’incertain (2/3)
23
Max: 400MW
High Fixed cost.
260MW140MW
420MW380MW
200MW
140MW
280MW
140MW
240MW
260MW
120MW
260MW
160MW
260MW140MW
420MW
380MW
400MW
Stochastic Optimization

24. Optimiser l’incertain (3/3)
24
Max: 400MW
High Fixed cost.
520MW280MW
820MW780MW
400MW
280MW
540MW
280MW
500MW
520MW
260MW
520MW
300MW
520MW280MW
820MW
780MW
800MW
Robust Optimization

25. Summary Results: Models Comparison Heatmap
− 3 formulations compared over 7 days, based on the electrical system of a typical utility company
− Ex-post analysis captures the performance of optimal plan (recommended before a day-of-operations)
versus the actual situation (registered at the end of the same day)
https://ree-dashboard.eu-de.mybluemix.net/#

26. Summary Results: What we learned
− 3 formulations compared over 7 days, based on the electrical system of a typical utility company
− Ex-post analysis captures the performance of optimal plan (recommended before a day-of-operations)
versus the actual situation (registered at the end of the same day)
The introduced robust formulation improves the quality of
recommendations by 45.7%
− Estimated daily cost savings of 3.7%
− Increased utilization rate of renewable energy by 5.1%
− Average computational time < 1 sec
(12 times faster than the stochastic program)

27. Conclusions
27
Les Data Sciences: différentes techniques pour différents
traitements de différents types de données…
Watson Studio: une plateforme incluant les différentes
techniques et permettant de les combiner.
https://www.ibm.com/cloud/watson-studio
Alain.chabrier@ibm.com
@AlainChabrier