1. WIND AND ENERGY
STORAGE
Optimal Control of Power Systems in Context of Wind
Energy Generation and Storage
Shuyang Li

2. Overview
Motivation
 Model
 Algorithms
 Methodology
 Data
 Results
 Further Exploration

3. MOTIVATION
 Increasing reliance on wind energy
 Wind Energy Variance
 Avoid waste
 Storing Wind
 Storage/Unit Allocation

4. MOTIVATION – Dynamic Programming
 Solving Bellman’s Equation:
𝑉 𝑆 𝑛 = max
𝑎
(𝐶 𝑆 𝑛, 𝑎 + 𝛾𝔼 𝑉 𝑆 𝑛+1 |𝑆 𝑛 )
From Optimal Learning (Powell & Ryzhov 2012)
𝑺 𝑻𝑺 𝑻−𝟏
𝑺 𝑻−𝟐
𝑺 𝑻−𝟑
. . .𝑺 𝟎
Backward Pass

5. MOTIVATION – ADP/RL
 Monte Carlo Simulation
𝑺 𝟎
Forward Pass
𝑺 𝟏
𝑺 𝟐
𝑺 𝟑 . . .
𝑺 𝑻

6. MOTIVATION - Literature
 “A Comparison of Approximate Dynamic Programming
Techniques on Benchmark Energy Storage Problems: Does
Anything Work?”
 Jiang, Pham & Powell, 2014
 “Benchmarking a Scalable Approximate Dynamic
Programming Algorithm for Stochastic Control of
Multidimensional Energy Storage Problems”
 Salas & Powell, 2014

7. PROBLEM
 Algorithmic Performance
 Q-Learning
 SARSA(λ)
 Step Sizes
Ryzhov, Frazier & Powell (2014)

8. Overview
 Motivation
Model
 Algorithms
 Methodology
 Data
 Results
 Further Exploration

9. MODEL
Demand
Grid
Battery
Wind
𝒂 𝒕
𝑾𝑫
𝒂 𝒕
𝑹𝑫
𝒂 𝒕
𝑮𝑫
𝒂 𝒕
𝑾𝑹
𝒂 𝒕
𝑮𝑹
𝒂 𝒕
𝑹𝑮
𝑺 𝒕 = (𝑹 𝒕, 𝑬 𝒕, 𝑫 𝒕, 𝑷 𝒕)
𝒂 𝒕 = (𝒂 𝒕
𝑾𝑫
, 𝒂 𝒕
𝑹𝑫
, 𝒂 𝒕
𝑮𝑫
, 𝒂 𝒕
𝑾𝑹
, 𝒂 𝒕
𝑮𝑹
, 𝒂 𝒕
𝑹𝑮
)

10. MODEL – Action Constraints
Storage capacity:
 𝑎 𝑡
𝑊𝑅
+ 𝑎 𝑡
𝐺𝑅
≤ 𝑅 𝑚𝑎𝑥 − 𝑅𝑡
 𝑎 𝑡
𝑅𝐷
+ 𝑎 𝑡
𝑅𝐺
≤ 𝑅𝑡
Demand satisfied:
 𝑎 𝑡
𝑊𝐷
+ 𝜂 𝑑 𝑎 𝑡
𝑅𝐷
+ 𝑎 𝑡
𝐺𝐷
= 𝐷𝑡
Charging / Discharging
 𝑎 𝑡
𝑊𝑅
+ 𝑎 𝑡
𝐺𝑅
≤ 𝛾 𝑐
 𝑎 𝑡
𝑅𝐷
+ 𝑎 𝑡
𝑅𝐺
≤ 𝛾 𝑑
Flow Conservation
 𝑎 𝑡
𝑊𝑅
+ 𝑎 𝑡
𝑊𝐷
≤ 𝐸𝑡
Maximal Wind Usage:
 𝑎 𝑡
𝑊𝐷
= min(𝐷𝑡, 𝐸𝑡)  𝑎 𝑡
𝑊𝑅
= min(𝑅 𝑚𝑎𝑥 − 𝑅𝑡, 𝐸𝑡 − 𝑎 𝑡
𝑊𝐷
)
From Benchmarking a Scalable Approximate Dynamic Programming Algorithm for Stochastic Control of Multidimensional Energy
Storage Problems (Salas, Powell) [Except Maximal Wind Usage]

11. MODEL – Transition Functions
Storage:
𝑅𝑡+1 = 𝑅𝑡 + 𝜙 𝑇
𝑎 𝑡 ; 𝜙 = (0, −1,0, 𝜂 𝑐
, 𝜂 𝑐
, −1)
Simplified in the stochastic model:
𝑅𝑡+1 = 𝑅𝑡 − 𝑎 𝑡
𝑅𝐷
+ 𝑎 𝑡
𝑊𝑅
+ 𝑎 𝑡
𝐺𝑅
− 𝑎 𝑡
𝑅𝐺
From Benchmarking a Scalable Approximate Dynamic Programming Algorithm for Stochastic Control of Multidimensional Energy
Storage Problems (Salas, Powell)

12. Wind:
 𝐸𝑡+1 = 𝐸𝑡 + 𝐸𝑡+1
 𝐸𝑡~𝒩(𝜇 𝐸, 𝜎 𝐸
2
)
Demand:
 𝐷𝑡 = max 0,3 − 4 sin
2𝜋𝑡
𝑇
From Benchmarking a Scalable Approximate Dynamic Programming Algorithm for Stochastic Control of Multidimensional Energy
Storage Problems (Salas, Powell)
Price:
 𝑃𝑡+1 = 𝑃𝑡 + 𝑃0,𝑡+1 + 1{𝑢 𝑡+1≤0.031} 𝑃1,𝑡+1
 𝑃0,𝑡~𝒩(𝜇 𝑃, 𝜎 𝑃
2
)
 𝑃1,𝑡~𝒩(0, 502
)
 𝑢 𝑡~𝒰(0,1)
MODEL – Transition Functions

13. MODEL – Objective Function
Reward Function:
𝐶 𝑆𝑡, 𝑎 𝑡 = 𝑃𝑡 𝐷𝑡 − 𝑃𝑡 𝑎 𝑡
𝐺𝑅
− 𝜂 𝑑
𝑎 𝑡
𝑅𝐺
+ 𝑎 𝑡
𝐺𝐷
− 𝑐ℎ
𝑅𝑡+1
Objective Function:
𝐹 𝜋∗
= max
𝜋∈Π
𝔼
𝑡∈𝑇
𝐶(𝑆𝑡, 𝐴 𝑡
𝜋
( 𝑆𝑡))
From Benchmarking a Scalable Approximate Dynamic Programming Algorithm for Stochastic Control of Multidimensional Energy
Storage Problems (Salas, Powell)

14. Overview
 Motivation
 Model
Algorithms
 Methodology
 Data
 Results
 Further Exploration

15. ALGORITHMS – Q-Learning
 Action-value function Q
𝑄𝑡+1 𝑆𝑡, 𝑎 𝑡 = 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 + 𝛼[𝑅𝑡+1 + 𝛾 𝐦𝐚𝐱
𝒂
𝑸 𝒕 𝑺 𝒕+𝟏, 𝒂 − 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 ]
Off-policy
 Selection policy: ε-greedy
 Evaluation policy: pure greedy

16. ALGORITHMS – Q-Learning
𝑺 𝒕
𝒂 𝒕
𝑸 𝒕+𝟏 𝑺 𝒕, 𝒂 𝒕 = 𝑸 𝒕 𝑺 𝒕, 𝒂 𝒕 + 𝛼[𝑹𝒕+𝟏 + 𝛾 𝒎𝒂𝒙
𝒂
𝑸 𝒕 𝑺 𝒕+𝟏, 𝒂 − 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 ]
𝑎1
𝑎2
𝒂 𝟑
𝑺 𝒕+𝟏
𝑄𝑡(𝑆𝑡+1, 𝑎1)
𝑸 𝒕(𝑺 𝒕+𝟏, 𝒂 𝟑)
𝑄𝑡(𝑆𝑡+1, 𝑎2)𝑸 𝒕(𝑺 𝒕, 𝒂 𝒕)

17. ALGORITHMS - SARSA
 SARSA [𝑺 𝑡 𝒂 𝑡 𝑹 𝑡+1 𝑺 𝑡+1 𝒂 𝑡+1]
𝑄𝑡+1 𝑆𝑡, 𝑎 𝑡 = 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 + 𝛼[𝑅𝑡+1 + 𝛾𝑸 𝒕 𝑺 𝒕+𝟏, 𝒂 𝒕+𝟏 − 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 ]
On-policy
 Evaluation policy is selection policy

18. 𝑺 𝒕
𝒂 𝒕
𝑸 𝒕+𝟏 𝑺 𝒕, 𝒂 𝒕 = 𝑸 𝒕 𝑺 𝒕, 𝒂 𝒕 + 𝛼[𝑹𝒕+𝟏 + 𝛾𝑸 𝒕 𝑺 𝒕+𝟏, 𝒂 𝒕+𝟏 − 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 ]
𝑸 𝒕(𝑺 𝒕, 𝒂 𝒕) 𝑺 𝒕+𝟏
𝒂 𝒕+𝟏
𝑸 𝒕(𝑺 𝒕+𝟏, 𝒂 𝒕+𝟏)
ALGORITHMS - SARSA

19. ALGORITHMS – SARSA(λ)
 SARSA(λ) includes a backward pass along a path (eligibility
trace)
𝑄𝑡+1 𝑆, 𝑎 = 𝑄𝑡 𝑆, 𝑎 + 𝛼𝛿𝑡 𝑍𝑡(𝑠, 𝑎)
𝛿𝑡 = 𝑅𝑡+1 + 𝛾𝑄𝑡 𝑆𝑡+1, 𝑎 𝑡+1 − 𝑄𝑡 𝑆𝑡, 𝑎 𝑡
𝑍𝑡 =
𝛾𝜆𝑍𝑡−1 + 1, 𝑆 = 𝑆𝑡, 𝑎 = 𝑎 𝑡
𝛾𝜆𝑍𝑡−1, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

20. ALGORITHMS – SARSA(λ)
Reinforcement Learning: An Introduction 2nd Ed. (Richard Sutton & Andrew Barto; 2014)

21. ALGORITHMS – Step Sizes
 Updating equation:
𝑄𝑡+1 𝑆𝑡, 𝑎 𝑡 = 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 + 𝜶[𝑅𝑡+1 + 𝛾 max
𝑎
𝑄𝑡 𝑆𝑡+1, 𝑎 − 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 ]
 Rearranged:
𝑄𝑡+1 𝑆𝑡, 𝑎 𝑡 = 1 − 𝜶 𝑄𝑡 𝑆𝑡, 𝑎 𝑡 + 𝜶[𝑅𝑡+1 + 𝛾 max
𝑎
𝑄𝑡 𝑆𝑡+1, 𝑎 ]

22. ALGORITHMS – Step Sizes
Target
Target
High 𝜶
Low 𝜶

23. ALGORITHMS – Step Sizes
 Constant : 𝛼 = 𝑘
 Harmonic : 𝛼 =
𝑎
𝑎+𝑛
 1/n : 𝛼 =
1
𝑛
 Ryzhov Formula
 Ryzhov, Frazier & Powell (2014)

24. Overview
 Motivation
 Model
 Algorithms
Methodology
 Data
 Results
 Further Exploration

25. METHODOLOGY - Simulation
 1 Deterministic Problem
 17 Stochastic Problems
 256 Sample Paths per stochastic problem
 Training iterations: transitions by Monte Carlo simulation
 ε-greedy action selection
 Evaluation: Averaged over all sample paths

26. Overview
 Motivation
 Model
 Algorithms
 Methodology
Data
 Results
 Further Exploration

27. DATA – Deterministic Parameters
 𝑅 𝑚𝑎𝑥
= 100
 𝑅 𝑚𝑖𝑛 = 0
 𝑅0 = 0
 𝜂 𝑐 = 𝜂 𝑑 = 0.90
 𝛾 𝑐
= 𝛾 𝑑
= 0.10
 𝑻 = 𝟐𝟎𝟎𝟎

28. DATA – Deterministic Problems

29. DATA – Deterministic Problems

30. DATA – Deterministic Problems

31. DATA – Stochastic Parameters
 𝑅 𝑚𝑎𝑥 = 30
 𝑅 𝑚𝑖𝑛 = 0
 𝑅0 = 25
 𝜂 𝑐
= 𝜂 𝑑
= 1.00
 𝛾 𝑐 = 𝛾 𝑑 = 5
 𝑻 = 𝟏𝟎𝟎
 𝑃 𝑚𝑎𝑥 = 70
 𝑃 𝑚𝑖𝑛 = 30
 𝐸 𝑚𝑎𝑥 = 7.00
 𝐸 𝑚𝑖𝑛
= 1.00

32. DATA – Stochastic Problems

33. DATA – Stochastic Problems

34. DATA – Stochastic Problems

35. Overview
 Motivation
 Model
 Algorithms
 Methodology
 Data
Results
 Further Exploration

36. RESULTS – METRICS
 Action Traces (Deterministic)
 Step Size over Updates
 Stored Energy over Time vs. Benchmark
 Performance over # of Training Iterations

37. DETERMINISTIC ACTIONS
Energy Charged (WR) over Time vs. Benchmark
EnergyUnits
Time Step

38. DETERMINISTIC ACTIONS
Energy Discharged (RD) over Time vs. Benchmark
EnergyUnits
Time Step

39. DETERMINISTIC ACTIONS
Energy Sold (RG) over Time vs. Benchmark
EnergyUnits
Time Step

40. DETERMINISTIC STORAGE
Energy in Storage vs. Benchmark
EnergyUnits
Time Step

41. DETERMINISTIC PERFORMANCE
Performance of Constant Step Sizes, D2
EnergyUnits
Training Iteration

42. STEP SIZE TUNING
 Declining Step Sizes
 Tunable parameters
 Harmonic:
𝒂
𝒂+𝑛
 Ryzhov: Estimator update factor 𝝂

43. HARMONIC STEP SIZES
Harmonic Step Sizes vs. Iteration
StepSize
Training Iteration

44. HARMONIC PERFORMANCE
Harmonic Step Size Performance
ProportionofOptimal
Training Iteration

45. RYZHOV STEP SIZE
 No-discount formula:
𝛼 𝑛−1 =
( 𝑐 𝑛
)2
( 𝑐 𝑛)2+( 𝜎 𝑛)2
 Tunable parameter:
𝑐 𝑛 = 1 − 𝝂 𝒏−𝟏 𝑐 𝑛−1 + 𝝂 𝒏−𝟏 𝑐 𝑛
( 𝜎 𝑛)2= 1 − 𝝂 𝒏−𝟏 ( 𝜎 𝑛−1)2+𝝂 𝒏−𝟏( 𝑐 𝑛− 𝑐 𝑛−1)2

46. Ryzhov Step Sizes (Iteration 1)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 128)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 2k)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 4k)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 8k)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 16k)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 32k)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 65k)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 131k)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 262k)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 524k)
StepSize
Update Step
RYZHOV STEP SIZES
Ryzhov Step Sizes (Iteration 1M)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 2M)
StepSize
Update Step
Ryzhov Step Sizes (Iteration 4M)
StepSize
Update Step

47. Ryzhov explanation
 It should even out at a certain constant depending on the
variance in the rewards
 For this problem, the rewards are deterministic, so the
variance measured is the variance in the different states that
we reach/are in

48. RYZHOV STEP SIZES vs.
ESTIMATOR VARIANCE
Update Step
StepSize
E.Variance

49. RYZHOV PERFORMANCE
Ryzhov Step Size Performance
ProportionofOptimal
Training Iteration

50. AGGREGATE PARAMETERS
From Benchmarking a Scalable Approximate Dynamic Programming Algorithm for Stochastic Control of Multidimensional Energy
Storage Problems (Salas, Powell)

51. AGGREGATE STORAGE
Storage vs. Benchmark
StepSize
Time Step

52. AGGREGATE PERFORMANCE
Performance of All Algorithms, S13
ProportionofOptimal
Training Iteration

53. AGGREGATE PERFORMANCE
Performance of Q-Learning, S13
ProportionofOptimal
Training Iteration

54. AGGREGATE PERFORMANCE
Performance of SARSA, S13
ProportionofOptimal
Training Iteration

55. Explanations
 Ryzhov flattens out very noticeably because the step size is
too high, it repeatedly overcompensates and bounces
around.
 Harmonic looks like it’s still getting better! Need to have it
decline more slowly / not go to 0 so quickly. Ryzhov harmonic
constant maybe?

56. STEP SIZE PERFORMANCE
Performance of Constant 0.1, All Problems
ProportionofOptimal
Training Iteration
Performance of Harmonic, All Problems
ProportionofOptimal
Training Iteration
Performance of 1/n, All Problems
ProportionofOptimal
Training Iteration
Performance of Ryzhov, All Problems
ProportionofOptimal
Training Iteration

57. AGGREGATE PERFORMANCE

58. Overview
 Motivation
 Model
 Algorithms
 Methodology
 Data
 Results
Further Exploration

59. FURTHER EXPLORATION
 Additional renewable energy sources / storage devices
 Finer levels of discretization
 Remove wind usage restriction
 Additional algorithms
 Actor-Critic
 Gradient Descent (Linear/Non-Linear VFAs)

60. Questions?