Y. Chen, A. Busic, and S. Meyn. In 54th IEEE Conference on Decision and Control, Dec. 2015. See also journal version of the paper, http://arxiv.org/abs/1504.00088

1. State Estimation and Mean Field Control
with Application to Demand Dispatch
Yue Chen, Ana Buˇsi´c, and Sean Meyn
Inria & ENS – Paris, France ECE, UF
Thanks to our sponsors:
National Science Foundation & Google

2. Virtual Energy Storage
through Distributed Control of Flexible Loads
1 Grid Control Problems
2 Demand Dispatch
3 State Estimation and Demand Dispatch
4 Conclusions
5 References

3. March 8th 2014: Impact of wind
and solar on net-load at CAISO
Ramp limitations cause price-spikes
Price spike due to high net-load ramping
need when solar production ramped out
Negative prices due to high
mid-day solar production
1200
15
0
2
4
19
17
21
23
27
25
800
1000
600
400
0
200
-200
GWGW
Toal Load
Wind and Solar
Load and Net-load
ToalWind Toal Solar
Net-load:Toal Load, lessWind and Solar
$/MWh
24 hrs
24 hrs
Peak ramp Peak
Peak ramp Peak
Grid Control Problems

4. Grid Control Problems
Challenges from Renewable Energy
Volatility from solar and wind energy has impacted markets
New “ramping products”
Greater regulation needs
March 8th 2014: Impact of wind
and solar on net-load at CAISO
Ramp limitations cause price-spikes
Price spike due to high net-load ramping
need when solar production ramped out
Negative prices due to high
mid-day solar production
1200
15
0
2
4
19
17
21
23
27
25
800
1000
600
400
0
200
-200
GWGW
Toal Load
Wind and Solar
Load and Net-load
ToalWind Toal Solar
Net-load:Toal Load, lessWind and Solar
$/MWh
24 hrs
24 hrs
Peak ramp Peak
Peak ramp Peak
1 / 18

5. Grid Control Problems
Frequency Decomposition
Example: Serving the Net-Load in Bonneville Power Administration
Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06
GW
0
1
2
3
4
Net-load curve = G1 + G2 + G3
G1
G2
G3
2 / 18

6. Grid Control Problems
Frequency Decomposition
Example: Serving the Net-Load in Bonneville Power Administration
Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06
GW
0
1
2
3
4
Net-load curve = G1 + G2 + G3
G1
G2
G3
Low frequency component: traditional generation
Remainder: “storage” (batteries, flywheels, ... smart fridges)
2 / 18

7. Local feedback loop
Local
Control
Load i
ζt Y i
tUi
t
Xi
t
Gridsignal
Localdecision
Powerdeviation
Demand Dispatch Design

8. Demand Dispatch
Demand Dispatch
Gr
Gr = G1 + G2 + G3
G1
G2
G3
?
3 / 18

9. Demand Dispatch
Demand Dispatch
Gr
Gr = G1 + G2 + G3
G1
G2
G
Traditional generation
3
3 / 18

10. Demand Dispatch
Demand Dispatch
Gr
Gr = G1 + G2 + G3
G1
G2
G
Traditional generation
Water pumping (e.g. pool pumps)
Fans in commercial HVAC3
Demand Dispatch: Power consumption from loads varies automatically
and continuously to provide service to the grid, without impacting QoS to
the consumer
3 / 18

11. Demand Dispatch
Demand Dispatch
Responsive Regulation and desired QoS
– A partial list of the needs of the grid operator, and the consumer
High quality AS? (Ancillary Service)
Reliable?
Cost effective?
Customer QoS constraints satisfied?
4 / 18

12. Demand Dispatch
Demand Dispatch
Responsive Regulation and desired QoS
– A partial list of the needs of the grid operator, and the consumer
High quality AS? (Ancillary Service)
Reliable?
Cost effective?
Customer QoS constraints satisfied?
Virtual energy storage: achieve these goals simultaneously
through distributed control
4 / 18

13. Demand Dispatch
General Principles for Design
Two components to local controlLocal feedback loop
Local
Control
Load i
ζt Y i
tUi
t
Prefilter Decision
ζt Ui
t
Xi
t
Xi
t
Each load monitors its state and a regulation signal from the grid.
Prefilter and decision rules designed to respect needs of load and grid
5 / 18

14. Demand Dispatch
General Principles for Design
Two components to local controlLocal feedback loop
Local
Control
Load i
ζt Y i
tUi
t
Prefilter Decision
ζt Ui
t
Xi
t
Xi
t
Each load monitors its state and a regulation signal from the grid.
Prefilter and decision rules designed to respect needs of load and grid
Randomized policies required for finite-state loads
5 / 18

15. Demand Dispatch
MDP model
MDP model
The state for a load is modeled as a controlled Markov chain.
Controlled transition matrix:
Pζ(x, x ) = P{Xt+1 = x | Xt = x, ζt = ζ}
Two components to local controlLocal feedback loop
Local
Control
Load i
ζt Y i
tUi
t
Prefilter Decision
ζt Ui
t
Xi
t
Xi
t
6 / 18

16. Demand Dispatch
MDP model
MDP model
The state for a load is modeled as a controlled Markov chain.
Controlled transition matrix:
Pζ(x, x ) = P{Xt+1 = x | Xt = x, ζt = ζ}
Two components to local controlLocal feedback loop
Local
Control
Load i
ζt Y i
tUi
t
Prefilter Decision
ζt Ui
t
Xi
t
Xi
t
Previous work:
• How to design Pζ? • How to analyze aggregate of similar loads?
6 / 18

17. Demand Dispatch
Aggregate Model
≈ Mean field model
State process:
µN
t (x) =
1
N
N
i=1
I{Xi
t = x}, x ∈ X
Evolution: µN
t+1 = µN
t Pζt + ∆t
7 / 18

18. Demand Dispatch
Aggregate Model
≈ Mean field model
State process:
µN
t (x) =
1
N
N
i=1
I{Xi
t = x}, x ∈ X
Evolution: µN
t+1 = µN
t Pζt + ∆t
Output (mean power): yt =
x
µN
t (x)U(x)
Nonlinear state space model Linearization useful for control design
7 / 18

19. Demand Dispatch
Aggregate Model
≈ Mean field model
Reference Output deviation (MW)
−300
−200
−100
0
100
200
300
0 20 40 60 80 100 120 140 160
t/hour
0 20 40 60 80 100 120 140 160
State process:
µN
t (x) =
1
N
N
i=1
I{Xi
t = x}, x ∈ X
Evolution: µN
t+1 = µN
t Pζt + ∆t
Output (mean power): yt =
x
µN
t (x)U(x)
Nonlinear state space model Linearization useful for control design
7 / 18

20. Demand Dispatch
Nonlinear state space model: µt+1 = µtPζt
, yt = µt, U
Linearization useful for control design
Bode Diagram
Magnitude(dB)
-10
0
10
20
30
Myopic Passive Optimal
10
-4
10
-5
10
-3
10
-2
Frequency (rad/s)
10
-1
onehournominalcycle
Three designs for a refrigerator: transfer function ζt → yt
8 / 18

21. Demand Dispatch
Grid Control Architecture: ζt = f(?)
ζ = f(∆ω)
ζ = f(y)
ζ
grid freq (Schweppe ...)
load power dev (Inria/UF 2013+)
load histogram (Montreal/Berkeley)= f(µ)
Increasing
Information
9 / 18

22. Demand Dispatch
Grid Control Architecture: ζt = f(?)
ζ = f(∆ω)
ζ = f(y)
ζ = f(y)
ζ
grid freq (Schweppe ...)
load power (Inria/UF 2013+)
load histogram (Montreal/Berkeley)
This work:
= f(µ)
Increasing
Information
ˆ
Goals: Estimate yt for control and QoS distribution
9 / 18

23. Demand Dispatch
Grid Control Architecture: ζt = f(?)
ζ = f(∆ω)
ζ = f(y)
ζ = f(y)
ζ
grid freq (Schweppe ...)
load power (Inria/UF 2013+)
load histogram (Montreal/Berkeley)
This work:
Linear state space model subject to white noise
State estimation using Kalman Filter
= f(µ)
Increasing
Information
ˆ
ζt ytLoads
µt
Goals: Estimate yt for control and QoS distribution
9 / 18

24. State Estimation and Demand Dispatch

25. State Estimation and Demand Dispatch
Linear State Space Model
State space model:
µN
t+1 = µN
t Pζt + ∆t
yN
t = µN
t , U =
1
N
N
i=1
Y i
t
Observations: Randomly sample a fixed percentage of {Y i
t }
Yt =
1
m
m
k=1
Y
st(k)
t = yN
t + Vt
Samples {st} i.i.d. and uniform.
10 / 18

26. State Estimation and Demand Dispatch
Linear State Space Model
State space model:
µN
t+1 = µN
t Pζt + ∆t
yN
t = µN
t , U =
1
N
N
i=1
Y i
t
Observations: Randomly sample a fixed percentage of {Y i
t }
Yt =
1
m
m
k=1
Y
st(k)
t = yN
t + Vt
Samples {st} i.i.d. and uniform.
Kalman filter requires second-order statistics of (∆t, Vt).
See proceedings
10 / 18

27. State Estimation and Demand Dispatch
Linear State Space Model
State-observation model:
µN
t+1 = µN
t Pζt + ∆t
Yt = yN
t + Vt
Two versions of the Kalman filter considered,
differentiated by Kalman gain Kt
11 / 18

28. State Estimation and Demand Dispatch
Linear State Space Model
State-observation model:
µN
t+1 = µN
t Pζt + ∆t
Yt = yN
t + Vt
Two versions of the Kalman filter considered,
differentiated by Kalman gain Kt
1 Assumption: µN
t+1 is conditionally Gaussian given Yt = (Yk, ζk) |t
k=0.
Under this assumption, the Kalman filter = optimal nonlinear filter.
The gain is a nonlinear function of observed variables.
11 / 18

29. State Estimation and Demand Dispatch
Linear State Space Model
State-observation model:
µN
t+1 = µN
t Pζt + ∆t
Yt = yN
t + Vt
Two versions of the Kalman filter considered,
differentiated by Kalman gain Kt
1 Assumption: µN
t+1 is conditionally Gaussian given Yt = (Yk, ζk) |t
k=0.
Under this assumption, the Kalman filter = optimal nonlinear filter.
The gain is a nonlinear function of observed variables.
2 The filter that is optimal over all linear estimators
similar to [Krylov, Lipster, and Novikov, 1984]
11 / 18

30. State Estimation and Demand Dispatch
Linear State Space Model
State-observation model:
µN
t+1 = µN
t Pζt + ∆t
Yt = yN
t + Vt
Two versions of the Kalman filter considered,
differentiated by Kalman gain Kt
1 Assumption: µN
t+1 is conditionally Gaussian given Yt = (Yk, ζk) |t
k=0.
Under this assumption, the Kalman filter = optimal nonlinear filter.
The gain is a nonlinear function of observed variables.
2 The filter that is optimal over all linear estimators
similar to [Krylov, Lipster, and Novikov, 1984]
The first is more easily calculated, and worked well in experiments.
See proceedings for details
11 / 18

31. State Estimation and Demand Dispatch
Observability fails?
Observed in models of residential pools, HVACs, fridges ...
One example for residential pools:
λ0 λζ
96 Eigenvalues of the
Observability Grammian
961
10
-10
10-5
100
105
i48 7224
|λi|
In general, all states are not recoverable from observations:
µa
0 − µb
0 = 1, yet
∞
t=0
|ya
t − yb
t |2
< 10−12
12 / 18

32. State Estimation and Demand Dispatch
Key features are observable
1. yN
t : total power consumption of loads
-3
0
3
Inputζt
Output deviation Reference
t/hour
0 20 40 60 80 100 120 140 160
−100
−50
0
50
100
MW
300,000 residential pools, with 0.1% sampling
13 / 18

33. State Estimation and Demand Dispatch
Key features are observable
1. yN
t : total power consumption of loads
2. Discounted QoS (quality of service)
Li
t =
t
k=0
βt−k
(Xi
k),
for residential pools: (x) ∝ [power consumption − desired mean]
13 / 18

34. State Estimation and Demand Dispatch
Key features are observable
1. yN
t : total power consumption of loads
2. Discounted QoS (quality of service)
t/hours
x103
−100
−50
0
50
0 100 200 300 400 500 600 700
0
2
4
6
Estimate Empirical
Lt
ΣL
t
VarianceMean
13 / 18

35. State Estimation and Demand Dispatch
Sampling rate, N, and closed-loop performance
Goal is to track reference signal rt.
Normalized error: et =
yN
t − rt
r 2
0
2
4
6
8
10
12
14
16
18
0.1%
1.0%
10%
100%
Sampling Rate
3 × 103
3 × 105
3 × 104
3 × 106
N
RMSNormalizedError(%)
14 / 18

36. State Estimation and Demand Dispatch
Un-modeled dynamics and closed-loop performance
Setting: 0.1% sampling, and
1 7th-order reduced-order observer (state is dimension 96)
2 Large uncertainty in heterogeneous population of loads
3 And, load i opts-out when QoS Li
t is out of bounds
15 / 18

37. State Estimation and Demand Dispatch
Un-modeled dynamics and closed-loop performance
Setting: 0.1% sampling, and
1 7th-order reduced-order observer (state is dimension 96)
2 Large uncertainty in heterogeneous population of loads
3 And, load i opts-out when QoS Li
t is out of bounds
0
0.5−10
−5
0
5
10
MW
100 120110 130
optout%
N = 300,000N = 30,000
100 120110 130
Closed-loop tracking
−100
−50
0
50
100
0.5
0
Output deviation Reference
t/hour t/hour
15 / 18

38. Conclusions
Conclusions
Observability provably fails in many cases,
yet important features can be estimated in-spite of large modeling error
Much more in the paper:
“Half of the states are unobservable for symmetric models”
Kalman filter for joint ensemble-individual (µt, Xi
t)
More on pools and fridges
16 / 18

39. Conclusions
Conclusions
Observability provably fails in many cases,
yet important features can be estimated in-spite of large modeling error
Much more in the paper:
“Half of the states are unobservable for symmetric models”
Kalman filter for joint ensemble-individual (µt, Xi
t )
More on pools and fridges
Outstanding question: What information is needed for successful
application of these methods?
ζ = f(∆ω)
ζ = f(y)
ζ
grid freq (Schweppe ...)
load power (Inria/UF 2013+)
load histogram (Montreal/Berkeley)= f(µ)
Increasing
Information
ˆ
ˆ
Purely local control may not be effective for primary control, but ...
stay tuned
16 / 18

40. Conclusions
Conclusions
Thank You!
17 / 18

41. References
Selected References
S. Meyn, P. Barooah, A. Buˇsi´c, Y. Chen, and J. Ehren. Ancillary service to the grid using
intelligent deferrable loads. IEEE Trans. on Auto. Control, 2015, and Conf. on Dec. &
Control, 2013.
P. Barooah, A. Buˇsi´c, and S. Meyn. Spectral decomposition of demand-side flexibility for
reliable ancillary services in a smart grid. In Proc. 48th Annual Hawaii International
Conference on System Sciences (HICSS), pages 2700–2709, Kauai, Hawaii, 2015.
N. V. Krylov, R. S. Lipster, and A. A. Novikov, Kalman filter for Markov processes, in
Statistics and Control of Stochastic Processes. New York: Optimization Software, inc.,
1984, pp. 197–213.
J. Mathieu, S. Koch, and D. Callaway, State estimation and control of electric loads to
manage real-time energy imbalance, IEEE Trans. Power Systems, vol. 28, no. 1, pp.
430–440, 2013.
P. Caines and A. Kizilkale, Recursive estimation of common partially observed disturbances
in MFG systems with application to large scale power markets, in 52nd IEEE Conference
on Decision and Control, Dec 2013, pp. 2505–2512.
R. Malham´e and C.-Y. Chong, On the statistical properties of a cyclic diffusion process
arising in the modeling of thermostat-controlled electric power system loads, SIAM J.
Appl. Math., vol. 48, no. 2, pp. 465–480, 1988.
18 / 18