new optimization algorithm for topology optimization
slides-defense-jie
1. Control Overhead Optimization in Wireless
Resource Allocation Problems
Jie Ren
Advisor: Dr. John MacLaren Walsh
Department of Electrical and Computer Engineering
Drexel University, Philadelphia, PA 19104
June 23, 2016
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 1 / 95
2. Major Contribution 1
user1 X
Controller
user2 Y
x=1
x=3
x=2
· · ·f2
(x, y) =
8
<
:
user1 : x = y = 1
user2 : x = y = 2
user1 : x = y = 3
f1
(x, y) =
8
<
:
user1 : x = y = 1
user1 : x = y = 2
user1 : x = y = 3
f(x, y) =
8
<
:
user1 : x > y
user2 : x < y
either : x = y
Who has higher value?
X = {1, 2, 3}
Y = {1, 2, 3}
Distinguish the ties
23
functions
Characteristic Graph
For independent channel qualities
greedy coloring achieves the limit
One-shot Minimum Control Overhead
Achievability: Candidate Functions
Converse: Graph Entropy
3. Major Contribution 2
2 3 4 5 6 7 8 9 10 11 12
5
10
15
20
25
30
35
40
45
50
number of users
overheads
Non−interaction
Interaction
Sending Threshold
Sending Number of Uers
One−way Limit
3 dB
2 dB
Ut( t = 3dB)
X1
XN
V 1
t = 1
V N
t = 0
·
·
·
Multi-threshold Interactive Scheme
Solve by Dynamic Programming
Substantial Rate Savings
than the One-shot Limit
4. Major Contribution 3
An Alternating Optimization Framework
Joint Optimization of Control and Communication
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
for all(c1, c2, b) 2 S
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Optimal
Control c⇤
(c1, c2, b)
c⇤
(i) = c⇤
(j)
if q(i) = q(j)
Find Nash Equilibrium!
Minimum number
of information
exchanged?
Buffer
Information
Channel
Information
Omniscient
Controller
0 1 2 3 4 5
0
0.5
1
1.5
Communication cost[bits]
Throughput
greedy scheme
round−robin scheme
Omniscient Control of a Distributed MDP
Minimum Communication to Simulate the MDP
5. Motivations and Backgrounds
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 5 / 95
6. Motivations and Backgrounds
Motivation
• Which user to assign the
physical resource block (PRB)
to
• Which modulation and coding
scheme to employ
X1
X2
X3
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 6 / 95
7. Motivations and Backgrounds
Control Overhead in LTE Downlink Resource
Allocation1
• Overhead: Reference Signals, CQIs,
Control Decisions
• Occupy the OFDMA
resource blocks
1
Gwanmo Ku and John MacLaren Walsh, “Resource Allocation and Link Adaptation
in LTE & LTE Advanced: A Tutorial,” IEEE Commun. Surveys Tuts., 2015.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 7 / 95
8. Motivations and Backgrounds
Lossless Function Computation with Side
Information
• Discrete Sources X, Y
• Function Computation Result f (X, Y )
• Vanish Error
Enc Dec
X U f(X, Y )
Y
lim
N→∞
1
N
Pr.(ˆf (UN
, Y N
) = f (XN
, Y N
)) = 0
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 8 / 95
9. Motivations and Backgrounds
Fundamental Limit: the Conditional Graph Entropy1
One-way function computation
Enc Dec
X U f(X, Y )
Y
R ≥ HG (X|Y )
= min
p(w|x),w∈Γ(G)
W −X−Y
I(W ; X|Y )
1
Alon Orlitsky and James R. Roche, “Coding for Computing,” IEEE Trans. I.T., vol.
47, no. 3, pp. 903 -917, Mar. 2001.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 9 / 95
10. Motivations and Backgrounds
Characteristic Graph
• G(V , E) and f
• The vertex set is the support set of X
• The edge (x, x ) exists if there is a y ∈ Y such that
p(x, y) > 0 and p(x , y) > 0
f (x, y) = f (x , y)
• Independent Set: A set of vertices that no two vertices has an edge
• Maximum Independent Set: An independent set that will be
dependent if adding one more vertex
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 10 / 95
11. Motivations and Backgrounds
The CEO Setup
Enc1
Dec
X U
f(X, Y )
Y
Enc2
V
• Separate Encoders, Joint Decoder
• Interested in the Minimum Sumrate
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 11 / 95
12. Motivations and Backgrounds
Distributed Lossless Function Computation1
Enc1
Dec
X U
f(X, Y )
Y
Enc2
V
• Outer bound: sum of conditional graph entropies
RX + RY ≥ HGX
(X|Y ) + HGY
(Y |X)
• Achievable scheme: each individually color the characteristic graph,
Send the colors by a Slepian-Wolf code
1
Vishal Doshi, Devavrat Shah, Muriel Medard and Sidharth Jaggi, “Functional
Compression through Graph Coloring,” IEEE Trans. I.T., vol. 56, no. 8, pp. 3901 -
3917, Aug. 2010.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 12 / 95
13. Motivations and Backgrounds
Interactive Function Computation1
Terminal Terminal
X Y
U1
Ut
·
·
·
lim
n!1
1
n
E[dn
(fn
(Xn
, Y n
), ˆfn
(Ut
, Y n
))] = 0
• Two terminals
• Multi-rounds
• Lossless computation
Rt ={R|∃Ut
, s.t.∀i = 1, · · · , t
Ri ≥ I(X; Ui |Y , Ui−1
), Ui − (X, Ui−1
) − Y , i odd
Ri ≥ I(Y ; Ui |Y , Ui−1
), Ui − (Y , Ui−1
) − X, i even
H(f (X, Y )|Y , Ut
) = 0}
1
Nan Ma and Prakash Ishwar, “Some Results on Distributed Source Coding for
Interactive Function Computation,” IEEE Trans. I.T., vol. 57, no. 9, pp. 6180 - 6195,
Sep. 2011.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 13 / 95
14. Motivations and Backgrounds
Interaction in Colocated Networks1
Source
Source
Source
Source
X1
X2 X3
X4
Sink
f(X1, X2, X3, X4)
U1, U7U2, U5
U3U4, U6
• Independent sources
• Lossless function computation
• Error free channels
• Communication order matters
Ui Ui 1
, Xj Xj 1, Xm
j+1
H(f(X1, . . . , Xm)|U1, . . . , Ut) = 0Rk
sum,t = min
pUt|Xm 2Pk
t (pXm )
I(Xm
; Ut
)
where Pk
t (pXm ) is the set of all pUt|Xm such thatRk
t = [pUt|Xm 2Pk
t (pXm )
8
>><
>>:
R
8i 2 [t]
j = (k + i 1 mod m)
Ri I(Xj; Ui|Ui 1
)
9
>>=
>>;
1
Nan Ma and Prakash Ishwar, “Interactive source coding for function computation in
collocated networks,” IEEE Trans. I.T., vol. 58, no. 7, pp. 4289 - 4305, July 2012.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 14 / 95
15. Motivations and Backgrounds
Interaction in Colocated Networks
Compute the minimum sum-rate Rsum,∞
• The rate reduction functional
ρk
t = H(Xm
) − Rk
sum,t = max
pUt |Xm ∈Pk
t (pXm )
H(Xm
|Ut
)
• The hypograph of a function f : Rn → R
hypof = {(x, µ) : x ∈ Rn
, µ ∈ R, µ ≤ f (x)} ⊂ Rn+1
• Iteratively update ρk
t for fixed pXmXk
ch hypo∆(Xk )ρk+
t−1(
m
i=1
pXi
) = hypo∆(Xk )ρk
t (
m
i=1
pXi
)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 15 / 95
16. Control Overhead in wireless Resource allocation
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 16 / 95
17. Control Overhead in wireless Resource allocation
References
1 Jie Ren and John MacLaren Walsh, “Interactive Communication for
Resource Allocation,” Conf. Information on Sciences and Systems
(CISS), Mar. 2014.
2 Jie Ren, Bradford Boyle, Steven Weber and John MacLaren Walsh,
“Overhead Performance Tradeoffs - A Resource Allocation
Perspective,” IEEE Trans. Inf. Theory., vol. 62, no. 6, pp.
3243–3269, Jun. 2016.
3 Bradford Boyle, Jie Ren, John MacLaren Walsh and Steven Weber,
“Interactive Scalar Quantization for Distributed Resource Allocation,”
IEEE Trans. Sig. Proc., vol. 64, no. 5, pp. 1243–1256, Mar. 2016.
18. Control Overhead in wireless Resource allocation Problem Setup
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 18 / 95
19. Control Overhead in wireless Resource allocation Problem Setup
Motivation – Wireless Resource Allocation
MS 1
MS 2
BS
Encoder
Encoder
Decoder
Subband index
User
subband
gain
1 2 3
Subband index
User
subband
gain
1 2 3
X
(1)
1 , . . . , X
(M)
1
X
(1)
2 , . . . , X
(M)
2
S1
Subband index
User
subband
gain
1 2 3
Z(1)
, . . . , Z(M)
Z(j)
= arg max
n
X
(j)
1 , X
(j)
2
o
Z = g(X1, X2)
ˆZ = f(S1, S2)
S2
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 19 / 95
20. Control Overhead in wireless Resource allocation Problem Setup
Problem Model
Z 2 g(XS
1 , . . . , XS
N )
lim
S!1
P(S)
e = 0
R1
R2
RN
Enc1
Dec
XS
2
XS
N
XS
1
Enc2
EncN
• Channel capacity: modeled as
discrete i.i.d. sources
• Assume rateless data
transmission
• The CEO needs to compute
{i|Xi = max{Xi : i ∈ [N]}}
• Distortion Measure
dA((X1,s, . . . , XN,s), ˆZA(s)) =
0 if ˆZA ∈ ZA
ZM(s) − XˆZA(s),s otherwise
• Distributed Lossless Computation
E d((X1, . . . , XN), ˆZ) = 0
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 20 / 95
21. Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 21 / 95
22. Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
Lossless One-way Results
Theorem
The minimum sum-rate for determining the argmax by losslessly
calculating a candidate arg-max function is
RA = min
Z∈ZA,N
N
n=1
min
cn∈C(Gn(Z))
H(cn(Xn))
• Achievability: build f ∗
N recursively, graph coloring
• Converse: graph entropy
• i.i.d. sources: do not need the OR-product graph
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 22 / 95
23. Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
The ArgMax Function Ambiguities and
Characteristic Graphs
user1 X
Controller
user2 Y
x=1
x=3
x=2
VX = X
· · ·f2
(x, y) =
8
<
:
user1 : x = y = 1
user2 : x = y = 2
user1 : x = y = 3
f1
(x, y) =
8
<
:
user1 : x = y = 1
user1 : x = y = 2
user1 : x = y = 3
f(x, y) =
8
<
:
user1 : x > y
user2 : x < y
either : x = y
Who has higher value?
Characteristic Graph of X w.r.t. Y and f
X = {1, 2, 3}
Y = {1, 2, 3}
Distinguish the ties
23
functions
xi ⇠ xj if 9 y 2 Y
s.t. f(xi, y) 6= f(xj, y)
GX(VX , EX )
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 23 / 95
24. Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
Fundamental Limit: the Conditional Graph Entropy
HG(X|Y ) = min
w2 (G), p(w|x),W X Y
I(W; X|Y )
The Conditional Graph Entropy
Distributed function computation2
f(X, Y )
RY
DEC
ENC
Source X
Source Y
ENC
RX + RY HGX
(X|Y ) + HGY
(Y |X)
R HG(X|Y ) f(X, Y )
R
DEC
ENC
Source X
Y
RX
Function computation with side information1
2
M. Sefidgaran and A. Tchamkerten “Distributed Function Computation over a Rooted
1
A. Orlitsky and J. R. Roche, “Coding for Computing”, IEEE Trans. I.T., Mar. 2001.
Directed Tree”, IEEE Trans. I.T., 2016
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 24 / 95
25. Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
Build the OR-Product Graph
x=1
x=3
x=2
(3,3)
(1,1)
(1,2)
(1,3)
(2,1)
(2,2)
(2,3)
(3,1)
(3,2)
Connect the graph entropy with graph coloring
lim
N!1
1
N
⇢
lim
✏!0+
min
c2C(G✏)
H(c(XN
)|Y N
) = HG(X|Y )
G2
x(V 2
x , E2
x)
V 2
x = X2
(u1, u2) ⇠ (v1, v2) if u1 ⇠ v1 or u2 ⇠ v2
The OR-product Graph
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 25 / 95
26. Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
Graph Colorings for different ArgMax Functions
x=1
x=3
x=2
y=1
y=3
y=2
f1
(x, y) =
8
<
:
user1 : x = y = 1
user1 : x = y = 2
user1 : x = y = 3
Assume uniformly distributed
sum-rate
Rx + Ry = 1.8337 bitsRx + Ry = 3.17 bits
x=1
x=3
x=2
Characteristic
Graph X
Characteristic
Graph Y
y=1
y=3
y=2
f2
(x, y) =
8
<
:
user1 : x = y = 1
user2 : x = y = 2
user1 : x = y = 3
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 26 / 95
27. Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
Theorem: Graph Coloring Achieves the Limit
x=1
x=3
x=2 y=1
y=3
y=2
f(x, y) =
8
<
:
user1 : x = y = 1
user2 : x = y = 2
user1 : x = y = 3
(GY ) = {{1}, {2, 3}}(GX ) = {{1, 2}, {3}}
Characteristic graph GX Characteristic graph GY
Rx + Ry = HGX
(X) + HGY
(Y )
HGX
(X|Y ) = HGX
(X) = 0.9183 HGY
(Y |X) = HGY
(Y ) = 0.9183
Assume X, Y are uniformly distributed
Greedy coloring achieves the rate limit for independent sources1
1
J. Ren, B. Boyle, S. Weber and J. M. Walsh, “Overhead Performance Tradeo↵s - A Resource
Allocation Perspective”, IEEE Trans. I.T., Jun. 2016.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 27 / 95
28. Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
Main Results
Encode the channel information for computing the distributed
argmax function
• Lossless one-way regime
• Achievability and converse
• At most 2-bits saving, does not scale with N
lim
N→∞
∆A
N
= 0
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 28 / 95
29. Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 29 / 95
30. Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Main Contribution
Achievable Interactive Communication Scheme for Resource
Allocation
• Determine the arg-max (use rateless codes for data transmission)
• Solve by dynamic programming
• Show huge savings
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 30 / 95
31. Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Problem Model
3 dB
2 dB
Ut( t = 3dB)
X1
XN
V 1
t = 1
V N
t = 0
·
·
·
Notations
• Xi ∈ Xt = {at, . . . , bt}
• Ut Broadcasting message at
round t
• V i
t Replied message from MS i
at round t
Achievable Interaction Scheme
1: BS broadcasts a threshold λt at
round t
2: MS i replies a 1 if Xi ≥ λt and 0
otherwise
3: Stops when BS knows arg-max
reliably
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 31 / 95
32. Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Analysis
Non-increasing Support set of X
If some users reply 1
at+1 = λt
bt+1 = bt
Ft+1(x) =
Ft(x) − Ft(λt)
Ft(bt) − Ft(λt)
If no user replies 1
at+1 = at
bt+1 = λt
Ft+1(x) =
Ft(x) − Ft(at)
Ft(λt) − Ft(at)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 32 / 95
33. Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Analysis
Interested in the minimum expected sum-rate min
λ1,...,λt
Rt
Rt(λ) = H(λ|λ1, · · · , λt−1) + Nt + (Ft(λ))Nt
R∗
(Nt, at, λ)
+
Nt
i=1
(1 − Ft(λ))i
Ft(λ)Nt −i Nt!
i!(Nt − i)!
R∗
(i, λ, bt)
Policy Iteration
λ∗
t = arg min
λ
Rt(λ)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 33 / 95
34. Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Analysis
• Efficiently Encode the Threshold
H(λt|λ1, . . . , λt−1)
• Why H(Nt|Nt−1) works?
• Xt and Nt determines λ∗
t
• Xt−1,Nt−1 and Nt determines Xt
• Two other strategies
• Non-conditioning Encode
the Threshold: H(λt)
• Encode the Number of
Users: H(Nt|Nt−1)
Xt =
{λ∗
t−1, bt−1} if Nt < Nt−1
{at−1, λ∗
t−1} if Nt = Nt−1 and λ∗
t−1 > xi
{λ∗
t−1, bt−1} if Nt = Nt−1 and λ∗
t−1 ≤ xi
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 34 / 95
35. Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Results
• X = {1, . . . , 16}
2 3 4 5 6 7 8 9 10 11 12
5
10
15
20
25
30
35
40
45
50
number of users
overheads
Non−interaction
Interaction
Sending Threshold
Sending Number of Uers
One−way Limit
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 35 / 95
36. Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Some Extensions
• Interaction with Distortion
E[max{X1, . . . , XNt } − Xi ] ≤ D
• Bits Cost Vs. Time Cost
C = f (R, T)
• Interactive Quantization1
• Interaction in Collocated Networks2
1
B. Boyle, J. Ren, J. M. Walsh and S. Weber, “Interactive Scalar Quantization for
Distributed Resource Allocation,” IEEE Trans. Sig. Proc., Mar. 2016.
2
S. Torabi, J. Ren and J. M. Walsh, “Practical Interactive Scheme for Extremum
Computation in Distributed Networks,” ISIT, July 2016.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 36 / 95
37. Rate Efficient Control of Distributed Discrete Systems
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 37 / 95
38. Rate Efficient Control of Distributed Discrete
Systems
References
1 Jie Ren, Solmaz Torabi and John MacLaren Walsh, “A Framework for
Rate Efficient Control of Distributed Discrete Systems”, plan to
submit to IEEE Transactions on Automatic Control.
39. Rate Efficient Control of Distributed Discrete Systems An Extension of the Rate Efficient Control Problem
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 39 / 95
40. Rate Efficient Control of Distributed Discrete Systems An Extension of the Rate Efficient Control Problem
Rate Efficient Control for Extremization Functions
• One way communication:
• Distributed function computation
• Graph colorings, Candidate functions
• Interaction communication:
• Multi-threshold Interaction Scheme
• Dynamic programming
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 40 / 95
41. Rate Efficient Control of Distributed Discrete Systems An Extension of the Rate Efficient Control Problem
Consider Other Control Decision Functions
• Distinguish high channel quality users and low channel quality users
• Maximize expected capacity
• Distinguish heavy traffic users and light traffic users:
• Maximize expected throughput
• Include control overhead cost in making control decisions
• cost-throughput tradeoff
• Fairness, package delay, etc.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 41 / 95
42. Rate Efficient Control of Distributed Discrete Systems An Extension of the Rate Efficient Control Problem
A Size Limited Buffer at the Basestation
User 1 User 2
B1(t) B2(t)
UE 1
UE 2
channel quality: C1(t)
channel quality: C2(t)
• Base station has a shared buffer containing all coming data packages
B(t) = (B1(t), . . . , BN(t))
N
n=1
Bn(t) ≤ Bm, for all t ∈ {1, 2, . . .}
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 42 / 95
43. Rate Efficient Control of Distributed Discrete Systems An Extension of the Rate Efficient Control Problem
Problem Model
UE1 UE2 UE1 UE2
UE2 UE2 UE1 UE1
UE1 UE1 UE2 UE1
UE2 UE1 UE2 UE2
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
time
frequency
···
···
c1,1
c1,2 c1,3
c2,1
c2,2
c2,3
capacity
capacity
PRB index
PRB index
1
1 2
2
3
3
Encoder2
M1
Resource
optimal coding schemes?
optimal control decisions?
Allocation
Problem 2
physical resource blocks
Encoder3
Encoder1
M3
M2
Problem 1
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 43 / 95
44. Rate Efficient Control of Distributed Discrete Systems An Extension of the Rate Efficient Control Problem
The Problems We Are Interested In
Encoder2
M1
Resource
optimal coding schemes?
optimal control decisions?
Allocation
Problem 2
Encoder3
Encoder1
M3
M2
Problem 1
Problem 1
• Minimizing control overhead
• One-shot distributed function
computation
• Interactive Communication
Problem 2
• Optimal control mapping
• Which user should be
scheduled?
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 44 / 95
45. Rate Efficient Control of Distributed Discrete Systems An Extension of the Rate Efficient Control Problem
The Problems We Are Interested In
Find optimal control mapping
Encoder2
M1
Resource
optimal coding schemes?
optimal control decisions?
Allocation
Problem 2
Encoder3
Encoder1
M3
M2
Problem 1
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 45 / 95
46. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 46 / 95
47. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
The Omniscent Controller Setup
Control decision with fully observing system state
Minimum number
of information
exchanged?
Buffer
Information
Channel
Information
Omniscient
Controller
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 47 / 95
48. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
Background: The Markov Decision Process1
• Discrete states s ∈ S
• Actions a ∈ A
S1 S2
S4S3
a1
a0
a0
a0
a0
a1
a1
a1
1
Richard Bellman, “A Markovian Decision Process,” No. P-1066. RAND CORP
SANTA MONICA CA, 1957.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 48 / 95
49. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
Background: The Markov Decision Process
• Random move from states to states with probability pa(s, s )
• A transition reward w.r.t. each move under each action Ra(s, s )
S1 S2
S4S3
a1
a0
a0
a0
a0
a1
a1
a1
p = 0.6
p = 0.4
p = 0.33
p = 0.67
p = 0.9
p = 0.1
p = 1
p = 1
p = 1
p = 1
p = 1
TR = 2
R = 2
R = 2
R = 3.5
R = 3.5
R = 1.7
R = 1.7
R = 3
R = 3
TR = 4
TR = 4
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 49 / 95
50. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
Background: The Markov Decision Process
Select the actions maximizing the discounted expected reward
max
c:S→A
∞
t=0
γt
E[RAt (St, St+1)]
Define state value V (S)
V (i) =
∞
t=0
γt
E[Rc(St )(St, St+1)|S0 = i]
The optimal control A∗ = c∗(S) and optimal state value V ∗(S)
satisfies
V ∗
(i) =
j∈S
pc∗(i)(i, j)[Rc∗(i)(i, j) + γV ∗
(j)], ∀i ∈ S
c∗
(i) = arg max
a∈A j∈S
pa(i, j)[Ra(i, j) + γV ∗
(j)], ∀i ∈ S
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 50 / 95
51. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
Solve the MDP: Value Iteration and Policy Iteration1
Value Iteration
• Update state values iteratively
Vk+1(i) = max
a∈A
j
pa(i, j)[Ra(i, j) + γVk(j)], ∀i ∈ S
• Determine the limit
V ∗
(i) = lim
k→∞
Vk(i), ∀i ∈ S
• Solve for the control policy
c∗
(i) = arg max
a∈A
{
j
pa(i, j)[Ra(i, j) + γV ∗
(j)]}, ∀i ∈ S
1
Dimitri P. Bertsekas, “Dynamic Programming and Optimal Control,” Athena
Scientific, 2005.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 51 / 95
52. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
Solve the MDP: Value Iteration and Policy Iteration
Policy Iteration
• Recursively update the control mappings
ck(i) = arg max
a∈A
{
j
pa(i, j)[Ra(i, j) + γVk(j)]}, ∀i ∈ S
• and the linear equations
Vk+1(i) =
j
pck (i)(i, j)[Rck (i)(i, j) + γVk+1(j)], ∀i ∈ S
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 52 / 95
53. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
The Omniscent Controller Setup
Control decision with fully observing system state
Minimum number
of information
exchanged?
Buffer
Information
Channel
Information
Omniscient
Controller
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 53 / 95
54. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
The Omniscent Controller Setup
• Actions: select one of the N users
A = {1, . . . , N}
• Global system state S ∈ S
S = (S1, . . . , SN, SN+1) = (C1, . . . , CN, B)
= (C1, . . . , CN, B1, . . . , BN)
• Transition rewards Ra(S): the amount of data successfully
transmitted
Ra(S) = min(Sa, Ba)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 54 / 95
55. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
The Omniscent Controller Setup
• New traffic arriving during time slot t
Xt = (X1,t, . . . , XN,t)
• Future buffer state after the packet dropping process D
SN+1,t+1 = D(SN+1,t − T(At, St) + XN+1,t+1)
• where T(At, St) has nth element
Tn(At, St) =
min{SAt ,t, BAt ,t} n = At
0 otherwise
• Transition probability
p(St = i, St+1 = j) = pB(jN+1|iN+1)
N
n=1
pC (jn)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 55 / 95
56. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
The Packet Dropping Process
Bm
UE1
UE1 UE2 UE2 UE1
UE1
UE1
UE2
UE2
New Arrival Data
From the Internet
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 56 / 95
57. Rate Efficient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
The Packet Dropping Process
Update next round buffer status SN+1,t+1 = (B1,t+1, . . . , BN,t+1)
1: SN+1,t+1 = SN+1,t
2: Remaining = BUmax − N
n=1 Bn,t
3: New = N
n=1 Xn,t+1
4: n = 1
5: while Remaining > 0 & New > 0 do
6: if Xn,t+1 > 0 then
7: Bn,t+1 = Bn,t+1 + 1
8: Xn,t+1 = Xn,t+1 − 1
9: Remaining = Remaining − 1
10: New = New − 1
11: end if
12: end while
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 57 / 95
58. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 58 / 95
59. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
The Problems We Are Interested In
Rate efficient control for the omniscient controller
Encoder2
M1
Resource
optimal coding schemes?
optimal control decisions?
Allocation
Problem 2
Encoder3
Encoder1
M3
M2
Problem 1
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 59 / 95
60. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
The Communication Model
• User node n observes channel quality cn
• Basestation observes buffer status b
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
Global State Partially Observable
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 60 / 95
61. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
The Communication Model
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
Omniscient
Controllerc⇤
(c1, c2, b)
for all(c1, c2, b) 2 S
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 61 / 95
62. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
The Communication Model
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
Omniscient
Controllerc⇤
(c1, c2, b)
for all(c1, c2, b) 2 S
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 62 / 95
63. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
The Communication Model
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
for all(c1, c2, b) 2 S
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Determine the control w/ current messages
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 63 / 95
64. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
One-shot, Non-interactive Distributed Simulation of
the Omniscient MDP1
objective:
find encoder mappings
q1(s1), . . . , qN (sN ), (s1, . . . , sN ) 2 S
Communication Optimization Part
build characteristic graphs
list all maximal independent sets of Gn
MISn(1) · · · MISn(mn)
build the encoder qn(·)
Gn(Vn, En)
For all n 2 N [ {N + 1}
qn : Sn ! {1, . . . , mn}
qn(sn) = ml if sn 2 MISn(ml)
1
J. Ren, B. Boyle, S. Weber and J. M. Walsh, “Overhead performance tradeoffs - a
resource allocation perspective,” IEEE Trans. I.T., Jun. 2016.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 64 / 95
65. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
Example
Minimum sum-rate to simulate the omniscient MDP: parameters
• User number: N = 2
• Buffer size: BUmax = 3
• Discount factor: γ = 0.9
• Channel quality Sn,t
• Identical, independently, and uniformly distributed on {0, 1, 2, 3}
• New data traffic Xn,t
• Identical and independently distributed on {0, 1, 2}
• with probabilities {1/2, 1/3, 1/6}
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 65 / 95
66. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
Example
Minimum sum-rate to simulate the omniscient MDP: results
• Maximum discounted reward from the all 0 state: 9.249
• Expected system throughput: 1.076 PDU per time-slot
• Expected amount of data dropped: 0.257 PDU per time-slot
• Sum-rate cost by using Huffman codes: 3.5175 bits
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 66 / 95
67. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
Example
Minimum sum-rate to simulate the omniscient MDP: results
• User encoder mapping that achieves the minimum sum-rate
q1(S1) =
1 if S1 = 0
2 if S1 ∈ {1, 2, 3}
q2(S2) =
1 if S2 ∈ {0, 1}
2 if S2 ∈ {2, 3}
• Basestation encoder mapping that achieves the minimum sum-rate
q3(S3) =
1 if S3 ∈ {(1, 0), (2, 0), (3, 0), (1, 1), (2, 1)}
2 if S3 ∈ {(0, 1), (0, 2), (0, 3), (0, 0)}
3 if S3 = (1, 2)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 67 / 95
68. Rate Efficient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
Interactive Distributed Simulation of the Omniscient
MDP
• Lower bound1
• Nodes take turns sending a message
• Based on its observation and all previous messages
• Compute Rate required for R rounds communication by evaluating the
rate reduction functional
• Practise interaction schemes2
• Scalar quantization based interaction
• Performance close to the fundamental limit
1
N. Ma and P. Ishwar, “Interactive source coding for function computation in
collocated networks,” IEEE Trans. I.T. 2002.
2
S. Torabi, J. Ren and J. M. Walsh, “Practical Interactive Scheme for Extremum
Computation in Distributed Networks,” ISIT 2016.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 68 / 95
69. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 69 / 95
70. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Problem Model
UE1 UE2 UE1 UE2
UE2 UE2 UE1 UE1
UE1 UE1 UE2 UE1
UE2 UE1 UE2 UE2
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
time
frequency
···
···
c1,1
c1,2 c1,3
c2,1
c2,2
c2,3
capacity
capacity
PRB index
PRB index
1
1 2
2
3
3
Encoder2
M1
Resource
optimal coding schemes?
optimal control decisions?
Allocation
Problem 2
physical resource blocks
Encoder3
Encoder1
M3
M2
Problem 1
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 70 / 95
71. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Jointly Consider the Control and the
Communication Problem
• Decentralized control: global state not available at any node
• Decentralized POMDP (belief states, approximations)
• Relationships between communication and control
• Control a linear system through a noisy channel1
• Communication in distributed MDP
• Communication cost in the reward function2
• Information theoretic limits not incorporated
1
A. Sahai and S. Mitter, “The necessity and sufficiency of anytime capacity for
stabilization of a linear system over a noisy communication link part i: Scalar systems,”
IEEE Trans. I.T. 2006.
2
P. Xuan and V. Lesser and S. Zilberstein, “Communication decisions in multi-agent
markov decision processes,” ICMAS 2000.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 71 / 95
72. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
An Augmented Reward Function
Incorporate the communication cost
Ra(i, j) = Ra(i, j) − λ|q(i)|
Jointly solving
max
(c,q)∈F
t
γt
(E[Rc(St )(St, St+1) − λ|q(St)|])
F := {(c, q)|q : S → B, ∃c : q(S) → A, c(i) = c (q(i)) ∀i ∈ S}
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 72 / 95
73. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Throughput Vs. Rate-cost Tradeoff
Find the optimal (c, q) pair for each λ
Ra(i, j) = Ra(i, j) − λ|q(i)|
rate cost
throughput
= 0
th
0 < < th
throughput vs. cost
tradeo↵
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 73 / 95
74. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Solving the Joint Optimization Problem: Search All
Space
The objective function
t∈N
γt
E Rc(St )(St, St+1) − λ|q(St)|
=
t∈N
γt
i,j∈S
P[St = i, St+1 = j] Rc(i)(i, j) − λ|q(i)|
=
t∈N
γt
i,j∈S
πP(c)t
(i)
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
=
i,j∈S
π
t∈N
γt
P(c)t
(i)
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
=
i,j∈S
π(I − γP(c))−1
(i)
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 74 / 95
75. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Example: Search All Space
Jointly optimize the controller and the encoders, tiny model
• User number: N = 2
• Buffer size: BUmax = 2
• Discount factor: γ = 0.9
• Channel quality Sn,t
• Identical, independently, and uniformly distributed on {0, 1}
• New data traffic Xn,t
• Identical, independently, and uniformly distributed on {0, 1}
• Find minimum rate achieving quantizers for each control c(·)
• Possible control maps |A||S|
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 75 / 95
76. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Example: Search All Space
Jointly optimize the controller and the encoders, tiny model
• Control overhead versus expected throughput tradeoff with
λ ∈ [0, 0.36]
0 1 2 3 4
0
0.1
0.2
0.3
0.4
0.5
Communication cost[bits]
Throughput[packets]
Throughput
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 76 / 95
77. Rate Efficient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Example: Search All Space
Jointly optimize the controller and the encoders, tiny model
• Control overhead versus expected packet dropping cost with
λ ∈ [0, 0.36]
0 1 2 3 4
0.5
0.6
0.7
0.8
0.9
1
Communication cost[bits]
Packetdropcost
Packets dropped per time−slot
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 77 / 95
78. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 78 / 95
79. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Alternatively Optimize the Controller and the
Encoders
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
for all(c1, c2, b) 2 S
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Optimal
Control c⇤
(c1, c2, b)
R0
a(i, j) = Ra(i, j) |q(i)| c⇤
(i) = c⇤
(j)
if q(i) = q(j)
Augmented Reward Function
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 79 / 95
80. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Alternatively Optimize the Controller and the
Encoders
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
for all(c1, c2, b) 2 S
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Optimal
Control c⇤
(c1, c2, b)
R0
a(i, j) = Ra(i, j) |q(i)| c⇤
(i) = c⇤
(j)
if q(i) = q(j)
Augmented Reward Function
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 80 / 95
81. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Alternatively Optimize the Controller and the
Encoders
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
for all(c1, c2, b) 2 S
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Optimal
Control c⇤
(c1, c2, b)
c⇤
(i) = c⇤
(j)
if q(i) = q(j)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 81 / 95
82. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Alternatively Optimize the Controller and the
Encoders
Communication Optimization Part
Find q1(·), . . . , qN (·) s.t.
X
H(qn(Sn)) is minimized with the constraint:
8in, i0
n s.t. qn(in) = qn(i0
n) n 2 [N + 1],
c(i1, . . . , iN , iN+1) = c(i0
1, . . . , i0
N , i0
N+1)
Control Part
Maximizing:
X
t
t
(E[Rc(st)(st, st+1) |q(st)|])
Find c : S ! A
with the constraint:
c(s) = c(s0
) if q(s) = q(s0
)
control
update
communication
update
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 82 / 95
83. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
The Communication Problem
objective:
find encoder mappings
q1(s1), . . . , qN (sN ), (s1, . . . , sN ) 2 S
Communication Optimization Part
build characteristic graphs
list all maximal independent sets of Gn
MISn(1) · · · MISn(mn)
build the encoder qn(·)
Gn(Vn, En)
For all n 2 N [ {N + 1}
qn : Sn ! {1, . . . , mn}
qn(sn) = ml if sn 2 MISn(ml)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 83 / 95
84. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
The Control Problem
t∈N
γt
E Rc(St )(St, St+1) − λ|q(St)|
=
i,j∈S
π(I − γP(c))−1
(i)
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
• Direct search, possible control maps |A||q(S)|
• Alternating optimization:
• Update control map for each quantized symbol m ∈ q(S)
• Complexity |q(S)||A|
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 84 / 95
85. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
An Approximation Scheme:Iteratively Solve the
Constrained MDP
Round-robin iteration scheme
ck, = arg max
c∈F(mk, ,ck, −1,qk )
i∈q−1
k (mk, ) j∈S
π(I − γP(c))−1
(i)
·
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
ck+1 = lim
→∞
ck, , ck,0 := ck. (1)
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 85 / 95
86. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
An Approximation Scheme:Iteratively Solve the
Constrained MDP
Greedy iteration scheme
mk, = arg max
m ∈qk (S)
max
c∈F(m ,ck, −1,qk )
i∈q−1
k (m ) j∈S
π(I − γP(c))−1
(i)
· pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 86 / 95
87. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Alternatively Optimize the Controller and the
Encoders
Theorem
The two iterative methods for solving the constrained MDP yields a
monotone increasing sequence of expected rewards which converges.
Additionally, when the sequence of control maps and quantizations
converges, (c∗, q∗) = limk→∞(ck, qk), the convergent pair (c∗, q∗) are a
Nash equilibrium1, in the sense that no unilaterial deviation in any of the
axes q or c (m) for each m ∈ q(S), can yield an increase in the expected
reward.
1
Roger B. Myerson, “Game Theory,” Harvard University Press, 2013.
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 87 / 95
88. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Example: Solve by Alternative Optimization
Jointly optimize the controller and the encoders, larger model
• User number: N = 2
• Buffer size: BUmax = 4
• Discount factor: γ = 0.9
• Channel quality Sn,t
• Identical and independently distributed on {0, 1, 2, 3, 4}
• with probabilities {1/8, 2/8, 3/8, 1/8, 1/8}
• New data traffic Xn,t
• Identical and independently distributed on {0, 1, 2}
• with probabilities {1/2, 1/3, 1/6}
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 88 / 95
89. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Example: Solve by Alternative Optimization
Jointly optimize the controller and the encoders, larger model
• Control overhead versus expected throughput tradeoff with λ ∈ [0, 4]
0 1 2 3 4 5
0
0.5
1
1.5
Communication cost[bits]
Throughput
greedy scheme
round−robin scheme
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 89 / 95
90. Rate Efficient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Example: Solve by Alternative Optimization
Jointly optimize the controller and the encoders, larger model
• Control overhead versus expected packet dropping cost with λ ∈ [0, 4]
0 1 2 3 4 5
0
0.2
0.4
0.6
0.8
Communication cost[bits]
Packet−dropcost
greedy scheme
round−robin scheme
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 90 / 95
91. Conclusion & Future Work
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Efficient Control of Distributed Discrete Systems
An Extension of the Rate Efficient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 91 / 95
92. Conclusion & Future Work
• Rate efficient control for deciding the extremization functions
• One way communication:
• Distributed function computation
• Graph colorings, Candidate functions
• Interaction communication:
• Multi-threshold Interaction Scheme
• Dynamic programming
• Rate efficient control for distributed Markov decision process
• Motivation: distinguish users demand, buffer setup
• Omniscient control: the MDP
• Minimum sum-rate to learn the omniscient control decision
• A joint optimization setup: incorporate rate cost in the reward
• An alternating algorithm to solve the joint optimization problem
• Future work:
• Time varying messages and control schemes
• Incorporate the communication cost into a POMDP framework
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 92 / 95
93. Publications
1 Jie Ren and John MacLaren Walsh, “Interactive Communication for
Resource Allocation,” Conf. Information on Sciences and Systems
(CISS), Mar. 2014.
2 Jie Ren, Bradford Boyle, Steven Weber and John MacLaren Walsh,
“Overhead Performance Tradeoffs - A Resource Allocation
Perspective,” IEEE Trans. Inf. Theory., vol. 62, no. 6, pp.
3243-3269, June 2016
3 Gwanmo Ku, Jie Ren and John MacLaren Walsh, “Computing the
Rate Distortion Region for the CEO Problem With Independent
Sources,” IEEE Trans. Sig. Proc., vol. 63, no. 3, pp. 567-575, Feb.
2015.
4 Bradford Boyle, Jie Ren, John MacLaren Walsh and Steven Weber,
“Interactive Scalar Quantization for Distributed Resource Allocation,”
IEEE Trans. Sig. Proc., vol. 64, no. 5, pp. 1243-1256, Mar. 2016
94. Publications
5 Solmaz Torabi, Jie Ren and John MacLaren Walsh, “Practical
Interactive Scheme for Extremum Computation in Distributed
Networks,” 2016 IEEE International Symposium on Information
Theory (ISIT), Barcelona, Spain, July 2016
6 Solmaz Torabi, Jie Ren and John MacLaren Walsh, “Interactive
Quantization for Extremum Computation in Collocated Networks,”
Data Compression Conference, Snowbird, UT, March 2016.
7 Jie Ren, Solmaz Torabi and John MacLaren Walsh, “A Framework for
Rate Efficient Control of Distributed Discrete Systems,” plan to
submit to IEEE Transactions on Automatic Control.