slides-defense-jie

Control Overhead Optimization in Wireless
Resource Allocation Problems
Jie Ren
Advisor: Dr. John MacLaren Walsh
Department of Electrical and Computer Engineering
Drexel University, Philadelphia, PA 19104
June 23, 2016
Jie Ren (Drexel ASPITRG) Control Overhead Optimization June 23, 2016 1 / 95

Major Contribution 1
user1 X
Controller
user2 Y
x=1
x=3
x=2
· · ·f2
(x, y) =
8
<
:
user1 : x = y = 1
user2 : x = y = 2
user1 : x = y = 3
f1
(x, y) =
8
<
:
user1 : x = y = 1
user1 : x = y = 2
user1 : x = y = 3
f(x, y) =
8
<
:
user1 : x > y
user2 : x < y
either : x = y
Who has higher value?
X = {1, 2, 3}
Y = {1, 2, 3}
Distinguish the ties
23
functions
Characteristic Graph
For independent channel qualities
greedy coloring achieves the limit
One-shot Minimum Control Overhead
Achievability: Candidate Functions
Converse: Graph Entropy

2 3 4 5 6 7 8 9 10 11 12
5
10
15
20
25
30
35
40
45
50
number of users
overheads
Non−interaction
Interaction
Sending Threshold
Sending Number of Uers
One−way Limit
3 dB
2 dB
Ut( t = 3dB)
X1
XN
V 1
t = 1
V N
t = 0
·
·
·
Multi-threshold Interactive Scheme
Solve by Dynamic Programming
Substantial Rate Savings
than the One-shot Limit

An Alternating Optimization Framework
Joint Optimization of Control and Communication
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
for all(c1, c2, b) 2 S
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Optimal
Control c⇤
(c1, c2, b)
c⇤
(i) = c⇤
(j)
if q(i) = q(j)
Find Nash Equilibrium!
Minimum number
of information
exchanged?
Buﬀer
Information
Channel
Information
Omniscient
Controller
0 1 2 3 4 5
0
0.5
1
1.5
Communication cost[bits]
Throughput
greedy scheme
round−robin scheme
Omniscient Control of a Distributed MDP
Minimum Communication to Simulate the MDP

Motivations and Backgrounds
Outline
1 Motivations and Backgrounds
2 Control Overhead in wireless Resource allocation
Problem Setup
Optimized Lossless Coding for Control Overhead
Interactive Communications for Control Overhead
3 Rate Eﬃcient Control of Distributed Discrete Systems
An Extension of the Rate Eﬃcient Control Problem
Omniscient Control of a Distributed Markov Decision Process
Minimal Communication Required to simulate the MDP
Incorporate Communication Cost in the Reward Function
An Alternating Optimization Framework for Distributed Control
4 Conclusion & Future Work

Motivation
• Which user to assign the
physical resource block (PRB)
to
• Which modulation and coding
scheme to employ
X1
X2
X3

Control Overhead in LTE Downlink Resource
Allocation1
• Overhead: Reference Signals, CQIs,
Control Decisions
• Occupy the OFDMA
resource blocks
1
Gwanmo Ku and John MacLaren Walsh, “Resource Allocation and Link Adaptation
in LTE & LTE Advanced: A Tutorial,” IEEE Commun. Surveys Tuts., 2015.

Lossless Function Computation with Side
Information
• Discrete Sources X, Y
• Function Computation Result f (X, Y )
• Vanish Error
Enc Dec
X U f(X, Y )
Y
lim
N→∞
1
N
Pr.(ˆf (UN
, Y N
) = f (XN
, Y N
)) = 0

Fundamental Limit: the Conditional Graph Entropy1
One-way function computation
Enc Dec
X U f(X, Y )
Y
R ≥ HG (X|Y )
= min
p(w|x),w∈Γ(G)
W −X−Y
I(W ; X|Y )
1
Alon Orlitsky and James R. Roche, “Coding for Computing,” IEEE Trans. I.T., vol.
47, no. 3, pp. 903 -917, Mar. 2001.

Characteristic Graph
• G(V , E) and f
• The vertex set is the support set of X
• The edge (x, x ) exists if there is a y ∈ Y such that
p(x, y) > 0 and p(x , y) > 0
f (x, y) = f (x , y)
• Independent Set: A set of vertices that no two vertices has an edge
• Maximum Independent Set: An independent set that will be
dependent if adding one more vertex

The CEO Setup
Enc1
Dec
X U
f(X, Y )
Y
Enc2
V
• Separate Encoders, Joint Decoder
• Interested in the Minimum Sumrate

Distributed Lossless Function Computation1
Enc1
Dec
X U
f(X, Y )
Y
Enc2
V
• Outer bound: sum of conditional graph entropies
RX + RY ≥ HGX
(X|Y ) + HGY
(Y |X)
• Achievable scheme: each individually color the characteristic graph,
Send the colors by a Slepian-Wolf code
1
Vishal Doshi, Devavrat Shah, Muriel Medard and Sidharth Jaggi, “Functional
Compression through Graph Coloring,” IEEE Trans. I.T., vol. 56, no. 8, pp. 3901 -
3917, Aug. 2010.

Interactive Function Computation1
Terminal Terminal
X Y
U1
Ut
·
·
·
lim
n!1
1
n
E[dn
(fn
(Xn
, Y n
), ˆfn
(Ut
, Y n
))] = 0
• Two terminals
• Multi-rounds
• Lossless computation
Rt ={R|∃Ut
, s.t.∀i = 1, · · · , t
Ri ≥ I(X; Ui |Y , Ui−1
), Ui − (X, Ui−1
) − Y , i odd
Ri ≥ I(Y ; Ui |Y , Ui−1
), Ui − (Y , Ui−1
) − X, i even
H(f (X, Y )|Y , Ut
) = 0}
1
Nan Ma and Prakash Ishwar, “Some Results on Distributed Source Coding for
Interactive Function Computation,” IEEE Trans. I.T., vol. 57, no. 9, pp. 6180 - 6195,
Sep. 2011.

Interaction in Colocated Networks1
Source
Source
Source
Source
X1
X2 X3
X4
Sink
f(X1, X2, X3, X4)
U1, U7U2, U5
U3U4, U6
• Independent sources
• Lossless function computation
• Error free channels
• Communication order matters
Ui Ui 1
, Xj Xj 1, Xm
j+1
H(f(X1, . . . , Xm)|U1, . . . , Ut) = 0Rk
sum,t = min
pUt|Xm 2Pk
t (pXm )
I(Xm
; Ut
)
where Pk
t (pXm ) is the set of all pUt|Xm such thatRk
t = [pUt|Xm 2Pk
t (pXm )
8
>><
>>:
R
8i 2 [t]
j = (k + i 1 mod m)
Ri I(Xj; Ui|Ui 1
)
9
>>=
>>;
1
Nan Ma and Prakash Ishwar, “Interactive source coding for function computation in
collocated networks,” IEEE Trans. I.T., vol. 58, no. 7, pp. 4289 - 4305, July 2012.

Interaction in Colocated Networks
Compute the minimum sum-rate Rsum,∞
• The rate reduction functional
ρk
t = H(Xm
) − Rk
sum,t = max
pUt |Xm ∈Pk
t (pXm )
H(Xm
|Ut
)
• The hypograph of a function f : Rn → R
hypof = {(x, µ) : x ∈ Rn
, µ ∈ R, µ ≤ f (x)} ⊂ Rn+1
• Iteratively update ρk
t for ﬁxed pXmXk
ch hypo∆(Xk )ρk+
t−1(
m
i=1
pXi
) = hypo∆(Xk )ρk
t (
m
i=1
pXi
)

Control Overhead in wireless Resource allocation
Outline
Problem Setup

Control Overhead in wireless Resource allocation
References
1 Jie Ren and John MacLaren Walsh, “Interactive Communication for
Resource Allocation,” Conf. Information on Sciences and Systems
(CISS), Mar. 2014.
2 Jie Ren, Bradford Boyle, Steven Weber and John MacLaren Walsh,
“Overhead Performance Tradeoﬀs - A Resource Allocation
Perspective,” IEEE Trans. Inf. Theory., vol. 62, no. 6, pp.
3243–3269, Jun. 2016.
3 Bradford Boyle, Jie Ren, John MacLaren Walsh and Steven Weber,
“Interactive Scalar Quantization for Distributed Resource Allocation,”
IEEE Trans. Sig. Proc., vol. 64, no. 5, pp. 1243–1256, Mar. 2016.

Control Overhead in wireless Resource allocation Problem Setup
Outline
Problem Setup

Motivation – Wireless Resource Allocation
MS 1
MS 2
BS
Encoder
Encoder
Decoder
Subband index
User
subband
gain
1 2 3
Subband index
User
subband
gain
1 2 3
X
(1)
1 , . . . , X
(M)
1
X
(1)
2 , . . . , X
(M)
2
S1
Subband index
User
subband
gain
1 2 3
Z(1)
, . . . , Z(M)
Z(j)
= arg max
n
X
(j)
1 , X
(j)
2
o
Z = g(X1, X2)
ˆZ = f(S1, S2)
S2

Problem Model
Z 2 g(XS
1 , . . . , XS
N )
lim
S!1
P(S)
e = 0
R1
R2
RN
Enc1
Dec
XS
2
XS
N
XS
1
Enc2
EncN
• Channel capacity: modeled as
discrete i.i.d. sources
• Assume rateless data
transmission
• The CEO needs to compute
{i|Xi = max{Xi : i ∈ [N]}}
• Distortion Measure
dA((X1,s, . . . , XN,s), ˆZA(s)) =
0 if ˆZA ∈ ZA
ZM(s) − XˆZA(s),s otherwise
• Distributed Lossless Computation
E d((X1, . . . , XN), ˆZ) = 0

Control Overhead in wireless Resource allocation Optimized Lossless Coding for Control Overhead
Outline
Problem Setup

Lossless One-way Results
Theorem
The minimum sum-rate for determining the argmax by losslessly
calculating a candidate arg-max function is
RA = min
Z∈ZA,N
N
n=1
min
cn∈C(Gn(Z))
H(cn(Xn))
• Achievability: build f ∗
N recursively, graph coloring
• Converse: graph entropy
• i.i.d. sources: do not need the OR-product graph

The ArgMax Function Ambiguities and
Characteristic Graphs
user1 X
Controller
user2 Y
x=1
x=3
x=2
VX = X
· · ·f2
(x, y) =
8
<
:
user1 : x = y = 1
user2 : x = y = 2
user1 : x = y = 3
f1
(x, y) =
8
<
:
user1 : x = y = 1
user1 : x = y = 2
user1 : x = y = 3
f(x, y) =
8
<
:
user1 : x > y
user2 : x < y
either : x = y
Who has higher value?
Characteristic Graph of X w.r.t. Y and f
X = {1, 2, 3}
Y = {1, 2, 3}
Distinguish the ties
23
functions
xi ⇠ xj if 9 y 2 Y
s.t. f(xi, y) 6= f(xj, y)
GX(VX , EX )

Fundamental Limit: the Conditional Graph Entropy
HG(X|Y ) = min
w2 (G), p(w|x),W X Y
I(W; X|Y )
The Conditional Graph Entropy
Distributed function computation2
f(X, Y )
RY
DEC
ENC
Source X
Source Y
ENC
RX + RY HGX
(X|Y ) + HGY
(Y |X)
R HG(X|Y ) f(X, Y )
R
DEC
ENC
Source X
Y
RX
Function computation with side information1
2
M. Seﬁdgaran and A. Tchamkerten “Distributed Function Computation over a Rooted
1
A. Orlitsky and J. R. Roche, “Coding for Computing”, IEEE Trans. I.T., Mar. 2001.
Directed Tree”, IEEE Trans. I.T., 2016

Build the OR-Product Graph
x=1
x=3
x=2
(3,3)
(1,1)
(1,2)
(1,3)
(2,1)
(2,2)
(2,3)
(3,1)
(3,2)
Connect the graph entropy with graph coloring
lim
N!1
1
N
⇢
lim
✏!0+
min
c2C(G✏)
H(c(XN
)|Y N
) = HG(X|Y )
G2
x(V 2
x , E2
x)
V 2
x = X2
(u1, u2) ⇠ (v1, v2) if u1 ⇠ v1 or u2 ⇠ v2
The OR-product Graph

Graph Colorings for diﬀerent ArgMax Functions
x=1
x=3
x=2
y=1
y=3
y=2
f1
(x, y) =
8
<
:
user1 : x = y = 1
user1 : x = y = 2
user1 : x = y = 3
Assume uniformly distributed
sum-rate
Rx + Ry = 1.8337 bitsRx + Ry = 3.17 bits
x=1
x=3
x=2
Characteristic
Graph X
Characteristic
Graph Y
y=1
y=3
y=2
f2
(x, y) =
8
<
:
user1 : x = y = 1
user2 : x = y = 2
user1 : x = y = 3

Theorem: Graph Coloring Achieves the Limit
x=1
x=3
x=2 y=1
y=3
y=2
f(x, y) =
8
<
:
user1 : x = y = 1
user2 : x = y = 2
user1 : x = y = 3
(GY ) = {{1}, {2, 3}}(GX ) = {{1, 2}, {3}}
Characteristic graph GX Characteristic graph GY
Rx + Ry = HGX
(X) + HGY
(Y )
HGX
(X|Y ) = HGX
(X) = 0.9183 HGY
(Y |X) = HGY
(Y ) = 0.9183
Assume X, Y are uniformly distributed
Greedy coloring achieves the rate limit for independent sources1
1
J. Ren, B. Boyle, S. Weber and J. M. Walsh, “Overhead Performance Tradeo↵s - A Resource
Allocation Perspective”, IEEE Trans. I.T., Jun. 2016.

Main Results
Encode the channel information for computing the distributed
argmax function
• Lossless one-way regime
• Achievability and converse
• At most 2-bits saving, does not scale with N
lim
N→∞
∆A
N
= 0

Control Overhead in wireless Resource allocation Interactive Communications for Control Overhead
Outline
Problem Setup

Main Contribution
Achievable Interactive Communication Scheme for Resource
Allocation
• Determine the arg-max (use rateless codes for data transmission)
• Solve by dynamic programming
• Show huge savings

Problem Model
3 dB
2 dB
Ut( t = 3dB)
X1
XN
V 1
t = 1
V N
t = 0
·
·
·
Notations
• Xi ∈ Xt = {at, . . . , bt}
• Ut Broadcasting message at
round t
• V i
t Replied message from MS i
at round t
Achievable Interaction Scheme
1: BS broadcasts a threshold λt at
round t
2: MS i replies a 1 if Xi ≥ λt and 0
otherwise
3: Stops when BS knows arg-max
reliably

Analysis
Non-increasing Support set of X
If some users reply 1
at+1 = λt
bt+1 = bt
Ft+1(x) =
Ft(x) − Ft(λt)
Ft(bt) − Ft(λt)
If no user replies 1
at+1 = at
bt+1 = λt
Ft+1(x) =
Ft(x) − Ft(at)
Ft(λt) − Ft(at)

Analysis
Interested in the minimum expected sum-rate min
λ1,...,λt
Rt
Rt(λ) = H(λ|λ1, · · · , λt−1) + Nt + (Ft(λ))Nt
R∗
(Nt, at, λ)
+
Nt
i=1
(1 − Ft(λ))i
Ft(λ)Nt −i Nt!
i!(Nt − i)!
R∗
(i, λ, bt)
Policy Iteration
λ∗
t = arg min
λ
Rt(λ)

Analysis
• Eﬃciently Encode the Threshold
H(λt|λ1, . . . , λt−1)
• Why H(Nt|Nt−1) works?
• Xt and Nt determines λ∗
t
• Xt−1,Nt−1 and Nt determines Xt
• Two other strategies
• Non-conditioning Encode
the Threshold: H(λt)
• Encode the Number of
Users: H(Nt|Nt−1)
Xt =



{λ∗
t−1, bt−1} if Nt < Nt−1
{at−1, λ∗
t−1} if Nt = Nt−1 and λ∗
t−1 > xi
{λ∗
t−1, bt−1} if Nt = Nt−1 and λ∗
t−1 ≤ xi

Results
• X = {1, . . . , 16}
2 3 4 5 6 7 8 9 10 11 12
5
10
15
20
25
30
35
40
45
50
number of users
overheads
Non−interaction
Interaction
Sending Threshold
Sending Number of Uers
One−way Limit

Some Extensions
• Interaction with Distortion
E[max{X1, . . . , XNt } − Xi ] ≤ D
• Bits Cost Vs. Time Cost
C = f (R, T)
• Interactive Quantization1
• Interaction in Collocated Networks2
1
B. Boyle, J. Ren, J. M. Walsh and S. Weber, “Interactive Scalar Quantization for
Distributed Resource Allocation,” IEEE Trans. Sig. Proc., Mar. 2016.
2
S. Torabi, J. Ren and J. M. Walsh, “Practical Interactive Scheme for Extremum
Computation in Distributed Networks,” ISIT, July 2016.

Rate Eﬃcient Control of Distributed Discrete Systems
Outline
Problem Setup

Rate Eﬃcient Control of Distributed Discrete
Systems
References
1 Jie Ren, Solmaz Torabi and John MacLaren Walsh, “A Framework for
Rate Eﬃcient Control of Distributed Discrete Systems”, plan to
submit to IEEE Transactions on Automatic Control.

Rate Eﬃcient Control of Distributed Discrete Systems An Extension of the Rate Eﬃcient Control Problem
Outline
Problem Setup

Rate Eﬃcient Control for Extremization Functions
• One way communication:
• Distributed function computation
• Graph colorings, Candidate functions
• Interaction communication:
• Multi-threshold Interaction Scheme
• Dynamic programming

Consider Other Control Decision Functions
• Distinguish high channel quality users and low channel quality users
• Maximize expected capacity
• Distinguish heavy traffic users and light traffic users:
• Maximize expected throughput
• Include control overhead cost in making control decisions
• cost-throughput tradeoff
• Fairness, package delay, etc.

A Size Limited Buﬀer at the Basestation
User 1 User 2
B1(t) B2(t)
UE 1
UE 2
channel quality: C1(t)
channel quality: C2(t)
• Base station has a shared buﬀer containing all coming data packages
B(t) = (B1(t), . . . , BN(t))
N
n=1
Bn(t) ≤ Bm, for all t ∈ {1, 2, . . .}

Problem Model
UE1 UE2 UE1 UE2
UE2 UE2 UE1 UE1
UE1 UE1 UE2 UE1
UE2 UE1 UE2 UE2
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
time
frequency
···
···
c1,1
c1,2 c1,3
c2,1
c2,2
c2,3
capacity
capacity
PRB index
PRB index
1
1 2
2
3
3
Encoder2
M1
Resource
optimal coding schemes?
optimal control decisions?
Allocation
Problem 2
physical resource blocks
Encoder3
Encoder1
M3
M2
Problem 1

The Problems We Are Interested In
Encoder2
M1
Resource
Allocation
Problem 2
Encoder3
Encoder1
M3
M2
Problem 1
Problem 1
• Minimizing control overhead
• One-shot distributed function
computation
• Interactive Communication
Problem 2
• Optimal control mapping
• Which user should be
scheduled?

Find optimal control mapping
Encoder2
M1
Resource
Allocation
Problem 2
Encoder3
Encoder1
M3
M2
Problem 1

Rate Eﬃcient Control of Distributed Discrete Systems Omniscient Control of a Distributed Markov Decision Process
Outline
Problem Setup

The Omniscent Controller Setup
Control decision with fully observing system state
Minimum number
of information
exchanged?
Buﬀer
Information
Channel
Information
Omniscient
Controller

Background: The Markov Decision Process1
• Discrete states s ∈ S
• Actions a ∈ A
S1 S2
S4S3
a1
a0
a0
a0
a0
a1
a1
a1
1
Richard Bellman, “A Markovian Decision Process,” No. P-1066. RAND CORP
SANTA MONICA CA, 1957.

Background: The Markov Decision Process
• Random move from states to states with probability pa(s, s )
• A transition reward w.r.t. each move under each action Ra(s, s )
S1 S2
S4S3
a1
a0
a0
a0
a0
a1
a1
a1
p = 0.6
p = 0.4
p = 0.33
p = 0.67
p = 0.9
p = 0.1
p = 1
p = 1
p = 1
p = 1
p = 1
TR = 2
R = 2
R = 2
R = 3.5
R = 3.5
R = 1.7
R = 1.7
R = 3
R = 3
TR = 4
TR = 4

Background: The Markov Decision Process
Select the actions maximizing the discounted expected reward
max
c:S→A
∞
t=0
γt
E[RAt (St, St+1)]
Deﬁne state value V (S)
V (i) =
∞
t=0
γt
E[Rc(St )(St, St+1)|S0 = i]
The optimal control A∗ = c∗(S) and optimal state value V ∗(S)
satisﬁes
V ∗
(i) =
j∈S
pc∗(i)(i, j)[Rc∗(i)(i, j) + γV ∗
(j)], ∀i ∈ S
c∗
(i) = arg max
a∈A j∈S
pa(i, j)[Ra(i, j) + γV ∗
(j)], ∀i ∈ S

Solve the MDP: Value Iteration and Policy Iteration1
Value Iteration
• Update state values iteratively
Vk+1(i) = max
a∈A
j
pa(i, j)[Ra(i, j) + γVk(j)], ∀i ∈ S
• Determine the limit
V ∗
(i) = lim
k→∞
Vk(i), ∀i ∈ S
• Solve for the control policy
c∗
(i) = arg max
a∈A
{
j
pa(i, j)[Ra(i, j) + γV ∗
(j)]}, ∀i ∈ S
1
Dimitri P. Bertsekas, “Dynamic Programming and Optimal Control,” Athena
Scientiﬁc, 2005.

Solve the MDP: Value Iteration and Policy Iteration
Policy Iteration
• Recursively update the control mappings
ck(i) = arg max
a∈A
{
j
pa(i, j)[Ra(i, j) + γVk(j)]}, ∀i ∈ S
• and the linear equations
Vk+1(i) =
j
pck (i)(i, j)[Rck (i)(i, j) + γVk+1(j)], ∀i ∈ S

Control decision with fully observing system state
Minimum number
of information
exchanged?
Buﬀer
Information
Channel
Information
Omniscient
Controller

• Actions: select one of the N users
A = {1, . . . , N}
• Global system state S ∈ S
S = (S1, . . . , SN, SN+1) = (C1, . . . , CN, B)
= (C1, . . . , CN, B1, . . . , BN)
• Transition rewards Ra(S): the amount of data successfully
transmitted
Ra(S) = min(Sa, Ba)

• New traﬃc arriving during time slot t
Xt = (X1,t, . . . , XN,t)
• Future buﬀer state after the packet dropping process D
SN+1,t+1 = D(SN+1,t − T(At, St) + XN+1,t+1)
• where T(At, St) has nth element
Tn(At, St) =
min{SAt ,t, BAt ,t} n = At
0 otherwise
• Transition probability
p(St = i, St+1 = j) = pB(jN+1|iN+1)
N
n=1
pC (jn)

The Packet Dropping Process
Bm
UE1
UE1 UE2 UE2 UE1
UE1
UE1
UE2
UE2
New Arrival Data
From the Internet

The Packet Dropping Process
Update next round buﬀer status SN+1,t+1 = (B1,t+1, . . . , BN,t+1)
1: SN+1,t+1 = SN+1,t
2: Remaining = BUmax − N
n=1 Bn,t
3: New = N
n=1 Xn,t+1
4: n = 1
5: while Remaining > 0 & New > 0 do
6: if Xn,t+1 > 0 then
7: Bn,t+1 = Bn,t+1 + 1
8: Xn,t+1 = Xn,t+1 − 1
9: Remaining = Remaining − 1
10: New = New − 1
11: end if
12: end while

Rate Eﬃcient Control of Distributed Discrete Systems Minimal Communication Required to simulate the MDP
Outline
Problem Setup

Rate eﬃcient control for the omniscient controller
Encoder2
M1
Resource
Allocation
Problem 2
Encoder3
Encoder1
M3
M2
Problem 1

The Communication Model
• User node n observes channel quality cn
• Basestation observes buﬀer status b
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
Global State Partially Observable

UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
Omniscient
Controllerc⇤
(c1, c2, b)

UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Determine the control w/ current messages

One-shot, Non-interactive Distributed Simulation of
the Omniscient MDP1
objective:
ﬁnd encoder mappings
q1(s1), . . . , qN (sN ), (s1, . . . , sN ) 2 S
Communication Optimization Part
build characteristic graphs
list all maximal independent sets of Gn
MISn(1) · · · MISn(mn)
build the encoder qn(·)
Gn(Vn, En)
For all n 2 N [ {N + 1}
qn : Sn ! {1, . . . , mn}
qn(sn) = ml if sn 2 MISn(ml)
1
J. Ren, B. Boyle, S. Weber and J. M. Walsh, “Overhead performance tradeoﬀs - a
resource allocation perspective,” IEEE Trans. I.T., Jun. 2016.

Example
Minimum sum-rate to simulate the omniscient MDP: parameters
• User number: N = 2
• Buﬀer size: BUmax = 3
• Discount factor: γ = 0.9
• Channel quality Sn,t
• Identical, independently, and uniformly distributed on {0, 1, 2, 3}
• New data traﬃc Xn,t
• Identical and independently distributed on {0, 1, 2}
• with probabilities {1/2, 1/3, 1/6}

Example
Minimum sum-rate to simulate the omniscient MDP: results
• Maximum discounted reward from the all 0 state: 9.249
• Expected system throughput: 1.076 PDU per time-slot
• Expected amount of data dropped: 0.257 PDU per time-slot
• Sum-rate cost by using Huﬀman codes: 3.5175 bits

Example
Minimum sum-rate to simulate the omniscient MDP: results
• User encoder mapping that achieves the minimum sum-rate
q1(S1) =
1 if S1 = 0
2 if S1 ∈ {1, 2, 3}
q2(S2) =
1 if S2 ∈ {0, 1}
2 if S2 ∈ {2, 3}
• Basestation encoder mapping that achieves the minimum sum-rate
q3(S3) =



1 if S3 ∈ {(1, 0), (2, 0), (3, 0), (1, 1), (2, 1)}
2 if S3 ∈ {(0, 1), (0, 2), (0, 3), (0, 0)}
3 if S3 = (1, 2)

Interactive Distributed Simulation of the Omniscient
MDP
• Lower bound1
• Nodes take turns sending a message
• Based on its observation and all previous messages
• Compute Rate required for R rounds communication by evaluating the
rate reduction functional
• Practise interaction schemes2
• Scalar quantization based interaction
• Performance close to the fundamental limit
1
N. Ma and P. Ishwar, “Interactive source coding for function computation in
collocated networks,” IEEE Trans. I.T. 2002.
2
S. Torabi, J. Ren and J. M. Walsh, “Practical Interactive Scheme for Extremum
Computation in Distributed Networks,” ISIT 2016.

Rate Eﬃcient Control of Distributed Discrete Systems Incorporate Communication Cost in the Reward Function
Outline
Problem Setup

Problem Model
UE1 UE2 UE1 UE2
UE2 UE2 UE1 UE1
UE1 UE1 UE2 UE1
UE2 UE1 UE2 UE2
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
time
frequency
···
···
c1,1
c1,2 c1,3
c2,1
c2,2
c2,3
capacity
capacity
PRB index
PRB index
1
1 2
2
3
3
Encoder2
M1
Resource
Allocation
Problem 2
physical resource blocks
Encoder3
Encoder1
M3
M2
Problem 1

Jointly Consider the Control and the
Communication Problem
• Decentralized control: global state not available at any node
• Decentralized POMDP (belief states, approximations)
• Relationships between communication and control
• Control a linear system through a noisy channel1
• Communication in distributed MDP
• Communication cost in the reward function2
• Information theoretic limits not incorporated
1
A. Sahai and S. Mitter, “The necessity and suﬃciency of anytime capacity for
stabilization of a linear system over a noisy communication link part i: Scalar systems,”
IEEE Trans. I.T. 2006.
2
P. Xuan and V. Lesser and S. Zilberstein, “Communication decisions in multi-agent
markov decision processes,” ICMAS 2000.

An Augmented Reward Function
Incorporate the communication cost
Ra(i, j) = Ra(i, j) − λ|q(i)|
Jointly solving
max
(c,q)∈F
t
γt
(E[Rc(St )(St, St+1) − λ|q(St)|])
F := {(c, q)|q : S → B, ∃c : q(S) → A, c(i) = c (q(i)) ∀i ∈ S}

Throughput Vs. Rate-cost Tradeoﬀ
Find the optimal (c, q) pair for each λ
Ra(i, j) = Ra(i, j) − λ|q(i)|
rate cost
throughput
= 0
th
0 < < th
throughput vs. cost
tradeo↵

Solving the Joint Optimization Problem: Search All
Space
The objective function
t∈N
γt
E Rc(St )(St, St+1) − λ|q(St)|
=
t∈N
γt
i,j∈S
P[St = i, St+1 = j] Rc(i)(i, j) − λ|q(i)|
=
t∈N
γt
i,j∈S
πP(c)t
(i)
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
=
i,j∈S
π
t∈N
γt
P(c)t
(i)
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
=
i,j∈S
π(I − γP(c))−1
(i)
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|

Example: Search All Space
Jointly optimize the controller and the encoders, tiny model
• Identical, independently, and uniformly distributed on {0, 1}
• Identical, independently, and uniformly distributed on {0, 1}
• Find minimum rate achieving quantizers for each control c(·)
• Possible control maps |A||S|

• Control overhead versus expected throughput tradeoﬀ with
λ ∈ [0, 0.36]
0 1 2 3 4
0
0.1
0.2
0.3
0.4
0.5
Throughput[packets]
Throughput

• Control overhead versus expected packet dropping cost with
λ ∈ [0, 0.36]
0 1 2 3 4
0.5
0.6
0.7
0.8
0.9
1
Packetdropcost
Packets dropped per time−slot

Rate Eﬃcient Control of Distributed Discrete Systems An Alternating Optimization Framework for Distributed Control
Outline
Problem Setup

Alternatively Optimize the Controller and the
Encoders
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Optimal
Control c⇤
(c1, c2, b)
R0
a(i, j) = Ra(i, j) |q(i)| c⇤
(i) = c⇤
(j)
if q(i) = q(j)
Augmented Reward Function

Encoders
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Optimal
Control c⇤
(c1, c2, b)
R0
a(i, j) = Ra(i, j) |q(i)| c⇤
(i) = c⇤
(j)
if q(i) = q(j)
Augmented Reward Function

Encoders
UE 1
UE 2
UE1 UE2
Bm
UE1 UE2
c1
c2
Encoder1
Encoder2
q1(c1)
q2(c2) = c⇤
(c1, c2, b)
c0
(q1(c1), q2(c2), q3(b))
q3(b)
Optimal
Control c⇤
(c1, c2, b)
c⇤
(i) = c⇤
(j)
if q(i) = q(j)

Encoders
Find q1(·), . . . , qN (·) s.t.
X
H(qn(Sn)) is minimized with the constraint:
8in, i0
n s.t. qn(in) = qn(i0
n) n 2 [N + 1],
c(i1, . . . , iN , iN+1) = c(i0
1, . . . , i0
N , i0
N+1)
Control Part
Maximizing:
X
t
t
(E[Rc(st)(st, st+1) |q(st)|])
Find c : S ! A
with the constraint:
c(s) = c(s0
) if q(s) = q(s0
)
control
update
communication
update

The Communication Problem
objective:
ﬁnd encoder mappings
q1(s1), . . . , qN (sN ), (s1, . . . , sN ) 2 S
build characteristic graphs
list all maximal independent sets of Gn
MISn(1) · · · MISn(mn)
build the encoder qn(·)
Gn(Vn, En)
For all n 2 N [ {N + 1}
qn : Sn ! {1, . . . , mn}
qn(sn) = ml if sn 2 MISn(ml)

The Control Problem
t∈N
γt
E Rc(St )(St, St+1) − λ|q(St)|
=
i,j∈S
π(I − γP(c))−1
(i)
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
• Direct search, possible control maps |A||q(S)|
• Alternating optimization:
• Update control map for each quantized symbol m ∈ q(S)
• Complexity |q(S)||A|

An Approximation Scheme:Iteratively Solve the
Constrained MDP
Round-robin iteration scheme
ck, = arg max
c∈F(mk, ,ck, −1,qk )
i∈q−1
k (mk, ) j∈S
π(I − γP(c))−1
(i)
·
pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|
ck+1 = lim
→∞
ck, , ck,0 := ck. (1)

An Approximation Scheme:Iteratively Solve the
Constrained MDP
Greedy iteration scheme
mk, = arg max
m ∈qk (S)
max
c∈F(m ,ck, −1,qk )
i∈q−1
k (m ) j∈S
π(I − γP(c))−1
(i)
· pc(i)(i, j) Rc(i)(i, j) − λ|q(i)|

Encoders
Theorem
The two iterative methods for solving the constrained MDP yields a
monotone increasing sequence of expected rewards which converges.
Additionally, when the sequence of control maps and quantizations
converges, (c∗, q∗) = limk→∞(ck, qk), the convergent pair (c∗, q∗) are a
Nash equilibrium1, in the sense that no unilaterial deviation in any of the
axes q or c (m) for each m ∈ q(S), can yield an increase in the expected
reward.
1
Roger B. Myerson, “Game Theory,” Harvard University Press, 2013.

Example: Solve by Alternative Optimization
Jointly optimize the controller and the encoders, larger model
• Identical and independently distributed on {0, 1, 2, 3, 4}
• with probabilities {1/8, 2/8, 3/8, 1/8, 1/8}
• Identical and independently distributed on {0, 1, 2}
• with probabilities {1/2, 1/3, 1/6}

• Control overhead versus expected throughput tradeoﬀ with λ ∈ [0, 4]
0 1 2 3 4 5
0
0.5
1
1.5
Throughput
greedy scheme

• Control overhead versus expected packet dropping cost with λ ∈ [0, 4]
0 1 2 3 4 5
0
0.2
0.4
0.6
0.8
Packet−dropcost
greedy scheme

Conclusion & Future Work
Outline
Problem Setup

Conclusion & Future Work
• Rate efficient control for deciding the extremization functions
• One way communication:
• Distributed function computation
• Graph colorings, Candidate functions
• Interaction communication:
• Multi-threshold Interaction Scheme
• Dynamic programming
• Rate efficient control for distributed Markov decision process
• Motivation: distinguish users demand, buffer setup
• Omniscient control: the MDP
• Minimum sum-rate to learn the omniscient control decision
• A joint optimization setup: incorporate rate cost in the reward
• An alternating algorithm to solve the joint optimization problem
• Future work:
• Time varying messages and control schemes
• Incorporate the communication cost into a POMDP framework

Publications
1 Jie Ren and John MacLaren Walsh, “Interactive Communication for
Resource Allocation,” Conf. Information on Sciences and Systems
(CISS), Mar. 2014.
2 Jie Ren, Bradford Boyle, Steven Weber and John MacLaren Walsh,
“Overhead Performance Tradeoﬀs - A Resource Allocation
Perspective,” IEEE Trans. Inf. Theory., vol. 62, no. 6, pp.
3243-3269, June 2016
3 Gwanmo Ku, Jie Ren and John MacLaren Walsh, “Computing the
Rate Distortion Region for the CEO Problem With Independent
Sources,” IEEE Trans. Sig. Proc., vol. 63, no. 3, pp. 567-575, Feb.
2015.
4 Bradford Boyle, Jie Ren, John MacLaren Walsh and Steven Weber,
“Interactive Scalar Quantization for Distributed Resource Allocation,”
IEEE Trans. Sig. Proc., vol. 64, no. 5, pp. 1243-1256, Mar. 2016

Publications
5 Solmaz Torabi, Jie Ren and John MacLaren Walsh, “Practical
Interactive Scheme for Extremum Computation in Distributed
Networks,” 2016 IEEE International Symposium on Information
Theory (ISIT), Barcelona, Spain, July 2016
6 Solmaz Torabi, Jie Ren and John MacLaren Walsh, “Interactive
Quantization for Extremum Computation in Collocated Networks,”
Data Compression Conference, Snowbird, UT, March 2016.
7 Jie Ren, Solmaz Torabi and John MacLaren Walsh, “A Framework for
Rate Eﬃcient Control of Distributed Discrete Systems,” plan to
submit to IEEE Transactions on Automatic Control.

slides-defense-jie

More Related Content

What's hot

Similar to slides-defense-jie

slides-defense-jie