This is a discussion of a paper of Alparslan et al. (ICML 2023) for the ellis un-conference

1. Differentially Private Distributed Bayesian Linear Regression with MCMC
Differentially Private Distributed Bayesian Linear
Regression with MCMC
by Barıs Alparslan, Sinan Yıldırım¸ and Ilker Birbil
ICML 2023

2. Differentially Private Distributed Bayesian Linear Regression with MCMC
Linear regression with privacy
Standard linear model
yi = x⊤
i θ + ϵi ϵi
i.i.d.
∼ N(0, σ2
y)
Further assumption on regressors
xi
i.i.d.
∼ Px
Summarising
y = Xθ + ϵ
by
S = X⊤
X z = X⊤
y ∼ N(Sθ, Sσ2
y)
[Why?]

3. Differentially Private Distributed Bayesian Linear Regression with MCMC
Noisy version
Randomise summaries
Ŝ = S + σsM Mij ∼ N(0, 1) (1)
ẑ = z + σzv v ∼ N(0, I) (2)
[Dwork et al., 2014]
Differentially private: algorithm M : D 7→ O (ε, δ)-differentially
private when
P[M(D) ∈ O] ≤ eε
P[M(D′
) ∈ O] + δ
for any pair of neighbouring data sets D, D′ ∈ D
[Dwork et al., 2006]

4. Differentially Private Distributed Bayesian Linear Regression with MCMC
Privacy
Constraint
σs = σz = ∆szσ(ε, δ)
where σ(ε, δ) given in Balle & Wang (2018, Algorithm 1), and
∆2
sz = |X|4
∞ + |X|2
∞|Y |2
∞

5. Differentially Private Distributed Bayesian Linear Regression with MCMC
Distributed Setting
[Alparslan et al., 2023]
J agents with separate and private datasets, independently
randomised

6. Differentially Private Distributed Bayesian Linear Regression with MCMC
Specificities
Node-specific observations (Ŝ1, ẑ1), · · · , (ŜJ , ẑJ statistically more
informative than their aggregates
J
X
j=1
Ŝj and
J
X
j=1
ẑj
that are insufficient statistics
“one should not, in principle, trivially aggregate them and
apply an inference method designed for J = 1 using those
aggregates”
[Just follow Bayesian principles]

7. Differentially Private Distributed Bayesian Linear Regression with MCMC
MCMC for Normally distributed X’s
Simulation from genuine posterior
π(θ, σy, Σx, z1:J , S1:J |ẑ1:J , Ŝ1:J )
when Px is N(0, Σx) and Σx ∼ IW(Λ, κ)
[explain why Px is needed]
Gibbs sampling slow, replaced with integrated version marginalising
in z1:J
Extension to non-Normal case by setting S to Ŝ (!!)

8. Differentially Private Distributed Bayesian Linear Regression with MCMC
Alternative approaches
▶ Exploit approximate posteriors of all agents in a divide
& conquer manner
▶ Perturbate individual observations with covariates simulated
from entire population
▶ Learn from other agents in iterative manner
▶ Assess confidentiality as part of the Bayesian framework /
MCMC algorithm

9. Differentially Private Distributed Bayesian Linear Regression with MCMC
Another take on Bayesian privacy