R package 'bayesImageS': a case study in Bayesian computation using Rcpp and OpenMP

R packages Potts model Bayesian computation Conclusion
bayesImageS: a case study in Bayesian
computation using Rcpp and OpenMP
Matt Moores
OxWaSP mini-symposium
December 2, 2016

Outline
1 R packages
Performance of BLAS
2 Potts model
3 Bayesian computation
Chequerboard Gibbs sampler
Pseudolikelihood
Thermodynamic integration
Exchange algorithm
Approximate Bayesian Computation (ABC)

Why write an R package?
Portability
Test bed for new statistical methods
Build on existing code
Research impact
Kudos
Hadley Wickham (2015) R packages

Why C++?
Most statistical algorithms are iterative
Markov chain Monte Carlo
Scalability for large datasets
Rcpp
OpenMP
Eigen or Armadillo
Dirk Eddelbuettel (2013) Seamless R and C++ integration with Rcpp

Inline
One function at a time:
§
library ( i n l i n e )
sum_logs ← cxxfunction ( signature ( log_vec = "numeric") , plugin = "RcppArmadillo" , body=’
arma::vec log_prob = Rcpp::as<arma::vec>(log_vec);
double suml = 0.0;
double maxl = log_prob.max();
for (unsigned i=0; i < log_prob.n_elem; i++)
{
if (arma::is_finite(log_prob(i)))
suml += exp(log_prob(i) - maxl);
}
return Rcpp::wrap(log(suml) + maxl);
’)

Package Skeleton
Create a new R package:
package.skeleton("myPackage", path=".")
Speciﬁc skeletons for each C++ library:
Rcpp.package.skeleton("myRcppPackage")
RcppArmadillo.package.skeleton("MyArmadilloPackage")
RcppEigen.package.skeleton("MyEigenPackage")

Annotations
Rcpp wrappers generated automatically:
compileAttributes("myRcppPackage")
R package documentation generated automatically:
roxygenize("myRcppPackage")
§
/ / ’ Compute the effective sample size (ESS) of the particles.
/ / ’
/ / ’ The ESS is a ‘‘rule of thumb’’ for assessing the degeneracy of
/ / ’’ the importance distribution:
/ / ’ deqn {ESS = frac { ( sum_ { q=1}^Q w_q ) ^ 2 } { sum_ { q=1}^Q w_q ^2}}
/ / ’
/ / ’’ @param log_weights logarithms of the importance weights of each particle.
/ / ’’ @return the effective sample size, a scalar between 0 and Q
/ / ’’ @references
/ / ’’ Liu, JS (2001) "Monte Carlo Strategies in Scientific Computing." Springer’
/ / [ [ Rcpp : : export ] ]
double effectiveSampleSize ( NumericVector log_weights )
{
double sum_wt = sum_logs ( log_weights ) ;
double sum_sq = sum_logs ( log_weights + log_weights ) ;
double res = exp (sum_wt + sum_wt − sum_sq ) ;
i f ( std : : i s f i n i t e ( res ) ) return res ;
else return 0;
}

Common Problems
Rcpp parameters are passed by reference (not copied):
Can rely on R for garbage collection
Memory allocation is slower
Can crash R (and Rstudio (and your OS))
R is not thread safe
Cannot call any R functions (even indirectly)
within parallel code!
Drew Schmidt (@wrathematics, 2015) Parallelism, R, and OpenMP

Performance
A simple performance comparison:
§
library ( RcppEigen )
library ( RcppArmadillo )
library (RcppGSL)
library ( rbenchmark )
source ( system . f i l e ("examples" , "lmBenchmark.R" ,
package ="RcppEigen" ) )
Fitting a linear regression model
(by default, n=100000 rows × p=40 columns,
with 20 repeat measurements)

Results with default R BLAS
Table: lm benchmark for Mac OS X with reference R BLAS
test relative elapsed user.self sys.self
3 LDLt 1.000 3.611 3.448 0.164
7 QR 1.356 4.896 4.801 0.077
8 LLt 1.421 5.130 5.007 0.123
11 gpuLm.ﬁt 2.634 9.510 8.332 0.395
1 lm.ﬁt 3.760 13.576 13.550 0.025
6 SymmEig 5.475 19.770 19.634 0.126
2 PivQR 5.517 19.923 19.760 0.162
9 arma 20.416 73.723 73.459 0.255
4 GESDD 24.815 89.607 89.429 0.175
10 GSL 198.975 718.500 718.157 0.272
5 SVD 201.717 728.401 727.958 0.423
matrix dimensions:1650 × 875

Accelerate Umbrella Framework
Table: lm benchmark for Mac OS X with vecLib
test relative elapsed user.self sys.self
3 LDLt 1.000 3.682 3.517 0.165
7 QR 1.330 4.898 4.809 0.088
8 LLt 1.413 5.204 5.081 0.121
1 lm.ﬁt 1.832 6.745 6.702 0.024
9 arma 2.396 8.821 20.572 1.229
11 gpuLm.ﬁt 2.655 9.776 8.757 0.344
4 GESDD 3.313 12.199 22.120 0.884
6 SymmEig 5.405 19.902 19.775 0.127
2 PivQR 5.506 20.273 20.095 0.159
10 GSL 195.244 718.889 718.565 0.273
5 SVD 198.514 730.930 730.403 0.495
Iacus, Urbanek, Goedman & Ripley (2016) R for Mac OS X FAQ, §10.5

Nested Models
β ← 0
Hidden MRF
θ : scale
D > 1 : dimension
∂i : neighbourhood
ψ : noise param.
p (yi | zi, ψ)
p zi | zi, θ
p (θ | z)
p(ψ) p(θ)
y, z, θ, ψ
Hidden Ising/Potts
z ∈ {1 . . . k}
θ = β : inverse
temperature
p zi | zi, β
p (β | z) p(β)
Mixture of Gaussians
λ = {λ1, . . . , λk} :
k
j=1 λj = 1
ψ = µ, σ2
: noise param.
p yi | zi, µj, σ2
j
p (λ | z) p(λ)

Hidden Markov Random Field
Joint distribution of observed pixel intensities y = {yi}n
i=1
and latent labels z = {zi}n
i=1:
p(y, z|µ, σ2
, β) = p(y|µ, σ2
, z)p(z|β) (1)
Additive Gaussian noise:
yi|zi =j
iid
∼ N µj, σ2
j (2)
Potts model:
π(zi|zi, β) =
exp {β i∼ δ(zi, z )}
k
j=1 exp {β i∼ δ(j, z )}
(3)
Potts (1952) Proceedings of the Cambridge Philosophical Society 48(1)

Inverse Temperature

Doubly-intractable likelihood
p(β|z) ∝ C(β)−1
π(β) exp {β S(z)} (4)
The normalising constant has computational complexity O(nkn):
C(β) =
z∈Z
exp {β S(z)} (5)
S(z) is the sufﬁcient statistic of the Potts model:
S(z) =
i∼ ∈L
δ(zi, z ) (6)
where L is the set of all unique neighbour pairs.

Chequerboard Gibbs
A 2D or 3D regular lattice with ﬁrst-order neighbourhood ∂i:
◦ ◦ ◦ ◦ ◦
◦ ◦ • ◦ ◦
◦ • × • ◦
◦ ◦ • ◦ ◦
◦ ◦ ◦ ◦ ◦
can be partitioned into 2 blocks:
• ◦ • ◦ • ◦ • ◦ • ◦
◦ • ◦ • ◦ • ◦ • ◦ •
• ◦ • ◦ • ◦ • ◦ • ◦
◦ • ◦ • ◦ • ◦ • ◦ •
• ◦ • ◦ • ◦ • ◦ • ◦
◦ • ◦ • ◦ • ◦ • ◦ •
• ◦ • ◦ • ◦ • ◦ • ◦
◦ • ◦ • ◦ • ◦ • ◦ •
• ◦ • ◦ • ◦ • ◦ • ◦
◦ • ◦ • ◦ • ◦ • ◦ •
so that z◦ are conditionally independent, given z•
Roberts & Sahu (1997) JRSS B 59(2): 291–317
Winkler (2nd
ed., 2003) Image analysis, random ﬁelds and MCMC methods

Chequerboard Gibbs II
Algorithm 1 Chequerboard sampling for z
1: for all blocks b do
2: for all pixels i ∈ b do
3: for all labels j ∈ 1 . . . k do
4: Compute λj ← p(yi | zi = j)π(zi = j | zi∼ , β)
5: end for
6: Draw zi ∼ Multinomial(λ1, . . . , λk)
7: end for
8: end for

Gibbs sampler in C++
§
void gibbsLabels ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
arma : : umat & z , arma : : umat & alloc , const double beta ,
const arma : : mat & log_ x f i e l d )
{
const Rcpp : : NumericVector randU = Rcpp : : r u n i f ( neigh . n_rows ) ;
for ( unsigned b=0; b < blocks . size ( ) ; b++)
{
const arma : : uvec block = blocks [ b ] ;
arma : : vec log_prob ( z . n_cols ) ;
#pragma omp p a r a l l e l for private ( log_prob )
for ( unsigned i =0; i < block . size ( ) ; i ++)
{
for ( unsigned j =0; j < z . n_cols ; j ++)
{
unsigned sum_neigh = 0;
for ( unsigned k=0; k < neigh . n_cols ; k++)
{
sum_neigh += z ( neigh ( block [ i ] , k ) , j ) ;
}
log_prob [ j ] = log_ x f i e l d ( block [ i ] , j ) + beta∗sum_neigh ;
}
double t o t a l _ l l i k e = sum_logs ( log_prob ) ;
double cumProb = 0.0;
z . row ( block [ i ] ) . zeros ( ) ;
for ( unsigned j =0; j < log_prob . n_elem ; j ++)
{
cumProb += exp ( log_prob [ j ] − t o t a l _ l l i k e ) ;
i f ( randU [ block [ i ] ] < cumProb )
{
z ( block [ i ] , j ) = 1;
a l l o c ( block [ i ] , j ) += 1;
break ;

Pseudolikelihood (PL)
Algorithm 2 Metropolis-Hastings with PL
1: Draw proposal β ∼ q(β |β◦)
2: Approximate p(β |z) and p(β◦|z) using equation (7):
ˆpPL(β|z) ≈
n
i=1
exp{β i∼ δ(zi, z )}
k
j=1 exp{β i∼ δ(j, z )}
(7)
3: Calculate the M-H ratio ρ = ˆpPL(β |z)π(β )q(β◦|β )
ˆpPL(β◦|z)π(β◦)q(β |β◦)
4: Draw u ∼ Uniform[0, 1]
5: if u < min(1, ρ) then
6: β ← β
7: else
8: β ← β◦
9: end if
Rydén & Titterington (1998) JCGS 7(2): 194–211

Pseudolikelihood in C++
§
double pseudolike ( const arma : : mat & ne , const arma : : uvec & e , const double b ,
const unsigned n , const unsigned k )
{
double num = 0.0;
double denom = 0.0;
#pragma omp p a r a l l e l for reduction ( + :num, denom)
for ( unsigned i =0; i < n ; i ++)
{
num=num+ne ( e [ i ] , i ) ;
double tdenom =0.0;
for ( unsigned j =0; j < k ; j ++)
{
tdenom=tdenom+exp ( b∗ne ( j , i ) ) ;
}
denom=denom+log ( tdenom ) ;
}
return b∗num−denom ;
}

Approximation Error
PL for n = 12, k = 3 in comparison to the exact likelihood
calculated using a brute force method:
0 1 2 3 4
6810121416
β
µ
exact
pseudolikelihood
(a) Expectation
0 1 2 3 4
0.00.51.01.52.02.5
β
σ
exact
pseudolikelihood
(b) Standard deviation

Thermodynamic Integration (TI)
Path sampling identity:
log
C(β◦)
C(β )
=
β◦
β
E z|β [S(z)] dβ (8)
0.0 0.5 1.0 1.5 2.0
500000100000015000002000000
ϕ
S(x)
Gelman & Meng (1998) Stat. Sci. 13(2): 163–185.

TI algorithm
Algorithm 3 Random walk Metropolis with TI
1: Draw random walk proposal β ∼ q(β |β◦)
2: Estimate S(z|β◦) and S(z|β ) by interpolation
3: Evaluate the deﬁnite integral in equation (8)
4: Calculate the log M-H acceptance ratio:
log{ρ} = log
C(β◦)
C(β )
+ (β − β◦
)S(z) (9)
7: β ← β
8: else
9: β ← β◦
10: end if

TI in C++
§
unsigned pathBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
const arma : : mat & path , const arma : : umat & z , double & beta ,
const double p r i o r _beta [ 2 ] , const double bw)
{
double bprime = rwmh( beta , bw, p r i o r _beta ) ; / / truncated Gaussian
/ / approximate log (Z( bprime ) / Z( beta ) )
double log_ r a t i o = quadrature ( bprime , beta , path )
+ ( bprime−beta ) ∗ sum_ ident ( z , neigh , blocks ) ;
/ / accept / r e j e c t
i f ( u n i f _rand ( ) < exp ( log_ r a t i o ) )
{
beta = bprime ;
return 1;
}
return 0;
}

Approximate Exchange Algorithm (AEA)
Algorithm 4 AEA
1: Draw random walk proposal β ∼ q(β |β◦)
2: Generate w|β by sampling from eq. (3)
3: Calculate the M-H acceptance ratio according to eq. (4):
ρ =
π(β ) exp {β S(z)} C(β◦)
π(β◦) exp {β◦S(z)} C(β )
exp {β◦S(w)} C(β )
exp {β S(w)} C(β◦)
(10)
6: β ← β
7: else
8: β ← β◦
9: end if
Murray, Ghahramani & MacKay (2006) Proc. 22nd
Conf. UAI, 359–366

AEA in C++
§
unsigned exchangeBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
const arma : : uvec & slice , const arma : : umat & z , double & beta ,
const double p r i o r _beta [ 2 ] , const unsigned aux , const bool useSW,
const bool swapAux , const double bw)
{
double bprime = rwmh( beta , bw, p r i o r _beta ) ;
arma : : umat a l l o c = arma : : zeros <arma : : umat >(z . n_rows−1, z . n_cols ) ;
arma : : umat w;
i f ( swapAux ) w = z ;
else w = randomIndices ( z . n_rows−1, z . n_cols ) ;
for ( unsigned i =0; i <aux ; i ++)
{
i f (useSW)
{
swLabelsNoData ( neigh , blocks , bprime , w. n_cols , w, a l l o c ) ;
}
else
{
gibbsLabelsNoData ( neigh , blocks , w, alloc , bprime ) ;
}
}
double sum_z = sum_ ident ( z , neigh , blocks ) ;
double sum_w = sum_ ident (w, neigh , blocks ) ;
double log_ r a t i o = ( bprime−beta )∗sum_z + ( beta−bprime )∗sum_w;
/ / accept / r e j e c t
i f ( u n i f _rand ( ) < exp ( log_ r a t i o ) )
{
beta = bprime ;
return 1;
}
return 0;
}

Approximate Bayesian Computation
Algorithm 5 ABC rejection sampler
1: Draw independent proposal β ∼ π(β)
3: if S(w) − S(z) < then
4: β ← β
5: else
6: β ← β◦
7: end if
Grelaud, Robert, Marin, Rodolphe & Taly (2009) Bayesian Analysis 4(2)
Marin & Robert (2014) Bayesian Essentials with R §8.3

ABC with Metropolis-Hastings
Algorithm 6 ABC-MCMC
1: Draw proposal β ∼ q(β |β◦)
4: if u < π(β )q(β◦|β )
π(β◦)q(β |β◦) and S(w) − S(z) < then
5: β ← β
6: else
7: β ← β◦
8: end if
Marjoram, Molitor, Plagnol & Tavaré (2003) PNAS 100(26): 15324–28

ABC-MCMC in C++
§
unsigned abcBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
const arma : : umat & z , double & beta , const double p r i o r _beta [ 2 ] ,
const unsigned aux , const bool useSW, const bool swapAux ,
const double bw, const double epsilon )
{
double bprime = rwmh( beta , bw, p r i o r _beta ) ;
arma : : umat a l l o c = arma : : zeros <arma : : umat >(z . n_rows−1, z . n_cols ) ;
arma : : umat w;
i f ( swapAux ) w = z ;
else w = randomIndices ( z . n_rows−1, z . n_cols ) ;
for ( unsigned i =0; i <aux ; i ++)
{
i f (useSW)
{
swLabelsNoData ( neigh , blocks , bprime , w. n_cols , w, a l l o c ) ;
}
else
{
gibbsLabelsNoData ( neigh , blocks , w, alloc , bprime ) ;
}
}
double sum_z = sum_ ident ( z , neigh , blocks ) ;
double sum_w = sum_ ident (w, neigh , blocks ) ;
double delta = fabs (sum_w − sum_z ) ;
i f ( delta < epsilon )
{
beta = bprime ;
return 1;
}
return 0;
}

Summary
bayesImageS supports methods for updating the latent labels:
Chequerboard Gibbs sampling (Winkler 2003)
Swendsen-Wang (1987)
and also methods for updating the inverse temperature β:
Pseudolikelihood (Rydén & Titterington 1998)
Thermodynamic integration (Gelman & Meng 1998)
Exchange algorithm (Murray, Ghahramani & MacKay 2006)
Approximate Bayesian computation (Grelaud et al. 2009)
Sequential Monte Carlo (ABC-SMC) with pre-computation
(Del Moral, Doucet & Jasra 2012; Moores et al. 2015)

Appendix
Acknowledgements
Principal supervisor: Kerrie Mengersen
Associate supervisor: Fiona Harden
Radiation Oncology Mater Centre (ROMC), Queensland Health:
Cathy Hargrave
Mike Poulsen
Tim Deegan
Other co-authors:
Christopher Drovandi (QUT)
Anthony N. Pettitt (QUT)
Christian Robert
(University of Warwick & Université Paris Dauphine)
PyMCMC team at QUT:
Clair Alston
Christopher Strickland

Appendix
For Further Reading I
D. Eddelbuettel
Seamless R and C++ integration with Rcpp
Springer-Verlag, 2013.
H. Wickham
R packages
O’Reilly, 2015.
D. Bates & D. Eddelbuettel
Fast and elegant numerical linear algebra using the RcppEigen
package.
J. Stat. Soft. 52(5): 1–24, 2013.
D. Eddelbuettel & C. Sanderson
RcppArmadillo: Accelerating R with high-performance C++ linear
algebra.
Comput. Stat. Data Anal. 71: 1054–63, 2014.

Appendix
For Further Reading II
M. Moores & K. Mengersen
bayesImageS: Bayesian methods for image segmentation using a
hidden Potts model.
R package version 0.3-3
https://CRAN.R-project.org/package=bayesImageS
M. Moores, A. N. Pettitt & K. Mengersen
Scalable Bayesian inference for the inverse temperature of a hidden
Potts model.
arXiv:1503.08066 [stat.CO], 2015.
M. Moores, C. C. Drovandi, K. Mengersen & C. P. Robert
Pre-processing for approximate Bayesian computation in image
analysis.
Statistics & Computing 25(1): 23–33, 2015.
M. Moores & K. Mengersen
Bayesian approaches to spatial inference: modelling and computational
challenges and solutions.
In Proc. 33rd Int. Wkshp MaxEnt, AIP Conf. Proc. 1636: 112–117, 2014.

Appendix
For Further Reading III
G. Winkler
Image analysis, random ﬁelds and Markov chain Monte Carlo methods
2nd
ed., Springer-Verlag, 2003.
J.-M. Marin & C. P. Robert
Bayesian Essentials with R
Springer-Verlag, 2014.
G. O. Roberts & S. K. Sahu
Updating Schemes, Correlation Structure, Blocking and
Parameterization for the Gibbs Sampler
J. R. Stat. Soc. Ser. B 59(2): 291–317, 1997.
T. Rydén & D. M. Titterington
Computational Bayesian Analysis of Hidden Markov Models.
J. Comput. Graph. Stat., 7(2): 194–211, 1998.

Appendix
For Further Reading IV
A. Gelman & X.-L. Meng
Simulating normalizing constants: from importance sampling to bridge
sampling to path sampling.
Statist. Sci., 13(2): 163–185, 1998.
I. Murray, Z. Ghahramani & D. J. C. MacKay
MCMC for Doubly-intractable Distributions.
Proc. 22nd
Conf. UAI, 359–366, 2006.
A. Grelaud, C. P. Robert, J.-M. Marin, F. Rodolphe & J.-F. Taly
ABC likelihood-free methods for model choice in Gibbs random ﬁelds.
Bayesian Analysis 4(2): 317–336, 2009.
R. H. Swendsen & J.-S. Wang
Nonuniversal critical dynamics in Monte Carlo simulations.
Physical Review Letters, 58: 86–8, 1987.

R package 'bayesImageS': a case study in Bayesian computation using Rcpp and OpenMP

More Related Content

What's hot

Similar to R package 'bayesImageS': a case study in Bayesian computation using Rcpp and OpenMP

More from Matt Moores

Recently uploaded

R package 'bayesImageS': a case study in Bayesian computation using Rcpp and OpenMP