The widespread popularity of Bayesian tree-structured regression methods has raised considerable interest in theoretical understanding of their empirical success. However, theoretical literature on methods such as Bayesian CART and BART is still in its infancy. This paper affords new insights about Bayesian CART in the context of structured wavelet shrinkage under the white noise model. We exhibit precise connections between tree-shaped sparsity priors and unstructured spike-and-slab priors, which are regarded as ideal but are rather theoretical in nature. We show that the more practical Bayesian CART priors lead to adaptive rate-minimax posterior concentration in the l∞sense, performing nearly as well as the theoretical ideal (up to a log term). To further explore the benefits of structured shrinkage, we propose the g-prior for trees, which departs from the typical wavelet product priors by harnessing correlation structure induced by the tree topology. While the majority of wavelet type theoretical results for CART focus on dyadic trees, here we do not require that splits are at dyadic locations. We introduce the library of weakly balanced Haar wavelets and show that Bayesian CART is equivalent to Bayesian basis selection from this library. To illustrate that l∞adaptation is an intricate phenomenon, where internal sparsity plays a key role, we show that dense trees are incapable of adaptation. While one of the major appeals of BART is uncertainty quantification via credible sets, asymptotic normality justifications have thus far been unavailable. Building on the l∞adaptation property, we provide new fully non-parametric and adaptive Bernstein-von Mises statements for Bayesian CART using multiscale techniques. (Joint work with Ismael Castillo)

1. Multiscale Analysis of
BART Priors
Veronika Roˇcková1
Joint work with Ismael Castillo 2
Bayesian, Fiducial, and Frequentist (BFF) Conferences
Duke
April 29th
, 2019
1
Booth School of Business, University of Chicago
2
Sorbonne Universite

2. Rise of the Machines and BART
1 / 23

3. Related Existing Work
Regression trees are one of the more widely used learning methods.
Theoretical Developments
Regression Histograms
Nobel (1996), Devroye and Gyorfi (1996), Stone (1985)
CART
Gordon and Olshen (1980), Donoho (1997), Breiman (1984)
Random Forests
Biau (2008), Biau et al. (2008), Scornet et al. (2015), Wager and Walther (2016)
Bayesian Regression Histograms
Coram and Lalley (2006)
Bayesian CART
Rockova and van der Pas (2017)
BART
Rockova and van der Pas (2017), Rockova and Saha (2018)
1 / 23

4. Bayesian CART
CART: Regression trees partition [0,1]p
with nested axis
parallel splits at observed values.
2 / 23

5. Bayesian Additive Regression Trees
BART: Superposition of trees to obtain more flexible partitions.
3 / 23

6. Bayesian CART (a)
PRIOR: Bayesian CART à la Denison, Mallick and Smith (1998)
(1) Poisson prior on the tree size K:
Π(K) =
λK
(eλ − 1)K!
, K = 1,2,..., for some λ > 0. (T3)
(2) Uniform prior over valid trees T :
Π(T K) =
1
card(VK )
I(T ∈ VK
)
where VK are all valid trees with K leaves and splits at X
(3) Gaussian prior on steps β:
Π(β K) =
K
∏
k=1
φ(βk )
4 / 23

7. Bayesian CART (b)
PRIOR: Bayesian CART à la Chipman et al. (1998)
(1*) Splitting probability:
Each node at level d is split with probability
π(d) = Γ−d
for Γ ∈ (2,n)
This corresponds to a heterogeneous Galton-Watson process.
(2*) Prior on splitting variables:
Each predictor chosen with prob 1/p
(3*) Prior on splitting rules:
Uniform prior on available splits
(4*) Gaussian prior on steps β:
5 / 23

8. The White Noise Model
For the Wiener process W(t) for t ∈ [0,1] and f0 ∈ Hα
we have
dY(n)
(t) = f0(t)dt +
1
√
n
dW(t).
Given basis functions ψlk (x), one obtains
Ylk = β0
lk +
1
√
n
εlk , for εlk
iid
∼ N(0,1),
and for multiscale coefficients
β0
lk = ∫
1
0
f0(t)ψlk (t)dt l ≥ 0,0 ≤ k < 2l
.
A prototypical example is the standard Haar wavelet basis
ψ−10(x) = I[0,1](x) and ψlk (x) = 2l/2
ψ(2l
x − k) (1)
obtained with orthonormal dilation-translations of a wavelet function
ψ = I(0,1/2] − I(1/2,1].
6 / 23

9. Haar Wavelets and Dyadic Trees
Ψ00
0.0 0.2 0.4 0.6 0.8 1.0
−1.0−0.50.00.51.0
Ψ00
Ψ10 Ψ11
0.0 0.2 0.4 0.6 0.8 1.0
−1.5−1.0−0.50.00.51.01.5
Ψ10
0.0 0.2 0.4 0.6 0.8 1.0
−1.5−1.0−0.50.00.51.01.5
Ψ11
Ψ20 Ψ21 Ψ22 Ψ23
0.0 0.2 0.4 0.6 0.8 1.0
−2−1012
Ψ20
0.0 0.2 0.4 0.6 0.8 1.0
−2−1012
Ψ21
0.0 0.2 0.4 0.6 0.8 1.0
−2−1012
Ψ22
0.0 0.2 0.4 0.6 0.8 1.0
−2−1012
Ψ23
7 / 23

10. Haar Wavelets and Dyadic Trees
A tree T = Tint ∪ Text is a collection of nodes (l,k).
Two sets of coefficients:
(a) the internal coefficients βlk for wavelets ψlk for (l,k) ∈ Tint
(b) the external coefficients ̃βlk for intervals Ilk for (l,k) ∈ Text .
(0,0)
(1,1)
(2,3)(2,2)
(3,5)(3,4)
(1,0)
The tree partitions (0,1] into
{Ilk }(l,k)∈Text
= {(0,1/2],(1/2,5/8],(5/8,3/4],(3/4,1]}. (2)
8 / 23

11. Haar Wavelets and Dyadic Trees
Histogram reconstruction
fT ,̃β(x) = ∑
(l,k)∈Text
̃βlk IIlk
(x), (3)
where Ilk are the dyadic intervals of the terminal nodes (l,k) ∈ Text .
One can rewrite (3) in terms of the wavelet coefficients as
fT ,̃β(x) = β−10ψ−10(x) + ∑
(l,k)∈Tint
βlk ψlk (x). (4)
From (4), it can be seen that
̃βlk = β−10 +
l−1
∑
j=0
2j/2
βj⌊k/2l−j ⌋s⌊k/2l−j−1⌋, (5)
where sk = (−1)k+1
.
There is a pinball game metaphor behind (5).
9 / 23

12. Haar Wavelets and Dyadic Trees
The Pinball Game
β−10
1
β00
−1 1
β10
−
√
2
√
2
β11
−
√
2
√
2
β20
−2 2
β21
−2 2
β22
−2 2
β23
−2 2
̃β30
̃β31
̃β33
̃β34
̃β35
̃β36
̃β37
̃β38
Binary flat tree obtained from dyadic splits.
The edges are weighted by the “size" of the Haar wavelets.
10 / 23

13. Tree-shaped Sparsity
We propose a prior on Haar wavelet coefficients, that weeds out
coefficients βlk that are not supported by the tree skeleton
Such tree-shaped prior is defined as
π ({βlk }l,k T ) ∼ ∏
(l,k)∈Tint
π(βlk ) × ∏
(l,k)∉Tint
δ0(βlk ), (6)
where π(βlk ) is some suitable density.
The major appeal of (6) is tree-shaped sparsity which has two
shrinkage implications:
(a) global: all coefficients at higher resolutions are zero,
(b) local: not all coefficients at lower resolutions are nonzero.
Denote with βT the internal wavelet coefficients. We will assume a
more general prior
π ({βlk }l,k T ) ∼ π(βT ) × ∏
(l,k)∉Tint
δ0(βlk ). (7)
11 / 23

14. Tree-shaped Sparsity
β00
−1 1
β10
−
√
2
√
2
̃β11
̃β20 β21
−2 2
̃β32
̃β33
We can rewrite (5) in matrix notation as
̃βT = AT βT . (8)
and where AT is the pinball matrix
⎛
⎜
⎜
⎜
⎜
⎝
̃β11
̃β20
̃β32
̃β33
⎞
⎟
⎟
⎟
⎟
⎠
=
⎛
⎜
⎜
⎜
⎜
⎝
1 1 0 0
1 −1 −
√
2 0
1 −1
√
2 −2
1 −1
√
2 2
⎞
⎟
⎟
⎟
⎟
⎠
⎛
⎜
⎜
⎜
⎝
β−10
β00
β10
β21
⎞
⎟
⎟
⎟
⎠
.
12 / 23

15. Introducing the g-prior for trees
The classical wavelet prior starts with βT ∼ N(0,I Text
). This yields
̃βT ∼ N(0,DT ), where DT = AT A′
T . (9)
The Bayesian CART prior starts with ̃βT ∼ N(0,gnI Text
)for some
gn > 0. This yields a g-prior for trees
βT ∼ N (0,gn (A′
T AT )−1
) (10)
Note that AT is not necessarily orthogonal, where the prior seizes the
correlation structure between the neighboring wavelet coefficients.
Denote with (l1,k1) the deepest right-most internal node in the tree T
Let T −
be a tree obtained from T by turning (l1,k1) into a terminal
node. Then
A′
T AT = (
A′
T − AT − + vv′
0
0′
2l1+1) (11)
13 / 23

16. Connection to Spike-and-Slab Priors
Chipman, Kolaczyk and McCulloch (1997) propose a spike-and-slab
prior for wavelet shrinkage.
Hoffmann et al. (2015), Ray (2018) study the following variant:
π(βlk γlk ) = γlk π(βlk ) + (1 − γlk )δ0(βlk ), (12)
where γlk ∈ {0,1} for whether or not the coefficient is active with
P(γlk = 1 θl ) = θl n−a
≤ θl ≤ 2−l(1+b)
.
There is a link: One can regard γlk as the node splitting indicator
γlk = I[(l,k) ∈ T ].
(a) Under the spike-and-slab prior, γlk ’s are independent,
(b) Under the Bayesian CART, γlk ’s are hierarchically constrained,
where the pattern of non-zeroes encodes the tree oligarchy.
14 / 23

17. Bayesian Dyadic CART is Multiscale
We define Hölder-type space of continuous functions on [0,1] as
H(α,M) ≡ {f ∈ C([0,1]) max
0≤k<2l
⟨f,ψlk ⟩ ≤ M 2−l( 1
2 +α)
}. (13)
Consider one of the Bayesian CART tree priors, i.e either
(i) Bayesian CART (a) prior with π(d) = Γ−d
for 2 < Γ < n,
(ii) Bayesian CART (b) prior with λ = 1/nc
for some c > 0 in (T3),
Moreover, consider the tree-shaped wavelet prior (7) with
π(βT ) ∼ N(0,ΣT ),
where ΣT is either I Text
or gn(A′
T AT )−1
with gn = n.
Then for any Mn → ∞, we have for n → ∞
Ef0
Π[ fT ,β − f0 ∞ > Mnεn Y] → 0,
where
εn =
√
logn (
logn
n
)
α
2α+1
. (14)
15 / 23

18. Adaptation in ∞
β−10
1
β00
−1 1
β10
−
√
2
√
2
β11
−
√
2
√
2
β20
−2 2
β21
−2 2
β22
−2 2
β23
−2 2
̃β30
̃β31
̃β33
̃β34
̃β35
̃β36
̃β37
̃β38
A tempting strategy to attain adaptation for flat trees would be to treat
depth L as random with a prior.
Such a prior can be also written as
π ({βlk }lk L) = ∏
l,k
̃π(βlk L), (15)
where
̃π(βlk L) = π(βlk )I(l ≤ L) + δ0(βlk )I(l > L).
Assuming that L ∼ π(L), this prior has spike-and-slab marginals
π(βlk ) = π(βlk )P(l ≤ L) + δ0(βlk )P(l > L).
16 / 23

19. Flat Trees are Not Multiscale
Let ψlk be a wavelet basis with S ≥ 1 vanishing moments (e.g. Haar
and S = 1, or CDV-type boundary corrected and S ≥ 1 an integer).
Let Π be the flat-tree prior with πL(L) ∝ e−L
with L ∈ {0,1,...,log2 n}
and Gaussian iid (0,1) heights. For any 0 < α ≤ S, M > 0, there exists
f0 ∈ H(α,M) such that
Ef0
Π [ ∞(f,f0) < ζn X] → 0,
where the lower-bound rate ζn is given by
ζn = (
logn
n
)
α
2α+2
.
17 / 23

20. Unbalanced Haar Wavelets
(a) The first breakpoint b00 is selected from X ∩ (0,1) and one sets
l00 = 0,r00 = 1.
(b) For each 1 ≤ l ≤ Lmax and 0 ≤ k < 2l
set
llk = l(l−1)⌊k/2⌋, rlk = b(l−1)⌊k/2⌋, if k is even,
llk = b(l−1)⌊k/2⌋, rlk = r(l−1)⌊k/2⌋, if k is odd.
Choose blk from X ∩ (llk ,rlk ].
Each collection of split locations B = (blk )(l,k) gives rise to nested
intervals
Llk = (llk ,blk ] and Rlk = (blk ,rlk ].
Starting with the mother wavelet ψB
−10 = ψ−10 = I(0,1), one then
recursively constructs wavelet functions ψB
lk from Llk and Rlk as
ψB
lk (x) =
1
√
Llk
−1 + Rlk
−1
(
ILlk
(x)
Llk
−
IRlk
(x)
Rlk
). (16)
18 / 23

21. Haar Wavelets and Non-Dyadic Trees
Ψ00
0.0 0.2 0.4 0.6 0.8 1.0
−0.50.00.51.0
Ψ00
3 8
Ψ10 Ψ11
0.0 0.2 0.4 0.6 0.8 1.0
−1.0−0.50.00.51.01.52.0
Ψ10
1 8
0.0 0.2 0.4 0.6 0.8 1.0
−1.0−0.50.00.51.01.5
Ψ11
5 8
Ψ20 Ψ21 Ψ22 Ψ23
0.0 0.2 0.4 0.6 0.8 1.0
−3−2−10123
Ψ20
1 16
0.0 0.2 0.4 0.6 0.8 1.0
−3−2−101
Ψ21
0.3
0.0 0.2 0.4 0.6 0.8 1.0
−10123
Ψ22
7 16
0.0 0.2 0.4 0.6 0.8 1.0
−1.5−1.0−0.50.00.51.01.5
Ψ23
0.8
19 / 23

22. Weakly Balanced Haar Wavelets
In order to control the combinatorial complexity, we require that the
UH wavelets are not too imbalanced.
Consider a collection of UH wavelets ΨB
= {ψB
−10,ψB
lk (l,k)}.
We say that ΨB
is weakly balanced with constants C,D ∈ N/{0} if
max( Llk , Rlk ) =
Mlk
2l+D
for some Mlk ∈ {1,...,C + l} ∀(l,k).
For instance, setting D = 2 and C = 3 one can build weakly
unbalanced wavelets by first choosing
b00 ∈ {
1
4
,
2
4
,
3
4
}.
If we pick b00 = 3/4, the next split b10 can be chosen from among
b10 ∈ {
2
8
,
3
8
,
4
8
}
while b11 would have to be set as 7/8.
20 / 23

23. Bayesian Non-dyadic CART is Multiscale
We regard the non-dyadic CART prior on regression functions f as
arising from the following three steps:
↝ Step 1. (Basis) Sample B = (blk )0≤k<2l −1,l≤L so that the system
ΨB
is weakly balanced.
↝ Step 2. (Tree) Independently of B, sample a binary tree T .
↝ Step 3. (Coefficients) Given T , we obtain the coefficients {βlk }
from the tree-shaped prior (7).
Let f0 ∈ H(α,M) for some M > 0 and 0 < α ≤ 1 and define
εn = (logn)2
(
logn
n
)
α
2α+1
. (17)
Then for any Mn → ∞, we have for n → ∞
Ef0
Π[ fT ,β − f0 ∞ > Mn εn X] → 0.
21 / 23

24. From ⋅ ∞ Convergence to Non-parametric BvM
Infinite-dimensional BvM fails to hold in a basic Gaussian
conjugate 2-sequence space (Freedman (1986))
In weaker topologies, non-parametric BvM can be formalized
(Castillo and Nickl (2015))
The multi-scale space: for an increasing sequence w = {wl }
M(w) = {f = {βlk } f M ≡ sup
l
maxk βlk
wl
< ∞}
We work in a subspace
M0 = {f ∈ M(w) lim
l→∞
maxk βlk
wl
= 0}.
Define ̃Πn(B) = Π(
√
n(f − Y) ∈ B Y) for any Borel set B.
The weak non-parametric BvM phenomenon occurs when
βM0
(̃Πn,N) → 0 in Pn
0-probability,
where N is the law of the white noise W in M0 and where βM0
is the
bounded Lipschitz metric.
22 / 23

25. Adaptive BvM’s
The non-parametric BvM theorem holds for suitable product priors
when α is known (the prior has to undersmooth) (Castillo and Nickl
(2015))
Adaptive BvM results have already been obtained for
spike-and-slab priors on wavelet coefficients (Ray (2016))
We show that adaptive BVM occurs also for trees!
23 / 23

26. Some References
Castillo, I. and Roˇcková, V. (2019)
Multiscale analysis of BART Priors
Coming soon
Roˇcková, V. (2019)
On Semi-parametric Bernstein-von Mises Theorems for BART
Manuscript
Roˇcková, V. and Saha, E. (2019)
On Theory for BART
Proceedings of the Artificial Intelligence and Statistics Conference 2019
Roˇcková, V. and van der Pas, S. (2017)
Posterior Concentration for Bayesian Regression Trees and Forests
The Annals of Statistics (In Revision)
Liu, Y., Roˇcková, V. and Wang, Y. (2018)
ABC Variable Selection with Bayesian Forests
Submitted
23 / 23