For the discovery of a regression relationship between y and x, a vector of p potential predictors, the flexible nonparametric nature of BART (Bayesian Additive Regression Trees) allows for a much richer set of possibilities than restrictive parametric approaches. To exploit the potential monotonicity of the predictors, we introduce mBART, a constrained version of BART that incorporates monotonicity with a multivariate basis of monotone trees, thereby avoiding the further confines of a full parametric form. Using mBART to estimate such effects yields (i) function estimates that are smoother and more interpretable, (ii) better out-of-sample predictive performance and (iii) less post-data uncertainty. By using mBART to simultaneously estimate both the increasing and the decreasing regions of a predictor, mBART opens up a new approach to the discovery and estimation of the decomposition of a function into its monotone components. (This is joint work with H. Chipman, R. McCulloch and T. Shively).

1. Multidimensional Monotonicity Discovery
with mBART
Ed George
Wharton, University of Pennsylvania
edgeorge@upenn.edu
Collaborations with:
Hugh Chipman (Acadia)
Rob McCulloch (Arizona State)
Tom Shively (UT Austin)
Bayesian, Fiducial, and Frequentist Conference
Duke University
April 29, 2019
1 / 45

2. Plan
I) Review BART
II) Introduce Monotone BART: mBART
III) Monotonicity Discovery with mBART
2 / 45

3. Part I. BART (Bayesian Additive Regression Trees)
Data: n observations of y and x = (x1, ..., xp)
Suppose: Y = f (x) + , symmetric with mean 0
Bayesian Ensemble Idea: Approximate unknown f (x) by the form
f (x) = g(x; θ1) + g(x; θ2) + ... + g(x; θm)
θ1, θ2, . . . , θm iid ∼ π(θ)
and use the posterior of f given y for inference.
BART is obtained when each g(x; θj ) is a regression tree.
Key calibration: Using y, set π(θ) so that Var(f ) ≈ Var(y).
3 / 45

4. Beginning with a Single Tree Model
4 / 45

5. Bayesian CART: Just add a prior π(M, T)
Bayesian CART Model Search
(Chipman, George, McCulloch 1998)
π(M, T) = π(M | T)π(T)
π(M | T) : (µ1, µ2, . . . , µb) ∼ Nb(0, τ2I)
π(T): Stochastic process to generate tree skeleton plus uniform
prior on splitting variables and splitting rules.
Closed form for π(T | y) facilitates MCMC stochastic search for
promising trees.
5 / 45

6. Moving on to BART
Bayesian Additive Regression Trees
(Chipman, George, McCulloch 2010)
The BART ensemble model
Y = g(x; T1, M1)+g(x; T2, M2)+. . .+g(x; Tm, Mm)+σz, z ∼ N(0, 1)
Each (Ti , Mi ) identifies a single tree.
E(Y | x, T1, M1, . . . , Tm, Mm) is the sum of m bottom node µ’s,
one from each tree.
Number of trees m can be much larger than sample size n.
g(x; T1, M1), g(x; T2, M2), ..., g(x; Tm, Mm) is a highly redundant
“over-complete basis” with many many parameters.
6 / 45

7. Complete the Model with a Regularization Prior
π((T1, M1), (T2, M2), . . . , (Tm, Mm), σ)
π applies the Bayesian CART prior to each (Tj , Mj ) independently
so that:
Each T small.
Each µ small.
σ will be compatible with the observed variation of y.
The observed variation of y is used to guide the choice of the
hyperparameters for the µ and σ priors.
π is a “regularization prior” as it keeps the contribution of each
g(x; Ti , Mi ) small, explaining only a small portion of the fit.
7 / 45

8. Build up the fit, by adding up tiny bits of fit ..
8 / 45

9. Connections to Other Modeling Ideas
Bayesian Nonparametrics:
Lots of parameters (to make model flexible)
A strong prior to shrink towards simple structure (regularization)
BART shrinks towards additive models with some interaction
Dynamic Random Basis:
g(x;T1,M1), ..., g(x;Tm,Mm) are dimensionally adaptive
Gradient Boosting:
Overall fit becomes the cumulative effort of many “weak learners”
Connections to Other Modeling Ideas
Y = g(x;T1,M1) + ... + g(x;Tm,Mm) + & z
plus
#((T1,M1),....(Tm,Mm),&)
12
Bayesian Nonparametrics:
Lots of parameters (to make model flexible)
A strong prior to shrink towards simple structure (regularization)
BART shrinks towards additive models with some interaction
Dynamic Random Basis Elements:
g(x; T1, M1), ..., g(x; Tm, Mm) are dimensionally adaptive
Gradient Boosting:
Fit becomes the cumulative effort of many weak learners
9 / 45

10. A Sketch of the BART MCMC Algorithm
Bayesian Nonparametrics:
Lots of parameters (to make model flexible)
A strong prior to shrink towards simple structure (regularization)
BART shrinks towards additive models with some interaction
Dynamic Random Basis:
g(x;T1,M1), ..., g(x;Tm,Mm) are dimensionally adaptive
Gradient Boosting:
Overall fit becomes the cumulative effort of many “weak learners”
Connections to Other Modeling Ideas
Y = g(x;T1,M1) + ... + g(x;Tm,Mm) + & z
plus
#((T1,M1),....(Tm,Mm),&)
12
Bayesian Backfitting: Outer loop is a “simple” Gibbs sampler
(Ti , Mi ) | Y , all other (Tj , Mj ), and σ
σ | Y , (T1, M1, . . . , . . . , Tm, Mm)
To draw (Ti , Mi ) above, subtract the contributions of the other
trees from both sides to get a simple one-tree model.
We integrate out M to draw T and then draw M | T.
10 / 45

11. For the draw of T we use a Metropolis-Hastings within Gibbs step.
Our proposal moves around tree space by proposing local
modifications such as the “birth-death” step:Because p(T | data) is available in closed form (up to a norming constant),
we use a Metropolis-Hastings algorithm.
Our proposal moves around tree space by proposing local modifications
such as
=>
?
=>
?
propose a more complex tree
propose a simpler tree
Such modifications are accepted according to their compatibility
with p(T | data).
20
Simulating p(T | data) with the Bayesian CART Algorithm
... as the MCMC runs, each tree in the sum will grow and shrink,
swapping fit amongst them ....
11 / 45

12. Using the MCMC Output to Draw Inference
Each iteration d results in a draw from the posterior of f
ˆfd (·) = g(·; T1d , M1d ) + · · · + g(·; Tmd , Mmd )
To estimate f (x) we simply average the ˆfd (·) draws at x
Posterior uncertainty is captured by variation of the ˆfd (x)
eg, 95% credible region estimated by middle 95% of values
Can do the same with functionals of f .
12 / 45

13. Out of Sample Prediction
Predictive comparisons on 42 data sets.
Data from Kim, Loh, Shih and Chaudhuri (2006) (thanks Wei-Yin Loh!)
p = 3 to 65, n = 100 to 7,000.
for each data set 20 random splits into 5/6 train and 1/6 test
use 5-fold cross-validation on train to pick hyperparameters (except
BART-default!)
gives 20*42 = 840 out-of-sample predictions, for each prediction, divide rmse
of different methods by the smallest
+ each boxplots represents
840 predictions for a
method
+ 1.2 means you are 20%
worse than the best
+ BART-cv best
+ BART-default (use default
prior) does amazingly
well!!
RondomForestsNeuralNetBoostingBART−cvBART−default
1.0 1.1 1.2 1.3 1.4 1.5
13 / 45

14. Automatic Uncertainty Quantification
A simple simulated 1-dimensional example
−1.0 −0.5 0.0 0.5 1.0
−1.0−0.50.00.51.0
x
y
95% pointwise posterior intervals, BART
posterior mean
true f
−1.0 −0.5 0.0 0.5 1.0
−1.0−0.50.00.51.0
x
y
95% pointwise posterior intervals, mBART
Note: mBART on the right plot to be introduced next
14 / 45

15. Part II. Monotone BART - mBART
Multidimensional Monotone BART
(Chipman, George, McCulloch, Shively 2018)
Idea:
Approximate multivariate monotone functions by the sum of many
single tree models, each of which is monotonic.
15 / 45

16. An Example of a Monotonic Tree
x1
x2
f(x)
Three different views of
a bivariate monotonic
tree.
16 / 45

17. What makes this single tree monotonic?
x1
x2
f(x)
A function g is said to be monotonic in xi if for any δ > 0,
g(x1, x2, . . . , xi + δ, xi+1, . . . , xk; T, M)
≥ g(x1, x2, . . . , xi , xi+1, . . . , xk; T, M).
For simplicity and wlog, let’s restrict attention to monotone
nondecreasing functions.
17 / 45

18. Constraining a tree to be monotone is easy: we simply constrain
the mean level of a node to be greater than those of its below
neighbors and less than those of its above neighbors.
0.2 0.4 0.6 0.8
0.20.40.60.8
x1
x2
4
10
11
12
13
7
node 7 is disjoint from node 4.
node 10 is a below neighbor of node 13.
node 7 is an above neighbor of node 13.
The mean level of node 13 must be greater than those of 10 and
12 and less than that of node 7.
18 / 45

19. The mBART Prior
Recall the BART parameter
θ = ((T1, M1), (T2, M2), . . . , (Tm, Mm), σ)
Let S = {θ : every tree is monotonic in a desired subset of xi s}
To impose the monotonicity we simply truncate the BART prior
π(θ) to the set S
π∗
(θ) ∝ π(θ) IS (θ)
where IS (θ) is 1 if every tree in θ is monotonic.
19 / 45

20. A New BART MCMC “Christmas Tree” Algorithm
π((T1, M1), (T2, M2), . . . , (Tm, Mm), σ | y))
Bayesian backfitting again: Iteratively sample each (Tj , Mj ) given
(y, σ) and other (Tj , Mj )’s
Each (T0, M0) → (T1, M1) update is sampled as follows:
Denote move as
(T0, M0
Common, M0
Old ) → (T1, M0
Common, M1
New )
Propose T∗ via birth, death, etc.
If M-H with π(T, M | y) accepts (T∗, M0
Common)
Set (T1
, M1
Common) = (T∗
, M0
Common)
Sample M1
New from π(MNew | T1
, M1
Common, y)
Only M0
Old → M1
New needs to be updated.
Works for both BART and mBART.
20 / 45

21. Example: Product of two x’s
Let’s consider a very simple simulated monotone example:
Y = x1 x2 + , xi ∼ Uniform(0, 1).
Here is the plot of the true function f (x1, x2) = x1 x2
0.2 0.4 0.6 0.8 1.0
0.20.40.60.81.0
x1
x2
x1
x2
f(x)
21 / 45

22. First we try a single (just one tree), unconstrained tree model.
Here is the graph of the fit.
0.2 0.4 0.6 0.8
0.20.40.60.8
x1
x2
x1
x2
f(x)
The fit is not terrible, but there are some aspects of the fit which
violate monotonicity.
22 / 45

23. Here is the graph of the fit with the monotone constraint:
0.2 0.4 0.6 0.8
0.20.40.60.8
x1
x2
x1
x2
f(x)
We see that our fit is monotonic, and more representative of the
true f .
23 / 45

24. Here is the unconstrained BART fit:
0.2 0.4 0.6 0.8
0.20.40.60.8
x1
x2
x1
x2
f(x)
Much better (of course) but not monotone!
24 / 45

25. And, finally, the constrained BART fit:
0.2 0.4 0.6 0.8
0.20.40.60.8
x1
x2
x1
x2
f(x)
Not Bad!
Same method works with any number of x’s!
25 / 45

26. Example: MSE Reduction by Monotone Regularization
Y = x1 x2
2 + x3 x3
4 + x5 + ,
∼ N(0, σ2
), xi ∼ Uniform(0, 1).
For various values of σ, we simulated 5,000 observations.
26 / 45

27. RMSE improvement over unconstrained BART
qqq
q
q
q
q
qq
q
qq
q
qq
q
q
bart−1 mbart−1 bart−2 mbart−2 bart−3 mbart−3 bart−4 mbart−4
0.100.150.200.250.300.35
σ = 0.2, 0.5, 0.7, 1.0
27 / 45

28. Part III. Discovering Monotonicity with mBART
Suppose we don’t know if f (x) is monotone up, monotone down or
even monotone at all.
Of course, a simple strategy would be simply compare the fits from
BART and mBART.
Good news, we can do much better than this!
As we’ll now see, mBART can be deployed to simultaneously
estimate all the monotone components of f .
With this strategy, monotonicity can be discovered rather than
imposed!
28 / 45

29. The Monotone Decomposition of a Function
To begin simply, suppose x is one-dimensional and f is of bounded
variation.
Any such f can be uniquely written (up to an additive
constant) as the sum of a monotone up function and a
monotone down function
f (x) = fup(x) + fdown(x)
where
when f (x) is increasing, fup(x) increases at the same
rate and is flat otherwise,
when f (x) is decreasing, fdown(x) decreases at the
same rate and is flat otherwise.
29 / 45

30. More precisely, when f is differentiable,
fup(x) =
f (x) when f (x) > 0
0 when f (x) ≤ 0
and
fdown(x) =
f (x) when f (x) < 0
0 when f (x) ≥ 0
Notice the orthogonal decomposition of f
f (x) = fup(x) + fdown(x)
30 / 45

31. The Discovery Strategy with mBART
Key Idea: To discover the monotone decomposition of f , we
embed f (x) as a two-dimensional function in R2,
f (x) = f ∗
(x, x) = fup(x) + fdown(x).
Letting x1 = x2 = x be duplicate copies of x, we apply mBART to
estimate f ∗(x1, x2)
constrained to be monotone up in the x1 direction, and
constrained to be monotone down in the x2 direction.
We are effectively estimating the monotone projections of
f ∗(x1, x2) onto the x1 and x2 axes
P[x1]f ∗(x1, x2) = fup(x1)
P[x2]f ∗(x1, x2) = fdown(x2)
31 / 45

32. Example: Suppose Y = x3 + .
q
q
q
q
qq
q
q
q q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qqqq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q qq
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
qq
q
q
qq
q
q
q
q
q
q
q
−1.0 −0.5 0.0 0.5 1.0
−1.0−0.50.00.5
x
y
BART and mBART
BART
mBART
q
q
q
q
qq
q
q
q q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qqqq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q qq
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
qq
q
q
qq
q
q
q
q
q
q
q
−1.0 −0.5 0.0 0.5 1.0
−1.0−0.50.00.5
x
y
mBARTD, fup, fdown
mBART: fup
mBART: fdown
mBARTD: overall fit
Note that ˆfdown ≈ 0 (the red in the right plot), as we would expect
when f is monotone up.
32 / 45

33. As the sample size is increased from 200 to 1,000, ˆfdown gets even
flatter.
q
q
q
qq
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
qq
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
qq
q
q
q
qqq
q
q
q
qq
q
q
q
q
q
qq
qq
q
q
q
q
qq
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
qqq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
qq
q
qqq
qq
q
q
q
q
q
qq
q
qq
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
qq
qq
q
q
qq
q
q
q
q
qq
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqqqq
q
q
q
q
q
qq
q
q
qq
q
q
q
qq
q
q
q
qq
q
q
qq
q
q
q
q
q
qq
qq
qq
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
qq
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
qqq
q
q
q
q
q
q
qqqq
q
q
q
qq
q
q
q
q
q
qq
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
qqq
q
qq
q
q
q
q
q
q
q
q
qq
q
qqq
qq
q
q
q
qq
q
q
qq
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
qqq
q
qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
qqq
qq
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
qq
q
q
q
q
qq
q
q
−1.0 −0.5 0.0 0.5 1.0
−1.0−0.50.00.51.0
x
y
BART and mBART
BART
mBART
q
q
q
qq
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
qq
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
qq
q
q
q
qqq
q
q
q
qq
q
q
q
q
q
qq
qq
q
q
q
q
qq
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
qqq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
qq
q
qqq
qq
q
q
q
q
q
qq
q
qq
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
qq
qq
q
q
qq
q
q
q
q
qq
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqqqq
q
q
q
q
q
qq
q
q
qq
q
q
q
qq
q
q
q
qq
q
q
qq
q
q
q
q
q
qq
qq
qq
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
qq
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
qqq
q
q
q
q
q
q
qqqq
q
q
q
qq
q
q
q
q
q
qq
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
qqq
q
qq
q
q
q
q
q
q
q
q
qq
q
qqq
qq
q
q
q
qq
q
q
qq
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
qqq
q
qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
qqq
qq
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
qq
q
q
q
q
qq
q
q
−1.0 −0.5 0.0 0.5 1.0
−1.0−0.50.00.51.0
x
y
mBARTD, fup, fdown
mBART: fup
mBART: fdown
mBARTD: overall fit
Suggests consistent estimation of the monotone components!!
33 / 45

34. Example: Suppose Y = x2 + .
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
qq
q
q
qq
q
q
q
q
q
q
q
−1.0 −0.5 0.0 0.5 1.0
0.00.20.40.60.81.0
x
y
BART and mBART
BART
mBART
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
qq
q
q
qq
q
q
q
q
q
q
q
−1.0 −0.5 0.0 0.5 1.0
0.00.20.40.60.81.0
x
y
mBARTD, fup, fdown
mBART: fup
mBART: fdown
mBARTD: overall fit
On the left, BART is good, but simple mBART is not.
On the right, ˆfup and ˆfdown are spot on.
And mBARTD = ˆfup + ˆfdown seems just as good as BART!
34 / 45

35. Example: Suppose Y = sin(x) + .
qq
qqqq
qqqq
qqqq
q
q
qq
q
q
q
q
qq
qqq
qq
qqq
qq
q
q
qq
q
qq
q
q
q
qqq
q
q
q
q
q
q
qqqqqq
qq
q
qqq
qq
q
q
q
q
q
q
qq
q
qqqqqqqqq
q
qqqqq
q
q
qqqq
qqqqq
q
q
q
q
qqqq
qq
qqqq
q
qqq
q
q
q
qqq
qq
qq
q
q
q
qqq
q
q
q
q
qqq
q
q
q
qq
q
q
qqqqqqqqqq
qq
q
q
q
qq
qqq
q
q
qqq
q
q
q
q
q
q
qq
q
q
q
q
q
qqqq
q
qqq
qq
q
q
qq
qqq
q
qqq
qq
q
qqq
qq
q
q
q
qq
q
q
q
qq
qq
qqqq
q
qqqqq
q
qq
q
q
q
qq
q
q
q
q
q
qqq
q
q
q
qq
q
q
qqqqq
q
q
qq
q
qq
q
qqq
qqq
q
qqqqqq
q
qq
q
qqqq
q
qqqq
qqqq
qq
q
qqq
qq
q
qqq
q
qq
qqqqq
qqqq
q
qq
q
q
q
q
q
qqqq
qqq
qq
qq
q
q
q
qqq
q
q
q
qq
qq
q
qq
qq
q
q
qq
qqq
qqqqq
qqqq
qq
q
q
qqq
q
qqq
q
q
q
q
q
q
qqqq
q
q
qqqqqqq
q
q
q
q
qq
q
qqq
q
qq
q
qqq
qq
q
qqqq
qqq
qqqqq
q
q
q
q
q
q
q
qq
qqqqq
q
q
qqq
qqqqqqqq
q
q
qq
qq
q
q
qqqq
q
qqq
qqqq
q
qq
q
q
q
q
qq
−10 −5 0 5 10
−3−2−10123
x
y
BART and mBART
BART
mBART
qq
qqqq
qqqq
qqqq
q
q
qq
q
q
q
q
qq
qqq
qq
qqq
qq
q
q
qq
q
qq
q
q
q
qqq
q
q
q
q
q
q
qqqqqq
qq
q
qqq
qq
q
q
q
q
q
q
qq
q
qqqqqqqqq
q
qqqqq
q
q
qqqq
qqqqq
q
q
q
q
qqqq
qq
qqqq
q
qqq
q
q
q
qqq
qq
qq
q
q
q
qqq
q
q
q
q
qqq
q
q
q
qq
q
q
qqqqqqqqqq
qq
q
q
q
qq
qqq
q
q
qqq
q
q
q
q
q
q
qq
q
q
q
q
q
qqqq
q
qqq
qq
q
q
qq
qqq
q
qqq
qq
q
qqq
qq
q
q
q
qq
q
q
q
qq
qq
qqqq
q
qqqqq
q
qq
q
q
q
qq
q
q
q
q
q
qqq
q
q
q
qq
q
q
qqqqq
q
q
qq
q
qq
q
qqq
qqq
q
qqqqqq
q
qq
q
qqqq
q
qqqq
qqqq
qq
q
qqq
qq
q
qqq
q
qq
qqqqq
qqqq
q
qq
q
q
q
q
q
qqqq
qqq
qq
qq
q
q
q
qqq
q
q
q
qq
qq
q
qq
qq
q
q
qq
qqq
qqqqq
qqqq
qq
q
q
qqq
q
qqq
q
q
q
q
q
q
qqqq
q
q
qqqqqqq
q
q
q
q
qq
q
qqq
q
qq
q
qqq
qq
q
qqqq
qqq
qqqqq
q
q
q
q
q
q
q
qq
qqqqq
q
q
qqq
qqqqqqqq
q
q
qq
qq
q
q
qqqq
q
qqq
qqqq
q
qq
q
q
q
q
qq
−10 −5 0 5 10
−3−2−10123
x
y
mBARTD, fup, fdown
mBART: fup
mBART: fdown
mBARTD: overall fit
BART is great, but simple mBART reveals nothing.
ˆfup and ˆfdown have discovered the monotone decomposition.
And mBARTD = ˆfup + ˆfdown is great too.
To extend this approach to multidimensional x, we simply
duplicate each and every component of x !!!
35 / 45

36. Example: House Price Data
Let’s look at a simple real example where y = house price,
and x = three characteristics of each house.
> head(x)
nbhd size brick
[1,] 2 1.79 0
[2,] 2 2.03 0
[3,] 2 1.74 0
[4,] 2 1.98 0
[5,] 2 2.13 0
[6,] 1 1.78 0
> dim(x)
[1] 128 3
> summary(x)
nbhd size brick
Min. :1.000 Min. :1.450 Min. :0.0000
1st Qu.:1.000 1st Qu.:1.880 1st Qu.:0.0000
Median :2.000 Median :2.000 Median :0.0000
Mean :1.961 Mean :2.001 Mean :0.3281
3rd Qu.:3.000 3rd Qu.:2.140 3rd Qu.:1.0000
Max. :3.000 Max. :2.590 Max. :1.0000
> summary(y)
Min. 1st Qu. Median Mean 3rd Qu. Max.
69.1 111.3 126.0 130.4 148.2 211.2
y: dollars (thousands).
x: nbhd (categorial), size (sq ft thousands), brick (indicator).
36 / 45

37. Call:
lm(formula = price ~ nbhd + size + brick, data = hdat)
Residuals:
Min 1Q Median 3Q Max
-30.049 -8.519 0.137 7.640 36.912
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 18.725 10.766 1.739 0.0845 .
nbhd2 5.556 2.779 1.999 0.0478 *
nbhd3 36.770 2.958 12.430 < 2e-16 ***
size 46.109 5.527 8.342 1.25e-13 ***
brickYes 19.152 2.438 7.855 1.69e-12 ***
---
Residual standard error: 12.5 on 123 degrees of freedom
Multiple R-squared: 0.7903,Adjusted R-squared: 0.7834
F-statistic: 115.9 on 4 and 123 DF, p-value: < 2.2e-16
If the linear model is correct, we are monotone up in all three
variables.
Remark: For the linear model we have to dummy up nbhd, but for
BART and mBART we can simply leave it as an ordered numerical
categorical variable.
37 / 45

38. Just using x = size, y = price appears to be marginally increasing
in size. (ˆfdown ≈ 0).
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
1.6 1.8 2.0 2.2 2.4 2.6
80100120140160180200
x
y
BART, mBART, and mBARTD
BART
mBART
mBARTD
linear
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
1.6 1.8 2.0 2.2 2.4 2.6
80100120140160180200
x
y
mBARTD: y on (x up, x down)
mBARTD: monotone up fit
mBARTD: monotone down fit
mBARTD
linear
mBART and mBARTD seem much better than BART.
38 / 45

39. Using x = (nbhd, size, brick), here are the relationships between
the fitted values from various models.
y
100 120 140 160 180
q qq
q
q
q
qq
q
q
q
q
q
q
q
q q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q qq
q
q
q
qq
q
q
q
q
q
q
q
q q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
100 120 140 160 180
q qq
q
q
q
qq
q
q
q
q
q
q
q
q q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q qq
q
q
q
qq
q
q
q
q
q
q
q
q q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
85 90 95
80120160200
qq q
q
q
q
q q
q
q
q
q
q
q
q
qq
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
100140180
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
bart
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q q
q q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
qq
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
mbart
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
qq
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
qq
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
100140180
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
100120140160180
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
mbartD
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
fup
100140180
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
80 120 160 200
859095
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
100 120 140 160 180
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
100 120 140 160 180 200
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
fdown
Note the high correlation between mBART, mBARTD and ˆfup.
39 / 45

40. x axis: mBARTD = ˆfup + ˆfdown.
y axis: red: ˆfup, green: ˆfdown.
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
80 100 120 140 160 180 200
80100120140160180200
yhat−mbartd
fupandfdown
q fup
fdown
mBARTD ≈ ˆfup suggests f is multivariate monotonic !!!
40 / 45

41. Let’s now look at the effect of size conditionally on the six possible
values of (nbdh, brick)
1.7 1.8 1.9 2.0 2.1 2.2 2.3
100120140160180
size
price
N
N
N
N N
N
N
N
N
N
N
N
N
N
N
N
N
N
N N
N
N
N
N
N
N
N
N
N
N
N
N
N
N N
N
N
N
N
N
N
N
N
N
N
B
B
B
B B
B
B B
B
B
B
B
B
B
B
B
B
B
B B
B
B
B
B
B
B
B
B
B
B
B
B
B
B B
B
B
B
B
B
B
B
B
B
B
BART: conditional effect of size
NBHD1
NBHD2
NBHD3
1.7 1.8 1.9 2.0 2.1 2.2 2.3
100120140160180
size
price
N
N
N
N N N N N
N
N N N
N
N
N
N
N
N
N N N N N
N
N N N
N
N
NN
N
N
N N N N N
N
N N N
N
N
N
B
B
B
B B B B B
B
B B B
B
B
B
B
B
B
B B B B B
B
B B B
B
B
B
B
B
B
B B B B B
B
B B B
B
B
B
mBART: conditional effect of size
1.7 1.8 1.9 2.0 2.1 2.2 2.3
100120140160180
size
price
N
N N
N N N N N
N
N N N
N
N
N
N
N N
N N N N N
N
N N N
N
N
N
N
N N
N N N N N
N
N N N
N
N
N
B
B B
B B B B B
B
B B B
B
B
B
B
B B
B B B B B
B
B B B
B
B
B
B
B B
B B B B B
B
B B B
B
B
B
mBARTD: conditional effect of size
The conditionally monotone effect of size is becoming clearer!
41 / 45

42. And finally, the effect of size conditionally on the six possible
values of (nbdh, brick) via ˆfup and ˆfdown
1.7 1.8 1.9 2.0 2.1 2.2 2.3
80100120140160180200
size
price
N
N N N N N N N
N
N N N N
N N
N
N N N N N N N
N
N N N N
N NN
N N N N N N N
N
N N N N
N N
B
B B B B B B B
B
B B B B
B B
B
B B B B B B B
B
B B B B
B BB
B B B B B B B
B
B B B B
B B
mBART, fup: conditional effect of size
1.7 1.8 1.9 2.0 2.1 2.2 2.3
80100120140160180200
size
price
N N N N N N N N N N N N N N NN N N N N N N N N N N N N N NN N N N N N N N N N N N N N NB B B B B B B B B B B B B B BB B B B B B B B B B B B B B BB B B B B B B B B B B B B B B
mBARTD, fdown: conditional effect of size
Price is clearly conditionally monotone up in all three variables!
By simultaneously estimating ˆfup + ˆfdown, we have discovered
monotonicity without any imposed assumptions!!!
42 / 45

43. Concluding Remarks
mBARTD = ˆfup + ˆfdown provides a monotone assumption free
approach for the discovery of the monotone components of f
in multidimensional settings.
Discovering such regions of monotonicity may of scientific
interest in real applications.
We have used informal variable selection to identify the
monotone components here. More formal variable selection
can be used in higher dimensional settings.
As a doubly adaptive shape-constrained regularization
approach,
mBARTD will adapt to mBART when monotonicity is present,
mBARTD will adapt to BART when monotonicity absent,
mBARTD seems at least as good and maybe better, than the
best of mBART and BART in general.
43 / 45

44. Concluding Remarks
The fully Bayesian nature of BART greatly facilitates
extensions such as mBART, mBARTD and many others.
Despite its many compelling successes in practice, theoretical
frequentist support for BART only now beginning to appear.
For example, Rockova and van der Pas (2017) Posterior
Concentration for Bayesian Regression Trees and Their
Ensembles recently laid out a framework for the first
theoretical results for Bayesian CART and BART, showing
near-minimax posterior concentration when p > n for classes
of Holder continuous functions.
Monotone BART paper is available on Arxiv. Software for
mBART is available at https://bitbucket.org/remcc/mbart.
44 / 45

45. Thank You!
45 / 45