This talk gives an overview of the research work done in working group IV. It will briefly introduce some of the problems in Deterministic Bayesian computation, big data reduction, estimation of normalizing constants in Bayesian sampling, and stochastic programming. This talk will briefly present two ongoing works in WG4. In the first part of the talk, we propose a new active data reduction method for large-scale Gaussian process (GP) modeling. GP modeling with big data is well-known to be computationally demanding, since it requires O(N^3) work and O(N^2) memory (N>>1 is the number of data points). The goal here is to first reduce this big data to a smaller dataset of n<<N points, then use the smaller data for efficient GP modeling. Our reduction method is guided by a useful bound on GP prediction error, which provides an exploration-exploitation trade-off for sequentially selecting the reduced data. We demonstrate the effectiveness of this approach in simulations and a climate model application. In the second part of this talk, we present a new adaptive modeling framework for estimating normalizing constants of an (unnormalized) posterior density, which we assume is expensive to evaluate. This method first uses posterior samples to fit an approximating surface, then employs this fit to obtain a closed-form estimate for the normalizing constant. A key novelty is the use of semi-definite (convex) programming, which allows for efficient and adaptive estimation with a small number of posterior samples. We explore the effectiveness of our approach in simulations and a real-world application.

1. Overview of Research in WG IV:
Representative Points for Small-Data and Big-Data Problems
V. Roshan Joseph and Simon Mak
1
Supported by NSF DMS 1712642

2. Big-Data Problems
• Reduce big data to reduce future
computational cost
2
Big Data Representative points

3. Small-Data Problems
• Obtain expensive small data with minimum
cost
3
Expensive data
generating mechanism
Representative points

4. Big Data: An Example
4
Kernel Ridge Regression
(Inputs)
90 song features
(Output)
Song release date
Loudness Pitch Timbre
2007
E.g.,
(Data)
N=515345 songs
Computation: 𝑂(𝑁3)
Storage: 𝑂(𝑁2)
https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD
• Mak and Joseph (2017)

5. How to Reduce Big Data?
• Stratified sampling (Dalenius 1950; Cox
1957)
• Principal points (Flury 1990)
• Quantizers (Lloyd 1957; Max 1960)
• MSE-rep points (Fang and Wang 1994)
– K-means Clustering
– Can’t produce a reduced point set that retains
the original distribution -> Not a “true”
representative point set!
5

6. Support Points
6
• Can be efficiently optimized using difference-of-convex
program.
• Mak and Joseph (2018)

7. Support Points-continued
7

8. Reducing Big Data
8

9. Example: N2(0,1)
9

10. Small Data: An Example
10
Physical Experiment
FEM Experiment
• Friction drilling (Miller and Shih 2007)

11. Model Calibration
11

12. Bayesian Model
where
• One evaluation of the unnormalized posterior takes
about 15.4 seconds in a 3.2 GHz laptop=> 10,000
MCMC samples would take 43 hours!
12

13. Minimum Energy Design
• Joseph, Dasgupta, Tuo, and Wu (2015)
• Posterior density
• Normalizing constant C is not needed!
13
𝑓 𝑥 =
1
𝐶
ℎ(𝑥)

14. 14

15. 15
• #evaluations=654 (Joseph, Wang, Gu, Lv, and Tuo 2017)

16. 16

17. Marginal Distribution
17

18. MED+MCMC
• Approximate the log-unnormalized posterior using
Gaussian Process and use MCMC
18

19. Support Points
min
2
𝑛𝐶
𝑖=1
𝑛
𝑥𝑖 − 𝑥 ℎ 𝑥 𝑑𝑥 −
1
𝑛2
𝑖=1
𝑛
𝑗=1
𝑛
𝑥𝑖 − 𝑥𝑗
• Normalizing constant C doesn’t factor out!
19

20. Research Questions
• Can we do fast Gaussian Process
approximation with big data?
• Can we adaptively estimate the
normalizing constant?
– Simon’s talk!
20

21. Thanks
21
• Lulu Kang
• Lester Mackey
• Fred Hickernell
• Mac Hyman
• Scott Schmilder
• Joe Marion
• Raaz Dwivedi
• Kan Zhang
• Cheng Cheng
• Matthias Sachs

22. References
Support points
• Mak, S. and Joseph, V. R. (2018). “Support Points,” Annals of Statistics, to appear,
https://arxiv.org/abs/1609.01811.
• Mak, S. and Joseph, V. R. (2017) “Projected Support Points: A New Method for High-
Dimensional Data Reduction”. Under review, https://arxiv.org/abs/1708.06897.
Minimum energy designs
• Joseph, V. R., Dasgupta, T., Tuo, R., and Wu, C. F. J. (2015). “Sequential
Exploration of Complex Surfaces Using Minimum Energy Designs”. Technometrics,
57, 64-74.
• Joseph, V. R., Wang, D., Gu, L., Lv, S., and Tuo, R. (2017) “Deterministic Sampling
of Expensive Posteriors Using Minimum Energy Designs”.
https://arxiv.org/abs/1712.08929
22