Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimization

1. SigOpt. Confidential. Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimization KDD 2019 2nd Annual Workshop Online and Offline Evaluation of Interactive Systems Michael McCourt, Research Engineer, SigOpt

2. SigOpt. Conﬁdential. About me ● Research engineering at SigOpt ● Focus on applied Bayesian optimization ● PhD from Cornell ● Avid Cleveland Cavaliers fan About SigOpt ● Leading software solution for parameter optimization and model experimentation ● Customers in ﬁnance, trading, media, technology, consulting, energy, industry ● Free version of our solution for academia available at sigopt.com/edu

3. SigOpt. Confidential. Abstract for KDD 2019 2nd Annual Workshop on Offline and Online Evaluation of Interactive Systems Many real world applications (ML models, simulators, etc.) have multiple competing metrics that define performance; these require practitioners to carefully consider potential tradeoffs. However, assessing and ranking this tradeoff is nontrivial, especially when the number of metrics is more than two. Often times, practitioners scalarize the metrics into a single objective, e.g., using a weighted sum. In this talk, we pose this problem as a constrained multi-objective optimization problem. By setting and updating the constraints, we can efficiently explore only the region of the Pareto efficient frontier of the model/system of most interest. We motivate this problem with the application of an experimental design setting, where we are trying to fabricate high performance glass substrate for solar cell panels. 3

4. SigOpt. Confidential. Most Metrics are Impacted by Free Parameters How can these free parameters be chosen? Generally, these are chosen to yield good future performance. • This discussion only covers offline metrics. • Some of the elements apply in an online setting as well. Given a computable metric defining future performance, a search can be conducted for the free parameters yielding acceptable/optimal performance. • In many circumstances, evaluating this performance metric is costly. • Example: Train a classification model and evaluate a validation accuracy. • Example: Use financial data from the past year for a trading strategy and evaluate its profit on last month’s data. 4

5. SigOpt. Confidential. Searching for Free Parameters Requires Efficiency Intelligently searching a fixed domain Many searches benefit from efficiently (actively) learning about the circumstances of the search. • Active learning -- “Active learning is closely related to experimental design … is most often adaptive … employs an oracle for data labelling … is usually used to learn a model for classification.” -- [Brochu et al 2010] Two adjacent fields of research have evolved. • Bayesian optimization -- “Bayesian optimization is a sequential model-based approach to [optimizing a function].” -- [Shahriari et al, 2016] • Active search -- “Active search is an active learning setting with the goal of identifying as many members of a given class as possible under a labeling budget.” -- [Jiang et al, 2017] How we conduct this active learning will greatly impact efficiency of the search. 5

6. SigOpt. Conﬁdential. Bayesian Optimization A graphical depiction of the iterative process 6 Build a statistical model Build a statistical model Choose a next point Choose a next point

7. SigOpt. Confidential. Bayesian Optimization Efficiently Optimize a Scalar Function To quote [Frazier 2018]: Bayesian optimization (BayesOpt/BO) is a class of machine-learning-based optimization methods focused on [maximizing/minimizing a function with] the following properties ... • Typically the dimension d is less than 20. • The objective function f is continuous, as is the domain (which is likely a d-dimensional rectangle). • f is expensive to evaluate: e.g., time, money, access • f is black-box: if lacks known special structure like concavity or linearity. • When we evaluate f, we observe only f(x); that is, the optimization is gradient-free. • f is often observed in the presence of noise. • Our focus is on finding a global rather than local optimum. 7

8. SigOpt. Confidential. Many Metrics may Contribute to Success How do we execute under these ambiguous circumstances? Defining/measuring future performance is imprecise. • Generally, several metrics will contribute to a sense of future performance. • Not all metrics are equally important -- some may only need to reach a threshold. • The feasible performance and preferred interaction between metrics may not be known a priori. Popular multiobjective optimization strategies are often population-based (and not often sample-efficient). 8

9. SigOpt. Confidential. Multiobjective Bayesian Optimization Balancing competing metrics to find the Pareto frontier Adaptations to BO to search for the efficient frontier: • Change the problem to an active search problem [Jiang et al, 2018]. • Search for diverse points near the efficient frontier. • Scalarize the problem with linear combinations of the metrics [Knowles, 2006]. • Define a hypervolume based acquisition function [Hernandez-Lobato et al, 2016, Emmerich et al, 2016]. • Scalarization through prior beliefs [Astudillo, 2017]. Guiding points • Users wanted to interactively update the search process. • Users felt uncomfortable stating a priori preferences. 9

10. SigOpt. Confidential. Multiobjective Bayesian Optimization Our strategy We apply a strategy similar to what was discussed in [Letham et al, 2019]. 1. Model all metrics independently. • Requires no prior beliefs on how metrics interact. • Missing data removed on a per metric basis if unrecorded. 2. Expose the efficient frontier through constrained scalar optimization. • Enforce user constraints when given. • Iterate through sub constraints to better resolve efficient frontier, if desired. • Consider different regions of the frontier when parallelism is possible. 3. Allow users to change constraints as the search progresses. • Allow the problems/goals to evolve as the user’s understanding evolves. 10 Variation on Expected Improvement

11. SigOpt. Conﬁdential. One strategy can be to randomly apply constraints. Multiobjective Bayesian Optimization Our strategy 11 Note: There are GIFs that do not show up in this version of the presentation. For a copy that includes them, please email contact@sigopt.com

12. SigOpt. Conﬁdential. Another strategy can be to “walk” up and down the constraint domain. Multiobjective Bayesian Optimization Our strategy 12 Note: There are GIFs that do not show up in this version of the presentation. For a copy that includes them, please email contact@sigopt.com

13. SigOpt. Conﬁdential. It can help to alternate which metric the constraint is imposed on. Multiobjective Bayesian Optimization Our strategy 13 Note: There are GIFs that do not show up in this version of the presentation. For a copy that includes them, please email contact@sigopt.com

14. SigOpt. Conﬁdential. Users can enforce their own bounds to focus on the desired outcome. Multiobjective Bayesian Optimization Our strategy 14 Note: There are GIFs that do not show up in this version of the presentation. For a copy that includes them, please email contact@sigopt.com

15. SigOpt. Conﬁdential. Users can also update their own bounds as the experiment goes on. Multiobjective Bayesian Optimization Our strategy 15 Note: There are GIFs that do not show up in this version of the presentation. For a copy that includes them, please email contact@sigopt.com

16. SigOpt. Conﬁdential. Awesome Applications of Bayesian Optimization Who is using, and can use, BO? 16 ● ML/DL hyperparameter tuning [Snoek et al, 2012; Feurer et al, 2015; Kandasamy et al, 2018] ● Engineering system design [Mockus, 1989; Jones et al, 1998; Forrester et al, 2008] ● Drug design [Negoescu et al, 2011; Frazier and Wang, 2016] ● Material design [Packwood, 2017; Haghanifar et al, 2019] ● Model calibration [Shoemaker et al, 2007; Shi et al, 2013; Letham et al. 2019] ● Reinforcement learning [Lizotte, 2008; Brochu et al, 2010; Martinez-Cantin et al, 2018] There are so many others!

17. SigOpt. Conﬁdential.17 A Joint Collaboration with University of Pittsburgh [Haghanifar et al, 2019] Metrics • Light transmission • Clarity (low haze) • Water resistance Constraints updated on all metrics during the search. Note: There is a video that does not show up in this version of the presentation. For a copy that includes this, please email contact@sigopt.com

18. SigOpt. Conﬁdential. Future Work How can we improve this process? When black-box constraints exist, how can we encourage our search to respect them? • Hallucinate bad function values at points which violate the constraints. • Attenuate the expected improvement by the probability of failure [Gelbart, 2015]. • Model the constraints and average out the noisy behavior [Letham et al, 2019]. • Model the Lagrangian [Picheny et al, 2016]. Question: Exactly how black-box/expensive are these constraints (or the objective)? • We can adapt to expensive constraints but a cheap objective [Gramacy et al, 2106]. Question: Can we help focus on the important region using preferences? • Joint work extending [Astudillo, 2017] with Raul and Peter. 18

19. SigOpt. Conﬁdential. Thank you for having me!