Voronoi optimistic optimization (VOO) is extended to Monte Carlo tree search (MCTS) to solve problems with continuous and high-dimensional action spaces. VOO implicitly constructs Voronoi cells around sampled points to balance exploration and exploitation without selecting partition dimensions. This Voronoi optimistic optimization applied to trees (VOOT) method outperforms Bayesian optimization and hierarchical partitioning methods on benchmark functions. The paper also proves a regret bound for VOOT, showing the number of simulations needed to guarantee a given level of performance. VOOT is tested on a packing problem with continuous box placements, outperforming state-of-the-art reinforcement learning and continuous-action MCTS algorithms.

1. Monte Carlo Tree Search in continuous spaces
using Voronoi optimistic optimization
with regret bounds
Beomjoon Kim
(with Kyungjae Lee, Sungbin Lim, Leslie Pack Kaelbling, Tomas Lozano-Perez)

2. Monte Carlo Tree Search in discrete action
spaces

5. Characteristics of robotics problems

6. Characteristics of robotics problems
Continuous and
high-dimensional
action spaces

7. Main question addressed in our paper
How can we extend MCTS to
high-dimensional continuous
action space problems?

8. First, consider the easier problem
• Assume planning horizon is 1
The problem is reduced to a budgeted-black-box-
function optimization (BBFO) problem

9. First, consider the easier problem
• Assume planning horizon is 1
• The problem is reduced to a budgeted-black-
box-function optimization (BBFO) problem

10. First, consider the easier problem
• Assume planning horizon is 1
• The problem is reduced to a budgeted-black-
box-function optimization (BBFO) problem

11. First, consider the easier problem
• Assume planning horizon is 1
• The problem is reduced to a budgeted-black-
box-function optimization (BBFO) problem
within n number of evaluations

12. BBFO: general problem formulation

13. BBFO: general problem formulation
• Given:
• Budget
• Bounded search space
• Objective function

14. BBFO problem formulation
• Goal: minimize the simple regret

15. A class of BBFO methods
Bayesian
Optimization
(BO)
EI (Mockus, 1974)
PI (Kushner, 1964)
GP-UCB(Srinivas et al., 2010)
ES (Hennig & Schuler, 2012)
PES (Hernandez-Lobato et al., 2014)
EST (Wang et al., 2016)

16. Drawback in high-dimensional search spaces
Bayesian
Optimization
(BO)
Hierarchical
Partitioning
(HP)
• EI (Mockus, 1974)
• PI (Kushner, 1964)
• GP-UCB( Auer, 2002; Srinivas et
al., 2010)
• ES (Hennig & Schuler, 2012)
• PES (Hernandez-Lobato et al.,
2014)
• EST (Wang et al., 2016)
Requires global
optimization of the
acquisition function

17. Another class of BBFO methods
Bayesian
Optimization
(BO)
Hierarchical
Partitioning
(HP)
• EI (Mockus, 1974)
• PI (Kushner, 1964)
• GP-UCB( Auer, 2002; Srinivas et
al., 2010)
• ES (Hennig & Schuler, 2012)
• PES (Hernandez-Lobato et al.,
2014)
• EST (Wang et al., 2016)
Requires global
optimization of the
acquisition function
Hierarchical
Partitioning
(HP)
DOO(Munos, 2011)
SOO(Valko et al., 2013)
HOO(Bubeck et al.,
2010)

18. How hierarchical partitioning methods work
• Consecutively partitioning the search space

19. How hierarchical partitioning methods work
• Consecutively partitioning the search space

20. • First evaluated point
How hierarchical partitioning methods work

21. • Partition space by constructing K cells. Let’s say K =
3
How hierarchical partitioning methods work

22. • Evaluate points in the new cells
How hierarchical partitioning methods work

23. • Choose the most promising cell
How hierarchical partitioning methods work

24. • Construct a new partition, and evaluate points
How hierarchical partitioning methods work

25. How to select the most promising cell?

26. Selecting the most promising cell
• Deterministic Optimistic Optimization [Munos 2011]
Compute b-values at each cell, where

27. Selecting the most promising cell
• Deterministic Optimistic Optimization [Munos 2011]
• Compute the b-value of each cell, where

28. Selecting the most promising cell
• Deterministic Optimistic Optimization [Munos 2011]
• Compute the b-value of each cell, where

29. Selecting the most promising cell
• Deterministic Optimistic Optimization [Munos 2011]
• Compute the b-value of each cell, where
Exploration bonus
Prefers larger cells

30. Diameter of a cell

31. Diameter of a cell

32. Diameter of a cell

33. DOO: Choose the cell with the highest b-value

34. DOO: exploration vs. exploitation

35. DOO: exploration vs. exploitation
Exploitation of current function knowledge
prefer cells that have high function values

36. DOO: exploration vs. exploitation
Exploration of the search
space
prefer cells with large diameters
Exploitation of current function knowledge
prefer cells that have high function values

37. BUT: it requires deciding which dimension to cut

38. BUT: it requires deciding which dimension to cut

39. BUT: it requires deciding which dimension to cut

40. BUT: it requires deciding which dimension to cut
VS.

41. BUT: it requires deciding which dimension to cut
In -dimensional search space, you
have number of choices

42. BUT: it requires deciding which dimension to cut
Finding the
optimal sequence of dimensions
to cut is not trivial
In -dimensional search space, you
have number of choices

43. DOO’s intuitive idea
• DOO idea:
• The larger the cell, the more the exploration
bonus
• The larger the value of the cell, the more the
exploitation bonus

44. Our main question
DOO idea:
The larger the cell, the more the exploration bonus
The larger the value of the cell, the more the
exploitation bonus
How can we use this idea
without cell constructions?

45. Our method: Implicitly construct cells

46. Our method: Implicitly construct Voronoi cells

47. Our method: Implicitly construct Voronoi cells
Voronoi cell:
A set of points closer to its
generator than all the other
generators

48. Our method: Implicitly construct Voronoi cells
Example
generator
Voronoi cell:
A set of points closer to its
generator than all the other
generators

49. Our method: Implicitly construct Voronoi cells
Example
generator
Voronoi cell:
A set of points closer to its
generator than all the other
generators

50. Our method: Implicitly construct Voronoi cells
Example
generator
Voronoi cell:
A set of points closer to its
generator than all the other
generators
Evaluated points
represent generators

51. Exploration
How do you prefer the larger cells
without constructing Voronoi cells?

52. Exploration
Use Voronoi bias
How do you prefer the larger cells
without constructing Voronoi
cells?

53. Exploration: Don't construct Vcells, use Voronoi bias

54. Exploration: Don't construct Vcells, use Voronoi bias
Volume of a cell
Volume of the search space

55. Exploration: Don't construct Vcells, use Voronoi bias

56. Exploration: Don't construct Vcells, use Voronoi bias

57. Exploration: Don't construct Vcells, use Voronoi bias

58. Exploration: Don't construct Vcells, use Voronoi bias

59. Exploration: Don't construct Vcells, use Voronoi bias

60. Exploitation
How do you select the cell with the highest value
without constructing Voronoi cells?

61. Exploitation
Use rejection sampling
How do you select the cell with the highest value
without constructing Voronoi cells?

62. Use rejection sampling to do exploitation
Assume best point found
so far

63. Use rejection sampling to do exploitation
Assume best point found
so far

64. Use rejection sampling to do exploitation
Assume best point found
so far
Randomly Sample:

65. Use rejection sampling to do exploitation
Reject until the sampled
point is closest to the
best point
Assume best point found
so far
Randomly Sample:

66. VOO: Voronoi optimistic optimization

67. VOO: Voronoi optimistic optimization
Search
space

68. VOO: Voronoi optimistic optimization
Search
space Exploration
parameter

69. VOO: Voronoi optimistic optimization
Search
space Exploration
parameter
Distance
metric

70. VOO: Voronoi optimistic optimization
Search
space Exploration
parameter
Distance
metric
Budget

71. VOO: Voronoi optimistic optimization
With probability

72. With probability
VOO: Voronoi optimistic optimization
Exploration

73. VOO: Voronoi optimistic optimization
With probability
With probability
Exploration

74. VOO: Voronoi optimistic optimization
With probability
With probability
Exploration
Exploitation

75. VOO: Voronoi optimistic optimization
Evaluate
With probability
With probability
Exploration
Exploitation

76. Optimizing Rastrigin function

77. Bestvaluefoundsofar
Number of evaluations
Optimizing 10 dimensional Rastrigin function

78. 10D – Bayesian optimization methods
Bestvaluefoundsofar
Number of evaluations
REMBO
BaMSOO
GPUCB

79. 10D – Hierarchical partitioning methods
Bestvaluefoundsofar
Number of evaluations
SOO
DOO

80. 10D – Evolutionary algorithm
Bestvaluefoundsofar
Number of evaluations
CMA-ES

81. 10D – Voronoi optimistic optimization
Bestvaluefoundsofar
Number of evaluations
VOO

82. 20D – even more drastic difference
Bestvaluefoundsofar
Number of evaluations
VOO

83. From BBFO to MCTS
• So that was when planning horizon is 1
• What about planning horizon >= 1, for
deterministic environments?

84. Assign a VOO agent at each node
Solve for the optimal action with respect to the
estimated Q-function
But…we have access to , not the true
VOO applied to Trees (VOOT): high-level
idea

85. VOO applied to Trees (VOOT): high-level
idea
Assign a VOO agent at each node
Solve the action optimization problem using the
estimated Q-function
But…we have access to , not the true

86. Our main theoretical result
How many simulations do we need to guarantee

87. Our main theoretical result
How many simulations do we need to guarantee
Under mild assumptions,

88. Packing problem

89. Packing problem
• Actions: placements for three boxes (9
dim)
Reward function:
1/d if feasible,
r_min otherwise
Planning horizon: 20

90. Packing problem
• Actions: placements for three boxes (9
dim)
• Reward function:
• 1/d if feasible,
• r_min otherwise
Planning horizon: 20

91. Packing problem
• Actions: placements for three boxes (9
dim)
• Reward function:
• 1/d if feasible,
• r_min otherwise
• Total boxes to pack: 20

92. State-of-the-art RL algorithms
PPO
DDPG

93. State-of-the-art continuous-action MCTS
PW-UCT
DOOT

94. VOO applied to Trees
VOOT

95. Takeaways
Voronoi Optimistic Optimization applied to Trees
(VOOT) for deterministic environment
• Empirical contribution: improvement over the
state-of-the-art, especially in high-dimensional
action-spaces
• Theoretical contribution: regret-bound for
VOOT