機械学習と機械発見：自然科学研究におけるデータ利活用の再考

ichigaku.takigawa@riken.jp
https://itakigawa.github.io/
IBISML 2022.12.22 @

( )
( )
/ (7 )
(7 )
JST (3.5 )
( )
⾒ (4 )
(4 ) ( )
( )
AIP ATR ( )

MaterialsInformatics
(Bioinformatics)
Materials Informatics
( )

.
. o -the-shelf
.
PDF https://itakigawa.page.link/IBISML taki

( )
<latexit sha1_base64="rREik5SVI+19We5hP0uVzuGffGw=">AAAB/nicbVDJSgNBEK2JW4xb1KOXxiCIhzAjbsegF48RzALJGHo6PUmT7p6hu0cMw4Df4FXP3sSrv+LRP7GzHEzig4LHe1VU1QtizrRx3W8nt7S8srqWXy9sbG5t7xR39+o6ShShNRLxSDUDrClnktYMM5w2Y0WxCDhtBIObkd94pEqzSN6bYUx9gXuShYxgY6VmOxDpU/Zw0imW3LI7Blok3pSUYIpqp/jT7kYkEVQawrHWLc+NjZ9iZRjhNCu0E01jTAa4R1uWSiyo9tPxvRk6skoXhZGyJQ0aq38nUiy0HorAdgps+nreG4n/ea3EhFd+ymScGCrJZFGYcGQiNHoedZmixPChJZgoZm9FpI8VJsZGNLMlEJnNxJtPYJHUT8veRfn87qxUuZ6mk4cDOIRj8OASKnALVagBAQ4v8ApvzrPz7nw4n5PWnDOd2YcZOF+/0HGWjQ==</latexit>
x⇤ <latexit sha1_base64="ggumTYNxk72HcgOPYsD+vGiQOG0=">AAAB/HicbVC7TsNAEFzzDOEVoKQ5ESFRRTbiVUbQUAaJPKTEis6Xc3Lk7mzdnRGRFb6BFmo6RMu/UPInnBMXJGGklUYzu9rdCWLOtHHdb2dpeWV1bb2wUdzc2t7ZLe3tN3SUKELrJOKRagVYU84krRtmOG3FimIRcNoMhjeZ33ykSrNI3ptRTH2B+5KFjGBjpUYnEOnTuFsquxV3ArRIvJyUIUetW/rp9CKSCCoN4VjrtufGxk+xMoxwOi52Ek1jTIa4T9uWSiyo9tPJtWN0bJUeCiNlSxo0Uf9OpFhoPRKB7RTYDPS8l4n/ee3EhFd+ymScGCrJdFGYcGQilL2OekxRYvjIEkwUs7ciMsAKE2MDmtkSiCwTbz6BRdI4rXgXlfO7s3L1Ok+nAIdwBCfgwSVU4RZqUAcCD/ACr/DmPDvvzofzOW1dcvKZA5iB8/ULqrqV8Q==</latexit>
x
<latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit>
x
<latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit>
y
<latexit sha1_base64="TEyxI6Ffe+b+OEFRulxYJ0sUB8w=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBGBdk8IVxRXTjkoc8EkTS1gEbStu0hQSJP2DiVhauNHFh/AA/wI0/4IJPMC4xcePCS2lilIi3mc6ZM/fcOTNXMlTFshnreISR0bHxCe+kb2p6ZtYfmJvPWnrdlHlG1lXdzEuixVVF4xlbsVWeN0wu1iSV56Tqfm8/1+Cmpejaod00eLEmVjSlrMiiTVS6ebxWCoRYhDkRHARRF4TgRkIPPOIIJ9Aho44aODTYhFWIsOgrIAoGg7giWsSZhBRnn+McPtLWKYtThkhslf4VWhVcVqN1r6blqGU6RaVhkjKIMHth96zLntkDe2Wff9ZqOTV6Xpo0S30tN0r+i8X0x7+qGs02Tr9VQz3bKGPH8aqQd8NhereQ+/rGWbub3k2FWyvslr2R/xvWYU90A63xLt8leep6iB+JvNCLUYOiv9sxCLLrkeh2ZCO5GYrvua3yYgnLWKV+xBDHARLIUP0KLnGFtuAVIsKWEOunCh5Xs4AfIcS/AEWBkJQ=</latexit>
y⇤
x⇤

( )
x
x
y
y⇤
<latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit>
?
?
?
?
?
x⇤

( )
x
y
y⇤
x⇤
x
<latexit sha1_base64="YCdrxHBq8hWthguuj4LwkMwyoug=">AAAB93icdVDLSsNAFJ3UV62vqks3g0VwFZKatM2u6MZlC9YW2lAm00k7dCYJMxMhhH6BW127E7d+jkv/xOlDsKIHLhzOuZd77wkSRqWyrA+jsLG5tb1T3C3t7R8cHpWPT+5lnApMOjhmsegFSBJGI9JRVDHSSwRBPGCkG0xv5n73gQhJ4+hOZQnxORpHNKQYKS21w2G5YpmW6zn1KrRM17I950oTz2s4NRfaprVABazQGpY/B6MYp5xECjMkZd+2EuXnSCiKGZmVBqkkCcJTNCZ9TSPEifTzxaEzeKGVEQxjoStScKH+nMgRlzLjge7kSE3kb28u/uX1UxU2/JxGSapIhJeLwpRBFcP513BEBcGKZZogLKi+FeIJEggrnc3aloDPdCbfj8P/yX3VtGum23YqzetVOkVwBs7BJbBBHTTBLWiBDsCAgEfwBJ6NzHgxXo23ZWvBWM2cgjUY71/5QpPk</latexit>
f
<latexit sha1_base64="DcbiaPwQEMQnrQxSiqXKigyS2IY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6Agw4z2YReC6MZlBdsKM0PJpJk2mMwMSUYsYb7Gj/Ab3OrKhQtB3PoLpo+FWg8EDuecm3s5YcqoVI7zas3Mzs0vLC4tF1ZW19Y3iptbLZlkApMmTlgibkIkCaMxaSqqGLlJBUE8ZKQd3l4M/fYdEZIm8bUapCTgqBfTiGKkjNQpnvp9pPQgh6dQ+6PvPNELA+3YTqVerh0dOnbFcevlY0Pq9ZNytZLDKN/3Q67v84NOsWSCI8Bp4k5ICUzQ6BTf/W6CM05ihRmS0nOdVAUaCUUxI3nBzyRJEb5FPeIZGiNOZKBHd+VwzyhdGCXCvFjBkfpzQiMu5YCHJsmR6su/3lD8z/MyFZ0EmsZppkiMx4uijEGVwGFnsEsFwYoNDEFYUHMrxH0kEFam2V9bQp6bTty/DUyT1pHtVu3KVbl0dj5pZwnsgF2wD1xQA2fgEjRAE2DwAJ7AM3ixHq0368P6HEdnrMnMNvgF6+sbdw6oMA==</latexit>
ŷ = f(x)

<latexit sha1_base64="ou7gs1Bt14yFxh1zGfB0tITvvDk=">AAACpXicbVFda9swFJW9ry77aNY+7kUsjKVQgp04H34YlJaNvQw6aNKCbYKkyK6oJBtJLgvCP2k/aI/7J1OcPKzpLggdzr3n3KsrXHGmTRD89vwnT589f3HwsvPq9Zu3h913Rwtd1orQOSl5qW4w0pQzSeeGGU5vKkWRwJxe47uLTf76nirNSnll1hXNBCokyxlBxlHL7i+btiaJKnBmg0EwjqPp8DQYjIMwjkYOxPEsmowbmDfwM0yRKmAqmFzaHKZMOozMLUHcfm2azp7XNA4mYegs3DWKJg6Ew2A0i5qtCmP7xYlSzAqedFpKCUuVKlXTX5/CvJ9iYX82JydtjcpgZ9ntObc24GMQ7kAP7OJy2f2TrkpSCyoN4UjrJAwqk1mkDCOcuua1phUid6igiYMSCaoz276igR8ds4J5qdyRBrbsvwqLhNZrgV3lZni9n9uQ/8sltclnmWWyqg2VZNsorzk0Jdz8EVwxRYnhawcQUczNCsktUogY95MPumDRuJ2E+xt4DBbDQTgZjH9EvbPz3XYOwHvwAfRBCKbgDHwDl2AOiHfsxd65d+F/8r/7V/5iW+p7O80xeBD+8i9BGsel</latexit>
f = arg min
f2F
E
⇥
error(y, f(x))
⇤
( )
x
y
y⇤
x⇤
x
<latexit sha1_base64="YCdrxHBq8hWthguuj4LwkMwyoug=">AAAB93icdVDLSsNAFJ3UV62vqks3g0VwFZKatM2u6MZlC9YW2lAm00k7dCYJMxMhhH6BW127E7d+jkv/xOlDsKIHLhzOuZd77wkSRqWyrA+jsLG5tb1T3C3t7R8cHpWPT+5lnApMOjhmsegFSBJGI9JRVDHSSwRBPGCkG0xv5n73gQhJ4+hOZQnxORpHNKQYKS21w2G5YpmW6zn1KrRM17I950oTz2s4NRfaprVABazQGpY/B6MYp5xECjMkZd+2EuXnSCiKGZmVBqkkCcJTNCZ9TSPEifTzxaEzeKGVEQxjoStScKH+nMgRlzLjge7kSE3kb28u/uX1UxU2/JxGSapIhJeLwpRBFcP513BEBcGKZZogLKi+FeIJEggrnc3aloDPdCbfj8P/yX3VtGum23YqzetVOkVwBs7BJbBBHTTBLWiBDsCAgEfwBJ6NzHgxXo23ZWvBWM2cgjUY71/5QpPk</latexit>
f
<latexit sha1_base64="DcbiaPwQEMQnrQxSiqXKigyS2IY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6Agw4z2YReC6MZlBdsKM0PJpJk2mMwMSUYsYb7Gj/Ab3OrKhQtB3PoLpo+FWg8EDuecm3s5YcqoVI7zas3Mzs0vLC4tF1ZW19Y3iptbLZlkApMmTlgibkIkCaMxaSqqGLlJBUE8ZKQd3l4M/fYdEZIm8bUapCTgqBfTiGKkjNQpnvp9pPQgh6dQ+6PvPNELA+3YTqVerh0dOnbFcevlY0Pq9ZNytZLDKN/3Q67v84NOsWSCI8Bp4k5ICUzQ6BTf/W6CM05ihRmS0nOdVAUaCUUxI3nBzyRJEb5FPeIZGiNOZKBHd+VwzyhdGCXCvFjBkfpzQiMu5YCHJsmR6su/3lD8z/MyFZ0EmsZppkiMx4uijEGVwGFnsEsFwYoNDEFYUHMrxH0kEFam2V9bQp6bTty/DUyT1pHtVu3KVbl0dj5pZwnsgF2wD1xQA2fgEjRAE2DwAJ7AM3ixHq0368P6HEdnrMnMNvgF6+sbdw6oMA==</latexit>
ŷ = f(x)
<latexit sha1_base64="0fXeD1cakvAs7WfPkj1UglydrqI=">AAACPXicbVDNSgMxGMz6b/2revQSLEIVKbu1VXtQiiJ4VLAqdNeSTbNtaLK7JFlxCftEPoTP4FE9efAmXr2a1h60OhAYZr7J9zF+zKhUtv1kjY1PTE5Nz8zm5uYXFpfyyyuXMkoEJg0csUhc+0gSRkPSUFQxch0LgrjPyJXfO+77V7dESBqFFyqNicdRJ6QBxUgZqZU/cbtI6TSDB1C7g++aouN72i7Z1Vplr7xtl6q2U6vsGFKr7Vd2qxkMsqLrc32XbcJDmN7orayVL5jAAPAvcYakAIY4a+Vf3XaEE05ChRmSsunYsfI0EopiRrKcm0gSI9xDHdI0NEScSE8P7svghlHaMIiEeaGCA/VnQiMuZcp9M8mR6spRry/+5zUTFex7moZxokiIvxcFCYMqgv3uYJsKghVLDUFYUHMrxF0kEFam4V9bfN7vxBlt4C+5LJec3VL1vFKoHw3bmQFrYB0UgQP2QB2cgjPQABjcg0fwDF6sB+vNerc+vkfHrGFmFfyC9fkFIG+q9w==</latexit>
ŷ = f(x) > y⇤
<latexit sha1_base64="ggumTYNxk72HcgOPYsD+vGiQOG0=">AAAB/HicbVC7TsNAEFzzDOEVoKQ5ESFRRTbiVUbQUAaJPKTEis6Xc3Lk7mzdnRGRFb6BFmo6RMu/UPInnBMXJGGklUYzu9rdCWLOtHHdb2dpeWV1bb2wUdzc2t7ZLe3tN3SUKELrJOKRagVYU84krRtmOG3FimIRcNoMhjeZ33ykSrNI3ptRTH2B+5KFjGBjpUYnEOnTuFsquxV3ArRIvJyUIUetW/rp9CKSCCoN4VjrtufGxk+xMoxwOi52Ek1jTIa4T9uWSiyo9tPJtWN0bJUeCiNlSxo0Uf9OpFhoPRKB7RTYDPS8l4n/ee3EhFd+ymScGCrJdFGYcGQilL2OekxRYvjIEkwUs7ciMsAKE2MDmtkSiCwTbz6BRdI4rXgXlfO7s3L1Ok+nAIdwBCfgwSVU4RZqUAcCD/ACr/DmPDvvzofzOW1dcvKZA5iB8/ULqrqV8Q==</latexit>
x

(UncertaintyQuantification,UQ)
x
y
x⇤
x
y <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit>
y
<latexit sha1_base64="uHwbcp1I7cfAojOCGzVZ2+fOJ7M=">AAACh3ichVE7TsNAEH2Yb/glQINEExGBoAnjACGh4tNQQiCAFKLINhuwcGzLdiKFiAtQ0AaJCiQKxAE4AA0XoMgRECVINBSMnSBEEZjV7s6+mTf7dke1Dd31iBodUmdXd09vX6h/YHBoOBwZGd11rbKjiaxmGZazryquMHRTZD3dM8S+7QilpBpiTz1Z9+N7FeG4umXueFVb5EvKkakXdU3xfMieqc4WIjGKp1NJWkhEKU6USlCSnUWS03I6KjPiWwwt27QiDzjAISxoKKMEARMe+wYUuDxykEGwGcujxpjDnh7EBc7Qz9wyZwnOUBg94fWIT7kWavLZr+kGbI1vMXg6zIxiip7pjt7oie7phT7b1qoFNXwtVd7VJlfYhfD5+PbHv6wS7x6Of1h/avZQRCrQqrN2O0D8V2hNfuW0/ra9nJmqTdMNvbL+a2rQI7/ArLxrt1sic/WHHpW18I9xg767EG3v7CbicjI+v7UQW1lrtaoPE5jEDPdjCSvYwCayXP8YF6jjUgpJc1JSSjVTpY4WZwy/TFr9AnUXkR0=</latexit>
p(y)
<latexit sha1_base64="xAYAr5eatCRkt85fJM86k8bLQd0=">AAAChnichVE7TsNAEH2YXwi/AA0STUQEQhTWOCQhoYqgoSSEABI/2WYTLBzbsp1IIeICSLRQUIFEgTgAB6DhAhQcAVGCREPB2AlCFMCsdnf2zbzZtzuaYxqeT/TUIXV2dff0Rvqi/QODQ8OxkdF1z665uijptmm7m5rqCdOwRMk3fFNsOq5Qq5opNrTDpSC+UReuZ9jWmt9wxE5VrVhG2dBVn6FiY3d2L5YgOZfNUCoZJ5kom6QMO2lSckourjASWAJtW7Fjd9jGPmzoqKEKAQs++yZUeDy2oIDgMLaDJmMue0YYFzhGlLk1zhKcoTJ6yGuFT1tt1OJzUNML2TrfYvJ0mRnHFD3SDb3SA93SM338WqsZ1gi0NHjXWlzh7A2fjBff/2VVefdx8M36U7OPMrKhVoO1OyESvEJv8etH56/FhdWp5jRd0Qvrv6QnuucXWPU3/bogVi/+0KOxFv4xbtBXF+K/O+tJWcnIc4VUIr/YblUEE5jEDPdjHnksYwUlrl/BKc5wLkUkWUpL861UqaPNGcMPk/Kf1ZuQ2g==</latexit>
y⇤
<latexit sha1_base64="Cx8vtQI6wD6rFemnJtzP0VxR74M=">AAACp3ichVE9T9tQFD0Y2vJRSqALUheLiCphiK5TGhIkKlSWboSPhEh8RLbzQp9wbMt+ieRG7FX/QIdOrdQBIXWFnYU/wMBPqDqCxMLQaydVxQBc6/mdd9499533ruU7MlRElwPa4NCTp8+GR0bHno+/mEhNTlVDrx3YomJ7jhfULDMUjnRFRUnliJofCLNlOWLLOliJ97c6Igil526qyBe7LXPflU1pm4qpemqmnIneRXtz2aUd6ap6l+HhHsOminQ/E2X1RlRPpSlXKhZoPq9TjqiYpwKDt2SUjJJuMBNHGv0oe6kT7KABDzbaaEHAhWLswETI3zYMEHzmdtFlLmAkk32BQ4yyts1ZgjNMZg/4v8+r7T7r8jquGSZqm09xeASs1DFLF3REV3ROx/Sbbu+t1U1qxF4inq2eVvj1iS/TGzePqlo8K3z8r3rQs0ITxcSrZO9+wsS3sHv6zqevVxuL67Pd1/SD/rD/73RJZ3wDt3Nt/1wT698e8GOxF34xbtC/Luj3g2o+ZxRyb9bm08vv+60axivMIMP9WMAyPqCMCtf/jF84wamW1Va1qlbrpWoDfc1L3AnN/AuqB50Q</latexit>
P(y > y⇤
) =
Z 1
y⇤
p(y)dy
<latexit sha1_base64="svoh2wRjAJ9IgkLWLNY4QZ6NPX0=">AAACwnichVHLTttAFL24vMsjlA0Sm1EjKlpF0XWANEECIWilLnk0gBSHyHYmMOBX7UkkY/ID/AALVi3qouoH9AO6abetWPAJqEsqsWHRayeoYgFcazxnztxz58xcw7NEIBEvupQn3T29ff0Dg0+HhkdGU2PPNgO34Zu8ZLqW628besAt4fCSFNLi257Pdduw+JZxsBLvbzW5HwjXeS9Dj1dsfdcRdWHqkqhqakGzdblnGNHbVlnLhEzLHGkZFi6GO68qC5pwZDUi2NohWJchowSz5krmTYcvWS2sptKYLRbyOJtjmEUs5DBPYA7VolpkKjFxpKETq27qG2hQAxdMaIANHByQhC3QIaCvDCogeMRVICLOJySSfQ4tGCRtg7I4ZejEHtB/l1blDuvQOq4ZJGqTTrFo+KRkMIXn+AWv8Ad+xUu8ubdWlNSIvYQ0G20t96qjxxMb14+qbJol7P1XPehZQh0KiVdB3r2EiW9htvXNw5Orjfn1qegFfsI/5P8jXuB3uoHT/Gt+XuPrpw/4McgLvRg16LYL7H6wmcuq+ezM2mx6abnTqn6YhOcwTf14DUvwDlahRPXP4Cf8gt/KG2Vf+aAE7VSlq6MZhzuhHP0DArmnyA==</latexit>
E[ y | y > y⇤
] =
Z 1
y⇤
y · p(y)dy
Probability of
improvement (PI)
Expected
Improvement (EI)
Upper Conﬁdence
Bound (UCB)
(UQ)

( )
<latexit sha1_base64="mpuRLmJhhTRKlPt+QsNIZJRw5HU=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkN+IiVuxeDoeSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVLKQnA==</latexit>
x1
<latexit sha1_base64="C1ohMOVeKO2tnLKPo8dfEeHHC5o=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDQTGuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmM9jzAxOTU94531zc0vLPoDS8t5S2+aMs/JuqqbRUm0uKpoPGcrtsqLhsnFhqTyglTfH+wXWty0FF07tNsGLzfEmqZUFVm0icqeVmKVQIhFmBPBURB1QQhupPTAI45wDB0ymmiAQ4NNWIUIi74SomAwiCujQ5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPalqOWqZTVBomKYMIsxd2z/rsmT2wV/b5Z62OU2PgpU2zNNRyo+K/WM1+/Ktq0Gzj5Fs11rONKnYcrwp5NxxmcAt5qG+ddfvZ3Uy4s8Fu2Rv5v2E99kQ30Frv8l2aZ67H+JHIC70YNSj6ux2jIB+LRLcj8XQ8lNxzW+XFGtaxSf1IIIkDpJCj+jVc4gpdwStEhC0hMUwVPK5mBT9CSH4BVtKQnQ==</latexit>
x2
<latexit sha1_base64="ms/uPGAGeTg0lt5op+vO01P7rjg=">AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYq2SwFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkC2GOP+Q==</latexit>
y
<latexit sha1_base64="WQjRG4bPVEcrBhk0IPMU7zTCz9E=">AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS4FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCsAOP5g==</latexit>
f
f
y
x1
x2 <latexit sha1_base64="mpuRLmJhhTRKlPt+QsNIZJRw5HU=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkN+IiVuxeDoeSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVLKQnA==</latexit>
x1
x2
y

( )
x1
x2
y
f
f
y
x1
x1
x2
y
‣
(= )
(x , x ) y
( )

( )
x1
x2
y
f
f
y
x1
x1
x2
y
‣
(= )
(x , x ) y
( )
‣ (x , x ) ( )
y (x , x )

x1
x2
y
f
f
y
x1
x2
⾒ ( )
N = = N = = N = =
10 10 =100
Full Factorial Design

Fisher
( )
. (Replication)
⾒ ( )
. (Local Control)
. (Randomization) Randomized Controlled Trial (Fisher )
Data Leakage

(or ?)
UQ ⾒
( ) ⾒ ( ?)
有川節夫
Herbert A. Simon
Simon HA, Machine Discovery. ( )
Langley PW, Simon HA, Bradshaw G, Zytkow JM, Scienti c Discovery:
Computational Explorations of the Creative Process ( ).
, . (1996)
, . (2001)

( big data )
( )
interpolator ( )
o -the-shelf

KernelRidge(kernel='rbf', alpha=0.05, gamma=0.1)
KernelRidge(kernel='rbf', alpha=1e-4, gamma=0.1)
ExtraTreesRegressor(n_estimators=50)
DecisionTreeRegressor()

DecisionTreeRegressor() ( )

DecisionTreeRegressor() ( )
/
(Activity Cli s,
Selectivity Cli s, )

(UQ)
skopt.learning.
skopt.learning.
ExtraTreesRegressor(n_estimators=50,
bootstrap=True)
GaussianProcessRegressor(kernel=1*RBF(),
alpha=1e-2)
skopt.learning.
RandomForestRegressor(n_estimators=50)
base = ExtraTreesRegressor(n_estimators=50,
bootstrap=True)
MapieRegressor(base, method="plus", cv=-1)
alpha=1e-5)
alpha=1e-4)
MapieRegressor(base, method="plus",
cv=Subsample(n_resamplings=50))

(>100)
( ) ( !?)
⾒
Balestriero R, Pesenti J, LeCun Y.
Learning in High Dimension Always Amounts to Extrapolation.
arXiv [cs.LG]. 2021. http://arxiv.org/abs/2110.09485
(interpolation)
y
• “Our goal in this paper is to demonstrate both theoretically and
empirically for both synthetic and real data that interpolation almost
surely never occurs in high-dimensional spaces (>100) regardless of the
underlying intrinsic dimension of the data manifold.“
• "Those results challenge the validity of our current interpolation/
extrapolation deﬁnition as an indicator of generalization performances. "

/ 1/2
d (r= )
<latexit sha1_base64="mchf2gvio17p/Fz9iF84xhnMH3c=">AAAC73ichVHNThRBEK4dVFZAWeBi4mXiBrJ72dQSIsTEhOhBj/ztQkLjpqenZ2mZv53p3WSdzAvwAhxMjJB4ID6AD+DFoxcOPILhCMGLidbOTiJKxJ7M9Fdf1VfzdZcVuirWiKcFY+TW7Tujxbtj4xP37k+WpqabcdCNhGyIwA2iLYvH0lW+bGilXbkVRpJ7lis3rb3ng/xmT0axCvwN3Q/ljsfbvnKU4JqoVul1s2JXzacmcyIuEhaqV8kQ2mkyn6bp1YgJO9DsBfc8zlzp6MrVXKTau7qamkwHJpqs0+W2WbGzkCnf0f1qq1TGGmbLvA7qOShDvlaC0idgYEMAArrggQQfNGEXOMT0bEMdEELidiAhLiKksryEFMZI26UqSRWc2D36tinazlmf4kHPOFML+otLb0RKE2bxBI/xHL/gR/yGP/7ZK8l6DLz0abeGWhm2JvcfrH//r8qjXcPub9WNnjU4sJR5VeQ9zJjBKcRQ33tzcL7+ZG02mcMjPCP/h3iKn+kEfu9CfFiVa29v8GORF7oxGlD973FcB835Wv1xbWF1obz8LB9VER7CI6jQPBZhGV7CCjSo/1e4hJ8FMDrGgfHOeD8sNQq5Zgb+WMbxL1/dugg=</latexit>
V (d) =
⇡
d
2
d
2 · d
2
! 0 (d ! 1)
https://www.math.ucdavis.edu/~strohmer/courses/180BigData/180lecture1.pdf
<latexit sha1_base64="eWTwm+uSU1d/PxikW7VncIaHMRY=">AAAChHichVHLSsNAFD2Nr/ps1Y3gRiwVF1JutD5wIUU3LltrVdAiSZzW0DQJSVqoxR/QreLClYIL8QP8ADf+gAs/QVxWcOPC2zQgWtQbJnPmzD13zsxVbUN3PaLnkNTR2dXdE+7t6x8YHIpEh0e2XKviaCKnWYbl7KiKKwzdFDlP9wyxYztCKauG2FZLa8397apwXN0yN72aLfJlpWjqBV1TPKYy8n40RgnyY6IdyAGIIYi0Fb3HHg5gQUMFZQiY8BgbUODytwsZBJu5POrMOYx0f1/gGH2srXCW4AyF2RL/i7zaDViT182arq/W+BSDh8PKCcTpiW6pQY90Ry/08Wutul+j6aXGs9rSCns/cjKWff9XVebZw+GX6k/PHgpY8r3q7N32meYttJa+enTRyC5vxOtTdE2v7P+KnumBb2BW37SbjNi4/MOPyl74xbhB8s92tIOt2YS8kEhmkrHUatCqMMYxiWnuxyJSWEcaOa4vcIoznEvd0ow0J823UqVQoBnFt5BWPgE/Y4+x</latexit>
1
<latexit sha1_base64="z3Ba/qsiF6tEgbjc9DFSGYR9udI=">AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQYZhFEUhuX/uQPqMjMeLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWnrDlFlG1lXdzEuixVRFYxmucJXlDZOJdUllOal20N/PNZlpKbp2yFsGK9XFqqYcKbLIiSpgaLNonZi8XemUfQEMoR3+YRB2QACcSOi+RyhCBXSQoQF1YKABJ6yCCBZ9BQgDgkFcCdrEmYQUe59BBzykbVAWowyR2Br9q7QqOKxG635Ny1bLdIpKwySlH4L4gvfYw2d8wFf8/LNW267R99KiWRpomVH2ni2nP/5V1WnmcPytGumZwxFs214V8m7YTP8W8kDfPL3opXdSwfYa3uIb+b/BLj7RDbTmu3yXZKnLEX4k8kIvRg0K/27HMMhuhMLRUCQZCcT2nVa5YQVWYZ36sQUxiEMCMvaLnsMVXAs+ISrsCnuDVMHlaJbgRwjxLwzik/k=</latexit>
0.5
p
d
<latexit sha1_base64="rzd8+rC/ag8/gv3jUp4Y6gUrSiI=">AAAChnichVHLSsNAFD3GV62PVt0IbopFcRVupVVxVXTjsg9rC1okidMamiYhSQta/AHBrV24UnAhfoAf4MYfcOEniEsFNy68TQOixXrDZM6cuefOmbmqbeiuR/Q8IA0ODY+MhsbC4xOTU5Ho9MyuazUcTRQ0y7Cckqq4wtBNUfB0zxAl2xFKXTVEUa1tdfaLTeG4umXueMe2KNeVqqlXdE3xmMqTnDqIxkkmP2K9IBGAOILIWNF77OMQFjQ0UIeACY+xAQUuf3tIgGAzV0aLOYeR7u8LnCLM2gZnCc5QmK3xv8qrvYA1ed2p6fpqjU8xeDisjGGRnuiW3uiR7uiFPv+s1fJrdLwc86x2tcI+iJzN5T/+VdV59nD0rerr2UMF675Xnb3bPtO5hdbVN0/ab/mN3GJria7plf1f0TM98A3M5rt2kxW5yz5+VPbCL8YNSvxuRy/YXZETq3Iym4ynN4NWhTCPBSxzP9aQxjYyKHD9Ks5xgbYUkmQpJa11U6WBQDOLHyGlvwBbQpAn</latexit>
0.5
<latexit sha1_base64="Nn/M5/H5iD9Z/hOTHxuHUqAZH+I=">AAAChnichVHLSsNAFD2Nr1ofrboR3BRLxVWZlmpFEIpuXPZhH1BLSdKxhqZJSNJCLf6A4NYuXCm4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzJUNVLJuxnkeYmJyanvHO+ubmFxb9gaXlvKU3TZnnZF3VzaIkWlxVNJ6zFVvlRcPkYkNSeUGqHwz2Cy1uWoquHdltg5cbYk1TThRZtInKVvdilUCIRZgTwVEQdUEIbqT0wCOOUYUOGU00wKHBJqxChEVfCVEwGMSV0SHOJKQ4+xzn8JG2SVmcMkRi6/Sv0arkshqtBzUtRy3TKSoNk5RBhNkLu2d99swe2Cv7/LNWx6kx8NKmWRpquVHxX6xmP/5VNWi2cfqtGuvZxgl2HK8KeTccZnALeahvnXX72d1MuLPBbtkb+b9hPfZEN9Ba7/Jdmmeux/iRyAu9GDUo+rsdoyAfi0S3I/F0PJTcd1vlxRrWsUn9SCCJQ6SQo/o1XOIKXcErRIQtITFMFTyuZgU/Qkh+AeO5kGc=</latexit>
d = 2
<latexit sha1_base64="LVcqe4i4MA2A5XBAmTvFXnqLayw=">AAAChnichVHLSsNAFD2Nr1ofrboR3BRLxVWZSrUiCEU3LvuwD6ilJOm0hqZJSNJCLf6A4NYuXCm4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzJUNVLJuxnkeYmJyanvHO+ubmFxb9gaXlnKU3TZlnZV3VzYIkWlxVNJ61FVvlBcPkYkNSeV6qHw728y1uWoquHdttg5caYk1Tqoos2kRlKvuxciDEIsyJ4CiIuiAEN5J64BEnqECHjCYa4NBgE1YhwqKviCgYDOJK6BBnElKcfY5z+EjbpCxOGSKxdfrXaFV0WY3Wg5qWo5bpFJWGScogwuyF3bM+e2YP7JV9/lmr49QYeGnTLA213Cj7L1YzH/+qGjTbOP1WjfVso4pdx6tC3g2HGdxCHupbZ91+Zi8d7mywW/ZG/m9Yjz3RDbTWu3yX4unrMX4k8kIvRg2K/m7HKMhtRaI7kVgqFkocuK3yYg3r2KR+xJHAEZLIUv0aLnGFruAVIsK2EB+mCh5Xs4IfISS+AOf5kGk=</latexit>
d = 4
1
<latexit sha1_base64="UR9ymFZxc/Q/TCzOORzighhjAlM=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDQTEuDNGNSx7ySJCQtgzYUNqmLSRI/AETt7JwpYkL4wf4AW78ARd8gnGJiRsXXkoTo0S8zXTOnLnnzpm5kqEqls1YzyNMTE5Nz3hnfXPzC4v+wNJyztKbpsyzsq7qZkESLa4qGs/aiq3ygmFysSGpPC/VDwf7+RY3LUXXju22wUsNsaYpVUUWbaIylf1YORBiEeZEcBREXRCCG0k98IgTVKBDRhMNcGiwCasQYdFXRBQMBnEldIgzCSnOPsc5fKRtUhanDJHYOv1rtCq6rEbrQU3LUct0ikrDJGUQYfbC7lmfPbMH9so+/6zVcWoMvLRploZabpT9F6uZj39VDZptnH6rxnq2UcWu41Uh74bDDG4hD/Wts24/s5cOdzbYLXsj/zesx57oBlrrXb5L8fT1GD8SeaEXowZFf7djFOS2ItGdSCwVCyUO3FZ5sYZ1bFI/4kjgCElkqX4Nl7hCV/AKEWFbiA9TBY+rWcGPEBJf6hqQag==</latexit>
d > 4
<latexit sha1_base64="nHSiJ7YlA7Y9nChVeD/tJhyz2yo=">AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQoZhFEUhuX/uQPqMjMdLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWXrDlFlW1lXdLEiixVRFY1mucJUVDJOJdUlleam239/PN5lpKbp2wFsGK9fFqqYcKbLIiSpiaKNknZi8HelUfAEMoR3+YRB2QACcSOq+RyjBIeggQwPqwEADTlgFESz6ihAGBIO4MrSJMwkp9j6DDnhI26AsRhkisTX6V2lVdFiN1v2alq2W6RSVhklKPwTxBe+xh8/4gK/4+Wettl2j76VFszTQMqPiPVvOfPyrqtPM4fhbNdIzhyPYsr0q5N2wmf4t5IG+eXrRy2yng+01vMU38n+DXXyiG2jNd/kuxdKXI/xI5IVejBoU/t2OYZCLhMKxUDQVDcT3nFa5YQVWYZ36sQlxSEASsvaLnsMVXAs+ISbsCLuDVMHlaJbgRwiJL6Jhk8c=</latexit>
0.5
p
2
<latexit sha1_base64="z3Ba/qsiF6tEgbjc9DFSGYR9udI=">AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQYZhFEUhuX/uQPqMjMeLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWnrDlFlG1lXdzEuixVRFYxmucJXlDZOJdUllOal20N/PNZlpKbp2yFsGK9XFqqYcKbLIiSpgaLNonZi8XemUfQEMoR3+YRB2QACcSOi+RyhCBXSQoQF1YKABJ6yCCBZ9BQgDgkFcCdrEmYQUe59BBzykbVAWowyR2Br9q7QqOKxG635Ny1bLdIpKwySlH4L4gvfYw2d8wFf8/LNW267R99KiWRpomVH2ni2nP/5V1WnmcPytGumZwxFs214V8m7YTP8W8kDfPL3opXdSwfYa3uIb+b/BLj7RDbTmu3yXZKnLEX4k8kIvRg0K/27HMMhuhMLRUCQZCcT2nVa5YQVWYZ36sQUxiEMCMvaLnsMVXAs+ISrsCnuDVMHlaJbgRwjxLwzik/k=</latexit>
0.5
p
d
!?
Unit Cube in
Unit Ball
1
<latexit sha1_base64="PtefnFPcLJximPXvkayt2U9xPRk=">AAAChXichVHLSsNAFD2Nr1pfVTeCm2KpuLHcSFFxoUU3LvuwWqhFkjjVYJqEJC1o8QfErbpwpeBC/AA/wI0/4KKfIC4ruHHhbRoQFfWGyZw5c8+dM3NV29Bdj6gZkrq6e3r7wv2RgcGh4ZHo6Nima9UcTRQ0y7Ccoqq4wtBNUfB0zxBF2xFKVTXElnqw1t7fqgvH1S1zwzu0Rbmq7Jl6RdcUj6ncsrwTjVOS/Ij9BHIA4ggiY0XvsY1dWNBQQxUCJjzGBhS4/JUgg2AzV0aDOYeR7u8LHCPC2hpnCc5QmD3g/x6vSgFr8rpd0/XVGp9i8HBYGUOCnuiWWvRId/RM77/Wavg12l4OeVY7WmHvjJxM5N/+VVV59rD/qfrTs4cKFn2vOnu3faZ9C62jrx9dtPJLuURjmq7phf1fUZMe+AZm/VW7yYrc5R9+VPbCL8YNkr+34yfYnEvK88lUNhVPrwatCmMSU5jhfiwgjXVkUOD6FZziDOdSnzQrpaT5TqoUCjTj+BLSygfo4Y/5</latexit>
> 1
!?
( ⾒) 0

/ 2/2
Bronstein MM, Bruna J, Cohen T, Veličković P.
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.
arXiv [cs.LG]. 2021. http://arxiv.org/abs/2104.13478
for all
<latexit sha1_base64="NqBYbofjRLhsRuy5gs3o5FfAjVo=">AAAConichVG7SgNBFD1Z3/EVtRFECEbFKkwkqKQK2ohVHkaFJIbddRKX7IvdSUCDlZ0/YGGlYCF2WvgBNv6ART5BLBVsLLzZLAQN6l1m58yZe+6cmavYuuYKxpoBqae3r39gcCg4PDI6Nh6amNxxrZqj8pxq6Zazp8gu1zWT54QmdL5nO1w2FJ3vKtWN1v5unTuuZpnb4sjmRUOumFpZU2VBVCk0W04UDFkcKkojc7J/EC4IK9whSqEIizIvwt0g5oMI/EhZoQcUcAALKmowwGFCENYhw6UvjxgYbOKKaBDnENK8fY4TBElboyxOGTKxVfpXaJX3WZPWrZqup1bpFJ2GQ8owFtgzu2Fv7Indshf2+Wuthlej5eWIZqWt5XZp/Gw6+/GvyqBZ4LCj+tOzQBlrnleNvNse07qF2tbXj8/fsonMQmORXbFX8n/JmuyRbmDW39XrNM9c/OFHIS/0YtSg2M92dIOd5WhsJRpPxyPJdb9Vg5jBHJaoH6tIYhMp5Kj+KW5wh3tpXtqS0lK2nSoFfM0UvoVU+AKNf5vS</latexit>
f : Rd
! R
<latexit sha1_base64="bLp7Jqe5Ke+bRuLIs0xt/zwFMHo=">AAACmnichVHLLgRBFD3T3u9BJBIWExOPhUxqRBCrCRtiM8a8EsOkuxUq+pXumgk68wN+wMKKxAIf4ANs/ICFTxBLEhsLd3o6EQS3U12nTt1z61RdzTGEJxl7jChNzS2tbe0dnV3dPb190f6BvGdXXJ3ndNuw3aKmetwQFs9JIQ1edFyumprBC9rBcn2/UOWuJ2wrK48cvmWqe5bYFboqiSpHhw6nDydjJWHFSqYq9zXNz9S2d8rROEuwIGI/QTIEcYSRtqO3KGEHNnRUYILDgiRsQIVH3yaSYHCI24JPnEtIBPscNXSStkJZnDJUYg/ov0erzZC1aF2v6QVqnU4xaLikjGGcPbAr9sLu2Q17Yu+/1vKDGnUvRzRrDS13yn0nwxtv/6pMmiX2P1V/epbYxULgVZB3J2Dqt9Ab+urx6cvGYmbcn2AX7Jn8n7NHdkc3sKqv+uU6z5z94UcjL/Ri1KDk93b8BPmZRHIuMbs+G08tha1qxwjGMEX9mEcKK0gjR/V9nOMaN8qosqSsKmuNVCUSagbxJZTsB6MZmAU=</latexit>
x, x0
2 Rd
-Lipschitz
<latexit sha1_base64="VAwN9BEfNuoROOZ0f78ZUuUItlA=">AAACqHichVFNLwNRFD0d39/FRmIz0fhM2ryKIFYNGwuL+qgSFZkZr0y8zoyZ16a0fgB/wMKKxEIsLLG28Qcs/ASxJLGxcDudRBDcybx73nn33Dlvru4I05OMPYaUmtq6+obGpuaW1rb2jnBn17Jn512Dpwxb2O6KrnlcmBZPSVMKvuK4XMvpgqf1nZnKebrAXc+0rSW55/D1nLZlmVnT0CRRG+FIOTtUHI7SMjhcVjOC73pCs6Q6p2bKajFaHKRMVSzG/FB/gngAIggiaYevkcEmbBjIIwcOC5KwgAaPnjXEweAQt44ScS4h0z/nOEAzafNUxalCI3aH1i3arQWsRftKT89XG/QVQa9LShX97IFdsBd2zy7ZE3v/tVfJ71HxskdZr2q5s9Fx1LP49q8qR1li+1P1p2eJLCZ9ryZ5d3ymcgujqi/sH78sTi30lwbYGXsm/6fskd3RDazCq3E+zxdO/vCjkxf6YzSg+Pdx/ATLo7H4eGxsfiySmA5G1Yhe9GGI5jGBBGaRRIr6H+IKN7hVRpSkklZWq6VKKNB040so+gci3ZxP</latexit>
|f(x) f(x0
)| 6 Lkx x0
k
<latexit sha1_base64="+ZiSkRyCDmMX1dfLuFjrVp2j6hw=">AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdpYWJjERCGK7K6TOLjZXXYngRj8AW0VCysFC/ED/AAbf8AinyCWEWwsvNksiAb1LrNz5sw9d87M1WxDuJKxekDp6OwKdod6evv6BwaHwsMjOdcqOzrP6pZhOTua6nJDmDwrhTT4ju1wtaQZfFs7Wmvub1e44wrL3JJVm++V1KIpCkJXJVGpjf1wlMWYF5F2EPdBFH5sWuEH7OIAFnSUUQKHCUnYgAqXvjziYLCJ20ONOIeQ8PY5TtBL2jJlccpQiT2if5FWeZ81ad2s6XpqnU4xaDikjGCSPbM71mBP7J69sI9fa9W8Gk0vVZq1lpbb+0OnY5n3f1UlmiUOv1R/epYoYMnzKsi77THNW+gtfeX4spFZTk/WptgNeyX/16zOHukGZuVNv03x9NUffjTyQi9GDYr/bEc7yM3G4guxRCoRTa76rQphHBOYpn4sIol1bCJL9TnOcI4LJajMKHPKfCtVCfiaUXwLZeUTeMOPzA==</latexit>
L
Donoho DL,
High-dimensional data analysis: The curses and blessings of dimensionality.
Plenary Lecture, AMS National Meeting on Mathematical Challenges of the 21st Century. 2000.
(Lipschitz ) ⾒
ε ( /ε)d Sobolev class
d= 100 d= 10 =100

Underspecification Rashomon
( )
CV
Underspeci cation
(
specify )
D’Amour et al., Underspeciﬁcation Presents Challenges for Credibility in Modern Machine Learning.
J Mach Learn Res, 2022; 23: 1-61. https://ai.googleblog.com/2021/10/how-underspeciﬁcation-presents.html

Underspeci cation
(
specify )
specify

Underspeci cation
(
specify )
specify
(Underspeci cation)
α

Kernel Ridge (RBF)
Neural Network (MLP)
Gradient Boosted Trees
SVR (RBF) Gaussian Process (RBF)
Random Forest
Nearest Neighbors Decision Tree
⾒
Nearest Neighbor
(Neural Network
)

=
if-then
<latexit sha1_base64="KLK3uKDnOLsAzd1uz4imbactEN0=">AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho=</latexit>
X2  ✓1
yes no
<latexit sha1_base64="fUwPPBtoir3sq20H215QULGdmUg=">AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho=</latexit>
X1  ✓2
yes no
Blue <latexit sha1_base64="TQ4GrszeMgM294G1rFaPjchY/V8=">AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd</latexit>
X2  ✓4
yes no
Red <latexit sha1_base64="FaDvqDDJ/DHLyBEhkovQS4NqCn0=">AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd</latexit>
X1  ✓5
yes no
Red
<latexit sha1_base64="j6aRObEml8aKBOmlM9//dP7CY5I=">AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4=</latexit>
X1  ✓6
yes no
Red
Blue
<latexit sha1_base64="3d4/BR7YXLLt+AHkvejt6OKrM6I=">AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb</latexit>
X1  ✓3
yes no
Red
<latexit sha1_base64="bG79VND5DJn1e15l/47tj0nzmbk=">AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykWHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm4/JcPJFJxFJLQav6MIFJzFA/kkhhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bz4+lh8=</latexit>
X1  ✓7
yes no
Blue Blue
if x2 ≤ θ1 then
if x1 ≤ θ2 then
return Blue
else
if x2 ≤ θ4 then
return Red
else
if x1 ≤ θ5 then
return Red
else
if x1 ≤ θ6 then
return Blue
else
return Red
else
if x1 ≤ θ3 then
if x2 ≤ θ7 then
return Blue
else
return Blue
else
return Red
<latexit sha1_base64="tvPTQvEnR42QahYYPRslvA0PAkQ=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5CUMugjWVE8wHJEfY2e8mS3b1jd08IR36CrdZ2YuuvsfSfuLlcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzN/c4zVZpF8slMY+oLPJIsZAQbKz12B7VBueJW3QxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n2akzdGGVIQojZUsalKl/J1IstJ6KwHYKbMZ61ZuL/3m9xIQ3fspknBgqyWJRmHBkIjT/Gw2ZosTwqSWYKGZvRWSMFSbGprO0JRAzm4m3msA6adeq3lW1/lCvNG7zdIpwBudwCR5cQwPuoQktIDCCF3iFNyd13p0P53PRWnDymVNYgvP1C4sglCA=</latexit>
X2
<latexit sha1_base64="wDeZ5yQA5kysGCdsNSdw9QhGcj8=">AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzQOSJcxOZpMh81hmZoWw5BO86tmbePVrPPonTpI9mMSChqKqm+6uKOHMWN//9gpr6xubW8Xt0s7u3v5B+fCoaVSqCW0QxZVuR9hQziRtWGY5bSeaYhFx2opGd1O/9Uy1YUo+2XFCQ4EHksWMYOukx3Yv6JUrftWfAa2SICcVyFHvlX+6fUVSQaUlHBvTCfzEhhnWlhFOJ6VuamiCyQgPaMdRiQU1YTY7dYLOnNJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC079Rn2lKLB87golm7lZEhlhjYl06C1siMXGZBMsJrJLmRTW4ql4+XFZqt3k6RTiBUziHAK6hBvdQhwYQGMALvMKbl3nv3of3OW8tePnMMSzA+/oFiY2UHw==</latexit>
X1
<latexit sha1_base64="FVULVmCNg47QKtvg46uxkYeCQJE=">AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSTayFoyRO3IiMAtEDUfAibYQfoQInYIACLdCAgg4csQoS2PiVQQQCJnJV6CJnIWLuPoVzCKC2hVkUMyRkm/iv46rssTqunZq2q1bwFBWHhcoIxMgLuScD8kweyCv5/LNW163heOngLA+11KyFLlZyH/+qNJw5NL5VYz1zOIVd1ytD76bLOLdQhvr22dUgl8zGuuvklryh/xvSJ094A739rtxlaLY3xo+MXvDFsEHi73aMgsJmXNyOJzKJaGrfa5UfVmENNrAfO5CCI0hD3u3DJfTgWggKW0JS2BumCj5Psww/Qjj8Ak3rksg=</latexit>
✓1
<latexit sha1_base64="OBBuUzKAHrHcbt2xZlrMzfgelZg=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRAooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4V4NLYTTWQS4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXtoWksD9KFTyuZhV+hHD0BVALksk=</latexit>
✓2
<latexit sha1_base64="587jdLEUJKtkMM/L+3DOcHwPLKw=">AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSLVELR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvFfx1XZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzaHyrxnrmcAq7rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BVRLkss=</latexit>
✓4
<latexit sha1_base64="e77Z2aqe8KZFIqh8R+ZYHDOKN+I=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQfIRYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixPS9wgTk1PTM95Z39y8fyEQXFzKW3rLlGlO1hXdLEqiRRWm0RxnXKFFw6SiKim0IDUP7f1Cm5oW07UT3jFoRRXrGqsxWeRIFcu8QblY3a4GwyRKnAiNgpgLwuBGSg8+QhlOQQcZWqACBQ04YgVEsPArQQwIGMhVoIuciYg5+xTOwYfaFmZRzBCRbeK/jquSy2q4tmtajlrGUxQcJipDECEv5J4MyDN5IK/k889aXaeG7aWDszTUUqMauFjJfvyrUnHm0PhWjfXMoQZ7jleG3g2HsW8hD/Xts6tBNpGJdNfJLXlD/zekT57wBlr7Xb5L00xvjB8JveCLYYNiv9sxCvKb0dhONJ6Oh5MHbqu8sAprsIH92IUkHEMKck4fLqEH14Jf2BISwv4wVfC4mmX4EcLRF1Zrksw=</latexit>
✓5
<latexit sha1_base64="eu7hkk7xInDVK5lVQXXLPV+MjYw=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQgoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaCwRjWfi4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BViLks0=</latexit>
✓6
<latexit sha1_base64="YU9dEL0zbRAMaEFNhHkGnXZKQpA=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmQQooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGwnmsgkwqkDt1VeWIcN2MJ+7EIKjiENeacPl9CHa8EvxIWksD9KFTyuZhV+hHD0BVIrkso=</latexit>
✓3
<latexit sha1_base64="VJpvibITWQ0L0zaDACUdk6SsPiA=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQIoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGw3Gs/Ew6kDt1VeWIcN2MJ+JCAFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BVqrks4=</latexit>
✓7
<latexit sha1_base64="wDeZ5yQA5kysGCdsNSdw9QhGcj8=">AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzQOSJcxOZpMh81hmZoWw5BO86tmbePVrPPonTpI9mMSChqKqm+6uKOHMWN//9gpr6xubW8Xt0s7u3v5B+fCoaVSqCW0QxZVuR9hQziRtWGY5bSeaYhFx2opGd1O/9Uy1YUo+2XFCQ4EHksWMYOukx3Yv6JUrftWfAa2SICcVyFHvlX+6fUVSQaUlHBvTCfzEhhnWlhFOJ6VuamiCyQgPaMdRiQU1YTY7dYLOnNJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC079Rn2lKLB87golm7lZEhlhjYl06C1siMXGZBMsJrJLmRTW4ql4+XFZqt3k6RTiBUziHAK6hBvdQhwYQGMALvMKbl3nv3of3OW8tePnMMSzA+/oFiY2UHw==</latexit>
X1
<latexit sha1_base64="tvPTQvEnR42QahYYPRslvA0PAkQ=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5CUMugjWVE8wHJEfY2e8mS3b1jd08IR36CrdZ2YuuvsfSfuLlcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzN/c4zVZpF8slMY+oLPJIsZAQbKz12B7VBueJW3QxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n2akzdGGVIQojZUsalKl/J1IstJ6KwHYKbMZ61ZuL/3m9xIQ3fspknBgqyWJRmHBkIjT/Gw2ZosTwqSWYKGZvRWSMFSbGprO0JRAzm4m3msA6adeq3lW1/lCvNG7zdIpwBudwCR5cQwPuoQktIDCCF3iFNyd13p0P53PRWnDymVNYgvP1C4sglCA=</latexit>
X2

( )
RandomForestRegressor
(n_estimators=6,
max_leaf_nodes=8)
6 × DecisionTreeRegressor(max_leaf_nodes=8)
= + +
+ + +
8
⾒
8
( )
min_samples_leaf

/ ⾒
PolyReg(1)
RMSE 0.299
PolyReg(3)
RMSE 0.28
PolyReg(5)
RMSE 0.225
PolyReg(7)
RMSE 0.113
PolyReg(10)
RMSE 0.0189
PolyReg(15)
RMSE 0.00737
PolyReg(20)
RMSE 0.000
PolyReg(30)
RMSE 0.000
ExtraTrees (no bootstrap)
RMSE 0.000
ExtraTrees (bootstrap)
RMSE 0.0121
Random Forest
RMSE 0.012
LGBM
RMSE 0.0508
95%予測区間 95%予測区間 95%予測区間 95%予測区間
Problematic overﬁtting by polynomial regression of order k
clearly overﬁtted but harmless (still informative)

Interpolator
0 ( )
ExtraTrees (no bootstrap) ExtraTrees (no bootstrap) ExtraTrees (no bootstrap) ExtraTrees (no bootstrap)
Gradient Boosted Trees Gradient Boosted Trees Gradient Boosted Trees Gradient Boosted Trees

Interpolator
Nearest Neighbor (k=1) Nearest Neighbor (k=1) Nearest Neighbor (k=1) Nearest Neighbor (k=1)
Decision Tree Decision Tree Decision Tree Decision Tree
0 ( GBDT, -NN, DT )

ExtraTrees
Geurts, Ernst, Wehenkel, Extremely randomized trees. Mach Learn 63, 3–42 (2006).
https://doi.org/10.1007/s10994-006-6226-1
RandomForestRegressor(n_estimators=10)
( )

Local-averagingestimators
Nearest Neighbor

Local-averagingestimators
Nearest Neighbor
Nearest Neighbor
(X, y)
(Stone , 1977)
Nadaraya-Watson
Attention Mechanism
ICML A Tutorial on Attention in Deep Learning
https://icml.cc/Conferences/ /ScheduleMultitrack?event=
Histogram rules on data-dependent partitions (or data-driven histogram methods)
90 (Nobel, Ann. Statist. ( ), ; Lugosi & Nobel,
Ann. Statist. (2), 1996) ( Yellow Terror)
Devroye, Györ , Lugosi, A Probabilistic Theory of Pattern Recognition, . https://www.szit.bme.hu/~gyor /pbook.pdf

Interpolation
⾒ ( 0 or 0)
Harmless over tting Benign over tting
( )

Interpolation
⾒ ( 0 or 0)
( )
Hastie et al, Surprises in high-dimensional ridgeless least squares interpolation (2020)
https://arxiv.org/abs/1903.08560
Muthukumar et al, Harmless interpolation of noisy data in regression (2019)
Bartlett et al, Benign overfitting in linear regression (2019)
Belkin et al, Overfitting or perfect fitting? Risk bounds for classification and regression rules
that interpolate (2018)
<latexit sha1_base64="V6gnlRc34dPuxi8leGWVweNz3Zg=">AAACG3icbVDLSsNAFJ3UV62vqks3g0VaEUsivjZC0Y3uKtg20MQymUzaoZMHMxMhhHyAH+E3uNW1O3HrwqV/4rTNwloPDBzOOZd75zgRo0Lq+pdWmJtfWFwqLpdWVtfWN8qbW20RxhyTFg5ZyE0HCcJoQFqSSkbMiBPkO4x0nOHVyO88EC5oGNzJJCK2j/oB9ShGUkm9csVyiETwAtbMqgkPoMXUrIvgzf59emhk0KzCRKX0uj4GnCVGTiogR7NX/rbcEMc+CSRmSIiuoUfSThGXFDOSlaxYkAjhIeqTrqIB8omw0/FnMrinFBd6IVcvkHCs/p5IkS9E4jsq6SM5EH+9kfif142ld26nNIhiSQI8WeTFDMoQjpqBLuUES5YogjCn6laIB4gjLFV/U1scP1OdGH8bmCXto7pxWj+5Pa40LvN2imAH7IIaMMAZaIBr0AQtgMEjeAYv4FV70t60d+1jEi1o+cw2mIL2+QPP7p7D</latexit>
= (X0
X + I) 1
X0
y
<latexit sha1_base64="VsNVIh2b6vsuVSv8LjXaQ8stH3Y=">AAACDnicbVDLSsNAFJ3UV62vqODGzWCRVoSSiK+NUHTjsoJtA20sk+m0HTqZhJmJEGL+wW9wq2t34tZfcOmfOG2zsK0HLhzOuZdzOV7IqFSW9W3kFhaXllfyq4W19Y3NLXN7pyGDSGBSxwELhOMhSRjlpK6oYsQJBUG+x0jTG96M/OYjEZIG/F7FIXF91Oe0RzFSWuqYe22PKASvYNkpOUcPyXEKnRKMO2bRqlhjwHliZ6QIMtQ65k+7G+DIJ1xhhqRs2Vao3AQJRTEjaaEdSRIiPER90tKUI59INxn/n8JDrXRhLxB6uIJj9e9FgnwpY9/Tmz5SAznrjcT/vFakepduQnkYKcLxJKgXMagCOCoDdqkgWLFYE4QF1b9CPEACYaUrm0rx/FR3Ys82ME8aJxX7vHJ2d1qsXmft5ME+OABlYIMLUAW3oAbqAIMn8AJewZvxbLwbH8bnZDVnZDe7YArG1y+Uw5p9</latexit>
= (X0
X)+
X0
y
Ridgeless
Ridge

Interpolation
⾒ ( 0 or 0)
( )
Belkin M, Hsu D, Mitra PP.
Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. NIPS 2018.
https://dl.acm.org/doi/10.5555/3327144.3327157
"Many modern machine learning models are trained to achieve zero or near-zero training error in order to obtain
near-optimal (but non-zero) test error. This phenomenon of strong generalization performance for "overfitted" /
interpolated classifiers appears to be ubiquitous in high-dimensional data, having been observed in deep networks,
kernel machines, boosting and random forests. Their performance is consistently robust even when the data
contain large amounts of label noise."
Belkin M.
Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation.
Acta Numerica, 2021; 30: 203-248: https://doi.org/10.1017/S0962492921000039
Weyner AJ, Olson M, Bleich J, and Mease D.
Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers. J Mach Learn Res.
2017; 18(48): 1-33. https://jmlr.org/papers/v18/15-240.html

(LSH)
.
( )
ReLU = max(x, )
(= )
cf.

(LSH)
Balestriero, Randall. "Max-Aﬃne Splines Insights Into Deep Learning." (2021)
Diss., Rice University. https://hdl.handle.net/1911/110439
Balestriero & Baraniuk. A Spline Theory of Deep Learning (ICML 2018)
https://proceedings.mlr.press/v80/balestriero18b.html
Balestriero et al., The Geometry of Deep Networks: Power Diagram Subdivision (NeurIPS 2019)
.
( )
ReLU = max(x, )
(= )
cf.

https://www.kaggle.com/kaggle-survey-2021
State of Data Science and Machine Learning 2021 (Kaggle Survey)
Q17. Which of the following ML algorithms do you use on a regular basis?
Linear or Logistic Regression
Decision Trees or Random Forests
Gradient Boosting Machines (xgboost, lightgbm, etc)
Convolutional Neural Networks
Dense Neural Networks (MLPs, etc)
Bayesian Approaches
Recurrent Neural Networks
Transformer Networks (BERT, gpt-3, etc)
None or Other
Generative Adversarial Networks
Evolutionary Approaches
0 3,500 7,000 10,500 14,000
13,852
11,863
7,566
7,410
4,468
4,392
4,228
2,273
1,953
1,353
963
# respondents: 25,973
( ⾒ ?)

ExtraTrees Random Forests
( Boosting ⾒ Quantile )
scikit-optimize
https://scikit-optimize.github.io/

ConformalPrediction
https://github.com/scikit-learn-contrib/MAPIE
regressor = ExtraTreesRegressor(max_leaf_nodes=32, bootstrap=True)
MapieRegressor(regressor,
method="plus", cv=-1)
method="plus",
cv=Subsample(n_resampling
s=50))
method="plus", cv=-1)
method="plus",
cv=Subsample(n_resampling
s=50))
95% prediction intervals
Jacknife+
Jacknife+ after bootstrap
Jacknife+
Jacknife+ after bootstrap
Conformal Prediction

SWED-8
SWED-3
Wang et al., Accelerated discovery of multi-elemental reverse water-gas shift catalysts using extrapolative machine
learning approach. (2022) https://doi.org/10.26434/chemrxiv-2022-695rj
( )

CV ( )
CV
( )
f
(Xi, yi
) yi
≈ ̂
yi = f(Xi)
( ) ⾒ ( )

Physics-informedML,GeometricML,CausalML
https://doi.org/10.1038/s42254-021-00314-5

( ) ( ) ⾒
( ) ⾒
⾒
( Symmetry Geometric ML )

( ) ( ) ⾒
( ) ⾒
⾒
( Symmetry Geometric ML )
⾒
課
https://doi.org/10.1038/s42254-021-00314-5

. Two Cultures ( vs )
‣ Data Modeling ( ) vs. Algorithmic Modeling ( )
L. Breiman Statistical Modeling: The Two Cultures ( ) + Cox Efron
‣ 20 Observational Studies (2021) +
Breiman's main point is: If you want prediction, do prediction for its own sake and
forget about the illusion of representing nature. (Judea Pearl)

. Two Cultures ( vs )
‣ Data Modeling ( ) vs. Algorithmic Modeling ( )
L. Breiman Statistical Modeling: The Two Cultures ( ) + Cox Efron
‣ 20 Observational Studies (2021) +
Breiman's main point is: If you want prediction, do prediction for its own sake and
forget about the illusion of representing nature. (Judea Pearl)
. p ( HARKing )
‣ (A. Lang, )
‣ 2019 Nature Retire statistical signi cance Don t say statistically signi cant
(ASA) (2016) (2019)
(800 )

vs
(= )
• “Theory-driven models can be wrong. But data-driven models cannot be wrong or
right. Data-driven are not trying to describe an underlying reality.”
David Hand, KDD2018 (Keynote Talk)
http://videolectures.net/kdd2018_hand_data_science/
⾒
( )

⾒ (NP-hard )
Schrittweiser et al. Mastering Atari, Go, chess and shogi by planning with a learned model.
(“MuZero”, Nature, 2020)
Davies et al. Advancing mathematics by guiding human intuition with AI. (Nature, 2021)
Fawzi et al. Discovering faster matrix multiplication algorithms with reinforcement learning.
(“AlphaTensor”, Nature, 2022)
×
closed /

( )
( )
( )
⾒ (Physics-infomed ML, Causal ML, Geometric ML, )

.
UQ ⾒
. o -the-shelf
( ) Underspeci cation
.
⾒ (Physics-informed ML, Causal ML, Geometric ML)
+ (Pure Math, Pure CS, ⾒ , / )
PDF https://itakigawa.page.link/IBISML taki

機械学習と機械発見：自然科学研究におけるデータ利活用の再考

機械学習と機械発見：自然科学研究におけるデータ利活用の再考

More Related Content

What's hot

More from Ichigaku Takigawa

Recently uploaded

機械学習と機械発見：自然科学研究におけるデータ利活用の再考