決定森回帰の信頼区間推定, Benign Overfitting, 多変量木とReLUネットの入力空間分割

ichigaku.takigawa@riken.jp
2022 2 24

if x2 ≤ θ1 then
if x1 ≤ θ2 then
return Blue
else
if x2 ≤ θ4 then
return Red
else
if x1 ≤ θ5 then
return Red
else
if x1 ≤ θ6 then
return Blue
else
return Red
else
if x1 ≤ θ3 then
if x2 ≤ θ7 then
return Blue
else
return Blue
else
return Red
<latexit sha1_base64="KLK3uKDnOLsAzd1uz4imbactEN0=">AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho=</latexit>
X2  ✓1
yes no
<latexit sha1_base64="fUwPPBtoir3sq20H215QULGdmUg=">AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho=</latexit>
X1  ✓2
yes no
Blue <latexit sha1_base64="TQ4GrszeMgM294G1rFaPjchY/V8=">AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd</latexit>
X2  ✓4
yes no
Red <latexit sha1_base64="FaDvqDDJ/DHLyBEhkovQS4NqCn0=">AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd</latexit>
X1  ✓5
yes no
Red
<latexit sha1_base64="j6aRObEml8aKBOmlM9//dP7CY5I=">AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4=</latexit>
X1  ✓6
yes no
Red
Blue
X1  ✓3
yes no
Red
X1  ✓7
yes no
Blue Blue
<latexit sha1_base64="FVULVmCNg47QKtvg46uxkYeCQJE=">AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSTayFoyRO3IiMAtEDUfAibYQfoQInYIACLdCAgg4csQoS2PiVQQQCJnJV6CJnIWLuPoVzCKC2hVkUMyRkm/iv46rssTqunZq2q1bwFBWHhcoIxMgLuScD8kweyCv5/LNW163heOngLA+11KyFLlZyH/+qNJw5NL5VYz1zOIVd1ytD76bLOLdQhvr22dUgl8zGuuvklryh/xvSJ094A739rtxlaLY3xo+MXvDFsEHi73aMgsJmXNyOJzKJaGrfa5UfVmENNrAfO5CCI0hD3u3DJfTgWggKW0JS2BumCj5Psww/Qjj8Ak3rksg=</latexit>
✓1
<latexit sha1_base64="587jdLEUJKtkMM/L+3DOcHwPLKw=">AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSLVELR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvFfx1XZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzaHyrxnrmcAq7rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BVRLkss=</latexit>
✓4
<latexit sha1_base64="OBBuUzKAHrHcbt2xZlrMzfgelZg=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRAooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4V4NLYTTWQS4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXtoWksD9KFTyuZhV+hHD0BVALksk=</latexit>
✓2
<latexit sha1_base64="e77Z2aqe8KZFIqh8R+ZYHDOKN+I=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQfIRYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixPS9wgTk1PTM95Z39y8fyEQXFzKW3rLlGlO1hXdLEqiRRWm0RxnXKFFw6SiKim0IDUP7f1Cm5oW07UT3jFoRRXrGqsxWeRIFcu8QblY3a4GwyRKnAiNgpgLwuBGSg8+QhlOQQcZWqACBQ04YgVEsPArQQwIGMhVoIuciYg5+xTOwYfaFmZRzBCRbeK/jquSy2q4tmtajlrGUxQcJipDECEv5J4MyDN5IK/k889aXaeG7aWDszTUUqMauFjJfvyrUnHm0PhWjfXMoQZ7jleG3g2HsW8hD/Xts6tBNpGJdNfJLXlD/zekT57wBlr7Xb5L00xvjB8JveCLYYNiv9sxCvKb0dhONJ6Oh5MHbqu8sAprsIH92IUkHEMKck4fLqEH14Jf2BISwv4wVfC4mmX4EcLRF1Zrksw=</latexit>
✓5
<latexit sha1_base64="eu7hkk7xInDVK5lVQXXLPV+MjYw=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQgoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaCwRjWfi4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BViLks0=</latexit>
✓6
<latexit sha1_base64="YU9dEL0zbRAMaEFNhHkGnXZKQpA=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmQQooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGwnmsgkwqkDt1VeWIcN2MJ+7EIKjiENeacPl9CHa8EvxIWksD9KFTyuZhV+hHD0BVIrkso=</latexit>
✓3
<latexit sha1_base64="VJpvibITWQ0L0zaDACUdk6SsPiA=">AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQIoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGw3Gs/Ew6kDt1VeWIcN2MJ+JCAFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BVqrks4=</latexit>
✓7
<latexit sha1_base64="JcXOTtKAO5x/t4uZH9BQrQClbjc=">AAACsnichVFNLwNBGH6sr/ouLhKXjYa4aGZpEIlEuDi2KEJls7sdNbHd3exOK9X4A64ODk4kDuIHiCsXf8DBTxBHEhcHb7crguDdzM4zz7zPO8/Ma3q2CCRjDw1KY1NzS2usrb2js6u7J97btxq4Jd/iWcu1XX/dNAJuC4dnpZA2X/d8bhRNm6+Zuwu1/bUy9wPhOiuy4vGtolFwxLawDEmUHh9L60KdmVUzuqbm9ni+wAmOf8KJD5jLuzLQ4wmWZGGoP4EWgQSiSLvxK+SQhwsLJRTB4UAStmEgoG8TGhg84rZQJc4nJMJ9jgO0k7ZEWZwyDGJ36V+g1WbEOrSu1QxCtUWn2DR8UqoYZvfsgj2zO3bJHtnbr7WqYY2alwrNZl3LPb3ncGD59V9VkWaJnU/Vn54ltjEdehXk3QuZ2i2sur68f/y8PLM0XB1hZ+yJ/J+yB3ZLN3DKL9Z5hi+d/OHHJC/0YtQg7Xs7foLV8aQ2mUxlUom5+ahVMQxiCKPUjynMYRFpZKn+Ea5xg1slpWwohmLVU5WGSNOPL6HY7+R7n+w=</latexit>
Pi := Q1 ^ Q2 ^ Q3 ^ . . .
<latexit sha1_base64="zBiZIFwyoN/DGVfflNKDwKhdwmg=">AAACynichVHLTttAFD24DyB9kJYNUjdWIxCraIIQIKRWUatKLFgQIBAJI8s2lzBkYrv2JFJqZceqP9BFV0ViUfEB/YBuumXBgvIFiCWVuumCG8cSAgS91njOnLnnzpm5bqhkrIU4GTAePHz0eHBoOPfk6bPnI/kXL9fioBV5VPUCFUQ114lJSZ+qWmpFtTAip+kqWncb73v7622KYhn4q7oT0mbTqftyW3qOZsrOf6jYu+b8G9NyqS79xONScTdXsxumpeijaekd0o6tTMtKybdXTM4ifysT2PmCKIo0zNuglIECslgK8j9gYQsBPLTQBMGHZqzgIOZvAyUIhMxtImEuYiTTfUIXOda2OIs4w2G2wf86rzYy1ud1r2acqj0+RfGIWGliXByL7+JC/BKH4kz8u7NWktboeenw7Pa1FNojn8dW/v5X1eRZY+dKda9njW3MpV4lew9TpncLr69vf/pysTK/PJ5MiH1xzv6/iRPxk2/gt/94BxVa/nqPH5e98Itxg0o323EbrE0VSzPF6cp0ofwua9UQXuE1JrkfsyhjAUuocv1DHOE3To1FIzI6RtJPNQYyzSiuhbF3CZqVqlw=</latexit>
Qj :=
(
Xk  ✓l
Xk > ✓l
Blue
<latexit sha1_base64="p+ll+ob/bB8deuXgcYGzmiYX5Zc=">AAACoXichVHLSsNAFD3G97NVN4IugqXiqkyKqAiC6EZ3tbZaaEtJ0rGGpklIpgUtblz6Ay5cKQiKK934AW78ARd+gris4MaFt2lAtFhvmMyZM/fcOTNXc0zDE4y9dEndPb19/QODQ8Mjo2Oh8PjErmdXXZ2nddu03Yymetw0LJ4WhjB5xnG5WtFMvqeVN5r7ezXueoZtpcShw/MVtWQZ+4auCqIK4ZmUvLIqJwqKnKtxTiDeArmiLbxCOMJizA+5HSgBiCCIhB1+QA5F2NBRRQUcFgRhEyo8+rJQwOAQl0edOJeQ4e9zHGOItFXK4pShElumf4lW2YC1aN2s6flqnU4xabiklBFlz+yGNdgTu2Wv7PPPWnW/RtPLIc1aS8udQuh0aufjX1WFZoGDb1VHzwL7WPa9GuTd8ZnmLfSWvnZ01thZSUbrc+ySvZH/C/bCHukGVu1dv9rmyfMOfjTyQi9GDVJ+t6Md7MZjymJsYXshsrYetGoA05jFPPVjCWvYRAJpqn+Ca9zhXopIW1JCSrZSpa5AM4kfIWW/AAtRmVk=</latexit>
T := P1 _ P2 _ . . .
<latexit sha1_base64="+WntnxsFSxJvfdpwMjPE/ws7ZhE=">AAACqXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoxCSlpmfmVScDzSiu5QqxNYyJUQBSBlwxqXkpUOF4AWUDPQMwUMBkGEIZygxQEJAvsJ0hhiGFIZ8hmaGUIZchlSGPoQTIzmFIZCgGwmgGQwYDhgKgWCxDNVCsCMjKBMunMtQycAH1lgJVpQJVJAJFs4FkOpAXDRXNA/JBZhaDdScDbckB4iKgTgUGVYOrBisNPhucMFht8NLgD06zqsFmgNxSCaSTIHpTC+L5uySCvxPUlQukSxgyELrwurmEIY3BAuzWTKDbC8AiIF8kQ/SXVU3/HGwVpFqtZrDI4DXQ/QsNbhocBvogr+xL8tLA1KDZeNyTBHQLMMSAEWSIHh2YjDAjPUMzPZNAE2UHJ2hUcTBIMygxaADjw5zBgcGDIYAhFGh+F8MGhp0Mu5i0mQKZIpiiIEqZGKF6hBlQAFMyAOxdnYo=</latexit>
(
T = 1
T = 0
Path
Query
Not Blue (= Red)

https://www.kaggle.com/kaggle-survey-2021
State of Data Science and
Machine Learning 2021
The 2021 Kaggle DS & ML Survey received 25,973 usable responses from participants in 171 different countries and territories.
Q17. Which of the following ML algorithms do you use on a regular basis? (Select all that apply)

Object recognition
Game play
J’aime la
musique I love music
Speech recognition
Machine translation
Super resolution

p1 p2 p3 p4
Random Forest
Gaussian Process
Logistic Regression

Breiman L, Statistical Modeling: The Two Cultures. Statist. Sci. 16(3): 199-231, 2001.
https://doi.org/10.1214/ss/1009213726
Rashomon
Occam
Bellman

<latexit sha1_base64="mpuRLmJhhTRKlPt+QsNIZJRw5HU=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkN+IiVuxeDoeSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVLKQnA==</latexit>
x1
<latexit sha1_base64="C1ohMOVeKO2tnLKPo8dfEeHHC5o=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDQTGuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmM9jzAxOTU94531zc0vLPoDS8t5S2+aMs/JuqqbRUm0uKpoPGcrtsqLhsnFhqTyglTfH+wXWty0FF07tNsGLzfEmqZUFVm0icqeVmKVQIhFmBPBURB1QQhupPTAI45wDB0ymmiAQ4NNWIUIi74SomAwiCujQ5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPalqOWqZTVBomKYMIsxd2z/rsmT2wV/b5Z62OU2PgpU2zNNRyo+K/WM1+/Ktq0Gzj5Fs11rONKnYcrwp5NxxmcAt5qG+ddfvZ3Uy4s8Fu2Rv5v2E99kQ30Frv8l2aZ67H+JHIC70YNSj6ux2jIB+LRLcj8XQ8lNxzW+XFGtaxSf1IIIkDpJCj+jVc4gpdwStEhC0hMUwVPK5mBT9CSH4BVtKQnQ==</latexit>
x2 <latexit sha1_base64="mpuRLmJhhTRKlPt+QsNIZJRw5HU=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkN+IiVuxeDoeSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVLKQnA==</latexit>
x1
x1
x2
x1
x1
x1
x2
N = 32 = 9 N = 52 = 25 N = 102 = 100
x1
x2
<latexit sha1_base64="ms/uPGAGeTg0lt5op+vO01P7rjg=">AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYq2SwFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkC2GOP+Q==</latexit>
y
<latexit sha1_base64="WQjRG4bPVEcrBhk0IPMU7zTCz9E=">AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS4FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCsAOP5g==</latexit>
f
<latexit sha1_base64="WQjRG4bPVEcrBhk0IPMU7zTCz9E=">AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS4FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCsAOP5g==</latexit>
f
y
x1
x2
y <latexit sha1_base64="ms/uPGAGeTg0lt5op+vO01P7rjg=">AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYq2SwFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkC2GOP+Q==</latexit>
y
y

Bronstein MM, Bruna J, Cohen T, Veličković P.
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.
arXiv [cs.LG]. 2021. http://arxiv.org/abs/2104.13478
for all
<latexit sha1_base64="NqBYbofjRLhsRuy5gs3o5FfAjVo=">AAAConichVG7SgNBFD1Z3/EVtRFECEbFKkwkqKQK2ohVHkaFJIbddRKX7IvdSUCDlZ0/YGGlYCF2WvgBNv6ART5BLBVsLLzZLAQN6l1m58yZe+6cmavYuuYKxpoBqae3r39gcCg4PDI6Nh6amNxxrZqj8pxq6Zazp8gu1zWT54QmdL5nO1w2FJ3vKtWN1v5unTuuZpnb4sjmRUOumFpZU2VBVCk0W04UDFkcKkojc7J/EC4IK9whSqEIizIvwt0g5oMI/EhZoQcUcAALKmowwGFCENYhw6UvjxgYbOKKaBDnENK8fY4TBElboyxOGTKxVfpXaJX3WZPWrZqup1bpFJ2GQ8owFtgzu2Fv7Indshf2+Wuthlej5eWIZqWt5XZp/Gw6+/GvyqBZ4LCj+tOzQBlrnleNvNse07qF2tbXj8/fsonMQmORXbFX8n/JmuyRbmDW39XrNM9c/OFHIS/0YtSg2M92dIOd5WhsJRpPxyPJdb9Vg5jBHJaoH6tIYhMp5Kj+KW5wh3tpXtqS0lK2nSoFfM0UvoVU+AKNf5vS</latexit>
f : Rd
! R
<latexit sha1_base64="bLp7Jqe5Ke+bRuLIs0xt/zwFMHo=">AAACmnichVHLLgRBFD3T3u9BJBIWExOPhUxqRBCrCRtiM8a8EsOkuxUq+pXumgk68wN+wMKKxAIf4ANs/ICFTxBLEhsLd3o6EQS3U12nTt1z61RdzTGEJxl7jChNzS2tbe0dnV3dPb190f6BvGdXXJ3ndNuw3aKmetwQFs9JIQ1edFyumprBC9rBcn2/UOWuJ2wrK48cvmWqe5bYFboqiSpHhw6nDydjJWHFSqYq9zXNz9S2d8rROEuwIGI/QTIEcYSRtqO3KGEHNnRUYILDgiRsQIVH3yaSYHCI24JPnEtIBPscNXSStkJZnDJUYg/ov0erzZC1aF2v6QVqnU4xaLikjGGcPbAr9sLu2Q17Yu+/1vKDGnUvRzRrDS13yn0nwxtv/6pMmiX2P1V/epbYxULgVZB3J2Dqt9Ab+urx6cvGYmbcn2AX7Jn8n7NHdkc3sKqv+uU6z5z94UcjL/Ri1KDk93b8BPmZRHIuMbs+G08tha1qxwjGMEX9mEcKK0gjR/V9nOMaN8qosqSsKmuNVCUSagbxJZTsB6MZmAU=</latexit>
x, x0
2 Rd
-Lipschitz
<latexit sha1_base64="VAwN9BEfNuoROOZ0f78ZUuUItlA=">AAACqHichVFNLwNRFD0d39/FRmIz0fhM2ryKIFYNGwuL+qgSFZkZr0y8zoyZ16a0fgB/wMKKxEIsLLG28Qcs/ASxJLGxcDudRBDcybx73nn33Dlvru4I05OMPYaUmtq6+obGpuaW1rb2jnBn17Jn512Dpwxb2O6KrnlcmBZPSVMKvuK4XMvpgqf1nZnKebrAXc+0rSW55/D1nLZlmVnT0CRRG+FIOTtUHI7SMjhcVjOC73pCs6Q6p2bKajFaHKRMVSzG/FB/gngAIggiaYevkcEmbBjIIwcOC5KwgAaPnjXEweAQt44ScS4h0z/nOEAzafNUxalCI3aH1i3arQWsRftKT89XG/QVQa9LShX97IFdsBd2zy7ZE3v/tVfJ71HxskdZr2q5s9Fx1LP49q8qR1li+1P1p2eJLCZ9ryZ5d3ymcgujqi/sH78sTi30lwbYGXsm/6fskd3RDazCq3E+zxdO/vCjkxf6YzSg+Pdx/ATLo7H4eGxsfiySmA5G1Yhe9GGI5jGBBGaRRIr6H+IKN7hVRpSkklZWq6VKKNB040so+gci3ZxP</latexit>
|f(x) f(x0
)| 6 Lkx x0
k
<latexit sha1_base64="+ZiSkRyCDmMX1dfLuFjrVp2j6hw=">AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdpYWJjERCGK7K6TOLjZXXYngRj8AW0VCysFC/ED/AAbf8AinyCWEWwsvNksiAb1LrNz5sw9d87M1WxDuJKxekDp6OwKdod6evv6BwaHwsMjOdcqOzrP6pZhOTua6nJDmDwrhTT4ju1wtaQZfFs7Wmvub1e44wrL3JJVm++V1KIpCkJXJVGpjf1wlMWYF5F2EPdBFH5sWuEH7OIAFnSUUQKHCUnYgAqXvjziYLCJ20ONOIeQ8PY5TtBL2jJlccpQiT2if5FWeZ81ad2s6XpqnU4xaDikjGCSPbM71mBP7J69sI9fa9W8Gk0vVZq1lpbb+0OnY5n3f1UlmiUOv1R/epYoYMnzKsi77THNW+gtfeX4spFZTk/WptgNeyX/16zOHukGZuVNv03x9NUffjTyQi9GDYr/bEc7yM3G4guxRCoRTa76rQphHBOYpn4sIol1bCJL9TnOcI4LJajMKHPKfCtVCfiaUXwLZeUTeMOPzA==</latexit>
L
Donoho DL,
High-dimensional data analysis: The curses and blessings of dimensionality.
Plenary Lecture, AMS National Meeting on Mathematical Challenges of the 21st Century. 2000.

<latexit sha1_base64="mchf2gvio17p/Fz9iF84xhnMH3c=">AAAC73ichVHNThRBEK4dVFZAWeBi4mXiBrJ72dQSIsTEhOhBj/ztQkLjpqenZ2mZv53p3WSdzAvwAhxMjJB4ID6AD+DFoxcOPILhCMGLidbOTiJKxJ7M9Fdf1VfzdZcVuirWiKcFY+TW7Tujxbtj4xP37k+WpqabcdCNhGyIwA2iLYvH0lW+bGilXbkVRpJ7lis3rb3ng/xmT0axCvwN3Q/ljsfbvnKU4JqoVul1s2JXzacmcyIuEhaqV8kQ2mkyn6bp1YgJO9DsBfc8zlzp6MrVXKTau7qamkwHJpqs0+W2WbGzkCnf0f1qq1TGGmbLvA7qOShDvlaC0idgYEMAArrggQQfNGEXOMT0bEMdEELidiAhLiKksryEFMZI26UqSRWc2D36tinazlmf4kHPOFML+otLb0RKE2bxBI/xHL/gR/yGP/7ZK8l6DLz0abeGWhm2JvcfrH//r8qjXcPub9WNnjU4sJR5VeQ9zJjBKcRQ33tzcL7+ZG02mcMjPCP/h3iKn+kEfu9CfFiVa29v8GORF7oxGlD973FcB835Wv1xbWF1obz8LB9VER7CI6jQPBZhGV7CCjSo/1e4hJ8FMDrGgfHOeD8sNQq5Zgb+WMbxL1/dugg=</latexit>
V (d) =
⇡
d
2
d
2 · d
2
! 0 (d ! 1)
https://www.math.ucdavis.edu/~strohmer/courses/180BigData/180lecture1.pdf
<latexit sha1_base64="eWTwm+uSU1d/PxikW7VncIaHMRY=">AAAChHichVHLSsNAFD2Nr/ps1Y3gRiwVF1JutD5wIUU3LltrVdAiSZzW0DQJSVqoxR/QreLClYIL8QP8ADf+gAs/QVxWcOPC2zQgWtQbJnPmzD13zsxVbUN3PaLnkNTR2dXdE+7t6x8YHIpEh0e2XKviaCKnWYbl7KiKKwzdFDlP9wyxYztCKauG2FZLa8397apwXN0yN72aLfJlpWjqBV1TPKYy8n40RgnyY6IdyAGIIYi0Fb3HHg5gQUMFZQiY8BgbUODytwsZBJu5POrMOYx0f1/gGH2srXCW4AyF2RL/i7zaDViT182arq/W+BSDh8PKCcTpiW6pQY90Ry/08Wutul+j6aXGs9rSCns/cjKWff9XVebZw+GX6k/PHgpY8r3q7N32meYttJa+enTRyC5vxOtTdE2v7P+KnumBb2BW37SbjNi4/MOPyl74xbhB8s92tIOt2YS8kEhmkrHUatCqMMYxiWnuxyJSWEcaOa4vcIoznEvd0ow0J823UqVQoBnFt5BWPgE/Y4+x</latexit>
1
<latexit sha1_base64="z3Ba/qsiF6tEgbjc9DFSGYR9udI=">AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQYZhFEUhuX/uQPqMjMeLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWnrDlFlG1lXdzEuixVRFYxmucJXlDZOJdUllOal20N/PNZlpKbp2yFsGK9XFqqYcKbLIiSpgaLNonZi8XemUfQEMoR3+YRB2QACcSOi+RyhCBXSQoQF1YKABJ6yCCBZ9BQgDgkFcCdrEmYQUe59BBzykbVAWowyR2Br9q7QqOKxG635Ny1bLdIpKwySlH4L4gvfYw2d8wFf8/LNW267R99KiWRpomVH2ni2nP/5V1WnmcPytGumZwxFs214V8m7YTP8W8kDfPL3opXdSwfYa3uIb+b/BLj7RDbTmu3yXZKnLEX4k8kIvRg0K/27HMMhuhMLRUCQZCcT2nVa5YQVWYZ36sQUxiEMCMvaLnsMVXAs+ISrsCnuDVMHlaJbgRwjxLwzik/k=</latexit>
0.5
p
d
<latexit sha1_base64="rzd8+rC/ag8/gv3jUp4Y6gUrSiI=">AAAChnichVHLSsNAFD3GV62PVt0IbopFcRVupVVxVXTjsg9rC1okidMamiYhSQta/AHBrV24UnAhfoAf4MYfcOEniEsFNy68TQOixXrDZM6cuefOmbmqbeiuR/Q8IA0ODY+MhsbC4xOTU5Ho9MyuazUcTRQ0y7Cckqq4wtBNUfB0zxAl2xFKXTVEUa1tdfaLTeG4umXueMe2KNeVqqlXdE3xmMqTnDqIxkkmP2K9IBGAOILIWNF77OMQFjQ0UIeACY+xAQUuf3tIgGAzV0aLOYeR7u8LnCLM2gZnCc5QmK3xv8qrvYA1ed2p6fpqjU8xeDisjGGRnuiW3uiR7uiFPv+s1fJrdLwc86x2tcI+iJzN5T/+VdV59nD0rerr2UMF675Xnb3bPtO5hdbVN0/ab/mN3GJria7plf1f0TM98A3M5rt2kxW5yz5+VPbCL8YNSvxuRy/YXZETq3Iym4ynN4NWhTCPBSxzP9aQxjYyKHD9Ks5xgbYUkmQpJa11U6WBQDOLHyGlvwBbQpAn</latexit>
0.5
<latexit sha1_base64="Nn/M5/H5iD9Z/hOTHxuHUqAZH+I=">AAAChnichVHLSsNAFD2Nr1ofrboR3BRLxVWZlmpFEIpuXPZhH1BLSdKxhqZJSNJCLf6A4NYuXCm4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzJUNVLJuxnkeYmJyanvHO+ubmFxb9gaXlvKU3TZnnZF3VzaIkWlxVNJ6zFVvlRcPkYkNSeUGqHwz2Cy1uWoquHdltg5cbYk1TThRZtInKVvdilUCIRZgTwVEQdUEIbqT0wCOOUYUOGU00wKHBJqxChEVfCVEwGMSV0SHOJKQ4+xzn8JG2SVmcMkRi6/Sv0arkshqtBzUtRy3TKSoNk5RBhNkLu2d99swe2Cv7/LNWx6kx8NKmWRpquVHxX6xmP/5VNWi2cfqtGuvZxgl2HK8KeTccZnALeahvnXX72d1MuLPBbtkb+b9hPfZEN9Ba7/Jdmmeux/iRyAu9GDUo+rsdoyAfi0S3I/F0PJTcd1vlxRrWsUn9SCCJQ6SQo/o1XOIKXcErRIQtITFMFTyuZgU/Qkh+AeO5kGc=</latexit>
d = 2
<latexit sha1_base64="LVcqe4i4MA2A5XBAmTvFXnqLayw=">AAAChnichVHLSsNAFD2Nr1ofrboR3BRLxVWZSrUiCEU3LvuwD6ilJOm0hqZJSNJCLf6A4NYuXCm4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzJUNVLJuxnkeYmJyanvHO+ubmFxb9gaXlnKU3TZlnZV3VzYIkWlxVNJ61FVvlBcPkYkNSeV6qHw728y1uWoquHdttg5caYk1Tqoos2kRlKvuxciDEIsyJ4CiIuiAEN5J64BEnqECHjCYa4NBgE1YhwqKviCgYDOJK6BBnElKcfY5z+EjbpCxOGSKxdfrXaFV0WY3Wg5qWo5bpFJWGScogwuyF3bM+e2YP7JV9/lmr49QYeGnTLA213Cj7L1YzH/+qGjTbOP1WjfVso4pdx6tC3g2HGdxCHupbZ91+Zi8d7mywW/ZG/m9Yjz3RDbTWu3yX4unrMX4k8kIvRg2K/m7HKMhtRaI7kVgqFkocuK3yYg3r2KR+xJHAEZLIUv0aLnGFruAVIsK2EB+mCh5Xs4IfISS+AOf5kGk=</latexit>
d = 4
1
<latexit sha1_base64="UR9ymFZxc/Q/TCzOORzighhjAlM=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDQTEuDNGNSx7ySJCQtgzYUNqmLSRI/AETt7JwpYkL4wf4AW78ARd8gnGJiRsXXkoTo0S8zXTOnLnnzpm5kqEqls1YzyNMTE5Nz3hnfXPzC4v+wNJyztKbpsyzsq7qZkESLa4qGs/aiq3ygmFysSGpPC/VDwf7+RY3LUXXju22wUsNsaYpVUUWbaIylf1YORBiEeZEcBREXRCCG0k98IgTVKBDRhMNcGiwCasQYdFXRBQMBnEldIgzCSnOPsc5fKRtUhanDJHYOv1rtCq6rEbrQU3LUct0ikrDJGUQYfbC7lmfPbMH9so+/6zVcWoMvLRploZabpT9F6uZj39VDZptnH6rxnq2UcWu41Uh74bDDG4hD/Wts24/s5cOdzbYLXsj/zesx57oBlrrXb5L8fT1GD8SeaEXowZFf7djFOS2ItGdSCwVCyUO3FZ5sYZ1bFI/4kjgCElkqX4Nl7hCV/AKEWFbiA9TBY+rWcGPEBJf6hqQag==</latexit>
d > 4
<latexit sha1_base64="nHSiJ7YlA7Y9nChVeD/tJhyz2yo=">AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQoZhFEUhuX/uQPqMjMdLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWXrDlFlW1lXdLEiixVRFY1mucJUVDJOJdUlleam239/PN5lpKbp2wFsGK9fFqqYcKbLIiSpiaKNknZi8HelUfAEMoR3+YRB2QACcSOq+RyjBIeggQwPqwEADTlgFESz6ihAGBIO4MrSJMwkp9j6DDnhI26AsRhkisTX6V2lVdFiN1v2alq2W6RSVhklKPwTxBe+xh8/4gK/4+Wettl2j76VFszTQMqPiPVvOfPyrqtPM4fhbNdIzhyPYsr0q5N2wmf4t5IG+eXrRy2yng+01vMU38n+DXXyiG2jNd/kuxdKXI/xI5IVejBoU/t2OYZCLhMKxUDQVDcT3nFa5YQVWYZ36sQlxSEASsvaLnsMVXAs+ISbsCLuDVMHlaJbgRwiJL6Jhk8c=</latexit>
0.5
p
2
<latexit sha1_base64="z3Ba/qsiF6tEgbjc9DFSGYR9udI=">AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQYZhFEUhuX/uQPqMjMeLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWnrDlFlG1lXdzEuixVRFYxmucJXlDZOJdUllOal20N/PNZlpKbp2yFsGK9XFqqYcKbLIiSpgaLNonZi8XemUfQEMoR3+YRB2QACcSOi+RyhCBXSQoQF1YKABJ6yCCBZ9BQgDgkFcCdrEmYQUe59BBzykbVAWowyR2Br9q7QqOKxG635Ny1bLdIpKwySlH4L4gvfYw2d8wFf8/LNW267R99KiWRpomVH2ni2nP/5V1WnmcPytGumZwxFs214V8m7YTP8W8kDfPL3opXdSwfYa3uIb+b/BLj7RDbTmu3yXZKnLEX4k8kIvRg0K/27HMMhuhMLRUCQZCcT2nVa5YQVWYZ36sQUxiEMCMvaLnsMVXAs+ISrsCnuDVMHlaJbgRwjxLwzik/k=</latexit>
0.5
p
d
!?
Unit Cube in
Unit Ball
1
<latexit sha1_base64="PtefnFPcLJximPXvkayt2U9xPRk=">AAAChXichVHLSsNAFD2Nr1pfVTeCm2KpuLHcSFFxoUU3LvuwWqhFkjjVYJqEJC1o8QfErbpwpeBC/AA/wI0/4KKfIC4ruHHhbRoQFfWGyZw5c8+dM3NV29Bdj6gZkrq6e3r7wv2RgcGh4ZHo6Nima9UcTRQ0y7Ccoqq4wtBNUfB0zxBF2xFKVTXElnqw1t7fqgvH1S1zwzu0Rbmq7Jl6RdcUj6ncsrwTjVOS/Ij9BHIA4ggiY0XvsY1dWNBQQxUCJjzGBhS4/JUgg2AzV0aDOYeR7u8LHCPC2hpnCc5QmD3g/x6vSgFr8rpd0/XVGp9i8HBYGUOCnuiWWvRId/RM77/Wavg12l4OeVY7WmHvjJxM5N/+VVV59rD/qfrTs4cKFn2vOnu3faZ9C62jrx9dtPJLuURjmq7phf1fUZMe+AZm/VW7yYrc5R9+VPbCL8YNkr+34yfYnEvK88lUNhVPrwatCmMSU5jhfiwgjXVkUOD6FZziDOdSnzQrpaT5TqoUCjTj+BLSygfo4Y/5</latexit>
> 1
!?

Balestriero R, Pesenti J, LeCun Y.
Learning in High Dimension Always Amounts to Extrapolation.
arXiv [cs.LG]. 2021. http://arxiv.org/abs/2110.09485 https://youtu.be/86ib0sfdFtw
• "on any high-dimensional (>100) dataset,
interpolation almost surely never happens."
• "Those results challenge the validity of our
current interpolation/extrapolation deﬁnition as
an indicator of generalization performances. "

!
Fan J, Han F, Liu H.
Challenges of Big Data Analysis. Natl Sci Rev. 2014;1: 293–314.
<latexit sha1_base64="hsRBbjUZq2fjH2VINZUiaiPtLiw=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIaETEG+rEpOSXFOtExKdoxhRn5ir4xadoGOh4asYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IXXzSUMaQwWYLdmAt1eABYB+SIZor+savrnYKsg1Wo1g0UGr4HuX2hw0+Aw0Ad5ZV+SlwamBs3G454koFuAIQaMIEP06MBkhBnpGZrpmQSaKDs4QaOKg0GaQYlBAxgf5gwODB4MAQyhQPObGFYwbGTYxKTM5MkUwBQEUcrECNUjzIACmKIBAhaZTQ==</latexit>
(X1, . . . , Xd) ⇠ Nd(0, I)
<latexit sha1_base64="Ay3UPGyVpdQRyf/zf3IimzcaXzc=">AAACvnichVFNSxtBGH6y9atp1VgvBS9Lg0VBwqxIbQtCaC89amw0YGSZXSfJ6n51dhK0m/yB/oEeeigVeij+AH+ACJ7aUw/+BPFooZce+u5mQVTUd5mdZ555n3eemdcKXSdSjJ3mtAcDg0PDIw/zjx6Pjo0XJp6sRUFb2qJqB24gaxaPhOv4oqoc5YpaKAX3LFesWztvk/31jpCRE/jv1V4oNj3e9J2GY3NFlFl4VW9xFcuevqTXPb5rxtv1pvgQudxX+nxP7yasakkvtgMpezM105jTa+b2rN41C0VWYmnoN4GRgSKyWA4Kh6hjCwFstOFBwIci7IIjom8DBhhC4jYREycJOem+QA950rYpS1AGJ3aH/k1abWSsT+ukZpSqbTrFpSFJqWOa/WY/2AU7YQfsjP27tVac1ki87NFs9bUiNMc/PV39e6/Ko1mhdam607NCAy9Trw55D1MmuYXd13c+fr5YfV2Zjp+zfXZO/r+xU3ZEN/A7f+zvK6Ly5Q4/FnmhF6MGGdfbcROszZeMF6WFlYVi+U3WqhFM4RlmqB+LKOMdllGl+l9xjJ/4pZW1huZpQT9Vy2WaSVwJbfc/j5mmYg==</latexit>
r̂ = max
j>2
|corr(X1, Xj)|
<latexit sha1_base64="DII9mhRSnTvcUJHdfuik1ohKbRg=">AAAC8nichVHNSiNBEK6Muv6srlEvgpfBoLggoUeCiiCIXjxqstGAI0PP2Ek6zh89naBO8gK+gKAnlT0su3cfYC97Xzz4CMseXRDBgzU/ICqrNfRU1df1VX/dZfo2DyQhNxmlq7vnQ29f/8DHwaFPw9mR0a3AawqLlS3P9kTFpAGzucvKkkubVXzBqGPabNvcX4v2t1tMBNxzv8hDn+06tObyKreoRMjIOnqdyrDYUZdV3aEHRtgutZcLnTTRTSap0cDUZlXZjlBZF05oeUJ0YmymYmizqh40HSNs6NxVS1icsNSK0dAFr9XlZzXxbSObI3kSm/o60NIgB6lteNkr0GEPPLCgCQ4wcEFibAOFAL8d0ICAj9guhIgJjHi8z6ADA8htYhXDCoroPv5rmO2kqIt51DOI2RaeYuMSyFRhilyTb+SW/CLfyR/y8N9eYdwj0nKI3ky4zDeGj8dLd++yHPQS6k+sNzVLqMJirJWjdj9GoltYCb91dHJbWipOhdPkgvxF/efkhvzEG7itf9bXTVY8e0OPiVrwxXBA2stxvA625vLafL6wWcitrKaj6oMJmIQZnMcCrMA6bEAZ+/+G+4yS6VKkcqqcK5dJqZJJOWPwzJQfj/5suoY=</latexit>
R̂ = max
|S|=4
max
j
corr
0
@X1,
X
j2S
jXj
1
A
0.3 0.4 0.5 0.6 0.5 0.6 0.7 0.8
<latexit sha1_base64="dsksfNj+oQDjs49biWmppWLfS4s=">AAACinichVHLTsJAFL3UFyIK6sbEDZFgXJEpIT43RF245CGPBAhp6wANfaUdSLDhB9y5MpGVJi6MH+AHuPEHXPAJxiUmblx4W5oYJeJtpnPmzD13zswVDUW2GCEDHzc1PTM7558PLAQXl0Lh5ZWCpbdNieYlXdHNkihYVJE1mmcyU2jJMKmgigotiq0jZ7/YoaYl69op6xq0qgoNTa7LksCQKlaaArOzvVo4SuLEjcg44D0QBS/SevgRKnAGOkjQBhUoaMAQKyCAhV8ZeCBgIFcFGzkTkezuU+hBALVtzKKYISDbwn8DV2WP1XDt1LRctYSnKDhMVEYgRl7IPRmSZ/JAXsnnn7Vst4bjpYuzONJSoxa6WMt9/KtScWbQ/FZN9MygDruuVxm9Gy7j3EIa6TvnV8PcfjZmb5Jb8ob+b8iAPOENtM67dJeh2f4EPyJ6wRfDBvG/2zEOCok4vx1PZpLR1KHXKj+swwZsYT92IAUnkIa8W/8SrqHPBbkEt8cdjFI5n6dZhR/BHX8B43OSnw==</latexit>
R̂
<latexit sha1_base64="XcYeasdspQvBninjyfi4DGqRhKY=">AAACinichVHLTsJAFL3UFyIK6sbEDZFgXJEpIT43RF245CGPBAhp6wANfaUdSLDhB9y5MpGVJi6MH+AHuPEHXPAJxiUmblx4W5oYJeJtpnPmzD13zswVDUW2GCEDHzc1PTM7558PLAQXl0Lh5ZWCpbdNieYlXdHNkihYVJE1mmcyU2jJMKmgigotiq0jZ7/YoaYl69op6xq0qgoNTa7LksCQKlaaArPNXi0cJXHiRmQc8B6IghdpPfwIFTgDHSRogwoUNGCIFRDAwq8MPBAwkKuCjZyJSHb3KfQggNo2ZlHMEJBt4b+Bq7LHarh2alquWsJTFBwmKiMQIy/kngzJM3kgr+Tzz1q2W8Px0sVZHGmpUQtdrOU+/lWpODNofqsmemZQh13Xq4zeDZdxbiGN9J3zq2FuPxuzN8kteUP/N2RAnvAGWuddusvQbH+CHxG94Ithg/jf7RgHhUSc344nM8lo6tBrlR/WYQO2sB87kIITSEPerX8J19DnglyC2+MORqmcz9Oswo/gjr8AJ6KSvw==</latexit>
r̂

K. Beyer+, When Is “Nearest Neighbor” Meaningful? ICDT’99
V. Pestov, On the geometry of similarity search: dimensionality curse and concentration of measure,
Information Processing Letters, 1999.

Decision
Tree
Random
Forest GBDT
Nearest
Neighbor
Logistic
Regression
SVM
Gaussian
Process
Neural
Network

Journal of the American Statistical Association , Jun., 2006, Vol. 101, No. 474 (Jun., 2006), pp. 578-590
https://www.jstor.org/stable/27590719
"We introduce a concept of potential nearest neighbors (k-PNNs) and show that random
forests can be viewed as adaptively weighted k-PNN methods. "

TheBibleor"TheYellowTerror"
byLuc,Laci,Gábor

byLuc,Laci,Gábor
<latexit sha1_base64="jdzy7O9ZT5s5Y1wkxQhz0FOJMwM=">AAACnHichVHLSsNAFL2Nr1ofrbpQEKRYlAoaboovBEHUhSCKr7aCLSWJ0zqYF0laqMUf8AdcuFJxIeraD3DjD7jwE8SlghsX3qYB0WK9YTJnztxz58xcxdK44yI+B4Sm5pbWtmB7qKOzqzsc6elNOWbRVllSNTXT3lVkh2ncYEmXuxrbtWwm64rG0srhUnU/XWK2w01jxy1bLKvLBYPnuSq7ROUi/evxjF6cn5DExHg04/CCLs8nxKmxXCSGInoRrQeSD2Lgx4YZuYcM7IMJKhRBBwYGuIQ1kMGhbw8kQLCIy0KFOJsQ9/YZHEOItEXKYpQhE3tI/wKt9nzWoHW1puOpVTpFo2GTMgoj+ITX+IaPeIMv+PlnrYpXo+qlTLNS0zIrFz4Z2P74V6XT7MLBt6qhZxfyMOt55eTd8pjqLdSavnR0+rY9tzVSGcULfCX/5/iMD3QDo/SuXm2yrbMGfhTyQi9GDZJ+t6MepBKiNC1Obk7GFhb9VgVhEIYhTv2YgQVYgQ1Ier4u4RbuhCFhWVgV1mqpQsDX9MGPEFJfeQKW8A==</latexit>
N(µ = 1.2, = 2.5)

Given , a real i.i.d. sequence, estimate
byLuc,Laci,Gábor
NeurIPS 2021 Invited Talk (Breiman Lecture)
Do we know how to estimate the mean?
<latexit sha1_base64="3NbZouAktoxRn4pfXQgJR8+DKAA=">AAACnHichVG7SgNREJ2s7/hI1EJBkGBQLCTMRo3GStRCECE+EgNJCLubm7i4L3ZvAhr8AX/AwkrFQtTaD7DxByzyCWKpYGPh7GZFLBJnuXtnzsyZe+4d2dJUhyM2AkJHZ1d3T29fsH9gcCgUHh7JOGbVVlhaMTXTzsqSwzTVYGmuco1lLZtJuqyxA/lo3c0f1JjtqKaxz48tVtCliqGWVUXiBBXDY3lZr2dPi+JcvmRyZ84PKRPFGCYWk/MYwdgiikvJJDmIieX5eEQkx7Uo+JYyw4+QhxKYoEAVdGBgACdfAwkc+nIgAoJFWAHqhNnkqV6ewSkEiVulKkYVEqFH9K9QlPNRg2K3p+OxFTpFo2UTMwLT+IK3+I7PeIev+NWyV93r4Wo5pl1ucplVDJ2N733+y9Jp53D4y2qrmUMZlj2tKmm3PMS9hdLk107O3/dWdqfrM3iFb6T/Ehv4RDcwah/KzQ7bvWijRyYt9GI0oJ8pRFo7mXhMTMQWdhaiq2v+qHphAqZgluaxBKuwCSlIe7qu4R4ehElhQ9gStpulQsDnjMIfEzLfLdyZlw==</latexit>
X1, . . . , Xn
<latexit sha1_base64="luqROJ+77OljeLhhMh6h7Ajj7RY=">AAACo3ichVFNSxtRFD1O/aq2JtqNUJDBkOIq3PEjmkJBLAXBjUajARPCvPGpg/PFzEvADtl15R/owlULXbQuW/AHuPEPuPAniEsFNy56ZxIpLrR3ePPOO++e+857VwSOHSmiyx7tRW9f/8Dgy6HhV69HMtnRsc3Ib4aWrFi+44dVYUbSsT1ZUbZyZDUIpekKR26Jg4/J/lZLhpHtexvqMJB119zz7F3bMhVTjexETbhxzW229Q96zTXVvhDxp7aesNV2w2hkc1Sg4lxphnQqzJExXyoxICouzEzrBoMkcujGqp89RQ078GGhCRcSHhRjByYi/rZhgBAwV0fMXMjITvcl2hhibZOzJGeYzB7wf49X213W43VSM0rVFp/i8AhZqSNPF/STbuicTuiK7p+sFac1Ei+HPIuOVgaNzNH4+t1/VS7PCvv/VM96VtjFQurVZu9ByiS3sDr61uevN+vvy/n4HX2na/b/jS7pjG/gtW6tH2uyfPyMH8Fe+MW4QQ9d0J8Gm9MFo1iYXZvNLS51WzWIt5jEFPdjHotYxioqXP8LfuE3/mh5bUUraxudVK2nq3mDR6HV/wJh+5ws</latexit>
µ = EX1
<latexit sha1_base64="ve2fKPQE43VaTVIusM8fwhBZiOg=">AAACqHichVG7ThtBFD1sSHjkgZM0SDQrLBBJ4dzlYTASEgpNSvMwdoSJtbsZYMS+tDu2MCt/APwARapEokApKIGaJj9AwScgSpBoKHJ37ShKAbmr2Tlz5p47Z+ZagSMjRXTZpT3pfvqsp7ev//mLl68GMq/frEZ+PbRFyfYdP6xYZiQc6YmSksoRlSAUpms5omxtLyT75YYII+l7K6oZiHXX3PTkhrRNxVQtk60qsaMi1XTEWDWqu7VYzhmtL55etdy40qrJdx+SLMpRfqowQTrlpsiYLhQYEOVnJsZ1g0ESWXSi6GdOUMVX+LBRhwsBD4qxAxMRf2swQAiYW0fMXMhIpvsCLfSzts5ZgjNMZrf5v8mrtQ7r8TqpGaVqm09xeISs1DFCF3REN/SLftIV3T9YK05rJF6aPFttrQhqA/uDy3f/Vbk8K2z9VT3qWWEDM6lXyd6DlEluYbf1jd2Dm+XZpZF4lH7QNfv/Tpd0zjfwGrf24aJY+vaIH4u98Itxg/50QX8YrI7njHxucnEyO/+x06peDGEYY9yPaczjE4oocf09HOMUZ9p7raiVtc/tVK2ro3mLf0KzfgMt4J6c</latexit>
(
Pn
i=1 Xi)/n

Journal of Machine Learning Research, 9, 2015-2033, 2008 The Annals of Statistics, 43(4), 1716-1741, 2015.

Breiman L, Statistical Modeling: The Two Cultures. Statist. Sci. 16(3): 199-231, 2001.
https://doi.org/10.1214/ss/1009213726
• "What I call the Rashomon Effect is that there is often a multitude of different descriptions
[equations f︎(x)] in a class of functions giving about the same minimum error rate. The most
easily understood example is subset selection in linear regression."
• "The Rashomon Effect also occurs with decision trees and neural nets."
• "This effect is closely connected to what I call instability (Breiman,1996a) that occurs when
there are many different models crowded together that have about the same training or test
set error."
Rashomon Effect:

Kernel Ridge (RBF)
Neural Network (MLP)
Gradient Boosted Trees
SVR (RBF) Gaussian Process (RBF)
Random Forest
Nearest Neighbors Decision Tree

https://arxiv.org/abs/2011.03395
https://ai.googleblog.com/2021/10/
how-underspecification-presents.html
"While ML models are validated on held-out data, this validation is often insufficient to
guarantee that the models will have well-defined behavior when they are used in a new setting. "

Kernel Ridge (RBF)
Neural Network (MLP)
SVR (RBF) Gaussian Process (RBF)
Random Forest
Benign？
Benign？ Benign？
Malignant Malignant

Hutter et al, Algorithm Runtime Prediction: Methods & Evaluation (2014) https://arxiv.org/abs/1211.0906

https://scikit-optimize.github.io/stable/_modules/skopt/learning/forest.html
def _return_std(X, trees, predictions, min_variance):
# This derives std(y | x) as described in 4.3.2 of arXiv:1211.0906
std = np.zeros(len(X))
for tree in trees:
var_tree = tree.tree_.impurity[tree.apply(X)]
# This rounding off is done in accordance with the
# adjustment done in section 4.3.3
# of http://arxiv.org/pdf/1211.0906v2.pdf to account
# for cases such as leaves with 1 sample in which there
# is zero variance.
var_tree[var_tree < min_variance] = min_variance
mean_tree = tree.predict(X)
std += var_tree + mean_tree ** 2
std /= len(trees)
std -= predictions ** 2.0
std[std < 0.0] = 0.0
std = std ** 0.5
return std
Hutter et al, Algorithm Runtime Prediction: Methods & Evaluation (2014) https://arxiv.org/abs/1211.0906

RandomForestRegressor
(n_estimators=10, max_leaf_nodes=12)
ExtraTreesRegressor
(n_estimators=10, max_leaf_nodes=32)
GradientBoostingRegressor
(n_estimators=30, learning_rate=0.1)

GradientBoostingRegressor(loss='quantile', alpha=α)

GradientBoostingRegressor(
(n_estimators=30, learning_rate=0.1)
LGBMRegressor (n_estimators=30, learning_rate=0.1,
max_leaf_nodes=8, min_child_samples=5)
aka "Pinball loss"

Rando Forest ExtraTrees (w/o bootstrap)
ExtraTrees (w/ bootstrap)
Rando Forest ExtraTrees (w/o bootstrap)
ExtraTrees (w/ bootstrap)

Gaussian Process Gaussian Process

MLP
(Quantile Regression)
MLP
(Quantile Regression)
MLP
(MC Dropout)
MLP
(MC Dropout)
arch:
(Linear(1, 100), ReLU,
Dropout(p),
Linear(100, 100), ReLU,
Dropout(p),
Linear(100, 1))
p=0.25
#sampling=30
p=0.25
#sampling=30
p=0.0
p=0.0

MLP
(30 Ensemble)
MLP
(30 Ensemble)
p=0.0
p=0.0
MLP
(30 Ensemble)
MLP
(30 Ensemble)
p=0.25
p=0.25
arch:
(Linear(1, 100), ReLU,
Dropout(p),
Linear(100, 100), ReLU,
Dropout(p),
Linear(100, 1))

• Tolstikhin IO, Houlsby N, Kolesnikov A, Beyer L, Zhai X, Unterthiner T, et al.
MLP-Mixer: An all-MLP Architecture for Vision. NeurIPS 2021.
• Kadra A, Lindauer M, Hutter F, Grabocka J.
Well-Tuned Simple Nets Excel on Tabular Datasets. NeurIPS 2021.
• Liu H, Dai Z, So D, Le Q.
Pay Attention to MLPs. NeurIPS 2021.
• Melas-Kyriazi L.
Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly
Well on ImageNet. 2021. http://arxiv.org/abs/2105.02723

PolyReg(1)
RMSE 0.299
PolyReg(3)
RMSE 0.28
PolyReg(5)
RMSE 0.225
PolyReg(7)
RMSE 0.113
PolyReg(10)
RMSE 0.0189
PolyReg(15)
RMSE 0.00737
PolyReg(20)
RMSE 0.000
PolyReg(30)
RMSE 0.000
ExtraTrees (no bootstrap)
RMSE 0.000
ExtraTrees (bootstrap)
RMSE 0.0121
Random Forest
RMSE 0.012
LGBM
RMSE 0.0508
95%-CI 95%-CI 95%-CI 95%-CI
Problematic overﬁtting by polynomial regression of order k
clearly overﬁtted but harmless (still informative)
also we can assess
the uncertainty

ExtraTrees (no bootstrap) ExtraTrees (no bootstrap) ExtraTrees (no bootstrap) ExtraTrees (no bootstrap)
Gradient Boosted Trees Gradient Boosted Trees Gradient Boosted Trees Gradient Boosted Trees

Nearest Neighbor (k=1) Nearest Neighbor (k=1) Nearest Neighbor (k=1) Nearest Neighbor (k=1)
Decision Tree Decision Tree Decision Tree Decision Tree

Random Forest Random Forest Random Forest Random Forest
Nearest Neighbor (k=3) Nearest Neighbor (k=3) Nearest Neighbor (k=3) Nearest Neighbor (k=3)

"our experiments establish that state-of-the-art convolutional networks for image classiﬁcation trained with
stochastic gradient methods easily ﬁt a random labeling of the training data. This phenomenon is qualitatively
unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured
random noise. "
Berner J, Grohs P, Kutyniok G, Petersen P.
The Modern Mathematics of Deep Learning. arXiv [cs.LG]. 2021. http://arxiv.org/abs/2105.04026
Zhang C, Bengio S, Hardt M, Recht B, Vinyals O.
Understanding Deep Learning (Still) Requires Rethinking Generalization. Commun ACM. 2021;64: 107–115.

Berner J, Grohs P, Kutyniok G, Petersen P.
The Modern Mathematics of Deep Learning. arXiv [cs.LG]. 2021. http://arxiv.org/abs/2105.04026

PNAS (2020) Ann. Statist. (2020)
NeurIPS (2018) arXiv (2019)

PolyReg(10)
Train RMSE 0.0189
PolyReg(15)
Train RMSE 0.00737
KernelRidge RBF(γ=1)
Train RMSE 0.103
Train RMSE 0.0139
Train RMSE 0.00726
Train RMSE 0.00726

ExtraTrees (#trees=1, #leaves=8,
bootstrap=off), Train RMSE 0.0189
GBDT w/ 1 ExtraTree (#leaves=8)
Train RMSE 0.29
GBDT w/ 10 ExtraTree (#leaves=8)
Train RMSE 0.17
GBDT w/ 102, ExtraTree (#leaves=8)
Train RMSE 0.00597
GBDT w/ 103, ExtraTree (#leaves=8)
Train RMSE 0.000997

Random Forest (#trees=100)
Train RMSE 0.0109
Train RMSE 0.0274
ExtraTrees (#trees=100, bootstrap)
Train RMSE 0.0109
Train RMSE 0.0243
Train RMSE 0.0112
Train RMSE 0.0114
Train RMSE 0.0099
Train RMSE 0.00977
With Random Fourier Features (100-dim Random Kitchen Sinks)

1-1-1 ReLU MLP
Train RMSE 0.287
1-5-1 ReLU MLP
Train RMSE 0.287
1-10-1 ReLU MLP
Train RMSE 0.0191
1-100-1 ReLU MLP
Train RMSE 0.00816
Train RMSE 0.0139 Train RMSE 0.0066 Train RMSE 0.0121 Train RMSE 0.0139
…
<latexit sha1_base64="OtDdB3Z2MZrUEFM+jHf2FtPg0CM=">AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS0FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCuqOP6w==</latexit>
k
Optimized With L-BFGS
arch:
(Linear(1, k),
ReLU,
Linear(k, 1))

1-1-1 ReLU MLP
Train RMSE 0.289
1-5-1 ReLU MLP
Train RMSE 0.297
1-10-1 ReLU MLP
Train RMSE 0.0237
1-100-1 ReLU MLP
Train RMSE 0.0243
…
<latexit sha1_base64="OtDdB3Z2MZrUEFM+jHf2FtPg0CM=">AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS0FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCuqOP6w==</latexit>
k
Optimized With Adam
arch:
(Linear(1, k),
ReLU,
Linear(k, 1))

k=1 ReLU MLP
Train RMSE 0.305
k=5 ReLU MLP
Train RMSE 0.303
k=10 ReLU MLP
Train RMSE 0.0131
k=100 ReLU MLP
Train RMSE 0.00979
Optimized With Adam
arch:
(Linear(1, k),
ReLU,
Linear(k, k),
ReLU,
Linear(k, 1))
x 9
Train RMSE 0.0443 Train RMSE 0.0124 Train RMSE 0.0811 Train RMSE 0.00864

Daubechies, I., DeVore, R., Foucart, S. et al. Nonlinear Approximation and (Deep) ReLU Networks.
Constr Approx 55, 127–172 (2022). https://doi.org/10.1007/s00365-021-09548-z
Ingrid Daubechies Ronald DeVore

Daubechies, I., DeVore, R., Foucart, S. et al. Nonlinear Approximation and (Deep) ReLU Networks.
Constr Approx 55, 127–172 (2022). https://doi.org/10.1007/s00365-021-09548-z

1-1-1 ReLU MLP
Train RMSE 0.289
1-5-1 ReLU MLP
Train RMSE 0.297
1-10-1 ReLU MLP
Train RMSE 0.0237
1-100-1 ReLU MLP
Train RMSE 0.0243
Random Forest

1-1-1 ReLU MLP
Train RMSE 0.289
1-5-1 ReLU MLP
Train RMSE 0.297
1-10-1 ReLU MLP
Train RMSE 0.0237
1-100-1 ReLU MLP
Train RMSE 0.0243
Train RMSE 0.0243

Geurts, P., Ernst, D. & Wehenkel, L. Extremely randomized trees. Mach Learn 63, 3–42 (2006). https://doi.org/10.1007/s10994-006-6226-1

"A large class of DNs can be written as a composition of max-aﬃne spline operators (MASOs)"

<latexit sha1_base64="hTsNc7Yxq+3zKV9wCSgyaPQAP+M=">AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K6TOGRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqiY1dFZnisSRKlUMjTakmlgLR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvHfwFXZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzOP1WjfXMoQ67rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BTIfkrs=</latexit>
!1
<latexit sha1_base64="Tgm67LizZXft8YrsZzmIczqCO8g=">AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmY1AJFkERLPMwD0hC2F3HOGRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUtI2WpdCCYqiGVZYlm6pMpwXOuErLpkUlTVZpSW7uO/ulNrVsZuhHvGPSmiY1dHbCFIkjVa4aGm1I9UQ9HCVx4kZkFIgeiIIXGSP8CFU4BgMUaIEGFHTgiFWQwMavAiIQMJGrQRc5CxFz9ymcQwC1LcyimCEh28R/A1cVj9Vx7dS0XbWCp6g4LFRGIEZeyD0ZkGfyQF7J55+1um4Nx0sHZ3mopWY9dLGS//hXpeHM4fRbNdYzhxPYcb0y9G66jHMLZahvn10N8qlcrLtObskb+r8hffKEN9Db78pdluZ6Y/zI6AVfDBsk/m7HKCgm4uJWPJlNRtN7Xqv8sAprsIH92IY0HEIGCm4fLqEH10JQ2BRSwu4wVfB5mmX4EcLBFzQ/krw=</latexit>
!2
<latexit sha1_base64="FFDeu0DUpj47hM4y6m8eBm4BZtQ=">AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmTVAJFkERLPMwD0hC2F3HOGRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUtI2WpdCCYqiGVZYlm6pMpwXOuErLpkUlTVZpSW7uO/ulNrVsZuhHvGPSmiY1dHbCFIkjVa4aGm1I9UQ9HCVx4kZkFIgeiIIXGSP8CFU4BgMUaIEGFHTgiFWQwMavAiIQMJGrQRc5CxFz9ymcQwC1LcyimCEh28R/A1cVj9Vx7dS0XbWCp6g4LFRGIEZeyD0ZkGfyQF7J55+1um4Nx0sHZ3mopWY9dLGS//hXpeHM4fRbNdYzhxPYcb0y9G66jHMLZahvn10N8qlcrLtObskb+r8hffKEN9Db78pdluZ6Y/zI6AVfDBsk/m7HKChuxsWteDKbjKb3vFb5YRXWYAP7sQ1pOIQMFNw+XEIProWgkBBSwu4wVfB5mmX4EcLBFzZfkr0=</latexit>
!3
<latexit sha1_base64="sVWa2gbxZQIcX+Uuo5Pa/ZykHGo=">AAAChnichVHLSsNAFD2Nr1ofrboR3BRLxVWZarUiCEU3LvuwD6ilJHFaQ9MkJGmhFn9AcGsXrhRciB/gB7jxB1z0E8RlBTcuvE0DosV6w2TOnLnnzpm5kqEqls1Y1yOMjU9MTnmnfTOzc/P+wMJiztIbpsyzsq7qZkESLa4qGs/aiq3ygmFysS6pPC/VDvr7+SY3LUXXjuyWwUt1saopFUUWbaIy6b3NciDEIsyJ4DCIuiAEN5J64BHHOIEOGQ3UwaHBJqxChEVfEVEwGMSV0CbOJKQ4+xzn8JG2QVmcMkRia/Sv0qroshqt+zUtRy3TKSoNk5RBhNkLu2c99swe2Cv7/LNW26nR99KiWRpouVH2XyxnPv5V1Wm2cfqtGunZRgU7jleFvBsO07+FPNA3zzq9zG463F5jt+yN/N+wLnuiG2jNd/kuxdPXI/xI5IVejBoU/d2OYZDbiES3I7FULJTYd1vlxQpWsU79iCOBQySRpfpVXOIKHcErRIQtIT5IFTyuZgk/Qkh8Ab91kFY=</latexit>
R = 3
<latexit sha1_base64="OjCfKqSNh49UObVnmIiT+2dVPoU=">AAACjnichVHLSsNAFL2Nr1ofjboR3ARLRRDKREoVQSy66bIP+4BaShKndWheJGmxFn/AvbgQFAUX4gf4AW78ARf9BHFZwY0Lb9KAaLHeMJkzZ+65c2aubKrMdgjpBriR0bHxieBkaGp6ZjbMz80XbKNpKTSvGKphlWTJpirTad5hjkpLpkUlTVZpUW7sufvFFrVsZuj7TtukFU2q66zGFMlBqixVReFYWBPkqljlIyRGvBAGgeiDCPiRNvhHOIBDMECBJmhAQQcHsQoS2PiVQQQCJnIV6CBnIWLePoVTCKG2iVkUMyRkG/iv46rsszqu3Zq2p1bwFBWHhUoBouSF3JMeeSYP5JV8/lmr49VwvbRxlvtaalbDZ4u5j39VGs4OHH2rhnp2oAabnleG3k2PcW+h9PWtk4tebisb7ayQW/KG/m9IlzzhDfTWu3KXodnLIX5k9IIvhg0Sf7djEBTWY2IiFs/EI8ldv1VBWIJlWMV+bEASUpCGvPei53AF1xzPJbhtbqefygV8zQL8CC71BYbukso=</latexit>
a1x + b1
<latexit sha1_base64="ypt4OullvQj05vXVFfrmSxA2QBQ=">AAACjnichVHLSsNAFL2Nr1ofjboR3ARLRRDKpJQqglh002Uf9gG1lCSOdWheJGmxFn/AvbgQFAUX4gf4AW78ARf9BHFZwY0Lb9KAaLHeMJkzZ+65c2aubKrMdgjpBriR0bHxieBkaGp6ZjbMz80XbaNpKbSgGKphlWXJpirTacFhjkrLpkUlTVZpSW7suvulFrVsZuh7TtukVU2q6+yQKZKDVEWqxYVjYU2Qa/EaHyEx4oUwCEQfRMCPjME/wj4cgAEKNEEDCjo4iFWQwMavAiIQMJGrQgc5CxHz9imcQgi1TcyimCEh28B/HVcVn9Vx7da0PbWCp6g4LFQKECUv5J70yDN5IK/k889aHa+G66WNs9zXUrMWPlvMf/yr0nB24OhbNdSzA4ew4Xll6N30GPcWSl/fOrno5Tdz0c4KuSVv6P+GdMkT3kBvvSt3WZq7HOJHRi/4Ytgg8Xc7BkExHhOTsUQ2EUnt+K0KwhIswyr2Yx1SkIYMFLwXPYcruOZ4Lsltcdv9VC7gaxbgR3DpL4s2ksw=</latexit>
a2x + b2
<latexit sha1_base64="KJcELS5QlR94EdZgQoFs9xFpLGc=">AAACjnichVHLSsNAFL2Nr1ofjboR3ARLRRDKxJYqglh002Uf9gG1hCROa2heJGmxFn/AvbgQFAUX4gf4AW78ARf9BHFZwY0Lb9KAaLHeMJkzZ+65c2auZKqK7RDSDTAjo2PjE8HJ0NT0zGyYnZsv2kbTkmlBNlTDKkuiTVVFpwVHcVRaNi0qapJKS1Jjz90vtahlK4a+77RNWtXEuq7UFFl0kKqIQpw75tY4SYgLbITEiBfcIOB9EAE/Mgb7CAdwCAbI0AQNKOjgIFZBBBu/CvBAwESuCh3kLESKt0/hFEKobWIWxQwR2Qb+67iq+KyOa7em7allPEXFYaGSgyh5IfekR57JA3kln3/W6ng1XC9tnKW+lppC+Gwx//GvSsPZgaNv1VDPDtRg0/OqoHfTY9xbyH196+Sil9/KRTsr5Ja8of8b0iVPeAO99S7fZWnucogfCb3gi2GD+N/tGATF9RifjCWyiUhq129VEJZgGVaxHxuQgjRkoOC96DlcwTXDMklmm9nppzIBX7MAP4JJfwGPfpLO</latexit>
a3x + b3

<latexit sha1_base64="cMrtMzvn4NG+7k/rWFifrZke5Q4=">AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdoINiYxUYgiu+skDm52l91JIAZ/QFvFwkrBQvwAP8DGH7DIJ4hlBBsLbzYLokG9y+ycOXPPnTNzNdsQrmSsHlA6OruC3aGe3r7+gcGh8PBIzrXKjs6zumVYzo6mutwQJs9KIQ2+YztcLWkG39aO1pr72xXuuMIyt2TV5nsltWiKgtBVSVRqYz8cZTHmRaQdxH0QhR+bVvgBuziABR1llMBhQhI2oMKlL484GGzi9lAjziEkvH2OE/SStkxZnDJUYo/oX6RV3mdNWjdrup5ap1MMGg4pI5hkz+yONdgTu2cv7OPXWjWvRtNLlWatpeX2/tDpWOb9X1WJZonDL9WfniUKWPK8CvJue0zzFnpLXzm+bGSW05O1KXbDXsn/NauzR7qBWXnTb1M8ffWHH4280ItRg+I/29EOcrOx+EIskUpEk6t+q0IYxwSmqR+LSGIdm8hSfY4znONCCSozypwy30pVAr5mFN9CWfkEdqOPyw==</latexit>
K
<latexit sha1_base64="SX+cUk0bt832iGox/GTAFgu/TOQ=">AAACnHichVG7SgNBFL2urxgfiVooCBIMEaswkaBiJZpCEMEk5gGJhp11kgzZF7uToC75AX/AwkrFQtTaD7DxByzyCWIZwcbCm82CqKh3mZ0zZ+65c2YuNVVuC0JaPVJvX//AoG/IPzwyOhYIjk9kbaNuKSyjGKph5alsM5XrLCO4UFnetJisUZXlaG2js59rMMvmhr4rjky2p8kVnZe5IgukSsGpItWcw2aoyPVQUZNFlVIn1dxPlIJhEiVuhH6CmAfC4MWOEbyHIhyAAQrUQQMGOgjEKshg41eAGBAwkdsDBzkLEXf3GTTBj9o6ZjHMkJGt4b+Cq4LH6rju1LRdtYKnqDgsVIYgQp7INWmTR3JDnsn7r7Uct0bHyxHOtKtlZilwMp1++1el4Syg+qn607OAMqy4Xjl6N12mcwulq28cn7bTq6mIM08uyAv6Pyct8oA30BuvylWSpc7+8EPRC74YNij2vR0/QXYxGluKxpPx8Nq61yofzMAcLGA/lmENNmEHMq6vS7iFO2lWSkhb0nY3VerxNJPwJaTsB6B/mVE=</latexit>
x 2 RD

<latexit sha1_base64="9hP3p02sWXHHDzXWLshlHDq6Jj0=">AAAC7nichVFNa9RQFD1NrdapOmPdCG6CQ2VmM7yUYotQKLpx4aJf0xaaNiSvLzOP5ovkzdAa8gf8AboQEQVB8Qf4A9y4c+WiP6F02YqbUr3JpJRarDck79zz7rk5710n8mSiGNsb0oavjFy9Nnq9Mnbj5q1q7fb4ShL2Yi7aPPTCeM2xE+HJQLSVVJ5Yi2Jh+44nVp3tJ/n+al/EiQyDZbUbiQ3f7gTSldxWRFk16Vrmclcou2E6frqTNfVZ3fSEqxoVdzNtPGtmVmqqvGCQZbrJZcx1cytUSYnzQuNcIWVZxYxlp6uap42tWp21WBH6RWCUoI4y5sPaF5jYQgiOHnwIBFCEPdhI6FmHAYaIuA2kxMWEZLEvkKFC2h5VCaqwid2mb4ey9ZINKM97JoWa0188emNS6phgP9gndsi+sc9snx3/s1da9Mi97NLqDLQisqov7i79+q/Kp1Whe6a61LOCi5nCqyTvUcHkp+ADff/5q8OlR4sT6QP2nh2Q/3dsj32lEwT9I/5hQSy+vsSPQ17oxmhAxt/juAhWJlvGw9bUwlR97nE5qlHcw300aB7TmMNTzKNN/b/jJ07wW4u0l9ob7e2gVBsqNXdwLrSPfwCSMriX</latexit>
f⇥(x) =
⇣
f
(L)
✓(L) · · · f
(1)
✓(1)
⌘
(x)
<latexit sha1_base64="up8ecpPgYfR7pYjRae+QJ3x7AHo=">AAACvHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuYx4RkpJYkKtgqxOSkppXEVMeUgPhx1RqGmrU6MSn5JcU6cCEfzdqYosz0jJKY2ngBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvC6uYQhjcEC7NZMoNsLwCIgXyRD9JdVTf8cbBWkWq1msMjgNdD9Cw1uGhwG+iCv7Evy0sDUoNl43JMEdAswxIARZIgeHZiMMCM9QzM9k0ATZQcnaFRxMEgzKDFoAOPDnMGBwYMhgCEUaP5chsMMZxjOMtkxpTBlM+VClDIxQvUIM6AApjIALx+l6g==</latexit>
⇥ =
n
✓(1)
, . . . , ✓(L)
o
<latexit sha1_base64="CrSrBbNDDRBNHbWah4z26wv6p/I=">AAACm3ichVHLSsNAFD2N73dVEEEEsVTqpkxFVFyJbkRc9GFVsLUkcaqDaRKSaUFjf8AfcOFKwYXo3g9w4w+46CeISwU3LrxNA6LFekNyz5y55+bMXM02hCsZq4WUtvaOzq7unt6+/oHBofDwyLZrlR2dZ3XLsJxdTXW5IUyelUIafNd2uFrSDL6jHa/V93cq3HGFZW7JE5vnS+qhKYpCVyVRhfBYseDl5BGX6r4XE7PVaiMVwhEWZ35MNYNEACIIImmFH5DDASzoKKMEDhOSsAEVLj17SIDBJi4PjziHkPD3OaroJW2ZqjhVqMQe0/eQVnsBa9K63tP11Tr9xaDXIeUUouyZ3bI39sTu2Av7/LOX5/eoezmhrDW03C4MnY9nPv5VlShLHH2rWnqWKGLJ9yrIu+0z9VPoDX3l9OIts5yOejPsmr2S/ytWY490ArPyrt+kePqyhR+NvNCN0YASv8fRDLbn4omF+HxqPrKyGoyqGxOYRozmsYgVrCOJLPU/wzXucK9MKmvKhrLZKFVCgWYUP0LJfgF7g5jS</latexit>
f
(i)
✓(i)
• fully-connected
• convolution
• activation
• ReLU
• leaky ReLU
• absolute value
• pooling (max, average, channel, etc)
• recurrent
• skip connection
MASO
DNs are signal-dependent aﬃne
transformations. The particular aﬃne
mapping applied to x depends on which
parrtition of the spline it falls in Rd.

ReLU
x1
x2
y
x1
x2
<latexit sha1_base64="wThlqV4N/f6TJkASaopeeXy6hGk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64gNpW3aQgLEHzBxKwtXmrgwfoAf4MYfcMEnGJeYuHHhpTQxSsTbTOfMmXvunJkrm5pqO4z1fMLE5NT0jH82MDe/sBgMLS3nbaNhKTynGJphFWXJ5pqq85yjOhovmhaX6rLGC3LtYLBfaHLLVg39yGmZvFyXqrp6qiqSQ1S2XREroQiLMTfCo0D0QARepIzQI45xAgMKGqiDQ4dDWIMEm74SRDCYxJXRIc4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq0HNW1XrdApGg2LlGFE2Qu7Z332zB7YK/v8s1bHrTHw0qJZHmq5WQlerGY//lXVaXZw9q0a69nBKXZdryp5N11mcAtlqG+2u/3sXiba2WC37I3837Aee6Ib6M135S7NM9dj/MjkhV6MGiT+bscoyG/FxJ1YPB2PJPe9VvmxhnVsUj8SSOIQKeSofhWXuEJX8AsxYVtIDFMFn6dZwY8Qkl9Y9pCe</latexit>
z1
<latexit sha1_base64="eRVZ9lBHtuomnSZ9U9icTCKNkZM=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDQTGuiG5c8pBHgoS0dcCG0jZtIQHiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVDFWxbMb6HmFqemZ2zjvvW1hcWvYHVlbzlt40ZZ6TdVU3i5JocVXReM5WbJUXDZOLDUnlBal+ONwvtLhpKbp2bLcNXm6INU2pKrJoE5XtVGKVQIhFmBPBcRB1QQhupPTAI05wCh0ymmiAQ4NNWIUIi74SomAwiCujS5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPa1qOWqZTVBomKYMIsxd2zwbsmT2wV/b5Z62uU2PopU2zNNJyo+K/WM9+/Ktq0Gzj7Fs10bONKvYcrwp5NxxmeAt5pG91eoPsfibc3WK37I3837A+e6IbaK13+S7NM9cT/EjkhV6MGhT93Y5xkI9ForuReDoeSh64rfJiA5vYpn4kkMQRUshR/RoucYWe4BUiwo6QGKUKHlezhh8hJL8AWxaQnw==</latexit>
z2
<latexit sha1_base64="ujDIq6S6SKYxF3DBHU4kFLGtayo=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDohhXRDcuecgjQULaOmBDaZu2kADxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmM9jzAxOTU94531zc0vLPoDS8s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFQ7HOznm9y0FF07tlsGL9XFqqZUFFm0icq0y9vlQIhFmBPBURB1QQhuJPXAI05wCh0yGqiDQ4NNWIUIi74iomAwiCuhQ5xJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVoPalqOWqZTVBomKYMIsxd2z/rsmT2wV/b5Z62OU2PgpUWzNNRyo+y/WM18/Kuq02zj7Fs11rONCvYcrwp5NxxmcAt5qG+2u/3Mfjrc2WC37I3837Aee6IbaM13+S7F09dj/EjkhV6MGhT93Y5RkNuKRHcjsVQslDhwW+XFGtaxSf2II4EjJJGl+lVc4gpdwStEhB0hPkwVPK5mBT9CSHwBXTaQoA==</latexit>
z3
<latexit sha1_base64="RfN7iAoKUyccE78XRExQP3n1Z8Y=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhgUIwV0caShzwSJGR3HXHDvrK7kADxB0xspbDSxML4AX6AjT9gwScYS0xsLLwsmxgl4t3Mzpkz99w5M1cyVcV2GOv5hInJqekZ/2xgbn5hMRhaWi7YRsOSeV42VMMqSaLNVUXneUdxVF4yLS5qksqLUv1gsF9scstWDP3IaZm8ook1XTlVZNEhKteuJqqhCIsxN8KjIO6BCLxIG6FHHOMEBmQ0oIFDh0NYhQibvjLiYDCJq6BDnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wg9q2q5aplNUGhYpw4iyF3bP+uyZPbBX9vlnrY5bY+ClRbM01HKzGrxYzX38q9JodnD2rRrr2cEpdl2vCnk3XWZwC3mob7a7/dxeNtrZYLfsjfzfsB57ohvozXf5LsOz12P8SOSFXowaFP/djlFQ2IrFd2KJTCKS2vda5cca1rFJ/UgihUOkkaf6NVziCl3BL8SEbSE5TBV8nmYFP0JIfQFfVpCh</latexit>
z4
<latexit sha1_base64="jGmtKxPXGnV1kZ3bfATsRPXurLo=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDATGuiG5c8pBHgoS0dcCG0jZtIQHiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVDFWxbMb6HmFqemZ2zjvvW1hcWvYHVlbzlt40ZZ6TdVU3i5JocVXReM5WbJUXDZOLDUnlBal+ONwvtLhpKbp2bLcNXm6INU2pKrJoE5XtVOKVQIhFmBPBcRB1QQhupPTAI05wCh0ymmiAQ4NNWIUIi74SomAwiCujS5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPa1qOWqZTVBomKYMIsxd2zwbsmT2wV/b5Z62uU2PopU2zNNJyo+K/WM9+/Ktq0Gzj7Fs10bONKvYcrwp5NxxmeAt5pG91eoPsfibc3WK37I3837A+e6IbaK13+S7NM9cT/EjkhV6MGhT93Y5xkN+JRHcjsXQslDxwW+XFBjaxTf1IIIkjpJCj+jVc4go9wStEhLiQGKUKHlezhh8hJL8AYXaQog==</latexit>
z5
z1
z2
z3
z4
z5

z1
z2
z3
z4
z5
-12.43 15.48 2.04 2.05 -2.48

arch:
(Linear(2, 5),
ReLU,
Linear(5, 1))
ReLU
x1
x2
y
x1
x2
x1
x2
x1
x2
x1
x2

Power diagram (PD)
aka Laguerre–Voronoi diagram

Balestriero, Randall. "Max-Aﬃne Splines Insights Into Deep Learning." (2021) Diss., Rice University. https://hdl.handle.net/1911/110439.

✓1
X2  ✓1
yes no

✓1
X2  ✓1
yes no
✓2
X1  ✓2
yes no
Blue

✓1
X2  ✓1
yes no
✓2
X1  ✓2
yes no
Blue
✓3
<latexit sha1_base64="3d4/BR7YXLLt+AHkvejt6OKrM6I=">AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb</latexit>
X1  ✓3
yes no
Red

✓1
X2  ✓1
yes no
✓2
X1  ✓2
yes no
Blue
<latexit sha1_base64="587jdLEUJKtkMM/L+3DOcHwPLKw=">AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSLVELR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvFfx1XZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzaHyrxnrmcAq7rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BVRLkss=</latexit>
✓4
<latexit sha1_base64="TQ4GrszeMgM294G1rFaPjchY/V8=">AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd</latexit>
X2  ✓4
yes no
Red
✓3
<latexit sha1_base64="3d4/BR7YXLLt+AHkvejt6OKrM6I=">AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb</latexit>
X1  ✓3
yes no
Red

決定森回帰の信頼区間推定, Benign Overfitting, 多変量木とReLUネットの入力空間分割

決定森回帰の信頼区間推定, Benign Overfitting, 多変量木とReLUネットの入力空間分割

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

More from Ichigaku Takigawa

More from Ichigaku Takigawa (20)

Recently uploaded

Recently uploaded (20)

決定森回帰の信頼区間推定, Benign Overfitting, 多変量木とReLUネットの入力空間分割