GEX: A flexible method for approximating influence via Geometric Ensemble

1. GEX: A flexible method for approximating influence
via Geometric Ensemble (NeurIPS 2023)
Sung-Yub Kim (presenter), Kyungsu Kim, Eunho Yang
Oct. 6th, 2023

2. TL;DR
• We empirically verify that the proposed method improves the downstream task performance and
relieves the framework dependency of IF (and its approximations).

3. Overview

4. Related works
[1]Cook,R.Dennis."Detectionofinfluentialobservationinlinearregression."Technometrics19.1(1977):15-18.
[2] Koh, Pang Wei, and Percy Liang. "Understanding black-box predictions via influence functions." International conference on machine learning. PMLR, 2017.
[3]Pruthi,Garima,etal."Estimatingtrainingdatainfluencebytracinggradientdescent."AdvancesinNeuralInformationProcessingSystems33(2020):19920-19930.
[4]Schioppa,Andrea,etal."Scalingupinfluencefunctions."ProceedingsoftheAAAIConferenceonArtificialIntelligence.Vol.36.No.8.2022.
[5]Sorscher,Ben,etal."Beyondneuralscalinglaws:beatingpowerlawscalingviadatapruning."AdvancesinNeuralInformationProcessingSystems35(2022):19523-19536.
[6]Paul,Mansheej,SuryaGanguli,andGintareKarolinaDziugaite."Deeplearningonadatadiet:Findingimportantexamplesearlyintraining."AdvancesinNeuralInformationProcessingSystems34(2021):
20596-20607.
[7]Song,Jaeyun,Kim,SungYub,andYang,Eunho."RGE:ARepulsiveGraphRectificationforNodeClassificationviaInfluence."InternationalConferenceonMachineLearning.PMLR,2023.
[8]Kim,SungYub,Kim,Kyungsu,Yang,Eunho.“GEX:AflexiblemethodforapproximatinginfluenceviaGeometricEnsemble”.AdvancesinNeuralInformationProcessingSystems(2023)
[9]Song,Jaeyun,etal.“Post-TrainingRecoveryfromInjectedBiaswithSelf-Influence”,Underreview,2023
[10]Daxberger,Erik,etal."Laplaceredux-effortlessbayesiandeeplearning."AdvancesinNeuralInformationProcessingSystems34(2021):20089-20103.
[11]Kong,Zhifeng,andKamalikaChaudhuri."Understandinginstance-basedinterpretabilityofvariationalauto-encoders."AdvancesinNeuralInformationProcessingSystems34(2021):2400-2412.
[12]Terashita,Naoyuki,etal."InfluenceEstimationforGenerativeAdversarialNetworks."InternationalConferenceonLearningRepresentations.2020.
[13]Chen,Zizhang,etal."CharacterizingtheInfluenceofGraphElements."TheEleventhInternationalConferenceonLearningRepresentations.2022.
[14]Grosse,Roger,etal."StudyingLargeLanguageModelGeneralizationwithInfluenceFunctions."arXivpreprintarXiv:2308.03296(2023).
[15]https://medium.com/data-science-in-your-pocket/random-projection-for-dimension-reduction-27d2ec7d40cd
[16]Arnoldi,WalterEdwin."Theprincipleofminimizediterationsinthesolutionofthematrixeigenvalueproblem."Quarterlyofappliedmathematics9.1(1951):17-29.
[17]Garipov,Timur,etal."Losssurfaces,modeconnectivity,andfastensemblingofdnns."Advancesinneuralinformationprocessingsystems31(2018).
[18]Otsu,Nobuyuki."Athresholdselectionmethodfromgray-levelhistograms."IEEEtransactionsonsystems,man,andcybernetics9.1(1979):62-66.
[19]Wei,Jiaheng,etal."LearningwithNoisyLabelsRevisited:AStudyUsingReal-WorldHumanAnnotations."InternationalConferenceonLearningRepresentations.2021.
[20]Kong,Shuming,YanyanShen,andLinpengHuang."Resolvingtrainingbiasesviainfluence-baseddatarelabeling."InternationalConferenceonLearningRepresentations.2021.
[21]He,Kaiming,etal."Deepresiduallearningforimagerecognition."ProceedingsoftheIEEEconferenceoncomputervisionandpatternrecognition.2016.
[22]Vaswani,Ashish,etal."Attentionisallyouneed."Advancesinneuralinformationprocessingsystems30(2017).
[23]Kim,SungYub,etal."Scale-invariantBayesianNeuralNetworkswithConnectivityTangentKernel."TheEleventhInternationalConferenceonLearningRepresentations.2023.
[24]Andriushchenko,Maksym,etal."Amodernlookattherelationshipbetweensharpnessandgeneralization." InternationalConferenceonMachineLearning.PMLR,2023.
[25]Cohen,Jeremy,etal."GradientDescentonNeuralNetworksTypicallyOccursattheEdgeofStability."InternationalConferenceonLearningRepresentations.2020.
[26]Damian,Alex,EshaanNichani,andJasonD.Lee."Self-Stabilization:TheImplicitBiasofGradientDescentattheEdgeofStability."TheEleventhInternationalConferenceonLearningRepresentations.
2022.
[27]Song,Minhak,andChulheeYun."TrajectoryAlignment:UnderstandingtheEdgeofStabilityPhenomenonviaBifurcationTheory."arXivpreprintarXiv:2307.04204(2023).

5. 1. Introduction

6. Influence Function

7. Influence Function
<latexit sha1_base64="AJLAeDwPQM3rbP2oKayvxEaWark=">AAACWXicbVBdaxNBFJ1dbY3xo9E++jIYhCg17IraVhEKvvggEj/SFjJpmJ3cTYbOziwzdwvpsH/SB0H8Kz44m+yDth64cDjnXu69JyuVdJgkP6P4xs2t7Vud2907d+/d3+k9eHjsTGUFjIVRxp5m3IGSGsYoUcFpaYEXmYKT7Px9459cgHXS6G+4KmFa8IWWuRQcgzTrlQyXgPzs2eySvnnHKj0PzYB+I1MmNWUFx2WW+S/12aj2zJRgORqreQGe20UhdV2zt/Tj4Ose3Yw9pc8pyy0XnoFSg8u9Vq79p3rW6yfDZA16naQt6ZMWo1nvO5sbURWgUSju3CRNSpyGzSiFgrrLKgclF+d8AZNAm7Pc1K+TqemToMxpbmwojXSt/j3heeHcqshCZ/Olu+o14v+8SYX5wdRLXVYIWmwW5ZWiaGgTM51LCwLVKhAurAy3UrHkIRIM+XZDCOnVl6+T4xfD9PXw1eeX/aPDNo4OeUQekwFJyT45Ih/IiIyJID/I72gr2o5+xVHcibub1jhqZ3bJP4h3/wCCJbTC</latexit>
✓⇤
z := argmin
✓2RP
L(S, ✓)
`(z, ✓)
N
on another instance z!
∈ ℝ"
<latexit sha1_base64="rqko3S5rwHjmaf4C5ABZwHraGWY=">AAACNHicbVDLSgNBEJz1bXxFPXoZDGIiGnbFNwiCF0VBBaNCNi6zk44ZnH0w0yskSz7Kix/iRQQPinj1G5zEHNRYMFBUddNT5cdSaLTtZ6uvf2BwaHhkNDM2PjE5lZ2eudBRojiUeCQjdeUzDVKEUEKBEq5iBSzwJVz6t/tt//IOlBZReI6NGCoBuwlFTXCGRvKyR27AsM6ZTA9bXocjpscnJ618c7m5WKA7u9QFKfPNxWXqYh2QXS95zQJd6ZULXjZnF+0OaC9xuiRHujj1so9uNeJJACFyybQuO3aMlZQpFFxCK+MmGmLGb9kNlA0NWQC6knZCt+iCUaq0FinzQqQd9edGygKtG4FvJtux9F+vLf7nlROsbVVSEcYJQsi/D9USSTGi7QZpVSjgKBuGMK6E+SvldaYYR9NzxpTg/I3cSy5Wi85Gcf1sLbe33a1jhMyReZInDtkke+SAnJIS4eSePJFX8mY9WC/Wu/XxPdpndXdmyS9Yn1/ttah3</latexit>
ILOO(z, z0
) := `(z0
, ✓⇤
z ) `(z0
, ✓⇤
)
<latexit sha1_base64="NhtnBVAUrxqI/4QkdlBFLBS0k0U=">AAACO3icbVBNSxtBGJ7V+tHUatSjl6GhYKWEXfFbhIAXDx5ibFTIxjA7eZMMzs4sM+8KYdn/1Uv/hDcvvXhoEa/enU1yqNoXBh6e5/2Y54kSKSz6/r03Nf1hZnZu/mPp08LnxaXy8sqF1anh0ORaanMVMQtSKGiiQAlXiQEWRxIuo5vjQr+8BWOFVj9wmEA7Zn0leoIzdFSn3AhxAMiuNw6OwlR1XSdgNuZoKBQNY4aDKMoa+XU9z0KdgGGojWIxZMz0Y6HyPDykp+vn3+l47FunXPGr/qjoexBMQIVMqt4p34VdzdMYFHLJrG0FfoJttx4Fl5CXwtRCwvgN60PLweK2bWcj7zn96pgu7WnjnkI6Yv+dyFhs7TCOXGdhxb7VCvJ/WivF3l47EypJERQfH+qlkqKmRZC0KwxwlEMHGDfC/ZXyATOMowux5EII3lp+Dy42q8FOdftsq1Lbn8QxT9bIF7JOArJLauSE1EmTcPKT/CZ/yF/vl/fgPXpP49YpbzKzSl6V9/wCkeuu8g==</latexit>
✓⇤
:= argmin
✓2RP
L(S, ✓)
Since retraining is computationally intractable, [2]
proposed an efficient approximation,
named Influence Function (IF):
This can be interpreted as a two-step approximation of
retraining effect
<latexit sha1_base64="m+8X1kuK18A0q/05NrLtZWw8sZ0=">AAACFXicbVDLSgMxFM3UV62vUZdugkVaoZYZ8Q1CwU3dVbAP6ItMmmlDMw+SjNAO8xNu/BU3LhRxK7jzb8y0s9DWA4HDOfdyc47lMyqkYXxrqYXFpeWV9GpmbX1jc0vf3qkJL+CYVLHHPN6wkCCMuqQqqWSk4XOCHIuRujW8if36A+GCeu69HPmk7aC+S22KkVRSVy+0HCQHGLHwNsqPC+PcIby6hv1uOM5FnZb0fFjuhEdmFCtRV88aRWMCOE/MhGRBgkpX/2r1PBw4xJWYISGapuHLdoi4pJiRKNMKBPERHqI+aSrqIoeIdjhJFcEDpfSg7XH1XAkn6u+NEDlCjBxLTcYZxKwXi/95zUDaF+2Qun4giYunh+yAQenBuCLYo5xgyUaKIMyp+ivEA8QRlqrIjCrBnI08T2rHRfOseHp3ki1dJnWkwR7YB3lggnNQAmVQAVWAwSN4Bq/gTXvSXrR37WM6mtKSnV3wB9rnD7fAneo=</latexit>
I(z, z0
) := g>
z0 H 1
gz
<latexit sha1_base64="fumyYRuOF/qnlPSScjWGdh/lGrM=">AAAC83icbVJda9swFJW9rzb7aLo97kUsbElKG+zCvh4Khb10MGgHS1uIEiMrSiIqy0KSxxKhv7GXPWyMve7P7G3/ZrITQ9rsguH43HOu75GVSs60iaK/QXjr9p2797a2G/cfPHy009x9fK7zQhHaJznP1WWKNeVM0L5hhtNLqSjOUk4v0qt3Zf/iM1Wa5eKTmUs6zPBUsAkj2Hgq2Q22UUqnTFjM2VTsuQb0hTJsZgRz+94lFTbGfjg9dZ3FPly0u5XmBcJSqvwLRJTzUaVSmfVruMQiM6MGj/a8ob0P67fELlwXHmwarqm6CC3nH8Gpd7TdCJlcws76lHJILYe1vt5n3XUysgexqygHj9Zz+ShlEkTFuI6eNFtRL6oKboJ4BVpgVWdJ8w8a56TIqDCEY60HcSTN0GJlGOHUNVChqcTkCk/pwEOBM6qHtvpnDj73zBhOcuUfYWDFrjsszrSeZ6lXllvrm72S/F9vUJjJm6FlQhaGCrL80KTg0OSwvABwzBQlhs89wEQxvyskM6wwMf6aNPwhxDcjb4Lzw178qvfy42Hr+O3qOLbAU/AMdEAMXoNjcALOQB+QQAZfg+/Bj7AIv4U/w19LaRisPE/AtQp//wOR/+cp</latexit>
ILOO(z, z0
) ⇡ `lin
✓⇤ (z0
, ✓⇤
z ) `lin
(z0
, ✓⇤
)
= g>
z0 (✓⇤
z ✓⇤
)
⇡ g>
z0 H 1
gz = I(z, z0
)

8. Influence Function

9. Two limitations of IF

10. Two limitations of IF

11. 2. Identifying distributional bias in IF and its approximations

12. Distributional bias in IF and its approximations
<latexit sha1_base64="lL3vvGifEpow471EGLQ7F4WhACM=">AAACR3icbVBNaxsxFNQ6Seu4X25z7EXElCRQzG7pRxowGHxJbgnEdsDrLFpZ64hotYv0NmAL/btccs2tf6GXHhJCjtFufGjtDgjNm3kP6U2cC67B9395tbX1jRcv65uNV6/fvH3XfP9hoLNCUdanmcjUWUw0E1yyPnAQ7CxXjKSxYMP4slf6wyumNM/kKcxyNk7JVPKEUwJOiprnYUrgghJhjmxUcQBzqgg9knZ3/nm+s9fADged6goT55jAmp4NdZFGhnYCe+6qaWTmO47RELLc4rIsKxs1W37br4BXSbAgLbTAcdS8DScZLVImgQqi9SjwcxgbooBTwWwjLDTLCb0kUzZyVJKU6bGpcrD4k1MmOMmUOxJwpf49YUiq9SyNXWe5qV72SvF/3qiAZH9suMwLYJI+P5QUAkOGy1DxhCtGQcwcIVRx91dML4jLClz0DRdCsLzyKhl8aQff299Ovra6Pxdx1NFHtI12UYB+oC46RMeojyi6Rr/RHbr3brw/3oP3+Nxa8xYzW+gf1LwnZf2xHw==</latexit>
ITracIn(z, z0
) :=
1
C
C
X
c=1
gc>
z0 gc
z
<latexit sha1_base64="Xjnu88gy+RPwO3H+1aB3E6C0IIg=">AAACVnicbVFNaxsxENWumyZ1mnbbHnsRNSEpBLNb+g2GQC71zQ5xEvA6i1bWOiKSdpFmC7bQn2wv7U/ppVS79qF1OjDozXszSPOUV4IbiOOfQdh5sPNwd+9Rd//xwZOn0bPnl6asNWUTWopSX+fEMMEVmwAHwa4rzYjMBbvK784a/eor04aX6gKWFZtJslC84JSAp7JIppLALSXCDl3WYgB7oQkdqvORO16drI5ed7GPz4P2SAuv2cTZM5eaWmaWDhJ346tFZldHHtEUysrhcXbe5E1T4UZrJJdFvbgft4Hvg2QDemgToyz6ls5LWkumgApizDSJK5hZooFTwVw3rQ2rCL0jCzb1UBHJzMy2tjh86Jk5LkrtUwFu2b8nLJHGLGXuO5vFzbbWkP/TpjUUH2eWq6oGpuj6oqIWGErceIznXDMKYukBoZr7t2J6S7xx4H+i601Itle+Dy7f9JP3/Xfjt73TTxs79tBL9AodowR9QKfoCxqhCaLoO/oVhEEn+BH8DnfC3XVrGGxmXqB/Ioz+AJLfsqw=</latexit>
ITracInRP(z, z0
) :=
1
C
C
X
c=1
gc>
z0 QRQ>
Rgc
z
<latexit sha1_base64="C5MaCNLjqlxsGTzGITmp5Mmztak=">AAACRnicbVBNaxRBEK1Zv5L1I6s5emlcJBF0mRE1MSBEcongIQY3CexMhp6e3k2Tnu6hu0bYbebXefGcmz/Bi4eEkKs9s3vQxAdNP96roqpeVkphMQx/Bp1bt+/cvbe03L3/4OGjld7jJwdWV4bxIdNSm6OMWi6F4kMUKPlRaTgtMskPs9Odxj/8xo0VWn3FacmTgk6UGAtG0UtpL4kLiieMSvepTluO6D4apWUu6vXZy9naiy7x2PrQfpPUzdbq4xh1SYbpPok/+1E5Td1+fexeRXUjzt2msk57/XAQtiA3SbQgfVhgL+2dxblmVcEVMkmtHUVhiYmjBgWTvO7GleUlZad0wkeeKlpwm7g2hpo890pOxtr4p5C06t8djhbWTovMVzaH2uteI/7PG1U43kycUGWFXLH5oHElCWrSZEpyYThDOfWEMiP8roSdUEMZ+uS7PoTo+sk3ycHrQfRu8PbLm/72+0UcS/AUnsE6RLAB27ALezAEBt/hF5zDRfAj+B1cBlfz0k6w6FmFf9CBPyenry4=</latexit>
IArnoldi(z, z0
) := g>
z0 UR⇤ 1
R U>
R gz

13. Distributional bias in IF and its approximations
<latexit sha1_base64="f47u6XDbCSPi2grflbpZy6fnKhM=">AAAFzHicjVTbbhMxEJ1CAqXcWnjkxaJFaqU0Soq4PSBVKgIqAQqoN6kJkdfrTVb17ka2tzRd5ZUP5I1P4JE/4NjZtuklFSt5PT4z9sw5491goGJjG43fMzduVqq3bs/embt77/6Dh/MLj3ZMlmsht0WmMr0XcCNVnMptG1sl9wZa8iRQcjc42HD+3UOpTZylW3Y4kJ2E99I4igW3gLKFmb/UJkuSjvAOKKKCWqQpowGGoRhoDCslRs+pTk0awVqmd0ANfBpzQPlpFCcFf4AVh59hTsu18rYEroEYWArZVj0awc6BpCTwXkEeRp9gWcy5P8edZmAnQBktUY+6dIyZYW+G3QrvH2Wk87cwVinE6Yk/15yrz1XOUZXyp4VTuDieErX0MGpYfcCe3Nfh2KUe2/CYoD4N/drhoa/+Dx0Cm376ykT0En3Gu12q1UbNHHF9xAfoxzdo/t33hflejTk5ri14lrzKJ8wnu3bo2S17rROsV5EzhB35PLHvOitzueqOvO5byCtLdtd16aS31seP+6BxGsMceU5nPITXvaBN1NsF12X0robhNHiL4bpZYO14Oobu9jGvyZlnWq9zzyUBFpbdnaa4Y/cea13WrP0uDavm2UT/cbMu3oHp2Wqn2kyqdL2iV2Xseb+rlvtv6Bh2WOrr7l0MzdZOb/uVvLvzi416wz/sstEsjUUqn1Z3/lc7zESeyNQKxY3ZbzYGtlNwbWOh5GiunRs54OKA9+Q+zJQn0nQK/zMasWdAQhZlGiO1zKOTOwqeGDNMAkQm3PbNRZ8Dr/Lt5zZ63SnidJBbmYpxoihXzGbM/dlYGGsprBrC4ELHqJWJPtdcWPz/5iBC8yLly8bOWr35sv7i69ri+ptSjll6Qk9xZZv0itbpIz66bRKVzUpWOaoMq1+qtlpUR+PQGzPlnsd07qn+/AdL/z9U</latexit>
Proposition 3.1 (Distributional bias in bilinear self-influence). Let us as-
sume gz follows a P-dimensional stable distribution (e.g., Gaussian, Cauchy, and
Lévy distribution) and M 2 RP ⇥P
is a positive (semi-)definite matrix. Then,
self-influence in the form of IM (z, z) = g>
z Mgz follows a unimodal distribution.
Furthermore, if gz follows a Gaussian distribution, then the self-influence follows
a generalized 2
-distribution.

14. A simple case study in a modified two-circle dataset
<latexit sha1_base64="tEVn5IcyeMefnGdLs/hWXGoL6PM=">AAACAXicbZDLSsNAFIYn9VbrLepGcBMsgquSSL0sC24UhFawF2hDmEwn7dDJhZkToYS48VXcuFDErW/hzrdxkmahrT8MfPznHOac3404k2Ca31ppaXllda28XtnY3Nre0Xf3OjKMBaFtEvJQ9FwsKWcBbQMDTnuRoNh3Oe26k6us3n2gQrIwuIdpRG0fjwLmMYJBWY5+MPAxjAnmyU3q5AyQ3DabqaNXzZqZy1gEq4AqKtRy9K/BMCSxTwMgHEvZt8wI7AQLYITTtDKIJY0wmeAR7SsMsE+lneQXpMaxcoaGFwr1AjBy9/dEgn0pp76rOrMd5XwtM/+r9WPwLu2EBVEMNCCzj7yYGxAaWRzGkAlKgE8VYCKY2tUgYywwARVaRYVgzZ+8CJ3TmnVeO7urVxv1Io4yOkRH6ARZ6AI10DVqoTYi6BE9o1f0pj1pL9q79jFrLWnFzD76I+3zBwqwlz0=</latexit>
ILOO
<latexit sha1_base64="sRYssw0GitVG1c+smryccc5XV6I=">AAACA3icbVDLSsNAFJ34rPUVdaebwSLUTUnEF4hQdKO7CvYBTSg300k7dPJgZiKUUHDjr7hxoYhbf8Kdf+OkzUJbD1w4nHMv997jxZxJZVnfxtz8wuLScmGluLq2vrFpbm03ZJQIQusk4pFoeSApZyGtK6Y4bcWCQuBx2vQG15nffKBCsii8V8OYugH0QuYzAkpLHXPXCUD1CfD0doSdi7IDPO7DpW1Zhx2zZFWsMfAssXNSQjlqHfPL6UYkCWioCAcp27YVKzcFoRjhdFR0EkljIAPo0bamIQRUuun4hxE+0EoX+5HQFSo8Vn9PpBBIOQw83ZldLKe9TPzPayfKP3dTFsaJoiGZLPITjlWEs0BwlwlKFB9qAkQwfSsmfRBAlI6tqEOwp1+eJY2jin1aObk7LlWv8jgKaA/tozKy0RmqohtUQ3VE0CN6Rq/ozXgyXox342PSOmfkMzvoD4zPH+s/lmg=</latexit>
I (↵ = 100)
<latexit sha1_base64="ZmlcEXoPjl2BxdNi7DFGhbT3CY8=">AAACBHicbVDLSsNAFJ34rPUVddnNYBHqpiTiC0QoutFdBfuAJpSb6aQdOnkwMxFK6MKNv+LGhSJu/Qh3/o2TNgttPTDM4Zx7ufceL+ZMKsv6NhYWl5ZXVgtrxfWNza1tc2e3KaNEENogEY9E2wNJOQtpQzHFaTsWFAKP05Y3vM781gMVkkXhvRrF1A2gHzKfEVBa6polJwA1IMDT2zF2LioO8HgAl1bVsg+7Zln/E+B5YuekjHLUu+aX04tIEtBQEQ5SdmwrVm4KQjHC6bjoJJLGQIbQpx1NQwiodNPJEWN8oJUe9iOhX6jwRP3dkUIg5SjwdGW2spz1MvE/r5Mo/9xNWRgnioZkOshPOFYRzhLBPSYoUXykCRDB9K6YDEAAUTq3og7Bnj15njSPqvZp9eTuuFy7yuMooBLaRxVkozNUQzeojhqIoEf0jF7Rm/FkvBjvxse0dMHIe/bQHxifP17olqA=</latexit>
I (↵ = 0.01)
<latexit sha1_base64="AoSyz/4d860DiF5Wn/pRjBOqz3U=">AAACCnicbVC7SgNBFJ2Nrxhfq5Y2o0GICGFXfDVC0CZ2EcwDsku4O5lNhsw+mJkVwpLaxl+xsVDE1i+w82+cTVJo4oGBM+fcy733eDFnUlnWt5FbWFxaXsmvFtbWNza3zO2dhowSQWidRDwSLQ8k5SykdcUUp61YUAg8Tpve4Cbzmw9USBaF92oYUzeAXsh8RkBpqWPuV0sO8LgPR/gKV/ExnvywE4Dqe356O+qYRatsjYHniT0lRTRFrWN+Od2IJAENFeEgZdu2YuWmIBQjnI4KTiJpDGQAPdrWNISASjcdnzLCh1rpYj8S+oUKj9XfHSkEUg4DT1dmG8pZLxP/89qJ8i/dlIVxomhIJoP8hGMV4SwX3GWCEsWHmgARTO+KSR8EEKXTK+gQ7NmT50njpGyfl8/uTouV62kcebSHDlAJ2egCVVAV1VAdEfSIntErejOejBfj3fiYlOaMac8u+gPj8wcybpi7</latexit>
H(↵) = H + ↵I

15. 3. Resolving distributional bias via Geometric Ensemble

16. Removing Linearization
<latexit sha1_base64="P2Y6+MzRDhxgW0qX2P+Pb+C/7Uk=">AAAEkXiclVNtTxNBED5oVayKIB/9MpEqraFNS1CBpAZEDURiMOEt6bXN3nau3bC3e+7u8Xbe7/H/+M1/4961RQok4iSXm8w88/bMjhdypk2t9ntiMpe/d//B1MPCo8dPpp/OzD470DJSFPep5FIdeUQjZwL3DTMcj0KFJPA4HnrHm6n/8ASVZlLsmfMQWwHpCeYzSow1dWYnf7oGz4znx3t9lAoDWK7WEyhtSiGQphjw0JwiCtj+DER0YWejXIWiGxDTp4TH20kRmIDm0koLKEnRgGe2Ba2xC0RDwfWwx0RMOOuJ10kBrFwJLl0swsVCOTO/akD2jzO/58Wfkk7shpqBq1kAYSezGxPvbCTDRBx90xyoH5EbAi5y3s5wKogtKWkG00dD2q+zWmm6Mri0K83dQhZGMYMqivX6pjWqTjzkMX5fY36lbyeuUHmSFFwU3ctxT/uo0NL1P92tNe4IrPwLN9LL0IBe56LtGhlCKaPUxo6cxWyvxTF+7VI1WADskJATikDCUMkzZhF2r6HUBhWT6rbtjuUp/R1ptPSvSba2UjYE/LjsYxG22nHFPr4ByeXqGJOdmflatZYJ3FTqQ2XeGcpuZ+aX25U0ClAYyonWzXotNK2YKMMoR7umSGNI6DHpYdOqggSoW3F2UQm8jNLn60tlP2Egs16NiEmg9XngWWQ6lr7uS423+ZqR8VdaMRNhZFDQQSE/4mAkpOcJXabs2fFzqxCqmO0VaJ8oQi3fumBJqF8f+aZysFStv62++bY0v746pGPKee68cEpO3XnnrDtbzq6z79DcdG4518i9z8/lV/Pr+Q8D6OTEMGbOGZP8lz9F7nSe</latexit>
Theorem 4.1 (Connection between IF and LA). I in [28] can be expressed
as
I(z, z0
) = E ⇠pLA
⇥
`lin
✓⇤ (z, ) · `lin
✓⇤ (z0
, )
⇤
where `lin
✓⇤ (z, ) := `lin
✓⇤ (z, ) `lin
✓⇤ (z, ✓⇤
) = g>
z ( ✓⇤
) and pLA is the Laplace
approximated posterior
pLA( ) := N |✓⇤
, H 1
.
<latexit sha1_base64="M+LVLxJYyTlnS/UIE0jcBjaofaM=">AAACknicdVHbatwwEJXdpkm3l2wvb30RXUqSUha7JL0EciMNpJCHFLpJYO2asXa8KyJfkMaBjfEH5Xfy1r+p7N1CLu0BweHMmZF0Ji6UNOR5vx33wcOFR4tLjztPnj57vtx98fLE5KUWOBC5yvVZDAaVzHBAkhSeFRohjRWexuf7Tf30ArWRefaTpgWGKYwzmUgBZKWoexWkQBMBqvpeRy0nqo726tXLD5cra3xzi7diHFcHdVQFhZE8MDLlxU1z3eEWgcKEhjP6DRUBD1Ap20QTJPj1vpnJmwlrPBCjnP7rWvlrm83ScjyhsB91e17fa8HvE39OemyO46h7HYxyUaaYkVBgzND3Cgor0CSFwroTlAYLEOcwxqGlGaRowqqNtObvrDLiSa7tyYi36s2OClJjpmlsnU0O5m6tEf9VG5aUfAkrmRUlYSZmFyWl4pTzZj98JDUKUlNLQGhp38rFBDQIslvs2BD8u1++T04+9v1P/Y0f673dr/M4ltgb9patMp99ZrvskB2zARPOsrPhbDs77mt3091z92dW15n3vGK34B79Abh5w8Q=</latexit>
ILA(z, z0
) := E ⇠pLA
[ `✓⇤ (z, ) · `✓⇤ (z0
, )] .
<latexit sha1_base64="srbKo/FwyKi7kz4Ynx9o8Z1llTs=">AAACLXicbVDLSgMxFM3UV62vqks3wSJU0TIjvkEo6MKlgtVCp5ZMetuGZh4kd4Q69Ifc+CsiuKiIW3/DtFbU6oHAyTnnktzjRVJotO2elRobn5icSk9nZmbn5heyi0tXOowVhxIPZajKHtMgRQAlFCihHClgvifh2muf9P3rW1BahMEldiKo+qwZiIbgDI1Uy566pyCRURekrCUutgDZzUY3f7dJ3UiLdXp0TAfmt7L1df9Kr9eyObtgD0D/EmdIcmSI81r2ya2HPPYhQC6Z1hXHjrCaMIWCS+hm3FhDxHibNaFiaMB80NVksG2XrhmlThuhMidAOlB/TiTM17rjeybpM2zpUa8v/udVYmwcVBMRRDFCwD8fasSSYkj71dG6UMBRdgxhXAnzV8pbTDGOpuCMKcEZXfkvudouOHuF3YudXPFwWEearJBVkicO2SdFckbOSYlwck8eSY+8WA/Ws/VqvX1GU9ZwZpn8gvX+ARRPpYk=</latexit>
`✓⇤ (z, ) := `(z, ) `(z, ✓⇤
)

17. Utilization of Geometric Ensemble
<latexit sha1_base64="gRUshho/+GpXwFSAGbUgD/oKvMw=">AAAEIXicbVNLbxMxEN4mPEp4tXDkMqKLlJY0SipBOVCpAglVahUF+pS6beR1ZrNWvfZie0uT1f4VLvwVLhxAqDfEn2GyaUtbmIM1+jyPb8afw1QK61qtX1OV6o2bt25P36ndvXf/wcOZ2Uc7VmeG4zbXUpu9kFmUQuG2E07iXmqQJaHE3fDo7fh+9xiNFVptuWGKBwkbKBEJzhxBvdnKcuDwxIVR3jU61VaM4QLqa2itYAqsUINMMiPcECJtQFOxxZQZlqBDI0bYh07HzjdhAx1kFpi1WYLAgFgsOsOIVh8u4sEPXIyOHS5AIBQECXNxGOYfisO8W/jAeCzwGC2M0Ggos6k9SG0t+Bv1zQacp8/DCrT8kpH9mDFDTdAYbZqwFaNqgL/mQ8yIjgOJzDrwu4uddb8svIhigOqYyYw6jSvQAGAzHoOLKd7vrMNr6PpNeJcZamcSbbABIgL/xAdBNRUIlWYOdPSXo2VJKhH8UTnYpt+gYqjGB7WQUn8aB8Va9ictx/iEBnKnDY0X5JD1BARFLxcrnfXn7clOakGIA6FyJsVALRQ1IBv0RoeB02mZsAKBYqFkJdLLAxohHxYFBChlfXRpY0Gm+mhCwzjmV1PKiCKqnzQugqkyEZm8T5S3qB7tuxag6p8z6c3MtZqt0uBfp33mzHln1u3NnAZ9zUkcynFJMtlvt1J3kDPjBJdY1ILMYsr4ERvgPrmKFGMP8lLhBTwjpF9uLtLKQYlezshZYu0wCSlyzNlevxuD/7vbz1z06iAv3xMVnzSKMgmO9EffBfrC0AvJITmM0y8QHHhMcuakZlujJbSvj/yvs7PUbL9svni/NLf65mwd094T76lX99resrfqrXldb9vjlc+Vr5XvlR/VL9Vv1Z/V00loZeos57F3xaq//wD+QVa0</latexit>
Proposition (Hessian singularity for over-parameterized NNs). Let us as-
sume a pre-trained parameter ✓⇤
2 RP
achieves zero training loss L(S, ✓⇤
) = 0
for squared error. Then, H has at least P NK zero-eigenvalues for NNs such
that NK < P. Furthermore, if x is an input of training sample z 2 S, then the
following holds for the eigenvectors {ui}P
i=NK+1
g>
z ui = r>
ŷ `(z, ✓⇤
) r>
✓ f(x, ✓⇤
)ui
| {z }
0
= 0
<latexit sha1_base64="PpgwH7Ake9PCmrRDLUlAeqSlFn8=">AAACEHicbVDJSgNBEO1xjXGLevTSGMR4CTPiBiJEvSh4iGAWyIRQ0+kkTXoWumuEMOQTvPgrXjwo4tWjN//GznLQxAdNP96roqqeF0mh0ba/rZnZufmFxdRSenlldW09s7FZ1mGsGC+xUIaq6oHmUgS8hAIlr0aKg+9JXvG6VwO/8sCVFmFwj72I131oB6IlGKCRGpk91wfsMJDJTb8x5IjJ7UWfumc5F2TUgXM7bzv7jUzW/EPQaeKMSZaMUWxkvtxmyGKfB8gkaF1z7AjrCSgUTPJ+2o01j4B1oc1rhgbgc11Phgf16a5RmrQVKvMCpEP1d0cCvtY93zOVg5X1pDcQ//NqMbZO64kIohh5wEaDWrGkGNJBOrQpFGcoe4YAU8LsSlkHFDA0GaZNCM7kydOkfJB3jvNHd4fZwuU4jhTZJjskRxxyQgrkmhRJiTDySJ7JK3mznqwX6936GJXOWOOeLfIH1ucPChab6g==</latexit>
ILA (↵ = 0.01)
<latexit sha1_base64="JY+U0ZmSLnSoGiMXkCJ3WQ+ilXc=">AAACD3icbZDJSgNBEIZ7XGPcoh69NAYlXsKMuIEIUS8KHiKYBTIh1HQ6SZOehe4aIQx5Ay++ihcPinj16s23sbMcNPGHho+/quiq34uk0Gjb39bM7Nz8wmJqKb28srq2ntnYLOswVoyXWChDVfVAcykCXkKBklcjxcH3JK943atBvfLAlRZhcI+9iNd9aAeiJRigsRqZPdcH7DCQyU2/MWTE5PaiT92znAsy6sC5Y9v7jUzWzttD0WlwxpAlYxUbmS+3GbLY5wEyCVrXHDvCegIKBZO8n3ZjzSNgXWjzmsEAfK7ryfCePt01TpO2QmVegHTo/p5IwNe653umc7CxnqwNzP9qtRhbp/VEBFGMPGCjj1qxpBjSQTi0KRRnKHsGgClhdqWsAwoYmgjTJgRn8uRpKB/kneP80d1htnA5jiNFtskOyRGHnJACuSZFUiKMPJJn8krerCfrxXq3PkatM9Z4Zov8kfX5A5GDm7I=</latexit>
ILA (↵ = 100)
<latexit sha1_base64="s0zEbOYkkBwPLESPadaN5uCYuU0=">AAACCXicbZC7SgNBFIZn4y3G26qlzWAQYhN2xaggQtRGwSKKuUASwuxkkgyZvTBzVgjLtja+io2FIra+gZ1v42SzhSb+MPDxn3OYc34nEFyBZX0bmbn5hcWl7HJuZXVtfcPc3KopP5SUVakvfNlwiGKCe6wKHARrBJIR1xGs7gwvx/X6A5OK+949jALWdknf4z1OCWirY+KWS2BAiYiu407CANHNeYxbp4W7s5K13zHzVtFKhGfBTiGPUlU65ler69PQZR5QQZRq2lYA7YhI4FSwONcKFQsIHZI+a2r0iMtUO0ouifGedrq450v9PMCJ+3siIq5SI9fRneNd1XRtbP5Xa4bQO2lH3AtCYB6dfNQLBQYfj2PBXS4ZBTHSQKjkeldMB0QSCjq8nA7Bnj55FmoHRfuoWLo9zJcv0jiyaAftogKy0TEqoytUQVVE0SN6Rq/ozXgyXox342PSmjHSmW30R8bnDwB/mTo=</latexit>
ILA (R = 50)
<latexit sha1_base64="IF4DsSdOVW+Hs34gt59F9n075OY=">AAAB/XicbVDLSgMxFL3js9bX+Ni5CRahbsqMWBVEKLrRXRX7gM5QMmnahmYeJBmhDsVfceNCEbf+hzv/xkw7C209EDiccy/35HgRZ1JZ1rcxN7+wuLScW8mvrq1vbJpb23UZxoLQGgl5KJoelpSzgNYUU5w2I0Gx73Ha8AZXqd94oEKyMLhXw4i6Pu4FrMsIVlpqm7uOj1WfYJ7cjJBzXry7KFuHbbNglawx0CyxM1KADNW2+eV0QhL7NFCEYylbthUpN8FCMcLpKO/EkkaYDHCPtjQNsE+lm4zTj9CBVjqoGwr9AoXG6u+NBPtSDn1PT6ZZ5bSXiv95rVh1z9yEBVGsaEAmh7oxRypEaRWowwQlig81wUQwnRWRPhaYKF1YXpdgT395ltSPSvZJqXx7XKhcZnXkYA/2oQg2nEIFrqEKNSDwCM/wCm/Gk/FivBsfk9E5I9vZgT8wPn8Ad7eT8A==</latexit>
I (R = 50)

18. Utilization of Geometric Ensemble
<latexit sha1_base64="pnaUFL7AQVg8XQ793gRTJ7w7D+E=">AAAClXicdVFdT9RAFJ1WFFw/WPDBB18mbgxgzKYliryQIPKhT2Liwibb2tzO3u5OmH5k5tZkafqP/DW88W+YdtcEQU8yyck5596ZuTculDTkedeO+2Dp4aPllcedJ0+fPV/trq2fmbzUAgciV7kexmBQyQwHJEnhsNAIaazwPL743Pjnv1AbmWc/aFZgmMIkk4kUQFaKur+DFGgqQFVf66jlRNXJ0bDevHx3ubHV4RZ7vDXiuDqqoyoojOSBkSkvbhXUdRsNFCY0mtNDVAQ8QKVsEU2R4Ofbpi1vOmzxQIxz+m9q409s3kvLyZTCqNvz+l4Lfp/4C9JjC5xG3atgnIsyxYyEAmNGvldQWIEmKRTWnaA0WIC4gAmOLM0gRRNW7VRr/sYqY57k2p6MeKverqggNWaWxjbZjMHc9RrxX96opGQ3rGRWlISZmF+UlIpTzpsV8bHUKEjNLAGhpX0rF1PQIMgusmOH4N/98n1ytt33d/ofvr/v7R8sxrHCXrHXbJP57CPbZ1/YKRsw4aw7u84n58B96e65h+7xPOo6i5oX7C+4324A8G/EQw==</latexit>
IGEX(z, z0
) = E ⇠pGE
[ `✓⇤ (z, ) · `✓⇤ (z0
, )]
<latexit sha1_base64="M+LVLxJYyTlnS/UIE0jcBjaofaM=">AAACknicdVHbatwwEJXdpkm3l2wvb30RXUqSUha7JL0EciMNpJCHFLpJYO2asXa8KyJfkMaBjfEH5Xfy1r+p7N1CLu0BweHMmZF0Ji6UNOR5vx33wcOFR4tLjztPnj57vtx98fLE5KUWOBC5yvVZDAaVzHBAkhSeFRohjRWexuf7Tf30ArWRefaTpgWGKYwzmUgBZKWoexWkQBMBqvpeRy0nqo726tXLD5cra3xzi7diHFcHdVQFhZE8MDLlxU1z3eEWgcKEhjP6DRUBD1Ap20QTJPj1vpnJmwlrPBCjnP7rWvlrm83ScjyhsB91e17fa8HvE39OemyO46h7HYxyUaaYkVBgzND3Cgor0CSFwroTlAYLEOcwxqGlGaRowqqNtObvrDLiSa7tyYi36s2OClJjpmlsnU0O5m6tEf9VG5aUfAkrmRUlYSZmFyWl4pTzZj98JDUKUlNLQGhp38rFBDQIslvs2BD8u1++T04+9v1P/Y0f673dr/M4ltgb9patMp99ZrvskB2zARPOsrPhbDs77mt3091z92dW15n3vGK34B79Abh5w8Q=</latexit>
ILA(z, z0
) := E ⇠pLA
[ `✓⇤ (z, ) · `✓⇤ (z0
, )] .

19. Utilization of Geometric Ensemble

20. Empirical evaluations

21. Empirical evaluations

22. 4. Discussion points

23. Conclusion and Future directions
• AsshowninTheorem4.1,IF isclosely relatedto the Laplace Approximation.
• Onthe otherhand,[23] formalizedthe relationshipbetweensharpnessandLaplace ApproximationthroughPAC-Bayes.
• Althoughthere iscriticismthat sharpnessisnot directly relatedto generalization[24],anaccurate understandingof the evolution
(dynamics) of sharpnesswillhelpclarify the relationshipbetweensharpnessandgeneralization.
• Recent worksonEdge-of-Stability (EOS)[25,26,27] wouldbe agoodstartingpoint.

24. Thank you !
Any Questions ?