Primer for Linearized Encoding Analysis. NCML lab meeting, MPI-EA, Frankfurt, Germany.

1. 2021-10-27 Seung-Goo Kim
Linearized encoding modeling
NCML lab meeting
Review on Nunez-Elizalde et al., 2019, NeuroImage.

2. 2021-10-27 Seung-Goo Kim
Primer for
Linearized encoding modeling
NCML lab meeting

3. Kay et al., 2008, Nature.
Static images
Nishimoto et al., 2011, Curr Biol.
Movie frames

4. Huth et al., 2016, Nature.
Semantics in natural speech
de Heer et al., 2017, J Neurosci.
Acoustic vs. linguistic
contributions

5. What is a “linearized encoding model”?
Nunez-Elizalde et al., 2019, NeuroImage.

6. How is it different from GLM?
• Estimation (ridge regression)
• Delay modeling (finite impulse response;
sometimes used in GLM)
• Optimization (regularization) and evaluation
(cross-validation)

7. 1/3 Ridge regression

8. Consider a linear model for a single-pixel time series*
<latexit sha1_base64="o/CZnJ+k4lphQmUXsvbmK6kJ5PI=">AAACHXicbVDLSsNAFJ34rPVVdelmsAiCUBIp6kYounFZwT6gCWUyvWmHTh7MTIQQ8iNu/BU3LhRx4Ub8Gydt8NF6YODMufdw7z1uxJlUpvlpLCwuLa+sltbK6xubW9uVnd22DGNBoUVDHoquSyRwFkBLMcWhGwkgvsuh446v8nrnDoRkYXCrkggcnwwD5jFKlJb6lXpq+0SNXA8nGb7A379u9sNtFxTJ8DG2IZKM57aqWTMnwPPEKkgVFWj2K+/2IKSxD4GinEjZs8xIOSkRilEOWdmOJUSEjskQepoGxAfppJPrMnyolQH2QqFfoPBE/e1IiS9l4ru6M19YztZy8b9aL1beuZOyIIoVBHQ6yIs5ViHOo8IDJoAqnmhCqGB6V0xHRBCqdKBlHYI1e/I8aZ/UrNOaeVOvNi6LOEpoHx2gI2ShM9RA16iJWoiie/SIntGL8WA8Ga/G27R1wSg8e+gPjI8vhiKhkQ==</latexit>
y = X + ✏
* Without autocorrelation
🤖 🤔

9. Consider a linear model for a single-pixel time series*
<latexit sha1_base64="o/CZnJ+k4lphQmUXsvbmK6kJ5PI=">AAACHXicbVDLSsNAFJ34rPVVdelmsAiCUBIp6kYounFZwT6gCWUyvWmHTh7MTIQQ8iNu/BU3LhRx4Ub8Gydt8NF6YODMufdw7z1uxJlUpvlpLCwuLa+sltbK6xubW9uVnd22DGNBoUVDHoquSyRwFkBLMcWhGwkgvsuh446v8nrnDoRkYXCrkggcnwwD5jFKlJb6lXpq+0SNXA8nGb7A379u9sNtFxTJ8DG2IZKM57aqWTMnwPPEKkgVFWj2K+/2IKSxD4GinEjZs8xIOSkRilEOWdmOJUSEjskQepoGxAfppJPrMnyolQH2QqFfoPBE/e1IiS9l4ru6M19YztZy8b9aL1beuZOyIIoVBHQ6yIs5ViHOo8IDJoAqnmhCqGB6V0xHRBCqdKBlHYI1e/I8aZ/UrNOaeVOvNi6LOEpoHx2gI2ShM9RA16iJWoiie/SIntGL8WA8Ga/G27R1wSg8e+gPjI8vhiKhkQ==</latexit>
y = X + ✏
* Without autocorrelation
🤔
🤖

10. Consider a linear model for a single-pixel time series*
<latexit sha1_base64="o/CZnJ+k4lphQmUXsvbmK6kJ5PI=">AAACHXicbVDLSsNAFJ34rPVVdelmsAiCUBIp6kYounFZwT6gCWUyvWmHTh7MTIQQ8iNu/BU3LhRx4Ub8Gydt8NF6YODMufdw7z1uxJlUpvlpLCwuLa+sltbK6xubW9uVnd22DGNBoUVDHoquSyRwFkBLMcWhGwkgvsuh446v8nrnDoRkYXCrkggcnwwD5jFKlJb6lXpq+0SNXA8nGb7A379u9sNtFxTJ8DG2IZKM57aqWTMnwPPEKkgVFWj2K+/2IKSxD4GinEjZs8xIOSkRilEOWdmOJUSEjskQepoGxAfppJPrMnyolQH2QqFfoPBE/e1IiS9l4ru6M19YztZy8b9aL1beuZOyIIoVBHQ6yIs5ViHOo8IDJoAqnmhCqGB6V0xHRBCqdKBlHYI1e/I8aZ/UrNOaeVOvNi6LOEpoHx2gI2ShM9RA16iJWoiie/SIntGL8WA8Ga/G27R1wSg8e+gPjI8vhiKhkQ==</latexit>
y = X + ✏
* Without autocorrelation
🤔
🤖
∅
???

11. Consider a linear model for a single-pixel time series*
<latexit sha1_base64="o/CZnJ+k4lphQmUXsvbmK6kJ5PI=">AAACHXicbVDLSsNAFJ34rPVVdelmsAiCUBIp6kYounFZwT6gCWUyvWmHTh7MTIQQ8iNu/BU3LhRx4Ub8Gydt8NF6YODMufdw7z1uxJlUpvlpLCwuLa+sltbK6xubW9uVnd22DGNBoUVDHoquSyRwFkBLMcWhGwkgvsuh446v8nrnDoRkYXCrkggcnwwD5jFKlJb6lXpq+0SNXA8nGb7A379u9sNtFxTJ8DG2IZKM57aqWTMnwPPEKkgVFWj2K+/2IKSxD4GinEjZs8xIOSkRilEOWdmOJUSEjskQepoGxAfppJPrMnyolQH2QqFfoPBE/e1IiS9l4ru6M19YztZy8b9aL1beuZOyIIoVBHQ6yIs5ViHOo8IDJoAqnmhCqGB6V0xHRBCqdKBlHYI1e/I8aZ/UrNOaeVOvNi6LOEpoHx2gI2ShM9RA16iJWoiie/SIntGL8WA8Ga/G27R1wSg8e+gPjI8vhiKhkQ==</latexit>
y = X + ✏
* Without autocorrelation

12. Ordinary least squares (OLS) estimation
<latexit sha1_base64="p18gm1H43Oqg8NhgtYlRnGeYDNw=">AAACC3icbVDLSsNAFJ3UV62vqEs3Q4tQF5akFHUjFN24cFHRPqCJZTKdtENnkjAzEULo3o2/4saFIm79AXf+jdM2C209cOFwzr3ce48XMSqVZX0buaXlldW1/HphY3Nre8fc3WvJMBaYNHHIQtHxkCSMBqSpqGKkEwmCuMdI2xtdTvz2AxGShsGdSiLicjQIqE8xUlrqmcXUERxeh1KO4Tl0bumAI1hO4DF0hkjB5Oi+2jNLVsWaAi4SOyMlkKHRM7+cfohjTgKFGZKya1uRclMkFMWMjAtOLEmE8AgNSFfTAHEi3XT6yxgeaqUP/VDoChScqr8nUsSlTLinOzlSQznvTcT/vG6s/DM3pUEUKxLg2SI/ZlCFcBIM7FNBsGKJJggLqm+FeIgEwkrHV9Ah2PMvL5JWtWKfVGo3tVL9IosjDw5AEZSBDU5BHVyBBmgCDB7BM3gFb8aT8WK8Gx+z1pyRzeyDPzA+fwCfipjf</latexit>
Loss = ⌃(y ŷ)2
<latexit sha1_base64="3nuOE8IFzxGi2m03bw4P8YIG8UQ=">AAADZHicjZLPa9swFMcVe1s7r1uTlZ0GQywUkkODbcrWS6Gslx12aFnTBqIkyIqciEq2keSC5+if3G3HXfp3TE5Sk7Wj6wPD0/d9pPfDL8o4U9r3fzUc99nzF1vbL71XO6/f7DZbby9VmktC+yTlqRxEWFHOEtrXTHM6yCTFIuL0Kro+reJXN1QqliYXusjoSOBZwmJGsLbSpNX4sV8iKeC3VCkDjyH6zmYCw04BDyCaYw2L7jj0NphOpaKIaty1+GJRIoH1PIphYQ5gfRgYWHOLxTi0aOf/ZHe8zHNhnsJ6KJaYlCjDUjPMVxUaUws1WbUVbj5yl2VDqmHbdvgYWphqGE976vjRpIWZNNt+z18afOgEa6cN1nY2af5E05TkgiaacKzUMPAzPSqrfgmnxkO5ohkm13hGh9ZNsKBqVC6XxMB9q0xhnEr7JRou1c0bJRZKFSKyZFWhuh+rxH/FhrmOj0YlS7Jc04SsEsU5hzqF1cbBKZOUaF5YBxPJbK2QzLH9d9rupWeHENxv+aFzGfaCT73D88P2yZf1OLbBe/ARdEAAPoMT8BWcgT4gjd/OltN0Ws6tu+Puue9WqNNY39kDf5n74Q/hThLL</latexit>
@L
@ ˆ
= 2XT
Xˆ 2XT
y
<latexit sha1_base64="jouuFJzebpPgK4oXg9NRFpiWvPs=">AAADoHicjVJda9swFFXsfXTeR9PtcQxEQyAZNNimbHsplPWlDx1rWdOGxUmQFTkRlWwjyQPj6G/th+xt/2ZSknlZO7pcEFyde6R77uHGOaNS+f7PhuM+ePjo8c4T7+mz5y92m3svr2RWCEz6OGOZGMRIEkZT0ldUMTLIBUE8ZuQ6vjmx9etvREiapZeqzMmIo1lKE4qRMtBkr/G9XUWCw7NMSg2PYPSFzjiCnRIewGiOFCy749Db4HQsGsVEoa6hLxZVxJGaxwks9QGsLwMNa95iMQ4NtfN/Zne87HOpt+F67SgRCFdRjoSiiK0kal0DNdXOFW7+8rvNBlSTzdzhfdRSe1v3/ePV+K11y7dGbqfi6F69pZ40W37PXwa8mwTrpAXWcT5p/oimGS44SRVmSMph4OdqVFnJmBEzVCFJjvANmpGhSVPEiRxVywXTsG2QKUwyYU6q4BLdfFEhLmXJY8O0CuXtmgX/VRsWKvkwqmiaF4qkeNUoKRhUGbTbCqdUEKxYaRKEBTVaIZ4jY78yO+0ZE4LbI99NrsJe8K53eHHYOv64tmMHvAb7oAMC8B4cg1NwDvoAO2+cE+fM+eTuu6fuZ/diRXUa6zevwF/hfv0FWvgpFQ==</latexit>
@L
@ ˆ
(ˆ⇤
) = 0
<latexit sha1_base64="ePM4g4AC3dq7Y45oeJpkbK7WDSE=">AAADoniclZJda9swFIYVex+d99G0u+zFxLJAUmiwTVl7UygrgzHG1tKmDURxkBU5EZU/kOSBcfS/9jt2t38zOcm8rB1dd8Bw/J5HOq8OJ8w4k8p1fzQs+8HDR483njhPnz1/sdnc2r6UaS4I7ZOUp2IQYkk5S2hfMcXpIBMUxyGnV+H1SVW/+kqFZGlyoYqMjmI8TVjECFZGGm81vrVLJGL4KZVSwyOIztk0xrBTwD2IZljBohv4zhrTqVQUUoW7Bp/PSxRjNQsjWOg9WP8MNKy5+TzwDdr5N9kNFn0u9H1Yp40igUmJMiwUw3xpUetaqNHqXf76Lb/arEk1bN7t34UW+j8a/x5WsFuNy3Xu6SLYNfTdNsbNlttzFwFvJ94qaYFVnI6b39EkJXlME0U4lnLouZkalZVnwql2UC5phsk1ntKhSRMcUzkqFyumYdsoExilwnyJggt1/USJYymLODRk5VDerFXi32rDXEWHo5IlWa5oQpaNopxDlcJqX+GECUoUL0yCiWDGKyQzbOavzFY7ZgjezSffTi79nve2t3+23zp+txrHBtgBr0EHeOAAHIMP4BT0AbFeWe+tz9YX+4390T6zz5eo1VideQn+CBv9BKCsKbE=</latexit>
XT
Xˆ⇤
= XT
y
<latexit sha1_base64="4cetCmY5E0S0MgizpjDBYsjqyRw=">AAAEOniclZPLjtMwFIY9CZchXKYDSzYWVaV0pFZJNAI2SCPYsGBRxHSmUtNUjuu01jgX2Q5SlPq52PAU7FiwYQFCbHkAnLaEzgxqi6VIJ//57HPOLzvMGBXScT7vGeaNm7du79+x7t67/+CgcfjwTKQ5x6SPU5byQYgEYTQhfUklI4OMExSHjJyHF6+q/Pl7wgVNk1NZZGQUo2lCI4qR1NL40Oi1Sp/H8E0qhIIvoP+OTmME7QJ2oD9DEhbtwLPWGLtS/ZBI1Nb4fF76MZKzMIKF6sD6Z6Bgzc3ngadRezvZDhZ1TtUurNXyI45w6WeIS4rYskWlaqFGq7m89VP+lFmTaljP7W1CC/Ufhf+aFRxVdjmVkzu1ERxpfEsf9uaztJsdV22DLlfc6cjNbY0bTafrLBa8HriroAlWqzdufPInKc5jkkjMkBBD18nkqKy8xIwoy88FyRC+QFMy1GGCYiJG5eLqK9jSygRGKddfIuFCXd9RoliIIg41WXUoruYq8V+5YS6j56OSJlkuSYKXhaKcQZnC6h3BCeUES1boAGFOda8Qz5C+F1K/Nkub4F4d+Xpw5nXdp93jt8fNk5crO/bBY/AE2MAFz8AJeA16oA+w8cH4Ynwzvpsfza/mD/PnEjX2VnsegUvL/PUblStl5w==</latexit>
(XT
X) 1
(XT
X)ˆ⇤
= (XT
X) 1
XT
y
<latexit sha1_base64="c2go9MUvKOfmugtZVrS7XQHShhk=">AAAEjXicrZPfbtMwFMa9JsAIf9bBJTcWVaV0UqskGrAhQBNcwAUXQ6xbpaapHNdprdlJZDtIUZqn4Y24421w2hK6Da1FwlKkk88/n/OdkzhMGZXKcX7uNAzzzt17u/etBw8fPd5r7j85l0kmMOnjhCViECJJGI1JX1HFyCAVBPGQkYvw8kO1f/GNCEmT+EzlKRlxNI1pRDFSWhrvN763C19w+DmRsoRvof+VTjmCdg670J8hBfNO4FlrjF2pfkgU6mh8Pi98jtQsjGBedmH9Mihhzc3ngadRezPZCRZ1zsptWKvtRwLhwk+RUBSxpcWyrIUarfry1rP8LrMm1bDu27sNzct/KPxnWMFBNS6nmuRWNoIDjW/yYd+eTI+z65aboKslt0q5wdd/zzhutpyes1jwZuCughZYrdNx84c/SXDGSawwQ1IOXSdVo6L6PJgR7TCTJEX4Ek3JUIcx4kSOisVtKmFbKxMYJUI/sYILdf1EgbiUOQ81WTmU1/cq8W97w0xFR6OCxmmmSIyXhaKMQZXA6mrCCRUEK5brAGFBtVeIZ0j/akpfYEsPwb3e8s3g3Ou5L3uHXw5bJ+9X49gFz8BzYAMXvAIn4BM4BX2ADctwjGPjtblnvjDfmO+WaGNndeYpuLLMj78AgLZ+Gw==</latexit>
ˆ⇤
= (XT
X) 1
XT
y
<latexit sha1_base64="1koSaSHqeiu1M6knayOKEC7fOho=">AAAEj3icrZPdbtMwFMe9JsAIg7XjkhtrU6V0qFUSIYY0QBXcgMTFEOtWqW4rx3Vaa86HYgcpSv0WvAIvxN3eBqcdWbehtUgcKdLJ//x8vhL7CWdCOs7lVs0wHzx8tP3YerLz9NluvbF3JuIsJbRHYh6nfR8LyllEe5JJTvtJSnHoc3ruX3ws4+ffaSpYHJ3KPKHDEE8jFjCCpZbGjdrPZoHSEH6JhVDwHUTf2DTE0M5hG6IZljBvjTzrGrFLEflU4pam5/MChVjO/ADmqg2rl76CFTefj72Rp2F7PdsaLSqdqk1Yq4mCFJMCJTiVDPNlk0pVQoWWg3mrWf6UWZEqWA/u3Yfm6h8KX69rdFguzLGaG7YxOtT4uj7s+5PpdbZdtQ66WXKjlGv3879TjusHTsdZGLzruFfOQXcfvfxx2c1PxvVfaBKTLKSRJBwLMXCdRA6L8vsQTpWFMkETTC7wlA60G+GQimGxuE8KNrUygUGc6ieScKGunihwKEQe+posOxS3Y6X4t9ggk8GbYcGiJJM0IstCQcahjGF5OeGEpZRInmsHk5TpXiGZYf2vSX2FLb0E9/bId50zr+O+7rz6qrfxASxtG7wA+8AGLjgCXfAJnIAeIMaO4RnHxluzYR6Z783uEq1tXZ15Dm6Y+fk3zUeByw==</latexit>
Loss(ˆ) = ||y Xˆ||2
2 = (y Xˆ)T
(y Xˆ)

13. What if (XTX)-1 doesn’t exist?
When the columns of X are not independent
<latexit sha1_base64="YUmrGvwPEV7SuFDowrs4Y+ja1ls=">AAAEWXicrVNNa9tAEF3LSuqqX05z7GWpMbUJNlIoTS6G0F56KDSlcWKwbLFar+wl+mJ3VRBr/ckeCqV/pYeuZNHIsmlT6IBg5r2ZN08L48Y+5cI0vze0pn5w+KD10Hj0+MnTZ+2j59c8ShgmYxz5EZu4iBOfhmQsqPDJJGYEBa5Pbtzbdzl/84UwTqPwSqQxmQVoGVKPYiQU5BxpUVfaARIr14NpBkfwdzXJoO0SgaDRtVdIlMUI2ogtnU1hBzSE63VVYLBHYL2enyoR1VzMygLNYC8dFMJpv6CrOyoac3mSpUa3jtSaTipFIVIT7FX4V5W8P5cDK9tLptnW1h1rOd/bsmWzAF5Vxfp3HkZ/6ryPVGH0H/bdS+8vpu7kHPnxw+fsf4g67Y45NIuAu4lVJh1QxqXT/movIpwEJBTYR5xPLTMWM4mYoNgnymTCSYzwLVqSqUpDFBA+k8VlZLCrkAX0Iqa+UMACrU5IFHCeBq7qzB3yOpeD+7hpIrzzmaRhnAgS4s0iL/GhiGB+ZnBBGcHCT1WCMKPKK8QrxBAW6hgN9QhW/Zd3k+vTofVmaH163bl4Wz5HC7wAL0EPWOAMXID34BKMAda+aT+bB83D5g+9obd0Y9OqNcqZY7AV+vEv5VJqPQ==</latexit>
ˆOLS = (XT
X) 1
XT
y
<latexit sha1_base64="hpW5tsCj3xUVMs6ZnnCcEwBXLnM=">AAADA3icbVLLattAFB2pr1R9xGlXpZuhxtQh2EihJNkEQrvJMoU6MVi2uRqP7CGjkZgZBYQs6Ka/0k0XLaXb/kR2/ZuOZFFUxRcGzj3nnnvnFSScKe26fyz73v0HDx/tPHaePH32fLez9+JSxakkdERiHstxAIpyJuhIM83pOJEUooDTq+D6Q6lf3VCpWCw+6Syh0wiWgoWMgDbUfM961cv9CPQqCHFW4FP8LxsX2A+oBuz4K9A1PsU+yOV8k/gRE3i9bvoHW/zr9ezQ6ZXFlTev2AL3s0HVONuv5OaMRo9ZflBkTq/NtIoOGknVpNWw39DfNvD+LB94xVYxK+adrjt0q8B3gVeDLqrjYt659RcxSSMqNOGg1MRzEz3NQWpGOC0cP1U0AXINSzoxUEBE1TSv3rDAPcMscBhLs4TGFdt05BAplUWBqSx3qNpaSW7TJqkOT6Y5E0mqqSCbQWHKsY5x+SHwgklKNM8MACKZ2SsmK5BAtPk2jrkEr33ku+DycOgdDb2P77pn7+vr2EGv0RvURx46RmfoHF2gESLWZ+ur9d36YX+xv9k/7V+bUtuqPS/Rf2H//gt/cO/J</latexit>
ˆ = arg min ||y X ||2
2
Tikhonov regularization
<latexit sha1_base64="QOhbYLpeATDyYUdW/7ks9+lnB1k=">AAAFEXicrVRNb9MwGPaWACN8dXDkYlFV61StSiYEXCpNcIADEkO0W6WmjRzXaa3GSWQ7SFGav8CFv8KFAwhx5caNf0O+YGlasYKwFOl93+fx8z52bNuBS4XU9R87u4p65eq1vevajZu3bt9p7N89E37IMRlg3/X50EaCuNQjA0mlS4YBJ4jZLjm3F88y/Pwt4YL6Xl9GARkzNPOoQzGSacnaVw5ascmQnNsOjBLYg7+zYQJNm0gEtZY5R7JMetBEfGYVicmoB5fLqsDRBoHlcnJcE6mQJnEnibRWvVIjdSpJLlITbFfwg0p8OImPjGQjGCUrXdesZXh7xZbJGexXxQ4vPPT+xNxGKjf6F/220rvM1IWeFb96+Sb536r/fF5gp0o0nyPGUFL9N/mhqtrv08Wl9lPVQuoXVmTbrMpqNPWung+4Hhhl0ATlOLUa382pj0NGPIldJMTI0AM5jhGXFLsk0cxQkADhBZqRURp6iBExjvMbncBWWplCx+fp50mYV6szYsSEiJidMjOHoo5lxU3YKJTOk3FMvSCUxMNFIyd0ofRh9jzAKeUESzdKA4Q5Tb1CPEccYZk+Ilq6CUZ9yevB2XHXeNQ1Xj9snjwtt2MP3AcPQBsY4DE4AS/AKRgArLxTPiiflM/qe/Wj+kX9WlB3d8o598DKUL/9BMwHsBE=</latexit>
ˆT ik = (XT
X + T
) 1
XT
y
<latexit sha1_base64="z89y30GU7pVMq/REjOUJs6FTyzY=">AAAGcnic1VTfi9NAEN67a+tZf/VOfFHQ1VLsUa82RdSXwqEPKgie2N4VmjZsNtt2aTYJuxshpHn37/PNv8IX/wA3ac5L06I96IsLgZn5Zr/5ZpKM6dlUyFbrx87uXqFYurZ/vXzj5q3bdyoHh2fC9TkmPezaLu+bSBCbOqQnqbRJ3+MEMdMm5+bsbYyffyVcUNfpysAjQ4YmDh1TjKQKGQeFb7VQZ0hOzTEMItiBf7x+BHWTSATLNX2KZOp0oI74xFg4OqMOnM+zBMdrCObzUTtHkkkahY0oKNfykVxSI+MkJDnCegZ/mrGPRuGxFq0Fg2ip6oq0GK8vydI5g90s2dGlhs7fMjehSoReod5GfP+lqEs+I/z08Uu0FdYtfMOwkU3U3yHGUIrnP3Ej7NLZP4UrwgXLBbbwrjqlbTVkq7VhIVVNe96OVv5aI+TUmpCNmlowwYvgh006MirVVrOVHLhqaKlRBek5NSrfdcvFPiOOxDYSYqC1PDkMEZcU20S9cV8QD+EZmpCBMh3EiBiGycqMYE1FLDh2uXocCZNo9kaImBABM1VmrFDksTi4Dhv4cvx6GFLH8yVx8KLQ2LehdGG8f6FFOcHSDpSBMKdKK8RTxBGWakuX1RC0fMurxlm7qb1sap9fVE/epOPYBw/AE1AHGngFTsB7cAp6ABd+Fu8VHxYfFX+V7pcel9LZ7e6kd+6CpVN69hvTmSvZ</latexit>
ˆ = arg min ||y X ||2
+ || ||2
2
2
Ridge regularization (Hoerl & Kennard, 1970)
<latexit sha1_base64="ZRPBfp0omrpv5doMyHPisv6yae8=">AAAFfXicrVRdb9MwFPXWFEb56sYjLxZVRUtZlUyI7aXSBA+AhMQQ7VapaSPHdVurcRLZDlKU5lfwz3jjr/ACTlJYmlYsICxFuvee63PPdexr+w4VUte/7e1XtOqt2wd3anfv3X/wsH54dCm8gGMywJ7j8aGNBHGoSwaSSocMfU4Qsx1yZS9fJ/jVZ8IF9dy+DH0yZmju0hnFSKqQdVj50oxMhuTCnsEwhj342xvG0LSJRLDWNBdIrp0eNBGfW5ljMurC1SpPcLyDYLWanBRIckmTqBOHtWYxUkjq5JyUpEDYyuFPc3Z7Eh0b8U4wjDeqbklL8NaGLJMz2M+Tta819P6UWYYqFfoX9Urx3STqms+KPrz/FP9v1n++L7CTTzTfIMZQnP83xUtlRX26vFG/os24fmGZV6qtfC1Op3NSqpqj3uI0aT0LvitTyqo39K6eLrhtGGujAdbrwqp/NaceDhhxJXaQECND9+U4QlxS7BAlPRDER3iJ5mSkTBcxIsZROj1i2FSRKZx5XH2uhGk0vyNCTIiQ2SozUSiKWBLchY0COTsbR9T1A0lcnBWaBQ6UHkxGEZxSTrB0QmUgzKnSCvECcYSlGlg1dQhGseVt4/Kka7zsGh9fNM5frY/jADwGT0ALGOAUnIO34AIMAK5816DW1p5pP6rN6vNqN0vd31vveQQ2VvX0J6Em1HY=</latexit>
ˆridge = (XT
X + I) 1
XT
y
<latexit sha1_base64="tBe+7BFQBWb2QQSb9Cr0O/Ws/nA=">AAAGd3ic1VRNj9MwEPUubVnKx3bhBgcsSqGraktTIeBSaQUHQEJiEe1upaaNHMdtrcZJZDtIUZqfwJ/jxv/gwg3nY9k0LVDQXhjJ0sy88Zs3ljWmZ1MhO52vO7tXSuXK1b1r1es3bt7arx3cPhWuzzEZYNd2+dBEgtjUIQNJpU2GHieImTY5MxevYvzsE+GCuk5fBh4ZMzRz6JRiJFXKOCh9boQ6Q3JuTmEQwR78GQ0jqJtEIlht6HMks6AHdcRnRhrojDpwucwTHG0gWC4n3QJJrmgStqKg2ihmCkWtXJCQFAibOfxxzj+chEdatBEMopWua9JivLkiS+cM9vNkhxcaer+r3IYqEfoX/bbi+y9FXfAZ4ft3H6PLZv3nTwxb+UL9NWIMZXjxjxthny7+qFwRpiznWBptNdDlz2OrtWEh1Ux70v3lWJxaM7LVYCkdPE++3WYqo1bvtDuJwXVHy5w6yOzEqH3RLRf7jDgS20iIkdbx5DhEXFJsE/VKviAewgs0IyPlOogRMQ6TvRnBhspYcOpydRwJk2z+RoiYEAEzVWWsUBSxOLkJG/ly+mIcUsfzJXFw2mjq21C6MF7C0KKcYGkHykGYU6UV4jniCEu1qqvqEbTiyOvOabetPWtrH57Wj19mz7EH7oEHoAk08BwcgzfgBAwALn0r3y3Xyw/L3yv3K48qzbR0dye7cwesWEX7Aa2RLgM=</latexit>
ˆ = arg min ||y X ||2
+ || 1/2
||2
2 2

14. MATLAB demo1

15. 2/3 Finite impulse response

16. FIR modeling for deconvolution
Transfer function (i.e., receptive field) estimation
<latexit sha1_base64="5LPCDZT9C0l7FM4OHvOZolqSE8A=">AAACG3icdVDLSgMxFM34rPVVdekmWAptwTJT1NaFUHTjsoJ9QFtLJs20oZkHyR2xDP0PN/6KGxeKuBJc+Dem01JU9EDIzTnncnOPHQiuwDQ/jYXFpeWV1cRacn1jc2s7tbNbV34oKatRX/iyaRPFBPdYDTgI1gwkI64tWMMeXkz0xi2TivveNYwC1nFJ3+MOpwQ01U0VR1nI4TN8N7ny2MkOcslM1B4QwM5YP7QUO/I30aE1jm3dVNosnJZLJfMYWwUzBtaMRsmcM2k0Q7Wbem/3fBq6zAMqiFItywygExEJnAo2TrZDxQJCh6TPWrr0iMtUJ4p3G+OMZnrY8aU+HuCY/d4REVepkWtrp0tgoH5rE/IvrRWCU+5E3AtCYB6dDnJCgcHHk6Bwj0tGQYx0Qajk+q+YDogkFHScSR3CfPf/i3qxYJ0UrKujdOV8FkcC7aMDlEUWKqEKukRVVEMU3aNH9IxejAfjyXg13qbWBWPWs4d+wPj4AhponIs=</latexit>
y(t) = x(t) ⇤ f(h)
<latexit sha1_base64="aFfjLgvB+TXECBfMAWN7+ZmkdY0=">AAACMXicdVDLSgMxFM34dnxVXboJlkIVLDOiVheC6MZlBVsLnVoyacYGMw+SO+IwzC+58U/EjQtF3PoTZqalqOiBkJtzziX3HjcSXIFlvRgTk1PTM7Nz8+bC4tLySml1raXCWFLWpKEIZdsligkesCZwEKwdSUZ8V7Ar9/Ys16/umFQ8DC4hiVjXJzcB9zgloKle6bySVGELH+P7/NrGXnWwZVZSZ0AAe5l+aKlwbF+nO3ZW2MxEk6njExi4Hm5njsuA9Eplq3Z0WK9b+9iuWQWwZjTq1pgpoxEavdKT0w9p7LMAqCBKdWwrgm5KJHAqWGY6sWIRobfkhnV0GRCfqW5abJzhimb62AulPgHggv3ekRJfqcR3tTMfU/3WcvIvrRODd9hNeRDFwAI6/MiLBYYQ5/HhPpeMgkh0QajkelZMB0QSCjpkU4cw3v3/orVbsw9q9sVe+eR0FMcc2kCbqIpsVEcn6Bw1UBNR9ICe0St6Mx6NF+Pd+BhaJ4xRzzr6AePzC+KHpMI=</latexit>
y = X
Assumption:
<latexit sha1_base64="YmTE+POcqiGRgw0iMj00P+dloaM=">AAACG3icdVDLSgMxFM34rPVVdekmWAptwTJT1NaFUHTjsoJ9QFtLJs20oZkHyR2xDP0PN/6KGxeKuBJc+Dem01JU9EDI4ZxzSe6xA8EVmOansbC4tLyymlhLrm9sbm2ndnbryg8lZTXqC182baKY4B6rAQfBmoFkxLUFa9jDi4nfuGVScd+7hlHAOi7pe9zhlICWuqliZpSFHD7Dd5Mrj53sIJeM2gMC2Blrrp04kL+JDq1xnOqm0mbhtFwqmcfYKpgxsFY0SuZcSaMZqt3Ue7vn09BlHlBBlGpZZgCdiEjgVLBxsh0qFhA6JH3W0tQjLlOdKN5tjDNa6WHHl/p4gGP1+0REXKVGrq2TLoGB+u1NxL+8VghOuRNxLwiBeXT6kBMKDD6eFIV7XDIKYqQJoZLrv2I6IJJQ0HUmdQnz3f8n9WLBOilYV0fpyvmsjgTaRwcoiyxUQhV0iaqohii6R4/oGb0YD8aT8Wq8TaMLxmxmD/2A8fEFGJyciw==</latexit>
ˆ
f(h) = y(t) ⇤ 1
x(t)
Objective:
Töplitz matrix
<latexit sha1_base64="zCpTTFVyja+uTLx+Aw7XSK+aGSw=">AAADKnicdVJNb9NAEF2br2K+UjhyWREFnFaN7ApIOSBFcOFYJNJGitNovVnHq67X1u64wrL8e7jwV7j0AKq48kNYu6Y4iI608pv33ozH4w0zwTV43oVl37h56/adrbvOvfsPHj7qbT8+0mmuKJvSVKRqFhLNBJdsChwEm2WKkSQU7Dg8fV/rx2dMaZ7KT1BkbJGQteQRpwQMtdy2JoPChSF+iz/Xjx0cufHQGZRBTABHlUmM1Dh2Tso9v2pszqAwbBkkBOIwwrMqCBmQP1VNUhmD23G86ODhyW5XKZxOZsqcQLAIXMf0WXNZEqVIUZW0jspppnyOvfYEdJWCxkFQC7t+LXUcm+p+q27Y/lqCswYZ6jpQm5hctRM5geLrGIbLXt8bvTkYj71X2B95TWDDmBh7V0wftXG47J0Hq5TmCZNABdF67nsZLExX4FQw0zfXLCP0lKzZ3EBJEqYXZfOrKzwwzApHqTJHAm7YbkVJEq2LJDTOeqf6X60m/6fNc4gOFiWXWQ5M0ssXRbnAkOL63uAVV4yCKAwgVHEzK6YxUYSCuV2OWcLVt18PjvZH/uuR//Flf/KuXccWeoqeIRf5aIwm6AM6RFNErS/WN+u79cP+ap/bF/bPS6tttTVP0EbYv34DFW/yIQ==</latexit>
X =
0
B
B
B
@
x(t) 0 0 · · ·
x(t + 1) x(t) 0 · · ·
x(t + 2) x(t + 1) x(t) · · ·
.
.
.
.
.
.
.
.
.
.
.
.
1
C
C
C
A
a.k.a. FIR filter
<latexit sha1_base64="4OAVFMQllCRbg3tozBawFkw8xV0=">AAADUXicdVLLbtNAFB07PIrLI4UlmxGRwWlEZFdAygKpgg3LIpE2UpxE4/HYHtUvzVxXWJZ/kQWs+A82LECMXRO5Eb3SaM4959x5XF0vj7kE2/6h6YNbt+/c3btn7N9/8PDR8ODxmcwKQdmcZnEmFh6RLOYpmwOHmC1ywUjixezcu/jQ6OeXTEiepZ+hzNkqIWHKA04JKGpzoIVmacEYv8Nfmu0QB1Y0NszKjQjgoFaJklrH4bp66dStzTBLxVZuQiDyAryoXY8B+VfVJrUyWD3Hix4eryd9pTR26nrielI2x25zpRqmG7MALLV7LORpRYQgZV3RJmrDbP/xHNvdcqmfgcSu2yoTp9F6lh35qJOv+Xoe97KFirsJtC6W+t2zVCJ4GMF4MxzZ07fHs5n9GjtTuw2sGBUze8uMUBenm+E3189okbAUaEykXDp2Dit1LHAas9pwC8lyQi9IyJYKpiRhclW1E1FjUzE+DjKhVgq4ZfsVFUmkLBNPOZveyl2tIf+nLQsIjlcVT/MCWEqvLgqKGEOGm/HCPheMQlwqQKjg6q2YRkQQCmoIDdWE7d9vBmdHU+fN1Pn0anTyvmvHHnqKniELOWiGTtBHdIrmiGpftZ/ab+2P/l3/NUAD/cqqa13NE3QtBvt/Adhu/Gc=</latexit>
ˆ = X+
y

17. Toy example: causal FIR filter
Stimulus
Kernel
Response
Response = sum (kernel .* stimuli)

18. MATLAB demo2

19. 3/3 Optimization & evaluation via
cross-validation

20. TRAINING SET
(Train/fit a model with/to given data)
TEST SET
(Test its prediction on unseen data)

21. How do we find the optimal lambda?
Optimization methods
• Ridge trace (where the shrinkage is stable): ≤ # lambdas
• Cross-validation (a lambda such that maximizes prediction accuracy): #
lambdas x # CV-folds
• PRESS (predicted residual sum of squares; Allen, 1971): # lambdas (only for
LOOCV)
• Generalized cross-validation: # lambdas

22. Over-optimization
“Cross-validation failure”
• If you find a regularization with a TEST SET, then can you use the same set to
evaluate the model performance?
• NO. Optimization itself introduces a bias towards the SET you used for optimization.
• You need THREE partitions: Training / Optimization / Evaluation sets (eg. Nested
CV)
Varoquaux et al., 2017, NeuroImage.

23. So what’s up with the
multivariate normal prior?
Nunez-Elizalde et al., 2019, NeuroImage.

24. OLS in Bayesian: MLE
<latexit sha1_base64="hpW5tsCj3xUVMs6ZnnCcEwBXLnM=">AAADA3icbVLLattAFB2pr1R9xGlXpZuhxtQh2EihJNkEQrvJMoU6MVi2uRqP7CGjkZgZBYQs6Ka/0k0XLaXb/kR2/ZuOZFFUxRcGzj3nnnvnFSScKe26fyz73v0HDx/tPHaePH32fLez9+JSxakkdERiHstxAIpyJuhIM83pOJEUooDTq+D6Q6lf3VCpWCw+6Syh0wiWgoWMgDbUfM961cv9CPQqCHFW4FP8LxsX2A+oBuz4K9A1PsU+yOV8k/gRE3i9bvoHW/zr9ezQ6ZXFlTev2AL3s0HVONuv5OaMRo9ZflBkTq/NtIoOGknVpNWw39DfNvD+LB94xVYxK+adrjt0q8B3gVeDLqrjYt659RcxSSMqNOGg1MRzEz3NQWpGOC0cP1U0AXINSzoxUEBE1TSv3rDAPcMscBhLs4TGFdt05BAplUWBqSx3qNpaSW7TJqkOT6Y5E0mqqSCbQWHKsY5x+SHwgklKNM8MACKZ2SsmK5BAtPk2jrkEr33ku+DycOgdDb2P77pn7+vr2EGv0RvURx46RmfoHF2gESLWZ+ur9d36YX+xv9k/7V+bUtuqPS/Rf2H//gt/cO/J</latexit>
ˆ = arg min ||y X ||2
OLS MLE (maximum likelihood estimate)
Nunez-Elizalde et al., 2019, NeuroImage.
2

25. Ridge in Bayesian: imposing a prior on β
Ridge MAP (maximum a posteriori)
<latexit sha1_base64="tBe+7BFQBWb2QQSb9Cr0O/Ws/nA=">AAAGd3ic1VRNj9MwEPUubVnKx3bhBgcsSqGraktTIeBSaQUHQEJiEe1upaaNHMdtrcZJZDtIUZqfwJ/jxv/gwg3nY9k0LVDQXhjJ0sy88Zs3ljWmZ1MhO52vO7tXSuXK1b1r1es3bt7arx3cPhWuzzEZYNd2+dBEgtjUIQNJpU2GHieImTY5MxevYvzsE+GCuk5fBh4ZMzRz6JRiJFXKOCh9boQ6Q3JuTmEQwR78GQ0jqJtEIlht6HMks6AHdcRnRhrojDpwucwTHG0gWC4n3QJJrmgStqKg2ihmCkWtXJCQFAibOfxxzj+chEdatBEMopWua9JivLkiS+cM9vNkhxcaer+r3IYqEfoX/bbi+y9FXfAZ4ft3H6PLZv3nTwxb+UL9NWIMZXjxjxthny7+qFwRpiznWBptNdDlz2OrtWEh1Ux70v3lWJxaM7LVYCkdPE++3WYqo1bvtDuJwXVHy5w6yOzEqH3RLRf7jDgS20iIkdbx5DhEXFJsE/VKviAewgs0IyPlOogRMQ6TvRnBhspYcOpydRwJk2z+RoiYEAEzVWWsUBSxOLkJG/ly+mIcUsfzJXFw2mjq21C6MF7C0KKcYGkHykGYU6UV4jniCEu1qqvqEbTiyOvOabetPWtrH57Wj19mz7EH7oEHoAk08BwcgzfgBAwALn0r3y3Xyw/L3yv3K48qzbR0dye7cwesWEX7Aa2RLgM=</latexit>
ˆ = arg min ||y X ||2
+ || 1/2
||2
2 2
Nunez-Elizalde et al., 2019, NeuroImage.

26. How to solve it? Linear transform (whitening)
Nunez-Elizalde et al., 2019, NeuroImage.
Generalized least squares
Linear transform:
Tikhonov -> Ridge
Tikhonov regression

27. Time for discussion!☕