Mar. 23, 2023•0 likes## 0 likes

•3 views## views

Be the first to like this

Show More

Total views

0

On Slideshare

0

From embeds

0

Number of embeds

0

Download to read offline

Report

Education

GLMs_printable

Elio LaureanoFollow

Estrategias de Marketing & Desarrollo de Nuevos NegociosThe Fundamental theorem of calculus AhsanIrshad8

Statistical Inference Part II: Types of Sampling DistributionDexlab Analytics

CS229 Machine Learning Lecture NotesEric Conner

Intro to Approximate Bayesian Computation (ABC)Umberto Picchini

X01 Supervised learning problem linear regression one feature theorieMarco Moldenhauer

Week 7EasyStudy3

- Week 2 Generalized Linear Models Applied Statistical Analysis II Jeffrey Ziegler, PhD Assistant Professor in Political Science & Data Science Trinity College Dublin Spring 2023
- Road map for today Generalized Linear Models (GLMs) I Why do we need to think like this? I What type of distributions can we use? I Getting our parameters and estimates Next time: Maximum Likelihood Estimation (MLE) By next week, please... I Begin working on problem set #1 I Read assigned chapters This has "been done already", but I want y’all to understand what’s going on, especially w.r.t. to theory & programming 1 38
- What are GLMs, why does they matter? Remember from last week, we want to use same tools of inference and probability for non-continuous outcomes So, we need a framework for estimating parametric models: yi ∼ f(θ, xi) where: θ is a vector of parameters xi is a vector of exogenous characteristics ofith observation Specific functional form, f, provides an almost unlimited choice of specific models I As we will see today, not quite 2 38
- What do we need to make this work? For a given outcome, we need to select a distribution (we’ll narrow down set) and select correct 1. parameter, and 2. estimate We’ll also want measure of uncertainty (variance) 3 38
- GLM Framework: Gaussian Example 2. Generalized Linear Model Linear Examples: 1. Gaussian 2. Poisson Noise (exponential family) Nonlinear y = f(~ ✓ · ~ x) + ✏ 11 4 38
- GLM Framework: Gaussian Example 2. Generalized Linear Model Linear Noise (exponential family) Nonlinear Terminology: “distribution function” “parameter” = “link function” 12 5 38
- GLM Framework: Gaussian Example 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 1 01 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 stimulus response From s ike counts to spike trains: linear filter vector stimulus at time t time response at time t first idea: linear-Gaussian model! yt = ~ k · ~ xt + ✏t yt = ~ k · ~ xt + noise N(0, 2 ) 13 6 38
- GLM Framework: Gaussian Example ~ xt yt 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 1 01 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 stimulus response linear filter vector stimulus at time t yt = ~ k · ~ xt + noise time response at time t t = 1 walk through the data one time bin at a time N(0, 2 ) 14 7 38
- GLM Framework: Gaussian Example ~ xt yt 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 1 01 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 linear filter vector stimulus at time t yt = ~ k · ~ xt + noise time response at time t t = 2 walk through the data one time bin at a time stimulus response N(0, 2 ) 15 8 38
- GLM Framework: Gaussian Example ~ xt yt 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 1 01 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 linear filter vector stimulus at time t yt = ~ k · ~ xt + noise time response at time t t = 3 walk through the data one time bin at a time stimulus response N(0, 2 ) 16 9 38
- More familiar maybe in matrix version Build up to following matrix version: 0 … Y X~ k = + noise = time design matrix … ~ k 1 0 10 38
- More familiar maybe in matrix version Build up to following matrix version: 0 1 0 … Y X~ k = + noise = time k̂ = (XT X) 1 XT Y stimulus covariance spike-triggered avg (STA) (maximum likelihood estimate for “Linear-Gaussian” GLM) least squares solution: … ~ k 21 11 38
- Towards a likelihood function 12 38
- Towards a likelihood function Formal treatment: scalar version model: N(0, 2 ) yt = ~ k · ~ xt + ✏t equivalent to writing: yt|~ xt,~ k ⇠ N(~ xt · ~ k, 2 ) p(yt|~ xt,~ k) = 1 p 2⇡ 2 e (yt ~ xt·~ k)2 2 2 or p(Y |X,~ k) = T Y t=1 p(yt|~ xt,~ k) For entire dataset: (independence across time bins) = (2⇡ 2 ) T 2 exp( PT t=1 (yt ~ xt·~ k)2 2 2 ) Guassian noise with variance 2 log P(Y |X,~ k) = PT t=1 (yt ~ xt·~ k)2 2 2 + const log-likelihood 22 13 38
- Towards a likelihood function Formal treatment: vector version 0 … Y X~ k = … ~ k = time + ~ ✏ N(0, 2 I) … + iid Gaussian noise vector ✏1 ✏2 ✏3 equivalent to writing: or Y |X,~ k ⇠ N(X~ k, 2 I) P(Y |X,~ k) = 1 |2⇡ 2I| T 2 exp ⇣ 1 2 2 (Y X~ k)> (Y X~ k) ⌘ Take log, differentiate and set to zero. 1 0 23 14 38
- Towards a likelihood function 15 38
- Towards a likelihood function 0 … … ~ k ≈ time probability of spike at bin t Bernoulli GLM: pt = f(~ xt · ~ k) (coin flipping model, y = 0 or 1) p(yt = 1|~ xt) = pt nonlinearity Equivalent ways of writing: yt|~ xt,~ k ⇠ Ber(f(~ xt · ~ k)) p(yt|~ xt,~ k) = f(~ xt · ~ k)yt ⇣ 1 f(~ xt · ~ k) ⌘1 yt or But noise is not Gaussian! log-likelihood: L = PT t=1 ⇣ yt log f(~ xt · ~ k) + (1 yt) log(1 f(~ xt · ~ k)) ⌘ f( ) 1 0 16 38
- GLM Framework: Logit too! Logistic regression Logistic regression: f(x) = 1 1 + e x logistic function • so logistic regression is a special case of a Bernoulli GLM 0 … … ~ k ≈ time probability of spike at bin t Bernoulli GLM: pt = f(~ xt · ~ k) (coin flipping model, y = 0 or 1) p(yt = 1|~ xt) = pt nonlinearity f( ) 1 0 25 17 38
- Where to start? Exponential Family Intro We need to narrow down set of functions I Set we use is called ’exponential family form’ (EFF), which we can characterise in ’canonical form’ Nice properties: I All have "their moments" We should be able to characterise (1) center and (2) spread of data generating distribution based on data More specifically, by putting PDFs and PMFs into EFF, we are able to isolate subfunctions that produce a small # of statistics that succinctly summarize large data using a common notation Exceptions: Student t’s or uniform distributions can’t transform into EFF, they’re dependent on bounds (sometimes Weibull) I Allows us to use log-likelihood functions in replace of likelihood function because they have same mode (maximum of function) for θ 18 38
- Exponential Family: Canonical Form The general expression is f(y|θ) = exp[yθ − b(θ) + c(y)] where yθ multiplicative term have both y and θ b(θ) ’normalising constant’ We want to isolate and derive b(θ)! 19 38
- Next, construct joint distribution This is important, we need this for likelihood function f(y|θ) = exp " n X yiθ − nb(θ) + n X c(yi) # 20 38
- Example: Poisson f(y|µ) = e−µµy y! = e−µ µy (y!)−1 (1) Let’s take log of expression, place it within an exp[] = exp [−µ + ylog(µ) − log(y!)] = exp [ylog(µ) − µ − log(y!)] (2) where yθ ylog(µ) b(θ) µ c(y) log(y!) 21 38
- Example: Poisson yθ ylog(µ) b(θ) µ c(y) log(y!) In canonical form, θ = log(µ) = canonical link Parameterized form of b(θ) by θ is done by taking inverse of canonical link whereby b(θ) = exp(θ) = µ 22 38
- Likelihood Theory Awesome, we have a way to calculate our parameters of interest, now what? How do we calculate our estimates? For sufficiently large samples, likelihood surface is unimodal in k dimensions for exponential forms I Process is equivalent to finding a k-dimensional mode I We want a posterior distribution of unknown k-dimensional θ coefficient vectors given observed data, f(θ|X) 23 38
- Likelihood Theory f(θ|X) = f(X|θ) p(θ) p(X) where f(X|θ) represents the joint PDF p(θ) is posterior produced by Bayes rule p(X) is unconditional probabilities Determines the most likely values of a θ vector 24 38
- Likelihood Theory We can regard f(X|θ) as a function for θ given observed data, where p(X) = 1 since we observed data We then stipulate a prior distribution of θ to allow for a direct comparison of observed data versus prior Gives us our likelihood function, L(θ|X) = f(X|θ), where we want to find value of θ that maximises likelihood function 25 38
- Likelihood Theory If θ̂ is estimate of θ that maximizes the likelihood function, then L(θ̂|X) ≥ L(θ|X)∀θ ∈ Θ To get expected value of y, E[y], we first need to differentiate b(θ) with respect to θ whereby ∂ ∂θ b(θ) = E[y] We can follow these steps: 1. Take ∂ ∂θ b(θ) 2. Insert canonical link function for θ 3. Obtain θ̂ 26 38
- Likelihood Theory To get uncertainty estimate of θ̂ (its variance), we can take the second derivative of b(θ) with respect to θ such that ∂2 ∂θ2 b(θ) = E[(y − E[y])2] We can then re-write the variance as1 1 a2(ψ) var[y] → var[y] = a(ψ) ∂2 ∂θ2 b(θ) 1 It’s useful to re-write canonical form to include a scale parameter, a(ψ). When a(ψ) = 1, then ∂2 ∂θ2 b(θ) is unaltered, (y|θ) = exp[ yθ−b(θ) a(ψ) + c(y, ψ). 27 38
- Likelihood Theory Ex: Poisson We will also use canonical equation that includes a scale parameter for Poisson We know that inverse of canonical link gives us b(θ) = exp[θ] = µ, which we will insert in exp [ylog(µ) − µ − log(y!)] (3) a(ψ) ∂2 ∂θ2 b(θ) = 1 ∂2 ∂θ2 expθ|θ= log(µ) = exp(log(µ)) θ̂ = µ (4) 28 38
- Notation side note: ∝ versus = As Fisher defines it, likelihood is proportional to joint density of data given parameter value(s) I This is important in distinguishing likelihood from inverse probability or Bayesian approaches I However, “likelihood function” that we maximize is equal to joint density of data When talking about a likelihood function that will be maximized, we’ll use L(θ|y) = Q f(y|θ) from now on I But we’ll remember that proportionality means we can only compare relative sizes of likelihoods I Value of likelihood has no intrinsic scale and so is essentially meaningless except in comparison to other likelihoods 29 38
- From parameter to estimate: Link Functions We have essentially created a dependency connecting linear predictor and θ (via µ in our Poisson example) We can begin by making a generalization where V = Xβ + e such that V represents a stocastic component, X denotes model matrix, and β are estimated coefficients We can then denote expected value as a linear structure, E[V] = θ = Xβ 30 38
- From parameter to estimate: Link Functions Let’s now imagine that expected value of stocastic component is some function, g(µ) that is invertible Information from explanatory variables is now expressed only through link (Xβ) to linear predictor, θ = g(µ), which is controlled by link function, g() We can then extend Generalized linear model to accomodate non-normal response functions by transforming functions linearly This is achieved by taking inverse of link function, which ensures Xβ̂ maintains linearity assumption required of standard linear models g−1 (g(µ)) = g−1 (θ) = g−1 (Xβ) = µ = E[Y] 31 38
- Basics of MLE: Setup Begin with likelihood function Function of parameters that represents probability of witnessing observed data given value of parameter Likelihood function : P(Y = y) = Y f(yi|θ) = L(yi|θ) 32 38
- Basics of MLE: Setup Awesome, and...? So far, we have a way to think about I Which distributions we want to work with I How to characterise center & spread I Link data to those moments I Now, we need a way to actually calculate our estimates 33 38
- Basics of MLE: Setup Maximum likelihood estimate (MLE) is value of parameter that gives largest probability of observing data I Score function u(θ) is derivative of log-likelihood function with respect to the parameters I Fisher information var(u(θ)) measures uncertainty of estimate, θ̂ I To find Fisher information, take second derivative of likelihood function 34 38
- Basics of MLE: Computational Estimation MLE is typically found by using Newton-Raphson method, which is an iterative process of mode finding I More on this next week! We begin by estimating k-dimensional β̂ estimates by performing an iterative least squares method with diagonal elements of an A matrix of weights These diagonal element are typically Fisher information of exponential family distribution 35 38
- Wrap-up What is exponential family form? What is a link function? Why are we performing MLE? 36 38
- Next week Unfortunately there isn’t a closed form solution for β (except in very special cases) Newton-Raphson method is an iterative method that can be used instead Computationally convenient to solve on each iteration by weighted least squares 37 38
- Class business Read required (and suggested) online materials Problem set # 1 is up on GitHub Next time, we’ll talk about how to actually maximise our likelihood functions ! 38 / 38