• Save
variational bayes in biophysics
Upcoming SlideShare
Loading in...5
×
 

variational bayes in biophysics

on

  • 456 views

talk by chris.wiggins@columbia.edu on variational methods / graphical modeling in biophysics. comments appreciated.

talk by chris.wiggins@columbia.edu on variational methods / graphical modeling in biophysics. comments appreciated.

Statistics

Views

Total Views
456
Views on SlideShare
447
Embed Views
9

Actions

Likes
1
Downloads
0
Comments
0

1 Embed 9

https://twitter.com 9

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

variational bayes in biophysics variational bayes in biophysics Presentation Transcript

  • bio+stats vbem/networks hierarchicalvariational and hierarchical modelingfor biological datachris wigginscolumbiaapril 23, 2012chris.wiggins@columbia.edu 4/23/12Chris Wiggins• APAM: Department of Applied Physics and Applied Mathematics;• C2B2: Center for Computational Biology and Bioinformatics;• CISB: Columbia University Initiative in Systems Biology• ISDE: Institute for Data Sciences and EngineeringColumbia UniversitySeptember 28, 2012
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionthanks. . .- jake hofman (vbmod,vbfret)- jonathan bronson (vbfret)- jan-willem van de meent (hfret)- ruben gonzalez (vbfret, hfret)for more info:- vbfret.sourceforge.net- vbmod.sourceforge.net- hfret.sourceforge.net (soon)chris.wiggins@columbia.edu 4/23/12BMC bioinformatics, 2010;PNAS 2009;Biophysical Journal 2009;
  • bio+stats vbem/networks hierarchical1 biology and statisticsgenomicsgenerative modeling2 variational/biological networksvariational Bayesian expectation maximizationinferencemodel selection3 hierarchical/time seriesbiological challengesinferencemodel selectionchris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical genomics generative modelingbiology and statistics:chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical genomics generative modelinggenomics:chris.wiggins@columbia.edu 4/23/12plos comp bio `10, nyas `07, bmc bioinfo `06a, bmc bioinfo `06b,bioinfo `04, regulatory genomics 04, IEEE `05; NIPS (MLCB`03, `06 )
  • bio+stats vbem/networks hierarchical genomics generative modelinggenerative modeling:chris.wiggins@columbia.edu 4/23/12bmc bioinfo 10, PNAS`09, biophysj`09, PNAS`07, PNAS`06 NIPS(MLCB 09, 10, 11) IEEE sig. proc.`12; plos one `08,cell `07,prl `06, jcs`06, biophys j `06pami `08, prl `08, NIPS (MLCB08),PNAS `05, bmc bioinfo `04
  • bio+stats vbem/networks hierarchical vbem inference model selectionvariational/biological networks:- variational bayesian expectation maximization- inference- model selectionchris.wiggins@columbia.edu 4/23/12
  • introduction formulation results extensions motivation historyintroduction:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netHartwell, Hopfield, Leibler and MurrayNATURE|VOL 402 | SUPP | 2DECEMBER 1999 | www.nature.com
  • introduction formulation results extensions motivation historymotivation:community detection in networkssocial networksbiological networksproblem: over-fitting/resolution limitchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions motivation historyhistory:by communitymath/cs: spectral methods (Fiedler ’74, Shi + Malik ’00)math/cs: clustering generally (Taskar, Koller, Getoor)physics: modularitycommon thread: test w/ stochastic block model (’76, ’83)ergo: use as inference tool (Hastings 0604429,Newman+Liecht 061148)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions generative model max likelihood max evidence algoformulation:generative modelmaximum likelihoodmaximum evidencecomplexity control. . .variational/mft. . .algorithmin physics: “test hamiltonian”in ML “variational bayesian methods” (Jordan, Mackay)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions generative model max likelihood max evidence algogenerative model:foreach node roll K-sided die with bias π to choosezi {1, . . . , K}foreach edge flip coin with bias ϑ+ if zi = zj , else ϑ−draw edge if coin lands heads upchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netStochastic block models (Holland, Laskey, Leinhardt 1983; Wang and Wong, 1987)i≠jzi zjAijπθ
  • introduction formulation results extensions generative model max likelihood max evidence algogenerative model. . . (bis)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netDie rolling, coin flipping, and priors: where counts are:non-edges withinmodulesedges withinmodulesedges betweenmodulesnon-edgesbetween modulesnodes in eachmodule
  • introduction formulation results extensions generative model max likelihood max evidence algomax likelihood:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netH ⇥ ln p(A, ⌦z|⌦⇤, ⌦⇥) =i,j(JLAij JG) zi,zj +Kµ=1hµNi=1zi,µJG ⇥ ln ⇥c/⇥dJL ⇥ ln(1 ⇥d)/(1 ⇥c) + JGhµ ⇥ ln µExtends Newman (2004, 2006), Hastings (2006), Bornholdt & Reichardt (2006)•Die rolling, coin flipping <-> infinite-range spin-glass Potts model:
  • introduction formulation results extensions generative model max likelihood max evidence algoformulation:generative modelmaximum likelihoodmaximum evidencecomplexity control. . .variational/mft. . .algorithmin physics: “test hamiltonian”in ML “variational bayesian methods” (Jordan, Mackay)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions generative model max likelihood max evidence algoformulation:generative modelmaximum likelihoodmaximum evidencecomplexity control. . .variational/mft. . .algorithmin physics: “test hamiltonian”in ML “variational bayesian methods” (Jordan, Mackay)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netIncreasing complexity
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.nethttp://research.microsoft.com/~minka/statlearn/demo/
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netcf. “BIC” Schwartz, 1978
  • introduction formulation results extensions generative model max likelihood max evidence algogenerative model:foreach node roll K-sided die with bias π to choosezi {1, . . . , K}foreach edge flip coin with bias ϑ+ if zi = zj , else ϑ−draw edge if coin lands heads upchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netStochastic block models (Holland, Laskey, Leinhardt 1983; Wang and Wong, 1987)i≠jzi zjAijπθ
  • introduction formulation results extensions generative model max likelihood max evidence algogenerative model:foreach node roll K-sided die with bias π to choosezi {1, . . . , K}foreach edge flip coin with bias ϑ+ if zi = zj , else ϑ−draw edge if coin lands heads upchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netStochastic block models (Holland, Laskey, Leinhardt 1983; Wang and Wong, 1987)i≠jzi zjAijπθ cn
  • introduction formulation results extensions generative model max likelihood max evidence algogenerative model. . . (bis)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netDie rolling, coin flipping, and priors: where counts are:non-edges withinmodulesedges withinmodulesedges betweenmodulesnon-edgesbetween modulesnodes in eachmodule
  • introduction formulation results extensions generative model max likelihood max evidence algogenerative model. . . (bis)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netDie rolling, coin flipping, and priors: where counts are:non-edges withinmodulesedges withinmodulesedges betweenmodulesnon-edgesbetween modulesnodes in eachmodule
  • introduction formulation results extensions generative model max likelihood max evidence algomax likelihood:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netExtends Newman (2004, 2006), Hastings (2006), Bornholdt & Reichardt (2006)•Die rolling, coin flipping <-> infinite-range spin-glass Potts model:•Infer distributions over spin assignments, coupling constants, andchemical potentials and find number of occupied spin statesJG ⇥ ln ⇥c/⇥dJL ⇥ ln(1 ⇥d)/(1 ⇥c) + JGhµ ⇥ ln µ•Die rolling, coin flipping <-> infinite-range spin-glass Potts model:•Infer distributions over spin assignments, coupling constants, andchemical potentials and find number of occupied spin statesH ⇥ ln p(A, ⌦z|⌦⇤, ⌦⇥) =i,j(JLAij JG) zi,zj +Kµ=1hµNi=1zi,µ
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netExtends Newman (2004, 2006), Hastings (2006), Bornholdt & Reichardt (2006)•Die rolling, coin flipping <-> infinite-range spin-glass Potts model:•Infer distributions over spin assignments, coupling constants, andchemical potentials and find number of occupied spin statesH ⇥ ln p(A, ⌦z|⌦⇤, ⌦⇥) =i,j(JLAij JG) zi,zj +Kµ=1hµNi=1zi,µJG ⇥ ln ⇥c/⇥dJL ⇥ ln(1 ⇥d)/(1 ⇥c) + JGhµ ⇥ ln µp(A|K) =⇥z⇥d⌦⇥d⌦⇥ p(A,⌦z,⌦⇥, ⌦) =⇥z⇥d⌦⇥d⌦⇥ e Hp(⌦)p(⌦⇥)
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netExtends Newman (2004, 2006), Hastings (2006), Bornholdt & Reichardt (2004 & 2006)•Die rolling, coin flipping <-> infinite-range spin-glass Potts model:•Infer distributions over spin assignments, coupling constants, andchemical potentials and find number of occupied spin statesH ⇥ ln p(A, ⌦z|⌦⇤, ⌦⇥) =i,j(JLAij JG) zi,zj +Kµ=1hµNi=1zi,µJG ⇥ ln ⇥c/⇥dJL ⇥ ln(1 ⇥d)/(1 ⇥c) + JGhµ ⇥ ln µp(A|K) =⇥z⇥d⌦⇥d⌦⇥ p(A,⌦z,⌦⇥, ⌦) =⇥z⇥d⌦⇥d⌦⇥ e Hp(⌦)p(⌦⇥)Can do integrals,but sum isintractable, O(KN);use mean-field
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net• Gibbs’/Jensen’s inequality (log of expected value bounds expected value of log) for any distribution qp(A|K) =⇥z⇥d⌦⇥d⌦⇥ p(A,⌦z,⌦⇥, ⌦) =⇥z⇥d⌦⇥d⌦⇥ e Hp(⌦)p(⌦⇥)Variational Bayes (MacKay, Jordan, Ghahramani, Jaakola, Saul 1999; cf. Feynman 1972)
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netwhy would you do this? (A1):Beal, 2003
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netwhy would you do this? (A2):Beal, 2003
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netwhy would you do this? (A3):Beal, 2003
  • introduction formulation results extensions generative model max likelihood max evidence algomax evidence:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net• Gibbs’/Jensen’s inequality (log of expected value bounds expected value of log) for any distribution qVariational Bayes (MacKay, Jordan, Ghahramani, Jaakola, Saul 1999; cf. Feynman 1972)• F is a functional of q; find approximation to posterior by optimizing approximation toevidence• Take q(z, π, θ)=q(z)q(π)q(θ); Qiμ is probability node i in module μ where expected countsare:
  • introduction formulation results extensions generative model max likelihood max evidence algoalgo:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netwhere expected countsare:
  • introduction formulation results extensions generative model max likelihood max evidence algoalgo:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netwhere expected countsare:
  • introduction formulation results extensions generative model max likelihood max evidence algoalgo:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions generative model max likelihood max evidence algoalgo:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netsuggests hard limit in step 3; sparse in step 1
  • introduction formulation results extensions run time consistency good vs easy real dataresults:run timeconsistencyrequired plot: good vs. easyreal datakaratebiologyamerican footballchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions run time consistency good vs easy real datarun time:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net• Main loop runtime for 104 nodes in MATLAB ~30 seconds
  • introduction formulation results extensions run time consistency good vs easy real dataconsistency:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions run time consistency good vs easy real dataconsistency:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1024681012θN=8, K=2, distribution after 2 iterationsp(θ+)p(θ−)
  • introduction formulation results extensions run time consistency good vs easy real dataconsistency:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net• K=4?• Automatic complexity control: probability of occupation for extraneous modulesgoes to zero
  • introduction formulation results extensions run time consistency good vs easy real dataconsistency:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net• K=4?• Automatic complexity control: probability of occupation for extraneous modulesgoes to zero
  • introduction formulation results extensions run time consistency good vs easy real dataconsistency:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netThe “resolution limit” problem1234557678 910111213141516171819202122232425262728293031323334353637383940414243 444546474849505152535455565859601234557678 910111213141516171819202122232425262728293031323334353637383940414243 44454647484950515253545556585960Variational BayesGirvan-Newmanmodularity
  • introduction formulation results extensions run time consistency good vs easy real dataconsistency:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netThe “resolution limit” problem10 12 14 16 18 208101214161820KtrueK*K*=KtrueVariational BayesModularity optimization10 12 14 16 18 200.720.740.760.780.80.820.84KtrueGNmodularityResolution limit problem on ring of 4−node cliquesSingle−clique communities (correct)Double−clique communities (incorrect)GN modularity (Clauset’s algorithm)Girvan-Newman modularity or Potts model w/ fixed parameters suffers from a resolution limit,where size of detected modules depends on network sizeFortunato et. al. (2007), Kumpula et. al. (2007),
  • introduction formulation results extensions run time consistency good vs easy real datagood vs easy:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensions run time consistency good vs easy real datareal data:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net• Correctly infer K=12 conferencesValidation: NCAA football schedule12 3456789101112131415161718192021222324252627282930313233343536373839404142434445 464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115nodes: teamsedges: gamesshape: conferencecolor: inferred module
  • introduction formulation results extensions run time consistency good vs easy real datareal data:chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netAPS march meeting 2008superconductivity(experimentalists)Nanotubes, Graphenesuperconductivity(theorists)
  • introduction formulation results extensionsextensions:model extensionsfull SBM (done)hierarchical model p(Aij = 1|zi = zi ± 1)hierarchical modeling (ensemble of graphs)p(D|u, K) = ΠLi dϑi p(D|ϑi )p(ϑi |u, K)Rdembedding (latent are real)more ‘rigorous’ SOM?‘correct’ for degree (allow variable affinity) (cf. Bader, Karrer)algorithm extensionsBP (see earlier talks)map-reducecvmod (model selection via cross validation)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net• Nodes belong to “blocks” ofvarying size• Roll die for assignment ofnodes to blocks• Probability of edge between twonodes depends only on blockmembership• Flip (one of K2) coins for edges• Result: mixture of Erdos-Renyigraphs0 20 40 60 80 100 120020406080100120nz = 2275adjacency matrix
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netvs
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net0 20 40 60 80 100 120020406080100120nz = 2803adjacency matrix
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netvs
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net>> vbsbm_vs_vbmod(0)running vbmod ...Elapsed time is 1.136925 seconds.running vbsbm ...Elapsed time is 1.398904 seconds.Fmod=13089.158019 Fsbm=13144.445782vbmod wins
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net>> vbsbm_vs_vbmod(0.25)running vbmod ...Elapsed time is 1.557298 seconds.running vbsbm ...Elapsed time is 1.759527 seconds.Fmod=20457.142416 Fsbm=19457.306022vbsbm wins
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net>> vbsbm_vs_vbmod(0.5)running vbmod ...Elapsed time is 2.624886 seconds.running vbsbm ...Elapsed time is 1.440242 seconds.Fmod=26133.351210 Fsbm=23921.797625vbsbm wins
  • introduction formulation results extensionsfull SBM. . .probability of edge depends only on block membership:p(Aij |zi = µ, zj = ν) = ϑµνchris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net• Using same framework we can compare theunconstrained and full stochastic block models via p(D|M,K*)0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.100.10.20.30.40.50.60.70.80.91perturbation to constrained modelwinpercentageforunconstrainedmodel0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.100.10.20.30.40.50.60.70.80.91perturbation to constrained modelwinpercentageforunconstrainedmodel0 20 40 60 80 100 120020406080100120nz = 2100adjacency matrix0 20 40 60 80 100 120020406080100120nz = 2048adjacency matrix0 20 40 60 80 100 120020406080100120nz = 2108adjacency matrix
  • introduction formulation results extensionsextensions:model extensionsfull SBM (done)hierarchical model p(Aij = 1|zi = zi ± 1)hierarchical modeling (ensemble of graphs)p(D|u, K) = ΠLi dϑi p(D|ϑi )p(ϑi |u, K)Rdembedding (latent are real)more ‘rigorous’ SOM?‘correct’ for degree (allow variable affinity) (cf. Bader, Karrer)algorithm extensionsBP (see earlier talks)map-reducecvmod (model selection via cross validation)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensionsextensions:model extensionsfull SBM (done)hierarchical model p(Aij = 1|zi = zi ± 1)hierarchical modeling (ensemble of graphs)p(D|u, K) = ΠLi dϑi p(D|ϑi )p(ϑi |u, K)Rdembedding (latent are real)more ‘rigorous’ SOM?‘correct’ for degree (allow variable affinity) (cf. Bader, Karrer)algorithm extensionsBP (see earlier talks)map-reducecvmod (model selection via cross validation)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensionsextensions:model extensionsfull SBM (done)hierarchical model p(Aij = 1|zi = zi ± 1)hierarchical modeling (ensemble of graphs)p(D|u, K) = ΠLi dϑi p(D|ϑi )p(ϑi |u, K)Rdembedding (latent are real)more ‘rigorous’ SOM?‘correct’ for degree (allow variable affinity) (cf. Bader, Karrer)algorithm extensionsBP (see earlier talks)map-reducecvmod (model selection via cross validation)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensionsextensions:model extensionsfull SBM (done)hierarchical model p(Aij = 1|zi = zi ± 1)hierarchical modeling (ensemble of graphs)p(D|u, K) = ΠLi dϑi p(D|ϑi )p(ϑi |u, K)Rdembedding (latent are real)more ‘rigorous’ SOM?‘correct’ for degree (allow variable affinity) (cf. Bader, Karrer)algorithm extensionsBP (see earlier talks)map-reducecvmod (model selection via cross validation)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netStochastic block models (Holland, Laskey, Leinhardt 1983; Wang and Wong, 1987)i≠jzizjAijπθ cn
  • introduction formulation results extensionsextensions:model extensionsfull SBM (done)hierarchical model p(Aij = 1|zi = zi ± 1)hierarchical modeling (ensemble of graphs)p(D|u, K) = ΠLi dϑi p(D|ϑi )p(ϑi |u, K)Rdembedding (latent are real)more ‘rigorous’ SOM?‘correct’ for degree (allow variable affinity) (cf. Bader, Karrer)algorithm extensionsBP (see earlier talks)map-reducecvmod (model selection via cross validation)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.netStochastic block models (Holland, Laskey, Leinhardt 1983; Wang and Wong, 1987)i≠jzizjAijπθ cnL
  • introduction formulation results extensionsextensions:model extensionsfull SBM (done)hierarchical model p(Aij = 1|zi = zi ± 1)hierarchical modeling (ensemble of graphs)p(D|u, K) = ΠLi dϑi p(D|ϑi )p(ϑi |u, K)Rdembedding (latent are real)more ‘rigorous’ SOM?‘correct’ for degree (allow variable affinity) (cf. Bader, Karrer)algorithm extensionsBP (see earlier talks)map-reducecvmod (model selection via cross validation)chris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • introduction formulation results extensionsfor more info. . .code: MATLAB & python (inc. “full” SBM) (vbmod.sf.net)paper: arxiv 08 / prl 08Hofman soon to come (not by me)code in C++, inc. full ‘vblabel propagation’ algotwitter-scale analysischris.wiggins@columbia.edu 22.2.12 vbmod.sourceforge.net
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhierarchical/time series:- biological challenges- inference- model selectionchris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Jan-Willem van de Meent, Ruben Gonzalez, Chris WigginsColumbia University
  • Ramakrishnan et al – http://www.mrc-lmb.cam.ac.uk/ribo/
  • Ramakrishnan et al – http://www.mrc-lmb.cam.ac.uk/ribo/
  • Ramakrishnan et al – http://www.mrc-lmb.cam.ac.uk/ribo/
  • Ramakrishnan et al – http://www.mrc-lmb.cam.ac.uk/ribo/
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12FRET = cy5 / (cy3 + cy5)Tinoco and Gonzalez, Genes Dev, 2011
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Tinoco and Gonzalez, Genes Dev, 2011
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12UnboundEF-G boundTinoco and Gonzalez, Genes Dev, 2011 Fei et al, PNAS, 2009(short-lived GS1 states correspond to an EF-G + GDPNP binding event)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12UnboundEF-G boundTinoco and Gonzalez, Genes Dev, 2011 Fei et al, PNAS, 2009
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/121. Identify states2. Estimate Kinetic Rates3. Average over many time series4. Detect subpopulations
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12FRET Signal
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12FRET SignalHistogram
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12FRET SignalHistogram
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12FRET SignalHistogram
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12FRET SignalHistogramIdea: Find probability of belonging to each state
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Expectation Maximization1. calculate p(z | x, θi)2. calculate θi+1 from p(z | x, θi)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Log-LikelihoodL = log p(x ￿ θ) = log￿￿zp(x, z ￿ θ)￿Expectation Maximization1. calculate p(z | x, θi)2. calculate θi+1 from p(z | x, θi)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Learned TruthAccurate for occupancy of states,not so good for rate estimates
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12p(x, z ￿ µ, σ, π) = p(x ￿ z, µ, σ)p(z ￿ π)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12probability of state depends on previous statep(zt+￿ =l ￿ zt =k) = Akl
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12p(z￿ =k) = πk
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12p(zt+￿ =l ￿ zt =k) = Akl
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12p(xt ￿ zt = k) = N(xt ￿ µk , σk)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Learned RealWe’ve learned:parameters: θ = {µ, σ, π, A} states: p(z | x, θ)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/122 States 3 States
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Log-LikelihoodL = log p(x ￿ θ) = log￿￿zp(x, z ￿ θ)￿Log-EvidenceL = log p(x ￿ u) = log￿￿z∫ dθ p(x, z ￿ θ)p(θ￿u)￿Log-Evidence
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Log-LikelihoodL = log p(x ￿ θ) = log￿￿zp(x, z ￿ θ)￿Log-EvidenceL = log p(x ￿ u) = log￿￿z∫ dθ p(x, z ￿ θ)p(θ￿u)￿Prior
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Log-LikelihoodL = log p(x ￿ θ) = log￿￿zp(x, z ￿ θ)￿Log-EvidenceL = log p(x ￿ u) = log￿￿z∫ dθ p(x, z ￿ θ)p(θ￿u)￿Ensemble
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Log-LikelihoodL = log p(x ￿ θ) = log￿￿zp(x, z ￿ θ)￿Log-Evidencebest model has highest average likelihoodL = log p(x ￿ u) = log￿￿z∫ dθ p(x, z ￿ θ)p(θ￿u)￿Log-Evidence
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Log-Evidence31L = log p(x ￿ u) = log￿￿z∫ dθ p(x, z ￿ θ)p(θ￿u)￿Lower BoundL = ￿z∫ dθ q(z)q(θ ￿ w)log￿p(x, z, θ ￿ u)q(z)q(θ ￿ w)￿≥ log p(x ￿ u)q(z)q(θ ￿ w) ￿ p(z, θ ￿ x)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231Lower bound tight for true posteriorL = ￿z∫ dθ p(z, θ ￿ x)log￿p(x, z, θ ￿ u)p(z, θ ￿ x)￿= ￿z∫ dθ p(z, θ ￿ x)log[p(x ￿ u)]= log p(x ￿ u)L = log p(x ￿ u) − Dkl [q(z)q(θ ￿ w) ￿￿ p(z, θ ￿ x)]
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231We’ve learned:parameters: q(θ | w) states: p(z | x, θ)δLnδq(zn)= ￿VBEM UpdatesδLnδq(θn)= ￿
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231variability: photophysical/experimental
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231Hierarchical Updates∂∂u￿nLn = ￿
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1231Hierarchical Updates∂∂u￿nLn = ￿ “two-stage PEB model/CIHM”-Kass & Steffey JASA 1989
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/122. Update p(θ | u)Until Σ Ln convergesUntil Ln converges• Update q(zn)• Update q(θn | wn)1. Run VBEM on each traceδLnδq(zn)= ￿Hierarchical UpdatesVBEM UpdatesδLnδq(θn)= ￿∂∂u￿nLn = ￿
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/122. Update p(θ | u)Until Σ Ln convergesUntil Ln converges• Update q(zn)• Update q(θn | wn)1. Run VBEM on each traceWe’ve learned:p(θn, zn | xn) ≃ q(θn) q(zn)(for each trace)p(θ | u)(for ensemble)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12UnboundEF-G boundTinoco and Gonzalez, Genes Dev, 2011 Fei et al, PNAS, 2009
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12ξntkl = p(znt = k, znt+￿ = l ￿ xn)1. Run mixture model on posterior countsp(ξn￿A) = ￿tklAξntklklp(ξn ￿um) = ∫ dA p(A￿um)p(ξn ￿ A)2. Rerun with M x K block-diagonal formuA=￿￿￿￿￿￿￿￿￿uA￿uA￿￿uAM￿￿￿￿￿￿￿￿￿
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12τfastτslow248163264
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12τfastτslow248163264reality
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12no EF-G 50 nM EF-G 500 nM EF-GFei, Bronson, Hofman, Srinivas, Wiggins, Gonzalez, PNAS, 2009
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12no EF-G 50 nM EF-G 500 nM EF-Gp(zk) ∼ e−Gk ￿kB Tlog p(zk) − log p(zl ) = −(Gk − Gl )￿kBT + cst.
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12no EF-G 50 nM EF-G 500 nM EF-Gp(zk) ∼ e−Gk ￿kB TΔ∆G=logit(p)
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12no EF-Gbound fraction and life-times
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/125 nM EF-Gbound fraction and life-times
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/1250 nM EF-Gbound fraction and life-times
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12250 nM EF-Gbound fraction and life-times
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12500 nM EF-Gbound fraction and life-times
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionmodel selection:chris.wiggins@columbia.edu 4/23/12
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Low Noise, UnderfittedInf Out <-> Inf In
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Low Noise, CorrectOut vs In
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12Low Noise, OverfittedInf Out <-> Inf In
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12High Noise, UnderfittedInf Out <-> Inf In
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12High Noise, CorrectInf Out <-> Inf In
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12High Noise, OverfittedInf Out <-> Inf In
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12High Noise, OverfittedInf Out <-> Inf In
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionhfret. . .chris.wiggins@columbia.edu 4/23/12the future, in progress:X
  • bio+stats vbem/networks hierarchical biological challenges inference model selectionthanks. . .- jake hofman (vbmod,vbfret)- jonathan bronson (vbfret)- jan-willem van de meent (hfret)- ruben gonzalez (vbfret, hfret)for more info:- vbfret.sourceforge.net- vbmod.sourceforge.net- hfret.sourceforge.net (soon)chris.wiggins@columbia.edu 4/23/12BMC bioinformatics, 2010;PNAS 2009;Biophysical Journal 2009;
  • traditional role of statistics in biophysics“if your experiment needsstatistics, you ought tohave done a betterexperiment”-lord rutherford