Overview of the course. Introduction to image sciences, image processing and computer vision. Basics of machine learning, terminologies, paradigms. No-free lunch theorem. Supervised versus unsupervised learning. Clustering and K-Means. Classification and regression. Linear least squares and polynomial curve fitting. Model complexity and overfitting. Curse of dimensionality. Dimensionality reduction and principal component analysis. Image representation, semantic gap, image features, and classical computer vision pipelines.
A 3hrs intro lecture to Approximate Bayesian Computation (ABC), given as part of a PhD course at Lund University, February 2016. For sample codes see http://www.maths.lu.se/kurshemsida/phd-course-fms020f-nams002-statistical-inference-for-partially-observed-stochastic-processes/
Overview of the course. Introduction to image sciences, image processing and computer vision. Basics of machine learning, terminologies, paradigms. No-free lunch theorem. Supervised versus unsupervised learning. Clustering and K-Means. Classification and regression. Linear least squares and polynomial curve fitting. Model complexity and overfitting. Curse of dimensionality. Dimensionality reduction and principal component analysis. Image representation, semantic gap, image features, and classical computer vision pipelines.
A 3hrs intro lecture to Approximate Bayesian Computation (ABC), given as part of a PhD course at Lund University, February 2016. For sample codes see http://www.maths.lu.se/kurshemsida/phd-course-fms020f-nams002-statistical-inference-for-partially-observed-stochastic-processes/
A discussion on sampling graphs to approximate network classification functionsLARCA UPC
The problem of network classification consists on assigning a finite set of labels to the nodes of the graphs; the underlying assumption is that nodes with the same label tend to be connected via strong paths in the graph. This is similar to the assumptions made by semi-supervised learning algorithms based on graphs, which build an artificial graph from vectorial data. Such semi-supervised algorithms are based on label propagation principles and their accuracy heavily relies on the structure (presence of edges) in the graph.
In this talk I will discuss ideas of how to perform sampling in the network graph, thus sparsifying the structure in order to apply semi-supervised algorithms and compute efficiently the classification function on the network. I will show very preliminary experiments indicating that the sampling technique has an important effect on the final results and discuss open theoretical and practical questions that are to be solved yet.
Approximate Bayesian computation for the Ising/Potts modelMatt Moores
Bayes’ formula involves the likelihood function, p(y|theta), which is a problem when the likelihood is unavailable in closed form. ABC is a method for approximating the posterior p(theta|y) without evaluating the likelihood. Instead, pseudo-data is simulated from a generative model and compared with the observations. This talk will give an introduction to ABC algorithms: rejection sampling, ABC-MCMC and ABC-SMC. Application of these algorithms to image analysis will be presented as an illustrative example. These methods have been implemented in the R package bayesImageS.
This is joint work with Christian Robert (Warwick/Dauphine), Kerrie Mengersen and Christopher Drovandi (QUT).
The (fast) component-by-component construction of lattice point sets and polynomial lattice point sets is a powerful method to obtain quadrature rules for approximating integrals over the dimensional unit cube.
In this talk, we present modifications of the component-by-component algorithm and of the more recent successive coordinate search algorithm, which yield savings of the construction cost for lattice rules and polynomial lattice rules in weighted function spaces. The idea is to reduce the size of the search space for coordinates which are associated with small weights and are therefore of less importance to the overall error compared to coordinates associated with large
weights. We analyze tractability conditions of the resulting quasi-Monte Carlo rules, and show some numerical results.
Binary classification and linear separators. Perceptron, ADALINE, artifical neurons. Artificial neural networks (ANNs), activation functions, and universal approximation theorem. Linear versus non-linear classification problems. Typical tasks, architectures and loss functions. Gradient descent and back-propagation. Support Vector Machines (SVMs), soft-margins and kernel trick. Connexions between ANNs and SVMs.
In this article we consider macrocanonical models for texture synthesis. In these models samples are generated given an input texture image and a set of features which should be matched in expectation. It is known that if the images are quantized, macrocanonical models are given by Gibbs measures, using the maximum entropy principle. We study conditions under which this result extends to real-valued images. If these conditions hold, finding a macrocanonical model amounts to minimizing a convex function and sampling from an associated Gibbs measure. We analyze an algorithm which alternates between sampling and minimizing. We present experiments with neural network features and study the drawbacks and advantages of using this sampling scheme.
Inference for stochastic differential equations via approximate Bayesian comp...Umberto Picchini
Despite the title the methods are appropriate for more general dynamical models (including state-space models). Presentation given at Nordstat 2012, Umeå. Relevant research paper at http://arxiv.org/abs/1204.5459 and software code at https://sourceforge.net/projects/abc-sde/
A likelihood-free version of the stochastic approximation EM algorithm (SAEM)...Umberto Picchini
I show how to obtain approximate maximum likelihood inference for "complex" models having some latent (unobservable) component. With "complex" I mean models having a so-called intractable likelihood, where the latter is unavailable in closed for or is too difficult to approximate. I construct a version of SAEM (and EM-type algorithm) that makes it possible to conduct inference for complex models. Traditionally SAEM is implementable only for models that are fairly tractable analytically. By introducing the concept of synthetic likelihood, where information is captured by a series of user-defined summary statistics (as in approximate Bayesian computation), it is possible to automatize SAEM to run on any model having some latent-component.
ActiveBoard presentation for introducing area of a square. The ruler may not transfer into ActiveInspire well, so check it out before you present. I have also included notes for how I intended the presentation to be... er... presented.
A discussion on sampling graphs to approximate network classification functionsLARCA UPC
The problem of network classification consists on assigning a finite set of labels to the nodes of the graphs; the underlying assumption is that nodes with the same label tend to be connected via strong paths in the graph. This is similar to the assumptions made by semi-supervised learning algorithms based on graphs, which build an artificial graph from vectorial data. Such semi-supervised algorithms are based on label propagation principles and their accuracy heavily relies on the structure (presence of edges) in the graph.
In this talk I will discuss ideas of how to perform sampling in the network graph, thus sparsifying the structure in order to apply semi-supervised algorithms and compute efficiently the classification function on the network. I will show very preliminary experiments indicating that the sampling technique has an important effect on the final results and discuss open theoretical and practical questions that are to be solved yet.
Approximate Bayesian computation for the Ising/Potts modelMatt Moores
Bayes’ formula involves the likelihood function, p(y|theta), which is a problem when the likelihood is unavailable in closed form. ABC is a method for approximating the posterior p(theta|y) without evaluating the likelihood. Instead, pseudo-data is simulated from a generative model and compared with the observations. This talk will give an introduction to ABC algorithms: rejection sampling, ABC-MCMC and ABC-SMC. Application of these algorithms to image analysis will be presented as an illustrative example. These methods have been implemented in the R package bayesImageS.
This is joint work with Christian Robert (Warwick/Dauphine), Kerrie Mengersen and Christopher Drovandi (QUT).
The (fast) component-by-component construction of lattice point sets and polynomial lattice point sets is a powerful method to obtain quadrature rules for approximating integrals over the dimensional unit cube.
In this talk, we present modifications of the component-by-component algorithm and of the more recent successive coordinate search algorithm, which yield savings of the construction cost for lattice rules and polynomial lattice rules in weighted function spaces. The idea is to reduce the size of the search space for coordinates which are associated with small weights and are therefore of less importance to the overall error compared to coordinates associated with large
weights. We analyze tractability conditions of the resulting quasi-Monte Carlo rules, and show some numerical results.
Binary classification and linear separators. Perceptron, ADALINE, artifical neurons. Artificial neural networks (ANNs), activation functions, and universal approximation theorem. Linear versus non-linear classification problems. Typical tasks, architectures and loss functions. Gradient descent and back-propagation. Support Vector Machines (SVMs), soft-margins and kernel trick. Connexions between ANNs and SVMs.
In this article we consider macrocanonical models for texture synthesis. In these models samples are generated given an input texture image and a set of features which should be matched in expectation. It is known that if the images are quantized, macrocanonical models are given by Gibbs measures, using the maximum entropy principle. We study conditions under which this result extends to real-valued images. If these conditions hold, finding a macrocanonical model amounts to minimizing a convex function and sampling from an associated Gibbs measure. We analyze an algorithm which alternates between sampling and minimizing. We present experiments with neural network features and study the drawbacks and advantages of using this sampling scheme.
Inference for stochastic differential equations via approximate Bayesian comp...Umberto Picchini
Despite the title the methods are appropriate for more general dynamical models (including state-space models). Presentation given at Nordstat 2012, Umeå. Relevant research paper at http://arxiv.org/abs/1204.5459 and software code at https://sourceforge.net/projects/abc-sde/
A likelihood-free version of the stochastic approximation EM algorithm (SAEM)...Umberto Picchini
I show how to obtain approximate maximum likelihood inference for "complex" models having some latent (unobservable) component. With "complex" I mean models having a so-called intractable likelihood, where the latter is unavailable in closed for or is too difficult to approximate. I construct a version of SAEM (and EM-type algorithm) that makes it possible to conduct inference for complex models. Traditionally SAEM is implementable only for models that are fairly tractable analytically. By introducing the concept of synthetic likelihood, where information is captured by a series of user-defined summary statistics (as in approximate Bayesian computation), it is possible to automatize SAEM to run on any model having some latent-component.
ActiveBoard presentation for introducing area of a square. The ruler may not transfer into ActiveInspire well, so check it out before you present. I have also included notes for how I intended the presentation to be... er... presented.
Polynomials And Linear Equation of Two VariablesAnkur Patel
A complete description of polynomials and also various methods to solve the Linear equation of two variables by substitution, cross multiplication and elimination methods.
For polynomials it also contains the description of monomials, binomials etc.
Machine Learning can often be a daunting subject to tackle much less utilize in a meaningful manner. In this session, attendees will learn how to take their existing data, shape it, and create models that automatically can make principled business decisions directly in their applications. The discussion will include explanations of the data acquisition and shaping process. Additionally, attendees will learn the basics of machine learning - primarily the supervised learning problem.
BackTracking Algorithm: Technique and ExamplesFahim Ferdous
This slides gives a strong overview of backtracking algorithm. How it came and general approaches of the techniques. Also some well-known problem and solution of backtracking algorithm.
The X•Y Chart: Multiplication and CoordinatesJoe Speier
The X•Y Chart is a visual presentation of the single digit multiplication facts presented in the first quadrant of the Cartesian coordinate system. Its advantages over the traditional multiplication tables are outlined.
1. Using Algebraic Geometry for
Solving Polynomial Problems
in Computer Vision
David Nistér and Henrik Stewénius
2. Very Rough Outline
• 14:00-15:30 “1st
Round", administered by David.
• - What is this Tutorial About?
• - Motivation, RANSAC, Example Problems
• - Gröbner Bases
• - Action Matrix
• -15:30-16:00 Coffee break
• 16:00-18:00-“2nd
Round“, administered by Henrik.
• - Background Material
• - Gröbner Bases
• - Exercises in Macaulay2
• - Q & A
3. What is this Tutorial About?
• If you are interested in solving polynomial
equations, this tutorial is for you
• This tutorial is a ‘real’ tutorial
• The main focus will be on Gröbner bases
and how the theory of Gröbner bases can be
used to solve polynomial equations
5. RISC
Research
Institute for
Symbolic
Computation
Linz, Austria
Wolfgang Gröbner (1899-1980) Bruno Buchberger
1966: Ph.D. in mathematics. University of Innsbruck,
Dept. of Mathematics, Austria.
Thesis: On Finding a Vector Space Basis of the Residue Class Ring
Modulo a Zero Dimensional Polynomial Ideal (German).
Thesis Advisor: Wolfgang Gröbner
6. Bezout’s Theorem
2x2=4
Mixed Volume, see for example [CLO 1998]
provides a generalization for non-general polynomials.
With two variables,
a solution according to
the Bezout bound
can typically be
realized with resultants.
With three or more
variables, things
are less simple.
8. Various States of Mind You May Have, I:
• Skip the theory, what is this all about? –Answers
from David
1st
Quadrant2st
Quadrant
3rd
Quadrant 4th
Quadrant
Difficult TheoryEasy Theory
Useful Theory
Useless Theory
Useful but HardUseful and Easy
Super!
Useless and Easy
Useless and impossible
to penetrate – Unfortunately
survives
Algebraic Geometry and
Gröbner Bases
9. Proofs of the Power of Gröbner Bases
• So far, approx. 600 publications and 10 textbooks have been devoted to
Gröbner Bases.
• Gröbner Bases are routinely available in current mathematical software
systems like Mathematica, Maple, Derive, Magma, Axiom, etc.
• Special software systems, like CoCoa, Macaulay, Singular, Plural, Risa-Asir
etc. are, to a large extent, devoted to the applications of Gröbner Bases.
• Gröbner Bases theory is an important section in all international conferences
on computer algebra and symbolic computation.
• Gröbner Bases allow, for the first time, algorithmic solutions to some of the
most fundamental problems in algebraic geometry but are applied also in such
diverse areas as functional analysis, statistics, optimization, coding theory,
cryptography, automated theorem proving in geometry, graph theory,
combinatorial identities, symbolic summation, special functions, etc.
10. • Stop the handwaving, what is the rigorous
theory?
Answers from David and Henrik + books
Various States of Mind You May Have, II:
11. Suggested Literature
• D. Cox, J. Little, D. O’Shea, Ideals, Varieties,
and Algorithms, Second Edition, 1996.
• D. Cox, J. Little, D. O’Shea, Using Algebraic
Geometry, Springer 1998.
• T. Becker and Weispfennig, Gröbner Bases, A
Computational Approach to commutative Algebra,
Springer 1993.
12. Suggested Literature
• B. Buchberger, F. Winkler (eds.) Gröbner
Bases and Applications
Cambridge University Press, 1998.
• Henrik’s Thesis: H. Stewénius, Gröbner
Basis Methods for Minimal Problems in
Computer Vision, PhD Thesis, 2005
• Planned scaffolding paper, keep a lookout
on the tutorial web page
13. • Skip the theory, how do I use it? – Answers
from Henrik, Exercises in Macaulay 2.
Various States of Mind You May Have, III:
21. For Which Problems Did We Use
Gröbner Bases?
The Generalized
3-Point Problem
YesNo Yes, you bet
The 3 View 4-Point
Problem
8(4)
10
2048
The 5-Point Relative
Pose Problem
0 (or thousands)
47
3 View
Triangulation
Unknown Focal
Relative Pose
15
64
Generalized Relative
Pose
Microphone-Speaker
Relative Orientation
8-38-150-344-??
201. 5-Point Matlab Executable
Recent Developments on Direct Relative Orientation,
Henrik Stewenius, Christopher Engels, David Nister,
To appear in ISPRS Journal of Photogrammetry and Remote Sensing
www.vis.uky.edu/~dnister
202. 6-point generalized relative orientation (64 solutions) (Stewenius,
Nistér, Oskarsson and Åström, Omnivis ICCV 2005)
6-point relative orientation with common but unknown focal length
(15 solutions) (Stewenius, Nistér, Schaffalitzky and Kahl,
to appear at CVPR 2005)
Further Examples of Solved Problems
221. For Which Problems Did We Use
Gröbner Bases?
The Generalized
3-Point Problem
YesNo Yes, you bet
The 3 View 4-Point
Problem
8(4)
10
2048
The 5-Point Relative
Pose Problem
0 (or thousands)
47
3 View
Triangulation
Unknown Focal
Relative Pose
15
64
Generalized Relative
Pose
Microphone-Speaker
Relative Orientation
8-38-150-344-??
222. Camera Geometry
• Often leads to polynomial formulations,
or can at least very often be formulated in
terms of polynomial equations
224. Main Point
• Gröbner Basis Gives Action Matrix
(because it provides the ability to compute unique ‘smallest’ possible
unique remainders)
• Action Matrix Gives Solutions
Polynomial
Equations
Gröbner
Basis
Action
Matrix
Solutions
Intensively Studied
225. Algebraic Ideal
• I(p1 , … , pn)= The set of polynomials
generated by the input polynomials
(through additions and multiplications by a polynomial)
p and q in I => p+q in I
p in I => pq in I
The ideal I consists of ‘Almost’ all the polynomials implied
by the input polynomials
(More precisely, the radical of the ideal consists of all)I
226. Basis (for Ideal)
• A basis for the ideal J is a set of polynomials
{p1 , … , pn} such that J=I(p1 , … , pn)
227. Algebraic Variety
• The solution set
(the vanishing set of the input polynomials)
V(I)={x:I(x)=0}
More precisely
p(x)=0 for all p in I
228. Quotient Ring J/I
• The set of equivalence classes of
polynomials when only the values on V are
considered (i.e. polynomials are equivalent iff p(V)=q(V))
V(I)
p in J/I
229. Action Matrix
• For multiplication by polynomial on finite
dimensional solution space
V(I)
231. An ‘Equivalence’
Finding the Roots
of a Polynomial
Finding the Eigenvalues
of a Matrix
Compute Companion
Matrix
Compute
Characteristic Polynomial
Finding the Roots
of a Multiple
Polynomial Equations
Finding the Eigenvalues
of a Matrix
Compute Action Matrix in Quotient Ring
(Polynomials modulo Input Equations)
Compute
Characteristic Polynomial
Requires
Gröbner
Basis for
Input Equations
232. Companion Matrix
x2
1x3
x4
x5
x6
x7
+ a6x6
+a5x5
+ a4x4
+ a3x3
+ a2x2
+ a1x+a0
-a6
1
1
1
1
1
-a0-a4 -a3 -a2 -a1-a5
x
1
Multiplication by x modulo the seventh degree polynomial
can be expressed as left-multiplication by the matrix
240. Action Matrix
Multiplication by a polynomial q is a linear
operator Aq
(αp+βr)q=α(pq)+β(rq)
The matrix Aq is called the action matrix for
multiplication by q
244. Action Matrix
I
The values q(xi) of q at the solutions xi are the
eigenvalues of the action matrix
q(x1)b1
q(x2)b2
q(x0)b0
q in J
245. Action Matrix
The values q(xi) of q at the solutions xi are the
eigenvalues of the action matrix
If we choose q=y1 , the eigenvalues are the
solutions for y1
246. Action Matrix
b’(x)Aq p=q(x)b’(x)p
for all p in J/I and x in V(I)
b’=[r1 … ro]
b’(x)Aq =b’(x)q(x)
b(x) is a left nullvector of Aq corresponding to eigenvalue q(x)
247. Monomial Order
• Needed to define leading term of a polynomial
• Grevlex (Graded reverse lexicographical) order
usually most efficient
y_1
y_2
248. Gröbner Basis
• A basis for ideal I that exposes the leading
terms of I (hence unique well defined remainders)
• Easily gives the action matrix for
multiplication with any polynomial in the
quotient ring
y_1
y_2
249. A Reduced Gröbner Basis is a Basis
in the normal sense
• A polynomial in the ideal I can be written
as a unique combination of the polynomials
in a reduced Gröbner basis for I
• The monic Gröbner basis for I is unique
251. Gaussian Elimination
Exposes all the leading terms, which are simply
all the variables in the case of general linear equations
xn x1
252. Remember Row Operations:
• Multiplying a row by a scalar
• Subtracting a row from another
• Swap rows
Add:
• Multiplying a row by any polynomial
253. More General Elimination
With non-linear equations, there are relations between
the monomials that matter when multiplying
x2
xyy2
xy
Multiply by x
Multiply by y
x2
xyy2
xy
x2
xyy2
xy
254. Multiplying by a Scalar
p(x)
3.8p(x)
Transitions through zero remain
268. Elimination
Schedule
Approach
Pose Problem over R
Compute Gröbner basis
End
Compute Action Matrix
Solve Eigenproblem
Backsubstitute
Begin (online)
Pose Problem. Port to Zp
Compute number of solutions
End
Build matrix based
Gröbner basis code
Port to R
Begin (offline)
269. Prime Field Formulation
• Reals => Cancellation unclear
• Rationals => Grows unwieldy
• Prime Field => Cancellation clear, size is
limited, only small risk of incorrect
cancellation if prime is large
Now, lets look at the animation. In order to bring the three points to the three rays, we are allowed to translate which makes for three degrees of freedom, and rotate, which makes for three degrees of freedom.
If we assume that we have the aligment in place, and forget about the constraint from the third ray for a minute, what are two degrees of freedom that are left?
Well, if we consider the axis through the first two points, and we rotate the point configuration around the axis through the first two points, we will clearly not disturb the first two constraints. This causes the third point to move in a circle around the axis.
So what about the second degree of freedom? The key is to consider the perpendicular axis between the first two rays. If we perform a slight 3D transformation of everything so that we are looking straight down the perpendicular axis, like this, and we look at this in an orthographic projection,
then we find that we can apply Euclid’s proposition III:21.
We wish to keep the first two rays through their corresponding points. Euclid III:21 tells us that we can do this with a revolving motion that moves the perpendicular axis around in a circle. This is the second degree of freedom. The interesting thing to study is now what happens to the third ray when we perform this revolving motion.
It turns out that every point on the third ray moves around on a plane curve called a Limacon, like this. Moreover, the whole ray traces out a quartic surface. Being traced out by a line, the surface is of course ruled.
The solutions to the problem therefore correspond to the points where the quartic surface meets the circle.
The shape of the surface varies with the configuration of the first two points and the three rays. It is built up by limacons in each plane,
and has a curve of self-intersection, which turns out to be parts of a twisted cubic.
An interesting question is what happens to the geometry in the classical case.
It turns out that in the classical case, the problem attains a certain symmetry.
The surface becomes symmetric across the plane that appears like a waist in this figure. The self-intersection curve becomes a circle that looks like a waist, and an infinite vertical line. Moreover, the circle is also upright and symmetric with respect to the plane of the waist. So everything has reflection symmetry across the plane of the waist. Because of this, the solutions come in pairs in the classical case. The reflected solutions correspond to reflecting the projection centre across the plane of the three points.
Because of the reflection symmetry, it is possible to find the solutions in closed form, since there is a quartic polynomial whose roots correspond to pairs of solutions.