This document discusses applying machine learning techniques to automated theorem proving and formal proof checking. It begins by providing background on logic-based theorem proving and efforts to formally prove mathematical theorems. It then discusses using machine learning to help guide automated theorem provers by selecting optimal heuristics and recommending useful lemmas. The document concludes by noting the challenges of developing mathematical languages that are both natural for humans and amenable to formal verification.

1. Proving theorems: still a major test for AI
Dr Sean Holden
University of Cambridge
Department of Computer Science and Technology
and
Trinity College
Codiax 2019

2. How did I get here?
Machine learning research since 1989:
• Initially on computational learning theory, particularly sample complexity of
the cross-validation and other estimates.
• Applications in drug design with Glaxo-SmithKline and others.
• Organelle proteomics in collaboration with the Cambridge Department of Bio-
chemistry.
• Various other areas in Bayesian inference, survey propagation, protein interac-
tions, gene expression and so on.
Most recently: I’ve been working on the question of…

3. How did I get here?
How can machine learning be applied to proving mathematical theorems?

4. How did I get here? Principia Mathematica…
Logicism: is all mathematics just logic?
• Alfred North Whitehead and Bertrand Russell attempted to demonstrate this.
• Three volumes, published in 1910, 1912 and 1913.
• Initially, very basic logical axioms. Then a systematic development.
• Proofs in their system can easily be searched for by an automated theorem
prover.
Famously, the important theorem
1 + 1 = 2
appears on page 83 of Volume 2.
Take-home message: being precise about mathematics is hard!
However, this has formed a fundamental challenge in AI since its earliest days:
the General Problem Solver, Herbert A. Simon, Cliff Shaw and Allen Newell, 1959.

5. Proofs in propositional/Boolean logic
Propositional logic:
• Variables can be true (T) or false (F).
• I have connectives ∧ (AND), ∨ (OR) and ¬ (NOT).
1. ¬x is T when x is F, and F when x is T.
2. x1 ∧ x2 is T precisely when both x1 and x2 are T.
3. x1 ∨ x2 is T precisely when x1 or x2 (or both) are T.
• For example, the formula (x1 ∨ x2) ∧ x3 is T when x1 = T, x2 = F and x3 = T.
Here is the game: I give you a HUGE formula with n variables:
(x1 ∨ x2 ∨ ¬x3) ∧ (¬x2 ∨ · · · ∨ x10) ∧ · · · ∧ (¬x1,000,000)
and you have to tell me whether or not there is a way of setting the variables to
make the formula T.
This is the satisfiability (SAT) problem.
There is no known method that can guarantee to be better than testing all 2n
possibilities!

6. Proofs in first-order logic
Am I a WEIRDO?
∀x . Pierced(x) ∧ Male(x) −→ Weirdo(x)
Pierced(sean)
Male(sean)
Does Weirdo(sean) follow?
We should probably agree that it does. (Hooray!)

7. First-order logic
The reason this is hard is simple:
{W(sean)}
{}
{P(sean)} {M(sean)}
x = sean
The set of clauses grows
There is a choice of which pair of clauses to resolve
{¬W(sean)}{¬P(x), ¬M(x), W(x)}
{¬M(sean), W(sean)}
And it’s worse than it seems:
• The full problem is semidecidable.
• (You can prove a valid theorem, but you might not be able to disprove an invalid
one.)
• The propositional subproblem is NP-complete.

8. Have computers managed to prove anything? The Robbins conjecture
So automated theorem proving is known to be hard. But has it been used to prove
any original mathematics?
Robbins algebras:
• Commutative:
∀a, b . a ∨ b = b ∨ a
• Associative:
∀a, b, c . (a ∨ b) ∨ c = a ∨ (b ∨ c)
• Robbins axiom
∀a, b . ¬(¬(a ∨ b) ∨ ¬(a ∨ ¬b)) = a
Robbins conjecture:
All Robbins algebras are Boolean.
The proof was found by William McClune using the EQP theorem prover. (Jour-
nal of Automated Reasoning, 1997.)

9. So what’s the problem?
So, what’s the problem?
Although the Robbins conjecture represents a proof of a genuine theorem by an
automated theorem prover:
• It is something of an isolated result.
• It essentially represents an application of brute-force search.
• The proof has little if any resemblance to the kind of proof a person might
construct.
• It’s not enough to impress the mathematicians.
What will it take to impress them?
We could aim to prove something BIG!
We could aim to prove multiple results with the same system without human
guidance. (Domain expertise used to tune the prover.)

10. First-order logic + machine learning
There’s a growing interest in improving ATPs by adding machine learning to
them.
I’ve been interested in this for some time:
• The E theorem prover is a state-of-the-art prover for predicate logic.
• One way it tries to address the problem is to use features of the theorem of
interest to choose from 82 heuristics for choosing what to resolve.
• Can we improve on this?
We found that, yes, we could. (Bridge, Holden and Paulson, Journal of Automated
Reasoning, 2014).
Specifically, applying support vector machines and Gaussian process classifiers we
achieved comparable performace with only 5 heuristics.
Take-home message: machine learning can provide useful guidance for automated
theorem provers.

11. This proof is quite big…
The classification theorem for finite simple groups
Every finite, simple group is isomorphic to one of:
• a cyclic group with prime order
• an alternating group with degree ≥ 5
• a group of Lie type
• one of 26 sporadic groups.

12. This proof is quite big…
The proof of the theorem was announced in the early 1980s. BUT it was, er, quite
big:
• Estimated at > 10, 000 pages long.
• After announcement, it turned out there was a gap related to quasithin groups…
• … which took another 7 years and 1221 pages.
• There is an effort at present to shorten the proof .
• As of 2019 this is 8 volumes, with another 5 likely to be needed.
QUESTION: Is this proof correct?
(The mathematicians don’t seem too worried, but I’m not so sure…)

13. Proof checking: formalising mathematical results
Systems such as Isabelle provide a means of constructing formally verified proofs:
https://www.cl.cam.ac.uk/research/hvg/Isabelle/
Such systems are often presented as proof assistants.
There is a body of formalized mathematics: the Archive of Formal Proofs (AFP):
https://www.isa-afp.org/
This provides us with another target for machine learning:
Lemma recommendation

14. Machine learning for selecting premises
Current work on lemma recommendation (Słowik, Mangla, Jamnik, Holden, Paul-
son, AAAI 2019).
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x
-2
-1
0
1
2
3
4
5
f(x)
Squared exponential prior, l = 1/2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x
-3
-2
-1
0
1
2
3
4
5
f(x)
γ-exponential prior, l = γ = 1/2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x
-2
-1
0
1
2
3
4
5
6f(x) Rational quadratic prior, l = a = 3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x
-2
-1
0
1
2
3
4
5
f(x)
Neural network prior, σ1 = 15, σ2 = 10
• We build on the existing Sumo Inference Engine (SInE).
• We use Bayesian optimization to tune SInE.
• The AFP is a natural source of training data.

15. Mathematical language is tricky
Question: why don’t more mathematicians use proof assistants?
Answer: mathematicians don’t necessarily want to learn a new formal language.
CHALLENGE
Is there a way of writing mathematics that looks natural while being formally
verifiable?
It turns out that this is REALLY HARD!
See for example The Language of Mathematics: A Linguistic and Philosophical
Investigation, by Mohan Ganesalingam, Springer, 2013.