Deep dive into Visual Causal Feature Learning by Krzysztof Chalupka, Pietro Perona, Frederick Eberhardt

1. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Visual Causal Feature Learning
Chalupka et al. 2015
Kojin Oshiba
Department of Computer Science & Department of Statistics
Harvard University
October 12, 2018

2. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Table of Contents
1 Paper Overview
2 Theory of Visual Causal Learning
3 Causal Feature Learning Algorithm
4 Experiment
5 Discussion

3. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Table of Contents
1 Paper Overview
2 Theory of Visual Causal Learning
3 Causal Feature Learning Algorithm
4 Experiment
5 Discussion

4. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Overview
Goal: Understanding the visual cause of humans.
• A framework for causal learning from macro variables (e.g. groups of pixels).
• Observational data + minimal experiment = visual cause
• Applicable to any aggregate of micro variables, e.g. auditory, olfactory data.

5. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Overview
Goal: Understanding the visual cause of humans.
• A framework for causal learning from macro variables (e.g. groups of pixels).
• Observational data + minimal experiment = visual cause
• Applicable to any aggregate of micro variables, e.g. auditory, olfactory data.
Technically,
• Define a macro-variable C, which contains all the causal information available
in an image I ∈ I about a given behavior T.
• I is generated from unobserved discrete variables H.

6. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Table of Contents
1 Paper Overview
2 Theory of Visual Causal Learning
3 Causal Feature Learning Algorithm
4 Experiment
5 Discussion

7. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Observational Partition
Definition: Observational Partition
Πo(T, I) ⊂ I is a partition based on the equivalence relation
i ∼ j ⇔ P(T|I = i) = P(T|I = j) ∀i, j ∈ Πo(T, I)
Knowing the observational partition of an image allows us to predict the value of T.

8. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Partition
Definition: Causal Partition
Πc(T, I) ⊂ I, is a partition based on the equivalence relation
i ∼ j ⇔ P(T|man(I = i)) = P(T|man(I = j)).
Manipulation changes the image, but not T or H.

9. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Partition
Definition: Causal Partition
Πc(T, I) ⊂ I, is a partition based on the equivalence relation
i ∼ j ⇔ P(T|man(I = i)) = P(T|man(I = j)).
Manipulation changes the image, but not T or H.
Definition: Visual Cause
C(i) = P(T|man(I = i)).
Has one-to-one correspondence with Πc(T, I).

10. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Coarsening Theorem
Theorem: Causal Coarsening
Πc is a coarsening of Πo, i.e. causal labels do not change within each
observational class.

11. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Coarsening Theorem
Theorem: Causal Coarsening
Πc is a coarsening of Πo, i.e. causal labels do not change within each
observational class.
The visual causes do not contain all the information in the image that predict T.
∃ information, not itself causal, can be informative about non-visual causes of T.
This is called the spurious correlate.

12. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Coarsening Theorem
Theorem: Causal Coarsening
Πc is a coarsening of Πo, i.e. causal labels do not change within each
observational class.
The visual causes do not contain all the information in the image that predict T.
∃ information, not itself causal, can be informative about non-visual causes of T.
This is called the spurious correlate.
Definition: Spurrious Correlate
S is a discrete random variable whose value differentiates between Πo(T, I)
contained in Πc(T, I).

13. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Coarsening Theorem
Theorem: Causal Coarsening
Πc is a coarsening of Πo, i.e. causal labels do not change within each
observational class.
The visual causes do not contain all the information in the image that predict T.
∃ information, not itself causal, can be informative about non-visual causes of T.
This is called the spurious correlate.
Definition: Spurrious Correlate
S is a discrete random variable whose value differentiates between Πo(T, I)
contained in Πc(T, I).
Theorem: Complete Macro-variable Description
C and S together contain all and only the visual information in I relevant to T,
but only C contains the causal information

14. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Intervention on Macro-variables
Definition: Visual Manipulation man(I = i)
An operation that changes (the pixels of) the image to image i ∈ I, while not
affecting any other variables (such as H or T).

15. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Intervention on Macro-variables
Definition: Visual Manipulation man(I = i)
An operation that changes (the pixels of) the image to image i ∈ I, while not
affecting any other variables (such as H or T).
Definition: Causal Intervention on Macro-variables do(C = c )
man(I = i ) such that C(i ) = c and S(i ) = s.
This is not always possible.

16. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Intervention on Macro-variables
Definition: Visual Manipulation man(I = i)
An operation that changes (the pixels of) the image to image i ∈ I, while not
affecting any other variables (such as H or T).
Definition: Causal Intervention on Macro-variables do(C = c )
man(I = i ) such that C(i ) = c and S(i ) = s.
This is not always possible.
Phew! We have the theory down. Now the question is: how can we learn C?

17. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Table of Contents
1 Paper Overview
2 Theory of Visual Causal Learning
3 Causal Feature Learning Algorithm
4 Experiment
5 Discussion

18. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Effect Prediction
Steps:
0 Observational class for each data is
necessary (do we estimate?).
1 Picks a representative member of each
observational class.
2 Need to design an experiment here.
3 Coarsen observational partitions to
construct causal paritions.
4 Training a neural net on this imputed data
gives the causal effect.
5 NN is our C.

19. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Feature Manipulation
Steps:
0 Oracle A can be obtained from Algo 1.
1 Train manipulation function every iter.
2 Train a "causal neural net".
3 Choose images to be manipulated.
4 Choose target causal partition.
5 For each data,
6 Since C is hardly invertible, this
approximates argminˆi∈C−1(k) d(i, ˆi)
7
8 Augment the causal data every iteration.
9 I think this algo doesn’t really learn MC but
this algo itself is MC ...

20. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Table of Contents
1 Paper Overview
2 Theory of Visual Causal Learning
3 Causal Feature Learning Algorithm
4 Experiment
5 Discussion

21. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Causal Intervention on Macro-variables
• T = 1 if a human answers affirmatively to
the question "does this image contain the
digit ’x’?", where x is the actual digit on
the image.
• Assume for simplicity that
P(T = 1|man(I)) = 0 or
P(T = 1|man(I)) = 1.
• Here, they already know the causal data:
the labels are assigned by default. There
were no experiments training Algo 1, so
unclear if it works in practice.

22. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Table of Contents
1 Paper Overview
2 Theory of Visual Causal Learning
3 Causal Feature Learning Algorithm
4 Experiment
5 Discussion

23. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Other Types of "Causality" on Visual Data
• Understanding the causal structure within an image: Discovering Causal
Signals in Images (Lopez-Paz et al. 2017), CausalGAN (Kocaoglu et al. 2017)

24. Visual Causal
Feature
Learning
Kojin Oshiba
Paper
Overview
Theory of
Visual Causal
Learning
Causal Feature
Learning
Algorithm
Experiment
Discussion
Fin.