...

1. Guided Transfer Learning
Danko Nikolić
evocenta, Robots Go Mental

2. The problem
The great AI solutions today are gigantic.
There is a huge demand for effective algorithms: Lean, fast
learning.
We conquer AI through:
- increasing model sizes,
- applying massive data,
- using excessive compute.

3. Strong Machine Learning
Weak ML Strong ML
Data Large amounts Small amounts
Parameters Ever more Get away with small
amounts
Compute Super computers Edge devices

4. Strong
Machine
Learning
Smart learner
Weak
Machine
Learning
Dumb learner

5. Transfer learning
starts from a point
nearby a minimum. It
reaches it quickly.
Standard deep learning begins training from a distant
point – but uses a large dataset to overcome bumps and
to reach high performance i.e., a deep error minimum.
Classical transfer learning

7. Transfer learning: Pre-learning weights
Much better: Smart learning algorithms
The network
Learning mechanisms

8. Guided Transfer Learning

9. Learning wisdom:
GTL is a domain expert and
knows in which direction to go
(inductive bias).
GTL guides transfer learning to
overcome a bump and reach a
deeper minimum – a better
solution.
The ordinary transfer learning does
not necessarily find the best solution.
What does GTL bring?

11. 𝑤1,1 ⋯ 𝑤1,𝑛
⋮ ⋱ ⋮
𝑤𝑚,1 ⋯ 𝑤𝑚,𝑛 𝑡+1
=
𝑤1,1 ⋯ 𝑤1,𝑛
⋮ ⋱ ⋮
𝑤𝑚,1 ⋯ 𝑤𝑚,𝑛 𝑡
+
∆𝑤1,1 ⋯ ∆𝑤1,𝑛
⋮ ⋱ ⋮
∆𝑤𝑚,1 ⋯ ∆𝑤𝑚,𝑛
𝑤1,1 ⋯ 𝑤1,𝑛
⋮ ⋱ ⋮
𝑤𝑚,1 ⋯ 𝑤𝑚,𝑛 𝑡+1
=
𝑤1,1 ⋯ 𝑤1,𝑛
⋮ ⋱ ⋮
𝑤𝑚,1 ⋯ 𝑤𝑚,𝑛 𝑡
+
∆𝑤1,1 ⋯ ∆𝑤1,𝑛
⋮ ⋱ ⋮
∆𝑤𝑚,1 ⋯ ∆𝑤𝑚,𝑛
*
𝑔1,1 ⋯ 𝑔1,𝑛
⋮ ⋱ ⋮
𝑔𝑚,1 ⋯ 𝑔𝑚,𝑛
𝑔 𝜖{0 . . 1}
where
1
2
m
1
2
n
𝑤1,1
𝑤𝑚,𝑛

14. GTL creates inductive biases.
Your standard multi-layer
deep learning neural
network
GTL lays a “expertise
blanked” that imposes
inductive biases over the
network
(Inductive bias ≡ prior ≡ assumption)

15. Step one
Becoming an expert for a domain.

16. G = (M - min(M)) / (max(M) - min(M)),
mW = 1/n Σ (wB – wn).
2

22. …
100 ms 100 ms
A more effective
model
Model adjusts to
new visual
conditions
GTL-assisted

23. …
100 ms 100 ms

24. Weak ML has reached its limits.
Strong ML is the future.
We want to be at the forefront of AI.

25. Thank you

26. resource TL GTL Combination
Data Full data set Light data set Full pre-train data
set
Compute Full effort pre-
training
Light effort 20% increase in
compute
Parameters No additional
parameters
2x parameters 2x parameters