XR COLOMBIA & NETHERLANDS 2020

1. The True
Arti
fi
cial Intelligence Revolution
Eric Postma
Cognitive Science & AI, Tilburg University
Data Analytics, Jheronimus Academy of Data Science, ’s-Hertogenbosch
XR COLOMBIA & NETHERLANDS 2020, November 19

3. A brief history of
Arti
fi
cial Intelligence

4. Alan Turing

6. “Biology is irrelevant to AI”
Substrate-independence of computation

7. Symbolic AI (1957-2000)
• Emphasis on formal logic and symbolic representations
• Neuroscience is irrelevant
• Focus on high-level cognition
(reasoning, problem-solving, …)

8. “Statistics is irrelevant to AI”
Deterministic inference rules operating on explicit knowledge representations

9. The “new AI”
The Sojourner Rover (1997)

10. “Sub-symbolic” AI (1988-2016)
• Emphasis on machine learning (neural networks)
• Neuroscience is relevant
• Emphasis on low-level cognition (e.g., perception)

11. How to recognise a dog?

12. Biological Inspiration:
Arti
fi
cial Neural Networks

14. Convolutional Neural Network
Deep Learning

16. INCREASE THE NUMBER OF PARAMETERS!!!
DECREASE THE NUMBER OF PARAMETERS!!!

17. The Lottery Ticket Hypothesis
Frankle & Carbin (2018). The Lottery Ticket Hypothesis: Finding Small, Trainable Neural Networks. arXiv:1803.03635v3 [cs.LG] 20 May.

18. arXiv:1608.08710v3
arXiv:1608.08710v3

19. Alternative approach
1. Train the network (randomly initialise weights: Wr)
2. Remove super
fl
uous structure (i.e., small weights)
3. Fine-tune Retrain the network by initialising with Wr
This does work…..

20. Over-parameterisation
• After pruning, subnetworks can do the job
• These subnetworks are typically between 1%-15% of the original size
• Require considerably less training time
• Perform at or around the same level
This raises the question: Why is over-parameterisation useful?

21. Training speed Validation accuracy
random reinitialisation destroys the advantage

22. Is learning necessary?
Ramanujan et al. (2019). What’s hidden in a randomly weighted neural network? arXiv:1911.13299v2 [cs.CV]

23. Conclusions
• The huge number of connections incorporates the solution
• The main challenge is to
fi
nd it (needle in a haystack)
• In nature, this happens during early development