The document discusses the "black box problem" in artificial intelligence and neural networks. Specifically, it notes that while these systems can perform complex tasks, the inner workings and decision-making processes are not fully understood. It argues that developing theoretical frameworks grounded in other domains, like physics, could help increase transparency and interpretability of these technologies. More work is needed to better understand and explain how artificial intelligence systems learn and operate.

1. Dr. Marcel Blattner - The Black Box Problem
The Black Box Problem 1
Dr. Marcel Blattner
Chief Data Scientist TX Group
marcel.blattner@tx.group
@blattnerma

2. Dr. Marcel Blattner - The Black Box Problem
The Black Box Problem 2
5 Ways Artificial Intelligence Will Change the World by 2050
https://news.usc.edu/trojan-family/five-ways-ai-will-change-the-world-by-2050/
How artificial intelligence will change every aspect of our lives
https://nypost.com/2018/09/07/how-artificial-intelligence-will-change-every-aspect-of-our-lives/
How Artificial Intelligence Is Totally Changing Everything
https://science.howstuffworks.com/artificial-intelligence.htm
AI will write a best-seller by 2049, experts predict
https://www.weforum.org/agenda/2018/03/timeline-of-creative-ai/
- What have opinion leaders to offer?
X → YWe can summarise these statements: Such statements are
useful only if X is
understood! Sadly,
X is not enough
understood in most
cases
Hypeitup!OMG!!

3. Dr. Marcel Blattner - The Black Box Problem
The Black Box Problem 3
- Who are the opinion leaders?
• Nearly 60 percent of news articles across industries are
indexed to industry products, initiatives, or announcements.
33 percent of unique sources across all articles are affiliated
with industry, almost twice as many as those from
academia, and six times as many as those from
government. Nearly 12 percent of all articles reference Elon
Musk.
You can safely count me in here!
https://reutersinstitute.politics.ox.ac.uk/our-research/industry-led-debate-how-uk-media-cover-artificial-intelligence
That’s bad news. Opinion
in public is driven
mainly by completely
biased
industry players

4. Dr. Marcel Blattner - The Black Box Problem
The Black Box Problem 4
“…..the dynamics of learning observed in deep neural networks
remain much of a mystery to this day.”
- What does a giant (Yoshua Bengio) say about AI and ANN?
On the Learning Dynamics of Deep Neural Networks (2019)
I have some questions!

5. Dr. Marcel Blattner - The Black Box Problem
5
Input Output
You don’t know the inner working.
Usually we do not care about using black boxes!
Why?
The Black Box Problem
- What we mean by Black Box

6. Dr. Marcel Blattner - The Black Box Problem
What is a black box? 6
- How to make a computer smart?
Data New
Data
Deductive
Algorithms
Model
Abstraction
Inductive
Training Application
Connectionists
• Rules are based on abstractions created by machines
• Rules are mostly not interpretable by humans
• Expressive advantages
• System stability and general behaviour is not
fully predictable
Algorithms
Experten-
Knowledge
Application
Rules
New
Data
Deductive
if/then
else
Symbolic
• Rules are human knowledge based
• Rules are transparent
• Expressive disadvantage

7. Dr. Marcel Blattner - The Black Box Problem
7
The Black Box Problem
- Object-Detector (CNN)
- Trained on 1.8 Mio. images
- Ported to iOS for real time
object recognition
- How to confuse a Artificial Neural Network

8. Dr. Marcel Blattner - The Black Box Problem
8
Source: Adversarial Patch: https://arxiv.org/pdf/1712.09665.pdf
Adversarial Attack - example
The Black Box Problem
- How to confuse a Artificial Neural Network

9. Dr. Marcel Blattner - The Black Box Problem
9
The Black Box Problem
Making a Tesla 50mph faster than allowed
https://electrek.co/2020/02/19/tesla-autopilot-tricked-accelerate-speed-limit-sign/
- How to confuse a Artificial Neural Network

10. Dr. Marcel Blattner - The Black Box Problem
10
The Black Box Problem
- What is needed to make a black box interpretable?
InterpretabilityTransparency
Based on
meta theories, e.g.
real understanding of the learning
dynamics
Explainability
Based on post-hoc
experiments, e.g.
increase experience of
input-output relation
Operator Executor
Decision
subject
Creator
According Tomsett et al. (2018)

11. Dr. Marcel Blattner - The Black Box Problem
11
The Black Box Problem
- Explainability
Explainability
Based on post-hoc
experiments, e.g.
increase experience of
input-output relation
Layer-Wise Relevance
Propagation
pug, pug-dog (254): 18.939
French bulldog (245): 11.256
bull mastiff (243): 10.136
doormat, welc ome mat (539): 9.809
Norwegian elk hound, (174): 9.591

12. Dr. Marcel Blattner - The Black Box Problem
12
Theoretical framework
ANN
(specific architecture)
Mapping to a framework
(possibly outside CS)
Predictions /
new insights
Guided experiments
Inverse mapping
Hypothesis
Proposal
Testing by experiments
with ANN
The Black Box Problem
- Transparency

13. Dr. Marcel Blattner - The Black Box Problem
Theoretical framework
ANN
(specific architecture)
Mapping to a framework
(possibly outside CS)
Metastable
configurations
(mini/max Landscape
Guided experiments
Inverse mapping Hypothesis
Loss Surface
Testing by experiments
with ANN
H = −
∑
i1>⋯>ip=1
Ji1...ip
σi1
⋯σip
Y = q
n0
∑
i=1
γ
∑
j=1
Xi, j Ai, j
K
∏
k=1
w(k)
i, j
13
- What could be a good candidate for a theory?
The Black Box Problem
TheorySimulations

14. Dr. Marcel Blattner - The Black Box Problem
Summary
14
• The Black Box Problem is important because of the
technology adoption in industry, government and private
sectors
• Try harder!! We need better models (theories) to
understand these technologies
• Theoretical physics could make valuable
contributions