These slides, based on a presentation at distinguished lecture at IBM Almaden in March, 2017 explore some of the challenges to machine learning and some recent work. It is a newer version of the slides originally presented at IJCAI 2016.

1. Tetherless World Constellation, RPI
The Future of AI: Going Beyond
Deep Learning, Watson, and the
Semantic Web
Jim Hendler
Tetherless World Professor of Computer, Web and Cognitive Sciences
Director, Institute for Data Exploration and Applications
Rensselaer Polytechnic Institute
http://www.cs.rpi.edu/~hendler
@jahendler (twitter)
Major talks at: http://www.slideshare.net/jahendler

2. Tetherless World Constellation, RPI
Knowledge and Learning
Knowledge representation in the age of
Deep Learning, Watson, and the Semantic Web
Jim Hendler
Tetherless World Professor of Computer, Web and Cognitive Sciences
Director, Institute for Data Exploration and Applications
Rensselaer Polytechnic Institute
http://www.cs.rpi.edu/~hendler
Major talks at: http://www.slideshare.net/jahendler

3. Tetherless World Constellation, RPI
Talk derives in part from a recent book

4. Tetherless World Constellation, RPI
New Journal: Data Intelligence
Knowledge graph is one of the
topics we are interested in, please
consider submitting a paper!
(handout in your conference
bag)

5. Tetherless World Constellation, RPI
What has happened?
• Several important AI technologies have
moved through “knees in the curve” bringing
much of the attention to AI again
–Deep Learning (eg AlphaGo, vision processing)
–Associative learning (eg Watson)
–Semantic Web (eg search and schema.org)

6. Tetherless World Constellation, RPI
A) Deep Learning
“phase transition” in capabilities of neural
networks w/machine power

7. Tetherless World Constellation, RPI
Trained on lots of categorized images
Imagenet: Duck Imagenet: Cat

8. Tetherless World Constellation, RPI
Impressive results
Increasingly powerful techniques have yielded
incredible results in the past few years

9. Tetherless World Constellation, RPI
B) Associative knowledge (text mining/QA)
© IBM, used with permission.

10. Tetherless World Constellation, RPI
Impressive Results
Watson showed the power of “associative knowledge”

11. Tetherless World Constellation, RPI
C) Semantic Web

12. Tetherless World Constellation, RPI
“Knowledge graphs” mined from extracted
and learned data

13. Tetherless World Constellation, RPI
Impressive Results
Google finds embedded metadata on >30% of its crawl – Guha, 2015

14. Tetherless World Constellation, RPI
Not just Google…

15. Tetherless World Constellation, RPI
Summary: AI has done some way cool stuff
• Deep Learning: neural learning from data with high
quality,
• Watson: Associative learning from data with high
quality
• Semantic Web/Knowledge Graph: Graph links
formation from extraction, clustering and learning
but, there are still problems…

16. Tetherless World Constellation, RPI
Combining these technologies

17. Tetherless World Constellation, RPI
still a long way to go

18. Tetherless World Constellation, RPI
Many intermediate steps
(P. Norvig, WWW 4/2016, w. permission)

19. Tetherless World Constellation, RPI
Why did knowledge graph need
“”Human Judgments”?
Association ≠ Correctness
P. Mika, 2014 w.permission
Michelangelo
Leonardo
Raphael
Donnatello

20. Tetherless World Constellation, RPI
Knowledge Representation?
• A knowledge representation (KR) is most fundamentally a surrogate, a substitute for the thing
itself, used to enable an entity to determine consequences by thinking rather than acting,
i.e., by reasoning about the world rather than taking action in it.
• It is a set of ontological commitments, i.e., an answer to the question: In what terms should I think
about the world?
• It is a fragmentary theory of intelligent reasoning, expressed in terms of three components: (i) the
representation's fundamental conception of intelligent reasoning; (ii) the set of inferences the
representation sanctions; and (iii) the set of inferences it recommends.
• It is a medium for pragmatically efficient computation, i.e., the computational environment in which
thinking is accomplished. One contribution to this pragmatic efficiency is supplied by the guidance a
representation provides for organizing information so as to facilitate making the recommended
inferences.
• It is a medium of human expression, i.e., a language in which we say things about the
world.
R. Davis, H. Shrobe, P. Szolovits (1993)

21. Tetherless World Constellation, RPI
KR: Human Expression
Cute kid story: first two words

22. Tetherless World Constellation, RPI
Telling cats from ducks doesn’t need KR
!

23. Tetherless World Constellation, RPI
“Saying things about the world” does
"If I was telling it to a
kid, I'd probably say
something like 'the cat
has fur and four legs
and goes meow, the
duck is a bird and it
swims and goes
quack’. "
How would you explain the difference between a
duck and a cat to a child?
Woof

24. Tetherless World Constellation, RPI
Learning semantic inferencing
Bassem Makni, 2018 Phd (now at IBM Yorktown)

25. Tetherless World Constellation, RPI
In noisy data
Bassem Makni, 2018 Phd (now at IBM Yorktown)

26. Tetherless World Constellation, RPI
The challenge of “background knowledge”
What is the relationship
between this man and
this woman?

27. Tetherless World Constellation, RPI
AI systems coming along well…
What is the relationship
between this man and
this woman?
Deep learning produced Scene Graph w/relationships
(Klawonn & Heims, 2018)

28. Tetherless World Constellation, RPI
But the challenges remain
What is the relationship
between this man and
this woman?
Deep learning produced Scene Graph w/relationships
(Klawonn, 2018)
Seeing the bride adds
significant information
that cannot be easily
learned w/o background
knowledge

29. Tetherless World Constellation, RPI
A major problem to deploying AI in key areas

30. Tetherless World Constellation, RPI
Adding knowledge to scene graphs
Matthew Klawonn, PhD, 2019

31. Tetherless World Constellation, RPI
KR: Surrogate knowledge?
Which could you sit in?
What is most likely to bite what?
Which one is most likely to become a computer
scientist someday?
…

32. Tetherless World Constellation, RPI
“Surrogate” knowledge
Which could you sit in?
What is most likely to bite what?
Which one is most likely to become a computer
scientist someday?
How would they go about doing it?

33. Tetherless World Constellation, RPI
KR: Recommended vs. Possible inference
Which one would you save if the house was on
fire?

34. Tetherless World Constellation, RPI
Ethical AI systems need certainty
Which one would you save if the house was on
fire? Would you use a robot baby-sitter
without knowing which of the three
possibilities it would choose?

35. Tetherless World Constellation, RPI
Human-Aware AI
• Context is key
– AI learning systems still perform best in well-defined
contexts (or trained situations, or where their
document set is complete, etc.)
– Humans are good at recognizing context and deciding
when extraneous factors don’t make sense
• Or add extra “inferencing” (the bride example)

36. Tetherless World Constellation, RPI
The challenge
• If we want to implement KR systems on top of
neural and associative learners we have an issue
– The numbers coming out of Deep Learning and
Associative graphs are not probabilities
– They don’t necessarily ground in human-meaningful
symbols
• ”sub-symbolic” learning …
• Association by clustering …
• Errorful extraction …

37. Tetherless World Constellation, RPI
The challenges
• Can we avoid throwing out the reasoning baby
with the grounding bathwater?
– Long-term planning
– Rules that need to be followed
– Human Interaction
• Even if computers don’t need to be symbolic communicators,
WE DO!!!
– Background knowledge (context) is symbolic

38. Tetherless World Constellation, RPI
Human-AI interaction
• Evidence that “centaurs” win
– Human(s) and computer(s) teams currently beat either
at chess (Go centaurs underway)
– Anecdotal evidence that humans w/Watson top choices
outperform Watson or human alone at Jeopardy
– Medical study (diagnostic):
• Doctor w/computer outperformed just doctor, just computer,
two doctors

39. Tetherless World Constellation, RPI
And this matters!
“There was no rule about how long we were allowed to think before we reported a strike …
but we knew that every second of procrastination took away valuable time, that the Soviet
Union’s military and political leadership needed to be informed without delay. All I had to do
was to reach for the phone; to raise the direct line to our top commanders — but I couldn’t
move. I felt like I was sitting on a hot frying pan … when people start a war, they don’t
start it with only five missiles …”
We must all strive to be like Petrov and learn to trust the
combination of AI training and human intuition.
Stanislav Petrov: The man who saved the world