For more AI talks, visit: nyai.co These slides are from NYAI #27: Cognitive Architecture & Natural Language Processing w/ Dr. Catherine Havasi, which took place Tues, 12/18/19 at Kirkland & Ellis NYC. [Speaker Bio] Dr. Catherine Havasi is a technology strategist, artificial intelligence researcher, and entrepreneur. In the late 90s, she co-founded the Common Sense Computing Initiative, or ConceptNet, the first crowd-sourced project for artificial intelligence and the largest open knowledge graph for language understanding. ConceptNet has played a role in thousands of AI projects and will be turning 20 next year. She has started several companies commercializing AI research, including Luminoso where she acts as Chief Strategy Officer. She is currently a visiting scientist at the MIT Media Lab where she works on computational creativity and previously directed the Digital Intuition group. [Abstract] People who build everything from entertainment experiences to financial management face a dilemma: how can you scale what you’re building for broader consumption, yet maintain the personalization that makes it special? A fundamental tension exists between building something individualized, and scaling it to consumers such as visitors at a theme park, or gamers exploring the latest Zelda adventure. True disruption happens when we overcome the idea that one must sacrifice personalization to achieve mass production — like it has in advertising, recommendations, and web search. Artificial Intelligence practitioners, especially in natural language understanding, dialogue, and cognitive modeling, face the same issue: how can we personalize our models for all audiences without relying on unscalable efforts such as writing specific rules, building dialogue trees, or designing knowledge graphs? Catherine Havasi believes we can remove this dichotomy and achieve “mass personalization.” In this session we’ll discuss how to understand domain text and build believable digital characters. We’ll talk about how adding a little common sense, cognitive architectures, and planning is making this all possible. nyai.co

1. COGNITIVE ARCHITECTURE &
NATURAL LANGUAGE
UNDERSTANDING
Dr. Catherine Havasi
Director, Common Sense Computing Initiative
Visiting Scientist, MIT Media Lab
Chief Strategy @ Luminoso
Photo by: Allen Watkin CC BY-SA 2.0.

2. It’s long been our dream to talk to a computer.

3. Advances have transformed language understanding.
1960s
Pattern Matching
& Keywords
1990s
Rule Sets &
Ontologies
2000s
Unsupervised
ML (Bayes)
Word Embeddings
(“Deep Learning”)
2010s
Photo by: Wing-Chi Poon CC BY-SA 2.5.

4. ...but it matters where they came from
Vectors are great
• Compare words by what they
mean, not just exact matches
• A convenient form for
machine learning
• Make your deep learning one
step shallower
Fragment of ConceptNet 5.5

5. Distributional semantics
• “You shall know a word by the company it
keeps.” – J. R. Firth
a word by the company it keeps

6. For natural language understanding to be
successful, it must be:
Multilingual &
Global
ExplainableUnbiased &
Ethical
Adaptable Automated

7. The flaw in deep learning is that it requires
more data, time, and compute than is practical
in most circumstances.

8. This data is needed because the
model must learn from scratch
each time it learns anything.
This isn’t how people work.
Photo by: Carl Hoffman

9. “I don’t have to actually experience crashing
my car into a wall a few hundred times before I
slowly start avoiding to do so.”
- Andrej Karpathy, Open AI

10. People learn and adapt quickly and
from few examples. We share a
common understanding.

11. Like us, AI needs to draw on its domain general
knowledge to learn new domains and skills
…without without the need for someone to keep
explaining.
Photo by: Mateus S. Figueiredo CC BY-SA 4.0

12. If machine learning isn’t automated, what’s the point?
We’ve seen enough brittle systems that work for one task
and nothing else.
Photo by: Eistropfen CC BY-SA 4.0

13. How do people do it?

14. ALMOST TWENTY
YEARS YEARS AGO…

15. Photo by: Fir0002 CC BY-SA 3.0.
We generalize from known examples…

16. We build analogies to understand complex concepts…
Photo by: Nicholas A. Tonelli CC BY-SA 2.0.

17. “To make real progress in A.I., we have to overcome the big challenges
in the area of common sense”
- Paul Allen, Microsoft co-founder

18. “You don’t understand anything
until you learn it in more than
one way.”
− Marvin Minsky

19. We have built models of how people think about the world in 73
languages – called ConceptNet.

21. COLLECTING COMMON-SENSE KNOWLEDGE
Open Mind Common Sense,
around 2006

22. Linked data
ConceptNet
OpenCyc
WordNet
UMBEL
YAGO
DBPedia
Wikidata Wiktionary
Lexvo
Wikipedia
UBY

23. LANGUAGES IN CONCEPTNETMultilingual coverage
English
6.5 million edges
French
4.9 million edges
German
1.6M
Italian
1.1M
Spanish
830k
Japanese
740k
Russian
620k
Portuguese
540k
Chinese
500k
Finnish
420k
Dutch
400k
Swedish
300k
bg pl cs sh eo ms sl ar
Total: 24.6 million edges in 70+ languages
. . .
ces
mon Sense
sh, French, German)
d knowledge
h)
se)
e (Chinese)
purpose
l WordNet
ces
Open code a
At http://conceptnet.io,
• Code on GitHub to rep
• A browsable Web inter
• A Linked Data REST AP
All data is available unde
Creative Commons Attrib
ShareAlike 4.0 license.

24. ConceptNet is multilingual glue
How do we get word embeddings that can be
compared across languages?

25. money
bank
location
IT’S POSSIBLE TO KNOW THE SAME FACT IN DIFFERENT
LANGUAGES

26. money
bank
location
KNOWLEDGE BECOMES STRONGER WHEN IT’S
CONNECTED BETWEEN LANGUAGES

27. Retrofitting
• Created by Manaal Faruqui in 2015
• Apply knowledge-based constraints after training distributional
word vectors
• It works better than during training, for some reason

28. RETROFITTING
• Terms that are connected in the knowledge graph should have
vectors that are closer together
• Many extensions now, such as “antonyms should be farther
apart” (Mrkšić et al., 2016)-
oak
tree
furniture

29. RETROFITTING JUST WORKS
• On intrinsic evaluations, the top-performing systems almost
always use retrofitting
– If you see a purely distributional algorithm claim “state of
the art on SimLex”, it may be “state of the art assuming no
knowledge graph”

30. • State-of-the-art word vectors
• Hybrid of ConceptNet and distributional
semantics
• Multilingual by design
• Open source, open data

31. Building ConceptNet Numberbatch
Common
Crawl
Open
Subtitles
ConceptNet
Google
News
word2vec GloVe fastText
Retrofit Retrofit Retrofit
Join
Reduce
dimensionality
De-bias
ConceptNet
Numberbatch
Many data
sources

32. SEMEVAL 2017
• Semantic Evaluation – head-to-head evaluations of
NLP systems
• Systems submit results to be evaluated against
unseen test labels

33. SemEval 2017 (multilingual)

34. Why ConceptNet in particular?
• Represents multiple registers of knowledge
– “fire is oxidation” vs. “fire is hot”
• Common words are represented with lots of edges;
rare words are represented at all
• Avoids wasting feature space on highly specific
trivia

35. Distinguishing attributes using ConceptNet
• A task at SemEval 2018
• We got 74% accuracy (2nd
place) by directly querying
ConceptNet Numberbatch
• Additional features trained on
the data didn’t help on the test
set
• All top systems used
knowledge graphs

36. AI2 & THE MOSAIC PROJECT
• Initial focus on evaluations for common sense – building a common set of
benchmarks to understand higher level reasoning
• Collecting some data, working with DARPA

37. Photo by: David Lapetina CC BY-SA 3.0.
In order to beat a
human player at
chess, Google’s
AlphaZero had to
play 68 million
games against itself.

38. You cannot simulate your call center
calling itself 68 million times.
-SA 4.0.

39. Making text quantifiable

40. For more info: havasi@luminoso.com

41. WHAT IS DOMAIN ADAPTATION?
domain
general
data
domain
specific
data
customer intents,
product names,
industry jargon,
specific issues
common words,
multiple languages,
paraphrases,
general sentiment

42. What is transfer learning?
Photo by: Chris Rodley

43. How do we adapt to a domain?
• You probably don’t need a general NLP system, you
need a specific one
• Starting from scratch isn’t feasible for most
applications
• We can take advantage of general knowledge to
quickly learn specific knowledge

44. Domain-specific learning
ConceptNet
Numberbatch
Text from your
particular domain
Luminoso NLP
pipeline
Merge
Domain-specific
word vectors

45. w w w . l u m i n o s o . c o m
Supervised classification
Remaining
examples are
used as
unlabeled
Luminoso input

46. Supervised classification
With 7 million
docs, VW is
just getting
started
Luminoso
outperforms
others with
1/1000 of the
labeled examples
Remaining
examples are
used as
unlabeled
Luminoso input

47. WHAT IS GROUNDING?
Photo by: Evan-Amos CC BY-SA 3.0.

48. ADDITIONAL TYPES OF COMMON SENSE
• Physical common sense
• Higher order reasoning
• Social common sense
• What about “folk” knowledge?
• Pete Clark’s Aristo project

49. COMPLICATED COMMON
SENSE
Allen Institute AI2
ATOMIC: An Atlas of Machine Commonsense
for If-Then Reasoning
Maarten Sap, Ronan LeBras, Emily
Allaway, Chandra Bhagavatula, Nicholas
Lourie, Hannah Rashkin, Brendan Roof, Noah
A. Smith, Yejin Choi

50. MIT’S NARRATARIUM

51. STORY UNDERSTANDING
USING CONCEPTNET
• SemEval 2018 also included a reading
comprehension task, designed to require
common sense
• The winning system (Yuanfudao
Research) used ConceptNet to find
unstated connections, on top of an
attention model

52. • The Story Cloze Test evaluates common sense
• Five-sentence stories, two possible endings, only one makes
sense
– Previous state of the art (OpenAI Transformer): 86.5%
– Jiaao Chen et al., adding ConceptNet as an input: 87.6%
STORY UNDERSTANDING USING
CONCEPTNET

53. What would we need to do improv with a chatbot? Have a dynamic
conversation with a NPC in a video game?

54. When we have a conversation with an intelligent agent, those
conversations are not natural or creative.

55. What would it mean to have a
conversation with a character?

56. mass production personalization

57. What we need to
do this sort of thing
is real cognitive
modeling…

58. Marvin Minsky envisioned a Society of Mind: many individual processes
and workflows which make an emergent and robust whole.
This is a Cognitive Architecture.

59. WHAT ARE WE MISSING?
• Ability to negotiate conversational goals
• Recover from errors without starting over
• Bond and personalize to users without individual customization
• Adaptability and Scalability

60. WE NEED TO ASK NEW QUESTIONS

61. WHY DO
WE MAKE
SOLO
AGENTS?
Xu, W., Hargood, C., Tang, W. and Charles, F.,
2018. Towards Generating Stylistic Dialogues for
Narratives using Data-Driven Approaches. In:
International Conference for Interactive Digital
Storytelling 5-8 December 2018 Trinity College
Dublin, Ireland.
Photo by: Saad Akhtar CC BY-SA 3.0.

62. FOCUS ON
EXPERIENCE
VERSUS
AGENT
Jason Alonso,Angela Chang, David Robert, Cynthia Breaeal.Toward a dynamic dramaturgy: an art of presentation in interactive storytelling, 2011

63. CONVERSATION IS A NEGOTIATION.
NEGOTIATIONS REQUIRE PLANNING.

64. WE NEED A
FRAMEWORK
THAT KNOWS
HOW TO
GROW UP

65. WE WANT TO BRING ALL OF THIS
TOGETHER
Photo by: Charles Hamm CC BY-SA 3.0.

66. “One can report steady progress,
all the way to the top of the tree.”
Google’s Peter Norvig and UCSF’s Stuart Russell
worried progress in AI can seem like trying to get to the
moon by climbing a tree.
Photo by: Evan-Amos CC BY-SA 3.0.

67. “SOCIETY FOR HIGH
HANGING FRUIT”
No one is motivated to work on very long
term problems that might not be lucrative in
the short term.

68. conceptnet.io – a browsable interface

69. api.conceptnet.io – a Linked Data API

70. A VERY, VERY MERRY THANK YOU
• Robyn Speer, Joanna Lowry-Duda, Robert Beaudoin, and everyone who has
built, contributed to, or used ConceptNet/OMCS.
• Everyone at the MIT Media Lab
• Pedro Colon, Nina Lutz, Pip Mothersill, Emily Salvador and all of my patient
students past and present
• Jeff Foley,Ying Chen,Vivian Shih, and the amazing (and patient) product &
marketing teams @ Luminoso

71. I’M LOOKING FOR COLLABORATORS
• Luminoso is hiring someone who knows Hybris & Solr to be the technical co-
founder of a project to transform ecommerce search
• Luminoso has a machine learning scientist slot open in May and a data
scientist slot open now.
• Topher (digital characters) is looking for collaborators and interested parties
and users
• ConceptNet is looking for users and contributors
• We may be looking for ConceptNet staff in 2019 and Creative Computation
students for 2020 - email me to be kept up to date
havasi@media.mit.edu