Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Developing

Reading Machines
Sebastian Riedel

UCL Machine Reading 

Bloomsbury AI
Machine Reading
@riedelcastro
@uclmr
UCL Machine Reading
Research Team at UCL Computer Science
Teaching Machines how to read!
2
Guillaume
[Citations]
3
Overview
What is Machine Reading?
How is it done?
In the past: knowledge and rules
Today: a lot of data and generic Deep L...
5
“Which proteins interact with wild-type TES2?”
Machine Reading for Science …
6
“Which proteins interact with wild-type TES2?”
NF-kB is a protein
complex that controls
th...
Machine Reading for Science …
7
“Which proteins interact with wild-type TES2?”
Reading Machine
“TRADD, …”
NF-kB is a prote...
For Journalists …
8
“Who is in the upper echelon of the Iranian
Military?”
Reading Machine
“Ataollah Salehi, … “
TEHRAN — ...
For Everyone
9
“What pension scheme should I use?”
Reading Machine
“A Self-Invested Personal Pension (SIPP)”
It makes sens...
10
Developing Reading Machines

from the 1960s to now
The Reading Machine User
11
NF-kB is a protein
complex that controls
the transcription of
DNA. ... It plays a key
role in ...
And the Developer
12
“Which proteins interact with wild-type TES2?”
“TRADD, …”
Reading Machine
“I want to build
the best
r...
And the Developer
13
Reading Machine
“I want to build
the best
reading machine!”
And the Developer
14
Reading Machine
1960
1990
“I have a lot of knowledge about language!”
If Protein A is the syntactic
s...
And the Developer
Reading Machine
1990
“I have a lot of knowledge about language!”
2014
15
“I want to build the best readi...
And the Developer
Reading Machine
2014
16
“I have some knowledge about language”
“I also have some data though!”
“I have n...
17
But usually we don’t have lots of data
Reading Machine
2014
18
“I have some knowledge about language”
“I also have some data though!”
“I have no knowledge about ...
Reading Machine
19
“I have no knowledge about language*”
“But I have a high capacity learner…”
Encoder
Decoder
“and lots o...
Math Word Problems
20
“I have very little data!”
A has X items, (gives away | gets) Y
items. How many does (he|she) have
n...
Learning to Generate Training Data
21
Reading
Machine
generates data for
to do well on
A has X items, (gives away | gets) ...
How to Generate What?
22
A has X items, (gives away | gets) Y
items. How many does (he|she) have
now? X (+|-) Y
67
69.75
7...
Information Extraction
23
Sebastian is a Reader at UCL
Sebastian lives in London
Who does Sebastian work for?
UCL
“Readers...
Regularisation
24
“Readers are University employees”
“But I know something!”
Knowledge Loss
optimise on
optimise on
conver...
Regularisation
25
“Readers are University employees”
“But I know something!”
0
15
30
45
60
Data Knowledge K+D Ours
Sebasti...
Learning to Program
26
Sort animal, pug, dog, mammal
animal, mammal, dog, pub
The program is recursive, and uses
an (unkno...
Learning to Program
27
Sort animal, pug, dog, mammal
animal, mammal, dog, pub
def sort(input):
???
compare(???, ???)
???
s...
Knowledge Compilation
28
Sort animal, pug, dog, mammal
animal, mammal, dog, pub
def sort(input):
???
compare(???, ???)
???...
Knowledge Compilation
29
Sort animal, pug, dog, mammal
animal, mammal, dog, pub
“And I have very little data!”
Machine
opt...
Summary
Machine Reading today:
high capacity deep learner & a lot of data
But Machine Reading reality: Not much data
How t...
Upcoming SlideShare
Loading in …5
×

Developing Reading Machines

324 views

Published on

Slides presented by Sebastian Riedel of UCL Machine Reading at an Iris. AI's meetup.
Slides cover the evolution of approaches to machine reading and describe a way to address the problem of working with very limited data.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Developing Reading Machines

  1. 1. Developing
 Reading Machines Sebastian Riedel
 UCL Machine Reading 
 Bloomsbury AI Machine Reading @riedelcastro @uclmr
  2. 2. UCL Machine Reading Research Team at UCL Computer Science Teaching Machines how to read! 2 Guillaume
  3. 3. [Citations] 3
  4. 4. Overview What is Machine Reading? How is it done? In the past: knowledge and rules Today: a lot of data and generic Deep Learning But data is scarce How do we get knowledge back in? 4
  5. 5. 5 “Which proteins interact with wild-type TES2?”
  6. 6. Machine Reading for Science … 6 “Which proteins interact with wild-type TES2?” NF-kB is a protein complex that controls the transcription of DNA. ... It plays a key role in regulating the immune response to infection. ... Incorrect regulation of NF-kB has been linked to many diseases and disorders. read
  7. 7. Machine Reading for Science … 7 “Which proteins interact with wild-type TES2?” Reading Machine “TRADD, …” NF-kB is a protein complex that controls the transcription of DNA. ... It plays a key role in regulating the immune response to infection. ... Incorrect regulation of NF-kB has been linked to many diseases and disorders. read
  8. 8. For Journalists … 8 “Who is in the upper echelon of the Iranian Military?” Reading Machine “Ataollah Salehi, … “ TEHRAN — Iran’s supreme leader, in an unexpected move, replaced the general in charge of the Iranian armed forces on Tuesday with the general’s deputy, a member of the Islamic Revolutionary Guards Corps… read
  9. 9. For Everyone 9 “What pension scheme should I use?” Reading Machine “A Self-Invested Personal Pension (SIPP)” It makes sense to put some money away for when you’re older and that’s what pension schemes help you do. You save a little of your income regularly during your working life so you can have an income in later life, when you want to work less or retire. read
  10. 10. 10 Developing Reading Machines
 from the 1960s to now
  11. 11. The Reading Machine User 11 NF-kB is a protein complex that controls the transcription of DNA. ... It plays a key role in regulating the immune response to infection. ... Incorrect regulation of NF-kB has been linked to many diseases and disorders. Reading Machine “Which proteins interact with wild-type TES2?” “TRADD, …”
  12. 12. And the Developer 12 “Which proteins interact with wild-type TES2?” “TRADD, …” Reading Machine “I want to build the best reading machine!”
  13. 13. And the Developer 13 Reading Machine “I want to build the best reading machine!”
  14. 14. And the Developer 14 Reading Machine 1960 1990 “I have a lot of knowledge about language!” If Protein A is the syntactic subject of an “activates” verb and Protein B is the object, and … then A interacts with B … “I want to build the best reading machine!”
  15. 15. And the Developer Reading Machine 1990 “I have a lot of knowledge about language!” 2014 15 “I want to build the best reading machine!” “I have some knowledge about language” “I also have some data though!” “I want to build the best reading machine!” distance < 10 “activates” in between
  16. 16. And the Developer Reading Machine 2014 16 “I have some knowledge about language” “I also have some data though!” “I have no knowledge about language*” “But I have a high capacity learner…” *“rather: I don’t want to bother” “I want to build the best reading machine!” Encoder Decoder “and lots of data!” Generic Recurrent Neural Network
  17. 17. 17 But usually we don’t have lots of data
  18. 18. Reading Machine 2014 18 “I have some knowledge about language” “I also have some data though!” “I have no knowledge about language*” “But I have a high capacity learner…” “I want to build the best reading machine!” Encoder Decoder “and lots of data!” 2015
  19. 19. Reading Machine 19 “I have no knowledge about language*” “But I have a high capacity learner…” Encoder Decoder “and lots of data!” 2015 “I want to build the best reading machine!” "Cody ran to Melanie ' s farm. The distance is 18 yards from Cody ' s farm to Melanie ' s farm. It took Cody 2 hours to get there. How fast did Cody go?” 18/2 ~ 200 Training instances Math Word Problems
  20. 20. Math Word Problems 20 “I have very little data!” A has X items, (gives away | gets) Y items. How many does (he|she) have now? X (+|-) Y “But I know something!” "Cody ran to Melanie ' s farm. The distance is 18 yards from Cody ' s farm to Melanie ' s farm. It took Cody 2 hours to get there. How fast did Cody go?” 18/2 Reading Machine Encoder Decoder
  21. 21. Learning to Generate Training Data 21 Reading Machine generates data for to do well on A has X items, (gives away | gets) Y items. How many does (he|she) have now? X (+|-) Y Data Generation Model “But I know something!” "Cody ran to Melanie ' s farm. The distance is 18 yards from Cody ' s farm to Melanie ' s farm. It took Cody 2 hours to get there. How fast did Cody go?” 18/2 “I have very little data!” defines Bouchard & Stenetorp, EMNLP 2016
  22. 22. How to Generate What? 22 A has X items, (gives away | gets) Y items. How many does (he|she) have now? X (+|-) Y 67 69.75 72.5 75.25 78 Prev. Work Data Knowledge K+D “I have very little data!” "Cody ran to Melanie ' s farm. The distance is 18 yards from Cody ' s farm to Melanie ' s farm. It took Cody 2 hours to get there. How fast did Cody go?” 18/2 “But I know something!”
  23. 23. Information Extraction 23 Sebastian is a Reader at UCL Sebastian lives in London Who does Sebastian work for? UCL “Readers are University employees” “But I know something!” “I have very little data!” Reading Machine
  24. 24. Regularisation 24 “Readers are University employees” “But I know something!” Knowledge Loss optimise on optimise on convert Sebastian is a Reader at UCL Sebastian lives in London Who does Sebastian work for? UCL “I have very little data!” Reading Machine Rocktäschel, Demeester, Singh, NAACL 2015, EMNLP 2016
  25. 25. Regularisation 25 “Readers are University employees” “But I know something!” 0 15 30 45 60 Data Knowledge K+D Ours Sebastian is a Reader at UCL Sebastian lives in London Who does Sebastian work for? UCL “I have very little data!”
  26. 26. Learning to Program 26 Sort animal, pug, dog, mammal animal, mammal, dog, pub The program is recursive, and uses an (unknown) comparison function “But I know something!” “I have very little data!” Machine
  27. 27. Learning to Program 27 Sort animal, pug, dog, mammal animal, mammal, dog, pub def sort(input): ??? compare(???, ???) ??? sort(???(input)) “But I know something!” “I have very little data!” Machine
  28. 28. Knowledge Compilation 28 Sort animal, pug, dog, mammal animal, mammal, dog, pub def sort(input): ??? compare(???, ???) ??? sort(???(input)) “But I know something!” Machine compile to optimise on “I have very little data!” Bosnjak, Rocktäschel, 2016, arxiv
  29. 29. Knowledge Compilation 29 Sort animal, pug, dog, mammal animal, mammal, dog, pub “And I have very little data!” Machine optimise on A Neural Program Trace:
  30. 30. Summary Machine Reading today: high capacity deep learner & a lot of data But Machine Reading reality: Not much data How to inject knowledge into deep learning? By (generating) data By regularising model By compiling knowledge into model structure 30

×