Paper Presentation: A Pendulum Swung Too Far

581 views
334 views

Published on

A paper presentation made by me for the paper 'A Pendulum Swung Too Far' by Kenneth Church at IIT Bombay as a part of preparation for the MTech Seminar.

Get the paper on which this presentation is based here: http://languagelog.ldc.upenn.edu/myl/ldc/swung-too-far.pdf

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
581
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
10
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Paper Presentation: A Pendulum Swung Too Far

  1. 1. Paper Presentation A Pendulum Swung Too Far (2011) by Kenneth Church Sagar Ahire [133050073]
  2. 2. Roadmap ● Introduction ● History of NLP ● Objections to Empiricism ○ Chomsky ○ Minsky ○ Pierce ● Reasons for the Problem and Solutions
  3. 3. Roadmap: We Are Here ● Introduction ● History of NLP ● Objections to Empiricism ○ Chomsky ○ Minsky ○ Pierce ● Reasons for the Problem and Solutions
  4. 4. Introduction ● The paper deals with the oscillation between the predominance of theory-driven approaches vs data-driven approaches in the history of NLP and its reasons. ● Specifically, it predicts a surge in rationalism in the 2010s and explains why and how researchers need to be prepared for it.
  5. 5. Rationalism vs. Empiricism Rationalism Empiricism 1. Emphasizes on theory 2. Assumes an “innate language faculty” 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky 1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig
  6. 6. Rationalism vs. Empiricism Rationalism Empiricism 1. Emphasizes on theory 2. Assumes an “innate language faculty” 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky 1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig
  7. 7. Rationalism vs. Empiricism Rationalism Empiricism 1. Emphasizes on theory 2. Assumes an “innate language faculty” 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky 1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig
  8. 8. Rationalism vs. Empiricism Rationalism Empiricism 1. Emphasizes on theory 2. Assumes an “innate language faculty” 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky 1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig
  9. 9. Rationalism vs. Empiricism Rationalism Empiricism 1. Emphasizes on theory 2. Assumes an “innate language faculty” 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky 1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig
  10. 10. Roadmap: We Are Here ● Introduction ● History of NLP ● Objections to Empiricism ○ Chomsky ○ Minsky ○ Pierce ● Reasons for the Problem and Solutions
  11. 11. History of NLP
  12. 12. 1950s: Empiricism ● Empiricism dominated across several fields ● Words were classified on the basis of their co-occurrence with other words (“You shall know a word by the company it keeps” Firth, 1957)
  13. 13. 1970s: Rationalism ● Several authors such as Chomsky, Minsky, etc criticized the Empirical approach ● Failure of the Empirical approach led to funding cutbacks (“winters”) ○ 1966: Machine Translation Failure ○ 1970: The abandonment of connectionism ○ 1971-75: Speech Recognition Failure
  14. 14. 1990s: Empiricism ● Large amounts of data became available ● Several specialized problems could be solved by statistical frameworks, without concentration on the general problems
  15. 15. 2010s: Rationalism? ● Most of the low-hanging fruit has been picked up ● But the original criticisms of the empirical approach are still as valid
  16. 16. Roadmap: We Are Here ● Introduction ● History of NLP ● Objections to Empiricism ○ Chomsky ○ Minsky ○ Pierce ● Reasons for the Problem and Solutions
  17. 17. Objections to Empiricism ● Several common empirical frameworks were opposed by rationalists in the 70s, including: ○ ○ ○ ○ Linear Separators (Machine Learning) Vector Space Model (Information Retrieval) n-grams (Language Modeling) HMMs (Speech Recognition) ● Many of these are mere approximations of complex phenomena
  18. 18. Chomsky’s Objections ● n-gram Language Modeling ● Finite State Methods
  19. 19. Chomsky’s Objections: n-gram Language Modeling ● Chomsky showed that n-grams cannot learn long-distance dependencies (dependencies spanning more than n words) ● For practical purposes ‘n’ needs to be a small value (3 or 5) ● However, such small values fail to capture several interesting facts
  20. 20. Chomsky’s Objections: Finite State Methods ● Examples of Finite State Methods include ○ Hidden Markov Models (HMMs) ○ Conditional Random Fields (CRFs) ● Finite State Methods can capture dependencies beyond n words ● However, they may require infinite memory to process certain sentences
  21. 21. Chomsky’s Objections: Center Embedded Grammars ● A center embedded grammar is of the form: ○ A -> x A y ● Chomsky proved that a center embedded grammar will require infinite memory and thus cannot be handled by finite state methods ● Center embedding is common in English, for example: ○ A man that a woman that a child that a bird that I heard saw knows loves
  22. 22. Minsky’s Objection ● Linear Separators
  23. 23. Minsky’s Objections: Perceptrons ● Minsky showed that perceptrons (and linear separators in general) cannot learn functions that are not linearly separable such as XOR.
  24. 24. Minsky’s Objections: Perceptrons ● This has implications for several tasks including: ○ ○ ○ ○ Word Sense Disambiguation Information Retrieval Author Identification Sentiment Analysis ● For instance, this is the reason why sentiment analysis ignores loaded terms
  25. 25. Minsky’s Objections: Sentiment Analysis ● Loaded terms can be either positive or negative depending on whom it is addressed to. This is an XOR dependency: Loaded Term Addressed to us Sentiment Positive Y Positive Positive N Negative Negative Y Negative Negative N Positive
  26. 26. Pierce’s Objections ● Evaluation by Demos ● Pattern Matching
  27. 27. Pierce’s Objections: Evaluation by Demos ● According to Pierce, evaluation of projects should be based on scientific principles rather than laboratory demos. ● Projects give good results in laboratory conditions, but have much higher error rates in real-world conditions.
  28. 28. Pierce’s Objections: Pattern Matching ● Pierce stated that pattern matching is “artful deception”, i.e. it is based on heuristics rather than scientific theory. ● Examples: ○ The ELIZA effect ○ The Chinese Room argument
  29. 29. Pierce’s Objections: Pattern Matching ● While pattern matching produces better results in the short term, it does so only by ignoring real scientific questions. ● While ambitious approaches may require time to deliver, they are backed by hard science.
  30. 30. Roadmap: We Are Here ● Introduction ● History of NLP ● Objections to Empiricism ○ Chomsky ○ Minsky ○ Pierce ● Reasons for the Problem and Solutions
  31. 31. Reason for the Oscillations: Gaps in Teaching ● The “losing” side of the debate (currently Rationalism) is never mentioned in textbooks/courses ● Leads to “reinventing the wheel” by each generation of NLP researchers
  32. 32. Reason for the Oscillations: Gaps in Teaching ● Currently most courses concentrate on Statistical methods, ignoring linguistic and scientific questions ● This prepares students only for “low-hanging fruit” but not the real scientific questions
  33. 33. Solution ● Introduce the following in NLP courses: ○ ○ ○ ○ ○ ○ Syntax Morphology Phonology Phonetics Historical Linguistics Language Universals ● Create parallels between computational linguistics and formal linguistics
  34. 34. Solution ● Teach both sides of the rationalism vs. empiricism debate ● Educate students about the challenges ahead of the “low-hanging fruit”
  35. 35. Major References ● A Pendulum Swung Too Far by Kenneth Church, 2001
  36. 36. Other References ● ● ● ● Papers In Linguistics 1934-1951 by JR Firth, 1957 Syntactic Structures by Noam Chomsky, 1957 Whither Speech Recognition by John Pierce, 1969 ELIZA - A Computer Program for the Study of Natural Language Communication between Man and Machine by Joseph Weizenbaum, 1966 ● Minds, Brains and Programs by John Searle, 1980

×