This document provides an introduction to natural language processing (NLP). It discusses how IBM Watson works by extracting relevant information from questions using named entity recognition, relation extraction, and information retrieval to find answers. Next, it describes how text generation works using Markov chains, hidden Markov models, conditional random fields, recurrent neural networks, and long short-term memory networks to predict the next character or word. Finally, it notes some limitations of current AI and provides extra resources for further reading.

1. Text Analysis and Knowledge Engineering Lab
Faculty of Electrical Engineering and Computing
University of Zagreb
takelab.fer.hr

2. Introduction to NLP

3. Introduction to NLP

4. Introduction to NLP

5. Introduction to NLP

6. 
In 2011. IBM Watson defeated two best US
players in Jeopardy

8. Shakespeare-like text
PANDARUS:
Alas, I think he shall be come approached and the day
When little srain would be attain'd into being never fed,
And who is but a chain and subjects of his death,
I should not sleep.
Second Senator:
They are away this miseries, produced upon my soul,
Breaking and strongly should be buried, when I perish
The earth and thoughts of many states.
DUKE VINCENTIO:
Well, your wit is in the care of side and that.
Second Lord:
They would be ruled after this chamber, and
my fair nues begun out of the fact, to be conveyed,
Whose noble souls I'll have the heart of the wars.
Clown:
Come, sir, I will make did behold your worship.
VIOLA:
I'll drink it.

9. Shakespeare-like text

v*AI*lentine (vailentine.com)

10. Predicting the next character

1913. A. A. Markov
analyzed
Eugene Onegin
by Pushkin

Markov chains

11. How does it work?

Machine learning based

Given datapoints X, predict label y

12. How does it work?

13. How does it work?

Requires having previously labelled data
(algorithms need to be told 16 + 3 = 19)

How does predicting labels of X relate to
making up new Shakespeare?

Framing the problem differently for text
generation

X = characters written so far

y = character to be written

14. Predicting the next character

Generating Shakespeare-like text

Same idea as Markov, but try to predict the
actual character

15. Generating names

Example
− input: character
− output: character

Compare input to output
− if different then adjust
parameters

16. Predicting the next word

17. Predicting the next word

Hidden Markov Model (HMM)

Conditional Random Fields (CRF)

Recurrent Neural Networks (RNN)

Long Term Short Memory Networks (LSTM)

18. How does IBM Watson work?

Q: This is the capital of Croatia

Information extraction
− Extract relevant information from question

Named Entity Recognition: Croatia

Relation extraction: A capitalOf B

Information retrieval
− Fetch me all the documents with information on
(capital AND Croatia)
− Find matching relation 'A isCapital B' in retrieved
document with B = Croatia

19. How does IBM Watson work?

IBM Watson considered outdated in
comparison to Neural Networks (7 years ago)

Lot of similar systems today work without
explicit Named Entity Recognition, Relation
Extraction
− Input: Question
− Output: Answer

20. That's all great, but...

No real understanding of multiple domains

No common sense reasoning

Very little understanding of how Neural
Networks work
− https://www.quantamagazine.org/new-theory-cracks-open-the-black-box-of-deep-learning-20170921/

Requires a lot of data usually
− depends on task

Turing award winner Judea Pearl says modern
AI is all just “curve fitting”
− https://www.quantamagazine.org/to-build-truly-intelligent-machines-teach-them-cause-and-effect-20180515/?mc_cid=26a384ff44&

21. AI fails

22. AI fails (2)

23. AI fails (3)

24. Extra resources

History of Markov chains
− https://www.americanscientist.org/article/first-links-in-the

Text generation
− http://karpathy.github.io/2015/05/21/rnn-effectiveness/

IBM Watson
− https://www.youtube.com/watch?v=DywO4zksfXw

Generating names in PyTorch
− https://pytorch.org/tutorials/intermediate/char_rnn_gene

25. Hiring!
PhD: http://takelab.fer.hr/dont-miss-out/
Internship: http://takelab.fer.hr/interns-wanted-juneseptember-2018/

26. Text Analysis and Knowledge Engineering Lab
Faculty of Electrical Engineering and Computing
University of Zagreb
takelab.fer.hr