The document provides an overview of the Natural Language Toolkit (NLTK), detailing its features, installation guidelines, and basic functionalities in natural language processing. It covers various tasks such as tokenization, part-of-speech tagging, parsing, and chunking, along with code snippets for implementation. Additionally, it includes resources for further learning and contact information for the author.

1. NLTK
Mohammed Shokr
16-Mar-16

2. Natural Language Toolkit (NLTK)
■ A collection of Python programs, modules, data set and tutorial to
support research and development in Natural Language Processing
(NLP)
■ Written by Steven Bird, Edvard Loper and Ewan Klien
■ NLTK is
– Free and Open source
– Easy to use
– Modular
– Well documented
– Simple and extensible

3. Installation of NLTK
■ NLTK requires Python versions 2.7 or 3.2+

4. Python REPL
• Read Eval Print Loop
• REPL: This is a procedure that just
loops, accepts one command at a
time, executing it, and printing the
result.
• GUI OR CLI

5. Installation of NLTK
1. Start a Command Prompt as an Administrator ( Windows User )
1. Click Start.
2. In the Start Search box, type cmd, and then press CTRL+SHIFT+ENTER.
3. If the User Account Control dialog box appears, confirm that the action it
displays is what you want, and then click Continue.
2. changing from user to Superuser ( linux user )
– sudo su

6. Installation of NLTK
Install NLTK: run
pip install nltk

7. Test installation
run
python
then type
import nltk

8. Installing NLTK Data
■ NLTK comes with many corpora, toy grammars, trained models, etc. A complete
list is posted at: http://nltk.org/nltk_data/
■ Run the Python REPL and type the commands:
■ A new window should open, showing the NLTK Downloader. Click on the File menu and select Change Download Directory. For central
installation, set this to C:nltk_data (Windows), /usr/local/share/nltk_data (Mac), or /usr/share/nltk_data (Unix). Next, select the
packages or collections you want to download.
>>> import nltk
>>> nltk.download()

9. NLTK Downloader

10. Getting Started with NLTK

11. NLTK Text

12. Text Pre-processing

13. Sentence splitter
from nltk.tokenize import sent_tokenize
input_sting = ‘Hello Every One. How Are you ? life is not
easy.’
all_sent = sent_tokenize(input_sting)
print (all_sent)

14. Tokenization
sent = "Hi Everyone ! How do you do ?"
# Split() built-in string function
print (sent.split())
# word_tokenize
from nltk.tokenize import word_tokenize
print (word_tokenize(sent))

15. Stemming

16. Lemmatization

17. Morphology

18. Edit-Distance
We can create a very basic spellchecker
by just using a dictionary lookup.

19. Part of Speech Tagging

20. Part of Speech
Tagging

21. Penn Bank Part-of-Speech Tags

22. Part of Speech Tagging
■ Stanford tagger
■ N-gram tagger
■ Regex tagger
■ Brill tagger
■ Machine learning based tagger
■ NER tagger
– Named Entity Recognition (NER)
– NLTK provides the ne_chunk() method

23. NER tagger
 Tokens
 Tagged
 Entities

24. Parsing Structure in Text

25. Shallow VS deep parsing
■ In deep or full parsing, typically, grammar concepts such as CFG, and probabilistic
context-free grammar (PCFG), and a search strategy is used to give a complete
syntactic structure to a sentence.
■ Shallow parsing is the task of parsing a limited part of the syntactic information
from the given text.

26. The two approaches in parsing
The rule-based approach The probabilistic approach
This approach is based on rules/grammar In this approach, you learn rules/grammar
by using probabilistic models
Manual grammatical rules are coded down
in CFG, and so on, in this approach
This uses observed probabilities of linguistic
features
This has a top-down approach This has a bottom-up approach
This approach includes CFG and Regexbased
parser
This approach includes PCFG and the
Stanford parser

27. context-free grammar (CFG)
■ Generating sentences from context-free grammars:

29. Different types of parsers
■ Recursive descent parser
■ Shift-reduce parser
■ Chart parser
■ Regex parser
■ Dependency parsing

30. Different types of parsers
Recursive descent parser One of the most straightforward forms of parsing is recursive
descent parsing. This is a top-down process in which the parser
attempts to verify that the syntax of the input stream is correct,
as it is read from left to right.
Shift-reduce parser The shift-reduce parser is a simple kind of bottom-up parser.
Chart parser We will apply the algorithm design technique of dynamic
programming to the parsing problem.
Regex parser A regex parser uses a regular expression defined in the form of
grammar on top of a POS-tagged string.
Dependency parsing Dependency parsing (DP) is a modern parsing mechanism. The
main concept of DP is that each linguistic unit (words) is
connected with each other by a directed link.

31. Chunking
■ Chunking is shallow parsing where instead of reaching out to the deep structure of
the sentence, we try to club some chunks of the sentences that constitute some
meaning.
■ For example, the sentence "the President speaks about the health care reforms"

32. So, let's write some code snippets to do some basic
chunking:

33. Display a parse tree
# import treebank corpus
from nltk.corpus import treebank
t = treebank.parsed_sents('wsj_0001.mrg')[0]
t.draw()

34. Relation Extraction
Output

35. Resources
■ NLTK 3.0 documentation
– http://www.nltk.org/
■ NLTK Essentials
– https://www.packtpub.com/big-data-and-business-intelligence/nltk-essentials
■ nltk_tutorial_repo [Code]
– https://git.io/vaRIR

36. Thanks.
Questions? Send me an email! (I love talking!)
Mohammed Shokr
mohammedshokr2014@gmail.com
@MShokr1