The document discusses large-scale natural language processing (NLP) using Python's NLTK on Azure, outlining the fundamental processes including language recognition, tokenization, and entity detection. It highlights the use of Azure's cloud services for running Python and NLTK jobs, detailing setup procedures like downloading necessary corpora and configuring web jobs. Additionally, it presents communication methods through queues and debugging strategies in both local and cloud environments.

1. large scale NLP using python's NLTK on Azure

2. I saw Mr. Washington.
This is your saw… I told you!
Is this really a chainsaw?

3. fundamentals of nlp
natural language toolkit (nltk)
running python and nltk on Azure

4. source: http://www.nltk.org/book_1ed/ch01.html
simple pipeline architecture for a spoken dialogue system

5. dialogue with a chatbot

7. identify language
tokenize & tag part of speech (pos)
identify named entities

8. corpora and lexical resources
corpus is a large body of text
lexical resource is a collection words
associated with additional information

9. e.g. brown corpus
first million-word electronic corpus of
english, created in 1961 at brown university

10. segmentation
tokenize
tag part of speech (pos)
identify named entities
source: http://www.nltk.org/book_1ed/ch07.html

11. entity detection using chunking

12. fundamentals of nlp
natural language toolkit (nltk)
running python and nltk on Azure

13. text as a sequence of words and punctuation
represented as a list
sent = [‘I', ‘love', ‘Dublin', ‘!']
upper_sent = [w.upper() for w in sent]

14. downloading corpus and lexical resources
nltk.download(‘all’)
nltk.download(‘brown’)

15. segment text into sentences
from nltk.tokenize import sent_tokenize
sent_tokenize_list = sent_tokenize(text)

16. tokenize sentence
from nltk.tokenize import word_tokenize
tokens = word_tokenize(sentence)

17. tag part of speech (pos)
tags = nltk.pos_tag(tokens)

19. identify named entities
entities = nltk.ne_chunk(tags)
entities.draw()

22. language recognition
import langid
lang = langid.classify(text)[0]

23. fundamentals of nlp
natural language toolkit (nltk)
running python and nltk on Azure

24. azure cloud services
azure webjobs
azure functions

25. azure cloud services & python
pip’s requirements.txt
PowerShell scripts for setup and launch

26. azure webjobs & python
upload zip (inc. dependencies)
runs run.py (or the first py file it finds)

27. configuration settings
key = os.environ["STORAGE_KEY"]

28. publish webjob
pip packages into site-packages
zip application (inc. depended packages)
upload zip file

29. add package location to sys.path
p = os.path.join(os.getcwd(), "site-packages")
sys.path.append(p)

30. downloading corpus
D:localAppDatanltk_data
if os.getenv("DOWNLOAD", True) == True :
dest = os.environ[“NLTK_DATA_DIR"]
nltk.download('all', dest)

31. using queues for communication
reads text from input queue
writes processed text into output queues

32. auto scale
based on queue length

33. debugging python webjobs
local: vs and webjob simulator
cloud: use kudu (xyz.scm.azurewebsites.net)
and logs

38. nltk is a great toolkit to perform nlp tasks
azure provides an elastic and scalable
platform to run python nltk jobs

39. http://www.nltk.org/
http://www.nltk.org/book_1ed
http://azure.com/