This document discusses building a large text corpus from various data sources for natural language processing. It covers ingesting data through RSS feeds and saving it as text files. It also covers preprocessing the raw text corpus by extracting paragraphs, sentences, words, and part-of-speech tags to create a tokenized corpus. Finally, it discusses managing the corpus by mapping categories to subdirectories and creating readers for the raw and processed text corpora.

1. Building a Gigaword Corpus
Data Ingestion, Management, and Processing for
Natural Language Processing with Python

2. Rebecca Bilbro

3. ●
●
○
○
○
●
●
●

4. Natural language
processing

5. Everyone’s doing it!

6. …
✓
✗
…
第13届上海国际电影节开幕…
Dan Jurafsky

7. NLTK

8. chat.py

9. User Tweet
Deep Learning Model
Response
Tweet
Past Tweets
Train Phase Deploy Phase
retrain as dataset grows

10. Except ...

11. learning = knowing + growing

12. Health
Weather
Cooking
Books
medical texts
WebMD
nutrition information
meteorological reports
NOAA data
AllRecipes
food blogs
Amazon Book Reviews
GoodReads

13. Fine then,
we’ll make our own corpus

14. Ingestion

15. import bs4
import slugify
import feedparser
TAGS = ['h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'h7', 'p', 'li']
def parse(feed):
content = feedparser.parse(feed)
posts = content.entries
for post in posts:
html = post.content[0].get('value')
soup = bs4.BeautifulSoup(html, "lxml")
filename = slugify.slugify(post.title).lower() + '.txt'
for tag in soup.find_all(TAGS):
paragraph = tag.get_text()
with open(filename, 'a') as f:
f.write(paragraph + "n n")
Given a list of RSS feeds, download all the posts and save them as text files

16. Uh...

18. Processing

19. From each doc, extract html, identify paras/sents/words, tag with part-of-speech
Raw Corpus
HTML
corpus = [(‘How’, ’WRB’),
(‘long’, ‘RB’),
(‘will’, ‘MD’),
(‘this’, ‘DT’),
(‘go’, ‘VB’),
(‘on’, ‘IN’),
(‘?’, ‘.’),
...
]
Paras
Sents
Tokens
Tags

20. Oops

21. Management

22. Streaming
Corpus Preprocessing
Tokenized
Corpus
Extract and stream corpus, preprocess and store, read from transformed corpus
HTML
Paras
Sents
Tokens
Tags

23. Map corpus categories to subdirectories, corpus readers for raw and processed text
corpus
├── citation.bib
├── feeds.json
├── LICENSE.md
├── manifest.json
├── README.md
└── books
├── 56d629e7c1808113ffb87eaf.html
├── 56d629e7c1808113ffb87eb3.html
└── 56d629ebc1808113ffb87ed0.html
└── business
├── 56d625d5c1808113ffb87730.html
├── 56d625d6c1808113ffb87736.html
└── 56d625ddc1808113ffb87752.html
└── cinema
├── 56d629b5c1808113ffb87d8f.html
├── 56d629b5c1808113ffb87d93.html
└── 56d629b6c1808113ffb87d9a.html
└── cooking
├── 56d62af2c1808113ffb880ec.html
├── 56d62af2c1808113ffb880ee.html
└── 56d62af2c1808113ffb880fa.html
Preprocessing
Transformer
Raw
CorpusReader
Tokenized
Corpus
Post-processed
CorpusReader

24. Spoilers!

25. Training Text,
Documents
Labels
Feature
Vectors
Classifier
Algorithm
New Document Feature
Vector
Predictive
Model
Classification

26. Instances
Feature
Vectors
Clustering
Algorithm
New Instance Feature
Vector
Topic A Topic B Topic C
Similarity

27. Data Management Layer
Raw Data
Feature Engineering Hyperparameter Tuning
Algorithm Selection
Model Selection Triples
Instance
Database
Model Storage
Model
Family
Model
Form

28. Data Loader
Text
Normalization
Text
Vectorization
Feature
Transformation
Estimator
Data Loader
Feature Union Pipeline
Estimator
Text
Normalization
Document
Features
Text Extraction
Summary
Vectorization
Article
Vectorization
Concept Features
Metadata Features
Dict Vectorizer

29. Data Ingestion Wrangling Preprocessing
WORM Store
Analytics
Corpus Reader
Preprocessing
Corpus Reader
Raw Corpus
Tokenized
Corpus
Text
Vectorization
Model Fitting
Model Store
Application
Lexical
Resources
Feedback

30. Baleen & Minke

31. Minke Corpus
Processing
Extract noun keyphrases
weighted by TF-IDF.
Baleen
Corpus Ingestion
Routine Document
Collection Every Hour

33. Yellowbrick

34. Shameless
Plug

35. Thank you!