Have Data? What now?! Hilary Mason @hmason

(Focused) Data == Intelligence

Common Problems Gathering data Parsing, Entity Extraction and Disambiguation Clustering Document classification NLP

Gathering data

Parsing, Entity Extraction and Disambiguation

Clustering

Document classification

NLP

Text is MESSY

Do you need to parse it? Parsing unstructured data is hard . (we’ll get to this) CHEAT. Open Calais ( www.opencalais.com ) currently supports: Anniversary, City, Company, Continent, Country, Currency, EmailAddress, EntertainmentAwardEvent, Facility, FaxNumber, Holiday, IndustryTerm, MarketIndex, MedicalCondition, MedicalTreatment, Movie, MusicAlbum, MusicGroup, NaturalFeature, OperatingSystem, Organization, Person, PhoneNumber, Position, Product, ProgrammingLanguage, ProvinceOrState, PublishedMedium, RadioProgram, RadioStation, Region, SportsEvent, SportsGame, SportsLeague, Technology, TVShow, TVStation, URL

Parsing unstructured data is hard . (we’ll get to this)

CHEAT.

Open Calais ( www.opencalais.com ) currently supports:

Anniversary, City, Company, Continent, Country, Currency, EmailAddress, EntertainmentAwardEvent, Facility, FaxNumber, Holiday, IndustryTerm, MarketIndex, MedicalCondition, MedicalTreatment, Movie, MusicAlbum, MusicGroup, NaturalFeature, OperatingSystem, Organization, Person, PhoneNumber, Position, Product, ProgrammingLanguage, ProvinceOrState, PublishedMedium, RadioProgram, RadioStation, Region, SportsEvent, SportsGame, SportsLeague, Technology, TVShow, TVStation, URL

Entity Disambiguation This is important .

This is important .

ME UGLY HAG

Entity Disambiguation This is important . Company disambiguation is a very common problem – Are “Microsoft”, “Microsoft Corporation”, and “MS” the same company?

This is important .

Company disambiguation is a very common problem – Are “Microsoft”, “Microsoft Corporation”, and “MS” the same company?

A Practical Approach – Path101 Human classification Data APIs Automatic classification model Example: Company Name External data from Open Calais, Freebase Based on industry , location , and type of job , we can differentiate between MS Volt (Microsoft) and Volt (Volt Information Sciences, Inc.)

SPAM sucks

Supervised Classification Text Feature Extractor Trained Classifier Cats Dogs Fire Training Data Feature Extractor

Classification Example: Movie Reviews! [['plot', ':', 'two', 'teen', 'couples', 'go', 'to', 'a', 'church', 'party', ',', 'drink', 'and', 'then', 'drive', '.'], ['they', 'get', 'into', 'an', 'accident', '.'], ...] …tagged ‘positive’ and ‘negative’.

[['plot', ':', 'two', 'teen', 'couples', 'go', 'to', 'a', 'church', 'party', ',', 'drink', 'and', 'then', 'drive', '.'], ['they', 'get', 'into', 'an', 'accident', '.'], ...]

…tagged ‘positive’ and ‘negative’.

#!/usr/bin/env python # encoding: utf-8 "””classification_example.py""" from __future__ import division import sys, os, random, nltk, re, pprint from nltk.corpus import movie_reviews def document_features( document, word_features): document_words = set(document) features = {} for word in word_features: features[ 'contains( %s )' % word] = (word in document_words) return features def main(): all_words = nltk.FreqDist(w.lower() for w in movie_reviews.words()) word_features = all_words.keys()[: 2000] documents = [( list(movie_reviews.words(fileid)), category) for category in movie_reviews.categories() for fileid in movie_reviews.fileids(category)] featuresets = [(document_features(d, word_features), c) for (d,c) in documents] train_set, test_set = featuresets[ 100:], featuresets[:100] classifier = nltk.NaiveBayesClassifier.train(train_set) print nltk.classify.accuracy(classifier, test_set) classifier.show_most_informative_features( 20) if __name__ == '__main__': main()

#!/usr/bin/env python # encoding: utf-8 "””classification_example.py""" from __future__ import division import sys, os, random, nltk, re, pprint from nltk.corpus import movie_reviews def document_features( document, word_features): document_words = set(document) features = {} for word in word_features: features[ 'contains( %s )' % word] = (word in document_words) return features def main(): all_words = nltk.FreqDist(w.lower() for w in movie_reviews.words()) word_features = all_words.keys()[: 2000] documents = [( list(movie_reviews.words(fileid)), category) for category in movie_reviews.categories() for fileid in movie_reviews.fileids(category)] featuresets = [(document_features(d, word_features), c) for (d,c) in documents] train_set, test_set = featuresets[ 100:], featuresets[:100] classifier = nltk.NaiveBayesClassifier.train(train_set) print nltk.classify.accuracy(classifier, test_set) classifier.show_most_informative_features( 20) if __name__ == '__main__': main()

Clustering immunity ultrasound medical imaging medical devices thermoelectric devices fault-tolerant circuits low power devices

Hierarchical Clustering

<3 Data Thank you!

Thank you!