Practical Data Analysis in PythonHilary Mason@hmasonwww.hilarymason.comhilary@path101.com

Data is ubiquitous.The ability and tools to use it are not.

(Focused) Data == Intelligence

Data Analysis on the WebData items change rapidly.Data items are not independent.There’s a lot of semi-structured data around.There’s a LOT of data around.==Too many problems, few tools, and few experts.

Entity DisambiguationThis is important.

UGLY HAGME

Entity DisambiguationThis is important.Company disambiguation is a very common problem – Are “Microsoft”, “Microsoft Corporation”, and “MS” the same company?This is a hard problem.

SPAM sucks

ClassificationDocument classification.Image recognition.Topic recognition.

Text Parsing

Recommendation SystemsProduct recommendations.Disease predictions.Behavior analysis.

IEEE Tag Clusteringimmunityfault-tolerant circuitsmedical imagingultrasoundlow power devicesthermoelectric devicesmedical devices

Python for Data Analysisimport why_python_is_awesomePython is readable.Easy to transition from Matlab or R.Numerical computing support.Growing set of machine learning libraries.

LibrariesNLTK (Natural Language Toolkit) – www.nltk.orgmlpy (Machine Learning PY) – mlpy.fbk.eunumpy & scipy – scipy.org

An EC2 AMI provisioned with all of the toys you need:http://blog.infochimps.org/2009/02/06/start-hacking-machetec2-released/MachetEC2

Demo: Classifying Tweets

Supervised ClassificationSpamTraining DataFeature ExtractorTextFeature ExtractorTrainedClassifierNot Spam

Data: TweetsHand-classified. For example, some spam:| don't disrespect me. I just wanted yall to get a head start so don't feel bad when I have more followers in two days. http://xyyx.eu/a1ha || oh yay more new followers..hiii...if u want go to http://xyyx.eu/a1hb || My friend made this new tool to get more twitter followers, http://xyyx.eu/a1ht || Yes, Twitter is doing some Follower/Following count corrections. Get it back at: http://xyyx.eu/a1h8 || man if i see one more person cry about losing followers!!! http://xyyx.eu/a1h4 |

Featuresdef document_features(self, document):document_words= set(document) features = {} for word in self.word_features:features['contains(%s)' % word] = (word in document_words) return featuresBreak tweets into lists of relevant words.

Naïve Bayesian ClassiferP(A|B) = the conditional probability of A given Bhttp://yudkowsky.net/rational/bayeshttp://blog.oscarbonilla.com/2009/05/visualizing-bayes-theorem/classifier = nltk.NaiveBayesClassifier.train(train_set)

Classifer AccuracyUse a hand-classified test set to see the accuracy of the classifier:nltk.classify.accuracy(classifier, test_set)

Feature Relevance contains(') = True not_s : spam = 53.6 : 1.4 contains(") = True not_s : spam = 32.2 : 1.1 contains(#) = True not_s : spam = 22.0 : 1.0 contains(!) = True not_s : spam = 10.8 : 1.0 contains(*) = True spam : not_s = 7.4 : 1.0 contains(=) = True not_s : spam = 5.5 : 1.0contains(i) = False spam : not_s = 5.2 : 1.0 contains(?) = True not_s : spam = 2.4 : 1.0 contains(:) = True spam : not_s = 2.3 : 1.0 contains(&) = True not_s : spam = 1.8 : 1.0 contains(;) = True not_s : spam = 1.6 : 1.0 contains($) = True spam : not_s = 1.5 : 1.0contains(u) = True spam : not_s = 1.5 : 1.0 contains(2.0) = False not_s : spam = 1.4 : 1.0contains(saw) = False not_s : spam = 1.4 : 1.0contains(noble) = False not_s : spam = 1.4 : 1.0contains(sound) = False not_s : spam = 1.3 : 1.0contains(approach) = False not_s : spam = 1.3 : 1.0contains(finally) = False not_s : spam = 1.3 : 1.0contains(more) = False spam : not_s = 1.3 : 1.0

Kitchen Sinkwash, rinse, repeat

Results90% accuracy on spam tweets – not bad!Other possibilities: categorization – what do you tweet about? human vsbot? which celebrity tweeter are you?

<3 DataThank you!