Presentation at Web Directions 2010, Atlanta, GA.

1. Machine Learning for Web
Data
Hilary Mason
Web Directions USA 2010

3. = new capacities
(superpowers)
Machine learning is a way of
thinking about data.

5. http://www.meetup.com/NYC-Tech-Talks/calendar/12939544/
?from=list&offset=0
http://bit.ly/9N7VB1

6. 6

7. wicked hard problem
10s of
millions of
URLs /day
100s of
millions of
events /
day
1000s of
millions of

11. @hmason

12. [archive photo]

13. ELIZA

17. ML Today

18. Algorithms +
On-demand computing +
Ubiquitous data

19. Algorithms
New frames for modeling the world with
data.

21. [moar data and new kinds of data]

22. Examples

23. [spam filters]

24. [netflix movie recommendations]

25. Language Identification

26. Face Identification

27. Machine Learning

28. Supervised Learning
Vs
Unsupervised Learning

29. Clustering
immunity
ultrasound
medical
imaging
medical
devices
thermoelectric
devices
fault-tolerant
circuits
low power
devices

30. Entity disambiguation
This is important.

31. ME
UGLY HAG

32. Entity disambiguation
This is important.
Company disambiguation is a very common
problem – Are “Microsoft”, “Microsoft
Corporation”, and “MS” the same
company?

33. Classification

34. classification
Text
Feature
Extractor
Trained
Classifier
Cats
Dogs
Fire
Training
Data
Feature
Extractor

35. <math>

36. Probability
P(A) is the probability that A is true.

37. Axioms of Probability
0 ≤ P(A) ≤ 1
P(True) = 1
P(False) = 0
P(A or B) = P(A) + P(B) – P(A and B)

38. P(A or B) = P(A) + P(B) – P(A and B)
P(A)
P(B)
P(A and B)

39. Bayes Law

40. Example
There are
10,000 people.
1% have a rare
disease.

41. Example
• Population of 10,000
• 1% have rare disease
• There’s a test that is 99% effective.
– 99% of sick patients test positive
– 99% of healthy patients test negative

42. Given a positive test result, what is the probability
that the patient is sick?

44. Disease Diagnosis
99 sick patients test positive, 99 healthy
patients test positive
Given a positive test, there is a 50%
probability that the patient is sick.

45. Bayesian Disease
Know the prob. of testing sick given healthy,
and healthy given sick
Use Bayes theorem to invert probabilities

46. </math>

47. Obtain
Scrub
Explore
Model
iNterpret

49. 1. Obtain Data
“pointing and clicking does not scale!”
http://www.delicious.com/pskomoroch/dataset

50. lynx –dump
http://www.nytimes.com
Lynx: http://bit.ly/a6Pumm
2. Scrub

52. 3. Explore
http://vis.stanford.edu/protovis/

53. 4. Model
Google Prediction API
http://code.google.com/apis/predict/

54. 4. Model
Python
• NLTK - http://www.nltk.org/
• Scikits Learn - http://scikit-
learn.sourceforge.net/

55. 4. Model
http://www.alchemyapi.com/

56. 5. Interpret
Andrew Vande Moore – Visual Poetry 06

57. http://www.dataists.com

58. One Final Example
Twitter is full of noise.
Sports – down
Math – UP!
Narcissism - down

59. Code!

60. Filtering & Relevance Ordering
http://github.com/hmason/tc

61. What’s next?

62. Soon:
Natural Language Generation
Rich media classification
Contextual everything

63. Algorithms-As-A-Service

64. infer links in data

65. Filtering

66. Relevance

67. h@bit.ly @hmason
Thank you!