1. Text classification
With Apache Mahout and Lucene

2. Isabel Drost-Fromm
Software Engineer at Nokia Maps*
Member of the Apache Software Foundation
Co-Founder of Berlin Buzzwords and
Berlin Apache Hadoop GetTogether
Co-founder of Apache Mahout
*We are hiring, talk to me or mail careers@here.com

5. TM

6. https://cwiki.apache.org/confluence/display/MAHOUT/Powered+By+Mahout
… provide your own success story online.

7. TM

8. Classification?

10. January 8, 2008 by Pink Sherbet Photography
http://www.flickr.com/photos/pinksherbet/2177961471/

11. By freezelight, http://www.flickr.com/photos/63056612@N00/155554663/

12. http://www.flickr.com/photos/29143375@N05/3344809375/in/photostream/
http://www.flickr.com/photos/redux/409356158/

13. http://www.flickr.com/photos/29143375@N05/3344809375/in/photostream/
http://www.flickr.com/photos/redux/409356158/

14. Image by jasondevilla
http://www.flickr.com/photos/jasondv/91960897/

15. How a linear classifier sees data

16. Image by ZapTheDingbat (Light meter)
http://www.flickr.com/photos/zapthedingbat/3028168415

17. Instance*
(sometimes also called example, item, or in databases a row)

18. Feature*
(sometimes also called attribute, signal, predictor, co-variate, or column in databases)

19. Label*
(sometimes also called class, target variable)

23. Image taken in Lisbon/ Portugal.

24. Image by jasondevilla
http://www.flickr.com/photos/jasondv/91960897/

26. ●
Remove noise.

28. ●
Remove noise.
●
Convert text to vectors.

29. Text consists of terms and phrases.

30. Encoding issues?
Chinese? Japanese?
“New York” vs. new York?
“go” vs. “going” vs. “went” vs. “gone”?
“go” vs. “Go”?

31. Terms? Tokens? Wait!

33. Now we have terms – how to turn them
into vectors?

34. If we looked at two phrases only:
Sunny weather
High performance computing

35. Aaron
Zuse

36. Binary bag of words
●
Imagine a n-dimensional space.
●
Each dimension = one possible word in texts.
●
Entry in vector is one, if word occurs in text.
●
Problem:
–
bi , j =
{
1 ∀ x i ∈d j
0 else
}
How to know all possible terms in unknown text?

37. Term Frequency
●
Imagine a n-dimensional space.
●
Each dimension = one possible word in texts.
●
Entry in vector equal to the words frequency.
bi , j =ni , j
●
Problem:
–
Common words dominate vectors.

38. TF with stop wording
●
Imagine a n-dimensional space.
●
Each dimension = one possible word in texts.
●
Filter stopwords.
●
Entry in vector equal to the words frequency.
●
Problem:
–
bi , j =ni , j
Common and uncommon words with same weight.

39. TF- IDF
●
Imagine a n-dimensional space.
●
Each dimension = one possible word in texts.
●
Filter stopwords.
●
Entry in vector equal to the weighted frequency.
●
Problem:
–
bi , j =ni , j ×log 
∣D∣

∣{ d : t i ∈d }∣
Long texts get larger values.

40. Hashed feature vectors
●
Imagine a n-dimensional space.
●
Each word in texts = hashed to one dimension.
●
Entry in vector set to one, if word hashed to it.

42. <

43. How a linear classifier sees data

45. HTML
Tokenstream+x
Apache Tika
FeatureVector
Encoder
Fulltext
Lucene
Analyzer
Vector
Online
Learner
Model

46. Image by ZapTheDingbat (Light meter)
http://www.flickr.com/photos/zapthedingbat/3028168415

47. Goals
●
Did I use the best model parameters?
●
How well will my model perform in the wild?

48. Tune model
Parameters,
Experiment with
Tokenization,
Experiment with
Vector Encoding
Compute expected
performance

50. Performance
●
Use same data for training and testing.
●
Problem:
–
Highly optimistic.
–
Model generalization unknown.

51. Performance
●
Use same data for training and testing.
DON'T
●
Problem:
–
Highly optimistic.
–
Model generalization unknown.

52. Performance
●
Use just a fraction for training.
●
Set some data aside for testing.
●
Problems:
–
Pessimistic predictor: Not all data used for training.
–
Result may depend on which data was set aside.

53. Performance
●
Partition your data into n fractions.
●
Each fraction set aside for testing in turn.
●
Problem:
–
Still a pessimistic predictor.

54. Performance
●
Use just a fraction for training.
●
Set some data aside for tuning and testing.
●
Problems:
–
Highly optimistic.
–
Parameters manually tuned to testing data.

55. Performance
●
Use just a fraction for training.
●
Set some data aside for tuning and testing.
DON'T
●
Problems:
–
Highly optimistic.
–
Parameters manually tuned to testing data.

56. Performance
●
Use just a fraction for training.
●
Set some data aside for tuning.
●
Set another set of data aside for testing.
●
Problems:
–
Pretty pessimistic as not all data is used.
–
May depend on which data was set aside.

57. Performance Measures

58. Correct prediction: negative
Model
prediction:
negative
Model
prediction:
positive
Correct prediction: positive

59. Accuracy
ACC=
●
true positivetrue negative
true positive false positive false negativetrue negative
Problems:
–
What if class distribution is skewed?

60. Precision/ Recall
true positive
Precision=
true positive false positive
true positive
Recall=
true positive false negative
●
Problem:
–
Depends on decision threshold.

61. ROC Curves

62. ROC Curves
Orange rate

63. ROC Curves
True orange rate
False orange rate

64. ROC Curves
True orange rate
False orange rate

65. ROC Curves
True orange rate
False orange rate

66. ROC Curves
True orange rate
False orange rate

67. ROC Curves
True orange rate
False orange rate

68. AUC – area under ROC
True orange rate
False orange rate

69. Foto taken by fras1977
http://www.flickr.com/photos/fras/4992313333/

70. Image by Medienmagazin pro
http://www.flickr.com/photos/medienmagazinpro/6266643422

72. http://www.flickr.com/photos/generated/943078008/

73. Apache Hadoop-ready
Recommendations/
Collaborative filtering
kNN and matrix factorization
based Collaborative filtering
Classification/
Naïve Bayes, random forest
Frequent item sets/
(P)FPGrowth
Classification/
Logistic Regression/ SGD
Clustering/ Mean shift, k-Means,
Canopy, Dirichlet Process,
Co-Location search
Sequence learning/
HMM
Math libs/ Mahout collections
LDA

74. Libraries to have a look at:
Vowpal Wabbit Mallet
LibSvm
LibLinear
Libfm
Incanter
GraphLab
Skikits learn
Where to get more information:
“Mahout in Action” - Manning
“Taming Text” - Manning
“Machine Learning” - Andrew Ng
https://cwiki.apache.org/confluence/dis
play/MAHOUT/Books+Tutorials+and+T
alks
https://cwiki.apache.org/confluence/dis
play/MAHOUT/Reference+Reading
Image by pareeerica
http://www.flickr.com/photos/pareeerica/3711741298/
Frameworks worth mentioning:
Apache Mahout
Matlab/ Otave
Shogun
RapidI
Apache Giraph
R
Weka
MyMedialight
Get your hands dirty:
http://kaggle.com
https://cwiki.apache.org/confluence/dis
play/MAHOUT/Collections
Where to meet these people:
RecSys
NIPS
KDD
PKDD
ApacheCon
O'Reilly Strata
ICML
ECML
WSDM
JMLR
Berlin Buzzwords

75. Get started today with the right tools.
January 8, 2008 by dreizehn28
http://www.flickr.com/photos/1328/2176949559

76. Discuss ideas and problems online.
November 16, 2005 [phil h]
http://www.flickr.com/photos/hi-phi/64055296

77. Images taken at Berlin Buzzwords 2011/12/13 by
Philipp Kaden. See you there end of May 2014.
Discuss ideas and problems in person.

79. Become a committer yourself

80. BerlinBuzzwords.de – End of May 2014 in Berlin/ Germany.
http://
Online – user/dev@mahout.apache.org, java-user@lucene.apache.org,
dev@lucene.apache.org
Interest in solving hard problems.
Being part of lively community.
Engineering best practices.
Bug reports, patches, features.
Documentation, code, examples.
Image by: Patrick McEvoy

84. http://www.flickr.com/photos/29143375@N05/3344809375/in/photostream/
http://www.flickr.com/photos/redux/409356158/

87. http://www.flickr.com/photos/29143375@N05/3344809375/in/photostream/
http://www.flickr.com/photos/redux/409356158/

88. By freezelight, http://www.flickr.com/photos/63056612@N00/155554663/