Presentation describes approach with word embeddings to the Box-Plots for Education competition.

1. Word embeddings for
social goods
Kyiv Deep Learning Study Group #1
Sergii Gavrylov

2. Overview
● Problem description
● Data preprocessing
● Bag-of-words
● Continuous bag-of-words
● Weighted continuous bag-of-words
● Convolutional neural network

3. www.drivendata.org

4. Box-Plots for Education

5. Dataset features

6. Dataset labels
Function
Object_Type
Operating_Status
Position_Type
Pre_K
Reporting
Sharing
Student_Type
Use

7. Dataset labels

8. Loss function
Multi-multi-class log loss

9. Preprocessing
● Tokenize all text columns with OpenNLP, lowercase tokens
and filter out stop words ["‐", "-", "-", "‒", "–", "—", "―", "－", "+", "/", "*",
".", ",", "'", "(", ")", """, "&", ":", "to", "of", "and", "or", "for", "the", "a"]

10. Preprocessing
● Tokenize all text columns with OpenNLP, lowercase tokens
and filter out stop words ["‐", "-", "-", "‒", "–", "—", "―", "－", "+", "/", "*",
".", ",", "'", "(", ")", """, "&", ":", "to", "of", "and", "or", "for", "the", "a"]
● Perform softmax normalization for all float columns
● Replace all NaNs in float columns with 0

11. Preprocessing
● Tokenize all text columns with OpenNLP, lowercase tokens
and filter out stop words ["‐", "-", "-", "‒", "–", "—", "―", "－", "+", "/", "*",
".", ",", "'", "(", ")", """, "&", ":", "to", "of", "and", "or", "for", "the", "a"]
● Perform softmax normalization for all float columns
● Replace all NaNs in float columns with 0
● Keep floats intact, replace NaNs with the specified value
or

12. Word representation
One-hot encoding
social [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
public [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]

13. Bag-of-words
Text column features

14. Bag-of-words
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

15. Bag-of-words
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]

16. Bag-of-words
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Sub_Object_Description_bow [0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0]
∑

17. Bag-of-words
● Concatenated BOW features and floats comprise final feature
vector.

18. Bag-of-words
● Concatenated BOW features and floats comprise final feature
vector.
● Replace FTE field NaNs with -1, Total filed NaNs with -20000

19. Bag-of-words
● Concatenated BOW features and floats comprise final feature
vector.
● Replace FTE field NaNs with -1, Total filed NaNs with -20000
● Train sklearn.ensemble.RandomForestClassifier

20. Bag-of-words
● Concatenated BOW features and floats comprise final feature
vector.
● Replace FTE field NaNs with -1, Total filed NaNs with -20000
● Train sklearn.ensemble.RandomForestClassifier
● Score: 0.8671

21. Bag-of-words
(+) Pros
● Simplicity

22. Bag-of-words
(+) Pros
(-) Cons
● Simplicity
● Notion of word similarity is undefined with one-hot encoding

23. Bag-of-words
(+) Pros
(-) Cons
● Simplicity
● Notion of word similarity is undefined with one-hot encoding
social [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
public [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]

24. Bag-of-words
(+) Pros
(-) Cons
● Simplicity
● Notion of word similarity is undefined with one-hot encoding
social [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
public [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
● Impossible to generalize to unseen words

25. Bag-of-words
(+) Pros
(-) Cons
● Simplicity
● Notion of word similarity is undefined with one-hot encoding
social [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
public [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
● Impossible to generalize to unseen words
● One-hot encoding can be memory inefficient

26. Word representation
Distributed representation
social [-0.56, 8.65, 5.32, -3.14]
public [-0.42, 9.84, 4.51, -2.71]

27. Word representation
Cosine similarity

28. Stanford GloVe
Trained on the Common Crawl (840B tokens)
Vector dimensionality is 300
nlp.stanford.edu/projects/glove

29. Sub_Object_Description Function_Description...Text columns
Continuous bag-of-words

30. personal
employees
wages
salaries
services
instructional
staff
training
services
words words
Sub_Object_Description Function_Description...Text columns
...
Continuous bag-of-words

31. personal
employees
wages
salaries
services
instructional
staff
training
services
words vectors words vectors
Sub_Object_Description Function_Description...Text columns
...
Continuous bag-of-words

32. personal
employees
wages
salaries
services
instructional
staff
training
services
words vectors words vectors
Sub_Object_Description Function_Description
∑ ∑
...Text columns
...
Continuous bag-of-words

33. personal
employees
wages
salaries
services
instructional
staff
training
services
words vectors words vectors
Sub_Object_Description Function_Description
∑ ∑
CBOW features
...Text columns
...
...
Continuous bag-of-words

34. personal
employees
wages
salaries
services
instructional
staff
training
services
words vectors words vectors
Sub_Object_Description Function_Description
∑ ∑
CBOW features
...Text columns
...
...
Continuous bag-of-words

35. Continuous bag-of-words
● Concatenated CBOW features and floats comprise the final
feature vector.

36. Continuous bag-of-words
● Concatenated CBOW features and floats comprise the final
feature vector.
● Replace FTE field NaNs with -1, Total filed NaNs with -20000

37. Continuous bag-of-words
● Concatenated CBOW features and floats comprise the final
feature vector.
● Replace FTE field NaNs with -1, Total filed NaNs with -20000
● Train sklearn.ensemble.RandomForestClassifier

38. Continuous bag-of-words
● Concatenated CBOW features and floats comprise the final
feature vector.
● Replace FTE field NaNs with -1, Total filed NaNs with -20000
● Train sklearn.ensemble.RandomForestClassifier
● Score: 0.6616

39. Continuous bag-of-words
(+) Pros
● Simplicity

40. Continuous bag-of-words
(+) Pros
● Simplicity
● Possible to generalize to unseen words

41. Continuous bag-of-words
(+) Pros
● Simplicity
● Possible to generalize to unseen words
(-) Cons
● All words are equal

42. Continuous bag-of-words
(+) Pros
● Simplicity
● Possible to generalize to unseen words
(-) Cons
● All words are equal, but some words are more equal than others

43. Weighted CBOW
Text column features

44. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

45. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

46. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

47. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

48. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

49. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

50. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

51. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

52. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

53. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

54. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

55. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

56. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal

57. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal ×
×
×
×
×

58. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal ×
×
×
×
×
∑
Sub_Object_Description_wcbow

59. Weighted CBOW
Text column features
Sub_Object_Description
employees
wages
salaries
services
personal ×
×
×
×
×
∑
Sub_Object_Description_wcbow

60. Weighted CBOW
● Concatenated WCBOW features and floats compose the final
feature vector.

61. Weighted CBOW
● Concatenated WCBOW features and floats compose the final
feature vector.
● Use softmax normalization for float columns

62. Weighted CBOW
● Concatenated WCBOW features and floats compose the final
feature vector.
● Use softmax normalization for float columns
● Replace all float NaNs with 0

63. Weighted CBOW
● Concatenated WCBOW features and floats compose the final
feature vector.
● Use softmax normalization for float columns
● Replace all float NaNs with 0
● Train softmax jointly with θc parameters

64. Weighted CBOW
● Concatenated WCBOW features and floats compose the final
feature vector.
● Use softmax normalization for float columns
● Replace all float NaNs with 0
● Train softmax jointly with θc parameters
● Score: 0.9159

65. Weighted CBOW
Why so poorly?

66. Weighted CBOW
Why so poorly?
H1: Softmax is not so powerful as Random forest

67. Weighted CBOW
Why so poorly?
H1: Softmax is not as powerful as Random Forest
H2: Model assumes that for describing relevant words it is
enough to use one direction per column in the word space

68. How many directions
should a good model
have?

69. Convolutional NN
personal services wagessalariesemployees

70. Convolutional NN
personal services wagessalariesemployees

71. Convolutional NN
personal services wagessalariesemployees

72. Convolutional NN
personal services wagessalariesemployees

73. Convolutional NN
personal services wagessalariesemployees

74. Convolutional NN
personal services wagessalariesemployees

75. Convolutional NN
personal services wagessalariesemployees

76. Convolutional NN
personal services wagessalariesemployees

77. Convolutional NN
personal services wagessalariesemployees

78. Convolutional NN
personal services wagessalariesemployees

79. Convolutional NN
personal services wagessalariesemployees

80. Convolutional NN
personal services wagessalariesemployees

81. Convolutional NN
personal services wagessalariesemployees

82. Convolutional NN
personal services wagessalariesemployees

83. Convolutional NN
personal services wagessalariesemployees

84. Convolutional NN
personal services wagessalariesemployees

85. Convolutional NN
personal services wagessalariesemployees

86. Convolutional NN
personal services wagessalariesemployees

87. Convolutional NN
personal services wagessalariesemployees

88. Convolutional NN
personal services wagessalariesemployees

89. Convolutional NN
personal services wagessalariesemployees

90. Convolutional NN
personal services wagessalariesemployees

91. Convolutional NN
personal services wagessalariesemployees

92. Convolutional NN
personal services wagessalariesemployees

93. Convolutional NN
personal services wagessalariesemployees

94. Convolutional NN
personal services wagessalariesemployees

95. Convolutional NN
personal services wagessalariesemployees

96. Convolutional NN
personal services wagessalariesemployees

97. Convolutional NN
personal services wagessalariesemployees

98. Convolutional NN
personal services wagessalariesemployees

99. Convolutional NN
personal services wagessalariesemployees

100. Convolutional NN
personal services wagessalariesemployees

101. Convolutional NN
personal services wagessalariesemployees

102. Convolutional NN
personal services wagessalariesemployees

103. Convolutional NN
personal services wagessalariesemployees

104. Convolutional NN
personal services wagessalariesemployees

105. Convolutional NN
personal services wagessalariesemployees
mean
max

106. Convolutional NN
personal services wagessalariesemployees

107. Convolutional NN
personal services wagessalariesemployees

108. Convolutional NN
personal services wagessalariesemployees
mean
max

109. Convolutional NN
personal services wagessalariesemployees

110. Convolutional NN
personal services wagessalariesemployees

111. Convolutional NN
personal services wagessalariesemployees
max
mean

112. Convolutional NN
personal services wagessalariesemployees

113. Convolutional NN
employees
wages
salaries
services
personal

114. Convolutional NN
employees
wages
salaries
services
personal

115. Convolutional NN
employees
wages
salaries
services
personal
×

116. Convolutional NN
employees
wages
salaries
services
personal
× =

117. Convolutional NN
employees
wages
salaries
services
personal
× =
Stride size = word dimensionality

118. Convolutional NN
● Concatenated mean and max values for feature maps and floats form the final
feature vector

119. Convolutional NN
● Concatenated mean and max values for feature maps and floats form the final
feature vector
● Use softmax normalization for float columns

120. Convolutional NN
● Concatenated mean and max values for feature maps and floats form the final
feature vector
● Use softmax normalization for float columns
● Replace all float NaNs with 0

121. Convolutional NN
● Concatenated mean and max values for feature maps and floats form the final
feature vector
● Use softmax normalization for float columns
● Replace all float NaNs with 0
● Train softmax with Wf parameters jointly

122. ● Concatenated mean and max values for feature maps and floats form the final
feature vector
● Use softmax normalization for float columns
● Replace all float NaNs with 0
● Train softmax with Wf parameters jointly
● Score: 0.6932
Convolutional NN

123. Convolutional NN
Why is it not as good as CBOW + RF?

124. Convolutional NN
Why is it not as good as CBOW + RF?
● There is fewer parameters

125. Convolutional NN
Why is it not as good as CBOW + RF?
● There is fewer parameters
● The performance is still comparable to CBOW+RF. Therefore, using cnn
is a sensible idea.

126. Convolutional NN
Why is it not as good as CBOW + RF?
● There is fewer parameters
● The performance is still comparable to CBOW+RF. Therefore, using cnn
is a sensible idea.
● Probably, we could gain more from this type of feature learner if we go
deeper

127. Final model
● Train RF on the concatenated CBOW features and NN logits

128. Final model
● Train RF on the concatenated CBOW features and NN logits
● Train 2 CBOW classifiers, 2 NN classifiers, 2 meta-classifiers
and blend them

129. Final model
● Train RF on the concatenated CBOW features and NN logits
● Train 2 CBOW classifiers, 2 NN classifiers, 2 meta-classifiers
and blend them
● Score: 0.5228

130. Results

131. Conclusion
● Explore your data before doing any analysis
● Keep trying
● Ensembles are powerful
● Participating in competitions provides a great
learning opportunity