a survey on convolutional neural network based language modeling

1. DCNN for text
B01902004 蔡捷恩

2. A CNN for modeling Sentences
Kalchbrenner, Nal, Edward Grefenstette, and Phil Blunsom. "A convolutional neural network for modelling sentences."arXiv:1404.2188
(2014).

3. Sentence model
• Sentence -> feature vector, that’s all !
• However, it is the core of:
• Sentiment analysis, paraphrase detection,
entailment recognition, summarisation,
discourse analysis, machine translation,
grounded language learning, image retrieval …

4. contribution
• Does not rely on parse tree
• Easily applicable to any language ?

5. How to model a sentence?
• Composition based method
• Need human knowledge to compose
• Automatically extracted logical forms
• Ex. RNN, TDNN

6. Brief network structure
• Interleaving k-max pooling & 1-dim-conv. +
TDNN => generate a sentence graph
A kind of syntax tree ?

7. NN sentence model with syntax tree
(Recursive NN, RecNN)
Reference syntax tree
while training
Share weight
and stack up to
form the network

8. RNN for sentence model
Linear “structure”

9. Back to DCNN
• Convolution
• TDNN
• K-max pooling( Dynamic k-max pooling)

10. Convolution
Narrow type, win=5
wide type, win=5 (0-padding)

11. Max-TDNN
GOAL: recognize features independent of time-shift
(i.e. sequence position)

12. Take a look at
DCNN
Need to be optimized during training
If we use Max-TDNN

13. K-max pooling
• Given k, no matter how many dimension an
input get, pool the top-k ones as output, “the
order of output corresponds to their input”
• Better than max-TDNN by:
– Preserve the order of features
– Discern more finely how high activated feature
react
• Guarantee the length of input to FC
independent of sentence length

14. Only fully connected need fix length
• Intermediate layers can be more flexible
• Dynamic k-max Pooling !

15. Dynamic k-max Pooling
• K is a function of length of the input sentence
and depth of the network
The k of currently concerned layer
Fixed k-max pooling’s k at the top
Total # of conv. in the network ( the depth)
Input sentence length

16. Folding
• Feature detectors in different rows are
independent of each other until the top fully
connected layer
• Simply do vector sum

17. +

18. Properties
• Sensitive to the order of words
• Filters of the first layer model n-grams, n ≤m
• Properties invariance of absolute position
captured by upper layer convs.
• Induce feature graph property

19. Experiments
Sentiment analysis
Stanford Sentiment Treebank
Movie review, 5 scense, +/- label

20. Experiments
Question type prediction
on TREC

21. Experiments
Twitter sentiment dataset, binary label

22. Experiments
• Visualizing feature detectors

23. Think about it
• Can this kind of k-max pooling apply to image
tasks ?

24. A CNN for matching nature language
sentences
Hu, Baotian, et al. "Convolutional neural network architectures for matching natural language sentences." Advances in Neural Information Processing Systems. 2014

25. Why convolution approach
• No need prior knowledge

26. Contribution
• Hierarchical sentence modeling
• The capturing of rich matching patterns at
different levels of abstraction

27. Convolutional Sentence Modeling
Word2vec pre-trained
2-window max pooling
Fixed input len

28. A trick on zero-padding
• The variable length of sentence may be in a
fairly broad range
• Introduce gate operation
• g(z) = <0> while z = <0>, otherwise, <1>
• No bias !

29. Conv + Max pool
Composition

30. RNN vs ConvNet
ConvNet RNN
Hierarchical
structure
W L
Parallelism W L
Capture far away
information
- -
Explainable W L
Variety L W

31. Architecture-I
• Drawback: in forward phase, the representation of each sentence
Is built without knowledge of each other

32. Architecture-II
• Build directly on the interaction space between 2 sentences
• From 1D to 2D convolution
Good trick at pooling

33. 2D max-pooling

34. Model Generality
• Arc-II subsumes Arc-I as a special case

35. Cost function
• Large margin objective
e(.)

36. Experiment – Sentence Completion

37. Experiment – Matching Response to
Tweet

38. Experiment – Paraphrase Identification
• Determine whether two sentences have the
same meaning

39. Discussion
• Sequence is important

40. Zhang, Xiang, and Yann LeCun. "Text Understanding from Scratch." arXiv preprint arXiv:1502.01710 (2015)
Text Understanding from Scratch

41. Contribution
• Character-level input
• No OOV
• Work for both English and Chinese

42. The model
character encoding space
Not encoded character or space
=> All-zero vector
Fixed length window
H e l l o w o r l

43. More detail

44. What about various input length?
• Set to the longest sentence we are going to
see (1014 character used in their experiments)

45. Data augmentation - Thesaurus
• Thesaurus: “a book that lists words in groups
of synonyms and related concepts”
• http://www.libreoffice.org/

46. Comparison models
• Bag-of-word: 5000 most freq. words
• Bag-of-centroids: 5000-means word vectors
on Google News corpus

47. DBpedia Ontology Classification

48. DBpedia Ontology Classification

49. Amazon review sentiment analysis
• 1~5 indicating user’s subjective rating of a
product.
• Collected by SNAP project

50. Amazon review sentiment analysis

51. Amazon review sentiment analysis

52. Yahoo! Answer Topic Classification

53. Yahoo! Answer Topic Classification

54. News Categorization in English

55. News Categorization in English

56. News Categorization in Chinese
• SogouCA and SogouCS
• pypinyin package + jieba Chinese
segmentation system

57. News Categorization in Chinese

58. Conclusion
• We can play a lot of trick with Pooling

59. Thank you