1.
Concept Extraction with
Convolutional Neural
Networks
Andreas Waldis, Luca Mazzola, and Michael Kaufmann
HSLU - Lucerne University of Applied Sciences,
School of Information Technology,
6343 - Rotkreuz,
Switzerland
7th International Conference on Data Science, Technology and Applications
DATA 2018 27/07/2018

2.
Slide 2, 27-Jul-18
- XMAS: Cross-platform Mediation, Association and
Search engine
- Knowledge Management Tool
- Automatic document tagging
- Recognition of Concepts
- Represented as N-Grams (sequences of words)
- Objective: create an index based model for
Keyconcept extraction
Context
• XMAS
• Knowledge Management Tool
• Automatic Keywords extraction
DATA 2018 27/07/2018

3.
Slide 3, 27-Jul-18
X-MAS example
• Concepts extracted
• Automatic summarization (KW)
DATA 2018 27/07/2018

4.
Slide 4, 27-Jul-18
- Part of Speech (NLP)
- Based on syntactical characteristics of
language and frequency of typical constructs
- Requires the exhaustive creation of words n-
grams combinations (over linear) and
frequency filtering
- POS limitations
- Language dependent
- Manually laborious to design the acceptable
pattern
- Including longer n-grams reduces significantly
the precision (even if increases coverage)
POS solution
• POS limitations
DATA 2018 27/07/2018

5.
Slide 5, 27-Jul-18
Examples
• POS performances
TP + FP = positive
TN + FN = negative
Pos/Neg class
True/False match
DATA 2018 27/07/2018
P
N
T
F
Concepts
Positive
True
Rutgers Preparatory School
Watts 103rd Street Rhythm Band
Twinkle Twinkle Little Star
Accademia di Belle Arti di Roma
False
Oricon Weekly Albums Chart
Grand Forks-ND-MN Metropolitan Statistical
Apple CEO Steve Jobs
Negative
True
the rims of the
in consonance with the
in which they were written
was interred in Spring Grove Cemetery
False
Los Angeles Film Critics Association Awards
United States Citizenship and Immigration Services
State of North Carolina
1917 October Revolution

6.
Slide
Neural Network

7.
Slide 7, 27-Jul-18
- Capability of identifying automatically:
- Regularities in the data
- Meaning of particular constructs
- Possibilities of add non-linearity by means of ReLU
activation units
- Deep model allows extremely compact network to
understand very complex problems.
- Can use any encoding of data
- We relied on the Word2Vec-plus by Google
Neural Network motivation
• Automatic knowledge extraction
• Multiple hidden layers
• Compatible with every data
encoding available
DATA 2018 27/07/2018

8.
Slide 8, 27-Jul-18
Data preprocessing
DATA 2018 27/07/2018

9.
Slide 9, 27-Jul-18
- Use the Word2Vec-plus
- Holds the word vector, including also some
contextual information
- Can provide a representation for unseen words:
a) Computation based on 4 surronding words
b) Vector update
Word Embedding
• Vector representation of word
• Holds some context, also
• Can also represent unseen word
DATA 2018 27/07/2018

10.
Slide 10, 27-Jul-18
Training Pipeline
DATA 2018 27/07/2018

11.
Slide 11, 27-Jul-18
Vertical vs. Horizontal layers
• Types of convolutions
• Vertical vs. Horizontal
DATA 2018 27/07/2018

12.
Slide
Configurations

13.
Slide 13, 27-Jul-18
Hyperparametrization
• Network configurations
• Parameters setting/limits
DATA 2018 27/07/2018

14.
Slide 14, 27-Jul-18
Evaluation
• Evaluation Procedure
DATA 2018 27/07/2018

15.
Slide 15, 27-Jul-18
- Lenght of N-Gram influences the results
- Percentage of valid concepts different per class:
Data Set distribution
• Dataset characterization
DATA 2018 27/07/2018

16.
Slide
Results
DATA 2018 27/07/2018

17.
Slide 17, 27-Jul-18
K-fold evaluation
• Cross evaluation
• 4-fold, 2 runs per config, 100
epochs training limit
DATA 2018 27/07/2018
F1 = 2*(Recall *
Precision) / (Recall +
Precision)
V6H3 precision along epochs
outlier

18.
Slide 18, 27-Jul-18
Word embedding comprehension
DATA 2018 27/07/2018

19.
Slide 19, 27-Jul-18
Examples
True False
Positive American Educational Research
Journal
Tianjin Medical University
carry out
Bono and The Edge
Sons of the San Joaquin
Glastonbury Lake Village
Earl of Darnley
Regiment Hussars
University of Theoretical Science
Inland Aircraft Fuel Depot
NHL and
Mexican State Senate
University of
Ireland Station
In process
Negative to the start of World War II
must complete their
just a small part
a citizen of Afghanistan who
itself include
NFL and the
a Sky
therefore it is
use by
in conversation with
Council of the Isles of Scilly
Xiahou Dun
The Tenant of Wildfell Hall
DATA 2018 27/07/2018

20.
Slide 20, 27-Jul-18
Cross checking POS vs. CNN
DATA 2018 27/07/2018
Concepts
True (CNN) False (CNN)
Positive
True
Rutgers Preparatory School
Watts 103rd Street Rhythm Band
Twinkle Twinkle Little Star
Accademia di Belle Arti di Roma
Capitanes de Arecibo
Fort Belknap Indian Reservation
False
Republican President Richard
Senator Ted
East Stroudsburg Senior High School North
Charles Bender High School
The New York Times Guide
Zombie Movie Encyclopedia
Negative
True
Toronto was the
in which they were written
are a family of passerine birds which
the Art Center College of Design
the NWA World Middleweight Championship
language novel
False
Legislative Council of New South Wales
1917 October Revolution
EAFF East Asian Cup
West Surrey College of Art and Design
Federal University of Rio Grande do Sul
Los Angeles Film Critics Association Awards
United States Citizenship and Immigration
Services
State of North Carolina
1917 October Revolution

21.
Slide 21, 27-Jul-18
Averaged Performances
DATA 2018 27/07/2018

22.
Slide 22, 27-Jul-18
Learning Curves
DATA 2018 27/07/2018

23.
Slide 23, 27-Jul-18
Performances w.r.t. the N-Gram length
• Dependency from lenght (n)
AUC= Area under Curve
 Global comparison
metric
DATA 2018 27/07/2018

24.
Slide 24, 27-Jul-18
- We presented a CNN approach for automatic
concept extraction
- We demonstrate its competitiveness w.r.t. POS,
holding a slightly better F1 measure
- Increase in recall with loss of precision, with
increasing length of N-Gram.
- Possible next steps:
- Adopt other words embedding models
- Use different n-Gram sources, extracting them
from real world documents
- Use a different architecture (RNN) to try
capturing latent and long running relationship
(LSTM)
- Train individual instances for different n and
using then the aggregated results.
Conclusions
• Results achieved
• Limits still existing
• Next research possibilities
DATA 2018 27/07/2018

25.
T direct
Research
Dr. Luca Mazzola
Research Associate
+41 41 757 68 90
luca.mazzola@hslu.ch
Rotkreuz
Questions
DATA 2018 27/07/2018