Slides for my talk at Research Seminar in Language Technology (http://www.helsinki.fi/~tiedeman/teach/ResearchSeminar/)

1. Convolutional neural networks for
text classification
Lidia Pivovarova
Research Seminar in Language Technology
1st June 2017

2. PULS Project
● Web-scale surveillance of news
● Current topic: Business. (Previous topics: Security,
Epidemics, ...)
● Tracking news from thousands of news sites about
business activities
– ≈6000–8000 news items per day
– among hundreds of thousands of entities:
● companies, persons, products, organizations, ...
– tracking many kinds of activities:
● merger, buyout, bankruptcy, layoff, product launch
and recall, ...
TEKES Project, led by Roman Yangarber
http://newsweb.cs.helsinki.fi

3. Neural network
● Each node computes a function on its inputs to produce an
output
● A network ~ a huge formula with many parameters
● Adjustment of parameters given an output and a true value
(back propagation)
● A network structure and the inference are separated
An image from http://www.opennn.net/

4. Convolutional neural networks
images found in the data science blog: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

5. Convolutional neural networks
images found in the data science blog: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

6. Convolutional neural networks
images found in the data science blog: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

7. Convolutional neural networks
images found in the data science blog: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

8. CNN for NLP
images found in the WildML blog:
http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/
also very good tutorial on CNN for NLP with Tensorflow
http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/

9. Polarity detection in business news
General Motors:
Daimler:

10. ● The task: to determine the sentiment polarity of a
mention of a given company in a business news
article
● Similar to (aspect-based) sentiment detection
● But:
– business news articles typically do not aim to express
emotions or subjectivity
– business news contains genre-specific word usages
● Thus we cannot apply existing resources
(dictionaries, labelled corpora) developed for more
general sentiment analysis
Polarity detection in business news

11. Manual annotation
● ~18,000 documents, ~20,000 names
● annotation by groups (Escoter at. el. EACL 2017)
● post-proccessing:
D1: Valeant Pharmaceuticals International Inc., the embattled
Canadian drugmaker, agreed to sell about $2.1 billion in assets to
get cash to streamline its businesses and begin easing its debt
burden.
D2: L’Oreal to buy three skincare brands from Valeant for $1.3 billion.
The French cosmetics giant paid nearly eight times the brand’s
combined annual revenue of $168 million

12. Knowledge transfer
● 2M collection of short business reports
● Manuall annotation with events labels
– 291 labels in total
– 26 labels imply positive polarity: Investment, New
Product, Sponsorship,...
– 12 labels imply negative polarity: Fraud, Layoff,
Bankruptcy,...
– 200,000 documents with exactly one company name
and non-ambiguous polarity
● High-level feature transfer:
– train a model for event labels
– replace the last layer of the network and continue
training for polarity labels

13. Token-based model
– Y.Kim (EMNLP 2014)
+ focus – position of the target company

14. Region-based model
– R. Johnson & T. Zhang (NIPS 2015, NAACL 2015)
+ focus – position of the target company

15. Experiments
● Tune: train on 200,000 documents with mapped polarity labels
when tune using 12,000 manually annotated documents
● Combine: train on 200,000 mapped + 12,000 manually
annotated documents
● Feature transfer: train on 2M documents with original event
labels, replace the last layer, tune to 12,000 manually annotated
documents

16. Examples
● Valeant to sell Dendreon unit to China’s
Sanpower for $820 million. Canada’s Valeant
Pharmaceuticals International Inc. said its
affiliate will sell its Dendreon cancer business to
China’s Sanpower Group Co. Ltd. for $819.9
million, as the drugmaker continues to shed its
non-core assets to repay debt.
● True score: -1.0
● With focus: 0.022
● Without focus: -0.322

17. Examples
● Valued less than Toshiba in 2004, Apple today
has a market capitalization of US$700-billion,
as shares hit a record high close yesterday.
Toshiba shares could soon be relatively
worthless, as they may have to declare
bankruptcy.
● True score: 1.0
● With focus: 0.310
● Without focus: -0.197

18. Examples
● Bailed-out Lloyds Banking Group reports highest annual profit for
ten years. Bottom line profits at the taxpayer-backed lender more
than doubled to £4.24 billion last year, partly due to lower PPI
compensationpayouts. The result marks its best performance at the
UK’s biggest retail banking group since 2006. The government put
£20.3b into the banking group, acquiring a 43 per cent stake to save it
from collapse at the height of the financial crisis. This has now
reduced to less than five per cent following a series of share sales
and the government has indicated that it aims to shed its remaining
stake this year. Announcing the results, Lloyds shares jumped 3.6
per cent and the group said its performance was “inextricably linked
to the health of the UK economy, which has been more resilient than
the market expected” since the referendum on EU membership.
● True score: 0.4
● With focus: 0.179
● Without focus: -0.162

19. Examples
● Facebook CEO Mark Zuckerberg and his wife
are dropping controversial suits they filed in
December to buy small plots of land that are
part of a 700-acre waterfront estate they own
on the island of Kauai in Hawaii.
● True score: 0.0
● With focus: 0.-743
● Without focus: -0.397

20. Examples
● Valued less than Toshiba in 2004, Apple today
has a market capitalization of US$700-billion,
as shares hit a record high close yesterday.
Toshiba shares could soon be relatively
worthless, as they may have to declare
bankruptcy.
● True score: 1.0
● With focus: 0.310
● Without focus: -0.197

21. Examples
● Valued less than Toshiba in 2004, Apple today
has a market capitalization of US$700-billion,
as shares hit a record high close yesterday.
Toshiba shares could soon be relatively
worthless, as they may have to declare
bankruptcy.
● True score: 1.0
● With focus: 0.310
● Without focus: -0.197

22. Behind the scenes

24. Thanks for your attention!
● More details can be found in:
– L. Pivovarova, L. Escoter, A. Klami, & R. Yangarber SemEval 2017
– Also in my future publications...