Presentation of the Master thesis defense on 04/13/2020. Title: Context-aware Graph Embedding for Session-based News Recommendation Abstract: Existing methods on session-based news recommendation mainly focus on extracting features from news articles and sequential user-item interactions, but they usually ignore the semantic-level structural information among news articles and do not explore external knowledge sources. In this paper, we propose a novel Context-Aware Graph Embedding (CAGE) framework for session-based news recommendation, which builds an auxiliary knowledge graph to enrich the semantic meaning of entities involved in articles, and further refines the article embeddings by graph convolutional networks. Experimental results on real-world news datasets demonstrate the effectiveness of our method compared with the state-of-the-art methods on session-based news recommendation

1. Context-aware Graph Embedding for Session-based
News Recommendation
Master Thesis By Heng-Shiou Sheu
Advisory committee
Dr. Sheng Li (Major Professor)
University of Georgia
Dr. Le Guan
University of Georgia
Dr. Jaewoo Lee
University of Georgia
April 13 2020
(University of Georgia) CAGE April 13 2020 1 / 34

2. Table of Contents
1 Problem Statement and Example
2 Related Work
3 Our Approach
4 Experiments
5 Conclusions
6 References
(University of Georgia) CAGE April 13 2020 2 / 34

3. Table of Contents
1 Problem Statement and Example
2 Related Work
3 Our Approach
4 Experiments
5 Conclusions
6 References
(University of Georgia) CAGE April 13 2020 3 / 34

4. Introduction
Figure: News portal
(University of Georgia) CAGE April 13 2020 4 / 34

5. Introduction
Research trends on News Recommender System(NRS) 1
1
Karimi, Jannach, and Jugovac, “News recommender systems–Survey and roads ahead”.[Information Processing and
Management, 2018]
(University of Georgia) CAGE April 13 2020 4 / 34

6. Introduction
News recommendation challenges
Fast growing number of items - Hundreds of new articles are added
daily in news media.
Accelerated decay of item’s value - information value decays over
time. Especially in news domain. Each article is expected to have a
short shelf life.
Sparse user profiles - The majority of readers are anonymous and they
actually read only a few articles in a session. It leads that an extreme
levels of sparsity in the user-item interactions matrix.
User’s preferences shift - User’s current interest in a session may be
affected by the context (e.g., location, access time) or by global
context (e.g. breaking news).
(University of Georgia) CAGE April 13 2020 5 / 34

7. Put our question into real context
Figure: Instance of news recommendation problem
(University of Georgia) CAGE April 13 2020 6 / 34

8. Introduction
News recommendation challenges
Fast growing number of items - Hundreds of new articles are added
daily in news media.
Accelerated decay of item’s value - information value decays over
time. Especially in news domain. Each article is expected to have a
short shelf life.
Sparse user profiles - The majority of readers are anonymous and they
actually read only a few articles in a session. It leads that an extreme
levels of sparsity in the user-item interactions matrix.
User’s preferences shift - User’s current interest in a session may be
affected by the context (e.g., location, access time) or by global
context (e.g. breaking news).
(University of Georgia) CAGE April 13 2020 7 / 34

9. Introduction
News recommendation challenges
Fast growing number of items - Hundreds of new articles are added
daily in news media.
Accelerated decay of item’s value - information value decays over
time. Especially in news domain. Each article is expected to have a
short shelf life.
Sparse user profiles - The majority of readers are anonymous and they
actually read only a few articles in a session. It leads that an extreme
levels of sparsity in the user-item interactions matrix.
User’s preferences shift - User’s current interest in a session may be
affected by the context (e.g., location, access time) or by global
context (e.g. breaking news).
Thus, short-term interested concepts have to be modeled by exploiting a
few user-article interactions, leading to the session-based
recommendation problem.
(University of Georgia) CAGE April 13 2020 7 / 34

10. Introduction
Session-based recommendation
Goal - Recommend a user the next possible interested item based on
sequential information within a short time period.
Recent research
Recurrent neural networks methods: GRU4Rec 2
, KSR 3
Attention mechanism: STAMP 4
Graph neural networks method: SR-GNN 5
2
Hidasi et al., “Session-based recommendations with recurrent neural networks”.[ICLR ’16]
3
Huang et al., “Improving sequential recommendation with knowledge-enhanced memory networks”.[SIGIR’18]
4
Liu et al., “STAMP: short-term attention/memory priority model for session-based recommendation”.[KDD’18]
5
Wu, Shu et al. “Session-based recommendation with graph neuralnetworks”.[AAAI’19]
6
Souza Pereira Moreira, Ferreira, and Cunha, “News session-based recommendations using deep neural networks”.[DLRS’18]
7
Moreira, Jannach, and Cunha, “Contextual hybrid session-based news recommendation with recurrent neural
networks”.[IEEE Access’19]
(University of Georgia) CAGE April 13 2020 8 / 34

11. Introduction
Session-based recommendation
Goal - Recommend a user the next possible interested item based on
sequential information within a short time period.
Recent research
Recurrent neural networks methods: GRU4Rec 2
, KSR 3
Attention mechanism: STAMP 4
Graph neural networks method: SR-GNN 5
News domain: CHAMELEON 6 7
2
Hidasi et al., “Session-based recommendations with recurrent neural networks”.[ICLR ’16]
3
Huang et al., “Improving sequential recommendation with knowledge-enhanced memory networks”.[SIGIR’18]
4
Liu et al., “STAMP: short-term attention/memory priority model for session-based recommendation”.[KDD’18]
5
Wu, Shu et al. “Session-based recommendation with graph neuralnetworks”.[AAAI’19]
6
Souza Pereira Moreira, Ferreira, and Cunha, “News session-based recommendations using deep neural networks”.[DLRS’18]
7
Moreira, Jannach, and Cunha, “Contextual hybrid session-based news recommendation with recurrent neural
networks”.[IEEE Access’19]
(University of Georgia) CAGE April 13 2020 8 / 34

12. Table of Contents
1 Problem Statement and Example
2 Related Work
3 Our Approach
4 Experiments
5 Conclusions
6 References
(University of Georgia) CAGE April 13 2020 9 / 34

13. Related work
Main inspirations:
GRU4Rec - The seminal work on the usage of Recurrent Neural
Networks(RNN) on session-based recommendations. 2
CHAMELEON - A Deep Learning Meta-Architecture for
Session-based news recommendation, by extracting textual
features from news article and taking sequential information into
account. 7
DKN - A deep knowledge-aware network that takes advantage of
knowledge graph representation in news recommendation. 8
2
Hidasi et al., “Session-based recommendations with recurrent neural networks”.[ICLR ’16]
7
Moreira, Jannach, and Cunha, “Contextual hybrid session-based news recommendation with recurrent neural
networks”.[IEEE Access’19]
8
Wang et al., “DKN: Deep Knowledge-Aware Network for News Recommendation”.[WWW’18]
(University of Georgia) CAGE April 13 2020 10 / 34

14. Limitations of existing work
Unaware of external knowledge sources
Ignore the semantic-level structural information among news articles
Lackness of discovering latent knowledge-level connections among the
news.
Figure: Illustration of real world example
(University of Georgia) CAGE April 13 2020 11 / 34

15. Research Challenges
In this work, our goal is answering that
How to extract semantically meaningful embeddings for news articles?
How to uncover the structural information among news articles?
(University of Georgia) CAGE April 13 2020 12 / 34

16. Table of Contents
1 Problem Statement and Example
2 Related Work
3 Our Approach
4 Experiments
5 Conclusions
6 References
(University of Georgia) CAGE April 13 2020 13 / 34

17. To address the above research challenges
Proposed context-aware graph embedding (CAGE) framework
Take advantage of semantic information from knowledge graph to
enrich the embeddings of news articles.
Model the similarities among articles using graph convolutional
networks to uncover the structural information.
Conduct experiments on real-world benchmark datasets.
Proposed Strategy
Exploit semantically structure information into account to enrich the
textual information.
Exploit the neighborhood information amongs the articles.
(University of Georgia) CAGE April 13 2020 14 / 34

18. Proposed Strategy
1. Exploit semantically structure information into account to enrich the
textual information.
Figure: Illustration of discovering semantic-level latent connections
(University of Georgia) CAGE April 13 2020 15 / 34

19. Proposed Strategy
1. Exploit semantically structure information into account to enrich the
textual information.
Figure: Illustration of discovering semantic-level latent connections
(University of Georgia) CAGE April 13 2020 15 / 34

20. Proposed Strategy
2.Exploit the neighborhood information amongs the articles.
Figure: Illustration of exploiting neighborhood structual information
(University of Georgia) CAGE April 13 2020 16 / 34

21. Framework: overview
Figure: Workflow of our CAGE framework
(University of Georgia) CAGE April 13 2020 17 / 34

22. Framework: Represent textual feature
Figure: Represent Article Textual Feature
Goal:
Learn news representations
from textual content
Steps:
Get word embedding from
pre-trained model
Instantiated by Deep Natural
Language Process architecture
(University of Georgia) CAGE April 13 2020 18 / 34

23. Framework: Extract semantically meaningful embeddings
Figure: Represent semantic-level Feature
Goal:
Represent semantic-level
embedding from mentioned
entity
Steps:
Disambiguate the mentioned
entity in an open knowledge
graph.
Adopt entity-linking
technology to distinguish the
meaning in the news article.
Extract triples from open
knowledge graph
Construct a sub-knowledge
graph
Apply KG embedding method
to learn entity vector.
(University of Georgia) CAGE April 13 2020 19 / 34

24. Framework: Refinement Learning
Figure: Exploit the neighborhood
information among articles
Goal:
Refine the article embeddings
with similar concepts among
the session.
Steps:
Concatenate obtained
textual-level and
semantic-level article
embeddings with side
information.
Represent each article in a
session as a node in graph.
Using graph neural networks
to further refine the article
embeddings.
(University of Georgia) CAGE April 13 2020 20 / 34

25. Framework: Represent sequential information
Figure: Represent sequential information
Goal:
Leverage the user session
information, as the sequential
interactions indicate user’s
preferences for session-based
recommmendations.
Steps:
Take refined article embedding
as input for GRU network
Apply soft-attention
mechanism to capture user
preference within the session
(University of Georgia) CAGE April 13 2020 21 / 34

26. Framework: Train the framework
Goal:
Train the proposal framework to maximize the similarity between user’s
session and next-click article, and minimize the similarity between
session and articles not read by user.
Steps:
Evaluate the posterior probability of clicking a candidate news article
P(item+
|ˆht) =
exp(γRel(ˆht, item+
))
Σ∀item∈U exp(γRel(ˆht ,item))
, (1)
where γ is a smoothing factor and U is the union of item+
and U−
.
loss function
L(θ) = − log
(ht ,item+)
P(item+
|ht), (2)
where θ denotes the model parameters. Since L(θ) is differentiable
w.r.t to θ, the proposed CAGE framework can be trained using gradient
descent based optimization algorithms.
(University of Georgia) CAGE April 13 2020 22 / 34

27. Table of Contents
1 Problem Statement and Example
2 Related Work
3 Our Approach
4 Experiments
5 Conclusions
6 References
(University of Georgia) CAGE April 13 2020 23 / 34

28. Dataset
Adressa
We evaluate our proposed framework on real-world news dataset. This
dataset contains approximately 20 million page visits from a Norwegian
news, which was collected by the Norwegian University of Science and
Technology and Adressavisen. 9
Language Norwegian(Bokm˚al) Norwegian(Bokm˚al)
Days 7 16
Users 251k 314k
Sessions 805k 928k
Clicks 2,286k 2,648k
Articles 11k 13k
Avg. session length* 2.26 2.70
9
Jon Atle Gulla et al. “The Adressa dataset for news recommendation”. [In:Proceedings of the international conference on
web intelligence.’17]
(University of Georgia) CAGE April 13 2020 24 / 34

29. Metrics
Accuracy
Hit Rate (HR)@10 - Check whether the positive item is among the
top-N ranked items.
Mean Reciprocal Rank (MRR)@10 - Ranking metric which assigns
higher at top ranks
Normalized Discounted Cumulative Gain (NDCG)@10 - Considering
the checkin rating value of the user as the graded relevance label.
(University of Georgia) CAGE April 13 2020 25 / 34

30. Parameter Setting
Knowledge graph embedding: 50, 100
Graph convolutional Networks layer1 output: 100,125,150,250
Graph convolutional Networks layer2 output: 50,60,70,150
Batch size: 32, 64, 128, 256
Finally, we set the dimensions for KG, GCN layer1, GCN layer2, and batch
size to 100, 125, 60, 64. The key parameter settings for baselines are same
as configurations reported in Chameleon 7
7
Moreira, Jannach, and Cunha, “Contextual hybrid session-based news recommendation with recurrent neural
networks”.[IEEE Access’19]
(University of Georgia) CAGE April 13 2020 26 / 34

31. Baseline algorithms
Deep Learning approach
Chameleon: The state-of-the-art method for session-based news
recommendation
SR-GNN: Model sessional information by applying graph neural
network.
GRU4Rec: The First RNN-based approach in session-based
recommendation problem.
Association Rules-based approach
Sequential Rules: Considers the order of interactions within the session
Co-Occurrence: Recommend the most read articles together with the
last read articles in other user sessions.
Neighborhood based approach
Item-skNN: Most similar articles to the last read one.
V-kNN: Compares the entire session with past sessions to detemine
items to be recommended.
Non-personalized approach
Recently Popular: Most viewed articles during the last N interactions.
Content-Based: Most similar articles to the last N interactions.
(University of Georgia) CAGE April 13 2020 27 / 34

32. Results in Quality Metrics
(University of Georgia) CAGE April 13 2020 28 / 34

33. Results in Quality Metrics
(University of Georgia) CAGE April 13 2020 28 / 34

34. Discussion
Non-personalized methods are both obtained poor performance without
considering user preferences
(University of Georgia) CAGE April 13 2020 29 / 34

35. Discussion
Neighborhood-based methods outperform non-personalized methods by
taking similar articles into account
(University of Georgia) CAGE April 13 2020 29 / 34

36. Discussion
Methods based on association rules perform better than previous strategies
(University of Georgia) CAGE April 13 2020 29 / 34

37. Discussion
It shows the advantage of using deep models for session-based news
recommendation. As GRU4Rec and SR-GNN are not designed to make
recommendations for first seen items during training, they cannot
outperform Chameleon.
(University of Georgia) CAGE April 13 2020 29 / 34

38. Table of Contents
1 Problem Statement and Example
2 Related Work
3 Our Approach
4 Experiments
5 Conclusions
6 References
(University of Georgia) CAGE April 13 2020 30 / 34

39. Conclusions
Proposed framework
Proposed a context-aware graph embedding framework for session-based
news recommendation by considering structured information and refining
article embeddings with graph neural networks.
Uncover semantic-level structured information
Build an sub knowledge graph for capturing semantic-level information and
take external knowledge resources into account.
Experiments
Extensive experiments on real-word news datasets demonstrated the
effectiveness of the proposed CAGE framework, compared with the
state-of-the-art methods on session-based news recommendation
Submitted Conference
SIGIR’20
(University of Georgia) CAGE April 13 2020 31 / 34

40. Table of Contents
1 Problem Statement and Example
2 Related Work
3 Our Approach
4 Experiments
5 Conclusions
6 References
(University of Georgia) CAGE April 13 2020 32 / 34

41. 1. Mozhgan Karimi, Dietmar Jannach, and Michael Jugovac. “News recommender
systems–Survey and roads ahead”. In: Information Processing & Management 54.6
(2018), pp. 1203–1227
2. Bal´azs Hidasi et al. “Session-based recommendations with recurrent neural networks”.
In: arXiv preprint arXiv:1511.06939 (2015)
3. Yong Kiam Tan, Xinxing Xu, and Yong Liu. “Improved recurrent neural networks for
session-based recommendations”. In: Proceedings of the 1st Workshop on Deep
Learning for Recommender Systems. 2016, pp. 17–22
4. Qiao Liu et al. “STAMP: short-term attention/memory priority model for
session-based recommendation”. In: Proceedings of the 24th ACM SIGKDD
International Conference on Knowledge Discovery & Data Mining. 2018, pp. 1831–1839
5. Shu Wu et al. “Session-based recommendation with graph neural networks”. In:
Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019,
pp. 346–353
6. Gabriel de Souza Pereira Moreira, Felipe Ferreira, and Adilson Marques da Cunha.
“News session-based recommendations using deep neural networks”. In: The 3rd
Workshop on Deep Learning for Recommender Systems. 2018, pp. 15–23
7. Gabriel de Souza Pereira Moreira, Dietmar Jannach, and Adilson Marques da Cunha.
“Contextual hybrid session-based news recommendation with recurrent neural
networks”. In: arXiv preprint arXiv:1904.10367 (2019)
8. Hongwei Wang et al. “DKN: Deep Knowledge-Aware Network for News
Recommendation”. In: Proceedings of the 2018 World Wide Web Conference. WWW
’18. Lyon, France: International World Wide Web Conferences Steering Committee,
2018, 1835–1844. isbn: 9781450356398
(University of Georgia) CAGE April 13 2020 33 / 34

42. Thanks for your attendance
(University of Georgia) CAGE April 13 2020 34 / 34