EDGEFORMERS: GRAPH-EMPOWERED TRANSFORMERS FOR REPRESENTATION LEARNING ON TEXTUAL EDGE NETWORKS.pptx

•Download as PPTX, PDF•

0 likes•111 views

ssuser2624f71

EDGEFORMERS: GRAPH-EMPOWERED TRANSFORMERS FOR REPRESENTATION LEARNING ON TEXTUAL EDGE NETWORKS

Education

Van Thuy Hoang
Dept. of Artificial Intelligence,
The Catholic University of Korea
hoangvanthuy90@gmail.com
Jin et al., ICLR 2023

2
INTRODUCTION
 Networks are ubiquitous and are widely used to model interrelated
data in the real world, such as user-user and user-item interactions
on social media
 There is rich information associated with edges in a network.
 For example:
 When a person replies to another on social media, there will be a
directed edge between them accompanied by the response texts
 When a user comments on an item, the user’s review will be
naturally associated with the user-item edge.
 Edgeformers, that leverage graph-enhanced Transformers to model
edge texts in a contextualized way

3
Contributions
 Identify the importance of modeling text information on network
edges and formulate the problem of representation learning on
textual-edge networks.
 Edgeformers (i.e., Edgeformer-E and Edgeformer-N), two graph
enhanced Transformer architectures, to deeply couple network and
text information in a contextualized way for edge and node
representation learning.

4
TEXTUAL-EDGE NETWORKS
 Textual-edge network

5
TRANSFORMER
 Each Transformer layer utilizes a multi-head self-attention mechanism
to obtain a contextualized representation of each text token:

6
PROBLEM DEFINITIONS
 The first task is edge classification
 The second task is link prediction

7
PROPOSED METHOD
 Edgeformer-E for edge representation learning
 Edgeformer-N for node representation learning

8
EDGE REPRESENTATION LEARNING (EDGEFORMER-E)
 Introduce virtual node tokens:
 To let text token representations carry node signals, we adopt a
multi-head attention mechanism:
 Representation of Virtual Node Tokens.

9
TEXT-AWARE NODE REPRESENTATION LEARNING (EDGEFORMER-N)
 Aggregating Edge Representations:
Enhancing Edge Representations with the Node’s Local
Network Structure

10
TRAINING
 Edge Classification: For Edgeformer-E (i.e., edge representation
learning), adopt supervised training, the objective function of which
is as follows
 Link Prediction: For Edgeformer-N (i.e., node representation learning),
conduct unsupervised training, where the objective function is as
follows

11
EXPERIMENTS
 Baselines
 a bag-of-words method (TF-IDF)
 a pretrained language model BERT
 Datasets
 Amazon (i.e., 1-star, ..., 5-star)
 Goodreads (i.e., 0-star, ..., 5-star).

12
Link prediction performance
 Link prediction performance (on the testing set) on Amazon-Movie,
Amazon-Apps, Goodreads-Crime, Goodreads-Children, and
StackOverflow

13
TASKS FOR NODE REPRESENTATION LEARNING
Node classification performance on Amazon-Movie and Amazon-App.

14
CONCLUSIONS
 Tackle the problem of representation learning on textual-edge
networks
 Integrates local network structure information into each Transformer
layer text encoding for edge representation learning and aggregates
edge representation fused by network and text signals for node
representation

Similar to EDGEFORMERS: GRAPH-EMPOWERED TRANSFORMERS FOR REPRESENTATION LEARNING ON TEXTUAL EDGE NETWORKS.pptx

Computing with Directed Labeled Graphs

Marko Rodriguez

NS-CUK Seminar: V.T.Hoang, Review on "Relphormer: Relational Graph Transforme...

ssuser4b1f48

Lately, deep learning has improved the algorithms and the architectures of several natural language processing (NLP) tasks. In spite of that, the performance of any deep learning model is widely impacted by the used optimizer algorithm; which allows updating the model parameters, finding the optimal weights, and minimizing the value of the loss function. Thus, this paper proposes a new convolutional neural network (CNN) architecture for text classification (TC) and sentiment analysis and uses it with various optimizer algorithms in the literature. Actually, in NLP, and particularly for sentiment classification concerns, the need for more empirical experiments increases the probability of selecting the pertinent optimizer. Hence, we have evaluated various optimizers on three types of text review datasets: small, medium, and large. Thereby, we examined the optimizers regarding the data amount and we have implemented our CNN model on three different sentiment analysis datasets so as to binary label text reviews. The experimental results illustrate that the adaptive optimization algorithms Adam and root mean square propagation (RMSprop) have surpassed the other optimizers. Moreover, our best CNN model which employed the RMSprop optimizer has achieved 90.48% accuracy and surpassed the state-of-the-art CNN models for binary sentiment classification problems.

Optimizer algorithms and convolutional neural networks for text classification

IAESIJAI

A technical paper presentation on Evaluation of Deep Learning techniques in S...

VarshaR19

In image scene, text contains high-level of important information that helps to analyze and consider the particular environment. In this paper, we adapt image mask and original identification of the mask region based convolutional neural networks (R-CNN) to allow recognition at 3 levels such as sequence, holistic and pixel-level semantics. Particularly, pixel and holistic level semantics can be utilized to recognize the texts and define the text shapes, respectively. Precisely, in mask and detection, we segment and recognize both character and word instances. Furthermore, we implement text detection through the outcome of instance segmentation on 2-D feature-space. Also, to tackle and identify the text issues of smaller and blurry texts, we consider text recognition by attention-based of optical character recognition (OCR) model with the mask R-CNN at sequential level. The OCR module is used to estimate character sequence through feature maps of the word instances in sequence to sequence. Finally, we proposed a fine-grained learning technique that trains a more accurate and robust model by learning models from the annotated datasets at the word level. Our proposed approach is evaluated on popular benchmark dataset ICDAR 2013 and ICDAR 2015.

Customized mask region based convolutional neural networks for un-uniformed ...

IJECEIAES

Assigning the submitted text to one of the predetermined categories is required when dealing with application-oriented texts. There are many different approaches to solving this problem, including using neural network algorithms. This article explores using neural networks to sort news articles based on their category. Two word vectorization algorithms are being used — The Bag of Words (BOW) and the word2vec distributive semantic model. For this work the BOW model was applied to the FNN, whereas the word2vec model was applied to CNN. We have measured the accuracy of the classification when applying these methods for ad texts datasets. The experimental results have shown that both of the models show us quite the comparable accuracy. However, the word2vec encoding used for CNN showed more relevant results, regarding to the texts semantics. Moreover, the trained CNN, based on the word2vec architecture, has produced a compact feature map on its last convolutional layer, which can then be used in the future text representation. I.e. Using CNN as a text encoder and for learning transfer.

TEXTS CLASSIFICATION WITH THE USAGE OF NEURAL NETWORK BASED ON THE WORD2VEC’S...

ijsc

Texts Classification with the usage of Neural Network based on the Word2vec’s...

ijsc

Big Data Intelligence: from Correlation Discovery to Causal Reasoning

Wanjin Yu

Understanding Large Social Networks | IRE Major Project | Team 57

Raj Patel

Most of the Arabic Named Entity Recognition (NER) systems depend massively on external resources and handmade feature engineering to achieve state-of-the-art results. To overcome such limitations, we proposed, in this paper, to use deep learning approach to tackle the Arabic NER task. We introduced a neural network architecture based on bidirectional Long Short-Term Memory (LSTM) and Conditional Random Fields (CRF) and experimented with various commonly used hyperparameters to assess their effect on the overall performance of our system. Our model gets two sources of information about words as input: pre-trained word embeddings and character-based representations and eliminated the need for any task-specific knowledge or feature engineering. We obtained state-of-the-art result on the standard ANERcorp corpus with an F1 score of 90.6%.

Arabic named entity recognition using deep learning approach

IJECEIAES

Nowadays irony appears to be pervasive in all social media discussion forums and chats, offering further obstacles to sentiment analysis efforts. The aim of the present research work is to detect irony and its types in English tweets We employed a new system for irony detection in English tweets, and we propose a distilled bidirectional encoder representations from transformers (DistilBERT) light transformer model based on the bidirectional encoder representations from transformers (BERT) architecture, this is further strengthened by the use and design of bidirectional long-short term memory (Bi-LSTM) network this configuration minimizes data preprocessing tasks proposed model tests on a SemEval2018 task 3, 3,834 samples were provided. Experiment results show the proposed system has achieved a precision of 81% for not irony class and 66% for irony class, recall of 77% for not irony and 72% for irony, and F1 score of 79% for not irony and 69% for irony class since researchers have come up with a binary classification model, in this study we have extended our work for multiclass classification of irony. It is significant and will serve as a foundation for future research on different types of irony in tweets.

Fine grained irony classification through transfer learning approach

CSITiaesprime

Deep Learning in Text Recognition and Text Detection : A Review

IRJET Journal

Feature Extraction and Analysis of Natural Language Processing for Deep Learn...

Sharmila Sathish

Unit 6 Uncertainty.pptx

DrYogeshDeshmukh1

IRJET - Visual Question Answering – Implementation using Keras

IRJET Journal

IMAGE TO TEXT TO SPEECH CONVERSION USING MACHINE LEARNING

IRJET Journal

Matching networks for one shot learning

Kazuki Fujikawa

Automatic natural description generation of an image is currently a challenging task. To generate a natural language description of the image, the system is implemented by combining with the techniques of computer vision and natural language processing. This paper presents different deep learning models for generating the natural language description of the image. Moreover, we discussed how the deep learning model, which works for the natural language description of an image, can be implemented. This deep learning model consists of Convolutional Neural Network CNN as well as Recurrent Neural Network RNN . The CNN is used for extracting the features from the image and RNN is used for generating the natural language description. To implement the deep learning model in generating the natural language description of an image, we have applied the Flickr 8K dataset and we have also evaluated the performance of the model using the standard evaluation matrices. These experiments show that the model is frequently giving accurate natural language descriptions for an input image. Phyu Phyu Khaing | Mie Mie Aung | Myint San "Natural Language Description Generation for Image using Deep Learning Architecture" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-5 , August 2019, URL: https://www.ijtsrd.com/papers/ijtsrd26708.pdfPaper URL: https://www.ijtsrd.com/computer-science/other/26708/natural-language-description-generation-for-image-using-deep-learning-architecture/phyu-phyu-khaing

Natural Language Description Generation for Image using Deep Learning Archite...

ijtsrd

A Review on Natural Scene Text Understanding for Computer Vision using Machin...

IRJET Journal

SNAwithNeo4j

Sadhana Singh

Similar to EDGEFORMERS: GRAPH-EMPOWERED TRANSFORMERS FOR REPRESENTATION LEARNING ON TEXTUAL EDGE NETWORKS.pptx (20)