The document summarizes a research paper that compares the performance of MLP-based models to Transformer-based models on various natural language processing and computer vision tasks. The key points are:
1. Gated MLP (gMLP) architectures can achieve performance comparable to Transformers on most tasks, demonstrating that attention mechanisms may not be strictly necessary.
2. However, attention still provides benefits for some NLP tasks, as models combining gMLP and attention outperformed pure gMLP models on certain benchmarks.
3. For computer vision, gMLP achieved results close to Vision Transformers and CNNs on image classification, indicating gMLP can match their data efficiency.
AAAI2023「Are Transformers Effective for Time Series Forecasting?」と、HuggingFace「Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)」の紹介です。
本スライドは、弊社の梅本により弊社内の技術勉強会で使用されたものです。
近年注目を集めるアーキテクチャーである「Transformer」の解説スライドとなっております。
"Arithmer Seminar" is weekly held, where professionals from within and outside our company give lectures on their respective expertise.
The slides are made by the lecturer from outside our company, and shared here with his/her permission.
Arithmer株式会社は東京大学大学院数理科学研究科発の数学の会社です。私達は現代数学を応用して、様々な分野のソリューションに、新しい高度AIシステムを導入しています。AIをいかに上手に使って仕事を効率化するか、そして人々の役に立つ結果を生み出すのか、それを考えるのが私たちの仕事です。
Arithmer began at the University of Tokyo Graduate School of Mathematical Sciences. Today, our research of modern mathematics and AI systems has the capability of providing solutions when dealing with tough complex issues. At Arithmer we believe it is our job to realize the functions of AI through improving work efficiency and producing more useful results for society.
The document summarizes a research paper that compares the performance of MLP-based models to Transformer-based models on various natural language processing and computer vision tasks. The key points are:
1. Gated MLP (gMLP) architectures can achieve performance comparable to Transformers on most tasks, demonstrating that attention mechanisms may not be strictly necessary.
2. However, attention still provides benefits for some NLP tasks, as models combining gMLP and attention outperformed pure gMLP models on certain benchmarks.
3. For computer vision, gMLP achieved results close to Vision Transformers and CNNs on image classification, indicating gMLP can match their data efficiency.
AAAI2023「Are Transformers Effective for Time Series Forecasting?」と、HuggingFace「Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)」の紹介です。
本スライドは、弊社の梅本により弊社内の技術勉強会で使用されたものです。
近年注目を集めるアーキテクチャーである「Transformer」の解説スライドとなっております。
"Arithmer Seminar" is weekly held, where professionals from within and outside our company give lectures on their respective expertise.
The slides are made by the lecturer from outside our company, and shared here with his/her permission.
Arithmer株式会社は東京大学大学院数理科学研究科発の数学の会社です。私達は現代数学を応用して、様々な分野のソリューションに、新しい高度AIシステムを導入しています。AIをいかに上手に使って仕事を効率化するか、そして人々の役に立つ結果を生み出すのか、それを考えるのが私たちの仕事です。
Arithmer began at the University of Tokyo Graduate School of Mathematical Sciences. Today, our research of modern mathematics and AI systems has the capability of providing solutions when dealing with tough complex issues. At Arithmer we believe it is our job to realize the functions of AI through improving work efficiency and producing more useful results for society.
This document discusses generative adversarial networks (GANs) and their relationship to reinforcement learning. It begins with an introduction to GANs, explaining how they can generate images without explicitly defining a probability distribution by using an adversarial training process. The second half discusses how GANs are related to actor-critic models and inverse reinforcement learning in reinforcement learning. It explains how GANs can be viewed as training a generator to fool a discriminator, similar to how policies are trained in reinforcement learning.
This document discusses generative adversarial networks (GANs) and their relationship to reinforcement learning. It begins with an introduction to GANs, explaining how they can generate images without explicitly defining a probability distribution by using an adversarial training process. The second half discusses how GANs are related to actor-critic models and inverse reinforcement learning in reinforcement learning. It explains how GANs can be viewed as training a generator to fool a discriminator, similar to how policies are trained in reinforcement learning.
CluBERT: A Cluster-Based Approach for Learning Sense Distributions in Multipl...禎晃 山崎
CluBERT: A Cluster-Based Approach for Learning Sense Distributions in Multiple Languages
Word Sense Disambiguation, BERT, clustering
ということで読みました.
p. 7 は「solid は glass の上位語,glassware は glass の下位語」でした。。。
Robust Neural Machine Translation with Doubly Adversarial InputsSho Takase
This document discusses using adversarial training methods to improve neural machine translation. Specifically, it explores training a Transformer model with both original inputs and their "doubly adversarial" corrupted versions to learn robust representations. Evaluation on two translation tasks showed this approach improved BLEU scores over the baseline Transformer model, demonstrating the effectiveness of adversarial training for more robust neural machine translation.
Harnessing Deep Neural Networks with Logic RulesSho Takase
This document summarizes a method for harnessing deep neural networks with logic rules. The goal is to incorporate general rules and human intuitions into neural networks. Rules are expressed using first-order predicate logic and incorporated into training as constraints. The method alternates between calculating the model distribution subject to constraints (q(y|x)) and updating the model parameters (θ). Experiments on sentiment analysis and named entity recognition show the approach improves performance by enforcing linguistic rules during training.
Generating Automatic Feedback on UI Mockups with Large Language Models
Retrofitting Word Vectors to Semantic Lexicons
1. Retrofitting Word Vectors to
Semantic Lexicons
Manaal Faruqui, Jese Dodge, Sujay K. Jauhar,
Chris Dyer, Eduard Hovy, Noah A. Smith
NACL 2015
読む人:高瀬翔
知識獲得研究会2015/4/21
1
5. 提案手法
• やりたいことは2つ
– コーパスから得たベクトル(入力)と似たベクトルとする
– 外部知識上で関連する単語は似たベクトルとする
• 関連:同義語,上位下位語,言い換え
• 目的関数
– 似せたいベクトル間のユークリッド距離を最小化
• 一項目:コーパスの情報(入力ベクトルに近づける)
• 二項目:外部知識(外部知識上での関連語に近づける)
– E:外部知識上で関連している単語間に張ったエッジの集合
– α,β:ハイパーパラメータ(α=1,β=1 / エッジの次数)
5
en related words
inferred (white)
method works
ord vector mod-
tors to beretrofitted (and correspond to V⌦); shaded
nodes are labeled with the corresponding vectors in
ˆQ, which areobserved. Thegraph can beinterpreted
as a Markov random field (Kindermann and Snell,
1980).
The distance between a pair of vectors is defined
to be the Euclidean distance. Since we want the
inferred word vector to be close to the observed
value ˆqi and close to its neighbors qj , 8j such that
(i, j ) 2 E, theobjectiveto beminimized becomes:
(Q) =
nX
i= 1
2
4↵i kqi − ˆqi k2
+
X
(i,j )2E
βij kqi − qj k2
3
5
where ↵ and β values control the relative strengths
of associations (moredetails in §6.1).
コーパスから得たベクトル(入力)
改良後のベクトル
6. 解き方
• 反復更新で解を求める
– 各 qi について,目的関数を最小化する値への更
新を繰り返す
– qi は入力ベクトルで初期化
• 経験的には10回の反復で近づけたいベクトル
間のユークリッド距離は0.01未満になる
6
orma-
o mul-
gives
valua-
engths
ect of
fitting
com/
s
heset
desse-
resent
ex for
V ⇥ V
lution can be found by solving a system of linear
equations. To do so, we use an efficient iterative
updating method (Bengio et al., 2006; Subramanya
et al., 2010; Das and Petrov, 2011; Das and Smith,
2011). The vectors in Q are initialized to be equal
to thevectorsin ˆQ. Wetakethefirst derivativeof
with respect to one qi vector, and by equating it to
zero arriveat thefollowing onlineupdate:
qi =
P
j :(i,j )2E βij qj + ↵i ˆqi
P
j :(i,j )2E βij + ↵i
(1)
In practice, running this procedure for 10 iterations
converges to changes in Euclidean distance of ad-
jacent vertices of less than 10− 2. The retrofitting
approach described above is modular; it can be ap-
plied to word vector representations obtained from
更新式: