SlideShare a Scribd company logo
NEXUS Network:
Connecting the Preceding
and the Following in
Dialogue Generation
Xiaoyu Shen, Hui Su, Wenjie Li and Dietrich Klakow
紹介者:M2 尾形朋哉
概要
• 短く一般的な応答を好む問題
• 良い応答は前の発話と次の発話が滑らかに接続
• 相互情報量を通して接続を強める
• 補助連続符号空間cの導入
• 生成目的の学習事前分布へ投射
(cの導入)のモチベーション
𝐻𝑖−1:対話履歴(𝑢1, 𝑢2, … , 𝑢𝑖−1)
𝐹𝑖+1:未来の会話(𝑢𝑖+1, … , 𝑢 𝑇)
𝑢𝑖:𝑖番目の発話
相互情報量を最大化するような𝑝 𝜃を見つける
→ uを連続符号空間cで置き換える
問題:相互情報量の個々の系列からの逆伝搬での学習は難しい
Gumbel-Softmaxやsemantic-hashing、ベクトル量子化は勾配推定にバイアスがかかる
モンテカルロシミュレーションはバイアスはかからないが、高い分散がある
不連続なのでreparameterization trickは直接使えない
連続符号空間
𝐻𝑖−1
~
と𝐹𝑖+1
~
はcが与えられた時のガウス分布と仮定し、平均と共分散を予測
𝑝 𝜙 𝑐 𝐻𝑖−1, 𝐹𝑖 →相互情報量 高
𝜆1 𝐼 𝐻𝑖−1, 𝑐 + 𝜆2 𝐼(𝑐, 𝐹𝑖+1)の下限
𝐻𝑖−1
~
:階層的エンコーダ𝐸𝑓によるベクトル
𝐹𝑖
~
:階層的エンコーダ𝐸 𝑏によるベクトル
連続空間からのデコード
毎ステップで𝐻𝑖−1
~
と連続空間cを結合して追加で入力
𝐼 𝑢𝑖, 𝑐 𝐻𝑖−1 の下限
𝑐は𝑢𝑖について可能な限り多くの情報を保持するよう学習
→ cの次元をエンコードされたベクトルより小さくする
cの異なる次元で履歴、現在、未来の発話の情報を保持して、
それぞれの依存関係を共有しない可能性
未知の未来に対する事前分布
テスト時には未来の発話は使えない→ 𝑝 𝜃(𝑐|𝐻𝑖−1)を学習する
全ての起こりうる未来の発話を足し合わせなくてはいけないので扱いづらい
KL情報量を近づけ
実験
• データセット
• DailyDialog
• 13,118対話
• 高品質でノイズが少ないが、分量も少ない
• Twitter Corpus
• 750,000対話
• 分量は多いが、ノイズも多い
• 語彙:20,000
• 符号空間c:100
モデル
• Seq2Seq :greadyかbeam search (beam=5)
• MMI :200候補サンプル&リランク
• VHRED : CVAEを用いた対話モデル
• RL : 3つの報酬関数による強化学習
• NEXUS-H : 履歴のみを用いる
• NEXUS-F : 未来のみを用いる
• NEXUS : 履歴と未来両方を用いる
評価
• Embedding score:
• 正解と生成の応答のベクトルのcos類似度
• BLEU Score
• 前のつながり
• AdverSuc:一貫性を敵対的に学習したDiscriminator
• Neg-PMI:−log𝑝(𝑐|𝑟)/𝑝(𝑐)
• 後のつながり
• #Turns:一般的な応答を出力するまでのターン
• Distinct-1, Distinct-2:n-gramの異なり数の割合
• Human Evaluation
• Pri : 前の文脈と一貫するかどうか
• Post: 対話を続けて面白いかどうか
• Flu : 流暢かどうか
Embedding score
BLEU
つながりの評価&人手評価
RLは一貫性を高くするように報酬関数を設定しているのに弱い
priとpostはcoherence(AdverSus, Neg-PMI)と#turnsと相関がある
出力例
VHREDは割と多様な応答を生成できているが、次につながるような応答ではない
結論
• 対話の前後の文脈への接続の役割を強める目的
• 提案手法は他のモデルより高品質な応答
• 学習しやすく、特別なトリックを必要としない

More Related Content

Featured

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
Marius Sescu
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
Expeed Software
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
Pixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
marketingartwork
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
Skeleton Technologies
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
SpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Lily Ray
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
Rajiv Jayarajah, MAppComm, ACC
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
Christy Abraham Joy
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
Vit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
MindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
RachelPearson36
 

Featured (20)

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 

Nexus network connecting the preceding and the following in dialogue generation

Editor's Notes

  1. a diagonal covariance matrix conditioning on the whole dialogue 密ベクトルの推論は並列化可能で、計算量的に安価 系列ベクトルは個々のトークンより全体の意味の類似性を捉える 事後崩落問題を避ける