SlideShare a Scribd company logo
1 of 31
Download to read offline
-
- -
:D - BE
0 : - G A A 5 , A 1 A A 2 1C D A 4 A G
. 2 A
C G ) ) ( D ,28 )
• - , 1 , -
)
• m r ) Dei f c
A A O
O D( ) ( D n
•
• O p SQT
• l t
2018/11/25 Journal Club sod 2
ba u SOTA
(state-of-the-art) D
2018/11/25 Journal Club 3
• d
• T
• nb uB
• tsb eB
• T
• i B
• : F g t
• R B i
• ) )2 - f
• (E-
2018/11/25 Journal Club r a 4
I played tennis.
-> <S> I play ed tennis . </S>
• Ei s
: eM
• ( )) )
• n T ( (
l u b L a m
• i d S
: g
2018/11/25 Journal Club of 5
https://kamujun.hatenablog.com/entry/2018/07/06/173931
<S> I play ed tennis . </S>
target
target
n n target r
• I O :
• - - - -
• G : - I
• a A: T
2018/11/25 Journal Club P 6
• h Lg y TL -.
• t S Rf
• d L. / :3
• 3 7 c blRn
• M a r Rm N
• p j sy
• i L
• 233 3/ 7 5 / 3 /0
5 1 : 3 / :3
• / /::/ 57 A0 7 7 A
/ 32 / :3
• /5/:7 / 3 /0 5 1 : 3
2018/11/25 Journal Club uoe 7
2018/11/25 Journal Club 8
• t 3 4: L p
• 13 2 : , 03 T OM Ll
e Rr .
• 13- 1 S r Go r
2 : :G S rL IB G
Rr .
• nsf RP L E a
• P A T
• i
2018/11/26 Journal Club m 9
• 0 0 .0 ,
• T 2 E cr WT
Sg S q P
• 30 0 0 2 1 20.0 ]d
S o se 3
• 01 0 : 0 0 2 1 n
• 2:2 0 0 2 1 kn T
2018/11/26 Journal Club mib[ 10
• TG I
• A
E
•
• O
e M
2018/11/26 Journal Club L 11
I see a little silhouetto of a man
3-gram
[ P , 2016]
• 0 5 0 8 05
• a [eS , - i M
t ] [
•
• K 1 0 : , - M
L% , - de l
kS l i
S l k A t S
u
• S Am S go
s S
2018/11/25 Journal Club nf 12
• % , 2 ,2 , 0 2 ,
• :5
• N : , 5
• :
• : P
• 5
2018/11/25 Journal Club S 13
• - B GL ) 2 M = U
ABAG =A 2 M = P irom t
l O
• L E< B E
• < ( L E< T 0 G
• G A A ,=
• bl dk CL 0 L A E CAE LEA
• p dk SW W
nawW PR
• -,6 ( 48 1,50
( 48T scWeg
2018/11/26 Journal Club uqh 14
• F - + B
• E S
•
2018/11/27 Journal Club TR 15
2018/11/25 Journal Club 16
2018/11/27 Journal Club 17
•
• 8
https://twitter.com/_Ryobot/status/1050925881894400000
MNLI match :
/ /
[Premise] At the other end of Pennsylvania Avenue, people began to line up for a
White House tour.
[Hypothesis] People formed a line at the end of Pennsylvania Avenue.
CoLA :
Books were sent to each other by the
students.
• A
•
• 8B /
2018/11/27 Journal Club E 18
•
2018/11/27 Journal Club 19
The Amazon rainforest (Portuguese: Floresta Amazônica or Amazônia; Spanish:
Selva Amazónica, Amazonía or usually Amazonia; French: Forêt amazonienne;
Dutch: Amazoneregenwoud), also known in English as Amazonia or the Amazon
Jungle, is a moist broadleaf forest that covers most of the Amazon basin of South
America. This basin encompasses 7,000,000 square kilometres (2,700,000 sq mi),
of which 5,500,000 square kilometres (2,100,000 sq mi) are covered by the
rainforest. This region includes territory belonging to nine nations. The majority of
the forest is contained within Brazil, with 60% of the rainforest, followed by Peru
with 13%, Colombia with 10%, and with minor amounts in Venezuela, Ecuador,
Bolivia, Guyana, Suriname and French Guiana. States or departments in four
nations contain "Amazonas" in their names. The Amazon represents over half of
the planet's remaining rainforests, and comprises the largest and most biodiverse
tract of tropical rainforest in the world, with an estimated 390 billion individual
trees divided into 16,000 species.
Which name is also used to describe the Amazon rainforest in English?
•
• ia O a
)
• ( (
nl
2018/11/27 Journal Club e S 20
EM: exact match
ia F1
T , Q m
b A, O
https://ai.googleblog.com/2018/11/open-sourcing-bert-
state-of-art-pre.html
• , / /
/ /
• L E
• A
R S T G
•
2018/11/27 Journal Club O 21
• ,
•
a
•
•
2018/11/27 Journal Club 22
.
A girl is going across a set of monkey bars. She
(a) jumps up across the monkey bars.
(b) struggles onto the bars to grab her head.
(c) gets to the end and stands on a wooden
plank.
(d) jumps up and does a back flip.
• &, & R Q
• S g f
• , B EP 2
Q
• E E h
• S NM
L e hA
D B
T: :
2018/11/25 Journal Club 23
• L
• 05 2 41 aM A ,
• a 05 2 41 A ,
• - E ,
• - - E ,
• S f G M
AEaT BM
i
• R A
B A
2351 51A
2018/11/27 Journal Club e 24
• T
• - ,- N 1
• 1 IR LM
2018/11/27 Journal Club 25
- -
• s B L Lb
C Le
• CM NT S L B
• a r
• uo Li C
• 2
• fd L
• 0 , 3 -,4 B
2018/11/27 Journal Club t 26
2018/11/25 Journal Club 27
• u0 p LN TS
ek MP
• p db TM , ,T-/. l
• c ba fTp ek ): St
oM G
( : 4 4
• ms
• s
• rs rs
• rs W1: : : Pa hbS
W N L H i g rsW G
2018/11/26 Journal Club n 28
• z u K R a
• i e oSTI I
• -/35 ) e K bK
Oc h i s 52/ .
I b q b a
• t z Ib f
km
CC B, C 8 1 B
4E C B (7 (7)88 (:( 7 )
2018/11/26 Journal Club p 29
• BI D TB S
• T A AB E
:: - - : . / /
• R ,
2018/11/27 Journal Club Q 30
• : : :
• : : :
• :
: : :
• A :
: :
2018/11/26 Journal Club ED 31

More Related Content

Similar to BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Event:由 Salesforce 收購 Slack 觀察 SaaS 產業併購趨勢
Event:由 Salesforce 收購 Slack 觀察 SaaS 產業併購趨勢Event:由 Salesforce 收購 Slack 觀察 SaaS 產業併購趨勢
Event:由 Salesforce 收購 Slack 觀察 SaaS 產業併購趨勢
Collaborator
 

Similar to BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (7)

[DL輪読会] Revealing the Dark Secrets of BERT (EMNLP-IJCNLP, 2019)
[DL輪読会] Revealing the Dark Secrets of BERT (EMNLP-IJCNLP, 2019)[DL輪読会] Revealing the Dark Secrets of BERT (EMNLP-IJCNLP, 2019)
[DL輪読会] Revealing the Dark Secrets of BERT (EMNLP-IJCNLP, 2019)
 
[DL輪読会]Quantization and Training of Neural Networks for Efficient Integer-Ari...
[DL輪読会]Quantization and Training of Neural Networks for Efficient Integer-Ari...[DL輪読会]Quantization and Training of Neural Networks for Efficient Integer-Ari...
[DL輪読会]Quantization and Training of Neural Networks for Efficient Integer-Ari...
 
re:Invent 2018 recap for Gaming ~ゲーム関連セッションのご紹介~
re:Invent 2018 recap for Gaming ~ゲーム関連セッションのご紹介~re:Invent 2018 recap for Gaming ~ゲーム関連セッションのご紹介~
re:Invent 2018 recap for Gaming ~ゲーム関連セッションのご紹介~
 
katagaitai CTF workshop #10 AESに対する相関電力解析
katagaitai CTF workshop #10 AESに対する相関電力解析katagaitai CTF workshop #10 AESに対する相関電力解析
katagaitai CTF workshop #10 AESに対する相関電力解析
 
190207 top-k off-policy_correction_for_a_reinforce_recommender_system
190207 top-k off-policy_correction_for_a_reinforce_recommender_system190207 top-k off-policy_correction_for_a_reinforce_recommender_system
190207 top-k off-policy_correction_for_a_reinforce_recommender_system
 
[DL輪読会]Addressing Failure Prediction by Learning Model Confidence
[DL輪読会]Addressing Failure Prediction by Learning Model Confidence[DL輪読会]Addressing Failure Prediction by Learning Model Confidence
[DL輪読会]Addressing Failure Prediction by Learning Model Confidence
 
Event:由 Salesforce 收購 Slack 觀察 SaaS 產業併購趨勢
Event:由 Salesforce 收購 Slack 觀察 SaaS 產業併購趨勢Event:由 Salesforce 收購 Slack 觀察 SaaS 產業併購趨勢
Event:由 Salesforce 收購 Slack 觀察 SaaS 產業併購趨勢
 

More from Shion Honda

More from Shion Honda (11)

BERTをブラウザで動かしたい! ―MobileBERTとTensorFlow.js―
BERTをブラウザで動かしたい!―MobileBERTとTensorFlow.js―BERTをブラウザで動かしたい!―MobileBERTとTensorFlow.js―
BERTをブラウザで動かしたい! ―MobileBERTとTensorFlow.js―
 
Bridging between Vision and Language
Bridging between Vision and LanguageBridging between Vision and Language
Bridging between Vision and Language
 
Graph U-Nets
Graph U-NetsGraph U-Nets
Graph U-Nets
 
Deep Learning Chap. 12: Applications
Deep Learning Chap. 12: ApplicationsDeep Learning Chap. 12: Applications
Deep Learning Chap. 12: Applications
 
Deep Learning Chap. 6: Deep Feedforward Networks
Deep Learning Chap. 6: Deep Feedforward NetworksDeep Learning Chap. 6: Deep Feedforward Networks
Deep Learning Chap. 6: Deep Feedforward Networks
 
画像認識 第9章 さらなる話題
画像認識 第9章 さらなる話題画像認識 第9章 さらなる話題
画像認識 第9章 さらなる話題
 
Towards Predicting Molecular Property by Graph Neural Networks
Towards Predicting Molecular Property by Graph Neural NetworksTowards Predicting Molecular Property by Graph Neural Networks
Towards Predicting Molecular Property by Graph Neural Networks
 
画像認識 6.3-6.6 畳込みニューラル ネットワーク
画像認識 6.3-6.6 畳込みニューラルネットワーク画像認識 6.3-6.6 畳込みニューラルネットワーク
画像認識 6.3-6.6 畳込みニューラル ネットワーク
 
深層学習による自然言語処理 第2章 ニューラルネットの基礎
深層学習による自然言語処理 第2章 ニューラルネットの基礎深層学習による自然言語処理 第2章 ニューラルネットの基礎
深層学習による自然言語処理 第2章 ニューラルネットの基礎
 
IaGo: an Othello AI inspired by AlphaGo
IaGo: an Othello AI inspired by AlphaGoIaGo: an Othello AI inspired by AlphaGo
IaGo: an Othello AI inspired by AlphaGo
 
Planning chemical syntheses with deep neural networks and symbolic AI
Planning chemical syntheses with deep neural networks and symbolic AIPlanning chemical syntheses with deep neural networks and symbolic AI
Planning chemical syntheses with deep neural networks and symbolic AI
 

Recently uploaded

Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
UK Journal
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  • 1. - - - :D - BE 0 : - G A A 5 , A 1 A A 2 1C D A 4 A G . 2 A C G ) ) ( D ,28 )
  • 2. • - , 1 , - ) • m r ) Dei f c A A O O D( ) ( D n • • O p SQT • l t 2018/11/25 Journal Club sod 2 ba u SOTA (state-of-the-art) D
  • 4. • d • T • nb uB • tsb eB • T • i B • : F g t • R B i • ) )2 - f • (E- 2018/11/25 Journal Club r a 4 I played tennis. -> <S> I play ed tennis . </S>
  • 5. • Ei s : eM • ( )) ) • n T ( ( l u b L a m • i d S : g 2018/11/25 Journal Club of 5 https://kamujun.hatenablog.com/entry/2018/07/06/173931 <S> I play ed tennis . </S> target target n n target r
  • 6. • I O : • - - - - • G : - I • a A: T 2018/11/25 Journal Club P 6
  • 7. • h Lg y TL -. • t S Rf • d L. / :3 • 3 7 c blRn • M a r Rm N • p j sy • i L • 233 3/ 7 5 / 3 /0 5 1 : 3 / :3 • / /::/ 57 A0 7 7 A / 32 / :3 • /5/:7 / 3 /0 5 1 : 3 2018/11/25 Journal Club uoe 7
  • 9. • t 3 4: L p • 13 2 : , 03 T OM Ll e Rr . • 13- 1 S r Go r 2 : :G S rL IB G Rr . • nsf RP L E a • P A T • i 2018/11/26 Journal Club m 9
  • 10. • 0 0 .0 , • T 2 E cr WT Sg S q P • 30 0 0 2 1 20.0 ]d S o se 3 • 01 0 : 0 0 2 1 n • 2:2 0 0 2 1 kn T 2018/11/26 Journal Club mib[ 10
  • 11. • TG I • A E • • O e M 2018/11/26 Journal Club L 11 I see a little silhouetto of a man 3-gram [ P , 2016]
  • 12. • 0 5 0 8 05 • a [eS , - i M t ] [ • • K 1 0 : , - M L% , - de l kS l i S l k A t S u • S Am S go s S 2018/11/25 Journal Club nf 12
  • 13. • % , 2 ,2 , 0 2 , • :5 • N : , 5 • : • : P • 5 2018/11/25 Journal Club S 13
  • 14. • - B GL ) 2 M = U ABAG =A 2 M = P irom t l O • L E< B E • < ( L E< T 0 G • G A A ,= • bl dk CL 0 L A E CAE LEA • p dk SW W nawW PR • -,6 ( 48 1,50 ( 48T scWeg 2018/11/26 Journal Club uqh 14
  • 15. • F - + B • E S • 2018/11/27 Journal Club TR 15
  • 17. 2018/11/27 Journal Club 17 • • 8 https://twitter.com/_Ryobot/status/1050925881894400000 MNLI match : / / [Premise] At the other end of Pennsylvania Avenue, people began to line up for a White House tour. [Hypothesis] People formed a line at the end of Pennsylvania Avenue. CoLA : Books were sent to each other by the students.
  • 18. • A • • 8B / 2018/11/27 Journal Club E 18
  • 19. • 2018/11/27 Journal Club 19 The Amazon rainforest (Portuguese: Floresta Amazônica or Amazônia; Spanish: Selva Amazónica, Amazonía or usually Amazonia; French: Forêt amazonienne; Dutch: Amazoneregenwoud), also known in English as Amazonia or the Amazon Jungle, is a moist broadleaf forest that covers most of the Amazon basin of South America. This basin encompasses 7,000,000 square kilometres (2,700,000 sq mi), of which 5,500,000 square kilometres (2,100,000 sq mi) are covered by the rainforest. This region includes territory belonging to nine nations. The majority of the forest is contained within Brazil, with 60% of the rainforest, followed by Peru with 13%, Colombia with 10%, and with minor amounts in Venezuela, Ecuador, Bolivia, Guyana, Suriname and French Guiana. States or departments in four nations contain "Amazonas" in their names. The Amazon represents over half of the planet's remaining rainforests, and comprises the largest and most biodiverse tract of tropical rainforest in the world, with an estimated 390 billion individual trees divided into 16,000 species. Which name is also used to describe the Amazon rainforest in English?
  • 20. • • ia O a ) • ( ( nl 2018/11/27 Journal Club e S 20 EM: exact match ia F1 T , Q m b A, O https://ai.googleblog.com/2018/11/open-sourcing-bert- state-of-art-pre.html
  • 21. • , / / / / • L E • A R S T G • 2018/11/27 Journal Club O 21
  • 22. • , • a • • 2018/11/27 Journal Club 22 . A girl is going across a set of monkey bars. She (a) jumps up across the monkey bars. (b) struggles onto the bars to grab her head. (c) gets to the end and stands on a wooden plank. (d) jumps up and does a back flip.
  • 23. • &, & R Q • S g f • , B EP 2 Q • E E h • S NM L e hA D B T: : 2018/11/25 Journal Club 23
  • 24. • L • 05 2 41 aM A , • a 05 2 41 A , • - E , • - - E , • S f G M AEaT BM i • R A B A 2351 51A 2018/11/27 Journal Club e 24
  • 25. • T • - ,- N 1 • 1 IR LM 2018/11/27 Journal Club 25
  • 26. - - • s B L Lb C Le • CM NT S L B • a r • uo Li C • 2 • fd L • 0 , 3 -,4 B 2018/11/27 Journal Club t 26
  • 28. • u0 p LN TS ek MP • p db TM , ,T-/. l • c ba fTp ek ): St oM G ( : 4 4 • ms • s • rs rs • rs W1: : : Pa hbS W N L H i g rsW G 2018/11/26 Journal Club n 28
  • 29. • z u K R a • i e oSTI I • -/35 ) e K bK Oc h i s 52/ . I b q b a • t z Ib f km CC B, C 8 1 B 4E C B (7 (7)88 (:( 7 ) 2018/11/26 Journal Club p 29
  • 30. • BI D TB S • T A AB E :: - - : . / / • R , 2018/11/27 Journal Club Q 30
  • 31. • : : : • : : : • : : : : • A : : : 2018/11/26 Journal Club ED 31