BERT応用勉強会の資料です。 https://nlpaper-challenge.connpass.com/event/174957/

1. BERTをブラウザで動かしたい！
―MobileBERTとTensorFlow.js―
本⽥ 志温
@shion_honda

2. 発表内容
• TensorFlow.jsのMobileBERT Q&A モデルの紹介
• デモ
• 応⽤例（プロトタイプ）の紹介
• MobileBERTとは
• TensorFlow.js Q&Aモデル
2

3. ⾃⼰紹介
3
• 本⽥ 志温（ほんだ しおん）
• データサイエンティストっぽい仕事@リクルートライフスタイル
• 3⽉までは東京⼤学⼤学院にてAI創薬の研究
• これまでも[cv/nl]paper.challengeで何度か発表してきました
• ブログ: https://hippocampus-garden.com/

4. MobileBERT Q&A モデルのデモ
https://muppet.fandom.com/wiki/Bert
4
Bert is Ernie's best friend and roommate on Sesame Street. The pair share the basement apartment at 123 Sesame Street. In
contrast to the practical-joking, extroverted Ernie, Bert is serious, studious, and tries to make sense of his friend's actions.
His own passions include reading Boring Stories, collecting paper clips and bottle caps (especially the rare Figgy Fizz),
consuming oatmeal, and studying pigeons. While Ernie's best companion outside of Bert is Rubber Duckie, Bert has his pet
pigeon, Bernice. True to his detail-oriented, organized mind, Bert is president of the National Association of W Lovers.
Unfortunately for Bert, his nose has come off of his face on a number of occasions. He has a distinctive, bleating laugh.
While Ernie has appeared in many of scenes without Bert, Bert has not been seen without Ernie quite so often. Notable
exceptions include the songs "Doin' the Pigeon," "The Subway," "Bert's Blanket," and "Stick Out Your Hand and Say Hello"
with Rick Moranis, as well as occasional Muppet/Kid scenes, such as discussing emotions with John John.
Some of Bert's memorable moments include riding with Ernie in a plane in Follow That Bird, and interrupting The
Adventures of Elmo in Grouchland whenever it looked like the movie was becoming unsettling.
Bert's extended family includes his identical twin brother salesman Bart, infant nephew Brad, and several unseen relatives,
including his Aunt Matilda, Uncle Bart (who gave Bert his first paper clip), and great-great-grandfather Mountain Mike. The
album Sesame Street Sing-Along! also refers to Bert's Uncle Louie, who lives on a farm.
Bert and Ernie were the only Muppets to appear in the Sesame Street Pilot Episodes, and they also hosted This Way to
Sesame Street, a special sneak preview special promoting Sesame Street.
Passage (289 words)

5. Question #1
5
Question: Where does Bert live?

6. Answer #1: 正解
https://muppet.fandom.com/wiki/Bert
6
Bert is Ernie's best friend and roommate on Sesame Street. The pair share the basement apartment at 123 Sesame Street. In
contrast to the practical-joking, extroverted Ernie, Bert is serious, studious, and tries to make sense of his friend's actions.
His own passions include reading Boring Stories, collecting paper clips and bottle caps (especially the rare Figgy Fizz),
consuming oatmeal, and studying pigeons. While Ernie's best companion outside of Bert is Rubber Duckie, Bert has his pet
pigeon, Bernice. True to his detail-oriented, organized mind, Bert is president of the National Association of W Lovers.
Unfortunately for Bert, his nose has come off of his face on a number of occasions. He has a distinctive, bleating laugh.
While Ernie has appeared in many of scenes without Bert, Bert has not been seen without Ernie quite so often. Notable
exceptions include the songs "Doin' the Pigeon," "The Subway," "Bert's Blanket," and "Stick Out Your Hand and Say Hello"
with Rick Moranis, as well as occasional Muppet/Kid scenes, such as discussing emotions with John John.
Some of Bert's memorable moments include riding with Ernie in a plane in Follow That Bird, and interrupting The
Adventures of Elmo in Grouchland whenever it looked like the movie was becoming unsettling.
Bert's extended family includes his identical twin brother salesman Bart, infant nephew Brad, and several unseen relatives,
including his Aunt Matilda, Uncle Bart (who gave Bert his first paper clip), and great-great-grandfather Mountain Mike. The
album Sesame Street Sing-Along! also refers to Bert's Uncle Louie, who lives on a farm.
Bert and Ernie were the only Muppets to appear in the Sesame Street Pilot Episodes, and they also hosted This Way to
Sesame Street, a special sneak preview special promoting Sesame Street.
Answer: 123 Sesame Street

7. Question #2
7
Question: What does Bert like to collect?

8. Answer #2: 正解
https://muppet.fandom.com/wiki/Bert
8
Bert is Ernie's best friend and roommate on Sesame Street. The pair share the basement apartment at 123 Sesame Street. In
contrast to the practical-joking, extroverted Ernie, Bert is serious, studious, and tries to make sense of his friend's actions.
His own passions include reading Boring Stories, collecting paper clips and bottle caps (especially the rare Figgy Fizz),
consuming oatmeal, and studying pigeons. While Ernie's best companion outside of Bert is Rubber Duckie, Bert has his pet
pigeon, Bernice. True to his detail-oriented, organized mind, Bert is president of the National Association of W Lovers.
Unfortunately for Bert, his nose has come off of his face on a number of occasions. He has a distinctive, bleating laugh.
While Ernie has appeared in many of scenes without Bert, Bert has not been seen without Ernie quite so often. Notable
exceptions include the songs "Doin' the Pigeon," "The Subway," "Bert's Blanket," and "Stick Out Your Hand and Say Hello"
with Rick Moranis, as well as occasional Muppet/Kid scenes, such as discussing emotions with John John.
Some of Bert's memorable moments include riding with Ernie in a plane in Follow That Bird, and interrupting The
Adventures of Elmo in Grouchland whenever it looked like the movie was becoming unsettling.
Bert's extended family includes his identical twin brother salesman Bart, infant nephew Brad, and several unseen relatives,
including his Aunt Matilda, Uncle Bart (who gave Bert his first paper clip), and great-great-grandfather Mountain Mike. The
album Sesame Street Sing-Along! also refers to Bert's Uncle Louie, who lives on a farm.
Bert and Ernie were the only Muppets to appear in the Sesame Street Pilot Episodes, and they also hosted This Way to
Sesame Street, a special sneak preview special promoting Sesame Street.
Answer: paper clips and bottle caps

9. Question #3
9
Question: Who is the best friend of Ernie?

10. Answer #3: 不正解
https://muppet.fandom.com/wiki/Bert
10
Bert is Ernie's best friend and roommate on Sesame Street. The pair share the basement apartment at 123 Sesame Street. In
contrast to the practical-joking, extroverted Ernie, Bert is serious, studious, and tries to make sense of his friend's actions.
His own passions include reading Boring Stories, collecting paper clips and bottle caps (especially the rare Figgy Fizz),
consuming oatmeal, and studying pigeons. While Ernie's best companion outside of Bert is Rubber Duckie, Bert has his pet
pigeon, Bernice. True to his detail-oriented, organized mind, Bert is president of the National Association of W Lovers.
Unfortunately for Bert, his nose has come off of his face on a number of occasions. He has a distinctive, bleating laugh.
While Ernie has appeared in many of scenes without Bert, Bert has not been seen without Ernie quite so often. Notable
exceptions include the songs "Doin' the Pigeon," "The Subway," "Bert's Blanket," and "Stick Out Your Hand and Say Hello"
with Rick Moranis, as well as occasional Muppet/Kid scenes, such as discussing emotions with John John.
Some of Bert's memorable moments include riding with Ernie in a plane in Follow That Bird, and interrupting The
Adventures of Elmo in Grouchland whenever it looked like the movie was becoming unsettling.
Bert's extended family includes his identical twin brother salesman Bart, infant nephew Brad, and several unseen relatives,
including his Aunt Matilda, Uncle Bart (who gave Bert his first paper clip), and great-great-grandfather Mountain Mike. The
album Sesame Street Sing-Along! also refers to Bert's Uncle Louie, who lives on a farm.
Bert and Ernie were the only Muppets to appear in the Sesame Street Pilot Episodes, and they also hosted This Way to
Sesame Street, a special sneak preview special promoting Sesame Street.
Answer: (No Answer) *SQuADにはNo Answerの問題がある

11. MobileBERT Q&A モデル 実⽤例
11
ブラウザのページ内検索（Ctrl+F）に質問を⼊⼒できる
Chrome拡張機能のプロトタイプ.
Google Developers Japan: ブラウザと Tensorflow.js を使った BERT の活⽤⽅法を考える より.

12. 常に上⼿くいくわけではない
12
https://www.bedbathandbeyond.com/store/product/kitchenaid-reg-5-speed-diamond-blender/3242409
質問 What is the pitcher made of?
正答 BPA-free polycarbonate pitcher
BERT Ice mode pulses at staggered intervals to uniformly crush a pitcher of ice in seconds

13. MobileBERTとは
13
モバイル端末向けに, 汎⽤性と精度はそのままに
軽量化・⾼速化を⾏ったBERT
ボトルネック型
（アーキテクチャの⼯夫）
Progressive knowledge transfer
（学習⽅法の⼯夫）
→ 汎化性能と各タスクでの精度を保ちつつ, BERT BASEの4倍⾼速

14. 学習⽅法（蒸留）の⼯夫
14
• ⼩さいモデルをスクラッチで訓練してもうまくいかないため,
BERT Largeからの蒸留を採⽤
• Progressive Knowledge Transfer: 下層から順に⼀層ずつ訓練
• 各層の特徴マップと注意マップを近づけていく
• Pre-training Distilation: PKTの後にMLMの出⼒を合わせるような訓
練も⾏う

15. アーキテクチャの⼯夫
15
BERT Large IB-BERT
(Teacher)
MobileBERT
(Student)
• PKTのために, 教師と⽣徒で各層の⼊出⼒サイズは⼀致させな
ければならない
• 教師は逆ボトルネック型にして, 性能を維持しつつ⼊出⼒サイズ縮⼩
• ⽣徒はボトルネック型にして, ⼊出⼒サイズを保ちつつモデル圧縮

16. 性能評価
16
ハイパラ探索の結果, 以下の設定を採⽤
x0.25
GLUE SQuAD
BERT BASEの4倍⾼速 (367ms)
汎化性能と各タスクの精度も維持 *BERT LARGEには負けている

17. TensorFlow.jsとは
• ブラウザで機械学習を実⾏するためのJavaScriptライブラリ
• GPUはWebGL経由で使える
• TensorFlowやKerasの学習済みモデルをインポートできる
• React Nativeもサポート
17

18. TensorFlow.js Q&Aモデルの使い⽅
18
https://github.com/tensorflow/tfjs-models/tree/master/qna
SQuAD 1.1でファインチューニング済みのBERTを呼ぶだけ

19. おまけ: Q&A以外の学習済みモデル
19
姿勢推定 (PoseNet) 有害テキストの検出
他にも⾳声コマンド認識など
ブラウザで動く機械学習アプリケーションが作れる！
https://www.tensorflow.org/js/models

20. 20
BERTを使ってアプリケーションを
作ってみよう！ｔ
https://muppet.fandom.com/wiki/Bert

21. 参考⽂献など
[1] Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou. “MobileBERT:
a Compact Task-Agnostic BERT for Resource-Limited Devices.” ACL. 2020.
[2] Google Developers Japan: ブラウザと Tensorflow.js を使った BERT の活⽤⽅法を考える
[3] MobileBERT Q&Aモデルのデモ
[4] tfjs-models/qna at master · tensorflow/tfjs-models
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. “BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding.” NACCL. 2019.
[6] アバター: Ultraman Mebius by sousaguilherme5
21