BERT Explained: What You Need to Know About Google’s New Algorithm

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

BERT Explained: What you
need to know about
Google’s new algorithm
Dawn Anderson
#SEJThinktank
@dawnieando

About Me
#SEJThinktank
@dawnieando

Also…Meet)Bert)and)Tedward
#SEJThinktank
@dawnieando

We’re%talking%about%BERT%in%Search%today
#SEJThinktank
@dawnieando

What%BERT%really%is
#SEJThinktank
@dawnieando

Important:*BERT*is*many*
things
#SEJThinktank
@dawnieando

BERT%is%a%Google%search%
algorithm%‘ingredient’%/%tool%
/%framework%called%‘Google%
BERT’
#SEJThinktank
@dawnieando

BERT%is%also%an%open%
source%research%
project%&%academic%
paper
#SEJThinktank
@dawnieando

Google&
BERT&Paper
• Devlin,&J.,&Chang,&M.W.,&Lee,&K.&and&Toutanova,&K.,&2018.&Bert:&PreD
training&of&deep&bidirectional&t...

BERT%
(Bidirectional%
Encoder%
Representation%
from%
Transformers)
#SEJThinktank
@dawnieando

Probably…)Most)mentions)of)
BERT)online)are)NOT)about)
THE)Google)BERT)update
#SEJThinktank
@dawnieando

BERT%Has%
Dramatically%
Accelerated%NLU%
(natural%language%
understanding)
#SEJThinktank
@dawnieando

Google’s(move(to(open(
source(BERT(has(probably(
changed(natural(language(
processing(forever
#SEJThinktank
@dawnieando

THE$ML$&$NLP$
COMMUNITY$
ARE$VERY$
EXCITED$
ABOUT$BERT
#SEJThinktank
@dawnieando

BERT%has%been%pre.trained%on%a%lot%of%words%…%on%the%whole%of%the%English%
Wikipedia%(2,500%million%words)
#SEJThinktank
@...

VANILLA&BERT&PROVIDES&A&
PRE/TRAINED&STARTING&
POINT&LAYER&FOR&NEURAL&
NETWORKS&IN&MACHINE&
LEARNING&&&NATURAL&
LANGUAGE&D...

EVERYBODY(WANTS(TO(‘BUILD2A2
BERT.((NOW(THERE(ARE(LOADS(OF(
ALGORITHMS(WITH(BERT
#SEJThinktank
@dawnieando

Whilst'BERT'
has'been'pre2
trained'on'
Wikipedia'it'is'
fine2tuned'on'
‘questions'
and'answer'
datasets’

MS#MARCO:#A#
Human#
Generated#
MAchine#Reading#
Comprehension#
Dataset
• Rajpurkar,#P.,#Zhang,#J.,#Lopyrev,#K.#and#Liang,#...

MS#MARCO
#SEJThinktank
@dawnieando

Real%Bing%
Questions%
Feed%MS%
MARCO
From%real%Bing%anonymized%
queries
#SEJThinktank
@dawnieando

Researchers(compete(over(Natural(Language(Understanding(with(e.g.(SQuAD(
(Stanford(Question(&(Answering(Dataset)
#SEJThink...

BERT%now%even%beats%the%human%reasoning%benchmark%on%SQuAD
#SEJThinktank
@dawnieando

Lots%of%the%major%AI%
companies%are%also%
building%BERT%versions
#SEJThinktank
@dawnieando

Microsoft)extends)on)BERT)with)MT4DNN
#SEJThinktank
@dawnieando

RoBERTa'
from'
Facebook
#SEJThinktank
@dawnieando

SuperGLUE Benchmark1was1created1because1GLUE1became1too1easy
#SEJThinktank
@dawnieando

What%challenges%does%
BERT%help%to%solve?
#SEJThinktank
@dawnieando

The$Problem$with$
Words
#SEJThinktank
@dawnieando

#SEJThinktank
@dawnieando

Words&are$problematic.$$
Ambiguous…$polysemous…$
synonymous
#SEJThinktank
@dawnieando

Ambiguity)and)
Polysemy
• Almost)every)other)word)in)the)
English)language)has)multiple)
meanings
#SEJThinktank
@dawnieando

In#spoken#word#it#
is#even#worse#
because#of#
homophones#and#
prosody
#SEJThinktank
@dawnieando

Like%“four%
candles”%and%
“fork%
handles”
#SEJThinktank
@dawnieando

Which%does%not%bode%well%for%conversational%
search%into%the%future
#SEJThinktank
@dawnieando

Word’s'Context
• ”The'meaning'of'a'word'is'its'use'in'a'language”'(Ludwig'
Wittgenstein,'Philosopher,'1953)
• Image'attrib...

Word’s'Context'Changes'
As'A'Sentence'Evolves
• The'meaning'of'a'word'changes'(literally)'as'a'
sentence'develops
• Due'to...

Like%“like”
We%can%see%in%just%this%
short%sentence%alone%using%
Stanford%Part%of%Speech%
Tagger%Online%that%the%
word%lik...

Like%“like”
• For%example:%The%word%”like”%has%
several%possible%parts%of%speech%
(including%‘verb’,%‘noun’,%‘adjective’)
...

Natural'Language'Recognition'is'
NOT'Understanding
• Natural'language'understanding'requires'
understanding'of'context'and...

Natural'language'
understanding'is'NOT'
structured'data
#SEJThinktank
@dawnieando

Structured(data(helps(
to(disambiguate(but(
what(about(the(‘hot(
mess’(in(between?
#SEJThinktank
@dawnieando

AND$NOT$EVERYONE$
OR$THING$IS$
MAPPED$TO$THE$
KNOWLEDGE$GRAPH
#SEJThinktank
@dawnieando

#SEJThinktank
@dawnieando

Ontology(Driven(Natural(Language(Processing
Image&credit:&IBM
#SEJThinktank
@dawnieando

How$can$search$
engines$fill$in$the$
gaps$between$
named$entities?
#SEJThinktank
@dawnieando

Natural'Language'Disambiguation
#SEJThinktank
@dawnieando

Word’s'Company
“You'shall'know'a'word'by'the'company'it'
keeps”'(John'Rupert'Firth,'Linguist,1957)
Image'Attribution:'Wiki...

Words&That&Live&
Together&Are&
Strongly&
Connected
• Co7occurrence
• Co7occurrence&provides&context
• Co7occurrence&change...

Language'models'are'trained'on'very'large'
text'corpora'or'collections'(loads'of'words)'
to'learn'distributional'similarit...

Vector'representations'of'words'(Word'Vectors)

And$build$vector$
space$models$
for$word$
embeddings
king$7 man$+$
woman$=$queen

Models'learn'the'
weights'of'the'
similarity'and'
relatedness'distances
#SEJThinktank
@dawnieando

EVEN$IF$WE$UNDERSTAND$THE$ENTITY$
(THING)$ITSELF$WE$NEED$TO$UNDERSTAND$
WORD’S$CONTEXT
#SEJThinktank
@dawnieando

#SEJThinktank
@dawnieando

They%need%
‘Text%
cohesion’
Cohesion is%the grammatical and%
lexical linking%within%a%text%
or sentence that%holds%a%text%...

Semantic)context)matters
• He)kicked)the)bucket
• I)have)yet)to)cross)that)off)my)bucket)list
• The)bucket)was)filled)with...

An#important#
part#of#this#is#
‘Part#of#
Speech’#(POS)#
tagging
#SEJThinktank
@dawnieando

Chunking(and(Tokenization
#SEJThinktank
@dawnieando

Example(Part(of(Speech(Tagging((POS)
#SEJThinktank
@dawnieando

How$BERT$works
#SEJThinktank
@dawnieando

PAST%LANGUAGE%MODELS%
(E.G.%WORD2VEC%&%
GLOVE2VEC)%BUILT%
CONTEXT:FREE%WORD%
EMBEDDINGS
#SEJThinktank
@dawnieando

BERT%provides%
‘context’
#SEJThinktank
@dawnieando

BERT%has%been%pre.trained%on%a%lot%of%words%…%on%the%whole%of%the%English%
Wikipedia%(2,500%million%words)
#SEJThinktank
@...

B"#>"Bi#directional
#SEJThinktank
@dawnieando

A"Moving"
Word"
‘Context"
Window’
#SEJThinktank
@dawnieando

Example(context(window(size(3
Source'Text Training'
Samples
The quick brown fox jumps over the lazy dog (the,(quick)(
(the...

Previously+Uni.Directional
#SEJThinktank
@dawnieando
Previously+all+language+
models+were+uni.
directional+so+could+only+
...

Most%language%modellers%are%uni0
directionalSource'Text
Writing a list of random sentences is harder than I Initially thou...

BERT%is%different.%%BERT%uses%bi1directional%
language%modelling.%%The%FIRST%to%do%thisSource'Text
Writing a list of rando...

BERT%can%see%the%WHOLE%
sentence%on%either%side%of%a%
word%(contextual%language%
modelling)%and%all%of%the%
words%almost%a...

Did$you$mean$“bank”?
Or$did$you$mean$“bank”?
#SEJThinktank
@dawnieando

ER #> Encoder
Representations
#SEJThinktank
@dawnieando

T"#>"Transformers
#SEJThinktank
@dawnieando

Bert%uses%‘Transformers’%&%
’Masked%Language%Modelling’
#SEJThinktank
@dawnieando

Masked'Language'
Modelling'Stops'
The'Target'Word'
From'Seeing'Itself
#SEJThinktank
@dawnieando

Transformers*
(Attention*
simultaneously)
#SEJThinktank
@dawnieando

Attention'is'all'you'
need
#SEJThinktank
@dawnieando

Type%of%natural%language%
tasks%BERT%helps%with
#SEJThinktank
@dawnieando

Things'
Like:
• Named'entity'determination
• Textual'entailment'(next'sentence'prediction)
• Coreference'resolution
• Ques...

11"NLP"
Tasks
BERT"advanced"the"State"of"the"Art"(SOT)"of"11"
NLP"Tasks
#SEJThinktank
@dawnieando

Polysemy(&(
Homonyms
#SEJThinktank
@dawnieando

Coreference(
resolution
#SEJThinktank
@dawnieando

Pronouns'can'be'
problematic
#SEJThinktank
@dawnieando

Computer)
programs)lose)
track)of)who)is)
who)easily
I’m)confused…)Here…)
Have)some)flowers)instead)
#SEJThinktank
@dawnie...

Anaphora(&(Cataphora
#SEJThinktank
@dawnieando

Named&Entity&
Determination
#SEJThinktank
@dawnieando

Named&Entity&Recognition&is&
NOT&Named&Entity&
Disambiguation
#SEJThinktank
@dawnieando

Named&entities&can&be&
polysemic
#SEJThinktank
@dawnieando

Did$you$mean?
• Amadeus$Mozart$(composer)
• Mozart$Street
• Mozart$Cafe
#SEJThinktank
@dawnieando

AND$VERBALLY…WHO$(WHAT)$ARE$
YOU$TALKING$ABOUT?
”LYNDSEY$DOYLE”$OR$”LINSEED$
OIL”?
#SEJThinktank
@dawnieando

#SEJThinktank
@dawnieando

#SEJThinktank
@dawnieando

Textual(Entailment((Next(
sentence(prediction)
#SEJThinktank
@dawnieando

BERT%can%
identify%which%
sentence%likely%
comes%next%
from%two%
choices
#SEJThinktank
@dawnieando

OFTEN&THE&NEXT&
SENTENCE&REALLY&
MATTERS
#SEJThinktank
@dawnieando

I"Remember"
When"My"
Grandad"
Kicked"The"
Bucket
#SEJThinktank
@dawnieando
BERT"is"able"to"
understand"the"NEXT"
sentence
...

“How%far%do%you%
reckon%I%could%kick%
this%bucket?”
#SEJThinktank
@dawnieando

How$BERT$will$impact$
search?
#SEJThinktank
@dawnieando

BERT%will%help%Google%
to%better%understand%
human%language
#SEJThinktank
@dawnieando

More%able%to%scale%
conversational%search%
(Pygmalion%alternative)
#SEJThinktank
@dawnieando

Expect'big'leaps'
for'international'
SEO
#SEJThinktank
@dawnieando

Google&will&be&better&
able&to&understand&
‘contextual&nuance’&&&
ambiguous&queries
#SEJThinktank
@dawnieando

Should'you'try'(or'can)'
optimize'our'content'
for'BERT?
#SEJThinktank
@dawnieando

PROBABLY(
NOT
#SEJThinktank
@dawnieando

‘Bertology’+– ‘The+
study+of+why+BERT+
does+things+(Hugging+
Face)
#SEJThinktank
@dawnieando

BLACK&BOX&
ALGORITHMS
#SEJThinktank
@dawnieando

Layers'
Everywhere
#SEJThinktank
@dawnieando

BERT%by%name%
but%not%by%nature
#SEJThinktank
@dawnieando

More%efficient%BERT’s%have%
been%developed
• DistilBERT
• ALBERT
• Fast BERT
#SEJThinktank
@dawnieando

ALBERT
#SEJThinktank
@dawnieando
BERT’s)successor)from)
Google
Joint)work)between)
Google)Research)&)Toyota)
Technological...

Algorithmic+
Bias+
Concerns
Ricardo+Baeza5Yates'+work+5 Bias+on+the+Web
NoBIAS+Project
IBM+initiatives+to+prevent+bias
BER...

#SEJThinktank
@dawnieando

Keep$in$Touch
@dawnieando
@BeBertey
#SEJThinktank
@dawnieando

References
• Rajpurkar,.P.,.Zhang,.J.,.Lopyrev,.K..and.Liang,.P.,.2016..Squad:.100,000+.questions.for.machine.comprehensio...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1033581 views
Storytelling For The Web: Integrate... by Chiara Aliotta 945137 views
Artificial Intelligence, Data and C... by OECD Directorate ... 865593 views
How to Leverage AI to Boost Employe... by SocialHRCamp 363144 views
2024 State of Marketing Report – by... by Marius Sescu 206375 views
Everything You Need To Know About C... by Expeed Software 226977 views

View on Slideshare

1 of 115 Ad

View on Slideshare

1 of 115 Ad