You can use GPT-J, GPT-3 or Jurassic-1 to create human-like texts and automate SEO. But not all that glitters is gold...what are the limitations of these language models, and how can you get the best of it? In his speech, SEO Automation using GPT3 and Transformer based language models, he shows you some examples of how to "hack" text-to-text transformer-based models to combine human intuition with artificial intelligence and what returns you can get in terms of traffic for your website. In this presentation I will: - explain the transformer architecture to SEO specialists and marketers; - show the limits of deep autoregressive language models created with this architecture; - provide some tips on how you can use them and manage the conditions.

1. SEO automation
using GPT-3
transformer based
language models
Andrea Volpini | WordLift
SLIDESHARE.NET/CYBERANDY
@CYBERANDY

2. #brightonSEO
@cyberandy
WHAT IS NLP?

3. #brightonSEO
@cyberandy
"Hi, my name is Andrea
Volpini, I am the CEO of
WordLift, you can find
me on Twitter as
@cyberandy and here is a
webpage
https://wordlift.io/blog
/en/entity/andrea-volpin
i"
UNSTRUCTURED

4. #brightonSEO
@cyberandy
STRUCTURED
{"@context":"https://schema.org
",
"@type":"Person",
"name":"Andrea Volpini",
"url":"https://wordlift.io/blog
/en/entity/andrea-volpini",
"sameAs":"https://twitter.com/c
yberandy",
"jobTitle":"CEO",
"worksFor":"https://wordlift.io
/blog/en/organization/wordlift}

5. #brightonSEO
@cyberandy
STRUCTURED
{"@context":"https://schema.org
",
"@type":"Person",
"name":"Andrea Volpini",
"url":"https://wordlift.io/blog
/en/entity/andrea-volpini",
"sameAs":"https://twitter.com/c
yberandy",
"jobTitle":"CEO",
"worksFor":"https://wordlift.io
/blog/en/organization/wordlift}
"Hi, my name is Andrea
Volpini, I am the CEO of
WordLift, you can find
me on Twitter as
@cyberandy and here is a
webpage
https://wordlift.io/blog
/en/entity/andrea-volpin
i"
UNSTRUCTURED

6. #brightonSEO
@cyberandy
STRUCTURED
{"@context":"https://schema.org
",
"@type":"Person",
"name":"Andrea Volpini",
"url":"https://wordlift.io/blog
/en/entity/andrea-volpini",
"sameAs":"https://twitter.com/c
yberandy",
"jobTitle":"CEO",
"worksFor":"https://wordlift.io
/blog/en/organization/wordlift}
"Hi, my name is Andrea
Volpini, I am the CEO of
WordLift, you can find
me on Twitter as
@cyberandy and here is a
webpage
https://wordlift.io/blog
/en/entity/andrea-volpin
i"
UNSTRUCTURED
NLP

7. #brightonSEO
@cyberandy
STRUCTURED
{"@context":"https://schema.org
",
"@type":"Person",
"name":"Andrea Volpini",
"url":"https://wordlift.io/blog
/en/entity/andrea-volpini",
"sameAs":"https://twitter.com/c
yberandy",
"jobTitle":"CEO",
"worksFor":"https://wordlift.io
/blog/en/organization/wordlift}
"Hi, my name is Andrea
Volpini, I am the CEO of
WordLift, you can find
me on Twitter as
@cyberandy and here is a
webpage
https://wordlift.io/blog
/en/entity/andrea-volpin
i"
UNSTRUCTURED
NLP
NLG

8. #brightonSEO
@cyberandy
STRUCTURED
{"@context":"https://schema.org
",
"@type":"Person",
"name":"Andrea Volpini",
"url":"https://wordlift.io/blog
/en/entity/andrea-volpini",
"sameAs":"https://twitter.com/c
yberandy",
"jobTitle":"CEO",
"worksFor":"https://wordlift.io
/blog/en/organization/wordlift}
"Hi, my name is Andrea
Volpini, I am the CEO of
WordLift, you can find
me on Twitter as
@cyberandy and here is a
webpage
https://wordlift.io/blog
/en/entity/andrea-volpin
i"
UNSTRUCTURED
NLP
NLU

9. #brightonSEO
@cyberandy
STRUCTURED
{"@context":"https://schema.org
",
"@type":"Person",
"name":"Andrea Volpini",
"url":"https://wordlift.io/blog
/en/entity/andrea-volpini",
"sameAs":"https://twitter.com/c
yberandy",
"jobTitle":"CEO",
"worksFor":"https://wordlift.io
/blog/en/organization/wordlift}
"Hi, my name is Andrea
Volpini, I am the CEO of
WordLift, you can find
me on Twitter as
@cyberandy and here is a
webpage
https://wordlift.io/blog
/en/entity/andrea-volpin
i"
UNSTRUCTURED
NLP
NLG
NLU

10. #brightonSEO
@cyberandy

11. #brightonSEO
@cyberandy

12. #brightonSEO
@cyberandy
WHAT IS SO
COMPLEX
ABOUT IT?

13. #brightonSEO
@cyberandy
WHAT IS
LINGUISTIC
SEMANTICS?
WHAT IS
LINGUISTIC
SEMANTICS?

14. THE MEANING
OF A WORD IS
ITS USE IN THE
LANGUAGE
Ludwig Wittgenstein
“
”

15. #brightonSEO
@cyberandy
HOW CAN WE DEFINE A WORD BY ITS USAGE?
YOU SHALL KNOW A WORD
BY THE COMPANY IT KEEPS.
John Rupert Firth
“
”

16. QUICK INTRO TO
TRANSFORMERS
#brightonSEO
@cyberandy

17. RECURRENT NEURAL NETWORKS
(RNN)
GATED RECURRENT UNIT
(GRU)
LONG SHORT TERM MEMORY
(LSTM)

18. ...AND WHY THEY
RECEIVED SO MUCH
ATTENTION?
#brightonSEO
@cyberandy

19. ...AND WHY THEY
RECEIVED SO MUCH
ATTENTION?
#brightonSEO
@cyberandy

21. “Pioggia a
catinelle”
“Rain cats
and dogs”
ENCODER
Input sequence to
embeddings
EMBEDDINGS
DECODER
Intermediate state to
output sequence

22. “Pioggia a
catinelle”
“Rain cats
and dogs”
ENCODER
Input sequence to
embeddings
EMBEDDINGS
DENCODER
Intermediate state to
output sequence
Multi-headed
Self Attention
Multi-headed
Self Attention
Multi-headed
Self Attention
Multi-headed
Self Attention

23. “Pioggia a
catinelle”
“Rain cats
and dogs”
ENCODER
Input sequence to
embeddings
EMBEDDINGS
DENCODER
Intermediate state to
output sequence
Multi-headed
Self Attention
Multi-headed
Self Attention
Multi-headed
Self Attention
Multi-headed
Self Attention

24. EMBEDDINGS
https://bit.ly/transformer-intro

25. #brightonSEO
@cyberandy
LET’S PREDICT THE WORD COMING AFTER...
“Pulp” 🤔

26. #brightonSEO
@cyberandy
INPUT: “Pulp” → OUTPUT: “Pulp Fiction” 😎
[[47, 29528]] [[47, 29528, 26756]]

27. ADDING MEANINGS
#brightonSEO
@cyberandy

28. ADDING MEANINGS
#brightonSEO
@cyberandy
From
Vectors
[[47, 29528]]

29. ADDING MEANINGS
#brightonSEO
@cyberandy
From
Vectors
[[47, 29528]]
To
Embeddings

30. #brightonSEO
@cyberandy
LET’S RUN ONE MORE PREDICTION
THIS TIME WE WILL COMPLETE A SENTENCE...

31. #brightonSEO
@cyberandy
LET’S RUN ONE MORE PREDICTION THIS TIME
TO COMPLETE A SENTENCE...

33. THE ATTENTION
MECHANISM

35. “The future
of SEO”
EMBEDDINGS
DECODER
Intermediate state to
output sequence
GPT architecture
● One single model
for a variety of
tasks
● Very simple to
use (priming is
done with text)
● Primarily trained
on English but
works also on
other languages
(sort of)

36. ● 175 billion
parameters
● closed-source
● very flexible can
be applied to
different tasks
GPT-3
#brightonSEO

37. GPT-J
● 6 billion
parameters
● the most
advanced
open-source
NLP model
#brightonSEO

38. Jurassic-1
● 178 billion
parameters
● closed-source
● larger model
vocabulary (3
times GPT-3)
#brightonSEO

39. OUTPUT
INPUT “LANGUAGE
MODELS DON’T
KNOW WHAT
THEY ARE
TALKING ABOUT”
Gary Marcus
"they're approximations [...] of language use
rather than language understanding.”

40. LANGUAGE MODELS ARE NOT
GROUNDED.
#brightonSEO

41. DALL·E is able to draw multiple
copies of an object, but is unable to
reliably count past three.

42. LANGUAGE MODELS CAN BE
RACIST.
#brightonSEO

43. LANGUAGE MODELS
INTRODUCE UNWANTED
BIASES.
#brightonSEO

44. HOW CAN
WE USE
THEM
#brightonSEO

45. ANYONE FLUENT IN
NATURAL
LANGUAGE IS A
PROGRAMMER

46. few-shot
zero-shot
fine-tuning
Q: Is Carbonara an Italian or an Indian recipe?
A: Carbonara is an Italian recipe.
NO
SAMPLES

47. few-shot
zero-shot
fine-tuning
Q: is Carbonara an Italian or an Indian recipe?
A: Carbonara is an Italian recipe.
Carbonara is an Italian recipe.
Duck Samosas is an Indian recipe.
Burritos are a Mexican recipe.
ONLY FEW
SAMPLES

48. few-shot
zero-shot
fine-tuning
Q: is Carbonara an Italian or an Indian recipe?
A: Carbonara is an Italian recipe.
Carbonara is an Italian recipe.
Duck Samosas is an Indian recipe.
Burritos are a Mexican recipe.
Carbonara is an Italian recipe.
Duck Samosas is an Indian recipe.
Burritos are a Mexican recipe.
Shish Tawook is an Egyptian recipe.
Sichuan Pork is a Chinese recipe.
Pizza Margherita is an Italian recipe.
Chicken tikka is an Indian recipe.
HUNDREDS
OF
SAMPLES

49. WHAT
SEO TASKS
CAN WE
AUTOMATE
#brightonSEO

50. title tag generation
search intent classification
frequently asked questions
product descriptions
query: I want to subscribe to wordlift
classes:
[informational,commercial,navigational,transactional]
label: transactional
topic: vaccine hesitancy, america
title: How America’s Vaccine Hesitancy is Hurting
Our Children
Q: how many boxes of daily contacts is one year of
supply?
title: One box is a year’s supply of contacts.
Montblanc | minimalist wallet | made of top quality
| abrasion-proof fabric | 5 times thinner than a
traditional leather wallet | holds up to 15 cards |
available in any color
Montblanc is a slim, lightweight wallet that allows
you to carry more credit cards at once and safely
walk around without fear of damaging your wallet.

51. TIPS & TRICKS
1 Provide an informative
context at the beginning of
the prompt to explain the
task and to condition the
tone of voice 👇
Describe the iconic
sunglasses, from a
list of features, to a
generation Y.
1

52. TIPS & TRICKS
1 Always remember
that any form of
incorrect grammar,
spelling, and
punctuation mistakes
will badly affect
completion.
2

53. TIPS & TRICKS
1 Add as many
examples as the
prompt will allow.
Diversification is
highly needed to
mitigate overfitting.
3

54. TIPS & TRICKS
1
Always consider for
your task an alternative
0-shot implementation.
This will improve the
ability of the model to
generalize the task and
force it to rely on its
vast acquired
knowledge rather than
on few examples.
4

55. TIPS & TRICKS
1
Add always a clear
stop sequence
(usually ### or ↵↵)
to prevent semantic
contamination.
5

56. TIPS & TRICKS
generate -> curate ->
generate is your
mantra.
“To generate
high-quality writing
you need to create a
prompt that seeds it
with high-quality
writing”
6

57. WHAT
IMPACT
CAN WE
EXPECT
#brightonSEO

58. GOOGLE DOESN’T LIKE AUTOMATICALLY
GENERATED CONTENT WHEN...
#brightonSEO

59. GOOGLE DOESN’T LIKE AUTOMATICALLY
GENERATED CONTENT WHEN...
IT’S INTENDED TO MANIPULATE SEARCH
RANKINGS AND NOT TO HELP USERS

60. R
O
I THE ROI OF AI-GENERATED FAQ?
Source: Google Search Console data from an e-commerce in
2K
CLICKS
2
MONTHS
$ 6.7K
CPC ADV
In 2 months, we generated
2.08K additional clicks that
would have otherwise cost
our client $ 6.7K on Google
Ads.
#brightonSEO

61. #brightonSEO
@cyberandy

62. 1
Content, even when
automatically generated,
needs to be updated and
maintained on regular basis.
2
We can seed questions from
both PAA or
what|how|when|why queries
that we’re already intercepting
in GSC.
3
Validation is both for humans
and for machines and helps
improving content quality.
#brightonSEO

63. #brightonSEO
@cyberandy
VALIDATION
FLOW
Here is how we do it
👉

64. #brightonSEO
THANK
YOU!
👉 wordlift.io