Richard Lawrence discusses how to create your own search quality evaluation algorithms by following Google's process. He explains that Google likely uses human raters to label content quality, then finds features that distinguish high and low quality content to build ranking algorithms. Lawrence proposes doing the same by labeling a sample of content, then using NLP models or multivariate techniques to build algorithms that measure how well content meets a search query's intent ("Needs Met") or achieves overall page quality. Creating such algorithms provides a way to evaluate content quality at scale.

1. How to create your own
search quality
evaluation algorithms
Richard Lawrence
Sanity.io
@richlawre

2. @richlawre
● Principal SEO at
Sanity
Who the hell is this guy anyway?

3. Who the hell is this guy anyway?
@richlawre
● Sanity is a headless
CMS and more!

4. @richlawre
● Doing a Data Science
degree in my spare
time
Who the hell is this guy anyway?

5. Onto some context
@richlawre

6. The ‘helpful content update’ might have
been a bit of a damp squib…
@richlawre

7. …but Google is always working towards
ranking helpful content more highly
@richlawre

8. So wouldn’t it be great to know if your
content is helping your audience - at scale?
@richlawre

9. The search rater guidelines hold the key
@richlawre
167 page document
that says what good
looks like!

10. Google says it doesn’t directly use the
ratings in its ranking algorithms
“We use responses from Raters
to evaluate changes, but they
don’t directly impact how our
search results are ranked.”
bit.ly/ratings-answer
@richlawre

11. But it will use the rated content to help find
features of what ‘good’ looks like
@richlawre

12. Similar methods have been used for years
in various areas - like counterfeit notes
@richlawre

13. Features are found that best separate
authentic and counterfeit notes
Distance between edge & watermark
Width of
shaded area
Counterfeit
Authentic
@richlawre

14. Features for high vs. low quality content will
likely be more complex
@richlawre

15. Bing confirmed this is how it works in 2019
bit.ly/bing-confirmation @richlawre

16. With 90% of its algorithms being ML based
@richlawre
bit.ly/bing-features

17. Plus it revealed its process
@richlawre
bit.ly/bing-process

18. So how can we harness this as an industry?
@richlawre

19. We can try to create our own!
@richlawre

20. 1. Label the content
2. Create a ‘Needs Met’ algorithm
3. Create a ‘Page Quality’ algorithm
What we need to do
@richlawre

21. Labelling the content
@richlawre

22. Get a representative sample of searches
448 million search queries
bit.ly/448-million @richlawre

23. Here’s how to play around with the file
@richlawre
bit.ly/large-file

24. Then gather the top 20 rankings for each
sample query
Likely available
feature of your
favourite rank
tracking software
@richlawre

25. Use some search raters to rate the content
Collect
labels
Choose
provider
Create
guidelines
Must not be
identical to
Google’s…
Needs Met &
Page Quality
2 search raters
with 3rd called in
for disagreements
@richlawre

26. Creating a Needs Met algorithm
@richlawre

27. This measures fulfilling search intent
Features will mainly be
relating to relevance
and structure
@richlawre

28. GPT language models are perfect for this
The open source option
@richlawre

29. GPT-3 became cheaper in September too
@richlawre

30. We need to create a pattern for GPT-J to learn
Content:
<h1>Compare car insurance quotes</h1>
<p>It's quick and easy to compare car insurance
and find cheaper cover – we just need a few
details about you and your vehicle.</p>
Target query: car insurance
Needs Met rating: Good
@richlawre

31. It will then rate new content
Content:
<h1>Car insurance</h1>
<p>From theft to write-offs and even lost keys,
you'll be covered with us. Here's what you'll like
about our comprehensive cover </p>
Target query: car insurance
Needs Met rating: ?????
@richlawre

32. We need to scrape content from each page to
give to the language model - with the rating
@richlawre

33. Then use this info to train GPT-J
@richlawre
bit.ly/finetune-gptj

34. You can also use existing services
@richlawre
NLP Cloud Forefront.ai

35. NLP Cloud also became cheaper!
@richlawre

36. Validate performance with a test set
@richlawre

37. Judge performance with a Confusion Matrix
@richlawre
Correct
Wrong
Correct Wrong
True positive False negative
False positive True negative
Actual
Prediction

38. Few shot learning can help improve
performance
@richlawre
Prompt
Example 1
Rating: Excellent
Example 2
Rating: Poor
Example 3
Rating: ????
GPT-J
Good

39. As can explaining to the model what it
needs to do!
@richlawre
Consider the content to rate.
Rate it according how well it
fits the search query.

40. We’ve done this for you within Sanity Studio
@richlawre

41. And lots of other great features
@richlawre

42. Contact us for more info about the beta for
these features:
bit.ly/sanity-beta
@richlawre

43. This isn’t perfect of course - though still very
useful
@richlawre
● Only text content
● Useful indication only
● Great at scale

44. Creating a Page Quality algorithm
@richlawre

45. This is much more difficult!
@richlawre

46. It measures how well a page achieves its
purpose
@richlawre
This is about quality of
content, independent
of search queries

47. So features can relate to a large number of
areas!
@richlawre
‘Main Content’ vs
‘Supplementary
Content’
Website
background
information
Amount of Main Content
Position of Main Content
Depth of ‘about’ info
Wikipedia presence

48. And you have to work out how to measure
them
@richlawre
Amount of Main
Content
Length of Main
Content area
Number of words
in Main Content

49. It becomes a huge multivariate challenge
@richlawre
Page
Length of
MC area
‘About us’
word count
Clicks to
‘About us’
Page 1 17cm 500 2
Page 2 20cm 300 1
Page 3 15cm 1000 2
Page 4 25cm 750 3

50. Then we need to find features that best
separate the groups
Number of words in ‘About’ section
Length of
‘Main Content’
area
High quality
Low quality
@richlawre

51. But with a large number of features!
@richlawre

52. This can be explored with a number of
potential models
@richlawre
Linear Discriminant Analysis

53. @richlawre
This can be explored with a number of
potential models
Random Forest

54. @richlawre
This can be explored with a number of
potential models
Neural Network

55. This is a huge challenge!
@richlawre

56. Which features?
@richlawre

57. How to measure them?
@richlawre

58. Which model?
@richlawre

59. The work is ongoing here!
@richlawre

60. Let’s sum up
@richlawre

61. Google likely uses its raters to gather
labelled data on content quality
@richlawre

62. It will then likely use that to find features of
‘good’ and ‘bad’ content
@richlawre

63. And creates algorithms to distinguish
between the two
@richlawre

64. You can do the same!
@richlawre

65. Get your own labelled content and create
your own scoring algorithms
@richlawre

66. We have created a ‘Needs Met’ score within
Sanity Studio
@richlawre

67. So that you can get an indication of content
calibre directly in your publishing workflow
@richlawre

68. Contact us to get more info about the beta
here:
bit.ly/sanity-beta
@richlawre

69. Richard Lawrence
Principal at Sanity.io
@richlawre
@richlawre