Avoid duplicate content and don’t leave money on the table with unoptimized groups of pages linked by canonical declarations! Particularly in e-commerce, you can increase Google’s confidence by making sure your groups of product URLs are perfectly canonicalized and clear to search engines.

1. Automated Duplicate
Content Consolidation with
Google Cloud Functions

2. Speaking today /
Présenté par
Automating Google
Lighthouse
Hamlet Batista // RankSense
slidehare.net/hamletbatista
@hamletbatista

3. https://jamesclear.com/marginal-gains

4. Agenda
ØFinding marginal but
repeatable success
ØScaling it with automation

5. Cruiseline.com
Success Story

9. Ø No www to non-www
redirects
Ø No canonicals
Ø Redundant parameter
URLs
Ø Only 1.40% of
indexed pages with
search clicks (out of
+300k pages)

11. The Google SEO
Scorecard Report

13. Ø Duplicate content
consolidation can be
executed relatively quickly,
as it requires a small set of
technical changes
Ø You will likely see improved
rankings within weeks after
the corrections are in place
Ø New changes and
improvements to your site
are picked up faster by
Google

14. Ø Natzir found the
total traffic to
pages ranking
for the same
keyword was
less than when
consolidated
with redirects
Ø Same idea but
from a
keywords’
perspective
https://www.youtube.com/watch?v=zI_jkhSyAew

15. Cruiseline.com
Reverse
Engineering

16. Ø Finding repeatable success
Ø Searching for a machine
learning model to connect
new visits to technical SEO
changes
Ø We focused on the impact
of links, indexing, and
canonical clustering

18. Our best predictive model
achieved 85% test accuracy
Ø Canonicalization drives
repeatable success
Ø The size of the canonical
cluster turned out to be a
strong predictor

19. One oversimplified way to
think about a machine
learning model is to
picture a linear regression
function in Excel/Sheets.
We predicted new users
(Y) within canonicalized
clusters dependent on the
size of the clusters (X).
Machine Learning 101
https://bit.ly/3lGyeqA

20. To Canonicalize
or Not to
Canonicalize

21. Current canonical clustering is
mostly self-referential (orange)
Every product variant
canonicalizes to itself.

22. Their optimal canonical setup is the
inverse.
Most clusters should canonicalize to one
product “leader”

23. For some products, people
specific the color they want
directly in Google. But, for other
products, they don’t.
They decide the color they want
after seeing the options
available in the site.

24. https://bit.ly/36ZxXel

25. Technical Plan
Ø Build clusters using OnCrawl
Ø Get search demand using SEMrush
Ø Canonicalization algorithm
Ø Experiment on CDN using RankSense
Ø Automate everything using Cloud Functions and
Pub/sub queues

26. Coupled vs
Decoupled
Systems

28. Pub/Sub is an asynchronous
messaging service that
decouples services that
produce events from services
that process events.
It allows us to connect
OnCrawl, SEMrush, and
RankSense asynchronously to
complete a custom workflow.

30. Cloud Scheduler acts as a
single pane of glass, allowing us
to manage all our automation
tasks from one place.
It allows us to trigger our custom
workflow on recurring times as
search demand changes with
seasons.

31. Technical Plan

32. Technical Plan

33. Clustering with
OnCrawl

39. Search Demand
Tracking with
SEMrush

41. Ø Cloud Scheduler triggers
OnCrawl Cloud Function
which uploads each craw
export to Cloud Storage
Ø Cloud Storage update
triggers SEMrush Cloud
Function which then exports
search demand data to
Cloud Storage

43. Canonicalization
Algorithm

44. Ø We are going to perform an
intermediate step and force
all product groups to
canonicalize to the “leader”
URL in the group.
Ø The “leader” could be the
URL with most search
traffic, more
internal/external links or
most frequently crawled

46. We end up with one cluster that
we need to update, which
means that David Yurman is
leaving a lot of money on the
table with their current setup
that relies on self-referential
canonicals.

47. Deploying to
Cloudflare’s CDN
with RankSense

49. We are going to use the
RankSense API to publish our
new canonical clusters as
experiments in the Cloudflare
CDN
https://bit.ly/3jWm4JP

50. Ø We automatically populate
a Google Sheet with the
changes
Ø We submit the Sheet to
RankSense’s
PRODUCTION environment

52. Resources to Learn More
Ø Python code covered in this presentation
https://github.com/ranksense/weloveseo
Ø Advanced Duplicate Content Consolidation with
Python
https://www.searchenginejournal.com/advanced-
duplicate-content-consolidation-python/314471/
Ø Cloud Functions https://cloud.google.com/functions
Ø Google PubSub https://cloud.google.com/pubsub
Ø Introduction to Python for SEO Pros
https://www.searchenginejournal.com/introduction-
to-python-seo-spreadsheets/342779/

53. Thank you!