SlideShare a Scribd company logo

Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017

You have a shiny new site and your brand is looking for a fresh start with their offering. It may be one of many past migrations, protocol switches and redirections you're undertaken historically. But then you find that things didn't quite go as you expected. You never really got back to where you wanted to be in organic search. Part of this is because 'Gone is never Gone'. Every URL that ever was known of on your site is listed in the history logs in the Google search engine system and history logs are used to determine the amount of time your site will be apportioned crawling. You inherited technical SEO debt and generational cruft where everything gets blurred for Google in understanding which is the target URL for a particular term. This can be particularly prevalent when you migrate from one ecommerce platform to another because past crawling rules developed for your site are now not applicable but are still in the history and crawl patterns discovered.

1 of 61
@dawnieando #BigDigitalADL
Generational	
  Cruft	
  &	
  Technical	
  Debt	
  in	
  
SEOGONE	
  IS	
  
NEVER	
  GONE Dawn	
  Anderson	
  @	
  dawnieando
@dawnieando #BigDigitalADL
A New Beginning
§ “A	
  new	
  website	
  will	
  solve	
  ALL	
  our	
  
problems”
§ “Let’s	
  start	
  again”
§ “We’ll	
  just	
  migrate…	
  and	
  redirect	
  
everything”
@dawnieando #BigDigitalADL
404 Page Not Found
§ “Of	
  course,	
  we	
  won’t	
  
redirect	
  everything…”
§ “Not	
  everything	
  will	
  
be	
  worth	
  redirecting”
@dawnieando #BigDigitalADL
410 Gone
§ “Some,	
  we’ll	
  just	
  kill	
  
off	
  with	
  a	
  410…”
§ “Then	
  the	
  URLs	
  will	
  
be	
  gone”
@dawnieando #BigDigitalADL
But in Reality…it Still Exists
@dawnieando #BigDigitalADL
Because…Web Crawler System’s History Logs
SEARCH	
  ENGINES	
  HAVE	
  A	
  	
  BIG	
  MEMORY	
  AND	
  A	
  LOT	
  OF	
  
STORAGE

Recommended

SEO Cannibalisation of Your Own SEO Success
SEO Cannibalisation of Your Own SEO SuccessSEO Cannibalisation of Your Own SEO Success
SEO Cannibalisation of Your Own SEO SuccessDawn Anderson MSc DigM
 
Bringing in the family to emphasise importance and win during crawling
Bringing in the family to emphasise importance and win during crawlingBringing in the family to emphasise importance and win during crawling
Bringing in the family to emphasise importance and win during crawlingDawn Anderson MSc DigM
 
SEO - Stop Eating Your Words - Avoid Cannibalisation Of Your Sites
SEO - Stop Eating Your Words - Avoid Cannibalisation Of Your SitesSEO - Stop Eating Your Words - Avoid Cannibalisation Of Your Sites
SEO - Stop Eating Your Words - Avoid Cannibalisation Of Your SitesDawn Anderson MSc DigM
 
Negotiating crawl budget with googlebots
Negotiating crawl budget with googlebotsNegotiating crawl budget with googlebots
Negotiating crawl budget with googlebotsDawn Anderson MSc DigM
 
Infinite Loops Dirty Architecture And Too Many Indexed URLs
Infinite Loops Dirty Architecture And Too Many Indexed URLsInfinite Loops Dirty Architecture And Too Many Indexed URLs
Infinite Loops Dirty Architecture And Too Many Indexed URLsDawn Anderson MSc DigM
 
Creating Commerce Reviews and Considering The Case For User Generated Reviews
Creating Commerce Reviews and Considering The Case For User Generated ReviewsCreating Commerce Reviews and Considering The Case For User Generated Reviews
Creating Commerce Reviews and Considering The Case For User Generated ReviewsDawn Anderson MSc DigM
 
Sasconbeta 2015 Dawn Anderson - Talk To The Spider
Sasconbeta 2015 Dawn Anderson - Talk To The SpiderSasconbeta 2015 Dawn Anderson - Talk To The Spider
Sasconbeta 2015 Dawn Anderson - Talk To The SpiderDawn Anderson MSc DigM
 
Digital Olympus Technical SEO Findings Whilst Taming An SEO Beast
Digital Olympus Technical SEO Findings Whilst Taming An SEO BeastDigital Olympus Technical SEO Findings Whilst Taming An SEO Beast
Digital Olympus Technical SEO Findings Whilst Taming An SEO BeastDawn Anderson MSc DigM
 

More Related Content

What's hot

Duplicate Content Myths Types and Ways To Make It Work For You
Duplicate Content Myths Types and Ways To Make It Work For YouDuplicate Content Myths Types and Ways To Make It Work For You
Duplicate Content Myths Types and Ways To Make It Work For YouDawn Anderson MSc DigM
 
SEO Crawl Rank And Crawl Tank - Brighton SEO April 2016
SEO Crawl Rank And Crawl Tank - Brighton SEO April 2016SEO Crawl Rank And Crawl Tank - Brighton SEO April 2016
SEO Crawl Rank And Crawl Tank - Brighton SEO April 2016Dawn Anderson MSc DigM
 
Pubcon florida 2018 logs dont lie dawn anderson
Pubcon florida 2018 logs dont lie dawn andersonPubcon florida 2018 logs dont lie dawn anderson
Pubcon florida 2018 logs dont lie dawn andersonDawn Anderson MSc DigM
 
Dawn Anderson SEO Consumer Choice Crawl Budget Optimization Conflicts
Dawn Anderson SEO Consumer Choice Crawl Budget Optimization ConflictsDawn Anderson SEO Consumer Choice Crawl Budget Optimization Conflicts
Dawn Anderson SEO Consumer Choice Crawl Budget Optimization ConflictsDawn Anderson MSc DigM
 
Technical SEO Myths Facts And Theories On Crawl Budget And The Importance Of ...
Technical SEO Myths Facts And Theories On Crawl Budget And The Importance Of ...Technical SEO Myths Facts And Theories On Crawl Budget And The Importance Of ...
Technical SEO Myths Facts And Theories On Crawl Budget And The Importance Of ...Dawn Anderson MSc DigM
 
SEO and The Mobile-First Paradigm Shift
SEO and The Mobile-First Paradigm ShiftSEO and The Mobile-First Paradigm Shift
SEO and The Mobile-First Paradigm ShiftDawn Anderson MSc DigM
 
SEO - The Rise of Persona Modelled Intent Driven Contextual Search
SEO - The Rise of Persona Modelled Intent Driven Contextual SearchSEO - The Rise of Persona Modelled Intent Driven Contextual Search
SEO - The Rise of Persona Modelled Intent Driven Contextual SearchDawn Anderson MSc DigM
 
Technical SEO - Generational cruft in SEO - there is never a new site when th...
Technical SEO - Generational cruft in SEO - there is never a new site when th...Technical SEO - Generational cruft in SEO - there is never a new site when th...
Technical SEO - Generational cruft in SEO - there is never a new site when th...Dawn Anderson MSc DigM
 
AMP Accelerated Mobile Pages - To AMPFinity And Beyond
AMP Accelerated Mobile Pages - To AMPFinity And BeyondAMP Accelerated Mobile Pages - To AMPFinity And Beyond
AMP Accelerated Mobile Pages - To AMPFinity And BeyondDawn Anderson MSc DigM
 
Cruft busting technical debt code smell and refactoring for seo - state of ...
Cruft busting   technical debt code smell and refactoring for seo - state of ...Cruft busting   technical debt code smell and refactoring for seo - state of ...
Cruft busting technical debt code smell and refactoring for seo - state of ...Dawn Anderson MSc DigM
 
Owl and The Hummingbird - Ontology and SEO
Owl and The Hummingbird - Ontology and SEOOwl and The Hummingbird - Ontology and SEO
Owl and The Hummingbird - Ontology and SEODawn Anderson MSc DigM
 
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUKeeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUJason Mun
 
Crawl Budget - Some Insights & Ideas @ seokomm 2015
Crawl Budget - Some Insights & Ideas @ seokomm 2015Crawl Budget - Some Insights & Ideas @ seokomm 2015
Crawl Budget - Some Insights & Ideas @ seokomm 2015Jan Hendrik Merlin Jacob
 
Lots of ways to speed up your site
Lots of ways to speed up your siteLots of ways to speed up your site
Lots of ways to speed up your siteIan Lurie
 
Google Search Engine Ranking Position - 200 Top Ranking Factors for SEO Marke...
Google Search Engine Ranking Position - 200 Top Ranking Factors for SEO Marke...Google Search Engine Ranking Position - 200 Top Ranking Factors for SEO Marke...
Google Search Engine Ranking Position - 200 Top Ranking Factors for SEO Marke...Ronald Soh
 
SearchLove San Diego 2018 | Will Critchlow | From the Horse’s Mouth: What We ...
SearchLove San Diego 2018 | Will Critchlow | From the Horse’s Mouth: What We ...SearchLove San Diego 2018 | Will Critchlow | From the Horse’s Mouth: What We ...
SearchLove San Diego 2018 | Will Critchlow | From the Horse’s Mouth: What We ...Distilled
 
Google's Search Signals For Page Experience - SMX Advanced 2021 Patrick Stox
Google's Search Signals For Page Experience - SMX Advanced 2021 Patrick StoxGoogle's Search Signals For Page Experience - SMX Advanced 2021 Patrick Stox
Google's Search Signals For Page Experience - SMX Advanced 2021 Patrick StoxAhrefs
 
Moving URLs: Structural Web changes 
without losing rankings #SearchLove
Moving URLs: Structural Web changes 
without losing rankings #SearchLoveMoving URLs: Structural Web changes 
without losing rankings #SearchLove
Moving URLs: Structural Web changes 
without losing rankings #SearchLoveAleyda Solís
 
How We Make Websites
How We Make WebsitesHow We Make Websites
How We Make Websitesfantasticlife
 

What's hot (20)

Duplicate Content Myths Types and Ways To Make It Work For You
Duplicate Content Myths Types and Ways To Make It Work For YouDuplicate Content Myths Types and Ways To Make It Work For You
Duplicate Content Myths Types and Ways To Make It Work For You
 
SEO Crawl Rank And Crawl Tank - Brighton SEO April 2016
SEO Crawl Rank And Crawl Tank - Brighton SEO April 2016SEO Crawl Rank And Crawl Tank - Brighton SEO April 2016
SEO Crawl Rank And Crawl Tank - Brighton SEO April 2016
 
Pubcon florida 2018 logs dont lie dawn anderson
Pubcon florida 2018 logs dont lie dawn andersonPubcon florida 2018 logs dont lie dawn anderson
Pubcon florida 2018 logs dont lie dawn anderson
 
Dawn Anderson SEO Consumer Choice Crawl Budget Optimization Conflicts
Dawn Anderson SEO Consumer Choice Crawl Budget Optimization ConflictsDawn Anderson SEO Consumer Choice Crawl Budget Optimization Conflicts
Dawn Anderson SEO Consumer Choice Crawl Budget Optimization Conflicts
 
Technical SEO Myths Facts And Theories On Crawl Budget And The Importance Of ...
Technical SEO Myths Facts And Theories On Crawl Budget And The Importance Of ...Technical SEO Myths Facts And Theories On Crawl Budget And The Importance Of ...
Technical SEO Myths Facts And Theories On Crawl Budget And The Importance Of ...
 
SEO and The Mobile-First Paradigm Shift
SEO and The Mobile-First Paradigm ShiftSEO and The Mobile-First Paradigm Shift
SEO and The Mobile-First Paradigm Shift
 
SEO - The Rise of Persona Modelled Intent Driven Contextual Search
SEO - The Rise of Persona Modelled Intent Driven Contextual SearchSEO - The Rise of Persona Modelled Intent Driven Contextual Search
SEO - The Rise of Persona Modelled Intent Driven Contextual Search
 
Technical SEO - Generational cruft in SEO - there is never a new site when th...
Technical SEO - Generational cruft in SEO - there is never a new site when th...Technical SEO - Generational cruft in SEO - there is never a new site when th...
Technical SEO - Generational cruft in SEO - there is never a new site when th...
 
AMP Accelerated Mobile Pages - To AMPFinity And Beyond
AMP Accelerated Mobile Pages - To AMPFinity And BeyondAMP Accelerated Mobile Pages - To AMPFinity And Beyond
AMP Accelerated Mobile Pages - To AMPFinity And Beyond
 
Cruft busting technical debt code smell and refactoring for seo - state of ...
Cruft busting   technical debt code smell and refactoring for seo - state of ...Cruft busting   technical debt code smell and refactoring for seo - state of ...
Cruft busting technical debt code smell and refactoring for seo - state of ...
 
Owl and The Hummingbird - Ontology and SEO
Owl and The Hummingbird - Ontology and SEOOwl and The Hummingbird - Ontology and SEO
Owl and The Hummingbird - Ontology and SEO
 
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUKeeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
 
Crawl Budget - Some Insights & Ideas @ seokomm 2015
Crawl Budget - Some Insights & Ideas @ seokomm 2015Crawl Budget - Some Insights & Ideas @ seokomm 2015
Crawl Budget - Some Insights & Ideas @ seokomm 2015
 
SEO Benchmark
SEO BenchmarkSEO Benchmark
SEO Benchmark
 
Lots of ways to speed up your site
Lots of ways to speed up your siteLots of ways to speed up your site
Lots of ways to speed up your site
 
Google Search Engine Ranking Position - 200 Top Ranking Factors for SEO Marke...
Google Search Engine Ranking Position - 200 Top Ranking Factors for SEO Marke...Google Search Engine Ranking Position - 200 Top Ranking Factors for SEO Marke...
Google Search Engine Ranking Position - 200 Top Ranking Factors for SEO Marke...
 
SearchLove San Diego 2018 | Will Critchlow | From the Horse’s Mouth: What We ...
SearchLove San Diego 2018 | Will Critchlow | From the Horse’s Mouth: What We ...SearchLove San Diego 2018 | Will Critchlow | From the Horse’s Mouth: What We ...
SearchLove San Diego 2018 | Will Critchlow | From the Horse’s Mouth: What We ...
 
Google's Search Signals For Page Experience - SMX Advanced 2021 Patrick Stox
Google's Search Signals For Page Experience - SMX Advanced 2021 Patrick StoxGoogle's Search Signals For Page Experience - SMX Advanced 2021 Patrick Stox
Google's Search Signals For Page Experience - SMX Advanced 2021 Patrick Stox
 
Moving URLs: Structural Web changes 
without losing rankings #SearchLove
Moving URLs: Structural Web changes 
without losing rankings #SearchLoveMoving URLs: Structural Web changes 
without losing rankings #SearchLove
Moving URLs: Structural Web changes 
without losing rankings #SearchLove
 
How We Make Websites
How We Make WebsitesHow We Make Websites
How We Make Websites
 

Similar to Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017

Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...
Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...
Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...Thiago de Faria
 
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...Codemotion
 
Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...
Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...
Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...Thiago de Faria
 
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...Codemotion
 
David Brown - Crawl Efficiency & Fixing Common Crawl Issues
David Brown - Crawl Efficiency & Fixing Common Crawl Issues David Brown - Crawl Efficiency & Fixing Common Crawl Issues
David Brown - Crawl Efficiency & Fixing Common Crawl Issues tmwi
 
SEO Table Stakes - Clarity 14
SEO Table Stakes - Clarity 14SEO Table Stakes - Clarity 14
SEO Table Stakes - Clarity 14Keith Goode
 
SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs
SearchLove Boston 2016 | Mike King | Developer Thinking for SEOsSearchLove Boston 2016 | Mike King | Developer Thinking for SEOs
SearchLove Boston 2016 | Mike King | Developer Thinking for SEOsDistilled
 
Tips and tactics that generate clicks and impressions - Daniel Brooks
Tips and tactics that generate clicks and impressions - Daniel BrooksTips and tactics that generate clicks and impressions - Daniel Brooks
Tips and tactics that generate clicks and impressions - Daniel BrooksSearchNorwich
 
Search Norwich #7 - Keyword Research Tips & Tactics To Increase Clicks & Impr...
Search Norwich #7 - Keyword Research Tips & Tactics To Increase Clicks & Impr...Search Norwich #7 - Keyword Research Tips & Tactics To Increase Clicks & Impr...
Search Norwich #7 - Keyword Research Tips & Tactics To Increase Clicks & Impr...Daniel Brooks
 
Winning SEO when doing Web Migrations #SEO4Life
Winning SEO when doing Web Migrations #SEO4LifeWinning SEO when doing Web Migrations #SEO4Life
Winning SEO when doing Web Migrations #SEO4LifeAleyda Solís
 
What To Do When You Can't Do Anything - SEO Soft Skills - Clarity '14
What To Do When You Can't Do Anything - SEO Soft Skills - Clarity '14What To Do When You Can't Do Anything - SEO Soft Skills - Clarity '14
What To Do When You Can't Do Anything - SEO Soft Skills - Clarity '14Keith Goode
 
The Best Employee Communications Tool You Already Own But Aren't Using
The Best Employee Communications Tool You Already Own But Aren't UsingThe Best Employee Communications Tool You Already Own But Aren't Using
The Best Employee Communications Tool You Already Own But Aren't UsingJWTINSIDE
 
Do we know our data, as good as we know our tools
Do we know our data, as good as we know our tools Do we know our data, as good as we know our tools
Do we know our data, as good as we know our tools Jeremie Charlet
 
How Search Works
How Search WorksHow Search Works
How Search WorksAhrefs
 
devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...
devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...
devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...Thiago de Faria
 
SMX West: Future-Proof Your Site for Google's Core Algorithm Updates by Lily Ray
SMX West: Future-Proof Your Site for Google's Core Algorithm Updates by Lily RaySMX West: Future-Proof Your Site for Google's Core Algorithm Updates by Lily Ray
SMX West: Future-Proof Your Site for Google's Core Algorithm Updates by Lily RayLily Ray
 
SEO In 2022: Google Discover and Microsite SERPs - (SEMrush Webinar)
SEO In 2022: Google Discover and Microsite SERPs - (SEMrush Webinar)SEO In 2022: Google Discover and Microsite SERPs - (SEMrush Webinar)
SEO In 2022: Google Discover and Microsite SERPs - (SEMrush Webinar)Evolving SEO
 
International SEO #SEBC Stockholm
International SEO #SEBC StockholmInternational SEO #SEBC Stockholm
International SEO #SEBC StockholmLisa Myers
 
Website Migrations at SMX Munich 2019 - Patrick Stox
Website Migrations at SMX Munich 2019 - Patrick StoxWebsite Migrations at SMX Munich 2019 - Patrick Stox
Website Migrations at SMX Munich 2019 - Patrick Stoxpatrickstox
 

Similar to Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017 (20)

Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...
Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...
Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...
 
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
 
Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...
Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...
Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...
 
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...
 
David Brown - Crawl Efficiency & Fixing Common Crawl Issues
David Brown - Crawl Efficiency & Fixing Common Crawl Issues David Brown - Crawl Efficiency & Fixing Common Crawl Issues
David Brown - Crawl Efficiency & Fixing Common Crawl Issues
 
SEO Table Stakes - Clarity 14
SEO Table Stakes - Clarity 14SEO Table Stakes - Clarity 14
SEO Table Stakes - Clarity 14
 
SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs
SearchLove Boston 2016 | Mike King | Developer Thinking for SEOsSearchLove Boston 2016 | Mike King | Developer Thinking for SEOs
SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs
 
Tips and tactics that generate clicks and impressions - Daniel Brooks
Tips and tactics that generate clicks and impressions - Daniel BrooksTips and tactics that generate clicks and impressions - Daniel Brooks
Tips and tactics that generate clicks and impressions - Daniel Brooks
 
Search Norwich #7 - Keyword Research Tips & Tactics To Increase Clicks & Impr...
Search Norwich #7 - Keyword Research Tips & Tactics To Increase Clicks & Impr...Search Norwich #7 - Keyword Research Tips & Tactics To Increase Clicks & Impr...
Search Norwich #7 - Keyword Research Tips & Tactics To Increase Clicks & Impr...
 
Winning SEO when doing Web Migrations #SEO4Life
Winning SEO when doing Web Migrations #SEO4LifeWinning SEO when doing Web Migrations #SEO4Life
Winning SEO when doing Web Migrations #SEO4Life
 
What To Do When You Can't Do Anything - SEO Soft Skills - Clarity '14
What To Do When You Can't Do Anything - SEO Soft Skills - Clarity '14What To Do When You Can't Do Anything - SEO Soft Skills - Clarity '14
What To Do When You Can't Do Anything - SEO Soft Skills - Clarity '14
 
The Best Employee Communications Tool You Already Own But Aren't Using
The Best Employee Communications Tool You Already Own But Aren't UsingThe Best Employee Communications Tool You Already Own But Aren't Using
The Best Employee Communications Tool You Already Own But Aren't Using
 
Do we know our data, as good as we know our tools
Do we know our data, as good as we know our tools Do we know our data, as good as we know our tools
Do we know our data, as good as we know our tools
 
How Search Works
How Search WorksHow Search Works
How Search Works
 
devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...
devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...
devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...
 
Technical SEO Audit
Technical SEO AuditTechnical SEO Audit
Technical SEO Audit
 
SMX West: Future-Proof Your Site for Google's Core Algorithm Updates by Lily Ray
SMX West: Future-Proof Your Site for Google's Core Algorithm Updates by Lily RaySMX West: Future-Proof Your Site for Google's Core Algorithm Updates by Lily Ray
SMX West: Future-Proof Your Site for Google's Core Algorithm Updates by Lily Ray
 
SEO In 2022: Google Discover and Microsite SERPs - (SEMrush Webinar)
SEO In 2022: Google Discover and Microsite SERPs - (SEMrush Webinar)SEO In 2022: Google Discover and Microsite SERPs - (SEMrush Webinar)
SEO In 2022: Google Discover and Microsite SERPs - (SEMrush Webinar)
 
International SEO #SEBC Stockholm
International SEO #SEBC StockholmInternational SEO #SEBC Stockholm
International SEO #SEBC Stockholm
 
Website Migrations at SMX Munich 2019 - Patrick Stox
Website Migrations at SMX Munich 2019 - Patrick StoxWebsite Migrations at SMX Munich 2019 - Patrick Stox
Website Migrations at SMX Munich 2019 - Patrick Stox
 

More from Dawn Anderson MSc DigM

Natural Semantic SEO - Surfacing Walnuts in Densely Represented, Every Increa...
Natural Semantic SEO - Surfacing Walnuts in Densely Represented, Every Increa...Natural Semantic SEO - Surfacing Walnuts in Densely Represented, Every Increa...
Natural Semantic SEO - Surfacing Walnuts in Densely Represented, Every Increa...Dawn Anderson MSc DigM
 
Passage indexing is likely more important than you think
Passage indexing is likely more important than you thinkPassage indexing is likely more important than you think
Passage indexing is likely more important than you thinkDawn Anderson MSc DigM
 
Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn And...
Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn And...Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn And...
Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn And...Dawn Anderson MSc DigM
 
Google BERT - SMX London 2020 Virtual Conference
Google BERT - SMX London 2020 Virtual ConferenceGoogle BERT - SMX London 2020 Virtual Conference
Google BERT - SMX London 2020 Virtual ConferenceDawn Anderson MSc DigM
 
Google BERT - What SEOs and Marketers Need to Know
Google BERT - What SEOs and Marketers Need to KnowGoogle BERT - What SEOs and Marketers Need to Know
Google BERT - What SEOs and Marketers Need to KnowDawn Anderson MSc DigM
 
Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020
Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020
Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020Dawn Anderson MSc DigM
 
2019 Tech SEO Boost Dawn Anderson Contextual Recommender Search
2019 Tech SEO Boost Dawn Anderson Contextual Recommender Search2019 Tech SEO Boost Dawn Anderson Contextual Recommender Search
2019 Tech SEO Boost Dawn Anderson Contextual Recommender SearchDawn Anderson MSc DigM
 
Connecting The Worlds of Information Retrieval & SEO - Search solutions 2019 ...
Connecting The Worlds of Information Retrieval & SEO - Search solutions 2019 ...Connecting The Worlds of Information Retrieval & SEO - Search solutions 2019 ...
Connecting The Worlds of Information Retrieval & SEO - Search solutions 2019 ...Dawn Anderson MSc DigM
 
Planning an SEO Strategy for a New Website - SMXL Milan 2019
Planning an SEO Strategy for a New Website - SMXL Milan 2019Planning an SEO Strategy for a New Website - SMXL Milan 2019
Planning an SEO Strategy for a New Website - SMXL Milan 2019Dawn Anderson MSc DigM
 
Google BERT and Family and the Natural Language Understanding Leaderboard Race
Google BERT and Family and the Natural Language Understanding Leaderboard RaceGoogle BERT and Family and the Natural Language Understanding Leaderboard Race
Google BERT and Family and the Natural Language Understanding Leaderboard RaceDawn Anderson MSc DigM
 
The User is the Query - The Rise of Predictive Proactive Search
The User is the Query - The Rise of Predictive Proactive SearchThe User is the Query - The Rise of Predictive Proactive Search
The User is the Query - The Rise of Predictive Proactive SearchDawn Anderson MSc DigM
 
Natural Language Processing and Search Intent Understanding C3 Conductor 2019...
Natural Language Processing and Search Intent Understanding C3 Conductor 2019...Natural Language Processing and Search Intent Understanding C3 Conductor 2019...
Natural Language Processing and Search Intent Understanding C3 Conductor 2019...Dawn Anderson MSc DigM
 
Using topic modelling frameworks for NLP and semantic search
Using topic modelling frameworks for NLP and semantic searchUsing topic modelling frameworks for NLP and semantic search
Using topic modelling frameworks for NLP and semantic searchDawn Anderson MSc DigM
 
Voice Search and Conversation Action Assistive Systems - Challenges & Opportu...
Voice Search and Conversation Action Assistive Systems - Challenges & Opportu...Voice Search and Conversation Action Assistive Systems - Challenges & Opportu...
Voice Search and Conversation Action Assistive Systems - Challenges & Opportu...Dawn Anderson MSc DigM
 
The Iceberg Approach - Power from what lies beneath in SEO for a mobile-first...
The Iceberg Approach - Power from what lies beneath in SEO for a mobile-first...The Iceberg Approach - Power from what lies beneath in SEO for a mobile-first...
The Iceberg Approach - Power from what lies beneath in SEO for a mobile-first...Dawn Anderson MSc DigM
 
Voice Search Challenges For Search and Information Retrieval and SEO
Voice Search Challenges For Search and Information Retrieval and SEOVoice Search Challenges For Search and Information Retrieval and SEO
Voice Search Challenges For Search and Information Retrieval and SEODawn Anderson MSc DigM
 
MOZCON 2017 WINNING WITH CHOICE & INFORMATION SYSTEMS FOR BOTH CRAWLERS & CON...
MOZCON 2017 WINNING WITH CHOICE & INFORMATION SYSTEMS FOR BOTH CRAWLERS & CON...MOZCON 2017 WINNING WITH CHOICE & INFORMATION SYSTEMS FOR BOTH CRAWLERS & CON...
MOZCON 2017 WINNING WITH CHOICE & INFORMATION SYSTEMS FOR BOTH CRAWLERS & CON...Dawn Anderson MSc DigM
 
AMP Accelerated Mobile Pages - The Next Generation SMX London 2017 Dawn Anderson
AMP Accelerated Mobile Pages - The Next Generation SMX London 2017 Dawn AndersonAMP Accelerated Mobile Pages - The Next Generation SMX London 2017 Dawn Anderson
AMP Accelerated Mobile Pages - The Next Generation SMX London 2017 Dawn AndersonDawn Anderson MSc DigM
 

More from Dawn Anderson MSc DigM (20)

Natural Semantic SEO - Surfacing Walnuts in Densely Represented, Every Increa...
Natural Semantic SEO - Surfacing Walnuts in Densely Represented, Every Increa...Natural Semantic SEO - Surfacing Walnuts in Densely Represented, Every Increa...
Natural Semantic SEO - Surfacing Walnuts in Densely Represented, Every Increa...
 
Passage indexing is likely more important than you think
Passage indexing is likely more important than you thinkPassage indexing is likely more important than you think
Passage indexing is likely more important than you think
 
Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn And...
Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn And...Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn And...
Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn And...
 
Google BERT - SMX London 2020 Virtual Conference
Google BERT - SMX London 2020 Virtual ConferenceGoogle BERT - SMX London 2020 Virtual Conference
Google BERT - SMX London 2020 Virtual Conference
 
Google BERT - What SEOs and Marketers Need to Know
Google BERT - What SEOs and Marketers Need to KnowGoogle BERT - What SEOs and Marketers Need to Know
Google BERT - What SEOs and Marketers Need to Know
 
Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020
Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020
Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020
 
2019 Tech SEO Boost Dawn Anderson Contextual Recommender Search
2019 Tech SEO Boost Dawn Anderson Contextual Recommender Search2019 Tech SEO Boost Dawn Anderson Contextual Recommender Search
2019 Tech SEO Boost Dawn Anderson Contextual Recommender Search
 
Connecting The Worlds of Information Retrieval & SEO - Search solutions 2019 ...
Connecting The Worlds of Information Retrieval & SEO - Search solutions 2019 ...Connecting The Worlds of Information Retrieval & SEO - Search solutions 2019 ...
Connecting The Worlds of Information Retrieval & SEO - Search solutions 2019 ...
 
Planning an SEO Strategy for a New Website - SMXL Milan 2019
Planning an SEO Strategy for a New Website - SMXL Milan 2019Planning an SEO Strategy for a New Website - SMXL Milan 2019
Planning an SEO Strategy for a New Website - SMXL Milan 2019
 
Google BERT and Family and the Natural Language Understanding Leaderboard Race
Google BERT and Family and the Natural Language Understanding Leaderboard RaceGoogle BERT and Family and the Natural Language Understanding Leaderboard Race
Google BERT and Family and the Natural Language Understanding Leaderboard Race
 
The User is the Query - The Rise of Predictive Proactive Search
The User is the Query - The Rise of Predictive Proactive SearchThe User is the Query - The Rise of Predictive Proactive Search
The User is the Query - The Rise of Predictive Proactive Search
 
Natural Language Processing and Search Intent Understanding C3 Conductor 2019...
Natural Language Processing and Search Intent Understanding C3 Conductor 2019...Natural Language Processing and Search Intent Understanding C3 Conductor 2019...
Natural Language Processing and Search Intent Understanding C3 Conductor 2019...
 
Using topic modelling frameworks for NLP and semantic search
Using topic modelling frameworks for NLP and semantic searchUsing topic modelling frameworks for NLP and semantic search
Using topic modelling frameworks for NLP and semantic search
 
SEO in a Mobile First World
SEO in a Mobile First WorldSEO in a Mobile First World
SEO in a Mobile First World
 
Modern Ecommerce SEO
Modern Ecommerce SEOModern Ecommerce SEO
Modern Ecommerce SEO
 
Voice Search and Conversation Action Assistive Systems - Challenges & Opportu...
Voice Search and Conversation Action Assistive Systems - Challenges & Opportu...Voice Search and Conversation Action Assistive Systems - Challenges & Opportu...
Voice Search and Conversation Action Assistive Systems - Challenges & Opportu...
 
The Iceberg Approach - Power from what lies beneath in SEO for a mobile-first...
The Iceberg Approach - Power from what lies beneath in SEO for a mobile-first...The Iceberg Approach - Power from what lies beneath in SEO for a mobile-first...
The Iceberg Approach - Power from what lies beneath in SEO for a mobile-first...
 
Voice Search Challenges For Search and Information Retrieval and SEO
Voice Search Challenges For Search and Information Retrieval and SEOVoice Search Challenges For Search and Information Retrieval and SEO
Voice Search Challenges For Search and Information Retrieval and SEO
 
MOZCON 2017 WINNING WITH CHOICE & INFORMATION SYSTEMS FOR BOTH CRAWLERS & CON...
MOZCON 2017 WINNING WITH CHOICE & INFORMATION SYSTEMS FOR BOTH CRAWLERS & CON...MOZCON 2017 WINNING WITH CHOICE & INFORMATION SYSTEMS FOR BOTH CRAWLERS & CON...
MOZCON 2017 WINNING WITH CHOICE & INFORMATION SYSTEMS FOR BOTH CRAWLERS & CON...
 
AMP Accelerated Mobile Pages - The Next Generation SMX London 2017 Dawn Anderson
AMP Accelerated Mobile Pages - The Next Generation SMX London 2017 Dawn AndersonAMP Accelerated Mobile Pages - The Next Generation SMX London 2017 Dawn Anderson
AMP Accelerated Mobile Pages - The Next Generation SMX London 2017 Dawn Anderson
 

Recently uploaded

Advertising and Sales Promotion concepts
Advertising and Sales Promotion  conceptsAdvertising and Sales Promotion  concepts
Advertising and Sales Promotion conceptsparamanjegatheeswari
 
Pay Per Click - PPC Foundation
Pay Per Click - PPC FoundationPay Per Click - PPC Foundation
Pay Per Click - PPC FoundationAdriannaBednarz
 
Weekly Media Update_19_02_2024. This document comprises news clips from vario...
Weekly Media Update_19_02_2024. This document comprises news clips from vario...Weekly Media Update_19_02_2024. This document comprises news clips from vario...
Weekly Media Update_19_02_2024. This document comprises news clips from vario...BalmerLawrie
 
Redseer_Ear-It-All_Report Design 2_16022024_final.pdf
Redseer_Ear-It-All_Report Design 2_16022024_final.pdfRedseer_Ear-It-All_Report Design 2_16022024_final.pdf
Redseer_Ear-It-All_Report Design 2_16022024_final.pdfSocial Samosa
 
ChatGPT : How to use ChatGPT and Get updated Prompts
ChatGPT : How to use ChatGPT and Get updated PromptsChatGPT : How to use ChatGPT and Get updated Prompts
ChatGPT : How to use ChatGPT and Get updated PromptsSaurav Srivastava
 
Viral Vibes in 2024: How to Elevate Your Business' Social Media Marketing
Viral Vibes in 2024: How to Elevate Your Business' Social Media MarketingViral Vibes in 2024: How to Elevate Your Business' Social Media Marketing
Viral Vibes in 2024: How to Elevate Your Business' Social Media MarketingJomer Gregorio
 
Porter's Five Forces Analysis: Starbucks vs. Costa Coffee
Porter's Five Forces Analysis: Starbucks vs. Costa CoffeePorter's Five Forces Analysis: Starbucks vs. Costa Coffee
Porter's Five Forces Analysis: Starbucks vs. Costa Coffeeakshayytanwar123
 
SEO Levers - Small Tweaks That Lead To Big Ranking Wins Across Any Industry.pptx
SEO Levers - Small Tweaks That Lead To Big Ranking Wins Across Any Industry.pptxSEO Levers - Small Tweaks That Lead To Big Ranking Wins Across Any Industry.pptx
SEO Levers - Small Tweaks That Lead To Big Ranking Wins Across Any Industry.pptxPeter Macinkovic
 
Research - Women's Jeans in Colombia (Feb-24).pdf
Research - Women's Jeans in Colombia (Feb-24).pdfResearch - Women's Jeans in Colombia (Feb-24).pdf
Research - Women's Jeans in Colombia (Feb-24).pdfJuan Renteria
 
Advertising concepts , Importance and its Types
Advertising concepts , Importance and its TypesAdvertising concepts , Importance and its Types
Advertising concepts , Importance and its Typesparamanjegatheeswari
 
The Resurgence of Direct Mail as a Growth Marketing Strategy
The Resurgence of Direct Mail as a Growth Marketing StrategyThe Resurgence of Direct Mail as a Growth Marketing Strategy
The Resurgence of Direct Mail as a Growth Marketing StrategyAggregage
 
2024 Insights: Role of Social Media Marketing in Business Growth
2024 Insights: Role of Social Media Marketing in Business Growth2024 Insights: Role of Social Media Marketing in Business Growth
2024 Insights: Role of Social Media Marketing in Business GrowthJomer Gregorio
 
Snapshot of Consumer Behaviors of January 2024-EOLiSurvey (EN).pdf
Snapshot of Consumer Behaviors of January 2024-EOLiSurvey (EN).pdfSnapshot of Consumer Behaviors of January 2024-EOLiSurvey (EN).pdf
Snapshot of Consumer Behaviors of January 2024-EOLiSurvey (EN).pdfEastern Online-iSURVEY
 
Hootsuite Platform Certification Course
Hootsuite Platform Certification CourseHootsuite Platform Certification Course
Hootsuite Platform Certification CourseAdriannaBednarz
 
UAE Latest digital marketing insights of 2024
UAE Latest digital marketing insights of 2024UAE Latest digital marketing insights of 2024
UAE Latest digital marketing insights of 2024medraradigitals
 
Advertising in Music Genre 2023 - TAM AdEx
Advertising in Music Genre 2023 - TAM AdExAdvertising in Music Genre 2023 - TAM AdEx
Advertising in Music Genre 2023 - TAM AdExSocial Samosa
 
SEO In 2024: Top Priorities, Challenges, And Opportunities
SEO In 2024: Top Priorities, Challenges, And OpportunitiesSEO In 2024: Top Priorities, Challenges, And Opportunities
SEO In 2024: Top Priorities, Challenges, And OpportunitiesSearch Engine Journal
 
Marketing and trading on the Internet to obtain significant profits
Marketing and trading on the Internet to obtain significant profitsMarketing and trading on the Internet to obtain significant profits
Marketing and trading on the Internet to obtain significant profitsمعا الى الافضل
 
KSA Latest digital marketing insights of 2024
KSA Latest digital marketing insights of 2024KSA Latest digital marketing insights of 2024
KSA Latest digital marketing insights of 2024medraradigitals
 

Recently uploaded (19)

Advertising and Sales Promotion concepts
Advertising and Sales Promotion  conceptsAdvertising and Sales Promotion  concepts
Advertising and Sales Promotion concepts
 
Pay Per Click - PPC Foundation
Pay Per Click - PPC FoundationPay Per Click - PPC Foundation
Pay Per Click - PPC Foundation
 
Weekly Media Update_19_02_2024. This document comprises news clips from vario...
Weekly Media Update_19_02_2024. This document comprises news clips from vario...Weekly Media Update_19_02_2024. This document comprises news clips from vario...
Weekly Media Update_19_02_2024. This document comprises news clips from vario...
 
Redseer_Ear-It-All_Report Design 2_16022024_final.pdf
Redseer_Ear-It-All_Report Design 2_16022024_final.pdfRedseer_Ear-It-All_Report Design 2_16022024_final.pdf
Redseer_Ear-It-All_Report Design 2_16022024_final.pdf
 
ChatGPT : How to use ChatGPT and Get updated Prompts
ChatGPT : How to use ChatGPT and Get updated PromptsChatGPT : How to use ChatGPT and Get updated Prompts
ChatGPT : How to use ChatGPT and Get updated Prompts
 
Viral Vibes in 2024: How to Elevate Your Business' Social Media Marketing
Viral Vibes in 2024: How to Elevate Your Business' Social Media MarketingViral Vibes in 2024: How to Elevate Your Business' Social Media Marketing
Viral Vibes in 2024: How to Elevate Your Business' Social Media Marketing
 
Porter's Five Forces Analysis: Starbucks vs. Costa Coffee
Porter's Five Forces Analysis: Starbucks vs. Costa CoffeePorter's Five Forces Analysis: Starbucks vs. Costa Coffee
Porter's Five Forces Analysis: Starbucks vs. Costa Coffee
 
SEO Levers - Small Tweaks That Lead To Big Ranking Wins Across Any Industry.pptx
SEO Levers - Small Tweaks That Lead To Big Ranking Wins Across Any Industry.pptxSEO Levers - Small Tweaks That Lead To Big Ranking Wins Across Any Industry.pptx
SEO Levers - Small Tweaks That Lead To Big Ranking Wins Across Any Industry.pptx
 
Research - Women's Jeans in Colombia (Feb-24).pdf
Research - Women's Jeans in Colombia (Feb-24).pdfResearch - Women's Jeans in Colombia (Feb-24).pdf
Research - Women's Jeans in Colombia (Feb-24).pdf
 
Advertising concepts , Importance and its Types
Advertising concepts , Importance and its TypesAdvertising concepts , Importance and its Types
Advertising concepts , Importance and its Types
 
The Resurgence of Direct Mail as a Growth Marketing Strategy
The Resurgence of Direct Mail as a Growth Marketing StrategyThe Resurgence of Direct Mail as a Growth Marketing Strategy
The Resurgence of Direct Mail as a Growth Marketing Strategy
 
2024 Insights: Role of Social Media Marketing in Business Growth
2024 Insights: Role of Social Media Marketing in Business Growth2024 Insights: Role of Social Media Marketing in Business Growth
2024 Insights: Role of Social Media Marketing in Business Growth
 
Snapshot of Consumer Behaviors of January 2024-EOLiSurvey (EN).pdf
Snapshot of Consumer Behaviors of January 2024-EOLiSurvey (EN).pdfSnapshot of Consumer Behaviors of January 2024-EOLiSurvey (EN).pdf
Snapshot of Consumer Behaviors of January 2024-EOLiSurvey (EN).pdf
 
Hootsuite Platform Certification Course
Hootsuite Platform Certification CourseHootsuite Platform Certification Course
Hootsuite Platform Certification Course
 
UAE Latest digital marketing insights of 2024
UAE Latest digital marketing insights of 2024UAE Latest digital marketing insights of 2024
UAE Latest digital marketing insights of 2024
 
Advertising in Music Genre 2023 - TAM AdEx
Advertising in Music Genre 2023 - TAM AdExAdvertising in Music Genre 2023 - TAM AdEx
Advertising in Music Genre 2023 - TAM AdEx
 
SEO In 2024: Top Priorities, Challenges, And Opportunities
SEO In 2024: Top Priorities, Challenges, And OpportunitiesSEO In 2024: Top Priorities, Challenges, And Opportunities
SEO In 2024: Top Priorities, Challenges, And Opportunities
 
Marketing and trading on the Internet to obtain significant profits
Marketing and trading on the Internet to obtain significant profitsMarketing and trading on the Internet to obtain significant profits
Marketing and trading on the Internet to obtain significant profits
 
KSA Latest digital marketing insights of 2024
KSA Latest digital marketing insights of 2024KSA Latest digital marketing insights of 2024
KSA Latest digital marketing insights of 2024
 

Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017

  • 1. @dawnieando #BigDigitalADL Generational  Cruft  &  Technical  Debt  in   SEOGONE  IS   NEVER  GONE Dawn  Anderson  @  dawnieando
  • 2. @dawnieando #BigDigitalADL A New Beginning § “A  new  website  will  solve  ALL  our   problems” § “Let’s  start  again” § “We’ll  just  migrate…  and  redirect   everything”
  • 3. @dawnieando #BigDigitalADL 404 Page Not Found § “Of  course,  we  won’t   redirect  everything…” § “Not  everything  will   be  worth  redirecting”
  • 4. @dawnieando #BigDigitalADL 410 Gone § “Some,  we’ll  just  kill   off  with  a  410…” § “Then  the  URLs  will   be  gone”
  • 5. @dawnieando #BigDigitalADL But in Reality…it Still Exists
  • 6. @dawnieando #BigDigitalADL Because…Web Crawler System’s History Logs SEARCH  ENGINES  HAVE  A    BIG  MEMORY  AND  A  LOT  OF   STORAGE
  • 8. @dawnieando #BigDigitalADL Web Crawler System GOOGLE  NEVER  FORGETS The  history  logs  play  a   role  in  deciding  when   every  URL  that  was  EVER   discovered  gets  visited   again
  • 9. @dawnieando #BigDigitalADL History Log Records Include: • URL  fingerprint • Timestamp  (last  crawl  or  download   attempt) • Crawl  status  (success  or  error)  (Response   code) • Content  checksum  (binary  code) • Source  ID  (accessed  from  cache  or   downloaded) • Segment  identifier  (Crawl  segment  assigned   to??) • Page  importance  (a  measure  of  importance   assigned  to  the  URL) May  be   calculated  by   identifying   historical   importance   scores  based  on   past  X  number  of   crawls
  • 10. @dawnieando #BigDigitalADL Gone Is Never Gone “We  knew  there  was  content   there  at  some  point  so  we   just  swing  by  every  now  and   then  to  see  if  anything  came   back”  (John  Mueller,  2016)
  • 12. @dawnieando #BigDigitalADL The Generational ’Snail Trail’ • Old  XML  sitemaps • Redirects  drop  away  on  old  site   .htaccess • DNS  issues • People  link  to  old  site  but  wrong   protocol • Old  sites  not  verified  in  GSC • Not  all  protocols  redirecting Leaving  it’s   slithery     footprint
  • 13. @dawnieando #BigDigitalADL The Generational ’Snail Trail’ • All  eating  away  at   Googlebot’s attention  on   your  server’s  IP WATCH  OUT  FOR  THE  SNAIL  TRAIL  &   GENERATIONAL  CRUFT
  • 14. @dawnieando #BigDigitalADL The Slow Page Evolution of Near Duplicates In  a  study  over  11  weeks  Denis  Fetterly and   Mark  Najork found  that  near-­‐duplicate  pages   rarely  change  and  that  they  are  still  near-­‐ duplicates  of  each  other  10  weeks  later.   Therefore  once  identified  their  download   priority  may  be  reduced  so  that  resources  may   be  used  more  efficiently  /  productively   elsewhere (Fetterly &  Najork,  2003) Fetterly &   Najork,   2003
  • 15. @dawnieando #BigDigitalADL ‘Transitive’?? Transitive  -­‐ A  ==  B  +  B  ==  C  then  A  ==  C THEORY:  Maybe  for  some  types  of   content  more  than  others  – e.g.   ecommerce/directories  but  not  news THEORY  ALERT  !!!!!!!
  • 16. @dawnieando #BigDigitalADL DUSTBUSTER & DUST CRAWLING RULES DO  NOT   CRAWL  IN   THE  DUST BUILDS   ‘HINTS’  ON   WHAT  NOT   TO  CRAWL
  • 17. @dawnieando #BigDigitalADL ‘Sampling’in Crawling for Efficiency ‘SMALL  TEST  VISITS  TO  A  SITE  TO   UNDERSTAND  WHETHER  IT  IS  WORTH   CRAWLING’
  • 18. @dawnieando #BigDigitalADL Every Site Will Have Its Own Crawling Rules UNSURE  AS  TO  WHETHER   THIS  IS  BEING  USED DUSTBUSTER   CRAWLING RULES
  • 19. @dawnieando #BigDigitalADL Popular CMS ’Rule Patterns’ (URL Parameters) ALL  WILL  HAVE  COMMON   CANONICALIZATION  PATTERNS  WHICH   CAN  BE  LEARNED
  • 20. @dawnieando #BigDigitalADL CRAWLING RULES BUILT OVER TIME Crawl  Frequency  Patterns No  two  sites  will  have  the  same  crawl   schedules  or  rules  built Moving  from  one  CMS  to  another  may   mean  that  different  parameters  are   created.    New  parameters  =  new  rules  
  • 21. @dawnieando #BigDigitalADL Every Version of Your Past Ecommerce Sites “Exponentially   multiplicative   URLs” Had  potential  to  spew…  at  some  point…
  • 22. @dawnieando #BigDigitalADL URLs Take Their Place in Crawling Queues The  Queue  Gets  Long  &   Congested
  • 23. @dawnieando #BigDigitalADL YOU INHERITED SEO TECHNICAL DEBT • Previous  content  /  link  manual  actions • Previous  algorithmic  suppressions • Past  infinite  loops • “We’ll  SEO  it  after  launch” • “SEO  is  dead…  so  we  won’t  optimise” • Dodgy  URL  parameters • Misconfigured  URL  parameters • Old  URL  crawling  ‘rules  /  hints’
  • 25. @dawnieando #BigDigitalADL IT WASN’T ME – PASSING THE BUCK
  • 26. @dawnieando #BigDigitalADL … with Interest AT  SOME  POINT  IT   MUST  BE  REPAID
  • 27. @dawnieando #BigDigitalADL GENERATIONAL CRUFT EVERY  SINGLE  TIME  YOU  MIGRATE,  CHANGE  DESIGN,  REDIRECT,  REINVENT  A  SITE  /  URL A  CLEAN  START REDIRECTIONS ANOTHER  STRUCTURE FIRST  SITE   STRUCTURE NEW  CRAWLING  ‘RULES’   BUILT CRAWLING   ‘RULES’  BUILT EVERYTHING   IS  ‘200  OK’ MORE  URLs MIXED  RESPONSE  CODES REDIRECTIONS ‘FUZZINESS’  IS  EMERGING NEW  CRAWLING  ‘RULES’  BUILT MORE  URLs REDIRECT  CHAINS  &  MIXED   RESPONSE  CODES NEW  SEO’s  DON’T   KNOW  THE  ‘HISTORY’ TARGET  URLs  NOW  ‘VERY  FUZZY’
  • 28. @dawnieando #BigDigitalADL Aged ‘Patchwork Quilt’Sites A  LITTLE  BIT  OF  THIS  CMS  AND  A   LITTLE  BIT  OF  THAT  CMS MANY  HISTORICAL  PARAMETERS   CREATED
  • 29. @dawnieando #BigDigitalADL ’Fuzzy’ URL Targets with Each Site Generation EVERYTHING  GETS   A  BIT  BLURRED ‘Which  is  the  target  URL   again?
  • 30. @dawnieando #BigDigitalADL Time Seems To Fly… The Older You Get Your  new  site  URL  is  just   one  of  very  many  historical   URLs  on  your  IP  to  be   visited  periodically A  tiny  fish  in  a  very   big  URL  pond  queue
  • 31. @dawnieando #BigDigitalADL The Great 302s Pass PageRank Debate
  • 32. @dawnieando #BigDigitalADL SOLUTION - THE BELOVED CANONICAL § 30X  redirects § Canonical  tag § Href lang § HTTPS  protocol § Global  canonicalization   rules In  ’ALL’  its  forms
  • 34. @dawnieando #BigDigitalADL Advanced Technical SEO? 50%  of  SEOs  surveyed   considered;; “CANONICALIZATION   IS  ADVANCED   TECHNICAL  SEO”
  • 35. @dawnieando #BigDigitalADL Oh Yeah – Canonicalization is Easy 76%  of  SEOs  surveyed   considered;; “CANONICALIZATION   IS  AN  EASY  CONCEPT   TO  UNDERSTAND”
  • 36. @dawnieando #BigDigitalADL REL NEXT REL PREV is NOT Canonicalization 47%  of  SEOs   categorizing  themselves   as  ‘TECHNICAL  SEO’s   considered;; “REL=NEXT  /  REL  =   PREV”  IS  A  FORM  OF   CANONICALIZATION
  • 37. @dawnieando #BigDigitalADL On Redirections as Canonicalization Forms Lots  were  unsure  about;; “301s  and  302s  are   BOTH  forms  of   canonicalization”
  • 38. @dawnieando #BigDigitalADL On Href Lang as Canonicalization Only  64%  of  ’Technical SEOs’  thought  HRef Lang  was  a  form  of Canonicalization IT  IS
  • 39. @dawnieando #BigDigitalADL URL Parameter Handling is Your Friend Help  Google  Build  ‘Crawling   Rules’  for  your  site  rather   than  wasting  time  on   ‘sampling’  and  giving  a  bad   impression GIVE  HELP  AND   GUIDANCE  WITH  THE   CRAWL  RULE  AND   HINT  BUILDING
  • 40. @dawnieando #BigDigitalADL SOLUTION - Understand URL Parameters ACTIVE  PARAMETERS  ==  CHANGE  THE  CONTENT  ON   YOUR  PAGE (e.g.  sort,  filter,  translate,  paginate,  specify) PASSIVE  PARAMETERS  ==  DO  NOT  CHANGE  THE   CONTENT  ON  YOUR  PAGE (Often  used  for  tracking)    (ALIAS:  REPRESENTATIVE)
  • 41. @dawnieando #BigDigitalADL ACTIVE Parameters (CHANGE CONTENT) SORT  ==  Sorts  dynamic  items  and  reorders  in  descending  /  ascending  price  /   popularity  /  added NARROWING  ==  Filters  dynamically  added  items  down  to  include  only   features  &  attributes  in  a  chosen  consideration  set SPECIFYING  ==  Identifies  a  particular  dynamically  variable  populated   content  set  within  a  site  section  (e.g.  store=women) TRANSLATING  ==  Indicates  a  language  driven  translation  URL  (e.g.  lang=fr) PAGINATING  ==  Indicates  a  paginated  display  of  long  content  (e.g.  page=2)
  • 42. @dawnieando #BigDigitalADL Understand How URLs with Multiple Parameters Are Handled The  most  restrictive  parameter  blocked  overrules   lesser  restrictions
  • 43. @dawnieando #BigDigitalADL Examples of Multiple Parameter Handling KNOW  THE   RULES http://www.example.com?shopping-­‐category=DVD-­‐movies&sort-­‐ by=production-­‐year&sort-­‐order=asc WILL  BE  CRAWLED http://www.example.com?shopping-­‐category=shoes&sort-­‐by=size&sort-­‐ order=asc WILL  NOT  BE  CRAWLED  (production-­‐year  blocks)
  • 44. @dawnieando #BigDigitalADL Help Googlebot Get Round its Shopping List OPEN  MORE  CHECKOUTS WIDEN  THE  AISLES MAKE  THINGS  EASY  TO  FIND DON’T  CONFUSE   GOOGLEBOT HELP  FILL  THE  TROLLEY   QUICKLY SPEED,  SPEED,  SPEED
  • 45. @dawnieando #BigDigitalADL SOLUTION - XML SitemapsAre Your Friend… (Strong Foundations) They  help  to   pass   ‘importance’   signals  within   a  site But…  never   leave  them  to   just   autogenerate without   periodically   checking ‘The   foundations’   underneath  a   site
  • 46. @dawnieando #BigDigitalADL Validate & Retain in GSC ALL Past Domains & Past Site Versions (Protocols (HTTPS / HTTP) THERE  MAY  STILL  BE  UNDETECTED  ACTIVITY  GOING  ON  THERE
  • 47. @dawnieando #BigDigitalADL Server Log FileAnalysis is Your Friend… You’ll  be  surprised  by  what  you  find Find  out  what  Googlebot is   visiting  and  when  (how   often)  and  whether  it   should  be  visiting  it  at  all
  • 48. @dawnieando #BigDigitalADL SOLUTION - Save & Grow The URLs Not  EVERYTHING  is   worthy  of  its  own  URL VARIANTS STEMMINGS PLURALS RANDOM  TAGS LONG,  LONG,  LONG   TAIL  PARAMETERS
  • 49. @dawnieando #BigDigitalADL SOLUTION - Save & Grow The URLs A  URL  is  like  a   fine  wine Maturing  over   time
  • 50. @dawnieando #BigDigitalADL Pass Strong Clues - Highly Relevant New Structure STRONG SEMANTICS
  • 51. @dawnieando #BigDigitalADL Wiki Page Redirects https://dbpedia.org/sparql Wikipedia   Redirects thesaurus.com OR  A  GOOD  OLD  FASHIONED  THESAURUS
  • 52. @dawnieando #BigDigitalADL Increase ‘Importance’ quickly of target URLs • Internal  link  optimization • Canonicalise to  (if  relevant) • Strengthen  up  importance  signals • Inclusion  in  front  facing  and  XML   sitemaps • Improve  the  content  &  keep  it   updated • 301  redirect  to  (if  relevant   redundant  content)
  • 53. @dawnieando #BigDigitalADL Reduce ‘Importance’ quickly of old URLs • Internal  link  unoptimization • 410 • Dig  out  URLs  with  links  to  them • Orphan  URLs • Canonicals  to  HTTPs • Exclusion  from  XML  sitemaps  (even   old  ones  on  the  server) • Strip  out  content
  • 54. @dawnieando #BigDigitalADL 304 IF MODIFIED HEADERS ONLY  DOWNLOAD  IF  THE   CONTENT  CHECKSUM  HAS   CHANGED
  • 56. @dawnieando #BigDigitalADL SOLUTION – Be Careful About Creating New Dynamic Parameters QUEUEING…  AGAIN Waiting  for  good  URLs  to  be   visited…  AGAIN
  • 57. @dawnieando #BigDigitalADL REVISIT PAST .HTACCESS FILES Can  you  rewrite  the  rules  to  be  more   efficient  or  cut  out  some  old  rules  still   firing  unnecessarily?
  • 58. @dawnieando #BigDigitalADL SOLUTION - NEVER TRY TO ‘OUTRUN’ GOOGLEBOT
  • 59. @dawnieando #BigDigitalADL You Have a Shiny New Site… So What? You  may  still  have… GENERATIONAL  CRUFT  &  TECHNICAL   DEBT  TO  PAY  OFF GONE  IS  NEVER  GONE
  • 60. @dawnieando #BigDigitalADL REMEMBER ”Gone  is   Never  Gone” “Google  Has  A   Big  Memory” Dawn  Anderson  @  dawnieando THANK  YOU
  • 61. @dawnieando #BigDigitalADL Sources & References • https://patentimages.storage.googleapis.com/US8042112B1/US08042112-­‐ 20111018-­‐D00000.png • Randall,  K.H.,  Google  Inc.,  2010. Scheduler  for  search  engine  crawler.  U.S.  Patent   7,725,452.