Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Indexing Your Content is Optional (2020 Edition)

123 views

Published on

Imagine having a big brand website with over 50% of its content not indexed by Google after weeks? Now imagine that the issue causing this severe decline in online visibility was self-induced? In the new digital marketing landscape, big brands losing money this way is shockingly normal.

This deck from a Searchmetrics webinar Bartosz Goralewicz did with Marcus Tober on December 4, 2019.

Want to watch the video? https://youtu.be/3XENVxVpyoo

Published in: Marketing
  • Be the first to comment

  • Be the first to like this

Indexing Your Content is Optional (2020 Edition)

  1. 1. Bartosz Góralewicz linkedin.com/in/goralewicz @bart_goralewicz Indexing Your Content is Optional 2020 Edition
  2. 2. 63% of Walmart’s pages are not indexed in Google. H&M is losing 4,938,885 visits per month due to their indexing issues. 24% of Topshop products are not indexed in Google. Now that I have your full attention – let’s begin J
  3. 3. Back in 2016 I published Hulu’s case study… … that I wrote in SearchMetrics’ office in Berlin J one.ly/hulu2016
  4. 4. In 2016 JavaScript was… exotic?
  5. 5. Hulu vs. Netflix – the first JavaScript SEO battle
  6. 6. Fast forward to 2019
  7. 7. HTML HTMLHTML There is no such thing as an HTML website!
  8. 8. In 2019 * This means that their content isn’t visible with JavaScript switched off. ** this also means that Google needs to render the page to see that content. Rendering is optional though. of the websites use JavaScript for their crucial content.*80%
  9. 9. Brands with content relying on JavaScript
  10. 10. 82% 0% 0% 42% 73% 0% 22% Percentage of JavaScript content indexed
  11. 11. Let’s have a look at the indexing issue in the wild
  12. 12. case study
  13. 13. Let’s compare H&M’s crawl with and without JS rendering
  14. 14. Let’s compare H&M’s crawl with and without JS rendering
  15. 15. Read more here: one.ly/hm
  16. 16. What usually relies on JavaScript? 01 Pagination 02 “You might be also interested in” (internal linking) 03 Top products 04 Reviews 05 Comments 06 Main content (rarely)
  17. 17. The Disconnect Google’s announcements this year didn’t help… Statement: Martin and John are nothing but helpful. Google is a large organization though and sometimes they make our life a little bit too complex.
  18. 18. But… but 90% of the websites get indexed in 5 seconds?
  19. 19. one.ly/martin5s
  20. 20. What is the exact timeframe for JavaScript indexing? Unfortunately - NO. 5 seconds for 90% of the pages?
  21. 21. Yeaaaah, my JS content can get indexed in 5 seconds. Well, not exactly.
  22. 22. Today’s episode was sponsored by the phrase: Queue latency
  23. 23. What Martin actually meant
  24. 24. Ranking
  25. 25. There is no timeframe for JavaScript indexing!
  26. 26. JS indexing delays are still an issue Good Bad
  27. 27. It is 2019 and our research shows that there are hundreds of thousands of domains not fully indexed. Even after… Months of publishing the content
  28. 28. Let’s get technical…
  29. 29. Web development evolved. Most websites use ready-to-go components Slider Faceted navigation Related Products Comments Main content Menu The roots of the problem
  30. 30. Slider Faceted navigation Related Products Comments Main content Menu So when Google isn’t rendering your website, it may see…well, this:
  31. 31. 2 completely different websites and 2 different website graphs. Rendering delays lead to
  32. 32. Category Ladies/Men/DividedCategory Category H&M.comHOMEPAGE Sub-Category Sub-Category Sub-Category Jeans Products Products Products Products Shaping Skinny High Jeans JavaScript
  33. 33. Category Ladies/Men/DividedCategory Category H&M.comHOMEPAGE Sub-Category Sub-Category Sub-Category Jeans Products Products Products Products Shaping Skinny High Jeans JavaScript
  34. 34. Let’s compare H&M’s crawl with and without JS rendering
  35. 35. Let’s compare H&M’s crawl with and without JS rendering
  36. 36. Let’s talk rendering…
  37. 37. Rendering is not Google’s indexer default mode. Rendering is triggered by... Heuristics
  38. 38. Rendering heuristics So, you look at the difference between the initial HTML, and, then, if after rendering you see extra content? Yeah.
  39. 39. There are certain heuristics, that, if we see after a while, like, oh, this page, actually, the renderer does not diff as much or doesn’t diff, it looks the way before The Heuristics I still haven’t fully, like, grasped what exactly triggers the heuristics
  40. 40. are FAR from perfect though. …those heuristics
  41. 41. Every website, when we see them for the first time, goes to rendering. ALL NEW SITES GET RENDERED
  42. 42. But what is a new website exactly? What is a new website?
  43. 43. Rendering Heuristics Logic behind those heuristics Limitations Changes in how we should approach web development • Staging • Experiments
  44. 44. Rendering vs. Crawler budget
  45. 45. Rendering vs. CPU cost
  46. 46. WRONG
  47. 47. CPU is not affecting rendering* *within what we would call a “reasonable” load
  48. 48. Is the JavaScript rendering cost affecting crawling and indexing? It’s complicated
  49. 49. How to check the rendering cost of a page TL;DR – Too Long Didn’t Render onely.com/tools
  50. 50. TL;DR – Too Long; Didn’t Render onely.com/tools
  51. 51. onely.com/tools
  52. 52. onely.com/tools
  53. 53. Content indexing - timeframe
  54. 54. Percentage of JavaScript content indexed after 14 days 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 100% 96% 70% 34% 0,5% New York Post Eventbrite Target The Guardian CNBC
  55. 55. TGIF - The Google Indexing Forecast onely.com/tools/tgif
  56. 56. HTML vs. JavaScript indexing
  57. 57. HTML vs. JavaScript indexing
  58. 58. Diagnosing rendering issues
  59. 59. WWJD - What Would JavaScript Do
  60. 60. WWJD – BBC & meta tags vs. rendering
  61. 61. WWJD – BBC & meta tags vs. rendering
  62. 62. WWJD – BBC & meta tags vs. rendering
  63. 63. WWJD – BBC & meta tags vs. rendering
  64. 64. WWJD – Links added by JS
  65. 65. WWJD – Links removed by JS
  66. 66. JSLet's talk about HTML
  67. 67. Even without JavaScript dependencies big players struggle with indexing 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Indexed after 1 day Indexed after 2 days Indexed after 1 week Indexed after 2 weeks The Guardian Target Reuters Eventbrite 98.5% 3.5% 68.5% 32% 99.9% 33% 88% 47% 99.9% 78% 91% 57% 99.9% 84% 91% 58%
  68. 68. Wrapping up
  69. 69. Parts of your website depend on JS 01 Webmaster updates with new products 02 Google crawls your website without seeing all the links 03 Only crawls a part of the domain without finding products (valuable content) 05 Budget is too low to render JS 04 Google indexer is confused - crawler budget falls
  70. 70. Which leads to this… 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Indexed after 1 day Indexed after 2 days Indexed after 1 week Indexed after 2 weeks The Guardian Target Reuters Eventbrite 98.5% 3.5% 68.5% 32% 99.9% 33% 88% 47% 99.9% 78% 91% 57% 99.9% 84% 91% 58%
  71. 71. TO DO: Go to onely.com/tools and check your website (100% free) Crawl your website with and without JS and compare two crawls
  72. 72. We are going like: ”Oh, all right, we are gonna skip rendering.” It is not as frequently happening anymore. (...) Many websites even if they do not run JavaScript, they might still go through the render phase, because it doesn’t make a difference as much Crawling and indexing to come together I expect, eventually rendering, crawling and indexing will come closer together
  73. 73. Summary Indexing your content is one of the key challenges for 2020 All indexing issues we are seeing are 100% self-induced by website owners We need to redefine how we are looking at the crawler budget and the factors affecting it JavaScript SEO = Technical SEO
  74. 74. To do: 1Closely monitor your indexing stats (daily, weekly, biweekly). 2 If your content relies on JavaScript, monitor it closely and make sure that: • The code is light and efficient (cheap to render) • Your code doesn’t affect web performance on low-end devices 3 Make sure that no content within your structure relies on JavaScript (check all the page types within your domain). 4 Make sure that no meta data is changed during rendering.
  75. 75. To do: 1Closely monitor your indexing stats (daily, weekly, biweekly). 2 If your content relies on JavaScript, monitor it closely and make sure that: • The code is light and efficient (cheap to render) • Your code doesn’t affect web performance on low-end devices 3 Make sure that no content within your structure relies on JavaScript (check all the page types within your domain). 4 Make sure that no meta data is changed during rendering. Subscribe to Onely.com – be the first one to know about our crawl budget monitoring toolset.
  76. 76. More data is coming soon
  77. 77. Cloaking in 2019 bartosz@onely.com linkedin.com/in/goralewicz @bart_goralewicz THANK YOU one.ly/15min

×