Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Bartosz Góralewicz
linkedin.com/in/goralewicz @bart_goralewicz
Indexing Your
Content is Optional
2020 Edition
63% of Walmart’s pages are not indexed in Google.
H&M is losing 4,938,885 visits per month due to their indexing issues.
2...
Back in 2016 I published Hulu’s case study…
… that I wrote in SearchMetrics’ office in Berlin J
one.ly/hulu2016
In 2016 JavaScript
was… exotic?
Hulu vs. Netflix – the first JavaScript SEO battle
Fast forward to 2019
HTML HTMLHTML
There is no such thing
as an HTML website!
In 2019
* This means that their content isn’t
visible with JavaScript switched off.
** this also means that Google
needs t...
Brands with content relying
on JavaScript
82%
0%
0%
42%
73%
0%
22%
Percentage
of
JavaScript
content
indexed
Let’s have a look
at the indexing issue
in the wild
case study
Let’s compare H&M’s crawl
with and without JS rendering
Let’s compare H&M’s crawl
with and without JS rendering
Read more here:
one.ly/hm
What usually relies on JavaScript?
01
Pagination
02
“You might be also
interested in” (internal linking)
03
Top products
0...
The Disconnect
Google’s announcements this year didn’t help…
Statement:
Martin and John are nothing but helpful.
Google is...
But… but 90% of the websites get
indexed in 5 seconds?
one.ly/martin5s
What is the exact timeframe for
JavaScript indexing?
Unfortunately - NO.
5 seconds
for 90% of
the pages?
Yeaaaah, my JS content can get indexed in 5 seconds.
Well, not exactly.
Today’s episode was sponsored by the phrase:
Queue latency
What Martin actually meant
Ranking
There is no timeframe for
JavaScript indexing!
JS indexing delays are still an issue
Good Bad
It is 2019 and our
research shows that
there are hundreds
of thousands of
domains not fully
indexed.
Even after… Months of...
Let’s get technical…
Web development evolved. Most websites use ready-to-go components
Slider
Faceted navigation
Related Products
Comments
Main...
Slider
Faceted navigation
Related Products
Comments
Main content
Menu
So when Google isn’t rendering your website,
it may ...
2 completely different websites
and 2 different website graphs.
Rendering delays lead to
Category Ladies/Men/DividedCategory Category
H&M.comHOMEPAGE
Sub-Category Sub-Category Sub-Category Jeans
Products Product...
Category Ladies/Men/DividedCategory Category
H&M.comHOMEPAGE
Sub-Category Sub-Category Sub-Category Jeans
Products Product...
Let’s compare H&M’s crawl
with and without JS rendering
Let’s compare H&M’s crawl
with and without JS rendering
Let’s talk rendering…
Rendering is not Google’s
indexer default mode.
Rendering is triggered by... Heuristics
Rendering heuristics
So, you look at the difference
between the initial HTML, and,
then, if after rendering you
see extra ...
There are certain heuristics, that,
if we see after a while, like, oh, this
page, actually, the renderer does not
diff as ...
are FAR from
perfect though.
…those heuristics
Every website, when we see
them for the first time, goes
to rendering.
ALL NEW SITES
GET RENDERED
But what is
a new website exactly?
What is a new website?
Rendering Heuristics
Logic behind
those heuristics
Limitations Changes in how we should
approach web development
• Staging...
Rendering vs. Crawler budget
Rendering vs. CPU cost
WRONG
CPU is not
affecting
rendering*
*within what we would call
a “reasonable” load
Is the JavaScript rendering cost
affecting crawling and indexing?
It’s complicated
How to check the rendering
cost of a page
TL;DR – Too Long Didn’t Render
onely.com/tools
TL;DR – Too Long; Didn’t Render
onely.com/tools
onely.com/tools
onely.com/tools
Content indexing
- timeframe
Percentage of
JavaScript
content
indexed
after 14 days
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
100%
96%
70%
34%
0,5%
N...
TGIF - The Google Indexing Forecast
onely.com/tools/tgif
HTML vs. JavaScript indexing
HTML vs. JavaScript indexing
Diagnosing
rendering issues
WWJD - What Would JavaScript Do
WWJD – BBC & meta tags vs. rendering
WWJD – BBC & meta tags vs. rendering
WWJD – BBC & meta tags vs. rendering
WWJD – BBC & meta tags vs. rendering
WWJD – Links added by JS
WWJD – Links removed by JS
JSLet's talk about
HTML
Even without JavaScript dependencies
big players struggle with indexing
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Indexe...
Wrapping up
Parts of
your website
depend
on JS
01
Webmaster updates
with new products
02
Google crawls your
website without
seeing all...
Which leads to this…
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Indexed after 1 day Indexed after 2 days Indexed after 1 ...
TO DO:
Go to onely.com/tools
and check your website
(100% free)
Crawl your website with
and without JS and
compare two cra...
We are going like: ”Oh, all right, we
are gonna skip rendering.” It is not
as frequently happening anymore.
(...) Many web...
Summary
Indexing your content
is one of the key
challenges for 2020
All indexing issues
we are seeing are
100% self-induce...
To do:
1Closely monitor your
indexing stats (daily,
weekly, biweekly).
2
If your content relies on
JavaScript, monitor it ...
To do:
1Closely monitor your
indexing stats (daily,
weekly, biweekly).
2
If your content relies on
JavaScript, monitor it ...
More data is coming soon
Cloaking in 2019
bartosz@onely.com
linkedin.com/in/goralewicz @bart_goralewicz
THANK YOU
one.ly/15min
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
Indexing Your Content is Optional (2020 Edition)
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Indexing Your Content is Optional (2020 Edition)

Download to read offline

Imagine having a big brand website with over 50% of its content not indexed by Google after weeks? Now imagine that the issue causing this severe decline in online visibility was self-induced? In the new digital marketing landscape, big brands losing money this way is shockingly normal.

This deck from a Searchmetrics webinar Bartosz Goralewicz did with Marcus Tober on December 4, 2019.

Want to watch the video? https://youtu.be/3XENVxVpyoo

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Indexing Your Content is Optional (2020 Edition)

  1. 1. Bartosz Góralewicz linkedin.com/in/goralewicz @bart_goralewicz Indexing Your Content is Optional 2020 Edition
  2. 2. 63% of Walmart’s pages are not indexed in Google. H&M is losing 4,938,885 visits per month due to their indexing issues. 24% of Topshop products are not indexed in Google. Now that I have your full attention – let’s begin J
  3. 3. Back in 2016 I published Hulu’s case study… … that I wrote in SearchMetrics’ office in Berlin J one.ly/hulu2016
  4. 4. In 2016 JavaScript was… exotic?
  5. 5. Hulu vs. Netflix – the first JavaScript SEO battle
  6. 6. Fast forward to 2019
  7. 7. HTML HTMLHTML There is no such thing as an HTML website!
  8. 8. In 2019 * This means that their content isn’t visible with JavaScript switched off. ** this also means that Google needs to render the page to see that content. Rendering is optional though. of the websites use JavaScript for their crucial content.*80%
  9. 9. Brands with content relying on JavaScript
  10. 10. 82% 0% 0% 42% 73% 0% 22% Percentage of JavaScript content indexed
  11. 11. Let’s have a look at the indexing issue in the wild
  12. 12. case study
  13. 13. Let’s compare H&M’s crawl with and without JS rendering
  14. 14. Let’s compare H&M’s crawl with and without JS rendering
  15. 15. Read more here: one.ly/hm
  16. 16. What usually relies on JavaScript? 01 Pagination 02 “You might be also interested in” (internal linking) 03 Top products 04 Reviews 05 Comments 06 Main content (rarely)
  17. 17. The Disconnect Google’s announcements this year didn’t help… Statement: Martin and John are nothing but helpful. Google is a large organization though and sometimes they make our life a little bit too complex.
  18. 18. But… but 90% of the websites get indexed in 5 seconds?
  19. 19. one.ly/martin5s
  20. 20. What is the exact timeframe for JavaScript indexing? Unfortunately - NO. 5 seconds for 90% of the pages?
  21. 21. Yeaaaah, my JS content can get indexed in 5 seconds. Well, not exactly.
  22. 22. Today’s episode was sponsored by the phrase: Queue latency
  23. 23. What Martin actually meant
  24. 24. Ranking
  25. 25. There is no timeframe for JavaScript indexing!
  26. 26. JS indexing delays are still an issue Good Bad
  27. 27. It is 2019 and our research shows that there are hundreds of thousands of domains not fully indexed. Even after… Months of publishing the content
  28. 28. Let’s get technical…
  29. 29. Web development evolved. Most websites use ready-to-go components Slider Faceted navigation Related Products Comments Main content Menu The roots of the problem
  30. 30. Slider Faceted navigation Related Products Comments Main content Menu So when Google isn’t rendering your website, it may see…well, this:
  31. 31. 2 completely different websites and 2 different website graphs. Rendering delays lead to
  32. 32. Category Ladies/Men/DividedCategory Category H&M.comHOMEPAGE Sub-Category Sub-Category Sub-Category Jeans Products Products Products Products Shaping Skinny High Jeans JavaScript
  33. 33. Category Ladies/Men/DividedCategory Category H&M.comHOMEPAGE Sub-Category Sub-Category Sub-Category Jeans Products Products Products Products Shaping Skinny High Jeans JavaScript
  34. 34. Let’s compare H&M’s crawl with and without JS rendering
  35. 35. Let’s compare H&M’s crawl with and without JS rendering
  36. 36. Let’s talk rendering…
  37. 37. Rendering is not Google’s indexer default mode. Rendering is triggered by... Heuristics
  38. 38. Rendering heuristics So, you look at the difference between the initial HTML, and, then, if after rendering you see extra content? Yeah.
  39. 39. There are certain heuristics, that, if we see after a while, like, oh, this page, actually, the renderer does not diff as much or doesn’t diff, it looks the way before The Heuristics I still haven’t fully, like, grasped what exactly triggers the heuristics
  40. 40. are FAR from perfect though. …those heuristics
  41. 41. Every website, when we see them for the first time, goes to rendering. ALL NEW SITES GET RENDERED
  42. 42. But what is a new website exactly? What is a new website?
  43. 43. Rendering Heuristics Logic behind those heuristics Limitations Changes in how we should approach web development • Staging • Experiments
  44. 44. Rendering vs. Crawler budget
  45. 45. Rendering vs. CPU cost
  46. 46. WRONG
  47. 47. CPU is not affecting rendering* *within what we would call a “reasonable” load
  48. 48. Is the JavaScript rendering cost affecting crawling and indexing? It’s complicated
  49. 49. How to check the rendering cost of a page TL;DR – Too Long Didn’t Render onely.com/tools
  50. 50. TL;DR – Too Long; Didn’t Render onely.com/tools
  51. 51. onely.com/tools
  52. 52. onely.com/tools
  53. 53. Content indexing - timeframe
  54. 54. Percentage of JavaScript content indexed after 14 days 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 100% 96% 70% 34% 0,5% New York Post Eventbrite Target The Guardian CNBC
  55. 55. TGIF - The Google Indexing Forecast onely.com/tools/tgif
  56. 56. HTML vs. JavaScript indexing
  57. 57. HTML vs. JavaScript indexing
  58. 58. Diagnosing rendering issues
  59. 59. WWJD - What Would JavaScript Do
  60. 60. WWJD – BBC & meta tags vs. rendering
  61. 61. WWJD – BBC & meta tags vs. rendering
  62. 62. WWJD – BBC & meta tags vs. rendering
  63. 63. WWJD – BBC & meta tags vs. rendering
  64. 64. WWJD – Links added by JS
  65. 65. WWJD – Links removed by JS
  66. 66. JSLet's talk about HTML
  67. 67. Even without JavaScript dependencies big players struggle with indexing 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Indexed after 1 day Indexed after 2 days Indexed after 1 week Indexed after 2 weeks The Guardian Target Reuters Eventbrite 98.5% 3.5% 68.5% 32% 99.9% 33% 88% 47% 99.9% 78% 91% 57% 99.9% 84% 91% 58%
  68. 68. Wrapping up
  69. 69. Parts of your website depend on JS 01 Webmaster updates with new products 02 Google crawls your website without seeing all the links 03 Only crawls a part of the domain without finding products (valuable content) 05 Budget is too low to render JS 04 Google indexer is confused - crawler budget falls
  70. 70. Which leads to this… 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Indexed after 1 day Indexed after 2 days Indexed after 1 week Indexed after 2 weeks The Guardian Target Reuters Eventbrite 98.5% 3.5% 68.5% 32% 99.9% 33% 88% 47% 99.9% 78% 91% 57% 99.9% 84% 91% 58%
  71. 71. TO DO: Go to onely.com/tools and check your website (100% free) Crawl your website with and without JS and compare two crawls
  72. 72. We are going like: ”Oh, all right, we are gonna skip rendering.” It is not as frequently happening anymore. (...) Many websites even if they do not run JavaScript, they might still go through the render phase, because it doesn’t make a difference as much Crawling and indexing to come together I expect, eventually rendering, crawling and indexing will come closer together
  73. 73. Summary Indexing your content is one of the key challenges for 2020 All indexing issues we are seeing are 100% self-induced by website owners We need to redefine how we are looking at the crawler budget and the factors affecting it JavaScript SEO = Technical SEO
  74. 74. To do: 1Closely monitor your indexing stats (daily, weekly, biweekly). 2 If your content relies on JavaScript, monitor it closely and make sure that: • The code is light and efficient (cheap to render) • Your code doesn’t affect web performance on low-end devices 3 Make sure that no content within your structure relies on JavaScript (check all the page types within your domain). 4 Make sure that no meta data is changed during rendering.
  75. 75. To do: 1Closely monitor your indexing stats (daily, weekly, biweekly). 2 If your content relies on JavaScript, monitor it closely and make sure that: • The code is light and efficient (cheap to render) • Your code doesn’t affect web performance on low-end devices 3 Make sure that no content within your structure relies on JavaScript (check all the page types within your domain). 4 Make sure that no meta data is changed during rendering. Subscribe to Onely.com – be the first one to know about our crawl budget monitoring toolset.
  76. 76. More data is coming soon
  77. 77. Cloaking in 2019 bartosz@onely.com linkedin.com/in/goralewicz @bart_goralewicz THANK YOU one.ly/15min

Imagine having a big brand website with over 50% of its content not indexed by Google after weeks? Now imagine that the issue causing this severe decline in online visibility was self-induced? In the new digital marketing landscape, big brands losing money this way is shockingly normal. This deck from a Searchmetrics webinar Bartosz Goralewicz did with Marcus Tober on December 4, 2019. Want to watch the video? https://youtu.be/3XENVxVpyoo

Views

Total views

171

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

9

Shares

0

Comments

0

Likes

0

×