Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings

17,064 views

Published on

Slides from my session at Brighton SEO 2017

Published in: Marketing

Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings

  1. 1. Bartosz Góralewicz ELEPHATE Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings @bart_goralewicz https://www.linkedin.com/in/goralewicz/
  2. 2. Why is JavaScript SEO such a complex topic?
  3. 3. DATANO Why is JavaScript SEO such a complex topic?
  4. 4. DATANO Making JavaScript SEO easy
  5. 5. JS SEO
  6. 6. JS SEO
  7. 7. What is the problem with JavaScript? HTML JS
  8. 8. What is the problem with JavaScript? HTML HTML
  9. 9. What is the problem with JavaScript? HTML HTML
  10. 10. JavaScript
  11. 11. and JavaScript
  12. 12. JavaScript & SEO
  13. 13. 50 shades of JavaScript
  14. 14. *more about it later not as “forgiving” as HTML*
  15. 15. It has all with…
  16. 16. Reverse engineering algorithms is no longer an option. Times have changed. Today, as long as you're not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers. To reflect this improvement, we recently updated our technical Webmaster Guidelines to recommend against disallowing Googlebot from crawling your site's CSS or JS files. ele.ph/ajax
  17. 17. ele.ph/scotch
  18. 18. Can this backfire?
  19. 19. Hulu.com JavaScript & SEO backfire
  20. 20. -37%
  21. 21. ele.ph/hulu Hulu case study
  22. 22. Google and JavaScript – difficult relationship
  23. 23. The only solution… Java Script Experiment
  24. 24. Kamil Grymuza
  25. 25. http://jsseo.expert
  26. 26. Content! OK.
  27. 27. ele.ph/AI
  28. 28. Inspect Element vs. View Source
  29. 29. Inspect element
  30. 30. View source
  31. 31. First JavaScript rendering issues
  32. 32. Timeout, GSC bug or real ?
  33. 33. /test/
  34. 34. JavaScript links were not parsed by Googlebot Google kept us waiting…
  35. 35. Fail?
  36. 36. Re-fetched all URLs via GSC
  37. 37. /html/test/ (duh!) /plain-inline/test/ /jquery-inline/test/ /jquery-ajax/test/ /vue/test/ /react-inline/test/ And… that’s it! /test/
  38. 38. Let’s start with JS frameworks created by
  39. 39. Version 1
  40. 40. Checklist Check if ”framework/test/” URL was crawled Search for unique content from framework’s page Is URL indexed by Google? Fetch and render test (GSC)
  41. 41. Fetch and render test (GSC)
  42. 42. Is URL indexed by Google?
  43. 43. Search for unique content from framework’s page
  44. 44. Check if ”framework/test/” URL was crawled
  45. 45. V2 Inline GSC fetch URL Indexed? Content Indexed? Test URL Inline Inline
  46. 46. GSC fetch URL Indexed? Content Indexed? Test URL External Ajax callInline
  47. 47. GSC fetch URL Indexed? Content Indexed? Test URL Inline External
  48. 48. GSC fetch URL Indexed? Content Indexed? Test URL Inline External
  49. 49. Inline vs. External JS makes a huge difference Not all JavaScript frameworks are crawled and indexed in the same way Mindblowing findings JS generated links aren’t always crawled
  50. 50. ele.ph/angularU Also… Regardless of what your developers say, Angular JS 2 always has to be server rendered… according to Angular JS 2 community.
  51. 51. If you search for any competitive keyword terms, it’s always going to be server rendered sites. And the reason is because although Google does index client-side rendered HTML, it’s not perfect yet and other search engines don’t do it as well. So if you care about SEO, you still need to have server-rendered content. Jeff Whelpley Angular U conference, June 22-25, 2015, Hyatt Regency, San Francisco Airport “Angular 2 Server Rendering” ele.ph/angularU
  52. 52. ?How about Search Engines other than
  53. 53. 63,4% 22,8% 11,7% 1,3% 0,8% Google Bing Yahoo! Ask AOL Search Engine market share Source: https://www.statista.com/statistics/267161/market-share-of-search-engines-in-the-united-states/
  54. 54. 83,5% 11,1% 3,8% 0,8% 0,6% Google Bing Yahoo! MSN DuckDuckGo Search Engine market share Source: https://www.statista.com/statistics/280269/market-share-held-by-search-engines-in-the-united-kingdom/
  55. 55. ele.ph/beyondgoogle
  56. 56. ele.ph/jsforum
  57. 57. Tell me why I can’t… Clear your mind of questions *not the actual conversation
  58. 58. GSC fetch URL Indexed? Content Indexed? Test URL External Ajax callInline
  59. 59. http://jsseo.expert/plain-external/test/ http://jsseo.expert/react-external/test/ http://jsseo.expert/jquery-external/test/ we found this URL from a link ca 3 weeks ago, but we haven't crawled it we picked this one up ~19 days ago, and similar to above, our systems just didn't feel like crawling it yet picked up ~21 days ago, also not crawled yet.
  60. 60. Thanks, John! You’re the best, Bartosz!* *he didn’t say that John Mueller’s feedback got me thinking…
  61. 61. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce JavaScript vs. crawler budget Some JavaScript frameworks are indexable, but there is a fine print…
  62. 62. JavaScript
  63. 63. BUT… HTML
  64. 64. Do you already know where I am going with this?
  65. 65. Kamil Grymuza
  66. 66. Crawler budget experiment (early beta)
  67. 67. We repeated this experiment a few times and the results were always the same.
  68. 68. JavaScript, even when indexable, can kill your crawler budget…
  69. 69. JS is less effective than HTML JavaScript is not bad for SEO, if done right* *he said exactly that Ilya Grigorik
  70. 70. Understanding technology behind
  71. 71. ele.ph/ilya
  72. 72. ele.ph/chrome41
  73. 73. ele.ph/chrome41 Download Chrome 41 here:
  74. 74. Using Chrome 41 for debugging JS indexing
  75. 75. http://jsseo.expert/angular2/ in Chrome 41
  76. 76. http://jsseo.expert/angular2/ in up-to-date Chrome
  77. 77. JavaScript SEO isn’t only affecting technical
  78. 78. At the same times those links count just as HTML ones. Most link indexes (e.g. Majestic, MOZ etc.) can’t crawl JavaScript generated backlinks.
  79. 79. Also… ...there is more JavaScript generated links out there than I would imagine (I’ll tell you more about it in a second)
  80. 80. Problems? Manual penalties Tracking mentions Google Penguin Link audits Link building
  81. 81. Crawling JavaScript is expensive…
  82. 82. To execute JS for every page at our scale [would] require 10,000-15,000 servers, and we believe our customers are not ready to pay for that yet” Dmitry Gerasimenko, CEO of Ahrefs ~ July 2017
  83. 83. No hope?
  84. 84. Ahrefs guys changed their mind after reading my article… ele.ph/js-ahrefs
  85. 85. ele.ph/js-ahrefs
  86. 86. …We accepted the challenge and crawled this same website with our JS crawler, which rendered all of the pages that were there. ele.ph/js-ahrefs
  87. 87. ele.ph/js-ahrefs
  88. 88. ele.ph/js-ahrefs So earlier this year Dmitry (our Founder & CEO) decided that Ahrefs has to be the first among its rivals to crawl JavaScript links. Which we achieved with ease last month.
  89. 89. JS ele.ph/js-ahrefs
  90. 90. JS ele.ph/js-ahrefs
  91. 91. 787622915746127654196745167546245246157964127432546 12794214512161504918805916076096067461350967619097 645684795464436557433443675698347856654756434587 6554364655437876954878796865645654456453212543321 5645466556762615655244675988799980879876424223314 003132425654342698754736755645654465474587698374 7876229 4322546 12794214512161504918805916076096067461350967619097 645684795464436557433443675698347856654756434587 6554364655437876954878796865645654456453212543321 5645466556762615655244675988799980879876424223314 003132425654342698754736755645654465474587698374 787622915746127654196745167546245246157964127432546 So how many JS links are out there?
  92. 92. JS LINKS PER DAY!
  93. 93. Is there a perfect JavaScript SEO scenario?
  94. 94. YES
  95. 95. The setup for that is called Isomorphic JavaScript on React & Universal on Angular. That's what I'd recommend doing in general, if you can. Isomorphic JavaScript ele.ph/Iso-John
  96. 96. ele.ph/iso
  97. 97. What if it’s too late?
  98. 98. Prerender.io
  99. 99. user server middleware Prerendering crawlers JS HTML JS ? ?
  100. 100. Experiments continue…
  101. 101. JS Only Google and Ask can handle JavaScript Key takeways
  102. 102. JS Only Google and Ask can handle JavaScript Not all JavaScript frameworks are indexable Key takeways
  103. 103. Not all JavaScript frameworks are indexable Indexing JavaScript is only half the battle Key takeways
  104. 104. Indexing JavaScript is only half the battle JavaScript even when rendered and indexed, can kill crawler budget Key takeways
  105. 105. JavaScript even when rendered and indexed, can kill crawler budget Use Chrome 41 to debug JS Key takeways
  106. 106. Use Chrome 41 to debug JS If too late, use prerendering Key takeways
  107. 107. If too late, use prerendering Key takeways Isomorphic JS = perfect scenario
  108. 108. Not all JavaScript frameworks are indexable Indexing JavaScript is only half the battle JavaScript even when rendered and indexed, can kill crawler budget Use Chrome 41 to debug JS JS Only Google and Ask can handle JavaScript If too late, use prerendering Key takeways Isomorphic JS = perfect scenario
  109. 109. @bart_goralewicz /in/goralewicz/ Thanks! :*

×