1.
Bartosz Góralewicz
ELEPHATE
Can Google properly crawl and index JavaScript?
SEO Experiments - Results and findings
@bart_goralewicz
https://www.linkedin.com/in/goralewicz/

2.
Why is JavaScript SEO such a complex topic?

3.
DATANO
Why is JavaScript SEO such a complex topic?

4.
DATANO
Making JavaScript SEO easy

5.
JS SEO

6.
JS SEO

7.
What is the problem with JavaScript?
HTML
JS

8.
What is the problem with JavaScript?
HTML
HTML

9.
What is the problem with JavaScript?
HTML
HTML

10.
JavaScript

11.
and JavaScript

12.
JavaScript & SEO

13.
50 shades of JavaScript

14.
*more about it later
not as “forgiving” as HTML*

15.
It has all with…

16.
Reverse
engineering
algorithms is
no longer an
option.
Times have changed. Today, as
long as you're not blocking
Googlebot from crawling your
JavaScript or CSS files, we are
generally able to render and
understand your web pages like
modern browsers. To reflect this
improvement, we recently updated
our technical Webmaster
Guidelines to recommend against
disallowing Googlebot from
crawling your site's CSS or JS files.
ele.ph/ajax

17.
ele.ph/scotch

18.
Can this backfire?

19.
Hulu.com
JavaScript & SEO backfire

20.
-37%

21.
ele.ph/hulu
Hulu case study

22.
Google and JavaScript – difficult relationship

23.
The only solution…
Java Script Experiment

24.
Kamil
Grymuza

25.
http://jsseo.expert

26.
Content!
OK.

27.
ele.ph/AI

28.
Inspect Element
vs.
View Source

29.
Inspect element

30.
View source

31.
First JavaScript rendering issues

32.
Timeout, GSC bug or real ?

33.
/test/

34.
JavaScript links were
not parsed by Googlebot
Google
kept us
waiting…

35.
Fail?

36.
Re-fetched all URLs via GSC

37.
/html/test/ (duh!)
/plain-inline/test/
/jquery-inline/test/
/jquery-ajax/test/
/vue/test/
/react-inline/test/
And… that’s it! /test/

38.
Let’s start with JS frameworks created by

39.
Version 1

40.
Checklist
Check if ”framework/test/” URL was crawled
Search for unique content from framework’s page
Is URL indexed by Google?
Fetch and render test (GSC)

41.
Fetch and render test (GSC)

42.
Is URL indexed by Google?

43.
Search for unique content from framework’s page

44.
Check if ”framework/test/” URL was crawled

45.
V2 Inline
GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
Inline Inline

46.
GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
External Ajax callInline

47.
GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
Inline External

48.
GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
Inline External

49.
Inline vs. External JS makes a huge difference
Not all JavaScript frameworks are crawled
and indexed in the same way
Mindblowing findings
JS generated links aren’t always crawled

50.
ele.ph/angularU
Also…
Regardless of what your
developers say, Angular JS 2
always has to be server
rendered…
according to
Angular JS 2 community.

51.
If you search for any competitive
keyword terms, it’s always going
to be server rendered sites. And
the reason is because although
Google does index client-side
rendered HTML, it’s not perfect
yet and other search engines
don’t do it as well. So if you care
about SEO, you still need to have
server-rendered content.
Jeff Whelpley
Angular U conference, June 22-25, 2015, Hyatt Regency, San Francisco Airport
“Angular 2 Server Rendering”
ele.ph/angularU

52.
?How about Search Engines other than

53.
63,4%
22,8%
11,7%
1,3% 0,8%
Google Bing Yahoo! Ask AOL
Search Engine market share
Source: https://www.statista.com/statistics/267161/market-share-of-search-engines-in-the-united-states/

54.
83,5%
11,1%
3,8% 0,8% 0,6%
Google Bing Yahoo! MSN DuckDuckGo
Search Engine market share
Source: https://www.statista.com/statistics/280269/market-share-held-by-search-engines-in-the-united-kingdom/

55.
ele.ph/beyondgoogle

56.
ele.ph/jsforum

57.
Tell me
why I
can’t…
Clear your
mind of
questions
*not the actual conversation

58.
GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
External Ajax callInline

59.
http://jsseo.expert/plain-external/test/
http://jsseo.expert/react-external/test/
http://jsseo.expert/jquery-external/test/
we found this URL from a link ca 3 weeks ago, but we haven't crawled it
we picked this one up ~19 days ago, and similar to above,
our systems just didn't feel like crawling it yet
picked up ~21 days ago, also not crawled yet.

60.
Thanks,
John!
You’re the
best,
Bartosz!*
*he didn’t say that
John Mueller’s feedback got me thinking…

61.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit
amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada
fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce
posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus.
Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros
quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin
pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar
ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi
tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra
imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy
pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus
malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et
netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor
congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce
est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem
ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet
commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac
turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere,
magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus.
Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros
quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin
pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar
ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi
tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra
imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy
pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus
malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et
netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor
congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce
JavaScript vs. crawler budget
Some JavaScript frameworks are indexable,
but there is a fine print…

62.
JavaScript

63.
BUT…
HTML

64.
Do you already know where I am going with this?

65.
Kamil
Grymuza

66.
Crawler budget experiment (early beta)

67.
We repeated this experiment a few times and
the results were always the same.

68.
JavaScript,
even when indexable,
can kill your crawler budget…

69.
JS is less
effective than
HTML
JavaScript is not
bad for SEO, if
done right*
*he said exactly that
Ilya Grigorik

70.
Understanding technology behind

71.
ele.ph/ilya

72.
ele.ph/chrome41

73.
ele.ph/chrome41
Download Chrome 41 here:

74.
Using Chrome 41
for debugging JS indexing

75.
http://jsseo.expert/angular2/
in Chrome 41

76.
http://jsseo.expert/angular2/
in up-to-date Chrome

77.
JavaScript SEO isn’t only affecting technical

78.
At the same times those links count just as HTML ones.
Most link indexes
(e.g. Majestic, MOZ etc.)
can’t crawl JavaScript generated backlinks.

79.
Also…
...there is more JavaScript generated links out there
than I would imagine
(I’ll tell you more about it in a second)

80.
Problems?
Manual penalties
Tracking mentions
Google Penguin
Link audits
Link building

81.
Crawling JavaScript is expensive…

82.
To execute JS for every page at
our scale [would] require
10,000-15,000 servers, and
we believe our customers are
not ready to pay for that yet”
Dmitry Gerasimenko, CEO of
Ahrefs
~ July 2017

83.
No hope?

84.
Ahrefs guys changed their mind after
reading my article…
ele.ph/js-ahrefs

85.
ele.ph/js-ahrefs

86.
…We accepted the challenge
and crawled this same
website with our JS crawler,
which rendered all of the
pages that were there.
ele.ph/js-ahrefs

87.
ele.ph/js-ahrefs

88.
ele.ph/js-ahrefs
So earlier this
year Dmitry (our
Founder & CEO) decided that
Ahrefs has to be the first
among its rivals to crawl
JavaScript links.
Which we achieved with
ease last month.

89.
JS
ele.ph/js-ahrefs

90.
JS
ele.ph/js-ahrefs

91.
787622915746127654196745167546245246157964127432546
12794214512161504918805916076096067461350967619097
645684795464436557433443675698347856654756434587
6554364655437876954878796865645654456453212543321
5645466556762615655244675988799980879876424223314
003132425654342698754736755645654465474587698374
7876229 4322546
12794214512161504918805916076096067461350967619097
645684795464436557433443675698347856654756434587
6554364655437876954878796865645654456453212543321
5645466556762615655244675988799980879876424223314
003132425654342698754736755645654465474587698374
787622915746127654196745167546245246157964127432546
So how many JS links are out there?

92.
JS LINKS PER DAY!

93.
Is there a perfect JavaScript SEO scenario?

94.
YES

95.
The setup for that is called
Isomorphic JavaScript on
React & Universal on
Angular. That's what I'd
recommend doing in general,
if you can.
Isomorphic JavaScript
ele.ph/Iso-John

96.
ele.ph/iso

97.
What if it’s too late?

98.
Prerender.io

99.
user
server
middleware
Prerendering
crawlers
JS
HTML
JS
?
?

100.
Experiments continue…

101.
JS
Only Google and Ask
can handle JavaScript
Key takeways

102.
JS
Only Google and Ask
can handle JavaScript
Not all JavaScript
frameworks are indexable
Key takeways

103.
Not all JavaScript
frameworks are indexable
Indexing JavaScript
is only half the battle
Key takeways

104.
Indexing JavaScript
is only half the battle
JavaScript even when
rendered and indexed,
can kill crawler budget
Key takeways

105.
JavaScript even when
rendered and indexed,
can kill crawler budget
Use Chrome 41
to debug JS
Key takeways

106.
Use Chrome 41
to debug JS
If too late,
use prerendering
Key takeways

107.
If too late,
use prerendering
Key takeways
Isomorphic JS
=
perfect scenario

108.
Not all JavaScript
frameworks are indexable
Indexing JavaScript
is only half the battle
JavaScript even when
rendered and indexed,
can kill crawler budget
Use Chrome 41
to debug JS
JS
Only Google and Ask
can handle JavaScript
If too late,
use prerendering
Key takeways
Isomorphic JS
=
perfect scenario

109.
@bart_goralewicz
/in/goralewicz/
Thanks! :*