Deep crawl the chaotic landscape of JavaScript

<h2>The Chaotic Landscape of JavaScript </h4>

What is JavaScript SEO?
What it takes for a
modern JavaScript-
powered website to
be properly indexed
by search crawlers?

7 Random Statements
1. SEO was never this dynamic
2. Google is pushing more changes than ever, they aren’t as popular as
Penguin and Panda
3. I still see too many SEOs who live in a fantasy world made out out of
MOZ DA scores, keyword density and SEO “hacks”
4. That tiny pocket in jeans was designed to store pocket watches
5. Looking at some of the biggest websites in the world, it seems that most
didn’t hear about Technical SEO yet.
6. McDonald’s once made bubblegum-flavored broccoli
7. JavaScript is here to stay and JavaScript SEO is not a geeky option
anymore (YAY!)

Have you heard about the
2 waves of JavaScript
indexing?

Web development
HTML → HTML + CSS → HTML + a little JS → HTML + a lot of JS → JS

Why isn’t this process instant?

“…Rendering pages at the scale of the web
requires a lot of time and computational
resources. And make no mistake, this is a
serious challenge for search crawlers,
Googlebot included.”

“Rendering the JavaScript powered
web pages takes processor power
and memory. While Googlebot is very
powerful, it doesn’t have infinite
resources.”

1 user = 60 Watt lightbulb
lit for 3 hours*
*Data from 2011

JavaScript “lives”
in your CPU

“The European Union Experience”

The Guardian vs. CPU
A Case Study

It is believed that
The Guardian
& Amazon are
performance
superstars.

“…Many of us still look at sites
like Amazon’s or the Guardian’s in awe…”

How fast do you think
The Guardian loads?

But this performance comes with a price…

Go to Chrome Dev Tools → Performance
CMD + OPT + I
CTRL + ALT + I

HIGH END MACHINE - THE BEST CPU YOU
CAN HAVE IN A MACBOOK - 10 seconds

Slower CPU
Load Time – 90 seconds

Source: https://medium.com/dev-channel/the-cost-of-javascript-84009f51e99e

Unfortunately not all websites are
like The Guardian…

SECONDS TO FIRST
MEANINGFUL PAINT HIGH
END CPU

SECONDS TO FIRST
MEANINGFUL PAINT (Moto
G4’s CPU)

13 SECOND DIFFERENCE
BETWEEN A HIGH END CPU
AND A SLOW CPU (!)

• Technical SEO in 2018 - New metrics, new technologies and new challenges.

What did they need
React for?
Language switcher
Buttons on the bottom of
the page
Client side login library

~300 lines of code written
~200 kB removed

Netflix’s performance (and
SEO) doesn’t depend on the
client’s CPU that much

AMP – BBC.COM
Netflix.com Netflix.com @ Moto G4

~200 KB to process
4 seconds
0.065 seconds

How do you ensure you are not caught up
between the waves?
You need to embrace chaos!

There are
And a lot of those “documents” are getting more and more
complex!

With the exponential
growth of JavaScript
frameworks...

artoszCoin
barti
How to be cool in 2018

The Big Three
Source: https://javascriptreport.com/the-ultimate-guide-to-javascript-frameworks/

?
?
?
??
??
??
?
?
?
?
?
?? ??
?
??
?
???
?
?
?
?
?
?
Let’s hear the framework
creators talk about
JavaScript & SEO

...if you care about
SEO, you still need to
have server-rendered
content.
Jeff Whelpley
Angular U conference, June 22-25, 2015, Hyatt Regency, San Francisco Airport
“Angular 2 Server Rendering”
ele.ph/angularU

...we are generally able
to render and
understand your web
pages like modern
browsers.

Can you
rank with a
JS website?

As Googlers would say
“there are factors at play here”

Let’s search for Hulu’s exclusive shows.

Everyone hates him for telling the truth!
Here’s why you’re
wasting your time on
client-rendered JS
websites…
Findoutnow!

SEO – where the giants fight!

Why is their
visibility declining?

Vimeo’s
homepage
without JS!

I’m about to
reveal the
YouTube secret
no one ever
told you.

YouTube is a
client-rendered
JS website…

…and they are
prerendering for
Googlebot

Why would YouTube prerender
for Googlebot? Doesn’t
Google render JS just fine?

SEO – where the giants fight

JavaScript vs. Crawler budget (crawl demand)

Oh no!
ele.ph/budget

Oh no!
(…) Also, crawling &
indexing is currently
a bit slower than
static HTML (...)
ele.ph/crawldemand

Crawling and
indexing
JavaScript is not
a 0/1 thing

GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
External Ajax callInline

JavaScript
dozens of different frameworks
their different configurations
transpiling
inline vs. external

On top of that, there are hundreds of factors
affecting crawling and indexing

?How about Search Engines other than

statements
reality
guy @ Pubcon
our experiment

THE BING THING
Fabrice Canel of
Bing said that
Bing processes
JS.
True story!
Patrick Stox @ DTD Conference 2018

Rich media cautions
don’t bury links to
content inside JavaScript

Rich media cautions
Graceful degradation
enable a clean down-
level experience so
crawlers can see your
content

Rich media cautions
Graceful degradation
Down-level experience
enhances discoverability
avoid housing content
inside Flash or JavaScript
– these block crawlers
from finding the content

It can process JS vs. It DOES process JS

2. Copy any text from the website
CTRL+C

3. Try to find it in Bing.com
CTRL+V
4. Find everything BUT the angular.io
website

Or search for any content from jsseo.expert

63,4%
22,8%
11,7%
1,3% 0,8%
Google Bing Yahoo! Ask AOL
Search Engine market share
Source: https://www.statista.com/statistics/267161/market-share-of-search-engines-in-the-united-states/

83,5%
11,1%
3,8% 0,8% 0,6%
Google Bing Yahoo! MSN DuckDuckGo
Source: https://www.statista.com/statistics/280269/market-share-held-by-search-engines-in-the-united-kingdom/

94,5%
4,2% 1,0% 0,1% 0,1%
Google Bing Yahoo! T-Online Ask.com
Source: https://www.statista.com/statistics/445002/market-shares-leading-search-engines-germany/

Let’s take a closer look at Google

JavaScript is REALLY easy to break

It’s all very shaky,
based on duct tape rather than…

Let’s start from Angular.io
and their GitHub issue

https://angular.io/guide/language-service
https://angular.io/guide/security
https://angular.io/guide/architecture
https://angular.io/api/animations/group
https://angular.io/guide/quickstart

https://angular.io
https://angular.io/events
https://angular.io/api
https://angular.io/resources
https://angular.io/about?group=Angular

https://angular.io
https://angular.io/events
https://angular.io/api
https://angular.io/resources
https://angular.io/about?group=Angular
url / / url /

UPSIDE DOWN
The only thing [those] pages (…)
have in common is the number of
path segments in the url.
Igor Minar

http://yourdomain.com/category/shoes/
url / / url /
How much would that cost you?
http://yourdomain.com/category/

Prerender.io – is partial indexing in Google possible?

vs. Developers
SEOs
vs. Search Engines
vs. Framework creators

…We’ve created what will be
remembered as the world’s
most confusing JavaScript
survey ever.

…it contained only
1 question…

…but proved to be the most
difficult question you can
ask your developer.

Can client-side rendered JavaScript
websites rank high in Google?

Can you rank well with a Client
Rendered JavaScript Website?

Is there hope for client-rendered JS websites?

Ranking with a CSR JavaScript
website
Ranking without it

What is partial indexing?
This means that if your site is using a heavy amount of client side
JS, you can be tripped up at times when the content is being
indexed due to this two phase indexing process.
It’s possible that some details may
be missed.

And this effectively means that if
your site is using a heavy amount of
client-side JavaScript for rendering,
you could be tripped up at times
when your content is being indexed
due to the nature of this two phase
indexing process.
…it’s possible some details might be
missed.
Tom Greenaway

2 waves
WAVE 1 WAVE 2
• JS dependent
content only
• HTML content
• Canonicals
• Meta data
• HTTP Codes

Indexing of modern JavaScript
websites is a challenge.
John Mueller

Client Side rendering
Source: Google I/O 2018

Server-side rendering

Hybrid Rendering*
*Google’s Long-Term
recommendation

Dynamic Rendering
(aka prerendering)

Prerendering/Dynamic Rendering issues
1. Computing power – A LOT of servers
2. Prone to issues (often load related)
3. Downtime = ranking loss
4. More complex and difficult from an SEO perspective (crawls, 2 sets
of code, etc.)
5. Requires a lot of SEO knowledge
6. Requires a great dev team to make it run smoothly

When to use dynamic
rendering (according
to Google)
A dynamic website
“Is if your site is large and
rapidly changing for, example
if you have a news website.”
Your website is relying on
features that are not
supported in Chrome 41
Libraries that cannot be
transpiled back to ES5, APIs
that don’t suport Chrome 41

Troubleshooting JavaScript indexing

Google Search
Console – fetch
and render

Google mobile-friendly test
g.co/MobileFriendly

ele.ph/hulu-test

ele.ph/hulu-test
COPY THIS CODE

#SMX #XXA @McCieslak
1. The Source Code vs. DOM
Source code DOM

#SMX #XXA @McCieslak
GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
InlineInlineV2 Inline
2. Crawling - server log

Make sure your content is indexed
ctrl+c
ctrl+v
hulu.com/casual

Under the right URL
https://angular.io/features

To Do:
1. Diff Check is your new best friend
2. Experiment and make sure to monitor
your server logs
3. Compare your setup with Chrome 41
4. Make sure that your content is indexed
in Google
5. Under the right URL

Wrap up JS
1
JS can kill
crawler budget

Wrap up JS
1
JS can kill a
crawler budget
2
JS websites have
ranking problems

Ranking well with
CSR JavaScript
websites is very
hard if not
impossible.

2
yet
Make sure you choose a right frawework

We are still looking for a pioneer!

Bartosz@Goralewicz.com
@bart_Goralewicz

Deep crawl the chaotic landscape of JavaScript

More Related Content

What's hot

Similar to Deep crawl the chaotic landscape of JavaScript

Recently uploaded

Deep crawl the chaotic landscape of JavaScript