Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021

Crawling &
Indexing for
JavaScript
Heavy Sites
{TametheBots}
@davewsmart #brightonSEO

In the beginning ...
There was the web.
It served ﬂat documents,
and the SEOs said it was
good
{TametheBots}

But then we wanted to do stuff ...
We wanted to buy things,
blog, chat on forums.
{TametheBots}
So dynamic server
side scripting was
born, and SEOs were
like “Cool!”

So then we wanted to interact ...
Right in the Browser!
We wanted to break away from
ﬂat documents.
{TametheBots}
JavaScript was
born, and the SEOs
said ...

AAAAAAAAAAAA
HHHHHHHHHH
AAAAAAAARGH!

Which is kinda understandable
Search engines took a while to catch up
But the universe is different now*
*For some search engines!
{TametheBots}

Evergreen to the rescue!
Google used to render with an OLD browser.
In May ‘19 it got a BIG upgrade
It now runs up-to-date
Chromium!
{TametheBots}

Now executes modern JavaScript!
No more testing against Chrome 41!
{TametheBots}

What about Bing?
October ‘19, Evergreen too!
{TametheBots}

So it works just like my browser?
Nearly, but not quite …
The goals are different.
{TametheBots}

So it works just like my browser?
Humans want to see some pages.
Search engines want to gather
{TametheBots}
billions

How do they do that?
By accessing the web differently,
fetching only what they need, somewhat at
their own leisure.
{TametheBots}

How do they do that?
{TametheBots}
From: developers.google.com/search/docs/guides/javascript-seo-basics

Myths about the render queue
It’s not days, it’s mins.
Pretty much everything
goes through the WRS.
{TametheBots}

Some ground rules
A page needs a proper URL
Googlebot needs <a href=""> to get around.
{TametheBots}

Some ground rules
Non-200, or pages with noindex will not get
rendered.
{TametheBots}

Some ground rules
JavaScript, and API resources cannot be
blocked by robots.txt
(if you want them to work)
{TametheBots}

So, what’s the fuss about?
Some is about nothing.
But there are potential
lumps in
the custard.
{TametheBots}

Permission Denied!
Service Workers
Location
Notiﬁcations
...
{TametheBots}

Websockets? Nope!
Not designed to provide initial content.
Googlebot will tell you it does support, but
then fail anyway.
{TametheBots}

Web Workers? Kinda, Mostly!
Great for loading off heavy processes.
But … some unpredictable behaviour,
especially if it performs
fetches.
{TametheBots}

Solution?
Fail Gracefully!
Make sure the important
content loads anyway.
{TametheBots}

How to test?
● Mobile-Friendly Test
● Rich Results Test
● URL inspection tool (live)
{TametheBots}

We need to talk about caching
Google cache “aggressively”
They probably won’t listen to your
Cache-Control headers
{TametheBots}

We need to talk about caching
It is what allows their scale, and it’s a good
thing for you.
Less fetched = more budget
to fetch the good stuff.
{TametheBots}

But you need to work with it
Images, CSS, JavaScript & API crawls can
all be cached.
{TametheBots}

But you need to work with it
Some things it doesn’t matter for.
Some things it does!
{TametheBots}

JavaScript files
● Your site needs /app.js
● You update /app.js
{TametheBots}

JavaScript files
● Googlebot tries the new
page with the old /app.js
{TametheBots}

Who ya gonna call? Cache-busters!
● “Fingerprint” your ﬁles.
● /app.5787ee49.js
{TametheBots}

Who ya gonna call? Cache-busters!
Many frameworks & bundlers
can do this for you!
{TametheBots}

What about CSS?
● Less critical
● But can cause mobile
usability issues
{TametheBots}

What about CSS?
Same cache-busting solution
{TametheBots}

What about APIs?
If your content is loaded in via an API call,
this can be cached too.
You have a decision to make.
{TametheBots}

What about APIs?
Is freshness actually needed?
{TametheBots}

What about APIs?
Something like related
products? It might not, so
perhaps just let it cache.
{TametheBots}

What about when it matters?
If freshness does matter ...
Timestamp the call i.e:
/api/latest-news?ts=123456
POST not GET
{TametheBots}

How to test?
Live tools are made to bypass cache
Use URL Inspection Tool (not the live test)
Look at the rendered HTML
{TametheBots}

In your search console
{TametheBots}

Is the content up to date?
Is the content there?
Does the content look as up to date as the
last crawl date?
{TametheBots}

I’m a human, not a browser!
Ok, HTML isn’t always easy to
read. Click the copy button, then
go to that page.
{TametheBots}

Open devTools, right click on
<html> in the elements panel,
Select Edit as HTML
{TametheBots}
Hat tip to Oliver H.G. Mason (@ohgm), he's also
mentioned this solution, in a less clumsy way at:
https://www.contentkingapp.com/academy/url-in
spection-tool/#view-tested-page

Select all the code in the box, paste the
code from URL Inspection tool & Enter.
{TametheBots}
The page should now be
as Google rendered it.

Measure it in your log files!
No changes mean
gbot is hitting mostly
pages / robots.txt
{TametheBots}

Measure it in your log files!
I pushed a change,
with new ﬁlenames.
All the resources!
{TametheBots}

Some final thoughts
The dev team knows and cares about
users, they might not know about
googlebot.
{TametheBots}

Some final thoughts
Be their friendly guide,
not their nemesis.
{TametheBots}

Some final thoughts
I am not a JavaScript salesman.
Sometimes JavaScript isn’t the best way. If a pure HTML
solution is better & you can advocate for it, do!
{TametheBots}

Some final thoughts
TheOldWays™ are still valid!
{TametheBots}

Some final thoughts
The web is growing though!
JavaScript hasn’t killed the document web, it’s added to it.
As SEOs, we might be called on to support that.
{TametheBots}

Some great resources:
Google’s Dev Docs for JavaScript related SEO:
https://developers.google.com/search/docs/guides/javascript-seo-basics
Opening devTools:
https://developers.google.com/web/tools/chrome-devtools
SEO Mythbusters Video on JavaScript:
https://youtu.be/EZtCgrpa6ss
Martin Splitt’s JavaScript Hangouts, ask live questions!
Keep an eye out here:
https://developers.google.com/search/events#calendar
{TametheBots}

Bye Bye!
I’ve been Dave Smart, sorry about that :D
You can reach me at @davewsmart on
twitter, or ﬁnd me at tamethebots.com
Ta-Ta For Now!
{TametheBots}

Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021

Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021

More Related Content

What's hot

Similar to Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021

Recently uploaded

Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021