BrightonSEO April 2019 - Tom Pool - Chrome Puppeteer, Fake Googlebo...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

slideshare.net/Tom-Pool
How To Use Chrome Puppeteer To
Fake Googlebot And Monitor Your Site
Tom Pool // BlueArray //
@cptn...

Who Am I?
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

Look After
Technical
Output Of
The
Agency
@cptntommy #BrightonSEO

Always Trying
To Find Ways
To Make My
Teams Job
Easier
@cptntommy #BrightonSEO

So I Was
Watching
Google I/O 18
(Which Is
Awesome
BTW)
@cptntommy #BrightonSEO

And I Saw A
Really
Really
Really Cool
Talk
@cptntommy #BrightonSEO

Eric Bidelman
@cptntommy #BrightonSEO

This Got Me Thinking
@cptntommy #BrightonSEO

I Can Use This
To Help Me
With My Job!
@cptntommy #BrightonSEO

So I Went Away &
Did A Shit Ton Of
Research
@cptntommy #BrightonSEO

That Included
@cptntommy #BrightonSEO

Headless
Chrome
@cptntommy #BrightonSEO

Chrome
@cptntommy #BrightonSEO

And A Little Bit
Of Coding
@cptntommy #BrightonSEO

(Not Much!)
@cptntommy #BrightonSEO

I Want All Of You
To At Least Take
@cptntommy #BrightonSEO

A Small
Piece Of
Knowledge
From This
@cptntommy #BrightonSEO

I’ll Also Tweet Out
This Deck
@cptntommy #BrightonSEO

So...
@cptntommy #BrightonSEO

What Is
Headless
Chrome?
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

Headless
Chrome
=
None Of That
Shit
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

Google Chrome Is
Running, But With
No User Interface
@cptntommy #BrightonSEO

So It Is ‘Headless’
@cptntommy #BrightonSEO

Why Should You Even Care?
@cptntommy #BrightonSEO

You Can:
@cptntommy #BrightonSEO

Scrape The Shit Out Of (JS)
Websites
@cptntommy #BrightonSEO

Copy The DOM, & Paste To A
Text File
@cptntommy #BrightonSEO

Compare Source Code With
DOM & Export Differences
@cptntommy #BrightonSEO

Generate
Screenshots
of Pages
@cptntommy #BrightonSEO

Crawl Single Page
Applications
@cptntommy #BrightonSEO

I Know, JS Is Evil, But
It Ain’t Going Away!
@cptntommy #BrightonSEO

Screaming Frog Does Have JS
Rendering Features.
Utilises (Something Like)
Headless Chrome
@cptntommy #BrightonSEO

Google Can Render JS, But It Is
In No Way Perfect, Or Even
That Effective
@cptntommy #BrightonSEO

Countless Case Studies
@cptntommy #BrightonSEO

Crawl Single Page
Applications
@cptntommy #BrightonSEO

Automate WebPage Checks
@cptntommy #BrightonSEO

Used For Webpage Testing
(Clicking On Buttons, Filling
In Forms, General Fuckery)
@cptntommy #BrightonSEO

Great For Emulating User
Behaviour!
@cptntommy #BrightonSEO

Great For Seeing How Much
Shit A Website Can Take
Before It Breaks!
@cptntommy #BrightonSEO

The Problem Is...
@cptntommy #BrightonSEO

You Have To Run
Basic Headless
Chrome From
Command Line
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

/Applications/Google
Chrome.app/Contents/MacOS/Google
Chrome
@cptntommy #BrightonSEO

/Applications/Google
Chrome.app/Contents/MacOS/Google
Chrome --headless
@cptntommy #BrightonSEO

/Applications/Google
Chrome.app/Contents/MacOS/Google
Chrome --headless --remote-debugging-
port=9222
@cptntommy #Brighton...

/Applications/Google
Chrome.app/Contents/MacOS/Google
Chrome --headless --remote-debugging-
port=9222 --disable-gpu
@cptnt...

/Applications/Google
Chrome.app/Contents/MacOS/Google
Chrome --headless --remote-debugging-
port=9222 --disable-gpu
https:...

Now
@cptntommy #BrightonSEO

I
Really
Really
Love Using
Command
Line
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

But This
Really
Really
Made
Me
Cry
@cptntommy #BrightonSEO

So How Do I Make It
Easy?
@cptntommy #BrightonSEO

Like I Said - I’m
Always Trying To
Make My Job Easier
@cptntommy #BrightonSEO

And This
Was Not
Easy!
@cptntommy #BrightonSEO

So I Went Away &
Did A Bigger Shit
Ton Of
Research
@cptntommy #BrightonSEO

Eric Bidelman
@cptntommy #BrightonSEO

What Is
Chrome
Puppeteer?
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

BlahBlahBlahBlahBlahBlahBlah
BlahBlahBlahBlahBlahBlahBlahBlahBlah
BlahBlahBlahBlahBlahBlahBlahBlahBlahBlahBlahBlahBlahBlah...

OOOOOOOO API
@cptntommy #BrightonSEO

Node Can Be Used
For Making
Applications
@cptntommy #BrightonSEO

And It Can Also
Be Used To help
Control Headless
Chrome
@cptntommy #BrightonSEO

And Trust Me
It’s Easy!
@cptntommy #BrightonSEO

So How Can I
Get Chrome
Puppeteer?
@cptntommy #BrightonSEO

If You Want To
Run Tests On
Your Local
Machine
@cptntommy #BrightonSEO

You Have To
Install NPM &
Node.js
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

Someone’s
Made This
Easy!
@cptntommy #BrightonSEO

So If
You
Are On
PC
@cptntommy #BrightonSEO

It’s Pretty
Straightforward
@cptntommy #BrightonSEO

Just Install From
The Node.js
Websites
@cptntommy #BrightonSEO

bit.ly/pc-pup-brighton19
@cptntommy #BrightonSEO

If You
Are On
Mac
@cptntommy #BrightonSEO

(Like Me)
@cptntommy #BrightonSEO

It’s Not That
Easy
@cptntommy #BrightonSEO

bit.ly/pupbrighton19
@cptntommy #BrightonSEO

You Wanna
Open Up
Terminal
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

ruby -e "$(curl -fsSL
https://raw.githubusercontent.com/Homebrew/
install/master/install)"
@cptntommy #BrightonSEO

This Installs
Homebrew,
That Makes
Everything E-Z
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

When
This
Has
Done Its
Thing
@cptntommy #BrightonSEO

You Have To
Install 2 More
Things, And
We’ll Be Ready
To Rock
@cptntommy #BrightonSEO

brew install node
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

And Then
@cptntommy #BrightonSEO

npm i puppeteer
@cptntommy #BrightonSEO

Now You Are All
Good!
@cptntommy #BrightonSEO

You Can Now Run
Chrome Puppeteer On
Your Machine!
@cptntommy #BrightonSEO

For Example
@cptntommy #BrightonSEO

If I Wanted To Take A
Screenshot Of A
Single Webpage
@cptntommy #BrightonSEO

There Is A Bunch Of
Code Coming Up
@cptntommy #BrightonSEO

That Can All Be Seen
In The Following Link
(I’ll Also Tweet It)
@cptntommy #BrightonSEO

https://bit.ly/Brighton
SEO19
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

let browser = await
puppeteer.launch({headless:
true});
@cptntommy #BrightonSEO

let page = await
browser.newPage();
@cptntommy #BrightonSEO

await
page.goto('https://www.
bluearray.co.uk/');
@cptntommy #BrightonSEO

await
page.screenshot({
@cptntommy #BrightonSEO

await
page.screenshot({ path:
'./testimg.jpg',
@cptntommy #BrightonSEO

await
page.screenshot({ path:
'./testimg.jpg', type:
'jpeg'});
@cptntommy #BrightonSEO

await page.close();
await
browser.close();
@cptntommy #BrightonSEO

File Is Saved As
Screenshot.js
@cptntommy #BrightonSEO

So To Run This Small
Piece Of Code
@cptntommy #BrightonSEO

Go To Terminal (In
Same Folder As Code),
And Type In
@cptntommy #BrightonSEO

Node Screenshot.js
@cptntommy #BrightonSEO

And Then, 5 Seconds
later,
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

If You Wanted To See
The Browser Do These
Steps
@cptntommy #BrightonSEO

let browser = await
puppeteer.launch({headless:
True});
@cptntommy #BrightonSEO

let browser = await
puppeteer.launch({headless:
False});
@cptntommy #BrightonSEO

You Can Also Provide
A List Of URLs
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO
And Get A Shit Ton Of
Screenshots!

Now I’m Sure You Can
See Where This Is
Headed
@cptntommy #BrightonSEO

Faking Googlebot!
@cptntommy #BrightonSEO

With A Few Tweaks to
The Code
@cptntommy #BrightonSEO

await
page.setUserAgent
('Googlebot');
@cptntommy #BrightonSEO

Googlebot’s User
Agent Is Not Just
‘Googlebot’
@cptntommy #BrightonSEO

It’s Fuck*** Huge
@cptntommy #BrightonSEO

Mozilla/5.0 (Linux; Android 6.0.1;
Nexus 5X Build/MMB29P)
AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/41.0.2272.96
Mobil...

And Then You Gotta
Set Googlebot’s
Viewport
@cptntommy #BrightonSEO

await
page.setViewport
@cptntommy #BrightonSEO

await
page.setViewport
({width: 1024, height:
1024});
@cptntommy #BrightonSEO

FYI This Is Not Really
Googlebot
@cptntommy #BrightonSEO

As Unfortunately
@cptntommy #BrightonSEO

Can’t Change Chrome
Version That
Puppeteer Uses To 41
:(
@cptntommy #BrightonSEO

As Chrome Puppeteer
Was Released After
Chrome 41
(*Not Backwards Compatible)
@cptntommy #BrightonSEO

However!
@cptntommy #BrightonSEO

Can Be Persuasive In
Getting A Client To
Ensure Their Content
Is SSR’d
(If Needed)
@cptntommy #BrightonSEO

Chrome Puppeteer
Can Be Installed On
The Server
@cptntommy #BrightonSEO

We Can Then Provide
Puppeteer With A List
Of URLs, And It Can
Work Through Them
All
@cptntommy #BrightonSEO

And Show How They
Would Appear To
Google, Instead Of
@cptntommy #BrightonSEO

In The Case Of Some
JS Sites
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

A Blank Page
@cptntommy #BrightonSEO

Which Is Cool & A
Nice Trick
@cptntommy #BrightonSEO

But The Really Cool
Stuff Is Yet To Come
@cptntommy #BrightonSEO

So Who Here
Has Heard
Of (Or Used)
ContentKing?
@cptntommy #BrightonSEO

It’s Fairly Awesome
@cptntommy #BrightonSEO

Allows You To
Monitor A Site In
Real-Time
@cptntommy #BrightonSEO

With It Letting you
Know Of Any Issues
@cptntommy #BrightonSEO

Meta Changes, New
404 Errors, Updated
Links….
@cptntommy #BrightonSEO

BUT
@cptntommy #BrightonSEO

Like Most Good Tools,
It Costs Money
@cptntommy #BrightonSEO

Maybe You
Don’t Wanna
Eat Into Your
Budget
@cptntommy #BrightonSEO

This Next Example
Shows How We Can
Use Puppeteer
@cptntommy #BrightonSEO

Monitor Your Site
When You Want
&
Report Of Any
Changes To Key Areas
@cptntommy #BrightonSEO

Including
@cptntommy #BrightonSEO

Title Changes
@cptntommy #BrightonSEO

Description Changes
@cptntommy #BrightonSEO

Word Count
Increases/Decreases
@cptntommy #BrightonSEO

Robots Directives
@cptntommy #BrightonSEO

Canonicals
@cptntommy #BrightonSEO

So Basically The
REALLY Important
Shit In The HTML
@cptntommy #BrightonSEO

So I Wrote Some Code
@cptntommy #BrightonSEO

As With All Code, Required A
Bit Of Research
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

And With A Bit Of Luck,
@cptntommy #BrightonSEO

We Now Have A Way To
Monitor Basic Areas Of Sites!
@cptntommy #BrightonSEO

So.
@cptntommy #BrightonSEO

There Is About 200 Lines Of
Code
@cptntommy #BrightonSEO

@cptntommy #BrightonSEO

And I Don’t Have Time To Go
Through The Full Thing
@cptntommy #BrightonSEO

But
@cptntommy #BrightonSEO

There Are A Few Interesting
Snippets I’d Like To Share
@cptntommy #BrightonSEO

We Launch Headless Chrome
& Puppeteer As Highlighted A
Minute Ago
@cptntommy #BrightonSEO

const browser = await
puppeteer.launch();
const page = await
browser.newPage();
@cptntommy #BrightonSEO

Provide A List Of URLs For
Puppeteer To Go And Play
With
@cptntommy #BrightonSEO

try {data =
fs.readFileSync('/Users/tomp
ool/Desktop/PuppeteerRender
ing/PageMonitor/urls.txt','utf
8');}
@cptntommy #Brig...

And Then Pull Relevant Meta
Data
@cptntommy #BrightonSEO

For Example
@cptntommy #BrightonSEO

Meta Title
@cptntommy #BrightonSEO

try {title = await page.title();}
catch (e1) {title = 'n/a';}
@cptntommy #BrightonSEO

Then Create An Array Of All
The Meta Data
@cptntommy #BrightonSEO

let retArray =
[date,url,title,description
,canonical,robots,wordC
ount];
@cptntommy #BrightonSEO

And Pushed This To A txt File
@cptntommy #BrightonSEO

The Script Then Loops
Through All Provided URLs
@cptntommy #BrightonSEO

And Checks For Differences In
The Returned Data
@cptntommy #BrightonSEO

If There Are Any Differences,
These Get Saved In Another
txt File
@cptntommy #BrightonSEO

That I Can Check Whenever
@cptntommy #BrightonSEO

So I Can See What Has
Changed From
Yesterday/When I Last Ran
The Code.
@cptntommy #BrightonSEO

This Required Me To Run The
Code Each Day
@cptntommy #BrightonSEO

(That I Forgot To Do)
@cptntommy #BrightonSEO

So I Went One Step Further
@cptntommy #BrightonSEO

Chucked It On A Raspberry Pi
@cptntommy #BrightonSEO

And Set Up A CronJob To
Automatically Run The Script
At The Same Time
@cptntommy #BrightonSEO

Every Day
@cptntommy #BrightonSEO

And Then
@cptntommy #BrightonSEO

(This Was The Longest Bit)
@cptntommy #BrightonSEO

Email Me If Anything Changed
@cptntommy #BrightonSEO

This Is By No Means A
Finished Product, And Is Still
An Ongoing Project
@cptntommy #BrightonSEO

These Usages Of Chrome
Puppeteer
@cptntommy #BrightonSEO

Barely Scratch The Surface Of
What Is Possible
@cptntommy #BrightonSEO

So, To Recap
@cptntommy #BrightonSEO

Today We Have Covered
@cptntommy #BrightonSEO

Headless Chrome
@cptntommy #BrightonSEO

Puppeteer
@cptntommy #BrightonSEO

Basic Scripts Using Node.js
@cptntommy #BrightonSEO

And Automation Of All Of
These To Save You Valuable
Time
@cptntommy #BrightonSEO

And Hopefully, Allow You To
@cptntommy #BrightonSEO

And Hopefully, Allow You To
@cptntommy #BrightonSEO

THANKS!
@cptntommy #BrightonSEO

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1036216 views
Storytelling For The Web: Integrate... by Chiara Aliotta 948112 views
Artificial Intelligence, Data and C... by OECD Directorate ... 870219 views
How to Leverage AI to Boost Employe... by SocialHRCamp 365506 views
2024 State of Marketing Report – by... by Marius Sescu 207694 views
Everything You Need To Know About C... by Expeed Software 227895 views

View on Slideshare

1 of 210 Ad

View on Slideshare

1 of 210 Ad