Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

Tom Pool
BlueArray
Command Line Hacks For SEO
@cptntommy
https://www.slideshare.net/TomPool

@cptntommy
Technical SEO Manager
@ BlueArray
#BrightonSEO @cptntommy

Look after
technical
output for all
clients @
BlueArray
@cptntommy@cptntommy

This means I get to work on
loads of different clients, doing
loads of different tasks

Checking Response
Codes

Tech Audits

Analysis Of

Keyword Gap

Server Log Files

Crawl Data

Command Line Hacks For SEO
Insert picture of hours ticking away?
Full screen
Each of
these tasks
takes a lot
of time

Command Line
Hacks for SEO
Terminal

What Is Command
Line?

It is a basic interface between
you and the computer

The following references are
related to Mac OS’s Terminal
application.

What Commands Do I
Use?

CURL
SORT
CAT
SPLIT
SED
AWK

Keyword Analysis
Keyword Gap Analysis
Checking Response Codes
Log File Analysis
Crawling & Analysis

What is curl?

“curl is a tool to transfer data
from or to a server”

“curl is a tool to transfer data
from or to a server”
*Examples shown are not the full usage, for full info check out the manual page
(man curl)

Checking Response Codes

Toms-MacBook-Pro:~ tompool$
https://www.bluearray.co.uk

https://www.bluearray.co.ukcurl

https://www.bluearray.co.ukhttps://www.bluearray.co.ukcurl

What If I Just Want To See The
HTTP Header?
@cptntommy

curl -I
Modifier - Just The HTTP
Header!

curl -I https://bluearray.co.uk

curl -I -L

curl -I -L https://bluearray.co.uk

curl-O
https://download.screami
ngfrog.co.uk/products/seo
-spider/ScreamingFrogSEO
Spider-8.3.dmg

Here we have used CURL TO:
Download Files,
Check HTML,
Check HTTP Header &
Follow Redirects

Sort ‘sorts’

Navigate to Folder and use ‘ls - “list”’
command to make sure the data is
there

Then run the sort command

filename.csvsort

“sort keyworddata.csv”

sort filename > newfilename

Did It Work?

WTF is Head/Tail?

‘Head’ views the first 10 Rows
‘Tail’ views the last 10 Rows

head filename.csv

head filename

sort filename.csv >
Z-A_SortedData.csv

sort filename.csv >
Z-A_SortedData.csv
-r

We want to sort by the
second column

sort -k2 -t,

sort -k2 -t, -n

sort -k2 -t, -n -r

sort -k2 -t, -n -r filename.csv

sort -k2 -t, -n -r filename.csv >
volumesorteddata.csv

head volumesorteddata.csv >
top10KW.csv

head -n100
volumesorteddata.csv >
top100KW.csv

Here sort has been used to:
Sort by Number,
Sort by A-Z,
Sort by Z-A

We have also used head/tail to;
Extract the top 10,
Top 100
Save these to a file

cat - short for “concatenate”

Used to display, combine &
create files

cat *.csv

cat *.csv > alldata.csv

>
User/Desktop/DataToCombine

Short for "stream editor",
allows you to filter and
transform text.

Adding text to rows

Use sed to add protocol &
domain at start of each line

sed -e
's/^/http://www.domain
.com/' file.csv

sed -e 's/^/

sed -e
's/^/http://www.domain.co
m/'

sed -e
.com/'

sed -e
.com/' file.csv > full.csv

use sed to find & replace

Replace http with https

sed -e 's/http file.csv
> newfile.csv
/https/'

Here we have:
Added in protocol & domain
Replaced http with https

Programming language,
used to process text

Following example is one of
the easier applications of
awk

awk -F “,”

awk -F “,”‘{print $1 “,” $2}’

awk -F “,”‘{print $1 “,” $2}’ kw.csv
> betterkw.csv

awk -F “,”‘{print $1 “,” $2 “,” $4}’

How to
combine
all this?

@cptntommy
About 1%
of given
Log Files

@cptntommy
1.cat
Combine
files
together

@cptntommy
Cat *.log > combinedlogs.log

@cptntommy
2. Use sed to add in full domain
name to log file

@cptntommy
sed -e 's //
file.log > newfile.log
/domain.com/'

@cptntommy
sed -e 's //
Forward Slash

@cptntommy
sed -e 's // /domain.com/'
Text to replace with

@cptntommy
sed -e 's //
file.log > newfile.log
/domain.com/'
Then the files

@cptntommy
3.Extract 404 errors
(awk)

@cptntommy
awk '/404/ {print $0}'
combinedlogs.log > log404.log

@cptntommy
4.Extract any
specific
status code
(awk)

@cptntommy
awk '/301/ {print $0}' file.log >
newfile.log

@cptntommy
newfile.log

@cptntommy
5. Extract all
Googlebot/
Mobile Bot
hits (awk)

@cptntommy
awk '/Googlebot/ {print $0}'
all.log > gbot.log

@cptntommy
What if you wanted to know
how many hits each bot had,
without opening the file?

@cptntommy
6. Extract all hits of a specific
URL, or all img requests, all css
requests, all with modification of
1 command

@cptntommy
Replace ‘Googlebot’ with
‘BingBot’, ‘anybot’...

@cptntommy
awk '/Bingbot/ {print $0}' all.log >
gbot.log

@cptntommy
awk '/anybot/ {print $0}' all.log >
gbot.log

@cptntommy
awk '/.css/ {print $0}' all.log >
gbot.log

@cptntommy
awk '/.jpeg/ {print $0}' all.log >
gbot.log

@cptntommy
awk '/whatever-you-want/ {print
$0}' all.log > file.log

@cptntommy
This Barely
Scratches
the
Surface

Extract columns
of data
@cptntommy

Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

Similar to Command Line Hacks For SEO - Brighton April 2018 - Tom Pool (20)

Recently uploaded

Recently uploaded (20)

Command Line Hacks For SEO - Brighton April 2018 - Tom Pool