SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
A presentation discussing the use of command line automation to save time on repetitive SEO tasks. Covers basic command line usage and the use of PowerShell with a number of useful scripts that you can start using today. Presented at BrightonSEO in April 2019
A presentation discussing the use of command line automation to save time on repetitive SEO tasks. Covers basic command line usage and the use of PowerShell with a number of useful scripts that you can start using today. Presented at BrightonSEO in April 2019
1.
Mike Osolinski
@Fearless_Shultz
Edit
CLI Automation – Using the
Command Line to
automate repetitive tasks
https://www.slideshare.net/MikeOsolinski
2.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
What the command line
is and how it helps us to
work more efficiently
Basic examples of how
time and effort can be
saved by using CLI Scripts
Chaining multiple scripts /
modules together to create
automated solutions
What I’m Going to Talk About.
5.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
The Command
Line and how it
helps us work
more efficiently.
6.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
A command-line interface or command
language interpreter, allows a user to
issues commands in the form of successive
lines of text
7.
@Fearless_Shultz #brightonSEO
What is the Command Line?
Command Prompt
http://bit.ly/windows-command-prompt
8.
@Fearless_Shultz #brightonSEO
What is the Command Line?
Terminal
http://bit.ly/open-mac-terminal
9.
@Fearless_Shultz #brightonSEO
What is the Command Line?
Bash
http://bit.ly/open-bash-linux
10.
@Fearless_Shultz #brightonSEO
What is the Command Line?
Cygwin
http://bit.ly/open-bash-linux
11.
@Fearless_Shultz #brightonSEO
What is the Command Line?
12.
@Fearless_Shultz #brightonSEO
What is the Command Line?
13.
@Fearless_Shultz #brightonSEO
What is the Command Line?
14.
@Fearless_Shultz #brightonSEO
What is the Command Line?
15.
@Fearless_Shultz #brightonSEO
What is the Command Line?
16.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Extremely powerful (in both good and bad
ways…)
17.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
“With great power comes great
responsibility….”
- Uncle Ben
18.
@Fearless_Shultz #brightonSEO@Fearless_Shultz #brightonSEO
Quick and easy to write ad-hoc scripts
that would otherwise take much longer
19.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Take advantage of a huge library of pre-
built scripts.
20.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Chain different scripts and libraries
together to create fully automated
solutions.
21.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Save time on
repetitive tasks.
25.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
26.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .
27.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Create a new
master file
Open each
individual csv
Select and
copy and
paste the data
Double check
everything has
been copied
28.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .
29.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
copy *.csv > merged.csv
cat *.csv > merged.csv
Windows
Linux / Mac
30.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
31.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
32.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .
33.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Work through all files and
rename them
34.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .
35.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
for /f "Tokens=*" %f in ('dir /l/b/a-d') do (rename "%f" "%f")
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
Windows
Linux / Mac
36.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
37.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
38.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .
39.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Download
images via FTP
Upload in
batches to an
image
processing
service
Tweak image
settings as
required
Download
optimised images
40.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .
42.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
https://github.com/tjko/jpegoptim
43.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
44.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .
45.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Create a
new SF Log
File project
Import log
file data
Filter out
Googlebot
User Agent
Export the
data to
Excel
46.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .
48.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
49.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .
50.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Install Lighthouse
extension
Open each URL
in a Browser
Select options
and run
performance
test
Once each
report has
finished running,
save as a HTML
file
51.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .
52.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
urls.txt
53.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
test.bat
55.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
• This does require some additional modules installing but it is generally very quick and
easy to get set up. Full instructions here https://edit.co.uk/blog/save-time-by-batch-
processing-lighthouse-audit-files-from-the-command-line/
56.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
57.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .
58.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Open
Screaming
Frog
Set
Configuration
Options
Wait For
Crawl to Run
Select Filter
100 KB
Images
Export the
Report
59.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .
60.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
cd "C:Program Files (x86)Screaming Frog SEO Spider"
ScreamingFrogSEOSpiderCli.exe ^
--crawl https://edit.co.uk ^
--headless ^
--save-crawl ^
--config “C:image-crawl.seospiderconfig" ^
--output-folder “C:scriptsScreamingFrog" ^
--export-tabs "Images:Over X KB"
62.
@Fearless_Shultz #brightonSEO
Introducing PowerShell
Scripting Panel
Command Line
cmdlet Library
PowerShell is a task-based command-line shell and scripting language built on .NET. PowerShell helps
system administrators and power-users rapidly automate tasks that manage operating systems (Linux,
macOS, and Windows) and processes
63.
@Fearless_Shultz #brightonSEO
Introducing PowerShell - cmdlets
A cmdlet (pronounced "command-let") is a lightweight Windows PowerShell script that performs a
single function.
Import-CSV Invoke-RestMethodExport-CSV
64.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv
The Import-CSV cmdlet turns a
CSV file into an object which
has properties which can easily
be referenced with filters
including:
-lt (Less than)
-gt (Greater than
-eq (Equal to)
-like (Like)
65.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv
Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'|
select Address, Status,'Title 1 Length'|
Where Address -like *services*|
where Status -eq 'OK'|
where 'Title 1 Length' -gt 55
70.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .
71.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv
Open redirect
report from SF
in Excel
Create filter to
find any rows
with a redirect
count of more
than one.
Create a new
column to
contain the
redirect
Use a
concatenation
formula to
prepend the
rule to the first
and last URLs
Remove all the
other columns
which aren’t
required and
save.
72.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .
74.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Invoke-RestMethod
Invoke-RestMethod consumes
REST APIs and parses the result
and where JSON is returned
the data is converted into a
PowerShell object that can be
easily queried.
86.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
What is the Command Line?
cd "C:Program Files (x86)Screaming Frog SEO Spider"
ScreamingFrogSEOSpiderCli.exe ^
--crawl https://edit.co.uk ^
--headless ^
--save-crawl ^
--config “C:image-crawl.seospiderconfig" ^
--output-folder “C:scriptsScreamingFrog" ^
--export-tabs “Internal:ALL"
Chaining-lighthouse.bat
87.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
What is the Command Line?
@echo off
cd C:scriptsChainingLighthouselighthouse-reports
for /f "delims=" %%a in
(C:scriptsChainingLighthousetest-urls.txt) DO (
ECHO Line is: %%a
lighthouse --quiet --chrome-flags="--headless" %%a
)
Test.bat
89.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
This is just the tip of the iceberg
90.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Useful Resources
For Learning and
Working With CLI
Scripts.
91.
@Fearless_Shultz #brightonSEO
CLI Scripting Resources
Title Link
A beginners guide to batch scripting in
Windows http://steve-jansen.github.io/guides/windows-batch-scripting/
Distilled Beginners guide to command line for
SEO https://www.distilled.net/blog/seo/the-beginners-guide-to-using-the-command-line-for-seo/
SEO Strategies to impress your boss https://elite-strategies.com/impress-seo-boss-linux-command-line-tricks/
Beginners guide to Terminal on Mac https://www.makeuseof.com/tag/beginners-guide-mac-terminal/
Tom Pool CLI Hacks presentation http://bit.ly/tom-pool-command-line-hacks-presentation
Tom Pool CLI Cheat Sheet http://bit.ly/command-line-cheat-sheet
Tom Pool Podcast http://bit.ly/tom-pool-presentation-podcast
64 Resources for learning to code online https://skillcrush.com/2016/03/15/64-online-resources-to-learn-to-code-for-free/
The 100 days of code project https://www.100daysofcode.com/
PowerShell download packages https://github.com/PowerShell/PowerShell
Beginners guide to PowerShell https://blog.netwrix.com/2018/02/21/windows-powershell-scripting-tutorial-for-beginners/
Guide to using Arrays in Poweshell https://4sysops.com/archives/arrays-in-powershell-create-change-read-sort-delete/
PowerShell Foreach-Object https://www.youtube.com/watch?v=BoutUXsBGcU
PowerShell out-File https://stackoverflow.com/questions/19450616/export-csv-exports-length-but-not-name
92.
@Fearless_Shultz #brightonSEO
CLI Scripting Resources
Title Link
CSV kit for PowerShell https://searchenginewatch.com/2019/02/06/technical-seo-2019-statelessness/
Loop through files in directory in powershell
https://stackoverflow.com/questions/18847145/loop-through-files-in-a-directory-using-
powershell
Understanding invoke-rest method https://4sysops.com/archives/understanding-the-invoke-restmethod-powershell-cmdlet
Working With Csvs in PowerShell https://www.petri.com/powershell-import-csv-cmdlet-parse-comma-delimited-csv-text-file
Video example of connecting to REST Method https://www.youtube.com/watch?v=7mEmQgGowMY
PowerShell add-ons https://4sysops.com/archives/my-favorite-windows-powershell-ise-add-ons/
Powershell substrings
https://4sysops.com/archives/strings-in-powershell-replace-compare-concatenate-split-
substring/
Screaming Frog Command Scheduler
https://www.greenlanemarketing.com/blog/schedule-your-screaming-frog-crawls-with-
punctual-frog/
Screaming Frog Command Line
Documentation https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#commandlineoptions
Running Screaming Frog CLI On a Server https://www.databulle.com/blog/seo/screaming-frog-headless.html
GTMetrix API https://gtmetrix.com/api/
Cygwin to run Linux and Mac commands on
Windows https://www.cygwin.com/
Huge searchable DB of REST Apis https://www.programmableweb.com/apis/directory
Regex to strip out domain from URL (For
Redirects) https://stackoverflow.com/questions/4791700/regex-needed-to-strip-out-domain-name
Content and sentiment analysis https://www.pveller.com/content-work-automation-with-text-analytics/
94.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Repetitive tasks are often PERFECT
candidates for automation
95.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
If a tool doesn’t already exist to perform a
task don’t assume you have to do things
the hard way. BUILD IT.
96.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Writing scripts is not as HARD as you might
imagine it to be.
97.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Remember the words of Uncle Ben and BE
CAREFULL!