1.
Mike Osolinski
@Fearless_Shultz
Edit
CLI Automation – Using the
Command Line to
automate repetitive tasks
https://www.slideshare.net/MikeOsolinski

2.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
What the command line
is and how it helps us to
work more efficiently
Basic examples of how
time and effort can be
saved by using CLI Scripts
Chaining multiple scripts /
modules together to create
automated solutions
What I’m Going to Talk About.

3.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
@fearless_Shultz

4.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
@fearless_Shultz

5.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
The Command
Line and how it
helps us work
more efficiently.

6.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
A command-line interface or command
language interpreter, allows a user to
issues commands in the form of successive
lines of text

7.
@Fearless_Shultz #brightonSEO
What is the Command Line?
Command Prompt
http://bit.ly/windows-command-prompt

8.
@Fearless_Shultz #brightonSEO
What is the Command Line?
Terminal
http://bit.ly/open-mac-terminal

9.
@Fearless_Shultz #brightonSEO
What is the Command Line?
Bash
http://bit.ly/open-bash-linux

10.
@Fearless_Shultz #brightonSEO
What is the Command Line?
Cygwin
http://bit.ly/open-bash-linux

11.
@Fearless_Shultz #brightonSEO
What is the Command Line?

12.
@Fearless_Shultz #brightonSEO
What is the Command Line?

13.
@Fearless_Shultz #brightonSEO
What is the Command Line?

14.
@Fearless_Shultz #brightonSEO
What is the Command Line?

15.
@Fearless_Shultz #brightonSEO
What is the Command Line?

16.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Extremely powerful (in both good and bad
ways…)

17.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
“With great power comes great
responsibility….”
- Uncle Ben

18.
@Fearless_Shultz #brightonSEO@Fearless_Shultz #brightonSEO
Quick and easy to write ad-hoc scripts
that would otherwise take much longer

19.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Take advantage of a huge library of pre-
built scripts.

20.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Chain different scripts and libraries
together to create fully automated
solutions.

21.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Save time on
repetitive tasks.

22.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO

23.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Examples Of
Using the
Command Line to
Save Time.

24.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
• Merging Files
• Optimising Images
• Extracting Data
• Running Lighthouse Reports
• Running Screaming Frog Reports
• Renaming Files
• Manipulating CSV Files
* I know, I know, bullet points kill Kittens

25.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples

26.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .

27.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Create a new
master file
Open each
individual csv
Select and
copy and
paste the data
Double check
everything has
been copied

28.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .

29.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
copy *.csv > merged.csv
cat *.csv > merged.csv
Windows
Linux / Mac

30.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples

31.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples

32.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .

33.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Work through all files and
rename them

34.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .

35.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
for /f "Tokens=*" %f in ('dir /l/b/a-d') do (rename "%f" "%f")
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
Windows
Linux / Mac

36.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples

37.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples

38.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .

39.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Download
images via FTP
Upload in
batches to an
image
processing
service
Tweak image
settings as
required
Download
optimised images

40.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .

41.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
jpegoptim –size 500k *.jpg
wget -i large-image-urls.txt

42.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
https://github.com/tjko/jpegoptim

43.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples

44.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .

45.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Create a
new SF Log
File project
Import log
file data
Filter out
Googlebot
User Agent
Export the
data to
Excel

46.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .

47.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
cat *.log | grep Googlebot >new-file.csv
Select-String -Path "C:a_ex190207.log" -Pattern
'Googlebot'

48.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples

49.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .

50.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Install Lighthouse
extension
Open each URL
in a Browser
Select options
and run
performance
test
Once each
report has
finished running,
save as a HTML
file

51.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .

52.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
urls.txt

53.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
test.bat

54.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
test.bat

55.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
• This does require some additional modules installing but it is generally very quick and
easy to get set up. Full instructions here https://edit.co.uk/blog/save-time-by-batch-
processing-lighthouse-audit-files-from-the-command-line/

56.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples

57.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .

58.
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
Open
Screaming
Frog
Set
Configuration
Options
Wait For
Crawl to Run
Select Filter
100 KB
Images
Export the
Report

59.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .

60.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Basic Command Line Examples
cd "C:Program Files (x86)Screaming Frog SEO Spider"
ScreamingFrogSEOSpiderCli.exe ^
--crawl https://edit.co.uk ^
--headless ^
--save-crawl ^
--config “C:image-crawl.seospiderconfig" ^
--output-folder “C:scriptsScreamingFrog" ^
--export-tabs "Images:Over X KB"

61.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Introducing PowerShell…

62.
@Fearless_Shultz #brightonSEO
Introducing PowerShell
Scripting Panel
Command Line
cmdlet Library
PowerShell is a task-based command-line shell and scripting language built on .NET. PowerShell helps
system administrators and power-users rapidly automate tasks that manage operating systems (Linux,
macOS, and Windows) and processes

63.
@Fearless_Shultz #brightonSEO
Introducing PowerShell - cmdlets
A cmdlet (pronounced "command-let") is a lightweight Windows PowerShell script that performs a
single function.
Import-CSV Invoke-RestMethodExport-CSV

64.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv
The Import-CSV cmdlet turns a
CSV file into an object which
has properties which can easily
be referenced with filters
including:
-lt (Less than)
-gt (Greater than
-eq (Equal to)
-like (Like)

65.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv
Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'|
select Address, Status,'Title 1 Length'|
Where Address -like *services*|
where Status -eq 'OK'|
where 'Title 1 Length' -gt 55

66.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv

67.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Export-Csv
$myData = Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'|
select Address, Status,'Title 1 Length'|
Where Address -like *services*|
where Status -eq 'OK'|
where 'Title 1 Length' -gt 55
$myData | Export-CSv 'C:BrightonSEOscriptscsv examplesinternal_html-new.csv'

68.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Export-Csv

69.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv

70.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
You Could . . .

71.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv
Open redirect
report from SF
in Excel
Create filter to
find any rows
with a redirect
count of more
than one.
Create a new
column to
contain the
redirect
Use a
concatenation
formula to
prepend the
rule to the first
and last URLs
Remove all the
other columns
which aren’t
required and
save.

72.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Or . . .

73.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Import-Csv
$myObjectsArray = Import-Csv "C:powershell-scriptsredirect_and_canonical_chains.csv" |
Where { $_."Number of Redirects" -gt 1}
$newArray = @()
$myObjectsArray |
ForEach-Object{
$redirectType = "Redirect 301 "
$addressToRedirectFrom = $_.Address -replace
'^(?:https?://)?(?:[^@/n]+@)?(?:www.)?([^:/n]+)', ''
$addressToRedirectTo = $_.'Final Address'
$rewriteRule = $redirectType + $addressToRedirectFrom + ' ' + $addressToRedirectTo
$newArray += $rewriteRule
}
$newArray | Out-File "C:powershell-scriptstesting-export.csv"

74.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Invoke-RestMethod
Invoke-RestMethod consumes
REST APIs and parses the result
and where JSON is returned
the data is converted into a
PowerShell object that can be
easily queried.

75.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Invoke-RestMethod
$username = “user-name-here"
$password = “api-key-here"
$uri = "https://gtmetrix.com/api/0.1/test"
$testURL = "https://edit.co.uk"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}"
-f $username,$password)))
$testing = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f
$base64AuthInfo)} `
-Method Post -Uri $uri -Body @{url="https://edit.co.uk"} `
$testID = $testing.test_id
$testURLWithID = $testing.poll_state_url
$testing

76.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Invoke-RestMethod

77.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Invoke-RestMethod

78.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Invoke-RestMethod
$fullData = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f
$base64AuthInfo)} `
-Method GET -Uri $uri+$testURLWithID
$testScore = $fullData.resources.pagespeed.Length
$testDoMTime = $fullData.results.dom_content_loaded_time
$testFullyLoaded = $fullData.results.fully_loaded_time
'The test score is:'+ $testScore
'The DOM content loaded time is: ' +$testDoMTime
'The fully loaded time is: ' +$testFullyLoaded

79.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Invoke-RestMethod

80.
@Fearless_Shultz #brightonSEO
Introducing PowerShell – Invoke-RestMethod

81.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Linking things together

82.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Generate Redirect Report and Create
Redirects.

83.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
What is the Command Line?
cd "C:Program Files (x86)Screaming Frog SEO Spider"
ScreamingFrogSEOSpiderCli.exe ^
--crawl https://edit.co.uk/ ^
--headless ^
--save-crawl ^
--config "C:redirect.seospiderconfig" ^
--output-folder "C:redirects" ^
--save-report "Redirect & Canonical Chains"
chaining-redirects.bat

84.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
What is the Command Line?
cd C:redirects
C:redirectschaining-redirects.bat
Get-Content C:redirectsredirect_and_canonical_chains.csv |
Select-Object -Skip 1 | Out-File C:redirectsredirects.csv
$myObjectsArray = Import-Csv "C:redirectsredirects.csv" |
Where { $_."Number of Redirects" -gt 1}
$newArray = @()
$myObjectsArray |
ForEach-Object{
$rType = "Redirect 301 "
$rFrom = $_.Address -replace
'^(?:https?://)?(?:[^@/n]+@)?(?:www.)?([^:/n]+)', ‘’
$rTo = $_.'Final Address'
$rewriteRule = $rType + $rFrom + ' ' + $rTo
$newArray += $rewriteRule}
$newArray | Out-File "C:redirectsredirect-rules.csv"

85.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Generate Lighthouse Reports.

86.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
What is the Command Line?
cd "C:Program Files (x86)Screaming Frog SEO Spider"
ScreamingFrogSEOSpiderCli.exe ^
--crawl https://edit.co.uk ^
--headless ^
--save-crawl ^
--config “C:image-crawl.seospiderconfig" ^
--output-folder “C:scriptsScreamingFrog" ^
--export-tabs “Internal:ALL"
Chaining-lighthouse.bat

87.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
What is the Command Line?
@echo off
cd C:scriptsChainingLighthouselighthouse-reports
for /f "delims=" %%a in
(C:scriptsChainingLighthousetest-urls.txt) DO (
ECHO Line is: %%a
lighthouse --quiet --chrome-flags="--headless" %%a
)
Test.bat

88.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
What is the Command Line?
cd C:lighthouse
Get-Content C:scriptsScreamingFroginternal_html.csv |
Select-Object -Skip 1 | Out-File C:lighthouseurls-to-test.csv
$myObjectsArray = Import-Csv "C:lighthouseurls-to-test.csv"
C:Lighthousechaining-lighthouse.bat
$newArray = @()
$myObjectsArray |
ForEach-Object{
$newArray += $_.Address
}
$newArray | Out-File "C:lighthousetest-urls.txt" -Encoding ascii
C:Lighthousetest.bat

89.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
This is just the tip of the iceberg

90.
@Fearless_Shultz #brightonSEO
Confidential
@Fearless_Shultz #brightonSEO
Useful Resources
For Learning and
Working With CLI
Scripts.

91.
@Fearless_Shultz #brightonSEO
CLI Scripting Resources
Title Link
A beginners guide to batch scripting in
Windows http://steve-jansen.github.io/guides/windows-batch-scripting/
Distilled Beginners guide to command line for
SEO https://www.distilled.net/blog/seo/the-beginners-guide-to-using-the-command-line-for-seo/
SEO Strategies to impress your boss https://elite-strategies.com/impress-seo-boss-linux-command-line-tricks/
Beginners guide to Terminal on Mac https://www.makeuseof.com/tag/beginners-guide-mac-terminal/
Tom Pool CLI Hacks presentation http://bit.ly/tom-pool-command-line-hacks-presentation
Tom Pool CLI Cheat Sheet http://bit.ly/command-line-cheat-sheet
Tom Pool Podcast http://bit.ly/tom-pool-presentation-podcast
64 Resources for learning to code online https://skillcrush.com/2016/03/15/64-online-resources-to-learn-to-code-for-free/
The 100 days of code project https://www.100daysofcode.com/
PowerShell download packages https://github.com/PowerShell/PowerShell
Beginners guide to PowerShell https://blog.netwrix.com/2018/02/21/windows-powershell-scripting-tutorial-for-beginners/
Guide to using Arrays in Poweshell https://4sysops.com/archives/arrays-in-powershell-create-change-read-sort-delete/
PowerShell Foreach-Object https://www.youtube.com/watch?v=BoutUXsBGcU
PowerShell out-File https://stackoverflow.com/questions/19450616/export-csv-exports-length-but-not-name

92.
@Fearless_Shultz #brightonSEO
CLI Scripting Resources
Title Link
CSV kit for PowerShell https://searchenginewatch.com/2019/02/06/technical-seo-2019-statelessness/
Loop through files in directory in powershell
https://stackoverflow.com/questions/18847145/loop-through-files-in-a-directory-using-
powershell
Understanding invoke-rest method https://4sysops.com/archives/understanding-the-invoke-restmethod-powershell-cmdlet
Working With Csvs in PowerShell https://www.petri.com/powershell-import-csv-cmdlet-parse-comma-delimited-csv-text-file
Video example of connecting to REST Method https://www.youtube.com/watch?v=7mEmQgGowMY
PowerShell add-ons https://4sysops.com/archives/my-favorite-windows-powershell-ise-add-ons/
Powershell substrings
https://4sysops.com/archives/strings-in-powershell-replace-compare-concatenate-split-
substring/
Screaming Frog Command Scheduler
https://www.greenlanemarketing.com/blog/schedule-your-screaming-frog-crawls-with-
punctual-frog/
Screaming Frog Command Line
Documentation https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#commandlineoptions
Running Screaming Frog CLI On a Server https://www.databulle.com/blog/seo/screaming-frog-headless.html
GTMetrix API https://gtmetrix.com/api/
Cygwin to run Linux and Mac commands on
Windows https://www.cygwin.com/
Huge searchable DB of REST Apis https://www.programmableweb.com/apis/directory
Regex to strip out domain from URL (For
Redirects) https://stackoverflow.com/questions/4791700/regex-needed-to-strip-out-domain-name
Content and sentiment analysis https://www.pveller.com/content-work-automation-with-text-analytics/

93.
@Fearless_Shultz #brightonSEO
Closing Thoughts.

94.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Repetitive tasks are often PERFECT
candidates for automation

95.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
If a tool doesn’t already exist to perform a
task don’t assume you have to do things
the hard way. BUILD IT.

96.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Writing scripts is not as HARD as you might
imagine it to be.

97.
@Fearless_Shultz #brightonSEO
@Fearless_Shultz #brightonSEO
Remember the words of Uncle Ben and BE
CAREFULL!

98.
@Fearless_Shultz #brightonSEO
Thank You.