Command Line Automation for Repetitive Tasks

1. Mike Osolinski @Fearless_Shultz Edit CLI Automation – Using the Command Line to automate repetitive tasks https://www.slideshare.net/MikeOsolinski

2. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO What the command line is and how it helps us to work more efficiently Basic examples of how time and effort can be saved by using CLI Scripts Chaining multiple scripts / modules together to create automated solutions What I’m Going to Talk About.

3. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples @fearless_Shultz

4. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples @fearless_Shultz

5. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO The Command Line and how it helps us work more efficiently.

6. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO A command-line interface or command language interpreter, allows a user to issues commands in the form of successive lines of text

7. @Fearless_Shultz #brightonSEO What is the Command Line? Command Prompt http://bit.ly/windows-command-prompt

8. @Fearless_Shultz #brightonSEO What is the Command Line? Terminal http://bit.ly/open-mac-terminal

9. @Fearless_Shultz #brightonSEO What is the Command Line? Bash http://bit.ly/open-bash-linux

10. @Fearless_Shultz #brightonSEO What is the Command Line? Cygwin http://bit.ly/open-bash-linux

11. @Fearless_Shultz #brightonSEO What is the Command Line?

16. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Extremely powerful (in both good and bad ways…)

17. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO “With great power comes great responsibility….” - Uncle Ben

18. @Fearless_Shultz #brightonSEO@Fearless_Shultz #brightonSEO Quick and easy to write ad-hoc scripts that would otherwise take much longer

19. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Take advantage of a huge library of pre- built scripts.

20. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Chain different scripts and libraries together to create fully automated solutions.

21. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Save time on repetitive tasks.

22. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO

23. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Examples Of Using the Command Line to Save Time.

24. @Fearless_Shultz #brightonSEO Basic Command Line Examples • Merging Files • Optimising Images • Extracting Data • Running Lighthouse Reports • Running Screaming Frog Reports • Renaming Files • Manipulating CSV Files * I know, I know, bullet points kill Kittens

25. @Fearless_Shultz #brightonSEO Basic Command Line Examples

26. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO You Could . . .

27. @Fearless_Shultz #brightonSEO Basic Command Line Examples Create a new master file Open each individual csv Select and copy and paste the data Double check everything has been copied

28. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Or . . .

29. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples copy *.csv > merged.csv cat *.csv > merged.csv Windows Linux / Mac

33. @Fearless_Shultz #brightonSEO Basic Command Line Examples Work through all files and rename them

35. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples for /f "Tokens=*" %f in ('dir /l/b/a-d') do (rename "%f" "%f") for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done Windows Linux / Mac

39. @Fearless_Shultz #brightonSEO Basic Command Line Examples Download images via FTP Upload in batches to an image processing service Tweak image settings as required Download optimised images

41. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples jpegoptim –size 500k *.jpg wget -i large-image-urls.txt

42. @Fearless_Shultz #brightonSEO Basic Command Line Examples https://github.com/tjko/jpegoptim

45. @Fearless_Shultz #brightonSEO Basic Command Line Examples Create a new SF Log File project Import log file data Filter out Googlebot User Agent Export the data to Excel

47. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples cat *.log | grep Googlebot >new-file.csv Select-String -Path "C:a_ex190207.log" -Pattern 'Googlebot'

50. @Fearless_Shultz #brightonSEO Basic Command Line Examples Install Lighthouse extension Open each URL in a Browser Select options and run performance test Once each report has finished running, save as a HTML file

52. @Fearless_Shultz #brightonSEO Basic Command Line Examples urls.txt

53. @Fearless_Shultz #brightonSEO Basic Command Line Examples test.bat

54. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples test.bat

55. @Fearless_Shultz #brightonSEO Basic Command Line Examples • This does require some additional modules installing but it is generally very quick and easy to get set up. Full instructions here https://edit.co.uk/blog/save-time-by-batch- processing-lighthouse-audit-files-from-the-command-line/

58. @Fearless_Shultz #brightonSEO Basic Command Line Examples Open Screaming Frog Set Configuration Options Wait For Crawl to Run Select Filter 100 KB Images Export the Report

60. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples cd "C:Program Files (x86)Screaming Frog SEO Spider" ScreamingFrogSEOSpiderCli.exe ^ --crawl https://edit.co.uk ^ --headless ^ --save-crawl ^ --config “C:image-crawl.seospiderconfig" ^ --output-folder “C:scriptsScreamingFrog" ^ --export-tabs "Images:Over X KB"

61. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Introducing PowerShell…

62. @Fearless_Shultz #brightonSEO Introducing PowerShell Scripting Panel Command Line cmdlet Library PowerShell is a task-based command-line shell and scripting language built on .NET. PowerShell helps system administrators and power-users rapidly automate tasks that manage operating systems (Linux, macOS, and Windows) and processes

63. @Fearless_Shultz #brightonSEO Introducing PowerShell - cmdlets A cmdlet (pronounced "command-let") is a lightweight Windows PowerShell script that performs a single function. Import-CSV Invoke-RestMethodExport-CSV

64. @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv The Import-CSV cmdlet turns a CSV file into an object which has properties which can easily be referenced with filters including: -lt (Less than) -gt (Greater than -eq (Equal to) -like (Like)

65. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'| select Address, Status,'Title 1 Length'| Where Address -like *services*| where Status -eq 'OK'| where 'Title 1 Length' -gt 55

66. @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv

67. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Export-Csv $myData = Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'| select Address, Status,'Title 1 Length'| Where Address -like *services*| where Status -eq 'OK'| where 'Title 1 Length' -gt 55 $myData | Export-CSv 'C:BrightonSEOscriptscsv examplesinternal_html-new.csv'

68. @Fearless_Shultz #brightonSEO Introducing PowerShell – Export-Csv

69. @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv

71. @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv Open redirect report from SF in Excel Create filter to find any rows with a redirect count of more than one. Create a new column to contain the redirect Use a concatenation formula to prepend the rule to the first and last URLs Remove all the other columns which aren’t required and save.

73. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv $myObjectsArray = Import-Csv "C:powershell-scriptsredirect_and_canonical_chains.csv" | Where { $_."Number of Redirects" -gt 1} $newArray = @() $myObjectsArray | ForEach-Object{ $redirectType = "Redirect 301 " $addressToRedirectFrom = $_.Address -replace '^(?:https?://)?(?:[^@/n]+@)?(?:www.)?([^:/n]+)', '' $addressToRedirectTo = $_.'Final Address' $rewriteRule = $redirectType + $addressToRedirectFrom + ' ' + $addressToRedirectTo $newArray += $rewriteRule } $newArray | Out-File "C:powershell-scriptstesting-export.csv"

74. @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod Invoke-RestMethod consumes REST APIs and parses the result and where JSON is returned the data is converted into a PowerShell object that can be easily queried.

75. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod $username = “user-name-here" $password = “api-key-here" $uri = "https://gtmetrix.com/api/0.1/test" $testURL = "https://edit.co.uk" $base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password))) $testing = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} ` -Method Post -Uri $uri -Body @{url="https://edit.co.uk"} ` $testID = $testing.test_id $testURLWithID = $testing.poll_state_url $testing

76. @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod

78. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod $fullData = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} ` -Method GET -Uri $uri+$testURLWithID $testScore = $fullData.resources.pagespeed.Length $testDoMTime = $fullData.results.dom_content_loaded_time $testFullyLoaded = $fullData.results.fully_loaded_time 'The test score is:'+ $testScore 'The DOM content loaded time is: ' +$testDoMTime 'The fully loaded time is: ' +$testFullyLoaded

81. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Linking things together

82. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Generate Redirect Report and Create Redirects.

83. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? cd "C:Program Files (x86)Screaming Frog SEO Spider" ScreamingFrogSEOSpiderCli.exe ^ --crawl https://edit.co.uk/ ^ --headless ^ --save-crawl ^ --config "C:redirect.seospiderconfig" ^ --output-folder "C:redirects" ^ --save-report "Redirect & Canonical Chains" chaining-redirects.bat

84. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? cd C:redirects C:redirectschaining-redirects.bat Get-Content C:redirectsredirect_and_canonical_chains.csv | Select-Object -Skip 1 | Out-File C:redirectsredirects.csv $myObjectsArray = Import-Csv "C:redirectsredirects.csv" | Where { $_."Number of Redirects" -gt 1} $newArray = @() $myObjectsArray | ForEach-Object{ $rType = "Redirect 301 " $rFrom = $_.Address -replace '^(?:https?://)?(?:[^@/n]+@)?(?:www.)?([^:/n]+)', ‘’ $rTo = $_.'Final Address' $rewriteRule = $rType + $rFrom + ' ' + $rTo $newArray += $rewriteRule} $newArray | Out-File "C:redirectsredirect-rules.csv"

85. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Generate Lighthouse Reports.

86. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? cd "C:Program Files (x86)Screaming Frog SEO Spider" ScreamingFrogSEOSpiderCli.exe ^ --crawl https://edit.co.uk ^ --headless ^ --save-crawl ^ --config “C:image-crawl.seospiderconfig" ^ --output-folder “C:scriptsScreamingFrog" ^ --export-tabs “Internal:ALL" Chaining-lighthouse.bat

87. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? @echo off cd C:scriptsChainingLighthouselighthouse-reports for /f "delims=" %%a in (C:scriptsChainingLighthousetest-urls.txt) DO ( ECHO Line is: %%a lighthouse --quiet --chrome-flags="--headless" %%a ) Test.bat

88. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? cd C:lighthouse Get-Content C:scriptsScreamingFroginternal_html.csv | Select-Object -Skip 1 | Out-File C:lighthouseurls-to-test.csv $myObjectsArray = Import-Csv "C:lighthouseurls-to-test.csv" C:Lighthousechaining-lighthouse.bat $newArray = @() $myObjectsArray | ForEach-Object{ $newArray += $_.Address } $newArray | Out-File "C:lighthousetest-urls.txt" -Encoding ascii C:Lighthousetest.bat

89. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO This is just the tip of the iceberg

90. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Useful Resources For Learning and Working With CLI Scripts.

91. @Fearless_Shultz #brightonSEO CLI Scripting Resources Title Link A beginners guide to batch scripting in Windows http://steve-jansen.github.io/guides/windows-batch-scripting/ Distilled Beginners guide to command line for SEO https://www.distilled.net/blog/seo/the-beginners-guide-to-using-the-command-line-for-seo/ SEO Strategies to impress your boss https://elite-strategies.com/impress-seo-boss-linux-command-line-tricks/ Beginners guide to Terminal on Mac https://www.makeuseof.com/tag/beginners-guide-mac-terminal/ Tom Pool CLI Hacks presentation http://bit.ly/tom-pool-command-line-hacks-presentation Tom Pool CLI Cheat Sheet http://bit.ly/command-line-cheat-sheet Tom Pool Podcast http://bit.ly/tom-pool-presentation-podcast 64 Resources for learning to code online https://skillcrush.com/2016/03/15/64-online-resources-to-learn-to-code-for-free/ The 100 days of code project https://www.100daysofcode.com/ PowerShell download packages https://github.com/PowerShell/PowerShell Beginners guide to PowerShell https://blog.netwrix.com/2018/02/21/windows-powershell-scripting-tutorial-for-beginners/ Guide to using Arrays in Poweshell https://4sysops.com/archives/arrays-in-powershell-create-change-read-sort-delete/ PowerShell Foreach-Object https://www.youtube.com/watch?v=BoutUXsBGcU PowerShell out-File https://stackoverflow.com/questions/19450616/export-csv-exports-length-but-not-name

92. @Fearless_Shultz #brightonSEO CLI Scripting Resources Title Link CSV kit for PowerShell https://searchenginewatch.com/2019/02/06/technical-seo-2019-statelessness/ Loop through files in directory in powershell https://stackoverflow.com/questions/18847145/loop-through-files-in-a-directory-using- powershell Understanding invoke-rest method https://4sysops.com/archives/understanding-the-invoke-restmethod-powershell-cmdlet Working With Csvs in PowerShell https://www.petri.com/powershell-import-csv-cmdlet-parse-comma-delimited-csv-text-file Video example of connecting to REST Method https://www.youtube.com/watch?v=7mEmQgGowMY PowerShell add-ons https://4sysops.com/archives/my-favorite-windows-powershell-ise-add-ons/ Powershell substrings https://4sysops.com/archives/strings-in-powershell-replace-compare-concatenate-split- substring/ Screaming Frog Command Scheduler https://www.greenlanemarketing.com/blog/schedule-your-screaming-frog-crawls-with- punctual-frog/ Screaming Frog Command Line Documentation https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#commandlineoptions Running Screaming Frog CLI On a Server https://www.databulle.com/blog/seo/screaming-frog-headless.html GTMetrix API https://gtmetrix.com/api/ Cygwin to run Linux and Mac commands on Windows https://www.cygwin.com/ Huge searchable DB of REST Apis https://www.programmableweb.com/apis/directory Regex to strip out domain from URL (For Redirects) https://stackoverflow.com/questions/4791700/regex-needed-to-strip-out-domain-name Content and sentiment analysis https://www.pveller.com/content-work-automation-with-text-analytics/

93. @Fearless_Shultz #brightonSEO Closing Thoughts.

94. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Repetitive tasks are often PERFECT candidates for automation

95. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO If a tool doesn’t already exist to perform a task don’t assume you have to do things the hard way. BUILD IT.

96. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Writing scripts is not as HARD as you might imagine it to be.

97. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Remember the words of Uncle Ben and BE CAREFULL!

98. @Fearless_Shultz #brightonSEO Thank You.