Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Command Line Automation for Repetitive Tasks

4,665 views

Published on

A presentation discussing the use of command line automation to save time on repetitive SEO tasks. Covers basic command line usage and the use of PowerShell with a number of useful scripts that you can start using today. Presented at BrightonSEO in April 2019

Published in: Technology

Command Line Automation for Repetitive Tasks

  1. 1. Mike Osolinski @Fearless_Shultz Edit CLI Automation – Using the Command Line to automate repetitive tasks https://www.slideshare.net/MikeOsolinski
  2. 2. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO What the command line is and how it helps us to work more efficiently Basic examples of how time and effort can be saved by using CLI Scripts Chaining multiple scripts / modules together to create automated solutions What I’m Going to Talk About.
  3. 3. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples @fearless_Shultz
  4. 4. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples @fearless_Shultz
  5. 5. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO The Command Line and how it helps us work more efficiently.
  6. 6. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO A command-line interface or command language interpreter, allows a user to issues commands in the form of successive lines of text
  7. 7. @Fearless_Shultz #brightonSEO What is the Command Line? Command Prompt http://bit.ly/windows-command-prompt
  8. 8. @Fearless_Shultz #brightonSEO What is the Command Line? Terminal http://bit.ly/open-mac-terminal
  9. 9. @Fearless_Shultz #brightonSEO What is the Command Line? Bash http://bit.ly/open-bash-linux
  10. 10. @Fearless_Shultz #brightonSEO What is the Command Line? Cygwin http://bit.ly/open-bash-linux
  11. 11. @Fearless_Shultz #brightonSEO What is the Command Line?
  12. 12. @Fearless_Shultz #brightonSEO What is the Command Line?
  13. 13. @Fearless_Shultz #brightonSEO What is the Command Line?
  14. 14. @Fearless_Shultz #brightonSEO What is the Command Line?
  15. 15. @Fearless_Shultz #brightonSEO What is the Command Line?
  16. 16. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Extremely powerful (in both good and bad ways…)
  17. 17. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO “With great power comes great responsibility….” - Uncle Ben
  18. 18. @Fearless_Shultz #brightonSEO@Fearless_Shultz #brightonSEO Quick and easy to write ad-hoc scripts that would otherwise take much longer
  19. 19. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Take advantage of a huge library of pre- built scripts.
  20. 20. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Chain different scripts and libraries together to create fully automated solutions.
  21. 21. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Save time on repetitive tasks.
  22. 22. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO
  23. 23. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Examples Of Using the Command Line to Save Time.
  24. 24. @Fearless_Shultz #brightonSEO Basic Command Line Examples • Merging Files • Optimising Images • Extracting Data • Running Lighthouse Reports • Running Screaming Frog Reports • Renaming Files • Manipulating CSV Files * I know, I know, bullet points kill Kittens
  25. 25. @Fearless_Shultz #brightonSEO Basic Command Line Examples
  26. 26. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO You Could . . .
  27. 27. @Fearless_Shultz #brightonSEO Basic Command Line Examples Create a new master file Open each individual csv Select and copy and paste the data Double check everything has been copied
  28. 28. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Or . . .
  29. 29. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples copy *.csv > merged.csv cat *.csv > merged.csv Windows Linux / Mac
  30. 30. @Fearless_Shultz #brightonSEO Basic Command Line Examples
  31. 31. @Fearless_Shultz #brightonSEO Basic Command Line Examples
  32. 32. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO You Could . . .
  33. 33. @Fearless_Shultz #brightonSEO Basic Command Line Examples Work through all files and rename them
  34. 34. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Or . . .
  35. 35. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples for /f "Tokens=*" %f in ('dir /l/b/a-d') do (rename "%f" "%f") for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done Windows Linux / Mac
  36. 36. @Fearless_Shultz #brightonSEO Basic Command Line Examples
  37. 37. @Fearless_Shultz #brightonSEO Basic Command Line Examples
  38. 38. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO You Could . . .
  39. 39. @Fearless_Shultz #brightonSEO Basic Command Line Examples Download images via FTP Upload in batches to an image processing service Tweak image settings as required Download optimised images
  40. 40. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Or . . .
  41. 41. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples jpegoptim –size 500k *.jpg wget -i large-image-urls.txt
  42. 42. @Fearless_Shultz #brightonSEO Basic Command Line Examples https://github.com/tjko/jpegoptim
  43. 43. @Fearless_Shultz #brightonSEO Basic Command Line Examples
  44. 44. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO You Could . . .
  45. 45. @Fearless_Shultz #brightonSEO Basic Command Line Examples Create a new SF Log File project Import log file data Filter out Googlebot User Agent Export the data to Excel
  46. 46. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Or . . .
  47. 47. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples cat *.log | grep Googlebot >new-file.csv Select-String -Path "C:a_ex190207.log" -Pattern 'Googlebot'
  48. 48. @Fearless_Shultz #brightonSEO Basic Command Line Examples
  49. 49. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO You Could . . .
  50. 50. @Fearless_Shultz #brightonSEO Basic Command Line Examples Install Lighthouse extension Open each URL in a Browser Select options and run performance test Once each report has finished running, save as a HTML file
  51. 51. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Or . . .
  52. 52. @Fearless_Shultz #brightonSEO Basic Command Line Examples urls.txt
  53. 53. @Fearless_Shultz #brightonSEO Basic Command Line Examples test.bat
  54. 54. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples test.bat
  55. 55. @Fearless_Shultz #brightonSEO Basic Command Line Examples • This does require some additional modules installing but it is generally very quick and easy to get set up. Full instructions here https://edit.co.uk/blog/save-time-by-batch- processing-lighthouse-audit-files-from-the-command-line/
  56. 56. @Fearless_Shultz #brightonSEO Basic Command Line Examples
  57. 57. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO You Could . . .
  58. 58. @Fearless_Shultz #brightonSEO Basic Command Line Examples Open Screaming Frog Set Configuration Options Wait For Crawl to Run Select Filter 100 KB Images Export the Report
  59. 59. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Or . . .
  60. 60. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Basic Command Line Examples cd "C:Program Files (x86)Screaming Frog SEO Spider" ScreamingFrogSEOSpiderCli.exe ^ --crawl https://edit.co.uk ^ --headless ^ --save-crawl ^ --config “C:image-crawl.seospiderconfig" ^ --output-folder “C:scriptsScreamingFrog" ^ --export-tabs "Images:Over X KB"
  61. 61. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Introducing PowerShell…
  62. 62. @Fearless_Shultz #brightonSEO Introducing PowerShell Scripting Panel Command Line cmdlet Library PowerShell is a task-based command-line shell and scripting language built on .NET. PowerShell helps system administrators and power-users rapidly automate tasks that manage operating systems (Linux, macOS, and Windows) and processes
  63. 63. @Fearless_Shultz #brightonSEO Introducing PowerShell - cmdlets A cmdlet (pronounced "command-let") is a lightweight Windows PowerShell script that performs a single function. Import-CSV Invoke-RestMethodExport-CSV
  64. 64. @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv The Import-CSV cmdlet turns a CSV file into an object which has properties which can easily be referenced with filters including: -lt (Less than) -gt (Greater than -eq (Equal to) -like (Like)
  65. 65. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'| select Address, Status,'Title 1 Length'| Where Address -like *services*| where Status -eq 'OK'| where 'Title 1 Length' -gt 55
  66. 66. @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv
  67. 67. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Export-Csv $myData = Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'| select Address, Status,'Title 1 Length'| Where Address -like *services*| where Status -eq 'OK'| where 'Title 1 Length' -gt 55 $myData | Export-CSv 'C:BrightonSEOscriptscsv examplesinternal_html-new.csv'
  68. 68. @Fearless_Shultz #brightonSEO Introducing PowerShell – Export-Csv
  69. 69. @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv
  70. 70. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO You Could . . .
  71. 71. @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv Open redirect report from SF in Excel Create filter to find any rows with a redirect count of more than one. Create a new column to contain the redirect Use a concatenation formula to prepend the rule to the first and last URLs Remove all the other columns which aren’t required and save.
  72. 72. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Or . . .
  73. 73. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Import-Csv $myObjectsArray = Import-Csv "C:powershell-scriptsredirect_and_canonical_chains.csv" | Where { $_."Number of Redirects" -gt 1} $newArray = @() $myObjectsArray | ForEach-Object{ $redirectType = "Redirect 301 " $addressToRedirectFrom = $_.Address -replace '^(?:https?://)?(?:[^@/n]+@)?(?:www.)?([^:/n]+)', '' $addressToRedirectTo = $_.'Final Address' $rewriteRule = $redirectType + $addressToRedirectFrom + ' ' + $addressToRedirectTo $newArray += $rewriteRule } $newArray | Out-File "C:powershell-scriptstesting-export.csv"
  74. 74. @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod Invoke-RestMethod consumes REST APIs and parses the result and where JSON is returned the data is converted into a PowerShell object that can be easily queried.
  75. 75. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod $username = “user-name-here" $password = “api-key-here" $uri = "https://gtmetrix.com/api/0.1/test" $testURL = "https://edit.co.uk" $base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password))) $testing = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} ` -Method Post -Uri $uri -Body @{url="https://edit.co.uk"} ` $testID = $testing.test_id $testURLWithID = $testing.poll_state_url $testing
  76. 76. @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod
  77. 77. @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod
  78. 78. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod $fullData = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} ` -Method GET -Uri $uri+$testURLWithID $testScore = $fullData.resources.pagespeed.Length $testDoMTime = $fullData.results.dom_content_loaded_time $testFullyLoaded = $fullData.results.fully_loaded_time 'The test score is:'+ $testScore 'The DOM content loaded time is: ' +$testDoMTime 'The fully loaded time is: ' +$testFullyLoaded
  79. 79. @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod
  80. 80. @Fearless_Shultz #brightonSEO Introducing PowerShell – Invoke-RestMethod
  81. 81. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Linking things together
  82. 82. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Generate Redirect Report and Create Redirects.
  83. 83. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? cd "C:Program Files (x86)Screaming Frog SEO Spider" ScreamingFrogSEOSpiderCli.exe ^ --crawl https://edit.co.uk/ ^ --headless ^ --save-crawl ^ --config "C:redirect.seospiderconfig" ^ --output-folder "C:redirects" ^ --save-report "Redirect & Canonical Chains" chaining-redirects.bat
  84. 84. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? cd C:redirects C:redirectschaining-redirects.bat Get-Content C:redirectsredirect_and_canonical_chains.csv | Select-Object -Skip 1 | Out-File C:redirectsredirects.csv $myObjectsArray = Import-Csv "C:redirectsredirects.csv" | Where { $_."Number of Redirects" -gt 1} $newArray = @() $myObjectsArray | ForEach-Object{ $rType = "Redirect 301 " $rFrom = $_.Address -replace '^(?:https?://)?(?:[^@/n]+@)?(?:www.)?([^:/n]+)', ‘’ $rTo = $_.'Final Address' $rewriteRule = $rType + $rFrom + ' ' + $rTo $newArray += $rewriteRule} $newArray | Out-File "C:redirectsredirect-rules.csv"
  85. 85. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Generate Lighthouse Reports.
  86. 86. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? cd "C:Program Files (x86)Screaming Frog SEO Spider" ScreamingFrogSEOSpiderCli.exe ^ --crawl https://edit.co.uk ^ --headless ^ --save-crawl ^ --config “C:image-crawl.seospiderconfig" ^ --output-folder “C:scriptsScreamingFrog" ^ --export-tabs “Internal:ALL" Chaining-lighthouse.bat
  87. 87. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? @echo off cd C:scriptsChainingLighthouselighthouse-reports for /f "delims=" %%a in (C:scriptsChainingLighthousetest-urls.txt) DO ( ECHO Line is: %%a lighthouse --quiet --chrome-flags="--headless" %%a ) Test.bat
  88. 88. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO What is the Command Line? cd C:lighthouse Get-Content C:scriptsScreamingFroginternal_html.csv | Select-Object -Skip 1 | Out-File C:lighthouseurls-to-test.csv $myObjectsArray = Import-Csv "C:lighthouseurls-to-test.csv" C:Lighthousechaining-lighthouse.bat $newArray = @() $myObjectsArray | ForEach-Object{ $newArray += $_.Address } $newArray | Out-File "C:lighthousetest-urls.txt" -Encoding ascii C:Lighthousetest.bat
  89. 89. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO This is just the tip of the iceberg
  90. 90. @Fearless_Shultz #brightonSEO Confidential @Fearless_Shultz #brightonSEO Useful Resources For Learning and Working With CLI Scripts.
  91. 91. @Fearless_Shultz #brightonSEO CLI Scripting Resources Title Link A beginners guide to batch scripting in Windows http://steve-jansen.github.io/guides/windows-batch-scripting/ Distilled Beginners guide to command line for SEO https://www.distilled.net/blog/seo/the-beginners-guide-to-using-the-command-line-for-seo/ SEO Strategies to impress your boss https://elite-strategies.com/impress-seo-boss-linux-command-line-tricks/ Beginners guide to Terminal on Mac https://www.makeuseof.com/tag/beginners-guide-mac-terminal/ Tom Pool CLI Hacks presentation http://bit.ly/tom-pool-command-line-hacks-presentation Tom Pool CLI Cheat Sheet http://bit.ly/command-line-cheat-sheet Tom Pool Podcast http://bit.ly/tom-pool-presentation-podcast 64 Resources for learning to code online https://skillcrush.com/2016/03/15/64-online-resources-to-learn-to-code-for-free/ The 100 days of code project https://www.100daysofcode.com/ PowerShell download packages https://github.com/PowerShell/PowerShell Beginners guide to PowerShell https://blog.netwrix.com/2018/02/21/windows-powershell-scripting-tutorial-for-beginners/ Guide to using Arrays in Poweshell https://4sysops.com/archives/arrays-in-powershell-create-change-read-sort-delete/ PowerShell Foreach-Object https://www.youtube.com/watch?v=BoutUXsBGcU PowerShell out-File https://stackoverflow.com/questions/19450616/export-csv-exports-length-but-not-name
  92. 92. @Fearless_Shultz #brightonSEO CLI Scripting Resources Title Link CSV kit for PowerShell https://searchenginewatch.com/2019/02/06/technical-seo-2019-statelessness/ Loop through files in directory in powershell https://stackoverflow.com/questions/18847145/loop-through-files-in-a-directory-using- powershell Understanding invoke-rest method https://4sysops.com/archives/understanding-the-invoke-restmethod-powershell-cmdlet Working With Csvs in PowerShell https://www.petri.com/powershell-import-csv-cmdlet-parse-comma-delimited-csv-text-file Video example of connecting to REST Method https://www.youtube.com/watch?v=7mEmQgGowMY PowerShell add-ons https://4sysops.com/archives/my-favorite-windows-powershell-ise-add-ons/ Powershell substrings https://4sysops.com/archives/strings-in-powershell-replace-compare-concatenate-split- substring/ Screaming Frog Command Scheduler https://www.greenlanemarketing.com/blog/schedule-your-screaming-frog-crawls-with- punctual-frog/ Screaming Frog Command Line Documentation https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#commandlineoptions Running Screaming Frog CLI On a Server https://www.databulle.com/blog/seo/screaming-frog-headless.html GTMetrix API https://gtmetrix.com/api/ Cygwin to run Linux and Mac commands on Windows https://www.cygwin.com/ Huge searchable DB of REST Apis https://www.programmableweb.com/apis/directory Regex to strip out domain from URL (For Redirects) https://stackoverflow.com/questions/4791700/regex-needed-to-strip-out-domain-name Content and sentiment analysis https://www.pveller.com/content-work-automation-with-text-analytics/
  93. 93. @Fearless_Shultz #brightonSEO Closing Thoughts.
  94. 94. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Repetitive tasks are often PERFECT candidates for automation
  95. 95. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO If a tool doesn’t already exist to perform a task don’t assume you have to do things the hard way. BUILD IT.
  96. 96. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Writing scripts is not as HARD as you might imagine it to be.
  97. 97. @Fearless_Shultz #brightonSEO @Fearless_Shultz #brightonSEO Remember the words of Uncle Ben and BE CAREFULL!
  98. 98. @Fearless_Shultz #brightonSEO Thank You.

×