Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Brighton SEO Sept 2019 PowerShell

6,130 views

Published on

My Talk on PowerShell from Brighton SEO Sept 2019

Published in: Internet

Brighton SEO Sept 2019 PowerShell

  1. 1. @Fearless_Shultz #brightonSEO Utilizing the power of PowerShell for SEO Mike Osolinski // Technical SEO Consultant SLIDESHARE.NET/MikeOsolinski @Fearless_Shultz
  2. 2. @Fearless_Shultz #brightonSEO https://www.slideshare.net/MikeOsolinski/command-line-automation-for-repetitive-tasks
  3. 3. @Fearless_Shultz #brightonSEO What I’m Going to Talk About Today A Brief Introduction to what PowerShell is, it's components and interface Some Examples of Using PowerShell to Automate SEO Tasks Extending PowerShell Beyond It’s Native Capabilities
  4. 4. @Fearless_Shultz #brightonSEO Why Automate SEO Processes?
  5. 5. @Fearless_Shultz #brightonSEO Spend Less Time Gathering and Formatting Data and More Time on Analysis
  6. 6. @Fearless_Shultz #brightonSEO Make your business more profitable by being able to do the same work in a shorter period of time
  7. 7. @Fearless_Shultz #brightonSEO Allow teams to easily handle greater workloads without having to recruit new staff
  8. 8. @Fearless_Shultz #brightonSEO Have an Easier Life…
  9. 9. @Fearless_Shultz #brightonSEO So Why Don’t More People Automate Processes?
  10. 10. @Fearless_Shultz #brightonSEO
  11. 11. @Fearless_Shultz #brightonSEO Lack of Awareness of the potential time savings and that many repetitive tasks can be easily automated
  12. 12. @Fearless_Shultz #brightonSEO Fear either that scripting is too hard for you to learn and you won’t be able to do it or that you might break something *Spoiler – It’s not and You Will. Everyone does
  13. 13. @Fearless_Shultz #brightonSEO Not having / wanting to spend time learning to write scripts
  14. 14. @Fearless_Shultz #brightonSEO Why PowerShell?
  15. 15. @Fearless_Shultz #brightonSEO
  16. 16. @Fearless_Shultz #brightonSEO Already Installed on Most Windows Computers so a lack of complicated set up processes
  17. 17. @Fearless_Shultz #brightonSEO Open Source Meaning it is Cross Platform and has a Large Development Community
  18. 18. @Fearless_Shultz #brightonSEO Makes Working with Data Easy and Reduces the Need to Write Complex Formulas and Expressions
  19. 19. @Fearless_Shultz #brightonSEO Highly Extendable Via Gallery of Custom Modules
  20. 20. @Fearless_Shultz #brightonSEO What is PowerShell?
  21. 21. @Fearless_Shultz #brightonSEO PowerShell Gives You Superpowers!!!
  22. 22. @Fearless_Shultz #brightonSEO Actually, PowerShell is a task-based command-line shell and scripting language built on .NET which helps system administrators and power- users rapidly automate tasks that manage operating systems and processes But that doesn’t sound as cool…
  23. 23. @Fearless_Shultz #brightonSEO So Why Should SEO’s Care?
  24. 24. @Fearless_Shultz #brightonSEO • Scrape Websites • Automate Website Interactions • Run Screaming Frog Reports • Consume Rest APIS • Extract and Manipulate Data • And Much More……..
  25. 25. @Fearless_Shultz #brightonSEO Getting Started
  26. 26. @Fearless_Shultz #brightonSEO https://www.peppercrew.nl/index.php/2014/06/powershell- coding-on-a-mac/ https://wilsonmar.github.io/powershell-on-mac/ https://docs.microsoft.com/en- us/powershell/scripting/install/installing-powershell-core-on- linux?view=powershell-6 https://docs.microsoft.com/en- us/powershell/scripting/install/installing-powershell-core-on- macos?view=powershell-6 https://www.starwindsoftware.com/blog/using-powershell- on-linux
  27. 27. @Fearless_Shultz #brightonSEO Scripting Panel Command Line Cmdlets
  28. 28. @Fearless_Shultz #brightonSEO Cmdlets
  29. 29. @Fearless_Shultz #brightonSEO Command Line A cmdlet (pronounced "command-let") is a lightweight Windows PowerShell script that performs a single function. Invoke-WebRequest Import / Export CSVInvokeRestMethod
  30. 30. @Fearless_Shultz #brightonSEO Command Line https://www.powershellgallery.com
  31. 31. @Fearless_Shultz #brightonSEO Command Line https://www.powershellgallery.com
  32. 32. @Fearless_Shultz #brightonSEO Invoke-WebRequest
  33. 33. @Fearless_Shultz #brightonSEO The Invoke-WebRequest cmdlet sends HTTP and HTTPS requests to a web page or web service. It parses the response and returns collections of links, images, and other significant HTML elements. https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-6
  34. 34. @Fearless_Shultz #brightonSEO $w = Invoke-WebRequest -Uri 'https://moz.com/learn/seo/what-is-seo' $w | Get-Member $w = $w.RawContent $w
  35. 35. @Fearless_Shultz #brightonSEO
  36. 36. @Fearless_Shultz #brightonSEO
  37. 37. @Fearless_Shultz #brightonSEO
  38. 38. @Fearless_Shultz #brightonSEO $w = Invoke-WebRequest -Uri 'https://moz.com/learn/seo/what- is-seo' $w | Get-Member $w = $w.Links $w
  39. 39. @Fearless_Shultz #brightonSEO
  40. 40. @Fearless_Shultz #brightonSEO $w = Invoke-WebRequest -Uri 'https://moz.com/learn/seo/what- is-seo' $w | Get-Member $w = $w.Links | Select innerHTML, href | WHERE innerHTML -like *SEO* $w | Export-Csv 'C:scriptslinks.csv' -NoTypeInformation
  41. 41. @Fearless_Shultz #brightonSEO innerHTML href Free SEO Tools https://moz.com/tools https://moz.com/learn/seo https://moz.com/training https://moz.com/beginners-guide-to-seo SEO Learning Center https://moz.com/learn Free SEO Tools https://moz.com/free-seo-tools What is SEO? https://moz.com/learn/seo/what-is-seo On-Site SEO /learn/seo/on-site Local SEO /learn/seo/local Mobile SEO /learn/seo/mobile International SEO /learn/seo/international Beginner's Guide to SEO /beginners-guide-to-seo
  42. 42. @Fearless_Shultz #brightonSEO $w = Invoke-WebRequest -Uri 'https://moz.com/learn/seo/what- is-seo' $w | Get-Member $w = $w.Links | Select innerHTML $w | Out-File 'C:scriptslink-text.txt' Get-Content C:scriptslink-text.txt | New-WordCloud -Path C:scriptswordcloud.svg -ImageSize 1080p
  43. 43. @Fearless_Shultz #brightonSEO https://github.com/vexx32/PSWordCloud
  44. 44. @Fearless_Shultz #brightonSEO $currentRequest = Invoke-WebRequest -Uri https://moz.com/learn/seo/what-is-seo $currentRequest.AllElements
  45. 45. @Fearless_Shultz #brightonSEO
  46. 46. @Fearless_Shultz #brightonSEO $w = Invoke-WebRequest -Uri 'https://moz.com/learn/seo/what- is-seo' $w.AllElements | where tagname -EQ "p" | select innerText | where innerText -Like *seo*
  47. 47. @Fearless_Shultz #brightonSEO
  48. 48. @Fearless_Shultz #brightonSEO $w = Invoke-WebRequest -Uri 'https://moz.com/learn/seo/what- is-seo' $w.AllElements | where tagname -EQ "p" | select innerText | where innerText -Like *seo*| Out-File 'C:scriptsscriptfilesparagraphs.txt' Get-Content C:scriptsscriptfilesparagraphs.txt | New-WordCloud -Path C:scriptsscriptfilesparagraph-cloud.svg -ImageSize 1080p
  49. 49. @Fearless_Shultz #brightonSEO
  50. 50. @Fearless_Shultz #brightonSEO $path = 'C:scriptsscriptfiles' Import-Csv -Path 'C:scriptsscriptfilesseo-competitors.csv'| ForEach-Object{ foreach ($property in $_.PSObject.Properties) { $currentRequest = Invoke-WebRequest -Uri $property.Value $pageTitle = $currentRequest.ParsedHtml.title $currentSavePath = $path + $pageTitle + '.txt' $currentSavePath = $currentSavePath -replace("?","") $currentSavePath = $currentSavePath -replace("/","") $currentSVGPath = $currentSavePath -replace("txt","svg") $paragraphs = $currentRequest.AllElements | where tagname -EQ "p"| select innerText| Out-File $currentSavePath Get-Content $currentSavePath | New-WordCloud -Path $currentSVGPath -ImageSize 1080p } }
  51. 51. @Fearless_Shultz #brightonSEO
  52. 52. @Fearless_Shultz #brightonSEO Invoke-RestMethod
  53. 53. @Fearless_Shultz #brightonSEO The Invoke-WebRestMethod cmdlet sends HTTP and HTTPS requests to a web page or web service. It parses the response and returns collections of links, images, and other significant HTML elements. https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-6
  54. 54. @Fearless_Shultz #brightonSEO $pageSpeedData = Invoke-RestMethod -Uri 'https://www.googleapis.com/pagespeedonl ine/v5/runPagespeed?url=https://www.brig htonseo.com/'
  55. 55. @Fearless_Shultz #brightonSEO
  56. 56. @Fearless_Shultz #brightonSEO $pageSpeedData = Invoke-RestMethod -Uri 'https://www.googleapis.com/pagespeedonl ine/v5/runPagespeed?url=https://www.brig htonseo.com/' $pageSpeedData.lighthouseResult.audits
  57. 57. @Fearless_Shultz #brightonSEO
  58. 58. @Fearless_Shultz #brightonSEO
  59. 59. @Fearless_Shultz #brightonSEO
  60. 60. @Fearless_Shultz #brightonSEO $pageSpeedData = Invoke-RestMethod -Uri 'https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url =https://www.dailymail.co.uk' $offScreenImageCount = $pageSpeedData.lighthouseResult.audits.'offscreen- images'.details.items.count for($i = 0; $i -le $offScreenImageCount; $i++){ $element = $pageSpeedData.lighthouseresult.audits.'offscreen- images'.details.items.getvalue($i) $element | export-csv 'C:scriptsoffscreen-images.csv' - notypeinformation -append }
  61. 61. @Fearless_Shultz #brightonSEO
  62. 62. @Fearless_Shultz #brightonSEO totalBytes wastedBytes wastedPercent url requestStartTime 43337 43337 100 https://i.dailymail.co.uk/1s/2019/09/04/14/18057384-0-image-m-2_1567604298582.jpg 363649.4834 33206 33206 100 https://i.dailymail.co.uk/1s/2019/09/05/01/18085400-0-image-a-1_1567644632414.jpg 363649.2957 18845 18845 100 https://i.dailymail.co.uk/1s/2019/09/02/16/17978778-0-image-a-36_1567436926465.jpg 363650.0165 18714 18714 100 https://i.dailymail.co.uk/1s/2019/09/03/12/18010078-0-image-a-2_1567509730138.jpg 363650.6832 17963 17963 100 https://i.dailymail.co.uk/1s/2019/09/02/18/17982924-0-image-a-48_1567445797522.jpg 363650.6857 17524 17524 100 https://i.dailymail.co.uk/1s/2019/09/04/09/18050766-0-image-a-12_1567585569391.jpg 363650.6837 14205 14205 100 https://i.dailymail.co.uk/1s/2019/09/04/18/18072588-0-image-a-35_1567619706814.jpg 363650.6849 14101 14101 100 https://i.dailymail.co.uk/1s/2019/09/05/00/18082194-0-image-a-40_1567638380685.jpg 363650.6838 13642 13642 100 https://i.dailymail.co.uk/1s/2019/09/04/14/18062208-0-image-a-30_1567605415945.jpg 363650.6846 13164 13164 100 https://i.dailymail.co.uk/1s/2019/09/02/22/17987956-0-image-a-9_1567458265073.jpg 363650.0176 13063 13063 100 https://i.dailymail.co.uk/i/pix/2018/10/03/tips_v3_opt1_308.png 363649.4832 12987 12987 100 https://i.dailymail.co.uk/1s/2019/09/03/22/18033866-0-image-a-10_1567547692640.jpg 363650.6835
  63. 63. @Fearless_Shultz #brightonSEO Import-Csv 'C:test-urls.csv' | Foreach-Object { foreach ($property in $_.PSObject.Properties) { $currentURL = 'https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=' + $property.Value $testResults = Invoke-restMethod -Uri $currentURL$testResults.lighthouseResult.audits.'render-blocking- resources'.details.items $arrayCount = $testResults.lighthouseResult.audits.'render-blocking- resources'.details.items.Count for($i = 0; $i -le $arrayCount; $i++){$element = $testResults.lighthouseResult.audits.'render-blocking- resources'.details.items.GetValue($i) $element | Export-Csv 'C:scriptsrender-results.csv' -NoTypeInformation - Append } }
  64. 64. @Fearless_Shultz #brightonSEO https://www.programmableweb.com/apis/directory
  65. 65. @Fearless_Shultz #brightonSEO + +
  66. 66. @Fearless_Shultz #brightonSEO # run the screaming frog crawl cd "C:Program Files (x86)Screaming Frog SEO Spider" $env:Path = $env:Path + ";C:Program Files (x86)Screaming Frog SEO Spider" $startingURL = "https://mikeosolinski.co.uk" ScreamingFrogSEOSpiderCli.exe --crawl $startingURL --headless --save-crawl --output-folder “C:scriptssf" --export-tabs "Internal:HTML,Response Codes:Client Error (4xx)" https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#commandlineoptions
  67. 67. @Fearless_Shultz #brightonSEO #create an excel document to contain the data $xl = New-Object -ComObject Excel.Application $xl.Visible = $true #add a workbook to the excel object $workbook1 = $xl.WorkBooks.Add() #add the sheet to contain long page titles $lngPageTitles = $workbook1.Worksheets.Item(1) $lngPageTitles.Name = 'Long Page Titles' #create the column headers for long page titles $lngPageTitles.Cells.Item(1,1) = 'Address' $lngPageTitles.Cells.Item(1,2) = 'Status' $lngPageTitles.Cells.Item(1,3) = 'Title Length'
  68. 68. @Fearless_Shultz #brightonSEO #import the title length data from the screaming frog crawl $titledata = Import-Csv -Path ‘C:scriptssfinternal_html.csv'| select Address, Status,'Title 1 Length'| where Status -eq 'OK'| where 'Title 1 Length' -gt 55 #iterate through the records and add data to the sheet $i = 2 foreach($record in $titledata) { $titledata.Address $titledata.'Title 1 Length' $lngPageTitles.cells.item($i,1) = $record.Address $lngPageTitles.cells.item($i,2) = $record.Status $lngPageTitles.cells.item($i,3) = $record.'Title 1 Length' $i++ }
  69. 69. @Fearless_Shultz #brightonSEO https://github.com/mikeosolinski/powershell/blob/master/screaming-frog-excel
  70. 70. @Fearless_Shultz #brightonSEO https://github.com/mikeosolinski/powershell/blob/master/site-speed-data
  71. 71. @Fearless_Shultz #brightonSEO # CHANGE THE URL BELOW TO THE ONE THAT YOU WANT TO TEST $urltotest = 'https://mikeosolinski.co.uk' #update the path to save files to from C:SiteSpeedDataTOOL-OUTPUT to whatever you want. avoid spaces in path/filenames #any questions on how to run this ping me on twitter at https://twitter.com/Fearless_Shultz https://github.com/mikeosolinski/powershell/blob/master/site-speed-data
  72. 72. @Fearless_Shultz #brightonSEO Extending PowerShell
  73. 73. @Fearless_Shultz #brightonSEO Install-Module -Name GoogleMap -Scope CurrentUser https://geekeefy.wordpress.com/2016/05/17/powershell-module-for-google-map/
  74. 74. @Fearless_Shultz #brightonSEO $env:GoogleGeocode_API_Key = “INSERT API KEY" $env:GooglePlaces_API_Key = “INSERT API KEY" ("Brighton, UK" | Get-GeoCoding).coordinates | Get-NearbyPlace -Radius 1000 -TypeOfPlace hotel
  75. 75. @Fearless_Shultz #brightonSEO
  76. 76. @Fearless_Shultz #brightonSEO Install-Module -Name PSTwitterAPI -Scope CurrentUser https://github.com/mkellerman/PSTwitterAPI
  77. 77. @Fearless_Shultz #brightonSEO Import-Module PSTwitterAPI #set keys $apiKey = INSERT KEY HERE' $apiSecret = INSERT API SECRET HERE' $AccessToken = INSERT ACCESS TOKEN HERE' $tokenSecret = INSERT TOKEN SECRET HERE' # Provide Authentication for the Twitter API # https://twittercommunity.com/t/how-to-get-my-api-key/7033 Set-TwitterOAuthSettings -ApiKey $apiKey -ApiSecret $apiSecret -AccessToken $AccessToken -AccessTokenSecret $tokenSecret # Get user twitter profile $followers = Get-TwitterFollowers_List -count 200 $followers.users.name
  78. 78. @Fearless_Shultz #brightonSEO
  79. 79. @Fearless_Shultz #brightonSEO Import-Module PSTwitterAPI #set keys $apiKey = INSERT KEY HERE' $apiSecret = INSERT API SECRET HERE' $AccessToken = INSERT ACCESS TOKEN HERE' $tokenSecret = INSERT TOKEN SECRET HERE' # Provide Authentication for the Twitter API # https://twittercommunity.com/t/how-to-get-my-api-key/7033 Set-TwitterOAuthSettings -ApiKey $apiKey -ApiSecret $apiSecret -AccessToken $AccessToken -AccessTokenSecret $tokenSecret # Get user twitter profile $followers = Get-TwitterFollowers_List -count 200 $followers.users
  80. 80. @Fearless_Shultz #brightonSEO
  81. 81. @Fearless_Shultz #brightonSEO Import-Module PSTwitterAPI #set keys $apiKey = INSERT KEY HERE' $apiSecret = INSERT API SECRET HERE' $AccessToken = INSERT ACCESS TOKEN HERE' $tokenSecret = INSERT TOKEN SECRET HERE' # Provide Authentication for the Twitter API # https://twittercommunity.com/t/how-to-get-my-api-key/7033 Set-TwitterOAuthSettings -ApiKey $apiKey -ApiSecret $apiSecret -AccessToken $AccessToken -AccessTokenSecret $tokenSecret # Get user twitter profile $followers = Get-TwitterFollowers_List -count 200 $followers.users| select name, screen_name, description| where description -Like '*SEO*' Export-Csv 'C:scriptsseo-followers.csv'-NoTypeInformation
  82. 82. @Fearless_Shultz #brightonSEO
  83. 83. @Fearless_Shultz #brightonSEO
  84. 84. @Fearless_Shultz #brightonSEO Resources
  85. 85. @Fearless_Shultz #brightonSEO https://github.com/mikeosolinski/powershell/ My Github profile which contains commented versions of all of the scripts mentioned https://docs.microsoft.com/en-us/skypeforbusiness/set-up-your-computer-for- windows-powershell/download-and-install-windows-powershell-5-1 The download page to install PowerShell https://www.powershellgallery.com/ PowerShelll Cmdlet Gallery https://www.adamtheautomator.com/invoke-webrequest-powershell/ Guide to Invoke-WebRequest https://www.gngrninja.com/script-ninja/2016/7/8/powershell-getting-started- utilizing-the-web Another guide to invoke WebRequest https://stackoverflow.com/questions/11885246/how-do-i-loop-through-a-line- from-a-csv-file-in-powershell Loops in PowerShell https://www.youtube.com/watch?v=PXBMdIkH24I Associative Arrays in PowerShell https://vwiki.co.uk/MySQL_and_PowerShell Accessing MySQL With PowerShell https://mcpmag.com/articles/2018/08/08/replace-text-with-powershell.aspx Finding and Replacing Text https://techblog.dorogin.com/generate-word-documents-with-powershell- cda654b9cb0e Generating Word Documents with PowerShell https://www.powershellbros.com/powershell-tip-of-the-week-create-invoke- webrequest-from-chrome/ PowerShell and Google Chrome https://docs.google.com/spreadsheets/d/1psz6SvRqv7fjIFIiAySPNshObMINjjFeCvGqTjluGLs/edit?usp=sharing
  86. 86. @Fearless_Shultz #brightonSEO Closing Thoughts
  87. 87. @Fearless_Shultz #brightonSEO PowerShell is Not Just for Systems Administrators and Network Admins
  88. 88. @Fearless_Shultz #brightonSEO PowerShell is An Incredibly Versatile Tool and you are Only Limited By Your Own Imagination
  89. 89. @Fearless_Shultz #brightonSEO Even If You Don’t Want to Learn to Script Yourself you Should Understand the Opportunities
  90. 90. @Fearless_Shultz #brightonSEO But. . .
  91. 91. @Fearless_Shultz #brightonSEO You CAN do it
  92. 92. @Fearless_Shultz #brightonSEO https://mikeosolinski.co.uk http://twitter.com/Fearless_Shultz

×