Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

4,600 views

Published on

Slide deck from Brighton SEO April 2018 - 1st Session of the day (Technical SEO).

Published in: Marketing

Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

  1. 1. Tom Pool BlueArray Command Line Hacks For SEO @cptntommy https://www.slideshare.net/TomPool
  2. 2. Who Am I?
  3. 3. @cptntommy Technical SEO Manager @ BlueArray #BrightonSEO @cptntommy
  4. 4. Look after technical output for all clients @ BlueArray @cptntommy@cptntommy
  5. 5. This means I get to work on loads of different clients, doing loads of different tasks @cptntommy@cptntommy
  6. 6. Checking Response Codes @cptntommy@cptntommy
  7. 7. Tech Audits @cptntommy@cptntommy
  8. 8. Analysis Of @cptntommy@cptntommy
  9. 9. Keywords @cptntommy@cptntommy
  10. 10. Keyword Gap @cptntommy@cptntommy
  11. 11. Server Log Files @cptntommy@cptntommy
  12. 12. Crawl Data @cptntommy@cptntommy
  13. 13. The Problem Is... @cptntommy
  14. 14. Command Line Hacks For SEO Insert picture of hours ticking away? Full screen Each of these tasks takes a lot of time
  15. 15. @cptntommy@cptntommy
  16. 16. @cptntommy@cptntommy
  17. 17. < <? @cptntommy@cptntommy
  18. 18. < < @cptntommy@cptntommy
  19. 19. Command Line Hacks for SEO @cptntommy@cptntommy Terminal
  20. 20. What Is Command Line? @cptntommy@cptntommy
  21. 21. It is a basic interface between you and the computer @cptntommy@cptntommy
  22. 22. What Is Command Line?
  23. 23. What Is Command Line?
  24. 24. The following references are related to Mac OS’s Terminal application. @cptntommy@cptntommy
  25. 25. @cptntommy@cptntommy
  26. 26. @cptntommy@cptntommy
  27. 27. @cptntommy@cptntommy
  28. 28. What Commands Do I Use? @cptntommy@cptntommy
  29. 29. CURL SORT CAT SPLIT SED AWK @cptntommy@cptntommy
  30. 30. Keyword Analysis Keyword Gap Analysis Checking Response Codes Log File Analysis Crawling & Analysis @cptntommy@cptntommy
  31. 31. curl
  32. 32. What is curl? @cptntommy@cptntommy
  33. 33. “curl is a tool to transfer data from or to a server” @cptntommy@cptntommy
  34. 34. “curl is a tool to transfer data from or to a server” @cptntommy@cptntommy *Examples shown are not the full usage, for full info check out the manual page (man curl)
  35. 35. Checking Response Codes @cptntommy@cptntommy
  36. 36. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.uk @cptntommy@cptntommy
  37. 37. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.ukcurl @cptntommy@cptntommy
  38. 38. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.ukhttps://www.bluearray.co.ukcurl @cptntommy@cptntommy
  39. 39. @cptntommy@cptntommy
  40. 40. @cptntommy@cptntommy
  41. 41. What If I Just Want To See The HTTP Header? @cptntommy
  42. 42. @cptntommy@cptntommy
  43. 43. curl @cptntommy@cptntommy
  44. 44. curl -I @cptntommy@cptntommy Modifier - Just The HTTP Header!
  45. 45. curl -I https://bluearray.co.uk @cptntommy@cptntommy
  46. 46. @cptntommy@cptntommy
  47. 47. @cptntommy@cptntommy
  48. 48. @cptntommy@cptntommy
  49. 49. @cptntommy@cptntommy
  50. 50. curl -I -L @cptntommy@cptntommy
  51. 51. curl -I -L https://bluearray.co.uk @cptntommy@cptntommy
  52. 52. @cptntommy@cptntommy
  53. 53. curl-O @cptntommy@cptntommy
  54. 54. curl-O https://download.screami ngfrog.co.uk/products/seo -spider/ScreamingFrogSEO Spider-8.3.dmg @cptntommy@cptntommy
  55. 55. @cptntommy@cptntommy
  56. 56. Here we have used CURL TO: Download Files, Check HTML, Check HTTP Header & Follow Redirects @cptntommy@cptntommy
  57. 57. @cptntommy@cptntommy
  58. 58. sort
  59. 59. Sort ‘sorts’ @cptntommy@cptntommy
  60. 60. @cptntommy@cptntommy
  61. 61. Sort - A-Z
  62. 62. @cptntommy@cptntommy
  63. 63. @cptntommy@cptntommy
  64. 64. Navigate to Folder and use ‘ls - “list”’ command to make sure the data is there @cptntommy@cptntommy
  65. 65. Then run the sort command @cptntommy@cptntommy
  66. 66. sort @cptntommy@cptntommy
  67. 67. @cptntommy@cptntommy filename.csvsort
  68. 68. “sort keyworddata.csv” @cptntommy@cptntommy
  69. 69. @cptntommy@cptntommy
  70. 70. sort filename > newfilename @cptntommy@cptntommy
  71. 71. @cptntommy@cptntommy
  72. 72. @cptntommy@cptntommy Did It Work?
  73. 73. head/tail
  74. 74. WTF is Head/Tail? @cptntommy@cptntommy
  75. 75. ‘Head’ views the first 10 Rows ‘Tail’ views the last 10 Rows @cptntommy@cptntommy
  76. 76. head filename.csv @cptntommy@cptntommy
  77. 77. head filename @cptntommy@cptntommy
  78. 78. tail filename @cptntommy
  79. 79. Sort - Z-A
  80. 80. sort filename.csv > Z-A_SortedData.csv @cptntommy@cptntommy
  81. 81. sort filename.csv > Z-A_SortedData.csv @cptntommy@cptntommy -r
  82. 82. @cptntommy@cptntommy
  83. 83. Sort - By Volume
  84. 84. @cptntommy@cptntommy
  85. 85. @cptntommy@cptntommy We want to sort by the second column
  86. 86. @cptntommy@cptntommy
  87. 87. sort @cptntommy@cptntommy
  88. 88. sort -k2 @cptntommy@cptntommy
  89. 89. sort -k2 -t, @cptntommy@cptntommy
  90. 90. sort -k2 -t, -n @cptntommy@cptntommy
  91. 91. sort -k2 -t, -n -r @cptntommy@cptntommy
  92. 92. sort -k2 -t, -n -r filename.csv @cptntommy@cptntommy
  93. 93. sort -k2 -t, -n -r filename.csv > volumesorteddata.csv @cptntommy@cptntommy
  94. 94. head volumesorteddata.csv > top10KW.csv @cptntommy@cptntommy
  95. 95. head -n100 volumesorteddata.csv > top100KW.csv @cptntommy@cptntommy
  96. 96. Here sort has been used to: Sort by Number, Sort by A-Z, Sort by Z-A @cptntommy@cptntommy
  97. 97. We have also used head/tail to; Extract the top 10, Top 100 Save these to a file @cptntommy@cptntommy
  98. 98. Title
  99. 99. @cptntommy@cptntommy cat - short for “concatenate”
  100. 100. @cptntommy@cptntommy Used to display, combine & create files
  101. 101. @cptntommy@cptntommy
  102. 102. @cptntommy@cptntommy
  103. 103. @cptntommy@cptntommy >
  104. 104. @cptntommy@cptntommy cat
  105. 105. @cptntommy@cptntommy cat *.csv
  106. 106. @cptntommy@cptntommy cat *.csv > alldata.csv
  107. 107. @cptntommy@cptntommy > User/Desktop/DataToCombine
  108. 108. sed
  109. 109. @cptntommy@cptntommy Short for "stream editor", allows you to filter and transform text.
  110. 110. @cptntommy@cptntommy Adding text to rows
  111. 111. @cptntommy
  112. 112. @cptntommy@cptntommy
  113. 113. @cptntommy@cptntommy Use sed to add protocol & domain at start of each line
  114. 114. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv
  115. 115. @cptntommy@cptntommy REGEX!!
  116. 116. @cptntommy@cptntommy sed -e 's/^/
  117. 117. @cptntommy@cptntommy sed -e 's/^/http://www.domain.co m/'
  118. 118. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/'
  119. 119. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv
  120. 120. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv > full.csv
  121. 121. @cptntommy@cptntommy
  122. 122. @cptntommy@cptntommy use sed to find & replace
  123. 123. @cptntommy@cptntommy Replace http with https
  124. 124. @cptntommy@cptntommy sed -e 's/http file.csv > newfile.csv /https/'
  125. 125. @cptntommy
  126. 126. @cptntommy@cptntommy Here we have: Added in protocol & domain Replaced http with https
  127. 127. awk
  128. 128. @cptntommy@cptntommy Programming language, used to process text
  129. 129. @cptntommy@cptntommy Following example is one of the easier applications of awk
  130. 130. @cptntommy@cptntommy
  131. 131. @cptntommy@cptntommy 1 7
  132. 132. @cptntommy@cptntommy
  133. 133. @cptntommy@cptntommy awk -F “,”
  134. 134. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’
  135. 135. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’ kw.csv > betterkw.csv
  136. 136. @cptntommy
  137. 137. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’
  138. 138. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2 “,” $4}’
  139. 139. @cptntommy@cptntommy How to combine all this?
  140. 140. Log File Analysis @cptntommy
  141. 141. @cptntommy About 1% of given Log Files
  142. 142. @cptntommy 1.cat Combine files together
  143. 143. @cptntommy Cat *.log > combinedlogs.log
  144. 144. @cptntommy 2. Use sed to add in full domain name to log file
  145. 145. @cptntommy
  146. 146. @cptntommy
  147. 147. @cptntommy sed -e 's // file.log > newfile.log /domain.com/'
  148. 148. @cptntommy sed -e 's Search
  149. 149. @cptntommy sed -e 's // Forward Slash
  150. 150. @cptntommy sed -e 's // /domain.com/' Text to replace with
  151. 151. @cptntommy sed -e 's // file.log > newfile.log /domain.com/' Then the files
  152. 152. @cptntommy 3.Extract 404 errors (awk)
  153. 153. @cptntommy awk '/404/ {print $0}' combinedlogs.log > log404.log
  154. 154. @cptntommy 4.Extract any specific status code (awk)
  155. 155. @cptntommy awk '/301/ {print $0}' file.log > newfile.log
  156. 156. @cptntommy awk '/200/ {print $0}' file.log > newfile.log
  157. 157. @cptntommy awk '/503/ {print $0}' file.log > newfile.log
  158. 158. @cptntommy awk '/418/ {print $0}' file.log > newfile.log
  159. 159. @cptntommy 5. Extract all Googlebot/ Mobile Bot hits (awk)
  160. 160. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  161. 161. @cptntommy What if you wanted to know how many hits each bot had, without opening the file?
  162. 162. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  163. 163. @cptntommy wc
  164. 164. @cptntommy wc -l gbot.log
  165. 165. @cptntommy
  166. 166. @cptntommy 6. Extract all hits of a specific URL, or all img requests, all css requests, all with modification of 1 command
  167. 167. @cptntommy Replace ‘Googlebot’ with ‘BingBot’, ‘anybot’...
  168. 168. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  169. 169. @cptntommy awk '/Bingbot/ {print $0}' all.log > gbot.log
  170. 170. @cptntommy awk '/anybot/ {print $0}' all.log > gbot.log
  171. 171. @cptntommy awk '/.css/ {print $0}' all.log > gbot.log
  172. 172. @cptntommy awk '/.jpeg/ {print $0}' all.log > gbot.log
  173. 173. @cptntommy awk '/whatever-you-want/ {print $0}' all.log > file.log
  174. 174. @cptntommy This Barely Scratches the Surface
  175. 175. @cptntommy So, To Recap
  176. 176. You can now hack: @cptntommy
  177. 177. Server Logs @cptntommy
  178. 178. Keyword data @cptntommy
  179. 179. .csv files @cptntommy
  180. 180. Extract columns of data @cptntommy
  181. 181. Sort data @cptntommy
  182. 182. General SEO Shit @cptntommy
  183. 183. And @cptntommy
  184. 184. #BrightonSEO @cptntommy
  185. 185. Thanks! @cptntommy

×