Successfully reported this slideshow.
Your SlideShare is downloading. ×

Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 186 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to Command Line Hacks For SEO - Brighton April 2018 - Tom Pool (20)

Advertisement

Recently uploaded (20)

Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

  1. 1. Tom Pool BlueArray Command Line Hacks For SEO @cptntommy https://www.slideshare.net/TomPool
  2. 2. Who Am I?
  3. 3. @cptntommy Technical SEO Manager @ BlueArray #BrightonSEO @cptntommy
  4. 4. Look after technical output for all clients @ BlueArray @cptntommy@cptntommy
  5. 5. This means I get to work on loads of different clients, doing loads of different tasks @cptntommy@cptntommy
  6. 6. Checking Response Codes @cptntommy@cptntommy
  7. 7. Tech Audits @cptntommy@cptntommy
  8. 8. Analysis Of @cptntommy@cptntommy
  9. 9. Keywords @cptntommy@cptntommy
  10. 10. Keyword Gap @cptntommy@cptntommy
  11. 11. Server Log Files @cptntommy@cptntommy
  12. 12. Crawl Data @cptntommy@cptntommy
  13. 13. The Problem Is... @cptntommy
  14. 14. Command Line Hacks For SEO Insert picture of hours ticking away? Full screen Each of these tasks takes a lot of time
  15. 15. @cptntommy@cptntommy
  16. 16. @cptntommy@cptntommy
  17. 17. < <? @cptntommy@cptntommy
  18. 18. < < @cptntommy@cptntommy
  19. 19. Command Line Hacks for SEO @cptntommy@cptntommy Terminal
  20. 20. What Is Command Line? @cptntommy@cptntommy
  21. 21. It is a basic interface between you and the computer @cptntommy@cptntommy
  22. 22. What Is Command Line?
  23. 23. What Is Command Line?
  24. 24. The following references are related to Mac OS’s Terminal application. @cptntommy@cptntommy
  25. 25. @cptntommy@cptntommy
  26. 26. @cptntommy@cptntommy
  27. 27. @cptntommy@cptntommy
  28. 28. What Commands Do I Use? @cptntommy@cptntommy
  29. 29. CURL SORT CAT SPLIT SED AWK @cptntommy@cptntommy
  30. 30. Keyword Analysis Keyword Gap Analysis Checking Response Codes Log File Analysis Crawling & Analysis @cptntommy@cptntommy
  31. 31. curl
  32. 32. What is curl? @cptntommy@cptntommy
  33. 33. “curl is a tool to transfer data from or to a server” @cptntommy@cptntommy
  34. 34. “curl is a tool to transfer data from or to a server” @cptntommy@cptntommy *Examples shown are not the full usage, for full info check out the manual page (man curl)
  35. 35. Checking Response Codes @cptntommy@cptntommy
  36. 36. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.uk @cptntommy@cptntommy
  37. 37. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.ukcurl @cptntommy@cptntommy
  38. 38. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.ukhttps://www.bluearray.co.ukcurl @cptntommy@cptntommy
  39. 39. @cptntommy@cptntommy
  40. 40. @cptntommy@cptntommy
  41. 41. What If I Just Want To See The HTTP Header? @cptntommy
  42. 42. @cptntommy@cptntommy
  43. 43. curl @cptntommy@cptntommy
  44. 44. curl -I @cptntommy@cptntommy Modifier - Just The HTTP Header!
  45. 45. curl -I https://bluearray.co.uk @cptntommy@cptntommy
  46. 46. @cptntommy@cptntommy
  47. 47. @cptntommy@cptntommy
  48. 48. @cptntommy@cptntommy
  49. 49. @cptntommy@cptntommy
  50. 50. curl -I -L @cptntommy@cptntommy
  51. 51. curl -I -L https://bluearray.co.uk @cptntommy@cptntommy
  52. 52. @cptntommy@cptntommy
  53. 53. curl-O @cptntommy@cptntommy
  54. 54. curl-O https://download.screami ngfrog.co.uk/products/seo -spider/ScreamingFrogSEO Spider-8.3.dmg @cptntommy@cptntommy
  55. 55. @cptntommy@cptntommy
  56. 56. Here we have used CURL TO: Download Files, Check HTML, Check HTTP Header & Follow Redirects @cptntommy@cptntommy
  57. 57. @cptntommy@cptntommy
  58. 58. sort
  59. 59. Sort ‘sorts’ @cptntommy@cptntommy
  60. 60. @cptntommy@cptntommy
  61. 61. Sort - A-Z
  62. 62. @cptntommy@cptntommy
  63. 63. @cptntommy@cptntommy
  64. 64. Navigate to Folder and use ‘ls - “list”’ command to make sure the data is there @cptntommy@cptntommy
  65. 65. Then run the sort command @cptntommy@cptntommy
  66. 66. sort @cptntommy@cptntommy
  67. 67. @cptntommy@cptntommy filename.csvsort
  68. 68. “sort keyworddata.csv” @cptntommy@cptntommy
  69. 69. @cptntommy@cptntommy
  70. 70. sort filename > newfilename @cptntommy@cptntommy
  71. 71. @cptntommy@cptntommy
  72. 72. @cptntommy@cptntommy Did It Work?
  73. 73. head/tail
  74. 74. WTF is Head/Tail? @cptntommy@cptntommy
  75. 75. ‘Head’ views the first 10 Rows ‘Tail’ views the last 10 Rows @cptntommy@cptntommy
  76. 76. head filename.csv @cptntommy@cptntommy
  77. 77. head filename @cptntommy@cptntommy
  78. 78. tail filename @cptntommy
  79. 79. Sort - Z-A
  80. 80. sort filename.csv > Z-A_SortedData.csv @cptntommy@cptntommy
  81. 81. sort filename.csv > Z-A_SortedData.csv @cptntommy@cptntommy -r
  82. 82. @cptntommy@cptntommy
  83. 83. Sort - By Volume
  84. 84. @cptntommy@cptntommy
  85. 85. @cptntommy@cptntommy We want to sort by the second column
  86. 86. @cptntommy@cptntommy
  87. 87. sort @cptntommy@cptntommy
  88. 88. sort -k2 @cptntommy@cptntommy
  89. 89. sort -k2 -t, @cptntommy@cptntommy
  90. 90. sort -k2 -t, -n @cptntommy@cptntommy
  91. 91. sort -k2 -t, -n -r @cptntommy@cptntommy
  92. 92. sort -k2 -t, -n -r filename.csv @cptntommy@cptntommy
  93. 93. sort -k2 -t, -n -r filename.csv > volumesorteddata.csv @cptntommy@cptntommy
  94. 94. head volumesorteddata.csv > top10KW.csv @cptntommy@cptntommy
  95. 95. head -n100 volumesorteddata.csv > top100KW.csv @cptntommy@cptntommy
  96. 96. Here sort has been used to: Sort by Number, Sort by A-Z, Sort by Z-A @cptntommy@cptntommy
  97. 97. We have also used head/tail to; Extract the top 10, Top 100 Save these to a file @cptntommy@cptntommy
  98. 98. Title
  99. 99. @cptntommy@cptntommy cat - short for “concatenate”
  100. 100. @cptntommy@cptntommy Used to display, combine & create files
  101. 101. @cptntommy@cptntommy
  102. 102. @cptntommy@cptntommy
  103. 103. @cptntommy@cptntommy >
  104. 104. @cptntommy@cptntommy cat
  105. 105. @cptntommy@cptntommy cat *.csv
  106. 106. @cptntommy@cptntommy cat *.csv > alldata.csv
  107. 107. @cptntommy@cptntommy > User/Desktop/DataToCombine
  108. 108. sed
  109. 109. @cptntommy@cptntommy Short for "stream editor", allows you to filter and transform text.
  110. 110. @cptntommy@cptntommy Adding text to rows
  111. 111. @cptntommy
  112. 112. @cptntommy@cptntommy
  113. 113. @cptntommy@cptntommy Use sed to add protocol & domain at start of each line
  114. 114. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv
  115. 115. @cptntommy@cptntommy REGEX!!
  116. 116. @cptntommy@cptntommy sed -e 's/^/
  117. 117. @cptntommy@cptntommy sed -e 's/^/http://www.domain.co m/'
  118. 118. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/'
  119. 119. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv
  120. 120. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv > full.csv
  121. 121. @cptntommy@cptntommy
  122. 122. @cptntommy@cptntommy use sed to find & replace
  123. 123. @cptntommy@cptntommy Replace http with https
  124. 124. @cptntommy@cptntommy sed -e 's/http file.csv > newfile.csv /https/'
  125. 125. @cptntommy
  126. 126. @cptntommy@cptntommy Here we have: Added in protocol & domain Replaced http with https
  127. 127. awk
  128. 128. @cptntommy@cptntommy Programming language, used to process text
  129. 129. @cptntommy@cptntommy Following example is one of the easier applications of awk
  130. 130. @cptntommy@cptntommy
  131. 131. @cptntommy@cptntommy 1 7
  132. 132. @cptntommy@cptntommy
  133. 133. @cptntommy@cptntommy awk -F “,”
  134. 134. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’
  135. 135. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’ kw.csv > betterkw.csv
  136. 136. @cptntommy
  137. 137. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’
  138. 138. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2 “,” $4}’
  139. 139. @cptntommy@cptntommy How to combine all this?
  140. 140. Log File Analysis @cptntommy
  141. 141. @cptntommy About 1% of given Log Files
  142. 142. @cptntommy 1.cat Combine files together
  143. 143. @cptntommy Cat *.log > combinedlogs.log
  144. 144. @cptntommy 2. Use sed to add in full domain name to log file
  145. 145. @cptntommy
  146. 146. @cptntommy
  147. 147. @cptntommy sed -e 's // file.log > newfile.log /domain.com/'
  148. 148. @cptntommy sed -e 's Search
  149. 149. @cptntommy sed -e 's // Forward Slash
  150. 150. @cptntommy sed -e 's // /domain.com/' Text to replace with
  151. 151. @cptntommy sed -e 's // file.log > newfile.log /domain.com/' Then the files
  152. 152. @cptntommy 3.Extract 404 errors (awk)
  153. 153. @cptntommy awk '/404/ {print $0}' combinedlogs.log > log404.log
  154. 154. @cptntommy 4.Extract any specific status code (awk)
  155. 155. @cptntommy awk '/301/ {print $0}' file.log > newfile.log
  156. 156. @cptntommy awk '/200/ {print $0}' file.log > newfile.log
  157. 157. @cptntommy awk '/503/ {print $0}' file.log > newfile.log
  158. 158. @cptntommy awk '/418/ {print $0}' file.log > newfile.log
  159. 159. @cptntommy 5. Extract all Googlebot/ Mobile Bot hits (awk)
  160. 160. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  161. 161. @cptntommy What if you wanted to know how many hits each bot had, without opening the file?
  162. 162. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  163. 163. @cptntommy wc
  164. 164. @cptntommy wc -l gbot.log
  165. 165. @cptntommy
  166. 166. @cptntommy 6. Extract all hits of a specific URL, or all img requests, all css requests, all with modification of 1 command
  167. 167. @cptntommy Replace ‘Googlebot’ with ‘BingBot’, ‘anybot’...
  168. 168. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  169. 169. @cptntommy awk '/Bingbot/ {print $0}' all.log > gbot.log
  170. 170. @cptntommy awk '/anybot/ {print $0}' all.log > gbot.log
  171. 171. @cptntommy awk '/.css/ {print $0}' all.log > gbot.log
  172. 172. @cptntommy awk '/.jpeg/ {print $0}' all.log > gbot.log
  173. 173. @cptntommy awk '/whatever-you-want/ {print $0}' all.log > file.log
  174. 174. @cptntommy This Barely Scratches the Surface
  175. 175. @cptntommy So, To Recap
  176. 176. You can now hack: @cptntommy
  177. 177. Server Logs @cptntommy
  178. 178. Keyword data @cptntommy
  179. 179. .csv files @cptntommy
  180. 180. Extract columns of data @cptntommy
  181. 181. Sort data @cptntommy
  182. 182. General SEO Shit @cptntommy
  183. 183. And @cptntommy
  184. 184. #BrightonSEO @cptntommy
  185. 185. Thanks! @cptntommy

×