Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Tom Pool
BlueArray
Command Line Hacks For SEO
@cptntommy
https://www.slideshare.net/TomPool
Who Am I?
@cptntommy
Technical SEO Manager
@ BlueArray
#BrightonSEO @cptntommy
Look after
technical
output for all
clients @
BlueArray
@cptntommy@cptntommy
This means I get to work on
loads of different clients, doing
loads of different tasks
@cptntommy@cptntommy
Checking Response
Codes
@cptntommy@cptntommy
Tech Audits
@cptntommy@cptntommy
Analysis Of
@cptntommy@cptntommy
Keywords
@cptntommy@cptntommy
Keyword Gap
@cptntommy@cptntommy
Server Log Files
@cptntommy@cptntommy
Crawl Data
@cptntommy@cptntommy
The Problem Is...
@cptntommy
Command Line Hacks For SEO
Insert picture of hours ticking away?
Full screen
Each of
these tasks
takes a lot
of time
@cptntommy@cptntommy
@cptntommy@cptntommy
< <?
@cptntommy@cptntommy
< <
@cptntommy@cptntommy
Command Line
Hacks for SEO
@cptntommy@cptntommy
Terminal
What Is Command
Line?
@cptntommy@cptntommy
It is a basic interface between
you and the computer
@cptntommy@cptntommy
What Is Command Line?
What Is Command Line?
The following references are
related to Mac OS’s Terminal
application.
@cptntommy@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
What Commands Do I
Use?
@cptntommy@cptntommy
CURL
SORT
CAT
SPLIT
SED
AWK
@cptntommy@cptntommy
Keyword Analysis
Keyword Gap Analysis
Checking Response Codes
Log File Analysis
Crawling & Analysis
@cptntommy@cptntommy
curl
What is curl?
@cptntommy@cptntommy
“curl is a tool to transfer data
from or to a server”
@cptntommy@cptntommy
“curl is a tool to transfer data
from or to a server”
@cptntommy@cptntommy
*Examples shown are not the full usage, for ful...
Checking Response Codes
@cptntommy@cptntommy
Toms-MacBook-Pro:~ tompool$
https://www.bluearray.co.uk
@cptntommy@cptntommy
Toms-MacBook-Pro:~ tompool$
https://www.bluearray.co.ukcurl
@cptntommy@cptntommy
Toms-MacBook-Pro:~ tompool$
https://www.bluearray.co.ukhttps://www.bluearray.co.ukcurl
@cptntommy@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
What If I Just Want To See The
HTTP Header?
@cptntommy
@cptntommy@cptntommy
curl
@cptntommy@cptntommy
curl -I
@cptntommy@cptntommy
Modifier - Just The HTTP
Header!
curl -I https://bluearray.co.uk
@cptntommy@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
curl -I -L
@cptntommy@cptntommy
curl -I -L https://bluearray.co.uk
@cptntommy@cptntommy
@cptntommy@cptntommy
curl-O
@cptntommy@cptntommy
curl-O
https://download.screami
ngfrog.co.uk/products/seo
-spider/ScreamingFrogSEO
Spider-8.3.dmg
@cptntommy@cptntommy
@cptntommy@cptntommy
Here we have used CURL TO:
Download Files,
Check HTML,
Check HTTP Header &
Follow Redirects
@cptntommy@cptntommy
@cptntommy@cptntommy
sort
Sort ‘sorts’
@cptntommy@cptntommy
@cptntommy@cptntommy
Sort - A-Z
@cptntommy@cptntommy
@cptntommy@cptntommy
Navigate to Folder and use ‘ls - “list”’
command to make sure the data is
there
@cptntommy@cptntommy
Then run the sort command
@cptntommy@cptntommy
sort
@cptntommy@cptntommy
@cptntommy@cptntommy
filename.csvsort
“sort keyworddata.csv”
@cptntommy@cptntommy
@cptntommy@cptntommy
sort filename > newfilename
@cptntommy@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
Did It Work?
head/tail
WTF is Head/Tail?
@cptntommy@cptntommy
‘Head’ views the first 10 Rows
‘Tail’ views the last 10 Rows
@cptntommy@cptntommy
head filename.csv
@cptntommy@cptntommy
head filename
@cptntommy@cptntommy
tail filename
@cptntommy
Sort - Z-A
sort filename.csv >
Z-A_SortedData.csv
@cptntommy@cptntommy
sort filename.csv >
Z-A_SortedData.csv
@cptntommy@cptntommy
-r
@cptntommy@cptntommy
Sort - By Volume
@cptntommy@cptntommy
@cptntommy@cptntommy
We want to sort by the
second column
@cptntommy@cptntommy
sort
@cptntommy@cptntommy
sort -k2
@cptntommy@cptntommy
sort -k2 -t,
@cptntommy@cptntommy
sort -k2 -t, -n
@cptntommy@cptntommy
sort -k2 -t, -n -r
@cptntommy@cptntommy
sort -k2 -t, -n -r filename.csv
@cptntommy@cptntommy
sort -k2 -t, -n -r filename.csv >
volumesorteddata.csv
@cptntommy@cptntommy
head volumesorteddata.csv >
top10KW.csv
@cptntommy@cptntommy
head -n100
volumesorteddata.csv >
top100KW.csv
@cptntommy@cptntommy
Here sort has been used to:
Sort by Number,
Sort by A-Z,
Sort by Z-A
@cptntommy@cptntommy
We have also used head/tail to;
Extract the top 10,
Top 100
Save these to a file
@cptntommy@cptntommy
Title
@cptntommy@cptntommy
cat - short for “concatenate”
@cptntommy@cptntommy
Used to display, combine &
create files
@cptntommy@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
>
@cptntommy@cptntommy
cat
@cptntommy@cptntommy
cat *.csv
@cptntommy@cptntommy
cat *.csv > alldata.csv
@cptntommy@cptntommy
>
User/Desktop/DataToCombine
sed
@cptntommy@cptntommy
Short for "stream editor",
allows you to filter and
transform text.
@cptntommy@cptntommy
Adding text to rows
@cptntommy
@cptntommy@cptntommy
@cptntommy@cptntommy
Use sed to add protocol &
domain at start of each line
@cptntommy@cptntommy
sed -e
's/^/http://www.domain
.com/' file.csv
@cptntommy@cptntommy
REGEX!!
@cptntommy@cptntommy
sed -e 's/^/
@cptntommy@cptntommy
sed -e
's/^/http://www.domain.co
m/'
@cptntommy@cptntommy
sed -e
's/^/http://www.domain
.com/'
@cptntommy@cptntommy
sed -e
's/^/http://www.domain
.com/' file.csv
@cptntommy@cptntommy
sed -e
's/^/http://www.domain
.com/' file.csv > full.csv
@cptntommy@cptntommy
@cptntommy@cptntommy
use sed to find & replace
@cptntommy@cptntommy
Replace http with https
@cptntommy@cptntommy
sed -e 's/http file.csv
> newfile.csv
/https/'
@cptntommy
@cptntommy@cptntommy
Here we have:
Added in protocol & domain
Replaced http with https
awk
@cptntommy@cptntommy
Programming language,
used to process text
@cptntommy@cptntommy
Following example is one of
the easier applications of
awk
@cptntommy@cptntommy
@cptntommy@cptntommy
1 7
@cptntommy@cptntommy
@cptntommy@cptntommy
awk -F “,”
@cptntommy@cptntommy
awk -F “,”‘{print $1 “,” $2}’
@cptntommy@cptntommy
awk -F “,”‘{print $1 “,” $2}’ kw.csv
> betterkw.csv
@cptntommy
@cptntommy@cptntommy
awk -F “,”‘{print $1 “,” $2}’
@cptntommy@cptntommy
awk -F “,”‘{print $1 “,” $2 “,” $4}’
@cptntommy@cptntommy
How to
combine
all this?
Log File
Analysis
@cptntommy
@cptntommy
About 1%
of given
Log Files
@cptntommy
1.cat
Combine
files
together
@cptntommy
Cat *.log > combinedlogs.log
@cptntommy
2. Use sed to add in full domain
name to log file
@cptntommy
@cptntommy
@cptntommy
sed -e 's //
file.log > newfile.log
/domain.com/'
@cptntommy
sed -e 's
Search
@cptntommy
sed -e 's //
Forward Slash
@cptntommy
sed -e 's // /domain.com/'
Text to replace with
@cptntommy
sed -e 's //
file.log > newfile.log
/domain.com/'
Then the files
@cptntommy
3.Extract 404 errors
(awk)
@cptntommy
awk '/404/ {print $0}'
combinedlogs.log > log404.log
@cptntommy
4.Extract any
specific
status code
(awk)
@cptntommy
awk '/301/ {print $0}' file.log >
newfile.log
@cptntommy
awk '/200/ {print $0}' file.log >
newfile.log
@cptntommy
awk '/503/ {print $0}' file.log >
newfile.log
@cptntommy
awk '/418/ {print $0}' file.log >
newfile.log
@cptntommy
5. Extract all
Googlebot/
Mobile Bot
hits (awk)
@cptntommy
awk '/Googlebot/ {print $0}'
all.log > gbot.log
@cptntommy
What if you wanted to know
how many hits each bot had,
without opening the file?
@cptntommy
awk '/Googlebot/ {print $0}'
all.log > gbot.log
@cptntommy
wc
@cptntommy
wc -l gbot.log
@cptntommy
@cptntommy
6. Extract all hits of a specific
URL, or all img requests, all css
requests, all with modification of
1 command
@cptntommy
Replace ‘Googlebot’ with
‘BingBot’, ‘anybot’...
@cptntommy
awk '/Googlebot/ {print $0}'
all.log > gbot.log
@cptntommy
awk '/Bingbot/ {print $0}' all.log >
gbot.log
@cptntommy
awk '/anybot/ {print $0}' all.log >
gbot.log
@cptntommy
awk '/.css/ {print $0}' all.log >
gbot.log
@cptntommy
awk '/.jpeg/ {print $0}' all.log >
gbot.log
@cptntommy
awk '/whatever-you-want/ {print
$0}' all.log > file.log
@cptntommy
This Barely
Scratches
the
Surface
@cptntommy
So, To Recap
You can now
hack:
@cptntommy
Server Logs
@cptntommy
Keyword data
@cptntommy
.csv files
@cptntommy
Extract columns
of data
@cptntommy
Sort data
@cptntommy
General SEO Shit
@cptntommy
And
@cptntommy
#BrightonSEO @cptntommy
Thanks!
@cptntommy
Command Line Hacks For SEO - Brighton April 2018 - Tom Pool
Upcoming SlideShare
Loading in …5
×

Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

7,279 views

Published on

Slide deck from Brighton SEO April 2018 - 1st Session of the day (Technical SEO).

Published in: Marketing
  • Visit this site: tinyurl.com/sexinarea and find sex in your area for one night)) You can find me on this site too)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Command Line Hacks For SEO - Brighton April 2018 - Tom Pool

  1. 1. Tom Pool BlueArray Command Line Hacks For SEO @cptntommy https://www.slideshare.net/TomPool
  2. 2. Who Am I?
  3. 3. @cptntommy Technical SEO Manager @ BlueArray #BrightonSEO @cptntommy
  4. 4. Look after technical output for all clients @ BlueArray @cptntommy@cptntommy
  5. 5. This means I get to work on loads of different clients, doing loads of different tasks @cptntommy@cptntommy
  6. 6. Checking Response Codes @cptntommy@cptntommy
  7. 7. Tech Audits @cptntommy@cptntommy
  8. 8. Analysis Of @cptntommy@cptntommy
  9. 9. Keywords @cptntommy@cptntommy
  10. 10. Keyword Gap @cptntommy@cptntommy
  11. 11. Server Log Files @cptntommy@cptntommy
  12. 12. Crawl Data @cptntommy@cptntommy
  13. 13. The Problem Is... @cptntommy
  14. 14. Command Line Hacks For SEO Insert picture of hours ticking away? Full screen Each of these tasks takes a lot of time
  15. 15. @cptntommy@cptntommy
  16. 16. @cptntommy@cptntommy
  17. 17. < <? @cptntommy@cptntommy
  18. 18. < < @cptntommy@cptntommy
  19. 19. Command Line Hacks for SEO @cptntommy@cptntommy Terminal
  20. 20. What Is Command Line? @cptntommy@cptntommy
  21. 21. It is a basic interface between you and the computer @cptntommy@cptntommy
  22. 22. What Is Command Line?
  23. 23. What Is Command Line?
  24. 24. The following references are related to Mac OS’s Terminal application. @cptntommy@cptntommy
  25. 25. @cptntommy@cptntommy
  26. 26. @cptntommy@cptntommy
  27. 27. @cptntommy@cptntommy
  28. 28. What Commands Do I Use? @cptntommy@cptntommy
  29. 29. CURL SORT CAT SPLIT SED AWK @cptntommy@cptntommy
  30. 30. Keyword Analysis Keyword Gap Analysis Checking Response Codes Log File Analysis Crawling & Analysis @cptntommy@cptntommy
  31. 31. curl
  32. 32. What is curl? @cptntommy@cptntommy
  33. 33. “curl is a tool to transfer data from or to a server” @cptntommy@cptntommy
  34. 34. “curl is a tool to transfer data from or to a server” @cptntommy@cptntommy *Examples shown are not the full usage, for full info check out the manual page (man curl)
  35. 35. Checking Response Codes @cptntommy@cptntommy
  36. 36. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.uk @cptntommy@cptntommy
  37. 37. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.ukcurl @cptntommy@cptntommy
  38. 38. Toms-MacBook-Pro:~ tompool$ https://www.bluearray.co.ukhttps://www.bluearray.co.ukcurl @cptntommy@cptntommy
  39. 39. @cptntommy@cptntommy
  40. 40. @cptntommy@cptntommy
  41. 41. What If I Just Want To See The HTTP Header? @cptntommy
  42. 42. @cptntommy@cptntommy
  43. 43. curl @cptntommy@cptntommy
  44. 44. curl -I @cptntommy@cptntommy Modifier - Just The HTTP Header!
  45. 45. curl -I https://bluearray.co.uk @cptntommy@cptntommy
  46. 46. @cptntommy@cptntommy
  47. 47. @cptntommy@cptntommy
  48. 48. @cptntommy@cptntommy
  49. 49. @cptntommy@cptntommy
  50. 50. curl -I -L @cptntommy@cptntommy
  51. 51. curl -I -L https://bluearray.co.uk @cptntommy@cptntommy
  52. 52. @cptntommy@cptntommy
  53. 53. curl-O @cptntommy@cptntommy
  54. 54. curl-O https://download.screami ngfrog.co.uk/products/seo -spider/ScreamingFrogSEO Spider-8.3.dmg @cptntommy@cptntommy
  55. 55. @cptntommy@cptntommy
  56. 56. Here we have used CURL TO: Download Files, Check HTML, Check HTTP Header & Follow Redirects @cptntommy@cptntommy
  57. 57. @cptntommy@cptntommy
  58. 58. sort
  59. 59. Sort ‘sorts’ @cptntommy@cptntommy
  60. 60. @cptntommy@cptntommy
  61. 61. Sort - A-Z
  62. 62. @cptntommy@cptntommy
  63. 63. @cptntommy@cptntommy
  64. 64. Navigate to Folder and use ‘ls - “list”’ command to make sure the data is there @cptntommy@cptntommy
  65. 65. Then run the sort command @cptntommy@cptntommy
  66. 66. sort @cptntommy@cptntommy
  67. 67. @cptntommy@cptntommy filename.csvsort
  68. 68. “sort keyworddata.csv” @cptntommy@cptntommy
  69. 69. @cptntommy@cptntommy
  70. 70. sort filename > newfilename @cptntommy@cptntommy
  71. 71. @cptntommy@cptntommy
  72. 72. @cptntommy@cptntommy Did It Work?
  73. 73. head/tail
  74. 74. WTF is Head/Tail? @cptntommy@cptntommy
  75. 75. ‘Head’ views the first 10 Rows ‘Tail’ views the last 10 Rows @cptntommy@cptntommy
  76. 76. head filename.csv @cptntommy@cptntommy
  77. 77. head filename @cptntommy@cptntommy
  78. 78. tail filename @cptntommy
  79. 79. Sort - Z-A
  80. 80. sort filename.csv > Z-A_SortedData.csv @cptntommy@cptntommy
  81. 81. sort filename.csv > Z-A_SortedData.csv @cptntommy@cptntommy -r
  82. 82. @cptntommy@cptntommy
  83. 83. Sort - By Volume
  84. 84. @cptntommy@cptntommy
  85. 85. @cptntommy@cptntommy We want to sort by the second column
  86. 86. @cptntommy@cptntommy
  87. 87. sort @cptntommy@cptntommy
  88. 88. sort -k2 @cptntommy@cptntommy
  89. 89. sort -k2 -t, @cptntommy@cptntommy
  90. 90. sort -k2 -t, -n @cptntommy@cptntommy
  91. 91. sort -k2 -t, -n -r @cptntommy@cptntommy
  92. 92. sort -k2 -t, -n -r filename.csv @cptntommy@cptntommy
  93. 93. sort -k2 -t, -n -r filename.csv > volumesorteddata.csv @cptntommy@cptntommy
  94. 94. head volumesorteddata.csv > top10KW.csv @cptntommy@cptntommy
  95. 95. head -n100 volumesorteddata.csv > top100KW.csv @cptntommy@cptntommy
  96. 96. Here sort has been used to: Sort by Number, Sort by A-Z, Sort by Z-A @cptntommy@cptntommy
  97. 97. We have also used head/tail to; Extract the top 10, Top 100 Save these to a file @cptntommy@cptntommy
  98. 98. Title
  99. 99. @cptntommy@cptntommy cat - short for “concatenate”
  100. 100. @cptntommy@cptntommy Used to display, combine & create files
  101. 101. @cptntommy@cptntommy
  102. 102. @cptntommy@cptntommy
  103. 103. @cptntommy@cptntommy >
  104. 104. @cptntommy@cptntommy cat
  105. 105. @cptntommy@cptntommy cat *.csv
  106. 106. @cptntommy@cptntommy cat *.csv > alldata.csv
  107. 107. @cptntommy@cptntommy > User/Desktop/DataToCombine
  108. 108. sed
  109. 109. @cptntommy@cptntommy Short for "stream editor", allows you to filter and transform text.
  110. 110. @cptntommy@cptntommy Adding text to rows
  111. 111. @cptntommy
  112. 112. @cptntommy@cptntommy
  113. 113. @cptntommy@cptntommy Use sed to add protocol & domain at start of each line
  114. 114. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv
  115. 115. @cptntommy@cptntommy REGEX!!
  116. 116. @cptntommy@cptntommy sed -e 's/^/
  117. 117. @cptntommy@cptntommy sed -e 's/^/http://www.domain.co m/'
  118. 118. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/'
  119. 119. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv
  120. 120. @cptntommy@cptntommy sed -e 's/^/http://www.domain .com/' file.csv > full.csv
  121. 121. @cptntommy@cptntommy
  122. 122. @cptntommy@cptntommy use sed to find & replace
  123. 123. @cptntommy@cptntommy Replace http with https
  124. 124. @cptntommy@cptntommy sed -e 's/http file.csv > newfile.csv /https/'
  125. 125. @cptntommy
  126. 126. @cptntommy@cptntommy Here we have: Added in protocol & domain Replaced http with https
  127. 127. awk
  128. 128. @cptntommy@cptntommy Programming language, used to process text
  129. 129. @cptntommy@cptntommy Following example is one of the easier applications of awk
  130. 130. @cptntommy@cptntommy
  131. 131. @cptntommy@cptntommy 1 7
  132. 132. @cptntommy@cptntommy
  133. 133. @cptntommy@cptntommy awk -F “,”
  134. 134. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’
  135. 135. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’ kw.csv > betterkw.csv
  136. 136. @cptntommy
  137. 137. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2}’
  138. 138. @cptntommy@cptntommy awk -F “,”‘{print $1 “,” $2 “,” $4}’
  139. 139. @cptntommy@cptntommy How to combine all this?
  140. 140. Log File Analysis @cptntommy
  141. 141. @cptntommy About 1% of given Log Files
  142. 142. @cptntommy 1.cat Combine files together
  143. 143. @cptntommy Cat *.log > combinedlogs.log
  144. 144. @cptntommy 2. Use sed to add in full domain name to log file
  145. 145. @cptntommy
  146. 146. @cptntommy
  147. 147. @cptntommy sed -e 's // file.log > newfile.log /domain.com/'
  148. 148. @cptntommy sed -e 's Search
  149. 149. @cptntommy sed -e 's // Forward Slash
  150. 150. @cptntommy sed -e 's // /domain.com/' Text to replace with
  151. 151. @cptntommy sed -e 's // file.log > newfile.log /domain.com/' Then the files
  152. 152. @cptntommy 3.Extract 404 errors (awk)
  153. 153. @cptntommy awk '/404/ {print $0}' combinedlogs.log > log404.log
  154. 154. @cptntommy 4.Extract any specific status code (awk)
  155. 155. @cptntommy awk '/301/ {print $0}' file.log > newfile.log
  156. 156. @cptntommy awk '/200/ {print $0}' file.log > newfile.log
  157. 157. @cptntommy awk '/503/ {print $0}' file.log > newfile.log
  158. 158. @cptntommy awk '/418/ {print $0}' file.log > newfile.log
  159. 159. @cptntommy 5. Extract all Googlebot/ Mobile Bot hits (awk)
  160. 160. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  161. 161. @cptntommy What if you wanted to know how many hits each bot had, without opening the file?
  162. 162. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  163. 163. @cptntommy wc
  164. 164. @cptntommy wc -l gbot.log
  165. 165. @cptntommy
  166. 166. @cptntommy 6. Extract all hits of a specific URL, or all img requests, all css requests, all with modification of 1 command
  167. 167. @cptntommy Replace ‘Googlebot’ with ‘BingBot’, ‘anybot’...
  168. 168. @cptntommy awk '/Googlebot/ {print $0}' all.log > gbot.log
  169. 169. @cptntommy awk '/Bingbot/ {print $0}' all.log > gbot.log
  170. 170. @cptntommy awk '/anybot/ {print $0}' all.log > gbot.log
  171. 171. @cptntommy awk '/.css/ {print $0}' all.log > gbot.log
  172. 172. @cptntommy awk '/.jpeg/ {print $0}' all.log > gbot.log
  173. 173. @cptntommy awk '/whatever-you-want/ {print $0}' all.log > file.log
  174. 174. @cptntommy This Barely Scratches the Surface
  175. 175. @cptntommy So, To Recap
  176. 176. You can now hack: @cptntommy
  177. 177. Server Logs @cptntommy
  178. 178. Keyword data @cptntommy
  179. 179. .csv files @cptntommy
  180. 180. Extract columns of data @cptntommy
  181. 181. Sort data @cptntommy
  182. 182. General SEO Shit @cptntommy
  183. 183. And @cptntommy
  184. 184. #BrightonSEO @cptntommy
  185. 185. Thanks! @cptntommy

×