This document discusses how analyzing website log files can provide insights into how a site is crawled by Google and reveal opportunities to improve search engine optimization. It recommends asking hosting providers about log file access and server setup. Excel and log file analysis tools are presented as options for analyzing log files. Specific metrics that can be analyzed include the most and least crawled pages, crawl frequency by depth and internal links, low-value URLs wasting crawl budget, and server response errors by page that could impact site health.

1. @FaisalAnderson
Spying on Google
How to Use Log File Analysis to Reveal
Invaluable SEO Insights

2. @FaisalAnderson
Imagine if you could
Spy on Google...

3. @FaisalAnderson
Understand how your
Site is Crawled

4. @FaisalAnderson
Quickly find why
content isn’t ranking

5. @FaisalAnderson
Find Quick Tech SEO Wins

6. @FaisalAnderson
See if fresh content being
discovered

7. @FaisalAnderson
LOG FILE ANALYSIS

8. @FaisalAnderson

9. @FaisalAnderson
Log Files
You Say?

10. @FaisalAnderson
WTF really is
Log File Analysis?

11. @FaisalAnderson

12. @FaisalAnderson
Log File

13. @FaisalAnderson
HUGE POTENTIAL

14. @FaisalAnderson
SEARCH CONSOLE IS…
____________________________

15. @FaisalAnderson
SEARCH CONSOLE IS…
____________________________
KINDA SHIT
- FAISAL ANDERSON 2K19

16. @FaisalAnderson
1) Data accuracy is NOT guaranteed

17. @FaisalAnderson
1) Data accuracy is NOT guaranteed
2) Only 1000 rows available in the GUI

18. @FaisalAnderson
1) Data Accuracy Is NOT Guaranteed
2) Only 1000 rows available in the GUI
3) Google owns it

19. @FaisalAnderson
4) Fresh AF

20. @FaisalAnderson

21. @FaisalAnderson

22. @FaisalAnderson

23. @FaisalAnderson

24. @FaisalAnderson
CONVINCED?
LET’S DIVE IN

25. @FaisalAnderson
3 THINGS

26. @FaisalAnderson
1) Getting the Log Files

27. @FaisalAnderson
1) Getting the Log Files
2) Choosing the best Tool

28. @FaisalAnderson
1) Getting the Log Files
2) Choosing the best Tool
3) Do Our Analysis

29. @FaisalAnderson
GETTING THE LOG
FILES

30. @FaisalAnderson

31. @FaisalAnderson

32. @FaisalAnderson

33. @FaisalAnderson
1) Ask about Host Setup

34. @FaisalAnderson
COMMON
HOSTING SCENARIOS

35. @FaisalAnderson
Load Balancing

36. @FaisalAnderson

37. @FaisalAnderson

38. @FaisalAnderson
CDNs

39. @FaisalAnderson
1) Ask about Server setup
2) Ask about the hostname

40. @FaisalAnderson
faisalfuntime.com

41. @FaisalAnderson
https://

42. @FaisalAnderson

43. @FaisalAnderson
/faisal/loves/bananas

44. @FaisalAnderson

45. @FaisalAnderson
1) Ask about Server setup
2) Ask about the hostname
3) Get 6 months of Data

46. @FaisalAnderson

47. @FaisalAnderson

48. @FaisalAnderson

49. @FaisalAnderson

50. @FaisalAnderson

51. @FaisalAnderson
copy *.log mylogfile.csv
bash$ cat file*.txt > mergedfile.csv

52. @FaisalAnderson
2.
Choosing the
Right Tool

53. @FaisalAnderson

54. @FaisalAnderson
Tools

55. @FaisalAnderson
Excel

56. @FaisalAnderson
Easy to use

57. @FaisalAnderson

58. @FaisalAnderson

59. @FaisalAnderson
Log File Analyser

60. @FaisalAnderson

61. @FaisalAnderson

62. @FaisalAnderson

63. @FaisalAnderson
Easy to use and quick

64. @FaisalAnderson
Can’t create custom reports

65. @FaisalAnderson

66. @FaisalAnderson

67. @FaisalAnderson

68. @FaisalAnderson
HUGE FILES

69. @FaisalAnderson
Larger Learning Curve

70. @FaisalAnderson

71. @FaisalAnderson

72. @FaisalAnderson

73. @FaisalAnderson
3.
Do Some Analysis

74. @FaisalAnderson
3
CATEGORIES

75. @FaisalAnderson
CRAWL BEHAVIOUR

76. @FaisalAnderson
CRAWL BUDGET WASTE

77. @FaisalAnderson
SITE HEALTH

78. @FaisalAnderson

79. @FaisalAnderson
CRAWL BEHAVIOUR

80. @FaisalAnderson
CASE STUDY

81. @FaisalAnderson

82. @FaisalAnderson

83. @FaisalAnderson

84. @FaisalAnderson

85. @FaisalAnderson

86. @FaisalAnderson

87. @FaisalAnderson

88. @FaisalAnderson

89. @FaisalAnderson

90. @FaisalAnderson
MOST AND LEAST
CRAWLED URLS

91. @FaisalAnderson

92. @FaisalAnderson

93. @FaisalAnderson

94. @FaisalAnderson

95. @FaisalAnderson
CRAWL FREQUENCY
BY DEPTH AND
INTERNAL LINKS

96. @FaisalAnderson

97. @FaisalAnderson

98. @FaisalAnderson

99. @FaisalAnderson

100. @FaisalAnderson

101. @FaisalAnderson

102. @FaisalAnderson

103. @FaisalAnderson

104. @FaisalAnderson

105. @FaisalAnderson

106. @FaisalAnderson

107. @FaisalAnderson

108. @FaisalAnderson
CRAWL BUDGET
WASTE

109. @FaisalAnderson

110. @FaisalAnderson

111. @FaisalAnderson
CASE STUDY #2

112. @FaisalAnderson

113. @FaisalAnderson

114. @FaisalAnderson
E > NACHOS > CHEESE

115. @FaisalAnderson
CHEESE

116. @FaisalAnderson
CHEESE
CHEESE

117. @FaisalAnderson
CHEESE
CHEESE
CHEESE

118. @FaisalAnderson

119. @FaisalAnderson

120. @FaisalAnderson
2 CHECKS

121. @FaisalAnderson
Find Low
Value-Add URLs

122. @FaisalAnderson

123. @FaisalAnderson

124. @FaisalAnderson

125. @FaisalAnderson

126. @FaisalAnderson
Crawl Frequency
by Subdirectory

127. @FaisalAnderson

128. @FaisalAnderson

129. @FaisalAnderson

130. @FaisalAnderson

131. @FaisalAnderson
SITE HEALTH

132. @FaisalAnderson
CASE STUDY #3

133. @FaisalAnderson
LARGE
NEWS SITE

134. @FaisalAnderson
SMARTPHONE
DESKTOP

135. @FaisalAnderson

136. @FaisalAnderson

137. @FaisalAnderson

138. @FaisalAnderson

139. @FaisalAnderson

140. @FaisalAnderson
LET’S ANALYSE...

141. @FaisalAnderson
INCONSISTENT
SERVER RESPONSES

142. @FaisalAnderson
200

143. @FaisalAnderson

144. @FaisalAnderson
500

145. @FaisalAnderson

146. @FaisalAnderson

147. @FaisalAnderson

148. @FaisalAnderson

149. @FaisalAnderson

150. @FaisalAnderson

151. @FaisalAnderson

152. @FaisalAnderson

153. @FaisalAnderson
Errors by Subdirectory

154. @FaisalAnderson

155. @FaisalAnderson

156. @FaisalAnderson

157. @FaisalAnderson

158. @FaisalAnderson
Finito!

159. @FaisalAnderson
Download the Scripts

160. @FaisalAnderson
bit.ly/log-file-scripts

161. @FaisalAnderson
WE’RE HIRING!