12 advanced uses of Screaming Frog's SEO Spider tool for SEO audits and general web development jobs. Links included to our indepth guides.

1. 12 Advanced Uses
of the Screaming Frog
SEO Spider
@screamingfrog
@oliverbrett
@stephhugman

2. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV
@screamingfrog
@oliverbrett
@stephhugman

3. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV
@screamingfrog
@oliverbrett
@stephhugman

4. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV
@screamingfrog
@oliverbrett
@stephhugman

5. @screamingfrog
@oliverbrett
@stephhugman

6. Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

7. Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

8. Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

9. Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

10. Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

11. Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

12. Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

13. Introduce SF, explain agency

14. Introduce SF, explain agency

15. Introduce SF, explain agency

16. @screamingfrog
@oliverbrett
@stephhugman

17. Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

18. @screamingfrog
@oliverbrett
@stephhugman

19. Explain history of tool
2011
@screamingfrog
@oliverbrett
@stephhugman

20. Explain fame
@screamingfrog
@oliverbrett
@stephhugman

21. “The Swiss
Army Knife
of SEO”
SUN TZU, PROBABLY
@screamingfrog
@oliverbrett
@stephhugman

22. Screaming
Frog Can…
@screamingfrog
@oliverbrett
@stephhugman

23. https://www.seerinteractive.com/blog/screaming-frog-guide/
@screamingfrog
@oliverbrett
@stephhugman

24. https://www.seerinteractive.com/blog/screaming-frog-guide/
@screamingfrog
@oliverbrett
@stephhugman

25. https://www.seerinteractive.com/blog/screaming-frog-guide/
@screamingfrog
@oliverbrett
@stephhugman

26. https://www.seerinteractive.com/blog/screaming-frog-guide/
@screamingfrog
@oliverbrett
@stephhugman

27. @screamingfrog
@oliverbrett
@stephhugman

28. We don’t have the
answers.
We have the data.
@screamingfrog
@oliverbrett
@stephhugman

29. @screamingfrog
@oliverbrett
@stephhugman

30. @screamingfrog
@oliverbrett
@stephhugman

31. A tool built by SEOs.
For SEOs. And you!
@screamingfrog
@oliverbrett
@stephhugman

32. And you!
@screamingfrog
@oliverbrett
@stephhugman

33. 3 Rules For Today:
@screamingfrog
@oliverbrett
@stephhugman

34. 1. This isn’t a
sales pitch.
@screamingfrog
@oliverbrett
@stephhugman

35. @screamingfrog
@oliverbrett
@stephhugman

36. @screamingfrog
@oliverbrett
@stephhugman

37. €239 A Year!
@screamingfrog
@oliverbrett
@stephhugman

38. €239 A Year!
Just 60 Pints!!
Just 24 plates of Käsespätzle!!
@screamingfrog
@oliverbrett
@stephhugman

39. €239 A Year!
Just 24 plates of Käsespätzle!!
@screamingfrog
@oliverbrett
@stephhugman

40. 2. This isn’t a
walkthrough.
@screamingfrog
@oliverbrett
@stephhugman

41. @screamingfrog
@oliverbrett
@stephhugman

42. Links will be
provided for in-depth
guides. Don’t panic.
@screamingfrog
@oliverbrett
@stephhugman

43. 3. Just ask if you
need help!
@screamingfrog
@oliverbrett
@stephhugman

44. @screamingfrog
@oliverbrett
@stephhugman

45. Setting Up and Getting Started
@screamingfrog
@oliverbrett
@stephhugman

46. Download the software: it runs locally, not online
@screamingfrog
@oliverbrett
@stephhugman

47. "You need a
supercomputer to
run the Spider"
"You just need 4GB
of RAM and an
SSD"
@screamingfrog
@oliverbrett
@stephhugman
Download the software: it runs locally, not online

48. Default Screen
Navigating the Interface:
@screamingfrog
@oliverbrett
@stephhugman

49. Entering a Website
@screamingfrog
@oliverbrett
@stephhugman
Navigating the Interface:

50. Populating Data
Navigating the Interface:
@screamingfrog
@oliverbrett
@stephhugman
Navigating the Interface:

51. Crawl Progress
Navigating the Interface:
@screamingfrog
@oliverbrett
@stephhugman
Navigating the Interface:

52. Tabs
Navigating the Interface:
@screamingfrog
@oliverbrett
@stephhugman
Navigating the Interface:

53. Filtering by Feature
Navigating the Interface:
@screamingfrog
@oliverbrett
@stephhugman
Navigating the Interface:

54. Crawl Modes
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

55. Crawl Modes
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

56. Crawl Modes: Spider Mode vs List Mode
Spider Mode – whole site:
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

57. Crawl Modes: Spider Mode vs List Mode
Spider Mode – whole site:
List Mode – specific set of URLs:
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

58. Storage Options
Memory Storage Mode Database Storage Mode
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

59. Storage Options
Memory Storage Mode
Uses your RAM to store your
crawl data
Database Storage Mode
Uses your SSD to store your
crawl data
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

60. Storage Options
Memory Storage Mode
Uses your RAM to store your
crawl data
Database Storage Mode
Uses your SSD to store your
crawl data
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

61. Benefits of Database Storage Mode
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:
● Automatically save your crawls

62. Benefits of Database Storage Mode
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:
● Automatically save your crawls
● Recover previously lost crawls

63. Benefits of Database Storage Mode
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:
● Automatically save your crawls
● Recover previously lost crawls
● Open crawls significantly quicker

64. Benefits of Database Storage Mode
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:
● Automatically save your crawls
● Recover previously lost crawls
● Open crawls significantly quicker
● Crawl more URLs (millions)

65. ● Automatically save your crawls
● Recover previously lost crawls
● Open crawls significantly quicker
● Crawl more URLs (millions)
● Enable extra features including crawl comparison and change detection
Benefits of Database Storage Mode
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

66. @screamingfrog
@oliverbrett
@stephhugman

67. The Issues Tab
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

68. The Issues Tab
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

69. The Issues Tab
Issues = an error or issue that should ideally be fixed.
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

70. The Issues Tab
Issues = an error or issue that should ideally be fixed.
Opportunities = potential areas for optimisation and improvement.
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

71. The Issues Tab
Issues = an error or issue that should ideally be fixed.
Warnings = Not necessarily an issue, but worth checking and fixing
if needed.
Opportunities = potential areas for optimisation and improvement.
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

72. @screamingfrog
@oliverbrett
@stephhugman

73. 12 Advanced Uses
@screamingfrog
@oliverbrett
@stephhugman

74. Crawling Complex Sites
@screamingfrog
@oliverbrett
@stephhugman

75. Crawling Complex Sites
@screamingfrog
@oliverbrett
@stephhugman

76. The number one
question our support
team get asked:
@screamingfrog
@oliverbrett
@stephhugman

77. Why won’t this Logfile
Analyser crawl my
site?
@screamingfrog
@oliverbrett
@stephhugman

78. @screamingfrog
@oliverbrett
@stephhugman

79. @screamingfrog
@oliverbrett
@stephhugman

80. Why won’t this Logfile
Analyser crawl my
site?
@screamingfrog
@oliverbrett
@stephhugman

81. The number one
question our support
team get asked:
@screamingfrog
@oliverbrett
@stephhugman

82. Q: Why isn’t the
spider crawling more
than one page?
@screamingfrog
@oliverbrett
@stephhugman

83. ???
@screamingfrog
@oliverbrett
@stephhugman

84. ???
@screamingfrog
@oliverbrett
@stephhugman

85. ???
@screamingfrog
@oliverbrett
@stephhugman

86. ???
@screamingfrog
@oliverbrett
@stephhugman

87. A: JAVASCRIPT!!
@screamingfrog
@oliverbrett
@stephhugman

88. Tricky site 1:
JavaScript websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/CRAWL-
JAVASCRIPT-SEO/
@screamingfrog
@oliverbrett
@stephhugman

89. @screamingfrog
@oliverbrett
@stephhugman

90. @screamingfrog
@oliverbrett
@stephhugman

91. @screamingfrog
@oliverbrett
@stephhugman

92. @screamingfrog
@oliverbrett
@stephhugman

93. @screamingfrog
@oliverbrett
@stephhugman

94. Tricky site 2:
big websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/HOW-TO-CRAWL-
LARGE-WEBSITES/
@screamingfrog
@oliverbrett
@stephhugman

95. What do we mean by
big?
@screamingfrog
@oliverbrett
@stephhugman

96. @screamingfrog
@oliverbrett
@stephhugman

97. (Over) Five Million URLs
@screamingfrog
@oliverbrett
@stephhugman

98. Do you really need to
crawl the whole
damn site?
@screamingfrog
@oliverbrett
@stephhugman

99. That’s gonna be a big
Excel doc…
@screamingfrog
@oliverbrett
@stephhugman

100. That’s gonna be a big
Excel doc…
@screamingfrog
@oliverbrett
@stephhugman

101. Crawl in sections,
subdomains, or
subfolders.
@screamingfrog
@oliverbrett
@stephhugman

102. Use the exclude
function to avoid
faceted navigation.
@screamingfrog
@oliverbrett
@stephhugman

103. https://www.johnlew
is.com/browse/men/
mens-
trousers/adidas/alls
aints/gant-
rugger/hymn/kin/sel
ected-
femme/homme/size
=36r/_/N-
ebiZ1z13yvxZ1z0g0g6
Z1z04nruZ1z0s0laZ1z
0vl67Z1z01kl3Z1z0sw
k1 @screamingfrog
@oliverbrett
@stephhugman

104. https://www.johnlew
is.com/browse/men/
mens-
trousers/adidas/alls
aints/gant-
rugger/hymn/kin/sel
ected-
femme/homme/size
=36r/_/N-
ebiZ1z13yvxZ1z0g0g6
Z1z04nruZ1z0s0laZ1z
0vl67Z1z01kl3Z1z0sw
k1 @screamingfrog
@oliverbrett
@stephhugman

105. Database Storage
Mode
CONFIGURATION > STORAGE > DATABASE STORAGE MODE
@screamingfrog
@oliverbrett
@stephhugman

106. @screamingfrog
@oliverbrett
@stephhugman

107. @screamingfrog
@oliverbrett
@stephhugman

108. 8GB RAM
@screamingfrog
@oliverbrett
@stephhugman

109. 8GB RAM
@screamingfrog
@oliverbrett
@stephhugman

110. SSD
@screamingfrog
@oliverbrett
@stephhugman

111. SSD
@screamingfrog
@oliverbrett
@stephhugman

112. SSD
@screamingfrog
@oliverbrett
@stephhugman

113. Tricky site 3:
password protected
websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/CRAWLING-
PASSWORD-PROTECTED-WEBSITES/
@screamingfrog
@oliverbrett
@stephhugman

114. Disclaimer:
@screamingfrog
@oliverbrett
@stephhugman

115. Disclaimer:
You need to know what
the password is.
@screamingfrog
@oliverbrett
@stephhugman

116. Disclaimer:
You need to know what
the password is.
We can’t help you hack it…
@screamingfrog
@oliverbrett
@stephhugman

117. (Maybe like a dev site
for example…)
@screamingfrog
@oliverbrett
@stephhugman

118. Forms Based
Authentication
CONFIGURE > AUTHENTICATION
@screamingfrog
@oliverbrett
@stephhugman

119. @screamingfrog
@oliverbrett
@stephhugman

120. Disclaimer 2:
@screamingfrog
@oliverbrett
@stephhugman

121. Disclaimer
2:
This will
click every
link on the
page.
@screamingfrog
@oliverbrett
@stephhugman

122. Disclaimer 2:
This will click every
link on the page.
Yes, really… @screamingfrog
@oliverbrett
@stephhugman

123. Installed 108
WordPress
Themes
@screamingfrog
@oliverbrett
@stephhugman

124. Installed 108
WordPress
Themes
Activated 8
themes
@screamingfrog
@oliverbrett
@stephhugman

125. Installed 108
WordPress
Themes
Activated 8
themes
Deleted posts
@screamingfrog
@oliverbrett
@stephhugman

126. Installed 108
WordPress
Themes
Activated 8
themes
Deleted posts
Logged itself off
@screamingfrog
@oliverbrett
@stephhugman

127. Installed 108
WordPress
Themes
Activated 8
themes
Deleted posts
Logged itself off
@screamingfrog
@oliverbrett
@stephhugman

128. Installed 108
WordPress
Themes
Activated 8
themes
Deleted posts
Logged itself off
@screamingfrog
@oliverbrett
@stephhugman

129. @screamingfrog
@oliverbrett
@stephhugman

130. @screamingfrog
@oliverbrett
@stephhugman

131. Limit the admin
rights of the
account
@screamingfrog
@oliverbrett
@stephhugman

132. Limit the admin
rights of the
account
Blacklist URLs
using the
exclude feature @screamingfrog
@oliverbrett
@stephhugman

133. Limit the admin
rights of the
account
Blacklist URLs
using the
exclude feature
Sense Check the
site first…
@screamingfrog
@oliverbrett
@stephhugman

134. Limit the admin
rights of the
account
Blacklist URLs
using the
exclude feature
Unplug your fax
machine
Sense Check the
site first…
@screamingfrog
@oliverbrett
@stephhugman

135. API Integration
@screamingfrog
@oliverbrett
@stephhugman

136. API Integration
@screamingfrog
@oliverbrett
@stephhugman

137. Google
Analytics
HTTPS://WWW.SCREAMINGFROG.CO.UK
/SEO-SPIDER/U SER-
GUIDE/CONFIGURATION/#GOOGLE -
ANALYTICS-INTEGRATION
@screamingfrog
@oliverbrett
@stephhugman

138. Google
Analytics
HTTPS://WWW.SCREAMINGFROG.CO.UK
/SEO-SPIDER/U SER-
GUIDE/CONFIGURATION/#GOOGLE -
ANALYTICS-INTEGRATION
@screamingfrog
@oliverbrett
@stephhugman

139. Google
Analytics
HTTPS://WWW.SCREAMINGFROG.CO.UK
/SEO-SPIDER/U SER-
GUIDE/CONFIGURATION/#GOOGLE -
ANALYTICS-INTEGRATION
@screamingfrog
@oliverbrett
@stephhugman

140. Google
Analytics
HTTPS://WWW.SCREAMINGFROG.CO.UK
/SEO-SPIDER/U SER-
GUIDE/CONFIGURATION/#GOOGLE -
ANALYTICS-INTEGRATION

141. Google Analytics (4)
Sessions
% New Sessions
New Users
Bounce Rate/Engagement Rate
Page Views Per Session
Avg Session Duration
Page Value
Goal Conversion Rate
Goal Completions All
Goal Value All
Sessions Above 0
Bounce Rate Above 70%
No GA Data Pages
Non-Indexable with GA Data
Orphan URLs
@screamingfrog
@oliverbrett
@stephhugman

142. Google Search
Console
HTTPS://WWW.SCREAMINGFROG.C
O.UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#GOOGLE -
SEARCH-CONSOLE-INTEGRATION
@screamingfrog
@oliverbrett
@stephhugman

143. Google Search Console
Clicks
Impressions
CTR
Position
Clicks above 0
No GSC Data
Non-Indexible with GSC Data
Orphan URLs
@screamingfrog
@oliverbrett
@stephhugman

144. ahrefs
H T T P S : / / W W W . S C R E A M I N G F R O G
. C O . U K / S E O - S P I D E R / U S E R -
G U I D E / C O N F I G U R A T I ON / # A H R E F
S
@screamingfrog
@oliverbrett
@stephhugman

145. Majestic
H T T P S : / / W W W . S C R E A M I N G F R O G
. C O . U K / S E O - S P I D E R / U S E R -
G U I D E / C O N F I G U R A T I ON / # M A J E S
T I C
@screamingfrog
@oliverbrett
@stephhugman

146. Moz
HTTPS://WWW.SCREAMINGFRO
G.CO.UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#MOZ
@screamingfrog
@oliverbrett
@stephhugman

147. Get Ya DAs
Done
1. Put the spider into list
mode
2. Connect Moz API, enable
DA
3. Paste in list of coverage
URLs
4. Crawl
5. Export list of URLs + DAs
6. Make link builders happy
@screamingfrog
@oliverbrett
@stephhugman

148. PageSpeed
Insights
HTTPS://WWW.SCREAMINGFROG.C
O.UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#PAGESPE
ED-INSIGHTS-INTEGRATION
@screamingfrog
@oliverbrett
@stephhugman

149. @screamingfrog
@oliverbrett
@stephhugman

150. Custom Search
@screamingfrog
@oliverbrett
@stephhugman

151. Custom Search
@screamingfrog
@oliverbrett
@stephhugman

152. @screamingfrog
@oliverbrett
@stephhugman

153. @screamingfrog
@oliverbrett
@stephhugman

154. @screamingfrog
@oliverbrett
@stephhugman
How many of our
products are out of
stock?

155. @screamingfrog
@oliverbrett
@stephhugman
How many of our
products are out of
stock?
Uhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhh
hhhhhhhhhhhh

156. @screamingfrog
@oliverbrett
@stephhugman
Which pages are
missing in GA4?

157. @screamingfrog
@oliverbrett
@stephhugman
Which pages are
missing in GA4?
GA444444444444444
44444444444444444
44444444444444444
4444444444444444?

158. Check the HTML for:
Contains: Text- ‘Out of Stock’
Does Not Contain: Text- [Google Tag Manager Tracking Code]
@screamingfrog
@oliverbrett
@stephhugman

159. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV
@screamingfrog
@oliverbrett
@stephhugman

160. @screamingfrog
@oliverbrett
@stephhugman

161. @screamingfrog
@oliverbrett
@stephhugman

162. @screamingfrog
@oliverbrett
@stephhugman

163. @screamingfrog
@oliverbrett
@stephhugman

164. Custom Extraction
@screamingfrog
@oliverbrett
@stephhugman

165. Custom Extraction
https://ww
w.screami
ngfrog.co.
uk/web-
scraping/ @screamingfrog
@oliverbrett
@stephhugman

166. Custom
Search
Custom
Extraction
@screamingfrog
@oliverbrett
@stephhugman

167. Xpath/Regex
@screamingfrog
@oliverbrett
@stephhugman

168. @screamingfrog
@oliverbrett
@stephhugman

169. @screamingfrog
@oliverbrett
@stephhugman

170. @screamingfrog
@oliverbrett
@stephhugman

171. @screamingfrog
@oliverbrett
@stephhugman

175. Visualising Data
@screamingfrog
@oliverbrett
@stephhugman

176. Visualising Data
@screamingfrog
@oliverbrett
@stephhugman

177. HTTPS://WWW.SCREAMINGFROG.CO.UK/SEO-SPIDER-10/
HTTPS://WWW.SCREAMINGFROG.CO.UK/SITE -ARCHITECTURE-CRAWL-
VISUALISATIONS/
@screamingfrog
@oliverbrett
@stephhugman

178. Crawl Tree Graph
SHORTEST PATH TO THE PAGE THAT THE SPIDER TOOK
@screamingfrog
@oliverbrett
@stephhugman

179. @screamingfrog
@oliverbrett
@stephhugman

180. Directory Tree Graph
FOLLOWS URL STRUCTURE
@screamingfrog
@oliverbrett
@stephhugman

181. @screamingfrog
@oliverbrett
@stephhugman

182. Force-Directed Crawl
Diagram
SHORTEST PATH TO THE PAGE THAT THE SPIDER TOOK
@screamingfrog
@oliverbrett
@stephhugman

183. @screamingfrog
@oliverbrett
@stephhugman

184. Force-Directed
Directory Tree
Diagram
FOLLOWS URL STRUCTURE
@screamingfrog
@oliverbrett
@stephhugman

185. @screamingfrog
@oliverbrett
@stephhugman

186. Inlink Anchor Text
Word Cloud
WHAT TEXT ARE YOU LINKING WITH?
@screamingfrog
@oliverbrett
@stephhugman

187. @screamingfrog
@oliverbrett
@stephhugman

188. Body Text Word
Cloud
WHAT IS YOUR CONTENT ABOUT?
@screamingfrog
@oliverbrett
@stephhugman

189. @screamingfrog
@oliverbrett
@stephhugman

190. Are these pointless?
@screamingfrog
@oliverbrett
@stephhugman

191. Yeah…
@screamingfrog
@oliverbrett
@stephhugman

192. @screamingfrog
@oliverbrett
@stephhugman

193. Can we make more of
em?
@screamingfrog
@oliverbrett
@stephhugman

194. Also yes!
@screamingfrog
@oliverbrett
@stephhugman

195. 3D Force-Directed
Crawl Diagram
SHORTEST PATH TO THE PAGE THAT THE SPIDER TOOK,
IN ✨3D✨
@screamingfrog
@oliverbrett
@stephhugman

196. @screamingfrog
@oliverbrett
@stephhugman

197. @screamingfrog
@oliverbrett
@stephhugman

198. 3D Force-Directed
Directory Tree
Diagram
FOLLOWS URL STRUCTURE, IN ✨3D✨
@screamingfrog
@oliverbrett
@stephhugman

199. @screamingfrog
@oliverbrett
@stephhugman

200. @screamingfrog
@oliverbrett
@stephhugman

201. @screamingfrog
@oliverbrett
@stephhugman

202. @screamingfrog
@oliverbrett
@stephhugman

203. @screamingfrog
@oliverbrett
@stephhugman

204. Segments
@screamingfrog
@oliverbrett
@stephhugman

205. Segments
@screamingfrog
@oliverbrett
@stephhugman

208. ● X
X
@screamingfrog
@oliverbrett
@stephhugman

209. ● X
X
@screamingfrog
@oliverbrett
@stephhugman

210. X
@screamingfrog
@oliverbrett
@stephhugman

211. @screamingfrog
@oliverbrett
@stephhugman

213. Auditing Redirects in a Site
Migration
@screamingfrog
@oliverbrett
@stephhugman

214. Common Issues in a Site Migration

215. Auditing Redirects in a Site
Migration
@screamingfrog
@oliverbrett
@stephhugman

216. Common Issues with Redirects:
@screamingfrog
@oliverbrett
@stephhugman

217. Common Issues with Redirects:
Multiple hops in a redirect chain
Start
Finish
@screamingfrog
@oliverbrett
@stephhugman

218. Start
End
point
Common Issues with Redirects:
Redirecting to the wrong target
Target

219. Start
Finish
Common Issues with Redirects:
Redirecting to a 404 page

220. Upload your Original URLs in List Mode
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

221. Configure the Crawl to Follow Redirects
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

222. Export a Report of all the Redirects
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

223. The report includes
every URL in the original
upload and the response codes in a
single report,
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

224. recording the start and final URLs
and statuses,
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

225. as well as the full hops if there are
several redirects in
a chain.
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

226. You can easily see if any URLs are
missing when they shouldn’t be,
or weren't redirected correctly.
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

227. Sitemaps
@screamingfrog
@oliverbrett
@stephhugman

228. Sitemaps
Helping Google better understand your website
@screamingfrog
@oliverbrett
@stephhugman

229. Sitemaps
Helping Google better understand your website
@screamingfrog
@oliverbrett
@stephhugman

230. Sitemaps
Helping Google better understand your website
@screamingfrog
@oliverbrett
@stephhugman

231. Generating and Auditing
Sitemaps
@screamingfrog
@oliverbrett
@stephhugman

232. Generating and Auditing
Sitemaps
@screamingfrog
@oliverbrett
@stephhugman

233. Crawl a Site,
then Click "Sitemaps" in the Main Nav
@screamingfrog
@oliverbrett
@stephhugman
How to
create
sitemaps:

234. @screamingfrog
@oliverbrett
@stephhugman
How to
create
sitemaps:
● All pages with a 200 response included
Crawl a Site,
then Click "Sitemaps" in the Main Nav

235. ● All pages with a 200 response included
● Can include images, PDFs, and canonicalised/paginated/noindex URLs
Crawl a Site,
then Click "Sitemaps" in the Main Nav
@screamingfrog
@oliverbrett
@stephhugman
How to
create
sitemaps:

236. Generating and Auditing
Sitemaps
@screamingfrog
@oliverbrett
@stephhugman

237. Generating and Auditing
Sitemaps
@screamingfrog
@oliverbrett
@stephhugman

239. BONUS:
Orphan
pages!

240. Configure the Spider
to Crawl Sitemaps Pre-crawl
@screamingfrog
@oliverbrett
@stephhugman
How to
audit
sitemaps:

241. Enable Post-crawl Analysis
to see Sitemap Data Reports
Reports:
● URLs in sitemap
● URLs not in sitemap
● Orphan URLs
● Non-indexable URLs in sitemap
● URLs in multiple sitemaps
● XML sitemap with over 50k URLs
● XML sitemap with over 50mb
How to
audit
sitemaps:

242. Auditing Content (with Data)
@screamingfrog
@oliverbrett
@stephhugman

243. Auditing Content (with Data)
@screamingfrog
@oliverbrett
@stephhugman

244. How to Audit Content with Spider Data
@screamingfrog
@oliverbrett
@stephhugman

245. Content-related Data Reports
Word count and average words per sentence
@screamingfrog
@oliverbrett
@stephhugman

246. Content-related Data Reports
Spelling and grammar errors
@screamingfrog
@oliverbrett
@stephhugman

247. Content-related Data Reports
Readability levels
@screamingfrog
@oliverbrett
@stephhugman

248. Content-related Data Reports
Lorem ipsum placeholder text
@screamingfrog
@oliverbrett
@stephhugman

249. Content-related Data Reports
Exact/near duplicates
@screamingfrog
@oliverbrett
@stephhugman

250. Content-related Data Reports
Low content pages
@screamingfrog
@oliverbrett
@stephhugman

251. Auditing Structured Data
@screamingfrog
@oliverbrett
@stephhugman

252. Auditing Structured Data
@screamingfrog
@oliverbrett
@stephhugman

253. Structured Data
Helping Google better understand your pages and their content,
enabling rich results in the SERPs
@screamingfrog
@oliverbrett
@stephhugman

254. Structured Data
Helping Google better understand your pages and their content,
enabling rich results in the SERPs
@screamingfrog
@oliverbrett
@stephhugman

255. Structured Data
Helping Google better understand your pages and their content,
enabling rich results in the SERPs
Review
stars
Cooking
time
@screamingfrog
@oliverbrett
@stephhugman

256. Structured Data
Helping Google better understand your pages and their content,
enabling rich results in the SERPs
Review
stars
Cooking
time
@screamingfrog
@oliverbrett
@stephhugman

257. Configure the Spider
to Enable Structured Data Options
@screamingfrog
@oliverbrett
@stephhugman
How to audit
structured data:

258. Export Relevant Structured Data Reports
● Contains structured data
● Missing
● Validation errors/ warnings
● Parse errors
● Microdata/JSON-LD/RDFa
URLs
@screamingfrog
@oliverbrett
@stephhugman
How to audit
structured data:

259. @screamingfrog
@oliverbrett
@stephhugman

260. Scheduling Crawls
@screamingfrog
@oliverbrett
@stephhugman

261. Scheduling Crawls
@screamingfrog
@oliverbrett
@stephhugman

263. Choose the Regularity, URLs, and Give it a Name
How to Schedule Crawls:
@screamingfrog
@oliverbrett
@stephhugman
How to schedule crawls:

264. @screamingfrog
@oliverbrett
@stephhugman

265. Crawling in the Cloud
@screamingfrog
@oliverbrett
@stephhugman

266. Crawling in the Cloud
@screamingfrog
@oliverbrett
@stephhugman

267. Benefits of Running the Spider in the Cloud
@screamingfrog
@oliverbrett
@stephhugman
● Can run crawls without your computer
even being switched on.

268. Benefits of Running the Spider in the Cloud
@screamingfrog
@oliverbrett
@stephhugman
● Can run crawls without your computer
even being switched on.
● Frees up your resources (RAM) to do
other things.

269. Benefits of Running the Spider in the Cloud
@screamingfrog
@oliverbrett
@stephhugman
● Can run crawls without your computer
even being switched on.
● Frees up your resources (RAM) to do
other things.
● Can run several crawls at once in the
cloud on just the one license.

270. Benefits of Running the Spider in the Cloud
● Can run crawls without your computer
even being switched on.
● Frees up your resources (RAM) to do
other things.
● Can run several crawls at once in the
cloud on just the one license.
● Easily access your crawls and data
anywhere, anytime.
@screamingfrog
@oliverbrett
@stephhugman

271. Summary
@screamingfrog
@oliverbrett
@stephhugman

272. In summary...
● 1. Crawling complex sites
● 2. Integrating with APIs
● 3. Custom Search
● 4. Custom Extraction
● 5. Visualising Data
● 6. Making Segments
● 7. Auditing Migration Redirects
● 8. Generating/Auditing Sitemaps
● 9. Auditing Content with Data
● 10. Auditing Structured Data
● 11. Scheduling Crawls
● 12. Crawling in the Cloud
@screamingfrog
@oliverbrett
@stephhugman

273. Resources
● support@screamingfrog.co.uk
● @screamingfrog on Twitter/X
● @oliverbrett
● @stephhugman
● https://www.screamingfrog.co.uk/seo-spider/user-guide/
● https://www.screamingfrog.co.uk/seo-spider/tutorials/
● https://www.screamingfrog.co.uk/seo-spider/faq/

274. Thank you!
https://www.screamingfrog.co.uk/
info@screamingfrog.co.uk
@screamingfrog
@oliverbrett
@stephhugman