The most popular topics SF get asked about when helping our customers with their questions. Also a nice overview of the SF Spider.

1. Lessons Learned
from 11,000
Spider Support Tickets
OLIVER BRETT, HEAD OF STRATEGY, SCREAMING FROG

8. Lots of Languages =
Lots of Languages
Two Languages =
Bilingual
One Language =
English

9. Lots of Languages =
Polylingual
Two Languages =
Bilingual
One Language =
English

10. Lots of Languages =
Polylingual
Two Languages =
Bilingual
One Language =
English

11. Lots of Languages =
Polylingual
Two Languages =
Bilingual
One Language =
English

12. Lots of Languages =
Polylingual
Two Languages =
Bilingual
One Language =
English

13. Lots of Languages =
Polylingual
Two Languages =
Bilingual
One Language =
English

15. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV

16. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV

17. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV

18. 3 Rules For Today:

19. 1. This isn’t a sales
pitch.

26. 2. This isn’t a
walkthrough.

28. https://www.screamingfrog.co.uk/seo-spider/tutorials/

29. https://www.screamingfrog.co.uk/seo-spider/tutorials/

36. 3. Just ask if you
need help!

38. 1.Was ist dein
Lieblingsfacht?
2.What is
Screaming Frog?
3.Spider
Support
4.What People
Ask Us

39. Part One:What isScreaming Frog?

42. Screaming Frog
Log File Analyser

44. Screaming Frog
Log File Analyser
Screaming Frog
SEO Spider

47. Why is it called
‘Screaming Frog’?

49. Screaming Frog is
built by SEOs, for
SEOs.

58. Screaming Frog
(usually) runs locally.
But that’s a good thing.

63. “The Swiss
Army Knife
of SEO”
(IT’S CUTTING EDGE…)

64. https://www.seerinteractive.com/blog/screaming-frog-guide/

65. https://www.seerinteractive.com/blog/screaming-frog-guide/

66. https://www.seerinteractive.com/blog/screaming-frog-guide/

67. https://www.seerinteractive.com/blog/screaming-frog-guide/

70. Screaming
Frog Can…

71. Timeline:
V.1(2011)-V.16(2022)

72. V1.0 (2011)
LAU N CH

73. V2.0 (2012)
Word Counts
URL Rewrites
Crawl Control

74. Nobigupdates
(2013/2014)

75. V3.0 (2015)
Tree View
Insecure Content Audits
Image Sitemaps + Enhanced XML Sitemaps

76. V4.0 (2015)
Google Analytics Integration
Custom Extraction

77. V5.0 (2015)
Webmaster Tools (Google
Search Console) integration
Better Robots.txt
Auditing/Control

78. http://taylortomita.com/seo-graph-portfolio/

79. V6.0 (2016)
JavaScript Rendering
Rendered Crawling
Crawl Indexed Sitemaps
SERP Emulator

80. V7.0 (2016)
Fetch and Render
Custom Robots.txt Rules
Hreflang
Web Form Authentication

81. V8.0 (2017)
Made it look nice(r)
More APIs
Custom Profiles

83. V9.0 (2018)
Database Storage
Store HTML and Rendered HTML
Better Search Functions

84. V10.0 (2018)
Scheduling
Visualisations
CLI (Headless Mode)

85. V11.0 (2019)
Structured Data- Aud it,
Valid ate and Expor t

86. V12.0 (2019)
Page Speed Insights API
Configurable Tabs
Custom Search
Improvements

89. http://taylortomita.com/seo-graph-portfolio/

90. V13.0 (2020)
N ear Duplicate Content
Spelling & Grammar

91. V14.0 (2020)
Dark M ode
Google Sheets Expor t
Cookies

92. V15.0 (2021)
Crawl Comparison
Change Detection

93. http://taylortomita.com/seo-graph-portfolio/

94. V16.0 (2021)
Improved JavaScript
Auto Crawl Repor ts for
Data Stud io
Translated U I

95. V16.0 (2021)
Improved JavaScript
Auto Crawl Repor ts for
Data Stud io
Translated U I

96. Lots of Updates…
V1.0 -2011- Launch
V2.0-2012- Word Counts, URL Rewrites, Crawl
Control
V3.0- 2015- Tree View, Insecure Content, Image
Sitemaps
V4.0- 2015- GA Integration, Custom Extraction
V5.0- 2015- GSC Integration, Better Robots.txt
V6.0- 2016- Rendered Crawling, Crawl Indexed
Sitemaps, SERP Emulator
V7.0- 2016- Fetch and Render, Custom Robots.txt,
hreflang, Web Form Authentication
V8.0- 2017- Made it look nice(r), More APIs,
Custom Profiles
V9.0- 2018- Database Storage, Store and Render
HTML, Better Search
V10.0- 2018- Scheduling, Visualisations,
Headless Mode
V11.0- 2019- Structured Data- Audit, Validate and
Export

97. Lots of Updates…
V1.0 -2011- Launch
V2.0-2012- Word Counts, URL Rewrites, Crawl
Control
V3.0- 2015- Tree View, Insecure Content, Image
Sitemaps
V4.0- 2015- GA Integration, Custom Extraction
V5.0- 2015- GSC Integration, Better Robots.txt
V6.0- 2016- Rendered Crawling, Crawl Indexed
Sitemaps, SERP Emulator
V7.0- 2016- Fetch and Render, Custom Robots.txt,
hreflang, Web Form Authentication
V8.0- 2017- Made it look nice(r), More APIs,
Custom Profiles
V9.0- 2018- Database Storage, Store and Render
HTML, Better Search
V10.0- 2018- Scheduling, Visualisations,
Headless Mode
V11.0- 2019- Structured Data- Audit, Validate and
Export

98. Lots of Updates…
V1.0 -2011- Launch
V2.0-2012- Word Counts, URL Rewrites, Crawl
Control
V3.0- 2015- Tree View, Insecure Content, Image
Sitemaps
V4.0- 2015- GA Integration, Custom Extraction
V5.0- 2015- GSC Integration, Better Robots.txt
V6.0- 2016- Rendered Crawling, Crawl Indexed
Sitemaps, SERP Emulator
V7.0- 2016- Fetch and Render, Custom Robots.txt,
hreflang, Web Form Authentication
V8.0- 2017- Made it look nice(r), More APIs,
Custom Profiles
V9.0- 2018- Database Storage, Store and Render
HTML, Better Search
V10.0- 2018- Scheduling, Visualisations,
Headless Mode
V11.0- 2019- Structured Data- Audit, Validate and
Export

99. Part Two: Spider Support

101. Slide about Ewa on the phone

102. Slide about Ewa on the phone
Ewa

105. Blah blah javascript blah blah
html?

106. Blah blah javascript blah blah
html?
What?

108. Spider Support

109. Spider Support
0.25€

110. 48% of queries
replied to within
1 hour

113. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV

114. 97% within
24 hours

117. Slide about Ewa on the phone

118. 0
5000
10000
15000
20000
25000
30000
35000
2015/16 2016/17 2017/18 2018/19 2019/20 2020/21 2021/22
Messages Received

119. Lessons Learned
from 11,000
Spider Support Tickets
OLIVER BRETT, HEAD OF STRATEGY, SCREAMING FROG

120. 1.6
1.65
1.7
1.75
1.8
1.85
1.9
1.95
2015/16 2016/17 2017/18 2018/19 2019/20 2020/21 2021/22
Replies to Resolve

121. November is our busiest
month

122. Screaming Frog™
Log File
Analyser™!!!11!!

123. Screaming Frog
Log File Analyser
Screaming Frog
SEO Spider

125. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV

126. Part Three: People Always Ask

127. Many spider support
questions
are secretly
SEO questions

130. People who don’t
People who like pie chats

131. People who
speak German
Dieser Junge auf der Bühne.

132. Useful Information

133. I looked at 108
common topics
we get asked
about

134. All 108 Topics

135. 108 topics
ranked by
no. of
conversations

136. 15% licence

137. 15% licence
11% license

140. 15% licence
11% license
6% crawling

141. Operating
Systems

148. 25% Mac
13% Ubuntu
7% Linux
56% Windows

154. Search Engines

156. 95% Google
2% Yandex
2% Yahoo
1% Bing

158. Functions

159. 17% Memory
12% Storage
9% Install
6% Opening
38% Crawling

161. Adjusting
Your RAM
Increasing allocated
memory

162. Changing
Storage
Choosing the right mode

163. MemoryModevs
Database Storage Mode
Memory Mode:
Crawl is saved to RAM as you go
Works on older PCs
More likely to explode
If your PC shuts down… good luck
Beyond several 1,000’s you might
struggle…
Database Storage Mode:
Crawl is saved right to your SSD/HDD
You need a faster SSD ideally
Can save the crawl if your computer goes
to sleep/crashes
Can crawl millions of URLs

164. Modes

165. 58% Spider Mode
32% List Mode
9% Command Line

166. 58% Spider Mode
32% List Mode
9% Command Line 1% SERP Mode 

169. Technical Terms

170. 10% User Agent
9% Redirect
14% JavaScript
7% Speed
8% Security

173. 10% User Agent
9% Redirect
14% JavaScript
7% Speed
8% Security
2% hreflang

174. 10% User Agent
9% Redirect
14% JavaScript
0.1% AMP
7% Speed
8% Security
2% hreflang

175. 10% User Agent
9% Redirect
14% JavaScript
7% Speed
8% Security
2% hreflang

176. 10% User Agent
9% Redirect
14% JavaScript
0.1% AMP
7% Speed
8% Security
2% hreflang

177. https://www.mediavine.com/amp-is-dead-miley-cyrus-was-only-loosely-involved/

178. Status Codes

181. 404

182. 200

183. 301

184. 302

185. 500

187. 30% 200s
22% 404s
14% 301s
11% 302s
10% 403s
4% 503s
4% 410s
3% 500s 2% 502s

188. 200

189. 30% 200s
22% 404s
14% 301s
11% 302s
10% 403s
4% 503s
4% 410s
3% 500s 2% 502s
36% 4XX
30% 200
25% 3XX
9% 5XX

190. 404

191. Jobs

192. 19% Custom Extraction
17%
Custom
Search
13% Large Site
2% Migration

194. Yes, there’s a
guide for that!

196. Yes, there’s a
guide for that!

198. Yes, there’s a
guide for that!

200. Content

203. 46% Images
17% Page Title
12% Meta
Descriptions
9% Alt Text

204. 46% Images
17% Page Title
12% Meta
Descriptions
9% Alt Text
4% Meta Keywords

209. <Heading Tags>

210. H1
H2
H3
H4

211. 60% H1
27% H2
13% H3

213. APIs

214. 48% Google
Analytics
24% Search Console
11% ahrefs
10%
PageSpeed
Insights
8% Moz

215. Review
PageSpeed
View stats and
opportunities to improve

217. We scraped all the
headlines we could…

219. (And you can use the Moz
API to bulk-check the
DAs of coverage.)

220. Other SEO Tools

221. 28% Moz
17% SEMRush
6% DeepCrawl
4% Sistrix
4% Sitebulb
38% ahrefs
3% OnCrawl 1% Ryte

223. Cheeky Asks

224. 71% Discount
26% Stickers
2% T-shirt 1% Hoodie

227. Slide about Ewa on the phone
Ewa

228. Slide about Ewa on the phone

231. The Fan Club

232. 6% Chocolate
45% Beer
27% Wine
4% Whiskey
2% Gin
1% Pimms

233. 6% Chocolate
45% Beer
27% Wine
4% Whiskey
2% Gin
1% Pimms
15% Egg

234. http://taylortomita.com/seo-graph-portfolio/

235. @OliverBrett
@ScreamingFrog Danke!