Practical guide to using Screaming Frog custom extractions to find new SEO opportunities and automate data collection tasks.

1. E R U D I T E

2. ERUDITE
Screaming Frog
Custom Extractions
J E S S I C A J A M E S

3. 1. What is Screaming Frog? And what’s
included in a standard crawl?
2. What are Custom Extractions and why are
they useful?
3. What is Custom Search and why is it
useful?
4. How it works
5. Other ideas you could try
6. Using different types of extractor
@Jessica_James01

4. 1. What is
Screaming Frog?
@Jessica_James01

5. Website crawler that
gathers data from either
URLs it discovers based
on your chosen criteria
or a predefined list of
URLs.
@Jessica_James01

6. Page Titles
Meta Descriptions
Meta Keywords
H1
H2
Indexability status
Word Count
Page Size
Structured Data
Response Time
Meta Robots
@Jessica_James01

7. 2. What is a
Custom Extraction?
@Jessica_James01

8. But what happens if you
want to gather data that’s
specific to a particular
site?
@Jessica_James01

9. You can use a custom
extraction to gather
information with custom
selectors using REGEX, CSS
Selectors or XPaths.
@Jessica_James01

10. 3. What is Custom
Search?
@Jessica_James01

11. Custom search is similar
to custom extractions.
Custom search finds a
specific string in the
HTML rather than
extracting data based on
an identifier.
@Jessica_James01

12. Tracking IDs and Pixels
(GA, GTM, Hotjar,
Facebook)
Text Strings (Out of
Stock, Properties with
Hot Tubs)
@Jessica_James01

13. 4. For Example. .
.
@Jessica_James01

14. Say we want to start
writing a blog about home
improvements.
We want to do some
competitor research to
gather content ideas.
@Jessica_James01

15. Collecting Pricing
Data from Rated
People
@Jessica_James01

16. Rated People
@Jessica_James01

17. Rated People
@Jessica_James01

18. Rated People
@Jessica_James01

19. Open Screaming
Frog and go
to:
Configuration
> Custom >
Extraction
Rated People
@Jessica_James01

20. Rated People
@Jessica_James01

21. Before you crawl:
Robots.txt
Include/Exclude
User Agent
Spider Configuration
@Jessica_James01

22. Rated People
@Jessica_James01

23. Collecting Q&A
Content Ideas from
Homebase
@Jessica_James01

24. @Jessica_James01

25. @Jessica_James01

26. @Jessica_James01

27. Collecting
Questions & Answers
+ Number of Replies
from MyBuilder
@Jessica_James01

28. H1 = Question
Number of Replies
Answer
@Jessica_James01

29. @Jessica_James01

30. @Jessica_James01

31. 5. What Else Could
You Use it For?
@Jessica_James01

32. 1. Identifying product pages without
reviews
2. Extract all the body copy on a page
(<p> tags)
3. Pull author names from
articles/identify blog posts without
authors
4. Find GA/GTM Tags/check for multiples
5. Extract and verify Hreflang tags
6. Extract headings below H2
7. Mark up (OL UL etc.)
@Jessica_James01

33. 6. Other Extractor
Types
@Jessica_James01

34. XPaths can be extracted using
the ‘Inspect’ feature in
Chrome.
This works well when the CSS
Selector isn’t unique to the
element you need to extract.
@Jessica_James01

35. XPaths are ideal for
scraping, and there are loads
of different XPath
expressions you can use for
more sophisticated
extractions.
@Jessica_James01

36. Syntax Function
// Searches anywhere
/ Only searches within the root
@ Selects a specific attribute
* Wildcard
[] Finds a specific element
. Selects current element
.. Selects parent element
@Jessica_James01

37. //*[@itemprop='availability']
/@content
=
“Find all elements with the
itemprop ‘availability’ and
return the content.”
@Jessica_James01

38. Returns the
availabilit
y status
for each
product
page.
@Jessica_James01

39. REGEX is particularly useful
when you want to extract
something that isn’t rendered
on the page, or to extract
non-HTML data.
JSON-LD Structured Data, Tags
and Pixel IDs…
@Jessica_James01

40. ["'](GTM-.*?)["']
=
Extract the GTM ID
@Jessica_James01

41. ["']@type["']: *["'](.*?)["']
=
Extract the types of JSON-LD
structured data on each page.
@Jessica_James01

42. ERUDITE
jess.james@erudite.agency
+44 (0) 1256 384890
https://erudite.agency
@Jessica_James01
J E S S I C A J A M E S
Ready to
talk to us?