Screaming Frog: Little-Known Features In The SEO Spider

Screaming Frog: Little-
Known Features In The SEO
Spider

1. Crawls Comparison
● Comparison helps you monitor the progress of SEO
bugs and opportunities and provides useful data about
what has changed between crawls

1.1 Crawls Comparison
● To compare crawls, you need to be in database storage
mode and have a licence. You can switch to database
storage by selecting ‘Config > System > Storage Mode’
and ‘Database Storage’.

1.3 Adjust Comparison Config For Change Detection
Компания
Введите свой текст
здесь Введите свой
текст здесь Введите
свой текст здесь
здесь.
здесь
Контекст
здесь
● Введите свой
текст здесь
Задача
свой текст здесь.

1.4 URL Mapping
● You’re able to compare two different URL
structures using the ‘URL Mapping’ feature
introduced in crawl comparison
* Learn more about Regax

1.4 (‘Config > Compare’) and ‘URL Mapping’
Сложность 1
Расширение
аудитории
свой текст здесь
здесь.
Лимит в 30 дней
● Введите свой
Повышение
конверсии
свой текст здесь.

2. Google Analytics
● Sessions Above 0 – This simply means the URL
in question has 1 or more sessions.
● Bounce Rate Above 70% – This means the URL
has a bounce rate over 70%, which you may
wish to investigate. In some scenarios this is
normal though!
● No GA Data – This means that for the metrics
and dimensions queried, the Google API didn’t
return any data for the URLs in the crawl. So the
URLs either didn’t receive any visits sessions, or
perhaps the URLs in the crawl are just different
to those in GA for some reason.
● Non-Indexable with GA Data – This means the
URL is non-indexable, but still has data from GA.
● Orphan URLs – This means the URL was only
discovered via GA, and was not found via an
internal link during the crawl.

2. Google Search Console
● If you wish to crawl new URLs discovered from Google Search Console to find any potential
orphan pages, remember to enable the configuration shown below.
● You can navigate to the ‘URL Inspection’ tab and ‘Enable URL Inspection’ to collect data about the
indexed status of up to 2,000 URLs in the crawl.
● The ‘Ignore Non-Indexable URLs for URL Inspection’ means any URLs in the crawl that are
classed as ‘Non-Indexable’, won’t be queried via the API

2.1 Google Search Console
● If you wish to crawl new URLs discovered from Google Search Console to find any potential
orphan pages, remember to enable the configuration shown below.
● You can navigate to the ‘URL Inspection’ tab and ‘Enable URL Inspection’ to collect data about the
indexed status of up to 2,000 URLs in the crawl.
● The ‘Ignore Non-Indexable URLs for URL Inspection’ means any URLs in the crawl that are
classed as ‘Non-Indexable’, won’t be queried via the API

● Clicks Above 0 – URL in question has 1 or more clicks.
● No GSC Data – Search Analytics API didn’t return any data for the URLs in the crawl.
● Non-Indexable with GSC Data – URLs that are classed as non-indexable, but have Google Search
Analytics data.
● Orphan URLs – URLs that have been discovered via Google Search Analytics, rather than internal links
during a crawl.
● URL Is Not on Google – The URL is not indexed by Google and won’t appear in the search results. This
filter can include non-indexable URLs (such as those that are ‘noindex’) as well as Indexable URLs that
are able to be indexed. It’s a catch all filter for anything not on Google according to the API.
● Indexable URL Not Indexed – Indexable URLs found in the crawl that are not indexed by Google and
won’t appear in the search results.
● URL is on Google, But Has Issues – The URL has been indexed and can appear in Google Search
results, but there are some problems with mobile usability, AMP or Rich results that might mean it
doesn’t appear in an optimal way.

● User-Declared Canonical Not Selected – Google has chosen to index a different URL
to the one declared by the user in the HTML. Canonicals are hints, and sometimes
Google does a great job of this, other times it’s less than ideal.
● Page Is Not Mobile Friendly – The page has issues on mobile devices.
● AMP URL Is Invalid – The AMP has an error that will prevent it from being indexed.
● Rich Result Invalid – The URL has an error with one or more rich result enhancements
that will prevent the rich result from showing in the Google search results. To export
specific errors discovered, use the ‘Bulk Export > URL Inspection > Rich Results’ export.

3.1 ‘Configuration > Spider > Extraction’

4. Log File Analyser — I love this topic I can talk about
it for hours…
● https://docs.google.com/document/d/1D6Sum8xTO7MpP_KGJ0F
PRq_tv4yx2qBjhIwmPdQWfiY/edit#heading=h.fihvxqo2f1i2 —
Instruction on how to perform the analysis

5.1 Mode-List
In this mode you can check a predefined list of URLs. This list can
come from a variety of sources — a simple copy and paste, or a .txt,
.xls, .xlsx, .csv or .xml file.

6. Custom Extraction — ‘Configuration > Custom >
Extraction’
— allows you to scrape any data from the HTML of a web page using CSSPath, XPath
and regex.

Thx you! Happy crawling!
Questions?

Screaming Frog: Little-Known Features In The SEO Spider

Recommended

Recommended

More Related Content

Similar to Screaming Frog: Little-Known Features In The SEO Spider

Similar to Screaming Frog: Little-Known Features In The SEO Spider (20)

Recently uploaded

Recently uploaded (20)

Screaming Frog: Little-Known Features In The SEO Spider