Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
@rachellcostello brightonSEO
TALK ABOUT WHAT YOU KNOW
What we do every day:
Help people to crawl websites
to get insights,...
Recommended fixes for a better site.
@rachellcostello brightonSEO
WHAT WE’LL COVER
Tips and tactics for crawling large sit...
CRAWLING LARGE SITES
Effectively & Efficiently
@rachellcostello brightonSEO
The sheer scale of enterprise sites and
knowing where to start can be daunting.
@rachellcostello brightonSEO
That’s why you need to start smaller.
@rachellcostello brightonSEO
Targeted crawls can run regularly, allowing
even the largest sites to bypass resource
and time constraints.
@rachellcostel...
Good news: you don’t
need to crawl every
URL every time.
TACTICAL CRAWLING
@rachellcostello brightonSEO
You only need enough
data to validate issues.
TACTICAL CRAWLING
@rachellcostello brightonSEO
TACTICAL CRAWLING =
@rachellcostello brightonSEO
Getting the data you need as
quickly as possible.1
Building the bigger pi...
@rachellcostello brightonSEO
@rachellcostello brightonSEO
With tactical crawling methods, you can
use segments to give a representative
picture of the whole site.
@rachellcostello ...
Learn more about a site by breaking it into
manageable, predictable chunks.
@rachellcostello brightonSEO
Depending on the vertical, each site should
be able to be split into 5-7 categories:
@rachellcostello brightonSEO
Category PagesProduct Pages Blog Pages Vanity Pages
Navigational
Pages
Transactional
Pages
FAQ Pages
@rachellcostello brig...
SAMPLING SLICING
@rachellcostello brightonSEO
SAMPLING
@rachellcostello brightonSEO
SAMPLING
This method is about
collecting a percentage
of the whole site.
@rachellcostello brightonSEO
SAMPLING METHOD #1
Start a small crawl and find
areas of crawl waste to
exclude for whole crawl.
@rachellcostello brighton...
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Tracking URLs
Pagination URLs
Product Page Parameter ...
@rachellcostello brightonSEO
SAMPLING METHOD #2
Use level limitation to
assess site breadth without
getting lost in the weeds.
@rachellcostello brighto...
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Category Page
Subcategory
Page
Subcateg...
@rachellcostello brightonSEO
SAMPLING METHOD #3
Crawl a certain number of
examples of each page
type.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Product Page 4Category Page 4 Blog Post 4
Category Page 3
Category Page 2
Category Page 1
Pro...
@rachellcostello brightonSEO
SLICING
@rachellcostello brightonSEO
SLICING
This method is about
analysing an isolated
section of the site.
@rachellcostello brightonSEO
SLICING METHOD #1
All of the products but none
of the categories
(horizontal slice).
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Category Page
Subcategory
Page
Subcateg...
@rachellcostello brightonSEO
SLICING METHOD #2
Crawl a distinct section that
serves a separate function
(e.g. blog or separate
mobile site).
@rachellco...
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Blog
Blog Category
Subcategory
Page
Lev...
@rachellcostello brightonSEO
@rachellcostello brightonSEO
1. A strong history of organic traffic.
2. Consistent levels of customer engagement.
+
3. Clo...
Because when you see the needle move
it will be more meaningful.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Supplement your segments in external data sources too.
Use custom properties,
inclusion rules...
@rachellcostello brightonSEO
Tactical crawling allows you to get to know
your website and its structure better.
@rachellcostello brightonSEO
Establishing patterns means you can crawl
more efficiently in future.
@rachellcostello brightonSEO
But, remember to also run annual or
semi-annual full crawls and tech SEO
audits.
@rachellcostello brightonSEO
The takeaways from these are the primary
baselines for planning tactical crawls.
@rachellcostello brightonSEO
HOW THE BIGGEST
BRANDS MEASURE UP
@rachellcostello brightonSEO
I analysed the most recent audits
we’ve completed using tactical
crawling methods for our biggest
enterprise clients.
Here...
FOOD
DELIVERY
TRANSPORTATION
ONLINE
CLASSIFIEDS
@rachellcostello brightonSEO
The key overlapping problem
areas...
@rachellcostello brightonSEO
1
2
3
4
5 Mobile
Internal Linking
Site Speed
Indexing
Internationalisation
@rachellcostello brightonSEO
MAIN
INTERNATIONALISATION
FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
@ra...
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
Add...
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
Add...
MAIN INDEXING FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
@rachellco...
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all...
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all...
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all...
MAIN SITE SPEED FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
@rach...
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
Reduc...
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
Reduc...
MAIN INTERNAL LINKING
FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
no...
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
no...
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
no...
MAIN MOBILE FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
@rachellcos...
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all...
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all...
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all...
Luckily, there are already
resources on these topics!
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Internationalisation - bit.ly/internationalisation-dc
Indexing - bit.ly/indexing-dc
Site Spee...
@rachellcostello brightonSEO
TO WIN IN ENTERPRISE SEO:
Run targeted, agile crawls on smaller sections
of your site.
Better...
@rachellcostello brightonSEO
THANK YOU!
Any questions? Send me a tweet:
@DeepCrawl www.deepcrawl.com
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl
Upcoming SlideShare
Loading in …5
×

How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl

2,962 views

Published on

How do you extract meaningful insights from the largest websites, spanning millions of URLs? Digging into that amount of data to find insights, or even knowing where to start can be daunting, but at DeepCrawl we've already done the hard work for you. In this talk, Rachel will share real-world examples of how we worked with some top-tier, world-class brands to dissect and analyse their enterprise sites, and how that data was used to inform impactful changes that improved the quality of their websites.

Published in: Marketing
  • GIVE HER A BIGGER PACKAGE THIS VALENTINE'S DAY ◆◆◆ https://tinyurl.com/yy3nfggr
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Did you know that once you lose your Ex, there is still a good chance you can get them back? Learn how ♣♣♣ http://t.cn/R50e2MX
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl

  1. 1. @rachellcostello brightonSEO TALK ABOUT WHAT YOU KNOW What we do every day: Help people to crawl websites to get insights, no matter how many URLs they have.
  2. 2. Recommended fixes for a better site. @rachellcostello brightonSEO WHAT WE’LL COVER Tips and tactics for crawling large sites. Common pitfalls of the biggest brands.
  3. 3. CRAWLING LARGE SITES Effectively & Efficiently @rachellcostello brightonSEO
  4. 4. The sheer scale of enterprise sites and knowing where to start can be daunting. @rachellcostello brightonSEO
  5. 5. That’s why you need to start smaller. @rachellcostello brightonSEO
  6. 6. Targeted crawls can run regularly, allowing even the largest sites to bypass resource and time constraints. @rachellcostello brightonSEO
  7. 7. Good news: you don’t need to crawl every URL every time. TACTICAL CRAWLING @rachellcostello brightonSEO
  8. 8. You only need enough data to validate issues. TACTICAL CRAWLING @rachellcostello brightonSEO
  9. 9. TACTICAL CRAWLING = @rachellcostello brightonSEO Getting the data you need as quickly as possible.1 Building the bigger picture from smaller parts.2
  10. 10. @rachellcostello brightonSEO
  11. 11. @rachellcostello brightonSEO
  12. 12. With tactical crawling methods, you can use segments to give a representative picture of the whole site. @rachellcostello brightonSEO
  13. 13. Learn more about a site by breaking it into manageable, predictable chunks. @rachellcostello brightonSEO
  14. 14. Depending on the vertical, each site should be able to be split into 5-7 categories: @rachellcostello brightonSEO
  15. 15. Category PagesProduct Pages Blog Pages Vanity Pages Navigational Pages Transactional Pages FAQ Pages @rachellcostello brightonSEO
  16. 16. SAMPLING SLICING @rachellcostello brightonSEO
  17. 17. SAMPLING @rachellcostello brightonSEO
  18. 18. SAMPLING This method is about collecting a percentage of the whole site. @rachellcostello brightonSEO
  19. 19. SAMPLING METHOD #1 Start a small crawl and find areas of crawl waste to exclude for whole crawl. @rachellcostello brightonSEO
  20. 20. @rachellcostello brightonSEO Homepage Category Page Subcategory Page Tracking URLs Pagination URLs Product Page Parameter URLs Social Tag URLs
  21. 21. @rachellcostello brightonSEO
  22. 22. SAMPLING METHOD #2 Use level limitation to assess site breadth without getting lost in the weeds. @rachellcostello brightonSEO
  23. 23. @rachellcostello brightonSEO Homepage Category Page Subcategory Page Category Page Category Page Subcategory Page Subcategory Page Level 1 Level 2 Level 3 Product Page Product Page Product PageLevel 4
  24. 24. @rachellcostello brightonSEO
  25. 25. SAMPLING METHOD #3 Crawl a certain number of examples of each page type. @rachellcostello brightonSEO
  26. 26. @rachellcostello brightonSEO Product Page 4Category Page 4 Blog Post 4 Category Page 3 Category Page 2 Category Page 1 Product Page 3 Product Page 2 Product Page 1 Blog Post 3 Blog Post 2 Blog Post 1
  27. 27. @rachellcostello brightonSEO
  28. 28. SLICING @rachellcostello brightonSEO
  29. 29. SLICING This method is about analysing an isolated section of the site. @rachellcostello brightonSEO
  30. 30. SLICING METHOD #1 All of the products but none of the categories (horizontal slice). @rachellcostello brightonSEO
  31. 31. @rachellcostello brightonSEO Homepage Category Page Subcategory Page Category Page Category Page Subcategory Page Subcategory Page Level 1 Level 2 Level 3 Product Page Product Page Product PageLevel 4
  32. 32. @rachellcostello brightonSEO
  33. 33. SLICING METHOD #2 Crawl a distinct section that serves a separate function (e.g. blog or separate mobile site). @rachellcostello brightonSEO
  34. 34. @rachellcostello brightonSEO Homepage Category Page Subcategory Page Category Page Blog Blog Category Subcategory Page Level 1 Level 2 Level 3 Product Page Product Page Blog PostLevel 4
  35. 35. @rachellcostello brightonSEO
  36. 36. @rachellcostello brightonSEO 1. A strong history of organic traffic. 2. Consistent levels of customer engagement. + 3. Close monitoring of all changes made. + HAVE SUBSETS WITH:
  37. 37. Because when you see the needle move it will be more meaningful. @rachellcostello brightonSEO
  38. 38. @rachellcostello brightonSEO Supplement your segments in external data sources too. Use custom properties, inclusion rules and filtered reports to pre-filter data in: Analytics tools Log files Sitemaps Google Search Console
  39. 39. @rachellcostello brightonSEO
  40. 40. Tactical crawling allows you to get to know your website and its structure better. @rachellcostello brightonSEO
  41. 41. Establishing patterns means you can crawl more efficiently in future. @rachellcostello brightonSEO
  42. 42. But, remember to also run annual or semi-annual full crawls and tech SEO audits. @rachellcostello brightonSEO
  43. 43. The takeaways from these are the primary baselines for planning tactical crawls. @rachellcostello brightonSEO
  44. 44. HOW THE BIGGEST BRANDS MEASURE UP @rachellcostello brightonSEO
  45. 45. I analysed the most recent audits we’ve completed using tactical crawling methods for our biggest enterprise clients. Here are the key takeaways of 3... @rachellcostello brightonSEO
  46. 46. FOOD DELIVERY TRANSPORTATION ONLINE CLASSIFIEDS @rachellcostello brightonSEO
  47. 47. The key overlapping problem areas... @rachellcostello brightonSEO
  48. 48. 1 2 3 4 5 Mobile Internal Linking Site Speed Indexing Internationalisation @rachellcostello brightonSEO
  49. 49. MAIN INTERNATIONALISATION FIXES @rachellcostello brightonSEO
  50. 50. @rachellcostello brightonSEO
  51. 51. Removing URLs for nonsensical locales. @rachellcostello brightonSEO
  52. 52. Removing URLs for nonsensical locales. Use location to show relevant results. @rachellcostello brightonSEO
  53. 53. Removing URLs for nonsensical locales. Use location to show relevant results. Allow users to toggle between languages. @rachellcostello brightonSEO
  54. 54. Removing URLs for nonsensical locales. Use location to show relevant results. Allow users to toggle between languages. Add x-default for non- regionalised URLs. @rachellcostello brightonSEO
  55. 55. Removing URLs for nonsensical locales. Use location to show relevant results. Allow users to toggle between languages. Add x-default for non- regionalised URLs. Translate all titles and descriptions to the correct language. @rachellcostello brightonSEO
  56. 56. MAIN INDEXING FIXES @rachellcostello brightonSEO
  57. 57. @rachellcostello brightonSEO
  58. 58. Replace 302 redirects with 301 redirects. @rachellcostello brightonSEO
  59. 59. Replace 302 redirects with 301 redirects. Fix any server errors in Google Search Console ‘Crawl Errors’ report. @rachellcostello brightonSEO
  60. 60. Replace 302 redirects with 301 redirects. Fix any server errors in Google Search Console ‘Crawl Errors’ report. Ensure all content elements are unique and relevant. @rachellcostello brightonSEO
  61. 61. Replace 302 redirects with 301 redirects. Fix any server errors in Google Search Console ‘Crawl Errors’ report. Ensure all content elements are unique and relevant. @rachellcostello brightonSEO Increase pagination to show more results per page.
  62. 62. Replace 302 redirects with 301 redirects. Fix any server errors in Google Search Console ‘Crawl Errors’ report. Ensure all content elements are unique and relevant. Increase pagination to show more results per page. Review orphaned URLs in the sitemaps. @rachellcostello brightonSEO
  63. 63. MAIN SITE SPEED FIXES @rachellcostello brightonSEO
  64. 64. @rachellcostello brightonSEO
  65. 65. Lazy-load below-the-fold images. @rachellcostello brightonSEO
  66. 66. Lazy-load below-the-fold images. Eliminate use of unnecessary plugins. @rachellcostello brightonSEO
  67. 67. Lazy-load below-the-fold images. Eliminate use of unnecessary plugins. Remove unused and/or render-blocking scripts. @rachellcostello brightonSEO
  68. 68. Lazy-load below-the-fold images. Eliminate use of unnecessary plugins. Remove unused and/or render-blocking scripts. Reduce page HTML size for high priority templates. @rachellcostello brightonSEO
  69. 69. Lazy-load below-the-fold images. Eliminate use of unnecessary plugins. Remove unused and/or render-blocking scripts. Reduce page HTML size for high priority templates. Increase caching time for static pages from minutes to days. @rachellcostello brightonSEO
  70. 70. MAIN INTERNAL LINKING FIXES @rachellcostello brightonSEO
  71. 71. @rachellcostello brightonSEO
  72. 72. Remove redirect loops and chains. @rachellcostello brightonSEO
  73. 73. Remove redirect loops and chains. Reinstate links to pages with 10+ impressions and/or visits. @rachellcostello brightonSEO
  74. 74. Remove redirect loops and chains. Reinstate links to pages with 10+ impressions and/or visits. Remove internal links to nofollow pages. @rachellcostello brightonSEO
  75. 75. Remove redirect loops and chains. Reinstate links to pages with 10+ impressions and/or visits. Remove internal links to nofollow pages. Avoid using parameter URLs for tracking. @rachellcostello brightonSEO
  76. 76. Remove redirect loops and chains. Reinstate links to pages with 10+ impressions and/or visits. Remove internal links to nofollow pages. Avoid using parameter URLs for tracking. Remove internal links to canonicalised search pages. @rachellcostello brightonSEO
  77. 77. MAIN MOBILE FIXES @rachellcostello brightonSEO
  78. 78. @rachellcostello brightonSEO
  79. 79. Avoid using ‘maximum-scale =1’ in viewport settings. @rachellcostello brightonSEO
  80. 80. Avoid using ‘maximum-scale =1’ in viewport settings. Change ‘user-scalable= no’ to ‘yes’ in viewport settings. @rachellcostello brightonSEO
  81. 81. Avoid using ‘maximum-scale =1’ in viewport settings. Change ‘user-scalable= no’ to ‘yes’ in viewport settings. Migrate all pages to responsive ASAP. @rachellcostello brightonSEO
  82. 82. Avoid using ‘maximum-scale =1’ in viewport settings. Change ‘user-scalable= no’ to ‘yes’ in viewport settings. Migrate all pages to responsive ASAP. Ensure each mobile-desktop reciprocal URL pair has 200 status. @rachellcostello brightonSEO
  83. 83. Avoid using ‘maximum-scale =1’ in viewport settings. Change ‘user-scalable= no’ to ‘yes’ in viewport settings. Migrate all pages to responsive ASAP. Ensure each mobile-desktop reciprocal URL pair has 200 status. Fix content mismatches between mobile and desktop URLs. @rachellcostello brightonSEO
  84. 84. Luckily, there are already resources on these topics! @rachellcostello brightonSEO
  85. 85. @rachellcostello brightonSEO Internationalisation - bit.ly/internationalisation-dc Indexing - bit.ly/indexing-dc Site Speed - bit.ly/site-speed-dc Internal Linking - bit.ly/internal-linking-dc Mobile - bit.ly/mobile-dc RESOURCES
  86. 86. @rachellcostello brightonSEO TO WIN IN ENTERPRISE SEO: Run targeted, agile crawls on smaller sections of your site. Better understand internationalisation, indexing, site speed, internal linking and mobile. Use the findings to deliver quicker and more impactful insights into site health. 1 2 3
  87. 87. @rachellcostello brightonSEO THANK YOU! Any questions? Send me a tweet: @DeepCrawl www.deepcrawl.com

×