A brief overview of a few big things we've done on Envato Elements to improve crawl budget, or as I'm calling it here: Crawl Capacity Management. This slide deck is presented at the Melbourne SEO meetup on Tuesday, the 8th of November.

1. MELB SEO Meetup
Crawl capacity management
on Envato Elements
Gastón Riera - @gastonriera

2. That's how I used to look,
all well dressed and all.
Gastón Riera - @gastonriera
Gastón Riera

3. Everything you need to get your
creative projects done.
Gastón Riera - @gastonriera
The big names:
Other very cool products:

4. Gastón Riera - @gastonriera
When should we care about crawl [capacity|budget]?
Then there is a shit ton of pages ingested by Google.
Here some examples of websites I've worked on

5. Let's get into SEO 🔎
Gastón Riera - @gastonriera

6. The problem:
A big part of the site was
not being indexed 👎
Gastón Riera - @gastonriera

7. ☹
Gastón Riera - @gastonriera

8. Gastón Riera - @gastonriera
It looked like Google was not recrawling a
large amount of pages!

9. Gastón Riera - @gastonriera
And taking very long to recrawl
some pages

10. Was Google actually
crawling the site?🤔
Gastón Riera - @gastonriera

11. Gastón Riera - @gastonriera
Yes, it was👍

12. But, it wasn't getting to
crawl the entire site 😕
Gastón Riera - @gastonriera

13. And, why was that?󰤈
Gastón Riera - @gastonriera

14. No idea
Gastón Riera - @gastonriera

15. Just kidding 😆
Gastón Riera - @gastonriera

16. We came up with two
theories to work on 😎
Gastón Riera - @gastonriera

17. - Content quality
- Internal linking
Gastón Riera - @gastonriera
We needed to work on

18. Gastón Riera - @gastonriera
We needed to work on
I'll get to them in a bit.
- Content quality
- Internal linking

19. What did we do? 🤔
Gastón Riera - @gastonriera

20. The basics!
Gastón Riera - @gastonriera

21. The basics!
- Noindex
Gastón Riera - @gastonriera

22. The basics!
- Noindex
- Redirects
Gastón Riera - @gastonriera

23. The basics!
- Noindex
- Redirects
- Nofollow
Gastón Riera - @gastonriera

24. The basics!
- Noindex
- Redirects
- Nofollow
- Crawl paths (more/less)
Gastón Riera - @gastonriera

25. Gastón, that's nothing new!
😡
Gastón Riera - @gastonriera

26. I know! 😉
Gastón Riera - @gastonriera

27. How are we using them?
Let's get to it
Gastón Riera - @gastonriera

28. Battle_1:
Content quality
Gastón Riera - @gastonriera

29. Content is not just text on the
page, but everything on it.
Every page is content.
Gastón Riera - @gastonriera
A key takeaway

30. Battle_1: Content quality
Gastón Riera - @gastonriera
Two options:
1. Add content focussing on quality over
quantity.
2. Remove content from Google's index.
We already had +9M items!

31. Battle_1: Content quality
Gastón Riera - @gastonriera
Two options:
1. Add content focussing on quality over
quantity. ❌
2. Remove content from Google's index.
We already had +9M items!

32. Battle_1: Content quality
Gastón Riera - @gastonriera
Two options:
1. Add content focussing on quality over
quantity. ❌
2. Remove content from Google's index. ✅
We already had +9M items!

33. Do you know what reduces the
content quality of any site?
Gastón Riera - @gastonriera

34. Do you know what reduces the
content quality of any site?
DUPLICATE CONTENT!
Gastón Riera - @gastonriera

35. Noindex and remove duplicates,
RUTHLESSLY
Gastón Riera - @gastonriera
Noindex a good part of
the items library.
-> Several million less
discoverable pages!
Why we decided to
noindex instead of
a fancier solution?
Ask me later
😉

36. A few tips on how to get what to noindex?
- Use google's crawled not indexed as a proxy
- Check duplicate titles/urls/content description
- Just a different image doesn't make it a different page to
the eyes of Google!
Gastón Riera - @gastonriera
Battle_1: Content quality

37. Noindex and remove duplicates,
RUTHLESSLY
Gastón Riera - @gastonriera
Why the redirected path
had 15% of site's traffic
and 20x the destination.
Ask me later
😉
Merged two translations that ended up being way
more similar that intended
-> A few millions pages removed from Google.

38. Other big things we did
● Turned Tag pages into Search pages
● Search pages are noindex by default
The overall result? Decreased the index size to a half
without impacting organic traffic.
Gastón Riera - @gastonriera
50%
Battle_1: Content quality

39. Battle_2
Internal linking
Gastón Riera - @gastonriera
Reference

40. Battle_2: Internal linking
Gastón Riera - @gastonriera
Out of many tactics:
1. Reduce the number of crawl paths
2. Nofollow on links to low-value pages

41. Basically,
Be intentional and smart
about crawl paths.
Gastón Riera - @gastonriera

42. Gastón Riera - @gastonriera
Link to only valuable pages
Added links between related
search pages
10% Organic traffic!
If it's a useful search page,
it will not have a noindex.
Note that

43. Gastón Riera - @gastonriera
Link to only valuable pages
Remove hreflang when you're uncertain
of the quality on other languages
15% size of index!
hreflang tags are
bidirectional, remove
them on every language.
Remember 😉

44. What are valuable pages?
Gastón Riera - @gastonriera
In short, pages we want Google to index.

45. Gastón Riera - @gastonriera
Link to only valuable pages
As per nofollow:
● Nofollow on links to noindex pages
● Filters and facets, all nofollow
The overall result? Google re-crawled more pages.
60%

46. So, why crawl capacity
management?
Gastón Riera - @gastonriera

47. Crawl budget stayed the
same.
Gastón Riera - @gastonriera
*On average, over the last 2yrs.

48. BONUS TRACK
and unpopular opinion.
Gastón Riera - @gastonriera
Sitemaps didn't help indexing AT ALL 󰤃

49. But actually, it is
Gastón Riera - @gastonriera

50. An unpopular
Gastón Riera - @gastonriera

51. BONUS TRACK
and unpopular opinion.
We learnt that
● Sitemaps didn't help indexing AT ALL
󰤃
● Helpful only for debugging 🤓
Gastón Riera - @gastonriera

52. Gastón Riera - @gastonriera
Gracias! - Thank You!