With so many data sources available today that need bringing together into one place we will explore how to best get your website in order using external/internal data sources for technical SEO optimisation but also setting yourself up for the consumer and driving ROI.

1. Sam Marsden | @sam_marsden #TechSEOBoost
Building Your Search Universe
Links, Log Files, GSC, and Everything in Between

2. Sam Marsden | @sam_marsden #TechSEOBoost
What are we going to cover?
What the hell is THE Search Universe?
Defining YOUR Search Universe
Getting the bigger picture -
Bringing together different parts of your Search Universe

3. Sam Marsden | @sam_marsden #TechSEOBoost
What is the Search Universe?

4. Sam Marsden | @sam_marsden #TechSEOBoost
What is the Search Universe?
The Search Universe encompasses
every URL that could possibly exist…
On every domain…
Regardless of whether the URL has been invoked

5. Sam Marsden | @sam_marsden #TechSEOBoost
What are the limits of the Search Universe?

6. Sam Marsden | @sam_marsden #TechSEOBoost
The Search Universe Limits
[scheme]://[domain-name].[TLD]/[path-name]
4-5 chars
max 253 chars
(37 allowed
chars)
max 2083 chars
(74 safe chars)
1002 delegated TLDs

7. Sam Marsden | @sam_marsden #TechSEOBoost
The Search Universe Formula
Sum[(37^k)* 1002 * Sum[(74^j), {j, 0, 2071-k}], {k, 2, 248}]

8. Sam Marsden | @sam_marsden #TechSEOBoost
The Search Universe Number
179,769,000,000,000,000,000,000,000,000,000,000,0
00,000,000,000,000,000,000,000,000,000,000,000,00
0,000,000,000,000,000,000,000,000,000,000,000,000
,000,000,000,000,000,000,000,000,000,000,000,000,
000,000,000,000,000,000,000,000,000,000,000,000,0
00,000,000,000,000,000,000,000,000,000,000,000,00
0,000,000,000,000,000,000,000,000,000,000,000,000
,000,000,000,000,000,000,000,000,000,000,000,000,
000,000,000,000,000

9. Sam Marsden | @sam_marsden #TechSEOBoost
The Search Universe Formula
Sum[(37^k)* 1002 * Sum[(74^j), {j, 0, 2071-k}], {k, 2, 248}]

10. Sam Marsden | @sam_marsden #TechSEOBoost
The Total Potential Number of URLs

11. Sam Marsden | @sam_marsden #TechSEOBoost
The Total Potential Number of URLs
784737662469851839238148157424821056960535254243427627032347798165496590186509884497079384545087250131665584732323029020968115044049340
768543077768887062516865578298543159515068147740651321339532877103281797982207062306788002155785858244242703373813166502999634913312928
838087456741086414034991347610789451621292215819421206230373165721570143065828756021435250420465590663912250173143392605384240521094509
0225235406974408247782644232526453597686504890883120620025029541522595784851103947840712521244960821970473309489871806139323628511478109
3048478937100163386870742309944803828198142115597744260799842370049714101533321653987455533474637527682316517735147306547970318482989911
407338722923669510926184000478998360445332499134094165564828770289301898201983043413216064835308108100584808070944855546137628250813070
8715287823006991789011101013014065967376808332969730973111260335109195323647999664691086192644145655664297809157770755494468954508549991
2503488413248949420999093389634411514708494873093137642745334512172794166215863679650206883098883226412412533869285362147086071202117736
490211788264888681452274934359153841488546263138401896337683585389050270080870068418135565720338609662210492382318337339465653803862441
249841096889669616484371004203818240779568891912908101663250492910813569596565948548119083274215815263073536980547590587202256883696429
3750528524111465179017254098375891444809395423322623381562374364064921804515376110901801125644306168029529923017144168645316227165186820
5695281253062063702690940608310049007740262369995145777807519645444722739929804647837095500046983063332876102471983151557113494680741150
705304549419006358172575934724486579166905034524117854803988871731643331832172215688107182454235342897872698976925843330496924879781201
331866454817166340353976506555313050551422879631900661955176009370524056148264570326183431154940364497991840150797755640661393584347567
948259966222242877453704587946060357440794133591539481561352841818910578995726182051279403877796603375577704655607957739037822676397174
1103668658340937787207678595764049472541639850668308214718389338503906551773245613371845840239806767118205800247278697290752189441765248
862274895925673982114883503869030458314473172140554676494756489818840995609703076827798833683149161064523337508268966360801478282219295
4761146383619813567775665914205466620904320412628933491779736067190170866141598196543711333955456271619495898249311653834779554527851162
7142351880721678464572361476663390962861071847525999474193850600711465076544469329807817220619380631187767945863516056151401186321613983
5838339672998526777425133509232797096718499332977023724431397811162858074645188379031737160223429361865851850424753251386653758920715637
986720239086772979960209708861948184281890071182723819202470880426982866185067084325025580477788198921379069735555849106745009300190834
002649919903577917693250242333909124984336882837568845358161718340609339021797414092804662616357542749961727319788252218375967095306189
492593295079531388962447278635681408836605517787830704331729989273264308323581996904655528690142571442849263229665810083897152521941415
635081532824679516159569208167011293103975331302623420352543498604231833236255337260488073591957033032737060848219299552982268131634972
997775653590402337186319344193126688466665414531721424348285089808078879210612359926328926510369021666589635262899634650327427470607242
5912051864572184323895929171020369037079884842508041313605830741852392486897104194949423244475931177116494727605862918047498560295178911
0263542186649287469147910904540226838414377113164665512166232983591485171778953606918455092405846698006559708683018725614163482694711273
607896494524339876891457930635148543343443572578913237452360217337958300907091238202126481680666289211018573484312786272606967133344066
848518380084707242221409750271173333084049965367203904578
7.95636796670821… x 103848

12. Sam Marsden | @sam_marsden #TechSEOBoost
The Googol Number
10,000,000,000,000,000,000,000,000,000,000,000,
000,000,000,000,000,000,000,000,000,000,000,00
0,000,000,000,000,000,000,000,000,000,000
10^100

13. Sam Marsden | @sam_marsden #TechSEOBoost
Defining YOUR Search Universe

14. Sam Marsden | @sam_marsden #TechSEOBoost
Vast choice of tools

15. Sam Marsden | @sam_marsden #TechSEOBoost
Are you getting the most out of your data?

16. Sam Marsden | @sam_marsden #TechSEOBoost
How can you bring different data sources together efficiently?

17. Sam Marsden | @sam_marsden #TechSEOBoost
Introducing Your Search Universe

18. Sam Marsden | @sam_marsden #TechSEOBoost
Building Your Search Universe

19. Sam Marsden | @sam_marsden #TechSEOBoost
I want to tell you a story

20. Sam Marsden | @sam_marsden #TechSEOBoost
Simple Story about one URL in Search Universe

21. Sam Marsden | @sam_marsden #TechSEOBoost
Starring:
Me as The URL

22. Sam Marsden | @sam_marsden #TechSEOBoost
Also Starring…
as the Backlink Source
as the Traffic Source
as the Log Summary Source
as the Consumer Source

23. Sam Marsden | @sam_marsden #TechSEOBoost

24. Sam Marsden | @sam_marsden #TechSEOBoost
Backlinks to broken pages
404 myurl.com/my-page 1000s of Backlinks!

25. Sam Marsden | @sam_marsden #TechSEOBoost
Backlinks to a nofollow page
myurl.com/my-page 1000s of Backlinks!
myurl.com/page-1
myurl.com/page-4
Meta Nofollow

26. Sam Marsden | @sam_marsden #TechSEOBoost
Where do backlinks land on your site?
myurl.com/my-page 1000s of Backlinks!

27. Sam Marsden | @sam_marsden #TechSEOBoost
Backlinks to non-indexable pages
Disallowed myurl.com/my-page 1000s of Backlinks!

28. Sam Marsden | @sam_marsden #TechSEOBoost
Duplicate pages with backlinks

29. Sam Marsden | @sam_marsden #TechSEOBoost
Duplicate pages with backlinks
myurl.com/my-page 876 Backlinks
myurl.com/my-page.html 456 Backlinks
Duplicates Canonical
456 Backlinks!
1332 Backlinks!

30. Sam Marsden | @sam_marsden #TechSEOBoost

31. Sam Marsden | @sam_marsden #TechSEOBoost
How is crawl budget distributed/utilized?

32. Sam Marsden | @sam_marsden #TechSEOBoost
Identify where crawl budget is being wasted
Indexable
Pages without
Googlebot
requests
Non-indexable
Pages
with
Googlebot
requests
Indexable
Pages
with
Googlebot
requests

33. Sam Marsden | @sam_marsden #TechSEOBoost
Identify crawl budget wastage
myurl.com/non-indexable 100 requests
myurl.com/broken 330 requests
myurl.com/page?sort=price 56 requests

34. Sam Marsden | @sam_marsden #TechSEOBoost
Optimising crawl budget – Low value pages with requests
myurl.com/my-page
2,000
impressions
Low DeepRank300 page
views
1,000 requests

35. Sam Marsden | @sam_marsden #TechSEOBoost
Determine most crawled site sections
myurl.com/sitemap_index.xml 2,300 requests
myurl.com/categories 330 requests
myurl.com/products 232 requests

36. Sam Marsden | @sam_marsden #TechSEOBoost
Now the fun begins…
myurl.com/dress
myurl.com/ dress?colour=red
2,768 Visits
1000s of Backlinks
50 Googlebot
requests
2,423 Clicks
myurl.com/ dress?colour=blue
myurl.com/ dress?colour=green
myurl.com/ dress?colour=green&price=1000+
Meta Canonical
myurl.com/ dress?colour=…&price=…
(…)

37. Sam Marsden | @sam_marsden #TechSEOBoost
Understand how bots crawl with different user agents

38. Sam Marsden | @sam_marsden #TechSEOBoost
Understand how bots crawl with different user agents
Responsive Dynamic
Would expect Google to use both
user-agents on the same URL to
validate that the same content is
returned.
Google needs to crawl with both
user agents to validate the mobile
version.
Hint: Use the Vary HTTP header!
Google needs to crawl the dedicated
mobile URLs with a mobile user agent to
validate the pages and confirm the content
matches the desktop pages.
Separate Mobile

39. Sam Marsden | @sam_marsden #TechSEOBoost

40. Sam Marsden | @sam_marsden #TechSEOBoost
What can Google Search Console add?

41. Sam Marsden | @sam_marsden #TechSEOBoost
URL story with Google Search Console
myurl.com/my-page
1,000 clicks
Disallowed

42. Sam Marsden | @sam_marsden #TechSEOBoost
Find poor experience pages with traffic
myurl.com/my-page
5,320 Clicks

43. Sam Marsden | @sam_marsden #TechSEOBoost
Improve site quality by pruning low quality content
myurl.com/my-page
5,320 Clicks
Indexed!
100 words
88% bounce
rate
Avg time on
page = 18s

44. Sam Marsden | @sam_marsden #TechSEOBoost
All well and good knowing what Google sees…

45. Sam Marsden | @sam_marsden #TechSEOBoost
But don’t forget about consumer power

46. Sam Marsden | @sam_marsden #TechSEOBoost
Google drives awareness
but people drive ROI!

47. Sam Marsden | @sam_marsden #TechSEOBoost
What do visitors see and convert on?

48. Sam Marsden | @sam_marsden #TechSEOBoost
A simple URL story with visits
Soft-404 myurl.com/my-page 1000s of Visits!

49. Sam Marsden | @sam_marsden #TechSEOBoost
Another simple URL story with visits
Blog post myurl.com/my-page 1000s of Visits!
No conversion
points

50. Sam Marsden | @sam_marsden #TechSEOBoost
An advanced story with visits & backlinks
Disallowed
myurl.com/my-page
2768 Visits!
301 redirection
myurl.com/redirect-page
301 redirection
myurl.com/other-page
No Indexed!
No Backlinks!
1000s of Backlinks!

51. Sam Marsden | @sam_marsden #TechSEOBoost
Scaling up
1 URL
100k URLs
1M URLs

52. Sam Marsden | @sam_marsden #TechSEOBoost
Merging Excel Spreadsheets

53. Sam Marsden | @sam_marsden #TechSEOBoost

54. Sam Marsden | @sam_marsden #TechSEOBoost
Automation is needed!

55. Sam Marsden | @sam_marsden #TechSEOBoost

56. Sam Marsden | @sam_marsden #TechSEOBoost
Start Building Your Search Universe…

57. Sam Marsden | @sam_marsden #TechSEOBoost
Thank you Boston!
Email: sam.marsden@deepcrawl.com
Twitter: @sam_marsden

58. Sam Marsden | @sam_marsden #TechSEOBoost