Boaz Sasson – One of the original 7 Samurai at SimilarWeb and currently Head of Performance Marketing. His main focus has always been technical SEO, but he’s spending more and more time on PPC and Social. SEO is a 4 legged animal, it stands on links/buzz, content, usability and crawlability. This webinar will cover crawlability, or the art of making love to spiders.

1. Boaz Sasson | Head of Performance
s a s s o n@ s i m i l a r w eb .c o m
Server-side SEO: The Art of Making
Love to Spiders

2. Every Website
 Traffic Trends Desktop & Mobile
 Referring Traffic
 Keywords
 Advertising Analysis
 Popular Pages
Every App
 Current User Installs
 Active Users
 Engagement per app
 Retention Analysis
 App store optimization
Every Country
We Reveal the Secrets of Online Success

3. Do
You
Know
?

4. Not Even a Spider: Googlebot is a headless
browser, not a
simple link crawler
Can render pages visually
Traverses the DOM
Executes AJAX, JS & forms
*Spotted in the wild as
early as 2010
( h t t p : / / s e a r c h e n g i n e l a n d . c o m / g o o g l e s - p r o p o s a l
- f o r - c r aw l i n g - a j a x - m a y - b e - l i v e - 3 4 4 11 )

5. Meet the Cookie Monster
Googlebot DOES seem able to
accept cookies
No longer a reliable way to
Segregate/sniff bot traffic

6. Mr. Greedybot Requires
Access to All Your Files,
and Quickly
Do NOT block JS, CSS, scripts or
images from Googlebot if they
are needed to render a page
Avoid setting crawl rate limits, if at
all possible
Search bot traffic can eat your
bandwidth, let them

7. Impact on PR/Linkjuice??
Not clear how it flows on links in JS,
forms, code, etc…
Consider all the links/filepaths in
code, as well as text links, as
part of the page’s link graph
Internal links can either promote
Indexation, or pass juice, or do both

8. What is a crawl
budget and how
does is affect me?

9. More Pages Crawled = More Pages Indexed =
More Traffic (*If site is healthy)

10. Depth Prob Based On:
Amount of incoming links/buzz
Content creation rate/amounts
Trust – more of it results in wider
and deeper crawling
Think in terms of both crawling a
site (laterally) and crawling a page
(depth)
Bad/low quality content getting
crawled is a waste of your crawl
budget

11. Indicator of Site’s Health
Crawl stats can be used as a quick
indicator of a site’s general SEO
health
How many pages indexed?
Trends?
What errors/parameters are
indexed?

12. Life is Hard
Not easy to get many pages
indexed quickly on a new site

13. What Should I
Block?

14. Golden rule: “One filepath per specific
content piece”

15. Low quality/trust pages

16. Duplicate (many forms), sorting, multi-
category, non-existent, framed content

17. How to Block
Content &
Some
Misconceptions

18. Better to delete crap than to block it

19. Assume that anything in the
DOM is technically accessible
to modern search bots, even
though it may not pass juice

20. Robots.txt only works on internally

21. Don't block with both robots.txt and meta
robots together

22. Best to block with meta robots & delete via
GWT (*renew every 6 months)

23. X-robots tag, is in document HEAD, useful
for PDFs, XML, etc…

24. Play around with blocking elements via
frames, tabs, forms, animations, lazyloading

25. Redirect Logic

26. Links lose juice with each hop

27. Catch as many instances as possible in
single rules

28. Default redirect should be 301

29. If no other options, use meta refresh set to
zero seconds for 301s, & 5 seconds for 302s

30. Google Really Dislikes Broken Links
Check using a scheduled 404 report or spider
Scan on a regular basis

31. Sessions,
parameters and
cookies

32. Do NOT:
Print session IDs/parameters on
filepaths in code
Pass session IDs via filepaths

33. Be mindful of parameters used, each is
considered to be a unique page

34. Use cookies to pass session info

35. If no other alternative, block
parameters via GWT

36. Supercookies (flash, browser cache,
fingerprinting, E Tags, etc)

37. URL
Structures

38. Filepaths can be flat, deep, or with several
parameters if needed, all seem to work fine

39. Have a clear hierarchy in terms of directory
structure, use internal links to emphasize
relationships

40. Be consistent - all lower case, hyphens not
underscores, avoid empty spaces

41. Think of clickability of the filepath
when seen by a human

42. Avoid foreign language encoding on URLs

43. Two Great & Free
Tools for Crawling

44. IIS SEO Toolkit
http://www.iis.net/downloads/microsoft/search-engine-optimization-toolkit

45. Xenu
http://home.snafu.de/tilman/xenulink.html

46. Quick Tests:
Technical,Penalty, or
Market?

47. 47/20
Thank You
Boaz Sasson
s a s s o n@ s i m i l a r w eb .c o m