1. Thursday, October 21st, 2010. Dealing with Crawlers Search Engine Optimization The Good, The Bad, & The Ugly. WELCOME CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT

2. 2 What to Discuss? Dealing with the Crawlers the good & the bad robots ………………………….. What is robots.txt file? The Definition ………………………………... Structure of robots.txt file for SEO purpose The Syntax ………. Standard User-agent, Disallow ……………………………..…. Nonstandard Crawl-delay, Allow, Sitemap …………………. Extended Request-rate, Visit-time, Comment ………..……. Effective use of robots.txt file Best Practices ……………………… Be aware of rel=“nofollow” Comment Spammers …...……………… User Generated Spam How to avoid …………………....……………… 4 5 6 7 9 10 11 12 13 CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

3. 3 Dealing with the Crawlers The good & the bad robots CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

5. Make sure that you place the robots.txt file in the main directory.

6. Restrict crawling where it's not needed.

7. Common robot traps “Forms, Logins, Session IDs, Frames”.CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

10. Allow:

12. Visit-time:

13. Comment: CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

21. Disallow: /dir_name/

22. Disallow: /users/login/CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

26. Sitemap: http://www.domain.com/dir/s/names-sitemap.xml.gzCONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

28. Disallow: /admin/

30. Disallow: /admin/

32. Comment: because Yahoo sucks :PCONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

34. Use more secure methods for sensitive content.

35. Avoid allowing search result-like pages to be crawled.

36. Avoid allowing URLs created as a result of proxy services to be crawled.

37. Create separate robots.txt file for each subdomain.CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

40. Turn on comment moderation.

41. Use “nofollow” tags.

42. Disallow hyperlink in comments.

43. Block comments pages using robots.txt or META tags.

44. Think twice before enabling guestbook or comments.

45. Use blacklist to prevent repetitive spamming attempts.

46. Add a “report spam” feature to user profiles and friend invitations.

47. Monitor your site for spammy pages.CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

49. On Page Factors.

52. Make use of free webmasters toolsCONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X NEXT BACK

53. THANK YOU CONTACT: 2 Saint Genny Street, Kafr Abdou. Suite 103 - 105. Alexandria 21311, Egypt. Phone: +2 (03) 546-7622. URL: www.eSpace.com.eg X BACK