The document explains the purpose and importance of a robots.txt file, which directs web spiders on how to crawl a website. It provides guidelines for editing a robots.txt file, especially in the context of WordPress, and emphasizes best practices to avoid negatively impacting search engine indexing. Additionally, it mentions tools for checking content accessibility and outlines common mistakes to avoid when configuring the file.

1. How to Optimize Robots.txt
www.syscomminternational.com 079 6181 0430

2. What is Robots.txt ?
➢ A Robots.txt file is a file containing restrictions for web spiders,
defining all the locations they are permitted to crawl
➢ A Robots.txt file defines rules for search engine spiders(robots)
regarding what to follow and what not to follow
➢ It is not mandatory for web robots to obey your Robots.txt file,
however most of the web spiders follow the rules you define in it
➢ Your Robots.txt file is located at
http://www.’yourwebsitename’.com/robots.txt
www.syscomminternational.com 079 6181 0430

3. Why to use Robots.txt ?
➢ The Robots.txt file allows you to control how search
engines crawl your store.
➢ Your store already has a default Robots.txt file for search
engines, however you can rewrite it, if you wish to do so
and know how to do it.
www.syscomminternational.com 079 6181 0430

4. WordPress Robots.txt
➢ WordPress robots.txt file plays a major role in search
engine ranking.
➢ It helps to block search engine bots to index and crawl
important part of our blog. Though, some time a wrong
configured Robots.txt file can let your presence
completely go away from search engines.
➢ So, it’s important when you make changes in your
robots.txt file, it should be well optimized and should not
block access to important part of your website.
www.syscomminternational.com 079 6181 0430

5. www.syscomminternational.com 079 6181 0430

6. Robots.txt for WordPress
➢ You can either edit your WordPress Robots.txt file by logging
into your FTP account of server or you can use plugin like
Robots meta to edit robots.txt file from WordPress dashboard.
➢ There are few things, which you should add in your robots.txt
file along with your sitemap URL. Adding sitemap URL helps
search engine bots to find your sitemap file and thus faster
indexing of pages.
www.syscomminternational.com 079 6181 0430

7. ➢ Example of Robots.txt file for any domain.
www.syscomminternational.com 079 6181 0430

8. How to make sure no content is affected by new Robots.txt file?
➢ You can use Google Webmaster tool ‘Fetch as bot tool’ to see if
your content can be accessed by Robots.txt file or not. This step is
simple, login to Google Webmaster tool and go to diagnostic and
Fetch as Google bot. Add your site posts and check if there is any
issue accessing your post.
www.syscomminternational.com 079 6181 0430

9. ➢ You can also check for the crawl errors caused due to Robots.txt
file under Crawl error section of GWT. Under diagnostic >Crawl
error select Restricted by Robots.txt and you will see what all
links has been denied by Robots.txt file.
www.syscomminternational.com 079 6181 0430

10. What You Shouldn’t do?
• Don’t use comments in Robots.txt file.
• Don’t keep space in the beginning of any line and don’t
make ordinary space in file.
• Don’t change rules of command.
Bad Practice:
Disallow: /support
User-agent: *
Good Practice:
User-agent: *
Disallow: /support
www.syscomminternational.com 079 6181 0430

11. What You Shouldn’t do?
• If you want no index more then one directory or page don’t
write along with these names:
Bad Practice:
User-agent: *
Disallow: /support /cgi-bin /images/
Good Practice:
User-agent: *
Disallow: /support
Disallow: /cgi-bin
Disallow: /images
www.syscomminternational.com 079 6181 0430

12. ➢ Strategic planning for content, communications and community
➢ Campaign design and deployment services
➢ Conversation monitoring and optimisation
➢ Building and maintaining a social media newsroom.
➢ Establishing, integrating, and updating your presence in media
communities like Flickr and YouTube.
➢ Optimising Blogs and RSS feeds.
➢ Regularly submitting RSS feeds and Blogs to the latest
aggregators, news sites and social book marking sites.
➢ Recommendations for advertising and marketing opportunities
within social networking and media communities.
➢ Influencer outreach and buzz-building tactics
Services We Offer
www.syscomminternational.com 079 6181 0430

13. Join and Follow us:
Address:
Syscomm Consulting Limited
67, Loudoun Rd, London
NW8 0DQ
United Kingdom
Contact
Website:
www.syscomminternational.com
Email:
info@syscomminternational.com
Tel/fax:
079 6181 0430
www.syscomminternational.com 079 6181 0430

14. Thank you
www.syscomminternational.com 079 6181 0430