Role Of Robots In Search Engine Optimization
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Role Of Robots In Search Engine Optimization

  • 528 views
Uploaded on

Robots.txt file are believed to be used for instigating search engines to crawl web sites. But this is a myth! The fact is, robot.txt files are used for one sole purpose of telling the search engines ...

Robots.txt file are believed to be used for instigating search engines to crawl web sites. But this is a myth! The fact is, robot.txt files are used for one sole purpose of telling the search engines spiders which web pages not to index.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
528
On Slideshare
528
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
4
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Role of Robots.txt in Search Engine Optimization
    Robots.txt file are believed to be used for instigating search engines to crawl web sites. But this is a myth! The fact is, robot.txt files are used for one sole purpose of telling the search engines spiders which web pages not to index. If any web page meets the specifications mentioned earlier then there is no need to encourage search engine spiders to index them for these pages will automatically get indexed.
     
    Search Engine Optimization is the process of getting your web pages indexed and improves upon the imperfections of your web site. So why would you ever want to keep your web pages away from being indexed? While indexing web pages, search engines function with restricted amount of time and resources. Therefore, your web site will benefit if you concentrate on getting the “important” customer-development, sales pages and products indexed.
    Note: Why do you want your shopping cart to get indexed? Indexing them brings you no benefit. So making them indexed by search engine spiders is unnecessary. Therefore make use of Robot.txt file to inform search engines not to waste time in indexing these pages. Rather get your sales, products or informative pages indexed.
    There are other pages that get indexed and have no benefit or rather carry sensitive data that you may not want to share with search engine spiders. Thus, make use of Robot.txt files for hiding them from search engine spiders. Those pages can be the cgi-bin folder and directories containing images. Though this will be profitable in the form that search
  • 2. Role of Robots.txt in Search Engine Optimization
    engines will spend increased amount of resources on your web pages, this can lead to consequences. For, search engine spiders index every possible web page on the internet including your password files that makes it easy for hackers to get access to your company’s personal data and misuse them.
     
    Another major concern with regard to Robot.txt file is that several web sites have created robot.txt files that automatically avert search engine spiders to crawl their sites.
     
    Did you know that by just adding the below mentioned two lines in your robot.txt file will keep all the search engines from crawling your website?
     
    User-agent: *
    Disallow: /
     
    Most people are not aware of the fact and therefore get baffled when they do not find their web page listed on search engines. Therefore, be careful of how you make use of Robot.txt files.
    Looking forHire SEO Expertor want SEO Consultant In India? Semaphore Software Offers Affordable SEO Services. Semaphore Software has highly qualified and Expert SEO Staff that will help your business to increase Online Visibility.