Learn how to modify the Robots.txt file that resides on your web server to eliminate issues with duplicate content.

1. By Chris J. Everett, Principal & Founder
Captivate Search Marketing

2. When reading blogs that discuss topics related to SEO, most of the time you'll
read about on-site strategies such as keyword optimization, image optimization,
cross linking, etc.
Most of these on-site strategies can be easily executed through the administration
interface of your Exponent CMS website.
Some of the mission critical items for SEO that you can't execute through the
admin interface of your Exponent CMS must be done on the web server through
a FTP or cPanel/WHM type interface.

3. This post is the first in a series of submissions I'll write over the next few days
discussing advanced server-side SEO tips for your Exponent CMS website
include editing your robots.txt and .htaccess files, as well as pushing webmaster
tools verification files and XML sitemaps to your web server.
In this edition, Part I, I'll discuss how modifying the Robots.txt file that resides on
your web server can eliminate issues with duplicate content.

5. The Robots.txt file on your web server defines crawling parameters for robots that
crawl websites all over the Internet.
For SEO, the Robots.txt file is a way to allow/disallow search engine robots (such
as Googlebot) that index your web pages from crawling specific directories on
your website.

6. If you're working with a
new install of Exponent
CMS, the Robots.txt file
will be included in your
installation with a basic
set of disallow
parameters that will look
like this:

7. Because Exponent CMS is a dynamic platform, there are several modules
within the system such as blog/news/portfolio or ecommerce that can pose
some SEO headaches revolving around duplicate content.
Some of the most frequent issues that I encounter from Exponent CMS
websites in terms of duplicate title tags, content, etc., come from the tagging
functionality (blogs in specific) and when login/view cart links are hard coded for
users on ecommerce websites.

8. If a robot can crawl a link to a blog tag or a link to login, this often times causes
that URL to be indexed – more often than not with a duplicate Title or Description
tag as other dynamic pages on your site.
The example below shows an ecommerce Exponent CMS website that has had
its login module link indexed. As you can see, the meta description is exactly the
same as the homepage meta description:
You can easily prevent this type of SEO issue by disallowing the /login module (or
for newer versions of Exponent the /login) or any other troublesome directories
within your Robots.txt file.

9. In the screenshot below, you can see where this Robots.txt file has added more
parameters than the initial file that comes with your Exponent CMS install.
Simply pull down this file
from public_html through
your FTP or cPanel/WHM
interface, make these
modifications and push that
file back up to your web
server to ensure you don't
experience these easily
avoidable SEO headaches.

10. In Part II of these server-side SEO tips for Exponent CMS sites, I'll discuss how
to modify the .htaccess in order to redirect URLs and set a preferred domain URL
structure.