This document discusses search engine optimization (SEO) and how to manage robots.txt files in DNN for single-site and multi-site instances. It covers common SEO techniques, the purpose and components of robots.txt, and how to configure multiple robots.txt files for different sites in a multi-portal DNN environment using URL rewrite rules. The presenter provides examples of robots.txt directives and files and demonstrates their use in DNN.

1. Robots.txt & Multi-Site
DNN Instances
A Primer on SEO & Robots.txt in DNN 9.8+

2. Thank You to Our Sponsors!

3. Will Strohl
• Using DNN since 2003
• CEO & Founder, Upendo Ventures
• Former DNN Corp Employee
• Former CEO, Hotcakes Commerce (2013-2017)
• DNN MVP Since 2013
• Leader of the Awareness Group
• DNN Author & Technical Editor
• Founder of Day of DNN & DNNCon
• Maintainer of 40+ Open-Source Projects on GitHub
@WillStrohl
WillStrohl.com

4. Upendo Ventures
• DNN Platform & Evoq
• E-Commerce
• Implementations
• Integrations
• Custom Modules
• Upgrades & Edition Migrations
• Custom Themes
• Training
• Fixing Things Other People Did ;)
@UpendoVentures
UpendoVentures.com

5. Robots.txt & Multi-Site
DNN Instances
A Primer on SEO & Robots.txt in DNN 9.8+

6. Objectives
I Hope You Learn…
• The Basics of SEO
• How to Manage Robots.txt
• The Do’s & Don’ts of Robots.txt
• How to Have More than One
Robots.txt in Multi-Portal
Environments
Prerequisites
• You Can Spell SEO ;)
• General Knowledge of How SEO
Works
• General Understanding of Your
Website Implementation
• General Understanding of How to
Administrate DNN

7. What is SEO?
Search Engine Optimization (“SEO”)
The process of maximizing the number of visitors to a particular
website by ensuring that the site appears high on the list of
results returned by a search engine.

8. Common SEO Techniques
• Digital Marketing Strategy
• Content Creation
• Regular Content Updates
• Visitor Engagement
• Offsite Content & Links
• Multiple Mediums
• Multiple Channels
• Use HTTPS for All Requests
• Generate a Video Playlist
• Reduce 404 Errors
• Decrease Page Load Times
• Keyword Research
• Keyword Acquisition
• Refine Targeted Content
• Use Schema Markup
• Properly Implement Semantic
Markup
• Optimize for Voice Search

9. Common SEO Techniques
Keep the Search Engines & Crawlers Informed

10. Page Meta Data
<meta name="robots" content="noindex" />
<meta name="robots" content="nofollow" />
<meta name="robots" content="noindex,nofollow" />
Default: Index, Follow

11. Page Meta Data
<meta id="MetaRobots" name="robots" content="INDEX, FOLLOW">
<meta id="MetaRobots" name="robots" content="NOINDEX, NOFOLLOW">

12. What is Robots.txt?
The robots exclusion standard, also known as the robots
exclusion protocol or simply robots.txt, is a standard used by
websites to communicate with web crawlers and other web
robots. The standard specifies how to inform the web robot
about which areas of the website should not be processed or
scanned.

13. User-agent
User-agent: [bot identifier/crawler name]
Specifies a bot or crawler to instruct.
User-agent: msnbot
User-agent: Googlebot
User-agent: Slurp
User-agent: *

14. Allow/Disallow
Allow: [path to crawl]
Disallow: [path to ignore]
Specifies a path where crawlers are or are not allowed to go.
Allow: /
Disallow: /activity-feed/

15. Crawl-delay
Crawl-delay: [milliseconds]
Indicates a period of time to wait between page requests.
Crawl-delay: 120

16. Sitemap
Sitemap: [path to sitemap]
Allows you to specify the location of one or more sitemaps for
the search engine to follow.
Sitemap: https://dnncommunity.org/sitemap.xml

17. Robots.txt Example

18. Pattern Matching
* is a wildcard pattern, matching any character
$ is the end of the URL
Allow: *.html
Disallow: *.php
Allow: *.aspx$

19. Robots.txt in DNN

20. Robots.txt in DNN
• Included since DNN 7.x
• Update for Every Instance
• Difficult to Support Multi-Site
• Used to Require Direct File System Access
• DnnSoftware.com has Good Examples

21. Multi-Site Robots.txt in DNN
• Requires Rewrite Rules Module
• https://www.iis.net/downloads/microsoft/url-rewrite
• Create a Robots.txt for Each Site
• example.com.robots.txt
• Add a Rewrite Rule for Each Site
• example.com.robots.txt becomes robots.txt

22. Multi-Site Robots.txt in DNN

23. Demo
Let’s See This Stuff in Action!

24. Thank You to Our Sponsors!

25. Upendo Ventures
Summary
• SEO important on all public sites
• There are many ways to improve SEO (all
are important)
• DNN makes all of these easy
• Robots.txt is fully supported
• Robots.txt files can be edited directly in
the Config Manager
• Be sure to throttle bots using Crawl-
delay
• Multi-site instances require some more
configuration
• Review robots.txt after any major
roll-out
We use technology to help your business
change people’s lives
UpendoVentures.com
solutions@upendoventures.com
(650) 381-9160
548 Market St. #65401
San Francisco, CA 94104