Successfully reported this slideshow.

DNN Summit: Robots.txt & Multi-Site DNN Instances

0

Share

Upcoming SlideShare
SEO and Marketing
SEO and Marketing
Loading in …3
×
1 of 25
1 of 25

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

DNN Summit: Robots.txt & Multi-Site DNN Instances

  1. 1. Robots.txt & Multi-Site DNN Instances A Primer on SEO & Robots.txt in DNN 9.8+
  2. 2. Thank You to Our Sponsors!
  3. 3. Will Strohl • Using DNN since 2003 • CEO & Founder, Upendo Ventures • Former DNN Corp Employee • Former CEO, Hotcakes Commerce (2013-2017) • DNN MVP Since 2013 • Leader of the Awareness Group • DNN Author & Technical Editor • Founder of Day of DNN & DNNCon • Maintainer of 40+ Open-Source Projects on GitHub @WillStrohl WillStrohl.com
  4. 4. Upendo Ventures • DNN Platform & Evoq • E-Commerce • Implementations • Integrations • Custom Modules • Upgrades & Edition Migrations • Custom Themes • Training • Fixing Things Other People Did ;) @UpendoVentures UpendoVentures.com
  5. 5. Robots.txt & Multi-Site DNN Instances A Primer on SEO & Robots.txt in DNN 9.8+
  6. 6. Objectives I Hope You Learn… • The Basics of SEO • How to Manage Robots.txt • The Do’s & Don’ts of Robots.txt • How to Have More than One Robots.txt in Multi-Portal Environments Prerequisites • You Can Spell SEO ;) • General Knowledge of How SEO Works • General Understanding of Your Website Implementation • General Understanding of How to Administrate DNN
  7. 7. What is SEO? Search Engine Optimization (“SEO”) The process of maximizing the number of visitors to a particular website by ensuring that the site appears high on the list of results returned by a search engine.
  8. 8. Common SEO Techniques • Digital Marketing Strategy • Content Creation • Regular Content Updates • Visitor Engagement • Offsite Content & Links • Multiple Mediums • Multiple Channels • Use HTTPS for All Requests • Generate a Video Playlist • Reduce 404 Errors • Decrease Page Load Times • Keyword Research • Keyword Acquisition • Refine Targeted Content • Use Schema Markup • Properly Implement Semantic Markup • Optimize for Voice Search
  9. 9. Common SEO Techniques Keep the Search Engines & Crawlers Informed
  10. 10. Page Meta Data <meta name="robots" content="noindex" /> <meta name="robots" content="nofollow" /> <meta name="robots" content="noindex,nofollow" /> Default: Index, Follow
  11. 11. Page Meta Data <meta id="MetaRobots" name="robots" content="INDEX, FOLLOW"> <meta id="MetaRobots" name="robots" content="NOINDEX, NOFOLLOW">
  12. 12. What is Robots.txt? The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
  13. 13. User-agent User-agent: [bot identifier/crawler name] Specifies a bot or crawler to instruct. User-agent: msnbot User-agent: Googlebot User-agent: Slurp User-agent: *
  14. 14. Allow/Disallow Allow: [path to crawl] Disallow: [path to ignore] Specifies a path where crawlers are or are not allowed to go. Allow: / Disallow: /activity-feed/
  15. 15. Crawl-delay Crawl-delay: [milliseconds] Indicates a period of time to wait between page requests. Crawl-delay: 120
  16. 16. Sitemap Sitemap: [path to sitemap] Allows you to specify the location of one or more sitemaps for the search engine to follow. Sitemap: https://dnncommunity.org/sitemap.xml
  17. 17. Robots.txt Example
  18. 18. Pattern Matching * is a wildcard pattern, matching any character $ is the end of the URL Allow: *.html Disallow: *.php Allow: *.aspx$
  19. 19. Robots.txt in DNN
  20. 20. Robots.txt in DNN • Included since DNN 7.x • Update for Every Instance • Difficult to Support Multi-Site • Used to Require Direct File System Access • DnnSoftware.com has Good Examples
  21. 21. Multi-Site Robots.txt in DNN • Requires Rewrite Rules Module • https://www.iis.net/downloads/microsoft/url-rewrite • Create a Robots.txt for Each Site • example.com.robots.txt • Add a Rewrite Rule for Each Site • example.com.robots.txt becomes robots.txt
  22. 22. Multi-Site Robots.txt in DNN
  23. 23. Demo Let’s See This Stuff in Action!
  24. 24. Thank You to Our Sponsors!
  25. 25. Upendo Ventures Summary • SEO important on all public sites • There are many ways to improve SEO (all are important) • DNN makes all of these easy • Robots.txt is fully supported • Robots.txt files can be edited directly in the Config Manager • Be sure to throttle bots using Crawl- delay • Multi-site instances require some more configuration • Review robots.txt after any major roll-out We use technology to help your business change people’s lives UpendoVentures.com solutions@upendoventures.com (650) 381-9160 548 Market St. #65401 San Francisco, CA 94104

Editor's Notes

  • https://en.wikipedia.org/wiki/Robots_exclusion_standard
    https://developers.google.com/search/docs/advanced/robots/intro
  • Generally, it’s considered a best practice to have this at the end of the robots.txt file.
  • Always been able to add a robots.txt to the root of the website.
    Now able to edit the robots.txt directly in the DNN Config Manager as of version 9.8.
  • Where do I edit the meta tags for a specific page?
    Where is the Config Manager and how do I use it?
  • ×