Robots and-sitemap - Version 1.0.1

979 views
921 views

Published on

Introduction to robots.txt and sitemap.xml files.

Published in: Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
979
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Robots and-sitemap - Version 1.0.1

  1. 1. Robots and SitemapBy Naji El Kotob.Information Technology Consultant_______________________________naji [@] DotNETHeroes.comReview 1.0.1 - 13.Business Intelligence Group BIG SARL
  2. 2. by Naji El KotobOutlines SEO Robots.txt Sitemap.xml Q&A
  3. 3. by Naji El KotobSEO
  4. 4. by Naji El KotobWhat is SEO SEO is short for search engine optimization orsearch engine optimizer. Search engine optimization is a methodology ofstrategies, techniques and tactics used toincrease the amount of visitors to a website byobtaining a high-ranking placement in the searchresults page of a search engine (SERP)-- including Google, Bing, Yahoo and other search engines.Source | http://www.webopedia.com/TERM/S/SEO.html
  5. 5. by Naji El KotobRobots.txt
  6. 6. by Naji El KotobRobots.txt Robots.txt is the file that Search Enginesuse to see what they should index. This file and site maps help make yoursite easier to navigate by the bots andlet them know what is legal and what youwould rather not have the published intheir engines.
  7. 7. by Naji El KotobRobots.txt
  8. 8. by Naji El KotobRobots.txt SyntaxUser-agent: *Disallow:Sitemap: http://dotnetheroes.com/sitemap.xml
  9. 9. by Naji El KotobRobots.txt Generatorhttp://tools.seobook.com/robots-txt/generator/
  10. 10. by Naji El KotobInteractive Demo
  11. 11. by Naji El KotobDemo Design and Create a Dynamic RobotsSource | http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/Adding a Dynamic Robots.txt file to an ASP.NET MVC siteRobots.txt is required to allow search engines to properly index yoursite, and more importantly not index it.If you have a public-facing staging or preliminary site that you don’twant to show up in Google results, you need to make sure that itreturns the correct robots.txt with theDisallow: /line to prevent indexing. However, manually adding robots.txt filesto staging and production environments as a manual process can beimproved with the process below – the same code can serve up alocked down robots.txt in staging or internal URLs, and allowindexing in production.
  12. 12. by Naji El KotobDemo (contd) You’ll also need to make sure thatrunAllManagedModulesForAllRequests istrue in web.config as normally text filesbypass the ASP.NET pipeline:<system.webServer><modules runAllManagedModulesForAllRequests="true"></modules>...</system.webServer>
  13. 13. by Naji El KotobDemo (contd)routes.MapRoute("Robots.txt","robots.txt",new{controller = "Robots",action = "RobotsText"});
  14. 14. by Naji El KotobDemo (contd)The create a new controller called “RobotsController” with a single action “RobotsText”. All requests to /robots.txt will gohere:public class RobotsController : Controller{public FileContentResult RobotsText(){var contentBuilder = new StringBuilder();contentBuilder.AppendLine("User-agent: *");// change this to however you want to detect a production URLvar isProductionUrl = Request.Url != null &&!Request.Url.ToString().ToLowerInvariant().Contains("elasticbeanstalk");if (isProductionUrl){contentBuilder.AppendLine("Disallow: /elmah.axd");contentBuilder.AppendLine("Disallow: /admin");contentBuilder.AppendLine("Disallow: /Admin");contentBuilder.AppendLine("Sitemap: http://www.mysite.com/sitemap.xml");}else{contentBuilder.AppendLine("Disallow: /");}return File(Encoding.UTF8.GetBytes(contentBuilder.ToString()), "text/plain");}}
  15. 15. by Naji El KotobSitemap
  16. 16. by Naji El KotobSitemap Sitemap (XML) is a list of pages on your Website.It is easiest way to give Google, and other searchengines, information about your site and mayspeed up Google’s crawlers finding you. There are many advantages of submitting aSitemap to Google, especially if your site is newand doesn’t have many backlinks.Source | http://diywebjem.com/webdefinitions.html
  17. 17. by Naji El KotobSitemap Format<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>http://www.example.com/</loc><lastmod>2005-01-01</lastmod><changefreq>monthly</changefreq><priority>0.8</priority></url></urlset>Source | http://www.sitemaps.org/protocol.html
  18. 18. by Naji El KotobSitemap Generators There are hundreds of online tools thatcan be used to create an XML sitemapthat can be submitted to Google, Bing,Yahoo and other◦ E.g. http://www.xml-sitemaps.com/ http://www.sitemapdoc.com/ And more…
  19. 19. by Naji El KotobReferences http://geekswithblogs.net/jjulian/archive/2012/04/25/adding-robots.txt-to-your-asp.net-mvc-3-applications.aspx http://tools.seobook.com/robots-txt/ http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/
  20. 20. by Naji El KotobQnA?
  21. 21. by Naji El KotobThank You  Please send your feedback andsuggestions to:◦ naji [@] DotNETHeroes.com Join us atwww.facebook.com/DotNETHeroes

×