Your SlideShare is downloading. ×
0
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Robots and-sitemap - Version 1.0.1
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Robots and-sitemap - Version 1.0.1

789

Published on

Introduction to robots.txt and sitemap.xml files.

Introduction to robots.txt and sitemap.xml files.

Published in: Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
789
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Robots and SitemapBy Naji El Kotob.Information Technology Consultant_______________________________naji [@] DotNETHeroes.comReview 1.0.1 - 13.Business Intelligence Group BIG SARL
  • 2. by Naji El KotobOutlines SEO Robots.txt Sitemap.xml Q&A
  • 3. by Naji El KotobSEO
  • 4. by Naji El KotobWhat is SEO SEO is short for search engine optimization orsearch engine optimizer. Search engine optimization is a methodology ofstrategies, techniques and tactics used toincrease the amount of visitors to a website byobtaining a high-ranking placement in the searchresults page of a search engine (SERP)-- including Google, Bing, Yahoo and other search engines.Source | http://www.webopedia.com/TERM/S/SEO.html
  • 5. by Naji El KotobRobots.txt
  • 6. by Naji El KotobRobots.txt Robots.txt is the file that Search Enginesuse to see what they should index. This file and site maps help make yoursite easier to navigate by the bots andlet them know what is legal and what youwould rather not have the published intheir engines.
  • 7. by Naji El KotobRobots.txt
  • 8. by Naji El KotobRobots.txt SyntaxUser-agent: *Disallow:Sitemap: http://dotnetheroes.com/sitemap.xml
  • 9. by Naji El KotobRobots.txt Generatorhttp://tools.seobook.com/robots-txt/generator/
  • 10. by Naji El KotobInteractive Demo
  • 11. by Naji El KotobDemo Design and Create a Dynamic RobotsSource | http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/Adding a Dynamic Robots.txt file to an ASP.NET MVC siteRobots.txt is required to allow search engines to properly index yoursite, and more importantly not index it.If you have a public-facing staging or preliminary site that you don’twant to show up in Google results, you need to make sure that itreturns the correct robots.txt with theDisallow: /line to prevent indexing. However, manually adding robots.txt filesto staging and production environments as a manual process can beimproved with the process below – the same code can serve up alocked down robots.txt in staging or internal URLs, and allowindexing in production.
  • 12. by Naji El KotobDemo (contd) You’ll also need to make sure thatrunAllManagedModulesForAllRequests istrue in web.config as normally text filesbypass the ASP.NET pipeline:<system.webServer><modules runAllManagedModulesForAllRequests="true"></modules>...</system.webServer>
  • 13. by Naji El KotobDemo (contd)routes.MapRoute("Robots.txt","robots.txt",new{controller = "Robots",action = "RobotsText"});
  • 14. by Naji El KotobDemo (contd)The create a new controller called “RobotsController” with a single action “RobotsText”. All requests to /robots.txt will gohere:public class RobotsController : Controller{public FileContentResult RobotsText(){var contentBuilder = new StringBuilder();contentBuilder.AppendLine("User-agent: *");// change this to however you want to detect a production URLvar isProductionUrl = Request.Url != null &&!Request.Url.ToString().ToLowerInvariant().Contains("elasticbeanstalk");if (isProductionUrl){contentBuilder.AppendLine("Disallow: /elmah.axd");contentBuilder.AppendLine("Disallow: /admin");contentBuilder.AppendLine("Disallow: /Admin");contentBuilder.AppendLine("Sitemap: http://www.mysite.com/sitemap.xml");}else{contentBuilder.AppendLine("Disallow: /");}return File(Encoding.UTF8.GetBytes(contentBuilder.ToString()), "text/plain");}}
  • 15. by Naji El KotobSitemap
  • 16. by Naji El KotobSitemap Sitemap (XML) is a list of pages on your Website.It is easiest way to give Google, and other searchengines, information about your site and mayspeed up Google’s crawlers finding you. There are many advantages of submitting aSitemap to Google, especially if your site is newand doesn’t have many backlinks.Source | http://diywebjem.com/webdefinitions.html
  • 17. by Naji El KotobSitemap Format<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>http://www.example.com/</loc><lastmod>2005-01-01</lastmod><changefreq>monthly</changefreq><priority>0.8</priority></url></urlset>Source | http://www.sitemaps.org/protocol.html
  • 18. by Naji El KotobSitemap Generators There are hundreds of online tools thatcan be used to create an XML sitemapthat can be submitted to Google, Bing,Yahoo and other◦ E.g. http://www.xml-sitemaps.com/ http://www.sitemapdoc.com/ And more…
  • 19. by Naji El KotobReferences http://geekswithblogs.net/jjulian/archive/2012/04/25/adding-robots.txt-to-your-asp.net-mvc-3-applications.aspx http://tools.seobook.com/robots-txt/ http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/
  • 20. by Naji El KotobQnA?
  • 21. by Naji El KotobThank You  Please send your feedback andsuggestions to:◦ naji [@] DotNETHeroes.com Join us atwww.facebook.com/DotNETHeroes

×