SlideShare a Scribd company logo
1 of 8
Download to read offline
Robots.txt
Introduction
• The robots exclusion protocol (REP), or robots.txt is a standard used
by websites to communicate with web crawlers and other web
robots.
• It is a text file webmasters create to instruct search engine robots
how to crawl and index pages on their website.
History
• The standard was proposed by Martijn Koster, when working for Nexor in February,
1994 on the www-talk mailing list, the main communication channel for WWW-related
activities at the time.
• Charles Stross claims to have provoked Koster to suggest robots.txt, after he wrote a
badly-behaved web crawler that caused an inadvertent denial of service attack on
Koster's server.
• The /robots.txt is a de-facto standard, and is not owned by any
standards body. There are two historical descriptions:
• the original 1994 A Standard for Robot Exclusion document.
• a 1997 Internet Draft specification A Method for Web Robots Control
Examples
• Block all web crawlers from all content
User-agent: *
Disallow: /
• Block a specific web crawler from a specific folder
User-agent: Googlebot
Disallow: /no-google/
• Block a specific web crawler from a specific web page
User-agent: Googlebot
Disallow: /no-google/blocked-page.html
The "User-agent: *"
means this section
applies to all robots. The
"Disallow: /" tells the
robot that it should not
visit any pages on the
site.
* - which is a wildcard
that represents any
sequence of
• To exclude all robots from part of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
• Sitemap Parameter
User-agent: *
Disallow:
Sitemap: http://www.example.com/none-standard-location/sitemap.xml
• Crawl-delay directive
Several major crawlers support a Crawl-delay parameter, set to the number of seconds to wait
between successive requests to the same server
User-agent: *
Crawl-delay:
• Allow directive
If one wants to allow single files inside an otherwise disallowed directory, it is necessary to place
the Allow directive(s) first, followed by the Disallow.
Allow: /directory1/myfile.html
Disallow: /directory1/
• Host
Some crawlers (Yandex, Google) support a Host directive, allowing websites with multiple mirrors
to specify their preferred domain.[26]
Host: www.example.com
Important Rules
• In most cases, meta robots with parameters "noindex, follow" should be employed as a
way to restrict crawling or indexation.
• It is important to note that malicious crawlers are likely to completely ignore robots.txt
and as such, this protocol does not make a good security mechanism.
• Only one "Disallow:" line is allowed for each URL.
• Each subdomain on a root domain uses separate robots.txt files.
• The filename of robots.txt is case sensitive. Use "robots.txt", not "Robots.TXT.“
• Spacing is not an accepted way to separate query parameters. For example,
"/category/ /product page" would not be honored by robots.txt.
Thank You

More Related Content

What's hot

Eye catching HTML BASICS tips: Learn easily
Eye catching HTML BASICS tips: Learn easilyEye catching HTML BASICS tips: Learn easily
Eye catching HTML BASICS tips: Learn easilyshabab shihan
 
Off Page SEO Strategies
Off Page SEO StrategiesOff Page SEO Strategies
Off Page SEO StrategiesShimanto Arif
 
Intro to HTML & CSS
Intro to HTML & CSSIntro to HTML & CSS
Intro to HTML & CSSSyed Sami
 
Html character entities
Html character entitiesHtml character entities
Html character entitiesnobel mujuji
 
Search Engine Optimization
Search Engine OptimizationSearch Engine Optimization
Search Engine OptimizationShashank Varun
 
Introduction to html
Introduction to htmlIntroduction to html
Introduction to htmlveena parihar
 
Html interview-questions-and-answers
Html interview-questions-and-answersHtml interview-questions-and-answers
Html interview-questions-and-answersMohitKumar1985
 
The Yoast SEO Plugin for WordPress
The Yoast SEO Plugin for WordPressThe Yoast SEO Plugin for WordPress
The Yoast SEO Plugin for WordPressBobWP.com
 
How Does Organic SEO Work?
How Does Organic SEO Work?How Does Organic SEO Work?
How Does Organic SEO Work?ITS-SEO.com
 

What's hot (20)

Eye catching HTML BASICS tips: Learn easily
Eye catching HTML BASICS tips: Learn easilyEye catching HTML BASICS tips: Learn easily
Eye catching HTML BASICS tips: Learn easily
 
Html basics
Html basicsHtml basics
Html basics
 
SEO PPT
SEO PPTSEO PPT
SEO PPT
 
Off Page SEO Strategies
Off Page SEO StrategiesOff Page SEO Strategies
Off Page SEO Strategies
 
How to learn HTML in 10 Days
How to learn HTML in 10 DaysHow to learn HTML in 10 Days
How to learn HTML in 10 Days
 
Intro to HTML & CSS
Intro to HTML & CSSIntro to HTML & CSS
Intro to HTML & CSS
 
Html character entities
Html character entitiesHtml character entities
Html character entities
 
Google Keyword Planner Tool
Google Keyword Planner ToolGoogle Keyword Planner Tool
Google Keyword Planner Tool
 
Search Engine Optimization
Search Engine OptimizationSearch Engine Optimization
Search Engine Optimization
 
Introduction to html
Introduction to htmlIntroduction to html
Introduction to html
 
Html interview-questions-and-answers
Html interview-questions-and-answersHtml interview-questions-and-answers
Html interview-questions-and-answers
 
Web Development using HTML & CSS
Web Development using HTML & CSSWeb Development using HTML & CSS
Web Development using HTML & CSS
 
On page seo PDF
On page seo PDFOn page seo PDF
On page seo PDF
 
Html Slide Part-1
Html Slide Part-1Html Slide Part-1
Html Slide Part-1
 
Html for beginners
Html for beginnersHtml for beginners
Html for beginners
 
The Yoast SEO Plugin for WordPress
The Yoast SEO Plugin for WordPressThe Yoast SEO Plugin for WordPress
The Yoast SEO Plugin for WordPress
 
Off page seo
Off page seoOff page seo
Off page seo
 
On-Page SEO
On-Page SEOOn-Page SEO
On-Page SEO
 
Html presentation
Html presentationHtml presentation
Html presentation
 
How Does Organic SEO Work?
How Does Organic SEO Work?How Does Organic SEO Work?
How Does Organic SEO Work?
 

Similar to Robots.txt: The Standard for Web Crawlers

Robots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeRobots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeLets Get Digital
 
Difference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagDifference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagParidhi Infotech
 
05.m3 cms list-ofwebserver
05.m3 cms list-ofwebserver05.m3 cms list-ofwebserver
05.m3 cms list-ofwebservertarensi
 
Controlling crawler for better Indexation and Ranking
Controlling crawler for better Indexation and RankingControlling crawler for better Indexation and Ranking
Controlling crawler for better Indexation and RankingRajesh Magar
 
Robots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationRobots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationJahid Hasan
 
Introduction to "robots.txt
Introduction to "robots.txtIntroduction to "robots.txt
Introduction to "robots.txtIshan Mishra
 
Search engine and web crawler
Search engine and web crawlerSearch engine and web crawler
Search engine and web crawlervinay arora
 
CNIT 129S - Ch 3: Web Application Technologies
CNIT 129S - Ch 3: Web Application TechnologiesCNIT 129S - Ch 3: Web Application Technologies
CNIT 129S - Ch 3: Web Application TechnologiesSam Bowne
 
Open Source Libraries for.NET developers
Open Source Libraries for.NET developersOpen Source Libraries for.NET developers
Open Source Libraries for.NET developersAndrei Marukovich
 
DNN Summit: Robots.txt & Multi-Site DNN Instances
DNN Summit: Robots.txt & Multi-Site DNN InstancesDNN Summit: Robots.txt & Multi-Site DNN Instances
DNN Summit: Robots.txt & Multi-Site DNN InstancesWill Strohl
 
Training Webinar: Enterprise application performance with server push technol...
Training Webinar: Enterprise application performance with server push technol...Training Webinar: Enterprise application performance with server push technol...
Training Webinar: Enterprise application performance with server push technol...OutSystems
 

Similar to Robots.txt: The Standard for Web Crawlers (20)

Robots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeRobots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can See
 
Burp suite
Burp suiteBurp suite
Burp suite
 
Difference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagDifference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tag
 
webcrawler.pptx
webcrawler.pptxwebcrawler.pptx
webcrawler.pptx
 
Robots.txt
Robots.txtRobots.txt
Robots.txt
 
05.m3 cms list-ofwebserver
05.m3 cms list-ofwebserver05.m3 cms list-ofwebserver
05.m3 cms list-ofwebserver
 
Controlling crawler for better Indexation and Ranking
Controlling crawler for better Indexation and RankingControlling crawler for better Indexation and Ranking
Controlling crawler for better Indexation and Ranking
 
Robots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationRobots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml Creation
 
Webcrawler
Webcrawler Webcrawler
Webcrawler
 
Introduction to "robots.txt
Introduction to "robots.txtIntroduction to "robots.txt
Introduction to "robots.txt
 
Web crawler
Web crawlerWeb crawler
Web crawler
 
Search engine and web crawler
Search engine and web crawlerSearch engine and web crawler
Search engine and web crawler
 
Web crawler
Web crawlerWeb crawler
Web crawler
 
CNIT 129S - Ch 3: Web Application Technologies
CNIT 129S - Ch 3: Web Application TechnologiesCNIT 129S - Ch 3: Web Application Technologies
CNIT 129S - Ch 3: Web Application Technologies
 
WebCrawler
WebCrawlerWebCrawler
WebCrawler
 
Open Source Libraries for.NET developers
Open Source Libraries for.NET developersOpen Source Libraries for.NET developers
Open Source Libraries for.NET developers
 
Web Crawlers
Web CrawlersWeb Crawlers
Web Crawlers
 
DNN Summit: Robots.txt & Multi-Site DNN Instances
DNN Summit: Robots.txt & Multi-Site DNN InstancesDNN Summit: Robots.txt & Multi-Site DNN Instances
DNN Summit: Robots.txt & Multi-Site DNN Instances
 
Training Webinar: Enterprise application performance with server push technol...
Training Webinar: Enterprise application performance with server push technol...Training Webinar: Enterprise application performance with server push technol...
Training Webinar: Enterprise application performance with server push technol...
 
Ruby on-rails-security
Ruby on-rails-securityRuby on-rails-security
Ruby on-rails-security
 

Recently uploaded

Gen Z and Millennial Debit Card Use Survey.pdf
Gen Z and Millennial Debit Card Use Survey.pdfGen Z and Millennial Debit Card Use Survey.pdf
Gen Z and Millennial Debit Card Use Survey.pdfMedia Logic
 
Dave Cousin TW-BERT Good for Users, Good for SEOsBrighton SEO Deck
Dave Cousin TW-BERT Good for Users, Good for SEOsBrighton SEO DeckDave Cousin TW-BERT Good for Users, Good for SEOsBrighton SEO Deck
Dave Cousin TW-BERT Good for Users, Good for SEOsBrighton SEO DeckOban International
 
The Fall of Social Media Marketing :- Global brand magazine
The Fall of Social Media Marketing :- Global brand magazineThe Fall of Social Media Marketing :- Global brand magazine
The Fall of Social Media Marketing :- Global brand magazineglobalbrandmagazines1
 
Content Marketing: How To Find The True Value Of Your Marketing Funnel
Content Marketing: How To Find The True Value Of Your Marketing FunnelContent Marketing: How To Find The True Value Of Your Marketing Funnel
Content Marketing: How To Find The True Value Of Your Marketing FunnelSearch Engine Journal
 
Lesotho-Botswana Water Project Brand Manual developed with new logo
Lesotho-Botswana Water Project Brand Manual developed with  new logoLesotho-Botswana Water Project Brand Manual developed with  new logo
Lesotho-Botswana Water Project Brand Manual developed with new logonelaohaimbodi
 
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...Social Samosa
 
Catálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaCatálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaEsteller
 
Llanai Buyer Persona & Segmentation Strategy
Llanai Buyer Persona & Segmentation StrategyLlanai Buyer Persona & Segmentation Strategy
Llanai Buyer Persona & Segmentation StrategyMarianna Nakou
 
Catálogo HYD 2024 gama completa Hydrapak
Catálogo HYD 2024 gama completa HydrapakCatálogo HYD 2024 gama completa Hydrapak
Catálogo HYD 2024 gama completa HydrapakEsteller
 
Professional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxProfessional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxSahil Srivastava
 
2024 WTF - what's working in mobile user acquisition
2024 WTF - what's working in mobile user acquisition2024 WTF - what's working in mobile user acquisition
2024 WTF - what's working in mobile user acquisitionJohn Koetsier
 
Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...VereigenMedia1
 
Paul Russell Confidential Resume for Fahlo.pdf
Paul Russell Confidential Resume for Fahlo.pdfPaul Russell Confidential Resume for Fahlo.pdf
Paul Russell Confidential Resume for Fahlo.pdfpaul8402
 
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Nugget Global
 
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptxv100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptxjeromecuyos1
 
TAM Sports IPL 17 Advertising Report- M01 - M23
TAM Sports IPL 17 Advertising Report- M01 - M23TAM Sports IPL 17 Advertising Report- M01 - M23
TAM Sports IPL 17 Advertising Report- M01 - M23Social Samosa
 
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Associazione Digital Days
 
Fritschi Collection 2022/23 EN gama completa
Fritschi Collection 2022/23 EN gama completaFritschi Collection 2022/23 EN gama completa
Fritschi Collection 2022/23 EN gama completaEsteller
 
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...Associazione Digital Days
 
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Valters Lauzums
 

Recently uploaded (20)

Gen Z and Millennial Debit Card Use Survey.pdf
Gen Z and Millennial Debit Card Use Survey.pdfGen Z and Millennial Debit Card Use Survey.pdf
Gen Z and Millennial Debit Card Use Survey.pdf
 
Dave Cousin TW-BERT Good for Users, Good for SEOsBrighton SEO Deck
Dave Cousin TW-BERT Good for Users, Good for SEOsBrighton SEO DeckDave Cousin TW-BERT Good for Users, Good for SEOsBrighton SEO Deck
Dave Cousin TW-BERT Good for Users, Good for SEOsBrighton SEO Deck
 
The Fall of Social Media Marketing :- Global brand magazine
The Fall of Social Media Marketing :- Global brand magazineThe Fall of Social Media Marketing :- Global brand magazine
The Fall of Social Media Marketing :- Global brand magazine
 
Content Marketing: How To Find The True Value Of Your Marketing Funnel
Content Marketing: How To Find The True Value Of Your Marketing FunnelContent Marketing: How To Find The True Value Of Your Marketing Funnel
Content Marketing: How To Find The True Value Of Your Marketing Funnel
 
Lesotho-Botswana Water Project Brand Manual developed with new logo
Lesotho-Botswana Water Project Brand Manual developed with  new logoLesotho-Botswana Water Project Brand Manual developed with  new logo
Lesotho-Botswana Water Project Brand Manual developed with new logo
 
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
 
Catálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaCatálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compelta
 
Llanai Buyer Persona & Segmentation Strategy
Llanai Buyer Persona & Segmentation StrategyLlanai Buyer Persona & Segmentation Strategy
Llanai Buyer Persona & Segmentation Strategy
 
Catálogo HYD 2024 gama completa Hydrapak
Catálogo HYD 2024 gama completa HydrapakCatálogo HYD 2024 gama completa Hydrapak
Catálogo HYD 2024 gama completa Hydrapak
 
Professional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxProfessional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptx
 
2024 WTF - what's working in mobile user acquisition
2024 WTF - what's working in mobile user acquisition2024 WTF - what's working in mobile user acquisition
2024 WTF - what's working in mobile user acquisition
 
Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...
 
Paul Russell Confidential Resume for Fahlo.pdf
Paul Russell Confidential Resume for Fahlo.pdfPaul Russell Confidential Resume for Fahlo.pdf
Paul Russell Confidential Resume for Fahlo.pdf
 
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
 
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptxv100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
 
TAM Sports IPL 17 Advertising Report- M01 - M23
TAM Sports IPL 17 Advertising Report- M01 - M23TAM Sports IPL 17 Advertising Report- M01 - M23
TAM Sports IPL 17 Advertising Report- M01 - M23
 
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
 
Fritschi Collection 2022/23 EN gama completa
Fritschi Collection 2022/23 EN gama completaFritschi Collection 2022/23 EN gama completa
Fritschi Collection 2022/23 EN gama completa
 
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
 
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
 

Robots.txt: The Standard for Web Crawlers

  • 2. Introduction • The robots exclusion protocol (REP), or robots.txt is a standard used by websites to communicate with web crawlers and other web robots. • It is a text file webmasters create to instruct search engine robots how to crawl and index pages on their website.
  • 3. History • The standard was proposed by Martijn Koster, when working for Nexor in February, 1994 on the www-talk mailing list, the main communication channel for WWW-related activities at the time. • Charles Stross claims to have provoked Koster to suggest robots.txt, after he wrote a badly-behaved web crawler that caused an inadvertent denial of service attack on Koster's server. • The /robots.txt is a de-facto standard, and is not owned by any standards body. There are two historical descriptions: • the original 1994 A Standard for Robot Exclusion document. • a 1997 Internet Draft specification A Method for Web Robots Control
  • 4. Examples • Block all web crawlers from all content User-agent: * Disallow: / • Block a specific web crawler from a specific folder User-agent: Googlebot Disallow: /no-google/ • Block a specific web crawler from a specific web page User-agent: Googlebot Disallow: /no-google/blocked-page.html The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. * - which is a wildcard that represents any sequence of
  • 5. • To exclude all robots from part of the server User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /junk/ • Sitemap Parameter User-agent: * Disallow: Sitemap: http://www.example.com/none-standard-location/sitemap.xml • Crawl-delay directive Several major crawlers support a Crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server User-agent: * Crawl-delay:
  • 6. • Allow directive If one wants to allow single files inside an otherwise disallowed directory, it is necessary to place the Allow directive(s) first, followed by the Disallow. Allow: /directory1/myfile.html Disallow: /directory1/ • Host Some crawlers (Yandex, Google) support a Host directive, allowing websites with multiple mirrors to specify their preferred domain.[26] Host: www.example.com
  • 7. Important Rules • In most cases, meta robots with parameters "noindex, follow" should be employed as a way to restrict crawling or indexation. • It is important to note that malicious crawlers are likely to completely ignore robots.txt and as such, this protocol does not make a good security mechanism. • Only one "Disallow:" line is allowed for each URL. • Each subdomain on a root domain uses separate robots.txt files. • The filename of robots.txt is case sensitive. Use "robots.txt", not "Robots.TXT.“ • Spacing is not an accepted way to separate query parameters. For example, "/category/ /product page" would not be honored by robots.txt.