This document discusses ways for bots to better locate actual content on webpages. It proposes using CSS with a "robots" media type to explicitly specify the main content for bots, which would give more granular control over what is shown to bots and help machines better understand and analyze webpage content. This could potentially lead to improved search engines that have a better understanding of content on the open web.

1. Assisting bots in crawling the web better Hello, I’m @KahWee

2. To index or not to index Robots looks at robots.txt to check which are the paths allowed and disallowed for indexing User-Agent: * Disallow: /folder1/ Allow: /folder1/myfile.html Sometimes only portions of the page contain actual content

3. Machines want to be smarter

4. Where’s the actual content?

5. We all know it’s here

6. The bot thinks it’s here

7. How to locate the actual content? Search engines tries to make an intelligent guess where the main content is. This approach is implicit.

8. Specifying actual content explicitly Google uses this in HTML using a comment. <!--googleon: all--> <div id="actual"> Loremipsum </div> <!--googleoff: all> This is proprietary.

9. Proposal to specify using CSS <link rel="stylesheet" media="robots" href="robots.css"> Or @media robots { * { display: none } #actual { display: block } }

10. Benefits Using CSS, we have more granular control of what to show to robots and what not to explicitly. Simple enough to enable development of better content-analysis-based applications. Potentially translate to better search, letting machines understand the open web better.