A search engine crawler must be able to access websites to read pages and index them so they appear in search results. Websites should have an architecture that is easy for users to understand and load quickly on all devices including mobile to enhance the user experience and receive a recommendation from search engines.