Rendering: Or why your perfectly optimized content doesn't rank

1. Rendering: Or why your perfectly optimized content doesn't rank

2. Websites are made of code. Code is written in languages. 3 languages are the foundation of websites.

3. HTML Content CSS layout, design, & visual effects JavaScript web version of programming code

4. How it is then so many of our optimization efforts slips between the cracks?

5. 75% of product pages for an ecommerce site were not indexed in a case study by @TomkeRudzki

6. Rendering

7. “When we crawl your page with Googlebot, we go fetch the content and then we give it to chrome. Then Chrome runs all the scripts. It loads additional content. Once everything's loaded we take a snapshot of the page and that's the content that actually gets indexed.” - Erik Hendriks, Software Engineer at Google Rendering (WMConf MTV '19)

8. Initial HTML Rendered HTML Rendering

9. The Mechanics of Google's Web Rendering Service

10. Google uses Chromium as its web rendering service.

12. Web Rendering Service (WRS) Blink Browser Engine Ignition TurboFan Liftoff Display backend GoogleMagic Chromium Headless Browser V8 Rendering Engine

13. Web Rendering Service (WRS) Blink Browser Engine Ignition TurboFan Liftoff Display backend GoogleMagic Chromium Headless Browser V8 Rendering Engine Chromium Headless Browser

14. Chromium, headless browser ● Headless means that there is no GUI (visual representation) ● Used to load web pages and extract metadata ● reading from and writing to the DOM ● observing network events ● capturing screenshots ● inspecting worker scripts ● recording Chrome Traces

15. Web Rendering Service (WRS) Blink Browser Engine Ignition TurboFan Liftoff Display backend GoogleMagic Chromium Headless Browser V8 Rendering Engine Blink Browser Engine

16. Blink, browser engine ● Allows for querying and manipulating the rendering engine settings (ex: mobile vs. desktop) ● responsible for 2 major elements: 1. Memory heap: stores the result of script execution (Memory Heap results are added to DOM.) 2. Call stack: queue of sequential next steps (Each entry in the call stack is called a Stack Frame.) Local storage and Session storage are key-value pairs that can store any JS objects and functions in the browser These keys are a weak point in your rendering offense against a stateless Googlebot.

17. Web Rendering Service (WRS) Blink Browser Engine Ignition TurboFan Liftoff Display backend GoogleMagic Chromium Headless Browser V8 Rendering EngineV8 Rendering Engine

18. V8, JavaScript engine Each thread will runs through a process of: 1. Loading 2. Parsing 3. Compiling 4. Executing ● open-source JavaScript engine and WebAssembly engine ● developed by Google & The Chromium Project ● Use in Node.js, Google Chrome, and Chromium web browsers

19. Let’s look at how Google renders.

20. 1. A URL is pulled from the crawl queue Crawl Queue Crawler

21. 2. The URI is requested using the Googlebot user-agent Crawl Queue Crawler URI

22. 3. Read HTTP response and headers Crawl Queue Crawler URI Q. Does the thing I asked for exist? A. HTTP Status Codes Q. Anything I should know before looking at this? A. Cache-Control, and Directives

23. 4. Download and parse the HTML Crawl Queue Crawler Initial HTML Use View Page Source to see the initial HTML Crawled DOM

24. 5. The Crawled DOM is passed to the index processor Crawl Queue Crawler Resources Crawled DOM Index Processor Initial HTML

25. 6. The processor extracts resources links from the crawled DOM and places them in the crawl queue Crawl Queue Crawler Resources Crawled DOM Index Processor Initial HTML

26. 7. The index processor passes valuable content to the index Crawl Queue Crawler Crawled DOM Index Processor Initial HTML Index Resources

27. Crawler 8. Once resources are crawled, the page queues for rendering Crawl Queue Scripts from Crawled DOM

28. Crawler Crawled DOM Render 9. When resources become available, the URI moves from the render queue to the renderer Crawl Queue Resources

29. Crawler Crawled DOM Render Rendered DOM 9. WRS assembles the page using the Crawled DOM and resources Crawl Queue Resources Use Inspect to see rendered HTML

30. Crawler Crawled DOM Render Rendered DOM 10. Renderer passes the rendered HTML to the index processor Crawl Queue Resources Index Processor

31. Crawler Crawled DOM Render Rendered DOM 11. The processor extracts resources links from the Rendered DOM and places them in the crawl queue Crawl Queue Crawled DOM Resources Index Processor Rendered DOM Resources

32. Crawler Crawled DOM Render Rendered DOM 12. The index processor passes valuable rendered content to the index Crawl Queue Crawled DOM Resources Index Processor Rendered DOM Resources Index

33. Great! Now let's break it!

34. 1. A URL is pulled from the crawl queue Crawl Queue Crawler Create a SPA that doesn't update URLs with history.pushState()

35. 2. The URI is requested using the Googlebot user-agent Crawl Queue Crawler URI Forget to pay your prerendering service

36. 3. Read HTTP response and headers Crawl Queue Crawler URI Use whichever status code whenever

37. 3. Read HTTP response and headers Crawl Queue Crawler URI Contradictory signals

38. 4. Download and parse the HTML Crawl Queue Crawler Initial HTML Store content in the ShadowDOM <!doctype html> <head> <script type="text/javascript" src="/main.js?ver=1.1.0"> </script> </head> <body> <app-root></app-root> </body> </html>

39. 5. The Crawled DOM is passed to the index processor Crawl Queue Crawler Resources Crawled DOM Index Processor Initial HTML

40. 6. The processor extracts resources links from the crawled DOM and places them in the crawl queue Crawl Queue Crawler Resources Crawled DOM Index Processor Initial HTML <a href="/good link">Will be crawled</a> <span onclick="changePage('bad-link')">Not crawled</span> <a onclick="changePage('bad-link')">Not crawled</a> <a href="/good-link" onclick="changePage ('good-link')">Will be crawled</a> Undiscoverable links

41. 7. The index processor passes valuable content to the index Crawl Queue Crawler Crawled DOM Index Processor Initial HTML Index Resources Don't answer user intent with the Crawled DOM

42. Crawler 8. Once resources are crawled, the page queues for rendering Crawl Queue Scripts from Crawled DOM Blocked Resources / CORS Errors

43. Crawler Crawled DOM Render Rendered DOM 9. WRS assembles the page using the Crawled DOM and resources Crawl Queue Resources Use Inspect to see rendered HTML iframes, SPOFs in <head> / Render blocking resources

44. 3 types of rendering-blocking resources A <script> tag that: ● Is in the <head> of the document. ● Does not have a defer attribute. ● Does not have an async attribute. A <link rel="import"> tag that: ● Does not have an async attribute. A <link rel="import"> tag that: ● Does not have an async attribute.

45. Crawler Crawled DOM Render Rendered DOM 9. WRS assembles the page using the Crawled DOM and resources Crawl Queue Resources Use Inspect to see rendered HTML Look for knock-on effects

46. Crawler Crawled DOM Render Rendered DOM 9. WRS assembles the page using the Crawled DOM and resources Crawl Queue Resources Use Inspect to see rendered HTML Don't prioritize resources

47. Crawler Crawled DOM Render Rendered DOM 9. WRS assembles the page using the Crawled DOM and resources Crawl Queue Resources Use Inspect to see rendered HTML Dependencies in the critical rendering path

48. JQuery script1.js script2.js script3.js script4.js

49. JQuery script1.js script2.js script3.js script4.js script1.js {Product name} {Product sku} {Product price}

50. JQuery script1.js script2.js script3.js script4.js {Product name} {Product sku} {Product price} {Product category} {Product availability} {Shipping times} {Breadcrumbs} {Product description} {Product images} {Product specs}

53. Crawler Crawled DOM Render Rendered DOM 9. WRS assembles the page using the Crawled DOM and resources Crawl Queue Resources Use Inspect to see rendered HTML Rely on Client-Side Rendering <!doctype html> <head> <script type="text/javascript" src="/main.js?ver=1.1.0"> </script> </head> <body> <app-root></app-root> </body> </html>

54. <!doctype html> <head> <script type="text/javascript" src="/main.js?ver=1.1.0"> </script> </head> <body> <app-root></app-root> </body> </html> script1.js script2.js script3.js script4.js {Product name} {Product sku} {Product price} {Product category} {Product availability} {Shipping times} {Breadcrumbs} {Product description} {Product images} {Product specs} Each script has a probability to fail.

55. JQuery script1.js script2.js script4.js {Product name} {Product sku} {Product price} {Product category} {Product availability} {Shipping times} {Breadcrumbs} script3.js {Product description} {Product images} {Product specs}

56. <rel="buyproduct.js"> = 99.5% success rate

57. Crawler Crawled DOM Render Rendered DOM 9. WRS assembles the page using the Crawled DOM and resources Crawl Queue Resources Use Inspect to see rendered HTML Ship every script on every request

58. Everyone has to wait while JS does it’s thing.

59. Crawler Crawled DOM Render Rendered DOM 9. WRS assembles the page using the Crawled DOM and resources Crawl Queue Resources Use Inspect to see rendered HTML Ignore performance

60. ● Google will limit CPU consumption ● Google will interrupt scripts ● Excessive CPU consumption negatively impacts indexing

61. Crawler Crawled DOM Render Rendered DOM 10. Renderer passes the rendered HTML to the index processor Crawl Queue Resources Index Processor Require permissions

62. Crawler Crawled DOM Render Rendered DOM 11. The processor extracts resources links from the Rendered DOM and places them in the crawl queue Crawl Queue Crawled DOM Resources Index Processor Rendered DOM Resources Wreck your crawl budget with indexable API endpoints

63. Crawler Crawled DOM Render Rendered DOM 12. The index processor passes valuable rendered content to the index Crawl Queue Crawled DOM Resources Index Processor Rendered DOM Resources Index

64. Let's make it better

65. ● Visibility Can I see and reproduce what ishappening? SEO for JavaScript frameworks requires

66. ● Visibility Can I see and reproduce what is happening? ● Diagnostics Can I measure what is happening? SEO for JavaScript frameworks requires

67. ● Visibility Can I see and reproduce what is happening? ● Diagnostics Can I measure what is happening? ● Iteration Can take meaningful steps to improve it? SEO for JavaScript frameworks requires

68. / / / '._ (_/) _.'/ |.''._ '--(o.o)--'_ .''.| _ / `;=/ " =;` _/ `__| ___/ |__/` (_|_)/ " ` " / |￣￣￣￣￣￣￣￣￣￣| WEAR A MASK AND BE KIND TO EACH OTHER |＿＿＿＿＿＿＿＿＿＿|

69. Thank you! Any questions? @Jammer_Volts