SMX Advanced Europe, June 2021 - With the advent of new technologies and the massive use of Javascript on the internet, search engines have started using Web Rendering Services to better understand the content of pages on the internet. What are the difficulties in building a WRS? Are tools we use every day replicating what search engines do? In this session, Giacomo will drive you on a discovery journey digging in some techy implementation details of a search engine like web rendering service building process, covering edge cases such as infinite scrolling, iframe, web component, and shadow DOM and how to approach them.

1. Challenges of building a search
engine like web rendering service
Giacomo Zecchini | Verve Search
@giacomozecchini

2. Hi, I’m Giacomo
Technical Director at
Technical background and
previous experiences in
development
Love: understanding how things work and Web
Performance
@giacomozecchini

3. @giacomozecchini
As always happens in all the
worst stories, everything started
with a website migration.

4. One of the largest brands in the
UK, with millions of pounds on the
line.
@giacomozecchini

5. The client was migrating platform,
moving from a Server Side
rendering website to a Client Side
Rendering one.
@giacomozecchini

6. ..and of course SSR was going to
be put in place.
@giacomozecchini

7. But.. two days before the migration
the client told us “SSR is not
working”.
@giacomozecchini

8. We were already at code freeze.
@giacomozecchini

9. We implemented a short term
solution* static rendering the site
and using the user-agent to serve
the right content.
*A sort of customized Rendertron script.
@giacomozecchini

10. After the migration we helped the
client to move to a medium-long
term solution (Prerender.io).
@giacomozecchini

11. Implementation is always harder
than it seems.
@giacomozecchini

12. My curiosity made me start to
research web rendering services.
@giacomozecchini

13. * Icons made by Freepik from www.flaticon.com
In the past the html was the most important thing to
download in order to access the content of a page
@giacomozecchini

14. Today, JavaScript is a big part of the web and
makes everything more complex
* Icons made by Freepik from www.flaticon.com @giacomozecchini

15. In the past we had a Crawling-Indexing process
Crawler Processing Index
URLs
Crawl
Queue
URL HTML
@giacomozecchini

16. Now, we’ve moved to a
crawling-rendering-indexing process
https://developers.google.com/search/docs/guides/javascript-seo-basics
Crawler Processing Index
Renderer
URLs
Crawl
Queue
URL HTML
Render Queue
@giacomozecchini

17. Google calls the rendering element WRS
Crawler Processing Index
Renderer
URLs
Crawl
Queue
URL HTML
Render Queue
WRS
@giacomozecchini

18. Martin Splitt’s TechSEO Boost 2019 talk
https://www.youtube.com/watch?v=Qxd_d9m9vzo
In his presentation, Martin
covered a lot of interesting
implementation details.
If you’re interested in
Google’s WRS, this is the
presentation to watch.
@giacomozecchini

19. These are three of the most important thing you
can get from a Web Rendering Service
DOM Tree
Render Tree +
Layout
Rendered
HTML
@giacomozecchini

20. DOM Tree & Render Tree
https://developers.google.com/web/fundamentals/performance/critical-rendering-path/render-tree-construction @giacomozecchini

21. Layout information
https://youtu.be/WjMSfTK1_SY?t=239
The layout information
helps to understand where
elements are positioned on
a page, their dimensions,
and their importance.
@giacomozecchini

22. The layout information is useful when it comes to:
- Understand the semantics of a page
- Check if a page is mobile friendly
- Find intrusive interstitials
- Understand above the fold content
https://youtu.be/WjMSfTK1_SY?t=239 @giacomozecchini

23. Crawler
Renderer
Render
queue
Fetch
Server
Cache
Server
Chrome
instances
Robots.txt
Server
DNS
Server
My toy web rendering service implementation
Fetchers
@giacomozecchini

24. Crawler
Renderer
Render
queue
Fetch
Server
Cache
Server
Chrome
instances
Robots.txt
Server
DNS
Server
It uses a first in first out queue
Fetchers
@giacomozecchini

25. Crawler
Renderer
Render
queue
Fetch
Server
Cache
Server
Chrome
instances
Robots.txt
Server
DNS
Server
...utilises Chrome DevTools Protocol
Fetchers
https://chromedevtools.github.io/devtools-protocol/ @giacomozecchini

26. Crawler
Renderer
Render
queue
Fetch
Server
Cache
Server
Chrome
instances
Robots.txt
Server
DNS
Server
I’m currently working on the fetch server
Fetchers
@giacomozecchini

27. Crawler
Renderer
Render
queue
Fetch
Server
Cache
Server
Chrome
instances
Robots.txt
Server
DNS
Server
It uses the cache when possible
Fetchers
@giacomozecchini

28. Crawler
Renderer
Render
queue
Fetch
Server
Cache
Server
Chrome
instances
Robots.txt
Server
DNS
Server
If the URL is not cached, the crawler fetches it
Fetchers
@giacomozecchini

29. The real problems start when you
have to make choices about the
actual rendering of pages!
@giacomozecchini

30. What about the viewport? Do you
want to limit the number of
fetches? Are you going to render a
page multiple times?
@giacomozecchini

31. Developing software is a matter
of choices and context.
@giacomozecchini

32. The same group of people in a
different context may end up
developing the same project in a
totally different way.
@giacomozecchini

33. You can’t replicate Google’s WRS
without having the same data they
have.
@giacomozecchini

34. This is where I began to realise
the case to build your own rather
than relying on tools.
@giacomozecchini

35. You can’t replicate Google’s WRS
but you can learn from it.
@giacomozecchini

36. Understanding Google's WRS behaviour
@giacomozecchini

37. Understanding Google's WRS behaviour
@giacomozecchini

38. If you need JavaScript console message data you
can use the Mobile Friendly test or Search Console
Live Test but be careful!
@giacomozecchini

39. Mobile-Friendly Test, Search
Console Live Test, AMP Test, and
Rich Results Test are using the
WRS infrastructure, but bypassing
cache, using shorter timeouts,
and few other differences.
@giacomozecchini

41. Hic sunt dracones / Here be Dragons
What follows is based on my own tests and
assumptions, results may be false positives.
Google can change implementation details at any
time without notice, explanation, or justification.
@giacomozecchini

42. How we’ll approach the edge cases
1. Define the edge case
2. Understand Google's WRS support and
behaviour (personal assumption)
3. Check for tools support and behaviour
4. Propose a solution
@giacomozecchini

43. Some of the tested tools
I did multiple tests for each edge case. @giacomozecchini

44. This is not an evaluation of those
tools, but just a comparison
between their results and those of
Google’s WRS.
@giacomozecchini

45. Edge Case #1
HTTPS/HTTP mixed content
@giacomozecchini

46. Mixed content occurs when initial
HTML is loaded over a secure
HTTPS connection, but other
resources are loaded over an
insecure HTTP connection.
https://youtu.be/WjMSfTK1_SY?t=239 @giacomozecchini

47. Website:
https://www.example.com
CSS:
http://www.example.com/style.css
@giacomozecchini

48. Chrome will automatically upgrade mixed content
from HTTP to HTTPS. If the fetch fails that asset
won’t be loaded.
@giacomozecchini

49. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
* this was not the case until recently
@giacomozecchini
Google’s WRS seems to behave like Chrome

50. Tools support
10% of tests showed a
different result
@giacomozecchini

51. Solution
When visiting an HTTPS website, upgrade the
URLs of assets from HTTP to HTTPS.
Using Chromium-based browsers you should
already have the right solution in place.
@giacomozecchini

52. Edge Case #2
Infinite scrolling / Lazy loading
@giacomozecchini

53. @giacomozecchini
SCROLL

54. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
VIEWPORT
Google starts the rendering
using a fixed viewport:
Mobile: 412 X 732
Desktop: 1024 x 1024
@giacomozecchini

55. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
VIEWPORT
PAGE
HEIGHT
Then calculate the new viewport:
Viewport = Page Height + pixels
The amount of additional pixels
depends on the page, it could be
thousands of pixels
@giacomozecchini

56. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
VIEWPORT
PAGE
HEIGHT
A bigger viewport
triggers Infinite
loading or lazy loading
events.
@giacomozecchini

57. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
10,000,000 px
This seems to be the maximum viewport height
@giacomozecchini

58. Tools support
95% of tests showed a
different result
* for very tall pages @giacomozecchini

59. Solution
Start with a fixed viewport
@giacomozecchini

60. Solution
Wait for an event:
onload
DOMContentLoaded
If you’re using puppeteer:
networkidle0
networkidle2
@giacomozecchini

61. Solution
VIEWPORT
PAGE
HEIGHT
Check for the page height
and compare it to the initial
viewport.
https://chromedevtools.github.io/devtools-protocol/tot/Page/#method-getLayoutMetrics @giacomozecchini

62. Solution
VIEWPORT
PAGE
HEIGHT
If the viewport is shorter
than the page
Viewport = Page Height + pixels
@giacomozecchini

63. Solution
The simplest solution is then
to wait for X seconds and
stop rendering or check
viewport and Page Height
again.
VIEWPORT
PAGE
HEIGHT
* for more complex solutions you can look at ongoing requests or an
event-based approach. @giacomozecchini

64. Edge Case #3
Content-visibility
@giacomozecchini

65. https://web.dev/content-visibility/
content-visibility is a
CSS property that enables
the browser to skip an
element's rendering.
@giacomozecchini

66. https://web.dev/content-visibility/
content-visibility is used together
with contain-intrinsic-size,
a CSS property allow you to
specify natural size of an
element if the element is
affected by size containment.
@giacomozecchini

67. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
VIEWPORT
Google starts the rendering
using a fixed viewport:
Mobile: 412 X 732
Desktop: 1024 x 1024
@giacomozecchini

68. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
VIEWPORT
PAGE
HEIGHT
Viewport = Page Height + pixels
When the browser starts the
rendering the Page Height is
calculated using the
contain-intrinsic-size
@giacomozecchini

69. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
VIEWPORT
PAGE
HEIGHT
A bigger viewport
makes the browser
rendering the element
affected by size
containment.
@giacomozecchini

70. Tools support
97% of tests showed a
different result
* for very tall pages @giacomozecchini

71. Solution
Start with a fixed viewport
@giacomozecchini

72. Solution
Wait for an event:
onload
DOMContentLoaded
If you’re using puppeteer:
networkidle0
networkidle2
@giacomozecchini

73. Solution
VIEWPORT
PAGE
HEIGHT
Check for the page height
and compare that to the
initial viewport.
https://chromedevtools.github.io/devtools-protocol/tot/Page/#method-getLayoutMetrics @giacomozecchini

74. Solution
VIEWPORT
PAGE
HEIGHT
If the viewport is shorter
than the page
Viewport = Page Height + Pixels
@giacomozecchini

75. Solution
The simplest solution is then
to wait for X seconds and
stop rendering or check
viewport and Page Height
again.
VIEWPORT
PAGE
HEIGHT
@giacomozecchini

76. Edge Case #4
Shadow DOM
@giacomozecchini

77. Shadow DOM
https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_shadow_DOM @giacomozecchini

78. Google is able to render
and use Shadow DOM content.
N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow. @giacomozecchini

79. Tools support
?
93% of tests showed a
different result
@giacomozecchini

80. Solution
Using the document.documentElement.outerHTML
returns a DOMString containing an HTML
serialization of the element and its descendants
but not the Shadow DOM.
*Puppeter’s Page.content() returns the outerHTML
@giacomozecchini

81. Solution
The solution is to get
the DOM tree, traverse it
and serialize it into HTML.
https://www.w3schools.com/js/js_htmldom_navigation.asp @giacomozecchini

82. Solution
Document
HTML
BODY
DIV
P
P
Text
Text
HEAD
@giacomozecchini

83. Solution
dom2html library:
https://github.com/GoogleChromeLabs/dom2html
Chrome DevTools Protocol:
DOM.getDocument
DOMSnapshot.getSnapshot
DOMSnapshot.captureSnapshot
https://chromedevtools.github.io/devtools-protocol/
* If interested in Shadow DOM, have a look at: https://web.dev/declarative-shadow-dom/
@giacomozecchini

84. Edge Case #5
Iframe
@giacomozecchini

85. IFRAME
Page A
Page B
@giacomozecchini

86. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
Google is able to render
an Iframe inlining the <body>
content in a <div>.
@giacomozecchini

87. N.B. These are just personal assumptions based on tests.
Tests could be wrong and implementation details may change tomorrow.
If the page included through the
Iframe has a noindex, the content
is not included in the page.
@giacomozecchini

88. Iframe - Tools support
?
95% of tests showed a
different result
@giacomozecchini

89. Solution
Get the DOM tree, traverse it, and serialize it into
HTML.
N.B. When traversing the DOM you only need the
content of <body>, remove other HTML elements
and tags such as the <head>. Remember to check
for the noindex.
@giacomozecchini

90. Solution
dom2html library:
https://github.com/GoogleChromeLabs/dom2html
Chrome DevTools Protocol:
DOM.getDocument
DOMSnapshot.getSnapshot
DOMSnapshot.captureSnapshot
https://chromedevtools.github.io/devtools-protocol/ @giacomozecchini

91. Are those the only problems that
exist?
@giacomozecchini

92. Web rendering services

93. What we can learn from this?
@giacomozecchini

94. Sometimes you should reinvent
the wheel.
It’s fun and you can learn a lot
from that!
@giacomozecchini

95. When you change the way you look
at things, the things you look at
change.
Understanding these limitations
should change, in those edge cases,
the advice that you provide.
@giacomozecchini

96. Don’t use tools blindly!
Tools are great and save us a huge
amount of time in all our tasks. The
majority of pages on the web are
not affected by those edge cases.
@giacomozecchini

97. If your website uses or is affected
by one of the mentioned edge
cases, you can open a support
ticket to check with your tool
provider if they are already covering
that.
@giacomozecchini

98. Thank You!
Got questions? DM on Twitter are open @giacomozecchini