Giacomo Zecchini discusses the challenges of debugging rendering problems for web pages at scale in his presentation. He outlines how search engines render pages and the common issues that can arise during this process, including various network and resource errors. Zecchini proposes a solution using JavaScript and post requests to log rendering errors efficiently, allowing for improved debugging and monitoring of web page rendering issues.

1. Debugging rendering
problems at scale
Giacomo Zecchini | Verve Search
SLIDESHARE.NET/GIACOMOZECCHINI
@GIACOMOZECCHINI

2. Hi, I’m Giacomo. Technical
Director at Verve Search.
Technical background and
previous experiences in
development.
@giacomozecchini
#brightonSEO

3. Today we are going to talk about
rendering errors, the challenges
of debugging at scale and a new
approach to solve these issues.
@giacomozecchini
#brightonSEO

4. The search engine's rendering
process is very similar to
Schrödinger's cat paradox.
https://en.wikipedia.org/wiki/Schrödinger's_cat
@giacomozecchini
#brightonSEO

5. A hypothetical cat page may be
considered simultaneously both
alive correctly rendered and
dead not correctly rendered.
@giacomozecchini
#brightonSEO

6. Let me explain..
@giacomozecchini
#brightonSEO

7. Search engines get web pages
and put them in web rendering
services.
https://developers.google.com/search/docs/guides/javascript-seo-basics
@giacomozecchini
#brightonSEO

8. Inside the web rendering
services, the pages are rendered
similarly to a browser.
https://developers.google.com/search/docs/guides/javascript-seo-basics
@giacomozecchini
#brightonSEO

9. Then, the search engines can
extract all information they
need from those rendered
pages.
https://developers.google.com/search/docs/guides/javascript-seo-basics
@giacomozecchini
#brightonSEO

10. This is an oversimplification of a
complex process.
https://www.youtube.com/watch?v=Qxd_d9m9vzo
@giacomozecchini
#brightonSEO

11. If you want to know more about
this I’d suggest to watch Martin
Splitt’s TechSEO Boost 2019
talk.
https://www.youtube.com/watch?v=Qxd_d9m9vzo
@giacomozecchini
#brightonSEO

12. What’s the problem
with Web Rendering
Services?
@giacomozecchini
#brightonSEO

13. What happens inside the web
rendering services is something
hidden from our eyes, like in a
closed box.
@giacomozecchini
#brightonSEO

14. You don’t know if a page has
been correctly rendered until you
check it manually.
@giacomozecchini
#brightonSEO

15. Search engines are capable of
rendering your pages and most
of the time the process will be
fine.
@giacomozecchini
#brightonSEO

16. Nonetheless, some pages have
rendering problems.
@giacomozecchini
#brightonSEO

17. When is a page not
correctly rendered?
@giacomozecchini
#brightonSEO

18. A page is “not correctly
rendered” when is not possible
for the WRS to get an asset or
when an error blocks the
process.
@giacomozecchini
#brightonSEO

19. Not only pages with Javascript
have problems!
@giacomozecchini
#brightonSEO

20. Let's have a look at a few
examples...
@giacomozecchini
#brightonSEO

21. HTTP / DNS / Network errors
@giacomozecchini
#brightonSEO
https://developers.google.com/search/docs/advanced/crawling/http-network-errors
Crawler
WRS
Cache
SEARCH ENGINE
* Icons made by Freepik from www.flaticon.com

22. Robots.txt blocks a resources
@giacomozecchini
#brightonSEO
https://developers.google.com/search/docs/advanced/robots/intro
Crawler
WRS
Cache
SEARCH ENGINE
* Icons made by Freepik from www.flaticon.com

23. Fetch timeout
@giacomozecchini
#brightonSEO
Crawler
WRS
Cache
SEARCH ENGINE
* this doesn’t seem very common,
but it can happen
* Icons made by Freepik from www.flaticon.com

24. https/http mixed content
@giacomozecchini
#brightonSEO
If your website has an HTTPS
URL but one of the Javascript
files has an HTTP URL and the
HTTPS version is not available,
the script won't be used!

25. Cache mismatch, user
permission for specific
features (e.g. geolocation),
service worker registration,
Javascript syntax errors, etc.
@giacomozecchini
#brightonSEO

26. What if a page is not
correctly rendered?
@giacomozecchini
#brightonSEO

27. If WRS can’t get your CSS the
page layout won’t be correct and
you may also have Mobile
Usability issues.
@giacomozecchini
#brightonSEO

28. If WRS can’t get or execute your
JS files correctly, your page may
be blank or broken.
@giacomozecchini
#brightonSEO

29. Eventually, WRS may need to
render again your page, which
means slower indexing.
@giacomozecchini
#brightonSEO

30. Debugging at scale
@giacomozecchini
#brightonSEO

31. Manually checking page per
page might work on very small
websites.
@giacomozecchini
#brightonSEO

32. When you start having a lot of
pages.. That’s a problem!
@giacomozecchini
#brightonSEO

33. You can prioritise and group
pages with similar HTML and
resources together, but..
@giacomozecchini
#brightonSEO

34. ..the rendering of a page can fail
regardless of what happens to
other similar pages.
@giacomozecchini
#brightonSEO

35. You still have to manually check
pages to be 100% sure those are
correctly rendered.
@giacomozecchini
#brightonSEO

36. WRS capabilities
vs Debugging
@giacomozecchini
#brightonSEO

37. Understanding what a Web
Rendering Service can or can’t
do is a one time task.
@giacomozecchini
#brightonSEO

38. You can build a page with a
specific feature and test it. If it
works once it will work again on
other pages.
@giacomozecchini
#brightonSEO

39. When debugging issues you are
not focusing on a single feature
but on having an overall correct
rendering.
@giacomozecchini
#brightonSEO

40. View crawled page is the way.
@giacomozecchini
#brightonSEO

41. A lot of information.
@giacomozecchini
#brightonSEO

42. But that’s not enough, you want
more. For instance, Javascript
console messages are
coalesced and not shown.
@giacomozecchini
#brightonSEO

43. Yes, you can get JavaScript
console errors from the Mobile
Friendly test or other live tests
but it’s not the same!
@giacomozecchini
#brightonSEO

44. Mobile-Friendly Test and the
other live tests bypass the
cache, have shorter timeouts,
and few other differences.
@giacomozecchini
#brightonSEO

45. A new hope
approach
@giacomozecchini
#brightonSEO

46. I started my research by getting
and printing the information I
needed on the page with some
Javascript, in a hidden <DIV>.
@giacomozecchini
#brightonSEO

47. <html>
…
<div id="info" style="display:none"></div>
…
<script>
…
function getInformation(){
// do stuff!
}
…
var div = document.getElementById("info");
var p = document.createElement("p");
p.innerText = getInformation();
div.appendChild(p);
…
</script>
…
</html>
@giacomozecchini
#brightonSEO
This prints the
information you need
in the DIV at
rendering time and
then you can get
them in Search
Console view crawled
page HTML.

48. But waiting for a page to be
crawled, rendered and indexed
again is time consuming and not
scalable.
@giacomozecchini
#brightonSEO

49. It’s a nice way of discovering
new things but you still have to
manually check all pages.
@giacomozecchini
#brightonSEO

50. Then, I thought of using 1x1 px
images, appending errors or
information in the URL:
https://www.example.com/image.jpg
?u=page_url&e=error
@giacomozecchini
#brightonSEO

51. The idea was to look in the
server access log and find all
errors that occurred during the
rendering.
@giacomozecchini
#brightonSEO

52. But Google’s WRS doesn’t
download images during the
rendering of a page.
@giacomozecchini
#brightonSEO

53. But then..
@giacomozecchini
#brightonSEO

54. The answer was always in
front of my eyes:
Javascript + POST requests!
@giacomozecchini
#brightonSEO

55. Google’s WRS cache GET
requests.
@giacomozecchini
#brightonSEO
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/GET

56. But doesn’t cache POST
requests.
@giacomozecchini
#brightonSEO
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST

57. Say welcome to the shiny new
Search Engine Rendering
Errors Logging framework!
@giacomozecchini
#brightonSEO

58. @giacomozecchini
#brightonSEO
Crawler
WRS
Cache
SEARCH ENGINE YOUR WEBSITE
Search Engines download or use the cache of
the resources they need to render your pages.
* Icons made by Freepik from www.flaticon.com

59. @giacomozecchini
#brightonSEO
CHROMIUM INSTANCE
SEARCH ENGINE
Crawler
INTERNET
During the rendering the website, WRS executes
Javascript and downloads additional resources
a website might need or request.
* Icons made by Freepik from www.flaticon.com

60. @giacomozecchini
#brightonSEO
CHROMIUM INSTANCE
* Icons made by Freepik from www.flaticon.com
SEARCH ENGINE
Crawler
SERVER
What if one of those Javascript sends a non
cacheable POST request to an external server?!
POST
REQUEST

61. @giacomozecchini
#brightonSEO
There are multiple ways of
sending POST requests in JS:
Fetch API
https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch
Navigator.sendBeacon()
https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon
XMLHttpRequest.send()
https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/send

62. {
"page":"https://www.example.com",
"timestamp": 1592568000000,
"category": "Fetch",
"error": "https://www.example.com/style.css"
}
@giacomozecchini
#brightonSEO
The message (or beacon) contains the information you want to store
in your database.

63. @giacomozecchini
#brightonSEO
TIME URL CATEGORY ERROR
25/10/1985 09:00:00 https://www.example.com Fetch https://www.example.com/style.css
21/10/2015 07:28:00 https://www.example.com/about.html Fetch https://www.example.com/app.js
12/11/1955 06:38:00 https://www.example.com Javascript File: https://www.example.com/app.js Line: 3 Col: 2
Error: Uncaught ReferenceError: APP is not defined
When you have everything in a database you can query the tables
and do all your analysis. You can also have automatic alerts, etc.

64. Debugging in
practice
@giacomozecchini
#brightonSEO

65. !! Warning !!
Don’t use this code on your
website, these are just (bad)
examples.
@giacomozecchini
#brightonSEO

66. Debugging example #1
Check if a page has been
rendered
@giacomozecchini
#brightonSEO

67. <html>
…
<script>
sendMessageToServer();
</script>
…
</html>
@giacomozecchini
#brightonSEO
When the WRS executes the script, the
function sends a message back to the
server.

68. Debugging example #2
Know if there is a problem
downloading CSS or JS files
@giacomozecchini
#brightonSEO

69. <html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<script>
…
window.addEventListener('error', function(err) {
if (isDownloadError(err)){
sendMessageToServer(err);
}
}, true);
…
</script>
…
</head>
…
</html>
@giacomozecchini
#brightonSEO
If there is an error and it's a CSS or
JS load error you can send a
message back to the server. This
works for HTTP/DNS/Network errors,
Robots.txt, fetch timeouts, etc.

70. Caveats
@giacomozecchini
#brightonSEO

71. There are some products out
there but all of them focus on
users and not on search
engines.
@giacomozecchini
#brightonSEO

72. Search engines are different
and you need to solve
different problems.
@giacomozecchini
#brightonSEO

73. You should be careful adding
new code to your website!
@giacomozecchini
#brightonSEO

74. Web Performance issues
You don’t want to slow down
the user experience with
something you need only for
search engines.
@giacomozecchini
#brightonSEO

75. Web Performance issues
Check for the User-Agent and
run the script only for search
engines.
@giacomozecchini
#brightonSEO

76. Crawl budget
You don’t want to consume
your crawl budget on these
requests.
@giacomozecchini
#brightonSEO

77. Crawl budget
Host your debugging server on
a different domain or
subdomain.
@giacomozecchini
#brightonSEO

78. There are many other possible
problems, you just need to find
a solution for them.
@giacomozecchini
#brightonSEO

79. Conclusions
@giacomozecchini
#brightonSEO

80. The simpler a page is, the
more chances it will render
correctly. The majority of
pages are just fine.
@giacomozecchini
#brightonSEO

81. If you work on big or complex
websites you may encounter
rendering problems.
@giacomozecchini
#brightonSEO

82. Debugging rendering
problems is a very time
consuming task..
@giacomozecchini
#brightonSEO

83. ..but, if you use the right
approach you can cut down
the time it takes.
@giacomozecchini
#brightonSEO

84. You can use this approach as
a one time debugging script to
get more information or as a
monitoring system.
@giacomozecchini
#brightonSEO

85. Thank You!
Got questions? DM me on Twitter.
@giacomozecchini