React.js & SEO - A Use Case Study

Editor's Notes

1. Intro - me Hottest topic ->
2. What why & experience testing, go live, issues, a solution? Used Botify for analysis & throughout
3. What & why - JS framework by FB. Easily share components, all sounds pretty good right? Frankie Goes to Hollywood 1984 single relax, change lyrics React don’t do it. Why so concerned? ->
4. Not able to process successfully or immediately ->
5. Similarly Bing states Rich Media e.g JS can lead to Bing not being able to crawl
6. We know Google then has issues with JS when rendered on client side, takes Google longer to index content as needs to go through 2 step indexing approach. 1st wave indexing, at which point tries to render the content. If struggles, goes away adn comes back when has the resources available. Problem = caches a page before rendering complete. Lot of issues if pages change frequently. Horror stories - not caching, dips in visibility ->
7. Put a page live anyway! So what did that look like? not so great. Not evergreen version of Googlebot, used Chrome 41 emulator and just as we saw there ->
8. Google cached an empty page. Not able to render the JS and cache that or index it. Story of this particular page real heartbreaker ->
9. See here, nice boost from broad core update in March, it wSee here, nice boost from broad core update in March, it was added to our mega menu, made it to site links then ->as added to our mega menu, made it to site links then ->
10. goes live in react, and visibility takes a sharp dip
11. Crawl data in Botify shows that before we went live in react, crawled approx 109x per day, then goes live and you can see delay in crawls, and now page still crawled but not at same volume
12. Unfortunately doesn't deter anyone from developing in react. Have to test as soon as possible to push more pages live in react. Now in the meantime the IT team have been working on a Server Side Rendering solution, to identify bots and serve them a rendered version of the pages from on the server side. So when it comes to testing this list of pages with examples of different templates across all 3 sites, patient Nick Tyler last min changes, cancel crawl 6am, set up to crawl & render in Botify spoofing Google UA. ->
13. Early to test, and snapshot of the really sexy testing doc we put together, looking in Botify for elements such as h1 tag, canonicals, internal links, manual checks on content, JS rendering in Chrome 41 and a whole host of other things, including bot requests hitting node stack correctly for our SSR solution. Mixed bag, back to the drawing board. Then next big issue 2nd time ->
14. response code errors, 301s not pulling through. Getting frustrated as developer, but saw drop in traffic. Under pressure to sign off pages even though no solution in place still. ->
15. along comes Google with a big announcement at I/O 19 ->
16. Googlebot evergreen, runs on latest version of Chromium, Chrome 74 at this time ->
17. Gets a lot of people talking. No longer look at Chrome 41, should be better at handling react - more modern frame work, more modern crawler. May be true for Googlebot, but what about other crawlers? Already saw what Bing have to say. Or real users, our customers who may well use older devices or browers which are not capable of rendering react correctly. So what did we decide to do? ->
18. put more pages live! Lower traffic, less risky pages, put live in react with no SSR solution to see how evergreen Googlebot handles them. Chrome 74 little to no issues with react, difference between being able to do something & actually doing it. Maths, me and Googlebot the same->
19. But with an unexepected unpleasant twist ->
20. Monitored closely in crawl logs in Botify, also manually checked the cached pages. ->
21. And this is when we got a 404. Tried to cache the page and got a 404 response and cached that, on every one of these pages. Not to say I told you so, monitor for longer. Cached like this full 24 hours. Not end of world for small set of low risk pages? 1) cant handle these, how can it handle them at scale? 2) still important pages, links, content, do get traffic. Imagine putting your biggest traffic page live like that and gets a 404 ->
22. Also saw huge drops in rankings, for one of the pages a position drop of over 50 in 24 hours, and these are for branded terms. 24 hours later starts to resolve and cache the page. Assume second wave of indexing. So h
23. So how long to wait? John Muelleur tweeted - was a year ago & a lot can happen in a year, but point around time is still important. We won't know when that second wave is going to hit. Looking at the crawl data in Botify, it seemed that even for these pages Google was crawling it more than once a day, but at least daily so if crawled less frequently could be looking at more like a week if there is no dynamic rendering solution in place ->
24. We worked closely with our IT teams to develop a server side rendering solution. While this may work for TUI, it may not be the right choice for everyone. 1) warm up the cache, using Botify, unique UA string to store a cached version of the page within our internal cache 2) Google/other bots can crawl the site and our identified by the CDN as credible bots 3) they are then served the rendered version of that page which has been stored in the internal cache. same content as user sees but pre-rendered, so the indexing & rendering is more instant and not asking them to go away and come back 4) customers still render the pages on client side ->
25. Does it work? In theory yes, still in testing, planned on going live Tuesday but response times were too high. Still under investigation with the IT team managing our CDN
26. this is probably more what our timeline looked like in the end, a lot of back and forth, and also a fair amount of skipping the middle steps which obviously not best pleased about but has given us a good before and after view of having a SSR solution in place ->
27. React don't do it might have been a bit of an over reaction in hindsight, but done incorrectly and without taking SEO into consideration, could have a huge impact on your sites visibility if crawlers struggle to index your content More modern versions of JS are seemingly pretty inevitable, and while we wait for the crawlers to catch up, you have to make sure your developers and IT team are your best friends in order to make sure your content is seen