SEO AJAX Crawlability in a Responsive Publisher World

5,110 views

Published on

With the emergence of heavy javascript / AJAX heavy frameworks and the growing popularity of things like AngularJS, Ember, Backbone.js, CanJS, and even JQuery; making sites and single page apps crawlable to search engines are becoming increasingly difficult. It doesn't have to be.

This presentation takes a look at some of the largest and trending publishers and some of the AJAX features they employ.

Published in: Technology, Design

SEO AJAX Crawlability in a Responsive Publisher World

  1. 1. AJAXin a responsive publishing world CRAWLABILITY
  2. 2. google.me/+EricWu @eywu linkedin.com/in/ericywu github.com/eywu
  3. 3. AJAXis like violence if it doesn't solve your problems, you’re not using enough of it.
  4. 4. 16ms 10kb 100ms SPEED, PERFORMANCE, HUMAN PERCEPTION
  5. 5. PAGES CRAWLED PER DAY 80% INCREASE 20,000 80,000 May 2013 – Aug 2013
  6. 6. 80% INCREASE ORGANIC SEARCH SESSIONS 600,000 1,200,000 May 2013 – Aug 2013
  7. 7. Mar 2004:“Googlebot/Test” External JS Mar 2006: Googlebot Uses Onsite Live Chat June 2010: Caffeine (Full Rollout) Nov 2010: Instant Preview May 2014: GWT Fetch & Render May 2012: Matt PSA. Don’t Block JS & CSS Oct 2009:AJAX Crawlability _escaped_fragment_ Nov 2007: Spider’sView on Web 2.0 May 2013: MattVideo. Googlebot & AJAX
  8. 8. SCROLL
  9. 9. http://googlewebmastercentral.blogspot.com/2014/02/infinite-scroll-search-friendly.html THINK BASIC PAGINATION NAVIGATE / PAGINATE
  10. 10. NAVIGATE / PAGINATE <a href="/page/2/">Load More</a> LOAD MORE LINK
  11. 11. NAVIGATE / PAGINATE <a href="/page/2/">Load More</a> LOAD MORE LINK
  12. 12. NAVIGATE / PAGINATE REL=NEXT / PREV <link href="/page/10/" rel="prev" /> <link href="/page/11/" rel="canonical" /> <link href="/page/12/" rel="next" />
  13. 13. POINTER URLS* example.com/?last-post=42 example.com/?time=1401424133
  14. 14. CONTINUOUS CONTENT
  15. 15. PUSHSTATE OR SHEBANG #! http://caniuse.com/#search=history history.pushState( {}, titleOfPage, newURL ); history.replaceState();
  16. 16. AJAX GALLERIES
  17. 17. CONTINUOUS CONTENT 1. PUSHSTATE 2. VIEW-SOURCE NEXT SLIDE LINKS 3. REL=NEXT/ PREV
  18. 18. DEFERED IMAGE LOADING I WISH CRAWLERS SUPPORTED REL=CANONICAL HTTP HEADERS FOR IMAGES
  19. 19. 80% DECREASE IMAGE SEARCH SESSIONS 400,000 800,000 Nov 2012 – Mar 2013
  20. 20. LAZY LOADING •  1x1 “BLANKS” •  SKELETON SCREENS •  LOW RESOLUTIONZ Z Z
  21. 21. RESPONSIVE IMAGES •  <PICTURE> •  SRCSET •  POLYFILL •  UA DETECTION
  22. 22. NOSCRIPT <img data-src='<480:small.jpg, <960:medium.jpg, >960:large.jpg' /> <noscript> <img src='full.jpg' /> </noscript>
  23. 23. CRAWLABLE PARTIALS
  24. 24. CRAWLABLE PARTIALS LYRIC SEARCH LINKS TOhttp://rock.rapgenius.com/1884632/Pharrell-williams- happy/ It-might-seem-crazy-what-im-bout-to-say-sunshine-shes- here-you-can-take-a-break
  25. 25. CRAWLABLE PARTIALS JS ENABLED REDIRECTShttp://rock.rapgenius.com/Pharrell-williams- happy-lyrics#note-1884632
  26. 26. CRAWLABLE PARTIALS JS DISABLED RENDERShttp://rock.rapgenius.com/1884632/Pharrell-williams- happy/It-might-seem-crazy-what-im-bout-to-say-sunshine- shes-here-you-can-take-a-break
  27. 27. 4X PAGES CRAWLED PER DAY QUADRUPALED 75,000 300,000 May 2013 – Aug 2013
  28. 28. +1MILLION ORGANIC SEARCH SESSIONS 2,000,000 4,000,000 May 2013 – Aug 2013
  29. 29. Technical SEO Community HelpDeskHangouts.com SearchSignals.com bit.ly/ajax-crawlability
  30. 30. APPENDIX
  31. 31. ARCHITECTURE PRERENDER SERVERSIDEVS
  32. 32. PRERENDER
  33. 33. PRERENDER
  34. 34. PRERENDER SEO.js CONSPROS • SINGLE VIEW TEMPLATE • SINGLE ROUTING • REVERSE PROXY CACHING • _ESCAPED_FRAGMENT_ • INTERPOLATED SOURCE
  35. 35. SERVER SIDE
  36. 36. SERVER SIDE
  37. 37. SERVER SIDE
  38. 38. SERVER SIDE
  39. 39. SERVER SIDE
  40. 40. SERVER SIDE
  41. 41. SERVER SIDE RENDER CONSPROS • CLEAN URLS* • ALL REQUESTS ARE EQUAL • TRUE DYNAMIC RESPONSE • 2 ROUTERS • 2 VIEW TEMPLATES*
  42. 42. Quote conflated from my favorite ruby XML parser » http://nokogiri.org/ Speed, Performance, and Human Perception » https://www.youtube.com/watch?v=7ubJzEi3HuA SERoundtable Timeline Links » http://www.seroundtable.com/google-javascript-webmaster-tools-18602.html Googlebot/Test External JS » http://www.seroundtable.com/archives/000236.html Googlebot Uses Onsite Live Chat » http://www.seroundtable.com/archives/003492.html Spider’sView on Web 2.0 » http://googlewebmastercentral.blogspot.com/2007/11/spiders-view-of-web-20.html AJAX Crawlability Proposal » http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html Caffine Rollout » http://googlewebmastercentral.blogspot.com/2010/06/our-new-search-index-caffeine.html Instant Previews » http://googleblog.blogspot.com/2010/11/beyond-instant-results-instant-previews.html http://googlewebmastercentral.blogspot.com/2010/11/instant-previews.html http://googlewebmastercentral.blogspot.com/2011/11/get-post-and-safely-surfacing-more-of.html https://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews Matt Cutts PSA: Don’t Block JS & CSS » http://www.seroundtable.com/googlebot-javascript-css-14930.html MattVideo: How Does Googlebot handle content loaded via AJAX? » https://www.youtube.com/watch?v=_6mtiwQ3nvw REFERENCES
  43. 43. GWT Fetch & Render » http://googlewebmastercentral.blogspot.com/2014/05/rendering-pages-with-fetch-as-google.html Google Blog: Infinite Scroll Recommendations & Example » http://googlewebmastercentral.blogspot.com/2014/02/infinite-scroll-search-friendly.html LA Times Reimagined by Code and Theory » http://www.codeandtheory.com/things-we-make/the-los-angeles-times-reimagined Google Blog: Specify your canonical » http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html Google Blog: Pagination with rel=“next” and rel=“prev” » http://googlewebmastercentral.blogspot.com/2011/09/pagination-with-relnext-and-relprev.html Google Blog:Video about Pagination » http://googlewebmastercentral.blogspot.com/2012/03/video-about-pagination-with-relnext- and.html One Page Wonder: Coverage on QZ » http://www.foliomag.com/2013/one-page-wonder-infinite-scroll The Next Web Redesign Coverage » http://www.niemanlab.org/2012/10/the-next-web-redesigns-to-be-more-app-like/ The Next Web Press Release » http://thenextweb.pr.co/ 010a893a11df2bb61d981b2b0607c1b6784a5ab07b5ab100790b2bb3168a35f8 REFERENCES
  44. 44. USA Today Redesign » http://blog.f-i.com/usatoday-com-redesigning-one-of-americas-most-popular-news-site/ http://designenvy.aiga.org/usa-today-website-redesign-fantasy-interactive/ http://www.businessinsider.com/usa-todays-homepage-redesigns-2012-9 Gawker 1Year Later Success » http://thenextweb.com/insider/2012/02/02/remember-that-gawker-redesign-a-years-worth-of- data-says-it-worked/ http://www.businessinsider.com/nick-denton-loses-bet-that-the-gawker-redesign-wouldnt-hurt- traffic-2011-10 http://www.businessinsider.com/gawker-media-traffic-numbers-2011-4 Gawker Failed Coverage » http://www.catchmyfame.com/2013/05/02/how-gawker-sabotaged-their-own-network-with-a- horrible-new-layout/ http://www.theatlantic.com/technology/archive/2011/04/gawkers-traffic-numbers-are-worse-than- anyone-anticipated/237594/ http://www.webmonkey.com/2011/02/gawker-learns-the-hard-way-why-hash-bang-urls-are-evil/ Paul Irish to Matt CuttsVideo » https://www.youtube.com/watch?v=yiAF9VdvRPw Google Developer Documentation on AJAX Crawlability » https://developers.google.com/webmasters/ajax-crawling/ Browser Compatibility Chart » http://caniuse.com/#search=history Breaking The Web With Hash Bangs » http://isolani.co.uk/blog/javascript/BreakingTheWebWithHashBangs REFERENCES
  45. 45. Vox Cards: Legalization of Marijuana » http://www.vox.com/cards/marijuana-legalization/learn-more-about-marijuana-legalization Bing’s Duane Forrester says still no rel=canonical in http headers » https://twitter.com/DuaneForrester/status/459387860358295552 Google Blog:A Faster Image Search » http://googlewebmastercentral.blogspot.com/2013/01/faster-image-search.html Google Says It’s Better for Webmasters » http://www.seroundtable.com/google-image-search-design-16259.html Ilya Grigork discussion around <plaintext> injection » https://plus.google.com/+IlyaGrigorik/posts/S6j45VxNESB Vox Workflow for Creating SVG Images » http://product.voxmedia.com/2013/11/25/5426880/polygon-feature-design-svg-animations-for-fun- and-profit One Solution to Responsive Images » http://www.smashingmagazine.com/2014/02/03/one-solution-to-responsive-images/ Truly Responsive Images » http://davidwalsh.name/responsive-design AngularJS NYC Meetup: Server-side Template Rendering by HBO » http://youtu.be/iB7hfvqyZpg?t=58m20s REFERENCES
  46. 46. Vox Cards: Legalization of Marijuana » http://www.vox.com/cards/marijuana-legalization/learn-more-about-marijuana-legalization Bing’s Duane Forrester says still no rel=canonical in http headers » https://twitter.com/DuaneForrester/status/459387860358295552 Google Blog:A Faster Image Search » http://googlewebmastercentral.blogspot.com/2013/01/faster-image-search.html Google Says It’s Better for Webmasters » http://www.seroundtable.com/google-image-search-design-16259.html Ilya Grigork discussion around <plaintext> injection » https://plus.google.com/+IlyaGrigorik/posts/S6j45VxNESB Vox Workflow for Creating SVG Images » http://product.voxmedia.com/2013/11/25/5426880/polygon-feature-design-svg-animations-for-fun- and-profit One Solution to Responsive Images » http://www.smashingmagazine.com/2014/02/03/one-solution-to-responsive-images/ Truly Responsive Images » http://davidwalsh.name/responsive-design Serious Angular SEO » http://www.ng-newsletter.com/posts/serious-angular-seo.html AngularJS NYC Meetup: Server-side Template Rendering by HBO » http://youtu.be/iB7hfvqyZpg?t=58m20s REFERENCES
  47. 47. Josh Kadis Quartz onVIP WordpressVideo » http://vip.wordpress.com/2013/09/26/josh-kadis-qz-wordpress/ https://docs.google.com/file/d/0B2Z4K6ynFLg5TVdvWVV1aTRmYUU/edit?pli=1 AirBNB: Our First Node.js App » http://nerds.airbnb.com/weve-launched-our-first-nodejs-app-to-product/ AirBNB: Rendr (Backbone in the Browser and Node) » http://nerds.airbnb.com/weve-open-sourced-rendr-run-your-backbonejs-a/ StackOverflow: PushState, Backbone, and Node » http://stackoverflow.com/questions/7098130/reusing-backbone-views-routes-on-the-server-when- using-backbone-js-pushstate-for Google: How do I create an HTML Snapshot (HIJAX) » https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot REFERENCES

×