Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Distilled, profile picture
Uploaded byDistilled
PDF, PPTX4,953 views

SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs

The document discusses the evolving landscape of technical SEO, emphasizing the need for SEOs to adopt developer-like thinking in response to advancements in web technologies, particularly JavaScript and frameworks like AngularJS and ReactJS. It highlights the importance of understanding contextual rankings, crawlability, and the utilization of HTTP headers over traditional SEO practices. The author encourages leveraging log file analysis and internal linking strategies while cautioning against outdated techniques like AJAX crawling and improper redirection methods.

Marketing◦
Download as PDF, PPTX
DEVELOPER THINKING FOR SEOS MICHAEL KING MANAGING DIRECTOR, IPULLRANK @IPULLRANK
Baby Sign Language …and how it relates to technical SEO
IPULLRANK.COM @ IPULLRANK So, I’m Having a Baby (She is too)… It’s a girl!
IPULLRANK.COM @ IPULLRANK Last Night I Learned Infants Can Learn Sign Language …
IPULLRANK.COM @ IPULLRANK Which is Great Because They Act like Drunks Otherwise …
IPULLRANK.COM @ IPULLRANK Which is Astonishingly Similar to the Perception of Marketers
SEOs generally know what they want, just not how to effectively ask for it.
As of Late People Started Saying You Don’t Need to Be Technical
IPULLRANK.COM @ IPULLRANK Yet, the Web is More Technical than Ever Before
The modern SEO needs to think like a developer because we are in a technical SEO renaissance.
IPULLRANK.COM @ IPULLRANK Lucky for You This Is The Point Where I have My Own Agency Shameless plug.
Changes in Web Technologies What is pushing this renaissance?
IPULLRANK.COM @ IPULLRANK JavaScript is Not Going Away Nearly every site on the web is using JavaScript to perform some type of transformation on the page. It’s obvious that Google needed to solve for this.
IPULLRANK.COM @ IPULLRANK Growth of AngularJS AngularJS, the single page application, MVW framework continues to grow dramatically in the top million websites.
IPULLRANK.COM @ IPULLRANK AngularJS is Popular in Most Verticals It’s likely that you’ll run into an AngularJS site soon enough in your optimization duties
IPULLRANK.COM @ IPULLRANK ReactJS is Growing Too Similarly, Facebook’s MVW framework is growing as well. Although, it has a built in feature for making sure that it renders proper HTML, but it’s still a framework you should know how to optimize for. Check out this post https://builtvisible.com/react-js-seo/
IPULLRANK.COM @ IPULLRANK Ironically, React (Facebook) is SEO-ready, Angular (Google) is Not Angular 1 did not have anything to make it easier to crawl. React has ReactDOMServer.renderToString to allow React to serve fully rendered HTML from the server.
View Source is Dead
Inspect Element Only
IPULLRANK.COM @ IPULLRANK HTTP/2 – Perhaps the Most Important Thing All you need to know right now is that it’s fast as hell.
Adoption is Super Low For Now Though
We Need More from Our SEO Tools Many of them are far behind search engines
What are rankings in 2016?
So, You’re #1? Cool Story, Bro. Where they at tho?
IPULLRANK.COM @ IPULLRANK On What Device, OS and Location? Different browser and OS configurations will show you different results. So what truly is a mobile ranking? + + = Ranking #3+ Nexus 5 Android NYCChrome Safari + + = Ranking #1+ iPhone 6 iOS NYC
Are we measuring for context and actionability or just vanity?
Please Give Me Position 0 in My Rankings
Please Give Me My Results in Context of Paid #1 is really #8 in this SERP
What is cloaking in 2016?
IPULLRANK.COM @ IPULLRANK 304 Response Codes The Not-Modified response code tells clients that they do not need to download a page again if it has already been downloaded.
IPULLRANK.COM @ IPULLRANK Googlebot Respects the 304 What is cloaking when Google accepts 304 response codes and does not revisit that content after indexation?
IPULLRANK.COM @ IPULLRANK What is Cloaking in the Adaptive/Responsive Era? Many people show less or more content based on the context of the browser/user, so what is cloaking now?
Google accepts directives in HTTP headers.
IPULLRANK.COM @ IPULLRANK X-Robots-Tag HTTP Header Many of the popular crawling tools do not look for this in the response headers.
IPULLRANK.COM @ IPULLRANK Hreflang HTTP Header I have not seen a tool that looks for this HTTP header. However, this implementation is preferred over the many lines of hreflang tags on every page.
IPULLRANK.COM @ IPULLRANK Rel-Canonical HTTP Header Many tools do not look for the rel-canonical HTTP header.
All of these HTTP headers are better than clogging up the <head> of the code.
IPULLRANK.COM @ IPULLRANK JavaScript Redirects for Mobile Google allows client side redirects for automation redirection of mobile to desktop and vice versa. Many crawling tools do not account for this.
Many SEO tools do not account for these things
IPULLRANK.COM @ IPULLRANK Chrome DevTools Does Natively in Chrome’s Developer Tools you can see all of the HTTP Request and Response Headers in the Network tab section.
IPULLRANK.COM @ IPULLRANK Geolocation Emulation is Built-in to Chrome If you’re running into issues with finding local rankings now that Google has gotten rid of nearEquals you can just change your location on the browser level.
IPULLRANK.COM @ IPULLRANK Remote Debugging You can also see this for a specific mobile devices with the remote debugging features of Chrome.
Crawlability What can be crawled at this point?
IPULLRANK.COM @ IPULLRANK Google is Crawling Headless Make no mistake that Google has not been crawling with just text driven crawlers for a long time.
IPULLRANK.COM @ IPULLRANK Adam Audette Said It So You KNOW It’s True Adam ran a few tests a little under a year ago further proving out these capabilities
IPULLRANK.COM @ IPULLRANK Also an Ex-Googler Confirmed It A former Google developer said the headless crawling was his primary project from 2006 – 2010.
IPULLRANK.COM @ IPULLRANK I Said it in 2011
IPULLRANK.COM @ IPULLRANK JsCrawlability.com Eric Wu has been doing tests to get a better determination of what it is that Google can actually see. http://www.jscrawlability.com
But Mike, what does it matter?
IPULLRANK.COM @ IPULLRANK Well I Captured Video of Headless Googlebot It’s a question of what does Googlebot do on your site. Using LuckyOrange I captured Googlebot sessions from Search Console’s Fetch and Render and then I submitted it to the index. They load the page and leave. The mouse never moves, the headless browser never came back.
IPULLRANK.COM @ IPULLRANK Get a Free Trial of LuckyOrange and Try it Yourself This code will only place LuckyOrange on the page if a JS-enabled Googlebot comes to your site. My test did not show what Googlebot from the wild does when it’s crawling.
IPULLRANK.COM @ IPULLRANK Log Files Went out of Style? No, log file analysis has not gone out of style. In fact, it’s more valuable than ever before. LOOK AT YOUR LOG FILES!
IPULLRANK.COM @ IPULLRANK Log File Analysis We parse the logs and query them in MySQL, but you can use any number of tools for this.
IPULLRANK.COM @ IPULLRANK Screaming Frog Log File Analyzer The Screaming Frog team rolled out an awesome new tool recently for log file analysis.
IPULLRANK.COM @ IPULLRANK Search Console’s Crawl Stats are Mediocre I personally find what Google gives us to be pretty much useless. Don’t less this take the place of going through your logs.
IPULLRANK.COM @ IPULLRANK Log Files Help You Spot What Really Happens I had a client come last year that was unsure whether it was Penguin or changes in their offline media buying habits had led to the downturn in traffic. Layering the Googlebot logs made it clear. 0 50000000 100000000 150000000 200000000 250000000 0 500000 1000000 1500000 2000000 2500000 Unknown organic organic home organic dyor organic er organic home,dyor,er organic cushion organic net of home organic ritani organic diamonds direct home paid brand sessions tv spend web sessions (as reported by adaptv) Top 800 Monthly SV impressions Googlebot Visits Penguin Update
IPULLRANK.COM @ IPULLRANK Social Shares Influence Crawl Patterns We find that there is a higher correlation with crawling and social shares than crawling and links. Commonly held SEO knowledge would make you believe this is entirely dictated by links. 0 1000000 2000000 3000000 4000000 5000000 6000000 0 100000 200000 300000 400000 500000 600000 700000 Googlebot Visits Links Social Shares FB Twitter G+
IPULLRANK.COM @ IPULLRANK Prerendering is the Best Practice Using a solution like prerender.io is the best practice.
IPULLRANK.COM @ IPULLRANK Prerender Can Cause Problems However, we’ve seen Prerender setups misidentify Googlebot as a human users and serve the wrong the AngularJS versions. 0 20000 40000 60000 80000 100000 120000 Googlebot Perceived as Human
IPULLRANK.COM @ IPULLRANK Prerender Setup Serving Non-200 Prepping the prerender cache can cause outages and lead to pages falling in and out of the index. 0 2000 4000 6000 8000 10000 12000 14000 16000 206 301 302 304 403 404 405 410 500 503 504 206 301 302 304 403 404 410 503 504 206 301 302 304 404 405 410 500 503 504 206 301 302 304 401 404 405 410 500 503 504 206 301 302 304 403 404 405 410 502 503 504 1 2 3 4 5 - hit miss pass
IPULLRANK.COM @ IPULLRANK Also Prerendering is Not a Requirement We got rid of prerender on 2 sections of the site and saw the traffic from Organic Search increase YoY. Google can index it just fine.
IPULLRANK.COM @ IPULLRANK AJAX Crawling Deprecated Google has gotten rid of the AJAX crawling scheme because they don’t need it anymore. They have perfected their ability to crawl headless at reasonable scale. They say don’t block the CSS or JS because it’s a browser.
They Say Don’t Prerender Just for Google
IPULLRANK.COM @ IPULLRANK Prerender Does Speed Up Crawling There is certainly value in prerendering for speed, but then that makes me wonder, why not just not use Angular?
IPULLRANK.COM @ IPULLRANK A Better Idea Is a Library that Doesn’t Need Prerender Let’s stop going after the shiny object and go with the things that are built to work, shall we? Check out MeteorJS – http://www.meteor.com
IPULLRANK.COM @ IPULLRANK Or go with Angular2 Brad Green is the Google engineering product manager for AngularJS. He says Angular2 is better in every way.
IPULLRANK.COM @ IPULLRANK Don’t Forget Google Wants Progressive Enhancement Thinking with the IoT in mind, progressive enhancement still makes sense. However, frameworks like AngularJS are completely contradictory in that they don’t offer graceful degradation.
Scraping The backbone of a lot of our analysis as SEOs
IPULLRANK.COM @ IPULLRANK Most Tools use cURL or Similar There are many similar libraries, but many SEO tools are built on library called cURL for managing HTTP requests. This is a text based crawler.
This is a fundamental flaw of many SEO tools because they are not constructing the DOM and its CSS/JS transformations. Search Engines are.
IPULLRANK.COM @ IPULLRANK PhantomJS – For Scraping the Modern Web There are many headless browsers out there such as Selenium, but PhantomJS is the go to. http://www.phantomjs.org
IPULLRANK.COM @ IPULLRANK HorsemanJS – The Easy PhantomJS I prefer HorsemanJS which is a NodeJS module that is incredibly simple. http://www.horsemanjs.org
IPULLRANK.COM @ IPULLRANK In-Browser Scraping I wrote a post on how you can scrape using your browser’s Console.
IPULLRANK.COM @ IPULLRANK Scrape the Unscrapable Nothing is in unscrapable when you’re using a browser.
IPULLRANK.COM @ IPULLRANK Multi-Page In-browser Scraping If you want to scrape multiple pages check out ArtooJS https://medialab.github.io/artoo/
IPULLRANK.COM @ IPULLRANK WebScraper.io WebScraper is another high-powered in-browser scraping tool. No code required. http://www.webscraper.io
IPULLRANK.COM @ IPULLRANK Or You Can Make Screaming Frog Headless You can make Screaming Frog headless by placing a proxy with a headless browser in front of it. From my initial findings, there is not much different aside from the order it identifies URLs and its ability to scrape content from pages.
IPULLRANK.COM @ IPULLRANK Here’s the Code Here’s the node code for how I did it. HorsemanJS is not the best tool for the proxy, but it’s great for a proof of concept. Screaming Frog pings for robots.txt a lot. Make sure you handle it with a cURL request rather than a headless one. For speed, don’t load images Make sure you’re setting the right headers before your return the HTML. HorsemanJS does not return headers.
IPULLRANK.COM @ IPULLRANK PHP Wrapper for PhantomJS For those of you that prefer PHP, the PHP PhantomJS wrapper is pretty decent as well. http://jonnnnyw.github.io/php-phantomjs/ For speed, don’t load images
Content & Linking You knew I wasn’t going to get up here and not talk about these things, right?
IPULLRANK.COM @ IPULLRANK Google Looks at Entities First According to Paul Haahr, Google looks at entities first and it appears that we as an industry have talked about this in theory, but not truly worked it into the optimization workflow.
IPULLRANK.COM @ IPULLRANK AlchemyAPI I encourage you to work AlchemyAPI into your keyword research and content optimization process.
IPULLRANK.COM @ IPULLRANK Term Relevance There’s also been a lot of talk about co-relevance, topic modeling and TF*IDF. This is a great opportunity for technical elements to influence content creation.
IPULLRANK.COM @ IPULLRANK In Fact Just Do Everything Cyrus Says https://moz.com/blog/7-advanced-seo-concepts
IPULLRANK.COM @ IPULLRANK Moz’s New Tool Will Help Moz has made the related keywords concept more accessible by adding it to Moz Pro. Searchmetrics also has a similar solution.
IPULLRANK.COM @ IPULLRANK Onpage.Org’s TF-IDF Tool is Also Great http://onpage.org
IPULLRANK.COM @ IPULLRANK John Mueller Says 302s Pass PageRank Don’t let Google mislead you about 302s vs. 301s.
IPULLRANK.COM @ IPULLRANK These Rankings Say He’s Wrong We had a client with millions for 302s and all we did was change them to 301s June August September HyperTraffic (1-3) 1272 1,283 1,473 Traffic (4-10) 979 983 837 Striking Distance (11-20) 167 161 110 Emerging (21-40) 106 103 98 Developmental (41-100) 156 90 87 Unranked 0 60 75 0 200 400 600 800 1000 1200 1400 1600 301 redirects
IPULLRANK.COM @ IPULLRANK This Traffic Improvement Says He’s Wrong Traffic went up pretty dramatically despite seasonality. August 27th 301 redirects 11,039,188 10,455,517 11,348,902 12,049,481 11,250,148 12,902,847 11,040,192 13,158,296 11,287,790 12,203,998 12,508,164 13,158,296 14,041,313 15,609,390 0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 12,000,000 14,000,000 16,000,000 18,000,000 April May Jun Jul Aug Sep Oct Organic Visits 2014 2015
But. We need to stop saying ‘just do 301 redirects.’
IPULLRANK.COM @ IPULLRANK Instead Do Optimal 301 Redirects Pattern matching on the .htaccess level is always faster than a list of redirects and always better than doing it on the code/page level. Developers will respect that you understand what you’re talking about. ReWriteRule url_pattern file_reference [R=301, L] redirect 301 /relative/path/to/file.php http://www.yoursite.com/path/to/new/file.php redirect 301 /relative/path/to/file.php http://www.yoursite.com/path/to/new/file.php redirect 301 /relative/path/to/file.php http://www.yoursite.com/path/to/new/file.php >
IPULLRANK.COM @ IPULLRANK Internal Linking Structures Internal Linking Structures are still an incredibly powerful asset. I’d like to see more tools like what Portent has built to visualize them and understand where there is opportunity. https://www.portent.com/blog/seo/seo-force-directed-diagrams.htm
IPULLRANK.COM @ IPULLRANK Searchmetrics Does a Great Job of This I wish all the link indices had a stronger focus on internal linking structure and visualizing where there are opportunities to flow more link equity throughout the site. Searchmetrics’ Site Structure tool does this well.
IPULLRANK.COM @ IPULLRANK Paul’s Method is Good for Rolling Your Own http://searchengineland.com/improve-internal-linking-calculate-internal-pagerank-r-246883
If Possible, Always Update your internal links on the database level WordPress Example UPDATE wp_posts SET post_content = REPLACE (post_content, 'http://www.oldsiteurl.com', 'http://www.newsiteurl.com');
Google is becoming the presentation layer of the web. Help them do it.
IPULLRANK.COM @ IPULLRANK Schema.org / JSON-LD Schema.org is a lot more reasonable to implement now that they are allowing JSON-LD. Being in knowledge boxes and such yields a lot more traffic than it takes away.
Page Speed Google’s next big play
IPULLRANK.COM @ IPULLRANK Google Expects Ludicrously Speedy Google has an expectation that the above the fold content on the page will load within 1 second. You basically get 400 ms to make that happen.
IPULLRANK.COM @ IPULLRANK The Critical Rendering Path