how_first_click_free_can_help_in_SEO

  • 1,393 views
Uploaded on

First click free a service by Google for restricted content based site. for example a financial data site or a market research site where research data are either paid or subject to access post …

First click free a service by Google for restricted content based site. for example a financial data site or a market research site where research data are either paid or subject to access post registration. How can these businesses take organic ranking advantage by installing first click free

More in: Marketing
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,393
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Amaliegade 3-5, 3. sal 1256 Copenhagen K Tel: 7027 8089 Mail: info@guava.dk Copenhagen, Wednesday, 25 May 2011 FIRST CLICK FREE IMPLEMENTATION Client: NE.SE Website: http://www.ne.se Report date: Version: Author: th 26 April, 2011 1.0 Mark Edmondson & Kristoffer Ewald (Guava DK) Introduction This report outlines a set of recommended updates for the existing website that will enable Google’s “First Click Free” implementation. (FCF) Version: 1.0- Created: April, 2011 - Last modified: May, 2011 Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 1
  • 2. SEO recommendations Page First Click Free Implementation........................................................................................................................................... 1 Introduction ....................................................................................................................................................................... 1 Technical Implementation ...................................................................................................................................................... 3 1. Detecting Referrals .................................................................................................................................................. 4 Verifying Googlebot with reverse DNS lookup .................................................................................................................. 5 2. Meta tag – noarchive................................................................................................................................................ 6 3. Robots.txt ............................................................................................................................................................... 7 4. Cookies for FCF ...................................................................................................................................................... 8 Implementing Flash LSO cookies ........................................................................................................................................ 9 USER ABUSE CONSIDERATIONS .................................................................................................................................... 10 3rd PARTY WEBSITES ....................................................................................................................................................... 12 Other Search Engines ....................................................................................................................................................... 12 Social Media websites ...................................................................................................................................................... 12 SEO considerations with backlinks .................................................................................................................................... 12 Partner Websites / Content Syndication .............................................................................................................................. 12 Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 2
  • 3. TECHNICAL IMPLEMENTATION First Click Free for Web Search - http://googlewebmastercentral.blogspot.com/2008/10/first-click-free-for-websearch.html This first section of the report focuses on the technical requirements to have Google’s First Click Free (FCF) working on the website. Google’s guidelines for First Click Free are: Google’s crawlers must be able to access the content of the site without the need for cookies or filling out forms Suggested implementation is to configure webservers to not serve the registration pages to the crawler Allow crawler in robots.txt Multi-paged articles must all be accessible for the user Page displayed must be identical to users to that which Google sees This document will also look at what ways to implement enhancements to this package such as: Implementing five articles per day with cookies Complementary meta-tags Using similar techniques for other traffic sources apart from Google Blocking user’s attempts to abuse FCF Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 3
  • 4. 1. Detecting Referrals The Referrer request-header - RFC: http://tools.ietf.org/html/rfc2616#section-14.36 All implementations of First Click Free need a mechanism to detect where the visitor has arrived from. This is preferably implemented by using the HTTP referrer in the server headers, rather than on-page scripting such as JavaScript, since this could be circumnavigated. For FCF, the referrer is recommended to look for user agents of the form: *.google.* Since this will cover the combinations of its search engines around the world (www.google.se , www.google.com, etc.) A full list of Google User agents is here: http://www.google.com/support/webmasters/bin/answer.py?answer=1061943 – this may be useful if looking to only give access to Googlebot-Video or Googlebot-Mobile, for example Using Apache and .htaccess as an example, this would typically be implemented similar to the following: #Have a 403 error document for Denied users with subscription page ErrorDocument 403 /subscribe-now.html #set the first click free variable SetEnvIfNoCase Referer "*google*" first_click_free=1 #Send first_click_free past registration screen, others get 403 <FilesMatch "(/subscription-content/.*)"> Order Deny, Allow Allow from env=first_click_free Deny from all </FilesMatch> The above is intended as example only, and it may be preferred to use a server side scripting language such as JSP to handle the referral requests – however, the principle remains the same: detect if the referral is from *.google.* and redirect past a subscribe request. Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 4
  • 5. Verifying Googlebot with reverse DNS lookup Google Help files: http://www.google.com/support/webmasters/bin/answer.py?answer=80553 Users could try to circumnavigate the restrictions by setting their browser’s User-Agent to Googlebot – to prevent this a reverse DNS lookup can be used to verify that the bot comes from googlebot.com and action taken accordingly. The example given in the link above is: > host 66.249.66.1 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. > host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 66.249.66.1 Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 5
  • 6. 2. Meta tag – noarchive NoArchive Information: http://noarchive.net/meta/ It is also recommended to place a “noarchive” meta tag on the pages under the FCF regime. This metatag tells Google not to keep a version of the webpage which could be accessed via a “cache:” command in Google to access page content: To prevent this, place this tag in each page to be available under FCF: <meta name="robots" content="noarchive" /> Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 6
  • 7. 3. Robots.txt http://www.robotstxt.org/ Ensure that Googlebot is allowed in the robots.txt file for all FCF content – that is the User-agent in the file found at ne.se/robots.txt does not block Googlebot – the current robots.txt file is: User-agent: * Disallow: /admin/ Disallow: /clusterjsp/ Disallow: /dev/ Disallow: /dns/ Disallow: /error/ Disallow: /modals/ Disallow: /resources/ Disallow: /temp/ Disallow: /xhp Disallow: /static/xhp Disallow: /google69eed29ee822c5b1.html Disallow: /ip.jsp Disallow: /newnormalpage.jsp Disallow: /testads.html Disallow: /testads.jsp Disallow: /testanalytics.jsp Disallow: /testlayout.jsp Disallow: /user/ Disallow: /openschool/ Disallow: /sok/ Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=f&to=l Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=l&to=q Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=q Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&to=f Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 7
  • 8. 4. Cookies for FCF Cookies are not strictly necessary for FCF, however may be needed if: FCF content is available across paginated articles, and a user needs to be tracked across a session The five (or more) articles a day per user implementation is required In such cases, a first party cookie is needed to be set to track users within and between sessions. For paginated articles a session cookie (expire=0) is all that is necessary For the Five clicks a day the expiration should be 24 hrs and include a counter on article views. Cookie Page Request Diagram Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 8
  • 9. Implementing Flash LSO cookies Using cookies for tracking users isn’t ideal as it can be circumnavigated quite easily – a user can use another browser, flush their browser cookies etc., however if the cookie information can be moved to a Flash LSO the cookie information is more rarely removed and can be more reliable. Guava have a solution that can provide this extra robustness in the cookies, which also allows more advanced implementation of Google Analytics tracking. How these files communicate between the Flash and the client javascript has been included with this report – please review this code and the examples provided. The example in the files (ne.se.example.html) shows very simple the methods of setting, retrieving and resetting a variable in the LSO using Javascript. Left out is the actual synchronisation with a possible browser cookie, but it should be fairly simple to build this upon the methods in the example: 1) If a browser cookie exists: Store the cookie value in the LSO 2) If a browser cookie does not exist but the LSO exists: Create browser cookie with the value in the LSO 3) If neither exists: Create browser cookie and store the value in the LSO Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 9
  • 10. USER ABUSE CONSIDERATIONS Some web-savvy web users may look to work around the measures put in place for First Click Free. An example of some techniques that are used against the New York Times is shown here http://www.makeuseof.com/tag/5-ways-york-times-paywall/ In this section we look at common methods and what technical measures can be in place to help prevent them: Disabling Javascript – if the referral or cookie setting scripts are done via JavaScript, simply turning JavaScript off will enable users to carry on as if the script wasn’t there. Likelihood: 5 out of 5 Prevention: Avoid security based on JavaScript. Faking Referrals – with plugins or other methods, a user can make his browser look like it is a Google crawler when surfing the site. This way any privileges given to Googlebot will also apply to that user, including access to FCF content. To prevent this, only techniques such as reverse DNS lookups to verify the User Agent will work. Likelihood: 2 out of 5 Prevention: reverse DNS lookup to verify user-agents Blocking/Flushing Cookies – the most common method, this will allow a user to navigate restrictions based on browser cookies, such as those tracking how many articles they have seen that session or day. If the Flash cookies LSO are used this will go a long way to stopping this occurring, since Flash cookies are a lot harder to remove. Also, tracking the user’s IP address as well and comparing them across cookies should flag up users who are holding multiple sessions. Likelihood: 4 out of 5 Prevention: Compare cookie to IP address, LSO flash cookies Using another Browser – this is related to the cookie issue above: if a user uses another browser such as Internet Explorer vs Firefox, then a new set of cookies are needed. The only way to prevent this if they are using the same PC is to also track the IP addresses and compare them across cookie sessions. Likelihood: 5 out of 5 Prevention: Compare cookie to IP address, LSO flash cookies Using Google search – a common method which prompted the Five Clicks a day policy, users could take advantage of the first click free from Google domains to do searches in Google that would bring up the content they knew the user could see. They could then click through and read the article at will. This would need to be prevented by tracking a user across sessions, either via IP address or cookies. Likelihood: 5 out of 5 Prevention: Cookies to limit access Using Google cache – whilst Google crawls the web, it keeps a copy of every page available in its search results in the Google cache (sitting in RAM on its numerous servers) – a user could access this information by using a “cache:www.urlto-look-at.com” command to get access to all content Google can see. The use of the “noarchive” meta-tag is meant to prevent this. Likelihood: 3 out of 5 Prevention: “noarchive” metatag Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 10
  • 11. Printing out the web article – A user may look to take the article and print out a hard copy using the browsers “Print” command. If this is not wanted for non-paying subscribers, look at using stylesheets to make the print page blank for non-logged in users – i.e. <style type="text/css" media="print"> body {visibility: hidden; display: none;} </style> Likelihood: 2 out of 5 Prevention: CSS Hacks Copy-Paste – A user may also use their PC to copy-paste article information and place in their own format – an even worse scenario will be for them to publish this information on the web giving away the content with no restrictions. Whilst little can be done to prevent this against a determined user, steps such as disabling right click on a page will prevent the casual user (Although create usability concerns) – these commands are usually still easily circumnavigated by turning JavaScript off. Other methods include making it harder by turning off the copy-paste highlighting for your page: ondragstart="return false" onselectstart="return false" Another solution is to place a transparent gif across the entire article that is copied instead when users try to. A final and more marketing focused solution would be to use something like the Tynt service, which inserts links back to the website when someone copy-pastes – in this way there may at least be a link back to your website. http://www.tynt.com/publisher-tools Likelihood: 5 out of 5 Prevention: Disabling right click? Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 11
  • 12. 3rd PARTY WEBSITES rd This section deals with how First Click Free affects other 3 party websites that are not Google, and suggested strategies on how www.ne.se can best deal with them. Other Search Engines Whilst Google is the dominate search engine for the moment, other search engines such as Bing and Yahoo are still used and should be catered for if at all possible. In many cases this will simply mean include those search engine User agents in the same process as Google, including cookies and referrals. The most pertinent user agents will be: Google Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Full list including mobile - http://www.google.com/support/webmasters/bin/answer.py?answer=1061943 Bing msnbot/1.1 (+http://search.msn.com/msnbot.htm) Yahoo Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Social Media websites Getting mentions on other social media sites can be good for marketing and page purposes, and it may be considered worthwhile payment in getting a social mention to give some free access to users who follow the link. Again, this could be restricted on a per user basis, or per referral by only allowing 10 visits per day from that site etc. SEO considerations with backlinks To compete well in SEO, backlinks are part of the essential mix. If the site restricts content then naturally less people will link to the content as it will be less useful for readers hitting a paywall. A good incentive to increase linking to the site would be to offer users viewing the content from that page a free view of the page. Factor in the cost of that link if it had to be bought for marketing and SEO purposes verses the revenue lost from potential subscribers, and it may look to be worthwhile. This could be coupled with only allowing users with a paid for subscription to be able to use the links for their own websites, ensuring that at least one subscriber has paid per link and also providing additional incentives to become a member of www.ne.se Partner Websites / Content Syndication Rather than let any website that links to www.ne.se gain free access, websites that are pre approved partners could appear on a “white-list” of websites that allow its users to see www.ne.se content. Effectively this is what Google will be receiving, and other websites will generally give better converting traffic than Google. Nuera Limited is registered in England number 3849708. Registered Office: 70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S. International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263 12