This document summarizes a lecture on search engines. It discusses the history of search engines like Google, Yahoo and Bing. It covers how search engines work by crawling websites, indexing pages and serving search results. Key factors that affect search engine rankings are discussed like on-page elements like titles, links and metadata as well as off-page elements like backlinks. The document provides an overview of the search engine results page and global search market shares.

1. 8 — Search Engines
From Code to Product
gidgreen.com/course

2. The Google bomb
From Code to Product Lecture 8 — Search Engines — Slide 2 gidgreen.com/course

3. Feeling lucky?
From Code to Product Lecture 8 — Search Engines — Slide 3 gidgreen.com/course

4. Lecture 8
• Introduction
• How search works
• Anatomy of HTML
• On-page factors
• Off-page factors
• PageRank
From Code to Product Lecture 8 — Search Engines — Slide 4 gidgreen.com/course

5. History of web search
• 1994 — WebCrawler — first full text
• 1994 — Yahoo — directory then portal
• 1995 — AltaVista — first big index
• 1997 — Google — link citation analysis
• 2000 — 2004 — Yahoo uses Google
• 2000 — Baidu — now leader in China
• 2006 — Microsoft Live Search
• 2009 — Bing, switched to by Yahoo
From Code to Product Lecture 8 — Search Engines — Slide 5 gidgreen.com/course

6. Importance of search
• US: 17.1B “core searches” in April 2012
– 55 per US citizen [comScore]
• 92% of online US adults use search
– 96% of college graduates
– 98% with income $75k+
• 70–80% ignore paid ads on right
– (but only 10% ignore ads on top)
• 80% of sessions begins with a search
From Code to Product Lecture 8 — Search Engines — Slide 6 gidgreen.com/course
Sources:comScore,PewInternetReport,UserCentric,PCMagazine,
http://www.searchenginejournal.com/24-eye-popping-seo-statistics/42665/

7. Googling google
From Code to Product Lecture 8 — Search Engines — Slide 7 gidgreen.com/course

8. Search as traffic source
From Code to Product Lecture 8 — Search Engines — Slide 8 gidgreen.com/course

9. Global market share
From Code to Product Lecture 8 — Search Engines — Slide 9 gidgreen.com/course
Google,
81.73%
Yahoo, 6.42%
Baidu, 5.65%
Bing, 4.15% Other,
2.05%
Global
Source:May2012figuresfromhttp://www.netmarketshare.com/

10. USA market share
From Code to Product Lecture 8 — Search Engines — Slide 10 gidgreen.com/course
Google,
76.57%
Bing, 10.46%
Yahoo, 9.83%
AOL,
1.47%
Ask, 1.33%
USA
Source:May2012figuresfromhttp://www.netmarketshare.com/

11. China market share
From Code to Product Lecture 8 — Search Engines — Slide 11 gidgreen.com/course
Baidu,
78.50%
Google,
16.60%
Sougou,
2.80%
SoSo, 1.40%
Others, 0.70%
China
Source:http://chineseseoshifu.com/china-search-engine-market-share/
Also: Japan,
Czech Republic,
South Korea,
Russia,

12. Search engine results page
From Code to Product Lecture 8 — Search Engines — Slide 12 gidgreen.com/course

13. Where do people look?
From Code to Product Lecture 8 — Search Engines — Slide 13 gidgreen.com/course

14. Where do people click?
From Code to Product Lecture 8 — Search Engines — Slide 14 gidgreen.com/course
http://www.seomoz.org/blog/mission-imposserpble-
establishing-clickthrough-rates

15. Black-hat vs white-hat
From Code to Product Lecture 8 — Search Engines — Slide 15 gidgreen.com/course
Black-hat SEO White-hat SEO
Tricking Google Working with Google
Hidden keywords Prominent keywords
Cloaking for search Structured for search
Content scraping Unique content
Link spam and farms Attracting links
Short-lived boost (maybe) Long-term results

16. Google’s recommendations
From Code to Product Lecture 8 — Search Engines — Slide 16 gidgreen.com/course
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35769

17. Lecture 8
• Introduction
• How search works
• Anatomy of HTML
• On-page factors
• Off-page factors
• PageRank
From Code to Product Lecture 8 — Search Engines — Slide 17 gidgreen.com/course

18. How search works
• Crawling
– Finding content to index
• Indexing
– Preparing content for search
• Searching
– Showing results to user
From Code to Product Lecture 8 — Search Engines — Slide 18 gidgreen.com/course

19. Basic crawling
• Create an empty URL queue (“frontier”)
• Add one good URL, e.g. wikipedia.org
• Repeat:
– Select random URL from queue
– Retrieve content for that URL
– Add links in content to queue
– Keep track to prevent repeat visits
From Code to Product Lecture 8 — Search Engines — Slide 19 gidgreen.com/course

20. Crawling issues
• Link prioritization
• Duplicate content
– Print versions, sorting
• Infinite loops
– Database-driven sites
• Revisiting pages
• Site overloading
• Parallelization
From Code to Product Lecture 8 — Search Engines — Slide 20 gidgreen.com/course

21. Indexing
From Code to Product Lecture 8 — Search Engines — Slide 21 gidgreen.com/course

22. Indexing
From Code to Product Lecture 8 — Search Engines — Slide 22 gidgreen.com/course

23. Inverted index
From Code to Product Lecture 8 — Search Engines — Slide 23 gidgreen.com/course
https://developer.apple.com/library/mac/#documentation/userexperience/
Conceptual/SearchKitConcepts/searchKit_basics/searchKit_basics.html

24. Other indexed information
• Page metadata
• More about words
– Prominence
– Position
– Frequency
• Links between pages
– Including anchor text
• Images, etc…
From Code to Product Lecture 8 — Search Engines — Slide 24 gidgreen.com/course

25. Other formats
From Code to Product Lecture 8 — Search Engines — Slide 25 gidgreen.com/course
http://support.google.com/webmasters/bin/
answer.py?hl=en&answer=35287
Forms?
Javascript?

26. Stemming, proximity, ANDs
From Code to Product Lecture 8 — Search Engines — Slide 26 gidgreen.com/course

27. Recent Google changes
• Aug 2012: sometimes 7 results
• May 2012: knowledge graph
• Jan 2012: top heavy ads penalty
• Nov 2011: rewarding freshness
• Feb 2011: hitting content farms
• Dec 2010: social media signals
• Dec 2009: real-time search
From Code to Product Lecture 8 — Search Engines — Slide 27 gidgreen.com/course
http://www.seomoz.org/google-algorithm-change

28. Google web history (2005–2009)
From Code to Product Lecture 8 — Search Engines — Slide 28 gidgreen.com/course

29. Search + your world (2012)
From Code to Product Lecture 8 — Search Engines — Slide 29 gidgreen.com/course
http://www.ubergizmo.com/2012/01/google-now-searches-your-world/

30. Keyword research
But: consider also long tail (referrer logs)
From Code to Product Lecture 8 — Search Engines — Slide 30 gidgreen.com/course

31. Keyword research
From Code to Product Lecture 8 — Search Engines — Slide 31 gidgreen.com/course

32. Lecture 8
• Introduction
• How search works
• Anatomy of HTML
• On-page factors
• Off-page factors
• PageRank
From Code to Product Lecture 8 — Search Engines — Slide 32 gidgreen.com/course

33. HTTP protocol
GET /wiki/Hypertext_Transfer_Protocol HTTP/1.1
Host: en.wikipedia.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/
534.50.2 (KHTML, like Gecko) Version/5.0.6 Safari/533.22.3
Referer: http://www.rexswain.com/httpview.html
Connection: close
HTTP/1.0 200 OK
Date: Sun, 17 Jun 2012 06:05:03 GMT
Server: Apache
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
Content-Language: en
Last-Modified: Sat, 16 Jun 2012 03:14:24 GMT
Content-Length: 164814
Content-Type: text/html; charset=UTF-8
Connection: close
From Code to Product Lecture 8 — Search Engines — Slide 33 gidgreen.com/course

34. Page structure
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01
Transitional//EN" "http://www.w3.org/TR/html4/
loose.dtd">
<html lang="en">
<head>
<title>Uber List Manager</title>
</head>
<body>
<h1>Uber List Manager</h1>
<p>The world's leading &amp; best priced list
management software.</p>
</body>
</html>
From Code to Product Lecture 8 — Search Engines — Slide 34 gidgreen.com/course

35. Key <HEAD> elements
<head>
<meta http-equiv="content-type" content="text/
html; charset=utf-8">
<link rel="stylesheet" type="text/css"
href="styles.css">
<script type="text/javascript" src=”script.js"></
script>
<title>Uber List Manager</title>
<meta name="description" content="An excellent
and well priced list management program.">
<meta name="keywords" content="lists, list
manager, uber, mailing software">
</head>
From Code to Product Lecture 8 — Search Engines — Slide 35 gidgreen.com/course

36. Key <BODY> elements
From Code to Product Lecture 8 — Search Engines — Slide 36 gidgreen.com/course
<body>
<h1>Uber List Manager</h1>
<p>The world's leading &amp; best priced list
management software.</p>
<h2>Features</h2>
<h2>Customer stories</h2>
<img src="images/ulm.jpeg" width="320" height="240"
alt="Screenshot" title="ULM in action">
<form action="form.php" method="post">
<input type="submit" value="Submit">
</form>
<iframe src="iframe.html" width="300"
height="300"></iframe>
</body>

37. External file summary
From Code to Product Lecture 8 — Search Engines — Slide 37 gidgreen.com/course
index.html
<html lang="en">
<head>
<title>
Uber List...
</title>
</head>
<body>
...
</body>
</html>
styles.css
body
{background-
color:yellow;}
h1 {font-size:
24px;}
a:hover {text-
decoration:
underline;}
script.js
document.onkey
down=okd;
window.onbefor
eunload=obl;
if (t>=0)
d=new Date();
Iframe.html
<html>
<head>
</head>
<body>
...
</body>
</html>
ulm.jpeg

38. Links
From Code to Product Lecture 8 — Search Engines — Slide 38 gidgreen.com/course
Click <a href="more-information.html">here</a> for
ULM benefits and pricing.
Click for <a href="more-information.html">ULM
benefits and pricing</a>.
Click for <a href="more-information.html" title="ULM
benefits and pricing">more about ULM</A>.
Better than <a href="http://slowlists.com/"
rel="nofollow">our competitors</a>.
<a href="pricing.html"><img src="dollar-bill.jpeg"
alt="Pricing"></a>

39. Internal targets
From Code to Product Lecture 8 — Search Engines — Slide 39 gidgreen.com/course
<a href="#history">2 History</a>
.
.
.
<h2 id="history">History</h2>
<a name="history"></
a><h2>History</h2>

40. Rich snippets
<span itemprop="reviewCount">808</span>
<span itemprop="streetAddress">204 E 43rd St</
span>
<span id="bizPhone" itemprop="telephone">(212)
972-1001</span>
<span itemprop="priceRange">$$$$</span>
<meta itemprop="ratingValue" content="4.5">
From Code to Product Lecture 8 — Search Engines — Slide 40 gidgreen.com/course

41. HTML5
<article>
<section>
<hgroup>
<aside>
<header>
<footer>
<nav>
From Code to Product Lecture 8 — Search Engines — Slide 41 gidgreen.com/course
http://www.michaelcropper.co.uk/2012/03/html5-
seo-best-practices-1032.html

42. Lecture 8
• Introduction
• How search works
• Anatomy of HTML
• On-page factors
• Off-page factors
• PageRank
From Code to Product Lecture 8 — Search Engines — Slide 42 gidgreen.com/course

43. Page title
From Code to Product Lecture 8 — Search Engines — Slide 43 gidgreen.com/course
No more than ~70 characters / 10 words

44. URLs
SEO Cheat Sheet: Anatomy of A URL
©2009 SEOmoz ·∙ www.seomoz.org ·∙ Read SEOmoz. Rank Be er.
http://store.example.com/topics/subtopic/descriptive-product-name#top
1 Protocol
Subdomain
Domain
Top-Level Domain
Folders / Paths
Page
Named Anchor
2
1
3
4
5
6
7
2 3 4 5 5 6 7
http://www.example.com/index.php?product=1234&sort=price&print=1
1 2 3 4 5 6 7
1 Protocol
Subdomain
Domain
Top-Level Domain
Page / File Name
File Extension
CGI Parameters
2
3
4
5
6
7
7 7
Popular TLDs2
.com
.net
.org
.edu
.info
.biz
.name
Popular ccTLDs*
.cn
.de
.uk
.nl
.eu
.ru
.ar
- China
- Germany
- United Kingdom
- Netherlands
- European Union
- Russian Federation
- Argentina
Popular Extensions
.htm
.html
.php
.asp
.aspx
.cfm
.jsp
- Static HTML
- Static HTML
- PHP code
- ASP code
- ASP.NET
- ColdFusion
- Java Code
Keyword Priority1
Observed Google priority
of keyword placement:
- commercial
- infrastructure
- non-profit
- schools
- informational
- small business
- personal sites
SEO-FRIENDLYURLOLDDYNAMICURL
(1) Domain
(2) Subdomain
(3) Folder
(4) Path/Page
1 SEOmoz correlational data (2009)
2 Verisign domain report (2009) * ccTLD = Country Code TLD
SEO Tips for URLs
• Use subdomains carefully.They may be treated as
separate entities,splitting domain authority.
• Separate path & page keywords with hyphens (”-”).
• Anchors may help engines understand page structure.
• Keyword effectiveness in URLs decreases as URL
length and keyword position increases.1
2
1
From Code to Product Lecture 8 — Search Engines — Slide 44 gidgreen.com/course
http://www.seomoz.org/blog/seo-cheat-
sheet-anatomy-of-a-url

45. URLs: good vs bad
www.really-cheap-great-mailing-list-manager.info
www.mailingmanager.com
googleblog.blogspot.com/view?
post_id=3982098&section_id=231
googleblog.blogspot.com/2012/04/introducing-google-
drive.html
amazon.com/store/products/books/computing/internet/
seo/Eric+Edge/The%20Art%20Of%20SEO/details
amazon.com/The-Art-SEO-Eric-Edge
From Code to Product Lecture 8 — Search Engines — Slide 45 gidgreen.com/course

46. Meta descriptions
From Code to Product Lecture 8 — Search Engines — Slide 46 gidgreen.com/course
Used for display but not for ranking
Length: 150~160 characters
Avoid duplication across many pages

47. Keyword density
From Code to Product Lecture 8 — Search Engines — Slide 47 gidgreen.com/course

48. Formatting
From Code to Product Lecture 8 — Search Engines — Slide 48 gidgreen.com/course
and <b>good value</b>
and <span style="font-weight:bold;">good value</span>
and <span class="emboldened">good value</span>
and <em>good value</em>
and <strong>good value</strong>
<font size="+2>Features</font>
<big>Features</big>
<p style="font-size:24px;">Features</p>
<h2>Features<h2>
<h2 style="font-size:24px;">Features</h2>

49. Freshness and speed
• Freshness determined by:
– Date the page appeared
– Frequency of content change
– Amount of content change
– Rate of new incoming links
From Code to Product Lecture 8 — Search Engines — Slide 49 gidgreen.com/course

50. Javascript and Flash
From Code to Product Lecture 8 — Search Engines — Slide 50 gidgreen.com/course

51. Lecture 8
• Introduction
• How search works
• Anatomy of HTML
• On-page factors
• Off-page factors
• PageRank
From Code to Product Lecture 8 — Search Engines — Slide 51 gidgreen.com/course

52. Links from external sites
• From high ranking sites
– Hard to manipulate
• From .edu or .gov
– No commercial motivation
• From topic-related sites
• From many sites
– Diversity of subject
– Different ownership / IP block
From Code to Product Lecture 8 — Search Engines — Slide 52 gidgreen.com/course

53. Links on external pages
• All-important anchor text
– First appearance counts
– Diversity of anchors
• Higher on linking page
• From core text content
– Not navigation/footers
– Image ALT text weaker
• Page has other good links
From Code to Product Lecture 8 — Search Engines — Slide 53 gidgreen.com/course

54. Power of anchors
From Code to Product Lecture 8 — Search Engines — Slide 54 gidgreen.com/course

55. Titles and URLs in anchors
Wikipedia, the free encyclopedia — 450 saves
Visit Wikipedia for more information
Recent referrers: en.wikipedia.org
http://en.wikipedia.org/wiki/Main_Page
From Code to Product Lecture 8 — Search Engines — Slide 55 gidgreen.com/course

56. Attracting links
• (Directories e.g. dmoz)
• Inbound marketing
– Great on-site content
– Post articles elsewhere
– Request reviews
• Viral marketing
– Banners + widgets
– Social network sharing
From Code to Product Lecture 8 — Search Engines — Slide 56 gidgreen.com/course

57. Link bait
From Code to Product Lecture 8 — Search Engines — Slide 57 gidgreen.com/course

58. Domain information
From Code to Product Lecture 8 — Search Engines — Slide 58 gidgreen.com/course

59. User monitoring
From Code to Product Lecture 8 — Search Engines — Slide 59 gidgreen.com/course

60. Duplicate content
• Other sites stealing your content
• www.domain.com vs domain.com
• domain.com/ vs domain.com/index.html
• Printer-friendly versions
• URL parameters
From Code to Product Lecture 8 — Search Engines — Slide 60 gidgreen.com/course

61. robots.txt
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/
User-agent: BadBot
Disallow: /
Sitemap: http://www.example.com/sitemap.xml
From Code to Product Lecture 8 — Search Engines — Slide 61 gidgreen.com/course
Or in <HEAD> of page:
<meta name="robots"
content="noindex">

62. XML sitemaps
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/
schemas/sitemap/0.9">
<url>
<loc>http://www.example.com/</loc>
<lastmod>2005-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
</urlset>
From Code to Product Lecture 8 — Search Engines — Slide 62 gidgreen.com/course

63. Redirects and rel=canonical
From Code to Product Lecture 8 — Search Engines — Slide 63 gidgreen.com/course
http://www.seomoz.org/blog/an-seos-guide-to-http-status-codes

64. Putting it all together
From Code to Product Lecture 8 — Search Engines — Slide 64 gidgreen.com/course
http://www.seomoz.org/article/search-ranking-factors#predictions

65. Lecture 8
• Introduction
• How search works
• Anatomy of HTML
• On-page factors
• Off-page factors
• PageRank
From Code to Product Lecture 8 — Search Engines — Slide 65 gidgreen.com/course

66. A random walk
From Code to Product Lecture 8 — Search Engines — Slide 66 gidgreen.com/course
A
B
C
D
E

67. Probability distribution
From Code to Product Lecture 8 — Search Engines — Slide 67 gidgreen.com/course
http://en.wikipedia.org/wiki/File:PageRanks-Example.svg

68. The maths
From Code to Product Lecture 8 — Search Engines — Slide 68 gidgreen.com/course
http://en.wikipedia.org/wiki/PageRank

69. PageRank sculpting?
From Code to Product Lecture 8 — Search Engines — Slide 69 gidgreen.com/course
http://www.seomoz.org/blog/google-says-yes-you-can-still-
sculpt-pagerank-no-you-cant-do-it-with-nofollow

70. PageRank in reality
• Domain authority signals
• Nofollow links are clicked by people
• Interval vs external links
• Paid link and link farm detection
• Removed from toolbar in 2009
From Code to Product Lecture 8 — Search Engines — Slide 70 gidgreen.com/course
https://sites.google.com/site/webmasterhelpforum/en/faq--crawling--indexing---ranking#pagerank

71. Tools
From Code to Product Lecture 8 — Search Engines — Slide 71 gidgreen.com/course