Your SlideShare is downloading. ×
0
Importance of Web Pages

          PageRank
   Topic-Specific PageRank
     Hubs and Authorities
    Combatting Link Spam
...
PageRank
 Intuition: solve the recursive equation:
“a page is important if important pages
link to it.”
 In high-falutin’ ...
Transition Matrix of the Web
Enumerate pages.
Page i corresponds to row and column i.
M [i, j ] = 1/n if page j links to n...
Example: Transition Matrix
Suppose page j links to 3 pages, including i but not x.
                            j

        ...
Random Walks on the Web
Suppose v is a vector whose i th
component is the probability that each
random walker is at page i...
Random Walks – (2)
Starting from any vector v, the limit
M (M (…M (M v ) …)) is the long-term
distribution of walkers.
Int...
Example: The Web in 1839
                                 y a m
             Yahoo            y 1/2 1/2 0
                ...
Solving The Equations
Because there are no constant terms,
the equations v = M v do not have a
unique solution.
In Web-siz...
Simulating a Random Walk
Start with the vector v = [1, 1,…, 1]
representing the idea that each Web
page is given one unit ...
Example: Iterating Equations
Equations v = M v :
y = y /2 + a /2           Note: “=” is
a = y /2 + m              really “...
The Walkers
         Yahoo




Amazon           M’soft




                          11
The Walkers
         Yahoo




Amazon           M’soft




                          12
The Walkers
         Yahoo




Amazon           M’soft




                          13
The Walkers
         Yahoo




Amazon           M’soft




                          14
In the Limit …
         Yahoo




Amazon           M’soft




                          15
Real-World Problems
 Some pages are “dead ends” (have no
links out).
  Such a page causes importance to leak out.
Other gr...
Microsoft Becomes Dead End
                                  y a m
              Yahoo            y 1/2 1/2 0
            ...
Example: Effect of Dead Ends
 Equations v = M v :
 y = y /2 + a /2
 a = y /2
 m = a /2

  y       1    1     3/4   5/8    ...
Microsoft Becomes a Dead End
               Yahoo




      Amazon           M’soft




                                19
Microsoft Becomes a Dead End
               Yahoo




      Amazon           M’soft




                                20
Microsoft Becomes a Dead End
               Yahoo




      Amazon           M’soft




                                21
Microsoft Becomes a Dead End
               Yahoo




      Amazon           M’soft




                                22
In the Limit …
         Yahoo




Amazon           M’soft




                          23
M’soft Becomes Spider Trap
                                  y a m
              Yahoo            y 1/2 1/2 0
            ...
Example: Effect of Spider Trap
 Equations v = M v :
 y = y /2 + a /2
 a = y /2
 m = a /2 + m

  y       1    1     3/4   5...
Microsoft Becomes a Spider Trap
                Yahoo




       Amazon           M’soft




                             ...
Microsoft Becomes a Spider Trap
                Yahoo




       Amazon           M’soft




                             ...
Microsoft Becomes a Spider Trap
                Yahoo




       Amazon           M’soft




                             ...
In the Limit …
         Yahoo




Amazon           M’soft




                          29
PageRank Solution to Traps, Etc.
   “Tax” each page a fixed percentage at
  each interation.
   Add a fixed constant to al...
Example: Microsoft is a Spider
      Trap; 20% Tax
 Equations v = 0.8(M v ) + 0.2:
  y = 0.8(y /2 + a/2) + 0.2
  a = 0.8(y...
General Case
 In this example, because there are no
dead-ends, the total importance
remains at 3.
 In examples with dead-e...
Solving the Equations
Because there are constant terms, we
can expect to solve small examples by
Gaussian elimination.
Web...
Finding a Good Starting Vector
1. Newton-like prediction of where
   components of the principal
   eigenvector are headin...
Predicting Component Values
 Three consecutive values for the
importance of a page suggests where
the limit might be.
  1....
Exploiting Substructure
Pages from particular domains, hosts,
or directories, like stanford.edu or
infolab.stanford.edu/~u...
Strategy
 Compute local PageRanks (in parallel?).
 Use local weights to establish weights on
edges between clusters.
 Comp...
In Pictures
         1.5                     2.05


      3.0 2.0

          0.15      0.1

Local ranks
Intercluster weigh...
Topic-Specific Page Rank
Goal: Evaluate Web pages not just
according to their popularity, but by
how close they are to a p...
Teleport Sets
Assume each walker has a small
probability of “teleporting” at any tick.
Teleport can go to:
1. Any page wit...
Example: Topic = Software
Only Microsoft is in the teleport set.
Assume 20% “tax.”
  I.e., probability of a teleport is 20...
Only Microsoft in Teleport Set
               Yahoo

                                Dr. Who’s
                           ...
Only Microsoft in Teleport Set
               Yahoo




      Amazon           M’soft




                                ...
Only Microsoft in Teleport Set
               Yahoo




      Amazon           M’soft




                                ...
Only Microsoft in Teleport Set
               Yahoo




      Amazon           M’soft




                                ...
Only Microsoft in Teleport Set
               Yahoo




      Amazon           M’soft




                                ...
Only Microsoft in Teleport Set
               Yahoo




      Amazon           M’soft




                                ...
Only Microsoft in Teleport Set
               Yahoo




      Amazon           M’soft




                                ...
Picking the Teleport Set
1. Choose the pages belonging to the
   topic in Open Directory.
2. “Learn” from examples the typ...
Application: Link Spam
Spam farmers today create networks of
millions of pages designed to focus
PageRank on a few undeser...
Hubs and Authorities
Mutually recursive definition:
  A hub links to many authorities;
  An authority is linked to by many...
Transition Matrix A
 H&A uses a matrix A [i, j ] = 1 if page i
links to page j, 0 if not.
 AT, the transpose of A, is simi...
Example: H&A Transition Matrix
                                 y   a m
                Yahoo      y     1   1   1
       ...
Using Matrix A for H&A
Powers of A and AT have elements of
exponential size, so we need scale factors.
Let h and a be vect...
Consequences of Basic Equations
  From h = λAa; a = µAT h we can
  derive:
    h = λµAAT h
    a = λµATA a
  Compute h and...
Example: Iterating H&A

   111         110          321              212
A= 101    AT = 1 0 1   AAT= 2 2 0       ATA= 1 2 ...
Solving H&A in Practice
Iterate as for PageRank; don’t try to
solve equations.
But keep components within bounds.
  Exampl...
Solving H&A – (2)
You may be tempted to compute AAT
and ATA first, then iterate these
matrices as for PageRank.
Bad, becau...
H&A Versus PageRank
If you talk to someone from IBM, they
may tell you “IBM invented PageRank.”
  What they mean is that H...
Spam on the Web
Search has become the default
gateway to the web.
Very high premium to appear on the
first page of search ...
What is Web Spam?
Spamming = any action whose purpose
is to boost a web page’s position in
search engine results, without ...
Web-Spam Taxonomy
Boosting techniques :
 Techniques for increasing the probability a
 Web page will be a highly ranked ans...
Hiding techniques
Content hiding.
  Use same color for text and page background.
Cloaking.
  Return different page to craw...
Boosting Techniques
Term spamming :
 Manipulating the text of Web pages in
 order to appear relevant to queries.
  • Why? ...
Term Spamming – (1)
Repetition : of one or a few specific
terms e.g., “free,” “cheap,” “Viagra.”
Dumping of a large number...
Term Spamming – (2)
Weaving :
  Copy legitimate pages and insert spam
  terms at random positions.
Phrase Stitching :
  Gl...
The Google Solution to Term
        Spamming
 In addition to PageRank, the original
Google engine had another innovation:
...
The Google Solution – (2)
Today, the Google formula for
matching terms to documents involves
over 250 factors.
  E.g., doe...
Link Spam
Three kinds of Web pages from a
spammer’s point of view:
1. Own pages.
  •   Completely controlled by spammer.
2...
Spam Farms – (1)
Spammer’s goal:
   Maximize the PageRank of target page t.
Technique:
1. Get as many links from accessibl...
Spam Farms – (2)
                Accessible       Own

                                       1
Inaccessible
 Inaccessible...
Analysis – (1)
                     Accessible        Own
                                             1
   Inaccessible
 ...
Analysis – (2)
                     Accessible       Own       Tax share
                                            1   f...
Analysis – (3)
                    Accessible       Own
                                           1
  Inaccessible
   Ina...
Detecting Link-Spam
Topic-specific PageRank, with a set of
“trusted” pages as the teleport set is
called TrustRank.
Spam M...
Picking the Trusted Set
Two conflicting considerations:
 Human has to inspect each seed page, so
 seed set must be as smal...
Approaches to Picking the
         Trusted Set
1. Pick the top k pages by PageRank.
     It is almost impossible to get a ...
Upcoming SlideShare
Loading in...5
×

Page rank

2,811

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,811
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
25
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Page rank"

  1. 1. Importance of Web Pages PageRank Topic-Specific PageRank Hubs and Authorities Combatting Link Spam 1
  2. 2. PageRank Intuition: solve the recursive equation: “a page is important if important pages link to it.” In high-falutin’ terms: importance = the principal eigenvector of the transition matrix of the Web. A few fixups needed. 2
  3. 3. Transition Matrix of the Web Enumerate pages. Page i corresponds to row and column i. M [i, j ] = 1/n if page j links to n pages, including page i ; 0 if j does not link to i. M [i, j ] is the probability we’ll next be at page i if we are now at page j. 3
  4. 4. Example: Transition Matrix Suppose page j links to 3 pages, including i but not x. j i 1/3 x 0 4
  5. 5. Random Walks on the Web Suppose v is a vector whose i th component is the probability that each random walker is at page i at a certain time. If each walker follows a link from i at random, the probability distribution for walkers is then given by the vector M v. 5
  6. 6. Random Walks – (2) Starting from any vector v, the limit M (M (…M (M v ) …)) is the long-term distribution of walkers. Intuition: pages are important in proportion to how likely a walker is to be there. The math: limiting distribution = principal eigenvector of M = PageRank. 6
  7. 7. Example: The Web in 1839 y a m Yahoo y 1/2 1/2 0 a 1/2 0 1 m 0 1/2 0 Amazon M’soft 7
  8. 8. Solving The Equations Because there are no constant terms, the equations v = M v do not have a unique solution. In Web-sized examples, we cannot solve by Gaussian elimination anyway; we need to use relaxation (= iterative solution). Can work if you start with a fixed v. 8
  9. 9. Simulating a Random Walk Start with the vector v = [1, 1,…, 1] representing the idea that each Web page is given one unit of importance. Repeatedly apply the matrix M to v, allowing the importance to flow like a random walk. About 50 iterations is sufficient to estimate the limiting solution. 9
  10. 10. Example: Iterating Equations Equations v = M v : y = y /2 + a /2 Note: “=” is a = y /2 + m really “assignment.” m = a /2 y 1 1 5/4 9/8 6/5 a = 1 3/2 1 11/8 ... 6/5 m 1 1/2 3/4 1/2 3/5 10
  11. 11. The Walkers Yahoo Amazon M’soft 11
  12. 12. The Walkers Yahoo Amazon M’soft 12
  13. 13. The Walkers Yahoo Amazon M’soft 13
  14. 14. The Walkers Yahoo Amazon M’soft 14
  15. 15. In the Limit … Yahoo Amazon M’soft 15
  16. 16. Real-World Problems Some pages are “dead ends” (have no links out). Such a page causes importance to leak out. Other groups of pages are spider traps (all out-links are within the group). Eventually spider traps absorb all importance. 16
  17. 17. Microsoft Becomes Dead End y a m Yahoo y 1/2 1/2 0 a 1/2 0 0 m 0 1/2 0 Amazon M’soft 17
  18. 18. Example: Effect of Dead Ends Equations v = M v : y = y /2 + a /2 a = y /2 m = a /2 y 1 1 3/4 5/8 0 a = 1 1/2 1/2 3/8 ... 0 m 1 1/2 1/4 1/4 0 18
  19. 19. Microsoft Becomes a Dead End Yahoo Amazon M’soft 19
  20. 20. Microsoft Becomes a Dead End Yahoo Amazon M’soft 20
  21. 21. Microsoft Becomes a Dead End Yahoo Amazon M’soft 21
  22. 22. Microsoft Becomes a Dead End Yahoo Amazon M’soft 22
  23. 23. In the Limit … Yahoo Amazon M’soft 23
  24. 24. M’soft Becomes Spider Trap y a m Yahoo y 1/2 1/2 0 a 1/2 0 0 m 0 1/2 1 Amazon M’soft 24
  25. 25. Example: Effect of Spider Trap Equations v = M v : y = y /2 + a /2 a = y /2 m = a /2 + m y 1 1 3/4 5/8 0 a = 1 1/2 1/2 3/8 ... 0 m 1 3/2 7/4 2 3 25
  26. 26. Microsoft Becomes a Spider Trap Yahoo Amazon M’soft 26
  27. 27. Microsoft Becomes a Spider Trap Yahoo Amazon M’soft 27
  28. 28. Microsoft Becomes a Spider Trap Yahoo Amazon M’soft 28
  29. 29. In the Limit … Yahoo Amazon M’soft 29
  30. 30. PageRank Solution to Traps, Etc. “Tax” each page a fixed percentage at each interation. Add a fixed constant to all pages. Models a random walk with a fixed probability of leaving the system, and a fixed number of new walkers injected into the system at each step. 30
  31. 31. Example: Microsoft is a Spider Trap; 20% Tax Equations v = 0.8(M v ) + 0.2: y = 0.8(y /2 + a/2) + 0.2 a = 0.8(y /2) + 0.2 m = 0.8(a /2 + m) + 0.2 y 1 1.00 0.84 0.776 7/11 a = 1 0.60 0.60 0.536 . . . 5/11 m 1 1.40 1.56 1.688 21/11 31
  32. 32. General Case In this example, because there are no dead-ends, the total importance remains at 3. In examples with dead-ends, some importance leaks out, but total remains finite. 32
  33. 33. Solving the Equations Because there are constant terms, we can expect to solve small examples by Gaussian elimination. Web-sized examples still need to be solved by relaxation. 33
  34. 34. Finding a Good Starting Vector 1. Newton-like prediction of where components of the principal eigenvector are heading. 2. Take advantage of locality in the Web. Each technique can reduce the number of iterations by 50%. Important – PageRank takes time! 34
  35. 35. Predicting Component Values Three consecutive values for the importance of a page suggests where the limit might be. 1.0 Guess for the next round 0.7 0.6 0.55 35
  36. 36. Exploiting Substructure Pages from particular domains, hosts, or directories, like stanford.edu or infolab.stanford.edu/~ullman tend to have many internal links. Initialize PageRank using ranks within your local cluster, then ranking the clusters themselves. 36
  37. 37. Strategy Compute local PageRanks (in parallel?). Use local weights to establish weights on edges between clusters. Compute PageRank on graph of clusters. Initial rank of a page is the product of its local rank and the rank of its cluster. “Clusters” are appropriately sized regions with common domain or lower-level detail. 37
  38. 38. In Pictures 1.5 2.05 3.0 2.0 0.15 0.1 Local ranks Intercluster weights 0.05 Ranks of clusters Initial eigenvector 38
  39. 39. Topic-Specific Page Rank Goal: Evaluate Web pages not just according to their popularity, but by how close they are to a particular topic, e.g. “sports” or “history.” Allows search queries to be answered based on interests of the user. Example: Query Maccabi wants different pages depending on whether you are interested in sports or history. 39
  40. 40. Teleport Sets Assume each walker has a small probability of “teleporting” at any tick. Teleport can go to: 1. Any page with equal probability. As in the “taxation” scheme. 2. A topic-specific set of “relevant” pages (teleport set ). For topic-specific PageRank. 40
  41. 41. Example: Topic = Software Only Microsoft is in the teleport set. Assume 20% “tax.” I.e., probability of a teleport is 20%. 41
  42. 42. Only Microsoft in Teleport Set Yahoo Dr. Who’s phone booth. Amazon M’soft 42
  43. 43. Only Microsoft in Teleport Set Yahoo Amazon M’soft 43
  44. 44. Only Microsoft in Teleport Set Yahoo Amazon M’soft 44
  45. 45. Only Microsoft in Teleport Set Yahoo Amazon M’soft 45
  46. 46. Only Microsoft in Teleport Set Yahoo Amazon M’soft 46
  47. 47. Only Microsoft in Teleport Set Yahoo Amazon M’soft 47
  48. 48. Only Microsoft in Teleport Set Yahoo Amazon M’soft 48
  49. 49. Picking the Teleport Set 1. Choose the pages belonging to the topic in Open Directory. 2. “Learn” from examples the typical words in pages belonging to the topic; use pages heavy in those words as the teleport set. 49
  50. 50. Application: Link Spam Spam farmers today create networks of millions of pages designed to focus PageRank on a few undeserving pages. To minimize their influence, use a teleport set consisting of trusted pages only. Example: home pages of universities. 50
  51. 51. Hubs and Authorities Mutually recursive definition: A hub links to many authorities; An authority is linked to by many hubs. Authorities turn out to be places where information can be found. Example: course home pages. Hubs tell where the authorities are. Example: CS Dept. course-listing page. 51
  52. 52. Transition Matrix A H&A uses a matrix A [i, j ] = 1 if page i links to page j, 0 if not. AT, the transpose of A, is similar to the PageRank matrix M, but AT has 1’s where M has fractions. 52
  53. 53. Example: H&A Transition Matrix y a m Yahoo y 1 1 1 A= a 1 0 1 m 0 1 0 Amazon M’soft 53
  54. 54. Using Matrix A for H&A Powers of A and AT have elements of exponential size, so we need scale factors. Let h and a be vectors measuring the “hubbiness” and authority of each page. Equations: h = λAa; a = µAT h. Hubbiness = scaled sum of authorities of successor pages (out-links). Authority = scaled sum of hubbiness of predecessor pages (in-links). 54
  55. 55. Consequences of Basic Equations From h = λAa; a = µAT h we can derive: h = λµAAT h a = λµATA a Compute h and a by iteration, assuming initially each page has one unit of hubbiness and one unit of authority. Pick an appropriate value of λµ. 55
  56. 56. Example: Iterating H&A 111 110 321 212 A= 101 AT = 1 0 1 AAT= 2 2 0 ATA= 1 2 1 010 110 101 212 a(yahoo) = 1 5 24 114 . . . 1+√3 a(amazon) = 1 4 18 84 . . . 2 a(m’soft) = 1 5 24 114 . . . 1+√3 h(yahoo) = 1 6 28 132 . . . 1.000 h(amazon) = 1 4 20 96 . . . 0.735 h(m’soft) = 1 2 8 36 . . . 0.268 56
  57. 57. Solving H&A in Practice Iterate as for PageRank; don’t try to solve equations. But keep components within bounds. Example: scale to keep the largest component of the vector at 1. Trick: start with h = [1,1,…,1]; multiply by AT to get first a; scale, then multiply by A to get next h,… 57
  58. 58. Solving H&A – (2) You may be tempted to compute AAT and ATA first, then iterate these matrices as for PageRank. Bad, because these matrices are not nearly as sparse as A and AT. 58
  59. 59. H&A Versus PageRank If you talk to someone from IBM, they may tell you “IBM invented PageRank.” What they mean is that H&A was invented by Jon Kleinberg when he was at IBM. But these are not the same. H&A does not appear to be a substitute for PageRank. But may be used by Ask.com. 59
  60. 60. Spam on the Web Search has become the default gateway to the web. Very high premium to appear on the first page of search results. 60
  61. 61. What is Web Spam? Spamming = any action whose purpose is to boost a web page’s position in search engine results, without providing additional value. Spam = Web pages used for spamming. Approximately 10-15% of Web pages are spam. 61
  62. 62. Web-Spam Taxonomy Boosting techniques : Techniques for increasing the probability a Web page will be a highly ranked answer to a search query. Hiding techniques : Techniques to hide the use of boosting from humans and Web crawlers. 62
  63. 63. Hiding techniques Content hiding. Use same color for text and page background. Cloaking. Return different page to crawlers and browsers. Redirection. Redirects are followed by browsers but not crawlers. 63
  64. 64. Boosting Techniques Term spamming : Manipulating the text of Web pages in order to appear relevant to queries. • Why? You can run ads that are relevant to the query. Link spamming : Creating link structures that boost PageRank. 64
  65. 65. Term Spamming – (1) Repetition : of one or a few specific terms e.g., “free,” “cheap,” “Viagra.” Dumping of a large number of unrelated terms. E.g., copy entire dictionaries. 65
  66. 66. Term Spamming – (2) Weaving : Copy legitimate pages and insert spam terms at random positions. Phrase Stitching : Glue together sentences and phrases from different sources. • E.g., use the top-ranked pages on the topic you want to look like. 66
  67. 67. The Google Solution to Term Spamming In addition to PageRank, the original Google engine had another innovation: it trusted what people said about you in preference to what you said about yourself. Give more weight to words that appear in or near anchor text than to words that appear in the page itself. 67
  68. 68. The Google Solution – (2) Today, the Google formula for matching terms to documents involves over 250 factors. E.g., does the word appear in a header? As closely guarded as the formula for Coke. 68
  69. 69. Link Spam Three kinds of Web pages from a spammer’s point of view: 1. Own pages. • Completely controlled by spammer. 2. Accessible pages. • E.g., Web-log comment pages: spammer can post links to his pages. 3. Inaccessible pages. 69
  70. 70. Spam Farms – (1) Spammer’s goal: Maximize the PageRank of target page t. Technique: 1. Get as many links from accessible pages as possible to target page t. 2. Construct “link farm” to get PageRank multiplier effect. 70
  71. 71. Spam Farms – (2) Accessible Own 1 Inaccessible Inaccessible 2 t M Goal: boost PageRank of page t. One of the most common and effective organizations for a spam farm. 71
  72. 72. Analysis – (1) Accessible Own 1 Inaccessible Inaccessible 2 t M Suppose rank from accessible pages = x. PageRank of target page = y. Share of “tax” Taxation rate = 1-β. Rank of each “farm” page = βy/M + (1-β)/N. From t Size of 72 Web
  73. 73. Analysis – (2) Accessible Own Tax share 1 for t. Inaccessible Inaccessible 2 Very small; t ignore. M y = x + βM[βy/M + (1-β)/N] + (1-β)/N y = x + β2y + β(1-β)M/N y = x/(1-β2) + cM/N where c = β/(1+β) PageRank of each “farm” page 73
  74. 74. Analysis – (3) Accessible Own 1 Inaccessible Inaccessible 2 t M y = x/(1-β2) + cM/N where c = β/(1+β). For β = 0.85, 1/(1-β2)= 3.6. Multiplier effect for “acquired” page rank. By making M large, we can make y as large as we want. 74
  75. 75. Detecting Link-Spam Topic-specific PageRank, with a set of “trusted” pages as the teleport set is called TrustRank. Spam Mass = (PageRank – TrustRank)/PageRank. High spam mass means most of your PageRank comes from untrusted sources – you may be link-spam. 75
  76. 76. Picking the Trusted Set Two conflicting considerations: Human has to inspect each seed page, so seed set must be as small as possible. Must ensure every “good page” gets adequate TrustRank, so all good pages should be reachable from the trusted set by short paths. 76
  77. 77. Approaches to Picking the Trusted Set 1. Pick the top k pages by PageRank. It is almost impossible to get a spam page to the very top of the PageRank order. 2. Pick the home pages of universities. Domains like .edu are controlled. 77
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×