Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Like this presentation? Why not share!

- Data structures by Pranav Gupta 1048 views
- Page-Rank Algorithm Final by William Keene 399 views
- Pagerank and hits by Shatakirti Er 9265 views
- Maximum Likelihood Estimation by guestfee8698 2604 views
- Large Scale Graph Processing with A... by sscdotopen 17685 views
- Link analysis .. Data Mining by Mustafa Salam 1383 views

No Downloads

Total views

4,436

On SlideShare

0

From Embeds

0

Number of Embeds

89

Shares

0

Downloads

40

Comments

0

Likes

1

No embeds

No notes for slide

- 1. Importance of Web Pages PageRank Topic-Specific PageRank Hubs and Authorities Combatting Link Spam 1
- 2. PageRank Intuition: solve the recursive equation: “a page is important if important pages link to it.” In high-falutin’ terms: importance = the principal eigenvector of the transition matrix of the Web. A few fixups needed. 2
- 3. Transition Matrix of the Web Enumerate pages. Page i corresponds to row and column i. M [i, j ] = 1/n if page j links to n pages, including page i ; 0 if j does not link to i. M [i, j ] is the probability we’ll next be at page i if we are now at page j. 3
- 4. Example: Transition Matrix Suppose page j links to 3 pages, including i but not x. j i 1/3 x 0 4
- 5. Random Walks on the Web Suppose v is a vector whose i th component is the probability that each random walker is at page i at a certain time. If each walker follows a link from i at random, the probability distribution for walkers is then given by the vector M v. 5
- 6. Random Walks – (2) Starting from any vector v, the limit M (M (…M (M v ) …)) is the long-term distribution of walkers. Intuition: pages are important in proportion to how likely a walker is to be there. The math: limiting distribution = principal eigenvector of M = PageRank. 6
- 7. Example: The Web in 1839 y a m Yahoo y 1/2 1/2 0 a 1/2 0 1 m 0 1/2 0 Amazon M’soft 7
- 8. Solving The Equations Because there are no constant terms, the equations v = M v do not have a unique solution. In Web-sized examples, we cannot solve by Gaussian elimination anyway; we need to use relaxation (= iterative solution). Can work if you start with a fixed v. 8
- 9. Simulating a Random Walk Start with the vector v = [1, 1,…, 1] representing the idea that each Web page is given one unit of importance. Repeatedly apply the matrix M to v, allowing the importance to flow like a random walk. About 50 iterations is sufficient to estimate the limiting solution. 9
- 10. Example: Iterating Equations Equations v = M v : y = y /2 + a /2 Note: “=” is a = y /2 + m really “assignment.” m = a /2 y 1 1 5/4 9/8 6/5 a = 1 3/2 1 11/8 ... 6/5 m 1 1/2 3/4 1/2 3/5 10
- 11. The Walkers Yahoo Amazon M’soft 11
- 12. The Walkers Yahoo Amazon M’soft 12
- 13. The Walkers Yahoo Amazon M’soft 13
- 14. The Walkers Yahoo Amazon M’soft 14
- 15. In the Limit … Yahoo Amazon M’soft 15
- 16. Real-World Problems Some pages are “dead ends” (have no links out). Such a page causes importance to leak out. Other groups of pages are spider traps (all out-links are within the group). Eventually spider traps absorb all importance. 16
- 17. Microsoft Becomes Dead End y a m Yahoo y 1/2 1/2 0 a 1/2 0 0 m 0 1/2 0 Amazon M’soft 17
- 18. Example: Effect of Dead Ends Equations v = M v : y = y /2 + a /2 a = y /2 m = a /2 y 1 1 3/4 5/8 0 a = 1 1/2 1/2 3/8 ... 0 m 1 1/2 1/4 1/4 0 18
- 19. Microsoft Becomes a Dead End Yahoo Amazon M’soft 19
- 20. Microsoft Becomes a Dead End Yahoo Amazon M’soft 20
- 21. Microsoft Becomes a Dead End Yahoo Amazon M’soft 21
- 22. Microsoft Becomes a Dead End Yahoo Amazon M’soft 22
- 23. In the Limit … Yahoo Amazon M’soft 23
- 24. M’soft Becomes Spider Trap y a m Yahoo y 1/2 1/2 0 a 1/2 0 0 m 0 1/2 1 Amazon M’soft 24
- 25. Example: Effect of Spider Trap Equations v = M v : y = y /2 + a /2 a = y /2 m = a /2 + m y 1 1 3/4 5/8 0 a = 1 1/2 1/2 3/8 ... 0 m 1 3/2 7/4 2 3 25
- 26. Microsoft Becomes a Spider Trap Yahoo Amazon M’soft 26
- 27. Microsoft Becomes a Spider Trap Yahoo Amazon M’soft 27
- 28. Microsoft Becomes a Spider Trap Yahoo Amazon M’soft 28
- 29. In the Limit … Yahoo Amazon M’soft 29
- 30. PageRank Solution to Traps, Etc. “Tax” each page a fixed percentage at each interation. Add a fixed constant to all pages. Models a random walk with a fixed probability of leaving the system, and a fixed number of new walkers injected into the system at each step. 30
- 31. Example: Microsoft is a Spider Trap; 20% Tax Equations v = 0.8(M v ) + 0.2: y = 0.8(y /2 + a/2) + 0.2 a = 0.8(y /2) + 0.2 m = 0.8(a /2 + m) + 0.2 y 1 1.00 0.84 0.776 7/11 a = 1 0.60 0.60 0.536 . . . 5/11 m 1 1.40 1.56 1.688 21/11 31
- 32. General Case In this example, because there are no dead-ends, the total importance remains at 3. In examples with dead-ends, some importance leaks out, but total remains finite. 32
- 33. Solving the Equations Because there are constant terms, we can expect to solve small examples by Gaussian elimination. Web-sized examples still need to be solved by relaxation. 33
- 34. Finding a Good Starting Vector 1. Newton-like prediction of where components of the principal eigenvector are heading. 2. Take advantage of locality in the Web. Each technique can reduce the number of iterations by 50%. Important – PageRank takes time! 34
- 35. Predicting Component Values Three consecutive values for the importance of a page suggests where the limit might be. 1.0 Guess for the next round 0.7 0.6 0.55 35
- 36. Exploiting Substructure Pages from particular domains, hosts, or directories, like stanford.edu or infolab.stanford.edu/~ullman tend to have many internal links. Initialize PageRank using ranks within your local cluster, then ranking the clusters themselves. 36
- 37. Strategy Compute local PageRanks (in parallel?). Use local weights to establish weights on edges between clusters. Compute PageRank on graph of clusters. Initial rank of a page is the product of its local rank and the rank of its cluster. “Clusters” are appropriately sized regions with common domain or lower-level detail. 37
- 38. In Pictures 1.5 2.05 3.0 2.0 0.15 0.1 Local ranks Intercluster weights 0.05 Ranks of clusters Initial eigenvector 38
- 39. Topic-Specific Page Rank Goal: Evaluate Web pages not just according to their popularity, but by how close they are to a particular topic, e.g. “sports” or “history.” Allows search queries to be answered based on interests of the user. Example: Query Maccabi wants different pages depending on whether you are interested in sports or history. 39
- 40. Teleport Sets Assume each walker has a small probability of “teleporting” at any tick. Teleport can go to: 1. Any page with equal probability. As in the “taxation” scheme. 2. A topic-specific set of “relevant” pages (teleport set ). For topic-specific PageRank. 40
- 41. Example: Topic = Software Only Microsoft is in the teleport set. Assume 20% “tax.” I.e., probability of a teleport is 20%. 41
- 42. Only Microsoft in Teleport Set Yahoo Dr. Who’s phone booth. Amazon M’soft 42
- 43. Only Microsoft in Teleport Set Yahoo Amazon M’soft 43
- 44. Only Microsoft in Teleport Set Yahoo Amazon M’soft 44
- 45. Only Microsoft in Teleport Set Yahoo Amazon M’soft 45
- 46. Only Microsoft in Teleport Set Yahoo Amazon M’soft 46
- 47. Only Microsoft in Teleport Set Yahoo Amazon M’soft 47
- 48. Only Microsoft in Teleport Set Yahoo Amazon M’soft 48
- 49. Picking the Teleport Set 1. Choose the pages belonging to the topic in Open Directory. 2. “Learn” from examples the typical words in pages belonging to the topic; use pages heavy in those words as the teleport set. 49
- 50. Application: Link Spam Spam farmers today create networks of millions of pages designed to focus PageRank on a few undeserving pages. To minimize their influence, use a teleport set consisting of trusted pages only. Example: home pages of universities. 50
- 51. Hubs and Authorities Mutually recursive definition: A hub links to many authorities; An authority is linked to by many hubs. Authorities turn out to be places where information can be found. Example: course home pages. Hubs tell where the authorities are. Example: CS Dept. course-listing page. 51
- 52. Transition Matrix A H&A uses a matrix A [i, j ] = 1 if page i links to page j, 0 if not. AT, the transpose of A, is similar to the PageRank matrix M, but AT has 1’s where M has fractions. 52
- 53. Example: H&A Transition Matrix y a m Yahoo y 1 1 1 A= a 1 0 1 m 0 1 0 Amazon M’soft 53
- 54. Using Matrix A for H&A Powers of A and AT have elements of exponential size, so we need scale factors. Let h and a be vectors measuring the “hubbiness” and authority of each page. Equations: h = λAa; a = µAT h. Hubbiness = scaled sum of authorities of successor pages (out-links). Authority = scaled sum of hubbiness of predecessor pages (in-links). 54
- 55. Consequences of Basic Equations From h = λAa; a = µAT h we can derive: h = λµAAT h a = λµATA a Compute h and a by iteration, assuming initially each page has one unit of hubbiness and one unit of authority. Pick an appropriate value of λµ. 55
- 56. Example: Iterating H&A 111 110 321 212 A= 101 AT = 1 0 1 AAT= 2 2 0 ATA= 1 2 1 010 110 101 212 a(yahoo) = 1 5 24 114 . . . 1+√3 a(amazon) = 1 4 18 84 . . . 2 a(m’soft) = 1 5 24 114 . . . 1+√3 h(yahoo) = 1 6 28 132 . . . 1.000 h(amazon) = 1 4 20 96 . . . 0.735 h(m’soft) = 1 2 8 36 . . . 0.268 56
- 57. Solving H&A in Practice Iterate as for PageRank; don’t try to solve equations. But keep components within bounds. Example: scale to keep the largest component of the vector at 1. Trick: start with h = [1,1,…,1]; multiply by AT to get first a; scale, then multiply by A to get next h,… 57
- 58. Solving H&A – (2) You may be tempted to compute AAT and ATA first, then iterate these matrices as for PageRank. Bad, because these matrices are not nearly as sparse as A and AT. 58
- 59. H&A Versus PageRank If you talk to someone from IBM, they may tell you “IBM invented PageRank.” What they mean is that H&A was invented by Jon Kleinberg when he was at IBM. But these are not the same. H&A does not appear to be a substitute for PageRank. But may be used by Ask.com. 59
- 60. Spam on the Web Search has become the default gateway to the web. Very high premium to appear on the first page of search results. 60
- 61. What is Web Spam? Spamming = any action whose purpose is to boost a web page’s position in search engine results, without providing additional value. Spam = Web pages used for spamming. Approximately 10-15% of Web pages are spam. 61
- 62. Web-Spam Taxonomy Boosting techniques : Techniques for increasing the probability a Web page will be a highly ranked answer to a search query. Hiding techniques : Techniques to hide the use of boosting from humans and Web crawlers. 62
- 63. Hiding techniques Content hiding. Use same color for text and page background. Cloaking. Return different page to crawlers and browsers. Redirection. Redirects are followed by browsers but not crawlers. 63
- 64. Boosting Techniques Term spamming : Manipulating the text of Web pages in order to appear relevant to queries. • Why? You can run ads that are relevant to the query. Link spamming : Creating link structures that boost PageRank. 64
- 65. Term Spamming – (1) Repetition : of one or a few specific terms e.g., “free,” “cheap,” “Viagra.” Dumping of a large number of unrelated terms. E.g., copy entire dictionaries. 65
- 66. Term Spamming – (2) Weaving : Copy legitimate pages and insert spam terms at random positions. Phrase Stitching : Glue together sentences and phrases from different sources. • E.g., use the top-ranked pages on the topic you want to look like. 66
- 67. The Google Solution to Term Spamming In addition to PageRank, the original Google engine had another innovation: it trusted what people said about you in preference to what you said about yourself. Give more weight to words that appear in or near anchor text than to words that appear in the page itself. 67
- 68. The Google Solution – (2) Today, the Google formula for matching terms to documents involves over 250 factors. E.g., does the word appear in a header? As closely guarded as the formula for Coke. 68
- 69. Link Spam Three kinds of Web pages from a spammer’s point of view: 1. Own pages. • Completely controlled by spammer. 2. Accessible pages. • E.g., Web-log comment pages: spammer can post links to his pages. 3. Inaccessible pages. 69
- 70. Spam Farms – (1) Spammer’s goal: Maximize the PageRank of target page t. Technique: 1. Get as many links from accessible pages as possible to target page t. 2. Construct “link farm” to get PageRank multiplier effect. 70
- 71. Spam Farms – (2) Accessible Own 1 Inaccessible Inaccessible 2 t M Goal: boost PageRank of page t. One of the most common and effective organizations for a spam farm. 71
- 72. Analysis – (1) Accessible Own 1 Inaccessible Inaccessible 2 t M Suppose rank from accessible pages = x. PageRank of target page = y. Share of “tax” Taxation rate = 1-β. Rank of each “farm” page = βy/M + (1-β)/N. From t Size of 72 Web
- 73. Analysis – (2) Accessible Own Tax share 1 for t. Inaccessible Inaccessible 2 Very small; t ignore. M y = x + βM[βy/M + (1-β)/N] + (1-β)/N y = x + β2y + β(1-β)M/N y = x/(1-β2) + cM/N where c = β/(1+β) PageRank of each “farm” page 73
- 74. Analysis – (3) Accessible Own 1 Inaccessible Inaccessible 2 t M y = x/(1-β2) + cM/N where c = β/(1+β). For β = 0.85, 1/(1-β2)= 3.6. Multiplier effect for “acquired” page rank. By making M large, we can make y as large as we want. 74
- 75. Detecting Link-Spam Topic-specific PageRank, with a set of “trusted” pages as the teleport set is called TrustRank. Spam Mass = (PageRank – TrustRank)/PageRank. High spam mass means most of your PageRank comes from untrusted sources – you may be link-spam. 75
- 76. Picking the Trusted Set Two conflicting considerations: Human has to inspect each seed page, so seed set must be as small as possible. Must ensure every “good page” gets adequate TrustRank, so all good pages should be reachable from the trusted set by short paths. 76
- 77. Approaches to Picking the Trusted Set 1. Pick the top k pages by PageRank. It is almost impossible to get a spam page to the very top of the PageRank order. 2. Pick the home pages of universities. Domains like .edu are controlled. 77

No public clipboards found for this slide

×
### Save the most important slides with Clipping

Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.

Be the first to comment