PageRank and The Google Matrix

Deriving “The Google Matrix”: G = αS + (1- α)1/neeT Lecture 4

 B.S Physics 1993, University of Washington M.S EE 1998, Washington State (four patents) 10+ Years in Search Marketing Founder of SEMJ.org (Research Journal) Blogger for SemanticWeb.com President of Future Farm Inc.

 Build a focused crawler in: Java, Python, PERL Point at MSU home page. Gather all the URLs and store for later use. http://www.montana.edu/robots.txt Store all the HTML and label with DocID. Read Google’s Paper. Next time Page Rank & the Google Matrix. Contest: Who can store the most unique URLS? Due Feb 7th (Next week). Send coded and URL list.

 #! /user/bin/python ### Basic Web Crawler in Python to Grab a URL from command line ## Use the urllib2 library for URLs, Use BeautifulSoup # from BeautifulSoup import BeautifulSoup import sys #allow users to input string import urllib2 ####change user-agent name from urllib import FancyURLopener class MyOpener(FancyURLopener): version = BadBot/1.0 print MyOpener.version # print the user agent name httpResponse = urllib2.urlopen(sys.argv[1])

 #store html page in an object called htmlPage htmlPage = httpResponse.read() print htmlPage htmlDom = BeautifulSoup(htmlPage) # dump page title print htmlDom.title.string # dump all links in page allLinks = htmlDom.findAll(a, {href: True}) for link in allLinks: print link[href]#Print name of Bot MyOpener.version

 Open source Java-based crawler https://webarchive.jira.com/wiki/display/H eritrix/Heritrix;jsessionid=AE9A595F01C AAB59BBCDC50C8A3ED2A9 http://www.robotstxt.org/robotstxt.html http://www.commoncrawl.org/

1 2 3 6 5 4

r(Pi) = Σr(Pj)/|Pj| PjΞBPi• r(Pi) is page rank of Page Pi• Pj is number of outlinks from page Pj• BPi is set of pages pointing into Pi

 r(Pj) values of Inlinking page is unknown. Need a starting value.  Could initialize the values to 1/n (number of pages) R0(Pi) = 1/n for all pages Pi Process is repeated until a stable value is obtained (Will not happen in all cases). Will this converge?

R k + 1(Pi) = Σrk(Pj)/|Pj| PjΞBPi• R k + 1 PageRank at of Pi at iteration K + 1• Ro(Pi) = 1/n, where in is all nodes• r(Pi) is page rank of Page Pi• Pj is number of outlinks from page Pj• BPi is set of pages pointing into Pi

Iteration 0 Iteration 1 Iteration2 Rankk= 2r0(P1) = 1/6 r1(P1) = 1/18 r2(P1) = 1/36 5r0(P2) = 1/6 r1(P2) = 5/36 r2(P2) = 1/18 4r0(P3) = 1/6 r1(P3) = 1/12 r2(P3) = 1/36 5r0(P4) = 1/6 r1(P4) = 1/4 r2(P4) = 17/72 1r0(P5) = 1/6 r1(P5) = 5/36 r2(P5) = 11/72 3r0(P6) = 1/6 r1(P6) = 1/6 R2(P6) = 14/72 2

[nxm] * [mxr] = nxr

• Non-zero row elements i are outlinking pages of page i• Non-zero column elements I are inlinking pages of page i

π (k + 1) T = π (k)T*HWhere: πT is a 1x n row vector

• Rank sinks & Convergence• Resembles work done on Markov Chains • H = transitional probability matrix • Converges to a unique positive vector if • Stochastic: Each row sum = 1 • Irreducible: Non-zero probability of transitioning (even if more than one state) to any other state. • Aperiodic: No requirements on how many steps to get to a state i. Can be irregular. • Primitive: Irreducible and Periodic

Next state depends on current state (no memory)

• “Random Surfer” Model • Following hyperlinks • Time spent on a page is proportional to its importance. • Fixes the “dangling node” problem. Surfer gets stuck on a node. Pdf files, images, etc. • Need to allow surfer to “teleport” or make random jumps.

S = H + a(1/n *eT)Where: ai = 1 if page i is dangling otherwise 0. eT(1x6) = all 1’s, n = number of nodes

Serendipity?: Page and Brin introduced an “adjustment”. Random Surfer can “teleport” and enter a new destination into a browser.

• Teleportation matrix: E = 1/n * eeT• α controls the proportion of time a “rand surfer” follows hyperlinks as opposed to teleporting. If = 0.5 then half the time is spent doing both.• At 0.5 about 34 iterations required to converge to a tolerance of 10^-10.• Originally set at 0.85. As it -> 1 computation time grows. Sensitivity issue.

G = αS + (1- α)1/nee T

π (k + 1) T = π (k)T*G*2002 World’s largest matrix computation. Order in 2002 ~8.1 x10^9 !

G = αH + (αa + (1-α)e)1/neT

PageRank and The Google Matrix

by sgollihe on Feb 01, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

PageRank and The Google Matrix — Presentation Transcript