This is the class project of Stanford Univ. online course Mining Massive Datasets. The mission is to write a Hadoop program that calculates Pagerank for 2002 Google Programming Contest Web Graph Data.
2. HOW DOES GOOGLE FIGHT WITH
SPAMMERS ?
• The old version search engine usually
relies on the information (e.g.,word
frequency) shown on each page itself.
• A spammer who want to sell hisT-shirt
may create his own web page which has
words like“movie” 1000 times.But he can
make these words invisible by setting the
same color as the background.
• When you search“movie”,the old search
engine will find this page unbelievably
important,so you click it and only find his
ad forT-shirt.
• “While Google was not the first search
engine,it was the first able to defeat the
spammers who had made search almost
useless. ”
• The key innovation that Google has
introduced is a measurement of web page
importance,called PageRank.
3. PAGERANK IS ABOUT WEB LINKS.
WHY WEB LINKS?
• People usually like to add a tag or a link to
a page he/she thinks is correct,useful or
reliable.
• For spammers,they can create their own
page as whatever they like,but it’s usually
hard for them to ask other pages to link
to them.
• Even though he can create a link farm
where thousands of pages link to one
particular page which he want to emphasis,
that thousands of pages he has control are
still not linked by billions of web pages in
the out side of world.
For example,a
Chinese web user
who see the left
picture on site will
probably add a tag as
“MilkTea Beauty”
(A Chinese young
celebrity whose
reputation is disputed).
4. WHAT IS PAGERANK?• PageRank is a vector whose j th element is
the probability that a random surfer is
travelling at the j th web page at the final
static state.
• At the beginning,you can set each page
onto the same value ( Vj=1/N ). Then you
multiply the PageRank vectorV with
transition matrix M to get the next
moment’s probability distribution X.
• In the final state,PageRank will converge
and vector X will be the same as vectorV.
For web that does not contains dead end
or spider trap,vectorV now represents
the PageRank.
A B C D
A
B
C
D
J: from
I: to
5. SPIDER TRAP
• Once you come to page C, you have no
way to leave C. The random surfer get
trapped at page C, so that everything
becomes not random.
• Finally all the PageRank will be taken by
page C.
6. DEAD END • In the real situation,a page can be a
dead end (does not link to any other
pages).Once the random surfer
comes to a dead end,it stops
travelling and has no more chance to
go out to other pages,so the
random assumption is violated.
• The column correspond to it in
transition matrix will be an empty
column,for the previous definition.
• Keeping on multiplying this matrix
will leave nothing left.
7. TAXATION
For loop iterations:
𝑉1 = 𝜌 ∗ 𝑀 ∗ 𝑉0
𝑉1 = 𝑉1 + (1 − 𝑠𝑢𝑚(𝑉1))/𝑁
𝑉0 = 𝑉1
The modified version algorithm:
• The modification to solve the above 2
problems is adding a possibility 𝜌 that the
surfer will keep on going through the
links, so there is (1 − 𝜌) possibility the
surfer will teleport to random pages.
• This method is called taxation.
8. HOWEVER, THE REAL WEB HAS BILLIONS
OF PAGES, MULTIPLICATION BETWEEN
MATRIX AND VECTOR IS OVERHEAD.
• By using partitioned matrix and vector,the
calculation can be paralleled onto a
computing cluster that has more than
thousands of nodes.
• And such large a magnitude of computing
is usually managed by a mapreduce system,
like Hadoop.
beta=0 1 2 3 4
alpha=0
1
2
3
4
beta=
0
1
2
3
4
10. BEFORE THE PAGERANK CALCULATING
TRANSLATING THE WEB TO NUMBERS
• 𝐴 → 𝐵
• 𝐴 → 𝐶
• 𝐴 → 𝐷
• 𝐵 → 𝐴
• 𝐵 → 𝐷
• 𝐶 → 𝐴
• 𝐷 → 𝐵
• 𝐷 → 𝐶
• A 0
• B 1
• C 2
• D 3
LINKS ID
• Performing Inner Join twice, where
the 1st time’s key is FromNodeID,the
2nd time’s key isToNodeID.
• 𝐴, 𝐵, 0
• 𝐴, 𝐶, 0
• 𝐴, 𝐷, 0
• 𝐵, 𝐴, 1
• 𝐵, 𝐷, 1
• 𝐶, 𝐴, 2
• 𝐷, 𝐵, 3
• 𝐷, 𝐶, 3
• 𝐴, 𝐵, 0, 1
• 𝐴, 𝐶, 0, 2
• 𝐴, 𝐷, 0, 3
• 𝐵, 𝐴, 1, 0
• 𝐵, 𝐷, 1, 3
• 𝐶, 𝐴, 2, 0
• 𝐷, 𝐵, 3, 1
• 𝐷, 𝐶, 3, 2
After 1st
inner join
After 2nd
inner join
After the PageRank is
calculated,the same thing can
be done to translate index
back to node names.
From
Node
ID
To Node
ID
Web
Node ID
in data
Index used in
program
11. 2002 GOOGLE PROGRAMMING
CONTEST WEB GRAPH DATA
• 875713 pages, 5105039 edges
• 72 MB txt file
• Hadoop program iterates 75 times (“For
the web itself, 50-75 iterations are
sufficient to converge to within the error
limits of double precision”).
• 𝜌 = 0.85 as the possibility to follow the
web links and 0.15 possibility to teleport.
• The program has a structure of for loop,
each of which has 4 map-reduce job inside.
• The first 2 MR job are for matrix
multiplying vector.
• The 3rd MR job is to calculate the sum of
the product vector beta*M*V.
• And the final MR job does the shifting.
12. PAGERANK RESULT
• A Python program is written to compare the result from Hadoop:
13. RESULT ANALYSIS
• The value not sorted is noisy
and hard to see.
• But sorting by PageRank value and plotting in
log-log provides a linear line.
14. RESULT ANALYSIS
• The histogram has exponentially decaying
counts for large PageRankvalue.
• The largest 1/9 web pages contains 60% of
PageRank importance over the whole dataset.
15. REFERENCE
• Mining of Massive Datasets, Chapter 5
Jure Leskovec, Anand Rajaraman and Jeffrey D. Ullman
The code will be attached as the following files.
FINALLY, A TOP K PROGRAM IN HADOOP
• 1st column is the index used in this
program;
• 2nd column is the web node ID
within the original data;
• 3rd column is the PageRank value.
The right table shows the top 15
PageRank value.
