1. Data Mining Wikipedia Yongqian Li

3. Short term (successful): given a page, find related pages

4. I will give an overview of how I accomplished this and demo

6. Procedure:

7. Create sub-graph consisting of nodes in the query's neighborhood.

8. Hub(x) = Sum(Auth(y) where x->y)

9. Auth(x) = Sum(Hub(y) where y->x)

10. Normalized at end of iteration

11. Eventually converges to fixed point

13. Links are easier to create, so there is more “noise”

14. On the page “Internet”, there is link to “2007”

15. Many pages (about 5%) point to “2007”, so “2007” has high rank even in “Internet”-subgraph

17. Rank(“2007” | “Internet”) is still high

18. Maybe Bayesian concepts can be applied...

19. Calculate Rank(“2007” | “Internet”) / Rank(“2007”)

20. This works

23. Database – 18+ GiB – This is after heroic optimizations

24. Hard to debug when dealing with user-generated and large data sets.

25. Parsing, etc

27. “Given that the user has visited the pages “Unemployment rate”, “Resume”, and “Cover letter”, what can we infer?”

28. “What is the connection between “Internet” and “Economy”?”

29. “Internet -> Dotcom bubble -> Stock market -> Economy”, “Internet -> New economy -> Economy”, etc

30. Beat Google ;)