This document discusses machine learning challenges posed by hypertext and the web. It presents two examples of applying machine learning to hypertext documents: 1) semi-supervised learning to classify topics of hypertext documents using both text and hyperlinks, and 2) classifying interconnected entities by labeling graphs with many classes. The author proposes models that combine text and link information to better learn from hypertext documents and address issues like "topic drift".

1. Machine learning for the Web: Applications and challenges Soumen Chakrabarti Center for Intelligent Internet Research Computer Science and Engineering IIT Bombay www.cse.iitb.ernet.in/~soumen

2. Traditional supervised learning Training instance Test instance Independent variables x mostly continuous, maybe categorical Predicted variable y discrete (classification) or continuous (regression) Statistical models, inference rules, or separators Learner Learner Prediction y

3. Traditional unsupervised learning No training / testing phases Input is a collection of records with independent attributes alone Measure of similarity Partition or cover instances using clusters with large “self-similarity” and small “cross-similarity” Hierarchical partitions Large self- similarity Small cross- similarity

4. Learning hypertext models Entities are pages, sites, paragraphs, links, people, bookmarks, clickstreams… Transformed into simple models and relations Vector space/bag-of-words Hyperlink graph Topic directories Discrete time series occurs(term, page, cnt) cites(page, page) is-a(topic, topic) example(topic, page)

5. Challenges Large feature space in raw data Structured data sets: 10s to 100s Text (Web): 50 to 100 thousand Most features not completely useless Feature elimination / selection not perfect Beyond linear transformations? Models used today are simplistic Good accuracy on simple labeling tasks Lose a lot of detail present in hypertext to fit known learning techniques

6. Challenges Complex, interrelated objects Not a structured tuple-like entity Explicit and implicit connections Document markup sub-structure Site boundaries and hyperlinks Placement in popular directories like Yahoo! Traditional distance measures are noisy How to combine diverse features? (Or, a link is worth a ? words) Unreliable clustering results

7. This session Semi-supervised clustering (Rich Caruana) Enhanced clustering via user feedback Kernel methods (Nello Cristianini) Modular learning systems for text and hypertext Reference matching(Andrew McCallum) Recovering and cleaning implicit citation graphs from unstructured data

8. This talk: Two examples Learning topics of hypertext documents Semi-supervised learning scenario Unified model of text and hyperlinks Enhanced accuracy of topic labeling Segmenting hierarchical tagged pages Topic distillation (hubs and authorities) Minimum description length segmentation Better focused topic distillation Extract relevant fragments from pages

9. Classifying interconnected entities Early examples: Some diseases have complex lineage dependency Robust edge detection in images How are topics interconnected in hypertext? Maximum likelihood graph labeling with many classes Finding edge pixels in a differentiated image ? ? ? ? ? ? .3 red .7 blue

10. Naïve Bayes classifiers Decide topic; topic c is picked with prior probability  ( c );  c  ( c ) = 1 Each c has parameters  ( c , t ) for terms t Coin with face probabilities  t  ( c , t ) = 1 Fix document length n ( d ) and toss coin Naïve yet effective; can use other algos Given c , probability of document is

11. Enhanced models for hypertext c =class, d =text, N =neighbors Text-only model: Pr( d | c ) Using neighbors’ text to judge my topic: Pr( d , d ( N ) | c ) Better recursive model: Pr( d , c ( N ) | c ) Relaxation labeling over Markov random fields Or, EM formulation ?

12. Hyperlink modeling boosts accuracy 9600 patents from 12 classes marked by USPTO Patents have text and prior art links Expand test patent to include neighborhood ‘ Forget’ and re-estimate fraction of neighbors’ classes (Even better for Yahoo)

13. Hyperlink Induced Topic Search Radius-1 expanded graph Response Keyword Search engine Query a = E T h h = Ea ‘ Hubs’ and ‘ authorities’ h a h h h a a a

14. “Topic drift” and various fixes Some hubs have ‘mixed’ content Authority ‘leaks’ through mixed hubs from good to bad pages Clever: match query with anchor text to favor some edges B&H: eliminate outlier documents Vector-space document model Centroid × Cut-off radius Query term Activation window ‘ Thick’ links

15. Document object model (DOM) Hierarchical graph model for semi-structured data Can extract reasonable DOM from HTML A fine-grained view of the Web Valuable because page boundaries are less meaningful now <html><head> <title>Portals</title> </head><body><ul> <li><a href=“…”>Yahoo</a></li> <li><a href=“…”>Lycos</a></li> </ul></body></html> html head body title ul li li a a

16. A model for hub generation Global hub score distribution  0 w.r.t. given query Authors use DOM nodes to specialize  0 into local  I At a certain ‘cut’ in the DOM tree, local distribution directly generates hub scores Global distribution Progressive ‘distortion’ Model frontier Other pages

17. Optimizing a cost measure H v v Reference distribution  0 Data encoding cost is roughly Distribution distortion cost is (for Poisson distribution)

18. Modified topic distillation algorithm Will this (non-linear) system converge? Will segmentation help in reducing drift? Initialize DOM graph Let only root set authority scores be 1 Repeat until reasonable convergence: Authority-to-hub score propagation MDL-based hub score smoothing Hub-to-authority score propagation Normalization of authority scores Segment and rank micro-hubs Present annotated results

19. Convergence 28 queries used in Clever and by B&H 366k macro-pages, 10M micro-links Rank converges within 15 iterations

20. Effect of micro-hub segmentation ‘ Expanded’ implies authority diffusion arrested As nodes outside rootset start participating in the distillation… #Expanded increases #Pruned decreases Prevents authority leaks via mixed hubs

21. Rank correlation with B&H Positively correlated Some negative deviations Pseudo- authorities downgraded by our algorithm These were earlier favored by mixed hubs (Axes not to same scale)

22. Conclusion Hypertext and the Web pose new modeling and algorithmic challenges Locality exists in many guises Diverse sources of information: text, links, markup, usage Unifying models needed Anecdotes suggest that synergy can be exploited