Link extraction and classification

http://www.ﬂickr.com/photos/pio1976/3330670980/ Link extraction and classiﬁcation Bruno Pedro December 2010

Bruno PedroA n e x p e r i e n c e d We b d e v e l o p e r a n dentrepreneur. Has extensive background inlarge scale projects and technical writing.http://tarpipe.com/user/bpedro

What is tarpipe? User

What is tarpipe?

twitter source: mashable.com• Average ~1.1M tweets/hour• ~300 new tweets every second

Challenge• ~300 reads/second• 160 X 300 = 48 KB/second = 4 GB/day (approximate calculation)• How to process all this information?

Strategy• Read and store immediately: • High performance write storage • No locks allowed • Prepare for lots of reading errors

Strategy• Process later: • Regular expressions • Term extraction • Machine learning

Link extraction

Link extraction extract user extract URL new tweet yeslaunch process has URL? save show no dump

Content extraction • Short URLs • HTTP redirects • HTTP errors — retry? • Content analysis

Content extraction URL retry? yes yes nofetch contents redirect? error? yes no save

Content analysis• Assume malformed (X)HTML• Regular expressions or• Convert to XHTML• DOM traversing

Content classiﬁcation • HTML title, description, keywords tag cloud { • H1, H2, H3, ... • Paragraphs graphplacement { • Who shared the link? • Internal and external links

Content classiﬁcation (X)HTML extract head extract text extract text elements yes yes yes H1,H2,... paragraphshead found? save found? found? no no

The big picturenew tweet classify content save extract URL extract content

Food for thought• PubsubHubbub http://code.google.com/p/pubsubhubbub/• Activity Streams http://activitystrea.ms/• twitter streaming API http://dev.twitter.com/pages/streaming_api

tarpipe streamlines your tarpipe is one of the most Today I had a chance toupdates to various social curious experiments in spend time experimentingweb sites, creating simple social media that Ive with tarpipe and I have toor complex workﬂows to seen lately. The service say that I am intrigued byupdate several buckets in has the potential to be the concept and impressedone fell swoop. the answer to the lament by the implementation. I ﬁrst talked about in TheAdam Pash looming crisis: Personal Jeff Barrlifehacker syndication overload. Amazon.com Rafe Needleman CNET newsthank you automated publishing

Link extraction and classification

by Bruno Pedro on Dec 14, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 6

Statistics

Link extraction and classification — Presentation Transcript

http://www.howl.or.kr	3
http://admin.howl.or.kr	2
http://new.howl.or.kr	1