SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
1.
http://www.flickr.com/photos/pio1976/3330670980/
Link extraction and classification
Bruno Pedro
December 2010
2.
Bruno Pedro
A n e x p e r i e n c e d We b d e v e l o p e r a n d
entrepreneur. Has extensive background in
large scale projects and technical writing.
http://tarpipe.com/user/bpedro
12.
Content extraction
URL retry?
yes
yes
no
fetch contents redirect? error?
yes
no
save
13.
Content analysis
• Assume malformed (X)HTML
• Regular expressions
or
• Convert to XHTML
• DOM traversing
14.
Content classification
• HTML title, description, keywords
tag
cloud
{ • H1, H2, H3, ...
• Paragraphs
graph
placement { • Who shared the link?
• Internal and external links
15.
Content classification
(X)HTML extract head
extract text extract text
elements
yes yes yes
H1,H2,... paragraphs
head found? save
found? found?
no no
16.
The big picture
new tweet classify content save
extract URL
extract content
17.
Food for thought
• PubsubHubbub
http://code.google.com/p/pubsubhubbub/
• Activity Streams
http://activitystrea.ms/
• twitter streaming API
http://dev.twitter.com/pages/streaming_api
18.
tarpipe streamlines your tarpipe is one of the most Today I had a chance to
updates to various social curious experiments in spend time experimenting
web sites, creating simple social media that I've with tarpipe and I have to
or complex workflows to seen lately. The service say that I am intrigued by
update several buckets in has the potential to be the concept and impressed
one fell swoop. the answer to the lament by the implementation.
I first talked about in The
Adam Pash looming crisis: Personal Jeff Barr
lifehacker syndication overload. Amazon.com
Rafe Needleman
CNET news
thank you automated publishing
Editor's Notes
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
0 likes
Be the first to like this
Views
Total views
3,492
On SlideShare
0
From Embeds
0
Number of Embeds
14
You have now unlocked unlimited access to 20M+ documents!
Unlimited Reading
Learn faster and smarter from top experts
Unlimited Downloading
Download to take your learnings offline and on the go
You also get free access to Scribd!
Instant access to millions of ebooks, audiobooks, magazines, podcasts and more.
Read and listen offline with any device.
Free access to premium services like Tuneln, Mubi and more.