212 building googlebot - deview - google drive

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Building Googlebot
Youngjin Kim
October 15, 2013

http://www.creditwritedowns.com/2011/07/european-monetary-union-titanic.html

From the web to your query
● Query processing
1. Lookup keywords in the index => every relevant page
2. Rank pages and dis...

All of the pages on the web!?!

60 Trillion Pages And Counting!

Our local copy of the web
● Crawling
○ Googlebot
● Storage
○ Google File System (GFS), BigTable
● Processing
○ MapReduce
●...

Finding every page with googlebot
● Basic discovery crawl
1. Start with the set
of known links
2. Crawl every link
(pages ...

Key considerations in crawling
● Polite crawling
○ Do not overload websites and DNS (no DoS!)
○ Understand web serving inf...

Mirrors
● Hosts with exactly the same content
deview.kr
www.deview.kr

● Paths within hosts with the same content
www.cs.u...

Optimizing our crawling
● Efficient crawling requires duplicate handling
○ Predict whether a newly discovered link points ...

Duplicates in Dynamic Pages
● Duplicates are most common in dynamic links
http://foo.com/forum/viewtopic.php?t=3808&sid=12...

Equivalence rules and class names
● Equivalence rule for a cluster
○ Set of relevant parameters
○ Set of irrelevant parame...

Modified crawl algorithm
● Representative table
○ Equivalence class name => representative link
● Given a new link
1. Iden...

Equivalence rule generation
● Find every crawled link under a cluster
cluster = { link1 : content1, link2 : content2, ... ...

1. Insignificance analysis
● Group links by content
content1 = { link11, link21, ... }
content2 = { link21, link22, ... }
...

2. Significance analysis
● For each parameter
○ Remove the parameter from every link
■ Group content by remainder link
rem...

3. Parameter classification
● For each parameter
○ Compute content relevance (or irrelevance) value
Significance_Index
Con...

Example: P is content-irrelevant
http://foo.com/directory?P=1&Q=3
http://foo.com/directory?P=2&Q=3
Cluster

Content A

htt...

Example: Q is content-relevant
http://foo.com/directory?P=1&Q=3
http://foo.com/directory?P=2&Q=3
Cluster

Content A

http:...

Facing the Real World
● Limitations
○ Co-changing parameters
○ Noisy data
○ Parameters not used in the standard way
○ Need...

Defining duplicates
● Identical pages
● Identical visible content
● Essentially identical visible content
○ Ignore page ge...

Q&A

Thank You!

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Strata NY 2014 - Architectural cons... by hadooparchbook 9520 views
Application Architectures with Hado... by hadooparchbook 1176 views
Architectural considerations for Ha... by hadooparchbook 9943 views
Apache Spark and the Emerging Techn... by Paco Nathan 3312 views
Data Architectures for Robust Decis... by Gwen (Chen) Shapira 13278 views

View on Slideshare

1 of 25 Ad

View on Slideshare

1 of 25 Ad