Google Cluster Innards

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Google Cluster Innards Martin Dvorak [email_address] http://www.e-mental.com/dvorka

Agenda <ul><li>Inventing Google The Anatomy of a Large-Scale Hypertextual Web Search Engine Sergey Brin and Lawrence Page...

Inventing Google

Inventing Google <ul><li>Sergey & Larry - Ph.D. students at Stanford University </li></ul><ul><li>Prototype (1998) </li>...

Inventing Google: Foundation <ul><li>PageRank*: </li></ul><ul><ul><li>We assume page A has pages T1...Tn which point ...

Inventing Google: Foundation <ul><li>Page Rank formula informally </li></ul><ul><ul><li>PR(A) = (1-d) + d (PR(T1)/C(T1) + ...

Inventing Google: Foundation <ul><li>PageRank relevancy tuning </li></ul><ul><ul><li>Page title </li></ul></ul><ul><ul><li...

Inventing Google: Anatomy

Inventing Google: Anatomy <ul><li>URL Server </li></ul><ul><ul><li>Providers list of URLs to be fetched to crawlers </li>...

Inventing Google: Anatomy <ul><li>Indexer </li></ul><ul><ul><li>Reads pages from Repository (uncompress) </li></ul></ul>...

Inventing Google: Anatomy <ul><li>Searcher uses… </li></ul><ul><ul><li>Lexicon </li></ul></ul><ul><ul><ul><li>Keeps map s...

Cluster Innards

Cluster Innards: Global Google <ul><li>Over 30 Google clusters around the world. </li></ul><ul><ul><li>DNS based & geo ...

Cluster Innards: HW <ul><li>Basics cluster design insights </li></ul><ul><ul><li>Reliability in SW rather then server-clas...

Cluster Innards: HW <ul><li>Optimistically, a consumer PC might crash once in three years from a software glitch or hard...

Cluster Innards: SW <ul><li>Stripped-down version of Linux , which is based on the Red Hat distribution but is really ...

Programming for Cluster

Programming For Cluster <ul><li>Google's MapReduce is a programming model and an associated implementation for proc...

Programming For Cluster <ul><li>Map Reduce runtime… </li></ul><ul><ul><li>takes care of the details of partitioning the ...

Programming For Cluster <ul><li>Special purpose computations examples: </li></ul><ul><ul><li>Inputs: </li></ul></ul><ul><...

Programming For Cluster <ul><li>LISP roots </li></ul><ul><ul><li>Remind map and reduce primitives: </li></ul></ul><ul>...

Programming For Cluster <ul><li>Example (pseudocode): </li></ul><ul><ul><li>map(String key, String value): // key: docum...

Programming For Cluster <ul><li>Example solves the problem of counting the number of occurrences of each word in a large...

Programming For Cluster

Programming For Cluster <ul><li>Also… </li></ul><ul><ul><li>Multiple tasks performed by single worker ( load-balancing ) <...

Putting Things Together

I’m Feeling Lucky <ul><li>Pre-phase: </li></ul><ul><ul><li>Browser requests e.g. http://www.google.com/search?q=edu+sessi...

Bonus

Stanford lab (around 1996)

The Original Google Storage: 10x4GB (1996)

Google San Francisco (2004)

A cluster of coolness Google History

Google Results Page Per Day

References <ul><li>Sergey Brin, Lawrence Page; The Anatomy of a Large-Scale Hypertextual Web Search Engine; 1998 </li></...

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

Share
Email

Cloud Computing Clusters for Dummies by Liberteks 1020 views
Grid by Fajar Zain 289 views
Clusters (Distributed computing) by Sri Prasanna 1519 views
Evaluation of Virtual Clusters Perf... by EuroCloud 1554 views
Parallel and Distributed Computing ... by Vittorio Giovara 1307 views
Chapter16 new by vmummaneni 2335 views

Top clipped slide

1 of 34 Ad

1 of 34 Ad