View stunning SlideShares in full-screen with the new iOS app!Introducing SlideShare for AndroidExplore all your favorite topics in the SlideShare appGet the SlideShare app to Save for Later — even offline
View stunning SlideShares in full-screen with the new Android app!View stunning SlideShares in full-screen with the new iOS app!
Some other Voices It’s stupidity. It’s worse than stupidity: it’s a marketing hype campaign. Somebody is saying this is inevitable — and whenever you hear somebody saying that, it’s very likely to be a set of businesses campaigning to make it true. Richard Stallman , quoted in The Guardian, September 29, 2008 The interesting thing about Cloud Computing is that we’ve redefined Cloud Computing to include everything that we already do. . . . I don’t understand what we would do differently in the light of Cloud Computing other than change the wording of some of our ads. Larry Ellison , quoted in the Wall Street Journal, September 26, 2008
What you want to do with 1000 pcs, or even 100,000 pcs?
Cloud is coming… Google alone has 450,000 systems running across 20 datacenters , and Microsoft's Windows Live team is doubling the number of servers it uses every 14 months, which is faster than Moore's Law “ Data Center is a Computer” Parallelism everywhere Massive Scalable Reliable Resource Management Data Management Programming Model & Tools
NERSC User George Smoot wins 2006 Nobel Prize in Physics Smoot and Mather 1992 COBE Experiment showed anisotropy of CMB Cosmic Microwave Background Radiation (CMB): an image of the universe at 400,000 years
Use web crawler to gather 151M HTML pages weekly 11 times
Generated 1.2 TB log information
Analyze page statistics and change frequencies
“ Moreover, we experienced a catastrophic disk failure during the third crawl, causing us to lose a quarter of the logs of that crawl.”
Fetterly, Manasse, Najork, Wiener (Microsoft, HP), “A Large-Scale Study of the Evolution of Web Pages,” Software-Practice & Experience, 2004
G A T G C TT A C T A T G C GGG CCCC C GG T C T AA T G C TT A C T A T G C G C TT A C T A T G C GGG CCCC TT AA T G C TT A C T A T G C GGG CCCC TT T AA T G C TT A C T A T G C AA T G C TT A G C T A T G C GGG C AA T G C TT A C T A T G C GGG CCCC TT AA T G C TT A C T A T G C GGG CCCC TT C GG T C T A G A T G C TT A C T A T G C AA T G C TT A C T A T G C GGG CCCC TT C GG T C T AA T G C TT A G C T A T G C A T G C TT A C T A T G C GGG CCCC TT Subject genome Sequencer Reads ?
C GG T C T A G A T G C TT A G C T A T G C GGG CCCC TT Reference sequence Alignment G C TT A T C T A T TT A T C T A T G C A T C T A T G C GG A T C T A T G C GG G C TT A T C T A T T C T A G A T G C T C T A T G C GGG C C T A G A T G C TT A T C T A T G C GG C T A T G C GGG C A T C T A T G C GG Subject reads
C GG T C T A G A T G C TT A T C T A T G C GGG CCCC TT G C TT A T C T A T TT A T C T A T G C A T C T A T G C GG A T C T A T G C GG G C TT A T C T A T GG CCCC TT G CCCC TT CC TT C GG C GG T C C GG T C T C GG T C T A G T C T A G A T G C T C T A T G C GGG C C T A G A T G C TT C TT A T G C GGG CCC Reference sequence Subject reads
2004 - Initial versions of what is now Hadoop Distributed File System and Map-Reduce implemented by Doug Cutting & Mike Cafarella
December 2005 - Nutch ported to the new framework. Hadoop runs reliably on 20 nodes .
January 2006 - Doug Cutting joins Yahoo!
February 2006 - Apache Hadoop project official started to support the standalone development of Map-Reduce and HDFS.
March 2006 - Formation of the Yahoo! Hadoop team
May 2006 - Yahoo sets up a Hadoop research cluster - 300 nodes
April 2006 - Sort benchmark run on 188 nodes in 47.9 hours
May 2006 - Sort benchmark run on 500 nodes in 42 hours (better hardware than April benchmark)
October 2006 - Research cluster reaches 600 Nodes
December 2006 - Sort times 20 nodes in 1.8 hrs, 100 nodes in 3.3 hrs, 500 nodes in 5.2 hrs, 900 nodes in 7.8
January 2006 - Research cluster reaches 900 node
April 2007 - Research clusters - 2 clusters of 1000 nodes
Sep 2008 - Scaling Hadoop to 4000 nodes at Yahoo!
From Theory to Practice You 1. Scp data to cluster 2. Move data into HDFS 3. Develop code locally 4. Submit MapReduce job 4a. Go back to Step 3 5. Move data out of HDFS 6. Scp data from cluster Hadoop Cluster
 J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," in Osdi , 2004, pp. 137-150.
 G. Sanjay, G. Howard, and L. Shun-Tak, "The Google file system," in Proceedings of the nineteenth ACM symposium on Operating systems principles . Bolton Landing, NY, USA: ACM Press, 2003.
 O. Christopher, R. Benjamin, S. Utkarsh, K. Ravi, and T. Andrew, "Pig latin: a not-so-foreign language for data processing," in Proceedings of the 2008 ACM SIGMOD international conference on Management of data . Vancouver, Canada: ACM, 2008.