Google jeff dean lessons learned while building infrastructure software at google

3,288 views

Published on

Published in: Technology
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,288
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
81
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Google jeff dean lessons learned while building infrastructure software at google

  1. 1. Lessons Learned While Building Infrastructure Software at Google Jeff Dean jeff@google.com Tuesday, September 10, 13
  2. 2. “Google” Circa 1997 (google.stanford.edu) Tuesday, September 10, 13
  3. 3. “Corkboards” (1999) Tuesday, September 10, 13
  4. 4. Google Data Center (2000) Tuesday, September 10, 13
  5. 5. Google Data Center (2000) Tuesday, September 10, 13
  6. 6. Google Data Center (2000) Tuesday, September 10, 13
  7. 7. Google (new data center 2001) Tuesday, September 10, 13
  8. 8. Google Data Center (3 days later) Tuesday, September 10, 13
  9. 9. Google’s Computational Environment Today • Many datacenters around the world Tuesday, September 10, 13
  10. 10. Google’s Computational Environment Today • Many datacenters around the world Tuesday, September 10, 13
  11. 11. Zooming In... Tuesday, September 10, 13
  12. 12. Lots of machines... Tuesday, September 10, 13
  13. 13. Cool... Tuesday, September 10, 13
  14. 14. Low-Level Systems Software Desires • If you have lots of machines, you want to: • Store data persistently – w/ high availability – high read and write bandwidth • Run large-scale computations reliably – without having to deal with machine failures • GFS, MapReduce, BigTable, Spanner, ... Tuesday, September 10, 13
  15. 15. Masters C0 C5 C1 C2 Chunkserver 1 • • • • Replicas Google File System (GFS) Design Client GFS Master Client Client C1 C5 Misc. servers GFS Master C3 Chunkserver 2 C0 … C5 C2 Chunkserver N Master manages metadata Data transfers are directly between clients/chunkservers Files broken into chunks (typically 64 MB) Chunks replicated across multiple machines (usually 3) Tuesday, September 10, 13
  16. 16. GFS Motivation and Lessons • Indexing system clearly needed a large-scale distributed file system – wanted to treat whole cluster as single file system • Developed by subset of same people working on indexing system • Identified minimal set of features needed – e.g. Not POSIX compliant – actual data was distributed, but kept metadata centralized • Colossus: Follow-on system developed many years later distributed the metadata • Lesson: Don’t solve everything all at once Tuesday, September 10, 13
  17. 17. MapReduce History • 2003: Sanjay Ghemawat and I were working on rewriting indexing system: – starts with raw page contents on disk – many phases: • (near) duplicate elimination, anchor text extraction, language identification, index shard generation, etc. – end result is data structures for index and doc serving • Each phase was hand written parallel computation: – hand parallelized – hand-written checkpointing code for fault-tolerance Tuesday, September 10, 13
  18. 18. MapReduce • A simple programming model that applies to many large-scale computing problems – allowed us to express all phases of our indexing system – since used across broad range of computer science areas, plus other scientific fields – Hadoop open-source implementation seeing significant usage • Hide messy details in MapReduce runtime library: – automatic parallelization – load balancing – network and disk transfer optimizations – handling of machine failures – robustness – improvements to core library benefit all users of library! Tuesday, September 10, 13
  19. 19. Typical problem solved by MapReduce • • • • • Read a lot of data Map: extract something you care about from each record Shuffle and Sort Reduce: aggregate, summarize, filter, or transform Write the results Outline stays the same, User writes Map and Reduce functions to fit the problem Tuesday, September 10, 13
  20. 20. MapReduce Motivation and Lessons • Developed by two people that were also doing the indexing system rewrite – squinted at various phases with an eye towards coming up with common abstraction • Initial version developed quickly – proved initial API utility with very simple implementation – rewrote much of implementation 6 months later to add lots of the performance wrinkles/tricks that appeared in original paper • Lesson: Very close ties with initial users of system make things happen faster – in this case, we were both building MapReduce and using it simultaneously Tuesday, September 10, 13
  21. 21. BigTable: Motivation • Lots of (semi-)structured data at Google – URLs: Contents, crawl metadata, links, anchors, pagerank, … – Per-user data: User preferences, recent queries, … – Geographic locations: Physical entities, roads, satellite image data, user annotations, … • Scale is large • Want to be able to grow and shrink resources devoted to system as needed Tuesday, September 10, 13
  22. 22. BigTable: Basic Data Model • Distributed multi-dimensional sparse map (row, column, timestamp) → cell contents Columns Rows • Rows are ordered lexicographically • Good match for most of our applications Tuesday, September 10, 13
  23. 23. BigTable: Basic Data Model • Distributed multi-dimensional sparse map (row, column, timestamp) → cell contents “contents:” Columns Rows “www.cnn.com” “<html>…” • Rows are ordered lexicographically • Good match for most of our applications Tuesday, September 10, 13
  24. 24. BigTable: Basic Data Model • Distributed multi-dimensional sparse map (row, column, timestamp) → cell contents “contents:” Columns Rows “www.cnn.com” “<html>…” t17 Timestamps • Rows are ordered lexicographically • Good match for most of our applications Tuesday, September 10, 13
  25. 25. BigTable: Basic Data Model • Distributed multi-dimensional sparse map (row, column, timestamp) → cell contents “contents:” Columns Rows t11 “www.cnn.com” “<html>…” t17 Timestamps • Rows are ordered lexicographically • Good match for most of our applications Tuesday, September 10, 13
  26. 26. BigTable: Basic Data Model • Distributed multi-dimensional sparse map (row, column, timestamp) → cell contents “contents:” Columns Rows t11 “www.cnn.com” “<html>…” t3 t17 Timestamps • Rows are ordered lexicographically • Good match for most of our applications Tuesday, September 10, 13
  27. 27. Tablets & Splitting “language:” “contents:” EN “<html>…” “aaa.com” “cnn.com” “cnn.com/sports.html” … “website.com” … “zuppa.com/menu.html” Tuesday, September 10, 13
  28. 28. Tablets & Splitting “language:” “contents:” EN “<html>…” “aaa.com” “cnn.com” “cnn.com/sports.html” Tablets … “website.com” … “zuppa.com/menu.html” Tuesday, September 10, 13
  29. 29. Tablets & Splitting “language:” “contents:” EN “<html>…” “aaa.com” “cnn.com” “cnn.com/sports.html” Tablets … “website.com” … “yahoo.com/kids.html” … “yahoo.com/kids.html0” … “zuppa.com/menu.html” Tuesday, September 10, 13
  30. 30. Tablets & Splitting “language:” “contents:” EN “<html>…” “aaa.com” “cnn.com” “cnn.com/sports.html” Tablets … “website.com” … “yahoo.com/kids.html” … “yahoo.com/kids.html0” … “zuppa.com/menu.html” Tuesday, September 10, 13
  31. 31. BigTable System Structure Bigtable Cell Bigtable master Bigtable tablet server Tuesday, September 10, 13 Bigtable tablet server … Bigtable tablet server
  32. 32. BigTable System Structure Bigtable Cell Bigtable master performs metadata ops + load balancing Bigtable tablet server Tuesday, September 10, 13 Bigtable tablet server … Bigtable tablet server
  33. 33. BigTable System Structure Bigtable Cell Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data serves data Tuesday, September 10, 13 … Bigtable tablet server serves data
  34. 34. BigTable System Structure Bigtable Cell Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data serves data Cluster scheduling system Tuesday, September 10, 13 Cluster file system … Bigtable tablet server serves data Lock service
  35. 35. BigTable System Structure Bigtable Cell Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data serves data Cluster scheduling system schedules tasks onto machines Tuesday, September 10, 13 Cluster file system … Bigtable tablet server serves data Lock service
  36. 36. BigTable System Structure Bigtable Cell Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data serves data … Cluster scheduling system Cluster file system schedules tasks onto machines holds tablet data, logs Tuesday, September 10, 13 Bigtable tablet server serves data Lock service
  37. 37. BigTable System Structure Bigtable Cell Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data serves data Cluster scheduling system schedules tasks onto machines Tuesday, September 10, 13 … Bigtable tablet server serves data Cluster file system Lock service holds tablet data, logs holds metadata, handles master-election
  38. 38. BigTable System Structure Bigtable client Bigtable Cell Bigtable client library Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data serves data Cluster scheduling system schedules tasks onto machines Tuesday, September 10, 13 … Bigtable tablet server serves data Cluster file system Lock service holds tablet data, logs holds metadata, handles master-election
  39. 39. BigTable System Structure Bigtable client Bigtable Cell Bigtable client library Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data Open() serves data Cluster scheduling system schedules tasks onto machines Tuesday, September 10, 13 … Bigtable tablet server serves data Cluster file system Lock service holds tablet data, logs holds metadata, handles master-election
  40. 40. BigTable System Structure Bigtable client Bigtable Cell Bigtable client library Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data read/write serves data Cluster scheduling system schedules tasks onto machines Tuesday, September 10, 13 … Open() Bigtable tablet server serves data Cluster file system Lock service holds tablet data, logs holds metadata, handles master-election
  41. 41. BigTable System Structure Bigtable client Bigtable Cell metadata ops Bigtable client library Bigtable master performs metadata ops + load balancing Bigtable tablet server Bigtable tablet server serves data read/write serves data Cluster scheduling system schedules tasks onto machines Tuesday, September 10, 13 … Open() Bigtable tablet server serves data Cluster file system Lock service holds tablet data, logs holds metadata, handles master-election
  42. 42. BigTable Status • Production use for 100s of projects: – Crawling/indexing pipeline, Google Maps/Google Earth/Streetview, Search History, Google Print, Google+, Blogger, ... • Currently 500+ BigTable clusters • Largest cluster: – 100s PB data; sustained: 30M ops/sec; 100+ GB/s I/O • Many asynchronous processes updating different pieces of information –no distributed transactions, no cross-row joins –initial design was just in a single cluster –follow-on work added eventual consistency across many geographically distributed BigTable instances Tuesday, September 10, 13
  43. 43. Spanner • Storage & computation system that runs across many datacenters – single global namespace • names are independent of location(s) of data • fine-grained replication configurations – support mix of strong and weak consistency across datacenters • Strong consistency implemented with Paxos across tablet replicas • Full support for distributed transactions across directories/machines – much more automated operation • automatically changes replication based on constraints and usage patterns • automated allocation of resources across entire fleet of machines Tuesday, September 10, 13
  44. 44. Design Goals for Spanner • Future scale: ~105 to 107 machines, ~1013 directories, ~1018 bytes of storage, spread at 100s to 1000s of locations around the world – zones of semi-autonomous control – consistency after disconnected operation – users specify high-level desires: “99%ile latency for accessing this data should be <50ms” “Store this data on at least 2 disks in EU, 2 in U.S. & 1 in Asia” Tuesday, September 10, 13
  45. 45. Spanner Lessons • Several variations of eventual client API • Started to develop with many possible customers in mind, but no particular customer we were working closely with • Eventually we worked closely with Google ads system as initial customer – first real customer was very demanding (real $$): good and bad • Different API than BigTable – Harder to move users with existing heavy BigTable usage Tuesday, September 10, 13
  46. 46. Designing & Building Infrastructure Identify common problems, and build software systems to address them in a general way • Important to not try to be all things to all people – Clients might be demanding 8 different things – Doing 6 of them is easy – …handling 7 of them requires real thought – …dealing with all 8 usually results in a worse system • more complex, compromises other clients in trying to satisfy everyone Tuesday, September 10, 13
  47. 47. Designing & Building Infrastructure (cont) Don't build infrastructure just for its own sake: • Identify common needs and address them • Don't imagine unlikely potential needs that aren't really there Best approach: use your own infrastructure (especially at first!) • (much more rapid feedback about what works, what doesn't) If not possible, at least work very closely with initial client team • ideally sit within 50 feet of each other • keep other potential clients needs in mind, but get system working via close collaboration with first client first Tuesday, September 10, 13
  48. 48. Thanks! Further reading: • Ghemawat, Gobioff, & Leung. Google File System, SOSP 2003. • Barroso, Dean, & Hölzle. Web Search for a Planet: The Google Cluster Architecture, IEEE Micro, 2003. • Dean & Ghemawat. MapReduce: Simplified Data Processing on Large Clusters, OSDI 2004. • Chang, Dean, Ghemawat, Hsieh, Wallach, Burrows, Chandra, Fikes, & Gruber. Bigtable: A Distributed Storage System for Structured Data, OSDI 2006. • Corbett et al. • Burrows. Spanner: Google’s Globally Distributed Database, OSDI 2012. The Chubby Lock Service for Loosely-Coupled Distributed Systems. OSDI 2006. • Pinheiro, Weber, & Barroso. Failure Trends in a Large Disk Drive Population. FAST 2007. • Barroso & Hölzle. The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, Morgan & Claypool Synthesis Series on Computer Architecture, 2009. • Malewicz et al. Pregel: A System for Large-Scale Graph Processing. PODC, 2009. • Schroeder, Pinheiro, & Weber. • Protocol Buffers. DRAM Errors in the Wild: A Large-Scale Field Study. SEGMETRICS’09. http://code.google.com/p/protobuf/ See: http://research.google.com/papers.html http://research.google.com/people/jeff Tuesday, September 10, 13

×