Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hypertable Berlin Buzzwords


Published on

This presentation was given by Doug Judd at BerlinBuzzwords 2010.

Published in: Technology
  • awesome
    Are you sure you want to  Yes  No
    Your message goes here

Hypertable Berlin Buzzwords

  1. 1. Hypertable Doug Judd CEO, Hypertable, Inc.
  2. 2. High Performance, Open Source Scalable Database <ul><li>Modeled after Bigtable </li></ul><ul><li>High Performance Implementation (C++) </li></ul><ul><li>Project Started in March 2007 </li></ul><ul><li>Thrift Interface for all popular languages </li></ul><ul><ul><li>Java </li></ul></ul><ul><ul><li>PHP </li></ul></ul><ul><ul><li>Ruby </li></ul></ul><ul><ul><li>Python </li></ul></ul><ul><ul><li>Perl, etc. </li></ul></ul>
  3. 3. Bigtable: the infrastructure that Google is built on <ul><li>YouTube </li></ul><ul><li>Blogger </li></ul><ul><li>Google Earth </li></ul><ul><li>Google Maps </li></ul><ul><li>Orkut (social network) </li></ul><ul><li>Gmail </li></ul><ul><li>Google Analytics </li></ul><ul><li>Google Book Search </li></ul><ul><li>Google Code </li></ul><ul><li>Crawl Database </li></ul><ul><li>… plus 90 other Google services … </li></ul>
  4. 4. Functionality <ul><li>Massive sparse tables of information </li></ul><ul><li>Single primary key index </li></ul><ul><li>Cells can have mulitple timestamped versions </li></ul><ul><li>Not Relational </li></ul><ul><ul><li>No joins (not yet) </li></ul></ul><ul><ul><li>No secondary indexes (not yet) </li></ul></ul><ul><ul><li>Not a transaction system (not yet) </li></ul></ul>
  5. 5. Hypertable Deployments
  6. 6. Other Architectures
  7. 7. Auto-Sharding MongoDB AsterData Greenplum
  8. 8. MongoDB
  9. 9. Dynamo-based Hash Table Architectures Cassandra Project Voldemort Riak
  10. 10. Eventual Consistency
  11. 11. Consistent Hashing
  12. 12. Order Preserving Partitioner (Cassandra) 1091721999…629750272 + 1091721999…965293103 / 2 = www.rgb????i?pQdp ?.??? 1091721999…297521687
  13. 13. Order Preserving Partitioner Balance Problem
  14. 14. Hypertable Architecture
  15. 15. Conceptual Table Layout
  16. 16. Table: Actual Representation
  17. 17. Range Distribution
  18. 18. Google Stack
  19. 19. Google File System
  20. 20. Google File System
  21. 21. System Overview
  22. 22. Log Structured Merge (LSM) Tree <ul><li>Eliminates random I/O on writes </li></ul><ul><li>Converts random I/O to sequential I/O </li></ul><ul><li>Write path </li></ul><ul><ul><li>Commit log on disk (DFS) </li></ul></ul><ul><ul><li>In-memory map </li></ul></ul><ul><li>In-memory map gets “compacted” to disk </li></ul><ul><li>Disk files periodically get merged </li></ul>
  23. 23. Range Server <ul><li>Manages ranges of table data </li></ul><ul><li>CellCache: In-memory map containing recent updates </li></ul><ul><li>CellStore: On-disk (DFS) file containing “compacted” cell cache </li></ul>
  24. 24. Range Server: CellStore <ul><li>Sequence of 65K blocks of compressed key/value pairs </li></ul>
  25. 25. Compression <ul><li>Cell Store blocks are compressed </li></ul><ul><li>Commit Log updates are compressed </li></ul><ul><li>Supported Compression Schemes </li></ul><ul><ul><li>zlib (--best and --fast) </li></ul></ul><ul><ul><li>lzo </li></ul></ul><ul><ul><li>quicklz </li></ul></ul><ul><ul><li>bmz </li></ul></ul><ul><ul><li>none </li></ul></ul>
  26. 26. Bloom Filter <ul><li>Probabilistic data structure associated with every CellStore </li></ul><ul><li>Indicates if key is not present </li></ul>
  27. 27. Caching <ul><li>Block Cache </li></ul><ul><ul><li>Caches CellStore blocks </li></ul></ul><ul><ul><li>Blocks are cached uncompressed </li></ul></ul><ul><ul><li>Dynamically adjusted size based on workload </li></ul></ul><ul><li>Query Cache </li></ul><ul><ul><li>Caches query results </li></ul></ul>
  28. 28. Dynamic Memory Adjustment
  29. 29. Performance Evaluation Hypertable vs. HBase
  30. 30. Test Setup <ul><li>Hypertable v0.9.3.2 (not yet released) </li></ul><ul><li>HBase 0.20.3 </li></ul><ul><li>HDFS 0.20.2 </li></ul><ul><li>10 machines </li></ul><ul><ul><li>3 Hyperspace / Zookeeper replicas </li></ul></ul><ul><ul><li>1 Master / 4 Tablet Servers (5GB RAM) </li></ul></ul><ul><ul><li>1 Test Dispatcher / 4 Test Clients </li></ul></ul><ul><li>Machine profile </li></ul><ul><ul><li>1 X 1.8 GHz Dual-core Opteron </li></ul></ul><ul><ul><li>10 GB RAM </li></ul></ul><ul><ul><li>3 X 250 GB SATA drives </li></ul></ul>
  31. 31. Random Write / Sequential Read
  32. 32. Random Read
  33. 33. Project Resources <ul><li>Twitter: hypertable </li></ul><ul><li> </li></ul>
  34. 34. Professional Support