Nosql series-part-3-hypertable


Published on

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • -Why use the HT Hive extension? -select from and insert to Hypertable from Hive -currently HT doesn’t support types natively everything is a string, whereas Hive has an extensive type system -Hive allows you to do joins across multiple tables, some in HT, some in Hive or other sources -Keep fast changing data in HT and combine with other less frequently updated data sources -This is an initial working implementation , we have a fairly detailed roadmap of features we want to add going forward.
  • Nosql series-part-3-hypertable

    1. 1. Hypertable Doug Judd CEO, Hypertable, Inc.
    2. 2. High Performance, Open Source Scalable Database <ul><li>Modeled after Bigtable </li></ul><ul><li>High Performance Implementation (C++) </li></ul><ul><li>Project Started in March 2007 </li></ul><ul><li>Runs on top of HDFS </li></ul><ul><li>Thrift Interface for all popular languages </li></ul><ul><ul><li>Java </li></ul></ul><ul><ul><li>PHP </li></ul></ul><ul><ul><li>Ruby </li></ul></ul><ul><ul><li>Python </li></ul></ul><ul><ul><li>Perl, etc. </li></ul></ul>
    3. 3. Hypertable Deployments
    4. 4. Architecture
    5. 5. Underlying Data Representation
    6. 6. Scaling (part I)
    7. 7. Scaling (part II)
    8. 8. Scaling (part III)
    9. 9. Request Routing
    10. 10. Query Handling
    11. 11. Features
    12. 12. <ul><li>Load data from HT to Hive and vice-versa </li></ul><ul><li>Use Hive types </li></ul><ul><li>Use Hive QL (joins, aggregations) </li></ul><ul><li>Low latency data warehousing </li></ul><ul><li>Uses Hypertable’s native MapReduce Input/Output format </li></ul>
    13. 13. Namespaces /development user tweet /testing user tweet /production /v1 user tweet /v2 user tweet
    14. 14. Column Family Options <ul><li>TTL=<t> </li></ul><ul><ul><li>“ time to live” </li></ul></ul><ul><ul><li>Remove cells that are older than <t> </li></ul></ul><ul><li>MAX_VERSIONS=<n> </li></ul><ul><ul><li>Keep only most recent <n> cell versions </li></ul></ul>
    15. 15. Access Groups <ul><li>Provides control over physical layout </li></ul><ul><ul><li>Row oriented </li></ul></ul><ul><ul><li>Column oriented </li></ul></ul><ul><ul><li>Hybrid </li></ul></ul><ul><li>Reduces I/O </li></ul>CREATE TABLE MyTable ( a, b, c, d, ACCESS GROUP first(a), ACCESS GROUP second (b, c, d) );
    16. 16. Regular Expression Filtering <ul><li>Google’s RE2 regular expression engine </li></ul><ul><ul><li>Extremely fast (up to 50X Java regex) </li></ul></ul><ul><ul><li>Searches run in time linear in the size of the input </li></ul></ul><ul><ul><li>Searches constrained to a fixed amount of memory </li></ul></ul><ul><li>Supported Searches: </li></ul><ul><ul><li>Row key </li></ul></ul><ul><ul><li>Column qualifier </li></ul></ul><ul><ul><li>Value </li></ul></ul>SELECT CELLS tag:/(?i)(nosql|bigtable)/ FROM MyTable WHERE ROW REGEXP &quot;^D+&quot; AND VALUE REGEXP ”(?i)hypertable&quot;;
    17. 17. Atomic Counters <ul><li>New column option: </li></ul><ul><li>Modified via existing API using specially formatted values: </li></ul>create table counts ( url COUNTER, ); Reset counter to n =n Decrement counter by n -n Increment counter by n [+]n Description Value Format
    18. 18. Group Commit <ul><li>Supports highly concurrent updates </li></ul><ul><li>Trades minimum latency for better throughput </li></ul><ul><li>Configurable commit interval per-table: </li></ul>CREATE TABLE counts ( url, domain ) GROUP_COMMIT_INTERVAL=100;
    19. 19. Compression <ul><li>Block compression </li></ul><ul><ul><li>Cell Store (SSTable) blocks </li></ul></ul><ul><ul><li>Commit Log blocks </li></ul></ul><ul><li>Supported Compression Schemes: </li></ul><ul><ul><li>zlib </li></ul></ul><ul><ul><li>lzo </li></ul></ul><ul><ul><li>quicklz </li></ul></ul><ul><ul><li>bmz </li></ul></ul><ul><ul><li>none </li></ul></ul>
    20. 20. Bloom Filter <ul><li>Dramatically reduces disk access </li></ul><ul><li>Associated with each Cell Store </li></ul><ul><li>Tells you if key is definitively not present </li></ul>
    21. 21. Performance Evaluation
    22. 22. Setup <ul><li>Modeled after Test described in Bigtable paper </li></ul><ul><li>1 Test Dispatcher, 4 Test Clients, 4 Tablet Servers </li></ul><ul><li>Test was written entirely in Java </li></ul><ul><li>Hardware </li></ul><ul><ul><li>1 X 1.8 GHz Dual-core Opteron </li></ul></ul><ul><ul><li>10 GB RAM </li></ul></ul><ul><ul><li>3X 250GB SATA drives </li></ul></ul><ul><li>Software </li></ul><ul><ul><li>HDFS 0.20.2 running on all 10 nodes, 3X replication </li></ul></ul><ul><ul><li>HBase 0.20.4 </li></ul></ul><ul><ul><li>Hypertable </li></ul></ul>
    23. 23. Latency
    24. 24. Throughput 220 Scan 10 byte values 75 Scan 100 byte values 58 Scan 1KB values 2 Scan 10KB values 129 Sequential Read 100 byte values 68 Sequential Read 1KB values 1060 Sequential Read 10KB values 931 Random Write 10 byte values 427 Random Write 100 byte values 102 Random Write 1KB values 51 Random Write 10KB values 100 Random Read Zipfian 2.5 GB 777 Random Read Zipfian 20 GB 925 Random Read Zipfian 80 GB Hypertable Advantage Relative to HBase (%) Test
    25. 25. Why does Performance Matter? $$$
    26. 26. Upcoming Release (0.9.5) <ul><li>Last “alpha” release </li></ul><ul><li>Release Date: February 15th 2011 </li></ul><ul><li>Features </li></ul><ul><ul><li>Automatic range balancing </li></ul></ul><ul><ul><li>Asynchronous API </li></ul></ul><ul><ul><li>Improved Monitoring System </li></ul></ul>
    27. 27. Resources blog Blog: Project Site: hypertable Twitter:
    28. 28. Professional Support
    29. 29. Q&A