Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

In-Memory Database System Built for Speed and Scale


Published on

- MemSQL workloads
- HTAP - Hybrid Transaction/Analytical Processing MemSQL key innovations

Published in: Data & Analytics
  • Be the first to comment

In-Memory Database System Built for Speed and Scale

  2. 2. Nikita’s Background PhD in CS from St. Petersburg ITMO Microsoft SQL Server Kernel Engineer Facebook Performance and Site Cost Since 2011 MemSQL CTO and co-founder
  3. 3. What will this talk cover MemSQL workloads HTAP - Hybrid Transaction/Analytical Processing MemSQL key innovations Live demo of MemSQL and Q/A
  4. 4. Why in-memory? OLTP Improve transactional throughput Ingest more per second OLAP Load data faster Generate reports faster
  5. 5. Modern Workloads Ingest and serve data in real-time Generate reports over changing datasets Anomaly detection as events occur Sub-second response for hundreds of users Integrating with the eco system (Connectivity, Spark, Tableau, Hadoop, Splunk, etc)
  6. 6. Gartner Introduces HTAP Hybrid Transactional/Analytical Processing Analytics over concurrently changing data Predictable performance and scalability ETL is irrelevant
  7. 7. In-memory Delivers HTAP Transactions Higher throughput, lower latencies Predictable SLAs Analytics Can access data as it is written (no ETL) Velocity -> Volume Columnar format can run on flash
  8. 8. Ankur’s Background CMU SCS Undergrad Internships at Microsoft and PDL MemSQL Employee #5 Director of Engineering
  9. 9. MemSQL Overview In-memory row store Distributed Online operations Fast ANSI SQL Agg 1 Agg 2 Leaf 1 Leaf 2 Leaf 3 Leaf 4
  10. 10. Key Innovations Lock-Free Skip Lists Code Generation Durability and Replication Clustering Distributed Query Execution
  11. 11. What is a Skip List Invented in 1990 by William Pugh Expected O(log(n)) lookup, insert, delete Much simpler than a balanced tree
  12. 12. Skip List Indexes Memory optimized Simple Lock free Fast Flexible Multiversion Concurrency Control
  13. 13. Common Concerns Memory overhead CPU cache efficiency Scan performance Reverse iteration struct Table_Row { int col_a; char* col_b; … Tower* idx_1_ptrs; Tower* idx_2_ptrs; };
  14. 14. Code Generation Inline scans Expression execution Need a powerful plan cache OLTP vs. data exploration
  15. 15. MemSQL Code Generation Versioning and type-system Templates .h file per table .cpp file per plan Precompiled headers
  16. 16. Expression Snippet memsql>  select  concat("foo",  "bar");   +-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐+   |  concat("foo",  "bar")  |   +-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐+   |  foobar                              |   +-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐+   1  row  in  set  (0.81  sec)   memsql>  select  concat("foo",  "bar");   +-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐+   |  concat("foo",  "bar")  |   +-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐+   |  foobar                              |   +-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐+   1  row  in  set  (0.00  sec) bool  overflow  =  false;   VarCharTemp  result1("abc",  3,  threadId);   VarCharTemp  result2("def",  3,  threadId);   opt<TemporaryImmutableString>  result3;   op_Concat(result3,  result1,  result2,  overflow,  threadId);
  17. 17. Durability Transaction log Periodic snapshots Replay to recover state Sequential I/O (reads and writes)
  18. 18. Replication Built on top of durability Stream snapshot and log files Slave == continuous recovery Very simple and robust
  19. 19. Clustering Two-tier architecture Scalable on both sides Intra- and inter-datacenter HA Stays out of the way
  20. 20. Failover Leaf 1 Leaf 2 Leaf 4Leaf 3 Agg 1 Agg 2 Leaf 1 Leaf 2 Leaf 4>_< Agg 1 Agg 2
  21. 21. Cluster Replication Leaf 1 Leaf 2 Agg Leaf 1 Leaf 2 Agg Primary Secondary
  22. 22. Query Execution Shard keys SQL between nodes ToSQL() Reference tables Foreign shard key joins
  23. 23. Example Query Agg 1 Agg 2 select&count(1)&from&orders;& leaf1>&using&memsql_demo_0& select&count(1)&from&orders;& & leaf2>&using&memsql_demo_1& select&count(1)&from&orders;& & leaf3>&using&memsql_demo_2& select&count(1)&from&orders;& ...& Leaf 1 Leaf 2 Leaf 3 Leaf 4
  24. 24. New QE/QO Projects SQL-based reshuffling ARRANGE Remote tables Optimizer SQL-SQL tree transforms Cost-based distributed optimizer ARRANGE          t.a,  t.b,  SUM(t.price)     FROM            t     GROUP  BY            t.a,  t.b     SHARDED  BY            t.a,  t.b     AS  t_reshuffled
  25. 25. Conclusions In-memory unlocks new workloads Old problems with new solutions Call for Data Structures/Algorithms Storage systems
  26. 26. DEMO AND Q/A