SlideShare a Scribd company logo
1 of 32
Download to read offline
Faster Column-Oriented Indexes

                        Daniel Lemire

  http://www.professeurs.uqam.ca/pages/lemire.daniel.htm
            blog: http://www.daniel-lemire.com/

Joint work with Owen Kaser (UNB) and Kamel Aouiche (post-doc).




                    February 10, 2010




                   Daniel Lemire   Faster Column-Oriented Indexes
Some trends in business intelligence (BI)




                                Low-latency BI, Complex Event
                                Processing [Hyde, 2010]
                                Commotization, open source software:
                                Pentaho, LucidDB
                                (http://www.luciddb.org/)
                                Column-oriented databases ←
source: gooddata.com




                       Daniel Lemire   Faster Column-Oriented Indexes
Row Stores




    name, date, age, sex, salary


    name, date, age, sex, salary


    name, date, age, sex, salary                   Dominant paradigm
    name, date, age, sex, salary
                                                   Transactional: Quick append and delete

    name, date, age, sex, salary




                                   Daniel Lemire      Faster Column-Oriented Indexes
Column Stores




                                             Goes back to StatCan in the
                                             seventies [Turner et al., 1979]
                                             Made fashionable again in Data
name   date   age   sex   salary
                                             Warehousing by
                                             Stonebraker [Stonebraker et al., 2005]
                                             New: Oracle Exadata hybrid columnar
                                             compression




                             Daniel Lemire      Faster Column-Oriented Indexes
Vectorization




                                       Modern superscalar CPUs support
const i n t N = 2048;                  vectorization (SSE)
i n t a [N] , b [N ] ;                 This code is four times faster with
i n t i =0;                            -ftree-vectorize (GNU GCC)
f o r ( ; i <N ; i ++)                 Need long streams, same data type, and
    a [ i ] += b [ i ] ;               no branching.
                                       Columns are good candidates!




                       Daniel Lemire      Faster Column-Oriented Indexes
Main column-oriented indexes




     (1) Bitmap indexes [O’Neil, 1989]
     (2) Projection indexes [O’Neil and Quass, 1997]
     Both are compressible.




                       Daniel Lemire   Faster Column-Oriented Indexes
Bitmap indexes




  SELECT * FROM
  T WHERE x=a                            Vectors of booleans
  AND y=b;
  Above, compute
   {r | r is the row id of a row where x = a} ∩
   {r | r is the row id of a row where y = b}




                         Daniel Lemire    Faster Column-Oriented Indexes
Other applications of the bitmaps/bitsets




      The Java language has had a bitmap class since the
      beginning: java.util.BitSet. (Sun’s implementation is based
      on 64-bit words.)
      Search engines use bitmaps to filter queries, e.g. Apache
      Lucene: org.apache.lucene.util.OpenBitSet.java.




                        Daniel Lemire   Faster Column-Oriented Indexes
Bitmaps and fast AND/OR operations


     Computing the union of two sets of integers between 1 and 64
     (eg row ids, trivial table). . .
     E.g., {1, 5, 8} ∪ {1, 3, 5}?
     Can be done in one operation by a CPU:
     BitwiseOR( 10001001, 10101000)
     Extend to sets from 1..N using N/64 operations.
     To compute [a0 , . . . , aN−1 ] ∨ [b0 , b1 , . . . , bN−1 ] :
     a0 , . . . , a63 BitwiseOR b0 , . . . , b63 ;
     a64 , . . . , a127 BitwiseOR b64 , . . . , b127 ;
     a128 , . . . , a192 BitwiseOR b128 , . . . , b192 ;
     ...
     It is a form of vectorization.


                            Daniel Lemire   Faster Column-Oriented Indexes
What are bitmap indexes for?




     Myth: bitmap indexes are for low cardinality columns (e.g.,
     SEX).
              the Bitmap index is the conclusive choice for data
              warehouse design for columns with high or low
              cardinality [Zaker et al., 2008].




                       Daniel Lemire   Faster Column-Oriented Indexes
Projection indexes




                 name


                        date
        city


                                                  Write out the (normalized)
                                                  column values sequentially.
                                                  It is a projection of the table
                                                  on a single column.
               name
                                                  Best for low selectivity queries
                                                  on few columns:
                               date               SELECT sum(number*price)
         city                                     FROM T;.




                                  Daniel Lemire     Faster Column-Oriented Indexes
How to compress column indexes?




     Must handle long streams of identical values efficiently ⇒
     Run-length encoding? (RLE)
     Bitmap: a run of 0s, a run of 1s, a run of 0s, a run of 1s, . . .
     So just encode the run lengths, e.g.,
     0001111100010111 →
      3, 5, 3, 1,1,3
     It is a bit more complicated (more another day)




                        Daniel Lemire   Faster Column-Oriented Indexes
What about other compression types?




     With RLE, we can often process the data in compressed form
     Hence, with RLE, compression saves both storage and
     CPU cycles!!!!
     Not always true with other techniques such as Huffman,
     LZ77, Arithmetic Coding, . . .




                      Daniel Lemire   Faster Column-Oriented Indexes
How do we improve performance?




     Smaller indexes are faster.
     In data warehousing: data is often updated in batches.
     So spend time at construction time optimizing the index.




                        Daniel Lemire   Faster Column-Oriented Indexes
Modelling the size of an index




      Any formal result?
      Tricky: There are many variations on RLE.
      Use: number of runs of identical value in a column
      AAABBBCCAA has 4 runs




                       Daniel Lemire   Faster Column-Oriented Indexes
Improving compression by reordering the rows



      RLE is order-sensitive:
      they compress sorted tables better;
      But finding the best row ordering is
      NP-hard [Lemire et al., 2010].
      Actually an instance of the Traveling Salesman Problem
      (TSP)
      So we use heuristics:
          lexicographically
          Gray codes
          Hilbert, . . .




                         Daniel Lemire   Faster Column-Oriented Indexes
How many ways to sort? (1)




      Lexicographic row sorting is                        a     a
          fast, even for very large                       a     b
          tables.                                         a     c
          easy: sort is a Unix staple.
                                                          b     a
      Substantial index-size reductions                   b     b
      (often 2.5 times, benefits grow                      b     c
      with table size)




                       Daniel Lemire   Faster Column-Oriented Indexes
How many ways to sort? (2)




      Gray Codes are list of tuples                       a     a
      with successive (Hamming)                           a     b
      distance of 1 [Knuth, 2005].                        a     c
                                                          b     c
      Reflected Gray Code order is
                                                          b     b
          sometimes slightly better
          than lexicographical. . .                       b     a




                       Daniel Lemire   Faster Column-Oriented Indexes
How many ways to sort? (3)




                                                         a     a
      Reflected Gray Code order is not                    a     b
      the only Gray code.                                a     c
                                                         b     c
      Knuth also presents Modular
                                                         b     a
      Gray-code.
                                                         b     b




                      Daniel Lemire   Faster Column-Oriented Indexes
How many ways to sort? (4)




                                       Hilbert Index
                                       [Hamilton and Rau-Chaplin, 2007].
                                       Also a Gray code
                                       (conditionnally)
                                       Gives very bad results for
                                       column-oriented indexes.




                  Daniel Lemire   Faster Column-Oriented Indexes
Recursive orders




      Lexicographical, reflected Gray code and modular Gray
      code belong to a larger class: recursive orders.
      They sort on the first column, then the second and so on.
      Not all Gray codes are recursive orders: Hilbert is not.




                         Daniel Lemire   Faster Column-Oriented Indexes
Best column order?


  Column order is important for recursive orders.
  We almost have this result [Lemire and Kaser, 2009]:
      any recursive order
      order the columns by increasing cardinality (small to
      LARGE)

  Proposition

  The expected number of runs is minimized (among all possible
  column orders).




                        Daniel Lemire   Faster Column-Oriented Indexes
How do you know when the lexicographical order is good
enough?




     Even though row reordering is NP-hard, we find it hard to
     improve over recursive orders.
     Sometimes, fancier alternatives (to be discussed another day)
     work better, but not always.




                       Daniel Lemire   Faster Column-Oriented Indexes
Thankfully, we can detect cases where recursive orders are
good enough



  We can bound the suboptimality of all recursive orders.
  Proposition

  Consider a table with n distinct rows and column cardinalities Ni
  for i = 1, . . . , c. Recursive ordering is µ-optimal for the problem of
  minimizing the runs where

            min(N1 , n) + min(N1 N2 , n) + · · · + min(N1 N2 · · · Nc , n)
   µ =                                                                     .
                                        n




                           Daniel Lemire   Faster Column-Oriented Indexes
Bounding the optimality of sorting: the computation




      How do you compute µ very fast so you know lexicographical
      sort is good enough?
      Trick is to determine n, the number of distinct rows without
      sorting the table.
      Thankfully: n can be estimated quickly with probabilistic
      methods [Aouiche and Lemire, 2007].




                        Daniel Lemire   Faster Column-Oriented Indexes
Bounding the optimality of sorting: actual numbers




                         columns        µ
   Census-Income 4-D         4         2.63
   DBGEN 4-D                 4         1.02
   Netflix                    4         2.00
   Census1881               7          5.09




                       Daniel Lemire   Faster Column-Oriented Indexes
Take away message




     Column stores are good because of vectorization and
     RLE/sorting
     Sorting is sometimes nearly optimal, but not always but we
     can sometimes tell when sorting is optimal




                      Daniel Lemire   Faster Column-Oriented Indexes
Future direction?




      Minimizing the number of runs it the wrong problem! We
      want to maximize long runs!
      Must study fancier row-reordering heuristics.




                        Daniel Lemire   Faster Column-Oriented Indexes
Questions?




                             ?




             Daniel Lemire       Faster Column-Oriented Indexes
Aouiche, K. and Lemire, D. (2007).
A comparison of five probabilistic view-size estimation
techniques in OLAP.
In DOLAP’07, pages 17–24.
Hamilton, C. H. and Rau-Chaplin, A. (2007).
Compact Hilbert indices: Space-filling curves for domains with
unequal side lengths.
Information Processing Letters, 105(5):155–163.
Hyde, J. (2010).
Data in flight.
Commun. ACM, 53(1):48–52.
Knuth, D. E. (2005).
The Art of Computer Programming, volume 4, chapter fascicle
2.
Addison Wesley.
Lemire, D. and Kaser, O. (2009).

                   Daniel Lemire   Faster Column-Oriented Indexes
Reordering columns for smaller indexes.
in preparation, available from
http://arxiv.org/abs/0909.1346.
Lemire, D., Kaser, O., and Aouiche, K. (2010).
Sorting improves word-aligned bitmap indexes.
Data & Knowledge Engineering, 69(1):3–28.
O’Neil, P. and Quass, D. (1997).
Improved query performance with variant indexes.
In SIGMOD ’97, pages 38–49.
O’Neil, P. E. (1989).
Model 204 architecture and performance.
In 2nd International Workshop on High Performance
Transaction Systems, pages 40–59.
Stonebraker, M., Abadi, D. J., Batkin, A., Chen, X.,
Cherniack, M., Ferreira, M., Lau, E., Lin, A., Madden, S.,
O’Neil, E., O’Neil, P., Rasin, A., Tran, N., and Zdonik, S.
(2005).
                   Daniel Lemire   Faster Column-Oriented Indexes
C-store: a column-oriented DBMS.
In VLDB’05, pages 553–564.
Turner, M. J., Hammond, R., and Cotton, P. (1979).
A DBMS for large statistical databases.
In VLDB’79, pages 319–327.
Zaker, M., Phon-Amnuaisuk, S., and Haw, S. (2008).
An adequate design for large data warehouse systems: Bitmap
index versus B-Tree index.
IJCC, 2(2).




                  Daniel Lemire   Faster Column-Oriented Indexes

More Related Content

Similar to Faster Column-Oriented Indexes

AWS July Webinar Series - Getting Started with Amazon DynamoDB
AWS July Webinar Series - Getting Started with Amazon DynamoDBAWS July Webinar Series - Getting Started with Amazon DynamoDB
AWS July Webinar Series - Getting Started with Amazon DynamoDBAmazon Web Services
 
Spring one2gx2010 spring-nonrelational_data
Spring one2gx2010 spring-nonrelational_dataSpring one2gx2010 spring-nonrelational_data
Spring one2gx2010 spring-nonrelational_dataRoger Xia
 
Faster Practical Block Compression for Rank/Select Dictionaries
Faster Practical Block Compression for Rank/Select DictionariesFaster Practical Block Compression for Rank/Select Dictionaries
Faster Practical Block Compression for Rank/Select DictionariesRakuten Group, Inc.
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQLYan Cui
 
Scala collections wizardry - Scalapeño
Scala collections wizardry - ScalapeñoScala collections wizardry - Scalapeño
Scala collections wizardry - ScalapeñoSagie Davidovich
 
An Efficient Language Model Using Double-Array Structures
An Efficient Language Model Using Double-Array StructuresAn Efficient Language Model Using Double-Array Structures
An Efficient Language Model Using Double-Array StructuresJun-ya Norimatsu
 
EarGram: an Application for Interactive Exploration of Large Databases of Aud...
EarGram: an Application for Interactive Exploration of Large Databases of Aud...EarGram: an Application for Interactive Exploration of Large Databases of Aud...
EarGram: an Application for Interactive Exploration of Large Databases of Aud...Gilberto Bernardes
 
Introduction of c_language
Introduction of c_languageIntroduction of c_language
Introduction of c_languageSINGH PROJECTS
 
Scaling Out With Hadoop And HBase
Scaling Out With Hadoop And HBaseScaling Out With Hadoop And HBase
Scaling Out With Hadoop And HBaseAge Mooij
 
Apache Cassandra, part 1 – principles, data model
Apache Cassandra, part 1 – principles, data modelApache Cassandra, part 1 – principles, data model
Apache Cassandra, part 1 – principles, data modelAndrey Lomakin
 
Wrapper induction construct wrappers automatically to extract information f...
Wrapper induction   construct wrappers automatically to extract information f...Wrapper induction   construct wrappers automatically to extract information f...
Wrapper induction construct wrappers automatically to extract information f...George Ang
 
Topological Data Analysis
Topological Data AnalysisTopological Data Analysis
Topological Data AnalysisDeviousQuant
 
Dynamodb Presentation
Dynamodb PresentationDynamodb Presentation
Dynamodb Presentationadvaitdeo
 
Cassandra Tutorial
Cassandra TutorialCassandra Tutorial
Cassandra Tutorialmubarakss
 

Similar to Faster Column-Oriented Indexes (20)

What the C?
What the C?What the C?
What the C?
 
AWS July Webinar Series - Getting Started with Amazon DynamoDB
AWS July Webinar Series - Getting Started with Amazon DynamoDBAWS July Webinar Series - Getting Started with Amazon DynamoDB
AWS July Webinar Series - Getting Started with Amazon DynamoDB
 
Spring one2gx2010 spring-nonrelational_data
Spring one2gx2010 spring-nonrelational_dataSpring one2gx2010 spring-nonrelational_data
Spring one2gx2010 spring-nonrelational_data
 
Faster Practical Block Compression for Rank/Select Dictionaries
Faster Practical Block Compression for Rank/Select DictionariesFaster Practical Block Compression for Rank/Select Dictionaries
Faster Practical Block Compression for Rank/Select Dictionaries
 
Sql rally 2013 columnstore indexes
Sql rally 2013   columnstore indexesSql rally 2013   columnstore indexes
Sql rally 2013 columnstore indexes
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQL
 
Scala collections wizardry - Scalapeño
Scala collections wizardry - ScalapeñoScala collections wizardry - Scalapeño
Scala collections wizardry - Scalapeño
 
An Efficient Language Model Using Double-Array Structures
An Efficient Language Model Using Double-Array StructuresAn Efficient Language Model Using Double-Array Structures
An Efficient Language Model Using Double-Array Structures
 
Realtime Analytics
Realtime AnalyticsRealtime Analytics
Realtime Analytics
 
Data representation
Data representationData representation
Data representation
 
Nosql
NosqlNosql
Nosql
 
EarGram: an Application for Interactive Exploration of Large Databases of Aud...
EarGram: an Application for Interactive Exploration of Large Databases of Aud...EarGram: an Application for Interactive Exploration of Large Databases of Aud...
EarGram: an Application for Interactive Exploration of Large Databases of Aud...
 
Introduction of c_language
Introduction of c_languageIntroduction of c_language
Introduction of c_language
 
Scaling Out With Hadoop And HBase
Scaling Out With Hadoop And HBaseScaling Out With Hadoop And HBase
Scaling Out With Hadoop And HBase
 
Apache Cassandra, part 1 – principles, data model
Apache Cassandra, part 1 – principles, data modelApache Cassandra, part 1 – principles, data model
Apache Cassandra, part 1 – principles, data model
 
Bayesian Counters
Bayesian CountersBayesian Counters
Bayesian Counters
 
Wrapper induction construct wrappers automatically to extract information f...
Wrapper induction   construct wrappers automatically to extract information f...Wrapper induction   construct wrappers automatically to extract information f...
Wrapper induction construct wrappers automatically to extract information f...
 
Topological Data Analysis
Topological Data AnalysisTopological Data Analysis
Topological Data Analysis
 
Dynamodb Presentation
Dynamodb PresentationDynamodb Presentation
Dynamodb Presentation
 
Cassandra Tutorial
Cassandra TutorialCassandra Tutorial
Cassandra Tutorial
 

More from Daniel Lemire

Accurate and efficient software microbenchmarks
Accurate and efficient software microbenchmarksAccurate and efficient software microbenchmarks
Accurate and efficient software microbenchmarksDaniel Lemire
 
Fast indexes with roaring #gomtl-10
Fast indexes with roaring #gomtl-10 Fast indexes with roaring #gomtl-10
Fast indexes with roaring #gomtl-10 Daniel Lemire
 
Parsing JSON Really Quickly: Lessons Learned
Parsing JSON Really Quickly: Lessons LearnedParsing JSON Really Quickly: Lessons Learned
Parsing JSON Really Quickly: Lessons LearnedDaniel Lemire
 
Next Generation Indexes For Big Data Engineering (ODSC East 2018)
Next Generation Indexes For Big Data Engineering (ODSC East 2018)Next Generation Indexes For Big Data Engineering (ODSC East 2018)
Next Generation Indexes For Big Data Engineering (ODSC East 2018)Daniel Lemire
 
Ingénierie de la performance au sein des mégadonnées
Ingénierie de la performance au sein des mégadonnéesIngénierie de la performance au sein des mégadonnées
Ingénierie de la performance au sein des mégadonnéesDaniel Lemire
 
SIMD Compression and the Intersection of Sorted Integers
SIMD Compression and the Intersection of Sorted IntegersSIMD Compression and the Intersection of Sorted Integers
SIMD Compression and the Intersection of Sorted IntegersDaniel Lemire
 
Decoding billions of integers per second through vectorization
Decoding billions of integers per second through vectorizationDecoding billions of integers per second through vectorization
Decoding billions of integers per second through vectorizationDaniel Lemire
 
Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...
Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...
Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...Daniel Lemire
 
Engineering fast indexes (Deepdive)
Engineering fast indexes (Deepdive)Engineering fast indexes (Deepdive)
Engineering fast indexes (Deepdive)Daniel Lemire
 
Engineering fast indexes
Engineering fast indexesEngineering fast indexes
Engineering fast indexesDaniel Lemire
 
Roaring Bitmaps (January 2016)
Roaring Bitmaps (January 2016)Roaring Bitmaps (January 2016)
Roaring Bitmaps (January 2016)Daniel Lemire
 
Roaring Bitmap : June 2015 report
Roaring Bitmap : June 2015 reportRoaring Bitmap : June 2015 report
Roaring Bitmap : June 2015 reportDaniel Lemire
 
La vectorisation des algorithmes de compression
La vectorisation des algorithmes de compression La vectorisation des algorithmes de compression
La vectorisation des algorithmes de compression Daniel Lemire
 
Decoding billions of integers per second through vectorization
Decoding billions of integers per second through vectorization  Decoding billions of integers per second through vectorization
Decoding billions of integers per second through vectorization Daniel Lemire
 
Extracting, Transforming and Archiving Scientific Data
Extracting, Transforming and Archiving Scientific DataExtracting, Transforming and Archiving Scientific Data
Extracting, Transforming and Archiving Scientific DataDaniel Lemire
 
Innovation without permission: from Codd to NoSQL
Innovation without permission: from Codd to NoSQLInnovation without permission: from Codd to NoSQL
Innovation without permission: from Codd to NoSQLDaniel Lemire
 
All About Bitmap Indexes... And Sorting Them
All About Bitmap Indexes... And Sorting ThemAll About Bitmap Indexes... And Sorting Them
All About Bitmap Indexes... And Sorting ThemDaniel Lemire
 
A Comparison of Five Probabilistic View-Size Estimation Techniques in OLAP
A Comparison of Five Probabilistic View-Size Estimation Techniques in OLAPA Comparison of Five Probabilistic View-Size Estimation Techniques in OLAP
A Comparison of Five Probabilistic View-Size Estimation Techniques in OLAPDaniel Lemire
 

More from Daniel Lemire (20)

Accurate and efficient software microbenchmarks
Accurate and efficient software microbenchmarksAccurate and efficient software microbenchmarks
Accurate and efficient software microbenchmarks
 
Fast indexes with roaring #gomtl-10
Fast indexes with roaring #gomtl-10 Fast indexes with roaring #gomtl-10
Fast indexes with roaring #gomtl-10
 
Parsing JSON Really Quickly: Lessons Learned
Parsing JSON Really Quickly: Lessons LearnedParsing JSON Really Quickly: Lessons Learned
Parsing JSON Really Quickly: Lessons Learned
 
Next Generation Indexes For Big Data Engineering (ODSC East 2018)
Next Generation Indexes For Big Data Engineering (ODSC East 2018)Next Generation Indexes For Big Data Engineering (ODSC East 2018)
Next Generation Indexes For Big Data Engineering (ODSC East 2018)
 
Ingénierie de la performance au sein des mégadonnées
Ingénierie de la performance au sein des mégadonnéesIngénierie de la performance au sein des mégadonnées
Ingénierie de la performance au sein des mégadonnées
 
SIMD Compression and the Intersection of Sorted Integers
SIMD Compression and the Intersection of Sorted IntegersSIMD Compression and the Intersection of Sorted Integers
SIMD Compression and the Intersection of Sorted Integers
 
Decoding billions of integers per second through vectorization
Decoding billions of integers per second through vectorizationDecoding billions of integers per second through vectorization
Decoding billions of integers per second through vectorization
 
Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...
Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...
Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...
 
Engineering fast indexes (Deepdive)
Engineering fast indexes (Deepdive)Engineering fast indexes (Deepdive)
Engineering fast indexes (Deepdive)
 
Engineering fast indexes
Engineering fast indexesEngineering fast indexes
Engineering fast indexes
 
Roaring Bitmaps (January 2016)
Roaring Bitmaps (January 2016)Roaring Bitmaps (January 2016)
Roaring Bitmaps (January 2016)
 
Roaring Bitmap : June 2015 report
Roaring Bitmap : June 2015 reportRoaring Bitmap : June 2015 report
Roaring Bitmap : June 2015 report
 
La vectorisation des algorithmes de compression
La vectorisation des algorithmes de compression La vectorisation des algorithmes de compression
La vectorisation des algorithmes de compression
 
OLAP and more
OLAP and moreOLAP and more
OLAP and more
 
Decoding billions of integers per second through vectorization
Decoding billions of integers per second through vectorization  Decoding billions of integers per second through vectorization
Decoding billions of integers per second through vectorization
 
Extracting, Transforming and Archiving Scientific Data
Extracting, Transforming and Archiving Scientific DataExtracting, Transforming and Archiving Scientific Data
Extracting, Transforming and Archiving Scientific Data
 
Innovation without permission: from Codd to NoSQL
Innovation without permission: from Codd to NoSQLInnovation without permission: from Codd to NoSQL
Innovation without permission: from Codd to NoSQL
 
Write good papers
Write good papersWrite good papers
Write good papers
 
All About Bitmap Indexes... And Sorting Them
All About Bitmap Indexes... And Sorting ThemAll About Bitmap Indexes... And Sorting Them
All About Bitmap Indexes... And Sorting Them
 
A Comparison of Five Probabilistic View-Size Estimation Techniques in OLAP
A Comparison of Five Probabilistic View-Size Estimation Techniques in OLAPA Comparison of Five Probabilistic View-Size Estimation Techniques in OLAP
A Comparison of Five Probabilistic View-Size Estimation Techniques in OLAP
 

Recently uploaded

Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clashcharlottematthew16
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostZilliz
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 

Recently uploaded (20)

Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 

Faster Column-Oriented Indexes

  • 1. Faster Column-Oriented Indexes Daniel Lemire http://www.professeurs.uqam.ca/pages/lemire.daniel.htm blog: http://www.daniel-lemire.com/ Joint work with Owen Kaser (UNB) and Kamel Aouiche (post-doc). February 10, 2010 Daniel Lemire Faster Column-Oriented Indexes
  • 2. Some trends in business intelligence (BI) Low-latency BI, Complex Event Processing [Hyde, 2010] Commotization, open source software: Pentaho, LucidDB (http://www.luciddb.org/) Column-oriented databases ← source: gooddata.com Daniel Lemire Faster Column-Oriented Indexes
  • 3. Row Stores name, date, age, sex, salary name, date, age, sex, salary name, date, age, sex, salary Dominant paradigm name, date, age, sex, salary Transactional: Quick append and delete name, date, age, sex, salary Daniel Lemire Faster Column-Oriented Indexes
  • 4. Column Stores Goes back to StatCan in the seventies [Turner et al., 1979] Made fashionable again in Data name date age sex salary Warehousing by Stonebraker [Stonebraker et al., 2005] New: Oracle Exadata hybrid columnar compression Daniel Lemire Faster Column-Oriented Indexes
  • 5. Vectorization Modern superscalar CPUs support const i n t N = 2048; vectorization (SSE) i n t a [N] , b [N ] ; This code is four times faster with i n t i =0; -ftree-vectorize (GNU GCC) f o r ( ; i <N ; i ++) Need long streams, same data type, and a [ i ] += b [ i ] ; no branching. Columns are good candidates! Daniel Lemire Faster Column-Oriented Indexes
  • 6. Main column-oriented indexes (1) Bitmap indexes [O’Neil, 1989] (2) Projection indexes [O’Neil and Quass, 1997] Both are compressible. Daniel Lemire Faster Column-Oriented Indexes
  • 7. Bitmap indexes SELECT * FROM T WHERE x=a Vectors of booleans AND y=b; Above, compute {r | r is the row id of a row where x = a} ∩ {r | r is the row id of a row where y = b} Daniel Lemire Faster Column-Oriented Indexes
  • 8. Other applications of the bitmaps/bitsets The Java language has had a bitmap class since the beginning: java.util.BitSet. (Sun’s implementation is based on 64-bit words.) Search engines use bitmaps to filter queries, e.g. Apache Lucene: org.apache.lucene.util.OpenBitSet.java. Daniel Lemire Faster Column-Oriented Indexes
  • 9. Bitmaps and fast AND/OR operations Computing the union of two sets of integers between 1 and 64 (eg row ids, trivial table). . . E.g., {1, 5, 8} ∪ {1, 3, 5}? Can be done in one operation by a CPU: BitwiseOR( 10001001, 10101000) Extend to sets from 1..N using N/64 operations. To compute [a0 , . . . , aN−1 ] ∨ [b0 , b1 , . . . , bN−1 ] : a0 , . . . , a63 BitwiseOR b0 , . . . , b63 ; a64 , . . . , a127 BitwiseOR b64 , . . . , b127 ; a128 , . . . , a192 BitwiseOR b128 , . . . , b192 ; ... It is a form of vectorization. Daniel Lemire Faster Column-Oriented Indexes
  • 10. What are bitmap indexes for? Myth: bitmap indexes are for low cardinality columns (e.g., SEX). the Bitmap index is the conclusive choice for data warehouse design for columns with high or low cardinality [Zaker et al., 2008]. Daniel Lemire Faster Column-Oriented Indexes
  • 11. Projection indexes name date city Write out the (normalized) column values sequentially. It is a projection of the table on a single column. name Best for low selectivity queries on few columns: date SELECT sum(number*price) city FROM T;. Daniel Lemire Faster Column-Oriented Indexes
  • 12. How to compress column indexes? Must handle long streams of identical values efficiently ⇒ Run-length encoding? (RLE) Bitmap: a run of 0s, a run of 1s, a run of 0s, a run of 1s, . . . So just encode the run lengths, e.g., 0001111100010111 → 3, 5, 3, 1,1,3 It is a bit more complicated (more another day) Daniel Lemire Faster Column-Oriented Indexes
  • 13. What about other compression types? With RLE, we can often process the data in compressed form Hence, with RLE, compression saves both storage and CPU cycles!!!! Not always true with other techniques such as Huffman, LZ77, Arithmetic Coding, . . . Daniel Lemire Faster Column-Oriented Indexes
  • 14. How do we improve performance? Smaller indexes are faster. In data warehousing: data is often updated in batches. So spend time at construction time optimizing the index. Daniel Lemire Faster Column-Oriented Indexes
  • 15. Modelling the size of an index Any formal result? Tricky: There are many variations on RLE. Use: number of runs of identical value in a column AAABBBCCAA has 4 runs Daniel Lemire Faster Column-Oriented Indexes
  • 16. Improving compression by reordering the rows RLE is order-sensitive: they compress sorted tables better; But finding the best row ordering is NP-hard [Lemire et al., 2010]. Actually an instance of the Traveling Salesman Problem (TSP) So we use heuristics: lexicographically Gray codes Hilbert, . . . Daniel Lemire Faster Column-Oriented Indexes
  • 17. How many ways to sort? (1) Lexicographic row sorting is a a fast, even for very large a b tables. a c easy: sort is a Unix staple. b a Substantial index-size reductions b b (often 2.5 times, benefits grow b c with table size) Daniel Lemire Faster Column-Oriented Indexes
  • 18. How many ways to sort? (2) Gray Codes are list of tuples a a with successive (Hamming) a b distance of 1 [Knuth, 2005]. a c b c Reflected Gray Code order is b b sometimes slightly better than lexicographical. . . b a Daniel Lemire Faster Column-Oriented Indexes
  • 19. How many ways to sort? (3) a a Reflected Gray Code order is not a b the only Gray code. a c b c Knuth also presents Modular b a Gray-code. b b Daniel Lemire Faster Column-Oriented Indexes
  • 20. How many ways to sort? (4) Hilbert Index [Hamilton and Rau-Chaplin, 2007]. Also a Gray code (conditionnally) Gives very bad results for column-oriented indexes. Daniel Lemire Faster Column-Oriented Indexes
  • 21. Recursive orders Lexicographical, reflected Gray code and modular Gray code belong to a larger class: recursive orders. They sort on the first column, then the second and so on. Not all Gray codes are recursive orders: Hilbert is not. Daniel Lemire Faster Column-Oriented Indexes
  • 22. Best column order? Column order is important for recursive orders. We almost have this result [Lemire and Kaser, 2009]: any recursive order order the columns by increasing cardinality (small to LARGE) Proposition The expected number of runs is minimized (among all possible column orders). Daniel Lemire Faster Column-Oriented Indexes
  • 23. How do you know when the lexicographical order is good enough? Even though row reordering is NP-hard, we find it hard to improve over recursive orders. Sometimes, fancier alternatives (to be discussed another day) work better, but not always. Daniel Lemire Faster Column-Oriented Indexes
  • 24. Thankfully, we can detect cases where recursive orders are good enough We can bound the suboptimality of all recursive orders. Proposition Consider a table with n distinct rows and column cardinalities Ni for i = 1, . . . , c. Recursive ordering is µ-optimal for the problem of minimizing the runs where min(N1 , n) + min(N1 N2 , n) + · · · + min(N1 N2 · · · Nc , n) µ = . n Daniel Lemire Faster Column-Oriented Indexes
  • 25. Bounding the optimality of sorting: the computation How do you compute µ very fast so you know lexicographical sort is good enough? Trick is to determine n, the number of distinct rows without sorting the table. Thankfully: n can be estimated quickly with probabilistic methods [Aouiche and Lemire, 2007]. Daniel Lemire Faster Column-Oriented Indexes
  • 26. Bounding the optimality of sorting: actual numbers columns µ Census-Income 4-D 4 2.63 DBGEN 4-D 4 1.02 Netflix 4 2.00 Census1881 7 5.09 Daniel Lemire Faster Column-Oriented Indexes
  • 27. Take away message Column stores are good because of vectorization and RLE/sorting Sorting is sometimes nearly optimal, but not always but we can sometimes tell when sorting is optimal Daniel Lemire Faster Column-Oriented Indexes
  • 28. Future direction? Minimizing the number of runs it the wrong problem! We want to maximize long runs! Must study fancier row-reordering heuristics. Daniel Lemire Faster Column-Oriented Indexes
  • 29. Questions? ? Daniel Lemire Faster Column-Oriented Indexes
  • 30. Aouiche, K. and Lemire, D. (2007). A comparison of five probabilistic view-size estimation techniques in OLAP. In DOLAP’07, pages 17–24. Hamilton, C. H. and Rau-Chaplin, A. (2007). Compact Hilbert indices: Space-filling curves for domains with unequal side lengths. Information Processing Letters, 105(5):155–163. Hyde, J. (2010). Data in flight. Commun. ACM, 53(1):48–52. Knuth, D. E. (2005). The Art of Computer Programming, volume 4, chapter fascicle 2. Addison Wesley. Lemire, D. and Kaser, O. (2009). Daniel Lemire Faster Column-Oriented Indexes
  • 31. Reordering columns for smaller indexes. in preparation, available from http://arxiv.org/abs/0909.1346. Lemire, D., Kaser, O., and Aouiche, K. (2010). Sorting improves word-aligned bitmap indexes. Data & Knowledge Engineering, 69(1):3–28. O’Neil, P. and Quass, D. (1997). Improved query performance with variant indexes. In SIGMOD ’97, pages 38–49. O’Neil, P. E. (1989). Model 204 architecture and performance. In 2nd International Workshop on High Performance Transaction Systems, pages 40–59. Stonebraker, M., Abadi, D. J., Batkin, A., Chen, X., Cherniack, M., Ferreira, M., Lau, E., Lin, A., Madden, S., O’Neil, E., O’Neil, P., Rasin, A., Tran, N., and Zdonik, S. (2005). Daniel Lemire Faster Column-Oriented Indexes
  • 32. C-store: a column-oriented DBMS. In VLDB’05, pages 553–564. Turner, M. J., Hammond, R., and Cotton, P. (1979). A DBMS for large statistical databases. In VLDB’79, pages 319–327. Zaker, M., Phon-Amnuaisuk, S., and Haw, S. (2008). An adequate design for large data warehouse systems: Bitmap index versus B-Tree index. IJCC, 2(2). Daniel Lemire Faster Column-Oriented Indexes