Column oriented Transactions


Published on

Row Store Implementation of Database

Published in: Education, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Column oriented Transactions

  1. 1. Column-Oriented Query Processing for Row Stores
  2. 2. INTRODUCTION OBJECTIVE AND SCOPE Column-oriented DBMSs have gained increasing interest dueto their superior performance for analytical workloads. Priorefforts tried to determine the possibility of simulating the Query processing techniques of column-oriented systems inrow-oriented databases, in a hope to improve their performance,especially for OLAP and data warehousing applications. We show that column-oriented queryprocessing can significantly improve the performance of row-orientedDBMSs. We introduce new operators that take intoaccount the unique characteristics of data obtained from indexes,and exploit new technologies such as flash SSDs andmulti-core processors to boost the performance. We demonstrateour approach with an experimental study using a prototypebuilt on a commercial row-oriented DBMS. Recently, column-oriented database systems (also knownas column stores) have been receiving a lot of attention. The main difference between column stores and the traditionalrow-oriented database systems (also known as row stores)is in the way the data is physically stored. As the namesimply, both kinds of database systems store data differently. In row-stores, the DBMS stores all the attribute values ofa single row in contiguous space on disk. Column-orientedDBMSs, on the other hand, store the data in a column-wisemanner (the values of a column are stored contiguously).This difference in data storage has an implication on dataaccess. In row stores, a whole row of data has to be read fromdisk, even if only few columns of that row are needed to answera query. This means that the system may have to readmuch more data than it actually needs. In column stores,only the required columns are read from disk, although someextra processing is needed to construct the output tuplesfrom these individual columns. However, when it comes todata updates, column stores tend to be at a disadvantage.Since data updates (or inserts) usually happen at row granularity,multiple accesses are needed to insert or update therelevant columns of each updated row. In row stores, thewhole row is written in a single operation.This difference in access patterns and performance hasmade column stores more suitable for workloads that areread-intensive with few or no updates, e.g., analytical processingworkloads, such as those found in data warehousesor decision support systems.
  3. 3. Related Area We describe a column-oriented query processingmechanism for row stores based on indexonly plans,operating on single-column indexes. In our approach, onlythe indexes relevant to the query are read, without readinganything from the underlying base tables. This leads toavoiding the cost of reading the entire table rows when onlya few columns are referenced in the query. Seminar is related to DBMS and DATA WAREHOUSING.
  4. 4. Process Description These column-oriented designs include: _ vertically partitioning the tables in the database intoa set of two-column tables, each consisting of (key,attribute) pairs, so that only the necessary columnsneed to be read to answer a query. The key is used tojoin these tables to reconstruct the output tuples. Using index-only plans; by creating a collection of indexes that cover all of the columns used in a query; it is possible for the DBMS to answer a query without ever going to the underlying (row-oriented) tables. Such column-orientedphysical designs do not improve the performance of rowstores. In fact, in their results, these techniques result inworse performance than the baseline row-oriented processing. Above depicts the results of their experiments (normalizedrelative to the row store time). This behaviour wasmainly because they tried to employ these techniques withoutany changes to the underlying systems, which causedthe optimizer to make some bad decisions and use very expensiveoperations. However, if there are operators that arespecially designed for column-oriented processing, this canlead to a significant performance improvement. We demonstrate the potential of column-orientedquery processing in row stores, focusing on usingindex-only plans, but with query operators that take advantageof the characteristics of the data obtained from theindexes. Our prototype is based on a commercial row store(which we shall refer to as System A). We compare the performanceof our technique to the baseline performance ofSystem A, as well as to that of a leading open-source columnstore (which we shall refer to as Col-DB). Our resultsshow that column-oriented processing can significantly improvethe performance of row stores for analytical workloads,and even approach the performance of a column store. Insummary, we make the following contributions:
  5. 5. 1. We show that column-oriented query processing techniques,when employed using proper operators andmechanisms, can significantly improve the performanceof a row store, and even approach the performance ofa column-store. Our main focus is index-only plans. 2. We introduce new ways of performing index intersectionthat leverage parallel processing, to be used inquery execution using index-only plans; namely theindex-merge, index-merge-join, and indexhash-join operators.We describe the algorithms, and present a costanalysis for each operator. 3. We highlight the advantage of exploiting new hardwaretechnologies – namely flash SSDs and multi-coreprocessors – to reduce query processing time. 4. We demonstrate the effectiveness of our approach withan extensive experimental study using a prototype basedon System A, and we compare the performance of ourapproach to those of a baseline row store (System A)and column store (Col-DB). We describe a column-oriented query processingmechanism for row stores based on index-only plans,operating on single-column indexes. In our approach, onlythe indexes relevant to the query are read, without readinganything from the underlying base tables. This leads toavoiding the cost of reading the entire table rows when onlya few columns are referenced in the query. Weintroduce new specialized ways of combining the data obtainedfrom the indexes that take advantage of the uniquecharacteristics of database indexes, instead of completely relyingon the existing database operators.Our algorithmsare designed specifically to take advantage of parallelprocessing whenever possible, which leads to better performance. The new operators are: Index Merge Merges the sorted RID-lists associated withindividual values from the index, producing a (RID,value) list for the whole index, in RID order. Index Merge Join Performs an N-way merge join operationbetween the (RID, value) lists coming from severalindexes. The input lists have to be in RID order (e.g.,produced by the Index Merge operator). The outputis a (RID, data) list, also in RID order, where the datais a collection of values). Index Hash Join Performs an N-way hash join operationbetween N (RID, data) lists. The input lists can bethe output of any of the previous two operators, anotherIndex Hash Join operator, or the traditional Index Scan operator that exists in most row stores. Theinput lists do not need to be in any order. One of theinput lists is used to build the hash table while theothers are used for probing. The output is a (RID,data) list.
  6. 6. Testing Technologies  INDEX MERGE (IXMG)  INDEX MERGE JOIN (IXMGJN)  Parallel Processing     INDEX HASH JOIN (IXHSJN) In-place update (IPU) Linked List (LL) Linked List with Partitioning (LLP)
  7. 7. Resources and Limitations All of our experiments were run on an IBM System x3400(Model 7974) server, with a 1.6 GHz single processor, dualcore Intel Xeon CPU with 4 GB of RAM running FedoraLinux. The server has a 1 TB Seagate SATA disk, with105 MB/sec sustained data rate at 7.2K RPM. The serveralso has an 80 GB Fusion-IO IO-Disk SSD with a read bandwidthof 700 MB/sec and a write bandwidth of 550 MB/sec. The numbers we report are the average of several runs, andare based on a cold buffer pool. We built a prototype usingSystem A. We used two TPC-H [2] data sets for ourexperiments, with scale factors 10 and 30. The lineitemtable(fact table with 16 columns) contains 60 million rowsin the 10GB database, and 180 million rows in the 30GBdatabase. A typical data warehousing query involves thefact table (lineitem) and one or more dimension tables, withpredicates on the fact and/or dimension tables.
  8. 8. Future scope and further enhancement Several recent papers [3, 5, 8] have tackled the issue ofcomparing the performance of row stores and column stores,and proposing optimizations for row stores in order for themto compete with column stores for analytical workloads. The work in [5] compares row stores and column stores.In particular, the authors try to determine whether columnstore performance can be achieved in row stored usingcolumnoriented query processing. They simulate column-orientedprocessing by using both vertical partitioning andindex-only plans in a commercial database system (referredto in the paper as System X"), and compare the performanceof this system to the performance of C-Store [22]. In above Figure, System X performed very poorly withvertical partitioning, and even worse with index-only plans.The reason for such a poor performance is because SystemX had to join the (RID, value) lists coming from each indexusing a series of 2-way hash joins, each of which builds anew hash table, which can be very expensive. Bruno [8] argues that it is possible to achieve the performanceof column stores (or very close to it) within row storeswithout any changes to the DBMS. The author proposes amethod to store data in a compressed form, using a separatetable for each column in the original schema (similar tothe vertical partitioning idea), but with these tables (called“c-tables”) storing data in a format similar to run-length encoding. The results show great performance improvement.However, this scheme requires creating a c-table for everycolumn, and creating multiple indexes on every c-table, thusthe storage requirements can grow rapidly. Additionally, thedependency between the tuples in the c-table causes insertingnew rows to become very expensive, even for workloadswith infrequent updates.Several optimizations have been implemented or proposedfor row-stores to approach column-store performance, suchas super-tuples, using a column-based layout within eachdata page, operating on compressed data, block processing,mirroring, invisible joins, late materialization, and column-storeindexes [4, 12, 13, 15, 19, 20, 23, 24]. The idea of using multiple indexes to access data has beenstudied for some time. Mohan et al. [16] introduced methodsfor using multiple indexes together to access base tables. Raman et al. [21] presented two algorithms to perform theRID-list intersection operation, which are comparable to thealgorithms we use for the low-cardinality indexes. Flash memory has received increasing interest as a stablestorage media that can overcome the access bottlenecks ofmagnetic disks. Researchers have considered modifying existingalgorithms and data structures to make use of flashmemory. Flash-DB [17] is a selftuning flash-based index thatdynamically adapts to the mix of reads and writes in theworkload. Flash-Logging [10] exploits using flash in appendmode for synchronous logging.
  9. 9. Conclusion Column-oriented processing can indeed improve query performancein row stores significantly. Such performance improvementscan only be seen when changes are made to thedatabase system. These changes should also make use ofnewly emerging technologies. For example, multi-core processorsprovide a level of parallelism that can achieve betterperformance, and new media such as flash memory canbe particularly used as temporary storage, as it providesfast random access. Our work exploits these technologies tobring the performance of a row store close to that of a columnstore while retaining roworiented capabilities (such asfast single-row lookups). This extends the scope for which asingle database system can be used, where one does not needseparate software for the warehouse and the OLTP system.
  10. 10. References [1] MySQL, http:// [2] TPC-H Benchmark, [3] D. J. Abadi, P. A. Boncz, and S. Harizopoulos. Column oriented Database Systems. PVLDB, 2(2):1664–1665, 2009. [4] D. J. Abadi, S. Madden, and M. Ferreira. Integrating Compression and Execution in Column-oriented Database Systems.In SIGMOD, 2006. [5] D. J. Abadi, S. R. Madden, and N. Hachem. Column-Stores vs. Row-Stores: How Different Are They Really? In SIGMOD, 2008. [6] P. A. Boncz and M. L. Kersten.MIL Primitives for Querying a Fragmented World.VLDB Journal, 8(2):101–119, 1999. [7] P. A. Boncz, M. Zukowski, and N. Nes.MonetDB/X100: Hyper-Pipelining Query Execution. In CIDR, 2005. [8] N. Bruno. Teaching an Old Elephant New Tricks.In CIDR, 2009. [9] M. Canim, G. A. Mihaila, B. Bhattacharjee, C. A. Lang, and K. A. Ross.Buffered Bloom Filters on Solid State Storage.In ADMS, 2010. [10] S. Chen. FlashLogging: Exploiting Flash Devices for Synchronous Logging Performance. In SIGMOD, 2009. [11] B. Debnath, S. Sengupta, and J. Li. FlashStore: High Throughput Persistent Key-Value Store. In VLDB, 2010. [12] G. Graefe. Efficient Columnar Storage in B-trees. SIGMOD Record, 36(1):3–6, 2007. [13] A. Halverson, J. L. Beckmann, J. F. Naughton, and D. J. Dewitt.A Comparison of CStore and Row-Store in a Common Framework.Technical Report TR1570, University of Wisconsin-Madison, 2006. [14] Y.-R. Kim, K.-Y.Whang, and I.-Y. Song. Page-Differential Logging: An Efficient and DBMS-Independent Approach for Storing Data into Flash Memory. In SIGMOD, 2010. [15] P.-˚ A. Larson, C. Clinciu, E. N. Hanson, A. Oks, S. L. Price, S. Rangarajan, A. Surna, and Q. Zhou. SQL Server Column Store Indexes.In SIGMOD, 2011. [16] C. Mohan, D. J. Haderle, Y. Wang, and J. M. Cheng. Single Table Access Using Multiple Indexes: Optimization, Execution, and Concurrency Control Techniques. In EDBT, 1990. [17] S. Nath and A. Kansal.FlashDB: Dynamic Self-Tuning Database for NAND Flash. In IPSN, 2007. [18] S. Padmanabhan, B. Bhattacharjee, T. Malkemus, L. Cranston, and M. Huras. MultiDimensional Clustering: A New Data Layout Scheme in DB2. In SIGMOD, 2003. [19] S. Padmanabhan, T. Malkemus, R. C. Agarwal, and A. Jhingran. Block Oriented Processing of Relational Database Operations in Modern Computer Architectures. In ICDE, 2001. [20] R. Ramamurthy, D. J. DeWitt, and Q. Su.A Case for Fractured Mirrors.In VLDB, 2002. [21] V. Raman, L. Qiao, W. Han, I. Narang, Y.-L. Chen, K.-H.Yang, and F.-L. Ling. Lazy, Adaptive RID-List Intersection, and Its Application to Index Anding.In SIGMOD, 2007. [22] M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden, E. O’Neil, P. O’Neil, A. Rasin, N. Tran, and S. Zdonik. C-Store: A ColumnOriented DBMS. In VLDB, 2005. [23] D. Tsirogiannis, S. Harizopoulos, M. A. Shah, J. L. Wiener, and G. Graefe. Query Processing Techniques for Solid State Drives.In SIGMOD, 2009.