Column orientation - rotate your thinking 90 degrees
Upcoming SlideShare
Loading in...5
×
 

Column orientation - rotate your thinking 90 degrees

on

  • 4,086 views

With ever increasing data and greater analytics requirements, a new breed of databases is becoming popular - column-based databases. Some popular real world examples of column based DBs are - Sybase ...

With ever increasing data and greater analytics requirements, a new breed of databases is becoming popular - column-based databases. Some popular real world examples of column based DBs are - Sybase IQ, Vertica, and to some degree, Infobright - MySQL's column based storage engine. These databases store data "column-wise" in pages instead of "row-wise". This re-orientation claims to provide significant advantages over row-based storage for read type analytics queries. In my talk, I will discuss the technicalities, benefits and motivating use-cases for column-based databases. We shall also see why more indexing or partitioning in a row-based storage won't achieve the same effect.

Statistics

Views

Total Views
4,086
Views on SlideShare
4,058
Embed Views
28

Actions

Likes
3
Downloads
85
Comments
1

4 Embeds 28

http://www.linkedin.com 12
http://www.slideshare.net 10
https://www.linkedin.com 5
http://www.lmodules.com 1

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Column orientation - rotate your thinking 90 degrees Column orientation - rotate your thinking 90 degrees Presentation Transcript

  • Column based Databases Shweta Agrawal    
  • What is a column based DB? ID NAME SEX AGE SALARY ADDRRESS PHONE PAN... 1 Sunil Sharma M 40 10,000 ... ... ... 2 Neha Agarwal F 25 12,000 ... ... ... 3 Anant Agarwal M 28 15,000 ... ... ... 4 Vishal Mehta M 30 8,000 ... ... ... One page of the table storage 1|Shweta Agrawal|M| 1|2|3|4...|Shweta  40|10000...|2|Neha  Agrawal|Neha Agrawal| Agrawal|F|25| Anant Agarwal|Vishal  12000...|3|Anant  Mehta...|M|F|M|M...| Agarwal|M|28| 40|25|28|30...|10000| 15000...|4|Vishal  12000|15000|8000... Mehta|M|30|8000... Row based storage Column based storage    
  • Column stores 1|2|3|4|5| Shweta  M|F|M|M|M|F| 40|25|28|30| 10000|12000| 6|.... Agrawal|Neha  F... 45|20... 15000|8000| Agrawal| 15000| Anant  5000... Agarwal| Vishal  ... Mehta| Srinivas  Pathak| Rubina  Mehta.... 1st page of each column store    
  • Query processing on row store SELECT name, salary FROM employee WHERE age > 40  Evaluate condition age>40 possibly using an index  on age.  Get a found­set containing row number/ID of rows  that satisfy above condition.  Retrieve all rows in the above  found­set.  Send only name, and salary from the rows as result  to client    
  • Query processing on  a column  store SELECT name, salary FROM employee WHERE age > 40  Evaluate condition age > 40 on column age, using an  index if present  Get a found­set containing row number/ID of rows that  satisfy above condition  Retrieve name's from name's column store for all rows in  the found­set  Retrieve salary's from salary column for all rows in the  found­set  Associate name with salary by row id/number for final  result    
  • A quick calculation of IO  Table has 10 columns  1 million rows.  Each row is 100 bytes   30% of employees are above age 40  Total amount of data read in row based store =  100MB * 0.3 = 30MB  Total amount of data read in column based  store 100MB * 0.3 * 0.2 (only 2 columns) = 6MB    
  • Why is it important?  Wide fact tables in data­warehouses  Analytics queries on data­warehouse tend to  aggregate/analyse a few columns but a large  number of rows.  Full table scans for analytics queries in row  stores  Normalization means more joins    
  • An example star schema    
  • Benefits of column based DB  Low pages read = Less IO = faster queries  Processes CPU bound instead of IO bound  Compression  Page level compression  Column level compression (lookup tables)  Natural intra­query parallelism on conditions on  different columns    
  • Row based equivalents  Index every column?  Maintenance: updates/insert/deletes  Storage  Most importantly: Index is value=>id, column is  id=>value  Useful for selective queries only    
  • Row based equivalents  Vertical partitioning?  Joins (although fast ones)  Table overhead  Cannot use horizontal partitioning  Row based query engine not geared up to make  use of the column based storage.    
  • Summary  For adhoc analytics queries, column based  storage reduces IO, and makes queries faster  Column based query engines written ground up  for analytics queries make good use of this  storage.  Indexing every column, or vertical partioning not  same as column based storage.    
  • References  Commercial products  Sybase IQ  Vertica  MySQL's InfoBright storage engine  To know more, read  http://databasecolumn.vertica.com/