1) Current database management systems (DBMS) vendors take a "one size fits all" approach that is no longer suitable for different data and usage types. 2) Specialized architectures can outperform general-purpose row-oriented DBMSs by over 50 times in domains like data warehousing and online transaction processing (OLTP). 3) The DBMS market will likely transition over the next decade from "one size fits all" systems to specialized architectures tailored for individual domains like analytics, OLTP, scientific data, and streaming data. This will challenge traditional DBMS vendors.

1. One Size Fits AllAn Idea Whose Time Has Come and GonebyMichael Stonebraker

2. DBMS Vendors (The Elephants) SellOne Size Fits All (OSFA)It’s too hard for them to maintain multiple code bases for different specialized purposes * engineering problem * sales problem * marketing problem

3. The OSFA ElephantsSell code lines that date from the 1970’sLegacy code

4. Built for very different hardware configurations

5. And some cannot adapt to grids….That was designed for business data processing (OLTP)Only market back then

6. Now warehouses, science, real time, embedded, ..Current DBMS Gold StandardStore fields in one record contiguously on disk

7. Use B-tree indexing

8. Use small (e.g. 4K) disk blocks

9. Align fields on byte or word boundaries

10. Conventional (row-oriented) query optimizer and executorTerminology -- “Row Store” Record 1Record 2Record 3Record 4E.g. DB2, Oracle, Sybase, SQLServer, Greenplum, Netezza,Teradata,…

11. At This Point, RDBMS is “long in the tooth”There are at least 6 (non trivial) markets where a row store can be clobbered by a specialized architecture

12. Warehouses (Vertica, SybaseIQ, KX, …)

13. OLTP (VoltDB)

14. RDF (Vertica et. al.)

15. Text (Google, Yahoo, …)

16. Scientific data (MatLab, SciDB)

17. Streaming data (StreamBase Coral8, …)Definition of “Clobbered” A factor of 50 in performanceCurrent DBMSs 30 years of “grow only” bloatware

18. That is not good at anything

19. And that deserves to be sent to the “home for tired software”Pictorially:Other appsDBMS appsOLTPData Warehouse

20. The DBMS Landscape – Performance NeedsOther appshighlowhighhighOLTPData Warehouse

21. One Size Does Not Fit All -- PictoriallyElephants get only “the crevices”SciDB, etc Open sourceVertica et. al.VoltDB, etc.

22. Stonebraker’s PredictionThe DBMS market will move over the next decade or so from OSFA

23. To specialized (market-specific) architectures

24. And open source systems

25. Presumably to the detriment of the elephantsA Couple of Slides of Color on Two of the MarketsData warehousesOLTP

26. Column Store:IBM 60.25 10,000 1/15/2006MSFT 60.53 12,500 1/15/2006Used in: Sybase IQ, Vertica Row Store:IBM 60.25 10,000 1/15/2006MSFT 60.53 12,500 1/15/2006Used in: Oracle, SQL Server, DB2, Netezza,…Data Warehouses – Column Stores are the Answer

27. Data Warehouses – Column Stores Clobber Row StoresRead only what you need

28. “Fat” fact tables are typical

29. Analytics read only a few columns

30. Better compression

31. Execute on compressed data

32. Materialized views help row stores and column stores about equallyExample of “Clobber”Vertica on an 2 processor system costing ~$2K

33. Netezza on a 112 processor system costing ~$1M

34. Customer load time benchmark

35. Vertica 2.8 times faster – per processor/disk

36. Customer query benchmark

37. Vertica 34X on 1/56th the hardware (factor of 1904)Things to Demand From ANY BI DBMSScalable

38. Runs on a grid (MPP), with partitioning

39. Replication for HA/DR

40. “no knobs” operation (more than index selection)

41. Cannot hire enough DBAs

42. On-line update – in parallel with query

43. Ability to run multiple analyses on compatible data

44. Time travel

45. On-the-fly reprovisioningOLTP – The Big PictureWhere the time goes (TPC-C) (Sigmod ’07)24% -- the buffer pool

46. 24% -- locking

47. 24% -- latching

48. 24% -- recovery

49. 4% -- useful workOLTP – The Big PictureHave to focus on overhead!!!!!Better data structures only affects 4%Have to get rid of all four sources to go really fastGet rid of one, and you win 25%H-Store/VoltDB AssumptionsMain memory operation

50. 1 TB is a VERY big OLTP data base

51. No disk stalls

52. No user stalls (disallowed in all apps)

53. Professional terminal operator replaced by “Aunt Martha”

54. A BIG OLTP transaction reads or write 200 records

55. microsecondsH-Store/VoltDB AssumptionsRun transactions to completion

56. Single threaded

57. Eliminate “latch crabbing”

58. And lockingH-Store/VoltDB AssumptionsBuilt-in high availability and disaster recovery

59. Failover to a replica

60. No redo logH-Store/VoltDB AssumptionsScalability requires MPP and partitioning

61. Most transactions are naturally “single-sited”

62. Place my order

63. Read my reservation

64. Update my profileH-Store/VoltDB AssumptionsCan play tricks to make transactions single-sites

65. E.g. replicate read-almost-always data

66. And some companies mandate ‘single-sitedness”H-Store/VoltDB SummaryNo buffer pool overheadThere isn’t oneNo crash recovery overheadDone by failover

67. (optional) Asynchronous data transmission to reporting system

68. (optional) Asynchronous local data archiveNo latching or locking overheadTransactions are run to completion – single threadedOLTP Performance (TPC-C) Elephant

69. 850 TPS (1/2 the land speed record per processor)

70. H-Store

71. 70,416 TPS (82X)

72. VoltDB

73. (33X) My Vision – RestatedOLTP Specialized SystemJelly bean grid33X ElephantHA/DRLimited DBAWarehouse Specialized SystemJelly bean grid50X elephantHA/DRLimited DBAInternalETL

74. The Data Center of the Future Specialized DBMSs

75. Perhaps a half dozen

76. Good at a specific task

77. Running on “jelly bean” MPP

78. Private or public cloud

79. Virtualization (sharing of resources) allows you to run 50% headroom rather than 90%

80. Uniformity lowers DBA costs