• Save
SAP World Tour 2010: Impact of Column-Oriented Main-Memory Databases on Enterprise Applications
Upcoming SlideShare
Loading in...5
×
 

SAP World Tour 2010: Impact of Column-Oriented Main-Memory Databases on Enterprise Applications

on

  • 2,387 views

 

Statistics

Views

Total Views
2,387
Views on SlideShare
2,310
Embed Views
77

Actions

Likes
5
Downloads
0
Comments
0

3 Embeds 77

http://www.slideshare.net 56
http://epic.hpi.uni-potsdam.de 18
https://epic.hpi.uni-potsdam.de 3

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • ccdcdMoore’s Law: “…number of transistors … doubling approximately every two years”CPU frequency hit limitin 2002, but Mooreslaw holds todayHow? Multi-Core and Parallelization
  • Select required attributes only
  • X: number of aggregatesY: log. time required for aggregate calculation
  • ordered/few: tarif ratesUnordered/few: sexOrdered/Distinct: temperature values
  • Partitioning!
  • Remove data redundancy
  • Partioning!
  • Insert-Only
  • Stress on:Materialized aggregatesMaterialized viewsIndicesRedudant data in cubes, change history, …
  • Analysis of accounting tablesBkpf= accounting document headersBseg = accounting document line items
  • VieleSELECTs: bspw. Dunning ablauf (mituntersehr complex) row-oriented, relational programming pattern select via attributes (column-wise)  cp. to OLAP needs a rewrite!!!
  • What about rescheduling for high-prio customer now: manual rescheduling necessary dank main-memory jedes mal neuberechnenmöglich rescheduling on-demand ATP combining with pricing, e.g. customer demands for a certain price per product you can name shipping date (cupper, metals, oil, etc.)
  • Aggregates narrow your flexibility interactive planning

SAP World Tour 2010: Impact of Column-Oriented Main-Memory Databases on Enterprise Applications SAP World Tour 2010: Impact of Column-Oriented Main-Memory Databases on Enterprise Applications Presentation Transcript

  • Impact of Column-OrientedMain-Memory Databases on Enterprise Applications
    Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld
    Hasso Plattner Institute
    March 02, 2010
  • © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 2
    Disclaimer
    This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.
  • Agenda
    The Hasso Plattner Institute
    Technical Foundation of Columnar In-Memory Databases
    Impact on Enterprise Applications
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 3
  • Agenda
    The Hasso Plattner Institute
    Technical Foundation of Columnar In-Memory Databases
    Impact on Enterprise Applications
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 4
  • Key Facts about the Hasso Plattner Institute
    Founded as a public private partnershipin 1998 in Potsdam near Berlin, Germany
    Institute belongs to theUniversity of Potsdam
    Ranked 1st in “CHE”
    340 B.Sc. and M.Sc. students
    10 professors, 91 PhD students
    Course of study: IT Systems Engineering
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 5
  • Prof. Dr. h.c. Hasso Plattner / Dr. Alexander Zeier
    Research focuses on the technical aspects of enterprise software anddesign of complex applications
    Memory-Based Data Management for Enterprise Applications
    Human-Centered Software Design and Engineering
    Maintenance and Evolution of Service-Oriented Enterprise Software
    Integration of RFID Technology in Enterprise Platforms
    Architecture-based Performance Simulation
    Research co-operations with
    Stanford, MIT, etc.
    Industry co-operations with
    SAP, Siemens, Audi, etc.
    Research GroupEnterprise Platform & Integration Concepts
    Partner of Stanford Center for Design Research
    Partner of MIT in Supply Chain Innovation
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 6
  • Agenda
    The Hasso Plattner Institute
    Technical Foundation of Columnar In-Memory Databases
    Impact on Enterprise Applications
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 7
  • Two separate worlds: OLTP and OLAP?
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 8
  • Two separate worlds: OLTP and OLAP?
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 9
  • Dominant Hardware Trends
    Multi-Core Technology
    Moore’s Law: “…number of transistors … doubling approximately
    CPU frequency hit limitin 2002, but Moore’s law holds today
    In-Memory Technology
    Increased size: up to 2TB of main-memory on one main board in 2010
    Constantly dropping costs
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 10
  • 3 Aspects for a Hybrid Solution
    Columnar Storage
    New database layout accessing only needed portions of data
    Improve access for subsets of attributes
    In-Memory
    Fastest possible data access
    Spatial proximity
    Compression
    Reduce amount of data to fit in main memory
    Use cache and bus capacities more efficient
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 11
  • Row Store
    Column Store
    Storages: Row vs. Column
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 12
  • Columnar Storage: Architecture
    Claim: Columnar storage is suited for update-intensive applications
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 13
  • In-Memory: Aggregate Processing Time
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 14
    The value of an attribute changes by calculation
  • Compression: Types
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 15
  • Dictionaries
    Compression:Advantages ofColumnar Storages
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 16
  • Scalability: Multiple CPU Cores
    Set processing is most frequent access type in EAs(scan is dominant pattern)
    Sequential column-wise scans show best bandwidth utilization between CPU cores and main memory
    Independence of tuples per column allows:
    easy partitioning, and
    parallel processing (see Hennessy [1])
    Faster memory scans by improved memory bandwidth in next generation CPUs
    Neither materialized views nor aggregateseverything is calculated on-the-fly
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 17
    [1] John L. Hennessy, David A. Patterson: Computer Architecture: A Quantitative Approach
  • Myth 1: Adapting existing databases leverages column-oriented perfomance improvement
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 18
    Column-Oriented
    Traditional
    • Neitherapplicationnordatabasecachesarenecessary
    • Redundant dataobjectsareeliminiated
    • Neitherindicesnoraggregatesneed to bemaintained
    • Number of layersisminimized
    • No updates
    • Applicationlogicisadjacent to rawdata
    • No databaselocksrequired
    • Data movementsareminimzed
    • Sustainuse of existingresources
    Application
    Cache
    DatabaseCache
    Pre-BuiltAggregates
    Raw Data
    + Stored Procedures
    + Mathematical Algorithms
  • Myth 2: The entire set of business data does not fit into main memory
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 19
    SRM
    SCM
    etc.
    CRM
    FI
    Use cumulated memory capacity of various blades
    • Only relevant data in memory
    • Partitioning across hardware
    • Redundant-free data
    • Only few columns have high many different attribute values
    • Up to ten times higher compression possible
  • Myth 3: Update/Insert of Huge Amounts of Data Degrades Columnar Performance
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 20
    Columnar Storing
    Traditional Storing
    Updates
    Insert
    • Ourresearchactivities at the HPI in Potsdam showed:
    • Updates areperformed rare
    • Onlyveryfewcolumnsareaffectedbyupdates
    Furtherinsightsavailable at SAP World Tour 2010 HPI booth 1.19.
    Insert Only
  • Agenda
    The Hasso Plattner Institute
    Technical Foundation of Columnar In-Memory Databases
    Impact on Enterprise Applications
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 21
  • Architecture of ExistingFinancials Systems
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 22
  • Architecture of Simplified Financials Systems
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 23
    Only base tables, algorithms, and some indices
  • Analyzing Real Customer Data
    1M records in BSEG ~ 1GB disk storage
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 24
  • Results:Distinct Values per Attribute
    Results on analyzing Financials
    Distinct values in accounting document headers (99 attributes)
    CPG
    Logistics
    Banking
    High Tech
    Discrete Manufacturing
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 25
  • Results:Accounting Document Updates
    Percentage of rows updated
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 26
  • Dunning
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 27
  • Available to Promise
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 28
  • Demand Planning
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 29
  • Insert Only
    Tuple visibility indicated by timestamps (POSTGRES-style time-travel [2])
    Additional storage requirements can be neglected due to low update frequency
    Timestamp columns are not compressed to avoid additional merge costs
    Snapshot isolation
    Application-level locks
    Insert Only
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 30
  • Memory Consumption
    Experiments show a general factor 10 in compression (using dictionary compression and bit vector encoding)
    Additional storage savings by removing materialized aggregates, save ~2×
    Keep only the active partition of the data in memory (based on fiscal year), save ~5×
    Next generation blade servers will allow up to 512 GB RAM.
    Arrays of 100 blades already available
    50 TB main memory would allow to cover the majority of SAP Business Suite customers
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 31
  • Impact on Application Development
    Formalized logic must be moved close to the engine
    Calculations must take place close to the data
    Reduction of application code
    OLTP queries must use minimal projections (SELECT * is not allowed)
    No caching necessary anymore
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 32
  • Conclusion
    Technology improvements allow re-thinking of how we build enterprise apps:
    A combined OLTP and OLAP system can share the same in-memory column store data base
    Our experiments with real applications and data prove it
    Open research challenges:
    Disaster recovery, extension for unstructured data, life cycle based data management
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 33
  • Further Information
    è
    SAP Public Web:
    EPIC@HPI: https://epic.hpi.uni-potsdam.de
    Hasso Plattner Institute: http://www.hpi-web.de
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 34
  • Thank you! Contact us!
    Hasso Plattner Institute
    EA²L / Enterprise Platform & Integration Concepts
    Matthieu-P. Schapranow
    August-Bebel-Str. 88
    D-14482 Potsdam, Germany
    Matthieu-P. Schapranow
    matthieu.schapranow@hpi.uni-potsdam.de
    Responsible: Deputy Prof. of Prof. Hasso PlattnerDr. Alexander Zeierzeier@hpi.uni-potsdam.de
    © SAP 2008 / SAP TechEd 08 / <Session ID> Page 35
  • Feedback
    Please complete your session evaluation.
    Be courteous — deposit your trash, and do not take the handouts for the following session.
    Thank You !
    © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 36