Balancing Replication and Partitioning in a Distributed Java Database

In-memory databases offer significant gains in But three issues performance as all data is have stunted their freely available. There is no uptake: AddressTraditional disk- spaces only being need to page to and from disk.oriented database large enough for aarchitecture is subset of a typicalshowing its age. This makes joins a user’s data. The ‘one problem. When data more bit’ problem must be joined across and durability. multiple machines performance degradation Snowflake is inevitable. Distributed in- Schemas allow memory databases us to mix But this model only goes solve these three Partitioning so far. “Connected problems but at the and Replication Replication” takes us a price of loosing the so joins never step further allowing us single address space. hit the wire. to make the best possible use of replication.

The lay of the land: The main architecturalconstructs in the database industry

Shared Disk

ms μs ns ps1MB Disk/Network 1MB Main Memory 0.000,000,000,000Cross Continental Main Memory L1 Cache RefRound Trip Ref Cross Network L2 Cache Ref Round Trip * L1 ref is about 2 clock cycles or 0.7ns. This is the time it takes light to travel 20cm

Distributed Cache

Taken from “OLTP Throughthe Looking Glass, and WhatWe Found There”Harizopoulos et al

Shared Nothing Teradata, Vertica, Greenplumb… SN Regular In-Memory In-MemoryDatabase Database Drop Disk Exasol, VoltDB,Oracle, Sybase, MySql Times Ten, HSQL, KDB Distribute Hana ODC Distributed Caching Coherence, Gemﬁre, Gigaspaces

Distributed Architecture Simplify the Contract. Stick to RAM

450 processes 2TB of RAM Oracle CoherenceMessaging (Topic Based) as a system of record (persistence)

Access Layer Java Java client client API API Query Layer Transactions Data Layer Mtms CashflowsPersistence Layer

IndexingPartitioning Replication

But your storage is limited bythe memory on a node

Keys Fs-Fz Keys Xa-YdScalable storage, bandwidthand processing

Trader Party Version 1 Trade Trader Party Version 2 Trade Trader Party Version 3 Trade Trader Party Version 4 Trade…and you needversioning to do MVCC

Trade Trader Party Party TraderTrade Party TraderTrade Party

So better to usepartitioning, spreading data around the cluster.

Trader Party TradeTrade Trader Party

!This is what using Snowﬂake Schemas and the Connected Replication pattern is all about!

Crosscutting Keys Common Keys

ReplicatedTrader Party Trade Partitioned

Valuation Legs Valuationsart Transaction Mapping Cashflow Mapping Facts: Party Alias Transaction =>Big, Cashflows common Legs Parties keys Ledger Book Source Book Dimensions Cost Centre Product =>Small, Risk Organisation Unit Business Unit crosscutting HCS Entity Keys Set of Books 0 37,500,000 75,000,000 112,500,000 150,000,000

Coherence’s KeyAssociation gives us thisTrades MTMs Common Key

ReplicatedTrader Party Trade Partitioned (

Query LayerTrader Party Trade Transactions Data Layer Mtms Cashflows Fact Storage (Partitioned)

Dimensions (repliacte) Transactions Mtms Facts Cashflows (distribute/ partition)Fact Storage(Partitioned)

Valuation Legs Valuations Facts:art Transaction Mapping Cashflow Mapping Party Alias =>Big =>Distribute Transaction Cashflows Legs Parties Ledger Book Source Book Dimensions =>Small Cost Centre Product Risk Organisation Unit => Replicate Business Unit HCS Entity Set of Books 0 37,500,000 75,000,000 112,500,000 150,000,000

We use a variant on a Snowﬂake Schema to partition big stuff, that hasthe same key and replicate small stuff that has crosscutting keys.

ReplicateDistribute

Select Transaction, MTM, ReferenceData FromMTM, Transaction, Ref Where Cost Centre = ‘CC1’

Select Transaction, MTM, ReferenceData FromMTM, Transaction, Ref Where Cost Centre = ‘CC1’ LBs[]=getLedgerBooksFor(CC1) SBs[]=getSourceBooksFor(LBs[]) So we have all the bottom level dimensions needed to query facts Transactions Mtms Cashflows Partitioned

Select Transaction, MTM, ReferenceData FromMTM, Transaction, Ref Where Cost Centre = ‘CC1’ LBs[]=getLedgerBooksFor(CC1) SBs[]=getSourceBooksFor(LBs[]) So we have all the bottom level dimensions needed to query facts Transactions Get all Transactions and Mtms MTMs (cluster side join) for the passed Source Books Cashflows Partitioned

Select Transaction, MTM, ReferenceData From MTM, Transaction, Ref Where Cost Centre = ‘CC1’Populate raw facts LBs[]=getLedgerBooksFor(CC1)(Transactions) with SBs[]=getSourceBooksFor(LBs[])dimension data So we have all the bottom levelbefore returning to dimensions needed to query factsclient. Transactions Get all Transactions and Mtms MTMs (cluster side join) for the passed Source Books Cashflows Partitioned

Replicated Partitioned Java clientDimensions Facts APIWe never have to do a distributed join!

So all the big stuff is held paritioned And we can joinwithout shipping keys around and having intermediate results

Trader Party TradeTrade Trader Party

Trader Party Version 1 Trade Trader Party Version 2 Trade Trader Party Version 3 Trade Trader Party Version 4 Trade

Trade Trader Party Party TraderTrade Party TraderTrade Party

Valuation Legs Valuationsrt Transaction Mapping Cashflow Mapping Party Alias Facts Transaction Cashflows Legs Parties This is a dimension Ledger Book •  It has a different Source Book Cost Centre key to the Facts. Dimensions Product •  And it’s BIG Risk Organisation Unit Business Unit HCS Entity Set of Books 0 125,000,000

Party Alias Parties Ledger Book Source Book Cost Centre ProductRisk Organisation Unit Business Unit HCS Entity Set of Books 0 1,250,000 2,500,000 3,750,000 5,000,000

Party Alias Parties Ledger Book Source Book Cost Centre ProductRisk Organisation Unit Business Unit HCS Entity Set of Books 20 1,250,015 2,500,010 3,750,005 5,000,000

So we only replicate‘Connected’ or ‘Used’ dimensions

Processing Layer Dimension Caches (Replicated) Transactions Data LayerAs new Facts are added Mtmsrelevant Dimensions thatthey reference are moved Cashflowsto processing layer caches Fact Storage (Partitioned)

Query Layer Save Trade (With connected dimension Caches) Data LayerCache Trade (All Normalised)Store Partitioned Trigger Source Cache Party Ccy Alias Book

Query Layer (With connected dimension Caches) Data Layer Trade (All Normalised)Party Source CcyAlias Book

Query Layer (With connected dimension Caches) Data Layer Trade (All Normalised)Party Source CcyAlias Book Party Ledger Book

‘Connected Replication’ A simple pattern whichrecurses through the foreign keys in the domain model, ensuring only ‘Connected’ dimensions are replicated

Java clientJava schema API Java ‘Stored Procedures’ and ‘Triggers’

Partitioned Storage

http://www.benstopford.com	2718
http://lonrs05720	8
http://feeds.feedburner.com	3
http://a0.twimg.com	2
http://paper.li	1
http://webcache.googleusercontent.com	1
http://translate.googleusercontent.com	1

Balancing Replication and Partitioning in a Distributed Java Database

by Ben Stopford on Oct 02, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 2,734

Statistics

Balancing Replication and Partitioning in a Distributed Java Database — Presentation Transcript