2011-11-02 | 02:25 PM - 03:15 PM In 2009 RBS set out to build a single store of trade and risk data that all applications in the bank could access simultaniously. This talk discusses a number of novel techniques that were developed as part of this work. Based on Oracle Coherence the ODC departs from the trend set by most caching solutions by holding its data in a normalised form making it both memory efficient and easy to change. However it does this in a novel way that supports most arbitrary queries without the usual problems associated with distributed joins. We'll be discussing these patterns as well as others that allow linear scalability, fault tolerance and millisecond latencies.

6. The lay of the land:
The main architectural
constructs in the
database industry

8. Shared
Disk

11. ms μs ns ps
1MB Disk/Network 1MB Main Memory
0.000,000,000,000
Cross Continental Main Memory L1 Cache Ref
Round Trip Ref
Cross Network Round L2 Cache Ref
Trip * L1 ref is about 2 clock cycles or 0.7ns. This is the
time it takes light to travel 20cm

22. Taken from “OLTP Through
the Looking Glass, and
What We Found There”
Harizopoulos et al

24. In-Memory TimesTen,
Database HSQL, KD
B
Shared Vertica, Gre
Nothing enplumb
Regular Distributed Exasol,
Database In-Memory VoltDB, Hana
Oracle, Sybase,
MySql
NoSQL Mongo, Cass
andra
Data Grid Coherence,
Teracotta

25. Distributed
Architecture
Simplify the Contract
Stick to RAM

26. 450 processes 2TB of RAM
Oracle
Coherenc
e
Messaging (Topic Based) as a system of record
(persistence)

27. Access Layer Jav Jav
a a
clie clie
nt
API nt
API
Query Layer
Transaction
Data Layer
s
Mtms
Cashflows
Persistence
Layer

28. Indexing
Partitioning Replication

30. But your storage is limited by
the memory on a node

31. Associating data in
different partitions implies
moving it.
Scalable storage, bandwidth
and processing

36. Trade
r Part Version 1
y
Trad
e Trade
r Part Version 2
y
Trad
e Trade
r Part Version 3
y
Trad
e Trade
r Part Version 4
y
Trad
…and you need e
versioning to do MVCC

37. Trad Trade
Part
e r
y
Part Trade
Trad y r
e
Part
y Trade
r
Trad
e Part
y

38. So we want to hold
entities separately
(normalised) to alleviate
concerns around
consistency and space
usage

39. Independently Data is
Versioned Trade
Singleton
r Part
y
Trad
e
Trad Trade Part
e r
y

40. Trade
r Part
y
Trad
e
Trad Trade Part
e r
y

44. This is what using Snowflake Schemas and
the Connected Replication pattern is all
about!

47. Crosscuttin
g
Keys
Common
Keys

48. Replicated
Trader
Party
Trade
Partitioned

55. Facts:
=>Big,
common
keys
Dimensions
=>Small,
crosscutting
Keys

57. Use a Key
Assignment
Trade
Policy
MTMs (e.g. KeyAssociation
s
in Coherence)
Common
Key

58. Replicated
Trader
Party
Trade
Partitioned

59. Query Layer
Trader
Party
Trade
Transactions
Data Layer
Mtms
Cashflows
Fact Storage
(Partitioned)

60. Dimension
s
(repliacte)
Transactions
Facts
Mtms
Cashflows
(distribute/
partition)
Fact Storage
(Partitioned)

61. Facts:
=>Big
=>Distribut
e
Dimensions
=>Small
=> Replicate

62. We use a variant on a
Snowflake Schema to
partition big entities that can
be related via a partitioning
key and replicate small stuff
who’s keys can’t map to our
partitioning key.

63. Replicate
Distribute

64. Select Transaction, MTM,
RefrenceData From MTM,
Transaction, Ref Where Cost Centre
= ‘CC1’

65. Get Get Get Get Get Get Get
Cost Ledger Source Transa MTMs Legs Cost
Center Books Books c-tions Center
s s
Network
Time

66. Get Get Get Get Get Get Get
Cost Ledger Source Transac MTMs Legs Cost
Centers Books Books -tions Centers
Network

68. Select Transaction, MTM, ReferenceData From
MTM, Transaction, Ref Where Cost Centre =
‘CC1’
Join
Dimensions in
Query Layer
Transactions
Mtms
Cashflows
Partitioned

69. Select Transaction, MTM, ReferenceData From
MTM, Transaction, Ref Where Cost Centre =
‘CC1’
Join
Dimensions in
Query Layer
Transactions
Join Facts Mtms
acrossCashflows
cluster
Partitioned

71. Select Transaction, MTM, ReferenceData From
MTM, Transaction, Ref Where Cost Centre =
‘CC1’
Join Join Dimensions
Dimensions in Query Layer
in Query
Layer
Transactions
Join FactsMtms
across Cashflows
cluster
Partitioned

73. Jav
Replicated a
clie
Partitioned
nt
API
Dimensions Facts
We never have to do a distributed join!

74. So all the big stuff is
held partitioned
And we can join
without shipping
keys around and
having intermediate
results

75. Trade
r Part
y
Trad
e
Trad Trade Part
e r
y

76. Trade
r Part Version 1
y
Trad
e Trade
r Part Version 2
y
Trad
e Trade
r Part Version 3
y
Trad
e Trade
r Part Version 4
y
Trad
e

77. Trad Trade
Part
e r
y
Part Trade
Trad y r
e
Part
y Trade
r
Trad
e Part
y

83. Facts
This is a dimension
• It has a different
key to the Facts. Dimensions
• And it’s BIG

91. So we only replicate
‘Connected’ or ‘Used’
dimensions

92. Processing Layer
Dimension
Caches
(Replicated)
Transactions
Data Layer
As new Facts are added Mtms
relevant Dimensions that
they reference are moved
Cashflows
to processing layer
caches
Fact Storage
(Partitioned)

94. Query Layer
Save Trade (With connected
dimension Caches)
Data Layer
Cache
Trad (All Normalised)
Store e
Partitioned
Trigger Cache
Party Sourc Ccy
Alias e
Book

95. Query Layer
(With connected
dimension Caches)
Data Layer
Trad (All Normalised)
e
Party Sourc Ccy
Alias e
Book

96. Query Layer
(With connected
dimension Caches)
Data Layer
Trad (All Normalised)
e
Party Sourc Ccy
Alias e
Book
Party Ledge
rBook

97. ‘Connected Replication’
A simple pattern which
recurses through the foreign
keys in the domain model,
ensuring only ‘Connected’
dimensions are replicated

98. With ‘Connected
Replication’ only
1/10th of the data
needs to be replicated
(on average).

100. Jav
a
Java schema clie
nt
API Java ‘Stored
Procedures’
and
‘Triggers’

102. Query with more
than twenty joins
conditions:
2GB per min /
250Mb/s 3ms latency
(per client)

105. Partitioned
Storage