Learn how MongoDB can help ingest high volume financial services messages. See how we can consume FIX, FpML and Swift message formats and handle their ever-changing nature. We will be delivering this session with our partner C24.

1. Title Text
Processing High Volume
Data Feeds with
MongoDB & C24
• John Davies (@jtdavies) | CTO - C24
• Trisha Gee (@trisha_gee) | 10gen
• Webinar
• 4th December 2012
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 1

2. Agenda...
• C24 iO - Introduction
• C24 iO - Demo
Confidential InformationInformation of C24 Technologies Ltd.
Confidential of C24 Technologies Ltd. © 2012 C24 Technologies © 2012 C24 Technologies
Tuesday, 4 December 12 2

3. Typical Usage
Choose from hundreds of
pre-modelled Standards Create and manage syntactic
Import or model any
SWIFT-MT, SWIFT-MX, ISO-20022, SEPA, & semantic validation rules
FIX, FpML, DTCC, MINOS, CBI
Flexible technology, Java-based, Full XPath 2.0
data formats support, Scala, Groovy, Clojure etc.
XML Schema, DTD, XML, Relax-NG,
RDBMS, Java Classes, Text Files, Tag/
Value, Delimited (CSV), Fixed, Binary,
Proprietary Canonical Formats Compare models &
manage versions
Graphical hierarchical
comparison
For Developers
Bound Java Objects
Fully Documented API For Business Analysts
Getters / Setters A Powerful GUI for managing,
modelling and transformation
XPath & XQuery API
Thousands of JUnit tests
ANT & Maven Build for C.I.
Integration with Camel / Fuse, Spring
Integration / Batch, Mule, GemFire,
GigaSpace, Coherence, Oracle, Export Documents & Reports
MongoDB and others
Production
Fully tested JAR
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
7
Tuesday, 4 December 12 3

4. SWIFT Example (MT541)
{1:F01DRESGB2LAXXX0548034693}
{2:I541DRESDEFFXXXXN2}{3:
{108:invalid:c3}}{4:
:16R:GENL
:20C::SEME//FRTJ123REC2
:23G:NEWM
:99B::TOSE//100
:16S:GENL
:16R:TRADDET
:98A::TRAD//20000519
:98A::SETT//20000524
:90A::DEAL//PRCT/101,001283
:35B:ISIN GB0987654321
:16S:TRADDET
...
...
...
:95P::PSET//CRSTGB22
:16S:SETPRTY
:16R:AMT
:19A::SETT//GBP4047151,3
:16S:AMT
:16S:SETDET
-}
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 4

5. Transformation Example
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 5

6. Model Visualisation
• Our old friend the MT541 - Receive Against Payment
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 6

7. Graphical Navigation
• Some more complex than others
• A Fix 5.0 SP2 Trade Capture Report (One of the
simpler messages)
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 7

8. So why are we here?
• Take any one of these technologies...
• MongoDB, Neo4J, Coherence, HazelCast, Redis, GemFire, GigaSpaces,
Terracotta, MemcacheD etc.
• Even a classic RDBMS, in fact “sadly”, usually a RDBMS
• A bunch of complex messages...
• XML Schema, DTD, XML, Relax-NG, RDBMS, Java Classes, Text Files, Tag/
Value, Delimited (CSV), Fixed, Binary, Proprietary Canonical Formats
• SWIFT MT/MX, ISO-20022/SEPA, FIX, FpML, DTCC, MINOS, CBI etc.
• And you have an integration problem...
• How, for example, do you get your internal canonical messages into
MongoDB?
• How do you store your front office Fix messages, in any way?
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 8

9. It’s a Very Simple Solution
• To load a CSV into an RDBMS is simple - 2D data anyway
• There are loads of tools to do the job, usually from the database vendor
• ETL (Extract, Transform, Load) is the classic tool for the RDBMS
• Complex (Hierarchical or 3D) data is difficult to map to a RDBMS
• Minor changes in the data require awkward table redesign every time
• MongoDB is to complex data what an RDBMS is to a CSV
• We are no longer restricted by table schema, multi-versions possible
• But we need new ETL tools
• We could chuck everything in as a String or byte array
• But we still need to understand the content if we are to be able to build
queries and extract what interests us
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 9

10. Technical Overview
• As I said, it’s very simple with this technology combination...
• Load the messages into a pre-built C24 model and convert into JSON
• The JSON format is built in
•
C24 Validate
Message queue
MQ Series / JMS
Save to
C24 Parse MongoDB
Files Reader
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 10

11. Fix
• Take a FIX message...
8=FIXT.1.1 9=1 35=AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING
57=STRING 143=STRING 144=STRING 145=STRING 52=20020101-00:00:00.000
122=20020101-00:00:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING
A reminder that it’s
629=20020101-00:00:00.000 630=1 571=STRING 487=0 568=STRING 150=0 572=STRING
17=STRING 527=STRING 378=0 570=Y 55=STRING 65=CD 48=STRING 22=1 454=1 455=STRING
456=1 460=1 461=STRING 167=FAC 200=200201 541=20020101 224=20020101 225=20020101
not flat, even Fix
239=CORP 226=1 227=1 228=1 255=STRING 543=STRING 470=AF 471=GB 472=STRING
240=20020101 202=1 206=0 231=1 223=1 207=XLON 106=STRING 348=1 349=D 107=STRING
350=1 351=D 32=1 31=1 15=AFA 120=AFA 194=1 195=1 30=XLON 75=20020101
has hierarchy...
60=20020101-00:00:00.000 768=1 769=20020101-00:00:00.000 1033=A 1034=1 1035=E.W 63=0
64=20020101 573=0 574=A1 552=1 54=1 453=1 448=STRING 447=B 452=1 802=1 523=STRING
1=STRING 581=1 81=0 575=Y 576=1 577=0 578=STRING 579=STRING 376=STRING 377=Y 582=1
336=STRING 625=STRING 12=1 13=1 479=AFA 497=Y 157=1 230=20020101 158=1 159=1 238=1
237=1 118=1 119=1 155=1 156=M 77=O 58=STRING 354=1 355=D 518=1 519=1 520=1 521=AFA
136=1 137=1 138=AFA 139=1 37=STRING 198=STRING 11=STRING 38=1 468=0 469=1 528=A
529=1 483=20020101-00:00:00.000 381=1 10=000
• Relatively simple as they go but to get this into a database in a
useful format is not trivial
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 11

12. Fix
ssage...
AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING
NG 144=STRING 145=STRING 52=20020101-00:00:00.000
:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING
:00.000 630=1 571=STRING 487=0 568=STRING 150=0 572=STRING
NG 378=0 570=Y 55=STRING 65=CD 48=STRING 22=1 454=1 455=STRING
RING 167=FAC 200=200201 541=20020101 224=20020101 225=20020101
=1 228=1 255=STRING 543=STRING 470=AF 471=GB 472=STRING
206=0 231=1 223=1 207=XLON 106=STRING 348=1 349=D 107=STRING
1=1 15=AFA 120=AFA 194=1 195=1 30=XLON 75=20020101
00.000 768=1 769=20020101-00:00:00.000 1033=A 1034=1 1035=E.W 63=0
574=A1 552=1 54=1 453=1 448=STRING 447=B 452=1 802=1 523=STRING
0 575=Y 576=1 577=0 578=STRING 579=STRING 376=STRING 377=Y 582=1
ING 12=1 13=1 479=AFA 497=Y 157=1 230=20020101 158=1 159=1 238=1
155=1 156=M 77=O 58=STRING 354=1 355=D 518=1 519=1 520=1 521=AFA
A 139=1 37=STRING 198=STRING 11=STRING 38=1 468=0 469=1 528=A
-00:00:00.000 381=1 10=000
ple as they go but to get this into a database in a
s not trivial
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 12

13. Using C24-iO
• Select or import the model (Fix in this example)
• Deploy the code (one click)
• This generates a Fix endpoint jar that is capable of reading, validating and
writing out Fix messages - in multiple formats
• Plus of course a rich API, XPath 2.0 and Spring Integration
• XPath, like QL offers a single language to filter/query/sort any data structure
• From a small snippet of code we can read in a Fix message (into
our Jar) and write it directly into MongoDB
• It’s even simpler with Spring Integration & Spring Data - no code (just XML)
• With Spring Integration / Data we can load Fix messages into
MongoDB without any code at all
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 13

14. Seconds later...
• Select the output format as JSON and...
"StandardHeader":{
"SendingTime":{
"withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002"
},
"OrigSendingTime":{
"withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002"
},
"XmlData":{
"XmlDataLen":1,
"XmlData":"D"
},
"MessageEncoding":"ISO-2022-JP",
"LastMsgSeqNumProcessed":1,
"HopGrp":{
"NoHops":1,
"HopGrpFields":{
"HopCompID":"STRING",
"HopSendingTime":{
"withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002"
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 14

15. Fix into MongoDB
• In code (for the techies)...
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 15

16. Or if you prefer in Spring...
• We literally can take Fix (SWIFT, FpML, CSV etc.) messages from a
queue/file and write them to MongoDB with NO CODE!
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 16

17. Some Important Links...
• The C24 iO and MongoDB page
• White Paper, Log4J & Fix examples
http://www.c24.biz/io-mongodb
• C24-iO
• http://www.c24.biz
• John Davies (@jtdavies) | CTO - C24
• 10gen
• http://www.10gen.com
• Trisha Gee (@trisha_gee) | 10gen
Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies
Tuesday, 4 December 12 17

18. Thank you
Confidential InformationInformation of C24 Technologies
Confidential of C24 Technologies Ltd. © 2012 C24 Technologies © 2012 C24 Technologies
Tuesday, 4 December 12 18