Title Text                                                                 Processing High Volume                         ...
Agenda...              • C24 iO - Introduction              • C24 iO - Demo             Confidential InformationInformatio...
Typical Usage                                                Choose from hundreds of                                      ...
SWIFT Example (MT541)                                                                                           {1:F01DRES...
Transformation Example             Confidential Information of C24 Technologies Ltd.   © 2012 C24 TechnologiesTuesday, 4 D...
Model Visualisation • Our old friend the MT541 - Receive Against Payment             Confidential Information of C24 Techn...
Graphical Navigation • Some more complex than others      • A Fix 5.0 SP2 Trade Capture Report (One of the          simple...
So why are we here? • Take any one of these technologies...      • MongoDB, Neo4J, Coherence, HazelCast, Redis, GemFire, G...
It’s a Very Simple Solution • To load a CSV into an RDBMS is simple - 2D data anyway      • There are loads of tools to do...
Technical Overview • As I said, it’s very simple with this technology combination...      • Load the messages into a pre-b...
Fix • Take a FIX message...     8=FIXT.1.1 9=1 35=AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING     57...
Fixssage...AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRINGNG 144=STRING 145=STRING 52=20020101-00:00:00....
Using C24-iO • Select or import the model (Fix in this example) • Deploy the code (one click)      • This generates a Fix ...
Seconds later... • Select the output format as JSON and... "StandardHeader":{             "SendingTime":{                "...
Fix into MongoDB • In code (for the techies)...             Confidential Information of C24 Technologies Ltd.   © 2012 C24...
Or if you prefer in Spring... • We literally can take Fix (SWIFT, FpML, CSV etc.) messages from a     queue/file and write ...
Some Important Links... • The C24 iO and MongoDB page      • White Paper, Log4J & Fix examples                            ...
Thank you             Confidential InformationInformation of C24 Technologies                        Confidential of C24 T...
Upcoming SlideShare
Loading in …5
×

Webinar: Processing High Volume Data Feeds with MongoDB - FIX, FpML and Swift in association with C24

3,597 views
3,348 views

Published on

Learn how MongoDB can help ingest high volume financial services messages. See how we can consume FIX, FpML and Swift message formats and handle their ever-changing nature. We will be delivering this session with our partner C24.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,597
On SlideShare
0
From Embeds
0
Number of Embeds
1,592
Actions
Shares
0
Downloads
69
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Webinar: Processing High Volume Data Feeds with MongoDB - FIX, FpML and Swift in association with C24

  1. 1. Title Text Processing High Volume Data Feeds with MongoDB & C24 • John Davies (@jtdavies) | CTO - C24 • Trisha Gee (@trisha_gee) | 10gen • Webinar • 4th December 2012 Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 1
  2. 2. Agenda... • C24 iO - Introduction • C24 iO - Demo Confidential InformationInformation of C24 Technologies Ltd. Confidential of C24 Technologies Ltd. © 2012 C24 Technologies © 2012 C24 TechnologiesTuesday, 4 December 12 2
  3. 3. Typical Usage Choose from hundreds of pre-modelled Standards Create and manage syntactic Import or model any SWIFT-MT, SWIFT-MX, ISO-20022, SEPA, & semantic validation rules FIX, FpML, DTCC, MINOS, CBI Flexible technology, Java-based, Full XPath 2.0 data formats support, Scala, Groovy, Clojure etc. XML Schema, DTD, XML, Relax-NG, RDBMS, Java Classes, Text Files, Tag/ Value, Delimited (CSV), Fixed, Binary, Proprietary Canonical Formats Compare models & manage versions Graphical hierarchical comparison For Developers Bound Java Objects Fully Documented API For Business Analysts Getters / Setters A Powerful GUI for managing, modelling and transformation XPath & XQuery API Thousands of JUnit tests ANT & Maven Build for C.I. Integration with Camel / Fuse, Spring Integration / Batch, Mule, GemFire, GigaSpace, Coherence, Oracle, Export Documents & Reports MongoDB and others Production Fully tested JAR Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies 7Tuesday, 4 December 12 3
  4. 4. SWIFT Example (MT541) {1:F01DRESGB2LAXXX0548034693} {2:I541DRESDEFFXXXXN2}{3: {108:invalid:c3}}{4: :16R:GENL :20C::SEME//FRTJ123REC2 :23G:NEWM :99B::TOSE//100 :16S:GENL :16R:TRADDET :98A::TRAD//20000519 :98A::SETT//20000524 :90A::DEAL//PRCT/101,001283 :35B:ISIN GB0987654321 :16S:TRADDET ... ... ... :95P::PSET//CRSTGB22 :16S:SETPRTY :16R:AMT :19A::SETT//GBP4047151,3 :16S:AMT :16S:SETDET -} Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 4
  5. 5. Transformation Example Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 5
  6. 6. Model Visualisation • Our old friend the MT541 - Receive Against Payment Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 6
  7. 7. Graphical Navigation • Some more complex than others • A Fix 5.0 SP2 Trade Capture Report (One of the simpler messages) Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 7
  8. 8. So why are we here? • Take any one of these technologies... • MongoDB, Neo4J, Coherence, HazelCast, Redis, GemFire, GigaSpaces, Terracotta, MemcacheD etc. • Even a classic RDBMS, in fact “sadly”, usually a RDBMS • A bunch of complex messages... • XML Schema, DTD, XML, Relax-NG, RDBMS, Java Classes, Text Files, Tag/ Value, Delimited (CSV), Fixed, Binary, Proprietary Canonical Formats • SWIFT MT/MX, ISO-20022/SEPA, FIX, FpML, DTCC, MINOS, CBI etc. • And you have an integration problem... • How, for example, do you get your internal canonical messages into MongoDB? • How do you store your front office Fix messages, in any way? Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 8
  9. 9. It’s a Very Simple Solution • To load a CSV into an RDBMS is simple - 2D data anyway • There are loads of tools to do the job, usually from the database vendor • ETL (Extract, Transform, Load) is the classic tool for the RDBMS • Complex (Hierarchical or 3D) data is difficult to map to a RDBMS • Minor changes in the data require awkward table redesign every time • MongoDB is to complex data what an RDBMS is to a CSV • We are no longer restricted by table schema, multi-versions possible • But we need new ETL tools • We could chuck everything in as a String or byte array • But we still need to understand the content if we are to be able to build queries and extract what interests us Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 9
  10. 10. Technical Overview • As I said, it’s very simple with this technology combination... • Load the messages into a pre-built C24 model and convert into JSON • The JSON format is built in • C24 Validate Message queue MQ Series / JMS Save to C24 Parse MongoDB Files Reader Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 10
  11. 11. Fix • Take a FIX message... 8=FIXT.1.1 9=1 35=AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING 57=STRING 143=STRING 144=STRING 145=STRING 52=20020101-00:00:00.000 122=20020101-00:00:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING A reminder that it’s 629=20020101-00:00:00.000 630=1 571=STRING 487=0 568=STRING 150=0 572=STRING 17=STRING 527=STRING 378=0 570=Y 55=STRING 65=CD 48=STRING 22=1 454=1 455=STRING 456=1 460=1 461=STRING 167=FAC 200=200201 541=20020101 224=20020101 225=20020101 not flat, even Fix 239=CORP 226=1 227=1 228=1 255=STRING 543=STRING 470=AF 471=GB 472=STRING 240=20020101 202=1 206=0 231=1 223=1 207=XLON 106=STRING 348=1 349=D 107=STRING 350=1 351=D 32=1 31=1 15=AFA 120=AFA 194=1 195=1 30=XLON 75=20020101 has hierarchy... 60=20020101-00:00:00.000 768=1 769=20020101-00:00:00.000 1033=A 1034=1 1035=E.W 63=0 64=20020101 573=0 574=A1 552=1 54=1 453=1 448=STRING 447=B 452=1 802=1 523=STRING 1=STRING 581=1 81=0 575=Y 576=1 577=0 578=STRING 579=STRING 376=STRING 377=Y 582=1 336=STRING 625=STRING 12=1 13=1 479=AFA 497=Y 157=1 230=20020101 158=1 159=1 238=1 237=1 118=1 119=1 155=1 156=M 77=O 58=STRING 354=1 355=D 518=1 519=1 520=1 521=AFA 136=1 137=1 138=AFA 139=1 37=STRING 198=STRING 11=STRING 38=1 468=0 469=1 528=A 529=1 483=20020101-00:00:00.000 381=1 10=000 • Relatively simple as they go but to get this into a database in a useful format is not trivial Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 11
  12. 12. Fixssage...AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRINGNG 144=STRING 145=STRING 52=20020101-00:00:00.000:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING:00.000 630=1 571=STRING 487=0 568=STRING 150=0 572=STRINGNG 378=0 570=Y 55=STRING 65=CD 48=STRING 22=1 454=1 455=STRINGRING 167=FAC 200=200201 541=20020101 224=20020101 225=20020101=1 228=1 255=STRING 543=STRING 470=AF 471=GB 472=STRING 206=0 231=1 223=1 207=XLON 106=STRING 348=1 349=D 107=STRING1=1 15=AFA 120=AFA 194=1 195=1 30=XLON 75=2002010100.000 768=1 769=20020101-00:00:00.000 1033=A 1034=1 1035=E.W 63=0574=A1 552=1 54=1 453=1 448=STRING 447=B 452=1 802=1 523=STRING0 575=Y 576=1 577=0 578=STRING 579=STRING 376=STRING 377=Y 582=1ING 12=1 13=1 479=AFA 497=Y 157=1 230=20020101 158=1 159=1 238=1155=1 156=M 77=O 58=STRING 354=1 355=D 518=1 519=1 520=1 521=AFAA 139=1 37=STRING 198=STRING 11=STRING 38=1 468=0 469=1 528=A-00:00:00.000 381=1 10=000ple as they go but to get this into a database in a s not trivial Confidential Information of C24 Technologies Ltd. © 2012 C24 Technologies Tuesday, 4 December 12 12
  13. 13. Using C24-iO • Select or import the model (Fix in this example) • Deploy the code (one click) • This generates a Fix endpoint jar that is capable of reading, validating and writing out Fix messages - in multiple formats • Plus of course a rich API, XPath 2.0 and Spring Integration • XPath, like QL offers a single language to filter/query/sort any data structure • From a small snippet of code we can read in a Fix message (into our Jar) and write it directly into MongoDB • It’s even simpler with Spring Integration & Spring Data - no code (just XML) • With Spring Integration / Data we can load Fix messages into MongoDB without any code at all Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 13
  14. 14. Seconds later... • Select the output format as JSON and... "StandardHeader":{             "SendingTime":{                "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002"             },             "OrigSendingTime":{                "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002"             },             "XmlData":{                "XmlDataLen":1,                "XmlData":"D"             },             "MessageEncoding":"ISO-2022-JP",             "LastMsgSeqNumProcessed":1,             "HopGrp":{                "NoHops":1,                "HopGrpFields":{                   "HopCompID":"STRING",                   "HopSendingTime":{                      "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002" Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 14
  15. 15. Fix into MongoDB • In code (for the techies)... Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 15
  16. 16. Or if you prefer in Spring... • We literally can take Fix (SWIFT, FpML, CSV etc.) messages from a queue/file and write them to MongoDB with NO CODE! Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 16
  17. 17. Some Important Links... • The C24 iO and MongoDB page • White Paper, Log4J & Fix examples http://www.c24.biz/io-mongodb • C24-iO • http://www.c24.biz • John Davies (@jtdavies) | CTO - C24 • 10gen • http://www.10gen.com • Trisha Gee (@trisha_gee) | 10gen Confidential Information of C24 Technologies Ltd. © 2012 C24 TechnologiesTuesday, 4 December 12 17
  18. 18. Thank you Confidential InformationInformation of C24 Technologies Confidential of C24 Technologies Ltd. © 2012 C24 Technologies © 2012 C24 TechnologiesTuesday, 4 December 12 18

×