Webinar: Managing Real Time Risk Analytics with MongoDB

1,976 views
1,716 views

Published on

Learn how customers are using MongoDB to aggregate risk data from multiple front office sources to calculate exposure.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,976
On SlideShare
0
From Embeds
0
Number of Embeds
280
Actions
Shares
0
Downloads
32
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Webinar: Managing Real Time Risk Analytics with MongoDB

  1. 1. Webinar : Managing Real Time Risk Analytics with MongoDBwill begin at 14:00 GMT / 7:00 AM PDT / 2:00 PM UTCAudio should start immediately when you log into the event viaAudio Broadcast. You will need a VOID headset and reliableinternet connection for Audio Broadcast. If you are having issuesconnecting, please dial 1-877-668-4493; Access code: 666 722454.There is a Q&A following the webinar. You can enter questions inthe chat box to the Host and Presenter.A recording of the webinar will be available 24 hours after theeventi s complete.For any other issues please email webinars@10gen.com.
  2. 2. Easy to Start, Easy to Develop, Easy to Scale Managing Real Time Risk Analytics with MongoDB 10gen, Inc. November 2012
  3. 3. @dmrobertsdaniel.roberts@10gen.com Solution Architect Based in London http://www.10gen.com/
  4. 4. Last Time•Document-Oriented •High Volume Data Feeds •Dynamic schema •Tick Data capture •Agile •Risk Analytics •Flexible •Product Catalogs & Trade•High Performance Capture•Highly Available •P&L Reporting•Horizontal Scale Out •Portfolio Management •Reference Data Management •Quantitative Analysis •Automated Trading
  5. 5. Key Features for reporting / analytics•Dynamic Query Language•Aggregation Framework•Dynamic & Flexible Schemas•Atomic Updates to documents •Upserts•Horizonal Scale Out•Map Reduce•Hadoop Integration
  6. 6. Sharded Architecture
  7. 7. Risk Analytics & ReportingUse Case:•Collect and aggregate risk data•Calculate risk / exposures•Potentially real timeWhy MongoDB?•Collect data from a single or multiple sources •Different formats•Documents used to create ‘pre-aggregated’ reports •Real Time•Aggregation Framework for reporting •e.g. exposure for a counter party•Internal MR or Hadoop connector •Batch process risk data
  8. 8. Portfolio / Position reportingUse Case:•Store positions or portfolio information•Query to find current positions/portfolios•Query by client or traderWhy MongoDB?•Customer/client my have many different products•Aggregation Framework to calculate values and views•Work on extremely large data sets •Current and historic data
  9. 9. Reporting / Analytics requirements•How quickly do you need answers?•How often do you need updates?•Requirements will drive which methods to utilise. •Generally the high the latency tolerance the greater the insight.•Choices •Batch calculations - large complex data volumes •Pre-Aggregated - specific and very fast •Real-time calculations - As needed reports and calculations
  10. 10. Batch Processing•MongoDB internal Map Reduce•Hadoop Map Reduce with MongoDB connector raw•Insight after batch run hourly •For instance every hour or day •Output to documents/collection daily •Fast read once data produced monthly•Results not up to last millisecond•Can generate insight from huge datasets•Rolled up stats •Source collections -> reporting collection
  11. 11. Sharded MongoDB + HadoopShard 1 Shard 2 Shard 3 Shard 4 Shard 5 c z t f v w ya s u g e h d b x Hadoop Hadoop Hadoop Hadoop Hadoop Node Node Node Node Node Hadoop Hadoop Hadoop Hadoop Node Node Node Node
  12. 12. Use Query Language•Query across documents using MongoDB JSON query language •Infer results in the application code. •Dynamic - but what happens when we have 1 billion documents.•Indexing strategy key•var data = db.pl.find({ positionId: 1234 })[0] { "_id" : ObjectId("50990a10fd421cb025407cb1"), "positionId" : 1234, "security" : "ORCL", "quantity" : 1000, "price" : 30.23, "currency" : "USD" } data.price * data.quantity = 30230.00
  13. 13. Leverage schema design•Group useful data together into documents•Utilise upsert and $inc functionality of MongoDB •Pre-aggregate reports •$inc incrementing counters is light weight.•Fast pre calculated data•Low latency retrieval•http://docs.mongodb.org/manual/use-cases/pre-aggregated-reports/
  14. 14. Pre-Aggregated Reports -Daily trade volumes// daily buckets, each hour a sub-document{ _id: "2012-11-06-1231", security:‘ORCL’ ts: ISODate("2012-11-06T00:00:00.000Z") daily: 67345234, minute: { 0: { 0: 2034, 1: 735, ... 59: 2644 }, ... 7: { 0: 15434, 1: 334, ... 59: 64234 } }}// Increment counters in document. Automaticallyinsert new document (upsert : true)> db.trades.update( { _id: "2012-11-06-1231", security:‘ORCL’ }, { $inc: { "minute.4.09": 1034, “daily” : 1034 } }, true)
  15. 15. Aggregation Framework•Much simpler and faster than MongoDB map reduce•Replaces common MR use cases in MongoDB•Native operators in the MongoDB coredb.pl.aggregate([! {$match:{"clientId" : 4321}}, { $project : { value : { $multiply:["$quantity", "$price"] } } }]);
  16. 16. Aggregation FIX Execution reportCalculating average price for a fulfilled order:db.ExecutionReport.aggregate({ $match : {"Instrument.Symbol":"MSFT"}, $group : { _id : "$ClOrdID", reportsPerOrd : { $sum : 1 }, totalNumOrdered : { $sum : "$OrderQtyData.OrderQty"}, avgPrice : { $avg : "$AvgPx" } }});
  17. 17. Summary•Number of choices for Aggregating data. •MongoDB Map Reduce •Pre-Compute - Schema Design •Hadoop Connector •Aggregation Framework
  18. 18. download at mongodb.org @dmroberts daniel.roberts@10gen.comFree online training - http://education.10gen.com/www.meetup.com/London-MongoDB-User-Group/ Facebook Twitter LinkedInhttp://bit.ly/mongodb @dmroberts http://linkd.in/joinmongo

×