Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

MongoDB in FS


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Easy to Start, Easy to Develop, Easy to Scale MongoDB In FS 10gen, Inc. October 2012
  • 2. Solution(Architect Based(in(London sdf
  • 3. 10gen$is$the$company$behindMongoDB.• Foundedin2007 • DwightMerriman,EliotHorowitz Set$the Foster direction$& community$&• $73M+infunding contribute ecosystem code$to • Flybridge,Sequoia,UnionSquare, MongoDB NEA• WorldwideExpanding Team • 170+employees Provide Provide • Locations: MongoDB$cloud MongoDB • NewYork&CA, services support services • London&Dublin • Sydney
  • 4. MongoDB is a....•Document Oriented•High Performance•Highly Available•Horizontally Scalable ...Operational Datastore
  • 5. Changes(impacting(the(traditional RDBMS. Agile(Development • Iterative&continuous • NewandemergingAppsVolume(and(Type(ofData • Trillionsofrecords • 10’sofmillionsof queriespersecond • Volumeofdata • Semi?structuredand unstructureddata New(Architectures • Systemsscalinghorizontally,not vertically • Commodityservers • CloudComputing 6
  • 6. Technology stack addssignificant complexity complexity • custom sharding • caching • vertical scaling
  • 7. Technology stack reduces productivity • denormalize • remove joins • remove transactions productivity
  • 8. What MongoDB solves
  • 9. • memcachedscalability & performance • key/value • RDBMS depth of functionality
  • 10. Complex Tables to Documents { title: ‘MongoDB’, contributors: [ { name: ‘Eliot Horowitz’, email: ‘’ }, { name: ‘Dwight Merriman’, email: ‘’ } ], model: { relational: false, awesome: true } }
  • 11.
  • 12. HiAv & Scale Out - Shardsread shard1 shard2 shard3 node_c1 node_c2 node_c3 node_b1 node_b2 node_b3 node_a1 node_a2 node_a3 write
  • 13. Key Features•Indexes on any attribute•Dynamic Query Language•Aggregation Framework•Dynamic & Flexible Schemas•Atomic Updates to documents•Impedance Mismatch removed•High Availability & Failover•Strong consistency of data•Horizonal Scale Out•Hadoop Integration
  • 14. Use Cases Content%Management Operational%Intelligence E<CommerceUser%Data%Management High%Volume%Data%Feeds Mobile
  • 15. FinancialServicesUseCases•High Volume Data Feeds•Tick Data capture•Risk Analytics & Reporting•Product Catalogs & Trade Capture•P&L Reporting•Reference Data Management•Portfolio Management•Quantitative Analysis•Automated Trading
  • 16. High Volume Data FeedsUse Case:•Ingesting data from different feeds sources - Internal / External •Risk data, Market data, Order data etc Example,)FIX)to)JSON •Any format Fix / FpML / Swift or own "NewOrder-Single" : {•Formats vary over time "Header" : { "BeginString" : "FIX.4.2", • Eg Fix 4.2 to 5.0 "BodyLength" : 190, "MsgType" : "D",•Tick Data "HeaderFields" : { "SenderCompID" : "Client",•Aggregate data from feeds "TargetCompID" : "TradingGateway", "MsgSeqNum" : 4, "SendingTime" : { "UTCFormat_2051-100" :Why MongoDB? "Fri Jun 01 09:36:26 BST 2012" }.....•High ingestion rates of data } }•Flexible schema - no db change when messages change •eg Single collection can maintain multiple FIX formats•Query: •Query Language / Aggregation Framework •Or use internal MR or hadoop to batch process - quantative analysis
  • 17. Risk Analytics & ReportingUse Case:•Collect and aggregate risk data•Calculate risk / exposures•Potentially real timeWhy MongoDB?•Collect data from a single or multiple sources •Different formats•Documents used to create ‘pre-aggregated’ reports •Real Time•Aggregation Framework for reporting •e.g. exposure for a counter party•Internal MR or Hadoop connector •Batch process risk data
  • 18. Product Catalogs and Trade CaptureUse Case:•Catalogs of complex financial products •‘Exotics’ difficult to model in relational db.•‘On-boarding’ new products in hours. •RDBMS less flexible for complex products that may require >~50 tables. What’s the impact from technology•Once create how do we capture the details of a new trade?Why MongoDB?•Flexible schema means we don’t need to go back to the databasewhen we have a product to sell.•Single collection for all products... even if they vary greatly•Trades potentially exist for long periods •newer trade can have different data with not impact on the db.
  • 19. Portfolio / Position reportingUse Case:•Store positions or portfolio information•Query to find current positions/portfolios•Query by client or traderWhy MongoDB?•Customer/client my have many different products•Aggregation Framework to calculate values and views•Work on extremely large data sets •Current and historic data
  • 20. Reference Data ManagementUse Case:•Global distribute Reference Data across organisation•Manage replication across of data centres•Provide fast read accessWhy MongoDB?•Sharding and replication to distribute data•Access locally for high performance reads•Fast replication of data•Unstructured reference data easily replicated. •New items/formats replicated without schema migrations NYC LON HK Primary Secondary Secondary Secondary Primary Secondary
  • 21. Quantative Analysis / Automated TradingUse Case:• Real time and history tick data. BID/OFFER)to)Candlesticks• Strategy testing { "_id" : ObjectId("4f4b8916fb1c80e141ea6201"),• Automated signals "ask" : 1.30028, "bid" : 1.3002, "ts" : ISODate("2012-02-16T12:48:00Z") }Why MongoDB?•Aggregation Framework for shape data 18018 •Bid/Offer -> Candlesticks•MR for batch processing data •Internal MR or Hadoop
  • 22. Aggregations•Number of choices •MongoDB Map Reduce •Pre-Compute - Schema Design •Hadoop Connector •Aggregation Framework
  • 23. Aggregation Framework•Much simpler and faster than MongoDB map reduce•Replaces common MR use cases in MongoDB•Native operators in the MongoDB coredb.portfolio.aggregate( { $match : { userid : “roberts123” } } , { $group : { _id : "$position" , total : { $sum : “$val” } } } )
  • 24. Sharded MongoDB + HadoopShard&1 Shard&2 Shard&3 Shard&4 Shard&5 c z t f v w ya s u g e h d b x Hadoop Hadoop Hadoop Hadoop Hadoop Node Node Node Node Node Hadoop Hadoop Hadoop Hadoop Node Node Node Node
  • 25. SummaryDocument-Oriented Dynamic schema High Volume Data Feeds Agile Tick Data capture Flexible Risk AnalyticsHigh Performance Product Catalogs & TradeHighly Available CaptureHorizontal Scale Out P&L Reporting Reference Data Management Portfolio Management Quantitative Analysis Automated Trading
  • 26. download at @dmroberts Free+online+training+1+ sdf Facebook+++++++++Twitter+++++++++LinkedIn @dmroberts