MongoDB in FS

2,575 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,575
On SlideShare
0
From Embeds
0
Number of Embeds
1,337
Actions
Shares
0
Downloads
49
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

MongoDB in FS

  1. 1. Easy to Start, Easy to Develop, Easy to Scale MongoDB In FS 10gen, Inc. October 2012
  2. 2. @dmrobertsdaniel.roberts@10gen.com Solution(Architect Based(in(London sdf http://www.10gen.com/
  3. 3. 10gen$is$the$company$behindMongoDB.• Foundedin2007 • DwightMerriman,EliotHorowitz Set$the Foster direction$& community$&• $73M+infunding contribute ecosystem code$to • Flybridge,Sequoia,UnionSquare, MongoDB NEA• WorldwideExpanding Team • 170+employees Provide Provide • Locations: MongoDB$cloud MongoDB • NewYork&CA, services support services • London&Dublin • Sydney
  4. 4. MongoDB is a....•Document Oriented•High Performance•Highly Available•Horizontally Scalable ...Operational Datastore
  5. 5. Changes(impacting(the(traditional RDBMS. Agile(Development • Iterative&continuous • NewandemergingAppsVolume(and(Type(ofData • Trillionsofrecords • 10’sofmillionsof queriespersecond • Volumeofdata • Semi?structuredand unstructureddata New(Architectures • Systemsscalinghorizontally,not vertically • Commodityservers • CloudComputing 6
  6. 6. Technology stack addssignificant complexity complexity • custom sharding • caching • vertical scaling
  7. 7. Technology stack reduces productivity • denormalize • remove joins • remove transactions productivity
  8. 8. What MongoDB solves
  9. 9. • memcachedscalability & performance • key/value • RDBMS depth of functionality
  10. 10. Complex Tables to Documents { title: ‘MongoDB’, contributors: [ { name: ‘Eliot Horowitz’, email: ‘eliot@10gen.com’ }, { name: ‘Dwight Merriman’, email: ‘dwight@10gen.com’ } ], model: { relational: false, awesome: true } }
  11. 11. bsonspec.org
  12. 12. HiAv & Scale Out - Shardsread shard1 shard2 shard3 node_c1 node_c2 node_c3 node_b1 node_b2 node_b3 node_a1 node_a2 node_a3 write
  13. 13. Key Features•Indexes on any attribute•Dynamic Query Language•Aggregation Framework•Dynamic & Flexible Schemas•Atomic Updates to documents•Impedance Mismatch removed•High Availability & Failover•Strong consistency of data•Horizonal Scale Out•Hadoop Integration
  14. 14. Use Cases Content%Management Operational%Intelligence E<CommerceUser%Data%Management High%Volume%Data%Feeds Mobile
  15. 15. FinancialServicesUseCases•High Volume Data Feeds•Tick Data capture•Risk Analytics & Reporting•Product Catalogs & Trade Capture•P&L Reporting•Reference Data Management•Portfolio Management•Quantitative Analysis•Automated Trading
  16. 16. High Volume Data FeedsUse Case:•Ingesting data from different feeds sources - Internal / External •Risk data, Market data, Order data etc Example,)FIX)to)JSON •Any format Fix / FpML / Swift or own "NewOrder-Single" : {•Formats vary over time "Header" : { "BeginString" : "FIX.4.2", • Eg Fix 4.2 to 5.0 "BodyLength" : 190, "MsgType" : "D",•Tick Data "HeaderFields" : { "SenderCompID" : "Client",•Aggregate data from feeds "TargetCompID" : "TradingGateway", "MsgSeqNum" : 4, "SendingTime" : { "UTCFormat_2051-100" :Why MongoDB? "Fri Jun 01 09:36:26 BST 2012" }.....•High ingestion rates of data } }•Flexible schema - no db change when messages change •eg Single collection can maintain multiple FIX formats•Query: •Query Language / Aggregation Framework •Or use internal MR or hadoop to batch process - quantative analysis
  17. 17. Risk Analytics & ReportingUse Case:•Collect and aggregate risk data•Calculate risk / exposures•Potentially real timeWhy MongoDB?•Collect data from a single or multiple sources •Different formats•Documents used to create ‘pre-aggregated’ reports •Real Time•Aggregation Framework for reporting •e.g. exposure for a counter party•Internal MR or Hadoop connector •Batch process risk data
  18. 18. Product Catalogs and Trade CaptureUse Case:•Catalogs of complex financial products •‘Exotics’ difficult to model in relational db.•‘On-boarding’ new products in hours. •RDBMS less flexible for complex products that may require >~50 tables. What’s the impact from technology•Once create how do we capture the details of a new trade?Why MongoDB?•Flexible schema means we don’t need to go back to the databasewhen we have a product to sell.•Single collection for all products... even if they vary greatly•Trades potentially exist for long periods •newer trade can have different data with not impact on the db.
  19. 19. Portfolio / Position reportingUse Case:•Store positions or portfolio information•Query to find current positions/portfolios•Query by client or traderWhy MongoDB?•Customer/client my have many different products•Aggregation Framework to calculate values and views•Work on extremely large data sets •Current and historic data
  20. 20. Reference Data ManagementUse Case:•Global distribute Reference Data across organisation•Manage replication across of data centres•Provide fast read accessWhy MongoDB?•Sharding and replication to distribute data•Access locally for high performance reads•Fast replication of data•Unstructured reference data easily replicated. •New items/formats replicated without schema migrations NYC LON HK Primary Secondary Secondary Secondary Primary Secondary
  21. 21. Quantative Analysis / Automated TradingUse Case:• Real time and history tick data. BID/OFFER)to)Candlesticks• Strategy testing { "_id" : ObjectId("4f4b8916fb1c80e141ea6201"),• Automated signals "ask" : 1.30028, "bid" : 1.3002, "ts" : ISODate("2012-02-16T12:48:00Z") }Why MongoDB?•Aggregation Framework for shape data 18018 •Bid/Offer -> Candlesticks•MR for batch processing data •Internal MR or Hadoop
  22. 22. Aggregations•Number of choices •MongoDB Map Reduce •Pre-Compute - Schema Design •Hadoop Connector •Aggregation Framework
  23. 23. Aggregation Framework•Much simpler and faster than MongoDB map reduce•Replaces common MR use cases in MongoDB•Native operators in the MongoDB coredb.portfolio.aggregate( { $match : { userid : “roberts123” } } , { $group : { _id : "$position" , total : { $sum : “$val” } } } )
  24. 24. Sharded MongoDB + HadoopShard&1 Shard&2 Shard&3 Shard&4 Shard&5 c z t f v w ya s u g e h d b x Hadoop Hadoop Hadoop Hadoop Hadoop Node Node Node Node Node Hadoop Hadoop Hadoop Hadoop Node Node Node Node
  25. 25. SummaryDocument-Oriented Dynamic schema High Volume Data Feeds Agile Tick Data capture Flexible Risk AnalyticsHigh Performance Product Catalogs & TradeHighly Available CaptureHorizontal Scale Out P&L Reporting Reference Data Management Portfolio Management Quantitative Analysis Automated Trading
  26. 26. download at mongodb.org @dmroberts daniel.roberts@10gen.com Free+online+training+1+http://education.10gen.com/ www.meetup.com/London1MongoDB1User1Group/ sdf Facebook+++++++++Twitter+++++++++LinkedInhttp://bit.ly/mongodb @dmroberts http://linkd.in/joinmongo

×