0
How Capital Markets Firms UseMongoDB as a Tick DatabaseAntoine Girbal, Technical Account ManagerEmail: antoine@10gen.comTw...
2• MongoDB Introduction• FS Use Cases• Writing/Capturing Market Data• Reading/Analyzing Market Data• Performance, Scalabil...
3Introduction10gen is the company behind MongoDB –the leading next generation databaseDocument-OrientedOpen-SourceGeneralP...
410gen Overview200+ employees 500+ customersOver $81 million in fundingOffices in New York, Palo Alto, WashingtonDC, Londo...
5Database Landscape• No Automatic Joins• Document Transactions• Fast, Scalable Read/Writes
6MongoDB Business BenefitsIncreased Developer Productivity Better Customer ExperienceFaster Time to Market Lower TCO
7MongoDB Technical BenefitsHorizontally Scalable-ShardingAgile &FlexibleHighPerformance-Indexes-RAMApplicationHighlyAvaila...
8Most Common FS Use Cases1. Tick Data Capture & Analysis2. Reference Data Management3. Risk Analysis & Reporting4. Trade R...
9Tick Data Capture & Analysis -Requirements• Capture real-time market data (multi-asset, top ofbook, depth of book, even n...
10Tick Data Capture & Analysis –Why MongoDB?• High throughput => can capture real-time feeds for allproducts/asset classes...
Writing/Capturing Tick Data
12Trades/metricsHigh Level Trading ArchitectureFeed HandlerExchanges/Markets/BrokersCapturingApplicationLow LatencyApplica...
13Trades/metricsHigh Level Trading ArchitectureFeed HandlerExchanges/Markets/BrokersCapturingApplicationLow LatencyApplica...
14{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),bidPrice: 55.37,offerP...
15{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),bidPrices: [55.37, 55....
16{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),bids: [{price: 55.37, ...
17{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),spreadPrice: 0.58leg1:...
18{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),title: “Disney Earning...
19{_id : ObjectId("4e2e3f92268cdda473b628f6"),timestamp: ISODate("2013-02-15 10:00"),twitterHandle: “jdoe”,tweet: “Heard @...
20{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS”,openTS: Date("2013-02-15 10:00"),closeTS: Date("2013-02-15 10...
Querying/Analyzing Tick Data
22Architecture for Querying DataHigher LatencyTradingApplicationsBacktestingApplications• Ticks• Bars• Other analysisResea...
23Index any fields: arrays, nested, etc// Compound indexes> db.ticks.ensureIndex({symbol: 1, timestamp:1})// Index on arra...
24Query for ticks by time; pricethreshold// Ticks for last month for media companies> db.ticks.find({symbol: {$in: ["DIS",...
25• Custom application code– Run your queries, compute your results• Aggregation framework– Declarative, pipeline-based ap...
26//Aggregate minute bars for Disney for this monthdb.ticks.aggregate({ $match: {symbol: "DIS”, timestamp: {$gt: new ISODa...
27…//then count the number of down bars{ $project: {downBar: {$lt: [“$close”, “$open”] },timestamp: 1,open: 1, high: 1, lo...
28var mapFunction = function () {emit(this.symbol, this.bidPrice);}var reduceFunction = function (symbol, priceList) {retu...
29• MongoDB’s Hadoop Connector• Supports Map/Reduce, Streaming, Pig• MongoDB as input/output storage for Hadoopjobs– No ne...
Performance, Scalability, andHigh Availability
31Why MongoDB is fast and scalableBetter data localityRelational MongoDBIn-MemoryCachingAuto-ShardingRead/write scaling
32Auto-Sharding for Horizontal ScalemongodRead/Write ScalabilityKey RangeSymbol: A…Z
33Auto-Sharding for Horizontal ScaleRead/Write Scalabilitymongod mongodKey RangeSymbol: A…JKey RangeSymbol: K…Z
34Shardingmongod mongodmongod mongodRead/Write ScalabilityKey RangeSymbol: A…FKey RangeSymbol: G…JKey RangeSymbol: K…OKey ...
35PrimarySecondarySecondaryPrimarySecondarySecondaryPrimarySecondarySecondaryPrimarySecondarySecondaryMongoS MongoS MongoS...
36SubscriptionsProfessional Support, Enterprise Edition and Commercial License10gen Products and ServicesConsultingExpert ...
37• MongoDB is high performance for tick data• Scales horizontally automatically by auto-sharding• Fast, flexible querying...
38Resource LocationMongoDB Downloads www.mongodb.org/downloadFree Online Training education.10gen.comWebinars and Events w...
How Capital Markets Firms UseMongoDB as a Tick DatabaseMatt Kalan, Sr. Solution ArchitectEmail: Matt.kalan@10gen.comTwitte...
Webinar: How Banks Use MongoDB as a Tick Database
Upcoming SlideShare
Loading in...5
×

Webinar: How Banks Use MongoDB as a Tick Database

2,380

Published on

Learn why MongoDB is spreading like wildfire across capital markets (and really every industry) and then focus in particular on how financial firms are enjoying the developer productivity, low TCO, and unlimited scale of MongoDB as a tick database for capturing, analyzing, and taking advantage of opportunities in tick data.

1 Comment
9 Likes
Statistics
Notes
No Downloads
Views
Total Views
2,380
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
70
Comments
1
Likes
9
Embeds 0
No embeds

No notes for slide
  • Mention tick databases
  • JSON document – contains key value pairs, different types, values can also be arrays and other documents
  • because of the way MongoDB lets you update documents atomically we can be sure totals and list of voters will stay in sync
  • because of the way MongoDB lets you update documents atomically we can be sure totals and list of voters will stay in sync
  • because of the way MongoDB lets you update documents atomically we can be sure totals and list of voters will stay in sync
  • comments is an array of JSON documentswe can query by fields inside embedded documents as well as array members.
  • secondary indexes, compound indexes, multikey indexes.why is it important to have all of document together? data locality
  • secondary indexes, compound indexes, multikey indexes.why is it important to have all of document together? data locality
  • Fewer reads, data is together, memory mapped files, caching handled by OS, naturally leaves most frequently accessed data in RAM (have enough RAM to fit indexes and working data set into RAM for best performance), horizontal scaling is "built-in" to the product by design from the start.
  • Full deployment. As many mongoS processes as you have app servers (for example); Config DBs are small but hold the critical information about where ranges of data are located on disk/shards.
  • Transcript of "Webinar: How Banks Use MongoDB as a Tick Database"

    1. 1. How Capital Markets Firms UseMongoDB as a Tick DatabaseAntoine Girbal, Technical Account ManagerEmail: antoine@10gen.comTwitter: @antoinegirbal
    2. 2. 2• MongoDB Introduction• FS Use Cases• Writing/Capturing Market Data• Reading/Analyzing Market Data• Performance, Scalability, & High Availability• Q&AAgenda
    3. 3. 3Introduction10gen is the company behind MongoDB –the leading next generation databaseDocument-OrientedOpen-SourceGeneralPurpose
    4. 4. 410gen Overview200+ employees 500+ customersOver $81 million in fundingOffices in New York, Palo Alto, WashingtonDC, London, Dublin, Barcelona and Sydney
    5. 5. 5Database Landscape• No Automatic Joins• Document Transactions• Fast, Scalable Read/Writes
    6. 6. 6MongoDB Business BenefitsIncreased Developer Productivity Better Customer ExperienceFaster Time to Market Lower TCO
    7. 7. 7MongoDB Technical BenefitsHorizontally Scalable-ShardingAgile &FlexibleHighPerformance-Indexes-RAMApplicationHighlyAvailable-Replica Sets{ author: “roger”,date: new Date(),text: “Spirited Away”,tags: [“Tezuka”, “Manga”]}
    8. 8. 8Most Common FS Use Cases1. Tick Data Capture & Analysis2. Reference Data Management3. Risk Analysis & Reporting4. Trade Repository5. Portfolio Reporting
    9. 9. 9Tick Data Capture & Analysis -Requirements• Capture real-time market data (multi-asset, top ofbook, depth of book, even news)• Load historical data• Aggregate data into bars, daily, monthly intervals• Enable queries & analysis on raw ticks oraggregates• Drive backtesting or automated signals
    10. 10. 10Tick Data Capture & Analysis –Why MongoDB?• High throughput => can capture real-time feeds for allproducts/asset classes needed• High scalability => all data and depth for all historical time periodscan be captured• Flexible & Range-based indexing => fast querying on time rangesand any fields• Aggregation Framework => can shape raw data into aggregates(e.g. ticks to bars)• Map-reduce capability (Native MR or Hadoop Connector) => batchanalysis looking for patterns and opportunities• Easy to use => native language drivers and JSON expressions thatyou can apply for most operational database needs as well• Low TCO => Low software license cost and commodity hardware
    11. 11. Writing/Capturing Tick Data
    12. 12. 12Trades/metricsHigh Level Trading ArchitectureFeed HandlerExchanges/Markets/BrokersCapturingApplicationLow LatencyApplicationsHigher LatencyTradingApplicationsBacktesting andAnalysisApplicationsMarket DataCached Static &Aggregated DataNews & socialnetworkingsourcesOrdersOrders
    13. 13. 13Trades/metricsHigh Level Trading ArchitectureFeed HandlerExchanges/Markets/BrokersCapturingApplicationLow LatencyApplicationsHigher LatencyTradingApplicationsBacktesting andAnalysisApplicationsMarket DataCached Static &Aggregated DataNews & socialnetworkingsourcesOrdersOrdersData Types• Top of book• Depth of book• Multi-asset• Derivatives (e.g. strips)• News (text, video)• Social Networking
    14. 14. 14{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),bidPrice: 55.37,offerPrice: 55.58,bidQuantity: 500,offerQuantity: 700}> db.ticks.find( {symbol: "DIS",bidPrice: {$gt: 55.36} } )Top of book [e.g. equities]
    15. 15. 15{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),bidPrices: [55.37, 55.36, 55.35],offerPrices: [55.58, 55.59, 55.60],bidQuantities: [500, 1000, 2000],offerQuantities: [1000, 2000, 3000]}> db.ticks.find( {bidPrices: {$gt: 55.36} } )Depth of book
    16. 16. 16{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),bids: [{price: 55.37, amount: 500},{price: 55.37, amount: 1000},{price: 55.37, amount: 2000} ],offers: [{price: 55.58, amount: 1000},{price: 55.58, amount: 2000},{price: 55.59, amount: 3000} ]}> db.ticks.find( {"bids.price": {$gt: 55.36} } )or any way your app uses it
    17. 17. 17{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),spreadPrice: 0.58leg1: {symbol: “CLM13”, price: 97.34}leg2: {symbol: “CLK13”, price: 96.92}}db.ticks.find( { leg1 : “CLM13” },{ leg2 : “CLK13” },{ spreadPrice : {$gt: 0.50 } } )Synthetic spreads
    18. 18. 18{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS",timestamp: ISODate("2013-02-15 10:00"),title: “Disney Earnings…”body: “Walt Disney Company reported…”,tags: [“earnings”, “media”, “walt disney”]}News
    19. 19. 19{_id : ObjectId("4e2e3f92268cdda473b628f6"),timestamp: ISODate("2013-02-15 10:00"),twitterHandle: “jdoe”,tweet: “Heard @DisneyPictures is releasing…”,usernamesIncluded: [“DisneyPictures”],hashTags: [“movierumors”, “disney”]}Social networking
    20. 20. 20{_id : ObjectId("4e2e3f92268cdda473b628f6"),symbol : "DIS”,openTS: Date("2013-02-15 10:00"),closeTS: Date("2013-02-15 10:05"),open: 55.36,high: 55.80,low: 55.20,close: 55.70}Aggregates (bars, daily, etc.)
    21. 21. Querying/Analyzing Tick Data
    22. 22. 22Architecture for Querying DataHigher LatencyTradingApplicationsBacktestingApplications• Ticks• Bars• Other analysisResearch &AnalysisApplications
    23. 23. 23Index any fields: arrays, nested, etc// Compound indexes> db.ticks.ensureIndex({symbol: 1, timestamp:1})// Index on arrays>db.ticks.ensureIndex( {bidPrices: -1})// Index on any depth> db.ticks.ensureIndex( {“bids.price”: 1} )// Full text search> db.ticks.ensureIndex ( {tweet: “text”} )
    24. 24. 24Query for ticks by time; pricethreshold// Ticks for last month for media companies> db.ticks.find({symbol: {$in: ["DIS", “VIA“, “CBS"]},timestamp: {$gt: new ISODate("2013-01-01")},timestamp: {$lte: new ISODate("2013-01-31")}})// Ticks when Disney’s bid breached 55.50 this month> db.ticks.find({symbol: "DIS",bidPrice: {$gt: 55.50},timestamp: {$gt: new ISODate("2013-02-01")}})
    25. 25. 25• Custom application code– Run your queries, compute your results• Aggregation framework– Declarative, pipeline-based approach• Native Map/Reduce in MongoDB– Javascript functions distributed across cluster• Hadoop Connector– Offline batch processing/computationAnalyzing/Aggregating Options
    26. 26. 26//Aggregate minute bars for Disney for this monthdb.ticks.aggregate({ $match: {symbol: "DIS”, timestamp: {$gt: new ISODate("2013-02-01")}}},{ $project: {year: {$year: "$timestamp"},month: {$month: "$timestamp"},day: {$dayOfMonth: "$timestamp"},hour: {$hour: "$timestamp"},minute: {$minute: "$timestamp"},second: {$second: "$timestamp"},timestamp: 1,price: 1}},{ $sort: { timestamp: 1}},{ $group :{ _id : {year: "$year", month: "$month", day: "$day", hour: "$hour", minute: "$minute"},open: {$first: "$price"},high: {$max: "$price"},low: {$min: "$price"},close: {$last: "$price"} }} )Aggregate into min bars
    27. 27. 27…//then count the number of down bars{ $project: {downBar: {$lt: [“$close”, “$open”] },timestamp: 1,open: 1, high: 1, low: 1, close: 1}},{ $group: {_id: “$downBar”,sum: {$sum: 1}}} })Add analysis on the bars
    28. 28. 28var mapFunction = function () {emit(this.symbol, this.bidPrice);}var reduceFunction = function (symbol, priceList) {return Array.sum(priceList);}> db.ticks.mapReduce(map, reduceFunction, {out: ”tickSums"})Map-Reduce Example: Sum
    29. 29. 29• MongoDB’s Hadoop Connector• Supports Map/Reduce, Streaming, Pig• MongoDB as input/output storage for Hadoopjobs– No need to go through HDFS• Leverage power of Hadoop ecosystem againstoperational data in MongoDBProcess Data on Hadoop
    30. 30. Performance, Scalability, andHigh Availability
    31. 31. 31Why MongoDB is fast and scalableBetter data localityRelational MongoDBIn-MemoryCachingAuto-ShardingRead/write scaling
    32. 32. 32Auto-Sharding for Horizontal ScalemongodRead/Write ScalabilityKey RangeSymbol: A…Z
    33. 33. 33Auto-Sharding for Horizontal ScaleRead/Write Scalabilitymongod mongodKey RangeSymbol: A…JKey RangeSymbol: K…Z
    34. 34. 34Shardingmongod mongodmongod mongodRead/Write ScalabilityKey RangeSymbol: A…FKey RangeSymbol: G…JKey RangeSymbol: K…OKey RangeSymbol: P…Z
    35. 35. 35PrimarySecondarySecondaryPrimarySecondarySecondaryPrimarySecondarySecondaryPrimarySecondarySecondaryMongoS MongoS MongoSApplicationKey RangeSymbol: A…F,TimeKey RangeSymbol: G…J,TimeKey RangeSymbol: K…O,TimeKey RangeSymbol: P…Z,Time
    36. 36. 36SubscriptionsProfessional Support, Enterprise Edition and Commercial License10gen Products and ServicesConsultingExpert Resources for All Phases of MongoDB ImplementationsTrainingOnline and In-Person, for Developers and Administrators
    37. 37. 37• MongoDB is high performance for tick data• Scales horizontally automatically by auto-sharding• Fast, flexible querying, analysis, & aggregation• Dynamic schema can handle any data types• MongoDB has all these features with low TCO• 10gen can support you with anything discussedSummary
    38. 38. 38Resource LocationMongoDB Downloads www.mongodb.org/downloadFree Online Training education.10gen.comWebinars and Events www.10gen.com/eventsWhite Papers www.10gen.com/white-papersCustomer Case Studies www.10gen.com/customersPresentations www.10gen.com/presentationsDocumentation docs.mongodb.orgAdditional Info info@10gen.comFor More InformationResource User Data Management
    39. 39. How Capital Markets Firms UseMongoDB as a Tick DatabaseMatt Kalan, Sr. Solution ArchitectEmail: Matt.kalan@10gen.comTwitter: @matthewkalan
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×