Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company

876 views

Published on

Proximus is one of the biggest Telecom companies in the Belgian market. This year the company began developing a new IoT network using LoRaWan technology. The talk will detail our development team’s search for a database suited to meet the needs of our IoT project, the selection and implementation of MongoDB as a database, as well as well as how we built a system for storing a variety of sensor data with high throughput by leveraging sleepy.mongoose. The talk will also discuss how different decisions around data storage impact applications in regards to both performance and total cost.

Published in: Data & Analytics
  • Login to see the comments

MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company

  1. 1. ENABLING IOT AT PROXIMUS
  2. 2. HI, MY NAME IS DAVE. Software Engineer Ordina Belgium dave.degroote@ordina.be
  3. 3. IN THIS SESSION How to implement mongoDB in a IoT M2M use case. Data modelling is key when storing time series data. How to insert data into mongoDB using Rest interface.
  4. 4. INITIAL QUESTIONS How to store data while keeping storage amount minimal? What storage engine to choose? How to store my data using a third party system? How can I easily deploy my mongoDB solution?
  5. 5. ESSENTIAL PROBLEMS How do we model our data? De ning storage engine Setting up your mongo Instances How do we insert our data in mongoDB? How do we query our data?
  6. 6. SETTING THE SCENE Mythings Project at Proximus What is IoT? LoRa Technology
  7. 7. 2 Front-end 3 Back-end MYTHINGS DEVELOPMENT TEAM Small team: Mongo experience: 1,5 year
  8. 8. MYTHINGS AT PROXIMUS
  9. 9. MYTHINGS AT PROXIMUS
  10. 10. INTERNET OF THINGS Very broad, not contained to a single technology. Not what but HOW ! Internet is the keyword Collecting, analyzing and above all communicating data.
  11. 11. LORA TECHNOLOGY
  12. 12. LORA TECHNOLOGY VS ...
  13. 13. LORA ARCHITECTURE
  14. 14. SENSORS Any type is possible as long as it is connected with a LoRa chip CO2, PIR, Humidity, Luminance, Temperature, Pressure, …
  15. 15. SENSORS multiple containers per sensor possible
  16. 16. LORA ALLIANCE https://www.lora-alliance.org
  17. 17. ESSENTIAL PROBLEMS How do we model our data? De ning storage engine Setting up your mongo Instances How do we insert our data in mongoDB? How do we query our data?
  18. 18. DATA MODELING SENSOR HISTORY company info sensor info container info noti cation value noti cation timestamp
  19. 19. SENSORHISTORY COLLECTION { "_id": ObjectID("5613d33401e101c27f8e668d"), "company": { "name": "Proximus", "companyId": 2 }, "container": { "container": "0x0050.0x0006.0.m2m", "type": "temperature", "containerId": 597, "name": "temperature" }, "sensor": { "sensorId": 773, "mac": "F03D291000001301" }, "data": { "type": "bool", "ts": 1444139821000, "value": "1" } }
  20. 20. Daily document size/sensor = SENSORHISTORY COLLECTION SIZING Document size = 496 Bytes Sensor will send 48 time a day On average 3 containers/sensor 496 B * 48 documents * 3 = 71,4 KB 71,4 KB 1 M sensors => 68.09 GB/day
  21. 21. Document with 3 additions: BUCKETING TECHNIQUE Data array (data values) Count metadata (Array size) Date eld (day timestamp) Use in combination with Upsert mechanism!
  22. 22. SENSORHISTORY COLLECTION (BUCKETING) { "_id": ObjectID("5613d33401e101c27f8e668d"), "company": { "name": "Proximus", "companyId": 2 }, "container": { "container": "0x0050.0x0006.0.m2m", "type": "temperature", "containerId": 597, "name": "temperature" }, "date": 1444086000000, "sensor": { "sensorId": 773, "mac": "F03D291000001301" }, "count": 48, "data": [ { "type": "bool", "ts": 1444139821000, "value": "1" }, { "type": "bool", "ts": 1444152754000, "value": "1" },... ] }
  23. 23. daily document size/sensor = SENSORHISTORY COLLECTION SIZING (BUCKETING) Document size = 3.98 KB Sensor will send 48 time a day On average 3 containers/sensor 3.98 KB * 1 document * 3 = 11,9 KB 11,9 KB 1 M sensors => 11.34 GB/day
  24. 24. Use if possible (depending on use-case) vs DATA MODELING BUCKETING 11GB/day 68GB/day
  25. 25. Concurrent updates on the bucket document may lead to queue operations Do not overdue it! DATA MODELING BUCKETING Advantages: Collection size will be smaller Indexes size will be considerable reduced Reads will be faster Remarks: ! Maximum document size 16MB !
  26. 26. DEFINING STORAGE ENGINE APPLICATION REQUIREMENTS: Write intensive Keep storage volume to a minimum Read and write simultaniously to collection
  27. 27. DEFINING STORAGE ENGINE WIREDTIGER 3.2 Document level lock instead of collection level Data is stored in a compressed format Index size is smaller Matched the application requirements WiredTiger was recommended by MongoDB
  28. 28. SETTING UP MONGODB INSTANCES CLOUD MANAGER
  29. 29. Once automation agent is installed, smooth sailing! SETTING UP YOUR MONGODB INSTANCES CLOUD MANAGER Easy setup Easy management Metrics
  30. 30. INSERTING DATA MYTHINGS ARCHITECTURE
  31. 31. INSERTING DATA REST INTERFACE Sleepy Mongoose (Python) https://github.com/mongodb-labs/sleepy.mongoose Rest Heart (Java) latest feature support http://restheart.org/ Advised to change to Rest Heart by MongoDB
  32. 32. Latest feature support Uses PATCH method for updates (Upsert) => Not supported current release Not mature enough: no bucketing for upsert! INSERTING DATA REST HEART
  33. 33. Back to square one! INSERTING DATA SLEEPY MONGOOSE
  34. 34. Interchange Rest heart <> Sleepy Mongoose Easy transition (1-2 days development)! INSERTING DATA SLEEPY MONGOOSE
  35. 35. Simple insert: Advanced upsert: INSERTING DATA $ curl --data 'docs=[{"x":2},{"x":3}]' 'http://localhost:27080/foo/bar/_insert' $ curl --data 'criteria= { "sensor":{"sensorId":1533,"name":"Binary mesurement","mac":"020000FFFF00B151"}, "company":{"companyId":134,"name":"Opinum"}, "container":{"containerId":1223,"name":"Door counter"}, "date":1456354800000 }, "count":{"$lt":100} &newobj={ "$push":{"data":{"ts":1456403435000,"value":"456","type":"int"}}, "$inc":{"count":1}, "$setOnInsert":{"location":{"latitude":"-1","longitude":"-1"}}} &upsert=true' 'http://localhost:27080/foo/bar/_insert'
  36. 36. QUERING DATA Query framework Aggregation framework
  37. 37. Retrieve sensorhistory for a week for all companies: AGGREGATION FRAMEWORK "op" : "command", "ns" : "sensoringlabs.sensorHistory", "command" : { "aggregate" : "sensorHistory", "pipeline" : [{ "$match" : { "company.companyId" : { $in" : [,54,61,76,77,79,80,81,82,83,86,87,90,122,123 ,124,125,126,127,128,129,130,131,132,133,134,135,136, 137,138,146,147,148,149,150,151,152,153,154] } } }, { "$match" : { "date" : {"$gte" : NumberLong("1457049600000")}, "$and" : [{"date" : {"$lte" : NumberLong("1457568000000")}}] } }, {"$skip" : NumberLong(0)}, {"$limit" : NumberLong(100)} ], "cursor" : {} } "responseLength" : 278888, "protocol" : "op_command", "millis" : 366
  38. 38. Retrieve sensorhistory for a week for all companies: QUERY FRAMEWORK "op" : "query", "ns" : "sensoringlabs.sensorHistory", "query" : { "find" : "sensorHistory", "filter" : { "company.companyId" : { "$in" : [2,54,61,76,77,79,80,81,82,83,86,87,90,122,123,124, 125,126,127,128,129,130,131,132,133,134,135,136,137, 138,146,147,148,149,150,151,152,153,154] }, "date" : { "$gte" : NumberLong("1457049600000"), "$lte" : NumberLong("1457568000000") } }, "limit" : 100, "singleBatch" : false } "nreturned" : 100, "responseLength" : 278888, "protocol" : "op_command", "millis" : 1
  39. 39. Same search: Query framework : Aggregation framework : QUERING DATA 1ms 360 ms
  40. 40. -> fast for retrieving data -> data aggregations that take more time Read concern: read from secondary QUERING DATA Query framework Aggregation framework
  41. 41. Query example: QUERING DATA SLEEPY MONGOOSE $ curl -X GET 'http://localhost:27080/foo/bar/_find ?criteria={"_id":{"$oid":"4f8c6f05db61e2a72600001d"}} &sort={"date”:-1} &limit=1'
  42. 42. INDEX STRATEGY COMPOUND INDEXES Multitenant system: company id Time based data: date
  43. 43. Data retention policy: -> Deterministic way to drop documents once expired -> Fixed size collections CC Not suitable because time range of documents will depend on throughput! (high throughput will decrease the time range stored). CAPPED COLLECTION VS TTL INDEXES GETTING RID OF THE EXPIRED DATA 3 months TTL index Capped collection
  44. 44. PUTTING IT ALL TOGETHER
  45. 45. Bucketing DATA MODELING
  46. 46. Wired tiger 3.2 STORAGE ENGINE
  47. 47. Sleepy Mongoose RestHeart (future) Java mongo Driver DATA INSERTION
  48. 48. query framework aggregation framework DATA EXTRACTION
  49. 49. compound indexes TTL index INDEXES
  50. 50. SHARDING WHEN There is more data than one machine can hold on its drives The application is write heavy and you could experience too much latency The working set outgrows the memory can be allocated to a single machine
  51. 51. CONCLUSION WHAT WE LEARNED Take enough time to model new collections (bucketing) => Can decrease the storage volume when done right The use of cloud manager drastically decreases deployment and maintenance time => More time to think about your application There are endless ways of getting your data into mongoDB => Mongo supplies and support many di erent solutions
  52. 52. THANKS FOR LISTENING! Questions?

×