1.
ENABLING IOT AT
PROXIMUS

2.
HI, MY NAME IS DAVE.
Software Engineer
Ordina Belgium
dave.degroote@ordina.be

3.
IN THIS SESSION
How to implement mongoDB in a IoT M2M use case.
Data modelling is key when storing time series data.
How to insert data into mongoDB using Rest interface.

4.
INITIAL QUESTIONS
How to store data while keeping storage amount minimal?
What storage engine to choose?
How to store my data using a third party system?
How can I easily deploy my mongoDB solution?

5.
ESSENTIAL PROBLEMS
How do we model our data?
De ning storage engine
Setting up your mongo Instances
How do we insert our data in mongoDB?
How do we query our data?

6.
SETTING THE SCENE
Mythings Project at Proximus
What is IoT?
LoRa Technology

7.
2 Front-end
3 Back-end
MYTHINGS DEVELOPMENT TEAM
Small team:
Mongo experience: 1,5 year

8.
MYTHINGS AT PROXIMUS

9.
MYTHINGS AT PROXIMUS

10.
INTERNET OF THINGS
Very broad, not contained to a single technology.
Not what but HOW !
Internet is the keyword
Collecting, analyzing and above all communicating data.

11.
LORA TECHNOLOGY

12.
LORA TECHNOLOGY VS ...

13.
LORA ARCHITECTURE

14.
SENSORS
Any type is possible as long as it is connected with a LoRa chip
CO2, PIR, Humidity, Luminance, Temperature, Pressure, …

15.
SENSORS
multiple containers per sensor possible

16.
LORA ALLIANCE
https://www.lora-alliance.org

17.
ESSENTIAL PROBLEMS
How do we model our data?
De ning storage engine
Setting up your mongo Instances
How do we insert our data in mongoDB?
How do we query our data?

18.
DATA MODELING
SENSOR HISTORY
company info
sensor info
container info
noti cation value
noti cation timestamp

19.
SENSORHISTORY COLLECTION
{
"_id": ObjectID("5613d33401e101c27f8e668d"),
"company": {
"name": "Proximus",
"companyId": 2
},
"container": {
"container": "0x0050.0x0006.0.m2m",
"type": "temperature",
"containerId": 597,
"name": "temperature"
},
"sensor": {
"sensorId": 773,
"mac": "F03D291000001301"
},
"data":
{
"type": "bool",
"ts": 1444139821000,
"value": "1"
}
}

20.
Daily document size/sensor =
SENSORHISTORY COLLECTION SIZING
Document size = 496 Bytes
Sensor will send 48 time a day
On average 3 containers/sensor
496 B * 48 documents * 3 = 71,4 KB
71,4 KB
1 M sensors => 68.09 GB/day

21.
Document with 3 additions:
BUCKETING TECHNIQUE
Data array (data values)
Count metadata (Array size)
Date eld (day timestamp)
Use in combination with Upsert mechanism!

22.
SENSORHISTORY COLLECTION (BUCKETING)
{
"_id": ObjectID("5613d33401e101c27f8e668d"),
"company": {
"name": "Proximus",
"companyId": 2
},
"container": {
"container": "0x0050.0x0006.0.m2m",
"type": "temperature",
"containerId": 597,
"name": "temperature"
},
"date": 1444086000000,
"sensor": {
"sensorId": 773,
"mac": "F03D291000001301"
},
"count": 48,
"data": [
{
"type": "bool",
"ts": 1444139821000,
"value": "1"
},
{
"type": "bool",
"ts": 1444152754000,
"value": "1"
},...
]
}

23.
daily document size/sensor =
SENSORHISTORY COLLECTION SIZING
(BUCKETING)
Document size = 3.98 KB
Sensor will send 48 time a day
On average 3 containers/sensor
3.98 KB * 1 document * 3 = 11,9 KB
11,9 KB
1 M sensors => 11.34 GB/day

24.
Use if possible (depending on use-case)
vs
DATA MODELING
BUCKETING
11GB/day 68GB/day

25.
Concurrent updates on the bucket document may lead to queue
operations
Do not overdue it!
DATA MODELING
BUCKETING
Advantages:
Collection size will be smaller
Indexes size will be considerable reduced
Reads will be faster
Remarks:
! Maximum document size 16MB !

26.
DEFINING STORAGE ENGINE
APPLICATION REQUIREMENTS:
Write intensive
Keep storage volume to a minimum
Read and write simultaniously to collection

27.
DEFINING STORAGE ENGINE
WIREDTIGER 3.2
Document level lock instead of collection level
Data is stored in a compressed format
Index size is smaller
Matched the application requirements
WiredTiger was recommended by MongoDB

28.
SETTING UP MONGODB INSTANCES
CLOUD MANAGER

29.
Once automation agent is installed, smooth sailing!
SETTING UP YOUR MONGODB
INSTANCES
CLOUD MANAGER
Easy setup
Easy management
Metrics

30.
INSERTING DATA
MYTHINGS ARCHITECTURE

31.
INSERTING DATA
REST INTERFACE
Sleepy Mongoose (Python)
https://github.com/mongodb-labs/sleepy.mongoose
Rest Heart (Java)
latest feature support
http://restheart.org/
Advised to change to Rest Heart by MongoDB

32.
Latest feature support
Uses PATCH method for updates (Upsert)
=> Not supported current release
Not mature enough: no bucketing for upsert!
INSERTING DATA
REST HEART

33.
Back to square one!
INSERTING DATA
SLEEPY MONGOOSE

34.
Interchange Rest heart <> Sleepy Mongoose
Easy transition (1-2 days development)!
INSERTING DATA
SLEEPY MONGOOSE

35.
Simple insert:
Advanced upsert:
INSERTING DATA
$ curl --data 'docs=[{"x":2},{"x":3}]' 'http://localhost:27080/foo/bar/_insert'
$ curl --data
'criteria=
{
"sensor":{"sensorId":1533,"name":"Binary mesurement","mac":"020000FFFF00B151"},
"company":{"companyId":134,"name":"Opinum"},
"container":{"containerId":1223,"name":"Door counter"},
"date":1456354800000
},
"count":{"$lt":100}
&newobj={
"$push":{"data":{"ts":1456403435000,"value":"456","type":"int"}},
"$inc":{"count":1},
"$setOnInsert":{"location":{"latitude":"-1","longitude":"-1"}}}
&upsert=true'
'http://localhost:27080/foo/bar/_insert'

36.
QUERING DATA
Query framework
Aggregation framework

37.
Retrieve sensorhistory for a week for all companies:
AGGREGATION FRAMEWORK
"op" : "command",
"ns" : "sensoringlabs.sensorHistory",
"command" : {
"aggregate" : "sensorHistory",
"pipeline" : [{
"$match" : {
"company.companyId" : {
$in" : [,54,61,76,77,79,80,81,82,83,86,87,90,122,123
,124,125,126,127,128,129,130,131,132,133,134,135,136,
137,138,146,147,148,149,150,151,152,153,154]
}
}
},
{
"$match" : {
"date" : {"$gte" : NumberLong("1457049600000")},
"$and" : [{"date" : {"$lte" : NumberLong("1457568000000")}}]
}
},
{"$skip" : NumberLong(0)},
{"$limit" : NumberLong(100)}
], "cursor" : {}
}
"responseLength" : 278888,
"protocol" : "op_command",
"millis" : 366

38.
Retrieve sensorhistory for a week for all companies:
QUERY FRAMEWORK
"op" : "query",
"ns" : "sensoringlabs.sensorHistory",
"query" : {
"find" : "sensorHistory",
"filter" : {
"company.companyId" : {
"$in" : [2,54,61,76,77,79,80,81,82,83,86,87,90,122,123,124,
125,126,127,128,129,130,131,132,133,134,135,136,137,
138,146,147,148,149,150,151,152,153,154]
},
"date" : {
"$gte" : NumberLong("1457049600000"),
"$lte" : NumberLong("1457568000000")
}
},
"limit" : 100,
"singleBatch" : false
}
"nreturned" : 100,
"responseLength" : 278888,
"protocol" : "op_command",
"millis" : 1

39.
Same search:
Query framework :
Aggregation framework :
QUERING DATA
1ms
360 ms

40.
-> fast for retrieving data
-> data aggregations that take more time
Read concern:
read from secondary
QUERING DATA
Query framework
Aggregation framework

41.
Query example:
QUERING DATA
SLEEPY MONGOOSE
$ curl -X GET 'http://localhost:27080/foo/bar/_find
?criteria={"_id":{"$oid":"4f8c6f05db61e2a72600001d"}}
&sort={"date”:-1}
&limit=1'

42.
INDEX STRATEGY
COMPOUND INDEXES
Multitenant system: company id
Time based data: date

43.
Data retention policy:
-> Deterministic way to drop documents once expired
-> Fixed size collections
CC Not suitable because time range of documents will depend on
throughput!
(high throughput will decrease the time range stored).
CAPPED COLLECTION VS TTL INDEXES
GETTING RID OF THE EXPIRED DATA
3 months
TTL index
Capped collection

44.
PUTTING IT ALL
TOGETHER

45.
Bucketing
DATA MODELING

46.
Wired tiger 3.2
STORAGE ENGINE

47.
Sleepy Mongoose
RestHeart (future)
Java mongo Driver
DATA INSERTION

48.
query framework
aggregation framework
DATA EXTRACTION

49.
compound indexes
TTL index
INDEXES

50.
SHARDING
WHEN
There is more data than one machine can hold on its drives
The application is write heavy and you could experience too much
latency
The working set outgrows the memory can be allocated to a single
machine

51.
CONCLUSION
WHAT WE LEARNED
Take enough time to model new collections (bucketing)
=> Can decrease the storage volume when done right
The use of cloud manager drastically decreases deployment and
maintenance time
=> More time to think about your application
There are endless ways of getting your data into mongoDB
=> Mongo supplies and support many di erent solutions

52.
THANKS FOR LISTENING!
Questions?