Proximus is one of the biggest Telecom companies in the Belgian market. This year the company began developing a new IoT network using LoRaWan technology. The talk will detail our development team’s search for a database suited to meet the needs of our IoT project, the selection and implementation of MongoDB as a database, as well as well as how we built a system for storing a variety of sensor data with high throughput by leveraging sleepy.mongoose. The talk will also discuss how different decisions around data storage impact applications in regards to both performance and total cost.
MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company
ENABLING IOT AT
HI, MY NAME IS DAVE.
IN THIS SESSION
How to implement mongoDB in a IoT M2M use case.
Data modelling is key when storing time series data.
How to insert data into mongoDB using Rest interface.
How to store data while keeping storage amount minimal?
What storage engine to choose?
How to store my data using a third party system?
How can I easily deploy my mongoDB solution?
How do we model our data?
De ning storage engine
Setting up your mongo Instances
How do we insert our data in mongoDB?
How do we query our data?
SETTING THE SCENE
Mythings Project at Proximus
What is IoT?
MYTHINGS DEVELOPMENT TEAM
Mongo experience: 1,5 year
daily document size/sensor =
SENSORHISTORY COLLECTION SIZING
Document size = 3.98 KB
Sensor will send 48 time a day
On average 3 containers/sensor
3.98 KB * 1 document * 3 = 11,9 KB
1 M sensors => 11.34 GB/day
Use if possible (depending on use-case)
Concurrent updates on the bucket document may lead to queue
Do not overdue it!
Collection size will be smaller
Indexes size will be considerable reduced
Reads will be faster
! Maximum document size 16MB !
DEFINING STORAGE ENGINE
Keep storage volume to a minimum
Read and write simultaniously to collection
DEFINING STORAGE ENGINE
Document level lock instead of collection level
Data is stored in a compressed format
Index size is smaller
Matched the application requirements
WiredTiger was recommended by MongoDB
Sleepy Mongoose (Python)
Rest Heart (Java)
latest feature support
Advised to change to Rest Heart by MongoDB
Latest feature support
Uses PATCH method for updates (Upsert)
=> Not supported current release
Not mature enough: no bucketing for upsert!
Back to square one!
Interchange Rest heart <> Sleepy Mongoose
Easy transition (1-2 days development)!
Query framework :
Aggregation framework :
-> fast for retrieving data
-> data aggregations that take more time
read from secondary
$ curl -X GET 'http://localhost:27080/foo/bar/_find
Multitenant system: company id
Time based data: date
Data retention policy:
-> Deterministic way to drop documents once expired
-> Fixed size collections
CC Not suitable because time range of documents will depend on
(high throughput will decrease the time range stored).
CAPPED COLLECTION VS TTL INDEXES
GETTING RID OF THE EXPIRED DATA
There is more data than one machine can hold on its drives
The application is write heavy and you could experience too much
The working set outgrows the memory can be allocated to a single
WHAT WE LEARNED
Take enough time to model new collections (bucketing)
=> Can decrease the storage volume when done right
The use of cloud manager drastically decreases deployment and
=> More time to think about your application
There are endless ways of getting your data into mongoDB
=> Mongo supplies and support many di erent solutions