Cosmos DB is among the top databases, with its strengths being in a flexible, extremely scalable hosted model, high SLA, low latency, globally distributed, automatic indexing, 2-dimensional redundancy and granular access level. But how does it suit IoT scenarios and for what scenarios is it appropriate?

1. MVP Unplugged #6
Sofia, Mar 16th 2022
Azure Cosmos DB
and IoT Scenarios

2. IoT Scenarios
for Azure Cosmos DB

3. Databases for Time Series Workloads
• Writes
o 95%-99% of all operations
o Streaming live data from multiple devices
o Typically sequential appends
• Updates to modify values are rare
• Deletes are bulk on large ranges (days, months, years)
• Queries
o Typically sequential
o Concurrent reads are common
• Performance issues are typically I/O bound
o Caching does not work well for BigData
o Systems are typically distributed by design
Credits: Baron Schwartz

4. Cosmos DB
• Globally distributed, horizontally scalable, multi-model database service
Database Model DB Engines Rank
Document store #3
Key-Value store #3
Wide column store #3
Graph #2

5. Azure Reference Time Series Solution
• Real-time Data Capture Layer
o IoT Hub
o Event Hub
o Kafka for HDInsight
• Stream Processing Layer
• Analytical Data Layer
o Azure Cosmos DB
o Azure Data Lake
o Azure Time Series Insights
• Cosmos DB
o Hot store at a cost and SLA
• TSI
o Cold store at a budget

6. IoT Reference (Use Case 1)
ExxonMobil (2021): Monitoring of 50K dispersed oil field assets
Scenario: Store telemetry for fast storage, retrieval and analytics
Cosmos DB
• Hot store
• Fast and indexed
• 50’000 oil wells’ health tracked

7. IoT Reference (Use Case 2)
Toyota: Car assembly line and high availability requirements to command robots
Scenario: Device catalog with high availability for critical processes
Cosmos DB
• Low latency
• 99.999% global availability
• <10ms latency for read/write

8. Cosmos DB for IoT
Pros and Cons

9. CosmosDB Key Features for IoT
• PaaS - Hosted model, High availability
• Ingest semistructured data at very high rate (various JSON messages)
• Partitioning (H-scale)
o Determines how data are routed to partitions
o Pick a partition key during design phase (i.e. DeviceID, company, city) to load partitions evenly
• Scaling
o Request Unit (RU) = CPU + Memory + IOPs
o Scaling without downtime
o Option to autoscale within maximum throughput (i.e. 500-5000 RU/s)
• Retention Policy (TTL)
o Automatically delete items after some time(seconds) using left over RUs in the background (for free)
• Change Feed
o Monitors CosmosDB for changes, outputs sorted list of changed documents

10. CosmosDB SQL API
• SQL / Core API
o Point Read (1KB=1 RUs, 10KB=10RUs)
o SELECT (2.3+ RUs) incl. Subqueries and Joins
o INSERT / DELETE (1KB=5 RUs, 100KB = 50RUs)
o UPDATE (1KB=10 RUs)
o All queries could have parameters
o Enable .NET SDK Metrics for real cost
• Indexing; Auto Indexing; User Defined Functions
o Impacts RU costs for INSERT/UPDATE/DELETE
• Aggregations
o AVG, COUNT, MIN, MAX, SUM
• Integrated Cache (€270+ / month)
o Read-through, write-through cache with a Least Recently Used (LRU) eviction policy.
o Requires dedicated gateway (D4/D8/D16)

11. CosmosDB Limitations for IoT Scenarios
• Analytical functions
o integral, sdtdev, forecasting, derivative, moving average, exponential MA, difference, covariance, histogram
• Cost
o RU (Request Unit) based, depends on consistency level, size of document, indexes
o Difficult to predict (despite a capacity calculator tool)
o Free Tier is available; Cost alerts are a must
• 20GB partition limit
• High rate of DB filling up
o 1KB document / second / device
o 1 device = 2.5GB per Month
• No Downsampling; Scheduled jobs with AZ functions
• Hot vs Cold Store
o Store old data to cold (BLOB) storage or TSI
• Indexing is both a curse and a blessing

12. Takeaways
• Reference Use Cases
https://customers.microsoft.com/en-us/story/exxonmobil-mining-oil-gas-azure
https://docs.microsoft.com/en-us/azure/cosmos-db/use-cases
• Azure Cosmos DB for IoT Solutions
https://techcommunity.microsoft.com/t5/internet-of-things-blog/iot-solutions-and-azure-cosmos-db/ba-p/1015605
https://docs.microsoft.com/en-us/azure/architecture/solution-ideas/articles/iot-using-cosmos-db
• Cosmos DB RUs Optimization
https://medium.com/@thomasweiss_io/how-i-learned-to-stop-worrying-and-love-cosmos-dbs-request-units-92c68c62c938
• Developer Resources, Architecture, Tools, Media
https://developer.azurecosmosdb.com/
• Cosmos DB Certification
https://docs.microsoft.com/en-us/learn/certifications/azure-cosmos-db-developer-specialty/
• Azure Samples
https://github.com/Azure-Samples

13. Upcoming Events
Azure Bootcamp Bulgaria 2022
May 14 (online)