This document discusses building multi-tenant systems using Cosmos DB and NServiceBus. It explores options for tenant segregation in Cosmos DB, including using separate accounts, containers, or partition keys per tenant. Partitioning in Cosmos DB is also covered, emphasizing the importance of choosing a high-cardinality partition key. The change feed is demonstrated as a way to perform live migrations. Integrating NServiceBus with Cosmos DB for messaging and transactions is then explained, specifically around the use of the outbox pattern for exactly-once processing.

1. Building multi-tenant
systems using Cosmos DB
and NServiceBus
Ivan Lazarov

2. Cosmos DB account
• Set throughput (RUs)
• Geo-replication settings
• Define partition key
• Set throughput (RUs)
• Max 25 containers per DB
• Max 20GB per partition

3. Tenant segregation
u What is a tenant?
u Goals
u Balance between cost & isolation of tenants
u Avoid noisy neighbors
u Options
u Account per tenant
u Container per tenant (with dedicated throughput)
u Container per tenant (shared database throughput)
u Partition keys per tenant
u Combination of the above

4. Tenant segregation options
u Account per tenant
u Costly (minimum 400 RUs)
u Different geo-replication settings per tenant
u Different throughput per tenant

5. Tenant segregation options
u Container per tenant (with dedicated throughput)
u Costly (minimum 400 RUs)
u Different throughput per tenant
u Can group tenants based on regional needs

6. Tenant segregation options
u Container per tenant (shared database throughput)
u Same throughput sharing as partition key per tenant model below
u Unpredictable performance
u Noisy neighbor problem, throughput shared between tenants
u 25 container limit

7. Tenant segregation options
u Partition keys per tenant
u Most cost effective
u Does not necessarily mean all tenant data in one logical partition
u Noisy neighbor problem, throughput shared between tenants
u Cost per tenant marginal
u 20GB limit per partition

8. Tenant data isolation
u Tenant data isolation & apply the least privilege principle
u Using Cosmos resource tokens instead of master keys
u Token broker concept from Terry Duchastel (Citrix)
https://github.com/terry-
citrix/TokenBroker/blob/master/docs/TokenBrokerConcept.md
u Full repository (Java-based)
https://github.com/terry-citrix/TokenBroker

9. Partitioning
Source: https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview

10. Partitioning continued
u Partition key considerations
u High cardinality
u Try to spread RU consumption and data storage across all logical partitions
u Transactions in Cosmos can only be scoped to one logical partition
u Partition key as filter predicate for read-heavy containers – avoid cross-partition queries
u Partition key can include tenant ID, eg {tenantId}-{aggregateRootId}
u Disadvantages of partition key = item ID
u If you get your partition key wrong, you need to migrate to new collection with
the new partition key settings
u Live migration possible using change feed
u Change feed can be used for live migrating tenants to new "premium tier" like
container/database per tenant with dedicated throughput

11. Demo
Live migrations using change feed

12. Live migration resources
u How to change your partition key in Azure Cosmos DB (Theo van Kraay)
u Blog post
https://devblogs.microsoft.com/cosmosdb/how-to-change-your-partition-key
u Repository
https://github.com/Azure-Samples/azure-cosmosdb-live-data-migrator

13. NServiceBus
u We use NServiceBus to build our distributed systems and use Azure Cosmos DB
u NServiceBus allows us to use messaging patterns and not have transport-
specific concerns leak into our code
u Outbox & sagas very useful
u Useful tooling like Service Pulse and Service Insight
u Flexible recoverability policies

14. Data and messaging transactions
public class AcceptQuoteCommandHandler : IHandleMessages<AcceptQuoteCommand>
{
private readonly IRepository<Quote> _repository;
...
public async Task Handle(
AcceptQuoteCommand message,
IMessageHandlerContext context)
{
var quote = await _repository.Load(
message.QuoteNumber,
new PartitionKey(message.QuoteNumber));
quote.Accept();
await _repository.Upsert(quote);
await context.Publish(new QuoteAcceptedEvent(message.QuoteNumber));
}
}
public interface IRepository<TDocument>
{
Task<TDocument> Load(string id,
PartitionKey partitionKey);
Task Upsert(TDocument document);
}
public void Accept()
{
if (IsAccepted)
throw new Exception("...");
IsAccepted = true;
}
Boom!
No txn Transient failure

15. Data and messaging transactions
public class AcceptQuoteCommandHandler : IHandleMessages<AcceptQuoteCommand>
{
private readonly IRepository<Quote> _repository;
...
public async Task Handle(
AcceptQuoteCommand message,
IMessageHandlerContext context)
{
var quote = await _repository.Load(
message.QuoteNumber,
new PartitionKey(message.QuoteNumber));
quote.Accept();
await _repository.Upsert(quote);
await context.Publish(new QuoteAcceptedEvent(message.QuoteNumber));
}
}
public interface IRepository<TDocument>
{
Task<TDocument> Load(string id);
void Upsert(TDocument document);
}

16. Data and messaging transactions
public class AcceptQuoteCommandHandler : IHandleMessages<AcceptQuoteCommand>
{
private readonly IRepository<Quote> _repository;
...
public async Task Handle(
AcceptQuoteCommand message,
IMessageHandlerContext context)
{
var quote = await _repository.Load(
message.QuoteNumber);
quote.Accept();
_repository.Upsert(quote);
await context.Publish(new QuoteAcceptedEvent(message.QuoteNumber));
}
}
public interface IRepository<TDocument>
{
Task<TDocument> Load(string id);
void Upsert(TDocument document);
}
No DB write

17. Outbox
u Why does one need Outbox?
u De-duplication (exactly once processing)
u Outgoing communications can transact with business data
u What if we did not use Outbox?
u No exactly-once processing guarantee
u Need to make all operations idempotent
u Remember, transactions in Cosmos DB that can only be scoped to one logical
partition
u NServiceBus Cosmos DB persistence package brings support for outbox & sagas
u Outbox & saga records live in same logical partition

18. Outbox

19. Message partition key extraction
u For the NServiceBus Cosmos DB persistence to work, it needs a partition key
(and optionally container information)
u Extract partition key from:
u Message headers during transport receive stage
u Message headers or message body at logical stage
u NServiceBus Cosmos Persistence overview
https://docs.particular.net/previews/cosmosdb
u Transactional processing sample
https://docs.particular.net/samples/previews/cosmosdb/transactions

20. Demo
NServiceBus Cosmos DB Persistence & Outbox

21. Wrap up
u Tenant segregation strategies
u Cosmos DB partitioning
u Using the change feed to fix a mess, or to live migrate tenants
u Integrating NServiceBus with Cosmos DB
u Cosmos DB transactionality within handlers
Attribution for first slide image: Luzern Altstadt3 by Julia Leijola — Attribution-ShareAlike 3.0 Unported
(CC BY-SA 3.0)

22. Questions?