When it comes to the question: "Where do we actually store our application data?", we are spoilt for choice, especially when it comes to the major cloud providers. The simple and often completely valid answer is still the classic relational database! It is very suitable for many areas of application, as the technology is tried and tested and can cover a very broad spectrum. It is therefore not surprising that all major cloud providers offer this as a "managed service". For some years now, however, there have also been so-called cloud-native databases that have been specially developed for the requirements of the cloud. The big promise: "Infinite scalability" In a large customer project, we have been using such a database productively for over 4 years with Azure CosmosDB. The presentation will deal with the following questions, among others What does "upscalability" mean in practice ? What do you have to pay attention to when designing? What are the actual limits? What other special features do I get? When do I need a cloud-native database? But that's not all! We also look beyond Azure to the other two major cloud providers: AWS and Google Cloud. With DynamoDB and Datastore/Firestore, they have similar products on offer.

1. qaware.de
Fully-managed Cloud-native Databases: The
path to indefinite scale
Dirk Kröhan
dirk.kroehan@qaware.de

2. 2
Dirk Kröhan
Software-Architect
#qaware #mainz
#cloudnative #spring
#cleanarchitecture
QAware

3. 3
The history of rail gauges: https://parovoz.com/spravka/gauges-standard.html
What does a horse have to do with a space shuttle?
QAware

4. 4
Where do we actually store our application data?
QAware

5. MagentaTV Voice @ Deutsche Telekom
5
QAware

6. High-level architecture
6
CVI Event-History BI Analytics
Invoke API History API
WRITE
READ
READ
QAware

7. 7
QAware

8. 8
We want a scalable NoSQL database and
preferably fully managed!
QAware

9. 9
Runs
on
Isn't there something provided by Azure?
QAware

10. Azure Cosmos DB

11. Azure Cosmos DB
11
Azure Cosmos DB is a globally distributed, multi-model database service offered by Microsoft Azure. It is
designed to provide high availability, scalability, and low-latency access to data for mission-critical
applications.
Cosmos DB is a NoSQL database, which means it can handle unstructured and semi-structured data types.
Cosmos DB supports multiple data models, including document, key-value, column-family, and graph, making
it suitable for a wide range of use cases.
QAware

12. Globally distributed
12
Quelle: Azure Portal
QAware

13. Multi-Model
13
■ API for NoSQL (native)
■ API for MongoDB
■ API for PostgreSQL (new)
■ API for Apache Cassandra
■ API for Apache Gremlin
■ API for Table
👍
QAware

14. Consistency Levels
14
https://learn.microsoft.com/en-us/azure/cosmos-db/consistency-levels#consistency-levels-and-latency
QAware

15. High Availability, Scalability, Low Latency
15
■ SLA with 99,999 % read availability
■ Scalability via guaranteed throughput
– Manual scaling
– Automatic instant scaling based usage
■ Low Read-/Write-Latency
– Guaranteed < 10ms at 99th percentile
– Average Read-Latency ~4 ms (read) / 5 ms (write) at 50th percentile
https://uptime.is/99.999
QAware

16. Request Unit as scaling and throughput unit
16
A point read for a 1-KB Item costs exactly 1
Request Unit.
point read: Reading a single Item per ID and
Partition-Key value
QAware

17. Scaling based on provisioned throughput
17
■ Throughput on database level
– Shared throughput across all containers
– No throughput guarantees for a specific container
■ Throughput on container level
– Exclusive throughput for a single container
If a database operation requires more RUs than the currently available throughput, rate limiting applies, i.e. the request is
rejected with a 429 response. The response contains hints as to when the request should be repeated.
The required RU throughput is always linked to the underlying physical partition.
–> Hot Partition
QAware

18. Indefinite scale?

19. 19
SCALE
LIKE A BOSS
QAware

20. 20
Quelle: https://tenor.com/view/yaburnt-joker-burn-burnward-calltheburnunit-gif-5274995
QAware

21. High scalability in the real world

22. Indefinite scaling!?
22
¹ You can increase Maximum RUs per container or database by filing an Azure support ticket.
QAware

23. Yes, but you have to know what you're doing!
23
Quelle:https://cdn.prod.www.spiegel.de/images/6c9a2605-8b97-4545-a815-926da609c698_w1600_r1.5_fpx58_fpy55.01.jpg
vs.
Quelle: https://de.wikipedia.org/wiki/Kombinationskraftwagen
QAware

24. Data modeling and partitioning is the key!
24
■ Scalability relies on the partitioning of data
■ Choosing the right partition keys is essential
■ A partition key can not be changed
■ The partition key defines a logical partition
■ Goal: All database operations should hit one
logical partition
QAware

25. Examples from our project

26. High level architecture
26
Azure
CosmosDB
Container: event
Touchpoints
APIs
ETL APIs
PDM APIs
Customer-
Care APIs
VLP APIs
QAware
Get cards
Get conversations

27. Cons
27
■ Event container stores two different types of items
■ Touchpoint APIs require only a small portion of the whole JSON data
– Get single card by ID
– Get cards paginated by size
– Get conversations paginated by time + various filter criteria
– Vote a single conversation
– Delete bulk
– Delete all + by filter criteria
■ Vote requires an update of an existing conversation item
■ Delete requires an update of an existing conversation to set a delete flag and to set sensitive data to NULL
QAware

28. Cons TL;DR
28
■ Database queries are expensive in terms of required database throughput
■ Database must be scaled higher to provide sufficient throughput
■ Higher code complexity
QAware

29. Goal: Separation of concerns
29
■ Store data that is needed from the customer's perspective separately from the data stored for analysis and debugging purposes
■ Reduce complexity
– Simplify the deletion process: Delete data that the customer sees immediately, delete remaining data asynchronously
QAware

30. Azure
CosmosDB
Container: event
Container: user_conversation
Consumer APIs of the Event-History
30
Touchpoints
APIs
ETL APIs
PDM APIs
Customer-
Care APIs
VLP APIs
QAware

31. RU Throughput before
31
QAware

32. RU Throughput after
32
~90% RU throughput savings!!
QAware

33. Some of the features

34. TTL-based Deletion
34
■ Globally on container level
■ Individually per Item via ttl property
– Global TTL still has to be activated,
at least with “-1” (Items do not expire)
QAware

35. ChangeFeed
35
QAware

36. A look outside the box:
AWS DynamoDB & Google Firestore

37. AWS DynamoDB
37
■ “Fully managed NoSQL database service that provides fast and predictable performance with seamless scalability”
■ Provides a single own API
■ Scaling is also based on provisioned throughput
– Autoscaling is possible
■ 2 Consistency Levels (eventually consistent, strongly consistent)
– Can be selected for each individual read access
■ Data is automatically replicated to multiple availability zones in an AWS region
– Global tables enable multi-region replication
QAware

38. AWS DynamoDB
38
■ Maximum item size: 400KB (CosmosDB 2MB)
■ Throughput is measured in read capacity unit (RCU) und write capacity unit (WCU)
– RCU: 1 strongly consistent / 2 eventually consistent read access per second for items up to 4KB
– WCU: 1 write access per second for items up to 1 KB
■ Throughput limits per partition: 3000 RCUs and up to 1000 WCUs
■ DynamoDB Streams is similar to CosmosDB ChangeFeed, but only for the last 24h. Contains all changes to an item including deletions!
■ Supports ACID transactions across multiple items, also across multiple tables
QAware

39. Google Firestore
39
■ “Serverless, maintenance-free document database that can be easily scaled to meet any need”
■ Automatic replication across multiple regions. Guaranteed availability of 99.999 %
■ Strong focus on App and Mobile development
– Live-synchronisation of changes
– Offline-Mode
■ Maximum item size: 1 MB
■ ACID transaction support
■ Automatic scaling
– Up to 1 million concurrent connections and 10000 writes/second
■ Pricing is mainly based on database operations (read, write, delete) plus some extra costs for bandwidth and storage
QAware

40. 1. + 2. Juli 2024
https://www.kcdmunich.de/

41. qaware.de
QAware GmbH Mainz
Rheinstraße 4 C
55116 Mainz
Tel. +49 6131 21569-0
info@qaware.de
twitter.com/qaware
linkedin.com/company/qaware-gmbh
xing.com/companies/qawaregmbh
slideshare.net/qaware
github.com/qaware