The document discusses change data capture (CDC) in distributed systems. CDC allows capturing changes to data and notifying actors so they can react accordingly. Some considerations for CDC include reliability, scalability, performance, consistency, and fault tolerance. Common strategies involve using timestamps, triggers, diffs, or log scraping. CDC has various use cases such as caching, search indexing, offline processing, and simplifying monolithic applications. Popular frameworks for implementing CDC include pg2k4j, Maxwell, AWS Database Migration Service, SpinalTap, DataBus, and Debezium.

3. Change Data Capture in
Distributed Systems
@petr_zapletal

4. Change Data Capture in
Distributed Systems
@petr_zapletal

6. CDC is a design pattern that enables
capturing changes to data and notifying
actors so they can react accordingly

7. Things to Consider

8. Things to Consider
● Reliability

9. Things to Consider
● Reliability
● Scalable

10. Things to Consider
● Reliability
● Scalable
● Performant

11. Things to Consider
● Reliability
● Scalable
● Performant
● Consistent

12. Things to Consider
● Reliability
● Scalable
● Performant
● Consistent
● Fault Tolerant

13. Strategies

14. Strategies
● Timestamps or version
numbers

15. Strategies
● Timestamps or version
numbers
● Triggers

16. Strategies
● Timestamps or version
numbers
● Triggers
● Diffs

17. Strategies
● Timestamps or version
numbers
● Triggers
● Diffs
● Log scraping

18. Polling

19. Server
override def routes: Routes = {
case GET(p"/v1/rules/state") =>
DB.fetchAllRules
.transact(readTx.tx)
.map { rs =>
...
}

20. Server
override def routes: Routes = {
case GET(p"/v1/rules/state") =>
DB.fetchAllRules
.transact(readTx.tx)
.map { rs =>
...
}

21. Server
override def routes: Routes = {
case GET(p"/v1/rules/state") =>
DB.fetchAllRules
.transact(readTx.tx)
.map { rs =>
...
}

22. Client
val url = URLHelper
.parseAbsoluteUrl(target.base +
“/v1/rules/state”)
val response = client
.get(url, Map.empty,
Map.empty, timeout)
updateState(response)

23. Client
val url = URLHelper
.parseAbsoluteUrl(target.base +
“/v1/rules/state”)
val response = client
.get(url, Map.empty,
Map.empty, timeout)
updateState(response)

24. Client
val url = URLHelper
.parseAbsoluteUrl(target.base +
“/v1/rules/state”)
val response = client
.get(url, Map.empty,
Map.empty, timeout)
updateState(response)

25. Client
val url = URLHelper
.parseAbsoluteUrl(target.base +
“/v1/rules/state”)
val response = client
.get(url, Map.empty,
Map.empty, timeout)
updateState(response)

26. Log Scraping

27. Schema
CREATE TABLE dynamo_source (
id BIGSERIAL PRIMARY KEY,
rules VARCHAR(255) NOT NULL,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
);

28. Schema
CREATE TABLE dynamo_source (
id BIGSERIAL PRIMARY KEY,
rules VARCHAR(255) NOT NULL,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
);

29. Source
val writeCols =
fr"id, rules, created_at, updated_at"
val writeValues =
fr"$id, $rules, $createdAt, $updatedAt"
(fr"""INSERT INTO
dynamo_source(""" ++ writeCols ++ fr""")
VALUES(""" ++ writeValues ++ fr""")
RETURNING id""").update

30. Source
val writeCols =
fr"id, rules, created_at, updated_at"
val writeValues =
fr"$id, $rules, $createdAt, $updatedAt"
(fr"""INSERT INTO
dynamo_source(""" ++ writeCols ++
fr""")
VALUES(""" ++ writeValues ++ fr""")
RETURNING id""").update

31. DMS Mapping Samples
{ "rule-type": "selection",
"object-locator": {
"schema-name": "sql-schema-somewhere",
"table-name": "dynamo_source" }, },
{ "rule-type": "object-mapping",
"rule-name": "TransformToDynamo",
"rule-action": "map-record-to-record",
"target-table-name": "dynamo-db-table",
...

32. DMS Mapping Samples
{ "rule-type": "selection",
"object-locator": {
"schema-name":
"sql-schema-somewhere",
"table-name": "dynamo_source" }, },
{ "rule-type": "object-mapping",
"rule-name": "TransformToDynamo",
"rule-action": "map-record-to-record",
"target-table-name": "dynamo-db-table",
...

33. DMS Mapping Samples
{ "rule-type": "selection",
"object-locator": {
"schema-name": "sql-schema-somewhere",
"table-name": "dynamo_source" }, },
{ "rule-type": "object-mapping",
"rule-name": "TransformToDynamo",
"rule-action": "map-record-to-record",
"target-table-name": "dynamo-db-table",
...

34. Target
final case class DynamoItem(id: String, rules: String, createdAt:
Epoch, updatedAt: Epoch)
val table: Table[DynamoItem] = Table[DynamoItem](tableName)
override def get(key: String): IO[Option[DynamoItem]] = {
val op = table.get('id -> key)
IO.fromFuture(IO(ScanamoAsync.exec(client)(op)))(cs)
}

35. final case class DynamoItem(id: String, rules: String,
createdAt: Epoch, updatedAt: Epoch)
val table: Table[DynamoItem] = Table[DynamoItem](tableName)
override def get(key: String): IO[Option[DynamoItem]] = {
val op = table.get('id -> key)
IO.fromFuture(IO(ScanamoAsync.exec(client)(op)))(cs)
}
Target

36. final case class DynamoItem(id: String, rules: String, createdAt:
Epoch, updatedAt: Epoch)
val table: Table[DynamoItem] = Table[DynamoItem](tableName)
override def get(key: String): IO[Option[DynamoItem]] =
{
val op = table.get('id -> key)
IO.fromFuture(IO(ScanamoAsync.exec(client)(op)))(cs)
}
Target

37. Use cases

38. Use Cases
● Caching

39. Use Cases
● Caching
● Search Indexing

40. Use Cases
● Caching
● Search Indexing
● Offline Processing

41. Use Cases
● Caching
● Search Indexing
● Offline Processing
● Simplifying monolithic
applications

42. Use Cases
● Caching
● Search Indexing
● Offline Processing
● Simplifying monolithic
applications
● CQRS

43. Use Cases
● Caching
● Search Indexing
● Offline Processing
● Simplifying monolithic
applications
● CQRS
● ...

44. Frameworks

45. Frameworks
● pg2k4j

46. Frameworks
● pg2k4j
● Maxwell

47. Frameworks
● pg2k4j
● Maxwell
● AWS Database Migration
Service

48. Frameworks
● pg2k4j
● Maxwell
● AWS Database Migration
Service
● SpinalTap

49. Frameworks
● pg2k4j
● Maxwell
● AWS Database Migration
Service
● SpinalTap
● DataBus

50. Frameworks
● pg2k4j
● Maxwell
● AWS Database Migration
Service
● SpinalTap
● DataBus
● Debezium

51. Frameworks
● pg2k4j
● Maxwell
● AWS Database Migration
Service
● SpinalTap
● DataBus
● Debezium
● ...

52. Summary

53. Questions

54. References

55. References
● https://github.com/disneystreaming/pg2k4j
● https://aws.amazon.com/dms/
● https://github.com/airbnb/SpinalTap
● https://github.com/linkedin/databus
● https://github.com/zendesk/maxwell
● https://github.com/debezium/debezium