Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)
•
3 likes•1,884 views
Presenter: Tim Berglund, Senior Director of Developer Experience, Confluent It has become a truism in the past decade that building systems at scale, using non-relational databases, requires giving up on the transactional guarantees afforded by the relational databases of yore. ACID transactional semantics are fine, but we all know you can’t have them all in a distributed system. Or can we? In this talk, I will argue that by designing our systems around a distributed log like Apache Kafka®, we can in fact achieve ACID semantics at scale. We can ensure that distributed write operations can be applied atomically, consistently, in isolation between services, and of course with durability. What seems to be a counterintuitive conclusion ends up being straightforwardly achievable using existing technologies, as an elusive set of properties becomes relatively easy to achieve with the right architectural paradigm underlying the application.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)
Similar to Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®) (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)
- 1. Dissolving Problem the @tlberglund (making an ACID-compliant database out of Apache Kafka®)
- 3. What is a Database?
- 4. A program that remembers things.
- 5. A program that remembers things and has a data model.
- 6. A program that remembers things and has a data model and ACID transactional properties.
- 9. Durability
- 12. Broker 4Broker 3Broker 2Broker 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 4 Topic 1 Partition 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 3 Topic 1 Partition 4 Topic 1 Partition 4 Topic 1 Partition 4
- 13. Atomicity
- 14. Database Transaction BEGIN; UPDATE account SET balance += 100 WHERE username = tlberglund; UPDATE account SET balance -= 100 WHERE username = gwenshap; COMMIT;
- 16. Isolation
- 17. process 1 process 2 Is there a Tim? nope Is there a Tim? Awesome, make a Tim nope No prob! Okay, hoss Cool, make a Tim
- 18. process 1 process 2 Is there a Tim? nope Is there a Tim? Awesome, make a Tim nope No prob! Okay, hoss Cool, make a Tim
- 19. Consistency
- 20. Consistency • Invariants/constraints • Unique usernames • Account balances greater than zero
- 22. What is Kafka?
- 23. Topics
- 24. K V
- 25. K V
- 26. K V
- 27. K V
- 28. K V
- 29. K V
- 30. K V
- 31. K V • Log of events • Strict ordering guarantee • Constant-time reads and writes • Persistent on disk
- 32. Partitioning
- 33. ………
- 34. … … … Partition 0 Partition 1 Partition 2 K V Run the key through a hash function
- 35. … … … Partition 0 Partition 1 Partition 2 K V
- 36. … … … Partition 0 Partition 1 Partition 2 K V
- 37. … … … Partition 0 Partition 1 Partition 2 K V
- 38. … … … Partition 0 Partition 1 Partition 2 K V
- 39. • Provides scalable writes, storage, and consumption • Ordering is within partition only • Key selection becomes a data modeling concern … … …
- 40. Replication
- 41. Broker 4Broker 3Broker 2Broker 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 4
- 42. Broker 4Broker 3Broker 2Broker 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 4 Topic 1 Partition 1 Topic 1 Partition 1
- 43. Broker 4Broker 3Broker 2Broker 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 4 Topic 1 Partition 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 2
- 44. Broker 4Broker 3Broker 2Broker 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 4 Topic 1 Partition 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 3
- 45. Broker 4Broker 3Broker 2Broker 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 4 Topic 1 Partition 1 Topic 1 Partition 1 Topic 1 Partition 2 Topic 1 Partition 2 Topic 1 Partition 3 Topic 1 Partition 3 Topic 1 Partition 4 Topic 1 Partition 4 Topic 1 Partition 4
- 46. Producers
- 47. … … … partition 0 partition 1 partition 2 Partitioned Topic producer
- 48. … … … partition 0 partition 1 partition 2 Partitioned Topic producer • A client application • Puts messages into topics • Handles partitioning, network protocol • Java, Go, .NET, C/C++, Python • Also every other language
- 49. Consumers
- 50. consumer A … … … partition 0 partition 1 partition 2 Partitioned Topic
- 51. consumer A consumer B … … … partition 0 partition 1 partition 2 Partitioned Topic
- 52. consumer A consumer B … … … partition 0 partition 1 partition 2 Partitioned Topic
- 53. consumer A consumer B … … … partition 0 partition 1 partition 2 Partitioned Topic consumer A
- 54. consumer A consumer B … … … partition 0 partition 1 partition 2 Partitioned Topic consumer A consumer A
- 55. consumer A consumer B … … … partition 0 partition 1 partition 2 Partitioned Topic consumer A consumer A • A client application • Reads messages from topics • Horizontally, elastically scalable (if stateless) • Java, Go, .NET, C/C++, Python, everything else
- 59. Kafka Connect
- 63. Connect Worker Connect Worker Connect Worker task task task task task
- 64. { “name": "jdbc_source_postgres_movies", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "connection.url": “jdbc:postgresql://database:5432/WORKSHOP…”, "table.whitelist": "movies", "mode": "incrementing", "incrementing.column.name": "id", "validate.non.null": "false", "topic.prefix": "postgres-" } }
- 66. • Declarative data integration framework • Extensive community library of connectors • Horizontally scalable and fault- tolerant • Pretty easy to extend Kafka Connect broker broker broker broker data source data sink Kafka Connect
- 67. Kafka Streams
- 68. consumer A consumer B … … … partition 0 partition 1 partition 2 Partitioned Topic consumer A consumer A
- 70. Turn streams into tables Enrich a stream with a table Aggregate streams Join one stream with another Scale stateful applications
- 71. Functional Java API Abstractions for streams and tables Scalable, fault-tolerant state
- 74. • Java API • Filter, join, aggregate, etc. • Locates stream processing with your application • Scales like a Consumer Group (but better!) KTable<Long, Movie> movies = builder.table(“movies”, Materialized. <Long, Movie,KeyValueStore< Bytes, byte[]>> as(“movies-store") .withValueSerde(movieSerde) .withKeySerde(Serdes.Long()) );
- 75. KSQL
- 76. CREATE TABLE movie_ratings AS SELECT title, SUM(rating)/COUNT(rating) AS avg_rating, COUNT(rating) AS num_ratings FROM ratings LEFT OUTER JOIN movies ON ratings.movie_id = movies.movie_id GROUP BY title;
- 78. • Declarative stream processing language • Provides stream and table abstractions • Filter, join, aggregate • Run on horizontally scalable KSQL cluster CREATE TABLE movie_ratings AS SELECT title, SUM(rating)/COUNT(rating) AS avg_rating, COUNT(rating) AS num_ratings FROM ratings LEFT OUTER JOIN movies ON ratings.movie_id = movies.movie_id GROUP BY title;
- 80. Durability
- 81. consumer A consumer B … … … partition 0 partition 1 partition 2 Partitioned Topic consumer A consumer A
- 82. Atomicity
- 93. Isolation
- 94. process 1 process 2 Is there a Tim? nope Is there a Tim? Awesome, make a Tim nope No prob! Okay, hoss Cool, make a Tim
- 95. process 1 process 2 I would like a Tim I would like a Tim Tim-1 Tim-2
- 98. process 1 process 2 Users Ale Yeva Vik Tim yesIs there a Tim? Is there a Tim? yes
- 99. Consistency
- 102. SQL
- 103. Tabular Model
- 104. Storage Engine
- 105. Commit Log
- 106. You are not just writing microservices.
- 107. You are building an inside-out database
- 108. with ACID semantics
- 109. and that is a good thing.
- 111. 25%OFF Standard Priced Conference Pass Confluent Community Discount Code KS19Meetup