Serio? Kolejna prezentacja o Event Sourcingu? I tak, i nie. Zamiast na gloryfikacji tej jakże zacnej koncepcji, skoncentrujemy się na możliwych problemach przy jej implementacji. Co może wybuchnąć (często z opóźnionym zapłonem)? Gdzie i jak przechowywać eventy? Jak osiągnąć najwyższy poziom wtajemniczenia i skalowalności? Jak żyć z eventual consistency? I wiele innych pytań, z którymi spróbujemy się zmierzyć. Postaram się unikać jednoznacznych odpowiedzi - każde rozwiązanie jest dobre, dopóki “działa”. Przerywniki z doświadczeniami słuchaczy będą bardzo mile widziane.

1. Event Sourcing
- what could go wrong?
Andrzej Ludwikowski

2. About me
➔
➔ aludwikowski.blogspot.com
➔ github.com/aludwiko
➔ @aludwikowski

3. What is Event Sourcing?
DB
Order {
items=[itemA, itemB]
}

4. What is Event Sourcing?
DB
DB
Order {
items=[itemA, itemB]
}
ItemAdded(itemA)
ItemAdded(itemC)
ItemRemoved(itemC)
ItemAdded(itemB)

5. History

6. History
● 9000 BC, Mesopotamian Clay Tablets,
e.g. for market transactions

7. History
● 2005, Event Sourcing
“Enterprise applications that use Event Sourcing
are rarer, but I have seen a few applications (or
parts of applications) that use it.”

8. Why Event Sourcing?
● complete log of every state change
● debugging
● performance
● scalability

9. ES and CQRS
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models

10. ES and CQRS level 1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Transaction

11. ES and CQRS level 1
● Entry-level, synchronous & transactional event sourcing
● slick-eventsourcing

12. ES and CQRS level 1
+ easy to implement
+ easy to reason about
+ 0 eventual consistency
- performance
- scalability

13. ES and CQRS level 2
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
Transaction

14. ES and CQRS level 2
+/- performance
+/- scalability
- eventual consistency
- increased events DB load
- lags

15. ES and CQRS level 3
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
Transaction
event
bus

16. ES and CQRS level 3.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

17. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

18. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
At-least-once delivery

19. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

20. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

21. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

22. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

23. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

24. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
?

25. ES and CQRS level 3.2
Events
Client
Query Service
Data access
Commands
Queries
Read
modelRead
modelRead
models
Updater
event
bus
Command Service
Domain
Command Service
Domain
Command Service
Domain
Transaction
Sharded
Cluster

26. ES and CQRS level 3.x
+ performance
+ scalability
- eventual consistency
- complex implementation

27. ES and CQRS alternatives
● Change Capture Data (CDC) logging instead of message queue?
● message queue instead of DB?

28. ES implementation?
● custom
● library
● framework

29. ES from domain perspective
● commands, events, state
● 2 main methods on state
○ process(command: Command): List[Event]
○ apply(event: Event): State

30. ES from application perspective
● snapshotting
● fail-over
● recover
● debugging
● sharding
● serialization & schema evolution
● concurrency access
● etc.

31. import javax.persistence.*;
import java.util.List;
@Entity
public class Issue {
@EmbeddedId
private IssueId id;
private String name;
private IssueStatus status;
@OneToMany(cascade = CascadeType.MERGE)
private List<IssueComment> comments;
...
public void changeStatusTo(IssueStatus newStatus) {
if (this.status == IssueStatus.DONE
&& newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW
&& newStatus == IssueStatus.DONE) {
throw new RuntimeException(String.format("Cannot change issue status from %s to %s",
this.status, newStatus));
}
this.status = newStatus;
}
...
}

32. import org.axonframework.commandhandling.*
import org.axonframework.eventsourcing.*
@Aggregate(repository = "userAggregateRepository")
public class User {
@AggregateIdentifier
private UserId userId;
private String passwordHash;
@CommandHandler
public boolean handle(AuthenticateUserCommand cmd) {
boolean success = this.passwordHash.equals(hashOf(cmd.getPassword()));
if (success) {
apply(new UserAuthenticatedEvent(userId));
}
return success;
}
@EventSourcingHandler
public void on(UserCreatedEvent event) {
this.userId = event.getUserId();
this.passwordHash = event.getPassword();
}
private String hashOf(char[] password) {
return DigestUtils.sha1(String.valueOf(password));
}
}

33. import akka.Done
import com.lightbend.lagom.scaladsl.*
import play.api.libs.json.{Format, Json}
import com.example.auction.utils.JsonFormats._
class UserEntity extends PersistentEntity {
override def initialState = None
override def behavior: Behavior = {
case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] {
case (GetUser, ctx, state) => ctx.reply(state)
}.onReadOnlyCommand[CreateUser, Done] {
case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists")
}
case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] {
case (GetUser, ctx, state) => ctx.reply(state)
}.onCommand[CreateUser, Done] {
case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done))
}.onEvent {
case (UserCreated(name), state) => Some(User(name))
}
}
}

34. ES packaging

35. import java.time.Instant
import info.ludwikowski.es.user.domain.UserCommand.*
import info.ludwikowski.es.user.domain.UserEvent.*
import scala.util.{Failure, Success, Try}
final case class User private[domain] (userId: UserId,
createdAt: Instant,
name: String,
email: Email) {
def applyEvent(userEvent: UserEvent): Try[User] = ??? //pattern matching
def process(userCommand: UserCommand): Try[List[UserEvent]] = ??? //pattern matching
}
object User {
def from(u: UserCreated): User = User(u.userId, u.createdAt, u.name, u.email)
}

36. ES packaging
● snapshotting
● fail-over
● recover
● debugging
● sharding
● serialization & schema evolution
● concurrency access
● etc.

37. ES packaging
● domain logic
● domain validation
● 0 framework/library imports

38. library vs. framework
● Akka Persistence vs. Lagom

39. Event storage
● file
● RDBMS
● Event Store
● MongoDB
● Kafka
● Cassandra

40. Event storage for Akka Persistence
● file
● RDBMS
● Event Store
● MongoDB
● Kafka
● Cassandra

41. akka-persistence-jdbc trap
val theTag = s"%$tag%"
sql"""
SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber",
"#$message", "#$tags"
FROM (
SELECT * FROM #$theTableName
WHERE "#$tags" LIKE $theTag
AND "#$ordering" > $theOffset
AND "#$ordering" <= $maxOffset
ORDER BY "#$ordering"
)
WHERE rownum <= $max"""

42. akka-persistence-jdbc trap
SELECT * FROM events_journal
WHERE tags LIKE ‘%some_tag%’;

43. Cassandra perfect for ES?
● partitioning by design
● replication by design
● leaderless (no single point of failure)
● optimised for writes (2 nodes = 100 000 tx/s)
● near-linear horizontal scaling

44. ScyllaDB ?
● Cassandra without JVM
○ same protocol, SSTable compatibility
● C++ and Seastar lib
● 1,000,000 IOPS
● not fully supported by Akka Persistence

45. Event serialization
● plain text
○ JSON
○ XML
○ YAML
● binary
○ java serialization
○ Avro
○ Protocol Buffers (Protobuf)
○ Thrift
○ Kryo

46. Plain text Binary
human-readable deserialization required

47. Plain text Binary
human-readable deserialization required
problems with precision (JSON IEEE 754) -

48. Plain text Binary
human-readable deserialization required
problems with precision (JSON IEEE 754) -
storage consumption compress

49. Plain text Binary
human-readable deserialization required
problems with precision (JSON IEEE 754) -
storage consumption compress
slow fast

50. Plain text Binary
human-readable deserialization required
problems with precision (JSON IEEE 754) -
storage consumption compress
slow fast
poor schema evolution support full schema evolution support

51. Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

52. Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

53. Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

54. Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

55. Multi-language support
● Avro
○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala
● Protocol Buffers (Protobuf)
○ even more than Avro

56. Speed
https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki

57. Size
https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki

58. Full compatibility
Application
Events
● backward - V2 can read V1
V1
V2

59. ● forward - V2 can read V3
Full compatibility
Application
Events
V1, V2
V2
Application
Application
V2
V3

60. Schema evolution - full compatibility
Protocol Buffers Avro
Add field + (optional) + (default value)
Remove field + + (default value)
Rename field + + (aliases)
https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

61. Protobuf schema management
//user-events.proto
message UserCreatedEvent {
string user_id = 1;
string operation_id = 2;
int64 created_at = 3;
string name = 4;
string email = 5;
}
package user.application
UserCreatedEvent(
userId: String,
operationId: String,
createdAt: Long,
name: String,
email: String
)

62. Protobuf schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
package user.application
UserCreatedEvent(
userId: String,
operationId: String,
createdAt: Long,
name: String,
email: String
)

63. Protobuf schema management
● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated
● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent

64. Protobuf schema management
+ clean domain
- a lot of boilerplate code

65. Avro schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}

66. Avro deserialization
Bytes Deserialization Object
Reader SchemaWriter Schema

67. Avro writer schema source
● add schema to the payload
● custom solution
○ schema in /resources
○ schema in external storage (must be fault-tolerant)
● Schema Registry

68. Avro schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}

69. Protocol Buffers vs. Avro
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}
message UserCreatedEvent {
string user_id = 1;
string operation_id = 2;
int64 created_at = 3;
string name = 4;
string email = 5;
}

70. Avro schema management
+ less boilerplate code
+/- clean domain
- reader & writer schema distribution

71. Memory consumption

73. Immutable vs. mutable state?
● add/remove ImmutableList 17.496 ops/s
● add/remove TreeMap 2201.731 ops/s

75. Fixing state
● healing command

76. Updating all aggregates
User(id)Command(user_id) Event(user_id)Event(user_id)Event(user_id)

77. Event + seq_noEvent + seq_no
Handling duplicates
Events
Read
modelRead
modelRead
models
Updater
event
bus
Event + seq_no
At-least-once delivery

78. Broken read model
Events
ad model
ead model
Read
models
Updater
event
bus

79. Broken read model
Events
ad model
ead model
Read
models
Updater
event
bus
read model update + offset

80. ES and CQRS level 3.2
Events
Client
Query Service
Data access
Commands
Queries
Read
modelRead
modelRead
models
Updater
event
bus
Command Service
Domain
Command Service
Domain
Command Service
Domain
Transaction
Sharding
Clustering

81. Cluster = split brain
1
5 4
3
Load balancer
2

82. Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)

83. Cluster = split brain
1
5 4
3
Load balancer
2
User(1)

84. Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)
User(1)

85. Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)
User(1)
Command(1)

86. Cluster best practises
● very good monitoring & alerting
● a lot of failover tests
● cluster also on dev/staging
● keep it as small as possible (code base, number of nodes, etc.)

87. Summary
● understand event/state schema evolution
● eventual consistency is your friend
● scaling is complex
● log-based processing mindset

89. WE
WANT
YOU

90. About me
➔
➔ aludwikowski.blogspot.com
➔ github.com/aludwiko
➔ @aludwikowski