Akka is a library implementing the actor model on the JVM. It enables the development of distributed, concurrent, fault-tolerant and scalable applications. Akka persistence enables actors to persist their internal state so that it can be recovered where an actor is restarted. We demonstrate the practical application of Akka persistence to build a configuration datastore using CQRS and Event Sourcing

1. Using Akka Persistence to build
a configuration datastore
Anargyros Kiourkos – Software engineer @ CERN

2. Agenda
• Problem statement
• CRUD architecture, why not?
• Event sourcing & CQRS
• Actor model, Akka & Akka persistence
• Supervision & Testing
• Akka persistence in production

3. Problem statement
• Configuration management system for large industrial
control systems
• Thousands of devices, millions of datapoints and properties
• Complex equipment, devices with a lot of configuration data
• Full audit trail
• Historical data
• “Undo” functionality

4. Some numbers (one of these systems)
80 dc’s
21.000 devices
250.000 data-points
> 100.000.000 properties

5. CRUD, why not?
At least in this case…

6. CRUD architecture
User interface
Business layer
DAO/ORM
Database
• Usually 3-layers
• UI or any other client
• Business layer
• Data layer
• Perfectly OK for most situations
• But…

7. CREATE
Database
Datapoint
id:1
Hysteresis: 2.0F
INSERT
Datapoint
id:1
Hysteresis: 2.0F

8. UPDATE
Database
Datapoint
id:1
Hysteresis: 1.0F
UPDATE
Datapoint
id:1
Hysteresis: 2.0F
Datapoint
id:1
Hysteresis: 1.0F
Data
lost

9. Traditional solutions
• Auditing solutions (e.g. Hibernate Envers, AOP etc.)
• “Soft” deletes
• Valid solutions
• Not an inherent characteristic of the system, usually add-ons
• Separate architecture considerations
• Sometimes difficult to test
• Often, considerable amount of extra code

10. CRUD summary
• Data loss on every UPDATE and DELETE (maybe ok)
• Auditing is hard
• Locking contention
• Database is shared mutable state
• Object-relation impedance mismatch
• Compromise on READ-WRITE performance

11. Event sourcing to the rescue
Sometimes…

12. Event sourcing – what is it?
• Do not save current state
• Store the events that lead to current state
• Immutable events in an append-only storage ( event-log)
• Restore state by replaying events
• Full audit trail of every change in the system
• No mapping of objects to tables
• Allows to “retrofit” new functionality to past data

13. Commands and Events
• Command
• Command = Wish
• Do something => UpdateHysteresis
• Needs to be validated => ValidateHysteresis
• Can be rejected => InvalidHysteresis
• Event
• Event = Fact
• It has already happened => HysteresisUpdated
• Cannot be deleted

14. Event sourcing example - create
CreateDP
id:1
Hysteresis: 2.0F
Commands Validation Events
CreateDP
id:1
Hysteresis: 2.0F
DPCreated
id:1
Hysteresis: 2.0F
persist
DPCreated

15. Event sourcing example - update
UpdateHysteresis
id:1
Hysteresis: 1.0F
Commands Validation Events
UpdateHysteresis
id:1
Hysteresis: 1.0F
HysteresisUpdated
id:1
Hysteresis: 1.0F
persist
DPCreated
HysteresisUpdated

16. Event sourcing - considerations
• Read side, eventually consistent
• Event data cannot be updated, compensating events to
“undo”
• Schema evolution (more on this later)
• Serialization overhead
• Event consistency is vital
• Increased storage requirements
• Not a “silver bullet”

17. Quiz
The largest, event sourced system in world?

18. CQRS
Command Query Responsibility Segregation

19. How do you query an event log?
• Unrealistic to go through millions of records for a single read
• Events optimized for write operations & sequential reading

20. CQRS
• Segregate data operations:
• Read side => optimized for reading
• Write side => optimized for writing
• Essentially 2 or more separate domain models
• Read and Write sides of the application can be optimized &
scale independently
• No “one-size-fits-all” domain model
• Works well with ES, but not mandatory

21. CQRS without Event Sourcing (one approach)
Client
Write model Read model
Database
Data access layer

22. CQRS with Event Sourcing
Client
Write model Read model
Journal
Database
Events Projections

23. CQRS - Considerations
• Eventual consistency (depends on the implementation)
• Increased storage requirements
• Can become complex
• Different mindset than CRUD

24. Actor model, actors and Akka
Actors everywhere…

25. Actor model and Actors
Actor A Actor B
Actor C
Internal
State
Internal
State
Internal
State
Mailbox Mailbox
Mailbox
The actor model is a mathematical
model of concurrent computation
that treats "actors" as the universal
primitives of concurrent
computation
An actor is the primitive unit of
computation. It receives messages
and does something depending on
the type of message received
Actor System

26. Akka
• A library implementing the actor model on the JVM
• Multi-threaded behavior without atomics or locks
• Less complex than “traditional” concurrency solutions
• Supports both scale-up and scale-out
• Embraces failure
• Scala and Java APIs available

27. Akka and actors in 6 lines
import akka.actor.{Actor, ActorLogging}
class MyActor extends Actor with ActorLogging {
var myState = 0
override def receive: Receive = {
case x: UpdateState => myState = x.value
sender() ! StateUpdated
}
}

28. Akka persistence
• Enables stateful actors to save their internal state with
events
• Events are saved in a journal
• Community plugins for
• Cassandra, MongoDB, Kafka, DynamoDB & JDBC
• Optional snapshots can be triggered on-demand
• Persistence Query allows the migration of data to the read
side of the application

29. Persistent actor
sealed trait Cmd
sealed trait Event
case class UpdateHysteresis(hysteresis: Float) extends Cmd
case class HysteresisUpdated(hysteresis: Float) extends Event
case class DpState(id: Long = 1L, name: String = "my-name",
hysteresis: Float = 0.0F, events: Int = 0)

30. Persistent actor
class MyPersistentActor extends PersistentActor {
override def persistenceId = "my-unique-id“
override def receiveCommand = {
case UpdateHysteresis(hys) => persist(HysteresisUpdated(hys)) { evt =>
updateState(evt)
sender() ! HysteresisUpdatedAck
if (lastSequenceNr % snapshotInterval == 0 && lastSequenceNr != 0)
saveSnapshot(state)
//any other side effects
}
}

31. Persistent actor - recovery
override def receiveRecover = {
case x: HysteresisUpdated => updateState(x)
case SnapshotOffer(_, snapshot: DpState) => state = snapshot
}

32. Persistent actor - complete
sealed trait Cmd
sealed trait Event
case class UpdateHysteresis(hysteresis: Float) extends Cmd
case class HysteresisUpdated(hysteresis: Float) extends Event
case class DpState(id: Long = 1L, name: String = "my-name", hysteresis: Float = 0.0F, events: Int = 0)
class MyPersistentActor extends PersistentActor {
override def persistenceId = "my-unique-id"
var state = DpState()
def updateState(event: Event): Unit = {
case x: HysteresisUpdated => state = state.copy(hysteresis = x.hysteresis, events = state.events + 1)
}
val snapshotInterval = 100
override def receiveCommand = {
case UpdateHysteresis(hys) => persist(HysteresisUpdated(hys)) { evt =>
updateState(evt)
if (lastSequenceNr % snapshotInterval == 0 && lastSequenceNr != 0)
saveSnapshot(state)
//any other side effects
}
}
override def receiveRecover = {
case x: HysteresisUpdated => updateState(x)
case SnapshotOffer(_, snapshot: DpState) => state = snapshot
}
}

33. Persistent Query
• Read side of events
• Limited operations ( e.g. eventsBypersistenceId )
• Unless you have very limited query needs, mostly used to
populate a “query friendly” datastore (e.g. RDBMS)
• Stream and non-stream queries
• Stream allows to subscribe to events from specific
persistence Ids

34. Akka persistence considerations

35. Schema evolution
• As application evolves over time schema evolves as well
• It should be possible to read “old” events
• Transparently promote events to the latest version
• Most of cases can be handled at the serialization level

36. Schema evolution
• Rename fields
• Easy with IDL based serializers
• Field name is not part of the binary representation
• Add fields
• Handle in the de-serialization phase
• When replaying old events set a default value for the new field
• Delete fields (not common)
• Handle at de-serialization level (ignore)
• Remove events
• Serializer aware of events no longer needed and skip de-
serialization

37. Serialization format
• By default Akka uses Java serialization (not recommended)
• Alternatives
• Google Protobuf
• Apache Thrift
• Apache Avro
• Kryo
• JSON
• Your own (not recommended)

38. Serialization - configuration
actor {
serializers {
dpEvent = “myPackage.DpEventSerializer"
}
serialization-bindings {
“myPackage.HysteresisUpdatedEvent" = dpEvent
}
}
Define your serializer
Bind it to your events

39. Serialization - ScalaPB
• Protocol buffer compiler for Scala
• Supports proto2 and proto3
• It generates:
• Case classes
• Parsers
• Serializers

40. Serialization – protobuf
import "scalapb/scalapb.proto";
import "google/protobuf/wrappers.proto";
package ch.cern.ensdm;
message HysteresisUpdatedEvent {
option (scalapb.message).extends =
“myParckage.HysteresisUpdatedEvent";
float hysteresis = 1;
}

41. Fault tolerance

42. Actor supervision
• Akka uses separate flows to handle normal & recovery code
• Normal flow handles normal messages
• Recovery flow consists of actors that monitor actors in the normal
flow (supervisors)
• Akka enforces parental supervision
• 3 types of failures:
1. Systematic (e.g. programming error)
2. Transient (e.g. unavailable resources, database etc.)
3. Corrupt internal state
• Various recovery strategies

43. Backoff supervisor
• Recommended to use a backoff supervisor with persistence
• When a higher level actor restarts all children restart as well
• Depending the level, this can mean thousands of actors
restarting at the same time
• Backoff supervisor can vary the intervals and restart time

44. Backoff supervisor
val supervisor = BackoffSupervisor.props(
Backoff.onStop(
childProps,
childName = "myActor",
minBackoff = 3.seconds,
maxBackoff = 30.seconds,
randomFactor = 0.2
))

45. Testing Akka applications

46. Testing Actors
• Akka TestKit
• Allows asynchronous testing in a controlled environment
• akka-persistence-inmemory plugin
• Stores journal and snapshot messages in-memory
• In general testing procedure goes like:
1. Instantiate the actor to test
2. Send commands to the actor
3. Actor persists events in memory
4. Restart the actor
5. Check actor’s state

47. Akka persistence in production

48. Configuration System implementation
• Domain driven actor application
• Akka persistent actors represent the current state of
configuration
• Domain driven messages notify actors of configuration
changes
• Read side updated using persistence query

49. Topology
Configuration
Management
System
AMSRDBMS
DC DC DC DCOther systems
REST
API
DC
SVN
SCADA
CASSANDRA

50. Technologies
• Akka with Cassandra for the Persistence Journal
• Play framework
• DI using Guice
• WS Library
• Eureka
• Automatic discovery, registration of data concentrators
• Serialization
• Google protobuf

51. Actor hierarchy
DC
Supervisor
DC
Actor
DC
Actor
Device
Actor
Device
Actor Device
Actor
Device
Actor
Tag
Actor
Tag
Actor
Tag
Actor
Tag
Actor
Tag
Actor
Tag
Actor
Tag
Actor
Tag
Actor

52. Our experience - async

53. Our experience - Scala

54. Summary
• Asynchronous operations => difficult to test, debug
• Careful consideration of actor supervision strategy
• Event granularity => varies per use case
• Schema evolution => handle at serialization level
• Longer startup times => reduce snapshot interval
• If state does not fit in memory you need to use sharding
• Increased disk and memory usage

55. Questions?