Using Cassandra for event logging
Presentation

Using Cassandra for event logging
Ivan Burmistrov
SKB Kontur

December 9th...
Using Cassandra for event logging
EDI

We develop EDI (Electronic data interchange) system.
Using Cassandra for event logging
EDI

We develop EDI (Electronic data interchange) system.
Using Cassandra for event logging
EventLogger

interface IEventLogger
{
void Write ( Event [] events ) ;
Event [] Read ( E...
Using Cassandra for event logging
EventLogger

interface IEventLogger
{
void Write ( Event [] events ) ;
Event [] Read ( E...
Using Cassandra for event logging
EventLogger

interface IEventLogger
{
void Write ( Event [] events ) ;
Event [] Read ( E...
Using Cassandra for event logging
EventLogger

interface IEventLogger
{
void Write ( Event [] events ) ;
Event [] Read ( E...
Using Cassandra for event logging
EventLogger

interface IEventLogger
{
void Write ( Event [] events ) ;
Event [] Read ( E...
Using Cassandra for event logging
EventLogger

interface IEventLogger
{
void Write ( Event [] events ) ;
Event [] Read ( E...
Using Cassandra for event logging
EventLogger

interface IEventLogger
{
void Write ( Event [] events ) ;
Event [] Read ( E...
Using Cassandra for event logging
Obvious solution

Event
string EventId;
long Timestamp;
byte[] Content;
Using Cassandra for event logging
Obvious solution

Write(Event[] events)
Write columns for each event e:
e.Timestamp = No...
Using Cassandra for event logging
Obvious solution

Write(Event[] events)
Write columns for each event e:
e.Timestamp = No...
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Easy implementation.
Write action is very fast....
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Easy implementation.
Write action is very fast....
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Easy implementation.
Write action is very fast....
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Easy implementation.
Write action is very fast....
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Easy implementation.
Write action is very fast....
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Easy implementation.
Write action is very fast....
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Easy implementation.
Write action is very fast....
Using Cassandra for event logging
Our algorithm

Event
string EventId;
long Timestamp;
byte[] Content;
State State (Good o...
Using Cassandra for event logging
Our algorithm

Write(Event event)
Timestamp = NowTicks, State = Bad, write event.
Are th...
Using Cassandra for event logging
Our algorithm

Write(Event event)
Timestamp = NowTicks, State = Bad, write event.
Are th...
Using Cassandra for event logging
Our algorithm

Write(Event event)
Timestamp = NowTicks, State = Bad, write event.
Are th...
Using Cassandra for event logging
Our algorithm

Write(Event event)
Timestamp = NowTicks, State = Bad, write event.
Are th...
Using Cassandra for event logging
Our algorithm

Read
We can read events (same as obvious solution) while it’s Good.
Using Cassandra for event logging
Our algorithm

Read
We can read events (same as obvious solution) while it’s Good.
Using Cassandra for event logging
Our algorithm

TimeLag = ?
TimeLag do not depends on timeouts.
Less write intensity −→ l...
Using Cassandra for event logging
Our algorithm

TimeLag = ?
TimeLag do not depends on timeouts.
Less write intensity −→ l...
Using Cassandra for event logging
Our algorithm

TimeLag = ?
TimeLag do not depends on timeouts.
Less write intensity −→ l...
Using Cassandra for event logging
Improvements

Improvements
"Are there Good events with bigger Timestamp?" for O(1).
Queu...
Using Cassandra for event logging
Improvements

Improvements
"Are there Good events with bigger Timestamp?" for O(1).
Queu...
Using Cassandra for event logging
Improvements

Improvements
"Are there Good events with bigger Timestamp?" for O(1).
Queu...
Using Cassandra for event logging
Sharding

Simple sharding
We can divide all entities by independent parts.
For each part...
Using Cassandra for event logging
Sharding

Simple sharding
We can divide all entities by independent parts.
For each part...
Using Cassandra for event logging
Sharding

Sharding for reading
Timestamp = NowTicks, State = Bad, write events in
differe...
Using Cassandra for event logging
Sharding

Sharding for reading
Timestamp = NowTicks, State = Bad, write events in
differe...
Using Cassandra for event logging
Sharding

Sharding for reading
Timestamp = NowTicks, State = Bad, write events in
differe...
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Small size of time lag. For 3500 writes per sec...
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Small size of time lag. For 3500 writes per sec...
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Small size of time lag. For 3500 writes per sec...
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Small size of time lag. For 3500 writes per sec...
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Small size of time lag. For 3500 writes per sec...
Using Cassandra for event logging
Advantages and disadvantages

Advantages
Small size of time lag. For 3500 writes per sec...
Using Cassandra for event logging
Another applications

Another applications
Indexing. We can use event log as notification...
Using Cassandra for event logging
Another applications

Another applications
Indexing. We can use event log as notification...
Upcoming SlideShare
Loading in …5
×

Иван Бурмистров "Строго ориентированная последовательность временных событий". Выступление на Cassandra Сonf 2013

1,323 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,323
On SlideShare
0
From Embeds
0
Number of Embeds
386
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Иван Бурмистров "Строго ориентированная последовательность временных событий". Выступление на Cassandra Сonf 2013

  1. 1. Using Cassandra for event logging Presentation Using Cassandra for event logging Ivan Burmistrov SKB Kontur December 9th, 2013
  2. 2. Using Cassandra for event logging EDI We develop EDI (Electronic data interchange) system.
  3. 3. Using Cassandra for event logging EDI We develop EDI (Electronic data interchange) system.
  4. 4. Using Cassandra for event logging EventLogger interface IEventLogger { void Write ( Event [] events ) ; Event [] Read ( Event exclusiveEvent , int count ) ; } Availability for Read and Write. Fast Write. Read with minimal time lag. No events in past while we making Read. Same events chronology in any thread. For any thread events orders for write and read are equals.
  5. 5. Using Cassandra for event logging EventLogger interface IEventLogger { void Write ( Event [] events ) ; Event [] Read ( Event exclusiveEvent , int count ) ; } Availability for Read and Write. Fast Write. Read with minimal time lag. No events in past while we making Read. Same events chronology in any thread. For any thread events orders for write and read are equals.
  6. 6. Using Cassandra for event logging EventLogger interface IEventLogger { void Write ( Event [] events ) ; Event [] Read ( Event exclusiveEvent , int count ) ; } Availability for Read and Write. Fast Write. Read with minimal time lag. No events in past while we making Read. Same events chronology in any thread. For any thread events orders for write and read are equals.
  7. 7. Using Cassandra for event logging EventLogger interface IEventLogger { void Write ( Event [] events ) ; Event [] Read ( Event exclusiveEvent , int count ) ; } Availability for Read and Write. Fast Write. Read with minimal time lag. No events in past while we making Read. Same events chronology in any thread. For any thread events orders for write and read are equals.
  8. 8. Using Cassandra for event logging EventLogger interface IEventLogger { void Write ( Event [] events ) ; Event [] Read ( Event exclusiveEvent , int count ) ; } Availability for Read and Write. Fast Write. Read with minimal time lag. No events in past while we making Read. Same events chronology in any thread. For any thread events orders for write and read are equals.
  9. 9. Using Cassandra for event logging EventLogger interface IEventLogger { void Write ( Event [] events ) ; Event [] Read ( Event exclusiveEvent , int count ) ; } Availability for Read and Write. Fast Write. Read with minimal time lag. No events in past while we making Read. Same events chronology in any thread. For any thread events orders for write and read are equals.
  10. 10. Using Cassandra for event logging EventLogger interface IEventLogger { void Write ( Event [] events ) ; Event [] Read ( Event exclusiveEvent , int count ) ; } Availability for Read and Write. Fast Write. Read with minimal time lag. No events in past while we making Read. Same events chronology in any thread. For any thread events orders for write and read are equals.
  11. 11. Using Cassandra for event logging Obvious solution Event string EventId; long Timestamp; byte[] Content;
  12. 12. Using Cassandra for event logging Obvious solution Write(Event[] events) Write columns for each event e: e.Timestamp = NowTicks; RowKey = e.Timestamp / Partition; ColumnName = e.Timestamp + ":" + e.EventId; ColumnValue = e.Content; Read(Event exclusiveEvent, int count) Execute get_slice from exclusive column; Skip events with Timestamp > NowTicks - TimeLag.
  13. 13. Using Cassandra for event logging Obvious solution Write(Event[] events) Write columns for each event e: e.Timestamp = NowTicks; RowKey = e.Timestamp / Partition; ColumnName = e.Timestamp + ":" + e.EventId; ColumnValue = e.Content; Read(Event exclusiveEvent, int count) Execute get_slice from exclusive column; Skip events with Timestamp > NowTicks - TimeLag.
  14. 14. Using Cassandra for event logging Advantages and disadvantages Advantages Easy implementation. Write action is very fast. Disadvantages TimeLag = TimeLag > TimeLag > TimeLag > ? WriteTimeout + SyncDiff; ReadTimeout + SyncDiff; Cassandra worst write time; We can’t cache events in unstable zone, if we want to read events from it. I.e. more requests −→ more CPU for cassandra.
  15. 15. Using Cassandra for event logging Advantages and disadvantages Advantages Easy implementation. Write action is very fast. Disadvantages TimeLag = TimeLag > TimeLag > TimeLag > ? WriteTimeout + SyncDiff; ReadTimeout + SyncDiff; Cassandra worst write time; We can’t cache events in unstable zone, if we want to read events from it. I.e. more requests −→ more CPU for cassandra.
  16. 16. Using Cassandra for event logging Advantages and disadvantages Advantages Easy implementation. Write action is very fast. Disadvantages TimeLag = TimeLag > TimeLag > TimeLag > ? WriteTimeout + SyncDiff; ReadTimeout + SyncDiff; Cassandra worst write time; We can’t cache events in unstable zone, if we want to read events from it. I.e. more requests −→ more CPU for cassandra.
  17. 17. Using Cassandra for event logging Advantages and disadvantages Advantages Easy implementation. Write action is very fast. Disadvantages TimeLag = TimeLag > TimeLag > TimeLag > ? WriteTimeout + SyncDiff; ReadTimeout + SyncDiff; Cassandra worst write time; We can’t cache events in unstable zone, if we want to read events from it. I.e. more requests −→ more CPU for cassandra.
  18. 18. Using Cassandra for event logging Advantages and disadvantages Advantages Easy implementation. Write action is very fast. Disadvantages TimeLag = TimeLag > TimeLag > TimeLag > ? WriteTimeout + SyncDiff; ReadTimeout + SyncDiff; Cassandra worst write time; We can’t cache events in unstable zone, if we want to read events from it. I.e. more requests −→ more CPU for cassandra.
  19. 19. Using Cassandra for event logging Advantages and disadvantages Advantages Easy implementation. Write action is very fast. Disadvantages TimeLag = TimeLag > TimeLag > TimeLag > ? WriteTimeout + SyncDiff; ReadTimeout + SyncDiff; Cassandra worst write time; We can’t cache events in unstable zone, if we want to read events from it. I.e. more requests −→ more CPU for cassandra.
  20. 20. Using Cassandra for event logging Advantages and disadvantages Advantages Easy implementation. Write action is very fast. Disadvantages TimeLag = TimeLag > TimeLag > TimeLag > ? WriteTimeout + SyncDiff; ReadTimeout + SyncDiff; Cassandra worst write time; We can’t cache events in unstable zone, if we want to read events from it. I.e. more requests −→ more CPU for cassandra.
  21. 21. Using Cassandra for event logging Our algorithm Event string EventId; long Timestamp; byte[] Content; State State (Good or Bad);
  22. 22. Using Cassandra for event logging Our algorithm Write(Event event) Timestamp = NowTicks, State = Bad, write event. Are there Good events with bigger Timestamp? No? State = Good, rewrite event. Yes? Delete event and try again.
  23. 23. Using Cassandra for event logging Our algorithm Write(Event event) Timestamp = NowTicks, State = Bad, write event. Are there Good events with bigger Timestamp? No? State = Good, rewrite event. Yes? Delete event and try again.
  24. 24. Using Cassandra for event logging Our algorithm Write(Event event) Timestamp = NowTicks, State = Bad, write event. Are there Good events with bigger Timestamp? No? State = Good, rewrite event. Yes? Delete event and try again.
  25. 25. Using Cassandra for event logging Our algorithm Write(Event event) Timestamp = NowTicks, State = Bad, write event. Are there Good events with bigger Timestamp? No? State = Good, rewrite event. Yes? Delete event and try again.
  26. 26. Using Cassandra for event logging Our algorithm Read We can read events (same as obvious solution) while it’s Good.
  27. 27. Using Cassandra for event logging Our algorithm Read We can read events (same as obvious solution) while it’s Good.
  28. 28. Using Cassandra for event logging Our algorithm TimeLag = ? TimeLag do not depends on timeouts. Less write intensity −→ less TimeLag. Less cassandra write time −→ less TimeLag.
  29. 29. Using Cassandra for event logging Our algorithm TimeLag = ? TimeLag do not depends on timeouts. Less write intensity −→ less TimeLag. Less cassandra write time −→ less TimeLag.
  30. 30. Using Cassandra for event logging Our algorithm TimeLag = ? TimeLag do not depends on timeouts. Less write intensity −→ less TimeLag. Less cassandra write time −→ less TimeLag.
  31. 31. Using Cassandra for event logging Improvements Improvements "Are there Good events with bigger Timestamp?" for O(1). Queue write requests and process its by batches. We could remove the dependency from time sync, if NowTicks will be monotonous.
  32. 32. Using Cassandra for event logging Improvements Improvements "Are there Good events with bigger Timestamp?" for O(1). Queue write requests and process its by batches. We could remove the dependency from time sync, if NowTicks will be monotonous.
  33. 33. Using Cassandra for event logging Improvements Improvements "Are there Good events with bigger Timestamp?" for O(1). Queue write requests and process its by batches. We could remove the dependency from time sync, if NowTicks will be monotonous.
  34. 34. Using Cassandra for event logging Sharding Simple sharding We can divide all entities by independent parts. For each part we can use separate eventLogger.
  35. 35. Using Cassandra for event logging Sharding Simple sharding We can divide all entities by independent parts. For each part we can use separate eventLogger.
  36. 36. Using Cassandra for event logging Sharding Sharding for reading Timestamp = NowTicks, State = Bad, write events in different rows. Are there Good events with bigger Timestamp in any row? No? State = Good, rewrite events. Yes? Delete event and try again.
  37. 37. Using Cassandra for event logging Sharding Sharding for reading Timestamp = NowTicks, State = Bad, write events in different rows. Are there Good events with bigger Timestamp in any row? No? State = Good, rewrite events. Yes? Delete event and try again.
  38. 38. Using Cassandra for event logging Sharding Sharding for reading Timestamp = NowTicks, State = Bad, write events in different rows. Are there Good events with bigger Timestamp in any row? No? State = Good, rewrite events. Yes? Delete event and try again.
  39. 39. Using Cassandra for event logging Advantages and disadvantages Advantages Small size of time lag. For 3500 writes per sec worst time lag is 50ms and 10 events. More cacheable. Easy sharding. Can be independed from time sync. Disadvantages More slow writes. But still enough fast. Limit for writing threads.
  40. 40. Using Cassandra for event logging Advantages and disadvantages Advantages Small size of time lag. For 3500 writes per sec worst time lag is 50ms and 10 events. More cacheable. Easy sharding. Can be independed from time sync. Disadvantages More slow writes. But still enough fast. Limit for writing threads.
  41. 41. Using Cassandra for event logging Advantages and disadvantages Advantages Small size of time lag. For 3500 writes per sec worst time lag is 50ms and 10 events. More cacheable. Easy sharding. Can be independed from time sync. Disadvantages More slow writes. But still enough fast. Limit for writing threads.
  42. 42. Using Cassandra for event logging Advantages and disadvantages Advantages Small size of time lag. For 3500 writes per sec worst time lag is 50ms and 10 events. More cacheable. Easy sharding. Can be independed from time sync. Disadvantages More slow writes. But still enough fast. Limit for writing threads.
  43. 43. Using Cassandra for event logging Advantages and disadvantages Advantages Small size of time lag. For 3500 writes per sec worst time lag is 50ms and 10 events. More cacheable. Easy sharding. Can be independed from time sync. Disadvantages More slow writes. But still enough fast. Limit for writing threads.
  44. 44. Using Cassandra for event logging Advantages and disadvantages Advantages Small size of time lag. For 3500 writes per sec worst time lag is 50ms and 10 events. More cacheable. Easy sharding. Can be independed from time sync. Disadvantages More slow writes. But still enough fast. Limit for writing threads.
  45. 45. Using Cassandra for event logging Another applications Another applications Indexing. We can use event log as notification about change state of some objects −→ indexing. Cacheable is very useful here! Message queues.
  46. 46. Using Cassandra for event logging Another applications Another applications Indexing. We can use event log as notification about change state of some objects −→ indexing. Cacheable is very useful here! Message queues.

×