Your SlideShare is downloading. ×
0
Indexing Cassandra data in SQL-storage

Indexing Cassandra data in SQL-storage

Kurpilyansky Eugene
SKB Kontur
December 9t...
Indexing Cassandra data in SQL-storage
What do we want?

Suppose, we want to store objects of dierent types in
Cassandra.
...
Indexing Cassandra data in SQL-storage
What do we want?

Suppose, we want to store objects of dierent types in
Cassandra.
...
Indexing Cassandra data in SQL-storage
What do we want?

Suppose, we want to store objects of dierent types in
Cassandra.
...
Indexing Cassandra data in SQL-storage
What do we want?

Suppose, we want to store objects of dierent types in
Cassandra.
...
Indexing Cassandra data in SQL-storage
What do we want?

Suppose, we want to store objects of dierent types in
Cassandra.
...
Indexing Cassandra data in SQL-storage
Using native Cassandra indexes

We can use native Cassandra indexes.
Advantages
The...
Indexing Cassandra data in SQL-storage
Using native Cassandra indexes

We can use native Cassandra indexes.
Advantages
The...
Indexing Cassandra data in SQL-storage
Using native Cassandra indexes

We can use native Cassandra indexes.
Advantages
The...
Indexing Cassandra data in SQL-storage
Using native Cassandra indexes

We can use native Cassandra indexes.
Advantages
The...
Indexing Cassandra data in SQL-storage
Using native Cassandra indexes

We can use native Cassandra indexes.
Advantages
The...
Indexing Cassandra data in SQL-storage
Using synchronization with SQL-storage
Main idea

Main idea
Run IndexService applic...
Indexing Cassandra data in SQL-storage
Using synchronization with SQL-storage
Main idea

Main idea
Run IndexService applic...
Indexing Cassandra data in SQL-storage
Using synchronization with SQL-storage
Implementation of EventLog

Create event log...
Indexing Cassandra data in SQL-storage
Using synchronization with SQL-storage
Implementation of EventLog

Create event log...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of EventLog

Event
string E...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of EventLog

Event
string E...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of EventLog

Event
string E...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of EventLog

EventLog.AddEv...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of EventLog

EventLog.AddEv...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of EventLog

We should spli...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

IndexServi...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

IndexServi...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

IndexServi...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

IndexServi...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

IndexServi...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Periodic s...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Periodic s...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Periodic s...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Periodic s...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

ProcessEve...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

ProcessEve...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Event[] Pr...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Event[] Pr...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Event[] Pr...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 1
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 1
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 1
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 1
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 1
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 3
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 3
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 3
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 3
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 3
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Example 4
...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Event[] Pr...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Periodic s...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

What event...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

What event...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

What event...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

What event...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

What event...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

What event...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

What event...
Indexing Cassandra data in SQL-storage
Synchronizing SQL-storage with Cassandra
Implementation of IndexService

Executing ...
Indexing Cassandra data in SQL-storage

Advantages.
Scalability.
Availability.
Fault tolerance.
Sharding.
Indexing Cassandra data in SQL-storage

Advantages.
Scalability.
Availability.
Fault tolerance.
Sharding.
Indexing Cassandra data in SQL-storage
Questions

Thank you for your attention. Any questions?
Upcoming SlideShare
Loading in...5
×

Евгений Курпилянский "Индексирование поверх Cassandra". Выступление на Cassandra conf 2013

528

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
528
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Евгений Курпилянский "Индексирование поверх Cassandra". Выступление на Cassandra conf 2013 "

  1. 1. Indexing Cassandra data in SQL-storage Indexing Cassandra data in SQL-storage Kurpilyansky Eugene SKB Kontur December 9th, 2013
  2. 2. Indexing Cassandra data in SQL-storage What do we want? Suppose, we want to store objects of dierent types in Cassandra. Any object has a primary string key. Cassandra is well-suited for using it as key-value storage. But we usually want to search among all objects of same type by some criterion. Results of searching must be consistent and reect current state of database. How can we implement storage which satises these requirements?
  3. 3. Indexing Cassandra data in SQL-storage What do we want? Suppose, we want to store objects of dierent types in Cassandra. Any object has a primary string key. Cassandra is well-suited for using it as key-value storage. But we usually want to search among all objects of same type by some criterion. Results of searching must be consistent and reect current state of database. How can we implement storage which satises these requirements?
  4. 4. Indexing Cassandra data in SQL-storage What do we want? Suppose, we want to store objects of dierent types in Cassandra. Any object has a primary string key. Cassandra is well-suited for using it as key-value storage. But we usually want to search among all objects of same type by some criterion. Results of searching must be consistent and reect current state of database. How can we implement storage which satises these requirements?
  5. 5. Indexing Cassandra data in SQL-storage What do we want? Suppose, we want to store objects of dierent types in Cassandra. Any object has a primary string key. Cassandra is well-suited for using it as key-value storage. But we usually want to search among all objects of same type by some criterion. Results of searching must be consistent and reect current state of database. How can we implement storage which satises these requirements?
  6. 6. Indexing Cassandra data in SQL-storage What do we want? Suppose, we want to store objects of dierent types in Cassandra. Any object has a primary string key. Cassandra is well-suited for using it as key-value storage. But we usually want to search among all objects of same type by some criterion. Results of searching must be consistent and reect current state of database. How can we implement storage which satises these requirements?
  7. 7. Indexing Cassandra data in SQL-storage Using native Cassandra indexes We can use native Cassandra indexes. Advantages There is no need to support additional storage. Disadvantages Every custom query may require new CF-structure for eective searching. SQL-indexes are more ecient than Cassandra's indexes. There exist a lot of complex indexes (e.g. full-text search indexing).
  8. 8. Indexing Cassandra data in SQL-storage Using native Cassandra indexes We can use native Cassandra indexes. Advantages There is no need to support additional storage. Disadvantages Every custom query may require new CF-structure for eective searching. SQL-indexes are more ecient than Cassandra's indexes. There exist a lot of complex indexes (e.g. full-text search indexing).
  9. 9. Indexing Cassandra data in SQL-storage Using native Cassandra indexes We can use native Cassandra indexes. Advantages There is no need to support additional storage. Disadvantages Every custom query may require new CF-structure for eective searching. SQL-indexes are more ecient than Cassandra's indexes. There exist a lot of complex indexes (e.g. full-text search indexing).
  10. 10. Indexing Cassandra data in SQL-storage Using native Cassandra indexes We can use native Cassandra indexes. Advantages There is no need to support additional storage. Disadvantages Every custom query may require new CF-structure for eective searching. SQL-indexes are more ecient than Cassandra's indexes. There exist a lot of complex indexes (e.g. full-text search indexing).
  11. 11. Indexing Cassandra data in SQL-storage Using native Cassandra indexes We can use native Cassandra indexes. Advantages There is no need to support additional storage. Disadvantages Every custom query may require new CF-structure for eective searching. SQL-indexes are more ecient than Cassandra's indexes. There exist a lot of complex indexes (e.g. full-text search indexing).
  12. 12. Indexing Cassandra data in SQL-storage Using synchronization with SQL-storage Main idea Main idea Run IndexService application which is synchronizing data in SQL-storage with data in Cassandra (constantly, in background thread). To perform a search we should make a query to IndexService which will return the search result after nishing SQL-storage synchronization process.
  13. 13. Indexing Cassandra data in SQL-storage Using synchronization with SQL-storage Main idea Main idea Run IndexService application which is synchronizing data in SQL-storage with data in Cassandra (constantly, in background thread). To perform a search we should make a query to IndexService which will return the search result after nishing SQL-storage synchronization process.
  14. 14. Indexing Cassandra data in SQL-storage Using synchronization with SQL-storage Implementation of EventLog Create event log One event per one write-request or delete-request. Event log sorted by time of event.
  15. 15. Indexing Cassandra data in SQL-storage Using synchronization with SQL-storage Implementation of EventLog Create event log One event per one write-request or delete-request. Event log sorted by time of event.
  16. 16. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of EventLog Event string EventId; long Timestamp; string ObjectId; interface IEventLog void AddEvent(Event event); IEnumerableEvent GetEvents(long fromTicks); New implementation of IObjectStorage Before writing or deleting objects call method IEventLog.AddEvent.
  17. 17. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of EventLog Event string EventId; long Timestamp; string ObjectId; interface IEventLog void AddEvent(Event event); IEnumerableEvent GetEvents(long fromTicks); New implementation of IObjectStorage Before writing or deleting objects call method IEventLog.AddEvent.
  18. 18. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of EventLog Event string EventId; long Timestamp; string ObjectId; interface IEventLog void AddEvent(Event event); IEnumerableEvent GetEvents(long fromTicks); New implementation of IObjectStorage Before writing or deleting objects call method IEventLog.AddEvent.
  19. 19. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of EventLog EventLog.AddEvent(Event event) Create column: ColumnName = event.Timestamp + ':' + event.EventId ColumnValue = event EventLog.GetEvents(long fromTicks) Execute get_slice from exclusive column for one row.
  20. 20. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of EventLog EventLog.AddEvent(Event event) Create column: ColumnName = event.Timestamp + ':' + event.EventId ColumnValue = event EventLog.GetEvents(long fromTicks) Execute get_slice from exclusive column for one row. We should split all event log into rows using PartitionInterval to limit size of rows. PartitionInterval is some constant period of time (e.g. one hour, or six minutes).
  21. 21. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of EventLog We should split all event log into rows using PartitionInterval to limit size of rows. PartitionInterval is some constant period of time (e.g. one hour, or six minutes). EventLog.AddEvent(Event event) Create column: RowKey = event.Timestamp / PartitionInterval.Ticks ColumnName = event.Timestamp + ':' + event.EventId ColumnValue = event EventLog.GetEvents(long fromTicks) Execute get_slice from exclusive column for one or more rows.
  22. 22. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService IndexService It has a local SQL-storage (one storage per one service replica). There is one SQL-table per one type of object. There is one specic SQL-table for storing times of last synchronization for each type of object. There is one background thread per one type of object, which is reading event log and updating SQL-storage. For executing incoming SQL-query, we can use data from SQL-storage and a little range of events.
  23. 23. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService IndexService It has a local SQL-storage (one storage per one service replica). There is one SQL-table per one type of object. There is one specic SQL-table for storing times of last synchronization for each type of object. There is one background thread per one type of object, which is reading event log and updating SQL-storage. For executing incoming SQL-query, we can use data from SQL-storage and a little range of events.
  24. 24. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService IndexService It has a local SQL-storage (one storage per one service replica). There is one SQL-table per one type of object. There is one specic SQL-table for storing times of last synchronization for each type of object. There is one background thread per one type of object, which is reading event log and updating SQL-storage. For executing incoming SQL-query, we can use data from SQL-storage and a little range of events.
  25. 25. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService IndexService It has a local SQL-storage (one storage per one service replica). There is one SQL-table per one type of object. There is one specic SQL-table for storing times of last synchronization for each type of object. There is one background thread per one type of object, which is reading event log and updating SQL-storage. For executing incoming SQL-query, we can use data from SQL-storage and a little range of events.
  26. 26. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService IndexService It has a local SQL-storage (one storage per one service replica). There is one SQL-table per one type of object. There is one specic SQL-table for storing times of last synchronization for each type of object. There is one background thread per one type of object, which is reading event log and updating SQL-storage. For executing incoming SQL-query, we can use data from SQL-storage and a little range of events.
  27. 27. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Periodic synchronization action Set startSynchronizationTime = NowTicks. Find all events which should be processed. Process these events: update SQL-storage and keep unprocessed events (they should be processed on the next iteration). Update time of last synchronization to startSynchronizationTime in SQL-storage.
  28. 28. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Periodic synchronization action Set startSynchronizationTime = NowTicks. Find all events which should be processed. Process these events: update SQL-storage and keep unprocessed events (they should be processed on the next iteration). Update time of last synchronization to startSynchronizationTime in SQL-storage.
  29. 29. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Periodic synchronization action Set startSynchronizationTime = NowTicks. Find all events which should be processed. Process these events: update SQL-storage and keep unprocessed events (they should be processed on the next iteration). Update time of last synchronization to startSynchronizationTime in SQL-storage.
  30. 30. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Periodic synchronization action Set startSynchronizationTime = NowTicks. Find all events which should be processed. Process these events: update SQL-storage and keep unprocessed events (they should be processed on the next iteration). Update time of last synchronization to startSynchronizationTime in SQL-storage.
  31. 31. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService ProcessEvents(Event[] events) This function actualizes values of related objects in SQL-storage. Remember, that we update object after creating an event. So, we can not process some of events at the moment, because correspoding object isn't updated yet.
  32. 32. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService ProcessEvents(Event[] events) This function actualizes values of related objects in SQL-storage. Remember, that we update object after creating an event. So, we can not process some of events at the moment, because correspoding object isn't updated yet.
  33. 33. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Event[] ProcessEvents(Event[] events) This function actualizes values of related objects in SQL-storage and returns events, which have not been processed. How will this function be implemented? For every event we should analyze corresponding objects from both Cassandra and SQL-storage.
  34. 34. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Event[] ProcessEvents(Event[] events) This function actualizes values of related objects in SQL-storage and returns events, which have not been processed. How will this function be implemented? For every event we should analyze corresponding objects from both Cassandra and SQL-storage.
  35. 35. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Event[] ProcessEvents(Event[] events) This function actualizes values of related objects in SQL-storage and returns events, which have not been processed. How will this function be implemented? For every event we should analyze corresponding objects from both Cassandra and SQL-storage.
  36. 36. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 1 event = {Timestamp: 2008} cassObj = {Timestamp: 2008, School: 'USU'} sqlObj = {Timestamp: 2005, School: 'AESÑ USU'} What should we do?
  37. 37. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 1 event = {Timestamp: 2008} cassObj = {Timestamp: 2008, School: 'USU'} sqlObj = {Timestamp: 2005, School: 'AESÑ USU'} Write cassObj in SQL-storage and mark event as processed.
  38. 38. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 1 event = {Timestamp: 2008} cassObj = {Timestamp: 2008, School: 'USU'} sqlObj = {Timestamp: 2005, School: 'AESÑ USU'} Write cassObj in SQL-storage and mark event as processed. Example 2 event = {Timestamp: 2012} cassObj = {Timestamp: 2008, School: 'USU'} sqlObj = {Timestamp: 2005, School: 'AESÑ USU'} What should we do?
  39. 39. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 1 event = {Timestamp: 2008} cassObj = {Timestamp: 2008, School: 'USU'} sqlObj = {Timestamp: 2005, School: 'AESÑ USU'} Write cassObj in SQL-storage and mark event as processed. Example 2 event = {Timestamp: 2012} cassObj = {Timestamp: 2008, School: 'USU'} sqlObj = {Timestamp: 2005, School: 'AESÑ USU'} Timestamp of event is greater than timestamp of cassObj. Probably, it needs to wait for updating of object.
  40. 40. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 1 event = {Timestamp: 2008} cassObj = {Timestamp: 2008, School: 'USU'} sqlObj = {Timestamp: 2005, School: 'AESÑ USU'} Write cassObj in SQL-storage and mark event as processed. Example 2 event = {Timestamp: 2012} cassObj = {Timestamp: 2008, School: 'USU'} sqlObj = {Timestamp: 2005, School: 'AESÑ USU'} Timestamp of event is greater than timestamp of cassObj. Probably, it needs to wait for updating of object. Write cassObj in SQL-storage and mark event as unprocessed.
  41. 41. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 3 event = {Timestamp: 1997} cassObj is missing sqlObj is missing What should we do?
  42. 42. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 3 event = {Timestamp: 1997} cassObj is missing sqlObj is missing Probably, that event corresponds to the creation of object.
  43. 43. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 3 event = {Timestamp: 1997} cassObj is missing sqlObj is missing Probably, that event corresponds to the creation of object. Mark event as unprocessed.
  44. 44. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 3 event = {Timestamp: 1997} cassObj is missing sqlObj is missing Probably, that event corresponds to the creation of object. Mark event as unprocessed. Example 4 event = {Timestamp: 2017} cassObj is missing sqlObj = {Timestamp: 2012, School: 'UFU'} What should we do?
  45. 45. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 3 event = {Timestamp: 1997} cassObj is missing sqlObj is missing Probably, that event corresponds to the creation of object. Mark event as unprocessed. Example 4 event = {Timestamp: 2017} cassObj is missing sqlObj = {Timestamp: 2012, School: 'UFU'} Two cases are possible: 1 That event corresponds to the deletion of object. 2 That event corresponds to the creation of object. sqlObj is not missing, because there were two operationsin a row: delete and create.
  46. 46. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Example 4 event = {Timestamp: 2017} cassObj is missing sqlObj = {Timestamp: 2012, School: 'UFU'} Two cases are possible: 1 That event corresponds to the deletion of object. 2 That event corresponds to the creation of object. sqlObj is not missing, because there were two operationsin a row: delete and create. Delete sqlObj from SQL-storage and mark event as unprocessed.
  47. 47. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Event[] ProcessEvents(Event[] events) Read objects, which occured in these events, from Cassandra and SQL-storage (some of them can be missing). For each (event, cassObj, sqlObj) do If cassObj is not missing cassObj in SQL-storage event.Timestamp = cassObj.Timestamp Save If then mark else mark event as processed; event as unprocessed. else (i.e. cassObj is missing) sqlObj from SQL-storage event as unprocessed. Delete Mark if it's not missing. Return events which has been marked as unprocessed.
  48. 48. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Periodic synchronization action Set startSynchronizationTime = NowTicks. Find all events which should be processed. Process these events: update SQL-storage and keep unprocessed events (they should be processed on the next iteration). Update time of last synchronization to startSynchronizationTime in SQL-storage.
  49. 49. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService What events should we use as arguments in ProcessEvents function? Of course, all unprocessed events from previous iteration. Also all new events, i.e. IEventLog.GetEvents(fromTicks). What is fromTicks? fromTicks = lastSynchronizationTime? No. Unfortunately, any operation with Cassandra can be executed for a long time. This time is limited by writeTimeout = attemptsCount · connectionTimeout. We should make undertow back, otherwise we can lose some events. fromTicks = lastSynchronizationTime - writeTimeout
  50. 50. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService What events should we use as arguments in ProcessEvents function? Of course, all unprocessed events from previous iteration. Also all new events, i.e. IEventLog.GetEvents(fromTicks). What is fromTicks? fromTicks = lastSynchronizationTime? No. Unfortunately, any operation with Cassandra can be executed for a long time. This time is limited by writeTimeout = attemptsCount · connectionTimeout. We should make undertow back, otherwise we can lose some events. fromTicks = lastSynchronizationTime - writeTimeout
  51. 51. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService What events should we use as arguments in ProcessEvents function? Of course, all unprocessed events from previous iteration. Also all new events, i.e. IEventLog.GetEvents(fromTicks). What is fromTicks? fromTicks = lastSynchronizationTime? No. Unfortunately, any operation with Cassandra can be executed for a long time. This time is limited by writeTimeout = attemptsCount · connectionTimeout. We should make undertow back, otherwise we can lose some events. fromTicks = lastSynchronizationTime - writeTimeout
  52. 52. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService What events should we use as arguments in ProcessEvents function? Of course, all unprocessed events from previous iteration. Also all new events, i.e. IEventLog.GetEvents(fromTicks). What is fromTicks? fromTicks = lastSynchronizationTime? No. Unfortunately, any operation with Cassandra can be executed for a long time. This time is limited by writeTimeout = attemptsCount · connectionTimeout. We should make undertow back, otherwise we can lose some events. fromTicks = lastSynchronizationTime - writeTimeout
  53. 53. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService What events should we use as arguments in ProcessEvents function? Of course, all unprocessed events from previous iteration. Also all new events, i.e. IEventLog.GetEvents(fromTicks). What is fromTicks? fromTicks = lastSynchronizationTime? No. Unfortunately, any operation with Cassandra can be executed for a long time. This time is limited by writeTimeout = attemptsCount · connectionTimeout. We should make undertow back, otherwise we can lose some events. fromTicks = lastSynchronizationTime - writeTimeout
  54. 54. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService What events should we use as arguments in ProcessEvents function? Of course, all unprocessed events from previous iteration. Also all new events, i.e. IEventLog.GetEvents(fromTicks). What is fromTicks? fromTicks = lastSynchronizationTime? No. Unfortunately, any operation with Cassandra can be executed for a long time. This time is limited by writeTimeout = attemptsCount · connectionTimeout. We should make undertow back, otherwise we can lose some events. fromTicks = lastSynchronizationTime - writeTimeout
  55. 55. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService What events should we use as arguments in ProcessEvents function? Of course, all unprocessed events from previous iteration. Also all new events, i.e. IEventLog.GetEvents(fromTicks). What is fromTicks? fromTicks = lastSynchronizationTime? No. Unfortunately, any operation with Cassandra can be executed for a long time. This time is limited by writeTimeout = attemptsCount · connectionTimeout. We should make undertow back, otherwise we can lose some events. fromTicks = lastSynchronizationTime - writeTimeout
  56. 56. Indexing Cassandra data in SQL-storage Synchronizing SQL-storage with Cassandra Implementation of IndexService Executing search request
  57. 57. Indexing Cassandra data in SQL-storage Advantages. Scalability. Availability. Fault tolerance. Sharding.
  58. 58. Indexing Cassandra data in SQL-storage Advantages. Scalability. Availability. Fault tolerance. Sharding.
  59. 59. Indexing Cassandra data in SQL-storage Questions Thank you for your attention. Any questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×